Log file Viewer
#mongodb logs for Wednesday the 20th of April, 2016
[02:59:23] <Ryzzan> any good tutorial to definitely learn how to deal with aggregation after a populate result?
[03:29:20] <Progster> I'm trying to create a web app using the MEAN stack, with mongoose for ODM. It seems that an ODM isn't really an ORM, or at least not like I've used them before. With ORMs, I could translate from domain models to DB and back, but mongoose doesn't really do that type of mapping. It seems that I have to hand-roll a set of domain models that I then write translators from mongoose models to. Or does developing on the MEAN stack presuppose that all
[03:29:20] <Progster> of your models, the way they're shown in views, are the way they should be saved in the DB? That would be a big issue with separation of concerns
[05:36:53] <Skaag> I have a situation where I'm replicating data from PostgreSQL to a MongoDB collection. I know that in the PG table, the 'id' field is unique, since PG handles that just fine
[05:37:47] <Skaag> And I see in the mongodb docs that: "The field name _id is reserved for use as a primary key; its value must be unique in the collection, is immutable, and may be of any type other than an array"
[07:23:26] <mk1> hello there. I'm trying to export a collection from my db and would like to EXCLUDE a single field. mongoexport seems to only provide a projection (i.e. include fields). Any ideas without resorting to a scripting language?
[07:27:36] <kurushiyama> mk1: Projecttion can be used to include and exclude fields. http://pastebin.com/YAHa8S5Z
[07:33:31] <kurushiyama> Well, then denote the fields. Or do an aggregation, with an according $project and an $out stage and dump that collection.
[07:36:27] <mk1> any pointers? I'm sorry asking dumb questions. I'm usually not in the db department of the project
[07:45:06] <mk1> I figured. Well, it's only a few so that's okay. hopefully there won't be any changes in the next version. thanks anyway
[09:05:49] <siyb> Derick: as long as it's official, I am ok with it, that's the channels ops call, personally I dislike logging services like that, but it's not my call to make ;)
[09:19:12] <siyb> does morphia understand JPA annotation, i.e. can I use the JPA annotation equivalents instead of the mongodb ones?
[09:47:10] <kurushiyama> siyb: Actually, I liked it. It is only awful when you try to use it as a RDBMS replacement ;)
[09:48:51] <siyb> kurushiyama: the annotation API is aweful, you have very little control over what you are doing, because the annotations do not support getters where it would be required, in addition, they could have used the JPA standard where applicable, so that you do not have to write adapter objects for your pojo
[09:50:59] <siyb> kurushiyama: the @Id annotation cannot be used on a getter for instance, JPA @Entity is not recognized, etc. I understand that some things are necessary design decisions, but the annotations seem broken to me
[09:52:07] <kurushiyama> siyb: In case you try to use MongoDB as a drop in replacement (which is what I conclude from your statements), you are preparing yourself to get into trouble.
[09:53:14] <kurushiyama> siyb: Taking an SQL approach to data modelling (Identify entities, their properties and relations to each other) is a pretty sure way of breaking sclability and performance.
[09:53:42] <siyb> kurushiyama: haha, nope that's not what i am doing, but i am actually contemplating switching to couchbase later on (actually my sysop is doing that)
[09:54:30] <kurushiyama> siyb: Well, I came from the Java world myself and had to make the same decision. tbh, there are very few things CouchDB does better than MongoDB.
[09:55:48] <siyb> i have been using mongodb for a few years now, in general i am very happy but there are a few quirks that annoy me and our sysop is not happy about the possibilities of mongodb when it comes to the ops part of things
[09:56:17] <siyb> soooo, i am using mongodb now but i am planning ahead for a possible switch in the future
[09:57:23] <siyb> kurushiyama: for me it's some of the broken API, (don't use map/reduce, etc), for our sysop, it's many fear of dataloss.
[09:58:34] <kurushiyama> siyb: People ranting about loosing data with MongoDB _always_ used a write concern of 0 or 1.
[09:58:36] <siyb> kurushiyama: he has had to deal with dataloss due to messed up sharding before, afaik, it's still an issue, even if picking the correct WC
[10:01:33] <kurushiyama> Well, I chose a different approach. Unix-inspired, I chose on document DB and learned it to a point where I understood it.
[10:12:10] <jokke> i've read into schema design for time-series data and i'd be happy to receive any sources for comparisons between MMAPv1 and WiredTiger regarding these schema design approaches. As in: since WiredTiger, what has changed?
[10:13:00] <jokke> the talks and articles i saw/read were from 2014 which was before WiredTiger afair.
[10:51:39] <kurushiyama> jokke: The default compression of the data files enhances hardware utilization and tends to give you more bang for the buck.
[10:52:22] <kurushiyama> jokke: Bottom line: Unless you have _very_ good reasons to do so, use WT instead of MMAPv1.
[10:54:46] <kurushiyama> Derick: I tend to refer to https://objectrocket.com/blog/company/mongodb-wiredtiger but this is a bit shallow, imho.
[10:59:56] <kurushiyama> Well, I'll have to google that. My personal experience is similar or even better, though. In one project, we achieved order of magnitudes better insert performance.
[12:08:11] <owg1> I#m looking into benchmarking mongodb on different filesystems, do you think that tool would be a good place to start? The issue I am facing is creating a test that will actually use the disk not just RAM for reads.
[14:05:19] <quattro_> is there any difference in performance between $inc and $set? I'm keeping a collection with average memory usage (in bytes) and using $inc the values are getting very large
[15:15:49] <Ryzzan> ok... after populating, how am i supposed to aggregate ($group) the results of this population?
[15:23:28] <mkjgore> hey folks, so I seem to be having an issue where my replica sets in my shard cluster are always (based on iotop) reporting constant disk reads of ~800KB
[15:24:33] <zylo4747> We are running MongoDB v2.6.7. For the past week we have seen a stack trace occur on different nodes of a single 3 node replica set a few times. I don't know how to analyze it to find the issue. Can someone guide me?
[15:24:36] <mkjgore> is there a way to see what's been going on in the mongo instance to cause all this?
[15:28:20] <mkjgore> I've tried to turn on profiling via "db.setProfilingLevel(2)" for a short while (30 seconds) to capture "normal" bad behavior up until "very bad" behavior then shut it off with "db.setProfilingLevel(0)" but to no effect
[15:28:44] <mkjgore> I observed the behavior with shell tools but whenI went to admin.system.profile the collection was empty
[15:44:31] <deathanchor> turning it on your admin db is pointless unless you have issues with that db
[15:46:40] <deathanchor> mkjgore: use suspectDBname; db.setProfileLevel(2); db.system.profile.find();
[15:47:39] <deathanchor> on secondary the system.profile collection gets removed when you turn off the profiler and attept to drop it and drop the connection from your shell.
[15:56:06] <owg1> StephenLynx: Yeah it is looking like the fastest at the moment, we wanted to investigate how ZFS performs with various options though, so that is the main focus.
[15:57:17] <bros> I am running 10 threads that are querying a subdocument (an array of objects inside a doucment)
[15:57:38] <bros> I'm noticing 200% CPU usage on a two core machine. Is it due to the fact that I am using subdocuments?
[16:12:29] <zylo4747> Can someone look at this stack trace I'm getting in production? (MongoDB v2.6.7, Win2012). http://pastebin.com/U1qwxFRU
[16:12:51] <zylo4747> We've gotten it twice this week already and we've had it a few times in the past. I don't know if there's anything I can do to identify / prevent the issue.
[16:13:12] <zylo4747> When it happens, the replica set member crashes and the service has to be restarted
[16:14:21] <bros> https://www.airpair.com/postgresql/posts/sql-vs-nosql-ko-postgres-vs-mongo Does MongoDB still lose data like mentioned in this article?
[16:16:32] <mkjgore> @deathanchor: is there a way to get a server wide analysis? the wall we're hitting is that there are ~100 dbs on this server though they're all mostly tiny making per db performance an issues
[16:29:47] <deathanchor> mkjgore: you could change the server startup option: https://docs.mongodb.org/manual/reference/configuration-options/#operationProfiling.slowOpThresholdMs
[16:56:37] <owg1> cheeser: I'm using this benchmarking tool: https://github.com/tmcallaghan/sysbench-mongodb
[16:56:56] <owg1> And I want to make sure that the data is actually get synced to teh disk, but not sure what teh best way of checkgn that is
[16:57:59] <cheeser> https://github.com/tmcallaghan/sysbench-mongodb/blob/master/src/jmongosysbenchexecute.java#L111
[16:58:59] <cheeser> everything gets fsync'd eventually. that's how things get to disk. the fsync WC level simply waits for that to happen before returning.
[17:00:26] <owg1> Annoyingly I've ran all my tests with the default SAFE option, I shall re-run one with FSYNC_SAFE and hope there isn't a massive difference!
[17:09:36] <kurushiyama> I would like to add that calling fsync in write concern reduces the interval in which the data is synced to disk, iirc. owg1 A write concern > 1 usually is enough – it rarely happens that 2 servers fail the same time. If you want to be really sure, set wc = ceil(members/2)
[18:11:30] <mkjgore> @deathanchor: It seems that the performance has nothing to do with ops coming from without but from some sort of internal ops. is there anything like this that we can look into?
[18:12:18] <mkjgore> I'm wondering if some "housekeeping" functionality might have gotten borked after we ran the restore operation
[18:26:11] <mikebommarito> How come I cannot do a mongorestore on 3.2 with a .gz exported from another 3.2 instance/
[18:26:48] <mikebommarito> here is the command that is failing str8 from the docs: sudo mongorestore --gzip --archive=NAMEOFARCHIVE.gz -db NAMEOFNEWDATABASE
[18:28:25] <mikebommarito> ya when I run it I get this but doesn’t create DB…guess I’ll do the old mongodump / tar / transfer to new server untar / restore
[18:28:44] <mikebommarito> here is what the output is tho: 2016-04-20T18:02:53.018+0000 creating intents for archive
[18:34:42] <mikebommarito> Ya I am getting crazy output now I think it has to do with the .gz with all the collections .gz’ed too
[18:35:35] <mikebommarito> 2016-04-20T18:26:11.530+0000 ending restore routine with id=1, no more work to do
[18:36:23] <mikebommarito> I’ll go the old way I have been doing it…..just glad I have a place to ping people that are way smarter than me.
[20:32:26] <Doyle> Hey. In the mongodb docs for mongodump --oplog, it says "no effect when running mongodump against a mongos instance to dump the entire contents of a sharded cluster" Will --oplog work when a collection isn't sharded?
[22:45:32] <kurushiyama> Doyle: access via mongos == no effect, regardless of sharding status of collection. --oplog only has an effect when run against a replset directly.