Log file Viewer
#mongodb logs for Tuesday the 28th of April, 2015
[00:23:53] <GothAlice> T_T Today was the first day I've ever incinerated security tokens within minutes of receiving the package. All because of this one conditional: if (!pw1.isValidated() && pw1_modes[PW1_MODE_NO81]) ISOException.throwIt(…)
[01:10:22] <bros> How can I query for only the length of an array in a subdocument, not the actual contents?
[01:13:14] <GothAlice> Two methods: map/reduce, or an aggregation pipeline involving $unwind and $group.
[01:23:09] <GothAlice> Unwind emits one document per array element to the next pipeline stage, duplicating the other fields in the document.
[01:23:28] <GothAlice> I.e. {foo: 1, bar: [2, 3]} -> $unwind on bar -> {foo: 1, bar: 2}, {foo: 1, bar: 3}
[01:26:27] <bros> What do I do if I want to query certain fields of the documents I also want to aggregate?
[01:27:55] <GothAlice> bros: http://docs.mongodb.org/manual/core/aggregation-introduction/ < the documentation is quite in-depth, and there are numerous tutorials available on Google.
[01:32:23] <bros> I think the logic I want to achieve is too complicated to fit into a match/group/project sort of aggregation.
[01:32:24] <bros> loop through all orders, according which user the order, its scans, misscans, and time elapsed belong to
[01:32:24] <bros> if the order belongs to a batch, divide the time elapsed by the number of orders in the batch
[01:38:03] <GothAlice> Step 1: get it done using plain application logic. Step 2: benchmark it and find out where it's slow as a dog. Step 3: optimize the slowest bits. If attempting to refactor into a single monolithic map/reduce or aggregate, you have a "known good" process to compare against.
[01:40:31] <bros> I got it done in plain application logic. It falls apart in the fact that it takes 3 seconds over 200 records.
[02:11:13] <JonGorrono> Anyone know a way to disable auto-creation of db' s in mongoshell? .... erytime I mistype a use <db name>... erytime
[03:57:27] <dunkel2> is it possible to sort by a field string with -number suffix? lie mystring-1 mystring-2 mystring-10
[05:10:07] <dreamdust> People say skip() is not performant because mongo has to walk the whole collection… but isn't that true of any find query ? What I mean is, is I have a find() query with some constraint based on an indexed field, is adding skip() to it any *less* performant?
[05:12:20] <dreamdust> IE find({ some: 'indexedField'}) and find({ some: 'indexedField'}).skip(0) should have the same performance, correct?
[06:06:26] <morenoh149> dreamdust: you should test it but I'd image first it does the filtering then the skipping. Wouldn't make much sense if it worked otherwise, unless mongodb intends for you to not use it that way
[06:06:41] <morenoh149> this is pretty in depth http://docs.mongodb.org/manual/reference/method/cursor.skip/
[10:07:25] <Johnade> i'm using nodejs with express, i'm trying to render an image (jpg) from db to my browser
[11:29:48] <newbsduser> spliting data and using 4 mognodb instance on the same machine VS using all data with single mongodb instance on the same machine
[12:16:42] <m4th> does anyone know if there is a clean way of doing a tailf-like on a collection that is not capped (i.e tailable cursors are not an option) ?
[12:17:13] <m4th> is there an API like a replication one, that could inform a client that the collection has been updated
[12:17:51] <m4th> deathanchor: I won't loose entries with that ? If there are a lot of insertions between my queries ?
[12:19:22] <deathanchor> hmm.. looks like if you want guarantees, you need to add a date/time stamp and index by that so you can get a quick lookup
[12:22:09] <deathanchor> yeah, but I don't think you can get cursor that will just keep polling for new entries.
[12:22:40] <StephenLynx> because I just read you can make a tail for capped collections, and capped collections will make room for new documents when needed.
[12:22:50] <m4th> StephenLynx: I have disk space, I want the collection to store as much as data as possible, and I can lvresize if ever I need it
[12:24:12] <StephenLynx> hm, it seems it isn't something directly related to mongo, but to filesystems.
[12:24:39] <StephenLynx> ok, so you want to be able to use this to resize the volume which holds your database?
[12:27:35] <m4th> cheeser: you spoke of the time component that's used to build the ObjectId I guess (didn't spot that at first sight)
[12:49:32] <StephenLynx> 1: I suggest upgrading to io.js. 2: I have done it with gridfs, will link in a moment
[12:54:05] <StephenLynx> but because I needed it on disk for resizing with imagemagick, I changed to writing from disk to gridfs.
[12:55:02] <Johnade> ok for the saving, but how render it with node or io ? When i tried to rendre,i have the buffer only [32,32,34,154 etc...]
[13:02:18] <StephenLynx> instead of reading, storing to RAM then writing to the response, you just stream it directly.
[13:04:11] <Johnade> StephenLynx: i'm making a craiglist kind of website, do you think that mongoDB is a good choice for this
[13:08:30] <StephenLynx> you could generate html on the back-end if you want to support non-js users.
[13:11:26] <Johnade> yes i used it before it's cool. but i will to work with a lot of json, but it will be simple, no inscription at all, just posting and reading list of ad
[13:29:36] <StephenLynx> from all problems in software development, that one is the easiest one to solve.
[13:37:00] <StephenLynx> afaik, wiredtiger is not the default db and several people wouldn't recommend it for production.
[13:49:19] <bros> I have a collection called orders with subdocuments called: log, scans, misscans, and shipments. I want to extract the size of each of these subdocuments over a set of data that's about ~300 entries long without overloading my servers.
[13:55:06] <jigax> hi everyone. was wondering if someone could kindly help me with this gist https://gist.github.com/fpena06/206d67acf4a5b4d3cd20
[13:57:46] <griffindy> does anyone here have experience running mongo with many many collections in one db? i'm talking about hundreds of thousands
[13:58:03] <StephenLynx> GothAlice how come I never saw OVH on your list? They have great prices from what I noticed both for small stuff https://www.ovh.com/us/vps/vps-classic.xml and dedicated are pretty great too.
[13:59:52] <StephenLynx> griffindy having hundreds of thousands is only possible if you have dynamic collection creation, which is a HUGE no no.
[14:00:15] <jigax> trying to run a async.waterfall to query some data then update it. but when i save it i dont see the upate.
[14:00:46] <griffindy> StephenLynx out of curiosity, what's wrong with dynamic collection creation (not trolling, I just haven't found anything online)
[14:01:29] <jigax> greyTEO: trying to run a async.waterfall to query some data then update it. but when i save it i dont see the upate.
[14:01:36] <griffindy> maintain because mongo cannot keep up with creation? or there's some sort of garbage collection that's expensive?
[14:03:39] <StephenLynx> because it is much harder to query, because it would require joins to get the whole data.
[14:03:49] <greyTEO> jigax, it is hard to say what could be wrong. are you sure customer lines is not empty?
[14:04:41] <jigax> greyTEO: no they are not. i actually did a console log after each update to see the lines and update was made
[14:09:03] <griffindy> StephenLynx one of my colleagues was under the impression it would offer better performance for indices fitting in memory
[14:10:04] <griffindy> the docs also say "Generally, having a large number of collections has no significant performance penalty and results in very good performance. Distinct collections are very important for high-throughput batch processing."
[14:13:23] <mrmccrac> maybe but if you're having to constantly query from all these multiple collections wont it be the same net result?
[14:15:50] <griffindy> i think my colleague's thoughts are if one collection is much hotter than the others, it would be easier to hold just that one index in memory, rather than a subset of the entire index, if that makes sense
[14:16:13] <griffindy> I'm also not 100% certain how mongo behaves with its indices when they can't fit in memory
[14:16:35] <GothAlice> StephenLynx: No joins needed if you aren't silly about it (i.e. client segregation means one _doesn't_ want data pollution) and it's automatically handled by the app; a new client signs up, it populates the indexes in a new collection.
[14:17:59] <griffindy> although at the end of the day, there seems like a very hard cap of ~3m namespaces
[14:19:58] <GothAlice> Though because most collections come with an _id index by default, that doesn't equate to 3.338 million collections.
[14:22:19] <griffindy> my understanding is that a collection counts as a namespace, and each index on that collection is also a namespace
[14:37:34] <jmeister> is there a way to query the date/time from a mongodb server? Just to use it as a sort of a centralized time server
[14:39:31] <jmeister> @mrmccrac we're running several instances on AWS and they all have ~2-5 seconds difference between them
[14:42:41] <jigax> guys i've honestly been googling all night to try and figure this one out and can't find much. I have a async waterfall which queries a line price and is supposed to update the line price. I can see the change in the first asymc call when doing a console.log, however when saving the document in the second call the document is being saved without the price that was previously updated.
[14:52:17] <jr3> is there a convention to follow on schemas? like I prefer to fully qualify an id so: new CarSchema({ carId: Number }) vs new CarSchema({ id: Number})
[15:06:58] <jigax> can someone please help me figure out why this gist ins't working as expected https://gist.github.com/fpena06/206d67acf4a5b4d3cd20 thanks
[15:10:17] <snowcode> Is possible to use an expression inside the $group aggregate operation. Let me explain. I've a group operator which count a set of field by day (number of events per day): { $group: { _id: { $dayOfYear: "$time"}, eventscount: { $sum: 1 } }} Now I would to get a detailed info: number of events in this day with a property x > value and number of events in this day with a property y > value2
[15:18:30] <snowcode> "Expression $gte takes exactly 2 arguments. 1 were passed in" .... ?? two args?
[15:45:45] <djoanes> Hey, how can i create a converter in spring to save a particular field ina document as a java serialized object
[16:07:06] <bros> Any criticisms on the aggregation/use? https://gist.github.com/brandonros/97f5ebba0be2667e5345
[16:31:03] <GothAlice> For use outside of a sharded cluster, see: http://docs.mongodb.org/manual/reference/command/serverStatus/#globallock-activeclients
[16:34:58] <GothAlice> Hmm; the docs aren't explicit on which perm is needed. Let me dig around my production cluster.
[16:36:09] <GothAlice> For conPoolStats you'll need clusterMonitor, clusterManager, clusterAdmin, or root. These are the ones I checked for that command.
[16:37:30] <GothAlice> The first user you add should be an "admin" (i.e. "root" role), but it isn't actually enforced, I don't think.
[16:37:47] <fontanon> Hi everybody, I'm migrating from a single-node mongodb to a mongodb-cluster. To migrate data, I added temporarily the single-node as a shard, so the sharded collections started to drain data to the cluster, but ... what about the not-sharded collections? Once I remove the old-single-node shard would the not-sharded collection be copied to the rest of the shards in my cluster?
[16:39:18] <GothAlice> tpayne: http://docs.mongodb.org/manual/reference/built-in-roles/ should be of use.
[16:54:41] <fontanon> Hi everybody, given a mongodb cluster, how can I force to move not-sharded collections from one shard to another ?
[16:59:38] <fxmulder> Tue Apr 28 09:38:14.221 [rsSync] Socket flush send() errno:9 Bad file descriptor
[16:59:59] <fxmulder> this is now the second time this has happened and my week-long replica cloning is starting over for the second time now
[17:01:00] <fxmulder> I'm going to have to rsync this, if there is any part of mongodb that needs to be looked at it is replica clong, this is horrible
[17:10:20] <daidoji> cheeser: hmm, does that mean I have to run multiple instances instead of just one?
[17:11:45] <daidoji> cheeser: really? I thought I would just shutdown mongod, mount a snapshot (with expanded space) to the dbpath, start mongod
[18:15:47] <StephenLynx> well, how can you have a binary value for your document id? then you would just hold two documents in the collection?
[18:17:31] <ah-> and it's very convenient, since my primary key/identifier for that collection is that binary string, which doesn't fit into an objectid
[18:20:07] <ah-> it's just a sha hash, and i'm pretty sure it worked in previous experiments with python
[18:21:59] <StephenLynx> I find _id to be such a pain in the ass that I use another field to make my unique indexes.
[19:43:30] <christo_m> design question: i have users, and many queues belong to a user, and many queue items belong to a queue. sholud these all be nested subdocuments?
[19:45:19] <christo_m> currently there are two types of queues.. so the number per user shouldn't change (right now)
[19:48:04] <christo_m> cheeser: from my understanding you should embed your child objects within parent objects as much as possible
[19:48:31] <GothAlice> http://www.javaworld.com/article/2088406/enterprise-java/how-to-screw-up-your-mongodb-schema-design.html goes into a few of the factors involved in choosing when to embed.
[19:50:21] <GothAlice> From one app I wrote, MongoDB-powered forums, embedding replies to a thread in a thread? Great idea. Embedding replies in threads in forums? Terrible idea.
[19:51:29] <GothAlice> It's a combination of factors: when looking at one thread, the user could care less about the others, so you'd need to project them out. It's also nigh-on impossible to update beyond a single nesting without losing your sanity.
[19:52:24] <christo_m> whats the method in joining them together after? is that what mapreduce is about
[19:52:33] <GothAlice> And it'd also restrict the amount of content available per forum, not per thread. (I.e. 16MB threads = ~3.3 million words per thread. 16MB forums? 3.3 million words in _all threads in that forum_.) There are ways around this, i.e. having "continuation" documents, but that adds a lot of complexity to the app.
[19:54:19] <christo_m> gonna read this: http://blog.mongodb.org/post/87200945828/6-rules-of-thumb-for-mongodb-schema-design-part-1
[19:54:29] <christo_m> seems to explain that the 1 to many problem needs to be thought about a bit more in depth
[20:08:30] <dreamdust> Is the performance of find({ some: 'indexedField'}) and find({ some: 'indexedField' }).sort(0) equivalent ?
[20:12:12] <GothAlice> I.e. a new queryset instance would start with a skip of zero, and changing it to zero should have no impact.
[21:41:43] <drags> anything recommended for visualizing shard layout besides shard-viz? browsed through mongodb-tools.com and been searching, but not coming up with much
[22:13:32] <greyTEO> GothAlice, you mentioned you replaced redis(and/or)memcache with mongo. Is it as simple as creating a collection for calculated data? Same as memcache?
[22:14:25] <GothAlice> greyTEO: Depends. If you want to replicate a redis queue, use a capped collection. If you want to replicate a memcache auto-expiring key-value store, use TTL index (with zero delay) on a date field you set to the expiry time.
[22:15:30] <greyTEO> I am wanting to avoid introducing a new system that I have to manage. I think I can do it mongo.
[22:17:52] <GothAlice> greyTEO: As a note, MongoDB TTL indexes are culled once a minute, and if a pass doesn't complete in time it won't clean up all expired data all at once.
[22:18:31] <GothAlice> greyTEO: Ref. my own cache implementation's "getter" which needs to handle the TTL edge case: https://github.com/marrow/cache/blob/develop/marrow/cache/model.py#L115-L133
[22:20:10] <greyTEO> I dont necessarily care that it doesn't expire. I will have to purge the queue on events.
[22:21:12] <coucou77> I sent a request in stackoverflow regarding mongodb, if anyone has any clue, thanks in advance!
[22:21:12] <coucou77> http://stackoverflow.com/questions/29929338/mongodb-performing-a-group-on-mapreduce-output-inline
[22:24:42] <greyTEO> Any reason to not update the cache on cachemiss? I do not know the interworkings on your system though?
[22:26:19] <GothAlice> greyTEO: It's an early exit from https://github.com/marrow/cache/blob/develop/marrow/cache/model.py#L75-L79
[22:27:12] <GothAlice> ^_^ This lib is 100% tested across Python 2.6+ and Pypy2+ and is used in production at the moment. It's MIT. Feel free to steal any ideas that seem good. ;)
[22:36:54] <greyTEO> I will definitely reference it. Solid extension. Ill have to figure out how to navigate through python...
[22:46:20] <bros> GothAlice, Thanks for helping me bring my server usage from 100% to 5% and response times from 3s to 30ms with aggregation.
[22:52:18] <GothAlice> bros: The .0. stuff is… potentially concerning. (I take it you are intentionally only checking the first array element in that first $match, line 70?)
[22:53:08] <GothAlice> $unwind on the array, $last the values you care about in a $project stage, bam, you'll (eventually) get the last value of each document's array.
[22:57:22] <bros> I would have never known about $project if it wasn't for you. All of the tutorials/documentation tried to lead me toward $group it felt
[23:04:42] <GothAlice> For certain edge cases involving lists, an $unwind/$group pair is effectively the only solution. I.e. your "get the last array element" issue.