Log file Viewer
#mongodb logs for Wednesday the 17th of June, 2015
[03:24:49] <Jonno_FTW> I keep getting an error that mongodb can't find a spatial index when using $maxDistance, I'm using 2.4.13, do I need to upgrade?
[07:22:34] <m3t4lukas> if anybody needs it, the answer lies here: https://github.com/mongodb/mongo-c-driver/blob/master/src/mongoc/mongoc-matcher-op.c#L1196
[09:52:09] <jar3k> I just wanted to know what is the difference between those two engines. But now I know, wiredTiger is better wor writes and MMAPv1 for reads.
[11:15:01] <sterichards> How can I write a query that will search “where username = ‘x’” and also “where id in iDs” from the following data structure - http://pastebin.com/QumTF82h
[11:22:02] <joannac> skuaskua: i volunteer my time, and I prefer to help people who make some effort to help themselves first :)
[11:25:50] <joannac> skuaskua: btw i'm not ignoring your question, but i don't use GUIs so I don't have an answer for you
[13:18:36] <dreamreal> I apologize for the newbie question, but if I have a document that's mapped like a/b/c (a contains a b, which contains a c, each of which have unique identifiers) can I access c using b's unique id?
[13:35:35] <dreamreal> StephenLynx: I'm at a POC stage right now. It's basically a master/detail/detail structure; an ontology has concepts, which have relationships and individuations
[13:36:20] <dreamreal> I don't have a document yet, no. I'm trying to figure out how to organize the documents.
[13:36:57] <dreamreal> I apologize for being vague. I am not meaning to be - I just don't have more information yet.
[13:37:43] <rakan> If i had a dictionary of different key names such as {"names": {"name_first": "1", "name_second": "2", "name_third": "3"}} as a subdocument. How do i fetch all document with only if the key of the subdocument matches name_f* ?
[13:38:07] <StephenLynx> instead you should use either subarrays or a separate collection with fake relations to parents. either option have its pros and cons.
[13:40:44] <rakan> StephenLynx: the use case is a more of a key value store... and i want to return keys that match a certain structure
[13:42:54] <StephenLynx> what you could do then, is to remove the 'names' object and have all three directly under the document.
[13:45:02] <deathanchor> the best documents are flat, subdocs for "organization" just adds complexity elsewhere.
[13:46:38] <StephenLynx> plus if the keys are going to be same across documents, how can you query them?
[14:09:15] <Someguy123> can anyone help me with $pull? I've put full details of my issue on this gist: https://gist.github.com/Someguy123/c8be45323e014144244d
[14:16:29] <jacksnipe> http://docs.mongodb.org/manual/reference/operator/update/pull/ it appears that pull only supports one field, yeah
[14:19:04] <Someguy123> jacksnipe, but if you look, they show an example where they use $lt/$gt against the fields
[14:19:25] <Someguy123> surely if they can use functions like that, multiple selection should be fine...
[14:20:21] <jacksnipe> looks like you're not using $in correctly: http://docs.mongodb.org/manual/reference/operator/query/in/#op._S_in
[14:20:57] <jacksnipe> the array you pass $in seems to be an array of literals, I don't know that yo ucan specify other document fields
[14:33:55] <jacksnipe> yeah but it's pulling out the elements of tags that are in the js literal array [ "appliances", "school" ]
[14:48:37] <Someguy123> hey cheeser, any idea why my update $pull query works great on the mongo CLI, but doesn't work from NodeJS?
[14:48:46] <cheeser> so depending on how complex the agg pipeline is and how much data you're processing, it might come back "instantly" or it might take some time.
[14:49:33] <Someguy123> dealing with a strange issue where if I copy and paste the query above into the mongo CLI, swapping out the variables for the actual usernames literally, it works great
[14:49:57] <cheeser> svm_invictvs: i was extra careful to not say map reduce since that's a mongo thing. :)
[14:50:29] <cheeser> you *can* wait around for m/r jobs. but since they can write to collections, usually you'd fire it off and check back after a while.
[14:52:22] <Someguy123> yeah I'm completely lost, especially considering the node query actually says nModified 2
[14:53:24] <Someguy123> I'm slowly regretting my choice of mongo more and more as I build my application
[14:54:28] <Someguy123> because I keep running into strange bugs like this, or realizing something that's a 20 character query in SQL, takes 50 lines in mongodb node
[14:55:25] <svm_invictvs> Someguy123: 'cause, I'm fighting with MySQL qeuries with just as hugely complicated queries.
[14:55:26] <Someguy123> StephenLynx, I'm running the same query in mongo CLI as I am in nodeJS, yet it doesn't work in my node application
[14:55:49] <Someguy123> both return nModified 2, claiming they worked fine, yet the node query didn't actually changer anything
[14:56:46] <Someguy123> StephenLynx, it's node-mongodb, there isn't really any good documentation outside of the very basics
[14:58:24] <StephenLynx> and your query is wrong because you are trying to match in the update block.
[14:59:36] <Someguy123> StephenLynx, the documentation for $pull shows them using $lt and other queries within the update param
[15:00:53] <StephenLynx> $in by itself is just checking if the array matches the one you are passing
[15:01:15] <Someguy123> oh, I misunderstood pullAll, from a quick look it sounded like something that would clear the array
[15:06:17] <Someguy123> that code right there claims nModified: 2, but doesn't actually modify the user, yet from the mongo CLI it works fine
[15:43:12] <Someguy123> StephenLynx, here's the full code I'm using, I really have no clue what's going wrong here: https://gist.github.com/Someguy123/a53e39520aea636c0c1d
[15:44:42] <Someguy123> StephenLynx, async is a library similar to promises, makes things easier when dealing with a lot of mongo queries
[15:45:50] <StephenLynx> try without this bullshit and print what you have in the collection before and after the update.
[15:45:58] <GothAlice> StephenLynx: Remember: practicality beats purity. "It's crap" is neither constructive, nor helpful. Also watch your language.
[15:48:56] <GothAlice> Indeed, simplifying problems down to "let's get it working in the mongodb shell" is a valid support approach. "stop it, it's crap" is not.
[15:49:29] <StephenLynx> I just gave him the whole code he needed. he couldn't make it work and said "I'm confused"
[15:49:35] <GothAlice> And looking at it, it's no more unreadable than any other JS I've seen suffering from callback hell. ;)
[15:51:10] <StephenLynx> its like saying "I saw people dying from shots to the head". Well, I dare to say shots to the head are quite lethal then.
[15:51:41] <GothAlice> Someguy123: In the example you provided (https://gist.github.com/Someguy123/6105a7ef5e22d308473c) I see a "callback" third argument to collection.update. In the shell, that's not a thing, FYI.
[15:51:59] <Someguy123> GothAlice, correct, that is NodeJS code, without the callback in the shell it's perfectly fine
[15:56:45] <GothAlice> Someguy123: I will admit, my lack of familiarity with JS, and async.parallel in particular, is thwarting my brain's attempt to grok the code. Is that defining new functions for later execution, or is that saying "actually run all three of these in parallel right now"?
[15:57:05] <Someguy123> GothAlice, it means run all three in parallel, and once all three have completed, run callback
[15:57:28] <Someguy123> so once they've all finished and returned data to a_callback, the final callback paramater of parallel runs
[15:57:52] <Someguy123> if any gives an error as the first arg, it automatically stops execution of any of the others and returns the callback
[15:59:15] <GothAlice> MongoDB can't do parallel, BTW. All database operations are handled in a linear fashion.
[15:59:33] <GothAlice> Someguy123: Actually… http://docs.mongodb.org/manual/tutorial/perform-two-phase-commits/
[16:00:56] <GothAlice> Someguy123: I'm sad to say I don't see anything technically wrong with that code. All of the MongoDB operations appear to be OK, if not following a code style guide. ;)
[16:01:24] <Someguy123> it's strange because I use AsyncJS all over my application for MongoDB without issue
[16:02:18] <GothAlice> "If I throw away the entire module that contains one bug somewhere in it, it works!" Not exactly bulletproof logic.
[16:09:40] <Someguy123> as soon as I added in "added 1" and "added 2" where it adds the persons to eachothers contacts, it suddenly stops removing the person from their pending/sent contacts
[16:09:42] <GothAlice> You can code those as separate functions, then just pass the function names in.
[16:10:22] <Someguy123> normally I use async which makes it look so much more cleaner (though still not perfect)
[16:11:48] <saml> insert({_id:1}) find({_id:1}) gives me no result sometimes. remove({_id:1}) find({_id:1}) gives me result sometimes
[16:12:12] <GothAlice> Aye, the default write concern says "confirm the server got my message", not "confirm the server actually wrote my changes".
[16:12:52] <GothAlice> saml: "Correct" depends on the situation. The default is "confirm receipt". Ref: http://docs.mongodb.org/manual/core/write-concern/
[16:13:50] <GothAlice> So testing is important. But it's also important to not need to do things like re-query for data immediately after inserting it.
[16:15:35] <Someguy123> StephenLynx, any ideas? I don't understand how updates that happen within the callback, can affect the results of the parent function which already executed...
[16:17:10] <GothAlice> And because of the nesting, the closures (functions) you're defining will inherit scope.
[16:18:00] <Someguy123> callbacks are the only time I've really had scoping issues like that, so any nested callbacks usually get renamed, e.g. a_callback, cv_callback
[16:18:13] <GothAlice> Someguy123: Have you tried issuing the three updates at the same "level", i.e. not chaining them like this?
[16:18:39] <GothAlice> The first query is modifying fields different from the other two, so shouldn't conflict, and the other two are writing to different documents, so again, shouldn't be a conflict.
[16:18:42] <Someguy123> GothAlice, I assumed chaining them would be the "safest" way that would avoid any weird issues
[16:19:29] <Someguy123> so all three things can run seperately, then run the callback once all three are done running independant of each other
[16:20:07] <GothAlice> A simple way to handle the "call when actually done" thing is to have a counter which each separate update callback increments then checks if === 3, run the final callback.
[16:20:31] <GothAlice> But that's beyond the scope of diagnosing things. Ignore the final callback for now. ;)
[16:20:52] <GothAlice> Oh, you also need to make sure you app isn't quitting before those operations finish.
[16:30:10] <Someguy123> GothAlice, I'm lost now, so with plain JS callback hell, it works, as long as the other updates happen outside of the first updates callback, but when ran in series using async, the two final updates seem to run, but not the thing that removes from pending/sent
[17:16:50] <MacWinne_> what do folks typically set their driver connection timeouts to? I currently have mine at 500ms for php client for no particular reason.
[17:17:18] <MacWinne_> i think I noticed some rare connection timeout issues, so was thinking of increasing this... we have 2 datacenters that have about 30-40ms latency
[17:17:49] <MacWinne_> does the number of nodes in a replicaset or cluster have any bearing on the timeout value that should be set?
[17:18:14] <GothAlice> Haven't really ever needed to, and I typically only deviate from service defaults when an actual need arises. ;P
[17:20:15] <GothAlice> MacWinne_: Obv. on PHP it'll be naturally restricted by the PHP allowed runtime duration.
[17:20:53] <MacWinne_> yep.. that too.. i'm almost complete moving to nodejs.. just trying ot keep the php side going on crutches until complete
[17:23:15] <MacWinne_> GothAlice, how are the wiretiger patches looking so far.. Are the major bugs you were monitored fixed up in the recent patch set?
[17:24:30] <MacWinne_> if you store an empty file in gridfs, will it take a minimum of 256k for 1 chunk plus whatever bytes the fs.files document needs?
[17:25:56] <GothAlice> Though the document for that minimum chunk will likely be sparse, i.e. not filled with zeroes, so it'll actually take the smallest document size… size.
[17:31:42] <MacWinne_> if the vast majority of my files in gridfs are beneath 100kb, is it reasonable to reduce the chunksize?
[17:35:22] <GothAlice> No. On-disk allocation is entirely handled by the storage driver, independent of how you use those collections.
[17:35:40] <GothAlice> And there are side-effects for manipulating the chunk size. For example, power-of-two sizes.
[17:36:02] <GothAlice> If you set your chunk size to 1024 bytes, you've actually constructed 2048-byte documents because the _id and BSON structure bumps it over a power of two.
[17:37:02] <GothAlice> This gets more dramatic the larger you go. ;) a 16KB chunk size would accidentally give you 32KB documents…
[18:32:55] <fxmulder> so I've setup sharding to split up my cluster between 2 different replica sets, it appears the original replica set is still growing though
[18:34:09] <fxmulder> how might I go about fixing that so mongodb reuses space that was migrated to the other shard?
[18:34:31] <fxmulder> I'm at 92% usage on this replica set so I'm going to be running out of space soon
[18:41:15] <StephenLynx> I have never ran replicas or clusters, but from what I understand, replicas do not split data.
[18:42:08] <StephenLynx> so if you set up just replica sets, it wouldn't decrease the amount of data per machine, right?
[18:43:47] <fxmulder> StephenLynx: I have 6 machines, 2 different replica sets of 3 machines each setup with sharding
[18:43:50] <Epichero> yeah, i could do that in my code... I'm just not sure how random that would work out to be
[18:44:21] <StephenLynx> another thing about space, I read that mongo do not free space back to the system.
[18:44:41] <fxmulder> no, I didn't expect it to free space, I expected it to reuse space, but it keeps consuming more space
[19:07:17] <GothAlice> My face. https://gist.github.com/amcgregor/5863ea6d848d192df03c < this query generated by my code is heinous. >_<
[19:42:33] <GothAlice> https://gist.github.com/amcgregor/5863ea6d848d192df03c#file-query2-py < mostly fixed, I think.
[20:10:02] <saml> write concern is per operation? not per connection? is there a way to connect to mongod with default write concern?
[20:13:38] <GothAlice> Hmm, well, I don't know about that, but in your connection URI, ?w=1 would be what you would add to the end.
[20:14:33] <GothAlice> Confirms that even in the event of a power failure, on startup the record will be found in the journal, and committed. (Generally.)
[20:15:12] <saml> res0: scala.util.Try[reactivemongo.api.MongoConnection.ParsedURI] = Success(ParsedURI(List((localhost,27017)),MongoConnectionOptions(0,None,true,true,10),List(jornal, w),Some(foo),None)
[20:17:21] <GothAlice> The official driver would seem to parse additional arguments correctly. Several are documented on this page alone.
[20:19:23] <GothAlice> Or realtime needs. ReactiveMongo does some truly insane in the membrane things.