Log file Viewer
#mongodb logs for Wednesday the 8th of May, 2013
[01:59:34] <danlamanna> anyone have a clue why i get a syntax error on this? https://gist.github.com/danlamanna/5537650
[02:08:32] <danlamanna> not sure what that means. I know its a bit weird because I want to use the object id as a key
[02:10:50] <lqez> danlamanna: http://stackoverflow.com/questions/4555228/mongodb-can-i-use-objectid-as-key FYI
[05:11:55] <krz> if i wanted to do an aggregation on comments: http://docs.mongodb.org/manual/core/aggregation/ instead of tags. how would i use $unwind here?
[06:19:12] <Garo_> Is it normal that the oplog is almost completely in res memory (ie. it's "hot") in a master of a replica set? I used mongomem to determine that currently around 95% of the oplog collection is mapped into physical memory.
[06:23:56] <krz> anyone: http://stackoverflow.com/questions/16418348/how-would-i-filter-out-sub-documents
[07:28:05] <Nodex> 6250 Wed May 8 12:50:55.759 [initandlisten] exception in initAndListen: 12596 old lock file, terminating
[07:29:22] <Nodex> 6248 Please visit http://dochub.mongodb.org/core/repair for recovery instructions.
[07:29:57] <Nodex> http://docs.mongodb.org/manual/tutorial/recover-data-following-unexpected-shutdown/#indications
[09:55:32] <Garo_> Here's a nice tip how to get quickly statistics about top collections which are queried with tcpdump: http://www.juhonkoti.net/2013/05/08/quick-way-to-analyze-mongodb-frequent-queries-with-tcpdump
[10:23:18] <kali> Garo_: mongodb comes with something called mongotop... from what i'm reading, it gives more or less the same information, without the impact of the promiscuous mode
[10:35:09] <Zelest> Derick, I noticed that if I have 3 nodes in a replicaset.. and one goes down and is listed as unhealthy/unavailable via rs.status(), the php-driver tries to connect to it anyway, waiting for the timeout to occur..
[10:36:37] <Nodex> what I'm trying to work out is.... if I upgrade will somehting that worked in an old version now be buggy
[10:37:13] <Derick> Zelest: we need to connect to it in order to find out what is happening, but as I said, 1.4 has a lot better logic for avoiding that situation.
[10:39:05] <Zelest> but if a master is available and you have readpreferences to read from the master only, ignore the slaves altogether :P
[10:40:46] <Derick> but we do need to do a discovery of the replicaset to even find out whether there are secondaries...
[10:41:56] <Zelest> i know it's tricky, but if you connect to let's say 3 nodes.. once it's connected to one node and the rs.status() is done, abort/cancel the other connects/discovery-probes.
[10:42:47] <Zelest> now i had 3 nodes, one secondary was down.. master was up and everything was working fine.. yet, the application was unavailable/unusable since it had to wait for the timeout to occur of the secondary that was down.
[10:43:08] <Zelest> which kind of defeat the whole point of using a replicaset as a sense of failover/redundancy :)
[12:24:21] <yields> basically there is "companies" collection, where each "user" can create multiple companies, each company has "customers", each "customer" has two properties "company_id" and "code" the "code" must be unique to each company.
[12:24:33] <romenov> hi, I have a collections of lists, and I want to store settings of these lists in a different collection because there will be attributes that I really don't want in the lists collection. Can I use the _id of the list to create a document in the settings collection (so that they have the same ID over two different collections) or is that a nono?
[12:30:19] <yields> I need to get the "company" and then the "customers" of those company, each "user" may have multiple companies, he basically picks which company to manage and then he sees it's customers.
[12:32:54] <yields> but i still need to query for specific customers of a company, for example i may want to look for a customer whose name is "david" within a "company".
[12:50:52] <theRoUS> it's set in the /etc/mongodb.conf file, but i don't see '--auth' on the command-line from ps -- and i can use a database and use 'db.system.users.find()' without authing first
[12:55:16] <woozly> guys, why if I use DBOject (which returned from DBCursor.next()), why if I try to update this object like: myCollection.update(myDBObject, new BasicDBOject("$inc", new BasicDBOject("someField", 1))); - I can do it only once, on second, it will not increment field. Why??
[12:59:42] <converser> Hi all. Quick question, what are the repercussions of using $ or . as keys in mongo. I ask because I've noticed something a bit unusual.
[13:00:00] <converser> Generally the mongo driver won't let you use $ or . in key names, but why is this exactly?
[13:02:42] <converser> Alright well, decided to try out Mongoid (Using the moped driver) and it doesn't actually enforce this.
[13:10:57] <Nodex> when updating with the positional operator "$" you will run into all sorts of problems
[13:20:42] <converser> There is a structure in place in our app where the key is a button, and the value would be the number of clicks. EG {"button-1" => 5}
[13:21:27] <converser> But its actually a questionare builder. So anyone could type in anything. EG {"$20 Euros" => 5}
[13:23:18] <kali> converser: it's cumbersome to have keys that are data and not application keywords... you'll run into all kind of problems
[13:25:38] <converser> Nodex: You're talking about sanitizing the characters as they go in and out of the database. Are those special characters or would I need to read serialize them in and out at application level.
[13:26:29] <Nodex> but if it's tightly nested in your app already then a hack is what you must do sometimes
[13:26:39] <converser> kali: You're structure above is perfect but its not going to work right now. Changing the structure like that isn't really an option
[13:28:17] <kali> converser: well, if you want to look for all {button-1 : 5}, you will want an index of button-1
[13:29:24] <converser> Thankfully thats not going to be an issue though as there are no circumstances where we need to use button-1 or anything similar as an index.
[13:30:11] <converser> So it would be possible to use what Nodex mentioned as a solution. (:DLR: :DOT:)
[13:31:14] <converser> Sanitizing the data as it goes in and out of the database seems the only option for now. Until we can get a better hash structure in place.
[13:31:20] <coin3d> hello. i have a collection "lecture_types" which stores documents with only one attribute: "name", which is a simple string. so i guess it doesn' make sense to reference to these documents. should i denormalize my other collections to store the name of a lecture_type directly? if so, would it be better to store the lecture_type_id as well?
[13:33:36] <converser> kali: Don't misunderstand, changing the hash is definitely the right way to go. The only reason it cant work is because there is a mobile element involved. Meaning there are a ton of mobile apps in the wild which would break if they received that new structure before they could be upgraded
[13:34:04] <kali> coin3d: once again, it depends on the access pattern. but in that case it does not sounds like you're gonna alter these lecture_types anytime soon. so embedding seems to make sense
[13:35:01] <coin3d> kali: yup, okay. my access pattern is: i also need to have access to "all lecture types". so should i additionaly store a lecture_types collection with a unique name?
[13:36:21] <kali> coin3d: mmmm i'm not sure hos "distinct" is optimized since 2.4: maybe running a distinct will only access the index (assuming there is on on the lecture_type field of the lecture collection)
[13:37:36] <kali> converser: try to get the app developers to get a "force update now" feature in the protocol. it only took me three years to get them to do it for our apps :)
[13:38:23] <kali> converser: so i still have three years worth of hacks and cludge to deals with the version we can not force update, but at least we are not adding un-removable cludges anymore
[13:42:41] <converser> kali: Just one last question. Nodex mentioned escaping the keys earlier on. Did he mean like strings?
[13:44:09] <kali> ha no. you need a different kind of escaping. backslash will not work. what Nodex suggests is "$akey" -> ":DLR:akey"
[13:46:02] <converser> I see. So when you read whatever object that belongs to, your turning '€a' into a '$' or some other character. Right?
[13:58:19] <converser> kali: I get why the less profitable way bugs you bug I'm kind of stuck. If I was to change the structure of the model in the way you suggested then everything in the wild would break. We cant really afford to take that knock on the chin.
[13:58:53] <zokko> is it safe to HUP mongodb? i need to reload config, verbose logging was turned on :-)
[13:59:49] <kali> zokko: i'm not sure it reloads the config. i think it will only cycle its file descriptors on log files
[14:03:13] <converser> If using $ is so destructive to the mongoid database, What do you think about Mongoid letting something like that through? Isn't that kind of important?
[14:11:33] <Nodex> why dont you just store the thigns you need in distinct in a separate array somewhere in another collection and $addToSet if you ever add a new one?
[14:14:38] <kali> coin3d: http://docs.mongodb.org/manual/reference/command/distinct/ the last line says it will use the index if there is one
[14:29:39] <failshell> hello. does anyone know of a cluster backup tool? something that can orchestrate the stopping of the balancer, locking a config + N nodes to then create a LVM snapshot?
[14:55:10] <starfly> There was some talk of 10gen offering MMS for in-house installations, but I'm not sure where that went
[15:15:53] <Nodex> I've decided it's really stupid to -have- to put in delays in code to show that something has happened due to the speed of an app being so fast that it appears as if nothing happened :/
[16:45:41] <Nodex> quick Q. ... if you guys could pick from scratch ONE transactional DB (for transactions only) to compliement MongoDB which would it be?
[16:47:34] <starfly> Nodex: MySQL if you're LAMP-oriented or Oracle if you're not LAMP-oriented and have buckets of money
[16:49:56] <Nodex> putting things into a queue and updating Mongo when the queue has ran (i/e transaction complete)
[16:52:52] <starfly> Nodex: We're just saying… If you want no bloat, roll your own, but that's usually not in the best long-term interest
[16:54:40] <starfly> Nodex: Cool, whatever, but if you want ACID out of the box, MySQL should be fine
[16:58:40] <Nodex> I recently had to fix a freinds mysql based site... I have not touched SQL in ~3 years - I was lost for a day LOL
[17:02:12] <leifw> Nodex: you can email us at support at tokutek dot com, or come to the #tokutek channel, we'd love to tell you more about what we're doing with transactions in mongo if you're curious
[18:24:20] <leostera2112> Has anybody knowledge of a entry-level to good read on performing ACID-like transactions with Mongo?
[18:25:16] <leifw> leostera2112: I'm working on adding native multi-statement transactions at tokutek
[18:25:58] <leifw> leostera2112: if you'd like to try it let me know, we have a prototype that we're hoping to have people try and give us feedback on
[18:26:06] <TommyCox> richcollins: Comparable to MySQL, you can string together multiple statements in a single query, such as FINDing all documents WHERE the title equals foo, SORT by bar, then LIMIT to 100 records would be a psuedo example
[18:26:21] <leifw> leostera2112: http://www.tokutek.com/2013/04/mongodb-multi-statement-transactions-yes-we-can/
[18:32:46] <starfly> leifw: How do you ensure atomicity across multiple statements like updates? What about the cooperative, multi-processing that MongoDB does (writes yielding to reads)?
[18:33:44] <leifw> starfly: we're replacing the storage code with a fully transactional, concurrent engine that uses MVCC, and row locks for serializable isolation, so we don't need yielding
[18:35:04] <starfly> liefw: So you're using MongoDB as a shell for your own storage engine, interesting, but how will you keep in lockstep with the evolution of MongoDB?
[18:37:15] <leifw> starfly: most of the mongodb ecosystem is pretty well abstracted from the storage code, I backported hashed shard keys to our 2.2 branch in a couple hours for example
[18:42:21] <starfly> leifw: I don't know your other published products, but presume you took the same general tack with them as well. I'm all for extending products to fill feature gaps and wish you well.
[18:43:26] <leifw> starfly: our other product is a storage engine for mysql, the api layer there is a lot clearer and well tested so it's a fairly different strategy
[20:13:04] <JoeyJoeJo> I'm trying to use $or in php, but it gives me this error - MongoCursorException: localhost:27017: invalid operator: $or in MongoCursor->rewind()
[20:23:29] <scoutz> would it be ok to host my 3 config servers in different datacenters? (50-75ms between them)
[20:26:10] <starfly> scoutz: That should be OK, but do you have a large number of shards and lots of document migration activity between them? The 2-phase commit activity could be high if so.
[20:26:14] <fxhp> How would I sum the tick field?: http://pad.yohdah.com/111/f952e8d0-c797-47de-996b-33512e6db5bb
[20:28:34] <kali> fxhp: aggregation framework... but i think you'll want to denormalize it in the long term
[20:28:56] <scoutz> starfly: well i'm sharding a single collection of user sessions (shardkey: sessionid)
[20:29:34] <scoutz> i'm not really sure of migration activity tho, i haven't setup sharding yet i'm only using a replica set for now
[20:29:52] <fxhp> kali: I was looking at the aggregation framework, I couldnt get the syntax right, mind giving it a shot?
[20:30:53] <kali> scoutz: it's one of the deployment scenarios in several presentation by 10gen people
[20:32:33] <kali> fxhp: not in its own collection, but having the sum of the votes as a field in the document can be very convenient
[20:33:00] <starfly> scoutz: I'd guess that sessionID are transitory and only persist for a particular Web app session during it's life. Unless you configure for many shards, I think you'll be OK. Nothing like a trial, though.
[20:38:44] <scoutz> starfly: actually they are sessions but sessions for an adserver so maybe more like 'profiles', the session will stay in the collection for as long the cookie times out (1 year+), i have thought of different shard keys but since i will only need to get 1 object from the collection each time i thought the sessionid would be the best shard key
[20:42:10] <starfly> scoutz: are the shard keys hash-like or monotonically increasing? The latter, of course, should be avoided to cut down on document migration and bottlenecking on document creation writes.
[20:44:23] <starfly> kali: true enough, unless scoutz isn't at the release (2.4) where that's available
[20:51:00] <starfly> scoutz: good, but back to the earlier question, I personally wouldn't be too concerned about how much latency is involved in the config server updates unless you have a perfect storm of a lot of shards and a lot of document migrations between those shards (or a lot of splits inside shards, which also require config server updates). I think you need to dive in and try some sharded configurations.
[21:08:16] <fxhp> kali: db.posts.aggregate([{$group:{_id: "$user_id", ticks: {$sum: "$votes.tick"}}}])
[21:20:48] <TommyCox> fxhp: Try this db.posts.aggregate([ { $group: { _id: { "$user_id", ticks: {$sum: "$votes.tick" }} }} ])
[21:30:58] <iwaffles> so if I have a document that has user_ids can I do a find that does if the user_id exists in user_ids[] ?
[21:36:11] <iwaffles> I have an array in the document, and I want to pass in one value and check if that document contains it inside that array
[21:40:06] <TommyCox> db.posts.aggregate( [ { $unwind: "$votes" }, { $group: { _id: "$user_id", ticks: { $sum: "$items.tick" } } } ] )
[21:42:47] <fxhp> TommyCox: So that sort of works, but it ends up querying all documents from the collection, unwinding the embedded votes object, and summing them all together
[21:43:00] <TommyCox> db.posts.aggregate( [ { $unwind: "$votes" }, { $group: { _id: "$user_id", ticks: { $sum: "$votes.tick" } } } ] )
[21:48:10] <TommyCox> fxhp: Did it not work? what's the output look like? What do you want it to look like?
[21:50:36] <freezey> in the pecl-mongo driver mongo.native_long => no value .. previously you were able to set this to false
[21:53:29] <TommyCox> fxhp: maybe you'd like to count the ticks for a document instead of the user mentioned in the document. You'd use this instead:
[21:53:40] <TommyCox> db.posts.aggregate( [ { $unwind: "$votes" }, { $group: { _id: "$_id", ticks: { $sum: "$votes.tick" } } } ] )
[21:58:46] <TommyCox> fxhp: If you can provide some more information on what your end goal is, I can help you further, but for now I'm not entirely sure what you're trying to achieve. Cheers!
[22:07:15] <TommyCox> db.posts.aggregate( [ { $unwind: "$votes" }, { $group: { _id: "$_id", ticks: { $sum: "$votes.tick" } } } ] )
[22:09:20] <fxhp> TommyCox: with that said, I only need the work done on one post_id, so I'm going to try to add the $match
[22:10:33] <TommyCox> db.posts.aggregate( [ { $unwind: "$votes" }, { $group: { _id: "$_id", ticks: { $sum: "$votes.tick" } } }, { $match: "YOUR_POST_ID"} ] )
[22:15:28] <TommyCox> db.posts.aggregate( [ { $match: { _id : "YOUR_POST_ID" } }, { $unwind: "$votes" }, { $group: { _id: "$_id", ticks: { $sum: "$votes.tick" } } } ] )
[22:19:03] <TommyCox> oh, so group does it's grouping based on the keys in _id, so I'm grouping on the unique key $_id. You can group on multiple keys, but not needed in this case. It's really only needed to group back together the unwound votes in this case
[22:24:28] <fxhp> TommyCox: thanks, now that I have this monster of a query, I'm wondering if it would be better to just do that math from my programming lang
[22:25:52] <TommyCox> Lol, could be. The question is, with this awesome query doing for you, why would you want to reprogram the wheel?
[22:32:44] <TommyCox> fxhp: If you want to do it all in one query, you can use $project to get the other fields you need from the resulting aggregation document, no need to query it twice. But honestly, I imagine a foreach - or whatever your language uses - would probably be more readable than a complicated aggregation query. The aggregate framework might be a little overkill for your sitch
[23:24:47] <Baribal> Hi. How would you structure documents that represent coordinates in 64-dimensional space to find those that are within a certain euclidean distance of each other?