Log file Viewer
#mongodb logs for Monday the 9th of July, 2012
[01:49:52] <arrdem> data model design question: in MySQL I understand that it is more "propper" to maintain seperate tables for users, passwords and permissions. This makes sense because in an unabstracted data store you want to querry over the minimum amount of data possible. Is this still the "best practice" for a document store like Mongo?
[07:23:14] <kali> Bartzy: you can go back to the shell, and use db.currentOp() / db.killOp() to cleanup the mess
[07:23:58] <Bartzy> kali: I have another question. The index size from db.shares.totalIndexSize , is in bytes ?
[07:24:28] <Bartzy> Also, for a big setup that we're now migrating from MySQL, around 200 million documents, 100GB +- of data - What should I be aware of / read ? :)
[07:28:25] <Bartzy> So with the 'mongo' linux shell, if I use ctrl+c, it asks me if I want to end the current operation or something, and no matter what I say, it exits.
[07:49:14] <akaIDIOT> i saw a presentation by a 10gen-guy who briefly mentioned mongodb having a problem with data sizes over 2P
[07:51:18] <akaIDIOT> think it was this presentation http://www.10gen.com/presentations/mongosf2011/sharding
[07:59:40] <Bartzy> "You may use sort() to return data in order without an index if the data set to be returned is small (less than 32 megabytes in version 2.0, or less than four megabytes in version 1.8 and earlier). For these cases it is best to use limit() and sort() together."
[08:00:25] <Bartzy> db.things.find().sort({myid:1}.limit(50), returned data set should be a few KB at most, and it takes forever!
[08:02:37] <jondot> i have a use case where I would go over all records in a collection and mark the last time I accessed each. this is done by around 10 workers. so each will grab a subset of the collection, read each record and update each record. i'm wondering how is this fit with mongodb
[08:09:35] <Null_Route> Can any wizards out there help me with an update query to change the second octet of an IP address?
[08:15:01] <Bartzy> vsmatck: This is good to do? Or should I just create an index for the ts field I have an sort by that?
[08:15:32] <Bartzy> It needs to be a compound index anyway, so there's no benefit of using the existing single _id index.
[08:15:54] <Bartzy> So if ObjectId is bigger than ISODate, maybe it's better to use the timestamp index ? Does that make sense ?
[09:25:54] <brahman_work> Hi, Am about to create an index on a sharded mongodb cluster. Documentation indicates that the best way to create indexes whilst causing minimum perfomance hit is to run the indexes on each secondary in the replica set and then demote the primary and so on. Is the process the same for a sharded setup?
[09:50:45] <remonvv> A little off topic but is anyone aware of any blogs/articles that have measured Amazon EC2's instance failure rate and such?
[10:23:20] <Bartzy> 'millis' in explain() are valuable only on the first execution ? After that it's in cache or something and returns 0 ?
[10:37:58] <augustl> I'm storing an array of "users" that has access to a certain document. Is it important that I store actual ObjectIds and not just the hex strings of the ObjectIds?
[10:38:28] <augustl> working with strings seems much easier, that's what I expose through my HTTP API anyway.
[10:43:52] <Bartzy> I have another question: I'm trying to use $in with ~400 values. I get : SyntaxError: unterminated string literal (shell):1 in the mongo shell
[10:44:57] <algernon> paste the stuff to something that can syntax highlight JS, and see where the runaway string is
[10:45:49] <augustl> Bartzy, ron: well it might be that they're more efficient in space, a string of hex encoded bytes typically stores as twice as much data since two characters/bytes are used to represent one byte
[10:46:32] <algernon> OIDs *are* more compact. That's about their only advantage of storing them as binary instead of hex strings.
[10:47:02] <augustl> I suppose querying is slightly faster since mongodb can know it's a pure byte compare, no string comparison fancyness required
[10:47:21] <augustl> obviously now I'm just prematurely optimizing without measuring a single ting :)
[10:49:23] <algernon> augustl: mongodb strings have a length encoded, so comparing them is fairly efficient. besides, in the best case, you compare once (to get the index)
[10:57:54] <augustl> algernon: ah good point, indexes makes it pretty much the same operation anyway I guess
[10:58:25] <augustl> ron: I'm talking about giving it a hex string directly, though: users: ["123abc...."], I suppose mongodb doesn't store ObjectIDs in those scenarios?
[10:59:06] <Bartzy> Now I want to show to a user all the photos of his/her friends, sorted by insert time. I have an index on {uid:1, _id:1} (_id used for sorting by insert time).
[10:59:37] <Bartzy> But if I do uid: {$in : [big list of uid of friends here] }, it takes up to 500ms to the query to execute!
[11:25:08] <Bartzy> remonvv: Just checked without the sort, it's much faster. Perhaps the sort is making it slow for some reason ?
[11:57:04] <Bartzy> I'm trying to benchmark the queries with PHP now, because of the large dataset in $in
[11:57:22] <Bartzy> It works, but I don't know what to time ? The cursor returns Immediately, I think ?
[12:29:21] <augustl> are there any atomic operations built-in for moving a document from one collection to another?
[12:31:30] <Bartzy> I seem to be unable to connect to a mongod instance @ localhost using the mongo shell
[12:32:34] <augustl> considering moving docs to a separate collection for archived documents, but I suppose setting a flag is more than good enough for now
[12:33:22] <Bartzy> augustl: If it's a big collection with lots of indexes... and you're not using the archived documents, maybe it's better to archive ?
[12:37:00] <Bartzy> It's really weird - I rebooted the server, and the port seems up in netstat , and mongodb is up, but telnet localhost 27017 doesn't work
[12:37:51] <ninja_p> Has anyone had much luck testing the new aggregation stuff with the PHP driver?
[12:55:26] <ninja_p> I can see no way of sending new aggregation pipeline queries using the php driver :(
[13:01:42] <skot> There may not be a helper in the shell yet but that is just a convenience, not a requirement.
[13:04:11] <ninja_p> was having issues with the format in which to pass the keys to the argument it takes
[13:49:24] <fredix> I try db.payloads.find({'steps._id': ObjectId("4ffadb1b62770d4e13724937")}) but it returns all the payload bsonobj not only the embedded
[13:49:43] <adamcom> you can definitely do it in the shell - http://www.mongodb.org/display/DOCS/Retrieving+a+Subset+of+Fields
[13:51:05] <adamcom> specifically you need to exclude _id of the root doc, so {_id : 0, steps._id : 1}
[13:53:16] <fredix> db.payloads.find({'steps._id': ObjectId("4ffadb1b62770d4e13724937")}, {_id :0, steps._id :1}) ?
[13:57:22] <fredix> ok this is works > db.payloads.find({'steps._id': ObjectId("4ffadb1b62770d4e13724937")}, {steps: 1})
[14:20:41] <Bartzy> Is that incorrect? http://www.deanlee.cn/programming/mongodb-optimize-index-avoid-scanandorder/
[14:21:43] <Bartzy> I'm using $in on user_id field and sorting by _id. When I use the index {uid:1, _id:1}, I get scanAndOrder:true in explain(), but the query is (pretty) fast. When I use the index {_id:1, uid:1}, the explain() doesn't even end in order for me to see what happens.
[15:04:43] <lolwtfyeh> Hey, how can I query an array of ISODates to find all of the timestamps that are in the hour 13?
[15:24:19] <fredix> db.payloads.find({"steps._id": ObjectId("4ffaf4b3ee838f6fd4135398")}, {_id: 0, "steps.order": 1}) return { "steps" : [ { "order" : 1 } ] }
[15:37:57] <fredix> I obtain {"order": 1} with tihis awfull code : BSONElement l_order = step_order.getField("steps").embeddedObject().getFieldDotted("0.order");
[15:39:36] <adamcom> you can use $slice to make sure only one value is returned, for example {_id: 0, "steps.order": {$slice : -1}} in the profection (or 1) but I don't think you can isolate just that sub element in the way you mean
[16:06:25] <looopy> how would i go about dealing with a list of embedded documents? like...user has many companies. I have my company document mapped out. so now i'm wondering what's a good way to reference each company a user may create...ListFields looks ok but i'm not sure. I'm using python/mongoengine by the way
[16:17:20] <npa> I have a Mongo 2.0.4 question. I'm writing a Ruby script to call mongodump on a sharded collection. if I lock the cluster according to the manual, and I write to the collection with another script while executing the dump, the dump just hangs. the backup script works fine when I do not use the locks. is there a good way to still create consistent backups with a script without using fsync+lock (besides using journaling and snapshots)? also, has anyone else en
[16:26:02] <Sim0n> Hiya. I have a big collection with two compund indexes on field_1_date_1 and field_1_date_-1. And I noticed with explain() than when doing a query "db.col.find({'field':'something'}).sort({'date':1}) the field_1_date_-1 index is used, but in reverse! So my question is can i drop the field_1_date_1? It doesnt seem to be used by mongodb and is taking up close to 6gb.
[16:52:33] <amontalenti> if you create a mongodb hidden secondary (e.g. for a backup) and remove documents from some of its collections, replications of new documents from the oplog will continue to work, right? I am thinking of a use case where we have a secondary that stores "last 30 days of data" as a backup by pruning old documents only on the hidden secondary.
[17:09:49] <jonwage> I have a situation where I $pushAll many elements on to an array, a field inside that array is indexed and when I $pushAll hundreds of items it takes ~2 seconds locking mongod, if I $pushAll in batches of 20 or so will that be better or will I still face the same problem? When I test locally the lock % reports actually being higher when I push them in batches instead of all at once
[17:26:11] <jonwage> anyone have any experience with $pushAll and pushing on hundreds/thousands of elements?
[17:32:45] <adamcom> question - did anyone see my response to npa re: mongodump or did my client disconnect me at the worst possible moment?
[17:35:11] <adamcom> npa: the mongodump is just a read, albeit a long running one - if you have writes waiting, as soon as it yields the read lock for any reason, the write locks will grab it and they are greedy, so will hold the lock until there's a gap in writes - then the readers get another look
[17:35:29] <adamcom> ow are you running the dump though? if you point it at a replica set it will run with slaveOK on by default and you would be reading from the secondaries - might get a bit better performance there, if not you can fsync/lock them without stopping writes on the primary
[17:35:31] <sebastiandeutsch> I just added a new replicaSet Node to my conf. When I take a look at rs.status() is says that the node is recovering, after about 20s something fails "DBClientBase::findN: transport error". Then the uptime is at 0 again and the node is recovering. Is this normal? (Info the db is pretty large ~13gb)
[17:36:36] <adamcom> sebastiandeutsch: the transport error - that from the primary or from the new member?
[17:38:15] <sebastiandeutsch> adamcom: it looks like this: transport error: wla3 :27017 query: { replSetHeartbeat: \"watchlater\", v: 7, pv: 1, checkEmpty: false, from: \"wla2:27017\" }
[17:39:44] <adamcom> transport error usually means the connection between the primary and the new member is flaky - any socket resets?
[17:41:13] <adamcom> the fact that it happens every time though……can you connect using a mongo shell from the new member to the primary and run commands? (similarly the other way around - mongo shell from primary to new member, run a few basic finds etc.)
[17:46:47] <sebastiandeutsch> adamcom: when I try to run a command it says: "error: { "$err" : "not master and slaveok=false", "code" : 13435 }" I guess I have to wait for a while until it's fully recovered. Just wondering if an error occurs if mongo is so clever and restarts from an incremental position.
[17:47:29] <npa> adamcom: I'm using the Ruby driver to obtain the lock - there's an API call to get/release it. from there I connect to the proper secondary node for both shards and run a dump against them in parallel (I have tried running it serially and ran into the same issue)
[17:50:31] <rydgel> I tried to remove a shard, but the drain stopped. Any way to see what's going on?
[17:56:33] <adamcom> sebastiandeutsch: during normal replication that would be the case - I think the copyDB command is a bit more fragile though
[17:57:19] <adamcom> sebastiandeutsch: if you suspect the network is not up to it, you could try seeding the new member with the data files from another member first - then it would not be copying so much data when it joins
[17:57:40] <sebastiandeutsch> adamcom: then I'll wait and see what's happening… copying 13gb over network can flaky. thanks for your time.
[17:59:13] <rydgel> adamcom: I know, but that's like that since 2 months. And yes the balancer is on. And yes chunks are keep moving between the remaining shards
[18:04:39] <adamcom> one of the issues with migrations (which is what has to happen to drain) is that a shard can only participate in one at a time
[18:05:12] <adamcom> if you have a lot of writes (and hence splits, migrates) happening, then they are going to compete with the drain
[18:07:16] <rydgel> adamcom: Yes I can explain the hotspots, we got an issue where one chunks was unable to be moved. So we stopped the balancer a few day, fixing the issue, and then relaunching it like 2 days ago
[18:08:03] <rydgel> adamcom: ok so you think I will have to wait, waiting medias to be balanced and then the draining will start again?
[18:08:18] <adamcom> right - which will again contribute to the contention - the balancer is now trying to do several things: 1. drain 002, 2. balance .popular, 3. migrate max chunk values around
[18:08:51] <adamcom> in terms of what to do, depends on priorities - you could turn off the balancer, then issue moveChunk commands manually
[18:09:11] <rydgel> adamcom: As you can see popular and the other one are already fully drained, but I guess you may be right, I will wait
[18:10:06] <adamcom> rydgel: sure - but for the drain at least it should be relatively easy - you need to move them all off 0002
[18:10:43] <rydgel> adamcom, yes. So I think I may be able to query the chunks on shard0002 and then move them
[18:18:10] <revnoah> I'm having an issue with mongoose and manual references. My Event model has a manual reference to a place id. What I would like to do is query the Event model and retrieve the Place title along with the rest of the Event data. I thought the simplest approach would be to use a virtual in the Event and look up the Place by using the findOne() function but the virtual appears to the synchronous while the findOne()
[18:23:48] <revnoah> aheckmann: so it only works with ObjectIds though? That complicates things a bit since I'm originally pulling data from a mysql (with a Drupal backend) and I would prefer to simply use those references.
[18:24:52] <aheckmann> revnoah: yes object ids only. so the getters/setters/virtuals are all sync, your next option is to manually set up some helper methods
[18:25:25] <aheckmann> revnoah: yourSchema.methods.whatever = function (cb) {… cb(err, yourResults) }
[18:26:17] <revnoah> aheckmann: yes, which I understand in theory but am very unsure how to go about it.
[18:28:12] <aheckmann> revnoah: back to the object ids, they can be either a string, number or objected _id type
[18:28:45] <aheckmann> revnoah: then you'll have to manually do your queries and pass a custom result set back
[18:30:01] <aheckmann> revnoah: what is your "place id"? is it a string, number, objected, or buffer?
[18:31:17] <revnoah> aheckmann: that's quite alright. The confusion is mostly with the vague documentation.
[18:33:25] <revnoah> aheckmann: Stellar. Are you the primary author? I see you contrib quite a bit. For the most part, things are great and easy to use, and you certainly have my thanks for not only maintaining it but also giving help in irc
[18:33:42] <revnoah> aheckmann: I didn't realize that. I'll join that one for mongoose-specific questions
[18:40:43] <revnoah> aheckmann: excellent work thus far. it's taking some work to get my head around it, but it's very cool stuff. Old dogs can learn new tricks, it just takes longer :)
[19:18:25] <vguerra> I was wondering if there are any mongo users hanging around here from Vienna, Austria ?
[19:26:54] <jstout24> question: so if we had tons of writes (lets say a tracking table for instance)… if a user hits a page and clicks a link… the first page will write the impression and the link needs to do a lookup for the impression then update a field on it… would we have a problem doing this?
[19:29:49] <jstout24> currently, we're inserting, then when a user clicks, we're doing a find on the object id, then update.
[19:43:27] <amontalenti> jstout24, you probably want to look at modifier operations (http://www.mongodb.org/display/DOCS/Updating#Updating-ModifierOperations) and findAndModify (http://www.mongodb.org/display/DOCS/findAndModify+Command)
[19:49:41] <kenneth> hey--we're seeing a bunch of errors on the php driver with "couldn't determine master". what's up with that, it seems random and we can't figure it out!
[19:52:13] <kenneth> what's the official way to connect to a sharded cluster btw, is it the same as a replset?
[20:00:25] <sl00> My documents contain an array (plots) of objects with the syntax {type:<integer>,color:<integer>} and now I want to find all documents that have an object {plots.type:1,plots.color:0xffffff} in that array. How can I do that without also getting {plots.type:2,plots.color:0xffffff} ?
[20:05:46] <Bartzy> bjori: It's a new thing, I sent it via the web form of consulting. We need to get some consulting on our hardware spec before ordering it
[20:11:04] <bjori> Bartzy: http://www.mongodb.org/display/DOCS/Production+Notes#ProductionNotes-WhatHardware%3F :)
[20:12:55] <Bartzy> and I guess it doesn't need more IO then a single SSD, and RAID10 would not help a lot in performance, so that's just to make them a JBOD... I gues ?
[20:13:20] <Bartzy> bjori: So we were wondering, hardware RAID, or mdadm.. with respect to a mongo setup
[20:14:11] <Bartzy> Also, what about CPU? We're taking dual E5-2620 so we'll be able to upgrade to lots of RAM (The motherboard only has 4 DIMMs per CPU socket, so if we take only 1 CPU, we get 16GB x 4 which is 64GB... Not so much for upgrades, and 32GB DIMMs are very costly)
[20:24:18] <timing> in a webapp running on mongo, do you guys just pass the objectId arround in the get parms? or do you generate numeric id's and put an index on it?
[20:28:57] <bjori> Bartzy: I have a hard time imagining someone from 10gen will actually bill you for tips on hardware
[20:29:25] <Bartzy> bjori: Well, if I get to their support by the paid consulting way.. how they won't ?
[20:29:26] <bjori> Bartzy: but like the doc say.. we aren't hardware specialists.. and everything depends on your data and access pattern
[20:29:42] <Bartzy> I wish someone from 10gen would just pop here for 5 minutes for quick answers about my questions :)
[20:30:12] <Bartzy> And then for the schema consulting and more in depth consulting of our setup, we'd get the paid support after installing the server and migrating the data and doing our own testing :)
[20:31:03] <Bartzy> bjori: Between 100 and 200GB now, growing fast... 100+ million documents, growing at a pace of 500,000 per day
[20:31:32] <Bartzy> bjori: Don't know about traffic. Why does it matter if the network is enough for it ?
[20:31:52] <Bartzy> I don't want to run several shards at the beginning. We want to scale vertically as much as possible on reasonable price. 64GB RAM for now.
[20:32:15] <Bartzy> We're just looking for: "Most of our setups using SSDs are using THIS type, and THIS raid configuration"
[20:33:48] <bjori> Bartzy: sounds like you first want schema suggestions and then recommendation on a hardware :)
[20:43:00] <dstorrs> I've got a handful of collections I use for my job manager, and they don't need to be replicated. It would just spam the oplog.
[20:51:14] <dstorrs> what's the difference between "local to that database" and "do not replicate" ?
[20:52:41] <dstorrs> I'm gonna run and grab lunch, then I'll RTFM about the 'local' DB. Thanks for the pointer.
[20:53:08] <jonwage_> does anyone know why the default to MongoCursor::count() is false? http://us.php.net/manual/en/mongocursor.count.php
[20:53:33] <jonwage_> it seems error prone, if someone does this $cursor = $collection->find()->limit(5); echo count($cursor); // expect I would get 5
[20:57:19] <timing> If I want to store a very large array in a collection and I don't want to query on it. IS it better to use serialize the object as a string or can I just put the array into the collection as is
[20:59:03] <bjori> jonwage_: wait. re-reading your question.. your example and question don't match..
[21:00:32] <jonwage_> bjori: hmm I'm not sure what is going on yet, when i call echo count($cursor) right now it is still calling $cursor->count()
[21:01:08] <jonwage_> bjori: i was dbl checking that maybe it was my library on top of mongo pecl extn that implemented countable
[21:05:31] <bjori> jonwage_: and I don't see how it could return anything else since it doesn't implement the countable interface
[21:13:51] <bjori> jonwage_: I'm sure we can have MongoCursor implement Countable by tomorrow for you if you want (:
[21:14:15] <jonwage_> bjori: hmm, well i don't necessarily "want" it, it's just now it is already there
[21:15:22] <jonwage_> in our application sometimes cursor get created and passed through to the tempting layer
[21:15:58] <jonwage_> so i am thinking I'm gonna add a listener for all view params and convert cursors to an array
[21:17:22] <jonwage_> and i don't wanna change $foundOnly default to true, because i wanna keep it inline with the MongoCursor API
[21:27:08] <jonwage_> kchodorow: Derick so this ended up being my fault, I have a Doctrine\MongoDB\Cursor that implements countable
[21:27:27] <jonwage_> so doing $cursor = $collection->find()->limit(5); echo count($cursor); gives me results > 5
[21:31:43] <bjori> kchodorow: sorry.. thought jonwage_ was right for a second there and was gonna blame you for some weird intercept magic :)
[21:32:27] <dstorrs> kchodorow: <voice = "maternal"> and if he started jumping off a bridge, would you do THAT too, hmmmmmmmm? </voice>
[21:46:21] <nimijr> So when upgrading to a sharded server, does the data auto migrate between the two servers?
[21:51:30] <nemothekid> bjori: I just don't see any data on the second database, the database wasn't even created
[21:53:11] <bjori> nemothekid: you should be able to see the migrations from your server A if you check the logs carefully enough
[21:58:34] <nemothekid> bjori: From the logs alone I can't tell. I see accepted connections from my config box. Is there any other way I can tell the existing data is already being sharded. Thanks for your help btw