Log file Viewer
#mongodb logs for Wednesday the 25th of July, 2012
[00:31:49] <krz> using the aggregation framework. how do i return all visits with minute greater than 12 ?
[00:33:48] <crudson> krz: as they are within an array of a single document, you need to $unwind first
[00:41:15] <crudson> oh i saw that as an array, not a hash. you should probably avoid using variable values as hash keys as they become hard to query against and manage. Also is there a reason visits is a hash rather than an array? It's much easier to have [ {id:'134300236111rcbmbmvv','country_name:'Europe'},{}]
[00:43:39] <crudson> krz: but the document structure is my opinion. In my experience using { key: '12345', value: 'abcde' } is much preferable to { '12345':'abcde'}
[00:47:06] <crudson> krz: personally I'd address the document structure to facilitate querying, but others may have other suggestions. What was the ruby question?
[00:47:57] <krz> crudson yea so based on this structure. how am i able to extract only the visits with "minute" greater than 12
[00:50:12] <crudson> krz: that's the only native operation in aggregation to split a single document's attribute values
[00:50:35] <krz> crudson: btw, http://www.10gen.com/presentations/webinar/real-time-analytics-with-mongodb slide 19
[01:10:53] <crudson> yes, but their keys are static except for some related to hours and minutes, which they are handling as a special case. You are asking to get a list of document fragments (i.e. particular {key,value}s), and on brief examination your document format is not going to make this easy with aggregation. You could map reduce, emitting id=>document if minute>12 and have a null reducer.
[01:15:43] <krz> such reasons are also highlighted at this webinar: http://www.10gen.com/events/new-aggregation-framework
[01:19:29] <crudson> krz: I have no further advice other than to rethink your document design if you want to use the query type you suggest, or you just filter the nested documents on the client side.
[01:21:59] <krz> crudson: lets take it a step back. whats wrong with this then https://gist.github.com/3173689
[01:29:15] <crudson> krz: I loaded your document into a collection and ran that command fine. check your db and collection names
[01:40:55] <crudson> I ran the following and got a single document in 'result' db.command(:aggregate => 'uw', :pipeline => [{'$match' => {:_id => '20120723/foobar/song/custom-cred'}}])
[01:57:05] <krz> crudson: ill consider changing the structure to an array instead. just to understand this further, can you show me more or less how the visits should look like in array form?
[02:05:29] <crudson> krz: something like this. http://pastebin.com/zf3BcqpT note that I let mongodb assign its own _id.
[02:10:05] <crudson> krz: then execute: db.uw.aggregate({$unwind:'$visits'}, {$project:{visit:'$visits'}}, {$match:{'visit.minute':{$gte:12}}})
[02:13:49] <crudson> krz: this will give the result you are looking for: http://pastebin.com/tzdyVFeS
[02:30:56] <krz> crudson: how would i perform the aggregation on this to return visits with minute greater than 12?
[02:32:14] <crudson> krz: note whether to use your own _id or not is entirely up to you, providing they are unique
[03:42:53] <clu3> Suppose users.tags = [ [id : 1, name : tag1], [ id: 2, name :tag2] , ...] how do i remove the tag with id=2 out of users.tags array? Any help really appreciated
[04:50:44] <krz> crudson: i put the issue here on more detail with code for a better view http://stackoverflow.com/questions/11641358/how-do-i-execute-this-query-using-the-aggregation-framework-in-mongodb have a look at it when you have the time
[05:08:38] <crudson> krz: I wrote a number of times how to do that with your data model. check previous pastes and chat.
[05:09:35] <krz> crudson: we only spoke about document structure. nothing specific into filtering the results
[07:17:36] <vak> mongostat shows "locked %" 2.71e+03 in 4-core server, how come? http://pastebin.com/h7zJBF3j
[08:08:30] <kali> dstorrs: if you're still struggling with your query, there are a few quite simple document schema alteration you can make to make the indexing work better: for instance, maintain a "has_pages" boolean (<=> np>0), a "pending" boolean (<=> action != "FINISHED")
[08:09:14] <kali> dstorrs: then, make two separate queries: one for "relocking" and one for new locks to discard the $or
[08:09:57] <kali> dstorrs: and finally you need the lock_until criteria at the end of the index definition (the "range" parameter is only index-compatible in last position)
[08:11:42] <kali> dstorrs: but even with that, i think you're not out of the wood because indexing and $elemMatch are sometimes weird
[08:14:04] <dstorrs> I looked at that, actually. horizontal striping (all pages in array) seemed to outperform vertical (each in a separate doc)
[08:14:24] <dstorrs> although the code has changed enough since that it might be worth re-running the test.
[08:14:50] <kali> splitting pages and the alteration i suggested will make indexing work, i'm sure of that.
[08:15:17] <kali> and i'm quite sure you won't get efficient indexing with the array and that kind of criteria
[08:15:39] <dstorrs> does indexing actually matter? it doesn't matter what order jobs get grabbed in.
[08:17:08] <dstorrs> if there are no indices, the workers will all grab at the top, then each will have to walk past all the locked docs to find an unlocked one.
[08:19:02] <kali> dstorrs: yes. and i don't know when the write lock is actually taken: just at the time of the actual update, or for the scanning too
[08:21:23] <kali> dstorrs: but i'm happy to see i'm not the only crazy person to use mongodb for concurrency state :)
[08:24:02] <kali> dstorrs: like... everything. mongo-resque is the most important in terms of transaction, but we also have code for ad-hoc queues, tensions for multidoc updates, mutual exclusion like you're doing
[08:24:52] <kali> dstorrs: we used redis before, and loved it, but the lack of built-in failover was an operation nightmare
[08:30:17] <dstorrs> basically, I know that I was able to harvest 10M vids in 1 hr 11 mins when I initially launched.
[08:31:20] <dstorrs> (performance has since dropped due to code changes, sharding of DB, and various other changes)
[08:32:59] <dstorrs> more or less the same thing, since we're only doing one queue at a time due to the business rules.
[08:36:25] <kali> dstorrs: we're very far from these figures actually. a few dozen lock request every seconds right now
[08:38:36] <dstorrs> (not that this isn't useful, but it's a pretty thankless job and it never seems to end. it gets disheartening after a while)
[08:44:10] <dstorrs> and I'm the co-founder, so making things spin faster is very much in my interest
[08:50:16] <dstorrs> to sum up your recommendations, you're saying change the schema to look like so :
[08:51:58] <dstorrs> { _id : ObjectId(), has_pages : true | false, pending : true | false, locked_until : $epoch, (optionally: owner : $val | owner_host : $val) }
[08:57:49] <dstorrs> meaning I need to set an owner_host on every document, even if it doesn't need it?
[08:58:12] <vak> guys, vote, add watches, comments, whatever, but, please, let push this shame issue-1240 http://bit.ly/PiPkXW we don't want 3rd year celebration, or do we?
[08:59:31] <dstorrs> vak: you know that database-level is coming in 2.2 (rc's for which are out now) and that collection level is scheduled for 2.4, right?
[10:32:47] <kaikaikai> does anyone have experience running a mongodb update on each page request? i'm building a sort of custom analytics and i have two choices: use timestamps and inserts and then organizing all the data later...or using updates and keeping an embedded document
[10:33:30] <kaikaikai> there is about 400 concurrent users on average, i know i'll need to just test this but if anyone has insight it will help me choose what to try first
[10:35:18] <Mortah> looking at our sampler its not causing performance issues for us... even for 400 users at 1kb each this only adds up to 400kb so it stays in memory no problem :)
[10:35:39] <kaikaikai> ok i see, how do you check? my update would be running every single page request
[10:36:53] <Mortah> we do a get... check if the feature has already been marked as used, if so - do nothing, if not - do a $set
[10:55:10] <Derick> ankakusu: http://derickrethans.nl/indexing-free-tags.html (links to the talk too)
[10:57:23] <Derick> I'm a big OSM fan and I work with the MongoDB people... so really interested :-)
[11:01:09] <thewanderer1> hi. I'd like to have an application that depends on particular array ordering in MongoDB. if I save [1,2,3], is the order in which they are stored and retrieved preserved?
[11:03:30] <thewanderer1> and, if ordering is preserved, how can I query only for documents which have an array whose 0-th element is "green"?
[11:19:31] <thewanderer1> guys, I'm trying to find() all documents which have "pirate" as the first element of the "arr" array. it needs to be the first element, not any further. any suggestions?
[11:20:19] <ankakusu> such as: <way id='153665622' timestamp='2012-03-06T20:03:37Z' uid='609753' user='Chad Lawlis' visible='true' version='1' changeset='10893949'> <nd ref='1663030941' /> /way>
[11:24:02] <ankakusu> well, while I was writing, I realized that ref in <nd ref='1663030941' /> is just an attribute
[12:03:38] <lqez> I'm copying my mongodb into new mongodb instance by 'copyDatabase' command. (about 700M rows, 1.3TB)
[12:04:30] <lqez> I'm waiting for more 6h for '"msg" : "index: (2/3) btree bottom up 197232658/715742197 27%"'
[12:23:02] <NodeX> Bartzy : I think you'll have to run a function and setTimeout().. in the function to call the secnod count
[13:56:09] <Bartzy> So for big datasets it's usually better to use LVM snapshots or something like that ?
[14:01:39] <remonvv> Hm, beginning to realize why the CouchDB community isn't winning any prizes : Guy tweets "Yup: this is why I choose CouchDB. Macworld Developers dish on iCloud's challenges" + link to blog about some iCloud issues. Retweeted by @CouchDB.
[14:02:01] <remonvv> Nice chain of reasoning "Apple didn't do iCloud very well so now I like completely unrelated technology CouchDB"
[14:02:52] <Bartzy> kchodorow_: I already asked it here but didn't get a complete answer - I have an index on {uid: 1, id: -1}, and doing: db.shares.find({uid: {$in: [list of 10-5000 strings here]}).sort({_id: -1}).limit(50) , results in using that index, but it also shows scanAndOrder: true. Why is it not using the _id key for sorting ?
[14:07:33] <kali> Bartzy: index will only be able to sort if you're parsing a contiguous slice of the index. in your case, the $in spread the results all over the place, so the index is useless for sorting
[14:08:06] <kali> Bartzy: there are quite good presentations by 10gen staff on the index internals, it might be a good time to have a look at one :)
[14:08:42] <Bartzy> kali: I will look at them, thanks. Reading MongoDB in Action and the chapter about indexes was no help about this :)
[14:09:32] <Bartzy> I just need a list of the documents with those uids, ordered by descending time (_id, timestamp, natural)... and limit to the last 50
[14:11:40] <Mortah> I wonder if you could flip the index... but I believe the total number of UIDs you have would affect whether that would be useful
[14:13:00] <Bartzy> kali: And if I keep this structure - there is no point for _id in that index, if scanAndOrder is true, right? So I can just remove it and add a {uid:1} index ?
[14:13:21] <kchodorow_> i'm not sure if it works in 2.0, but {_id:1, uid:1} should know how to use the index in 2.2
[14:14:02] <Bartzy> kchodorow_: No need for _id: -1 ? Also, how would it know to use it , I thought sorting can be done only if is for the key after the last one used in the index ?
[14:15:53] <kchodorow_> on _id:-1: the query optimizer should be smart enough for that not to matter
[14:15:54] <kali> kchodorow_: with the sort key in first ? skipping the docs based on the index key bit ? as far as i know this is new
[14:16:20] <kchodorow_> on the reverse order: yeah, that's probably just in 2.2. but it is a new feature
[14:17:28] <Bartzy> And the performance of just getting the documents according to uid, without ordering, will be the same in {_id:1, uid:1} as in {uid:1, _id:1} ?
[14:18:34] <kchodorow_> looks like it doesn't automatically use the index if you don't include the sort
[14:20:22] <kchodorow_> well, it's like a sorted list like: [id1, uid1], [id2, uid2], ... [id50000, uid50000]
[14:21:09] <kali> Bartzy: so you scan the index instead of making random access in your whole collection
[14:21:53] <kchodorow_> it shouldn't have to, it'll sort the uids and then go from min->max uid value
[14:23:20] <Bartzy> in my scenario I'm searching for friends photos of a specific user. So all those UIDs are his/her friend UIDs
[14:24:12] <Bartzy> That will still be faster than getting all the photos of the friends, then sorting in "disk" ? I have enough RAM to hold the dataset
[14:25:40] <kali> Bartzy: another option is to completely denormalize and store a "friends pictures" history of ids in each user. every time a picture is posted, you'll have to propagate the new picture ids to all the friends
[14:33:43] <BurtyB> is there any easy way to change a shards _id? or would it be something like stopping the balancer, then in config updating shards._id and updating chunks.shard to the new name and restarting config servers?
[14:34:30] <Derick> BurtyB: and now to be more helpful, the ns*objects is just documents, the nscanned also includes index key searches
[14:39:15] <solars> hey, shouldn't this query use only indexes? https://gist.github.com/2798607b5ede61192887 or whats wrong with that?
[14:40:19] <Bartzy> I created a PHP script with that query - I measured the time for $cursor->next to come back (with microtime(true)), and it was 64ms. Then I did $cursor->reset(); print_r($cursor->explain()) and got 121 in millis.
[14:41:13] <kali> solars: no. the index must have the selecting part as a prefix of the sortering part for it to work
[14:41:31] <kali> solars: there are quite good presentations by 10gen staff on the index internals, it might be a good time to have a look at one :)
[14:43:34] <solars> does this mean the ordering part is always last? what happens if I have hotel_id and fubar_id which I use to select?
[14:43:41] <Mortah> Bartzy... it looks like its actually getting all the docs you need via the index and then doing the sort in memory... is that really so bad? sorting 5k objects shouldn't take too long... plus you could scale with more secondaries for reads :)
[14:44:44] <solars> e.g. I have: index([ [ :timestamp, Mongo::ASCENDING ], [ :hotel_id, Mongo::ASCENDING], ['rateplans.rateplan_id', Mongo::ASCENDING] ], background: true)
[14:45:10] <solars> so if I put the timestamp at the end, for ordering, does it also work if I only filter by hotel_id?
[14:45:32] <kali> solars: yes, i've seen your index. once again, take one hour, watch one of the presentation, you won't regret it if you want to understand what you're doing
[14:47:05] <kali> solars: http://www.slideshare.net/mongodb/mongodb-sharding-internals that one for instance
[16:08:47] <kchodorow_> diegok: situation improving, we hired someone to work on the driver full-time :)
[16:10:38] <kchodorow_> i might or might not get a chance to merge in your pull request before then, mostly working on 2.2 testing
[16:11:28] <diegok> kchodorow_: well, my changes are fine, but what I really need now is the ability to retry and query on secondaries mostly
[16:14:14] <kchodorow_> okay... so basically the driver gets a seed list of servers to connect to. it should call ismaster on them and populate any missing hosts (ones not in the seed list)
[16:14:43] <kchodorow_> ismaster returns either ismaster=>true or secondary=>true (or both of those fields false)
[16:15:57] <kchodorow_> the driver should also time how long it takes to call ismaster on the various servers
[16:18:05] <diegok> ok, so, it calls ismaster on startup and keep this ordered by "closer" secondaries. When it should refresh this list?
[16:19:44] <kchodorow_> let me check on how often to refresh, any driver ppl here know off the top of their head?
[17:37:03] <chubz> when i run a command like "db.isMaster()" in mongo, how can i limit the output to just a certain field?
[19:04:58] <Lawnchair_> In a sharded database's config.locks collection, what does "doing a balance round" mean? Is the balancer actually balancing the shards when it has this lock?
[19:16:23] <krz> anyone use rails and the mongo db driver? how can i run the same command https://gist.github.com/3177972 using a rails model?
[19:46:45] <tystr> we've started getting these errors in our webserver logs: MongoCursorException: couldn't get response header
[19:48:43] <Habitual> I am in need of some guidance for your package, the "mongo guy" (our client) is having one or more of "network latency, network limits, disk io speed, io wait" issues. I have been beating up sysstat tools all day. I have added counts for mysqd and mongod processes in zabbix. I really need to identify the cause. Thank you.
[19:49:12] <jiffe98> nagios is reporting the status as (null) but if I run the script with the proper arguments it checks ok
[20:45:33] <gheegh> Mongo and Ruby Question: Is it normal in my logs to see my clients connecting and disconnecting almost on a per request basis?
[21:59:19] <krz> i have the following structure https://gist.github.com/3161542. how do I find the visit with token_id "13432515303rwiaczcx" and update its minute to 1234?
[22:09:57] <krz> anyone can help me with this: http://stackoverflow.com/questions/11659334/how-do-i-update-an-item-in-an-array-in-this-document-structure
[22:18:34] <Tobsn> http://stackoverflow.com/questions/4669178/how-to-update-multiple-array-elements-in-mongodb
[22:44:47] <iwoj> It looks like you can do something like: db.orders.find({"items.sku":12345},{_id:1})
[23:10:39] <dstorrs> are there any gotchas about indexes on a sharded collection that might cause inserts to silently fail?
[23:11:02] <dstorrs> we added an index recently and suddenly many of our inserts are not hitting the DB
[23:17:46] <iwoj> but with the code example above it looks like documents can exist in more than one collection.
[23:23:00] <krz> Tobsn: with update( { visits.token_id: 'yourid' }, { $set: { visits.$.minute: 123 } } ); whats visits.$.minute ?