Log file Viewer
#mongodb logs for Wednesday the 22nd of August, 2012
[00:05:20] <nooga> i'd like to find certain documents and retrieve only a subset of fields that are in the objects in the array
[00:16:08] <geoffeg> how do i get a secondary to reconnect to a different secondary for syncing? i just took one secondary into recovery for compaction and there are two other secondaries syncing off it. which are now falling behind
[01:51:36] <doxavore> To be sure I understand correctly, using the 10gen apt repo, you can only install the latest version?
[01:52:15] <doxavore> So do people actually use that, or does everyone just install the same version from the binary zip to keep their servers running the same version?
[02:48:56] <vsmatck> svm_invictvs: wassssssup! .. Mongo uses a memory mapped file. If you keep your database smaller than your physical memory it's basically in-memory.
[04:02:25] <svm_invictvs> If I have a document like {$id:"some_id", foo:"bar"} and I want to ensure that for all documents foo is unique, how would I go about doing that?
[07:59:20] <ShishKabab> A unit test I wrote just broke in a way that puzzles me. I think it is best explained by some (Python) code. Could anyone tell me why this fails? http://pastebin.com/iJNcKrP9
[08:00:10] <NodeX> are you likely to be doing 250million inserts in a 4gb windows 7 box constantly ?
[08:13:04] <cubud> I'd be likely to scale it up to about 32GB, but I would expect Mongo to degrade performance rather than to crash
[08:21:20] <cubud> Using YouTube video comments as an example. If a web page were to show a list of 10 videos including "Total number of comments", "Total thumbs up", and also "Total thumbs down" would I run a map-reduce on the action data (liked, commented, etc) or would it be better to periodically count them up and store them in the Videos collection object?
[08:22:09] <cubud> I don't know yet, just trying to understand options so as to make an informed decision :)
[08:23:31] <cubud> So a new comment would go into a comments collection + store the most recent 20 in the video? Or do you mean store the comment count in the video?
[08:23:35] <NodeX> I would strongly suggest you store the count of the comments in the document too as count() of collections when using queries is costly
[08:24:08] <NodeX> pop the embedded comments, $push the comment, also add the comment to a comments collection
[08:27:37] <cubud> I am just reading a page on conflict resolution, it seems that there might be a programmatic way of dealing with 2 comments added to the same video on different DB servers
[08:27:46] <NodeX> it does mean you'll have 20 comments of duplicate data for every video but it stops the need for a second query and HDD's are cheaper than RAM so it will scale more efficiently
[08:29:37] <cubud> Cassandra let's you connect to any and then they distribute the data amongst themselves
[08:30:13] <cubud> So you could have servers in different parts of the world and they would be eventually consistent
[08:31:37] <NodeX> IIrc you -can- connect to a specific shard and read/write but I dont know what knock on effect it would have on your data
[08:33:15] <cubud> The purpose is for load balancing. You could have servers on opposite sides of the world serving IP ranges geographically close to the client and then replicating between themselves. It also means there is no single point of failure because the client will try another master node if it is unable to reach the one it used last
[08:33:39] <cubud> It also receives load statistics from the master and so knows which server will give the fastest response for the next request
[08:36:05] <cubud> Yes it will when I get that far, I am just participating in idle conversation atm :)
[09:01:24] <mbuf> is there a recommended way to run tests for a Rails project that uses MongoDB? any specific object mapper that you suggest that can also be used with the testing?
[09:03:48] <mbuf> NodeX: right now I am using Mongoid, but, I couldn't find documentation on running tests
[09:04:12] <mbuf> NodeX: someone mentioned that it is sort of difficult to mock the calls to mongo database, IIRC
[09:09:10] <mbuf> NodeX: just to keep a check on my code, have Continuous Integration going, for example
[09:09:57] <algernon> mbuf: dunno. I don't follow any rails stuff. I can show you C & python examples only, but it really is stupidly straightforward.
[09:12:09] <algernon> mbuf: https://github.com/algernon/mojology/blob/master/mojology/tests/browser.py & stuff under https://github.com/algernon/libmongo-client/tree/master/tests
[09:12:47] <algernon> output of the latter can be seen at http://travis-ci.org/#!/algernon/libmongo-client/jobs/2141943 for example.
[09:46:15] <NodeX> so you'll allways know... you can then do in one query $pop : where your count $gt : 20
[09:47:14] <cubud> yes I suppose I could run two queries couldn't I? Push the comment, inc the CommentCount, then another query to pop a comment but with a filter "CommentCount >= 10"
[09:47:45] <NodeX> It's probably more efficient to always push then every so often just pop the last few
[09:48:14] <NodeX> you're only every going to read the last 20 anyway so what does it matter if an extra 5 creep through if it maintains a faster app
[09:49:19] <cubud> What I might do is add them anyway, and have a thread which selects the videos with the highest RecentCommentCount then pops and resets the RecentCommentCount to 0
[09:51:12] <cubud> If I have a Video collection with the info in, then a VideoComments collection I could simply push the latest comment to the Comments list and inc CommentCount, and then when paging select a slice
[09:54:06] <NodeX> take my advice and store the first page of comments in both the Video document and the comments collection
[09:54:46] <cubud> Yes, considering what you have just told me about a 16mb limit and having to read all comments into memory even when splicing I think that would be best
[09:56:06] <cubud> only last 10 in video rather than all comments in a single document in another collection
[09:58:51] <cubud> The web app yes. The DB initially, but might move it to Linux if the load is high
[10:04:19] <cubud> By "Domain" I mean that I currently load a C# object in its entirety, update stuff, save it all. For Users this is okay, but for other stuff (such as Video) I will need my app to update individual column values only
[10:07:20] <_johnny> [11:57] < NodeX> I can tell you;re a windows based programmer from the CamelCasing <- lol, NodeX :)
[10:15:01] <_johnny> btw, i converted all my xml data (3gb) to json (800mb). roughly 2,4 mio docs, imported in half an hour. i have restored faith in mongoimport :)
[10:16:00] <_johnny> the biggest time delay was my ignorance (big surprise, huh) though. i was querying the upper case names with a compiled regex, rather than just the regex itself
[10:16:30] <_johnny> compiled added 0.5 secs of latency! removed it, got 0.008 in average on queries. d'oh
[11:15:40] <kali> justin_j: do you expect to perform operation on them inside mongodb ? or just store and retrieve ?
[11:17:49] <justin_j> and large, I'm talking up to 50,000 bits which is just over 6kb - so not that large :-)
[11:19:18] <kali> justin_j: the document size limit is 16MB, so if they grow much bigger, you can use gridfs
[11:21:38] <kali> I'm not sure what the exact terminology would be, I guess it depends on the driver... It's called Binary in java for instance
[11:36:01] <kali> justin_j: well casbah is a thin lay over the java driver anyway. for Binary, just use the java class
[11:54:03] <Littlex> NodeX: Sorry for my late reply, well the testsystem is just a small machine, without any action mongo generates a load of 0.7
[12:07:08] <Littlex> NodeX: in the current state its empty, with data it would contain about 2,2 gig
[12:08:27] <Littlex> [root@... logs]# ps -eo pcpu,pid,user,args | sort -k 1 -r %CPU PID USER COMMAND 1.0 20480 root ps -eo pcpu,pid,user,args 0.7 1870 mongod /usr/bin/mongod -f /etc/mongod.conf
[12:14:10] <NodeX> I'm just guessing because I would never run mongo on a 1gb RAM box but I would hazard a guess that some of the internals that go on normal in RAM are happening on disk
[12:15:18] <NodeX> 16.3 2337 mongodb /usr/bin/mongod --dbpath /home/mongodb --logpath /var/log/mongodb/mongodb.log run --config /etc/mongodb.conf
[14:40:50] <ShishKabab> Is there any way to query on a subobject without key order mattering? Because I use Python with Pymongo, the key order of my objects are not guaranteed. So if I do a query like db.coll.find({x: {y: 1, z: 2}}) it randomly fails. See http://pastebin.com/Mwpqx72g .
[14:42:21] <circlicious> can someone hlp me with this - https://gist.github.com/53039e8b3f209759d091 ?
[14:44:57] <algernon> circlicious: you'll have to do that client side (or use m/r), I think. Unless the new(ish) aggregation framework can do that.
[14:46:28] <algernon> looks like the aggregation framework can help, at a first blink: http://docs.mongodb.org/manual/applications/aggregation/
[14:47:49] <circlicious> oh NodeX was talking about this ? ok will have to read. but i am not using 2.1, i am using 2.0.6 i think, algernon
[14:49:04] <circlicious> for example, i wanted to filter data from the m/r results and get all my document fields
[14:50:14] <circlicious> first of all, i want all fields of document, does that make sense ? secondly i wanna filter the result set, where count > 1
[15:28:56] <scott1> I tried posting a question to the google group yesterday, but it appears not to have shown up. Is there some kind of moderation that I'm not getting to, or am I just delusional about having posted?
[15:32:50] <Derick> scott1: first post by a new person is indeed moderated - sadly, I don't have access to the moderation queue
[15:55:36] <jQuy> Hi all! I would like to find a good GUI for MongoDB. Can you help me? I have Windows 7.
[16:03:45] <Vile> I need an idea. I have an hierarchically arranged collection (using materialized path). I need to do a m/r on it, but… for proper processing of each document i need all of its parents
[16:07:27] <Vile> basically, i need to emit parent node for each of its direct and indirect child nodes (and for itself of course)
[17:51:56] <thedahv1> Might be nice to throw some ideas against the proverbial wall and see what sticks
[19:25:46] <jallan> if I have a field that holds a string, is there any easy way for me to go in and edit just part of the string?
[21:42:04] <dufflebunk> Is there a developer for the mongo Java API around? I think there's a bug in the RC2 version.
[22:19:13] <joshontheweb> basically I'm trying to build a list of an objects children when you query the object
[22:28:46] <owen1> i have a collection called questions with array of answers. how can i query for all answers of a specific user?
[23:07:36] <svm_invictvs> If I have a an object and I want to constrcut a query which will return that exact object, how would I do that?
[23:20:04] <vsmatck> It would be a object that contains one or more fields of the object you're looking for.
[23:20:48] <vsmatck> Like if you had a user object your query object might be a object with a UID, or username.
[23:22:17] <svm_invictvs> vsmatck: Well, what i'm trying to do is rig up the atomic compare and swap.
[23:26:46] <vsmatck> Ah, I see what is going on there. In that first example it's fetching the document and decrementing that "qty" field.
[23:27:09] <svm_invictvs> So I'ma ssuming that I'd basically do something like, original = find("_id":"myId"); changes = original.copy(); /* make changes */ insert(original, change, false, false);
[23:27:10] <vsmatck> The update command is trying to find a document with a specific _id and a specific qty. If it can't find that document it won't update.
[23:40:22] <crudson1> owen1: so you want not just the full documents that a certain user is in, but just their bits in answers array for all documents
[23:45:01] <rboyer> if w=majority is specified for a write, is that measured off of the same N as the majority vote calculation for Election? or something else?
[23:45:25] <rboyer> my gut would say it's the majority of N, where N is the total number of electable members (master, and non-hidden secondaries)
[23:47:24] <crudson1> owen1: if so you could do something like: db.u.aggregate({$match:{'answers.user':'josh'}}, {$unwind:'$answers'}, {$match:{'answers.user':'josh'}})