Log file Viewer
#mongodb logs for Wednesday the 13th of June, 2012
[01:02:24] <tystr> db.coll.update({ "_id": ObjectId("4fd7de77ab3c45e565000021") }, { "$pushAll": { "attributes": [ { "k": "favorite_foods", "v": "beer" } ] } });
[04:54:25] <heoa> Are the instructions here [1] outdated? What is the deb -cmd there? [1] http://docs.mongodb.org/manual/tutorial/install-mongodb-on-debian-or-ubuntu-linux/
[05:15:50] <Kane`> i'm looking into combining mongodb + postgresql to store my log data. any thoughts on my schema? http://codepad.org/c1zxZqfL
[07:57:32] <algernon> Kane`: why mongodb AND postgre? For logs, mongodb itself is enough, and not having to use two dbs makes things a whole lot simpler.
[08:39:04] <Kane`> algernon, i'm playing around with using just mongodb now. will see how it works out :}
[08:58:48] <timoxley> how do people deal with things like "when canonical data X changes, reflect changes in caches by updating A, B and C"
[08:59:42] <timoxley> my first idea was 'triggers', but this isn't yet implemented https://jira.mongodb.org/browse/SERVER-124
[09:04:07] <timoxley> Derick possibly, I had hoped to avoid tailing oplog, but perhaps it's the only way
[09:04:25] <timoxley> it sorta feels like sorting through someones garbage to see what they've been eating
[09:09:06] <NodeX> tailing cursors and adding an entire document to SOLR is a very bloated approach... if one doesn't need to search every field in the document then it should not be indexed
[09:09:49] <timoxley> NodeX yeah, but we're using this mongoose thing… with which you can easily sidestep the pre/post save hooks by doing bulk operations… or using update syntax directly, or using the driver directly. I feel like there's little point having post save hooks if it's easy to forget about them and have them not fire
[09:10:34] <timoxley> I've actually been struggling a lot with mongo recently as a result of not being able to know when particular events happen
[09:10:36] <NodeX> for me I just added a quick function to my update -(upsert) which took the fields I wanted and posted them to solr
[09:11:21] <NodeX> a very good tip I learnt with solr is if you have a large amount of docs to add o
[09:12:00] <Derick> timoxley: not the point. you can use the same thing to update your caches of course
[09:12:31] <Derick> NodeX: And of course, you don't have to index every field... your script should handle that ;-)
[09:37:39] <timoxley> Derick tailing oplog wasn't so scary after I looked at it… I think it's the way forward for me. thanks for the help
[09:43:56] <ro_st> performance-wise, is it better to load a whole document into memory and query that data in-memory, or better to query just the bits that i need each time i need to check something?
[09:44:38] <ro_st> i ask because i've got to check access control and status flags on a large document prior to passing back large parts of it
[09:45:43] <Derick> ro_st: the only difference is that if you request fewer fields, less data traffic happens
[09:45:45] <ro_st> the difficulty is that the checks need to be used for a wide variety of actions, some which don't need the large data blob and some which do. so i don't want to fetch the blob into memory just to perform the checks and then dump it
[09:47:43] <ro_st> so if i tell it only to return some fields, that filtering happens in the mongo process, or in the driver outside the mongo process?
[10:44:09] <timoxley> Derick a friend said "The biggest problem with reading the oplog to perform hooks is that by the time you read the oplog the database wont be in the same state as it was in when the update was run"
[10:47:35] <timoxley> For example. I want to take N most recent comments for a "Post" and embed them into the Post
[10:48:06] <timoxley> Derick whenever a new comment is added, I want to run the hook that adds the item to the post
[10:49:45] <Derick> but if you want to do things like "10 latest posts" I suggest you do that in your app
[10:50:55] <timoxley> Derick again, problem is I can't definitively "know" what the 10 latest posts are without making an expensive query
[10:54:37] <timoxley> 15 seconds is pretty extreme, the whole application could have changed by then…
[10:56:52] <Derick> You want your latest X posts added to the object. But you don't want to do that query to find out the X latest ones when you do a post. Right?
[10:58:20] <Tobsn> if it would be 1.6 i wouldve said that there was a major flaw in the replication.. i had in a single server basically 0-2ms and on a two server replica i got 50-120sec
[10:59:52] <Goopyo> do you guys turn on safemode by default? Or is it not needed when theres journaling?
[11:00:14] <Derick> Goopyo: two totally different things. You probably want safe mode on, journalling on, but fsync off.
[11:00:45] <Derick> with safe mode being off, you will never find out about f.e. duplicate key errors, or data not being stored... etc.
[11:01:10] <Goopyo> I thought safemode protects from a write operation loss during a server death in the middle of the write operation, and thought thats what journaling does too
[11:02:15] <Goopyo> Derick: besides parsing errors which are caught even when safemode is off, what problems can be caught with safemode?
[11:02:51] <Derick> Goopyo: duplicate keys for unique indexes, a primary disappearing or reverting to a secondary
[11:03:10] <Derick> Goopyo: the client will *never* check error conditions in data updates/inserts/deletes with safemode off
[11:17:36] <statusfailed> I'm using C#, is it possible to do a query, apply a function to each result, and save all the results?
[11:17:41] <spillere> how can i add a for to create a sub object? idk if its the right way to say it, but can check here: http://pastie.org/4079215
[11:17:48] <statusfailed> Basically I want to update a whole bunch of things at once, and i'm not sure what to use
[11:18:50] <statusfailed> At the moment i'm using foreach, AsQueryable, and writing doing many writes to save results
[11:29:42] <statusfailed> Goopyo: haven't quite figured it out, but I will keep looking- thanks for your help!
[11:30:44] <Goopyo> what I am suggesting you do is apply the function to the data, append it to the array and once you've done all of them, do a batch insert of the array that way its one trip and one operation
[11:33:30] <statusfailed> Sorry, I just meant i'm having trouble figuring out how to get the "upsert" to happen
[11:34:18] <Goopyo> are you updating the data or saving it? if you're saving it you dont need a upsert, if you are updating and want to insert if its not there, you upsert
[11:35:59] <statusfailed> Ok, so there's a collection of items in mongodb, I want to modify each item in that collection
[11:36:14] <statusfailed> Currently, i'm reading them all from mongo, modifying them all, and then saving back
[11:38:57] <Goopyo> then you can either do them individually or if you can categorize them do a update with multi flag set to true
[11:41:00] <Goopyo> it depends on your situtaion. Say you want to take all documents that have 'john' in their name and you want to set name_john to true in all of them you can simply do a db.collection.update({'name' : 'john'}, {'$set' : {'name_john' : True }, multi=True)
[11:43:00] <Goopyo> statusfailed: another option is server sided code: http://www.mongodb.org/display/DOCS/Server-side+Code+Execution#Server-sideCodeExecution-Using%7B%7Bdb.eval%28%29%7D%7D
[11:43:40] <statusfailed> Oh I see- i'm trying to hide the "mongo" layer from other C# code, so the actual function doing the modification is in C#
[11:45:17] <Goopyo> statusfailed: what do you mean by "But I can still do a batch save of the updated collection, right?"
[11:57:14] <pilgo> Hi All. I have a collection as such: {name: String, users: [user_id, ...]} and I want to find all the documents that have a specific user_id. I tried this: Lists.find({ users: uid }) but it doesn't work
[12:06:06] <NodeX> [12:54:49] <pilgo> Hi All. I have a collection as such: {name: String, users: [user_id, ...]} and I want to find all the documents that have a specific user_id. I tried this: Lists.find({ users: uid }) but it doesn't work
[12:06:40] <Goopyo> NodeX: lol relax man, I am asking for your professional second opinion not giving you blame
[12:09:02] <NodeX> the general principal is go with the least amount of disk reads .. if you query the overall document more than embedded arrays then scope them elsewhere else dont
[12:09:05] <pilgo> NodeX: my doc will look like this: {name: "my awesome list", users: ['dsfdssdfds', 'dsfdasdsfd','dsfdasfd']} . I want to find all the lists that have 'dsfdasfd' in the users.
[12:11:25] <Goopyo> you are better of creating documents like {'userid' : <SINGLE USER ID> , falls_in_list = Name of list they fall in } that way you can query both ways equally as fast
[12:12:56] <NodeX> he hasn;t explained HOW he queries his data, or which is queried more so no assumptions can be made
[12:13:17] <Goopyo> he did: {name: "my awesome list", users: ['dsfdssdfds', 'dsfdasdsfd','dsfdasfd']} . I want to find all the lists that have 'dsfdasfd' in the users.
[12:14:36] <pilgo> Ok, so for how I query, I need to find lists that belong to users, tasks that belong in a list
[12:17:02] <Goopyo> NodeX: your right that direction and frequency matters, B ut many to many is the best way to go, especially considering it does do query by users in the list sometimes
[12:17:19] <NodeX> if you're just asking for all documents users.foo = bar then embeded is probably best
[12:19:13] <Goopyo> pilgo my vote is for: Lists - {name: 'Happy' , users: user1}, (name: 'Happy' , users: user2} repeat for as many as you need, index by list name
[12:25:04] <Goopyo> I was saying he should make list a many-to-many kind of design like: {'list_name' : name, 'user' : single_user } and make multiple entries for each list
[12:26:16] <NodeX> and even though you're on ignore ...[13:18:44] <NodeX> more queries = slower performance <---- is fully correct
[12:30:13] <NodeX> as I said earlier .. if you just want names of lists for a given user then a relational collection {user:foo, list_name:bar} will probably work best
[12:30:40] <NodeX> index on user, and if in future you need to get all users for list_name:bar then index on list_name too
[12:47:37] <Goopyo> NodeX: That statement is wrong absolutely. It depends on the query types, indexing, etc
[13:09:09] <remonvv> "I want to do X", "But kind sir, this program only does Y", "Well that's just stupid!"
[13:59:30] <stevie-bash> Hello, I see a lot querys/updates (db.currentOp) which have no namespace "ns" set. How can this happen?
[14:10:48] <ro_st> is there a way to return a document based on whether or not an array within it has a length > 0?
[14:40:36] <remonvv> W0rmDrink, 2.0.6. I'm not aware of any regression issues and if there were any they'd be top priority fixes I would assume.
[14:45:29] <W0rmDrink> is this something I have to worry about https://jira.mongodb.org/browse/SERVER-5988 ?
[17:18:10] <diversario> I have a question about nodejs driver. Specifically, I want to stream results of multiple queries, one by one, somehow.
[17:26:01] <grizlo42> my assumption would be replace, but is there any way to get a merge functionality?
[17:37:31] <spillere> one thing i find really annoying on the mongo is that when I press up key, it doesnt go to the last command
[17:46:21] <algernon> spillere: rlwrap mongo. it will do magic, and make the shell have history and up/down keys working. usually, anyway.
[17:58:32] <diversario> Does node-mongodb-native support streaming from multiple cursors at once? From same collection over 1 connection.
[18:12:22] <mkmkmk> is there a way to specify which shard i want to be the primary for a database (or collection)
[18:26:27] <edvorkin> { "uid" : "1234567", "activity" : "search", "value" : "test article2", "date" : { "$date" : "2011-05-03T17:09:55.359Z" } }
[18:26:34] <edvorkin> This is because of my usage of "$date". What I am doing wring converting into mongoDB date?
[18:26:53] <kchodoro_> mkmkmk: no, but you can use the moveprimary command after the fact to change it
[19:03:49] <shadfc> what are people using for offline processing of large amounts of mongo data? it looks like the mongo-hadoop connector queries mongo every time... which i'd like to avoid
[19:10:58] <spillere> I have a save I wanna do, where I will save a userID, and on it's user, a bunch of checkins, like in foursquare, how do I make subobjects?
[19:11:03] <spillere> db.users.save({'name':'daniel', 'checkin': { 'name1': 'name1', 'geo1':'geo1' }, { 'name1': 'name2', 'geo1':'geo2'}})
[19:12:28] <spillere> db.users.save({'name':'daniel', 'checkin': [{ 'name1': 'name1', 'geo1':'geo1' }, { 'name1': 'name2', 'geo1':'geo2'}]})
[19:12:39] <Killerguy> hum I thought flock on primary would prevent all shard to do write, and then I could do backup
[19:14:23] <kali> Killerguy: you can backup, the low-level way, copying the files or making a FS-level snapshot
[19:16:20] <kali> Killerguy: by the way, where are you ? "héhé" sound french, but the whois hints farther north
[19:18:17] <Killerguy> got a question for config server backup, is that really good to offline one of the three?
[19:30:18] <kali> Killerguy: when we have to do this, we usually stop one, copy the files to the new server and start it :)
[19:31:19] <kali> Killerguy: at least, with one stopped, you know the state will not evolve, so you don't have to worry about consistency
[19:46:04] <multiHYP> how does bson look like? lets say bson of: {"name": "multiHYP", "occupation":"programmer"}
[19:53:29] <shadfc> anyone know if mongo-hadoop can read from BSON instead of querying mongo directly? current docs seem to say no, but i've seen some forum posts which seem to indicate it is possible
[20:15:23] <zirpu> anyone have a good rule of thumb about how much new data a replicaset can take? i keep overflowing the oplog and want to know if there's a way to avoid that.
[20:16:05] <zirpu> it's just log data (access and gc logs) that i'm throwing in. apparently too quickly. so if there's a way to back off and let the replicas catch up that's what i'd like.
[20:16:40] <zirpu> it's a 3 node replicaset. i have w=2, mostly to try to slow down the clients. w/o it i nuked the replicas w/in a few hours.
[20:22:28] <UForgotten> so I disabled authentication on my mongo instances and now my mms monitor agent is failing that it can't authenticate - how can I remove the credentials from the agent? I tried resubmitting the form empty but that does not appear to fix it.
[20:58:04] <ragnok> hi, i installed mongo and it was working fine with my node.js app. then i had to install the lamp stack in the same server. now i tried to run my node app, and i have some error from mongo. is there a possible conflict between mysql and mongodb in a ubuntu 10.04 ?
[21:43:43] <ribo> so, I'm working on a diaster recovery system, my production cluster has 6 nodes, (3 repl of 2 shards). If my disaster recovery server is just a single node, is doing a mongodump from a mongos gateway then a mongorestore on that single node backup server a good enough way to achieve this?
[21:46:51] <pmxbot> !8ball (8) !acronym (ac) !anchorman !annoy (a, bother) !bender (bend) !bitchingisuseless (qbiu) !blame !bless !boo !bottom10 (bottom) !calc !chain !cheer (c) !c
[21:46:51] <pmxbot> ompliment (surreal) !ctlaltdel (cad, controlaltdelete, controlaltdelete, quit, restart) !curse !dance (d) !deal !define (def) !demotivate (dm) !disembowel (dis,
[21:46:52] <pmxbot> eviscerate) !duck (ducky) !embowel (reembowel) !excuse (e ) !featurecreep (fc) !fight !flip !fml !gettowork (gtw) !golfclap (clap) !google (g) !googlecalc (gc)
[21:46:52] <pmxbot> !grail !hal (2001) !hangover !help (h) !hire !imotivate (im, ironicmotivate) !insult !job (card) !karma (k) !keelhaul (kh) !klingon (klingonism) !logo !lunch (
[21:46:52] <pmxbot> lunchpick, lunchpicker) !meaculpa (apologize, apology) !motivate (appreciate, m, thank, thanks) !murphy (law) !nailedit (n, nail) !nastygram (bcc, nerf, passive
[21:46:53] <pmxbot> ) !notify !oregontrail (otrail) !panic (pc) !password (passwd, pw) !paste !pick (p, p:, pick:) !progress !quote (q) !r (r) !resolv !roll !rubberstamp (approve)
[21:46:53] <pmxbot> !saysomething !simpsons (simp) !stab (shank, shiv) !storytime (story) !strategy !strike !tgif !therethere (comfort, poor) !ticker (t) !time !tinytear (cry, tear
[21:46:53] <pmxbot> , tt) !top10 (top) !troutslap (slap, ts) !urbandict (ud, urb, urbandef, urbandefine, urbandictionary, urbdef) !weather (w) !where (last, lastseen, seen) !zinger
[21:47:04] <millun> i am wondering about a little design thing. i have a table of Records. they can be of 2 types. now, lostRecords extend records class. so do foundRecords (@Entity). so i have 2 tables. i was wondering if it would be better if i marked both extended classes of mine this way: @Entity ("records"). so i would have 1 table (collection)
[21:52:41] <dstorrs> when I do findAndModify -- does it lock the entire collection throughout the request, or can it contain the lock + scan to indices such that only records that might be matches get locked?
[22:41:34] <dstorrs> does anyone know -- is there a notable speed diff between findAndModify()'s update ability and a simple db.coll.remove({ ... })
[22:42:48] <kenneth> btw mongo people, i just sent y'all a pull request to fix some small issues in the c driver: https://github.com/mongodb/mongo-c-driver/pull/45
[22:44:07] <heoa> kenneth: I want to open an HTML page and when it loads I want to execute "db.scores.save({a : 99})", for example.
[22:44:20] <dstorrs> I'm using Mongo as a job queue. Once I'm done with a job, I could set it to say "state: finished" or I could simply delete it
[22:47:37] <kenneth> dstorrs: that sounds like a bad idea, using mongodb as a job queue. we went that route and regretted it, it doesn't scale. removing jobs is too expensive
[22:48:45] <kenneth> i'd recommend using redis or a dedicated in memory job queuing system (beanstalk, gear man, resque)
[22:50:54] <kenneth> dstorrs: resque is pretty neat if you're only using php/ruby or don't mind implementing their relatively simple protocol
[22:50:56] <dstorrs> It would crash with an uninformative error in the driver, there's no support available, hardly any docs, etc
[22:51:38] <kenneth> but that's more of a communication library and you have to hold the queued jobs somewhere if you want to persist them
[22:52:00] <heoa> http://www.mongodb.org/display/DOCS/Http+Interface <--- is this tutorial correct or am I running wrong cmd?
[22:52:20] <kenneth> dstorrs: does it need to scale? if your traffic is small you;ll be fine on mongo
[22:52:28] <heoa> "This interface is disabled by default. Use --rest on the command line to enable."
[22:52:37] <kenneth> but if you're hammering it with hundreds of jobs per second like we were, it won't work
[22:54:17] <dstorrs> for each channel I process 1 job per page of the feed. most feeds are 1-4 pages, but a few are up to a 1000
[22:56:05] <dstorrs> ok...are you saying Mongo isn't going to support being our database at all, or just that it can't be the job queue?
[22:57:29] <dstorrs> I really did not want to have to introduce more infrastructure at this late date.
[23:04:14] <heoa> https://github.com/christkv/node-mongodb-native.git <--- I think I need mongodb driver with node.js to do what I want ?!
[23:18:27] <heoa> http://ajay555.wordpress.com/2011/06/08/mongodb-on-ubuntu-rest-is-not-enabled-use-rest-to-turn-on-error/ found solution in 2011 but not working anymore or conf -file changed...
[23:22:06] <heoa> (I got it running by adding line "rest = true" to the /etc/mongodb.conf, just guessed)