Log file Viewer
#mongodb logs for Friday the 22nd of June, 2012
[00:20:10] <mmlac> Is there a way to manipulate children of a model without hitting the database? Or more specific, how can I set a class ivar for all children and when they are lazily loaded call a hook (after_initialize?) on it?
[03:02:36] <jstout24> do i sacrifice speed if i have some indexed stuff not in the root top-level document?
[03:56:52] <daveluke> can i update a document where i just specify what needs to be changed, in opposed to inputting the entire revised document?
[04:26:49] <dstorrs> I've got a collection of 30,000 docs. I'd like to remove 90% of them, randomly selected. What's the best way to do that?
[04:48:27] <daveluke> and is there a callback for updates? i've been using a third parameter as a function, but the docs say thats an upset boolean
[04:48:54] <mids> dstorrs: if this is only a one time operation on such a smal set; I'd just iterate over all of them, throw a dice and delete in 90% of the cases
[04:49:44] <dstorrs> mids: is there a way to do that in the shell? I figured it out a week or so ago but since then for the life of me I can't reconstruct what I did.
[04:50:20] <dstorrs> daveluke: that's not a goal, that's a pseudocode description. What is the purpose of your actual code?
[04:50:50] <dstorrs> Guest63591: nuke a randomly selected 90% of the entries in a collection (30k items)
[04:50:56] <daveluke> i have players and lobbies.. lobbies have an array of players.. i want to put the lobby in the player object too though
[04:51:27] <daveluke> the point is that then i can later just get the player and not need to query the lobby
[04:53:01] <dstorrs> I see a lot of edge cases here -- can users be in multiple lobbies? can they be in zero? what happens if their connection drops mid-session?
[04:53:22] <dstorrs> (presumably that's why you want to persist the lobby info, but what about other state...?)
[04:53:57] <daveluke> if there connection drops, i have a disconnect event that deletes the player from the lobby
[04:55:10] <daveluke> the lobby info i want in the player, because i am identifying each player by their socket's given id
[04:55:41] <daveluke> so when i get a socket, and i want to broadcast only to players in his lobby, i can just take the ids of the players in the player's lobby!
[04:56:34] <daveluke> i'm wondering though if i need a player collection at all… if i can just do a find where lobby's players has an id of X
[04:57:19] <dstorrs> I would be more inclined to manage this as two separate collections of referenced docs. but that may be a matter of taste
[04:59:29] <dstorrs> well, if it's 4, I don't think it matters which way you do it. If it "doesn't really matter", then I'd say you better not use embedded docs -- if your site got popular, your docs would get very large, and very hard to work with / slow over the wire / possibly blow the 16M doc size limit
[05:03:07] <mids> dstorrs: db.x.find().forEach(function(o) { if (Math.random() > 0.5) { db.x.remove(o) }})
[05:03:39] <dstorrs> oh, nice. I didn't think removing from the collection you were iterating on was legit...?
[05:05:45] <daveluke> if i have an array as a field.. can i query for all docs where that array has an object with field X?
[05:09:09] <dstorrs> you can even use the $push, $pushAll, and $pull modifiers on update() to change that array in a performant way.
[05:15:21] <mids> dstorrs: alternative => db.x.remove({ $where : function() { return Math.random() > 0.5 }})
[08:17:04] <nopz> I'm using pymongo and i've got an OperationFailure: cursor id '6312762830493842190' not valid at server. I'm already using timeout=False on this request, I don't know what to do next to solve this problem.
[09:18:03] <jenner> guys, what might be the problem if I'm getting "[conn8] assertion 13436 not master or secondary, can't read ns:config.settings query:{ $query: {} }" on a former replica set primary which I restarted? looks like it lost the replica set config?
[09:47:19] <dstorrs> hey all. I've got a collection of hourly stats; there's about 30,000 users, each of whom has one stat every hour. I want to generate a "daily stats" collection with the latest stat from each day for each user.
[09:48:34] <dstorrs> "the latest stat for each day" might not be the same harvest for each user -- someone could have been missed. is there an easy way to do this?
[10:01:03] <JamesHarrison> Okay; is there any way I can get mongodb to sort by a field and be aware of a limit prior to filtering by an index to avoid a huge table scan for a small return set?
[10:04:35] <dstorrs> NodeX: ok, thanks. I'm already halfway through that, but it seems like a fair amount of code for something that is clearly a common use case. I was hoping there was a rollup I had missed.
[10:05:13] <JamesHarrison> ie "given a sorted list of most recent docs, find me 20, search at most 200"?
[10:06:00] <remonvv> I think scanAndSort : true/false in your explain() will notify you of the difference
[10:06:02] <dstorrs> JamesHarrison: see also http://www.mongodb.org/display/DOCS/Retrieving+a+Subset+of+Fields#RetrievingaSubsetofFields-CoveredIndexes
[10:06:35] <remonvv> if it scans and then sorts it will not meet your criteria, if it does a sorted b-tree walk it (accidently) will.
[10:07:38] <JamesHarrison> remonvv: currently I'm scanAndSorting, just over 16,000 entries which is a tad suboptimal, and the indexes won't reduce that since it's a sort
[10:08:19] <Derick> JamesHarrison: you can add the sorting field to the index as last part of your compound key and it should be able to use it
[10:09:33] <remonvv> JamesHarrison, perhaps post the index it's using now and youre query in a pastie so we can have a look.
[10:09:42] <Lujeni> db.foofindOne().sort({'_id':1}) , Fri Jun 22 12:04:38 TypeError: db.tag6545542.findOne().sort is not a function (shell):1
[10:10:05] <dstorrs> Derick: IIUC, if I have this index: { foo : 1, bar : 1 } then the query db.coll.find({ foo : 'bleh' }) can use the index but db.coll.find({ bar : 'bleh' }) cannot. is that right?
[10:10:43] <remonvv> a compound index A, B will overlap an index A but not B. If you query on B it cannot use an index because you don't have any index that starts with B.
[10:11:11] <dstorrs> thanks. I always thought was the case in both RDBMS and NoRelDBs, but then I used MySQL....
[10:11:55] <Derick> 11:08 <remonvv> if you change your index to {bar : 1, foo :1} both queries will hit the index
[10:12:18] <Lujeni> Guest63591, so why is compatible? how i can find first or last document with findone and sort ASCENDING / DESCENDING
[10:12:54] <dstorrs> if I have index { foo : 1, bar : 1 } is there a diff between these queries?: db.coll.find({foo:'a', bar : 'b' }) db.coll.find({bar :'b', foo : 'a' })
[10:13:28] <remonvv> I'd like to point out at this juncture that I actually agree with Guest63591 here. Sorting and returning the top document isn't that uncommon a use case. findOne() should allow sort().
[10:13:48] <remonvv> dstorrs, order of index fields is relevant (and their direction!), order in which you use them within a query isn't.
[10:14:35] <remonvv> Oh? I thought the issue was that findOne() returns a document rather than a cursor and as such you can't invoke .sort() on it.
[10:15:35] <remonvv> Oh I know but that doesn't really change the argument that much. All drivers and the shell support findOne so a consistent and usable function signature would be nice.
[10:16:00] <Lujeni> Derick, remonvv "All arguments to find() are also valid arguments for find_one()" in pymongo documentation
[10:16:22] <Derick> remonvv: sticking arguments to findOne() is a tad ugly... as I can dream up a lot more that we might have to add
[10:16:35] <remonvv> Lujeni, not familiar with pymongo but that might be an API design flaw if it implies you can sort a findOne atm.
[10:16:48] <Derick> Lujeni: not every driver is the same though. So I guess Python does support it, but the PHP one does not (and can't)
[10:17:17] <remonvv> Derick, I agree but the use case for sort is a LOT more common than, say, any addSpecial clause.
[10:20:55] <remonvv> also, a compound index that starts with a boolean will only really reduce candidate sets if it'll match only a relatively small part of the data. "hidden_from_users" sounds like an exception rather than the rule.
[10:21:30] <remonvv> I'm going to be bold and say this index : {created_at : 1} is going to give you the best results
[10:21:43] <Derick> yes, a very low cardinality isn't good, but not too much of a prob if the compound key works (which it doesn't now)
[10:22:51] <remonvv> Because having an index on an array field with a ton of possible values is going to result in a big index.
[10:23:13] <Derick> instead of having tags in an array, perhaps: { 'tagname': 1, 'tagname2' : 0 } in some form?
[10:23:37] <remonvv> Hm, I would go for a boolean that you update based on the contents of your tags array on every write
[10:24:12] <remonvv> but then you're still stuck with a double boolean compound index which has a grand total of 4 different paths
[10:26:32] <remonvv> But a quick step back. How many % of the total dataset is skipped by the hidden_from_users flag?
[10:26:33] <JamesHarrison> but how does that help in terms of reducing the collection size for $nin?
[10:27:46] <Derick> well, if that's the only query he's running the matching only has to be done on those 60
[10:28:24] <Derick> JamesHarrison: try an index on just hidden_from_users, it should make *this* case better
[10:29:37] <JamesHarrison> the hidden_from_users field is fairly irrelevant to this unless I'm much mistaken (which I may be)
[10:30:34] <JamesHarrison> we're asking for "what's not hidden from the users, and..." and only a small fraction is hidden
[10:31:29] <remonvv> index fields must eliminate a significant portion of the candidate set in the most common usecases to be effective
[10:32:51] <remonvv> Because that's "worst case" in that it'll stop scanning only if it found X elements on the sorted set
[10:33:30] <JamesHarrison> probably done by now, yeah, just a fair bit going on in currentOP so just giving it a second. ;)
[10:34:29] <JamesHarrison> so that's now using the created_at index, but still scans the entire dataset
[10:35:38] <JamesHarrison> mongodb at no point, given it can sort by the index, needs to go scanning the whole table
[10:37:20] <JamesHarrison> I am recreating by-hand my ORM's query in javascript here because my ORM doesn't give me a js query output, hang on
[10:41:23] <JamesHarrison> Huh, okay, and with the tag_ids_1_created_at_-1 and created_at_-1 indexes, it's gone for created_at_-1 and is doing well
[11:47:48] <remonvv> The sort only index was meant to scan down a sorted list but the scanning (in that case) isn't helped by any index. In other words it will scan linearly until it hits N items in your limit(N)
[11:51:22] <remonvv> Embed if you can, but you usually shouldn't in which case you have to store the _id value of the referred document in the referring document.
[11:54:06] <remonvv> MongoDB doesn't manage the relationships (relationships aren't a concept within MongoDB).
[11:54:16] <remonvv> And embedding comes with benefits and drawbacks that you need to take into account
[11:55:09] <remonvv> Google "MongoDB embedded versus references" or something and you should hit a few resources.
[13:19:31] <locojay> hi anyone using mongo-hadoop on osx? streaming works for me with dumbo... but with mongo-hadoop its chunks the collections but then fails (no avalable error) : https://gist.github.com/2969688
[14:21:29] <Guest63591> would anyone ask a question before expecting an answer to an unanswered question?
[14:23:04] <dnnsmanace> i am having trouble making unique pairs because some times the script ends up pairing two people to the same person
[14:23:40] <dnnsmanace> even though it searches for an unpaired person, by the time it updates the entry that person is already paired
[14:25:12] <Chel> question: i have a problem, my mongo client cant connect for a long time. this is a log: https://gist.github.com/7c4b7bc841ca837fb4c4
[14:26:08] <dnnsmanace> for the pair i tried setting { type: String, unique: true, sparse: true } but it keeps seeing null as a dup
[14:27:14] <dnnsmanace> there must be a general structure for databases when you pair entries on the go to ensure you don't pair to any entry except those two
[14:34:59] <dnnsmanace> or the logic behind it so the app doesnt link two users to a third by accident
[14:38:36] <Guest63591> i wonder why 75% of the people here are not capable asking a reasonable question or ask in complete sentences or providing reasonable information?
[14:42:30] <FerchoArg> haha you're right. Is there any way to do this within a query? : Suppose I have a collection of places, which has City Name, State, Street name. I want to make a serach of a regex. The regex should be compared to the three fields, City Name, State and Street, but documents in which City matches the regex should be listed first, or better weighted in some way
[14:43:18] <Chel> Guest63591: what is not correct in my question ? i have a problem, i've described it with logs and software versions. What else ?
[15:34:27] <ron> but he swears he sees the process running. you wouldn't be calling him a liar now... would you?
[15:36:56] <ron> NodeX: not really. I'm just a bit fed up of people who say 'yes' when they mean to say 'I don't need to check it since I *know* the answer'.
[15:39:20] <ron> but he was filled with shitload of sunshine, flowers, rainbows and flying unicorns!
[15:39:56] <NodeX> it painted a general bad cloud on the chan and is not good for them as a company (my opinion) and obviously thiers so he got banned
[15:39:59] <ron> zak_: please check the logs. the prompt is telling you that it has started, but it could have crashed (so to speak).
[15:41:25] <NodeX> carsten (who is on perma ignore) has done a good job of picking up the being a bad kid slack though
[15:44:41] <NodeX> idling is the internet's favourite past time, second only to the more pre-troll lurking
[15:45:15] <NodeX> whenI'm bored I like to burst into a chan and start a PHP vs the world debate or mac vs PC
[15:50:59] <NodeX> else it wouldv'e tried to kill me and realised it couldn't interoperate with anything and gave up
[16:18:08] <Pilate> Is it possible to define a function that is callable from within map/reduce without having to define it within the map/reduce functions?
[16:25:48] <sir> has anyone run into an issue with gridfs (using python driver) where you attempt to get a file using the objectid, and it says there is no file in the collection with that objectid?
[16:38:27] <pgentoo> hey guys, i'm looking at using mongo for keeping a large mysql table archived off, but still searchable. I see that craigslist does a simliar thing, and am just curious how this works. I have a lot of records per user in my system, so i can't just do a document per user, so curiuos what bestpractice is around this. For reference, think of something like bit.ly, where there are a lot of
[16:38:28] <pgentoo> users, but many many clicks that need to be kept track of per user. (i'm not working for bit.ly, but similar type service)
[16:42:35] <dnnsmanace> hey, i am having some trouble making pairing documents uniquely, and because of my query i get two different documents paired to one, whereas i am looking to have only unique pairs
[16:43:44] <addisonj> i am have really interesting performance on doing large numbers of upserts. I have ~100,000 records. I am processing 1,000 records at a time, the first batch took only .8 seconds, the 54th batch is up to ~39 seconds…is this just due to index building?
[16:44:00] <dnnsmanace> basically i want a user paired to another user, but when my query happens sometimes the script finds an unpaired user, but by the time it updates t another instance of the script already paired that user so i am getting double pairs to one person
[16:47:30] <kali> addisonj: it might be indexing. you can also check you mongod log for preallocation messages
[16:48:40] <kali> addisonj: are your upsert making the updated docs grow bigger ? that could be a factor too
[16:49:00] <dnnsmanace> the app attempts to pair a user who's "paired_email" is null to another who's "paired_email" is null
[16:51:08] <dnnsmanace> but what sometimes happens is, because first i do findOne, and then i update
[16:51:41] <dnnsmanace> findOne finds a null paired_user, but by the time it updates another instance of a pair attempt already paired that user so he is actually no longer null by the time it updates
[16:52:00] <dnnsmanace> so i end up having two different users thinking they are in a pair with a third
[16:53:35] <dnnsmanace> because it findAndModify, as far as i understand, just overwrites the whole document
[16:53:40] <kali> dnnsmanace: findAndModify will returns you the old or the new document (as you prefer) so you can check the update worked, and do something else if need be
[16:55:41] <addisonj> kali: to go in more depth, we are trying to import a few million documents daily from a partner and joining that to our own data. so our own data persists but each day the updated dat a from our partner is joined with an upsert
[16:59:03] <addisonj> kali: uhh, well mostly we are just replacing the meta-data with updated meta-data, typically, this data stays the same from day to day
[16:59:26] <dijonyummy> in pymongo how do i check if the result of a find() is null or empty? ie not found. == null doesnt work
[17:00:53] <kali> addisonj: ok, this is better... another thing to check is the %lock (run mongostats) and the page faults. if your collection and index outgrow your RAM... well, you need more RAM :)
[17:01:33] <addisonj> kali: lock percentage during the import is pegged at 100 basically all the time
[17:04:46] <charliekilo> System Information: Model: MacBook Air (13-inch Mid 2011) • CPU: Intel Core i7-2677M (4 Cores) @ 1.80 GHz • L2: 262.14 KB • L3: 4.19 MB • Memory: 4.00 GB • Uptime: 2 Days • Disk Space: Total: 249.82 GB; Free: 94.77 GB • Graphics: Intel HD Graphics 3000 • Screen Resolution: 2560 x 1440 • Load: 46% • OS: Mac OS X 10.7.4 (Build 11E53) (64-bit kernel)
[17:07:16] <kali> addisonj: for this kind of load, you will want the whole table and its index in RAM
[17:07:39] <kali> addisonj: check out the stats() from the collection and compare that to the available RAM in the sytem
[17:08:03] <kali> addisonj: on your mac, mongodb has to struggle for RAM with a kiloton of other stuff
[17:08:14] <dnnsmanace> kali: one more, how do i use $ne with more than one value? {email {$ne : [value1, value2]}}?
[17:11:40] <addisonj> hrm I am thinking there just isn't a good way to do this with upserts… just going to be slower than bulk inserts or mongoimporter, I think we may try a map-reduce or something to join the two data-sets together
[17:13:03] <kali> addisonj: if you can fit it in RAM, upsert will probably be more efficient than a m/r
[17:14:33] <addisonj> so you are saying the reason I am seeing degraded performance is just page faults huh… hrm… I will try on a big server then
[17:23:37] <mmlac> When I load an object that has_many OtherObjects, what callbacks are called when on the OtherObject? Is there a way to inject a variable into the "stub" and evaluate the "stub" before the model gets generated for real?
[17:26:41] <addisonj> yeah, even with that small of a dataset so far, the time to insert 1,000 documents after inserting 14,000 balloons to 15 seconds
[17:29:05] <jenner> guys, how can I make sure a certain node is elected as primary when I call rs.stepDown()? will a higher priority of the desired node help?
[17:31:01] <kali> jenner: yes. it will also steal primary status when you update your cofniguration if it is not primary
[17:31:48] <pgentoo> anyone have any comments on my earlier question (about structure of storing say 1billion click records tying back to say 10K total system users)? I realize its too big to store all a users clicks as a single document for that user, so just curiuos on how to attack the problem (new to mongodb...)
[17:32:56] <kali> jenner: the current primary will actualy step down when it will see somebody else has priority
[17:53:51] <sir> can anyone think of why a file would exist in gridfs, i can find it using mongo queries, but gridfs can't see it?
[19:00:39] <pgentoo> I have a single mysql table that i'd like to try moving into mongodb for testing. Currently the mysql data is on the filesystem in the form of a mysqldump file (insert statements for each record). is there some "easy" way of inserting this data into mongo?
[19:03:04] <pgentoo> I saw some tools (mongovue?) but requires the data to still be in mysql... would take ages to reimport this stuff..> :(
[19:09:21] <Azra-el> i have a listfiend of embedded fields inside my collection and I would like to know if its possible to query by one of the fields in that embedded fields and as a result get that embedded field to get more info from it. for example : http://pastie.org/4133935 I would like to find the embedded field with internal_id of 5 and get the "name" of that embedded field
[19:16:29] <mids> pgentoo: possibly you can change the file into a tsv format with some sed magic and then import it using mongoimport ?
[19:26:23] <dstorrs> hey all. I have a collection with a boat-ton of statistics (video metadata from youtube). I've got an index on { harvested_epoch : -1, publisher : 1 }. Will that index be used in the sort clause of a map_reduce that says:
[19:27:35] <dstorrs> IIUC it will be used by the query, it's just the sort that I'm not sure about -- does sort ever use indices, or is it always in RAM?
[19:35:51] <pgentoo> are there any documents out there on schema design for mongo? I have hundreds of millions of mysql records (single table) that sotre attributes about a click through my system. I'm curious how to best design the storage of this in mongo. Multiple collections, one per user? Just one huge collection? etc... Could easily be 1 billion click documents in a couple years...
[19:36:13] <pgentoo> would liek to be able to shard it out across servers to be able to scale that way in teh future, etc...
[19:37:04] <pgentoo> ok, do you have any hints on suggestions for hte design that i can keep in mind?
[19:37:57] <dstorrs> in general, be careful about embedded docs. They sound great in theory and sometimes they are the best solution, but they complicate the code a lot and introduce potential issues (e.g. "what if this grows beyond the 16M doc limit?" "how do I shard this efficiently?" etc)
[19:40:25] <pgentoo> i have clients that will be doing millions of clicks/day, so embedding those isn't possible. All my click "document" (current row in mysql) has a bunch of denormalized data already. bunch of bool values, some text strings, dates, etc. Everything i need to be able to generate my reports later.
[19:40:42] <pgentoo> in taht case, i'm assuming that each row would just be its own document, with no embedding or linking...
[19:42:23] <pgentoo> if i had a single collection of "clicks" and each click had say a "username" property or something similar that would distributed my writes/reads well, i could shard on that right? Its not sharding by colleciton right, by some value of the documents?
[19:42:28] <Azra-el> i have a listfiend of embedded fields inside my collection and I would like to know if its possible to query by one of the fields in that embedded fields and as a result get that embedded field to get more info from it. for example : http://pastie.org/4133935 I would like to find the embedded field with internal_id of 5 and get the "name" of that embedded field
[19:43:43] <spillere> I wanna add a information which is in a list inside like this {'name':name, 'photo':[{id:1, name:daniel},{id:2, name:pedro}]}, how do I add information after id:1?
[19:44:00] <spillere> so it would be {'name':name, 'photo':[{id:1, name:daniel, info:NEW},{id:2, name:pedro}]},
[19:47:44] <spillere> i'm trying an insert like this db.dataz.update({'username': 'dansku', "photos": [ {"filename":"5gzv38erpm1wxnfu4i0jq9ltk2b6yc.jpg"}]}, {"$push":{"name":"lalala"}})
[20:27:44] <sir> i'm getting a lot of "replSet syncTail: 13629 can't have undefined in a query expression" from slaves trying to replicate. i think someone tried copying bad data
[20:27:59] <sir> how do i remove all of those from the oplog on the primary so they secondaries stop trying to replicate them and failing?
[20:44:39] <spillere> db.dataz.update({'photos':{$elemMatch:{'filename':'5gzv38erpm1wxnfu4i0jq9ltk2b6yc.jpg'}}, {$set:{'photos.$.filename_tb240':'lala'}}})
[20:56:21] <php14> is it bad practice to use the mongodb assigned _id as say a username in a credential set for something?
[21:06:05] <spillere> db.dataz.update({'photos.filename':'4zbg5yqa80jol7em6nfwt2dvpuc9si.jpg'},{$push:{"filename_tb240" : "lala"}})
[21:18:02] <dijonyummy> how to specify a not equal in a find() for a property via pymongodb or mongodb in general find({"date" : not mydate}) like that but thats wrong or is it
[21:25:25] <dijonyummy> this doesnt work: "lastModified" : { $ne : os.path.getmtime(file_path) } } )
[21:34:06] <dijonyummy> its not possible to do that right? do i have to get the results, then further filter based on the lastModified being not equal?
[21:40:19] <dijonyummy> neverminded this fixed it in pymongo non-shell... "lastModified" : {"$ne" : os.path.getmtime(file_path) }} )
[23:23:28] <iamjarvo> i feel like if i could grab the whole document instead of doing multiple queries on a relational database
[23:50:11] <dstorrs> iamjarvo: what are you trying to achieve? by "like a Excel doc" do you mean an actual .xls file? or something with the data from an XLS ?
[23:50:55] <dstorrs> acidjazz: there's something in the indexing docs on exactly your problem, but I can't remember the keywords, sorry.
[23:51:24] <iamjarvo> im going to have all these data points and im going to show them in the format of an excel sheet
[23:52:02] <dstorrs> in general, IIRC, you want an index on your tags array and then db.coll.find on it
[23:54:58] <dstorrs> each row in the XLS becomes a document, each column is a key in the doc, each cell is a value
[23:56:02] <dstorrs> (obviously by "each row / column / cell" I actually mean "each of those that you're actually *using*" :>)