Log file Viewer
#mongodb logs for Tuesday the 7th of August, 2012
[01:13:10] <mappum> I have documents that are able to be "liked" by users, and I want to get a few fields from them. How can I check if a certain user has liked it without making a separate query or serving the whole "like" array?
[05:34:29] <macarthy> I have a report type model which has 1-2 million items contain a bunch of JSON data, I'll have 100k or so reports
[05:39:31] <macarthy> Each item will have an array of hashes, some deeply nested, generally queries will access the array and and the first level of the hashes
[05:49:03] <macarthy> Guess I'm thinking about this http://docs.mongodb.org/manual/use-cases/storing-comments/?highlight=embed
[05:49:47] <ron> okay, I'd really like to help but I'm afraid that any answer I'll give you will be unprofessional. I don't know what you mean by 'report' - you don't describe your application. but let's say this, assuming you have many data entries, each entry as its own document, and you want to collate those entries to a single document when you read from the database, don't hold an additional document with reference to those entries but rather query the database for those
[05:50:41] <ron> you should store the data so that reading it would be easy. assuming you come from the DRBMS world, you need to break free from some... hmm.. assumptions.
[05:52:05] <ron> depending on what you do more, storing or querying (or which is more important) - base your data model. aim for the more important one to be easier.
[05:52:38] <macarthy> and mongo can store 1.5 million * 100K * n where n is between say 4 - 10 with reasonable performance?
[05:53:24] <ron> one of the great things about mongo is that it can scale out, so if one server isn't enough, you can add another one.
[05:53:46] <ron> not saying it's a flyby operation, you need to take it into account, but it's still relatively easy to do.
[05:55:18] <ron> then you should definitely make your queries easier/more efficient than storing your data.
[05:56:04] <ron> again, it's difficult to give concrete suggestions without going into details, but I think that should give you the basics.
[05:57:06] <macarthy> lets say I an creating said report, the report has a date, author, title, should I create a report object and ref it in the items or … ?
[08:25:32] <augustl> more specifically, I want to know what I can do in the .js file. I'd like to access some environment variables, for example.
[08:34:28] <Bartzy> for(i=10000000000;i<=10000100000;i++) { db.shares.update({_id: ObjectId("4ff9c2f3733a582655000000")}, {$push: {likes: {uid: i, name: "Some Guy"} }}) }
[08:39:24] <Bartzy> Just wanted to let you know, maybe it's related to the fact that the document constantly needs to get moved ?
[08:41:42] <Bartzy> And why do you do it as a different document, and what does it have to do with mongo doing that at 120 updates per sec ?
[08:42:22] <Derick> Bartzy: we did explain yesterday that this was not going to be the best design I think?
[08:42:52] <ron> there's a 'ref' sortof, we do it in a different document since it's easier to control, and I have no idea regarding the performance.
[08:43:14] <Bartzy> Derick: But I still don't understand why is it so slow? After our talk yesterday, I decided to start checking my data.
[08:45:17] <Bartzy> Derick: The collection has 100 million documents (real data) with not likes/comments/views at all
[08:47:44] <Bartzy> The compression sucks, because the 'name' field is exactly the same on all elements :P
[08:48:11] <NodeX> and here is your trade off between UI and performance I pointed out yesterrday ;)
[08:48:30] <Bartzy> NodeX: Yeah, sorry for bailing out yesterday, I'm now in the part of realizing my mistake :D
[08:49:59] <Bartzy> Do you want me to provide a quick spec of the data before I ask more annoying questions? :)
[08:50:28] <Bartzy> I read the entire MongoDB in Action book and thought it would make me smarter! It did, but just a bit :)
[08:51:33] <Bartzy> So I have this: Pinterest like UI - each photo can be liked, viewed, commented on (each of those actions is recorded with the uid and name of the user who did it, and with comment some more data of course)
[08:52:37] <NodeX> I would save teh count of likes/views/comments with each photo, and probably save the bare minimum of meta information
[08:52:49] <Bartzy> 50 photos ("pins") per page, when you scroll down you get 50 more. Each of those photo boxes has a like/comment/view counters, and the last 3 comments. When you click on a photo, you can see all the comments, likes, views in some other UI.
[08:52:59] <NodeX> (that way you can display a page then users can drill down / paginate should they need to)
[08:53:27] <SisterArrow> Im doing a mongorestore from a dump.. Im at a specific db.collection now and when doing a db.collection.count() in the shell(on another machine that is being restored to) its reported that there are 82618776 objects.
[08:53:42] <Bartzy> Almost all (!!) of the photos will have no likes/comments/views, but popular ones can have hundreds of thousands of "views", thousands of likes... And there is a "Popular" tab so it shows 50 of those heavy photos each time :)
[08:54:13] <Bartzy> NodeX: OK I'm done explaining what are the needs, now to my (yours? :)) solution :
[08:54:49] <Bartzy> photos collection with metadata like _id, URL, image size... and - likes/comments/views counters , AND the an array of the last 3 comments.
[08:54:55] <NodeX> storing the last 50 likes/comments/views as embedded / nested data inside the photo document will take care of the initial view
[08:55:10] <NodeX> storing the rest in a separate collection will take care of drill downs and pagination
[08:56:05] <Bartzy> But now I have the issue of how to determine if the current user liked/viewed/commented on a photo (this is specified in the UI).
[08:57:26] <Bartzy> When it is in one collection (bad, like it is now), in order to know if the user liked/commented/viewed on each of those 50 photos displayed each time - We get the entire document into PHP, and in PHP we check if their UID is in likes/comments/views fields.
[08:57:26] <NodeX> it's probably less expensive to store with the user what he/she has liked/commented on (just the ID's)
[08:58:11] <Bartzy> NodeX: So a "users" collection with likes/comments/views of his, with just the _id of the photos in those arrays ?
[08:58:23] <NodeX> because you can initialise that data into the PHP session and you already have the ID of the photo so a quick "in_array()" and you know if they have or not
[08:59:11] <Bartzy> and in the (very) unlikely scenario where a user viewed 10,000 photos (in theory) - that in_array() on the "views" array of the user - will not be a problem ?
[08:59:55] <NodeX> I would persistently (with a timeout) cache it in redis personaly - easier to manage
[09:00:21] <NodeX> you can save the number of likes a user has (as a count) and if it's >N goto your cache
[09:01:09] <NodeX> having never done in_array() on 10k element's I could not speak of the performance
[09:03:50] <Bartzy> NodeX: OK - you said: you can save the number of likes a user has (as a count) and if it's >N goto your cache
[09:04:20] <NodeX> if the number of likes is greater than "N" (some large number), check in your cache
[09:15:08] <Bartzy> NodeX: But I still didn't understand why to cache like that in a non-mongo tool ?
[09:17:11] <Bartzy> I thought of just getting the user metadata (what he liked/viewed/commented on) when he first arrives at the page - save it in memcached/session/apc
[09:17:32] <Bartzy> and then for each photo, do in_array or isset (whatever) on that likes/comments/views array in PHP
[09:18:30] <Bartzy> NodeX: Also, why separate the comments to a different collection too - and not just keep them in the same document, if we assume that there would not be thousands of comments ?
[09:19:10] <Bartzy> We said that it would be best to have a photos collection and a metadata collection, that has the comments/likes/views.
[09:20:44] <Bartzy> Acceptable is some magic Mongo number of how much would be best to store nested - or is it only by my UI ?
[09:55:20] <Bartzy> I don't get why to put the last liikes/views/comments on the photos collection, and then the rest on another collection?
[09:55:36] <Bartzy> Why not only put 2 last comments on the photos collection and then everything else (including those comments) in the metadata collection.
[12:09:51] <Bartzy> NodeX: Hi again :) You think 2-3 hours of lightning consulting from 10gen will help me out with the schema design decisions ?
[12:23:47] <Bartzy> to represent a time and date - for example when a comment was created, or a post was updated - Should I use Timestamp or ISODate ?
[12:28:01] <kali> Bartzy: you want datetime (which is mapped to ISODate in shell). Timestamp is not really a timestamp
[12:28:46] <NodeX> the only reason I can think of using ISODAte is to visualy aid when viewing objects, unless there is a performance advantage I am unaware of
[12:30:46] <Derick> which gives it a range from 292 billion years in the past, to 292 billion years in the future
[12:33:59] <kali> Bartzy: no. timestamps does not make sense. datatime is good, and an integer from to epoch make sense too
[12:34:54] <kali> Bartzy: because it's meant as an internal type to manage replication and unique ids
[12:38:06] <Derick> Bartzy: no, "timestamp" is a wrong tiype, but yo ucan use an int to store unix timestamps
[12:59:30] <Bartzy> if I have an array in a document. I want to always have a maximum of 50 elements in it
[13:00:08] <Bartzy> i.e. when a new element needs to get in, get it in (first if possible, if not, then last), and remove the last one only if it's the 51th element. If we didn't reach 50 , don't do it
[13:04:15] <feklee> I want to store settings in a MongoDB database. Every setting has a key and a value. What's best design? (A) `settings` collection with one document containing all the settings, or (B) `settings` collection with one document per setting (`_id` = key, `value` = value)?
[13:06:11] <kali> feklee: http://www.mongodb.org/display/DOCS/Schema+Design you can start from there
[13:07:04] <feklee> kali: Thanks! Any suggestion concerning my particular problem (which should be common)?
[13:07:48] <kali> feklee: it's likely to be a small collection with an even smaller number of write, so anything should work, realyk :)
[13:11:34] <kali> feklee: i have a collection like that, with hierarchical content. I have one document per key at the first level of the tree...
[13:15:15] <feklee> kali: Well, in my case (settings), documents would really just consist of a single value. Therefore my idea to use: `_id`, `key`
[13:41:26] <Bartzy> can you explain the queries that would be needed? if I have 'likes' array field and a likes_c counter integer field.
[13:47:05] <NodeX> the trouble you will keep stumbling on is you're trying to use a k/v store / database for something that's designed to be handled by an edge graphing database
[13:56:54] <NodeX> if you fully believe that facebook and foursquare tell the truth on thier infrastructure you're mistaken
[13:57:13] <NodeX> they do not divuldge anything that can hurt them finiancialy or give any competitors an edge
[13:57:48] <NodeX> and facebook (as an example) have huge daat centers so can afford to be liberal with data flows
[14:03:51] <Bartzy> NodeX: in your metadata collection solution - each metadata document (like/comment/view) has a _id ?
[14:06:22] <ron> why redundant? you need a unique identifier, otherwise you can't address it. that identifier is the _id. that's it.
[14:10:02] <Bartzy> That is actually another great reason for separating the comments from the photos document
[14:10:41] <Bartzy> Ah, actually you can also use an id in the embedded document... But still referencing seems much more flexible
[14:10:45] <NodeX> Bartzy : you will find all sorts of good reasons when you develop your app - things you never thought of in the beginning
[14:11:31] <NodeX> As I said yetserday - I started with an app that nested the images in a gallery, even at 10 per gallery it was a nightmare
[14:12:26] <Bartzy> NodeX: We were now thinking that no one will ever want to know who is the 51th person who liked their photo. Then we just need the last 50 likes and a total likes counter (and save the likes of each user in their own user document, to avoid more than 1 like per user per photo). But we have that $push and $pull issue which is a bitch...
[14:13:19] <NodeX> the user may not want to know it but you -may- want to know it in your app where you can graph trends and things
[14:14:00] <NodeX> even if it's not indexed - you can pull it out and map/reduce it for those kinds of things
[14:15:16] <Bartzy> but what graphing can be done with that data. Averages can be done with counts. Averages per user can be done with the user document.
[14:16:18] <NodeX> for me personaly I have a history collection that tracks all that but some people like the raw daaty
[14:18:36] <NodeX> varying methods, the main one is grouping by session_id so I can have an array of what a user did from first contact to leaving the site
[14:20:12] <Bartzy> I have another question about my data. the photos collection I was talking about only contains photos that has been "shared" by the user that created those photos. There is another "results" collection which just has all the photos, regardless if they are shared or not. When a photo is created, it is inserted into 'results'. When it is shared, it is inserted (some of the data at least), into 'photos'.
[14:20:51] <Bartzy> I've done the results/photos separation, so it would be easier to get photos that are available for showing, without indexing on a "shared:1, shared:0" key...
[14:21:52] <NodeX> you ahve an _id for each photo's and results rather than one _id and another small field
[14:22:24] <Bartzy> But on results I do nothing - It's there for archive - I can actually delete it later, or just have it on some slower disks ....
[14:22:57] <NodeX> for example, if you query the results then goto a result and query the "photos" mongo -could- viably visit the cache rather than query again
[14:24:18] <Bartzy> Yes, but what if I never query 'results', all my queries are for shared photos only. So if I have only one collection, all my indexes need to have a 'shared' with them
[14:24:49] <Bartzy> the results collection is 600 million documents, increasing 2 million documents per day. Shared is %18 of that.
[14:25:59] <Bartzy> db.results.find({uid:UID_OF_USER, shared:1}), and a compound index on {uid:1, shared:1}. Right?
[14:28:45] <Bartzy> so adding 'shared' field in any index I have is better than having that _id index on results ?
[14:31:16] <Bartzy> so you say if I can delete them - just delete them from that one collection, not their own
[14:32:15] <Ushu_> NodeX : i have a log for my error _a != - 1 ... here http://pastebin.com/FNnkemig
[15:12:22] <NodeX> what if that 20gb pushes an important index out of memory .. when it doesn't need to ?
[15:15:21] <Bartzy> NodeX: But I wonder how much the extra "shared" index in all my indexes will take .
[15:17:18] <NodeX> then tomorrow when it sinks in why it's not as efficient as the other way we're here to help again ;)
[15:40:59] <kzinti> When I type show log in a shell I get a type error stating that res.names has no properties. Does this mean that I just don't have any logs?
[16:36:45] <doxavore> Is it possible to create circumstances where, just after inserting a document, a query for values (non-_id) matching that document returns no results?
[16:38:01] <TkTech> Then from the same connection (assuming no error is returned) and you aren't sharding or reading from a replica secondary, it shouldn't be.
[16:39:38] <doxavore> Hmmm - it's using a connection pool, but the query is definitely firing after the create, with no sharding and reading off of master.
[16:47:00] <doxavore> TkTech: It's basically utilizing Mongoid's find_or_create_by. Out of about 400,000 cases, 400 of them suffer from an issue where they've basically been created twice.
[16:49:24] <doxavore> Mongoid translates it to 2 calls to MongoDB: find(with my options) and if that returns null, create(with my options). I can't consistently get it to break, but for a month now it's been hovering at about 0.1% of cases.
[16:49:42] <TkTech> doxavore: https://github.com/mongoid/mongoid/blob/master/lib/mongoid/finders.rb#L143
[16:51:50] <doxavore> I will only ever have one thread executing a find_or at a time with given arguments, so I think that removes the chances of a race condition on the Mongoid side.
[16:53:07] <doxavore> What I don't know is if 1 connection inserts (with safe=true), gets an OK, and then potentially another connection queries, will Mongoid tell me it doesn't exist yet?
[16:56:33] <digitalfiz> I come from a long mysql background so I am sturggling with this new way of thinking, enjoying it but struggling
[16:57:09] <TkTech> doxavore: Nope. Safe just means the issuer waits until success and minimal replication occurs (w=3).
[16:58:09] <digitalfiz> I have 2 peices of data I want to store: users and devices. The devices are connected to the users so my thought was in mongo I should make a users collection and have an a user with a list in it of devices attached to that user but I'm not sure if I should keep them seperated like in a users collection and a devices collection
[16:59:51] <digitalfiz> but that seemed very relational to me so i wasnt sure if it was better to combine them somehow
[17:07:52] <doxavore> TkTech: A findAndModify can upsert, so I think I should be able to get that working instead of Mongoid less-than-exemplary implementation. Thanks.
[17:13:04] <digitalfiz> thanks for the help guys I'm just super new to mongo so I'm second guessing my decisions as I was told to forgot what I know about mysql so if it seems similar I try to think how it could be done differently
[17:34:29] <ron> if you want to just see documents in a collection, simply do db.collectionname.find()
[17:38:25] <Almindor> I know this is not mongoose channel but I couldn't find anything closer. Does anyone know what mongoose.remove(conditions, callback) calls on the callback? (the args) they don't say at http://mongoosejs.com/docs/finding-documents.html
[17:57:19] <jgornick> hey guys, in 2.2, how do you query for db references? before you were able to do something like field.$id, but that doesn't seem to exist anymore.
[20:41:02] <autolycus> I have a query like yItem= { "e" : 187, "options.id" : { '$exists': true } }; which works fine but now I want to add another condition which is that children.option.id has to be true too…how can I do that
[20:41:34] <autolycus> so I wanna do something like { "e" : 187, "options.id" : { '$exists': true } or { "e" : 187, "children.options.id" : { '$exists': true }
[20:44:49] <autolycus> so I wanna do something like { "e" : 187, "options.id" : { '$exists': true } or { "e" : 187, "children.options.id" : { '$exists': true }
[20:50:01] <ikiini> so <your_colleciton>.find( {"e" : 187, $or : [ {"options.id" : { '$exists': true }},"children.options.id" : { '$exists': true }]})
[20:52:12] <autolycus> gives me this Tue Aug 7 13:51:24 SyntaxError: missing ] after element list (shell):1
[20:58:33] <payal> jY: I was asked a question that went something like "On how many nodes are you running MongoDB"
[21:03:14] <autolycus> db.products.find({events" : 18775, $or : [{"custom_options.id" : { '$exists': true }},{"children.custom_options.id" :{ '$exists': true }}]});
[21:09:08] <ikiini> @autolycus. you can start by making sure your quotes are consistent. you are mixing double and single quotes.
[21:21:40] <dstorrs> hi all. I have a sharded DB where each shard is the same size (800G). Shard 1 is 60% full, shard 2 is 42% full. I am pretty sure that the extra space on shard 1 is deleted docs from an earlier schema change. Is it likely that, after compaction, shard 1 will be the same size as shard 2
[21:32:37] <jY> dstorrs: not 100% sure but compact just defrags.. i don't think it will un-allocate any of the files already created for use
[21:34:10] <dstorrs> I was thinking I would set up a new replicant, let the data copy over to that, and it SHOULD only copy the undeleted stuff. then I either take the primary out and run a repair (? I forget the command) on it, which will actually compact it, or I save it off somewhere and migrate it back from the replicant
[21:40:59] <dstorrs> jY: I'm really just focused on the "if I got rid of the deleted stuff, is it plausible that the two would end up the same size? what information is needed to assess that?"
[21:41:22] <dstorrs> oh, there are about a dozen unsharded collections on shard 1, totally maybe a gig between them
[21:45:24] <dstorrs> jY: db.runCommand({compact:'collectionname'}) will not shrink the datafiles. db.repairDatabase() will.
[21:45:31] <dstorrs> http://www.mongodb.org/display/DOCS/Excessive+Disk+Space#ExcessiveDiskSpace-RecoveringDeletedSpace
[21:46:45] <dstorrs> I'm just looking for a sanity check on whether there is some reason that one shard (the original one, from before the DB became sharded) would always be heavier.
[23:28:33] <acidjazz> i think i might just rock this https://github.com/micahwedemeyer/automongobackup
[23:29:46] <linsys> acidjazz: You could always stop mongodb on the 3rd node and snapshot the file system real quick then start it up again
[23:31:19] <linsys> yea I mean there is nothing wrong with that script but if/when you have gigs and gigs of data it becomes kind of a pain in the a**
[23:32:30] <acidjazz> even when i have gigs i will want at least a weekly full backup to put somewhere offline
[23:54:46] <icicled> in the mongo shell when I run: db.mycoll.find({ reading_date: { $gte: new Date(2012,1) } }).count() I get the correct count
[23:55:39] <icicled> however, when I do: db.mycoll.find({ reading_date: { '$gte': { '$date' : 1328007600000} } }).count() I get 0
[23:56:41] <icicled> I'm trying to build up a query to pass to mongodump (dump docs w/reading_date > than 31st Jan 2012)