Log file Viewer
#mongodb logs for Monday the 21st of April, 2014
[01:36:27] <digitalextremist> Anyone know why the README for mongo-ruby-driver shows Mongo::Client and everywhere else in the Wiki I see MongoClient ?
[01:49:38] <digitalextremist> Mark_: the driver seems to be radically out of sync with the Wiki. I suppose I’ll post an issue if one hasn’t been made already
[01:50:18] <Mark_> since i was just playing it wasnt a big concern for me, i was just playing converting some nosql friendly stuff to mongo
[01:51:17] <digitalextremist> I have mongodb running in production and the latest gem was a breaking change in many areas :-/
[03:18:46] <george2> My db has been working on the single command "db.data.remove({text: {$exists: false}})" for over 9 hours now. :/
[03:19:30] <george2> I mean, I have a few hundred million records, but advanced queries usually still only take a couple hours.
[03:21:26] <george2> just wondering if I should stop it now, create an index and restart, or just let it finish
[03:25:00] <cheeser> although indexing might take as long as deleting. but it you need that index long term, it'd be worth it.
[03:26:19] <cheeser> well, you could issue a killOp command on it. get the count on that collection, and try again.
[03:27:53] <george2> I thought maybe it found all matching records first, then started removing them, or something like that.
[03:28:41] <george2> I'm deleting all my old records, so I would expect at some point a findOne would return one of the new ones, with 'text'
[05:40:20] <NaN> (fedora 19) I created the .repo file and when I try to install it it says "No package mongodb-org available." is there any extra step to install it with yum?
[05:42:40] <NaN> ranman: http://docs.mongodb.org/manual/tutorial/install-mongodb-on-red-hat-centos-or-fedora-linux/
[05:43:01] <weeb1e> Hi everyone, my mongodb is freaking out, is there a faster way to clear a whole collection than .delete()?
[05:45:04] <weeb1e> Mongodb was using a crazy amount of resources and making all other databases insanely slow
[05:45:24] <ranman> if you're going to run multiple databases on one machine look into something like cgroups
[05:45:28] <weeb1e> I couldn't just delete some of the documents, since it took half a day to delete only a million or so
[05:45:39] <ranman> the software has no way of knowing that it's sharing resources with other databases
[05:46:25] <weeb1e> Mongodb was killing performance due to the collection size, even with nothing touching the database it was using
[05:47:05] <weeb1e> Though top did not say all applications were using 64GB, so I can't say exactly how much ram mongo was using
[05:47:14] <ranman> weeb1e: you have 64GB of ram on that machine ? or did it just say 64GB of virtual RAM also, it should have been paged out by your operating system if the other databases
[05:47:43] <ranman> weeb1e: then mongod was not what was killing the performance if it was only taking up 64gb...
[05:48:29] <ranman> I no longer understand what you're saying, hop in a PM if you'd like more help but what you're saying doesn't make a lot of sense to me sorry
[05:49:04] <ranman> I thought you meant you were running multiple databases on one machine as in postgresql, mongodb, etc.
[05:49:22] <weeb1e> I now know I cannot store that much data in mongdb in a useful way, and have to rethink things
[05:49:48] <ranman> so the next time something like that happens try to spin up mongoperf or run db.currentOp() to ascertain what exactly is taking up all that time
[05:50:01] <weeb1e> But I have a bunch of services that use mongo, and rely on a fast query time, so after a day it became very noticable
[05:50:03] <ranman> there's also a --slowqueries option that will log queries that take above a certain threshold
[05:50:29] <ranman> weeb1e: understood sorry that happened, if you still have the logs you can definitely get a better understanding of what was happening
[05:50:52] <weeb1e> Ok, once I rework this projects storage, I will pay better attention to performance once it is running again
[05:51:24] <weeb1e> I always knew I needed to change that collection, but by the time I got to it, it was already too large to migrate the data
[05:52:08] <weeb1e> I feel a column store would be much better suited for stats storage, but when I tested putting some of the data into postgres, it used even more disk space than mongos millions of documents
[05:52:16] <ranman> weeb1e: yikes :/ that's rough. there really a lot of options you have to prevent stuff like that
[05:52:38] <ranman> weeb1e: event data (non-entity data) like stats is particularly well suited to document stores like mongodb IMO
[05:53:06] <weeb1e> NaN: Yes, since I don't know of something else that can store the data more efficiently
[05:53:54] <weeb1e> I have a few integers that need to be recorded for ~20k servers every ~10 seconds
[05:54:12] <ranman> weeb1e: There's a really great presentation that mongodb has called Real Time Analytics With Bucketing by scott hernandez
[05:55:43] <weeb1e> My application queries ~20k servers and records integers representing server number, client count, ping and timestamp
[05:56:38] <weeb1e> The web backend application then renders graphs for the data on the web front end, and also receives realtime updates from the query backend
[05:58:19] <weeb1e> I feel that such small amounts of stats data is not really a good fit for mongo documents though
[05:58:38] <weeb1e> Because the keys and bson data will use up a lot more space than the actual data
[05:58:44] <ranman> weeb1e: if you go to mongodb world there's a talk about something similar on the scale of 4GB I think it's called weathermen or something
[05:59:58] <weeb1e> Again, when all I need to store is 4x integers, the keys will still use more space than the data, and the _id will use more than everything put together?
[06:02:18] <ranman> that's why I think you should look into bucketing, larger documents by minute / hour / etc.
[06:03:48] <weeb1e> Yeah that's true, though it still will end up using a lot of disk space, and at some point mongo performance will become an issue again
[06:05:17] <ranman> weeb1e: I think this might help? not sure: http://docs.mongodb.org/ecosystem/use-cases/pre-aggregated-reports/
[06:06:50] <weeb1e> That sounds like it's on the right track, I will read through it when I have time
[06:07:35] <the8thbit> I'm trying the query: Badge.find( { owners: { $in: [{ username: req.query.username }] } } )
[06:07:52] <the8thbit> where owners is an array of subdocuments containing a username and a date field
[06:08:22] <the8thbit> I'm trying to look for an entry in the subdocument array with a username equal to req.query.username
[06:15:48] <ranman> because then you could just do: Badge.find({"owners.username": req.query.username})
[06:16:15] <joannac> is req.query.username is actually an array of usernames, do it the way I had it
[06:16:24] <ranman> the8thbit: from the query's perspective you can think of all arrays of documents as being unrolled
[07:24:30] <Diplomat> is it possible to have like.. 3 read servers and 2 write servers.. like i write into 2 servers and 3 servers will have consistent data
[07:25:24] <Diplomat> so 3 servers will be optimized for reading (SSD etc) and 2 servers are optimized for receiving data and then sending that data into read servers for access
[07:56:18] <lkthomas> > db.runCommand({cloneCollectionAsCapped:log_lines, toCollection:log_lines_cloned, size:2147483648} )
[08:15:26] <kali> Diplomat: you can have one "write" server (the primary node in a replica set) and use the other to performs reads
[08:15:57] <kali> Diplomat: to have two "write" server, you need to shard, and in that case every node get only half of the data
[08:16:25] <Zelest> kali, as for my pull and push within the same query, that's not possible.. but nothing stops me from adding the new url first and remove all the old ones afterwards :D
[08:28:48] <kali> well, you already know the ns, you should not mess with the "v", and the name is usually generated ok. so you only need to call createIndex with the key
[08:39:15] <Diplomat> @Kali: well it's okay if it gets half of the data.. as long as it writes fast and stores info so i can use processor to process that data and store it in read only server
[08:41:36] <Diplomat> well i need to be able to write raw data into write server even when read server for queue is down
[08:42:35] <Diplomat> I'd need to have it like MongoDb (read only) -> Queue -> Raw processor -> Cassandra -> Processor -> MongoDb
[08:43:41] <Diplomat> So when MongoDb is down I still have queue which servers as like a little buffer between mongo and raw processor
[08:44:57] <Diplomat> because mongodb storage is most important part of the system and when it goes down I can use cassandra's data as until mongo is up again
[08:47:13] <Diplomat> hmm i could use mongodb really.. but i'd need it to be consistent at all the times
[09:12:17] <dragoonis> If I have a mongodb collection in production, and I want to run a background job to clear out that collection and update it with new results. What's a suggested technique for going about this to ensure there's no loss of services between when swapping the live collection with the newly generated collection
[10:26:02] <dragoonis> Derick, does the php extension for MongoDB support db.collection.copyTo()? I'm not seeing this in: http://www.php.net/manual/en/class.mongocollection.php
[11:08:48] <boll> has anyone had any experience using any of the unauthorized php drivers with hhvm and mongodb?
[12:15:51] <kali> Diplomat: is there an index on your update selector ? what write concern are you using ?
[13:47:47] <G1eb> I thought it was the same as a table scan in relational but that turns out to be false
[14:00:41] <saml> you start mongod with --notablescan and you are certain performance is good because you are forced to ensure proper index
[14:01:33] <saml> if you use mongo client to connect to replica set, it should be transparent, i think
[14:02:56] <tscanausa> so I have secondary preferred through my client, but load / ram has jumped 4x on the primary then the secondaries.
[14:15:55] <G1eb> saml, so if you do a find and mongo hits an index it is not considered a collection scan? (since it does not scan the whole collection)
[14:16:44] <G1eb> ah I get it now, it is except for probably the case when mongo only hits the index when specified in the sort
[14:17:41] <G1eb> I figured anytime it hits the index its not a collection scan but clearly its only the case when it uses find
[14:21:34] <tinco> though I'm not sure mongodb is the best document store out there for learning fundamentals
[14:27:27] <tinco> a proper document store allows you to store your entire database in a single document
[14:28:16] <tinco> I'm just saying there's cases where relational data and a powerful query language are pretty awesome
[14:30:21] <tinco> cool example, xquery lets you query DAG's in a single query without much of a performance issue
[14:35:07] <tinco> well you could use a graph database of course, but your graph database is just an abstraction over a column store, just like a document store is
[14:55:18] <gskill934> so, i thought a JSON format like mongodb have is better than a 2 dimenational table
[14:55:47] <boutell> Thinking about storing permissions in mongodb. Right now I have arrays holding the ids of people and groups who can, let’s say, edit a specific thing: editPersonIds and editGroupIds. By indexing these fields I can efficiently find out if Bob can edit something (in his own right or via a group he’s in).
[14:56:28] <boutell> but I’d like to generalize to more permissions, and I’m wondering if I can efficiently index it still. For instance, there might be “view”, “contribute” and “edit” permissions, and more later.
[14:57:03] <boutell> one possibility is a single array in which the values include both an id and a prefix indicating the permission, like this: permissions = [ ‘edit-57’, ‘contribute-42’ ]
[14:57:55] <boutell> I wonder if I can index a two-dimensional array instead though. permissions[‘edit’] = [ ’57’, ’92’ ]
[15:04:41] <tscanausa> I guess map reduc eonly works on the master which is different from what that docs say
[15:11:28] <boutell> Nodex: does it work like I’m suggesting, with the permission name and the id of the individual permitted to do it?
[15:16:41] <Nodex> I store a registry with a hmac of the file or whatever and a list of attributes associated with it
[16:14:02] <Mick271> helloy guys, anyone knows if statup script exists to run mongos/config server off the distro's packages
[16:31:08] <no_mind> I am using flask with mongoengine. I have a date input field that is mapped as DateTimeField in the model. When I try to validate the form, I consistently get an error "Not a valid datetime value" . How do I fix it ? The date is sent from the browser in Y-m-d format
[17:56:43] <proteneer> https://aws.amazon.com/marketplace/pp/B00CO7AVMY/ <-- do these AMIs support SSL?
[19:19:47] <travisgriggs> all of a sudden, pymongo.MongoClient.database_names() has quit working. it’s failing in createPrivateMap on a file that hasn’t been written to in months (don’t use that db anymore, just haven’t cleaned it out yet). any suggestions? I have zero mongo admin experience. til now, i only had to apt-get install it and it’s just worked wonderful. i *am* on a 32bit box (i know i need to switch to 64 bit some day), but i should be way below memory
[19:38:37] <travisgriggs> a reboot of my box doesn’t fix the error. any help much appreciated. is there a better place to ask these questions?
[19:42:05] <Joeskyyy> travisgriggs: Only time I've ever seen that kind of error is when your resources are being overutilized
[19:48:16] <Odin^^> Hi guys.. Anyone here ? I’m having some strange issues with mongo. Doing a query on two fields, one string and one collection. If I do either ones I get the right data, if I do them together I get 1/10th of results. And when I hint the query it uses some random index nothing to do with the query.
[19:49:39] <Odin^^> Sure. So I do this query: {"album" : "241052846483_1073741867","profilesRelation" : 1619}
[19:53:58] <Joeskyyy> try db.coll.find({$and: [{"album" : "241052846483_1073741867"},{"profilesRelation" : 1619}]})
[20:11:42] <Odin^^> Ok either this is not working or i’m a fucking retard. If i’ll need help can someone here do that for $$ ofcourse ? Needs some way of identifying bcz I have a client production server that I need give access to.
[20:30:08] <Odin^^> Does index creation sequence has anything to do with how indexes are slected when you query ?
[20:30:37] <cheeser> the order the indexes are created? or the order in which the fields are listed in an index?
[20:31:29] <cheeser> yes. the order they appear in the query should match the order they appear in the index
[20:34:27] <skot> You can order the fields in the query in whatever order you want and it will use the same indexes.
[20:35:52] <skot> Depending on the selectivity of the index you can get different indexes used and for different values in your query as well.
[20:36:55] <skot> It may turn out that {a:1, b:3} is better for an index with of b, than {b:1, a:10} is for an index of a
[20:37:14] <skot> It totally depends on which are tested by the query planner and picked (then cached).
[20:38:41] <skot> cheeser, the query plans are cached by canonical query, so the fields are basically reorded anywya.
[20:47:25] <Odin^^> https://dl.dropboxusercontent.com/u/6875944/Screen%20Shot%202014-04-21%20at%2022.45.59.png
[20:51:26] <Odin^^> It is, production. But I had the same problem when I didn’t have anyone changing data.
[20:52:48] <Odin^^> Shouldn’t matter - I need to trust it will return the right data. It’s a database.
[20:54:57] <skot> (unless you have sparse indexes, but that is another thing which you don't seem to be using)
[20:55:20] <skot> You should be able to run the query without using an index to test if the index is the problem.
[20:55:53] <skot> just add .hint({$natural:1}) before the explain, but that will cause it to look at all the documents and not use an index.
[20:59:24] <skot> $natural is a special "index" which will walk the documents forcing no index to be used.
[21:15:00] <skot> It basically causes that index to be sparse, and those docs with large values aren't in the index.
[21:16:06] <Odin^^> Ahh. Got it. I can probably survive w/o images in an index bcz not many of the documents don’t have images
[21:17:22] <skot> You should remove that index since it is not what you wanted/intended given your docs.
[21:23:06] <skot> should be trivial, but could be more complicated if you are using auth or sharding (just requires more steps to upgrade)
[21:23:55] <skot> I'd suggest reading this, http://docs.mongodb.org/manual/release-notes/2.6-upgrade/