Log file Viewer
#mongodb logs for Tuesday the 30th of June, 2015
[00:01:26] <pikapp> Hello, I have a document that contains an array of objects that contains an array of other objects. I need to loop through these inner-inner arrays and remove a field from each object in the inner-inner arrays. I have successfully looped through using db.collection.find() but how do I $unset them?
[00:02:11] <pikapp> Ultimately, I can access the objects I need using nested loops inside db.collection.find() but not a query that I can use in db.collection.update()
[00:03:00] <pikapp> Is there a way to perform an update inside the db.collection.find() function without using a query?
[00:18:20] <Freman> so, in mongo, I can call db.log_outputs.find({task_id: "550f3b73cd3b101400377d18"}, {_id : 1, date : 1, duration: 1, pid: 1, code: 1, server: 1, failed: 1}).sort({date: -1})
[00:18:45] <Freman> but mongoish can't call LogModelInstance.find({task_id: this.task._id}, '_id date duration pid code server failed', {sort: {date: -1 }, limit: limit}, function(err, docs))
[02:16:15] <Axy> Hey all, I have an object structure like {"_id":"uuid",source:{"link":"http://","desc":"description","content":{"p1":"a","p2":"b"}}}
[02:39:26] <Axy> {"_id":"uuid",source:{"link":"http://","desc":"description","content":[{"name":"one"},{"name":"two"},{"name":"three"}]}}
[02:51:16] <joannac> mantovani: http://docs.mongodb.org/manual/reference/program/mongoexport/#cmdoption--fields
[02:58:15] <joannac> For JSON output formats, mongoexport includes only the specified field(s) and the _id field, and if the specified field(s) is a field within a sub-document, the mongoexport includes the sub-document with all its fields, not just the specified field within the document.
[03:04:18] <mantovani> mongodb is very nice for process high volume of "unique transactions( like query something or update)"
[03:04:55] <mantovani> and it doesn't support pipe, I had to put an extra driver just do to export/import
[03:05:59] <mantovani> in any database you can use named pipe "mkfifo" and use ssh or any socket to import/export from differents machines without use extra space
[03:12:00] <joannac> we spent 10 minutes talking about mongoexport and then you switched to mongodump
[03:13:54] <mantovani> zcat /opt/backup/mongodump-2014-12-03/accounts.people.bson.gz | mongorestore --collection people --db accounts -
[03:32:41] <tehgeekmeister> it seems like mongo is spending a whole ton of time in write locks (almost all writes are to the same database, but over many collections). I'm queuing up a fair number (mean is like 50) of writes. mongo is the bottleneck so far in this system, so I need to figure out what to do to optimize it. where should I look?
[03:33:17] <tehgeekmeister> (we're only at 15-60MB/s of write, far from the capacity of the disk most of the time. averaging like 15-20% disk utilization, as measured by iostat.)
[03:35:39] <tehgeekmeister> (i'm just managing this app, didn't write it, hence not knowing that yet.)
[03:35:57] <mantovani> mongoexport --csv -d BigData -c TwitterStreaming --fields '_id,user.name' > twitter_names.csv
[03:36:01] <mantovani> 1193 be/4 mongodb 205.92 M/s 0.00 B/s 0.00 % 2.95 % mongod --confi~c/mongodb.conf
[03:37:18] <tehgeekmeister> we're also hitting page faults, but that's probably only because of the updates
[03:39:35] <mantovani> tehgeekmeister: if you need performance in bulk updates operations you should use Teradata, DB2 which has merge update
[03:39:59] <tehgeekm_> yeah, I don't expect magic. I just had been assuming they were inserts, because by my understanding they should be.
[03:40:12] <tehgeekm_> i'm glad you mentioned it, because i wouldn't have thought to check otherwise.
[03:40:22] <mantovani> I had a problem like this in a costumer, he was doing a lot of updates using an ETL tool
[03:41:06] <tehgeekm_> we're literally guaranteed that this data will always be unique by the time we do the update
[03:43:45] <tehgeekm_> varies widely, some are low (5-10 ish) some are higher (50 ish?) don't think any are huge
[03:44:46] <mantovani> and you have this insane updates, if you should check your logic how you are doing right now
[03:45:03] <mantovani> and you have this insane updates, you should check your logic how you are doing right now*
[03:45:18] <tehgeekm_> I think they may not be insane during normal workload, but we're trying to backfill a couple years worth of data
[03:45:50] <tehgeekm_> would be better to not block the backfill on those updates, and just do them all at once.
[03:52:36] <mantovani> tehgeekmeister: does your app need the update to be done in real-time or can be batch processing ?
[03:52:56] <tehgeekmeister> we could totally do batch, but we'll have to rewrite a decent chunk of this
[06:04:03] <aps> I'm looking for an automated way to backup a prod mongo sharded cluster of data size ~600GB. Is there a script available for this or what is the preferred method to do this?
[06:19:15] <prabampm> Guys, is there any way to restrict mongodb RAM usuage? i have goggled a lot and tried cgroup as well to limit the process memory but nothing seems to be working.. Application becomes dead slow after few days and sometimes getting "cannot allocate memory" error. As an interim solution, we are clearing the cache every one hour.. Any ideas to go forward???
[06:26:42] <Boomtime> mongodb uses (by default) memory mapped files - almost all of the memory used for these remains available to other processes to take
[06:27:06] <Boomtime> when this out-of-memory condition occurs, can you capture the output from ps -aux?
[07:21:51] <Jonno_FTW> i need help with a mapreduce, I have a colleciton of documents with readings:[{count:1234,id:12},...], I want to know the proportion of all readings with a count > 2000, readings has different length in each document
[07:26:14] <Boomtime> you probably can't get a "proportion" in a single query, but you can get a count of the number of documents with that criteria easily enough
[07:27:54] <Boomtime> 1. $match filter to those documents containing at least one array entry with count > 2000
[07:28:21] <Boomtime> 3. $match the same criteria again, to filter out those unwound which don't match
[07:53:57] <pschichtel> Hi! Is there an mongodb equivalent to MySQL's "INSERT INTO table (id, numer) VALUES (1, 10) ON DUPLICATE KEY UPDATE numer=number+10" ?
[07:58:07] <Derick> pschichtel: if you want auto-generated unique IDs for documents, just rely on the default _id generation
[08:03:06] <pschichtel> Derick: no I don't. My collections holds objects with a unique key and some counters. I want to either insert a new object if the key is not found or increment one of the counters by one
[08:05:25] <Derick> you can just do : db.col.update(( { id: 1 }, { $inc: { counter: 1 } }, { upsert: true } );
[08:05:54] <Derick> this will increase counter by one if a doc with id=1 exists, if not, it will create a document with {id: 1, counter:1}
[08:18:21] <Boomtime> be aware that an update matched against a unique indexed field with upsert:true has a most curious possibility (due to concurrency) of generating a duplicate-key exception - be prepared to re-try the operation in that event
[08:20:19] <Boomtime> you want to define both of those terms - for write durability guarantees, you should read up on WriteConcern - the word 'immediate' is difficult to even define in a network environment, so what do you mean by that?
[08:20:55] <Boomtime> if you have received a success reply to a write that you issued, then the write is definitely visible to all clients
[08:21:13] <Boomtime> however, it is entirely possible that clients see the write result before you receive the reply confirming it
[08:22:08] <burhan> in my application, there are scheduled writes (either and update, or an insert) for documents; and I want to make sure that these writes are immediately available to all clients. Right now, there is only one node, but eventually there will me many read-only nodes. My concern is how to prevent the read-only nodes from having stale data?
[08:24:58] <Boomtime> reading from secondaries means you read stale data, this is a fact, it is physically impossible to avoid since there is an undefinable delay between primary and secondary - if you want to ensure always-everytime-absolutely-guranteed-up-to-date data then you read from the primary only - which is the default mode for this reason
[08:25:33] <Boomtime> under most circumstances, the secondariers are milliseconds-at-most behind the primary
[08:25:53] <Boomtime> but if those milliseconds mean something to you, then you must not read from them
[08:27:21] <Boomtime> this says it better than i ever could: http://askasya.com/post/canreplicashelpscaling
[08:44:44] <burhan> How do you write a statement to fetch documents that do not match a regular expression?
[08:57:51] <burhan> okay, that one was easy - the really complicated one that I am trying to figure out is how to do a query like "if any of the sub-documents in a collection match a condition"
[08:59:00] <burhan> I have a document describing customers, and a list of other documents describing purchases. So I need to find all customer that have at least one purchase where the product code is 543.
[09:00:07] <burhan> for example: {'name': foo, purchases: [{'id': 543, name: "foo widget", price: 54}, ....]}
[09:02:05] <Derick> however, I think you should split that up into separate purchase documents, instead of an array embedded
[09:06:05] <burhan> is that recommended? I thought it was better to have a nested document rather than multiple documents.
[09:07:25] <Derick> how many purchases are you going to get? that's an array that's going to grow grow (hopefully)
[09:07:50] <Derick> perhaps you might hit the 16mb limit? and, if you're using the mmap storage engine you end up moving the document around a lot too
[09:08:13] <Derick> nested documents like this are mostly ok if it's not data that's going to change a lot - I suppose
[09:09:57] <Derick> added on perhaps more than updated - updating nested documents in more than two levels is not easy to do
[09:51:35] <bilkulbekar> I have a mongo sharded cluster, with two shards, three replicas in each, is it a good thing to have mongos running on each of it?
[11:07:57] <Axy> Hello channel. I have my documents in this structure: http://pastebin.com/Te1KqJWS -- I want to be able to pull the url's of te last "image" - how do I pick the last object in the array via find?
[11:08:01] <bilkulbekar> I have a mongo sharded cluster, with two shards, three replicas in each, is it a good thing to have mongos running on each of it?
[11:09:32] <Axy> I can do .fond({},{"images.url":1}) and only return url's but that is not for the "last image "only
[12:16:07] <Axy> http://docs.mongodb.org/manual/tutorial/upgrade-revision/ makes it look like it's rocketscience
[12:18:50] <Derick> secondly, you should be able to : stop mongodb; put new binaries in place; start mongodb — providing you have a single node, and no other complicated setups
[12:19:17] <Axy> no my setup was pretty straightforward, I followed a tutorial (I'm very new with all this)
[12:19:59] <Derick> then yeah, stop mongodb; apt-get update mongodb (or however you installed it); and start it back up again
[12:20:47] <chombium> hi Derik, I have an Ubuntu 12.04 with MongoDB 2.0.4 running and I want to upgrade to the current release. ist that possible?
[12:22:54] <chombium> i thought so, in the upgrade docs i've read that i need to upgrade first to 2.2, than 2.4, 2.6 and then to 3.0. i started with this http://docs.mongodb.org/manual/tutorial/install-mongodb-on-ubuntu/ but I can not find the packages for the older releases. is there some archive?
[12:37:08] <chombium> Derick: Thanks for the link. it seems like a good startpoint. unfortunatelly there are not packages, I couldn't fund ppa as well. so if I manually upgrade it to 2.2 and so on, the data should be preserved without any problems, or?
[12:40:56] <chombium> d-snp I don't have docker. it's a root server. from the last test i did this works: apt-get install mongodb-10gen=2.2.0 it seems that there are packages after all
[12:42:59] <Axy> Derick, sorry for asking such beginner questions but, how do I update mongodb binaries?
[12:47:08] <alexi5> guys I am new to mongodb and was wondering what applications is type of database more suitable for than RDBMS ?
[12:51:31] <Axy> d-snp, if there is a way to make a clean install or "right way to install" I can try that as well
[12:51:57] <Axy> but if "everything is fine anyway" then maybe you can help me update this, I really do have no idea what I am doing atm
[12:52:07] <d-snp> http://docs.mongodb.org/manual/tutorial/install-mongodb-on-linux/?_ga=1.195084522.796200131.1404994453 <-- try this
[12:52:52] <d-snp> http://docs.mongodb.org/manual/tutorial/install-mongodb-on-ubuntu/ <-- I meant this one
[12:54:18] <d-snp> simply install the newest version from that guide, and then follow the upgrade guide
[12:54:35] <d-snp> or well, follow the upgrade guide, and use that tutorial for the 'install the latest version' step
[12:55:53] <d-snp> in a professional setting btw, you would run this entire procedure on a staging server, so you don't have surprises
[13:01:20] <Axy> d-snp, I followerd http://docs.mongodb.org/manual/tutorial/install-mongodb-on-ubuntu/ but it's still the old version of mongod when I sudo start service
[13:12:15] <Axy> Derick, d-snp I followed the tutorials again to update my mongod from 2.6.6 to latest, I'm stuck here http://docs.mongodb.org/manual/tutorial/upgrade-revision/#upgrade-replace-binaries at step 3
[13:21:47] <Axy> Derick, when I do "sudo apt-get install -y mongodb-org" it says "mongodb-org is already the newest version.
[13:30:46] <Axy> Derick, I think for some reason files were in use or something, I just did purge and installed everything from scratch
[14:03:31] <watmm> Hey, would anyone have some suggestions as to how to avoid an uncaught MongoConnectionException with message Failed to connect to: localhost:27017: Connection refused when pm.max_children is reached?
[14:44:35] <Derick> iszak: 3.5million? sounds like you're hitting the maximum document size of 16MB there
[14:44:59] <iszak> Derick: it's a $match $in aggregate, so it's to filter out results, I wouldn't assume so?
[14:47:57] <iszak> so if I have all this data in mongodb how do I get access to it with a 16mb limit?
[14:49:04] <Derick> 3.5million elements in $in, is going to be roughly 16mb of data - that has to be send, as a document (with a 16mb limit), to the server
[14:52:18] <Derick> I think you just need to remember that "a query with all of its arguments and options" - is also a document
[14:55:39] <Derick> and that's mmapv1 only, The WiredTiger storage engine doesn't have tha limit I believe
[14:56:51] <Derick> most of the MMAP engine limitations are OS limitations though... not much we can do about it. But, you can scale to multiple shards to get around that.
[14:58:04] <Derick> then again, *one* node with 32TB of data is not easy to do anyway. As you need diskspace for it too.
[15:11:39] <Siamaster> and I'm gonna implement a chat service, so I thought I'm gonna make another entity called User User
[15:14:03] <Choreetso> I would like to pass documents from one collection into another, but I'm afraid of id clash
[15:15:23] <watmm> Is it normal for mongo to refuse connections and require a restart when php-fpm hits max.children?
[15:15:55] <watmm> It didn happen before (a mongo upgrade and bind_ip change from 127.0.0.1 to 0.0.0.0)
[15:17:13] <Choreetso> Siamaster, ok, thank you! I think I'll try regenerating ids with this anyway: http://stackoverflow.com/a/8723243/1781612
[15:58:25] <tehgeekmeister> in an aggregation, how do I say "make this field on the result be set this value, if it exists on the input document, or some default value otherwise"?
[16:03:21] <tehgeekmeister> hmm, for our use case that is impractical. i can do one aggregation pipeline pass, and a second collscan to reformat the output documents.
[16:06:08] <tehgeekmeister> so, if i understand correctly, you're saying that a single pass of the aggregation pipeline will be difficult to use to accomplish the full goal?
[16:09:26] <tehgeekmeister> so the gotcha might be in this detail, we select out like four attributes, and group on those. then, the project would be merging those into a single string, that becomes the new _id
[16:22:13] <Siamaster> Everytime the user is looking att their contacts they will see the pending contact requests
[16:23:40] <Siamaster> I wouldn't be able to index the contactrequests with status pending otherwise right?
[16:25:01] <Gemtastic> designing is for everything from within your application to within your database
[16:28:31] <Siamaster> and then always query on sender / receiver when I need to get forexample; all the pending requests received by user
[16:29:16] <Siamaster> when having an embedded collection I can have a boolean indicating if the request was sent/received
[16:31:03] <Siamaster> hmm no I think I'll go with collection feels really weird having duplicated message/date and stuff
[19:20:51] <pokEarl> Anyone with some Java/(MongoJack) experience that mind helping me out? https://www.reddit.com/r/learnprogramming/comments/3bnz0y/javamongojack_having_trouble_making_a_pojo_from/ Having trouble going from the MongoDB and back to java :(
[20:01:01] <StephenLynx> anyone here having issues installing mongo from http://docs.mongodb.org/manual/tutorial/install-mongodb-on-ubuntu/ on trisquel?
[20:27:41] <whaley> pokEarl: did you solve it by changing version? It's been a couple of years, but I used to be a mongojack user.
[23:34:36] <Freman> the collection contains some 45650 odd documents (depending on the time of day), some as big as the 16 meg limit
[23:40:25] <Freman> PHP Fatal error: Allowed memory size of 134217728 bytes exhausted (tried to allocate 23 bytes) in Eventing.php on line 139
[23:42:28] <Freman> PHP Fatal error: Allowed memory size of 134217728 bytes exhausted (tried to allocate 80 bytes) in /Users/shannon/Development/PHP/Eventing/Eventing.php on line 139
[23:43:37] <Boomtime> no, see you need to prove the memory leak is in the driver and not the hundreds of other lines of code that is yours
[23:44:37] <Boomtime> but i would be very interested if you have a dataset to which the answer is yes
[23:44:47] <Freman> PHP Fatal error: Allowed memory size of 134217728 bytes exhausted (tried to allocate 16 bytes) in foo.php on line 9
[23:45:54] <Freman> the only difference between the php in pastebin and php in foo.php is <?php and the actual connect url (shrub.logging.net is not the real servers name)
[23:46:37] <Boomtime> that is probably important - but i notice the amount of memory it has "allocated" seems to be incredibly low
[23:48:08] <Boomtime> allowed to use only ~130MB though? given that streaming just one large document in will take at least 20MB, and the cursor will start receiving the next one before you even get to it...
[23:49:45] <Boomtime> not too surprising, the document sizes still mean there are tens of megabytes involveed
[23:50:24] <Boomtime> the fact is you're manipulating very large documents in a very small space - why is the available memory so low?
[23:51:29] <Freman> because it's my dev env (production it's set to 512 but I don't like to push that because there's 200 php processes per machine in production)
[23:52:50] <Freman> so you're saying the library is streaming more data in the background than I'm consuming and it's garbage collection is cleaning up?
[23:53:57] <Freman> my understanding leads me to believe there should be no more than 2 or 3 documents loaded at once, which 128 meg should easily accomodate
[23:54:16] <Boomtime> what i'm saying is that if you know that the library is going to need ~40MB just to be able to provide the mimimal ability to iterate a cursor... having only 130MB available sounds like pushing your luck
[23:54:49] <Boomtime> you're expecting the PHP garbage collector to be awesome at it's job, turns out it might not be
[23:56:58] <Freman> PHP Fatal error: Allowed memory size of 268435456 bytes exhausted (tried to allocate 40 bytes) in foo.php on line 10
[23:57:21] <Freman> (the increase of line numbers is a counter variable and an ini_set(), nothing more)
[23:58:20] <Boomtime> if there is no upper limit then you should be able to construct a simple dataset, and iterate loop, that shows the problem - then you raise a driver bug
[23:59:00] <Boomtime> if what you're showing is simply an ineffeicency of memory re-use in the zend engine, then it's not really anything we do about it