Log file Viewer
#mongodb logs for Tuesday the 4th of November, 2014
[00:04:02] <izolate> when I apt-get install mongodb, it automatically starts the daemon. is this expected behavior?
[00:16:02] <darkblue_b> hi all - I will find out more tomorrow, but I think the Java PhD colleague of mine, might be re-allocated to be the search algorithms guy, with Mongo retained as the search platform for cluster analysis
[00:37:16] <hahuang65> what's the difference between doing a Bulk Write Operation for remove as opposed to just a remove (for a single collection)
[00:47:11] <joannac> hahuang65: a single remove does a getLastError each time. BulkWrite bathces the getLastError calls
[00:54:50] <hahuang65> I'm having issues because we need to bulk delete things and I think the index changes and replication fucks our performance.
[01:01:56] <hahuang65> Boomtime: not necessarily that replication is the problem, just a massive amounts of writes at once is the issue. Every operation associated with the writes (replication, index updating)
[01:12:58] <bpap> i definitely used the wrong syntax. but it looks implemented here: http://docs.mongodb.org/manual/reference/method/db.collection.findAndModify/
[01:13:00] <Boomtime> anyway, the standard solution to "slowing down" bulk deletes is to do a query for the thing you are removing, with a limt - then issue a remove using a $in for that set
[01:13:55] <Boomtime> if you $in using the _id it will be very efficient, despite the round-trip silliness
[01:14:35] <Boomtime> also, filter your results to provide only the _id in the query result so you reduce the amount of data returned/gathered
[01:15:18] <Boomtime> not that optimizing this will matter much since your delete has to hit disk no matter what
[02:08:49] <quuxman> I get: TypeError: "'dict' object is not callable" on calling cursor.next(), but it's not consistent
[02:09:22] <quuxman> it happens a lot, but only for certain queries, some of the time. Restarting my app consistently causes it to disappear, where it will then quickly reappear shortly after
[02:10:34] <quuxman> I've stepped through the entire code path in pymongo multiple times, and then it appears almost randomly later somewhere in my code where there is no dict object that's being called
[02:36:20] <Boomtime> quuxman: "I get: TypeError: "'dict' object is not callable" on calling cursor.next(), but it's not consistent"
[02:36:51] <Boomtime> this is not something that pymongo could possibly have done, since you own that object
[02:38:12] <quuxman> I was reassigning cursor, with `cursor = islice(cursor, limit)` and I think I just narrowed my problem to that somehow
[02:41:06] <Boomtime> you have asked a question whose answer seems to be "the mysteries of python" so i have no idea
[02:44:39] <Boomtime> http://api.mongodb.org/python/current/api/pymongo/cursor.html#pymongo.cursor.Cursor.limit
[02:56:42] <quuxman> ah, I have figured it out. It was an issue with an ifliter I had in my code. Now that I've turned the ifilter and islice into a loop, I actually get a meaningful traceback
[04:11:26] <annoymouse> cheeser: And how do I make an index if I'm using a db hosting service like mongolab?
[04:14:41] <annoymouse> cheeser: How does that differ from http://docs.mongodb.org/manual/reference/method/db.collection.ensureIndex/
[04:17:11] <cheeser> oh, i had it backwards. createIndex() is deprecated. ensureIndex() is what you want.
[07:06:21] <shushamna> just connected to feenode..its look like little bit different to communicate..
[07:33:14] <LouisT> so, if i need faster read/write access, would it be a bad idea to use ramdisk for mongodb? say i don't really care about keeping data on system restarts or w/e.. could always just make backups ever X minutes or w/e..
[07:34:18] <joannac> would the speed of read/writes outweigh the backing up (writing your whole dataset to disk) every X minutes?
[07:37:40] <LouisT> well, i could probably figure out a way to do something like rsync or w/e... idk
[07:55:08] <zamnuts> LouisT, you were talking about rsync'ing your db that would be stored on a hypothetical ram disk... i'm saying just tail the oplog
[07:58:58] <LouisT> zamnuts: yea but that wouldn't really help.. i could probably just set up mongodb on another server and use that there instead of reading the oplog manually but not really what i wanted to do
[10:27:20] <PirosB3> hi all, is it possible to select a super-set of a query in the aggregate pipeline
[10:28:07] <PirosB3> basically, I want a way to ask mongo: “get me ALL the coordinates or a specific ID, where at least one coordinate is near this point
[10:28:54] <PirosB3> with geonear it will correctly select the coordinates I want, but it will only return the coordinates that match
[11:19:03] <Lope2> I realized that I want to make one of my keys the _id field in my collection. But when I tried to rename myKey to _id, mongoDB told me _id is immutable.
[11:19:33] <Lope2> how can I easily dump the collection from the DB, run a search/replace in a text editor, and import it again?
[11:36:25] <joannac> Lope2: you basically have the steps, i'm not sure what else you're looking for?
[11:38:22] <ssarah> hei guys i make stuff like "sudo mkdir -p /srv/mongodb/rs0-0 /srv/mongodb/rs0-1" for my replicate set. then how do i change it's chown / chmod to make it work with mongo
[11:39:36] <Lope2> joannac: I used something like this and it wiped out my entire collection... oops. Luckily it was only 49 docs and I had dumped it in the terminal beforehand so I could just insert it again. db.status.find().sort({foo:1}).forEach(function(doc){ var id=doc._id; doc._id=doc.UserId; db.status.insert(doc); db.status.remove({_id:id}); })
[11:39:40] <kali> ssarah: you need to find out which users runs mongo (probably "mongo" or "mongod")
[12:33:50] <scruz> i’m looking at the aggregation pipeline for doing some analysis on some documents.
[12:34:49] <scruz> suppose my documents contain a field, myList, that could contain any subset of the list [1..5]
[12:44:43] <Coffe> someone have time to explain lite design hints from a sql noob ? as i cant seem to break my habit of relations .. having X sessions that connects to X number of maps.. and conencted to that map there is X number of zones in..
[12:47:32] <kali> Coffe: you need to think in terms of the queries you need to make efficient and the one you can afford to be expensive
[12:48:50] <Coffe> today a sector have the id of the track saved in it.. feels like the wrong way of doing it.
[13:46:05] <LinkRage> I added a replica set to be part of a shard by executing rs.add("test-sh03-rs/server3:27017,server1:27016") - server3:27017 is always the primary/priority=2 . But why then sh.status() shows test-sh03-rs's members in the opposite order? - output here: http://pastie.org/private/jwpcuzwuf3ygqaqyhhk0nq
[14:08:11] <lipiec> Hi! I have the following problem: https://groups.google.com/forum/#!topic/mongodb-user/m2fYbA4C-go
[14:18:27] <kali> scruz: mongodb is only marginally using javascript (in the shell or map/reduce). all the internal stuff works with BSON data, and BSON is typed.
[16:05:13] <Lonesoldier728> does anyone know if mongoose supports text search or if I have to use the plugin
[16:13:05] <GothAlice> kali: Ostensibly JavaScript (and PHP) are bad and should feel bad for treating them the same. See pages 80 and on, notably §11.9.3 and §11.9.4. It's insanity like that that forced me to prefix hex-encoded hashes with 'x' to prevent them being compared *numerically* in one application.
[16:13:17] <GothAlice> kali: (of http://www.ecma-international.org/publications/files/ECMA-ST/Ecma-262.pdf)
[17:14:15] <GothAlice> http://cl.ly/image/2Y1t1W022p2x coded in time for the presentation. 16 aggregate queries gave their short, short lives (3-10ms, 4ms on average) to produce that.
[19:47:20] <boutell> hi. I needed to process results from mongo queries in parallel, but only up to a certain limit, and there wasn’t really an API for that, so I wrote an npm module: https://github.com/punkave/broadband
[19:47:32] <boutell> it’s nice if you don’t want to pull all the data into RAM first and feed it to async.eachLimit.
[20:41:02] <agend> i'm looking for a db which would be optimized for key value storing - where value would always be a number with incrament function
[20:49:43] <Goopyo> agend: use redis. Though internally it stores ints at strings although it does allow for atomic incrementing of them (this is only a concern because your ints consume more memory)
[20:51:28] <bttf> A little confused ... I just kicked off mongod for the first time, then created an admin user. After that, I was still able to connect to mongod with mongo without using any credentials. This shouldn't happen, right?
[20:51:50] <GothAlice> bttf: Did you enable authentication on the command line or in the configuration file (and remembered to specify the configuration file on the command line…)
[20:52:30] <bttf> I just ran 'mongod' when I started the daemon, so no I don't think I enabled authentication on command-line nor did I load a config file
[20:53:03] <Goopyo> agend: Redis has dis persitance too. Its probably the fastest option for what you need
[20:54:25] <GothAlice> agend: As an alternate, I store per-document arbitrary key/value data as mappings or sometimes lists of nested {key: …, value: …} type things (depending on index needs). (Mostly for my CMS, where the arbitrary additional key/value pairs are tied to a more concrete document.)
[20:56:10] <GothAlice> Goopyo: https://gist.github.com/amcgregor/1ca13e5a74b2ac318017#file-sample-py Mine are. :)
[20:56:26] <Goopyo> if you have terrabytes of k/v data that are mostly ints, storing {key: …, value: …} would probably be 98% of your storage
[20:56:53] <agend> im using mongo now but its not enough fast - and I thought whole i need is string -> number with inc funcionality
[20:57:03] <GothAlice> Goopyo: Yeah, I tune specific metadata values depending on need and sparseness in the dataset.
[20:59:49] <GothAlice> Goopyo: Back in the day I joined a project that had Postgres, RabbitMQ, ZeroMQ, Redis (+ Celery), and membase. RabbitMQ had persistence, ZeroMQ was lower-latency. After some benchmarking and stress testing of failure scenarios, we replaced all of the aforementioned with MongoDB (for a 1M simultaneous user benchmarked Facebook game).
[21:00:42] <Goopyo> ah. Probably small zmq functionalities replaced. That is full fleged networking library though.
[21:02:52] <GothAlice> Goopyo: Most of our pub/sub for per-user eventing is handled entirely in Nginx, see: http://ecanus.craphound.ca/chat/ >:D
[21:03:15] <GothAlice> It's the 1.9 million dRPC requests per second replacement for Celery that really makes use of MongoDB capped collections (as a replacement for other queues).
[21:04:52] <GothAlice> Goopyo: https://gist.github.com/amcgregor/2712270 is my working prototype (running on my /chat/ link, above).
[21:05:34] <GothAlice> Goopyo: Uses webapp-mediated access to https://pushmodule.slact.net endpoints (internal redirect access only).
[21:06:02] <GothAlice> (Using X-Accel-Redirect returned if the webapp likes you; also obfuscates the channel names.)
[21:07:58] <GothAlice> Goopyo: https://github.com/bravecollective/forums/blob/develop/brave/forums/util/live.py is an example Python-side publisher for these channels; used by the forums software I wrote to live-update threads you have open when new comments are added (or comments are deleted/modified/etc.)
[21:29:08] <FIFOd[a]> { [MongoError: cursor killed or timed out] name: 'MongoError' } Any idea what would cause that? I'm iterating through my collection fine with cursor.nextObject() and then suddenly the cursor is killed
[21:29:45] <GothAlice> FIFOd[a]: Cursors are only allowed to live for a certain period of time before they are automatically cleaned up. You must have been .next'ing that cursor for nearly 10 minutes, though…
[21:30:10] <GothAlice> FIFOd[a]: http://docs.mongodb.org/manual/core/cursors/#closure-of-inactive-cursors
[21:32:10] <GothAlice> Well, either test cursor.alive and re-query when needed or handle the exception to re-query.
[21:34:34] <FIFOd[a]> I have like a 250k items in the collection. The documentation says it needs to be idle for 10 minutes, but that's not really possible unless the cursor is exhausted. According to the docs, if the cursor is exhausted I would not see that error.
[21:37:15] <GothAlice> FIFOd[a]: I'm not sure… the cursor *may* have been exhausted. I'm not sure how .nextObject handles it in your case. (In Python raising StopIteration from a generator being consumed by a for loop will exit the loop gracefully…)
[21:37:43] <GothAlice> You may need to check cursor.alive and cursor.hasNext yourself before calling nextObject.
[21:38:00] <FIFOd[a]> The documentation implies to me that there will not be an error but that with nextObject(function(err, obj) {}); obj will be null
[21:38:55] <FIFOd[a]> http://mongodb.github.io/node-mongodb-native/api-generated/cursor.html#nextobject
[21:40:06] <GothAlice> And all of my long-running (multi-second queries) are also maxTimeMS limited… so my code expects and handles timeout errors.
[21:41:29] <FIFOd[a]> I'm logging timestamps so I can see if it's 10 minutes between the last nextObject and the one that fails
[21:42:37] <FIFOd[a]> GothAlice: just changing how the collections are organized. So much for being schema-less =)
[21:43:15] <GothAlice> FIFOd[a]: http://www.slideshare.net/montrealpython/mp24-the-bachelor-a-facebook-game — see slide 13. XD
[21:43:49] <sheki> hey guys does anyone have any thoughts on why "$and" queries are not picking the right index.
[21:44:44] <Boomtime> also, do an example query using the mongo shell and add .explain(true) to the end, put the output in a gist
[21:46:36] <florinandrei> when doing db.serverStatus(), what is the meaning of network.numRequests? I thought it's the sum of everything under opcounters.*, but obviously that's not true. Clues, please?
[21:53:47] <Boomtime> if you have these two documents: a = {"_id":"NOvRVdV0ru","deviceIds":"none"}, b = {"_id":"yo","deviceIds":"id123"}
[21:56:09] <sheki> if i had to rewrite my good case to use "$and" it would result in the same query ?
[21:56:11] <Boomtime> "$or":[ {"_id":"NOvRVdV0ru","deviceIds":"id123"} ], <- this will only match a document which has both fields set to these values
[21:57:13] <Boomtime> your "bad" performing query looks like this: "$or":[ {"_id":"NOvRVdV0ru"}, {"deviceIds":"id123"} ]
[21:57:47] <Boomtime> that is a real $or, there are two possible paths to satisfying the query, indeed it must be fully evaluated twice and then the results sets merged
[21:58:49] <Boomtime> the only reason you need $and is to include the _id field more than once, otherwise it is implicit
[22:00:16] <sheki> i think there is a bug in how these queries are being generated, i will look into.
[23:09:34] <trco> Hello all. I am using mongo 2.4.8 and using the aggregation framework to group and count by a field. No problem. Now I would like to be able to a) put this into a collection (I don't have the $out command because that didn't come out until 2.6+) and b) manipulate the results (get a count of the count). Can anyone point me in the right direction?
[23:10:53] <joannac> 1. save the output of aggregation into a variable, and insert the result field into a collection
[23:12:49] <trco> I would like the results and then I would like to play with them. I think I actually would like them as a collection