Log file Viewer
#mongodb logs for Friday the 13th of July, 2012
[00:37:00] <jiffe1> is https://github.com/mikejs/gridfs-fuse/ still the best implementation of a fuse client for gridfs?
[00:50:37] <Honeyman> 1. As far as I understand, there are limitations in map/reduce which make it generally unusable for "real time" queries. That is, it is generally more intended as, say, an administration tool, rather than a system that can be called on every user request, or even once in a minute/several minutes. Please correct me if I am wrong.
[00:50:58] <Honeyman> 2. The main reason of this is the limitation that only 1 thread of map/reduce can be run at a time (not speaking about sharding, etc). The usual "key-value" queries are not limited with it. Please correct me, again (say, if there is some more important factor than that).
[00:54:57] <Honeyman> 3. Are these issues limiting only, say, the db.col.mapReduce() function, or are there any other subsystems which share the map/reduce internals and hence also should be assumed "non-realtime-safe"? db.col.count(), db.col.distinct(), db.col.group()? Does the aggregation framework from 2.2 has the same limitations as mapReduce()?
[00:55:02] <Honeyman> 4. Any known roadmap when the mapReduce can be considered "generally ready for realtime"?
[01:54:54] <Guest70203> why would a database created from within an app not be accessible when using the mongo client?
[01:55:52] <Guest70203> i can still use the app to access the db and various db related files can be seen in /data/db. but it is as if the db doesn't exist when i connect to mongod with the mongo client
[05:25:41] <IAD> pr0ton: mongodb used mmap(2) function of the Operating system to map files into memory. memory will be released auto
[08:09:17] <mids> trupheenix: using pymongo and asyncmongo myself; considering switching to motor once it gets more stable
[08:15:23] <trupheenix> mids, i cannot figure out how to do a find() like in pymongo where i can do a find({"id":"1234"}) to get the entire document associated with that id
[08:19:49] <mids> trupheenix: so the 'fields' attribute is to specify which fields to return; not what to query on
[08:20:24] <mids> if you query for an object id; you most likely have to use pymongo.objectid.ObjectId("4ffd3b65140f5669b3f35f13") (or bson.objectid in newer versions of pymongo)
[08:25:47] <trupheenix> however if i do docs = yield apps.find(fields={"_id":"4ffd3b65140f5669b3f35f13"})
[08:31:35] <mids> and then use it like this: apps.find({"_id": objectid.ObjectId("4fffdc2ba6e9c8a00bbc19a7")})
[08:32:14] <mids> https://github.com/fiorix/mongo-async-python-driver/blob/master/tests/test_objects.py
[10:25:51] <naiquevin> hi, I am connecting to mongodb from a django app using pymongo library. I create an instance of the connection object as a module-global due to which the connection leaves open between requests ie the mongod logs just shows "connection accepted". How bad it is from mongodb point of view to have an open connection like this or should I be connecting on a per request basis ?
[10:58:10] <NodeX> the docs are sparse on whether one needs to close the connection after getBytes()
[11:03:57] <NodeX> [11:56:26] <NodeX> anyone alive who knows the gridfs part fo the php driver alive
[11:04:02] <NodeX> [11:58:38] <NodeX> the docs are sparse on whether one needs to close the connection after getBytes()
[11:06:06] <Derick> I can check, but I'm afraid that will have to wait until I've this code working :)
[11:37:03] <remonvv> I was expecting 10gen to have some sort of huge marketing push about it...but no
[11:45:02] <remonvv> well I reluctantly admit I noticed and then immediately jumped to the conclusion I broke something.
[11:51:25] <remonvv> I thought this entire channel would be all excited and abuzz with this. But it's a little underwhelming I must admit.
[12:06:28] <remonvv> I don't get why people get all excited about eclipses and whatnot but when ObjectIds switch from 4 to 5, which is a once in every 8 years event, everybody's like "so..wait..is that like the Y2K bug?"
[12:06:53] <Derick> remonvv: actually, the leap second issue was more troublesome than the y2k issue :-)
[12:07:07] <remonvv> This is the first time since MongoDB was released that ObjectIds start with a 5
[12:29:59] <remonvv> Anyone know of any resources online that can help me do some availability/MTBF calculations for full (MongoDB) clusters?
[14:17:03] <TkTech> Heyo. My google-fu is failing me - is there any documentation on profiling and debugging MapReduce?
[14:18:42] <TkTech> In particular we may have hit an issue where a MapReduce op decides to take the read lock and never releases it
[14:40:21] <ranman> TkTech: as a quick hack you can check the rest interface it has some locking stats
[15:12:38] <warz> hi all. how would i query the last 10 records, for example, out of a collection, but still keep them in the natural sorting order, which is ascending for an _id field, i believe? i think i'd need to skip documents, but how would you dynamically skip the correct amount of documents?
[15:22:46] <warz> i understand that part. i think what i need to do is something like this, though: db.collection.find().skip(db.collection.count() - 10)
[15:23:01] <NodeX> http://www.mongodb.org/display/DOCS/Advanced+Queries#AdvancedQueries-%7B%7Bskip%28%29%7D%7D <----- all in the docs
[15:25:35] <warz> wouldnt the result set be sorted descending, then? i guess i could resort in my code.
[15:28:23] <NodeX> [16:27:39] <NodeX> you can probably sort({_id:-1}); seeing as it's timestamp based too <------ "based"
[15:31:52] <NodeX> you're joking around which is putting ideas in people's heads who dont really understand Mongo all that well
[15:33:53] <sinisa> i see. we post at almost the same time $natural thingy.. ok , you are the hero :)
[15:36:45] <Bofu2U> first time I'm looking in this room ... ever. Can't wait to watch it more frequently now. :)
[15:37:31] <NodeX> Bofu2U : It's supposed to be about Mongo, this kid thinks that he can confuse users and that it's funny
[15:39:26] <diegok> hello. Which is the best way to replace a collection?. I mean, I have a collection with 30M docs and a dump with different 7M. I want to switch the faster I can. I was about to test renaming the existing one and then restore the dump... it's ok?, is there a better way?
[15:40:57] <markgm> Hello everyone. I was wondering if there is any support in doctrine2/zend for flexible schema document collections?
[15:41:47] <diegok> NodeX: so, I rename the dump folder, then load it, then rename existing and then rename the just loaded. Right?
[15:42:57] <diegok> when I rename the existing one it will keep indexes and the new dump has no idx info
[15:43:27] <markgm> NodeX: I am looking to store documents with a few known fields, however, the rest will be determine by a client admin and will fluctuate from subdomian. So I am looking to use persistent collections in Doctrine with a flexible schema. Is there any way to do this? Or would it be better to rely on Mongo's driver
[15:43:59] <NodeX> I wasn't aware that drivers were forcing schemas in a schema free datastore to be honest
[15:45:10] <NodeX> markgm : I wrote my own wrapper to go around the driver to make light of most things.. took around a day but it gives greate flexibility
[15:45:28] <markgm> it seems like flexible schema is one of the biggest pluses of mongo but every example I have seen is a forced/expected schema
[15:46:24] <markgm> NodeX: Yes, I have pretty much come to the conclusion that I would have to write my own. But I didn't want to waste the time if there was one out there, so I thought I'd come here first
[15:48:11] <Bofu2U> Random Q for whoever may know: is it possible to take a record, divide a numeric value on each by a constant, and then sort the output by the result?
[15:49:18] <remonvv> Bofu2U, no, $inc is the only in-place mathematical operator currently supported
[15:51:35] <Bofu2U> sinisa, yeah. I have north of 100,000,000 in a collection and need to sort by "relevance to now" aka number divided by timestamp, etc.
[15:56:10] <diegok> NodeX: should I rename folder or bson file?... can I restore it on another db and then rename db?
[16:07:49] <sapht> hello, how do i run getLastError from pymongo? neither db.runCommand nor db.getLastError is a callable
[16:08:30] <sapht> and the documentation just says to pass "safe=True" to save/update and that options propagate to the "resultant getLastError command"
[16:10:31] <sapht> ah... db.command({"getlasterror":1}) it would seem, let's hope i haven't got this backwards
[16:21:00] <diegok> NodeX: ok, rename can move from one db namespace to another and it also move/rebuild indexes. But it takes long. Renaming on the same db is instantaneous.
[16:21:07] <TheEmpath> hi… if mongo 2.0.6 is fsynching, and another connection tries to write, what does mongo do to the other connection?
[16:22:34] <groundnuty> hey, is it possible to use mongodb without runnign a server? - similar to sqllite? if not you gyus know any nosql dbs like that?
[16:24:35] <algernon> there are a couple of no-server key-value stores and the like (kyotocabinet, for example)
[16:51:39] <TecnoBrat> Hows the rc0 release coming along? is there going to be an actually packaged release, or is it basically just the nightlies?
[17:00:40] <Blackonrain> I came for freedom..........freedom from opression. Freedom to make love and not war. Freedom to streaming porn
[17:03:00] <Blackonrain> So my live database is down, not even sure if it needs repairs. The guy who set up the server has since left. Awesomely enough, the dev who set up mongo apparently put some info about it on trac before he left
[17:07:39] <sapht> i get this message a ton in my mongod logs: "query not recording (too large)" -- but running getlasterror after the failing queries seem to yield {ok:1.0 ...}, am i missing something?
[17:08:23] <TecnoBrat> its not an error, its just not going to print the whole entire query to the log, since its large
[17:09:09] <sapht> ah, alright.. kinda weird though, the query is relately small, i run another one as many times which is larger
[17:10:29] <sapht> what determines the log? it's basically 2 queries in a row, going to different collections in the same db, with safe=True for both
[18:08:06] <TecnoBrat> Curious what improvements have been done, my understanding is that in 2.0 ... all locks are global to the server process
[18:10:55] <vptr> what does eplSet syncThread: 1000 replSet source for syncing doesn't seem to be await capable -- is it an older version of mongodb?
[18:11:48] <TecnoBrat> (talking about 2.2, which I know isn't complete, I'm just trying to plan if we should stay with mongo, or move away)
[18:12:21] <TecnoBrat> we have 110 million documents in it .. and we need aggregation and better concurrency
[18:19:55] <ranman> TecnoBrat: a lot of this is available in the changelogs if you're interested in specifics
[18:29:41] <TecnoBrat> ranman: I read the release notes, but it doesn't have details (besides linking to the tickets, which doesn't have a lot of detail)
[18:31:35] <ranman> TecnoBrat: you looked at this? http://docs.mongodb.org/manual/release-notes/2.2/
[18:37:20] <TecnoBrat> yea, it links to the tickets .. which I just looked through the commits, gave me a little more insight
[18:39:48] <ranman> TecnoBrat: 2.2: better aggregation stuff, DB level locking and better pagefault management, data center tags, improved queryoptimizer, new read preference semantics, TTL collections, oplog replay, 16MB documents, shell unicode support/multi-line support, bulk inserts, … a bunch of cool stuff
[19:17:58] <junipr> hello. im wondering, how would i update, $set a property, on all documents using that doc's current value for that property? simply, all my documents have a timestamp, but it's not an actual javascript date object. i want to update all docs, and make it a date object.
[19:18:30] <junipr> i can do this in a loop in code, but was wondering if this is possible via mongo shell using a simple update?
[19:31:13] <ConstantineXVI> how do you know if a secondary in a replica set is taking queries? setting slaveOk in drivers, but not seeing the slave register queries/sec
[19:33:22] <chubz> when a mongodb node goes down, is there a way that a new node is automatically created and replaces the dead node?
[19:35:49] <chubz> but i'mwondering how it can do it on its own, cause so far it looks like its manual?
[19:35:58] <jiffe98> is https://github.com/mikejs/gridfs-fuse/ still the best implementation of a fuse client for gridfs?
[20:16:40] <TecnoBrat> ranman: yea, I knew about the changes in general .. was looking mainly at the details on the locking in general. I figured it out now though
[20:17:59] <TecnoBrat> still may not be as fast as we need, we'll see (we are collecting lots of stat counters, and require lots of numerical sums and groups on keys)
[20:18:53] <TecnoBrat> but, from reading tickets .. looks like there is a lot of room for improvement still performance wise
[20:33:23] <TecnoBrat> well our dataset is 110 million documents, each document has 9 keys, and up to 41 incrementing fields
[20:34:37] <TecnoBrat> we have tried some different DBs, like columnar DBs, and we can do aggregation on the entire data set and group / sum the columns in 100ish seconds
[20:35:34] <tystr> instead of running reports on tons of data, we pre-aggregate it by minute, hour, day, and month
[20:36:20] <javvierHund> but its more real-time-oriented so 110 million "rows" would be kind of the numbers you are saying above
[20:36:23] <TecnoBrat> we aggregate that removing one of them, aggregate on 8, which reduces the set down by a factor of like 4
[20:36:59] <TecnoBrat> and when we run a report, the engine is smart enough to grab the collection with the least number of keys
[20:37:12] <javvierHund> TecnoBrat: the db i am using is worked in "causefindr" www.manifact.com / www.causefindr.com
[20:37:37] <tystr> if you pre-aggregate by months, pulling the reports is very light on the database
[20:38:29] <tystr> well you can pre-aggregate the counters for all the metrics, and to the math in the application
[20:38:46] <javvierHund> in our application we often have dynamically linked paretos with user selected transformations and filters (in matrice form)
[20:39:05] <TecnoBrat> tystr: so we took the other approach, we aggregate based on the grouping keys
[20:39:31] <tystr> true, our use case is mainly tracking impressions, clicks, conversions, revenue, etc
[20:40:43] <tystr> we're doing ours almost exactly the way that's described here: http://docs.mongodb.org/manual/use-cases/pre-aggregated-reports/#pre-allocate-documents
[20:40:44] <TecnoBrat> for a simple use case, say we were tracking 4 keys of: source, subsource, destination, and country
[20:41:04] <TecnoBrat> we would then pre-aggregate down to source, subsource, destination, and drop country
[20:41:31] <javvierHund> they tend to be written by someone who studied economics rather than science
[20:43:01] <tystr> we've just rebuilt our app on mongo, and we're gonna send traffic to it monday, so we'll see how it performs
[20:45:47] <TecnoBrat> well tystr our app handles 30,000 requests a second, and mongo handles the updates of the stats just fine (we aggregate in the app as well before we do upserts, so thats not 30,000/s to mongo)
[20:46:31] <TecnoBrat> and with the new aggregation changed, I think it will solve some short term issues
[20:46:51] <mw44118> I ran db.currentOp(), and I see a lot of queries that I would like to terminate, because at this point, there's no reason to still run them (they are from webservers). How to do that?
[20:46:54] <TecnoBrat> (the preaggregation technique was actually taking too long to update the sub-collections we use)
[20:57:46] <TecnoBrat> mw44118: I believe you can manually construct those queries, you have to check the docs
[21:05:16] <AlbireoX> Hi, is it faster to get documents by their ObjectID or could I use my own unique string and have it be the same speed?
[21:09:20] <TecnoBrat> its still not as fast as a columnar, they were like 25 seconds for the same thing