Log file Viewer
#mongodb logs for Friday the 18th of January, 2013
[00:02:24] <hadees> sirious: yeah but I don't think that is all there is. I think actually if limit it by everything less than created_at it would work most of the time. The problem is if multiple comments are created at the same time.
[02:12:59] <sam___> keep getting this - assertion: 9997 auth failed: { errmsg: "auth fails", ok: 0.0 } - when i try mongodump even with the right credential, any idea?
[02:25:13] <aep> hm if i query for arbitrary garbage like {'abcdef' : 1234' } which is not contained in any document, i get all documents. is that supposed to work this way?
[05:21:42] <jwilliams_> does mongodb's mapreduce perform full table (collection) scan? or if within map function, using indexed field will let map function perform more effectively?
[05:47:06] <timah> hey guys… looking for some opinions regarding the best (reliable, efficient, fast, etc.) method for mass transforming about 30 million documents from a single collection to many (about 40) collections… i wrote a very slick mongodb shell script that serves in such capacity, however i can only seem to get between 2-5k documents read/transformed/written every few seconds or so.
[07:22:48] <samurai2> hi there, anyone use mongo java driver and get this error : Jan 18, 2013 3:19:34 PM com.mongodb.DBPortPool gotError
[09:07:52] <Guest_1448> what would be the best way to to about creating a graph out of thousand of entries in a collection with a timestamp value?
[09:53:15] <Guest_1448> is there a better/faster way to do this: http://stackoverflow.com/a/9388189 with the new aggregation framework?
[09:56:33] <kali> Guest_1448: it will be much more compact, and many problems will find much simple solutions
[09:57:37] <kali> Guest_1448: the aggregation framework can make simple operation on dates, so my guess is you can achieve this kind of grouping
[10:19:27] <Guest_1448> "is there a better/faster way to do this: http://stackoverflow.com/a/9388189 with the new aggregation framework?"
[10:35:18] <kali> Guest_1448: well, it would be pretty standard stuff with the AF: you can extract components of a datetime with http://docs.mongodb.org/manual/reference/aggregation/#date-operators, then group
[10:42:16] <xcat> Derick: Are there PHP source files that can be loaded into Eclipse to provide intellisense for PHP Mongo projects?
[10:48:20] <Guest_1448> um can I do a batch update with .update({}) to convert all those values to native dates?
[11:38:38] <xcat> If I have a logging collection that expires via a TTL date field and a type field which I want to filter messages by, how many indexes would I need and on which fields?
[12:15:28] <xcat> NodeX: if I also want to sort by insert order (either ID or date) do I need to create a compound index on one of those fields?
[13:36:23] <NodeX> quick question regarding the _id index.... if I do somehting like this ... {_id:ObjectId("1234"),uid:"5678"} with an index on UID is this as efficient as just looking up on _id ?
[13:38:38] <jmar777> NodeX: you shouldn't need a compound index, assuming your query is only for the `uid`
[13:38:44] <kali> NodeX: it's as efficient when it find something, marginally less efficient when it does not match
[13:38:55] <vlad-paiu> Quick question : any way I can run a find and modify query with the C driver API ?
[13:39:46] <vlad-paiu> can't see any explicit method.. like mongo_find_modify or something like that :)
[13:40:33] <NodeX> thanks kali, it was not somehting I have thought about before, just wondered if anyone else had
[13:42:51] <NodeX> I just wondered if the query planner would see my index on uid and merge it with the _id
[15:12:23] <JoeyJoeJo> I need to performance test how long it takes to insert my data (tens of millions of documents). What's a good way of doing that? I was thinking about writing a python script and just doing finish time - start time. Does mongo have anything built in to handle that?
[15:15:16] <JoeyJoeJo> So then comparing the time before and after my insert loop runs is a good idea?
[15:20:37] <NodeX> http://www.itproportal.com/2013/01/18/google-ceo-larry-page-taunts-apple-hows-that-thermonuclear-war-against-android-going/ :D
[16:28:20] <timah> that looked something like this: { _id: ObjectId('…'), event: 'someLongEventKey', properties: { publisher_hash: <MD5>, merchant_hash: <MD5>, region_hash: <MD5>, longitude: '-45.93939', latitude: 'anotherStringEek', user_hash: <MD5>, time: <SECONDS_TIMESTAMP> } }
[16:30:21] <timah> longitude and latitude not geo indexable in their current state, not to mention strings.
[16:33:47] <timah> so publisher gets moved out of the schema altogether and its hashes mysql pkid value becomes part of the collection name… so… 1 collection per publisher… e.g. p.25.e
[16:37:19] <timah> { _id: { m: <int>, o: <oid> }, l: [ <int>, <int> ], p: <int>, k: <int>, u: <int> }
[16:42:20] <timah> which is tricking awesome considering i'm not eating storage for a separate index.
[16:44:04] <timah> then each publisher event collection (p.25.e) has a corresponding count collection (p.25.c).
[16:47:52] <timah> yeah… and that's exactly why i'm here… figured i'd drop a line before converting it to pymongo.
[16:49:12] <timah> yeah… ok… mind taking a quick peak and what i'm doing just to see if i'm missing something logically?
[16:55:39] <timah> i mean… i was even going to go as far as using mongoexport with --query (per publisher) and --fields (only the fields i need) and dumping each publisher in its own file during that export… then using something super simple like php fgets for buffer read on large files and have php just swap all the values out and basically just transform the exports into imports.
[16:56:58] <timah> but if you think that having php (or anything else, really) do the work will speed it up a bit… it surely wouldn't take too long to port it over.
[17:12:08] <JoeyJoeJo> I've inserted a few million documents to my new DB. How can I make sure that sharding is working properly?
[17:25:32] <ehershey> http://docs.mongodb.org/manual/administration/production-notes/#disk-and-storage-systems
[17:26:31] <webber_> :) Ok, cool ... anything else beyond this. Any dev blogs to follow... must read wiki... or the like?
[17:27:18] <webber_> Oh , wait. That's on using and the like. I was wondering how MongoDB stores its data on disk.
[17:36:02] <ehershey> http://docs.mongodb.org/manual/faq/fundamentals/#are-writes-written-to-disk-immediately-or-lazily
[17:36:08] <ehershey> http://docs.mongodb.org/manual/faq/developers/#why-are-mongodb-s-data-files-so-large
[17:37:54] <ehershey> http://www.slideshare.net/mdirolf/inside-mongodb-the-internals-of-an-opensource-database
[17:50:42] <JoeyJoeJo> Based on this output from the sh.status() command, can anyone tell if my database is using all four of my servers? It looks to me like it's only using one server. http://pastebin.com/zKK4Rvy2
[17:52:48] <JoeyJoeJo> How much is enough? I'm running my insertion script now and I'm up to 1.5 million documents out of 180+ million
[17:55:02] <JoeyJoeJo> Is there any way to force mongo to always write to all shards? Right now I seem to be inserting about 5000 documents per second on the one server. Couldn't I quadruple that if all servers were being used?
[20:31:20] <w3pm> why are there different functions to connect - connect() and rs_connect() in the Erlang driver for example.. i thought replica sets were supposed to be transparent to the app?
[21:26:02] <SEJeff_work> I've got a record that has a key named "roles" with a value like ["super_user", "secret-sauce"]
[21:26:56] <SEJeff_work> I thought something like this might work, but it doesn't: db.users.find({"roles": {"$in": ["super_users"]}})
[21:33:19] <SEJeff_work> This is what I was after: db.users.find({"roles": {"$in": ["super-users"]}}, {login: 1, _id: 0})
[22:45:38] <Ironhand> hello; is anyone able to give me a rough idea of the current status of MongoDB on ARM CPU's? the information I've found suggests some people have done some serious work on it, but with very few recent signs of activity
[22:49:29] <Ironhand> well, any information would be useful, just hope you won't waste too much time on whatever a cross search would actually amount to
[22:52:00] <Ironhand> after a lot of consideration I decided I wanted to migrate a personal Python-based project from SQL to MongoDB, read up on various libraries, got a taste for the whole thing, then when I finally decided to actually install stuff it turned out running MongoDB on my QNAP NAS running Debian wasn't quite that easy...
[22:52:43] <Ironhand> I'm kind of hoping the popularity of the Raspberry Pi might have sparked some new interest in MongoDB on non-x86
[23:10:46] <tjadc> Ironhand: and by cross search I just mean searching it up as a second person, I probably search differently and may stumble upon something you haven't
[23:11:13] <midinerd> I'm looking at http://mongoosejs.com/docs/index.html <--- where it starts with //getting-started.js (where exactly is this taking place, a javascript file located <somewhere>, within npm, in mongoose?)
[23:13:56] <midinerd> ehershey: Thanks! So I include node.js from somewhere and then continue on with javascript documentation as they've got shown there?
[23:14:59] <tjadc> Ironhand: You've obviously taken a look @ skrabban/mongo-nonx86 - what are your thoughts around it :?
[23:15:03] <ehershey> midinerd: you might want to step through some basic node.js stuff before diving into mongoose imho
[23:18:27] <w3pm> hey guys, why is there rs_connect and connect? my understanding was replica sets would be invisible to the app
[23:20:36] <midinerd> [17:18:30] <aheckmann> midinerd: once you have node installed you can execute javascript files as shown in getting-started: node someScript.js
[23:29:00] <ehershey> w3pm: I think that the replica set layout can be somewhat transparent to the app but the app needs to know that it's connecting to a replica set vs a single instance
[23:30:20] <ehershey> for example what if you were building an administrative tool that needed to connect to an individual instance that is also a member of a replica set
[23:30:32] <ehershey> you would need a way to specify a single instance and know for sure you are connecting to that instance
[23:31:26] <ehershey> is it that you don't have any secondaries yet and you don't want to have to chane your app code when you get some?
[23:32:22] <ehershey> one thing I can think of would be to use a connection string that is easy to change without modifying your code