Log file Viewer
#mongodb logs for Tuesday the 30th of July, 2013
[00:18:23] <bjorn248> hey guys, new to mongo here, but I didn't find anything clear in the documentation on this question...so I decided to come here. I'm trying to set up something very simple. Mongo replication with automatic failover. I have a node app using mongoose pointing at my database server, and the database is replicated on two other boxes (minimum replica set). No sharding at the moment. Is there a load balancer or something I have to set up between the node
[00:19:24] <rafaelhbarros> bjorn248: let me know what you find out because I'm interested on that one too.
[00:20:35] <bjorn248> I mean, if I set up a sharded and replicated cluster, I understand that mongos can act as a router in between the application and the database, but in the case where I am not using sharding, I am not sure how that is handled.
[00:32:58] <bjorn248> cheeser: oh really? awesome, so I can just set up mongos on the application box then? neat
[00:37:11] <cheeser> we did that at a former gig so that all the infrastructure was in place once we decided to shard.
[00:53:39] <tpayne> can anyone hope me figure out what's syntactically wrong with this line of code for aggregate?
[00:53:46] <tpayne> coll.aggregate(MongoDBObject("$group" -> MongoDBObject("_id" -> "$key", "value" -> MongoDBObject("$addToSet" -> "$value"))))
[00:56:24] <tpayne> i believe this translates to coll.aggregate({"$group": {"_id": "$key", "value": {"$addToSet": "$value"}}})
[00:57:38] <tpayne> DanWilson: can you take a look at my data, and tell me how to do a group by on it?
[01:05:52] <tpayne> not a count, i want the actual data. imagine a tree with a root node with id 1, and coming out of it is 2 branches
[01:06:31] <tpayne> DanWilson: I tried to explain it here: https://groups.google.com/forum/#!topic/mongodb-casbah-users/zp0izu4iaL8
[01:14:22] <tpayne> ok so i ran db.droplets.aggregate([{$group: {"_id": 0, "id": 1, "stuff": {"$addToSet":$list }}}])
[01:14:25] <tpayne> db.droplets.aggregate([{$group: {"_id": 0, "id": 1, "stuff": {"$addToSet":$list }}}])
[01:17:41] <DanWilson> db.droplets.aggregate([{$group: {"_id": 0, "id": 1, "stuff": {"$addToSet":list }}}])
[01:17:52] <DanWilson> db.droplets.aggregate([{$group: {"_id": 0, "id": 1, "stuff": {"$addToSet":"list" }}}])
[01:18:31] <tpayne> ok now i get this: "errmsg" : "exception: the group aggregate field 'id' must be defined as an expression inside an object",
[01:24:18] <DanWilson> db.tpayne.aggregate([{$group: {"_id": "$id", "stuff": {"$addToSet":"$list" }}}])
[01:32:43] <DanWilson> db.tpayne.aggregate([{$group: {"_id": "$id", "stuff": {"$addToSet":"$list" }}}])
[01:36:01] <DanWilson> that's a piece of piss. It's just a label in the statement: db.tpayne.aggregate([{$group: {"_id": "$id", "id": {"$addToSet":"$list" }}}])
[01:39:05] <DanWilson> you can call it whatever you like, just don't collide with anything else in the statement at that level.
[02:36:02] <Petrochus> does anyone know of a nice python mongodb wrapper that can transform things like `mongo.db.users.save({"_id": x}, {"$push": {"array": "value"}})`into, something like `users(id=x).array.append(value)`? pymongo, mongokit, and mongoengine don't really have such features
[03:20:47] <mscook> {'$and': [{'StrainID': 'S77EC'}, {'Class': 'substitution'}, {'$and': [{'Position': {'$gte': '1'}}, {'Position': {'$lte': '1000000'}}]}]}
[03:37:57] <tpayne> how do i write this query in casbah? db.droplets.aggregate([{$group: {"_id": "$id", "id": {"$addToSet":"$list" }}}])
[06:35:12] <ron> LoneSoldier728: never ask that on irc. if you have a question, just ask it and wait until someone responds.
[07:50:59] <trupheenix> anyone here know about MongoLab DB hosting service? I wanted to know if I can create a db with my free shared plan. I logged in and tried doing show dbs but it doesn't show me dbs.
[07:57:11] <cpu> My mongo server is entering a "state" I can't understand. 100CPU, locked all the time. Can someone help me understand the results of db.serverStatus?
[07:59:42] <Garo_> cpu: don't ask if you can ask, just ask your question (that includes the output of your db.serverStatus in this case)
[08:02:19] <cpu> in log I find: serverStatus was very slow: { after basic: 0, after asserts: 0, after backgroundFlushing: 0, after connections: 0, after cursors: 0, after dur: 0, after extra_info: 1460, after globalLock: 1460, after indexCounters: 1460, after locks: 1460, after network: 1460, after opcounters: 1460, after opcountersRepl: 1460, after recordStats: 1580, at end: 1580 }
[08:06:31] <cpu> Tue Jul 30 07:47:23.836 [conn78] insert crayon.samples_sec ninserted:628 keyUpdates:0 locks(micros) w:328706367 328706ms
[08:07:52] <trupheenix> I'm confused what kind of DB architecture I should use for my application. I'm trying to use MongoLab where I am limited by the number of databases I can have on a single connection. My current code assumes that multiple DBs can be accessed on the same connection. So is it a good idea to shift all my collections into one db and then connect to it?
[08:17:33] <trupheenix> Nodex, I have one data set for authentication another for storing images another for storing user data and so on
[08:21:02] <trupheenix> Nodex, what if I want to take out a collection and put it on a dedicated machine?
[09:10:07] <puppeh> is there the possibility to do a range query but searching on keys and not on values?
[09:11:47] <puppeh> for ex. I have documents like this: http://pastie.org/private/el4nquexdgla65cyttiqq what I want to do is to query for documents that are between "13_07_06" and "13_07_09" in their "d" fields
[09:38:11] <_Heisenberg_> just recognized that I built bullshit yesterday. adding members of my replica set (1 master 2 slaves) as shards to a sharded cluster makes no sense right? A shard should be a replica set (represented by the actual master), so what I would need to built is something like two or more replica sets which consits of 2 or more replica nodes each and then add the replica sets each as shards?!
[09:50:34] <_Heisenberg_> This would be the way to go right? https://www.lucidchart.com/documents/view/4295-3704-51f789dd-a4c3-65470a00507e (no config servers and multiple mongos modeled)
[09:57:18] <_Heisenberg_> Nodex: High Availability, Horizontal Scaling, Performance and Fault Tolerance are the main aspects. I'm building a prototype which makes use of polyglot persistence, MongoDB takeing the star role ;)
[09:58:53] <_Heisenberg_> just means you use several datastores for your application. for example a document store for a product catalog, a kv store for sessions and a rdbms for critical oerations where you need transactions
[09:59:59] <_Heisenberg_> Nodex: If you use sharding I assume that queries get routed to both masters, yes
[10:01:03] <_Heisenberg_> that's the key of horizontal scaling, isn't it? distributing your reads and writes over all available nodes...
[10:03:47] <_Heisenberg_> the master/slaves in a replicaset should have the same data, the shards will have different data depending on the sharding key
[10:05:44] <Nodex> rs0/1-slave = replica sets of each of the shards, meaning each side would have 50% of the data
[10:05:51] <_Heisenberg_> just wanted to show that the master handels the operations for the replica sets
[10:09:34] <_Heisenberg_> I think a multi-master architecture with replication between them would be much better :/
[10:10:13] <_Heisenberg_> if it comes to availability... of course you run into consistency problems then...
[10:18:00] <_Heisenberg_> so for a productive system you must guarantee to have the members of a replica set in different data centers to a
[10:19:21] <_Heisenberg_> because if you have a network partition between your replica sets or a datacenter fails, half of the data would be unavailable
[10:21:45] <Nodex> but replica sets of shards should be in a separate data center to get anywhere near it
[10:24:49] <_Heisenberg_> I don't get it why in this example (http://docs.mongodb.org/manual/core/replica-set-architecture-geographically-distributed/) the Secondary in Data Center 2 should never get primary?
[10:29:11] <_Heisenberg_> yes but it makes no sense to me. if datacenter 1 goes down the secondary in data center 2 should get primary to ensure availabilty?!
[10:46:54] <Andy80> hi I'm performing this aggregation on a collection that I've: db.campaigns.aggregate({'$group': {'_id': 'budget', 'daily_budget_sum': {'$sum': '$daily_budget'}}})
[10:47:47] <Andy80> I'd like to filter, for example, the campaign using some parameters.... example: campaign_status = 1, account_enabled = 1
[10:51:28] <Nodex> http://docs.mongodb.org/manual/tutorial/aggregation-examples/#states-with-populations-over-10-million <--- shows a $match
[10:55:14] <neophy> my mongodb document some thing like this: http://pastebin.com/xXR8DnUh .most of my query is based on timestamp, hostname and message. Now I want to build index for timestamp and hostname for existing collection.
[10:55:51] <neophy> does it build index for timestamp and hostname db.events.ensureIndex( { "timestamp": 1, "hostname": 1 } , {background: true, name: "events_time_host"} ) right?
[10:58:53] <neophy> existing collection right? I have to do this through client interface for new insert right?
[11:01:09] <Andy80> Nodex, from what I'm reading, the $match is more similar to the sql HAVING command. I would need something relative to WHERE instead.
[11:12:03] <neophy> Nodex: I have single node MongoDB instance which has last two months syslog. The actual syslog size is 20GB in flat file. but MongoDB takes around 600GB. There are lots of dbname.xxx file with 2GB size. Does it make sense?
[11:23:07] <rspijker> neophy: if you want more specifics there was a talk I saw recently. Talking about the storage engine, pretty interesting stuff
[11:24:10] <rspijker> And I recently learned that in some scenarios the freelist algorithm is just utter crap :)
[11:25:07] <cpu> OK.. I now know better something I've been researching that is happening to my mongo.
[11:25:29] <cpu> I'm using nodejs as client and I do roughlty 10K inserts per second (according to mongostat)
[11:26:08] <cpu> My nodejs service outputs that the submitted the documents to mongo, but they appear in the DB only after a few minutes
[11:40:03] <_Heisenberg__> the docs are a little confusing towards this. because if I would call sh.addShard() for each member of my replica set I would end up with 4 shards in my case (2 replica sets with two members each)
[11:42:38] <rspijker> _Heisenberg__: the docs aren;t confusing, they say exactly what you need to do
[11:43:12] <rspijker> Add each shard to the cluster using the sh.addShard() method, as shown in the examples below. Issue sh.addShard() separately for each shard. If the shard is a replica set, specify the name of the replica set and specify a member of the set. In production deployments, all shards should be replica sets.
[11:45:42] <rspijker> cpu: I'm guessing you are doing the find over a different connection than the insert?
[11:45:51] <_Heisenberg__> If the shard is a replica set, specify the name of the replica set and specify _a_ member of the set -> so, which one? don't I have to specify all members of my replica-set if I want the whole replica-set to be a shard?!
[11:46:17] <rspijker> _Heisenberg__: nope, any member should do. Remember, the members know what other members are in the RS
[11:46:53] <rspijker> back in the day (not that far back, actually). you had to specify all members of the RS
[11:48:08] <braoru> I have a collection of document and in each document a set of included object like ["stat1":"1","stat2":"3"] actually I can use aggregation to count number of occurence of stat2, stat3 etc.. but can I have the sum of all stat2, stat1, etc ?
[12:42:55] <hroark> so i have voting object like question.options.votes, where options and votes are arrays.
[13:01:06] <Nodex> col.save(doc); takes any document, I would be VERY surprised if it didn't allow arrays in the document
[13:02:43] <braoru> I got precision for my last unclear question :) I have record like that : http://paste.fedoraproject.org/28967/37518917/ and I would like to know if it's possible with aggregation api to get sum of "motivations" and get something like { "motivationXX":"44", "motivationTT":"66", ..} word are in french but if needed I can just make it completely English and generic..
[13:24:36] <aandy> having used coffeescript (not anymore, just toyed with it), it's not as bad as people try to make it out to be. everything is context
[13:24:54] <aandy> if you expect it to be a language, it's not. it's sugar and shorthand, use it as such :p
[13:25:50] <aandy> i deemed it more useful for one of two people: 1. those who are new to javascript, and 2. those who like the syntax
[13:27:26] <aandy> i write a lot of javascript, and while my cs projects are easy to read, it's not as easy to switch between the two
[13:28:00] <aandy> and coffee has its own quirks, but as per 1. it does help novice becoming better js writers
[13:29:46] <aandy> anyway, i don't use it. but it did teach me a few things about a complicated `this` nest i had :)
[13:30:15] <aandy> right, but the same can be done with checking.for && checking.for.nest, but i agree when it becomes more nested than that it's ugly and confusing
[13:30:48] <aandy> and if you go so nested, it's usually because you're *trying* to structure random data :p
[13:31:24] <aandy> which is yet another problem in itself. at best the checking.for?.nested?... ignores your failed initializers
[13:32:22] <aandy> but still, if people want to use coffee, let them. for smaller projects it can work very well
[13:33:06] <aandy> the only example i have is github's hubot. it's way more readable, and easier to follow the plugin/interface use than the js implementation would be
[13:33:37] <aandy> i didn't say not large, i said small. it can work for larger aswel, but i've never tried it, so can't say
[13:34:51] <Nodex> generaly I find people who like shortcutting and using these silly tools all tend to stick to whatever makes them type less
[13:37:15] <Nodex> I've looked at all these new angluar type things, seems a lot of work to do simple stuff
[13:39:18] <Nodex> I've not seen any SPA that can do things I can do faster or in a less / more coherant manner
[13:43:37] <Nodex> it's not difficult to mimic that in any app, you don't need to be an SPA for that
[13:45:04] <Nodex> an SPA cannot kill the actual latnecy between my app and the server. For example, a chat app between 2 people - one cannot update the other on the other side of the world any faster than actual network latency
[13:45:55] <Nodex> what point is an app with all that weight and bloat to just interact with yourself?
[13:48:05] <Nodex> I want to tweak my profile so ONLY my profile is changed and refelcted in the page
[13:49:08] <Nodex> but I've always written my apps like that, this is nothing new to me, I have been doing it this way for many yeas
[13:51:53] <Nodex> I don't agree that SPA's are the future, they have a place for those that want to use them sure but they are not the only way to accomplish the things they do and VV with non SPA's
[13:52:07] <chostrander> Hello all! Speaking of performance... I have a mongodb that consists of 525 million records... every day we must process ~40 million records that can either exist (update) or be new (insert).
[13:56:26] <chostrander> one more thing... has anyone used TokuMX? Are their claims of tremendous speed improvement for real?
[13:59:29] <leifw> chostrander: I'm one of the developers of TokuMX, you can ask me questions or come to the #tokutek channel if you like
[14:00:39] <chostrander> Hi leifw! I understand your response... just wanting to peoples experiences! :-)
[14:03:09] <leifw> chostrander: because of concurrent writers, we can't support a particular count() optimization that vanilla mongodb does, but for almost everything else, pretty much everyone we've heard from has seen big performance improvements
[14:04:16] <leifw> chostrander: these things are of course workload dependent, and I think most people that have tried us were already hitting the limits of mongodb's performance, so there's some selection bias
[14:15:14] <Andy80> is there a way to compare the _id against a list of ids? something like {'_id': ['id1', 'id2', 'id3']}
[14:15:48] <leifw> Andy80: I think you want an $in query: http://docs.mongodb.org/manual/reference/operator/in/#op._S_in
[14:42:58] <remonvv> leifw: Out of pure curiousity, can you explain a bit where the performance gains come from that are being claimed here?
[14:46:51] <Nodex> http://blog.mongodb.org/post/56876800071/the-most-popular-pub-names <--- I thought Derick wouldv'e written that tbh
[14:47:06] <aandy> Nodex: agreed. that's also why i don't try out that manye of the new <fancy libraries>, as i just can't see the use case, and *worse* that it's a lot of work to do less. it's a bad trend :)
[14:47:45] <aandy> it's basically, FORGET EVERYTHING YOU THOUGHT YOU KNEW ABOUT THE WEB, then learn how they think web should be, change all your habbits and then it's super easy to use
[14:49:14] <Nodex> aandy : I agree, to much abstraction going on whilst waiting for the browser vendors to catch up
[14:55:36] <remonvv> Not that odd until I tell you that it's in the red light district and does not sell banana related beverages
[14:56:29] <remonvv> leifw: You here? Where can I find technical details on TokuMX? I'm looking through the code but it's a little tricky to see what you've changed.
[14:57:28] <remonvv> NodeX: Really? I've been looking at that and all I can find is something about fractal trees instead of b-trees and some benchmarks.
[14:58:03] <remonvv> I need to know if it breaks contract with vanilla MongoDB and why it's so much faster.
[15:01:28] <remonvv> Hm, seems to focus on insert speed for a lot of the performance improvement claims.
[15:02:42] <remonvv> Ah Nintendo..I wrote a Gameboy Advance emulator in Java. A healthy mix of fun and masochism.
[15:04:30] <Nodex> hello world apps are kinda stupid because the only person seeing it is the developer and the developer is in no way the entire world
[15:05:02] <deepy> You're beeing too literal, the greeting is extended to the world, it's a wide audience, it doesn't mean that everyone has to read it just that everyone is encouraged to :-)
[15:26:54] <_jo_> I'm seeing this error come up on our staging servers: https://jira.mongodb.org/browse/SERVER-7768 If I pull a more recent Python driver, can that work around the problem?
[15:27:17] <_jo_> Let's assume I can't upgrade the MongoDB version because we're not hosting it in-house.
[15:43:38] <mrlami> does mongoDB have a php driver and if so what are the risks that in 6 months time it'll be invalid and unusable due to changes?
[15:44:41] <cheeser> yes, it does. and it's a pretty safe bet. the php driver author sits two desks down from me. :)
[16:02:19] <leifw> remonvv: if you'll be in the boston area tomorrow you can go see my coworker Zardosht give a technical talk about how we do it
[16:03:05] <leifw> remonvv: but basically what we did was we took mongodb, ripped out the storage code (b-trees, file allocation, extents, all that) and replaced it with our storage library that we use in tokudb
[16:03:39] <leifw> remonvv: it's an implementation of a fractal tree, which is a write-optimized data structure
[16:04:37] <leifw> remonvv: http://youtu.be/c-n2LGPpQEw is a description of why they're better than b-trees, we have a bunch of other posts about it if you google around
[16:05:28] <leifw> here's the talk tomorrow in boston if you're interested: http://www.meetup.com/Boston-MongoDB-User-Group/events/127576442/
[16:05:51] <leifw> there's also this: http://www.tokutek.com/2013/07/tokumx-fractal-treer-indexes-what-are-they/
[16:07:32] <leifw> the compatibility issues are: no geo/2d/2dsphere yet, no mixed vanilla/tokumx clusters, otherwise apps shouldn't notice a difference
[16:07:53] <leifw> I'll stop flooding the channel now, come ask in #tokutek if you have specific questions or want a longer lecture
[16:10:22] <remonvv> leifw: Thanks ;) I'll read up on it tonight. The only google hits on Fractal trees are to Toku so it's hard to find additional info that isn't directly from Toku ;)
[16:17:07] <flaper87> What does multi means here? (.explain()'s output) "cursor" : "BtreeCursor sub multi",
[18:35:41] <braoru> hello, I have a collection of document looking like that http://paste.fedoraproject.org/29073/37520921/ and I want to obtain a list of "motivations" containing the sum of each .. any idea how to proceed with aggregation or something else ? (I can change the shema if needed)
[18:36:44] <ninepointsix> Hey, is there a way to force an insert to behave like an upsert? (or have the update command take an array?)
[19:03:55] <braoru> sad .. ok will have to extract each line bring them back to python and do the sum by code ..
[19:55:00] <tg2> on a server with 256G of ram, seems a bit of a waste to limit batch inserts to 16MB total seeing as single document size limitation is 16Mb
[20:04:55] <braoru> DanWilson, thx a lot now I just need to extend that to be able to obtain the sum by id :)
[20:06:52] <ron> DanWilson: and for future reference, avoid pasting more than 2 lines in irc channels. that's why paste sites were invented.
[20:12:50] <tjmehta> How does mongo consider subdocuments distinct - will it compare values of keys or does distinct only work for fields that hold a literal value?
[20:13:09] <braoru> DanWilson, any idea on how I can get the sum by id ? instead of the global one ?
[20:21:05] <tjmehta> cheeser: field by field, so what about indexing fields that are subdocuments -- is it sufficient to index the key of the document or should each sub key be indexed?
[20:21:11] <DanWilson> braoru: did you figure out how to reinsert the document with the right types?
[20:21:46] <cheeser> tjmehta: you should test it but I would think if you're querying by subkeys you should index those explicitly.
[20:24:29] <tjmehta> cheeser: okay so last question if I do a distinct query for subdocuments and I have indexed the key for the subdocument (but not the children) - is that enough indexing for the distinct query if it is doing a field by field comparison?
[20:27:51] <braoru> no I get multiple record but data seem to be wrong .. I have to verify my insert
[22:05:34] <Industrial> Say I have a SQlite database with 500MB of records in 1 table with a date column with a unix timestamp (number) in it. I want to seach through this data set by start/end time. Currently takes 20-30 sec. How would mongodb do a better job at this?