Log file Viewer
#mongodb logs for Friday the 25th of July, 2014
[06:58:29] <rspijker> d-rock1: had to leave yesterday, did you manage to find out what was going on with the hidden collections?
[08:03:02] <movedx> SO if I enable sharding on an existing collection, one which is growing in size now, and then need to rollback during the maintenance, can I do so?
[08:26:12] <rspijker> you shoul really make your shards replica sets… Then you can actually do maintenance...
[08:28:36] <movedx> Yeah I'm new to this stuff, so I'm trying to get my head around the architecture.
[08:31:34] <movedx> rspijker: Can you explain the steps involved in that? I will review the architecture of this cluster now.
[08:33:24] <rspijker> well… if your shard is a RS, you can just bring down each secondary and the shard will continue to function fine. You perform maintenance on the secondary and bring it back up. Repeat for each secondary. Then you step down the primary, it becomes secondary and you do the same thing to that. Then you’te done. Maintenance performed on all nodes without a second of downtime
[08:34:40] <rspijker> for this to work properly (and you to have actual guarantees about data retention etc. you need 2 real secondaries in each RS)
[08:34:44] <movedx> Right, OK. SO I'm going to login to the system now and determine primaries/secondaries, etc.
[08:38:56] <movedx> I think so? We have Shard 1, Server 1, 2 and 3. Then Shard 2, Server 1, 2 and 3.
[08:42:46] <movedx> rspijker: So for each "secondary" (Server 2 + 3), I take them out the cluster and then follow joannac's steps?
[08:43:24] <rspijker> if you just want to perform maintenance on the hosts, you don;t have to remove the entire shard
[08:44:18] <movedx> OK, sorry I'm confused. I thought what you were recommended was supplementary to what joannac was suggesting. How do your steps reduce a sharded collection to a non-sharded collection?
[08:46:03] <movedx> Sorry, I never clarified "maintenance." So we have unsharded collections which we want to shard.
[08:50:50] <Katafalkas> Deploying 3 config servers for production - they guide says that http://docs.mongodb.org/manual/tutorial/deploy-config-servers/ i just start it with 'mongod --configsvr' all three of them ... does this mean that the config servers are not in replica set ?
[08:51:26] <Katafalkas> lets say if later I would like to increase the size of one of the config servers ... how do I add additional config server replica member ?
[08:53:09] <Katafalkas> and if later in production i decided to increase the size if config servers ? is there a nice way to migrate ? like adding replica set member ?
[08:55:52] <Katafalkas> we are starting project. so i dont want to get like xlarge servers for config servers, would like to get a micro maybe or a small servers
[08:59:21] <movedx> rspijker: Sorry had a meeting. So we have unsharded collections, now reaching 10s of GBs in size, so we want to shard them. We need a regression plan should things go south.
[08:59:55] <movedx> rspijker: So I need a regression plan to rollback the sharding in the event something fails or stops working.
[09:00:35] <movedx> rspijker: It's a maintenance in ITIL terms. Perhaps MongoDB has its own definition, but it is a maintenance in... the entire IT industry ;)
[09:03:51] <Nodex> regression is easy. Take a snap shot, if all fails then remove the shards and put the snapshot back
[09:03:55] <rspijker> maintenance, to me, is the act of keeping something maintained. As in, keeping it in good condition. What you are doing isn’t that…
[09:08:02] <movedx> So going off of joannac's instructions (and not bickering about the definition of an industry standard term), I should basically removeShard() on the replica servers (the non-primary ones for that shard) and then movePrimary() those shards to a single/current primary?
[09:09:39] <movedx> Why have I got to shutdown the mongoS instance and config servers after doing this?
[09:12:23] <Nodex> movedx : just a quick point on your "industry standard term" comment. I have been in the industry for a very very long time, I have never even heard of ITIL until today, also the (proper) definition from the dictionary of "maintenance" is closer to what rspijker stated which is what most "normal" people would also think of making certain questions ambiguous if you're throwing these terms
[09:14:29] <movedx> Luckily we're mkoving away from Mongo in a month or two after we've migrated the data.
[09:15:12] <rspijker> Well, it certainly looks like you put in the effort to understand it before arriving at a conclusion
[09:16:21] <movedx> I wish I could understand it - it's nice tech. I really like the idea and I love the scability of it.
[09:18:29] <Nodex> unfortunately it can't be learned on paper, you must develop 'things' with it to have a deep understanding of it
[09:19:57] <movedx> "The best suggestion I can offer is that you schedule a maintenance window where there are no writes to the system..." heh.
[09:21:35] <Nodex> the ONLY way to migrate this properly is to make the system read-only for the migration time
[09:22:12] <Nodex> take a dump, shard, if sharding fails then remove the shards and restore the dump
[09:23:39] <crs_> Hello All. A quick question. I am trying to insert a json file into mongodb. Everything seems to be working fine except the date. It is getting stored as string. This is the sample json {"d":{"t":"2014-07-24T01:00:00Z"}}. I read that by making this as {"d":{"t":new Date("2014-07-24T01:00:00Z")}} it will work fine. But is this the only way to insert a json date into mongodb?
[09:29:04] <rspijker> http://stackoverflow.com/questions/12186006/inserting-json-into-mongodb-converting-dates-from-strings-automatically this might be useful though crs_
[09:29:07] <movedx> Nodex: So these steps: http://docs.mongodb.org/manual/tutorial/manage-sharded-cluster-balancer/#disable-the-balancer
[09:30:13] <movedx> Nodex: Sorry wrong link: http://docs.mongodb.org/manual/tutorial/backup-sharded-cluster-with-database-dumps/#procedure
[09:33:09] <movedx> Nodex: That's a good point. Sorry I never clicked onto that. Yes so I only have to dump from the primary the collection exists on?
[09:33:45] <movedx> Then if it goes belly up, delete the shards in the sharded collection and bring the backup back in?
[09:34:14] <Nodex> if you have the disk space this is probably the best solution, certainly the easiest
[09:34:48] <rspijker> so… how will you deal with all the data that is added between making the backup and deciding you don;t want to shard anyway?
[09:36:44] <Nodex> well that's a lie, you can setup a single instance to accept the writes and read from a replica set then migrate the new writes into the new cluster
[09:37:29] <rspijker> Why are you assuming it goes belly-up? And, in the case that it does that the answer is to just stop sharding altogether and go back to unsharded operation?
[09:38:30] <movedx> rspijker: All other suggestions are welcome. ANd I believe having a plan to deal with failure is a good idea - you disagree with this? I shouldn't plan for this going wrong due to a bug or incorrect operation on the data?
[09:39:45] <arussel> I'm starting mongo using: mongod --dbpath, is there any option I can set to see all queries being ouput ?
[09:45:38] <rspijker> movedx: having a disaster recovery plan is good, of course. However, this isn’t that.. You are considering implementing a certain ‘technology’ or way of operating and are planning against that not working. In my opinion that’s not a decision you make on a production system. You test it on a copy of your data in a staging set up and if it all works out well there, then you can start using it in production. You are attempting to
[09:45:39] <rspijker> apply a mechanism you would use for dealing with bugs as a way to decide which technologies and techniques to use in your system. That won't end well.
[09:46:38] <movedx> rspijker: Yep, I agree. I wish we had the automated steps in place to bring up a staging MongoDB cluster.
[09:52:41] <rspijker> fairly sure the buffer is only 256 bytes, anything larger than that will generate the message you are seeing
[09:57:03] <rspijker> I think all of the regular built-in logging capabilities will run into the same 256B wall
[10:04:21] <movedx> Does dropping a sharded collection delete all replicas of the collection/shards? Basically does it "clean out the house?"
[10:05:32] <joannac> if you drop a collection, all data that the mongoS knows about in the collection gets deleted
[10:05:48] <movedx> Sorry, what I mean to ask is: if I delete a sharded collection, if it removed from the cluster as a whole in the single operation?
[10:06:24] <rspijker> joannac: I have to admit I’ve not read it anywhere official. Only in comments in mulitple places (e.g., on here https://jira.mongodb.org/browse/SERVER-1794). So not sure whether it’s the actual truth or just a repeated misconception
[10:08:41] <rspijker> joannac: you might very well be right… I think I’ve seen longer ones as well...
[10:20:43] <talbott> quick q, if i've removed millions of docs, do you know how i free up that space on disk?
[10:43:11] <dawik> even if it were pre-allocated it would need to allocate more (and free memory at times) you would think
[11:01:54] <Nodex> the space is pre-allocated to ensure things try to sit next to each other on disc to avoid massive disk seeks
[11:35:47] <rdsoze> Found a SO question which references to older docs stating the same: http://stackoverflow.com/questions/9009987/improve-querying-fields-exist-in-mongodb
[12:05:46] <rdsoze> joannac: My queries using $exists have slowed down significantly as the collection size has increased. Will using a sparse index here help ?
[12:09:19] <rdsoze> joannac: find({"results.rank": {$exists: 0}, "results.0": {$exists: 1}}).sort({ uid: 1})
[12:52:42] <rdsoze> joannac: ahh ok. But the cursor is a BtreeCursor which means it is using an index rite ?
[14:43:18] <remonvv> Anyone aware of why arrays are stored as index keyed maps in BSON. Always struck me as odd.
[14:45:41] <remonvv> kali: That's my theory. It's not a particularly convincing one though. We're doing something where we store quite a bit of integers in an array and the size overhead is dizzying.
[14:46:32] <rspijker> kinda.. It’s saying an array is actually a very specific instance of this underlying thing we already have all of the helper methods for...
[14:49:31] <rspijker> o, multi char strings are of course far less optimal to store than their integer representation, nvm
[14:50:41] <remonvv> The optimal way to store an array (int array in this example) is 1 byte for element type, 4 bytes for array size, and 4 bytes per value.
[14:52:33] <rspijker> I’ve never actually tried it, but can you have an array like [“string”, 12, true] ?
[14:53:16] <kali> remonvv: well, rspijker is right, your scheme only works for array of a single type
[14:53:17] <remonvv> so you get all the overhead, the total size in bytes, each element has an (unnecessary) utf8 key 0term key string, etc.
[14:54:08] <remonvv> there is no overhead between "this is a float" and "this is an array of floats"
[14:54:22] <rspijker> still, 4 bytes for array size and 5 bytes per value would be quite an improvement
[14:54:48] <remonvv> yes, but for primitive arrays especially (so, fixed size elements) it can be better still.
[14:55:06] <remonvv> Anyway, I understand we're not going to solve this now. Was just curious if someone ever caught the reasoning behind it
[14:55:32] <remonvv> kali: True, in such cases you need a type per element. It's still a good optimization.
[14:56:24] <remonvv> And for the additional optimization of "this is an array of type X" as I mentioned it'd need some sort of optional strict typing check.
[15:01:03] <remonvv> Hm, so an int array with a single value in current BSON is 1(type == array) + 4(docsize of array) + 1(cstring element type) + 2(smallest cstring key) + 1(x10 integer element type) + 4(integer value) + 1(0x00 doc terminator) = 14
[15:07:01] <remonvv> the overhead per document is 5 bytes, the minimum element key is then 2 bytes, an int value is 5 (1 for type, 4 for value)
[15:13:11] <plowe> I want to $set a field in all documents within all arrays within a document within a document. Basically, I want to do this
[15:14:35] <bdiu> Is someone with experience optimizing MapReduce/Aggregations available for a screenshare consulting session? (paid)
[15:17:52] <Derick> plowe: in that way, yes - set can only update *one* field. In your case, you need to do it client side (but you lose atomicity of the update)
[15:18:02] <bdiu> joannac: ... if you have experience and think you can help send me a msg with your rate and we can chat...
[15:18:45] <rspijker> remonvv: that’ll be a cheap session: http://www.ebay.com/itm/Was-Opened-FERRARI-F355-SPIDER-BLACK-1-18-DIECAST-MODEL-CAR-BY-HOTWHEELS-/141353645345
[15:46:12] <fontanon> Hi everyone, I'm looking for some advice on mongodb design patterns. Lets suppose I've a mongodb collection for storing a timeline of events, i mean, with a timestamp each. For example, a phone call registry. A requirement of the app using this mongodb collection is: displaying the last call per phone number with the sum of the call durations. Which design pattern makes more sense? [Design pattern 1] The app make a queries with the agg
[15:46:12] <fontanon> regation framework to calculate the sum of call-durations and returns it attached to the last phone-call (sort by timestamp:-1). [Design pattern 2] To keep a separate collection with one record per phone number. On each new phone-call, do mongodb findAndUpdate on the proper record.
[15:47:33] <Derick> I'd do: store an entry for each call in one collection *and* store an entry for each phone number in another containing the number, and the sum-of-calls
[15:47:48] <Derick> upon insertion of the call entry, also update the phone number entry to add the new time
[15:49:46] <fontanon> Derick, how is the sum-of-calls calculated in that colletion? I've read something about some mongodb facilities to do this.
[15:51:39] <Derick> db.phonenrs.update( { _id: '555-111-2345' }, { total_duration: { $inc: 6000 } } );
[15:51:42] <Derick> db.phonenrs.update( { _id: '555-111-2345' }, { total_duration: { $inc: 600 } } );
[15:58:22] <d0liver> Hi, I'm relatively new to mongo. I'm trying to store a relatively small amount of binary data as a field on one of my collections. The field is the binary data for a profile image. The data is being stored as a string for some reason. I can verify this with db.myCollection.find({imageBlob: {$type: 2}}). What do I need to do to make mongo store this as binary data?
[17:04:25] <s2013> does mongodb have limitation on the size of the collection being imported/exported?
[17:30:00] <in_deep_thought> Is there a mongodb equivalent of sqlite? like where the db can just be a file that you access?
[17:45:45] <zfjohnny> What can I add to my mongo.conf file iln order to get a full query profile log file in my development server?
[19:24:01] <in_deep_thought> zfjohnny: I ask questions from time to time. I bet you are more interested in answer givers than question askers but here I am!
[19:51:28] <in_deep_thought> s2013: I think cause its friday.. Im not sure. usually has some activity
[19:59:00] <SubCreative> Running into an issue with making an update using $set, I have an object like so { user: "bar" details: { item1: "bar" }}
[19:59:31] <SubCreative> Every time I try to make an update to item1, if there are other items in the same object, it will remove all the others and only keep my updated item.
[20:00:11] <SubCreative> { user: "bar" details: { item1: "bar", item2: "mongodb" }} Becomes ------> { user: "bar" details: { item1: "update" }}
[20:01:12] <SubCreative> How can I add or update to a single key/value, while still retaining all my others?
[20:14:32] <SubCreative> db.collection('users').updateById(user, {$set: {apiKeys: {TestKey: apiKey}}}, function(err, result){
[20:16:51] <SubCreative> db.collection('users').updateById(user, {$set: {apiKeys.TestKey: apiKey}}, function(err, result){
[20:32:55] <nicolas_leonidas> hi, I'm trying to learn how to aggregate, I have a collection with documents that look like this:
[20:33:31] <nicolas_leonidas> I need to aggregate on browsing_metadata.ipgeo.country for example to see how many people are from what countries
[20:34:12] <nicolas_leonidas> the problem I'm facing now is using the dots in a hierarchy, $project and $group don't seem to be doing it
[20:34:46] <nicolas_leonidas> I'd like to modify this command to do what I said http://paste.debian.net/111690/
[21:14:10] <leru> Hi. I have don't know anything of mongodb. Is it just like javascript but as an database?
[21:53:44] <mango_> I'm using MMS, and have 'Host is unreachable.' I've just started my MongoDB server, how do I verify the connection again?
[22:25:36] <AlexZanf> hey guys I have a user model, and i am trying to add profile images to users. Should i just add a images[int,int] list to the user model? with the first param being the image slot, and the 2nd param being the image name or id? or should i make a new image model and compose with it?