Log file Viewer
#mongodb logs for Thursday the 28th of April, 2016
[01:43:25] <xmad> I need to upgrade a 400gb database from 3.0 to 3.2, ideally with no downtime. I looked into doing it with https://github.com/compose/transporter which supports oplog tailing but it has some issues and can't spare too much time fixing them. What are my options?
[10:06:47] <kurushiyama> jokke: Sorry, just got up (long night). Lgtm, wouldn't have done any different.
[10:09:36] <jokke> i have a question about $out. I want to feed the result of the aggregation into a collection. problem: $out overwrites the collection if it exists. If i have different retentions in my collections that's a problem
[10:09:54] <kurushiyama> jokke: you could probably try "value.r" and "values.i" as keys in the group stage, thinking of it.
[10:10:28] <jokke> basically i want to create 15 min aggregates from the data by averaging the values
[10:12:11] <jokke> sth like 3 months worth of raw data, 12 months worth of 15 min aggregates, infinite amount of 1h aggregates or so
[10:12:19] <kurushiyama> jokke: Ah, so you want to downsample the data, and delete the original documents after the averages were created?
[10:13:09] <jokke> no the deleting i would do separately. for the first three months i'd have data in all three collections
[10:16:52] <kurushiyama> I'd probably have a collection for each granularity level and use $out for each level within the maximum retention time of the original data.
[10:17:26] <kurushiyama> As for "archive" data, you will be most likely better off when you do a double bulk
[10:18:11] <kurushiyama> jokke: You do not _have_ to, you can calc it on the fly. See the preaggregated data as a cache.
[10:19:45] <kurushiyama> Let's say you want a granularity of say 5minutes for the last day, 1h for up to a week, 1d for up to a month and 1w until max retention time.
[10:22:24] <kurushiyama> Ol, with this granularity and your max retention of the original data, you _could_ display everything up to 3 month with an "on-the-fly" aggregation
[10:23:36] <kurushiyama> Not necessarily, with proper early matches, good indexing and a good shard key, if applicable.
[10:25:24] <kurushiyama> So, an option would be to preaggregate the data every <insertMinGranularityHere>
[10:27:01] <kurushiyama> which – you are right with that – would cause quite some overhead (albeit not as much I would guess, since we'd do an average on <MinGranularity> at least), but access to the refined data would be blazing fast, then
[10:28:59] <kurushiyama> Ok, we have dealt with the downsampling of the data for the maxRetentionTime
[10:31:47] <kurushiyama> So, the system ran for 3 month, everyone lived happily and now we want to have the data surpasing the max retention time aggregated for the granularity chosen for +3month, the result merged with the according collection and the original data deleted.
[10:37:24] <kurushiyama> Get all documents with a var date = now - 3 month, build the average say for a calendaric month. For all documents each of the documents returned from the aggregation, find the according one in the +3m collection if it exists, insert otherwise.
[10:37:26] <jokke> i can still only think of not using $out but bulk inserting the results of the aggregation
[10:40:14] <kurushiyama> Just make sure you execute the bulk insert into the +3m collection _before_ you execute the bulk delete on the original data ;)
[10:42:01] <kurushiyama> Because here is the trick I used once: Simply add the processed document '_id's to a result field for the aggregation you use for outdating the data, and use this result set to add deletes to a bulk delete.
[10:43:20] <moneybeard> Is it bad to take a vmware snapshot of a running primary mongodb node in a cluster....
[10:46:31] <moneybeard> Yeah that is exactly what happened but not sure if it failed over completely.
[10:46:40] <kurushiyama> moneybeard: kali is right, it will block, which is the reason why I'd do it from a secondary
[10:46:51] <moneybeard> Then node01 and 02 complained about needing a reload as disk.service was out of sync
[10:47:45] <moneybeard> kurushiyama: So do I need to do the reload to get things back in a good state or will they sync up on their own in time?
[10:48:58] <kurushiyama> jokke: sh.status(true) or db.collection.getShardDistribution(). As *cough* documented ;)
[10:49:05] <moneybeard> kurushiyama: The issue is I needed a snapshot of the OS cause I was upgrading some packages so it had to be the primary, so in that case I was thinking my best bet is shut node 1 primary down , let a new master take over , then do the snap shot on 1
[10:49:55] <kurushiyama> Why does it have to be primary? You should always update a secondary first.
[10:54:41] <kurushiyama> moneybeard: Doesn't matter. Do not interrupt a service without need. Since you can do the update on secondaries first, fire them up again and have the primary step down prior to updating, there is no need.
[10:55:42] <moneybeard> kurushiyama: True that. But tell me why this says is journaling is enabled it can be done live?
[10:56:23] <moneybeard> From: https://docs.mongodb.org/manual/administration/production-notes/ It is possible to clone a virtual machine running MongoDB. You might use this function to spin up a new virtual host to add as a member of a replica set. If you clone a VM with journaling enabled, the clone snapshot will be valid. If not using journaling, first stop mongod, then clone the VM, and finally, restart mongod.
[10:56:57] <Industrial> I can only run this once (and i did it wrong) and now every time it says the replicaset already exists
[10:57:38] <kurushiyama> Industrial: restart machines without replset option, remove "local" database
[10:58:27] <kurushiyama> moneybeard: iirc, a clone is a _cow_-snapshot of the original data, with a dependency on the original image.
[10:58:29] <moneybeard> kurushiyama: NVM I found my answer : Clone> An exact copy of a VM at a specific moment in time, although this is usually performed on a powered off VM (a clone of a running VM is called as snapshot
[11:19:27] <jokke> do you guys know if i can get the total size of an index for a collection with the ruby mongodb driver?
[11:25:50] <FuzzySockets> What's the best way to seed some test data that has collections with references to other ObjectIds?
[11:31:43] <kurushiyama> FuzzySockets: Im out. I do not use Mongoose, and I strongly discourage to do so.
[11:35:42] <kurushiyama> FuzzySockets: It forces an SQL approach to data modelling: Define entities and their properties + the relations between the entities, model them, than bang your head against the wall to get the data you need to implement your use cases.
[11:36:23] <kurushiyama> FuzzySockets: Basically, it makes you treat MongoDB like an SQL-ish object store. Which could not be farer from the truth, imho
[11:36:53] <FuzzySockets> kurushiyama: I could see how that'd be a problem if you don't understand designing for a document store
[11:37:06] <kurushiyama> FuzzySockets: I fail to see any advantages. It is slower, it is bloated, and it abstracts away MongoDB's real strengths
[11:38:45] <kurushiyama> FuzzySockets: The slower and bloated part not from experience, but StephenLynx, who is usually rather active on the channel says so, and I take his word for fact.
[11:41:12] <kurushiyama> FuzzySockets: Imho, the proper approach to data modelling in MongoDB is to define your use cases, order them by how common they are, derive the questions you'd have to the data from them and create an optimized data model, with more common use cases having precedence over rarer ones when it comes to optimization.
[11:44:29] <kurushiyama> FuzzySockets: Not that this could not be achieved with Mongoose, but then you end up with an unnecessary abstraction layer.
[11:47:47] <FuzzySockets> kurushiyama: Speed of development is a higher concern for me right now. I already know mongoose syntax, and it'd be a fairly simple interface to replace if speed ever became an issue down the line.
[11:49:01] <kurushiyama> FuzzySockets: That is amassing technical debts. Just the other day there was some guy who used mongoose and now his application is getting some real momnetum, it fails to scale. I can understand your concern, just do not blame it on MongoDB ;)
[11:49:58] <kurushiyama> And the interface is not the problem: The data models produced by Mongoose are. And be aware of populate(), avoid it at all costs.
[11:49:59] <FuzzySockets> kurushiyama: I'm blaming it on my lack of knowledge on the native driver as well as already having 3-4 models written in mongoose. Besides, OOP principle 101, code to an interface, not an implementation.
[11:51:32] <kurushiyama> FuzzySockets: Well, you have been warned ;) Just be prepared that you'll need an ETL specialist later down the road.
[11:51:55] <FuzzySockets> kurushiyama: my application is actually fairly flat, and not incredibly write intensive
[11:54:22] <kurushiyama> FuzzySockets: You should be able to judge that better than I do. And I make a good part of my living from data-remodelling and (query) optimization, so I probably should not argue too much ;)
[11:55:22] <FuzzySockets> kurushiyama: Can't imagine you'd have much of a stance anyways considering you don't know what my application does ;)
[12:11:56] <scruz> $ifNull returns the original value if it’s not null, so it isn’t useful in this case.
[12:14:49] <scruz> each document has a field called status, which could be null. i want to count how many statuses are null
[12:25:16] <kurushiyama> scruz: Alternatively, if you want the count for all possible values of status, including _explicit_ null: db.checknull.aggregate({$group:{_id:"$status",count:{$sum:1}}})
[12:31:04] <scruz> it doesn’t have the status field and was included in the group for status == ‘other’
[12:31:49] <kurushiyama> scruz: Nope, it was included in the group null, since non-existant fields by definition evaluate to null
[12:32:40] <scruz> and i want to get a count specifically for missing data apart from the other explicit statuses.
[14:38:59] <jokke> kurushiyama: i see some weird stuff with some of my schemas when stress testing the cluster. it seems that some values are lost: db.flat.count() => 180000, db.panel_aggregate.aggregate([{$unwind: '$values'}, ..., { $group: { _id: '1', count: { $sum: 1 } } }]) => { _id: 1, count: 162340 }
[14:43:29] <jokke> flat is sth like { _id: { data_source_id: 'foo', timestamp: ... }, value: [1, 2] }
[14:44:29] <jokke> and panel_aggregate is sth like { _id: { panel_id: 'foo', timestamp: ... }, values: [{ timestamp: ..., data_source_id: 'foo', value: [1, 2] }, ...] }
[14:45:13] <jokke> here's the complete migration: db.panel_aggregate.aggregate([{$unwind: '$values'}, {$project: { _id: { panel_id: '$_id.panel_id', data_source_id: '$values.data_source_id', timestamp: '$values.timestamp' }, value: '$values.value' }}, {$group: { _id: '1', count: { $sum: 1 } } }])
[14:45:17] <kurushiyama> jokke: a) please make sure you have used the proper writeConcern. b) use pastebin if you paste code. c) In order to help you, I need what I request. I do not do this out of curiosity.
[14:50:10] <kurushiyama> jokke: MongoDB does not just "loose arbitrary data". There can be _very_ special circumstances in which that happens, and _only_ if your writeConcern does not match your durability needs: https://docs.mongodb.org/manual/reference/write-concern/ Every single time somebody claims that MongoDB "looses data" it can be tracked down to a user's fault. As a matter of fact, the control over durability on MongoDB is much higher than
[14:51:43] <kurushiyama> So, in order to debug what has happened, please do a .count() on both collections first
[14:54:23] <kurushiyama> jokke: And _please_ read about writeConcern, readPreferrence and readConcern.
[16:47:30] <kurushiyama> jokke: Sort of. If you have a replSet, you might want to make sure that the write made it to the majority of members. Reason: the member with the most recent data will be elected primary in a failover situation. With a wc:1, if an failover happens while the data has not made it to secondary yet, a so called rollback will happen when the member joins again.
[17:24:57] <saml> kurushiyama, given unstructured json objects, how would you insert them to mongo using mgo?
[17:49:46] <ramnes> mongo hostname/database --username 'user_name' --password 'WeIrDpAsSwOrD' --eval 'rs.slaveOk()' --shell
[17:51:30] <ramnes> we thought it could be that mongodb was compiled without SSL support on that machine
[17:56:44] <ramnes> but if I add an user on that database and try to connect with that user/password combo, it still asks me a password
[18:25:01] <kurushiyama> ramnes: Maybe more like "#ComeToMeAllYouWhoAreWearyAndBurdenedWithDatabaseRelatedQuestions"?
[18:25:35] <kurushiyama> ramnes: btw: giving passwords as a param is not necessarily the best idea, either ;)
[18:26:55] <ramnes> the command line is generated (we have a lot of databases) and executed in a subprocess, it doesn't usually go in bash/history
[18:40:58] <ramnes> (btw, that could be great to seperate both, so I don't have to recompile the whole DB each time :p)
[18:43:03] <kurushiyama> The idea was more that a buildflag caused a dependency to be built in a way not expected by MongoDB.
[18:48:14] <kurushiyama> ramnes: Would be great if you found out the per package build flags necessary and document it ;)
[18:55:00] <kurushiyama> And of course not about the impact of those buildflags on the dependencies. And an undefined buildflag does not necessarily mean "no impact"... ...iirc... *blush*
[19:04:42] <Derick> yes - except for openssl I think (which is not a problem for the link above, as it doesn't have ssl compiled in)
[19:12:39] <Yuri4_> Hi, guys! I need to train transfering databases between servers. Anybody know where I can find a simple database, that I can train on?
[19:13:15] <ramnes> https://raw.githubusercontent.com/mongodb/docs-assets/primer-dataset/primer-dataset.json
[19:14:17] <Yuri4_> ramnes, yeah, tried that one already. This is just a single .json file all you need to do is to press "import" and you are all done. Looking for a bit complex databse now.
[19:17:35] <ramnes> kurushiyama, Derick, if you guys have any idea, don't hesitate to send me a PM and I'll take a look tomorrow
[19:18:22] <Yuri4_> I have a freelance order to transfer databases. Probably will need to do that with remote connection to clients machine. Just wanna make sure that all goes smooth, don't do stupid mistakes and don't waste clients time.
[19:20:32] <kurushiyama> Yuri4_: Is there preexisting data on the target server? Do you need to merge it? Is the target server a production server? Is any downtime planned/acceptable? Are we talking of a sharded environment, a replica set or standalones?
[19:21:41] <Yuri4_> kurushiyama, standalone server, downtime is not acceptable. New server is a fresh install.
[19:22:09] <kurushiyama> Yuri4_: As a word from one freelancer to the other: accepting a contract you do not have any clue of is... not exactly a quality mark. You _live_ of your reputation. If you sack that, that's going to be major problem.
[19:25:08] <kurushiyama> Yuri4_: First: why is downtime not acceptable if the target instance is empty? I assume the source is production data then?
[19:26:41] <kurushiyama> Wait. What? Are we talking of a database migration from one server to the other, then?
[19:27:29] <kurushiyama> Ok, here is the problem: As long as the original server runs, it will get data, correct?
[19:31:17] <kurushiyama> Yuri4_: The only way to do a data migration without (or to be precise only minimal) downtime is to build a temporary replica set, consisting of the old server, the new server and an arbiter.
[19:32:22] <Yuri4_> kurushiyama, thanks for the input. Gonna take a break then set up a test lab. Cheers!
[19:42:10] <kurushiyama> Yuri4_: At least, you should bother to read the release notes. Not going to work with the way I suggested. So either, you need a downtime, to make sure the data does not get changed after or while you take the dump/export, or you need to downgrade the target machine to 2.6 first
[19:43:25] <Yuri4_> kurushiyama, thank you. I'm very ashamed. I deserved this. Going to get my head straight now. Gracias.
[19:44:34] <Yuri4_> kurushiyama, i'd rather don't waste anyones time and be very thourgh. You are too kind.
[19:46:13] <kurushiyama> Yuri4_: Ok, the idea is to synch to the new server, to 2.6. As soon as this is synced, you shut down the original server and update it to 3.0.x
[19:47:43] <kurushiyama> Yuri4_: removing the contents of dbpath before the update. Then you wait until the data is synced back to the original server, where we are on the target version.
[19:49:16] <Yuri4_> Oh, by the way, what tool to manage database would you recomend? I currently use mongochef
[19:49:29] <kurushiyama> Yuri4_: Then, you can simply update the new server to 3.0, wait until the data from the original server synced, have it step down. Then, you can restart the new server as a standalone and you are done. With minimal downtime, although this process is going to take a while.
[19:54:19] <kurushiyama> Yuri4_: So here is what has to happen _before_ the downtime: you need to set up the arbiter and the new server in 2.6, start them with the replset option as documented, but do _not_ initialize the replica set yet.
[19:57:09] <kurushiyama> Yuri4_: Then, downtime happens. The mongodb connection URI of the application needs to be changed so that it connects to the replica set we are going to create during the downtime. In parallel, you restart the original server with the same replset option you started the new servers with and initiate the replica set. Then, you add the new server and the arbiter to the replica set. At this point, the application can be restar
[20:00:53] <kurushiyama> FuzzySockets: if you have a better idea to minimize downtime for a migration from 2.4 to 3.0, I'd be happy to hear it!
[20:01:27] <FuzzySockets> kurushiyama: I was being facetious... you keep pinging some guy in here named downtime
[20:06:14] <kurushiyama> downtime: I am so sorry. I did not know.... But you have to admit that this is not exactly an ideal nick in a channel like this... ;)
[20:52:41] <tinylobsta> i'm using mongoose, and for some reason when i try to call methods such as findOne, findById, etc, on a model wrapper, nothing happens
[20:52:59] <deathanchor> for 2.4 mongo, to change a member from arbiter to secondary, I have to remove then add the member again?
[20:55:57] <kurushiyama> StephenLynx: Do you have your results for Mongoose performance somewhere? Took over the "job" of discouraging the use of Mongoose over the day, thought it might be helpful...
[20:56:32] <deathanchor> kurushiyama: this made me doubt it: http://dba.stackexchange.com/questions/61023/can-arbiteronly-replica-in-mongodb-become-secondary-and-what-it-means
[20:57:12] <deathanchor> kurushiyama: gist of the page it says you see this error with a reconfig: http://dba.stackexchange.com/questions/61023/can-arbiteronly-replica-in-mongodb-become-secondary-and-what-it-means
[20:57:15] <tinylobsta> i tried to move my models into a separate module and i'm requiring the module containing the models
[20:58:36] <kurushiyama> deathanchor: As per the dba question: Neil, in his one of a kind style, pretty much summed it up.
[21:01:17] <kurushiyama> deathanchor: You might need to force the reconfig. However, if you feel uncomfortable with it, simply remove the arbiter and readd it. should take seconds.