Log file Viewer
#mongodb logs for Monday the 11th of July, 2016
[07:35:10] <Lope> is there any way to flatten data when I do a query? Like I pull the data out with dot notation like this: 'foo.bar.baz'. can I make baz appear on the root level of the object?
[08:37:43] <Industrial> I have 15 million documents that may not have a property. What is the best/fastest way to add them?
[08:40:40] <Industrial> `db.getCollection('incoming').update({ $or: [ { eid: null }, { ev: null } ] }, { $set: { eid: 'abc', ev: 5 } }, { multi: true })`
[08:59:09] <krion> I got an error with the "balancer". A chunk is tried to move, but fail, "data transfer error" in the errmsg.
[09:38:26] <kurushiyama> Zelest Well, I am more into stuff like this nowadays https://open.spotify.com/track/0ObNJajSJNBTvnvoDBHWVP
[09:39:35] <kurushiyama> krion Ah. Uh... Have to think where the balancer errors were recorded there. You are aware that you are doing experimental archeology with 2.6?
[09:42:32] <kurushiyama> Zelest Well, for me, they could be from Mars – in my opinion, national states and stuff are more than outdated.
[09:43:18] <krion> kurushiyama: Well, yes. We want to migrate from 2.6 to 3 but we first need to migrate from.... squeeze to jessie.
[09:43:21] <Zelest> I'm more into happy hardcore (hixxy, brisk, etc) or more industrial/electronical stuff (new order, nine inch nails, etc)
[09:43:56] <kurushiyama> Zelest something more mainstream, is https://open.spotify.com/track/3g5Yhx4mIARae1U2fLHW6c (German lyrics)
[09:46:19] <krion> kurushiyama: Here is the error i got. https://gpldr.in/v/tfhtg4Oj9J/IO0eOg99ozHtgq6S
[09:46:32] <kurushiyama> krion Yeah. Do a rolling update. I'd stop the balancer for that procedure anyway.
[09:48:29] <Zelest> kurushiyama, what about https://open.spotify.com/track/7KQTXUvQdEUqDMKdN8MKaa ? ;)
[09:51:30] <kurushiyama> krion Here is what I would do: Stop the balancer, do a rolling update to 3.2 and see what happens.
[09:53:26] <kurushiyama> Zelest A bit too fast to my taste for working and chillout (as a father of one 14 month old, you do little more), but it definetly has some quality.
[09:53:53] <Zelest> kurushiyama, weakling! I was listening to gabber during the nighty nighty of my new born ;)
[09:54:26] <kurushiyama> krion remove one member from the replset, Take that replset member offline, update the os, update the mongodb rejoin the member.
[09:57:39] <Zelest> i hightly recommend https://open.spotify.com/track/6HJH2v5BIqpKW34nsLV1O7 if you like Ott :)
[10:09:05] <krion> https://docs.mongodb.com/manual/faq/sharding/#is-it-safe-to-remove-old-files-in-the-movechunk-directory
[11:43:13] <hvemder> Emergency! I need help! I launched a small hobbyproject using mongodb, node.js and gmaps. Now the idea is taking off
[13:07:44] <pamp> hey, we have slow performance in our bulk insert last night, we had a lot of errors like "[QueryFailure flag was Timeout while waiting for migration commit (response was { "$err" : "Timeout while waiting for migration commit", "code" : 24 }).]"
[13:21:57] <jayjo> If I have epoch timestamps as a field in my document, can I use the aggregation framework to get a count for different time periods? Like weekly, daily, monthly counts? A GROUP BY in SQL
[13:24:43] <kurushiyama> pamp That is a bit like saying "My car ran for years! How can it be broken now?" ;)
[13:29:42] <jayjo> kurushiyama: I'm looking around but can't find it - is this something like "epoch group by mongodb" ... any other search term rec?
[13:31:10] <kurushiyama> jayjo Sorry, was by somebody else http://stackoverflow.com/questions/35861602/aggregate-group-by-time-of-intervals-of-15-minutes
[13:41:55] <kurushiyama> jayjo hence my "adjust accordingly". You do a modulo operation. It does not really matter on what kind of number you do it. And I'd first get it to run and then optimize – As Donald Knuth once said "Premature optimization is the root of all evil."
[13:43:55] <kurushiyama> jayjo I just wonder why you store an epoch distance instead of an ISODate – internally, an ISODate is stored as an epoch distance, anyway.
[13:45:49] <jayjo> OK - I didn't know that. Right now I upload this data and can make changes accrodingly. In the future it sounds like converting to ISODate at that upload time would be beneficial
[13:56:33] <jayjo> kurushiyama: I think you're right - being able to use the date aggregators (https://docs.mongodb.com/manual/reference/operator/aggregation-date/) is a huge help... Can I convert the time field to ISODate inplace or do I need to add a field to each document in the mongo shell?
[13:58:28] <kurushiyama> jayjo I'd probably use something like var docs = db.mycoll.find(); bulk = db.bulk.InitializeUnorderedBulkOp(); docs.forEach(...); bulk.execute()
[14:00:09] <kurushiyama> jayjo Within the forEach, you can take the current value, convert it to an ISODate and then add an according bulk.find(docsUnderscoreId).update()
[14:21:25] <jayjo> kurushiyama: so docs.forEach( bulk.find(docsUnderscoreId).update() ) ? not clear on this second component
[14:27:22] <jayjo> I think I'm letting mysql (groan) influence my mongo development in very negative ways
[14:42:55] <kurushiyama> jayjo May well be – albeit there is a datetime type in MySQL as well, iirc. Here is what I hacked together. Please check against samples first!!! http://hastebin.com/izinunazef.coffee
[14:46:01] <kurushiyama> jayjo I just hacked it down from the top of my head – so please review it properly.
[14:50:15] <jayjo> kurushiyama: thanks this helps a lot - I just need to get in the habit of using these 1-off scripts to accomplish this stuff
[15:00:00] <chris|> does anyone know where I can get the mongosniff tool? I cannot find any download and it is not part of the distribution
[15:12:06] <chris|> it is sort of a very verbose binary diagnostic log, but I am not sure if it's any useful if I cannot read it :)
[16:55:46] <jorsto> When using Wiredtiger, for timeseries, if I'm storing all the points in a single document, does it store the data in a hash table for faster updates? or should I still split up the time into smaller sections? (i.e. 0..........4071... updating doc.stats.0 is faster than doc.stats.4071)... is this still relevant?
[17:00:25] <kurushiyama> jorsto If you store every data point in the same document, you will have a rather small database: there is a 16MB size limit.
[17:01:25] <kurushiyama> jorsto Then, you could store it any way you want, since chossing an effective or ineffective way would hardly make a difference.
[17:02:26] <jorsto> https://docs.mongodb.com/ecosystem/use-cases/pre-aggregated-reports/#add-intra-document-hierarchy
[17:02:43] <jorsto> super old post, but it says it's "not as a hash table", but wasn't sure if wiredtiger optimized this.
[17:05:27] <kurushiyama> jorsto Actually, afaik, there is no such thing as a hashtable. And I do not see the point in having one. Lets take MMAPv1 as an example, for the moment to make clear why embedded documents for time series do not make sense atm.
[17:07:43] <kurushiyama> jorsto Simplified, a data file consists of documents, which in turn consist of field descriptions: the type and (if necessary) the length of the field. The default _id index holds the position of an individual document
[17:08:42] <jorsto> yeah, last library i made for a previous company, time series were stored in different collections depending on the step and it worked fine, but this new use-case, there's more documents where i need the values such as "post title" and would rather not do a join on all of them, so it's kind of a trade-off. i guess i can store it in each single document and assume the latest one is the title, but that's just that much more data :P
[17:09:08] <kurushiyama> jorsto However logically related individual events may be, putting them into a single document holds not benefit
[17:10:25] <jorsto> the minute i almost started doing, { 0: { 0...60 }, 1: { 0...60} ... 24 } is what made me cringe. back to 1 doc per event
[17:10:47] <kurushiyama> jorsto If you really want to learn about time series data, InfluxDB, and the way it treats values and tags really is en eye opener
[17:14:17] <kurushiyama> jorsto Not to mention that storing time series data in subdocuments makes aggregations more complicated – and I guess aside from adding the data, this is your most common use case. So we have covered the most common use cases (adding the data and doing aggregations on it) to be more efficient with single docs.
[17:53:27] <jorsto> is there an easy way to download a dump from mongo atlas and restore it locally for debugging
[19:10:36] <zylo4747> how do i get the list of primary and secondary servers with pymongo? Do I have to run a rs.status() and just parse the results?
[19:24:22] <phutchins> Anyone good with aggregations that can piont me in the right direction with what I'm doing wrong here? db.reports.aggregate([ { $match: { $timestamp: { $gt: ISODate("2016-07-04T14:45:17.809Z"), $lt: ISODate("2016-07-04T14:55:17.809Z") } } }, { $group: { _id: "$nodeID", used: { $max: "$storage.used" } } } ] ) I'm trying to match 10 minutes worth of data, group by the 'nodeID' field, and get the max
[20:05:57] <phutchins> Actually, I think I got it working... Does this look right? db.reports.aggregate([ { $match: { timestamp: { $gt: ISODate("2016-07-11T14:45:17.809Z"), $lt: ISODate("2016-07-11T14:55:17.809Z") } } }, { $group: { _id: "$nodeID", storage_used: { $max: "$storage.used" } } }, { $group: { _id: "$nodeID", total_storage_used: { $sum: "$storage_used" } } } ] )
[21:25:20] <ctooley> Is there a way to turn off the WARNINGs that get reported on client connect for things like transparent_hugepage? We've got replicasets in docker containers and I can't turn those things off on the docker hosts. But, it's breaking the client's ability to connect cleanly.
[21:33:45] <teeka> http://stackoverflow.com/questions/38276461/mongodb-bulk-upsert-throwing-e11000-duplicate-key-error <-- can anyone take a peak ? :)
[22:34:23] <poz2k4444> hi guys, I'm trying to sync some data to elasticsearch with mongo-connector, but the oplog thread says "reading last checkpoint as None" I think this means it's not going to sync anything, or am I missing something here?
[22:37:17] <poz2k4444> does this mean mongo-connector haven't found the oplog entry for the collection I'm trying to sync?