Log file Viewer
#mongodb logs for Wednesday the 18th of May, 2016
[00:23:01] <Freman> kurushiyama: https://www.freedesktop.org/software/systemd/man/systemd.resource-control.html#TasksMax= is why mongod wasn't working for us :D
[07:21:32] <sivi> http://mongodb.github.io/node-mongodb-native/2.0/tutorials/enterprise_features/?_ga=1.41901026.1114324834.1461242741
[10:18:45] <m0rpho> hi there, my pymongo connections are dropped exactly after 10 seconds, do you guys have any guess what that could be?
[10:47:37] <m0rpho> its very difficult to trace as its a production server environment with lots of connections and it doesnt happen in the local testing environment
[10:50:34] <m0rpho> i just thought you guys might have any clues or experiences where there is a 10s timeout
[11:13:37] <m0rpho> do you have have any idea what I should look for? socketTimeoutMS? maxIdleTimeMS?
[11:13:58] <kurushiyama> m0rpho: I have no clue about pymongo, I just came up with a theory that might fit the facts... ;) maxIdleTimMS sounds about right.
[11:44:03] <Lumio> Hey guys! I was wondering what the best practice is here… I’m thinking of having all my invoices for my clients saved in a MongoDB and I would attach the generated PDF file within it. Do you think this is good practice? I would only store it, so i have everything in one place.
[12:17:36] <grug> Lumio: no - you have all the information in your database (presumably) already, so just generate it when you need it
[12:32:09] <Dlabz> Hi, all. If I load a huge json file with geometry in the mongodb , where one of the properties of root is an array of individual geometric objects, will I be able to querry for them? will that be efficient? Thanks.
[12:41:13] <Dlabz> I'm having issues parsing those files, so I'm planing to pass the file to mongodb, and use nodejs, protocol buffers and websockets to load individual pieces
[12:45:39] <Dlabz> yeah, I was considering that option... but I'd face a problem with older files, which I'd need to regenerate all
[12:47:05] <cheeser> compeman: please don't press enter every 4 words. complete a thought. or four. then hit enter.
[12:57:44] <Dlabz> so, since I need to change my json files, to be able to load them in the mongo in a correct way, what's my best option to programatically convert them to prefered format?
[12:59:39] <Dlabz> I'm messing with all this since php and clients are crashing due huge file size...
[13:00:10] <Dlabz> ... so loading the json file for conversion will still be a huge strain to the server...
[13:04:43] <Dlabz> obviously, end-users don't really think in terms of optimizing the graphics for web, so I end up with a whole hospital of flexy-tubes... easily thousand triangles
[13:07:13] <Dlabz> if I need to parse the huge files using node.js, there's really no point in using the mongodb...
[13:07:45] <Dlabz> I can just store parsed pieces in binary form, since I'll be using that to transfer data to client
[14:14:50] <JsonTooBig_> hi there, I wonder if anyone can give me a hand, I am getting the following error message while replicating to a new 3.2.6 node: [NetworkInterfaceASIO-BGSync-0] Assertion: 10334:BSONObj size: 17338193 (0x1088F51) is invalid. Size must be between 0 and 16793600(16MB) First element: id: 4408015349485
[14:16:40] <JsonTooBig_> this happens several hours into the replication, and afterwards the replication process has to restart
[14:21:48] <silviolucenajuni> JsonTooBig_: Can be a problem because of a large document to be replicated ? Maybe your master have a document limit higher of your secondary ??
[14:22:53] <JsonTooBig_> its possible, but the error context doesn't give us much information to go by, we don't even know the collection that contains the offending document
[14:24:22] <JsonTooBig_> also the elementid in the error message doesn't makes sense to us, because we don't use numeric ids for any of our documents
[14:27:11] <kurushiyama> JsonTooBig_: You might want to check other collections and or databases as well.
[14:27:19] <JsonTooBig_> that is the thing, that id isn't for any of our documents, we don't use integer IDs for anything
[14:28:13] <kurushiyama> Well, it has to come from somewhere. You should check, even if it does not make sense to you. Maybe somebody did something dtupid.
[14:29:30] <JsonTooBig_> the error context doesn't mention the collection, I will write a script to iterate and check them all
[14:33:17] <kurushiyama> silviolucenajuni: Iiirc, log level increase _what_ is logged, not the log details. But the idea is good. Lemme check.
[14:33:21] <JsonTooBig_> i have the verbose error, it is just a lot of noise, other than that assertion which triggers an exit with signal 6
[14:37:03] <kurushiyama> Whereas I would be more concerned about the fact that somebody managed to increase a doc beyond the BSON size limit. I have never actually tried that and can only assume that this could be caused through updates.
[14:46:24] <silviolucenajuni> JsonTooBig_: loop over all documents in all collections in all db to check size of documents is a options ?
[15:25:21] <StephenLynx> if I am not mistaken, it's kind of slow, but I might be wrong. haven't looked into it yet.
[15:27:37] <kurushiyama> ange7: The thing is that it _is_ slow. It is no replacement for a JOIN. It is just there for saving an extra query for small result sets. You should rather go with redundancy in most cases.
[15:30:14] <kurushiyama> ange7: That depends. It is not about the _data_ set, but the result set of your current stage in the pipeline.
[15:32:31] <kurushiyama> $lookup might well be used to save an additional query for small result sets. But since it has either to be called for every doc in the result set or as an $in query, I expect it to be less than perfect for medium result sets already, not speaking of large ones.
[15:34:52] <kurushiyama> oky: Morning. Well, it does correlations between tables. I have no clue about how it is implemented, but as described above, I doubt it will be good for more than just a couple of dozens docs in the result set.
[15:36:11] <kurushiyama> ange7: As said: That depends on the number of docs in the pipeline after $match. If you need to correlate, you are most likely _much_ better off utilizing redundancy properly.
[15:38:26] <kurushiyama> ange7: Say you have posts and authors and you want to display the posts with author names.
[15:39:20] <kurushiyama> ange7: If you would reference authors by something other than their name, say you use an ObjectId as _id in authors, you would need to look up the author name for each post
[15:40:55] <kurushiyama> ange7: But having something like {_id: new ObjectId(), title:"Use redundancy wisely", author:"Kurushiyama", text:"blah"}, you'd have the author name right away. Albeit the author name might be redundant here.
[15:43:54] <kurushiyama> ange7: Your use case might be different. The point is that post holds redundant data to save you one or more queries.
[15:47:14] <kurushiyama> ange7: Compare {_id: new ObjectId(), title:"Use redundancy wisely", author:authorObjectId, text:"blah"} and {_id: new ObjectId(), title:"Use redundancy wisely", author:"Kurushiyama", text:"blah"}
[15:51:59] <kurushiyama> ange7: Well, if your data model does not suit your needs, you _should_ change it. If you can not, you have to life with what you have.
[15:53:31] <silviolucenajuni> Have some diference in a simple query with db.collection.find({'a':'a', 'b':'b'}) and db.collection.find({'$and': [ {'a':'a'}, {'b':'b'} ] });
[15:54:26] <kurushiyama> silviolucenajuni: Not sure what the query optimizer would make of it, but they should be equal. albeit I find the first version much more readable.
[15:54:36] <StephenLynx> if you are figuring why $and exists, its for cases where you can't use the first syntax
[15:55:50] <Derick> StephenLynx: only a little, there are indeed cases where that trick doesn't work I believe
[15:56:23] <StephenLynx> anyway, your default syntax is the first and the second for exceptional cases.
[15:57:01] <Derick> https://docs.mongodb.com/manual/reference/operator/query/and/#and-queries-with-multiple-expressions-specifying-the-same-operator is a good example
[16:02:01] <ange7> kurushiyama: fo you think is it possible to make one update query to add author field in my collection from my author collection ? :/
[16:03:12] <kurushiyama> ange7: I do not think so. You need to do a migration. I am off, will be back in an hour.
[16:09:55] <StephenLynx> ange7, nope, you will have to fetch the data and then update the other collection on a separate query
[16:11:07] <StephenLynx> just keep in mind that it probably won't be able to handle a billion or so operations at once
[17:29:55] <cffworld> Mongo question: How can I tell if a member of a replication set is frozen or not? I can freeze with `rs.freeze()` but have no idea how to tell if it is frozen.
[17:33:31] <edrocks> cffworld: you can unfreeze it if you set to 0 https://docs.mongodb.com/v3.0/reference/command/replSetFreeze/#dbcmd.replSetFreeze
[17:34:29] <kurushiyama> cffworld: edrocks asked wether it is in rs.status(). You answered "yes". Then you said you can not tell. So IS it in rs.status or not?
[17:35:15] <cffworld> kurushiyama: ah sorry for the confusion. Yes, it is in rs.status. But there is nothing in rs.status that says anything about being frozen
[17:37:33] <kurushiyama> cffworld: "Yes, it is in rs.status. But there is nothing in rs.status" What?!?
[18:09:33] <kurushiyama> saml: in general $and is redundant except for special cases. can you show the complete query?
[18:12:48] <saml> kurushiyama, no i don't freelance. i can't market myself as competitive consultant. not raelly a good businessman
[18:18:13] <silviolucenajuni> saml: in query db.collection.find({'a': 'a', 'b':'b'}) is like an implicit $and. like db.collection.find({'$and': [{'a':'a'}, {'b':'b'}] }) is explicit $and. And in also query the filters are evaluated in short-circuit. If a document don't have field 'a' equal 'a', mongodb don't check if field 'b' is equal 'b'
[18:39:59] <saml> imagine a collection with docs like {authoredBy:'author name', tags: ['tag'], publishDate:ISODate()} and I have 20 different queries around authoredBy and tags. some use regex (authoredBy contains something ignoring case). I need to write a report about document count per each 20 query groups by month
[18:42:11] <saml> $group._id is like {year: {$year: '$publishDate'}, month: {$month: '$publishDate'}, and the query}
[18:42:35] <kurushiyama> saml: Under usual provisions, I'd probably write 20 aggregations using the original statements, then group as you described.
[18:45:10] <saml> number of documents per month by author = simple aggregation. now modify "by author" part with more involved query that business wants
[18:45:11] <kurushiyama> saml: Might well be that you can/should define reporting use cases, of which the aggregations derive rather naturally.
[18:46:25] <kurushiyama> saml: In this case, I'd really go for working out the use cases with the suits and write according aggregations from scratch.
[18:50:53] <kurushiyama> saml: Sorry that I just can deliver commonplaces, but for better answers I need more detailed info.
[19:31:29] <oky> kurushiyama: you into other dbs like influx or prometheus? i see a lot of talk about them
[19:32:31] <oky> f.e. a lot of these DBs are calling themselves 'time series DBs' - does that mean only time series queries can be made?
[19:36:20] <kurushiyama> oky: Right. Say you have tags host and region. So you would have queries that are _roughly_ comparable to db.measurement.find({host:"somehostname", date:{$gte:someDate,$lte:someOtherDate}})
[19:37:38] <kurushiyama> or db.measurement.find({region:"NCSA", date:{$gte:someDate,$lte:someOtherDate}})
[19:38:28] <silviolucenajuni> Anyone know if the rule # 27 of the book Tips and Tricks for Developer by Chorodow is still a valid rule in mongo 3.2? I'm trying to replicate the results but I can not see difference.
[19:38:52] <silviolucenajuni> Tip #27: AND-queries should match as little as possible as fast as possible
[19:40:06] <silviolucenajuni> Tip #27: AND-queries should match as little as possible as fast as possible
[19:41:45] <kurushiyama> silviolucenajuni: Out of context, I guess it is wise to follow Kristina's advice until she is proven wrong ;)
[19:42:52] <kurushiyama> oky: The interesting part of Influx (for me) is that it is rather easy group by time _intervals_, which can become a pita in other DBs
[19:44:00] <yopp> yeah, but main problem with influx that they don't have basic things like increments and stuff
[19:44:12] <kurushiyama> oky: Keeping the example above, you could do something like (pseudocode) db.measurement.find({host:"somehostname", date:{$gte:someDate,$lte:someOtherDate}}).groupBy("5 mins").average()
[19:46:11] <kurushiyama> saml: But to answer the question properly: Yes, there is sharding. Albeit this is a enterprise feature since recently.
[19:47:48] <edrocks> was trying to make dev builds faster by loading minified libraries instead of their source
[19:48:00] <yopp> hey database, show me the stats of requests by codes for last hour with 1 min resolution
[19:49:59] <saml> yopp, db.request_logs.aggregate({$match:{ts:{$lt:ISODate(1 hour ago)}}}, {$group:{_id:{$hour: '$ts', $minute: '$ts', status: '$status_code'}, count: {$sum:1}}, {$sort: ...
[19:51:04] <saml> https://docs.mongodb.com/v3.0/reference/operator/aggregation/minute/ it does exist
[19:51:48] <saml> A time series database (TSDB) is a software system that is optimized for handling time series data, arrays of numbers indexed by time (a datetime or a datetime range).
[19:53:41] <kurushiyama> yopp: How else would you store timestamps efficiently? Especially when we talk of comparisons?
[19:55:46] <yopp> saml, nope, but imagine you have a small factory and you need to store historical values of all sensors
[19:56:41] <yopp> when we speak about slow processes, like temperature that can't change like hundred times per seconds, its easy
[19:58:22] <kurushiyama> yopp: That is a problem, admitted. Though latency will kill you anyway. You'd probably need local window aggregation for electronics.
[20:01:34] <yopp> and you have a formula that allows you to use historical sensor values to predict the breakdown
[20:02:22] <saml> sure. archiving them is one thing. doing real time analysis of high throughput is another. and analysis yields manageable sized output
[20:02:58] <saml> all those events, you can archive somewhere. and you also run analysis transforming raw events into something more meaningful to you
[20:03:30] <yopp> it's not working like: okay, once a week we will put this shit in our hadoop cluster and see what it says
[20:03:36] <kurushiyama> yopp: I doubt that. exceeding stddev as a trigger would be sufficient. The amount by which stddev is exceeded allows breakdown projection.
[20:04:40] <saml> maybe not.. i don't think weather forecast requires scanning all historical values everyday to make forecast
[20:05:06] <yopp> you are analysing the process, not the "snapshot". you need to compare current state with prevoius state, with previous state trends
[20:06:02] <kurushiyama> yopp: So how do you take variance into account when only comparing two values?
[20:10:16] <yopp> and the problem was like: how we can do that no just on single site, but in our warm office to get a whole picture and to optimize the maintenance routine
[20:28:55] <cpama> as you can see, i have two functions... both update the same collection...but different fields in a document
[20:29:26] <cpama> my bug is that when I run function update_location_status it works... creates the record if it doesn't exist ... and i can it multiple time...everything looks good.
[20:29:39] <cpama> but then another module comes along and wants to add data to an existing location record.
[20:29:58] <cpama> when it calls the function update_location_data the existing document is wiped out and replaced with this new document.
[20:32:08] <cpama> i think maybe the problem is that i'm passing in an array with all the fields i want to update, but this is being interpreted as rewrite the entire doc using this new array. Really, what I want is to update the existing doc with updated values for only the fields defined in the array
[20:41:05] <cpama> i figured it out. had to pass not just the array of new data, but array('$set'=>$newdata)
[23:22:43] <jr3> is there a way to query the date inside of an objectid so I can find the number of docs made in a date range
[23:30:40] <Boomtime> https://www.kchodorow.com/blog/2011/12/20/querying-for-timestamps-using-objectids/