Log file Viewer
#mongodb logs for Monday the 25th of April, 2016
[04:02:00] <Mrgoose> Hey quick question, I have a bunch of sensor data thats collected every 3 seconds. WIth mongodb could i create aggregate queries that would show the average every minute?
[04:07:21] <Ameo> Mrgoose: you can just select all data from the past minute, add up the results, and divide by the number of results returned :p
[05:40:39] <Mrgoose> im currently watching https://www.mongodb.com/presentations/mongodb-time-series-data
[05:42:33] <kurushiyama> Mrgoose: Well, good entry point. Do not get me wrong: MongoDB might well fit your use case, but if you have the time, I'd probably checl InfluxDB, too.
[05:44:11] <kurushiyama> Mrgoose: It has some advantages and disadvantages compared to MongoDB, ofc.
[05:47:23] <kurushiyama> Mrgoose: From a very high level perspective, it is less flexible and very much less general purpose. The whole TICK stack combined, however, makes it very easy to get results. So you really have to decide. You can not use InfluxDB for anything else, basically.
[05:48:31] <Mrgoose> just from what im looking at a quick glance , influxdb seems to be easier to get that done
[05:55:57] <kurushiyama> Mrgoose: The querying? Not so much,imho. Personally, I find the aggregations easier, but then, I use MongoDB for much longer.
[05:57:07] <Mrgoose> > SELECT MEAN(water_level) FROM h2o_feet WHERE time >= '2015-08-18T00:00:00Z' AND time < '2015-09-18T17:00:00Z' GROUP BY time(4d
[06:03:47] <kurushiyama> A major drawback I see is that you almost always need another database for well, everything else like authentication and whatnot. I code in Go, so there is always boltdb embeddable, but it is a pita. With MongoDB, you only have one persistence technology.
[09:11:59] <Ruphin> Anyone here? I have an issue with a node failing to rejoin a replica set. More specifically, the HostnameCanonicalizationWorker fails to resolve the hostname in the replica set config so it does not recognise itself as a member
[12:43:08] <Onemorenickname> If I want to do some thing with the result of a .find in mongodb, I do db.collection.find(selector, cb)
[12:46:53] <Onemorenickname> there is the syntax db.Post.find({}, {}, {limit:10}, cb), but I'd like db.Post.find({}).limit(nb)
[13:13:05] <kurushiyama> Onemorenickname: It depends. First, you should probably state what stack you are using instead of making us guess.
[13:15:00] <tembrae> Hello, I have a question. I have a mongo db with replication enabled and 2 full disks. I am wondering how to reduce the amount of storage used as a temporary solution untill we can upgrade disk space
[13:23:55] <kurushiyama> tembrae: the other option would be to wipe dbpath , install the new hard disk, setup LVM and do a resync. You should only do this, however, when you are sure your current primary holds.
[13:24:07] <tembrae> kurushiyama: yes I have read about the repair and compact options but they do not seem like viable stategies
[13:25:33] <kurushiyama> tembrae: Well, another option of course would be to build an LVM besides it, copy the data over and use it
[13:29:48] <tembrae> kurushiyama: If you add a second server (with double the disk space) and add that as a replication set, how will this affect the existing replication?
[13:35:28] <kurushiyama> Since most likely your nodes are down because of the full disks, most likely not at all.
[13:44:45] <Ange7> i try to find how to insert document if not _id exists else ignore is it possible ?
[13:46:43] <kurushiyama> tembrae: Ok, give the new server the IP of one of the old ones, CREATE AN LVM ;), copy over the data and config, fire up an MongoDB.
[13:52:16] <yopp> kurushiyama, re last time, I've played with value keys, and everything is just fine, but I'm trying to follow the array approach as you advised.
[13:52:41] <yopp> But main roadblock is that $addToSet adds new element, because timestamps of events are not the same
[13:57:34] <kurushiyama> yopp: As said: you must do as you are pleased, but how an index of a fraction of the data should be bigger than the data itself eludes my comprehension, and I am not exactly a beginner.
[14:02:20] <yopp> kurushiyama, when you have small records, like around 64 bytes, even with primary index on _id you will have overhead of _id + reference to the page, which is already something like 24 bytes. Add timestamp index, and here you are : your data set is smaller than index size
[14:04:45] <kurushiyama> yopp: And objectIds, should you choose to use them, are internally stored as 12 bytes.
[14:07:39] <yopp> you are not just storing ids, you are storing the tree of references to the storage
[14:10:22] <kurushiyama> yopp: Well... what? Ok, you made up your mind, be it. But modelling data for constraints other then CRUD efficiency will impact said efficiency. And working against best practises (have a look at how dedicated TS databases handle it) should tell you something. But, be it, do as you please.
[14:15:55] <Ange7> i try to find how to insert document if not _id exists else ignore is it possible ?
[14:19:30] <jokke> i want to setup a sharded cluster with docker and i'd need to be able to call rs.initiate and sh.addShard in a way that it's run only once
[14:21:36] <kurushiyama> jokke: I would not do that. Use a docker container for MongoDB. then fire up the cluster via shell. You can script that, of course, but thing can too easily go south that way.
[14:22:33] <yopp> cheeser, say I have a document: {_id: oid, events: [{_id: 1, a: DateTime, b: DateTime} ] }. Is there any way to tell $addToSet to check for just "events._id" for uniqueness? Say I'm doing "$addToSet": { _id: 1, a: { DateTime + 1} } .
[14:22:41] <kurushiyama> Ange7: What is wrong with https://docs.mongodb.org/manual/reference/operator/update/setOnInsert/ ?
[14:27:33] <kurushiyama> jokke: And usually, MongoDB is not as volatile. It might be worth if you compose services (which you rarely do, since more often than not, you do not want to have much more than MongoDB running on a production server)
[15:08:59] <varunkvv> I ran into a crazy issue over the weekend. One of our applications was performing a query that hit the caveat: https://docs.mongodb.org/manual/core/index-intersection/#index-intersection-and-sort - this was on a large resulting dataset and the application side kept retrying after a timeout. This actually caused the mongodb server to crash with an "out of memory" error. Kind of a first for me.
[15:32:14] <jokke> i'm getting the error "database db_evaluation not found" when trying to enable sharding for the db
[15:42:04] <kurushiyama> jokke: How about adding an index for your shard key, which you are going to need either way?
[15:44:14] <kurushiyama> jokke: You _definetly_ do not want _id as a shard key, assuming you use ObjectId...
[15:51:14] <Derick> StephenLynx: no, the _id would often just mean you'll end up writing to one shard only
[15:56:01] <StephenLynx> ok, so from what I read https://docs.mongodb.org/manual/core/sharding-shard-key/ _id is good IF you use it hashed for sharding?
[15:56:52] <kurushiyama> StephenLynx: The problem is that you basically distribute equally across the shards and have little to no control.
[15:57:18] <kurushiyama> StephenLynx: Think of location based sharding, and you want all EMEA customers on the EMEA shard and so on.
[15:57:48] <kurushiyama> StephenLynx: I have found it to be slightly less performant because of the transparent hashing.
[16:02:36] <kurushiyama> jokke: It is. Performance optimization and finding performance bottlenecks is... ...tricky, to say the least.
[16:17:13] <kurushiyama> Derick: The wire proto has changed, iirc. Was that between 2.6 and 3.0 or 3.0 and 3.2?
[16:44:24] <varunkvv> kurushiyama, replying to this: http://irclogger.com/.mongodb/2016-04-25#1461596482, we're _not_ using swap on the servers, but is that a mistake for WiredTiger? I know it was not a requirement for mmapv1.
[16:45:00] <kurushiyama> varunkvv: It always was a requirement to prevent Linux OOM-Killer from kicking in
[16:50:58] <varunkvv> Yeah, it's something to consider and try out. This is the first time I've had OOM killer hit mongo though :). Thanks for the pointer.
[18:30:36] <ams_> kurushiyama: I'm trying to create a replicaset, but mongodb isn't happy (I think!) that the hostname I'm giving it doesn't resolve to 127.0.0.1
[18:31:21] <kurushiyama> ams_: And how should the other servers reach somthing that actually resolves to 127.0.0.1? ;)
[18:32:17] <kurushiyama> ams_: If all replica set members get the config that they should contact the other members on 127.0.0.1, what would happen?
[18:33:06] <kurushiyama> ams_: To put it another way: do not use the loopback interface to create a replica set, unless it is local only.
[18:34:25] <ams_> kurushiyama: I think you're misunderstanding me. I understand what a 127.0.0.1 is, I'm not trying to get different hosts to access different 127.0.0.1s
[18:36:33] <ams_> Derick: maybe I've misunderstood then. What causes that error ^ though? I don't have an interface on this server bound to 191.168.0.10. But traffic for that IP will be routed to that server.
[18:36:47] <ams_> (The reason for the contrived network set up is i'm trying to get this working in kubernetes/docker)
[18:41:25] <saml> so, you have a single machine, mongodba.infra.svc.cluster.local and want to set up replacaset with that one member?
[18:45:57] <Ryzzan> i want to $push to an array inside an array... let's say... {_id : Number, array : [{someProperty : String, anotherArray:[]}]}
[18:46:55] <saml> so you have replicaset already configured. maybe unconfigure replicaset (i don't know how to do this). and reconfigure with the hostname, not localhost
[18:47:22] <ams_> sam1: yeah i've tried that, same error. I don't see why that should have any impact?
[18:50:46] <Ryzzan> saml: ty... think i did it already, but let me check if I misstyped something... ;)
[18:59:55] <Ryzzan> saml: the _id would be enough to find the place to push the value... cause "anotherArray" is directly connected to "array"... got it?
[19:00:46] <Ryzzan> I need to find by array... i have an unique id in array, how would i filter it to get to anotherArray?
[19:01:37] <saml> looks like your document model might not be optimal for kinds of updates you want to perform
[19:04:00] <saml> it's helpful to use test database, insert simple documents that represents your problem and try different updates
[19:05:22] <Ryzzan> i thought about db.collection.find({_id : 1, "array.uniqueId" : 1}).update({$push:{'array.anotherArray': THE VALUE}})
[19:08:59] <saml> i thought it's db.collection.update({_id : 1, "array.uniqueId" : 1}, {$push:{'array.anotherArray': THE VALUE}})
[20:35:20] <WiuEmPe> i need to restore mongodb from backup. I dont know how this backup has been made, i have in tar files: metadata.json and bson
[20:56:31] <kurushiyama> WiuEmPe: It does not sound like a good idea to do a restore as the first task with MongoDB. Feel free to ask if in _any_ doubt.
[21:00:39] <kurushiyama> WiuEmPe: Well, StephenLynx helped you, I am just offering to help in case you need further assistance.
[21:05:20] <assert> All the user credentials (no matter which database) are stored in the admin db, if you backup the admin db, you backup all users
[21:13:23] <kurushiyama> assert: Learning something new. Always, on any topic, every day. Now I am _really_ puzzled what the authSource / authenticationDatabase params are there for...
[21:18:24] <WiuEmPe> I like this... When noob ask for simple things, and hard users begining dispute ;)
[21:21:00] <kurushiyama> WiuEmPe: I have to admit that I only use authentication, and most of the time I simply stored system wide users in "admin", whereas I stored per DB users in the respective DB. Since I do my backups with LVM snapshots, I never noticed or had to know the exact position. However, for people doing backups with mongodump, this might make a huge difference.
[21:22:52] <WiuEmPe> kurushiyama, previous admin doing only backup data, dont thinks about system, LVM (what is lvm?!) and VM... And after crash both SSD we must setup system from nothing...
[21:24:34] <WiuEmPe> kurushiyama, yes on servers setup by prevous admin i dont found any LVM, so i thinks that he dont hear about LVMs ;)
[21:26:04] <kurushiyama> WiuEmPe: LVM snapshots are suggested, and it does not make much difference for me. I guess this is because an LVM snapshot freezes the blockdev automagically.
[21:28:35] <kurushiyama> WiuEmPe: Actually, I am not too sure about this. As per the xfs mailing list, xfsdump is not atomic.
[22:12:40] <GothAlice> Veeeeeeery interesting. Just had a pymongo app which lost all pool connections due to socket timeout go "full Balmer" with 140% CPU use doing nothing.