Log file Viewer
#mongodb logs for Thursday the 7th of January, 2016
[01:14:08] <brotatochip> hey guys, does anyone know of a script or tool that I can use to run a diff against two separate mongodb databases?
[01:15:00] <brotatochip> I’m trying to recover a replica set with as little downtime as possible without incurring the 2+ hour downtime for a full resync
[01:35:12] <brotatochip> ok cheeser let me try to explain what I’d like to accomplish and maybe you can tell me if it’s possible
[01:36:23] <brotatochip> i’m not sure what the root cause was, but somehow my secondary fell days behind my primary and then the primary entered a FATAL state, causing the secondary to be elected
[01:36:48] <brotatochip> i was not able to return the primary to a healthy replicating state, so I disabled replication to get the app back online
[01:39:16] <cheeser> also, i'd consider signing up for monitorin so you get alerts if your secondaries are falling too far behind
[01:51:03] <brotatochip> Ok looks like I found the root cause cheeser, networking issue, somehow the primary lost connectivity to the secondary which caused it to step down
[01:54:51] <brotatochip> Two secondaries meaning one of them can step up to primary in case there is a problem with the primary, right?
[01:55:33] <brotatochip> Currently my secondary is only set up to be a geographically redundant failover
[02:04:19] <cheeser> now, if you can get your primary and arbiter talking again, you'd at least be live.
[02:04:41] <morenoh149> where can I read about mongodb behaviour at high speeds? many writes per second
[02:05:33] <morenoh149> like what to do if a mongo instance can't keep up with the requested number of writes
[02:17:04] <brotatochip> I highly doubt that… AWS spends a lot of money trying to get people to use RDS
[02:23:33] <brotatochip> cheeser: so I’ve done a full resync using another server with this morning’s backup. if I stop mongodb in the secondary, point it and the arbiter back at the original primary and then restart the original primary with replication enabled, will mongo fill in the gap in the data from this morning to tomorrow (when I’ll be performing the maintenance)?
[02:23:47] <brotatochip> it would be great if I could do that without having to worry about data loss
[02:24:48] <cheeser> the secondary will attempt apply anything in the primary's oplog that has not been applied to the secondary.
[02:25:12] <cheeser> if the last sync time is farther back than what's in the oplog, it'll have to do a full resync.
[02:25:55] <cheeser> you can also do this: https://docs.mongodb.org/manual/tutorial/resync-replica-set-member/#sync-by-copying-data-files-from-another-member
[02:27:58] <cheeser> if you init that mongod as a primary, you can then add secondaries but you'll have to do a full resync at that point
[02:31:38] <brotatochip> so it’s got an oplog but that oplog hasn’t been written to, so those changes won’t be in it
[02:39:58] <brotatochip> cheeser: it sounds like I need to remove the oplog from my intended primary as it is no longer valid and would probably cause an issue
[02:42:43] <brotatochip> but there are going to be over 3 days of writes to the DB that are not in the oplog
[02:43:35] <brotatochip> the problem with that is again, i would need to stop any writes to the DB while I copied the files over to the secondary, would I not?
[02:43:50] <brotatochip> the intended primary isn’t going to start writing to an oplog until it’s a replica set member
[02:48:20] <brotatochip> so should I remove the old oplog or local files, then restart it with replSet enabled in the config?
[02:51:06] <brotatochip> no, I’m not going to touch that DB until the maintenance window tomorrow night
[02:55:27] <brotatochip> Gonna have to be tomorrow though, it’s been time to head home for about an hour :s
[03:57:02] <Doyle> Hey. If a mongos is running 2.6, and the mongod's are 3.0, will that prevent queries hitting the mongos from writing?
[10:10:31] <Zelest> atm we have 3 nodes and I'm somewhat confused when "no candidate servers" happens or not
[10:16:12] <Derick> "no candidate servers" should only happen if a. you're writing, and b. two nodes are down or c. the latency to the servers is too high
[10:17:42] <Zelest> well, if b or c happens now and two nodes are down, "no candidate servers" happens.. with 5 nodes, that won't happen?
[11:02:48] <m3t4lukas> Derick: no problem. Seems kinda long though for something that can mostly be done automated :P
[12:01:27] <m3t4lukas> Derick, cheeser: the issue is closed but there is only a sum accumulator that takes a field :/
[12:05:31] <m3t4lukas> sources: http://grepcode.com/file/repo1.maven.org/maven2/org.mongodb.morphia/morphia/1.0.1/org/mongodb/morphia/aggregation/Group.java?av=f and http://grepcode.com/file/repo1.maven.org/maven2/org.mongodb.morphia/morphia/1.0.1/org/mongodb/morphia/aggregation/Accumulator.java?av=f
[12:06:52] <m3t4lukas> Derick, cheeser: never mind. I just saw that the accumulator has two constructors
[13:41:52] <Zelest> Derick, if I plan on using gridfs.. and wish to lets say md5 every file before storing it in order to find duplicate files, what is the best approach for this?
[13:47:08] <m3t4lukas> Zelest, not if you use the default GridFS driver. I would not recommend to change that though. What happens if you save to one of these previously similar files?
[13:54:23] <cheeser> so morphia uses the legacy api in the 3.x driver and i need to update it to use the new api
[13:56:02] <m3t4lukas> cheeser: I guess it makes sense to not make it that way. From a software enginering perspective
[15:27:36] <ut2k3> Hi Guys, we want to create daily collections that are sharded. Before we were only using "insert" and the new collection was automatically created. What are the best practices to create directly sharded collections?
[15:29:07] <cheeser> you'll to explicitly create those collections or at least make the explicit call to shard it.
[16:00:59] <basiclaser> do you know how to make a schema/document in mongoDB which accepts 0 or more instances of a bunch of different key:values?
[16:01:00] <basiclaser> My feature is this: A user can create a ‘result’. It can contain 0 or more instances of 8 different datatypes (text,link,image,video,sound,file,map,entity).
[16:01:00] <basiclaser> So one result could be [text,image,video,text,text,link] and another could be [image, image]
[16:01:00] <basiclaser> in frontend i think i know how to program this, but im not sure how to define the server router schema to allow for such flexible objects
[16:06:36] <basiclaser> that’s pretty much as defined as the project is at this point. a webapp constructs a collection of [text,image,video,text,text,link] and sends to server
[16:10:46] <basiclaser> cheeser: is there a formal term for this type of problem so i can read about it further ?
[18:11:50] <jingalala> I need to apply a transformation on one particular field of all records from python. What would be the best way?
[18:16:47] <cheeser> pulling docs in to your app just to make one simple change like that is terribly inefficient.
[18:17:51] <jingalala> that is fine, I am just running some statistics project. And change is not simple, it is a set of transformations to a string
[18:18:58] <StephenLynx> do a bulkwrite, which might not be possible, if you have too many documents.
[18:44:47] <freeone3000> When primary is lost, how long should it take before a secondary is elected primary?
[20:25:04] <brotatochip> hey cheeser, so I just swapped my testing primary’s storage volume with a more recent backup of my production (intended) primary, restarted it, and it came up as secondary while my intended secondary was elected as primary - even though my testing primary has a priority of 10. I restarted the (intended) secondary, which then came back up as secondary while the testing primary stepped up to again become primary
[20:25:33] <brotatochip> Is there any way for me to know if the secondary has the incremental changes from between yesterday and today?
[20:28:09] <eiabea> cheeser, thank you, i am doing something like this to create the nested object: Event.findOneAndUpdate({_id: out._id}, {$set: {athene: athene}}, {new: true}, eventUpdated)
[20:29:00] <eiabea> the nested object (athene) gets an _id, but i can't find it with: vent.find({"athene._id": atheneId}, callback)
[20:34:45] <brotatochip> ok, so how certain can I be that all of the data from between yesterday and today made it to the secondary cheeser?
[20:35:07] <brotatochip> this is super important, the client will be pretty pissed if there is data loss
[20:42:17] <brotatochip> The next thing to try is to reinit today’s backup of the intended primary with yesterday’s data on the intended secondary, but if I can be certain there is no data loss, I’m not going to bother
[21:02:00] <brotatochip> ok, so going forward, having a second secondary in the same location as the primary is probably the best long term solution
[21:14:16] <m3t4lukas> brotatochip: as far as I remember your case is covered by the basic dba course on university.mongodb.com
[21:15:26] <m3t4lukas> brotatochip: as far as I understood your problem you don't wanna loose the data the primary was ahead. The course actually teaches manual desaster recovery in that sitch
[21:16:12] <brotatochip> m3t4lukas: I’m not sure that’s true as my primary isn’t currently a primary, as it is currenly not a replica set member
[21:16:29] <brotatochip> which means the data that I want to get into the secondary is not in the oplog
[21:17:09] <brotatochip> i still don’t think there’s any way to avoid the full resync unless I can try a re init with a seeded secondary
[21:17:47] <m3t4lukas> borotatochip: don't pin me down to it but I think there is a directory for data that has not fully been synced. I think that's what is used during manual desaster recovery
[21:18:38] <m3t4lukas> otherwise you can export the collections and use the import/export tools and meld to do it.
[21:19:18] <brotatochip> from what I’ve read a mongodump is not only not possible for this but it would also take probably 15+ hours
[21:20:01] <m3t4lukas> in that case first step would be mongoexporting each collection on the current primary and on the crashed one you took out, running those files through meld and mongoimport the diff back into your current replicaset
[21:21:07] <brotatochip> i’ve attempted a mongorestore months ago when there was like 25gb and 9gb index and it took 10 hours
[21:22:42] <brotatochip> so i would export from the primary, and then import the same data back to that primary?
[21:23:54] <m3t4lukas> cheeser: no, that is the thing, it isn't, he took the crashed one out of rs in order to prevent rollback
[21:24:41] <m3t4lukas> he wants the data that would have been rolled back imported manually to the currently running rs
[21:24:55] <cheeser> well, i've lost track of which machine has what and what state the data is in so i'll just wash my hands of this one move on
[21:26:03] <m3t4lukas> brotatochip: you could do a diagram, so we are all on the same page. If you don't know any tool use draw.io
[21:26:58] <m3t4lukas> cheeser: what I just portrayed was how I understood what brotatochip is trying to do.
[21:28:43] <m3t4lukas> brotatochip: if you did not bring the crashed one back to the cluster the first thing you should do is spin up another secondary, so you have a cluster of three
[21:29:08] <brotatochip> i did not do that, what I did was restart the former primary with replication disabled
[21:29:48] <m3t4lukas> that was right, but you should realy set up a new instance of mongo quick (!!!) and add it to the rs
[21:30:48] <m3t4lukas> what you schould also do is look for the cause why the secondary lagged behind in the first place and eliminate the cause
[21:34:04] <m3t4lukas> cheeser: ah, okay, I did not read it :P Now he's heard it from two sides. brotatochip: if he told you yesterday you should have had another mongo added to the rs right now. If money is involved what you do is kinda dangerous
[21:41:49] <brotatochip> yeah the cause of the secondary falling behind is mostly obvious as the primary and secondary are on separate coasts of the US connected via an ipsec VPN tunnel
[21:43:29] <freeone3000> m3t4lukas: "Sure, they can LOG IN to our database, but they aren't allowed to run anything".
[21:44:24] <m3t4lukas> brotatochip: you can avoid vpn lag by adding processing power (only if you don't utilize the full bandwidth)
[21:47:02] <m3t4lukas> just use the source packages and let the package manager do the compile options.
[21:50:25] <m3t4lukas> if I have a server, let's say in Munich and one in Stockholm (both in data centers) I could transfer gigabytes in minutes. It actually is something like 20 megs per second. I'm just wondering that the US has that bad of an infrastructure
[21:50:47] <freeone3000> m3t4lukas: No. It's much faster. AWS internal transfer speeds are around 200MB/s internally.
[21:52:32] <freeone3000> m3t4lukas: I don't use AWS for the API. I use AWS because I can get servers instantly. OpenStack provides the API (and not as good of one, at that), not the servers.
[21:53:32] <brotatochip> freeone3000 how would I keep the mongodb traffic internal to AWS between separate regions?
[21:53:35] <m3t4lukas> freeone3000: with the openstack web interface you can spin up vm's in seconds. clone them etc. It's king combined with puppet or chef :P
[21:54:29] <freeone3000> brotatochip: VPC-to-VPC bridge is the AWS solution. A simpler solution is to hit it by its IP. It's a routing path, it's not magical. (Well, it is magical, but of the really simple variety.)
[21:54:47] <freeone3000> m3t4lukas: Right. For colocation, I have to actually buy a server, and physically put it somewhere. I can't get, say, 20 of them, in Seoul, in 20 minutes. That's what we use AWS for.
[21:55:54] <m3t4lukas> even if I plan on 30% capacity for eventualities and quick vm spinups, I'm cheaper off using openstack
[21:56:33] <freeone3000> m3t4lukas: You have to plan. our idea is that we plan less, so that we can save money by doing more things less efficienctly.
[21:57:30] <freeone3000> brotatochip: Well, if they're connected by their public IPs just do that. Shoot. ANd yes, it will go "outside" AWS, but it takes the AWS-to-AWS routing path because that's the most efficient.
[21:59:27] <m3t4lukas> brotatochip: what OS do you use? I know the process of automating building from sources can be automated pretty easily on RedHat systems.
[22:00:03] <freeone3000> brotatochip: Okay. So if your solution to that is going to be a VPN tunnel, you designate a left, a right, and then have it route. Again, it'll take the proper path.
[22:00:42] <freeone3000> It's not fedora because it doesn't have the fedora core enhancements and follows the release schedule. It's centos.
[22:02:03] <m3t4lukas> whoa, why fedora? fedora is desktop... Do you have X11 running on those machines? That might cause instabilities. Especially concerning ram usage over time.
[22:03:33] <brotatochip> but yeah, I could totally use FPM or YUM to automate the building of mongodb, however, I went with what I felt was the safer option at the time
[22:04:07] <brotatochip> Yeah but CentOS is so far behind package version wise, it’s really hard to develop software on it and maintain your dependencies
[22:05:57] <brotatochip> Another reason I’m using Amazon Linux is that before I spun up any servers at my current job I was a complete AWS noob, and AWS supposedly supports it
[22:06:48] <m3t4lukas> if you think CentOS is far behind package version wise you should try debian :D Try to install non omnibus gitlab on debian :D :D :D
[22:07:23] <brotatochip> I love Debian but if I’m going with that family of OS it’s probably gonna be Ubuntu
[22:09:17] <m3t4lukas> never tried gentoo :P I came from debian over to ubuntu (because I was annoyed from outdated stuff), once tried RHEL and stuck with it because of the freedesktop system.
[22:09:54] <brotatochip> you should build gentoo for fun m3t4lukas, roll the kernel and everything hehe
[22:10:23] <m3t4lukas> brotatochip: sounds like fun, but I imagine it to be time consuming without generating income :P
[22:12:12] <brotatochip> a lot of people run Ubuntu in prod, I’m not sure if they are running MongoDB on it but it’s a popular server OS
[22:13:35] <m3t4lukas> brotatochip: I like arch for testing how far they are on wayland. I'm really waiting for wayland to be shipped with the regular distros
[22:16:07] <m3t4lukas> I hate that ubuntu started mir. It's like an act of active community splitting. They should stay in bed with amazon and leave the community alone :D
[22:18:59] <m3t4lukas> well I never experienced a crash on a stable release. That's nothing I can't say about X11
[22:19:08] <brotatochip> ugh I can remember having to make manual changes to xorg.conf years and years ago, when I was just a linux noob
[22:20:11] <m3t4lukas> brotatochip: still have to do it. Actually after over a year nvidia still has not gotten to utilize dual GPU notebooks GPUs in an efficient manner
[22:25:20] <m3t4lukas> so I don't know that it is more fun. Only thing I know is that Windows is a pain in the butt
[22:26:07] <Muimi> i'll probably be doing almost everything on my server, itself, but I wish I had a local testing environment for offline stuff. I might uninstall a bunch of stuff and reinstall it in a vb.
[22:27:03] <m3t4lukas> Muimi: maybe you can free up some gigs if you use explorer search for "log" and delete all log files on our machine. Might even make it faster
[23:27:54] <brotatochip> hey cheeser, in this doc https://docs.mongodb.org/manual/core/backups/#backup-with-file-copies in the section under “Backup by Copying …” where it mentions snapshots and journaling, the journal it is referring to is the filesystem journal, correct?
[23:34:45] <ThisIsDog> I'm trying to find the intersection of two queries. I tried using $and, but its not working as I expected. Does anyone see what I'm doing wrong? https://dpaste.de/TyqA
[23:42:48] <ThisIsDog> The query from my example, but I'm afraid of a case where a complex query is sent and it explicity defines isActive False. I'm worried I can't just set isActive without changing its meaning.
[23:44:17] <ThisIsDog> I'd want to make sure I return nothing, instead of assuming what they wanted.
[23:46:11] <joannac> what does cls.query.find({$and: [{"isActive": True}, {"name": "Dog"}]}).all() return ?
[23:49:20] <ThisIsDog> I'd expect it be the same as cls.query.find({"isActive": True, "name": "Dog"}).all(), the problem I'm having is it is not
[23:49:58] <joannac> {u'_id': ObjectId('568ef7b8dfc3b7931bb4c255'), u'name': u'Dog', u'isActive': True}
[23:50:51] <joannac> if you can reproduce it with the python driver, then let me know because that would be a bug