Log file Viewer
#mongodb logs for Thursday the 4th of June, 2015
[00:07:40] <Boomtime> brotatochip: then it sounds like the right configuration for the use case you want addressed
[00:10:54] <brotatochip> unfortunately I can't seem to get a working replica set period. No matter what kind of configuration I use, the initial sync fails
[00:10:56] <brotatochip> replSet initial sync exception: 13106 nextSafe(): { $err: "assertion src/mongo/db/storage/extent_manager.cpp:109" } 4 attempts remaining
[01:13:52] <brotatochip> joannac there is no load at all as I'm using a test environment, and the /var/lib/mongo dir in the primary is 3.6gb
[01:14:43] <brotatochip> here is a pastebin of my config and log, but the log is the same regardless of the config settings, and logs the same thing any time I try to add a secondary http://pastebin.com/p7T4zZ5F
[02:39:12] <ctcg> Apparently you cannot modify multiple array elements in a single update operation, what would be the most efficient way to do so then?
[02:40:29] <ctcg> I would like to be able to update all matching elements in an array in case there are duplicated skus
[04:44:35] <ev4l> Is it possible to expire a single filed of a schema at a certain date / time? I see it is possible to expire the netire schema
[07:28:21] <techie28> just removed the .lock file manually and the database started again and I am able to connect to it
[09:02:18] <jelle> what's the best way to use the pymongo with flask, for example now I create a mongodb connection every time a request comes in, but might it be more performant / better to just keep it open?
[10:26:24] <Naeblis> Hi. I upgraded to latest mongoose and am getting mongoose.mongo.BSONPure as undefined.
[13:29:37] <miwa> as I understand it, to provide document-level access control I need to implement Field Level Redaction (basically redacting the whole document I assume). How does this apply when using CRUD-methods rather than aggregate?
[13:59:03] <snuggles> is this still valid? http://www.sarahmei.com/blog/2013/11/11/why-you-should-never-use-mongodb/
[14:00:44] <snuggles> ok, so i can have one document relate to others? and query that structure? awesome
[14:02:31] <snuggles> i have products and customers, and these are stored separately, yet i need to related customers to products so i can figure out which customers have purchased a given product, and which products a given customer has purchased
[14:02:37] <StephenLynx> so even if you use dbref, the driver will just perform multiple queries to "join" it.
[14:03:22] <StephenLynx> you either query just for that fake relation or you perform multiple queries.
[14:04:32] <StephenLynx> and if the customerId changes, you will have to update everywhere you related it.
[14:05:10] <snuggles> i'm talking about relating one document to another. You know, because, like, that's a basic need.
[14:05:38] <snuggles> StephenLynx: trying to determine if mongodb is useful for non-trivial stuff. Seems like it's not.
[14:06:17] <cheeser> he's a known agitator from other channels. i'm unsurprised he's unwilling/unable to consider larger concepts.
[14:06:25] <StephenLynx> I wonder if he will eventually find out that "trivial" is a very BROAD term.
[14:09:22] <snuggles> oh forgot to ask, is this still valid? https://aphyr.com/posts/284-call-me-maybe-mongodb
[14:11:07] <StephenLynx> and it actually suffered from corrupting lots of data. Don't got any sources though.
[14:11:40] <StephenLynx> I have been around this channel almost every day and haven't heard once about people having this issue.
[14:12:50] <StephenLynx> plus I wouldn't mind much a writer that tries so hard to be funny with images like in that first link.
[15:02:23] <Boomtime> there was a change from 2.4 to 2.6 from 123 to 120 bytes, that is the only one i know of
[15:04:06] <eba> Boomtime it is written as The maximum size of a collection name is 128 characters, including the name of the database. However, for maximum flexibility, collections should have names less than 80 characters.
[15:06:26] <Boomtime> anyway, i'm pretty sure that FAQ is wrong.. or at least, something is wrong because these docs references appear to be in conflict
[16:10:38] <__mph__> Hey guys quick question about a query. Is there a way to match where a subdocument field is ONLY EQUAL to a certain value?
[16:11:09] <__mph__> my db is threads. and I want to find where messages.author_id is ONLY equal to a certain value
[16:12:52] <__mph__> StephenLynx: messages.author_id: value will still return thread documents where message id is another value
[16:14:20] <deathanchor> or you only want threads that only has that author_id in the subdocs and no other?
[16:16:17] <deathanchor> StephenLynx: I don't know if there is a way to eliminate matchs where another message.author_id is there
[16:18:49] <redsand> 1. why can i use partial(true) on find cursor, but there is no equivilent to the aggregaion framework
[16:19:04] <__mph__> { '$unwind': '$messages' }, { '$unwind': '$recipient_ids' }, { '$match': { 'messages.author_id': membership.id } },
[16:21:06] <brotatochip> just an FYI, this documentation does not necessarily work: http://docs.mongodb.org/v2.6/tutorial/convert-standalone-to-replica-set/
[16:21:35] <__mph__> The unwinding and matching still returns documetns where the subdocument matches other cases :/
[16:22:01] <brotatochip> The only way to convert my standalone instance to a replica set that worked was to dump the db, wipe /var/lib/mongo, restart mongodb as a replica set, add the members, then mongorestore.
[16:22:10] <redsand> you can try pushing those elems that match into its own array maybe? with elemmatch and $push ?
[16:22:42] <__mph__> redsand: I want the whole document.. I just don't want the documents that match other cases
[16:23:01] <__mph__> Basically, I'm looking for the case where the current person I'm looking at sent a message and it was never ever responded to
[16:43:10] <__mph__> can I match on a case based off of values in the document? (i.e. like $eq: 1, $nin: '$recipient_ids)
[16:54:03] <brotatochip> is anyone available to help explain to me the best way to set up a failover for disaster recovery? i tried following this guide http://docs.mongodb.org/v2.6/tutorial/deploy-geographically-distributed-replica-set/ but when both the primary and the arbiter become unreachable, the secondary does not elect itself a primary.
[16:54:54] <brotatochip> what worked was spinning the arbiter back up, however, that document recommends that you keep the arbiter in the same DC as the primary, so if SHTF in that main DC, there is no failover...
[16:58:27] <deathanchor> and are you also talking about machine failure, DC failure or nuclear winter failover?
[16:58:28] <brotatochip> any of the above deathanchor, none work when following that guide with a 3 member set - 1 primary, 1 secondary, and 1 arbiter
[17:00:00] <brotatochip> i have redundant hardware in both sites, i will configure the DR environment to read/write from the secondary, spin them up in the event of a disaster
[17:01:01] <brotatochip> no, my app shouldn't, as my main production app is on the east coast and my DR site is on the west coast
[17:01:13] <deathanchor> assuming primary secondary and arbiter has 1 vote, you need two to communicate so that you have quorum
[17:02:03] <deathanchor> no, because if it ever loses communication with your primary, your primary will stepDown
[17:04:05] <brotatochip> right, my boss has tasked me with setting up geographical redundancy, so I'm just trying to find the best way to keep the DBs in sync so we lose as few writes as possible
[17:05:05] <brotatochip> well i tried master slave replication using a replica set, and I was unable to force the secondary to become primary
[17:06:08] <brotatochip> do you think I should try to use the legacy master/slave replication deathanchor ?
[17:06:14] <deathanchor> you can do secondary+arbiter in one DC, and primary in another. Then set priority to primary higher
[17:06:41] <brotatochip> yeah but like you said if the primary loses contact with the arbiter, won't it step down?
[17:06:50] <deathanchor> but know that any drop of network between DC will cause eletion and the secondary would take over
[17:07:23] <brotatochip> the secondary stepping up won't cause any headache, I can force it to step back down manually if needed, i just can't have the primary stepping down
[17:09:22] <brotatochip> right, so do you know of any configuration that would give me any kind of failover while keeping 2 members in sync deathanchor, without the risk of the primary stepping down?
[17:09:45] <jigax> hi, anyone know if its possible to use $set and $push in the same query? i'm currently getting an error
[17:12:11] <Boomtime> @jigax: yes, you can use $push and $set in the same *update* (i assume this is what you meant), but they must refer to different fields
[17:13:27] <brotatochip> so looks like my only option is to try legacy master/slave replication, or have a 3 member set with 2 arbiters
[17:15:20] <brotatochip> i'm just thinking maybe a 5 member set - 2 in the main DC with 1 arbiter, 1 in the DR site with the other arbiter - should prevent the primary from stepping down if my VPN goes down
[17:16:53] <Boomtime> if you have 5 members, where 2 are arbiters, and you remove the 2 arbiters, then you still have an odd number of votes
[17:16:53] <jigax> @Boomtime please check this jsbin http://jsbin.com/hepixoqazi/1/edit?html,css,js,output
[17:18:50] <Boomtime> @brotatochip: if you have only 2 DCs then one of them is critical, it is logistically impossible to have a majority in both places
[17:19:21] <Boomtime> the usual recommendation is to have 2 data-bearing DCs, and one arbiter (light-weight) DC to break the tie
[17:19:23] <brotatochip> http://docs.mongodb.org/v2.6/tutorial/deploy-geographically-distributed-replica-set/ so what's the point of this then?
[17:20:18] <deathanchor> brotatochip: the second DC has a replicamember with priority:0 it will never win.
[17:20:37] <Boomtime> @brotatochip: you are conflating disaster recovery with availability - with 2 DCs you get data protection, but you might not have a primary, you might not have availability
[17:21:46] <Boomtime> if you want to maintain availability in the face of any fault, then you need 3 DCs
[17:21:48] <brotatochip> how exactly does it provide disaster recovery if you can't elect a new primary?
[17:22:18] <jigax> @Boomtime also see undefined in the console but not sure what is undefine. because when i comment either line query works just fine
[17:22:29] <vinayus> Failed: error getting collections for database `nCloudR`: error running `listCollections`. Database: `nCloudR` Err: WSARecv tcp 127.0.0.1:49356: i/o timeout
[17:23:14] <Boomtime> @brotatochip: reconfig can be forced at any member, but it isn't recommended since that would cause that member to go rogue
[17:23:20] <brotatochip> so that command will work on a secondary, regardless of priority or voting deathanchor ?
[17:23:30] <MacWinner> do object id's need to be globally unique? eg, user1 = { _id: 1, name: 'joe', bikes: [{_id: 1, type: 'mountain'}, {_id: 2, type: 'racing'}]} i'm migrating from mysql and some of my tables have same ids..
[17:24:34] <brotatochip> like, could I just do another reconfigure and fix it? or do I need to rebuild the entire replica set deathanchor ?
[17:25:00] <deathanchor> let's use the setup on the webpage. if DC1 got nuked, you only have DC2, you make that member a single member replicaSet and spin up new machines to become secondaries.
[17:25:03] <Boomtime> @brotatochip: can you please explain what you think mongodb should do in the case of 2 DCs where either one is lost?
[17:28:05] <brotatochip> Boomtime I've explained my use case I think, but let me explain further - I have 2 DCs, one on the east coast, one on the west coast, the east coast DC hosts production with a standalone mongodb instance. I simply need any kind of failover so that if the east coast DC goes down, my DR site can take its place.
[17:31:34] <Boomtime> @brotatochip: impossible with only 2 DCs, how can the DR site know that it isn't simply a connectivity problem? i.e that east coast is actually operationally fine
[17:32:24] <Boomtime> the only physically possible way the two sites can determine which one is actually at fault, is by having an arbiter break the tie and vote for the one it can see
[17:33:21] <brotatochip> Boomtime: what about a manual election to primary in the event of a DC failure?
[17:33:22] <Boomtime> if you host that arbiter in one of the 2 DCs, then that DC becomes critical, the other one can never determine the difference between a connectivity problem and a meteorite
[17:34:11] <Boomtime> if you reconfigure the replica-set, then you have broken the symmentry - this can be done safely if you are batman
[17:34:51] <brotatochip> what if I have 2 arbiters - one in the prod site, one in the DR site, keep the DR arbiter offline until SHTF, spin it up, will it elect the secondary to become primary if it can't reach the primary?
[17:35:49] <Boomtime> it is mathematically impossible to divide a number by 2 and have more than a half in each of the two parts
[17:35:51] <brotatochip> no actually I'm thinking a 4 member set -1 primary, 1 secondary, 2 arbiters, but keep the second arbiter offline until I actually need it
[17:36:27] <Boomtime> if the other members know about it, they'll want it around, and expect it to vote
[17:37:15] <brotatochip> there will be 4 members in the set, one will just be offline until its vote is needed as the main DC goes down
[17:37:36] <Boomtime> if you have 4 members in the set, then you lose majority no matter which DC goes down
[17:40:04] <brotatochip> ok, how about in this 4 member set, the DR secondary is non voting, and I keep both arbiters up all the time?
[17:41:40] <Boomtime> it is mathematically impossible to divide a number by 2 and have more than a half in each of the two parts
[17:42:11] <Boomtime> you cannot achieve write availability with only 2 DCs under all possible fault conditions
[17:44:11] <MeStesso> Hi! I’m looking at setting up a local (in-house) MongoDB instance to replicate the contents of the database on EC2, for backup purposes. Our budget is very limited so I can’t get VPN equipment… We’re also behind a firewall so we can’t accept connections from outside; however, we do have a static ip. I’ve been reading all the docs and I assume that I can’t do a replica set (with an hidden slave), but I can do master-slave
[17:45:07] <brotatochip> oh actually your use case isn't so bad, sorry, we've been discussing mongodb replication in here for some time
[17:45:46] <Boomtime> replication across 2 DCs works just dandy, and no matter what happens to one DC your data will be safe in the other one
[17:46:03] <MeStesso> cheeser: some customers of ours are companies with some requirements. they authorized us to keep data in EC2 but not in MMS
[17:46:08] <Boomtime> and when you get the second DC rebuilt, or decide you don;t need it anymore, you can use the data in the second location
[17:48:10] <cheeser> does it require master/slave? because there's almost 0 reason to use that today.
[17:48:25] <brotatochip> cheeser I have 2 DCs, one on the east coast, one on the west coast, the east coast DC hosts production with a standalone mongodb instance. I simply need any kind of failover so that if the east coast DC goes down, my DR site can take its place.
[17:50:20] <brotatochip> again, i've been through this discussion with Boomtime, the maths don't work
[17:50:44] <cheeser> though Boomtime(?) is right that you'd need a 3rd location to ensure quorum voting
[17:50:45] <MeStesso> so can anyone answer my question. i don’t think it’s hard… do you think I can do what I’m planning to do? use master/slave for backing up, as I can’t use replSet because the office is behind a firewall?
[17:51:06] <Boomtime> that is only to assure a primary - if all you want to maintain is read capability then you get that already
[17:51:48] <MeStesso> deathanchor: but I can’t use hidden slaves (replset) if I’m behind a firewall… because they can’t communicate with each other. correct?
[17:52:22] <brotatochip> Boomtime if there are docs or guides or anything you could point me to to learn how to execute safe reconfigs, I'm all ears
[17:53:20] <deathanchor> set a replSet of { primary : votes 1 and hidden secondary votes 0 if anything}
[17:54:16] <MeStesso> that’s why i’m asking about master/slave. it’s my understanding only the slave needs to connect to the master, but not vice versa. am i correct?
[17:55:26] <MeStesso> brotatochip: if I had, i’d tell you :) I can control the firewall behind the master, but not behind the slave
[17:55:46] <brotatochip> damn, yeah, I don't think you can do anything there without a two way communication
[17:55:50] <MeStesso> brotatochip: we’re renting office space in a coworking place, so we share the network. that’s the best I can do
[17:55:54] <Boomtime> @brotatochip: http://docs.mongodb.org/manual/tutorial/reconfigure-replica-set-with-unavailable-members/
[17:56:39] <MeStesso> brotatochip: need new networking equipment and ~$40/month for Amazon VPC. boss says no
[17:57:09] <brotatochip> Thanks Boomtime ! also thanks deathanchor and cheeser and Boomtime again for explaining these things to me. You've really helped my understanding of MongoDB and replication and now I at least know what my options are, however limited
[17:57:18] <Boomtime> @brotatochip: you should consider forced reconfigruation to be dynokryptonite to be used in anger only if you are covered in petrol near open flames
[17:57:49] <brotatochip> right, because how would I get all the new writes back to the original primary after the fact
[17:58:42] <cheeser> new writes would sync back to the now secondaries in the original DC. when that's done, you can step down the DR primary
[18:00:03] <Boomtime> btw, if the original DC comes back, you must kill it fast unless it can communicate with the restore DC - otherwise the original DC wil believe it can see a majority of the original members
[18:00:48] <Boomtime> this is the danger of force reconfig - you should test this, make sure you understand the permutations, understanding is key to making that situation work
[18:00:49] <brotatochip> it's actually ok if that happens Boomtime because then I can just rs.stepDown that primary as no traffic will be hitting it until I reconfigure route 53
[18:01:47] <Boomtime> if the original DC votes in it's own primary, with you forcing another one to exist in the remote location, the replica-set will have diverged and cannot be reconciled easily
[18:03:03] <brotatochip> ok, the thing is, i think i'll just have to kill the VPN so that when the original primary comes back up, it will just pick up where it thinks it left off
[18:12:20] <MacWinner> hi, I'm using the latest versions of mongo 3.0.3 and mongoose 4.0.4. I have a simple model method to save some usersettings. http://pastebin.com/hdvwYFnR Anyone know why this hangs when I call self.save() ? my OS shows 100% cpu usage and the save never seems to return. I'm using iojs + promises as well
[18:13:04] <svm_invictvs> I have a silly question, is there a reason why the Java grid FS drivers can't just seek to any point in a file and open an input stream from there?
[18:21:29] <brotatochip> cheeser I can't go into read only mode during the downtime, mongodb is the back end of our social platform with over a million users
[18:21:49] <deathanchor> Boomtime: i do worse than force reconfig, I drop local dbs and set up new shardsets.
[18:23:06] <brotatochip> our client would be pretty pissed if their customers were unable to "like" each others posts :-P
[18:27:58] <MacWinner> StephenLynx, do you use something else for ODM? particularly interested in simple hooks into update/save/delete as well as having custom methods
[18:28:00] <StephenLynx> brotatochip hm, using mongo for a social platform seems like a bad decision, if you ask me. but I know that is unrelated to your issue.
[18:28:57] <MacWinner> StephenLynx, do you create your own custom object? that makes the mongo calls? any pointers to some example projects..
[18:30:07] <StephenLynx> lynxchan and bck_lynxhub uses mongo. io-boot is some sort of minimalist framework for an issue I had at work.
[18:31:29] <brotatochip> StephenLynx I've read the rant from the webapp programmer at Diaspora if that's what you are referring to, unless you have another reason that MongoDB is not a good backend for a social app
[18:33:49] <StephenLynx> that is why, from what I have seen so far, mongo is not a very good choice for general social applications.
[18:34:17] <brotatochip> no but it's a good point of discussion and I thank you for bringing it up StephenLynx
[18:35:39] <StephenLynx> I read that as soon as your data is larger than your RAM, it becomes really bad.
[18:36:06] <StephenLynx> I would personally just use a relational DB for that case and a cache for whatever needs a cache.
[18:36:49] <brotatochip> I'm not sure if that's true in our case as mongodb data dir is 32gb, the machine has 30gb ram and there is 3.2gb ram free StephenLynx
[18:39:00] <MacWinner> so if you have a document retreived from mongo, and you need custom methods like "sayHello()" or whatever other business logic you have, what's the best way to do this? I'm coming from mysql+php way of doing things, so I'm guessing I'm just htinking about it wrong with mongo+nodejs.
[18:41:25] <MacWinner> so is the pattern to apply some sort of decorator to the returned objects from mongo? or set the prototype of the returned object to something with all yoru custom methods you need?
[18:45:01] <MacWinner> StephenLynx, i'm looking at this project calling mongorito.. seems interesting too
[18:46:15] <StephenLynx> any project that starts with "awesome" in its description and says it is failing its tests goes straight to my trash bin.
[18:49:47] <StephenLynx> brotatochip funny thing you guys have that product. I released something similar a couple of months ago.
[19:56:52] <_ari> deathanchor: i'm outputing my aggregation and looping over the aggreagation to update
[20:39:18] <Iniglo> Hey guys, I'm trying to create a blog with Mongo as the database but it won't allow me to store blog post content in the database. I keep getting an ILLEGAL error because of the newlines inside the content entries. Anyone know how to work around that?
[20:40:24] <Iniglo> Yeah you're right I'm calling an insert using meteor, sorry that probably makes a huge difference lol
[20:46:12] <StephenLynx> unless you already already familiar with the base tools and know what kind of compromises you are making in using these additional tools.