Log file Viewer
#mongodb logs for Thursday the 7th of April, 2016
[08:04:33] <dimaj> Thanks for the reply kurushiyama! at my work, we have several lab networks and a production network. One lab network has access to production, but the other does not.
[08:04:50] <dimaj> it so happened that I have to deploy my application on the network that does not have access
[08:05:19] <dimaj> so, I was hoping that I could trick my application into thinking that I'm talking to the mongodb on the lab network
[08:06:31] <dimaj> oh, my web app is a node.js application and i'm using mongoose for mongodb queries
[08:08:15] <dimaj> My requirement is to have my application deployed on network C and read data from database on network A
[08:09:22] <dimaj> my ops department is super busy right now and they won't get to my ticket for at least another week, if not more
[08:17:16] <dimaj> I'm basically hoping that one node is going to delegate all of the heavy lifting to another node and then just return the results back to the application
[08:18:18] <Derick> dimaj: there is no such thing like that with MongoDB - unless you can get a mongos on that network? I suppose you can't install anything either?
[08:19:18] <dimaj> machine on network C is a Cloud Foundry... so, can't install... can only deploy to it... machine on network B, i have root privileges
[08:20:26] <Derick> on network B, install a mongos (sharding router). On network C, install a mongod, and 3 mongo config instances. Configure your app on network C to use the mongos on network B. Configure the sharded environment through mongos on network B to talk to things on network A
[08:22:11] <kurushiyama> Derick Since it is a temporary solution... ... and we are talking of a single shard...
[08:23:40] <dimaj> and my app won't even know which shard it is talking to? as far as it is concerned, mongodb is that mongos isntance?
[08:23:44] <kurushiyama> Derick collections and databases themselves are unaware of sharding, right?
[08:24:29] <Derick> yeah, I think this could work with just one mongos and one config server on network B
[08:25:25] <dimaj> so, again, just to clarify... my final setup is going to look like this: Network A (mongodb); Network B - mongos and mongodb; Network C - node app that is talking to mongos
[08:25:51] <dimaj> Derick, that's fine... it's temporary, until my opts department will have time to address my ticket
[08:26:39] <kurushiyama> dimaj Correct setup. However, terminology is _really_ important. In B, it will be mongos and a config server.
[08:28:35] <kurushiyama> dimaj mongod is the daemon, which – under certain cisrcumstances – may be used as a config server.
[08:28:59] <dimaj> also, my DB is currently 20GB in size... will it all this data be in the config server?
[08:29:17] <Derick> the config server only stores on which shard (only one in your case) your data lives
[08:30:34] <kurushiyama> dimaj the config server stores the cluster metadata. The router, called mongos, fetches the metadata from the config server.
[08:31:10] <kurushiyama> dimaj Simplified, it actually caches the metadata and reloads it under certain conditions.
[08:33:03] <kurushiyama> Actually, backing up cluster metadata is the only valid use case for mongodump, imvho.
[08:35:50] <kurushiyama> dimaj There was a guy hammering it into my head "ALWAYS use 3 config servers". But he was referencing to a proper cluster setup. You have nothing to worry about.
[08:37:14] <dimaj> if, and when, it'll take off, we'll set up a proper cluster to make sure that it's fast and reliable and not hacky :D
[08:37:22] <kurushiyama> And even if. The worst case scenario would be that you get corrupted metadata. Which, with only a single shard, is pretty easy to overcome.
[08:54:32] <Ange7> Someone can explain why i have error « duplicate key error » but i make an update with $inc and options upsert = true
[08:58:34] <kurushiyama> Ange7 can you show some code? In general, this should work. a sample doc would not hurt, either.
[09:07:43] <ToMiles> Any reason that when I mongorestore a database into a freshly initiated replset of mongo servers that my db name got prefixed with "b=" ?
[09:08:10] <kurushiyama> Ange7 Most likely, your match does not match the already existing document with _id $key, hence mongod tries to insert a document, but then it finds that a document with $key exists and throws an error.
[09:15:37] <kurushiyama> ToMiles you might want to provide us with the command lines used to dump and restore.
[09:34:24] <Ange7> so is it possible to match only with _id field and if not exists insert other fields ?
[09:36:36] <kurushiyama> Ange7 Well, depends on your use case.db.foo.update({_id:"asdf"},{$inc:{a:1}},{upsert:true}) should always work.
[09:38:39] <Ange7> if the document not exists, it will be insert. so i need to set the name field too ?
[09:38:53] <kurushiyama> db.foo.update({_id:"asdf"},{name:"bar", $inc:{a:1}},{upsert:true}) should always work. Get a spoon.
[09:40:11] <kurushiyama> Ange7 As you are told not for the first time, and not only by me. Really, it does not help you if you get every problem spoon fed. You need to learn how to fish, not how to beg for fish.
[09:44:42] <diegoaguilar> kurushiyama, do u think it would be too dificult to get automatic s3 backup ina custom server?
[10:04:46] <Ange7> db.foo.update({_id:"asdf"},{name:"bar", $inc:{a:1}},{upsert:true}) should always work. Get a spoon.
[10:18:40] <kurushiyama> eje_ So basically, it is correct. limit on the other hand, modifies the cursor. Which is what you want. So call the length method on the cursor, and Bob's your uncle.
[10:31:50] <crazyphil> anyone know if its possible to stripe mongodb data across multiple directories?
[10:34:44] <kurushiyama> https://docs.mongodb.org/manual/reference/configuration-options/#storage-options
[10:34:45] <crazyphil> I was looking for a global way to let mongo do it itself, like I can do with Elasticsearch or Kafka (basically you can tell them use these X directories)
[10:35:02] <kurushiyama> crazyphil https://docs.mongodb.org/manual/reference/configuration-options/#storage.directoryPerDB
[10:35:50] <kurushiyama> crazyphil Why do you want that? Unless you have _a lot_ of collections and indices, this is not necessary
[10:37:38] <kurushiyama> crazyphil Since you _really_ do not want to do backups with mongodump on that scale.
[10:41:03] <kurushiyama> crazyphil Again, data or file size? Or, to put it differently: Do you use wiredTiger or MMAPv1?
[10:41:58] <crazyphil> I have no idea, I inherited this setup, and got a crash course on mongo operation
[10:44:18] <strepsils> Hey folks, I have a replica set upgrade question: going from 2.4 to 3.2 by adding a 3.2 node to the 2.4 replica set, once the data is replicated over making the 3.2 node PRIMARY, removing the 2.4 nodes and adding more 3.2 nodes. This is skipping the 2.6 upgrade the docs recommend, are there any risks in doing so?
[10:44:21] <crazyphil> it was supposed to be a sharded setup, however when I told the primary db to start replicated, it told me something about it couldn't because there was some dbs that was local
[10:44:22] <kurushiyama> Which most likely means that you have less than 1TB worth of data. It is just that the datafiles do not shrink.
[10:46:44] <kurushiyama> strepsils Uhm, iirc you are supposed to update to 2.6 first, and not out of fun. Skipping it may render your data unusable, so I would not.
[10:47:25] <kurushiyama> crazyphil uhm. your data size is bigger than your files? We might have a misunderstanding here... ;)
[10:48:28] <kurushiyama> strepsils Well, cool if _you_ want to bet your data on being lazy. I would not. Ever.
[10:50:00] <kurushiyama> crazyphil A sharded cluster may be composed of replsets or standalones (or both). First step is to start a replica set (if you want to), then do the sharding.
[10:51:49] <crazyphil> I'm pretty sure a replica set is there, as all 3 servers show "RS0" at the mongo prompt
[10:52:34] <kurushiyama> crazyphil Please pastebin rs.status() from the primary. Do not forget to anonymize the host names, if publicly available.
[10:54:13] <strepsils> @kurushiyama I don't want to play with data, it's just that after upgrading a few replica sets using this approach I realized we missed the 2.6 step. RTFM didn't happen when it was supposed to :(
[11:00:16] <kurushiyama> crazyphil Well, now you can do a sharding setup. However, you should be very, _very_, _VERY_ careful with choosing a shard key. It is one of the very few things you can really screw up with MongoDB, since it is _pretty_ hard to change.
[11:05:17] <avner_> Hi all, question re ReplicaSet URI setup, from driver side. docs say: "When connecting to a replica set it is important to give a seed list of at least two mongod instances.."
[11:05:18] <avner_> The "at least 2" are needed part, is it so that during first connection I get better HA, or is there a runtime implications (because all servers are auto discoverd in any case)?
[11:06:19] <cheeser> if you give just one server the driver (at least java's) will only talk to that one server ever. and that one might not be primary at the moment.
[11:06:55] <cheeser> by giving it at least 2, it knows to discover the topology of your cluster and find the primary/mongos machines
[11:09:58] <avner_> I use python, and the discovery works with a single server. I get all 5 in our case.
[11:12:11] <cheeser> kurushiyama: i'll double check that with the java driver to be sure but i don't believe it works that way.
[11:12:53] <kurushiyama> cheeser Well, I am not too sure wether I just implied it would or wether it is documented that way in the uri docs.
[11:13:16] <avner_> So, using replicaSet, and just one server in the uri, discovery of all servers works. Do I need to worry because of the docs saying “at least 2”?
[11:13:29] <kephu> got a kinda tricky question, is there any way to get a result where keys are one field's values, and those key's matching vals are from aggregation's $push?
[11:14:35] <avner_> @cheeser, I suspect the “at least” two, is to get a better HA on startup time, e.g. first connection, but once replicasset is discovered, it’s meaningless.
[11:15:14] <kurushiyama> cheeser Well, actually it isn't too clear about that. It says that if you only set one server _and_ omit the replset option, the client creates a standalone connection. I have to admit that I would conclude that it creates a replSet aware connection (including discovery) if the replSet option is set.
[11:33:04] <cheeser> kurushiyama: you might be right. i'll check the source after i get my kids to school.
[12:26:12] <jdo_dk> kurushiyama: Any ideas to create a better data model? if i have a model like: id: ID
[12:54:40] <kurushiyama> jdo_dk Well, the question is wether it makes sense to have a subdoc in the first place.
[12:56:57] <cheeser> kurushiyama: the java driver doesn't seem to even acknowledge the replSet parameter on a URI
[12:57:36] <kurushiyama> cheeser :/ Well, easy to complain, but I think that should not be the case.
[12:58:26] <kurushiyama> cheeser The way it is documented, a single host plus the param should force discovery. Imho.
[13:00:04] <kurushiyama> https://docs.mongodb.org/manual/reference/connection-string/#urioption.replicaSet
[13:00:38] <kurushiyama> Since the java driver accepts an uri, it should behave accordingly. What do you think?
[13:08:42] <cheeser> kurushiyama: confirmed: https://github.com/mongodb/mongo-java-driver/blob/master/driver-core/src/main/com/mongodb/connection/ClusterSettings.java#L189-L189
[13:13:04] <kurushiyama> cheeser well, do not know the java driver too well, so I was cautious ;) I was actually pretty astonished that it could be otherwise, as the principle of least surprise seems to be an important one for MongoDB, and the Java driver is supported...
[13:52:42] <myaskevich> I have a collection of 46 docs and it takes me ~23 sec. to retrieve one of them by id (IDHACK)
[13:55:19] <myaskevich> It doesn't. I have an another collection with thousands of docs and it pulls out very fast
[14:06:52] <StephenLynx> even if it weren't indexed, you wouldn't notice on a collection with 20 or so documents
[14:06:58] <siddc> Hi, I am trying to troubleshoot a mongodb issue. We have one primary and 6 secondary(voting) and 4 secondary(non-voting) notes. During replication the secondaries become unresponsive causing the monogdb to break the cluster and the primary becomes secondary. When they reform the cluster, there is no primary and they drop their DB and start the replication again and this is stuck on an endless loop. How do we fix this? We are using 2.4.14
[14:07:37] <myaskevich> But why. Look, I can pull the same doc two ways, by using _id or using name fields
[14:41:38] <StephenLynx> I don't know what is your mongo version, mongod version, your query, your schema
[14:44:30] <kurushiyama> siddc make sure your can ping each member by name as defined in the replica set config from each member.
[14:46:46] <siddc> Kurushiyama: Yup that was one of the first things I checked when starting to TS :)
[14:48:00] <siddc> Kurushiyama: I am not a DB so I don't know how to get these parameters. How do i get them?
[14:50:43] <kurushiyama> pamp That sounds more like an Ubuntu problem than like a MongoDB thing, no?
[14:53:40] <kurushiyama> siddc If you do not have it, we are going to have a problem debugging it. Running an 11 member cluster without metrics is like doing an ultra-low altitude flight at supersonic speed with a mobility cane and a street map for navigation. ;)
[14:54:26] <siddc> kurushiyama: I found the MMS tool. Just looking for metrics :). So Replication headroom on one node that I found is 1717S
[14:57:01] <siddc> kurushiyama: Right looking the Manage Charts thing and I dont see it. I see OpCounters = Repl
[15:01:54] <kurushiyama> siddc Hm... That should be sufficient headroom to do the initial sync, albeit depending on your setup, it might be not enough.
[15:13:36] <kurushiyama> Which, admittedly is not very likely, but more of a theory that fits the facts
[15:15:19] <kurushiyama> And even if that theory proves to be correct, it is not exactly easy to overcome that problem. My guess is that the primary is overburdened.
[15:16:13] <siddc> And to resolve that, I tried replicating with only 6 of the 10 secondaries but still same issue
[15:17:07] <kurushiyama> siddc Well, as said – pretty hard to impossible to find a solution "remotely"
[16:44:55] <Echo6> I have a foundation question about Mongo. As I understand it I can have "nested" collections. I have a customer that has a hotel booking website. If they were storing the hotels in one collection, the customer information in another collection and the bookings in another collection and the bookings use the hotels and the customer does a copy of the information for the hotel and the customer get duplicated to the booking document?
[16:48:58] <kurushiyama> Echo6 a) No, you can not. You can have embedded documents, which are a double sided sword: http://blog.mahlberg.io/blog/2015/11/05/data-modelling-for-mongodb/ b) You probably better use references, be them implicit or explicit (you have to decide that per use case). c) https://docs.mongodb.org/manual/applications/data-models/
[16:50:38] <Echo6> I'm not saying that I want to do it that way. I'm checking to see if I understand how Mongo works as there is conflicting information on the internet about it.
[16:57:27] <quadHelix> Is there a way for me to get the last element of a nested array? I love mongo, yet my newbishness shows with this data structure :\ I have an array like: "status" : [
[17:00:18] <quadHelix> I can get the results if I know how many entries are in the array using: db.orders.find({'status.2.status':'Printing'})
[17:00:20] <kurushiyama> Echo6 Well, here is the most valuable piece of advice I can give you: Use the official docs _only_ until you feel confident enough to judge the quality of the advice given at other locations.
[17:01:41] <kurushiyama> quadHelix http://blog.mahlberg.io/blog/2015/11/05/data-modelling-for-mongodb/ Most likely, you overembedded.
[17:03:17] <kurushiyama> quadHelix That is no doc, just one opinion. However, I am a stern supporter of the "Be as flat as possible" fraction.
[17:06:30] <quadHelix> I have been trying to use slice -1 and elemMatch without success. Flatness is my future friend.
[17:08:09] <quadHelix> I knew I would never hit the BSON limit. It is a glorified log file contained in an order. I will change my model to be flatter.
[17:18:56] <FreeSpencer> is there a way I can make certain that a user can only see certain records where a value = value? In mysql you can kinda do it with views
[17:53:38] <quadHelix> I have been able to search in a nested array using: db.orders.find({'status':{$all:[{$elemMatch:{status:"Submitted"}} ] }} ) Does anybody know how I could limit this to only return the last element?
[20:34:19] <roonie> mongo 3.2 broke all my tests because i think i can no longer maintain as many connections to the db. any one else encounter this problem?
[21:01:42] <wayne> i don't believe {_id: {$in: [id1, id2]}} would do the trick, because it can be out of order
[21:03:01] <StephenLynx> the best way to do it would be to sort on application code if it's a sub-array.
[21:06:27] <kurushiyama> wayne So you want to have the documents returned in the order of the ids given? Why would you?
[23:51:05] <Freman> hey kurushiyama, just talked our boss into the enterprise support, can you link me to pricing? :D