Log file Viewer
#mongodb logs for Tuesday the 5th of July, 2016
[06:03:27] <7GHABHM56> Hi there, can I set [secondary to other hidden replica ] replication without touching anything on my primary node ?
[06:05:20] <kurushiyama> 7GHABHM56 Please explain in more detail what you want to achieve – and why.
[06:06:39] <7GHABHM56> I mean, is it possible like, I have a secondary replica, I want to make a clone of it,(yet another secondary hidden replica) but don't want (actually afraid) to touch primary node ? kurushiyama
[06:08:02] <kurushiyama> 7GHABHM56 You want to instruct a hidden _member_ to replicate from a specific secondary or you want a different replica set to read from a specific secondary (which would not make much sense).
[06:09:50] <kurushiyama> 7GHABHM56 You want a MEMBER to replicate from that secondary? Or a second replica set?
[06:11:17] <7GHABHM56> I want to make new replica set (fresh start) which syncs everything from a secondary node kurushiyama
[06:13:59] <kurushiyama> 7GHABHM56 You add one, you remove one, to keep an uneven number of members.
[06:16:02] <bambanx> i connect to my remote server on webfaction using ssh, then i run $HOME/webapps/application/mongodb-linux-architecture-version/bin/mongod --auth --dbpath $HOME/webapps/application/data/ --port number
[06:17:54] <kurushiyama> 7GHABHM56 Basically all you need is to start your new members with the same replSet name, and then use https://docs.mongodb.com/manual/reference/method/rs.add/ and https://docs.mongodb.com/manual/reference/method/rs.remove/
[06:19:32] <bambanx> the hosting recommend me this method , what u think guys? https://community.webfaction.com/questions/6157/watchdog-script-to-keep-process-running-with-cron
[06:21:00] <7GHABHM56> #eg: rs.add({_id: 4, host: "192.168.2.123:27017", priority: 0, hidden: true, votes: 0 })
[06:21:36] <kurushiyama> 7GHABHM56 Please, how does your current replset look like? How many nodes, is there an arbiter?
[06:22:24] <7GHABHM56> currently there are 3 nodes, (1 primary, 1secondary and 1 arbiter) kurushiyama
[06:23:07] <kurushiyama> 7GHABHM56 Ok, here is what I would do, fwiw. Add the first new node, remove the arbiter. No harm done, since no impact on the replication factor.
[06:24:13] <7GHABHM56> I want to add new set, such that I could sync from secondary, isn't it possible or not recommended method ?
[06:25:25] <kurushiyama> 7GHABHM56 After the new node is synced, add the next of the two nodes, remove the old secondary.
[06:25:57] <7GHABHM56> :( I thouht you would recommend sth like this: https://docs.mongodb.com/manual/reference/method/rs.syncFrom/#rs.syncFrom
[06:26:13] <7GHABHM56> 12:00 <kurushiyama> 7GHABHM56 After the new node is synced, add the next of the two nodes, remove the old secondary.
[07:10:50] <kurushiyama> 7GHABHM56 Not interested in it. wiredTiger is pretty good at what it does, and I prefer battle tested storage engines for my persistence. Forgive me, but given your skill level, you probably should not make any experiements until you have at least a bit more experience.
[07:14:06] <7GHABHM56> hmm, rocksdb seems pretty promissing, compared to what I am using (Memory Map)
[07:29:08] <7GHABHM56> We we very helpful from what percona offered (tools like online schema changer for mysql on live production env ).. so kinda more biased to experiment percona mongo with rocksdb.
[07:30:53] <7GHABHM56> kurushiyama: any awesome urls(blogs,youtube), book you want to recommend boosting my mongo skills ?
[07:38:45] <7GHABHM56> where could I jump dive to learn about storage engines and replication...for that course I will have to wait(which I will)... and time is what I have not
[07:54:20] <kurushiyama> 7GHABHM56 Rushing that kind of stuff usually bites you in the neck, and rather sooner than later.
[10:24:05] <cxeq> hey guys quick super noob question about mongodb: if I have a collection of clients with each client on an individual document, and I want to make a list of items sold to each particular client, or something along those lines
[11:32:36] <Zelest> the error I get is "Uncaught MongoDB\Driver\Exception\BulkWriteException: Cannot do an empty bulk write" o_O
[12:45:36] <chris|> is there a way to get a measure about the size of the queries from the mongocluster?
[12:51:37] <deathanchor> or you can do a db.collection.stats() to find out the average size of the docs and guesstimate from there and a count of your query
[12:51:52] <_matix> i have a db dump folder where all of the files are named {prefix}\{collectionname}.bson or {prefix}\{collectionname}.metadata.json. I'm certain that {prefix} is the db name. how do i restore this?
[12:54:05] <chris|> deathanchor: all true, but also not what I am looking for. I am looking for a metric or a stat of some kind that tells me about the size of the document that is sent to the cluster as part of the query
[13:26:03] <GothAlice> chris|: You can also easily do some napkin-calculations using the sizes given in: http://bsonspec.org/spec.html + protocol documentation in https://github.com/mongodb/specifications/tree/master/source
[13:27:32] <GothAlice> The protocol is itself BSON, so, it's just a matter of adding up the parts. Or, wiring together the BSON protocol parts from your language driver of choice (i.e. the "bson" module from "pymongo", in the Python case) to add an event listener for different types of communication, then use a similar bsonsize() thing on the messages.
[13:29:41] <GothAlice> Heh, the aggregate I'm working on this morning would take three separate IRC lines, minified. XD
[13:33:50] <GothAlice> In fact, you can. You can have any join you want as long as it's a left outer. (Ford joke.)
[13:34:55] <Zelest> the good news? i can do joins! the bad news? i have to redo parts of my application now :(
[13:35:01] <GothAlice> It's… still not recommended to structure your models this way, as it's still "inefficient" (just _slightly_ less, now), but sometimes it can't be avoided.
[13:35:47] <Zelest> in my case, i have one to many relation.. where the many _id's are stored in a sub-document array
[13:36:25] <Zelest> now i fetch the array and loop through it and fetch each _id individually.. which probably is way more inefficient as $lookup
[13:37:55] <GothAlice> Yes and no. If you have an array of IDs you need to look up, you'll need to $unwind on that, then $lookup the resulting singular values, then $unwind that, then $group on the original ID ($push'ing the individual foreign values back into an array). It _might_ be more efficient than doing it application-side, and will certainly benefit from future enhancements to $lookup (a number are planned).
[13:42:11] <GothAlice> Some are simply truisms, such as "Everything is air-droppable at least once."
[14:03:30] <chris|> GothAlice: the thing is, I am seeing some serious impact on the mongo with pretty small changes in oplog, so I am currently trying to verify that someone is just "doing it wrong", assuming he is just sending huge documents over the wire that contain next to no updatable data. I could of course review all the application codes and calculate data usage for data types I did not write, or hack on the application
[14:06:46] <GothAlice> Well, given most modern drivers incorporate listener (i.e. event, or callback) functionalities, monitoring communication shouldn't be too difficult.
[14:20:18] <GothAlice> chris|: I've had users store documents containing JSON strings as primary data storage, so I feel your pain. ("I wanted to store JSON… what's wrong with that?" :head-desk: ;)
[14:38:21] <yopp> um. I wonder how bad it will be to run mongo in docker with ceph-backed volume driver
[19:13:05] <cruisibesares> anyone know what this operation is doing replSetUpdatePosition I'm seeing that in the log taking 58 seconds
[19:43:39] <zylo4747> I am running MongoDB 2.6 and would like to migrate databases to another replica set. If I have all of the users setup in the admin database on the target and I restore a database with mongorestore, will the users and their permissions be there still or will it get reset?
[20:14:37] <GothAlice> zylo4747: The roles assigned to users should be preserved. Fine-grained permissions come down to the names of the "namespaces" used (typically just the collection names) and should be unaffected. (I.e. there's no hidden identifier used that gets reset.)
[20:14:52] <GothAlice> Unrelated: http://s.webcore.io/020e1y1n0G0r < a common MongoEngine problem I loathe. Instead of raising the validation error where it's checked, which would provide, y'know, a _useful_ stack trace, it instead gathers errors and raises them at the last (and shallowest) location possible, making it utterly useless. Given that message, which field exploded? What was the bad value used?
[20:26:35] <nalum> Hello all I'm having a bit of trouble running mongo-connector. I have a 3 node replicaset running on kubernetes and trying to get mongo-connector to sync data from that to elasticsearch but it is giving me the following error
[20:26:51] <nalum> pymongo.errors.ServerSelectionTimeoutError: mongodb-svc-02:27017: [Errno -3] Try again,mongodb-svc-01:27017: [Errno -3] Try again,mongodb-svc-00:27017: [Errno -3] Try again
[20:36:27] <GothAlice> kurushiyama: Exceptions in my libraries tend to be exceptionally informative. There's no reason not to be. (I.e. they don't just tell you what the problem is, and where the problem is, but often also explicitly how to fix the problem.)
[20:37:53] <GothAlice> They also don't defer such error generation until later, which obfuscates the source. If you assign an invalid value to the attribute, it immediately explodes in Marrow Mongo, rather than deferring to the Active Record .save() method in ME's case.
[20:39:07] <kurushiyama> @GothAlice I agree with your approach, and I guess most users appreciate that behaviour. But it requires discipline and dedication to quality. Qualities desperately missed in certain subgroups of devs ;)
[20:40:46] <GothAlice> kurushiyama: Hehehe, did you see my "migrate away from ME" meta-/tracking ticket? It's kinda astounding. https://github.com/marrow/contentment/issues/12
[20:41:42] <kurushiyama> @GothAlice Well, I am a Java and Go dev, so I am not actively following marrow, I am afraid.
[20:42:52] <kurushiyama> Albeit I still try to wrap my head about Mongoose. Maybe I am just missing its point, given my lack of knowledge regarding JS.
[20:44:33] <StephenLynx> the only reason mongoose took off was the dev working for mongodb and being able to promote it
[20:49:09] <kurushiyama> Well, I guess we all agree that enforces a bad, if not outright dangerous pattern of approaching data modelling. Aside from that, maybe it gives the JS developers something they miss within the driver. Dont know.
[20:51:10] <kurushiyama> StephenLynx Given your level of experience with it and MongoDB in general...
[20:52:28] <StephenLynx> being good at js is more about knowing what to not use than knowing what to use.
[20:54:55] <kurushiyama> It does all I need, and admirably so – but then, what I need to do is very basic: minify, uglify, htmlprocess a bit, LESS here and there.
[21:03:52] <nalum> I started out with JS in the mid 90s and loved it for a few years, I have no patients for it now
[21:05:28] <GothAlice> JS is a terrible anti-language, with multiple examples (some trivial, such as 'typeof "string" !== typeof new String("also a string")', or the "NaNNaN… Batman!" bit, others pointing straight to the specification, such as the weak comparison decision tree…)
[21:09:09] <nalum> Is there a tool other than mongo-connector that can sync data between mongodb and elasticsearch?
[21:09:57] <GothAlice> nalum: … depends on what you need, and what you mean by synchronization. https://github.com/richardwilly98/elasticsearch-river-mongodb/ is an alternate approach.
[21:12:19] <nalum> pymongo.errors.ServerSelectionTimeoutError: mongodb-svc-02:27017: [Errno -3] Try again,mongodb-svc-01:27017: [Errno -3] Try again,mongodb-svc-00:27017: [Errno -3] Try again
[21:12:47] <GothAlice> Hmm. Which mongod version are you using, and which version of pymongo is installed?
[21:14:40] <GothAlice> http://api.mongodb.com/python/current/tutorial.html < following the first few steps here (up to "getting a collection") works in a "python" REPL shell?
[21:15:14] <nalum> I'm running on kubernetes, let me get a pod up and running that I can test that with
[21:15:59] <GothAlice> Hmm; more layers I can't really assist with, if the problem lies somewhere in there. (As an aside: when testing issues, I tend to boil problems down to the absolute minimum number of moving parts to help in diagnosing the true cause.)
[21:22:29] <GothAlice> (On my work project, wrestling with aggregates, phase one refactor has improved report generation performance ~1700x. $lookup, you are my new best friend.)
[21:23:19] <GothAlice> It's glorious when stupidly complex reports generate with no perceptible delay between filter selection and result. \o/
[21:24:34] <GothAlice> Still takes 15 seconds for the "all companies" (unfiltered) resultset, but that only impacts staff, not users, so I'm okay with that. XP
[21:50:57] <GothAlice> nalum: Sorry, was walking home from work on this gorgeous late afternoon. Hmm. I'd have to dig in to how mongo-connector is set up to see what the issue is, and I have no ElasticSearch handy to test it against. :(
[21:52:04] <GothAlice> For a long time mongo-connector was stuck on the old connection driver, preventing use with 3.0 (making use of updated authentication) or 3.2 (updated protocol).
[21:54:41] <nalum> Ah. I've been using the connector with 3.2 for a little while now. I'm currently migrating to kubernetes and just started getting mongo-connector set up today
[21:56:22] <dsc_> trying to do 'between' query with '$lt' and '$gt', is this the correct way? http://pastie.org/pastes/10899758/text?key=w2oeoxhokgwsjpxrfrw
[22:00:51] <GothAlice> dsc_: So, that's the trick. 6-8 inclusive (6 and 8 are valid), or exclusive (6 and 8 are not valid)?
[22:01:20] <dsc_> first time doing mongodb queries, no idea what im doing. Is what I pasted correct?
[22:04:20] <GothAlice> However, more generally, you don't need the $and. 95% of the time, "and" is assumed.
[22:04:57] <GothAlice> That would be the simplified version, not using an $and. Now, is info an array of values, or a single sub-document?
[22:06:20] <GothAlice> If "info" were an array, the query would mean something slightly different. ;)
[22:07:04] <GothAlice> Almost the way you'd expect. info.score would refer to _any score_ in the info array.
[22:08:08] <GothAlice> Thus your range query would mean "match any document that contains any array element whose "score" field is < 5, and that contains any array element whose "score" field is < 8. That's where $elemMatch comes in handy; that pins the array subsection of your query to matching a single array element that matches all criteria, instead of splitting them up like the previous example.
[22:08:48] <GothAlice> (I keep using < / $lt and > / $gt because that's what's what was in your original example, pretend I'm using $gte/$lte and whatnot. ;)
[22:09:48] <nalum> Alright, gonna have to continue this tomorrow :(, night and thanks for the help GothAlice
[22:10:07] <GothAlice> nalum: It never hurts to help, and apologies for not fully solving your issue. We've at least ruled out certain things, though! :)
[22:12:03] <GothAlice> dsc_: To clarify, in the non-$elemMatch case, if "info" is an array of sub-documents with a score field, the two parts of the query don't necessarily mean it's a single array element matching both.
[23:09:42] <zylo4747> can i stop mongodb, copy the data files to a target replica set and have that replica set automatically "see" those database files as another database?
[23:18:04] <cheeser> like so? https://docs.mongodb.com/manual/tutorial/resync-replica-set-member/#sync-by-copying-data-files-from-another-member
[23:19:54] <fission6> this is kind of weird… i haven't used mongo in a year or so, i just went to startit up so i could run a web app i develop locally and I can't find the mongo command
[23:27:03] <cheeser> fission6: um. 1. your path is wrong. 2. mongodb isn't actually installed. 3. whut?