Log file Viewer
#mongodb logs for Tuesday the 2nd of February, 2016
[01:39:04] <Freman> is most simplistic... checks just that the first field in the query is the first field in any index
[02:50:33] <Waheedi> I've noticed something while using 2.4 server and client trying to read from nearest node using read preference
[03:21:22] <Boomtime> @Waheedi: when you use 'nearest' it is the driver that decides where to send queries - what driver (and version) are you using?
[03:24:45] <Boomtime> note you are using a very old version, so i can't be certain that behavior was intended - it isn't today
[03:45:55] <Freman> my proxy performs an explain on a given query and if no indexes are used it rejeects
[05:03:43] <Waheedi> If thats because its an old version then I can fix it, but if thats the default behavior I think something should be considered
[05:16:46] <Boomtime> @Waheedi: can you test with the latest version? the specification suggests that where the ping times are close (within 15ms by default) 'nearest' would round-robin but that only applies to later versions
[05:20:32] <Waheedi> sometimes i see words and characters coming after my sentences but they are not intended, this one is clear though
[13:15:28] <knuto> I'd like to add AUTH capabilites to an already running mongo database.. is this possible? (I'm getting "Unable to connect" etc.) is it possible in theory?
[13:57:37] <Lope> Apparently this is an unsupported projection? db.foo.find({_id:{$gt:ObjectId('56b0b138c16e3195036bb7f5')}});
[14:01:40] <Lope> What I wrote above was not my actual query. Sorry. My bad again. Maybe I should go to bed :)
[14:15:01] <Lope> is there a way I can mutate results in the mongoDB shell, so that my ObjectIds are returned as timestamps?
[14:19:04] <yossarianuk> hi - looking to setup a mongodb replica server- can this be done only using 2 servers or do I need 3 ?
[14:20:38] <StephenLynx> yossarianuk, I am not 100% sure, but I think the 3rd server is the arbiter that gets to pick one of the 2 other servers to step in when one fails
[14:21:10] <StephenLynx> so my GUESS is that while you don't need 3 servers, having 2 will only duplicate your data without adding resilience to your setup.
[14:21:29] <StephenLynx> but I am not experienced with replica sets so you shouldn't take just my word.
[14:22:12] <yossarianuk> whoops -> https://devops.profitbricks.com/tutorials/configure-mongodb-replica-set/
[14:24:06] <yossarianuk> however slightly confused now.... With Mysql for example we have MASTER -> MASTER replication bewteen 2 data centres, if a DC ever went offline we would still have the data, if we need one primary server set to replicate to2 secondary servers we won't have a failover for the primary?
[14:43:28] <pchoo> Hello all, we've got two stacks in two different DCs, looking to get a replica set, consisting of one on either side, and then a third party. We're looking at using an amazon Micro Instance for the arbiter, would that suffice, or would you recommend something with a higher spec?
[14:49:42] <StephenLynx> even if you slap some abstraction layer on top of mongo that does that behind the scenes
[14:50:05] <StephenLynx> it is just fetching data from two separate collections on two separate queries and doing some application-side operation to output a result
[15:40:14] <GothAlice> the_drow: You can "cask pour" using homebrew to install a binary version instead of compiling from source.
[15:40:51] <GothAlice> Additionally, the downloads on the official site are an (effectively, since it's technically not possible on Mac) statically compiled version of the command-line tools you can just untar somewhere and run in-place.
[15:43:24] <GothAlice> the_drow: brew search mongodb would indicate that the "cask to tap" (I love homebrew terminology!) Caskroom/cask/mongodb
[15:50:29] <GothAlice> Uh, wow. https://docs.mongodb.org/manual/core/index-unique/#unique-index-and-missing-field and https://docs.mongodb.org/manual/core/index-sparse/#sparse-and-unique-properties would seem, on the surface, to contradict each other directly.
[15:50:57] <GothAlice> "An index that is both sparse and unique prevents collection from having documents with duplicate values for a field but allows multiple documents that omit the key." vs. "Because of the unique constraint, MongoDB will only permit one document that lacks the indexed field. If there is more than one document without a value for the indexed field or is missing the indexed field, the index build will fail with a duplicate key error."
[15:51:55] <GothAlice> Aaaah, no, the sparse flag just wasn't actually getting set on index creation from the ODM layer. >____<
[15:59:43] <the_drow> GothAlice: it's kind of a bummer to install mongodb itself if you only need the tools
[16:00:19] <GothAlice> I mean, unless you launchctl load it, it's hardly consuming system resources.
[16:01:00] <GothAlice> the_drow: Well, that certainly bloats more than just un-tarring the tarball somewhere and running the resulting files.
[16:02:20] <GothAlice> the_drow: By running on Docker, you're hardly actually running MongoDB on Mac. You're running it on Linux in a VM. Any overhead is… self-inflicted.
[16:54:03] <csd_> How do I search for all collections that have a subtree that matches a certain pattern? (e.g. i dont know what the outer key will be)
[17:00:04] <Tomasso> db.getCollection('Accuchecks').aggregate({$group: {_id: "$algorithm",accuracy: { $avg: "$difference.close"}}})
[17:03:15] <Tomasso> haha me neither.. but the values from the field difference.close almost never are 0
[18:04:52] <uuanton> hi anyone can help find the documentation on how to rollback replica set to backup of amazon snapshot ?
[19:11:03] <renlo> how do you install / (specifically update) the mongo shell without installing mongod ?
[19:25:12] <idd2d> I've got some questions about modeling some data relationships in a new app with mongo, but it's likely kind of involved to give enough context to explain the relationships. What's the protocol for this?
[19:25:37] <idd2d> I guess I could just throw it out there, explanation and all, and hope someone bites? Just hate to blow the time if nobody is feeling up to helping out
[19:34:53] <GothAlice> Took a year, but https://jira.mongodb.org/browse/SERVER-15815 was fixed by https://jira.mongodb.org/browse/SERVER-15176
[19:44:33] <GothAlice> No, now I have fine-grained control over getMore timeouts across tailing cursors.
[19:45:19] <GothAlice> I.e. myAwesomeTask.result(timeout=2) — actually waits two seconds now instead of waiting for the cursor to die of old age and/or try, sleep a second, and repeat polling.
[19:47:07] <GothAlice> idd2d: And yeah, the general process is to throw ideas out to get feedback; "ask, don't ask to ask" being the general idea. IRC is asynchronous, and nobody can
[19:48:12] <GothAlice> "Modelling relationships" is sticky business when not using a relational database, though. I often link http://www.javaworld.com/article/2088406/enterprise-java/how-to-screw-up-your-mongodb-schema-design.html as a brief primer to some of the concerns.
[19:48:33] <GothAlice> StephenLynx: Memory mapped, efficient, secure. What's not to love about UNIX domain sockets?
[19:49:30] <StephenLynx> yeah, but I rather have it secured by the authentication method I am already using over mongo.
[19:50:28] <StephenLynx> ok, but when? when I start listening for the socket it won't automatically create the file
[19:50:44] <GothAlice> But the memory mapped thing is great, and the elimination of TCP overhead for local connections is awesome.
[19:51:08] <StephenLynx> while if I had a tailable cursor that never closes, there would be no code overhead.
[19:51:24] <StephenLynx> and since this feature doesn't need to perform fast, the performance is not an issue
[19:52:55] <GothAlice> Uhh, binding to a domain socket creates it if possible. It's then up to the application to chown/chmod it as needed.
[19:53:55] <StephenLynx> and my software can't change its permissions sanely, because I don't mandate permissions or ownerships.
[19:55:49] <GothAlice> Most daemons like MongoDB generally accept a "user/group to become" option and re-use that when constructing their on-disk sockets. Some, like the FastCGI on-disk socket adapter I use in Python, also lets you set a "umask" for the sockets; this sets default permissions configurably.
[19:57:36] <StephenLynx> I just assume the person is using the intended system user to run the application.
[19:57:50] <uuanton> Hey Alice, by any chance do you know strategy how to restore replica set from aws snapshot ?
[19:59:14] <GothAlice> uuanton: Alas, I switched away from AWS before we really started pickup up MongoDB at work. :(
[20:00:07] <GothAlice> uuanton: A first set of steps might be to clone a new EBS volume from the snapshot, delete the journal files, then spin up a standalone mongod across it to see if you can get your data readable; it may require a --repair depending on your snapshot process.
[20:00:42] <GothAlice> You could then follow the standalone-to-replaca-set tutorial to bring things back to production-worthy: https://docs.mongodb.org/manual/tutorial/convert-standalone-to-replica-set/
[20:05:54] <uuanton> i've done above steps, after i switch volume on all 3 replica set members and reconnect secondaries they resync from scratch
[20:08:17] <GothAlice> So you're swapping out the disk storage for three configured nodes then spinning them back up?
[20:09:13] <GothAlice> Re-syncs happen when a secondary knows it's a member of a set and knows that its data is older than the current primary by longer than the replication oplog time. (It does this because it knows there is literally no way for it to actually "catch up" and still see everything that changed.)
[20:17:30] <GothAlice> uuanton: The "bring it back up as a new primary, then promote back into a replica set" approach will sync the new replicas, unless you take a new snapshot and clone it in under the oplog time, but will be potentially more reliable/controllable a process than swapping out the underlying storage of existing nodes and just restarting them.
[20:19:12] <GothAlice> MongoDB itself now has schema validation, so the need for an ODM layer to do that is becoming excessive.
[20:20:02] <GothAlice> Ref: https://www.mongodb.com/blog/post/document-validation-part-1-adding-just-the-right-amount-of-control-over-your-documents which introduces the idea, and the documentation: https://docs.mongodb.org/manual/core/document-validation/
[20:21:02] <GothAlice> I'm a Python developer, and even I'm seriously considering switching back to the plain MongoDB-provided pymongo driver.
[20:22:20] <dylush> I have an issue where mongoose is throwing a cast error saying I have an undefined _id even though the update works
[20:26:15] <Spynxic> I'm using pymongo and I noticed that when I insert a document into a collection like so, db.collectionname.insert({ someid:{ someotherid:somevalue }, .. }) the someid document gets an _id but someotherid document doesn't -- does that mean it isn't a document?
[20:26:17] <GothAlice> dylush: Have you run into accidentally constructing a collection literally named "[object Object]" yet? How to fix that once it happens makes for a fun MongoDB challenge question.
[20:28:12] <GothAlice> Spynxic: To answer the question, though, only the topmost document has an _id field added to it (by default with a newly generated ObjectId generated on your application-side prior to pymongo sending the request off to the server). Embedded documents are stored as-is, and ObjectId generation is skipped if you provide your own _id at the top level.
[20:29:29] <GothAlice> http://www.javaworld.com/article/2088406/enterprise-java/how-to-screw-up-your-mongodb-schema-design.html
[20:30:44] <GothAlice> Adding your own _id to the embedded documents is OK, but that field won't be automatically indexed. (So remember to create that index, if possible.) from bson import ObjectId – simply creating a new instance of ObjectId() will generate one according to the standard.
[20:31:18] <GothAlice> (My forums do exactly this to add an _id to each reply, embedded in the thread.)
[20:33:16] <dylush> @GothAlice Not that I am aware :S it's strange the update works perfectly but I get the recurive cast error in my console
[20:40:42] <Spynxic> One more question, unrelated to the last. Is there a method to iterate the fields of a document similar to iterating through a dictionary (in Python) like so, for key,value in dict.iteritems()
[20:45:05] <GothAlice> (_Actually_ iterating dictionaries, and not the result of calling one of their iteration methods, iterates keys.)
[20:50:02] <Spynxic> If I'm iterating {"a":1, "b":2, "c":3} then I can get 1 2 and 3 from your example but I can't seem to find a way to get the a b or c from the value
[20:52:46] <Spynxic> One solution would be to put the key in the value like so, {'a': {'key':'a', 'value':1}, 'b':{... but that's my last resort
[20:54:18] <GothAlice> Order isn't preserved, but that's a Python quirk of how dictionaries work (by hash, not value).
[20:57:02] <Spynxic> Oh thank you, list(dict) is exactly what I'm looking for. I'm using the key to access a value in a different document with the same key
[20:59:47] <GothAlice> Spynxic: If iteration is your goal, getting the list of the dict is generally not what you want. It needlessly constructs a list prior to iteration if used like for field in list(somedict).
[21:00:08] <GothAlice> Spynxic: Similarly, if you're checking for the presence of a field, this'll do: "somefield" in somedict
[21:02:37] <GothAlice> (One almost never needs to turn the dictionary into something else just to get the keys, or just to get the values, or the combination of the two as items.)
[21:29:12] <Freman> figure out permissions so I can add a new role with the existing permissions and depreciate the existing role so it can only query not insert/delete
[21:49:42] <GothAlice> Hmm, Freman, db.serverStatus().connections only lists counts, not actual connections. But listing the current connections, if possible, should also list which identity they're authenticated as.
[21:52:19] <Freman> thanks GothAlice, it's not so important, the two services I know that should be writing have been moved to writer roles, just was curious to see if anything else was writing and should be updated before I dropped the writing role from the old user
[22:29:39] <decl> hi. quick question and i know this isn't the right channel to ask... but do you happen to know if there's any kind of database that'd perform good on dropbox?
[22:30:57] <GothAlice> decl: None. The per-file latency is too high, and the gods know what happens if two hosts try to modify the same file. No atomic locking, …
[22:32:01] <GothAlice> It took Apple more than six years before they worked out most of the bugs of trying to use SQLite files over synced iCloud storage (formerly Mobile Me, formerly .Mac).
[22:32:49] <GothAlice> Yeah. It was so bad after the iCloud name change that most app developers, when the APIs were fresh and newly public, refused to use it. Thus my task manager has Things Cloud instead of iCloud sync.
[22:34:54] <GothAlice> If something as "simple" as a to-do list can't get distributed file-based data storage to work…
[22:36:50] <uuanton> hey Alice I followed your advice to drop replica set settings and reconnect replica set. Secondaries start resyncing from scratch
[22:38:15] <GothAlice> https://docs.cloud.mongodb.com/tutorial/use-restore-to-seed-secondary/ < you'd need to follow something like this to make a new snapshot of the new primary to transfer to the new secondaries, as the snapshot the primary was constructed from is "too old" to meet the oplog requirement for the secondaries.
[22:41:34] <Freman> "$err" : "No index was found that could be used for your query try db.log_settings.getIndexes()"
[22:41:36] <uuanton> problem is that snapshot is from production with different replset settings. But restore is for staging and replicaset settings need to be modified
[22:42:12] <GothAlice> https://docs.mongodb.org/manual/tutorial/reconfigure-replica-set-with-unavailable-members/ should help with that last one, uuanton.
[22:44:53] <GothAlice> uuanton: Basically, restore a single member, convince it it's not a replica set, re set up the replica set, optionally with a fresh snapshot/backup to restore to the new secondaries. That's the process, and the last link there outlines how to force it if it won't cooperate. ;)
[22:49:41] <GothAlice> uuanton: This is why on particularly large datasets I don't use filesystem snapshots or mongodump as a primary backup mechanism. I use _another_ secondary. ;)
[22:50:53] <GothAlice> Bajix: mapReduce in MongoDB is accomplished by sending JavaScript code to the server for execution.
[22:51:55] <GothAlice> uuanton: Say I have three replicas in the datacenter where my app lives. I also have a pair of secondaries (hidden ones) in the office. And one at home.
[22:52:05] <Bajix> I get that part - what's confusing me here is that my map/reduce functions work, but if I add query, then I end up with output that hasn't been reduced
[22:52:13] <GothAlice> Absolute worst-case, I can serve our app from anywhere, with data coming from our office. ;)
[22:53:03] <GothAlice> https://docs.mongodb.org/manual/core/map-reduce/ < ensure you are specifying the query correctly; mapReduce is not like find.
[22:54:12] <Bajix> I'm just using Mongoose query casting - which has always worked for me when using aggregation
[22:54:18] <GothAlice> Bajix: Please avoid pasting in-channel; use a paste service like Gist. Also, that's the query part… but what's really important here is how you're putting together the actual mapReduce call.
[22:55:07] <GothAlice> Can you try getting your map/reduce and query working in a plain Mongo shell?
[23:10:47] <idd2d> I'm trying to implement tags in a new app. I've got a collection of people, and each person can have multiple tags. I need to be able to search by tags. The situation is similar to this example: http://programmers.stackexchange.com/questions/190802/how-to-model-hashtags-with-nodejs-and-mongodb
[23:11:33] <idd2d> However, there's one critical difference: I can't embed the tags on the person document. This is for scoping concerns. Basically, every user of the app will be drawing from the same large pool of People, but they should not allowed to make changes to these documents.
[23:12:07] <idd2d> So what's the best approach? Is it to create another tags collection, and associate each tag with a person via personID?
[23:19:57] <GothAlice> Well, an initial problem which may be unrelated that I notice is that your reduce function can't eat its own output.
[23:20:34] <GothAlice> I.e. it emits a reduced group document in a different format than the records it's grouping. This can be problematic.
[23:23:02] <GothAlice> Actually, not eating its own output may, in fact, be the issue. Without a query there's no reason to issue multiple reduces, you just stream the values as they're mapped. With a query, though, it might sparsely run the index issuing reduces in batches.
[23:23:23] <GothAlice> If that happens, a final reduce will be run with the output of the prior mini-reduces, leading to failure.
[23:25:45] <GothAlice> Generally experienced MongoDB users, including myself, will recommend using the Aggregate framework instead of map/reduce. Is there any particular reason holding you to this less efficient data processing model?
[23:27:29] <Bajix> This is my first time using MapReduce actually - I've only ever used aggregate, which I'm quite familiar with
[23:29:06] <Bajix> I'd like to use aggregate if possible, but for that to be viable, I would need to be able to count tokens grouped into all hour ranges between createdAt & lastActive
[23:29:38] <GothAlice> https://gist.github.com/amcgregor/1ca13e5a74b2ac318017 < pardon the field name shortening
[23:30:10] <GothAlice> This example groups by day of the week, but you can see some of the operator possibilities. You can extract date/time segments easily, $match query, etc.
[23:31:14] <GothAlice> Ref: https://docs.mongodb.org/manual/meta/aggregation-quick-reference/#date-expressions
[23:32:11] <GothAlice> See also https://docs.mongodb.org/manual/reference/operator/aggregation/cond/#exp._S_cond to allow you to $sum conditionally. :)
[23:34:10] <GothAlice> Just so I have a head for what's going on here, what is the value of query after line 10?
[23:34:58] <GothAlice> (Ignoring the exact date values being range-queried for; those obv. depend on the request time.) Ok, so nothing crazy coming from req.query. ;)
[23:35:01] <Freman> queryguard is live on our shards... I probably should have added statistics so I can see how many queries it rejects
[23:35:43] <GothAlice> Freman: Heh, without instrumentation how can you tell that it's not rejecting them all? ;)
[23:36:16] <Bajix> So, let's say for example the difference between createdAt & lastActive is 5 hours - that token would need to be counted in all 5 hours, which is where the date grouping falls flat
[23:37:00] <Bajix> i thought about using an earlier group to push each hour as a new token, and then to unwind it, but that seemed terrifying
[23:37:36] <Freman> tho I might add source ip to the request log cos I has no idea who's executing what
[23:38:03] <GothAlice> You have singular timed-duration events which you're trying to pivot to repeated per-hour events covering the durations.
[23:42:02] <GothAlice> Well, no, that's the thing. With a $group on null there's only one record per group thus only one $push issued per record.
[23:43:12] <Bajix> Making one giant array didn't seem sane, which is why I went over to use mapReduce
[23:44:23] <Bajix> Maybe there's a way to use $project + $unwind to accomplish pivoting here, but I'm not seeing it
[23:46:13] <GothAlice> Hmm. Within the standard flow we can easily calculate the initial "reporting period" and number of periods covered.
[23:46:41] <GothAlice> (That's a little bit of $mod action for the former, division of that result for the latter.)
[23:51:18] <GothAlice> Aye; transforming the rich date object into a UNIX timestamp integer, then subtract the value modulo the period size in seconds to snap it to the period size. Then, to determine how many periods are covered for that entry, subtract that snapped period from the end date, divide by the period size and ceil round.
[23:52:13] <GothAlice> Rather, subtract the original start date/time from the end. ¬_¬ Clearly sleep depravation isn't working for me, here. ;)
[23:54:55] <GothAlice> It gives a value we can use to easily identify the exact time periods of each affected slice, for each record.
[23:58:37] <uuanton> Alice, mebbe you know how to drop replset settings without dropping oplog inside local database