Log file Viewer
#mongodb logs for Wednesday the 31st of August, 2016
[01:02:53] <Jasper> Does anybody know why adding .explain() to a query in the MongoDB shell makes it so much slower?
[01:03:52] <Jasper> (Context: we're trying to debug why db.jobs.find({ status: 1 }) is so much slower than db.jobs.find({ status: { $in: [1] } }) -- and trying to explain both of them is failing)
[05:44:50] <_jaitaiwan> Hey guys, I'm using aggregate to combine a list of "timeslots" together to form a sort of schedule object. I'm trying to determine how to get one of the return fields to be a count of each status (int) from each timeslot. Any hints about where to look?
[07:27:05] <SimpleName> when I change the default data directory, it raise a error: Permission denied Is a mongod instance alread
[09:30:47] <jamieshepherd> So I'm getting Class 'MongoDB\Driver\Manager' not found when using PHP's latest MongoDB driver. Not using old deprecated driver, but this only happens on php7.0, not earlier versions - anyone have any ideas?
[09:58:24] <thapakazi> hi there, where can I get details on mongoid gem ? to be precise for this configuration options: https://github.com/mongoid/echo/blob/master/config/mongoid.yml#L42-L45
[11:08:50] <SimpleName> as we know, if we deploy the mongodb-replica set, we moidfy bind_ip to 0.0.0.0, now it become unsafe, so how to handle this situation
[11:13:05] <SimpleName> 😯， just let fixed-ip + 27017, through my server, is that right ???? Derick
[11:21:37] <SimpleName> For replica sets and sharded clsuter, you can also enable access control by enforcing internal authentication. For details, see Internal Authentication. Derick have you use this kind of auth
[11:25:26] <SimpleName> I have to write html5,js, python server, and handle mongodb, sql-database, fuck
[13:44:37] <kba> I want a collection of tags. Each object has an id and a collection of tags. A tag is a number and a tag name. So my collection will be a list of objects like { _id: "foo", tags: { 5: "bar", 81, "quux" }Â }, for instance
[13:48:44] <kba> I know little about the mechanics, but if I have an array with just 2 indices, say 10,000 and 20,000, won't it still allocate at least 20,000 fields?
[13:54:06] <kba> I assumed it used a hashmap in which case a loopup would be O(1) compared to stupidly running through an array O(n) or using an index O(log n)
[13:55:03] <kba> Anyway, I will create an array of objects with an id and tag name and index the id, thanks!
[13:57:34] <kba> Right now, I'm just creating my collection by inserting into it. I'd prefer not having to check whether it exists first and then create an index. Can I do that somehow?
[14:04:08] <cheeser> that's what I do, personally. if the index name already exists, it's generally a no-op.
[14:04:36] <cheeser> if it's a *new* index though on large sets of existing data, you'll either want to create that index in the background or build it in during some off time
[14:05:43] <kba> That won't be a problem, there should be no data before the program is run (and the index created)
[14:07:01] <kba> even though that wouldn't be the end of the world -- almost all interaction with the program is through a websocket
[14:50:29] <kba> my collection now has an array of tags [{ label: "foo", version: 1}, {label: "bar", version: 42}, ... ]. I've added indices using db.tags.ensureIndex({"tags.tag": 1, "tags.version": 1});
[14:51:26] <kba> However, the version does not seem to be unique. It's possible to have {label: "quux", version: 1} and {label: "baz", version: 1} in the same array
[14:52:20] <kba> What can I do to prevent this? I want version to be unique -- no two tags for the same version
[15:00:35] <kba> if an object already exists with the same label, can I overwrite the object with the new version? e.g. my collection contains ("foo", 5). Now I insert ("foo, 8"), so ("foo", 5) gets overwritten.
[15:05:31] <GothAlice> kba: AFIK your requirements, here, can not be fulfilled using an embedded array of sub-documents without adding race conditions.
[15:06:23] <GothAlice> I.e. looking something up, then deciding how to update. Between those two operations, other changes can sneak in and invalidate the initial lookup without the subsequent update realizing.
[15:06:52] <GothAlice> (Then it's a simple matter of a unique index and straight-forward inserts/updates/upserts.)
[15:08:36] <kba> my collection has the structure { id: String, tags: [ { label: String, version: Integer }, ... ] }.
[15:08:56] <kba> Do you suggest I create a collection that is just an array containing { id, label, version }?
[15:10:55] <GothAlice> That looks suspiciously like Mongoose. :'( But yes, that would seem to fulfill your uniqueness requirement. (It's typically _id, not id, in MongoDB, BTW.)
[15:12:39] <GothAlice> Now, is your original record example _just_ and ID and set of versioned tags? That's it?
[15:14:42] <GothAlice> kba: Curious. Can the same label be present under two different versions for the same ID?
[15:16:41] <GothAlice> This is the first time I've run into a 2-way uniqueness constraint like this. See, because _id is itself automatically uniquely indexed, often you can create "compound" _id values to contain the fields which must be unique *together*, but in your case things can't be quite as clear-cut.
[15:18:32] <kba> What I'm doing -- if you're curious -- is that I have a bunch of documents in my database including the document's history (i.e. all its versions)
[15:19:03] <GothAlice> So yes, {_id: <ignored>, identifier: "…", label: "…", version: …} will need to be your actual structure, with a unique index for (identifier, label) and another for (identifier, version), where "identifier", not "_id" points at the versioned document reference. That shouldn't really be a string, BTW, but should probably be an ObjectId.
[15:20:05] <GothAlice> (That's one of the reasons why I suspected Mongoose use. That DB adapter lib often messes treats ObjectIds just as strings… causes all sorts of headaches.)
[15:20:39] <kba> Ah yes, I actually made a pull request to mongoose just about that a couple years ago
[15:22:06] <GothAlice> In your case, since the versioned document reference and two unique combinations are the "primary key" — complicated, but seemingly necessary — the _id itself can be pretty much ignored. It provides a convenient way to load up specific document-label-version combinations directly, though.
[15:24:03] <Naeblis> Hi, is there some way to capture regex group in mongo aggregation? Say, I have a string "FOO XXX" where X values can vary, and I want to group them based on the pattern /FOO (\w+)/
[15:26:56] <kba> My document collection's _id doesn't follow the convention, so when I try to do mongodb.ObjectId(docId), I get: Error: Argument passed in must be a single String of 12 bytes or a string of 24 hex characters
[15:27:25] <kba> Is there a remedy? I'm doing: db.tags.insert({ webstrateId: mongodb.ObjectId(webstrateId), label, version })
[15:28:51] <GothAlice> kba: If it's not actually an ObjectId, i.e. it's "some-opaque-identifier", a number, or other non-OID identifier, because you've explicitly inserted records with custom _id's, you can't just arbitrarily cast them. If it's already actually an ObjectId, no need to cast it, either.
[15:29:21] <GothAlice> Naeblis: Not from what I can tell. You can $substr, though that seems to be the most powerful string manipulation function available during projection: https://docs.mongodb.com/manual/reference/operator/aggregation-string/
[15:30:03] <kba> Ah, they're not actully ObjectIds, I suppose. I just figured all _ids automatically were ObjectIds. Never mind then
[15:30:23] <GothAlice> kba: They are if you don't define them yourself when inserting, under every connection driver I'm aware of.
[15:33:19] <GothAlice> Naeblis: One might recommend extracting the values application-side for storage alongside the larger string, that way you can actually query on it, project just that value, group on it, etc, instead of storing a single compact value that can't easily be queried the way you are asking.
[15:34:05] <Naeblis> but this is something that nobody really gave a thought to while designing the schema and is now needed :(
[15:34:44] <Naeblis> basically the "location" of a store is stored in a format like "Country Code City Code Store Address" as a single string
[15:35:43] <GothAlice> We store addresses as embedded documents, a la {…, location: {country: "…", city: "…", address: "…"}} — multi-line addresses (e.g. office number on one line, street address on another) is just a string with a \n in it.
[15:37:26] <GothAlice> If so, this might actually work for you until you get a chance to… unpack those: https://docs.mongodb.com/manual/reference/operator/aggregation/substr/
[15:37:52] <GothAlice> Yeah, since it's a prefix, and of fixed length, you can get away with $substr.
[15:39:08] <GothAlice> Your initial example had a "FOO" prefix, which… didn't inspire confidence as to $substr use. ;)
[15:42:01] <kba> Just to be more of a pain: Now if I insert something with a label/version that already exists, it doesn't work, which is good, I suppose
[15:42:54] <kba> but that's assuming that I'm updating the version and not the label. I won't know which one is changing
[15:46:57] <GothAlice> kba: You probably want an upsert. "Update an existing value matching a query, or insert it if not there."
[15:47:41] <GothAlice> kba: https://docs.mongodb.com/manual/reference/method/db.collection.update/#upsert-option
[15:48:42] <GothAlice> That… doesn't make sense to me. Could you provide an example using the mongo REPL shell syntax?
[15:48:43] <kba> now I'm trying db.tags.update({ $and: [ webstrateId, { $or: [ label, version ] } ] }, { upsert: true })
[15:50:35] <kba> db.tags.update({ "webstrateId": webstrateId, "label": label, "version": version }, { upsert: true })
[15:50:39] <GothAlice> So, if db.tags.update({ $and: [ webstrateId, { $or: [ label, version ] } ] }, { upsert: true }) is what you're trying, that's not an update at all.
[15:50:55] <Derick> kba: you only have query, and options. You need query, update-spec, and options.
[15:50:57] <GothAlice> An update takes two arguments, the query, and the update operations, and optional "options" as a third argument.
[15:51:46] <GothAlice> This is why I ask for _complete_ examples rather than "it's this, oh, but with that other thing too" — no way to tell if you're actually doing things correctly. ;)
[15:52:54] <kba> But how would I create the query then? I thought something like db.tags.update({ $and: [ webstrateId, { $or: [ label, version ] } ] }, { label, version }, { upsert: true }) would make sense
[15:53:18] <GothAlice> db.tags.update({webstrateId: "…", $or: [label: "…", version: …]}]}, {$set: {label: "…", version: "…"}}, {upsert: true })
[15:53:39] <GothAlice> First, the top-level $and is unnecessary, because the top level is _naturally_ an $and of the operations contained therein.
[15:54:44] <GothAlice> Lastly, I have too many closing braces and stuff in there because I forgot to trim those out when removing the $and. Apologies.
[15:57:12] <GothAlice> However, this might not quite do what you expect, and might fail the unique constraints. You're setting both label and version, but the $or on label + version might find more than one match. Something with a different label but requested version, or with a different version for that same label.
[15:59:37] <kba> So if I'm working with a document and I'm trying to tag it, I'd expect the version I'm working with to recieve the tag
[16:00:12] <kba> If I'm trying to set a label and version, and those two are unique, it wouldn't matter
[16:02:28] <kba> So I'm inserting (id, "foo", 5), and (id, "foo", 2) and (id, "baz", 5) already exist, I'd want both of those to be removed in favor of (id, "foo", 5)
[16:03:22] <GothAlice> There may be some XY problem solving going on; you're versioning (storing a history of changes to) a value related to another document with this, correct? Do the version numbers not increment over time? I'm not seeing a complete enough picture.
[16:04:43] <kba> I can freely move back and forth between documents by just starting out from scratch and applying the operations from the operation log until I reach the desired version
[16:05:14] <kba> now, instead of saying "Give me version 14532 of documentA", I'd rather be able to say "Give me Draft 2 (tag) of documentA"
[16:06:16] <kba> So, now I want to be able to freely move in the history of the document and say "This version should be called Draft 2" or something
[16:06:54] <kba> Now, Draft 2 obviously has to be unique (the label), because otherwise if somebody asks for Draft 2, there may be multiple versions for that
[16:07:36] <GothAlice> So, your operations are: tag a version with a specific label, or remove a tag from a specific version, right?
[16:08:13] <kba> Yes, and in case somebody tries to tag version X with a label that already exists on Y, I just want to remove the tag from label Y and apply it to X (move the tag)
[16:09:33] <GothAlice> db.tags.update({webstrateId: "…", version: …}, {$set: {label: "…"}}, {upsert: true}) — tag a version
[16:12:27] <GothAlice> db.tags.update({webstrateId: "…", label: "…"}, {$set: {version: "…"}}, {upsert: true}) — change the version a tag points at
[16:13:27] <GothAlice> In the situations that arise causing a unique constraint failure your application can then decide how to proceed, in the "simple case", here.
[16:13:59] <kba> yeah, okay, I was hoping the first two operations could be one operation. Otherwise, I always have to see if the tag exists in the database first and then update, and that won't be atomic
[16:14:02] <GothAlice> Note that the "change" operation will also actually add a tag for that version if missing. (Upsert use.)
[16:14:24] <GothAlice> Don't check first. Optimistically try to set the label for a version, or the version for a label, and handle the unique constraint failure.
[16:16:28] <GothAlice> I might actually go with the "set the version for a label" path, then if the constraint fails, notify the user that that version already has a label, letting them decide what to do.
[16:18:01] <GothAlice> (The other path makes less sense from an end-user perspective: "set the label for a version", the constraint failure would be "the label is already set", which is less useful for end-user reporting of conflict since the user is presumably wanting to set _or_ reset the label.)
[16:18:39] <GothAlice> But you can see how each approach reports on a very slightly different error state. Up to you to pick which best fits your application. :)
[16:19:02] <kba> Hm, yeah, but on the other hand, it's easy for the user to verify how it works. They just see it as a property
[16:19:20] <kba> The document evolves and at any time they can either set/see the webstrate.tag variable
[16:19:52] <kba> Surprisingly often, the problem seems difficult to solve simply because is hasn't been properly defined
[16:20:04] <GothAlice> If you want to always set the label for a version, regardless of that version already being labelled, or the label already being used on another version, you can preemptively delete the label for the version and the version association with the new label, then perform the single operation to add it back.
[16:20:51] <kba> Yeah, but right now I'm just doing the set the label for the version, if that fails, set the version for the label
[16:21:20] <GothAlice> I.e. db.tags.delete_many({webstrateId: "…", $or: [{label: "…"}, {version: …}]}) first
[16:32:48] <Clemlar> Guys, is there a way to connect to mongo with the latest mongo-ruby-driver when using a self signed certificate? All I see is an option to use SSL
[16:35:50] <Derick> Clemlar: I don't think there is, I would recommend to file a ticket at http://jira.mongodb.org.browse/RUBY
[16:37:05] <Clemlar> thank @Derick I think someone already beat me to it, with a similar issue and was told this is expected behaviour: https://jira.mongodb.org/browse/RUBY-810
[16:38:06] <Clemlar> I may open a new ticket and ask if this can be reviewed. I'm pretty shocked that the driver doesn't support an option for invalid certificates :-/
[16:42:55] <Clemlar> hah, just checked the source for the driver, and I see this: set_cert_verification(context, options) unless options[:ssl_verify] == false
[16:43:50] <Derick> so ssl_verify as an option - it's not documented though, so certainly still worth a ticket
[18:04:20] <MacWinner> HI, i'm trying to understand a bunch of bandwith utilization that seems to be from my nodejs mongodb connection pool to a replica set.. My replica set has 4 members and 1 arbiter.. There are about 90 connections to each of the 4 replica set members. on an idle server, I'm still using 200kpbs bandwidth and when I do a packet capture, it looks like a lot of replical set polling traffic about who ismaster and the current replica set state..
[18:04:50] <MacWinner> should I be tuning something here? Seems likea waste to have a consistent 200k used for this... I feel like maybe my mongoose connections are misconfigured
[18:07:34] <GothAlice> Seconded, even. However, all drivers maintain a pool of connections and in a replica set need to maintain knowledge of who is primary, who is not available, etc.
[18:08:27] <GothAlice> To try to reduce that "who's who" traffic, we run a mongos in front of a replica set, and connect our application to the mongos.
[18:08:50] <GothAlice> It's potentially less reliable (in case the mongos service goes away), but greatly simplified life otherwise.
[18:09:18] <MacWinner> ahh.. i guess I should put in mongos into the artchitecture now since we wille eventually need to shard
[18:09:34] <GothAlice> The other reason I started with a mongos on this project; eventual sharding. :)
[18:10:11] <GothAlice> It's been rock-solid in this configuration so far, running this way since Jan 1st.
[18:11:04] <MacWinner> so if I move to mongos, then each of the 90 connects points to local mongos instance.. but they don't do any replicaset chattiness.. only mongos talks with the members? so theoreticall it should drastically reduce this traffic, right?
[18:11:21] <GothAlice> Remember, also, that you can have multiple mongos. If, for example, you run VMs on your own hardware, you can run the mongos at the dom0 level, then have the VM application workers connect to the physical-server-shared mongos router, one mongos per physical host instead of each VM tracking the entire replica set pool.
[18:11:52] <GothAlice> The amount of benefit will increase the more app worker processes you have connecting to one mongos.
[18:12:06] <GothAlice> (Since mongos itself needs to perform that chatter, but this helps reduce the "amplification" of chatter the more app workers you have.)
[18:14:50] <StephenLynx> and not that it doesn't have its uses, but it isn't the be all end all either.
[18:15:36] <StephenLynx> a good reason to use procedural code when possible is how it can be more performant.
[18:16:01] <GothAlice> Procedural code to manipulate and query the data is one thing, but the data itself (BSON) represents a rich, potentially structured object.
[18:17:29] <GothAlice> For example: {name: "home", path: "/home", title: "My Homepage", content: [{reference: '/include/header'}, {content: '<p>…</p>'}, {reference: '/include/footer'}]} — this represents a "Page" object with "Blocks" as contents.
[18:18:35] <MacWinner> any tips on a good tutorial on introducing mongos into a simple replica set architecture?
[18:19:10] <MacWinner> not looking to do a full sharded cluster just yet.. looking for minimal reliably introduce for mongos
[18:20:17] <GothAlice> No need to enable sharding on any of the databases/collections, but the mongod's need to be aware they'll be speaking through a mongos.
[18:21:56] <MacWinner> so do you have your app connect to multiple mongos servers for failover? or do you front-end multiple mongos servers with something like HAProxy?
[18:23:09] <GothAlice> MacWinner: As I described earlier, I have one mongos per physical host, and each app worker (Python process in my case) on that physical machine connects to it with its own connection pool. If the mongos fails on the app node, the app node has failed, so the potential for reduced reliability / single point of failure doesn't matter.
[18:23:56] <GothAlice> If you have VMs, run the mongos (if possible) at the top level (dom0) and have the processes in the VMs on that host connect through to the dom0's mongos for maximum efficiency.
[18:24:01] <MacWinner> ahh .. ok.. so the entire app node would be taken out of the cluster if the mongos for that node failed
[18:25:05] <GothAlice> I have never attempted to proxy anything other than HTTP through that. No idea how effective it would be.
[19:21:11] <ars3n> Hey guys, I am recording values by second and need a way to return the average by minute. Is there a query for this?
[20:05:20] <Doyle> Is there a way to fix db.currentOp() error Assertion: 16686:error converting js type to Utf8Value?
[20:15:39] <Jasper> Does anybody know why adding .explain() to a query in the MongoDB shell makes it so much slower?
[20:15:48] <Jasper> (Context: we're trying to debug why db.jobs.find({ status: 1 }) is so much slower than db.jobs.find({ status: { $in: [1] } }) -- and trying to explain both of them is failing)
[20:30:23] <rodd> Hi has anyone here worked with geo/2d indexes, comparing user's lat/long values with countries lat/long ranges?
[22:09:48] <gkze> haven't found a way to achieve the effect of db.printSlaveReplicationInfo() from the python driver