Log file Viewer
#mongodb logs for Tuesday the 8th of November, 2016
[02:24:34] <cornfeedhobo> cheeser: so, not to sound negative, but are the only one answering questions?
[02:55:35] <ghadi> Hi folks. Are there any successful prod deployments of Mongo on btrfs filesystems here?
[04:59:17] <energizer> I have a web scraper that checks a bunch of pages, case numbers A1-A1,000,000, B1-B1000000, ... . If there's a page for that case number, save its data in my mongodb. Otherwise, move on to the next case number.
[04:59:17] <energizer> If I stop the program in the middle, i'd like it to be able to pick up where it left off, even if the last 100 cases it checked were empty, and therefore aren't in the db.
[04:59:17] <energizer> How should I keep track of the checked case numbers? Insert empty documents with just the case number into the collection? Another collection to store just the latest one tested for each letter prefix?
[08:45:09] <deever> kali: so i can't directly upgrade 2.6.x to 3.2.x? this is a problem because i'm using freebsd and they don't provide any 3.0.x packages. how long is 2.6.x maintained?
[08:46:59] <kali> deever: i'm not sure. don't assume the required one is the one with the major version are they are not semver. you need to check the release notes, they state very clearly from what version you can bump
[09:26:18] <energizer> Python 2.6.9 is the final security-only source-only maintenance release of the Python 2.6 series. With its release on October 29, 2013, all official support for Python 2.6 has ended. Python 2.6 is no longer being maintained for any purpose.
[10:55:45] <kurushiyama> deever By the Inc? I doubt that. And although I like FreeBSD, I am not too convinced that it is a good choice for running MongoDB in production.
[11:41:49] <kurushiyama> deever Well first of all, you are doomed to xfs (not a bad choice in itself) if you want to stay on a supported (aka tested) file system. Newer versions of ext4 seem to exceed xfs in performance when it comes to MongoDB.
[11:43:08] <kurushiyama> deever Second: You might run into problems, and you really want to be able to call in professional services. I am not sure wether Inc would support FreeBSD, but even if they did, their experience would be much more Linux-based.
[11:45:07] <kurushiyama> deever Third: Depending on what you do and who your customers are, you want to have a certification chain (OS, DBServer, maybe even application server if applicable). Already pretty hard to get for Linux, but afaik basically impossible for FreeBSD.
[11:46:18] <kurushiyama> deever In sum, there is little to no advantage for running it on FreeBSD other than personal preference, but quite some disadvantages in quite common scenarios.
[12:31:38] <marianogg9> hi guys, i've cloned a working mongo 3.0.9 into other instance (aws), running the same hardware, same resources, same app app consuming/querying, now queries take up to ~600 secs in this new instance, when they should (and do) take around 100ms in the working environment
[12:32:46] <marianogg9> single deployment, no replica, no sharding, no tune, just installed and up in both scenarios, same resources, same hardware, same OS (centOS7), same dataset, same app, same queries
[12:33:32] <marianogg9> and load goes up to 9 in the new one, while load is stable at 0.5 in the working one
[12:34:44] <marianogg9> i'm looking for a hint or somewhere I could take information from, like profiling (done that) or debugging logs (done that), query explain (also done), there're no errors at sight, it's just taking much longer
[12:42:34] <kurushiyama> whirlibulf For starters: http://stackoverflow.com/questions/25502410/two-nodes-mongodb-replica-set-without-arbiter/25515526#25515526
[12:43:29] <kurushiyama> whirlibulf So, lets say you have two members. Of two, the _qualified_ majority is 2.
[12:44:35] <kurushiyama> whirlibulf Now, if one node fails, the other one checks for votes, finds out that a qualified majority can not be reached and reverts back to secondary state - eliminating the possibility to write.
[12:48:18] <kurushiyama> whirlibulf Of which the qualified majority would be 3. One node fails, no problem. Two nodes fail, and again a qualified majority can not be reached. So you are paying for 4 nodes, while two nodes failing will revert your replica set to secondary state. With a simple arbiter, which will only cost a fraction of a data bearing node, you could prevent that and actually afford 2 data bearing nodes failing out of your 4.
[12:48:49] <kurushiyama> whirlibulf Or, to put it another way: with 4 nodes, you actually have the same redundancy as with 3 nodes, only that you pay one node more.
[12:51:16] <marianogg9> kurushiyama: clone as in installed a new mongodb server, compressed all /mongo and then uncompressed in the new instance
[12:52:46] <kurushiyama> marianogg9 Have you had a look at the logs? How about file system permissions?
[12:53:41] <marianogg9> kurushiyama: yes I have, this query fails in just one collection, other ones are being done just fine
[12:56:40] <marianogg9> also, run with --repair flag in order to avoid any inconsistency, and logs show a correct data recovering
[12:58:39] <kurushiyama> marianogg9 That is calling mongodump on the new instance with the url of the old instance, and piping the result into mongorestore accordingly configured.
[12:59:57] <kurushiyama> marianogg9 I assume you had both MongoDB instances shutdown during the transfer?
[13:02:49] <marianogg9> kurushiyama: i've done this same process many times before, just copying over /mongo dir; never got an error
[13:37:27] <cheeser> mongodump will take out a lock so that secondary will stop syncing. that lock might yield so it might not be that catastrophic, though.
[13:37:56] <cheeser> if it yields you might get an inconsistent dump (an already dumped document might be updated behind you...)
[14:17:23] <kurushiyama> marianogg9 We are talking of memory mapped files. In order to have a consistent copy, stopping is mandatory. As per the time: it is a wise decision to have ntpd on both the client and the servers. Time deviances can lead to.... interesting behavior. And I could not agree more with cheeser that given the prices of today, running an prod database standalone is a bold choice, in his words. In my book, that is bordering neglegence.
[16:20:33] <qswz> https://www.mongodb.com/blog/post/document-validation-part-2-putting-it-all-together-a-tutorial they wrote { price: {$exists: true}, price: {$type: 1} } .. that's not serious
[16:26:21] <qswz> hmm {email: .., $or: [{ username: {$exists: 0} }, { username: {$regex: /^[\w-]{3,}$/} }]} works
[16:27:17] <qswz> username: {$or: [{$exists: 0}, {$regex: /^[\w-]{3,}$/}]} would have been fun, but i'll deal with tit
[17:24:24] <tenshihan> there _might_ be a back up I put in the request for one months ago but it's not my dept that handles it
[17:50:13] <qswz> var [osinfo, networkInterfaces ] = yield* [sysinfo.osInfo(), sysinfo.networkInterfaces()]
[18:08:07] <marianogg9> kurushiyama: well..i've done a backup by shutting down the db and then restored it in the new instance
[18:45:50] <blizzow> I used mongochef to copy collections between a sharded cluster, and a replica set. Is possible to use mongo-connector to sync them up after the collections have been copied?
[18:46:34] <blizzow> I'd like to use mongo-connector to do the whole sync, but have been unable to get that to work.
[19:44:51] <marianogg9> once a query is received by mongo, it's logged right away or is it logged only if and when it's answered by mongo?
[20:10:20] <ironpig> Should $size work on an array that has keys set, so rather than being 0 - 2 the array has One, Two, Three?
[21:14:51] <GothAlice> cheeser: Heh, without quotes around the attribute names / keys, mine isn't valid JSON, either. ;P
[21:33:24] <cpama> what's changed is that I'm now using a newer version of mongo. upgraded from 3.0.4 to 3.2.10
[21:34:30] <cpama> the find command still works... no errors. But the result set is missing the object id for every document.
[21:54:06] <arduinoob> What is the maximum number of collections in MongoDB with WiredTiger storage engine?
[21:57:39] <arduinoob> I'm not really getting a good idea from the documentation if WiredTiger has no limits on the number of collections. Appears so https://docs.mongodb.com/manual/reference/limits/#Number-of-Namespaces
[22:57:30] <j4f-shredder> hey....I want to store the reference to an object ID in mongo of another collection....I'm using mongoose...which datatype should I use for that field?
[23:15:16] <GothAlice> j4f-shredder: ObjectId. And make absolutely sure Mongoose actually saves real ObjectIds and not strings there.
[23:15:45] <GothAlice> (It has a tendency to treat IDs as hex-encoded strings… which don't compare the same against real IDs, causing many, many problems.)
[23:17:50] <GothAlice> qswz: As to your question, yes, but only sensibly if the whole update operation's purpose is to update that one field.
[23:19:08] <GothAlice> (Note that this is a slow operation; also filter on indexed fields to speed it up.)
[23:20:02] <GothAlice> Object.assign(data, user) returns the whole document, updated with the contents of `data`?
[23:21:33] <GothAlice> "returns the whole document, updated with the contents of `data`" is logically identical to "merging". Same thing.
[23:23:03] <GothAlice> But, by definition, the user must already exist to even run that query. You're filtering on the ID of an existing user.
[23:25:27] <qswz> if I have {_id:1, x:1} and I want to update it with {x:2, y:3} but have {_id:1, x:1, y:3} at the end
[23:46:52] <atbe> I created a mongodb database and pointed it to a a $mydir/data/db directory but the size of that directory has not changed. Running `show dbs` shows that the database is at 75gb. Where is all that data?
[23:48:03] <atbe> Is there some command I need to execute so mongodb can save all that data into the directory? The memory usage is growing to be rather large
[23:48:44] <qswz> https://docs.mongodb.com/v3.2/core/document-validation/ What are all the $type 's available?
[23:49:52] <qswz> https://docs.mongodb.com/manual/reference/operator/query/type/#document-type-available-types
[23:55:45] <atbe> https://usercontent.irccloud-cdn.com/file/GimMMcO5/Screenshot%20from%202016-11-08%2018-37-34.xcf
[23:57:05] <GothAlice> After any change to a document (e.g. inserting a new one, updating an existing one) if there's a validation document (and you didn't explicitly say "don't check" when inserting/updating) all MongoDB does is make sure the document being updated matches the query defined by the validation document.
[23:57:18] <atbe> https://usercontent.irccloud-cdn.com/file/vFDau8Lc/Screenshot%20from%202016-11-08%2018-37-34.jpg
[23:57:30] <GothAlice> You can use everything you can use in a normal query in a validation document; this includes $and, $or, etc.
[23:58:11] <atbe> Okay, that will be useful when I decide to validate documents, but thankfully that is the not the issue I am having at the moment.