Log file Viewer
#mongodb logs for Tuesday the 23rd of February, 2016
[00:26:24] <jiffe> if I flatten a maching with mongoc running (in a 3 mongoc node setup), it should resync from one of the other nodes right?
[00:43:06] <GothAlice> Doyle: I was going to suggest iotop, but apparently MongoDB doesn't name its threads, so that's not useful. :(
[00:48:32] <Doyle> inotifywatch -r -t 60 /dbpath shows a total that is far from the numbers I see from iostat -xmt /dbpath
[00:50:27] <GothAlice> Depends on how the kernel decides to multiplex filesystem events, i.e. if it waits for all receivers.
[00:53:04] <GothAlice> Yeah, I suspect inotifywatch is giving you the perceived operations, not actual operations.
[00:53:32] <GothAlice> I.e. it can't tell if the kernel actually had to hit disk to accomplish the operation.
[00:55:05] <GothAlice> cheeser: Hmm. That's actually something I hadn't considered. Do you know of any work to get MongoDB running under L4?
[00:58:17] <GothAlice> L4 is generally a microkernel architecture. Specific implementations take different directions, but seL4 went for the "formally verified in every operation" approach. It's the only 100% guaranteed secure, guaranteed bug-free kernel in the world.
[00:58:56] <GothAlice> Fun fact: its the kernel in use on the Security Enclave in that iPhone currently giving the FBI fits. ;P
[01:00:10] <GothAlice> "The functional correctness proof states that, if the proof assumptions are met, the seL4 kernel implementation has no deviations from its specification. The security proofs state that if the kernel is configured according to the proof assumptions and further hardware assumptions are met, this specification (and with it the seL4 kernel implementation) enforces a number of strong security properties: integrity, confidentiality, and
[01:02:52] <GothAlice> cheeser: L4 is more widely deployed, generally, than Linux, BSD (+Mac), and Windows combined. (It's in pretty much everything, from WiFi basebands and every iPhone with a Security Enclave, to cars and industrial machinery.)
[01:17:26] <GothAlice> But, ah, I want more trusted OS components and less inefficiency through indirection.
[01:27:04] <GothAlice> uuanton: Step 1 in both methods of removing a node involve shutting that node down.
[01:35:32] <uuanton> GothAlice whenever I remove a member from replica set it elect new primary using both methods ?
[01:36:08] <GothAlice> Yes; changes in the electorate call for an election. Sometimes the same server might reclaim primary, sometimes not.
[01:58:51] <joannac> GothAlice: that's fixed in 3.0 onwards, I think. Removal can happen without always requiring an election, under certain conditions
[02:02:11] <GothAlice> Indeed, dependence on such factors is sub-optimal; every client driver I am aware of can handle failover if configured with a set name.
[04:37:16] <gzn> http://pastebin.com/mP64aExd <--- interview question having trouble with. My background Lamp Dev. Initial ideas -> db sharding && elastic search. Never used nosql db before wondering if I could get some help here
[06:21:12] <WaRLoRD786> Hey guys, I'm testing a 3 sharded mongo instance with only an _id field and one other value which is a 32 character value. Using a single mongoimport process I'm able to load about 19k rec/sec. But on the query side (querying the mongos process), i'm only able to achieve about 1400 rec/sec when querying on the _id field. This seems off? Or does this seem right with a sharded mongo cluster?
[08:11:17] <partycoder> then you would like to use something like publish/subscribe each time you insert a document
[11:25:15] <Lope> lets say I have a document {_id:1, stuff: {a:{one:'blah'}}} now I want to update stuff.a. I want to add these fields {two:'blah', three:'blah'}. How can I do that?
[11:26:13] <Lope> When I tried an update with {$set:{stuff.a.two:'blah', stuff.a.three:'blah'}} it destroyed stuff.a.one ?
[11:41:57] <kalx> Hi All. Is there anyway to use $bitsAllSet in the other direction, meaning check if field_value & provided_value == field_value?
[12:03:51] <zerivael> Hi, I have a strange problem with old mongodb from 2.4 line. I have a production installation of 2.4.9 and it runs quite fast but when we try to copy it to another machine and restore databases from dump it runs almost 100x slower on querying, inserts are fast (mongostat). Funny thing is that we managed once to get the same speed as in production environment by upgrading mongo to 2.4.14. But when we tried to repeat the preocess it was still slow. Can anybo
[15:37:50] <zerivael> turned out dump was done using older mongo 2.0.9, i'm trying to get backup made with newer version and than restore and see what will happen
[16:48:03] <Pirschie> I keep getting "The application was unable to start correctly (0xc000007b). Click OK to close the application. "
[16:55:13] <GothAlice> Since I don't use Windows, I can only really point you at https://docs.mongodb.org/manual/tutorial/install-mongodb-on-windows/ Pirschie. AFIK "reinstalling .NET, C++ red." etc. is unnecessary as the MongoDB MSI installer bundles its dependencies.
[17:03:51] <GothAlice> Pirschie: Alas, then you are beyond my ability to assist. As StephenLynx so ineloquently puts, Windows isn't typically considered a "serious" service hosting platform (given the lack of POSIX or overall architecture sanity). I'd recommend getting VirtualBox or another VM system and running MongoDB under a Linux VM.
[17:06:05] <GothAlice> Pirschie: That is worrisome; is it a memory issue? Modern virtualization is paravirtualization, i.e. the CPU doesn't really make a distinction between host and VM instructions, making overhead extremely small.
[17:06:22] <StephenLynx> you can either stop working with windows or you can be a serious developer.
[17:07:24] <Pirschie> it's just, everything is going super slow nowadays and my ping dropped by half in every game since a few months
[17:07:38] <GothAlice> Pirschie: Right. Then I'd double, nay, quadruple check your OS is actually running in 64-bit mode, and that your BIOS has VT-x extensions enabled. (Older BIOSes might have that set to off by default.)
[17:09:53] <GothAlice> Wagh. https://support.microsoft.com/en-us/kb/827218 < clearly, Microsoft isn't particularly good at consistency or simplicity even when it comes to the simple question "am I actually running 64-bit".
[17:12:02] <Pirschie> msiexec.exe /q /i mongodb-win32-x86_64-2008plus-ssl-3.2.3-signed.msi ^ INSTALLLOCATION="C:\mongodb" ^ ADDLOCAL="all"
[17:13:42] <GothAlice> And I'd expect nothing to happen if your command prompt isn't running as Administrator.
[17:14:12] <Pirschie> just the line "msiexec.exe /q /i mongodb-win32-x86_64-2008plus-ssl-3.2.3-signed.msi" doesnt do anything
[18:08:25] <synthmeat> freaky. i rebooted server and mongod failed running with "No map found at top level of YAML config"
[18:12:56] <GothAlice> synthmeat: Run your config through yamllint.com, then remove the superfluous leading "---" line.
[18:15:10] <GothAlice> The amount of indentation is unusually wide. YAML is whitespace sensitive, except in specific wrapped sections, so its something that yamllint.com changes that might have an impact.
[18:17:56] <synthmeat> short of trying to rebuild mongodb (which is horrible experience on low-mem instances), i'm hitting a wall
[18:24:50] <Derick> synthmeat: you sure you don't have a BOF at the start? run: head -n 1 /etc/mongodb.conf | od -t x1
[18:31:55] <GothAlice> synthmeat: The lint site I linked gave modified output. It was my assumption that testing its output (with two-space indentation) would be a reasonable diagnostic step. Apologies if I was unclear on that.
[18:32:56] <synthmeat> GothAlice: no need for apologies, and yes, i've tried that. even making another file localy and uploading and various other things
[18:43:44] <kalx> Hi All. Is there anyway to use $bitsAllSet in the other direction, meaning check if field_value & provided_value == field_value?
[18:44:44] <GothAlice> synthmeat: Ah, you're… actually not entirely sure what you've got there. Statically linked, with symbols?
[18:45:10] <synthmeat> dunno really, i'm using this makefile ftp://ftp.wart.ru/pub/download/mongodb3.tar.gz
[18:45:48] <kalx> The only workaround I've been able to see is to use $where with a custom js function
[18:47:41] <GothAlice> synthmeat: So getting an order of magnitude larger binary size is likely a PEBKAC issue. ;P
[18:48:36] <GothAlice> I.e. there's something you're suppling to ./configure or friends causing that.
[18:49:43] <synthmeat> i'm having numerous other issues with this though, everything's kinda built to work with linux nowadays
[18:53:54] <GothAlice> kalx: Isn't https://docs.mongodb.org/manual/reference/operator/query/bitsAllClear/#op._S_bitsAllClear the inverse?
[18:55:21] <Gevox> Hello, I'm working on some HA architecure research and taking a deeper look into the MongoDB HA Architecure'. A question raised while reading docs about the replication model. What happens if the primary node have failed while the secondary are still reading the oplog (i.e they are not done of a complete sync)?
[18:56:21] <GothAlice> Gevox: Then they're stale, and election will take place only between nodes with the latest operation applied.
[18:57:00] <GothAlice> Whatever the "latest" is according to the cluster, that is; if all nodes are stale, the least stale one will win if it can.
[18:57:05] <Gevox> GothAlice, so there is a high potential that an incomplete version of the data goes live.
[18:58:10] <Gevox> Is there any supporting documents by the Mongo architectes demonstrating the pros and cons of the currently used architecure, why did they prefereed that one over others etc?
[18:59:18] <Gevox> Something from the labs, because the docs are full of cleared statement about what is actually going on, not why.
[18:59:58] <kalx> GothAlice: That would essentially be field_value & provided_value = 0. Not quite the same. I supposed I could take the inverse of my provided_value and then look for the !$bitsAnySet
[19:00:01] <GothAlice> But initial sync is equivalent to a RAID rebuild; if the process is interrupted, it's bad. How bad would vary from instance to instance. A clean set of nodes would initial sync almost instantly (no operations to apply) after which you could load it with data. This would be the "safest" way to set up a replicated cluster. Migrating an existing large dataset to a replicated cluster is… a different bucket of bolts.
[19:01:06] <GothAlice> Gevox: With WiredTiger being the default back-end, now, I'd check out the WiredTiger mailing list traffic.
[19:03:50] <uuanton> seems like when you install using yum /lib/systemd/system/mongod.service missing
[19:08:38] <GothAlice> As well as JIRA tickets relating to architectural decisions, such as https://jira.mongodb.org/browse/SERVER-14561 (though this particular example contains less discussion and more links to Github change sets.)
[20:27:31] <djMax> so I have a 3 member RS. 2 members have died, and I am trying to bring up new members while they try and figure out what blew up. Of course, I can't add them to the rs because there's only 1 member... Is there anything I can do to bring the RS back up?
[20:33:50] <Doyle> djMax, for quickest recovery, disengage the RS from your application layer, and kick off an initial sync with a new secondary
[20:34:21] <Doyle> Once that's complete and comes up as secondary, start another initial sync to bring up the second secondary
[20:35:21] <Doyle> djMax, with a single member you're in a high risk situation for complete failure. Priority one is typically the recovery of a secondary
[20:35:52] <djMax> agree. Just wanted to make sure reducing to one while no apps connected wouldn't blow up on principle
[20:35:52] <Doyle> As long as the single member comes up as primary, you can do an init-sync... as far as blowing shit up, I'm not sure in what capacity you mean.
[20:36:50] <Doyle> As far as apps go, they'll stall if you block their access to that RS. I only recommend that to help expedite the recovery of a secondary
[20:37:29] <Doyle> If the data to sync isn't much, you could also just leave it online. I think in terms of multiple TB/rs...
[20:39:00] <zeioth> using the C# driver, InsertMany() will continue inserting even if one document has a DuplicateKey?
[21:59:40] <kalx> Hi all. How can I figure out why mongo CPU would be jumping to 100% (across all 8 cores on a test machine) for a query. The query runs on an index, but the last condition requires a scan to look for matching values. However, we're only talking about 200k records here.
[22:05:00] <Doyle> It looks like there's no configuration for journal path. Do people typically add a simlink when mounting dedicated volume for it?
[22:52:57] <rook2pawn> i want to use jsonschema. Can i just remove/not use mongoose if I am going to be using jsonschema?
[22:56:00] <GothAlice> rook2pawn: Current advice seems to run along the lines of a) mongoose is terrible, and b) MongoDB now has document validation built-in, so requiring a DAO/ODM for that is… excessive.
[22:58:03] <GothAlice> MongoDB document validation is infinitely more expressive than typical field → type mapping.
[23:00:26] <GothAlice> Yeah, Derick, even I'm considering pumping out some "your ODM is now deprecated" inflammatory posts. ;P
[23:01:54] <rook2pawn> @GothAlice: The reason I like json-schema is that you can throw these schema's everywhere, not just for database stuff. Does using json schema instead of the built-in validator seem fine?
[23:02:32] <GothAlice> It's a question of integrity. Validation schemas within the database will better guarantee integrity.
[23:02:46] <GothAlice> If your DAO is required to touch your data, touching your data with things like the mongo shell will bypass your rules.
[23:02:53] <rook2pawn> I have json schema in my API routes, in form submissions, everywhere. This way i define them once, in a single way, taht is agnostic of a database type
[23:03:21] <GothAlice> rook2pawn: Then an obvious solution would seem to involve automatic translation of your JSON schema to a MongoDB validation schema.
[23:31:46] <kalx> I can reproduce it easily so source is known. Wondering if there's any tools/options to enable better logging to figure out exactly why the cpu spike occurs.
[23:32:14] <kalx> The query plan uses an index but requires a scan in the end for the final matching condition (so execution plan is a filter + idxscan). Running 30 of these queries in parallel pushes cpu to 100% across 8 cores for ~15 seconds.
[23:32:46] <GothAlice> kalx: It'd be important to identify the source for the "load". Under "top" is the CPU time for mongod spent mostly in "user", "system", or "wait"?
[23:35:03] <GothAlice> I.e. under "top" I get a third line like: "%Cpu(s): 0.0 us, 0.0 sy, 0.0 ni,100.0 id, 0.0 wa, 0.0 hi, 0.0 si, 0.0 st" — indicating 0% user, 0% system, 0% nice'd, 100% idle, 0% wait, 0% hardware interrupt, 0% software interrupt, 0% stolen by hypervisor.
[23:37:47] <GothAlice> Hmm. (Confirmed: Linux) I can assist with BSD/Mac process sampling/spindumping, but not so much on the Linux side, for capturing what mongod thinks its doing.
[23:40:08] <GothAlice> kalx: Can you gist/pastebin the .explain() of the query you are running, the query, and an example document? You mentioned earlier you had many such documents returned in response to this query.
[23:44:39] <bros> I currently have 1 tall MongoDB server. I read you can shard on the same machine to improve performance due to multiple CPU cores, is this correct?
[23:55:58] <GothAlice> cerjam: Cool beans, so it's probably not dying due to failing allocation. Memory utilization is predominantly virtual, with memory-mapped on-disk files and some "real" working memory for query space. Too-big queries should die gracefully, though. (If you are doing an aggregate query, investigate the allowDiskUse option of https://docs.mongodb.org/manual/reference/method/db.collection.aggregate/#definition )
[23:56:38] <GothAlice> cerjam: Uh, did you notice the "WARNING: You are running in OpenVZ. This is known to be broken!!!" note in there?
[23:58:47] <bros> StephenLynx: I have it on a 4 CPU machine now. Is it a step back to move it to a dedicated 2?
[23:59:29] <StephenLynx> depends, if your usage is IO intensive, then moving to a 4 core server will help.
[23:59:43] <StephenLynx> if its CPU intensive AND your current sofware makes good usage of all the cores, then no.
[23:59:57] <GothAlice> Running three copies on the same physical hardware, in VMs, protects you from nothing.