Log file Viewer

[08:24:45] <s10g> I am using mongodump with --archive /mongodumps/mongodump_20190918, and getting this error: positional arguments not allowed: [/mongodumps/mongodump_20190918]. What does this mean?

[13:42:19] <GothAlice> s10g: mongodump --help # syntax is mongodump <options> — no positional arguments are used. --archive=/path/to/archive is a single switch, which includes the path to archive into.

[13:42:30] <GothAlice> Apparently MongoDB doesn’t use GNU CLI parsing standards. ;P

[13:43:27] <s10g> GothAlice: Aye I found out that I had to do it with = in my version of mongodump/restore

[13:43:43] <GothAlice> “In my version”? In all.

[13:57:10] <s10g> ahh i probably misunderstood the bottom answer at https://stackoverflow.com/questions/37385729/mongodump-3-2-1-positional-arguments-not-allowed

[13:57:24] <s10g> but it worked nicely after sprinkling some = on it

[16:19:30] <blottoface> I have a 4 node MongoDB 3.6 cluster running on ubuntu 18.04, each node has 32GB RAM. I have added LimitNOFILE=128000 and a few other ulimit settings (https://pastebin.com/99C8Nz22) to my systemd startup config.

[16:19:50] <blottoface> My mongo cluster seems to exhaust it's connections at about 4800 active connections.

[16:20:22] <GothAlice> Exhaust as in “new connections stall or error”, or..?

[16:20:44] <blottoface> Exhaust, as in "new connections stall and/or error"

[16:21:25] <GothAlice> Exhausting TCP connections is a thing if you aren’t using connection pooling. The kernel itself can run out of outbound port numbers to allocate. (A problem I ran into attempting to test C10K on my HTTP/1.1 server implementation; 10,000 connections at a time exhausts *all* port numbers in around 6 seconds!)

[16:22:10] <blottoface> Eventually in the logs I see errors in the logs like this. "pthread_create failed: Resource temporarily unavailable"

[16:22:30] <blottoface> And my primary dies with a stacktrace.

[16:22:42] <blottoface> Hmm.. I haven't investigated connection pooling.

[16:22:48] <GothAlice> That’s… different. That’s running out of literal threads to allocate. :o

[16:23:30] <GothAlice> That pivots the concern from “not using connection pooling” to “a conneciton pool that never re-uses a connection, constantly allocating new ones”.

[16:24:51] <blottoface> I'm concerned about that for sure. There is a scala driver that's connecting and my bet is it's not re-using connections.

[16:25:53] <blottoface> I'm also wondering why I'm butting up against ~4800 connections. As I understand it, we should be able to handle about 80% of the LimitNOFILE number.

[16:26:31] <blottoface> pastebin to my the end of a logfile as it stacktraces: https://pastebin.com/uNErEmUp

[16:33:42] <GothAlice> (Domains involving ‘paste’, ‘bin’, ‘fiddle’, and literally anything that offers a POST or PUT endpoint for publicly accessible file upload without authenticaiton, …)

[16:44:36] <blottoface> GothAlice, do you see any identifiable/sensitive information in that pastebin?

[16:45:29] <GothAlice> I thought my messages were relatively clear: I can see nothing on that pastebin. I can not see pastebin. Because malware and corporate firewall policy.

[17:01:19] <blottoface> Oh, I see. I was not reading it that way.

[19:51:46] <blottoface> Anyone have a good sample /etc/sysctl.con and systemd start script for a three node cluster of decent size (32GB RAM 8core)? DB is probably on the order of 50GB used disk collections taking up about 160,000,000 records.

[19:55:23] <blottoface> *sysctl.conf*

[20:26:54] <GothAlice> Y’know, the only sysctl I’ve ever needed to tweak (“for real”) was the fopen limit, and that’s a ulimit, not sysctl. :^P The only actual value I’ve tweaked with a sysctl is the TCP_CLOSE_AWAIT timeout in order to perform 10,000 concurrency abusive testing of my web server project, once.

[20:28:44] <GothAlice> https://gist.github.com/amcgregor/4fb7052ce3166e2612ab#hosting a sysctl that would be worthwhile setting is the one to disable THP and Huge Pages in general.

[20:28:47] <GothAlice> blottoface: ^

[20:29:15] <GothAlice> Ref the https://gist.github.com/amcgregor/4fb7052ce3166e2612ab#whats-the-deal-with-this-transparent-huge-page-thing-i-keep-seeing-warnings-about section.