Log file Viewer
#mongodb logs for Monday the 7th of October, 2019
[07:46:16] <dandre> GothAlice: Yes got it in console but the script stops from there and I can't get any error consition inside my script
[14:51:10] <syntaxfree> I have this API that has some state to be recalculated everytime a server boots up. And I developed my bits on a digital ocean like scenario. but now Deployment is coming and they want me to have close to stateless servers that can be swiftly torn down and recreated.
[14:53:52] <GothAlice> syntaxfree: It is best practice to make your systems as durable as possible. Central coordination and configuration weakens that durability. An example I have is of running Postgres servers without any physical storage at all.
[14:55:13] <GothAlice> First node spins up, sees no other alive nodes, restores last hourly snapshot from archived backup, then applies per-minute WAL (oplog) logs until it’s caught up. Then it spins up two other nodes. Node spins up, sees alive node, replicates from that. Ultimately: the cluster can have any VM nuked from orbit without warning, it’ll self-heal.
[14:56:11] <GothAlice> RPO (recovery point objective; maximum data loss) of T-1 minute, RTO (recovery time objective; how long to recover from catastrophic failure) of T+15 minutes.
[14:57:20] <GothAlice> My MongoDB-powered primary app at work has a slightly less flagrant RPO/RTO of 1h and 15m, respectively.
[15:00:28] <syntaxfree> I'm not responsible for the systems stuff generally speaking. I'd been working in my digitalocean bubble and having trouble adjusting now.
[15:00:35] <GothAlice> That Postgres setup powered by WAL-e, Gentoo Linux, and a bunch of custom init.d automation. (On Amazon, you get two “local drives” for fast-access non-permanent storage. So on startup my Gentoo nodes logical-block-volume and RAID/MD them, and use them for temporary fast local storage.)
[15:01:21] <GothAlice> https://gist.github.com/amcgregor/4947201 — nice, I do still have an ancient copy of that init.d script.
[15:01:23] <syntaxfree> my API has state. It's an approximate nearest-neighbor structure. Takes a minute or so to build.I'm being pressured to have servers that are as stateless as possible now.
[15:14:20] <vegivamp> maybe a weird question: is it possible to have administrative commands (like rs.status() ) output clean json, without stuff like ISODate("2019-10-07T14:17:18.633Z") in it ?
[15:23:25] <GothAlice> vegivamp: Where are you making the invocation to get the results? In the Mongo REPL shell? Ultimately, pull in the results in your language of choice, serialize them however you choose.
[15:24:30] <vegivamp> GothAlice: yes, that's what I'm doing, but it barfs, and when I try to put the resulting json through a json validator it barfs at the isodate entry because it's got a colon and isn't surrounded by quotes
[15:25:06] <GothAlice> (Fun fact: JSON is actually a pretty terrible interchange format. It doesn’t have integers. Strings require some pretty extensive escaping, e.g. HTML-in-JSON is almost worst-case. It doesn’t serialize dates at all. Etc.)
[15:25:11] <vegivamp> for context, this is for ansible stuff; so I'm echoing the command into a mongo --quiet
[15:26:10] <GothAlice> You have all the choice. MongoDB connectors are widely available for nearly every language.
[15:27:28] <vegivamp> thus the question wasn't "can I program this", it was "can I get it to output proper json instead of mongo json
[15:28:59] <GothAlice> “I want JSON, and am not getting JSON, but am not willing to write something to give me JSON.” ¯\_(ツ)_/¯ It’s almost a one-liner in Python.
[15:29:22] <vegivamp> I'm sorry, I was rather thinking it's not too much to expect a json database to output json.
[15:29:50] <GothAlice> import json; import pymongo; c=pymongo.MongoClient(); print(json.dumps(c.adminCommand({‘replSetGetStatus’: 1})))
[15:30:38] <vegivamp> heplful. Now to convert that to a mongodb module for ansible and get in into the core. Will probably take me all of 5 minutes, including ACTUALLY LEARNING PYTHON.
[15:31:18] <vegivamp> and yes, yes, I know it's a document store that happens to input json. Same difference in this context.
[15:32:14] <GothAlice> Or pick a language you’re more familiar with. Or learn Python. It’s good for you, and the glue that holds the internet together. Learning to code is empowering, and code is magic, because you will be freed from the limitation of depending on others’ existing tools. Which don’t cover your specific needs.
[15:33:05] <vegivamp> All of wich is entirely irrelevant to me at this point. I asked a question, and you're diverting so far you're telling me learning python is good for me.
[15:36:55] <vegivamp> syntaxfree: granted. After having programmed for a decade and going on to do other stuff, I wasn't looking to get "you should learn to code" when asking a question about the mongodb shell. I apologise for the grumpiness.
[15:37:49] <GothAlice> On one hand, you can’t get what you want as things stand, partially due to a fundamental misunderstanding of what MongoDB actually **is**. On the other hand, writing one line of code, you can get exactly what you want. The pure approach gets you no solution, the practical one was solved ten minutes ago.
[15:37:52] <syntaxfree> tell people to take the long way around because you can only go so far with little answers.
[15:39:02] <vegivamp> Unfortunately, you never know if people are asking for little answers because they don't know any better or because they know exactly what they want.
[15:39:35] <vegivamp> So, is the answer to "can I get rs.status() to output clean json" a yes or a no?
[15:40:03] <syntaxfree> there's this problem I see a lot where people go look at a microwave at costco and keep asking questions about what kind of fuel it takes.
[15:40:05] <GothAlice> Little answer: you can’t. Moving on to the next problem, now? (I prefer to not just balk people with problems, but also to provide a viable path forwards. Writing one line of code would do. If you can invoke a Mongo REPL shell, you can invoke anything within that pipeline.)
[15:40:24] <syntaxfree> or why did it cook my food funny! because it's not a stove, it's a weird thing that's displaced some of the stove market.
[15:40:54] <vegivamp> ok, thank you. I'll see if it's worth it to incorporate that oneliner into the shell command, ugly as that is. Not like the shell call itself was clean to begin with :-p
[15:41:57] <GothAlice> (That one-liner does make a bunch of assumptions, e.g. it defaults to connecting to localhost. Pass a connection string to MongoClient() to point it elsewhere.)
[15:42:08] <syntaxfree> I've been on the internet since the mid-1990s. Somehow making people angry still passes for funny.
[15:42:57] <syntaxfree> it wouldn't work on a shitty prank tv show, irritating someone to the point where they're red screaming and then "you won 5000 dollars on deal with it". it's not funny.
[15:43:17] <vegivamp> I can figure those bits out, thank you :-) Have already been messing a little in the check_mk mongodb check which didn't support authentication.
[15:45:18] <GothAlice> I was quite prompt on giving the short answer. Not JSON, won’t get JSON, don’t use the REPL shell to try to get JSON. (You get REPL notation, which is JS, not JSON.) But hey, if syntaxfree interprets “you can’t, but here’s a way you might be able to” as a prank, not much I can do.
[15:46:10] <GothAlice> Noting a very particular point of problem: JSON can’t encode dates. So… that’ll be a fun thing to try to resolve, too. (Sorta… your next problem.)
[15:47:10] <GothAlice> Instead of json.dumps, might need to use bson.json_util.dumps to cover the date case. Ref: https://docs.mongodb.com/manual/reference/mongodb-extended-json/
[15:47:42] <vegivamp> No, I'm sorry, when you said "pull it into the language of your choice" I pointed out that I did that (ansible has a json-to-dictionary filter) and that it barfed. Then you basically went on to tell me to learn to code. Perhaps prefacing that with an explicit "no, it can't do that, but..." might have been clearer?
[15:48:56] <GothAlice> “Not JSON, won’t get JSON, don’t use the REPL shell to try to get JSON.” For example, what, specifically, does your Ansible system expect for dates… stored in JSON? Which can’t store dates?
[15:52:14] <vegivamp> Oh boy. I'll just chalk it down to my english not being good enough, because from my point of view that says exactly what I just said again: I never asked how to process dates, I asked how to output that info without the ISODate crap.
[15:52:52] <GothAlice> vegivamp: “Without the ISODate crap” = you want actual JSON, not REPL. >>>How do you expect to transfer dates in JSON, which can not transfer dates?<<<
[15:54:13] <vegivamp> the grep solution has exactly the same problem as the python oneliner - it's horrible, hard to read and hard to maintain.
[15:54:46] <vegivamp> At least the python solution will transform it into actual json so I can reuse the data in other parts of the playbook.
[15:55:02] <GothAlice> Even better, I grok it. Which is odd, given the questioner doesn’t, at all. “without ISODate crap” absolutely ignores what that is, what it means, and why it was there. And that “getting JSON back” **will explode**, because of those dates. “You can’t.” Not unless substantialyl more understanding is wrought.
[15:56:15] <vegivamp> I'm curious which bit you believe I don't understand, GothAlice? For all I care the date gets put into an ISO string, or an epoch, or a whatever.
[15:57:47] <GothAlice> vegivamp: Then you have no data ingress requirements? Grand! Throw anything at the wall, whatever sticks, use it. Most systems have… requirements, though. Easiest case: ignore the fact that it looks like JSON, store the REPL output as a string, eval() it any time you need the results. :3
[15:58:54] <GothAlice> Remembering that the output you are getting is REPL shell output, that is, valid input to that REPL.
[16:00:02] <vegivamp> I'm not working in the shell, I'm working in ansible playbooks. I was trying to get clean json, which I could easily transform into an actual dictionary for later use without a mess.
[16:00:42] <syntaxfree> Is it sunny where y'all are now? I for one am noticing that clouds have finally lifted and I'm going to get some vitamin D in me.
[16:00:53] <GothAlice> Of course, I’m somewhat sarcastically suggesting the worst possible solution it is possible to create, given the data would then become an opaque, unqueryable string. Also, you *are* working in a shell. You are invoking “mongo” and attempting to interpret its output. Which is REPL output.
[16:02:57] <vegivamp> I misinterpreted that, but evalling the output as a string in the shell simply yields the same thing.
[16:03:50] <vegivamp> Oh well, the python oneliner is probably the closest I'm going to get without writing my own ansible module. Thank you for that.
[16:03:59] <GothAlice> Indeed. REPL. Read, Eval, Print, Loop. It’s spitting out the repr representation of the input you gave it, demonstrating “identity transform”. Meaning you have demonstrated you can reconstitute the data from the output you have been given, that way.