Log file Viewer
#mongodb logs for Friday the 9th of August, 2019
[07:54:58] <benoitc> how is handled the transaction api at the protocol level? How can I simply add read consistency, ie read a snapshot of acollection
[08:14:29] <benoitc> is a session on majority + a transaction snapshot the way to have read consistency across multiple queries?
[09:08:49] <nukeu666> In a node application, how can i check if a mongo connection is still valid before querying?
[11:26:34] <blip99> hi all, I have a deeply nested collection with a field found at all levels named 'sys'. I'm looking for a way to remove that field wherever it occurs
[11:27:14] <blip99> what are your thoughts on the proposed solution? https://stackoverflow.com/questions/44254923/search-by-regex-on-field-name
[11:32:55] <GothAlice> blip99: To your initial question, you don't. You iterate documents, manipulate those documents in their entirety application-side, then ship the entirety back over the wire to save over the record. That SO answer will help you find the records to correct, with $where being a literal last ditch effort.
[11:36:16] <GothAlice> As this sounds like a one-time correction of a mistake, whole iteration of the collection with REPL or application-driven record replacement is likely going to be, by far, the simplest actual solution to cleaning up the records.
[11:36:44] <blip99> GothAlice, Thanks for the response. I'm basically cleaning up the DB during the population phase - and what we populate with changes but always has a 'sys' field. If I got you correctly, there's no more performant option than $where with javascript to go through each document and remove?
[11:37:10] <GothAlice> No. Again, $where will only *find* the records. It will not help you clean them up.
[11:37:39] <GothAlice> That will require application code you write to recursively and iteratively explore the document and delete the keys, then send the whole record back to MongoDB for replacement.
[11:40:59] <blip99> GothAlice, sorry I'm new to Mongo so need a bit more time. So I need app code that will 1) Call .aggregate($where...) to find the paths to the fields. 2) call .remove(paths)
[11:41:32] <blip99> I was hoping to be able to do it all with aggregations, like $where + $project somehow
[11:43:35] <GothAlice> No. Both steps you just described are wrong. a) find() with no query, literally iterate every single record. b) In your language, recursively explore the document, deleting the keys from that document application-side. (E.g. "del doc[key]" in Python). c) https://api.mongodb.com/python/current/api/pymongo/collection.html#pymongo.collection.Collection.replace_one
[11:45:29] <blip99> Ok. I get the above now - but I don't get why the emphasis on app code over doing it via mongo aggregations
[11:48:01] <GothAlice> blip99: Then I'm extremely sorry for you. There's a general rule: garbage in, garbage out. Repeatedly needing to compact the garbage is not a good sign. Do you have a working solution using aggregations? KISS (keep it simple, stupid) would suggest a) not populating the garbage in the first place ("strip" it before first insert), b) using the simplest operations available to meet the need for cleanup, now that damage has been done.
[11:48:53] <GothAlice> I caution against this because you think you're freeing up the space used by those deleted fields. But you are freeing up nothing, only fragmenting the records on-disk by adding sparse holes.
[11:49:36] <blip99> No the field removal is so that the REST endpoint on top doesn't serve it to the consumer
[11:49:55] <blip99> and it should be more performant to remove it from DB offline, than remove it in the REST code online
[11:50:27] <blip99> I will see if the garbage can be removed from source (however most likely it's actually not garbage there :) )
[11:51:05] <blip99> and you're right - I don't have an aggregation solution - would take me ages to figure one out. app code is simpler
[11:51:26] <GothAlice> Please learn what a "projection" is: https://docs.mongodb.com/manual/reference/method/db.collection.find/index.html — excluding a field from being returned is "free".
[11:51:46] <GothAlice> Sorry, more direct link: https://docs.mongodb.com/manual/reference/method/db.collection.find/index.html#find-projection
[11:52:26] <blip99> but this is tricky because field path will vary as the data changes, all i have is the name 'sys'
[11:52:59] <GothAlice> Uh… wait a minute. Your documents aren't polymorphic from document to document within the same collection, right? Like, entirely dynamic keys?
[11:54:36] <GothAlice> In this type of context, it's a short-form for: I can't help you. You are beyond my help.
[11:55:14] <GothAlice> https://www.javaworld.com/article/2088406/how-to-screw-up-your-mongodb-schema-design.html
[11:56:26] <blip99> Well, when the product is in early stages you're still figuring out the documents/format and how you model your data
[11:56:59] <GothAlice> E.g. this is naive: {greet: {en: "hello", fr: "bonjour"}} because as languages are added, the set of keys varies. It's unmanageable. This is correct: {greet: [{lang: 'en', greeting: "hello"}, {lang: 'fr', greeting: "bonjour"}]}.
[11:57:20] <blip99> which I don't like, given I have a research background. but industry is different, rush features, wow investors, buzz words
[11:58:35] <GothAlice> With code executing across that data like this: http://s.webcore.io/1a3b2B2B1e0K/match.pdf
[11:59:08] <GothAlice> Which then utterly implodes when 100 users sign up at the same time. (I.e. getting LifeHacker'd and HackerNews'd the same week. Like we did. Before curling up and dying.)
[12:00:03] <blip99> Yeah, I agree but we haven't been able to change management's mindset. I gotta jump ship
[12:00:22] <GothAlice> Sometimes that is the right thing to do. Sometimes that's the only thing to do.
[12:00:42] <blip99> You literally gave me more time, more insight and more help than any senior colleagues do
[12:01:52] <GothAlice> It galls my Linux-using friends and coworkers that my OS is the only one in the office actually certified UNIX.
[12:02:32] <GothAlice> "Wait, Linux isn't UNIX?" "Nope. It is not legally allowed to call itself that. There's a reason you see it written UN*X everywhere it's mentioned."
[12:02:50] <blip99> Been Linuxing since 2005 or so, moved to industry only few years ago. Still so much to learn, but can't find a place that gives room for transition - it's always RUSH rush rush with startups
[12:12:14] <GothAlice> Heh. If ever I've seen someone in need of a printed copy of my Laws, it's you. So that you can beat management around the ears with it: https://gist.github.com/amcgregor/9f5c0e7b30b7fc042d81
[12:12:42] <GothAlice> Alice's Law #158: Perfection is achieved not when nothing more can be added, but when nothing else can be taken away.
[15:39:01] <aksu> I have sharded environment where all the configuration servers got wiped. what would happen if I just setup new configuration servers and add the shards?