Log file Viewer

[07:05:18] <ocx32> hello, i i have a counter in mongo, which is accessed by nodejs and incremented, now when multiple requests from node come in, the counter increment becomes inconsistent as multiple requests are reading the initial counter (shaow copy) and incremention only by 1, how can i solve this to have the counter blocking and read correcly?

[07:15:04] <ocx32> anyone?

[07:15:20] <ocx32> its not easy to switch just like that

[08:26:00] <ocx32> hello, i i have a counter in mongo, which is accessed by nodejs and incremented, now when multiple requests from node come in, the counter increment becomes inconsistent as multiple requests are reading the initial counter (shaow copy) and incremention only by 1, how can i solve this to have the counter blocking and read correcly?

[08:26:01] <ocx32> anyone??

[08:54:50] <ocx32> ins a #inc atomic ?

[08:55:57] <kali> it is

[08:57:37] <ocx32> kali i am doing a mycol.update({'_id':x['_id']},{'$addToSet': {'channels': {'Name': ch} },'$inc': { 'available' : -1 } }) <-- if i get concurrent requests from node to do this, i am sometimes getting inconsistanies

[08:57:42] <ocx32> how can i solve it

[08:59:13] <kali> how do you observe inconsistencies ?

[08:59:46] <ocx32> so for instance if i have 2 requests, i should get available -2 i am getting an available -1

[09:00:09] <ocx32> this happens when transactions happen at the same time mainly

[09:00:11] <ocx32> hard to trace

[09:01:06] <kali> so how are you observing it ? performing a find() ?

[09:01:13] <ocx32> yes

[09:01:21] <ocx32> i check my counter afterward

[09:01:30] <ocx32> it should be say 18 , sometimes it shows 19

[09:03:00] <kali> do you have a replica set ?

[09:03:35] <ocx32> no

[09:06:27] <ocx32> kali any idea?

[09:07:54] <kali> you should check the logs of your server, but you, or something is doing somehting wrong, the $inc is atomic.

[09:08:30] <ocx32> kali here channels is considered a subdocument ?

[09:11:17] <kali> i don't know what you mean, and it should not matter anyway

[09:13:05] <ocx32> kali here is my code https://pastebin.com/rx3nPcYL

[09:13:44] <ocx32> if a requests comes in while request1 is at line6, the counter becomes inconsistent

[09:16:41] <kali> if you're doing the obvious thing, you should just check available : { $gt: 0 } in the update command

[09:16:54] <kali> the inc is atomic. your find then update code is not

[09:18:06] <kali> and you need to check the update_one return to check that one document was updated

[09:18:22] <kali> you may actually be able to do anything in one single update

[09:21:49] <ocx32> kali how would it like what function should i use?

[09:22:52] <kali> let's just keep the find then update for now. add "available : { $gt: 0 }" beside the id on the update

[09:25:26] <kali> that should stop the inconsistency, but you also need to know if the update was successful, so you'll have to replace your update to findAndUpdate

[09:26:09] <ocx32> kali mycol.update({'_id':x['_id'], "available : { $gt: 0 }"},{ ?

[09:26:44] <kali> yeah, this will prevent the update to happen entirely if available got down to zero in the meantime

[09:27:26] <ocx32> , but you also need to know if the update was successful, so you'll have to replace your update to findAndUpdate <-- sorry didnt get that

[09:28:13] <kali> findAndUpdate is a variant of update that allow you to know if the update actually happen

[09:28:59] <ocx32> so it woulg go like mycol.findAndUpdate({'_id':x['_id'], "available : { $gt: 0 }"},

[09:29:00] <ocx32> ?

[09:29:15] <kali> if the available gets down to zero in between your find and your update (with the available check on update) the it will fails silently. your code has no way yo check. unless, instead of using update, you use a find and update AND check the outcome

[09:29:25] <kali> yes

[09:29:41] <ocx32> do you recommend any book on mongo?

[09:30:29] <ocx32> kali so you mean i do that findandUpdate and do an if condition to check tis return value and act accordingly right?

[09:30:39] <kali> yes

[09:30:55] <ocx32> thanks

[09:32:30] <ocx32> y = mycol.findAndUpdate({'_id':x['_id'], "available : { $gt: 0 }"}, if ( y ): ...

[09:32:38] <ocx32> in python

[09:34:43] <kali> man, don't copy and paste, fix the syntax :)

[09:35:36] <ocx32> kali i have another one working on same document, happening on a different even : dbo.collection('servers').updateMany({ 'c': {'cName': pub } } , { $pull: {'c': {'cName': pC } } , $inc: {'available':1 } , $set: { 'status':'free'} } );

[09:35:40] <ocx32> this is atomic?

[09:38:05] <kali> yes and no: atomic inside a document, but not accrosss the servers the collection

[09:38:27] <ocx32> any way to fix that?

[09:38:56] <kali> mongodb transactions, i guess, but i have never used them

[09:43:02] <ocx32> kali if i have 1 document in servers i should be fine?

[09:48:29] <kali> yes

[09:52:10] <ocx32> so my non atomic was in the thing we fixed

[09:52:19] <ocx32> kali how can i understand all this more? do you recommend any book?

[09:52:56] <kali> i don't know the books, really

[09:53:00] <kali> i can't recommend one

[09:53:49] <ocx32> is mongo considered a good db? sql ppl hate it

[09:53:59] <kali> but i agree, you need some systematic in depth knowledge

[09:54:02] <ocx32> they say its only a hype

[09:54:17] <kali> ocx32: well, 10 years is not bad for a hype

[09:54:35] <ocx32> can i rely on it as my main db

[09:55:03] <kali> ocx32: the sql world was comfortably settle in a paradigm from the late 60's and had difficulites accepting other people challenging it

[09:55:30] <ocx32> they said my problem cant be solved, only transactions can solve it which is no existent and complex in mongo

[09:55:32] <ocx32> which makes me wonder

[09:55:59] <kali> ocx32: now that a few alternatives have shown that some thing could be done differently, some are making some progress

[09:56:22] <kali> ocx32: well, from the small examples you show, it looks like your app is quite transactional in nature

[09:56:33] <ocx32> yes

[09:56:47] <kali> ocx32: so it may be true that a relational, transactional db may be a better fit

[09:56:49] <ocx32> chains and flows

[09:57:06] <ocx32> where does mongo win over sql?

[09:58:04] <kali> it's geared towards applications with a massive read-to-write ratio

[09:58:15] <kali> as most modern web apps are

[09:59:15] <kali> it is relatively easier to manage, comes with easy replicate and fallover, can scale out if needed

[09:59:34] <ocx32> when it comes to disk io performance it wins then

[09:59:35] <kali> you can "change the schema" with no downtime

[10:00:49] <ocx32> right?

[10:00:54] <kali> sql system add isues staying up 24/7 for a long time... they were design for business with opening and closing hours

[10:01:39] <kali> disk io is not that relevant really, mongodb will operates more smoothly if the working set is in memory anyway

[10:07:15] <ocx32> kali will mongo survive a server crash if it runs in memory?

[10:08:59] <kali> depending on your configuration, you make loose the latest updates, but it will recover

[10:09:26] <ocx32> but sometimes that is critical you dont want to lose anything

[10:11:41] <kali> well, you should configure accordingly

[10:11:46] <kali> at least you have the choice

[10:13:19] <kali> in some setups, that were dramatically ignored by the old sql system, you will need to trade some persistence warranties for some performance

[10:13:54] <kali> mongodb lets you do that to some extent, while the old db are pretty radical on persistency

[10:14:31] <kali> once again, designing for a bank, not a web chat