Log file Viewer
#mongodb logs for Wednesday the 29th of June, 2016
[00:05:23] <TommyTheKid> OK, so I *know* I am being dense or missing something obvious. I am creating a new "user" in an existing mongo 3.0.x cluster, I can get the user to authenticate if I specify "admin" then change dbs, but I can't connect directly to its "application" db. Another user on the same system works properly :-/ (there is no system.users in the working application database)
[00:11:19] <TommyTheKid> I am authenticating with a user with role "root" to create the user (if that matters)
[00:23:28] <TommyTheKid> ugh, apparently you create the users on the database, but you can't see the collection "system.users" in it... however "show users" after "use appdb" does show appdb.appdbuser ... so ... wow suckify
[09:48:56] <spleen> could not verify that config servers are in sync :: caused by :: config servers xxxx27019 and xxxx:27019 differ
[10:22:17] <Mmike> Hi! Is there a way to change the log level verbosity on a RUNNING mongod instance? I am running mongo 2.4.9
[10:31:05] <thapakazi> Please recommend me best storage engine for my production: (app demanding optimised for fast disk I/O), I was looking into perconaft, but they are ditching it on future release :(
[10:59:51] <thapakazi> Please recommend me best storage engine for my production: (app demanding optimised for fast disk I/O), I was looking into perconaft, but they are ditching it on future release :(
[12:14:56] <HoierM> Is there anything for mongo that resembles the hibernate xml configuration? I'm using Java, but i can't annotate the model. Not possible to change the model, in no way.
[12:25:11] <StephenLynx> one document can be {banana:9001} while the other can be {zebra:{omg:'kek'}}
[12:27:52] <kurushiyama> StephenLynx Seriously, that one is good. It takes away boilerplate, but lets you go down to the driver in case you want to.
[12:28:02] <StephenLynx> to be honest, I see zero reason to use any kind of schema validation on a database that doesn't implement relations.
[12:28:21] <StephenLynx> at best is a crutch for developers that shouldn't be allowed to program to begin with.
[12:29:25] <StephenLynx> sure is not just validation, after all, how could they justify it without loads and loads of useless bloat.
[12:29:30] <kurushiyama> StephenLynx Well, I use it. I validate the user input multiple times. Frontend before submit, server on submit, and I use document validation to make sure the data stays as it should be, no matter what.
[12:29:56] <StephenLynx> the problem is adding a whole dependency for something that is not even a dozen LOC.
[12:30:13] <kurushiyama> StephenLynx But using SpringData just for the sake of it will cause serious problems ;)
[12:31:08] <HoierM> I'm more worried about the serialization/deserialization part, i'm not doing the web module, just the persistence layer... I wanted to use SpringData, just not sure if it's too big a dependency for this project.
[12:31:44] <kurushiyama> HoierM Depends on the scale of the project. It takes away all the basic CRUD stuff, that is not a bad thing.
[12:31:59] <HoierM> I wanted to use Morphia at first, but i couldn't find anything about it and external mapping.
[12:35:43] <kurushiyama> If it is a small project, I suggest using Morphia directly. If you have to, you can pass around the client object or (if the project has the potential of getting bigger) use Guice.
[12:36:52] <HoierM> Yeah, exactly that. I'm quite new to mongo, so i might be saying bullshit on this side... The thing is that i cannot annotate the classes directly. Any direct model change would be seen as a hack, so i can't do that. The model is on a separated module.
[12:36:53] <kurushiyama> HoierM Although I would not underestimate SpringData. If you just need basic CRUD, this can be achieved with a few dozen lines of codes on top of _all_ of your entity classes.
[12:38:05] <HoierM> I cannot change the model. That's the catch. Before i could control it, but they changed it to a module, so i cannot make changes to it.
[12:38:58] <HoierM> To explain better, this is for my university, and since there's a load of people doing different solutions, they decided to do this.
[12:39:57] <HoierM> kurushiyama well... that'd still be a model change, and my professor would hate it.
[12:40:59] <HoierM> Yeah, anything vendor-specific. It was kinda of absurd when i heard it the first time too, i cannot even picture this without annotation...
[12:41:32] <HoierM> But then i remembered that hibernate had those old xml external mappings, because i only used the annotation-based ones, so i was looking for something like it
[12:44:54] <HoierM> I pretty much wanted this: https://github.com/DavideD/ogm-mongodb-example/blob/master/src/main/resources/hibernate-contact.hbm.xml
[12:45:37] <kurushiyama> HoierM Would you answer my question? I might have a solution for you, if that is the case.
[12:45:39] <HoierM> They "are being converted into". They didn't even made get and set for all the fields, so i can't call it exactly a bean.
[12:46:37] <kurushiyama> HoierM Well, subclass them in a bean and override. Lombok might be helpful here.
[12:50:33] <HoierM> kurushiyama Heh, now that's one nice expression. I'll look into that, then. Thanks for the help.
[12:54:32] <kurushiyama> HoierM You probably can use Lombok for the boilerplate. But with hiding(where necessary only!) and subclassing, your problems should be solved.
[13:05:37] <kantoniak> I'm creating a playground for users and I'd like anyone to have their own database.
[13:56:50] <scruz> hi. how do i write a an expression (agg framework) checking if a value is null or missing?
[14:58:27] <scruz> i’m using an $eq, so as long as it’s not a valid value, it’s allowed to be null or empty
[16:03:36] <speedio> anyone know what i can use for wordpress that handles documentation for software with different versios(similar to msdn documentation)?
[16:08:24] <speedio> StephenLynx: i need to setup a documentation management system and thought maybe i could use wordpress for that..?
[18:20:09] <idioglossia> Hi everyone, do collection names have to be alphanumeric, like javascript identifiers?
[19:07:28] <kurushiyama> idioglossia Actually, just like JavaScript identifiers, if my lack of JS knowledge does not betray me,
[22:00:37] <bjpenn> using 2.6.9 mongodb, anyone know if theres a bug that causes operations to not get killed properly?
[22:05:22] <bjpenn> kurushiyama: any reason why there would be super long running process that doesnt end?
[22:06:00] <kurushiyama> bjpenn Depends on the process and what you call "super long". Can you be more specific?
[22:07:30] <kurushiyama> bjpenn Depending on your data size, an index creation for example can take quite a while.
[22:10:24] <kurushiyama> Oh. That is a long query.This can happen when you set the timeout for the query to a big number or to 0 (worst case) and the client does not properly close the cursor, iirc.
[22:11:56] <kurushiyama> bjpenn You are aware of the fact that each time you create a client, actually a connection pool is created?
[22:12:46] <kurushiyama> bjpenn You close cursors. And clsoing a connection should really return it to the pool.
[22:17:05] <kurushiyama> bjpenn As far as I can see (from my rather limited knowledge and interest in PHP), both open a connection pool.
[22:19:38] <bjpenn> Derick: so do php users use someting like maxTimeMS() to kill serverside connections?
[22:22:10] <Derick> the mongo log will tell you, as well as the place where you seeing where it says "query"
[22:24:01] <kurushiyama> @Derick Interesting. know php was single threaded. Per instance, I guess, but yes, that makes sense.
[22:31:43] <bjpenn> Derick: looks like this: https://gist.github.com/3283096e24cfef4cc8cf512eedb9d681
[22:32:34] <bjpenn> if its normal and its running along time could it be that the clinet died mid transaction?
[22:33:46] <bjpenn> but i guess the operation would just complete, and operation would finish right?
[22:34:46] <kurushiyama> Well, I am not so. If the timeout was set to 0 and the client just broke away?
[22:46:56] <kurushiyama> @Derick Curious. https://docs.mongodb.com/manual/reference/method/cursor.noCursorTimeout/#definition
[22:50:16] <kurushiyama> @Derick I would not, but in my (admittedly rusty) memory it was the server managing the server timeouts.
[22:54:54] <kurushiyama> bjpenn Well, you _could_ have your primary step down and restart it then...
[22:58:54] <bjpenn> kurushiyama: yeah... but i wonder why my queries running so long in teh first place :/
[23:02:15] <Derick> UberDuper: really? Can you write it down and email me? I did write it after all.
[23:03:49] <UberDuper> Oh? My perspective is as the mongo server operator dealing with apps that end up hitting connection limits.
[23:05:48] <UberDuper> The one thing I'll give it credit for is that you don't have cursor not found issues when load balancing mongos nodes.
[23:08:49] <kurushiyama> UberDuper Reaching the connection limit? Even on my 6 year old laptop with 4Gb of RAM, running several databases and my complete dev environment my mongod reports a max connection count of soome 7k...
[23:10:34] <kurushiyama> UberDuper May well be, but from my perspective, running into your max connection count is a question of lazy resource management and lack of tests therof.
[23:10:58] <UberDuper> So even if you get to 40k connections into your mongos nodes, you could still only have a couple hundred connections to the primary.
[23:11:59] <kurushiyama> UberDuper As long as RAM and fds are available, I do not see an issue. Not closing connections properly is a dev issue.
[23:12:29] <kurushiyama> UberDuper I am managing several high load clusters, and that _never_ happened.
[23:13:28] <kurushiyama> UberDuper If that happens, there is something severely wrong, either on the ressource side or on the configuration side.
[23:23:44] <bjpenn> Specifies a cumulative time limit in milliseconds for processing the operation on the server (does not include idle time). If the operation is not completed by the server within the timeout period, a MongoExecutionTimeoutException will be thrown.
[23:24:11] <bjpenn> if my php client connects to mongo to make a query, where in that example is the idle time?
[23:29:46] <kurushiyama> bjpenn To be aken with a pound of salt and then some, here is how I read it: When a query is sent to the MongoDB server, it does not get executed right away. It is run through the query optimizer and – as you noted, and especially for MMAPv1 – the requested ressource or parts thereof might not be available. Say you have an unindexed query, aka collscan. Even with WT, documents might be locked while the query traverses the collection.
[23:33:29] <kurushiyama> Basically, it is a wild guess taking into account a few things. I am most likely totally off, thinking of it twice. The collection or documents therein being locked could very well count to the timeout (and actually should, from my perspective.)
[23:34:53] <bjpenn> kurushiyama: do you know how to tell whats happening for a long running operation
[23:35:00] <UberDuper> If I had to guess, I'd say idle time has more to do with batches and time between getmores.
[23:36:16] <kurushiyama> UberDuper Dont know. Time between getmores is actually what I would call inactivity...
[23:36:17] <UberDuper> From the server perspective there's really no idle time when it comes to queries.
[23:36:39] <bjpenn> kurushiyama: reading this link now... http://blog.mlab.com/2014/02/mongodb-currentop-killop/
[23:37:58] <UberDuper> bjpenn: Best place to start with finiding out what why that query is slow is to check the mongo logs.
[23:39:02] <kurushiyama> bjpenn Well, that is not exactly new. The docs state pretty much the same. And UberDuper is right, especially when you go for mlogviz.
[23:44:18] <bjpenn> i guess one way is to find the time when i did the currentOp() and then minus the seconds its been running to get a rough estimate when it started running
[23:45:53] <UberDuper> Yea. I'm not sure what shows up in the log when you kill a long runner. I don't think anything.
[23:46:39] <UberDuper> grep "ms$" /path/to/log | grep -vi "writeback\|lockping" | awk '{print $NF" "$0}' | sort -n
[23:47:16] <kurushiyama> bjpenn Not sure wether they still work, but Thomas Rückstiess mtools might be quite handy : https://github.com/rueckstiess/mtools/wiki/mlogfilter
[23:50:22] <bjpenn> it doesnt sort, it just greps through and shows every matching one that ends with ms
[23:50:43] <UberDuper> grep "ms$" /path/to/log | grep -vi "writeback\|lockping" | awk '{print $NF" "$0}' | sort -n
[23:56:33] <UberDuper> kurushiyama: Yeah. My awk-fu is weak. So I just keep piping till I get what I want.