PMXBOT Log file Viewer

Help | Karma | Search:

#mongodb logs for Monday the 7th of September, 2015

(Back to #mongodb overview) (Back to channel listing) (Animate logs)
[00:02:37] <damoncasale> StephenLynx: Reading over what I can find on MongoDB performance vs. Mongoose performance. It seems like Mongoose can still work, but one would need to be careful to use it correctly.
[00:03:13] <StephenLynx> the problem is
[00:03:17] <StephenLynx> mongoose doesnt work correctly
[00:03:21] <StephenLynx> it screws up ids
[00:03:25] <damoncasale> For instance, one of the slowdowns appears to be the validation prior to writing out a record. Mongoose provides a bare query method, tho.
[00:03:28] <damoncasale> Huh??
[00:03:30] <damoncasale> Since when?
[00:03:40] <damoncasale> I can
[00:03:53] <damoncasale> I can't find any reference to that in the articles I've read.
[00:06:55] <StephenLynx> hold on
[00:07:23] <StephenLynx> Mongoose assigns each of your schemas an id virtual getter by default which returns the documents _id field cast to a string, or in the case of ObjectIds, its hexString.
[00:07:27] <StephenLynx> its fubar
[00:07:52] <StephenLynx> in the last year, every single issue people had with node and mongo was related somehow to mongoose.
[00:08:16] <StephenLynx> and since you just started learning mongo, I assume you are not bound by some legacy system using this cancer.
[00:08:19] <StephenLynx> but hey
[00:29:23] <mprelude> * (cl-scram-sha1:base64-encode (cl-scram-sha1:gen-client-initial-message :username "aurum" :nonce (cl-scram-sha1:gen-client-nonce)))
[00:29:25] <mprelude> "biwsbj1hdXJ1bSxyPXdaUnF6MnNuUUZ6UVdoNFVJb0RTOGpGampKbkJXUGhx"
[00:29:28] <mprelude> We're part of the way there ;)
[00:37:52] <mprelude> StephenLynx: Decided the only way forward is to implement SCRAM-SHA1 in lisp. :D
[00:38:15] <StephenLynx> good speed
[00:38:17] <StephenLynx> god*
[00:38:35] <mprelude> Hoping I can have that done, and the mongo driver updated to use it, this week.
[00:40:07] <mprelude> Not a fan of hacky fixes, heh.
[05:32:16] <_syn> how can I get the output of slow system profiling to go to a logfile
[05:59:33] <Rou> Is SQL an objectively more reliable data store than mongodb
[05:59:35] <Rou> And does it matter?
[06:01:05] <pEYEd> how would I query all of the usernames from this collection? http://paste.ee/p/o9xGf it's inside the subarray of 'data' and I can't get to it. :(
[07:25:45] <preaction> SQL is a data store? i thought it was a structured query language
[09:01:18] <jebis> Hello, is there a way in php to make a ordered insert of multiple documents? u know i want to have them in a ordered manner http://pastebin.com/EmwSsztD i dont know it is allways mixed even when i fetch them with sort "create_date" : -1
[09:14:16] <coudenysj> jebis: can't you just use the "natural" order ?
[09:14:40] <jebis> coudenysj natural order hmmm
[09:16:12] <coudenysj> https://docs.mongodb.org/manual/reference/operator/meta/natural/
[09:21:58] <jebis> coudenysj this natural works but why does not work a normall sort by create date ? i mean is mongo so fast that it creates 2 same dates ?
[09:22:25] <coudenysj> basically, yes
[09:22:46] <coudenysj> jebis: look at http://docs.mongodb.org/manual/reference/object-id/
[09:24:05] <coudenysj> the first part of the id is the time, but there are other values needed (mostly to solve multi server problems), but you can insert multiple docs in a given msec
[09:33:02] <jebis> coudenysj and $natural uses that part of the id ? or how it works if i can ask :P
[09:33:24] <coudenysj> jebis: https://docs.mongodb.org/manual/reference/operator/meta/natural/
[11:05:24] <agis-> hello
[11:07:36] <agis-> Isn't something wrong with my 2.4 server logs? https://gist.github.skroutz.gr/agis/212d40ee53e5ac7e3006
[11:07:48] <agis-> (I don't have any authentication enabled)
[11:11:32] <mprelude> Hi, sorry if this seems like a noob question. I'm updating one of the mongo drivers to use SCRAM-SHA1. I was wondering the easiest way to get into a communication with the server so I can try throwing requests at it and look at the responses.
[11:11:42] <mprelude> Would telnet work?
[11:12:43] <mprelude> I just want a raw TCP discussion with the server, I guess, so I can try things.
[11:42:26] <agis-> what's the way to force the client to connect to the server?
[11:54:51] <joannac> agis-: ...what?
[12:06:45] <StephenLynx> wtf
[12:06:57] <StephenLynx> I am getting a weird behavior on aggregation sort
[12:07:25] <StephenLynx> http://pastebin.com/CFtPvDLB
[12:07:32] <StephenLynx> joannac anything I am missing here?
[12:08:49] <StephenLynx> the $group isnt respecting the order of the documents after $sort
[12:08:53] <StephenLynx> wot
[12:09:33] <StephenLynx> >Order of the array elements is undefined.
[12:09:34] <StephenLynx> :v
[12:09:35] <StephenLynx> v:
[12:10:30] <StephenLynx> $push instead $addToSet did the trick
[12:28:15] <mprelude> Anyone here familiar with writing mongodb drivers?
[12:28:26] <StephenLynx> cheeser or Derick
[12:28:33] <StephenLynx> one of these writes the java driver at least.
[12:29:00] <StephenLynx> or something
[12:29:21] <mprelude> Thanks StephenLynx
[12:30:28] <mprelude> cheeser or Derick (or anyone else): If you can help me get pointed in the right direction to do a mongo auth over telnet or similar, that'd be great. I can't seem to find any tcp-level documentation.
[13:19:35] <Derick> mwhooker: what sort of auth?
[13:25:16] <mprelude> Derick: Assume that was for me. SCRAM auth.
[13:27:44] <nerder> hello
[13:28:56] <nerder> i'm using mongodb using spring
[13:29:10] <nerder> which one of this 2 solution can be better in terms of performance? https://gist.github.com/nerder/ee5edac69f8306f11bfd
[13:30:03] <nerder> they produce actually the same result, in the first case i use ArrayList because $in wants a List
[13:48:29] <StephenLynx> nerder I suggest you don't use spring.
[13:51:59] <corentin> hello
[13:52:18] <corentin> #any clue how to remove all documents from a capped collection?
[13:55:36] <corentin> I've seen about {emtpycapped: "collection"} but it seems to only be a test command
[14:05:42] <Derick> mprelude: let me see if I can find something
[14:06:21] <mprelude> Derick: The main thing I'm struggling with is just how to format my requests to the server. Usually I do this kind of stuff over telnet so that I understand what i'm implementing, then write them in code.
[14:06:56] <Derick> mprelude: this is what PHP does: https://github.com/mongodb/mongo-php-driver/pull/730/files
[14:08:34] <Derick> it implements http://tools.ietf.org/html/rfc5802
[14:08:52] <Derick> but not sure if you can do it with telnet, as it requires binary info to talk to mongodb
[14:09:09] <mprelude> Ah
[14:09:41] <Derick> mprelude: and https://github.com/mongodb/specifications/tree/master/source/auth is our spec for it
[14:12:27] <Derick> https://github.com/mongodb/specifications/blob/master/source/auth/auth.rst#scram-sha-1 to be about scram-sha-1
[14:15:08] <crispy_beef> Hi. Wondered if somebody here could shed some light on why we are getting the error message "no primary server available" now and then when connecting to our replica set?
[14:15:16] <crispy_beef> 90
[14:15:25] <crispy_beef> 90% of the time everything is fine.
[14:15:45] <crispy_beef> Using latest stable Node.js driver.
[14:30:25] <agis-> what's the way to get the collection_names;
[15:56:23] <corentin> Is there some way to automatically delete a collection after some timeout has expired?
[15:57:17] <cheeser> nope
[15:57:30] <cheeser> you can ttl away all the docs but the collection will remain
[16:02:53] <corentin> cheeser: thanks. Is there no way to automatically delete a collection when it's empty?
[16:03:22] <cheeser> there isn't
[16:03:30] <corentin> that would be nice
[16:04:17] <cheeser> it's a pretty niche use case
[16:13:20] <corentin> cheeser: I think it's mostly useful when using mongodb for web stuff, such as django or so
[16:17:08] <cheeser> "web stuff" doesn't really imply the need for this kind of feature, though.
[16:19:59] <corentin> cheeser: well in our case we have limited hard disk space, so we have to use capped collections and rotate the documents in our collection. This means when we display a document to a user, the document might have expired if/when the user request more data about this document
[16:20:21] <corentin> our solution was to copy the current collection to another collection with a name containing the user session ID
[16:21:19] <corentin> when the user session expires, we want to delete this tmp collection
[16:23:41] <StephenLynx> if you are using dynamic collection creation
[16:23:53] <StephenLynx> your system is beyond salvation.
[16:27:26] <cheeser> disk space is super cheap
[16:27:33] <cheeser> engineering time not so much
[16:28:07] <StephenLynx> not to mention theres no point to track anonymous browsing.
[16:28:27] <StephenLynx> either the data is there to be shown or isn't.
[16:43:21] <corentin> cheeser: we fill up like 10GB per day
[16:47:53] <cheeser> you'd be better off to not use dynamically created collections and just ttl away the data you don't want
[16:48:14] <StephenLynx> and don't store useless data.
[16:55:34] <synthmeat> hey, i'm failing to $inc stuff :(
[16:55:40] <synthmeat> here's my code https://gist.github.com/synthmeat/1bf80e26e4f12de9dc44
[16:57:21] <StephenLynx> update['facebook.authenticated'] = true;
[16:57:22] <StephenLynx> wat
[16:57:50] <StephenLynx> why don't you use {faceboot:{authenticated:true}}
[16:59:01] <StephenLynx> why dont you just use ... options,callback); instead of using a dummy callback?
[16:59:15] <StephenLynx> you forgot to use $set
[16:59:43] <synthmeat> because there's more in facebook i don't wanna overwrite
[17:00:01] <StephenLynx> hm
[17:00:14] <StephenLynx> it doesnt matter.
[17:00:15] <synthmeat> if you know of a more convenient workaround that, i'd be glad to refactor my stuff, since there's a lot of this ugly stuff around
[17:00:25] <StephenLynx> in the end you are passing the same object.
[17:00:51] <StephenLynx> the issue you have probably its because you don't use $set
[17:00:53] <synthmeat> no, there's in document alread stuff under facebook., and if i fill up facebook object and pass it as update, what was there disappears
[17:01:10] <StephenLynx> so it overwrites the whole thing
[17:01:16] <synthmeat> i'm all ears.
[17:01:21] <synthmeat> (this is mongoose, if it's not obvious)
[17:01:37] <StephenLynx> hm
[17:01:42] <StephenLynx> yeah, I suggest you don't use mongoose.
[17:01:46] <StephenLynx> theres no worse tool than it.
[17:01:53] <synthmeat> i get that a lot. :/
[17:02:17] <StephenLynx> so you have an idea on how its bad.
[17:03:08] <synthmeat> ok. how about this. i fo refacctor my stuff to not use mongoose and then i'll find this channel much more useful? :)
[17:03:36] <synthmeat> i always get "ugh. it's mongoose" :D
[17:05:18] <cheeser> only from StephenLynx, really.
[17:05:31] <synthmeat> oh, plot thickens
[17:05:40] <cheeser> i'm usually "ugh. it's javascript" but i usually keep that to myself. ;)
[17:05:42] <StephenLynx> ok, about that other think
[17:06:04] <StephenLynx> you can use {'facebook.thing':value,'facebook.otherthing':value}
[17:06:08] <StephenLynx> instead of what you have.
[17:06:21] <StephenLynx> it will achieve the same result you have not.
[17:06:30] <StephenLynx> either way, you have to use $set
[17:06:36] <synthmeat> how do i use $set?
[17:06:37] <StephenLynx> otherwise you overwrite the document.
[17:06:57] <StephenLynx> update({query},{$set:{field:value}})
[17:07:16] <StephenLynx> http://stackoverflow.com/questions/15724272/what-is-the-difference-between-id-and-id-in-mongoose
[17:07:19] <StephenLynx> Mongoose assigns each of your schemas an id virtual getter by default which returns the documents _id field cast to a string, or in the case of ObjectIds, its hexString.
[17:07:22] <StephenLynx> so yeah, nah
[17:07:24] <StephenLynx> mongoose is pure cancer.
[17:07:42] <StephenLynx> flawed by design.
[17:07:59] <synthmeat> lemme try your suggestions
[17:08:30] <StephenLynx> and no one from the people that I have ever seen here giving advice on anything uses it.
[17:09:40] <synthmeat> actually, i spent the other day debugging that _id thing
[17:09:51] <synthmeat> was hard to notice in logs
[17:11:07] <StephenLynx> :v
[17:11:09] <StephenLynx> v:
[17:11:16] <StephenLynx> telling you.
[17:11:21] <StephenLynx> mongoose is nothing but trouble.
[17:11:40] <synthmeat> dont get me started on Date!
[17:20:28] <synthmeat> StephenLynx: this worked amazingly so. thank you!
[17:21:51] <StephenLynx> now you just need to perform chemo on your project and remove mongoose.
[17:22:22] <synthmeat> :)
[17:22:39] <synthmeat> as soon as i manage to scrape an evening off. sounds like a weekend project
[17:34:37] <synthmeat> hm, "cannot have a mix of inclusion and exclusion"
[17:34:53] <synthmeat> that's with var options = { new: true, upsert: false }
[17:35:03] <synthmeat> on findOneAndUpdate
[17:38:35] <synthmeat> what was expected was new document if one was found and updated, and nothing if no document was found
[17:52:59] <synthmeat> this is the code producing that error https://gist.github.com/synthmeat/72ec11e75c06dd7e6fe9
[17:59:50] <synthmeat> (nvm, wasn't that :) )
[18:00:09] <StephenLynx> else if (!!response) {
[18:00:11] <StephenLynx> what
[18:00:13] <StephenLynx> wat
[18:00:14] <StephenLynx> wot
[18:00:15] <StephenLynx> :v
[18:00:15] <StephenLynx> v:
[18:00:39] <StephenLynx> dude, just pass the callback directly to findOneAndUpdate
[18:00:49] <StephenLynx> userModel.findOneAndUpdate(query, update, options, callback);
[18:01:34] <synthmeat> hm, yeah, right on
[18:02:20] <synthmeat> StephenLynx: i still have issue with that pasted code in that it returns response, creates document
[18:02:39] <synthmeat> (i have these ifs to debug this right now)
[18:11:48] <synthmeat> so, uses $set, new: true, upsert: false, findAndModify, creates document :/
[18:12:53] <StephenLynx> it shouldnt
[18:13:02] <StephenLynx> upsert is what should create a document.
[18:13:34] <synthmeat> i'll debug then further, maybe it's me :/
[18:13:40] <StephenLynx> it is you.
[18:13:45] <StephenLynx> or mongoose.
[18:14:19] <synthmeat> :)
[18:14:26] <synthmeat> that's helpful still, thank you
[18:20:34] <flok420> in an aggregate I want to group records by their timestamp per hour. timestamp in seconds so I thought I just do ts - (ts % 3600). Am I right that that should be written like this: $group: { '_id': { name : "$name", ts : { $subtract : [ "$ts", { $mod : [ "$ts", 3600 ] } ] } }
[18:20:56] <flok420> because mongodb shell says "SyntaxError: Unexpected string" (with no hints where this faultive string is)
[18:20:57] <StephenLynx> nope
[18:21:13] <StephenLynx> {_id:'$yourpivotingfield',otherstuffgoeshere}
[18:21:57] <StephenLynx> you put the name along in the _id
[18:22:12] <StephenLynx> so it will always be different even if the part relating to time is the same
[18:22:25] <StephenLynx> and are you storing dates as actual dates?
[18:22:28] <StephenLynx> that would be the optimal.
[18:23:07] <flok420> my bad I meant that i want to group on the name-field together with the timestamp. yes, storing names together with epoch-times (seconds between now and 1970)
[18:23:55] <synthmeat> StephenLynx: fyi, yes, it was me. query value was undefined. not sure why it acted like it found something and created still, but that was it
[18:24:13] <StephenLynx> hm
[18:24:21] <StephenLynx> ok, I get the part of using the name along it.
[18:24:39] <StephenLynx> but consider using date objects, that allows you to perform operations on the date at a database level.
[18:24:43] <StephenLynx> that are more intuitive.
[18:25:07] <StephenLynx> and the problem is
[18:25:14] <StephenLynx> that mongo tries to just parse the javascript of the query.
[18:25:31] <StephenLynx> it can't tell you very well where the error is because of how json is formatted.
[18:25:39] <StephenLynx> try formatting it somewhere
[18:25:51] <StephenLynx> that should allow you to see better where you made a mistake.
[18:27:45] <flok420> oh hang on it indeed has nothing to do with that maths I added. if I remove it (and only group on name) it still complains about a string! ok thanks sofar, I now have something to work with
[19:31:38] <najah> Hi all, I'm a young sysadmin and I use mongodb with docker. I would like to know things about journal in mongodb. I would like to know what are pros and cons about journal in mongodb for a little application. It is working like RDBMS ? I mean... Can we start to a special point in the journal before error humain ? Thanks very much
[19:33:42] <treta> I'm getting this message all the time, it's not fun anymore "Caused by: com.mongodb.MongoTimeoutException: Timed out while waiting for a server that matches AnyServerSelector{} after 1000 ms"
[20:10:40] <frenchiie> might someone be able to help me with this question? http://stackoverflow.com/questions/32444803/pymongo-parameters-of-function-find-one-and-update
[20:53:58] <sqram> Is it possible to prepend a value to a document's field if it's an array?
[20:56:05] <sqram> for instance, if i have a document like {id:4, children: [4,3] } ... can i update the values of children in a single update command? or do i have to findOne({id:4}), alter the chuldren array, and then update
[21:00:15] <StephenLynx> you can update elements in arrays.
[21:00:46] <sqram> ah thank you.update operators
[21:01:25] <sqram> now going a step futher..say i want to update multiple documents - determined by their unique ids. is it possible?
[21:02:29] <sqram> for instance, toupdate = [3,7,8,5] <-- these would be id fields of different documents. must i run a loop to update each one, or can a single update call do it?
[21:02:51] <sqram> i di dsee multi: true in the docs, but seems like it would have to be a field value shared among the documents. these are unique
[21:04:43] <mprelude> Derick: You still around?
[21:17:46] <mprelude> Derick: When I'm creating HMACs, what format am I looking to get the digest out in?
[21:18:21] <mprelude> Derick: Obviously, hashing a string of a hex num will be entirely different to hashing the same num in decimal or binary.
[21:27:30] <Derick> mprelude: I believe binary, but not near a computer for me to check it
[21:31:55] <mprelude> Derick: Doesn't seem to say anywhere in the spec. :(
[21:32:20] <mprelude> Would be nice if the example showed all the intermediate values so we had something to test against.
[21:32:49] <mprelude> And could easily debug where an implementation is going wrong.
[21:37:16] <Derick> yeah :-/ I remember us (PHP people) struggling with it too
[21:38:20] <mprelude> Like, the proof in the spec, MC2T8BvbmWRckDw8oWl5IVghwCY= ... appears to base64_decode to something utterly useless.
[21:38:30] <mprelude> "0-“ð™d\\<<¡iy!X!À&"
[21:38:42] <Derick> yeah, binary data does that
[21:39:12] <mprelude> I'm not sure what it represents, I don't know how to test whether my implementation is right :(
[21:39:17] <Derick> mprelude: maybe you could step through the PHP driver's running of things?
[21:39:29] <Derick> with a debugger (like gdb)?
[21:39:29] <mprelude> Would help if I knew PHP. *grin*
[21:39:35] <Derick> don't need PHP, just C ;-)
[21:40:03] <mprelude> Even worse for me ;) I could probably step through PHP with perl knowledge.
[21:40:45] <Derick> the PHP driver is written in C though
[21:41:38] <mprelude> Yeah, my C isn't good.
[21:42:26] <Derick> which language are you doing this in?
[21:42:33] <mprelude> Common lisp
[21:42:39] <Derick> ok
[21:42:45] <Derick> now that is something I don't udnerstand :-)
[21:43:10] <mprelude> I get that a lot :D
[21:43:25] <mprelude> I'm trying to
[21:43:31] <mprelude> a. Implement scram-sha1 as a library.
[21:43:39] <mprelude> b. Implement mongo driver auth based on it.
[21:43:59] <Derick> i wonder whether our c driver implements it
[21:44:46] <Derick> it does
[21:46:05] <mprelude> There's an existing cl driver, but it uses the old auth
[21:46:13] <mprelude> So I'm forking the library and trying to implement SCRAM.
[21:46:59] <Derick> reading through the c-driver... this is not easy
[21:47:13] <mprelude> Yeah, not expecting easy. :)
[21:47:19] <Derick> but it reads easily enough
[21:47:36] <mprelude> Can I see the part that generates the client final message?
[21:48:02] <Derick> is that step2 or?
[21:48:05] <mprelude> http://stackoverflow.com/questions/29298346/xmpp-sasl-scram-sha1-authentication I've been working off this answer and the rfc
[21:48:07] <mprelude> Client step2
[21:48:14] <mprelude> See point 8 on that answer
[21:48:35] <Derick> i believe we have a difference to the RFC...
[21:49:22] <Derick> mprelude: pasting that in a PM, if you don't mind?
[21:49:29] <mprelude> Literally the only part I'm really stuck on is what format these hashes should be on. I have to do logical XORs and re-hashes, which could be on binary, decimal, octets, hex idk.
[21:49:31] <mprelude> Sure
[21:50:02] <Derick> I expect it to be binary
[21:51:11] <Derick> mprelude: https://github.com/mongodb/mongo-c-driver/blob/master/src/mongoc/mongoc-scram.c#L395 is step 2 in the c driver