PMXBOT Log file Viewer

Help | Karma | Search:

#mongodb logs for Saturday the 20th of February, 2016

(Back to #mongodb overview) (Back to channel listing) (Animate logs)
[00:39:57] <kurushiyama> synapse: Here! Had to track down a huge performance problem, 16h straight
[03:46:38] <kalx> Hi all. Is there a way to query a document to see if my query input array contains all the values that are in a document's array value? So like $.all except matching in the other direction.
[03:47:00] <kalx> eg. document is: { tags: ['b','c','d'] }. I want to query with ['a','b','c','d','e'] and find documents where all the document's tags are in my input array
[03:52:47] <Boomtime> @kalx: https://jira.mongodb.org/browse/SERVER-16896
[03:53:11] <Boomtime> however, you can use aggregation to do it today; https://docs.mongodb.org/manual/reference/operator/aggregation/setIsSubset/#exp._S_setIsSubset
[03:53:35] <kalx> will check those out, much appreciated Boomtime
[04:16:54] <kalx> @Boomtime: Tested $setIsSubset, worked but only in the context of an aggregate call. I'm trying to do this within a findOneAndDelete call (atomic grab+delete for perfomance reasons). Any idea if that's possible? Seems not, but researching in meantime
[04:41:33] <kalx> @Boomtime: Just to share. Found a workaround but will likely destroy performance for larger collections. As a find condition -> {my_field: {$not: {$elemMatch: {$nin: ['a','b','c','d','e']}}}} . Too many negatives makes it confusing... essentially, finds any document with my_field array where none of the array elements can satisfy not being in the input list.
[04:42:28] <kalx> so not happy about it (even the other one $setIsSubset likely would have been not good performance wise)
[04:48:53] <Boomtime> @kalx: yeah, i know of that workaround and it suffers the problem you recognized - it cannot use an index, literally there isn't one that can be used - the aggregation pipeline _can_ - it won't be terribly efficient, but it should be much better than the $not/$elemMatch/$nin abomination
[05:07:44] <Trinity> i'm working with mongo in node.js and i'm trying to use a string variable for a key in the document
[05:07:46] <Trinity> is this possible?
[05:08:30] <Logicgate> Show some code Trinity
[05:10:04] <Trinity> Logicgate, http://pastebin.com/yMD1bs6G
[05:10:34] <Trinity> shell shows that the key 'throttleKey' is being created/updated instead of 'myIndex'
[05:10:58] <Trinity> ah i think I might be able to create the object and just pass it along?
[05:13:40] <Trinity> nvmd I dont think that's working
[05:14:55] <Boomtime> @Trinity: you can't do what you want in JSON - you'll need to use a real javascript array to create it
[05:15:20] <Boomtime> right now you are describing the update using JSON syntax - which will assume that all keys are literals
[05:16:01] <Boomtime> create a javascript object instead, even if that means instantiating all other members using JSON first, then adding one more entry using javascript array syntax
[05:16:04] <kalx> @Boomtime: Unfortunately, aggregate leads to other trouble. Need to find+delete or mark processed at a rapid pace which means atomicity is best (Findanddelete or findandupdate). Aggregate would lead to a multistep process which is ultimately doomed.
[05:17:23] <Boomtime> you know that all findAndUpdate type operations are multi-step right?
[05:17:51] <kalx> Sorry, FindOneAndDelete/FindOneAndUpdate*
[05:17:52] <Boomtime> i mean, they can do what they say on the packet - which is assure atomicity, but so can you
[05:18:57] <Boomtime> if you find a document that matches your special query and then use an update which specifies the _id you can hit only that document - even if there is a chance of an ABA problem, you can just use another field as a sequencer to avoid it
[05:22:10] <Trinity> Boomtime, thanks got it working
[05:23:22] <Logicgate> Trinity, you would have to create an object first.
[05:23:39] <Logicgate> Then object[throttleKey] = ..
[05:23:40] <Trinity> Logicgate, thanks for the help I got it working with Boomtime's suggestion
[05:24:14] <kalx> Issue is more to avoid multiple processors from identifying the same task to process. Sure, we can easily prevent multiple from grabbing the same record, but then one fails and has to re-identify + grab again. With very high concurrency, it's easy to run into perf problems there.
[05:28:19] <Boomtime> so don't re-identify - use a cursor - then loop 'update' until you get one that succeeds - reads are much cheaper than writes, so a miss on the read portion of an efficient update doesn't matter so much
[05:28:46] <Boomtime> also, given that in order to 'miss' one other _must_ have succeeded, the system must be making progress
[12:33:19] <cdunklau> i don't get this... i'm trying to do the update data part of the mongodb getting started thing (https://docs.mongodb.org/getting-started/node/update/), here's my code https://gist.github.com/cdunklau/7eef9f4a3280007bf3ff
[12:33:23] <cdunklau> when i run that, it just outputs undefined, and the object isn't changed
[12:33:39] <cdunklau> what am i missing, and what things could I do to find out what's going on?
[12:34:07] <cdunklau> using the same filter and update in the mongo shell, it works
[12:56:56] <cdunklau> aha. i didn't have an assert.equal(err, null) so the error wasn't getting outputted
[13:02:42] <cdunklau> so here's the error https://gist.github.com/cdunklau/ec2060bfc5efd9d6c742
[13:06:34] <cdunklau> looks like there's something up with my mongodb package. when i do "npm version mongodb" it says it can't find the package.json but when i look in the node_modules/mongodb i find it, and it shows me this: https://gist.github.com/8e0610dadbd39be40f01
[13:07:40] <synapse> kurushiyama are you still there
[13:07:51] <kurushiyama> Aye
[13:08:06] <synapse> re: our performance issue regarding AJAX and large responses
[13:08:49] <kurushiyama> synapse: ... ...Had we? You were asking for other poor lads spending their Friday... I was the other one ;)
[13:09:09] <kurushiyama> synapse: But shoot, and I will try to help.
[13:09:27] <synapse> As I said I done some tests. I have a small db with under 100 entries, and I also messed about with using the "restaurants" collection example dataset which is very big. The app works fine with the former and on the latter the whole app gets killed on like 4 queries
[13:09:47] <kurushiyama> synapse: MongoU?
[13:09:50] <synapse> ahh So you was only responding to the last part of my query
[13:10:54] <synapse> did you track down your issue lol
[13:11:14] <kurushiyama> synapse: Yes. But go on. This is a MongoU assignment, iirc?
[13:11:44] <synapse> This is just pure JS server side using simple mongodb driver and AJAX client side, no frameworks or libraries
[13:12:02] <synapse> node.js/js
[13:12:40] <synapse> So basically my questions is A) is this normal behaviour for an app that has no server side result restrictions?
[13:13:02] <synapse> Is limiting the results and frequency of queries imperative?
[13:13:24] <synapse> In every scenario or just when DB's are monsterous in size
[13:13:57] <kurushiyama> No and no.
[13:14:40] <synapse> Well I am shocked to hear that because I have no idea then why the app would die when returning large responses
[13:14:54] <kurushiyama> If just a few queries are killing you... you might want to have a look at the queries and the according indices.
[13:15:13] <synapse> in what sense?
[13:15:26] <synapse> like limiting the broadness?
[13:15:28] <kurushiyama> synapse: And you definetly want to use a cursor and not load thw hole result set ;)
[13:15:31] <kurushiyama> synapse: No
[13:15:46] <kurushiyama> synapse: I dont know about node, so lets talk in shell terms
[13:15:50] <synapse> ok
[13:16:22] <kurushiyama> for example, if you have db.coll.find({foo:"bar",answer :42})
[13:16:46] <olivervscreeper> Hi - I'm trying to get the PHP Mongo driver installed. I've installed the MongoDB driver (not Mongo), and phpinfo() shows it as enabled, but if I try to create a connection with new Mongo() it says that the class is not found. Any ideas? [13:09] == Cannot send to channel: #mongodb
[13:17:08] <kurushiyama> you should have an index on those two fields (order matters!). So you need to check wether an according index exists.
[13:17:42] <synapse> I am lost when you talk about indexes
[13:18:01] <synapse> kurushiyama I already am using a cursor
[13:18:02] <kurushiyama> synapse: Have a look at the docs.
[13:18:15] <kurushiyama> synapse: Indexing is _extremely_ important
[13:18:26] <kurushiyama> synapse: But let me explain.
[13:18:28] <synapse> I see
[13:18:31] <synapse> would I know if I was indexing
[13:18:38] <kurushiyama> synapse: Yes.
[13:18:48] <kurushiyama> synapse: Take the above query
[13:18:53] <synapse> yes
[13:19:15] <kurushiyama> without an index, mongod would need to read each and every document from disk to determine wethe the document matches.
[13:19:16] <olivervscreeper> My issue's on StackOverflow if that explains it better: http://stackoverflow.com/questions/35514409/uncaught-error-class-mongoclient-not-found
[13:19:41] <kurushiyama> olivervscreeper: Sorry, when I want Pasta, I see an Italian ;)
[13:19:44] <synapse> I have db.users.find({ $name: {$regex: 'query', $options: 'i'}});
[13:19:57] <kurushiyama> $name?
[13:20:23] <synapse> I have db.users.find({ 'name': {$regex: 'query', $options: 'i'}}); I mean
[13:20:27] <olivervscreeper> kurushiyama: I don't really know what you mean :/
[13:20:54] <kurushiyama> olivervscreeper: Programmable Hyperlink Pasta. Just kidding a bit... ;)
[13:21:01] <kurushiyama> ok
[13:21:11] <olivervscreeper> Do you know what the problem might be?
[13:21:33] <kurushiyama> synapse: ok. Since persumably you do not have an index on "name", think for a sec what this query does...
[13:21:44] <synapse> kurushiyama In simple terms can you describe how I'd alter the query to include indexing
[13:21:53] <kurushiyama> olivervscreeper: No, since I dont use PHP
[13:22:07] <olivervscreeper> Should I ask in the PHP channel then?
[13:22:16] <synapse> kurushiyama I know it's extremely taxing on the database
[13:22:18] <kurushiyama> synapse: You have to understand first. I teach you how to fish, not simply give you one.
[13:22:24] <synapse> of course
[13:22:27] <synapse> :)
[13:22:38] <synapse> I like that saying
[13:22:45] <synapse> it's new to me
[13:22:48] <synapse> but please continue
[13:23:01] <kurushiyama> synapse: So, what happens is that each and every document is read from disk and a regex is applied then...
[13:23:32] <kurushiyama> synapse: (a bit simplified, since there are RAM caches, but take it as a good rule of thumb)
[13:23:39] <synapse> yep
[13:23:47] <kurushiyama> synapse: So what you want is to create an index on that field.
[13:23:48] <synapse> so how is this stopped
[13:24:01] <synapse> ok
[13:25:44] <kurushiyama> synapse: An index basically is a specialized data structure which can be searched efficiently and will be kept in RAM, if at all possible. It holds the values of the indexed field(s) and references the position of the document in the data files.
[13:26:28] <synapse> like a pointer?
[13:28:38] <kurushiyama> synapse: Yep.
[13:28:48] <kurushiyama> But it goes even further
[13:30:02] <kurushiyama> When you do a query on an indexed field, the match is done against the index, and only then the matching docs are read from the disc, (vastly) reducing latency (RAM is orders of magnitude faster than any disk)
[13:31:15] <synapse> Ahh so basically if the query checks a significant amount of docs more than what is returned eg, checked 10,000 docs to return 2 results, this is a good example where an index would be ideal
[13:31:28] <kurushiyama> synapse: exactly!
[13:31:46] <synapse> I didn't know about execution stats
[13:31:51] <synapse> :D
[13:32:16] <synapse> I believe that's what my other query was doing, checking like a million docs to return 2 results lol
[13:32:34] <synapse> So I need to implement a solution for this
[13:32:49] <kurushiyama> synapse: So, now you go figure on how to create a proper index for your query. aka: read the acording docs!
[13:33:03] <synapse> I do a lot of reading in my life
[13:33:10] <kurushiyama> synapse: It is well documented...
[13:33:11] <synapse> it never ends :)
[13:33:26] <kurushiyama> synapse: It gets worse.
[13:33:26] <synapse> yeah I pulled a page up on execution stats
[13:33:32] <kurushiyama> synapse: Leave em
[13:33:41] <kurushiyama> synapse: Learn the basics first.
[13:34:08] <kurushiyama> synapse: If in doubt, user ".explain()" on your query.
[13:34:14] <synapse> I see
[13:34:36] <kurushiyama> synapse: COLLSCAN is what you dont want to see. IXSCAN is desired.
[13:37:08] <synapse> I see COLLSCAN atm but that's expected with no index
[13:38:20] <synapse> an index is basically an extra entry on the key:value that incorporates a built in limitation?
[13:46:02] <kurushiyama> synapse: No...
[13:46:51] <kurushiyama> synapse: https://docs.mongodb.org/manual/core/index-creation/
[13:47:10] <kurushiyama> synapse: it is a serverside construct.
[13:47:58] <kurushiyama> synapse: after your index is built, the query looks exactly the same, but mongod will utilize the index to make the query faster ;)
[13:54:10] <synapse> kurushiyama incredible improvement
[13:54:29] <kurushiyama> synapse: I guess several orders of magnitude...
[13:54:34] <synapse> Before it was looking at 25,300 docs to return like <10 results
[13:54:50] <synapse> now it looks at 400 on the same query
[13:55:10] <kurushiyama> synapse: how long does it take?
[13:55:24] <synapse> 30ms
[13:55:28] <kurushiyama> ;)
[13:55:32] <synapse> :D
[13:55:41] <synapse> I need to try it out though in my app
[13:55:45] <synapse> see if it still dies
[13:55:56] <synapse> if so maybe ad more indexes
[13:56:00] <kurushiyama> If it does, there is another problem.
[13:56:00] <synapse> add*
[13:56:14] <synapse> ok lets see
[13:57:45] <kurushiyama> synapse: Ok, a basic: Identify the questions you have on your data, model your data ___according to your questions___ (as opposed to model by entities) and add the indexes for the fields your query. That's about all there is to it (details and intricacies ommited) and this is what about 90% of the people ranting about MongoDB get wrong.
[13:57:57] <synapse> should I add .hint to the query to force the use of the index I added
[13:58:08] <kurushiyama> synapse: This usually is not necessary.
[13:58:14] <synapse> mmm
[13:58:32] <synapse> I might add it for verbosity in my code
[13:58:36] <kurushiyama> synapse: But on the other hand: you are asking a JS illiterate... ;)
[13:59:06] <kurushiyama> synapse: I wouldnt. If you change the index (NoSQL is pretty fluid), the hint will go wrong.
[13:59:28] <synapse> comment then :D
[13:59:40] <synapse> or bleh just leave it
[13:59:57] <kurushiyama> Test first
[13:59:58] <synapse> that's a good reason not to use it
[14:00:11] <kurushiyama> If it does not use the index, add a commented hint
[14:03:08] <synapse> didn't work with the app, it died
[14:03:20] <synapse> worked on the shell when i tested
[14:03:46] <kurushiyama> synapse: Conclusion?
[14:06:47] <synapse> with .hint it hasn't dies yet upon testing
[14:06:58] <synapse> when added to the app
[14:08:30] <synapse> I know when i tested on the shell without .hint it still used the index but for some reason in the code i can only assume it wasn't using it
[14:08:48] <synapse> but with the addition of it it hasn't crashed
[14:08:56] <kurushiyama> synapse: well... Dont ask me about JS.
[14:09:10] <synapse> you more of a PHP kinda guy? :)
[14:10:13] <kurushiyama> synapse: Yeah. Absolutely. I'd rather change my job for a promising career as a loaf of bread.
[14:10:41] <synapse> If I had a little timer to the onkeyup AJAX query so it doesn't send off queries every single time it should be a lot more solid
[14:11:06] <kurushiyama> synapse: Chinese? Urdlu?
[14:11:34] <synapse> I prefer kebabs
[14:11:50] <kurushiyama> I once got a mouseover working... And that is about my JS experience.
[14:12:09] <synapse> I'm using it client side and server side
[14:12:26] <synapse> it's nice :)
[14:12:38] <synapse> event driven and non-blocking
[14:13:11] <synapse> I do like PHP when it first became popular it was very handy
[14:13:12] <kurushiyama> synapse: Single-threaded, no type safety and callback hell. I tried node _very_ shortly.
[14:13:30] <synapse> PHP is type safe??
[14:13:37] <kurushiyama> I dont do PHP
[14:13:40] <synapse> oh
[14:13:42] <kurushiyama> I hate it
[14:13:46] <synapse> what do you do?
[14:13:51] <kurushiyama> Programmable Hyperlink Pasta.
[14:13:57] <synapse> lol
[14:14:02] <kurushiyama> I do Go and used to do Java
[14:14:07] <synapse> Oh cool
[14:14:15] <synapse> I'm very curious to try Go
[14:14:34] <synapse> Is it really the rolls royce of server side web app building?
[14:15:14] <synapse> kurushiyama callback hell is being phased out with the use of chained methods / promises :)
[14:31:41] <kurushiyama> synapse: Rolls Royce? Not really, since it lacks mature frameworks, imho. The better comparison would be Ferrari – it is __fast__. And you compile static binaries. Deployment can be done by copying a (one) file from one machine to the next, if done properly.
[14:33:12] <kurushiyama> synapse: Where it excels is API building. This is where Go really rocks. Think of a Rolls Royce with a formula 1 engine and the capabilities of using it.
[14:34:03] <StephenLynx> kurushiyama:
[14:34:15] <kurushiyama> StephenLynx: Aye?
[14:34:22] <StephenLynx> 1- callback hell is just as bad as any other uncontrolled nesting. no differen than nesting too many loops or conditions
[14:34:40] <StephenLynx> 2- single threaded isn't an issue because of how you have the cluster native module to use all cores of the cpu
[14:35:07] <StephenLynx> 3- neither is type safety because you have unambiguous operators such as '==='
[14:35:58] <kurushiyama> StephenLynx: It still is concurrent instead of parallel. Agreed, node does an impressive job, but still it is single threaded ;) And type safety enforcing during runtime is not exactly the problem. ;)
[14:36:33] <StephenLynx> what is the problem with being concurrent instead of parallel?
[14:37:00] <StephenLynx> you can use the whole CPU if you need.
[14:38:54] <kurushiyama> StephenLynx: Well, let's say you have like 4 users and 4 cores. With true parallelism, those would be processed in parallel. With concurrency, each worker from time to time yields. It is not that this would not work, but it makes less good use of the resources. When it comes to scaling out, the difference will become noticeable sooner or later.
[14:39:20] <StephenLynx> you got it wrong.
[14:39:30] <kurushiyama> StephenLynx: So enlighten me.
[14:39:43] <StephenLynx> they will be processed in parallel because each one would be processed by a different process running on a different core.
[14:39:53] <StephenLynx> node has this feature built in.
[14:40:28] <StephenLynx> now if it were processing 8 requests on 4 cores, now you have 4 in parallel and the other 4 concurrently.
[14:40:49] <kurushiyama> StephenLynx: Huh? Wait a sec. As far as I know, v8 is inherently singlethreaded.
[14:40:53] <StephenLynx> yes.
[14:40:57] <StephenLynx> each process is single threaded
[14:41:04] <StephenLynx> you just have more processes to use all cores.
[14:41:34] <StephenLynx> 1 master process and as many worker processes, usually you spawn 1 worker for each core
[14:42:13] <StephenLynx> otherwise node software would be severely capped.
[14:43:24] <kurushiyama> StephenLynx: So, if I get you right (paraphrasing to check if I got you right): You have a single master node process, dispatching request to worker processes, for which one is spawned per CPU?
[14:44:15] <cdunklau> i'm following the getting started guide for node (https://docs.mongodb.org/getting-started/node/update/) and I've hit a wall with the update portion. code and error is here: https://gist.github.com/cdunklau/7eef9f4a3280007bf3ff
[14:44:36] <StephenLynx> yeah, pretty much that.
[14:44:53] <StephenLynx> you spawn the worker processes manually, though. so 1 per core is just what you usually want.
[14:45:17] <StephenLynx> and they can all share the same HTTP listener transparently.
[14:45:21] <StephenLynx> node handles that internally.
[14:45:44] <kurushiyama> StephenLynx: Interesting. One point less on my long list "Why not to do node..." ;) Thanks for the info! But I think cdunklau needs a node specialist, now ;)
[14:47:32] <synapse> kurushiyama well it looks like you have 3 points less not to do node now ;)
[14:47:52] <synapse> don't do loose comparisons === not == as an example
[14:48:58] <kurushiyama> synapse: Nah, that's runtime. Last but not least – I used to be a sysadmin. I like go for the easy way to have a natively compiled application including all resources deployed by merely copying a file.
[14:49:24] <kurushiyama> synapse: No dependencies aside from libstdc
[14:49:49] <synapse> cdunklau $currentDate should be 'currentDate' as it's a field not an in-built mongo option on line 13 of pastebin
[14:50:08] <StephenLynx> compiled application are not too practical for quickly iterating software.
[14:50:22] <StephenLynx> and deploying is not an issue on scripted software when you use git to deploy.
[14:50:30] <cdunklau> synapse: ah, looks like that's new in 2.6 :(
[14:50:40] <cdunklau> i'm on 2.4.
[14:50:42] <cdunklau> synapse: thanks!
[14:52:04] <synapse> cdunklau hmm it's actually an option I didn't know about that one
[14:52:43] <synapse> but there lies your issue with the versioning probably so see if it helps
[14:53:21] <cdunklau> synapse: currentDate isn't a field in the document, or did you mean something else?
[14:55:43] <synapse> $currentDate is valid
[14:57:36] <cdunklau> synapse: right i'm just curious why you suggested that $currentDate should be currentDate (before you realized it was valid)
[14:58:03] <cdunklau> synapse: did you think i just made a typo in trying to replace an embedded document or something?
[14:59:27] <synapse> I didn't think it was a recognised mongo field, but it's got an issue with that line
[14:59:46] <cdunklau> ah i gotcha
[15:00:27] <synapse> cdunklau I see it
[15:00:33] <synapse> Looks like you missed some braces
[15:01:48] <synapse> {$currentDate: {'lastModified':'true'}}
[15:01:58] <synapse> as it's an embedded key:value
[15:09:54] <synapse> cdunklau god I see it
[15:10:05] <synapse> you missed a bracket closing off the update
[15:10:13] <synapse> you have a comma there
[15:10:49] <cdunklau> synapse: where? https://gist.github.com/cdunklau/7eef9f4a3280007bf3ff
[15:11:04] <cdunklau> synapse: it's all good now, i removed the currentDate thing because it doesn't work
[15:11:10] <cdunklau> (in 2.4 which i have)
[15:11:30] <synapse> on line 14 of the pastebin
[15:11:35] <synapse> the original one you sent
[15:11:48] <synapse> the updateOne( <--- isn't closed
[15:12:17] <cdunklau> synapse: it's closed on line 20...
[15:12:54] <cdunklau> the original one meaning the one i sent ~30 min ago or the one i sent a while before
[15:14:27] <synapse> cdunklau the syntax you used is wrong, you don't chain a callback function on the end of the query like that
[15:14:33] <synapse> what example are you following
[15:14:51] <cdunklau> synapse: https://docs.mongodb.org/getting-started/node/update/
[15:15:08] <cdunklau> synapse: and the docs say i can...
[15:15:21] <cdunklau> http://mongodb.github.io/node-mongodb-native/2.0/api/Collection.html?_ga=1.217822224.729576732.1455824695#replaceOne
[15:15:28] <cdunklau> whoops wrong one
[15:15:29] <synapse> yeah that's some weirdness.
[15:15:37] <synapse> I see it but it looks odd
[15:15:41] <cdunklau> but it's basically the same signature
[15:16:02] <synapse> I tend to build the query as it would be sent on the shell cdunklau and then put it inside a function with a callback
[15:16:34] <cdunklau> yeah i'm *super* new to all this. i imagine i'll figure out or read about better strategies
[15:16:38] <synapse> because the query works fine on my shell I tried it on my restaurants collection cdunklau
[15:16:59] <cdunklau> sure, the query is fine now for me, now that i took out $currentDate
[15:17:15] <cdunklau> (because again i'm on 2.4)
[15:17:38] <synapse> Yes your error pointed to that line so your mongo version / node versions are important. I've had many errors with version issues
[15:17:40] <cdunklau> synapse: i really appreciate you going into depth on this, but i have a feeling i communicated the status or something badly
[15:18:05] <cdunklau> you helped me figure out the issue like 20 minutes ago :)
[15:18:13] <synapse> lol ok :D
[15:18:46] <synapse> I recently had an issue with socket.io that was all down to a node version
[15:18:55] <synapse> there's a serious bug in v4 of node
[15:20:05] <kurushiyama> cdunklau: Mongo 2.4?
[15:20:38] <cdunklau> kurushiyama: yes
[15:21:09] <kurushiyama> cdunklau: Archeologist out of hobby or profession? ;)
[15:21:31] <Tachaikowsky> Hi
[15:21:49] <kurushiyama> Tachaikowsky: Hoi!
[15:21:51] <Tachaikowsky> If something is written to mongodb using mongoose and secret key, can I use mongodb plugin now to read it without a secretkey?
[15:22:20] <cdunklau> the problem was originally difficult to figure out because the example with updates doesn't have the error === null assertion. i don't know any better so i took it out of my code. then i just got undefined in the console and was confused
[15:22:33] <kurushiyama> Tachaikowsky: If you are talking of the cluster key file: nafaik.
[15:22:51] <cdunklau> the examples before (querying and such) have assert.equal(err, null) in the callback
[15:23:06] <cdunklau> kurushiyama: it's just the one on debian sta(b)le
[15:23:17] <Tachaikowsky> kurushima, in my case - the key is a 'secretword' that is in my config.js (I am working with node.js)
[15:23:58] <kurushiyama> synapse! StephenLynx! node.js question...
[15:24:14] <synapse> I have never used mongoose
[15:24:33] <kurushiyama> cdunklau: Well. As said. Thats a bit like archeology.
[15:24:37] <cdunklau> sure
[15:24:57] <Derick> please don't run 2.4 anymore
[15:24:58] <cdunklau> i'm learning mongo and node with the hopes of moving from tech support to development at my company
[15:25:03] <kurushiyama> cdunklau: You might likely run into driver problems, unless you are very careful etc.
[15:25:06] <cdunklau> and i'll bet we're using an old old version
[15:25:18] <cdunklau> kurushiyama: okay, i'll keep that in mind. thanks!
[15:25:19] <synapse> cdunklau that syntax looks so odd to me, where the function was tagged onto the query, I'm a beginner too but that does look strange as
[15:26:02] <kurushiyama> cdunklau: Use a docker image of a more recent version of Mongo
[15:26:09] <cdunklau> synapse: seemed natural enough to me. i'm just getting started with mongo, but i've played with node and other async stuff to become fairly comfortable with that style
[15:26:31] <cdunklau> ...enough to become...
[15:27:36] <synapse> ahh so updateOne in the mongo driver api docs must define an addition callback as a parameter, something you wouldn't have on the shell of course
[15:27:46] <synapse> additional*
[15:28:09] <cdunklau> oh god one of our products is on 2.2
[15:28:38] <synapse> heh
[15:29:29] <kurushiyama> cdunklau: Well. Archeologist by profession, so much is clear
[15:29:35] <cdunklau> apparently
[15:31:05] <cdunklau> but hey it only listens on localhost, that stops all the hacks right?
[15:31:07] <cdunklau> ...right?
[15:31:46] <kurushiyama> cdunklau: As long as you assume localhost as being safe, sure. Though I wouldnt
[15:32:00] <cdunklau> that was mostly a joke
[15:33:03] <kurushiyama> cdunklau: Sorry ;) Well, as for your playing around, I'd use a docker image for a more recent version of MongoDB. And you might want to have a look at MongoU
[15:33:08] <synapse> I think my synapses are having a day off today
[15:35:42] <StephenLynx> kurushiyama, back
[15:35:47] <StephenLynx> what was the question?
[15:36:24] <cdunklau> kurushiyama: i'll look into it, thanks
[15:36:27] <kurushiyama> StephenLynx: cdunklau has some problems with node. I didnt really get it.
[15:36:32] <cdunklau> had*
[15:36:53] <cdunklau> and an archeological curiosity
[15:36:54] <StephenLynx> welp
[15:37:19] <kurushiyama> StephenLynx: vastly OT, but do you use bower?
[15:37:22] <synapse> Tachaikowsky just taking a random stab at this as it's not something I know much about but I assume the secret key is only there to validate a session cookie on the client, I don't see why it would affect your ability to read a document from the db?
[15:37:30] <StephenLynx> kurushiyama, god, no.
[15:37:37] <StephenLynx> I avoid any building system like the plague.
[15:37:38] <Tachaikowsky> synapse, okay let me try it
[15:37:46] <StephenLynx> building systems, web frameworks, ODM
[15:37:49] <StephenLynx> all cancer.
[15:37:58] <kurushiyama> StephenLynx: Hahahhaaa.
[15:38:29] <kurushiyama> StephenLynx: Do you know the times when men were real men and compiled their OS on their own?
[15:38:54] <StephenLynx> kek
[15:39:13] <StephenLynx> I just do my best to avoid unnecessary complexity.
[15:40:27] <kurushiyama> StephenLynx: I dont get the point of bower. In which way is it different than to have a git submodule?
[15:41:36] <synapse> kurushiyama when they wrote drivers upon boot? :)
[15:42:03] <kurushiyama> synapse: Pfft. Premature writing is the root of all evil. On demand!!!
[15:42:34] <synapse> lol
[15:43:44] <synapse> I need coffee
[15:43:57] <StephenLynx> kurushiyama, dont try to understand webdevs.
[15:44:15] <kurushiyama> StephenLynx: I have to. Have to write a frontend...:/
[15:44:16] <StephenLynx> they are just the thing about a thousand monkeys on typewriters
[15:44:31] <StephenLynx> eventually spitting out somewhat usable software
[15:45:07] <cdunklau> so there's a lot of mongo hate, some of it's based on old issues that have been fixed, but the arguments about data-you-think-isnt-relational-until-it-is seem to be pretty reasonable. I'm sure you guys have to deal with the hate a lot in here, so i'll try to keep it specific and opinion-free.... What are the use cases where mongo really shines?
[15:45:28] <kurushiyama> StephenLynx: Infinite Monkey theorem. Sounds about right... I could not describe how much I hate frontend development.
[15:45:48] <kurushiyama> cdunklau: Let me give you an example
[15:45:56] <StephenLynx> large datasets of data that are not too related and don't follow a strict schema, cdunklau
[15:46:07] <kurushiyama> cdunklau: Many people say MongoDB can not be used for financial transactions.
[15:46:23] <kurushiyama> cdunklau: Which could not be farther from the truth.
[15:46:42] <kurushiyama> cdunklau: When banks do bookings, they document the actual booking.
[15:46:57] <kurushiyama> cdunklau: So they have accounts and bookings.
[15:46:59] <StephenLynx> well, if your system depends deeply on transactions, mongo will be not too optimal. because you will have to implement the transactions on software code.
[15:48:06] <kurushiyama> cdunklau: A booking would be {_id:someId, deductFrom: acctId1, transferTo: acctId2, amount:200.00}
[15:48:25] <kurushiyama> cdunklau: Atomic insert
[15:49:03] <StephenLynx> is nothing inherent about financial software, though.
[15:49:05] <synapse> I used to use postgres but I recently switched to mongo purely for the experience of using it with node, the JSON format is handy but I think postgres supports that too
[15:49:06] <StephenLynx> its about transactions
[15:49:16] <kurushiyama> cdunklau: ever wondered why it takes even directly connected cash machines a while to find out wether you can get some money? Because they will do an aggregation on the bookings.
[15:49:40] <kurushiyama> StephenLynx: But that's an example you often see in the context "You cant do this with Mongo"...
[15:50:28] <StephenLynx> well, any "you can't do X" is a bad example.
[15:50:35] <kurushiyama> cdunklau: To make a long story short: There are _very_ few things you absolutely can not do with MongoDB. Everything else is prone to sensible data modelling.
[15:50:37] <StephenLynx> one thing is doing, the other is doing optimally.
[15:51:14] <kurushiyama> StephenLynx: Actualy, the format of bookings with a referall to accounts is how banks exchange bookings ;)
[15:51:15] <StephenLynx> if your implementation is X tool is a flawed hack, you might as well be using Y tool.
[15:51:27] <Tachaikowsky> synapse, to confirm - YES. It works. The secret key is associated with the plugin, and the db details can be directly accessed from mongoshell
[15:51:33] <StephenLynx> again, nothing about banks and stuff.
[15:52:30] <synapse> Tachaikowsky thanks for the update, that's how I expected it to work but as I say it's just on instinct as I don't know anything about mongoose. Secret key's on the server side are used to validate the clients session
[15:52:48] <StephenLynx> >mongoose
[15:52:49] <StephenLynx> don't.
[15:53:07] <Tachaikowsky> StephenLynx, yes I am moving away from mongoose - exactly why these questions are being asked :)
[15:53:11] <kurushiyama> StephenLynx: I get you. It was just an example. Transactions imho are overestimated. From my experience, if you need a transaction spanning multiple tables, that's a good candidate for an according document in MongoDB.
[15:53:32] <synapse> I was told fiercely to not use MongoDB but the truth is I didn't really understand why the said person was so adamant about it
[15:53:39] <StephenLynx> constant runtime aggregations are too slow, though.
[15:53:42] <cdunklau> kurushiyama: well i was wondering about the transaction/data integrity thing too, but i don't know enough about it to really know one way or the other
[15:53:44] <Tachaikowsky> synapse, I just want to confirm - any query that is executable on mongoshell can also be executed through node.js without much modification?
[15:53:56] <StephenLynx> yes
[15:53:57] <cdunklau> kurushiyama: i'm more interested in things i do understand, that is, relations
[15:54:07] <Tachaikowsky> StephenLynx, that yes was for me?
[15:54:09] <StephenLynx> yes
[15:54:13] <synapse> Tachaikowsky that's been my experience they basically match up perfectly
[15:54:23] <StephenLynx> if you use the native driver, that is.
[15:54:27] <kurushiyama> StephenLynx: Well, sort of. db.coll.aggregate([{$match... isnt.
[15:54:27] <Tachaikowsky> synapse, then one wonders the need for mongoose
[15:54:33] <synapse> yeah with the native driver Tachaikowsky
[15:54:35] <Tachaikowsky> synapse, then one wonders about the need for mongoose
[15:54:46] <Tachaikowsky> synapse, by native driver you mean mongodb and then assert right?
[15:54:47] <synapse> Tachaikowsky I've never used mongoose :)
[15:54:53] <StephenLynx> kurushiyama, that is because the terminal supports 2 kinds of aggregation
[15:54:55] <synapse> Tachaikowsky yes
[15:54:56] <StephenLynx> syntaxes, that is.
[15:55:03] <StephenLynx> that one is supported by the driver.
[15:55:06] <kurushiyama> cdunklau: Well, I have _started_ a series of blog posts. Need to finish the second article this WE... never seem to find the time.
[15:55:18] <StephenLynx> the other one isnt.
[15:55:23] <StephenLynx> where you don't have an array.
[15:55:54] <StephenLynx> Tachaikowsky, assert?
[15:59:26] <cdunklau> kurushiyama: please make time, there are not enough of these types of posts
[16:00:33] <kurushiyama> cdunklau: If you enjoyed it, please comment. Most important: Please add what you want to see in the future, so I can focus. Wrinting on data modelling for MongoDB can easily fill a book.
[16:01:21] <cdunklau> best practice documentation is sorely lacking for a *lot* of things
[16:07:27] <cdunklau> kurushiyama: i can't think of anything to contribute... this is so new to me i can't even have an opinion :)
[16:07:42] <kurushiyama> cdunklau: Aye
[16:08:24] <cdunklau> i'm still at the hobbyist-programmer stage, sorely lacking in experience
[16:08:32] <cdunklau> i mean in general, not just with mongo/node/tec
[16:08:34] <cdunklau> etc*
[16:09:17] <cdunklau> and i don't have any formal education, so i can't even really talk about this kind of stuff from a theoretical point of vier
[16:09:20] <cdunklau> view*
[16:19:39] <Tachaikowsky> synapse, can you please help me a second with this?
[16:19:41] <Tachaikowsky> So consider this db schema
[16:19:43] <Tachaikowsky> a document that has '_id' and an 'array of friends' (i.e. array of _ids)
[16:19:45] <Tachaikowsky> Now consider this post api - wherein, a 'friend' in the body.
[16:19:47] <Tachaikowsky> When successful, this 'friend' needs to be pushed into the arrayOfFriends
[16:19:49] <Tachaikowsky> db.users.find({"_id": ObjectId("req.decoded.id")}).update{ $push: { arrayOfFriends: req.body.friend } }
[16:19:51] <Tachaikowsky> The question is, how to write this thing in a 'callback style' in nodejs
[16:27:45] <cdunklau> Tachaikowsky: http://mongodb.github.io/node-mongodb-native/2.0/api/Collection.html#findOneAndUpdate ?
[16:30:27] <Tachaikowsky> cdunklau, thanks a bunch
[16:31:00] <StephenLynx> Tachaikowsky, you are using the worst syntax for that though
[16:31:16] <StephenLynx> it should be db.col.update({query to match},{update to perform})
[16:31:28] <cdunklau> Tachaikowsky: i do hope you're validating the incoming data :)\
[16:31:36] <Tachaikowsky> Oh yes, that's great thanks!
[16:32:35] <StephenLynx> on node you just pass your callback as the last parameter. or if you wish to perform multiple operations on the cursor, like sort and limit, you call toArray() as the latest function and pass the callback for it
[16:33:23] <StephenLynx> blabla.find({query},{projection, if desired}).sort({field and direction}).toArray(function gotArray(error, data){});
[16:33:51] <Tachaikowsky> StephenLynx, I dont quite understand the last part (from toArray)
[16:34:11] <StephenLynx> it gets all the data from the cursor as an array
[16:34:20] <StephenLynx> at once
[16:34:26] <Tachaikowsky> Yeah, that I figured, but why wud I need that?
[16:34:37] <StephenLynx> to get all the data from the cursor as an array :v
[16:34:47] <StephenLynx> v:
[16:35:16] <StephenLynx> your two other options are:
[16:35:25] <StephenLynx> keeping the cursor open and fetching one document at once
[16:35:39] <StephenLynx> performing multiple queries fetching one document at once on multiple cursors.
[16:36:13] <StephenLynx> usually I fetch all as a single array if I know there isn't too much data or use one cursor at a time if I know there might be too much data.
[16:36:44] <StephenLynx> using a single cursor and getting one document at once usually isn't worth.
[16:37:00] <StephenLynx> too much code and your cursor will timeout if your operations take too long
[16:37:11] <StephenLynx> you will never get a timeout on the other two options.
[17:48:20] <synapse> StephenLynx .toArray... damn I've been using a .push all this time
[17:48:56] <StephenLynx> :v
[17:49:03] <StephenLynx> how, exactly?
[17:49:16] <StephenLynx> fetching all the docs and assembling an array?
[17:49:27] <synapse> let me show you StephenLynx what I did in a pastebin.
[17:49:31] <synapse> It's simple
[17:50:33] <synapse> StephenLynx: http://pastebin.com/BZCciPrc .. so I have a private method _searchUsers which never gets called externally
[17:51:00] <StephenLynx> kek
[17:51:10] <StephenLynx> and btw
[17:51:20] <StephenLynx> using assert like that is awful
[17:51:28] <StephenLynx> because you have no means to handle errors.
[17:51:44] <StephenLynx> you should be sending the error on the callback
[17:51:55] <StephenLynx> and let whatever ran the function to handle it.
[17:51:57] <synapse> you don't like what I did? :)
[17:52:12] <StephenLynx> the part of reinventing the wheel is kind of bad
[17:52:16] <synapse> lol
[17:52:21] <StephenLynx> but using assert there is much worse
[17:52:39] <synapse> I like it because it makes the code I use elsewhere more concise
[17:52:53] <StephenLynx> for that case a simple findOne would suffice
[17:52:57] <StephenLynx> It doesn`t matter
[17:53:11] <StephenLynx> your application would just crash were an error to happen.
[17:53:27] <StephenLynx> any crash that is not intended is a failed software.
[17:53:57] <StephenLynx> actually, findOne wouldn't work there, nvm. but a find followed by toarray would
[17:54:09] <synapse> assert would abort the app if an error were to happen
[17:54:14] <StephenLynx> exactly.
[17:54:20] <StephenLynx> you should handle the error.
[17:54:32] <StephenLynx> and your function requires you to pass the db
[17:54:34] <StephenLynx> that is bad too
[17:54:36] <synapse> but I don't see how an error can happen if the query is filtered and processed prior to going in
[17:54:45] <StephenLynx> db failure
[17:54:49] <StephenLynx> plus
[17:54:58] <StephenLynx> you are wasting time by getting the collection every time
[17:55:03] <StephenLynx> you should just keep a reference to it.
[17:55:15] <StephenLynx> instead of passing around the db pointer around.
[17:55:19] <synapse> I appreciate the critique
[17:56:26] <StephenLynx> and I think you could also use $text. don`t know your use case, you should check if it works better than regex for that.
[17:56:38] <StephenLynx> regexes are not trivial.
[17:56:50] <synapse> my function StephenLynx that gets used inside the core of the app doesn't take a db argument, as that's handled on the module page I posted. I just do searchUsers(query, .....
[17:57:29] <StephenLynx> you connect to the db every time you get an instance of search users?
[17:57:47] <StephenLynx> wait
[17:57:49] <synapse> what's the alternative, connection pool?
[17:57:50] <StephenLynx> every time you run
[17:57:51] <StephenLynx> jesus
[17:57:52] <StephenLynx> christ
[17:57:57] <StephenLynx> yes
[17:58:00] <StephenLynx> you connect only once
[17:58:06] <StephenLynx> the driver holds its own connection pool.
[17:58:23] <StephenLynx> you are wasting huge amounts of time there
[17:58:31] <StephenLynx> your code would perform awfully under higher loads.
[17:58:33] <synapse> I only followed the example mongo uses for node.js usage
[17:58:45] <StephenLynx> those examples are just syntax examples.
[17:58:52] <StephenLynx> aside that they are garbage.
[17:59:05] <StephenLynx> proper code would be too large for those examples.
[17:59:08] <StephenLynx> and too abstract.
[17:59:14] <synapse> StephenLynx I stress tested it earlier and it was dying so I knew it needed improving that's why I joined the channel to ask about it, I was told to add indexing which I did
[17:59:32] <StephenLynx> the connection dance isn`t helping either.
[17:59:51] <StephenLynx> change so you only connect once and get a reference to the collections once.
[18:00:00] <StephenLynx> then you reuse the collection reference or db reference if needed.
[18:00:08] <synapse> so what it holds the collections in RAM?
[18:00:16] <StephenLynx> yeah
[18:00:18] <StephenLynx> let me show my code
[18:00:23] <synapse> ahh ty
[18:00:26] <StephenLynx> not the whole colelction of course
[18:00:28] <StephenLynx> just a pointer to it.
[18:00:34] <StephenLynx> and its even resilient
[18:01:10] <StephenLynx> https://gitgud.io/LynxChan/LynxChan/blob/master/src/be/db.js
[18:01:31] <StephenLynx> notice how I just provide functions that return the result of .collection()
[18:01:48] <StephenLynx> when the other modules start they call these functions and store the result.
[18:02:02] <synapse> I see what you mean about it being non-trivial
[18:02:21] <StephenLynx> that is how you make the most of your connections.
[18:02:30] <synapse> that's good of you, appreciate it
[18:02:34] <synapse> I knew the code was bad
[18:02:53] <synapse> The performance was awful on larger collections when i ran multiple queries
[18:02:57] <StephenLynx> yup
[18:03:05] <StephenLynx> the connection dance
[18:03:12] <synapse> connection dance :D
[18:03:47] <synapse> so as this was inside ajax, it's literally connect/dc every time key is pressed!
[18:04:00] <StephenLynx> yup
[18:06:27] <synapse> your code is very professional making lots of checks
[18:06:36] <StephenLynx> yup
[18:06:42] <synapse> It's really good
[18:06:46] <StephenLynx> even when errors are ignored its intentional.
[18:07:05] <StephenLynx> plus
[18:07:17] <StephenLynx> when your software runs multiple sites with some fair activity
[18:07:25] <StephenLynx> edge cases start to crop up :v
[18:07:41] <StephenLynx> endchan.xyz and freech.net are the main sites running this engine
[18:07:54] <StephenLynx> these two don`t get much activity lynxhub.com penumbra.network
[18:08:11] <synapse> edge cases?
[18:09:04] <StephenLynx> things that only happen every other bluemoon but only if you are wearing a rainbow wig on a bathtub made out of cheese
[18:09:49] <StephenLynx> so now you have to check if your bathtub is made out of cheese or porcelain and can`t just assume the later
[18:12:06] <StephenLynx> any software that handles external input works based on assumptions to some degree.
[18:12:18] <synapse> lol
[18:12:52] <StephenLynx> the more you assume, the more unpredictable behaviour you get.
[18:12:57] <synapse> yes it does, I try to assume every possibility if I'm accepting an input
[18:13:05] <StephenLynx> the more you check, the more predictable behaviour you get.
[18:13:13] <synapse> indeed
[18:13:46] <StephenLynx> the more usage a software gets, more of these unpredictable and undesirable behaviour is revealed.
[18:13:55] <synapse> I totally agree I was lazy on the error handling but it wouldn't be that way if it was a fully working app I'm literally just trying to get off the ground with mongo and node ya know?
[18:14:17] <StephenLynx> yeah, but I don't critique based on context.
[18:14:31] <synapse> of course and i appreciate it
[18:14:38] <StephenLynx> i just tell you how it is :v
[18:14:44] <synapse> as you should
[18:15:19] <StephenLynx> i have myself some code that would easily break if the input were some unexpected stuff.
[18:15:37] <StephenLynx> but given their context, I don't mind it.
[18:15:50] <StephenLynx> like how the admin of the site inputs settings.
[18:16:04] <StephenLynx> and how he could input weird stuff to make it break.
[18:16:11] <synapse> I have to admit StephenLynx, Node has been a peculiar language for me to get to grips with as it's very different to traditional OOP or procedural languages I've used before. I found coming up with a way to organise the code a minefield
[18:16:23] <StephenLynx> protip: dont use OOP on js.
[18:16:28] <StephenLynx> js OOP is cancer.
[18:16:36] <StephenLynx> you should use it as a procedural language.
[18:16:41] <synapse> mmm
[18:16:58] <synapse> So you don't think making use of the proto chain is good?
[18:17:02] <StephenLynx> no
[18:17:04] <StephenLynx> its cancer.
[18:17:07] <synapse> interesting
[18:17:18] <StephenLynx> good lord, the whole prototype thing
[18:17:25] <StephenLynx> should be thrown in a fire
[18:17:26] <synapse> I was kinda liking it
[18:17:45] <StephenLynx> anyone with a java or C++ background would look at that and say "lolwut"
[18:17:47] <synapse> But I've never managed a big project and used it
[18:18:03] <StephenLynx> prototyping makes code navigation and maintenance impossible.
[18:18:04] <synapse> StephenLynx I know lol but I just like it for some reason
[18:18:20] <StephenLynx> your code becomes extremely obfustated
[18:18:26] <StephenLynx> obfuscated*
[18:18:37] <StephenLynx> 'this' is useless too.
[18:18:40] <synapse> As I said, I've never had to manage a large scale app
[18:18:51] <StephenLynx> no matter the scale
[18:18:54] <synapse> 'this' was weird before I understood how it works
[18:18:56] <StephenLynx> it makes everything worse.
[18:19:05] <StephenLynx> 'this' is not useless as a concept, just on js.
[18:19:13] <StephenLynx> since you shouldn't be using OOP there in the first place.
[18:19:46] <synapse> The problem is not so much me, but what everyone else does you know? I'f I have to depend on libs I kinda need to follow how they're built
[18:20:00] <StephenLynx> most libs on node are garbage
[18:20:01] <synapse> Well I mean put another way
[18:20:05] <StephenLynx> so as most devs.
[18:20:15] <StephenLynx> and if the lib dictates how your code is written
[18:20:21] <StephenLynx> that is the biggest red flag you could get.
[18:20:24] <StephenLynx> never use a library like that.
[18:20:26] <synapse> I had to be able to understand them when I read them to know what they were doing, and they all make use of the proto chain
[18:20:43] <StephenLynx> you dont. you just have to understand what they spit out on their callback or their return.
[18:20:52] <StephenLynx> if you have to understand more than that, its a bad lib.
[18:21:07] <synapse> So I ended up implementing it in my own code not because I had to but because I figured well if libs are built this way it can't be that bad?
[18:21:14] <StephenLynx> the thing is
[18:21:15] <StephenLynx> it is.
[18:21:27] <StephenLynx> this strain of thought is pretty much what corrupts the PHP community
[18:21:33] <StephenLynx> most devs are bad and use these bad practices
[18:21:50] <StephenLynx> so newcomers start believing this is The One True Way(tm).
[18:22:09] <StephenLynx> and these bad practices become standard.
[18:22:21] <StephenLynx> protip: never do what a webdev does.
[18:22:33] <synapse> The problem for new comers is besides from official docs, where do we go for consensus on "the right way"
[18:22:43] <StephenLynx> UNIX philosophy and KIS
[18:22:50] <synapse> Seeing how API's are built I figured was a good place to start
[18:24:22] <synapse> I notice your code looks very procedural and damn i do like it
[18:24:28] <StephenLynx> yup.
[18:24:33] <StephenLynx> mu js is purely procedural.
[18:24:41] <StephenLynx> I use modules as namescapes.
[18:24:44] <StephenLynx> spaces*
[18:24:49] <StephenLynx> instead of using them as classes.
[18:24:55] <kurushiyama> Regarding obfuscation... $_=q(s%(.*)%$_=qq(\$_=q($1),$1),print%e),s%(.*)%$_=qq(\$_=q($1),$1),print%e
[18:25:00] <StephenLynx> kek
[18:25:27] <synapse> I think I've seen that before in a Wordpress codex
[18:26:07] <synapse> So what's your take on callbacks / promises
[18:26:14] <StephenLynx> callbacks all the way.
[18:26:17] <synapse> mmm
[18:26:22] <StephenLynx> promises are not only useless but slow.
[18:26:42] <synapse> How long have you been coding for StephenLynx?
[18:26:46] <StephenLynx> hm
[18:26:49] <StephenLynx> a decade or so
[18:27:12] <StephenLynx> pascal, java, C, C++, unrealscript, obj C, js and a little PHP
[18:27:25] <StephenLynx> I also checked out a little some other stuff
[18:27:27] <synapse> unrealscript! I was just doing some of that
[18:27:32] <StephenLynx> haskell, python
[18:28:09] <synapse> which language do you enjoy to use the most
[18:28:13] <StephenLynx> C
[18:28:18] <StephenLynx> followed by js
[18:28:25] <StephenLynx> but its hard to find a good place to use C
[18:28:26] <synapse> I like C as well
[18:28:41] <synapse> I use C to this microcontroller I bought from Ti
[18:28:52] <synapse> to = for*
[18:28:56] <StephenLynx> you should writing your js as you write your C
[18:29:03] <StephenLynx> this is the recipe to make js a sane language
[18:29:59] <synapse> The problem comes though as I said, lets say I end up in an environment that means I need to use JS but it's insisted I do it the "conventional" way.... I'd be lost
[18:30:09] <StephenLynx> not really
[18:30:18] <StephenLynx> because the base is the same.
[18:30:28] <synapse> I guess I'd have to stay on top of what changes
[18:30:33] <StephenLynx> and webdev have no standard whatsoever
[18:30:33] <synapse> as with anything
[18:30:43] <StephenLynx> if your shit works, its good.
[18:30:47] <synapse> lol
[18:30:53] <synapse> I hear that
[18:31:06] <StephenLynx> this is why only crap devs do nothing but webdev
[18:31:16] <StephenLynx> because few people there care about actual good software
[18:31:44] <StephenLynx> must just go with flavor of the month anything and 'good enough'
[18:32:03] <synapse> Do you think node is a viable language for real world deployment
[18:32:07] <StephenLynx> of course.
[18:32:19] <synapse> I get the feeling it can be kinda flakey, it needs hardening
[18:32:22] <StephenLynx> IMO its the single best scripted runtime environment for web currently.
[18:32:24] <StephenLynx> not at all.
[18:32:30] <Tachaikowsky> synapse great help you were
[18:32:32] <Tachaikowsky> One question
[18:32:36] <StephenLynx> it even has a LTS version.
[18:32:38] <Tachaikowsky> can I create schemas using mongodb native plugin
[18:32:47] <Tachaikowsky> or shud it be created on mongoshell
[18:32:50] <StephenLynx> neither.
[18:33:06] <StephenLynx> document your schema and ensure any index you might need.
[18:33:19] <Tachaikowsky> index is primary key?
[18:33:24] <StephenLynx> no.
[18:33:27] <Tachaikowsky> index: {
[18:33:28] <Tachaikowsky> unique: true
[18:33:29] <synapse> it's np Tachaikowsky but I really am not an expert with node or mongo but I will help if I can
[18:33:41] <StephenLynx> an index don`t have to be unique
[18:34:05] <StephenLynx> it can just be indexed so certain operations are faster at a cost of RAM
[18:34:27] <Tachaikowsky> ohh ok
[18:34:32] <Tachaikowsky> gotcha
[18:34:33] <StephenLynx> mongo is not designed with strict schemas in mind.
[18:34:39] <synapse> yeah
[18:34:42] <StephenLynx> so working in a way to enforce them is not optimal.
[18:35:08] <StephenLynx> if your programmer screws up and break the db because he is dumb
[18:35:12] <StephenLynx> its the programmer`s fault.
[18:35:22] <synapse> StephenLynx do you like Mongo? I here people on freenode with real split opinions on it's viability
[18:35:26] <StephenLynx> yes.
[18:35:29] <StephenLynx> I like it a lot.
[18:35:42] <StephenLynx> I don`t think its a silver bullet
[18:35:55] <StephenLynx> and I still think RDS and graph dbs have their place
[18:36:04] <StephenLynx> but so does mongo.
[18:36:34] <StephenLynx> its performance, feature rich, intuitive and got tools to easily able one to scale its deploy.
[18:37:22] <StephenLynx> specially with WT
[18:37:27] <StephenLynx> it got much better.
[18:37:34] <synapse> WT?
[18:37:38] <StephenLynx> wired tiger
[18:37:42] <synapse> Not heard of that
[18:38:12] <StephenLynx> the new engine
[18:38:18] <StephenLynx> made default on 3.2
[18:40:28] <synapse> ahh
[18:41:29] <synapse> StephenLynx what do you think is essential steps to deploying a node http server
[18:41:37] <synapse> eg, PM2?
[18:41:40] <StephenLynx> what is that
[18:41:41] <synapse> init.d
[18:41:43] <StephenLynx> wot
[18:41:59] <synapse> how would you deploy a node app to run dedicated as a background service
[18:42:03] <StephenLynx> yes
[18:42:05] <StephenLynx> ah
[18:42:14] <StephenLynx> I just create a script for the init system
[18:42:18] <synapse> there's PM2 or init.d
[18:42:18] <StephenLynx> let me link you mine
[18:42:22] <StephenLynx> wot
[18:42:23] <StephenLynx> no
[18:42:34] <StephenLynx> theres systemd, upstart, openrc or something
[18:42:38] <StephenLynx> never heard of pm2
[18:42:52] <StephenLynx> and isn`t init.d just a directory?
[18:42:59] <synapse> Pm2 is an alternative to using the inbuilt systemd or openrc
[18:43:08] <synapse> it is I just call it that :)
[18:43:08] <StephenLynx> cancer
[18:43:23] <synapse> lol
[18:43:29] <StephenLynx> if the init system is there and you use something else for the same thing
[18:43:34] <StephenLynx> you are just reinventing the wheel.
[18:43:57] <StephenLynx> https://gitgud.io/LynxChan/LynxChan/blob/master/aux/lynxchan.conf https://gitgud.io/LynxChan/LynxChan/blob/master/aux/lynxchan.rc https://gitgud.io/LynxChan/LynxChan/blob/master/aux/lynxchan.systemd
[18:44:03] <StephenLynx> my 3 init system scripts
[18:44:25] <StephenLynx> this is what I use
[18:45:15] <StephenLynx> and then I provide a install script https://gitgud.io/LynxChan/LynxChan/blob/master/aux/root-setup.sh
[18:45:20] <synapse> and this is all you'd need to deploy ? nothing else for fall over?
[18:45:52] <StephenLynx> no
[18:46:03] <StephenLynx> just this
[18:46:05] <StephenLynx> and the libraries
[18:46:18] <StephenLynx> that the other script is able to install along with other stuff
[18:46:23] <StephenLynx> https://gitgud.io/LynxChan/LynxChan/blob/master/aux/setup.sh
[18:46:37] <synapse> I bookmarked them
[18:46:47] <StephenLynx> you might as well bookmark the whole project
[18:46:51] <synapse> I'm not in the stage of deploying yet as you can tell
[18:46:56] <StephenLynx> all the things I linked to far are just for the same project
[18:46:59] <synapse> I'm still working out how to use things lol
[18:47:14] <synapse> that's handy
[18:47:20] <synapse> what was this project?
[18:47:24] <StephenLynx> a chan engine
[18:47:29] <StephenLynx> lynxhub.com
[18:49:39] <synapse> looks cool
[18:49:42] <synapse> lots of activity
[18:49:47] <synapse> who is Cat
[18:49:52] <StephenLynx> :v
[18:49:57] <StephenLynx> the board default name
[18:50:04] <StephenLynx> the engine allows each board to customize it.
[18:50:17] <StephenLynx> there is a site default
[18:50:24] <synapse> it's like 4chan?
[18:50:29] <StephenLynx> that will be used if the board doesnt have anything on place
[18:50:30] <StephenLynx> yes
[18:50:34] <StephenLynx> but its more inspired on 8chan
[18:50:46] <synapse> ok cool
[18:50:53] <StephenLynx> endchan.xyz is the most active deploy
[18:50:59] <StephenLynx> mine is quite slow
[18:51:10] <StephenLynx> I just have it running for about a year now
[18:51:28] <StephenLynx> http://lynxhub.com/graphs.js here you can see activity
[18:55:36] <synapse> InfinityNow
[18:55:51] <synapse> a spin off?
[18:55:54] <StephenLynx> not really
[18:55:57] <StephenLynx> they wanted to start a fork
[18:55:59] <StephenLynx> but never did
[18:56:07] <StephenLynx> now they put that name on their FE fork
[18:56:11] <StephenLynx> from 8tailedlynx
[18:56:20] <StephenLynx> this being a FE meant to look like 8chan
[18:56:32] <StephenLynx> because my engine works on a decoupled FE architecture
[18:56:47] <StephenLynx> where any template or FE file, such as js and css live on a separate project.
[18:57:03] <StephenLynx> look over my gitgud group for the engine
[18:57:13] <StephenLynx> the placeholder fe and the 3rd party fe list
[18:57:28] <StephenLynx> I use the placeholder one over lynxhub.com
[18:57:39] <synapse> I have these sites bookmarked so I'll come back to them when i start working on my project again
[18:57:57] <synapse> I can't be on the screen for too long as it kills my eyes
[18:58:02] <StephenLynx> kek
[18:58:13] <StephenLynx> you need better color schemes and display settings
[18:58:21] <StephenLynx> and set the distance too
[18:58:30] <StephenLynx> I can stay on computers the whole day with no issues
[18:58:45] <StephenLynx> you should get your sight checked too, you might need glasses.
[18:58:48] <synapse> yeah I used to all the time but recently I need to take breaks
[18:59:04] <StephenLynx> one should check for it yearly
[18:59:22] <synapse> the setup I have I know is not good at all
[18:59:33] <synapse> particularly on my back, I have a bad chair
[18:59:48] <synapse> but it's where I live now I have little space
[19:00:01] <synapse> my monitor is good, its set to low brightness
[19:19:54] <synapse> so StephenLynx, are you on irc every day?
[19:20:12] <StephenLynx> pretty much.
[19:20:17] <StephenLynx> I let it open while I work
[19:20:33] <StephenLynx> my company doesn't try and keep a tight grip on employees.
[19:20:53] <synapse> Ahh I see
[19:20:58] <synapse> well that's good
[19:21:00] <StephenLynx> aye
[19:21:17] <synapse> what country you in?
[19:21:22] <StephenLynx> since I get shit done (and shit few could, mind you) they just let me be :v
[19:21:23] <StephenLynx> brazil
[19:21:40] <synapse> ahh nice
[19:21:56] <synapse> Is there a big need for devs in Brazil? easy to find work?
[19:22:01] <StephenLynx> hm
[19:22:23] <StephenLynx> sort of.
[19:22:36] <StephenLynx> our economy is going alright.
[19:22:37] <cheeser> the largest JUG is there
[19:22:48] <StephenLynx> and most people don`t have enough education to work on IT
[19:23:03] <StephenLynx> so if you got some experience, isn`t too hard to get a job
[19:23:26] <StephenLynx> the problem is market saturation of code monkeys and people not knowing how to hire
[19:23:32] <synapse> You had that virus recently and set the army in to fight the mosquito, we had it on our news here in England
[19:23:42] <StephenLynx> yeah, zyka
[19:23:51] <synapse> yeah
[19:23:58] <StephenLynx> people are only freaking out because of MUH BABIES
[19:24:24] <StephenLynx> the disease itself is not an issue ata ll
[19:24:26] <synapse> crazy stuff
[19:24:30] <StephenLynx> regular dengue is much worse
[19:24:36] <synapse> but it's bad for the babies eh
[19:24:36] <StephenLynx> and we also got some super dengue of sorts
[19:24:42] <StephenLynx> fuck the babies.
[19:24:46] <StephenLynx> we already got enough of them.
[19:24:46] <synapse> lol
[19:25:01] <StephenLynx> specially from poor families
[19:25:13] <synapse> yes that is true but still once they're here they're here
[19:25:27] <StephenLynx> i still couldn't give a crap :v
[19:25:48] <synapse> Well I feel for the babies born with that illness zyka causes
[19:25:54] <synapse> fuck that shit
[19:26:10] <StephenLynx> microcephalia
[19:26:21] <synapse> yeah, tiny head
[19:26:28] <StephenLynx> v:
[19:26:40] <synapse> Life's hard enough
[19:26:48] <synapse> what a shitty start they got
[19:26:49] <StephenLynx> is not like I got reproduction on my life schedule anyway
[19:26:57] <synapse> me neither
[19:27:12] <StephenLynx> on the long term it could even improve the country.
[19:27:32] <StephenLynx> tipping the ratio of educated citizens up.
[19:27:43] <StephenLynx> so they push for a better country in the future.
[19:27:53] <synapse> There's many uneducated people?
[19:27:57] <StephenLynx> kek
[19:27:59] <StephenLynx> you ahve no idea.
[19:28:03] <StephenLynx> they pretty much run the country.
[19:28:09] <synapse> dang
[19:28:10] <StephenLynx> because they outvote everyone.
[19:28:39] <StephenLynx> its because of them we have homophobic religious dicks running our human rights comitee.
[19:28:41] <synapse> Latin America seems like an interesting place
[19:28:49] <StephenLynx> so yeah, the less they reproduce the better.
[19:28:59] <synapse> Wow that sucks
[19:29:07] <StephenLynx> its been like that for a while now.
[19:29:11] <synapse> heretics?
[19:29:16] <StephenLynx> hm?
[19:29:47] <synapse> you don't know about heretics? It was a period in history where anyone who questioned the religious doctrine in society was tortured :)
[19:29:53] <StephenLynx> aah
[19:29:55] <StephenLynx> the inquisition.
[19:29:58] <synapse> yeah
[19:30:00] <StephenLynx> is not that bad.
[19:30:08] <StephenLynx> they are just the biggest voice on politics.
[19:30:15] <synapse> minus the torture of course, is it a weirdly religious place
[19:30:17] <StephenLynx> when it comes to the ones that give a crap about social issues.
[19:30:31] <synapse> yeah
[19:30:35] <StephenLynx> if every single of them in the slums could get zyka right now
[19:30:38] <synapse> I can perhaps imagine
[19:30:51] <StephenLynx> brazil would be pretty much fixed on twenty years or so.
[19:31:49] <StephenLynx> because they are both the cause and the product of our issues.
[19:32:04] <StephenLynx> they are poor because of shitty politicians and they are the ones that keep electing these.
[19:32:19] <synapse> I do completely understand your view
[19:32:23] <StephenLynx> and as a byproduct we get less personal freedoms because of religious fanatics.
[19:32:48] <synapse> I often feel that way here that it seems like the majority of people don't think or question anything and they end up voting in the governments
[19:33:01] <synapse> so it;s like the idiots deciding power
[19:33:39] <synapse> People rarely form an opinion that goes beyong that of a tabloid newspaper
[19:33:45] <synapse> beyond*
[19:41:50] <synapse> StephenLynx your project avoided the use of things like express?
[19:42:00] <StephenLynx> ye.
[19:42:09] <StephenLynx> web frameworks are ebola cancer.
[19:42:11] <StephenLynx> :v
[19:42:16] <synapse> you handle the routing and everything yourself?
[19:42:18] <StephenLynx> yes.
[19:42:37] <synapse> StephenLynx I love this code style. It's the nices node code i've seen yet
[19:42:40] <StephenLynx> on engine/requestHandler
[19:42:54] <synapse> I avoided express too and built my own routing
[19:43:03] <StephenLynx> from there a request can be handled in 4 different ways
[19:43:07] <synapse> I just didn't like what I saw with express
[19:43:10] <StephenLynx> 1: static file serving
[19:43:15] <StephenLynx> 2: JSON RPC
[19:43:36] <StephenLynx> 3: standard url request
[19:43:45] <synapse> I built in 2 things either static html serving or a jade processor
[19:43:46] <StephenLynx> 4: gridfs file serving
[19:43:47] <vak> hi all
[19:44:01] <synapse> hi vak
[19:44:23] <StephenLynx> I also don`t like templating engines.
[19:44:28] <StephenLynx> I use jsdom
[19:44:41] <synapse> I was told jade is the way to go for dynamic content
[19:44:52] <synapse> I saw big projects using it too like Keystone
[19:44:57] <StephenLynx> pfffff
[19:44:59] <StephenLynx> it doesn`t matter.
[19:45:00] <synapse> lol
[19:45:01] <StephenLynx> big or small
[19:45:06] <StephenLynx> its all made by your standard webdev
[19:45:19] <StephenLynx> the problem with templating engines is that their syntax is non-standard.
[19:45:33] <StephenLynx> and I am pretty sure their performance is lacking too.
[19:45:41] <synapse> Some people would argue that by avoiding express, you reinvent the wheel
[19:45:46] <StephenLynx> nope
[19:45:52] <StephenLynx> because express doesn`t implement any standard.
[19:46:02] <StephenLynx> your standard is HTTP and node is already handling that.
[19:46:07] <synapse> yeah
[19:46:16] <StephenLynx> routing is business logic
[19:46:23] <StephenLynx> and libraries have no saying on that.
[19:46:57] <vak> how to pass an index hint to an aggregate() ? there is no info on this in docs and a call like this: db.c.aggregate(...).hint() leads to TypeError: Object #<Object> has no method 'hint'
[19:47:17] <StephenLynx> maybe you can't on aggregate?
[19:47:20] <vak> mongodb 2.6
[19:47:22] <StephenLynx> I never used hint so I don't know.
[19:47:48] <StephenLynx> maybe 2.6 doesn't support it?
[19:48:01] <synapse> Tachaikowsky ask here :)
[19:48:12] <vak> ?
[19:48:23] <Tachaikowsky> synapse - quick question, is the only way to verify whether something exists is to do a find, then pass output to array, then check length??
[19:48:40] <Tachaikowsky> synapse, sorry for pm - I asked in pm coz my question is very noobish
[19:48:40] <synapse> Tachaikowsky SQL has a query called COUNT(*) Mongo must have an equivalent
[19:48:52] <synapse> it's ok we're all here to learn from each other
[19:49:04] <Tachaikowsky> But I am a distilled idiot, I have nothing to offer :P
[19:49:19] <synapse> lol
[19:49:55] <synapse> Tachaikowsky try db.collection.count()
[19:50:24] <StephenLynx> or
[19:50:26] <StephenLynx> don`t.
[19:50:40] <synapse> or db.collection.find(query).count()
[19:50:45] <StephenLynx> use toArray
[19:50:56] <StephenLynx> and then if you are just traversing the result, your lenght doesn`t matter
[19:51:11] <StephenLynx> if you can skip code if you didn't find anything, just check the final array length
[19:51:19] <StephenLynx> .count() is not too useful.
[19:51:20] <Tachaikowsky> oh great
[19:51:23] <Tachaikowsky> let me try count
[19:51:41] <StephenLynx> and if you are looking for a single document
[19:51:46] <StephenLynx> then using .count is a horrible idea
[19:51:55] <StephenLynx> because mongo will traverse the whole damn collection
[19:52:04] <StephenLynx> if you want a single document, use findOne
[19:52:14] <StephenLynx> and just check if you got an document or not.
[19:52:35] <StephenLynx> .count is kind of slow usually
[19:52:42] <StephenLynx> and you will want to avoid it.
[19:53:01] <StephenLynx> unlike RDBs mongo can't figure out it too easily.
[19:53:04] <synapse> Ahh I'm just going by how it was done with SQL
[19:53:09] <StephenLynx> yeah
[19:53:09] <synapse> mmm
[19:53:21] <StephenLynx> rdbs can figure amounts because rows have the same size
[19:53:24] <StephenLynx> mongo can`t do that.
[19:53:29] <StephenLynx> so count is way less efficient.
[19:53:34] <synapse> ahh indeed
[19:54:24] <StephenLynx> if want to use limit usually to check for minimal amounts
[19:54:29] <StephenLynx> you want*
[19:54:31] <synapse> db.collection.findOne(query).toArray() is the best way?
[19:54:40] <StephenLynx> just findOne
[19:54:46] <Tachaikowsky> okay findone
[19:54:49] <StephenLynx> findOne give you a single document and not a cursor
[19:54:53] <StephenLynx> so you can`t use toArray
[19:55:10] <StephenLynx> toArray is for when you want to get all documents from a cursor
[19:55:22] <StephenLynx> and there might be more than one.
[19:55:28] <Tachaikowsky> I have a question
[19:55:39] <StephenLynx> findOne is also faster because it will stop traversing the collection on the first match.
[19:55:46] <StephenLynx> while a find will traverse the whole collection.
[19:55:56] <StephenLynx> the same could be achieved with a limit(1)
[19:56:01] <synapse> how will findOne know the length of the collection though ?
[19:56:06] <StephenLynx> it won`t.
[19:56:11] <StephenLynx> it will find One and return it.
[19:56:28] <StephenLynx> it traverses until finding a document that matches what you want.
[19:56:41] <synapse> iterate using a loop and count the length?
[19:56:48] <StephenLynx> no
[19:57:14] <StephenLynx> it just uses a limit(1) and returns you the document instead of an array of documents
[19:57:27] <Tachaikowsky> StephenLynx,
[19:57:28] <StephenLynx> so is more handy than using a find().limit(1), because that will give you an array
[19:57:34] <Tachaikowsky> My problem is more straightfwd
[19:58:18] <Tachaikowsky> let me pastie this
[19:59:01] <Tachaikowsky> http://pastie.org/private/45teuz134rjsjqgdrlctw
[19:59:21] <StephenLynx> wot
[19:59:25] <Tachaikowsky> StephenLynx, basically I am seeing if the person I am trying to add has me on their blocked list.
[19:59:37] <StephenLynx> ok
[19:59:39] <synapse> Tachaikowsky http://stackoverflow.com/questions/7658228/mongodb-count-is-very-slow-how-do-we-refine-work-around-with-it
[19:59:42] <Tachaikowsky> That person's blocked list has an _id and then an arrayOfBlockedUsers
[19:59:52] <StephenLynx> you should put a callback on that if its node
[19:59:59] <Tachaikowsky> Now all I need to know is I have been blocked or not
[20:00:41] <Tachaikowsky> StephenLynx, whats the best way to find if I figure in their blocked list?
[20:00:43] <Tachaikowsky> findOne?
[20:01:23] <synapse> Tachaikowsky if you're using that query in the pastebin then just check the length of the result (my opinion)
[20:01:34] <Tachaikowsky> here targetUser = db.collection('users')
[20:01:40] <StephenLynx> no
[20:01:47] <StephenLynx> use findOne to get the user with blocked people
[20:01:59] <StephenLynx> then if you get an user you will know if the target user exists or not.
[20:02:03] <StephenLynx> that is half of the job
[20:02:12] <synapse> that function will return a true or false right Tachaikowsky?
[20:02:18] <StephenLynx> the other one is finding of arrayOfBlockedUsers has the user you are trying to see if its blocked
[20:02:21] <StephenLynx> no
[20:02:28] <Tachaikowsky> No it doesnt return that
[20:02:47] <Tachaikowsky> yea StephenLynx
[20:02:49] <synapse> so it doesn't just check for those entries and if a record comes back the person is blocked?
[20:02:57] <Tachaikowsky> I need to look into their _id and see if their arrayOfBlockedUsers has me
[20:03:24] <Tachaikowsky> That's why I am using req.body.id for _id and req.decoded.id for $elemMatch
[20:03:34] <Tachaikowsky> req.decoded.id is my own id decoded from the token
[20:03:45] <StephenLynx> http://pastebin.com/50qkzbus
[20:04:02] <StephenLynx> unless the syntax of elemMatch is wrong there
[20:04:23] <StephenLynx> in that case the value of arrayOfBlockedUsers on user won`t be what I expect
[20:05:56] <Tachaikowsky> if user exists here, it means you got an user with the id on req.body.
[20:05:58] <Tachaikowsky> I dont need that
[20:05:59] <synapse> That's what I meant by the way of a true or false result, it's either blocked or isn't
[20:06:18] <StephenLynx> you need.
[20:06:31] <StephenLynx> what if the user doesn`t exist at all?
[20:06:43] <StephenLynx> you are just going to assume the user on req.body exists?
[20:07:27] <synapse> I'm sure he does that elsewhere in a different function
[20:07:33] <Tachaikowsky> If the user doesn't exist at all, I cannot be on their blocked list right?
[20:07:44] <StephenLynx> you might get a crash.
[20:07:47] <Tachaikowsky> Whether the target user exists, that is validated before that stage
[20:07:53] <StephenLynx> because you might want to access a field of an undefined value.
[20:08:15] <Tachaikowsky> for that I am using targetUser.find({
[20:08:16] <Tachaikowsky> "_id": req.decoded.id
[20:08:16] <Tachaikowsky> }).toArray(function(err, result) {
[20:08:19] <Tachaikowsky> things like that
[20:08:28] <StephenLynx> thats a completely different thing
[20:08:29] <Tachaikowsky> req.body.id
[20:08:33] <StephenLynx> that is just projection
[20:08:51] <StephenLynx> if the document isn`t found, that part won`t even be used.
[20:09:12] <synapse> Tachaikowsky why not pass the result of that back into the function that checks the blocking depending if the user is found first
[20:09:13] <Tachaikowsky> yeah and these parts lie INSIDE that aprt
[20:09:24] <StephenLynx> no
[20:09:25] <StephenLynx> they don`t
[20:09:29] <StephenLynx> its separate
[20:09:50] <StephenLynx> step one: match. step two, project.
[20:09:52] <StephenLynx> is not inside
[20:09:55] <StephenLynx> its sequential.
[20:10:11] <StephenLynx> if you try to access user.arrayOfblocked and user isn`t defined, you will crash.
[20:10:13] <StephenLynx> period.
[20:11:08] <synapse> Tachaikowsky pass it along down the chain
[20:11:30] <Tachaikowsky> Gotcha :)
[20:11:37] <Tachaikowsky> I see what you mean now
[20:11:56] <Tachaikowsky> StephenLynx, that will return some 'socketsish' error?
[20:12:04] <StephenLynx> no
[20:12:08] <synapse> That db query Tachaikowsky is literally only going to return either 0 or 1 results though right, or wrong?
[20:12:16] <StephenLynx> if it finds a document it will return a document. otherwise, no document
[20:12:22] <StephenLynx> no
[20:12:45] <StephenLynx> the projection is a completely different business.
[20:12:47] <synapse> it either matched or didn't
[20:12:58] <StephenLynx> so it returns either a document or not.
[20:13:01] <StephenLynx> not 0 or 1
[20:13:06] <StephenLynx> a document or undefined.
[20:13:13] <StephenLynx> but thats not all
[20:13:20] <synapse> I mean in terms of what he's getting it's a binary answer
[20:13:22] <StephenLynx> because he is also concerned with the projection of a field.
[20:13:29] <StephenLynx> no, it has 4 states
[20:13:39] <synapse> 4 states?
[20:13:41] <StephenLynx> 3 actually
[20:13:50] <synapse> oh?
[20:14:09] <StephenLynx> 1: no document, 2: document with no field for array of blocked users, 3: document with field for array of blocked users
[20:14:48] <synapse> ahh
[20:15:26] <Tachaikowsky> again feeling stupid
[20:15:27] <Tachaikowsky> http://pastie.org/private/tjh5s1kdr7tpac4ibpu43q
[20:15:31] <Tachaikowsky> will this work?
[20:15:33] <synapse> if he added an arrayOfBlockedUsers: true in there that would assert it needs to exist right StephenLynx?
[20:16:01] <StephenLynx> no
[20:16:12] <StephenLynx> it would only return a value if he had "true" on that array.
[20:16:26] <StephenLynx> actually
[20:16:32] <StephenLynx> it would just return that array at all.
[20:16:36] <StephenLynx> itwouldn't check whats in there.
[20:16:48] <Tachaikowsky> StephenLynx, what about the code I just pasted?
[20:16:51] <StephenLynx> wrong
[20:16:56] <synapse> find( {'user':'foo'}, {user: true, array: true}) no?
[20:16:57] <StephenLynx> you are just checking the length of na object
[20:17:12] <StephenLynx> and findone won`t give you an array
[20:17:15] <StephenLynx> but an object
[20:17:22] <StephenLynx> checking for length is pointless.
[20:17:26] <StephenLynx> and even then
[20:17:39] <StephenLynx> it doesn`t matter if you found a document or not
[20:17:49] <StephenLynx> because you check for the blocked user on the projection
[20:17:58] <StephenLynx> you could check that on the match itself, though
[20:18:08] <StephenLynx> se
[20:18:09] <StephenLynx> sec
[20:18:19] <StephenLynx> also
[20:18:23] <StephenLynx> you can`t use return like that.
[20:18:25] <StephenLynx> on async code
[20:18:28] <StephenLynx> you need a callback
[20:18:30] <Tachaikowsky> stpid syntax
[20:18:50] <synapse> you need to use callbacks Tachaikowsky
[20:19:06] <StephenLynx> you can`t use toArray either
[20:19:15] <StephenLynx> because its a findOne and not a find
[20:20:03] <synapse> Tachaikowsky do you know how to use callbacks?
[20:20:15] <StephenLynx> http://pastebin.com/ijUTt8sP
[20:20:23] <StephenLynx> your callback will receive error, document
[20:20:34] <StephenLynx> if the document exists at all, its because the user a blocked user b
[20:20:39] <StephenLynx> otherwise either
[20:20:44] <StephenLynx> 1: user a doesn`t exist
[20:20:44] <Tachaikowsky> I dont know how to use callbacks
[20:20:52] <StephenLynx> 2: user a haven`t blocked user b
[20:20:53] <Tachaikowsky> I think I dont know exactly what a callback means
[20:21:00] <StephenLynx> its just a function pointer.
[20:21:09] <StephenLynx> its a variable that holds a function and you can execute it.
[20:21:10] <StephenLynx> thats it.
[20:21:39] <StephenLynx> var x = function (){console.log(1); }; x();
[20:21:41] <StephenLynx> 1
[20:22:09] <Tachaikowsky> ok
[20:22:12] <Tachaikowsky> thats the callback?
[20:22:24] <StephenLynx> callbacks its just a mean of returning value where you execute something instead of returning a value.
[20:23:03] <StephenLynx> more specifically a function pointer.
[20:23:15] <StephenLynx> x in that example is just a function pointer.
[20:23:53] <StephenLynx> in C you wouldn`t be able to declare it like that, though, because js got closures.
[20:23:57] <StephenLynx> but it works the same.
[20:24:24] <StephenLynx> and why you use callbacks? because of asynchronous code.
[20:24:37] <StephenLynx> if you return a value you are returning to nothing.
[20:24:45] <StephenLynx> not nothing, but nothing you can control.
[20:25:43] <synapse> Tachaikowsky I show you here how to use it... http://pastebin.com/XZ86Nkd2
[20:26:19] <StephenLynx> you missed something
[20:26:23] <synapse> I did?
[20:26:27] <StephenLynx> you didn`t check if result actually exists.
[20:26:36] <Tachaikowsky> thanks synapse
[20:26:41] <synapse> I'm just giving him an idea of how to use the callback
[20:27:01] <synapse> they can be weird for people to understand at first but are easy
[20:28:02] <synapse> but Tachaikowsky take StephenLynx's advice check both the error and result exisit and if neither handle it all accordingly :)
[20:29:28] <Tachaikowsky> allright
[20:44:46] <Tachaikowsky> synapse, I am facing a problem
[20:45:04] <Tachaikowsky> this is more of node.js
[20:45:05] <Tachaikowsky> still
[20:45:31] <Tachaikowsky> http://pastie.org/private/hgvtnl2cdtbdims3rxrsog
[20:45:33] <Tachaikowsky> this WONT work
[20:45:35] <Tachaikowsky> but how can I make it?
[20:54:39] <synapse> Tachaikowsky well there are many problems with how that's constructed, one thing you should know is never place anything after a return, return will return you from the function, ntohing after will be executed
[20:54:59] <Tachaikowsky> ok
[20:55:11] <Tachaikowsky> so now I truly see why we need 'callback'
[20:55:26] <Tachaikowsky> basically we need to do var something = targetUser.find({
[20:55:29] <Tachaikowsky> right?
[20:56:29] <synapse> Second that will cause a connection to the db every single time the query is run, in a real world setting it will kill your app ( literally had this problem myself and learned the hardway), you need to create a connection cache / pointer something which I've yet to implement myself but StephenLynx just provided me with a well written project code in node js which I will take a look at to learn
[20:56:29] <synapse> how to implement such a thing
[20:56:57] <Tachaikowsky> ok
[20:57:14] <synapse> like a connection piool
[20:57:18] <synapse> pool*
[20:58:05] <StephenLynx> the driver handles its own pool though
[20:58:17] <Tachaikowsky> the driver = mongodb?
[20:58:20] <StephenLynx> yes.
[20:58:22] <Tachaikowsky> ok
[20:58:22] <synapse> The idea behind async code is that you can get on with other tasks whilst something is still being executed i.e it's non-blocking, eg we don't wait to read in a 10gb file before doing anything we get on with other things that's where callbacks and events come in Tachaikowsky
[20:58:53] <Tachaikowsky> how do I learn about async?
[20:59:00] <synapse> you're learning right now
[20:59:11] <Tachaikowsky> oo
[20:59:57] <synapse> Tachaikowsky when defining functions, you write them with a callback as you saw earlier what me and stephen posted
[21:00:17] <synapse> All it is is a function which takes another function as an argument
[21:00:25] <synapse> it's like a function pointer
[21:00:58] <synapse> By following that, you will be writing async code
[21:02:22] <Tachaikowsky> ok
[21:03:12] <Tachaikowsky> <synapse> All it is is a function which takes another function as an argument
[21:03:15] <Tachaikowsky> will think
[21:04:16] <Tachaikowsky> ahh so pass a function into a function
[21:09:43] <Tachaikowsky> http://www.tutorialspoint.com/nodejs/nodejs_callbacks_concept.htm
[21:09:48] <Tachaikowsky> ok I read that and got the concept of async
[21:10:17] <Tachaikowsky> but I dont really understand how we can do this in an async way
[21:14:47] <synapse> Tachaikowsky Basically when you define a function you specifiy a bunch of work to do as you would any other, the function solves a task
[21:15:10] <synapse> What's different is what we then do with the result
[21:16:40] <synapse> instead of running a return value; to the caller, we run a function (the callback) that's placed as an argument with the results of our work as the parameters of the function we callback.
[21:16:55] <synapse> Tachaikowsky By building your entire app this way, your app will be completely async
[21:17:09] <synapse> because noting waits for results before moving on
[21:17:40] <Tachaikowsky> ok
[21:18:14] <synapse> Do you understand now?
[21:18:36] <synapse> I will make you a quick example
[21:18:45] <synapse> if another callback if you want
[21:18:47] <synapse> of*
[21:22:04] <Tachaikowsky> I really feel guilty about wasting ur time synapse and steph*
[21:22:13] <Tachaikowsky> So I am trying to figure out as much as I can
[21:22:24] <synapse> It's ok I don't mind to help
[21:22:29] <Tachaikowsky> but I have a question, all my work happens in api
[21:22:39] <synapse> Tachaikowsky let me give you another example
[21:22:46] <Tachaikowsky> api.post('/addFriend', function(req, res) {
[21:22:48] <Tachaikowsky> Okay
[21:23:22] <synapse> lets say you have a function that needs to read in a 10GB file and do some complicated stuff with the bytes it read in....
[21:23:33] <Tachaikowsky> right..
[21:23:51] <synapse> Are you going to wait for that to occur and then give the result back to the caller?
[21:24:24] <Tachaikowsky> Nope
[21:24:34] <Tachaikowsky> I will run the code and let it happen in the background
[21:24:55] <Tachaikowsky> so i neeed it to be non-blocking
[21:24:56] <synapse> No... we'll make that file handler function a callback function so we can write other functions that do other tasks in our code that will also be callbacks. Our app will consist of these types of functions, our becomes async
[21:25:15] <synapse> our code*
[21:25:23] <Tachaikowsky> ok
[21:25:36] <synapse> This is async
[21:25:55] <Tachaikowsky> I get the theory
[21:25:57] <Tachaikowsky> But the thing is
[21:26:38] <Tachaikowsky> I am unable to explain it properly
[21:26:54] <synapse> If you look at the node.js API, look at the Filesystem API you will see there are two functions that read in files that node offers you to use. You can choose a syncronous .readFile() function or an async version of the same thing. They both achieve the same task, the difference is how the results are handled
[21:27:26] <synapse> One will make your app pause and wait, the other wont
[21:28:19] <Tachaikowsky> okay
[21:28:20] <Tachaikowsky> got it
[21:28:40] <synapse> I will write you one more example which may help you see it in action
[21:28:55] <synapse> Consider this example function I am writing that checks a name you pass in
[21:30:17] <Tachaikowsky> okay
[21:39:52] <synapse> Tachaikowsky if you don't understand something just ask.. http://pastebin.com/ZesL9Mic
[21:39:58] <Tachaikowsky> ok
[21:40:07] <Tachaikowsky> checkking
[21:43:24] <Tachaikowsky> return callback(error, result);
[21:43:27] <Tachaikowsky> dont understand that
[21:43:36] <synapse> ok so look at the function defintion
[21:43:44] <synapse> what parameters does it take
[21:44:03] <Tachaikowsky> name and callback
[21:44:12] <synapse> and what are we returning?
[21:44:14] <StephenLynx> the return there is probably useless though.
[21:44:35] <Tachaikowsky> we are returning error and result
[21:44:38] <StephenLynx> no
[21:44:42] <synapse> StephenLynx this is for clarity so he can learn only
[21:44:44] <StephenLynx> you are returning what callback returns.
[21:44:47] <StephenLynx> k
[21:44:59] <synapse> no Tachaikowsky
[21:45:08] <synapse> return callback(error, result);
[21:45:16] <synapse> we are returning callback
[21:45:23] <synapse> our 2nd parameter...
[21:45:34] <Tachaikowsky> ok
[21:45:39] <StephenLynx> you are not returning callback either
[21:45:42] <synapse> The parameter we expect our usage will pass in
[21:45:46] <Tachaikowsky> so the function returns iself?
[21:45:58] <StephenLynx> since you are executing callback you are returning what callback returns upon execution with those two parameters.
[21:46:14] <StephenLynx> return callback; << that returns the callback.
[21:46:45] <Tachaikowsky> o ok
[21:46:51] <synapse> penny dropped?
[21:46:55] <Tachaikowsky> in this case the callback is function(error, result){....}
[21:47:00] <synapse> exactly
[21:47:08] <Tachaikowsky> and since we are passing in error value of false
[21:47:13] <synapse> That's the argument and it's the 2nd parameter
[21:47:14] <Tachaikowsky> we get happy result
[21:47:22] <synapse> Tachaikowsky run it in your console
[21:47:25] <synapse> test it
[21:47:27] <Tachaikowsky> I did run it
[21:47:32] <synapse> so it makes sense?
[21:47:55] <Tachaikowsky> Yeah but this is complicated
[21:47:59] <Tachaikowsky> have to read it many many times
[21:48:01] <Tachaikowsky> but good example
[21:48:13] <Tachaikowsky> this isnt really straightforward though
[21:48:15] <synapse> lol not at all, it's very easy Tachaikowsky it's a function like any other
[21:48:27] <synapse> there is nothing different about this definition
[21:49:44] <synapse> Tachaikowsky you really need to understand this if you want to use node
[21:49:54] <synapse> it's a crucial part of the language
[21:50:09] <synapse> And I never had someone hold my hand through it when i was learning so you are lucky :)
[21:50:32] <Tachaikowsky> I am very lucky
[21:50:37] <Tachaikowsky> First thing, my prep is half-assed
[21:50:47] <synapse> It will save you time, you can get on and be productive
[21:50:47] <Tachaikowsky> No one helps people whose prep is weak
[21:50:54] <Tachaikowsky> Thanks a lot synapse
[21:51:00] <Tachaikowsky> I am very grateful
[21:51:07] <synapse> it's np, people help me so I am happy to help others