Log file Viewer
#mongodb logs for Sunday the 20th of September, 2015
[00:44:28] <jtthedev> Can someone point me in the right direction ? I would like to change the maximum default pool size
[00:49:30] <zamnuts> joannac, thanks for the help, turns out I had old mongodb IP addresses listed in the firewall, never updated them with the new api-agents.mongodb.com A records.
[00:50:17] <StephenLynx> I think your problem is with your application refusing to open more than 100 connections
[00:52:13] <StephenLynx> you don`t open a new connection for every single operation, thats a big no-no.
[00:52:36] <zamnuts> jtthedev, your driver should have a "poolSize" option for server, i.e. in the node.js mongodb driver, we have the following available: {options:{server:{poolSize:2}}}
[00:53:13] <zamnuts> jtthedev, but like StephenLynx said, your pool size should not reflect the number of operations.
[00:53:28] <jtthedev> if you are fluent in node... please review https://gist.github.com/jterrero/f70c68b6754ee55b353b
[00:53:44] <zamnuts> if you want to bulk insert, consider buffering into an UnorderedBulkOperation http://mongodb.github.io/node-mongodb-native/2.0/api/UnorderedBulkOperation.html
[00:54:12] <StephenLynx> you open it once and reuse as you wish. you can even store pointers to the collections.
[00:54:45] <jtthedev> thank you, the guidance is greatly appreciated... nodejs/mongo are new waters for me. i've been working with rails and relational backends for years now
[00:55:35] <StephenLynx> with the node driver, even if the connection drops, it will keep the references to the collections when it reconnects.
[00:59:10] <jtthedev> something that's going to require a lot more studying than me writing those few lines
[01:00:10] <zamnuts> jtthedev, are you using the native mongodb driver in nodejs (i.e. 'mongodb'), or are you using something like mongoose?
[01:00:57] <StephenLynx> and what I provided is just an example of how to handle connections in an effective manner
[01:01:19] <StephenLynx> how you can just open the connection, get the collection reference and keep using this reference.
[01:01:59] <zamnuts> jtthedev, your application should have a main file, usually "index.js" ... this is where you'll perform a MongoClient.connect(...), then store the "db" result of the connect callback in a "global" (*cringe*) variable of some sort
[01:03:37] <zamnuts> anyways, jtthedev, how you store the ref to db is wide-ranging, but then in any submodules of the application should reference that single db pointer
[01:03:38] <jtthedev> ok.. so, in rails I have written over the past 3 years about 16 - 18 apps that are in production and used every day within my organization. I am making the switch to node and am trying to tackle them one by one. I am doing it in order of easiest to dificult of what the development process was for me.
[01:04:53] <StephenLynx> every source file will be a module on its own that will be converted to a module in a singleton architecture.
[01:05:32] <StephenLynx> second: callbacks. they are the most efficient way to handle async, just make sure you don't nest them. call a separate function when you finish with a callback
[01:06:28] <jtthedev> hahaha, i can see why.. trust me, i've had my few share of issues with callbacks while making this transition
[01:06:40] <StephenLynx> personally, I nest them ONCE often, because I wrap the inner async in comment and then theres no way to confuse stuff, but is not exactly kosher.
[01:08:46] <StephenLynx> personally I also like to avoid using prototypes. but I won't preach that like the one true way.
[01:09:25] <StephenLynx> I understand their use in libraries, but I don't think its a good idea to use them on purely application code.
[01:10:00] <StephenLynx> IMO, it makes much harder to understand the system's logic when you use them
[01:10:41] <jtthedev> heh, funny enough. you guys seem to be cooler then the guys in node.js.... they preach to stay away from mongodb, not sure why. I personally am not a fan of mongodb, mostly due to my career as a dba.. the only nosql databases to impress me so far are graphdb's, but I am sure there are applications where mongo makes a lot more sense than a relational db
[01:11:03] <jtthedev> StephenLynx, do you know where I can start research for node/mongo on iterating over remote api's for my application ?
[01:12:04] <zamnuts> jtthedev, node.js guys are opinionated about everything, not just mongodb, i always get the worst advice from that channel
[01:12:45] <StephenLynx> and when they say "I don`t want to write code, I just want to let dependencies do all the work"
[01:13:13] <StephenLynx> "my application has only 100 LOC, don't mind the fact it uses over two dozens of dependencies"
[01:13:22] <jtthedev> i'm not easily swayed, i listen and read for as long as i can before forming my own opinion.... eg, they told me to stay away from mongodb and i was doing it all wrong... but here I am still trying to figure out how to go about solving my issue
[01:14:07] <StephenLynx> because only recently I figured out the proper way to iterate over something in async properly
[01:14:51] <StephenLynx> https://gitlab.com/mrseth/LynxChan/blob/master/src/be/engine/uploadHandler.js#L493
[01:15:40] <StephenLynx> so when the operation for each file finishes, I have to execute exports.saveUploads again
[01:16:05] <zamnuts> StephenLynx, dude, that "two dozens of dependencies" drives me nuts, when you have more deps than lines of actual logic, there's a problem IMO
[01:16:23] <StephenLynx> and the other is to ONLY let saveUploads call saveUploads once each file is finished.
[01:17:02] <StephenLynx> what is the problem with the former: you will have to pass around ALL the parameters saveUploads requires and are used only up to a certain point of the operation that processes each file
[01:17:32] <StephenLynx> when you make a separate callback to the operation that processes each file, you stop passing parameters once they are no longer used in the operation
[01:17:56] <StephenLynx> because the callback will be treated at saveUploads, which already has these parameters.
[01:20:25] <StephenLynx> when you apply the pattern I described, you separate the structure of the iteration from the structure of the processing of each iterated element.
[01:21:58] <StephenLynx> keep the iteration on a separate callback of the processing of each element.
[01:23:06] <StephenLynx> once you iterate over all the elements, have the iteration function call the callback it received.
[01:24:49] <StephenLynx> https://gitlab.com/mrseth/LynxChan/blob/master/src/be/engine/modOps/transferOps.js
[01:25:53] <StephenLynx> what I do is to each part of the process to initiate a new callback and pass it to the next part.
[01:26:34] <StephenLynx> then the part sends the received callback to the function that reverts what it did previously and have the reversal function to execute the callback
[01:26:52] <StephenLynx> the part where I pass the original error along is not a core part of this pattern
[01:27:22] <StephenLynx> if your callback returns an error, have it be the first parameter of the callback
[01:28:27] <jtthedev> interesting way to go about it, I can actually see a few ways I can play with that sort of logic
[01:29:56] <jtthedev> ruby i learned self taught. I'm a pretty decent developer, but with nodejs, I am not going to be going the same route. I am preparing as much as possible, before I start classes on Jan 10th, 2016. I was able to get in, (8% acceptance rate) and the test was a decent study, I had to take it twice before I finally passed. and even passing the test doesn't get you in, you still have to go through like 4 interviews.
[01:31:13] <jtthedev> some people learn differently, those people have their ish together.. im taking the courses to cut down the time
[01:33:46] <StephenLynx> I use eclipse because of the lint integration of nodeclipse and auto-formatting, but I think its awful.
[01:35:13] <StephenLynx> any framework that doesn't abstract something meant to be abstract is useless by design
[01:35:56] <jtthedev> have you ever dablled with the rails framework ??? Ideally I would like to use node with a framework that will give me a similar MVC development pattern
[01:37:01] <StephenLynx> the key to good mvc is ignoring the fact a GUI exists in your core back-end design.
[01:37:23] <preaction> sure. you need a limited scope. but how limited can be up to debate. the web framework i use, mojolicious, has its own implementation of a lot of stuff. like django does
[01:37:26] <StephenLynx> when you use a tool that tangles all that together, its not modular anymore.
[01:39:01] <jtthedev> if i could find a a framework that works the way the rails framework works, but gives me nodejs as the backend.... that's what I would be looking at... node is a lot more flexible then ruby is
[01:39:16] <jtthedev> StephenLynx, do you know what's the clsoest framework to rails that uses node as a backend ?
[01:40:27] <StephenLynx> from here you write data to the response object and fetch request data from the request object.
[01:41:04] <preaction> and then you build routines to help you write to the response and read from the request, and then you move those routines into a module, and perhaps start building abstraction, and now you've got a web framework!
[01:41:11] <jtthedev> i've written production apps in less then a week sometimes... not sure i would have been able to achieve that without a framework that already has a pattern in place
[01:43:04] <preaction> so what you're saying is that it's better to cobble together your own set of libraries and just not call it a framework
[01:43:36] <preaction> so i write a json library from scratch? a template library? a url routing library?
[01:44:07] <preaction> right, so i've got these libraries. at what point do i have my own "web framework"?
[01:44:44] <preaction> frequently that is exactly what it is, though they're generally called "components" when they're part of something bigger
[01:45:07] <StephenLynx> no, a framework streamlines the whole application through a certain process.
[01:46:32] <StephenLynx> you just used someone else's code instead of re writing the exact same thing.
[01:46:54] <StephenLynx> and this person`s code didn`t made any decision specific to your system`s use case.
[01:49:54] <jtthedev> if there's something already out there, that's going to cut down my time in putting an application into production
[01:50:12] <jtthedev> and I have to use some "modules" or "libraries" that someone else wrote (in my case gems for rails)
[01:50:31] <StephenLynx> ok, I will be a little flexible here and give my personal view on a valid use case:
[01:50:59] <StephenLynx> if you have already mastered the base tool and can easily figure out what a framework is doing under the hood
[01:51:42] <jtthedev> I work for a distributor, very fast pased environment... i'm not building top secret ish for nasa or the nsa... I don't need to write every single piece of code in my app if there's something already out there that will do a fucntion for me
[01:51:46] <StephenLynx> because you could easily do those things and if anything goes wrong, you know what went wrong.
[01:52:33] <jtthedev> then having to manually take care of a bunch of stuff that I already had a quicker route to get there and do the same exact thing
[01:52:36] <StephenLynx> I can agree on that, but IMO, when you use a framework without a solid experience with the base tool, you will eventually end up with some big issue that you can`t figure out.
[01:53:13] <jtthedev> anyone that tries to learn rails, before mastering ruby, isn't going to get too far
[01:53:31] <jtthedev> someone who doesn't master jd/node... has no business trying to learn express/angular etc
[01:53:38] <StephenLynx> the time that it takes to get the basic structure of the system working is not considerable.
[01:54:16] <jtthedev> you build some sort of base that has a core set of functions you know you will reuse over and over
[01:55:00] <StephenLynx> not really, I don`t usually design system`s that work as every other I designed often.
[01:55:23] <StephenLynx> I take in consideration their specific requirement and their impact in the system`s architecture.
[01:57:06] <jtthedev> but if you ended up in a culture like mine.. and you are 1 of 2 developers, and one developer uses a framework and leverages it, in you're both working on very similar apps for 2 departments.... he might outperform you most of the time
[01:57:08] <StephenLynx> I have recently introduced cutting edge tech in the company with a real-time chat module using websockets
[01:57:49] <StephenLynx> currently I am tasked with OCR research but I am not familiar at all with that, so is going slow.
[01:58:37] <StephenLynx> I take my time to see if I can learn anything from them, but usually I take myself as measurement.
[01:58:38] <jtthedev> i work for a distributor, my applications are usually built around getting rid of manual processes (it sucks, a lot of people have lost their jobs as a result of something i've developed)..
[01:59:24] <StephenLynx> so I rather have a slow build-up to the project`s productivity and ensure it will be the best possible when it`s complete.
[01:59:48] <StephenLynx> and the documentation and specs will be the closes thing to perfection too.
[02:00:16] <jtthedev> last application i built..... we have a customer who has 230+ locations in the eastern US... sends all his orders electronically via email attached as an html file.... prior to the app, the orders were printed out an manually entered into our ERP, it would take 3 customer service reps, all day to key in those orders
[02:01:06] <jtthedev> could I have built it without rails and just use ruby and a cron job to schedule it?
[02:01:34] <jtthedev> but building it with rails, made the development process structured, simpler and easier to go back to and make adjustments if needed one day
[02:02:23] <StephenLynx> yeah, I will be honest: for such a short development cycle, the impact of not using a framework is much larger.
[02:03:06] <jtthedev> i just resigned for 3 years, so i'll be here through 2019... one of my goals is to replace the erp with something we build in house
[02:03:31] <jtthedev> which is why, im going to go take that course that i showed you the curriculum for
[02:03:51] <jtthedev> it will help me start getting there faster, because I would rather leverage the time and learn node during the next 3 years
[02:04:18] <StephenLynx> well, at least you are not paying, but I don't think they will help you much.
[02:04:55] <StephenLynx> and what it is, they will go on, and on, and on in classes aimed at the lowest common denominator
[02:05:11] <StephenLynx> that you could figure out just using your free time to work hard on challenging projects.
[02:06:09] <jtthedev> i'm still challenbging myself, which is why im here now... instead of just waiting to get there on Jan 10th when class starts
[02:07:06] <jtthedev> do my work at work, get home and sit at my computer (couch and a 65 inch tv about 6 feed away) for another 10 hours or so
[02:07:47] <jtthedev> yeah, there's a bunch of stuff when i started really learning JS that i havnt found a use for nor shor that i ever will
[02:09:23] <jtthedev> alright, it's been aweoms chatting with you today, going to get back to my mongo / node issue with the remote api call... i've added this channel to the list, i'll be in here more frequently in the coming months
[02:24:55] <macwinner> any tips on a clever methodology to generate an array of tags and their associated posts.. i current have an array of posts which have a subdocument of array of tag objects. post = { title: String, tags: [{text: 'tagtext1'}, {text:'tagtext2'}]}
[02:25:22] <macwinner> i want to have a flat object or array where I can lookup a tag name and see all posts associated with it
[15:05:07] <whaley> m3t4lukas: eh? what are you looking at? DBCollection certainly has those defined: https://github.com/mongodb/mongo-java-driver/blob/master/driver/src/main/com/mongodb/DBCollection.java#L648-L765
[15:17:20] <m3t4lukas> if MongoCollection is deprecated it would be nice if it was marked as deprecated
[17:30:29] <Derick> THK: have you read all of http://docs.mongodb.org/ecosystem/platforms/amazon-ec2/? (not an expert here)
[17:32:45] <Derick> THK: apparently, you don't need to do anything as AWS is clever enough to internal resolve their DNS names to the "private IP"
[17:33:05] <Derick> you should use the public DNS name of your instances in your replicaset config and *never* the IP
[17:33:35] <THK> so you set the public ip for both, and aws will just handle the communication internally?
[17:34:01] <Derick> http://stackoverflow.com/questions/17601292/how-do-i-configure-mongodb-replicaset-using-elastic-ips-in-ec2 has hints (in the non accepted comments) too
[17:37:19] <THK> Derick:that looks like it is for ec2-classic.. as it mentions that SecurityGroups are ingress only.
[17:57:17] <BadApe> i know this probably isn't possible but is it possible to generate a json schema based on a document? yes i can do it in code, but can mongo do it for me?
[18:46:43] <BadApe> so mongo doesn't help with schema, so i am going to need someway to validate my json
[22:53:53] <StephenLynx> hey, is it possible to keep the cursor open and receive new documents without having to open a new tailable cursor?
[23:04:28] <StephenLynx> is there any trick to it or the driver is bugged? I am on node.js native driver
[23:08:06] <StephenLynx> http://pastebin.com/CxuBi9G0 I know I am using the right collection because I get the documents at first
[23:18:34] <StephenLynx> so I cut it completely trying to reduce stuff that could be going wrong and cut the flag and didn't try to put it