Log file Viewer
#mongodb logs for Thursday the 31st of March, 2016
[09:17:56] <einseenai> hello, everyone! I'm trying to build a basic blog with Mongoose, Node and Express. I can get the list of entries, create an entry and delete them, but when I proceed to an individual entry from the list, my server crashes and that's it. here's the piece of code responsible for retrieving a post and output that I get - https://jsfiddle.net/pcfyesr4/1/ Could you, please, have a look and tell what am I doing wrong here?
[09:21:14] <kurushiyama> einseenai Formally: you should check for error first, than access the result – that being said without knowing poo about node.
[09:21:54] <kurushiyama> einseenai And in your error check, you should probably log the error returned, so that we can find out more.
[09:40:05] <einseenai> although, one strange thing, I noticed - if I try to access an entry that doesn
[09:40:59] <einseenai> 't really exist - like, localhost:3000/blog/non-existant-entry, the server _does not_ crash, it just displays emptry entry
[09:47:43] <kurushiyama> einseenai I do not know too much about node, but I assume that the template language gracefully handles null values.
[10:11:00] <einseenai> kurushiyama, thank you a lot, it made my day :-), for the sake of logging of the channel non-existent entry problem is solved basically like this - https://jsfiddle.net/pcfyesr4/6/
[10:14:54] <einseenai> kurushiyama, one last question - should I cache database calls' results? I mean it has to be not a good thing to call a db on every page update, right? to json, for example?
[10:16:08] <kurushiyama> einseenai With adding cache, you add a moving part, and – please forgive me – you do not seem to be well experienced.
[10:16:44] <kurushiyama> einseenai So keep your application as simple as possible in the beginning. "Premature optimization is the root of all evil" -- Donald Knuth
[10:19:07] <kurushiyama> einseenai Not really. The problem with caching is that getting eviction right is _very_ hard to do across a cluster.
[10:19:43] <hyades> Hi. I have a aggregate cursor stream coming in. I then attempt to push this result into another collection. I am doing this using Bulk upserts.
[10:19:44] <hyades> I expect my stream results to run into millions, so bulk created would be huge. I want to know the recommended way of doing this.
[10:22:19] <kurushiyama> hyades Any reason for not using an $out stage? https://docs.mongodb.org/manual/reference/operator/aggregation/out/
[10:23:10] <hyades> kurushiyama: the out collection is on different DB, and I have updates are upserts with some logic going on.
[10:24:55] <kurushiyama> hyades No. You run your aggregation and have it written to some temporary collection.
[10:25:39] <kurushiyama> Then you simply do a find() on said temporary collection and do whatever you have to do.
[10:25:44] <hyades> kurushiyama: yea got that part. But how do I do my upserts? Should I iterate over this $out collection, and create bulk upserts?
[10:26:30] <kurushiyama> hyades However, it may make sense to commit your bulk ops from time to time. I tend to commit them every 10k ops.
[10:28:02] <hyades> kurushiyama: i guess then I should read stream till 10k, pause, bulk upsert, resume
[10:28:32] <kurushiyama> hyades The problem here is that the stream may be hellish expensive. Gimme a sec
[10:31:39] <kurushiyama> hyades Please, gimme a few mins, I am wrinting something up to make it more clear.
[10:37:47] <kurushiyama> hyades The problem you _may_ have is that your aggregation needs more than 100MB of RAM, in which case you need to basically swap to disk and you might run into scatter/gather problems in sharded environments.
[10:38:58] <kurushiyama> hyades It is not that my solution will prevent either of them, but you split them problem into individual parts. When implemented properly, you might be able to do parallel processing of your aggregation result set.
[10:41:52] <kurushiyama> hyades BTW: From my side, the alarm bells would ring when you need to do cross database data correlation or/and aggregation. This tends to be a good sign that there is something wrong with your data structure. imvho.
[10:43:51] <kurushiyama> hyades For example. Your language might be capable of using multiple cores, so you could start multiple threads/processes splitting the $out collection between them.
[10:54:26] <kurushiyama> hyades Chances are that you split data over different dbs which should not be split. It is sort of a circumstantial evidence.
[11:06:43] <ruohki> Hey, is it possible to cut down the character of a field that is returned. For example if you want to build a read more text
[11:10:24] <ruohki> yeh i could do that but iam allways trying to find alternative ways of doing things
[12:12:53] <kryo_> anyone using mongoose can explain this behavior? http://hastebin.com/uxuxudipuj.coffee
[13:36:35] <StephenLynx> unlike a tool, that either enables you to perform a task or improves your performance when performing said task.
[13:37:22] <StephenLynx> and specially not when you use a broken one that brings more trouble than saves you.
[13:42:59] <mkozjak> is there a way to make wiredtiger to sync data to another persistent storage like elasticsearch? so with every checkpoint (60 seconds), can it sync data to elasticsearch instead of its own default storage on disk?
[14:43:03] <pamp> I have the first match stage like this--> { "meta.parameterId": {$regex:"20160329_ericsson_threeg_.*eul.*"} }
[14:46:54] <kurushiyama> pamp a) it would help if you used pastebin for your full aggregation b) indices are only used for $match and $sort _at the beginning of the pipeline_. c) the whole result set of your $match is fed into the subsequent stages
[14:48:01] <kurushiyama> pamp An aggregation is not necessarily meant as a replacement for queries. If you need performance on the result, preaggregate and use an $out stage
[14:51:53] <kurushiyama> pamp A slightly more precise definition of "really fast" and "really slow" could make sense, too. And: Do you run the aggregation on a sharded collection?
[15:39:11] <kurushiyama> jr3 https://docs.mongodb.org/manual/reference/method/db.collection.findAndModify/#db-collection-findandmodify
[15:45:19] <kurushiyama> jr3 Since only the first document found will be updated, simply use a sort on your date field in ascending order, which means that the first document found will be the oldest. Set the update document accordingly and you are good to go.
[15:49:57] <kurushiyama> jr3 Bad luck if you are in a sharded cluster, though. In that case, it gets a wee bit more complicated.
[15:56:46] <Perseus0> Hey, I'm still wondering where the db is being stored because this is all I get when I run these cmds on the mongo shell: http://kopy.io/dp9OQ
[15:58:46] <crazyphil> anyone here aware of any utility/connector/api that would allow pulling documents from mongo into kakfa?
[20:33:53] <dimaj> Hello, I was wondering if there's a way to get the timing of an aggregation query
[22:06:06] <hackel> I've got an index and cannot even start mongod after upgrading to 3.2. How can I fix this? Do I have to downgrade mongod just to remove the invalid index?
[22:40:56] <hackel> kurushiyama: I actually just ended up downgrading it to remove the index, and that worked fine. Thanks.
[22:42:33] <hackel> kurushiyama: The log entry that prevented the server from starting was: Found an invalid index { v: 1, unique: true, key: { email: 1.0 }, name: "email_1", ns: "db.users", sparse: true, partialFilterExpression: { deleted_at: null } } on the quirks.users collection: cannot mix "partialFilterExpression" and "sparse" options
[22:43:33] <kurushiyama> ellistaa It is a misconception (and a severly wrong one) that MongoDB can not handle exceptions properly. They are just not as straightforward as people used to SQL know them to be ;)
[22:45:52] <kurushiyama> ellistaa In general, I probably would use a MongoDB (suprise, surprise) and use Cayley (a graph database which can use MongoDB as storage backend and has a REST API) for accessing certain relations.
[23:04:47] <kurushiyama> ellistaa I suggest to take cover. All JS specialists (I am not one of them) seem to doom mongoos. Personally, I see the problem that it makes some of the strengths very hard, since it misuses MongoDB as a simple object store.
[23:31:45] <kurushiyama> ellistaa MongoDB is a document oriented database, no object store. One of it's advantages is that with carefully crafted data models and queries, you can achieve amazing performance. With Mongoose, being an ODM, you take away that possibility.
[23:32:14] <ellistaa> kurushiyama: so how would i query the db from my node app if i dont use mongoose?
[23:36:55] <kurushiyama> Maybe ODMs actually are easier to grasp for people coming from relational data. Doing data modelling "on foot" can be hard to understand and is highly error prone, as proven by highly overembedded documents.
[23:39:47] <kurushiyama> Albeit my eyes start to bleed when I see some docs created by Mongoose, specifically.
[23:44:14] <kurushiyama> Used to use morphia for quite a while, that was actually a nice experience.
[23:52:34] <kurushiyama> Either you use it as a lightweight ODM using structs and tags or you use maps. And it is fast. Nice to use. And performance comparisons made me learn Go just to use mgo.
[23:54:21] <skot> It has a single designer with their own perspective, and that will get you that kind of taste...
[23:55:30] <cheeser> i haven't used it overly much though so I still have a lot of familiarization to do with it.
[23:58:38] <kurushiyama> Tbh, Gustavo did an _awesome_ job. Many of the weird things stem from the fact that he tried to be as close the Golangs stdlib as possible, imho,