Log file Viewer
#mongodb logs for Tuesday the 10th of May, 2016
[00:34:07] <kenalex> if i have a document like {'a':[{'id':2},{'id':4}],'b':[{'id':2},{'id':21}] } what query can I use to check if the lists refered to by fields 'a' and 'b' are equal ?
[00:38:05] <Boomtime> @kenalex: https://docs.mongodb.com/manual/reference/operator/aggregation/setEquals/
[05:45:46] <Keverw> Is Mongo 3.2 limited to one index per query? Was reading a post from 2012 mentioning that, but that's 4 years ago now.
[06:08:28] <kurushiyama> Keverw: As a rule of thumb, this still has its worth. Albeit there are index intersections (https://docs.mongodb.com/manual/core/index-intersection/), I did not find it easy to leverage them.
[06:13:37] <Keverw> @kurushiyama interesting. hmm that's a bit disappointing. I wonder in reality how much of a problem it is... I see the use of querying multiple fields at once.
[06:17:10] <kurushiyama> Keverw: Let me assure you: it is very,very rarely. I have yet to find a situation where this became a problem.
[06:18:03] <kurushiyama> Keverw: Correction: I had it once or twice, when a customer created an insufficient index. Remember: there are compound indices.
[06:18:37] <kurushiyama> Keverw: And said compound indices can be reused for single fields under certain conditions.
[06:22:54] <Keverw> @kurushiyama and I guess once collection could have multiple compound indexes? but only use one compound index per query? So say one collection but you wanted to beable to run a few different types of queries against it.
[06:22:55] <kurushiyama> Keverw: It is. Lets say you want to do sth like db.coll.find({foo:"bar",a:"b"}), you'd simply have an index {foo:1,a:1} (order matters)
[06:23:46] <Keverw> @kurushiyama Interesting. I'm starting to think everyone telling me to not use MongoDB just don't understand it or using out dated info.
[06:24:14] <kurushiyama> Keverw: The advantage with the example above is: Even when you only do a query like db.coll.find({foo:"bar"}), the index {foo:1,a:1} is used
[06:24:34] <kurushiyama> Keverw: "just don't understand it or using out dated info." I could not agree more
[06:25:07] <Keverw> @kurushiyama Oh nice. :) I think mongo is cool because it's scale options, and I kinda hated writing SQL and converting code back and forth.
[06:25:53] <Keverw> @kurushiyama but some friends told me it was a bad idea, and even a company I consulted with about their commercial database followed up with me and I told them I was going with Mongo, and said some things that got me rethinking.
[06:26:30] <kurushiyama> Keverw: The people claiming that MongoDB looses data simply did not read the docs. There is a notion of writeConcern, which gives very granular control over when a write is acknowledged
[06:27:45] <Keverw> @kurushiyama Interesting. I think there's a way to default that to write to a few repl. I know I seen some exmaples when they included it with the update statements.
[06:29:10] <kurushiyama> Keverw: There was a rather unfortunate decision (in hindsight) that the _default_ writeConcern used to be 0, basically saying "acknowledge that the write reached the server only". With a default writeConcern nowadays, this is different. And if people would have read the docs, it would have been different before.
[06:31:23] <kurushiyama> Keverw: Bottom line: You have more granular control (configurable in general and per write operation) over durability than with any other DBMS I have met.
[06:34:29] <kurushiyama> Keverw: And let me guess: You did what you used to do with RDBMS: You started with identifying your entities, their properties and relations and modelled accordingly?
[06:34:45] <Keverw> @kurushiyama I was wondering though. It mentions shardkey in it's schemas but I couldn't find where to set it for the native driver. So that's what I was about to ask about.
[06:35:54] <kurushiyama> Keverw: One of the major advantages of MongoDB is that you can model your data und the according queries optimized for your use cases.
[06:36:16] <Keverw> @kurushiyama hmm interesting. I figured it'd help stay organized and prevent typos of field names.
[06:36:39] <kurushiyama> Keverw: To leverage this advantage, however, you need to approach modelling differently
[06:37:56] <kurushiyama> Keverw: SQL way: Identify entities, properties, relationships, bang your head against the wall for some time to get upper left above and beyond JOINS right to get the questions answered you have to your data to cover the use cases.
[06:38:54] <kurushiyama> Keverw: MongoDB: Identify your use cases, derive the questions you have on your data for them, create optimized model to answer those questions.
[06:40:11] <Keverw> @kurushiyama hmm. other than that, anything else wrong with Mongoose? There was some other additions it had I liked, but I could always emulate them if I wanted with some custom functions.
[06:40:50] <kurushiyama> Keverw: With optimizations made for the most common use cases. I have to admit that this sometimes means that you have to take performance "penalties" for less common use cases, or accept redundancy (not a bad thing in MongoDB).
[06:41:44] <kurushiyama> Keverw: Other than that, if you model properly, Mongoose is not much more than an additional moving part, introducing an additional layer in your app eating performance.
[06:44:01] <kurushiyama> Keverw: That is _exactly_ the point. It emulates JOINS. What they do not tell you is that these are client side JOINS, inherently using rather inefficient queries.
[06:45:35] <kurushiyama> Keverw: Let's say you have users and posts. user {_id:someId, name:"Keverw"...}
[06:46:12] <kurushiyama> Keverw: Now, you have a use case in which you want to display the last 10 posts made by any user on a page
[06:47:17] <kurushiyama> Keverw: Of course, we could model a post like {_id:someId, user: userId, text:"foo"} and leverage Mongoose's populate to get the user names.
[06:48:56] <kurushiyama> Keverw: But lets say we only want to display the user names together with the posts. We can model our posts like {_id:someId, user:"Keverw", text:"foo"} and reduced the use case to a single query.
[06:52:20] <Keverw> Interesting. I really liked the idea of storing names or picture urls once. Updating thousands or even millions of records sounds wasteful compared to just a ID. but hmm. That would probably get tricker when checking if a user is suspended or not, etc
[06:53:54] <kurushiyama> Keverw: That example was heavily simplified for the sake of making it easy to grasp.
[06:54:57] <kurushiyama> Keverw: But what you should understand is that we optimized for a quite common use case (displaying posts) vs a rather uncommon one (changing of a username).
[06:56:14] <kurushiyama> Keverw: Oh, and btw, updating the username of a user in all posts would be a single query: db.posts.update({user:"Keverw"},{$set:{user:"KeverwMongoMonger"}})
[07:00:00] <Keverw> @kurushiyama hmm interesting. What about profile pictures? hmm. I guess I was always taught to store something once but I guess that's not always the case with Mongo then.
[07:01:20] <kurushiyama> Keverw: Many ways to do it. What I would do is have a fixed url: http://my.app/profiles/<username>/profile.jpg
[07:03:44] <kurushiyama> Keverw: If all you take out of this conversation is that one should model optimized for the most common use cases, that'd be great.
[07:04:33] <Keverw> @kurushiyama at least most things are post specific. things like friends probably not a good idea to store in every post but hmm. Even for some PHP projects i've done joins in PHP as I really hated writing joins. I'd just make to where WHERE IN instead of doing a loop to get multiple items with the same ID in the array.
[07:06:04] <Keverw> @kurushiyama Yeah. You surely do have me thinking a bit differently on how to approach things.
[07:16:25] <Keverw> @kurushiyama With the native driver, do you need to include the shardkey when doing queries, adding or updating content? I know Mongoose mentions it, but say you were mixing both drivers, or say later on you shareded and not apps restarted yet... I wonder if anything odd would happen with the queries not including the key. I think all it would do is hit all shards, so a bit slower but wouldn't break.
[07:19:08] <Keverw> @kurushiyama Considered doing it right away, also later. I was thinking maybe later.
[07:19:53] <kurushiyama> Keverw: In general, you never _need_ to include the shard key in your query. But you definetly _should_.
[07:20:34] <kurushiyama> Keverw: Picking a proper shard key for your use case can be quite tricky and needs careful consideration.
[07:20:45] <Keverw> @kurushiyama Oh I see. so just for optimizations. and I think you need them for queries only, not inserting new content?
[07:21:45] <Keverw> @kurushiyama Yeah. Probably shouldn't worry about it right away but as the user base increases I'll need to study on picking them but I think it should be fairy trivial to migrate.
[07:23:19] <kurushiyama> Keverw: Say you shard on the default _id, but hashed, and you use objectIds. In this case, you would not need to include the shardKey. If you omit the shard key, it evaluates to null and the resulting doc will be stored on the according shard (the one assigned $minKey, iirc)
[07:25:01] <Keverw> @kurushiyama Interesting. I had a feeling most cases would shard on the id probably but hmm
[07:26:24] <kurushiyama> Keverw: I am not too sure. Say you want a shard per region (EMEA, NCSA, APAC), then it might be reasonable to already include those reagions and use them as a shard key later. But choosing a shard key is probably one of the most difficult things to do in MongoDB, because there are so many factors to be taken into account
[07:26:35] <Keverw> @kurushiyama I guess as with everything, once I gain more experience with this I'll be moving a lot faster than I have been so far.
[08:40:22] <jokke> i have time series data with a timestamp in _id. Each doc has an attribute panel_id. I need to query the last docs (by timestamp) for a specified panel_id. I'm quite clueless on how to do this.
[08:40:49] <jokke> this won't work because memory limits get exceeded: https://p.jreinert.com/1NL/javascript
[11:51:12] <jlambert121> on a clean centos 7 box with all updates, mongodb 3.2.6-1.el7 starting mongo I get selinux errors (http://pastebin.com/Xb6QPCUB). setenforce 0 allows mongo to run correctly so i think the issue is just selinux. google hasn't been helpful in finding a solution to this, has anyone else had this issue?
[12:53:55] <Zelest> Derick, I've installed the new MongoDB client and try to $m = new MongoDB\Client('...'); and I get the error that class MongoDB\Client doesn't exist.. o_O
[12:59:34] <kurushiyama> Zelest: You sure about the backslash? I am not exactly an expert on PHP, but that looks strange to me.
[13:00:58] <Zelest> kurushiyama, That's what namespace to use (if I know my php correctly), neither MongoClient, MongoDB or Mongo works either though :S
[13:12:33] <Zelest> Derick, I've installed the pecl-mongodb driver (1.1.6) .. is there anything else I need to install?
[13:28:02] <Zelest> Now I get it.. the mongodb driver is just a very high-level framework to do magic mongodb stuff.. Then you use a php-written library on top of that, like https://github.com/mongodb/mongo-php-library
[14:06:31] <Derick> Zelest: \MongoDB\Client is part of the PHP library, which you need to fetch through composer
[14:13:13] <Zelest> Derick, anything new I need to know regarding the replicaset features in the new mongodb driver? E.g, automagic failover and such..
[14:53:32] <kurushiyama> Both are indexed (_id is indexed by default), so both can be searched for efficiently
[14:55:27] <kurushiyama> Ben_1: https://docs.mongodb.com/manual/reference/method/ObjectId/#description
[15:05:58] <kurushiyama> cheeser: Well, I see that. But if we assume that BSON is supposed to be a data representation independent of MongoDB... ;)
[17:48:58] <diogogmt> i’m trying to follow the steps in this article to create the first user with authentication enable using the LocalHostException but it doesn’t seem to work: https://docs.mongodb.com/v2.6/tutorial/enable-authentication-without-bypass/
[17:49:15] <diogogmt> i keep getting the message: “Error: couldn't add user: not authorized on admin to execute command”
[17:51:15] <diogogmt> this lines says: “After you enable authentication, only the user administrator can connect to the MongoDB instance. The user administrator must log in and grant further access to the instance by creating additional users.”
[17:51:41] <diogogmt> i’m connecting to the mongo shell without specifying a user, do i need to specify the admin user? what about its password?
[18:40:20] <wspider> for a node app, should I create one connection to mongodb and pass it as parameter to all the functions users will use?
[18:42:45] <wspider> not really, just found the answer in google, you create one connection object and reuse it through the entire app
[19:19:54] <tozz> When using $unset on a couple of nested objects the result is actually that the entire parent tree is removed too. "$unset": { "obj.level1.level2.level3": "" }, The resulting document has obj.level1 deleted, is this expected?
[22:55:47] <ayee> I'm trying to deploy ops manager. It worked initially, though after I added my cert through the webui, it no longer works.
[22:56:14] <ayee> The pem format is unclear to me, is it private key, domain cert, chain certs (arbitrary in length) all in the same file separated by line breaks?
[23:12:55] <ayee> Hmm, I've configured apache/nginx many times, but the certs and private key are always in separate files. it's non-standard to mix the certs and private key into an ascii pem file.
[23:15:46] <ayee> I tested these certs with nginx on the hello world page, and it works fine with a separate pem (with 3 certs in it), and different file for the key