PMXBOT Log file Viewer

Help | Karma | Search:

#mongodb logs for Sunday the 24th of April, 2016

(Back to #mongodb overview) (Back to channel listing) (Animate logs)
[00:35:52] <kexmex> how come mongodb doesn't overwrite password in arguments so that it's not seen in PS?
[00:35:56] <kexmex> mongodump*
[00:47:58] <StephenLynx> PS?
[03:59:00] <bros> My replica set is f*cked up.
[03:59:05] <bros> I can't have one node go down to do maintenance...
[04:39:30] <Ryzzan> testing
[04:41:31] <Ryzzan> good tutorial teaching how to populate and then aggregate ($group) population result (is it possible?)
[04:42:06] <Ryzzan> anyone?
[04:42:16] <Ryzzan> new to nosql... and sry about my english... i'm brazilian
[04:47:28] <Ryzzan> ????
[05:52:38] <Ryzzan> good tutorial teaching how to populate and then aggregate ($group) population result (is it possible?)... anyone
[05:52:41] <Ryzzan> ?
[09:08:20] <Avocher> Hello, I've made too many collections is there a way to merge two or more collections?
[09:09:28] <Derick> there is no automated way. You need to read all docs, and import them into the other collection
[09:10:58] <Avocher> @Derick: okay thx
[11:15:51] <yopp> hey
[11:16:06] <yopp> is it possible to make an index on subdocument key name?
[11:22:15] <kurushiyama> yopp: you mean you have a document like { Foo: {Bar:"Baz"}} and you want an index on "Foo.Bar"?
[11:23:19] <yopp> More like I have { foo: { bar1: { bar: 1} } } and { foo: { bar2: { bar: 3 } }
[11:23:37] <yopp> and I'd like to be able to quickly search for "foo.bar1"
[11:24:32] <kurushiyama> yopp: You mean you are using values as keys? Good luck.
[11:26:18] <kurushiyama> yopp: That is a good way to get you into trouble.
[11:26:23] <yopp> why is that?
[11:26:47] <kurushiyama> yopp: As you noted, you'd have to have the names in your code in order to access the values...
[11:27:03] <yopp> My code can handle that just fine
[11:27:15] <kurushiyama> yopp: But it wont be able to handle indices.
[11:27:45] <kurushiyama> yopp: Believe me or not. This will get you into trouble and slow and in the long run hard to maintain code.
[11:28:01] <yopp> As I said, code is just fine.
[11:28:41] <kurushiyama> yopp: Well, as you seem to know better than I do, I answer your question: You will need to create an index for each key
[11:28:44] <yopp> Say, I have an document with subdocument that is list of events happened
[11:29:00] <kurushiyama> so it would be an index on bar1 and bar2
[11:29:03] <yopp> Events can happen multiple times, and I need to update timestamp on that event
[11:29:18] <yopp> or create new one if it new event
[11:29:24] <kurushiyama> yopp: Please pastebin a sample.
[11:31:47] <yopp> http://pastie.org/private/wx1d0aebjjaalmi1vrvq
[11:32:34] <yopp> This one allows me to just do simple $set: { "events.<token>.foo": Datetime }
[11:32:44] <kurushiyama> yopp: As I said, you'd have to create an index for each unique token in this case _as you use a value as a key_...
[11:33:30] <yopp> Okay, so simple answer: mongo can't index keys.
[11:36:52] <kurushiyama> yopp: http://hastebin.com/ifepegixiv.m
[11:37:11] <yopp> ha-ha
[11:37:12] <kurushiyama> yopp: BS. You screwed your data model, do not blame it on MongoDB
[11:38:37] <yopp> BS is that in your case you have to _manually_ maintain uniqueness of the events
[11:39:03] <yopp> so you need to read and then write back on application level. with eventId as a key you don't need that.
[11:39:17] <yopp> plus, you don't need to store extra field, which is pricy in my case
[11:39:44] <kurushiyama> yopp: I do not know how to explain it different. If in doubt, you are of course right, which was why you asked for help in the first place.
[11:42:16] <yopp> anyway, that's a not a big deal, I can denormalize data and store ids in separate field or even collection. My fight is to keep the record footprint as small as possible, as mongo can efficiently deal with billions of small records. Overhead is just too big.
[15:10:00] <Ryzzan> how to populate and then aggregate ($group) population result (is it possible?)... anyone
[15:48:33] <jr3> If I'm updating a numeric field in a document is there an advantage of using
[15:49:00] <jr3> $inc over say an replacing the field within the update
[16:24:00] <cheeser> jr3: consecutive updates wouldn't overwrite eac other with $inc
[16:56:18] <palaska> hey all, is anyone available? got a question
[16:57:01] <kurushiyama> Don't ask to ask, just ask. If you are lucky, you'll get an answer ;)
[16:57:29] <palaska> {
[16:57:48] <palaska> so here is it, in my model i got a field which is an array of items
[16:58:06] <kurushiyama> Array of items or array of subdocuments?
[16:58:35] <palaska> basically array of objects
[16:59:07] <palaska> and i am trying to find the model whose array has a specific object
[16:59:31] <palaska> there is a status field in the object that i can match
[16:59:52] <kurushiyama> Can you pastebin a sample doc?
[17:00:04] <palaska> but there is also a date field that i need to filter out the older objects
[17:00:11] <palaska> sure
[17:02:21] <palaska> http://pastebin.com/D07g7cRV
[17:02:23] <palaska> here is it
[17:02:27] <palaska> this is my model structure
[17:03:18] <palaska> i am trying to get a document by matching a status string and updated_at newer than a date
[17:03:45] <kurushiyama> I am talking of a plain sample document
[17:03:49] <kurushiyama> ;)
[17:04:11] <kurushiyama> Or do you want your answer in like Go or Java? ;)
[17:05:11] <palaska> no no :)
[17:05:30] <palaska> i am using mongoose + express
[17:05:44] <kurushiyama> palaska: The lingua franca is shell + BSON ;)
[17:06:27] <kurushiyama> palaska: But in general, I see a time series there implied, no?
[19:05:26] <Yatekii> hey guys!
[19:06:21] <kurushiyama> Yatekii: Hey ... Person?
[19:06:22] <Yatekii> when I have let's say a mailbox and some mails belonging to it. would you do a key on the mailbox object which holds a list of all mail objects or would you do a separate collection for the mails?
[19:06:41] <Yatekii> I am a Person indeed!
[19:06:50] <kurushiyama> Yatekii: Separate collection of mails
[19:07:04] <Yatekii> kk ty
[19:07:13] <Yatekii> what would you use a subobject for?
[19:08:43] <kurushiyama> Yatekii: Good question. I have hardly ever needed one. Much less an array of subdocs. I can give you an artificial example, though: http://blog.mahlberg.io/blog/2015/11/05/data-modelling-for-mongodb/#why-would-one-use-embedding
[19:09:59] <Yatekii> hmm okay, ty :)
[19:10:32] <kurushiyama> Yatekii: As per your use case, I'd have an additional field named folder, which would be a materialized path by definition. with the recipients email and the path, you should be good.
[19:10:56] <Yatekii> ya
[19:16:03] <Yatekii> another question: how would you delete an object that has no name, id and nothing like that except for an attribute thats a huge list of other objects?
[19:16:26] <Yatekii> (I accidentially have a nameless object because I didn't know that I have to specifiy EVERYTHING in the update query)
[19:17:40] <kurushiyama> Yatekii: I'd probably use an aggregation to find the docs with the highest number of elements for said attribute.
[19:17:58] <Yatekii> hmm ok
[19:18:03] <Yatekii> (I am very new :D)
[19:18:06] <Yatekii> gotta read then
[19:18:40] <kurushiyama> Yatekii: Well, you know the saying about a fish and fishing? ;)
[19:18:51] <Yatekii> ?
[19:19:43] <kurushiyama> "Give a man a fish, and you feed him for a day. Teach a man _how_ to fish, and you feed him for a lifetime."
[19:20:20] <Yatekii> ya
[19:20:32] <kurushiyama> Yatekii: I could of course give you the aggregation, but you'll learn more when you google for "mongodb $size aggregation"
[19:21:16] <Yatekii> yeye, I just didnt think of that :)
[19:21:21] <Yatekii> I tried name: ""
[19:21:28] <Yatekii> but that's different to SQL
[19:21:36] <Yatekii> it's really nameless not just an empty name
[19:23:05] <kurushiyama> Uh. Wait.
[19:24:06] <kurushiyama> Yatekii: http://hastebin.com/itegugomot.rb Works like expected
[19:25:38] <Yatekii> kurushiyama: didn't have the name attribute
[19:25:43] <Yatekii> just some other
[19:25:57] <Yatekii> biut the other objects in that collection had the name tag
[19:26:04] <kurushiyama> Yatekii: However, empty string is searchable. I could have used "dsljkasldjadoqjwoidöjaklösdj" as key
[19:26:04] <Yatekii> *attribute
[21:01:40] <Ryzzan> how to populate and then aggregate ($group) population result (is it possible?)... anyone
[21:32:03] <Ryzzan> kurushiyama: is there any irc mongoose channel?
[21:34:56] <StephenLynx> i recommend you don`t use mongoose at all.
[21:39:15] <Ryzzan> StephenLynx: so tell how woul i populate and aggregate with mongodb, would u?
[21:39:35] <StephenLynx> insert
[21:39:38] <StephenLynx> and aggregate
[21:40:33] <StephenLynx> http://mongodb.github.io/node-mongodb-native/2.1/api/Collection.html#insertMany
[21:41:13] <StephenLynx> http://mongodb.github.io/node-mongodb-native/2.1/api/Collection.html#insertOne
[21:41:18] <StephenLynx> http://mongodb.github.io/node-mongodb-native/2.1/api/Collection.html#aggregate
[21:43:17] <Ryzzan> what about populate? insert and populate r totally different ideas
[21:44:59] <StephenLynx> what is populate?
[21:48:44] <Ryzzan> let's say i have a document like this { _id: {type: Number}, fk : [{ _creator : {type: Number, ref : "SomeCollection" } }] }
[21:49:45] <StephenLynx> so
[21:49:50] <StephenLynx> just insert it
[21:50:12] <Ryzzan> then i would like to get data (populate) from "SomeCollection" (_creator would work as a foreign key) using a valur to related to this other collection
[21:50:15] <Ryzzan> got it?
[21:50:18] <StephenLynx> no.
[21:50:21] <StephenLynx> what does populate does?
[21:50:38] <kurushiyama> Ryzzan: The problem with Mongoose is that it forces you to see MongoDB as an SQLish or document store. Neither could be farer from the truth
[21:50:56] <kurushiyama> Ryzzan: Plus: You basically abstract away all advantages MongoDB has.
[21:51:18] <kurushiyama> s/document/object/
[21:51:47] <StephenLynx> i still have no idea what populate does.
[21:52:21] <Ryzzan> StephenLynx: let me try to give u an use case
[21:52:24] <kurushiyama> StephenLynx: Client side joins, as far as I got it.
[21:52:38] <StephenLynx> that is pointless, since mongo doesn't implement relations.
[21:52:44] <StephenLynx> use insert.
[21:52:46] <Ryzzan> kurushiyama: pretty much this, but way cleaner
[21:53:50] <Ryzzan> StephenLynx: but when i'm dealing with data from different collection... what would be the best strategy?
[21:54:12] <StephenLynx> run multiple finds.
[21:54:16] <Ryzzan> let's say i have a Collection to "movies"
[21:54:19] <kurushiyama> Ryzzan: It is conceptually problematic, to say the least, and StephenLynx can tell you some details about performance. populate basically is a good way to keep you from proper data modelling, further decreasing performance. Try it over a sharded cluster. It wreaks havox on UX.
[21:54:22] <StephenLynx> if your data is highly relational, don`t use mongo.
[21:54:35] <StephenLynx> simple as that.
[21:54:37] <Ryzzan> and another to artists
[21:54:40] <kurushiyama> Or use cayley on top of it for graphs
[21:55:03] <StephenLynx> if your data is only slightly relational, just run multiple queries.
[21:55:14] <kurushiyama> Ryzzan: Here is the thing
[21:55:31] <kurushiyama> Let us assume you want to display all actor names of a movie.
[21:55:49] <Ryzzan> kurushiyama: reading...
[21:56:04] <kurushiyama> Good idea: Add the actor names to your movie document
[21:56:31] <kurushiyama> So you can just do a db.movies.find({name"Hitchikers Guide to the Galaxy"})
[21:56:32] <StephenLynx> also that. often you can use embedded data to represent 1 to many relations.
[21:57:03] <StephenLynx> or even many to many relations, given the scenario.
[21:57:07] <kurushiyama> And now you Bad idea: do a populate, which is _at least_ two queries.
[21:57:41] <kurushiyama> Now, let us assume you have displayed your movie, and you want details on Bill Nighe
[21:57:45] <Ryzzan> kurushiyama: got it... gonna try and stay away from structured ideas
[21:57:53] <kurushiyama> Ryzzan: Not quite.
[21:58:32] <Ryzzan> keep the idea, kurushiyama ... what if i wanted Bill Nighe details?
[21:58:37] <kurushiyama> Ryzzan: Get your use cases right. Derive the questions on your data from them. Now, start to model for optimized UX.
[21:58:55] <kurushiyama> Ryzzan: You want to display them _together_ with the movies?
[21:59:00] <kurushiyama> Ryzzan: Unlikely.
[21:59:33] <kurushiyama> Ryzzan: But somebody clicks on Bill Nighe, then you can load Bill Nighe's details.
[21:59:56] <kurushiyama> Ryzzan: WHEN and IF you need it.
[22:00:06] <Ryzzan> kurushiyama: ok... got it
[22:00:08] <Ryzzan> ty
[22:00:27] <Ryzzan> StephenLynx, kurushiyama ... obrigado
[22:00:29] <Ryzzan> ;)
[22:00:38] <kurushiyama> Ryzzan: All Kudos to StephenLynx, who made me analyze Mongoose ;)
[22:01:47] <Ryzzan> i'll share the kudos
[22:01:50] <Ryzzan> :)
[22:01:55] <Ryzzan> c ya