pmxbot IRC Log Viewer

[10:10:26] <crodjer> Does it make sense to use hashed index for saving space? I have long id keys. Also, what hashing algorithm will be used?

[10:13:35] <kurushiyama> crodjer: Well, hashing takes time. Not much, but it does. I would not. Not sure how index prefix compression works, but for me, that is usually enough.

[13:27:48] <gain_> hello to all, I need to search by id but find({_id:"asd123"}) obv not works because the _id is an object and not a string...

[13:28:45] <cheeser> is _id an ObjectId ?

[13:28:52] <gain_> yes...

[13:29:06] <cheeser> find({_id: ObjectId("asd123") })

[13:30:00] <gain_> cheeser: thanks

[13:31:06] <cheeser> np

[13:36:00] <gain_> cheeser: that works in mongo cli but not with nodejs driver... maybe you know the equivalent for the nodejs driver?

[13:37:15] <StephenLynx> gain_ et me see

[13:37:25] <StephenLynx> ok, hold on

[13:37:37] <cheeser> i don't js

[13:37:53] <StephenLynx> var mongo = require('mongodb');

[13:37:53] <StephenLynx> var ObjectID = mongo.ObjectID;

[13:38:11] <StephenLynx> _id : new ObjectID(yourstringid)

[13:38:56] <StephenLynx> if the string id is invalid, it will throw an exception

[13:39:04] <gain_> StephenLynx: I'm trying the same but got an error

[13:39:18] <gain_> uhm... what do you mean with invalid?

[13:39:18] <StephenLynx> what is your string?

[13:39:22] <StephenLynx> for example

[13:39:27] <StephenLynx> 'sdfjshfsjdfssd' is invalid.

[13:39:31] <StephenLynx> it will throw an exception.

[13:39:37] <gain_> 1047fbb7-d74a-4888-9a51-5f2830ad7d09

[13:39:39] <StephenLynx> yeah

[13:39:43] <StephenLynx> that is not a valid _id format.

[13:40:19] <StephenLynx> show me your document from the terminal

[13:40:30] <gain_> Error: Argument passed in must be a single String of 12 bytes or a string of 24 hex characters

[13:40:40] <StephenLynx> yup

[13:40:44] <StephenLynx> exactly that.

[13:41:20] <gain_> oh f*** it's a string...

[13:41:36] <StephenLynx> not with 12 bytes or 24 hex characters.

[13:41:59] <StephenLynx> if the string can be invalid by design (user input) you use a try-catch

[13:43:31] <gain_> if _id is a string on my db find({ _id: "1047fbb7-d74a-4888-9a51-5f2830ad7d09" }) should work... am I wrong?

[13:44:34] <StephenLynx> no

[13:44:41] <StephenLynx> if that's the case

[13:44:44] <StephenLynx> you just compare directly

[13:44:51] <StephenLynx> since your _id is not an ObjectId

[13:44:55] <StephenLynx> and just a string.

[13:45:13] <StephenLynx> _id: myStringId and off you go.

[13:45:46] <StephenLynx> I wouldn't fiddle with _id anyway and make it anything but the default, but that's just me.

[13:47:13] <gain_> in cli works but not in the nodejs driver... xD

[13:48:24] <gain_> StephenLynx: I would not fiddle with _id too... but that's what I have, so I think I have to fix a mistake... :)

[13:48:39] <StephenLynx> it should work just fine.

[13:49:03] <StephenLynx> your code is probably wrong or theres something really fucked up in the driver.

[13:49:13] <StephenLynx> which I find unlikely.

[13:49:19] <kurushiyama> StephenLynx: Well, you can use natural _ids. But one should be very aware of the implications...

[13:49:22] <StephenLynx> are you using mongodb or something else?

[13:49:28] <StephenLynx> I know, kurushiyama

[13:49:54] <kurushiyama> gain_: Weren't you using this minimongo thingy?

[13:49:55] <StephenLynx> the thing is how _id is a special field, which causes said implications.

[13:50:15] <gain_> kurushiyama: hahaha, no, that's not the case... server side now... :P

[13:52:52] <gain_> StephenLynx: it was a wrong import by someone... I just have to update some things, so I need to script a db update finding by id

[13:53:06] <StephenLynx> ok

[13:53:12] <StephenLynx> and what are you using as a driver?

[13:53:19] <gain_> than I'll try to recreate all ids with ObjectId in order to clean all... I hope...

[13:53:20] <StephenLynx> mongodb?

[13:53:33] <gain_> StephenLynx: yes, mongodb installed with npm

[13:53:36] <StephenLynx> ok

[13:53:42] <StephenLynx> then your code is probably wrong.

[13:54:56] <kurushiyama> > yes, mongodb installed with npm

[13:55:06] <kurushiyama> The driver, that is?

[13:55:20] <StephenLynx> kek

[13:55:22] <StephenLynx> it is.

[13:55:35] <StephenLynx> that's the driver's module name.

[13:55:38] <kurushiyama> Thanks to His Noodly Appendage!

[13:58:08] <gain_> kurushiyama: don't you like nodejs? xD

[13:58:34] <kurushiyama> gain_: You do not want to open this gate...

[13:58:37] <kurushiyama> gain_: ;)

[13:58:47] <gain_> ok, ok... sry... xD

[14:00:53] <gain_> StephenLynx: I've found the error... -.-"

[14:01:08] <StephenLynx> in your code? :V

[14:01:24] <gain_> y

[14:01:26] <StephenLynx> v:

[14:01:51] <kurushiyama> gain_: To be fair: It is not node. Not even the ecosystem. But sometimes, when I see node related questions on SO, I ask myself wether the self-proclaimed developers should spend less time with demanding their jobs to be be done for them and more with thinking.

[14:02:55] <cheeser> if people did that, we'd have a shot at getting rid of js and putting in something decent.

[14:03:01] <kurushiyama> gain_: (or reading docs, for that matter). Do not get me wrong though. Personally, I avoid node like the devil holy water, but I see the advantages.

[14:03:01] <cheeser> but, alas. inertia.

[14:03:09] <StephenLynx> yeah, node.js community is cancerous.

[14:03:18] <StephenLynx> I use node.js a lot, but I avoid the community wholly.

[14:03:46] <StephenLynx> the tool is great on itself, though.

[14:03:56] <StephenLynx> and the ecosystem is designed reasonably.

[14:04:54] <kurushiyama> StephenLynx: Just not the QA. one line npms? who came up with the "isPositiveInteger" package? cheeser, was that you?

[14:05:10] <StephenLynx> yeah, that's the community being full of dumbasses.

[14:05:24] <StephenLynx> the actual design of how npm works is ok.

[14:05:34] <cheeser> kurushiyama: i don't js. ;)

[14:05:34] <StephenLynx> if people misuse it, is a different issue.

[14:05:52] <kurushiyama> cheeser: More in a way "Did you mock about it?"

[14:06:04] <StephenLynx> the same could be done on any other ecosystem that supports dependency injection or w/e its called.

[14:06:13] <StephenLynx> you could also make dumb one-line C libraries.

[14:06:24] <cheeser> dependency management? DI is something else entirely.

[14:06:36] <StephenLynx> yeah, that.

[14:06:43] <kurushiyama> StephenLynx: Well, Perl had a thing that you had to announce a package on a mailing list. An "isPositiveInteger" would have been a very, very short joke.

[14:07:21] <cheeser> at least it's a one-liner

[14:07:26] <StephenLynx> that's bad, though.

[14:07:31] <kurushiyama> Oh, holy pun, Batman!

[14:07:52] <StephenLynx> to require centralized decision on what makes it or not.

[14:08:08] <StephenLynx> i still prefer the freedom that npm offers and all its woes.

[14:08:20] <StephenLynx> than having someone deciding on what should be there or not.

[14:08:30] <kurushiyama> StephenLynx: Well, it is a decision. Either have a QA or not. Have a look at Debian...

[14:08:42] <StephenLynx> brb lunch

[14:08:52] <gain_> got it...

[14:08:56] <StephenLynx> and even then

[14:09:01] <gain_> StephenLynx: have a good lunch

[14:09:02] <kurushiyama> StephenLynx: Well, it weren't just a few. Basically, every CPAN member could argue. It was a process of discussion.

[14:09:03] <StephenLynx> you don't have to use npm to use modules.

[14:09:16] <kurushiyama> StephenLynx: Good munch!

[14:09:20] <StephenLynx> npm is not part of the ecosystem design, actually.

[14:09:28] <StephenLynx> it just works based on how the modules are designed.

[14:19:37] <alexi5> what problems you guys have with the node.js community ?

[14:22:23] <cheeser> besides the obvious leftpad debacle? :)

[14:23:01] <alexi5> leftpad debacle ?

[14:23:05] <alexi5> i am new to node.js

[14:25:54] <cheeser> oh, man.

[14:26:25] <edrocks> kurushiyama: will wiredtiger keep everything in memory if I have enough?

[14:26:31] <cheeser> http://www.haneycodes.net/npm-left-pad-have-we-forgotten-how-to-program/

[14:26:43] <cheeser> edrocks: it should, i believe, yes.

[14:26:46] <kurushiyama> edrocks: Sorta.

[14:27:00] <edrocks> I'm maxing out all my servers

[14:27:29] <kurushiyama> edrocks: I'd _really_ have a deep look into the memory utilization before doing so.

[14:28:10] <edrocks> kurushiyama: I had to buy a 3rd one anyway and I got a good deal + It costs a lot to drive up and install stuff

[14:29:27] <kurushiyama> edrocks: Usually, people tend to max out memory because of problems.

[14:30:19] <kurushiyama> edrocks: And usually, those problems turn out to be either mas storage IO related or bad indices/query order/you know the stuff.

[14:30:54] <kurushiyama> edrocks: But ofc, more RAM would not hurt.

[14:31:29] <edrocks> kurushiyama: I wanted more ram. I was going to buy 64gb for ea server but I decided to max them

[14:31:30] <kurushiyama> edrocks: As long as you are still below the "sweet spot" adding RAM can actually save money ;)

[14:31:39] <kurushiyama> edrocks: o.O

[14:31:43] <edrocks> I got 32GB sticks for $140 ea

[14:32:04] <kurushiyama> edrocks: You are _buying_ them? O.O

[14:32:13] <edrocks> yea I bought 1.1TB yesterday

[14:32:37] <kurushiyama> Well, then the decision is already made.

[14:33:08] <edrocks> I wasn't going to do it unless I got them to $155 ea but I called the vendor and they said $140

[14:35:27] <kurushiyama> edrocks: Replset or sharded cluster?

[14:35:33] <edrocks> kurushiyama: replset

[14:35:44] <edrocks> finally get to have 3 nodes and do it right

[14:38:27] <StephenLynx> alexi5, simply put, most people there can't program at all.

[14:38:30] <StephenLynx> is not that they are bad

[14:38:36] <StephenLynx> they just can't program.

[14:39:30] <alexi5> for someone looking to learning node.js, what resource or forum is best ?

[14:40:14] <StephenLynx> official documentation.

[14:40:24] <StephenLynx> https://nodejs.org/api/

[14:40:28] <StephenLynx> don't use frameworks, either.

[14:40:35] <StephenLynx> they are all garbage.

[14:40:44] <alexi5> ok

[14:40:48] <StephenLynx> and try to keep dependencies to a minimum

[14:40:50] <StephenLynx> most are garbage too

[14:41:01] <alexi5> the leftpad library is very funny

[14:41:16] <StephenLynx> kek

[16:59:43] <kurushiyama> edrocks: Sorry, got distracted on #go-nuts. Uhm. so you'll have 128Gigs in the members?

[17:00:14] <edrocks> kurushiyama: I'll have 3 servers with 382GB ea

[17:00:59] <edrocks> kurushiyama: they run some other stuff too, so probably around 100GB ea for mongodb

[17:01:08] <kurushiyama> edrocks: My bet is that most likely, your disk IO is much more of a bottleneck.

[17:01:30] <edrocks> at this point I just bought ram disks

[17:02:10] <edrocks> I got a great deal on ssds too

[17:02:15] <kurushiyama> edrocks: What is the other stuff running on those servers?

[17:02:44] <edrocks> kurushiyama: elasticsearch, redis, influxdb(on one server), some internal stuff and the app servers

[17:03:23] <kurushiyama> edrocks: Not ideal. since both InfluxDB and MongoDB are quite IO-heavy.

[17:03:40] <kurushiyama> edrocks: I'd rather scale out than scale up.

[17:03:51] <edrocks> kurushiyama: was cheaper to go up at this point

[17:04:13] <edrocks> kurushiyama: I agree with scaling out later on

[17:04:17] <kurushiyama> edrocks: Do you have IOWaits?

[17:04:35] <edrocks> kurushiyama: I don't believe so

[17:04:56] <edrocks> kurushiyama: it was more of a reliability thing with getting a 3rd server so all the cluster stuff worked as intended

[17:04:57] <kurushiyama> edrocks: What is your SSD setup? RAID? Which level, if yes?

[17:05:30] <edrocks> kurushiyama: raid iirc 1. the third server will be raid 10 with 4 ssds the other 2 have 2 larger ssds each

[17:06:04] <kurushiyama> edrocks: Makes sense, assuming the RAID10 one will be the one bearing influxdb.

[17:06:22] <edrocks> the 1tb of ram was mostly so I could forget about going there for a while and I got the price down by buying a bunch at once

[17:38:01] <shlant> anyone know why I would get "SCRAM-SHA-1 authentication failed for blah on admin from client 172.17.0.2 ; AuthenticationFailed SCRAM-SHA-1 authentication failed, storedKey mismatch"

[17:38:12] <shlant> does that mean the password is wrong?

[17:39:45] <StephenLynx> or the account does not exist

[17:40:14] <StephenLynx> maybe authentication is disabled and the client expects it to be enabled? just guessing.

[18:01:59] <dino82> hi all o/ -- How long does mongodump take before I actually see files being written? Is it supposed to be immediate? Also, can I run a backup while the node I am connecting to is still indexing?

[18:04:58] <kurushiyama> dino82: Foreground or background index?

[18:05:45] <dino82> Foreground

[18:12:50] <kurushiyama> dino82: Bothered to read the docs?

[18:13:42] <kurushiyama> dino82: https://docs.mongodb.com/manual/reference/method/db.collection.createIndex/#behaviors, namely

[18:13:53] <kurushiyama> dino82: "Non-background indexing operations will block all other operations on a database."

[18:14:44] <dino82> Simple question simple answer, 'no'. Thanks

[18:15:51] <kurushiyama> dino82: Not so simple. I was actually trying to suggest that you read the docs _before_ you use an operation to understand what is going on.

[18:21:13] <dino82> o7

[18:21:37] <kurushiyama> dino82: Miss, no ship there. ;)

[18:23:10] <dino82> ;__;

[19:04:00] <alexi5> hello guys what are some good ways of implementing document versioning ?

[19:06:48] <cheeser> using java/morphia, @Version

[19:10:18] <alexi5> i am using c#

[19:10:59] <alexi5> basically what I have is a collection with documents for a product and price. but when price changes I want to store the older version of the document

[19:11:52] <alexi5> so far I am thinking of having an array call pricing version ing document, but that will laod all history when document is fetched or have a collect for storing the old product documents

[19:12:11] <kurushiyama> alexi5: I would actually treat this as a time series.

[19:13:04] <alexi5> as in a document has an array that contains all the old documents ?

[19:14:11] <kurushiyama> Nope

[19:14:16] <kurushiyama> alexi5: Because...

[19:14:32] <kurushiyama> ?

[19:15:10] <alexi5> can you give me an example of a possible document implementation ?

[19:17:07] <kurushiyama> {_id:someId, product:"Foo", price:12.95, date: today},{_id:someId, product:"Foo", price:11.95, date: yesterday}

[19:18:06] <alexi5> so store old documents in another collection ?

[19:21:36] <kurushiyama> alexi5: I'd do so. It is not that collections are expensive. If you need a pricing history, it does not get any cheaper. Use redundancy where necessary. I'd probably have a collection for storing the data I need for a product overview.

[19:22:42] <alexi5> and aggregation would can be easily done to see past history status on documents

[19:31:35] <kurushiyama> alexi5: db.prices.find().sort({date:-1}).limit(1) for the current price. Add the fields you need for an overview, and you might be where you want.

[19:33:19] <alexi5> so current pricing and history all in the same collection

[19:33:48] <Perseus0> I have users collection @mongodb with a createdAt date field but can't seem to be able to return it when I do Users.find().fetch()

[19:35:20] <Perseus0> any clue how to get this returned?

[21:53:09] <bofh> Hi all! I have a mongodb and I need to export certain records to another database, what is the best way?

[21:58:13] <oky> bofh: there is literally a command called 'mongoexport'

[21:59:16] <bofh> I don't need everything, just records that match some search criteria

[21:59:19] <oky> i guess you want filtering

[21:59:50] <oky> use "--query", bofh with mongoexport or write your own driver, or do a google search for existing ones

[22:01:44] <kurushiyama> That.

Log file Viewer

Help | Karma | Search:

#mongodb logs for Friday the 27th of May, 2016