PMXBOT Log file Viewer

Help | Karma | Search:

#mongodb logs for Sunday the 13th of December, 2015

(Back to #mongodb overview) (Back to channel listing) (Animate logs)
[12:50:50] <StephenLynx> GothAlice now that WT is the default engine, you think those issues you seen were fixed? are you using WT?
[13:14:44] <C0nundrum> hello
[13:20:36] <C0nundrum> in my program i want to check if an item exists using find and then add or update it using save(). How would i make it so that multiple processes don't end up adding entries between find and save ?
[13:23:31] <StephenLynx> >save
[13:23:39] <StephenLynx> where is that from?
[13:24:17] <C0nundrum> mongoose
[13:25:27] <StephenLynx> I strongly suggest you don't use mongoose.
[13:25:39] <StephenLynx> it is extremely slow and handles ids and dates poorly.
[13:26:46] <C0nundrum> What would you recommend ?
[13:27:01] <StephenLynx> the native driver.
[13:27:07] <StephenLynx> npm install mongodb
[13:27:58] <StephenLynx> great documentation and performance, intuitive
[13:29:04] <C0nundrum> ugh :<
[13:31:50] <C0nundrum> Well a unrelated question i have is i want to write a ecomerence site and i was going to use mongodb. Hoever after hours of reading, everywhere points to the downside of mongodb not supporting ACID. And that hacking toghether a solution at the application level makes projects unmaintinable in the longrun ?
[13:33:20] <StephenLynx> hm
[13:33:54] <StephenLynx> if ACID is vital for your software, the effort of implementing these features might not pay off, depending on what you gain from using mongo.
[13:34:19] <StephenLynx> however, it is possible to implement these without making the software unmaintainable.
[13:34:23] <StephenLynx> for example
[13:34:31] <StephenLynx> findAndModify
[13:34:43] <StephenLynx> or incremental operators
[13:35:18] <StephenLynx> there are tools that helps to keep atomicity without having actual ACID
[13:35:48] <StephenLynx> or bulkwrite, which enables you to rollback changes in a single operation.
[13:36:07] <StephenLynx> if you have to rollback several documents
[13:37:01] <StephenLynx> IMO, unless you REALLY know what you would gain from mongo and you have enough experience with the platform you are using (node.js, right?) I wouldn't use mongo.
[13:37:05] <StephenLynx> in your case.
[13:37:27] <StephenLynx> otherwise you might end up writing spaghetti code trying to compensate the missing features you need.
[13:39:28] <C0nundrum> what would you recommend then for a a ecomernece site that doesn't really have bigData requirements but supports easy horizontal scaling. Thinking of deploying it on heroku
[13:40:21] <StephenLynx> mariadb
[13:40:45] <StephenLynx> the horizontal scaling might not be as good as mongo's
[13:40:52] <StephenLynx> but I don't think it would be too much of an issue
[13:44:01] <StephenLynx> that kind of stuff doesn't cause legacy nor technical debt.
[13:44:06] <C0nundrum> Hm i will look into it. So about findandmodify the problem with that is that i don't want to waste resource populating and object before calling it is the a way atomize the findandmodify ?
[13:44:28] <StephenLynx> you can use upsert too
[13:45:40] <StephenLynx> in a single operation the document is created and has a field atomically modified.
[13:48:37] <C0nundrum> So this wouldb't be possible right ? t1: check if exist t2: check if exist t1:insert document t1: modify document
[13:50:25] <StephenLynx> t = thread?
[13:50:51] <C0nundrum> thread/mongodb query
[13:50:54] <StephenLynx> that would be possible, but unnecessary.
[13:51:02] <StephenLynx> if you need something to exist, just use upsert.
[13:51:20] <StephenLynx> not only, using multiple operations cause race conditions.
[13:51:45] <StephenLynx> both t1 and t2 could try and create the document if they have to check for it before.
[13:52:28] <StephenLynx> I am pretty sure mongo wouldn't allow both to insert in an upsert.
[13:54:56] <C0nundrum> Well yea i mean i want to avoid populating and object before call upsert as its exspensive.
[13:55:05] <C0nundrum> It that possible ?
[13:55:52] <StephenLynx> yes, that is what upsert is for.
[13:56:02] <StephenLynx> it populates for you if it doesnt exist.
[13:58:42] <C0nundrum> i mean on the appliaction level if i have 5 queries that run at the same time to check if an entry exists need only 1 to pass. At that point it will then do some intenstive tasks before it know what to set in the database. the calling findmondify twice be the correct option that way only on will pass the upsert ?
[13:59:04] <C0nundrum> would calling*
[13:59:37] <StephenLynx> what
[13:59:46] <StephenLynx> just call a find and modify once with upsert.
[14:09:12] <C0nundrum> StephenLynx ok maybe this puesduo code best explains what i mean
[14:10:27] <C0nundrum> http://pastie.org/private/0fewe70u1a2fedm804gxjw
[14:10:49] <C0nundrum> whoops
[14:11:02] <C0nundrum> http://pastie.org/private/v9no0rmqm6do1749dza
[14:11:46] <C0nundrum> in order to use upsert would line 9-14 have to be before line 5 ?
[14:11:52] <C0nundrum> wouldn't*
[14:12:20] <StephenLynx> no
[14:12:27] <StephenLynx> you dont need anything
[14:12:30] <StephenLynx> just use it.
[14:13:24] <C0nundrum> so findandmodify doesn't have to set any fields right ? and only 1 of those addObject calls will reach line 9 ??
[14:15:25] <StephenLynx> no.
[14:15:44] <StephenLynx> findandmodify doesnt have to set any fields.
[14:16:05] <StephenLynx> keep in mind that changing is not the same as setting
[14:16:14] <StephenLynx> $set, $inc, $push
[14:18:33] <C0nundrum> hmm, i see
[15:06:12] <C0nundrum> StephenLynx, do you know and good mongo db Admin Uis ? I like Mongo Mangement Studio atm but would be nice to have one with built in anylytics thats user friendly.
[15:06:39] <StephenLynx> no, I just use the terminal.
[16:16:02] <C0nundrum> how would i query a dubdocument
[16:16:19] <C0nundrum> WHen i do query it still returns the parent document
[16:16:32] <StephenLynx> yes, you have to fetch the document.
[16:16:37] <StephenLynx> and from it read the sub-document.
[16:16:40] <StephenLynx> unless
[16:16:51] <StephenLynx> you use aggregation and use $group
[16:17:27] <StephenLynx> and on it you make it give you documents with the data of the sub-document
[16:17:39] <StephenLynx> but still you are not getting the sub-documents directly
[16:21:10] <StephenLynx> R9K pushed.
[16:21:21] <StephenLynx> 1.4 scheduled.
[16:23:53] <StephenLynx> oops
[16:23:55] <StephenLynx> wrong channel
[16:23:55] <StephenLynx> v:
[17:04:19] <Rashad> Anyone using mongo express?
[17:04:46] <Rashad> I can't seem to be able to modify data through it.
[17:06:51] <StephenLynx> never heard about it.
[17:08:26] <Rashad> This is my conf file: http://hastebin.com/zezosageze.js
[17:10:59] <Rashad> I can login using the credentials but whenever I try to modify or delete something, it returns: "Document not found!".
[17:15:21] <Rashad> OK I figured it out. My documens' ids are numbers.
[17:15:41] <Rashad> I should start them with an underscore.
[17:15:58] <StephenLynx> _ids are not numbers.
[17:16:29] <Rashad> I understand.
[17:16:52] <Rashad> But for example I have the _id set to "123". Mongo express can't handle that.
[17:17:20] <StephenLynx> you manually set the _id?
[17:18:21] <Rashad> Yeah.
[17:18:23] <Rashad> Is that wrong?
[17:18:48] <Rashad> My _id's are basically the facebook user id's of my users.
[17:18:54] <Rashad> So they are unique.
[17:19:10] <Rashad> And it's easier to query the db that way.
[17:19:32] <StephenLynx> I wouldn't use the _id field though.
[17:19:39] <Rashad> Why not?
[17:19:45] <StephenLynx> because _id has some special properties.
[17:19:55] <Rashad> Like?
[17:20:01] <StephenLynx> projection
[17:20:13] <StephenLynx> it is projected by default
[17:20:13] <Rashad> OK so how am I affecting that?
[17:20:22] <Rashad> Oh that's OK.
[17:20:32] <Rashad> So I get an extra _id field? That's OK.
[17:20:34] <StephenLynx> just saying, it got more going on than a regular field.
[17:20:49] <StephenLynx> which might give you some headache.
[17:20:56] <Rashad> Sure but its meaning is a unique value that distinguishes the document from the others.
[17:21:00] <StephenLynx> what I do is define my own unique indexes.
[17:21:05] <Rashad> I am not violating this.
[17:21:14] <StephenLynx> no, is not exactly wrong.
[17:21:20] <StephenLynx> it just might cause you inconvenience in the future.
[17:21:29] <Rashad> Like?
[17:21:42] <StephenLynx> let me look for it
[17:21:46] <Rashad> Sure.
[17:22:38] <StephenLynx> btw, did you tell mongo to not automatically create the _id field?
[17:24:46] <Rashad> No...
[17:25:29] <Rashad> Why should I do that?
[17:25:53] <StephenLynx> you are setting the _id on creation then?
[17:26:07] <Rashad> Yes I am.
[17:26:26] <StephenLynx> hm, in that case I don't think mongo treats the field differently.
[17:26:27] <StephenLynx> nvm
[17:27:02] <StephenLynx> because it seems that if you provide it with something that is not an ObjectID, it doesn't do anything of the special behaviours
[17:30:05] <Rashad> Sounds good.
[17:30:15] <Rashad> Seems they expected my use case. No?
[17:31:27] <StephenLynx> nothing especial either way.
[17:36:46] <StephenLynx> but most people rather name their own unique index.
[17:37:32] <C0nundrum> StephenLynx my aggreation was working up until {$sort : { "$episodes.date" : -1 }}
[17:37:51] <C0nundrum> am i suppose to use what i rename it to in project ?
[17:38:03] <StephenLynx> what
[17:38:26] <C0nundrum> if i use that object as part of a aggreation query
[17:38:53] <C0nundrum> {"errormessage":"Mongo Error: exception: Error: command failed: {\n\t\"errmsg\" : \"exception: FieldPath field names may not start with '$'.\",\n\t\"code\" : 16410,\n\t\"ok\" : 0\n} : aggregate failed\n at Error (<anonymous>)\n at doassert (src/mongo/shell/assert.js:11:14)\n at Function.assert.commandWorked
[17:38:56] <StephenLynx> remove that $
[17:39:10] <C0nundrum> i'm selecting the subdocument
[17:39:12] <C0nundrum> field
[17:39:15] <StephenLynx> i know
[17:39:19] <StephenLynx> remove the $
[17:40:56] <C0nundrum> ah i see it the correction didn't work because i was still using project
[17:47:10] <GothAlice> StephenLynx: I'm evaluating 3.2 + WT today. So far things are looking promising; restore of a stock data template took 1/10th as long as expected.
[17:47:26] <StephenLynx> nice.
[17:47:43] <GothAlice> Now I'm about to rebuild the analytics pre-aggregate dataset.
[17:47:43] <StephenLynx> btw, do I have to change something to start using WT when upgrading from 3.0?
[17:48:28] <GothAlice> And mongod is now consuming 110% of CPU resources.
[17:48:35] <StephenLynx> kek
[17:48:39] <GothAlice> :/ And the pre-aggregate recalculation is going much slower than expected.
[17:48:54] <GothAlice> ooh, 280% CPU.
[17:49:57] <GothAlice> This is with four simultaneous processes working on per-client child tasks.
[17:51:07] <GothAlice> StephenLynx: A ha. Record lookup by _id is taking half a second. I need to set my slowms higher. :|
[17:51:21] <StephenLynx> v:
[17:52:02] <GothAlice> At least memory usage is fairly constant. No more "consume it all!" policy. ;P
[17:52:28] <StephenLynx> what about upgrading from 3.0? do I have to change anything if I never told which engine to use?
[17:53:01] <GothAlice> Existing collections won't be "automatically converted" or anything, AFIK.
[17:53:14] <GothAlice> I.e. to test this I dropped my test dataset then re-imported it.
[17:53:20] <StephenLynx> ok
[17:54:01] <StephenLynx> a simple mongodump, dropdatabase and mongorestore will work then?
[17:54:35] <GothAlice> Looks like.
[17:54:44] <GothAlice> 17% done reprocessing…
[18:03:32] <C0nundrum> what query would i use to check the difference between now and the date stored ?
[18:05:11] <StephenLynx> not sure that is possible.
[18:05:57] <StephenLynx> you can use arithmetic operators though
[18:07:32] <GothAlice> The question doesn't really make sense.
[18:08:05] <GothAlice> If you have data in hand (i.e. a variable in your code), and data in MongoDB, you'll have to use your language's available tools to perform the calculation of difference. (I.e. difflib in Python.)
[18:08:13] <GothAlice> MongoDB can't really help you with that.
[18:09:26] <GothAlice> On the other hand, there are ways to structure how you store your data, and how you store updates to your data, that preserve the history, but that's still you, not MongoDB, helping you out.
[18:09:43] <C0nundrum> Also looking at the docs select the difference using db.sales.aggregate( [ { $project: { item: 1, dateDifference: { $subtract: [ new Date(), "$date" ] } } } ] )
[18:09:59] <GothAlice> My gods that'd be tedious.
[18:10:12] <C0nundrum> but how do i return the whole document without having to manually select every field manualy in project ?
[18:10:12] <GothAlice> Also… aggregate is typically compared against many values. It still doesn't make sense.
[18:10:14] <GothAlice> http://stackoverflow.com/a/3510277/211827
[18:11:31] <GothAlice> C0nundrum: https://docs.mongodb.org/manual/reference/operator/aggregation/project/#considerations
[18:11:48] <GothAlice> Ah, scroll up a bit. Wrong anchor, sorry.
[18:12:57] <GothAlice> If you mark any fields for inclusion (1), it implies that non-marked fields aren't included. Simply remove your "item: 1" part of the projection to get all fields back, plus the one added as dateDifference. But this is a serious case of XY problem solving.
[18:13:04] <C0nundrum> explicitly define them is mandatory :(
[18:13:12] <GothAlice> http://xyproblem.info
[18:13:47] <C0nundrum> I tried that goth if remove it it says there must be at least 1 field marked
[18:15:00] <C0nundrum> o right any* :/
[18:15:04] <GothAlice> That feeling you're getting–it's telling you this is the wrong solution to whatever the problem is.
[18:15:19] <GothAlice> But you keep asking about specific solutions instead of the problem. :|
[18:17:38] <C0nundrum> its not that it isn't working for what i'm doing just trying to understand the api
[18:34:57] <GothAlice> StephenLynx: Initial stamp of approval given.
[18:35:24] <StephenLynx> k
[18:35:26] <GothAlice> Though I stressed the lights out of it, it didn't turn into grey goo like it did in prior tests.
[18:35:42] <StephenLynx> will move to it when I start by new beta on 19th
[18:35:48] <StephenLynx> my new beta*
[18:42:57] <GothAlice> StephenLynx: https://soundcloud.com/gothalice/metal-crusher?in=gothalice/sets/undertale ;) Now to explore the data storage differences. I'll be setting up a new production cluster next week, and I think I'll give WT a shot on it.
[19:15:51] <StephenLynx> GothAlice you know any condition where mongo would change the _id of a document for no good reason?
[19:15:59] <StephenLynx> I am 100% certain I a not editing the document _id
[19:16:04] <GothAlice> I know of none. Once set, AFIK internally it's immutable.
[19:16:19] <StephenLynx> well, it is changing.
[19:16:27] <GothAlice> (Only outside $set or other operations should affect it.)
[19:16:40] <StephenLynx> hold on
[19:18:27] <StephenLynx> ok, nvm v: the id is not changing
[20:10:14] <m3t4lukas> hey guys, I just installed kernel 4.2.0-19 on my MSI GE60 Notebook. Now wifi is not working anymore :/ rfkill shows only my bluetooth adapter and lshw shows an unclaimed Intel Wireless 3160 controller. Any ideas what my options are?
[20:10:37] <StephenLynx> wrong channel
[20:10:45] <m3t4lukas> I found something about some iwlwifi kernel oooops, sry :D
[20:10:52] <StephenLynx> :v
[20:11:42] <m3t4lukas> never mind, touchpad is weirdly misbehaving, too. Must have activated the mongodb channel