PMXBOT Log file Viewer

Help | Karma | Search:

#mongodb logs for Friday the 15th of January, 2016

(Back to #mongodb overview) (Back to channel listing) (Animate logs)
[00:26:41] <metame> jbrhbr: are you projecting out _id
[00:42:05] <jbrhbr> metame: no, just collection.count({state_enum: 1})
[00:43:48] <Boomtime> @jbrhbr: can you provide the explain output in a gist?
[00:43:58] <Boomtime> or pastebin etc.
[00:45:24] <jbrhbr> Boomtime: https://gist.github.com/jerryhebert/cbfedaf3f0f96fad55cd
[00:47:46] <jbrhbr> nothing fancy really, just slow. not sure if i'm missing something obvious but i thought this would be faster than 600ms
[00:49:03] <Boomtime> so did i.. and the explain output is odd
[00:49:19] <Boomtime> ""totalKeysExamined" : 4199257"
[00:49:37] <Boomtime> that's the entire collection presumably - it looped over every key in the index
[00:49:45] <jbrhbr> yeah
[00:49:55] <Boomtime> i expected it to range count the bucket entries.. weird
[00:50:01] <jbrhbr> .count() with no query is 4199256
[00:50:03] <Boomtime> but instead it walked
[00:50:11] <jbrhbr> maybe a bug?
[00:50:23] <Boomtime> certainly odd, i'm going to run some tests
[00:50:35] <Boomtime> does that index have only one entry?
[00:50:35] <jbrhbr> sweet, thanks :)
[00:50:56] <jbrhbr> https://gist.github.com/jerryhebert/3a92bd60f9e280a341e2
[00:50:59] <Boomtime> i.e is 1 the only value in the entire index?
[00:51:06] <jbrhbr> that's it
[00:51:22] <Boomtime> so every document (all 4+ million) have that field set to 1?
[00:51:45] <jbrhbr> yeah
[00:51:48] <jbrhbr> currently
[00:52:17] <Boomtime> since you already have this data set.. can you try adding 2 new documents, with values of 0 (zero) and 2
[00:52:26] <Boomtime> then re-run the count
[00:53:13] <jbrhbr> ok, one sec
[00:55:13] <jbrhbr> i have to bypass my validations for this which is clumsy for me
[00:59:51] <jbrhbr> Boomtime: https://gist.github.com/jerryhebert/e8f782c7a29f0463eda9
[01:00:20] <jbrhbr> (no difference)
[01:00:55] <jbrhbr> (that i can see...)
[01:37:59] <jbrhbr> Boomtime: find anything?
[02:55:01] <mocx> how can I return a mongoose promise for Model.populate()?
[03:01:17] <mocx> nevermind
[04:07:54] <Voyage> Why nosql dbs usually dont have consistency and durability?
[04:11:05] <Boomtime> well, i can't say about other nosqls, but mongodb has both of those
[04:17:41] <Voyage> Boomtime, so mongo has CAP fully? consistency, availability , partition tolerant? You only can be good at 2.
[15:05:32] <metame> Voyage: mongo is a CP db
[15:06:48] <metame> Voyage: it is eventually consistent
[15:32:27] <StephenLynx> CP db?
[15:35:19] <cheeser> C(A)P
[15:42:03] <metame> StephenLynx: Consistent & Partition Tolerant
[15:59:50] <metame> btw here's some (academic but short) reading on CAP: https://www.comp.nus.edu.sg/~gilbert/pubs/BrewersConjecture-SigAct.pdf
[16:01:43] <ThunderMuscle> Hi! Small question about c++ client library (legacy 1.1). I did a pretty small test with it and encounter a weird problem. I just insert a simple object with my program. It executes without any errors. when executing db.stats() in the db, it also tells me "objects": 1 (and consecutive execution of my program increments this) but doing db.test.test1.count() or db.test.test1.find() returns 0 resp. nothing. Do I have to set some
[16:01:43] <ThunderMuscle> specific options in my lib that it also can be seen by the 'normal' client?
[16:08:44] <StephenLynx> are you on the right db?
[16:15:56] <ThunderMuscle> Well - think so :) So This is the code: http://pastebin.com/eRw30ykz and this is my querying on the 'mongo' command line tool http://pastebin.com/LaRT42mf
[16:16:23] <ThunderMuscle> the query in the code prints each time one more
[16:16:34] <ThunderMuscle> where each time means each execution of my program ^
[16:18:02] <ThunderMuscle> http://pastebin.com/zpDaVyiV
[16:18:06] <ThunderMuscle> sorry - get get
[16:18:13] <ThunderMuscle> getConnection
[16:18:38] <StephenLynx> b.demo.xyz
[16:18:44] <StephenLynx> > db.demo.xyz.count()
[16:18:55] <StephenLynx> what is your collection name?
[16:19:03] <StephenLynx> use show collections
[16:19:47] <StephenLynx> usually you don't use dots in collection names.
[16:21:50] <ThunderMuscle> http://pastebin.com/f16vd6Ey
[16:22:03] <ThunderMuscle> so I've already 3 there :)
[16:22:31] <StephenLynx> use db.xyz.count()
[16:23:09] <ThunderMuscle> aahh damn it... sorry - obviously I'm new to mongodb :)
[16:23:16] <ThunderMuscle> it works, thanks a lot!
[16:23:31] <StephenLynx> v:
[16:23:32] <StephenLynx> np
[17:48:15] <dooderino> Hey, anyone here that can help me with my query?
[17:50:04] <m3t4lukas> I'm pretty tired. If it can wait till tomorrow I', down for it
[17:50:13] <m3t4lukas> *I'm
[17:52:39] <dooderino> Sure, though there might not be an adequate solution, just take a look here http://stackoverflow.com/questions/34811299/mongo-find-limit-each-match
[17:53:40] <cheeser> there's an answer there...
[17:57:36] <dooderino> yes? But it's not a very good solution performance wise if you haven't noticed, I'm wondering if there are any better ones
[18:02:30] <johnjohn101> hi mongo people
[18:03:15] <johnjohn101> is there a specific order for findOne()?
[18:05:19] <cheeser> johnjohn101: specific order of what?
[18:05:35] <cheeser> dooderino: aggregations are your only choice "bad" perf or not.
[18:05:50] <johnjohn101> well i have a collection. does it always return the 1st item that in inserted into the collection?
[18:05:59] <cheeser> you could always pre-aggregate and use $out to store the results in a collection then query that.
[18:20:36] <acedude> I have a pretty specific question: when this query "db.getCollection('checkins').find({"created_at": {"$gte": new ISODate("2013-10-01T00:00:00.000Z")}})" stopped being correct?
[18:21:03] <acedude> it works just fine on Mongo 2.4, but on 3.2 it returns nothing (until I add .toJSON() to the ISODate)
[18:21:29] <acedude> I don't know if I'm correct, but that might break PHP mongo library
[18:34:23] <acedude> looks like my issue might be driver-specific, but I'm not sure... I'm so desperate I'm downgrating to mongo 3.0, then 2.6
[18:49:46] <cheeser> what do your documents looks like?
[19:08:51] <acedude> @cheeser like that http://uploads.acedude.pl/2016-01/Robomongo_0.8.5_2016-01-15_20-02-50.png
[19:09:04] <acedude> everything's perfect @ 2.4
[19:09:14] <acedude> I'm now migrating to 3.0 to see if it will help
[19:14:23] <stickperson> i’m trying to use $sum in a projection in mongo 3.2.1, but i’m getting seeing an “unsupported projection” error
[19:17:23] <cheeser> pastebin your pipeline
[19:23:44] <stickperson> cheeser: http://pastebin.com/b1BegU5V
[19:28:19] <cheeser> $sum is an aggregation operator not a find operator
[19:32:39] <stickperson> cheeser: the docs say that $sum can be used in projections as of 3.2. i’m using db.myCollection.find(query, projection)
[19:34:31] <metame> stickperson: link to that in docs?
[19:34:52] <stickperson> metame: sure. here it is on the $sum page: https://docs.mongodb.org/manual/reference/operator/aggregation/sum/#grp._S_sum
[19:35:21] <cheeser> that's still an aggregation operator. you're not doing an aggregration. you're doing a find()
[19:35:32] <stickperson> here’s the release notes: https://docs.mongodb.org/manual/release-notes/3.2/#agg-accumulator-availability
[19:35:34] <metame> stickperson: ah ya $project is an aggregation step
[19:36:02] <stickperson> hm, a bit confused. doesn’t find look like find(query, projection) ?
[19:36:05] <metame> stickperson: different than projection in a query
[19:36:20] <stickperson> oh
[19:36:33] <metame> $project (aggregatiion) vs. projection in query
[19:36:45] <stickperson> oh shit
[19:36:51] <stickperson> that’s a bit confusing
[19:37:01] <metame> similar only in etymology :)
[19:37:29] <metame> well *only is a bit strong but definitely different things
[19:38:20] <metame> query projection is pretty light in operators/syntax https://docs.mongodb.org/manual/reference/operator/query/#projection-operators
[19:38:29] <jbrhbr> metame: did you have any more thoughts on my slow count(_query_) issue yesterday? Boomtime said he was testing something but never got back to me
[19:40:24] <metame> don't have time to test it, but make sure to project out _id
[19:40:35] <metame> jbrhbr: ^
[19:40:37] <jbrhbr> ok
[19:40:46] <jbrhbr> yea you asked me if i was or not but i wasn't sure if i should or not :)
[19:40:55] <metame> if I'm thinking about this correctly since count() is shorthand for find().count()...
[19:41:14] <metame> then projection is essential to leverage your index
[19:41:53] <stickperson> metame and cheeser thanks for the help
[19:42:15] <metame> stickperson: np
[19:42:20] <jbrhbr> metame: `c.count({state_enum: 1}, {_id: 0})` right? same behavior
[19:43:42] <metame> jbrhbr: don't remember if you have any other props, but also put state_enum: 1 in your projection
[19:44:00] <metame> by default a projection statement excludes all keys besides _id
[19:44:27] <metame> also if you haven't done an explain on your query check that
[19:47:16] <jbrhbr> i gisted the explain yesterday
[19:47:23] <jbrhbr> not sure how to explain a count with a projection though
[19:47:43] <jbrhbr> let's seeee
[19:48:42] <metame> just db.collection.explain(query,projection)
[19:50:11] <metame> sorry forgot the .count in there
[19:50:24] <metame> db.collection.explain().count(query,projection)
[19:50:30] <jbrhbr> ah cool
[19:50:45] <jbrhbr> i was using .runCommand(…) since .count(…).explain() doesn't work
[19:51:31] <jbrhbr> well, i can see the plan but not the exec stats
[19:52:02] <metame> https://docs.mongodb.org/manual/reference/method/db.collection.explain/#output
[19:52:32] <metame> there's how to get what you need :)
[19:52:39] <metame> well scroll up a bit
[19:53:55] <jbrhbr> thanks, i'm obviously pretty new to mongo
[19:55:03] <jbrhbr> this is the 3rd way that i've seen to explain stuff
[19:56:15] <metame> explain used to have a different syntax/api pre 3.0
[19:56:53] <jbrhbr> yea, i see the changed note at the top. but in the older version, i don't think that syntax supported .count() so i was using .runCommand() :p
[19:56:56] <jbrhbr> anyway, this way is much nicer
[19:57:07] <jbrhbr> and my .count is still 600ms :p c.explain("executionStats").count({ state_enum: 1}, {state_enum: 1, _id: 0})
[19:57:31] <jbrhbr> and my state_enum index is being used according to the planner
[19:57:41] <jbrhbr> i don't know...
[19:57:44] <cheeser> why are you doing a projection on a count?
[19:58:02] <cheeser> why does count() even support projections?
[19:58:03] <jbrhbr> metame suggested it might improve performance
[19:58:09] <cheeser> i don't know how...
[19:58:16] <jbrhbr> me neither, i'm throwing darts
[19:58:20] <jbrhbr> it's 600ms no matter how i slice it
[19:58:36] <metame> wasn't sure if it'd help since .count is like .find().count() and he's trying to use an index
[19:58:49] <cheeser> is state_enum indexed?
[19:58:58] <metame> but didn't know exactly how .count is executed
[19:59:05] <jbrhbr> cheeser: yes
[19:59:15] <cheeser> and how big is your collection?
[19:59:21] <jbrhbr> https://gist.github.com/jerryhebert/e8c87557b547e9d2e38c
[19:59:27] <metame> jbrhbr: dinner time. afk for a while
[19:59:30] <jbrhbr> 4.2mil
[20:00:15] <cheeser> is this not in the shell?
[20:00:27] <jbrhbr> the question i originally came here with was "is this normal performance?"
[20:00:36] <jbrhbr> the gist i just pasted is mongo shell
[20:01:08] <cheeser> "c." is not a shell thing.
[20:01:20] <jbrhbr> it's just my collection, from db.getCollection()
[20:02:55] <cheeser> ah
[20:03:39] <cheeser> i have a collection of just over 9M log records. count() is almost instantaneous.
[20:04:21] <jbrhbr> but is that count with a query?
[20:04:26] <jbrhbr> that's the issue here it seems
[20:04:42] <jbrhbr> plain count() is very fast, but with a query (on an indexed field), it's 600ms
[20:05:48] <cheeser> ah, yes. no, with a query it's slower, you're right.
[20:07:24] <jbrhbr> ok
[20:09:49] <jbrhbr> heh, it's 0ms if i query with state_enum:0 instead of 1 (all of my documents have it set to 1)
[20:10:08] <jbrhbr> it just seems surprising that this index doesn't know how many things it has in each path
[20:11:01] <jbrhbr> like it seems to be doing a full scan of the data just to get the count where this value is 1
[20:11:36] <cheeser> just confirmed that it does indeed have to do an index scan to generate the count
[20:12:07] <jbrhbr> thank you
[20:12:13] <cheeser> np
[20:12:36] <jbrhbr> does it have to for architectural reasons or do you just mean it does so currently?
[20:25:41] <saml> "$date" : 1442761653000 "$date" : 1391040000000 "$date" : 1376870400000 so mongoimport chokes on one of these
[20:25:58] <saml> 2016-01-15T05:59:23.782-0500 Failed: error getting extended BSON for document #9: invalid type for $date field
[20:26:04] <saml> why is that invalid type?
[20:27:13] <cheeser> those are numbers not dates
[20:27:30] <saml> oh https://docs.mongodb.org/manual/reference/mongodb-extended-json/ this changed
[20:27:39] <saml> it used to be {"$date: number}
[20:27:59] <saml> so after mongod upgrade in september, mongoimport has been failing hahahaha
[20:28:07] <cheeser> nice
[20:36:37] <saml> it's weird.. the other mongoimport accepts {"$date" : 1442761653000} ..
[20:37:00] <saml> mongoimport on same server. one to db1.collection another to db2.collection2
[20:37:26] <saml> on the same localhost mongod
[20:38:52] <saml> https://jira.mongodb.org/browse/TOOLS-874
[20:39:22] <saml> so it's not format issue. maybe the number is too big
[21:20:23] <ChALkeR> Anyone using mongoose here? You should update.
[21:25:39] <saml> The MongoDB JSON parser currently does not support loading ISO-8601 strings representing dates prior to the Unix epoch. When formatting pre-epoch dates and dates past what your system’s time_t type can hold, the following format is used
[21:25:58] <StephenLynx> ChALkeR you should stop using mongoose./
[21:26:10] <saml> {"date": {"$date": {"$numberLong": "68400000"}}} this works. {"date": {$date: 68400000}} this does not work cause number is pre-epoch
[22:27:31] <digitalvectorz> Ello all. Using mongodb c# 2.2.2 - has the MongoDB.Driver.Builders namespace been relocated or replaced with a more preferred practice?
[22:36:49] <ChALkeR> StephenLynx: Who told you that I'm using it?
[22:36:55] <ChALkeR> I don't.
[22:59:08] <Boomtime> @digitalvectorz: it's still there AFAIK: http://api.mongodb.org/csharp/current/html/N_MongoDB_Driver_Builders.htm