pmxbot IRC Log Viewer

[04:14:45] <antz> Hello

[04:15:17] <antz> I have an issue in my server, where the reads have become very slow

[04:15:20] <antz> "desc" : "conn26",

[04:15:20] <antz> "threadId" : "0x263c3c0",

[04:15:20] <antz> "connectionId" : 26,

[04:15:20] <antz> "opid" : 113858,

[04:15:21] <antz> "active" : true,

[04:15:21] <antz> "secs_running" : 1149,

[04:15:22] <antz> "microsecs_running" : NumberLong(1149081326),

[04:15:26] <antz> "op" : "query",

[04:15:28] <antz> "ns" : "215.$cmd",

[04:15:30] <antz> "query" : {

[04:15:32] <antz> "count" : "unbxdgetitu",

[04:15:34] <antz> "query" : {

[04:15:36] <antz> "doctype" : "category",

[04:15:38] <antz> "autosuggest" : "Mobile Accessories"

[04:15:40] <antz> },

[04:15:42] <antz> "fields" : {

[04:15:46] <antz> }

[04:15:48] <antz> },

[04:15:50] <antz> "client" : "127.0.0.1:48006",

[04:15:53] <antz> "killPending" : true,

[04:15:56] <antz> "numYields" : 0,

[04:15:58] <antz> "locks" : {

[04:16:00] <antz> "Global" : "r",

[04:16:02] <antz> "Database" : "r",

[04:16:04] <antz> "Collection" : "R"

[04:16:06] <antz> },

[04:16:08] <antz> "waitingForLock" : true,

[04:16:10] <antz> "lockStats" : {

[04:16:12] <antz> "Global" : {

[04:16:14] <antz> "acquireCount" : {

[04:16:16] <antz> "r" : NumberLong(2)

[04:16:18] <antz> }

[04:16:20] <antz> },

[04:16:22] <antz> "MMAPV1Journal" : {

[04:16:26] <antz> "acquireCount" : {

[04:16:28] <antz> "r" : NumberLong(1)

[04:16:30] <antz> }

[04:16:32] <antz> },

[04:16:34] <antz> "Database" : {

[04:16:36] <antz> "acquireCount" : {

[04:16:38] <antz> "r" : NumberLong(1)

[04:16:40] <antz> }

[04:16:42] <antz> },

[04:16:44] <antz> "Collection" : {

[04:16:46] <antz> "acquireCount" : {

[04:16:48] <antz> "R" : NumberLong(1)

[04:16:50] <antz> },

[04:16:52] <antz> "acquireWaitCount" : {

[04:16:56] <antz> "R" : NumberLong(1)

[04:16:58] <antz> },

[04:17:00] <antz> "timeAcquiringMicros" : {

[04:17:02] <antz> "R" : NumberLong("1318934351710")

[04:17:04] <antz> }

[04:17:06] <antz> }

[04:17:08] <antz> }

[04:17:10] <antz> I see Collection.timeAcquiringMicros is very high

[04:17:12] <antz> what can be the reason behind it

[04:37:28] <tejasmanohar> antz: holy shit man

[04:37:29] <tejasmanohar> use gist :P

[04:37:32] <tejasmanohar> !gist

[04:37:34] <tejasmanohar> fk

[04:37:38] <tejasmanohar> https://gist.github.com/

[04:41:05] <antz> https://gist.github.com/anantheshadiga/69b64a168d0ca9795a7c

[04:41:13] <antz> :P

[08:30:14] <Frenchiie> so if i want to query documents in a collection by date is there a specific method from the API or do i have to stick a time/date object in there for every documents

[08:30:25] <Frenchiie> well date down to the time

[08:30:56] <Frenchiie> like is there an api that lets me grab the last 10 inserted documents of a collection?

[08:31:07] <Frenchiie> api method

[08:31:25] <Frenchiie> pymongo specifically i suppose, if that makes a difference.

[08:32:42] <kali> Frenchiie: if you use standard ObjectId as _id, they contain a timestamp

[08:33:14] <Frenchiie> aha!

[08:33:30] <kali> Frenchiie: so objectId naturally sort themselves roughly by creation order (there can be permutation for objects created in the same second)

[08:34:36] <Frenchiie> hm looks like i'm not using any objectid/_id

[08:34:49] <Frenchiie> i guess i should if it i'll give me that sweet sweet timestamp of justice.

[08:36:13] <kali> Frenchiie: if you're not setting _id, mongodb will inject an ObjectId there

[08:36:44] <Frenchiie> oh cool

[08:36:55] <Frenchiie> so it's already done for me

[08:37:10] <kali> probably

[08:39:30] <diegoaguilar> For an schoolar game app, I gotta store the points users get through time

[08:41:13] <diegoaguilar> Points come from a particular subject and indeed, part of a block/topic

[08:41:40] <diegoaguilar> Id like to request points a user has got at anytime, and somehow also do a query or aggregations to get tops

[08:42:05] <diegoaguilar> would a {userId: 1, block: 1, theme: 1} index be beneful?

[09:49:55] <repxxl> Hello, i was using mongo now for more days, i don't know what happened but im no more able to connect to mongo i get " Fri Oct 9 04:44:57.086 Error: couldn't connect to server 127.0.0.1:27017 at src/mongo/shell/mongo.js:145"

[09:50:08] <repxxl> when i write "mongo" in shell to connect

[09:50:44] <Zelest> check if you have a mongod.log in your datadir?

[09:50:56] <Zelest> (or wherever you/your dist placed it)

[09:56:56] <repxxl> Zelest http://pastebin.com/qaU5Yxsg

[09:58:19] <Zelest> Yeah, it's quite obvious if you actually read it.

[09:59:44] <repxxl> Zelest ERROR: dbpath (/data/db/) does not exist. ?

[10:00:21] <repxxl> Zelest created that directory, and made mongo use it mongod --dbpath /data/db/ and restarted mongodb and still same result

[10:00:51] <Zelest> what user is mongod running as?

[10:01:00] <Zelest> does it has permission to write to /data/db ?

[10:28:25] <repxxl> i tryed to remove mongo completely from debian with purge

[10:28:29] <repxxl> and installed new

[10:28:36] <repxxl> the problem is still there.

[14:18:56] <deinspanjer> Hi folks, I'm wondering if I've missed something simple that is causing what should be an indexed query to take a very long time. Running Mongo 3.0.6 on OSX with a 46 GB database. Here is the explain for the query: https://gist.github.com/deinspanjer/383aa672f32b96fcb506

[14:20:18] <deinspanjer> Argh!! Okay, new question.. why *was* it taking forever? I just ran it again after getting my IRC issues settled and now it returns instantly. Before I waited at least 15 minutes for it to complete.

[14:37:04] <StephenLynx> maybe it was building the index?

[14:39:18] <deinspanjer> Hmm.. Maybe.

[15:28:47] <brianboyko> Hello. I have users... users have an array of deck _ids. If I want to get a list of all a user's decks, do I have to do a Deck.findOne for every Deck._id, or is there like a "Get all these deck records" . I'm using Mongoose.

[15:30:33] <StephenLynx> I suggest you don't use mongoose, it is about 6x times slower than the native driver and doesn't handle types right.

[15:30:48] <StephenLynx> you want to get the decks which the user's contains?

[15:31:07] <StephenLynx> you would have to get the users to retrieve said ids and then query the decks collection to get the actual document.

[15:31:11] <StephenLynx> you can't join on mongo.

[15:35:07] <StephenLynx> however

[15:35:26] <StephenLynx> if each deck can only be owned by a single person, you could just have the user's on the deck

[15:35:36] <StephenLynx> and just get all decks that are owned by an user.

[15:36:53] <deinspanjer> I'm working with a 46GB database for demo and learning purposes on a mac with 16 GB of memory. Am I okay working with the default storage engine or should I be looking at using something different?

[15:38:14] <StephenLynx> I would stick with the default until you need something the default doesn't provide.

[15:38:32] <StephenLynx> it doesn't matter for development purposes anyway, afaik.

[15:39:21] <cheeser> though 3.2 changes that default, fwiw

[15:39:56] <deinspanjer> Well, this is demo rather than development. I'm trying to get a few queries and agg pipelines set up so that I can run them and record the results for a presentation.

[15:40:24] <deinspanjer> But currently, running with the vanilla defaults, everything is taking 10 to 20 minutes to execute..

[15:40:55] <deinspanjer> Unless I immediately execute the same query again in which case it returns what I assume is cached results..

[15:42:12] <StephenLynx> hm

[15:42:44] <StephenLynx> it could be either bad code or you need some index somewhere. unless you diagnose what is taking so long, you can't tell for sure what needs to be done.

[15:43:20] <deinspanjer> At the moment, I'm not dealing with any app code, just raw queries in the shell.

[15:43:37] <deinspanjer> I'm reading the FAQ pages right now and checking sizes on my indexes..

[15:44:01] <deinspanjer> I ran explain on some find() commands earlier. I also need to see if something similar is possible for agg pipelines

[15:45:51] <deinspanjer> totalIndexSize() reports that my indices are currently just under 2 GB, so that sounds okay according to the FAQ..

[15:47:21] <deinspanjer> But if I start the server clean and then try an agg pipeline, mongod's memory usage doesn't seem to climb significantly..

[15:47:28] <deinspanjer> Right now it is still sitting at just 30 MB.

[15:48:15] <deinspanjer> Oh wait, maybe I'm reading the wrong thing in Activity Monitor.

[15:48:42] <deinspanjer> On the actmon details for the mongodb proc, it says Real Memory size 11 GB, so that sounds more like what I'd expect

[15:52:46] <deinspanjer> hmm.. okay, I read how to do an explain on an agg, and it looks like it isn't able to take advantage of one of the indices..

[15:53:08] <deinspanjer> Not surprising since I'm projecting on that field first to get a substr of it.

[15:53:14] <StephenLynx> you REALLY need aggregation?

[15:53:25] <StephenLynx> very few things require an aggregation.

[15:53:47] <deinspanjer> I was just trying to do something simple like viewing how many documents for each date there were.

[15:54:48] <StephenLynx> try a regular find.

[15:55:11] <deinspanjer> But in the long run, I can probably skip agg pipeline and just do the aggregation in the app instead.

[16:00:56] <deinspanjer> It does seem a bit inefficient to have to use find to return 16m documents just to turn around and group them by day and count them in the application. A lot of data over the wire

[16:01:50] <StephenLynx> so you are using a $group?

[16:02:28] <deinspanjer> In the agg pipeline I was, yes..

[16:02:44] <StephenLynx> you should have told me. it changes a lot.

[16:02:55] <StephenLynx> since gruping without aggregation is not as simple.

[16:03:00] <StephenLynx> it is possible, though.

[16:03:03] <StephenLynx> http://docs.mongodb.org/manual/reference/method/db.collection.group/

[16:03:18] <StephenLynx> I wouldn't have suggested a find if I knew that.

[16:03:43] <deinspanjer> No worries. I thank you very much for the feedback.

[16:03:46] <StephenLynx> you have 2 choices:

[16:03:51] <StephenLynx> use the .group function

[16:03:59] <StephenLynx> pre-aggregate data as it is inserted.

[16:04:18] <deinspanjer> I was just reading group(). Wondering what the differences are between group(), mapReduce(), and aggregation(). They all seem to be solving pretty similar problems.

[16:05:57] <deinspanjer> Looks like group has some limitations that separate it, I'm assuming that within those limitations, it can typically be faster than the other two?

[16:06:19] <deinspanjer> Namely, less that 20k groups and can't operate on sharded clusters

[16:06:57] <StephenLynx> no idea, I have only used aggregate to group so far.

[16:19:08] <StephenLynx> v:

[16:19:11] <StephenLynx> yeah, you can

[16:19:28] <ducktape> awesome! :-)

[16:20:54] <ducktape> do these two function call mongo::socket::_send() and mongo::socket::_recv() invoke the actual data storing/fetching?

[16:21:35] <ducktape> or do these only reflect the actual socket-only portion

[16:31:47] <shlant> hi all. I have a general practice question: It's good to have all envs (dev/qa/prod) the same, so if we have a replica set in prod, then we should have it in dev/qa. But, would it make sense to do a replica set locally as well on developer workstations? or is that overkill? would a replica set on localhost even be possible?

[16:57:10] <ducktape> shlant: from my experience its really useful if you have at the very least QA as close to PROD as possible

[16:57:47] <ducktape> even if those environments are not physically identical, e.g. in QA you may have all the shards/replica on a smaller number of physical servers

[16:58:30] <ducktape> typically dev need not have all the characteristics like prod but QA/Testing should for sure

[17:01:26] <shlant> yea I have dev/qa using one replica set and prod using another. A dev was wondering why we don't have a replica on local, and I said it would be more complicated than it's worth (would only be useful for debugging very few edge cases)

[17:09:37] <deathanchor> shlant: local is special

[17:10:19] <deathanchor> http://docs.mongodb.org/master/reference/local-database/

[17:19:48] <deinspanjer> Is there a way I can make the mongo shell be forced to use strict JSON input format so I can compare queries from my app to queries in the shell?

[17:20:20] <shlant> deathanchor: I'm not referring to the mongo specific 'local' db, but 'local' as in 'developer workstation/laptop'

[17:33:28] <stig801> hello, i have a question about adding a replica.

[17:33:50] <stig801> I have a 2.5TB DB, which i'm trying to get a new replica for... old replica was running out of disk, so I recreated disk, but the time for syncing the old disk to the new disk was beyond the oplog time... so i deleted the old data, and just started mongo with the new disk, which started an inital sync

[17:34:05] <stig801> does the initial sync have to complete within the oplog window too?

[17:40:03] <stig801> by initial sync... i'm talking for a new replica, does its initial sync have to complete within the space/time permitted by the oplog size?

[18:59:14] <CmdrKilljoy> Hi guys… I’ve been trying to wrap my head around mongodb for a bit and I’m kind of confused with all the replica set + shards stuff. I was hoping someone could help clear it up for me?

[19:04:45] <CmdrKilljoy> does anyone have experience with replica sets and sharding?

[19:08:20] <saml> no

[19:08:42] <saml> doesn't mongodb automatically replicate and shard?

[19:08:48] <saml> i thought that's the point

[19:09:05] <CmdrKilljoy> you have to configure it.

[19:09:09] <joshua> No it doesn't automatically do it.

[19:09:11] <saml> meh

[19:09:42] <CmdrKilljoy> the blogs I’ve tried following are pretty confusing.

[19:09:51] <saml> and i bet documentation around those configuration isn't that good with lots of "depends on your situation"

[19:09:52] <joshua> You have to add config servers, pick a shard key based on your data. Enable sharding on your database, add mongos server and point it to your cluster

[19:10:12] <CmdrKilljoy> plus there’s lots to keep track of everything — config servers, mongod instances, mongos instances...

[19:10:36] <saml> why not put good default on those for mongodb 4.0?

[19:10:36] <StephenLynx> I would suggest learning the basics before tackling advanced topics.

[19:10:39] <joshua> Mongo isn't going to know how you plan on using your data

[19:11:04] <joshua> yeah start simple with a single server, then replica set

[19:11:40] <saml> with automatic rs.conf and on to mongos automagically in the cloud

[19:11:40] <CmdrKilljoy> I’ve managed to get a replica set going… I just get confused when it comes to shards, etc.

[19:11:48] <saml> they only sell those as paid service

[19:12:16] <CmdrKilljoy> -sigh- I realize everyone expects these questions to be coming from a noob, but that’s not what I am

[19:12:32] <joshua> Well the docs are pretty good http://docs.mongodb.org/manual/sharding/

[19:12:34] <saml> CmdrKilljoy, just get managed cloud service https://www.mongodb.com/cloud/

[19:13:26] <saml> we spend full time building the service. no matter how well you know mongodb, our service would outperform. just pay us.

[19:14:17] <CmdrKilljoy> saml, as much as I don’t want to do that (I’m a tinkerer, I want to know how everything works), the price isn’t bad and I might go that route.

[19:15:39] <saml> well. just reinvent database if you want to learn. at day job, for making money, it's much better to implement your GIANTideas(TM) instead of banging your head configuring db, docker, deployment... etc

[19:16:37] <CmdrKilljoy> -sigh- whats wrong with wanting to know what goes on under the hood?

[19:17:07] <saml> but our tools are pretty well documented. you can contribute as well. https://github.com/mongodb/docs

[19:44:23] <drsmith1> Hi. I've got a very interesting query where changing the .limit() from 37 to 38 changes the execution plan and slows down the execution by an order of magnitude.

[19:50:56] <drsmith1> here's the fast one: https://gist.github.com/devinrsmith/9ea1849a3f1347c55a2f

[19:51:14] <drsmith1> here's the slow one: https://gist.github.com/devinrsmith/7a6f8f2cd0cb6e0c3714

[19:51:34] <StephenLynx> that query is awful, imo.

[19:52:04] <StephenLynx> ah

[19:52:05] <StephenLynx> nvm

[19:52:09] <StephenLynx> I was looking at the results

[19:52:12] <StephenLynx> which are bad too

[19:52:21] <StephenLynx> that schema is overly complex.

[19:52:25] <drsmith1> i've got an index on geometry

[19:52:44] <drsmith1> 2dsphere.

[19:52:52] <drsmith1> if it's awful hopefully I can fix it!

[19:53:22] <StephenLynx> ah nvm

[19:53:28] <StephenLynx> that was the explain, right?

[19:53:32] <drsmith1> yeah

[19:53:32] <StephenLynx> and not your document?

[19:53:35] <drsmith1> correct!

[19:53:43] <StephenLynx> ok, I got it all wrong v:

[19:54:17] <drsmith1> so, it's executing a different plan between limit 37 and limit 38

[19:54:31] <drsmith1> assuming that's why it jumps up an order of magnitude in execution time…

[19:54:41] <drsmith1> but this is a bit lower level than i've treaded before

[19:57:44] <stig801> does an initial sync have to complete within the oplog window too? by initial sync... i'm talking for a new replica, does its initial sync have to complete within the space/time permitted by the oplog size?

[20:02:33] <devdvd> stig801, ntmk, initial sync is a full copy. The oplog is used the same way as binary logs in mysql

[20:03:11] <stig801> thats what I was thinking... but had others tell me differently...

[20:04:18] <drsmith1> i've posted my question here: https://stackoverflow.com/questions/33046222/order-of-magnitude-limit37-vs-limit38-nearsphere

[20:06:11] <devdvd> then maybe someone in here with more knowledge should answer

[20:06:20] <devdvd> cuz as i said, tmk that's how it works

[20:07:19] <drsmith1> leaving the channel, if somebody has any advice please post on stack overflow!

[20:07:35] <stig801> i just have a problem with the speed of the disks on the current setup... and a rsync from/to the disks isn't fast enough for the size of the oplog

[20:07:54] <deathanchor> stig801: raid!

[20:08:01] <stig801> from/to new disks

[20:08:08] <stig801> they are raided...

[20:08:32] <stig801> i would argue incorrectly... but... neither here nor there at the moment... and something i'm trying to rectify

Log file Viewer

Help | Karma | Search:

#mongodb logs for Friday the 9th of October, 2015