pmxbot IRC Log Viewer

[00:21:26] <LoOoD> I'm trying to change the priority of the primary member in my mongodb 2.0 replicaset (cfg.members[3].priority = 1) but get this error, "TypeError: cfg.members[3] has no properties (shell):1". Any ideas why?

[02:57:10] <hendry> How do I make mongodb by default listen only for local connections? TCP localhost:28017 & not TCP *:28017 (LISTEN), which is frankly dangerous ?

[03:00:11] <crudson> Did you try --bind_ip 127.0.0.1 ?

[03:00:18] <w3pm> only dangerous depending on your firewall setup

[03:00:29] <w3pm> mongodb says as much

[03:00:36] <hendry> w3pm: i don't have a firewall

[03:01:00] <w3pm> ok then :P

[03:07:52] <jrdn> no firewall hmmmmm

[05:09:56] <joshua> Does anyone know offhand if there is an example limits.conf using the suggested mongodb settings for ulimit? I'm lazy

[05:37:46] <joshua> I think all I have to adjust is number of open files in ulimit. Bingo bango

[07:41:41] <ankakusu> I want to delete a database in mongodb. When I enter the command db.dropDatabase(), I get the following error:

[07:41:45] <ankakusu> "Can't take a write lock while out of disk space"

[07:41:59] <ankakusu> How can I get over with this error?

[07:42:15] <ankakusu> and delete that database in mongodb

[07:43:18] <joshua> ankakusu: have you tried removing the collections first? Just wondering. Don't know enough about that issue to give expert advice

[07:43:33] <ankakusu> yes.

[07:43:50] <ankakusu> first tried to move collections but I get the same error.

[08:20:20] <joshua> I have to bounce my nodes and a stepdown had a few connections errors... poop. Was hoping failover would be seemless

[08:21:00] <joshua> or maybe it was when I restarted before I decided to run stepdown first

[08:44:57] <[AD]Turbo> hi there

[10:08:20] <NodeX> \quote pass ron_is_annoying

[10:08:24] <NodeX> damnit

[10:18:52] <Antiarc> I've got something of an index puzzler: https://gist.github.com/ad0f8b015b3d8e80ea44

[10:19:15] <Antiarc> Basically, when explaining a query, it's fast and appears to use an index, but if I drop the explain, then it is slow and appears to do an almost-full-collection scan

[10:19:35] <Antiarc> I'm really scratching my head on this one. Anyone have any ideas?

[10:21:14] <NodeX> can you gist your indexes

[10:22:33] <Antiarc> updated with the relevant indexes

[10:23:12] <Antiarc> I had a straight {tags: 1} index as well, and I noticed that the query optimizer would prefer it over the compound index, as well, resulting in a much slower query. Since the tags_1_post_date_-1 index covers the normal tags index, I dropped it.

[10:23:24] <Antiarc> But I feel like there's something fundamentally weird going on here that I am completely missing.

[10:24:19] <Antiarc> I can't for the life of me figure out why an explained query would only scan a few documents and would not scanAndOrder, while the non-explain version performs the scanAndOrder

[10:25:21] <Antiarc> What's even more fun is that if I explicitly .hint({post_date: -1, tags: 1}), it's very fast

[10:25:34] <NodeX> which is why I asked for your indexes

[10:25:42] <NodeX> it might be choking on another index

[10:25:52] <Antiarc> Well, I can dump the full set, sure. Sec.

[10:26:17] <Antiarc> gist updated with the full set.

[10:26:37] <kali> Antiarc: with explain(), the optimizer re-evaluates the plan

[10:26:47] <kali> Antiarc: without it, it can use the cached one

[10:27:08] <Antiarc> kali: But I've dropped and re-added indexes, and run reIndex(), which should cause it to recache the optimal plan according to the doc.

[10:27:24] <Antiarc> Additionally, explicitly hinting ({tags: 1, post_date: -1}) results in a slow query

[10:27:53] <Antiarc> the fact that that's what explain is saying it used is making me wonder if it's picking the wrong index somehow. I mean, I can just manually hint from my app, but that's annoying and seems like it'll cause maintenance issues.

[10:28:10] <kali> agreed

[10:28:59] <Antiarc> This is 2.2.0, fwiw.

[10:29:03] <kali> i'm not sure there is any way to inspect the query plan cache

[10:29:59] <Antiarc> Well, the fact that explain() is actually selecting the *wrong* index is a big ol' red flag for me, but I'm not sure what I'm doing to get it to settle on that index.

[10:31:32] <Antiarc> well, I can give it a post_date: $gt as a query condition, which gets it to use the right index.

[10:31:36] <Antiarc> But man, that's annoying.

[10:56:44] <oskie> background parameter only has an effect when creating the index, no?

[11:03:23] <NodeX> yes

[11:04:09] <Zelest> yay!g

[11:04:25] <Zelest> got $200 as a "thank you" for reporting a few issues at my VPS provider. :D

[11:04:31] <Zelest> or well, two fairly nasty security holes

[11:07:40] <oskie> nice of them!

[11:07:47] <NodeX> nice

[11:08:35] <solars> hey, is there any tool, admin interface, etc that provides a list of (slow) queries on a collection and whether an index is used or not? (other than the logfiles or explain output, to quickly get an overview of which indices should be created)

[11:09:26] <Zelest> indeed.. always nice to get some free hosting :D

[11:28:18] <joshua> solars: you can adjust the mongo profiler level to log the slower queries. If you can enable MMS though that probably would be handy to see things graphically

[11:38:30] <solars> MMS?

[11:39:28] <joshua> http://mms.10gen.com/help/

[11:42:04] <solars> oh, I didn't know that

[11:42:53] <Antiarc> This index issue has me stumped. I can make it work via hint, but I absolutely cannot get mongo to select the "fast" index on its own.

[11:44:42] <Antiarc> bah, I'll solve it tomorrow.

[11:44:44] <joshua> We went over that in the training but I forget the specifics.

[11:44:57] <joshua> Thats why I gotta take the online class coming up. heh

[11:45:20] <Antiarc> joshua: I thought I knew mongo's indexing stuff forwards and backwards, but this one is just behaving completely counter to everything I know about its indexing behaviors.

[11:45:32] <Antiarc> But. I can solve it tomorrow. Now I need sleep :)

[11:45:45] <NodeX> Antiarc : did you gist your indexes?

[11:45:52] <Antiarc> NodeX: Yes, I added them to the original gist.

[11:45:58] <NodeX> I dont see them

[11:46:02] <Antiarc> https://gist.github.com/ad0f8b015b3d8e80ea44

[11:46:03] <Antiarc> scroll down

[11:46:12] <NodeX> db.collection.getIndexes();

[11:46:18] <Antiarc> Yes, that's what I did :P

[11:46:25] <Antiarc> http://cl.ly/image/1G181T141b17

[11:46:29] <Antiarc> It's there! I promise!

[11:46:43] <NodeX> it's not on mine

[11:46:47] <Antiarc> o.0

[11:47:22] <Antiarc> I've got two files in that gist, the first being the output, and the second being the indexes. You're only seeing the one?

[11:49:09] <NodeX> I see them now

[11:49:21] <Antiarc> relevant indexes are the last two.

[11:49:25] <NodeX> you have 2 indexes, drop one

[11:49:52] <Antiarc> I've tried every combination of those indexes, individually and compound, no difference.

[11:50:04] <Antiarc> (as in, dropped them all, tried them individually)

[11:50:18] <NodeX> your sort should always be the last part of the index

[11:50:39] <NodeX> it's being confused because you haev 2 conflicting indexes

[11:50:53] <Antiarc> Well, it stays confused even if I drop one or the other. Happy to demonstrate. sec.

[11:51:15] <NodeX> drop the one that starts with -1

[11:51:21] <NodeX> reindex the collection

[11:51:32] <Antiarc> Bah, I'm gonna have to do a code revert before I can do that

[11:51:37] <Antiarc> Since I'll otherwise get bad hint exceptions.

[11:51:45] <Antiarc> Yeah, I'm just gonna do this tomorrow.

[11:52:05] <Antiarc> But yes, I really did try having only one index or the other and doing a reindex and running that test and it was still slow as hell.

[11:52:27] <Antiarc> Anyhow, I'll pick it up later. Have a good (evening/morning appropriate)

[12:30:48] <remonvv> t

[12:46:47] <NodeX> friday

[12:47:16] <joe_p> :)

[13:48:35] <solars> does anyone know if there is already a solution for the slow count query problem?

[13:49:25] <solars> I always have queries like this:

[13:49:35] <solars> Fri Jan 11 14:46:59 [conn815] command history_production.$cmd command: { count: "status_lists", query: { hotel_id: "IT_BRIX_PANOPOST" } } ntoreturn:1 keyUpdates:0 numYields: 90 locks(micros) r:20330726 reslen:48 10560ms

[13:54:03] <remonvv> solars, define "solution" ;)

[13:54:16] <remonvv> There is a rather important optimization in 2.3 for counts.

[13:55:21] <solars> well, a performance improvement I guess :)

[13:55:26] <solars> do you know if that improves the above query?

[13:55:57] <remonvv> https://jira.mongodb.org/browse/SERVER-7745

[13:56:06] <remonvv> And my test here : http://imgur.com/9nxyH

[13:56:30] <remonvv> solars, it should improve dramatically if the hotel_id is indexed

[13:58:00] <remonvv> The optimization becomes more effective the lower the cardinality of hotel_id is

[13:59:02] <solars> remonvv, those are my indexes: https://gist.github.com/c01913f88424098e1c31

[13:59:08] <solars> shouldn't the bottom one help already?

[13:59:26] <remonvv> Not until this optimization

[13:59:30] <solars> oh

[13:59:35] <solars> ok let me read, sec

[14:00:03] <solars> ah

[14:00:05] <solars> I didn't know this ticket

[14:00:21] <solars> only https://jira.mongodb.org/browse/SERVER-1752

[14:00:30] <solars> thats great, in this case I'll update the db

[14:00:33] <remonvv> right now it does a full index scan, the optimization allows MongoDB to use key value intervals

[14:00:39] <solars> yep, I read it

[14:00:46] <remonvv> if you look at my tests you can see it's quite a bit faster

[14:00:50] <solars> thanks a lot fo rthe pointer! that should help a lot

[14:00:54] <solars> yeah

[14:01:47] <remonvv> If you run this in shell :

[14:01:53] <remonvv> db.status_lists.count() / db.status_lists.distinct("hotel_id").length

[14:02:21] <remonvv> you get a value that is higher if the cardinality is lower

[14:02:42] <remonvv> the higher the value the more effective the optimization is

[14:03:13] <remonvv> It's not an exact science but it gives you some indication

[14:03:37] <solars> sec

[14:03:49] <remonvv> By the way the distinct can be very slow on large datasets so use with caution

[14:04:55] <solars> 2989.1000239865675

[14:05:09] <remonvv> ok, and total collection count()?

[14:05:34] <solars> 12461668

[14:05:46] <solars> the above value was the result of the division already

[14:06:54] <remonvv> I understand. My tests used data where that value was 1000 for the low cardinality test and 10 for the high one.

[14:07:13] <remonvv> 1000 gave about a 5x improvement so for you perhaps that's between 6-20.

[14:07:38] <remonvv> 2.3.x is binary compatible so updating mongod (temporarily) should tell you what you'd gain

[14:15:26] <solars> sounds good

[14:18:49] <solars> remonvv, hm is there a stable release for 2.3 or is this the development series?

[14:18:51] <solars> I only see 2.2

[14:18:59] <solars> I thought 2.3 is out already

[14:20:26] <solars> I'm on 2.2.2

[14:41:45] <remonvv> Odd minor versions are always the development version of the even (stable) versions

[14:41:52] <remonvv> So 2.3 is the dev version of 2.4

[14:42:03] <remonvv> So use in production at your own risk.

[14:42:14] <NodeX> are version specific relations coerced?

[14:42:24] <remonvv> In my experience dev version tend to be stable

[14:42:37] <remonvv> ish

[14:42:50] <remonvv> NodeX, what do you mean?

[14:43:06] <remonvv> Englishos not is language native thank you please

[14:48:26] <NodeX> I was hoping you could tell me what I meant :(

[15:00:09] <ehershey> I was just trying to build a dev version

[15:00:11] <ehershey> but interupted

[15:00:11] <ehershey> real 133m9.525s

[15:03:40] <solars> remonvv, alright, thanks!

[15:14:18] <solars> is there a rough date for the 2.4 release? otherwise I'll try the dev version

[15:25:45] <NodeX> no dates yet

[15:29:34] <solars> ok then I'll have to use the dev version

[15:46:20] <ehershey> I thought there was a rough timeline

[15:46:45] <ehershey> like 6 months after 2.2

[15:48:49] <ehershey> which I guess would be februaryish

[15:49:30] <NodeX> they have a roadmap on the website somewhere iirc

[15:49:43] <NodeX> 2.2 was delayed a few weeks iirc

[15:53:01] <joe_p> https://jira.mongodb.org/browse/SERVER#selectedTab=com.atlassian.jira.plugin.system.project%3Aroadmap-panel

[16:02:21] <ehershey> nice

[16:05:24] <lunali> hey dudes, im using the aggregate framework and i am having a match and group in the pipeline, i want the group function to limit the registers it uses to 100, for example

[16:05:28] <lunali> how can i do that?

[16:06:08] <lunali> i want to use the last 100 registers which "match" the $match statement

[16:06:15] <lunali> for grouping

[16:06:34] <lunali> and not all

[16:06:39] <lunali> or, instead of all.

[16:07:36] <NodeX> I am not sure that's possible if I understand you correctly

[16:08:38] <lunali> hmm, in the pipeline im having a match and the result of that match is used for grouping, right?

[16:11:37] <lunali> when a match returns 20 documents, the group function is grouping them, but i dont want 20 documents, i just want the last 10 of the all (20) documents

[16:12:18] <lunali> *of all (20) the documents

[16:12:24] <lunali> dsadsk

[16:33:29] <rekibnikufesin> lunali: are you looking for something like $limit? http://docs.mongodb.org/manual/reference/aggregation/limit/#_S_limit

[16:35:35] <NodeX> that limits everything I think, not a single match in a pipeline, I could be mistaken

[16:37:17] <lunali> no, you are right NodeX, that limits everything

[16:37:32] <lunali> reikalus1kka: i just want to limit the results of the match which the group is using

[16:37:34] <lunali> "just"..

[16:40:02] <NodeX> I really dont think you currently can, I would love to be told a way to do it also

[16:44:55] <iwaffles> What's the preferred method for sharing schema/structure between multiple people?

[16:45:24] <lunali> you can use cloneCollection

[16:45:31] <lunali> if you dont want to replicate

[16:49:43] <iwaffles> And dump that into a file?

[17:15:49] <linula> so its not possible to limit the matching results in the aggregation framework pipeline?

[17:24:28] <AlphaPinky> What does CRUD mean?

[17:24:38] <kali> Create ... Update Delete

[17:24:39] <anewkirk> CRUD == create, read, update and delete

[17:24:44] <kali> read, yeah

[17:24:58] <AlphaPinky> oh.

[17:25:29] <kali> AlphaPinky: that usually refers to the dozen pages of boring forms you have to code in your webapp

[17:25:55] <kali> AlphaPinky: and i tend to disagree with this statement. CRUD is not a database thing, it's a web framework thing

[17:30:43] <xbuzz> looking to see if there is a performance hit for $inc multiple fields in a single update. if i'm updating 3 fields versus 6 fields.

[17:32:54] <kali> xbuzz: i would expect the difference to be quite insignificant compared to the cost of locking, fetching, journaling, plumbing, etc

[17:34:08] <kali> maybe more significant if there are indexes on the fields

[17:36:20] <xbuzz> kali: gotcha. thanks. indexes are def important here. this will be ran about 5 million times a day. so at that rate it's updating 15-30 million fields. i figure a single update is more efficient. just wanted to make sure.

[17:37:03] <xbuzz> kali: or updating fields 15-30 million times. not 15-30 million different fields / docs :D

[17:45:54] <JoeyJoeJo> How can I add a field to an existing document?

[17:52:45] <kali> JoeyJoeJo: $set

[18:00:10] <JakePee> {$set:{field:value}}

[18:25:15] <iwaffles> So how do people share schemas when they're developing the same application?

[18:25:39] <iwaffles> Is there any sort of project that'll take one database and clone the structure for use with teams?

[18:26:08] <kali> the structure ?

[18:26:15] <iwaffles> Yeah

[18:26:31] <iwaffles> That's what I meant :P

[18:26:32] <kali> the only thing that is close to that is the list of indexes

[18:26:41] <kali> the structure is in each document

[18:26:46] <iwaffles> Hmm..

[18:27:09] <iwaffles> So how do teams share databases typically?

[18:27:23] <kali> we copy them, or part of them

[18:27:39] <kali> but the actual knowloedge about the schema is in the application(s)

[18:27:58] <efeller> 1

[18:28:01] <kali> particularly if you use a layer of ODM mapping

[18:28:46] <iwaffles> Right, I totally get that

[18:50:37] <remonvv> ORM/ODM, it's still an interesting debate which is the more appropriate term ;)

[18:51:36] <remonvv> ODM is accurate, ORM is more recognizable. The joy of development acronyms.

[19:17:19] <rev1> i'm looking for documentation on querying historical server status information, e.g. how can i find out what the network utilization was for our primary on cyber monday?

[19:22:40] <blankstare> hi all. i just recently compiled the mongo driver for xampp and every time I try and add the extension to php.ini, apache serves php as a d/l'able file. any suggestions as to what may have gone wrong here?

[19:27:32] <kchodorow_> blankstare: check your apache error log

[19:27:55] <blankstare> ah it looks like an architecture mixup

[19:28:05] <blankstare> thanks kchodorow_

[19:28:43] <blankstare> i'm guessing php is x86 and x86_64 was targeted

[19:29:21] <blankstare> how would i compile mongo.so as 32bit?

[19:33:15] <rev1> hey everyone; does anyone know how to query mongo for network utilization? mms is doing it by magic, which is fine, but i need to look at hourly utilization back in november, and mms only lets me look at daily charts back that far.

[20:15:47] <a|3x> hi

[20:17:36] <a|3x> my mongod process has a virtual memory size of 12 gb but resident and shared only 2 and 2, how can i tweak the setup to allow more stuff in memory (i have 12 gb of ram)?

[21:53:35] <tworkin> is there a tool for inspecting BSON blobs? I keep getting bad element type, and even trying to enumerate what fields are in the object is throwing the bad element error

[23:48:05] <w3pm> i thought replsets were supposed to be transparent to the application? why are there mongo_replset methods in the drivers?

[23:56:00] <PhanLord_> how could i get count/sum of the array sizes within documents

[23:56:43] <PhanLord_> and number of items within arrays within array of document :)

Log file Viewer

Help | Karma | Search:

#mongodb logs for Friday the 11th of January, 2013