Log file Viewer
#mongodb logs for Thursday the 26th of July, 2012
[00:12:24] <dstorrs> got a weird one here. if we do NOT index our jobs collection, we end up correctly processing all 27,000 jobs. if we DO index it, we end up correctly processing between 9,000 and 13,000
[00:14:30] <dstorrs> it's a pretty big issue -- runtime is 5 minutes with the index and 30 without, so it's a huge impact on our other systems.
[01:07:39] <krz> any help would be appreciated: http://stackoverflow.com/questions/11660931/how-do-i-group-by-id-and-country
[02:54:53] <sent-hil> how fast is mongodb's find when using time based selectors compared to regular find with string?
[02:56:16] <sent-hil> i've been working with relatively small amount of data (<100k records) and I don't see much of a diff.
[04:21:00] <at133> Hi, I have a map reduce query that all the sudden has stopped running. It is complaining about str.replace not being a function. If I trim the code down to only include the replace it will run, but if I try do a parseInt() on it it will complain about replace. This was running for days with no problems. I've stopped and restarted mongodb, it's 1.8.3. Any suggestions?
[04:36:12] <nemothekid> in your emit, remove the commas if you have to, then only deal with numbers in your reduce
[04:36:14] <at133> Thats what I thought, but I can run the replace on it without the parseInt, that works. Adding parseInt creates the replace not a function error.
[04:41:01] <at133> Yes, view_count is string, particularly values[0].view_count is a string because I've been echoing it out to confirm. Its "98,359". I can strip the comma out of it and print it as "98359" but parseInt of that causes mongo to return the replace is not a function nofile_b:0 error.
[04:58:08] <jamiel> Hi all, is it possible to upsert but not only update my "inserted_at" field if it's a new record?
[05:00:33] <krz> anyone can help me with this: http://stackoverflow.com/questions/11660931/how-do-i-group-by-id-and-country
[06:02:44] <nofxx> How would you model a "big queue"? Say like 10k users, need to some job on each, and keep that result for history.
[06:56:18] <krz> any ideas: http://stackoverflow.com/questions/11660931/how-do-i-group-by-id-and-country
[08:00:12] <Shadda> The big thing that sticks out to me is that mongostat is reporting that it's consuming nearly 5gb of ram
[08:02:27] <Shadda> The big thing that sticks out to me is that mongostat is reporting that it's consuming nearly 5gb of ram
[08:03:17] <Shadda> k i've no idea if those are sending but I won't spam the channel any longer with it
[08:04:13] <deoxxa> vsize is just the virtual memory size - the size of all the data mongo has mmap()'d and such
[08:04:42] <deoxxa> that's mongo mapping files into its address space - it doesn't actually use that much ram at one time
[08:06:24] <deoxxa> could it be that you've recently crossed a threshold where it does a lot more swapping?
[09:22:54] <fredix> I have a problem with 2 threads connected on mongodb, when the first thread write datas, the second cannot read datas
[09:23:26] <fredix> anyway I have set this options on mongodb connection : BSON("getlasterror" << 1 << "j" << true)
[09:44:08] <algernon> fredix: are you usiing two threads with a single connection, or two distinct connections?
[09:45:34] <fredix> algernon: I create two instance of nosql : https://github.com/nodecast/ncs/blob/master/nosql.cpp#L69
[09:50:58] <solars> hey, is there any way I can speed up this count query: https://gist.github.com/7dd239e41ec262257fdd - I'm not sure if the index is used also for count, is there a way to explain a count query?
[09:57:41] <krz> any ideas: http://stackoverflow.com/questions/11660931/how-do-i-group-by-id-and-country
[09:59:50] <remonvv> Is anyone aware of a method to force a mongos process to refresh cluster metadata from config servers without restarting mongos?
[10:05:16] <solars> NodeX, the queries are used for pagination, so its an unnecessary slowdown.. I guess I will just cache the count then
[10:29:57] <krz> "_id" => "20120725/foobar/song/test-pg3-long-title-here-test-lorem-ipsum-dolor-lo", will have
[10:52:24] <Habitual> I am in need of some guidance for your package, the "mongo guy" (our client) is having one or more of "network latency, network limits, disk io speed, io wait" issues. I have been beating up sysstat tools all day. I have added counts for mysqd and mongod processes in zabbix. I really need to identify the cause. Thank you.
[10:54:06] <Habitual> Well, where the issue lies mostly, he believes his setup "may" be not optimally configured. We're (I'm) looking for a CPU +IO spike of ~7 seconds.
[10:58:34] <Habitual> dunno exactly, it's a rather general complaint. He relies on the visuals in htop for this compaint.
[10:59:07] <NodeX> I would ask him to use explain() and see what is taking so long to make the CPU spike
[11:00:21] <NodeX> it could be any number of things, a bad / lack of index perhaps, the way he is querying maybe
[11:00:28] <remonvv> Anyone know why this is thrown? "shard version not ok in Client::Context: client in sharded mode, but doesn't have version set for this collectio...", code: 13388, ok: 0.0 }, ok: 0.0, errmsg: "should be sharded now"
[11:00:32] <Habitual> He won't or hasn't run or offered any speculation about what I see in htop -s PERCENT_CPU -u mysql or htop -s PERCENT_CPU -u mongodb and I have to ask why would mongostat be disabled? For a mongo guy, that seems counter-intuitive.
[11:08:11] <Habitual> typed mongod and it barfed ... Thu Jul 26 04:07:00 [initandlisten] closeAllFiles() finished...
[11:11:42] <kali> Habitual: also, you want to look for line talking about file preallocation in the logs
[11:12:09] <Rozza> I can do the counting fine - it does in the example its the collapsing into a single doc thats not working
[11:27:50] <Habitual> kali or NodeX can you peek at http://pastie.org/4335888 - log digging output. agent.log has 128 Traceback entries. Thanks.
[11:27:51] <krz> Rozza: nigel suggests this: "I'd try grouping by both _id and country first (letting you do the count you want), then group the result just by _id to give the structure you want."
[11:34:15] <kali> Habitual: mms is the mongo monitoring agent. it push monitoring data on a server belonging to 10gen somewhere in the cloud. I'm not using it, so I can't say if this is expected bahaviour
[11:37:40] <krz> Rozza: your answer at SO is right for grouping id and country. ill select it. but now i need to only group by country
[11:38:16] <kali> Habitual: file allocation lines look like that: 2012-07-02_05:20:17.74193 Mon Jul 2 05:20:17 [FileAllocator] allocating new datafile /xxx/yyy/zzzz, filling with zeroes...
[11:39:09] <Habitual> none of those that I saw. but the mnt/tmp/logs/mongodb/mongodb.log file is zero bytes, so I zgrep'd :)
[11:49:46] <kali> Habitual: for file allocation, you actualy want the "other" lines ("done allocating datafile") they say how much time it took
[11:50:32] <kali> Habitual: as for the queries, the nscan is not that high, so these queries are not lacking an index
[11:51:47] <kali> Habitual: it's a good thing as in "your developper knowns what he's doing" and a bad as in "we haven't found what we're looking for" :)
[11:55:43] <kali> Habitual: i think the next thing to do is run a db.currentOp() in the shell during one of these spikes
[11:58:13] <Habitual> is there a mechnism or c-li switch that can dump out result of db.currentOp() ?
[12:03:06] <NeoNmaNDK> If I must have "replication" set up on my Mongo server and put shared datasheed on, there are some that can send guides to this? I use CentOS 6.2 today.
[12:43:43] <DinMamma> I have a mongo instance that is reported as down in the replica-set. I have a SSH session alive(If I try to open a new one it times out).
[12:44:03] <DinMamma> The load on the machine is around 35 and I did a service mongodb stop but its not terminating..
[12:44:41] <DinMamma> To clarify "sudo service mongodb stop -> cursor is not returning".. Its been locked up for around 5 minutes.
[12:46:54] <Rozza> from a mongodb point of view a reboot should be fine. If its within the window of the oplog the node should auto recover
[12:54:13] <DinMamma> Before the machine died i did this query "db.database.collection({}, fields=({'product_hash':1, '_id':0})" In the collection there is ~80 million documents and there is a index on product_hash.
[12:54:20] <adamcom> when I see questions like DinMamma's I am so happy that journaling is default on in 2.0+ :)
[12:55:26] <adamcom> Rozza, if you are scanning every record (no criteria on product_hash) then it's still a table scan
[13:01:26] <adamcom> DinMamma: :) to expand a bit on my previous statement, if you use an index for a query that has to touch all (or nearly all) the documents in a collection, it can actually be slower than not using an index - the index has to be loaded into memory first, then the document, so it's more inefficient - of course, if everything is in memory already then the difference will be minor - if things have to be loaded from disk then the index scan + table scan is
[13:07:09] <DinMamma> adamcom: you cut off there but if I get you correctly it would be better to just db.col.find({}) rather than db.col.find({}, fields=....)?
[13:17:28] <adamcom> sorry, distracted - no, as I read your query (it's not shell format, and I don't recognize the driver language), the fields are just a projection, i.e. that's what you want returned
[13:18:19] <adamcom> my point was, that, in the case where you are going to have to touch all the records in a collection, using an index will be less efficient
[13:18:51] <adamcom> to expand a bit on my previous statement, if you use an index for a query that has to touch all (or nearly all) the documents in a collection, it can actually be slower than not using an index - the index has to be loaded into memory first, then the document
[13:19:07] <adamcom> hence it's more inefficient - of course, if everything is in memory already then the difference will be minor - if things have to be loaded from disk then the index scan + table scan is going to be slower
[13:19:47] <DinMamma> adamcom: Its python. From what I understood this will only look in memory so affectivly that would mean indexOnly: True?
[13:29:39] <adamcom> indexOnly: True is indicated in an explain when your entire query can be returned from the index (without touching the document itself) - the full index, in this case the one on product_hash will:
[13:30:29] <adamcom> I'd have to check if the index would be used for an empty criteria field, my gut says yes, but I'd have to check
[13:31:10] <adamcom> or, if you use explain() you can see for yourself - if IndexOnly : true is there and it's using the Btree cursor, then you are good
[13:32:24] <adamcom> note: the index doesn't have to all be in memory for it to be an indexOnly : true (i.e. covered index) query, but that's what you want for it to be nice and fast
[13:32:58] <adamcom> get the host into MMS and watch page faults and IO (if you install munin) to get a feel for it over time
[13:37:38] <joeritchey> The sync seems to finish but new server is showing less than 1/2 of the size of the original database
[13:38:57] <hashpuppy> was hoping scott was in here. do you guys know the state of morphia? i'm still on 1.6.3 and an considering upgrading. will morphia work with 2.0?
[13:47:17] <hashpuppy> there he is. skot, i just asked this moments before you joined: hashpuppy: was hoping scott was in here. do you guys know the state of morphia? i'm still on 1.6.3 and an considering upgrading. will morphia work with 2.0?
[13:57:48] <adamcom> joeritchey: is the host in MMS? (looking to see what the repl ops counters are like)
[13:59:19] <adamcom> and the smaller data size is normal - when you do an initial sync of a secondary, it writes the data files from scratch on the new secondary - all padding, empty space is removed
[14:00:04] <adamcom> hashpuppy: AFAIK morphia will work just fine with 2.0 but I don't know if there were any breaking changes off the top of my head
[14:01:21] <joeritchey> I understand the differences in padding. The original database is 17GB and on the new secondary it is showing 3.9GB
[14:48:21] <clone1018> I'm having quite a weird issue with MongoDB and it's PHP driver, basically I've enabled auth=true in the mongodb config file, and verified my script can login using the user information I selected, but you're still able to make a connection and read data using without using a user on localhost
[14:58:51] <NodeX> "To enable secure mode, run the mongod process with the --auth option (or --keyFile for replica sets and sharding). You must either (1) have added a user to the admin db before starting the server with -auth, or (2) add the first user from a localhost connection (you cannot add the first user from a connection that is not local with respect to the mongod process)."
[15:27:24] <sspy> I use MongoVUE to manage my DB and I see 2 columns - Size(40.8 MB) and Storage(51.3 MB). Which one handles my data and what is the purpose of another ?
[15:30:26] <linsys> sspy: one is the actual size on disk (storage) and the other is the actual size of your database
[15:31:30] <clone1018> I'm having quite a weird issue with MongoDB and it's PHP driver, basically I've enabled auth=true in the mongodb config file, and verified my script can login using the user information I selected, but you're still able to make a connection and read data using without using a user on localhost
[15:55:30] <balboah> yup. Maybe times 4-5, depends on how your setup looks like. Hard to translate 1 to 1
[15:56:55] <balboah> sspy: http://www.wikivs.com/wiki/MySQL_vs_MongoDB says between 2-10. Looks just about right as I have a version of the same data in both mongo and mysql
[16:40:28] <pdtpatrick> Question - i'm trying to make a node in a replica set arbiter only, but keep getting the following: http://pastie.org/private/chjsgfd5jdwaz1cjt9hpw
[17:29:19] <pdtpatrick> can u bing mongo on multiple ports? for instance, localhost and also eth0? right now i have it on eth0 and i can only connect on that nic and cannot connect via socket to localhost
[17:31:16] <pdtpatrick> right but how can i keep eth0 active cuz i need to bind to it from another host
[18:00:04] <Rhaven> Hi all, i'm currently stuck with some errors related to mongodb using replica sets and sharding.. Is there someone to help me?
[18:03:20] <joeritchey> I have been working through a few issue myself so I have been reading up the last few days
[18:04:13] <Rhaven> i'm actually running with 3 configs server, and 3 mongos and 1 mongos is stuck with "scoped connection to [xxxx] not being returned to the pool
[18:09:28] <joeritchey> are you seeing the error in the mongos log or is the driver returning a error?
[18:12:31] <Rhaven> but around 1/30 times the server ping successfully all config server and the next 10s after he say the same error
[18:12:58] <joeritchey> so the server having the problem can ping the hostname or ip address of each server as it appears in config servers
[18:13:53] <joeritchey> if you just run ping for a few minutes to the config servers does it respond with 100% success
[18:57:44] <pdtpatrick> Question - i've installed Edda and ran "edda /var/log/mongodb/mongodb.log" -- however the page that comes up on port 28000. Show just the frame and no data
[19:23:33] <idefine> JoeyJoeJo: http://www.mongodb.org/display/DOCS/Overview+-+The+MongoDB+Interactive+Shell#Overview-TheMongoDBInteractiveShell-Deleting
[20:08:50] <tphummel> hello folks. i'm trying to shard the resulting documents of a Map/Reduce job using "out: {sharded: true}". using both 2.0.x and 2.2 i get an error at the end of the job. curious if anyone else has seen this error: err { [MongoError: MR post processing failed: { errmsg: "exception: unknown out specifier [sharded]", code: 13522, ok: 0.0 }]
[20:08:51] <tphummel> errmsg: 'MR post processing failed: { errmsg: "exception: unknown out specifier [sharded]", code: 13522, ok: 0.0 }' }
[20:12:31] <tphummel> it appears that it doesn't like the key "sharded" in the out object. i'm wondering if is that necessary anymore
[20:29:41] <nofxxxx> Wow, 2.2.rc0... finally, mongodb is the only piece of my stack w/o remote syslog
[20:45:24] <krispyjala> hey guys, trying to do a group by, and it works with this command: db.discourses.group({key: {name:true}, cond: {evaluation:"OK", vlifecycle: {$ne:"retire"}}, reduce: function(obj,prev) {prev.csum++; }, initial:{csum:0}});
[21:12:20] <dstorrs> anyone here use the Perl language driver? how do I rename a collection using it?
[21:20:34] <nemothekid> dstorrs: $db->run_command({renameCollection => "db.collection", to => "db.newcollection"});
[21:23:50] <dstorrs> aha. say Dumper [ $db->run_command(Tie::IxHash->new(renameCollection => "dstorrs.foo", to => "dstorrs.bar" )) ];
[21:25:33] <dstorrs> tiripamwe: in the Perl client it would be like so: $db->coll->find({ field => { '$regex' => /foobarbz/ } })
[21:25:54] <krispyjala> i ran this and it was fine db.discourses.group({key: {name:true}, cond: {evaluation:"OK", vlifecycle: {$ne:"retire"}}, reduce: function(obj,prev) {prev.csum++; }, initial:{csum:0}});
[21:27:54] <tiripamwe> dstorrs: thanks a lot, it looks like this in python db.coll.find( { field : re.compile('foobarbz') } )
[21:32:29] <nemothekid> dstorrs: both work in both languages actually. The former is more powerful because you can do things like find({field : {$not : re.compile('foobar')}, otherwise you couldn't do a not on a regex
[21:32:39] <nemothekid> krispyjala: I think group returns an array so you can just filter it yourself
[21:32:59] <nemothekid> otherwise you could always add a finalize function like if (scum == 0) return null
[21:33:28] <dstorrs> nemothekid: well, unless you use a negated regex ("anything but" character class, negative lookahead, etc)
[22:07:57] <tphummel> ahh. figured out my problem above, in which i was getting: err { [MongoError: MR post processing failed: { errmsg: "exception: unknown out specifier [sharded]", code: 13522, ok: 0.0 }]
[22:08:54] <tphummel> not so good with c++ but looks like the order is significant in pulling properties off the "out" object
[22:08:58] <tphummel> https://github.com/mongodb/mongo/blob/master/src/mongo/db/commands/mr.cpp#L243
[22:10:05] <tphummel> out: { sharded: 1, reduce: "name"} failed while out: {reduce: "name", sharded: 1} works
[22:51:59] <dufflebunk> I have a simple set of data, but I need to perform a greatest not-less-than-X query on it. So I might search for "5" and the document it would return would have a key of "5" if it exists, or "6" if "5" doesn't exist, ...
[23:19:22] <vsmatck> dufflebunk: You'd need a index on the integer with the right sort order. I haven't thought it through fully but I think you can do it.
[23:21:30] <dufflebunk> I was thinking of indexing the field, then find({field:{$gte:<some value>}}).sort().limit(1)
[23:22:20] <dufflebunk> but I wasn't sure if the db is smart enough to use the index for the sorting, and also smart enough not to sort the entire set of results when I only want the first result.
[23:22:58] <krz> given this structure. how can i include the country_name to the result set? https://gist.github.com/3161542
[23:42:15] <dstorrs> embedded collections for something potentially unbounded like visits are a bad idea. They are hard to work with, and they risk blowing your 16M limit
[23:43:09] <dstorrs> what exactly are you trying to achieve here? that query is too complex to understand easily
[23:54:23] <krz> dstorrs: I'm using heroical documents because I'm following http://www.10gen.com/presentations/webinar/real-time-analytics-with-mongodb