Log file Viewer
#mongodb logs for Friday the 3rd of August, 2012
[00:06:09] <Godslastering> do mapreduce jobs run, in any way, concurrently? i.e. - will running mapreduce take advantage of multiple available cores?
[00:55:30] <dstorrs> I have a sharded DB. there are no backups or replicas. I tried to create an index, it ate the entire CPU and locked the DB. I ^C'd the operation (which I've been able to do before). That didn't work. I tried to killOp the operation.
[00:55:59] <dstorrs> then I tried to service mongoX stop my servers. one shard went down easily, the other took a while then said FAILED. it now has a lock file hanging around.
[01:03:43] <nofxx> anyone got a replset can issue `mongotop` so I can sleep in peace today that my db is ok? local.oplog.rs 1945ms 1945ms 0ms
[01:07:30] <dstorrs> I need to get some data out of the system for a customer ASAP. That's why I killed this index.
[01:10:29] <nofxx> dstorrs, wow... that's a good load. Grab all the info you have on a pastie, post here and in the mailist
[01:14:53] <dstorrs> last 500 lines of the log on shard 2, the one with slow shutdown. shard 1 went down obediently
[01:36:59] <ukd1> does mongodb ensure ordering of arrays in document? I.e. if {$set: {x: [1,2,3,4,5]}} will always return x in the same order?
[11:14:45] <akaIDIOT> so I need another roundtrip, but for the 'inner' query i only need to retrieve something to restrict the result set for the second part
[11:15:26] <akaIDIOT> can i tell a query (from Java) to give me just the objids and tell the second query to restrict itself to the objids
[11:31:42] <akaIDIOT> the $in operator does get very very slow if you feed a long list of things right? :P
[11:34:51] <nemothekid> we've found coll.remove({$in => [1,2,3…,50]}) is much slower than for i in 50 coll.remove(i)
[11:35:42] <akaIDIOT> i've currently just saved all the ids of parents, chidlren, ancestors and descendants as list properties
[11:37:17] <akaIDIOT> also that will only allow me to efficiently get the descendants of a single node
[12:26:16] <akaIDIOT> managed to translate an SQL generator to a DBObject generator in Java in just a few hours
[12:26:56] <akaIDIOT> ♥ to the team that designed a Java binding without ending up with >>>> generics anywhere :P
[13:44:28] <souza> I'm trying to insert a big BSON in my mongodb, but I won't get this :( - It runs normally, but doesn't insert in database, this is my code: http://pastebin.com/azpxbgbS and if i try to print the BSON that will be inserted in MongoDB, I got this stack > http://pastebin.com/tB6uTnJU
[13:47:27] <souza> MANCHUCK: i recommend you read this > http://www.mongodb.org/display/DOCS/Security+and+Authentication =)
[13:48:19] <MANCHUCK> i followed that however i still need to run the command on the database i basically want to create a new root user
[15:09:41] <simon___> Ive been using 10.04 for a long time without any big issues. But thinking about how much more problems Ive had with ubuntu on the desktop than with Debian Im thinking about moving over to Debian.
[15:10:20] <simon___> Reason I switched on desktop Is because out of date desktop packages. On a server I really could not care less about version of XFCE etc.. :)
[15:10:54] <simon___> I.e using Debian has always been a more stable experience for me when it comes to desktop, is there any real difference on the servers?
[15:12:06] <vsmatck> Ubuntu is debian unstable with few changes. So I just weigh it as debian stable vs debian unstable (and some people like to run testing, unstable, or mixed).
[15:16:47] <TubaraoSardinha> Any idea on why mongo won't save dates on local time using ruby driver?
[15:19:38] <kali> TubaraoSardinha: mongodb dates are stored as offset from 1970-01-01T00:00:00 utc, without timezone information
[15:21:34] <TubaraoSardinha> kali: I was expecting this, but the thing is that I need to store consolidated user actions per day and my fear is that some user actions might end up on the wrong day due to timezone differences.
[15:22:57] <TubaraoSardinha> I imagined something like actions: {03082012: 5, 04082012: 3} with the key representing the day
[15:32:08] <kali> TubaraoSardinha: i would not use mongodb dates for that, because there are actualy timestamps
[16:13:57] <jordanorelli> is there a tool out there that will let me specify a hash/dict/object with collection names for keys and query documents for values, that will dump all relevant documents?
[16:14:19] <jordanorelli> that is, for doing something like getting all the data for a particular week, to extract a portion of my data to make a testing database.
[16:23:11] <Godslastering> i'm attempting to do something like this ( http://paste.pound-python.org/show/ckzGbjCZHg3qOy5ixFEI/ ) in python using pymongo. this is running pretty slow to iterate over so many rows, and i'm wondering if i can do some custom advanced query in mongodb to speed it up. Basically, i want to peer into multiple collections with one query
[17:06:31] <nofxx> anyone got a replset can run `mongotop` on the master? I'm in doubt it's normal: local.oplog.rs 1094ms 1094ms 0ms
[17:30:55] <Godslastering> can anyone possibly help explain to me how the heck this is happening? http://paste.pound-python.org/show/uvXZxFrwxkdMvoFauLHk/
[18:00:30] <Godslastering> this is uh .... extremely annoying, does anyone have any clue how in the world this could happen? http://paste.pound-python.org/show/uvXZxFrwxkdMvoFauLHk/
[18:05:00] <Godslastering> mpobrien: {u'name':'bob',u'hostname':'bob.com'} it doesn't have u'ip' .. i'm just wondering how in the world mongodb is giving me the document if it doesn't have u'ip'. my query is wrong somehow, i'm guessing
[18:06:56] <Godslastering> mpobrien: here is what i want "every document where 'ip' is (1) not null and (2) not equal to the string '__unresolved__'
[18:09:22] <mpobrien> or simpler yet, just filter them out on the application side. if not 'ip' in entry: continue
[18:09:39] <Godslastering> mpobrien: that's what i was doing, and it was ... err. slow. i'll try that line you just said
[18:10:23] <mpobrien> it shouldn't really be too bad unless there are a LOT of those docs that are missing the ip field; is that the case?
[18:10:40] <Godslastering> mpobrien: yes. about 3 million are missing the ip field, and about 12 million have the ip field
[18:16:15] <Godslastering> also, if i have about 10 pre-allocated files, and i delete a 2gb chunk of data, can i tell mongodb to re-arrange the data and remove empty pre-allocated files?
[18:49:03] <Godslastering> mpobrien: http://paste.pound-python.org/show/uvXZxFrwxkdMvoFauLHk/ still having the same issue. i've tried the same query another place in my application and i'm getting the same error; this seems like a proper query, but it's acting wrongly
[18:55:16] <mpobrien> @godslastering can you update that pastebin with your latest? it looks like the same thing to me
[18:57:35] <chubz> mpobrien: rs.status() shows the members of mt replica set and their name,health, state, stateStr, uptime,optime etc
[18:58:16] <mpobrien> yeah, but i mean, whats the raw output of the function? are you sure you are getting the right key?
[18:58:34] <mpobrien> IIRC members is an array, you rpobably need something like rs.status().members[0].stateStr
[19:01:46] <chubz> mpobrien: is there a way to display more than one member's state? like instead of rs.status().members[0].state something like rs.status().members[0..3].state
[19:02:49] <mpobrien> var x = rs.status(); for(var i=0;i<x.members.length;i++){ print(i + ": " + x.stateStr) }
[19:08:50] <mpobrien> @godslastering what happens if you run that query from the shell, do those docs come up
[19:09:40] <Godslastering> mpobrien: actually what i pasted is wrong. i dont want those docs. i'm lost, i've gotta come back to this problem later or tomorrow.
[19:12:14] <slavik_> is it possible to recursively search for a key=value pair and return the top level doc if it is found somewhere?
[19:23:05] <Godslastering> mpobrien: i am having an issue though: viewing my mongod output log, i'm getting like 80% of my queries over 150ms ... it's running on a quad core 3.2GHz i7 ... these are pretty simple queries, and i'm quering on indexed data. is this normal?
[19:24:24] <mpobrien> you should double check what the query plan is with .explain() just to be sure
[19:25:15] <Godslastering> mpobrien: ok i'm running mongostat. what do i do now? what am i looking for?
[19:27:22] <Godslastering> mpobrien: mongod is using 1gb ram, and i've got about 6gb of ram free on the system. how would i know if the bottleneck here was I/O?
[19:28:38] <mpobrien> also in the log, what does it say when it reports a slow query? should have something like nscanned etc.
[19:29:57] <Godslastering> mpobrien: should it be lower? doesn't that mean that is how many entries are in that collection?
[19:30:39] <Godslastering> mpobrien: hm... again, is that bad though? i did ensure an index on the fields i'm querying on
[19:30:52] <mpobrien> a high number like that would probably mean index isn't being used at all, or its not an effective one
[19:31:21] <Godslastering> mpobrien: appears not to be a disk bottleneck either. MB/s is only at about 0.25
[19:32:17] <mpobrien> try your queries with .explain() and check the output carefully - also using hte profiler might help
[19:34:09] <Godslastering> ok so i did a .explain(), mpobrien, and it's giving me this http://paste.pound-python.org/show/24581/
[19:34:49] <Godslastering> mpobrien: hm, ok. that explains it. but if i do db.mycollection.ensureIndex({ip:1,hostname:1,name:1}), why wouldn't this work?
[19:41:20] <Godslastering> mpobrien: alright, db.mycoll.getIndexKeys() is telling me exactly what i'd expect, but it's still using a basiccursor in .explain()
[19:42:26] <Godslastering> mpobrien: and attempting to force an index with db.mycoll.find({ip:'192.168.0.1'}).hint({ip:1}).explain() is telling me bad hint
[19:44:39] <mpobrien> you can query on (name) or (name, ip), or (name, ip, hostname) and it will use the index
[19:44:57] <Godslastering> mpobrien: i just did db.mycoll.ensureIndex({ip:1}) and now it's using a BTreeCursor
[19:45:20] <Godslastering> mpobrien: before i was doing db.mycoll.ensureIndex({ip:1,hostname:1,name:1}) thinking it would create all of those
[19:45:46] <mpobrien> or if you are querying on multiple fields together, you can use that syntax to create a compound index
[19:47:44] <Godslastering> mpobrien: wow, since i did that i haven't seen a single slow query show up yet
[19:53:51] <Godslastering> mpobrien: ok, since i was confused before, let me be sure i know what i'm doing here: i only have to do ensureIndex once, even if i insert a lot of data after this? will it rebuild the index?
[19:59:04] <kali> Godslastering: yes. index are maintained when you write, (not on the first read a la couchdb)
[19:59:49] <Godslastering> kali: alright, thanks. i was pretty confused with the documentation that i read
[20:01:25] <kali> Godslastering: that said, if you have a big chunk of data to load, it might be a good idea to insert and then to setup the index
[20:02:48] <Godslastering> kali: i already have the bulk of my data inserted, from now on it will be just like 10-20000 entries an hour, compared to the initial chunk. should i check to be sure the indices exist at the beginning of my script before i run to make sure i dont get bad performance?
[20:04:12] <kali> Godslastering: you don't have to do anything. if the index it there, it wont go away :)
[20:05:01] <Godslastering> kali: real quickly, how can i get all documents where the key 'ip' is either null or doesn't exist?
[20:05:27] <Godslastering> i was doing db.mycoll.find({ip:none}) but someone said that may not work?
[20:06:59] <Godslastering> and db.mycoll.find({ip:{$nin:[null,'__unresolved__']}}) was giving me some documents in which 'ip' wasn't even defined though
[20:09:54] <Godslastering> kali: that's fine, yeah, but db.mycoll.find({ip:{$nin:[null,'__unresolved__']}}) is still giving me documents where ip might be null
[20:13:38] <kali> Godslastering: http://uu.zoy.org/v/bizeri#lang=smalltalk&style=solarized_light this is working, but not nice
[20:17:54] <Godslastering> kali: here's my situation: i want to find entries where ip is not null and it doesn't equal '__unresolved__
[20:19:55] <Godslastering> kali: will {unresolved:{$not:true}} give me where unresolved:false and where unresolved doesn't even exist?
[20:22:46] <kali> yeah, this is why i prefer resolved: true, but i think unresolved:true might be more efficient in term of space
[20:24:13] <Godslastering> kali: can i rename later, easily, when i'm out of the 'development' phase?
[20:24:52] <Godslastering> kali: hmm. i'm not using _long_ names per se, but not short either. i'll have to change them soon, though, i guess
[20:25:47] <Godslastering> kali: luckily this is a personal project, but because it _is_ personal, lots of disk usage can get annoying
[20:35:15] <Godslastering> kali: if mongodb allocated 20 files when i had a lot of data, and then i remove like 4gb, can i tell it to get rid of unecessary pre-allocated files?
[20:37:52] <mpobrien> if you do a repairDatabase, it will re-write the datafiles with the minimum space needed
[20:59:46] <Godslastering> if an insert takes a while, it's most likely disk I/O causing that, right?
[21:02:34] <pas256> is there a reason why MongoDB with a read only workload doing geospatial queries would not use all cores on a system?
[21:04:59] <Godslastering> kali: i'm using macosx, so ... i can't remember of the top of my head what i'm using
[21:19:33] <mpobrien> if the collection isn't too huge, do a count() to get the size of hte collection
[21:20:36] <mpobrien> well, depends on what acceptable performance is for you… the potential issue is that .skip() gets expensive if you have a lot of docs
[21:24:37] <mpobrien> you're basically just assigning a random value to each doc, and then choosing one based on another random value at runtime
[22:50:51] <mpobrien> it has a "remove" option which will return a document and remove it in one option
[22:53:42] <nofxx> My server is almost idle, got some spare ram for mongo, 90% queries are < 100ms , but every once in a while a got some find by ids, going 300-700ms ...
[22:53:57] <nofxx> and this local.oplog.rs 1973ms 1973ms 0ms on mongotop.... things are normal? first trip sailor...
[22:55:25] <pas256> is there a reason why MongoDB with a read only workload doing geospatial queries would not use all cores on a system?
[22:55:41] <nofxx> that and mongoid that always but an $order, even in find by ids...: { $query: { _id: ObjectId('501c46c343d8a44c26000163') }, $orderby: { _id: 1 } } ... wondering if it could hit performance in some way
[22:59:28] <jordanorelli> manveru: in mgo it's Find(…).Apply(). it might be fixed by now, but there's a bug on findAndModify if your session mode is set to eventual
[23:04:18] <Liquid-Silence> can I do db.collection.update( {}, {$unset: {fieldname : ResourceId} }, false, true)
[23:04:51] <mpobrien> can I do db.collection.update( {}, {$unset: {"ResourceId" : 1} }, false, true)
[23:25:53] <jordanorelli> what's the normal way to see if a field exists on an object in mongo's javascript implementation?
[23:26:10] <jordanorelli> e.g., i get a document, regardless of whether it has some key, and then i want to see, for that document that i've already retrieved, if i have that key.
[23:26:50] <mpobrien> just fetch the document by its _id (or whatever) and check to see if the value for the key is undefined.
[23:27:58] <mpobrien> e.g.: doc = db.collection.find({_id:'foo'); if( 'ResourceId' in doc ){ print ("it has the field") }