Log file Viewer
#mongodb logs for Monday the 15th of July, 2013
[03:25:02] <jonjon> I'm really new to NoSQL. Can I get some quick opinion on this? http://pastebin.com/crGF5CV5
[05:39:08] <racarter> hello, my java mongodb client is using this connection string: mongodb://127.0.0.1:20000/mydb, I have a fresh mongod server running on that port but when I start my client I get this message: couldn't connect to [localhost/127.0.0.1:27017] bc:java.net.ConnectException: Connection refused
[06:51:46] <racarter> imsc: mac. I actually figured it out. the connection string had an extra slash. it had to be mongodb:/127.0.0.1:2000/mydb instead of mongodb://127.0.0.1:2000/mydb
[07:34:41] <carl_> can I get some help, I'm having a problem with my dbpath in ubuntu (new to ubuntu)
[07:38:32] <das3in> I've tried editing the /etc/mongodb.conf file, but when I run 'mongod' it still says that it can't find it
[08:14:17] <tadasZ> Good morning everyone! strange thing happened today - I was doing some stuff with website I'm building, and from time to time I check my collections using RockMongo (or console), I inserted some records and tried to check them in RockMongo and they are not there, then tried using console and they are still not there, but my website is returning saved documents as if nothing happened
[08:18:29] <tadasZ> I tried to view my collections using RockMongo and console, both shows that they are empty, but my php find query returns data
[08:20:03] <Nodex> check the "test" db on the console - that's where default NON selected db inserts end up iirc
[08:22:05] <tadasZ> :D I was inserting documents in demo server, and checking my collections on localhost
[09:22:14] <rspijker> a mongos instance can only be part of 1 replica set. An arbiter is a mongos instance, so...
[09:23:21] <statu> Anyone knows why I get a "VersionError: Document not found" when I save a document?
[09:26:27] <rspijker> http://stackoverflow.com/questions/17499089/versionerror-no-matching-document-found-error-on-node-js-mongoose
[09:30:59] <statu> rspijker, It is supossed that the error occurs when you modify an array by index, but I get it using the push method :S
[10:15:26] <[AD]Turbo> Mon Jul 15 12:07:17.083 [conn183] authenticate db: mytestdb { authenticate: 1, user: "myadmin", nonce: "75a2390bdbe1c475", key: "61d19bc9d2ed17e2f628b6921721b0c4" }
[10:15:26] <[AD]Turbo> Mon Jul 15 12:07:17.083 [conn183] warning: No such role, "userAdminAnyDatabase", in database mytestdb. No privileges will be acquired from this role
[10:15:26] <[AD]Turbo> Mon Jul 15 12:07:17.093 [conn183] assertion 16550 not authorized for query on mytestdb.system.namespaces ns:mytestdb.system.namespaces query:{}
[10:15:27] <[AD]Turbo> but in that database there is myadmin user with userAdminAnyDatabase privileges:
[10:15:28] <[AD]Turbo> { "_id" : ObjectId("51d6c57657623078d952997b"), "user" : "myadmin", "pwd" : "d6d397eb49b89a807f7a640e9665d2b0", "roles" : [ "userAdminAnyDatabase" ] }
[10:26:45] <[AD]Turbo> mongo shell (that runs ok), is executed on the same machine where mongod runs, external admin tool in on another machine (inside the same LAN)
[10:44:08] <richthegeek> I'm having some trouble with data not being found in some test units... not sure what's going on
[10:45:01] <richthegeek> using the Node driver, with write concern set to 1, after inserted the data {_id: 1, name: "bob"}, a separate connection then queries for {_id: 1} and finds nothing. It then queries again (some condition) and finds the inserted row.
[10:49:28] <Derick> mhmm, sounds odd, I don't know how your code works of course, but do remember that node does lots of things asynchronously
[10:49:52] <richthegeek> yeah, this is using the async library to perform a series of steps in the test
[10:50:22] <richthegeek> inside the "insert data" step, there's a serial async array - one callback for each row inserted
[10:51:58] <richthegeek> oh wait i've figured it out... the data was a "join" operation so it was inserting stuff from the "left" side before the "right" side, and then finding rows in the right before they were properly inserted
[11:18:22] <remonvv2> Derick, okay. https://jira.mongodb.org/browse/SERVER-6749 is reoccuring for us and caused a rather spectacular failure.
[11:21:05] <remonvv> Are you aware of any issues being fixed with mongos metadata consistency in 2.2/2.4? This is 2.0.8
[11:23:45] <remonvv> I understand ;) We basically invoked 9 addShard commands and out of nowhere it started complaining about the error mentioned in that ticket, again.
[11:32:33] <remonvv> We've upgraded it this weekend but we had a complete cluster failure due to this.
[11:33:36] <remonvv> And now I have to find a tool to combine 25 mongos logs into one time sorted one ;)
[11:34:41] <remonvv> The worrying thing is that I can't find any 2.2-2.4 issues in JIRA that would have addressed this error.
[11:36:06] <remonvv> We're scale up and down constantly so it's likely we run into this sort of thing more often if it is still an issue.
[12:47:54] <greenmang0> hello all, we just cleaned a lot of data from our production mongo replica set. we were expecting it would free up a huge disk space, but it didn't, after looking up mongodocs we realized that we can't clean it up - http://docs.mongodb.org/manual/reference/method/db.repairDatabase/#db.repairDatabase , so my question is if there any way we can free up the disk space"
[12:55:22] <Infin1ty> if you go the repairpath and you don't have enough freespace, you can mount an nfs mount or additional storage and use repairpath so it'll store temp files there
[12:56:43] <Infin1ty> I am running this procedure now over 7 shards, got tons of freespace to claim :), i just resync, seems better
[12:57:59] <h0bbit> Derick, Infin1ty the compact docs say that it should be run on a collection. what if I've dropped a large collection and want to reclaim the space?
[12:58:14] <Infin1ty> "compact requires a small amount of additional disk space while running but unlike repairDatabase it does not free space on the file system."
[13:02:36] <remonvv> Derick, do you know what the process is to report an issue that involves logs that cannot be public?
[13:03:48] <remonvv> Derick, this isn't support. And I've noticed someone referring to Community Private as a way to do so for public issues (e.g. report issue public, but keep logs private)
[13:03:53] <Foxandxss> Hi, I said no to me. I am doing 10gen course and I really need a hint, just 1 ;( :P
[13:04:05] <remonvv> And to be honest I think it's about time 10gen starts paying me rather than the other way around ;)
[13:05:31] <remonvv> So I report public first without logs and you move it and the description to private?
[13:07:09] <Infin1ty> the slowest thing in the resync/repair is building the index after, seems that is sort twice or something, it takes ages on large collections, it's seems as it's only single-threaded
[13:07:37] <Infin1ty> it takes me around 6 hours to replicate a large collection, then the index building process (just one index, _id) takes almost 2 days
[13:08:52] <Foxandxss> So, if limit a query to 30 returns 30 but scan 40 and no limit returns and scan 40. How to create an index that when limit to 30, just scan 30? Hints welcome! :P
[13:29:34] <greenmang0> Infin1ty: what do you think about this approach - add one more member to replset, it will sync and will occupy less disk space, then make it a primary, then remove secodary members one by one, delete all the data on them, and add them back to replset
[14:20:38] <limpc> hi guys. im having trouble getting mongo to work with php. im using the apt repository (ubuntu) to install. everything installs fine, no errors, mongod is running, but when i try to connect to the db via php, it just hangs there, and there's no connection entry in the mongo log
[14:24:02] <Derick> limpc: and, which version of the php driver do you have installed (check the output of phpinfo())
[14:33:49] <kevinsimper> Should you always check if a key inside a document exist because you cant be sure? Or should you do a db.col.find().foreach and ensure all docs has a least a value?
[14:36:19] <kevinsimper> What do you do when you make new functions? Do you then update the old documents?
[14:37:52] <kevinsimper> You have 100 old documents that you already created, they have a "title" and "content"
[14:38:59] <kevinsimper> Now you made a comment sections, so now when you create new documents they have "title", "content", "comments", "comments_count"
[14:39:29] <Nodex> well OLD documents wont have the comments nor count so why waste space adding them as keys?
[14:39:32] <kevinsimper> You now have 100 old documents, which does not have the "comments_count" field.
[14:40:21] <kevinsimper> Do people A: if(comments_count != undefined OR B: db.content.find().forEach()
[14:40:22] <Nodex> mongo is not like *SQL - you don't have to define a field to be able to add it in the future
[14:43:02] <kevinsimper> Do people A: if(comments_count != undefined OR B: db.content.find().forEach()
[14:43:37] <Nodex> People do neither bcause it's not needed, if YOUR DRIVER requires it then put your own checks in
[14:43:48] <rspijker> kevinsimper: why not just do db.content.find({"comments_count":{$exists : true}}).forEach() ?
[14:47:03] <kevinsimper> rspljker: Mongodb is as you know schemaless, but either you update the old documents or you make a check in your code.
[14:48:00] <Nodex> if you don't know the answer already, you don't deserve to have a job that relates to databases
[14:48:21] <Nodex> if you think that foreaching every time to check for somehting is efficient then god help your boss
[14:48:37] <rspijker> in some cases you might have some field that doesn't have a clear default value
[14:49:08] <rspijker> then adding it to "old" documents with a null value seems the most logical approach
[14:49:52] <Nodex> space vs efficiency = the choice is the developers... for me efficiency is #1 priority, not the same for everyone thougbh
[14:51:10] <kevinsimper> rspijker: That is how i am doing it now, setting a default value on old documents,
[14:53:18] <rspijker> space considerations are also somewhat relative… what happens when things are moved/updated, you get fragmentation, etc. Especially when the amount of old documents is relatively low I would just update them...
[14:53:51] <kevinsimper> rspijker: Okay, that was just was i was looking for :-) In MySQL I would just set the default value to zero in the schema, but i know MongoDB is different.
[14:54:00] <rspijker> it will improve the readability of your code, make things easier to manage, with (imo) very little downside
[14:56:06] <rspijker> it depends heavily on what is important for you. If you need every byte and things aren't going to change much, do some checks in code. If space isn't as much of an issue for you and you prefer code readability, add the field...
[14:56:33] <kevinsimper> Or would you? I am just interested, because from MySQL where i am coming from it was not something i had to deal with, because it was just how it was with a schema
[14:57:38] <limpc> i cant seem to get a db.insert to work. what am i doing wrong? http://pastebin.com/Y817j7er
[14:58:28] <kevinsimper> rspijker: Thanks for your input :-) Much appreciated hearing how you are doing it with MongoDB
[15:00:53] <lxnay> src/mongo/client/sasl_client_session.cpp: In function 'mongo::Status mongo::{anonymous}::_mongoInitializerFunction_CyrusSaslAllocatorsAndMutexes(mongo::InitializerContext*)':
[15:00:56] <lxnay> src/mongo/client/sasl_client_session.cpp:77:28: error: invalid conversion from 'void* (*)(long unsigned int)' to 'void* (*)(size_t) {aka void* (*)(unsigned int)}' [-fpermissive]
[15:01:35] <lxnay> it looks like size_t is defined as unsigned int while somebody thought it's always long unsigned int...
[15:22:42] <JoeC1> If I was to use a mongo collection to store session data, after issuing an update to a record, can I reasonably assume it will be available for immediate read?
[15:24:17] <rspijker> so when you go db.auth.insert(… it will try accessing the insert element of the auth function and not of the db
[15:28:06] <rspijker> exactly, it's a function. You can't do function.insert(…), which is why it fails
[15:30:04] <rspijker> try doing db.eval.insert(…) or db.getName.insert(…). You'll get the exact same error ;)
[15:41:26] <dllama> hi guys, i'm fairly new to mongo, pretty sure that when i started it up, i only ran mongod. i didnt' specify a local folder or anything.. I'd like to restart mongo and reload it properly, with a full path to the db folder and of course maintain all my documents, etc. whats the safest way to do that?
[15:44:28] <Striki> do you have any points why 1 mongos (out of 30-40) is the source of almost all setShardVersion requests?
[15:44:36] <rspijker> just turning off mongos, moving the files to your desired dbpath and starting mongos up again with the correct dbpath should be fine, dllama
[15:45:56] <dllama> if i'm not mistaken, i think i may have just done apt-get install and then just ran it
[15:46:09] <rspijker> dllama: if you want no downtime, you can start a second mongod (with correct dbpath), configure it as secondary, wait till it syncs, then step down the original master
[15:46:42] <Nodex> edit that and it will show you the defaults it uses, change what you need, save it and do `/etc/init.d/mongodb restart`
[15:49:28] <dllama> from the running daemon, I dumped them to a folder in my home directory. mongo powers a cms that i'm using and my initial upgrade of the CMS failed to migrate the DB properly, forcing me to essentailly rewrite the entire site (mostly copy/paste but was still a tedious process). so i'm actually pretty nervous about this procedure
[15:49:59] <Nodex> you wont have lost your data. Changing values in the config does not erase the db's
[15:50:34] <Striki> do you know how I can find out what servers owns a certain serverId? the serverId is an ObjectId
[15:58:21] <Striki> { setShardVersion: "", init: true, configdb: "config1:27019,config2:27019,config3:27019", serverID: ObjectId('51bec2ff9e5fe2d77437b56d'), authoritative: true, $auth: {} }
[15:58:46] <Striki> since that mongos, that is issuing this request, is doing it all too often, I suppose
[16:38:37] <dllama> hmm, if my rails app is showing that as an error, would that a configuration error with the db or something really specific to the app?
[16:38:52] <dllama> i'm forced to use a cms here and its been giving me those errors all morning and i can't really seem to narrow down why
[16:41:54] <dllama> i just dont even know where to turn with this, either #rubyonrails or here, since the developers irc room has 3 people in it which never seem to answer :/
[16:42:38] <Nodex> the only time you should even see those words (bson dump) is when backing up or restoring a mongodb
[16:59:14] <limpc> hmm have another problem, when calling Session::instance(); i get an ' Error reading session data.' error from kohana
[17:03:26] <limpc> hmm when i print_r $e (from inside kohana's session class), it says the session was already started.
[17:03:30] <Nodex> dllama : I don't know ror enough to help you, if it was mongo or in the mongo log rather it would be better
[17:04:09] <dllama> from what i've read, that has something to do with MOPED (not sure what that is even)
[17:06:43] <dllama> Nodex, is there a way to enable more indepth logging? i'm not really getting much from mongodb.log
[17:10:58] <dllama> Thanks Nodex, seems like thats my best bet @ this point, just wish they actually were available in the irc room they advertise… i'd be furious if this were a commercial product :/
[17:11:36] <drag> Hi there, if I query mongodb with a BSON that like: {_id=,name=Joe}, while it ignore the empty _id value or will that cause an error?
[17:11:39] <dllama> went as far as adding every name/mention from their git pages on skype, and nobody has yet accepted my ivite lol
[17:13:54] <Nodex> my point still stands, mongo will not be your bottleneck for sessions, your php framework will melt before that happens
[17:17:00] <drag> Hi there, if I query mongodb with a BSON that like: {_id=,name=Joe}, will mongo ignore the empty _id value or will that cause an error?
[17:18:35] <Nodex> the php driver throws an exception if the _id is supposed to be an ObjectId and isn't, can't imagine the java driver differing much
[17:45:38] <silasdavis> db.licenses.find({ "distributions.key": "b41f7c82-756b-43c8-9435-15cff7498d15",}, {"distributions.name": 1})
[17:46:11] <silasdavis> I would like the same projection but only including the first element of the distributions array, I've tried:
[17:46:19] <silasdavis> db.licenses.find({ "distributions.key": "b41f7c82-756b-43c8-9435-15cff7498d15",}, {"distributions.$.name": 1})
[17:48:10] <remonvv> If the distributions array can be large and your requirement is to only fetch one you may want to rethink your schema.
[17:48:28] <remonvv> Meaning either do a document per distribution, or make the distribution key the field name (not recommended)
[17:48:36] <mrapple> in a map/reduce, i'm creating an array with a lot of values, with duplicates possible. should i remove duplicates in the reduce or finalize stage?
[17:49:08] <silasdavis> remonvv, ah it's not really a size issue, it's just for convenience at the command line and I wanted to project out the elements before doing a
[17:49:13] <remonvv> Not really. Field filtering is just not meant to do a lot more than exclude fields. It is not meant to be selective about which array elements ought to be returned.
[17:51:13] <remonvv> So, for example : db.licenses.update({ "distributions.key": "b41f7c82-756b-43c8-9435-15cff7498d15",}, {$set:{"distributions.$.name": "HI!"}}) works just fine
[17:51:52] <remonvv> Also have a look at $elemMatch for a somewhat more intuitive element matching if you're matching on more than one array element field.
[17:52:28] <silasdavis> it would be nice if find allowed the same semantics in projections, so you can do a sanity check with a find before an update
[17:54:37] <remonvv> silasdavis, you can, but the "fields" part of find(criteria, fields) is not meant to do that
[19:51:20] <idlemichael> i have a mongo collection that's about ~120GB large. i'm trying to export it to another mongo instance on a different server -- every time i import data from the old server to the new server, mongo seems to re-do the entire index. is that standard? is there a way to background the _id index creation, is that safe?
[19:54:08] <awpti> Could use some advice, doing a little toy project to learn Mongo better. Is this way of storing this data going to bite me in the rear? http://pastie.org/private/lxrgkrroig8jzgfvsrbg
[20:00:16] <remonvv> idlemichael, there is and it's safe but you'll have to start it prior to the import
[20:00:29] <remonvv> And obviously data access for that period will be significantly slower for that collection
[20:01:21] <remonvv> You can create indexes in the background but I'm not sure if that works for _id indexes
[20:16:24] <Chammmmm> Hi, I had to drop a collection due of a bug in 2.4.2 for corrupted indexes.. After doing a db repair I have been able to drop it.. but not the collection is still showing up in the shard status.. with chunks distributed and everything..
[20:16:50] <Chammmmm> Will recreating the collection and reshard it fix it? or I actually need to hack the config database
[20:25:14] <aandy> hi guys, i'm in a bit of a jam. i have a fairly simple replica set of three nodes. running rs.status() shows ALL nodes as secondary (no primary). so obviously i can't reconf to change the priority, or step down. i've tried freezing two of them, but the remaining doesn't seem to attempt to become primary. do i need to set up an arbeiter or can i solve it easier?
[20:29:14] <aandy> i'm afraid i'll need another hint. it's been running like this for about a month, and with heartbeats (and low latency), so elections aren't run (successfully anyway)
[20:31:46] <aandy> oh wait, i may have confused lastheartbeat with lastheartbeatrecv. here's my conf + status: http://pastie.org/private/ubn5lr1ytix3dmo8vji7ag
[20:34:32] <aandy> yeah that helped. turns out one of the nodes could not reach two of the others (short story: tunnel mixup). thanks, retran :)
[20:39:17] <awpti> Okay, I can't figure this out and the docs don't go into it.. how do I remove an item from a doc in a doc? http://pastie.org/8143662 (I want to get rid of created_time from login_data)
[20:44:45] <retran> since you have the added copmlexity of recursive doc, you would possibly need to control for concurency in a transaction
[20:48:14] <awpti> There's gotta be a way to do it without recreating the object. I'm guessing I'm just missing this on the docs.
[21:29:52] <retran> the _id is based on time, but if it finds there's another inserted at same time it does some other magic to it to make it different
[21:30:30] <retran> i guess that would make your inserts potentially perform table scans just to complete an insert
[21:31:51] <retran> if it was a hash of the data it would mean the user would always have to insert unique documents
[21:32:27] <retran> the mongodb devs would know, but i read it's most likely to be the time of insert
[21:34:50] <retran> anyway, that doesn't change that having _id be background indexed would be a weird idea
[21:35:09] <retran> since it could potentilally cause table scan (negating the speed of insert gain from background indexing)
[21:35:31] <idlemichael> retran: the reason i'm interested in this is -- i have a ~120GB mongo collection that I'm trying to replicate to another box. everytime i do a mongoexport --host remote.host | mongoimport it re-indexes the _id, and blocks
[21:35:37] <jblack> I don't believe it's indexed in the "background". I believe there's an app wide write lock
[21:38:22] <retran> idlemichael, are you certain that mongoexport is suitable tool for replicating such a large data set
[21:38:29] <jblack> I wish I could remember the vodcast, but if I remember right, there was a way to disable indexing
[21:41:48] <idlemichael> retran: mongoexport seems fine, may be it's not suitable -- i just assumed 10gen would provide suitable tools to work with their data
[21:42:15] <idlemichael> retran: mongoexport doesn't seem to trigger the reindex either, it seems to happen from within mongod
[21:43:23] <retran> i've done so in mysql, and mysql's equivelent isn't suitable for such large data sets
[21:43:57] <idlemichael> indeed retran; i tried doing the replicaset method and having a slave catch up to the master, but it never seems to catch up
[21:43:58] <Chammmmm> Question.. if I stop the balancer.. are there other ways that the config servers meta data can be modified? I want to know if I can restart the config server after a dump.. while the rest of mongo is being snapshotted
[21:45:07] <retran> i havn't impelmented a replication set yet in mongodb, but i have migrated a mongodb server's data via tarball of data dir
[21:47:02] <idlemichael> retran: that's the problem. primary is production right now -- stopping it is almost out of the question, but it might come to that
[21:49:15] <idlemichael> "If you want to simply copy a database or collection from one instance to another, consider using the copydb, clone, or cloneCollection commands, which may be more suited to this task. The mongo shell provides the db.copyDatabase() method."
[21:51:50] <retran> that looks like they mean for it to work well if it's part of the native functionality
[22:39:57] <jblack> If you look up the examples at mongoid.org (http://mongoid.org/en/mongoid/docs/querying.html), they're clearly passing in a type String.
[22:45:32] <crudson> hahuang65: IDs don't have to be ObjectID. For you two examples to be compared to each other you should be doing ObjectID.from_string('abc'), as that is what should check whether it's correct.
[22:46:20] <hahuang65> crudson: but in the old version of mongoid, it would automatically do that for me. Now I have to do it myself?
[22:47:25] <hahuang65> crudson: in Mongoid 2.x, if you do a find where it expects an ID, and that's not a valid format, it would throw InvalidObjectId… now it doesn't. You're saying that I have to check that myself before I pass it to .find or any other query where I use an ObjectId?
[22:47:54] <idlemichael> retran: db.cloneCollection seems to be doing the trick -- i can query and access the db
[22:50:38] <crudson> hahuang65: have you overridden _id's type in the model? If not, how about? field :_id, type: Moped::BSON::ObjectId
[22:54:07] <hahuang65> crudson: I don't think that's declared anywhere… Mongoid::Documents should handle it no?
[22:59:04] <hahuang65> crudson: looking at the code in mongoid, where it tries to call .mongoize(id) it's calling that on Mongoid::Fields::Standard, when I'm assuming it should be Moped::BSON::ObjectId
[23:01:18] <hahuang65> crudson: never mind, Mongoid::Fields::Standard delegates :mongoize to it's type. Lemme see where that leads.
[23:09:50] <hahuang65> crudson: thanks for your help. I've found the cause. It seems like an oversight on their part… ping me later if you want details.
[23:22:40] <poet> I am attempting to call rs.initiate() from the mongo shell and I get an error "no such cmd: replSetInitiate"
[23:48:06] <tyl0r> If you stop/shutdown mongodb properly will it flush to disk immediately? ie. Can I start taking a snapshot as soon as the shutdown returns? Or should I run db.rsyncLock() before shutting down? I'm having issues using fsyncLock/Unlock with authentication so I'd prefer not to use that
[23:49:13] <retran> dont know, i've always had good luck with my datadir snapshots, and i never wait any period of time or do have a particular command to run before the shutdown