Log file Viewer
#mongodb logs for Monday the 19th of May, 2014
[00:00:18] <phrearch> im trying to figure out the best way to update a counter field on a collection
[00:00:34] <phrearch> like i have an order field, and after i remove 1 item, i need to rebuild the order index again
[00:04:40] <phrearch> something like this, but i have to figure out how to wait async on all updates, or do all updates in one query(which i would prefer if possible)
[00:47:25] <__mark__> how do I issue a "slaveOk" command via runCommand or $cmd.findOne({})? I keep getting "no such cmd: slaveOk". The shell command is "rs.slaveOk()" but I'm a database-driver author, and I need to figure out how to do this via the wire spec.
[00:49:06] <__mark__> for example, db.$cmd.findOne({dropDatabase: 1}) is synonymous with db.dropDatabase()
[03:53:01] <chovy> does anyone know of a way to run data migrations on a db? Like add some default attributes to existing documents?
[04:03:31] <ranman> anyone here an AWS expert, with EBS? is it better to ask for a few tiny volumes or one larger one?
[04:05:43] <chovy> ranman: is there any tools/npm modules for this? All I could find was db versioning stuff. I'm also using Mongoose too.
[04:06:23] <ranman> chovy: you mean a stored migration? I typically write that in a script and throw it in a folder like /migrations, you could do all of this with mongoose
[06:11:31] <netQt> Hi all. I'm using nodejs to insert data int mongo(no external lib like mongoose). All my integers are being saved as double, how do I set type to int? Now I'm using mongo's Long, but it stores my numbers as NumberLong object.
[06:45:42] <spychalski> so i have an ObjectId type in my schema, is there a quick way of accessing that type's attributes?
[06:46:11] <ranman> netQt: "By default, the mongo shell treats all numbers as floating-point values. The mongo shell provides the NumberLong() wrapper to handle 64-bit integers."
[06:49:54] <spychalski> i'll try again. my schema has a category with type ObjectId. category is another schema which has, say, a name. i want to be able to do 'mainschema.category.name'
[06:50:42] <netQt> I can use NumberLong from mongo shell, and it works perfectly, but I can't find that in the node's mongo lib. It has Long, but it doesn't store correctly.
[06:51:53] <netQt> mongo.Long(new Date().getTime()) this breakes the timestamp, it doesn't store correctly
[06:56:01] <netQt> that's why it cannot store my timestamp, I need to somehow specify NumberLong that works in mongos shell
[06:58:02] <ranman> spychalski: that's because mongodb is not a relational database, if you want to fetch two documents, you do two queries, there is a concept of a DBRef but I would avoid that
[06:58:30] <ranman> spychalski: have you considered embedding the information you do need in that document along with the objectid reference?
[06:58:39] <spychalski> ranman: should I instead set my type to the category schema itself then? i had issues with uniqueness doing that earlier
[06:59:33] <ranman> spychalski: you're being a little to data specific for me to give a recommendation sorry :(, could you throw it in a gist/pastebin and maybe explain what your end goal is as well? just don't want to give you bad advice
[07:09:27] <ranman> ichetandhembre: maybe read the docs? http://mms.mongodb.com/help/tutorial/install-monitoring-agent/
[07:13:58] <ichetandhembre> so i can allow security group of agent to access my monogodb port on server
[07:19:39] <spychalski> ranman: apparently my code was wrong and thats why it wasn't working. sorry =(
[07:20:33] <ranman> spychalski: I purchased a rubber duck that I explain all of my problems to, sometimes that helps, sometimes your coworkers just mutter about the crazy guy talking to the rubber duck
[07:20:35] <spychalski> I was using mongoose, creating a schema (category) and later referencing it as a type in a 2nd model. it should work
[07:38:33] <ichetandhembre> in MMS which graph i should keep eye on to make sure every thing is working fine .. what these graph means ?
[07:42:23] <ichetandhembre> joannac: in MMS which graph i should keep eye on to make sure every thing is working fine .. what these graph means ? is there any doc about it ?
[07:45:16] <ranman> in clusters I typically watch memory usage, opcounters, lock %, queues, background flush average, and replication stats
[07:47:00] <ichetandhembre> ranman: thanks i would like to know more about these graph .. is there any doc explaining which and why graph is important ? srry for such naive question i am newbie in mongodb management stuff
[07:48:12] <ranman> ichetandhembre: this video might help http://www.mongodb.com/presentations/performance-tuning-and-monitoring-using-mms
[07:48:40] <ranman> ichetandhembre: mongolab has a good article: http://blog.mongolab.com/2013/12/tuning-mongodb-performance-with-mms/
[08:05:27] <spychalski> if i set unique in a collection, every other collection that references that first also become unique
[08:07:32] <spychalski> ok, help me out. i made a schema called cat { name: { type: String, unique: true } }
[08:09:14] <spychalski> because i think i can make this work in mongo but not using mongoose somehow
[08:18:37] <Zelest> what's the best way of making _id an int and have it auto-increment, can this be done automagically?
[08:18:37] <spychalski> see, thats what i assumed. like you can't create a category with the same name twice.
[08:19:18] <Derick> spychalski: *that* is what that index enforces, but it has nothing to do with your insert to your items collection
[08:19:38] <Derick> Zelest: http://docs.mongodb.org/manual/tutorial/create-an-auto-incrementing-field/
[08:19:58] <spychalski> yes but since items references categories, it becomes unique too, for some reason.
[08:20:06] <Zelest> Derick, oh, also, do you have any words on the "no candidate server" issue I had? like, it stays like that until I restart php-fpm..
[08:24:27] <spychalski> it seems just storing the ObjectId of the category would be the best way of doing things.
[08:25:28] <Derick> Zelest: but the driver will find a new primary on its own, it just takes a while (and has to do it for each php-fpm process)
[08:26:47] <Derick> http://docs.php.net/manual/en/mongo.configuration.php#ini.mongo.is-master-interval
[08:33:44] <Zelest> silly question, I created a function in the mongoshell.. can I use this function everywhere now? as in, are functions like "stored proceedures" or are they just local inside my shell?
[08:44:38] <Zelest> if two nodes hit it exactly the same, findAndModify will lock and return two different values?
[08:46:55] <Zelest> the wrong way to do it is "find the highest _id from collection, increase it with 1.. and use that"
[08:50:29] <freeMind> i tried c.ensureIndex("tutorial.testBD",mongo::fromjson("{\"$**\": \"text\", \"name\":\"textIndex\"}"));
[09:30:51] <Azitrex> in NoSql databases we dont have "LIMIT" option for get items limited by quantity ?
[09:38:18] <Azitrex> is true , i find this and worked as fine , but today we have a new NoSql DB (Perst) with JSQL !! http://www.garret.ru/jsql/readme.htm i dont know how can i to limited result set !
[09:39:44] <jdj_dk> not that we're not interested in helping.. But the change of us knowing the answer is very small
[09:40:36] <Azitrex> yes i know , i think if this is a nosql familiar and stractures is same as mongo
[09:41:25] <rspijker> from a quick look at the grammar, it doesn’t look like there is a limit in there
[10:42:30] <rspijker> kas84: not as such… You can mount a different volume on the journal dir though
[14:16:48] <[EddyF]_> is anyone looking for work in London,UK? http://careers.stackoverflow.com/jobs/50317/php-developer-rocabee?a=ZmJB2vAI&searchTerm=PHP
[14:19:27] <rspijker> perhaps some more specifics about your collection, the documents inside and the query you are running
[14:21:06] <ekristen> .find({ state : 'new', ts : { $lte : 1400501853000 } }).sort({ recordedAt : 1 }).limit(100)
[14:28:42] <ekristen> .find({ state : 'new', ts : { $lte : 1400501853000 } }).sort({recordedAt:1}).hint({ state: 1, ts: 1, recordedAt: 1 }).limit(100).explain()
[14:30:37] <rspijker> from what you told me about the index you are using, scanAndOrder should be false…
[14:46:49] <rspijker> also, it shouldn’t matter unless you have some weird definitions in your active shell, but try wrapping fields names in quotes
[14:47:33] <rspijker> so… .find({“state”:”new”, “ts”:{$lte : 1400501853000}}).sort({“recordedAt”:1}).limit(100)
[14:51:18] <rspijker> hmmm, so apparently it’s the sort that it can’t do with the index for some reason. Which brings the limit out of the sort as well and you’re sorting 17million records instead of 100
[15:05:41] <rspijker> ekristen: ugh, it’s because the query on ts is not an equality comparison but a range
[15:07:11] <rspijker> “The sort document can be a subset of a compound index that does not start from the beginning of the index. For instance, { c: 1 } is a subset of the index { a: 1, b: 1, c: 1, d: 1 } that omits the preceding index fields a and b. MongoDB can use the index efficiently if the query document includes all the preceding fields of the index, in this case a and b, in equality conditions.”
[15:40:41] <fewt> Quick question, is the 2.4.x OSS edition still being updated now that 2.6 is available?
[15:41:02] <dgarstang> Anyone awake? I'm trying to deploy 2.6. The docs say in 2.6 the config switched to yaml but it's backwards compatible. This seems to be untrue. Mongo complains if the config file is not in yaml format.
[15:43:51] <dgarstang> wonder why the docs say 2.6 is backards compat with the config file when it's not.
[15:45:21] <dgarstang> rspijker: Correct. If I deploy 2.6 with non yaml config file format, it complains. For example it complains about the 'nojournal' entry and then a few others
[15:49:40] <rspijker> dgarstang: I don’t know about all the options. The general format still works though. I’m not having any issues with a test deployment. Mind you, I’m not using the nojournal option.
[15:50:06] <rspijker> but if some of the options aren’t working, that does mean it’s not entirely backwards compatible
[16:09:35] <rspijker> well, like I said, smallfiles works fine here… Are you sure you don’t have other errors?
[17:09:00] <dgarstang> Why? .... "Starting mongos: Error parsing INI config file: unknown option nojournal"
[17:09:50] <dgarstang> And googling "Error parsing INI config file: unknown option nojournal" gets me nothing AT ALL
[17:17:18] <dgarstang> Something is very screwed up with mongos. I modified /etc/mongodb.conf to the yaml file format and now I'm getting "Unrecognized option: storage.journal.enabled".
[17:22:01] <kongthap> hi, i'm trying to setup authentication, i can use db.auth(), i can display admin.system.users content, but why i cannot use "show collections" it returns error (more: http://pastie.org/9190172), windows 7, mongodb 2.6.1
[17:34:53] <dwstevens> if i want to update some statistics fields on a document after save, can I do so with a post save hook? Specifically I want to summarize one of the arrays in the model
[18:21:57] <kongthap> i'm trying to setup authentication, i can use db.auth(), i can display admin.system.users content, but why i cannot use "show collections" it returns error (more: http://pastie.org/9190172), windows 7, mongodb 2.6.1
[19:56:37] <xofer_> hello, i am trying to add a 2.4.10 node to a replicaset with 2 x 2.4.9 nodes, and it's unable to sync (falls further and further behind) -- i've pretty much ruled out hardware and network -- any ideas?
[20:03:52] <xofer_> the other odd thing about the 2.4.10 is that its cpu usage is low, around 100% on a 16-core machine
[20:06:17] <federated_life> xofer_: you sure its not just your box having cleared out its FS cache when you upgraded?
[20:09:12] <xofer_> in contrast, we rebooted a dead secondary just last week which was down for 2h and it caught up
[20:11:14] <federated_life> otherwise, you can play with journalCommitInterval and syncDelay to reduce your IO wait and flushing
[20:12:36] <federated_life> xofer_: yea, thats one the 'newer' status's when its starting up…whats your network throughput like?
[20:12:59] <federated_life> also, something about 2.2+ is that it only clones from the primary…in 1.8, you could have it pull the initial sync from secondaries
[20:13:25] <jiffe98> I have a replica set with 3 nodes and I need to take one of them offline but when I do that, the 3rd node which is primary turns into a secondary and there's no primary nodes
[20:14:29] <MacWinner> is it possible to just called .update() on results from .find() eg… db.testdb.find().update(…)
[20:15:15] <federated_life> xofer_: its unlikely that such a minor revision did anything unless you see an error in rs.status or the logs. you can increase verbosity of the logs .. db.adminCommand({ set : paramter : 1 , logLevel : 3 })
[20:16:56] <dgarstang> okiday, what causes "about to fork child process, waiting until server is ready for connections" when starting mongod?
[20:17:37] <xofer_> federated_life: i see it's pushing a lot of ops: replSet initialSyncOplogApplication applied 34405073 operations
[20:18:49] <jiffe98> federated_life: that's what I said I needed to do, I need to take two of them down for repair and leave the 3rd running
[20:20:33] <jiffe98> reading back I didn't say that, I'm sorry, I meant I needed to take two of them offline, one was already offline so I needed to take one more offline
[20:20:47] <dgarstang> omg, this is so confusing. What config files do mongod and mongos use? mongodb.conf? mongod.conf? WHich one uses which!!?!?
[20:21:40] <jiffe98> the reason I need to take two of them offline is because one of them is corrupt so I need to sync the other secondary to it
[20:23:01] <federated_life> in older versions, like 1.8 and 2.0, you'd have to step it down from 60 to like 30 to 15 to 5
[20:23:02] <dgarstang> these docs http://docs.mongodb.org/manual/reference/configuration-options/ don't say which process uses which config file....
[20:23:50] <federated_life> xofer_: or…you can change which member you are receiving replication events form
[20:24:49] <dgarstang> What config files do mongos and mongod use? 'man mongos' and 'man mongod' also don't specify.
[20:25:00] <emgram> hi, I have a question about how to make a certain query efficiently. is it appropriate to ask that here?
[20:25:10] <federated_life> xofer_: http://docs.mongodb.org/manual/tutorial/configure-replica-set-secondary-sync-target/
[20:25:47] <federated_life> jiffe98: you need to fix your replica set…you can also rs.reconfig( {},{force:true})
[20:26:04] <dgarstang> xofer_: I'm installing this with a chef cookbook which is having issues, so I'd rather not reply on what's deployed. it might be wrong. it's written for 2.4 and stuff changed in 2.6
[20:26:41] <dgarstang> now, if there was just some docs, that you know, said what config file what processes used, that would be a helpful starting point
[20:26:43] <xofer_> dgarstang: well on my systems (ubuntu) the config path is specified in the command
[20:27:25] <dgarstang> federated_life: really? because certain options like smallfiles, nojournal mongos complains about
[20:28:53] <emgram> federated_life: I have a set of chats with topics and want to only ever have the 20 newest topics stored. thus if there are 21 topics I want to find the least used topic and delete every chat with that topic.
[20:28:53] <dgarstang> federated_life: if there's options that mongos doesn't like, seems to make sense there'd be more than one config file
[20:28:54] <federated_life> but they all go in the one config file…unless you add them to the startup string
[20:29:18] <emgram> federated_life: I have it working with javascript but thought there might be a pure mongo way to run it
[20:29:24] <dgarstang> federated_life: ok, any idea why mongo doesn't like smallfiles or nojourmal then?
[20:29:55] <dgarstang> federated_life: setting smallfiles in the config file parsed by mongos gives me an error
[20:31:51] <dgarstang> federated_life: I had to remove smallfiles and nojournal or else mongos would not start
[20:32:14] <dgarstang> there's also no mention of 'smallfiles' at http://docs.mongodb.org/manual/administration/configuration/
[20:33:29] <dgarstang> federated_life: Well, I removed those to get it to work. now I got new issues, but the whole config file thing.
[20:34:41] <federated_life> xofer_: yea, so with that extra verbosity, you see stuff … like, you can see the journal files getting created and removed, etc etc the repl write workers are, I think, the batch replication applies
[20:41:40] <federated_life> dgarstang: Id recommend, from experience, not to disable the journal, but instead, to put it on a second raid volume [ ie ssd's ] …since Im guessing you want to reduce IO
[20:44:21] <dgarstang> federated_life: How about now? http://pastebin.com/gkSJKacg has 'nojournal = false' but it doesn't like that
[20:45:57] <dgarstang> federated_life: you also suggested later I don't disable journing, hence nojournal = false
[20:46:40] <dgarstang> this would also suggest https://github.com/edelight/chef-mongodb/issues/289 that mongos doesn't like nojournal'
[20:47:25] <federated_life> conceptually, mongoS only routs traffic…you would only need a journal if your worried about data consistency…not in the case of routing
[20:50:13] <federated_life> dgarstang: you need to learn whats the different processes…mongoD vs mongoS
[20:50:42] <federated_life> dgarstang: probably, you should just use mongoctl ..its python scripts from mongoLabs…super easy and fast for mongo
[20:52:11] <dgarstang> oh come on. now mongod is reporting "Error parsing INI config file: unknown option configdb" when starting... pointing to that ONE config file that they allegedly all use
[20:53:44] <kongthap> if the first added user has userAdminAnyDatabase, this user has full power to the database right?
[20:53:56] <kongthap> if the first added user has userAdminAnyDatabase, this user has full power to the database right?
[20:54:00] <dgarstang> federated_life: trying to set up sharded cluster router. I thought it ran mongos and mongod.
[20:54:37] <federated_life> dgarstang: probably you;ll benefit form going to an office hours / meetup
[20:55:44] <dgarstang> federated_life: just basing it on this cookbook, which seems to start mongod and mongos on the router.
[20:56:00] <dgarstang> lots of people have used/forked it, so it's gotta be pretty close you'd think
[20:58:14] <dgarstang> certainly seems to want to connect to something local "WARN: Could not connect to database: 'localhost:27017', reason Failed to connect to a master node at localhost:27017"
[21:25:14] <abonilla> Hi, where is the documentation for mongodb.conf so that I can start a DB and its set already for replication?
[21:27:09] <emgram> how do I use these results to delete all chats that match any of these _ids? http://pastebin.com/LShsjN5J
[21:28:09] <emgram> it is an aggregate result that returns a list of topics that are candidate for deletion
[21:31:11] <dgarstang> Confused. I have 3 config servers and a running mongos. how do I connect to mongos since mongod isnt running on that box?
[22:14:40] <dgarstang> Weeee. What does "could not verify config servers were active and reachable before write" mean?
[22:19:06] <jimpop> any idea on why FTS for "red white -blue" would take 3 times longer than for "red white"? It seems that negating a few results should be fairly easy.
[22:25:11] <dgarstang> Ugh. "Error: nextSafe(): { $err: "error creating initial database config information :: caused by :: DBConfig save failed: { ok: 0, code: 25, errmsg: "could not verify config servers w...", code: 13396 } at src/mongo/shell/mongo.js:146"
[22:30:37] <palominoz> hello there. any good ideas on how to convert an HTTP GET to a mongoose/mongo query with javascript? sorting, population, and quite advanced queries are required
[22:31:09] <dgarstang> mongos says it starts fine but then trying to connect gets me "ould not verify config servers w...""
[22:32:17] <palominoz> @ranman in some way yes. but the main use cases i would like to support are like products?manufacturer=some-mango-id&sort=-price
[22:33:44] <palominoz> @ranman for those not trivial searches it may be a good idea to support things like products?query={ some: { mongodb: 'thing' }}
[22:34:21] <palominoz> @ranman but at the moment i have no idea about escaping and securing things on mongodb
[22:34:58] <ranman> palominoz: only thing you have to watch out for is $where, rest of the stuff is a non-issue
[22:35:50] <palominoz> ranman: is there any good node library you know that could help this sort of things?
[22:35:50] <ranman> palominoz: if you have the parameters I think it's pretty easy to create a method that will generate a query
[22:36:26] <ranman> palominoz: I'm pretty sure this is easy to write though. Let me take a stab at it.
[22:37:29] <NaN> you guys, do you know a good plugin for Vim to execute mongo shell commands (in vim)?
[22:38:17] <ranman> palominoz: if you're passing in raw query strings you want to make sure you parse them into json and strip out $where first
[22:38:58] <ranman> palominoz: but that's seriously the only escaping concern, very different from SQL injection
[22:41:30] <ranman> palominoz: don't allow them, too dangerous, who wants to execute arbitrary javascript anyway? if someone wants to do that a DBA or some such person should approve it.
[22:41:49] <ranman> palominoz: literally worked with 1000s of applications, infrequently does someone need where
[22:41:51] <dgarstang> What might this mean? "create ns_1_min_1 index on config db: could not verify config servers were active and reachable before write"
[22:42:14] <ranman> dgarstang: sharded cluster ... it seems like your config servers aren't reachable?
[22:43:04] <dgarstang> ranman: they are and the logs indicate they are. I just noticed that it logs "config servers and shards contacted successfully" AFTER above error
[22:44:15] <dgarstang> ranman: version 2.6.1, thats logged to /var/log/mongodb/mongodb.log on mongos node
[22:44:55] <dgarstang> it's logging the error at 076 and the connection is at 102... jeez... 76 (milliseconds) to contact all config server isn't fast enough!?
[22:47:33] <dgarstang> ranman: here's the log... http://pastebin.com/RBF7gFFM this is a new install... so i've only gotten as far as the config servers and the routing server so far
[22:51:18] <dgarstang> Ugh. Searches for the error "could not verify config servers were active and reachable before write" are getting me C++ code. That's never a good sign
[22:52:16] <dgarstang> thanks. running the shell gets me "2014-05-19T22:51:04.651+0000 Error: nextSafe(): { $err: "error creating initial database config information :: caused by :: DBConfig save failed: { ok: 0, code: 25, errmsg: "could not verify config servers w...", code: 13396 } at src/mongo/shell/mongo.js:146"
[22:52:44] <ranman> dgarstang: also sorry to do this but I have to run to a meeting, I'll check back again later tonight
[22:55:14] <dgarstang> This is in Amazon too, but I doubt that's an issue. 76 milliseconds is pretty fast to give up
[23:05:12] <federated_life> dgarstang: you need to add a shard before you can write any data to the clsuter
[23:05:25] <federated_life> sounds like you have your config servers and shard routers…but no where to put the data
[23:06:10] <dgarstang> federated_life: is that definitely required? was hoping, thinking, based on what I read, that I could set up the config servers first, mongos, verify that's working and then move on
[23:09:49] <dgarstang> I'm following the oreilly book here. it walks through the sharding setup. If they haven't configured sharding yet, what's their mongo shell connecting to? one of the config servers?
[23:10:07] <federated_life> dgarstang: conceptually, you have your config servers that hold meta data, then you have your routers [ mongoS ] that router requests adn keep the config data in memory…then you have your mongoD's that are replica sets and comprise a shard. so, each replcia set is a shard
[23:10:32] <federated_life> dgarstang: perhaps you did not specify the config server URI when starting the mongoS?
[23:11:10] <dgarstang> it sure looks like in the oreilly book like they are connecting to mongos with mongod before they've configured sharding
[23:11:35] <dgarstang> ie because they tell you to run the sh.addShard() command. gotta have a shell to run it on
[23:11:46] <federated_life> dgarstang: you have to understand, you cannot connect to mongoS with mongoD
[23:12:19] <federated_life> when you start up your mongoS process, you need to specify the config servers , configdb=host1:27019,host2:27019,host3:27019
[23:15:53] <dgarstang> not seeing any specific instructions on sharding at http://docs.mongodb.org/manual/core/sharding-introduction/
[23:18:06] <dgarstang> http://pastebin.com/9trKR7g4 ... so if the reason it's reporting "could not verify config servers were active and reachable before write" BEFORE it even tries to connect to the config servers is due to sharding not being configured, then ok.
[23:19:51] <dgarstang> The guide at http://docs.mongodb.org/manual/tutorial/deploy-shard-cluster/ CLEARLY says to set up the config servers first followed by mongos. Nothing about setting up sharding yet.
[23:23:09] <dgarstang> The guide at http://docs.mongodb.org/manual/tutorial/deploy-shard-cluster/ CLEARLY says to set up the config servers first followed by mongos. In their example, once the config servers have been set up, you should be able to use 'mongo' to connect to the local 'mongos'.
[23:37:51] <dgarstang> Don't think thats necessary. Starting mongos process returns "Starting mongos: warning: bind_ip of 0.0.0.0 is unnecessary; listens on all ips by default"
[23:40:04] <dwstevens> dgarstang: can you confirm that you can connect to your config servers from that host?
[23:45:03] <dgarstang> dwstevens: i restarted mongod on the first config server... got this... http://pastebin.com/Hg33ZUbv
[23:48:37] <dgarstang> wtf? I stopped and restarted all three config servers, which I had done at least ocne already, and restaretd mongos router and now I can connect with mongo. Ugh!
[23:49:16] <dwstevens> nah, probably just making a lot of changes, and simple little mistakes get in the way
[23:49:52] <dgarstang> dwstevens: Thanks for your help. I'l probably be back in 30min. :) federated_life, ditto
[23:50:02] <dwstevens> if you look back in your logs on the configdbs before the restart you may have some answers to what was going on