Log file Viewer
#mongodb logs for Thursday the 8th of November, 2012
[00:00:13] <rossdm> is it useful to be able to grab a specific URL? then maybe key-value pair where URL is value. If not, perhaps use an array?
[00:01:40] <spartango> its generally not all that useful to grab a specific url, although it'd be useful to have information about the ordering
[00:02:43] <spartango> note that the appends have no guarantee of being done in order (concurrent) so i'd be doing some kind of "put at position" with an array
[01:50:37] <cjhanks> ckd: First one I tried was Fedora. There was a ticket marked resolved about the issue with other people still claiming issue persisted -- I did not want to add noise. Debian installer I didn't bother since I assumed it must be endemic.
[01:51:10] <ckd> cjhanks: Interesting, thanks for the info… I'm so used to living in Ubuntu-land I completely forgot that there's a world outside :)
[02:29:57] <thomas___> rather then running users.find({gender:"male"}).count(); and getting 231 then running users.find({gender:"female"}).count(); getting 416 something that could return {male:"231",female:"416"} I know map reduce can do this, is that my best bet?
[02:45:42] <IAD> thomas___: look at http://www.mongodb.org/display/DOCS/Advanced+Queries#AdvancedQueries-{{count%28%29}} : a = users.find({gender:"male"}) && all = a.count(true) && m = a.count() && f = all - m
[03:01:05] <IAD> thomas___: this will work only if you have a men and a woman in collection(not gay, lesbian, ...) =)
[03:32:35] <thomas___> I just updated my database via homebrew, when I got back into the console all my dbs were gone. Are they still on my computer somewhere? Or are they whiped?
[03:41:36] <thomas___> I don't remember what the data directory was or if it changed from verison 2.0 to 2.2
[03:49:37] <thomas___> ckd : I'm not in linux, i'm on mac and installed with homebrew so my conf isn't in etc
[04:07:59] <ckd> does it look like your new install IS in fact using a different data dir than your original install?
[04:08:45] <ckd> yeah, shut her down, and replace your new dir with a copy of the orig and start it up again
[06:28:37] <samurai2> hi there, is making backup using mongodump for shard replicaset cluster is better than using lvm? thanks :)
[07:08:02] <kizzx2> insert query update delete getmore command flushes mapped vsize res faults locked % idx miss % qr|qw ar|aw netIn netOut conn
[07:08:23] <ckd> well, someone more informed can chime in, but basically… it just means that it's having to read from disk
[07:08:36] <ckd> so if you're ok with that and it's not visibly affecting performance, i wouldn't worry about it
[08:58:49] <codemagician> I'm using the PHP $obj = new \Mongo(); in my web app. I stop the mongod and I start my webapp again, without restarting the mongod but my app still gets a connection back. Is this possible?
[08:59:25] <codemagician> Is there some kind of persistence happening at the PHP level, that doesn't check to see if the mongod server has gone away?
[09:07:52] <codemagician> NodeX: is my web app going through the collection of PHP processes, or a connection pool of mongodb connections
[09:08:47] <codemagician> NodeX: is there a way to force the driver to check if the db has gone away
[09:11:30] <codemagician> NodeX: $this->connection = new \Mongo(); this line still gets a connection back regardless of selecting db
[09:11:53] <NodeX> great, but selecting a DB will throw an error if there is no database (no connection to the db)
[09:12:51] <codemagician> when running PHP to test it on the command line, as soon as the mongo server is gone, an exception is throw indicating that the connection failed
[09:13:57] <codemagician> NodeX: $this->connection = new \Mongo(); this line raises an exception so the line you are referring to is not relavant
[09:14:30] <codemagician> if I remove the selectDB() then it still gets a connection, so what's the point?
[09:16:24] <codemagician> PHP website says "When the PHP process exits, all connections in the pools will be closed."
[09:20:09] <NodeX> put a try catch on selectDB - this will Throw an exception if it can't select a DB
[09:24:12] <codemagician> NodeX: you idea of suggesting to test to see if I can select a DB from a connection that doesn't exists is dumb
[09:31:13] <codemagician> NodeX: I've been using it for 1 day, but long enough to know this is a connection pooling problem
[09:32:07] <ron> codemagician: I've been developing for 10+ years to know the problem is that you use PHP instead of a real programming language. that doesn't help solve your problem though.
[09:32:15] <remonvv> NodeX, basically all command parameters that you'd expect could be changed runtime.
[09:35:26] <codemagician> ron: listen, driver, PHP or C. The original question was about connection pooling, since I want my development server to recognise when the mongod instance exists. The PHP caches a pool of connections and this is the issue, not using workaround to attempt to selectDbs. The select works too, because PHP caches that also
[09:36:31] <codemagician> ron: PHP will not raise an exception for either attempting to create a new connection, nor select a DB because it caches both in a pool.
[09:39:55] <codemagician> Anyone else know why PHP MongoDb driver does not close the connections in the pool when the process exits?
[09:46:27] <remonvv> codemagician, most drivers will throw exceptions only if you perform an actual operation on the connection.
[09:46:41] <remonvv> Things like selecting database is local state and will not do anything to raise exceptions.
[09:47:44] <codemagician> remonvv: when I refresh my browser window again and again, the web app gets a connection every N cycles, which shows there is likely a pool of connections held inside PHP
[09:48:17] <codemagician> running it from the command line means that there is no connection pooling so the exception is thrown instantly
[09:48:36] <codemagician> the documentation says when the PHP process exists the connection is closed, but this doesn't happen
[09:51:52] <codemagician> NodeX: you don't even understand the problem. just be quiet I was programming whilst you were in nappies
[09:53:37] <stevie-bash> I once used a mongo shell command with the database name specified in it. Now I can't recall this. How is the syntax to use the db name in a db.'dbname'.collection.command()?
[09:56:22] <stevie-bash> I have the problem, that my shard config thinks, that databases exist on some shards, which are not there
[09:57:09] <IAD1> A day some of NoSQL guys walk into RDBMS bar. But they couldn't open the door! any one know why?
[09:57:40] <remonvv> like, doing something like setDB(..) or whatever the equivalent is in your driver does nothing with the server.
[09:58:33] <codemagician> remonvv: then the implementation of the PHP mongodb should disassociate the connection from the pool, but doesn't?
[09:58:45] <remonvv> If the connection has issues for whatever reason it will throw exceptions only if you actually do something that requires the driver to send operations to the server. Also note that most connection pool implementations will quietly attempt to create new connections if one timed out or had IO issues (rather than MongoDB oriented errors)(
[09:59:57] <stevie-bash> Thu Nov 8 09:57:08 TypeError: rest.trim is not a function src/mongo/shell/utils.js:1369
[10:00:13] <codemagician> remonvv: but in the case of the $obj = new Mongo(); for PHP it returns an object with a connection inside, even after the original PHP process has died, and the mongo server too.
[10:00:50] <remonvv> codemagician, I'm not familiar with the PHP implementation but the Java driver closes the connection wrapper (and obviously in that case the actual connection isn't there anymore)
[10:01:35] <stevie-bash> I found in the doc something like this db['name'] , does this refer to a collection?
[10:01:46] <remonvv> codemagician, highly unlikely given the usage. Again I'm really unfamiliar with PHP but in most drivers the Mongo object is a singleton and should not be created more than once.
[10:02:02] <Gargoyle_> stevie-bash: There's no reason for the mongo shell not to allow a DB called *
[10:03:45] <codemagician> whoops, Derick, maybe I can check with you on PHP implementation issue pleaes?
[10:03:46] <remonvv> Does PHP have global scope? If so I'm going to guesstimate your Mongo instance has to be assigned to a global variable.
[10:05:38] <codemagician> Derick: my connections appears to be living on after the PHP process has existed
[10:06:07] <codemagician> Derick: such that the exceptions are not caught even if the mongo db server goes away
[10:06:48] <Derick> codemagician: there are issues in the 1.2 release - the 1.3 series should solve quite a few things
[10:07:20] <codemagician> Derick: can you help explain something just to clear up some mystifying behaviour
[10:08:00] <IAD1> codemagician: use it before at destructing : http://www.php.net/manual/en/mongo.close.php
[10:08:09] <codemagician> Derick: when I stop the mongodb server and hit refresh on the browser, the page gives an exception, an exception, .. and then returns am apparently valid connection instance as if it's cycling through the connection pool
[10:08:55] <Derick> codemagician: possible - the pooling code is confusing. (Hence we got rid of it in 1.3)
[10:09:06] <codemagician> Derick: So before you mention a distinction between PHP request and PHP process. I assume the PHP process continues running, even when a new HTTP request occurs?
[10:11:42] <codemagician> Derick: I am correct in assuming that the PHP request isn't informing the PHP process to update it's connection state when the PHP program terminates?
[10:12:33] <codemagician> Derick: that is, currently in 1.2 the PHP script ends, but should inform its container PHP process and in tern update the connection state for that pool instance internally ?
[10:20:55] <codemagician> Derick: so there is no way for an exiting script to signal that the connection has terminated
[10:21:28] <Derick> codemagician: do you have the opportunity to try 1.3.0rc1? (or the very soon upcoming rc2?)
[10:23:47] <Derick> codemagician: IMO, that stuff should go fit in a proxy... it's very hard to do that from a apache/php application with so many distinct workers
[10:24:05] <codemagician> NodeX: why now do you care? You spent the last hour flaming me and telling me PHP isn't a real programming language?
[10:25:31] <codemagician> i've been ducking and bobbing the PHP abuse so much I've forgotten who was throwing
[10:25:43] <NodeX> Gargoyle must be the most unlucky person in the world with the number of bugs he's had
[10:26:35] <Derick> NodeX: It's exactly what type of users we (as devs) really value. People wanting to try out new shiny things in production.
[10:26:43] <codemagician> Derick: is there anyway for the mongodb to signal that it has quit back up to the PHP layer?
[10:28:35] <Gargoyle> NodeX: I only seem to get the problems when connecting to the RS. When running on dev to just a local copy of the db, it's fine.
[10:28:57] <codemagician> Derick: one last question. as a stop-gap, is there a way to force $obj = new MongoDb(); to throw an exception when the server has died
[10:31:54] <codemagician> Derick: http://php.net/manual/en/mongo.connecting.php connection pooling section. last few words of that
[10:35:00] <durre> I have a noob question: I have a document like this: http://pastebin.ca/index.php … how do I query all the documents that has "SE" in the markets array?
[10:37:32] <NinjaPenguin> Derick: can I ask how (if at all) this connection pooling is handled differently FPM
[10:43:11] <NinjaPenguin> when can we expect the article? and is there a rough ETA on the final realease?
[10:45:31] <Derick> I will fix something else today, which you probably want to use when connecting to an RS
[13:13:20] <NodeX> http://www.zdnet.com/microsoft-patent-spies-on-consumers-to-enforce-drm-7000007102/
[13:22:50] <remonvv> Microsoft has a lot of extremely smart people forced into implementing some rather bad ideas.
[13:25:17] <NodeX> this corproate mentality about always having to maximise profits really is not good
[13:31:32] <kali> NodeX: depends how big a place you consider... i've heard there is a village in france with 1 camera for 17 people
[13:32:12] <remonvv> You know. For a support channel for a NoSQL database we do tend to get horribly off topic on occasion.
[13:33:50] <remonvv> We should have a RL #mongodb meet and force everyone to wear nametags with their IRC handle
[13:36:11] <remonvv> Yes, can't help but feel somewhat happy I didn't go for my second name choice "inappropriatefondling"
[13:47:03] <durre> with this document: "filters" : { "markets" : [ "SE", "DK", "GB" ] } … how do I find all the documents that exist in the market SE or GB?
[14:27:03] <durre> using java, is there a way to see the query being generated by the mongodb driver?
[14:30:01] <rshade98> db.command("{ replSetInitiate : { _id: 'repl1', members: [{_id: 0, host: '10.202.18.48'},{_id: 1, host: '10.144.153.251'},{_id: 2, host: '10.144.15.96'}, ] } }")
[14:55:09] <rshade98> any idea why I get this: Mongo::MongoArgumentError: command must be given a selector
[14:56:54] <Derick> nopz: http://stackoverflow.com/questions/9201832/how-do-you-rename-a-mongodb-database
[14:58:00] <nopz> by chance my Database is 19 Go so it will takes only 1 or 2 hour, but it's really a problem anyway
[15:31:06] <squawknull> honestly, i think you're just going toh ave to iterate through the array and add them to a list
[15:32:06] <squawknull> ignore everything i've said as it's totally non-relevant unless you happen to be using java :)
[15:40:55] <rshade98> I am getting close, I got it to start the init, however it does not seem to like my config doc
[15:42:11] <rshade98> str="{ _id : 'repl1', members : [{_id : 0, host : ' 10.84.246.236'},{_id : 1, host : '10.190.235.118'},{_id : 2, host : '10.211.61.49'}, ]}"
[15:46:01] <rshade98> the second part of the config doc looks right. I have to have it quoted or ruby gripes. should make it into a hash and then parse it to json
[16:24:46] <jtomasrl> need some help with this plz http://stackoverflow.com/questions/13292393/find-object-inside-json-using-nested-id/13292960#13292960
[16:29:58] <Gargoyle> jtomasrl: You've not really explained what you are trying to do very well. So it's hard to follow!
[16:32:33] <jtomasrl> Gargoyle: From the songSchedule song_id's i need to thet the corresponding song name and artist from Songs
[16:36:44] <Gargoyle> jtomasrl: Are we talking lots of songs, and song Schedules? (It's looking like you might have your schema a little complicated)
[16:38:28] <Gargoyle> Well actually, considering your tob level item is an array, I'm going to suggest you have a total rethink of you schema
[16:39:06] <jtomasrl> Gargoyle: is there a guide for making this schemas? Im so used to relational DB's
[16:39:58] <Gargoyle> jtomasrl: There is a good section in the mongodb docs - I havent got the exact url.
[16:40:52] <NodeX> jtomasrl : here is a good tip, think of how you would design it for a relational DB and do the exact opposite
[16:44:25] <jdevelop> Hi all! given a list of ObjectId, I need to return only documents with those IDs and containing specific properties. How do I do that in mongo?
[16:45:56] <ckd> jdevelop: http://www.mongodb.org/display/DOCS/Advanced+Queries#AdvancedQueries-%24in
[16:47:13] <blueprints> if i run db.foo.find().sort({foo:1}).limit(int), can i be sure that sort() is applied before limit() on the collection?
[17:08:47] <eka> have a mongodb working on a 8gb machine... and is faulting... but mongostat shows that is only using 3.5gb any clue?
[17:24:52] <kali> eka: it's mapping 45g. the 3g are the process space, they don't contain the data, but the "running" stuff, threads stacks, connection, temporary stuff, that kind of thing
[17:25:38] <eka> kali: AFAIK mapping is the file size... so that doesn't map to the whole DB size... or I'm wrong?
[17:30:13] <kali> Tobsn: yep, but it's even worse in eka's case, because by sharding his data, he moved data away to other servers
[17:30:57] <kali> Tobsn: it means mongodb tries to access a memory mapped file wich is not im memory
[17:35:06] <eka> kali: running repairDB so it will do all the collections... then will see... and add shard accordingly
[17:37:38] <cyounkins> Is anyone using a PyMongo pool which does NOT create a new socket per greenlet?
[17:38:00] <kali> memory-mapped can be swapped out (and they are all the time, actually). res is for binary libraries, thread stacks, very alive stuff
[17:39:27] <eka> The amount of (resident) memory used by the process at the time of the last mongostat call.
[18:22:19] <nicobn_> hi, I'm trying to shard a collection and even if the index exists, I still get the following error message:
[18:26:11] <frsk> As the message says.. Create an index for the data you'll be using as a sharding key
[18:36:07] <jonno11> Hi - I have a node app accessing my MongoDB instance on my local machine. For some reason, when I log into mongod (shell) to the same database, the data doesn't seem to be the same as what the node instance is accessing. Any ideas as to why this is?
[18:56:31] <jonno11> Hi - I have a node app accessing my MongoDB instance on my local machine. When I log into mongod (shell) to the same database, the data doesn't seem to be the same as what the node instance is accessing. (ie. a find({}) won't return anything!) Any ideas as to why this is?
[19:03:18] <_m> jonno11: The commands "mongo" and "monogd" are quite different. You should be using the former to connect locally.
[19:05:25] <ckd> nicobn_: sorry, wasn't ignoring, you just happened to hit the rather shallow ceiling of my knowledge on sharding… your command LOOKS correct to me…. what's that last index though?
[19:05:54] <ckd> jonno11: make a pastie of the entire sequence of commands you're using in the shell
[19:08:24] <ckd> jonno11: thanks! in the shell, looks like, unless you're actually using the db "test" in node as well, you aren't switching to the appropriate DB
[20:38:25] <fatninja_> cod I found another way: obj = { xxx } , db.coll.save(obj) , obj._id is now the object id
[20:39:38] <eka> kali: found that I had 2 other collections with 10M docs each ... sharding those also
[21:15:57] <cdehaan> Hello! I don't know a ton about MongoDB, but I'm looking at using it to store data from Twitter firehose, as compared to using MySQL. What kind of performance tradeoff might I be looking at?
[21:17:00] <cdehaan> Cool. I'm thinking I might be better off with MongoDB since Twitter will be handing me JSON that I can just store as-is, instead of doing processing on it and then storing in MySQL
[21:22:07] <mspitz> May I ask a quick question about WriteResult in a sharded environment? I'm a little confused as to the format.
[21:23:49] <mspitz> And in my write result, I'll occasionally get result["updatedExisting"] = false, but the item was added to the set.
[21:24:17] <mspitz> The other thing of note is that in those writeresults, I'll also get "writeback": ObjectId("...")
[21:27:59] <crudson> cdehaan: you can treat it as json-in and json-out to a degree if you like. It's not 100% accurate, but that difference shouldn't be used as a dismissing factor for mongo if you value it in your case. Different languages will make it easier/harder more/less transparent.
[21:42:47] <cdehaan> crudson: That makes sense, I guess the only reason I know anything about is due to this Python I'm using that treats it like json-in json-out.
[22:20:37] <mspitz> Does anyone know what the 'writeback' field (an ObjectId) means in a WriteResult in a sharded environment?
[22:25:32] <fatninja_> I want to ask: if I choose to add more values after the map reduce, will they still be available in the new meta-collection ?
[22:27:56] <eka> fatninja_: http://stackoverflow.com/questions/5681851/mongodb-combine-data-from-multiple-collections-in-to-one-how
[22:28:33] <eka> fatninja_: but as said in a comment "I would probably never do that in production software, but it's still a wicked cool technique"
[22:28:59] <eka> fatninja_: http://tebros.com/2011/07/using-mongodb-mapreduce-to-join-2-collections/
[22:31:15] <fatninja_> the thing is that I can emulate this Join via Model and Db Model , but my main problem is that using js server side, the module for mongo is async, which basically means, when data is received a handler is called with the results. The only viable solution I see is to make this mod sync
[22:33:04] <fatninja_> yes, that is indeed a great option, to show a loader and when data is received to display it, but that means more http requests to the server
[22:34:30] <fatninja_> Wasn't concerned about MongoDB, I was referring to the http server, but now that I think about it, multiple messages can be send/received through a single connection, but don't know for sure
[22:51:46] <Baribal> Hi guys. I just set up a mongod, played with it a bit via pyongo, and then I thought... "SECURITY!" So I added a user (use test; db.addUser('name', 'password'), set auth=true in mongo.conf, service restart... yet still I can access the db without authentification. What did I miss?
[22:52:56] <eka> Baribal: you set the user on which db? test? and then you try to connect to which db ? test?
[22:57:27] <Baribal> { "_id" : ObjectId("509c35e0c0753218efb2959e"), "user" : "project64198", "readOnly" : false, "pwd" : "4b07178bb3b0fe357275ba13969ceab9" }
[22:58:07] <eka> Baribal: http://docs.mongodb.org/manual/tutorial/control-access-to-mongodb-with-authentication/
[22:58:39] <eka> Baribal: http://docs.mongodb.org/manual/tutorial/control-access-to-mongodb-with-authentication/#adding-users
[23:00:39] <Baribal> OperationFailure: db assertion failure, assertion: 'unauthorized db:test lock type:-1 client:127.0.0.1', assertionCode: 10057
[23:06:14] <frsk> *sigh*, just noticed I have to rewrite my scripts for log rotation :( <https://jira.mongodb.org/browse/SERVER-4739>
[23:23:41] <PedjaM> I have issue with sharding, I had capacity issues with single node and decided to go with sharding, but now balancer is making troubles with chunk movement
[23:24:33] <PedjaM> Thu Nov 8 15:21:38 [cursorTimeout] killing old cursor 7588257232297401619 idle for: 600147ms
[23:24:43] <PedjaM> Thu Nov 8 15:21:38 [Balancer] distributed lock 'balancer/mongo3:28017:1352406473:1804289383' unlocked.
[23:26:08] <PedjaM> with this speed it will balance chunks in couple of months, which is unacceptable
[23:27:01] <PedjaM> when i stop background jobs which wirk with mongo it is a lot lot faste (most of the chunks are moved over that time) but i just can't afford to stop background jobs for a long time
[23:28:14] <PedjaM> Anyway, if someone have time to help me a bit, it would be greatly appreciate ;)
[23:43:54] <Baribal> Okay, so I have to create an admin user, authenticate as him, then create a user on the actual db, and THEN authentification works, eka. :D
[23:44:18] <Baribal> And inbetween, after creating the admin user, I think, I have to restart the mongod.
[23:47:13] <Baribal> PedjaM, with the balancing, and yes, it looks like your cursor died because it wasn't used in 10 minutes.
[23:48:08] <eka> PedjaM: mmm sharding is hard... I'm dealing with it now... but doesn't take 24hs and my collection has > 10M docs
[23:49:37] <Baribal> But maybe only 88 chunks have moved because after that the cluster was balanced enough? Was it really chunk movement that made the cursor be idle?
[23:49:57] <eka> PedjaM: so... you are up to that... balancing takes resources and your app takes resources... and balancing locks also
[23:50:06] <PedjaM> well it is not balanced enough, I wouldn't say that 88 of 8000 is balanced enough ;)
[23:53:49] <eka> PedjaM: so there it talks about how to choose your shard key...you may want to add a random number to a new field... but you cant recreate the shard key
[23:55:13] <eka> that tells you how balanced are your dbs...but again ... with an incremental key you are kind of on a bad position
[23:56:12] <eka> PedjaM: the docs say that... don't start sharding when db is full... will make everything slower!!
[23:57:02] <eka> PedjaM: the problem is... your DB is slow cause you are overburden so... now you added sharding... that has to balance an overburden system
[23:57:40] <eka> It takes time and resources to deploy sharding, and if your system has already reached or exceeded its capacity, you will have a difficult time deploying sharding without impacting your application.
[23:58:13] <eka> As a result, if you think you will need to partition your database in the future, do not wait until your system is overcapacity to enable sharding.
[23:59:51] <eka> PedjaM: you need to 1. stop app.. so DB can speed up 2. since you stopped re do your shard key add a random value to a new field