Log file Viewer
#mongodb logs for Wednesday the 9th of October, 2013
[01:28:20] <dllama> i turned off mongod hoping to enable text search, but now i can't get it to turn back on with the db
[01:33:28] <dllama> i have 6gb of data in that folder, but when i login via mongo console, it shows the db as empty
[01:36:41] <crudson> http://docs.mongodb.org/manual/core/text-search/ http://docs.mongodb.org/manual/tutorial/enable-text-search/
[01:41:52] <ruphos> is there a page somewhere listing messages that go with the various error codes? I was having trouble finding one earlier and digging through source every time is quite the bother
[05:45:16] <jhonkola> I'm getting ConnectionFailure with "ids don't match" when trying to connect to mongo
[05:46:17] <jhonkola> connection is being made from django 1.4.5 running on gunicorn (4 workers) and nginx
[05:47:18] <jhonkola> Issue PYTHON-398 on JIRA claims that a similar error is fixed on pymongo 2.1.1
[06:08:31] <das3in> Hi, I'm trying to learn mongodb after a long time with SQL, I'm having trouble finding how query for specific fields. I want to set a variable (say var a = ) "Select ID From Users"
[06:09:02] <das3in> and I know I can do like, db.Users.find({name: "whatever"}) but I just want the id not everything
[06:24:09] <liquid-silence> what would be the best way to do this, I have a top level document and need to add a collection of documents into it, called files
[07:51:26] <liquid-silence> currently db.containers.find({"folders.name" : "another test folder"}) returns the whole document
[07:55:47] <liquid-silence> current query is db.containers.find({"folders.name" : "another test folder"}, { "assets": true})
[08:04:11] <ron> it gives you the structure you use, yes. the object id comes by default. you have to remove it if you want (that is, "_id":false)
[08:07:55] <ron> liquid-silence: either what HashMap suggested or use a different document structure. you can't have it all, man.
[08:10:15] <ron> no worries, it's a fair question. but that question can't be answered without knowing more details. like what kind of queries you want to do, how do you normally update your data and so on.
[08:11:19] <liquid-silence> well I would like to query the container, and get all the folders, assets that are part of the root
[08:11:36] <liquid-silence> then the user will navigate into a folder, and he will have subfolders and assets
[08:12:46] <ron> well, if you intend on reading it as a whole, then store it all in one document. if you intend on reading it one directory at a time, you'd probably want to flatten the structure.
[08:16:22] <liquid-silence> http://docs.mongodb.org/manual/tutorial/model-tree-structures-with-ancestors-array/
[08:16:57] <HashMap> wouldn't it be better to have a document for a folder/file and an array of subfolder id's?
[08:46:46] <talbott_> but when i add them, rs.status() says "lastHeartbeatMessage" : "still initializing" forever
[09:33:06] <Veejay> Hello everyone, what exactly is a mongo session? And is there a limit to the number of Mongo open sessions?
[09:37:09] <maasan> i uses mongoose with mongodb. If mongo db stops, server throws exceptions and stops accepting post request. How and where to handle mongodb connect issue?
[09:58:04] <remonvv> Veejay: The two concepts somewhat related to sessions areconnections which are a limited resource and live for as long as the connection is open, and cursors.
[10:12:08] <Veejay> remonvv: To be more precise I'm using Mongoid which has Mongoid::Sessions thus my asking. But I guess Mongoid is another topic that falls out of the scope of this channel. So thanks :)
[10:13:30] <remonvv> Well, mappers and drivers are valid topics, but I can't help you with that ;) Mongo doesn't have sessions though.
[11:06:01] <maasan> does it try to reconnect for every specific time perioed? does it happen on next mongo db request?
[11:11:52] <bmcgee> hey guys, I have a really simple tailable cursor in Java which seems to just refuse to work. I'm running 2.4.5, with collection.find().addOption(Bytes.QUERYOPTION_TAILABLE)
[11:12:14] <bmcgee> I'm periodically check if the cursor hasNext(), but it never does despite there being new entries in the capped collection
[12:25:22] <Neptu> hej have a question, I just tested the tutorial and created a replica set with a master and 3 replicas
[12:26:20] <supermang> Neptu can you send me the link for that? Im trying to setup a 3 cluster sharded environment also
[12:29:20] <Neptu> supermang, Im not sharding this is jusst a replica set... trying to play with it to understand it
[12:29:25] <joannac> supermang: You don't like the tutorial in the docs? http://docs.mongodb.org/manual/tutorial/deploy-shard-cluster/
[12:29:48] <Neptu> joannac, just playing aroudn and learning... but its curious you can not get as many arbiters as you want
[12:30:41] <Neptu> joannac,I mean if they only help you to vote the maximum should be the same as nodes you have in your replica
[12:32:38] <Neptu> joannac, a question now that I configured on the console the replica set if I turn all off and turn it on the configuration is still available?
[12:33:54] <Neptu> joannac, I setup for ubuntu machines with one mongo each. I created a replica set of 4 mongod and I created 4 arbiter mongod processes aswell
[12:35:42] <joannac> supermang: sign up for one of the courses, if you need a walkthrough on sharding?
[12:36:05] <joannac> M102 definitely walks through, i think the M101X courses do too but not 100% on that
[12:38:55] <Nomikos> new to mongodb, long-winded question.. http://stackoverflow.com/questions/19272209/mongodb-category-tallies-of-found-set
[12:43:12] <jhonkola_> I'm getting ConnectionFailure with "ids don't match" when trying to connect to mongo
[12:43:28] <jhonkola_> connection is being made from django 1.4.5 running on gunicorn (4 workers) and nginx
[12:43:34] <jhonkola_> Issue PYTHON-398 on JIRA claims that a similar error is fixed on pymongo 2.1.1
[12:48:35] <Nomikos> joannac: I would "select" the documents to be grouped with the _id field, as in the search params go in there?
[12:50:03] <Neptu> joannac, question, does the mongod that holds the database as secundary on port 27017 can be added as a arbiter or you really need to define an extra mongod process in the same machine?
[12:51:54] <joannac> jhonkola_: Not really surprising :) You need to figure out the simplest way to reproduce it :)
[12:52:01] <jhonkola_> joannac: Google search suggested that it has something to do with multiprocessing / multithreading, probably gunicorn in this case
[12:52:48] <jhonkola_> joannac: I'll try to set up a simple django / gunicorn setup and try to reproduce there
[12:53:22] <joannac> Neptu: you have a process running on 27017, 27018. Are they both already in your replSet config?
[12:54:09] <Neptu> joannac, Ok I setup a normal mongod in all my machines, and I created another mongod --port 30000 --dbpath /data/arb --replSet rs so on the same server I have 2 mongod one in the normal mongo port an another on port 30000 the first as replica and the second as arbiter. My question is If I can have only one mongod acting like arbiter and database
[12:55:41] <joannac> if they were the same process, once you killed that process, you would be down 2 members
[12:56:38] <Neptu> joannac, for the simple reason that you have only 4 machines so for the voting process to happend should be 4 arbiters
[12:58:32] <joannac> the only purpose of an arbiter is to get to an odd number so you don't get ties
[13:00:49] <Neptu> now I get the point, in any case makes more sense for a secundary to rely on an arbiter to decide than get a network problem and find yourslef with 2 masters out of sync
[13:28:47] <drag> Hi. I need to implement so sort of paging for some queries. What's th easiest way to do this?
[13:32:57] <drag> cheeser, the find with numToSkip and batchSize is deprecated. What's the new way to do this?
[13:34:13] <taf2> anyone have suggestions on migrating from sharded to NOT sharded? see https://jira.mongodb.org/browse/SERVER-9845
[13:34:20] <remonvv> drag: Neither numToSkip nor batchSize are related to this. Use skip() and limit(). And it is the easiest way to do paging. It's *potentially* problematic because skip is an O(n) operation which means if you need to page to very high page numbers performance will degrade.
[13:34:38] <taf2> have a production system running that was sharded but on a bad key also sharded way before it's needed
[13:35:07] <taf2> using the technique in that bug report looks reasonable but wondering if anyone can verify
[13:36:40] <remonvv> taf2: There's no such thing as sharded before you need it ;) Any production database that has to potentially scale up at some point should start out sharded. If you sharded on the wrong key refer to http://docs.mongodb.org/manual/faq/sharding/#can-i-change-the-shard-key-after-sharding-a-collection
[13:37:16] <remonvv> drag: I'm not saying there's a better way, I'm saying that there are times when skip() and limit() may not be the best approach for performance reasons.
[13:37:38] <remonvv> drag: If it's for a UI you'll never page to page 102 anyway so it's usually fine.
[13:38:23] <remonvv> drag: If you have a huge leaderboard with 400k players and you need to display the page with player 232,411 on it then you might have to do things differently.
[13:39:06] <taf2> remonvv: thanks - i've seen that document, but have seen the bug report i referenced here: https://jira.mongodb.org/browse/SERVER-9845 it would appear this strategy could work
[13:39:49] <taf2> let's say i take a server with 3 shards… and drain out 2 of the 3 into just 1 mongod instance
[13:40:06] <taf2> now i'm left with 1 mongod instance and all the data but did not have to suffer downtime
[13:40:44] <remonvv> taf2: That's not the end of it though, you still need to unshard the collection by manually removing config metadata, flush the routers, hope everything is still okay, and so forth.
[13:41:58] <remonvv> But there's no "unshard collection" command so you'll have to manipulate the config database manually to make that happen.
[13:43:02] <remonvv> That's the safe route. The risky route is to change config data directly and flush your mongos processes.
[13:43:37] <taf2> remonvv: got it… and that's because as data is being written it's possible data could be lost?
[13:45:07] <remonvv> taf2: The route is; stop all writes, stop balancer, remove your sharded collections from db.collections in config db, remove chunks in db.chunks for those collections, execute router flushes on all mongos processes and hope.
[13:45:34] <remonvv> I don't really see how you can lose records other than ones that are being written while it lost chunk information.
[13:46:33] <taf2> i might do that… so it sounds like the risky route would convert things faster than reloading
[13:47:37] <agend> have a problem- when I make update with query: query = {'_id': SON([('u', hash_pk), ('k', 'h'), ('d', ts)])} and upsert=True i get document with _id: { "u" : 567, "k" : "h", "d" : ISODate("2013-10-09T13:36:23.666Z") }. But when I make a update with query {'$or': [{'_id': SON([('u', hash_pk), ('k', 'h'), ('d', ts)])}]}, I got document with _id like this: ObjectId("52555cbdad20bca3410e995b"). So - how can I update(upsert) with query which selects mu
[13:50:39] <agend> have a problem- when I make update with query: query = {'_id': SON([('u', hash_pk), ('k', 'h'), ('d', ts)])} and upsert=True i get document with _id: { "u" : 567, "k" : "h", "d" : ISODate("2013-10-09T13:36:23.666Z") }. But when I make a update with query {'$or': [{'_id': SON([('u', hash_pk), ('k', 'h'), ('d', ts)])}]}, I got document with _id like this: ObjectId("52555cbdad20bca3410e995b").
[13:50:42] <agend> So - how can I update(upsert) with query which selects multiple documents and keep the _id with data I want?
[13:53:31] <Neptu> joannac, I just tested to put down all nodes adn turn on 2 of them... the problem is seems now both are secundary so voting happend
[13:54:26] <agend> kali: but should work the same in others - I need to update mulpitle documents - probably create them (upsert) but when I make query to select more then one I get ObjectId(....) as _id instead of my _id and only one document. What I want is to update/create many documents in one go
[13:56:32] <kali> agend: well, if you can reproduce the problem in the mongodb shell, maybe more people could help
[13:56:53] <agend> jonnac: like this with pymongo/python: SON([('u', hash_pk), ('k', 'd'), ('d', day_ts)])
[14:01:41] <joannac> agend: I don't understand how you think you can get unique _ids from a single-line update
[14:02:21] <Neptu> joannac, what I do not understand why should be the majority on all the servers and not the majority on all servers still alive
[14:02:44] <joannac> Neptu: because you have no idea if the other servers are dead on on theother side of a network partition
[14:03:22] <Kim^J> Neptu: When you have two servers, if one dies, how should the other one know it can be primary? It can't vote on itself.
[14:07:02] <agend> joannac: when I make: query = {'_id': SON([('u', 'blah'), ('k', 'h'), ('d', 'ts')])} mongo_collection.update(query, update, upsert=True) I get upserted document with _id = {u: 'blah', k: 'h', d: 'ts'}
[14:07:24] <agend> joannac: I thought I can get the same thing with query which selects mulitple documents
[14:07:52] <agend> joannac: I think I should precreate the documents first and then maybe it could work
[14:08:24] <Neptu> in case of network partition lets say on thsi setup you have 7 machines if 2 secondaris and arbiter are still alive can not decide for a master what means I'm n/2 -1 fail tolerant so if the net gets segmented one segment will be able to survive and sync later in case of failure of the other 4 boxes the rest alive can not survive
[14:11:06] <agend> joannac: nop - I want to update about 40 documents (which may not exist yet) at one go
[14:12:45] <agend> joannac: I thought if I make query with _id $in [40 ids] it would create and update the docs like for one doc - and keep the _id for me
[14:23:49] <bhangm> is there a way to force initial sync of a new replica set member from the primary?
[14:27:35] <bhangm> I need it to sync from primary b/c of some cleanup that was run to drop databases. I couldn't issue the drop on the primary b/c it takes a global write lock *and* does a flushAll for every single drop
[14:28:39] <joannac> bhangm: "To affect the sync target for the initial sync, run rs.syncFrom() operation before initial sync."
[14:28:53] <joannac> from http://docs.mongodb.org/manual/tutorial/configure-replica-set-secondary-sync-target/
[14:30:17] <joannac> bhangm: so now you have a primary with no db, and N secondaries with the db? and you're going to initial sync them all?
[14:31:05] <bhangm> I dropped a set of dbs, brought that secondary back in and made it the primary (after catching up)
[14:31:23] <bhangm> now I want the other secondary and the old primary to basically refresh their data set from the new primary
[14:31:43] <bhangm> seeding is one way to do it but unfortunately there are constraints that prevent me from doing that
[14:33:32] <bhangm> how exactly does one run rs.syncFrom before initial sync ? Doesn't initial sync happen as soon as the member starts up? Or do I have to remove it from the set, restart it and run rs.syncFrom and add it back to the set ?
[14:39:08] <agend> joannac: ok - it's preserved - for existing document. But when it creates a new one (thanks to upsert) we have 2 cases - when query for update selects one document by _id: i have a new doc with _id from query. But when I have query with _id $in list of 40(for example) expected _id's - I get 1 new document with _id = ObjectId. I guess it is how mongo works. Now I try to precreate the documents and update them with multi and it works
[14:41:12] <agend> when I make bulk insert - and some of the docs already exists - are they destroyed? Is there any way to make bulk insert and preserve already existing docs?
[14:48:12] <bakis> hey guys is there a way to skip the config user accounts on mongodump and mongorestore? I'm dumping from one branch and loading to another and it's overwriting the access keys.
[14:49:16] <drag> agend, from my quick testing, it throws an exception when I try to add a document that's already in the database with the java driver.
[14:53:44] <bhangm> joannac: no luck with that approach I'm afraid. Might have to try blackholing traffic to existing secondary instead
[14:59:06] <bakis> okay.. is there a way to specify multiple collections in mongodump? or specify NOT a collection? can't seem to find it in the docs
[15:13:16] <bakis> Could anyone tell me a better way to do this? I'm trying to transfer data from my production branch to my feature branch for real seed data. How is this not a common problem?
[15:15:05] <bakis> sorry. i'm using mongohq on heroku. the config variables are being pulled down and when i load it into the next branch it's overwriting the mongohq credentials
[15:41:47] <drag> in the Java driver, if I want to check if field A contains a value from a list of values, I can do that with BasicDBOBject().put("A", ["1", "2", ...]), correct?
[16:35:18] <jamesshijie> Quick beginner question about mongo find() -- Do I need to check the existence of a key before I try to access it? It seems to work if I don't, but is it slower to do that, or just bad practice?
[16:36:05] <cheeser> if you find() against a property that doesn't exist, you'll simply get no results.
[16:39:36] <jamesshijie> ok, that I understand. So in my collection, I have a variety of objects each containing the key module_id, course_id, or organization_id. If I search for module_id, and there are other documents that don't have that key, will they be intelligently ignored?
[16:41:17] <jamesshijie> Nomikos: Hahaha. Thank you. I've been at this for about a day or so. I love it so far, but a few paradigm shifts need to happen in my mind.
[16:44:12] <jamesshijie> kali: If I had a collection with equal amounts of documents containing one unique key (module_id, course_id, organization_id, etc...) Would indexing be the way to go for that?
[16:44:44] <Nomikos> jamesshijie: might it make sense to split the collection into different collections based on the type of data?
[16:46:57] <Nomikos> Q: in the Mongo CLI, if I do b.find({foo: 'bar'}) where b is a collection it outputs a bunch of docs. how do I store those docs in a variable?
[16:50:29] <Nomikos> if I just enter 'res' on the next line, it outputs the data, but then it's gone. I guess that's the cursor emptying itself
[16:51:40] <cheeser> calling .toArray() will exhaust the cursor and give you an array of the results
[17:14:28] <Nomikos> is there a shortcut to delete a certain attribute of all documents in a collection?
[17:21:31] <Nomikos> kali: cheers. maybe I should go home if I can't figure out those things anymore >.>
[17:40:36] <Nomikos> defaultro: you don't specify a size. I don't think it knows what size the data in the fields is
[17:41:49] <Nomikos> if you need larger, there is gridfs or somesuch, but now I'm talking about something which I heard late at night in a pub on a stormy night
[19:52:33] <dxd828_> I have started my mongodb install with the --auth option. When I try and add a user admin using: db.addUser() I get a throw "password can't be empty". Anyone know whats up with that?
[19:55:22] <dxd828_> db.addUser( { user: "admin", pwd: "mycoolpassword", roles: [ "userAdminAnyDatabase" ] } )
[19:56:32] <retran> http://docs.mongodb.org/manual/reference/privilege-documents/#<database>.system.users.pwd
[20:01:11] <dxd828_> cheeser, ah I did this install quickly, must of used the packages in the cents repo doh!
[20:02:52] <dxd828_> retran, yeah will do. I used them on my desktop and I managed to get auth working the same way. That's why i wondered why it wasn't working :) thanks for the pointers
[20:03:15] <cheeser> if you're on ubuntu, there's a 10gen repo you can add to get the latest ones we build.
[20:04:40] <dxd828_> Ah, I'm on centos. Ill just use http://downloads-distro.mongodb.org/repo/redhat/os/x86_64/
[20:58:13] <ZacGod> One of our clients is pushing back on using seperate vms for config servers but i can't find any documentation talking about the subject.
[21:06:51] <MattAFT> It can't write remotely (from my local) machine....the boxes are hosted on aws and from an app server on another ec2 instance, writes work fine
[21:07:15] <MattAFT> Whenever I try to write remotely I get "Unable to connect to a member of the replica set matching the read preference Primary".
[21:07:40] <MattAFT> connection string looks like mongodb://mongobox1,mongobox2,mongobox3/dbname?replicaSet=replicasetname&readpreference=primaryPreferred
[21:07:53] <MattAFT> I also found that the sometimes (not sure the conditions), when I take down the master, I can then write fine from local and when I bring the machine back up it syncs properly. But as soon as I bring the machine back up, I can no longer write from local
[21:08:35] <MattAFT> also If I change my connection string to only connect to primary, then the write goes through fine, but obviously does not replicate to the other boxes.
[21:08:58] <retran> is there an awesome book i can learn about mongo deployment stories and strategies
[21:11:02] <joannac> Your replica set doesn't cease to be a replica set just because your connection string changes...
[21:13:08] <ZacGod> Matt, if the C# driver is like the java driver it will check to see if there is a replicaset when it connects.
[21:15:35] <joannac> MattAFT: when you set up a replica set every node knows about all the other nodes in the replSet
[21:18:01] <MattAFT> but the main issue is being able to write at all when my connection string has all 3 instances
[21:24:13] <joannac> so something about the combination of your local setup + c# driver, most likely.
[21:25:50] <MattAFT> is there any other additional tools to use to investigate something like this or logs or support?
[21:42:50] <joannac> i would want to see the code you're running on both servers, to make Sure it is the same