Log file Viewer
#mongodb logs for Tuesday the 11th of September, 2012
[05:31:29] <Null_Route> I've remove some hosts from a replica set. When I use listshards on a mongos instance, I still see the old hosts.
[07:24:00] <Gargoyle> I'm pretty sure they are still available. They are just automatically handled for you.
[07:24:42] <gigo1980> and at the moment each call to the webapp creates an connection to the mongorouter ...
[07:25:20] <gigo1980> so is there an way to make an pool that the webapp handels about (40) persistens connections
[07:27:38] <Gargoyle> I guess that depends on the setup you have. I'm sure with apache + php the persistent connections and pooling is all handled automatically
[07:28:19] <LambdaDusk> I have some trouble with proper database design for mongo... if I have several users, each of them can have several blogs... should be the blogs and the posts always be their own collections? I have heard mongo is about redundancy everywhere
[07:29:03] <gigo1980> so there are to many file locks and this depends on the connections that the application makes to the mongos
[07:31:27] <gigo1980> the mongos creates for each connection from the webapp, an connection the shards behind
[07:33:04] <gigo1980> follwing command gives me 866 established connections: lsof | grep mongos | grep TCP | wc -l
[07:33:57] <gigo1980> but in realtime there not so much connections 52 gives me : lsof | grep mongos | grep TCP | grep localhost | wc -l
[07:34:44] <gigo1980> and the 52 can rise … so what i want is, to set from the application side an pool limit
[09:58:50] <gigo1980> hi, i will remove an shard. but the state is daining ongoing, and no chunks will be moved
[10:04:00] <Gargoyle> Derick: Driver is holding strong. pretty consistent 200ish connections to each of the 3 nodes in the RS. No segfaults spotted in apache, and delivered 25,000+ pageviews already this morning. :)
[10:08:50] <Gargoyle> if my munin traffic graphs are correct, PRIMARY is pumping out 70Mbps, and the Secondaries 30Mbps each.
[10:16:31] <Gargoyle> can't claim much credit for the design itself. But it's been a nightmare fitting the "business logic" into that design.
[10:17:17] <Gargoyle> if that fails, we use google to attempt to find the lat long for what you typed, and then find the closest match from the 500.
[10:19:27] <NodeX> if you like I can give you my database to help you out, it's made up of 2million or so places in the UK including every postcode and placename (towns/villages)
[10:20:38] <Gargoyle> I might take you up on that. At the moment, the focus is on franchisees promoting their area - and not so much the geo-coding
[10:21:08] <NodeX> http://www.theukjobsite.co.uk/seek <--- type in the area to see it in action - it suggests and also looks up on search
[10:21:50] <lupisak> you mean you personally collected the postcodes, or is it from the openpostcode project?
[10:22:47] <lupisak> NodeX: a lot of people are interested in postcodes, but the post office have copyrighted them, preventing free use (bastards!)
[10:23:02] <Mmike> Hi. How do I run --repair option? I had a server crash, and now when I start server I get 'mongo.lock file exists...'. It is stated that I run mongod wih --repair option, but it just spits the same error in the log file
[10:23:50] <NodeX> Derick : I am willing to open source my data, but in general UK postcode databases are made from the PAF which carries a licence charge from the Royal Mail (UK Post people)
[10:24:35] <NodeX> I geocoded (added a lat/long pair) to every single postcode, place, placename and geohashed it all too
[10:24:45] <lupisak> NodeX: I can probably tell you if that's possible if you tell us where the data is from - which various sources?
[10:25:54] <lupisak> NodeX: ah, so you mean you looped through your own address database, and ripped the postcodes?
[10:26:38] <NodeX> Gargoyle : did you get it from PAF ? because I released one into the wild a few years back
[10:26:57] <Zelest> Mmike, Perhaps the repair should remove the lock-file once its done.. but I think I had a similar issue when I expienced the same thing.
[10:27:33] <lupisak> NodeX: sorry to bug you on this, but could you explain exactly how you dreived the data? If done the right way, this data could be interesting for the projects mentioned.
[10:27:53] <Mmike> Zelest, yes, that's what I was expecting, that the lock file would be removed by repair
[10:28:15] <NodeX> it's the basis of alot of my current contracts and retainers so I keep it close to my chest
[10:31:37] <NodeX> Gargoyle : be careful with reverse geocoding with google, after a few requests they throttle like mad
[10:32:17] <nkuttler> how can i order a distinct() query? i need an ordered list of embedded objects, or should i not use distinct()?
[10:34:23] <Gargoyle> But a full gazzateer is on the future list anyway. but we also have to rollout to .au in the new year.
[10:36:49] <lupisak> regarding post codes. Seems the Ordnance Survey has released their data under an open license: http://www.freepostcodes.org.uk/
[10:39:38] <NodeX> that data ^^ is enough to reverse geocode and create a postcode database of every postcode in the UK
[11:20:55] <PDani> I have a replicated environment with two members and an arbiter. I'm experiencing periodical slowdowns when running relatively intensively insert queries. I'm using 2.2.0. I investigated this issue a lot, I tried to turn off journaling, and I set syncdelay to 0 (for testing purposes), but it didn't help. I'm trying to find out what could cause this issue.
[12:21:08] <BurtyB> heh Zelest I've started to add that kind of data to my db too.. tho you've got a head start!
[12:25:55] <NodeX> I can at least aggregate down into months for you, then it will take the load off
[12:27:11] <Zelest> Not that I personally care, but I've got a few mails where people have been quite picky who has access to the data. :P
[13:27:42] <yhpark> Use a DBRef when you need to embed documents from multiple collections in documents from one collection. DBRefs also provide a common format and type to represent these relationships among documents.
[13:30:27] <yhpark> if (documents from one collection) embed (documents from multiple collections) : Use DBRef ?
[13:40:25] <gigo1980> "errmsg" : "MR post processing failed: { errmsg: \"exception: could not initialize cursor across all shards because : socket exception [CONNECT_ERROR] for set12/mongo07.luan.local:10022,mongo08.luan....\", code: 14827, ok: 0.0 }"
[14:46:54] <W0rmDrink> Hi, I have a system consisting of n computers - request for each user can reach each system - but one request may be processed per user at any one time in the system overall - is there a name for mechanisms used to ensure this - I normally refer to it just as syncronization but its slightly vauge
[14:49:51] <sysdef> is there a way to get mongodb support? a mailing list or something like that? or an official support channel?
[14:51:56] <W0rmDrink> further - to achive this we typically do something like : try_lock: db.collection.update( { userId : ..., ownerId : NONE }, { $set : { ownerId : ?self? } }, true, false );
[14:56:56] <Derick> sysdef: you need to joind the google groups before you can access it - but we (10gen) also provide paid support plans
[14:58:42] <abourget> what's the primary shard used by mongos if sharding is not enabled ? is it the first server listed as 'configdb' in mongos.conf ?
[14:59:27] <sysdef> i asked teh questio two times and dien't got a response yet. that's why i asked for an other way
[15:06:56] <abourget> Derick, but where will mongos store the data, if it,s configured to deal with 3 shards, but none of them have been enabled with "enable sharding" ?
[15:07:35] <Derick> Will mongos even be able to talk to any shard in that case? I don't think it would.
[15:11:55] <abourget> Derick, can we connect directly to a server that has `shardsvr` == true ? and store data there ? or *must* be accessed through mongos ?
[15:51:14] <NodeX> I want to do something like db.foo.update({ymd:20120911,type:'geohash'},{$inc:{"foo.myhash":1}});
[15:52:03] <jsaacmk> http://pastebin.com/5JJUA3Uf is there a more concise way for me to do the query at the bottom of the post (ideally with mongodb directives). Also, why does the result print twice when I run it in the mongo CLI?
[15:59:12] <nostalgeek> Hi all, I am fairly new to Mongo. I think I've got a replicas between 2 mongod working (ports 27018). I also have 2 mongos routers (ports 27017). Now, if I understand right, when I connect to the mongos (27017) and issue slaveOk() this allow routing of find() queries to my secondary. How can I tell if the query was served by the secondary or by the master?
[16:27:39] <astropirate> I am working on a CMS. Documents can have child documents. There is no limit how deep the tree structure can be. Should I reference the child documents or embed them? Also what would be the way to get the child documents indexed too based their "slug" property
[16:31:35] <Samurai> @astro from personal experience i would reference them, it will make querying much easier
[16:36:09] <Samurai> i have a quesion.. is there a way to return the whole collection from a disctinct command match?
[16:53:00] <nostalgeek> answer to my previous question: I enabled profiling on the DB, and confirmed my read requests are logged on my secondary, and any inserts are logged on the primary!
[19:01:40] <iyp> Anyone have advice on dynamically generating json-schema compliant docs on homogeneous(ish) collections?
[19:16:09] <crudson> iyp: that's going to come from one of your application layers somewhere, and depending on what language you are using. There's nothing currently in mongodb to associate a schema with a collection.
[19:17:46] <iyp> crudson: Gotchya. Do you know if theres anything in the json-schema spec that allows for summarizing populated data? Or is that another problem entirely?
[19:19:29] <crudson> iyp: I have no idea, sorry. That would be a question for the json-schema folks.
[19:44:45] <iyp> I intend to keep a cache of the data sources with something like Hbase and then transform that data into mongodb collections to sit under a web interface for exploration.
[20:43:16] <Rhaven> hi all, i have just added a shard on a running shard cluster, since that i have got some high latency... and i got some error like that
[20:45:44] <_m> Have you checked here: https://groups.google.com/forum/?fromgroups=#!topic/mongodb-user/lm7Ht1yCpeQ
[20:46:48] <_m> There's a lot of information on the error messages you linked available via google. I suggest turning there first. Apologies in advance if you've already exhausted that route.
[21:08:26] <tomlikestorock> anyone here know how to connect to a replica set in mongohub over an ssh tunnel successfull?
[21:24:07] <Neptu> somethat that used pymongo how i can do db.lala.insert(..) or db.lele.insert() dynamically?
[21:37:28] <Gaddel> hello, i'm brand new to mongodb. if i have a collection of documents and i want a string from one of the fields to the be primary key, is my best option just duplicating the field value for "_id" as well?
[21:41:48] <crudson> Gaddel: you can just use it as _id providing it's unique and logically appropriate. Downside is that it may not be obvious to consumers of that document what that field means without further knowledge.
[21:42:50] <Gaddel> which isn't naturally very similar to an "ID" so to speak. but i could probably store it there and then re-assign it to "IP" when presenting the document
[21:47:34] <crudson> Gaddel: if your collection is 'ip_address_stats' or something, then it's obvious what the _id means, otherwise I personally see benefit in making it obvious what attributes mean. Just my opinion.
[22:22:40] <fg3> I can search for my-field true -- how can I search for my-field: false -- or does not exist