Log file Viewer
#mongodb logs for Tuesday the 4th of June, 2013
[03:25:53] <xilo> looking to see what type of database to use. i have this scenario: several objects of type Foo with predetermined values for properties. Object class Bar could have a reference to Foo.id=1 and object blass Blah could also have a reference to Foo.id=1. can you set this up without being wonky ie different collections so i don't duplicate data? or should i stick to a RDBMS
[07:18:37] <lessless> hi guys! how do I enable database listing in ubuntu? should I edit startup script and add --rest or there is an config option to do this?
[10:59:19] <yorickpeterse> Hey folks, is there an easy way to get an overview of all the used fields in a Mongo collection? Preferrably taking nested structures into account. For example:
[10:59:33] <yorickpeterse> Given you have something like this: [{title: "foo", author: {name: "john"}}, {title:"bar", author: {age: 10}}]
[10:59:57] <yorickpeterse> You'd expect something like this as the result: { title: 2, author: 2, author.name: 1, author.age:1 }
[11:15:50] <yorickpeterse> Hm, that would be an option. However, this will be done for a pretty big (30m objects or so) collection
[11:16:14] <yorickpeterse> You can probably do it using the aggregation framework/map reduce but I fear that will be something I can't look at 2 weeks after writing it
[11:16:52] <yorickpeterse> Hm, map/reduce performs better than the aggregation framework in that case?
[11:17:01] <yorickpeterse> In most of our cases we've had the exact opposite in terms of performance
[11:18:27] <crudson> bed here, if you wanted an example map reduce for this send me a pm, would be a nice little one to do
[12:13:05] <schlitzer|work> hey alles, what is the recommended way to have a singlehost shard, a config server & a mongos on the same ubuntu (12.04) host?
[12:41:46] <foofoobar> Hi. I just installed mongodb and took a look in the data dir. I have no entries in the database and the data folder has a size of 3.3GB??
[13:08:33] <foofoobar> double_p: so I set smallfiles to true and restarted, can I delete the old big journaling files now?
[14:31:15] <diegows> iterating over docs and updating a field that is in one index isn't a good idea right?
[14:36:33] <arthurnn> curiosity: doest mongodb uses Red-Black Tree for maintaining the order of keys on embed docs?
[14:38:42] <arthurnn> Snebjorn: sure.. but I am curious to know what do they use to maintain the order of keys in an embed doc.
[14:42:08] <Snebjorn> so it just stores the location of the main document and don't care about the order of sub docs
[14:45:14] <arthurnn> Snebjorn: actually i does keep the order. try something like this: db.entities.update({ "_id": ObjectId("51adf6957c71c1cc08000001") }, {"$set": {"links.0.a": 1370355778} }); db.entities.update({ "_id": ObjectId("51adf6957c71c1cc08000001") }, {"$set": {"links.0.c": 1370355778} }); db.entities.update({ "_id": ObjectId("51adf6957c71c1cc08000001") }, {"$set": {"links.0.b": 1370355778} })
[14:45:45] <arthurnn> when u do db.entities.find()['links'] , u will see that the first link has Orderd keys
[14:46:54] <ixti> arthurnn: gotta run, but I saw some explanation on docs that it "runs" through all nested docs
[14:48:20] <Snebjorn> arthurnn, try reading about indexing in this book http://it-ebooks.info/book/964/
[14:48:57] <arthurnn> Snebjorn: i am not sure if this alphabetical order on subdocs is related to index at all or not.
[14:49:26] <arthurnn> Snebjorn: in my example I dont have any index on my collections.. and the order on the keys are still there.
[14:50:04] <Nodex> are you asking how they're stored in the index or how they're stored by mongo in relation to how they appear in the shell?
[14:51:35] <ixti> arthurnn: google mongo docs re structure of big nested docs. not sure how it was called
[14:51:46] <arthurnn> Nodex: my question has nothing to do with indexes... i am just asking 1. Why do they keep the keys ordered on a embed doc? and HOW? RB tree?
[14:52:51] <arthurnn> yes they are... try the example I posted above.. u will see.. everytime u $set that new key wont go to the end of the doc...
[14:52:55] <Nodex> each object in an array after save -may- get ordered but embedded array's are not ordered for obvious reasons
[14:58:24] <arthurnn> Nodex: the doc is totally irrelevant. run those commands u will see in the last find that the keys on links.0 are ordered.
[15:02:54] <failshell> hey guys. im trying to restore a backup we did with LVM snapshots of a sharded cluster. on a different set of VMs. obviously, its failing because the machines names changed. can anyone give me a brief crash course on doing that?
[15:35:44] <jpfarias_> I have this collection where I added a coordinates field with [lat, lng] format
[15:39:08] <vargadanis> Nodex, is that a built in data type for mongo or ... linky please? :D I'd like to read about this
[15:40:24] <jpfarias_> To calculate geometry over an Earth-like sphere, store your location data on a spherical surface and use 2dsphere index.
[15:40:24] <jpfarias_> Store your location data as GeoJSON objects with this coordinate-axis order: longitude, latitude. The coordinate reference system for GeoJSON uses the WGS84 datum.
[15:41:06] <aandy> Nodex: true. not a factor for my pet project, but worth considering, right. probably an okay cost when prototyping though, to avoid these bugs :)
[15:41:44] <jpfarias_> I am just worried that it may stop working on future releases if they change something
[15:43:23] <jpfarias_> when I do a {$within: {$center: [[lat, lng], radius]}} query it takes a long that the first time I query it
[15:44:57] <vargadanis> I was reading about read performance of MongoDB. In a situation where it is not critical to have the most up to date information, I could direct the reads to specific replica sets... However the same docs page sais that I would be smarter to just direct all to the primary and use sharding instead. Are the 2 mutually exclusive?
[15:53:38] <jpfarias_> Tue Jun 4 11:48:15 [conn8645] getmore mls_data.mls_data query: { coordinates: { $within: { $center: [ [ 34.093418, -118.202126 ], 0.015 ] } }, $or: [ { listing_info.status: { $in: [ "active", "pending", "on hold", "backup" ] } }, { listing_info.status: "sold", listing_info.sold_date: { $gte: new Date(1338824884659) } } ] } cursorid:3277315563308696778 ntoreturn:0 keyUpdates:0 locks(micros) r:4588433 nreturned:161
[15:56:47] <vargadanis> it is soo damn hard to get started with MongoDB ! :) I wish I had a guy to whom I told what kind of data I want to store, what is the average read/write performance and he'd set it up to me and just told me: it's gonna work, go do your programming! :D
[15:59:03] <vargadanis> which will create a secondary index on the field (do you use this term at all?) called "Field"
[15:59:45] <dougb> I'm using MongoHub right now as a client, so I just enter in a query string under 'index' but I assume it's running it through ensureIndex
[16:00:15] <kali> vargadanis: you'll do it wrong anyway. it's always wrong the first time. so go do your programming, it won't work, we'll help you fix it, and you'll learn something :)
[16:02:10] <vargadanis> dougb, I guess that is an application specific thing, but I guess so. I mean it would be the Vulcan (logical) way to do it for me
[16:04:33] <double_p> so.. i have this offsite replica. lowest priority and all fine. but it _happens_ that it loses connectivy and then reelection blues (quick, but it DROPS connections). even if i dislike, i would put it to hidden. but that doesnt prevent election if lost.
[16:07:21] <vargadanis> I read that it is a good idea to create shards per core over shards per server for better perf. however is there a way to actually limit a mongod instance to a core or even "dedicate" a core to one? Working under linux system..
[16:24:16] <vargadanis> is there some kinda cool GUI like app for web or desktop I could use to monitor/manage what is going on with my DB?
[16:29:50] <vargadanis> jpfarias_, you mean you resend the same query and once it is 48ms and then 425?
[16:32:33] <vargadanis> I suppose the docs for the 2nd query were already in memory or the query was cached... or why the huge performance increase?
[16:36:10] <vargadanis> I've read - though it's a bit different - to optimize queries it is sometimes a good idea query the entire data set and then run the query which "looks" for something.. but it only works for smaller data sets that fit into the memory
[16:36:58] <vargadanis> guess it's due to the fact that the first query runs slow is because of disk IO? O_o dunno
[17:25:15] <failshell> im trying to reconfigure a sharded cluster from backups. problem is, the cluster sees the new shards and the old ones. every command i try fails because of that. tried db.runCommand( removeShard), but its failing because the balancer is not running
[17:27:35] <devastor> We just got "Tue Jun 4 17:08:29 [repl writer worker 2] production.messages Assertion failure type == cbindata src/mongo/db/key.cpp 585" with mongo db 2.2.3, any known bugs that might cause that?
[17:34:41] <devastor> [repl writer worker 2] ERROR: writer worker caught exception: assertion src/mongo/db/key.cpp:585 on { long document data here or part of it }
[17:37:34] <starfly> failshell: Dude, most folks probably don't want to weigh in on that dicey situation. I'd guess that they only way you can effectively recover is to build an environment that looks like (operating system, hostnames, hardware) very similar to what you backed up previously and recover into that. Comingling old and new shards sounds like a losing proposition.
[17:39:30] <starfly> failshell: so, build the staging environment out like it was production. You can use the same hostnames, just can't obviously do that with DNS, but only local to the staging hosts...
[17:41:34] <starfly> failshell: when hostnames are embedded in configurations, what else can be expected?
[17:42:33] <starfly> failshell: OK, well I've worked 25+ years with Oracle DBA and I wouldn't agree that it's always easy...
[20:36:59] <miskander> I need to find all records based on an attributes in a different collection. For example find all comments where the user role is 'author'
[20:37:16] <jpfarias__> I'm not sure if you are the right person for this, but why would mongo take too long to answer for some query, even when the index is there?
[20:37:42] <miskander> I come from Mysql land, so first reaction is one query with a join but that doesnt work out in Mongo :)
[20:42:35] <niemeyer> jpfarias__: I have never used the geo features of MongoDB, so I'm not a good person to walk you through this, but I'm telling you what that query you made is showing you
[20:46:17] <kali> miskander: you have to denormalize: add the user role alongside the user id in the comment collection
[20:49:00] <kali> miskander: you need to think and design your data model to fit the specific needs of your apps, not as a pristine theoretical thing
[20:55:48] <jpfarias__> there's an "Important: If you use longitude and latitude, specify longitude first."
[20:59:57] <niemeyer> jpfarias__: Again, without knowledge of the implementation, it shouldn't make a difference if the 2d index is indeed in an Euclidian space
[21:01:02] <niemeyer> jpfarias__: In other words, if it's flat and bounded on -180-180 on both values
[21:01:38] <niemeyer> jpfarias__: Geohashes defines how the points are encoded, not how the indexing works
[21:03:40] <niemeyer> jpfarias__: If everything just works on a flat 2d space, and you swap x/y, it'd still work, though.. but given the comment in the page, I'd ask someone that knows better than I do
[21:04:18] <niemeyer> jpfarias__: Once you start using spherical coordinates, that'll all be completely bogus for sure
[21:12:24] <niemeyer> jpfarias__: That query is running of 30k documents, yielding 600 times.. I don't know where you're running this, but it's definitely not surprising that it's misperforming.
[21:17:29] <tystr> the agent from this host shows up under the agents tab, but it never seems to submit any data
[21:25:41] <niemeyer> jpfarias__: Have you investigated queries around the latitude boundary (90 or -90)?
[21:26:13] <niemeyer> jpfarias__: The docs say it's the limits are bounded at 180, but I wouldn't be surprised if part of the logic is actually enforcing the correct boundaries.
[21:27:20] <niemeyer> jpfarias__: You might also just test the same query against a point without the correct boundaries
[21:32:06] <tystr> ah I'm trying to run 2 agents separately in the same group in mms…this is the problem
[21:37:29] <jpfarias__> hmm, do you think it would be much faster if I update them first then create the index?
[21:51:09] <jpfarias__> tomlikestorock: you can probably make a complicated enough regex that matches :D
[21:51:33] <tomlikestorock> jpfarias__: yeah, but I saw you could definitely do a $in on regexes, so did that instead
[21:58:14] <jcalvinowens> Quick question: is there a way to issue raw mongo shell commands from pymongo? Want to use forEach(), but not callable from Python.
[21:58:35] <jpfarias__> tomlikestorock: does it have any performance penalty for the $in instead of the big regex?