Log file Viewer
#mongodb logs for Friday the 20th of July, 2012
[00:56:04] <hadees> I'm writing a Facebook app where I want to cache the user's feed. I was thinking about doing this in the User model. Have you heard of anyone doing this? or does it really make more sense to create an associated document.
[03:34:29] <roycehaynes> The document is a User document. I have several type of users, that i've created as separate documents but i want to specify them as a attribute in the User Document.
[05:40:26] <trupheenix> >>> x = objectid.ObjectId("a407dae09ef4b786b889e8fe761887b8d510959f7649802aa8c22e00")
[06:10:37] <wereHamster> if you want to use your own IDs, then use a string, or UUIDs, or a binary value, or an Int, or whatever else suitable in your situation
[06:34:05] <kaikaikai> when using addToSet and updating/inserting a few values, is there a way to choose which value is being checked for existence when appending? ex: keyword + createdBy
[06:34:42] <kaikaikai> if my insert is the keyword with createdBy value, it will always consider the insert as if it does not already exist, when really i just want it to check the keyword
[06:59:23] <alexisabril> Morning. I'm writing a simple rest api(ruby), but I'm curious on the right way to return a json formatted list of documents with corresponding id's. e.g.: [{_id: "somekey", foo: "bar"}…]
[06:59:54] <alexisabril> I'm using the following code atm: coll.find().to_a().to_json, however this returns the bson formatted oid
[07:01:43] <alexisabril> Right, I was curious if there was something more elegant than looping through each item in the array to reformat/build up a custom response
[07:02:16] <crudson> alexisabril: exclude the _id from the query with {:fields => {:_id=>false}} if you don't want it
[07:03:29] <alexisabril> @crudson I'd like the id to be present, but it's a bit more intuitive on the client side not to have to worry about typing _id.objectid
[07:04:00] <wereHamster> and _id is quite mongodb specific, you may want to rename the field to just 'id'.
[07:05:52] <crudson> alexisabril: ah ok. so you just want a string? docs.each { |d| d['_id'] = d['_id'}.to_s } perhaps
[07:08:10] <crudson> as far as munging the _id to a string in the native mongo query, I don't think so, without passing it through .aggregate()
[07:11:28] <crudson> if you're passing a reasonable result size in an api response, doing a .each in ruby shouldn't be terrible
[08:51:15] <magegu> hi guys, is there any way to read all database names through a connection via the c-driver?
[08:55:58] <jwilliams> if i want to partitionly read data from mongodb, is there any efficient way other than skip?
[08:59:46] <jwilliams> i have several processes that will read the collection. for example, there are 3 processes which will read 10 docs.
[09:08:02] <jwilliams> thanks. btw, is there any way i can estimate probably how long the skip may take ?
[09:42:12] <McNulty_> In the PHP driver, what's the bet way of handling replica sets? It seems like if I list 3 nodes and the first two are down, it takes timeout * 2 ms to connect as it waits for the first couple to fail.
[09:47:46] <McNulty_> We're thinking about somehow maintaining the server list in our application, but that seems like the wrong approach
[09:48:12] <McNulty_> Well I know that but isn't the point of the replica set that the set lives on...
[09:49:15] <Derick> yes, and it will still work. Once the driver has found it, it will not use a down node of course
[09:51:34] <Derick> yes, it is good to have more nodes, but for replicasets you generally expect all those nodes to be up
[09:53:07] <McNulty_> But one of the reasons (not the only one) to run a replica set is automatic failover right
[09:53:40] <NinjaPenguin> surely the entire POINT of a replica set is that you are anticipating the scenario when one (or multiple) nodes are 'down'
[09:55:47] <Derick> yes, but a connection is reused, so not really a problem as it will only try to reconnect to that failed host once every 30-60 seconds
[09:56:34] <Derick> oh, the driver just checks every 60 seconds whether a failed connection starts working again
[09:58:58] <McNulty_> so in a typical web app, I set the timeout to 2s and once a minute, one request takes 2s longer than it should?
[09:59:57] <McNulty_> one secondary is in a different data centre to the web servers so I probably can't drop it too low
[10:30:18] <schlitzer|freihe> with an sharded setup, where will mongos put a nonsharded colelction, only on the primary shard?
[10:31:14] <schlitzer|freihe> so there is a way to put a non sharded collection, on a not pimary shard?
[10:32:11] <McNulty_> the primary is per-collection so you can change it for one collection if that's what you want
[10:32:31] <Derick> schlitzer|freihe: http://docs.mongodb.org/manual/administration/sharding-architectures/#sharded-and-non-sharded-data
[10:32:54] <Derick> moveprimary allows you to change the "primary shard" for a non sharded collection
[10:43:24] <newcomer_> basically I'm new to the NoSQL world / just managed to setup mongo & inserted some random rows
[10:46:22] <newcomer_> exactly this http://stackoverflow.com/questions/10224364/calculate-distance-in-java-using-mongodb
[10:47:16] <Derick> newcomer_: you need to show exactly what you're doing, use a pastebin for your code
[10:52:58] <newcomer_> I suppose that should be my collection,but since I got an err I added my own above
[11:35:12] <jwilliams> if i use mongo shell to find a set of doc, how can i directly point to the last doc with javascript function?
[12:52:50] <JamesHarrison> So I've just taken a 2.0 single server (my development server), shut it down, replaced binaries with 2.2, and started up
[12:53:48] <JamesHarrison> A quick count() shows my data is still there as expected, but none of my queries are returning any data any more. Is this a common issue or have I got some kind of odd bug here?
[13:19:06] <remonvv> Hi all. Question; can anyone think of a practical way to wait for unsafe (w=0) writes to have finished without db.fsyncLock()?
[13:19:51] <remonvv> Running into issues where a distinct() doesn't pick up the w=0 writes until a little (random) time later
[14:08:26] <JamesHarrison> spillere: Only if your machine is compromised or you allow unfettered network access to the machine
[14:08:38] <remonvv> No, nobody uses MongoDB auth. Just make sure your database server is only accessible by your app server
[14:09:06] <JamesHarrison> if you have a firewall that blocks the outside world (and iirc mongo listens on localhost by default only anyway) and the machine itself is not compromised (and if it is, wouldn't matter if there were a password) it's secure
[14:17:25] <JamesHarrison> and only open what you specifically need the outside world to have access to
[14:53:08] <Almindor> I have user objects with this field in then "date": { "$date": 1300935091000.000000 }, how can I best query for all since 01.01.2011 ?
[15:06:08] <Almindor> NodeX so you're telling me I have to manually convert say '01.01.2011' into a long number so I can get a date query done right?
[16:52:22] <rintoul> Are there quotes missing from the JSON describing the usage of "$exists"? http://www.mongodb.org/display/DOCS/Advanced+Queries#AdvancedQueries-%24exists
[16:59:55] <bjartek> rintoul: no exists should be used like that to check if a field exists. do you have an example of the finding you would like to do?
[17:01:29] <rintoul> I've got it working, but I had to do something like "find( {"a":{"$exists":"true"}})" with the quotes around "a" ...
[17:02:03] <rintoul> I just wondered if I *had* to put quotations around the "a" and "$exists", etc.
[17:02:41] <hdm> hi folks, I have been fighting a db corruption issue all week, just noticed that mongod has an --objcheck option, which is likely the solution to random memory corruption by clients. As a sanity check, does that mean mongo implicitly trusts the incoming bson data to be valid by default?
[17:30:29] <hdm> kchodorow: thanks - that does explain things, the thing that kills me about it is the size field (once you get one broken record, the entire db is hosed)
[17:30:52] <hdm> fwiw, this is ruby driver/bson_ext/linux64, seeing corruption happen once in a while under really high upsert loads
[17:37:13] <vsmatck> Hm. Doesn't mongo have to deserialize the bson doc to look to see if _id is present? If it's already doing that can't it see when there's a decode error?
[17:41:52] <kchodorow> hdm: that isn't really true, a single messed-up bson doc can't hose the db. other corruption can, though
[17:43:49] <scoates> kchodorow: do you know if there's a way to `blockdev --setra` via sysctl or similar? I had a box reboot during cloudstorm, and I see just now that the RA is the old value. I'd like to puppet this.
[17:44:22] <kchodorow> scoates: there is a way to set it permanently, don't remember what it is though
[17:45:56] <hdm> kchodorow: tried 2.0.5, 2.0.6, 2.1.x versions, two different sets of SSDs, a NAS, local DAS, and did a full RAM verify
[17:46:01] <vsmatck> kchodorow: Ah. That seems reasonable. I guess when you're decoding you can come across something like a nested doc and you can easily skip it instead of spending time decoding it.
[17:46:17] <hdm> 12-48 hours to load the data set and i start getting invalid bson size errors during any table scan query
[17:47:18] <hdm> (it takes ~5-6 weeks of 12 cores to generate the dataset from the raw files, so im bootstrappign with the last known good backup)
[17:48:02] <hdm> the mongorestore works, no issues querying, then i start loading the last week's of queued data and within 20-30m records get corruption
[17:48:35] <kchodorow> hdm: can you run db.collName.validate() on the collection before you start upserting?
[17:57:09] <kchodorow> hdm: pass in true (validate(true)) if you have the time, does a more thorough job
[17:58:24] <hdm> (and any recommendation on the version to use? pegged at 2.0.6 just in case it was a dev branch issue)
[18:08:13] <hdm> thanks - i can always export and load into 2.1.x for analysis, its just the new data load that seems to trigger it
[18:09:13] <hdm> sort of related, can you add a replicate set member with a different version? or is that madness
[19:22:18] <Jesmith17> Is there any performance difference in mongo between insert() and save() for new documents? using Java API
[19:29:05] <callen> so I have a MongoDB database in staging, for which I can query a GeoPointField (MongoEngine way of doing geospatial) from my local computer and it will work fine, but return nothing if the same code runs on an actual staging server
[19:29:38] <callen> also if I fall-back to pymongo, why does it not work unless I immediately follow the index creation with the query and then the processing of the cursor?
[19:50:34] <hdm> kchodorow: verify came back solid, --objcheck set and starting to reload new data, crossing fingers
[20:34:36] <ra21vi> ok. I was benchmarking MongoDB in my company to introduce it. And wrote some benchmark (as I was asked), but it really showed a poor performance. The Server is on x64 Ubuntu while the benchmarking clients are being run on some Windows 7 Workstations. I then wrote the same thing in Java, and there was quite clear distinction. Benchmark in pymongo could insert 100000 docs in 90-100 secs avg,
[20:34:36] <ra21vi> max being 170secs. While same benchmark in Java inserted 100000 docs in 6.6 secs average. Then I realized it may be because Pymongo is running in pure python version. Since I installed it using pip in virtualenv, how can I find if compiled C extensions were installed too??
[20:36:35] <ra21vi> Do i have to install the MS VS Express to get those C extensions compiled when I install pymongo
[21:39:55] <hdm> might need to break shops out into its own document if you want the list of shops, not the list of companies that are "near"
[21:40:37] <hdm> you can use a m/r to pivot the collection into a new one that is mapped shop -> company
[21:41:40] <hdm> without a m/r, you can probably handle that via an aggregate query, with its limitations on group
[21:42:14] <hdm> then just query that cached pivot table, probably less work than redesigning your entire db
[22:09:45] <jhaddad> we've got a 500GB db in a 4 shard configuration. aws just announced their high i/o servers, which look like they might be better for us (since EBS performance was killing us). i'd like to move our setup to a replica set, but our indexes are bigger than available memory. does anyone have any tips/advice on working with mongo indexes that are bigger than ram but sitting on SSDs?
[22:10:39] <jhaddad> i guess what i'm asking is: does mongo intelligently bring portions of the index into memory, or does it just become a complete mess
[22:31:09] <linsys> jhaddad: mongodb uses mem mapped files, so as indexes (portions of indexes) are leveraged they are called into memory
[22:31:18] <vsmatck> Mongo uses a btree for indexes on _id which is typically a increasing key so it's right leaning. It relies upon the OS to figure out what pages to send to memory.
[22:32:31] <jhaddad> linsys: so the portions of the indexes that aren't part of the working set don't really matter, in that they won't be brought into memory and waste my availble ram