Log file Viewer
#mongodb logs for Tuesday the 18th of September, 2012
[00:31:28] <_m> Also see: http://stackoverflow.com/questions/7021678/how-can-i-handle-datetime-in-mongodb-with-php
[00:32:19] <_m> Think the key is that PHP's DateTime doesn't know anything about Timezone information.
[02:59:05] <wachpwnski> Here is my schema, I am wondering if I can limit the size of the logins array. https://gist.github.com/b383ebc37311a93d4e23
[03:43:37] <jgornick> Hey guys, can I use the aggregation framework to properly query and return on embedded documents? For example, can I create an aggregation query to query embedded documents and return those as the result?
[03:55:20] <crudson> wachpwnski: you can't impose a 'rule' that results in mongodb enforcing that outside of application or maintenance code
[04:05:30] <wachpwnski> crudson: would it be better to just keep a circular queue and reference the user?
[04:15:12] <crudson> wachpwnski: If you want to retain n logins for each user then a single capped collection referencing user would not be a good solution. I'd maintain it on a per user basis. I'd look at, vote for, and keep an eye on this: https://jira.mongodb.org/browse/SERVER-991
[04:35:35] <crudson> wachpwnski: note that size of array isn't available unless you track it (other than querying $size). You will have to do some manually work to maintain the array. What do you mean by "global collection incrementer"? Like a traditional db sequence?
[04:39:17] <wachpwnski> crudson: I am building a forum, so i basically do /forum/<title>.<counter> for seo
[04:39:54] <wachpwnski> Using sql with an auto_incrementing id this makes sense. is there a way that is more practical with mongo?
[05:57:38] <ron> well, clucene is a port of lucene. I don't know how far back it dates. 'fastest' is a problematic description.
[06:01:16] <timeturner> http://clucene.git.sourceforge.net/git/gitweb.cgi?p=clucene/clucene;a=log;h=HEAD
[06:03:41] <ron> that's relatively old. lucene underwent quite a few changes since. however, that doesn't mean clucene isn't good. nevertheless, you should look at the functionality and not only the 'speed'.
[06:07:13] <timeturner> if the search bar in elasticsearch.org is using elasticsearch then it is really slow
[06:09:17] <timeturner> google is instantaneous. I would expect at least 1/6 - 1/4 the speed of that
[08:27:47] <VinX> hello all, I am looking about how to embed mongodb ( server ) in a c++ application.
[08:40:16] <wiherek> hi. is there a channel specifically for mondodb - nodejs (using 10gen drivers)? or is this the one?
[09:52:22] <xenocom> I cant find any info on this, Im' hoping someone can point me in the right direction
[09:52:28] <arussel> I'm implementing a search using the 'split words and put them in a keywords field' patter
[09:52:44] <xenocom> if I want to run mongodb on AWS... or at all, really... how can I compact without pulling my site offline?
[09:52:52] <arussel> ideally, I would do the splitting/setting in an create/update listener, is there something like this ?
[10:05:40] <NodeX> arussel : can you explain your problem any differently, what you are asking doesn't make sense
[12:08:58] <driek> one of my config servers fails on update config.mongos query, the logs tell me 'exception: Invalid BSONObj size: 0'. I restarted all the config servers but this does not help. Any clue where to look?
[12:10:32] <boll> Is there anything in the mongodb development pipeline focusing on making gridfs work with reasonably sized data sets (ie. 300+ GB worth of file data)?
[13:01:03] <boll> NodeX: Anyway, we have 300 GB of stuff in gridfs, and it is causing problems with other writes
[13:03:31] <boll> I have an mms-log showing what happens to lock percentages when I drop the 300+ GB gridfs collections
[13:10:12] <NodeX> boll: have you not thought of putting a cache in the middle of your app for this?
[13:13:24] <driek> I cant seem to disable my balancer, 'db.locks.find( { _id : "balancer" } ).pretty()' shows a lock in state 1. Should I just be more patient to let the balancing round finish, or is something wrong/stale?
[13:17:17] <amirc> NodeX : You mean to do a 'manual' counter? after each call to run update with $inc
[13:30:21] <driek> is it possible that stale locks are there because of a config server migration? I did stop the server before rsyncing it tho...
[13:36:10] <NodeX> if you shift the grid away from mongo you will only compound the problem, the files are mmapes by the operating system for a reason (MRU)
[13:37:41] <boll> NodeX: keeping the files in a flat file system with logarithmic access times currently seems a lot more plausible
[13:39:28] <boll> problem is that LRU isn't really a terribly good algorithm, as the files get hit more or less randomly
[13:41:44] <NodeX> you can be smart about it with somethign like redis ... use the id as an expring key, check the key before checking the DB - if exists then serve from memory/cache else serve from Mongo.. will eat a bit of ram / disk but will relieve your system a bit
[13:42:44] <boll> except if old files are as likely to get hit as old ones, that doesn't really gain me a lot, does it?
[13:44:26] <boll> it's a completely generic image manipulation service (written in php), which downloads images, manipulates them, and returns the manipulated image to the caller
[13:45:30] <boll> since all images are theoretically equally likely to be requested at any given time
[13:45:33] <NodeX> I imagine in your case the original is not really needed after it's been manipulated?
[13:46:21] <boll> The original may be 150 MB, and the caller usually requests 4 or 5 different "versions" of it, so keeping the original around makes a bunch of sense
[13:47:16] <boll> yeah, cause different callers (or client software) often requests different sizes of the same image (depending on the screen size of the client)
[13:48:08] <NodeX> if it was me I would keep hot data in a cache for X days (dependant on ram) and store the originals outside of mongo after manipulation
[13:48:41] <boll> Yeah, we may end up having to do that, it's just that gridfs is damn convenient :)
[13:48:52] <NodeX> if a new request to manipulate comes down the wire it get's added to the queue (load the original from the FS) and start all over again
[13:49:30] <NodeX> or... setup a second mongod to house originals - as they are infrequently accessed they won't hardly be mmapped
[16:03:05] <gyre007> anyone here uses mongo community cookbook ? we are and just found out that it keeps restarting mongo every time chef-client runs...needless to say the restart forces master to change to slave...and some other node becomes master..
[16:45:37] <Almindor> what's the fastest way to update references? say I have a list of users with ObjectIds pointing to some collection A but I want to "exchange" that with another collection B and A is linked to B (each A has some link to B, not vice versa tho!)
[16:47:13] <ron> you mean that updating ALL the records in your collection is slow? who would have guessed...
[17:00:23] <ron> well, that's not surprising considering... well... considering you CAN'T modify the _id.
[17:04:24] <Vile> due to the fact that you don't have to re-index, but rather create new indexes only
[17:04:27] <ron> well, the whole idea of actually having to update the _id field baffles me, but okay.
[17:05:54] <ron> well, just goes to teach you that you shouldn't fight against objectid as the _id :)
[17:07:39] <Vile> ron: in my case it makes sense. my ids are unique. Only problem that the software that was feeding them was feeding wrong values
[17:12:45] <Vile> anyway, in that case objectID is completely useless, because will not be used at all
[17:56:01] <Neptu> i have a collection where some fields are A = 1 another ones are A = 2, is it possible to get a query to retrieve all of them that has te bigger integer
[18:01:20] <cmendes0101> oh, I'm still new to mongo. but I saw that in the Aggregate framework. $max. That might be what you want.
[18:01:41] <cmendes0101> Or you can do a find, sort by A then pull the last or first value, however you sorted
[18:07:36] <cmendes0101> Well that was cli, but I would assume so. There very basic commands. You just need to convert to talk with the cpp driver
[18:08:48] <cmendes0101> I think thats where the aggregate framework comes into play, it can handle more (I think)
[18:11:34] <cmendes0101> I guess I dont know what your dataset looks like or what your doing but check out http://docs.mongodb.org/manual/reference/aggregation/, Search for $max. Its for grouping but maybe it will work for you
[18:12:36] <ludof> Hey, we need some help here for a "special" operation. We're moving from 1 mongos + 20 single shards to 3 mongos + 3 replicated shards. We're using mongodump from the old mongos to grab 800 gigs of data (one collection. Yet the process "almost" fail, it's terribly slow.
[18:14:17] <ludof> I/O are low on the active shard (100 reads/sec, less than 10 writes/sec), ram usage is low, load average is also low, mms give us 2 informations: average of 80 page faults and average of 4Mbits in network.
[18:40:22] <augustl> what's a good way to implement audit trails for documents? I was thinking of having a "history" array on the document itself, copying the entire document in on every change. I guess this'll work, a bit worried about the 16mb limit on document size though
[18:58:48] <augustl> ludof: the reason I liked the history array was that I thought that would be an atomic operatoin, but on second thought it won't.. So I might as well use that
[19:16:10] <arussel> I want to update a field of a document when the document is updated/created. Is there something like listener/trigger I could use.
[19:24:19] <arussel> is there a reason for not implementing it or is it just that no one motivated enough needed it
[19:24:52] <NodeX> because a key value store is built for speed and schemaless properties and most of all scalability
[19:25:30] <NodeX> things like that should -allways- be appside logic, it just so happens a lazy developer added them directly into SQL
[19:25:46] <ron> actually, I believe it's a matter of demand. should the community ask for it, they may consider implementing it. it _will_ complicate sharded databases and such.
[19:27:00] <ron> well, wrt to slowing things down, I believe it's up to the user to decide what's more important to them.
[19:27:43] <NodeX> it would slow down the core as i's effectivly an extra index to check for what to do / what to update
[19:27:51] <arussel> I'm very new to mongo so apologies for question, but isn't a document always persisted as a whole (so sharding doesn't really matter).
[19:28:37] <ron> well, some other nosql dbs offer triggers (to an extent), so I wouldn't say 'ever'.
[19:30:20] <NodeX> mongo is not trying to replace SQL, if people want triggers or other SQL based things then use SQL l;ol
[19:31:53] <arussel> no more like: That could be an appside problem, but if you want, you can do it inside mongo too.
[19:33:36] <NodeX> and regarding your sharding issue/comment .. documents are more analogous to rows in SQL and collections to tables
[19:34:43] <arussel> then updating a field of a doc using field from the same doc wouldn't be a problem in a sharded DB.
[19:34:53] <arussel> "Your code must split the title above into the keywords before saving. Note that this code (which is not part of Mongo DB) could do stemming, etc. too. (Perhaps someone in the community would like to write a standard module that does this...)"
[19:35:24] <arussel> when they say module, does this mean the module would update the doc, and the code would talk to the module ?
[19:38:04] <NodeX> if you're looking to do LIKE %% style queries in mongo you're going to be disapointed
[19:39:16] <Neptu> sort.($lala: -1) should give me a reverse list starting from the highest number to the smaller number right??
[19:39:22] <arussel> I need to implement a simple full text search and try to find the best way to do it
[19:39:30] <Neptu> sort.($lala: -1) should give me a reverse list starting from the highest number to the smaller number right??
[19:56:23] <NodeX> you can pretty much build it from the examples here http://docs.mongodb.org/manual/reference/aggregation/
[20:00:47] <NodeX> My brain is fried for the day, if you can wait until tomrrow I'll help you work it out
[20:05:20] <gigo1980> hi how can i make in an sharding collection inside an map of an MR an db.mycollection.find() ?
[20:09:00] <Neptu> mmmmmmm when u do a find().sort() the find gets executed first and then it sorts it out??
[20:30:10] <_Tristan> I have users and items. Users have many items, and most of my reads will be looking up an individual item by it's id, along with it's owner's name. My question is, should I store users and items in separate collections, or should I store the items within the user's document?
[20:46:14] <mcav> If I have an index of {a: 1, b: 1}, is it redundant to have separate indexes for {a: 1} and {b: 1}? A quick test of the repl seems like it uses a btreecursor even if querying for just "a".
[23:47:10] <_m> While they aren't exactly "aggregation framework" docs, you'll likely be able to do the same sort of operations.
[23:48:23] <cmendes0101> and im using aggregate(). Not sure how to execute something like that in there
[23:50:38] <cmendes0101> I've tried many options. I haven't specifically tried a map reduce since I didn't think I can execute that within the aggregate command but I'll see what that triggers
[23:51:15] <cmendes0101> Basically I'm doing $group to group by an ID and those records contain a number of seconds. I'm trying to get minutes rounded up. I have the minutes returning but there not rounded up, so the totals are not correct