Log file Viewer
#mongodb logs for Thursday the 1st of November, 2012
[00:46:06] <therealkoopa> What's the best datatype for an encrypted object? Store it as a string? Or as binary
[02:47:43] <hdm> sorry, not familiar with the PHP api for it, in ruby;s driver you use update or upserts
[02:50:26] <IAD> och, embedded. $addToSet http://www.mongodb.org/display/DOCS/Updating#Updating-%24addToSetand%24each
[02:55:09] <Determinist> If I try to pull the users being followed by 1234, this doesn't really work.
[02:56:16] <Determinist> If I have to maintain names for both the follower userID and the userID of the user being followed, that means I'll have to keep updating the names in a bunch of docs when a user changes their name.
[03:04:12] <Determinist> IAD: you mean embed the names into the followers collection? What happens if a single user is following 6000 users and that user changes his name? That means an update command for 6000 documents. Sounds like a bad recipe for scaling? Unless, of course there's no way around that...
[03:06:41] <IAD> Determinist: you can use the _id field for link users (not username). it will be a little easier
[03:07:18] <Determinist> IAD: still doesn't aleviate the main bottlenecks of having to do massive updates, but what can ya do.
[03:18:34] <IAD> it will be like: db.[your_collection].update({_id : your_id}, {$set : {'ratings' : {$push : new_data_array}}})
[03:27:37] <JoseffB> $r = $ratingRec->update(array('_id' => new MongoId($pid)),array('$set' => array('ratings' => array('$push' => $obj))));
[03:34:47] <IAD> JoseffB: may be it will be helpful: http://stackoverflow.com/questions/10420635/mongodb-php-how-to-push-item-to-array-with-a-spcecific-key
[03:40:27] <IAD> You can use a save with full overwrite: $a= $collection->findOne($where); $a['ratings'][] = $new_array; $collection->save($a);
[04:52:04] <hdm> i am seeing performance drops on insert for a single instance of mongod, even when writes occur to different databases, these go away when i manually restart mongodb, and dont represent as higher % of faults (but higher % of locks)
[04:52:36] <hdm> any ideas why an instance when a new db every ~6 hours would spiral down on performance, with similar contents for each db, and inserts only into the latest
[04:53:36] <hdm> performance drops from 12k to 5k to 3k and now 1k per second over 48 hours, but across 7+ dbs, with only the latest being inserted
[04:54:15] <hdm> ex: 1357 1 0 0 0 2 0 681g 1363g 138g 0 critical_201208:30.5% 0 0|0 0|19 1m 3k 20 23:51:24
[05:08:53] <neiz> anyone famliar with the C# driver? Been trying to retrieve the only document in my collection (.FindOne()), but I cannot seem to coerce it into working
[05:19:47] <tpae> using tailable collection, can i listen for changes in that collection, and execute a command?
[05:29:06] <socket> from what i see, mongo does not require user/password. how does it make sure data isnt hacked?
[06:06:53] <holly1> Anybody know any mongodb admin UI's that support batch updates?: http://www.mongodb.org/display/DOCS/Admin+UIs
[06:54:28] <IAD> ali__: pretend to be the slave, or to have a central point for communication with the base
[08:10:47] <socke> hey dudes, im migrating from rdbms to mongo, i have a user system and a gps tracking system that tracks the users (which are cars). when designing a mongo db, should the user system be a part of the tracking system?
[08:11:46] <ron> socke: when moving to nosql dbs, your data structure should almost always conform to the way you want to query your data rather than how you want to save it.
[08:14:30] <socke> the thing is, i need to do user authentication first and then start track the user. at later point i'd like to see the path the user took on a given time frame
[10:21:57] <samurai2> hi there, which one is faster : 1. filtering before map/reduce or 2. filtering inside map function? thanks
[10:23:53] <kali> samurai2: make sure you have a look at the aggregation framework too. anything you can do with it will be more efficient than map/reduce
[10:25:31] <samurai2> kali : but aggregation framework has limitation of only 16 MB on it's result right?
[10:26:24] <kali> samurai2: sure. but if you're generating a result bigger than 16MB, chances are you're abusing mongodb
[10:27:51] <samurai2> kali : and if I need to also filtering based on geospatial indexing it won't work as well right?
[10:28:23] <kali> samurai2: i'm not clear on what work and does not on geospatial indexing, i'm not using it
[10:30:27] <samurai2> I'm hoping it can directly output the result into a certain collection and also we can decide the output key, not only key and value. :)
[11:22:23] <topriddy> got this error from a java/morphia app. out of memory. mongodb top suspects http://pastebin.com/mFrgnWJz
[11:25:14] <kali> topriddy: but you need probably need to get some insight about what's going on in your app too
[11:26:12] <kali> topriddy: try -verbosegc, to start, but then you may need something more invasive (try yourkit profiler for instance, there is a trial)
[11:27:12] <kali> topriddy: the fact mongodb appears so much in the stack is not that suspect, a database driver is likely to allocate lots of things
[11:30:51] <topriddy> kali: hmm...am suspecting maybe the jvm is simply the problem. it probably needs more heapspace.
[11:31:44] <kali> topriddy: you can try and up Xmx, but if the problem does not go away, you'll have to profile gc and memory usage
[11:33:18] <kali> topriddy: well, unfortunately, this is a the real world of the jvm in production :)
[11:34:29] <topriddy> kali: thanks man. you have been helpful. i sure hope mongodb not good enough for production is not the culprit though
[11:36:28] <kali> topriddy: this is memory consumption in the jvm, not in mongodb. i'm pretty sure there is no hidden dragons in the mongo java driver
[11:38:17] <kali> topriddy: usually this is due to bad application code (like overenthusiastic caching, leading to reference/memory leak) or heap size not big enough for the jvm working set
[12:14:14] <topriddy> kali: how do you even cache explicitly? i'm not doing any caching or read about it in mongo yet
[12:34:20] <fatninja> I have the var a = "string"; and I want to search all fields that contain a, so I should do something like db.users.find({name: /a/}); But we know that doesn't work, (using server side javascript)
[12:39:01] <fatninja> NodeX, I use a driver to communicate to the db. But I think the solution is this: var regex = new RegExp(a, "g"); and {name: regex}
[13:28:42] <NodeX> 'Apple tried to argue that it would take at least 14 days to put a corrective statement on the site – a claim that one judge said he "cannot believe".'
[13:46:11] <robo> hello: right now we're using replica sets and I'm seeing heavy I/O across all are replica set nodes because of the overhead of replication itself I believe. I'm trying to talk our DBA's into going with shards but they are a bit nervous because of the testing. Anyone have any gems of wisdom to drop on this?
[13:47:14] <NodeX> or to put another way to put it is.... Apple are overpriced manufacturors of equipment that use open source cores and close them off for their own personal gain
[13:49:27] <robo> skot, https://mms.10gen.com/dashboard/view/4f5f957a87d1d86fa8b8821c#chartLinkMinuteOneHour
[13:55:34] <skot> If you look at MMS: https://mms.10gen.com/host/chartReplicaSetHosts/4f5f957a87d1d86fa8b8821c/mongodbprod#chartLinkMinuteTwelveHours
[13:58:15] <robo> n/m, i see that opcounter graph that shows what it was doing when you hold your mouse over it
[13:59:09] <robo> so they have to run these to keep our data fresh. Problem is with replica sets is that it has to replicate to all nodes. I figure that if you use shards it will cut down on the overall i/o used
[14:00:23] <skot> I wouldn't suggest sharding for this issue, just making the process less impactful or adding more resources to the nodes.
[14:01:25] <robo> skot, what do you think of sharding overall? I talked to a few people that hit some major bugs with it
[14:01:45] <robo> i think it's a must-do for us because it's going to be hard to scale with replica sets as we put more traffic on it
[14:02:25] <skot> your set doesn't seem to have traffic patterns where a single instance can't handle the load, does it?
[14:02:44] <robo> nope. But we only have about 20% of our traffic going to it right now (slowly moving off oracle.)
[14:03:51] <skot> there is probably lots you can optimize in your app/queries/indexes before you need sharding
[14:09:05] <kali> throttling imports on a production db is a must, or mongo will eat all it can until it chokes
[14:32:27] <wiseguysonly> I'm a bit unsure how I would do this in Mongo -> SELECT SUM(field1) GROUP BY DAY(date_field). So I get all days available
[15:07:21] <kali> mrpro: process memory is irrelevant. the indexes will be in the kernel cache and buffer zones
[16:24:29] <gshipley> Anyone know if 2.2.1 is just a bugfix release with the 3-4 issues listed in the change doc? I am not sure I am reading it correctly?
[16:41:43] <_m> According to their server's JIRA: Odd version numbers are unstable dev branches (1.1, 1.3, 1.5) Even numbers are stable branches and only get bug fixes, etc... (1.2, 1.4, 1.6)
[17:01:22] <mikejw> with doctrine odm is it possible to remove a reference to a document from within a executed querybuilder cursor?
[17:31:21] <wereHamster> does mongodb sort by unique documents or can a document appear multiple times if the query matches it multiple times?
[17:32:41] <wereHamster> I basically have: Game = { players: [...]}, each player has a score, and I want to rank games by the highest score any player has reached in it.
[17:33:17] <mikejw> on second thoughts I guess it doesn't really make sense retrieving objects from the cursor more than once :)
[17:33:19] <wereHamster> or should I create a field on the Game (maxScoreByAnyPlayer) and sort by that?
[17:35:07] <wereHamster> so it would be something like: db.games.find(...).sort({ 'players.score': -1 })
[17:35:53] <hdm> wereHamster: yup, just index it with db.games.ensureIndex({ 'players.score' : -1 })
[17:36:00] <wereHamster> I guess it's probably better to make a separate field on the Game document. Makes the index smaller.
[19:37:05] <andrecolin> have a setup running auth, i am using a simple ruby script, i can login to the admin db, change to my test db, can query the collections and i see the system.users collection, how can i access the user names in that collection using my code
[19:45:43] <_m> andrecolin: Pastie or gist the error you're getting and we'll be better able to help you.
[19:48:38] <_m> Is my_db really a hash? How did you create the connection? What are the actual errors? Pastie/gist so we have enough context to actually help.
[19:49:48] <bricker> Hello - at /var/lib/mongo I have a bunch of files of databasename.1, databasename.2, etc. Are these backups (i.e., safe to delete the older ones)?
[19:52:23] <crudson> bricker: that's your data, it gets split up into files of increasing size, maxing at 2gb(ish), but also preallocated before you hit the limit
[19:53:05] <crudson> bricker: not quite the equation I gave as they start out small, but that's the general idea
[19:53:48] <crudson> andrecolin: my_db["system.users"].each should be my_db["system.users"].find.each
[19:56:42] <crudson> bricker: you'll see 64M, 128M, 256M, 1G, 2G, 2G, 2G, 2G....... http://www.mongodb.org/display/DOCS/Excessive+Disk+Space#ExcessiveDiskSpace-DatafilePreallocation
[19:57:08] <andrecolin> crudson: my test box just went south, need to reboot, will be back with status
[20:14:55] <bricker> Forgive me, I'm new to mongodb and am afraid of deleting something that I didn't mean to. Will this query remove any entries with the timestamp before October 1? db.stream_minutes_events.remove("t" : { $lt: ISODate("2012-10-01T00:00:00.000Z") })
[20:15:26] <bricker> Forgive me, I'm new to mongodb and am afraid of deleting something that I didn't mean to. Will this query remove any entries with the timestamp before October 1? db.stream_minutes_events.remove({ "t" : { $lt: ISODate("2012-10-01T00:00:00.000Z") } })
[20:15:40] <bricker> db.stream_minutes_events.remove({ "t" : { $lt: ISODate("2012-10-01T00:00:00.000Z") } })
[20:16:35] <naquad> can i use mongodb for high load project (i'm talking about 10k-25k concurrent requests)?
[20:19:06] <hdm> naquad: yes, you might need a few shards, and make sure queries are working from indexes in ram, SSDs help with high throughput too
[20:19:15] <crudson> bricker: I'd shove a load of documents with varying dates into a test collection and check it.
[20:20:21] <hdm> its someone's experience, everyone has their own opinion, and until a few months ago i had unrelated headaches that drove me crazy
[20:20:44] <hdm> but mongo 2.2 has been a huge improvement overall and msot of the issues i ran into are now just a matter of allocation/sizing up front
[20:21:55] <hdm> the safe writes thing is overblown, you can enable it in your driver, or not, it depends if you need confirmation of every write, some uses cases dont
[20:22:25] <hdm> the sharding stuff is still suboptimal imo, ive seen weird/bad balancing, and adding/removing shards is a pain, especially compared to something like elastic
[20:23:47] <hdm> fwiw, i run a ~1.5 billion record / 3.5Tb dataset for a hobby project, my roadblocks tend to be things like concurrent cpu use since im focusing on single/huge nodes, or slow disk io once indexes go over ram
[20:24:27] <hdm> using mongo properly with horizontal scaling/replica sets/sharding would work better if i could afford to
[20:25:31] <hdm> https://speakerdeck.com/hdm/derbycon-2012-the-wild-west <- not much there about the mongo setup, but thats the source of it
[20:30:55] <mrichman> are there any sample databases out there to speak off, something like Northwind in SQL Server? Just need to sink my teeth into something real
[20:33:42] <mrichman> understood…but say i wanted to play with map reduce on a large dataset…kinda hard unless i contrive the data myself….seems like there should be some archetypal examples out there
[20:34:49] <hdm> http://aws.amazon.com/publicdatasets/ < lots of stuff there and there -> https://explore.data.gov/catalog/raw/
[20:42:27] <TecnoBrat> normally ISPs just kick you rather than letting you do educational stuff like that
[20:44:26] <hdm> TechCel: https://twitter.com/hdmoore/status/262080277915004928/photo/1 < two weeks ago
[20:48:12] <hdm> hmm, so back to mongo, how do you do the equivalent of "use database" in JS scripts?
[20:49:36] <hdm> single server for most of it, 2x8-core xeon, 192gb ram, 1T of 4xSSD, 40Tb NAS, 3x4Tb direct HDDs
[20:50:51] <hdm> sharding is nice on single-boxes since you get to use multiple cpus for queries that do calculations
[20:51:20] <hdm> yeah, changes depending on what set, im reloading it to a differetn schema on one of the boxes almost all the time
[20:52:03] <hdm> VMs failed miserably (kvm and vmware), weird storage issues, multiple mongods was ok, but sharding fell on its face to the point it wouldnt scale
[20:52:30] <hdm> the primary shard ended up with 3 x the data of the others and the disk i/o was maxed, so it couldnt slough it off and handle incoming inserts fast enough
[20:53:09] <hdm> even with only _id indexes, the read overhead got crazy, at the time drives were raid-0 x 4, so decent number of spindles
[20:53:45] <hdm> the 1T ssds (raid-0 as well) get 1000MB r/w and 25,000 IOPS, but only go to 1T, which doesnt work so well for the current size
[20:55:20] <hdm> not any more, it choked after 3 weeks with the primary unable to hand off fast enough
[20:55:44] <hdm> currently setup is to create new dbs on the fly for each month and put unique contraints into a multi-key _id
[20:56:45] <hdm> its reloading the data still, but going way faster, better than mongo's normal downward spiral as the index goes beyond ram, your insert performance dies
[21:14:30] <Guest46986> Following this information: http://docs.mongodb.org/manual/tutorial/deploy-replica-set/
[21:14:30] <Guest46986> I've tried time and time again. And all I can get is: exception: need most members up to reconfigure, not ok.
[21:14:30] <Guest46986> I was able to run rs.initiate() once, looked like it worked, so I did it again for the other instance,
[21:14:30] <Guest46986> Of course I did some looking around, I noticed mention of a "\etc\hosts" <--- The heck is that? Do I need it? Is there something else I'm supposed to review that I might have missed?
[21:28:16] <Guest46986> If anyone is available to help me solve my issue, I've posted it here as well: https://groups.google.com/forum/?fromgroups=#!topic/mongodb-user/QTqV1G3LUGU
[21:34:26] <Guest46986> Following this information: http://docs.mongodb.org/manual/tutorial/deploy-replica-set/
[21:34:26] <Guest46986> I've tried time and time again. And all I can get is: exception: need most members up to reconfigure, not ok.
[21:34:26] <Guest46986> All three instances are up, followed the documentation just fine. What's the deal?
[21:34:26] <Guest46986> I was able to run rs.initiate() once, looked like it worked, so I did it again for the other instance,
[21:34:27] <Guest46986> Of course I did some looking around, I noticed mention of a "\etc\hosts" <--- The heck is that? Do I need it? If so why didn't I come across it
[21:34:27] <Guest46986> in the installation or mongo setup documentation? Is there something else I'm supposed to read that I might have missed?
[21:35:15] <cyberfart> I have a quick question: are "$in" queries considered equality test or range query? I'm trying to work out the better compound_index ordering
[21:39:49] <TecnoBrat> cyberfart: its not a range ... its an array. You are finding any records which have a value that exists in the array
[21:54:40] <Guest46986> Following this information: http://docs.mongodb.org/manual/tutorial/deploy-replica-set/
[21:54:41] <Guest46986> I've tried time and time again. And all I can get is: exception: need most members up to reconfigure, not ok.
[21:54:41] <Guest46986> All three instances are up, followed the documentation just fine. What's the deal?
[21:54:41] <Guest46986> I was able to run rs.initiate() once, looked like it worked, so I did it again for the other instance,
[21:54:41] <Guest46986> Of course I did some looking around, I noticed mention of a "\etc\hosts" <--- The heck is that? Do I need it? If so why didn't I come across it
[21:54:41] <Guest46986> in the installation or mongo setup documentation? Is there something else I'm supposed to read that I might have missed?
[21:55:39] <raja> how can i do a query where i don't want a unique on a field but a limit.. for example.. i want to query all stories but limit it to 2 per userid
[22:02:54] <Guest46986> This is depressing. Can someone at least shoot me a link to a decent tutorial on how to set up replication sets?
[22:03:06] <raja> hdm: looking at aggregation docs but don't see it.. aggregation supports distinct by key but not a limit by key value
[22:03:13] <hdm> Guest46986: you ignored previous advice and then pasted the same thing twice, no thanks
[22:07:01] <Guest46986> Look I'm sorry I upset you, was not my intention. I just figured you where talking to someone else. Don't see why you would being up /etc/hosts is you where.
[22:11:50] <Guest46986> I do not understand where I should put these directories or configure them. THey where not covered in the documentation I followed.
[22:14:58] <Guest46986> Well thanks for responding at least. Then is there somehting I can do? Some other documentation I can follow for windows?
[23:17:29] <jrdn> if a value is known in a column, is it the same performance doing, x = 1; x++; db.foo.update(criteria, { $set: { bar: x } }
[23:18:42] <jrdn> i was just wondering if i should build increment functionality in my domain model persistence manager since the update needs the domain model
[23:19:12] <Zelest> somewhat odd question; a server with only ~10MB/s write-speed, is that usable for mongodb?
[23:19:59] <ron> jrdn: using the $inc is an atomic operation. using the first option is not. in a multi-threaded environment, that matters.
[23:21:58] <jrdn> Zelest, yeah we're on EC2 on a regular EBS volume and were getting high IO alerts every minute when data is being flushed to disk
[23:23:08] <jrdn> having my application do $x += 10, then update with $set vs $inc with value of 10 is micro micro micro
[23:30:19] <bricker> When I delete entries from a collection, when does that data get removed from disk?
[23:42:23] <bricker> wereHamster: Thanks, just wanted to make sure I didn't have to do it manually or something :)
[23:45:01] <bricker> wereHamster: Oh... because we have some data that is taking up a huge amount of space on our server, and we're just trying to clean it up