Log file Viewer
#mongodb logs for Wednesday the 8th of August, 2012
[01:08:54] <acidjazz> vsmatck: it points to the mongodb doc page which does not specify how to check if its on
[02:42:39] <warz> can you create an index on a property and put ObjectIds in it? ive got an index on a field, and am putting ObjectIds into it. my queries don't seem to be returning anything, though.
[04:41:02] <circlicious> my mongod wont start, this is the error, what should i do, should i repair ? http://pastie.org/private/fstcecyi0solcpxbgoppta i can see the /data/db/journal along with files in it, dunnpo why its happening, ugh
[05:16:39] <skot> Do you have the logs from last time it was shutdown, before the current error message?
[07:10:49] <lizzin> so i have a db full of collections that consist of info on various business including location(in lat, long),name, address, type of business and a few other misc info
[07:12:09] <lizzin> then the foursquare checkin provides my app with a lat and a long. but this lat and long rarely ever match up 100% with the data in the db. kind of expected...
[07:13:30] <lizzin> the best i can come up with is to take the foursquare lat+long then do a near query on the db and then do some sort of funky 'address/name' matching
[08:51:05] <mbuf> is there a tutorial that I can follow to write unit, functional tests for Rails with mongodb?
[08:58:41] <remonvv> there is no in-memory version and I'm not aware of any language that has a full MongoDB mock implementation.
[08:59:05] <remonvv> With difficulty. Our integration tests run mongod instances, load test data, run the test, and kill the instance.
[09:00:42] <remonvv> In Java we use basic JUnit for our tests where each test case starts a new mongod instance.
[09:01:51] <remonvv> Or someone else really. We need mock implementation of at least the basic subset of MongoDB functionality.
[09:21:08] <ron> last week we discovered that a guy from work things that indie games are made... in india.
[09:26:32] <remonvv> All Sparcs are big-endian iirc, hence MongoDB not compiling on them for the timing being ;)
[09:30:37] <ron> niemeyer: hello, and welcome to #mongodb. we hope you enjoy your stay. if you have any questions, please feel free to address them to the general public in the channel. please come again.
[09:50:20] <NodeX> and yet my apps/sites are top 1% of speed in the world, imagine what would happen if I wrote in a fast language :P
[09:51:17] <_johnny> NodeX: out of curiocity, do you use php -> bytecode at all in any of your projects?
[09:58:02] <_johnny> i'm out of the php loop, so not sure what's the latest, but yeah, that's what i meant
[10:00:07] <NodeX> my bottlenecks are -almost- always network card / bandwidth so it's not always needed
[10:00:57] <_johnny> at least in my experience. sure languages will vary, but it's not the most speed unfluencial :P
[10:11:01] <_johnny> well, not at the end, but wasn't the whole point of hiphop to use it before compilation?
[10:12:24] <_johnny> php was big back then, and rewriting everything just because people prefer ruby now, and then to scala in a few years because whatever, isn't "smart" :P
[10:46:56] <Bartzy> if the server crashed before the journal synced to disk - it is in dirty state ?
[10:49:49] <Bartzy> And still, if it the data files are corrupted during a fail - why the journal helps ?
[10:49:55] <Derick> the files are smaller, and it's append only, so that just makes it a lot faster operation
[11:15:15] <Bartzy> I read an example - if you have a million user, but only half are active, then only those half data is your working set
[11:16:35] <Bartzy> working set is different in time. The working set per second is very different than the working set per minute
[11:19:44] <remonvv> Bartzy, working set is your hot data. Hot data in the MongoDB sense is data that is in physical memory due to the OS swapping it in there based on frequent access/MRU.
[11:21:10] <remonvv> Bartzy, the working set isn't really defined per specific timespan. It's all the data MongoDB can keep into physical memory. That's why things like right balancing for large indexes is as important as limiting the amount of hot data you have at any point in time (if possible, which it isn't always)
[11:21:44] <remonvv> cmex, take a chill pill dude :) C# is not as common as other options. If someone is working with it and is reading this I'm sure they'll respond.
[11:22:03] <Bartzy> Ideally it's working set + indexes in RAM. But what is the working set. If it's the data mongodb can keep into RAM, then I can never know how much RAM I need :)
[11:22:47] <remonvv> Bartzy, hard to say. Easy answer is as much as you can afford, the slightly more complicated answer involves testing or estimating your data sizes and which chunk of it will be hot at any point in time.
[11:23:23] <S7> Hi, i'm coming from the sql world, we're usually creating a stored procedure and then calling it from the code, is it a good practise creating a stored javascript and calling from the code or it's best to avoid?
[11:23:58] <remonvv> There's a reason a lot of developers/companies lean towards "try it and see" ways of determining hardware costs/utilization. It's becoming increasingly hard to predict what kind of resources deliver what kind of performance.
[11:24:46] <remonvv> S7, anything JavaScript is a bad idea for anything where performance is an issue so if you'd normally write a stored procedure (performance optimalisation) then in MongoDB you should go for a native query.
[11:25:16] <remonvv> cmex, not sure. Just not that many C# developers here. Java, PHP, Python, C and Ruby are most common.
[11:26:17] <Bartzy> remonvv: We can afford a 512GB RAM server. Doesn't mean we want to waste money :|
[11:29:59] <remonvv> Bartzy, I'm not saying get the biggest server you can find. I'm just saying create a realistic test and run it on different hardware. Performance profiles/bottlenecks are pretty straightforward for MongoDB.
[11:30:27] <remonvv> Bartzy, you need more RAM if your pagefaults/second are non-zero/high continuously.
[11:30:33] <gigo1980> hi all, i have a map reduce function in the map reduce function i all create an dynamic array with hashes as key ? is this posible ?
[11:30:41] <Bartzy> remonvv: Right, but still, there are recommendations everywhere on the web to get as much RAM as the index size + working set. And I didn't find a clue about what a working set really is.
[11:30:55] <remonvv> A lot of people that ask these things tend to find out they can do what they need on relatively little hardware.
[11:33:13] <remonvv> Basically think of it like this; for queries that are important for your application speed as much *AS POSSIBLE* should be in memory. Most application we make at my company tend to have working sets of only a few Gb even though we serve over 100,000 concurrent users on a frequent basis.
[11:34:03] <remonvv> So look at your software, evaluate which queries are executed frequently and go from there.
[11:34:19] <remonvv> I'm assuming this is so relevant for you because you intend to buy hardware rather than rent it?
[11:34:57] <Bartzy> But I just reread some of the stuff in MongoDB in action and was curious, because it's so unintuitive.
[11:35:48] <remonvv> It's not. It's roughly similar to any other database. What makes it slightly less intuitive is that MongoDB's storage engine is built directly on top of OS mapped memory functionality.
[11:36:15] <Bartzy> Well, I know the queries that are executed frequently. Those are for active users. But what happens when a active user that didn't visit the app, visits it now and needs his/her data shown. It's not in RAM, so they have a bad experience. If they come back soon, it may be in RAM.... Very hard to know
[11:36:42] <remonvv> Well, keep in mind we're usually talking about the difference between 0.01ms and 1ms
[11:37:26] <Bartzy> We have SSDs as the storage backend, in RAID 10, and queries not from RAM takes 500ms+
[11:38:47] <remonvv> Bartzy, that isn't really relevant. The time added for a pagefault is the diskswap. Are you sure you're testing single doc queries and that your indexes are hitting?
[11:39:46] <Bartzy> I can test now for not-in-RAM single doc query. How do I clean the cache? Only by restart ?
[11:40:26] <remonvv> It's not a super useful test anyway. Just do a smoke test on your hardware and see what's bottlenecking
[11:41:31] <remonvv> well, high pagefaults means it's swappig a lot, in/out stats give you network bottlenecks if any, and lock % is extremely relevant to overall query performance
[11:43:25] <Derick> restart of mongodb doesn't really help either... you need to reboot the machine really :-)
[11:44:42] <Bartzy> remonvv: How do you did that one-doc test after restart - if the index is not in RAM ? :p
[11:45:20] <remonvv> Worst case scenario is nothing in ram, and the index pages are swapped in and out just like normal data.
[11:45:39] <Bartzy> when I do that one-doc search - it means the entire _id index is getting into RAM ? Or only part of it ?
[11:45:40] <remonvv> Yeah it's 0ms according to explain from SSD backed volumes, but it's slightly more
[11:47:34] <remonvv> Typically I see 1 or 2, and I think on sharded environment followed by a swap in of an entire chunk or something.
[11:47:49] <remonvv> Not exactly sure if MongoDB is prefetching something or if it actually needs 40-50 pages.
[11:48:51] <Bartzy> remonvv: Because how else can it know where in the btree it should look for - it will need an index for the index :p
[11:49:00] <NodeX> I was going to comment that all the performance you gain with micromanaging things you will lose in a framework
[11:49:27] <Derick> the index are btrees, so only the relevant parts of the btree have to be pulled into memory
[11:49:43] <remonvv> Bartzy, that's not quite how it works though is it ;) It knows where the index data is and is addressing that data in the virtualized memory. The OS will swap that page into physical memory if it isn't already (and is hot enough).
[11:49:45] <Bartzy> NodeX: I'm just learning that way about mongo. Of course I don't really care if a document takes 4ms to fetch or 0.1ms.
[11:50:44] <remonvv> Bartzy, it's more an OS MRU paging thing than a MongoDB thing. MongoDB uses memory mapped files. The data files (data and indexes) are addressed through that and hot pages are swapped in based on the OS memory management.
[11:51:46] <Bartzy> And what is in that metadata? where in the data files are parts of the indexes ?
[11:52:10] <Bartzy> If it's asking for a mapped file only then the OS would load the entire thing into memory
[11:53:25] <remonvv> OS figures out if that range of the memory mapped file is in memory and if it chooses to (usually based on an MRU scheme) it swaps those file pages to physical memory.
[11:53:41] <remonvv> MongoDB knows where to look for what because it maintains index metadata per collection.
[11:54:15] <remonvv> not really, it doesn't ensure it's loaded, not all OSs swap directly at the first read if physical memory is highly contended
[11:55:01] <Bartzy> And Mongo knows to ask for Address X to X+S because it has this metadata about all addresses ?
[11:55:40] <remonvv> it knows where the b-tree for that colleciton is located based on some metadata (I think .ns files right Derrick?) and it simply addresses that space.
[11:55:47] <NodeX> [12:49:19] <@Derick> and yes, there is an "indx" to teoo mongodb where everything is on disk
[11:56:09] <NodeX> I think translated that means "there is an index to tell mongod where everything is
[11:56:16] <remonvv> But b-tree offsets might be the exception. It would be a good reason why there's a 14k collection limit.
[11:57:06] <kali> as for the rationale behind it, there is a very good article there: https://www.varnish-cache.org/trac/wiki/ArchitectNotes
[11:58:03] <remonvv> Bartzy, also, if you're worried about performance start small but enable sharding from the start.
[11:58:25] <remonvv> Very easy to scale up as needed and sharding scales up memory along with all other hardware resources.
[11:58:48] <Bartzy> Yeah it was not our first thought and now we're already deep into "scale vertically as much as possible before sharding"
[11:58:51] <NodeX> [12:38:12] <NodeX> why dont you save a headache and get smaller servers and shard?
[12:01:10] <kali> i also like to have an easy "scale up" solution: i'm not using the biggest server available (on aws). if i find myself cornered, i can scale up easily and then start to find out what's wrong
[12:39:55] <remonvv> kali, we use that strategy too. The only concern is that 10 big nodes are much more reliable than 80 small nodes.
[12:41:34] <remonvv> cmex, ignore the haters! :) C# is pretty nice. Just not hugely popular around here.
[13:16:13] <PDani> i have a single-instance mongodb with a collection with 3 fields: _id, block_id, payload. payloads are always 4096byte binaries. _id is an ever-incremented unique integer. there is a secondary index on the collection, { "v" : 1, "key" : { "block_id" : 1, "_id" : -1 }, "ns" : "mongobd.testdev", "name" : "_block_id_id" }
[13:18:15] <PDani> i'm doing many queries like: query: { query: { block_id: 868413 }, orderby: { _id: -1 } } ntoreturn:1 nscanned:1 nreturned:1 reslen:4166 163ms, there's no other query during these. when i read sequentially by block_id, it's 10 times faster than when i query with random block_id
[13:19:53] <PDani> i have low cpu usage, low storage utilization. the collection is 2-3 times bigger than the memory size. I don't know what can be the bottleneck?
[13:30:57] <remonvv> PDane, all else being equal you're looking at the performance difference between reading from a virtualized page that is in physical memory and one that is not. Sequential data is likely to be in memory while randomly accessed data typically is not.
[13:31:30] <remonvv> run mongostat and see if faults/sec rises when you switch to random reads compared to sequential ones.
[14:31:19] <ranman> zakg: are you sure you're using the right python and that you're installing pymongo to the correct version of python?
[15:01:40] <Bilge> If I have a LOT of transaction records should I store precalculated totals in another collection?
[15:03:04] <Bilge> I'm assuming there would be a significant performance difference that would make maintaining stored totals more efficient even though they could be vulnerable to going out of sync
[15:04:55] <PDani> I have as much page faults as queries, and on pagefaults, mongodb seems to read more data than needed (aggregated size of read documents is 10 times smaller than actual data read from disk)
[15:07:01] <PDani> it means, that random reads will always have a big overhead on ~5000byte documents?
[15:11:20] <PDani> my index size is 300MB, and the instance have 3.7G memory, mongo seems to be using only 2G of it
[15:12:18] <linsys> Derick: index miss means the index wasn't in memory when it was asked for or referenced.
[15:14:51] <remonvv> and yes, index misses is similar to faults/sec in that it tells you how often a b-tree node is accessed that isn't in physical memory
[15:15:08] <PDani> is there any readahead in mongodb when i do a query like { $query: { block_id: 685233 }, $orderby: { _id: -1 } } ntoreturn:1 nscanned:1 nreturned:1 reslen:4166 134ms?
[15:16:16] <PDani> because it seems that mongodb reads from disk much more data than the size of documents i query for
[15:16:16] <remonvv> if there are multiple matching document it'll prepare an initial resultset that the returned cursor can iterate over. If it needs more it will issue getMore commands
[15:16:43] <linsys> PDani: How are you determining this? That mongodb reads from disk more data then the size of the document you query?
[15:17:26] <remonvv> limit(1) has nothing to do with it, that just tells mongo to only return 1 document. Is very speedy if you're not sorting but you are so it has to sort the resultset before it can determine what the first document is.
[15:17:28] <PDani> linsys, I calculate the size of read documents per sec on client side, and i see the iostat output
[15:18:07] <remonvv> PDani, you're looking at the wrong things. If you have performance issues insight into how MongoDB's mmap engine works isn't going to fix much for you. It's not something you can affect
[16:25:47] <cedrichurst> dumb question… if i wanted to compute the cross-product of keys in two collections, would i need to do that at the application layer
[16:26:02] <cedrichurst> for example if i had one collection containing salesperson and another containing fiscal quarters
[16:26:24] <cedrichurst> and i wanted to create a new collection with a compound key for every salesperson in every fiscal quarter
[16:58:07] <Bilge> If I have a LOT of transaction documents should I store precalculated totals in another collection or always calculate aggregate values every time?
[17:09:00] <nickswe> Annoying on Windows 7 when trying to run "mongod.exe" it says that I am missing folder /data/db/... but I am having "dbpath=C:\mongodb\data" in my mongod.cfg... why does it not understand this?
[17:45:19] <brynary> hello -- I'm experiencing extremely slow MongoDB reads under what I think is a relatively low write load (few hundred/s). in mongostat, locked % gets very high, faults is 0
[17:46:32] <brynary> I'm running fast hardware, plenty of RAM, 8 CPUs. 15k RPM SAS drives. I'm not sure what sort of perf I should expect. many simple inserts are taking 350ms+
[18:53:53] <Bilge> If I have a LOT of transaction documents should I store precalculated totals in another collection or always calculate aggregate values every time?
[19:17:54] <aster1sk> New to mongo, I'd love some feedback on whether this appears to be a reasonable aggregate object or if I'm totally doing it wrong.
[19:37:22] <federated_life> whats a good way to tell if a replica member was a step down, something about op counters ?
[19:40:13] <crudson> aster1sk: really depends on what problem you are trying to solve, what your input documents are etc.
[19:41:28] <aster1sk> crudson: Thanks. This model is what I want however I'm afraid of the ridiculous nesting.
[19:42:14] <aster1sk> For instance I couldn't (in one query) determine how many android device views from canada... I suppose the frontend could figure that out but I'm not sure the boss will buy it.
[19:43:21] <aster1sk> Also when upsert / incrementing the view counts will require two queries to determine if the indexed array exists.
[19:47:05] <crudson> aster1sk: That epends whether you want this updated in realtime. I wouldn't be afraid to have as many map reduce operations as suits your querying needs, which shoul be your primary concern.
[19:48:46] <crudson> If you have a specific aggregate question you can get advice here for sure (I have to run out now but there'll be plenty of experts around)
[19:49:04] <aster1sk> Hah, yeah close to realtime would be a plus. The aggregate documents are 'per day per issue'. Doesn't have to be up to the minute but I'm sure they'd be happier with near-realtime.
[21:33:37] <wereHamster> I literally have no idea how it works. But it works. Which was good enough for me :)
[21:47:31] <fdv> Hi guys. It seems to me that renameCollection is a privileged command, and needs to be run while using the admin db. Now, the command takes two parameters, the old name of the collection and the new name, but can anybody tell me how to specify the database?
[21:51:22] <fdv> when I try to add the db name like db.runCommand({renameCollection: "thedb.foo", to: "thedb.bar"}), I get an error stating "exception: source namespace does not exist"
[21:52:54] <crudson> fdv: additionally, it can be run from any db: db.fromCol.renameCollection('toCol')
[21:53:14] <fdv> crudson: but when I 'use thedb' and then try to rename "foo" to "bar", I get another error, { "errmsg" : "access denied; use admin db", "ok" : 0 }
[21:53:40] <fdv> and the docs say "You must run this command against the admin database. and thus requires you to specify the complete namespace (i.e., database name and collection name.)"
[21:56:52] <crudson> it's an "administrative helper", so even though it may get execute in that namespace, the translation is done for you. http://www.mongodb.org/display/DOCS/dbshell+Reference#dbshellReference-AdministrativeCommandHelpers
[22:12:01] <rossdm> question: I have a MongoDB replica set using IP over Infiniband. RCP'ing a file between mongo servers averages 200MB/s transfer speed. Sticking a file in GridFS and waiting for replication to two replicates yields 140MB/s out (70MB/s in for each server). Any ideas why replication would be slower?
[22:53:40] <Bilge> If I have a lot of transaction documents should I store precalculated totals in another collection or always calculate aggregate values every time?
[23:02:44] <dstorrs> if you can calculate on the fly (in the browser), do so. that reduces your storage and CPU needs
[23:03:26] <dstorrs> if you can get away with just caching the latest data in (e.g.) memcached and doing the totals from there, fall back to there.
[23:04:12] <dstorrs> but it's always best to distribute your processing onto the client if that doesn't compromise security and site performance. much cheaper than scaling your own hardware
[23:13:59] <Bilge> But it's conceivable that if you just maintain totals and records separately that they could somehow become out of sync, particularly if there are bugs in the code or transactions don't happen atomically