Log file Viewer
#mongodb logs for Monday the 25th of August, 2014
[02:28:13] <culthero_> is it common that a standalone instance on 40m records with a fulltext index takes 20-30 minutes to query on a very beefy machine?
[02:35:25] <culthero> I am somewhat at a loss, as this worked very well originally, and was hoping not to have to deal with sharding of data per 5-6 days of twitter data :S
[02:48:06] <Boomtime> @culthero, can you run db.tweets.count() in a shell a few times, a couple seconds apart
[02:49:06] <Boomtime> they show up in replication but unless you have a secondary to check with then the effect of TTL maintenance is only seen in the resulting effects such as lock%
[02:50:45] <Boomtime> i think you have a fixed IO allocation, say, 1000 or so IOPS by the looks of it (perhaps 2000) and you are hitting it
[02:57:52] <Boomtime> also, what do you mean the memory stays under 2GB, that output shows resident at 9...
[02:59:48] <culthero> I just don't know what is going on, I feel like mongo let me go up to a 40-50gb database without too many problems, then all of the sudden everything stopped and everything I'm doing doesn't make any sense as to what is slowing everything down
[03:00:16] <culthero> thanks for your insight, I am just curious if I have to wait until the TTL expiration is completed
[03:01:55] <Boomtime> that a benchmark of your system, showing again those similar big peaks for cached IO, but not for actual
[03:02:33] <Boomtime> and your count() checks showed more than a few thousand documents being deleted per check
[03:07:40] <culthero> So what I guess I am asking, is that if I need to somehow store lets say.. a rolling log of 30 days of tweets, with tweets coming in between 30 and 80 tweets per second, expiring the same amount, how do I factor what my hardware costs will be (based on one fulltext index on the collection)
[03:08:21] <culthero> or decide if it's even feasible on a standalone instance? like I didn't imagine I was going to be I/O bound
[03:08:24] <Boomtime> an empirical test like what you are doing is the best way, but right now you are in steady state
[03:09:24] <Boomtime> i realize that mongostate is effectively lying to you about that, the delete column does not show deletes
[03:10:32] <Boomtime> the slowness is in the document delete itself, not the fact that the TTL is doing it (though that is slightly more expensive)
[03:11:39] <Boomtime> right, so it is merrily deleting half the documents in your database based on a cursor.find() result
[03:12:20] <Boomtime> it will take awhile, you can probably work it out based on how many documents you had, or by running a find().count() on what should be kept
[03:12:47] <Boomtime> at this point, it will almost certainly be better to just wait for steady-state to be reached
[03:14:54] <culthero> I don't understand why, but I am curious how long I should have to wait for the stable state
[03:16:39] <culthero> insert rate (according to the other table recording it) varies from 75,000 an hour to 300k an hour
[03:18:13] <culthero> and I will also do a second query on a words table against the words in the tweet.
[03:19:30] <Boomtime> anyway, whatever you insert in 7 days is your target .count()... running .count() in the shell shows you how quickly you are approaching your target
[03:21:08] <Boomtime> if those numbers are right, and you currently have ~37 million documents, your TTL has a long way to go to catch up after the change
[03:21:57] <Boomtime> back-of-the-envelope calcs suggest you are looking at around 5 hours before steady-state is achieved :(
[03:24:25] <Boomtime> it depends a lot on the deletion-rate too, which i can't really tell, but you could measure
[03:30:25] <culthero> in general, if I sharded this over 4 20 dollar servers vs one 160 dollar one, am I going to potentially see a longer time of data that I could probably save>
[03:37:40] <Boomtime> @culthero, unless the sum of the hardware in those 4 servers is greater than the single server you had, then no, it will not improve anything
[03:40:56] <culthero> aren't you basically saying I'm i/o bound based on the number of inserts / deletes I would be doing
[04:14:19] <huleo> we have several types of content that share most of their fields, but not all of them - several are unique
[07:18:43] <Zelest> if I have a replicaset of 3 nodes.. and shut down a secondary member.. will that affect the other nodes in any way? or will operations continue as before?
[07:19:30] <joannac> Zelest: shouls be fine, you'll see more logging at the other 2 nodes report they can no longer see the one you shut down
[07:23:00] <Zelest> mostly it's default.. where it's consistency dependent, it reads from the primary.. on others where performance is preferred, secondaries.
[07:23:36] <joannac> since the secondary reads previously went evenly to 2 secondaries, but are now going to the only secondary left
[07:26:59] <Zelest> joannac, also, if I run a mongodb node on a server with 1 core vs a server with 2 cores.. will mongodb benefit from an extra core?
[09:30:24] <mccajm> Are there going to be any issues with running a replica set between two datacentres 80ms apart? Ideally the secondary datacentre would be a priority 1 slave.
[11:18:39] <EXetoC> I might've stumbled upon something like that before, but it's not exactly documented, is it?
[11:37:20] <EXetoC> kali: thanks. hopefully it'll simplify wrapper development, but I might be wrong
[11:44:17] <EXetoC> actually, the body of mongoc_collection_aggregate for example is fairly large, but others are simple
[12:00:42] <dragoonis> What are the differences between doing map/reduce in mongodb or doing it in hadoop ? The mongodb says you can send mongodb data to hadoop for map/reduce if you like.
[12:01:52] <kali> dragoonis: mongodb is desinged mostly for small, fast queries. hadoop map/reduce is oriented towards hour-long batch processing
[12:02:33] <dragoonis> I'm trying to pipe lots of data into mongodb and then use $.aggregate() to pull out data based on things like "date range" and "area_id"
[12:03:38] <kali> dragoonis: aggregation is faster than m/r so anything that can be done that way should be that way
[12:04:12] <dragoonis> kali, what do you think about this? https://gist.github.com/dragoonis/3a6a4b2b9bcc1543698b
[12:05:35] <kali> dragoonis: well, it all depends on what are your performance and scalability expectations on this use case
[12:06:19] <kali> dragoonis: heavy usage of aggregation (or worse, m/r) on important chunk of data is expensive
[12:08:41] <kali> dragoonis: so when you'll hit this wall, you may have no choice but redesign and refactor so everything that you need fast (because there is a user waiting for a page to load) must be pre-computed
[12:10:36] <dragoonis> kali, we pre-compile everything on the "date range". the problem we want to solve is we want them to choose an arbitrary date range on a date picker and miss the opportunity to pre-process.
[13:43:20] <flok420> I got this back from mongo: { result: [ { _id: null, total: 90 } ], ok: 1.0 } how do I get the value for "total" using the c++ api from this mongo::BSONObj instance?
[14:35:55] <flok420> I got this back from mongo: { result: [ { _id: null, total: 90 } ], ok: 1.0 } how do I get the value for "total" using the c++ api from this mongo::BSONObj instance?
[15:25:47] <feathersanddown> Hi, if I add an object to an array, is this an update or save operation?
[15:27:51] <ejb> Can anyone recommend a way to store "open hours" for a business? I need to be able to query currently open businesses.
[18:55:31] <culthero> Howdy, will sharding a standalone to replicasets allow me to scale an application that has high locking percentages when going over a specific size, given a uniform distribution of data across shards?
[20:59:09] <bob_11> hey guys, quick question: I am trying bulk updates of existing documents in a collection using UnorderedBulkOperation, it was my understanding this was the fast way to do bulk updates and while the wrapper function returned quickly, the database is still handing the bulk update (~40k updates) a half hour later
[20:59:11] <bob_11> my question is: is it normal for UnorderedBulkOperation to continue in the database for a "long" time?
[22:57:38] <melvinrram> I'm using mongoid. If a collection has a field called 'emails' of type Array, how would do a search for all documents that contain value 'blah@blah.com' in the email field? I'm trying to create a custom validator that ensures uniqueness of values that are going into the 'emails' field.
[23:23:58] <feathersanddown> How to write a 256kb fixed size binary file in mongodb ??? as a normal class attribute ???
[23:26:00] <feathersanddown> file: { name: "my_file.hs", data: <binary_data>, owner: "an user" } <--- something like this is possible ? or there is something else to take in consideration ?
[23:28:45] <feathersanddown> because bson save whole "file" object as a binary i think, is correct ??