Log file Viewer
#mongodb logs for Wednesday the 10th of December, 2014
[00:16:37] <joannac> modulus^: also, what do the lgos say interms of nscanned, nscanned objects, index use, time in read lock, etc?
[00:20:10] <bpap> will Mongo 2.8 allow you to choose storage engine on a per-collection basis? or is it one engine for the entire server?
[00:23:27] <modulus^> [conn212072] query VM.rec.files query: { query: { filename: "greet_522fb1d7e4b0d91b533cbef2.wav" }, orderby: { uploadDate: -1 } } ntoreturn:1 ntoskip:0 nscanned:1 keyUpdates:0 locks(micros) r:1420 nreturned:1 reslen:180 28571ms
[00:25:30] <modulus^> [conn210074] insert VM.rec.chunks ninserted:1 keyUpdates:0 locks(micros) w:28428610 28427ms
[00:42:15] <Boomtime> modulus^: it's hard to say, you have a 28 second single write - it doesn't matter how parallel the database is if it can't write to disk
[00:43:13] <Boomtime> you say you're only getting 40% lock, but that just means that we haven't observed the problem - a 28 second write that hits only a single document is ridiculous
[00:47:16] <Boomtime> modulus^: capture the output from mongostat for a minute or so, and pastbin/gist it
[00:54:53] <Boomtime> the number of faults is nearly equal to the number of ops, every operation page faults to disk
[00:56:01] <Boomtime> your write through-put is shockingly bad, it takes just 25 writes/s to hit 60% lock
[01:01:25] <Boomtime> i'm a little surprised it is as bad as we're seeing.. i suspect it all comes down to seek time
[01:01:53] <Boomtime> the trouble with seek time is that it doesn't improve no matter how many disks you add
[01:02:19] <Boomtime> (well, unless you add so many disks that their individual onboard caches collectively store everything)
[01:03:56] <Boomtime> it may also be partly the controller, a RAID controller can, at best, make the seek time for read/write unchanged - or, it can make it worse
[01:24:47] <Boomtime> outside of mongodb, use a disk testing tool to thrash the disk around a bit and observe which metric is letting you down
[01:26:21] <Boomtime> a disk IO testing tool - one that will read/write some random data, in varying sized blocks, in varying places, and give a performance report
[01:26:41] <modulus^> http://www.deploymentresearch.com/Research/tabid/62/EntryId/135/For-the-fun-of-it-Benchmarking-the-HP-Smart-Array-420i-controller.aspx
[01:29:55] <Boomtime> raid controllers are highly configurable (does yours even have the same bios revision as his?) and that is without considering the disks, the OS, other software or motherboard that backs it all
[01:30:14] <Boomtime> unless you have done performance tests yourself, on your own hardware, you have no idea what your performance actually is
[01:44:42] <HMill> Hello, all. Is there a way to get mongo to return a cursor from collection.aggregate() in v2.4.X?
[01:46:37] <joannac> upgrade to 2.6, or make sure your aggregation output doesn't exceed 16MB (and intermediate stages don't exceed 32MB)
[01:51:51] <joannac> Based on https://github.com/meteor/meteor/issues/2036, they're not really supporting 2.6 yet
[03:57:45] <kataracha> if querying on _id should there be any difference between the time it would take to query a very large collection compared to a small one or are they both constant lookup regardless?
[04:10:28] <GothAlice> kataracha: Indexes are generally stored as b-trees. This gives them predictable performance, ref: http://stackoverflow.com/questions/4694574/database-indexes-and-their-big-o-notation
[04:19:20] <bmillham> Hi all. I have some questions about the best way to keep a remote MongoDB in sync with a local copy.
[04:21:39] <Boomtime> be sure to make the priority zero for the remote secondary so it is never a valid option for primary
[04:21:43] <bmillham> So when I'm DJing, they make requests at the site, which I'd like to have updated locally
[04:21:51] <GothAlice> bmillham: At work we use a pair of replica secondaries, one in the office, one in my apartment. The one in my apartment is purely for backup purposes (it's actually delayed 24 hours to assist with data recovery in the event of deletion) and the one in the office gets queried.
[04:23:41] <GothAlice> bmillham: You can do interesting things like run the secondary's connections through an SSH tunnel. :)
[04:27:18] <bmillham> Can a replica-set be setup so that my local system periodically queries the remote for updates?
[04:28:00] <GothAlice> bmillham: Just make sure your "oplog" size is sufficient to cover the period of time between "syncs" (plus a fair bit of head room for safety), and your in-office secondary can be spun up when you need it.
[04:32:05] <bmillham> As you have have guessed, I live out in the country. But it's sad that I live 60 miles from Washington DC, and there is no high speed internet available here other than satellite.
[04:41:28] <bmillham> OK, looking at replica-sets, it looks like that won't work for what I'm hoping for. The remote replica can't accept updates. And I need that.
[06:27:33] <kenalex> i am new to mongodb and current testing it out. When is mongodb preffered over a RDBMS ?
[07:04:53] <sabrehagen> other than semantic meaning, is there value to storing an id field of a foreign object as ObjectID type rather than string?
[07:21:26] <logic> in mongoose, I could use new Schema.add({path: 'type'}) but I would like to do it from mongo shell
[07:21:44] <borjagvo> Hi. It seems I found a bug: http://stackoverflow.com/questions/27381041/text-search-not-working?noredirect=1#comment43219076_27381041. The $search doesn't work with the word "mesías" for example. Both words are not stop words.
[07:22:26] <borjagvo> Interestingly, I didn't see none of these two words in the spanish stop words list: https://github.com/mongodb/mongo/blob/master/src/mongo/db/fts/stop_words_spanish.txt
[07:25:22] <linocisco> logic, is it complete programming language? Node.js is one form of javascript. It is confusing
[07:26:36] <borjagvo> logic: I though I just did that: http://stackoverflow.com/questions/27381041/text-search-not-working?noredirect=1#comment43219076_27381041
[08:20:02] <Climax777> Hi all. Now that wiredtiger compression is to be available, how big effect does long field names have now?
[09:06:24] <Streemo> will $or:[{field:10},{field:10}] match anything with {field:10} am I allowed to fall back on redundancy like that?
[10:03:06] <kali> linocisco: mongodb atomicity is limited to one single document update. so typical banking transaction scenario get difficult to implement correctly
[10:15:21] <linocisco> kali, what is one single document update? is it only ok for one transaction at a time?
[12:41:07] <borjagvo> Hi. It seems I found a bug: http://stackoverflow.com/questions/27381041/text-search-not-working?noredirect=1#comment43219076_27381041. The $search doesn't work with the word "mesías" for example. Both words are not stop words.
[12:41:23] <borjagvo> Interestingly, I didn't see none of these two words in the spanish stop words list: https://github.com/mongodb/mongo/blob/master/src/mongo/db/fts/stop_words_spanish.txt
[12:57:35] <borjagvo> Any help please from any person involved in the project? I posted some more details on the comments: http://stackoverflow.com/questions/27381041/text-search-not-working?noredirect=1#comment43219076_27381041
[13:36:58] <rioch> I have a document like so: {'field1' : 'value', 'field2': [ {'key': 'a', 'name': 'a'}, {'key': 'b', 'name': 'b'}, {'key': 'c', 'name': 'c'}]}. How can I update it so that all key fields are set to null?
[15:10:43] <Constg> Good afternoon, I have a question: How would you update ($increase) a value in an object which is nested in a array?
[17:20:11] <harttho> Specifically, we have some slow queries run and the performance of un-related ones are affected too
[17:36:11] <rchickenman> I am running a very simple query that returns 20 or so simple documents with no filters, and it's taking 30 seconds in Node.JS, but only a few milliseconds when I do it from the Mongodb shell.
[17:41:39] <GothAlice> rchickenman: Is that MongoDB shell running on the same host as the server itself? Also, what are you doing with that data, and is it overly large? (I.e. is each document multi-megabyte?)
[17:43:15] <GothAlice> rchickenman: Could you gist the relevant query handling code from both the JS and shell?
[17:45:42] <rchickenman> Story.find().lean(true).select('url').exec(function(error, stories) { console.log(stories) })
[17:46:56] <rchickenman> I tried using the "lean" feature because my understanding is that it removes whatever overhead mongoose usually imposes.
[17:50:45] <GothAlice> rchickenman: Hmm. If it's working from the shell, but freaking out in Mongoose, sure sounds like a Mongoose issue. Have you asked in #mongoosejs? (They'd be more likely to be able to assist… I don't JS, so my help here is limited.)
[17:56:22] <rchickenman> Okay, before I go there I might take some time to try it with the vanilla node.js mongodb-native driver to isolate it as a node or a mongoose issue.
[18:09:45] <GothAlice> Welp, I now have a complete caching decorator library (both general memoize and Document-aware method decorator) for Python built on MongoDB ready for extraction from my work codebase. Mondays suck, but Tuesdays are very productive. :3
[19:58:47] <GothAlice> Does anyone know if MMS can be told to not stream-backup specific collections? I'd like to not have my cache (which can be rebuilt in its entirety or lazily) pointlessly transferred around. :/
[20:01:25] <cheeser> from one of the devs: on the backup page in the gear box of options on the right there should be a "Manage excluded namespaces". adding to that should take effect shortly
[20:02:52] <cheeser> it's still going to read changes to those collections from the oplog collection, but the agent will stop forwarding them onward
[20:36:10] <michaelq> Quick Mongoose.js question: User.count({}) is returning [Object, object]. How do I get it to instead return the results of the query?
[20:39:28] <ejb> Can anyone recommend some mongo based job queues? I'd like to queue jobs from meteor and process them on some other server(s)
[20:39:58] <Synt4x`> is there an easy way from mongo shell to output my .find (about 2,500 results) into a CSV or something that's easier to scroll through and look at?
[21:05:50] <GothAlice> ejb: Task queues in MongoDB are easy to roll for yourself: https://gist.github.com/amcgregor/4207375 is an extract from a presentation I gave on the process, with full Python implementation (supporting immediate and scheduled tasks) linked in the comments.
[21:14:35] <d-snp> hi, we have a high-write low-read environment, and I'm thinking of splitting our databases up to a database per customer, will this mess with the write throughput?
[21:15:17] <d-snp> yeah? what are the reasons for it improving? is it because of the smaller indexes?
[21:15:22] <GothAlice> d-snp: MongoDB currently has a collection-level write lock. (Future versions will have document-level write locks.) Splitting into separate DBs = spreading the locking around.
[21:15:53] <GothAlice> (No contention when simultaneously updating records for different clients in your case.)
[21:17:06] <GothAlice> d-snp: Smaller indexes will help update/find performance, too. Even lookups by ID should be faster, though depending on data size the difference might not be really measurable.
[21:17:13] <d-snp> I'm actually not sure if locking is an issue, it's not really important that the writes go through immediately, as long as the final throughput is optimal
[21:18:20] <d-snp> well, I was fearing maybe because the writes wouldn't be to the same file anymore, there would be less sequential writes which might reduce performance
[21:21:33] <GothAlice> Locking is an issue if you have high "waiting for write lock" times or write lock percentages. Read-intensive databases have low percentages here, write-intensive databases (where a write is likely to happen before a previous one finished) will have a high percentage. (Sometimes >50%.)
[21:22:21] <GothAlice> When you're spending so much time literally doing nothing and waiting, splitting across DBs can eliminate much of the waiting, and thus utilize disk IO more efficiently.
[21:22:59] <GothAlice> (SSDs are truly random access devices compared to spindle drives with a physical head that needs to move.)
[21:23:27] <d-snp> well, that certainly strengthens my case, I'll be able to defend my move to the CTO now I think :D thanks :P
[21:25:51] <d-snp> I hope rearchitecting our stuff to support a db per customer isn't going to be too hard
[21:27:47] <d-snp> do you know any companies who work like this? maybe I should build a middleware that makes it sort of transparent
[21:28:40] <GothAlice> (Data colocation laws in Canada require single-tenant databases for some of our clients.)
[21:29:35] <GothAlice> In our case, choice of database is derived from the requested host name over HTTP.
[21:31:09] <GothAlice> Aye; it's work-in-progress WSGI middleware sandwiched between the overall transactional middleware and the application proper.
[21:32:28] <GothAlice> (This middleware also swaps out master templates for CNAME whitelabling purposes.)
[21:34:55] <d-snp> hmm do you plan on having the databases all hosted by the same mongod/s instances, or instances per database?
[21:36:03] <GothAlice> Where the CNAME is set up as something like "jobs.example.com" or "careers.example.com".
[21:36:34] <d-snp> any specific reason for not doing a cluster per customer? or just sysadmin overhead?
[21:36:53] <GothAlice> Most clients don't have datasets worthy of that level of isolation and independent scaling.
[21:37:19] <GothAlice> (I.e. we have some clients with 10,000 jobs registered… and they're still on the shared infrastructure, since that comes to about 20MB of data. ;)
[21:38:05] <d-snp> our data is in the 10s of gb's per customer, and that's the small ones unfortunately
[21:40:14] <GothAlice> You mentioned a write-heavy load. Since your dataset for a single client won't fit in RAM anyway, the only benefit of splitting into a cluster-per-client is to have cache locality. (I.e. data cached in RAM for client A that client B's data can't push out if split, but can if shared.)
[22:03:37] <boxmein> hiya, stupid question but how the hell do I query for a list of possible parameter values efficiently?