Log file Viewer
#mongodb logs for Thursday the 4th of December, 2014
[02:10:49] <appledash> CPU seems fine, network isn't in the equation because this is a local server, and as for I/O... I'm not sure
[02:11:24] <appledash> dstat says I'm writing about 4000k/sec. Would that max out a standard desktop HDD? (I think the l is actually capital K as in kilobytes, not kilobits.
[03:50:16] <Noxywoxy> Quick Q: Is anyone able to recommend a consultancy service? I was looking here and I'm not really sure how to guage quality. Hoping a personal recommendation would help.
[03:50:34] <Boomtime> type "db.getReplicationInfo" at a shell prompt, it will show you the javascript
[07:18:21] <n^izzo> this is what I have at the moment db.histories.aggregate([{ $match : { archived : false } }, { $group:{ _id: "$_zone", amt: { $sum: 1 } } }])
[10:11:32] <CipherChen> hi all, do your mongodb cluster size execeed 12 nodes, is there exists an good solution to solve this
[10:35:24] <CipherChen> when my cluster exceed large enough that i need more than 12 slaves for querying
[10:41:20] <CipherChen> we cannot now, for that we deploy multi data center and the data in each center are totally the same
[11:19:56] <CipherChen> I'm running 2.4, if I want to upgrade to 2.8, when it says "To upgrade an existing MongoDB deployment to 2.8, you must be running 2.6.", do it mean that the WHOLE rs has to being in 2.6 first or just the upgrading-node being running in 2.6?
[12:20:22] <markotitel> What needs to be done in order for mongod logs queries to a file? I have enabled "profile =2" and "slowms = 30" in config file.
[13:08:18] <joannac> markotitel: you should see it in the mongod log as well as in the profile collection
[13:13:38] <markotitel> joannac, it seems that still there is no slow log, DB is not used at all for now so possibily no slow queries. Should I see read queries also?
[15:51:51] <Lingo__> hi. I have a collection with documents such as { id: 1, version: 1 }, { id: 1, version:2 }, { id: 2, version: 1 }. is it possible to construct a query that gets me the document with greatest version for each id?
[15:55:20] <amk> Can someone give me a hint where to look for causes of multiple lingering connections (ex. 810 open connections on a low low low traffic app db)? Is a the event of a connection not being closed always caused by the client?
[15:58:28] <Lingo__> I basically want select * from collection where id = X have version = max(version)
[16:03:39] <amk> Lingo__, in other words 'find documents with distinct ID, limit to 1, sorty by VERSION' ?
[16:04:45] <Lingo__> amk: yes. 99% sure it's not possible with find(), thinking it's not possible with aggregate either, but not sure.
[16:09:57] <amk> Lingo__, how about this http://stackoverflow.com/questions/14697332/mongodb-query-for-all-documents-with-unique-field
[16:20:46] <Lingo___> amk: trying that approach gives me "exception: the group aggregate field 'id' must be defined as an expression inside an object" and I have an id property defined.
[16:22:13] <amk> Lingo___, this is a bit over my head now; but sounds like you're throwing it an _id where you should be throwing an {something}
[17:25:58] <OrNix> hi. tell me please how it is better to make replica set with 2 nodes? On which node should be started arbiter: on master or on slave?
[17:35:25] <GothAlice> OrNix: Your question is non-sensical. An arbiter is its own thing, separate from a primary or secondary or any other data-storing process.
[17:36:09] <GothAlice> OrNix: http://docs.mongodb.org/manual/tutorial/deploy-replica-set/ — Give this a good read, in addition to http://docs.mongodb.org/manual/core/replication-introduction/
[17:36:41] <GothAlice> OrNix: Yes, it certainly is possible. It's not high-availability unless you have three or more nodes, though, since if one of the two goes offline the other has no way of knowing if it's buddy went offline or it did.
[17:40:00] <OrNix> i understand the arbiter's role. So if i need 2nodes rs - is it better to disable failover and select PRIMARY by hands?
[17:40:37] <GothAlice> OrNix: The difficulty is that in that arrangement, if the SECONDARY goes down, the primary will demote itself to a *read-only* secondary.
[17:40:55] <GothAlice> I.e. even if the non-critical part of the replica set goes away, the critical part fails.
[17:42:50] <GothAlice> (Using priority levels, i.e. a priority zero secondary, you can ensure a certain host will remain primary at all costs, or will remain primary if no other option is available. You do get control over how the cluster arranges itself, but it's best to use these provided tools to do that, rather than any form of manual intervention.)
[17:44:32] <GothAlice> A la: http://docs.mongodb.org/manual/tutorial/configure-secondary-only-replica-set-member/
[17:45:33] <GothAlice> (Still throw an arbiter in there to ensure a failure in the secondary doesn't bring down the primary, though.)
[18:07:58] <flyingkiwi> we added a new shard to our rs. when are the moved chunks deleted (from harddrive and freed) on the old shard?
[19:24:33] <GothAlice> RaceCondition: db.example.find({$and: [{$or: [{A conditions…}, {B conditions…}]}, {$or: [{D conditions}, {E conditions}]}]}) — not wrapping in $and would mean attempting to have two values for the same key ($or) at the top level. Ref: http://docs.mongodb.org/manual/reference/operator/query/and/ http://docs.mongodb.org/manual/reference/operator/query/or/
[19:46:04] <GothAlice> It's an officially supported client language, and most of the server is written in it. ;)
[19:50:43] <GothAlice> aliasc: As a note, since MongoDB documents are inherently dynamically typed, use within a statically typed language comes at a cost in terms of the amount of boilerplate code you will need to write. (This is especially true in Java which rather amplifies the effect.)
[19:50:45] <GothAlice> Having some form of RESTful service written in a dynamic scripting language which your game speaks to (service-oriented-architecture) may speed development, reduce bugs, and increase agility. :)
[19:52:10] <GothAlice> aliasc: Optimization without measurement is by definition premature. What're your expected pain points (things that may hit performance walls and require independent scaling)?
[19:53:07] <GothAlice> aliasc: I've used MongoDB with several gaming clients, mostly Facebook games, though. https://gist.github.com/amcgregor/4207375 is my section of a presentation on the distributed RPC system I built for one of them. (Link to the full Python+MongoDB+Facebook gaming presentation in the comments.)
[19:54:32] <aliasc> but if mongodb is fast and clients link directly then there should be no problems.
[19:54:47] <harttho> We're seeing load spike on our mongo shards (all three replica sets) every 15 minutes
[19:56:07] <aliasc> while testing currently the server shows coordinates of another player in 2D space
[19:58:08] <aliasc> however if we plan to coop we need to store object data from the game to database. for example if we want users to save their coop progress together
[20:33:55] <tim_t> can anyone advise the best approach to implementing a mail system in mongo? right now i have a collection holding mail documents with a to: field for single users… a giant inbox as it were. i get the feeling this is not the best use of mongo but i cannot put a fine point on why
[20:35:44] <tim_t> well, when a user gets their email i simply query using the user as a filter. each document has a single user reference on it
[20:38:40] <shoerain> any thoughts on extensions like https://github.com/gabrielelana/mongodb-shell-extensions? it adds jsonpath, lodash, and moment to the mongorc.js
[20:38:56] <tim_t> no, its more like… say you have a letterbox at a house and a bunch of people are living there. each person has a letter addressed only to them, but it is still held in the one letterbox. my collection is the letterbox, my documents are the letters and just like when you bring the letters in you look at who it is addressed to and give it to them.
[20:39:40] <GothAlice> tim_t: Since e-mail data is typically read-only (excluding flags on the message; those are entirely different) you could attempt to replicate the top-level MIME encoding of the message (headers vs. body, each header as a field, etc.) and index the ones you care to search on. What you have is referred to as a multi-tenant collection.
[20:40:16] <GothAlice> Generally this isn't an issue, esp. if the data is indexed. With a substantial enough user base, the cardinality (uniqueness) of any given user's ID in the pool of all messages will allow for highly efficient querying. (The index will be used to great effect.)
[20:42:14] <tim_t> nothing that i can tell. i am just curious if others had advice on a potentially better solution for issues i am not yet seeing
[20:44:10] <GothAlice> Maildir is better in that it's simpler, relies on existing atomic filesystem operations, and can exploit existing filesystem compression and encryption techniques without needing to even be aware of them. It's harder to network and scale. Using a collection-per-user will run into namespace count limits in MongoDB, but isolates user's data.
[20:44:11] <tim_t> i guess having wide-open choices has me second-guessing the wisdom of the choices i am making?
[20:45:14] <GothAlice> Index on [user_id, mailbox_path, -recieved] — you can now answer "what messages are in mailbox X for user Y, ordered with newest on top" very efficiently.
[20:46:09] <GothAlice> (Where mailbox_path is something like "INBOX", "Sent", or "Projects.Marrow", etc.)
[20:46:17] <tim_t> okay thanks. isolation is not really an issue in my case so it looks like this is a good approach
[20:46:45] <GothAlice> (I'd recommend following IMAP conventions on mailbox naming for interoperability and consistency reasons.)
[21:09:40] <GothAlice> That could very well be your issue. Compare the timestamps triggering cleanup vs. the periods of excessive load; you may have to begin staggering your TTL runs. (I.e. shift the dates around a bit. TTL indexes are minute-accurate.)
[21:11:08] <hicker> Hi Everyone :-) I'm getting "exception: Cannot apply $addToSet modifier to non-array" when executing Practice.findByIdAndUpdate(req.params.PracticeId, { $addToSet: { users: req.params.UserId } } ... but users is an array, so I'm not sure why it's throwing an exception.
[21:36:10] <harttho> command collectionname.$cmd command: { dbstats: 1 } ntoreturn:1 keyUpdates:0 locks(micros) r:926075 reslen:255 926ms
[21:40:06] <GothAlice> harttho: The system.profile collection will contain additional information; it includes client info, authenticated user, and the raw query, but you may need to enable profiling first.
[21:41:24] <GothAlice> (Indexing blocks other administrative $cmds from operating against the collection being indexed.)
[21:42:05] <GothAlice> You'd need to go spelunking through your mongod logs to find out what it was, though.
[21:44:40] <harttho> Any way to see current admin operations only (so as to not have to sift through the rest)?
[21:45:19] <GothAlice> The profiler data is a collection like any other; formulate a query and run it. ;) (You can query the logged query to only the ones you care about.)
[21:47:10] <harttho> Also seeing command admin.$cmd command: { writebacklisten: ObjectId('5473af0118c6a1efae65b96b') } ntoreturn:1 keyUpdates:0 reslen:44 300000ms
[21:47:59] <GothAlice> What type of cluster are you running? And what type of write concern are you setting on your queries?
[21:50:05] <GothAlice> So, {[shard:(replica, replica, replica), shard:(replica, replica, replica)], (config, config, config)} or {[replica:(shard, shard), replica:(shard, shard), replica:(shard, shard)], (config, config, config)}
[21:54:43] <GothAlice> Are you using capped collections and tailing cursors? (a la the "awaitData" option?)
[21:56:02] <harttho> getmore local.oplog.rs query: { ts: { $gte: Timestamp 1415929198000|76 } } cursorid:1662888153033004078 ntoreturn:0 keyUpdates:0 numYields: 1 locks(micros) r:226 nreturned:2 reslen:924 2167ms
[21:56:19] <harttho> and command admin.$cmd command: { listDatabases: 1 } ntoreturn:1 keyUpdates:0 locks(micros) R:799 r:34188 reslen:28407 9040ms
[21:56:51] <GothAlice> You must be missing an index on ts. Nothing in normal operation should really query listDatabases… and that read lock is nuts. There is certainly something up with your current configuration.
[22:03:56] <GothAlice> harttho: Could you run "mongotop/mongostat" and see what your read/write latencies are, and if they're coordinated with that bulk load operation? If so, to prevent hosing your database each time, you may have to slow that import down. If not… I'm at a loss. I'd spin up a staging environment for more vigorous testing that won't impact production.
[22:05:19] <GothAlice> Enabling more than slow query profiling will have a deleterious effect on performance, FYI. mongotop/mongostat require no changes to the profiling configuration; you're looking for excessive lock percentages.
[22:11:47] <GothAlice> http://cl.ly/image/3n312V3Z1B3N — also sometimes amusing. Apparently my DB host has been up for so long that I've exceeded 48 bits of integer accuracy in counting how long locks have been held. XD
[22:16:28] <harttho> (there is some missing data, sorry for the that, don't want to waste your time obfuscating the data)
[22:22:39] <GothAlice> accessesNotInMemory: 27543272, pageFaultExceptionsThrown: 17166926, page_faults: 2370184717, average_ms: 802, asserts.warning: 85896
[22:23:17] <harttho> Any way to clear the stats to see what is happening now vs the previous 300ish days?
[22:23:26] <GothAlice> Warning level assertions should appear in the mongod log. Page faults indicate a clear need to continue to scale up (add more RAM) or horizontally (further subdivide into additional shards).
[22:23:50] <GothAlice> 'Cause it's causing insane lag to wait for the kernel to pull in pages off-disk, swapping out others in the process.
[22:30:01] <GothAlice> http://cl.ly/2c2g2I2m1T10 are the issues resolved (37 of them) between 2.4.9 and 2.4.12, many of which are _extremely_ important fixes. (Data corruption, reliability, filesystem, security, and performance.)
[22:35:34] <styles> This query works, but if you remove MINUTE and only count by hour, it does nothing
[22:43:53] <GothAlice> http://blog.mongolab.com/2012/06/introducing-dex-the-index-bot/ is one of them (note the follow-up articles)
[22:44:39] <GothAlice> (We keep a copy of "notable events" in the oplog and store them as an audit log and version tracking.)
[22:47:27] <GothAlice> (I added full-text indexing to my codebase before such a thing existed in MongoDB itself. Same with compression.)
[22:48:17] <GothAlice> (iSCSI from Drobo to paired server, that MongoDB cluster exposing GridFS to a local FUSE filesystem.)
[22:48:48] <GothAlice> Because my filesystem represents an infinite number of nodes stored in an infinite number of directories infinitely deep.
[22:49:40] <styles> I have like 5 processes I consistantly have to run on my local machine that's just a daunting task & my make shift 6TB raid here (local comp) is running out
[22:49:44] <GothAlice> I priced it out; most cloud storage providers would cost me ~$2K/terabyte/year. (compose.io would cost me half a million a month)
[22:51:08] <styles> It's like the story of a guy that had a whole rack at home and was using liek 24TB of bandwith a month with Verizon
[22:52:20] <GothAlice> Bell keep coming by to try to sell me "Bell Fibe", and I keep having to teach them that they're not selling me fiber, they're selling me ADSL last-mile to the concentrator on my apartment complex roof, *then* it's fiber. They don't offer unlimited unless you bundle, and I can only tell them I don't own a home phone or television…
[22:53:19] <GothAlice> … they try to sell me the bundle anyway, as the "solution" to my "unlimited" problem. Then I tell them I have offsite backup of 25 TiB of data, and they back away from the door.
[22:53:28] <styles> If there's an emergency chances are you're mobile towers will go out or be over loaded fast
[22:55:46] <styles> most of the time people are just freaking out contacting loved ones making sure they're ok
[22:56:36] <GothAlice> styles: That's included in my exocortex. (The name for that at home project.)
[22:57:31] <GothAlice> (userAdmin only applies to an admin user of one database, adminUserAnyDatabase would apply to an admin in the 'admin' database.)
[22:58:21] <GothAlice> styles: If I attempted to open it, I'd be in violation of a dozen patents that I'm aware of, and probably many that I'm not.
[22:59:05] <GothAlice> http://www.freshpatents.com/Framework-for-the-dynamic-generation-of-a-search-engine-sitemap-xml-file-dt20081023ptan20080263005.php for one.
[23:00:35] <GothAlice> l2trace99: I hope you mean the userAdmin/userAdminAnyDatabase thing, not the patent thing. ¬_¬
[23:01:40] <GothAlice> https://gist.github.com/amcgregor/c33da0d76350f7018875#file-cluster-sh-L76-L79 is the "add_user" chunk of my own staging provisioning script.
[23:03:40] <GothAlice> styles: AFIK, no? I test the first few stages ($match, $sort, $skip, $limit) as a normal query first, anyway.
[23:04:10] <GothAlice> (Those are the only bits that'll be effected by an index in an aggregate query, and only if they lead the pipeline.)
[23:04:13] <styles> I have an agg function in Go and it's not returning the same results as me doing it by hand
[23:04:46] <GothAlice> Alas you'd have to inspect the implementation of the driver to see if you can get the "raw" SON being sent over the wire.
[23:06:22] <GothAlice> PHP is not a general purpose scripting language, no matter how much it tries to be one…
[23:06:35] <harttho> what's an acceptable accessesNotInMemory and pageFaultExceptionsThrown per second?
[23:07:20] <GothAlice> My at home dataset is highly "at rest", and waaaaay larger than any amount of RAM I could throw at it, so expect some when you have terabytes of data.
[23:07:23] <styles> Kinda off topic, but since we know Google has been running document stores for ages now, I wonder what they have running that people don't know about :P
[23:08:45] <GothAlice> But when NDAs are involved, it's easier to simply avoid the whole subject. ¬_¬
[23:12:05] <GothAlice> l2trace99: Feel free to adapt my script; all it really needs is some "ssh" commands wrapped around everything to execute these commands remotely.
[23:12:55] <GothAlice> l2trace99: Or, even more highly recommended, provision using MMS. https://mms.mongodb.com/
[23:17:45] <GothAlice> Bash: one of the most highly underrated, and single most deployed scripting language on the planet.
[23:18:50] <l2trace99> GothAlice: yes but it make me itch. When things get heavy I chicken out and go perl
[23:19:24] <GothAlice> Because perl is a light-weight solution? O_o Perl is the the peg that fits any hole… by changing the meaning of the hole. XP
[23:21:43] <GothAlice> Hey, look at the functions in my example here. One even defines a default value for an argument.
[23:21:56] <GothAlice> Functions are the least of bash's worries. Text manipulation… sure. That's why the gods invented sed/awk/ed/etc. ;)
[23:23:46] <GothAlice> (My favourite example from my automation is this: git diff --name-only @{1}.. | xargs qfile -Cq | xargs --no-run-if-empty equery files | grep init.d/ | sort -u — that's the pipeline that generates a list of services to reload/restart prior to doing so after a git pull. Yes, I automate using git hooks and the… #truepowerofunix.)
[23:25:43] <GothAlice> https://gist.github.com/amcgregor/a816599dc9df860f75bd — have some sample code.
[23:26:11] <GothAlice> It's uses a mutable tree-based parser that allows you to redefine the syntax as you go.
[23:26:29] <bmillham> Hi all, new to MongoDB, playing trying to learn, and I've run into a problem with aggregate and unicode.
[23:27:14] <styles> I wanted to make a lang as pratice one of these days, don't have the time this second
[23:27:17] <bmillham> I have a document, Artists and am trying to get a count of artists by the first letter of their name
[23:28:35] <bmillham> > db.albums.aggregate({$group: {_id: {$substr: [ {$toUpper: "$name"}, 0, 1]}, count: {$sum: 1}}}, {$sort:{_id:1}})
[23:42:29] <harttho> :GothAlice is there a way to set profiling level of all db's? or is it just the one you're currently using?
[23:42:33] <bmillham> I do know that the data in the database is good, because I can display a complete document just fine, with the unicode
[23:44:59] <Boomtime> harttho: if you are happy to read log files instead of the profiling colection: http://docs.mongodb.org/manual/reference/parameters/#param.logLevel
[23:49:01] <GothAlice> (Hint: the second is one character "\u00F9", the first is two: "\u0060\u0075".)
[23:51:26] <GothAlice> In raw UTF-8 these turn into the byte sequences (in hex): C3 B9, or CB 8B 75.
[23:51:38] <bmillham> I can handle little things like that. I have web page that's currently written in php with mysql. I'm considering switching to Django/MongoDB.
[23:51:56] <GothAlice> bmillham: "Slice" of the first byte will give you an invalid character. Slice of the first two will either give you the ù or only the `.
[23:52:36] <GothAlice> bmillham: Avoid Django. Django is what happens when a cluster bomb goes off in code: disjoint parts and shrapnel everywhere. (I recommend any form of µframework like WebCore, or Flask. Full disclosure: WebCore is mine.)
[23:53:28] <GothAlice> bmillham: If that particular query matters, extract the first literal character, strip accents (ensure ASCII), and store that separately.
[23:54:09] <GothAlice> bmillham: https://github.com/marrow/WebCore/tree/master/examples and http://web-core.org
[23:54:26] <styles> GothAlice, dex doesn't seem to show agg funcccnsnsnsn is there anything else I can do?
[23:54:28] <bmillham> I always have problems with frameworks. It seems like I spend more time trying to get the framework to do what I want...
[23:54:40] <GothAlice> styles: I find OO abstractions of request/response handy, and mapping URLs to controller methods handy, and that's about all WebCore does by default. ;)
[23:55:38] <bmillham> The site I'm looking to convert is: http://bmillham.millham.net It's a request site for listeners for my internet radio show.
[23:55:44] <styles> I had a problem w/ uhh crap I think CakePHP at one point w/ unicode. there wer emovies in the db not in english and it'd break
[23:55:57] <GothAlice> styles: And no, dex doesn't show aggregate function output. But again, extract the query and run as a normal find().
[23:57:33] <GothAlice> bmillham: https://github.com/amcgregor/TSA-Timeline is an example "light-weight" project that still has some minimal structure to it.
[23:58:01] <bmillham> If you want to see what I'm trying to replicate, go to my site, authenticate so you can browse the site. Select Browse: By Artist and you will see a list of letters at the top of the page
[23:59:40] <GothAlice> Yup, totally manually extract the first literal character and store that separately.