Log file Viewer
#mongodb logs for Tuesday the 21st of October, 2014
[00:20:34] <morenoh149> http://docs.mongodb.org/ecosystem/tools/administration-interfaces/ which do you use?
[00:47:29] <TylerWalts> I'm looking to scale out from a single datacenter with 3-node replica set and add 2 more data centers each with its own replica for better read performance in those regions. Is there any way to to also perform writes to a secondary member of a replica set and have them eventually sync with the rest of the nodes?
[00:51:19] <TylerWalts> Shorter question: Is it possible to configure mongo to allow writes to a secondary member of a replica set?
[00:51:59] <Boomtime> TylerWalts: you can never write to a secondary, you want to use sharding to achieve what you describe
[05:29:02] <jrbaldwin> if i run: db.users.ensureIndex({'email':1},{unique:true, dropDups:true}) ——> how do i turn off dropDups for future data written to users collection
[05:31:00] <Boomtime> it's a unique index, once you have a unique index you can't get duplicates in future
[05:33:14] <jrbaldwin> if mongoose rebuilds the index everytime the node server is restarted then yes
[05:35:01] <Boomtime> once the index is built, unique:true means it will reject any attempt to insert duplicates
[05:35:27] <Boomtime> while the index is building, dropDups:true means it will just straight up delete any documents that conflict
[10:15:51] <disaster_> I have followed this help : http://docs.mongodb.org/manual/tutorial/add-user-administrator/
[10:16:28] <disaster_> But after I enable auth and I have restart mongodb, I can auth with my account but I haven't admin right in mogodb
[10:29:20] <sachinaddy> hi.. i m adding user from frontend in localhost mongodb server. but when i m connecting mongodb from robomongo, i m not able to see that user.
[10:34:38] <Climax777> How does MMS generate alerts on the metrics it gathers? I can see how it may work for metrics as measured when the pings come in. But what about the avg/sec metrics?
[10:42:17] <joannac> Climax777: well, it knows when the last ping it got was. why can't it generate avg/sec metrics?
[10:45:10] <Climax777> I'm not talking about the graphs, I'm talking about configurable alerts. I can configure an alert for when page faults are xxx avg/sec
[10:46:43] <Climax777> It means, alerts for these metrics can't be generated when receiving a new ping (event driven). Some how, either by querying the last 5 minute's data, or by polling periodically, the alerts must be generated/updated
[10:50:00] <Climax777> Also if no new pings come in (the device is dead or something), averages will decay, and should affect the alerts
[10:55:03] <Climax777> relevant stackoverflow: http://stackoverflow.com/questions/26484963/mongodb-mms-like-alert-generation
[10:57:23] <disaster_> { "_id" : "admin.admin", "user" : "admin", "db" : "admin", "credentials" : { "MONGODB-CR" : "1c6421f54c02df81746601d0f7c108f" }, "roles" : [ { "role" : "userAdminAnyDatabase", "db" : "admin" } ] }
[10:58:55] <joannac> Climax777: i don't know what to tell you. if you want to be able to alert of metrics that must be calculated, you have to actually calculate them
[11:03:16] <Climax777> joannac: I want to try getting everything event driven, this however seems impossible to do with only event driven design
[11:05:57] <joannac> disaster_: cool. that's it. if you want other privileges, you need to give them to that user, or (better) create another user
[11:06:57] <joannac> Climax777: i don't understand why it can't be event driven. you get a new ping (event), you do your calculations, and then check alerts
[11:08:31] <Climax777> What is the alternative? poll for every account, every alert for every device metric
[11:09:20] <Climax777> The server is least of my problems, the units (being mobile and all) will definitely miss some pings
[11:10:33] <Climax777> true, but what would you suggest. one background task, iterating trough all alerts, or one per MMS group?
[11:13:07] <Climax777> one could make the assumption that if no pings come in, no alerts will further be generated. I'm just thinking that a down unit alert always trumps other metrics
[11:16:58] <sebastianlulu> if i have schema A and he can contain either B or C - but has to contain on of the two possibilities , how can i do that?
[11:18:06] <Climax777> and on app side you can determine which kind it is, and also in other queries
[11:19:24] <Climax777> to elaborate, you can say A contains a type B/C or you can say this sub document is type B/C
[11:20:27] <Climax777> It just doesn't work that great at this stage for embedding, but it gives you an idea
[12:27:44] <mlowicki> Hi, Is this the right place to ask about MMS? I'm adding new deploy after couple of months. In deployment view I get "Deployment is empty" but monitoring agent is working fine (syncing at least once per min). Any idea?
[14:18:41] <mlowicki> I want to upgrade MongoDB from 2.6.3 to 2.6.5 on debian wheezy. Launched sudo apt-get update and then sudo apt-get install -y mongodb-org but mongod --version still returns 2.6.3
[14:19:10] <mlowicki> dpkg --list | grep mongo shows that mongodb-org is in 2.6.5 (metapackage) but version of mongodb-org-server is 2.6.3
[15:35:01] <mike_edmr> Q: just upgraded 2.4->2.6. some of my indexes appear to be gone. Re-added one, watched it build, then it disappeared.
[15:35:32] <mike_edmr> what the hell could be causing it to quietly disappear when it gets done building??
[15:36:37] <GothAlice> mike_edmr: Possibly that it detected the index could not help a query. I.e. only one out of every 100 records has a value to be indexed in a non-sparse index. (Such an index would be a hinderance to most queries, not a help.)
[15:36:56] <GothAlice> However I wasn't aware that MongoDB does that type of rather glaring optimization. It may not.
[15:36:59] <mike_edmr> this one is absolutely essential and prevents people logging in when it's not there
[15:39:15] <GothAlice> mike_edmr: If you are reproducing the problem in development, try enabling the audit log in mongod, forcing an index build, and see what happens?
[15:42:48] <GothAlice> Alas, the audit log is probably your best hope for identifying the issue. I've never heard of or encountered your unique and beautiful problem. :/
[17:08:30] <doug1> According to http://docs.mongodb.org/manual/tutorial/deploy-shard-cluster/, the config servers should be installed first, followed by the mongos instance. However, if I do that, how can I add the admin user when there's nowhere to store anything yet?
[17:09:58] <GothAlice> doug1: If you have authentication enabled in your cluster, your sharding/replication behaviour is secured using a pre-shared key, not in-database authentication.
[17:11:28] <GothAlice> doug1: When bootstrapping a new cluster I've spun them up w/o authentication enabled, created the initial replication configuration, got replication working, then enabled authentication, added my admin user to the primary in isolation, then configured the PSKs. After restarting the cluster the authentication data would be propagated.
[17:12:50] <GothAlice> You could automate it. Spin up your initial master, configure its sharding and authentication (w/o authentication enabled at this point), *then* enable authentication and spin up the slaves.
[17:14:40] <GothAlice> (In fact, not a single step there can't be automated. We do this in-house, but alas, I can't share our scripten.)
[17:19:57] <GothAlice> Much of our MongoDB-based automation is template-generated JavaScript passed to the mongo interactive shell. Also note: rebalancing shards is cheap and easy if the dataset is empty. As with most IT problems, breaking it down into discrete steps can help clear things up. ;)
[17:21:16] <GothAlice> (Well, prior to enabling *final* secure replication, and prior to any sharding.)
[17:24:00] <GothAlice> doug1: It can be, but don't lose hope. I play around with my sharding/replication setups on a single testing VM (my finagling port numbers). Damn, my home internet connection must be down or I'd check my Exocortex code; it automatically constructs local three-node replication clusters for testing. T_T
[17:27:18] <GothAlice> Waaaitaminute. Are you trying to replicate shards, or shard a replication set? (I.e. client -> replication -> sharding or client -> sharding -> replication)
[17:27:56] <GothAlice> Sharding replication sets seems weird to me. (You could have three replication boxen for shard 1, but six for shard 2 for no reason.)
[17:29:27] <doug1> Because I work for a startup that expects me to automate the shit out of it and understand it without training and the docs are sketchy and we had the mongo folks come up and they weren't interested at all in automating mongo
[17:30:37] <GothAlice> Step one before "flipping all the switches" (sharding, replication, authentication) is getting a simpler setup automated. Without authentication is infinitely easier. Most deployments pretty explicitly do not need authentication.
[17:32:29] <GothAlice> … considering you'll need to include the password in your code, any explot that gains access to disk or memory will leak the DB password, making the password pointless from a security standpoint—that is, unless you fail to firewall MongoDB. If you don't firewall it (i.e. only allow connections from host X) you're doing the internet wrong and your ops guy will have a litter of kittens.
[17:33:16] <doug1> GothAlice: the app code will only have an app user. there are ways to obfuscate the admin password
[17:33:59] <GothAlice> doug1: Not if it's ever used. ;) The deobfuscation code has to be bundled with it in order for you to ever use the value.
[17:35:46] <GothAlice> … I worked for a startup for a while. 8 million SLoC later we had some pretty epic EC2 automation for Postgres—requiring no physical storage at all! Of course, we also failed and folded—the project wasn't to automate Postgres. ;P
[17:37:10] <doug1> GothAlice: That might work here if we didn't have a 1970's style division between ops and dev. Dev doesn't touch prod, which means dev has to give something shrink wrapped and boxed to QA and ops to deploy.
[17:37:23] <GothAlice> The only legitimate use I've had to date for authentication in MongoDB (since actual host-to-host security is handled properly via the firewall and IPSec) is to isolate multiple simultaneous users of a single cluster. I.e. my "cheap cheap cheap hosting" service uses a shared cluster set up this way. (It's also not sharded, just replicated, as these users' quotas won't ever require it.)
[17:38:43] <GothAlice> Vagrant FTW. Here, have a complete stable VM images representing "an application server" (automatically configuring the front-end proxy on startup) and another representing "a database cluster member" (automatically identifying the existing cluster and joining it, or recovering from latest snapshot + oplog replay.)
[17:39:55] <GothAlice> Packaging up a new VM snapshot is a 'git flow release' hook, or running 'tox' locally to build a temporary VM for local testing. Arguments (i.e. the DNS name to load the intiial configuration from) supplied to the Linux kernel and processed by init scripts. :)
[17:42:31] <GothAlice> However, authentication seems to be the key sticking point you're running into with your sharding setup. It's also something you probably don't actually need, given EC2's excellent control over firewalls. (Love the tag-based access control.) Hell, if you're running this on EC2 or any other "cloud" provider, you don't even have physical control over your data. Authentication (beyond external access control) is doubly pointless. :D
[17:44:11] <GothAlice> Not going the "golden master" route increases the likelihood that production and staging environments will differ in subtle, but issue-masking ways.
[17:45:00] <GothAlice> For continuous integration it's also helpful to have an *exact* snapshot of the failed build that is reproducable on any machine with VTx support. ;)
[17:49:46] <GothAlice> https://gist.github.com/amcgregor/4947201 — waaay old, but should demonstrate some of the ideas.
[17:54:10] <GothAlice> It's to demonstrate that a lot of what one might have as application-level deployment automation can in fact be VM startup automation, using VM snapshots as your "boxed up" solution.
[18:24:22] <mike_edmr> are there any major changes going from 2.4->2.6 which would have destroyed my write performance and cause excessive global write locks?
[19:17:10] <mike_edmr> are there any major changes going from 2.4->2.6 which would have destroyed my write performance and cause excessive global write locks?
[19:18:05] <mike_edmr> I have got 200 or so queries on average now waiting for a write lock on the database
[19:18:06] <GothAlice> Welp. That's one idea thrown out. Have you attempted a full database repair, and if so, I assume the symptoms persisted.
[19:18:57] <GothAlice> Ouch; this is production data? A repair would require being offline for as long as it takes to copy and re-pack the on-disk stripes.
[19:22:27] <GothAlice> O rly? Any diff between their configurations? (This includes sysctl settings at the kernel level.) You could try enabling slow query logging "live" (no downtime needed) and watching the logs to see exactly what MongoDB thinks its doing during those writes. There is also an audit log you can enable (requires a quick service restart to enable AFIK) which may provide more details.
[19:25:10] <GothAlice> Slow writes on my own cluster are generally attributable to the on-disk stripes becoming sparse, but with no holes large enough for the data to insert. (I.e. needs to investigate and reject potentially thousands of locations for the data.) Re-packing (via --repair or compact). Both repairing and compacting block use of the database.
[19:26:04] <GothAlice> Note that compact will not free space, unlike repair, and will require up to 2GB of additional disk space during operation. Repair requires free space equal to your current dataset size + 2GB.
[19:36:46] <GothAlice> mike_edmr: Are your writes using non-default write concerns? Your writes could all be waiting for on-disk sync()s that are slow to complete.
[19:38:39] <GothAlice> I'd be curious to see if 'diff' finds a difference between production and staging mongod.conf or the output of `sysctl -a` from each machine.
[19:38:55] <GothAlice> Open file handle limits are something I ran into a problem with a long time ago.
[19:39:57] <GothAlice> mike_edmr: Finally, how does your production and staging environment's on-disk storage differ, if at all? I.e. production runs on EC2 EBS volumes, staging runs on the EC2 ephemeral volume?
[20:06:47] <GothAlice> morenoh149: Yeah, _id is the only MongoDB-enforced key on documents, AFIK. (Excluding GridFS which is its own bucket of bolts.)
[20:08:02] <mike_edmr> perhaps after more testing on the staging cluster i'll be able to share something
[20:18:57] <speaker1234> if I have a list of objects, will mongo split them into separate records or do I need to do that?
[20:19:22] <GothAlice> speaker1234: MongoDB allows you to store lists within your documents, so it depends entirely on what you want it to do.
[20:19:50] <GothAlice> speaker1234: If you do want separate records, than either an iterative series of separate inserts or, if your driver offers it, use bulk_insert.
[20:20:33] <speaker1234> what I have is a bunch of individual records, all the same format, stored in a list so I could transport them as a single element
[20:21:38] <speaker1234> one thought was I should convert them from json to python native before injecting them into the database as a simple sanity check that I was getting good data
[20:31:49] <GothAlice> speaker1234: If you're using Python, I can highly recommend MongoEngine, which includes a bulk_insert utility. And yeah, it's a good idea to json.loads() before dumping into the DB.
[20:33:33] <speaker1234> okay, I need you to go slowly and speak small words because I'm new to mongodb. I installed the stock package for mongo from the mongo repository
[20:35:24] <speaker1234> is mongoengin that much better? all I'm doing is installing records, and retrieving them according to certain parameters as if it were a priority queue
[20:35:57] <GothAlice> MongoEngine is a layer on top of pymongo that gives you some goodies. https://github.com/marrow/marrow.task/blob/develop/marrow/task/model/query.py#L9-L58 is one of the goodies I added to MongoEngine. (Still need to submit that patch upstream, though.)
[20:36:00] <speaker1234> later I'll create reports on the state of the data for different customers.
[20:37:49] <speaker1234> the database entries go through several states such as queued, in process, delivered, failed, retry.
[20:39:05] <GothAlice> https://github.com/marrow/marrow.task/blob/develop/marrow/task/model/task.py#L162-L171 shows how to use that TaskQuerySet with a MongoEngine document schema.
[20:40:24] <GothAlice> Almost all of my states are programatically calculated, rather than storing a single string value to represent state: https://github.com/marrow/marrow.task/blob/develop/marrow/task/model/task.py#L221-L240
[20:40:25] <speaker1234> this is going to take a bit of study. It also tells me that I need to move the two entries, once they are finished on to a different database so that the queue won't grow overly large
[20:42:15] <GothAlice> Indeed. marrow.task handles this in two ways: collections that act like queues have an inherantly limited size, and are called capped collections for this reason. They're implemented as a ring buffer. (I.e. it gets to the end, returns to the beginning and keeps writing.) The second is that the Task records themselves get set to expire:
[20:42:16] <GothAlice> https://github.com/marrow/marrow.task/blob/develop/marrow/task/model/task.py#L188
[20:42:58] <speaker1234> I think that's where our models diverge. Tasks never expire. And I need to handle something like 2 million tasks a day
[20:43:31] <GothAlice> That's the real trick, my Tasks only expire after they are complete, and only 30 days after that.
[20:43:36] <speaker1234> because the systems of the far end sometimes get busy, tasks can queue up for a number of days but they still need to be delivered
[20:43:40] <GothAlice> https://github.com/bravecollective/core/blob/develop/brave/core/account/model.py#L198 < MongoDB can handle expiry of completed tasks for you.
[20:44:26] <GothAlice> speaker1234: Not an issue. The "remote" in your case would search the Tasks collection for ones that haven't been completed, ignoring the queue, and once caught up would then listen to the queue again.
[20:45:36] <speaker1234> I get my task lists from the database upstream (customer does geolocation stuff for handing out tasks to workers)
[20:45:53] <GothAlice> I.e. for task in Task.objects(complete=None): pass # do something \n for notice in TaskMessage.objects(task__gt=last_task.id): pass # process tasks added during catch-up
[20:46:31] <GothAlice> The length of the queue would be determined by how far behind you need to let the queue get before giving up and doing ^ as a more aggressive catch-up.
[20:46:44] <speaker1234> but when I pull things off the queue, but either pull them off sequentially (low-cost service) or, pull them off by customer name sequentially for the higher priority, i.e. they spent more $$ tasks
[20:47:34] <speaker1234> the idea is out have one fetcher that would always take the new, untried, tasks and the second for the stuff that filled the first time around
[20:47:50] <GothAlice> marrow.task internally handles priority levels by having the queue runner picking up tasks rapidly hand tasks off to a local distributor (i.e. concurrent.futures) that has been made priority-aware.
[20:48:13] <ejb> Hi, I need some design guidance for creating a weighted tag system. Essentially, every doc has an array of tags and I need to query and sort the docs based on how many tags are matched. The weighted aspect is that the tags in every query will vary in weight. So the tag weight for one query may be { foo: 10, bar: 5 } while the next one is { baz: 10, foo: 2 }. Does this make sense?
[20:48:46] <speaker1234> GothAlice, okay, I'm by doing. So I'm going to download the engine and your patch (where do I put it?)
[20:49:05] <GothAlice> m.task handles task retrying, that above "catch-up" process, and even RPC calling (and iteratively retrieving) remote generators, i.e. to indicate progress upstream.
[20:49:10] <ejb> In the former, docs tagged with foo should rank higher than those with bar. And in the latter, docs tagged with baz should rank higher than those with foo.
[20:49:29] <GothAlice> speaker1234: m.task is currently in the process of being extracted and sanatized from my work, propriatary codebase. It's incomplete.
[20:49:31] <speaker1234> can you give me a sample stone simple inject/pull off example I can work with an experiment from?
[20:50:33] <GothAlice> ejb: Store your tags as a list of subdocuments, i.e. [{t: 'foo', w: 1.0}, {t: 'bar', w: 2.0}]
[20:50:39] <speaker1234> mike_edmr, as much fun as it may be to tease us with little knowledge, it isn't a very kind. :-)
[20:50:57] <speaker1234> mike_edmr, just think, we may know something you don't and we may return the favor. :-)
[20:52:36] <GothAlice> ejb: Then your problem simplifies down to careful aggregate query construction.
[20:54:33] <GothAlice> Cool, so not *too* intense. I assume you only care about documents that match at least one of the tags referenced by the weights? (Or do you require all?)
[21:06:56] <ejb> These documents also have a geometry attribute (GeoJSON Point ... lat/lon). I want to weigh distance ($geoNear) into the sort order.
[21:12:21] <GothAlice> This aggregate query just keeps getting more funky. Now I've even got a $let in there.
[21:20:26] <ejb> GothAlice: Sorry, but I actually need to get going. Can you give me your GH username so I can find the gist when you're done?
[21:20:47] <GothAlice> I'm having difficulty internally debating between several different approaches.
[21:21:06] <ejb> GothAlice: Much appreciated. I'm sure I'll have some questions... so I'll be back to discuss. Thanks
[21:39:22] <speaker1234> is there a simple commandline tool that can dump the content of a database so I can see what's there?
[21:43:57] <speaker1234> I'm just trying get familiar with the basics and I'll probably move to your module after I make a very simple queue work
[21:48:10] <diegows> where is the log output documented? I can't find it... I don't remember the meaning of some information about slow queries
[21:51:57] <Boomtime> speaker1234: what do you want to know? the update command is atomic, for example
[21:52:38] <speaker1234> I'm just learning so I'm not sure what I don't know. using pymongo. I want to find a record, change a field, and write it back to the database as an atomic operation
[21:53:09] <GothAlice> diegows: /var/log/mongo… usually. Or syslog, or nowhere, all depends on your configuration and mongod command line arguments.
[21:53:34] <GothAlice> speaker1234: One of the most powerful patterns in MongoDB is "update-if-not-different", allowing you to implement things like locks on documents atomically.
[21:54:14] <GothAlice> speaker1234: Task.objects(owner=None).update(set__owner=me) — set me as the owner only if no other owner is set; gaurenteed not to conflict with other simultaneous attempts to do this. (One will win, the others will lose.)
[21:54:58] <speaker1234> looking at http://api.mongodb.org/python/current/api/pymongo/collection.html update method
[21:56:01] <Boomtime> if there are 2 such documents present, then running the update twice will update different documents
[21:56:11] <GothAlice> diegows: The mongod configuration documentation goes to some length to describe the available logging options. http://docs.mongodb.org/manual/reference/configuration-options/#systemLog and http://docs.mongodb.org/manual/reference/configuration-options/#auditLog
[21:56:22] <speaker1234> okay, I'm still missing some basic information. For example the definition of a spec or document
[21:59:21] <kellyp> heyo, is it possible to get an the _id field as a string inside of a aggregation $project?
[22:02:29] <GothAlice> ObjectID support in aggregate queries has been a bane of my existence, as I generally avoid adding explicit 'created' dates to my documents. ('Cause you can just pull that from the _id. Not so much in an aggregate query, though.)
[22:03:13] <speaker1234> How would you specify a set of values for the spec? In the updated example, the spec is {'x':'y'} but what if you wanted to match more than 'y'. What if you want to match something like {'x':'y'|'g'|'h'|'z'}
[22:04:28] <speaker1234> GothAlice, I'm assuming that it doesn't need to be string literals for the values?
[22:04:47] <GothAlice> http://docs.mongodb.org/master/core/read-operations-introduction/#query-interface
[22:05:46] <speaker1234> GothAlice, to answer your question, 'spec' is the first argument to update in pymongo.
[22:06:54] <speaker1234> GothAlice, oh I am orders of magnitude more successful with mongo in the first two hours of using it that I was in the first two days of using sqlite
[22:07:32] <GothAlice> Interesting SQLite fact: SQLite ignores your schema if you choose to write a non-conforming value to a field. (I.e. the column may be `int`, but you can write strings to it.)
[22:09:10] <GothAlice> If nothing else it'll let you rapidly prototype ideas like a bat out of hell. :D
[22:09:38] <speaker1234> It's also easy to use when your hands are broken. This is why I write code in Python. It's not just a good language, it is something I can dictate with minimal overhead
[22:10:33] <GothAlice> I managed to use Mac OS's built-in recognition to code JavaScript and Python for a while. Quite handy. "function hello open parenthesis name equals double quote world double quote close parenthesis open curly brace newline …"
[22:11:18] <speaker1234> I've been living with it for 20 years. And the problem is, people assume that writing code as yougave as an example, is an acceptable solution.
[22:11:50] <speaker1234> Is a big guy that showed program using speech recognition Python convention. Forget his name
[22:12:53] <speaker1234> yes. He uses what is called the "grunt, belch, fart" school of speech recognition
[22:13:15] <speaker1234> he is actually brutally harassing the language model and is reducing its effectiveness for ordinary speech.
[22:13:37] <speaker1234> I'm working on a different technique which works with the language model and is more intentional in its interface than expressive
[22:13:57] <speaker1234> look up on YouTube togglename and you will find a very raw demonstration of the first of the techniques
[22:14:07] <GothAlice> Can't argue with success, though. I could imagine his approach is speedier once you're acclimatized (or if you develop the linguistic set yourself from scratch, you'll know it to start with).
[22:15:42] <GothAlice> speaker1234: True that. I'm thinking that a metric ton of Automator bindings as Speakable Items *might* work for iMovie, but professional software? Not even close.
[22:15:55] <speaker1234> https://docs.google.com/presentation/d/1nKJu5m9B19FjpvWVZlvY0hJTKG61ohvNG5zkrViRyvM/edit?usp=sharing
[22:16:23] <speaker1234> this is the presentation showing some of the next stage. It's a way a template expansion can be driven through a question-and-answer session
[22:16:57] <GothAlice> speaker1234: Slide 3: No cursor insertion point support? (i.e. drop into the __init__ arglist?)
[22:18:01] <rainabba_> I have a dedicated mongodb instance on AWS and I need to move my data to a shared instance with Modulous.io where I expect that I'm lacking many permissions. Should I bother trying CopyDB (http://docs.mongodb.org/manual/reference/command/copydb/) ? Any suggestions on what likely would work that I should try if not that?
[22:18:01] <speaker1234> it's a work in progress. :-) There are other things I need to add like dock strings and integration into a programming environment
[22:19:01] <GothAlice> rainabba_: mongodump from source, mongorestore to destination. You can prune the sub-folder under the "dump" folder that gets created to remove collections you aren't allowed to write to to avoid errors during import.
[22:19:39] <GothAlice> morenoh149: db.collectionName.findAll() in the shell. Standard "find" query from a driver (i.e. from Python) will provide a cursor you can iterate over to retrieve all results.
[22:21:18] <GothAlice> rainabba_: At work I actually have to run one mongodump per collection I wish to export, since the production user has no access to the system collections in the database, so the pruning is done during export rather than import. It's a PITA, though.
[22:21:24] <Moogly2012> can anyone help me find a way to get the highest value of a field? I've tried sorting, but it returns back empty values (empty string), and I can't seem to do the right incantations to block them out
[22:22:38] <GothAlice> Moogly2012: {myfield: {$not: null, $exists: 1}} — add this to your query to exclude them with extreme predjuice. :) (Think I've got that right; the driver I use hides the raw queries from me.)
[22:22:45] <rainabba_> GothAlice: Dump creates a folder-structure where ever I'm running the command from so as long as I can connect to source and destination from there, I don't need to worry about moving the intermediate data?
[22:23:07] <speaker1234> GothAlice, another update question: 1) can multi be limited to a a finite number of matches and, 2) does update return the records it updated?
[22:23:13] <GothAlice> rainabba_: Run 'man mongodump' or `mongodump --help` for details of the setup.
[22:24:53] <GothAlice> Moogly2012: MongoDB, like most databases, has four-state logic. Values may be "truthy" (1, true, non-empty string), "falsy" (0, false, empty string), null (explicitly devoid of value), or completely missing (value not defined).
[22:25:20] <GothAlice> Moogly2012: This means when trying to exclude "empty" and "missing" values, you have to handle both cases to be safe.
[22:26:15] <GothAlice> Moogly2012: Sorting on your field -1 (descending) and limiting to a single result will find your "maximum" value. :)
[22:27:43] <Moogly2012> I still get negative numbers, been able to exclude empty values, but when it comes to sorting I have no such luck for whatever reason
[22:29:13] <Moogly2012> but I had gotten back empty values, now I just get negative values with $not: null, $exists: 1 included
[22:29:38] <GothAlice> Moogly2012: If you're only getting negative numbers, I suspect all your numbers are negative. ;)
[22:32:39] <Moogly2012> I'm acustomed to other databases, never used mongo, so I don't even know how to check for data types
[22:32:46] <shoerain> hm, 1. are there examples of migration scripts in mongodb (changing field names, removing fields, updating values, etc) and 2. examples of #1 using mongoose?
[22:33:40] <shoerain> I have some examples from work that look less than ideal, but I could just have a hard time following callbacks.
[22:34:03] <Moogly2012> odd, in some of the results the field is a string, while in others, it's a int
[22:34:38] <GothAlice> shoerain: Most of those operations are quite simple and straightforward. Changing isn't (you'll actually need to load the record up and issue an update per-record), but removing and mutating values is quite easy. $set to assign a new value, $unset to remove, and various atomic operations such as appending to lists, incrementing / decrementing, etc.
[22:35:17] <GothAlice> Moogly2012: MongoDB enforces no schemas. I use MongoEngine (Python driver wrapper) to provide those for me where needed.
[22:35:31] <GothAlice> Moogly2012: Even though I use an ODM, though, I still end up rolling a lot of manual queries.
[22:35:34] <Moogly2012> GothAlice: recommended way to remove documents with a specific empty field? Maybe that might help
[22:37:39] <GothAlice> The number of times I've accidentally my whole database with queries I thought were rock-solid I can't count on my hands.
[22:39:46] <GothAlice> The last screwup was an unbounded update() that nuked three different attributes worth of data for an entire collection. Before that was the incident that made me drop MySQL like a hot potato: after a lengthy EC2 failure I spent 36 hours straight (without sleep) recovering the data by reverse engineering the on-disk structure… prior to getting 8 teeth removed. (Wisdom teeth and impacted molars.)
[22:40:27] <GothAlice> The latter screwup was a doozie. Learned enough about innodb on-disk formats to make me literally fear storing data that way ever again.
[22:42:06] <GothAlice> (directory-per-db? Not if you use innodb! It lies straight to your face about it and pretends to work, but pools important data structures outside the DB folders.)
[22:44:49] <GothAlice> Moogly2012: That's because I'm a terrible person who uses abstractions waaaay too much. db.collection.delete({criteria})
[22:48:11] <GothAlice> Then you'll need to adjust your removal criteria to hanlde that case. The one I gave you only handles if the value is actually missing. :)
[22:48:11] <GothAlice> (You'll want to either run several remove() queries, or use $or to combine the criteria.)
[22:52:58] <GothAlice> For an advanced look at one of the niftiest things MongoDB can do, see: https://github.com/bravecollective/forums/blob/develop/brave/forums/component/thread/model.py#L99-L161
[22:53:57] <GothAlice> This is for forum software that stores all replies to a thread within the thread document. These methods handle appending a comment, retrieving a single nested comment, and updating a specific comment in the thread. (At the bottom are some queries that get either the first or last comment in a thread.)
[22:55:27] <GothAlice> Boomtime: We calculated it out. Migrating the entirety of the gaming group's forums (a few thousand users at the time) from the phpBB fork they were using (eveBB) to this would require a total of a single document. (< 16MB for *everything*.)
[22:55:53] <GothAlice> I estimated that the average thread size would have to grow by over 1000x before I would even consider adding automatic splitting of thread replies.
[22:56:59] <GothAlice> Also the migration (properly splitting the threads) ran great and now they're up to > 10,000 users. :)
[22:57:12] <Boomtime> yes, 16MB is lots of text, but the limit is not unreasonable for a popular forum thread to reach - what happens at that point?
[22:57:34] <Moogly2012> brb my nets being funky thanks for the help GothAlice I will possibly be back when my net is not being obnoxious
[22:58:09] <Boomtime> goodo, as long as it doesn't die and you have a response strategy, that is all that matters
[22:59:32] <GothAlice> Boomtime: By a growth of 1000x before I would have a concern I mean the average number of replies would have to grow to 40,000 on a single thread (based on the average length of a reply across all forums) before we hit 10MB per thread. Lots of headroom to actually implement proper thread spanning. ;)
[23:00:04] <GothAlice> I was more eager to add things like live push updates and other "sexy" features. ;^)
[23:08:11] <GothAlice> Huh; now that I check it, by the time I left the project they were already up to 82 words per comment on average. (1,323 threads containing 14,281 comments with 5,734 up-votes, viewed 380,187 times, containing 1,175,505 words.)
[23:09:24] <doug1> I'm getting "Error parsing INI config file: unknown option auth" ... I have auth = true in mongodb.conf. Why?
[23:10:50] <GothAlice> I would strongly recommend upgrading to the YAML-based configuration. http://docs.mongodb.org/manual/reference/configuration-options/
[23:12:28] <GothAlice> Ouch. It may also be that running as mongos, authentication isn't supported, or wasn't supported under the old configuration syntax. You either can't enable auth there, or must upgrade to YAML. :(
[23:14:52] <GothAlice> http://docs.mongodb.org/v2.4/reference/program/mongos/#bin.mongos — yup, no --auth option. You'd add it to the mongod (shard-specific) configurations, not the management process.
[23:15:20] <GothAlice> Note that enabling auth will require you to use a pre-shared key to secure communication between the shards.
[23:19:58] <speaker1234> I'm missing something update. How do you get the list of documents you changed
[23:23:55] <speaker1234> I guess I could create external lock for all access to the database I do find and update
[23:25:09] <GothAlice> joannac: localhost exception; I KNEW I was forgetting something important earlier, with another user who had authenticated sharding/replication questions.
[23:26:15] <GothAlice> speaker1234: It's less of a race condition than one might think. If you apply the same atomic operation (i.e. set X to 27 or append Y to a list) client-side after confirming the success of the update, you should be fine.
[23:26:26] <speaker1234> that's right. Find and modify was mention earlier. I have a bunch of records, each one indicates some is to be done. I changes state the record and hand it off to worker bee
[23:27:17] <GothAlice> speaker1234: https://github.com/marrow/marrow.task/blob/develop/marrow/task/model/task.py#L30-L90 is an example of implementing atomic locking on MongoDB records, FYI.
[23:27:46] <GothAlice> speaker1234: Also directly applicable to your use case, as this is part of a distributed task execution system. ;)
[23:28:40] <speaker1234> GothAlice, you keep throwing me all this code I don't quite understand yet. :-)
[23:28:55] <Boomtime> can you do an update that specifically sets an identifier field on documents that it touches? allowing you to find the modified ones afterwards?
[23:29:04] <GothAlice> speaker1234: https://gist.github.com/amcgregor/4207375 is the slides from the presentation I gave on this which truly simplify this type of thing down to the bare essentials. Link in the comments there to the more complete code which marrow.task is based on.
[23:31:52] <GothAlice> https://gist.github.com/amcgregor/4207375#file-4-job-handler-py-L7-L8 sets the `state` to "running", the `acquired` date to now, and the current "owner" of the job to the current process, but *only if* the job is still in the pending state, with no owner when we execute that query. Update-if-not-different is awesome. (Two processes can execute that statement at exactly the same time, only one will win.)
[23:32:30] <GothAlice> Note that this code doesn't even bother to get the record first, it waits for success on that lock before trying to find() the actual document.
[23:34:11] <speaker1234> I think I get the gist of it. You are still doing some magic that I don't understand where it comes from. For example the owner identity and using __class__ objects.
[23:34:53] <speaker1234> To be truthful, I'm seriously tempted to use portalocker on an external file just out of expediency
[23:35:01] <GothAlice> For the most part you can ignore my Python-fu. An "identity" (like https://gist.github.com/amcgregor/4207375#file-1-example-job-record-js-L10) is just a combination of [hostname, pid, parent_pid].
[23:36:31] <speaker1234> Actually, I was, in a previous life, a deep OS weenie. I could untangle lock and memory management problems others were baffled by
[23:36:44] <GothAlice> Boomtime: Not to mention optionally redundant (replicated) and optionally partitioned (sharding) which are both monumental tasks by themselves. ;)
[23:36:51] <Boomtime> GothAlice beat me to it, seriously do not implement your own lock system if you can even remotely avoid it
[23:37:10] <speaker1234> but in this case, I would use exclusive locks based on the type of state change
[23:37:30] <GothAlice> speaker1234: That's exactly what https://gist.github.com/amcgregor/4207375#file-4-job-handler-py-L7-L8 does. ;)
[23:38:11] <GothAlice> doug1: Spin up the new cluster with auth enabled, SSH into the master and use the mongo shell to add the user.
[23:40:14] <speaker1234> GothAlice, maybe I'm just tired but I'm not grokking how that Locke works. Maybe. Are you using the record ID and a separate container to record the lock status?
[23:41:26] <GothAlice> speaker1234: Because of the way MongoDB implements its "queue" analog I had to split storage of the actual task data into its own collection and only use the "capped collection" (queue) for push notification of events on those tasks. Thus the runners get woken up and pushed a new task ID, then execute that lock in a thread/process pool to try to grab the task so it can be worked on. Only IDs get passed around cross-process this way.
[23:42:46] <GothAlice> speaker1234: You would have to determine the size of your capped collection based on the estimated number of waiting jobs. For the Facebook game I wrote those presentation slides from we estimated 8GB capped collection to handle 1M simultaneous users active' gameplay.
[23:42:49] <doug1> joannac: they had no solutions to provide regarding the automated installation of a mongo cluster. They're main priority seemed to be seeing MMS. When asked about chef they said they heard the Edelight cookbook works, which it plain does not. That's the one I've spent 4 months trying to get to work
[23:42:50] <speaker1234> So maybe the trick for me is to find a record, update the state for just that record ID, I need to go back and get another record
[23:43:32] <doug1> I logged into the MMS GUI yesterday. I can't even work out how to start a cluster with it
[23:44:21] <speaker1234> doug1, remember that the only intuitive user interface is the mammalian nipple and as any nursing mom will tell you, there are some that still don't get it
[23:45:01] <GothAlice> speaker1234: That is the general approach that the runners use. They have a thread constantly pulling in "new job" messages and handing the IDs off to a pool of actual task executors which then do the locking, as quickly as possible. (In my case all runners are notified of all eneuqued tasks and all locally queue the task for execution. Failure, due to the lock, is very fast.)
[23:46:00] <GothAlice> I.e. you need a cluster which you then install the agent onto in order to get statistics and monitoring.
[23:46:48] <GothAlice> Rackspace and many other providers offer "rollouts", i.e. you can simply tell Rackspace you want a three-node cluster and it goes off and does it. Not very customizable in my experience, though. (I.e. no automation of enabling authentication.)
[23:46:56] <speaker1234> GothAlice, okay great. So I can use the find an updatesuccess or failure trick to guarantee that I get the record. Now which would be better, grabbing one record at a time or finding a set of records that match and updating one at a time to find out which ones I "own"
[23:47:45] <joannac> you have a host. you probably don't want to deploy a sharded cluster on a single host
[23:48:20] <doug1> "MMS can deploy MongoDB on any internet connected servers, but on Amazon Web Services (AWS), MMS does even more."
[23:49:00] <doug1> "Supported instance type provisioning: m3.*, c3.*, r3.*, g2.2xlarge, hs1.8xlarge, i2.xlarge"
[23:49:10] <joannac> doug1: after finding some hosts, and deploying a cluster / replica set, you can monitor them
[23:49:27] <doug1> "MMS reliably orchestrates the tasks you currently perform manually — provisioning a new cluster, upgrades, restoring your system to a point in time, and many other operational tasks."
[23:50:17] <GothAlice> speaker1234: Tasks tend to be atomic, i.e. each one can complete independently of the rest. Depending on how you distribute tasks within your runner (i.e. threads, subprocesses, or pure linear execution) will determine which approach you take. If you fill local queues to feed threads and subprocesses, you can batch it all together as each thread or subprocess will individually handle a single task.
[23:50:33] <doug1> I signed into the GUI. I tried to do something... like start a clustter... not apparently functionality could be found
[23:51:07] <GothAlice> speaker1234: A pure linear approach would only ever tail the capped collection, executing each task as it gets them. (You could then use a tiny 'catchup' script to re-inject tasks into the capped collection that were skipped because they were added before the runners were launched; or just increase the size of your capped collection to compensate.)
[23:51:21] <speaker1234> GothAlice, there will be a bunch of parallel requests. Think 100 worker bees wanting some more work
[23:52:11] <GothAlice> speaker1234: I'm basically saying that you can load the data at effectively any point: when your worker is initially notified of the task or when your worker actually gets around to running the task.
[23:52:48] <GothAlice> speaker1234: My "worker bees" each have their own thread pool, each thread handling the locking, loading, and execution of a single task.
[23:54:17] <GothAlice> speaker1234: I'm hoping by the end of this coming week-end to have marrow.task in an operational state, if you can wait that long. Patches are welcome and encouraged! ;)
[23:54:51] <speaker1234> I need to do a test run by the start of next week. This is I can prove to customers that we have a viable infrastructure, then I will have more money to go back and revisit implementation details.
[23:55:56] <speaker1234> I'm having to do the simple approximation first, prove that the idea works, get the money to do more, and then revisit with a second implementation. In other words, what mythical Man month tells you will happen
[23:56:55] <doug1> This http://www.mongodb.com/blog/post/installing-mms-monitoring-less-5-minutes just talks about it in the context of monitoring, not building a cluster
[23:57:02] <speaker1234> GothAlice, once I prove things,I have one customer that wants to use the queue similar to what you have for capability and I'll be at least able to either hire someone to help or do and contribute the work myself
[23:57:14] <GothAlice> Yeah, that's pretty tight. https://gist.github.com/amcgregor/4773553 is the "more complete" implementation of the snippits from the presentation. Won't run out of the box (has some hanging references to internal code) but should be pretty quick to patch and get running.