Log file Viewer
#mongodb logs for Friday the 24th of October, 2014
[00:17:11] <freeone3000> How do I get around "Error getting results.MongoError: exception: aggregation result exceeds maximum document size (16MB)"?
[00:17:41] <GothAlice> Adjust the query so that the results fit within a single document, or $out the results to a real collection.
[00:18:42] <freeone3000> It's an aggregation pipeline. The output I care about is a single number, I just have to run a $where, $where to do an and, a $group for uniqueness, then a $group for count.
[00:19:11] <freeone3000> Apparently if any stage of this goes over 16MB, including the reuslts after the *first* $where, it'll not give me results.
[00:19:39] <GothAlice> freeone3000: Having two $group operations is dubious to me, but. Individual stages have 100MB of RAM to work with unless you enable the option to let them use on-disk temporary files.
[00:20:41] <freeone3000> GothAlice: It looks like https://gist.github.com/freeone3000/6a46ac80c7dc661feeda
[00:22:02] <freeone3000> GothAlice: How would I combine those two groups? {"_id":"$requestInfo.userId", "userCount": {"$sum":1}} gives me results like [{"_id":"user1","userCount":22}]
[00:25:33] <GothAlice> [{$match: {dateAdded: {$gte: startDate}, source: {$in: routes}}}, {$project: {uid: "$requestInfo.userId"}}, {$group: {_id: "$uid"}}, {$group: {_id: 1, userCount: {$sum: 1}}
[00:26:02] <GothAlice> Might need to move the project in front of the match simply to avoid loading all that extra data and blowing out the 100MB-per-stage limit.
[00:32:05] <GothAlice> We do statistical aggregation at work, but we pre-aggregate all the values that user-triggered queries run against for performance. (Processing four weeks of click data to produce one chart takes ~50ms against a dataset pre-aggregated into one-hour slices by client, invoice, job, and click source.)
[00:32:50] <GothAlice> Processing clicks pre-aggregated into hourly chunks means I have far, far fewer records to handle.
[00:34:46] <freeone3000> These are already pre-aggregated by conversation. We just have a bit of load.
[00:36:12] <GothAlice> If your aggregate is running out of memory, that's a query and dataset problem, not a load problem. ;^P
[00:37:19] <GothAlice> freeone3000: [{$match: {dateAdded: {$gte: startDate}, source: {$in: routes}}}, {$project: {uid: "$requestInfo.userId"}}, {$group: {_id: "$uid"}}, {$group: {_id: 1, userCount: {$sum: 1}}]
[00:38:51] <GothAlice> The trick is combining the $match queries; you didn't actually want to "find all records with a dateAdded greater than startDate", store the temporary results, then filter those results to only matching routes.
[00:39:05] <GothAlice> Also $project to reduce the dataset down to only the values you need for the calculation.
[00:40:04] <GothAlice> freeone3000: My aggregate queries tend to look like: https://gist.github.com/amcgregor/7bb6f20d2b454753f4f7#file-aggregate-py-L6
[00:45:07] <GothAlice> freeone3000: https://gist.github.com/amcgregor/1ca13e5a74b2ac318017 is one of my aggregates a bit more like your test case here. The DOTW conversion is hideous. XD
[00:46:11] <GothAlice> (If anyone has any suggestions on how to improve that aggregate, I'm all ears. ^_^)
[01:39:17] <zak_> HI.... late to the party on mongo... is a document with a field which is an array of documents queryable at all...so { title : "something", authors : [{name : "bob", ...}, ...] , ... }
[01:41:54] <GothAlice> You could do it with a standard query, but with duplicates: db.foo.find({}, {authors: 1}) — loop through the result with a nested loop over .authors
[01:44:03] <GothAlice> Or you could do an aggregate query to only get the *unique* authors out: db.foo.aggregate([{$project: {a: "$authors"}}, {$unwind: '$a'}, {$group: {_id: '$a'}}])
[01:52:52] <zak_> GothAlice, joannac, i got kicked off for a second -- so that query returns something like a Map (book => (book.id, book.authors)). what's the mongo pattern here, is it better to have a second "authors" collection and refer to them like foreign keys
[01:53:22] <GothAlice> zak_: The answer to that question lies in how much duplication of author information you have.
[01:54:05] <zak_> but unfortunately it's not reliable, i have to make a model of some kind to disambiguate them
[01:54:15] <GothAlice> You would instead store a list of author IDs (rather than a list of author documents).
[01:55:08] <GothAlice> At work we have a process to merge (automatically or manually) records we come across that are duplicates. How are you "model"ing your data?
[01:56:53] <zak_> and i'm using mongo to store and index all these flat files i have that have details on books
[01:57:55] <GothAlice> You had authors: [{name: "bob", ...} in your model, so {name: "bob", …} would be one of these "authors" collection-level documents.
[02:00:42] <zak_> i'm sure the insert/select capabilities are the same as any. i was doing this with postgres a few days ago, but i inserted millions of records in minutes using mongo, so i'm convinced i should be learning it
[02:01:22] <zak_> whereas i'm used to building dbs with a ton of tables and massive inner join nonsense, so i guess it's the same problem
[02:01:52] <GothAlice> Yes. MongoDB seems to penalize bad data architecture more than other databases. Then again, the "flat file" mentality is pretty… restrictive. You end up inventing approaches like entity-attribute-value to work around the limitations, and you'd pretty much never do that in MongoDB.
[02:02:17] <GothAlice> http://docs.mongodb.org/ecosystem/use-cases/product-catalog/ is a good article about that, actually.
[02:03:22] <GothAlice> When I first told people I'm writing forums software that stores all replies to a thread within the thread, my relational friends looked at me like I was a crazy person. ^_^
[02:05:49] <zak_> GothAlice, well that seems like a great application of this right, since the root level document being the post itself is good
[02:07:25] <GothAlice> Well, that's "all threads and the first comment by that person on the thread". Aggregate $match, $unwind, $match, $project for the "all posts by a user" bit.
[02:08:45] <GothAlice> ($match to find all threads commented in by that user, $unwind to continue comment-by-comment-by-thread, $match to find just that user's comments, $project to clean up the data to just the fields I want.)
[02:11:08] <GothAlice> zak_: https://github.com/bravecollective/forums/blob/develop/brave/forums/component/thread/model.py#L99-L192 are some of the other things you can do. (I.e. find a specific comment anywhere, add a comment, modify one in-place, get the first and last (or any arbitrary index) comment, etc.
[02:12:22] <zak_> hey thanks for the link, i've been looking at solr for a new project too so i'm sure this will be good to look at
[02:13:25] <GothAlice> (https://github.com/bravecollective/forums/blob/develop/brave/forums/component/thread/model.py#L70-L80)
[08:53:05] <Mmike> That is, i can have 3 boxes in replset, and each of the members can be a master and have a slave attached to it?
[09:00:58] <alphamarc> What are the greatest advantages of Mongo (and NoSQL in general) when it comes to store log information
[09:03:30] <alphamarc> I would like to have a fast and generic system to store logs for different services within a firm.
[09:10:51] <kali> Mmike: master/slave is more or less being deprecated, and it becomes less and less relevant with the recent availibility of huge replica set (50 nodes)
[09:11:58] <Mmike> kali: still, you can have a slave on each of you replset nodes - that particular replset node is then 'master, right?
[09:54:15] <Tark> db.struct.find({'number': {$type: 1}}) returns all the collection. There is all values is double! Holy hell! I want int. What should I do?
[10:20:12] <jabclab> hey all, is it possible use functionality from an npm lib within a map() function of mapReduce (using http://mongodb.github.io/node-mongodb-native/contents.html)?
[10:59:37] <Mmike> Fri Oct 24 09:16:12.118 [initandlisten] ERROR: can't use --slave or --master replication options with --replSet
[12:55:07] <Luser> hey guys. I am getting invovled to project which uses mongodb on monday and I would like to get some experience with it this weekend before i start. Could you drop me a link or two with learning resources?
[13:03:44] <Di> Try this https://github.com/mongodb/mongo-ruby-driver/wiki/Tutorial but you need to learn mongo's query language documentation anyway
[13:39:55] <leenug> Hi, is anyone able to see why the follow query doesn't return any results: http://laravel.io/bin/LN8LO, there are plenty in the collection, I think I may be doign something wrong with dates, thanks
[13:58:24] <shoerain> hmm, what's the equivalent non nested version of `db.campaigns.find({'topics.included': true }).count()` ? I thought it would be `{topics: { included : true }}` or `{topics: [{ included : true }]}`, but I get 0 count for both.
[14:04:21] <shoerain> leenug: does some subset of the query work? Also you can merge $gt and $lt together, like so: http://cookbook.mongodb.org/patterns/date_range/
[14:49:48] <shoerain> kadosh: would you be looking for somtehing like this: http://dba.stackexchange.com/a/63295 ?
[16:06:42] <locojay_> hi need to run a repairDatabse() since disk size and db size do not match. i only have 200gb left on a 700gb db so i m using --repairpath You must use a --repairpath that is a subdirectory of --dbpath when using journaling
[16:28:38] <GothAlice> locojay_: I'm reeeally not sure what you'd be symlinking, and why that might have "helped", but there will always be a mismatch between the on-disk size and the actual size of the data contained within.
[16:29:28] <GothAlice> Not at all. There are now holes in the on-disk files that will get filled in as you add new data.
[16:30:05] <GothAlice> (GridFS chops files up, too, into separate records. This lets them be packed more efficiently.)
[16:30:33] <locojay_> k so you say i should not me woried about the actual filesize but just worry if i have enought dispace - dataSize?
[16:30:36] <GothAlice> However, what you want is "compact". http://docs.mongodb.org/manual/reference/command/compact/
[16:43:15] <GothAlice> (At least, vacuum full = compact. MongoDB normally self-vacuums dead objects.)
[16:44:39] <GothAlice> sub_pop: In the raw struct, yes, there's probably a BSON helper lib function to assist with that.
[17:01:32] <jaequery> ive heard stories of how mongo is not really suitable for huge databases? is this really true or just a misguided information?
[17:01:49] <GothAlice> jaequery: I have 24TiB (growing at 12-24GiB per day) in MongoDB without issue.
[17:02:52] <GothAlice> jaequery: I record everything, and have been since 2001. (Transparent HTTP proxy, natural language processing for automated tagging and content extraction, etc., etc.)
[17:03:54] <jaequery> do you ever come across times where you need to restructure your collection definitions? and how would you handle that?
[17:03:59] <GothAlice> Non-HTTP content gets slurped in on a regular basis (i.e. from my chat logs).
[17:04:50] <GothAlice> $unset (to remove fields), $set (to add new fields), etc. work, but across the entire dataset those queries can take hours.
[17:05:27] <jaequery> in SQL, you'd just create a new table and join after. can same be said for mongo?
[17:05:40] <GothAlice> However generally I don't need to ever do those against the whole dataset. Each document in the primary metadata collection has a different structure related to the type of data it referneces.
[17:08:26] <GothAlice> jaequery: For the most part, since I won't have data for newly added fields most of the time, I don't worry about adding a "null" or "empty string" value to all of the data. There's no point. It'll be used and present on any record I update where I set it.
[17:10:24] <GothAlice> jaequery: The version of my exocortex (the name of the project) that last used SQL used multiple table inheritance. It was unbelievably painful, but I hadn't learned of EAV at the time.
[17:20:24] <obeardly> Hello Mongoers; can anyone here tell me if it is possible to assign a role to multiple users at once?
[17:21:09] <GothAlice> obeardly: In theory you could execute an update statement against the system.users collection that would update several records at once. $push is the one you want to "add" a value to a list.
[17:23:56] <GothAlice> Where you can have as many attributes and values as you want, up to the document size limit. ;^)
[17:35:43] <ejb> GothAlice: Remember my query? I have that list of generated $cond's (in Python - or JS in my case)
[17:36:43] <ejb> GothAlice: That's fine. But if I add another item to the $add list like this: {$divide: [{$subtract: [radius, '$distance']}, radius]}
[17:37:21] <ejb> GothAlice: When I $sum the score in the $group stage, the number is a whole number, not a float like it is in the $project stage
[17:46:31] <ejb> Not sure. I broke the $cond array out into its own variable then did push(divideExp)
[20:08:57] <baegle> Is there a way to create a database, fill it with data, and then set it to read-only?
[20:09:42] <GothAlice> You can semi-enforce read-only access by only accessing the database on a replication secondary. (Writes must go to the primary.)
[20:10:36] <GothAlice> baegle: Otherwise you could have a boolean field on your documents indicating read/write status, and filter your update()s on that. (I.e. only allow updates on records whose ro: field is false or rw: property is true.)
[20:11:52] <baegle> No, the answer is I can prevent applications from writing to the database by various mechanisms but I cannot create a database, fill it with data, and then set it to read-only
[20:12:26] <GothAlice> Only connecting to a secondary has the effect of making the entire connection read-only. Access your read-only database through a secondary, and it's really read-only.
[20:13:16] <GothAlice> Thus careful management of your database connections (i.e. using a connection to the full cluster for most, a direct conenction to a secondary for read-only data) can secure your data in the way you want.
[20:13:38] <baegle> the equivalent conversation about file systems would be that no, you cannot set -w, but you CAN mount the entire FS with ro.
[20:13:55] <baegle> in that conversation the question "Can I set a file to read-only?" has the answer of "no"
[20:14:03] <GothAlice> baegle: Think of it more like having a filesystem, then mount -o bind,ro a subsection of it.
[20:14:18] <GothAlice> You can mount -o bind a single file if you wish, making that one file read-only.
[20:14:34] <baegle> but that's not actually true because the file system also has to be mounted writable for it to have a replicant, correct?
[20:14:46] <baegle> It's not like I can shut-down the writable database and leave the read-only one there
[20:15:19] <GothAlice> Spin up two shards and an arbitrator, populate the primary, kill it, the secondary will freak out and enter r/o mode.
[20:15:54] <GothAlice> At the same time, since it can't get a majority of votable nodes, it likely won't accept new connections either.
[20:16:54] <GothAlice> Really, though, at work we store frequently-accessed purely readonly data in SQLite whose on-disk file is non-writable.
[20:22:57] <GothAlice> (My managers keep using the words "big data"… I tried to make it clear to them we do not qualify for that phrase, yet.)
[20:25:03] <mike_edmr> http://spectrum.ieee.org/robotics/artificial-intelligence/machinelearning-maestro-michael-jordan-on-the-delusions-of-big-data-and-other-huge-engineering-efforts
[20:25:29] <mike_edmr> how we're in a phase of heightened expectations for what it can do, and theres likely going to be a bit of a backlash
[20:26:10] <GothAlice> At work here I've been tasked with producing analytics across our job/click/referral data. The key metric is cost per applicant. There is zero possibility we can calculate this value, but they still want it.
[20:26:32] <GothAlice> (We don't know if the user clicks on 'apply' and actually fills out the form after the tracked click to the job detail on the foreign ATS.)
[20:27:56] <GothAlice> So they were, like, well, we *can* measure intent to apply, right? No. Intent to apply would be clicking apply regardless of actually filling out the form. We can't get that either… just cost per click. So we're calling "cost per click" (CPC) "intent to apply" anyway. >_<
[20:29:32] <mike_edmr> GothAlice: when i figure out how to save myself from mine, you'll be next in line
[20:31:28] <GothAlice> Great article, BTW. That guy uses average sentence lengths that exceed mine, which is kinda nuts. XD
[20:45:04] <GothAlice> On the natural language bit, I just asked my phone "what was the phase of the moon on July 12th in Argentina?" (waxing gibbous, 99%), "what is the specific heat of hydrogen cyanide" (1.328 J/g K gas, 2.612 J/g K liquid), "what is the molecular structure of lithium cobalt oxide?" (it showed me), and "what is the speed of light in furlongs per fortnight" (about 1.8 trillion). The modern era is kinda mind-blowing when you think about it.
[20:58:40] <dbb> .. new to mongodb.. I am just walking through some examples.. question about roles/users in 2.6
[21:00:30] <dbb> I want to just make a few things and try them.. will I get tangled on this somehow ?
[21:00:36] <GothAlice> dbb: The mongo shell is a MongoDB "client", like a webapp would be. It connects to the mongod process over a network connection, even on the local machine.
[21:01:20] <dbb> ok yes.. is it advisable to make a dbb user and make it an admin, to simplify the exercises
[21:01:49] <GothAlice> As long as you either don't care about your playtesting data set, or have backups, you can always stop mongod, nuke /var/lib/mongodb/*, and start mongod back up again to start from scratch. When I run mongod locally for development, I don't even bother with authentication at all. (I do make sure it only accepts local connections, though. --bindHost 127.0.0.1 or similar.)
[21:03:01] <GothAlice> dbb: Almost all data in MongoDB, even when using the shell, are passed around as JSON/BSON structures. You can use the helpers, and you can also directly query against the collections that the helpers manipulate.
[21:03:39] <GothAlice> A la: http://docs.mongodb.org/manual/tutorial/add-user-to-database/#create-the-new-user
[21:03:40] <dbb> I have a VM with rockmongo in it.. but I wont want to depend on that going forward..
[21:04:24] <voidDotClass> Can you have nested information in a document? E.g a Family document which has nested documents for each child?
[21:05:32] <GothAlice> Yes. MongoDB stores rich, deep structures, and provides ways to query and manipulate that data. However often it's best to step back from the data and really examine how best to model it. Nesting is unlimited, but not useful beyond a certain point. Don't do that.
[21:05:58] <voidDotClass> What would be a better way to query for the children of a particular family, in my above example?
[21:06:13] <GothAlice> For example, consider simple forums. You have threads/topics, replies/comments to those threads, forums (that group threads), and categories (of forums).
[21:07:05] <GothAlice> You could store something like: {category: "General", forums: [{forum: "Random", threads: [{title: "General randomness.", comments: [{author: "bob", comment: "Oh boy."}]}]}]}
[21:07:50] <GothAlice> Instead, comments make sense to store in a thread (the finest-grained thing you can look at is a thread) while storing the forums and categories separately.
[21:08:33] <voidDotClass> GothAlice: What about if forums was an array of forum ids, so if you got a category, you'd have a list of its forum ids, and could then query for those ids on the forum collection?
[21:09:49] <GothAlice> {category: "General", forums: [ObjectId(…), …]} — this would let you manually order the forums within the category (good), but require two lookups if you want to search forums on anything else than category.
[21:10:11] <dbb> voidDotClass: "how do I use a document-oriented flat store to model my relational problem"
[21:10:58] <GothAlice> voidDotClass: And would require updating of a record in another collection if you add or remove a forum.
[21:11:56] <GothAlice> voidDotClass: It's a constructor for an ObjectId datatype. MongoDB doesn't use auto-increment integer keys, it uses something that scales a whole lot better.
[21:12:50] <GothAlice> voidDotClass: For further reading: http://docs.mongodb.org/manual/reference/object-id/
[21:13:52] <voidDotClass> GothAlice: Is that the same as my above example then, where I said it could category could store an array of forum ids?
[21:14:45] <voidDotClass> Is there any ORM or such for Java in Mongo, which would take care of automatically updating the forums array on categories, if any forums were added or removed to a category?
[21:15:21] <GothAlice> I'm sure there must be a client driver that simulates triggers for you. MongoEngine on Python does.
[21:15:45] <GothAlice> "Database triggers", i.e. "when I insert into this collection, run this function", etc.
[21:18:12] <voidDotClass> Mongo seems easier to use than couch. Only thing I'm considered about is clustering. Do I need to setup 5 db nodes even when I only need one, i.e when I have 0 users?
[21:18:33] <GothAlice> voidDotClass: https://gist.github.com/amcgregor/18e3180215736369b49e is an example of one of my triggers, and how to use it. Very simple in Python/MongoEngine.
[21:19:45] <GothAlice> voidDotClass: Replication is to handle reliability and failover, as well as backups and offloading read-only queries, and sharding is to help scale. You don't need these, but they do become useful in production environments.
[21:20:14] <GothAlice> voidDotClass: https://gist.github.com/amcgregor/c33da0d76350f7018875 is a tiny script to start a small authenticated, replicated, and sharded cluster on one machine. :)
[21:21:21] <voidDotClass> GothAlice: If the cluster is run on one machine.. i.e one node on AWS, how will it scale?
[21:21:49] <GothAlice> voidDotClass: Oh, it won't. It'll be pretty terrible, in fact. That script is to help test application response to an environment like that in development.
[21:22:27] <joannac> GothAlice: replication is not to scale reads! askasya.com/post/canreplicashelpscaling
[21:22:44] <GothAlice> This script constructs 10 processes, in theory three of them would be shared on machines doing other things, so that setup would need 7 nodes.
[21:23:55] <voidDotClass> So, i read somewhere that mongo can automatically start new instances to help scaling. Does that include auto starting new nodes on AWS / load balancing?
[21:24:50] <GothAlice> voidDotClass: There are ways of automating that, yes. MMS seems to have that capability these days, but not sure about actually spinning up new VMs.
[21:26:38] <voidDotClass> "Instead of writing its own sharding layer, Foursquare can rely on MongoDB's automated scaling infrastructure and spin up new nodes as its application grows"
[21:26:42] <dbb> I will certianly be doing this also.. though I expect to write ansible scripts to do that
[21:26:52] <GothAlice> voidDotClass: They likely rolled their own automation to scale VM nodes. That article mostly mentiones auto-failover.
[21:27:50] <GothAlice> dbb: The example script I gave you tells each replication set about all nodes in that set all at once.
[21:27:51] <voidDotClass> Lets say if you have your own automation and you spin up a new VM, how will you add that VM to your mongo cluster?
[21:28:26] <GothAlice> voidDotClass: My VMs announce their startup to a smaller offsite management cluster, which then pings back to the other members to let them know to add the member.
[21:28:38] <joannac> voidDotClass: thanks for the link. I'm not actually sure what that refers to; I think it refers to the fact you can add a new shard at any time, but MongoDB won't just start new instances automatically (because that would be a bit weird)
[21:29:12] <voidDotClass> GothAlice: The other members = members in your offsite management cluster or your main cluster?
[21:29:47] <GothAlice> https://gist.github.com/amcgregor/4342022 is the "name pool" collection for my automation, and https://gist.github.com/amcgregor/2032211 a monitoring document (also in MongoDB).
[21:30:04] <GothAlice> voidDotClass: Main cluster. The management cluster would just record the new member in the name pool.
[21:31:08] <GothAlice> The management cluster can, in theory, spin up VMs on any number of providers, making it much more resillient.
[21:32:37] <GothAlice> One mongod per vm, with the exception of the config servers which are piggybacked.
[21:33:32] <GothAlice> The servers also don't have permanent storage. (Amazon ephemeral storage is fscking awesome, and insanely fast compared to EBS volumes which are the bane of my existence.)
[21:34:18] <voidDotClass> GothAlice: Do you use any particular AMI for AWS? Or anything else to setup your vms when you launch them?
[21:34:20] <dbb> btw GA, I talked to my wizened admin neighbor about DNS.. he believes that I can do my cluster of Amazon VMs without actually naming them
[21:35:15] <GothAlice> My management cluster, you'll note, spends a lot of effort to manage DNS names. ;)
[21:35:36] <dbb> on each VM creation, send a msg to the remote with some profile info and the 10.x IP addr
[21:35:57] <GothAlice> voidDotClass: We run our own homogenous AMI. (One AMI for all servers regardless of role.) Eases distributing updates across the cluster.
[21:36:53] <voidDotClass> GothAlice: What AWS instance type and how many instances would you recommend for a completely new service, that has 0 traffic but might grow or might stay low / zero?
[21:37:14] <GothAlice> voidDotClass: On EC2 you can pass arguments to the VM at the time you request it, too. (We use that to inform the VM which role/branch to git checkout into.)
[21:38:33] <GothAlice> Storing large volumes of data? Running costly queries? Map/reduce? How many queries per minute/hour/etc estimate?
[21:39:16] <GothAlice> dbb: Hey, ignorance can be corrected and is nothing to be ashamed of. The ability to pass on knowledge is what made us human.
[21:40:25] <voidDotClass> No mapReduce or costly queries. Volume of data and queries will be 0, and will depend on traffic.
[21:40:52] <GothAlice> voidDotClass: So for a basic start you could begin with a single instance with a minimum of 2GB RAM, must be 64-bit (for a variety of reasons), and two cores. I'd recommend three in a replica set to make sure your data doesn't accidentally itself.
[21:42:06] <GothAlice> You can start with one and later replicate, but your data is "at risk" as long as it's only on one node.
[21:43:25] <voidDotClass> how would you determine if more VMs are needed? by checking the CPU / memory usage on each VM?
[21:43:38] <GothAlice> voidDotClass: It can seem silly to only have, say, one application server and *three* database servers, especially when first starting, but…
[21:45:22] <GothAlice> voidDotClass: Generally you want your entire dataset to fit in RAM. When you can't scale a single machine up (by adding more RAM) any more, you have to split (shard) your data into separate chunks on separate servers. There's a break-even point to it that I'll leave as an exersize for the reader. (Where the cost of adding ram is more than the cost of adding a new VM.)
[21:46:08] <GothAlice> However CPU load is a thing, too. That can be somewhat helped with sharding (because sub-sections of a query are run across different machines), and potentially improved by altering your data design to simplify expensive queries.
[21:47:00] <voidDotClass> GothAlice: But sharding is automatic in mongo, right? I.e I'll just need to spin up a VM, start the node, have it ping the master, and the rest will be taken care of automatically?
[21:48:32] <GothAlice> When you add a new node to a sharding cluster, data will re-balance across the cluster to spread things around as evenly as possible.
[21:48:54] <GothAlice> But it does this according to your sharding index—something you configure yourself. What you choose to shard *on* in a given collection will have a huge impact on if sharding is even useful or not.
[21:49:54] <GothAlice> voidDotClass: I spend a lot of my data architecting just meditating on the structures, plans, and repercussions.
[21:50:55] <GothAlice> Wrong key and all of your data may end up on shard A and not spread around. (Or related data gets spread around when it should actually be on the same server for performance reasons, etc., etc.)
[22:06:55] <Synt4x`> I have a bunch of test scores by date, example { 2014/10/05, mean, median, sample, [ {name, score}, {name, score}, {name, score}, ... ] }
[22:06:59] <voidDotClass> It seems cassandra's scaling is more automatic, seems like its just a matter of spinning a VM on it, and that's it.
[22:07:10] <Synt4x`> I want to track data from the individuals as they take tests over time (they take one every day)
[22:07:44] <GothAlice> joannac: Could you confirm for me that if you shard on _id ObjectIds, documents will be distributed effectively round-robin to the shards?
[22:07:53] <Synt4x`> should I store this in the {name, score} pair, so {name, score, percentile, %diff_mean, %diff_median}, {name, score, percentile, %diff_mean, %diff_median}...
[22:08:35] <GothAlice> Synt4x`: Depends on how frequently you need access to those stats (which would be "pre-aggregated" in a sense) vs. how expensive it is to calculate.
[22:09:24] <GothAlice> We try to pre-aggregate data just enough to keep our main user-facing queries (i.e. the ones the user actually has to wait for to see something) below 100ms.
[22:09:35] <Synt4x`> GothAlice: I might make the results available to the public, meaning they would be on a web server where people could look them up all the time, so it could end up getting read quite a bit
[22:10:13] <Synt4x`> so I wasn't sure just how expensive it is to store them vs to calculate them (which takes 2 $groups, and a couple of for loops atm)
[22:10:22] <GothAlice> Then the cost of storage is negligable to the potential cost to calculate repeatedly. Storing would be a good idea in that situation, and it makes your data more useful with less work on the part of the consumers. :)
[22:10:49] <GothAlice> (Whereas for something like a monthly report, there's little point in storing the values.)
[22:10:56] <Synt4x`> ok thanks :), that was the approach I was taking and wanted to make sure it wasn't something super newb :-p
[22:11:31] <GothAlice> MongoDB asks not what you think your data should look like but instead asks, how are you going to *use* the darn stuff?
[22:14:00] <dbb> aha - I see this in the mongo log Failed to authenticate dbb@mongo_fdw_regress with mechanism MONGODB-CR: AuthenticationFailed UserNotFound Could not find user dbb@mongo_fdw_regress
[22:19:23] <GothAlice> MongoDB authentication can be non-obvious. You have system-level users (i.e. a global admin) and database-level users (e.g. application user). Which database you "authenticate against" can be different than the one you wish to actually use. (I.e. to authenticate as the global admin on anything other than the admin table, you need to explicitly say authenticate as admin from the admin db but use 'foo'.)
[22:22:18] <GothAlice> (This is extremely useful, BTW. All of my clients have an "admin" user they can use that is only admin on their database.)
[22:28:56] <dbb> oh hm - apparently I was in the admin database when I made the dbb user, which has role 'root'
[22:29:19] <GothAlice> You may wish to remove that user and add the user to the correct database. :)
[22:29:24] <dbb> when I use the test db, that db.createUser() doesnt work, because there is no role root
[22:29:53] <GothAlice> dbb: Pro tip: create a generic global admin on the admin db. Use that to populate the other db users.
[22:30:23] <GothAlice> Sample MongoDB roles for an admin: https://gist.github.com/amcgregor/c33da0d76350f7018875#file-cluster-sh-L78
[22:36:32] <GothAlice> The number of catgirls being killed by processing a rich, potentially deeply nested document system through flat SQL tables is too darn high.
[22:38:03] <dbb> this will be running clustering analysis on tons of strings, anyway.. so no deeply nested structures here
[22:38:31] <GothAlice> (I nearly have an aneurism every time a client asks me "can I have that data in an Excel file?" ;)
[22:39:32] <dbb> side story - the academics here do SOOO much R.. the planners who get paid lots of money.. ? excel
[22:39:35] <GothAlice> dbb: Other than learning (which is why I do most of the things I do) any particular reason to not store that data in pg itself?
[22:40:08] <dbb> I had my conf call with the clients ... it appears they want to use map/reduce to do the clustering analysis..
[22:41:55] <GothAlice> dbb: Use https://gist.github.com/amcgregor/7bb6f20d2b454753f4f7 to convince them they really want aggregate queries, not map/reduce. (Same query in both… one is clearly more understandable!)
[22:42:28] <dbb> if I get stuck on some horrible thing, I can fall back to fixing it/understanding it/backing it up on pg
[22:42:41] <dbb> this same setup will work if I am talking to a master node in a cluster, across a wire
[22:47:03] <dbb> (I have to sign in to github to see that gist.. that means they track what I read.. )
[22:53:40] <GothAlice> And yes, the Python code at the end is a method of storing aggregate queries within documents.
[22:54:14] <GothAlice> (Our report engine builds reports from documents which describe them, including the aggregate pipeline stages.)
[23:52:57] <mongoooo> hello! i am trying to find a good way to store/check days of the week in mongo. my current solution is a weekday field which is an array of booleans for each weekday m-f (e.g., [true, false, false, false, false] to indicate Monday) -- does anyone have any susggestions for the best way to query this array for a match (e.g., if show two users who selected monday)?
[23:58:09] <GothAlice> We do click analytics at work. Day-of-the-week is somewhat painful due to differences of interperetation over when the week actually starts. ;)