Log file Viewer
#mongodb logs for Thursday the 13th of November, 2014
[00:00:57] <Boomtime> I cannot think of why you'd want to store a random number (in the document) for the purposes of selecting a random document
[00:01:39] <Boomtime> you want the documnents to form an perfectly flat distribution (no randomness) then select an index in that distribution
[00:03:38] <Boomtime> for example: a die has the 6 faces marked in a perfectly flat distribution (1,2,3,4,5,6, no randomness), the face is then selected at random from the set
[00:03:39] <GothAlice> Generate a pseudo-random number with a recorded starting seed, query the database for records with their recorded random number equal to or greater than that one. If multiple records returned, pick one using another step in the PRNG modulo the number of choices. You can then replay the same (random) choice using the original dataset and the starting seed, which would be actually randomly chosen.
[00:06:03] <GothAlice> Yes; which results in positive bias in returned results, which "slide" up the gap to the next approximate match.
[00:06:06] <Boomtime> you bias those documents which have the earlier luck of having a largely chunk of the number space
[00:06:33] <Boomtime> you should try your method with only a few documents and you'll see how disatrous it is
[00:09:19] <GothAlice> It is if it describes a balanced binary tree; the random choice, which might not exist, is between two evenly balanced sides.
[00:10:20] <Boomtime> you have a single number space 0-1, all documents have a "random" number that puts them on that line
[00:10:48] <Boomtime> you then select a "random" point on that line and slide up to hit the first document
[00:11:18] <Boomtime> you have bias, documents on the number line with slightly larger spaces than others get selected more frequently
[00:11:19] <GothAlice> Pseudo-random. ¬_¬ The first insert gets 0.5 — it divides the available space. Now there are two spaces. The next insert gets to divide either the left- or right-hand spaces—this is chosen randomly. Etc.
[00:13:02] <GothAlice> The first one isn't. The next one is. The one after that isn't (it takes the opposite space.) The one after that is (between four available equal spaces).
[00:13:55] <GothAlice> But the choice is between [0.25, 0.75] on the second insert; not the whole 0-1 range.
[00:16:12] <GothAlice> This evenly distributes the range… thus the "slide up gap" between each will be the same, excluding any orphan leaves in the tree, which will have a bias. However, for this application, such (manageable and minor) bias is acceptable.
[00:19:31] <Boomtime> curious system, so it still has bias but you are constantly in a process of reducing it
[00:20:09] <Boomtime> it seems to require significant maintenance.. how do you know what ranges to split on insert?
[00:21:16] <Boomtime> btw, there is a relatively simple method of selecting documents at random, requiring similar overhead as yours, and has more precision than you'd ever need
[00:21:50] <GothAlice> I have a tiny service generating the next division. It's a stateful generator (capable of resuming) — it's done algorithmically, so I don't really need to track the members of the sets, only the ever reducing range of the "current" set, and the "current" depth in the tree.
[00:23:08] <Boomtime> also, the bias condition is by far more common than the relatively rare in-balance/unbiased arrangements
[00:23:46] <Boomtime> but it is at least limited to 2x selectivity, so I suppose you at least knows the bounds of the bias
[00:24:05] <GothAlice> Indeed, but the level of imbalance graphed against number of records will resemble a sawtooth; simply maintain records at the 0 points.
[00:25:20] <GothAlice> (And yes; bias is purely calculable for any given number of records in the tree.)
[00:25:42] <Boomtime> since you have insert only, i expect you have a much simpler method of selecting a doc at random: just have a single 64bit int in each doc, a simple counter
[00:26:16] <Boomtime> first doc =1, second = 2... now pick a random number between 1 to count() inclusive
[00:26:49] <Boomtime> flat distribution at all times, and the random number is completely under your control at all times for replayability
[00:28:54] <GothAlice> The data is already a tree… I was trying to kill two birds with one stone. (The exact range of any given space in the tree or point of division, indexed linearly, is calculable as well…)
[00:29:44] <Boomtime> if your goal isn't to select a random document, then you have some other purpose in mind
[00:29:55] <GothAlice> I'm trying to both store the tree in a very queryable way, but also be able to find random points of entry into those trees.
[00:31:56] <GothAlice> Popular use has them as the storage mechanism for level data in 3D games, as it solves the "what can I see from point x/y/z" problem.
[00:37:20] <GothAlice> Streemo: At work we learned the hard way to use a graph database to store graphs… don't try to roll something under a different model.
[00:53:32] <bramgg> I can not for the life of me add anything but Strings to MongoDB documents in Java. Can anyone help? http://pastebin.com/80nNVLMm
[01:13:41] <bramgg> If I go AFK but someone has an answer, I've posted the question on StackOverflow: https://stackoverflow.com/questions/26899694/create-mongodb-document-with-non-string-values-in-java
[01:27:24] <leotr> hello. I have big collection (about 70Gb). I get poor performance when i scan it. I scan it in sorted order and i created index for that. I even try to hint it but that doesn't help. Looks like it's stuck in some place and that's all...
[02:31:30] <jchia> Hi, why is it recommended to run mongod with numactl --interleave=all on a NUMA machine? How does it help performance? I think it's going to allocate memory on multiple nodes without controlling the cores that the kernel schedules mongod threads to. Couldn't the kernel end up scheduling some mongod thread on a core that has slow access to some memory? Doesn't it make more sense to pin both the memory and the core to the same one for better affinity?
[04:51:03] <Lonesoldier728> hey keep getting an error error MongoError: Unsupported projection option: $meta ... anyone know why cant find too many complaints on google
[04:58:48] <Boomtime> if you have a full text index, and you are using 2.6+ then text search is enabled by default
[05:12:04] <Lonesoldier728> Boomtime just got version 2.6 on my terminal but I am assuming that does not affect what I have on my servers
[05:12:13] <Lonesoldier728> http://blog.mongolab.com/2013/02/experiment-with-mongodb-2-4-on-mongolab/
[05:12:31] <Lonesoldier728> that is the add-on for heroku I am using for mongodb which apparently has textsearch enabled
[05:13:15] <Boomtime> client version and server version are independant, though old clients may or may not work with new servers
[05:20:38] <Lonesoldier728> On November 19, 2014, all free Sandboxes databases running MongoDB 2.4 will be upgraded to MongoDB 2.6
[05:26:14] <irc_smirk> referencing this page https://thomashunter.name/blog/getting-the-php-mongodb-driver-installed-with-mamp-on-os-x/
[08:18:40] <ldksofdjk> Ive been learning, mongodb and am trying to get my head around how the expires ttl works. Ive read stackoverflow posts and have added code to make it work (from my understanding of how it works). Am pretty stumped as to why it does not work. Can any one point me in the direction of why my data is not deleting - http://pastebin.com/DGbbUeq0
[08:37:50] <nonuby> would a "text" index be right for field firstName and lastName where a user might search either/or and only partial so query would be {$or [ { "firstname": /nonub/ }, { "lastName": /nonub/ }]} at the moment it takes several seconds on 100k docs
[08:52:48] <ldksofdjk> Ive been learning, mongodb and am trying to get my head around how the expires ttl works. Ive read stackoverflow posts and have added code to make it work (from my understanding of how it works). Am pretty stumped as to why it does not work. Can any one point me in the direction of why my data is not deleting - http://pastebin.com/DGbbUeq0'
[09:47:03] <jchia> Hi, why is it recommended to run mongod with numactl --interleave=all on a NUMA machine? How does it help performance? I think it's going to allocate memory on multiple nodes without controlling the cores that the kernel schedules mongod threads to. Couldn't the kernel end up scheduling some mongod thread on a core that has slow access to some memory? Doesn't it make more sense to pin both the memory and the core to the same one for better affinity?
[09:55:48] <braz> @jchia MongoDB places an atypical workload on a system, where it as a single large multithread process consumes all the systems memory. If NUMA isn't disabled then strange performance problems can happen. This is the case when you are running a single mongod on the machine. NUMA rewards threads who access their own memory more frequently, while punishing threads who access each others memory. MongoDB, and any other multithreaded database, doesn't work well with
[09:57:13] <jchia> braz: so, why not just pin the CPU and memory to one node so that the CPUs that mongod gets scheduled on always access the most local memory?
[10:05:51] <braz> jchia: for multiple mongods on a host pinning with the likes of cgroups is a strategy
[10:06:22] <braz> jchia: however if you are running only a single mongod you should use the all option
[10:58:25] <kas84> does anybody know why object size in a capped collection is smaller than a regular one?
[11:10:59] <TCMSLP> Hi all, we've disabled anonymous login (auth=true) which has prevented anonymous users from querying anything useful.
[11:11:33] <TCMSLP> However, is there a way to actively deny connections to mongodb without auth, besides firewall rules?
[13:47:39] <slvrbckt> hi all, im having a very difficult time getting a shard key to take with my new sharded cluster. seems anything i choose I get a (rather obtuse) error message of " Uniqueness can't be maintained unless shard key is a prefix "
[13:48:08] <slvrbckt> i've tried several combinations. i have a few different "unique index" fields, and always one of them comes up in the error message
[13:48:48] <slvrbckt> and ive done a bunch of searching, but nothing really clarifies what im doing wrong
[13:51:39] <slvrbckt> does anyone know what on earth "unless shard key is a prefix" means? a prefix? of what?
[14:17:17] <omid8bimo> hello, i need help, i have 2 server in a datacenter in europe. my dataset is around 570 GB; i just installed a node in asia as a secondary-only node. it starts replicating when configured in replicaSet but when its near the end, like 500 GB already synced, im starting to get errors like these:
[14:17:23] <omid8bimo> [rsBackgroundSync] replSet not trying to sync from kookoja-db1:27017, it is vetoed for 477 more second
[14:17:30] <omid8bimo> conn13] assertion 13436 not master or secondary; cannot currently read from this replSet member ns:config.settings query:{}
[14:18:47] <omid8bimo> can somebody help me? i tried this approach 2 times; a full clean resync, but boths times failed near the end
[15:26:19] <wblu> Does anyone know if a package for 2.8.0-rc0 will be published to http://downloads-distro.mongodb.org/repo/ubuntu-upstart/?
[15:49:07] <omid8bimo> can somebody tell me how a full resync works? i have a primary and a slave, and adding a new secondary in a different datacenter in another country. but the full resync stops near the end and says its become stale!
[16:19:14] <kali> omid8bimo: it matters very much. it has to cover enough time for the main sync to take place
[16:42:38] <GothAlice> speaker1234: I can be; I'll be walking into another meeting once one of my bosses gets off the phone, though.
[16:43:33] <speaker1234> GothAlice, very quick question. Have potential customer with an old Informix database. They need to upgrade and one of the requirements is simple, non-technologist friendly report generation. Do we have the tools in this area?
[16:45:35] <GothAlice> speaker1234: Alas, right now I have those tools, but they're fairly deep down the chain for extraction and open-sourcing.
[16:47:15] <GothAlice> speaker1234: http://cl.ly/image/0n300z2e3p08 — sample report; can't screenshot real data, though. ;)
[17:16:08] <malev> hello! I'm having a super weird issue. I have a webapp with a mongodb and sometimes gives me a 500 because: [conn21] auth: couldn't find user
[17:16:25] <malev> but some others, just work: authenticate db: mapa76 { authenticate: 1, user: "...
[17:53:58] <braz> @ahawkins only where your journal's on another volume or you are running without journaling (v bad idea) http://docs.mongodb.org/manual/tutorial/backup-with-filesystem-snapshots/#create-backups-on-instances-that-do-not-have-journaling-enabled
[18:08:37] <braz> @ahawkins not sure I understand the context fully, the best place is to add a fuller description in https://groups.google.com/forum/#!forum/mongodb-user
[18:08:52] <GothAlice> ahawkins: Your application may have cached some old (and no longer valid) ObjectIds.
[18:09:09] <GothAlice> ahawkins: Generally a good idea to kick app servers in the pants after restoring data.
[18:11:07] <GothAlice> ahawkins: Also, no, mongodump communicating with a mongod process (rather than directly accessing the datafilee on-disk) should never require a filesystem lock; locking is needed in the aforementioned scenarios only when attempting filesystem-level (instead of database-level) backups.
[18:11:54] <ahawkins> GothAlice: wondering if I need to force a flush of some sort before doing mongodump
[18:13:16] <GothAlice> ahawkins: You should not. Depending on use of http://docs.mongodb.org/manual/reference/program/mongodump/#cmdoption--oplog either records added/modified/deleted during the backup are ignored or the additions/modifications/deletions happening during the time it takes to do the dump will also be recorded.
[18:14:49] <GothAlice> (And by ignored, I mean you'll actually have some undefined behaviour. Modifications may be partially applied, ignored, or successful, deletions may be noticed or not, etc.)
[18:36:07] <omid8bimo> kali: oh ok. so its like when the initial sync begins, it marks the state of the dataset, starts from begining up until that mark, and reads the rest from oplog? is this assumption correct?
[18:36:56] <GothAlice> Then on restore it restores the original records as received, then applies the oplog.
[18:39:07] <GothAlice> Ah, different things, but roughly the same process. Initial replica syncs are handled in pretty much the same way as mongodump --oplog, except the synchronization continues forever afterwords, streaming the oplog from the primary.
[18:42:06] <omid8bimo> GothAlice: ok, so what could be my problem here? i did the initial sync twice and both times, i was near the end of replication when i got the "member become stale" error message
[18:43:57] <kali> omid8bimo: check out you oplog depth (rs.printReplicationInfo()) should give it to you
[18:45:17] <omid8bimo> i thinks with that bandwidth, initial sync is far longer than oplog capacity! am i correct?
[18:46:20] <GothAlice> omid8bimo: In terms of raw bandwidth, that'd take 23.8 (base 1000) or 24.38 (base 1024) hours. It's cutting it way too close. (Since there is extra time overhead beyond just bandwidth transfer.)
[18:48:33] <kali> omid8bimo: at this side of DB and oplog, it start to make sense to consider sharding (but that is probably not something you want to do today)
[18:49:22] <GothAlice> omid8bimo: http://stackoverflow.com/a/13095913 is a SO answer to a slightly different question, but covers some important details such as maintaining equal oplog size amongst replica set members. http://www.briancarpio.com/2012/04/21/mongodb-the-oplog-explain/ for more general information.
[18:50:50] <kali> omid8bimo: if you have a 3-node replica set working on your current datacenter, you're not taking that big a risk
[18:51:22] <GothAlice> omid8bimo: I've got 25 TiB of data in a three-shard with three replicas for each shard. Heavily write-only dataset with narrow scope queries (i.e. I almost never table scan… uuuuugh it's bad when it does) and maintaining a 24-h oplog requires about 96 GiB of space dedicated to it.
[18:51:39] <omid8bimo> kali: its 2 node actually. one primary, one secondary, and i had a arbiter which i removed to replace it with a new secondary
[18:52:39] <kali> omid8bimo: if it is cloud based, you can create two new nodes in the current DC with the right oplog size, make them join and then remove the old ones
[18:53:53] <omid8bimo> kali: no its not cloud based. first question that im faced regarding the oplog size increase procedures, is that should take down the whole databases to do that?
[18:55:36] <GothAlice> kali: Yeah, with data stored over iSCSI across three Drobo arrays, one array per replica, shared between the shards, which is a weird setup.
[18:56:14] <GothAlice> (Lose array A, and the primary of set A goes away and a secondary from each of set B and C. ;)
[18:57:16] <kali> omid8bimo: the bottom line is: you need to have a majority of working nodes in your cluster at all time
[18:57:25] <GothAlice> omid8bimo: You'll need to scale up the oplog before replication can even succeed. :(
[18:57:45] <kali> omid8bimo: so i suggest you reconfigure to make the current remote node an arbiter, to restore some kind of sanity to your RS
[18:57:48] <GothAlice> omid8bimo: Until that's done, I'd add the arbiter back. (I only remove an arbiter after the node to replace it is live.)
[18:58:34] <kali> omid8bimo: then, you'll stop the secondary, and perform the oplog growing procedure on it. during that time, you'll have no redundancy and no secondary. everything will go to your primary
[18:59:26] <kali> omid8bimo: one the secondary is regrown back in the RS, and has catch up, you'll have to stepDown() the current primary. the secondary will become primary and your app should recover quickly
[18:59:56] <kali> omid8bimo: you'll stop your old primary and perform the oplog growing procedure on it, and restart it
[19:00:15] <kali> omid8bimo: then you'll reconfigure to make you arbiter a full data node, and it should resync :)
[19:00:27] <GothAlice> omid8bimo: The above should work nicely considering the active secondary will only have to catch up for the time needed to grow the oplog, which should be far less than 24h, then the old primary will catch up from the old secondary (which would become primary). :)
[19:00:52] <kali> if you don't make mistake, you'll have a few errors when you'll stop the secondary, and a nice bunch of errors when stopping the primary
[19:06:25] <GothAlice> Resizing is somewhat painful, and a little napkin calculation can save you a lot of time later.
[19:07:25] <GothAlice> (I.e. we benchmarked our distributed RPC capped collection to be 8GB to handle one million simultaneous users' activity based on average document sizes, estimates of the number of events for any particular game session and whatnot.)
[19:07:45] <GothAlice> omid8bimo: Yeah; you want the oplogs to be equally sized across the cluster. (Or, as the SO answer mentioned, expect the one with shortest oplog to win.)
[19:07:56] <kali> yep, the two nodes you have already need a resize, and the remote third too, but it should be easier :)
[19:08:31] <rendar> GothAlice: lack of inode limitations? what you mean? that reiserFS implements all inode functions, while other FS don't?
[19:09:20] <GothAlice> rendar: ext* family filesystems have a limit on the number of inodes based on a division of the block device size by block size chosen when initially formatting. (Usually 4KB.)
[19:10:07] <omid8bimo> GothAlice: kali: ok, so steps are that i dont touch the primary; add an arbiter to keep the vote/replicaset balance; bring down secondary, increase oplog, bring it up, step down primary so secondary with new oplog becomes primary, then increase oplog on this node, then add the new secondary with initial sync
[19:10:28] <rendar> GothAlice: ignoring that means that you can easily scale up to very huge files, without caring about block size and those stuff, i guess
[19:10:51] <GothAlice> rendar: And more importantly, never have "disk full" errors if some silly user of yours creates billions of one-byte files.
[19:11:32] <GothAlice> (Getting a disk full error and having df -h show free space is one of the most confusing sysop things I have ever encountered… long ago. ;)
[19:12:24] <rendar> GothAlice: ext* fs have a minimum 4KB block for each files, so a bunch of little files would take a bunch of 4KB blocks, instead in reiserfs a 6bytes file would take only 6bytes (plus some other structure) but you got the point
[19:13:28] <GothAlice> rendar: This is particularly critical since several of my applications make rather extensive use of hard links. Also that's only true if you enable tail packing; there is still a "block size" in Reiser, it's just hidden away for the most part. (Tail packing allows the last block of a file, with free space at the end, to store another sub-block file at the end efficiently. Makes disk recovery darn hard, though.)
[19:14:48] <GothAlice> My favourite call from a cleanroom recovery company ever: "Uhm… I don't know how you did it, but this drive you sent in has six petabytes of data on it, in 14 trillion files. Which ones do you want?" XD
[19:15:15] <omid8bimo> one question though, can i do these with the new secondary in STARTUP2 state? so i dont need to add back the arbiter?
[19:15:47] <GothAlice> omid8bimo: No. I'd nuke the new secondary until such time as the oplogs have been scaled up, adding the arbiter back.
[19:16:10] <GothAlice> rendar: Hourly backups for three years, with unmodified files hard-linked to their prior versions. :3
[19:16:31] <GothAlice> rendarn: (Their recovery process couldn't identify hard links, so it treated each link as a distinct file.)
[19:17:57] <omid8bimo> so i should do all the steps in here? -> http://docs.mongodb.org/manual/tutorial/change-oplog-size/ - such as starting secondary in standalone, saving the oplog etc?
[19:22:38] <kali> omid8bimo: yes, but only after making sure you can stop the secondary (which means your RS must have the primary and an arbiter)
[19:24:25] <omid8bimo> kali: last time i tried to stop secondary i got these errors -> http://paste.debian.net/131644/
[19:26:59] <GothAlice> (If you see messages about "read only" and "majority of nodes" on the primary, you may have a problem. If not, you'd be good to go to scale up that first secondary.)
[19:28:50] <kali> the whole point is that the primary stays primary. for that, it needs to stay in contact with a strict majority of running nodes and arbiters
[19:29:46] <kali> so with an arbiter, when you take the secondary down, as long as the primary "sees" the arbiter, it will stay up. if it disappear, it will step down and become a read only secondary
[19:36:11] <GothAlice> (The arbiter confirms for the primary that it was actually the secondary that lost its connection / went away, and not the primary itself disappearing into a black hole. ;)
[19:37:31] <GothAlice> (To which the primary effectively says, "so what?" and keeps going. Without the arbiter the primary would think it lost its connection and assumes a secondary took over primary duties.)
[19:54:10] <omid8bimo> kali: ok, secondary's oplog increased and it caught up to primary; should i wait for it to populate its oplog?
[19:54:53] <omid8bimo> db.getReplicationInfo() -> usedMB is increasing, should i wait for it to complete something?
[19:56:23] <GothAlice> omid8bimo: Both boxen will now contain the same actual data, and when you downgrade the primary to expand its oplog, it'll be able to catch up from what has been populated of the secondary's oplog so far.
[20:20:33] <omid8bimo> GothAlice: kali: all right, things are working with new oplog, i will start the new secondary for a full resync now. thanks guys
[20:22:03] <GothAlice> Also somewhat impressive that even with just two real data nodes (to start) you can do things like that with only extremely minor interruptions in service. (I.e. the time needed to elect the secondary as primary in the second step.)
[20:23:05] <GothAlice> OFC, you've got degraded network performance for the next 24h while that new secondary populates…
[20:24:19] <arussel> I'm using reactive mongo to connect. I've run a $set command, I was hoping to do {$set: {"a.b": "x"}} with result: {"a": {"b":"x"}} but I end up with {"a.b":"x"}
[20:27:07] <GothAlice> arussel: http://cl.ly/image/293b0t420t1V — with $set, it works correctly for me.
[20:28:59] <GothAlice> arussel: > db.deepsert.update({}, {"c.d": "x"}) -> "can't have . in field names [c.d] at src/mongo/shell/collection.js:155" — not even sure how you're managing to get {"a.b": "x"} at all.
[20:29:29] <GothAlice> John: Have you added /usr/local/bin to your PATH environment variable? Try this: export PATH="$PATH:/usr/local/bin"; mongo --version
[20:29:59] <arussel> GothAlice: sorry, I can set deep fields and it works, but somehow I've run a command that, instead of setting deep field manage to set a "a.b" field. I'm trying to find out which is the command that did that.
[20:29:59] <davis> i am trying to migrate a database from one server to another, I found some .bson files on the older server which appear to be backups. however when I run mongoimport on the new server with thesee files it appears that it needs to take either .csv or .json files.
[20:30:27] <Jonn> 2014-11-13T16:29:06.111-0400 warning: Failed to connect to 127.0.0.1:27017, reason: errno:61 Connection refused 2014-11-13T16:29:06.112-0400 Error: couldn't connect to server 127.0.0.1:27017 (127.0.0.1), connection attempt failed at src/mongo/shell/mongo.js:146
[20:31:06] <GothAlice> Jonn: Hmm; ps aux | grep mongod — anything there? (Or showing up in Activity Monitor?)
[20:31:25] <GothAlice> John: brew info mongodb — also run this, it has some instructions at the end on how to register mongodb as an auto-start service.
[20:31:37] <Jonn> ERROR: dbpath (/data/db) does not exist. Create this directory or give existing directory in --dbpath. See http://dochub.mongodb.org/core/startingandstoppingmongo
[20:32:27] <GothAlice> John: I seriously recommend running "brew info mongoldb" and following those instructions after updating /usr/local/etc/mongod.conf
[20:34:10] <GothAlice> Jonn: If you follow home-brew's instructions here, your data will go into /usr/local/var/mongodb, and permissions should already be correct.
[20:35:57] <GothAlice> Jonn: The key is that the directory will be created automatically with the correct permissions; your user should already own /usr/local. https://gist.github.com/amcgregor/23e0e36f80122223d565 is my /usr/local/etc/mongod.conf, FYI.
[20:37:12] <GothAlice> Jonn: All of that boils down to running "brew info mongodb" and following the instructions given.
[20:37:31] <GothAlice> (These instructions would have been printed out after the initial "brew install mongodb" command, too.)
[20:38:17] <GothAlice> And yes; those instructions require configuration via the mongod.conf file… unless you're particularly happy editing launchd .plist files.
[20:39:19] <GothAlice> Ugh; they restructured it. Caveats are it, but they removed the really useful line.
[20:39:54] <GothAlice> Jonn: cp /usr/local/Cellar/mongodb/*/homebrew.mxcl.mongodb.plist ~/Library/LaunchAgents/
[20:42:49] <GothAlice> Yes. Directory-per-db will allow you to symlink those into your project folders if you want.
[20:42:55] <GothAlice> (Launchd can even be configured to only start mongodb when a process tries to connect to it, but that's Very Advanced Technique™.)
[20:43:54] <GothAlice> Really… you don't want Time Machine backing up your mongod stripes every hour…
[20:45:21] <GothAlice> Jonn: However! My development environment isn't like my "development staging" environment… for that I use https://gist.github.com/amcgregor/c33da0d76350f7018875 (a BASH script of mine) to start up a complete sharded+replicated+authenticated mini-cluster on one machine. I do this because I need to be able to test and confirm correct behaviour of sharding indexes.
[20:46:05] <GothAlice> (And fault tolerance… after starting the script, I'll put my app under load and kill -9 mongod processes. :)
[20:47:24] <GothAlice> cheeser: My entire at-home dataset (25 TiB + random external drives) is backed up to Backblaze. It took three-four months for the initial backup!
[20:49:20] <Ontological> I'm having an issue where, if I try to execute commands via a script, I get an error. I can copy and paste the commands in, just fine. http://pastie.org/private/uvaylz54v4lbvez5t5jzg It yells and says "can't have . in field names [one.two.three]" Any suggestions?
[20:52:59] <GothAlice> Ontological: Works for me. https://gist.github.com/amcgregor/2cec36435d929cdca036
[20:53:55] <Ontological> If you copy what I wrote into an bash script and execute it, it will fail. That's my problem.
[20:56:44] <GothAlice> The problem is actually quite obvious. Replace "mongo" in your script with "cat" and re-run to see.
[20:57:34] <GothAlice> (Obvious primarily in hindsight if you weren't aware that variables are expanded even in string blocks.)
[20:59:01] <Ontological> http://pastie.org/private/7ku9cayoyrfk6xegqldpa Is *not* working for me on 2.6.4.
[21:00:38] <Ontological> Thanks for helping troubleshoot. At least I have an idea that it's most likely not mongo itself, but I'm still really confused
[21:01:32] <GothAlice> "ssh -t git@staging.server.com" — GIT? Also, "ssh -t git@staging.server.com mongo << EOF MONGO" — you can pass commands for SSH to execute…
[21:02:19] <Ontological> I was trying to improve my bash-ness, but you've discovered I've got some limitations :P
[21:03:00] <GothAlice> Ontological: http://cl.ly/image/2F383V3S2q3i — this approach does, however, work. A better approach in this instance would be to store the script on the other side, and just remotely execute it. ssh example.com ~/bin/process.sh
[21:03:27] <Ontological> But then I don't learn where I'm failing and the limitation of bash/ssh/mongo that I've discovered
[21:03:52] <GothAlice> (Nesting block strings is Danger Zone™… since you'll have to escape the escaping of the escaped $. ;)
[21:05:59] <GothAlice> Ontological: You can even create a secure user that can only ever execute that one command, as an example.
[21:06:32] <Ontological> I'll just have to disconnect and then re-connect to the server, which is less efficient, but I see your point
[21:07:21] <GothAlice> Ontological: You can get creative with interactive SSH sessions wrapped in screen—you can send commands for screen to emit into the remote terminal. (This will also result in madness, but it does technically work.)
[21:09:39] <Ontological> Okay, I'm taking your advice about not double-nesting, but I *really* appreciate you helping me understand the fix, because it helps me gain a better understanding of what's going on and how I failed.
[21:10:29] <GothAlice> \\\$ \\\\$ \\\\$ \\\\\\$ \\\\$ \\\\\\$ — at a glance, which of those is valid? ;^)
[21:11:11] <Ontological> Well, I know that the first one is valid in my current situation, but again, that depends on the situation
[21:21:57] <Mmike> If I have a mongodb server with bunch of data on it, may I just do rs.initiate() to start building replicaset of it?
[21:22:58] <GothAlice> Mmike: http://docs.mongodb.org/manual/tutorial/convert-standalone-to-replica-set/
[22:14:39] <GothAlice> Ontological: If you're trying to improve your BASH-fu, https://gist.github.com/amcgregor/c33da0d76350f7018875 might be worth a gander. It does some nifty things. (Note line 62…)
[22:27:30] <BlakeRG> Anyone know why i am getting this error with yum install? http://pastebin.com/raw.php?i=WBM2NP1B
[22:29:57] <Ontological> I'm not on RHEL, but, does 10gen have a repo for RHEL and have you tried that?
[22:38:01] <BlakeRG> Ontological: it's already installed, i believe it's trying to upgrade and giving these errors
[22:46:06] <bpap> is there a way to force MMS monitoring agent to poll db stats once per minute? right now it polls once every 3 minutes, and I don't see any option to set that.