Log file Viewer
#mongodb logs for Wednesday the 24th of February, 2016
[00:02:37] <GothAlice> cerjam: Also note the last comments on the JIRA ticket I linked. You may be running an old RHEL-derived CentOS, that inherited the overall bug.
[00:03:53] <GothAlice> Sharding is the approach MongoDB uses to ensure high availability and data redundancy.
[00:04:23] <GothAlice> Without it, there's only one copy of your data. You might periodically backup, but if it dies, there's downtime while you restore that backup.
[00:05:11] <GothAlice> You mix them, in a RAID 10-like arrangement for both data isolation between shards, and redundancy amongst the sharded replicas.
[00:08:30] <bros> GothAlice: What size server would you put for that kind of DB (in terms of CPU cores/memory), and what kind of replication and/or sharding would you do?
[00:09:13] <GothAlice> bros: https://gist.github.com/amcgregor/4fb7052ce3166e2612ab#how-much-ram-should-i-allocate-to-a-mongodb-server
[00:12:24] <GothAlice> Major faults require disk access. Minor faults are page allocations, which may require page defragmentation if you have transparent huge page support enabled, see the wiki link I just linked. :)
[00:18:52] <GothAlice> I have an uptime of 352 days, MIN=264464523 MAJ=4490680 or ~8.69 MIN/sec and 0.14 MAJ/sec for one city's web presence, vs. your 386/0.01. This would indicate to me that you have a very heavy query load vs. dataset size compared to the city. ;) You would need several nodes, but small ones, preferably on physically distinct hardware.
[00:19:52] <GothAlice> No, what types of queries do you run, and is lag/latency an issue on a majority of your queries? Additionally, does your data have some natural method of grouping like-documents within collections?
[00:20:36] <bros> GothAlice: It's a SaaS platform. Few aggregations based on past sales data, mainly just storing data into subdocuments
[00:21:38] <GothAlice> And yes, indexes are certainly a factor in query performance and load, but your dataset fitting in RAM somewhat offsets that. Certainly look into it, though. I highly recommend http://mongolab.org/dex/ to analyze your real queries to see how they might be improved.
[00:22:36] <bros> What's the first thing I should do? Move it onto a dedicates server? Yes. What do I do about replication/sharding?
[00:25:41] <GothAlice> Optimization without measurement is by definition premature. Yes, however, DBs should be isolated from other services.
[00:26:31] <GothAlice> Dex uses the profiler results to automatically catalog the most impactful queries to suggest indexes automatically.
[00:29:25] <GothAlice> bros: Choosing sharding has an impact on other optimizations, such as the use of "covered queries" (mentioned in that documentation). If your data is write-heavy (insert/update), sharding with a "sharding key" chosen to distribute those writes amongst nodes is an effective strategy. If it's find-heavy and a little replication lag is acceptable, one can reduce network traffic on the primary by querying secondaries in a replica set.
[00:31:26] <GothAlice> bros: Well, in ordinary operation I believe it watches and periodically emits what it thinks might be helpful indexes. It's a continuous process of refinement. Run as long as you need, during a period of time of typical usage, until you have indexes optimizing the queries noted.
[00:33:27] <GothAlice> cheeser: I liked Denver. I liked Boulder and Crested Butte better, though. Better biking. ;P
[00:35:25] <GothAlice> bros: Hmm. An important note: beware that creating indexes without specifying background creation will halt all other operations. So… be careful of that. (And background indexes can take a very, very long time to build, depending on load.)
[00:35:49] <bros> GothAlice: I'm running slow queries and dex isn't picking anything up. Yes, I set profiling to 1.
[00:43:15] <GothAlice> Also, what did you set "slowMs" to? It needs to be below the threshold for the queries you're finding slow.
[00:57:41] <bros> GothAlice: https://gist.github.com/brandonros/aee2716691e8d6b24573 I made these indexes a while ago. See anything that looks strange?
[00:58:10] <kalx> GothAlice: Sorry for the delay, got pulled into something else. Appreciate the help, don't want to take up too much of your time though so maybe you can just peek at the explain and see if anything stands out. (I'm fairly new to mongo but not DBs in general)
[01:00:04] <Doyle> Hey. I'm trying start up 3.2 with this config, but the service fails to start without any useful logs. http://pastebin.com/JeByEev8
[01:05:09] <Doyle> oh, better output. Expected boolean switch but found string: /mongodb/indexes for option: storage.wiredTiger.engineConfig.directoryForIndexes
[01:05:32] <kalx> GothAlice: Essentially, an index allows access to the records we're targeting already pre-sorted. However, in order to find a record with the proper "tags", a scan must occur. These is just bitfield comparisons so should still be pretty inexpensive.
[01:07:25] <bros> GothAlice: db.orders.stats(): totalIndexSize: 108054016 Is this bad? It seems bad.
[01:08:48] <bros> GothAlice: so it isn't huge transparent pages, it isn't slow queries, it isn't indexes not in RAM
[01:10:43] <bros> Doyle: GothAlice and I are trying to figure out why I have 308 page faults a second or something.
[01:10:47] <kalx> GothAlice: I tried an array approach too -> array of the tagids, and then the nasty looking: "tags: { $not: {$elemMatch: {$nin: some_input_tag_array}}}. Same result. I've also tried eliminating any fs by using a ramdisk, same result (and presumably the entire collection is in ram anyhow)
[01:12:12] <bros> Doyle: trying to figure out what kind of hardware I should roll out. I've got my app servers and database intertwined at the moment
[01:12:50] <kalx> That case -> that needs to check if the query input array contains all the tags that the tags field array does
[01:15:30] <Doyle> bros, ideally the working set fits in ram, indexes and data. Next best is indexes fitting in ram. at this point faults will be normal
[01:16:23] <bros> Doyle: I've got 8GB memory and 4GB swap. Mongo is eating up 12%, and my entire db is about 2-3GB.
[01:20:15] <GothAlice> So the same recommendation from earlier: three nodes, it's okay if they're VMs, on physically discrete underlying hardware (to isolate host failures to impacting only one node).
[01:23:14] <GothAlice> See if utilization is still an issue, and clearly identify the cause. The solution depends on the actual problem.
[01:23:46] <GothAlice> Still: isolation. There's no real way of knowing if any of the ~30 other things on there were impacting it.
[01:25:19] <Doyle> Exactly. You can run multiple arbiters on a single node also, just on diff ports to safe cash
[01:27:48] <Doyle> 1. Get mongo running (tune your ulimits, disable THP, adjust TCP keepalive, adjust zone reclaim, and set RA to 32), do a mongodump, then import to your new host
[01:28:13] <kalx> @GothAlice: Anyhow, thanks for taking a peek. I will investigate a little bit more but I may just throw this out in favor of a different solution. Mongo doesn't seem to be benefiting me much for this use-case. Haven't been able to identify anything I'm doing wrong (but still.. there must be, I can't imagine Mongo's performance being that bad...)
[01:29:17] <Doyle> if you want to go l33t, cut a separate disk for your journal (& indexes if using WT)
[01:45:09] <Doyle> I'm about to being playing with WT... So psyched. I'm really hoping to see the kind of perf benefits it's being hailed for
[02:40:44] <nyexpress> I have records collected from every minute, stored with a unix timestamp field (not mongodate), how can I retrieve the average value from values in the record on an hourly basis?
[02:52:59] <GothAlice> nyexpress: Aggregate query, $group stage with an _id: {$mod: ["$dateFieldName", {$literal: 60*60}]}
[02:54:34] <nyexpress> {"date":linux_timestamp, "cpuUsage": 10} <-- I have these records for each minute of every day
[02:57:43] <GothAlice> $subtract, rather. That should do; the _id field in each per-hour average document in the aggregate result will be "snapped" to the hour it belongs to.
[02:58:13] <nyexpress> should it be $avg instead of $sub to get the average value for the 60 values in the hour?
[02:58:56] <GothAlice> $group will average all of the documents that end up having the same _id, i.e. hour logged, as a unix timestamp.
[02:59:57] <GothAlice> So, the _id should be that $sub thing, and yes, you'll need to tell it which fields to average, and where to save them as part of the $group project.
[03:00:05] <GothAlice> https://docs.mongodb.org/manual/reference/operator/aggregation/group/#pipe._S_group
[03:02:24] <GothAlice> Right now, you want an hourly sample size, but have individual per-minute events. This means MongoDB needs to perform a lot of work each time you want to pull a given time range, scaled to a large multiple of the number of hours covered.
[03:03:20] <nyexpress> yeah I was planning on reducing the set even further if the time range is larger than a month, year, etc
[03:03:38] <GothAlice> Using pre-aggregation, you can save a "view" that automatically updates its averages as it goes. So, you might keep a capped collection of limited size around storing the last 24 hours of individual events, but when .insert()'ing the log entry, also .update() a collection for the view for the scaled out timeframe with upsert=True.
[03:04:14] <GothAlice> https://gist.github.com/amcgregor/1ca13e5a74b2ac318017#file-eek-py-L30 < like this in Python
[03:05:03] <GothAlice> https://gist.github.com/amcgregor/2032211 < this is what my per-minute logging records can grow to. :3
[03:05:56] <GothAlice> Though due to WiredTiger compression, key shortening isn't needed any more and it's much more readable. XD
[03:07:29] <GothAlice> There was a great writeup here: http://www.devsmash.com/blog/mongodb-ad-hoc-analytics-aggregation-framework
[03:08:41] <Doyle> http://webcache.googleusercontent.com/search?q=cache:BTrp0K_TZcEJ:devsmash.com/blog/mongodb-ad-hoc-analytics-aggregation-framework+&cd=1&hl=en&ct=clnk&gl=ca&client=ubuntu
[03:09:14] <GothAlice> Oh, Google, ignoring robots.txt. XD All the pretty charts are missing, though. T_T
[03:25:07] <Doyle> GothAlice, in your mongo faq, you mention that the thread attempting the read gets suspended during page fault. At what point are faults excessive? 1000s per second?
[03:25:54] <GothAlice> Depends on the hardware; when you start getting backlog it's bad. I.e. time spent in io-wait.
[03:28:02] <GothAlice> Frozen EBS volumes, cross-zone failure (despite SLA with guarantees against cascade failures), and snapshots locked restoring.
[03:28:06] <Doyle> Their regions are just collections of acquired DC's. It's not surprising they have some issues at times
[03:28:51] <Doyle> In cloud environments, what's your approach for getting good performance from your instances? striped LVM?
[03:28:57] <GothAlice> Quite the nightmare. I had to restore data at the time by reverse engineering the InnoDB on-disk format. ¬_¬ Took the 36 hours prior to wisdom tooth extraction. The reverse engineering without sleep was the more painful of the two. XP
[03:30:31] <GothAlice> It stores certain critical bits of information in a shared pool, a directory up.
[03:30:51] <GothAlice> Which were in a locked EBS volume (vs. the client home folders, which were accessible),
[03:34:45] <GothAlice> On the performance front, for MongoDB I also shard and replicate, with each node on distinct hardware (mostly… I have a complex VM setup with three physical servers… but each primary is on one box) with iSCSI RAID-ish. (Drobo 8-something-i.) One RAID per VM host. Durable to one server failure, or a lot of drives failing. XP
[03:37:35] <GothAlice> nils_: Paid for itself in ~7 months at initial purchase vs. cloud hosting this stupid amount of data. Compose.io would cost me just over half a million dollars a month for this dataset. :/
[03:38:33] <nils_> especially if you're already dealing with devs that want to throw hardware at any problem, if they can do it at the click of a button...
[03:39:47] <nils_> although I remember quite a few cases where I could have probably spent a fortune on a proprietary solution instead of haggling over components for something homegrown
[03:39:47] <Doyle> optimize everything!!! Start with the code before looking to scale hardware. See if your aggregates can do a larger reduction earlier in the pipeline...
[03:40:29] <Doyle> What I love is that a Macbook will always out-perform a cloud instance, and that everyone asks about it
[03:41:39] <GothAlice> Doyle: There's never been more than 15 second replication lag on that cluster, and that's only like, one operation lagging that far intermittently. (Never did track that down either…) It's almost purely inserts, with some chatter over a few capped collections (also only ever inserts). Very unusual load. ;P
[03:42:44] <GothAlice> I've only got one thing streaming on my network at the moment, and IRC chatter is light. :P
[03:43:42] <GothAlice> When I Duckduckgo things, though, there are larger bursts of traffic. Wikipedia browsing is deadly, sometimes rising to a few thousand/second.
[03:54:21] <cheeser> i'm building a Java FileSystemProvider to expose that as a FileSystem in the JVM.
[03:56:02] <cheeser> GothAlice: funny. my next step was to expose it all via REST for reading and writing :D
[03:56:13] <GothAlice> (All content is tagged, with each sub-directory containing all possible valid tags as sub-directories, "files" matching those tags also contained therein. Recursively forever.
[04:01:20] <GothAlice> The reason for this: finding something turns into a z-shell short-hand tab completion. :3 "open mo/da/ag/tu/Manu<tab>" → "open mongodb/database/aggregate/tutorials/MongoDB Tutorials - MongoDB\ Manual.url"
[05:18:09] <kalx> arussel: Not sure there's a convenient way to check that besides being a bit more explicit for each field, checkout $exists and $or.
[05:48:29] <arussel> I know how to check for a field, I was just hoping not to recursively have to test for each field
[05:49:09] <kalx> it wouldn't really be recursive, you just mean avoiding a long $exists:false OR $exists:false OR blabla chain I assume?
[05:50:45] <arussel> oops,forgot to say that I don't know the documents fiels, so I have to use the shell and iterate over every fields
[05:56:14] <arussel> {$type:6} seems to still work on my mongo version, what are you supposed to use instead considering it is deprecated ?
[08:13:35] <Sagar> hello i am trying to run this software https://github.com/mrvautin/adminMongo . as per config i have updated app.json and set the ip to my server's ip but still the adminMongo is accessible via localhost only. please help
[08:51:45] <kurushiyama> commenting out the "listen" line in the config file triggers the default behavior of listening to all interfaces.
[10:13:48] <dunk> Hi. I was wondering - I know that you can use a Postgres Foreign Data Wrapper to pull mongo data into postgres, but can you do the reverse? (without writing a bunch of custom application code - I mean like a true foreign data wrapper)
[10:21:45] <dunk> I'd rather not write the syncing in application code. It'll be fragile, and likely wrong.
[10:39:16] <dunk> kurushiyama: sounds like I'd be solving some nasty distributed systems problems myself right there
[11:20:59] <Sepho> I'm creating a Schema for events. Each event is only visible by user who has created the event. I'm thinking about to create something like "createdBy" field, which stores de current user _id... Is this the best approach? In that case, what value type should I use on "createdBy"? A Number? Thanks in advance!
[11:38:28] <kurushiyama> Sepho: You have your use cases. From these use cases questions are derived. Based on these questions, you should model your data.
[11:39:07] <kurushiyama> Sepho: And the aim is to get your questions answered in the most efficient way.
[13:51:04] <Sagar> hello, we are running mongodb on our dedicated server, it's getting slow in getting queries, can anyone help please?
[13:53:09] <Sagar> isn't there any tuner for the configuration file, we just saw that our mongodb.log was like 224MB
[14:30:22] <x4w3> Hello to everybody, please, could someone help me with this query? <db.devices.update({$set: {disableAccelsens:true}}, {multi:true})> it doenst change all collection, i want to add new column to devices collection...tk in adv
[14:34:03] <x4w3> it run now db.devices.update({},{$set: {disableAccelsens:true}}, {multi:true}) --> tk cheeser
[15:56:34] <Ben_1> sami: I want to update entries but if there are no entries to update I want to insert it.
[16:00:35] <saml> what are your entries? trying to see when it's reasonable to update entries.. but if nothing matches just create one
[16:18:33] <JustMozzy> hello everyone. I have a script that aggregates documents with duplicate values. when I loop over the cursor, I do a remove but the documents are not deleted. no error messages either. anyone an idea?
[16:26:10] <StephenLynx> your usage is weird, but I can't see anything actually wrong on the tools I am used to.
[16:27:50] <StephenLynx> you could use an unwind and a second group on that aggregation to get all the ids to be removed at once, though
[16:32:58] <JustMozzy> the print gives me the correct ID value. When I use the remove statement in the shell it actually removes the document
[16:34:28] <Ben_1> If I insert something to mongoDB with the async driver and maybe there is a duplicate _id, will it stop after the error or just skip that duplicated entry?
[16:38:29] <Ben_1> if it just skip that duplicate maybe there should be a list of throwables in the callback because there could be several keys duplicated
[16:44:03] <JustMozzy> StephenLynx: Your version also left the documents untouched. Nothing was printed to the console though
[16:48:01] <JustMozzy> when I need the allowDiskUse, what comes first, the callback or the option? I could not find it in the docs
[17:20:08] <Ben_1> cheeser: it seems that I got only one error and the whole operation is canceled, I am that right? would it not be better to insert all entries and track all errors
[17:34:58] <Ben_1> cheeser: unordered means it doesn't matter which entry is inserted after another, I am right?
[18:04:37] <JustMozzy> this script http://pastebin.ca/3380969 I am getting a lot of results, but the remove simply does not remove. WriteReulst has always nRemoved: 0, but when I type the remove manually in shell, it does remove
[18:08:22] <JustMozzy> Ben_1: but they clearly do, when I type the db.contents.remove(Object("sameidasincursor"); it does remove the document
[18:08:34] <Ben_1> bzw. you are using print("Removing " + todelete.valueOf()); but in var res you don't use valueOf, why?
[20:07:07] <TechIsCool> Hey everyone got a question about latency in Mogodb. Inserts are taking between 10-30s each right now on one of my hosts. I have no clue how mongodb works other than a few basic commands. Is there a way to figure out what happening?
[20:14:12] <kurushiyama> Let us come back to "one of my hosts". Maybe you should describe your setup a bit.
[20:16:34] <Doyle> If you dropped a collection that resided on the server being slow, it may be taking chunks, but that shouldn't impede operations
[20:17:01] <kurushiyama> Huh? In a replication, writes only go to the primary. So when writes are done when server A is primary, everything is fine, but when writes go to server B, they are slow?
[20:22:23] <TechIsCool> So it could be possible that the Primary failed over to another replica and then all servers are writing to the wrong host. Is that what you guys are saying?
[20:25:06] <kurushiyama> .Net – I am far gone and out. *And having a Jesus and Mary Chain Song in his head now*
[20:32:28] <dypsilon> Hie everyone, I'm thinking about using full text search capabilities of mongodb and have a question: how does it handle rich text content as in text with html tags?
[20:33:17] <kurushiyama> TechIsCool: just to make sure (I honestly dont know about the .Net driver), but the URI string should read "mongodb://db12,db13,db14/?replicaSet=set01"
[20:35:23] <kurushiyama> TechIsCool: Just a wild guess, but I think this is what happens: The URI string as provided by you does not inform the driver that he is talking to a replica set. So he tries to write to the hosts in order of their appearance. db12 refuses to take the write, as it is not primary. So the next one is tried.
[20:40:08] <kurushiyama> Ok, just for testing, I'd have the primary step down to make sure it is not a hardware problem. Next, check your SCM. "We haven't changed anything!" is one of the most common lies in IT industry.
[20:42:44] <kurushiyama> TechIsCool: connect to the primary and issue rs.stepDown(). We just want to make sure the primary changes – we do not care too much to which of the other nodes for now.
[20:44:00] <kurushiyama> TechIsCool: And dont forget the SCM. I cant count the hours I ran after problems caused by "just a minor change in some unimportant part of the code".
[20:51:22] <kurushiyama> TechIsCool: Well. I assume you checked the usual suspects (load, mem shortage, IOWait)?
[21:08:23] <kurushiyama> Ok. Since we are doing this with one-and-a-half hands tied behind our back, I can only guess.
[21:09:26] <kurushiyama> Looking into my christal ball, murmuring some gibberish, I see collection level locking might be the problem.
[21:10:08] <kurushiyama> Doyle: also keep in mind we are talking of shared storage, if I get that right?
[21:11:52] <Doyle> Yea, I've seen the mutex/concurrency issues with ext4 and the mongodb files before
[21:12:36] <Doyle> Between 3.2's file breakout, document level locking, and xfs, the perf is miles ahead of 2.6 on ext4
[21:12:44] <kurushiyama> TechIsCool: can you check wether the partition mongodb writes on has "noatime" in fstab?
[21:15:59] <kurushiyama> TechIsCool: ok, we have multiple problems here. a) Shared storage. More often than not, this is not ideal.
[21:16:19] <zivester> is it possible to use mongodump on two db's and to mongorestore all collections into a single DB ?
[21:16:42] <Doyle> You can go nuts and throw a Pure Storage all flash array at a set of mongodb servers, but that's a $100K+ game
[21:16:47] <kurushiyama> TechIsCool: b) Maybe not the optimal fs. I agree with Doyle that xfs is the FS to use
[21:17:52] <kurushiyama> c) atime. Basically what happens, every time the data files are _accessed_ the FS metadata is updated.
[21:19:51] <Doyle> Indexes won't be rebuilt... Shared storage... Susceptible to noisy neighbour issues.
[21:20:17] <kurushiyama> TechIsCool: attach additional storage to one of the machines, use XFS, set noatime in fstab, remove the contents of dbpath, mount the new FS to dbpath, update mongod to 3.0.9 and start mongod
[21:21:47] <kurushiyama> TechIsCool: let the devs approve https://docs.mongodb.org/ecosystem/drivers/driver-compatibility-reference/#c-net-driver-compatibility first
[21:22:21] <kurushiyama> TechIsCool: when the instance you changed has synced, repeat the process with the next secondary.
[21:24:30] <kurushiyama> TechIsCool: well, you need to change a few things anyway, so you can use the optimal FS, too
[21:26:22] <TechIsCool> So basically every other time this happens we drop the collection and everything is resolved
[21:26:29] <kurushiyama> TechIsCool: basically, it doesn't hurt then. If the database is small, it wont hurt either to keep it.
[21:27:08] <kurushiyama> TechIsCool: I wouldnt, since it is not necessary and the syncing of 62Gb should not take long.
[21:27:47] <TechIsCool> since we dropped the 29million records why has mongo not cleaned up the db size
[21:29:31] <kurushiyama> TechIsCool: you might want to disable WT's compression on the data files and/or journal during the update.
[21:33:02] <kurushiyama> TechIsCool: Ah, you need to shutdown the instances manually before doing the yum update, since mongod is not shut down preupdate or preuninstall.
[21:35:32] <TechIsCool> I totally want to upgrade to the newer version of mongo it just works better
[21:38:03] <kurushiyama> Bosses. Sometimes I ask myself why they pay people good money for their technical expertise when in the end they seem to know most things, if not everything better.
[21:38:58] <dypsilon> Hi everyone, I'm thinking about using full text search capabilities of mongodb and have a question: how does it handle rich text content as in text with html tags?
[21:42:23] <Doyle> TechIsCool, see the notes. https://docs.mongodb.org/v3.0/administration/production-notes/
[21:46:50] <kurushiyama> Doyle: I am working on a PR for the RPMs to include THP, ulimits and zone reclaim
[21:55:13] <kurushiyama> TechIsCool: nvm, you tried. In Germany, there is the notion of "Beratungsresistenz" which roughly translates to resistance against advice
[22:02:49] <Doyle> That'd be great, kurushiyama. It's annoying to script it all for each different OS (if you change os's)
[22:17:41] <dypsilon> kurushiy_, thank you for this simple test, looks good, but I have all kinds of html in there with classnames and so on. I think, I just test how it looks tomorrow.
[22:21:34] <kurushiyama> dypsilon: What this test was supposed to show you is that html tags wont be ignored
[22:22:45] <dypsilon> kurushiyama, oh, just missed the last line, almost midnight here... well that's the answer I need. Seems like I'll have to trim them before saving to the database.
[22:23:39] <dypsilon> kurushiyama, the problem is: I have all kinds of html with classnames and they will surely mess up the results.
[22:23:53] <kurushiyama> dypsilon: most dedicated search engines have the option to strip html before indexing
[22:25:29] <dypsilon> kurushiyama, oh, that's what you mean. Not sure I want to add all the admin overhead that comes with elasticsearch.
[22:26:14] <kurushiyama> dypsilon: Well, that is obviously your call. However, I do not find the overhead to be very small.
[22:27:56] <dypsilon> well, either way I'll try and go with mongodb, if html stripping is my only problem, everything is fine
[22:33:45] <kurushiyama> bros: you sure can. As a whole or for single fields. http://pastebin.com/A4BiJtfC
[22:34:53] <bros> kurushiyama: If I create indexes for field "status" and field "account_id", should I also create one for { status: 1, account_id: 1 } (not subdocument related)?
[23:26:53] <jiffe> if I flatten a maching with mongoc running (in a 3 mongoc node setup), it should resync from one of the other nodes right?
[23:26:59] <zeioth_> http://stackoverflow.com/questions/35615176/c-sharp-mongodb-driver-2-0-retrieving-filter-results-as-custom-model
[23:33:44] <jiffe> ah ok, so it may not resync itself but I can just rsync the data over from one of the other 3
[23:46:08] <bros> I just dropped all of my indexes. How can I have dex tell me which I need to build?