Log file Viewer
#mongodb logs for Friday the 24th of June, 2016
[00:41:09] <f00sh> Quick question - Anyone know how I can use .updateMany({ "Derp" : data.Hello}, data) ? data.Hello would be a different value every iteration
[00:52:41] <Boomtime> @f00sh: you mean you want to update many documents but based on a different filter for each one?
[08:06:38] <Industrial> Say I have a collection with a .location key and a .deviceID key, how do I create an aggregation that picks out all the last locations?
[08:07:30] <Industrial> db.devices.find({}).forEach(function(v) { db.incoming.find({ deviceID: v.deviceID, location: { $exists: 1 } }).sort({ activityID: -1 }).limit(1).forEach(function(w) { db.locations.insert(w); }); });
[08:08:29] <Industrial> So for each device I'm fetching the last document with a location key, and write all those to a new collection
[10:43:12] <fontanon> Hi. What is the usual procedure when you need to create temporary collections to perform further aggregations on that? I think creating temporary collections on the same prod-env databse is really dirty.
[11:42:15] <chris|> when looking at the stats of a capped collection, I see a maxSize, a size and a storageSize value. Which of the later two is relevant for when documents get deleted from the capped collection because of maxSize?
[13:15:44] <cheeser> the collection will "exist" on all shards. it's the documents that get shifted around
[13:22:14] <spleen> cheeser, my collections are present on one shard only.. or two shard when collections are chunked
[13:23:26] <cheeser> the collections should be "there" (as much as mongo supports the notion) everywhere regardless of document location. e.g., if you define a non-ID index, that index should exist on all shards.
[13:23:48] <spleen> I am looking for a way to know actually where a chunk of a collections is written ? shardA, shardB, or shardC..
[13:24:13] <cheeser> where things live in the sharded cluster is not usually something you have to care about.
[13:25:44] <spleen> cheeser, i have to move a collection to another shard. For that i need to know where a collection is
[14:14:41] <kurushiyama> spleen Non-sharded collections _always_ on the primary shard. Chunks are not written. Chunk ranges are assigned to shards – so you always know where a document gets written to. If a chunk is split, then it remains on the same shard until it gets balaned away.
[14:49:56] <dino82> So apparently I am replicating to a node that doesn't show up in rs.status() or rs.conf() ?
[14:52:31] <dino82> Well, if I rs.remove a node, it begins replicating, and if I rs.add it, it says "Our replication set configuration is invalid or does not include us"
[16:51:09] <jsheely> What would be the best way to store a javascript function in mongo? I'm using angular formly to schema and I want to store both JSON schema as well as functions
[16:58:57] <StephenLynx> if your model consist of a bunch of overly complicated strings, its a bad model.
[17:15:55] <jsheely> Well, It's not a matter of expressive or whatever. It's literally how the schema works. So I either need to write some complicated abstraction over it or use it as is
[17:26:58] <StephenLynx> and if that is not enough, you either don't implement it or you use a different tool.
[18:05:05] <GothAlice> StephenLynx: You are entirely wrong on the "MongoDB doesn't implement schemas" bit, BTW.
[18:05:28] <GothAlice> https://docs.mongodb.com/manual/core/document-validation/ https://www.mongodb.com/presentations/webinar-document-validation-in-mongodb-3-2
[18:06:52] <GothAlice> Considering it hooks every insert and update unless expressly disabled on the query, I'd call that pretty core.
[18:07:26] <GothAlice> (And even then, some, like SQLite, are almost no-SQL in that they don't enforce their schemas at all, really.)
[18:08:37] <GothAlice> My own suite of Python MongoDB tools (that are explicitly not going be Active Record ODM) I'm writing to instead of implement schema and validation application-side, have it generate validation documents for DB-side use. The Way Of The Future™ :D
[18:12:32] <teprrr> hi, I'm new to mongodb, and was wondering if this document structure look sane to you? https://pastee.org/c3pnu
[18:13:14] <GothAlice> teprrr: Deep nesting suffers from substantial issues in how you can manipulate that data.
[18:13:43] <teprrr> I have a list of domains which are being resolved regularly. now I'd like to do things like: 1) unique IPs per domain 2) augment those IP addresses with extra data
[18:13:45] <GothAlice> teprrr: There are quite a few points to cover, let me gist a copy of this and add notes.
[18:15:24] <teprrr> yeah, I'm struggling exactly with nesting here, when trying to do anything a bit more complex.
[18:16:10] <teprrr> and well, now I'd like to augment those IPs with extra information, but would/should it then be stored in another collection & joined together on the client-side, or what's a sane approach for it?
[18:18:20] <GothAlice> teprrr: The key is to not nest arrays; multiple arrays at the same level in a document _might_ be OK, but only if you never need to query subsets of both at the same time. (The $ projection operator only works on one array in a document at a time.)
[18:18:52] <teprrr> GothAlice: ah, thanks. good catch with ordering of the dns tree :) the entries in ips array come directly from elsewhere, although it's adaptable for sure.
[18:19:00] <GothAlice> Arrays of sub-documents are a-OK, as long as those don't also contain arrays. (So my advise there to turn "ips" into a list of strings instead, to avoid storing "address_type", can be ignored if you intend to add more fields there.)
[18:19:08] <teprrr> GothAlice: about having those answers in a separate collection, looks like JOINs are a pretty new feature?
[18:19:39] <GothAlice> That means two queries, and no need for use of the recent $lookup, but $lookup during aggregate queries would be useful here, too.
[18:21:36] <teprrr> and thanks StephenLynx, encoding those as ints is good, although PITA when doing adhoc debugging with robomongo/pymongo :/
[18:24:16] <GothAlice> Well, subnet searching is trivial. True, exact numeric range searching is not so much. teprrr: The question boils down to: how will you use that data? Will you search on IP ranges, exact matches, or not at all?
[18:24:42] <StephenLynx> yeah, if you are just going to use them as strings, you might as well store them as strings.
[18:24:59] <GothAlice> MongoDB forces you to really think about how you're going to _use_ and _manipulate_ the data as a focus instead of focusing on "purity of data structure" (which highly nested structures would be… very pure, but not very practical).
[18:27:40] <GothAlice> With IPs stored as integers, subnetting involves bit masks or math (modulo and friends).
[18:30:39] <teprrr> but how would you store augmented info about those ips? a separate collection too? would give me know a rdmbs-like setup with three collections: domains (with name & metadata), results (raw data + some extracted info such as that ip) + ip_info (or somesuch with the data gathered from elsewhere)
[18:31:02] <teprrr> or even so, that one collection for ips, which then have backlinks to their respective domains & when that info was gathered
[18:31:07] <GothAlice> As I mentioned, ignore my advise to turn it into a list of strings (instead of a list of sub-documents) if you're going to add more fields there.
[18:31:41] <StephenLynx> then you put the ip itself on one field, and the other data on other fields.
[18:34:41] <teprrr> StephenLynx: yeah, was thinking about that. but it still needs a backlink towards hostnames including when it was seen (there may be more than one host resolving to the same IP, which is something I'm interested in :)
[18:36:45] <teprrr> having forward links on domain collection for IPs seen for that domain, and viceversa from IP collection to domains it's been seen
[18:38:36] <GothAlice> teprrr: Often what appears to be redundant is required to facilitate or optimize a query.
[18:39:33] <GothAlice> For example, consider "owner" references for records. Since the #1 thing I do with the "owner" field is display it with the username (or actual name) as a link to their profile, it makes sense to save an extra query to look that data up and just store the username with the ObjectId reference. E.g. {…, "owner": {id: ObjectId(…), name: "GothAlice"}}
[18:40:05] <GothAlice> "Duplicated", but incredibly efficient to service the #1 typical use for the field.
[18:41:56] <teprrr> actually that data structure I have sucks also by not having the exact times when those IPs were resolved. ah, great :)
[18:42:40] <teprrr> what I'd like to do is to make some sort of timeline of those results. and also compare which all domains were pointing out to same addresses at some point etc.
[18:42:50] <GothAlice> teprrr: If you have document-per-ip, the _id ObjectId will contain the creation time automatically. (Not times of each resolution, but people often miss that they don't need a "created" time because of this.)
[18:43:43] <GothAlice> Also, for the "update some statistics, or create a document if missing" situation, you want upserts.
[18:43:57] <teprrr> however, the same IP is resolved again and again, so I need to keep that manually
[18:45:00] <teprrr> a nice "oneliner" incrementing the counter, updating the last checked and pushing the newest results into that array :)
[18:46:03] <GothAlice> teprrr: http://www.devsmash.com/blog/mongodb-ad-hoc-analytics-aggregation-framework is the article I normally link for analytics of time-series data (which is what the IP "hit" events sorta fall into)
[18:46:28] <GothAlice> This article is a touch old, but covers various concerns for different ways of storing such data, from performance of different types of query on the data, to data storage and index overhead size.
[18:47:46] <teprrr> GothAlice: awesome, thanks, will read it up later. what I atm have is a really adhoc hack I built yesterday, and already stumbled upon query problems after wanting more than "how many IPs there are in total"
[18:48:32] <GothAlice> Yeah, pre-aggregation (the technique the article covers) can help, but you'll need to include the statistics you want to gather in your upsert. (That's the pre- part of pre-aggregation.)
[18:49:06] <GothAlice> That makes asking questions of the data easier and more efficient by having the data handy, rather than needing to expensively re-calculate for each question.
[18:49:07] <teprrr> I'm thinking I'm just unsure how much logic should I build already to queries versus what to process in python after fetching the data from the db
[18:49:41] <GothAlice> teprrr: https://gist.github.com/amcgregor/d6a507cfba5eed0606d6cc78c2384d75 is an aggregate query from work, missing an initial $match, but you get the idea.
[18:50:03] <GothAlice> The processing done application-side is only to apply the "discount" field's instructions (add, subtract, multiply, etc. by an amount) to the totals.
[18:50:33] <teprrr> will need to read and experiment more. anyway, thanks a lot for your time and tips :)
[19:01:15] <GothAlice> Quite correct; being non-relational, there is no referential integrity checking.
[19:01:18] <StephenLynx> and you can crash into a wall if you misuse them, but as cheeser said, there are valid use cases.
[19:01:28] <GothAlice> (This means: make sure your client application can handle bad reference edge cases…)
[19:16:38] <jayjo> When I ran my code with line_profiler, I only get back one line that reads "Timer unit: 1e-06 s"
[19:44:08] <synthmeat> so, if i have a collection of user documents (autoindexed), and i have a "rating" Number field in them, how would one go constructing a query to get 10 users (including one particular user) that are sorted by rating around that user?
[19:51:52] <GothAlice> There's a way to do it, that also adds a slight race condition, but it is possible. Expensive, but possible. Mongoose… is unfortunate, but shouldn't overly get in the way. First, you need to determine where the record you want to target is in the sorted data. (Project just IDs, iterate until you find it.) Now that you know the offset, you can subtract 5 from it to find the offset to seek to in the larger query.
[19:52:35] <GothAlice> synthmeat: Unfortunately, skip use is… not very efficient. It requires visiting the leaves of the index to count results (if using an index, otherwise a table scan!) with logarithmically worse performance the further you go.
[19:53:30] <kurushiyama> @GothAlice Regarding your validation library that generates server side validations: That sounds like a pretty interesting concept. Would you mind me stealing it?
[19:53:58] <GothAlice> kurushiyama: https://github.com/marrow/mongo — go nuts. There's a few things in there, including a really nice capped collection tailing query helper thing.
[19:54:14] <GothAlice> kurushiyama: If you're targeting Python, though, pull requests are greatly appreciated! :D
[19:55:16] <GothAlice> kurushiyama: :P Also, it's not a validation library. It's a ODM, just not Active Mapper pattern. The "schema" stuff comes from its own isolated "declarative schema" package: https://github.com/marrow/schema (which does include validation and transformation tools)
[19:55:45] <synthmeat> StephenLynx: so you keep on saying and i keep agreeing, but i'm too far off in the project to drop it
[19:56:10] <GothAlice> synthmeat: "Don't use Mongoose" is the general advise; it's known to cause lots of problems, and occasionally encourages bad practices. But yeah, practicality beats purity, and for this problem, it's basically just two simple queries and a for loop.
[19:56:49] <synthmeat> GothAlice: i'm still not sure i understood the answer though. trying to parse it third time around
[19:58:17] <GothAlice> synthmeat: Sadly, my p-code here will be semi-Python, but the idea should transfer. for i, doc in enumerate(collection.find({… criteria …}, {"_id": 1}, order=[…])): if doc._id == target: offset = i
[19:58:52] <GothAlice> offset -= 5; results = collection.find({… criteria …}, {… desired projection …}, order=[… the same order …], skip=offset, limit=10)
[19:59:24] <GothAlice> And there you go. results will be the +/- 5 records around the target, and the target. Unless you get hit by the race condition…
[19:59:46] <GothAlice> Oh, also "break" in that "if doc._id ==" bit, no need to keep searching once found.
[20:00:00] <GothAlice> (Between the first query completing and the second starting the data might change, thus the race condition.)
[20:04:26] <GothAlice> No. The first time you're rapidly searching just the IDs for the one you're looking for, counting as you go to find the offset.
[20:04:41] <GothAlice> The second time you're doing your real query, and only getting 11 results back. ±5 around your target, and the target.
[20:20:12] <synthmeat> GothAlice: something like this? https://gist.github.com/synthmeat/1550579ba2ff9c4495b8c7df761c00f1
[20:21:24] <GothAlice> Additionally, that code is highly concerning on line 15, seeming to imply that _all results are loaded by that point_, which is going to be a terrifying amount of data, even just IDs, depending on how many users you have.
[20:21:51] <GothAlice> I.e. it'll have to wait to collect everything there, before continuing, instead of the "stop pulling in data once we have a match" process in my p-code example.
[20:24:58] <GothAlice> Load your user's score and ID, then perform two queries to load records whose score is >, inversely sorted, and a second query to load records whose score is <, sorted normally.
[20:25:27] <GothAlice> (Again, there's race condition, but the overall queries should be faster for… that many records.)
[20:31:38] <synthmeat> well, i'm just hoping 4GB mongodb instance will hold up for now, the rest should be fine
[20:32:08] <synthmeat> GothAlice: this is the infrastructure for now https://www.dropbox.com/s/19ebfymsa27kdj9/Screenshot%202016-06-15%2012.52.46.png?dl=0
[20:33:13] <GothAlice> Not to mention that htop is a pretty heavy load just to get general uptime/load/mem stats. ;P
[20:34:26] <synthmeat> GothAlice: that's just my "emergency" tmux layout, i don't have it running all the time
[20:34:30] <GothAlice> On one game project at work, after I was onboarded as a fresh hire, I replaced about a half dozen other tools with MongoDB, saving immensely on infrastructure and maintenance by reducing the moving part count.
[20:36:20] <GothAlice> Hehe, have you ever looked into Stackless Python, as powers the EVE Online client and persistent server universe?
[20:37:14] <synthmeat> tbh, i'm looking into something typed for beyond prototype/first iteration stuff
[20:41:04] <GothAlice> Plus, Pypy JIT, or for "full, strict typing that enforces singular typing of all variables", there's also RPython.
[20:41:32] <GothAlice> (RPython infers type, but ensures it's constant once inferred. Thus: no need to explicitly write any typing yourself, just be consistent and it won't explode.)
[20:47:32] <synthmeat> it is. i'm loving it. can't wait to start on the next project with it (in a week, once i release the game)
[22:39:45] <troydm> hey all! please suggest some open source free GUI tool for Windows, doesn't needs to have much, just to be able to browse database and edit data