Log file Viewer
#mongodb logs for Friday the 9th of September, 2016
[00:02:48] <GothAlice> With SSL, specifically using X.509: https://docs.mongodb.com/manual/core/security-internal-authentication/#member-certificate-requirements
[11:34:43] <SimpleName> new Date("2012-09-08T10:10:10") will be ok, and it will convert to iso format
[13:14:43] <w1nst0n> I was wondering if someone here uses ReactiveMongo and onSuccess{case someone => can I ask him/her a (probably) stupid question?} (there's no reactivemongo channel :( )
[13:48:55] <jelle> oh hmm maybe I've found it https://docs.mongodb.com/manual/core/document/#record-documents
[13:49:16] <LowWalker> Howdy all. So I'm writing up some automation for deploying a replica 3 member set. I'm seeing log messages and things look right but rs.conf() and rs.status() are showing this output - https://gist.github.com/lowwalker/4e226f573c72daa716f2aba1f8558f0c
[13:49:55] <LowWalker> One doc says to use rs.add("hosts:ports") but the keyfile method shows to use rs.initiate with members
[15:14:09] <spacecrab> i recently took up mongodb for a couple days with the approach of "how quickly can i learn how this works (minimally), and make a replica set?" i puked this out and it worked
[15:14:51] <spacecrab> it doesnt handle auth, and a lot of things are only minimally built features, and the initiate part of my js init script is completely unecessary (i think) but if that helps at all yay!
[15:15:20] <GothAlice> Hehehe, https://gist.github.com/amcgregor/c33da0d76350f7018875 is my version of that, setting up two three-node replicas as shards, with authentication. (It's a bit old at this point, though.)
[15:15:49] <spacecrab> but thanks to GothAlice i was able to grasp some basic concepts and from there hash out this basic replica thing-y
[15:19:50] <rockyh> I am completely new as regards databases and consequently mongodb, trying to document myself about them. Are there some examples about how mongodb can be used to store mails? I found some file archiving (mp4) with GridFS, but it seems not exactly to be the same thing
[15:21:16] <GothAlice> rockyh: E-mail messages are MIME documents. You can store them quite easily with a little bit of "header name mangling" (i.e. conversion to lower-case, replacing hyphens with underscores) as MongoDB documents. Individual e-mails shouldn't really ever exceed the 16MB document size limit under normal circumstances. (Many e-mail servers limit e-mails to 10MB or so.)
[15:22:13] <GothAlice> I.e. {from: "Alice <alice@…>", to: "Bob Dole <bdole@…>", sender: "…", subject: "…", root: "mime root, i.e. body"}
[15:23:38] <GothAlice> You can even handle multipart by making "root" a list of individual MIME parts.
[15:24:36] <rockyh> ok, and do you think is there a way to do the same without the 16 MB constraint?
[15:24:40] <GothAlice> {…, body: [{mime: "text/plain", root: "Plain text version."}, {mime: "text/html", root: "<p>HTML version.</p>"}]}
[15:25:26] <GothAlice> The above plain text + HTML example, with attachments stored in GridFS might look like:
[15:26:32] <GothAlice> {…, body: [{mime: "text/plain", root: "Plain text version."}, {mime: "text/html", root: "<p>HTML version.</p>"}, {mime: "application/octet-stream", filename: "test.zip", disposition: "attachment", grid: ObjectId(…)]}
[15:26:53] <GothAlice> Handling it this way would allow you to still make use of things like full text indexing of the text parts. :)
[15:27:14] <LowWalker> @spacecrab, I'll try those adds in my init and see if that bumps me to the right state. And no, I think everything is private by default and I clean them up :)
[15:27:17] <GothAlice> (And have a GridFS store of individual attachments, not just whole e-mail blobs.)
[15:27:52] <LowWalker> @spacecrab, I've got auth and everything handled in Ansible at the moment. I'm just ending at a weird state
[15:28:29] <rockyh> ...because if I put the entire e-mail into GridFS its text will not be indexable...
[15:29:00] <GothAlice> rockyh: Generally, yes. You can manipulate the GridFS collections a bit manually, but that's… note entirely supported. There be dragons, potentially. ;)
[15:31:54] <GothAlice> rockyh: Now, there is some duplication, here. GridFS can store filename and mime type with the file stored there, but I included it in the e-mail document itself to save needing extra lookups (no joins ;) to allow searching and easy display.
[15:32:31] <GothAlice> rockyh: https://docs.python.org/2/library/email.mime.html < this is how you manipulate MIME objects in Python, for example. MIME is a bit complex, thus this class hierarchy is a bit complex, but the idea is there.
[15:34:12] <GothAlice> rockyh: https://github.com/marrow/mailer/blob/develop/marrow/mailer/message.py#L214-L258 is an example of using that module and its classes. It's easier if you have an e-mail message as a string to parse, but this module is an abstraction around e-mail messages that's a bit easier to use.
[15:35:01] <GothAlice> However a statistical analysis of StackExchange questions would seem to indicate a slight lead for Python in terms of recommendations for a language to learn.
[15:36:01] <GothAlice> I would recommend using Python 3, however. It's been out for ~10 years, and has some really, really nice newer features. (Such as full core support for async in 3.5.)
[15:38:40] <GothAlice> Quite so. Interestingly, Ubuntu ship Python 3 out of the box (according to the new machine I just spun up ;), as do many other distributions.
[15:39:19] <GothAlice> (And contrary to popular belief, it's pretty easy to write code that's compatible with both. ;)
[15:39:22] <rockyh> so do you think that, with Python and your links, it would not be so complicated to parse an e-mail (given as a string) and to fill a mongodb item?
[15:40:49] <GothAlice> Definitely. My test machines have CPython 2.6, 2.7, 3.3, 3.4, 3.5, Pypy2, and Pypy3 all installed at the same time! :D
[15:42:02] <GothAlice> One big benefit of Python 3.4 and 3.5, though, is the built-in inclusion of the "venv" module to help isolate your application dependencies into a project folder, rather than needing to install modules globally. Once you have a python3.5 command working, try running: python3.5 -m venv
[15:44:31] <rockyh> this seems to be very useful, even if I maybe would not install 3.5 as first version
[15:45:16] <GothAlice> Hmm, then you're going to want to globally install the "virtualenv" package, which provides the functionality of "venv" in older versions before 3.4.
[15:46:22] <rockyh> so I would have in - say - 3.0 the same benefits as 3.5 as regards the application dependencies?
[15:47:29] <GothAlice> 3.4 is the earliest 3.x version I'd recommend using. 3.5 is the current one. (Also easy to grab a copy on Ubuntu, for example: add-apt-repository spa:fkrull/deadsnakes; apt-get install python3.5)
[15:48:36] <GothAlice> There is some newer syntax ("async def", "async for", "async …") that is not backwards compatible below 3.5.
[15:49:19] <GothAlice> (As with all things: it's usually best to use the "current" version, of anything, not just Python. ;)
[15:50:11] <rockyh> older versions (of Python) were suggested as better-documented and better-tested, this is the reason I never used the current one
[15:50:19] <GothAlice> Starting with MongoDB, one wouldn't choose to install 2.6.12 if 3.2.5 is available, no?
[15:50:51] <GothAlice> Tests get added over time, so newer versions will generally have more tests, and more complete coverage. Same with documentation. I'd be curious where you got that suggestion from?
[15:53:19] <GothAlice> But that's why there's that PPA provided by fkrull, which contains current versions and many older ones, too.
[15:56:32] <GothAlice> Exactly. Or install, as is the case on my test machines, all the versions. :D
[15:58:17] <rockyh> backward compatibility is often desirable. So you seem to be very skilled not only with mongodb, but also with Python!!!
[15:59:00] <GothAlice> It's my job. :) Over 100 (combined public and private) Python-related Github repositories so far.
[16:00:54] <StephenLynx> whever you need a version they don't provide, you just compile from source.
[21:35:00] <nienn> Hello. Why "unwinding": { preserveNullAndEmptyArrays: true } in $lookup return "field must be string"
[21:35:21] <nienn> it's from here https://docs.mongodb.com/manual/core/aggregation-pipeline-optimization/#lookup-unwind-coalescence
[23:11:44] <dtrott> In mongo is there anyway to automatically run a DB init script when a stand alone server (actually a docker instance) starts up, this is just a CI server instance and I need to pre-load some canned data / users
[23:14:11] <dtrott> Currently I copy a tiny (200k) database into the docker image before booting it, however I am getting feedback that its not “configuration as code” hence the desire to run a script instead.