PMXBOT Log file Viewer

Help | Karma | Search:

#pypa logs for Friday the 24th of April, 2015

(Back to #pypa overview) (Back to channel listing) (Animate logs)
[03:03:00] <dstufft> tdsmith: ping
[14:41:10] <jessamynsmith> I have an idea for pip, and I'm wondering if anyone would like to talk it through with me?
[15:52:31] <carljm> jessamynsmith: I'm not actively working on pip these days, so my buy-in doesn't mean a lot, but I at least have "historical perspective" :-) What's your idea?
[15:52:51] <jessamynsmith> Hi again, carljm :)
[15:52:59] <carljm> Hi :-)
[15:53:05] <jessamynsmith> historical perspective is definitely helpful
[15:53:34] <jessamynsmith> I routinely work on apps that have test requirements I don't need or want in production
[15:53:45] <jessamynsmith> As far as I know, pip has nothing built in to support environments
[15:53:51] <jessamynsmith> (let me know if I'm wrong on this)
[15:54:26] <carljm> not exactly, no. though the ability to include one reqs file from another is useful in that situation
[15:54:31] <jessamynsmith> I actually built myself a little tool because I was tired of manually sorting new requirements into files
[15:54:38] <jessamynsmith> yeah, I use multiple files with includes
[15:54:50] <carljm> yes, that's how I handle that as well
[15:54:57] <jessamynsmith> the main case that isn't handled nicely is I add a new requirement, and I want to auto-add it
[15:55:18] <carljm> yeah, like `npm install --save` and `npm install --save-dev`
[15:55:21] <jessamynsmith> or,more often, I add some new lib that adds 4 other libs, and then I was manually cutting those out of the output of pip freeze
[15:55:22] <jessamynsmith> yes
[15:55:29] <jessamynsmith> exactly like that
[15:55:47] <jessamynsmith> I had actually had the idea already, and then encountered it recently in node, and thought YES
[15:56:01] <carljm> so in the "4 other libs" case, my view is that you want those in your reqs file
[15:56:10] <carljm> so their versions are also pinned
[15:56:14] <jessamynsmith> that's what I do, yes
[15:56:24] <carljm> ah ok
[15:56:44] <carljm> yeah, I definitely think there is room in the Python world for a tool (probably built on top of pip) that does some of what you're talking about
[15:56:48] <jessamynsmith> I built a thing, which I use in my projects: pypi.python.org/pypi/pipreq
[15:57:00] <carljm> I think the main thing that would make a difference to me is something like Gemfile vs Gemfile.lock
[15:57:06] <jessamynsmith> oh?
[15:57:09] <jessamynsmith> I don't know about gemfiles
[15:57:24] <carljm> where you can have one file that is maintained manually that includes only your top-level direct dependencies
[15:57:35] <jessamynsmith> (btw, I realize that you can actually achieve the upgrade all option with a combination of pip commands)
[15:57:42] <carljm> and then another auto-generated file that pins everything all the way down
[15:57:45] <jessamynsmith> oh interesting
[15:57:55] <jessamynsmith> I haven't ventured into the ruby world
[15:57:58] <jessamynsmith> did not know about that
[15:57:59] <carljm> so you get the benefits of full pinning, with less maintenance headache
[15:58:10] <jessamynsmith> fascinating
[15:58:19] <jessamynsmith> I hadn't even thought of going that direction
[15:59:11] <jessamynsmith> my current thinking is that cleaning up a virtualenv to match a set of requirements should live in virtualenvwrapper -- I'm going to see about getting a pr accepted
[15:59:19] <jessamynsmith> upgrading all is achievable with just pip
[15:59:25] <jessamynsmith> (I didn't realize at the time)
[15:59:37] <jessamynsmith> the big thing is splitting up and maintaining requirements files
[16:00:08] <carljm> pipreq looks useful for sure
[16:00:18] <jessamynsmith> it solves my problem :)
[16:00:26] <jessamynsmith> but it requires an extra file, which I don't love
[16:00:36] <carljm> having both .requirementsrc and the actual requirements files seems like a potential source of issues, yeah
[16:00:53] <carljm> it seems like it ought to be possible (though maybe not so simple) to implement pipreq without .requirementsrc
[16:00:55] <jessamynsmith> it's also a simplistic tool, which doesn't handle installing from source well
[16:01:02] <carljm> just using the actual requirements files themselves as the source of truth
[16:01:20] <jessamynsmith> right, so, I think I could do something like that, with something like this: github.com/pypa/pip/issues/1433
[16:01:48] <jessamynsmith> in which case, you're back to a single requirements file, with sections
[16:02:00] <jessamynsmith> magical comments to delineate, perhaps
[16:02:15] <carljm> why couldn't it manage multiple files? just point it at a directory
[16:02:49] <jessamynsmith> hm, I am thinking
[16:03:10] <jessamynsmith> it certainly could
[16:03:20] <jessamynsmith> and it could still have the interactive bit to ask about new deps, i.e. which file they go into
[16:03:24] <carljm> yep
[16:03:25] <jessamynsmith> yes, I think this could work
[16:03:47] <carljm> it would be cool to have a wrapper around pip install
[16:03:54] <carljm> where you just give it an arg to tell it where to put it
[16:03:58] <jessamynsmith> yeah
[16:04:00] <carljm> so the equiv of npm install --save-dev
[16:04:05] <jessamynsmith> agreed
[16:04:11] <jessamynsmith> thank you *so* much
[16:04:21] <jessamynsmith> it's so helpful to talk things over with someone else who gets the problem
[16:04:33] <carljm> sure. thanks for working on this
[16:04:40] <jessamynsmith> so far this all just came out of my own head, and I felt it was unnecessarily complex
[16:04:56] <carljm> you might also be interested in Mozilla's "peep" tool, which solves a very different problem, but is still in the category of "wrappers around pip"
[16:05:04] <jessamynsmith> I experiment with packages a lot
[16:05:15] <jessamynsmith> and I found virtualenv/requirements maintenance was taking too much time
[16:05:24] <jessamynsmith> ok, good to know
[16:05:47] <jessamynsmith> I'm going to see about implementing this
[16:05:59] <carljm> yeah, for me it's always in that category of "I have ideas about what a better tool would look like, but the current tooling works just well enough that it never makes it to the top of the list"
[16:06:16] <jessamynsmith> maybe I install a lot of packages?
[16:06:31] <jessamynsmith> for me that line moved past the pain point
[16:07:14] <carljm> maybe. or maybe I've been doing this too long and my pain tolerance has gotten too high :)
[16:07:21] <jessamynsmith> hahaha
[16:07:28] <jessamynsmith> could be
[16:07:59] <jessamynsmith> I still think it would be great if pip eventually pushed people toward some standard naming
[16:08:14] <jessamynsmith> but for now, requirements files have so many names
[16:08:44] <carljm> funny story about npm install --save; the person on my team who is responsible for package.json wants the dependencies alphabetical, and npm doesn't do that, so we still have to go in and manually edit package.json anyway :-)
[16:08:52] <jessamynsmith> how do you organize your projects? requirements/env.txt or do you have them all top-level
[16:08:58] <carljm> there's always that one more step that the tooling doesn't quite handle automatically...
[16:09:13] <jessamynsmith> yeah, so, when pipreq divides them it makes it alphabetical
[16:09:17] <carljm> nice
[16:09:20] <jessamynsmith> because I am that way :)
[16:09:39] <carljm> I do a requirements/ dir
[16:09:43] <jessamynsmith> same here
[16:09:45] <jessamynsmith> I like it better
[16:09:48] <carljm> usually with base.txt, dev.txt, and prod.txt
[16:09:58] <jessamynsmith> I used to abbreviate, now I don't
[16:10:02] <carljm> I try to avoid getting any more complex than that
[16:10:06] <jessamynsmith> I have common.txt, development.text, production.text
[16:10:10] <jessamynsmith> very similar though
[16:10:11] <carljm> sure, same idea
[16:10:20] <carljm> I aim to minimize or get rid of prod.txt though
[16:10:29] <carljm> in the spirit of the 12-factor app
[16:10:30] <jessamynsmith> prod.txt is basically to support heroku stuff
[16:10:34] <jessamynsmith> (for me)
[16:10:36] <carljm> use as many of the same libs in dev as in prod
[16:10:44] <doismellburning> 12factor represent!
[16:10:45] <jessamynsmith> things like dj-database-url
[16:10:51] <doismellburning> jessamynsmith: see, I use those in dev
[16:10:53] <carljm> yeah - I use that locally too
[16:10:56] <jessamynsmith> ah
[16:11:04] <carljm> recently switched from runserver to gunicorn for local dev, too, in the same vein
[16:11:09] <jessamynsmith> hm
[16:11:13] <jessamynsmith> interesting
[16:11:13] <carljm> (now that gunicorn has built-in auto-reloading)
[16:11:13] <doismellburning> carljm: ooh interesting, not sure I could do that
[16:11:21] <doismellburning> oooh ok that makes more sense!
[16:12:04] <jessamynsmith> dj-database-url==0.3.0
[16:12:04] <jessamynsmith> gunicorn==19.3.0
[16:12:04] <jessamynsmith> newrelic==2.50.0.39
[16:12:08] <carljm> the one remaining problem is that gunicorn doesn't handle pdb gracefully, because of the multi-process setup and all
[16:12:10] <jessamynsmith> so yeah, that's a typical one
[16:12:30] <carljm> usually I don't need to pdb on an actual request, just in a test. but every now and then I fire up runserver so I can stick a pdb in a live request
[16:12:31] <jessamynsmith> I use pycharm which "just knows" how to run the django server, so I use that for debugging
[16:12:40] <jessamynsmith> oh, I debug the live server all the time
[16:12:43] <jessamynsmith> (and tests)
[16:13:00] <jessamynsmith> I wonder if newrelic can be there in dev without messing htings up
[16:13:04] <jessamynsmith> I haven't actually tired
[16:13:06] <jessamynsmith> *tried
[16:13:12] <jessamynsmith> I agree, though, my production stuff is minimal
[16:13:12] <carljm> jessamynsmith: yeah, my prod.txt just has newrelic; I've moved everything else out
[16:13:20] <jessamynsmith> *nod*
[16:13:40] <doismellburning> I've basically never used pdb
[16:13:56] <jessamynsmith> carljm: so you just export local vars to make things work like heroku?
[16:14:08] <doismellburning> I do need to work out how to get LiveServerTestCase using gunicorn though
[16:14:14] <doismellburning> jessamynsmith: fwiw, I use honcho
[16:14:16] <carljm> jessamynsmith: yeah, I've adapted the 12-factor "config via env vars" everywhere
[16:14:22] <carljm> and I also use honcho and a .env file
[16:14:33] <jessamynsmith> carljm: I still haven't read the 12-factor thing, but I do tend to do that
[16:14:36] <jessamynsmith> I ahve never heard of honcho
[16:14:40] <carljm> (though usually I aim to have my "default" settings - i.e. if there are no env vars - as close as possible to my desired dev config)
[16:14:56] <carljm> so generally on my projects you can fire them up for local dev with no env vars at all and it'll be reasonable
[16:15:09] <jessamynsmith> yeah, I learned that the hard way...
[16:15:25] <jessamynsmith> I have my django secret key in an env var, but then people can't just clone my repo and run without setting an env var
[16:15:28] <jessamynsmith> so now it defaults in dev
[16:15:45] <doismellburning> jessamynsmith: well hello ;)
[16:15:50] <jessamynsmith> hello!
[16:16:01] <doismellburning> jessamynsmith: (oh, wait, django12factor still requires setting an env var, sorry, carry on)
[16:16:08] <jessamynsmith> heh
[16:16:17] <doismellburning> (because it doesn't require SECRET_KEY if DEBUG is true, BUT it defaults to debug=false for safety
[16:16:20] <doismellburning> )
[16:16:26] <carljm> what I do to avoid mistakes in prod is I have a MODE setting
[16:16:26] <jessamynsmith> I will look up 12factor and honcho
[16:16:29] <doismellburning> I still might change that behaviour in v2
[16:16:35] <carljm> which can be either 'dev' or 'prod', defaults to 'dev'
[16:16:43] <jessamynsmith> when I get back to django
[16:16:57] <carljm> but if it's set to 'prod' (via an env var of course), then a bunch of other settings (including SECRET_KEY) error out if not set explicitly via env var
[16:17:04] <jessamynsmith> carljm: oh cool
[16:17:08] <jessamynsmith> that'd work
[16:17:20] <carljm> so as long as I remember to set the env var for MODE='prod' on my prod deployment, I can't forget to set anything else important
[16:17:26] <jessamynsmith> right now I have multiple settings files; I gather you've gotten everything to work with one?
[16:17:42] <doismellburning> jessamynsmith: oooh do you have a public project I can attempt to 12factor via PR? :D
[16:17:46] <doismellburning> jessamynsmith: eggtimer?
[16:17:47] <carljm> yeah, I just use one, with a utility function to pull stuff from the env (with the MODE-based logic I just described)
[16:17:54] <carljm> and then a bunch of env-based settings in it
[16:18:04] <jessamynsmith> github.com/jessamynsmith/eggtimer-server
[16:18:15] <jessamynsmith> doismellburning: that'd be amazing
[16:18:50] <carljm> doismellburning: I avoid using DEBUG as a proxy for dev vs prod, because DEBUG changes Django's behavior, and sometimes I want to test DEBUG=False locally without having to go into "full prod" mode on all my settings
[16:19:11] <jessamynsmith> I have the same issue
[16:19:28] <doismellburning> carljm: I don't use it as a proxy for that
[16:19:33] <carljm> (somehow this conversation veered from pip requirements handling to 12-factor and Django, making it a bit OT here...)
[16:19:45] <doismellburning> carljm: but I absolutely want it to be opt_in_
[16:19:50] <carljm> doismellburning: Oh, I just saw your comment about SECRET_KEY handling above
[16:19:52] <doismellburning> and yeah, sorry, probably my fault, most things are
[16:19:58] <doismellburning> ;(
[16:20:04] <doismellburning> er, ;), silly .de keyboard
[16:20:31] <jessamynsmith> yeah, a bit
[16:22:13] <carljm> doismellburning: nope, not your fault. but it does appear that either you were watching the discussion all along, or you have an IRC highlight set up for the word "12-factor" :-)
[16:22:29] <carljm> mention 12-factor and doismellburning appears, like a genie from a bottle.
[16:22:46] <jessamynsmith> like nedbat and coverage
[16:22:59] <carljm> jessamynsmith: now you've highlighted him twice!
[16:23:06] <jessamynsmith> I am a terrible human
[16:23:08] <carljm> except he's not here, so no
[16:23:15] <carljm> oh wait, he is
[16:23:15] <doismellburning> carljm: both actually ;)
[16:23:27] <doismellburning> I also have one for staticfiles
[16:30:01] <bowlofeggs> 12-factor auth is the new way to do it
[16:30:06] <bowlofeggs> 2 factor is so last year
[16:31:57] <jessamynsmith> bowlofeggs: heheh
[16:37:42] <doismellburning> bowlofeggs: ;)
[23:02:05] <pixelfog> _habnabit: Why is dh_virtualenv really bad?