Log file Viewer
#pypa logs for Thursday the 11th of September, 2014
[00:12:32] <crawln> yeah i came up with the command after some trial and failure with the wget command using -O -
[01:23:48] <Pinchiukas> Ok the installation of packages while offline is documented but how do I install pip itself offline? I don't have the ability to install system packages so --user is my only choice.
[02:10:06] <collinanderson> i subscribed to distutils-sig and send an email, and it's not showing up in the archives. did i do something wrong?
[12:15:53] <namsor_com> Hi! I've just installed Python, failed to install PIP to try and install 'hammock'. Any helping hand?
[12:23:24] <namsor_com> Ok found it's already installed. 'pip install something' needs be called in a regular command line, not in Python lang
[20:03:15] <requiredfield> Hi #pip, I ran "pip install --upgrade requests[secuirty]" in a python3 virtualenv yesterday and it seems to have broken pip? now when i pip install something it fails with a SyntaxError because it's running code in .../site-packages/ndg/httpsclient/ssl_peer_verification.py with a python2-style "except ImportError, e"
[20:07:46] <dstufft> pip internally uses requests, and on py2 if you do requests[security] you'll get better performance
[20:09:34] <requiredfield> didn't realize requests[security] didn't work on py3. wish it hadn't let me install it then.
[20:10:57] <requiredfield> of course now i can't "pip uninstall requests[security]" since pip is borked
[20:12:09] <Alex_Gaynor> requiredfield: yoru best bet is to just blow the the venv away and recreate it
[20:12:42] <requiredfield> oh man. Thanks Alex_Gaynor. I wonder if I can just hack site-packages/easy-install.pth?
[20:13:54] <requiredfield> Alex_Gaynor: do you happen to know who maintains ndg / why they're keeping all this so secret? Trolling unsuspecting newbs? :)
[20:14:41] <Alex_Gaynor> requiredfield: https://pypi.python.org/pypi/ndg-httpsclient is teh module you're looking for FWIW
[20:15:21] <requiredfield> while i'm here thanks for the great work you've been doing to make python 2 more secure Alex_Gaynor
[20:29:54] <buck2> hello friends. I'm trying to optimize the pip portion of our build process today, and would like to make an effort to make sure any changes are up-streamable
[20:30:27] <buck2> our process has the packages completely cached (via --download-dir) and all requirements are pinned
[20:30:47] <buck2> my plan is to tell pip not to check with the pypi server for packages that are cached and pinned
[20:31:13] <Alex_Gaynor> buck2: I think you'll be interested in the caching stuff for pip 1.6, it basically does this AFAIK /cc dstufft
[20:36:41] <DanielHolth> only having the correct versions available is a pretty effective form of pinning
[20:38:42] <buck2> pinning isn't the issue per-se, it's that it takes a good long while to install everything even when everything is cached and pinned
[20:40:36] <dstufft> with pip 1.5.x there shoudl only be 3 HTTP requests per package assuming a completely empty cache
[20:43:26] <dstufft> something sounds wrong, unless your on like the moon it shouldn't take 5 minutes if you're installing from wheels
[20:51:14] <buck2> this is how our build process works http://paste.pound-python.org/show/Y4zp3ZdaumOt77t4sJgr/
[20:51:19] <dstufft> Alex_Gaynor: honestly the biggest problem with a bulk API is I don't think it'll win us much, since a bulk API would need the ability to say "give me the stuff for all these deps" it'd mean that for each unique installation combination we'd end up having a seperate cache item in Fastly
[20:51:36] <buck2> most of the time is under pip wheel and the "file was already downloaded" messages
[20:51:50] <Alex_Gaynor> dstufft: honestly, that doesn't sound too bad, there'll be lots of really common combos
[21:00:30] <dstufft> buck2: i'd be interested to see what you have, I'm not familar with the wheel build code very well
[21:01:47] <dstufft> buck2: I mean what you have to make it faster, you said you were wanting to upstream something?
[21:02:00] <dstufft> (also you should try with the development branch, I wonder if it'd make it any faster)
[21:04:45] <buck2> i like to try to talk to upstream before i wind up walking down a very long dead end
[21:04:51] <dstufft> --download-cache is gone in the dev branch, there's an on by default cache now that operates at the HTTP request layer now
[21:06:04] <dstufft> buck2: do you want to be able to upload with setup.py upload / twine/ some other CLI, or are you ok with scp
[21:07:06] <dstufft> but all you _need_ is a bunch of stuff in a directory with nginx/apache/whatever and autoindex = on;
[21:16:39] <buck2> dstufft: now with individual timings http://paste.pound-python.org/show/NpxicBAM62PrsY1re38p/
[21:34:07] <qwcode> I haven't read this whole thread, but know that "pip wheel" currently rebuilds regardless if it exists in the --wheel-dir target https://github.com/pypa/pip/issues/855
[21:35:27] <qwcode> that skips if the pypi download is a wheel, but if the download is an sdist, then from there it always rebuilds
[21:38:21] <qwcode> buck2, yes, --find-links can work around the issue, but it is true that "pip wheel" rebuilds ignorantly
[21:48:44] <buck2> dstufft: this is profiling diagram of the problematic case https://gist.github.com/bukzor/48fffabea1484cf66ab0#file-slow-pip-png
[21:57:30] <dstufft> buck2: depends on what exactly it's requesting, the HTTP cache on pip will respect cache headers on *files* regardless of what it says (so a year or more is totally OK), but on the /simple/ pages it'll cache those for a max of 10 minutes
[21:57:45] <dstufft> (but if the cache expires it'll do a conditional GET instead of a full get if possible)
[22:02:02] <_habnabit> wickman, also, i've been pondering about how to serve static assets out of a pex... the pex contains the whole website, including static assets, and i was trying to figure out how to make nginx serve those files without requiring pyramid to expend effort on static files
[22:18:28] <wickman> habnabit: one cheat is just 'unzip -d foo foo.pex && python foo' -- the files will be on disk and you can refer to them relative to __file__
[22:18:56] <wickman> habnabit: PEX_FORCE_LOCAL=1 ./foo.pex also does this but writes assets to .pex/cache/<hash> i think
[22:22:34] <wickman> habnabit: i've seen people w/ django apps just use pkg_resources to dump static assets to a tmpdir on disk and point the webserver there
[22:24:43] <dstufft> (grnated it's a small problem in the grand scale of things, but hey it's a nice small thing)
[22:46:27] <dstufft> buck2: they effect whether it'll be cached or not, and for how long, the only thing is /simple/* stuff has a ceil of uh, 10 minutes I think on the cache
[22:46:42] <dstufft> and if your server supports conditional gets, we'll attempt to do that instead of a full GET
[22:47:35] <dstufft> buck2: there was concern that people would end up accidently caching a /simple/ page for a long time, that isn't a big deal on a file (because files shouldn't change), but the /simple/ pages *do* change
[22:48:58] <dstufft> if it's when it's downloading a file, it has no ceil, if it's the discovery of files / looking for versions, there's a 10 minute ceil
[23:05:43] <dstufft> so yea, probably the only way to get it down more is to implement the == thing then
[23:07:01] <buck2> dstufft: would you give me a hint as to which file that if statement would go into?
[23:07:46] <dstufft> that's gonna be a tough one, most of that logic is in pip/index.py... but that explicitly checks for stuff that's isntalled, it doesn't take a list of things that are installed
[23:10:44] <dstufft> https://github.com/pypa/pip/blob/develop/pip/wheel.py#L569 this is what triggers the download
[23:11:05] <buck2> dstufft: this is my critical path https://gist.github.com/bukzor/48fffabea1484cf66ab0#file-slow-pip-png
[23:11:24] <dstufft> self.requirement_set is an isntance of https://github.com/pypa/pip/blob/develop/pip/req/req_set.py#L53
[23:11:46] <dstufft> I think it makes the most sense to try and remove things from the requirement set before that line if you already have a wheel for it