Log file Viewer

[16:08:22] <russw> Hi all. I'm having frustrating cert issues I can't resolve with pip... curl works fine. --cert (and the pip.conf equivalent) don't seem to work (strace shows no load I can see).

[16:08:39] <russw> Here's a paste with a decent summary of the status:

[16:08:45] <russw> https://bpaste.net/raw/ce3aeacf0928

[16:09:24] <russw> Seems very similar to an open issue from a year ago: https://github.com/pypa/pip/issues/1517

[16:10:17] <dstufft> russw: do any of the things you're trying to install have a setup_requires?

[16:10:22] <russw> If anyone has any advice on how I can get pip running, I'd really appreciated it! I 've previously had no issues with pip, but am rebuilding a machine and the "new" cert usage in pip is blocking me.

[16:10:32] <dstufft> oh

[16:10:34] <dstufft> pip search

[16:10:36] <russw> dstufft: it is failing on a pip search

[16:10:37] <russw> yep

[16:11:03] <dstufft> I don't think pip 1.5 supported controlling the context (like proxy, ssl, etc) of ``pip search``

[16:11:09] <dstufft> I think that's new in pip 6

[16:12:09] <russw> hmm... I installed pip with the new `ensurepip` module in 2.7.9 . Why wouldn't that give me pip 6, I wonder?

[16:12:36] <russw> I used `python -m ensurepip --upgrade`

[16:16:41] <russw> sigh... that will teach me

[16:17:32] <russw> thx for the nudge to pip 6, dstufft. I was relying on ensurepip working. When I install with get-pip.py I get pip 6 and have no issues.

[16:18:16] <dstufft> russw: ensurepip uses whatever version is embedded inside of ensurepip itself (but that can be upgraded in X.Y.Z+1 releases of Python)

[16:18:32] <dstufft> so you could do ``python -m ensurepip --upgrade && pip install --upgrade pip`` or whatever

[16:18:39] <dstufft> or just use get-pip if you want too :)

[16:18:58] <russw> dstufft: I thought the --upgrade switch was supposed to pull down the latest

[16:19:17] <russw> I'll stick with the get-pip script

[16:19:33] <dstufft> russw: that's so if you run ensurepip after upgrading say 2.7.9 yo 2.7.10, you'll get upgraded to whatever the newer version of pip is that was bundled with ensurepip

[16:20:04] <russw> dstufft: ah. Thanks for clarifying.

[16:20:06] <dstufft> perhaps ensurepip should have a flag that allows you to pull down the latest from PyPI though

[16:21:10] <russw> dstufft: That is definitely how I interpreted it. However, in my case it would have died since there is clearly a cert issue with the 1.5 pip I had that would have prevented the download

[16:21:50] <dstufft> yea, if we add a --latest or something flag we'd probably also need to add --cert and --trusted-host and --index and such

[16:28:13] <russw> rofl - was just investigating the sketchy binary at the end of get-pip.py and saw the opening comments

[16:28:16] <russw> read my mind. :)

[19:09:56] <russw> I'm now having another weird pip/setuptools issue. The start and end of an attempted `pip install numpy` is here:

[19:10:00] <russw> https://bpaste.net/show/95e716617de1

[19:11:14] <russw> I've done this build many times, but have recently moved to python 2.7.9 and am having issues.

[19:11:41] <russw> Other package upgrades that have happened may also be the problem as well since I try and build with the latest (system build from scratch).

[21:24:47] <bowlofgrapes> hello! is there a way i can get a machine friendly view of all available pypi packages and their versions and their download URLs in a single request?

[21:42:41] <dstufft> bowlofgrapes: not currently

[22:22:37] <bowlofgrapes> dstufft: ah, that's somewhat unfortunate. i'm working on http://pulp-python.readthedocs.org/en/latest/ and i'm trying to add support for pulp to synchronize packages from PyPI

[22:23:01] <bowlofgrapes> i'm currently thinking that i might use the /pypi/<package_name>/json interface to find all the versions/urls for the packages i am synching

[22:23:07] <bowlofgrapes> but it means one request per package name

[22:23:19] <dstufft> that's the best way currently

[22:23:22] <bowlofgrapes> which i can do, but i wanted to avoid that if there were a more efficient way

[22:23:26] <bowlofgrapes> cool, thanks for the info!

[22:25:44] <apollo13> bowlofgrapes: I don't think those requests reach pypi at all

[22:26:31] <bowlofgrapes> apollo13: oh, like they just hit a CDN or something?

[22:26:40] <bowlofgrapes> i was kind of worried about being mean to the server

[22:27:00] <bowlofgrapes> i'm planning to only allow users to sync a specific list of packages in my first release

[22:27:15] <bowlofgrapes> so, the first release won't get all ~45k packages ☺

[22:27:23] <apollo13> bowlofgrapes: I'd imagine that they are terminated at fastly, dstufft would know the details

[22:27:36] <bowlofgrapes> i think most users would prefer that anyway, because it really only makes sense to sync the packages you want to use

[22:27:40] <apollo13> and uploading a new package purges the cache

[22:27:46] <dstufft> oh

[22:27:47] <bowlofgrapes> ah, cool

[22:27:53] <dstufft> yea that's all cached by fastly

[22:28:00] <bowlofgrapes> oh very nice

[22:28:09] <bowlofgrapes> so this won't bother anyone! good to know!

[22:28:17] <dstufft> yea we won't even notice it

[22:32:10] <dstufft> PyPI averages something like 2 million requests a day hitting Fastly and we average roughly 87% cache hit rate

[23:25:17] <dmlockhart> hello, I have a non-traditional use-case for virtualenv and I'm hoping to get some feedback on the best way to accomplish my task

[23:27:13] <dmlockhart> I'm helping TA a course in which students will be using our servers to do Python development. We have a "global" virtualenv that we use so we don't have to bug the sysadmin everytime we want to install a new Python package. There is a core set of packages our students need for the course, so our plan was to have them activate this "global" virtualenv

[23:28:56] <dmlockhart> However, we'd also like the students to be able to install their own python packages containing their course projects. But without impacting the "global" virtualenv. We are trying to avoid having a huge number of students create their own independent virtualenvs, we'd like them to be able to inherit from the global virtualenv so the same software isn't installed 50 times

[23:29:27] <dmlockhart> does virtualenv have support for anything like "nested virtualenvs"? is there a good way to accomplish my above use case?

[23:32:42] <tdsmith> as far as the "global virtualenv" is concerned you could just install the packages to an arbitrary globally accessible path using --target I think and add it to sys.path using PYTHONPATH or a .pth file or another mechanism; doesn't sound like you really need a virtualenv

[23:33:18] <_habnabit> dmlockhart, what's the problem with there being 50 virtualenvs containing the same packages?

[23:34:14] <dmlockhart> _habnabit: memory usage

[23:34:33] <_habnabit> dmlockhart, er, it doesn't use memory. did you mean disk usage?

[23:34:59] <dmlockhart> _habnabit: yes! my mistake

[23:35:10] <_habnabit> dmlockhart, disk is cheap, and python packages are small

[23:37:51] <dstufft> dmlockhart: I think the only real ways of doing that are either --target like tdsmith said, just eating the disk space usage like _habnabit said, or compiling your own python and using that as your "global" environment

[23:38:04] <dstufft> virtualenv doesn't really support nested environments like that

[23:38:21] <dmlockhart> _habnabit: personally I have couple gigs of venvs in my home directory, and I don't have more than 20 venvs

[23:38:57] <dmlockhart> _habnabit: a small part of it is quota set on student home directories. a larger part is the professors preference to not have students all have their own virtualenv.

[23:39:23] <dstufft> dmlockhart: is it science stuff?

[23:39:35] <_habnabit> dmlockhart, does each student have their own user account?

[23:39:50] <ionelmc> user sites!

[23:39:54] <dmlockhart> dstufft: some of it is, yes

[23:40:01] <dmlockhart> _habnabit: yes each student has their own account

[23:40:08] <dstufft> (normally when someone says gigs of packages that tends to mean the science stack, their packages tend to be bigger)

[23:40:20] <dmlockhart> dstufft: numpy/scipy are definitely a large part of it...

[23:44:46] <ionelmc> dmlockhart: prolly already mentioned, but i'm curious what's wrong with user sites

[23:45:32] <dmlockhart> ionelmc: I'm reading about them now. I have zero familiarity with them currently :)

[23:46:02] <tdsmith> doesn't solve the multiple-installations problem if that's a firm criterion

[23:47:08] <dstufft> well dmlockhart said he didn't want to bug the sysadmins

[23:47:11] <dstufft> but yea that's basically it

[23:47:12] <dstufft> https://bpaste.net/show/dd25aa246f6c

[23:48:02] <tomprince> Custom local package that installs a .pth file pointing at a shared site-directory?

[23:48:23] <dmlockhart> is there a way to have --system-site-packages pull from a virtualenv rather than the system installation?

[23:48:33] <dmlockhart> kind of silly i guess

[23:49:08] <dstufft> dmlockhart: not really, you can use .pth files to fake it though maybe

[23:49:14] <apollo13> run virtualenv with the python of the parent venv?

[23:49:25] <apollo13> or will that still find the normal site-packages

[23:49:27] <dstufft> virtualenv will find the real python

[23:49:27] <ionelmc> dmlockhart: from a *severely broken* virtualenv, it could

[23:49:44] <ionelmc> i doubt you'd want that, all sorts of stuff malfunction

[23:50:39] <tomprince> Is part of the reason for wanting shared virtualenv to avoid having everybody compile the same packages? A local wheel cache would address that.

[23:54:47] <dmlockhart> tomprince: part of it is it seems silly to have 50 copies of the same packages. part of it is we'd like all the students to have a sane starting point. part of it is a professor directive that he doesn't really want the students messing with virtualenv stuff too much. maybe I can try convincing him that separate virtualenvs is the way to go, but I'm not sure he'll bite.

[23:56:18] <dstufft> dmlockhart: if you can get past the "silly to have 50 copies of the same package", ``pip install --user numpy`` will install nupy into the user site packages, students won't need to mess with virtualenv unless they want two different versions of numpy

[23:56:52] <dstufft> (and if you bug the sysadmins to install stuff into the global python, that stuff will be visible too)