Log file Viewer
#pypa logs for Sunday the 11th of January, 2015
[22:11:39] <prologic> https%3A%2F%2Fpypi.python.org%2Fpackages%2F2.7%2FP%2FPygments%2FPygments-2.0.1-py2-none-any.whl
[22:11:39] <prologic> https%3A%2F%2Fpypi.python.org%2Fpackages%2F2.7%2FP%2FPygments%2FPygments-2.0.1-py2-none-any.whl.content-type
[22:15:00] <dstufft> prologic: yea just remove them unless you want to keep them around for pre pip 6's
[22:16:33] <prologic> and just wondering if pip will still see the cached tar.gz and .zip files there
[22:17:15] <dstufft> the new pip cache doesn't actually cache tar files itself, it caches HTTP responses which just so happen to contain a tarfile
[22:18:22] <dstufft> the --wheel-dir just tells pip where to _write_ wheel files too when you do pip wheel
[22:19:10] <dstufft> --find-links will cause pip to find .tar.gz's and .whls and such in that directory
[22:19:24] <prologic> The directory '/var/lib/pip/http' or its parent directory is not owned by the current user and the cache has been disabled. Please check the permissions and owner of that directory. If executing pip with sudo, you may want the -H flag.
[22:21:16] <dstufft> that runs pip as root, but doesn't change the $HOME environment variable, pip expands ~/.cache/pip/http out using that variable, and gets something like /home/myuser/.cache/pip/http and creates those directories if they don't exist, and writes some files into that directory, but as the root user
[22:22:28] <dstufft> the solution is to use sudo -H pip install, which sets the $HOME environment variable to root's home dir (or whatever user you're sudoing too), but the ownership check and the warning was designed to tell people about that
[22:23:18] <dstufft> maybe we should extend the check out to see if we own the directory or a group we're in owns the directory
[22:23:42] <dstufft> or flip it around and check explicitly for sudo somehow since that's the primariy situation in which you'll end up with the "bad" configuration
[22:25:22] <dstufft> prologic: for the moment your best options are either make the directory owned by the user invoking pip or downgrade to 6.0.3
[22:28:18] <prologic> what should I do with my existing source archives? Put them in /var/lib/pip/sources and add /var/lib/pip/sources to --find-links? I assume that'll still work?
[22:28:35] <dstufft> prologic: well pip's cache doesn'te xactly make devpi-server useless, but it can reduce some of the benefit
[22:29:02] <dstufft> for instance, pip's http cache doesn't attempt to make it possible to install stuff without internet access
[22:30:02] <dstufft> prologic: it'll look there yes, but it won't use a stale response if the network is down
[22:31:26] <dstufft> so it's mostly about attempting to minimize the amount of time spent pulling things over the wire, but not about eliminating the need for a working wire all together
[22:32:09] <dstufft> plus devpi-server has a bunch of extra functionality like hosting multple private indexes, inherienting indexes from one another, release management workflows, that sort of thing
[22:32:25] <dstufft> but if you're just using devpi-server as a caching pypi, then the offline thing is the biggest thing
[22:33:14] <dstufft> now a future feature we want to add to pip is that instead of having seperate installinaton paths for Wheel and sdist, we only every instal from Wheel, and if a Wheel isn't available we'll download the sdist, build a wheel, cache it, and then install from that
[22:33:57] <dstufft> and that'll work a lot better (probably, in theory) in an offline kind of world I think (maybe, it's not implemented so who knows for sure)
[22:33:57] <ronny> hmm, i wonder about having a official pypi where people can publish linux binary wheels
[22:34:48] <dstufft> ronny: I don't think pip 6 changes much about comibining devpi and pip, other than you might want to use --no-cache-dir if all of your access goes through devpi since devpi will handle caching for you and hitting the localhost is likely to be pretty fast
[22:35:26] <prologic> <dstufft> now a future feature we want to add to pip is that instead of having seperate installinaton paths for Wheel and sdist, we only every instal from Wheel, and if a Wheel isn't available we'll download the sdist, build a wheel, cache it, and then install from that <-- I love it
[22:35:27] <dstufft> ronny: sooner or later we'll allow linux binary wheels on PyPI itself, we just need to extend the wheel spec to better handle Linux first
[22:35:54] <dstufft> prologic: it'll probably break things for people in some scenarios, but I think long term it's for the better
[22:36:17] <dstufft> especially since we can cache those wheels and make it so somehting like ``pip install lxml`` only compiles once for that user
[22:37:29] <dstufft> ronny: binary compat is solvable in roughly as close as it is solved on OSX and Windows
[22:39:29] <dstufft> prologic: if you can come up with something better we'll be happy to hear it :)
[22:39:48] <ronny> as faras i can tell one needs to cover 3-5 versions of centos, fedora, debian and ubuntu
[22:40:00] <dstufft> as part of that, we'll also likely allow generic wheel uploads, because it's possible to make something that'll work on most linuxes if you build it against an old enough glibc and static link everything
[22:42:16] <dstufft> ionelmc: I didn't add the distro tag because it requires a PEP update and I have a finite amount of time and more pressing things :)
[22:42:42] <ronny> prologic: quite frankly, distro name based deploy is the only sane option, because everything else has worse fail
[22:43:00] <dstufft> prologic: it has to all fit in the filename, and we don't know ahead of time what things we need to check to compare against the filenames
[22:43:39] <dstufft> prologic: nope, the .whl spec was mostly concerned with compat with the Python ABI, it didn't really consider compatability with the platform itself or other libraries
[22:44:04] <ronny> prologic: whats wrong with just having whls for the most common user and server distros
[22:44:37] <ronny> people with funky distros and strange servers shouldnt create create unreasonable binary compat maintenance efford
[22:44:39] <prologic> can we not create a situation where we cover 80% of the distros and let the other 20% build their own wheels?
[22:45:35] <dstufft> prologic: fwiw I have some other ideas around this too, like getting an official build farm so people don't have to compile their own wheels (this makes it easier for any single project to cover a larger set of distros) and doing things like embedding a file inside of the Wheel that has the ABI infomration stored inside of it, then have pip start with the most specific wheel (a distro wheel), compare the libraries, fall back to a less specific
[22:45:35] <dstufft> wheel (a generic linux wheel with static compiled lbiraries), and finally fall back to an sdist
[22:47:18] <dstufft> prologic: the problem is that a Wheel built on Distro A that links against a C library may or may not work on Distro B
[22:47:39] <dstufft> this isn't a wheel probably really, it's a "C ABI compatability across linux distro compatability" problem
[22:49:04] <dstufft> Techincally Windows and OSX have the same problem, but in practice you're far less likely to actually have it occur because they are single vendor systems so you don't have "Red Hat Windows" and "Microsoft Windows" which have different ABIs
[22:49:26] <prologic> Also it's worth nothing that you can't really guarantee that two systems of the same Distor/version have the same libraries really -- we're making an assumption that the sysadmin(s) haven't done any manual tweaking :)
[22:50:09] <dstufft> if the sysadmin has installed an ABI incompat thing on their system, well they're going to need to use --no-use-wheel for right now on any system
[22:50:23] <tdsmith> (people keep trying to send homebrew patches to allow wide builds and we keep rejecting them)
[22:50:43] <prologic> All I'm saying is (I think) -- Fine embed the disto/version into the filename; but don't make it impossible for the less popular distro/usersbase to build wheels compatible for "their" distros in some way (without our/pypa's effort)
[22:51:18] <dstufft> prologic: oh you misunderstand I think, we aren't going to devise a list of hand picked distros and hardcode their names in
[22:51:34] <prologic> I guess just like I can build wheels now in my Vagrant VM(s) based on some not-so-widely-used distro and shared the *.whl(s) with my other colleagues
[22:51:52] <dstufft> we'll use lsb-release or something to discover to the distro name (like zero effort put into figuring out if this works, but that's the general idea)
[22:52:52] <dstufft> so if someone wants to use bob's totally cool linux distro that's totally fine, they can create a wheel that has that distro name/version embedded into it
[22:53:16] <dstufft> Wheels probably won't work well for Arch given that I think they are a rolling release and that might mean ABI breaks
[22:54:38] <prologic> https://gist.github.com/ca345e61ba239419b1d5 <-- my /var/lib/pip looks like this atm :)
[22:55:04] <tdsmith> ionelmc: whether characters are stored as 16 or 32 bit values; on narrow builds unicode representation is utf-16 and on wide builds it's ucs-4
[22:55:21] <prologic> dstufft> Wheels probably won't work well for Arch given that I think they are a rolling release and that might mean ABI breaks <-- yeah :)
[22:55:55] <dstufft> tdsmith: heh, I kind of want to try and backport the ABI tags to Python 2.7 which will at least allow pip to differentiate between wide, narrow, and disabled unicode, but it obviously won't solve what I think is the main reason for y'all doing that, in that even if pip can differentiate y'all want wheels for system python to work on homebrew
[22:57:42] <tdsmith> dstufft: yeah, we'd also have to add logic to make sure our precompiled packages aren't delivered to users who can't use them
[23:00:30] <dstufft> ironically, that probably means wheels would be broken for me if people were publishing them for OSX
[23:04:36] <prologic> hmm I have a problem I can install pip 6.0.3 but I'm not sure how to pin virtualenv to 12.0.3 (uses pip 6.0.3) as well as virtualenvwrapper (which I can't find a changelog for!)
[23:04:45] <dstufft> (I totally used to use homebrew python until y'all removed some older Pythons, I forget what. Might have been 2.5 or 2.6 or something)
[23:09:22] <dstufft> it probably doesn't help that I don't really know ruby and I don't know how homebrew itself works other than I type brew install a thing and I get a thing that makes me happier than the other options most of the time :D
[23:10:00] <tos9> dstufft: /nod -- FWIW, you basically just do brew install <a commit that has python26.rb in it>
[23:10:20] <tos9> dstufft: I dislike the part of it that is the same thing I think is absurd about rbenv, the "swapping" part
[23:10:28] <tos9> which most people seem to find to be the "main" part, which means I just don't use it
[23:10:56] <tos9> Making `python` get switched around seems just useless to me, I don't mind typing `python2.6`, that's not the thing that needs fixing
[23:11:39] <dstufft> because my ``python`` is 3.4 and gyp breaks on 2.7 and I don't feel like setting up a venv
[23:13:05] <tos9> I mean to replace virtualenvwrapper at some point with something smaller that does just the parts that I use / want.
[23:13:10] <dstufft> I'd probably be just as happy with python-build (the internal command in pyenv that actually builds things) except I'm lazy and I can brew install pyenv and it sets up my paths for me
[23:13:35] <dstufft> tos9: I never really figured out what pyenv-virtualenv does, I couldn't imagine anything it would do that would be useful to me so I didn't even install it
[23:14:25] <dstufft> I *think* it just makes it easy to create a virtual environment with a specific version of Python (like, down to the X.Y.Z specific)
[23:14:55] <dstufft> but I care about doing that like, once in a blue moon, and when I do, mkvirtualenv -p ~/.pyenv/versions/2.7.9 isn't that hard
[23:16:13] <tos9> (I have the sha saved so that in every new machine I use the commit that has them.)
[23:17:03] <dstufft> I'm also kind of wierd in that being able to install random point releases is something I use on a not uncommon (but not regular) basis
[23:18:00] <prologic> actually dstufft my particular scenario is kind of different re /var/lib/pip permissions -- see https://gist.github.com/therealprologic/929fcd8631e978e682f8
[23:18:33] <tdsmith> i think the problem was that python support in core was a moving target for a while and keeping the pythons from interfering with each other was tricky and nobody was really motivated to keep the versions tap current. i'd be delighted if a homebrew-deadsnakes tap appeared though
[23:31:11] <tos9> tdsmith: yeah I remember discussing it at the time -- there definitely was some justification to remove it, but I find it pretty frustrating ultimately how homebrew's update model has basically stayed git, because what happens is just that 3 months later a thing that was there isn't, and now you hunt it down with git to figure out why :P
[23:32:02] <tos9> a tap with the formulas put back is a good idea, I guess I should probably just put those back somewhere, if someone hasn't already
[23:34:23] <dstufft> prologic: I'm actually kind of out of it because I'm high on pain killers, I sprained my ankle this week and I started walking without a cane today so it's hurting again (but at least i can do it!) and i'm back on the pain killers. I'm having a little bit of trouble following what your gist is showing me
[23:36:01] <dstufft> thanks :) I just didn't want to sound like a pill head with my "I'm a little out of it because high on pain killers" thing heh