Log file Viewer
#pypa logs for Thursday the 3rd of April, 2014
[05:50:09] <glyph> dstufft: why do I need both wheel-dir and find-links? doesn't wheel-dir sort of imply find-links? also isn't there a *default* wheel-dir?
[05:52:34] <glyph> dstufft: but until then, sometimes I ssh into a server and need to do stuff and I don't want to write a pip.conf, I just want to remember the right command line ;)
[05:53:05] <glyph> dstufft: also I'm going to give a talk at pycon where I berate you all and mostly I just want to berate you accurately ;-)
[05:53:25] <dstufft> https://github.com/pypa/pip/issues/878 is the issue and https://github.com/pypa/pip/pull/1572 is a WIP implemenation
[06:04:15] <Alex_Gaynor> glyph: Sounds like you have a thing to do, I can review a PR for the builders repo.
[06:04:22] <glyph> Alex_Gaynor: oh jeez and I can't update them because I have an email that has some encrypted credentials I need to install except they're encrypted to my GPG key except my GPG key is on a different computer
[06:06:03] <glyph> I should be putting this into a fabric file, I am _already_ regretting not recording these steps programatically
[06:06:50] <glyph> Alex_Gaynor: cyli has now been fighting with getting coreos working under vagrant for 3x as long as I've been fighting with this
[06:08:44] <glyph> Successfully installed Twisted TxSNI cffi cryptography pyOpenSSL pycparser six wheel zope.interface
[06:10:51] <dstufft> I hope we're trending towards works at least, or something vaguely resembling works :/
[06:12:23] <Alex_Gaynor> dstufft: cryptography, can we upoad OS X wheels, and are we likely to hit issues with them?
[06:13:47] <glyph> dstufft: your comment on that twisted ticket is appreciated, but could you phrase it slightly more constructively?
[06:13:59] <glyph> dstufft: in other words, "here's what you do to avoid the implicit compile in your setup.py"
[06:14:35] <dstufft> Alex_Gaynor: so technically you can upload them, likely to hit issues is hard to predict. Probably the biggest thing is going to be the fact you have system Python, Python.org Python, Homebrew Python, MacPorts Python, and pyenv Python
[06:15:23] <dstufft> I haven't heard any major complaints, which means either people aren't doing it, or it's working
[06:30:44] <Ivoz> qwcode: yeah, py27->py2 by default for pure packages slipped in quietly thanks to pf_moore :)
[06:41:14] <dstufft> glyph: sorry my original thing was more ranty than useful. Sometimes the brain pain leaks out :)
[06:54:23] <glyph> dstufft: have you considered making a 'shenanigans' package that just contains what cryptography is doing to deal with all these workarounds?
[06:55:43] <glyph> dstufft: because as I was reading your post, I got the distinct feeling that I don't want to do those things
[06:56:02] <glyph> dstufft: oh also, I told the tahoe folks something which I'm sure is going to get you in trouble
[06:56:23] <glyph> dstufft: can you use the ctypes cffi backend to cheat at runtime and not do compilation if the user doesn't have a C compiler and also doesn't have binary packages?
[06:56:32] <Alex_Gaynor> dstufft: maybe we should just contribute to cffi 1.0 instead of developing so many workarounds :/
[06:57:16] <dstufft> glyph: I don't know if you can do that or not, I guess you can in some cases, I doubt it works in every case though
[06:57:50] <glyph> dstufft: if you have really exciting things going on in your verify section I would imagine not
[06:58:00] <dstufft> yea like https://github.com/pyca/cryptography/blob/master/cryptography/hazmat/primitives/padding.py#L31-L67
[06:59:53] <dstufft> Alex_Gaynor: I don't have enough time or motivation to tackle fixing those things upstream right now :( It's quicker in the moment to just tell folks about the work arounds. I think it would be far better to actually get those things fixed upstream though
[07:01:16] <dstufft> I finally got around to making a nice little cookiecutter thing so I can bootstrap new projects pretty quickly at least
[07:02:18] <dstufft> where help means I tell them how terrible everything is and maybe provide insight into making things work more
[07:04:23] <dstufft> oh I can delete https://github.com/pyca/pynacl/blob/master/src/nacl/_cffi_fix.py now
[07:05:14] <dstufft> glyph: Alex_Gaynor tomorrow i'll at least file bugs with cffi for all the gotchas I'm aware of
[15:51:00] <e0d> I'm building a set of wheels for our custom packages and depends that I would like to subsequently install using something like: pip install --use-wheel --no-index --find-links=https://wheelhouse.example.com/ pyramid
[15:54:06] <pf_moore> and https://wheelhouse.example.com/ returns a page containing links to archives?
[15:55:40] <pf_moore> as far as I know that's all you need. Not really a wheel question as such, this should be basic pip finder functionality.
[15:56:55] <pf_moore> Might be worth trying plain http just to see if that works (although you may hit fun with the new --allow-insecure stuff then, so that may just open a whole other can of worms :-()
[15:57:39] <e0d> They currently look like, say, <a href="amqp-1.0.13-py27-none-any.whl">amqp-1.0.13-py27-none-any.whl</a>
[15:58:12] <pf_moore> Looks like I'd expect. And clicking on that link in a browser downloads the file?
[15:59:19] <pf_moore> It *might* be related to the new security stuff, but I know next to nothing about that - dstufft may be able to comment if he's around
[16:01:07] <dstufft> plain http should work, but is inadvisible, nothing else should be affecting that though. you might try doing ./ampq-.....whl instead of just ampq-....whl
[16:03:07] <e0d> pip install --use-wheel --no-index --allow-insecure --find-links=https://s3.amazonaws.com/edx-wheel-test/index.html amqp
[16:04:08] <dstufft> if I go to https://s3.amazonaws.com/edx-wheel-test/index.html I see an index where it links ampq
[16:06:43] <e0d> That's helpful, let me see how far that gets me. I was suspicious of the index doc format, but sounds like that looks OK.
[16:08:06] <dstufft> if you're going to duplicate the PyPI scheme of BASEPATH, BASEPATH/project/ where BASEPATH has links to the BASEPATH/project/ urls, and BASEPATH/project/ has links to the file, you'll want to use --index-url instead of --no-index --find-links
[16:08:07] <e0d> but I assume it doesn't look for index.html by default if the server doesn't serve a default index doc.
[16:08:45] <dstufft> for --find-links it's going to go to exatly the url you gave it, and look on that page for links that point to filenames that look downloadable
[16:09:14] <dstufft> for --index-url it's going to go to the url you specify + /thingyouwanttoinstall/ and if that 404's it'll fall back to look at the url you specify
[16:10:19] <dstufft> http://www.pip-installer.org/en/latest/reference/pip_install.html#finding-packages has two links that describes the --index-url API
[16:10:38] <dstufft> but you can just make one big index.html that links directly to files and just point to that with ---find-links too
[16:10:52] <dstufft> also you can see exactly what urls pip is trying out by adding -vvv to your command
[17:07:21] <e0d> pip install --use-wheel --no-index --allow-insecure --find-links=http://edx-wheel-test.s3-website-us-east-1.amazonaws.com watchdog-0.7.1-py2-none-any.whl
[17:10:03] <e0d> I suppose moving to PyPi format will fix this, but would be nice to be able to do it from a single index.
[18:22:33] <e0d> pip install --use-wheel --no-index --find-links=http://edx-wheel-test.s3-website-us-east-1.amazonaws.com scipy==0.11
[21:22:09] <barry> yeah, except when i create a venv with pyvenv-3.4 /tmp/gg i don't get any pip scripts in /tmp/gg/bin
[21:22:11] <jezdez> barry: dstufft: please move the discussion of the development of a pypa tool into #pypa-dev as we're trying to have separate user and development channels
[21:25:12] <barry> and yet, when i build py34 from source and install that into /usr/local, and then run *its* pyvenv, i get pip and easy_install all living in /tmp/gg/bin right along side python3 as expected
[21:25:31] <jezdez> tomprince: this discussion discusses development topics that aren't interesting for end users
[21:25:43] <barry> dstufft: so that confuses me and i haven't quite been able to trace my way through all the layers to figure out why this is happening
[21:26:22] <barry> dstufft: maybe, but i'm not atm been able to find a patch that changes pyvenv's behavior here
[21:27:08] <barry> right, but if no system pip is installed, won't pyvenv use the bundled pip wheel?
[21:27:41] <tomprince> debian change the paths where python thinks stuff should be installed relative to its prefix.
[21:28:44] <tomprince> Outside of a venv, prefix=/usr and scripts are installed in /usr/local/bin. In a venv, prefix=<venv dir> and scripts are installed in <venv dir>/local/bin
[21:32:05] <barry> let me ask this slightly differently then. pip uses which variables to decide where to install things?
[21:32:36] <barry> (and i suppose installing into /whatever/local/bin wouldn't be so bad if activate added that path to $PATH)
[21:32:53] <tomprince> https://github.com/pypa/virtualenv/blob/develop/virtualenv.py#L1521 might be relevant?
[21:36:19] <tomprince> I'm not sure how pyvenv works, but I suspect that https://github.com/pypa/virtualenv/blob/develop/virtualenv_embedded/site.py may have been adjusted to deal with debians patches but pyvenv not.
[21:37:19] <barry> dstufft: my package changes are not. are you able to build debian/ubuntu packages? if not, i can put some .debs up somewhere
[21:37:54] <barry> maybe i should just hack the activate script to include $root/local/bin and be done with it. :/
[21:38:27] <dstufft> but if you can throw up some .deb and tell me which OS I need I can take a poke
[21:41:20] <barry> wget that, unpack it and you'll have .debs that should be installable on ubuntu 14.04
[21:45:35] <barry> cool. dstufft one small hitch i'm afraid. i'm a bad person and i started this conversation before realizing i have to leave in 15m :( i'll be back later tonight and tomorrow. can we perhaps continue again then? (in #pypa-dev i promise)
[21:46:19] <dstufft> barry: yea that's fine too, tonight infusion night for Alyssa (my daughter) so my time is limited and distracted too until that's done :) though atm i'm waiting for her to get ready
[21:47:57] <barry> yeah. here at canonical it's a job requirement to be able to do dst&utc math in your head :)
[21:48:44] <barry> dstufft: one last thing before i disappear. would it be totally insane to just add $root/local/bin to activate's $PATH?
[21:49:40] <dstufft> barry: well it'd work for some defintiion of work, but I think that'l cause headaches, people invoke from the bin path all the time
[21:49:43] <tomprince> I often try to run things via <venv>/bin/<whatever> rather than than activating. (Particularly from CI systems.
[21:57:15] <barry> f.e. gives you the source branch for python3.4 in trusty. then debian/patches has all the patches that are applied
[21:59:12] <tomprince> Looking at http://patch-tracker.debian.org/patch/series/view/python2.7/2.7.6-8/distutils-install-layout.diff I bet ensurepip is running outside of the created venv, and so sys.prefix is /usr and so the unix_local scheme is used.
[21:59:37] <tomprince> But once in the enviroment, sys.prefix != /usr and so the a differrent scheme is used.
[22:01:28] <tomprince> That is, the patches to sysconfig are smart enough to add /local/ to paths only when using the system python outside a virtualenv, but ensurepip is running there.
[22:04:20] <dstufft> I wonder if this is happening before pyvenv drops the config file that makes pyvenv look in the right paths
[22:12:39] <barry> but now i really do have to go. dstufft, tomprince thanks, and i'll catch up with you tomorrow
[22:22:05] <qwcode> dstufft, btw, what I'm seeing on debian with pyvenv is the pip script showing in <venv>/local/bin/pip, but not in <venv>/bin/pip
[22:23:58] <qwcode> dstufft, oh, ok, wasn't following along earlier. I gotta run, but will look later since somewhat familar with debian.