Log file Viewer

[09:09:57] <p0zt3r> hello there! could anyone help me with pip, SSL certs and my own index?

[09:11:50] <p0zt3r> I use distutils to build my package, which depends from another package installed into my own index. When I try to just install that another package using pip install, everything works fine. When I install main package, depending on another one, pip fails: Could not find any downloads that satisfy the requirement, Some externally hosted files were ignored (use --allow-external redis-cache to allow).

[09:57:58] <p0zt3r> downgraded to pip==1.4.1 and everything works fine. Guys, who can help? I can't easily find changes dedicated to Authorization in current version 1.5.x of pip

[10:12:37] <pf_moore> p0zt3r: You probably need pip install redis-cache --allow-external redis-cache --allow-unverified redis-cache

[10:13:27] <p0zt3r> but I have a lot of same packages. am i supposed to define them all in command line?

[10:14:46] <pf_moore> p0z3tr: Yes :-( That's the incantation to get things moving for you, but it'll need someone who understands ssl and the pip security model etc better than I to help you set things up so you don't need to type that. I think serving your index via ssl and including hashes in the links should work, but I don't know if there's a simpler approach for an index.

[10:15:18] <pf_moore> Of course if you don't need a full index, just having a page with links to all the package files and using --find-links should be simpler

[10:16:09] <pf_moore> Someone like dstufft can probably give you a better answer, if he's around...

[16:16:55] <p0zt3r> the problem was solved downgrading pip to 1.4.1 =( additionally you will need to add into pip.conf "cert = /path/to/certificates"

[16:17:32] <p0zt3r> the problem is in pip/download.py I see and I will try to propose best solution if will have free minute

[16:17:41] <p0zt3r> anyway thanks for help!

[17:07:32] <Ivo> p0zt3r: I actually made a small self-contained pip index to solve this https://pypi.python.org/pypi/pipa/0.1.0

[17:08:28] <Ivo> p0zt3r: the 'problem' being having a private index that A) provides "hash-protected" links and B) serves itself over SSL from the get-go

[17:09:53] <Ivo> p0zt3r: alternately, maybe the index you are using is not providing hash-protected links to pip

[19:57:13] <miketheman> !logs

[19:57:13] <pmxbot> http://chat-logs.dcpython.org/channel/pypa

[19:59:43] <miketheman> so….. logs show nothing along these lines.

[19:59:43] <miketheman> but

[20:00:54] <miketheman> using recent versions of setuptools and pip and wheel, on python 2.7, I randomly get a "yourpackage is in an unsupported or invalid wheel"

[20:01:25] <miketheman> This is during a jenkins deploy, where some number of nodes are pulling the same .whl file - and running it a second time passes flawlessly

[20:12:44] <Ivo> miketheman: could you paste the exact output? May be some kind of corruption or race condition

[20:13:30] <miketheman> Ivo: sure thing, compiling output now

[20:14:43] <miketheman> I pull in a buttload of dependencies, so I'm going to truncate a lot of that (for security reasons, too)

[20:20:11] <Ivo> miketheman: specifically that error means the wheel is corrupt in some way, or something else has gone wrong reading or generating it https://github.com/pypa/pip/blob/develop/pip/wheel.py#L408-L443

[20:25:51] <miketheman> Ivo: I read through that code a few times already - and can't come up with a reasoning why it would succeed on 119 nodes, fail on 1, and then pass on that 1. Then, succeed on 117 nodes, fail on 3 _other_ nodes.

[20:25:55] <miketheman> no rhyme or reason

[20:26:11] <miketheman> some more output here would be helpful

[20:26:12] <miketheman> https://github.com/pypa/pip/blob/develop/pip/wheel.py#L423-L424

[20:26:54] <miketheman> I'm guessing if it was an invalid wheel, it should fail consistently everywhere

[20:27:11] <miketheman> it's the inconsistency that has me worried

[20:28:57] <Ivo> miketheman: my first guess is a race condition somehow corrupting data

[20:29:51] <Ivo> miketheman: if that code fails somehow then it essentially means the files of the zip we've extracted aren't in a valid wheel format

[20:30:51] <Ivo> or can't be read currently

[20:31:58] <Ivo> miketheman: I would collect the wheel thats throwing that and see if it consistently can't be installed

[20:32:29] <miketheman> Ivo: https://dpaste.de/xMxa

[20:32:44] <miketheman> it's a custom wheel, our CI server builds it, and then 120 nodes install it.

[20:32:55] <miketheman> the failures are random, inconsistent and always with this message

[20:33:27] <miketheman> Ivo: if pip is trying to install a half-downloaded file, that's another problem?

[20:34:33] <Ivo> miketheman: that check should essentially occur on a folder of files resulting from a wheel thats been extracted (it's just a zip)

[20:35:20] <Ivo> so im thinking some sort of read error under load'

[20:36:48] <Ivo> miketheman: do you have to set --allow-insecure for your installs?

[20:37:47] <miketheman> Ivo: I don't

[20:38:30] <Ivo> reason i ask is if pip is installing from a pypi-like index then it should also be receving a hash of the wheel it downloads and verifying it before doing anything

[20:38:51] <miketheman> hm, where does that happen, and who generates the hash?

[20:38:57] <Ivo> the index

[20:39:36] <Ivo> miketheman: https://pypi.python.org/simple/boto/ see the actual url for each package

[20:39:58] <Ivo> contains an md5 that pip will check

[20:40:41] <miketheman> ohsnap, that's cool

[20:41:12] <miketheman> we've been using https://github.com/wolever/pip2pi to generate our pypi-like index

[20:41:14] <miketheman> is there a better (or simpler) way to do that?

[20:41:20] <Ivo> so if thats happening in your deployment you should be able to rule out a corrupt download...

[20:41:20] <miketheman> it also doesn't explain the random failure

[20:45:13] <Ivo> miketheman: that mirror doesn't include hashes in its links

[20:45:20] <miketheman> correct

[20:45:30] <miketheman> is there a better one?

[20:45:50] <Ivo> devpi is the most widely used

[20:47:41] <miketheman> Ivo: I'll look into that

[20:48:11] <miketheman> but repo and md5 hashes notwithstanding, why would pip try to unpack a failed download, or something like that?

[20:48:27] <Ivo> it could be a corrupted download

[20:48:35] <Ivo> or it could be something else

[20:49:00] <Ivo> executing the hash check code path would just rule that possible cause out

[20:53:12] <Ivo> I've made a very simple pypi-like server which I know does the right thing https://pypi.python.org/pypi/pipa

[20:58:55] <tomprince> miketheman: I'm not sure exactly how you are downloading the image, but do you have a shared cache that something is reading out of before a different process is done writting to?

[20:59:40] <Ivo> seems like some kind of race condition under load

[21:00:41] <miketheman> tomprince: the local index is being served by apache, as a simple directory

[21:00:45] <miketheman> I've turned up apache to debug to try and catch _anyhting_ out of the ordinary

[21:01:19] <tomprince> I mean, on the failing nodes, do you have a cache?

[21:15:12] <miketheman> tomprince: not that I know of? there's nothing settuing any cache settings

[21:15:42] <miketheman> tomprince: and there's no failure pattern - it works, then it doesn't then it fails on other nodes

[21:19:28] <miketheman> tomprince: would this failure occur if there's a running process using that code?

[21:30:54] <miketheman> Ivo: the client-side hashing is implemented? https://github.com/pypa/pip/issues/468

[21:33:22] <Ivo> maybe not, but server side is...

[21:33:29] <miketheman> heh, ok

[21:33:35] <Ivo> (ie when its retreiving from a pypi server)