Log file Viewer

[02:25:13] <BlueAidan> so it appears that pip 8.1.2 normalizes namespace package names differently than previously, leading us to be unable to install them

[02:25:21] <BlueAidan> <= 8.1.1 works fine

[02:25:39] <BlueAidan> it normalizes foo.bar to foo-bar when scanning indices

[02:26:01] <BlueAidan> I didn't see a mention in the changelog about changing this

[02:27:40] <dstufft> BlueAidan: from PyPI?

[02:29:39] <BlueAidan> dstufft: we use https://pypi.python.org/pypi/pypiserver for hosting internal packages

[02:30:12] <BlueAidan> let me find a namespace package to install from the public pypi

[02:31:38] <dstufft> BlueAidan: So it sounds like pypiserver doesn't correctly implement https://www.python.org/dev/peps/pep-0503/

[02:32:23] <BlueAidan> okay, I can buy that; just surprised it broke in a dot release and there's nothing in the changelog about it

[02:33:23] <BlueAidan> works as expected on 8.1.1

[02:34:25] <BlueAidan> dstufft: do you have a recommendation for a simple server implementation? I'm not opposed to changing what we have. Just seemed the lightest weight option at the time

[02:34:43] <dstufft> (which is understandable, previously to PEP 503 what the /simple/ API was terribly unspecified and unless you matched the name _exactly_ it may or may not work in a bunch of different scenarios that weren't PyPI, PEP 503 attempts to remedy that, but it means some things broke until they get updated)

[02:35:12] <dstufft> BlueAidan: Honestly, I'm kind of surprised it worked on 8.1.1

[02:35:18] <dstufft> I Thought we broke that ages ago heh

[02:35:43] <BlueAidan> it's worked on every release since like last october/november I'm pretty sure

[02:39:37] <dstufft> Yea, it looks like we "broke" it in 8.0 actually, but there was an edge case specific to . that didn't get broken... until 8.1.2 when we pulled in a new copy of pkg_resources from setuptools which I guess started normalizing the names by default

[02:40:57] <dstufft> so the reason it's not in the changelog, is it was actually a setuptools change and I just didn't notice

[02:41:06] <dstufft> BlueAidan: I've had luck with devpi in the past fwiw

[02:41:38] <BlueAidan> I was hoping to not have something that tries to cache the public pypi; I've had bad luck with those in the past caching things incorrectly

[02:42:30] <BlueAidan> I hear ya; this is a thankless task. Python packaging is a minefield.

[02:43:29] <dstufft> devpi is actually a multi index thing

[02:43:40] <dstufft> it has one repository endpoint that cached PyPI

[02:43:50] <dstufft> but you can add as many other endpoints as you want that don't have that

[02:44:09] <dstufft> (also your endpoints can inherent frome ach other, including inheriting from the pypi mirror)

[02:46:34] <dstufft> BlueAidan: depending on what features you need, a web server and a file system is enough fwiw

[02:47:19] <dstufft> just make a dir for each project, named how the code in PEP 503 says to name it, stick those packages inside of there, and autoindex your server at that dir

[02:50:30] <BlueAidan> yeah; just wasn't expecting this particular rabbit hole tonight... was testing standing up an entire environment and it failed spectacularly. ;)

[02:51:43] <dstufft> yea sorry :( I suspect if you open an issue on pypiserver they'll probably be willing to fix it too :]

[02:52:25] <BlueAidan> yeah; I imagine it's a pretty simple change

[02:52:42] <BlueAidan> guessing setuptools exposes the normalizing function?

[02:53:10] <dstufft> um

[02:53:12] <dstufft> I believe so

[02:53:17] <BlueAidan> I'll dig

[02:54:17] <dstufft> If not, packaging does

[02:54:18] <dstufft> http://packaging.readthedocs.io/en/latest/utils/

[02:56:25] <BlueAidan> okay

[02:57:21] <BlueAidan> fwiw, pkg_resources.safe_name doesn't seem to do the trick

[03:00:43] <BlueAidan> sigh, it appears pypiserver is doing a lot of stuff on it's own instead of using the proper api's

[03:00:48] <BlueAidan> https://github.com/pypiserver/pypiserver/blob/master/pypiserver/core.py

[03:37:23] <njs> dstufft: so uh btw maybe should have mentioend before that ruamel.yaml is also a fork of pyyaml that has already moved to single-source py2/py3

[03:37:35] <njs> dstufft: in case you get anyone else asking you to maintain such a thing :-)

[07:54:36] <ronny_> dstufft: ouch, that bugfix in distlib is a breaking behaviour change that warants a major release in semver ^^

[09:18:56] <MarkusH> We just ran into https://github.com/pypa/pip/issues/3666 -- From what I understand this is a backwards incompatibility change that was introduced in a patch release?

[15:28:09] <sontek> Hey, what changed in pip 8.1.2 that would make some packages not be found anymore?

[15:28:10] <sontek> http://paste.openstack.org/show/496755/

[15:28:54] <dstufft> sontek: upgrade yo devpi

[15:29:04] <sontek> :(

[15:29:06] <dstufft> https://github.com/pypa/pip/issues/3666

[15:29:36] <sontek> Did we miss a deprecation period or something?

[15:30:07] <sontek> Unscheduled infrastructure work is a bummer

[15:30:16] <sontek> I can upgrade devpi but we usually test our upgrades and shit before we do them

[15:31:09] <dstufft> sontek: The tl;dr of what happened is there was a deprecation warning for the overall normalization changes in pip 6, than went to a error in pip 8, BUT due to a bug that nobody noticed those warnings didn't apply to ``.`` only all the other characters

[15:31:21] <dstufft> so ``.`` kept working accidently

[15:31:54] <dstufft> until we pulled in an updated setuptools and packaging to fix issues with markers and projects being uninstallable depending on what markers they used, and those libs had been correctly implemented

[15:32:06] <dstufft> so they "fixed" the bug

[15:32:08] <sontek> and the decision is to hold people to the change instead of backing out the change until proper deprecation warnings are out?

[15:34:09] <dstufft> sontek: the issue is that either way people are broken, we need the newer libs to fix some projects on PyPI that are completely uninstallable right now, but if we pull in the newer libs we break the ``.`` for repositoreis that didn't correctly implement PEP 503. The decision is to keep the updated libs because those projects on PyPI don't have much of an option to unbreak themselves besides wait for pip, but peolpe affected by this can upgarade

[15:34:09] <dstufft> (or patch their server if it doens't have a new release) to fix it

[15:34:59] <dstufft> so basically, which bad choice do we want to make :(

[15:37:32] <sontek> Yeah, but the existing breakage is at least already known broken things

[15:37:51] <sontek> Waking up to 100 developers not being able to run their tests in our CI system is shitty

[15:38:06] <sontek> I'd rather have things that were broke before to continue being broken

[15:38:09] <sontek> Rather than new things

[15:43:31] <dstufft> sontek: I'd rather have broken things that can be fixed quicker, particularly when that broken thing is broken because it was wrong and where other people can control the fix than I would broken things that are blocked on a deprecation process... but at the end of the day neither one is wrong or right, it's just trade offs.

[15:43:37] <dstufft> It sucks though :/

[15:44:20] <dstufft> (alternatively, you can pin to pip 8.1.1 until you have time to upgrade your devpi)

[15:47:56] <sontek> Yeah, I just can't prevent all our devs from upgrading their pip, I can only prevent our CI system from doing it

[15:48:00] <sontek> so its still fucked

[15:48:09] <sontek> I'm doing the devpi upgrade, just hoping it doesn't break other stuff =)

[16:27:58] <sontek> and latest devpi throws errors :)

[16:28:02] <sontek> That was a fun upgrade

[16:29:10] <xafer> sontek, for what is worth, you can also upload pip-8.1.1 to your devpi: this will shadow PyPI pip 8.1.2 version

[16:30:46] <sontek> dstufft: looks like latest devpi doesn't work with pypi either: http://paste.openstack.org/show/496774/

[16:30:49] <sontek> Debugging now though

[16:31:14] <sontek> OH!, I pulled bad versions

[16:31:20] <sontek> because pypi doesn't sort latest

[18:35:11] <WesleySnipes> hello pipsters

[21:15:14] <sontek> xafer: It seems like tox downloads its own version of pip no matter what you do which is our biggest issue right now

[21:15:22] <sontek> xafer: so you can't really avoid getting the broken version

[21:15:25] <sontek> http://paste.openstack.org/show/496824/

[21:18:23] <sontek> Everyone in our company as resulted to updating tox.ini to do `commands= pip install pip==1.8.1` before their other commands

[21:18:40] <sontek> putting it in `deps=` doesn't work

[21:19:27] <xafer> In my company, I uploaded pip==8.1.1 to our devpi which effectively shadowed pip==8.1.2 for the day

[21:20:10] <sontek> ahh, so you just killed it so it can't find 8.1.2

[21:20:12] <xafer> and I upgraded devpi-common to 2.0.9 (with devpi-server==2.6.1) with effectively solved the issue with 8.1.2

[21:20:12] <sontek> that makes sense

[21:20:56] <xafer> :)

[21:23:17] <xafer> (and of course I removed the uploaded pip==8.1.1 from devpi to unshadow PyPI)

[21:23:21] <sontek> We were just talking about re-uploading 8.1.1 as 10.0.0 :P

[21:24:12] <xafer> could also work but I'd suggest to use 8.1.2.1 instead ^^

[21:25:19] <sontek> Well, we'd hate for an 8.3-9.9 to come out and break us again

[21:25:31] <sontek> version 99999.99.99 has also been suggested

[21:28:10] <xafer> ^^

[22:11:28] <MarkusH> sontek: put pip==8.1.1 as first line in your tox deps

[22:11:38] <MarkusH> deps =

[22:11:45] <MarkusH> pip==8.1.1

[22:51:04] <dstufft> sontek: MarkusH xafer or just do uhh, I think it's export VIRTUALENV_DOWNLOAD=no

[22:51:30] <dstufft> which will make it so tox virtualenv doesn't download an updated pip when it creates a new virtualenv

[23:47:47] <ianw> INTERNALERROR> AttributeError: 'PyCodeCheckItem' object has no attribute 'out'

[23:47:59] <ianw> ^ is this familiar to anyone? running tox with bandersnatch

[23:50:11] <dstufft> ianw: yea I was getting that too

[23:50:13] <dstufft> only on py27

[23:50:16] <dstufft> I don't know what it means

[23:50:34] <dstufft> I just ingored it and only ran the py 2.6 tests ¯\_(ツ)_/¯

[23:52:53] <ianw> dstufft: hmm, upstream looks different to my local code

[23:52:54] <ianw> https://bitbucket.org/RonnyPfannschmidt/pytest-codecheckers/src/ffa881a66ef64576247bbdd2895b3b737e99469f/codecheckers/plugin.py?at=default&fileviewer=file-view-default#plugin.py-28

[23:52:59] <ianw> maybe they fixed it

[23:55:25] <ianw> yep, a little patching and that works ... looks like they need a release