Log file Viewer
#pypa logs for Monday the 17th of August, 2015
[18:16:19] <sontek> Probably the PEP440 changes if I had to guess, we were pinned at 1.5.4 for ~6 months because of those changes and it was weeks of work to make our CI system and all our tooling around it to work
[18:16:52] <sontek> It was significant effort to move beyond 1.5.4 and I doubt many people thought it was worth it. We didn't think it was worth it but did it anyways :P
[18:22:14] <sontek> Yeah, for us the biggest pain point was being able to do ranges of versions. We have ~70 different pyramid applications we deploy and 200+ shared libraries across them and pretty much all of them do things like: `requests<=1.2,>=2.3` where there was a bug in a few releases and so we wanted to exclude them
[18:22:34] <sontek> But "," became "AND" instead of "OR" and that pretty much broke everything in our company
[18:23:09] <sontek> and so releasing 200+ libraries across the company and breaking compatibility where we had to drop support for older versions because we couldn't say "OR" was a big problem
[18:23:28] <sontek> So it was weeks of work and months of time as we rolled out the changes one lib at a time
[18:24:41] <sontek> and I'm talking weeks of work at the DevOps/tooling level. It was more combined time for each application developer and QA person as they upgraded each of these libraries to verify that nothing broke because they were getting forced upgrades
[18:32:33] <sontek> Is there a way to tell pip to rebuild the wheels in its cache? Our deploy script had an ordering problem so the C dependencies got installed after the package ran pip. So now I'm getting errors like this: ImportError: The C extension needed to use libev was not found.
[18:32:53] <sontek> So now we have the C dependencies installed but pip sees it already has the wheel so it doesn't recompile
[18:34:06] <sontek> I see there is --no-cache and --no-use-wheel but I want cache and want the wheels, I just want to rebuild the wheel for that package with C deps
[19:02:37] <dstufft> sontek: well, to be clear it was "," became AND instead of sometimes AND and sometimes OR and sometimes something else
[19:03:25] <dstufft> sontek: there isn't a way to rebuild the wheels in the cache except to just delete the cache
[20:18:24] <sontek> dstufft: Yeah, thats fine. It worked for us and made complete sense how it worked and when it changed it didn't provide a replacement
[20:21:24] <dstufft> sontek: well, it didn't actually make sense to how it worked, I don't know how it fully worked, I tried to work through what the full implications of the code was and after 15 minutes I gave up, someone i know figured it out and couldn't articulate what the actual behavior of the comma was. It followed a common problem that a lot of setuptools had, where it had one really sane behavior, one saneish behavior, and one or more completely insane
[20:23:17] <sontek> Yeah, so lets say you want to say "We support versions less than 1.2 and greater than 2.3" you can't do that today
[20:23:57] <sontek> We couldn't come up with any reason to have AND but we were actively using OR in pretty much every project
[20:24:59] <dstufft> I went through a ton of projects on github and PyPI and looked how people used them
[20:26:27] <carljm> dstufft: I do a fair amount of patched local "releases", and I bundle sdists with projects, using --no-index and --find-links with a filesystem path.
[20:26:52] <carljm> so if I patch FooProject 1.1, I'd (in the past) make an sdist for FooProject 1.1.oddbird1
[20:27:08] <carljm> which worked fine, and still works fine actually, but violates PEP 440 (and I think recent pip's warn about that)
[20:27:30] <carljm> the PEP 440 way to do that, AFAICT, would be something like FooProject 1.1+oddbird1
[20:28:10] <carljm> And requirements files don't seem to like the + in the version either, it crashes pip with an error parsing the reqs file
[20:29:08] <carljm> If you put it into the requirements file with the + translated to a -, it finds the sdist ok and installs it, but then you get the issue where pip thinks every time you're installing a different version, because 1.1+oddbird1 doesn't match 1.1-oddbird1, so it reinstalls it every time
[20:29:38] <carljm> Not asking you to fix this, for the moment I'm just ignoring PEP 440, and it's on my list of things to look into and try to fix at some point :-)
[20:30:32] <dstufft> sontek: anyways, you can do that today, it's just not as clean, !=1.3.*,!=.1.4.*,!=1.5.*,...,!=2.2.*. So still possible, just more annoying.
[20:32:02] <sontek> dstufft: Yeah, I think the problem is when you are looking at opensource projects they just use a major version bump and move on with their lives or never break compatibility
[20:32:29] <sontek> I'm sure if you looked at the other big python shops you would have seen more of what we are doing
[20:33:44] <sontek> I had never actually used that syntax in the opensource world because if there was a bug in one of the dependencies we would just document it
[20:34:03] <sontek> but in the corporate world we needed to prevent those versions from being pulled in or our software would break
[20:34:23] <carljm> dstufft: Huh. I wonder if I had an older setuptools in the virtualenv where I generated the sdist. I just ran into this last week.
[20:35:00] <carljm> I'll check it out again. (I think the requirements-file-parsing issue would still be a problem, but maybe that can be solved with some quoting, I didn't have time to look into it very far.)
[20:36:55] <carljm> Maybe that accounts for both problems, then; I guess pip defers to setuptools for parsing requirement versions in a reqs file.
[20:37:50] <carljm> Hmm. I was pretty sure I had the latest pip, which should have covered the reqs-file-parsing then. But maybe I didn't.
[20:38:44] <dstufft> sontek: fwiw I'm not disagreeing people used it as an OR, I couldn't find _many_ cases of it though. Sadly I can only look at stuff for which I have access too and hope that if there are major use cases that aren't being covered someone else will be participating or watching and will chime it before the trigger is pulled
[20:39:06] <dstufft> some of the ones I did find, I had a hard time figuring out what versions they actually wanted installed
[20:42:06] <sontek> dstufft: Our big use cases were for things like python-dateutil that had different versions for py2 and py3 for a bit and then merged it together
[20:42:28] <sontek> We never wanted the python3 only versions, so we said we only supported the python2 only or the combined source
[20:47:29] <sontek> The other things were like versions of packages that had bugs in them and we just wanted to skip them