Log file Viewer
#pypa logs for Sunday the 15th of February, 2015
[12:00:40] <mitsuhiko> the way this thing currently works with local paths seems not to scale well at all
[12:40:25] <mitsuhiko> i have 120 wheels in this wheelhouse and it spends about 1/3rd of my build time scanning some crap
[12:42:00] <mitsuhiko> ionelmc: also, i plan on having about 1000 wheels in this cache, so i am getting a bit concerned
[12:42:23] <ionelmc> mitsuhiko: `time pip wheel lazy-object-proxy` is 0.95s user 0.10s system 62% cpu 1.673 total
[18:18:24] <gchristensen> Hi, the password change reuqest email sent from `python setup.py register` had a link with a URL with the "httpss" scheme, ie: httpss://pypi.python.org/. that is all. Cheers!
[19:31:41] <mitsuhiko> i have so much extra boilerplate code all the sudden to make dev\d+ to be the same as dev
[19:34:00] <dstufft> mitsuhiko: so your problem isn't with PEP 440 then, it's with the fact that setuptools auto normalizes
[19:38:12] <ronny> mitsuhiko: hg and git tell you how many commits a tag is away from the current rev
[19:38:13] <mitsuhiko> (which now does not work well because all our tags are different than the version numbers that setuptools generates \o/)
[19:39:23] <dstufft> mitsuhiko: fundamentally there's no way to order a bunch of git commits without having git around to check what the order should be or unless you limit what you count as a reasonable version, like "from the master branch"
[19:41:09] <mitsuhiko> in any case this "tiny" change of pep440 made me spend about three or four days sifting through replacing our whole build infrastructure.
[19:42:45] <dstufft> there's nothing stopping you from using an invalid version and using them with ==
[19:43:10] <dstufft> (or in PEP 440 parlance, ===, but the way we implemented it == with an invalid version will use legacy matching)
[19:44:19] <dstufft> if you don't use a version we can't parse as PEP 440, you'll have to use == or === to install it, that's how it is today in pip 6
[19:45:10] <dstufft> you also have to refer to it inv ersion constraints as unconstrained or == or === because if we can't order it, we don't know if it's >=1.0 (for instance)
[19:47:58] <mitsuhiko> dstufft: to be honest. the reason i'm so unhappy currently is because of pep440 and not so much the infrastructure around it
[19:48:06] <mitsuhiko> if you already are going to force people on a versioning system, why not adopt semver?
[19:48:46] <dstufft> PEP 440 is almost 100% compatible with semver the only thing you can't really do is arbitrary pre-release names, we restrict what pre-release names you can use
[19:49:15] <dstufft> or rather, there is a common subset that contains almost all of semver sans that bit
[19:50:34] <mitsuhiko> dstufft: if i want to do anything with pep 440 strings, which api am i supposed to use btw?
[19:52:26] <dstufft> the reason we couldn't force everyone onto semver is that would be a massively breaking change for a ton of projects on PyPI. For instance anything that uses a date based version number. Semver forces a pretty major set of semantics around what version numbers mean, but we have a lot of versions which simply don't follow that. So PEP 440 attempts to take how versions were already sorted in Python-land and restrict it to remove the arbitrary bits
[19:52:26] <dstufft> (like versions like "dog") but without requiring major changes to how people are versioning things, and we got pretty close. We can successfully parse something like 98 or 99% of versions on PyPI which is more than we could with semver
[19:53:03] <dstufft> mitsuhiko: well pkg_resources.parse_version should still work, but under the covers it uses the packaging lib -> https://packaging.pypa.io/en/latest/
[19:53:53] <mitsuhiko> not breaking compat would have been to run both schemes in parallel with different ways to anchor the versions
[19:54:03] <dstufft> mitsuhiko: From what you've described the thing you've spent days fixing up is because setuptools normalizes not because of PEP 440
[19:54:42] <mitsuhiko> dstufft: it's also because different versions of wheel/setuptools have different behavior if mixed
[19:55:03] <dstufft> if setuptools didn't normalize and you had a version like "1.0-dev" you wouldn't have even noticed, all the tooling would have just internally converted it to "1.0.dev0"
[19:55:14] <mitsuhiko> and because depending on if we were using a dev release or a rc we got into either legacy version or version which caused all kinds of funky behavior
[19:56:32] <dstufft> mitsuhiko: so like I said earlier, you have to order them somehow to do that, and how do you order c1c638 and f0a599
[19:57:19] <mitsuhiko> or you just always install a version with a prerelease tag unless it matches the currently installed one
[19:57:53] <dstufft> You have 1.0-dev-f0a599 and 1.0.-dev-c1c638 available, and you do ``pip install foo``, which one do you install?
[20:00:28] <dstufft> Right, I didn't say it didn't break, but the stuff that broke was places where you had to rely on arbitrary ordering of things that conceptually we had no way to compare versions that contained them. Switching to semver would break things that conceptually would be entirely possible to order but we couldn't just because we adopted semver instead of PEP 440
[20:01:10] <dstufft> and in addition a bunch of other people with versions like "1.0" would break because semver mandates X.Y.Z
[20:02:39] <dstufft> we don't have a reasonable way of doing that, we only have the filename available, and how do you order oldversion:1.0 and semver:1.1.0?
[20:03:06] <dstufft> unless you want to force everyone to pull down every version off PyPI and re-upload them using the semver opt in character
[20:04:36] <dstufft> mitsuhiko: no I mean what do you do if a project has versions available which are both not semver (because they were uploaded before we supported semver) and which are semver
[20:05:29] <mitsuhiko> and in fact, i remember very clearly that part of this issue was already there with pytz and other libraries before
[20:05:44] <mitsuhiko> so yeah, if you only look at part two of the madness which was now then the situation is already a bit disarmed
[20:06:17] <dstufft> mitsuhiko: we do have the same problem now yes, the difference being almost every version available on PyPI (including pytz since 2013 or so) can be parsed as a valid PEP 440 version, instead of with semver wihch a signifcant number of versions cannot be parsed as
[20:08:10] <dstufft> so the problem now is only with versions we can't parse as PEP 440, instead of every version that someone didn't mark as semver (which would include several hundred thousand that exist on PyPI currently)
[20:09:31] <mitsuhiko> right now you do not know if a version is pep440ver or just happens to be pep440ver though
[20:10:42] <dstufft> sure, but the semantics of PEP 440 and what setuptools used to use is almost entirely the same, if I recall the only real difference is in the handling of local versions and the epoch versions, which there are like... 10 versions on PyPI that use the local version character and no versions that use the epoch character
[20:11:10] <dstufft> (in fact part of what we tried to ensure was that the difference in how we sorted a PEP 440 version and how legacy setuptools sorted it were the same)
[20:12:52] <dstufft> a big part of why we went with PEP 440 was that it's basically "take the setuptools behavior, instead of accepting arbitrary characters restrict it down to a sane subset, and explicit define it in a PEP" instead of the rule being "whatever setuptools implements, or something like it"
[20:14:49] <mitsuhiko> right now i need to build two different things for two different server environments because i didn't have the time so switch everything over
[20:16:07] <mitsuhiko> but yeah, given that this is going to be the norm forward from what i can tell i pretty much gave up disregarding setuptools and pip versions now
[20:16:59] <dstufft> from what i can tell most people either had no problems or were able to adjust with minimal impact, the most common change being to switch it so that their things generate a version with a +<hash> instead of a -<hash> or .<hash>. You're honestly the first person whose come to me or that i've seen that has had any sort of major problems due to PEP 440
[20:17:43] <dstufft> well if you go back to the pre-release change in pip 1.4 more people had problems with that if you count that as part of PEP 440
[20:24:31] <dstufft> Well wheel certainly isn't included in that since wheel hasn't had a release in a few months shy of a year. between pip and setuptools I'd be very surprised if (every since we started vendoring pkg_resources) there was much requirement to be synced. There may be some edge cases if you're using a setuptools older than 8 with pip 6+ but that would only be in cases where you're using pkg_resources to enforce available versions at runtime (such as
[20:24:31] <dstufft> entry_points when installed from sdist) and you hit an edge case where pip 6+ and setuptools <8 have different opinions about what is contained within a version specifier
[20:26:07] <dstufft> (A lot of people are far more likely to upgrade pip than setuptools because their system installs setuptools for them via system package mananger in a lot of cases, but they are likely to install pip from PypI)
[20:26:22] <dstufft> though it's certainly not a case we go out of our way to test very well (largely because of lack of resources)