Log file Viewer

[14:06:03] <dstufft> jaraco: ping

[14:06:38] <jaraco> hi dstufft. I'm just kicking off my Tuesday morning meeting spree, so I'll be indisposed for the next three hours or so.

[14:06:54] <jaraco> Can we chat later this afternoon or evening?

[14:07:01] <dstufft> jaraco: sure!

[14:07:11] <dstufft> jaraco: ping me when you're avail? :)

[14:07:19] <jaraco> sure.

[14:07:24] <dstufft> cool, sounds good

[14:07:24] <dstufft> thanks

[19:23:56] <jaraco> dstufft, ping

[19:32:46] <dstufft> jaraco: pong!

[19:33:01] <jaraco> What's up dstufft?

[19:33:26] <dstufft> jaraco: are you familar at all with PEP 440? (The versioning PEP)

[19:34:22] <jaraco> not much beyond what we've discussed before.

[19:36:51] <dstufft> jaraco: Ok, so just background b/c I don't recall exactly, I have a "packaging" library which has a PR that implements PEP 440. I have a PoC PR for pip which uses that to implement PEP 440 and I'm looking at doing something similar for setuptools/pkg_resources and I wanted to see how you thought hat might look

[19:39:28] <dstufft> jaraco: FWIW the compatability fallout is basically 499 projects go completely uninstallable on PyPI, and 210 additional projects have a different latest version.

[19:40:03] <jaraco> What is the pip approach?

[19:40:09] <jaraco> Does it maintain compatibility?

[19:41:24] <dstufft> So the pip approach in the PoC is it does not maintain compatability, however we added an escape hatch to PEP 440

[19:41:50] <dstufft> e.g. if you have some wierdo version that does not conform, you can always install exactly it with a === (triple equal)

[19:42:01] <jaraco> !acronym PoC

[19:42:02] <pmxbot> Arg! I couldn't expand that...

[19:42:08] <dstufft> Proof of concept

[19:42:10] <jaraco> aah

[19:42:50] <dstufft> so there are two basic levels of compat

[19:43:24] <jaraco> It may be possible to do the same with setuptools - simply declare it as backward incompatible and describe the steps to upgrade.

[19:43:54] <dstufft> things like pkg_resources.parse_version right now returns a tuple of strings that can be sorted with normal sorting, does that signature need to say the same? Or can it return a an object? If it needs to continue returning a tuple can it return a tuple containingi tems other than strings?

[19:44:01] <jaraco> I'd probably also start by factoring out the versioning and dependency management in pkg_resources such that it's essentially pluggable and matches the interface of 'packaging'.

[19:44:05] <dstufft> then there's the backwards compat in the actual how do versions support

[19:44:12] <dstufft> s/support/sort/

[19:44:37] <dstufft> I tried to add extra rules to PEP 440 which wil support a much wider array of things that pkg_resources currently supports btw

[19:45:26] <dstufft> the main things we don't support is, arbtirary alpanumeric patch releases (1.0-whatever1), only numeric, and there isn't arbitrary nested levels of stuff (1.0.dev1-r1.dev4 etc)

[19:45:45] <jaraco> hmm.

[19:46:15] <jaraco> What do you recommend for the use case where someone forks a project that already has a version number?

[19:46:41] <jaraco> i.e. I fork requests 2.0.0. I typically call it requests-2.0.0-jaraco-1

[19:48:03] <dstufft> so PEP 440 has a specific thing for this, the local version

[19:48:13] <dstufft> 2.0.0+jaraco1

[19:48:14] <dstufft> or whatever

[19:49:11] <dstufft> alpha numeric segments seperated by . after the +, no semantics other than it's ideally API compat and you split on the . and sorting is done using string sort if it's alpha numeric and int() sort if that segment is integer only

[19:49:25] <dstufft> (I say seperated by ., but we also allow - or _ for compat sake)

[19:50:10] <dstufft> jaraco: the idea being, everything to the left of the + is the official versioning numbers for that project, everything tot he right is versioning stuff that was modified by someone else

[19:51:05] <dstufft> (and for instance, PyPI won't allow uploads of versions with a + in their version, so that is semi enforced)

[19:53:40] <dstufft> jaraco: to be clear, PEP 440 isn't actually accepted yet, though I think it's close and/or ready for a final discussion + acceptance. so thats why this stuff is all PoC right now, because it shouldn't be merged until we have a final version laid down

[20:08:14] <jaraco> dstufft, That's good to hear that's in consideration (local versions). I would expect to be able to upload local versions to a local cheeseshop and have them discovered.

[20:08:44] <dstufft> jaraco: oh yea, there's nothing in pip that prevents it from finding a local version on an idnex

[20:08:46] <jaraco> But it's all good. Whatever the community settles on will be the canonical form.

[20:08:58] <dstufft> just that PyPI as a public index won't allow them

[20:09:13] <dstufft> and theortically the PEP states other public indexes shouldn't either

[20:09:20] <dstufft> but in pratice PyPI is the only public index that matters

[20:09:35] <jaraco> As to your question regarding compatibility of functions, I have to assume that unless otherwise documented, every public function is fixed.

[20:10:05] <jaraco> So any changes to that interface (such as parse_version(...)) must be published as a compatibility change.

[20:10:51] <jaraco> Of course, if 'packaging' or any other library becomes canonical for parsing versions, it may be that pkg_resources continues to supply the old behavior.

[20:11:06] <dstufft> that's a totally legit way of doing that too

[20:11:08] <jaraco> But _uses_ the PEP440 behavior.

[20:11:33] <jaraco> And the existing functionality is deprecated with a reference to packaging.

[20:11:50] <jaraco> The big challenge you're going to face with setuptools is vendoring/bootstrapping.

[20:12:13] <jaraco> There's currently no model for vendoring or requiring (non-optional) dependencies in setuptools.

[20:12:16] <dstufft> so the way packaging works under the cover is sort of similar to parse_version in that it ends up as a tuple, except the tuple is an implementation detailw wrapped inside a class just for a cleaner API

[20:12:33] <dstufft> and instead of a tuple of strings, it's a tuple of strings, integers, and a special Infinity object

[20:13:19] <dstufft> so parse_version could totally still return a tuple

[20:14:38] <jaraco> special infinity objec? Not float('Inf')?

[20:15:12] <jaraco> nevermind - I'll try not to care about the implementation details.

[20:15:30] <jaraco> I've started a branch to work on allowing dependencies in setuptools.

[20:15:50] <jaraco> https://bitbucket.org/pypa/setuptools/issue/229/add-support-for-dependencies-such-as-six

[20:15:58] <dstufft> packaging/version.py is ~600 lines of code for both version parsing and specifier parsing, but ~120 lines of that is just in regexes which are expanded out over multiple lines to make it easier to read

[20:16:02] <dstufft> and there are a ton of comments too

[20:16:09] <jaraco> The tests are currently failing, but I suspect it's mostly test-specific failures.

[20:17:38] <dstufft> like parse_version could be -> https://github.com/pypa/packaging/blob/pep440/packaging/version.py#L114-L147 and just return the cmpkey instead of assigning it to a var

[20:18:28] <dstufft> but that depends on if you prefer to leave parse_version alone completely, and just use packaging internally, or make some sort of pluggable interface, or you think it's better to implement pep440, but keep the interfaces mostly the same, or just go whole hog packaging

[20:19:10] <dstufft> jaraco: also I'm not sure why I made a special Infinity object instead of using float("inf") and float("-inf"), I wouldn't be opposed to just using that

[20:19:51] <jaraco> if we could bear the pain of supporting dependencies in setuptools, I'd like to do simply use 'packaging' for version comparison and resolution.

[20:21:14] <jaraco> I would not be opposed to simply copy/pasting that technique into setuptools until #229 could be fixed.

[20:23:44] <dstufft> jaraco: for the record, I did the vendoring stuff in pip and it's worked fairly well for us. We wouldn't unvendor stuff in setuptools in pip, we don't bother to try and deduplicate stuff like that

[20:24:08] <dstufft> I also worked with Debian on figuring out how to handle the fact we vendored stuff, and how that interacted with PEP 453 / ensurepip

[20:24:11] <dstufft> and virtualenv

[20:25:04] <dstufft> although setuptools is a bit different than pip in that regards

[20:25:25] <dstufft> since setuptools is an actual dependency of a lot of things vs just being something that people install seperately

[20:25:25] <jaraco> I think there's still fallout from that work (and other vendoring techniques). I see conditional imports in projects that use requests that conditionally import urllib3 from vendored version or unvendored.

[20:25:50] <dstufft> yes, vendoring stuff is less ideal than jsut depending on things

[20:25:57] <dstufft> for pip it was vendoring or nothing

[20:26:08] <dstufft> depending on stuff is a no go because of the bootstrap problem

[20:26:42] <dstufft> I would prefer if requests didn't vendor urllib3, but I don't control that situation :/

[20:27:50] <dstufft> oh the other reason we don't depend on stuff directly, is even if you solve the bootstrap problem, you still have the "I accidently uninstalled something or installed a differently compatible version and now the package manager itself is broken and you have to go reinstall the dependencies manually"

[20:28:13] <dstufft> setuptools doesn't have that problem as much since eggs support smulti version isntalls, but it could still happpen