Log file Viewer

[00:47:16] <kit> If anyone can give me some feedback on https://github.com/pypa/warehouse/pull/588 I'd appreciate it. Thanks!

[04:27:09] <lifeless> tchaypo: I need to go pick up cynthia; perhaps i nmy evening ?

[04:27:59] <tchaypo> Yep, I should be settled in an hour or so

[05:29:52] <tchaypo> lifeless: okay, this looks like less effort than I thought it would be

[05:30:47] <tchaypo> looking at the entry points in setuptools’ setup.py it looks like all the requirements are processed by setutools.dist:check_requirements, except extras

[05:31:18] <tchaypo> so simply enhancing check_requirements to support markers should do all three at once

[05:31:52] <tchaypo> but as you said yesterday

[05:31:53] <tchaypo> we need to think at least a little about the impact of metadata in wheels landing on a system with an older pkg_resources

[05:32:03] <tchaypo> Nakato: ping, this probably interests you

[05:58:46] <lifeless> tchaypo: indeedy

[05:58:56] <lifeless> tchaypo: so there are I think two ways forward

[05:59:21] <lifeless> tchaypo: one is to say 'if you use this feature, we'll generate an automatic install_requires of a suitable version of setuptools.

[05:59:38] <lifeless> tchaypo: another is to say 'buyer beware'

[05:59:50] <lifeless> tchaypo: a third is to put this on the back burner for some hypothetical future time

[06:01:49] <lifeless> tchaypo: another way forward is to map install_requires markers and markers-in-extras to the extras magic format automatically, leaving setup_requires and tests_require as separate cases

[06:02:05] <lifeless> setup_requires only needs to be handled by setuptools itself (today), so there's no compat issue to speak of.

[06:02:10] <lifeless> tests_require likewise.

[06:14:11] <tchaypo> okay, two things

[06:14:37] <tchaypo> one is, I think I remember hearing you say that in the past adding a setup_requires on a specific version of setuptools has had mixed results

[06:14:59] <tchaypo> but I don’t remember details. Is there anything in your experience that’s likely to be relevant here?

[06:16:12] <tchaypo> second - the existing code is quite stringent about what it accepts as a valid identifier; I can’t think of a simple way to jam the env markers in

[06:16:56] <tchaypo> if we assume the setup_requires=setuptools thing works, that still means we can’t ever put markers in setup_requires, because we have to assume setup_requires is being processed by an old version

[06:17:27] <tchaypo> but also, for it to work, the setuptools process is going to have to start running the code it just downloaded rather than the code that was already on disk. That sounds tricky..

[06:31:09] <lifeless> tchaypo: so

[06:31:21] <lifeless> tchaypo: setuptools in setup_requires doesn't work but its still useful.

[06:31:40] <lifeless> tchaypo: there are two forms of failure. Silently doing the wrong thing, and not-that.

[06:32:12] <lifeless> tchaypo: the point of a setuptools min-version in setup_requires would be to avoid silently doing the wrong thing, not to make it work :)

[06:36:08] <tchaypo> okay.

[06:36:16] <tchaypo> If it doesn’t work, what does the error look like?

[06:36:18] <tchaypo> *checks*

[06:37:24] <tchaypo> distutils.errors.DistutilsError: Could not find suitable distribution for Requirement.parse('setuptools>34')

[06:38:01] <tchaypo> That seems to make the error fairly clear, but maybe not how to resolve it

[06:38:42] <tchaypo> but wait, you said install_requires of a suitable version of setuptools

[06:40:30] <tchaypo> does that change anything?

[06:41:09] <tchaypo> no, because if I have two install_requires, one that has a marker and one that is for a version of setuptools that understands the marker, the old version of setuptools will try to parse both and barf on the marker

[06:43:30] <tchaypo> and that’s really annoying, because the error message then is going to be about the marker, not about the version of setuptools

[06:43:58] <tchaypo> as a side note, I just realised that this means PEP-0496 requires this work

[06:44:16] <tchaypo> because as currently written it has examples of using markers in install_requires and setup_requires

[06:47:20] <tchaypo> it seems to me there’s a fourth way, which is to do something similar to “setup_requires=[‘pbr’]"

[06:47:40] <tchaypo> ie, have the marker handling in an external dependency

[06:50:02] <tchaypo> if I’ve got the order of operations right, that would mean that the un-augmented code only parses setup_requires and installs the new plugin; then the augmented code can intercept and parse install_requires and test_requires

[06:50:36] <tchaypo> downside - yet another external bit of code that setuptools needs to download and run

[07:03:24] <tchaypo> how about we start off with an implementation that defaults to translating markers on install_requires into extras; so that people can use the new version of setuptools when building an sdist, but still end up with a backwards-compatible sdist. But we’ll add a flag that allows this behaviour to be turned off, in which case we’ll pass the markers through

[07:03:24] <tchaypo> to the requires.txt in egg-info and we’ll add the setup-requires to make sure older versions of setuptools fail noisely

[07:04:08] <tchaypo> at some point in the future we can make that flag the default; but even then packagers will still be able to produce a backwards-compatible package if they want

[07:04:45] <tchaypo> however, env markers in tests_require would always need the new format, because we don’t have an mechanism to read the existing tests_require in from the extras

[07:08:13] <lifeless> install_requires so that pkg_resources can handle the installed thing

[07:08:22] <lifeless> setup_requires to allow builds to work; may need both

[07:30:21] <tchaypo> *rethinks*

[07:30:57] <tchaypo> okay; even if we go the backwards-compatible route we’d want a setup_requires to make sure that the version of setuptools processing the install_requires is new enough to know how to mangle them into extras; but that way, we don’t need an install_requires

[07:33:51] <tchaypo> or at least, we only need one that ensures setuptools is new enough to know how to handle markers in extras

[07:34:18] <tchaypo> the non-backwards-compatible route will need both setup_requires and install_requires

[07:34:44] <tchaypo> my head is buzzing; I’m going to hit the gym and try to document this later on

[07:35:05] <tchaypo> I think that once I’ve documented the problem and the possible solutions this deserves an email

[07:57:10] <lifeless> tchaypo: actually, setup_requires does not imply install_requires

[07:57:30] <lifeless> tchaypo: and pip's bundled pkg_resources may be new enough without a new enough installed pkg_resources

[07:57:46] <lifeless> tchaypo: yes, email to distutils-sig

[07:58:03] <lifeless> tchaypo: it should perhaps be part of the markers PEP

[07:59:42] <tchaypo> Either we make it part of pep496 or we alter pep496 to not require it

[08:01:41] <tchaypo> So maybe what we need is a setup_requires, so we fail fast if Setuptools is too old to handle the markers; and an install_requires on pkg_resources in the case where we produce non-backwards-compatible install_requires?

[08:49:24] <lifeless> tchaypo: yes, except pkg_resources is delivered by setuptools, which is why i suggested install_requires on setuptools

[10:12:35] <tchaypo> Sigh

[10:53:12] <dstufft> lifeless: tchaypo old setuptools is already borken with wheels

[10:57:08] <lifeless> dstufft: tchaypo: sure, but old == the release before we put this feature in

[11:41:31] <dstufft> lifeless: I'm not sure what metadata wheels don't already have though.. wheels already support environment markers

[11:42:30] <dstufft> you'd only need a new enough version of wheel itself that understands how to use the new format for environment markers

[11:42:39] <dstufft> and only on wheel build

[13:45:21] <xafer> hello, any plan to cleanup the 500+ issues of pip ? Half of them are certainly invalid/incomplete/obsolete ones ...

[13:50:04] <xafer> would it be helpful to make a list of such issues ?

[15:10:14] <carljm> xafer: I think bringing issues to their proper conclusion is more useful than making lists of issues.

[15:11:07] <carljm> That is, if you have some time to spend cleaning up the pip issues (which is great!), and you find an issue that is invalid/incomplete/obsolete, it is more useful to comment on the ticket to that effect (with evidence if you have it) than to put it on some list.

[15:20:50] <xafer> well, the invalidity of a ticket is certainly subjective (since the creator - hopefully - thought it was valid), that's why I was thinking of going over the old tickets and making a list of invalid tickets (to me)

[15:22:45] <xafer> https://github.com/pypa/pip/issues/1383 for example ...

[15:33:10] <Wooble> Well, yeah, that's probably not pip's fault.

[15:34:06] <Wooble> (plus matplotlib has OS X wheels now, so it's obsolete anyway)

[15:40:19] <dstufft> xafer: carljm I might say that it'd be good to do both, to comment on the issue on why you think it's invalid, incomplete, or obsolete and then make a list of those to send to the ML

[15:40:38] <dstufft> since github doesn't allow just anyone to change ticket states a contributor has to do it

[15:41:07] <dstufft> a whole slew of emails from github is hard to keep track of for myself to go and double check the comment, make sure it sounds reason, and then close them (or whatever)

[15:41:18] <dstufft> a list makes it easy, but the actual reasoning should be on the ticke titself

[15:41:20] <dstufft> if that makes sense

[15:43:55] <xafer> dstufft, ok, I'll comment the issues and start a list

[15:45:42] <dstufft> xafer: cool, if there's a list I can commit to going through them and reviewing the comment and the tickets themselves

[15:45:52] <dstufft> maybe not all at once if it's a long list, but relatively quickly

[18:23:43] <lifeless> dstufft: IIRC the serialisation of markers is distinctly weird, and we're talking about how to affect -or not- that

[23:17:32] <lifeless> tchaypo: morning