Log file Viewer
#pypa-dev logs for Friday the 3rd of April, 2020
[14:09:10] <sumanah> I see from https://dev.azure.com/pypa/pipenv/_build/results?buildId=21264&view=logs&jobId=e5c2fdf2-ea94-5e63-a966-a1772ac249a4&j=e5c2fdf2-ea94-5e63-a966-a1772ac249a4 that Azure is failing _differently_ now?
[14:17:28] <sumanah> I updated the release tracking issue to reflect that April is now the projected release month and that you're cleaning up a big chunk of technical debt with fixing the CI setup
[14:20:21] <sumanah> (also, test suites are uncomfortably reminiscent of a Skinner box https://en.wikipedia.org/wiki/Operant_conditioning_chamber )
[19:48:02] <sumanah> di_codes: thank you for your work on improving the infrastructure of metadata validation!
[19:48:24] <sumanah> I used to work with someone who would sign off his emails with "peace, love, and metadata" and I wish you all of those things as well di_codes
[19:59:33] <sumanah> for those who were not watching: https://github.com/pypa/packaging/issues/147 and https://github.com/pypa/warehouse/pull/7582 reflect that there is a new standalone pkg https://pypi.org/project/trove-classifiers/
[20:06:42] <sumanah> as it approaches 5pm ET: techalchemy I hope you will consider taking the weekend off. But please ignore me if that's not what you want
[20:07:11] <sumanah> (I mean I know it's not even 4:10, but, like, in the grand scheme of things, it's near the end of the workweek)
[20:08:07] <techalchemy> also thanks sumanah, i'm just glad i got windows tests running... apparently azure has decided to start prompting for host key verification when using git over ssh
[20:08:12] <sumanah> di_codes: I am taking a fresh look at https://github.com/pypa/packaging-problems/issues/264 on metadata strictness and a bit of discussion from https://discuss.python.org/t/increasing-pips-pypis-metadata-strictness-summit-followup/1855/9 and trying to disentangle what else to ask for soon
[20:09:25] <techalchemy> yeah also i couldn't spin up any windows vms locally because of some weird interactions between hw virtualization and secure boot + changes to the kernel recently i think, so i wound up just spinning up a vm in GCE to figure that out
[20:09:46] <PSFSlack> <di> yeah, got tired of running my IRC bouncer so I asked ernest to enable the bridge for these channels as well
[20:14:10] <sumanah> di_codes: I am grateful to be able to communicate with you in a way that works for you. Thanks, Ernest.
[20:14:32] <sumanah> techalchemy: so we've made some progress in the last few months, which is great....
[20:15:32] <techalchemy> I think it doesn't resolve short paths on windows / thinks they are distinct from long paths
[20:16:23] <techalchemy> AssertionError: Egg-link c:\users\runner~1\appdata\local\temp\pipenv-mv0dt017-test\six does not match installed location of six (at c:\users\runneradmin\appdata\local\temp\pipenv-mv0dt017-test\six)
[20:24:19] <sumanah> Warehouse started hard failing package uploads on invalid markup with explicit description type https://github.com/pypa/warehouse/issues/3285 , and there's progress on the yanking of old releases https://github.com/pypa/warehouse/pull/5838 , and we now check that .tar.gz files are actually just that https://github.com/pypa/warehouse/pull/5609
[20:27:48] <PSFSlack> <di> "verifying metadata is like the single most important thing we can do imho" not sure I really understand this
[20:28:15] <PSFSlack> <di> we're already verifying metadata, we're just moving the logic to a place where others (like twine) can verify it before trying to upload it to PyPI and get it rejected there instead
[20:31:13] <techalchemy> di: i mean, verifying metadata is kind of meaningless if we still let people upload things
[20:31:29] <techalchemy> metadata verification needs to happen in warehouse imho, not in the upload client
[20:32:04] <PSFSlack> <di> it'll still happen there. we're just making it possible to do the exact same validation PyPI will do without the round-trip to pypi
[20:32:11] <techalchemy> yes, we should have tools to allow maintainers to know if they are meeting standards too, but accepting artifacts that don't conform is bad
[20:32:43] <pradyunsg> techalchemy: a GH issue (or zulip topic 😬) might be a better way to reach out to people with some idea of both pip's internals and Windows. :P
[20:33:48] <techalchemy> @di right, my only point is that preventing people from uploading things that don't follow metadata standards is the topic I have had the most conversations about with other package manager maintainers, etc
[20:33:51] <pradyunsg> @di IIUC, the plan is to enhance packaging to have those checks, and to make twine check do them as well, correct?
[20:34:18] <sumanah> so techalchemy I do recommend you take a fresh look at https://github.com/pypa/packaging-problems/issues/264
[20:34:54] <sumanah> "Increasing pip's & PyPI's metadata strictness" which is the issue summarizing TODOs on this topic, from the meeting at PyCon last year
[20:35:01] <PSFSlack> <di> techalchemy: sure, what i'm saying is that you already can't upload something that doesn't follow "metadata standards".
[20:35:29] <techalchemy> @di my complaint isn't that the sdist itself is invalid, just that the metadata can be -- e.g. requirement and extras specifiers, python_requires specifiers, etc
[20:36:39] <PSFSlack> <di> techalchemy: yeah, pypi verifies the specifiers too, so that shouldn't be possible. now, they could be over-constrained, but that's not "invalid" per the metadata specs
[20:37:32] <techalchemy> right -- i mean we have to believe what people say their package is and needs
[20:37:43] <PSFSlack> <di> techalchemy: which is why I don't understand discussions about "increasing PyPI's metadata strictness" -- it's already as strict as it can be to conform with the specs, and if there's something missing, that's a bug
[20:38:05] <techalchemy> @di I didn't realize it had been implemented since the conversations last year
[20:38:40] <sumanah> so maybe you would revise your statement to "verifying metadata is like the single most important thing we can do imho.... so I'm really glad we're doing it" ;-)
[20:42:51] <sumanah> di_codes: I've gotten a bit confused about what we've already achieved and what still needs checking/tightening up, lemme look again at this issue
[20:42:56] <techalchemy> @di I didn't realize the server side work was already sorted out -- thanks for your work on that then!
[20:45:08] <PSFSlack> <di> sumanah: eh, most of that was already done before I started working on warehouse. deciding what can/can't be uploaded is kind of central to a pypi implementation
[20:45:39] <sumanah> I see a note "For packages where no restrictions on Python version are desired, a “python_requires==*” would be satisfactory"
[20:46:55] <PSFSlack> <di> sumanah: I left a comment about python_requires specifically here: https://github.com/pypa/packaging-problems/issues/264#issuecomment-578950775
[20:47:23] <sumanah> (as I read multiple times, different things make sense to me or jump out at me. Sorry for the redundancy di_codes )
[20:47:46] <sumanah> I saw there was a note: "Also, can we fail when there's missing author, author_email, URL. Currently warnings on setup" so I should go check as to which of those are optional
[20:49:44] <techalchemy> yeah, wat least for what I personally care about, it's just that for things that _are_ populated, they should contain valid metadata (I guess warehouse parses this to make the json api anyway)
[20:49:58] <sumanah> Running auditwheel on new manylinux uploads https://github.com/pypa/warehouse/issues/5420 . di_codes I see you said last year "while I do think PyPI should eventually start auditing uploaded wheels, this is not going to happen overnight and/or without fair warning."
[20:50:54] <sumanah> (also di_codes if now is a bad time to dive into this, please say so and I'll switch mediums)
[20:52:41] <sumanah> di_codes: I remember a while back you were working on package preview/2-phase upload https://github.com/pypa/warehouse/issues/726 . Is that the case? I may be mistaken
[20:53:34] <PSFSlack> <di> sumanah: haven't started work on that yet. I don't think wheel-auditing is a blocker for that (or vice versa)
[20:53:56] <sumanah> then I would love for us to start auditing those wheels. I'll leave a comment on that issue
[20:54:20] <PSFSlack> <di> wheel-auditing just requires all the infra to run auditwheel in a sandbox or something. we don't have anything like that now
[20:54:22] <sumanah> pradyunsg: https://github.com/pypa/warehouse/issues/69 "Clean database of UNKNOWN and validates against it" is this something that would aid the pip resolver work? I don't think so but I want a 2nd opinion
[20:54:59] <PSFSlack> <di> or maybe a sandbox wouldn't be totally necessary? I'm not sure if auditwheel is 100% static analysis
[20:55:06] <techalchemy> the resolver stuff is mostly hampered by the Requires-Python metadata issue
[20:56:07] <techalchemy> the issue being that the field can? or at least used to be something you didn't have to populate with a specifier in all cases, so there are weird variants in there that require some additional cleaning
[20:57:13] <pradyunsg> techalchemy: I think what you're asking for is more consistency in the values of Python-requires when specified, *and separately* that the value be made non-option, correct?
[20:57:59] <techalchemy> I mean sure I would love things to be required but the more difficult challenge for me has always been the things that are already populated but populated with invalid values
[20:58:51] <PSFSlack> <di> definitely can't go back in time and edit metadata. yanks will help though
[20:59:50] <techalchemy> yeah, I don't think there's a real solution to that problem which is why I am just looking ahead at new uploads
[21:00:04] <techalchemy> as long as those are being validated it's probably the best we can do for now
[21:00:51] <sumanah> so in https://github.com/pypa/warehouse/issues/474#issuecomment-370986838 Nick talked about retrofitting metadata for old releases ... di_codes it sounds like you're saying we can't/won't do that. is that a policy thing, technical feasibility, both?
[21:02:05] <sumanah> pradyunsg: would it help the resolver work if PyPI ran auditwheel on manylinux uploads, stopping less-valid wheels from being uploaded? (I have suspicions that the answer is yes)
[21:02:23] <PSFSlack> <di> policy. we've done it literally once for a crucial django release and people flipped out
[21:03:54] <sumanah> yeah. I remember. dredging up those memories from the ancient Before Time back when I worked more on PyPI
[21:04:31] <sumanah> I see someone said in that discussion at PyCon last year: "Could we explore banning old upload clients from PyPI?" is there still any reason to do that, now that we've maybe solved the "bad metadata" upload problem at a different abstraction layer?
[21:05:00] <techalchemy> IMHO it shouldn't matter what is putting the bits into the api if the api validates them
[21:05:06] <PSFSlack> <di> sumanah: seems like it would make sense to ban old metadata versions instead
[21:05:38] <sumanah> Looking now at https://packaging.python.org/specifications/core-metadata/#metadata-version
[21:06:41] <PSFSlack> <di> yeah, IIRC wheel will use the minimum metadata version or something like that
[21:06:56] <dstufft> IIRC distutils (and I suspect setuptools does as well, as an extension to distutils) only emits the minimum metadata version that the project needs
[21:11:23] <sumanah> dstufft: can you help me understand -- when did we start versioning metadata with a proper Metadata-Version?
[21:12:14] <dstufft> maybe it's changed in the interim so we no longer emit the lowest common denominator for Metadata-Version
[21:13:10] <sumanah> I'm reading https://packaging.python.org/specifications/core-metadata/#metadata-version now "For broader compatibility, build tools MAY choose to produce distribution metadata using the lowest metadata version that includes all of the needed fields."
[21:13:11] <techalchemy> i can try older version of setuptools or i can try older versions of python
[21:13:13] <dstufft> techalchemy: the original logic was def in distutils, and I think setuptools inherited by nature of being an extension to distutils
[21:13:27] <PSFSlack> <di> don't forget about 2.0, the spooky metadata version that never was: https://github.com/pypa/warehouse/blob/master/warehouse/forklift/legacy.py#L401
[21:14:11] <dstufft> but regardless, I think the point still stands, we don't require people to emit the latest metadata
[21:15:16] <sumanah> dstufft: ok. I get that this has been our approach so far. I think I am having trouble thinking about the use case for favoring compatibility-with-old-tools over making-them-do-what-I-want
[21:15:20] <techalchemy> i mean, tools can still emit the lowest common denominator whether warehouse accepts them or not :p
[21:16:01] <dstufft> sumanah: in theory tools should warn and/or error if they get a package with a Metadata-Version that they don't understand right?
[21:16:16] <dstufft> because it might have important information in it that they don't now how to understand
[21:17:31] <dstufft> so if we release a Metadata-Version 2.2, all of those tools are going to suddenly reject /warn anything that uses Metadata-Version 2.2, even if that project is using nothing from Metadata 2.2 and is otherwise (besides the Metadata-Version) a 2.1 compatible version
[21:18:13] <techalchemy> and yeah pushing out new metadata versions should happen rarely and should be as backward compatible as possible
[21:19:05] <dstufft> I guess the other thing is, what value do you get from setting a minimum? All of those metadata <old> things still exist on PyPI and so tooling is going to generally still need to be able to understand them
[21:20:45] <techalchemy> a guarantee that new artifacts have more thorough and complete information since it helps with discovery, search, gives a more complete understanding of what is actually in pypi, i mean
[21:22:21] <techalchemy> I guess it's a question of do you really think poeple should still be uploading artifacts that only contain a name and a version? If we did a review of other packaging ecosystems we'd find a lot of other information like SPDX license specifiers, URLs, some even have mandatory screenshots (not really relevant to us but still)
[21:22:22] <sumanah> step a. increase what's necessary in Metadata, like, 2.2 to require a Requires-Python field
[21:23:39] <dstufft> This can be super useful to make it easier for projects that are not meant to be uploaded to PyPI
[21:24:14] <dstufft> like just for instance, SPDX license specifiers... if your package is never meant to be distributed to the public, being forced to set that is just make work
[21:24:25] <sumanah> and then step c. use the 2-phase upload to give a heads-up to the people who are uploading to PyPI....... ok I'll stop this thinking-aloud and answer your point
[21:24:54] <techalchemy> dstufft, but SPDX specifiers cover all of the licenses, you can still upload a project and type in the name of your license now
[21:24:55] <sumanah> dstufft: so I think this is where I was trying to respond to what di_codes had said, which is that if we want people to upload .... hold on
[21:25:27] <sumanah> if we want people to specify Requires_Python and we want Warehouse to mandate it, then we should change the metadata specification
[21:25:30] <techalchemy> at least if someone uses an spdx license and i'm programmatically consuming metadata i will know i can't redistribute
[21:26:09] <sumanah> techalchemy: I can't tell: are you arguing that we should mandate SPDX for PyPI upload?
[21:26:31] <dstufft> Like I said, Warehouse can mandate things more easily than the metadata spec can
[21:26:48] <dstufft> because the metadata spec has to function for all thing, even things that never get released to PyPI
[21:27:42] <techalchemy> does the information warehouse displays about a package even necessarily need to tie to package metadata?
[21:27:52] <sumanah> ok, I agree with dstufft that it's overkill to require the creators of all packages to do SPDX paperwork, and I get how techalchemy would really prefer that all new Warehouse packages have SPDX metadata
[21:28:37] <sumanah> techalchemy: so you're thinking that, alternatively, Warehouse could do some smart parsing of the repository or something? instead of tying to package metadata?
[21:29:06] <techalchemy> sumanah, no, thinking that there could be a web UI for a maintainer to add some extra info
[21:29:29] <sumanah> it sounds like di_codes and dstufft disagree about the levels of Warehouse mandates outside the metadata spec
[21:30:22] <sumanah> linking now: https://github.com/pypa/packaging-problems/issues/264#issuecomment-578950775
[21:30:39] <techalchemy> e.g. https://i.imgur.com/EtxVzau.png <- here is what the canonical snap store UI looks like for packages
[21:30:48] <sumanah> "For things like "warn on missing python_requires", we could have this today if we made this field required in the metadata specification." and I think earlier today in the backscroll he reiterated that would be the approach he would prefer
[21:31:25] <techalchemy> there is also a build metadata file but what you see in the UI doesn't come from there
[21:32:14] <dstufft> techalchemy: I think the answer to your question is there is nothing technically preventing it, but I'm not sure that it's something we would want to do. I would have to think about it more
[21:33:01] <EWDurbin> It feels like a pretty bad idea to blur the line we've drawn around modification of metadata post upload.
[21:33:14] <sumanah> So then we would have to explain to maintainers "there are 2 places you need to modify things, and one of them cannot be automated"
[21:33:40] <techalchemy> for one I find it kind of odd when I login to pypi and can't do anything really
[21:34:11] <techalchemy> yeah no i mean there is a lot of like actual important package maintenance i can do
[21:34:30] <dstufft> We certainly wouldn't want to allow you to modify metadata different than what is inside the package
[21:34:52] <sumanah> techalchemy: so it sounds like we are making different design choices than some other package management sites
[21:34:58] <dstufft> if we implemented it, it would have to be wholly distinct metadata that didn't exist at all
[21:36:12] <sumanah> and, again, there would be a support cost. Of time and maybe money someday. Of explaining to maintainers "here's what you can change where" and dealing with confusion about what can and cannot be automated, whether owners vs maintainers can do the web UI things..... I am not saying "veto" but there would be this cost
[21:36:20] <dstufft> I could see a world where we had made different choices and hadn't stuffed a lot of this metadata into the package and instead had made it some project level metadata on PyPI or something, but I think we're well past the point of no return on that
[21:36:32] <PSFSlack> <di> sumanah: yeah, I wasn't saying we should/shouldn't change python_requires, just that if the spec said that it was required, it should probably be required on PyPI too.
[21:38:01] <techalchemy> sure, I think the ship has sailed on most of these decisions for now. And the interface is super intuitive and such now as well so I can't really complain
[21:38:21] <sumanah> ok so then let's return to the question of whether we want to ban old metadata versions, or old upload clients, from uploading to PyPI
[21:38:45] <dstufft> Yea, I think that if we require something in the metadata spec, then PyPI should 100% require it, and we can also optionally opt to require (or fruther constrain) what PyPI accepts as valid metadata
[21:40:01] <dstufft> I don't think banning old metadata versions buys us anything unless we did something like change the meaning of a metadata field in a backwards incompatble way where the only discrminiator was metadata-version
[21:40:15] <dstufft> and i think it's a poor user experience to use metadata-version as the enforcement mechanism
[21:40:38] <dstufft> like let's say that we change it so that in 2.2 python-requires is required, and we say "ok PyPI requires metadata 2.2 now"
[21:41:39] <dstufft> From my POV, the metadata-version validation on PyPI is primarily there to ensure that PyPI gets a metadata-version that it understands how to parse
[21:42:11] <dstufft> any other constraints (like required fields, invalid field, etc) should just be validated on a field by field basis
[21:42:41] <sumanah> I don't have any particular notes on _why_ people were interested in exploring banning old upload clients last year -- the notes don't say I think
[21:44:11] <techalchemy> so if we want pypi to require more metadata that should be decided based on the specific metadata fields we want, and the error that it spits back should tell you which fields it expected to find but didn't
[21:44:53] <techalchemy> instead of talking about some indirect question about metadata versions which act as a surrogate for the real issue, and probably a poor one
[21:46:47] <techalchemy> yes that makes plenty of sense to me because as you and dustin both mentioned we still have a ton of packages with the old metadata so it's not like those are going anywhere
[21:46:56] <dstufft> sumanah: I think that came out of the idea of just having twine warn about requires-python
[21:47:14] <dstufft> and then it was "well if twine warns, but nobody is using the new version, wat do"
[21:48:48] <dstufft> I don't think blocking old versions is a very worhwhile change exept for maybe some edge case where we change the meaning of something that the old client will give us erroneous information that we can't otherwise detect, but I don't think we'd generally do that
[21:49:26] <sumanah> ok, so I'm going away with 3 main things: nudging staged releases along, nudging the auditwheel checking of new wheel uploads along, and filing an issue where Warehouse would block uploads that lack python_requires
[21:49:58] <sumanah> and maybe we won't do that last thing, but at least I figure we should have the discussion on GitHub instead of IRC
[21:50:15] <dstufft> Reading back through the history about too from before I sat back down, I think if we still have any issues that are like "mae things more strict" we should probably just close them unless those issues have specific completion criteria
[21:50:39] <dstufft> a lot of that is my fault, when I originally wrote a lot of the issues in Warehouse they were mostly just notes to myself, to "do this thing i know we're going to need at some point"
[21:51:17] <sumanah> I started https://github.com/pypa/packaging-problems/issues/264 as a place to list the things we wanted coming out of the summit last year; it is ripe for checklisting
[21:53:39] <dstufft> We *might* be able to modify the metadata in the database so that it more correctly matches the metadata in the files, in cases where they deviate
[21:54:21] <dstufft> like requires-dist currently a release specific field, but different wheel files can have different requirements
[21:54:57] <dstufft> and as part of that, we can *probably* "fix" the metadata in the database by backfilling it by looking inside the wheel files
[21:55:39] <dstufft> I guess a better way to word that, is the metadata inside the files should be considered authorative, and I'm pretty sure we can adjust the Database to match the authorative record
[21:55:44] <sumanah> ok, so, https://github.com/pypa/warehouse/issues/194 is a general "let's check more things upon upload" issue, which is the kind of thing I think you were talking about dstufft - not concrete
[21:58:33] <sumanah> dstufft: sometime next week if you want to get on a call, we can renew the "Donald's Responsibility Inventory" to get a fresh one
[22:00:04] <dstufft> I also spent all last week processing my like, many thousands of unread email so I actually can notice incoming email again :P
[22:05:59] <pradyunsg> dstufft: yay! I feel like that's an accomplishment that deserves a congratulations. :)
[22:08:01] <EWDurbin> sometimes you're just sitting watching the messages scroll by while you eat dinner, drink a beer, and watch the horror of a whitehouse press briefing.
[22:09:16] <pradyunsg> EWDurbin: I've been checking https://www.covid19india.org/ daily, which is better and worse in some ways.
[22:09:32] <dstufft> EWDurbin: my GH keys are not restored either :P I'm grabbing my backup drive to restore them
[22:09:37] <sumanah> oh for like the last hour I didn't think about the pandemic at all! I was actually absorbed in work!
[22:09:57] <EWDurbin> dstufft: contents of your authorized_keys on the bastion host now match https://github.com/dstufft.keys anyway
[22:10:14] <dstufft> and I don't give people my laptop (even though it has encrpytion) with any of my actual data on it
[22:10:34] <EWDurbin> oh dang, i should probably wipe and send my macbook for keyboard repair while i'm certainly not traveling
[22:12:02] <dstufft> they had it fixed and back on the truck in < 24 from when Fedex picked it up from my house
[22:13:58] <sumanah> pradyunsg: you're not the only one who reminded me of The News so no apology necessary ;-)
[22:15:57] <techalchemy> my thinkpad swings in the opposite direction and aggressively thermal throttles due to firmware issues making it think it is always in lap mode
[22:17:18] <sumanah> pradyunsg: would it help the resolver work if PyPI ran auditwheel on manylinux uploads, stopping less-valid wheels from being uploaded? (I have suspicions that the answer is yes)
[22:20:17] <EWDurbin> if we had 2-phase uploads, we could probably hold manylinux wheels from being public until results existed...
[22:22:46] <sumanah> "Determine new API URL structure for warehouse (starting with new JSON API)" is not quite it
[22:23:29] <sumanah> dstufft: I think today is your lucky day and there is no particular issue for this, IMO
[22:24:06] <dstufft> the hardest part about that issue is iguring out how we deal with the existing uploaders
[22:25:24] <dstufft> because we'd want some mechanism where twine could get an identifier back, and then poll for completion (and any errors / warnings)
[22:26:12] <dstufft> so it'd probably be some thing where we stand up a second API with the new semantics, then deprecate the old one and start pushing people to it
[22:26:31] <dstufft> but maybe spending omre than 30s thinking about it would come up with something clever
[22:31:02] <pradyunsg> sumanah: while it won't directly affect the resolver work (since we're only checking the tags for compatibility), it would help the actual compatibility issues that users face. Basically, not directly affecting but certainly nice to have!
[22:32:01] <sumanah> right .... it would help reduce the general support load and help reduce how often users get into bad situations
[22:37:06] <pradyunsg> I'mma go sleep now because I'd like to slowly pull my sleep timings to be more... appropriate for my timezone.
[22:41:36] <travis-ci> Change view : https://github.com/pypa/pip/compare/d53d3d6b24dd67dee7c89298ff192bbd9867bbbd...f7c5a69e69c96dadb50f182654135ba7a464d91b
[23:30:15] <travis-ci> Change view : https://github.com/pypa/pip/compare/f7c5a69e69c9...73bfea6d28b1