Log file Viewer
#pypa-dev logs for Wednesday the 3rd of July, 2019
[14:01:21] <sumanah> You mentioned being interested in helping prep the grant proposal so I would love to get some help from you on some info-gathering I need to do
[14:13:49] <sumanah> so: first off, I got your email and will update https://wiki.python.org/psf/Fundable%20Packaging%20Improvements
[14:14:22] <sumanah> pradyunsg: so do you have, like, 30 min right now so I can info-dump a lot at you and ask you for stuff?
[14:15:52] <pradyunsg> I will have 30 minutes after 20 minutes. Last meeting for the day, just finished so wrapping up work for today.
[14:26:14] <sumanah> for anyone following along, the thing I'm gonna be talking about a lot with pradyunsg is https://chanzuckerberg.com/rfa/essential-open-source-software-for-science/ . Applications are due August 1
[14:35:35] <sumanah> under "detailed application instructions" on https://chanzuckerberg.com/rfa/essential-open-source-software-for-science/ there's a section called "Open Source Project Details #1 – #5" and I could use help assembling the "Project Details" and "Project Metrics" and "Impact" for pip and Warehouse (let's start there, even though we may also want to assemble that info for other projects next)
[14:48:56] <sumanah> so I think the first step is to start a collaborative document. You get to choose! Etherpad, Google Docs, Dropbox Paper, something else :-)
[14:51:42] <pradyunsg> I took the liberty of making one: https://paper.dropbox.com/doc/2019-Chan-Zuckerberg-Initiative-WIP--AgN82JQFhUckYPuw6a_uBoR~Ag-rAnJfdSq8emDP1YJzWLMO
[14:52:41] <sumanah> pradyunsg: so I think a good thing for you to do is actually read https://chanzuckerberg.com/rfa/essential-open-source-software-for-science/ start to end. Stop after like 10 min and ask me any questions you have
[14:54:53] <pradyunsg> sumanah: I'd skimmed through that when you'd initally mentioned that's a possible grant to apply for.
[14:57:51] <pradyunsg> sumanah: I'm guessing we're applying under "Foundational tools and infrastructure"?
[15:00:39] <pradyunsg> fiscal sponsor: PSF is applying for and dispersing the funding (via Packaging WG); correct?
[15:01:40] <sumanah> So the pitch is going to be: We're seeking funding to make foundational improvements to Python packaging and distribution tools that biomedical researchers use.
[15:02:06] <sumanah> The amount we can ask for between $50k and $250k USD, covering up to 1 year of work
[15:09:41] <pradyunsg> Selection Process -> Impact -> "Demonstrated scientific impact of the project": hmm... not sure about this.
[15:10:47] <pradyunsg> Selection Process -> Quality -> "Existence, clarity, and recency of project roadmap": oooooh
[15:12:08] <toad_polo> sumanah: By the way, is "migrate distutils into setuptools" on your radar already? I think we didn't discuss it at the mini-summit because there's not much to discuss, but it's a pain point that is mainly stymied by lack of resources.
[15:12:24] <pradyunsg> "Number of open issues, and rate of issues both opened and closed" and "Time between opening and closing of pull requests": These are optional metrics; I don't think they really matter for us so it might be handy to mention how we can have issues linger for literally a decade and it's OK.
[15:13:01] <pradyunsg> sumanah: about what toad_polo said, https://github.com/pypa/packaging-problems/issues/127
[15:13:07] <toad_polo> We were thinking that it may be a roughly GSoC-sized project for a sufficiently advanced student, with the main downside being that there aren't good "half-done" waypoints. It's mostly all-or-nothing.
[15:13:08] <sumanah> (toad_polo thank you for confirming my biasesl this is an example of why I like to do this sort of discussion in public)
[15:15:02] <toad_polo> pradyunsg's link is I think right. The TL;DR version is that `setuptools` is a wrapper around `distutils`, but it also monkey-patches `distutils`, so most `distutils` fixes these days go directly into `setuptools` (which supports more versions of Python, etc).
[15:15:50] <toad_polo> We'd like to move `distutils` entirely into `setuptools` and turn all the monkey patches into real patches, but it's a big enough job that none of us can do it on a single weekend or sprint, and like I said it's kind of an all-or-nothing job.
[15:15:56] <pradyunsg> toad_polo: From experience, "GSoC-sized project" with no in-between isn't exactly super friendly for the student. :)
[15:16:16] <sumanah> pradyunsg: ok, so, I need to conserve my time on CZI application stuff today and move on to some backlogged OTF work. Do you feel like you know enough to make a start on the Metrics and Details portions we've started on?
[15:16:43] <toad_polo> pradyunsg: Yeah, that was my big reservation about it. It would be a super good accomplishment for someone to do it, but I know how fast an internship goes.
[15:17:45] <sumanah> pradyunsg: please do timebox this, I'd be curious to know what you can get done in about an hour
[15:18:29] <toad_polo> sumanah: The other part of it that can actually be done independently is the migration of the `distutils` docs out of the Python documentation and into `setuptools` and packaging.python.org, as appropriate. That can be done in smaller chunks, though at some point we should probably organize the work so that we know which chunks are already done and which ones need to be done together, etc.
[15:19:18] <toad_polo> Anyway, it came up again recently on CPython so I figured I'd ping you while I had it on my mind. It's not high priority, but it's worth having on the long list of things that are blocked on lack of resources.
[15:19:40] <pradyunsg> sumanah: toad_polo: I feel we should start maintaining a Wiki for good GSoC (code) and GSoD (docs) projects.
[15:20:18] <sumanah> and toad_polo I'd love to know more about what else is on that long list - I am ignorant of a lot, I am sure
[15:20:51] <toad_polo> In other news, this is a 4-day weekend, and then EuroPython (which has an associated sprint). I'm hoping that at some point in my "time off" I may be able to get around to the PEP 517 proof of concept.
[15:21:49] <pradyunsg> sumanah: I think I'll get ~60% done. I'll skip the prose bits (possibly leave notes of things worth mentioning in those).
[15:22:19] <toad_polo> sumanah: I suspect the `pip` and `warehouse` folks have more of it than we do in `setuptools`. It may be worth maintaining an actual real list somewhere as well that we re-assess bi-annually or something.
[15:23:45] <pradyunsg> We're refering the old-bits as legacy in pip; slowly changing references to disambiguate PEP 517/518 code paths from legacy ones.
[15:23:57] <toad_polo> pradyunsg: I was somewhat disheartened to learn that PEP 517 introduces multiple pain points for the `matplotlib` developers.
[15:24:37] <toad_polo> One of them is just a normal bug that affects a somewhat niche workflow, but people with somewhat complicated builds and build dependencies are kinda the core constituency for the modern build workflow.
[15:25:47] <toad_polo> And they're particularly important QA testers, so unblocking their transition would be nice. I will try this weekend to persuade them to either make issues on the `pip` tracker or packaging.python.org list or comment in the ones that already exist.
[15:27:30] <toad_polo> One of the problems was that it's difficult to build with unreleased versions of Cython and unreleased versions of Python because of the build isolation. That might just be a documentation issue. Another one that I've hit a decent amount is the lack of incremental builds, which might be fixed by the editable install stuff (though it's subtly different in the sense that I may not want an actual `-e` install, but that
[15:27:53] <sumanah> toad_polo: this sounds like stuff I'll want to add to https://wiki.python.org/psf/Fundable%20Packaging%20Improvements
[15:28:20] <sumanah> (I am very open to being told that stuff on that list should be organized differently, new stuff should be there, stuff that's there should be removed, etc.)
[15:28:58] <sumanah> I should say, I mean, _some_ of what you are saying sounds like potentially fundable to fix
[15:29:11] <pradyunsg> toad_polo: I am open to making it possible to override PEP 517 dependencies for certain packages from the CLI but someone has to figure it out.
[15:29:31] <toad_polo> sumanah: Ah, that seems like exactly the list I was looking for. I'll look at this tonight and see about adding the `distutils` thing to it.
[15:30:11] <sumanah> toad_polo: so the PSF wiki page there is not necessarily editable by you .... you could start a similar wiki page on the more open Python wiki though
[15:30:58] <toad_polo> sumanah: Yeah I figured I'd just send you additions if I couldn't edit myself.
[15:31:35] <toad_polo> pradyunsg: Yeah, I think that the Cython thing might be fixable without changes to `pip` by building Cython wheels that would take precedence over anything from `pip`. I haven't played around with it enough, to know how easy that is, though.
[15:34:37] <toad_polo> I have sometimes found that the isolation is a bit annoying when I am trying to do something like construct a specific environment where I run the build (e.g. run it with an editable install of `setuptools` or something), so a more general solution like, "Run the build in this environment that I've already constructed" might work. It could be that `--no-build-isolation` already does this?
[15:37:23] <pradyunsg> toad_polo: Yea.. If someone has a workflows that doesn't work with isolation, --no-build-isolation is an escape hatch for now.
[15:38:32] <pradyunsg> IMO ideally we'd want to improve things so that folks don't need to pass that for almost all use cases.
[16:29:13] <pradyunsg> sumanah: Was trying to figure out how to quantify the "Impact" is tricky; put down a bunch of notes on some (dumb?) ideas for getting some numbers (not sure how usable they'd be). :P
[16:35:45] <sumanah> pradyunsg: is that 14 billion downloads in the last 6 months? my goodness. I knew it was a lot, but the billion-with-a-b makes it super amazingly concrete
[16:44:51] <sumanah> toad_polo: pradyunsg: please do ping me if there's stuff you'd like me to follow up on (other than reading the "migrate distutils into setuptools" issue, which I will do)