Log file Viewer
#pypa-dev logs for Saturday the 22nd of June, 2019
[16:56:58] <sumanah> pradyunsg: I was thinking that maybe you & I could work together on an update you could make to your blog & to #988
[17:32:22] <sumanah> pradyunsg: I created a new issue https://github.com/pypa/pip/issues/6633 for pip support for yanking (PEP 592) -- I tried to apply the right labels but I might have gotten some wrong
[17:33:30] <sumanah> (the method by which I decide whom to root for in a sporting match is probably not nearly as convoluted as pip dependency resolution but it is still silly)
[17:36:49] <techalchemy> pradyunsg: did something significant change about vcs dependencies lately? all my editable installs in virtualenvs with pip fail now if i pass a vcs uri because it seems like it doesn't know how to find the remote urls properly'
[17:37:53] <pradyunsg> techalchemy: Chris is working a fair bit on those, and he's a lot more familiar with that bit of the codebase than me.
[17:38:36] <techalchemy> either way it's already released so i still need to sort out a fix even if it's actually broken on pip's end
[17:39:46] <pradyunsg> techalchemy: pip's issue tracker is a good place to catch his attention, plus it'll be easier to spot with a repro.
[17:41:03] <sumanah> pradyunsg: so I'm realizing that -- in my opinion -- this update probably gonna be more like 3 parts. 1 part on what you've been doing and News For You (like the new distributions package), one para on what is next, and then the "how you can help" section
[17:41:34] <techalchemy> everything works fine when I don't do this all in the test suite so i'm just trying to sort out whether this is an actual problem
[18:26:51] <pradyunsg> sumanah: thoughts on adding https://github.com/pypa/pip/milestone/25 (Improve pip's error messaging) as a fundable project? Being able to get someone with UX experience + dedicated time to work on improving pip's error messaging would be great!
[18:27:26] <pradyunsg> PS: That shouldn't really be a GitHub milestone but it works fine for now. Plus I started that list back when we didn't have "projects" so there's that.
[18:28:33] <pradyunsg> some issues from https://github.com/pypa/pip/milestone/10 are also relevant to that actually.
[18:30:02] <sumanah> pradyunsg: since this is more of a project than a time-based milestone I think you should go ahead and create a Project and put the relevant issues in there
[18:33:03] <pradyunsg> Looking at https://github.com/pypa/pip/projects/1 makes me feel like I've done useful thing, even if all of them aren't user facing. :)
[18:36:16] <sumanah> pradyunsg: go ahead and create a new Project and then if you want to give the user brainwane power to move stuff around into projects within the repo on GitHub, I can start
[18:38:12] <sumanah> pradyunsg: we can leave the issue-wrangling till later and I'll just point to those 2 milestones in the Fundable Projects page
[18:39:40] <pradyunsg> sumanah: some of the issues in the "improve our UX" are pretty broad issues -- things that can be like 2+ months of work; especially the ones near the end of the listing.
[18:40:42] <pradyunsg> sumanah: btw, if you don't have those abilities on GH, it's feedback we should provide to GitHub. :)
[18:49:51] <sumanah> pradyunsg: https://wiki.python.org/psf/Fundable%20Packaging%20Improvements#Improve_pip_user_experience
[19:15:23] <pradyunsg> sumanah: ping! could you give the etherpad a read? (thanks for the copy-edits earlier)
[19:25:29] <sumanah> pradyunsg: great. my edit on line 9, saying the build logic refactor will take multiple months: was that accurate?
[19:32:31] <sumanah> pradyunsg: so I suggest you blog this, and copy it to an update on the GitHub issue (you can do the same thing you did in May, editing the issue description and then commenting with a link to it), and post to discuss.python.org
[20:02:06] <sumanah> techalchemy: oh do you maybe mean https://wiki.python.org/psf/Fundable%20Packaging%20Improvements#Improve_pip_user_experience ?
[20:03:23] <sumanah> techalchemy: ah -- no, that's not just pip -- see https://wiki.python.org/psf/Fundable%20Packaging%20Improvements#Rewrite_virtualenv and the PyPI/Warehouse items on that page
[20:05:01] <sumanah> I'm the person within https://wiki.python.org/psf/PackagingWG who has put energy recently into finding/defining these but I don't have to be the only one
[20:05:08] <techalchemy> well we have a resolver for example, and we could move it under the pypa, since it will ultimately back pipenv, and we were hoping to share code but it seems unlikely
[20:06:07] <techalchemy> i mean me and tzu ping rewrote the entire dependency resolver that underlies pipenv, from the ground up, because needed one and pipenv's is only partially operational
[20:06:12] <sumanah> (also I am oversimplifying my role since a bunch of other Packaging WG members have put a LOT of time in the past few years into defining scope, writing proposals, and so on for grants & getting funding for projects)
[20:07:07] <sumanah> techalchemy: right -- I added the line "minimizing duplication of work between pip and pipenv" to https://wiki.python.org/psf/Fundable%20Packaging%20Improvements#Finish_dependency_resolver_for_pip and linked to https://mail.python.org/archives/list/distutils-sig@python.org/thread/2QECNWSHNEW7UBB24M2K5BISYJY7GMZF/#2QECNWSHNEW7UBB24M2K5BISYJY7GMZF
[20:07:16] <sumanah> and https://github.com/pypa/pipenv/issues?q=is%3Aopen+is%3Aissue+label%3A%22Category%3A+Dependency+Resolution%22
[20:07:29] <techalchemy> but part of why I constantly ask about what should be in the pypa is because like those are probably good targets for things that could be shared, but are likely to fall by the wayside since it seems like they are just going to be reimplemented from scratch if my understanding of the plans on pradyuns side is correct
[20:08:13] <techalchemy> so i'm kind of like, should i work on those, should i work on something else, will pip's things be shareable? I've been told in clear terms that i should never use pip internals
[20:09:31] <sumanah> techalchemy: ok. so it sounds like you have a few different questions, one of which I can help answer and at least 1 other of which I can only gesture at
[20:10:04] <sumanah> but in case you're not finished explaining your questions/thoughts/wonderings, I'll shut up for a moment :)
[20:10:06] <techalchemy> I mean I agree that the funding propositions are important generally but I have no idea if I should even be offering to help or anything
[20:13:15] <sumanah> so techalchemy first off let me say that it sounds frustrating and confusing to be in the situation you're in, and I sympathize
[20:13:40] <sumanah> where you don't want to duplicate work and you maybe already have and it's unclear how to proceed
[20:14:20] <techalchemy> you know i appreciate you and everything you do btw, right? definitely want to make sure that's clear
[20:14:45] <sumanah> thank you techalchemy -- I think I have heard you say that but I am happy to have it confirmed :)
[20:15:09] <sumanah> and techalchemy I am grateful for your work and for opening up this dialogue, as cheesy as that may sound
[20:15:39] <sumanah> I think pradyunsg can speak more about what the options 1, 2, 3, etc. are for the future of the resolver you coauthored, the future of his resolver project, what can learn from what, and so on
[20:18:06] <sumanah> as far as I am concerned, part of the scope for the pip resolver work should also include "figuring out how we coexist with/reusing/reusing code from/integrating/learning from/decommissioning and replacing?/etc." relevant tooling in our ecology, and it sounds like that includes pipenv & its components
[20:19:16] <pradyunsg> techalchemy: Sorry if I've not been communicating this clearly enough on this -- the plan from the start has been to do this in a manner that tools other than pip can the backing logic for the resolver.
[20:19:20] <sumanah> like, I'm sure there are a dozen hacky workaround things out in the world that exist because of deficiencies in the current resolver, and inventorying those, reaching out to those maintainers, learning from them, using and reusing their work ......
[20:20:24] <sumanah> and then there are more robust codebases that are basically implementations of Python dependency resolvers
[20:21:15] <sumanah> and similarly -- the scope for the pip resolver work should, in my opinion, include figuring out what the right way forward is, in coordination with the maintainers and users of those projects
[20:21:26] <pradyunsg> techalchemy: I'm super glad that you and TP have worked on resolvelib/passa, and I don't intend to redo all that hard work. What I'm trying to do right now is untangle pip's internals so that it's easier to slot in those bits, so that code sharing is easier.
[20:22:18] <techalchemy> i mean in all likelihood we will wind up with a new library/libraries for this but i'm fine with that
[20:22:55] <sumanah> techalchemy: this actually kind of reminds me of when Leonard changed up Beautiful Soup so that users could choose their own XML parsers for under-the-hood and BS was the interface
[20:26:52] <pradyunsg> techalchemy: I'm sure you're aware of how convoluted up pip's internals are. It gets almost hacky to work around resolution issues in it since it gets mixed up with the logic in InstallRequirement or RequirementSet; I just want to take out the current resolver properly, without breaking other things.
[20:27:57] <techalchemy> pradyunsg: I always forget RequirementSet exists, that bit is going to be rough
[20:30:28] <sumanah> pradyunsg: once you post your update I'll be able to point to it when people ask questions about this -- in it you explain pretty well what the sequence of next steps is
[20:30:38] <pradyunsg> techalchemy: part of the point of telling folks to not uses pip's internals is to allow us to make drastic changes (like breaking up RequirementSet!).
[20:36:24] <sumanah> techalchemy: so Pradyun is making it so his next resolver update post/comment will address the code reuse issue you brought up
[20:36:29] <pradyunsg> techalchemy: thanks for voicing your concern, glad we're on the same page again. :)
[20:37:19] <sumanah> techalchemy: Pradyun's post is gonna include like 6 "how you can help" bullet points :) and I suspect you are a good candidate for at least 3 of them
[20:38:31] <techalchemy> and pradyunsg, I obviously appreciate all the stuff you are doing too -- you are great
[20:38:40] <sumanah> techalchemy: so the Packaging WG is trying to support -- well I'll just quote the mission from the wiki pg
[20:38:41] <sumanah> The purpose of this working group is to support the larger efforts of improving and maintaining the packaging ecosystem in Python through fundraising and disbursement of raised funds. It largely focuses on efforts such as PyPI, pip, packaging.python.org, setuptools, and cross-project efforts.
[20:40:04] <sumanah> so I don't remember who originally had the idea that Warehouse should apply for a MOSS grant, maybe it was njs or dstufft or EWDurbin or Nicole or me or Nick Coghlan or somebody else, but we clearly needed extra help for the Warehouse rollout
[20:40:58] <sumanah> and then there was this moment in February 2018 where I was like "this project is going great!...... we should get more money because there is so much more that we will still need to do after the deployment"
[20:41:34] <sumanah> and I did some research and I told the rest of the WG: I think we're eligible for Open Tech Fund, and this Facebook Secure The Internet grant, plus one other thing
[20:42:16] <sumanah> and we put together some proposals and we got the OTF money, and Lukasz and Cooper helped us get Facebook money, and the third agency said basically: sorry, not right now, but you seem cool and we'll keep you on file and maybe ping you later
[20:43:26] <sumanah> techalchemy: so, in retrospect, I am having the same moment now as I did in Feb 2018: "the current funded project is going well!..... we are gonna need more money because Python packaging/distro has so many unmet needs! As I freshly remember as I dive back in!"
[20:44:03] <sumanah> and so I am, like, emailing people I last saw at conferences last year, or worked with years ago, because they are at these funding agencies and we actually have a shot
[20:44:28] <pradyunsg> techalchemy: I'm unsure on whether I should mention passa directly in the post -- do you have a preference?
[20:46:05] <pradyunsg> sumanah: ping, could you take a look at what I wrote? I moved a para to "waht's next"
[20:46:07] <sumanah> techalchemy: and when I started building this list of fundable Python packaging/distro things-that-need-doing, I started with the stuff that I already knew about
[20:47:41] <sumanah> pradyunsg: I think you should explicitly say the thing you said before in chat, something like "so that it's easier to slot in those bits" or "to reuse the resolvers they wrote" or something
[20:49:27] <techalchemy> and it makes no sense that everyone works for free and that is supposed to maintain all of the infrastructure around python
[20:49:29] <sumanah> pradyunsg: some free advice: be explicit and specific, and use the active voice ("x does y") instead of the passive voice ("y is implemented") and it's totally fine that it sounds less formal.
[20:51:21] <sumanah> pradyunsg: cool :) I've made an edit to be clearer on line 13 -- edit it in case it's inaccurate
[20:53:05] <sumanah> techalchemy: the imbalance, of the importance of this infrastructure and the "no one was getting paid for this" shoestring volunteer maintainer pool, was unsustainable. now we have more of a mix of paid & unpaid and I do want to be careful in case it causes heartache, unmet expectations, etc
[20:54:17] <sumanah> techalchemy: so I _started_ listing fundable projects I already knew about, but of course there's room for more and I want to learn more about how the WG can support different parts of the ecology
[20:57:09] <sumanah> pradyunsg: line 13 now contains more questions I have. truly the ancient superstition around the number 13 is rearing its head
[21:00:33] <sumanah> techalchemy: and getting existing maintainers paid to work on this stuff -- well, with my Packaging WG hat on, I say, that's why when we get grants, now we put out RFPs like https://github.com/python/request-for/blob/master/2019-Q1-PyPI/RFP.md , and eventually if we get enough sponsors/donations then we could support more of that work on an ongoing basis, and I'm glad Ernest is a fulltime staffer
[21:01:00] <sumanah> techalchemy: but with my Tidelift hat on (I do like 1 hr/month of consulting for them) I'm like: hope that takes off! that'll help
[21:05:12] <sumanah> Looking now pradyunsg and not annoying at all! I'm sorry that I maybe did something annoying by asking you to write this and thinking it would be fast
[21:08:16] <pradyunsg> Plus, the potential for miscommunication with techalchemy pumped enough adrenaline into my bloodstream, that I'm not exactly sleepy right now. :)
[21:10:13] <sumanah> pradyunsg: it sounds like we aren't 100% certain whether the future includes "the new 'pip resolver' is actually us calling out to/reusing the pipenv resolver" or "we mix and match some new code with the existing pipenv resolver code" or "pip encapsulates/defines a resolver interface and lets you the user Choose Your Own Resolver Adventure" or what. so my requests for more specificity may have been off
[21:15:47] <pradyunsg> sumanah: it's mostly gonna be the last one -- where resolvelib defines the interface, and some of the bits and pieces for the adventure are shared by different tools.
[21:16:57] <sumanah> pradyunsg: go ahead and Ship the update as written in the Etherpad and we will wordsmith this nuance and find someplace reasonable to say it
[21:34:34] <sumanah> great! pradyunsg will you also point to it in https://pradyunsg.me/gsoc-2017/ ?
[21:49:26] <sumanah> pradyunsg: well if you _felt_ like also notifying distutils-sig and pypa-dev I think that would be a very nice thing of you to do
[21:50:16] <sumanah> whether you notify, like, Twitter and TikTok and so on is between you and your branding agency
[21:51:18] <sumanah> pradyunsg: I would wager that there are people who care about this who _commented_ on #988 and yet unsubscribed from it because it felt noisy
[22:18:56] <techalchemy> pradyunsg: no miscommunications here btw just needed to try and understand where things are heading
[22:20:02] <techalchemy> also I do not in any way keep up with any of the numerous channels of communication on this topic but they are centered in like, pip's issue tracker (which I don't follow even if I am subscribed) or discourse (which is usually too much wall-of-text style)