Log file Viewer
#pypa-dev logs for Friday the 10th of April, 2020
[09:03:56] <pradyunsg> techalchemy: looks like a badly done file paths comparison. Feel free to file an issue on pip. I think the fix is trivial enough to release in pip 20.1.
[09:29:32] <travis-ci> Change view : https://github.com/pypa/pip/compare/c7bde5bf88c7...9360793a6c46
[11:27:39] <travis-ci> Change view : https://github.com/pypa/pip/compare/9360793a6c46...6c97645e2fe6
[11:47:57] <pradyunsg> techalchemy: I have good news for ya -- https://dev.azure.com/pradyunsg-azure/pipenv/_build/results?buildId=514&view=logs&j=2d4cb656-6149-5a33-f8b2-fd60a79c32e2&t=1bba0ae4-9f23-5a5e-e84f-b954474a96bc&l=29. Running the full test suite now. :)
[12:13:38] <pradyunsg> https://dev.azure.com/pradyunsg-azure/pipenv/_build/results?buildId=515&view=logs&j=2d4cb656-6149-5a33-f8b2-fd60a79c32e2
[13:19:27] <pradyunsg> techalchemy: I spent a few minutes to clean up the names, to make this make it nicer to read / understand the names -- https://github.com/pradyunsg/pipenv/pull/1/checks
[13:58:10] <travis-ci> Change view : https://github.com/pypa/pip/compare/6c97645e2fe6...81f1054865e0
[13:59:40] <travis-ci> Change view : https://github.com/pypa/pip/compare/9cbe8fbdd0a1...8cca170ceb40
[15:06:08] <pradyunsg> techalchemy: it was probably pipenv run; I'll be trying that tonight... Couldn't figure out a good way to directly invoke pytest from the venv. If I were to guess, it's due to some CLI-colors logic for the "enabling .env" command.
[15:11:39] <techalchemy> https://dev.azure.com/pypa/pipenv/_build/results?buildId=21742&view=results
[15:12:27] <techalchemy> https://github.com/pypa/pipenv/pull/4169/commits/e82b9ce083591a8bd48bff4140f939496b099229 <- also switched to using cmd
[16:17:24] <pradyunsg> https://github.com/pypa/pip/blob/master/.azure-pipelines/steps/run-tests-windows.yml#L31
[16:19:19] <techalchemy> @ pradyunsg the pip issue is so weird, the one with removing editable installs when PIP_EXISTS_ACTION="w"
[16:19:41] <techalchemy> my workaround is currently to just listen for the relevant error msg and rewrite all the egg links in the environment
[16:37:00] <travis-ci> Change view : https://github.com/pypa/pip/compare/3764736f20ef...5c9e83a3aadd
[16:46:51] <techalchemy> btw i stole that GA implementation from cooperlees so if you're riffing on it to make yours you should thank him :p
[16:51:00] <pradyunsg> I sure hope that there's a lotta stuff folks on that team would want to change if backwards compatibility wasn't a thing. :P
[16:51:26] <techalchemy> considering how much they break i'm not sure anyone is that concerned with backward compatibility anyway
[16:55:29] <techalchemy> pradyunsg, yeah i just realized that recently, we used to be capped at 10 at a time
[16:57:01] <cooperlees> RIghtio - Let me know if you want to see if we can ask Sean for some favors. I feel the last 5 major releases of Mac OS X is enough for pip to be tested on ...
[16:57:21] <techalchemy> i seriously don't understand how they managed to build CI infrastructure that is so good but so hard to use
[16:58:00] <pradyunsg> cooperlees: no no... We test multiple versions of Python on MacOS. And run linters, docs etc as well.
[16:59:15] <pradyunsg> cooperlees: I'm sure 100% of pip's would. I just wanna make sure someone working on pip on a single OS doesn't break things like that for folks in other OSes.
[16:59:55] <cooperlees> pradyunsg: I get it, but I feel running lint and different platforms is overkill
[17:00:08] <techalchemy> if anyone files an issue just tell them it works on your machine and close the issue
[17:00:41] <pradyunsg> techalchemy: soon enough, we'll give up to our automation overlords and hand over responsibility for code formatting.
[17:01:25] <techalchemy> massive project with many contributors, seems like it'd be really important to have a consistent style/format
[17:02:03] <techalchemy> i dont need to be all over reddit like 'packaging authority hates its own users' or something
[17:02:40] <techalchemy> that's pretty nice for pip, it takes us 20 mins before we can even run a test :p
[17:02:51] <pradyunsg> Just need to add the "don't run tests on docs-only changes" and other optimizations like that.
[17:03:03] <cooperlees> here is my fav azure screen shot of late from Steve Dowler (a m$ employee): https://twitter.com/zooba/status/1247939567593938944
[17:03:16] <pradyunsg> techalchemy: no shit. It's like, an 8 minute setup for tests. Please start caching stuff.
[17:06:08] <techalchemy> our tests are crap, we really need to re-engineer the test suite from scratch i guess
[17:06:09] <pradyunsg> https://docs.microsoft.com/en-us/azure/devops/pipelines/release/caching?view=azure-devops
[17:06:55] <techalchemy> i mean, that's a separate issue from the speed issue, they are related but that's not the reason the tests need to be redone
[17:07:02] <pradyunsg> I don't understand why you can't do "setup, ramdisk, pipenv install, pipenv run"
[17:08:22] <techalchemy> setup ramdisk, setup environment variables, install dependencies, run tests
[17:10:47] <pradyunsg> EWDurbin, @di, @dstufft, sumanah: OK to add pypi.org/sponsor as pip's GitHub Sponsor button?
[17:13:29] <EWDurbin> my understanding is that we can set the default sponsor button and then projects can override if they have other funding, is that correct pradyunsg ?
[17:16:29] <pradyunsg> EWDurbin: I don't think there's any reason to believe anyone would be opposed, but I'll leave it open for a bit to let folks chime in.
[17:17:47] <EWDurbin> the packaging-wg discussion was related to using the sponsors button where GH accepts $ on projects behalf
[17:18:38] <techalchemy> what are the mechanics of that? should everyone add buttons ? where will the $ go? PSF?
[17:19:59] <EWDurbin> 1) the "sponsor" button that a project can add via the funding.yml like https://github.com/pypa/pip/pull/8015
[17:19:59] <EWDurbin> 2) the "GitHub Sponsors" program where money is collected by GH and sent to a bank account.
[17:20:14] <pradyunsg> EWDurbin: yea. For the repo button, we can have a whole bunch of platforms that get linked to from there (including GitHub Sponsors, Patreon, OpenCollective and more).
[17:21:52] <pradyunsg> EWDurbin: so, use the repo sponsor button to accept money via GitHub Sponsors... Correct?
[17:21:52] <EWDurbin> the PyPA org is not setup with that yet. that's the discussion (as yet unresolved) on packaging-wg
[17:29:06] <pradyunsg> In case anyone wants to see the announcement of the "GitHub Sponsors for organizations" blog post: https://github.blog/2019-11-13-universe-day-one/#sponsors
[17:44:14] <travis-ci> pypa/pip#15626 (add-sponsor-button - f19f887 : Pradyun Gedam): The build passed.
[18:32:59] <pradyunsg> techalchemy: can we change packagebuilder and installer to be non-pipenv specific and become general libraries for all of Python, by moving code into them?
[18:33:23] <pradyunsg> techalchemy: I'd *really* like to spend some cycles to do this sometime soon-ish.
[18:35:32] <techalchemy> pradyunsg, added to both, both have repos in sarugaku, feel free to do whatever you want with them / move them to pypa
[18:54:23] <sumanah> I just happened to come into this discussion and it sounds like you are talking about multiweek projects that it would be nice to find funding for!
[18:54:58] <sumanah> "change packagebuilder and installer to be non-pipenv specific and become general libraries for all of Python, by moving code into them" being one. right?
[18:55:16] <techalchemy> 'change packagebuilder + installer to exist and become general libraries...
[18:56:08] <sumanah> techalchemy: maybe you could kind of start from the beginning and tell me the story of passa? I think I am missing something
[18:56:20] <techalchemy> that was an iteration me + TP did to rewrite the resolution logic of pipenv from scratch
[18:57:04] <techalchemy> basically we spent a few months and started from scratch putting a majority of our effort into rewriting the logic behind pipenv
[18:58:54] <sumanah> OK. So, https://github.com/sarugaku/passa dates from August 2018-Oct 2019 (with work mostly in 2018)
[18:59:22] <pradyunsg> sumanah: more like, "create reusable libraries for shared implementation of package build and install logic for Python Packages"
[18:59:25] <techalchemy> TP was still sort of new to the whole project, so I did a first pass of a backtracking resolver which i guess we are all calling pubgrub now or something like that since it doesn't sat solve and it doesn't require every constraint
[19:00:04] <techalchemy> my implementation still relied heavily on pip internals and was kind of thrown together in requirementslib
[19:00:17] <pradyunsg> It's just that we have `packagebuilder` and `installer` on PyPI and they're... good names for packaging-related libraries for once. :P
[19:00:50] <techalchemy> TP rewrote it into resolvelib and we built passa to be the CLI wrapper to manage projects etc, and to try to stop using pip internals
[19:01:32] <sumanah> techalchemy: I had to take a second and verify my understanding that Pubgrub is a dependency resolution algorithm for which there are many implementations
[19:02:04] <techalchemy> sumanah, i'm not 100% sure it translates directly, but maybe pradyunsg knows
[19:02:13] <sumanah> techalchemy: is "passa" actually being used, by pipenv or by any other Python tool, as far as you know?
[19:02:41] <pradyunsg> techalchemy, sumanah: they're not the same -- passa's algorithm and pubgrub. :P
[19:02:59] <pradyunsg> FWIW, I'd started refactoring pip's code/logic that handles installing and building, to make these libraries a real thing, so this is in a similar state as the resolver... except much smaller and it's actually 2 separate projects.
[19:03:14] <sumanah> I think I need one more "step back and tell Sumana the story starting in, like, 2015"
[19:04:15] <pradyunsg> sumanah: basically, https://discuss.python.org/t/building-distributions-and-drawing-the-platypus/2062 prompted me to decide -- fine, we need a single sharable implementation of the build logic stuff.
[19:04:49] <pradyunsg> not exactly, but definitely made me be certain that it's a good idea -- plus it's the easiest thing to point to. :P
[19:06:04] <sumanah> here's what I actually need: I need an inventory of the ~15 different GitHub repositories that we are talking about here. Some of them are in https://github.com/sarugaku , there is https://github.com/pradyunsg/zazo , are there any others?
[19:06:36] <pradyunsg> And, I realized install logic would be a much easier target to tackle in like, a week or two. So, I started refactoring both of them (they came out of the big blob of technical debt of pip, which is `InstallRequirement`).
[19:07:29] <pradyunsg> And then they both got shelved once after life hit me, and by the time I'd returned with a big enough chunk of time, CZI+MOSS funding happened. :)
[19:07:33] <techalchemy> sumanah, 'packagebuilder' and 'installer' are spinoffs of 'passa' which we started because we were looking to rework pipenv but were constrained by design decisions that had been made there
[19:07:57] <sumanah> techalchemy: when you say "spinoff" what do you mean? Also could you please link to them?
[19:08:23] <pradyunsg> Today's discussion was prompted by https://discuss.python.org/t/sans-i-o-implementation-of-pep-503-simple-repository-api/3862/4.
[19:08:56] <techalchemy> https://github.com/sarugaku/installer https://github.com/sarugaku/packagebuilder https://github.com/sarugaku/passa
[19:09:59] <techalchemy> <techalchemy> TP rewrote it into resolvelib and we built passa to be the CLI wrapper to manage projects etc, and to try to stop using pip internals
[19:10:17] <sumanah> techalchemy: help me understand Sarugaku as an org. Who started it, and should it still exist or should it be merged into PyPA?
[19:10:57] <travis-ci> Change view : https://github.com/pypa/pip/compare/5c9e83a3aadd...4416e88ef13d
[19:11:55] <pradyunsg> sumanah: since these projects are dependent on pip's internals, and I think we should wait until they're things pip uses to implement itself, before we move them to github/pypa.
[19:12:37] <pradyunsg> eg: resolvelib should move into pypa before the "stable" release of the new resolver.
[19:13:01] <sumanah> As far as we know, are there any other GitHub orgs that are like sarugaku? I mean, other orgs that also house/work on/collect Python dependency management/packaging/etc. tools/projects? I figure I may as well try to get a bird's eye view
[19:15:48] <sumanah> techalchemy: I see that the Netlify page says "The team is funded by Pipenv maintainers Dan Ryan and Tzu-ping Chung" - may I ask more about that?
[19:16:50] <techalchemy> i mean, in kind contributions count as contributions for every purpose i can think of
[19:17:38] <sumanah> I'm not saying you are actively looking to mislead anyone. I am saying it's confusing and, in my opinion, misleading.
[19:18:56] <sumanah> techalchemy: at this point it's clear you and I disagree, and I don't want to take up too much time here with that. happy to talk in another venue about it. Separately: I think I better understand its purpose
[19:19:39] <sumanah> a greenhouse... it's a set of more experimental projects that may make their ways into pipenv or other projects
[19:45:34] <techalchemy> sumanah, it's like 50/50 some of the projects are actively in use (resolvelib etc) but sarugaku basically was meant to be a place for us to spin up new projects to support packaging work
[19:47:37] <sumanah> techalchemy: ok. Is Sarugaku a legal entity -- does it have its own legal existence in any country?
[19:48:45] <sumanah> when I hear that a team is funded by [source], that really sounds like there are people, in a team, who are doing work, and who are being paid to do that work
[19:49:15] <sumanah> if I hear that a PROJECT is funded by [source] then I think maybe there is money headed to people, but maybe there is money flowing to, for instance, servers, etc.
[19:53:38] <sumanah> separate note techalchemy - here is a PR to fix a few other bits https://github.com/sarugaku/site/pull/1
[19:54:14] <techalchemy> sumanah, TP wrote and published the site so i'll probably prefer he looks it over
[19:55:49] <techalchemy> but given the team is made up of us and it also says we fund it I tend to think it was not that odd. We have covered costs as needed and certainly covered plenty of time which as you know is quite costly
[19:58:07] <techalchemy> not like we were paying ourselves to do open source, anyway, but i dont mind if it says something else
[19:58:16] <sumanah> I think that time is valuable, and I think it deserves for us to value it. I think that wording on this stuff can be tricky and I am happy to get in a conversation with you & TP about wording if you would like my input
[19:58:50] <techalchemy> ultimately I don't feel that strongly, I just tend to think it doesn't matter that much
[19:59:17] <sumanah> techalchemy: understood. back to me better understanding the sort of incubator work
[20:00:11] <techalchemy> (basically, if you think it's really an issue which it seems like you do, i have no problem with changing it, it's not like we're sitting here pouring money into something)
[20:02:02] <sumanah> techalchemy: so is there a way, from the GitHub repositories/descriptions, to better understand which projects are actively in use and which are more experimental?
[20:06:21] <techalchemy> sumanah, things toward the bottom are not in active use anything below pythonfinder isn't afaik)
[20:07:12] <sumanah> so it sounds like the answer is: implicitly, things that haven't been touched in a while are not in active use. But there is no explicit badge/explanation in READMEs or descriptions
[20:07:28] <sumanah> techalchemy: are you in the middle of pipenv work right now? I can take care of some of this myself
[20:52:37] <techalchemy> sumanah, actually i lost track of the tab with that, its' been a busy day for some reason
[20:53:02] <sumanah> techalchemy: lemme know when you have re-found it and we can catch up about what's the next step
[20:53:33] <techalchemy> some tests seem to randomly fail still but the majority of issues are solved
[21:44:15] <alanbato> pradyunsg: You mentioned async uploads to pypi regarding the issue about uploading unpublished packages to PyPI. I think that both features are complimentary and not exclusive, you can have one without the other, but implementing both will provide the best experience.
[21:45:30] <alanbato> For context on the comment above: https://github.com/pypa/warehouse/issues/726
[22:18:01] <tos9> Oy: https://github.com/actions/cache/blob/master/examples.md#using-a-script-to-get-cache-location
[22:42:47] <travis-ci> Change view : https://github.com/pypa/pip/compare/5c9e83a3aaddd5ada7b556125a0cffcebbd5b493...4416e88ef13d43b0178ac1697a2d2091eef80797
[22:54:21] <pradyunsg> @di, alanbato: I agree both would be good to have and aren't competing but complementary; I was mostly just wondering what di's thoughts were on the async uploads situation, but didn't wanna be pushy in asking. :)
[23:04:23] <alanbato> alanbato: Well, I don't have much info on async uploads, but if you have any q's about how that relates to what we're trying to do I'd be happy to answer them
[23:05:46] <alanbato> Something to note is that this shouldn't need any extra work on the client's side (like pip). At least for the time being.
[23:37:21] <dstufft> Hmm, what's the point of the date on Release.published, and the boolean on Project.auto_publish_releases-- I assumed that published would just be a boolean, and tht all projects would auto publish, and there would be a value you pass in the API to tell it not to immediately publish
[23:48:37] <alanbato> @dstufft: We thought that if we are going to make creating a release and publishing two separate steps, we needed two different timestamps. As for the is_published boolean, we can already know if a project is published by checking if it has a publish date, so we don't need the extra field.
[23:50:56] <alanbato> As for the auto_publish_releases, we wanted to make this opt-in for developers through their project's settings page. This would be True by default. If we relied on an API argument to determine if it's auto published or not, we would need the clients to add this option too for mantainers to actually use it. By leaving this responsability solely on the backend, clients get this feature for "free"
[23:52:10] <alanbato> While I agree that an option would give more control, we thought we could implement that in the future alongside with the upload API / the async uploads
[23:56:59] <alanbato> This is still in the idea stage with a tad of design, so your questions are really helpful, thank you.