Log file Viewer
#pypa-dev logs for Wednesday the 13th of May, 2020
[04:20:06] <McSinyx[m]> currently I'm using cibuildwheel and it kind of automate the wheel repairing part (auditwheel/delocate), however there's no equivalent on Windows AFAIK and delocate doesn't work for standalone modules
[04:21:19] <McSinyx[m]> basically building cross-platform, independent wheel is still a pain and there needs specific hack for each situation, and it'd be really nice when we have the build tool we can generalize that
[04:21:30] <toad_polo> Yeah, conda is basically a package manager, it has dependencies in all kinds of stacks, including C/C++.
[04:22:07] <McSinyx[m]> toad_polo: thanks that's good to know, so basically we can package C/C++ stuff there too?
[04:24:39] <toad_polo> https://github.com/xtensor-stack/xtensor/blob/master/README.md#package-managers
[04:30:04] <McSinyx[m]> it's good to know, although it might not be a solution for me (my module depends on a bunch of audio codecs and openal, it'd be too much for me to maintain all those)
[04:30:24] <McSinyx[m]> gutsytechster: : thanks for the congrats, I could not find your name under PSF projects
[04:30:57] <McSinyx[m]> I think you have other plans for the summer, but it's nice still seeing you on GitHub!
[04:37:39] <McSinyx[m]> sumanah: congratulations on WisCon, I'm a bit curious how auctions works with the latency when it's hosted online though d-:
[09:53:41] <travis-ci> Change view : https://github.com/pypa/packaging/compare/fbe51442f084...db291c7bdac5
[11:54:19] <travis-ci> Change view : https://github.com/pypa/pip/compare/0adeeef51e68...d911a9fcb8d2
[11:57:23] <pradyunsg> techalchemy: pipenv’s vendored pythonfinder tries to use `backports.functool_lru_cache` from the user’s environment. This results in pipenv not working (you’ll see more information on Zulip, from ei8fdb)
[13:59:28] <techalchemy> pradyunsg, pipenv doesn't do anything before it adds its own vendored libraries to the path
[13:59:52] <techalchemy> was leaving out the 's' in 'functools' a typo or is that actually what pythonfinder is doing
[15:36:41] <sumanah> toad_polo: I just read logs and my eyebrows went up when I saw you say "I'm considering jumping ship from PyPA tooling and going all in on conda. "
[15:38:00] <sumanah> I would love to know a bit more about that (and how close you are to making that decision) when you have time
[16:23:54] <toad_polo> sumanah: I haven't really used conda very much, so I can't really say that I'll like it more.
[16:25:01] <toad_polo> But it does seem to have had a lot of features that PyPA tooling has had on the "long list" for ages.
[16:25:25] <toad_polo> I also think there's a greater focus on reproducibility, which is something that I'm evidently more concerned about than other PyPA folks.
[16:25:36] <sumanah> toad_polo: would you be switching to also _working_ more on conda-y things and not working on PyPA tools any more?
[16:27:09] <toad_polo> Conda tools are more about package management and installation and such, so it would be a replacement for pip and possibly other things like pep517, maybe twine as well? Not sure.
[16:28:53] <sumanah> of course I would miss you as a pip/etc. user but I hope you find tooling that works well for you!
[16:29:28] <sumanah> and if you write out a brief list of the features/focus that make you want to switch, I would try to circulate it to help spur a bit more focus, funding, etc
[16:36:11] <toad_polo> From what I can tell from the feedback I'm getting on some of these issues, `pip` is not interested in supporting my use case.
[17:30:15] <dstufft> toad_polo: conda doesn't support either of those does it? afaik conda doesn't support anything but their version of installing from a wheel
[17:30:46] <toad_polo> Ah, the long list stuff I was talking about was support for native dependencies.
[17:34:05] <dstufft> yea, native dependencies is a fairly fundamental thing that we're likely to never fully support even if we hack it in some way, and conda is a reasonable tool to handle those in
[17:34:42] <dstufft> (I personally don't like conda for them, but that's a personal thing it's a reasonable thing to use in general)
[20:21:33] <travis-ci> Change view : https://github.com/pypa/pip/compare/0adeeef51e68...d911a9fcb8d2
[21:16:06] <toad_polo> dstufft: That issue you're talking about with not encoding enough information, that's specific to extension modules?
[21:17:08] <toad_polo> Yeah, I just didn't know if there was an issue with `.pyc` files or something.
[21:18:10] <toad_polo> I'm not even convinced that's a safety issue, you get the same thing if you don't clean artifacts from any build system, but if it's possible to encode that information in the file name I don't see a problem with it.
[21:18:51] <toad_polo> I thought the problems with the build/ directory were not related to that, though. Stuff like pytest picking up random crap in there or whatever.
[21:19:49] <dstufft> toad_polo: well the problem comes down to pip can't clean it because we don't know when to clean it nor do we know *what* to clean in abstract (likefor setuptools we do, but in the general case of "a build backend" we don't)
[21:20:10] <dstufft> there was a number of problem with in-place builds that are leading us to revert it atm
[21:20:22] <toad_polo> Sure. That was why I was saying that front-ends should be able to pick a directory they can clean.
[21:20:53] <toad_polo> setuptools probably needs it the most or possibly it's the only one that needs it, because setuptools is the only one that actually supports C extensions.
[21:20:53] <dstufft> the pytest build thing is one of them, but that's not really a safety issue it's just an annoying interaction
[21:21:17] <toad_polo> I guess enscons does in some way I don't understand, and they may have similar problems or not support incremental builds, I dunno.
[21:21:40] <toad_polo> flit doesn't, and poetry just basically uses distutils (which is the same thing anyway).
[21:22:02] <dstufft> I tested scikit-build the other day and their demo project just flat our errored for me if I didn't manually delete the build directory after each invocation
[21:22:38] <toad_polo> I mean poetry doesn't use anything, it just says, "Hey you can specify this .py file to run as part of the build"
[21:23:30] <dstufft> toad_polo: the other question is when does pip clean that build directory, how do we know when the cache has been poisoned and we now need to clean it
[21:26:05] <toad_polo> But having a `pip --clean` (or even the inverse, `pip --incremental`) to force invalidate (or prevent invalidation) would help.
[21:26:51] <toad_polo> Because even proper build systems with a bunch of people working on them have problems with build cache invalidation, and it's pretty common to have to run `make clean` or `git clean -idx` to do something in a reproducible way.
[21:28:31] <toad_polo> I dunno, the more I dig into this stuff, the more overwhelmed I am with how much work needs to be done to do anything reasonable.
[21:32:47] <toad_polo> I'm sort of tired of being the guy who explains to complaining people that "we're working it but it takes time", particularly when it seems like in fact no one is interested in working on any of it and there's effectively zero agreement on a direction.
[21:40:30] <dstufft> toad_polo: and this is the general thrust of the issue is that the pip developers generally prefer reproducable installs over "fast" / "incremental", which is basically what we've always done (sans the one edge case where you could do an incremental build out of band of pip before invoking pip and then that would get reused). So to us this isn't anything new with pip
[21:42:53] <toad_polo> dstufft: Well I may have been under a misapprehension about where the ecosystem was heading.
[21:43:55] <toad_polo> The real issue is that `pip` is really the only general-purpose PEP 517 front-end in town.
[21:45:30] <dstufft> toad_polo: I shouldn't make it sound like the pip developers are some unified block here tbh, this certainly represents my view, and It hink Paul's. I think we have one developer who does think we should support incremental builds
[21:45:55] <toad_polo> So we're pushing hard for PEP 517 / 518 and trying to get people into PEP 517 workflows.
[21:46:06] <dstufft> but I do think that pip being the only genral purpose PEP a517 frontend around is a bit of a drag
[21:47:01] <dstufft> to be clear, I think that PEP 517 enabling incremental builds is a wholly good thing
[21:48:13] <toad_polo> I really doubt that people will adopt anything other than `pip` for doing dev installs.
[21:49:15] <toad_polo> "Use `devpip` if you need incremental installs when doing development, otherwise use `pip`"? People aren't going to want to have a whole other tool around just for this one minor variation on exactly what `pip` does.
[21:49:30] <dstufft> it certainly feels to me like if there was a front end more focused on building / managing the development life cycle, and leaving pip more to installing published packages that we could better serve both cases
[21:50:16] <dstufft> like you'd imagine it'd have support for like invoking the build sdist / build wheel commands
[21:51:07] <dstufft> and yes, editable installs are something else I considered when I was thinking about this just now
[21:51:31] <dstufft> To be clear, I think wrestling with this question is something that the pip developers have beentrying to do for awhile now
[21:52:43] <dstufft> Well, I guess I mean more determining how wide we draw the circle around supported and not
[21:53:47] <toad_polo> I think if `pip` continues to have the ability to build from source and not from built artifacts, it should have the ability to use incremental builds.
[21:54:24] <toad_polo> It's a pretty basic feature and if `pip install .` remains possible, it will completely crowd out any other tool in the same space.
[21:55:12] <toad_polo> Particularly because a good chunk of all packages don't need incremental builds because building them is almost instantaneous, but a good chunk of the most important tools do need incremental builds.
[21:56:13] <toad_polo> Right now we're in this terrible world where `matplotlib`, `numpy`, and `pandas` need for example PEP 518 especially more than most, but their use case is horribly under-served because it's "too complicated"
[21:57:16] <toad_polo> I do see that PEP 517 is super useful for the "simple project" use case because the defaults for `flit` are a lot less tricky.
[21:58:38] <toad_polo> I dunno, arguing or caring about packaging is not really good for me, I think.
[21:58:54] <dstufft> Yea I don't know. Like I said where we draw the line is tough. One could argue we should only support isntaling from wheels and eschew building all togehter, you could also argue that building a wheel from a sdist and then installing that is close enough that we should support it... then it's like "well what about simple installs from file system that don't require any advanced build features" to "well if you're gonna do that you should support
[21:59:29] <dstufft> I think if we werelooking at adding something like editable today, pip wouldn't add support for it
[22:00:07] <dstufft> and it's just too wideapread a feature to take it back at this point without a lot of effort nbody has done
[22:39:08] <tos9> I don't know if this is helpful (let alone correct), and god do I hate this sentence but -- from a marketing perspective I still think it'd Look Nice if the next hot tool was a simplifying tool rather than a componentizing one
[22:39:51] <tos9> (like tool that can handle all steps in the process even if it doesn't do that for all cases)
[22:40:17] <tos9> but just wrap the tools that people in the pypa ecosystem are already using together, cover the missing cases if there are any, and now boom everyone has better high-level tool than they started with
[23:08:04] <travis-ci> Change view : https://github.com/pypa/pip/compare/89f248c7bfdae65c911e4efc5398823dc5d54c28...d911a9fcb8d29ed560ba0ccb16f83f914a37a2a7
[23:12:27] <techalchemy> dstufft, fwiw i think it's the right call to avoid breaking existing workflows, but i'm sorry for the fact there can't be an easy answer
[23:16:47] <techalchemy> I do agree with the point about the scientific libraries which require more extensive build support, that universe is definitely underserved -- and I have no issue with adding modularity or moving individual pieces of functionality into their own tools, but i tend to agree with tos9 that even if the functionality isn't built into pip it should likely be included/consumed by pip for presentation to users
[23:37:54] <dstufft> techalchemy: yea I mean, it's possible it's too late to do anything about pip being "the" tool that has to support every even half way popular thing people want to do (like incremental builds or what have you)
[23:39:16] <dstufft> I think the overarching point I was trying to get across is mostly even among the pip developers we dont really have consensus for what exactly pip's place in the ecosystem is
[23:40:42] <dstufft> and the answer for "should pip do this, or should some other tool" in large part depends on the answer to that question