Log file Viewer
#pypa logs for Wednesday the 6th of February, 2019
[20:43:42] <dude-x> i have a setup.py that is "littering" my site packages with the directories of the thing i am trying to package properly
[20:48:31] <ngoldbaum> dstufft: in my case it's packages who use versioneer, e.g. https://travis-ci.org/ngoldbaum/yt/jobs/489710257
[20:48:50] <ngoldbaum> if you expand the build log, you'll see when it tries to install cartopy it barfs
[20:53:00] <ngoldbaum> toad_polo: nope, ultimately because of a race condition in a C++ library it depends on, https://github.com/SciTools/cartopy/issues/805
[20:53:26] <ngoldbaum> they're pretty happy mostly supporting conda though so most of their users don't hit it
[20:54:44] <toad_polo> I mean there's other ways to fix it, but the problem is that they are assuming `setup.py` has its directory in the path, which is not a safe assumption.
[20:56:44] <toad_polo> Probably someone should see if versioneer can work without being actually in your build root, like as a declared dependency.
[20:56:56] <toad_polo> Because that's an even better way to fix it for all the people who get broken by this.
[20:57:19] <ngoldbaum> and people vendor it anyway so updating versioneer doesn't really help all the packages out there that use it
[20:58:15] <toad_polo> But my point is at some point people are going to actually start opting-in to setuptools.build_meta, and "add the local directory to the path" is not the only option we want to present, because it's not great.
[20:58:40] <toad_polo> But presumably the reason everyone vendors versioneer is because PEP 518 wasn't a thing.
[20:59:37] <toad_polo> So it's a two-birds, one-stone thing. And the nice thing about the fact that PEP 517 removes the source root from the PYTHONPATH is that if you keep vendoring versioneer but also add it to your build dependencies, setup.py will pick up the right one if you're in a PEP 517 build by default.
[21:14:44] <toad_polo> ngoldbaum: I +1-ed it if you want to name-check me, but I also subscribed already.
[21:16:26] <toad_polo> Though actually if you can edit out the IRC name from the GH post that would be nice. I haven't had the greatest opsec in the world, but I try not to link my IRC handle and my real name terribly publicly.
[21:16:57] <toad_polo> At some point I'll maybe have a workflow that allows me to have a separate IRC window where I do "real name" stuff :p
[21:18:26] <dude-x> found the problem, i was setting `package_dir` to my directory but that's only good if you have a src directory that should be excluded.
[21:23:34] <gsnedders> (I say that jokingly, except I'm about to do that… Decided to keep a "G" as a middle initial to avoid changing IRC nick…)
[21:27:56] <ngoldbaum> ultimately this is coming from me doing "pip install -r tests/test_requirements.txt", where test_requirements.txt looks like https://github.com/yt-project/yt/blob/master/tests/test_requirements.txt
[21:32:18] <ngoldbaum> https://github.com/SciTools/cartopy/pull/1264/commits/454a9118f711d111b36026b126724cdb8dd705e7
[21:32:51] <ngoldbaum> apparently there's some issue with pip reinstalling numpy if it's present in requires but is already installed on the build machine
[21:34:23] <toad_polo> pip creates an isolated build environment that starts empty, so it doesn't matter what you have installed.
[21:36:43] <ngoldbaum> toad_polo: i think this is the issue https://github.com/pypa/pip/issues/4582#issuecomment-313208565
[21:40:49] <toad_polo> They don't need a specific version of numpy, they need it to match the version of numpy that exists already in the system.
[21:42:05] <toad_polo> And presumably that gets more complicated because there's no great way to express ABI compatibility with numpy versions.
[21:42:32] <ngoldbaum> yup, it would be a very bad day if numpy did a release that broke ABI back compat
[21:42:57] <njs> ngoldbaum: they need to build-require the oldest version of numpy they want to support
[21:43:35] <toad_polo> At ${DAYJOB} we have a lot of trouble with ABI compatibility because there are no backwards compat guarantees like that.
[21:44:26] <njs> it would be lovely if we could have some way to track numpy ABIs in wheel metadata, but until that happens, numpy just follows a policy that if you build against X, then your binaries will always work against X+N
[21:45:50] <toad_polo> I imagine that with what little taste of it I've gotten in this PEP 517 bootstrapping discussion, bikeshedding over the name of the pyproject.toml keys was probably not high on the list of anyone's priorities.
[21:46:07] <njs> toad_polo: that wouldn't have resolved the ambiguity between build-requires and install-requires :-)
[21:47:14] <njs> we did think about this some, but there is a limit to how accurately one can predict these kinds of details based on pure reasoning
[21:47:39] <dstufft> (or more likely I didn't care very much about the name because they were all reasoanble)
[21:47:54] <njs> dstufft: it's also possible we just thought of it at the authoring stage and didn't tell anyone, to reduce bikeshedding :-)
[21:48:25] <toad_polo> I need to add a PEP 517 frontend to setuptools, there's not really a good PEP 517 option for building sdists at the moment.
[21:48:59] <njs> speaking of minor naming things, I still think we should tweak PEP 517 to say that you look up the backend by doing 'getattr(given_backend, "__pep517_backend__", given_backend)', so people can just write 'backend = "flit"' or 'backend = "setuptools"'
[21:50:13] <njs> we already have a standard way to do it for setuptools-based projects (setup.py sdist), and if you need to build an sdist for a non-setuptools-based project then setuptools would be a weird place to look :-)
[21:50:17] <toad_polo> Because setuptools has a bunch of other commands that aren't going away any time soon, and I'm actually fine with "setup.py-as-makefile".
[21:51:57] <toad_polo> Probably it's a pipe dream anyway because it's not mission-critical but it's also complicated and I've got a huge backlog of stuff to do.
[21:51:59] <njs> it would be lovely to integrate setup_requires with the new pep 517 stuff so that it stops breaking everyone's environments. I'm not sure how that relates to giving setuptools a build frontend though?
[21:52:46] <toad_polo> A setuptools build frontend would be able to invoke `sdist` inside an isolated build environment.
[21:52:49] <dstufft> njs: I think the thing is that ``python setup.py <foo>`` doesn't automatically install dependencies, and there's nothing out there that does that
[21:53:29] <dstufft> and even if we add that to pip/twine/whatever, setuptools supports people writing custom commands with setup.py, so setuptools probably wants something that'll continue to support arbitrary setup.py commands being passed through that same mechanism
[21:53:49] <njs> dstufft: I would be OK with eventually seeing 'python setup.py sdist' -> "error: you're using setup_requires, use pip sdist . instead"
[21:54:08] <njs> dstufft: pep 517 doesn't have a way to pass through arbitrary commands though, it's only sdist and bdist_wheel
[21:54:26] <dstufft> njs: sure, but a setuptools specific front end can add on any extensions it wants
[21:54:43] <toad_polo> PEP 517 also doesn't specify that you have to be able to install things but pip installs stuff.
[21:56:47] <njs> oh, so 'setuptools X .' is just another way to write 'virtualenv .something; .something/bin/pip install $build-requires; .something/bin/python setup.py X', and it works for all X? sure ok, why not
[21:57:29] <njs> probably don't need to even use PEP 517, just parsing for build-system.requires would be good enough
[21:58:06] <toad_polo> It would be weird to invoke setuptools to build flit projects, but whatever floats your boat.
[22:00:51] <toad_polo> jaraco has a `pip-runner` tool. If we don't want it actually *built in to `setuptools` (which may not be necessary at all), that might suffice.
[22:02:31] <jaraco> The pep517 project has a builder; `python -m pep517.build .` The `pip-run` tool doesn’t honor pep518 dependencies (any more than pip does).
[22:03:34] <jaraco> But yeah, no support for arbitrary `setup.py` commands, which I think is a deprecated model in the pep517/518 world.
[22:05:19] <toad_polo> I'm not entirely convinced it needs to be if we could solve the problem of resolving target dependencies.
[22:05:45] <njs> my personal interest these days is more in new non-distutils-based systems, that don't have that crushing weight of backcompat that makes it impossible to fix the original architectural flaws (e.g. the command/plugin system). But if setuptools wants to add new setuptools-specific stuff, that seems... fine?
[22:06:20] <njs> what makes pep 517 hard is that everyone uploads sdists and bdists to pypi, and everyone uses pip to install them, so *everyone* has to agree on how to invoke those
[22:07:08] <njs> for stuff related to hacking on the project, like running tests or whatever, it's OK if different projects use different tools; you just have to read the README or whatever to find out
[22:09:36] <toad_polo> Yeah, it's not super important, but the `cmdclass` model is actually a fine one to replace make files.
[22:09:47] <toad_polo> I don't want to write make files, but I don't mind writing the equivalent in Python.
[22:11:21] <dstufft> I mostly use `invoke` for that but there's a big body of commands already written for setuptools so it makes sense to think about how to support it in the new world