Log file Viewer
#pypa-dev logs for Monday the 12th of October, 2015
[02:25:13] <Nakato> RIght now I'm looking at how those old pip's installs function to get a better understanding of egg_info in the sdist.
[03:39:06] <dstufft> lifeless: fwiw a ton of this sdist thread is a lot of people talking past each other I think. I hope that call tomorrow will enable a much higher bandwidth of communication so we can figure out what the actual things everyone is trying to solve is
[03:58:53] <dstufft> I'm all over the place right now, house hunting and probably going to be putting in an offer
[04:00:34] <dstufft> it's still pretty hard to use a build system other than setuptools unless you can rely mostly on your users getting wheels from PyPI
[04:13:12] <dstufft> lifeless: one thing, I'm not sure the interface _is_ actually documented, or at least I can't think of where it would be documented at and I don't think it's currently considered part of the stable API of pip
[04:14:47] <dstufft> Unrelated: I hate setup.cfg and wish it to die with the burning intensity of a thousand suns
[04:20:32] <lifeless> dstufft: https://pip.pypa.io/en/latest/reference/pip_install/#build-system-interface
[04:20:37] <dstufft> lifeless: I hate the ini "format". If something is designed for human consumption it should use something better, like YAML or even god forbid, TOML (though I don't like TOML much), if something is designed for machines, it should use a format that is actually a format and has sensible parsing and data structures, like JSON
[04:21:33] <lifeless> dstufft: but for arguments sake, having a single key we read from setup.cfg seems like a low-impact-on-pip and you can then write your own fav system that uses setup.yaml
[04:22:06] <lifeless> holy crapsticks ubuntu! python-pip | 1.5.6-7ubuntu1 | wily/universe | source, all
[04:22:31] <dstufft> lifeless: this relates also to why I think VCSs and sdists need different formats, because a VCS format should be written for human consumption (and here, it may make sense to keep setup.cfg with just enough information to bootstrap querying the real build tool for the rest of the metadata and what not) and in a sdist it should be entirely within something like JSON
[04:22:59] <dstufft> lifeless: setup.cfg has some baggage with me, becuase distutils2 wanted setup.cfg to be both the VCS file format _and_ the sdist file format
[04:23:24] <lifeless> dstufft: well, if you're happy with the file content changing as it goes into the sdist, thats fine
[04:25:00] <lifeless> I'm still thinking about the implications of all-static or permitting intermediary code to generate deps
[04:28:56] <dstufft> I think there are two real avenues to attack this, long term I think setup.py needs to die and we need a sdist 2.0 which has static metadata wherever we can, though it doesn't need to be _all_ static, but we should have really good reasons why anything that isn't static isn't static. In this avenue I see it working like VCS: has a setup.cfg with a key to indicate your build system (possibly build requirements? idk) and a key to indicate how
[04:28:57] <dstufft> to invoke your build system (import name idk?), from a VCS checkout we use that to invoke the build system to build a sdist (or to do an inplace development install). That generates a "sdist 2.0" with static metadata, which also includes build dependency information and the way to invoke the build system, and we can uise that to turn the sdist into a wheel.
[04:29:49] <dstufft> The other avenue is layering just the setup.cfg build system (build requirements?) and rely on the current sdist behavior of invoking setup.py to make a hook.
[04:30:23] <dstufft> The second avenue is a lot less work, and I think it can be done before the first one jsut fine (as long as the setup.cfg stuff is done in a way that will mesh well with the first avenue, because I plan on making that happen at some point)
[04:33:50] <dstufft> the benefits of the first is that static metadata makes a lot of things way easier, not just pip but PyPI. It's a real problem that a lot of downstreams have been asking for as well. Also setup.py is a pretty terrible interface to implement in general, and finally the first one is obviously a packaging format and we remove the footgun of people unpackging the sdist and running ``setup.py install`` (which is almost never what they actually
[04:34:27] <dstufft> those are all really great benefits, but I don't think they need to block smaller, incremental changes like the second which can have a ton of great impact today unless someone feels like putting in that effort today
[04:35:06] <dstufft> what I don't want to see, is implementing something that will require the work and ecosystem impact of #1, but without getting all those benefits of #1
[04:35:57] <dstufft> I mean, they do sort of, but only if you're able to upload wheels and not all projects can do that
[04:36:22] <lifeless> right, but if they can't they probably can't use the static metadata we have defined so far either
[04:36:57] <dstufft> Why not? Surely name, version, project summary, project description, etc are not going to change.
[04:37:13] <dstufft> maybe dependencies change, so maybe dependencies need to be something that isn't static
[04:38:21] <dstufft> it only has it because when you do an upload, you send it alongside the actual file
[04:41:42] <dstufft> lifeless: it is sometimes in egg-info, it is sometimes not, it sometimes matches reality, it sometimes doesn't. We can't trust it on average because it's not the "source of truth" for that data, a script is and that data can be wildly inaccurate. (twine has to trust it, and when I sent out the emails about removing off site hosting on PyPI, I got a slew of people complaining my instructions to use twine didn't work, because their sdists
[04:43:29] <dstufft> VCS can be nothing but build dependencies and a description of what to import to call (or whatever interface we end up with)
[04:43:33] <lifeless> dstufft: ok; one thing I don't like in the discussion so far is the idea of a separate sdist artifact for VCS installs
[04:43:44] <lifeless> dstufft: I'm entirely happy with a logical step that has all the right metadata etc
[04:44:08] <lifeless> dstufft: just I don't want 'compress and decompress this big tree' to be in my local dev iterations inner loop
[04:45:49] <dstufft> lifeless: we already copy the entire tree to a temporary directory, compress and decompress can actually be a huge improvement in some situations, like when you have big files in your VCS tree that don't get included in the sdist
[04:47:32] <dstufft> lifeless: no, we only backed out the "when we do the copy, try to omit certain directories like .tox and .git from the copy because they are often big and not often needed"
[04:49:10] <dstufft> because if we don't, things like ``venv1/bin/pip install -e /path/to/my/repo && venv2/bin/pip install /path/to/my/repo/`` can cause the install in venv1 to break because we've potentionally changed the metadata in the egg-info
[04:49:34] <dstufft> I'm also really set on reducing the number of possible combinations for installations to actually happen in
[04:50:40] <lifeless> the install -e thing for instance, if pip did it rather than the build tool (a-la wheels), we could use a unique egg-info and prevent it breaking on subsequent ops
[04:50:53] <dstufft> like right now we have: you can do a development install, or you can install directly from a VCS, or you can install directly from a sdist, or you can build a wheel from a sdist and then install from the wheel, or you can build a wheel from the vcs and then install the wheel
[04:51:35] <lifeless> ok, so - I'd like to be able to get to a world where we don't call setup.py develop ever
[04:55:02] <dstufft> My dream goal is: For non -e installs, we have one path for install: VCS -> sdist -> Wheel -> installed, and the only thing that varies is where you enter this path (e.g. if you pip install foo, and foo has a wheel available, then you jump right to Wheel -> Install, if you have an sdist available, then you jump to Sdist -> Wheel -> Install, if you have a VCS available, then you start at begining and you get VCS -> sdist -> Wheel ->
[04:55:02] <dstufft> Install). Then, we ideally remove most of the "subtly different" from non -e installs
[04:58:16] <dstufft> My dream goal is: for -e installs, we have a situation where pip can install the build dependencies, then poke the build tool and be like "please build in place", then someone (pip? the build tool?) drops the needed metadata in place, and then pip handles making sure this directory is added to sys.path
[05:08:22] <lifeless> so the first one I can see why its attractive, but I'd like to be able to avoid what I see as needless byte-shuffling
[05:09:26] <lifeless> for the second one, sure, though I think we need to figure out some good semantics to avoid venv1 and venv2 stepping on each other - assuming thats really going to be a use case
[05:10:06] <lifeless> dstufft: I'll need to dig deep in to understand why you consider it essential; I'm sure I'm missing something ;)
[05:10:10] <dstufft> I think it's the only way to avoid subtly different outcomes (I see it happen that people who get confused why pip installing from their VCS works, but pip installing from an sdist doesn't because their MANIFEST.in wasn't quite right or so)
[05:11:01] <dstufft> a lot of the "weird" errors that people get in their Python installs are often attributed to mixing a bunch of packages installed in subtly different ways
[05:11:22] <dstufft> and alot of the messy spaghetti code in pip is there to deal with the subtly different paths the installations can take
[05:13:53] <dstufft> I'm not really concerned with the performance of a non -e VCS install compared to reducing that complexity, because I think it's an outlier case anyways and it's trivial to come up with scenarios where both generating an sdist makes it faster and makes it slower and I think supporting one "path" is the sanest way of doing it (how many packaging tools support more than one path? Almost none of them, and I don't think any that do support it
[05:14:51] <dstufft> but, the interface could be done in a way the avoids the compressing/decompress step at least, since you could make it possible to ask the build tool to create an "unpacked sdist" in some location
[05:15:31] <dstufft> (possibly you'd have ``pip sdist`` which just asked the build tool to create that unpacked sdist, and then pip would do the compressing and tarballing
[05:18:06] <dstufft> I went through a ton of different scenario's for pyca/cryptography and the only way I could force it so that when running under tox, we were _actually_ testing against the installed copy, was to add a src/ dir and move our importable package inside that, so that the thing we're importing wasn't directly in .
[05:19:51] <dstufft> we only noticed it because pyca/cryptography made a change that required an explicit build step to be runnable (compiling a .so) and so it would blow up when running because it'd try to import from . and there wasn't a .so there (unless the person previouslly had run ``python setup.py build --inplace`` in which case you might jsut get weird errors because who knows when you last build that in place .so)
[05:53:30] <dstufft> lifeless: so pip can't check that it does that that's correct, but people "test" things by doing ``pip install .`` and then get confused when that gives different results then installing the final sdist
[18:57:18] <ErikRose> The hashing PR is (hopefully) finished. :-) https://github.com/pypa/pip/pull/3137
[20:23:23] <ronny> lifeless, dstufft: did i just miss the sdist stuff hanngout/ is there a recording?
[23:36:10] <qwcode> dstufft, https://packaging.python.org/en/latest/ still broke : ) you control that, or should I email some other PSF team or something...
[23:42:15] <qwcode> still getting the packaging project... maybe need to wait for the CDN a few mins?