Log file Viewer
#pypa-dev logs for Thursday the 18th of February, 2016
[00:00:17] <dstufft> I'm not sure the overhead here is going to be massive. the API as written in my PEP right now doesn't actually handle the zipping up/down of files, the build tool emits directories that a higher level tool can then take and zip up. The reason for that is so something like ``twine wheel .`` can first build a sdist (in a directory) without having to compress the sdist, then decompress it and then build a wheel and then compress it all again.
[00:00:18] <dstufft> The overhead will be copying files + whatever computations the sdist creation tool needs to do (invoke git to get a version or whatever)
[00:02:00] <dstufft> if you're going straight from a directory to a wheel (as pip install . does) yes the sdist dir will be throw away
[00:02:22] <lifeless> so, I expect that nontrivial projects will do nontrivial things in the source -> sdist step
[00:03:17] <dstufft> I mean granted, people do them in the sdist steps today, because our build system is such that build time dependencies punish you
[00:03:54] <njs> dstufft: there's a long precedent for doing things like this in source distribution build steps, e.g. automake also has special provisions for code that needs to run at this stage
[00:04:10] <lifeless> to me, a source distribution is much more than 'tar up'; its about making the thing ready for consumption
[00:04:23] <njs> dstufft: a common case there is running generating the ./configure script, which is the same principle
[00:05:00] <lifeless> your developer source tree has the bare minimum - you don't want redundant generated files in VCS
[00:05:12] <dstufft> would Debian accept the .c files that were generated from a .pyx inside of a sdist, or are they going to delete the .c files and regenerate them?
[00:06:02] <njs> (of course most people are not going to be running autoreconf as part of their python build process. I hope. But the point is that this general need is not something that's unique to python and its build systems' particular historical limitations)
[00:06:32] <lifeless> dstufft: .c files that were compiled are still free, its not an issue (as long as the .pyx is still present)
[00:06:57] <lifeless> dstufft: separately, Debian wants to know that things *can* be edited and rebuilt entirely within Debian
[00:07:19] <lifeless> dstufft: so from that angle the .c files provenance is unknown, and they'd be deleted during package build.
[00:07:54] <lifeless> dstufft: but its not a policy problem for them to be present in the upstream source
[00:08:25] <lifeless> dstufft: anymore than having compiled docs is a problem; Debian chooses to spend CPU rebuilding for a bunch of reasons, but Debian != universe :)
[00:08:40] <njs> :-) oh hah, I'm wrong, they actually did catch this at some point and they do forcibly regenerate that .c file during package build (https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=710177)
[00:09:00] <dstufft> Right, I'm not saying we need to do X because Debian does, but rather that how Debian handles it I think matches what I think it should be
[00:09:12] <njs> but whatever, numpy is going to keep doing its thing and debian is going to keep doing its thing and everyone is happy with the current situation
[00:09:58] <lifeless> dstufft: people don't *want* installing numpy to be slow via pip, for instance, but they don't see the build time in the Debian build farm ever
[00:10:07] <njs> (in fact the person who filed and then fixed that bug report with debian is also one of numpy's core contributors)
[00:10:19] <dstufft> tunring a .pyx file to a .c file is a build time step is my assertion, and the only reason it makes sense to do that at the sdist stage is because we live in a world where distutils/setuptools made doing it at build time actively painful
[00:11:04] <lifeless> dstufft: in a big picture sense, VCS, tarballs, binary tarballs, distro packages are all shades of the same thing
[00:11:19] <njs> dstufft: freezing a single cython version for the whole QA->release was also a substantial contributing factor to that decision
[00:11:52] <dstufft> and in any case, if the .pyx has a .c file living alongside it, there's no reason the sdist step can't skip regenerating the .c to speed things up if it doesn't want to do that each time
[00:13:06] <lifeless> dstufft: as we do any computation on the source - both deriving things (like DISTINFO) or removing things (not copying .git)
[00:13:33] <lifeless> dstufft: we specialise the output, what can be done with it, and what use cases it can be used for
[00:13:58] <lifeless> dstufft: there's nothing fundamentally different about an sdist and a wheel for instance, except that we don't produce multiple sdists for a single version
[00:16:33] <njs> lifeless: I think in source dirs usually, which drives me crazy, but, y'know. setuptools.
[00:16:41] <lifeless> just getting a feel for 'folk could do X' vs 'what folk do today is going to be friendly to this proposal'
[00:21:56] <lifeless> dstufft: another stab at it - the audiences for sdists are - folk with platforms without wheels, folk that don't trust wheels, and distro packagers.
[00:22:18] <lifeless> dstufft: the audience for wheels are folk that can use them and trust them - doing installs via pip
[00:23:08] <lifeless> njs: the first category? Somewhat. github etc take most of the load there-- except for semi-proprietary shops of course
[00:23:27] <lifeless> but yes, folk do want to be able to pull down the source and browse around in it usefully
[00:23:44] <njs> yeah, this is less of a big deal now that public VCS is more stable/uniform/reliable as a thing
[00:25:08] <njs> but I do still use 'apt-get source' not too uncommonly, because it's a lot quicker than googling for some projects website, click around to find clone URL, wait for git clone, try to figure out how to check out the commit that corresponds to the version I'm using, etc.
[00:28:47] <dstufft> I think that the mandatory sdist -> wheel thing ends up with a much cleaner set of shared responsibilities
[00:30:03] <lifeless> I have in my head that you're really wanting build hygiene so folk don't upload broken wheels.
[00:36:17] <dstufft> it's not so much broken wheels (typically it's broken sdists actually), but surprising behavior. There are reguarly people who get confused because pip install . produces a different result than creating a sdist from that same directory and installing that. So this would represent trading some speed for reducing some complexity that end users generally aren't aware of or don't think about.
[00:42:11] <dstufft> lifeless: the other edge case is harder to pin down, and it normally breaks down into "the code in setup.py is expected to run inside of both a vCS repo, and inside of a sdist, and assumptions get made that don't hold up in both situations". The best example I can think of offhand is getting a VCS version from git instead of from the metadata, in a repo you want to pull from git always, in a sdist you want to pull from the sdist metadata
[00:42:57] <dstufft> setuptools-scm has this problem (or did, not sure if it was fixed) where it was giving the wrong information in Debian's workflow because the setup.py wasn'te xpecting to be ran when inside of someone _elses_ git repo (the packaging .git repo for Debian) when it was an unpacked sdist
[00:43:52] <dstufft> it's obviously not impossible to write a single chunk of code that can operate correctly both when running inside of a repo, and when running inside of a sdist, but a clear pipeline I think makes it _easier_
[00:44:36] <lifeless> the tension I feel is that traditionally there are other uses to sdists than just a vehicle to get to the next bit of compilation
[00:46:01] <lifeless> I'll catch you more tomorrow I imagine. Moving this forward is critical for me
[00:47:47] <dstufft> lifeless: cool, and I'm going to poke at what it'd look like if we don't mandate VCS -> SDIST -> Wheel but instead allow VCS -> Wheel as well as VCS -> SDIST -> Wheel
[00:49:39] <njs> a meta concern I have is that if we pack another pile of controversial bits into a single proposal right now, then we'll blow our collective stack and crash the discussion.
[00:50:20] <njs> dstufft: also I'd be interested at some point to get your thoughts on the worked-out rationale for python hooks > command line interfaces in my PR
[00:50:49] <dstufft> njs: So I think that you can't replace setup.py unless you have a replacement for all the things we use setup.py for
[00:50:54] <njs> (esp. since lifeless is happy to argue about it, but when pressed always says that his actual preference is to go with whatever makes you happy :-))
[00:52:37] <njs> I mean in some sense this is true, but it seems to gloss over a lot of details? Not all projects care about all features of setup.py, and not all features of setup.py should necessarily get replaced by a single PEP (or even any PEP -- parts could be replaced by build-system-specific stuff that distutils-sig doesn't even try to standardize)
[00:52:47] <dstufft> I think what would happen if we don't have a replacement for ``setup.py sdist``, is flit would implement enough to make VCS -> wheel work, and then nobody would bother to create sdists for it because they'd see sdists as extra effort and then distros would start yelling at me for incentiving people to not upload sdists
[00:53:12] <dstufft> njs: sorry, specifically I mean, I don't think you can't replace setup.py without having a replacement for all of the things, or a reason why we don't actually need a replacement for that thing
[00:54:13] <njs> yeah, but they don't really care :-). what started this whole thing was me whinging at Thomas that flit was useless if I couldn't do 'flit sdist'
[00:54:55] <njs> I mean, it would already be an improvement over the status quo if they supported 'pip install .' too
[00:55:12] <dstufft> I mean flit might not care,, but flit is just a stand in I'm using for a theortical consumer here :]
[00:57:06] <njs> I understand your worry about people yelling at you, because that's always unpleasant and doesn't always make sense. But it would be pretty unfair in this case I think: the distros should be yelling at the projects for not uploading sdists. There is literally no extra incentive to not uploading sdists in any of these proposals.
[00:57:50] <njs> there's zero demand for users to have 'pip install .' work -- the reason people don't switch to flit-like build tools right now is because users demand 'pip install pypiname' keep working.
[00:59:33] <njs> and in this case some projects are using it and giving up on sdists, which indeed sucks :-)
[01:02:22] <njs> can you say more specifically what exactly you see about the build system interface proposals as "incentivizing people not to upload sdists"?
[01:02:51] <njs> like what is the new incentive to not upload an sdists, versus uploading an sdist, if one of Robert or my's proposals are implemented?
[01:05:19] <njs> as far as I can think, the only way this would be true is if it (a) leads to the creation of new build systems that support 'pip install .', but (b) those build system authors are lazy, and don't bother to implement sdist creation code, and yet (c) they are sufficiently industrious that their build systems have compelling new features that convince package authors to adopt them even at the cost of giving up sdist support?
[01:05:30] <dstufft> because it pushes the onus on implementing a sdist command onto the build tool (and possibly the upload command too, depending on the specifics) without actually telling them that they need to do that... it's just kind of assumed they are going to.
[01:06:21] <dstufft> in particular, in a world where a "sdist" is not really defined anywhere as anything but "a thing that's like what setuptools/distutils outputs"
[01:06:37] <njs> so I think this requires that we assume there are lots of people who care a lot about 'pip wheel .' working (i.e. being able to do 'flit wheel' is not good enough, it must be spelling 'pip wheel .', but who don't care about distributing sdists?
[01:07:07] <njs> I agree that we should definitely tell the build tools what they need to do to implement an sdist command :-)
[01:07:46] <njs> but (while more specific documentation is always nice) is that not just a link to PEP 345?
[01:08:55] <dstufft> njs: PEP 345 doesn't mention sdists at all, PEP 314 sort of mentions them (in that it says a sdist is a thing, and you add a PEP 314 metadata file to a specific location inside of one)
[01:09:25] <dstufft> there is no formal defintiion of what a sdist is, except "a thing that looks like what distutils/setuptools outputs"
[01:10:04] <njs> so I definitely agree that together with Robert/my PEP we would definitely want to document what the actual status quo for sdists is
[01:11:39] <njs> I think it's basically {NAME}-{VERSION}.{zip/tar.gz}, unpacks into {NAME}-{VERSION}/, with a PKG-INFO file that contains the fields that twine cares about?
[01:11:51] <dstufft> so I think that people are incentivized by the fact that actually producing a "sdist" is the same sort of terribleness that implementing setup.py is, where you have to sit there and play whack a mole with a bunch of implementation defined behavior (because in practice, the defintiion of a sdist beyond what distutils/setuptools produces, is a set of adhoc assumptions that a variety of different tools makes about it)
[01:13:47] <njs> to make sure I understand your last line there -- the incentive is that right now trying to implement a sdist command would be painful and unpleasant, so build system authors won't want to do it unless we make it less painful and more pleasant?
[01:14:20] <dstufft> (which is part of my thinking I guess. Not to toot my own horn or anything, but I'm heavily steeped in all of this and the best answer I can give you for what a sdist is "go look at the main tools that consume them and see what assumptions they make")
[01:14:45] <dstufft> which is basically the same thing as why we're making this thing instead of just telling people to write a setup.py :D
[01:15:33] <dstufft> I mean, I think some build system authors will do it, but I'm not sure they'll do it correctly (because what even is correct?)
[01:16:03] <dstufft> in a world of a thousand build systems blooming, we don't want a thousand different variations for what a sdist is :]
[01:16:41] <dstufft> (sorry, I'm probably a bit all over the place because my migraine is still there, it's just subsidded enough I can manage to computer)
[01:16:59] <njs> well, the upside is that sdists already provide so few services that you can't miss much :-)
[01:18:34] <dstufft> part of why my pair of PEPs provide a .src.whl instead of trying to add this to the current setup is becasue it means that we don't have to really care about what exact assumptions every single tool makes, we can just declare what assumptions they can make
[01:18:49] <dstufft> and any tools that aren't updated to support a .src.whl will just ignore .src.whl's
[01:20:26] <dstufft> (we'll still probably want to implement some sort of shim, ideally one that can accept a .src.whl and turn it into a legacy sdist, but that won't need to be build system specific, and individual projects can choose if they want to support older pip's and at what point they don't care anymore)
[01:20:57] <dstufft> e.g. Wheel benefited from the fact it took the ideas of .egg, but gave itself a new name so it didn't have to try to maintain some sort of compatability with .egg files
[01:22:06] <njs> I'm not (necessarily) against the idea of defining a new sdist format, and I agree that if we want to do more than tighten up and formalize the current definition then we should probably should give it a new name etc.
[01:22:16] <dstufft> njs: aslso re: CLI vs Python API. I'm torn but I think I lean towards a CLI (specifically, one that has to be invoked as python -m <wahtever the user enters>)
[01:23:25] <dstufft> the process boundary is super easy to define (because it already exists, sys.stdin, sys.stdout, sys.stderr, sys.args, and return codes) and it comes with built in support for incremental output and seperated error/output
[01:24:01] <dstufft> to replicate that in a Python API requires a more complex specification as to what types you're going to pass into what, what exceptions are valid, how you handle incremental output, etc
[01:25:31] <dstufft> njs: I mean, as of right now, how would we implement the spinner you added to pip with ``def build_wheel(wheel_directory, config_settings, metadata_directory=None):``
[01:26:30] <dstufft> because aiui, that function is still going to take ~30+ minutes for numpy, and now we don't know if the function is still progressing or if it stalled
[01:26:41] <dstufft> (also speaking of the spinner, I've had a number of people mention they really like it ;))
[01:33:55] <njs> dstufft: the assumption is that pip will spawn a child and the child will run the hook, and the hook will blather whatever random status updates it wants to stdout/stderr, just like it does now
[01:34:01] <dstufft> njs: how does that work? or are you assuming that calling the python function is going to use sys.stdout anyways?
[01:34:45] <dstufft> yea I'm not the biggest fan of that, becasue it seems to be mixing the abstraction layers
[01:35:25] <njs> how so? it's part of the contract of that hook (in fact, all of the hooks) that they're allowed to write whatever random information they want to stdout/stderr
[01:36:06] <njs> but anyway, I assume yo uhaven't had time to read the PEP draft properly what with the migraine and all, so I won't grill you on all the detaisl :-) there's a chunk of text below that tries to lay out in detail why I think the hook approach is better that you might want to look at when you have the chance (if you haven't already)
[01:37:52] <dstufft> I don't like functions that write to sys.stdout :] (logging always bothers me in general :/) I like functions that take some arguments, operate on them, and return a value.
[01:56:47] <dstufft> StevenK: looks like xafer reviewed it and offered a suggestion. I went back to sleep this morning cuz of my migraine hanging around and then promptly forgot when I woke up :(
[02:04:41] <dstufft> StevenK: quick glance, other than the cannonicalize function, it seems ok to me
[02:05:59] <StevenK> The cannonicalize function wasn't a thing when I wrote it, but I shall rebase and fix it up
[02:12:32] <njs> dstufft: but, like I said, would appreciate thoughts based on the PEP draft, b/c there are some technical advantages to the hook approach :-)
[02:31:57] <lifeless> dstufft: I don't understand the -m preference: I don't think it helps anything
[03:01:48] <njs> lifeless: "I prefer a python API, because of the import etc etc stuff" <-- I don't understand this sentence :-)
[03:06:48] <njs> dstufft: also it looks like as far as distutils is concerned, the current sdist format really isn't so bad: it's just the tar/zipball with a PKG-INFO file, where the PKG-INFO format is as defined in whichever-PEP-is-current and is needed for wheels anyway, plus the rule that the contents of PKG-INFO are allowed to be lies. This doesn't seem super onerous to ask people to generate :-)
[03:07:09] <njs> also in your comments on my draft you wrote "Python interfaces that call out to external commands (as I expect every thunk to do) are going to have to deal with all the complexities of incremental output, dealing with stdout and stderr from the things they called, concurrency etc. We're not avoiding that, we're just failing to deal with it. Python code can just write to pip's output for instance, which is even more intrusive - as pip has no control over th
[03:07:09] <njs> at!" <-- which I'm baffled by and makes me wonder if you didn't understand the actual python hook proposal at all?
[03:11:30] <lifeless> njs: if pip doesn't (because the hook is python), then the hooks in each system do
[03:11:52] <lifeless> njs: if pip deals with it, then its not a benfit to the proposal's design, because pip is still having to deal with it
[03:12:03] <njs> I'm envisioning that it's like how make or whatever works -- the hooks print whatever they want to stdout/stderr, and they let their children do the same (unless they have some specific reason to try and suppress it)
[03:12:29] <njs> so yeah, pip deals with it, using exactly the same code paths it already uses, not a big deal
[03:12:50] <lifeless> njs: so if it deals with it, why is it a benefit to the proposal :> - you claim the UI stuff is better there
[03:15:02] <njs> there's a claim that trying to use stdout for pip<->build-backend communication (as in your build_requires and metadata commands) is a pain, because it puts the onus on build backends to make sure that nothing else scribbles on stdout
[03:16:14] <lifeless> there was a discussion on the list the last time we went around CLI vs Python contracts
[03:17:34] <njs> there was a fair amount of discussion on the list, and I think there were several claimed advantages of hooks in that discussion was not really advantages
[03:18:34] <njs> (e.g. the idea that we could do better error reporting using python exceptions... I mean, I guess, maybe somehow? having exceptions available isn't *bad*? but in practice I think it's pretty irrelevant)
[03:19:24] <njs> there may be real advantages that I did not think of to include in my draft PEP text, but all the ones that I could think of and feel comfortable defending are in the text :-)
[03:20:45] <njs> also regarding the bit about "Python code can just write to pip's output [...] pip has no control over that!": this would only be true if we were running the hooks in-process, right, which I definitely do not advocate?
[03:21:11] <pmxbot> jaraco pushed 4 commits to setuptools (https://bitbucket.org/pypa/setuptools/) :
[03:21:11] <pmxbot> Switch back to SyntaxError for invalid markers, stops consumers having to import packaging themselves
[03:22:46] <StevenK> lifeless: debconf's glue is easy, it's the gymnastics everyone else has to do because fds 0, 1, 2 aren't yours anymore
[03:23:13] <njs> lifeless: presumably under pip's regular subprocess handling -- so output attached to a spinner, unless -v is specified, etc.
[03:23:56] <lifeless> njs: so pip has to have a utility package it installs into the PYTHONPATH as the first thing it does in any operation
[03:24:43] <lifeless> njs: ? I concede it doesn't allow direct access to pip's address space, unless there are bugs in the thunk
[03:25:02] <lifeless> njs: well, it can't be in pip's source tree, because it has to work when pip's been installed
[03:26:05] <njs> I guess if you want to support running pip directly from a zipfile then you might have to extract the file somewhere
[03:27:04] <njs> I guess importing pip would be an error, unless you have listed pip as one of your bootstrap requirements
[03:27:11] <lifeless> njs: the pip that started the task, or the pip that has just been installed during an upgrade
[03:28:00] <lifeless> njs: it has to work when the pip in the target environment has already been upgraded and made arbitrary internal changes
[03:29:35] <lifeless> njs: sure; the attractive nuisance is that folk have to know they /cannot/ within that file
[03:29:38] <njs> but to the extent I understand the issue you're worrying about, I don't think it's a worry :-)
[03:30:15] <njs> okay? so you put a comment at the top, and also if you try the test suite would catch it instantly
[04:23:17] <njs> lifeless: if we support any kind of isolated build mode (i.e., you only have access to the packages you explicitly requested), then this would be blindingly obvious because all builds would fail instantly ('import pip' -> not found)
[06:01:14] <njs> well, it's trivial if we're allowed to depend on virtualenv. dunno if dstufft will let us though >.<
[06:57:25] <njs> actually on second thought, I guess it's not that hard to fake... just install into --prefix tmpdir/build-env, and then PYTHONPATH=tmpdir/build-env python -S
[07:09:25] <Ivo> you can also execute scripts with `<mypython> script` since they are just python code with a shebang
[09:13:32] <njs> lifeless: virtualenv creation only takes ~250 ms on this computer, and venv only takes ~50 ms
[10:45:40] <lifeless> njs: then when you have its wheel, you put together a new venv to build C, install D.
[10:47:20] <lifeless> njs: and now you can finally read the metadata for A, only to find that with that you can't use it, so you need to build A'. Rinse and repeat.
[11:59:17] <Ivo> dstufft, any reason https://www.python.org/dev/peps/pep-0517/ doesn't render but https://www.python.org/dev/peps/pep-0516/ does
[12:02:14] <Ivo> http://rst.ninjs.org/?n=5ac8e15da4391d6083e100e6d64cd3e8&theme=basic has some red boxes for its' rendering
[13:42:12] <xafer> We might want to wait for a fix of https://github.com/pypa/pip/issues/3486 but we already have 5 bugfixes available
[22:45:07] <njs> lifeless: but now you have wheels for B / C / D, so when you build A' you skip all those steps (basically the wheel cache acts a memoizer to convert the naive quadratic algorithm into the dynamic programming solution)
[22:45:54] <njs> lifeless: anyway, there are lots of ways to go about optimizing isolated builds (caching and reusing venvs, etc.) -- first make it right and then make it fast ;-)