Log file Viewer
#pypa-dev logs for Wednesday the 17th of February, 2016
[01:07:33] <StevenK> lifeless: I think the markers -> packaging card is done, the code has been reviewed, landed and and is in 16.1
[01:11:04] <StevenK> lifeless: Yes, I was starting to poke at marker support in install_requires yesterday
[01:24:42] <StevenK> lifeless: So I think the first step is to make sure .marker or something like it is available in pkg_resources.Requirement, and then search for all these hacks that look for ';' in extras and parse it off
[12:57:56] <StevenK> dstufft: Although, you're probably feeling better if you can bear to look at a computer screen without scratching your own eyes out.
[13:00:12] <StevenK> dstufft: You might have seen my previous message, or missed it, but https://github.com/pypa/pip/pull/3307 is ready for you whenever. If it helps, you probably wrote 70% of that, I just cleaned it up and made the tests pass.
[22:45:52] <dstufft> lifeless: I'm working on it right now (I was MIA most of yesterday because of a migraine :( ). I'm going to eat dinner (wife jsut called me) and after that Ithink I have enough here that I can push it and just fill in the rest over tonight
[23:16:57] <dstufft> Ok, I'm still working on this, but I think this gives the general idea -> https://github.com/pypa/interoperability-peps/compare/master...dstufft:build-pipeline
[23:20:41] <dstufft> if y'all want I can add you both to the PEP repo and assign numbers to your PEPs
[23:21:26] <njs> if we're not happy with sticking up out-of-date snapshots of the drafts in the PEP repo it might even make sense to just commit a placeholder document like "This is under discussion -- see <link>"
[23:22:34] <dstufft> (We actyually have two PEPs repos lol, there's the official one located at hg.python.org and the one that we have on github which we use because of PRs and such
[23:23:02] <dstufft> we generally treat the github one as the one to edit and then we sync that up to the hg.python.org one periodically (often whenever we post a new copy to distutils-sig)
[23:24:17] <njs> yeah, I don't really want to be hassling pep-editors by email every time I push a commit to the interoperability-peps PR/repo, but it also seems bad to have a stale snapshot up on python.org
[23:25:17] <dstufft> njs: I have commit bit and I'm happy to sync yours if you want. I think lifeless has commit already so he can sync his own :]
[23:25:45] <lifeless> dstufft: it is; I don't recall if I have interops commit bit or not; it will need synced to hg too - and my dev environment is messed up right now
[23:26:38] <dstufft> lifeless: do you feel comfortable (once your machine is setup) updating your build abstraction PEP to hg.python.org if I copy it over now and assign it a number?
[23:28:57] <dstufft> https://bpaste.net/show/5cef287b69fd this is now the diff between PEP 508 in the hg.python.org repo and the interop-peps repo
[23:31:53] <lifeless> dstufft: I will be happy to sync stuff once I'm all sorted - but its 200GB to copy on shitty wifi
[23:32:33] <lifeless> dstufft: its been 4 days so far - I've got a lot of code review, email, and the occasional ctrl-Z + quick-git-ops done in the meantime
[23:32:48] <dstufft> I'll go ahead and make numbers for all of these, and give you and njs commit rights on the interopt repo
[23:33:55] <dstufft> lifeless: it's easier to reference them, and numbers are free. We can always reject a PEP if we don't end up using one of them
[23:34:38] <njs> lifeless: my reason for bringing up numbers is that it gives us short handles to use during this discussion (and in the PEP texts when comparing different options). "Robert's pull request <link>" is kinda awkward :-)
[23:35:30] <dstufft> lifeless: so I see two things, either we want the sdist tool to be decoupled from the build tool, and then we must go through sdist or if we want to allow building a wheel straight from a VCS checkout, then the build tool has to handle building sdists
[23:35:54] <njs> dstufft: that is similar to my concern about the pipeline thing. Maybe this is a useful concrete aspect to push on. IMO, ability to support incremental rebuild of source directory -> [whatever] -> install is a mandatory feature for the numpy's and scipy's of the world. I believe that you want this to be used by the numpy's and scipy's of the world, so your PEP should have some documentation of how that feature is going to work (or not) :-)
[23:36:31] <lifeless> njs: I'm not sure thats a good enough answer - (in fact I'm quite sure it isn't)
[23:37:36] <njs> ...so, uh, personally I refuse to use editable installs in any circumstances whatsoever, because they are so flakey and error-prone :-) I am not sure what weight we should place on such personal preferences, but :-)
[23:37:36] <dstufft> both a sdist and a wheel needs to know metadata about the project (like version number) and need to know how to get that information. If we need to support making both a wheel and a sdist from a VCS checkout, then both a wheel and a sdist need to support the exact same mechanisms for getting a version number (or whatever other info) out of the VCS repo (for example, using flit - flit gets version number from __version__ in the pacakge, so
[23:37:37] <dstufft> any tool to build a wheel would need to implement that logic, and any tool to build a sdist would implement that logic)
[23:38:29] <dstufft> if you have a build pipeline like in my PEP, then only sdist needs ot know how to pull that information out of the VCS, and the wheel building just reuses that version number that the sdist already generated
[23:40:44] <dstufft> njs: that's why you can specify a build directory, keep the build directory around, make your build tool reuse already build objects from within that build directory.
[23:42:35] <njs> hmm. my assumption would be that in practice *of course* everyone will use a single tool inside their repo for preparing both binary and source dists. they want to share lots of information (which files are source files?), and have to solve tightly related problems (e.g., at sdist build time, run cython to convert .pyx files to .c files; at wheel build time, run compiler on those .c files)
[23:43:37] <lifeless> dstufft: In your proposal I'm worried about: skew in .dist-info like we've had in .egg-info in the past; performance for incremental builds;
[23:43:57] <lifeless> dstufft: I have some issues with your concerns about 'making an sdist' - I really don't understand the redistributor argument there
[23:43:59] <njs> in order to avoid requiring cython on user machines, and avoid problems due to cython version skew (they're a little flakey about such things historically)
[23:45:13] <dstufft> njs: why is avoiding cython on user machines a good thing? Isn't the whole point here that we can depend on stuff at build time reasonably now? :]
[23:45:17] <njs> this is the current consensus best practice; I know at least numpy and scipy do things this way, and I think that's true further downstream to (pandas, sklearn, astropy, etc.)
[23:45:35] <lifeless> dstufft: there's a tonne of bikesheddy stuff that I don't care about, as well as polish (like referenceing pep 508 for the dependency lists)
[23:46:21] <njs> anyway, the specific example doesn't matter -- the point is that in practice no-one uses two unrelated make-ish systems in the same project for generating source distributions and binary distributions, because they're tightly related things
[23:46:24] <lifeless> one of the issues we've had in the past is that someone runs setup.py egg_info
[23:47:54] <lifeless> dstufft: will it be possible to thunk through to setuptools as-is with your pipeline design?
[23:49:28] <dstufft> lifeless: so that can happen here yes, but the reason for splitting the metadata and the creation steps up, are so automated tools can inspect the metadata without having to build the entire artifact. ideally end users won't be invoking these commands directly, and something like ``twine wheel`` will ensure the commands get empty directories
[23:49:34] <lifeless> dstufft: so, pragmatic issues. editables are not addressed, so anyone that migrates to this breaks pip install -e .
[23:50:43] <lifeless> dstufft: so you see the sdist step as equivalent to 'metadata' in my proposal?
[23:52:29] <dstufft> lifeless: I'm not sure I'm following. you don't create a sdist in place, you give it a directory to tell it to create a sdist in that directory. When you build a wheel from that, you don't give it the original VCS directory anymorem you give it the sdist directory
[23:52:49] <dstufft> if you create a sdist directory in /tmp/sdist/ then it doesn't matter if you edit files in your VCS checkout
[23:53:13] <dstufft> I thought you meant because the create sdist and create wheel have seperate metadata / create/build commands sorry
[23:56:25] <dstufft> lifeless: editables I'm not sure on yet, that's why I have a ..note:: there asking myself if it's resaonable to just punt them for now or not. I haven't yet given it a lot of thought about how exactly this affects that and how to handle it (other than my current, "figure it out later", but I'm not sure if that's an OK stance to take or not)