Log file Viewer
#pypa-dev logs for Thursday the 26th of March, 2015
[01:43:21] <lifeless> dstufft: you seem to be getting torn both ways quite a lot :) - I've been getting whiplash reading your updates
[01:44:10] <dstufft> my comments tend to more or less be a lot of raw "wahtever my thoughts are at this exact moment"
[01:51:46] <lifeless> dstufft: I am fairly sure now though, that lots and lots of setup_requires are install_requires
[01:59:48] <dstufft> lifeless: hopefully this hasn't be too terribly painful, there's a lot of postel's tarpit going on so changes tend to get a lot of "on the fence"-ness as people try to decide whether the potential breakages are worth the change
[02:01:26] <lifeless> dstufft: I'd like the assessment to be 1) 'is (a) reasonable from first principles' and 2) 'will it harm the ecosystem if we do it' :)
[03:24:32] <dstufft> so things that appear first in the dependency list won't be installed before the setup.py gets executed
[03:28:31] <dstufft> but I don't think that (a) is reasonable for pip wheel, which means you'd need to implement (b) for pip wheel anyways, either inside of WheelBuilder or as part of RequirementSet
[03:45:51] <dstufft> lifeless: ah yea, I forget what all the letters were being stand ins for ;) (Or rather Iw as mentally associating multiple things together)
[20:57:53] <dstufft> lifeless: heya, I'm not going be here much longer. Have a ballet thing for Alyssa in a bit
[20:59:18] <lifeless> dstufft: I wanted to check out design for the iteration needed to align with your post
[20:59:52] <lifeless> dstufft: e.g. I have no idea how to sanely do the thing you asked for vis-a-vis *not* using a venv but still having isolated installs
[21:03:13] <dstufft> lifeless: I would do something like, make a temporary directory, install into that temporary directory using the same mechacnisms we use to implement ``pip install -t $TMPDIR/whatever``, then just do subprocess.Popen(..., env={"PYTHONPATH": "$TMPDIR/whatever"})
[21:13:27] <lifeless> dstufft: would it be reasonable to just whole-hogg it and build wheels for this case
[21:14:20] <dstufft> easier to implement? Yes probably, but it's going to break in strange situations and do unexpected things (we used to have a mode that pip would install "into" a virtual environment instead of being invoked inside of an environment, it didn't work very well) and it'll put a dependency of virtualenv on pip which is something i'd rather avoid (especially since we're starting to get Linux distros use us as part of their build chain)
[21:14:46] <dstufft> lifeless: you mean instead of invoking ``setup.py install`` invoke ``setup.py bdist_wheel`` and then install that wheel?
[21:17:41] <ionelmc> lifeless: so the setup_requires deps are getting installed in the current env or not?
[21:18:28] <lifeless> ionelmc: dstufft has decided against that, we're now discussing impl of the alternative
[21:18:44] <ionelmc> in case you haven't considered it, you could just dump em in a tmp dir, and site.addsitedir(the_tmp_dir) to get pth loading and all the stuff
[21:20:32] <ionelmc> lifeless: now i get it, you were talking about how to install the stuff in the tmpdir, right?
[21:22:06] <dstufft> lifeless: well anything else would have roughly equivalent performance to what we have today right? So it wouldn't be a regression in performance it just wouldn't get all of the possibly benefit of the change? I'm totally on board with building wheels and installing them, I'm just trying to determine if it can be done reasonably in a successive PR or if you'd end up needing to throw a bunch of code out.
[21:22:49] <dstufft> building a wheel first is going to break some things, I'm happy to tell those things they are doing it wrong, but we might need to phase it in
[21:23:26] <dstufft> I know it'll break old versions of Django, and anything that copied Django's setup.py because they did some monkeypatching in the setup.py
[21:24:56] <ionelmc> lifeless: what's wrong with --single-version-externally-managed (instead of wheels) ?
[21:25:26] <lifeless> ionelmc: my analysis of setup_requires showed that most of them are also install_requires
[21:26:06] <lifeless> if what we have now didn't suck, noone would complain or be wanting to change it :)
[21:31:47] <lifeless> let me rephrase. The limits to which you can optimise a particular thing are constrained by the design.
[21:32:40] <xafer> I'm all for the wheel building solution :) and if some projects do not support it, they'll not put their setup_requires inside setup.cfg
[21:33:56] <lifeless> ionelmc: there can be many correct designs for a given problem; picking the design to use needs to be done with performance in mind, otherwise you may well end up constrained to poor performance.
[21:34:44] <lifeless> xafer: dstufft has mentioned having pip build-wheels-and-install-wheels always, I was speculating on whether doing that now would make the overall work easier
[21:34:45] <ionelmc> on the other hand, can't you just use pip's install thingy to install those deps?
[21:35:29] <lifeless> ionelmc: and has constraints - you can't install anything other than a wheel without the necessary dependencies
[21:35:49] <lifeless> lots of setup.py's are written badly (e.g. they have undeclared setup_requires and fail if you don't install their install_requires first)
[21:36:27] <lifeless> most/all of which can be tracked back to /everyone/ hating how setup_requires works
[21:39:59] <xafer> lifeless, well if pip starts to take the setup_requires responsability it seems logical to directly build wheels
[21:43:15] <xafer> with ideally the recursive possibility to deal with setup_requires of a package present in a setup_requires
[21:43:53] <lifeless> the only issue with my current patch is that it does the pragmatic thing and installs into the target env
[22:40:41] <xafer> well via subprocess it seems like it would stay simple, an issue could be the detection of dependency loops among setup_requires