Log file Viewer
#pypa logs for Tuesday the 28th of July, 2015
[06:46:29] <fwilhelm> Hi there, I added setuptools>=9.0 to my install_requires and setup_requires but setuptools ignores it and stays with a much older version of setuptools. Is this an intended behavior?
[06:51:08] <mgedmin> like setuptools is already imported before it gets to process your setup_requires, so even if it installs a newer version of itself, your setup.py still has to deal with the old one
[06:54:54] <fwilhelm> @mgedmin, thanks, this is also what I expected... but it gets ignored and I find nothing on the web about it.
[08:23:19] <ronny> fwilhelm: ez_setup is drop in python script that will update/install setuptools before running setup, you have to ship it with your project
[08:40:27] <asfaltboy> our "setup.cfg" file contains: "[install] install-lib = /opt/nsfx" however only a non-wheel install seems to respect this
[08:44:12] <asfaltboy> but not all options are supported, as mentioned here https://docs.python.org/2/install/#custom-installation
[08:45:36] <asfaltboy> actually, it might still be in discussion, as there's this issue: https://github.com/pypa/pip/issues/2677
[08:45:56] <ronny> asfaltboy: wheels are always installed in site, you can use pip install --target to put them int a different location
[08:46:28] <ronny> asfaltboy: is there any reason why you employ that unusual sheme, such install path should be left to os native packages
[08:46:51] <ronny> asfaltboy: after all your setup.py deliberately breaks virtualenv usage or example
[08:48:00] <asfaltboy> what we're achieving by using the install-lib option is installing "supporting" (required) packages in the env site-packages, while the actual package (our app) being installed in a specified location
[08:48:54] <asfaltboy> it works perfectly well with source installation, and I am not aware of another way to achieve this...
[08:49:19] <asfaltboy> oh right, this is my last comment on that issue: https://github.com/pypa/pip/issues/2677#issuecomment-118139364
[08:49:59] <ronny> asfaltboy: bascially you deliberately break standart python tooling with your setup, wheel is not designed to support that
[08:50:53] <ronny> asfaltboy: either you install a wheel to a different target (pip install --target)
[08:51:22] <ronny> or you accept that what you did there was semi broken and working only by accident to begin with
[08:51:35] <ronny> asfaltboy: whats the reason you cant instal your lib into the standard python locations?
[08:52:25] <asfaltboy> I assume the better solution today is install everything to standard locations and hook with scripts ?
[08:53:13] <ronny> its certainly a better solution to go with standard locations, else you break tools like virtualenv
[08:55:41] <ronny> (or if you need the very special destination control, compose native operating system packages )
[08:56:27] <asfaltboy> I believe the requirement was to deploy the "app" to a given path on target machine, while still using pip
[08:57:59] <mgedmin> and if you're fine installing it into /usr/local/bin/ourapp, then pip will work wonderfully
[08:59:44] <asfaltboy> though it's not really a "single binary", in many cases we're talking about web apps with a wsgi module
[09:01:00] <mgedmin> have you considered a tool other than pip for installation? maybe wrapping pip (in a virtualenv)
[09:01:25] <asfaltboy> i.e ops felt it better to configure "gunicorn" to load the wsgi app from /opt/app/wsgi.py than /home/appuser/.venvs/app/lib/ptyon2.7/site-packages/app/wsgi.py
[09:01:26] <mgedmin> I believe the usual way of deploying python wsgi apps is virtualenv + pip install + write a wsgi script wrapper that sets up sys.path for the virtualenv
[09:02:05] <mgedmin> (the writing of a wsgi script part is the one I kind of hate, and sometimes you can avoid it)
[09:02:51] <asfaltboy> sounds like there isn't really a "usual way", hence our way was no worse then the other workarounds
[09:02:53] <mgedmin> (a typical wsgi wrapper execs the virtualenv's activate_this.py and then imports the wsgi app from the right package/module)
[09:03:55] <mgedmin> perhaps it would be sufficient to use pip install --no-use-wheel as a workaround for now?
[09:05:20] <mgedmin> there are plans to extend the wheel spec so you can install things like man pages etc
[09:05:38] <asfaltboy> for example, sentry installs some scripts into $HOME/.sentry while the app is in env/site-packages
[09:05:39] <mgedmin> I'm not sure the plans include the ability to write to arbitrary locations in the root file system
[09:07:04] <mgedmin> so, back to wheels: the desire to install stuff other than scripts and python packages exists
[09:09:18] <mgedmin> I believe the last thread on this topic was called "elements of setuptools/wheel/FHS file categories proposal"
[09:40:36] <fwilhelm> ronny: Would ez_setup also work with wheels? Can I run as a preprocessing script ez_setup from a wheel file?
[09:49:25] <fwilhelm> ronny: I know. In your setuptools_scm you have _warn_if_setuptools_outdated(). Would it be possible to instead enforce setuptools >= 12 when someone downloads the wheel of setuptools_scm? In the case that the sdist is installed due to setup_requires one would need to use ez_setup.py from within setup.py in the sdist if I got you right.
[09:51:52] <mgedmin> what happens if you have install_requires=['setuptools >= 12'] and someone with an older setuptools tries to pip install?
[09:53:28] <ronny> fwilhelm: its a really tricky domain, and i experimented with automation, however its fragile
[09:56:53] <fwilhelm> ronny: okay, you have surely done a lot more in this domain than I have. So if that's your conclusion I would rather remove setuptools from my requirements and let it fail late.
[09:58:06] <ronny> fwilhelm: because then you can mostly avoid it, setuptools_scm is just needed for creating the wheel
[09:58:54] <ronny> fwilhelm: i did some prelimitary work on creating a sdist command, which edits setuptools_scm out of setup.py on sdist but it was also very fragile
[10:00:47] <fwilhelm> ronny: since my package is also used in setup_requires I think it is not possible to ship wheel-only, right?
[10:01:15] <fwilhelm> ronny: I wonder when setuptools will use pip and wheel to fetch setup_requires
[10:01:40] <ronny> fwilhelm: when someone did implement it, its rather tricky and both jason and me are stripped for time
[10:06:35] <fwilhelm> ronny: thanks for your explanations! I really appreciate the work of the setuptools/pip developers. Of course there are still some rough edges but it is a huge step forward for python packaging.
[10:07:22] <ronny> unfortunately i reached the conclusing that the way virtualenv and pip work today is a huge step back a while a go (after understanding how egg and buildout work
[10:08:16] <fwilhelm> I think I am missing that understanding then... So you say the old eggs are better?
[10:08:59] <ronny> fwilhelm: eggs as format are horrible, but the ways buidouts used eggs was far more powerfull than what pip+virtualenv can do
[10:09:35] <ronny> i have some plans to bring some of the advantages of eggs back, but its still fuzzy, i lack time
[10:10:31] <ronny> different scripts within one environment can have mutally exclusive requirements/dependencies
[10:10:51] <ronny> so you could do things like have a script that has a completely different set of requireements than another script
[10:11:58] <ronny> however pythons import system is fundamentally lacking, its ipossible to get rid of sys.modules, so we will never ever get the ability to have different imported module sets in different contexts
[10:14:06] <ronny> you couldnt have multipe versions in the same process, but you could have them in scripts
[10:15:28] <fwilhelm> that really is an advantage. from a user's perspective having a separate venv for each project is really annoying. And if your library starts requiring min/max versions of other libraries, you end up in the dependency hell.
[10:16:16] <fwilhelm> but you say that this an inherently python related problem? Since you can import a module only once?
[10:18:20] <ronny> well, its much more manageable if you can have different sets of rependencies per process
[10:18:49] <ronny> like for xample if your test framework uses a version of the plugin anager that the app also uses and they are incompaible
[10:23:43] <ronny> (i.e. have a repo of materealized wheels, symlink the python packages and write a adapted dist-info folder)
[10:33:54] <ronny> most python packages you install in a virtualenv you dont change, it having them installed in a non-materealized fashion seems a nice space-safer
[10:34:26] <fwilhelm> ronny: I mean if two venvs use the same version of a library they access the some blob, if one install another version (write), the link points to the new version for some project
[10:36:33] <fwilhelm> but if a new version of a library is like a change to a blob, then one could see it like that I guess
[10:37:40] <ronny> fwilhelm: bascially you have a repository of materealized wheels, and you reference to one of those
[11:40:28] <fwilhelm> ronny: so in the case of a normal venv with pip a new version of a library would just replace the old version. With egg, instead of replacing(write) you copy the library to another place, this is why I brought the metaphor. But I agree that your interpretation is conceptionally more suitable than mine.
[12:27:42] <rrva> if I copy a whl file from another machine, which is also cp26-none-linux_x86_64, and this makes the python runnning pip sigsegv, in PyGC_Collect(), is this considered a pip bug, a python bug, or just wrong use of .whl files?
[12:38:53] <ronny> rrva: wrong use wf wheel files, linux abi compatibility is distribution/version speciic,
[12:54:22] <ronny> ionelmc: well, it exists, on a very personal level, and never look at it wrong, or it will fail you
[19:26:25] <agx> I have a git url git+git@....git#egg=MyPackage, it has a requirements.txt and a setup.py. inside its requirements.txt it has another private git+git... requirement (transitive dep). when I install 'MyPackage' i get a No matching distribution found for None