Log file Viewer
#pypa-dev logs for Tuesday the 5th of May, 2015
[02:09:18] <dstufft> lifeless: Um, not sure offhand. I suppose it's about time to take a look at the open issues and see if anything stands out as something to address
[02:09:43] <dstufft> lifeless: tomprince did notice the other day (I forget if I opened an issue..) that the develop branch breaks ``pip install .`` reinstalling itself always
[02:16:05] <lifeless> yeah, I don't recall it. But basically we want to treat any path based requirement as having an implied -U ?
[02:20:08] <dstufft> git checkout 1.5.6 && pip install pip==1.5.6 && pip install ~p/pip does the same thing
[02:53:32] <dstufft> lifeless: distutils-sig for higher level things that are more broader reaching, and pypa-dev for stuff that is more pip specific
[09:26:11] <ronny> lifeless, dstufft : btw, any plans to integrate between devpi caching and pip caching/wheel chaching (i for example notice that using devpi with different indexes spams my pip cache with identical files from different locations)
[19:22:44] <ronny> jaraco: im wondering if using wheels as zip unsafe eggs and a base package for optimizing toplevel imports for the norma setup could help with setuptools issue 229 (devendored deps)
[19:24:11] <ronny> jaraco: so basically all pip/setuptools requirements would always land in $SITE_PACKAGES/$name-$version-....whl/
[19:24:47] <ronny> then it would be more reasonably to split it up into tightly pinned versions of setuptools, pkg_ressources, packaging and so on
[19:25:12] <ronny> (the installer would bascialyl inpack the wheels into wheel folders and enable putting the right versions at the beginning of sys.path
[19:26:19] <ronny> jaraco: well yes, its kind of necessary for anything where upgrades/downgrades break the world
[19:26:21] <jaraco> One issue with that approach is if something else (pip, easy_install, other) has already installed a conflicting version into sys.path, you're still in trouble.
[19:26:57] <ronny> jaraco: thats why setuptools scripts invoke that heavy pkg_ressources machinery instead of
[19:27:41] <ronny> so the idea would be that the script ask for the exact versions, those set up the importer, then the tools can run with the correct lib versions right at the start of sys.path
[19:28:44] <ronny> an optimization could be to feed some informations about the base setup wrt versions into the scripts (to get inexpensive toplevels)
[19:29:02] <DanielHolth> I'd love to see an installer that extracts each package into its own versioned directory like buildout.
[19:29:41] <ronny> DanielHolth: i have a plan for that actually, doing it with wheels instead of eggs, just just couldnt do the sprint on it
[19:30:14] <DanielHolth> To reduce confusion just don't call them wheels after they are unzipped. You could call them dists
[19:30:17] <ronny> dstufft: with a setup like im describing, pip could just ship the unzipped wheels and add them to site
[19:31:41] <ronny> dstufft: bascailyl once pkg_ressources/setuptoops support a layout with unpacked wheels as "multiversion" packages, pip will no longer need to vender specific versions since its installation can just put them in place and the pip script will use them
[19:32:28] <DanielHolth> I think pip has the right idea, but your installer (a similar one someone else has written, but I forget the name) will be a great alternative to virtualenvs when you just want to use some app.
[19:35:34] <ronny> jaraco: but yeah, as first step, i'd like to suppport downloading and unpacking/caching wheels for details like setup_requires
[19:35:48] <DanielHolth> I thought pkg_resources had a public API for grabbing arbitrary files out of any package's .egg-info folder
[19:36:37] <ronny> DanielHolth: eggs are per efault installed as eggs within site-packages/custom folders, and a pth file prepends them to sys.path
[19:37:07] <ronny> DanielHolth: and easy-install will keep zip-safe eggs as zip file and zip-unsafe eggs as unpacked folers
[19:37:29] <ronny> DanielHolth: buildout is workign on top of that just unpakcing them all and constructing a own workingset
[19:38:50] <ronny> jaraco: one "feature" of pkg_resources is that it can hide a package in sys.path behind a package in a egg by putting the egg first in sys.path
[19:39:18] <ronny> jaraco: however there are some detail issues wrt package paths in workingsets when a non-egg installed version of something is required
[19:39:29] <jaraco> ronny: Have you seen the mechanism where pkg_resources will look at __main__.__requires__ and make sure those requirements are present?
[19:39:51] <DanielHolth> So I think the reason buildout works a little better is that the individual scripts for applications set the sys.path they needed, and then when you install anything else the existing scripts continue to work the same.
[19:40:13] <DanielHolth> Instead of the old "I need this package" on the top of the script, which would re-run dependency analysis for whatever you had installed just then.
[19:40:28] <ronny> DanielHolth: however as far as i understood that breaks plugin systems based on entrypoints
[19:41:22] <DanielHolth> Basically the first egg use case was as a file you'd drop into some plugin folder.
[19:43:04] <ronny> jaraco: i see what you mean, i'd completely do away with the easy_install.pth for something that also includes information about toplevels
[19:43:27] <ronny> jaraco: because one of the main issues with eggs is, that having quite a few of them makes import ______slow______ and a mess
[19:44:10] <dstufft> easy_install.pth was only added if I recall becasue people wanted to have default versions available when they did ``python``
[19:44:37] <dstufft> and easy_install didn't have an option to install things as pip installs them that didn't also turn off dependency resolution completely
[19:45:04] <dstufft> walking a largely sys.path is still expensive, it's better but still expensnive
[19:47:46] <ronny> basically there is need for some kind of meta importer that makes sys.path lean and instead dispatches straight into the toplevels of requested distributions
[19:48:18] <ronny> so that if there is like 300 packages in the workingset it will still only have to open one file
[19:51:56] <DanielHolth> So I'm looking at what I might have to do to write the per-package install scheme used into its .egg-info directory, and then pull it back out again at runtime. However it's annoying to figure out how to determine whether egg-info is being generated as part of a develop or regular install, or as building a bdist/sdist. https://bitbucket.org/dholth/wheel/commits/72eff6db19f918294154c2bfecf28493e4352bd7
[19:53:27] <ronny> dstufft: im writing something that will generate wheels commpletely without setuptools
[19:54:34] <DanielHolth> you and flit and this wscript https://bitbucket.org/dholth/wheel/src/tip/wscript?at=default and the old Bento patch I did that I never updated :)
[19:57:12] <ronny> DanielHolth: its called gumby_elf , atm im adding requirements writing, then i will add normal wheels and sdists
[19:57:45] <ronny> (currently i only support "develop mode wheels" which basically generates a wheel with a bootstraper to import from the source tree)
[19:59:56] <dstufft> DanielHolth: my new job doesn't really involve any openstack, other than in the sense they are a major consumer of python packaging (though I'm techincally on the cloud team, but that's mostly for the aforementioned reason)
[20:11:48] <DanielHolth> It's a relative of SCons, but I never understood it well enough to build a general purpose setupt.py replacement.
[20:13:04] <DanielHolth> But part of the problem with that project idea for me is that I don't actually write CPython extensions.
[20:14:22] <ronny> DanielHolth: yes, but it nethier does nice base metadata, nor backward-compatible sdists nor whole packages
[20:14:39] <DanielHolth> No, I definitely want you to write your own, just trying to provide some prior art.
[20:15:17] <DanielHolth> Don't reinvent the wheel unless you are prepared to learn a lot about wheels.
[20:15:43] <ronny> DanielHolth: im relatively fluent in pip, vrtualenv, setuptools and a few other things by now ^^
[20:25:13] <ronny> DanielHolth: i thik for the initial versions of gumby elf i'll actually steal your ideas from wheel and make the .in files + stuff
[20:27:43] <ronny> tol is a no-go, reminsd me of the cinfigparser mess (a non stdlib ini file parser)
[20:52:56] <jaraco> Daniel discovered it was broken, and it was fixed in a prior version, but a new issue had emerged with the latest pymongo, so I had to fix that.
[22:40:33] <dstufft> lifeless: no, I forgot. Will take a look after I sort out this health insurance crap