Log file Viewer
#pypa logs for Wednesday the 15th of April, 2015
[12:09:20] <ronny> dstufft: you didnt answer on my question of being able to declare different commands for pip via a tox.ini section
[12:10:16] <ronny> and adding on that, having a pip dist command that defauls to setup.py sdist and can be configured as well
[12:12:00] <dstufft> setup.cfg is a natural one to look at for a quick shim, longer term I dunno, not a file that's for a different third party thing though
[12:13:19] <ronny> dstufft: btw, can we have a notion of pip plugins (which are used to obtain the dependency metadata
[12:16:32] <dstufft> ronny: long term sdsits will always get metadata from PEP 426 metadata (except for legacy) and wheels will always get it from PEP 426 metadata (except for legacy), the only real thing we'll still need that we don't have a plan for yet is how do we go from development checkout to a sdist
[12:18:06] <dstufft> ronny: right, hence the "only real thing we don't have a plan for is how do we go from checkout to sdist" ;)
[12:22:43] <ronny> dstufft: tox creates dists for installing them so it can run tests against the installed packages
[12:23:41] <dstufft> ronny: tox also has to install dependencies, that doesn't mean we should rely on tox for dependency resolution. Tox should call _us_, we shouldn't be calling tox.
[12:24:42] <ronny> dstufft: but well, scm tree to metadata needs a solution, either you rely on something that knows how to do, or you need to invent something own
[12:25:28] <dstufft> and creating a sdist isn't a core part of tox, hence why it calls out to setuptools or distutils to do it
[12:26:16] <ronny> dstufft: one part of me making gumby elf is teaching tox about other things that make distributions and distribution building time dependencies
[12:27:39] <dstufft> ronny: I feel pretty strongly that tox has no business knowing how to do anything with sdists except call the tool that does know how to do it
[12:28:18] <ronny> dstufft: yes, which is exactly the job of tox, so why should pip replicate exactly that job instead of relying on tox to do it
[12:28:59] <ronny> dstufft: in the setup im proposing all tox will know is what commands to invoke to get metadata and distribution archives
[12:29:28] <dstufft> ronny: why wouldn't pip just learn how to do those things (and then tox call ``pip sdist`` or something)
[12:30:12] <dstufft> why is an environment and test runner the interface to creating sdists instead of the packaging tool
[12:33:21] <dstufft> ronny: pip's already in the business of doing more things than installing distributions, it installs things, it inspects the current environment, it uninstalls things, it builds wheels, in the future it'll absorb twine and then it'll upload distributions too, and either way pip will need to know how to go from a checkout to a sdist in order to handle pip install -e so it feels natural to expose that
[12:35:53] <ronny> dstufft: one needs at lease some kind of plugin interface and a way to specify the required plugins that pip needs to build a distribution or extract metadata from a source tree
[12:36:32] <dstufft> ronny: as far as how do we teach pip, ideally someone would need to propose a mechanism for doing this and it'll need to run through the paces of discussion (and likely a PEP). Ideally I think the way it would work is that you'd have some sort of configuration file which would say what tool you need to use to create an sdist (and possibly what the entry point is for calling that tool? or would there just be a standard entry point?) and then pip
[12:36:32] <dstufft> would inspect that file and install that tool (if it doesn't exist) and then call into it
[12:40:49] <dstufft> the basic sketch in my head is that we'd have a PEP for this, and it would define the file we use (setup.cfg? It's an easy choice but not sure it makes sense since it's historically tied to distutils/setuptools) and it'd do something like: https://bpaste.net/show/2ed89c635519 and then gumby elf defines some standard entry points (or maybe the entry points get specified in the setup.cfg too, idk)
[12:41:17] <dstufft> probably we'd need to define the API that gumby elf (or anything else) would need to expose too
[12:45:25] <dstufft> maybe! I haven't thought it through that far, to further push it, it needs someone to sit down and really think hard about how to enable a ``pip sdist`` command that could function with arbitrary sdist building mechanisms and do the research and experiments and write up a PEP. If nobody else gets to it I'll end up doing it but my own personal priority is working backwards so focusing on PEP 426 (and a new iteration on Wheel and a sdist 2.0
[16:45:39] <ronny> and for the other - different handling of setup_requires, and supporting to avoid easy_install invokation
[16:52:05] <ronny> jaraco: and the second one will do the normal egg fetching when one disabes auto-easy-install
[16:53:10] <ronny> jaraco: and if auto easy install is disable, the install command would error out printing the missing dependencies
[16:53:23] <jaraco> So currently, setup_requires are resolved very early, long before commands are parsed, so that setup_requires requirements are present for implementing commands.
[16:54:32] <jaraco> And it's distutils that implements the models for commands and command-line parsing.
[16:54:53] <ronny> jaraco: bascially opting out of them would put it in a state where any command but the command to fetch and list would bail out telling to fetch/install the eggs
[16:55:06] <jaraco> So you couldn't use a proper distutils command to print setup_requires without having already resolved them.
[16:55:10] <ronny> jaraco: ah, and the other thing i need is adding new subcommands to commands as extensions
[16:57:43] <jaraco> I imagine you could potentially pre-process the command line and if it has that command, bypass the normal distutils handling. That feels a little heavy-handed, but might work. It doesn't feel like a very general solution.
[16:59:20] <jaraco> Another approach could be to adopt distutils (vendor it) such that setuptools no longer depends on the stdlib version of it... and then customize the command handling such that some commands are explicitly processed prior to invoking setup_requires. However, I would be worried that vendoring distutils could cause other problems, such as with buildout and other packages that depend on specific exceptions derived from the stdlib.
[17:00:47] <ronny> jaraco: i would disable easy_installing the setup_requires eggs to the .egg folder
[17:01:09] <ronny> jaraco: its a opt out for a reason, packages that would completely fall apart cant opt out
[17:01:39] <ronny> and setuptools would have a check in all commands and simply refuse to act unless setup_requires are availiable
[17:04:43] <ronny> jaraco: but i suspect that practically most packages with setup_requires can opt out of the immediate processing
[17:05:22] <ronny> and then there could be a setup.py fetch_setup_requires wuth a --use-wheel option ^^
[17:06:19] <ronny> jaraco: the basci idea behind this is to get a reliable way to completely opt out of easy_install and potentially even eggs for setup_requires
[17:09:01] <jaraco> ronny, I have a lot of packages that have a setup_requires that looks like this - https://bitbucket.org/jaraco/jaraco.functools/src/4a79f0873d961dcecf7edee3ea3f9f2dd85ac97d/setup.py?at=default#cl-26
[17:09:43] <jaraco> In keyring, I was specifically requested to selectively include setup_requires only when necessary (using an imperfect heuristic).
[17:10:28] <ronny> jaraco: tbh, thats a shortcoming of setuptools, its impossible to replace builtin commands
[17:11:35] <ronny> jaraco: there is no way to specifiy what command requires what setup_requires dependencies
[17:12:58] <jaraco> And as far as I know, there's not even a draft solution in the upcoming PEPs and Metadata definitions.
[17:13:16] <ronny> jaraco: to set it up propperly you would need something like a command_requires={'test': ['pbr'], 'build_sphinx': [ 'sphinx>=1.2'], }
[17:13:31] <ronny> but tbh these days i dont see a problem with just fetching wheels for all of them
[17:17:07] <ronny> jaraco: thats basically lack of support for propper command overrides and requirements for that
[17:18:41] <jaraco> In the case of that last ticket, the command is 'ptr', and at the time, it didn't need to override anything. It added functionality.
[17:18:45] <ronny> jaraco: test_requires is part of the test command for example, so its ipossible to override it from a not yet instalede dependency in a aliased manner
[17:19:40] <jaraco> Maybe pytest-runner is a bad example. It confuses a lot of people because it seems like a test dependency. It's not a test dependency, but a setuptools plugin implementing another command.
[17:19:54] <jaraco> We have the same problem with setuptools_scm. Not all commands require resolution of a version number.
[17:23:07] <ronny> i tend to use only wheel based setup_requires, so these days, they would bascially be instantly availiable for any normal usage of pip
[17:23:55] <ronny> so not a battle worth fighting to be tetailed about setup_requires, but easy_install breakign stuff should be out-outable or swichable for invoking pip
[17:25:56] <jaraco> I guess what I'm saying is that setup_requires solves a very broad problem (so hit imperfectly). Because of that, it's created hacks like the aforementioned conditional inclusion in setup_requires.
[17:27:04] <jaraco> So if you want a package that can inspect setup_requires in advance, I feel like we have to solve those cases first, so those dependencies aren't missed.
[17:27:38] <jaraco> Because if you were to invoke "setup print_setup_requires" on keyring, you wouldn't get sphinx or pytest-runner.
[17:28:37] <jaraco> But if you subsequently installed those setup_requires and then tried to run `./setup --dont-download-setup-requires build_docs`, it would fail because sphinx isn't present.
[17:31:42] <ronny> where '' is for any case, and the other things can be command names abused as extras marker
[17:32:13] <ronny> jaraco: but still - the issue is orthogonal to opting out, for a package where you do conditionals you wouldnt opt out
[17:33:51] <ronny> jaraco: bascially i want to have disable_implied_easy_install=True, and thn it will bail out instead of installing setup_requires in .egg and install_requires via normal easy_install
[17:35:25] <jaraco> What happens if someone depending on your package tries to install it and hasn't already installed the setup_requires dependency?
[17:37:02] <ronny> and with easy_install as far as i understood it knows when its on the sandbox, so it would print a warning instead of opting out
[17:39:20] <ronny> jaraco: bascially opting out of setup would only be allowed by direct setup.py invocation
[17:43:30] <jaraco> I'm still unsure if that works. What if I clone the project and run ./setup develop on it. Does that also opt out? Would I need to install the dependency before developing the project/
[17:44:46] <ronny> it would print an error, telling that it won automatically install setup/install requires , and tell what commands list them/suggest pip install -e . / easy-install -e .
[17:45:00] <jaraco> I'm more inclined to think the opt-out should be something that happens at command-invocation and not in the package definition.
[17:46:18] <jaraco> That feels hostile to me - requiring a developer to take manual steps and affect his execution environment before developing a third-party package.
[17:47:14] <ronny> jaraco: that presumes eople are fine with eggs being shat into site-pacakges and .eggs
[17:50:22] <ronny> jaraco: activated setup requires eggs create strange install requires issues, even worse when install requires are stacked in some way
[17:50:49] <ronny> (a friend of mine created a wrokaround on top of setuptools for that one since he didnt want to dig into setuptools)
[17:52:18] <CVirus> Recent versions of pip don't show the compile output of the package being installed .. how can I force it to spit this output ?
[17:53:58] <jaraco> ronny: why if there are setup_requires issues wouldn't we first explore fixing those issues rather than bypassing the functionality?
[17:54:32] <ronny> jaraco: i dont recall the exact fix, i talked with him about it at the pythonbarcamp in cologne, but currently i cant reach him
[17:58:46] <jaraco> I just mean generally - why should we invest time in disabling setup_requires instead of characterizing the issues and solving the underlying problems so that it works smoothly?
[17:59:53] <ronny> jaraco: that still means implicit invocations of easy install on any setup call and on python setup.py install
[18:02:25] <ronny> jaraco: please keep in mind that what setuptools does is the same as if ./configure for a gtk app would download and compile gtk if missing ^^
[18:04:17] <jaraco> It's more as if ./configure for a gtk app would download a C compiler and use that to compile the GTK app. setup_requires is for setup-time dependencies, which can be discarded after the package is built/installed.
[18:05:04] <jaraco> They're intentionally transient, and that's important because you don't want a setup-time dependency for one package to affect the setup-time dependency for another package.
[18:05:05] <ronny> jaraco: but in any case, currently setup.py does invocations of a package manager, and even more strange ones if one has setup_requires
[18:06:23] <ronny> i want that to happen only if one does something like easy_install ./pip install .
[18:06:50] <jaraco> ronny, have you considered opting out of setup_requires altogether? That would have a similar effect (failing when the required package isn't already present).
[18:09:51] <ronny> jaraco: bascially i'd like to kill implicit invocations of packe managers, so people needto invoke the package manager to install the package and the dependencies
[18:10:37] <ronny> so setup.py develop/install/build/... would all refuse unless all is there, and easy_install ./pip install . would do the right things
[18:11:01] <ronny> where easy_install would use eggs, and pip uld use unpacked wheels for the setup_requires
[18:19:28] <jaraco> ronny: It still feels hostile to me, breaking use-cases that I see often in development and testing.
[18:21:27] <ronny> jaraco: from my side, the implicit invocation of package managers is problematic, it creates a mess for upstream packaging and for various use-cases
[18:21:35] <jaraco> Maybe it's viable. I worry that pip install would unpack wheels and break when namespace packages are involved. pip doesn't handle namespace packages in more than one location.
[18:22:13] <ronny> jaraco: activation of unpacked wheels would be a pkg_resources workingset operation
[18:23:02] <ronny> also many people avoid namespaces to begin with because they break in various situations
[18:25:41] <jaraco> Say you have 'jaraco.functools' in system site-packages, and you install a package that setup_requires 'jaraco.itertools' and 'jaraco.functools'. Can jaraco.itertools be installed and added to the working set without being in system site-packages?
[18:26:22] <ronny> jaraco: unpacked wheels in a own folders are practically just like a unpacked egg just with a dist-info instead of a egg-info
[18:26:58] <ronny> jaraco: also namespaced packages are half-broken on single-version-externally-managed
[18:28:31] <ronny> jaraco: namespaced packges in single_version_externally managed mode (as made by pip) combined with a develop install cannot work
[18:29:07] <jaraco> hmm. Is that because there's no hook to activate the distinct packages in the working set?
[18:29:12] <ronny> jaraco: a solution to propperly fix that would put each wheel inpacked in a own folder and add a new import hook that directly dispatches toplevel to the correct wheel
[18:29:31] <ronny> jaraco: its not a activation issue, its a pythonpath borked with strange things issue
[18:29:59] <ronny> afair activation will either be in a strange state or bail out for develop installs (
[18:30:35] <tdsmith> i've noticed that when namespace packages are installed to prefixes that are not the distribution site-packages, they are broken because the .pth file isn't processed
[18:33:57] <jaraco> ronny, you've convinced me there may be something to this idea. If you could prepare a PR, maybe with some documentation describing how this technique works best, I'd entertain it.
[18:34:17] <DanielHolth> unfortunately namespace packages are fragile, even in Python 3. It would be nice to get a warning from the installer if it thought you were breaking a namespace.
[18:34:28] <jaraco> Python 3.3 with PEP 420 is awesome. I can't wait until everyone else joins me there.
[18:37:28] <jaraco> ronny: by writing code and publishing packages. I'm not sure I understand the question.
[18:38:39] <ronny> jaraco: pep 420 namespaces are not supposed to have a __init__.py as far as i understood
[18:39:27] <DanielHolth> the answer is that you don't do that, the PEP 420 namespace is empty except for import namespace.subpackage
[18:50:49] <jaraco> It's safe to have a __init__.py in one package or even in all of them, but the behavior is undefined if more than one __init__.py exists with differing content.
[19:00:30] <ronny> jaraco: apipkg relaces the module in sys.module with something that can do lazy importation of functions/aliased modules/other apipackages
[19:01:41] <jaraco> ronny, I believe if you can do it in a standard package, you can do it in a PEP-420 namespace package... the only difference being that only one __init__.py is ever invoked for a given package.
[19:02:23] <ronny> jaraco: the thing is apipkg will replace sys.modules[__name__] with something else
[19:05:02] <ronny> (so import mypkg will return what mypkg/__init__.py did put into sys.modules['mypkg']
[19:05:59] <ronny> jaraco: is __path__ set to the correct specia type before __init__.py is executed?
[19:13:11] <ronny> jaraco: but pep420 namespaces have that special object that adopts based on sys.path?
[19:14:41] <jaraco> I'm not sure. I did print(__path__) in __init__.py and got: ['C:\\Users\\jaraco\\foo']
[21:42:30] <dstufft> nanonyme: depends on what version of pip you're using. We bundled certifi in some versions, but we removed it, if you're using the latest pip, then yes.
[21:46:27] <_habnabit> ummm does pip not give output from `setup.py install` anymore? i did `pip -vvv install -vvv ...` and all it says is 'Running setup.py install for ...' and then 'Running command ...' but it doesn't show the output
[22:08:55] <_habnabit> dstufft, somewhat relatedly, do i misunderstand how --no-clean works or is something broken with it? i see pip-build-{...} directories being created but they're empty after pip finishes
[22:12:34] <dstufft> _habnabit: I think we have multiple types of temporary build directories and only some of them are controlled by --no-clean
[22:25:07] <_habnabit> dstufft, so is there any other way to keep these directories around? ultimately the problem i'm trying to solve is saving gcov .gcno files, and i'm wondering if i have to do some crap in setup.py to save them then
[22:35:18] <_habnabit> huh, this might've been around for a long time? it appears to be there in commits from 2009
[22:52:04] <_habnabit> dstufft, are the pip integration tests not usually run on OS X? the unit tests run pretty quickly and all pass but the integration tests all fail hard on the shutil.rmtree(appdirs.user_cache_dir("pip")) line
[23:03:47] <_habnabit> dstufft, https://github.com/habnabit/pip/commit/85556a5441fa7c4b8663da7f74ad7b9dfa0e5e42 does this make sense? i'm trying to figure out the best way to make this test pass atm