pmxbot IRC Log Viewer

[00:38:31] <lifeless> dstufft: should 'tox' Just Work in the pip source tree ?

[00:38:49] <dstufft> sort of

[00:39:00] <lifeless> should I open an issue when it barfs?

[00:39:02] <dstufft> we don't isolate from the outside environment the greatest in our test

[00:39:12] <dstufft> and we require svn, bzr, hg, and git installed

[00:39:39] <lifeless> I have that

[00:39:47] <dstufft> you can open an issue, or pastebin in here

[00:40:20] <lifeless> https://github.com/pypa/pip/issues/2562

[00:40:40] <dstufft> that

[00:40:43] <dstufft> is an interesting error

[00:40:46] <lifeless> yup

[00:40:52] <dstufft> I assume you have setuptools installed?

[00:40:57] <lifeless> my python3 does not

[00:41:24] <dstufft> which environment has tox isntalled into it

[00:41:40] <lifeless> I'm just sitting in my outer terminal

[00:41:46] <lifeless> no venv or lxc

[00:42:02] <dstufft> ok, is tox installed into python 2 or python 3 I mean

[00:42:49] <lifeless> head /usr/bin/tox

[00:42:49] <lifeless> #!/usr/bin/python3

[00:42:52] <lifeless> looks like python3

[00:43:03] <dstufft> you probably need to either install it into python2 or install setuptools in python3

[00:43:06] <lifeless> installed as a dep

[00:43:08] <lifeless> erm

[00:43:09] <lifeless> deb

[00:43:12] <lifeless> dpkg -l python-tox

[00:43:16] <lifeless> ii python-tox 1.8.0-1 all virtualenv-based automation of test activities

[00:43:25] <dstufft> you probablt need python3-setuptools then

[00:43:30] <lifeless> sure

[00:43:36] <lifeless> but why? I mean

[00:43:39] <lifeless> if tox needs it

[00:43:43] <lifeless> then the debian package is broken

[00:43:59] <lifeless> if tox doesn't need it, and its this tox.ini, we can try to fix it here, no?

[00:44:24] <dstufft> It's not this tox.ini, tox does ``sys.executable setup.py sdist`` before it creates any environments

[00:44:49] <dstufft> if pip's setup.py didn't require setuptools it would work

[00:44:50] <lifeless> so its missing setup-requires

[00:44:59] <lifeless> which is ironically exactly what I'm poking at

[00:45:33] <lifeless> so not solvable until this is done, at which point it could make a venv, etc

[00:46:16] <dstufft> well this is before any of that happens

[00:46:32] <dstufft> python-tox should probably depend on python-setuptools

[00:46:50] <dstufft> it already depends on python-pkg-resources

[00:47:40] <dstufft> (or at the very least, it should recommends)

[00:47:45] <lifeless> it has to because it uses entry_points for its own CLI

[00:47:53] <lifeless> via # EASY-INSTALL-ENTRY-SCRIPT: 'tox==1.8.0','console_scripts','tox'

[00:47:54] <lifeless> etc

[00:48:01] <dstufft> yea

[00:48:16] <lifeless> I'm saying, lets assume that setup.cfg with setup-requires was a thing

[00:48:25] <lifeless> tox could make a 'build' venv

[00:48:38] <lifeless> and use declarative setup-requires from that to populate the build deps

[00:48:42] <lifeless> and build the sdist with that

[00:48:46] <dstufft> yea it could do that

[00:54:18] <lifeless> dstufft: is there an existing test that would be a good basis for copying and editing to make a functional test around setup-requires ?

[00:55:12] <dstufft> you're probably going to want one of the ones in functional/test_install.py

[00:58:08] <dstufft> lifeless: https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=780702

[01:19:08] <lifeless> dstufft: how does one run just one test?

[01:19:28] <lifeless> dstufft: python-fox?

[01:19:37] <dstufft> er

[01:19:40] <dstufft> fucking auto correct

[01:19:42] <dstufft> lifeless: http://pytest.org/latest/usage.html#specifying-tests-selecting-tests

[01:20:39] <lifeless> dstufft: test_install or test_install_reqs.py?

[01:21:03] <dstufft> tox -e py34 tests/functional/test_install_req.py will work

[01:21:24] <lifeless> dstufft: test_without_setuptools - should that change to install setuptools?

[01:22:21] <dstufft> lifeless: not sure what you mean, is that a test that already exists?

[01:22:25] <lifeless> yes

[01:22:35] <lifeless> there is a test that says if setuptools is not installed error

[01:23:03] <lifeless> which, if we're going to install build dependencies seems contradictory

[01:24:17] <dstufft> which file is that in

[01:24:23] <lifeless> test_install.py

[01:25:09] <dstufft> lifeless: well you're not going to just automatically install setuptools unless there's a setup.cfg that tells you too right?

[01:25:23] <dstufft> so it still makes sense to error out if we don't have setuptools and that setup.cfg doesn't exist

[01:26:06] <lifeless> well

[01:26:20] <lifeless> I guess I'm asserting that all well written setup.cfg's would list setuptools

[01:26:41] <dstufft> sure, but not all packages that depend on setuptools are going to have a setup.cfg

[01:26:50] <lifeless> meh, I'll ignore it for now. Just having cognitive friction

[01:30:17] <tomprince> dstufft: Doesn't pip introduce an implicit setuptools dependency?

[01:30:25] <dstufft> no

[01:30:32] <dstufft> well, yes and no

[01:30:35] <tomprince> I recall you talking about having setuptools automatically installed during the build.

[01:30:37] <dstufft> pip does not require pip in it's own running

[01:30:41] <dstufft> er

[01:30:44] <dstufft> setuptools

[01:30:53] <dstufft> and pip can install from Wheels without setuptools being present

[01:31:06] <dstufft> it's only when installing from an sdist that setuptools is required

[01:31:36] <tomprince> Sure, but this is for setup_requires, which seems to imply installing from sdist, hence depending on setuptools.

[01:31:56] <dstufft> yea, but it's a setup_requires that's in a setup.cfg, so it's a wholly new feature

[01:32:25] <dstufft> there's severeal hundred thousand sdists that don't have a setup.cfg that lists setuptools as a setup_requires

[01:36:06] <tomprince> I'm recalling <dstufft> But the ideal future is that pip itself no longer depends on setuptools, and for all th legacy stuff, pip just assumes the builder is ``setup.py install`` + setuptools in abscence of the sdist 2.0/wheel features that tells it to use something else, then end users won't even have to know setuptools exists, if pip needs it, it'll go out and get it, otherwise it won't, and if

[01:36:08] <tomprince> $RANDOMPACKAGE doesn't want to use setuptools, they don't have to

[01:38:06] <tomprince> It seems, if pip is handling setup_requires, that it should implictly add setuptools to it.

[01:38:20] <dstufft> pip isn't handling setup_requires across the board

[01:38:42] <dstufft> the feature lifeless is added is an opt in, per package where you can specify your setup_requires inside of a setup.cfg

[01:38:46] <dstufft> adding*

[01:39:15] <dstufft> for every sdist that doesn't use the opt in feature lifeless is adding, we still need an error for not having setuptools

[01:39:40] <tomprince> Or, just implicitly install setuptools?

[01:40:12] <dstufft> It would be reasoanble for the "you need setuptools" error to be modified so it only occurs if there's not a setup.cfg with setup_requires" or an another change could implicit install setuptools though i'd rather that be in a different pull request

[01:46:11] <tomprince> I'm just imagining that, in the new codepath, instead of defaulting to an empty list with the setup.cfg's setup_requires, it is ['setuptools'] + setup.cfg's setup_requires.

[01:47:37] <tomprince> So, it simplifies the code slightly, in that it always runs, instead of being conditional.

[01:48:49] <dstufft> that could work, though it has the danger of forcing a setuptools upgrade of -U is used so we'd want to be careful about that

[01:49:47] <dstufft> IOW we wouldn't want the implicit setuptools requirement to cause setuptools to get reuqired because you did ``pip install -U thing-that-is-a-sdist``

[01:50:39] <lifeless> well

[01:50:47] <lifeless> thats what will happen for other setup requires

[01:50:55] <lifeless> but separate discussion

[01:50:59] <lifeless> let me wrap this up first

[01:52:33] <dstufft> (that could be done by just conditionally adding the implicit setuptools to the list of things to be installed if it's not already installed)

[01:52:42] <dstufft> s/done/prevented/

[01:53:27] <dstufft> so your default would be try: import setuptools; setup_requires = []; except ImportError: setup_requires = ["setuptools"]

[02:01:45] <lifeless> dstufft: whats a suitable ini file handler for 2.6?

[02:01:54] <lifeless> dstufft: I presume pip doesn't support 2.5 any more

[02:02:02] <dstufft> 2.6 and 3.2+

[02:02:28] <dstufft> I think configparser is in 2.6?

[02:02:42] <lifeless> oh yeah

[02:02:44] <lifeless> freaking Case

[02:03:22] <dstufft> we have six

[02:03:34] <dstufft> from pip._vendor.six.moves import configparser should work IIRC

[02:04:15] <lifeless> yeah

[02:19:00] <lifeless> dstufft: does pip use a recursive approach to deps, or some sort of iterator?

[02:19:13] <lifeless> dstufft: e.g. when I find out that setup-requires is needed, do I call deeper to make it install it

[02:19:28] <lifeless> dstufft: or should I be raising some error and pip can call back in once its satisfied?

[02:19:59] <dstufft> during the collecting phase it downloads + iterates over it

[02:20:03] <dstufft> lemme link

[02:20:37] <dstufft> you're going to cry

[02:20:42] <dstufft> because this is some of the worst code in pip

[02:20:46] <lifeless> dstufft: of course I am

[02:21:43] <lifeless> this loop -

[02:21:44] <lifeless> reqs = list(self.requirements.values())

[02:21:44] <lifeless> while reqs or unnamed:

[02:21:45] <lifeless> ?

[02:21:48] <dstufft> https://github.com/pypa/pip/blob/develop/pip/req/req_set.py#L204-L457 that is how pip discovers things

[02:21:55] <dstufft> yup

[02:21:56] <dstufft> you found it

[02:22:10] <lifeless> ok, its iterative, easy.

[02:22:58] <lifeless> hmm

[02:23:06] <lifeless> buy me alcohol and I might make this nice for you

[02:24:56] <dstufft> lifeless: I will buy you alcohol if you make this code not the worst ever

[02:25:19] <dstufft> I'm pretty sure there's a branch path in there that summons cthulu

[02:27:28] <lifeless> dstufft: handling setup-requires means putting req_to_install back onto reqs

[02:27:38] <lifeless> dstufft: is the download etc stuff fairly idempotent ?

[02:27:50] <lifeless> dstufft: or is it likely to fall over in a heap?

[02:28:53] <dstufft> I don't think I can honestly answer that question, I don't know how well it's going to handle that

[02:29:06] <lifeless> will find out

[02:29:21] <dstufft> you're probably going to have to do an install right then and there

[02:29:32] <dstufft> which might be an argument for installing into a temporary directory

[02:29:44] <dstufft> because we iterate over the entire dependency list before we install anything

[02:30:01] <lifeless> thats ok

[02:30:03] <dstufft> but we can't process something that has your new setup_requires prior to installing the setup_requires

[02:30:25] <lifeless> I think...

[02:34:34] <lifeless> so

[02:34:37] <lifeless> thinking out loud

[02:34:39] <lifeless> if its editable

[02:34:45] <lifeless> or if we unpack it

[02:34:48] <lifeless> we run egg_info

[02:34:56] <lifeless> what do we pull out from the egg info ?

[02:37:04] <dstufft> the install_requires

[02:37:32] <dstufft> pip fundamentally works by first building up a list of everything it needs to install, and then, once that list is built, installing everything

[02:37:48] <lifeless> yup

[02:37:51] <dstufft> we use egg_info to force the static metadata that setuptools writes into the egg_info directory to be rewritten

[02:37:57] <lifeless> so we're going to have to do that multiple times here

[02:38:01] <lifeless> or

[02:38:14] <lifeless> we're going to have to insist that both the setup-requires and install-requires are present in setup.cfg

[02:38:23] <lifeless> do you have a preference?

[02:39:18] <dstufft> one of the reasons pip did that is because part of the problem with easy_install was that if you did ``easy_install foo`` it would start installing things before it had the entire dependency list, which meant if the install failed half way through you could easily end up in a half installed situation

[02:39:33] <lifeless> yeah

[02:39:44] <lifeless> so I'd want to preserve that property

[02:39:56] <lifeless> here's how I think you could do multiple passes and preserve it

[02:40:23] <lifeless> while collecting, if we can't run one or more egg_infos, we identify their needs

[02:40:54] <lifeless> then we recurse to do those

[02:41:08] <lifeless> any single thing should be either fully installed or not (moduleo ctrl-C)

[02:41:20] <dstufft> lifeless: I think either it's going to need to install the new setup_requires into a temporary location (at least for executing egg_info itself) or it's going to need to insist that you specify the install_requires too. In both situations if we fail at any time during collection we've not installed anything new (or uninstalled anything)

[02:41:50] <dstufft> ontop of that you can potentionally add the setup_requires stuff into the global list of things to be installed

[02:42:01] <dstufft> so that they get installed into the non temporary location

[02:42:12] <lifeless> I think insisting both sets of reqs is probably cleanest

[02:42:23] <lifeless> we'll add setup_requires to be installed

[02:42:40] <lifeless> and have to take care to order the install

[02:43:08] <dstufft> but I think it's important that we do not install or uninstall anything from the target environment until we've collected the entire list of dependencies to install

[02:43:15] <lifeless> with you

[02:43:19] <dstufft> cool

[02:44:30] <lifeless> is line 447 where the install_requirements are added ?

[02:46:12] <dstufft> 447 is where it gets added to the list (actually a dictionary of package name: specifiers IIRC) of things to install, also 445 does it in another branch

[02:46:24] <dstufft> 444 is where it gets added to the list of things to process for further dependencies

[02:48:51] <lifeless> ah, \.req is what I needed

[02:49:26] <lifeless> no

[02:59:29] <lifeless> dstufft: ok, can I phone a friend... what attribute on InstallRequirement has the install deps ?

[03:00:29] <dstufft> lifeless: sec

[03:02:35] <dstufft> lifeless: I think it's part of egg_info_data

[03:03:09] <lifeless> so add_requirement adds things to self.requirements

[03:03:34] <lifeless> thats called twice

[03:03:41] <dstufft> ah

[03:03:41] <dstufft> wait

[03:03:45] <lifeless> once for dist.requires(available_requested)

[03:03:48] <dstufft> yea

[03:03:51] <dstufft> that's where the dependencies are from

[03:03:53] <lifeless> and once for the thing itself

[03:03:53] <dstufft> dist.requires

[03:04:23] <dstufft> that's going to be from setuptools

[03:04:26] <lifeless> yeah

[03:04:34] <lifeless> so thats what we're going to short-circuit

[03:07:54] <lifeless> ah but adding requires-dist implies extras being added too

[03:08:07] <lifeless> I can't find an example of that via simple google

[03:08:39] <dstufft> that's what the argument being passed to dist.requires() is

[03:08:54] <lifeless> I mean the schema

[03:08:56] <lifeless> for the cfg

[03:09:09] <lifeless> we're going to be not running egg_info at all

[03:09:14] <dstufft> https://github.com/pypa/pip/blob/develop/pip/_vendor/pkg_resources/__init__.py#L2544-L2556

[03:09:20] <dstufft> ah

[03:09:25] <dstufft> there's an environment marker for that

[03:09:48] <dstufft> I think

[03:10:15] <dstufft> hm

[03:10:16] <dstufft> maybe not

[03:10:20] <dstufft> I must be remembering wrong

[03:11:26] <dstufft> ah

[03:11:34] <dstufft> Wheel uses a non standard environment marker for it

[03:11:42] <dstufft> Requires-Dist: pyOpenSSL; extra == 'security'

[03:12:30] <lifeless> so

[03:14:23] <lifeless> is that honoured by tooling ?

[03:14:37] <lifeless> should we drop the extras thing from pep-426

[03:15:26] <dstufft> it's honored by pip for Wheels version 1.0

[03:16:16] <dstufft> I don't think we should remove the better format for them from PEP 426

[03:16:45] <lifeless> distutils2 doesn't do extras at all in its metadata AFAICT

[03:17:05] <dstufft> yea I don't think it does

[03:17:16] <dstufft> another option is something like

[03:17:19] <lifeless> so metadata-2.0 assumes a json file

[03:17:27] <dstufft> [extras]

[03:17:32] <dstufft> security =

[03:17:37] <lifeless> I think thats what it will need to be

[03:17:40] <dstufft> pyOpenSSL

[03:17:41] <dstufft> etc

[03:18:22] <lifeless> right, I need to pick up C, will come back to this tomorrow, the inevitable scope creep seems tolerable so far

[03:18:44] <dstufft> cool

[03:18:47] <dstufft> have fun!

[03:19:06] <lifeless> how would you feel about a hack-n-refactor patch to req_set.py as part of this

[03:19:23] <lifeless> since if X elif Y elif Z just cries out for an abstraction to me

[03:20:02] <dstufft> refactors are good, if possible trying to split them into their own patches is better. This code is not super well tested so there's an ihnereient risk to that sadly :/

[03:20:07] <dstufft> but yea it's real bad and needs a refactor

[03:20:08] <lifeless> yeah

[03:20:38] <dstufft> even just some comments as you're figuring out what things do with a fresh-ish set of eyes would be a pretty big imporvement

[04:44:03] <lifeless> dstufft: one more q

[04:44:07] <lifeless> dstufft: the self requirement

[04:44:21] <lifeless> with version == req_to_install.req

[04:44:40] <lifeless> dstufft: that won't be possible until we've egg_info'd, which we can't in this model

[04:44:59] <lifeless> dstufft: (unless we do the temp environment thing, which might make it actually needed)

[12:01:00] <Wooble> sounds like a GPL violation :P

[12:11:31] <dstufft> heh

[12:11:41] <dstufft> A different GNU interestingly enough

[12:15:48] <Wooble> oh, nice. For some reason I didn't memorize the implementation details of the clacks when I read that book years ago :).

[12:16:48] <Wooble> (although I'm reasonably confident that Pratchett was referencing that GNU by choosing those codes :) )

[22:55:53] <lifeless> dstufft: so this nearly works, but hangs :(. And fails to import pip._vendor.pkg_resources.tests.__builtin__

[22:56:12] <lifeless> tox -e py27 -e py34 -- -k test_install_local_setup_requires -k test_install_local_editable_setup_requires -d --tx 8*ssh=homegwy --tx 8*ssh=a.b.c.d --tx 2*ssh=localhost -vs

[22:56:36] <dstufft> lifeless: your change fails to import that, or just the test suite in general does?

[22:56:46] <lifeless> just the test suite

[22:56:55] <lifeless> spent a few minutes poking at making xdist work

[22:57:10] <lifeless> oh, interesting, dropping to one -e gets it to not hand

[22:57:11] <lifeless> hang

[22:57:15] <lifeless> now to figure out the import issue

[22:57:22] <dstufft> we do some terrible magic in pip/_vendor/__init__.py

[22:57:36] <dstufft> it's probably related to that

[22:59:30] <lifeless> ooh nice

[22:59:35] <lifeless> someone using meta_path

[23:01:14] <dstufft> yea, that's our hack to make it possible for distros to unbundle our bundled things without having to go through and change all the imports

[23:01:36] <dstufft> requests has the same thing now (taken from us, though I think sigmavirus24_awa has modified that and idk if that's been synced back to pip yet or not)

[23:01:43] <lifeless> ok, well I probably need to ping hpk and whinge

[23:01:51] <lifeless> since this seems entirely opaque and undiagnosable

[23:02:08] <dstufft> the failing to import __butilin__ might be similar to an issue that sigmavirus24_awa fixed in requests metapath hackery if that hasn't been brought back over to pip yet

[23:02:35] <lifeless> yeah

[23:02:37] <lifeless> I'm now getting

[23:02:38] <lifeless> ConftestImportFailure: (local('/home/robertc/pyexecnetcache/tests/conftest.py'), (<type 'exceptions.ImportError'>, ImportError('No module named scripttest',), <traceback object at 0x7f24a0538e18>))

[23:02:44] <lifeless> which is pytest innards blowing up

[23:03:48] <dstufft> I don't use pytest-xdist to do anything but run the tests in multiple processes

[23:04:02] <dstufft> so I'm not the greatest help here :(

[23:04:19] <dstufft> I normally don't bother running the full test suite locally, I just run the tests I'm adding and let travis run the rest

[23:05:13] <lifeless> hmm, can yield return on 2.6 ?

[23:05:13] <lifeless> does pip support 2.5

[23:06:01] <dstufft> we don't support 2.5

[23:06:05] <dstufft> yield can't return in 2.x at all

[23:06:21] <dstufft> pip is 2.6+ and 3.2+

[23:06:38] <dstufft> probably once PyPy3 supports 3.3+ we'll drop 3.2

[23:07:01] <lifeless> yield can return in 2.7 for sure

[23:07:04] <lifeless> 2.6 is the question

[23:08:06] <lifeless> yes, 2.6 works

[23:09:49] <dstufft> lifeless: https://bpaste.net/show/941af3b4290b ?

[23:13:02] <lifeless> thats different

[23:13:23] <dstufft> maybe I misunderstand what you're askingt hen

[23:13:28] <lifeless> https://bpaste.net/show/519484529288

[23:13:37] <dstufft> ohh

[23:13:37] <dstufft> that

[23:13:40] <dstufft> yea sorry

[23:13:41] <lifeless> I'm probably not going to use it

[23:13:47] <lifeless> but yeah, that

[23:13:58] <lifeless> was considering it for the growing-of-reqs

[23:19:10] <lifeless> does pip have travisci setup ?

[23:19:13] <lifeless> or similar?

[23:19:30] <dstufft> yea

[23:19:31] <dstufft> travis CI

[23:19:41] <lifeless> ok, I'm going to be naughtyish :)

[23:19:48] <dstufft> lifeless: you can open a WIP PR or whatever

[23:19:50] <dstufft> we do it all the time

[23:19:52] <lifeless> yeah

[23:20:01] <dstufft> running our tests locally sucks

[23:21:18] <Wooble> You can have travis test a fork just fine without making a PR...

[23:22:33] <dstufft> that requires setting the fork up in travis, which is less fun than just making a PR ;)

[23:22:42] <lifeless> https://github.com/pypa/pip/pull/2570

[23:25:23] <Wooble> dstufft: it's literally flipping a switch in the UI. *shrug*

[23:25:36] <lifeless> Wooble: which UI? And can non-owners do it?

[23:25:44] <dstufft> in the travis-ci UI

[23:25:48] <dstufft> you can turn on travis for your fork

[23:25:52] <dstufft> instead of making a PR

[23:26:12] <lifeless> meh

[23:26:17] <lifeless> I like you knowing what I'm hacking on

[23:26:31] <dstufft> doing it has some problems with notifications which we've worked around on pip, but PRs are nice anyways since you can review the code before it's "done" easier

[23:32:14] <Wooble> (I mostly prefer it because I seem to mess up every PR I ever try to create :) )

[23:42:03] <lifeless> dstufft: so, just to get a taste - have a look at the pr diff

[23:43:45] <lifeless> dstufft: its a big diff because of the indent shift :(

[23:44:08] <lifeless> dstufft: but there's only one change in the indented code - accumulate to more_reqs and return that

[23:46:01] <lifeless> dstufft: also I'm going to keep a bunch of small commits - so if the little commits look good, I can split out targeted PR's for just the ones that are ready

[23:46:16] <lifeless> dstufft: e.g. if you like this unification of the iteration, I'll push that to a dedicated branch+PR now

[23:46:39] <lifeless> dstufft: I'd like to keep the aggregate pending work small

Log file Viewer

Help | Karma | Search:

#pypa-dev logs for Wednesday the 18th of March, 2015