Log file Viewer
#pypa-dev logs for Thursday the 14th of April, 2016
[04:01:47] <StevenK> If the kids spent an hour jumping around, wouldn't the sugar from the lollipop get burned pretty much immediately?
[04:04:14] <StevenK> lifeless: Anyway, the test case that fails is we install a wheel, assert it created some files, and then install the same wheel with [extra]
[04:07:22] <StevenK> What I think is happening is during the second call is we find the existing install, and don't inject the [extra] into WorkingSet.resolve(), but everytime I try and think through it my mental model collapses
[04:28:33] <StevenK> lifeless: Yeah, the problem is by 640 I'm confused by state since the function begins back at 430
[04:32:22] <lifeless> vcs update or unpack archive is where assessing whats already installed happens
[04:33:58] <lifeless> pdb: what you need to do is get rid of the buffering of the subprocess by the test suite
[04:34:47] <lifeless> to poke at that with pdb from inside the test suite, you need to have your stdout wired to its stdin etc
[04:35:24] <StevenK> rpdb doesn't work, since it's running that pip inside a venv which doesn't have it available
[04:37:50] <StevenK> I tried to dig into what makes that venv, got confused, gave up, switched to printf debugging via with open(file, 'a')
[05:00:01] <lifeless> StevenK: things to check a) that we hit the extras line under 'if not ignore_dependencies'
[05:01:52] <lifeless> StevenK: d) check you don't see 'does not provide the extra' in the output - if you do, thats a strong indicator that you're failing to parse the installed metadata correctly
[05:03:26] <StevenK> My line numbers are now off, but that's just before 648, probably 643 or so for you
[05:08:46] <StevenK> lifeless: Ah ha, that dist.requires call at the bottom of that function will hit WorkingSet
[05:10:22] <lifeless> StevenK: oh! this is probably just another callsite that similarly needs to loop over the given extras rather than evaluating without supplying the extra
[05:12:09] <StevenK> It looks like we work out the depends inside pip, and then pass *that* to resolve(), not including the thing that requires them
[05:49:49] <StevenK> Yeah, I'm right, check_if_exists() calls into get_distribution() which will call WorkingSet.resolve() without the information that it was required by requires_simple_extra[extra]
[21:01:06] <dstufft> lifeless: ah, I sorted it. Ubuntu 16.04 has setuptools 20.3 which is one of the partially broken releases. Was trying to see what exactly was broken and what patches were needed, but I think doko is just going to upload 20.7 to 16.04 (At least, he asked me to file a FFe to that affect)
[21:02:12] <lifeless> there may be more damage to sort - see the discussion w/stevenk yesterday here
[21:09:51] <dstufft> well, if there are more fixes I'll figure out how to convince Ubuntu to FFe those when the time comes :]
[21:27:49] <lifeless> dstufft: when he updates pkg_resources vendor copy in pip, dist.extras is throwing - hits requires, hits resolve, attempts to evaluate an extra marker without the extra set.
[21:28:26] <lifeless> dstufft: That resolve code is what we fixed in .7, so its either another implementation of resolve (possible -e.g. wheel) or its omfgweird
[21:28:47] <lifeless> dstufft: (or perhaps not using the latest pkg_resources to do the update - I dunno, and have'nt synced with StevenK since chatting yesterday
[22:46:26] <StevenK> lifeless: I think it's a bug in setuptools: http://paste.openstack.org/show/494138/
[22:49:20] <StevenK> lifeless: I was thinking about this last night (actually while trying to fall asleep, sigh), and I think resolve() needs to grow a evaluate kwarg that defaults to True, and that callsite sets it to False.
[22:53:40] <njs> dstufft: lifeless: I wonder if we should try to convince doko to enable ~/.local/bin in 16.04?
[22:57:15] <StevenK> lifeless: The second test case I thought about was get_distribution("foobar;python_version<'2'"), which I suspect will throw ValueError
[23:01:36] <njs> lifeless: the problems with ~/bin are: (a) pip doesn't use it currently, (b) everyone who does use it uses it to drop random personal scripts, so starting to automatically drop our junk there on top of their junk could cause problems, (c) for users who don't already know about ~/bin, pip probably shouldn't be creating user-visible directories with confusing names in ~, (d) that 'if' is brutal, because even if we get over all of the above issues and decide
[23:01:36] <njs> to start dropping pip-installed scripts in ~/bin, it means that for most users, the first time they do pip install, they then immediately need to re-log before they can run it.
[23:01:58] <lifeless> pjdelport: I'm supremely uninterested in that debate; we're dealing with a deployed base of millions - we need to figure out what will get out to those users most efficiently and adapt what we do to that need.
[23:02:41] <pjdelport> lifeless: Petitioning for the standardisation of ~/.local/bin is the best way to do that; ~/.local is already being standardised on for so many other things.
[23:02:42] <njs> it's (d) that's kinda the clincher for me -- even if pip does start dropping stuff into ~/bin, the user-experience is still awful until Debian/Ubuntu fix their stuff
[23:02:55] <pjdelport> And a bunch of distributions have already started making it part of the default PATH
[23:02:59] <njs> and if they're going to fix their stuff anyway then they should fix it to point to ~/.local/bin
[23:04:02] <pjdelport> The other thing is that it shouldn't be set by a shell hack, but properly, as part of the system.
[23:04:07] <lifeless> njs: so ~/.local/bin would be added the same way - and have the same (d) behaviour
[23:04:24] <lifeless> pjdelport: uhm, that part of the system *is* shell - its not a hack in any sense
[23:04:25] <njs> nah, for ~/.local/bin you can make the hack be: if [ ! -f ... ]; then mkdir -p ~/.local/bin; fi
[23:04:41] <pjdelport> Otherwise you get the situation where non-interactive things that don't start the shell or source that script don't get the PATH entry set.
[23:04:58] <lifeless> njs: mmmm, I'll let the skel maintainers argue that one, I don't have the context to channel them
[23:05:10] <pjdelport> lifeless: What I mean is it should happen on the PAM or whatever level, where the rest of the default environment is set. That all happens before the login shell is invoked.
[23:05:37] <pjdelport> For example, I have PATH DEFAULT=${HOME}/.local/bin:${PATH} in my ~/.pam_environment
[23:05:53] <njs> pjdelport: haha if only it were that simple. the login shell on these systems by default throws away any PATH stuff that you set in PAM
[23:06:54] <njs> I agree that in general a larger cleanup of this machinery would obviously be a good thing, but I'm suggesting that for right now we might want to think about whether we can convince the next Ubuntu LTS to at least handle things as well as the Fedora/RHEL ecosystem do
[23:07:00] <lifeless> StevenK: I'm not sure that > pkg_resources.get_distribution("foo;extra=='extra'") - makes sense though
[23:07:24] <lifeless> StevenK: since you wouldn't want to loop over get_distribution for foo[bar,baz]
[23:07:24] <pjdelport> njs: Right; I'd love to be able to remove the above from my standard system setup. :)
[23:10:53] <njs> so maybe we should try to convince Ubuntu to add that to the default PAM settings, I don't care
[23:11:19] <njs> I'm just wondering if there's any chance to sneak through a 90% solution before the next LTS ships
[23:11:20] <pjdelport> njs: Anyway, the PAM route is necessary for the PATH setting to work reliably, at least in my experience. Setting it at the shell level invariably fails for certain invocations that don't go through the relevant shell magic.
[23:11:48] <njs> pjdelport: yeah, I've run into the same thing -- I have a .env that gets sourced from both my .zsh stuff and my .gnomerc, to keep them in sync
[23:13:38] <pjdelport> So I try to advocate that whenever I can, since the .pam_environment hook is sadly not well known / documented.
[23:14:39] <njs> oh but don't worry -- there's a bug about this filed on base-files. it's been open since 2000.
[23:18:28] <lifeless> StevenK: so I'm guessing the call to get_distribution from pip was for requires_simple_extra[extra] ?
[23:20:21] <dstufft> I would very much like it if Debian and friends added ~/.local/bin, but my ability to poltics for that isn't super high
[23:20:23] <lifeless> StevenK: but in the absence of any requested extras, you have to evaluate with {'extra':''}
[23:26:53] <dstufft> I do think that it's not reasonable to change the PEP from ~/.local/bin to ~/bin though.
[23:27:16] <dstufft> This is made more :sadface: by the fact that I'm pretty sure Ubuntu has pip patched to default to --user
[23:35:32] <njs> I guess my question is, does anyone have pull with doko? he's also the maintainer for Debian's /etc/skel so :-)
[23:36:18] <njs> I filed a bug ont he debian side, but given that my only interactions with him have been him shouting at me about manylinux I'm not sure how helpful a direct appeal would be :-)