Log file Viewer
#pypa-dev logs for Tuesday the 24th of February, 2015
[10:35:29] <xafer> pf_moore, regarding your comment on https://github.com/pypa/pip/issues/1891, I've tried to extract the script generation logic from distlib into: https://github.com/xavfernandez/scriptlib/blob/master/scriptlib/__init__.py
[10:36:47] <xafer> still work in progress, but I was wondering if those 100s line codes shouldnt directly be part of pip
[10:48:40] <pf_moore> xafer: possibly, if we could remove the distlib dependency as a result. But 2 problems.
[10:50:26] <pf_moore> Second issue is the exe wrappers, which are C code from (IIRC) a separate project of Vinay's.
[10:50:47] <pf_moore> We'd still either have a dependency on that, or we'd end up supporting C code.
[10:51:27] <pf_moore> It would be nice if cpython core included an "official" exe wrapper implementation, then people could stop writing their own :-)
[10:51:57] <pf_moore> The launcher has the capability for that IIRC but we don't ship a built wrapper.
[10:52:51] <xafer> and actually took them: https://github.com/xavfernandez/scriptlib/tree/master/scriptlib/launchers with a link to vinay's simple_launcher project
[10:55:16] <pf_moore> simple_launcher, that's the one. As it's not a PyPI project, we can't simply vendor it like we do with distlib. So meh, not a simple issue
[10:56:14] <pf_moore> In principle, I don't think we use distlib enough to warrant vendoring, but I do thing the distlib exe wrappers are way better than the setuptools ones, so we're stuck at the moment.
[10:58:04] <pf_moore> And BTW, the launcher code is subtle to get right (the py.exe launcher uses Windows job objects to make sure things get shut down in groups correctly, I don't really understand the details) so it's not something I'd want to treat lightly...
[10:59:41] <xafer> well why not just embed the 4 executables and take simple_launcher under the pypa umbrella ?
[11:12:20] <pf_moore> Essentially because it's not clear who'd end up maintaining simple_launcher. Vinay does at the moment, but AFAICT mostly based on distlib bug reports.
[11:13:43] <pf_moore> The long and short if it is, at the moment with an issue like this we can say "raise a distlib bug" if that's the problem, and Vinay will pick it up.
[11:14:24] <pf_moore> If we have our own script wrapper code, we have to deal with these bugs ourselves.
[11:15:30] <pf_moore> If we take the exes from simple_launcher, how do we get the right version? That project doesn't seem to do formal releases
[11:16:00] <pf_moore> And if we nick the exes from distlib, Vinay may feel justified in saying doing that's not supported.
[11:53:58] <pjdelport> I'm playing around with wheel building, and i set the following in my ~/.pip/pip.conf: [wheel] wheel-dir = ~/.cache/pip-wheelhouse
[11:54:30] <pjdelport> With the idea that i can just say "pip wheel Pillow" to build wheels into that directory.
[11:56:08] <pjdelport> That seems to work perfectly fine, but in *addition* to creating that directory and storing the wheel there, pip also seems to be creating literal subdirectories of the working directory: ./~/.cache/pip-wheelhouse/
[11:57:15] <pjdelport> Is this a bug, or is ~ syntax not supposed to be supported in that pip.conf setting, or what?
[12:06:13] <pf_moore> pjdelport: it's a bug. Could you raise an issue on the pip tracker? I see the problem and I'll get a fix sorted now.
[12:15:11] <dstufft> pf_moore: xafer I do think we should drop distlib at some point, but I agree with paul that we have to wait to drop it until we have a satisfactory answer for what to do about the script entry points (and that Python should really just ship a script wrapper we can use)
[12:15:29] <dstufft> I don't do Windows though so idk if py.exe is usable for that (and if it is, how we'd use it)
[12:15:45] <pf_moore> pjdelport: Ta. http://git.io/AMaY will fix it once I stop crying about flake8 :-(
[12:16:21] <pf_moore> dstufft: AFAIK, py.exe needs a recompile with different options to do "wrapper mode".
[12:16:58] <dstufft> I'm probably going to cut a pip release shortly to pull in the new requests update since it does some security related things
[12:19:11] <pf_moore> dstufft: the new flake8 warning W503 is a PITA - I don't intend to fix them right now, am I ok to either commit with flake8 errors, or add a flake8 setting to suppress the damn thing for now?
[12:20:09] <pf_moore> Or https://github.com/pypa/pip/pull/2436 (pin the flake8 version) may be a better solution
[12:25:32] <pf_moore> dstufft: lol I beat you, I have an ignore in my PR and going through travis now.
[12:27:00] <pf_moore> Yay commit clashes. Looks like I got the flake8 config syntax wrong, so how about I back my tweak out, commit the actual fix, and then you do your update on that?
[12:31:09] <dstufft> pf_moore: btw, and windows related things on develop that should be pulled into a 6.0.9?
[12:32:04] <pf_moore> mph, not that I know of but let me check... (is there a git command for "log what's on develop since whatever ends up in 6.0.9"?)
[12:36:06] <pf_moore> fatal: ambiguous argument '6.0.8..develop': unknown revision or path not in the working tree
[12:40:37] <pf_moore> dstufft: PR #2397 (retry rmtree) and the win32_hidecursor ones should probably go in. What's the criterion for going into 6.0.9? Bug fixes? The wheel-dir normalisation one should probably go in as well, but it's not Windows-specific...
[12:41:06] <dstufft> pf_moore: yea, generally stuff that fixes bugs and are unlucky to break somehting
[12:45:41] <dstufft> pf_moore: you said that py.exe requires a recompile with different options to do "wrapper mode", do you know if there is anything that simple_launcher does that a py.exe in that mode doesn't do? I'd personally be OK with pip shipping a compiled py.exe with those correct compile flags
[13:01:06] <xafer> I'm tempted to change https://github.com/pypa/pip/blob/develop/pip/req/req_install.py#L578 into an "if"
[13:03:29] <dstufft> that doesn't sound like a bad thing to do, but it sounds like more than what we should do in a micro
[13:03:49] <xafer> maybe I should put back the same uninstall order as before https://github.com/pypa/pip/commit/c509c5c4aba060a64754c90e1687dd265f58ef77
[13:11:48] <pf_moore> dstufft: sorry went for lunch. py.exe and simple_launcher come from 2 different bases, and do different things. I'd need to check.
[13:14:37] <pf_moore> I don't think there's a way round us shipping some sort of exe, but I'd prefer the C source to be part of the source of the same project that has the exe.
[13:15:32] <pf_moore> We don't want pip to need a C compiler to build so we should vendor *something*
[13:16:04] <pf_moore> My issue with distlib is that it doesn't include the C source in its sdist, but rather contains an externally built binary blob.
[13:16:42] <pf_moore> Also, I'd like it if pip and setuptools used the same wrapper. The differences are annoying
[13:20:42] <pf_moore> dstufft: I'm sort of thinking my next project should be a "pywrapper" project that basically takes the script maker bit of distlib and isolates it, and integrating that into both pip and setuptools. Not sure how jaraco would feel about replacing the existing setuptools exe wrappers, though.
[13:21:32] <dstufft> pf_moore: I thought about that, I also thought about trying to write a PEP to standardize the idea of entry points, specifically the console entry points and then just adding it to the packaging lib
[13:24:01] <pf_moore> dstufft: yeah, that's another option, and possibly better (although wrappers have benefits outside of entry points, for example tack an exe wrapper on the front of a pyz archive and you have a standalone executable that only needs an installation of Python to run).
[13:25:37] <pf_moore> See, for another example of stuff like this, https://github.com/pfmoore/shimmy
[13:27:09] <dstufft> pf_moore: yea that's a good point as well, and it'd be possible to design the API to support that, we just can't add the general concept of script wrappers to packaging without being backed by a PEP. I'm happy with either option with a slight preference to putting it in packaging, but I recognize that PEPs are not often the funnest things to deal with :)
[13:29:04] <pf_moore> I'm all for a PEP (although I agree on the "not fun" aspect). I think too many people have invented the same wheel over the years, and we need to pick an implementation and stick with it.
[13:45:02] <xafer> dstufft, ok I changed the order for a reason: when the install is done via an egg-link, the distribution still gets an egg_info and is now detected via the "egg_info_exists and dist.egg_info.endswith('.egg-info')" case
[13:55:12] <xafer> a dirty way to detect this would be the difference between "classic" egg-info: "FSPkg-0.1.dev0-py2.7.egg-info" and the much simpler egg-info: "FSPkg.egg-info" in the case of egg-link
[15:56:02] <msabramo> I get it a lot in my browser the last week and just now Travis failed for one of my PRs because a 500 installing a test package from Bitbucket
[15:58:29] <pjdelport> msabramo: Apparently there's some system upgrading in progress: http://status.bitbucket.org/
[15:58:55] <msabramo> pjdelport: Ah, not just me. Good to know. For a while I thought I might've been a problem with my laptop because my networking was acting fiddly yesterday but I guess they are unrelated.
[15:59:09] <msabramo> I guess it doesn't really make sense that a problem on my end would cause a 500
[15:59:42] <pf_moore> pjdelport: you're just causing trouble now with directories in ~, aren't you? :-) (https://github.com/pypa/pip/issues/2453)
[16:00:28] <pf_moore> I'll have a look at that one, but it might be harder to address, as find-links takes a number of different formats...
[16:00:37] <msabramo> between bitbucket 500ing and periodic pep8 upgrades, it is getting a bit trying to contribute to pip unfortunately
[16:01:45] <pjdelport> pf_moore: Not on purpose! :) I just started poking at using wheels like this today.
[16:02:21] <pjdelport> Normally i would just move and and write it down to "Python package tooling is annoying and crappy", but i figure i don't get to complain about that unless i at least report the pain points i run into.
[16:07:40] <pf_moore> I'd be sad if you thought pip was crappy... :-( The bug reports are much appreciated, and I think what you're doing should work. Just might be a bit harder to fix this one.
[16:08:25] <pf_moore> (The problem is that find-links may be a URL, so it's not obvious where we'd do ~ normalisation)
[16:09:46] <sigmavirus24> The last known "stable" release afaiac was 1.5.7, 1.6.x has had thoroughly broken config issues
[16:10:24] <pjdelport> pf_moore: Oh, i don't think pip is crappy. It's the vanguard against crappiness, on the whole.
[16:12:43] <pf_moore> pjdelport: hmm, there's a really ugly hack I could easily do (if find-links starts with ~, normalise it) but I'm not sure how vile that is. I'm a Windows user myself, and don't typically put ~ in configs, so my instincts aren't good here...
[16:14:20] <pjdelport> pf_moore: How exactly does find-links detect when something is a path, if it isn't a file://-scheme URL?
[16:15:07] <pf_moore> pjdelport: os.path.exists. So there's a chicken and egg problem. We don't know to ~-expand unless we know it's a file, which we can't tell unless we ~-expand...
[16:15:50] <pjdelport> pf_moore: Erk, does that mean if you happen to have a local file path with the same name as a URL, pip will think the URL is that path?
[16:17:14] <pf_moore> Well, that's not possible on Windows (because : is a special character), so I don't know for sure, but maybe yes. Who the heck would have a directory called "http:" though?
[16:17:19] <sigmavirus24> msabramo: if you want to help move the 2.4.0 release along for flake8 feel free to look at the last blocking PR on gitlab ;)
[16:17:33] <sigmavirus24> msabramo: alternatively if you want to fix the config file problems in pep8 that'd be awesome =P
[16:18:08] <pf_moore> dstufft: any advice on this? Is it reasonable to special-case find-links values starting with ~, and try normalize_path on them to see if they are a real directory?
[16:19:04] <pf_moore> Or are we going to be endlessly chasing corner cases and we'd be better saying that we don't support ~ in filenames? (Which seems like giving up, but nevertheless...)
[16:19:52] <pf_moore> To put it another way, can ~ appear anywhere *other* than at the start of a filename in Unix, and be an abbreviation for a home directory?
[16:23:36] <pjdelport> pf_moore: My intuition tells me that stuff appearing in pip.conf should have a layer of "shell-like" expansion performed on them that doesn't also happen on the command line.
[16:24:15] <pjdelport> When you say "pip install --find-links ~/foo/..." in the shell, you expect the shell to be doing the expansion, not pip.
[16:24:59] <pjdelport> But when you use paths like ~/foo in pip.conf, you expect them to be expanded in a general way, before the expanded paths get consumed by the same places in the code.
[16:25:24] <pf_moore> pjdelport: Well, I have no intuition here as Windows doesn't do any of that shell expansion stuff, so *my* instincts say that pip should expand anything it wants expanding. Which essentially says I agree with you, but for different reasons.
[16:28:06] <pf_moore> The problem with find-links is that it's underspecified in that regard. But I'm not disagreeing, I think it should expand, just not sure when or how.
[16:37:25] <pjdelport> The more i think about it, the more my intuition tells me that the layer of shell-like expansion should happen during pip.conf parsing, before stuff gets passed further.
[16:37:47] <pjdelport> Rather than each setting consumer trying to do its own expansion independently later.
[16:39:10] <pjdelport> The latter seems like a guaranteed way to breed bugs and weird behavior down the line. Even if you disregard code complexity, you'll now have to deal with problems like double-expansion for values that come from the shell (already expanded) instead of pip.conf.
[16:42:44] <tomprince> Certainly it is somewhat confusing, as ~ works some places in pip.conf and not others.
[16:43:56] <pf_moore> Just uploaded https://github.com/pypa/pip/pull/2455 which is a pretty hacky fix for this. I can't test myself as I'm not on Unix, but let's see what Travis says. Then we can debate whether it's too ugly to live :-)