Log file Viewer
#pypa-dev logs for Sunday the 4th of January, 2015
[05:23:12] <dstufft> jaraco: there was a reddit thread that made me think about it, is there any interest at all in moving setuptools away from the installer bits (easy_install in particular)? I'm not saying we should, but if there's some interest I might try and work out all the niggly bits sometime
[05:25:07] <jaraco> I think that's the plan ultimately, but there are several aspects which are difficult. For example, the installer logic is necessary for 'develop' and 'test' (and plugins like pytest-runner).
[05:25:50] <jaraco> I still use easy_install because it has support for single-file, human-readable console scripts.
[05:26:45] <jaraco> The namespace package support in pip (issue #3) is also a bit of a blocker in my opinion.
[05:27:18] <dstufft> jaraco: so by "figuring out the niggly bits" I more or less mean figure out what parts of setuptools rely on that stuff so that we can at least get a list somewhere which is the first step to figuring out answers to those things, I just didn't want to waste my time if your opinion was that easy_install should stick around forever
[05:27:45] <jaraco> I don't have good answers for those things, but I believe there is a roadmap that does eventually replace these aspects of setuptools with other tools.
[05:28:15] <dstufft> (slightly related, what do you mean by "single-file, human-readable console scripts"?)
[05:28:57] <jaraco> I mean naturally-executed scripts, much like you get on Unix. Paul and I had a discussion on it in pypa-dev.
[05:29:53] <jaraco> And everyone else is happy with dual-file or opaque executable files, so maybe I should just be too.
[05:29:58] <dstufft> I was under the impression that PATHEXT wasn't globally respected in the Windows APIs
[05:31:19] <dstufft> how do you cope with that? do you just put up with the fact it doesn't work? or do you have something to work around it?
[05:31:25] <jaraco> It could also be honored by Python's Popen, which would fill 99% of the use cases that are currently problemmatic with using that model broadly.
[05:32:05] <jaraco> In my usage, it doesn't matter, because I don't expect to Popen my scripts... I always invoke them in the shell.
[05:32:45] <jaraco> It does occassionally bite me when a process tries to invoke Popen(['pip']) or Popen(['easy_install']) because that fails in my env.
[05:34:33] <jaraco> It's really the namespaced packages support that's the most problemmatic with the pip model. I also do appreciate the egg model (that you can have a single file installation). I think even wheels have to be expanded to be installed/imported.
[05:34:58] <dstufft> I'm not specifically opposed to an option to install scripts like that, other than I have a blanket opposition to options which are not broadly applicable and I don't know enough about Windows to know if it is :D, anyways, that's a bit of a tanget. It sounds like maybe sitting down and trying to enumerate the things blocking the ultimate removal of installer features from setuptools to at least document it would be a useful thing
[05:37:01] <dstufft> I didn't want to come across like you should totally rm -rf easy_install right this moment, I just wanted to see if that was a direction you wanted to go in at all before I bothered to spend any time on it
[05:38:57] <dstufft> jaraco: regarding single file installs: Wheels do currently "require" it as in, the PEP mentions that you can do it, but it's discouraged and it's not going to work generically because of C compiled things
[05:39:32] <dstufft> the PEP says you can do it, and we use that in ensurepip and virtualenv and get-pip.py actually
[05:42:00] <tomprince> Even if easy_install wants to stay around, it could be spun out to a seperate package.
[05:43:04] <dstufft> I'm on the fence about the general feasability of such a thing, alot of projects simply don't work with it (though I believe setuptools detects it? or attempts to anyways) and more than one API in the stdlib requires file paths not in memory things which leads to terrible things like -> https://github.com/pypa/pip/blob/develop/tasks/generate.py#L165-L169 just to deal with it
[05:44:13] <dstufft> tomprince: yea that's possibly any option too, and it may make sense to do that as part of the migration process too in a "sorry we removed it from setuptools but you can get it back if you do X andw e're going to maintain that for an extra whatever" way
[05:48:40] <jaraco> dstufft: That does seem slightly terrible. Doesn't pkg_resources.resource_filename address that issue in a less terrible way?
[05:52:18] <dstufft> jaraco: probably yea, get-pip.py itself can't use pkg_resources so it has to do it manually (and of course using pkg_resources itself ends up making a runtime dependency on setuptools if this was needed for something other than a bootstrap script)
[15:46:27] <sigmavirus24> So I'm really curious about a good way of implementing https://github.com/pypa/twine/pull/84
[15:53:09] <dstufft> sigmavirus24: I'm like half way convinced that register is the wrong abstraction anyways
[15:53:28] <dstufft> I think it's a pointless leftover from the time before uploading was implemented on PyPI
[15:55:22] <sigmavirus24> dstufft: true. But register is also useful for updating stuff like the long_description
[15:56:08] <dstufft> sigmavirus24: I wonder if it makes more sense to just have a dedicated command to upload a new long description
[15:56:56] <Ivo> I personally like the idea of claiming the name for a project from only uploading a release for it
[15:58:54] <dstufft> in the common case of "I want to claim a name" you typically register a version 0 and end up deleteing it from the webuio
[16:00:59] <dstufft> I don't particularly see a whole lot of value in making it easier for people to claim names for which they don't have a project to use with
[16:02:14] <Ivo> well as long as we have a free-for-all package index theres probably going to be some sort of squatting to some degree
[16:03:29] <dstufft> almost 100% of the time my use of setup.py register is ``twine upload -s dist/*`` -> gets an error -> gets mad and registers once
[16:27:25] <agronholm> there should be a filter to prevent empty projects from being shown in search results by default
[16:31:52] <sigmavirus24> dstufft: do we still want twine to upload docs? (https://github.com/pypa/twine/issues/12) ?
[16:34:12] <dstufft> it's a maybe! I'm not opposed to adding it, but I don't think it's super important either
[16:34:50] <sigmavirus24> So my reason for wanting to work on a register was to take care of https://github.com/pypa/twine/issues/21 with a suggestion of using twine register to ensure the package was created first
[16:36:41] <dstufft> sigmavirus24: if we were really smart, we'd pick a Wheel to fetch the data from in order to auto register
[16:37:16] <sigmavirus24> So my first approach was going to be: try upload, if that fails, register then retry
[16:37:59] <dstufft> sigmavirus24: that's doable too, could just sort the list of things to upload so wheels come first in that istuation to get the benefit of better metadata
[16:48:48] <dstufft> ionelmc: re-opened it, that's about all I have the mental capacity to do right now
[16:51:57] <dstufft> https://github.com/pypa/virtualenv/compare/1.11.6...12.0.5#diff-15 shows the diff of 1.11.6 and 12.0.5, I don't think anything in there should have changed sys.path handling
[17:03:00] <dstufft> because I think the way -p works is completely insane and is never going to work in every situation
[17:03:31] <dstufft> I tried to fix the sys.path thing when openstack reported that the patched in one of the earlier 12's broke things for them
[17:04:29] <dstufft> ionelmc: my plan is to finish making it work, then probably posting to pypa-dev about it and probably sticking it on pypi under a virtualenv-refactor name or something
[17:06:04] <ionelmc> dunno what to do about that, virtualenv's site.py doesn't seem to have any api to get the path to the real interpreter :-|
[17:07:12] <dstufft> we can easily add stuff to the rewrite's site.py to do it, the question then becomes if we need to support new-virtualenv-in-legacy-virtualenv
[17:08:06] <dstufft> and my answer to that is probably something like "see if we can make it work, and if not then error out"
[17:09:18] <dstufft> another option (if can't make it work) is to support them both in parallel for awhile
[17:12:27] <ionelmc> cause we have sys.real_prefix, but we'd have to have version/implementation -specific sys.path handling
[17:16:45] <dstufft> ionelmc: looking at your PR right now, why'd you move the destination from create() to __init__
[17:19:02] <dstufft> ionelmc: Ah, well not american to any extent right? I've never seen an american add u's to things like flavour
[17:21:36] <dstufft> sigmavirus24: obviously you're a fake american and we should eject you immediately
[17:26:42] <pmxbot> jaraco pushed 2 commits to setuptools (https://bitbucket.org/pypa/setuptools/) :
[17:26:42] <pmxbot> Skip integration tests when one of the packages under test is already installed, as the installation will fail in that case.
[17:29:28] <dstufft> ionelmc: I'm gonna merge your virtualenv-rewrite PR, but I'm going to make some changes I think
[18:00:08] <dstufft> (to be fair, I doubt the rewrite works on pypy or jython or anything like that at all right now)
[18:01:33] <dstufft> I'll have to figure out how to add hacks back in for those platforms, but long term the way I want to support them is by having them backport the venv module
[18:31:52] <sigmavirus24> there are different commands I'd much rather see like pip open which opens an editor with the source of the package you're looking at (c.f., bundle open in rubby)
[18:34:33] <sigmavirus24> I'm a buzzkill though. I like my tools well defined, composable, and practical
[18:40:17] <tomprince> I think pip would need API for that to be useful, unless your extension could just as easily be a seperate program.
[18:43:54] <dstufft> something like ``pip home`` could probably just as easily be a seperate program
[18:44:53] <dstufft> I mean the idea is a quick and easy way to get to the project's homepage, if I have to install another thing to do that I'd probably just rather do ``open https://warehouse.python.org/project/<foo>/`` and click on it
[18:49:20] <sigmavirus24> dstufft: my measure of usefulness is typically "How many people are actually going to use this? How many people have needed it and not had it?"
[18:50:17] <dstufft> sigmavirus24: sure, I don't know if ``pip home`` is generally useful on it's own, just saying that "install a seperate command" for something like pip home drives it's useful down a lot, it might not have been very useful to begin with i dunno :)
[18:50:38] <dstufft> the last few sentences is about the most i've thought about the proposal so far
[18:54:36] <sigmavirus24> I also don't think "pip open" belongs in pip until it proves it's popularity
[18:55:04] <sigmavirus24> I also think most people wouldn't use it because I'm probably one of the few people who ever used bundle open in ruby
[18:58:03] <dstufft> the hardest part about me using pip open is retraining myself not to type subl ~v/l/p/s-p/<whatever>/
[18:59:54] <dstufft> sigmavirus24: extensions is probably dosable at some point, not sure it makes sense right now since pip doesn't have an API that we offer to people
[19:00:22] <dstufft> without an API that we offer to people, the biggest thing that an extension point gives you is the ability to be executed as ``pip foo`` isntead of ``pip-foo``
[19:02:24] <sigmavirus24> honestly, one of my favorite and least favorite things about ruby gems is that by structuring your project one way, you can totally extend something without entry points
[19:02:51] <sigmavirus24> want an ssh library? lib/net/ssh/ is all the structure you need for someone to do "require 'net/ssh'"
[19:04:21] <dstufft> sigmavirus24: isn't the flipside of that it's super hard to figure out if someone _is_ extending your project or where they are extending it at?
[19:04:59] <dstufft> xafer: I haven't yet worked up the will to think about eggs and try to figure out if it's going to break anything yet ;(
[19:09:24] <xafer> well on old system that installed with not pep440 version it will certainly break :o
[19:10:57] <xafer> and the lack of feedback info due to the absence of the "else" clause makes it harder to debug than it could
[19:26:14] <pmxbot> jaraco pushed 10 commits to setuptools (https://bitbucket.org/pypa/setuptools/) :
[19:26:14] <pmxbot> Add test capturing expectation when dependencies conflict during resolve. Fixes #281
[19:38:39] <pmxbot> jaraco pushed 3 commits to setuptools (https://bitbucket.org/pypa/setuptools/) :
[20:33:13] <ionelmc> dstufft: i think i'm gonna take a shot at the virtualenv-in-virtualenv thing again
[20:34:05] <ionelmc> dstufft: yeah i have added some basic test with scriptest, but you've deleted it :-)
[20:35:24] <dstufft> I need to figure out if there is a saner way to doing functional tests for this than scripttest
[20:38:26] <dstufft> ionelmc: well functional/integration and unit tests do different things, it'll have unit tests that cover everything I can do, but it more or less needs functional/integration tests either way, especially for the legacy isolation builder
[20:39:48] <ionelmc> but it needs coverage, otherwise you'll endup with similar issues as virtualenv where you don't know what other things a bugfix will break
[20:40:01] <dstufft> they are also basically "run the virtualenv and see if the command errors and then go YOLO good enough"
[20:41:30] <dstufft> I mean this is basically the entire test suite : https://github.com/pypa/virtualenv/blob/develop/tests/test_virtualenv.py
[20:42:00] <dstufft> https://github.com/pypa/virtualenv/blob/develop/tox.ini#L11 this is the best test
[20:42:45] <dstufft> probably like 95% of the time the way we notice something broke in virtualenv is because the pip test suite broke
[20:44:10] <dstufft> (like half of the pip test suite ae functional tests which create virtualenvs, and the pip develop branch uses the virtualenv develop branch)
[20:46:45] <willingc> dstufft, sigmavirus24, ionelmc: Happy New Year. Lots of good stuff that you are doing.
[20:51:59] <ionelmc> dstufft: does a jython or pypy virtualenv have different layouts for windows/nix?
[21:02:32] <ionelmc> dstufft: when would sys.prefix be different than sys.exec_prefix ? is there any distro that configures python to have them different?
[21:03:05] <dstufft> ionelmc: dunno if anyone sets them different, but sys.prefix is where pure python crap goes, and sys.exec_prefix is where compiled crap goes
[21:03:59] <ionelmc> dstufft: cause virtualenv(legacy) only sets sys.real_prefix (and not sys.real_exec_prefix)
[21:22:23] <ionelmc> dstufft: you seem to have used `import os.path` explicitly a lot - is there a reason?
[21:24:34] <dstufft> mostly because I think it's clearer, os.path is documented as a seperate module (even though os.py does hacks with sys.modules that makes it that way) so I like to use it that way
[21:29:27] <sigmavirus24> ionelmc: I do the same because it more clearly conveys what specifically I'm likely to be using
[21:31:26] <ionelmc> sigmavirus24, dstufft: for me it signals that os doesn't do anything in it's __init__ (well actually, it's not an package, but it's the same thing)
[21:31:54] <ionelmc> and that expectation is mercilessly broken buy the terrible import code living in os.py
[21:34:15] <ionelmc> put it in other words, `import foo.bar` means that you can use `bar` without `foo` - as `foo` is a namespace package
[21:35:11] <dstufft> I don't think that really holds, because ``foo`` can have all sorts of code (and often times does)
[21:36:30] <ionelmc> dstufft: foo.bar in the hypothetical scenario where foo/__init__.py is empty, of course
[21:37:29] <ionelmc> just saying, if you're doing that for newbies that might look at the code, it's more confusing
[21:38:47] <dstufft> I know when I first started doing python I was confused that I had to import foo.bar and I couldn't just import foo; foo.bar because os.path worked that way
[21:39:11] <ionelmc> dstufft: import os.path only works cause the evil sys.modules patching in os.py
[21:39:50] <dstufft> ionelmc: I don't really care how it's implemented, all of the documentation treats it like it's written as os/__init__.py and os/path.py
[21:40:59] <dstufft> and there's a good chance that it would have been written like that except changing os.py to os/__init__.py would require changing other hacks in the python interpreter
[21:45:40] <dstufft> because CPython uses os.py as a sentinel to determine where everything else should be
[22:10:36] <r1chardj0n3s> dstufft: pypi search is quite slow now - could be related to the postgres upgrade maybe?
[22:10:59] <dstufft> r1chardj0n3s: PyPI didn't get upgraded, that's still on it's own DB cluster not on the PSF cluster
[22:11:02] <r1chardj0n3s> dstufft: in brighter news, we have a green light for shutting down the file replacement loophole
[22:11:24] <dstufft> it's a TODO item for me and EWDurbin to get PyPI off of it's own DB cluster though
[22:12:11] <dstufft> I have the start of what will become the object store mostly done too. or rather I have the abstractions that will enable it
[22:31:55] <r1chardj0n3s> yes, warehouse does have faster search results. because it uses a different architecture. and isn't ready yet. "there's not a lot we can do immediately" ugh
[22:52:36] <dstufft> well you'd still have the main PR, (it'd have to be a different main PR though), since it could have a PR from the main repo branch to the develop branch
[23:03:58] <r1chardj0n3s> (I'm trying to decipher your "there is no "default" comparison operator" commen)
[23:04:31] <r1chardj0n3s> since my slides don't mention a "default" operator, but I do have "== Version matching clause" is that it?
[23:07:45] <dstufft> the PEP had some of those left over from before we removed that, I assume you copied it from there
[23:10:42] <r1chardj0n3s> Now I just have "~= 2.4.3 is equivalent to >= 2.4.3, == 2.4.*" which I believe is correct
[23:11:19] <dstufft> originally PEP 426 and PEP 440 together had the requirement syntax being "foo (~=2.4.3)"
[23:11:51] <dstufft> so we had the default specifier being "foo (2.4.3)" being the same as "foo (~=2.4.3)"
[23:12:16] <dstufft> but we got rid of that because it was incompat for little reason to what setuptools and pip expected, and harder to type on the command line, to just foo~=2.4.3
[23:40:23] <ionelmc> you'd think you'd make a flavor for pypy, but then you still need the windows/nix differentiation :-|