Log file Viewer
#pypa-dev logs for Wednesday the 6th of May, 2015
[02:48:15] <tchaypo> trying to run pip’s tests on my own machine, I’m finding that it assumes a few things
[02:48:40] <tchaypo> eg, some of the functional tests assume it has subversion executables, other tests assume hg executables
[02:53:21] <tchaypo> I can’t see anything that installing these in travis, so I’m assuming the travis environment must just happen to come with those things installed
[06:46:20] <tchaypo> http://docs.travis-ci.com/user/ci-environment/#Environment-common-to-all-VM-images
[06:47:20] <tchaypo> subversion and mercurial were two things I had to install on my machine; the third was bzr, which I don’t see listed there or in the travis setup stuff
[07:48:35] <ronny> lifeless: btw, did you notice the talk about using unpacked wheels like zip-unsafe eggs yesterday?
[07:53:29] <ronny> lifeless: i see, i have a rough plan to make setuptools/pip support unpacked wheel as if they where unpacked eggs, then remove the need for eggs in setup_requires/install_requires when wheels are availiable
[08:00:24] <ronny> lifeless: thats why i thought it wuld be a good idea to talk a bit, in part since you seem to have actual work time on the topic
[08:00:49] <lifeless> ronny: so pr 2617 is my current WIP on the cache - heading up the resolver path t the moment
[08:01:23] <lifeless> the resolver needs a cache because it's going to be examining multiple versions for dep data before it can decide which one to choose and that needs to be cheap
[08:05:54] <lifeless> this means that in pip install scenarios, you end up building the setup_requires many times, once per package that has it
[08:06:42] <lifeless> so if using it with pip install -e . -- aka setup.py develop - it will grab the deps more often
[08:07:19] <lifeless> since we won't need to do C stuff, just unpack a wheel from the users ~/.cache
[08:09:24] <ronny> lifeless: i also plan to teach setuptools itself to treat wheels as zip-unsafe eggs
[08:09:56] <lifeless> ronny: something I think would be good is a way to request setup_requires statically from setuptools without invoking setup.py
[08:10:36] <ronny> lifeless: there also is a plan for invoking setuptools without implicit easy_install
[08:11:13] <ronny> lifeless: i made some issues to setuptools recently dealing with no implicit easy_install and no implicit setup_requires
[08:14:40] <ronny> lifeless: we should get together with jason and donal and figure a common scheme for caching downloaded python packages, (in particular to avoid re-downloads when one deals with a devpi based pypi mirror)
[08:16:09] <lifeless> ronny: I'd really like to just deprecate easy-install in setuptools, so that it can get out of the business of installing other packages
[08:17:03] <ronny> lifeless: hmm, true, i suppose once setuptools can list setup-requires and avoid implicit easy_install its doable
[08:22:39] <ronny> lifeless: btw, there was support for static setup_requires in sdists for a release, but pkg_resources is broken in some situations wrt distribute
[08:31:38] <ronny> lifeless: i still think opt out of easy_install + list via an option is a better way than setup.cfg additions that require a extra tool to function
[08:32:30] <lifeless> ronny: setup.cfg addresses 'cannot import setup.py because of missing requires' - it lets you write plain old python without the crazy gymnastics
[08:32:55] <lifeless> ronny: opting out of easy install and returning a list provides a contract that can work for existing setup.py using setup_requires
[08:38:39] <lifeless> ronny: lots of the packages I looked at choose to write regular python rather than gymnastics python
[08:39:25] <lifeless> ronny: so, I agree that its broken in the sense that 'python setup.py ...' fails for them in a clean venv with nothing installed
[08:39:53] <lifeless> ronny: but I don't agree that they are broken in principle: we've not delivered a good environment for them
[08:46:53] <ronny> lifeless: well, setuptools is quiirky - a setup.py erroring out instead of running is broken
[08:47:10] <ronny> lifeless: in particular all those packages that import not yet installed dependencies
[08:47:49] <ronny> many of them would be instantly fixed by just parsing the version from foopkg.version instead of importing
[08:51:16] <lifeless> ronny: but you have to use gymnastics to defer actual importing, which is fixable by setup.cfg, and thats why a bunch of folk are interested in setup.cfg
[08:51:18] <ronny> lifeless: they dont quite work if the code in question imports before even calling setup
[08:52:33] <ronny> lifeless: im just against tools that are already a mess getting even more messy to work around people messing things up
[08:53:28] <lifeless> ronny: they're messy because they attempted to be too clever; I want to offer people a way to avoid that
[08:55:31] <ronny> lifeless: you added another layer of abstraction, instead of enabling the tools to opt out of cleverness
[08:57:13] <ronny> lifeless: the static setup requires in setup.py adds a new layer of absttraction in different tools while not even remotely repairing the original problem or providing ways to better work with it
[08:57:53] <ronny> suddenly pip needs to know about extra file format,s easy_install needs a extra thing to deal with them, and the base situation of tricky gymnastic for the details is unchanged
[08:58:40] <lifeless> easy_install doesn't need to deal with them (because by this point pip will be able to own that)
[08:59:05] <lifeless> the tricky gynamnastics can be somewhat unwound, and at least the potential for clearer things is enabled
[09:04:20] <ronny> i think we are missing out on details, we should talk about it at a different point in time, but from my pov you are solving the problem at the wrong position thus adding technical debt and format cuft that will have to stay around for decades
[09:05:24] <tomprince> Clearly, adding an ad-hoc thing is something that will need to be maintained long-term.
[09:05:56] <tomprince> But I think that supporting static metadata of that kind is something that people will want to support in the sdist 2.0 world (whatever format that ends up taking).
[09:06:37] <tomprince> So the additional code will just be to handle the reading the ad-hoc data and converting it internally to sdist 2.0.
[09:08:58] <tomprince> `python setup.py` not working without dependencies isn't clearly broken. It breaks some peoples expectations (obviously, from this conversation). But it is the same as './configure; make; make install' not working without dependencies.
[09:09:34] <ronny> tomprince: configure tells you whats missing, setup.py just raises a confusing exception
[09:09:40] <tomprince> And it is a small fairly straightforward incremntal improvement that can be implemented well now.
[09:10:44] <tomprince> And the point of adding support for setup_requires in setup.cfg is so nobody need trip over that, or controt themselves to avoid that.
[09:11:50] <tomprince> https://github.com/pyca/cryptography/blob/master/setup.py#L146-L250 is clearly a code smell, and people shouldn't be forced to do something like that just to work-around setuptools.
[09:12:38] <tomprince> And, in any case, the ecosystem appears to be moving (deliberately) to a world where nobody ever runs 'setup.py' by hand anyway.
[09:13:08] <ronny> tomprince: yes, but adding workarounds on top of workarounds instead of making the workarounds unnecessary seems to be the easy way out that just adds technical debt
[09:14:36] <ronny> the last item of that kind was distribute, and that is creating such a mess that we still cant have setup_requires in the sdist metadata
[09:14:39] <tomprince> Well, this is partly a work-around, and partly a step in the right direction of making all the metadata declarative.
[09:15:00] <lifeless> and since pip has basic requirements that need to be present in all the proposed designs
[09:15:21] <lifeless> (resolver, setup_requires semantic handling independent of build system && query method)
[09:15:50] <lifeless> I don' tthink there's a lot to be gained by worrying about stuff much beyond that horizon, unless we think we can eliminate pip needing those things
[09:16:47] <tomprince> And, in any case, the bulk of the work is getting pip to handle setup_requires, however it gets that data from the sdist, whether that is by piling hack on top of setuptools or by reading a bit of static metadata.
[09:21:23] <tomprince> ronny: In any case, it seems you are arguing against adding "hacks" in pip by adding "hacks" in setuptools.
[09:22:46] <ronny> tomprince: to give pip full controll, setuptools needs an complete easy_install opt out anyway
[09:27:08] <ronny> tomprince: my plan is to propperly teach setuptools to deal with setup_requires printing, and having computed arguments for setup
[12:57:28] <dstufft> I don't feel too bad about adding a setup.cfg "hack" that we can eventually isolate to the "here's the legacy setuptools hack builder thing for the old and busted way" when we implement the ability to have different build tools.
[19:30:16] <dstufft> lifeless: hey! So trying to grok RequirementCache - Is this just a mapping of reuqirements (ignoring for a moment that it's a little more complicated than just a straight mapping) to a build location so that we reuse a build location if we've already downloaded + unpacked a thing?
[19:33:20] <lifeless> both setup-requires and resolver workloads require the ability to reuse a not-yet-installed package
[19:33:39] <lifeless> the former case is for packages with the same setup-requires as install-requires
[19:34:32] <lifeless> given A and A' that both require B, if we have to backtrack from the A to the A' branch, we don't want to re-process B
[19:35:16] <dstufft> lifeless: right, but REquirementsCache here doesn't itself actually do anything but map requirements to build dirs right?
[19:35:49] <lifeless> dstufft: I think it will also end up owning the list of available versions of a package
[19:37:59] <lifeless> one of the confounded bits I haven't really tackled yet is that req_to_install is both a concrete version of a package
[19:39:04] <lifeless> but anyhow, one class for constraints, one for 'things that may or may not satisfy'
[19:39:35] <lifeless> there's a lot of hidden interactions so its taking me a while to context switch in each time I pick up the work :(
[19:47:27] <lifeless> dstufft: :). C is up, I willping you when I get back from dropping her at kindy
[19:47:30] <dstufft> lifeless: so right now it looks OK, I'm a little concerned that the RequirementsCache might be better off as just a function (or a method since it needs to hold some state for the actual physical disk location of the tmp dir) that can turn a requirement into a build path, but I see you have two lookup functions but aren't using them that I can see yet, so I may just not be seeing future plans
[20:01:12] <dstufft> agronholm: I have it installed and working yes, I didn't get the tests running yet since there's a bit of a chicken/egg problem with tox and virtualenv and I got distracted by filling out paperwork
[22:32:42] <dstufft> the rhyme and reason behind what is and isn't indented is before my time, I don't think anyone is going to hardline any one stance, as long as the output makes sense
[22:33:54] <lifeless> and we can get rid of all the special handling of named/unnamed in the inner loop
[22:35:23] <dstufft> we can't remove them because there is a use case for them where we don't have the replacement done yet
[22:37:14] <dstufft> "I'm a company with internal projects and I'd rather not have to setup an index and develop tooling to create artifacts for all our internal projects and i'd rather just have them all install from their git sources"
[22:38:51] <dstufft> the replacement is being able to essentially do ``install_requires=["internalthing @ git+https://github.com/dstufft/internalhing.git"]``.. synax might be wrong on that
[22:39:11] <dstufft> so you can't use it on things you release to PyPI, so it'd only be for internal
[22:41:57] <tomprince> You need somewhere to host that index, which you don't need, if each package can have that information in it.
[22:43:16] <lifeless> tomprince: I can totally get the overhead of maintaining pacakge building etc stuff, but a single simple name:value index can be hosted on a github page, for instance
[22:44:49] <dstufft> lifeless: so you can do that, but it's more work for people who want to do this workflow. It's essentially just letting people work "index-less" by encoding direct references directly in their dependency metadata, a lot of companies are using this ability internally. I generally agree it's not super hard, but I don't believe supporting it is super hard to do (you just essentially skip the index step for that one) and it helps out some class
[22:45:32] <lifeless> what I want is to be in a position where we're referencing stuff by name always
[22:46:49] <dstufft> nothing has implemented direct references yet so if we need to change them, it shouldn't be very hard to do so
[23:27:14] <dstufft> it's really a container type for multiple Specifier instances, where a Specifier is like >=1.0
[23:38:25] <dstufft> lifeless: not currently, in the future we have wildcards (not wildcards like *foo, but just "*" for, all extras) and we have exclusion like A[foo,-bar] where (currently in PEP 426 at least) the exclusion operator says "do not install this extra at all, even if someone else asks for it"
[23:39:05] <dstufft> lifeless: I should note, this is all in PEP 426 which hasn't been accepted or tested for actual sensibility yet