Log file Viewer
#pypa-dev logs for Thursday the 19th of March, 2015
[00:33:45] <dstufft> trying to figure out the best way for me to review this refactor, this code is fiddly and the (reasonable) indentation change makes it slightly harder, maybe I'll fake a commit locally that reindents it with a no-op loop
[00:52:34] <dstufft> lifeless: Ok, so reviewing locally, the changes you've got so far look good to me
[01:26:35] <dstufft> lifeless: ok, I merged it locally, just waiting on tests to pass before I push it up
[01:45:48] <lifeless> e.g. locate_files looks to me like it should be doing 'if self.update or self.force_reinstall
[01:46:48] <lifeless> right now as I read it pip install --no-download --force-reinstall foo bar baz
[01:47:52] <lifeless> Requirement already satisfied (use --upgrade to upgrade): linecache2 in ./.virtualenvs/test/lib/python2.6/site-packages
[01:48:21] <lifeless> Requirement already satisfied (use --upgrade to upgrade): linecache2 in ./.virtualenvs/test/lib/python2.6/site-packages
[01:49:05] <dstufft> what you're looking at is a recent change which was designed to make it so pip install foo==1.0 didn't hit PyPI if foo 1.0 was already installed
[01:49:39] <dstufft> it sounds like maybe there's still edge cases that we need to work out with that PR
[01:51:47] <dstufft> -U doesn't imply --force-reinstall, -U on it's own won't uninstall + install something, it'll see the version we've discovered is already installed and skip it
[01:53:09] <lifeless> and that from a code perspective, a check for -U will always pass where a check for force-reinstall might also be done, pragmatically speaking
[01:58:38] <dstufft> the workflow for --no-download was: "first do pip install --no-install ..." which will download the thing, and compile it in a staticly named temporary directory, then copy that temp dir onto your prod machines and do ``pip install --no-download ...`` which will "finish" the install
[01:58:49] <dstufft> it was basically a terrible hack that only exists because we didn't have a binary format
[02:03:36] <dstufft> so we don't forget to delete it when we kill the options since that's it's only purpose
[02:26:25] <lifeless> dstufft: things guarded by is_download can become unconditional in 7 too, right?
[02:27:42] <dstufft> which isn't deprecated, that's just the "I want to download the files without installing it" case
[02:39:18] <sigmavirus24> dstufft: we were having problems where people were using requests in `python setup.py test` what was happening was that our modifications to sys.meta_path were lingering even though the module (requests and the vendored packages) had been gc'd
[02:39:33] <sigmavirus24> That tends to come back as an attribute error though in so much as NoneType not having an attribute "X"
[02:39:52] <sigmavirus24> Not being able to import some module sounds like the fix that I made to both pip and requests which is a bit more complicated to explain
[02:40:37] <sigmavirus24> What was happening was that a deferred import was happening on py2.x and was being executed as an implicit relative import so people were seeing errors as "failed to import requests.packages.chardet.sys"
[02:41:22] <sigmavirus24> That's fixed in both pip and requests in so much as the VendorAlias now properly knows what modules have been vendored (explicitly) and knows how to say "the import you're looking for needs to be provided by something else"
[02:41:24] <lifeless> sigmavirus24: if you want to poke around, I can give you reproduction instructions
[02:41:41] <sigmavirus24> lifeless: sure. I just read far enough back to get a gist of the conversation
[02:42:32] <sigmavirus24> one of my coworkers was complaining about something in neutron tonight and I kept itching to pull out my laptop at the bar and look into it
[02:42:51] <sigmavirus24> dstufft: this is why I work so passionately on urllib3 and maintain chardet
[02:44:12] <sigmavirus24> lifeless: while we're here, question for you. Does testr always import all of the discoverable test modules even if you specify a specific one?
[02:45:55] <lifeless> sigmavirus24: testr doesn't import anything. "python -m subunit.run discover ... --load-list X" does unit test upstream discovery, and then filters the result down by the list (set intersection)
[02:46:05] <lifeless> sigmavirus24: so yes, if you have many thousands of packages, this can be slow
[02:47:04] <sigmavirus24> A difference between python setup.py testr {args} (roughly) and python -m testtools.run {args}
[02:47:18] <sigmavirus24> so the graduation to oslo.log made debugging into tests in glance impossible for reasons
[02:47:33] <sigmavirus24> mostly it's glance's fault, but testr was working fine and testtools.run was failing hard
[02:55:12] <lifeless> dstufft: is_wheel && link.scheme in vcs.all_schemes is a logical impossibility, right ?
[03:08:43] <lifeless> its experimental - I haven't run everything locally, but hopefully better version pushed
[03:12:55] <lifeless> dstufft: any objection to calling assert_source_matches_version when doing --download
[03:43:24] <lifeless> _________________ ERROR at setup of test_install_upgrade_editable_depending_on_other_editable __________________
[21:07:50] <qwcode> lifeless, is there a master plan where you're headed? some feature you're trying to add?
[21:10:14] <lifeless> qwcode: I started poking directly at that, but in discussion here with dstufft realised that fixing the awfulness of the code first was a good idea
[21:12:14] <qwcode> lifeless, for the abstraction classes, you said "may grow, or be subsumed into someting else in time".... in your time in working on this feature you mean?
[21:13:22] <qwcode> lifeless, so you're done with those classes right now? just wanted just enough to clear up that routine?
[21:18:44] <qwcode> everything you're doing looks sharp, just trying to understand if you had master domain model in mind yet. sounds like right now it's more based on what's required to clean up the routine
[21:20:50] <dstufft> yea, I think long term we should look real close at these and figure out real good models
[21:21:07] <dstufft> but I don't think that we need to wait off for perfect to get improvements in code that we can easil change later
[21:24:03] <qwcode> roger that. I'll just keep watching the PRs. ditto on someone documenting a long term model at some point at least...
[21:25:50] <lifeless> we have a awkward thing where we track with unrelated variables the result from finder.find_requirement
[21:26:15] <lifeless> qwcode: thats the patch I want to be able to write, which this work is cleanup to prepare for
[21:27:27] <qwcode> lifeless, got it. that helps. that would be nice. do you expect to have to add a flag to setuptools to *not* double do what pip is taking over, or would it all work it due to setuptools detecting it's already done?
[21:28:40] <qwcode> lifeless, maybe I'll just wait for the feature... I won't get ahead of myself here...
[21:28:42] <dstufft> I'd have to think about memoizing find_requirements, off top of my head it sounds like a reasonable thing to do with just a lru cache or something
[21:28:43] <lifeless> if you hate the idea of easy_install, you don't, and your setup.py can avoid all the shenanigans needed to work
[21:30:11] <dstufft> I think it's a reasonable thing, I just don't know offhand if we ever expect find_requirement to return a different result, I can't imagine a sitaution where we do though
[21:31:49] <lifeless> dstufft: so, I propose to move the not_found caching logic into find_requirement fo rnow
[21:32:01] <lifeless> and we can decorator-enable that as a future step when we need it in more than one place
[21:36:53] <qwcode> dstufft, indirectly related, can the PyPI cdn give different results right after uploads... https://github.com/pypa/pip/issues/2408
[21:38:58] <qwcode> dstufft, I mean back and forth from the old and new on successive installs, until the upload stabilizes across the CDN
[21:40:14] <qwcode> "purge doesn't happen in lock step across the entire cluster", ok then that probably explains what he saw, then?
[21:41:10] <qwcode> he apparently uploaded a series of *many* versions... and then saw "random" versions installed
[21:42:00] <qwcode> any way, just remembered that, and wanted to know.... not that it means we shouldn't cache find_requirement results
[22:04:10] <dstufft> the HTTP access will be cached (assuming the correct headers) but we still execute a bunch of machinaery that we don't need to (including parsing HTML)
[22:27:16] <qwcode> seems like for pip to take over setup_requires, it will end up having to pass some kind of "--no-setup-requires" to "python setup.py" (assuming it's using setuptools to parse setup_requires)
[22:33:31] <qwcode> lifeless, oh..... so this is not a plan to let current projects work better... you'd need to configure in setup.cfg?
[22:34:55] <lifeless> qwcode: making existing projects Just Work better is possibly doable in a couple of evil ways
[22:35:21] <lifeless> qwcode: we could compile but not run and attempt a constant fold back from the setup() call in the bytecode.
[22:35:54] <lifeless> qwcode: or we could run it with import knobbled and magic mock objects everywhere and setup stubbed out and use that to figure out the setup_requires
[22:37:04] <lifeless> qwcode: but the basic problem is that executable code has no business installing its own requirements
[22:37:06] <dstufft> I'm still on the fence if the general idea is a good idea for a project to use it, but since it's opt in and when we switch to metabuild system we can just tie it as another legacy quirk of the setuptools builder I don't feel too bad about having the feature
[22:37:09] <qwcode> or add a new command or flag to setuptools, and have pip run setup.py 3 times, instead of just 2... the extra run to get the setup_requires information, but without running the logic to process setup_requires
[22:37:45] <lifeless> qwcode: the canonical example would be this - setuptools is a setup_requires for setuptools.
[22:39:38] <dstufft> On the other wise there's always some setup_requires dependency you can't really express
[22:40:17] <qwcode> 1) "python setup.py get_setup_requires", 2) pip intalls the setup_requires deps 3) python setup.py egg_info, 4) python setup.py install
[22:41:12] <dstufft> qwcode: you're thinking similar to what I thought, that's the best way if people want to maintain compatability with easy_install and ``setup.py build`` with a clean environment
[22:41:35] <dstufft> apparently people don't care so much about maintaing compat with those things, but they care a lot that having to defer imports in the setup.py makes it harder to write setup.py's
[22:41:56] <qwcode> dstufft yes, I recall you referring to some "3 time execution plan"... maybe in the thread the other day
[22:42:43] <dstufft> (realistcally easy_install could read the setup.cfg setup_requires too, in which case you'd only be breaking the case where you want to execute setup.py directly
[22:43:52] <qwcode> lifeless, why is setup.py not importable? because it's not written such that imports are nested or delayed you mean?
[22:45:16] <dstufft> in a way it's somewhat of an extension of how I've told people to think about setup.py's for awhile now: setup.py isn't for you to execute, it's for the packaging tools
[22:51:59] <lifeless> so my take here is I think very straight forward: I want pip to be able to figure out what to install without executing any in-tree code. Thats derived from a) don't install /anything/ until all deps are known, b) don't waste a bunch of cycles installing temp environments left right and centre, c) setuptools knows about /building/, pip knows about /installing/
[23:02:59] <qwcode> I agree with a/b/c, at least eventually. also agree that it's mind boggling that setup_requires was a setup arg in the first place. setup.cfg would have made more sense. But just trying to think about existing projects too. maybe do both solutions... something for old and new
[23:04:08] <qwcode> PEP426 intends to cover new projects... but I understand people don't want to wait
[23:05:49] <dstufft> PEP 426 handles it on the installer side, because it eliminates the need for setup.py all together when you're installing a package (much like wheel does)
[23:16:45] <lifeless> also as ncoghlan noted, I'm not aiming at ideal here: just *something* thats unambiguously better, which we can hide under an abstraction later if we don't like it. And folk can opt-in.
[23:24:50] <qwcode> if the plan turns out with pip reading setup_requires from setup.cfg, would be nice to have setuptools support it too...
[23:25:33] <lifeless> qwcode: and as a future iteration that makes a bunch of sense, for install-require and extras.
[23:25:47] <lifeless> qwcode: I don't think it makes any sense for setup-requires, since we don't /want/ setuptools doing setup-requires :)
[23:29:26] <lifeless> qwcode: OTOH many many many other people don't want setup.py installing stuff ever :).
[23:35:42] <qwcode> lifeless, when doing "pip wheel", wheel deps may be found that should prevent building them... so they get downloaded into the wheelhouse. although I'm not clear atm, why download_dir wasn't reused
[23:37:14] <lifeless> I think I could possibly come up with a cleaner way to represent it with a bit of effort, not sure if its worth it
[23:39:06] <lifeless> but you can assess yourself when i push, which I will once dstufft has reviewed the current passing PR
[23:40:25] <qwcode> do you want to chain PRs with different branchs, and just say "depends on PRXXX" vs trying to do all of this in develop?
[23:53:36] <lifeless> dstufft: didn't have one at hand, and having one would need twice the testing - not worth it until we have another call site
[23:55:32] <lifeless> (twice the testing - test the decorator, and test that its used on find_requirements)
[23:56:34] <dstufft> tbh find_requirements should probably return a sentinal value instead of raise exceptions, but that's probably a hard change since I think we use find_requirements all over