Log file Viewer

[00:00:35] <qwcode> "unit" for pip is not using scripttest

[05:29:11] <StevenK> lifeless: Continuing from -infra, you'll rescind your comment on Nakato's PR?

[05:29:29] <StevenK> Since it obviously requires a follow-up

[05:30:18] <lifeless> StevenK: Its not fundamentally my call (I'm not a committer on pip) but I think its worth seeing if the fix can be made a bit more general first

[05:31:26] <StevenK> It looks like Nakato's fix is only for extras

[05:32:31] <StevenK> But I've only been staring at this code for an hour, there could be more going on

[05:32:39] <lifeless> yes - your test is testing extras too

[05:34:25] <StevenK> lifeless: It doesn't even reach that code, though?

[05:34:56] <StevenK> lifeless: The exception is on line 240, the extra's union is on line 266

[05:35:04] <lifeless> so Nakato's patch handles the case where the first refence to the thing

[05:35:11] <lifeless> after the constraints file

[05:35:13] <lifeless> has the extras

[05:35:33] <lifeless> it won't handle the more general case - so you need the same basic code

[05:35:40] <lifeless> e.g. update the extras on the install_req

[05:36:12] <lifeless> but it needs to rescan the thing rather than erroring - if and only if the specifier has one more or extras that weren't present already

[05:38:25] <StevenK> I think I get it. If existing_req and install_req are the same, we can append to .extras on existing_req and then return it to be rescanned?

[05:39:26] <lifeless> yeah

[05:39:58] <lifeless> it may need some care reading through the rest of that nested if mess (sorry, I tried, but its way better in my next mega branch)

[05:40:02] <lifeless> but thats about it

[05:40:10] <lifeless> we still want that error

[05:40:16] <lifeless> if the extras were the same

[05:40:37] <StevenK> Yup

[05:40:52] <StevenK> Well, we could just deal if we're going to union them

[05:41:08] <lifeless> so

[05:41:24] <lifeless> there's this bug where pip doesn't honour <= / >= etc when the same package is specified twice

[05:41:44] <lifeless> the exception is there for the case where the user directly specified the thing

[05:42:00] <lifeless> to stop users reporting the fact that pip doesn't honour those rules

[05:42:50] <lifeless> I'm proposing that we teach pip how to handle a subset of the cases it fails and remove the exception there

[05:43:19] <lifeless> dstufft might object at the end of the day, in which case both nakato's work and yours will need to be chained after my resolver work, and we'll re-org the trello stuff

[05:43:43] <StevenK> Also, WTB a faster test suite

[05:43:44] <lifeless> so foo[] + foo[] should still error

[05:44:00] <lifeless> and thats a special case of foo[bar] + foo[bar]

[05:44:12] <lifeless> which is a special case of foo[bar,quux] + foo[bar]

[05:44:47] <lifeless> StevenK: yeah, I have a plan for the test suite. https://twitter.com/rbtcollins/status/656163846093049856

[05:47:49] <StevenK> tests/functional/test_install_reqs.py ....FF. Bleh

[08:07:50] <xafer> lifeless: thanks for the review, https://github.com/pypa/pip/pull/3196 was updated

[13:20:23] <xafer> Not sure what to do with https://github.com/pypa/pip/issues/3059 ? it seems related to https://github.com/pypa/pip/issues/2319

[21:18:04] <ErikRose> dstufft: qwcode: You guys are killing me with this cache stuff. :-/

[21:18:59] <qwcode> ErikRose, apologies : )

[21:19:09] <ErikRose> I've been trying all day to come up with a way to have security and every last little drop of build speed, but I don't think physics works that way.

[21:19:37] <ErikRose> Remember, it's only one little tiny part of caching we have to disable: the caching of locally built wheels made out of sdists.

[21:19:40] <qwcode> yea, I feel bad like we backed you into a corner

[21:19:51] <ErikRose> Everything else—the http cache, the caching of index-fetched wheels—stays on.

[21:20:26] <ErikRose> I'm mostly concerned that its going to blow up into an "all of #1175" type of thing, and I'm running out of time to work on this.

[21:20:51] <qwcode> the problem is once turn off the cache by default, then turning it back on later will break peoples requirements files.... so it's a commitment

[21:21:18] <ErikRose> hence my --yeah-yeah-turn-on-the-wheel-cache-even-though-I'm-hashing flag

[21:21:53] <ErikRose> Or if somebody implements something like a "pip lock" down the road, they can assume that behavior implicitly.

[21:22:33] <ErikRose> Now, I did have one crazy idea just before I came in here with my tongue hanging out. :-)

[21:22:52] <ErikRose> Let me start explaining it and see if it runs into a contradiction.

[21:23:22] <ErikRose> The trouble is that pip hash and pip install spit out only the sdist hashes, not the wheel ones.

[21:23:35] <ErikRose> Let's assume for the moment that we make wheels build deterministically.

[21:23:56] <qwcode> ErikRose, well, actually, I'm still in the middle of some work stuff, so not sure I can give you 100% over IRC.... although I'd be happy to respond later if it was in the issue

[21:24:03] <ErikRose> We could rig pip hash and pip install's error messages to not only spit out the sdist hashes but to also build the sdists into wheels and spit those out as well.

[21:24:12] <ErikRose> qwcode: I'm really rubberducking. :-) But feel free to feed back.

[21:24:20] <qwcode> ok,

[21:24:58] <ErikRose> That's a bit of work—I don't know how much, since it might violate some assumptions I made in pip.utils.hashes.

[21:25:22] <qwcode> but I thought you had a problem with immediately buiding wheels before validating

[21:25:25] <ErikRose> But it could work. You'd put both sets of hashes in your file, and whammo.

[21:25:38] <ErikRose> Ah, you're right!

[21:25:42] <ErikRose> Thanks for reminding me!

[21:25:51] <ErikRose> That moots it before I even got to my other concern,

[21:26:31] <ErikRose> which was that we'd have to come up with a way to get "pip install" not to sidestep the sdist altogether and happily go with a cached locally built wheel, spitting out only the wheel hash.

[21:26:42] <ErikRose> And that would probably involve some degree of surprise to the user.

[21:26:58] <ErikRose> Good. You've saved me writing up that bad idea. :-)

[21:43:23] <qwcode> ErikRose, I guess I don't really understand the security concern with building the wheel. under the current plan, you just spit out the hash, and users just paste into their requirements file... and off pip goes to run setup.py

[21:48:12] <ErikRose> qwcode: That's the "repeatability" part, but in peep's docs I recommend an interstitial step: actually examining the sdist to vet the package.

[21:48:35] <ErikRose> So no, I don't recommend just blithely pasting hashes in.

[21:48:41] <ErikRose> bbiab

[21:48:43] <qwcode> ErikRose, vet how?

[21:48:57] <ErikRose> By reading the code. There's no other way I'm aware of.

[21:49:25] <ErikRose> Someday we might have a distributed cache of known-decent hashes or something, but that's later.