pmxbot IRC Log Viewer

[01:56:17] <ErikRose> I'm going to upgrade encountering unknown hash names to a fatal error unless someone can think of a reason not to.

[02:03:26] <dstufft> ErikRose: seems reasonable (also another reason to use --hash, because otherwise --sha789 just gets you a CLI flag not found error, better error messages ftw!

[02:03:55] <ErikRose> Yep. I'll probably make your --hash change as a last step, once I get this hairy stuff working.

[02:04:13] <ErikRose> Tracing these trees of ifs and external state is intense. :-)

[02:04:20] <dstufft> lol

[02:04:24] <dstufft> our code is so bad

[02:04:31] <ErikRose> It honestly could be a lot worse.

[02:04:44] <dstufft> it could be a lot better too

[02:04:47] <ErikRose> I had all these glorious dreams of leaving pip.download cleaner than I found it.

[02:05:01] <dstufft> I've had that feeling before

[02:05:08] <dstufft> it passes quickly

[02:05:24] <ErikRose> I thought, why not determine what kind of URL (or whatever) it is outside (file:// pointing to file, file:// pointing to dir, VCS, etc.)

[02:05:45] <ErikRose> embody each in a class, tag each class with metadata like "Do I support hashing?"

[02:05:50] <dstufft> I do plan on refactoring, but it's hard because a lot of the code is interdependent and not well tested :/

[02:06:08] <ErikRose> And then bammo, things are easier to understand and to extend

[02:06:41] <ErikRose> methods: .go_hash_yourself() and go_unpack_yourself()

[02:06:44] <ErikRose> or some such

[02:06:59] <ErikRose> But I found a shorter way that I'm actually likely to get done, so…later.

[02:07:16] <ErikRose> I do believe I'll get all the hashing moved to a single place, though, rather than scattered around as it is atm.

[02:07:35] <ErikRose> I have this sort of hash bucket that you update, and internally it updates a bunch of different hashes.

[02:07:56] <ErikRose> I believe I've got it handling both URL hashes (#md5=...) and flag-based ones.

[02:08:37] <ErikRose> There's some interesting reasoning that allows us to get away with that: https://pastebin.mozilla.org/8845955

[04:02:25] <lifeless> ErikRose: oh man, I hope your work doesn't conflict with my resolver branch :(

[04:02:45] <lifeless> ErikRose: because - I rearranged a /tonne/ of stuff there to break the assumption of packagename-is-unique

[04:03:42] <ErikRose> lifeless: Oh no, I hope not, too!

[04:04:37] <lifeless> https://github.com/rbtcollins/pip/tree/issue-988

[04:04:51] <lifeless> I might see about merging that up with head now actually, I've a few minutes slack

[04:06:43] <lifeless> oh yeah

[04:06:51] <lifeless> dstufft: in pip/req/req_install.py

[04:06:59] <ErikRose> req_install looks okay to me.

[04:07:04] <lifeless> dstufft: paths_to_remove.add - why do we add .pyc and not .pyo ?

[04:07:22] <dstufft> oversight probably

[04:07:26] <ErikRose> lifeless: We're going to collide in prepare_files a bit.

[04:07:58] <lifeless> dstufft: so in my resolver branch

[04:08:24] <lifeless> meh, will deal

[04:08:35] <lifeless> my fault for not pushing it through to prod

[04:09:00] <ErikRose> And _prepare_file() went away? That's where I did some work.

[04:09:18] <ErikRose> But I don't think it will be disastrous.

[04:09:18] <lifeless> ErikRose: yeah, there's a whole new cache layer

[04:09:33] <ErikRose> My heaviest work is probably going to end up in pip.download.

[04:09:36] <lifeless> its a partial decouple from 'concept of a thing we install

[04:09:50] <lifeless> and 'cache on disk of something we downloaded/unpacked'

[04:10:18] <ErikRose> I take it that's a different kind of cache than the current cache of web requests?

[04:10:24] <lifeless> ErikRose: yes

[04:10:39] <lifeless> ErikRose: this is a cache of e.g. unpacked zip/wheel/whatever, prepped with egg_info

[04:10:51] <ErikRose> Nice docstring. :-)

[04:10:58] <lifeless> which ?

[04:11:04] <ErikRose> https://github.com/pypa/pip/compare/develop...rbtcollins:issue-988#diff-818fa0077e580ddb67a9dd4651035438R26

[04:11:10] <ErikRose> (on RequirementCache)

[04:11:27] <dstufft> pip is actually a thinly veiled attempt to see how many different caches we can put into one program

[04:11:45] <lifeless> ErikRose: heh, thanks :)

[04:12:34] <dstufft> I just want to get warehouse done so I can focus more time on pip again :(

[04:12:47] <ErikRose> Well, we've got the caching bit, we've got the naming bit (safe? trusted? verifiable? hashed?). What was that third?

[04:13:06] <ErikRose> Keep fighting the good fight, dstufft! We believe in you. :-)

[04:13:22] <ErikRose> Ah yes, off-by-ones. We need to get some more of those in there.

[04:14:32] <dstufft> ErikRose: luckily safe/trusted/verfifable and shit is all gone in pip 8

[04:14:50] <ErikRose> I feel like I should know about that ASAP before I write more code.

[04:15:45] <dstufft> ErikRose: PEP 470

[04:16:04] <ErikRose> Oh, this.

[04:16:04] <dstufft> the tl;dr is there's no more shit on /simple/ that isn't hosted on PyPI

[04:16:23] <dstufft> a repository has files, PyPI as a repository only allows files hosted on it

[04:16:31] <dstufft> pip finds files

[04:16:34] <dstufft> etc etc

[04:16:50] <ErikRose> That's fine with me. Where's the pip 8 work? Is it on some mysterious branch somewhere?

[04:17:02] <ErikRose> Or am I safe basing off develop?

[04:17:11] <dstufft> it's develop

[04:17:17] <ErikRose> Hooray. Saved.

[04:17:25] <dstufft> it's the next release of pip :)

[04:17:31] <dstufft> 8 because we've removed stuff

[04:17:37] <dstufft> it'll be ~ 3 months out or so

[04:18:26] <ErikRose> It's always a race to land your stuff before something else clobbers it. :-) I should make it just fine.

[04:22:58] <dstufft> (the stuff is already removed in develop, just we can't release for ~ 3 months because of deprecation time lines)

[04:23:33] <ErikRose> I was referring to the "trusted/safe" wording in pip.download, which seems to refer mostly to HTTPS.

[04:23:39] <ErikRose> That's still there, as I'd expect.

[04:24:41] <dstufft> ah

[04:25:05] <dstufft> that's probably centered around the fact we won't use a non HTTPS site by default, and even if you enable it we disable some features like caching

[04:25:18] <ErikRose> correct

[04:27:01] <dstufft> theortically we could allow it if you were using all hashed packages, but I'm -1 on relaxing it even then.

[04:27:59] <lifeless> hmm, half-fixed up

[04:28:26] <lifeless> dstufft: uhm, so meta. It would be great if we could keep develop always releasable

[04:28:42] <lifeless> dstufft: so e.g. put things like that in a branch and merge it at the time

[04:30:11] <lifeless> dstufft: are the PIP CI fails due to the PyPI search issue ?

[04:30:17] <lifeless> bblater, daughtertime

[04:30:34] <ErikRose> dstufft: That's such a corner case. I don't care about it until somebody yells.

[04:30:59] <dstufft> lifeless: they were yea, I xfailed them for now

[04:31:05] <ErikRose> Though the way I'm building this, it demands everything be hashed if anything is, so that'd be an easy thing to add.

[04:31:30] <dstufft> my traffic will be encrypted or it will be bullshit

[04:31:52] <ErikRose> Encryption is still nice to ensure privacy.

[04:33:22] <dstufft> lifeless: the problem is, if we don't plan to make a release in that time, we longer we allow it to remain, the more work it is to go back and find any changes to review them to make sure messages/etc don't need updated

[04:35:07] <dstufft> oh

[04:35:10] <dstufft> a new search failure

[04:35:11] <dstufft> sigh

[04:52:26] <ErikRose> G'nite, all.

[07:21:19] <lifeless> dstufft: well, if no other changes landed, then the cost is zero

[07:49:37] <lifeless> issue-988 branch updated (but haven't run tests yet or resolved the semantics around constraints

[07:58:23] <lifeless> ErikRose: ^ issue-988 branch updated (but haven't run tests yet or resolved the semantics around

[07:58:26] <lifeless> constraints

[12:22:43] <xafer> Why is https://pypi.python.org/pypi/INITools not found by pip search ? :-/

[12:23:31] <dstufft> search is fucked

[12:23:34] <dstufft> and I don't know why

[12:25:14] <xafer> will warehouse provide the same xmlrpc api ?

[12:25:55] <dstufft> yea

[12:26:17] <dstufft> Mostly anyways, a few XMLRPC methods that weren't being used I removd

[12:26:24] <dstufft> search is the only xmlrpc api method not implemented yet

[12:31:13] <xafer> well search seems to have improved a little lately, it seems to finally have indexed pip 6.0 proper description ;)

[12:31:14] <xafer> cf https://github.com/pypa/pip/pull/3096

[12:31:39] <dstufft> I made it so instead of taking 6 hours to reindex it only took 40 minutes

[12:31:43] <dstufft> still a crazy amount

[12:33:41] <xafer> well apparently for the last 9 months, it kept showing pre-version 6 description when searching for pip, as if pip was not reindexed :o

[12:46:05] <xafer> ok for https://github.com/pypa/pip/pull/3096 ?

[15:06:04] <xafer> dstufft, I'd like to merge https://github.com/pypa/pip/pull/2812 , what do we usually do with duplicate PR like https://github.com/pypa/pip/pull/3084 ?

[15:06:29] <dstufft> close them and say they were fixed in the other one

[15:06:43] <xafer> k

[15:12:21] <ErikRose> Heh, just found a 37-line dead procedure.

[15:13:20] <dstufft> doesn't surprise me!

[15:16:58] <xafer> _copy_dist_from_dir ?

[15:17:09] <ErikRose> You got it.

[15:18:46] <xafer> strange commit https://github.com/pypa/pip/commit/4e6d935eafab56695f74f89bde04aae885441b7a :o

[15:19:27] <ErikRose> That is strange.

[15:19:29] <xafer> Looks like a bit is missing

[15:19:37] <ErikRose> I can vouch that msabramo generally knows what he's doing, though.

[15:20:37] <xafer> I guess he was benchmarking with or without the change and commited the without :p

[15:21:06] <ErikRose> Yep

[15:21:27] <ErikRose> I'll comment on the commit; maybe we can wake him up. :-)

[15:42:14] <xafer> https://github.com/pypa/pip/blob/develop/pip/utils/appdirs.py#L100-L129 does not seem to be used anymore since https://github.com/pypa/pip/commit/3569a9b06003e0b27e625af315424619daa0f1da

[15:44:02] <dstufft> can probably delete it

[16:06:18] <qwcode> !t xafer

[16:06:19] <pmxbot> XAFER at N/A (ET): N/A (N/A)

[16:06:52] <ErikRose> dstufft: Did so. He can put it back if he is still interested.

[16:07:44] <xafer> ? :o

[16:08:51] <qwcode> xafer, oh, I was trying to give you kudos for doing so much tidying and goodness for the project

[16:11:11] <xafer> :) thanks

[16:18:42] <ErikRose> The more I remove mutability from objects, the less time and order matter, the fewer errors are possible, and the saner I become.

[16:19:54] <xafer> https://github.com/pypa/pip/pull/3098 and https://github.com/pypa/pip/pull/3099 seem ok to you ?

[16:26:25] <dstufft> ErikRose: have you seen https://github.com/tobgu/pyrsistent

[16:26:37] <ErikRose> nope

[16:26:56] <ErikRose> This looks familiar after studying CLojure. :-)

[16:27:09] <ErikRose> I wonder to what degree they're efficient COW.

[16:27:48] <ErikRose> Wow, they did C extension versions as well. Somebody's serious.

[16:28:11] <ErikRose> I wonder what their motivation was; they have quite a team there.

[16:31:39] <dstufft> ErikRose: I haven't actually used them yet, keep meaning to find a reason to

[16:32:13] <ErikRose> Especially gutsy is https://github.com/tobgu/pyrthon, which does AST transforms to retrofit the immutable versions as the std ones.

[16:32:26] <ErikRose> I wonder how many things run unmodified under it.

[20:20:00] <ErikRose> The hash integration appears to be working!

[20:23:51] <ErikRose> With that: weekend!

[20:25:35] <ErikRose> https://github.com/pypa/pip/compare/develop...erikrose:hashing?expand=1

[20:25:48] <ErikRose> Still needs tests added/fixed and some cosmetic work on error messages

[20:26:11] <ErikRose> And I'll see about --hash rather than --sha256

[21:41:36] <xafer> pfff, how did I not think of the search failure ? -.-

Log file Viewer

Help | Karma | Search:

#pypa-dev logs for Friday the 11th of September, 2015