Log file Viewer
#pypa-dev logs for Tuesday the 13th of May, 2014
[02:57:10] <dstufft> the categories are really hosted on pypi, hosted safely externally, hosted unsafely externally
[02:59:09] <dstufft> I can change it, I was trying to distinguish between externally safely and externally unsafely more obviously by using different words instead of an ajective
[12:47:54] <dstufft> jaraco: heya :) I was looking at easy_install, does it only offer --find-links for adding additional urls to find packages at? or does it support extra index URLs like pip does too and I'm just missing it?
[12:48:18] <dstufft> Alex_Gaynor: heh, well it was basically detailing all the possible ways pip can find a file, and if they are safe or not (and why)
[12:49:33] <dstufft> There's only 5 different combinations! 7 if you count MAL's proposal, and I think like 15 if you count TLS too
[12:49:50] <dstufft> jaraco: does that replace the default url of pypi, or does that add an additional one?
[12:50:45] <dstufft> jaraco: would you be opposed to an option that adds an additional one if a PR was added for it?
[13:01:23] <dstufft> pf_moore: it kind of lays out (I think) part of why I hate the external links, because you need to have very specific conditions for the links to be safe, and missing any link in the chain causes it to be unsafe
[13:02:02] <dstufft> plus then you have the whole TLS thing and varying defintions of what "safe" means (MAL's proposal is "safe", but not as implemented, TLS is actually "safe" as implemented, but we don't consider it safe)
[13:02:20] <pf_moore> dstufft: Yeah, I hadn't realised that the chain of links with hashes (MAL's scenario) wasn't explicitly covered by the PEP
[13:03:31] <dstufft> https://pypi.python.org/simple/foobar/ linking to https://downloads.example.com/ linking to http://archive.example.com/foobar-1.0.tar.gz#md5=hash is actually safe in the real world (and every combination where you have hashes and TLS as long as they all have them)
[13:03:49] <dstufft> but it gets really confusing as to what is safe or not if you're not intimately aware of all the rules :/
[13:04:34] <pf_moore> Actually no, IMO it's pretty simple (maybe just because you explained it ;-)) just hard to explain clearly.
[13:05:10] <pf_moore> Every link must be individually safe - safe means "source has a hash of the target" or "target uses https".
[13:06:15] <pf_moore> yeah, I threw that in because you did. Ignore it for the current state of affairs.
[13:06:37] <dstufft> largely because long term we don't want to rely on TLS for the security of the system, we're just using it t get the inital link in the chain of the simple index
[13:07:11] <pf_moore> AFAIK, the only other thing is that simple->PyPI is always considered safe even without a hash. Is that true? Or do we always add a hash on upload?
[13:07:53] <dstufft> anything hosted on PyPI always included a hash (unless someone manually adds a link to their simple page withou a hash to a file that is hosted on PyPI)
[13:08:24] <dstufft> in the current implementation that additional link would be considered unsafe (but ti wouldn't matter because the safe link which PyPI generates would still be there)
[13:08:33] <pf_moore> OK, so it's easy then :-) (Reminds me of the maths professor who spent 2 days proving something was "obvious"...)
[13:09:21] <dstufft> it's easy once you know the rules, it's sorting out what the rules are which gets a little confusing, especially if you consider that the most common response to me trying to explain the difference to people is "what's a simple index?"
[13:09:58] <dstufft> most people simply have no idea how pip discovers files, or that they can be hosted externally to PyPI at all
[13:10:20] <dstufft> so before you can explain how the safe/unsafe categorization works, I end up having to explain how pip finds files
[13:12:05] <dstufft> it means that your average user will have basically zero insight into what flags they require to install something without just trying and seeing if it fails or succeeds
[13:12:09] <pf_moore> There's bits of pip internals that look suspiciously like chicken entrails actually...
[13:13:41] <pf_moore> Yeah. The aim should be that people don't need any flags cos that's simple - that's sort of where this started, with people wanting --allow-all-external to be the default
[13:14:24] <dstufft> yea, the problem is --allow-all-external won't actually make things work in most situations ;(
[13:14:39] <pf_moore> By the time people understand why they might care that something is external, we've pretty much lost
[13:16:05] <pf_moore> It's why the extra index idea might work better. I trust PyPI but mx-tools is on egenix.com. Do I trust them? OK, go for it.
[13:16:24] <pf_moore> Whether "trust" means reliability or security only needs to matter if the end user wants it to matter.
[13:16:33] <dstufft> my PEP basically says "turn off external hosting on PyPI all together, provide a way for projects to say "my fiels are hosted on THIS index" and have pip/easy_install etc be able to query that information and if someone says ``pip install foobar`` and we don't find a download for foobar, but we find that "my files are hosted on https://pypi.example.com/" we can say "foobar uses a custom index, add --extra-index-url https://pypi.example.com/
[13:18:13] <dstufft> like the one where we can't know if a particular URL is supposed to be a hard fail or a soft fail if we can't find it, so if can't find it we assume soft fail and just ignore that URL isntead, so if the URL we actually need is down, we don't tell the user that and instead we say "we couldn't find any versions of foobar to install"
[13:19:51] <pf_moore> Yeah, but that's sort of a different problem. Error reporting is entirely pip's problem to handle well, and we don't. We're simultaneously too verbose and too terse.
[13:21:20] <dstufft> because we can't really rely on the external URLs existing because a ton of them simply don't
[13:21:50] <pf_moore> Yeah, we need a clear model of what we expect and how we respond to be able to describe things properly in the error
[13:22:12] <pf_moore> Hmm, I wonder if there are some GSOC projects in the user interface side of pip?
[13:22:26] <dstufft> yea, and the external URLs are a big unknown quantity where sometiems they are mandatory for a particular project, sometimes they are optional, and sometimes they will always fail and should be ignored
[13:23:11] <dstufft> one thing i'd really like to do is figure out how to remove some of our options
[13:24:56] <dstufft> this whole thing has been making me think about --index-url, --extra-index-url, and --find-links and how those three are related but slightly different and how that applies to the UX
[13:25:06] <dstufft> (try explaining the difference between an index and a find-links to someone :V)
[22:41:08] <dstufft> https://github.com/pypa/warehouse/issues/319 is going to be trickier than it seems