Log file Viewer
#pypa-dev logs for Friday the 2nd of January, 2015
[04:20:18] <dstufft> that's not expected, pip install --user on 2.7 should give ~/.local/lib/python2.7/site-packages/e.py
[04:20:53] <r1chardj0n3s> had to do this thing first as well :/ http://stackoverflow.com/questions/4495120/combine-user-with-prefix-error-with-setup-py-install
[04:21:06] <r1chardj0n3s> (and the error from pip isn't helpful, which is a shame, but tricky to fix)
[04:22:25] <r1chardj0n3s> lrwxr-xr-x 1 richard staff 38 2 Jan 15:13 /usr/local/bin/python2.7 -> ../Cellar/python/2.7.8_1/bin/python2.7
[04:25:24] <dstufft> since /Users/richard/Library/Python/2.7/lib/python/site-packages/e.py is certainly a user directory, it's not one I've noticed before but I compile python via pyenv
[04:26:53] <dstufft> https://github.com/Homebrew/homebrew/blob/master/Library/Formula/python.rb#L191-L200
[04:26:55] <r1chardj0n3s> \o/ I found my image for the "and then CI systems became popular" slide http://4.bp.blogspot.com/-QgVBW4nCvcg/Uyvn0QBZQRI/AAAAAAAAAOI/TDsFOTER9Z0/s1600/End+of+the+World.jpg
[04:28:55] <dstufft> r1chardj0n3s: pip 6 should mean we get less traffic \o/ (except from travis and other emphereal build systems *womp whomp*)
[04:30:33] <dstufft> r1chardj0n3s: it follows the CacheControl/expires headers with the caveat that the /simple/ page patches have a hardcoded max age of 10 minutes just to prevent a bad cachecontrol/expires header from breaking things
[04:30:57] <dstufft> since we expect the /simple/ pages to change periodically but we don't really expect the package files to change
[04:31:31] <dstufft> so if we have a cached response, but it's expired and it has an Etag it'll do a conditional get so we don't have to re-download the body response
[04:35:04] <dstufft> r1chardj0n3s: I forget, is there a PEP needed for disabling upload -> delete -> re-upload dance?
[04:37:09] <dstufft> r1chardj0n3s: you know how you can delete a file, change the contents, and upload a new file with the same name?
[04:41:16] <dstufft> https://mail.python.org/pipermail/distutils-sig/2014-September/024896.html a +1
[04:41:28] <r1chardj0n3s> PEP 458 and 480 would probably affect the editing of metadata post-release
[04:41:42] <dstufft> https://mail.python.org/pipermail/distutils-sig/2014-September/024897.html -1 from MAL
[04:42:07] <dstufft> r1chardj0n3s: not really, either the author would need to upload new signed metadata or PyPI would just sign the new metadata
[04:42:12] <r1chardj0n3s> (in fact, one of my huge red flags with PEP 458 / TUF is the requirement that EVERYTHING for a release is released at the same time)
[04:43:14] <dstufft> gonna call this a an implicit +1 https://mail.python.org/pipermail/distutils-sig/2014-September/024903.html
[04:43:34] <dstufft> r1chardj0n3s: I don't think it needs that, just the signed metadata has to contain everything, but you can upload new signed metadata
[04:44:18] <dstufft> https://mail.python.org/pipermail/distutils-sig/2014-September/024901.html another +1
[04:44:27] <dstufft> https://mail.python.org/pipermail/distutils-sig/2014-September/024910.html another
[04:44:36] <dstufft> https://mail.python.org/pipermail/distutils-sig/2014-September/024911.html another
[04:45:34] <r1chardj0n3s> ok, so assume that we *do* allow re-upload of a file with the same name, what's preventing it?
[04:45:52] <r1chardj0n3s> (sorry, we *currently* allow it, what's stopping us from doing so down the track?)
[04:46:45] <r1chardj0n3s> "When a project uploads a new transaction, the project transaction process MUST add all new targets and relevant delegated bins metadata. "
[04:47:37] <dstufft> r1chardj0n3s: I'm crossing the streams here, is the first two lines there about the delete loophole?
[04:48:58] <dstufft> becasue if authors are signing for like /simple/<foo>/, it removes the ability of PyPI to ever really evolve that API
[04:49:30] <r1chardj0n3s> that's what TUF ca;lls the "snapshot" isn't it? that is supposed to be signed by PyPI in their design
[04:49:45] <dstufft> r1chardj0n3s: so it's a bit complicated because there are two kinds of metadata at play
[04:50:11] <dstufft> there is the TUF metadata, the TUF metadata is essentially a signed document asserting the contents of some other file
[04:50:43] <dstufft> and in the current PEP, their TUF metadata is responsible for verifying /simple/<foo>/ and /packages/.../foo.tar.gz
[04:52:02] <dstufft> r1chardj0n3s: the way the current PEPs are written is that TUF more or less takes the place of requests.get(), except it verifies using TUF the thing that gets downloaded
[04:52:50] <dstufft> A claimed or recently-claimed project will need to upload in its transaction to PyPI not just targets (a simple index as well as distributions) but also TUF metadata. The project MAY do so by uploading a ZIP file containing two directories, /metadata/ (containing delegated targets metadata files) and /targets/ (containing targets such as the project simple index and distributions that are signed by the delegated targets metadata).
[04:54:26] <dstufft> r1chardj0n3s: the way they designed the PEPs is that TUF more or less takes the place of TLS
[04:55:27] <dstufft> r1chardj0n3s: that we should use TUF to verify everything we downloading, which includes the installer API
[04:55:56] <dstufft> currently in PEP 480 the author is responsible for signing the installer API files that pertain to their project
[04:56:05] <r1chardj0n3s> I don't see why /simple/ needs to *exist* let alone be signed by the project
[04:56:54] <dstufft> r1chardj0n3s: well it doesn't need to exist, you can use the TUF API itself to do that, they just choose this way becasue it represents the minimal amount of change in pip, since they can essentially just replace the requests.get() calls with a call to a TUF verifying thing
[04:57:13] <dstufft> I told them in my last email I don't think we need to optimize for minimal code changes in pip
[04:57:58] <dstufft> and if we have another file like /simple/, even if we're not relying directly on TUF for the installer metadata like "what versions are available", I tihnk that should be signed by PyPI not by the author
[04:58:38] <dstufft> and the attacks that makes possible are not much worse than the attacks that are possible if someone gets the snapshot or timestamp key
[04:59:45] <dstufft> but anyways, as far as is related to deleting files, I think the worst case scenario in a post TUF PEP 480 world is that you can't delete files from the web UI and you have to use a tool to generate metadata that you upload which triggers the delete
[05:03:26] <r1chardj0n3s> http://www.abc.net.au/news/specials/heatwave-tracker/ right at the end, that's why my aricon is now on
[05:03:46] <r1chardj0n3s> predicted to hit 40º today, 42º tomorrow and then back to 24º on Sunday :)
[05:05:03] <r1chardj0n3s> (I live in the larger Australia-shaped blob just above the smaller Tasmania-shaped blob ;)
[05:13:23] <dstufft> given that we had a bunch of +1, one +0, and one -1 do we need a PEP for it or is that enough of a consensus to just change it?
[05:13:44] <r1chardj0n3s> one of the other Rackers here lives right under where the hot, red blob is at the end of the timescale. he was going to be golfing today.
[05:13:50] <pmxbot> jaraco pushed 11 commits to setuptools (https://bitbucket.org/pypa/setuptools/) :
[05:14:40] <dstufft> (I'm not sure what exactly i'd write in a PEP tbh, other than "Don't allow delete + repload if the filename matches a previously uploaded file)
[05:14:42] <r1chardj0n3s> I know a few prominent folk have expressed their desire to continue to be able to replace files
[05:18:35] <r1chardj0n3s> I think I'm going to twit it, see what the general reaction is. Will be more muted than otherwise could be, given the time of day and time of year
[05:19:37] <dstufft> (FTR MAL is the only person i've ever met who was against it who I couldn't pretty quickly convince that it's a bad idea)
[05:20:34] <dstufft> I'm at least 25% convinced that half the reason I couldn't convince him is because it came from me
[05:21:12] <dstufft> for better or worse I've spearheaded some things that made things worse for his particular workflow
[05:22:25] <dstufft> (Also FTR, npmjs.org a few months ago removed the ability to republish files for the same reason)
[05:23:13] <r1chardj0n3s> I agree that the loophole is bad; my intention was to never allow people to do what they're doing, but I was lax in enforcing it once the loophole was discovered
[05:24:51] <dstufft> I don't think it's any secret that I'm far less convervative about breaking things I don't like than some people would like :D
[05:25:06] <dstufft> so I don't mind waiting a bit ;P just trying to close the gap on my PyPI TODO's now!
[05:29:21] <r1chardj0n3s> I'm just exporting my talk slides before I finish - will email 'em to you, thanks for agreeing to give 'em a read. the presenter notes are my usual ramblings, hope there's enough to make sense....
[12:57:21] <dstufft> pf_moore: ftr, I think the PEPs themselves (not what they are proposing, but the PEPs themselves) are pretty poor :(
[12:58:04] <dstufft> I'm writing up an email where I suggest focusing just on PEP 458 (which is more or less transparent to everyone but people writing tooling and people managing PyPI) and where I offer to help them write these things
[12:58:22] <pf_moore> dstufft: Yeah, clearly written by specialists who know their stuff but don't know how to put it in terms "normal people" can follow :-)
[12:59:13] <dstufft> I don't think it's really reasonable to require reading an academic whitepaper and the documentation/specification for the TUF project to get a high level overview of what the PEP proposes
[12:59:24] <dstufft> (I mostly know what I know about TUF because I've read those things, multiple times)
[12:59:37] <pf_moore> The only reason I'm banging on is that I don't want the PEPs (which are good ideas) getting lost in a huge debate over people thinking it'll make their life difficult
[13:00:23] <pf_moore> Agreed. I feel guilty that I'm commenting after barely skimming the proposals, but I think that's what most people will actually do...
[13:01:22] <dstufft> we have the problem where we have to handle everything from really serious projects that have people who care deeply about these issues to really fun hobby projects where it's just one guy who releases things once in a blue moon
[13:02:10] <pf_moore> And without branding the hobby stuff as "toy" or the serious stuff as "too big for PyPI"...
[13:03:56] <dstufft> and the sad fact is, that unless we get something that's good enough that we can either mandate, or provide warnings for or something, then PEP 480 isn't very useful
[13:04:18] <pf_moore> We're getting very, very close to a curated index here. Maybe that's a viable option. Have a second subset-index of PyPI that only contains things that meet a certain security standard. You get added by asking, but it's not the default, and people needing a level of security can switch to that index.
[13:04:18] <dstufft> because if I install 100 things, and 99 are signed by the end author, and 1 isn't, then someone with contorl over PyPI will just attack me with that 1 thing
[13:05:19] <dstufft> it's just instead of relying on PyPI to provide ACLs for who can upload a file, we rely on cryptography to do it
[13:06:29] <dstufft> (curated vs non curated has other connotions, like Linux repos are curated, PyPI is not and I sort of jumped the gun)
[13:06:35] <pf_moore> BTW, do you have any realistic idea of what the likelihood is of an attack that compromises PyPI in the way the PEP describes?
[13:07:39] <dstufft> PEP 458 == we protect against Fastly being compromised and people can use a mirror of PyPI without trusting that the mirror operator is a good person
[13:08:07] <dstufft> PEP 480 == we protect against the servers that Myself, Richard, EWDurbin etc run on Rackspace being compromised
[13:08:48] <pf_moore> Yes, ta. Say PEP 458 then (in the first instance). What's the odds of fastly being compromised for long enough to do real harm?
[13:10:19] <pf_moore> I've never even *heard* of a case where a library was maliciously compromised. As opposed to the many things you do hear of (bugs, DDOS on application systems, ...)
[13:10:30] <dstufft> Hopefully not very likely :) I do not know the details of Fastly's security but I honestly do not worry about them being a weak link, for me PEP 458 is primarily good about allowing people to use mirrors without trusting the mirror operators
[13:12:52] <dstufft> (I mean, right now China can create certificates that are valid for pypi.python.org and can serve packages to people trying to access pypi and pip would be none the wiser)
[13:13:29] <pf_moore> OK. But doesn't 99.99% of the world rely on TLS? Gmail, facebook, dropbox, skydrive...? Do they have something else as well? From the idiot's POV, those are better targets than PyPI...
[13:14:25] <pf_moore> Not to say we shouldn't solve the problem, but this is what makes it hard to keep the UX debate in perspective...
[13:14:53] <dstufft> yes the vast bulk of the world relies on TLS, and there are things that we (pip) can do that a lot of browsers are starting to implement to help close the gap on a lot of those things
[13:16:26] <dstufft> similarly to how most people don't need to care about the fact we're using TLS right now
[13:17:09] <dstufft> in PEP 458, authors continue to upload things how they do now, PyPI gets modified to sign things and serve the new TUF metadata, and pip gets modified to verify that and it ships the keys
[13:19:36] <pf_moore> Yeah, I think PEP 458 needs to be presented as a zero-impact infrastructure change that just improves things at no cost to anyone outside the PyPI admins.
[13:20:20] <pf_moore> I suspect there'll be issues for projects like devpi to look at, but that's the extent of the impact
[13:20:22] <dstufft> I mean it also imposes cost on people like holger who writes devpi and bandersnatch and them
[13:20:43] <dstufft> but for people who are installing things and people who are uploading hings, it's basically transparent except maybe error messages change
[13:21:13] <pf_moore> Does it affect external hosting at all? That's the other group who tend to get forgotten...
[13:22:39] <dstufft> pf_moore: yes and no! how we verify externals on PyPI is an open question. PEP 470 answers that question by saying "it's not our problem, the external hosting people need to setup a repo", there are other options where we keep the externally scraped links stuff
[13:23:40] <dstufft> beyond what PyPI presents about external links, it's mostly up to pip about what we want to mandate a repository looks like
[13:24:13] <pf_moore> Yeah, but that's 470, not 458. Let's keep things separate - "there's no impact on external hosting from 458" is the message.
[13:24:20] <dstufft> (e.g. do we want to mandate that a repository uses TUF? or are we happy to allow people to continue using TLS only repositories)
[13:24:42] <dstufft> pf_moore: as PEP 458 is currently implemented there i no impact to external hosting yea
[13:26:28] <pf_moore> Personally, I don't think pip should mandate anything more than the bare minimum if the users are opting into a repo, but that's a whole other debate...
[13:27:44] <dstufft> pf_moore: yea, that's why my draft email suggests focusing just on PEP 458 right now, which as it currently stands is mostly transparent except to us poor shmucks who agreed to maintain these tools
[13:28:35] <pf_moore> Cool - so I see the best thing as a reboot of PEP 458, explicitly and repeatedly asserting that it has no UX component and is purely an internal improvement affecting PyPI and the mirroring tools.
[13:29:04] <pf_moore> We table PEP 480 totally, and anything else that intrudes, until 458 is resolved.
[13:30:05] <dstufft> basically in these things there are chains of keys where you trust the root key, and the root key signs a document that says "hey trust these other keys for X", and those other keys can sign other keys that say "hey trust these other other keys for Y" and so on
[13:31:54] <dstufft> and if you want to verify Y, you have to verify the key that signed for Y, the key that signed for X (where X is a thing that says hey trust this other other key for Y), and then you have to verify that intiial document which was signed by the root key
[13:32:14] <dstufft> and the security of Y is basically equal to whatever the security is of the weakest key in that chain
[13:34:33] <pf_moore> Yeah, I sort of understood that (although the last bit about which keys you trust under the 2 PEPs is new to me, thanks)
[13:35:27] <pf_moore> But talk of key signing reminds me of the 90s, with security geeks talking about holding key-signing parties.
[13:36:05] <dstufft> pf_moore: In my opinion, if we do PEP 480 we should use something like what minilock has done, which is the key that an author needs to sign with is derived from a secondary password
[13:36:29] <dstufft> so the UX implications are that when you do ``setup.py upload`` it prompts you for your "signing password"
[13:39:49] <dstufft> pf_moore: it's kind of an interesting situation though, I'm talking to some other epople about PEP 458/ PEP 480 right now, and they are excited about both proposals (without having actually read them to ensure that they do what they claim), however they are part of a group of people who write software that helps fight against oppresive regimes in other countries and they distribute software via PyPI/pip for that purpose
[13:39:50] <pf_moore> Hmm, just googled minilock. I see what you mean now. Makes sense. I tend to think of a "pass phrase" for this. I'm aware there's no practical difference between a password and a pass phrase other than length and allowing spaces, but it's all about connotations...
[13:41:14] <pf_moore> wow - yeah, it's easy to forget that what's just a hobby to me can be so important to others...
[13:44:53] <dstufft> I have the good old, security paranoia so I tend to lean towards designing systems that can handle "I'm distributing software where if I don't get the right software someone might die" (hows that for motivation), but I'm super grateful for people who tend to lean towards "I'm just distribution some things i do for a hobby, if i don't get the exact right software I'm not to worried about it" who force me look at the UX side of things more
[13:46:21] <dstufft> (for the oppresive regimes types, since PyPI is hosted in the US, the only thing you need to worry about is the US.... whch isn't exactly a benign operator in the global online battlefield)
[13:49:13] <pf_moore> dstufft: Trouble with security stuff is the people like those you mention for whom it *is* life or death tend not to be able to comment. So having the specialists speaking on their part against the "who cares" mob like me is really important
[13:51:06] <dstufft> (reminder that it was roughly 2 years ago that PyPI had an untrusted TLS certiifcate, and pip/easy_install didn't even bother to verify TLS, and would just scrape random sites on the internet)
[13:53:09] <dstufft> (and we've still got a long way to go in some places, ``setup.py upload`` on anything but Python 2.7.9 doesn't verify TLS which means it's easy to get an author's password ;(
[14:13:50] <dstufft> pf_moore: if you have a moment to read https://bpaste.net/show/5a5bd70a5c97 do you mind? before I send it :D
[14:36:47] <collinanderson> just got a varnish "backend read error" on pypi. it worked after refreshing
[14:38:19] <dstufft> collinanderson: that more or less means that something went wrong between Fastly and the PyPI origin servers, where something went wrong can include things like "the request took too long" or "the internet sucks and sometimes connections get dropped"
[15:25:54] <pf_moore> dstufft: sorry got sidetracked, just posting a reply now but looks great to me.
[15:28:17] <pmxbot> jaraco pushed 13 commits to setuptools (https://bitbucket.org/pypa/setuptools/) :
[15:52:07] <pmxbot> jaraco pushed 5 commits to setuptools (https://bitbucket.org/pypa/setuptools/) :
[16:47:35] <pmxbot> jaraco pushed 8 commits to setuptools (https://bitbucket.org/pypa/setuptools/) :
[16:47:35] <pmxbot> Extend contexts environment patching logic based on context manager in test_msvc9compiler and use it.
[17:03:24] <pmxbot> jaraco pushed 5 commits to setuptools (https://bitbucket.org/pypa/setuptools/) :
[17:30:12] <_cjones_> Hi guys. Our team has run into an issue with PIP 6.0.3 Specifically this issue: https://github.com/xavfernandez/pip/commit/bd2a789578ea70e2207e9d4ee8e70f31f42b0957
[17:30:36] <_cjones_> Does anyone know approximately when another official release will be spun? (6.0.4)?
[18:16:29] <pmxbot> jaraco pushed 19 commits to setuptools (https://bitbucket.org/pypa/setuptools/) :
[18:16:29] <pmxbot> Use unicode literals to define files as text, and encode specifically when saving files.
[19:00:46] <msabramo> I guess they've agreed to maintain stuff for a while? https://crate.io/thanks-dstufft/
[19:02:53] <sigmavirus24> so long as we keep hitting /packages/* we can get them to serve an alternative index?
[19:03:15] <sigmavirus24> Nothing to stop them from eventually acting maliciously (or if their servers are compromised...)
[19:28:28] <pmxbot> jaraco pushed 8 commits to setuptools (https://bitbucket.org/pypa/setuptools/) :
[19:31:17] <pmxbot> jaraco pushed 3 commits to setuptools (https://bitbucket.org/pypa/setuptools/) :
[20:31:06] <sigmavirus24> dstufft: thoughts on https://bitbucket.org/pypa/setuptools/issue/323/install-issues-with-102 ?
[21:36:55] <sigmavirus24> I'd love some review on https://bitbucket.org/pypa/setuptools/pull-request/114/fix-regression-in-entry-point-name-parsing/diff by some people who are available
[21:48:18] <pmxbot> jaraco pushed 4 commits to setuptools (https://bitbucket.org/pypa/setuptools/) :
[22:49:20] <pmxbot> jaraco pushed 3 commits to setuptools (https://bitbucket.org/pypa/setuptools/) :