Log file Viewer
#pypa-dev logs for Monday the 13th of July, 2020
[14:59:08] <pradyunsg> cooperlees: regarding the PEP you're writing, I just read your comment and wanted to suggest HackMd to you. You'd probably want to be careful about the permissions when you share the link, but it's certainly a good platform for this kind of use (unless you want people to suggest exact wording, in which case Google Docs wins).
[15:12:35] <sumanah> cooperlees: thank you for the draft https://github.com/cooperlees/peps/blob/warehouse_json_api/pep-9999.rst !
[15:13:32] <sumanah> people can comment on the GitHub commit, e.g., https://github.com/cooperlees/peps/commit/ffd4ea5629f9bc62bff0280f720f35e18a037209
[15:14:23] <cooperlees> Ahh - I couldn't find the URL to get that mode! Thanks - That's the best. Let me edit my comment on the issue
[15:15:16] <cooperlees> sumanah: Main feedback I want - Is using the "JSON Schema" a sane way to document the JSON? There is no real standard for documenting JSON (still!)
[15:18:01] <cooperlees> That's the JSON standard - Not how to document your JSON data structures unless my quick look failed me ...
[15:18:22] <cooperlees> JSON Schema is ment to do what I want for our API - But isn't an official standard
[15:18:57] <cooperlees> So unless someone has a better way to document that's where I think I'll go, I just didn't want to go document every field in the JSON until that accepted by majority stakeholders
[15:19:43] <sumanah> hey -- sorry, stepped away for a few minutes to give short story recommendations to a friend who is recovering from COVID :\
[15:20:23] <cooperlees> My girlfriends Boxing coach got diagnosed, so I am yet to show symptoms - but will get tested later this week. Still not symptoms here!
[15:20:39] <sumanah> cooperlees: later today my spouse Leonard is gonna take a look -- he wrote a book about web API design so I figure he can comment a bit here on how to unambiguously describe this
[15:24:08] <cooperlees> If people are happy I'll finish the API fields this week. Then it's ready for di_codes + dstufft + EWDurbin critiquing for all the things I missed :)
[15:25:25] <cooperlees> sumanah: Can your partner also look at my proposal comment on https://github.com/pypa/packaging-problems/issues/367#issuecomment-657624768
[15:26:49] <cooperlees> I'm a big fan of standardizing on a `/json` so devpi and others can easily have a per index JSON API
[15:27:24] <cooperlees> Naturally we'll have to keep the legacy URL for the foreseeable future - But we should deprecate it
[15:30:58] <cooperlees> Not being in the core Python team @ Facebook anymore I don't even run it anywhere :(
[15:32:28] <EWDurbin> This is totally doable. Want to setup a call some time in the second half of this week to talk about it?
[15:35:50] <cooperlees> I am still there, just back in a Networking team (what I'm actually good at)
[15:42:01] <dstufft> cooperlees: small world, I play video games with someone who works at Facebook and knows you :p Not quite sure his actual name, just his in game name, but it made me laugh when I found out they worked at Facebook and knew you
[15:42:57] <cooperlees> If people work @ Facebook in infrastructure, especially networking or Python heavy shit - They've prob read my spam / ramblings
[15:43:57] <cooperlees> dstufft: Want contributor to my repo so you can just fix things / save time?
[15:44:02] <dstufft> cooperlees: I've been goofing around in EverQuest, old game that I'vee been playing mostly out of nostalgia
[15:45:45] <dstufft> cooperlees: I'm probably not gonna make direct changes, We're just kicking off our Q3 stuff and I'm running (well responsible for?) a sort of sub team thing so I've got a bunch on my plate with that atm
[15:46:21] <cooperlees> dstufft: No worries, just offering, with the length of your replies sometimes I feel you'd be quicker just to edit the .rst directly :P
[15:47:11] <dstufft> The JSON api is kinda weird, I don't feel like I have great insight into what people actually want to use it for
[15:47:47] <cooperlees> dstufft: I think all we should do here is document what it does and maybe just make it a little friendlier to per index API for devpi
[15:53:30] <dstufft> it has occureed to me since that discussion that we probably don't want to use the JSON api for resolving dependencies fwiw
[15:53:38] <sumanah> also techalchemy ping in case you missed the announcement in https://github.com/pypa/packaging-problems/issues/367 that now's a good time to take a look at https://github.com/cooperlees/peps/blob/warehouse_json_api/pep-9999.rst
[15:54:25] <dstufft> and we should be leaning in on TUF and further securing things, not moving to another API for dep resolution that isn't protected by TUF
[15:55:03] <sumanah> dstufft: so I hear what you are saying - since the current state is "Warehouse's API gives the user sometimes inaccurate dependency metadata for a project's release" should we still fix that?
[15:55:28] <dstufft> which probably means making the JSON api (or any non simple api) to be acceptable for dep resoltuion it will probably involve integration with TUF in that API too
[15:55:44] <dstufft> sumanah: I mean, we should till probably do what we can to fix inaccurate metadata
[15:57:00] <dstufft> if you're doing dep resolution you should be using something that is or will be secured by TUF
[16:00:13] <sumanah> cooperlees: ^ you ok with this? if so I'm gonna add it to the non-goals section of the PEP
[16:00:43] <cooperlees> sumanah: Yup - totally agree with it, but, on the inverse, people do use it cause we don't have a good alternative
[16:01:24] <cooperlees> pradyunsg: If you're still around, what does pip use the JSON API for? (although dstufft probably knows this too)
[16:01:34] <sumanah> cooperlees: well we can say "be aware that, in the next few years, we will aim to provide a TUF-protected API for you to use instead"
[16:01:43] <sangy> I wonder if there's a way to proxy the trust from the TUF-secured metadata to the JSON api, although I'd suggest we keep that future tense
[16:02:26] <dstufft> the new resolver might, but I'm pretty sure it's trying to use lazy wheels instead
[16:02:51] <dalley> I don't think it uses it for anything, simple API only. Because pip can install packages from Pulp and we're not providing any APIs yet :)
[16:05:27] <dstufft> I'd have to think hard about it a bit to figure out if there's a good way to do that
[16:06:20] <sangy> dstufft: range request as in http range request? Wouldn't zip-compressed wheels get in the way of that as well?
[16:06:35] <sangy> cooperlees: it is confusing at first, but I promise everything there is there for a good reason :P
[16:07:08] <dstufft> sangy: yea, but compression inside of a zip is done file by file, and as a matter of implementation detail, the metadata directory is always at the end of the zip file
[16:07:11] <cooperlees> sangy: I’m sure. It’s just something I haven’t had to sit down and understand the full protocol
[16:07:38] <dstufft> so if you read the last N bytes, you'll probably get the metadata from the wheel
[16:08:07] <sangy> cooperlees: yeah fair enough. I'm also on the "we should abstract the confusing parts out" camp, which I hope TUF did a better job at
[16:09:09] <dstufft> probably the smallest change we could make that would let dep solvers work without downloading the entire file is to add another file that is just the extracted METADATA file
[16:09:13] <sangy> dstufft: aha so in reality these slices are pre-defined to some extent right? (i.e., the entries in the zipfiles are known at the time of compressing). I wonder if we could use a MHT in interesting ways to provide a path to show membership over the signed content
[16:09:50] <dstufft> and the pip / poetry / pipenv / etc could just download the metadata file during resolution
[16:12:24] <cooperlees> Will we have to keep non updated (I.e. no new releases) packages TUF files you to date somehow?
[16:12:29] <dstufft> but using those files pip etc can verify those were the correct files that came from PyPI
[16:13:27] <dstufft> there are some top level files that will change regularly that BS will have to start mirroring
[16:15:05] <dstufft> basically there's a tree of signatures (slightly more complex than that, but good enough) for every package that gets updated on every upload for that package, then there's another file that basically signs those signatures
[16:15:32] <dstufft> the point of that signature of signature thing is that thee signature on that is only valid for a short period of time
[16:15:58] <dstufft> so you can make sure that someone isn't serving you a valid, but years old snapshot of PyPI filled with only vulnerable software or w/e
[16:16:31] <dstufft> "short period of time" could bee hours or days or something, I don't know the exact timeframe we chose for pypi
[16:16:59] <dstufft> but whatever it is, it means that a BS mirror will have to update at least that often, or TUF will fail
[16:20:13] <dstufft> but from BS PoV, TUF will be: Extra files to mirror, and a requirement to keep the mirror reasonably up to datee
[16:20:59] <dstufft> unless BS wants to start validating the files it's getting from PyPI are valid, but that's not really *required*, since thee clients hitting BS should be doing that anyways
[16:21:28] <dstufft> but it could if it wanted to, the "win" in that case would be maybe protecting clients that don't validate TUF metadata themselves
[16:36:49] <cooperlees> All possible - Lets just try do the best for security and if people complain see what we can do
[16:55:06] <woodruffw> inre: specific timeframes: most of the TUF metadata being served will have a 24-hour expiry
[16:55:25] <woodruffw> (the exception is the root and other long-lived metadata, which will expire yearly)
[17:05:35] <sumanah> cooperlees: I lost track a bit of that discussion because I was in another call. Which bits are you going to follow up on and which will Donald?
[17:08:28] <cooperlees> I didn't see any actions for me other than finish the PEP with feedback I eventually get
[17:57:37] <dstufft> https://github.com/pypa/warehouse/issues/8254 wrote up a thing about the idea for improving the simple API for dep solvers without a ton of extra work
[22:25:02] <travis-ci> Change view : https://github.com/pypa/pip/compare/8bf5731b8490...77ead320b02d
[23:34:46] <travis-ci> Change view : https://github.com/pypa/pip/compare/c5e19c01c46e4402ce3f7b24f0b17e2bcf2f20bd...77ead320b02d7922030f3fd36a41b05fa686f1e8