Log file Viewer
#pypa-dev logs for Wednesday the 10th of June, 2020
[14:52:32] <sumanah> dstufft: do you have time anytime soon to talk with me about https://github.com/pypa/warehouse/issues/7730 the async API PEP? I'm thinking it's a part of a bigger fundable thing
[14:52:33] <sumanah> https://github.com/psf/fundable-packaging-improvements/blob/master/FUNDABLES.md#revamp-pypi-api
[14:54:12] <sumanah> dstufft: so in order to go after money for something I like to make sure that it's something the Python packaging community wants
[14:54:38] <sumanah> the Python packaging community wants it: there's already consensus among project maintainers, and if a PEP is involved it's been accepted.
[14:54:38] <sumanah> fundable: would happen much faster if the Packaging Working Group got funding to implement the work. (So, it has to be legal and physically possible.)
[14:55:34] <sumanah> dstufft: (also, FUNDABLES is a fun filename for me because it reminds me of Lunchables, which I did not eat growing up because my family was vegetarian, but it is still a fun word)
[14:57:12] <sumanah> dstufft: anyway. so the API stuff .... if a bunch of the API work that I listed in https://github.com/psf/fundable-packaging-improvements/blob/master/FUNDABLES.md#revamp-pypi-api depends on getting your PEP into shape + accepted, then I think I wanna move it to like a Blocked/Deferred issue or something, off the list
[14:58:30] <dstufft> Well I certainly want it :) I suspect the group of people who care one way or anothe what the uplod API is basically Warehouse + Twine? Maybe flit/setuptools/poetry if they're reimplementing twine
[14:59:02] <dstufft> I would say it depends on a PEP being written, it doesn't have to be *my* PEP, but someone would need to write one
[15:06:09] <sumanah> dstufft: the upload API -- you said "I'm going to start writing this up into a proper PEP, but I wanted to give a sort of brain dump on my thoughts here to see if anyone else had any thoughts on it." is that still the case?
[15:07:03] <dstufft> I haven't meaningfully started writing the PEP yet, if nobody else does it before I do I still plan to do it, but if someone else wants to run with it that's cool too
[15:07:20] <sumanah> many years ago (2018) I thought I was gonna write a PEP for PyPI upload spec https://github.com/pypa/packaging-problems/issues/128 ...
[15:07:23] <sumanah> "I would write it in collaboration with the Warehouse maintainers who will develop the reference implementation per pypa/warehouse/issues/284 , and consulting with the maintainers of packaging and distribution tools such as zest.releaser, flit, devpi, pypiserver, etc."
[15:07:39] <dstufft> those thoughts I brain dumped still fairly accurately reflect what I think it should look like
[15:11:17] <sumanah> ok, so I'm thinking that the projects to overhaul the *upload* API & improve upload functionality can be separated out from the many ideas to improve the various download APIs, add RSS feeds, JSON stuff like in https://github.com/pypa/warehouse/issues/284 , etc.
[15:11:32] <sumanah> dstufft: are those logically separate enough that they could be worked on independently?
[15:12:44] <dstufft> sumanah: Yea. The one real tie in those things have with each other is it would be nice if some care was made to try to make them feel more unified rather than the mess of different APIs we have now
[15:13:15] <dstufft> which probably boils down to whichever one that happens first gets to decide on some new cultural normals for our APIs
[17:20:39] <sumanah> techalchemy: dstufft: if you have a moment to cast your eyes over https://github.com/pypa/pypa.io/pull/54 I would appreciate that -- documenting the current PyPA membership procedure
[17:21:03] <sumanah> I think it is not controversial, and 6 of your colleagues have already approved it
[17:25:54] <dstufft> part of me wanted to propose ammending the bit about being welcome to use it to tell fascists to fuck off but people will probably yell a lot and it won't acutally stop them from using it so I'll just make them unwelcome in othe rways :p
[17:27:54] <dstufft> it's cool to se people with commit bits that I don't even know through what avenue they're a pypa member
[17:29:17] <dstufft> not sarcasticlly, i think it's legit cool I can't keep the entire set of people i my head anymore and that people are jumping in and I don't even know it. growth etc
[17:29:40] <techalchemy> dstufft, is there any actual barrier to producing a PEP for a json api beyond that it hasn't been done yet?
[17:30:02] <sumanah> techalchemy: "a measure of success" was not directed at you! it was at dstufft about how growth beyond known people is a measure of success
[17:30:39] <techalchemy> sumanah, for sure, i just saw dstufft talking and remembered I wanted to check in about the thing we all keep talking about but nobody wants to actually do
[17:30:43] <dstufft> techalchemy: not really. just requies someone doing the work & getting consensus and all that normal PEP stuff
[17:31:53] <techalchemy> great dstufft I'll keep mentioning that it can't be relied on since there's no pep & hope that someone with free time writes one
[17:32:05] <dstufft> if want to write the upload 2.0 PEP if I manage to find time and someone else doesn't beat me to it, and use that to define our conventions for what our next gen APIs look like.... but there's no reason it has to go first or anything
[17:32:39] <dstufft> my stage in writing that PEP was basically experimenting with tooling and such that we can use to manage those APIs in a holistic way
[17:33:51] <techalchemy> though part of me wants to ask if you plan to rewrite the metadata pep while you're at it
[17:37:40] <dstufft> I don't plan on rewriting the entire API at once, but rather my hope is that whoever does the first "next gen" api (and certainly my plan if it's me) is to define the broad strokes of what a warehouse API looks like, how do we define the routes and urls and methods and what not that is available, do we do a RESTish thing, or do we go all in on HATEOS, or do we pull in anothe rdirection and do something like GraphQL or RPC based.
[17:38:57] <dstufft> (I have my own opinions on all of those things!' but they're not mandatory opinons)
[17:39:32] <sumanah> techalchemy: in https://github.com/pypa/packaging-problems/issues/128 I laid out a bit of a plan for how I'd write a PEP mostly to document the current state of the upload API
[17:40:19] <sumanah> it might be useful to clone the 6 steps I listed there and customize that for the sequence of writing the PEP for the new JSON API
[18:02:53] <sumanah> dstufft: techalchemy: ok I think I wanna better understand the current situation.....
[18:03:33] <sumanah> I think some developers of client apps have basically said "what is the risk of depending on the current Warehouse JSON API?" and have heard confusing things back from the main Warehouse developers
[18:04:12] <sumanah> like "we're gonna change the JSON API at some point in the future but...." and then have wildly different estimates of how long it will be till then
[18:04:44] <sumanah> and Warehouse maintainers haven't had the resources, the dedicated time, to collaborate with contributors who want to help develop the new JSON API, review branches, etc
[18:06:30] <sumanah> so this has left everyone in a state of uncertainty, so that developers of clients of Warehouse's current API have a hard time making engineering tradeoff decisions ("it's ok to write this and know we'll have to rewrite it in a year")
[18:06:56] <techalchemy> dstufft has actually been a major contributor to the confusion and i've likely amplified it because he told me not to rely on the json api
[18:07:44] <techalchemy> although i suspect he didn't realize the alternative was that i was going to use pip internals which are substantially less stable and also not defined in a pep
[18:08:52] <dstufft> if you want to work with things that aren't warehouse, you can't rely on the JSON API really, because it's non standard and support for it varies
[18:09:33] <techalchemy> yeah it's pretty nuanced, other indexes would have to implement the entire data model of warehouse
[18:10:36] <sumanah> ok so techalchemy when you started today by talking about "producing a PEP for a json api" did you mean writing a standard to describe the current Warehouse JSON API so that other indexes could also implement that standard?
[18:11:20] <sumanah> right. and that could be v2, and then the next generation could be v3, or something
[18:11:46] <dstufft> I think the current JSON API is also not a great API in general, but it is there and it exists so it's not wrong to document that as a standard for now
[18:12:46] <sumanah> techalchemy: so I am a ridiculous person in many ways but also my spouse wrote a book on API design and I read it
[18:13:26] <sumanah> he does practically all the cooking in our household so if he tells me I should version my APIs then isn't that the least that I could do to repay him?
[18:14:19] <dstufft> anyways, when I say you cant rely on it, it mostly comes down to you can't rely on things that aren't PyPI having the JSON API, or if they have it, having it in the same location as PyPI does with the same weird implementation details
[18:14:20] <sumanah> https://www.crummy.com/writing/ RESTful Web Services, and the sequel RESTful Web APIs (the former is now available to read for free online)
[18:15:24] <techalchemy> sumanah, oh see what he's doing, tease you with the old version so you just can't wait to pay for the new one!
[18:16:03] <dstufft> also I've slept like 4 hours in the last 50? 60? hours so I might not be entirely lucid
[18:16:31] <sumanah> dstufft: I could swear we versioned some part of the Warehouse API with a /v1/ in the URL. am I hallucinating? can't find it in the docs
[18:16:50] <dstufft> techalchemy: how we version our APIs is something that a standards document would do well to document
[18:18:07] <sumanah> I was foolish not to make this happen back when I was PMing the production deployment. crap
[18:20:27] <sumanah> di_codes: hey so I hear you write PEPs sometimes. about packaging. like, how many hours did it take you to write the first draft of each of the PEPs you've authored?
[18:22:20] <dstufft> that's about what PEP 503 took, and there wasn't much discussion other than some minor tweaks
[18:22:40] <dstufft> because it wasn't doing anything new, just documenting what exists and declaring it a standard
[18:22:43] <sumanah> yeah. techalchemy: dstufft: it sounds like a somewhat easier next step is for someone to write a PEP to describe the Warehouse JSON API as it exists right now, for the purpose of locking it in as a guarantee for consumers, and for the purpose of helping other indexes adopt it. And then there's a much more ambitious second PEP that would Architect The New Next-Gen JSON API
[18:24:45] <sumanah> techalchemy: I am NOT asking you to volunteer, but does this set of next steps feel right to you?
[18:25:43] <dstufft> I wouldn't say for sure that one path is better or worse (documenting current, waiting on next gen, or jumping straight to next gen). They're both reasonable paths to get to a better word, one just has a better partial? interim? result but it ultimately comes down to what we have people willing to work on
[18:26:29] <techalchemy> i will say that a draft pep means pip can move toward supporting something which will force adoption
[18:26:39] <sumanah> dstufft: right, it feels like a staffing/project management/etc. question rather than an engineering question
[18:27:16] <techalchemy> will force other index server implementations to start considering building support
[18:28:02] <sumanah> techalchemy: ok so this is an argument for getting momentum behind the "document current JSON API" PEP and/or for getting momentum behind writing a next-gen API PEP. correct?
[18:29:00] <techalchemy> right, either one is ok, I probably lean a bit in favor of documenting the existing one first just because, well, it exists, so is usable now if there is a pep
[18:29:46] <techalchemy> dstufft, well it has all kinds of metadata that you otherwise need to download packages to acquire
[18:30:33] <sumanah> as dstufft said, this all depends on what people will actually commit to work on
[18:30:35] <dstufft> it also requires making a signifcant amount more HTTP requests to actually get that information
[18:30:47] <techalchemy> i actually have code that resolves packages using the json api that I use supplementally but yeah, it's not at all fast
[18:30:54] <sumanah> I have a wacky idea which is that "document the current Warehouse JSON API in PEP form" is the kind of task that SOMEONE ELSE who is not one of us three could do. Like Hugo or Sviatoslav
[18:31:26] <techalchemy> i haven't asyncio'd it since it still supports python 2 but i would imagine if you like, cached the results you might get some speedup
[18:31:52] <techalchemy> also maybe if you didn't build massive nested object sets out of the results as i am doing, it could be faster
[18:34:09] <dstufft> but it's important I think to, when documenting it, figure out what we expect people to use it for and make sure it's actually usable for that
[18:34:22] <techalchemy> yeah i actually pulled all of that and made a partial copy of pypi locally (sorry for the massive number of connections) and was confused by that
[18:34:42] <sumanah> like, *as* we document it, that's an opportunity to find out what people expect, etc., but we need to not block the documenting work on the fixing work
[18:35:01] <techalchemy> by which i mean i pulled all the data but only implemented part of the data model
[18:35:40] <techalchemy> so which file do you pick for parsing dependencies? the first wheel that gets uploaded?'
[18:37:39] <dstufft> sumanah: mostly I mean that in standardizing it, we need to make sure we're standardizing something that makes sense for the use cases we expect it to be used. An example (really the main example) is currently dependencies are presented only at the release level in the API, but in python packaging we actyally do dependency information at the file level... just not in warehouse. So even if Warehouse doesn't *today* do it correctly, the PEP should
[18:38:23] <sumanah> I appreciate both of you going into painstaking detail in response to my super pedantic questions dstufft techalchemy
[18:38:30] <dstufft> but if we say expect pip to use this to resolve dependencies, it doesn't actually help to standardize something that has the wrong shape for pip to do that
[18:39:11] <dstufft> which is also problematic, if you upload a sdist first no dependency information is encoded
[18:40:57] <dstufft> in part because upload already takens forever and the more stuff we do the longer it takes
[18:41:14] <dstufft> I think our timeout on upload is multiple minutes, because that's how nlong it takes sometimes
[18:41:39] <techalchemy> i guess i'd really just propose populating the json api asynchronously for any wheel that doesn't provide metadata
[18:43:11] <dstufft> anyways, there's a lot of problems with the JSON API, all ofwhich are solvable, and can be solved piecemeal over time
[18:44:52] <dstufft> I suspect the hypothetical JSON API PEP would be a little bit more effort than PEP 503 was
[18:44:56] <techalchemy> the current code is incredibly easy to navigate, basically anyone with python web development experience should have no issues with it
[18:45:46] <dstufft> largely because whiel the simple api is weird and has problems with extending it, we had a very well defined target consumer for it, so we knew exactly what we needed to standardize
[18:46:30] <dstufft> the JSON API is a bit more vague, but probably the main driving force behind a desire to standardize it, is as either a repkacement or a suppliment to the simple api
[18:47:13] <techalchemy> the simple api is easy to produce, the main issue with it is that consuming it sucks
[18:47:15] <sumanah> hey so I want to head off a bit more of the discussion about the next gen JSON API and concentrate on the next step of "document the current API"
[18:47:52] <sumanah> I'm gonna repeat: I have a wacky idea which is that "document the current Warehouse JSON API in PEP form" is the kind of task that SOMEONE ELSE who is not one of us three could do. Like Hugo or Sviatoslav
[18:49:17] <dstufft> sumanah: you are correct with just the caveat I mentioned above that we might want to tweak the JSON API a bit just to make it suitable the primary use case I *think* people want it for
[18:49:58] <sumanah> ok. I have a phone call in 10 min, I'm gonna lay out some steps, dstufft techalchemy if you could help me I'd really appreciate it
[18:50:46] <sumanah> step 1, techalchemy or dstufft, file a bug about the tweak you just mentioned to the existing API
[18:51:20] <sumanah> step 2, talk with the volunteer about a reasonable deadline and interim checkins to write the first draft, and get the thread on Discourse started
[18:52:22] <sumanah> step 4 [and there's multiple back-and-forths here] get that PEP written, discussed, and accepted. now we have a standard that all the clients can feel guaranteed about using and that other indexes can implement
[18:53:34] <sumanah> real step 4 is: get some dedicated volunteer time committed, from a Warehouse expert, to write the next-gen JSON API PEP. because that is gonna be a bear, and I'm pretty wary of getting a grant for it because it's way more research than development
[18:53:53] <techalchemy> even if it's not accepted, as long as it makes it to draft status it can generally start being useful
[18:53:54] <sumanah> and then get that discussed and accepted. then step 5: get money to implement it
[18:54:07] <techalchemy> there's a bunch of other peps like that that have never been accepted but everyone uses anyway
[18:54:55] <sumanah> anyway, techalchemy dstufft does that sound like a good sequence? given that both of you already have commitments at like 170% of your capacity
[18:56:22] <sumanah> great. which of you can file that "add a field or two to a dict inside of alist" bug to improve the Warehouse API to help the dependency info stuff?
[21:44:43] <sumanah> dstufft: techalchemy: I filed https://github.com/pypa/warehouse/issues/8090 "Fix how Warehouse stores dependency (per-file, not per-release)". Please amend if I got things wrong
[22:11:54] <sumanah> ok, and now I have filed a public issue requesting that someone write the PEP to document the current state of things https://github.com/pypa/packaging-problems/issues/367 dstufft techalchemy