pmxbot IRC Log Viewer

[02:29:08] <asmacdo> cooperlees: ^

[15:28:33] <asmacdo> dstufft, cooperlees, the POC for the hypermedia api is up (tests are passing). i'm around, PM's are welcome. https://git.io/vhLCr

[15:33:16] <cooperlees> asmacdo: nice work. Will have a read today!

[16:06:13] <dwighthubbard> asmacdo that looks like a lot more fleshed out than last time I looked at it.

[16:37:34] <cooperlees> asmacdo has been hard at work :)

[16:38:57] <cooperlees> dstufft: Around today? :D

[16:40:49] <asmacdo> dwighthubbard: thanks! As this moves forward, I could use some help developing specific use cases and workflows to enable them. https://pad.sfconservancy.org/p/hypermedia_api_design

[16:44:09] <dstufft> cooperlees: ish, my computer is randomly going to sleep every like 5 minutes, so trying to figure out wtf is going on

[16:57:01] <techalchemy> dstufft: try pouring coffee in its mouth

[16:57:04] <cooperlees> LoL - what a lazy computer :) Would love to know you thoughts and what our chances are of merging https://github.com/pypa/pip/pull/5400

[16:57:47] <techalchemy> asmacdo: when you say PoC what are you implementing this for, pypi?

[16:58:54] <cooperlees> techalchemy: Yes, we want to get rid of the XMLRPC API. asmacdo, bizhang and I worked on a POC @ PyCon

[16:59:40] <techalchemy> yes i know about the xlmrpc plans and I saw the issue from 5 years ago on github but I didn't see any discussion of what specific features are going to be implemented here

[17:00:03] <techalchemy> so I'm not sure where that is happening but that conversation is one I have some thoughts about

[17:01:18] <asmacdo> we pulled a lot of ideas from https://github.com/pypa/warehouse/issues/284, and also got a fair amount of in-person feedback. the design document is https://pad.sfconservancy.org/p/hypermedia_api_design

[17:01:36] <cooperlees> di and dstufft were pretty verbose on https://github.com/pypa/warehouse/issues/284 and asmacdo spells out a lot here: https://pad.sfconservancy.org/p/hypermedia_api_design

[17:04:30] <techalchemy> right, I'd be happy to provide feedback to you guys, the last thing I read was that it didn't make sense to make a proof of concept yet so I didn't realize it was time to be giving feedback or that designs were being nailed down

[17:05:37] <techalchemy> if you are open to feedback that is

[17:05:39] <asmacdo> techalchemy: no worries. feedback is welcome, even if you completely hate the design :)

[17:05:44] <techalchemy> i won't

[17:05:52] <techalchemy> I only care about incredibly specific things

[17:06:17] <asmacdo> ah, for specific stuff, maybe the etherpad is best. ive got a TODO going with use cases

[17:06:30] <cooperlees> PyPI just wants a more CDN friendly API

[17:06:39] <cooperlees> XMLRPC is not

[17:06:59] <cooperlees> bandersnatch is a major user, I have the ability to change that :)

[17:07:11] <cooperlees> So is pip, but, we can change that too :)

[17:08:05] <techalchemy> I can help with implementation of the features I want, most likely

[17:08:11] <techalchemy> our use case is super specific

[17:08:16] <cooperlees> shoot

[17:08:19] <cooperlees> I'm interested now :)

[17:08:34] <techalchemy> i want dependency listings if packages have wheels available

[17:09:12] <techalchemy> i want package version listings by python version (major/minor/patch) if wheels are available

[17:09:21] <techalchemy> or project version in api lingo

[17:09:23] <techalchemy> i guess

[17:09:33] <cooperlees> That metadata is super best effort

[17:09:37] <techalchemy> yep

[17:09:42] <techalchemy> thats ok

[17:10:03] <cooperlees> That's why I want https://github.com/pypa/pip/pull/5400

[17:10:10] <cooperlees> So I can use pip to download and then tell me :)

[17:10:12] <techalchemy> it's best effort whether I do it on a user's machine or on a server :p

[17:10:16] <cooperlees> from my local mirror

[17:10:59] <asmacdo> techalchemy: I'm also interested in improving the dependency metadata. I have a use case to retrieve deps without having to crack open every wheel.

[17:11:16] <techalchemy> I am the primary maintainer of pipenv

[17:11:58] <techalchemy> so dependency metadata (and all package related metadata) being accessible via API is really important

[17:12:45] <asmacdo> techalchemy: The API can be improved there, but I think that the metadata itself is a problem

[17:12:51] <techalchemy> well yeah

[17:13:00] <techalchemy> it's a two pronged issue

[17:13:20] <cooperlees> dstufft: I dragged pjameson so we can all talk about PR 5400 :)

[17:13:20] <techalchemy> I figure the band-aid on the current situation is a much faster approach

[17:14:03] <techalchemy> we use piptools on the backend and we actually vendor and patch pip10 now quite heavily internally and we use our own wheel / build caches

[17:14:15] <techalchemy> and we actually just have cached dependency graphs

[17:14:23] <techalchemy> or we will, next release

[17:14:29] <cooperlees> :O

[17:14:43] <techalchemy> locally cached

[17:14:45] <techalchemy> *

[17:14:57] <cooperlees> techalchemy: Have you talked with the condo guys? They are super keen to help with all the dep calculation etc. ?

[17:15:10] <asmacdo> techalchemy: do you happen to know off hand which fields of which models you need?

[17:15:20] <techalchemy> yeah the problem is that implementing an actual resolver is basically impossible

[17:15:24] <techalchemy> like a real sat solver

[17:15:30] <techalchemy> unless you have a billion years

[17:15:41] <asmacdo> i have fridays ;)

[17:15:43] <techalchemy> lol

[17:15:59] <techalchemy> i mean a billion years per dependency resolution you attempt

[17:16:12] <cooperlees> asmacdo will change the world on Fridays

[17:16:18] <asmacdo> huzzah!

[17:17:24] <asmacdo> techalchemy: there are a few nasty db calls i want to do, and i am hopefully that good use of caching will make up for it

[17:17:47] <cooperlees> Postgresql == scale

[17:17:51] <techalchemy> fields we need would be like, install_requires, extras_require (if extras), markers, specifier / specifierset, sha256 / links and refs to the other uploads of the same version of the project but sdist/wheel for other python/etc

[17:24:45] <asmacdo> techalchemy: i'll need to dig into how `Release.requires` works under the hood to see if it is reasonable to expose. In a perfect world it sounds like you just need /projects/{name}/releases/{version}/ which returns a link to /projects/{name}/releases/{version}/files/

[17:31:28] <asmacdo> brainstorming here, but maybe it would be worth separating this into a new endpoint, like /dependencies/{name}/{version}/

[17:31:41] <techalchemy> possibly

[17:31:58] <techalchemy> it's somewhat complicated because of the various ways dependencies can be exposed

[17:32:04] <techalchemy> and the various formats artifacts come in

[17:33:12] <asmacdo> heres a silly question, for a single Release, can we assume that the dependencies of each File are all the same?

[17:33:23] <techalchemy> nope

[17:33:30] <asmacdo> ouch

[17:33:40] <techalchemy> technically you can't even assume that they're correct

[17:33:41] <techalchemy> lol

[17:33:45] <techalchemy> but we have to start somewhere

[17:35:16] <asmacdo> OK. to start with, how about /dependencies/{filename}/ , response would link to /projects/{name}/releases/{version}/files/

[17:36:26] <asmacdo> unfortunately though, i *think* any `requires` metadata is stored on the release, not the file. so we might be talking about cracking open a tarball.

[17:40:35] <techalchemy> the tarball release metadata is mostly useless

[17:40:56] <techalchemy> sdist 'requires' metadata needs to be supplemented by setup.py no matter what

[17:41:04] <techalchemy> so it's totally non-deterministic

[17:41:16] <techalchemy> might as well pretend it's not there

[17:42:46] <dstufft> I think maybe I got my computer to not be annoying

[17:42:52] <dstufft> so uh

[17:43:05] <dstufft> New API may or may not be what you want for dep resolving

[17:43:24] <dstufft> replacing the /simple/ API is more complex than replacing XMLRPC and the hacke dup JSON api

[17:43:43] <dstufft> because we're going to want to layer TUF ontop of it, which comes with it's own constraints

[17:44:17] <dstufft> but we don't gaf about TUF on the general purpose API

[17:49:12] <techalchemy> ok

[17:49:47] <techalchemy> i have no issue with that if you want to keep this fuckery out of the new api

[17:49:55] <techalchemy> other than that we don't do anything special with /simple/

[17:50:02] <techalchemy> besides look for packages :p

[17:50:59] <cooperlees> that's what it's simply for :D

[17:56:54] <dstufft> techalchemy: the thing about dep solving and downloading packages and suff is the intent is you shouldn't have to trust PyPI the service, there should be strong cryptographic signatures backing that

[17:57:09] <dstufft> but that's overhead that doesn't matter for a lot of use cases

[18:00:21] <techalchemy> yeah

[18:00:45] <techalchemy> well in practical terms at the end of the day you have to trust whoever wrote the the code to tell you waht it needs

[18:00:55] <techalchemy> like they are still writing it down somewhere

[18:01:10] <techalchemy> just following that chain shouldn't be as complicated as it is

[18:11:02] <cooperlees> LoL

[18:11:25] <cooperlees> https://www.irccloud.com/pastebin/FfIP87Ji/

[18:15:51] <dstufft> cooperlees: turns out being added as a list owner dind't subscribe me

[18:16:09] <cooperlees> Makes sense

[18:16:17] <dstufft> someone mentioned the thread on the list and I'm like... uh what? People are talking there??

[18:16:41] <cooperlees> There is people talking on there? Shit. I need to join too

[18:16:41] <cooperlees> haha

[18:17:05] <cooperlees> I've only approved 20 people

[18:17:27] <cooperlees> I can accept myself! w00t

[18:17:36] <techalchemy> we really should add our other maintainer at some point

[18:18:03] <cooperlees> techalchemy: Sorry - add what to who?

[18:18:23] <techalchemy> the other guy who maintains pipenv with me

[18:18:38] <techalchemy> to the mailing list and or pypa/pipenv-committers etc

[18:18:43] <cooperlees> O please

[18:19:00] <cooperlees> lol

[18:19:34] <cooperlees> seems di is going to be the man!

[18:19:39] <cooperlees> gg di_codes

[18:30:12] <cooperlees> Can anyone remember the gentlemen's name from GitHub @ PyCon?

[18:30:24] <techalchemy> definitely

[18:30:35] <dstufft> Bryan uh

[18:30:35] <techalchemy> oh you mean a github staff member

[18:30:53] <cooperlees> yeah

[18:31:20] <dstufft> Bryan Clark

[18:32:18] <dstufft> PM'd you his email

[18:32:19] <dstufft> at least I think that's who it was

[18:32:29] <dstufft> that's the person I have in this email thread who said they were coming lol

[18:32:33] <cooperlees> Yeah - thanks - Pinged him on Linked In :)

[20:19:32] <techalchemy> woops

[20:19:48] <techalchemy> forked something into pypa by accident :op

[21:30:52] <cooperlees> haah - I've almost done that

[21:32:35] <cooperlees> https://www.irccloud.com/pastebin/YKX90aQy/

[21:32:50] <cooperlees> I dislike how this client does that

[21:33:02] <cooperlees> dstufft: had any time for that pip PR or ours? https://github.com/pypa/pip/pull/5400

[21:33:02] <cooperlees> I really just want to know if it's going to be blocked or not. If not I'll fork for now and continue on internally with that.

[22:32:07] <njs> cooperlees: why do we have Yet Another pypa list

[22:33:15] <cooperlees> So active committers can have specific discussions that don't need to go to the world and are recorded - This was discussed @ PyCon - I thought you were there.

[22:47:08] <njs> cooperlees: that vaguely rings a bell

[22:48:22] <njs> cooperlees: but IMO it's not a great idea, except maybe as a short-term thing; we already have a lot of fragmentation of communication channels, and a lot of frustration comes down to people not feeling like it's possible to keep track of what's happening or what decisions are made – makes it hard to have a shared vision and move in the same direction

[22:50:01] <njs> if it's like, used for a focused discussion on how to refine and streamline our communications then I'm all for it, but it's frustrating to realize "oh surprise there's another list and discussions happening that you care about but you didn't know about it or have a chance to participate"

[22:52:49] <cooperlees> So these conversations were just happening in mail threads before this list.

[22:52:54] <cooperlees> So it's better than that.

[23:14:16] <dstufft> njs: it's basically "we need to figure out communication and governance and what that means for us, and probably a random thread with 20 people in CC's that Donald started isn't the best way to structure this until we figure out the long term real solution"

[23:16:22] <njs> I agree, the private email thread with lots of CCs is even worse on all the dimensions I'm talking about :-)

Log file Viewer

Help | Karma | Search:

#pypa-dev logs for Friday the 25th of May, 2018