pmxbot IRC Log Viewer

[13:14:58] <sumanah> Reminder that the next livechat is in a little under 2 hours, 1500 UTC.

[15:00:05] <sumanah> Welcome to the Warehouse/PyPI livechat, the third of four during this beta!

[15:00:18] <sumanah> Warehouse, the codebase behind the new PyPI, is now in beta https://pypi.org, and I'm one of the people working on it. I'm Sumana Harihareswara, project manager. We also have Dustin Ingram - introduce yourself, Dustin?

[15:00:52] <sumanah> We intend to make pypi.python.org redirect to Warehouse on Monday, April 16th, and then fully shut down the legacy PyPI site pypi.python.org on Monday, April 30th.

[15:01:33] <sumanah> Our developer roadmap: https://wiki.python.org/psf/WarehouseRoadmap

[15:01:35] <di_codes> hi everyone! I’m Dustin (@di), a PyPI maintainer/administrator and PyPA enthusiast

[15:01:47] <sumanah> We want to hear from you about what you like & what doesn't work for you in the new interface. https://pyfound.blogspot.com/2018/03/warehouse-all-new-pypi-is-now-in-beta.html#workflows has a list of stuff we especially want people to test.

[15:02:10] <sumanah> And, as a reminder, participating in this livechat means you agree to abide by the PyPA code of conduct: https://www.pypa.io/en/latest/code-of-conduct/

[15:02:10] <sumanah> So let's start with introductions. Who all is here for the livechat? Wave your hand: )

[15:02:26] <sumanah> \o

[15:03:39] <sumanah> oh no prob mgedmin

[15:03:46] <sumanah> [Also: we've made it to beta thanks to funding from Mozilla's Open Source Support Program https://pyfound.blogspot.com/2017/11/the-psf-awarded-moss-grant-pypi.html . Thanks, Mozilla! And thanks Python Software Foundation for supporting this work.]

[15:04:20] <sumanah> di_codes: today's really the day I've started publicizing the plan we agreed on yesterday, where April 16th is the redirect date and April 30th is the legacy shutdown date

[15:04:36] <sumanah> di_codes: I checked in with Ernest about it yesterday night and he was cool with it, so, yay :)

[15:04:57] <sumanah> !logs

[15:04:57] <pmxbot> http://kafka.dcpython.org/channel/pypa-dev

[15:05:11] <mgedmin> when is the right date to start redirecting automated tools to the new pypi?

[15:05:19] <sumanah> hi asx_ & pradyunsg - welcome.

[15:05:33] <asx_> Hi this is anurag

[15:05:37] <sumanah> mgedmin: I'll let di_codes speak to that.

[15:05:39] <pradyunsg> Hi sumanah! Hope it's going well. =)

[15:05:41] <mgedmin> (I recall hearing that it doesn't have the capacity yet to handle production traffic levels)

[15:05:41] <di_codes> mgedmin: pretty much now!

[15:05:43] <pradyunsg> !logs

[15:05:43] <pmxbot> http://kafka.dcpython.org/channel/pypa-dev

[15:05:45] <sumanah> Hi asx_ and thanks for contributing to Warehouse

[15:06:10] <di_codes> “production traffic” really means “traffic from pip” (with the exception of XML-RPC traffic)

[15:06:15] <asx_> Thanks for the help @sumanah

[15:06:26] <sumanah> di_codes: so I've been advising folks to wait till we say on pypi-announce "please update your automated tools to point to pypi.org" and I think we should say that on April 9th or 16th since we haven't said it yet

[15:06:27] <di_codes> but, we do have that capacity now!

[15:07:54] <mgedmin> cool!

[15:07:56] <di_codes> yeah I mean, unless your automated tool is generating a ton of requests, it’s _probably_ ok to do it now

[15:09:06] <asx_> !

[15:09:14] <di_codes> mgedmin: mind if I ask what tools you have that depend on PyPI?

[15:09:35] <sumanah> asx_: you're welcome. :) how did you hear about PyPI and Warehouse?

[15:09:55] <mgedmin> I have a set of ancient hacky Python scripts that update https://zope3.pov.lt/py3/ daily

[15:10:33] <asx_> I knew about PYPI before. I heard about Warehouse from @EWDurbin

[15:10:33] <mgedmin> this is the script that accesses PyPI's API: https://github.com/mgedmin/ztk-py3-status/blob/master/get_pypi_status.py

[15:12:05] <mgedmin> hm, does warehouse support the JSON API (https://wiki.python.org/moin/PyPIJSON)?

[15:12:18] <di_codes> mgedmin: yup!

[15:12:24] <sumanah> mgedmin: https://warehouse.readthedocs.io/api-reference/ might help!

[15:12:33] <alanbato> Hello everyone :)

[15:12:35] <mgedmin> thanks!

[15:12:57] <sumanah> *wave* alanbato

[15:13:06] <sumanah> !logs

[15:13:06] <pmxbot> http://kafka.dcpython.org/channel/pypa-dev

[15:13:49] <sumanah> I'm tweeting some screenshots at https://twitter.com/hashtag/newpypi?f=tweets&vertical=default&src=hash

[15:14:39] <sumanah> mgedmin: asx_ alanbato -- remind me, are any of you coming to PyCon North America in May, or EuroPython in July?

[15:15:09] <sumanah> We're going to hold some packaging sprints https://wiki.python.org/psf/PackagingSprints at those conferences

[15:15:56] <asx_> I will be at PYCON. I am from Cleveland!!

[15:16:23] <sumanah> asx_: oh awesome! Have you registered yet?

[15:16:50] <asx_> Yes. I dont know about my sprint commitment yet.

[15:16:58] <sumanah> asx_: Hope you can make it.

[15:17:04] <sumanah> di_codes will be speaking about Warehouse: https://us.pycon.org/2018/schedule/presentation/148/

[15:17:32] <alanbato> sumanah: I wish! Sadly, I won't be able to make it this time. However, I'll try to be active during the sprints remotely :)

[15:17:45] <sumanah> understood alanbato!

[15:18:13] <sumanah> asx_: alanbato: I'd love your feedback on https://github.com/pypa/warehouse/issues/2794 -- what do you think we ought to add to https://warehouse.readthedocs.io/application/ ?

[15:18:25] <sumanah> to help newer Warehouse developers get up to speed faster?

[15:20:14] <mgedmin> does anyone here have working wiki.python.org credentials?

[15:20:26] <mgedmin> my browser offers to autocomplete something that the server then rejects

[15:20:48] <alanbato> sumanah: I'm reading the docs and the GH issue, give me a sec :)

[15:20:55] <di_codes> asx_: are you a Warehouse contributor I should know? sorry, don’t have a great mapping between IRC nicks and github handles

[15:20:55] <mgedmin> I think it would be nice if https://wiki.python.org/moin/PyPIJSON linked to https://warehouse.readthedocs.io/api-reference/json for a more-recent API doc version

[15:20:56] <sumanah> sure alanbato :)

[15:21:17] <sumanah> mgedmin: I can edit some of wiki.python.org but not sure about that page. Will try

[15:21:22] <asx_> di_codes: I am saxenanurag on gh

[15:21:50] <mgedmin> also https://wiki.python.org/moin/PyPIXmlRpc could link to https://warehouse.readthedocs.io/api-reference/xml-rpc/

[15:21:55] <di_codes> asx_: oh awesome! howdy!

[15:22:29] <di_codes> mgedmin: good call… those wiki pages look extremely old and dusty

[15:22:33] <asx_> @sumanah: I think the some content from application should move to getting started page.

[15:22:51] <mgedmin> oh yeah, dust aplenty

[15:22:54] <sumanah> mgedmin: I am logged in but don't see an Edit link/button on https://wiki.python.org/moin/PyPIJSON

[15:22:57] <mgedmin> but people might have them bookmarked

[15:23:08] <sumanah> so I think I don't have the relevant privileges.

[15:23:17] <mgedmin> oh, I see it's an immutable page!

[15:23:31] <mgedmin> I'm not sure what's the process for changing those

[15:23:33] <asx_> @sumanah: especially the section on what the codebase is made of.

[15:23:39] <mgedmin> maybe I ought to file a bug on bugs.python.org

[15:24:16] <sumanah> asx_: What if the link to the application overview were higher up? Right now it's in https://warehouse.readthedocs.io/development/getting-started/#learn-about-warehouse-and-packaging

[15:25:04] <sumanah> asx_: I think "getting started" should probably be basically a long recipe that the reader should follow :) and prose/diagrams like the application overview should be on a different page. But I am happy to move the link so the reader sees it much earlier in the recipe.

[15:25:05] <asx_> @sumanah: Maybe the warehouse code base should be on the top in the table of contents.

[15:25:11] <sumanah> asx_: maybe!

[15:25:24] <sumanah> mgedmin: I think that's a good call

[15:26:58] <sumanah> di_codes: you said "unless your automated tool is generating a ton of requests, it’s _probably_ ok to do it now" -- if we could make a litmus test and publicize that, I think we ought to do that this week. "Under n requests/minute? Move it now. Over? Move it on the 16th."

[15:27:59] <asx_> I have a docker related question. What is the best way to make sure docker is not eating up all your resources? Mac user. 8GB ram.

[15:28:06] <asx_> using a SSD

[15:28:21] <asx_> docker noob

[15:29:45] <dstufft> the best litmus test is really something like

[15:29:53] <dstufft> are you going to yell at us if we go down

[15:30:00] <sumanah> This is dstufft, Donald Stufft, a founder of Warehouse and one of its maintainers

[15:30:04] <dstufft> we have the capacity for traffic now

[15:30:12] <dstufft> so it's really just what we promise in terms of uptime

[15:30:17] <dstufft> and weird bugs and such

[15:30:19] <sumanah> pradyunsg is a pip maintainer, and alanbato and asx_ are contributors to Warehouse

[15:30:44] <sumanah> dstufft: I recognize the truth in what you say

[15:30:49] <Wooble> dstufft: (the answer is always going to be "yes" even if it's clearly not your fault and if you told people not to use it, so...)

[15:31:05] <sumanah> dstufft: so perhaps the new question is: what is the actual promise we will be making on April 16th?

[15:31:15] <dstufft> Wooble: my body is ready

[15:31:34] <sumanah> dstufft: and I still think that the "n requests per [time period]" is a useful metric to be able to give hobbyists who have tools that hit our APIs

[15:31:39] <dstufft> sumanah: We'll care more about not taking the site down

[15:31:47] <dstufft> is basically the promise, as far as APIs is concerned

[15:31:59] <dstufft> (since I think all the APIs are feature complete?)

[15:32:04] <sumanah> dstufft: ok, yes

[15:32:11] <dstufft> if they're not, the other promise is we'll have things implemented :)

[15:32:14] <sumanah> I'll double check that, but basically yes

[15:32:16] <Wooble> dstufft: (also people will yell at you if it's working perfectly fine, for no apparent reason at all, and the community appreciates that you put up with that)

[15:32:20] <alanbato> sumanah: I think a Glossary regarding terms like "project, distribution, maintainer" could be helpful to clear confusions between similar concepts and synonims. Both in the codebase and the docs

[15:32:31] <sumanah> alanbato: ok, thanks, I will come back to this

[15:32:35] <dstufft> sumanah: to be clear, not opposed to a req/s thing

[15:32:42] <dstufft> just saying, it's going to be largely arbitrary

[15:32:49] <dstufft> so whatever folks want to suggest is fine

[15:32:54] <sumanah> dstufft: so I am in favor of nailing things down a bit and being able to say to people, here's what we promise (even if the promise is more about caring than about specific uptime numbers)

[15:33:44] <di_codes> sumanah: how about “we promise we tried out best”? :wink:

[15:33:48] <di_codes> our*

[15:33:49] <dstufft> sumanah: Yea, messaging SLA is hard here, because we don't really have one in general

[15:34:01] <sumanah> yes I know -- I'm trying to talk about that without actually being flippant

[15:35:06] <sumanah> on requests/second, so I could pull a number out of nowhere but I presume it would be wrong, so with that in mind, what if the advice were like this?

[15:36:13] <sumanah> If your tool hits pypi.python.org more often than once a second, and/or you care a lot about service interruption, you should go ahead and migrate between now and April 16th

[15:36:59] <sumanah> and if we want to add some specific different advice about XML-RPC we can do that

[15:37:55] <dstufft> sumanah: is the conditional on that inverted

[15:37:57] <di_codes> sumanah: don’t we want to say the opposite?

[15:38:11] <alanbato> sumanah: Also, I think it would be valuable to include architecture beyond the codebase, and include things like design preferences for tests, how the docker containers are setup right now, describe with detail what each `make` command does, and other "development" parts of the workflow for completeness.

[15:38:15] <sumanah> RIGHT, less often than once a second, that's what I meant

[15:38:34] <sumanah> alanbato: cool! could you add both your suggestions -- that and the glossary -- to the GitHub issue?

[15:39:09] <alanbato> Of course :)

[15:40:07] <sumanah> If your tool hits pypi.python.org less often than once a second, and/or you are fine with a little service interruption as we work out beta stuff, you should go ahead and migrate between now and April 16th

[15:40:10] <sumanah> like that?

[15:40:40] <di_codes> sounds good to me

[15:41:15] <sumanah> once we wordsmith this, I suggest I update https://pypi.org/help/#beta and send a heads-up to distutils-sig

[15:43:15] <dstufft> sumanah: yea seems reasonable

[15:44:13] <sumanah> and then when we launch/redirect on April 16th, we'll remove https://pypi.org/help/#beta and add something to https://pypi.org/help/#availability basically saying: there is no specific SLA for PyPI. Volunteers get paged if the site goes down, and we've never had an outage lasting longer than [8 hours?? not sure].

[15:44:29] <sumanah> ok not NEVER

[15:44:33] <dstufft> Depends on your definition of outage

[15:44:35] <sumanah> right

[15:44:48] <sumanah> we can't give actual predictions or promises but we can give people a little data

[15:44:50] <dstufft> we've def had PyPI in YOLO mode running screen for like 3 days before >:]

[15:45:00] <sumanah> dstufft: right i remember you telling me about that

[15:45:28] <sumanah> how about, since [year? 2015? whatever relevant year] we haven't had an outage lasting longer than [time]

[15:45:41] <dstufft> (in general, ``pip install`` is really really hard to get an outage on now adays, other parts of the service are easier to take down)

[15:45:54] <dstufft> but we really engineer/optimize for pip install not failing

[15:46:10] <dstufft> (I am fine with giving data to be clear)

[15:46:24] <sumanah> thanks for clarifying, on this and the reqs/second thing

[15:48:23] <sumanah> mgedmin: thanks for asking about this and getting us to clarify stuff

[15:49:26] <sumanah> Our livechat hour is winding up -- this is a 24-7 channel so you're welcome to continue to talk & hang out here, but Dustin and I may drift away in about 10 min. :)

[15:49:59] <sumanah> I see Anurag had to leave, but I'll probably write a quick PR to move the link to the application overview higher up in the Getting Started guide.

[15:50:58] <alanbato> sumanah: I added a comment with what I said to the issue

[15:51:43] <mgedmin> thanks for the answers!

[15:51:45] <sumanah> Thanks alanbato!

[15:51:53] <alanbato> If di_codes could take a look at https://github.com/pypa/warehouse/pull/3520 and give me some feedback I'd really appreciate it!

[15:52:38] <sumanah> pradyunsg: I'd also appreciate your take on https://github.com/pypa/warehouse/issues/2794 , on what we should add to application overview docs, if you have time and interest

[15:54:15] <di_codes> alanbato: looking at that now. seems to be a lot of duplication between that and <https://packaging.python.org/specifications/core-metadata/> with regards to the descriptions/examples. maybe it would be better to just link to the specification for each field?

[15:54:24] <di_codes> also would be great to know which ones are optional or not

[15:54:26] <sumanah> thea: ^

[15:54:38] <sumanah> in case you want to comment on some proposed API doc improvements

[15:54:38] <alanbato> sumanah: Feel free to point me or tag me in any other issue I can help with

[15:55:30] <di_codes> alanbato: also looks like it’s missing the dependency/digest fields but seems like you’re already aware of that

[15:55:39] <sumanah> alanbato: :) once you revise #3520 per Dustin's feedback here, perhaps you could take on https://github.com/pypa/warehouse/issues/3490 , Document User-Agent filtering for clients with broken redirect handling APIs/feeds documentation ?

[15:56:34] <thea> Yeah it's my strong preference that we have a single source of Truth for that. Warehouse should link to pypug for the metadata specs.

[15:57:17] <alanbato> di_codes: Yes, most of it was just me paraphrasing and trying to give it another format.

[15:57:18] <di_codes> alanbato: basically I’m imagining a plaintext version of this form: <https://github.com/pypa/warehouse/blob/8118bd3c5e9bbc72ef1f24aff26716df5ade1d5c/warehouse/forklift/legacy.py#L326-L558> with links to the metadata spec

[15:57:19] <sumanah> I'll +1 that. Thanks for prototyping and iterating on this thea and I'm sorry to have inadvertently led you astray, as I think I did with some early comments

[15:57:33] <sumanah> er, thanks for iterating, alanbato

[15:58:00] <sumanah> I'll also give a shoutout right now to thea for the GitHub-Flavored Markdown work http://blog.jonparrott.com/github-flavored-markdown-on-pypi/ :) thanks for the work and the kind words, thea

[15:58:08] <thea> Indeed, thanks alanbato

[15:58:21] <thea> My pleasure!

[15:59:32] <sumanah> ok it's 2 minutes till the hour by my clock -- pradyunsg alanbato mgedmin Wooble any last thoughts?

[16:00:04] <alanbato> di_codes: Yes, I was using the form as a reference. I'm sorry, but I'm not sure what do you mean by a plaintext versions. Maybe replacing the description and example with the link in the table?

[16:00:46] <di_codes> alanbato: ah, I meant: that form, but not in code (human-readable)

[16:00:54] <sumanah> Dustin & I may drift away now -- thanks for joining us. The next one:

[16:00:54] <sumanah> Thursday, April 5th, 5pm-6pm PDT, 8pm-9pm EDT, (April 5th) 8am-9am

[16:00:55] <sumanah> Manila, (April 5th) 10am-11am Melbourne, (April 5th) 0:00-1:00 UTC

[16:00:55] <sumanah> https://www.timeanddate.com/worldclock/fixedtime.html?p1=24&iso=20180405T19&msg=Warehouse/PyPI%20beta%20livechat&ah=1&low=4

[16:05:38] <alanbato> di_codes: I'm having a bit of trouble thinking how that would look like, I'll try to look for examples on other projects and submit a patch to the PR Soon™. Thanks for your input :)

Log file Viewer

Help | Karma | Search:

#pypa-dev logs for Tuesday the 3rd of April, 2018