Log file Viewer
#pypa-dev logs for Tuesday the 28th of June, 2016
[17:14:32] <sumanah> nlh: is there any particular issue that you think would benefit most from the first chunk of attention, to break it down/estimate it?
[17:16:30] <nlh> sumanah: I thought it might be good to break down the front side of things into 3 sections
[17:17:41] <dstufft> oh, jsut to make it clear here, bitbucket.org/pypa/pypi was the legacy PyPI code base/issues, which have since been migrated to https://github.com/pypa/pypi-legacy where mostly I attempt to ignore things and not feel too guilty because I don't have time to triage and github.com/pypa/warehouse is the new stuff that will hopefully kill legacy
[17:18:35] <dstufft> (2) is probably not *super* important to tackle right away until we have something that people can actually do once they're logged in
[17:19:11] <sumanah> So those are sort of three mini-milestones for the Warehouse work, necessary for replacing PyPI?
[17:19:50] <dstufft> well, for "replacing"-- we're likely still going to need to have legacy PyPI deployed somewhere for the "stuff you can do while logged in" aspect of things
[17:20:44] <dstufft> e.g. those three things don't represent getting to 100% parity, but they represent covering the vast bulk of things that people actually use, except for authors themselves
[17:20:44] <nlh> dstufft: I know that I have not started on any of the logged in side - is it ready on your side?
[17:21:24] <dstufft> nlh: None of the views are done yet no. gsb-eng has got the start of some of the emailing and password reset stuff but it's not merged yet
[17:21:57] <dstufft> I've been focusing at the moment on getting upload.pypi.io ready to go so people can start migrating over to that even before we do the switch
[17:22:55] <dstufft> (My organizational/prioritzation style is very much.. whatever makes me feel most guilty at the moment)
[17:23:23] <sumanah> dstufft: And what still needs doing in order for upload.pypi.io to be ready for people to try out? (like, even if it's not ready for people to actually migrate to)
[17:23:54] <dstufft> sumanah: it's actually ready now, and some folks have moved to it and started using it, twine's master branch defaults to using it (but that hasn't been released yet)
[17:24:52] <sumanah> [oh interesting, upload.pypi.io does not respond to ping at all! I presume as part of the limitation of what ports it responds to?]
[17:25:26] <sumanah> dstufft: so if upload.pypi.io is ready now, what is left for you to do with upload.pypi.io ?
[17:26:05] <sumanah> (maybe I've misunderstood, and you're done with that and ready to move on to other things)
[17:26:43] <dstufft> sumanah: We need to cut a release for twine to start moving more folks over to it by default, make the same change in CPython to default to it, and go and convince Debian, Fedora, etc to switch their URLs-- but the service itself is basically done (although we might end up needing to move it off of Heroku)
[17:27:48] <sumanah> dstufft: so I'm a noob here and you know a lot more than me, but it strikes me that people will be a lot more interested in switching to something that is under a proper python.org domain name and that has a web presence of some kind
[17:28:11] <sumanah> testing, yes, they might add a "also upload to upload.pypi.io" step for testing so you can shake out problems and figure out missing features
[17:29:06] <dstufft> We're moving things off of the python.org domain name for security reasons-- the browser sandbox doesn't protect pypi.python.org from somethingelse.python.org, and because of the crushing weight of legacy systems there are parts of python.org that I don't trust their security of and we can't mandate HTTPS on python.org, only on specific sub domains
[17:30:02] <dstufft> so PyPI is moving to pypi.io as part of this, and upload specifically is moving to upload.pypi.io (upload.pypi.io is a different subdomain from pypi.io largely to avoid passing uploads through our CDN)
[17:30:53] <sumanah> dstufft: ok, understood. I still do think having a web presence of *some* kind is pretty necessary in order to persuade maintainers to switch.
[17:31:56] <dstufft> we can stick something there yea, it's not hard. I just hadn't bothered because it's just an API endpoint, throwing some HTML up is as simple as writing a template file and adding a few lines of code
[17:33:17] <sumanah> dstufft: Great. nlh I've been looking at the issues in https://github.com/pypa/warehouse/issues and I don't see an issue for that already -- do you know if there's one?
[17:35:07] <dstufft> sumanah: testpypi.python.org // test.pypi.io - a second copy of PyPI that's running as a sort of sandbox for people to do wahtever with, without touhing the state of PyPI itself
[17:35:19] <sumanah> nlh: I'm guessing https://github.com/pypa/warehouse/milestones/Launch should include https://github.com/pypa/warehouse/issues/1272 which is adding the send_email feature -- would you mind adding it to that milestone?
[17:35:37] <dstufft> also in legacy PyPI it's used as a sort of staging environment because we don't have tests and running legacy PyPI locally is... hard
[17:36:57] <sumanah> dstufft: ok. And am I right in presuming that people are trying out https://test.pypi.io/ and filing bugs based on that? I saw https://github.com/pypa/warehouse/issues/1060 for instance (showing the author of a package in the metadata)
[17:37:32] <dstufft> sumanah: nah, Id on't think anyone is actually using test.pypi.io right now, it's folks using pypi.io
[17:37:46] <dstufft> when we shut down testpypi.python.org, the folks who use that will migrate to test.pypi.io
[17:39:46] <sumanah> nlh: finishing the current pages (step 1 that you mentioned): is that step covered by issues assigned to you in https://github.com/pypa/warehouse/issues/assigned/nlhkabu ?
[17:41:35] <dstufft> theortically "Launch" was meant to be that, but I'm not sure if we're using it greatly for that purpose
[17:41:45] <sumanah> nlh: dstufft: out of https://github.com/pypa/warehouse/milestones/Launch is there anything that we should actually take out because it's not necessary for Launch?
[17:43:28] <dstufft> If we define Launch as "the stuff we need to make pyp.python.org redirect to pypi.io, but not nescarily enough to shut down the legacy code base completely (e.g. the 90% but not all of the features) then I think #1228 is not required
[17:44:49] <dstufft> and a fair number of these issues are only in the launch task so we can hide them from view on the page (e.g. right now they're rendered using junk or hard coded data, and we should comment them out or something)
[17:45:37] <sumanah> I agree that we ought to break it down further into smaller milestones, and bump some things from "Launch" into something a bit later. Like, (1) Actively ask the community to stress-test pypi.python.org and try using it for real for real (2) Redirect pypi.python.org to pypi.io (1) Shut down legacy code base completely
[17:46:03] <sumanah> Uh, "(1) Shut down legacy code base completely" should be "(3) Shut down legacy code base completely"
[17:47:58] <dstufft> The current migration plan is something like 1) Get pypi.io in a state it's "good enough for now" 2) Deploy legacy-pypi to legacy.pypi.io for now 3) redirect pypi.python.org to pypi.io 4) Continue iterating fixing issues and implementing still missing features from legacy PyPI 5) Once (4) is complete, shut down legacy code base
[17:49:03] <dstufft> and I guess after that is (6) start working on honest to goodness new features rather than just riemplementing all the old features
[17:50:34] <sumanah> dstufft: so let me just ensure I understand: it is ok for Warehouse to replace old PyPI *before* we reach feature parity with the old system. Correct?
[17:52:05] <sumanah> nlh: yeah, that's key. OK I'm going to make a few milestones in GitHub in accordance with the 6-step plan dstufft just suggested
[17:52:09] <dstufft> nlh: Anything that requires a logged in user can be dropped for now, anything for anonymous users.... it's fuzzy I guess depending on how important we think the feature in question is for end users
[17:52:49] <nlh> dstufft: yeah. at this stage, I'd rather focus on cleaning up what we have than getting to the ideal
[17:52:51] <dstufft> e.g. we obviously can't ship to live if we hadn't implemented file serving, but stats? Meh that's a sort of auxillary feature (although if someone else implements it, I wouldn't be _opposed_ to it)
[17:54:16] <nlh> so maybe one of the next steps is going through the todos and making issues for actually removing/replacing stuff
[17:54:19] <dstufft> I think on the front end it's largely cleaning up what's there rather than a whole lot of brand new stuff, besides the misc pages like /sponsors/ -- though that's not a *required* thing either if we wanted to really get down to brass tacks, that's just stuff I think would be good to include
[17:56:05] <nlh> sumanah: it might be a good idea to have an extra milestone of 'post launch top priorities' or something similar
[17:56:21] <dstufft> (I am not great at keeping thoughts organized in the issue tracker, a lot of info still eixsts primarily in my head)
[17:56:42] <sumanah> dstufft: I understand that -- that's why people like me ask you questions and then write stuff down :)
[17:59:07] <dstufft> from my end, the sooner we can get to the "redirect pypi.python.org to pypi.io" step the better-- once the intial "everoyne is yelling at me" is over, reducing the use of legacy (and especially getting rid of it) is going to be a huge reduction in stress for me
[17:59:40] <sumanah> nlh: when you say "Launch" you mean the step where we redirect pypi.python.org to pypi.io? I just want to make sure I mean the same thing as you :)
[18:00:44] <nlh> I have not done a great deal of cross browser work... so... hopefully we are not in for too many surprises :P
[18:01:40] <dstufft> sumanah: yea, to be clear, we auto deploy the `master` branch to both pypi.io and test.pypi.io and pypi.io and pypi.python.org share data stores (so they're really just two diffrent views over the same data) (and test.pypi.io and testpypi.python.org also share data store)
[18:02:04] <dstufft> so launching is really just flipping the switch to make people go to the new site by default
[18:02:33] <sumanah> dstufft: Understood! And when you say "5) Once (4) is complete, shut down legacy code base" is that, like, freezing the repository?
[18:03:28] <dstufft> sumanah: we're going to have to keep the legacy code base running somewhere for people to be able to access it until we maintain feature parity, so that people depending on something that hasn't yet been implemented have some mechanism to continue to do that
[18:04:05] <dstufft> once we reach feature parity (minus and thing we've decided to deprecate/remove) we can shut down the copy of the legacy code base completely and have only warehouse running
[18:04:42] <dstufft> nlh: I think we need to do... something, maybe just comment out that page completely for now
[18:05:06] <dstufft> right now it just renders 3 "dependencies" and 3 "depdents" which are really just whatever the current page are in, just random junk placeholder
[18:05:27] <nlh> ok, well, I'm thinking that for the stats, badges and deps, we can create another issue of "remove/replace with placeholders"
[18:06:05] <sumanah> dstufft: Ok, so when you say "shut down legacy code base" you mean "shut down the legacy web *site* at legacy.pypi.io" as well?
[18:07:06] <dstufft> (legacy.pypi.io actually exists today but it doesn't work very well because PyPI has hard coded URLs in the configuration, so around the time we do the bump I'll have to deploy a config change to legacy pypi so it knows it's being served at legacy.pypi.io)
[18:10:57] <sumanah> Some of the things dstufft mentioned I turned into issues or combined in some way because they are basically conditions/tasks for a milestone
[18:12:14] <nlh> it would be good to also have some kind of system for prioritising the different issues in 5...
[18:12:16] <sumanah> nlh: I believe you wanted a sort of "top-priority after launch" category or milestone as well? sort of a 1.1?
[18:14:14] <dstufft> (5) might not be the most useful milestone I dunno, once we get to (5) we're basically at "chug along mode", or maybe it's useful to classify things as (5) just as a "this is a future task, but not sure when in the future" kind of milestone, i dunno
[18:14:49] <nlh> dstufft: my main thing is that I want to not 'lose' the issues we kick from the launch / parity milestones
[18:14:50] <sumanah> dstufft: I find it's useful to have a way to tag or label or milestone stuff as "this is for the future when we can get to new features"
[18:15:13] <sumanah> so that when that time comes, they are all in a bundle to consider together and prioritize
[18:15:28] <sumanah> and it's a way to be respectful to the people filing those issues. "Important! Next year."
[18:15:38] <dstufft> I guess my question would be what's the difference between something in (5) and something not in any milestone
[18:16:03] <sumanah> dstufft: sometimes something is in no milestone because no one has gotten around to triaging it yet
[18:17:09] <dstufft> (To be clear, I don't mind having it, I'm just trying to sort out what it's actually for:D )
[18:18:15] <dstufft> it feels a bit like a milestone that can never be closed/reached? Which makes me feel more like it's a label not a milestone
[18:19:12] <sumanah> basically I have been using these milestones as a way to have a project plan without having to write an extra document that would have to be kept in sync separately.
[18:20:51] <nlh> dstufft: I worry that having a stand alone "high priority" tag will direct new contributors to those issues
[18:22:28] <sumanah> dstufft: let's meet again sometime soon, like, tomorrow, so I can ask you questions and get stuff out of your head and into issues in GitHub
[18:22:48] <sumanah> I'll also go through my log of this conversation and turn a bunch of these "we should do ___" notes into GitHub issues
[18:23:02] <dstufft> sumanah: sure! I'm around most of the time, I'm Eastern time zone but I tend to stay up late and sleep in
[18:23:50] <sumanah> nlh: dstufft please feel free to move issues around the various milestones and change the deadlines for the milestones, remove them, etc
[18:25:11] <sumanah> also nlh if you feel like having a go at the TODOs that are code comments and turning the "must do by launch" ones into GitHub issues, that would be great
[18:25:49] <sumanah> dstufft: remind me, these days, are you basically available to work on Warehouse 30+ hours/week? because of life changes?
[18:27:24] <dstufft> sumanah: I am employed full time to work on Packaging related tasks, largely I'm trying to focus on Warehouse right now because legacy pypi is a constant source of stress in my life and I want it to die, but I also share that time with other packaging things (pip, legacy pypi, design discussions, yelling at Linux peope for doing things I hate)
[18:27:55] <dstufft> My time has not been super effective lately though, mostly because I'm constantly overwhelmed with it all anymore
[18:28:46] <sumanah> dstufft, if you'd like some advice from a project manager on the next thing to work on, I can give you some advice. I also recognize the overwhelmption and would be happy for you to take a vacation day -- I would stamp APPROVED on that if I could
[18:30:32] <dstufft> yea, it does :) I think it's just 70-90+ hour weeks and my inability to say "no" and my difficulty in delegating for the past... 3-4 years? catching up to me
[18:32:10] <dstufft> nlh: is the first person besides myself who had power to land changes to Warehouse :] and sumanah is the second person really with any meaningful delegation
[18:33:41] <nlh> I think part of the appeal for me in joining was your willingness to give me a fairly free reign
[18:34:28] <nlh> If we were to do another call out for someone to "own" another part of the codebase, then I think taking the same line would be appealing
[18:34:40] <sumanah> dstufft: if you want a suggestion for a next thing: code-reviewing https://github.com/pypa/warehouse/pull/1284 and maybe https://github.com/pypa/warehouse/pull/488 . Helping other contributors learn will be a good way to grow future maintainers.
[18:35:27] <sumanah> nlh: the braindump you need to do -- how about with me, this week? maybe tomorrow?
[18:35:52] <dstufft> sumanah: Yea, I've fallen behind on code reviews :/ I should carve out time today to do that
[18:36:08] <nlh> sumanah: I tend to work on warehouse at 6am in the morning (before work)... so I'm guessing you are asleep then :P
[18:36:28] <nlh> but I will ping you when I have opened some new issues and we can go through them at a more reasonable hour?
[18:36:45] <sumanah> dstufft: you are absolutely not alone. When I was at Wikimedia Foundation code review backlog was something I worked on for all 4 years. I was able to help sometimes with the structural slowness but we never cracked that nut
[18:38:07] <sumanah> see you soon, hope this has been helpful to you, it sure has to me nlh dstufft!
[18:38:38] <nlh> dstufft: I'm going to make dinner now - but I'll ping you once I've opened those new issues
[18:38:57] <dstufft> nlh: sounds good, I'm gonna run to the store to pick up some stuff for my wife, but I'll be back