Log file Viewer
#pypa-dev logs for Friday the 16th of March, 2018
[00:08:12] <sumanah> So far it seems like all the people who reported issues to us about the load testing were nice about it
[00:10:27] <sumanah> di_codes: I'm sorry for the wait. I'm testing with downstreams right now, e.g., finding this bug https://github.com/zestsoftware/zest.releaser/issues/271
[00:38:32] <clinth> hey, is there any sort of a/b testing going on with pypi.python.org and pypi.org?
[00:40:55] <clinth> it looks like the ordering changed on the simple index between the two, so previous a .tar.gz was first, and on pypi.org a .zip is first :\ this caused an issue where our requirements.txt hashes only specify the .tar.gz...
[00:45:54] <EWDurbin> clinth: can you file an issue with more details at https://github.com/pypa/warehouse/issues?
[01:26:16] <sumanah> I skimmed https://bugs.launchpad.net/ubuntu/+source/python-pip/+bug/1694560 and https://bugs.launchpad.net/ubuntu/trusty/+bugs?field.searchtext=pip&search=Search&field.status%3Alist=NEW&field.status%3Alist=INCOMPLETE_WITH_RESPONSE&field.status%3Alist=INCOMPLETE_WITHOUT_RESPONSE&field.status%3Alist=CONFIRMED&field.status%3Alist=TRIAGED&field.status%3Alist=INPROGRESS&field.status%3Alist=FIXCOMMITTED&field.assignee=&field.bug_reporter=&field.omit_dupes
[01:26:16] <sumanah> =on&field.has_patch=&field.has_no_package= but am too tired to really dig in and try to figure out whether anyone has already reported the issue we saw tonight
[02:15:19] <sumanah> I added a "help needed" label to the CAPTCHA item - the only issue in https://github.com/pypa/warehouse/milestone/10 that seems most head-scratching.
[04:04:53] <sumanah> dstufft: those pip PRs, https://github.com/pypa/pip/pull/5000 & https://github.com/pypa/pip/pull/4835 - on your plate for tomorrow?
[05:56:13] <pgadige__> hello! May I know what username should be given to login at http://localhost:80/account/login? I'm interested in contributing to Warehouse project. I read the post on user testing, and I thought it's an amazing project to be part of. Also, after listening to a Talk Python podcast on PyPI, I'm all the more curious to learn about what goes in building such a product.
[13:31:39] <di_codes> pgadige__: see <https://warehouse.readthedocs.io/development/getting-started/#logging-in-to-warehouse>
[14:29:59] <sumanah> di_codes: clearly in https://github.com/pypa/warehouse/issues/869 there are people who want to help in some way. I wonder whether we could ask them to pre-test anything with a fresh comment in that issue once your Warehouse PR is available to try?
[14:43:29] <di_codes> sumanah: yeah, I was planning to comment there once the last Warehouse PR is merged with my blog post, and tell eager folks that they could try it with the twine prerelease
[14:44:50] <sumanah> I'm guessing the setuptools issue with provides-extras won't affect Markdown support
[14:45:09] <sumanah> di_codes: I'm gonna post to wheel-builders asking them to try the twine prerelease
[14:46:46] <di_codes> sumanah: nope, it wont, and warehouse doesn’t do anything with `provides_extras` anyways so it doesn’t really matter there, but it is unfortunate. hopefully we can get that fix released soon
[18:01:31] <lghampton> sumanah: It's been ok. We did an event last night with NYC Python called IDEs of March where people showed off their editors
[18:01:59] <sumanah> I made an "IDEs of March" joke yesterday and it's wonderful to know it instantiated itself into an actual event
[18:02:31] <sumanah> lghampton: how about we spend a few minutes catching each other up, and then you get to work for a few hours :)
[18:03:08] <sumanah> lghampton: this week we did some load testing where we redirected some traffic, for a few periods, for `pip install`
[18:03:30] <sumanah> we learned various things about stuff that breaks on our end and stuff that is broken out among our users
[18:05:39] <sumanah> lghampton: I also found out that http://pypi.python.org/ , if you don't block the statuspage JS, shows you relevant status in the upper right, under the login box
[18:06:50] <sumanah> lghampton: folks have already made some improvements based on stuff we learned during the testing, like 7dcff19af4882c4e18bb7456fb2504cfdf08b13a
[18:07:42] <sumanah> lghampton: people running the 2014 LTS (long term service release) of Ubuntu are usually using a pretty old version of pip
[18:08:29] <sumanah> lghampton: people on some versions of the Mac OS have older versions of Python and old versions of security-related libraries that don't support the version of TLS that we want them to use
[18:09:16] <sumanah> if you want you can catch up on issues in the warehouse repo to see a lot of what we found out from users
[18:10:45] <lghampton> sumanah: I heard about the issue with old versions of pip. Doesn't it show a message to tell them to upgrade?
[18:11:08] <sumanah> of course we all want more monitoring, alerts, etc. to help us keep from breaking stuff, but also the MOSS grant is limited in scope and we have to prioritize
[18:12:26] <sumanah> there's now a classifiers-adding UI for PyPI admins https://github.com/pypa/warehouse/pull/3253
[18:12:52] <sumanah> and I added stuff about site admins to https://warehouse.readthedocs.io/application/#usage-assumptions-and-concepts
[18:13:15] <sumanah> there's more migration guidance regarding the API https://github.com/pypa/warehouse/pull/3240
[18:13:36] <sumanah> and I worked through this list of third-party services to tell them about the Warehouse changes https://github.com/pypa/warehouse/issues/2935
[18:14:41] <sumanah> and am drafting a better public-facing explanation of the changes https://wiki.python.org/psf/PackagingWG/PyPIBetaAnnouncement -- I know that says draft and don't publicize, but I'm ok sharing it with IRC and distutils-sig, as it is not SECRET
[18:15:59] <sumanah> the next milestone https://github.com/pypa/warehouse/milestone/10 is down to 4 open issues. 1 is very easy (removing the pre-prod warning), 1 we already have a PR for from Nicole (refactoring dropdowns), 1 we have part of a PR for (blake2 uniqueness)
[18:16:55] <sumanah> so my guess is that we will be able to publicize the beta sometime in the middle of next week and it really depends on the CAPTCHA situation
[18:17:54] <Wooble> lghampton: there are versions of pip in the wild that predate the version check that tells you they're out of date. :)
[18:18:14] <sumanah> Then, in Warehouse, 16 open issues till redirect/launch, then 12 more till legacy shutdown. Of course that will change as we get more testing.
[18:18:48] <sumanah> lghampton: another big chunk of news is how far we're getting on Markdown support
[18:19:11] <sumanah> Dustin got PEP 566 accepted weeks back, and since then has been getting PRs finished and merged into various parts of the stack
[18:20:17] <sumanah> he's working on a PR now for Warehouse so that, for instance, https://test.pypi.org/project/Forms990-analysis/1.0.5.dev1/ would render properly
[18:20:54] <sumanah> lghampton: in the past few days, with my volunteer twine maintainer hat on, I have learned a lot about wheels, tox, setuptools, core metadata, and so on
[18:21:51] <sumanah> lghampton: https://pypi.org/project/twine/1.11.0rc1/ is the pre-release of Twine that supports uploading READMEs in Markdown
[18:23:10] <sumanah> lghampton: so today, since your existing open PR has been merged, I think it would be good for you to do the scheduling for the Monday meeting, then review 1 or 2 PRs, then work on an open issue
[18:23:34] <sumanah> lghampton: is there anything else you want to mention to me or ask about before starting?
[18:24:07] <sumanah> great. Will ping you here in channel with a recommendation of a PR to review and an issue to work on
[18:27:32] <sumanah> lghampton: https://github.com/pypa/warehouse/pull/3291 "Move modal out of dropdown markup, ensure index matches slug" by yeraydiazdiaz - please test this in a few browsers and, if possible, see how it works on mobile
[18:28:18] <sumanah> Update on Markdown! https://github.com/pypa/warehouse/pull/3296 is created and merged! thank you di_codes
[18:29:33] <di_codes> handy guide: <https://dustingram.com/articles/2018/03/16/markdown-descriptions-on-pypi>
[18:29:58] <sumanah> di_codes: https://test.pypi.org/project/Forms990-analysis/1.0.5.dev1/ is still misrendered. That's expected, right?
[18:31:04] <sumanah> di_codes: GO YOU many happy faces, rockets, trophies, and cornucopia confetti horns here!!!
[18:36:57] <sumanah> di_codes: I was thinking of collaborating with lghampton today on trying to test the description_content_type side of Warehouse and be really tough on it and see if anything else breaks. :) would that be ok with you?
[18:40:49] <lghampton> sumanah: I have a quick UI question, is there a place you would recommend I can easily upload a picture?
[18:43:53] <lghampton> sumanah: sorry, try this one? https://send.firefox.com/download/8c808fb138/#bAqigwEh8lj_YLkpj1ksmQ
[18:45:04] <lghampton> sumanah: Ok. To delete the release, the user has to enter the version number of the release in the text box, and then the "delete release" button becomes usable
[18:45:30] <lghampton> sumanah: but the text box has the version number already in it, and I think it is not obvious that the user has to retype the number
[18:46:33] <sumanah> we found it in user testing https://github.com/pypa/warehouse/issues?q=is%3Aopen+label%3A%22user+testing%22
[18:47:01] <sumanah> lghampton: https://github.com/pypa/warehouse/issues/3162 and https://github.com/pypa/warehouse/issues/3163 are related to this
[18:50:59] <di_codes> sumanah: looks like description_content_type is not set for that release of Forms990-analysis
[19:05:14] <lghampton> sumanah: this: https://howchoo.com/g/mte2zgrhmjf/how-to-access-a-website-running-on-localhost-from-a-mobile-phone
[19:08:03] <sumanah> di_codes: I just now double-checked and re-uploaded. https://test.pypi.org/project/Forms990-analysis/1.0.5/
[19:09:48] <sumanah> di_codes: I did set description_content_type="text/markdown", in https://gitlab.com/brainwane/form990s/blob/master/setup.py
[19:10:38] <sumanah> di_codes: and the Markdown-specific bits (the links) still are not rendering as Markdown
[19:13:38] <sumanah> di_codes: setuptools 38.6.0, twine 1.11.0rc1, pkginfo 1.4.2. And if you download the sdist you can verify that I did in fact add that description_content_type in setup.py. Maybe the wheel is the issue?
[19:15:56] <sumanah> I'll try only building and uploading an sdist, and only building and uploading a wheel, for the next versions, to see what happens.
[19:20:18] <sumanah> Nope, that's not the problem. https://test.pypi.org/project/Forms990-analysis/1.0.6.dev1/#description and https://test.pypi.org/project/Forms990-analysis/1.0.6.dev2/#description both don't have properly rendered links.
[19:24:09] <lghampton> sumanah: That was surprisingly straightforward! I found my ip with ifconfig and went to it on my phone and voila!
[19:24:37] <sumanah> "surprisingly straightforward" says Laura Hampton of the New York Open Source Times
[19:28:26] <sumanah> lghampton: are you done reviewing the delete modal one on mobile? what browsers did you try?
[19:30:44] <sumanah> lghampton: I refreshed the PR and saw you mentioned Safari in your comment. got it
[19:31:01] <sumanah> lghampton: ok, so next, now that Dustin has landed his Warehouse PR for PEP 566 support, I would like for us to test that a bit
[19:31:36] <sumanah> lghampton: weirdly, even though I have set a description_content_type in https://test.pypi.org/project/Forms990-analysis/ , the Markdown is not rendering
[19:31:50] <sumanah> and when Dustin logs in as a PyPI admin and looks at the metadata, the description_content_type field is blank!
[19:35:19] <sumanah> lghampton: how about I make that list in its own GitHub issue. https://github.com/Tunous/TestingRepo/issues/33 is going to be part of it in case you want to start looking
[19:35:37] <sumanah> lghampton: as prep, set up a virtualenv on your computer where you install the prerelease of Twine
[19:36:08] <sumanah> lghampton: https://dustingram.com/articles/2018/03/16/markdown-descriptions-on-pypi is the instructions you will be working from, basically
[19:45:31] <sumanah> lghampton: I'm in a private chat with Dustin right now where he spies a potential bug. setuptools talks about long_description_content_type and twine talks about description_content_type. (I imagine he also needs to get pkginfo and Warehouse to call it the same thing too)
[19:46:15] <lghampton> sumanah: so you think that mismatch is what is causing the Markdown to dissapear?
[19:46:39] <sumanah> lghampton: in the meantime. https://github.com/pypa/warehouse/issues/3299 we can still prepare to test stuff
[19:48:01] <sumanah> I'm gonna read https://packaging.python.org/specifications/core-metadata/#provides-extra-optional-multiple-use again
[19:51:45] <sumanah> I think https://setuptools.readthedocs.io/en/latest/setuptools.html#declaring-extras-optional-features-with-their-own-dependencies applies too. reading this.
[19:53:24] <sumanah> and https://www.python.org/dev/peps/pep-0566/#provides-extra-optional-multiple-use has more explanation too
[19:54:07] <sumanah> when I do some git greps and some searches in the Warehouse repo on GitHub, I think the only place we show the Provides-Extra data is in the PyPI site admin view of a particular project
[19:54:14] <sumanah> warehouse/admin/templates/admin/projects/release_detail.html:159: {% for extra in release.provides_extras %}
[19:55:18] <sumanah> lghampton: I'm not even clear on whether we provide it in our API yet. it's not in https://warehouse.readthedocs.io/api-reference/xml-rpc/#package-querying
[19:57:11] <sumanah> lghampton: I may have seen it a few times but not understood it as well as I do now.
[19:58:40] <sumanah> lghampton: so may I tell you about the relationship, as far as I can gather, among pkginfo, setuptools, and twine?
[20:00:08] <sumanah> the setuptools docs https://setuptools.readthedocs.io/en/latest/setuptools.html#declaring-extras-optional-features-with-their-own-dependencies on "extras_require" suggests: your sample project here is Project-A
[20:00:37] <sumanah> if you want this optional extra "PDF" feature, you need to install the reportlab package
[20:01:19] <sumanah> https://packaging.python.org/specifications/core-metadata/#provides-extra-optional-multiple-use says: your sample project says, via Provides-Extra, that there exists this extra option/feature called "PDF"
[20:01:53] <sumanah> and if you want, you can also say, with Requires-Dist, that there is this required dependency for "PDF" which is the reportlab package
[20:02:38] <sumanah> Requires-Dist has been around for a while and is documented in https://packaging.python.org/specifications/core-metadata/#requires-dist-multiple-use
[20:04:26] <sumanah> as far as I can tell -- and I hope people in the channel will tell me if I'm getting this wrong -- it's the fundamental way that we say "this project depends on you installing that package"
[20:07:35] <sumanah> I think I know, in an intuitive way, but am not sure I can explain it yet, lghampton
[20:10:19] <sumanah> lghampton: I'm not sure. I'm gonna take 5 min to fix the long_description_content_type issue with my README and upload it
[20:10:49] <sumanah> lghampton: you may find it useful to take a look at http://www.curiousefficiency.org/posts/2016/09/python-packaging-ecosystem.html while I do that
[20:24:04] <nlh> di_codes, I don't know if you're still here - but I really can't emphasis enough how happy I am to see markdown descriptions ship! Thanks for all your work on this!!! SOOOOOO exciting!
[20:25:12] <nlh> ok, well, it doesn't need a response - I just wanted to say it in a public channel :D
[20:27:19] <sumanah> that was causing a problem for https://test.pypi.org/project/Forms990-analysis/
[20:27:39] <sumanah> lghampton: right now it looks like making and uploading an sdist makes everything go fine, but wheels, not so great
[20:28:07] <sumanah> lghampton: but in the interim, go ahead and try following his directions from his blog post to get a test package onto test PyPI that has a Markdown README
[20:28:32] <sumanah> and then go ahead and start working through the items in https://github.com/pypa/warehouse/issues/3299 "Test PEP 566 features"
[20:28:54] <lghampton> sumanah: can I have a few more minutes to finish Nick's blog post, or can I put it aside for now?
[20:29:14] <sumanah> oh please go ahead and finish that blog post first if you're not done, lghampton - sorry, I miscalculated
[20:43:34] <lghampton> sumanah: I am going to edit my test package to add some markdown, and start testing
[20:54:54] <lghampton> sumanah: I am going to test the first case, where I add Github Markdown to the README
[20:56:04] <lghampton> sumanah: It will either render it correctly, or there will be errors. It depends on the flavor of markdown Warehouse supports
[20:56:31] <sumanah> lghampton: ok. do you have an intuition about where to check what Markdown parser we use and thus what flavors we'll support?
[20:57:16] <lghampton> sumanah: not at the moment, I thought my experiment would give me more information
[21:16:12] <sumanah> dstufft: I went to a Debian packaging workshop once, about 8 years ago, so I have forgotten a lot
[21:17:08] <sumanah> dstufft: so https://tracker.debian.org/pkg/twine points to your fork of the repo instead of pypa/twine -- is that something any of us can change fairly easily, or is it something we have to ask one of the uploaders on the Debian end to do?
[21:30:28] <lghampton> I'm trying to upload my test package, but I'm getting the error message that says " HTTPError: 400 Client Error: File already exists". Which file does it refer to?
[21:34:33] <sumanah> lghampton: one of the principles of PyPI - clearly this should go into the usage assumptions documentation --
[21:34:46] <dstufft> lghampton: if they're named the same it will, so running the same command twice without making changes will overrite, but the old files might be named foo-1.0.tar.gz and your new files are now named foo-2.0.tar.gz
[21:35:14] <sumanah> dstufft: I think she's talking about *uploading* a new distro, not *creating* one locally
[21:35:35] <sumanah> lghampton: and even if the user then deletes the distribution on PyPI, we do not allow the user to then upload new packages with the same filename
[21:36:29] <sumanah> like: you can upload an sdist, and then later, for the same version number, delete the sdist and upload a wheel
[21:36:43] <sumanah> there's this https://github.com/MacPython/wiki/wiki/Build-Tags which I haven't looked into that much yet
[21:37:44] <sumanah> lghampton: but our policy is: once you have uploaded foo 1.0.0, other people can assume that *that* is what 1.0.0 was and is. And so, ideally, there's no way someone can pip install foo 1.0.0 and have a different version than you got when you pip installed foo 1.0.0 ten months ago.
[21:39:32] <sumanah> lghampton: the big controversy around this was represented in a conversation on distutils-sig a few months ago. Some people said they appreciated a feature on legacy PyPI: the ability to edit, in-browser, the project description page, to fix small errors.
[21:39:41] <dstufft> pf_moore: What do you think about me cutting a 9.0.2 that just backports the securetransport changes to 9.0.2-- Only reason not to wait for 10.0 is we're coming up on a deadline where some mac users won't be able to talk to PyPI anymore with pip without that change, so I figure getting something out asap even with 10.0 coming soon is a good idea?
[21:40:18] <lghampton> sumanah: so they had to make small version changes to fix their project description page?
[21:40:31] <dstufft> pf_moore: It'll be https://github.com/pypa/pip/pull/4454 and possibly upgrading the bundled requests
[21:40:58] <sumanah> lghampton: but our take is: release metadata, such as the long_description, is *part of the project*. And people need to be able to depend on it being immutable.
[21:41:18] <sumanah> lghampton: imagine if my README said "sponsored by EvilCorp" and then I went back and deleted all that
[21:41:36] <pf_moore> dstufft: reminds me, I probably need to check all our dependencies and make sure we have the latest version vendored for pip 10. I'll add that to my list :-)
[21:42:25] <pf_moore> dstufft: If it's hard, let me know and I'll start panicking now for pip 10 ;-)
[21:42:42] <lghampton> sumanah: in other news, Github-style markdown is not supported: https://test.pypi.org/project/list-printer-package/
[21:42:59] <pf_moore> dstufft: I'll update the docs once I've done the release. I'll make sure to add that one
[21:43:38] <dstufft> pf_moore: if you give me some forewarning when you actually plan on cutting the release, I can try to be around incase you run into any problems that you need advice on
[21:43:54] <lghampton> sumanah: Yes, it is. I am going to take a screenshot of my test results and put it in the PR
[21:44:33] <lghampton> sumanah: Sorry, I meant the thread on GitHub with the request for testing the Markdown
[21:45:09] <pf_moore> dstufft: Ta, I'm not sure exactly when I'll do it yet, the beta weekend is Easter and I tend to be busy that weekend, so I'll probably be fitting it around other things - but I'll let you know.
[21:45:40] <lghampton> sumanah: ah :) It just does a weird job rendering the Markdown. Should I make a list of what tags worked and which didn't, or just say "GFH is not supported?"
[21:46:15] <sumanah> lghampton: I think you should say "GFM is not supported" and a few sample tags that did not work, and mention that this is basically fine and expected
[21:46:46] <dstufft> pf_moore: sent you my cell, if you find time and run into an issue and I'm not around, just text me
[21:47:53] <pf_moore> dstufft: Thanks - but honestly I think I'll probably be OK. The process doesn't look that hard. Famous last words ;-)
[21:48:49] <dstufft> pf_moore: yea I think you'll be fine, I'm just wary of shit that accidently only lives in my head :)