Log file Viewer
#pypa-dev logs for Wednesday the 28th of March, 2018
[08:32:28] <pf_moore> pradunsg dstufft: Can you take a look at https://github.com/pypa/pip/issues/5121? Looks like we may have an issue with Jython in pip 9.0.2 and 10...
[14:28:00] <sumanah> lghampton: Ernest is working on mitigating a DoS attack (see http://status.python.org/ ) and I'm responding to some beta testers' feedback
[14:29:15] <lghampton> sumanah: I am working on filing the text encoding issue against setuptools, and then I will look at some past PR reviews
[14:36:13] <sumanah> waseem18: I'm a little busy because I'm taking some beta testing feedback from one place and putting it in another :) (from a list to Warehouse issues on GitHub)
[15:01:01] <pradyunsg> I'm not sure who to tell or if this would need a github issue -- It seems the "trending" projects on pypi.org haven't changed in the past full month.
[15:01:24] <sumanah> pradyunsg: there's an issue about trending PyPI projects in github.com/pypa/warehouse but I don't think it mentions that
[15:02:31] <pradyunsg> Does anyone know how often is that refreshed -- or _maybe_ it's not being updated for some reason?
[15:55:15] <lghampton> sumanah: it seems setuptools wants to allow users to pass in `long_description` files in non-UTF-8 formats, and not fail when they do so
[15:58:40] <toad_polo> I'll note that this exact same thing happened to me 2 days ago: https://github.com/dateutil/dateutil/pull/651
[15:59:16] <toad_polo> Llghampton , sumanah : If you pass it as a `unicode` / `str` it will always succeed, you don't have to worry about the encoding.
[15:59:54] <toad_polo> You do need to worry about the encoding when you are reading the file, and often times there's literally no way to know what that is because there's no consistent storage of metadata about file encoding :(
[16:00:19] <lghampton> toad_polo: Yes, I was reading the link you put in your comment about mojibake
[16:00:24] <sumanah> toad_polo: when lghampton was testing this I was warning her: you are now going to find monstrous Frankenfiles that lie about encodings
[16:01:08] <sumanah> and now you need to double-check, like, every layer of processing the file undergoes between your text editor and PyPI to see what might be reading/fiddling with stuff
[16:01:23] <toad_polo> I think if you are trying to write something generic, the `chardet` module will give a best guess heuristic about what the encoding is.
[16:01:35] <sumanah> toad_polo: are there likely encoding-related problems that readme_renderer *should* notice?
[16:01:59] <sumanah> better to target our efforts at that, rather than just fuzz the system with wacky files in arbitrary encodings
[16:03:24] <toad_polo> Though you'd probably have to open the file in `rb` mode and then `decode` afterward you know what encoding to use, rather than using `io.open` with a specific encoding.
[16:06:14] <toad_polo> I'm the world's shittiest setuptools maintainer, having just taken on that mantle very recently, so I don't really know, but my guess is that unless you want to try to auto-magically fix the encoding stuff (which I don't recommend), you don't bother.
[16:06:26] <toad_polo> I mean, it's good to fuzz it mind you to make sure there are no actual security vulnerabilities, but the contract is that PKG-INFO is in UTF-8.
[16:06:37] <toad_polo> And setuptools expects you to pass it something preferably as `unicode`, meaning any encoding/decoding shenanigans would happen *before* it ever gets to the metadata writing stage.
[16:06:59] <toad_polo> It's always possible that someone could craft their own non-compliant PKG-INFO in a weird encoding, but I think you can rightly call that a bug in their implementation of the PEP.
[16:08:09] <toad_polo> Theoretically you could try to do some herculean thing like ftfy does: https://ftfy.readthedocs.io/en/latest/
[16:13:29] <toad_polo> I'm going to close bug #1311, let me know if you have reason to believe it actually is something that needs to be changed on the setuptools side.
[16:16:14] <sumanah> toad_polo: and hey, thanks to the pmxbot logs, you can just link to this to explain!
[16:16:49] <sumanah> lghampton: congrats, you have now Learned More Things about the intricate messiness of file encodings
[16:18:01] <sumanah> lghampton: Leonard owns an interesting book he reviewed in a visually fun blog post https://www.crummy.com/2012/3/19/1
[16:21:27] <lghampton> sumanah: It is also interesting that languages like Japanese and Russian both have words for the random characters that result from text encoding being read incorrectly
[17:54:39] <sumanah> lghampton: I'm gonna take 5 min to catch up on a few things and then ping you to talk code review!
[17:55:25] <nlh> sumanah, if you're here - I'm not sure if you saw this: https://github.com/pypa/warehouse/pull/3434
[17:56:41] <nlh> lghampton, did you see my message on slack ? I need some help organising user test training and sessions, if you might be able to help me with that
[18:00:05] <pf_moore> quick question - when you did the pypi beta announcement, how did you get it in the Python insider blog? Was it just through posting it to python-announce?
[18:02:19] <sumanah> pf_moore: I believed the announcement was important enough to deserve that signal-boosting but let me know if you think I was wrong there
[18:02:45] <pf_moore> No, no - I was thinking about putting the pip 10 announcement up there, but wasn't sure how to do it
[18:03:19] <sumanah> pf_moore: :) I figure you can ask any of the people on the sidebar at https://blog.python.org/ to do it.
[18:03:22] <pf_moore> (Apparently I have editor rights on there, but I'm not sure I've ever used them :-))
[18:03:52] <sumanah> pf_moore: oh - maybe Ernest can tell you if he is not busy explaining things to Artifactory and fighting spammers
[18:04:29] <pf_moore> sumanah: Yeah - includes me... So I guess I just go for it. Cool. Really, I just didn't want to have a mail to python-announce get auto-posted there and then me duplicate it
[18:05:52] <pf_moore> EWDurbin - I'm just trying to get clear what I need to do to post the pip 10 announcement (beta end of this week) and wasn't sure if there was any protocol for posting to python insider
[18:06:08] <sumanah> pradyunsg: so lghampton has been doing a fair amount of pull request review recently for Warehouse, but so far has almost entirely been doing *testing* (fetch the branch, run Warehouse, verify functionality)
[18:06:39] <sumanah> pradyunsg: I thought you might have some tips and thoughts about what was helpful to you, in learning code review skills, as you grew from contributor to co-maintainer of pip
[18:06:44] <EWDurbin> pf_moore: i think it's generally just used for release announcements... seems fair to do major stuff there. the protocol is basically "someone does it" it seems
[18:08:22] <EWDurbin> pf_moore: i would reccommend working with two or three people to draft/edit before posting
[18:09:32] <pradyunsg> sumanah: I did do a bunch of reviews, so, I'll be happy to help in any way I can. :)
[18:13:25] <sumanah> lghampton: while we're waiting for pradyunsg to share any tips/thoughts about what was helpful to them, let's look at https://github.com/pypa/warehouse/pull/3405
[18:14:05] <sumanah> lghampton: through the lens of https://warehouse.readthedocs.io/development/reviewing-patches/
[18:15:59] <sumanah> pradyunsg: Different projects have different customs about what it means to have the commit bit
[18:16:19] <sumanah> lghampton: if you look at Anurag's two open PRs, if you look at the commit histories, I think you'll see what I mean
[18:16:43] <lghampton> sumanah: in the sense that he is pushing to Warehouse master and not his fork?
[18:17:04] <sumanah> lghampton: compare his two PRs and see what he says (in the PR message) he's trying to do in each
[18:17:34] <sumanah> pradyunsg: how long ago did you get the GitHub "pypa/pip wants to make you a collaborator on this repo" message?
[18:19:54] <sumanah> pradyunsg: I have found https://producingoss.com/ a useful reference in talking about what good maintainership looks like, in case you want more perspectives
[18:23:25] <lghampton> sumanah: oh, I see what you mean now, it does look like he pushed changes from his branch fixing https://github.com/pypa/warehouse/issues/3185 to the wrong branch
[18:24:40] <lghampton> sumanah: he also seems to have these extra commits in his history where he merged changes from upstream
[18:25:59] <sumanah> lghampton: I gave Anurag some tips in https://github.com/pypa/warehouse/pull/3406#issuecomment-376677254
[18:26:30] <sumanah> lghampton: so let's switch to looking at https://github.com/pypa/warehouse/pull/3408
[18:28:18] <sumanah> lghampton: so first let's do the same thing you've been doing in PR review - load up the branch locally and check that it does what the author says it does
[18:33:54] <sumanah> lghampton: great! I am checking that locally myself now. And now let's look at the code change.
[18:35:47] <sumanah> lghampton: ok, so now I think it's time for us to play with the API to verify that the new documentation is accurate!
[18:36:23] <sumanah> lghampton: https://warehouse.readthedocs.io/api-reference/feeds/#project-and-release-activity-details has a sample of how to connect in your interpreter
[18:37:03] <sumanah> actually lghampton before you start trying out that code, read https://warehouse.readthedocs.io/api-reference/feeds/#project-and-release-activity-details and tell me when you have done so
[18:40:00] <sumanah> Also lghampton I spaced a moment, sorry, we're talking about the /simple/ API rather than the XML-RPC API, so we don't need to use xmlrpc.client
[18:42:32] <pradyunsg> lghampton: so... the way I do PR reviews is that I skim the code, to see the scale of the changes. Then, I proceed try to verify that what the PR says is what it does. Looking for edge cases that OP would not have considered is usually what I do after that.
[18:44:28] <lghampton> pradyunsg: thank you, that is interesting. How do you know what edge cases to look for?
[18:46:05] <pradyunsg> lghampton: If I don't find anything fishy until then, I look at the code and try to see if the change makes sense -- was it made in the right place according to the system architecture (interrupt)
[18:47:49] <pradyunsg> lghampton: what edge cases to look for is mostly more of a good guess than a process for me. I just try to throw all kinds of inputs at the problem.
[18:50:05] <pradyunsg> lghampton: the obvious stuff at this point includes things like super longs strings or unicode strings and zeros.
[18:50:15] <sumanah> lghampton: I love http://thebraidytester.com/downloads/YouAreNotDoneYet.pdf as a source of edge cases :) and as you become more experienced as an engineer you start to pattern-match.
[18:51:34] <lghampton> sumanah: I'm sure. I found https://www.ricston.com/blog/testing-apis-postman/ but it seems like too much
[18:54:48] <pradyunsg> lghampton: Usually, you also get an idea of what kind of edge cases exist _after_ you look at the code -- some things are optional but the PR doesn't consider that and stuff like that.
[18:57:42] <pradyunsg> sumanah: I also like https://github.com/minimaxir/big-list-of-naughty-strings -- has some interesting string handling related edge cases. Maybe someone could try those out on Warehouse sometime?
[18:58:26] <pradyunsg> lghampton: (resuming from interrupt) and other than the architecture, this is also when I look at things like, do the variable names make sense, is the code "looking good" (readability) and all the style stuff.
[18:59:20] <lghampton> pradyunsg: Yes, I have done that kind of code review where someone critiques my code for style and readability
[19:00:41] <pradyunsg> lghampton: They're tricky but I guess you know how to go about those. Having a linter running like warehouse does, makes life easier in these cases. :)
[19:03:03] <pradyunsg> sumanah: it does check line lengths, newlines and whitespace properly so, it's an overall benefit.
[19:05:16] <pradyunsg> sumanah: i'll have to look. I'd made https://github.com/pypa/pip/pull/4651 at one point, and I think I made one for better linting too...
[19:06:46] <pradyunsg> Never got around to making the issue (coz I'd have linked it to the above PRs)
[19:07:52] <sumanah> lghampton: and I see: "'<!DOCTYPE html>\n<html>\n <head>\n <title>Simple Index</title>\n </head>\n <body>\n <a href="/sim
[19:07:52] <sumanah> ple/0/">0</a>\n <a href="/simple/0-core-client/">0-core-client</a>\n <a href="/simple/1234-hello-world
[19:18:54] <pradyunsg> lghampton: sooo, yeah. Mostly, it's skim code, test, do code review, test more and iterate on this as many times as needed.
[19:21:59] <pradyunsg> lghampton: this is assuming that a feature is gonna be useful and we're checking correctness.
[20:14:22] <sumanah> benjaoming: PyPI no longer has a UI for users to manage GPG or SSH public keys, and once the legacy site shuts down, GPG/PGP signatures for packages will no longer be visible in PyPI's web UI.
[20:14:43] <sumanah> benjaoming: you might want to look at https://mail.python.org/pipermail/distutils-sig/2018-March/032066.html for more context
[20:15:04] <sumanah> benjaoming: and Donald has written more about his thinking in https://caremad.io/posts/2013/07/packaging-signing-not-holy-grail/
[20:15:38] <sumanah> benjaoming: You'll still find GPG signatures for packages in the https://warehouse.readthedocs.io/api-reference/legacy/#simple-project-api Simple Project API, per https://www.python.org/dev/peps/pep-0503/ PEP 503
[20:17:35] <sumanah> benjaoming: if you talk a little about what you currently use PGP keys on your PyPI profile to do, we can talk about what other tools you could use to do that. Sorry for the difficulty
[20:37:09] <sumanah> and going to localhost in my browser gives me: sqlalchemy.exc.OperationalError: (psycopg2.OperationalError) could not translate host name "db" to address: Name or service not known
[20:39:07] <EWDurbin> sumanah: you _may_ just need `make serve` running in another tab before running `make initdb`
[20:39:30] <sumanah> EWDurbin: I do have make serve running in another terminal; make initdb still fails
[20:42:32] <sumanah> EWDurbin: at least as far as I can remember I've been using the `make build` and `make serve` commands in the repo root directory on the host machine, like usual
[20:42:50] <sumanah> EWDurbin: I ran docker-compose down -v, then ran `make purge` and got the exact same permission denied errors
[20:44:44] <sumanah> EWDurbin: this can wait several hours because it's blocking Nicole but the magic of timezones means she is probably not blocked *right now*
[20:49:00] <sumanah> I don't see why this should have worked, but I closed a bpython session where I'd used `requests` to make requests of the `/simple` API locally, and the terminal it was in. Ran `make build`, opened a new terminal and ran `make serve`, and now `make initdb` is beetling along happily
[20:51:13] <benjaoming> sumanah: thanks! I find consolation in the mail thread, a nice exchange. Also nice to know that plans of proper authenticity are in the making. Is the signing-disabled version of Wheel released yet? Or does Pypi.org just anticipate it to be a final decision?
[20:51:51] <sumanah> benjaoming: good questions! :) I can refer you to agronholm for wheel-related policy and roadmap questions
[21:00:32] <toad_polo> What would trigger this going off? https://github.com/pypa/setuptools/blob/v38.7.0/pkg_resources/__init__.py#L199
[21:03:14] <sumanah> and benjaoming if you would like, you're welcome to reply to distutils-sig and ask there as well
[21:45:12] <EWDurbin> cooperlees: a kind of silly bug is that you have to push the tar.gz first before the wheel due to wheel not supporing the Description-Content-Type stuff
[21:47:02] <cooperlees> I get no error but bitbucklet web never updates - I opened a ticket to see if they have issues again with that repo :(
[22:07:41] <njs> benjaoming: the wheel signing feature discussed in that distutils-sig thread is actually a *different* way of signing packages than gpg signatures... The basic logic is the same though -- neither really does anything useful, and better to clear the ground of (redundant!) useless stuff than keep it around pretending like it's useful
[22:10:02] <njs> cooperlees: this is off-topic, but I am super curious about your comment yesterday that you can't understand twisted at all, but you're excited about moving things to asyncio, since I've always thought of them as essentially the same modulo methodNaming_conventions :-)
[22:26:06] <warner> hey, so I'm seeing intermittent failures of a Travis-CI OS-X build. I'm aware of the brownout, but 1: I'm pretty sure we've got pip-9.0.3, 2: it fails when it gets to a specific dependent package ("incremental") on multiple builds, and 3: I'm seeing the tox process successfully hit https://pypi.python.org/ lots of times just a few moments before it fails to fetch Incremental
[22:26:24] <warner> of course with travis OS-X builds lagging by several hours, it's not been easy to experiment :)
[22:27:50] <warner> https://travis-ci.org/tahoe-lafs/tahoe-lafs/jobs/359242783#L2054 in case anyone has any ideas
[22:33:36] <njs> oh dear. well, I guess this might help convince people to stop using setup_requires...
[22:33:41] <warner> so.. am I looking for a way to upgrade setuptools, or would that help on an OS-X=10.12?
[22:35:15] <warner> has setup_requires= fallen out of favor? we're using it to require a setuptools that's new enough to understand things like python_requires=
[22:36:21] <dstufft> warner: you can't really use setup_requires to change the version of setuptols that is running setup.py
[22:36:54] <dstufft> you're going to get 2 different versions of setuptools imported, so you might have half of the imported setuptools.* namespace be one version, and the other half be another version
[22:37:19] <warner> yeah, like https://tahoe-lafs.org/buildbot-tahoe-lafs/builders/OS-X%2010.13/builds/14/steps/tox/logs/stdio
[22:38:40] <dstufft> warner: in the meantime, you can probably just ``pip install setuptools incremental`` beforehand
[22:38:58] <dstufft> sumanah: er, not officially, a friend of mine started calling it that and I picked it up as a habit
[22:39:19] <warner> hm, ok. This is being driven from tox, so I guess I need to do that manually from within my tox.ini instead of letting Twisted own it
[22:39:26] <sumanah> I'm actually liking it as long as there is actually a 10.0 thing, you know, for all the dependency management goodies
[22:40:34] <sumanah> njs: can I put you down as a maybe for https://wiki.python.org/psf/PackagingSprints ?
[22:41:51] <njs> sumanah: I'll be around, but if people want to work on trio stuff I'm going to prioritize that over packaging
[22:45:20] <warner> so under what circumstances do we get TLS-1.2? (pip-* under OS-X-10.13) OR (pip-9.0.3 under OS-X-*) and no version of setuptools-without-pip at all?
[22:47:39] <warner> so with a system python, linked against their ancient OpenSSL, how's the story change?
[22:49:23] <dstufft> in all cases here Python is jsut letting OpenSSL negiotiate the highest TLS it understands, just macOS was using 0.9.8<some letters> until it switched to libressl in 10.13
[22:50:05] <dstufft> and pip 9.0.3 has workarounds to use SecureTransport instead in the 0.9.8 using macs
[22:50:12] <warner> ok, so (mostly) any python on 10.13 should be ok because the system OpenSSL is really a modern-ish libressl, right?
[22:50:54] <warner> but on 10.12 you depend upon either 1: a better python, e.g. homebrew, that doesn't link against system OpenSSL because it's ancient, or 2: pip-9.0.3 which knows how to use OT
[22:50:56] <cooperlees> njs: TO be fair, I haven't twisted in year too. I'm sure it's a lot better. But I just understand asyncio naming and flows better I feel. Was there a particular question you have for me?
[22:57:02] <dstufft> I've talked at length with this to glyph & co, I'm pretty sure it boils down to (A) Twisted has a LOT of abstraactions at varying levels, and a lot of tutorials at using those abstractions, but not a lot of documentation at "unpealing" the abstractions, so as soon as an abstraction doesn't do what you want it to do, you're very very confused, whereas asyncio starts out without much, and most of the abstractions live outide of asyncio, so you
[22:57:02] <dstufft> start to layer them on as you understand things, versus trying to unpeal them and (B) asyncio makes coroutines first class, whereas Twisted really watns you to use callbacks, and callbacks do not fit my brain nearly as nicely as coro's do (and yes there is @inlineCallbacks, but none of the example code really uses it, and you used to often times get scoleded for using it)
[22:59:58] <techalchemy> dstufft: is pep518 going to become mandatory at some point/setup.py support dropped?
[23:00:36] <dstufft> techalchemy: setup.py will never be dropped, though it is very likely at some point that setup.py becomes implemented as an implicit PEP 518 backend.
[23:02:34] <techalchemy> okay, I think we may need to prioritize an upgrade path on our end because anything that avoids us having to download every package to resolve setup.py when generating lockfiles will be a massive gain
[23:03:40] <dstufft> PEP 518 still downloads packages of course, not doing that requires a lot more than just a static setup_requires
[23:06:57] <techalchemy> dstufft: for our resolver, though, all it really demands is that pyproject.toml has install_requires enumerated and that that would be available via some api
[23:06:58] <njs> techalchemy: it's PEP 517 that lets you avoid calling setup.py, PEP 518 is just a replacement for setup_requires=
[23:07:46] <njs> techalchemy: mostly relevant b/c pip 10 will come out within a few weeks with PEP 518 support, but PEP 517 isn't implemented yet
[23:08:05] <dstufft> PEP 517 doesn't have static install_requires either, you have to call code to get it
[23:08:31] <njs> cooperlees: I didn't have any specific question I guess, I'm just generally interested in people's experiences with these packages
[23:08:51] <techalchemy> it's currently a huge bottleneck for some larger projects, we are trying to consider ways to speed up lockfile generation
[23:10:01] <njs> cooperlees: the backstory is that I am for some reason obsessed with async usability, to the point that I have my own thing that competes with both twisted and asyncio :-)
[23:11:04] <njs> dstufft: there is something confusing about twisted's maze of abstractions, isn't there :-/
[23:16:35] <njs> dstufft: do you happen to know what the dividing line is for "thing you can use" vs not that? :-)
[23:18:42] <dstufft> njs: think it could handle 500 million "events" (where 1 event = 1 line of bytes sent to it) a month? :D
[23:18:48] <njs> dstufft: oh, you can totally use it, as long as you're okay with a somewhat aggressive deprecation policy and a really tiny library ecosystem
[23:20:02] <njs> dstufft: eh, speed-wise it should be within, like, a factor of 2 of anything else, and better than that if you can give us real benchmarks to target
[23:20:19] <dstufft> njs: I have a thing using LineReceiver currently that could be rewritten in trio :]
[23:23:59] <njs> dstufft: http is a little more complicated. For clients there's 'asks', which is a from-scratch requests-like client (though missing fancy stuff like proxy support), there's an in-progress branch to add native support to urllib3/requests, and there's trio-asyncio which lets you use asyncio libraries like aiohttp
[23:24:57] <njs> dstufft: for server-side trio-asyncio is kinda the only thing right now, someone (maybe me I guess) needs to come up with an async WSGI-equivalent API and servers and frameworks written against it etc etc sigh