Log file Viewer

[14:13:54] <sumanah> tylerdave: thanks for https://github.com/pypa/warehouse/issues/3174#issuecomment-373032387 :)

[15:21:05] <sumanah> I wonder why https://pypi.org/ says "132,202 Projects" and https://libraries.io/Pypi thinks there are 125,999?

[15:22:42] <di_codes> sumanah: possibly we are counting projects with no releases, while they aren’t.

[15:22:49] <sumanah> AH. Yeah.

[15:22:51] <dstufft> my first guess would be PyPI is-- yea that

[15:22:52] <dstufft> I know we are

[15:22:56] <sumanah> OK!

[15:22:58] <dstufft> no idea what they're doing

[15:23:19] <di_codes> 10K projects w/ no releases sounds about right

[15:37:51] <sumanah> the trending list at https://libraries.io/trending?platforms=PyPI is pretty different from ours

[15:38:23] <dstufft> I'm assuming completely different algo and data

[15:38:53] <dstufft> there is like a 75% chance that the PyPI one sucks

[15:38:59] <dstufft> I am not a stats guy

[15:39:41] <sumanah> I am not gonna take the time right now to dive into their methods & data, but yeah, I know it's very different

[15:40:09] <toad_polo> I was under the impression that libraries.io takes into account github stars, but also the stars of people in your dependency graph.

[15:40:14] <toad_polo> Like if you have 30 stars but 10 projects depend on you that each have 20,000 stars, you get some weighted boost to your influence.

[15:40:25] <toad_polo> Not sure if that is used for the trending part or just the influence part.

[15:40:59] <dstufft> toad_polo: that is cool

[15:41:05] <toad_polo> Oooh, this is useful: https://libraries.io/pypi/python-dateutil/usage

[15:41:09] <dstufft> ours is kinda dumb

[15:41:11] <toad_polo> (List of things that depend on a given library)

[15:41:19] <dstufft> it just computes a zscore of downloads

[15:42:23] <sumanah> please shout me down if this is a bad idea, but what if we looked a bit more at their trending list and maybe replaced ours with theirs?

[15:43:36] <dstufft> I have no loyalty to what exists now

[15:43:54] <dstufft> I only did it because "order by total number of downloads" sucked even harder

[15:44:15] <dstufft> if we think their data is more interesting or better then sure

[15:44:30] <sumanah> ok, I'll file a cool-but-not-urgent issue, thanks dstufft

[15:45:02] <sumanah> !logs

[15:45:02] <pmxbot> http://kafka.dcpython.org/channel/pypa

[15:51:11] <sumanah> toad_polo: the dependency tracing is potentially a gamechanger. I feel like looking at https://libraries.io/pypi/twine/usage and seeing what versions they require makes it possible for me to be a substantially better maintainer with my downstreams.

[15:51:41] <dstufft> I am confused by this list

[15:51:48] <sumanah> There are hundreds of projects that *require* old versions of Twine, evidently.

[15:51:50] <dstufft> or maybe by their definition of dependency

[15:53:06] <dstufft> cryptography does not have a runtime dependency on twine, but it claims it does

[15:53:56] <sumanah> https://github.com/librariesio/pydeps is what they use to check, I think

[15:55:03] <toad_polo> https://libraries.io/github/pyca/cryptography

[15:55:10] <toad_polo> It's in 'dev-requirements.txt'

[15:57:43] <sumanah> ok actually https://github.com/librariesio/bibliothecary/blob/50bf1e88a191fb8301ae6a5e00df4d22b05cce55/lib/bibliothecary/parsers/pypi.rb and https://github.com/librariesio/bibliothecary/blob/50bf1e88a191fb8301ae6a5e00df4d22b05cce55/spec/parsers/pypi_spec.rb look like where they look at requirements.txt files

[15:59:41] <sumanah> toad_polo: https://docs.libraries.io/overview#sourcerank has their ranking criteria

[16:11:22] <Rotonen> what's the intended upgrade path for bringing old deployments up to date now with enforced modern TLS? tsocks wrappering? setting up a local http proxy of pypi?

[16:11:57] <Rotonen> say a py2.4 based installation where the underlying cpython / openssl cannot have predicted the future

[16:13:36] <dstufft> Rotonen: uhhh

[16:14:01] <dstufft> undefined? I don't think there is one that been thought of

[16:14:51] <dstufft> A local mirror on a modern system could work

[16:15:29] <ngoldbaum> update cpython and openssl?

[16:15:32] <Rotonen> i get the rationales for the TLS only path, but still having http for the public read-only resources would help at times

[16:15:51] <Rotonen> ngoldbaum: well, yes, but that'll be a step-by-step process

[16:20:39] <Rotonen> dstufft: no reason to mirror as one can just reverse proxy and have the proxy terminate the "backend ssl", but out of curiosity, how large is the pypi repo currently?

[16:21:22] <sumanah> storage is more than a terabyte https://pypi.org/help/#mirroring

[16:21:33] <sumanah> I don't know the details, though

[16:21:40] <dstufft> 1270909974944

[16:21:42] <dstufft> bytes

[16:22:25] <Rotonen> that'd be somewhat manageable, thank you

[16:23:01] <dstufft> Rotonen: http only means arbitrary code execution for MITM resources

[16:23:57] <Rotonen> i'm aware of this, but it'd still need to be a valid zip payload, but sure, a full replacement mitm will get you

[16:24:58] <Rotonen> (and likewise for any checksum should there be a mechanism for that, turtles all the way down the plaintext route)

[17:32:55] <Rotonen> dug my self out of the pit with tsocks, thank you