pmxbot IRC Log Viewer

[06:45:35] <pradyunsg> Playing around w/ PyPI downloads data shows... a whole bunch of packages w/ an anomalous 80k downloads on Py2 w/ pip 9.0.3.

[06:45:47] <pradyunsg> https://usercontent.irccloud-cdn.com/file/3rShxEqI/image.png

[06:46:04] <pradyunsg> *19.2.3

[08:27:21] <graingert> ronny: what is an "entrypoint name" in the context of setuptools_scm

[08:27:44] <graingert> looking to move https://github.com/m-labs/nmigen/blob/master/setup.py#L5-L9 into a declarative config

[08:28:19] <graingert> https://github.com/m-labs/nmigen/pull/324/files#diff-380c6a8ebbbce17d55d50ef17d3cf906R27

[08:28:19] <graingert> should this be attr: local_scheme.local_scheme ?

[09:22:56] <graingert> ronny: how do I register entry point names?

[09:23:19] <graingert> Also is relative_to __file__ needed anymore?

[09:25:09] <graingert> I think a comprehensive porting guide from the legacy options to the modern declarative options is required

[12:42:46] <graingert> ronny: I've figured how to add an entrypoint but the maintainer doesn't want to add a dep

[12:43:15] <graingert> is there a way to define a version scheme inline without using the setuptools_scm legacy options?

[13:00:32] <graingert> it's fair enough that they don't want to add a dep - I think setuptools_scm might need updating to accept "attr: ..." options to allow inline definitions of callable get_version kwargs

[13:03:20] <ronny> graingert: thats a setuptools level thing, setuptools_scm doesnt touch setup.cfg wit a 10ft pole, as for pyproject.toml, there is not mechanism yet, for simple variants i beleive we can do a pretty simple declarative one, for anytihng complesx, use deps

[13:06:20] <graingert> ronny: yeah the way it parses "attr: ..." is after the config loads per config option

[13:06:42] <graingert> as I understand it, it's not a setup.cfg thing per-se but something that's processed by each config option

[13:07:18] <whitequark> one thing i hoped might be possible with setuptools_scm is a declarative way to describe versions, something like fmt-strings

[13:08:01] <graingert> eg it reads "attr: some.pkg:module" then something does .startswith("attr:") to use pkg_util.resolve_name

[13:08:33] <whitequark> setuptools_scm describes four cases here: https://github.com/pypa/setuptools_scm#default-versioning-scheme

[13:09:10] <graingert> ronny: whitequark is the maintainer I mentioned earlier

[13:09:12] <whitequark> if i could configure each of the four cases with options, something like `distance_no_clean = "{tag}+{node}.dirty"` that would be ideal

[13:10:13] <whitequark> (actually i'd be fine with two cases personally, but setuptools_scm seems to be built to handle more complex dirty suffixes than i need)

[13:10:38] <graingert> ronny: my current attempt moves the config from an inline python function to a pypi package: https://github.com/nmigen/nmigen/pull/358/files#diff-522adf759addbd3b193c74ca85243f7dR2

[13:12:07] <whitequark> being able to specify a function inline (or really, anywhere that is not "in a dependency") would also be fine with me

[13:12:22] <whitequark> but i understand this goes against the spirit of the declarative approach, which is fair

[13:40:23] <ronny> im fine with addin a declarative aporach for the formatting in general

[13:40:43] <ronny> currently its a bit hard to find time for it tho + there is the mess with pytest unfolding

[13:42:23] <graingert> ronny: I think it would be a case of updating callable_or_entry to support a pkg_util.resolve_name string

[13:43:04] <graingert> I've made a backport in pkg_util_resolve_name but it's py3 only atm

[13:45:26] <ronny> i just have to load a fake entrypoint from setuptools

[13:45:35] <ronny> but i dont think i want to do that

[13:45:55] <ronny> that attr: thing is something from setultools core for config loading, i wont reimplement that in setuptools_scm

[13:46:18] <graingert> a fake entrypoint?

[13:46:22] <ronny> what i can do however is adding support for declaring what you needed as a dict

[13:46:44] <ronny> aka instantiate a entrypoint from an attr string and then call load

[13:47:11] <graingert> ?

[13:47:20] <graingert> an entrypoint from the package intself?

[13:47:25] <graingert> itself*

[13:49:30] <ronny> no, it would be `pkg_resources.Entrypoint.parse("fake = {attr}".format(attr=thatattr)).load()`

[13:49:48] <ronny> evil, straiht forward, and done

[13:50:17] <ronny> but still, why do that stuff when i can provide a way to turn a declarative dict into something like that function

[13:50:38] <ronny> for funky stuff, make a package, for simple preference, change the dict

[13:50:50] <whitequark> i'd be very happy with that solution

[13:51:11] <whitequark> (the declarative one)

[13:56:20] <ronny> personally i cant dig deepr into it before the weekend

[13:56:41] <whitequark> absolutely no rush

[13:57:15] <ronny> btw, until it is fixed, its totally a absolutely valid solution to just leave that bit in setup.py, its not like setup.py will go away soon

[13:57:55] <whitequark> yup, that's what i'm going to do

[13:58:10] <ronny> graingert: can you open a issue about that, i have to run for sorting details with my toddler and i think iä'll forget

[14:15:17] <graingert> ronny: whitequark: see https://github.com/pypa/setuptools_scm/issues/419 and https://github.com/pypa/setuptools_scm/issues/418

[14:16:24] <whitequark> thank you graingert!

[14:20:30] <graingert> whitequark: might be worth asking the person who was having problems to try https://github.com/nmigen/nmigen/pull/358

[14:20:45] <graingert> eg pip installing from my graingert:use-declarative-setup-config branch

[14:21:58] <whitequark> graingert: could you ask them? I forget the syntax for asking pip to install from a branch...

[14:28:50] <graingert> done

[14:31:37] <whitequark> thanks!

[19:12:46] <PSFSlack> <deveshkusingh> pradyunsg: o/

[19:14:48] <PSFSlack> <deveshkusingh> So as per the comment at https://github.com/pypa/pip/issues/7951#issuecomment-614226534 , can new PRs still be made for existing issues, assuming that they will only be merged after April 30th?

[19:25:37] <pradyunsg> @deveshkusingh: o/

[19:25:45] <pradyunsg> Yup yup.

[19:26:55] <pradyunsg> Also, expect reviews on them to be delayed (at least from my end) since there's a bunch of other stuff being done around-and-toward the release. :)

[19:28:23] <PSFSlack> <deveshkusingh> Yes sure no worries, I see everyone of the pip maintainers hard at work doing fixes geared towards the release :slightly_smiling_face:

[19:29:49] <PSFSlack> <deveshkusingh> anyways, I will tag relevant folks in the PRs/issues so that they can see it whenever they can

[19:34:24] <pradyunsg> @deveshkusingh: could you be a little less eager in mentioning/adding folks for reviews? Almost all of pip's maintainers get notifications for new PRs and nearly all comments on existing issues/PRs.

[19:36:07] <graingert> ronny: how about instead of a dict, use a list of templates with extras_require style conditions?

[19:37:13] <PSFSlack> <deveshkusingh> pradyunsg: Oops I wasn't aware of maintainers getting automatic notifications. Thanks for letting me know :slightly_smiling_face:

[19:37:26] <graingert> ronny: scheme = [

[19:37:26] <graingert> "{node}; clean", "+{...}; distance",

[19:40:44] <pradyunsg> @deveshkusingh: No worries. :)

[19:42:43] <PSFSlack> <deveshkusingh> pradyunsg: So maintainers -> folks listed at https://github.com/orgs/pypa/people

[19:43:16] <PSFSlack> <deveshkusingh> I think there will be a way to see just pip maintainers, but perhaps that's not visible to regular contributors

[19:43:56] <pradyunsg> Correct. There's a GitHub "team" in the `pypa` organization for pip's maintainers.

[19:44:14] <pradyunsg> Those teams aren't visible publicly.

[19:44:54] <devesh> okay, but it's a subset of pypa folks.

[19:45:18] <devesh> Cool, I'll be less trigger happy tagging those folks :)

[19:46:15] <pradyunsg> :)

[19:46:23] <pradyunsg> Yes, it's a subset.

[19:49:48] <devesh> I guess things will kinda slow down after the release on the dev front and pick up on the fixing regressions/new issues front

[22:13:37] <pradyunsg> Anyone around who can run a PyPI downloads BigQuery for me? 🙈

[22:14:29] <pradyunsg> EWDurbin or @di maybe?

[22:15:02] <EWDurbin> pradyunsg: did you exhaust your free tier quota?

[22:16:05] <pradyunsg> yup.

[22:16:10] <EWDurbin> womp

[22:16:22] <EWDurbin> depending on how big of a query, i might be able to run it for you

[22:16:31] <EWDurbin> PSF doesn't have a magic unlimited bit on GCP

[22:16:46] <EWDurbin> if it's a gigantic query di might be able to help

[22:17:42] <pradyunsg> I didn't really realize that data studio use BigQuery behind it -- it makes sense in retrospect.

[22:18:06] <EWDurbin> what's the query pradyunsg ?

[22:18:33] <pradyunsg> Wanna see the PyPI downloads by implementation for past 3 weeks basically. https://www.irccloud.com/pastebin/B0iHLYSP/

[22:18:48] <pradyunsg> Made with pip 20.0.2

[22:18:54] <pradyunsg> Before you run it...

[22:21:35] <EWDurbin> pradyunsg: ?

[22:21:37] <pradyunsg> I wonder if there's any other thing that's worth adding here -- I don't know how big the query would get if we add `details.distro` or `details.system` to this.

[22:23:15] <pradyunsg> I guess `details.distro.name` and `details.distro.version` would likely be enough information? I know AWS downloads show up in `distro.name`. Does GCP information come through in this?

[22:26:02] <EWDurbin> pradyunsg: cost of BQ queries are based on how much data is queried... not columns so if you want more columns just holler

[22:27:50] <pradyunsg> EWDurbin: My understanding is that additional columns count as more data.

[22:28:10] <EWDurbin> makes sense, i guess more columns means reading more data.

[22:29:05] <EWDurbin> anyway.

[22:29:06] <EWDurbin> https://pastebin.com/PmJKDsGW

[22:29:08] <EWDurbin> ran that

[22:29:23] <pradyunsg> whee

[22:29:30] <EWDurbin> pradyunsg: do you want results as json or csv

[22:30:00] <pradyunsg> csv is easier -- I can dump it into a local SQL table. :3

[22:30:40] <pradyunsg> EWDurbin: ^

[22:30:47] <EWDurbin> yep, i saw

[22:30:51] <EWDurbin> working on getting it for ya

[22:31:01] <pradyunsg> techalchemy: this query ran on 2418112220 records.

[22:31:14] <EWDurbin> yeah... there's u

[22:31:16] <EWDurbin> uh

[22:31:20] <techalchemy> wow

[22:31:33] <EWDurbin> 79,103,964,798 rows in the downloads table

[22:31:34] <pradyunsg> EWDurbin: put in google drive? :P

[22:31:41] <EWDurbin> pradyunsg: yeah... working on it

[22:31:48] <EWDurbin> pradyunsg: google auth is garbage

[22:31:48] <techalchemy> we should just put it into yaml files in a gh repo

[22:31:52] <pradyunsg> EWDurbin: yaayayay! Thank you so much! ^>^

[22:32:45] <pradyunsg> EWDurbin: but, that's like, all of history, correct?

[22:33:02] <EWDurbin> well, i mean with tons of asterisks

[22:33:24] <pradyunsg> Like, since 2016, with all dropped data, all duplications etc?

[22:33:30] <EWDurbin> yeah

[22:33:39] <pradyunsg> :)

[22:33:44] <EWDurbin> https://docs.google.com/spreadsheets/d/1w5GD_aTRQbkNEduVoanYgEeJuWiAgXkIvWkeJ_fheOE/edit?usp=sharing

[22:33:49] <EWDurbin> pradyunsg: maybe that works?

[22:34:19] <pradyunsg> It does! \o/

[22:34:23] <EWDurbin> cool

[22:34:38] <EWDurbin> (just for reference, won't be able to run "unlimited" queries for ya until your quota refils"

[22:34:56] <pradyunsg> EWDurbin: yup yup.

[22:35:08] <EWDurbin> PSF pays for google (using a google donation, but still it is like actual dollars)

[22:36:11] <travis-ci> pypa/twine#1365 (master - f7402e0 : Brian Rutledge): The build passed.

[22:36:11] <travis-ci> Change view : https://github.com/pypa/twine/compare/5482d6877df0cd90545c4708e5abe13b359f8a08...f7402e0d2e1c1e6ecde61dc8bcba515b807a48a0

[22:36:11] <travis-ci> Build details : https://travis-ci.org/pypa/twine/builds/675513281

[22:36:23] <EWDurbin> pradyunsg: note i rewrote your query to use standardsql and query the new table (announcement tomorrow) https://pastebin.com/PmJKDsGW

[22:36:51] <EWDurbin> summary of new table: https://gist.github.com/ewdurbin/6ff69333e5cb7cb77b9b7db4e291f22b

[22:37:06] <EWDurbin> it also includes a new table that might interest you :-D

[22:37:38] <PSFSlack> <di> Same is not true for me though, so feel free to send any massive queries my way!

[22:37:46] <EWDurbin> horray!

[22:37:56] <EWDurbin> DI is actually BQaaS

[22:38:06] <pradyunsg> @di: \o/

[22:38:56] <pradyunsg> EWDurbin: I was looking at "pypidev" and was about to ask you. :)

[22:39:03] <pradyunsg> EWDurbin: thanks for sharing the query.

[22:39:21] <PSFSlack> <di> Also I think you can save queries and share them for other people to run? Might be easier in the future

[22:40:16] <pradyunsg> Yup yup -- I was fumbling w/ buttons and copy-paste was easier. 🤷

[22:41:08] <pradyunsg> Now to figure out what do people using the current pip version on Python 2 look like. :)

[22:47:07] <travis-ci> pypa/pip#15745 (master - 2f3a1be : Paul Moore): The build passed.

[22:47:07] <travis-ci> Change view : https://github.com/pypa/pip/compare/8c118c8f3a6fd850e49256c51440142526a3d7c6...2f3a1be1185e3434c0c8d9cc58d4271beff1d122

[22:47:07] <travis-ci> Build details : https://travis-ci.org/pypa/pip/builds/675508452

Log file Viewer

Help | Karma | Search:

#pypa-dev logs for Wednesday the 15th of April, 2020