pmxbot IRC Log Viewer

[04:08:34] <dtux> the only reason i can see to invoke setup.py directly anymore is `setup.py sdist` and `setup.py check`. the former is mostly addressed here: https://github.com/pypa/pip/issues/1323. anyone know if the latter is possible with pip? `pip check` seems to be different.

[21:17:30] <dtux> is `setup.py check` something specific to setuptools?

[21:30:34] <toad_polo> dtux: It's probably from distutils.

[21:31:17] <dtux> toad_polo: do you know of another way to invoke it? i'm trying to find a way to avoid ever invoking setup.py directly

[21:31:37] <toad_polo> No. What does it do?

[21:33:07] <dtux> checks package metadata (e.g. https://github.com/Isilon/isilon_sdk_python/issues/12)

[21:33:29] <dtux> i want that check, but in a build-system-agnostic way (just adopted pyproject.toml)

[21:33:40] <toad_polo> That seems very distutils specific.

[21:33:41] <dtux> `pip check` seems to do something different

[21:34:22] <dtux> is that not something every built package so have correctly set?

[21:34:29] <dtux> should*

[21:35:11] <dtux> (otherwise, why does distutils call at least some fields "required"?)

[21:35:25] <toad_polo> I guess the way it's described is.

[21:35:35] <techalchemy> pip isn't a build tool

[21:35:40] <techalchemy> it's an installation tool

[21:35:45] <toad_polo> But the thing it's checking is generic.

[21:35:53] <dtux> `pip wheel` is a thing tho, and sdist will probably be a thing eventually

[21:36:01] <techalchemy> sdist is a thing already

[21:36:08] <techalchemy> but not via pip

[21:36:09] <dtux> woah, really?? cool!

[21:36:13] <techalchemy> sdists are going away

[21:36:14] <dtux> oh.. lol nvm

[21:36:24] <techalchemy> that's the whole point of pep517 / pyproject.toml (long term)

[21:36:32] <techalchemy> well, sort of

[21:36:43] <techalchemy> the ideal packaging state would be to have wheels for everything

[21:36:43] <toad_polo> Probably if you want your pipeline to be totally generic, use tox.

[21:36:47] <dtux> https://github.com/pypa/pip/issues/1323

[21:37:19] <techalchemy> obviously wheels for everything is pretty impractical atm so yeah, sdists are a thing still

[21:37:20] <toad_polo> Add a "check" testenv that invokes whatever check command you use.

[21:37:33] <techalchemy> yeah, I agree with toad_polo

[21:37:37] <toad_polo> I don't see why wheels for everything is impractical.

[21:37:48] <techalchemy> only for now it is

[21:38:02] <techalchemy> until people port over all their bad practices

[21:38:05] <toad_polo> Well, I mean it will be impossible to get rid of sdists.

[21:38:31] <toad_polo> But you can make a wheel for any project.

[21:38:38] <techalchemy> to get rid of them entirely, but effectively we can pretty much eliminate them from the normal install pipeline

[21:38:53] <techalchemy> that's not completely right, wheels are a compressed format

[21:38:57] <techalchemy> not everything can be compressed

[21:39:02] <toad_polo> Yeah, they're largely eliminated.

[21:39:10] <dtux> toad_polo: tox currently runs that check (but by invoking setup.py directly)... if i were to ever swap out build systems, seems silly that i should have to edit tox.ini

[21:39:13] <techalchemy> well, they don't ahve to be compressed i guess

[21:39:26] <techalchemy> dtux: plan on editing your code.

[21:39:36] <techalchemy> nothing you write will remain constant

[21:39:46] <dtux> why should the tests care about the build?

[21:39:53] <toad_polo> dtux: There is not and probably will never be a generic "check" utility.

[21:40:09] <toad_polo> Well, maybe not never, just that it's not on anyone's radar.

[21:40:11] <techalchemy> pep517 literally just got rolled out within the last few weeks and it broke the build process of literally like 30% of packaging in python, maybe more

[21:40:18] <toad_polo> Seems easy to build, though.

[21:40:34] <techalchemy> 'seems easy' to implement the generic build process too

[21:40:37] <techalchemy> and yet....

[21:40:50] <toad_polo> If it's something you are interested in, I recommend giving it a shot.

[21:40:52] <dtux> techalchemy: oh ya? interesting... i just adopted it, np (but my project is really simple)

[21:41:10] <techalchemy> fixes are only _just_ getting rolled out now

[21:41:19] <toad_polo> If you build something shoot an email to distutils-sig and/or mention it in here.

[21:41:33] <techalchemy> twine already checks the built distribution

[21:41:38] <techalchemy> i.e. the wheel

[21:41:47] <techalchemy> and will check readme etc

[21:41:57] <dtux> toad_polo: could be fun :) not sure i understand enough to do it quite yet, tho. how does distutils fit into the post pep517 world?

[21:42:11] <dtux> techalchemy: oh ya?? that could be what im looking for

[21:42:15] <techalchemy> distutils is a core python library

[21:42:16] <dstufft> sdists are not going away

[21:42:20] <techalchemy> it just installs things

[21:42:41] <techalchemy> dstufft: yeah, but we can pretend they don't exist a bit more and push people toward wheels

[21:42:45] <toad_polo> dtux: I'll tell you a secret, most of us are learning this as we go along.

[21:43:01] <dtux> techalchemy: so, e.g. this could be added to `pip check` by calling into distutils?

[21:43:16] <techalchemy> dtux: there is a lot to unpack in your question

[21:43:23] <dtux> toad_polo: =p eh, not a problem, just trying to catching up and tag along!

[21:43:35] <dtux> catch*

[21:43:37] <toad_polo> dtux: https://www.bernat.tech/pep-517-and-python-packaging/

[21:43:44] <techalchemy> generally speaking I wouldn't expect new commands or a plugin system to be added to pip

[21:43:52] <techalchemy> they are trying to remove things from pip atm

[21:43:54] <toad_polo> That's a 3 part series that should hopefully get you caught up.

[21:44:07] <techalchemy> ah bernat, that will be a great series indeed

[21:44:12] <dtux> toad_polo: awesome, ty (super recent too!)

[21:44:24] <techalchemy> i have not read that but I can promise it will be very informative

[21:44:29] <toad_polo> techalchemy: It's already done. 😛

[21:44:44] <techalchemy> toad_polo: awesome

[21:44:53] <techalchemy> thanks for the link, adding it to my queue of tabs

[21:44:59] <toad_polo> I encouraged him to break it into three parts because it was so thorough and broke up cleanly.

[21:45:18] <toad_polo> But then he did it and published all three at once. So utilitarian.

[21:45:26] <techalchemy> i'm still trying to recover from the breakages in pipenv :

[21:45:28] <techalchemy> :|

[21:48:59] <techalchemy> well and the rollout in general i guess

[21:49:43] <dtux> ha, twine check is yet another thing it looks like

[21:50:38] <dstufft> techalchemy: I mean, we shouldn't be pushing people to _not_ upload sdists, that'll just cause headaches

[21:51:08] <techalchemy> dstufft: sure, people should upload both

[21:51:22] <techalchemy> one of the build tools is building broken sdists i think though

[21:51:29] <techalchemy> haven't had time to verify

[21:53:51] <toad_polo> Any PEP 517 front end that builds sdists needs to have setuptools >= 40.6.0

[21:54:52] <toad_polo> pyproject.toml also needs to be explicitly added to manifests, at least for now.

[21:55:30] <toad_polo> We were going to fix it but then all the PEP 517 breakages happened and we didn't want to cause a bunch more.

[21:56:24] <techalchemy> dstufft: have pypi download #s moved at all in line with the pip 19 rool?

[21:57:11] <dstufft> rool?

[21:57:17] <techalchemy> roll*

[21:57:20] <techalchemy> typing is hard

[21:57:38] <dstufft> you mean have people switchded to using pip 19?

[21:58:02] <techalchemy> no i mean did things break sufficiently to cause people to keep trying to redownload stuff that was broken

[21:58:15] <dstufft> (sorry have a migraine so a little slow)

[21:58:17] <dstufft> uhh

[21:58:38] <techalchemy> np that is me 99% of the time

[21:58:47] <dstufft> pip caches file downloads so trying to install the asme thing twice wouldn't register as two downloads anyways

[21:59:01] <techalchemy> after 6h it would though

[21:59:12] <dstufft> the new simple data would show that, but we don't have historical data to compare it

[21:59:13] <dstufft> nah, the pip cache is forever for file download

[21:59:21] <techalchemy> ah

[21:59:23] <dstufft> or effectively forever

[21:59:30] <dstufft> like 10 years or something lol

[21:59:40] <techalchemy> i guess only CI builds then for actual downloads

[21:59:43] <dstufft> (we probably need to spin a background thread up toclean it up from time)

[21:59:46] <techalchemy> so people who thought it was their own fault

[22:00:08] <dstufft> Dunno if you saw https://discuss.python.org/t/new-pypi-statistics-simple-api-requests/837

[22:00:15] <techalchemy> i didn't

[22:00:23] <techalchemy> is it for HEAD or OPTIONS or something

[22:00:35] <dstufft> old data was only storing the actual file downloads

[22:00:42] <dstufft> newdata is /simple/<whatever>/

[22:01:09] <techalchemy> hm, i have no idea how to access bigquery

[22:01:14] <techalchemy> last itme i tried the link was broken

[22:01:21] <techalchemy> or i couldn't figure out google

[22:01:41] <dstufft> I normally go to https://console.cloud.google.com/ and there's a BigQuery link on the left side

[22:02:01] <dstufft> you might have to put aCC in, since they'll bill you for queries over a threshold each month

[22:02:51] <techalchemy> is there a good number of people who know sql / help out with this?

[22:03:38] <techalchemy> this being warehouse i guess

[22:03:58] <dstufft> The BigQuery stats cycle through linehaul, but it's mostly me working on that

[22:12:51] <techalchemy> dstufft: did you write that in trio? I feel like i talked to someone about that

[22:13:12] <dstufft> it was in trio yeq

[22:13:15] <dstufft> new version is in rust

[22:13:15] <dstufft> lol

[22:16:02] <techalchemy> did you find rust hard to pick up? the syntax is kind of awkward imo

[22:16:18] <techalchemy> i didn't like it much when i looked at it, but maybe you get used to it

[22:16:44] <techalchemy> maybe you need to remind yourself that it's not CPP

[22:19:12] <dstufft> I don't know C++ so I didn't have that problem

[22:19:13] <dstufft> It took me like 2 weeks I guess to rewrite linehaul in rust

[22:19:18] <dstufft> which was bootstrapping from zero rust knowledge

[22:19:35] <dstufft> https://github.com/pypa/linehaul/tree/rust

[22:20:33] <techalchemy> dstufft: so you like it? you hit a constraint i guess running it in python? what was the issue?

[22:21:43] <dstufft> eh, old linehaul was running under PyPy, adding the new stats sent more traffic than a single pypy daemon could handle and we didn't have great ways to scale a TCP (vs http) server out to multiple machines in the pypy infra

[22:22:03] <dstufft> so I was moving to having fastly write logs to s3, and then s3 event -> lambda to have linehaul run as a lambda

[22:22:36] <dstufft> problem was, lambda means pypy didn't make sense, becuase jit warmup and we wer eusing pypy cuz parsing all the log lines were STUPID slow on cpython

[22:22:46] <dstufft> so rust was fast

[22:23:12] <dstufft> also deploying python to lambda is kinda annoying, but rust is dead simple

[22:27:19] <techalchemy> dstufft: had you considered any of the alternative options / have any experience with things like airflow for this kind of stuff? maybe overkill for just log transfers

[22:29:12] <dstufft> never eally bothered, linehaul itself is rpetty simple

[22:29:53] <techalchemy> i'm asking with my work ETL pipeline in mind

[22:30:54] <techalchemy> which we've rewritten about 10 times in the last 2 years, most recently from celery (incomprehensible) -> dramatiq (undebuggable with bizarre memory leaks) -> trio (didn't finish that yet) -> very simple implementation on top of airflow

[22:31:14] <techalchemy> which takes forever to spin up worker tasks but at least it actually runs

[22:31:38] <dstufft> for pip 6+ bigquery is basically just "load this bit from json, stick it in a struct, serialize back to json and write it out to bigquery"

[22:32:09] <techalchemy> do you have a sense of how much faster the serialization is?

[22:33:05] <techalchemy> that was a big bottleneck for us when we were looking at web frameworks, we ditched python entirely as a result

[22:33:33] <dstufft> I don't know for sure no

[22:34:31] <techalchemy> i should just make some time to mess with it

[22:35:20] <techalchemy> ok so from the dashboard, how do i get to pypi

[22:49:59] <dstufft> techalchemy: should be ableto just query it

[22:50:18] <techalchemy> whats the table name or whatever

[22:50:38] <techalchemy> or maybe the links are wrong but the info is still right

[22:51:36] <dstufft> https://bpaste.net/show/518b8caac680

[22:51:39] <dstufft> example query

[22:58:11] <techalchemy> thx

[22:58:30] <techalchemy> that looks expensive

[22:58:32] <techalchemy> ooh limit 100

[22:59:01] <techalchemy> oh god we still get like 30k downloads a day

[22:59:08] <techalchemy> that's awful i should fix things

[22:59:57] <techalchemy> super interesting, this is nice

[23:00:22] <techalchemy> i should configure my various codes to report user agents and such

[23:00:30] <techalchemy> is that how that gets collected?

[23:11:27] <dstufft> techalchemy: yea user agents, though it's a white list of supported user agents, so we'd have to add support for it

[23:11:47] <dstufft> I mean to write a PEP at some point to make the user agent format standard so other tools can implement it

[23:12:27] <techalchemy> oh cool i dont care that much

Log file Viewer

Help | Karma | Search:

#pypa logs for Saturday the 9th of February, 2019