pmxbot IRC Log Viewer

[00:25:12] <omry> Hi, I hit a new problem that I suspect is a result of the build-in-place change in 20.1.

[00:25:29] <omry> this is the output from my ci: http://paste.ubuntu.com/p/SmttnVKzPb/

[00:26:13] <omry> looks like the created build directory is confusing some tools

[00:27:12] <omry> in particular, pytest is blowing up once the build directory is there because it discovers the python files (or pyc) inside and they are in conflict with those in the directory.

[00:27:57] <omry> I can work around it by excluding build from pytest discovery, but this is likely to cause issue to many projects.

[00:46:21] <toad_polo> omry: I recommend raising an issue.

[01:46:14] <omry> toad_polo, done. https://github.com/pypa/pip/issues/8165

[04:03:57] <techalchemy> i wonder if that's actually intentional

[04:31:38] <pradyunsg> ronny: could we get a pytest fix for this? :puppy-face:

[04:40:18] <ronny> Yikes let me have the first coffee in the morning first

[04:42:18] <pradyunsg> ronny: whoop, sorry!

[04:42:32] <ronny> that one is a tricky one

[04:43:05] <ronny> its a entirely valid directory name

[04:43:20] <ronny> my initial suggestion would be to have a collect ogno

[04:43:29] <ronny> eh collect ignore for the build dir

[04:43:31] <pradyunsg> toad_polo, techalchemy: yea, the change-over in pip is intentional -- part of it is that the "copy" step was prone to issues/errors and generally slow. Another part is that pip has to start moving away from bodging to hide distutils-specific consistency issues, to "rely on the backend behaving correctly".

[04:44:13] <ronny> i really dont want to go and invent a "standard" for making directories "go"

[04:44:51] <ronny> pradyunsg: i suppose we can kind of assume that the build directory is going to be a python thing to ignore more often now?

[04:46:40] <techalchemy> I dont think pytest will be the only thing that has an issue with that

[04:47:07] <techalchemy> pradyunsg, why do you need to leave the build directory though?

[04:47:33] <techalchemy> like, can you not delete it when you're done

[04:47:35] <ronny> techalchemy: distutils leaves it, pip no longer ihdes grave distutils errors (which actually is awesome for setuptools_scm)

[04:48:28] <techalchemy> ronny, I just don't see why forcing garbage on users is good

[04:48:42] <ronny> tell that distutils

[04:48:45] <techalchemy> like, if we want to change distutils behavior we should change it

[04:49:00] <techalchemy> this isn't a war between us and the core developers avaik

[04:49:04] <techalchemy> afaik*

[04:50:17] <ronny> my personal impression is you cant really change distutils behaviour sanely - its all a big house of cards thats rather fragile, and even more so when setuptools is wrapped around

[04:50:43] <ronny> the build folder reappears because pip no longer "hides distutils dirty secret"

[04:51:02] <ronny> so now i hope someone comes around and fixes the intgration layer on the distutils/setuptools side

[04:51:44] <ronny> but thats going to be something error prone and time-consuming

[04:52:13] <techalchemy> i'm just not clear on what pip gains from holding the position that adding crap to a users directory that doesn't need to be there is good

[04:53:24] <pradyunsg> I wouldn't call it distutils' mess or crap.

[04:54:40] <pradyunsg> The benefit to this change, is less code in pip, faster build times for users and less distutils-specific workarounds in pip.

[04:54:46] <techalchemy> like, they have to use pip to do stuff, and decisions like these are confusing at best and leave users wondering whether developers are attentive to UX (not this one specifically but ones like it, in the aggregate) -- it's not crazy to clean up after a build that you invoked

[04:55:04] <ronny> techalchemy: thing is - pip has nothing to do with the folder that now appear, distutils has, pip is just no longer hiding everything in a tmpdir while breaking some other tools

[04:55:24] <techalchemy> i mean, if you don't have the folder, then you run pip, then you do have the folder

[04:55:29] <techalchemy> sorry to tell you but pip does have something to do with it

[04:56:35] <techalchemy> pip is choosing to use tools that produce the folder as part of its build process, and as the primary user facing packaging tool in python that unfortunately means any decision made in pip impacts basically everyone

[04:57:08] <ronny> so go and fix it in the right place?

[04:58:13] <techalchemy> idk, if it were up to me i'd delete the directory, sounds like it must be more complicated than that to be causing so much maintainability burden

[05:01:52] <ronny> you cannot safely know if its ok to delete the folder in pip

[05:02:33] <ronny> so far it was hidden by making a copy of the pakcage in tmp, breaking a few tools along the way and then dropping that tmpdir cause it was something under safe control

[05:03:06] <techalchemy> how can you not know if it's safe to delete a folder you created? that doesn't make any sense

[05:03:15] <techalchemy> either you've finished building from it or not

[05:04:04] <ronny> suit yourself, not going to repeat the details

[05:05:34] <pradyunsg> techalchemy: pip didn't create the folder? distutils did.

[05:05:35] <techalchemy> ronny, I'm super familiar with pip's old build logic, it was a bit annoying and there was a layer or two too many of indirection, but it was still better to build in isolation

[05:05:58] <techalchemy> pradyunsg, see above -- users don't care about the distinction

[05:06:08] <techalchemy> the folder is created as a result of invoking pip

[05:07:11] <techalchemy> what pip does under the hood, its 'internal api' if you will ;) isn't that important to someone who just wants to run things from the command line

[05:07:32] <pradyunsg> techalchemy: It's still a directory that's clearly not a big deal IMO. There's some tools that'd need to learn to ignore it if they haven't already. 🤷🏻‍♂️

[05:07:49] <pradyunsg> s/clearly//

[05:08:02] <techalchemy> idk, polluting a user's code seems bad

[05:08:15] <techalchemy> maybe nobody will care

[05:08:44] <pradyunsg> techalchemy: is creating a dist/ directory after setup.py sdist polluting the user's code?

[05:10:23] <pradyunsg> I don't see why build/ directory would be different. It's pretty well known that distutils / setuptools generate them, it's in the default Python.gitignore that GitHub maintains and doesn't affect anything other than the current build (unless you're doing a wheel build LMAO).

[05:11:13] <ronny> techalchemy: i actualy hope people make a big deal out of this and fix distutils/setuptools, pip hiding the problem is not a good solution, and a working solution is the worst enemy of a good solution

[05:11:17] <pradyunsg> toad_polo: any idea if we can get setup.py bdist_wheel to complain about dirty build directories?

[05:11:54] <techalchemy> pradyunsg, i got burned by that today actually while i was updating my patches

[05:12:02] <techalchemy> since you have a build directory in pip now

[05:12:03] <techalchemy> lol

[05:13:41] <ronny> pradyunsg: do you remember off-hand of the build isolation peps inform the backend of an intended tmpdir

[05:14:17] <ronny> the setuptools backend could be improved by setting the build-base to $THATTMPDIR/setuptools-build

[05:14:32] <pradyunsg> ronny: not sure what your question is.

[05:16:57] <pradyunsg> Ah. Okay, no, PEP 517 has no concept of a build directory / temporary directory.

[05:17:42] <pradyunsg> relevant line in PEP 517: "The source directory may be read-only. Backends should therefore be prepared to build without creating or modifying any files in the source directory, but they may opt not to handle this case, in which case failures will be visible to the user."

[05:17:42] <ronny> ok, thanks

[05:18:09] <ronny> that one makes ofr a horrendous ux ^^

[05:18:17] <pradyunsg> "The backend may store intermediate artifacts in cache locations or temporary directories. The presence or absence of any caches should not make a material difference to the final result of the build."

[05:22:49] <techalchemy> pradyunsg, how is your pandemic going btw

[05:36:59] <pradyunsg> techalchemy: haha. everyone around me and I am fine, as of right now, which is a good enough for me.

[05:37:06] <techalchemy> awesome

[05:37:23] <pradyunsg> Actively trying to not think about the burning world. :)

[05:38:43] <PSFSlack1> <deveshkusingh> ^ reminds me of the meme where the dog sits in the middle of the burning room, and says "it's fine"

[05:39:32] <techalchemy> lol, basically how i feel most of the time anyway

[06:07:30] <ronny> Ah, good old trauma anxiety, finally the external world adheres to the internal word

[07:11:26] <dstufft> i think changing away from isolation was a step in the wrong direction, but there isnt muuch to do about it now

[09:15:09] <pradyunsg> dstufft: I mean, we can still revert the patch and make a bugfix release -- though at that point, the discussion should really move over to GitHub to not exclude folks "not on #pypa-dev IRC". https://github.com/pypa/pip/issues/7555

[09:40:03] <ronny> dstufft: from my pov the directoy isolaton pip did was so far broken in a lot of edge cases and nobody fixed the details on the relevant places, so personally i welcome it going away

[11:09:42] <toad_polo> I definitely would have said something if I had gotten around to trying the beta...

[11:10:17] <toad_polo> Bernat and I were working on a PEP to solve this in a cleaner way.

[11:11:22] <toad_polo> I am surprised it's such a big deal though. Wouldn't the exact same thing happen for all non-PEP 517 builds? Or at least all editable installs?

[12:04:48] <ronny> Editable installs are built in place, so as far as I remember it is not always creating a build dir

[12:31:27] <toad_polo> Oh right.

[12:58:13] <toad_polo> pradyunsg: I think we can probably get rid of the build directory.

[12:59:21] <toad_polo> I'd like to do it as part of this, though: https://discuss.python.org/t/proposal-adding-a-persistent-cache-directory-to-pep-517-hooks/2303

[13:25:17] <cnx> in continuation of the build dir thing, could we please have pip clean?

[13:26:43] <cnx> afaik atm pip only create build/ during build process, but it'd be easier for users to be able to have `pip install . && pip clean .` or `pip install --auto-clean .`

[13:28:58] <cnx> wait this doesn't solve all issues like parallel build )-:

[13:35:48] <toad_polo> cnx: pip doesn't create that directory.

[13:35:55] <toad_polo> It wouldn't know what to clean.

[13:37:28] <cnx> i see, so to my understanding the linked discourse is similar to previous pip's behavior

[13:37:41] <cnx> but the directory is not deleted afterwards?

[13:37:41] <toad_polo> You're probably best off using `.gitignore` to exclude `build/`, then running `git clean` to remove files not tracked by git.

[13:38:21] <toad_polo> Not sure I follow. You're asking what the proposal above is?

[13:38:35] <cnx> yes the cache one

[13:39:07] <cnx> re git: that's what I'm doing at the moment

[13:39:36] <toad_polo> The proposal I linked to is that `pip` or any PEP 517 front-end would create a directory somewhere and pass it to the backend to use in arranging its build artifacts.

[13:40:40] <toad_polo> That could be a totally temporary place where the cache is deleted on every run, or one that persists between runs (i.e. the backend shouldn't assume it is empty, and should assume that if it is not empty that it contains the results of a previous build).

[13:41:00] <cnx> thanks, that seems clearer to me now

[13:41:22] <toad_polo> pip's previous behavior was to copy the entire build tree into a temporary directory then remove that directory once the build artifact was complete.

[13:42:17] <toad_polo> In the world of the proposal, `setuptools` would need to be updated to put the `build/` directory in that persistent cache directory instead of in CWD.

[13:44:09] <cnx> since I have a living resource to ask, sorry for being lazy, was the previous behavior copying the whole cwd instead of a generated sdist, since pip doesn't seem to have problem with setuptools' artifact anymore

[13:47:02] <toad_polo> The whole install root (which may or may not be the CWD).

[13:47:24] <toad_polo> `pip` doesn't go through a "generate sdist" step at the moment, but if it did you'd have the same problem that the build backend might generate random artifacts in the local directory.

[13:48:59] <cnx> thanks for clearing my mind and sorry for the incoherent thoughts

[13:49:42] <toad_polo> No worries, it can be complicated stuff.

[13:51:56] <pradyunsg> toad_polo: I'm on board.

[14:42:39] <techalchemy> the pep517 cache idea is interesting

[15:02:22] <dstufft> pradyunsg: ehh, im not convinced flip flopping after its been released is a desirable thing either. like it's out there now i think and unless a bunch of structural problems arise its probably worse to be like "lol just kidding"

[15:04:20] <dstufft> i think that moving away from that without a plan for dealing with dirty directories was suboptimal, but going back on it is probably even worse. at the end of the day i missed the discussion so i dont really get a say :)

[15:06:11] <techalchemy> ^

[15:07:26] <dstufft> i do think it probably means weve lost our chanveto use pip as a forcing function for ensuring building sdists work and to narrow down the total "install paths" that the python ecosystem has to support

[15:09:52] <dstufft> since it is unlikely we could ever make build an sdist then build a wheel as fast as just building a wheel without changes made to pep517 to support optimizations. optimizations which were rejected at the time because no tool currently had any need for them

[20:38:57] <Raito_Bezarius> Hi there, is there any reason the PATH_HASHER in https://github.com/pypa/warehouse/blob/master/warehouse/forklift/legacy.py#L1340 is always blake2_256 when the Simple API gives only info about SHA256

[20:38:59] <Raito_Bezarius> ?

[21:20:23] <dstufft> Raito_Bezarius: simple api gives sha256 because that is guaranteed to exist in a python install, so that is an obvious choice to use for a public api. The path hash it doesnt really matter, we had them both computed already and I just happened to pick blake2

[21:21:29] <Raito_Bezarius> dstufft: yes but it makes it impossible to guess the URL

[21:22:04] <Raito_Bezarius> given sha256 & filename & some metadata, I just cannot build the URL to the file

[21:22:12] <Raito_Bezarius> (I'm not using pip)

[21:23:13] <dstufft> youre not intended to guess the url. the url is not a stable api and may change in the future (and it has changed in the past).

[21:23:54] <Raito_Bezarius> dstufft: will there be a day where it'll stabilize itself?

[21:23:59] <Raito_Bezarius> are there any intended changes in the future?

[21:24:31] <Raito_Bezarius> is there a way to just go directly to the proper URL without going through an API first?

[21:25:36] <dstufft> its unlikely we will ever make any promises about the url structure of where files are hosted. youre inteded to use the api to discover it. There is a redirector for an url that doesnt have the hash in it thiugh

[21:25:50] <Raito_Bezarius> dstufft: where is this redirector?

[21:26:04] <Raito_Bezarius> that would solve all my problems

[21:26:17] <Raito_Bezarius> I can verify the hash locally

[21:31:14] <dstufft> https://warehouse.pypa.io/api-reference/integration-guide/#if-you-so-choose

[21:32:51] <Raito_Bezarius> Thanks so much dstufft :)

[21:35:51] <toad_polo> Raito_Bezarius: It doesn't really seem like it will solve "all your problems"

[21:36:02] <Raito_Bezarius> toad_polo: why?

[21:36:34] <Raito_Bezarius> indeed, https://files.pythonhosted.org/packages/source/k/keras-applications/Keras_Applications-1.0.8-py3-none-any.whl seems to be failing

[21:37:13] <dstufft> because a wheel file isnt a "source"

[21:37:22] <dstufft> you need to know what pyversion it was uploaded for

[21:37:32] <Raito_Bezarius> i know which python version

[21:37:38] <Raito_Bezarius> but i cannot guess the blake2 hash :D

[21:37:45] <dstufft> probably "py3" for that url

[21:38:15] <Raito_Bezarius> dstufft: there are no redirectors for wheels?

[21:38:42] <toad_polo> Because "here's some basically unsupported thing that might work incidentally for your use case" is not something to build on.

[21:39:14] <dstufft> note: pyversion is not exactly oython version, its a string sent along when the file was uploaded that is indended to represent the oython version

[21:39:19] <dstufft> but in practice it has varying values

[21:39:51] <toad_polo> You should build on top of the supported API, convince the python community to support your use case or accept that you are on your own IMO.

[21:39:54] <dstufft> py2.py3 or py3 or 3.5 or 3.6 etc

[21:40:01] <toad_polo> But I'll butt out. I think plenty of people have tried to steer you towards a more sensible course.

[21:40:17] <Raito_Bezarius> toad_polo: I believe multiple Nix/nixpkgs developers tried to explain the usecase years ago

[21:40:38] <Raito_Bezarius> But there were no explicit support for it, except use what is there

[21:41:09] <Raito_Bezarius> I definitely want to go for a more sensible course, but that's not always technically possible on my side, that's why I try to understand what's feasible

[21:41:30] <Raito_Bezarius> I understand what you say dstufft ; but I don't understand why it prevent a redirector for wheels

[21:41:41] <Raito_Bezarius> if I know exactly the python version I want

[21:41:53] <dstufft> https://bpaste.net/5TNQ

[21:42:24] <Raito_Bezarius> ah oops

[21:42:42] <dstufft> it works, but you have to know what value was sent in the pyversion field when it was uploaded

[21:43:00] <Raito_Bezarius> alright, makes sense

[21:43:13] <dstufft> not what version you're using

[21:43:13] <Raito_Bezarius> dstufft: do you know if there is a list of such values? somewhere? I can read the code if that's required

[21:43:47] <Raito_Bezarius> Okay, makes sense

[21:44:04] <dstufft> there is not, it was a free form field for uploaders to use. in practice the two popular uploaders have two different behaviors iirc

[21:44:29] <dstufft> twine i think will use whatever is in the wheel filename i think

[21:44:50] <Raito_Bezarius> dstufft: what about https://files.pythonhosted.org/packages/py2.py3/r/requests-oauthlib/requests_oauthlib-1.3.0-py2.py3-none-any.whl ?

[21:45:16] <dstufft> and i think setuptools might use the version of python that the person running setuo.py upload ran it under

[21:45:18] <Raito_Bezarius> it's a 404, but i think that py2.py3 is the right pyversion

[21:46:15] <Raito_Bezarius> Sorry to bother you dstufft, you're super helpful :)

[21:46:18] <dstufft> https://pypi.org/project/requests-oauthlib/#files

[21:46:29] <dstufft> it looks like it was set to 3.7

[21:46:36] <toad_polo> I don't understand why you don't want to use the API for discovering the packages...

[21:47:14] <dstufft> so they likely uploaded using setuptools instead of twine

[21:47:16] <toad_polo> How is that any better than using a legacy redirector service?

[21:47:50] <Raito_Bezarius> dstufft: makes sense

[21:48:00] <dstufft> problems around pyversion being inconsistent is why our redictor is only really recommended for sdists

[21:48:27] <dstufft> since that is always consistently "source"

[21:48:28] <Raito_Bezarius> toad_polo: I would like to avoid fixed output derivations

[21:48:32] <Raito_Bezarius> that's the reason

[21:48:47] <Raito_Bezarius> dstufft: makes sense, I'll see what I can do

[21:50:40] <dstufft> the other reason we discourage its use it its really only intended to suport the legacy URLs after we migrated to hash urls so the structure was forced on us, if we were designing it from scratch it would be a lot simpler

[21:53:58] <PSFSlack1> <di> FWIW I was curious how many unique values exist for that field: https://gist.github.com/di/93019b51582899c294cfd76ee2d0a4a3

[21:55:16] <dstufft> lol

[21:55:25] <dstufft> some of those are hilarious

[21:55:25] <Raito_Bezarius> :D

[21:56:09] <PSFSlack1> <di> yeah there's some real quality garbage in there

[21:56:34] <dstufft> that list gives a good indication of why we dont recommend the redirector for wheels

[21:56:41] <Raito_Bezarius> :'D

[21:56:47] <Raito_Bezarius> Okay, I'll resort to fixed output derivations

[21:56:55] <Raito_Bezarius> Cannot win against this

[21:57:06] <PSFSlack1> <di> my favorite is "2.7 (Tal Yarkoni's conflicted copy 2015" looks like someone's got their own personal 2.7 fork

[21:59:43] <dstufft> you could write your own redirector if you wanted

[21:59:58] <dstufft> you would have to host it somewhere

[22:00:19] <dstufft> but our redirector just uses the JSON api

[22:02:22] <dstufft> Make an URL like /{project}/{filename} look up /pypi/{project}/json read it for the url and then send a redirect

[22:04:16] <dstufft> https://github.com/pypa/conveyor/blob/eb15f0de91f945cfceedc52be8f8a6cd204b0e70/conveyor/views.py#L32-L103

[22:04:41] <dstufft> thats all we do. could adapt that and run it yourself somewhere

[22:06:42] <toad_polo> di: Tal Yarkoni is apparently a professor at UT Austin.

[22:10:12] <alanbato> My favorite is thon3.5, looking forward to the 3.6 release

[22:10:36] <Raito_Bezarius> dstufft: that's an option, indeed

[22:11:07] <Raito_Bezarius> thanks for the refs dstufft !

[22:11:10] <Raito_Bezarius> I appreciate it really

[22:15:23] <dstufft> Raito_Bezarius: no problem. i dont know much about nix to know what option is the least bad, but hopefully thats enough to get something doable working

[22:17:14] <Raito_Bezarius> dstufft: we're actually going for the API call, but it requires to do something like this: https://github.com/nix-community/poetry2nix/pull/94/files#diff-49b94f8a45fd693ca88fb2813d640536R108

[22:17:38] <Raito_Bezarius> it's not that bad I guess, we'll see how it works

[22:18:15] <dstufft> doesnt the poety lockfile put he urls in the lockfile

[22:18:32] <Raito_Bezarius> dstufft: unfortunately…

[22:18:54] <Raito_Bezarius> I terribly want to open an issue and submit such a PR for the lockfile, but I'm not sure it's a *very* good idea

[22:19:30] <dstufft> guess its just the hash and filename

[22:19:54] <dstufft> i dont use poetry so my memory was just bad

[22:21:02] <Raito_Bezarius> hash & filename indeed

[22:21:08] <Raito_Bezarius> and they include sha256 only

[22:21:14] <Raito_Bezarius> which is why we cannot predict URLs

[23:50:32] <techalchemy> dstufft, as you say, the urls aren't meant to be stable anyway

[23:51:16] <dstufft> techalchemy: to be clear, the urls themselves are. the method for generating them is not

[23:51:55] <dstufft> https://warehouse.pypa.io/api-reference/integration-guide/#querying-pypi-for-package-urls

[23:52:00] <techalchemy> well, sure, and for practical purposes the files at /a/b/c will be redirected, but it's not that relevant

[23:52:13] <dstufft> The URLs on PyPI for individual files are static and do not change.

[23:53:56] <techalchemy> dstufft, that's good to know. And I didn't realize that some aspects of the JSON api were ok to rely on

[23:55:30] <techalchemy> I don't see any major advantage to storing urls in lockfiles anyway

[23:56:21] <techalchemy> if i create a mirror and swap indexes it would just make my lockfile useless for example

[23:57:14] <dstufft> techalchemy: the json api is weird and undersoecified and probably not the right solution long term, but its supported and were not likely to make breaking changes in it at this point without good reason

[23:57:15] <dstufft> ehh

[23:57:28] <dstufft> reducing http requests by 50% in the common case seems worth it

Log file Viewer

Help | Karma | Search:

#pypa-dev logs for Wednesday the 29th of April, 2020