PMXBOT Log file Viewer

Help | Karma | Search:

#pypa logs for Wednesday the 29th of April, 2020

(Back to #pypa overview) (Back to channel listing) (Animate logs)
[05:03:53] <dude-x> huh. installed pip 20.1 thought i'd try the new beta resolver to upgrade a few packages. nothing gets upgraded. :/
[18:26:52] <Raito_Bezarius> Hello, anyone knows why files.pythonhosted.org/packages/py2.py3/r/requests-oauthlib/requests_oauthlib-1.3.0-py2.py3-none-any.whl is a 404 though on PyPI, it seems like such a file should exist (?)
[18:31:25] <tos9> Raito_Bezarius: You almost certainly aren't really supposed to hotlink to random files on PyPI in that way -- is that what you're trying to do there?
[18:32:21] <ngoldbaum> that file is available at files.pythonhosted.org/packages/a3/12/b92740d845ab62ea4edf04d2f4164d82532b5a0b03836d4d4e71c6f3d379/requests_oauthlib-1.3.0-py2.py3-none-any.whl FWIW
[18:32:27] <tos9> Raito_Bezarius: That wheel does appear here: pypi.org/project/requests-oauthlib/#files, so if you're .. what ngoldbaum said :)
[18:32:41] <tos9> Basically pip will find it. Whether it has some hotlink-available version I doubt is guaranteed
[18:32:44] <ngoldbaum> i don't think there's a stable URL scheme for wheels
[18:32:48] <tos9> But if it is,... also what ngoldbaum said :D
[18:32:51] <Raito_Bezarius> tos9: I don't have the choice
[18:32:52] <ngoldbaum> there is one for the sdist though
[18:32:55] <tos9> Raito_Bezarius: Why not?
[18:33:08] <Raito_Bezarius> tos9: I'm statically fetching dependencies, that requires predictable URLs
[18:33:14] <tos9> Raito_Bezarius: No it doesn't :)
[18:33:23] <tos9> Raito_Bezarius: `pip` is the way to statically fetch dependencies as well.
[18:33:28] <tos9> And knows how to find your wheel.
[18:33:34] <ngoldbaum> pip is the way to install wheels
[18:33:38] <Raito_Bezarius> tos9: pip is not reproducible and at the moment has no proper dependency solver
[18:33:41] <tos9> `pip download requests-oauthlib==1.3.0` or whatever
[18:33:47] <ngoldbaum> if you can live without the wheels there is a static URL scheme for the sdist
[18:33:49] <Raito_Bezarius> Also, pip has no --offline flag
[18:34:08] <Raito_Bezarius> This is what I meant by statically fetching the dependencies
[18:34:10] <tos9> Raito_Bezarius: What does this have to do with dependency solving? Or reproducibility
[18:34:19] <tos9> Raito_Bezarius: pip supports offline installs.
[18:34:21] <Raito_Bezarius> tos9: I'm using Nix/NixOS to do it
[18:34:24] <ngoldbaum> pypi.io/packages/source/o/requests-oauthlib/requests-oauthlib-{{ version }}.tar.gz
[18:34:30] <Raito_Bezarius> And I would like to let Nix do the proper dependency solving
[18:34:33] <tos9> Raito_Bezarius: Sure, NixOS knows how to do all this (and uses pip for part of it I believe)
[18:34:43] <Raito_Bezarius> tos9: poetry2nix performs exactly what I've showed
[18:34:51] <ngoldbaum> if you're packaging it for nix then you should use the sdist i would think
[18:34:51] <Raito_Bezarius> it uses files.pythonhosted.org/<guess the URL>
[18:34:52] <ngoldbaum> not the wheel
[18:35:11] <Raito_Bezarius> sdist is not always available or easily available due to pyproject.toml stuff
[18:35:14] <Raito_Bezarius> take SciPy
[18:35:17] <ngoldbaum> huh?
[18:35:27] <ngoldbaum> scipy's sdist is available?
[18:35:42] <Raito_Bezarius> It is available, but the setup.py has assumptions on build dependencies
[18:35:43] <ngoldbaum> do you mean it's not easily installable, sure, but if you're nix then you have a package for scipy
[18:35:43] <tos9> (there are other things for which the sdist isn't available, you could have said tensorflow)
[18:35:50] <ngoldbaum> and all of scipy's dependencies
[18:35:51] <tos9> But that doesn't change this being correct or supported :)
[18:36:07] <Raito_Bezarius> ngoldbaum: nixpkgs has a certain package for SciPy
[18:36:14] <Raito_Bezarius> I want to generate arbitrary expressions for any type of package
[18:36:23] <ngoldbaum> ok, well if you want to install wheels
[18:36:24] <tos9> Raito_Bezarius: You cannot.
[18:36:30] <ngoldbaum> you should probably use pip to do it
[18:36:50] <Raito_Bezarius> github.com/nix-community/poetry2nix/blob/4ebbfdacd1b14a6713c634c2af6e3f880e416fd3/lib.nix#L95
[18:36:55] <Raito_Bezarius> tos9: Why?
[18:37:15] <ngoldbaum> it may be the case that URL scheme worked at one point
[18:37:19] <ngoldbaum> but i don't think it works now...
[18:37:19] <Raito_Bezarius> Is it a "You cannot" due to arbitrary limitations in PyPI/pip or "You cannot" due to "it's a bit complex"?
[18:37:34] <tos9> Raito_Bezarius: Nothing is ever arbitrary about limitations :)
[18:37:35] <ngoldbaum> e.g. pypi moved to the URLs with the hashes for improved security a while ago
[18:37:36] <Raito_Bezarius> ngoldbaum: Is there any reason for the URL scheme to require those xx/yy/zzz, are those computable?
[18:37:48] <ngoldbaum> not without having the file already i think
[18:37:50] <Raito_Bezarius> I'm okay with hashes in the URL, but are those hashes documented?
[18:38:41] <ngoldbaum> there's a pep where all this is documented, i don't have a reference for it off the top of my head
[18:39:01] <Raito_Bezarius> python.org/dev/peps/pep-0503 this one?
[18:39:47] <Raito_Bezarius> ah maybe this one python.org/dev/peps/pep-0458
[18:40:25] <ngoldbaum> i'm not sure, also keep in mind that there are peps that are in the database but where never implemented or have since been superceded
[18:40:39] <ngoldbaum> unfortunately keeping all this straight approaches full-time job :/
[18:41:12] <Raito_Bezarius> Makes sense
[18:41:35] <Raito_Bezarius> I'm just trying to see how feasible it is to get metadata on PyPI, because that would make Python packaging on Nix so simple
[18:41:57] <Raito_Bezarius> ngoldbaum: there must be some way for pip to find those URLs, right?
[18:42:29] <ngoldbaum> i don't know but also keep in mind that as tos9 told you already they don't want you to reimplement what pip does
[18:42:44] <ngoldbaum> it's probably a lot easier just to use pip for whatever you're trying to do
[18:44:00] <Raito_Bezarius> i'd love to, but pip is just not doing the job alas
[18:44:12] <Raito_Bezarius> maybe in the future, with the new dependency resolving & stuff
[18:44:22] <ngoldbaum> you can tell pip to not install dependencies of a package
[18:44:27] <tos9> Raito_Bezarius: You're making incorrect assumptions
[18:44:37] <Raito_Bezarius> tos9: please, elaborate
[18:44:37] <tos9> Whatever you're doing yourself, youc an use pip to do
[18:44:52] <tos9> Raito_Bezarius: if you don't want pip doing the install, don't use pip to do the install
[18:44:56] <Raito_Bezarius> AFAIK, I cannot ask pip to dump a lockfile right now
[18:45:02] <tos9> If you don't want pip doing the dependency resolution, don't use pip to do the dependency resolution
[18:45:11] <ngoldbaum> what does a lockfile have to do with this?
[18:45:12] <tos9> If all you want is a supported client for PyPI that knows where to find files, pip is that
[18:45:20] <tos9> Everything else, go wild, use what you want
[18:45:23] <Raito_Bezarius> ngoldbaum: I want to pin my dependencies
[18:45:29] <tos9> Raito_Bezarius: pip already supports that
[18:45:33] <tos9> if again, you don't want to use it, don't
[18:45:33] <ngoldbaum> you want to use pip as your widget to convert a package name to a url
[18:45:35] <tos9> still use pip to find the files
[18:45:37] <ngoldbaum> where does a lockfile come in?
[18:45:55] <Raito_Bezarius> Sorry, the discussion just got confusing, I'll summary what I'm trying to do
[18:46:04] <tos9> Raito_Bezarius: That's exactly what I was hoping you'd do!
[18:47:18] <Raito_Bezarius> Given a requirements.txt or pyproject.toml, I'd like to ensure that dependencies are pinned (e.g. I need a lockfile, right?) and I want to be able to install them as a derivation from Nix, I cannot just do `pip install XXX` because it'll use Internet to install dependencies and I don't want to use fixed-output derivation because they're cumbersome for development
[18:47:33] <Raito_Bezarius> Also, I would like to install wheels and not sdist
[18:47:36] <Raito_Bezarius> Because of tensorflow & friends
[18:48:20] <Raito_Bezarius> (let us ignore special cases such as local URLs in requirements.txt, editable and git URLs for example.)
[18:49:26] <Raito_Bezarius> So what I tried is to use poetry2nix which uses the lockfile from Poetry to download the dependencies (and bypass pip I guess), the thing is that as you said, I cannot rely on the URL scheme for security reasons
[18:49:34] <ngoldbaum> isn't a requirements.txt with pinned dependencies exactly what you want?
[18:49:43] <Raito_Bezarius> I'm pretty sure the data in the lockfile from Poetry includes enough information to rebuild the hashes that PyPI uses
[18:49:58] <Raito_Bezarius> ngoldbaum: Well, that was until I encountered dependencies which installed contradictory versions
[18:50:16] <Raito_Bezarius> I didn't investigate really, but I'm okay with requirements.txt if it's enough
[18:50:20] <ngoldbaum> then the requirements.txt didn't have all the packages in the virtualenv
[18:51:10] <Raito_Bezarius> That's possible, I preferred to be on the safe side with the poetry lock
[18:51:12] <ngoldbaum> you want to have a listing of all of your full python dependency tree along with the versions of the packages, pip freeze in a virtualenv will give you that
[18:51:22] <Raito_Bezarius> but pip freeze require to have it already installed
[18:51:26] <Raito_Bezarius> right?
[18:51:33] <ngoldbaum> to have pip installed?
[18:51:35] <ngoldbaum> yeah
[18:51:38] <Raito_Bezarius> to have the dependencies installed
[18:51:43] <ngoldbaum> sure
[18:51:51] <ngoldbaum> you'd be starting from a known good set, right?
[18:52:08] <Raito_Bezarius> Yes
[18:52:24] <Raito_Bezarius> Well, let's admit the bootstrap problem is solved
[18:52:26] <ngoldbaum> so then in that virtualenv where you have the known good set of dependencies installed, you run pip freeze
[18:52:27] <Raito_Bezarius> That's not really hard
[18:52:33] <ngoldbaum> that's the requirements.txt file you want
[18:52:44] <ngoldbaum> the poetry lock file would presumably be the same information?
[18:52:48] <Raito_Bezarius> Indeed
[18:53:03] <Raito_Bezarius> But `pip` does not interface very well under a very hardened sandbox for installation phase
[18:53:23] <ngoldbaum> so wherever you're doing this doesn't have access to the internet?
[18:53:38] <Raito_Bezarius> I can have Internet but I don't want to use Internet
[18:53:42] <Raito_Bezarius> The rationale is extra caching, etc.
[18:53:44] <ngoldbaum> e.g. you can do "pip install package==version --single-version-externally-managed"
[18:53:58] <ngoldbaum> i think that's the name of the option you want
[18:54:13] <ngoldbaum> to tell pip to only install that version and no dependencies, because you are managing the dependencies manually
[18:54:15] <Raito_Bezarius> I don't find it in the man pip-install
[18:54:29] <Raito_Bezarius> (pip 20.0.2)
[18:54:45] <ngoldbaum> it looks like conda-forge does "{{ PYTHON }} -m pip install . --no-deps -vv"
[18:54:54] <ngoldbaum> to install python packages into the conda environment
[18:55:01] <ngoldbaum> so maybe you just want --no-deps
[18:55:24] <ngoldbaum> but i guess in that context it's not fetching data from pypi at all
[18:55:30] <ngoldbaum> so it's not really what you want to do
[18:55:32] <Raito_Bezarius> Indeed
[18:56:00] <Raito_Bezarius> I want to do the download phase, hash check phase, caching phase, myself
[18:56:22] <ngoldbaum> but of wheels, which is a little weird
[18:56:23] <Raito_Bezarius> And for the installation phase, I'd prefer to delegate to pip or whatever which has implemented more of PEPXXX than me regarding "installation mechanisms in Python"
[18:56:34] <Raito_Bezarius> ngoldbaum: Why so?
[18:57:02] <Raito_Bezarius> (thank you for your input/help BTW, it's super helpful, tos9 & ngoldbaum)
[18:57:06] <ngoldbaum> because of exactly the difficulties you're running into
[18:57:12] <ngoldbaum> which are purposely there for security reasons
[18:57:40] <ngoldbaum> there is a stable URL scheme for the sdist
[18:57:50] <ngoldbaum> because downstream packaging schemes want to consume sdists
[18:57:58] <ngoldbaum> it sort of sounds like that's what you're buildinb
[18:58:02] <ngoldbaum> but you want to use wheels
[18:58:16] <ngoldbaum> from pip/pypi perspective, wheels are a binary artifact that are supposed to be consumed by pip
[18:58:17] <Raito_Bezarius> Because I don't want to recompile everything each time I install a package for example
[18:58:27] <Raito_Bezarius> wheels are exclusively made for pip?
[18:58:50] <ngoldbaum> it's a bit like trying to use an rpm in a package manager that you're maintaining yourself on a redhat system
[18:59:22] <Raito_Bezarius> I don't see how it's bad
[18:59:26] <Raito_Bezarius> I'm not trying to fully reuse pip
[18:59:32] <Raito_Bezarius> But I'm trying to get machine-reusable output from pip
[18:59:36] <Raito_Bezarius> To build automation on the top of pip
[18:59:42] <Raito_Bezarius> But I can understand that pip is not made for that
[18:59:52] <Raito_Bezarius> But surely, there must be lower levels tooling or documentation
[18:59:53] <ngoldbaum> there's likely an option you can use
[19:00:26] <Raito_Bezarius> I read multiple times the man but I don't see such an option
[19:01:06] <ngoldbaum> ok, well, you've already explained it and i still don't really understand what you're doing, and i'm no expert myself
[19:01:14] <ngoldbaum> so good luck
[19:01:17] <Raito_Bezarius> Thanks :)
[20:45:26] <Raito_Bezarius> FWIW, I found how those hashes are generated
[20:45:32] <Raito_Bezarius> github.com/pypa/warehouse/blob/master/warehouse/forklift/legacy.py#L1340
[20:45:57] <Raito_Bezarius> The only question I have left, which I asked at #pypa-dev, is: why using blake2_256 when sha256 is used everywhere else (and only available to Simple API)
[20:46:20] <Raito_Bezarius> "curl pypi.org/pypi/requests-oauthlib/json" don't expose any blake2_256 hash
[21:18:56] <energizer> does pypi have any data about how often packages are installed using Poetry?
[21:31:21] <dstufft> energizer: does it have a unique user agent?
[21:38:21] <energizer> dstufft: i think no, it only sets the user agent when uploading. for downloading i think it wraps pip. does pip have a way to set the user agent?
[21:39:19] <energizer> oh i see, yes. PIP_USER_AGENT_USER_DATA
[21:40:09] <energizer> as of last week! :)
[21:40:16] <energizer> github.com/pypa/pip/pull/5550
[21:52:38] <dstufft> such a silly feature lol, but w/e - that doesnt really help if poetry is using pip until they populate that field and pypi is updated to understand it
[21:53:48] <dstufft> in anycase, i dont remembe what uas we support atm, but given a unique ua we can track metric against that
[21:54:31] <dstufft> packaging.python.org/guides/analyzing-pypi-package-downloads