Log file Viewer

[04:27:40] <McSinyx[m]> hi, I'm just curious, why would the oldest numpy help?

[04:28:33] <McSinyx[m]> btw it'd be easier to debug if we know what's with the line errored out

[10:45:44] <PSFSlack> <deveshkusingh> As per https://github.com/numpy/numpy/pull/432 , this seems to have been fixed with newer versions of numpy

[14:05:33] <abadger1999> Hey, I've run across a problem with pip in software that we're packaging that I've heard has occurred with other projects before but I can't find an open issue... Just wanted to ask if anyone here knows about a prior issue before I open a new one.

[14:05:54] <abadger1999> The prior occurrence I know about is about docker and docker-py.

[14:06:26] <abadger1999> The one I'm dealing with now can reproduced by pip install ansible==2.9.10 ; pip install ansible==2.10.0a1

[14:07:06] <abadger1999> What's happening in both cases is that files are being moved around from one package to another.

[14:07:41] <abadger1999> And pip is running an uninstall of the old package after the new package is installed, so the install ends up in a broken state.

[14:08:21] <abadger1999> The ansible package case isn't obvious from just the pip install commands... Here's what's going on behind those two commands:

[14:10:16] <abadger1999> ansible-2.9.10 is a large package that's been split into two packages for ansible-2.10.0... one package is named ansible-base and contains the bin scripts (these are not entrypoints) and the library in site-packages/ansible. The other package retains the ansible package name and contains all of the addons to the ansible package. It has a requirement on the ansible-base package.

[14:11:09] <FFY00> and they both install to site-packages/ansible?

[14:11:47] <abadger1999> FFY00: No.... for the 2.10 packages, ansible-base installs to site-packages/ansible ; ansible installs the addons to site-package/ansible_collections/

[14:12:08] <abadger1999> FFY00: But the ansible-2.9.10 package does use site-package/ansible.

[14:12:17] <abadger1999> (which is where the bug lies)

[14:13:51] <FFY00> this seems to be new

[14:13:56] <FFY00> https://pypi.org/project/ansible-base/#history

[14:13:58] <abadger1999> So what's happening is that 2.9.10 owns site-packages/ansible and bin/ansible*. pip installs ansible-base which is the new dependency of ansible-2.10.0a1. ansible-base gets placed into site-packages/ansible and into bin/ansible*. Then pip installs ansible-2.10.0a1 and uninstalls ansible-2.9.10

[14:14:04] <abadger1999> FFY00: correct.

[14:14:34] <abadger1999> The last step ends up erasing many of the files which are in the newly installed ansible-base package.

[14:15:37] <FFY00> hum

[14:17:21] <abadger1999> the way the system package managers I know fix this is they know that some files are owned by multiple packages. So when the uninstall ansible-2.9.10 happens, the files owned by both the new ansible-base and the old ansible-2.9.10 will be left on the filesystem as there's still a package which owns them.

[14:19:39] <FFY00> if the same files are owned by both packages then this is a bug

[14:19:42] <FFY00> it should not happen

[14:20:30] <abadger1999> They should be ownable by both packages during the pip "transaction" but the state after the transaction finishes should be that only one package owns them.

[14:21:36] <abadger1999> my definition of transaction here is roughly "while pip is handling install and erase of all the packages to install the package that the user requested on the cli"

[14:22:21] <FFY00> IMO if both packages own the same file pip should error out with file conflicts

[14:23:16] <abadger1999> (Tangent: system package managers allow multiple packages to own a file after the transaction finishes but only if both packages contain the exact same version of the file)

[14:23:39] <abadger1999> (Tangent: system package managers allow multiple packages to own a file after the transaction finishes but only if both packages contain the exact same version of the file)

[14:24:06] <abadger1999> FFY00: how would you do package splits and package renames, then?

[14:24:35] <FFY00> depends on the package manager

[14:24:37] <FFY00> pacman does not allow this

[14:24:39] <FFY00> it seems no ansible-base and ansible versions are compatible right now

[14:24:49] <FFY00> maybe they forgot to push them

[14:26:13] <FFY00> looks like we are missing a ansible-2.10.0b1, which should be compatible with ansible-base-2.10.0b1

[14:26:44] <abadger1999> I don't think that pacman would be able to handle package splits or package renames without this.... so if you can tell me how pacman does it, I can stand corrected ;-)

[14:26:59] <abadger1999> no.. ansible-2.10.0a1 and ansible-base-2.10.0b1 are compatible.

[14:27:05] <FFY00> my bad

[14:27:17] <FFY00> ansible-2.10.0a1 seems to be compatible

[14:27:21] <FFY00> yes

[14:27:40] <abadger1999> ( pip uninstall ansible ; pip install ansible==2.10.0a1 and everything would be fine. )

[14:27:51] <abadger1999> <nod>

[14:27:59] <FFY00> uh, this seems weird

[14:28:22] <FFY00> they uploaded a sdist that appears to be an egg?

[14:28:28] <abadger1999> Hmmm......

[14:28:33] <abadger1999> if so, I;ll fix it...

[14:28:37] <abadger1999> which of those is an egg?

[14:30:22] <FFY00> pip install ansible==2.10.0a1 is working for be

[14:30:48] <abadger1999> <nod>

[14:30:49] <FFY00> I think they just mistakenly included .egg-info in ansbible

[14:31:07] <abadger1999> Oh, you mean, the tarball has a pre-built .egg-info directory in it?

[14:31:48] <FFY00> yes

[14:32:01] <FFY00> it is not really a problem

[14:32:10] <FFY00> just looked weird to me :P

[14:32:17] <abadger1999> yeah :-)

[14:32:26] <FFY00> anyway, I seems to be able to install both packages

[14:32:35] <FFY00> and they do not seem to have conflicts

[14:32:37] <abadger1999> FFY00: yes.... So the things to check...

[14:33:02] <FFY00> since ansible-base installs to ansible and ansible installs to ansible_collections

[14:33:13] <FFY00> so what is really the problem :D?

[14:33:40] <abadger1999> (1) did you install ansible==2.9.10 before insalling ansible==2.10.0a1 ? (2) When you try to use ansible, have some of the files been removed? (can you do bin/ansible --version ? Is there a site-packages/ansible/__init__.py ?)

[14:34:03] <abadger1999> pip installs the pacakges, but corrupts the installation.

[14:34:33] <abadger1999> When it removes ansible-2.9.10, it ends up removing files that are owned by ansible-base-2.10.0b1

[14:35:00] <abadger1999> (the files are owned by both that older version of ansible and the new version of ansible-base)

[14:35:07] <FFY00> oh

[14:35:25] <FFY00> so the problem is having ansible-base and an older ansible

[14:35:39] <abadger1999> The problem is upgrading the older ansible to the newer one.

[14:35:52] <FFY00> right, that makes sense

[14:36:12] <FFY00> I installed directly the new version and the isn't any issue, as expected

[14:36:20] <abadger1999> Right :-)

[14:36:24] <FFY00> ansible --version works file

[14:36:27] <FFY00> *fine

[14:36:47] <FFY00> I blame this on pip not handling file conflicts properly :P

[14:37:11] <abadger1999> (We're going to need to document pip uninstall ansible ; pip install ansible as how to upgrade from 2.9 to 2.10)

[14:37:15] <abadger1999> Yeah :-)

[14:37:19] <FFY00> in a system package I would just add a conflict on the older version

[14:37:34] <FFY00> but this is not possible in python metadata

[14:38:10] <FFY00> you can add a circular import

[14:38:12] <abadger1999> I saw that there's an Obsoletes-dist metadata tag in one of the packaging PEPs but pip doesn't do anything with that yet. (That might have helped here... Don't know without the pip implementation)

[14:38:31] <FFY00> ansible-base would require ansible>=2.10

[14:38:42] <FFY00> that should make pip behave properly

[14:39:14] <FFY00> it would make sure you don't install ansible-base with an older ansible version

[14:39:17] <abadger1999> <nod> Yeah... Our purpose in splitting the packages, though, was to allow people to just install ansible-base if that's all they need in their environment.

[14:39:24] <FFY00> but it's a bad solution :/

[14:39:34] <abadger1999> <nod>

[14:39:52] <FFY00> unfortunately I am not seeing any other solution :(

[14:40:09] <abadger1999> Okay, well, I'll open a pip bug and we'll do the documentation thing.... I just wanted to see if anyone knew of an existing bug before I did that.

[14:40:24] <FFY00> I would definitely open a bug in pip if it is not preventing you from installing packages with file conflicts

[14:40:39] <abadger1999> (docker and coker-py ran into this before so I would have thought there was a pip bug but I couldn't come up with any search terms that found one)

[14:40:51] <abadger1999> *docker and docker-py

[14:42:53] <FFY00> alternatively you can call ansible-base ansible, and ansible something like ansible-extra

[14:43:11] <FFY00> but that is also not a great option

[14:45:20] <abadger1999> <nod> yeah.

[14:50:10] <FFY00> abadger1999, this is a bit unrelated but I would recommend you to also upload wheels, instead of just sdists (tarballs)

[14:52:20] <FFY00> well, actually I think it might be worth trying to install via wheels, since they contain a file list and pip might be checking that

[14:52:53] <FFY00> if pip just does setup.py install, it had no information about what files will be installed

[14:53:09] <FFY00> *it has

[15:06:51] <abadger1999> Hmm...

[15:07:27] <abadger1999> FFY00: I can do that for the ansible-2.10.x package but I can't for the ansible-2.9.x or the ansible-base-2.10.x package (which is where the problem is occuring :-(

[15:08:08] <FFY00> well, you can still upload right now

[15:08:14] <FFY00> but that shouldn't be needed

[15:08:23] <abadger1999> FFY00: Those packages need to install symlinks which setuptools and pip don't yet understand so they use scripted code in setup.py to achieve that.

[15:08:45] <FFY00> if I am correct, installing wheels in 2.10 should be enough

[15:08:53] <FFY00> ah

[15:09:00] <FFY00> okay

[15:09:29] <FFY00> anyway, I would test to see if wheels solve your problems

[15:09:38] <abadger1999> <nod> yeah, good idea.

[15:09:48] <FFY00> and if they do, start looking into solutions to the symlink problem

[15:09:59] <abadger1999> I'll try making some test wheels with similar files and dependencies to see what happens.

[15:10:21] <FFY00> you should be able to build the wheel locally and just install it via pip

[15:10:41] <FFY00> you just need to install ansible 2.9 and then try to install a 2.10 wheel :)