pmxbot IRC Log Viewer

[14:10:28] <mitsuhiko> dstufft: why does pip prevent installation to --user in a non system package virtualenv without being able to override it?

[14:10:46] <mitsuhiko> i was trying to use it for plugin installation into a custom folder through PYTHONUSERHOME but it does not want to work (obviously)

[14:40:09] <dstufft> mitsuhiko: I don't know

[14:40:32] <dstufft> mitsuhiko: that's been around for longer then I've been involved, not sure if there is a particular reason

[14:40:53] <mitsuhiko> i actually found out that i want --home and --install-lib anyways as install option

[14:40:57] <mitsuhiko> but obviously that no longer works with wheels

[14:41:09] <mitsuhiko> which apparently cannot be installed into specific folders with pip

[14:41:10] <mitsuhiko> is that true?

[14:41:21] <dstufft> other than the fact that the place you install too with --uiser isn't generally on the sys.path inside of a virutalenv

[14:41:23] <dstufft> uh

[14:41:29] <dstufft> you just want to install into a particular folder?

[14:41:43] <dstufft> does pip install -t /whatever/folder <thing> do what you want?

[14:42:38] <mitsuhiko> uh. maybe. let me check

[14:42:57] <dstufft> sorry I just woke up so i'm still a little out of it

[14:43:04] <mitsuhiko> dstufft: i could kiss you

[14:43:07] <mitsuhiko> how did i miss this

[14:43:29] <dstufft> awesome :D

[14:43:42] <dstufft> the long form is --target iirc

[14:43:52] <mitsuhiko> i'm so absuing pip right now but it feels awesome

[14:44:09] <mitsuhiko> i'm basically using it as a plugin manager for things that are totally misused setuptools package

[14:44:14] <mitsuhiko> but it works

[14:44:21] <mitsuhiko> and with --target even less ugly :)

[14:44:32] <dstufft> sounds cool. Python needs a good plugin solution

[14:44:55] <dstufft> is this the thing you were talking about needing coordinate from pip for?

[14:46:59] <mitsuhiko> dstufft: na, the coordination would be for multi version installations like npm

[14:47:07] <dstufft> ah

[14:47:10] <mitsuhiko> but that's out of scope for me right now

[14:47:17] <mitsuhiko> but the plugin system is shaping up nicely

[14:47:31] <mitsuhiko> it's not competing with setuptools entrypoints but it's a bit more controlled for what i do

[14:49:26] <mitsuhiko> dstufft: i assume the uninstall action would never work for such things, right

[14:49:48] <dstufft> mitsuhiko: if you add the path to sys.path it'll find them

[14:49:57] <dstufft> but otherwise it won't know it's installed

[14:50:20] <mitsuhiko> i tried that, but that does not work

[14:50:30] <dstufft> hm, I thought it should, moment

[14:50:47] <mitsuhiko> dstufft: well, it finds it but it says that it's not in the venv and does not want to uninstall

[14:51:03] <mitsuhiko> and as far as i can see there is no way to override this

[14:51:08] <mitsuhiko> unless pip is not installed into a venv

[14:51:17] <mitsuhiko> (which is the same problem i had with --user)

[14:52:04] <dstufft> mitsuhiko: ah, that's probably running into code that prevents people from uninstalling stuff in the global site-packages

[14:52:29] <dstufft> probably that could be made smarter so it allows uninstalling stuff on PYTHONPATH

[14:52:44] <dstufft> or rather, so it only prevents the global site-packages

[14:52:49] <dstufft> and not other random dirs that someone might add

[14:53:23] <mitsuhiko> looking how much work changing that is

[14:54:12] <mitsuhiko> i could monkeypatch it easily :D

[14:54:34] <mitsuhiko> dstufft: actually, the right thing to fix would be a --add-managed-dir flag to uninstall

[14:54:40] <mitsuhiko> instead of using PYTHONPATH

[14:54:43] <mitsuhiko> or just --target like for install

[14:54:54] <dstufft> mitsuhiko: ah yea, not a bad idea either

[14:55:04] <mitsuhiko> would you accept such a patch?

[14:56:15] <dstufft> mitsuhiko: probably yes, modulo I haven't thought it through yet and i'm still trying to wake up :) but I can't think of any reason not to allow it atm

[14:56:37] <mitsuhiko> as far as i can see it could be self contained logic that is not triggered unless you pass --target

[14:57:16] <dstufft> sounds good then

[15:00:40] <mitsuhiko> dstufft: how do you not get mad maintaining pip?

[15:01:01] <mitsuhiko> what looked like a 10 line change now looks like something that would require significant reengineering :-/

[15:02:13] <dstufft> mitsuhiko: I do sometimes, but someone needs to do it

[15:02:21] <dstufft> ah :( yea that happens, pip is poorly factored

[15:03:11] <dstufft> refactoring is on my TODO list, I started putting some stuff into place the last 2 weeks or so to start moving towards more unit tests and less functional tests to try and get tests covered better before that happens

[15:03:11] <mitsuhiko> it needs to go through PYTHONPATH because it uses pkg_resources

[15:03:25] <mitsuhiko> which funnily enough conflicts with how the rest of the code thinks uninstallation works

[15:05:55] <dstufft> I've started some movements to stop relying on pkg_resources too, though that's likely a longer term project

[15:10:08] <mitsuhiko> uninstall makes me cry

[15:10:19] <mitsuhiko> there is a workaround for a 3 year old debian patch bug in there

[15:14:57] <dstufft> yea

[15:14:59] <dstufft> it's real bad

[15:24:14] <mitsuhiko> dstufft: so pkg_resources working set seems to cache

[15:24:23] <mitsuhiko> so my attempt of just adding an entry to sys.path afterwards fails

[15:24:36] <mitsuhiko> now the clear workaround is to empty out the working set, but at one point the hack just gets too bad

[15:24:41] <mitsuhiko> do you see a better option?

[15:24:57] <mitsuhiko> i assume the correct thing to do would be to construct an empty working set

[15:25:04] <mitsuhiko> but then i would have to rewrite half of pip i guses

[15:29:56] <dstufft> mitsuhiko: I'm not sure without taking a closer look

[15:32:18] <mitsuhiko> actually. it's not as bad. i have something now

[15:33:40] <mitsuhiko> the only thing i need to change is that it removes an empty folder at the end

[15:33:41] <mitsuhiko> that's a bit odd

[15:50:47] <mitsuhiko> dstufft: part one: https://github.com/pypa/pip/pull/1912

[15:50:52] <mitsuhiko> i will add tests if you are happy with how it generally works

[15:51:30] <dstufft> mitsuhiko: will take a look in a minute, doing stand up right now

[15:51:41] <mitsuhiko> no worries

[16:06:33] <dstufft> mitsuhiko: hmm, is retaining the --target directory what should happen, or should pip install --target create the directory if it doesn't exist?

[16:26:15] <mitsuhiko> dstufft: not sure if pip install shuold create it

[16:26:19] <mitsuhiko> but i don't expect it to go away

[16:26:24] <dstufft> mitsuhiko: ok

[16:26:25] <mitsuhiko> because i create it manually

[16:26:32] <dstufft> looks fine to me

[16:26:59] <mitsuhiko> trying to add tests

[16:27:05] <mitsuhiko> but failing a little bit right now :)

[16:27:10] <dstufft> oh no

[16:27:15] <dstufft> you might be hitting this horrible bug

[16:27:35] <dstufft> the failure here https://travis-ci.org/pypa/pip/jobs/28882769

[16:27:46] <dstufft> I cannot figure out WTF is causing pip in a subprocess to fail on that one single test

[16:27:53] <dstufft> it's happening in other PRs

[16:28:05] <dstufft> s/fail/hang forever after sys.exit is called/

[16:28:58] <mitsuhiko> dstufft: ah. i have a theory

[16:29:13] <dstufft> it happens here too https://github.com/pypa/pip/pull/1901/files

[16:30:07] <mitsuhiko> dstufft: sorry, can't find the right code. where do you invoke the subprocess?

[16:30:54] <mitsuhiko> dstufft: the reason it most likely hangs is because you did not read the buffer empty

[16:31:38] <dstufft> It fails here -> https://github.com/pypa/pip/blob/develop/tests/functional/test_install_config.py#L153, the subprocess calls is in another library, https://github.com/pypa/scripttest/blob/master/scripttest.py#L242-L260

[16:31:48] <dstufft> mitsuhiko: isn't .communicate() supposed to handle that?

[16:32:17] <dstufft> (debug is False)

[16:33:09] <mitsuhiko> dstufft: the problem is that communicate assumes the process actually ends if you fully read

[16:33:17] <mitsuhiko> if it ever gets only slightly interactive beyond that you hang

[16:33:27] <mitsuhiko> and because you can't signal into it you can't interrupt it

[16:34:32] <dstufft> not sure what you mean, pip does make it the whole way though it's own code, and calls sys.exit(), or do you mean something confuses subprocess into thinking the EOF never happens?

[16:36:20] <dstufft> oh, if i swap sys.exit(0) for os._exit(0) in the subprocess it works

[16:36:48] <mitsuhiko> sec. phone

[16:36:52] <dstufft> np

[16:39:41] <mitsuhiko> dstufft: so the problem is when you get lots of output that comes after lots of input i believe

[16:40:55] <dstufft> mitsuhiko: hm, there shouldn't be any input, only output

[16:41:28] <dstufft> I thought it might be something with the pipes, turning them off the process exits fine, but piping to a file doesn't work either

[16:41:34] <mitsuhiko> dstufft: for fun. try setting close_fd=True

[16:42:01] <mitsuhiko> sorry. close_fds

[16:42:59] <dstufft> ok, moment

[16:57:20] <mitsuhiko> there is one bug, it leaves a dist info for wheel installs behind

[16:57:23] <mitsuhiko> investigating why

[16:59:12] <mitsuhiko> found it

[17:00:08] <dstufft> mitsuhiko: no change with close_fds btw

[17:05:35] <mitsuhiko> i totally fail writing those tests

[17:05:45] <mitsuhiko> for some reason it reports me that the scratch folder was deleted

[17:05:54] <mitsuhiko> but i cannot reproduce this on the command line

[17:07:49] <dstufft> mitsuhiko: I can take a look, the testing stuff is largely as bad as the rest of things

[17:11:39] <mitsuhiko> dstufft: updated pull request

[17:18:56] <mitsuhiko> the two failures are i think misreports or i am misunderstanding how this invokes it

[17:19:44] <dstufft> mitsuhiko: ok, i'll take a look shortly

[17:24:45] <mitsuhiko> sorry for causing extra work :)

[17:25:24] <dstufft> mitsuhiko: no problem :)

[18:07:37] <Celebi> Bonjour :D

[18:09:04] <Celebi> Bonjour :3

[18:09:14] <Celebi> I'm french

[21:39:31] <mitsuhiko> dstufft: you around?

[22:20:19] <e6iUkX> hey is there a way to upgrade a package with --user?

[22:21:22] <mitsuhiko> e6iUkX: as far as i can see --user is largely unsupported

[22:21:35] <mitsuhiko> i tried to get it working in a nice way the last few days but i think it's a lost cause

[22:25:17] <e6iUkX> the US is totally going to win it in a shootout.

[22:33:48] <e6iUkX> ah well.

Log file Viewer

Help | Karma | Search:

#pypa logs for Tuesday the 1st of July, 2014