PMXBOT Log file Viewer

Help | Karma | Search:

#pypa logs for Tuesday the 9th of June, 2015

(Back to #pypa overview) (Back to channel listing) (Animate logs)
[07:59:24] <vortec> hi, i have a question. i just upgraded to pip==7.0.3 and i am trying to install a package from a private pypi server, that has a dependency that also only exists on the private pypi server. now when i run "pip install -e ." it tries to download the dependency from pypi.python.org, even though my ~/.config/pip/pip.conf has "[global] index-url = ...."
[07:59:53] <vortec> actually i downloaded the sourcecode and run "pip install -e ." in the code folder
[08:00:17] <vortec> and i want pip to download the dependency from the private pypi, not the official one. what am i doing wrong?
[08:03:18] <vortec> i just debugged further and it seems that it doesnt use pip.conf at all:
[08:03:25] <vortec> gist.github.com/vortec/f7f9cf52fc8f1f3ff4f2
[08:07:11] <apollo13> interesting
[08:07:21] <apollo13> stracing pip shows no access to config here either
[08:08:13] <vortec> IIRC this worked in pip6
[08:09:09] <apollo13> oh wait my strace was borked
[08:10:00] <apollo13> run strace pip search Django 2>&1|grep pip.conf and show me the output
[08:10:25] <apollo13> not that it finds /etc/xdg/pip/pip.conf or /etc/pip.conf or…
[08:10:47] <vortec> sec i must install strace first
[08:10:58] <vortec> i'm on OSX
[08:11:01] <apollo13> oh
[08:11:21] <vortec> seems dtruss is the one people use
[08:11:37] <apollo13> jupp
[08:14:56] <vortec> apollo13: gist.github.com/vortec/3852567703d24c65158d
[08:15:44] <apollo13> hrmpf, dtruss is apparently not doing what you want it to
[08:16:02] <vortec> oh
[08:16:13] <anthony25> hello
[08:16:22] <vortec> hmm i don't know these tools, lemme search further
[08:16:23] <vortec> hi anthony25
[08:16:37] <vortec> apollo13: what do i want it to do?
[08:16:38] <apollo13> you could try with -f and see if it gets better
[08:16:49] <apollo13> you want it to show stat and open syscalls
[08:17:21] <vortec> i see plenty of stat64() calls, how do syscalls looks like?
[08:17:42] <apollo13> like that
[08:17:46] <anthony25> for my project, I'm using a little program coded in C to interact with iptables where I do a setuid(0) (because I don't want it to run as root), so I have to add the +s flag to the executable
[08:18:01] <anthony25> how can I include that in my setup.py using setuptools ?
[08:18:20] <apollo13> vortec: this is where pip searches for me dpaste.de/rrA7/raw
[08:18:49] <anthony25> hum, maybe I have to ask in #pypa-dev, sorry
[08:19:04] <apollo13> anthony25: given that your setup.py doesn't have to be run by root, probably not at all
[08:19:17] <vortec> apollo13: i see, can i see a full output of the dtrace?
[08:19:30] <vortec> strace*
[08:19:36] <apollo13> vortec: well yes, run with -f to follow children, as you can see by 10601/0xbb8d: 141539 3 2 execve("/usr/bin/bash\0", 0x7FFF58498BB8, 0x7FFF58498BE0) = -1 Err#2it opens a new process
[08:19:52] <anthony25> apollo13: ok, so what do you propose ?
[08:20:00] <apollo13> anthony25: as I said, run it with -f
[08:20:45] <apollo13> oh wait
[08:20:47] <apollo13> sorry :þ
[08:20:53] <apollo13> anthony25: I do not propose anything :)
[08:20:54] <anthony25> yeah that's for vortec :p
[08:21:52] <apollo13> especially if you would deploy wheels there would be no setup.py involved at all
[08:22:00] <apollo13> I think this is something which should just get documented
[08:22:10] <apollo13> or maybe do it via sudoers instead of setuid
[08:22:30] <vortec> dtruss -f: gist.github.com/vortec/48d293fae70440407e7b
[08:22:37] <anthony25> apollo13: that's what I did before
[08:22:49] <anthony25> (the sudoers way)
[08:23:07] <apollo13> vortec: guess I have no idea how dtrace works then…
[08:23:42] <apollo13> vortec: you could try "python -m pip" instead of just pip, maybe that does not spawn an extra shell which mucks it up?
[08:24:52] <apollo13> but aside from that I am somewhat out of ideas
[08:26:53] <vortec> apollo13: can i see a complete strace output from your "pip search Django" so that i can see what it should look like?
[08:27:02] <vortec> the python -m pip trick didnt work
[08:30:11] <vortec> ah nevermind
[08:30:25] <vortec> it should just follow the execve() call at the bottom
[08:31:52] <vortec> but it errors out?
[08:31:56] <vortec> it doesnt start a bash
[08:32:07] <vortec> exit code is -1 (no such file or directory)
[08:37:06] <vortec> anyway, i have no pip.conf in my activated virtualenv and no site-wide pip.conf. the only file that exists is ~/.config/pip/pip.conf, as described at pip.pypa.io/en/latest/user_guide.html#config-file
[08:37:22] <vortec> so i assume this feature is broken and i should submit a bug ticket?
[08:41:13] <apollo13> vortec: well does pip look there for a file?
[08:41:18] <apollo13> it does for me…
[08:42:32] <vortec> i cannot say for sure, because I dont know how to debug this
[08:42:46] <vortec> i understand that pip wants to fork a new shell, but it seems that it cant
[08:50:02] <vortec> ok i think i got something
[08:52:07] <vortec> apollo13: gist.github.com/vortec/efdc5f82af8f5229428c
[08:52:23] <vortec> it does not look for ~/.config/pip/pip.conf
[08:52:34] <apollo13> makes sense I guess
[08:52:45] <vortec> why?
[08:52:54] <apollo13> cause .config is freedesktop spec, ergo linux
[08:53:02] <apollo13> I don't think it looks for .config on windows either
[08:53:14] <apollo13> On Mac OS X the configuration file is $HOME/Library/Application Support/pip/pip.conf.
[08:53:21] <apollo13> right from the link you showed…
[08:53:28] <vortec> :-(
[08:53:31] <vortec> i'm so sorry
[08:53:40] <vortec> this worked in previous pip version
[08:54:13] <vortec> so i assumed it was a good idea to put it there (also i love the .config convention, doesnt pollute my home folder)
[08:54:20] <vortec> thanks for your (wasted) time apollo13
[08:54:27] <apollo13> no worries, glad it works now
[08:54:48] <apollo13> well I assume "Application Support" can be seen as similar to .config
[08:54:58] <apollo13> also doesn't pollute your home folder ;)
[08:55:30] <vortec> works perfect now
[08:55:37] <vortec> that's true
[08:56:57] <vortec> huzzah, successfully wasted an hour due assuming things and not reading the docs properly
[08:57:22] <vortec> at least i learned how to monitor file access
[08:57:31] <vortec> which is useful i guess :-)
[08:57:33] <apollo13> that is positive I guess :)
[08:57:41] <apollo13> jupp, I always say: know your tools and OS
[08:58:36] <vortec> there's so much to know, don't think one can ever finish learning
[08:58:40] <vortec> unless you've written it of course
[08:58:50] <apollo13> haha, not even then probably
[08:59:01] <apollo13> but strace/gdb etc are handy to know
[08:59:19] <vortec> heh right. linus doesn't know large portions of the kernel code i'm pretty sure
[08:59:21] <vortec> yeah
[08:59:23] <vortec> gdb i know
[08:59:37] <vortec> and strace/dtruss is very useful
[08:59:50] <apollo13> well then I got a new one, try using gdb to debug python processes including getting python traces!
[08:59:52] <vortec> just never really needed it so far
[09:00:04] <apollo13> cause pdb does not allow you to break into a running process, gdb does… :)
[09:00:21] <vortec> oh there are tools for that
[09:00:44] <apollo13> yeah, oddly enough the never seemed to work for me
[09:00:57] <apollo13> nowadays I just use a self compiled python + gdb and python support macros
[09:01:02] <apollo13> that literally rocks
[09:01:27] <vortec> wiki.python.org/moin/DebuggingWithGdb << like this?
[09:01:43] <apollo13> jupp
[09:01:58] <vortec> that's very nice
[09:02:15] <apollo13> but even there, installing python-dbg was not enough for me, gotta retry that at some point
[09:07:19] <apollo13> ah it didn't work in a venv cause it couldn't find the debug stuff, makes sense
[09:07:36] <vortec> this is a mess these days
[09:07:44] <vortec> i have no idea how many python interpreters i have installed
[09:07:50] <vortec> neither which one is being used
[09:10:16] <apollo13> I have 5
[09:13:38] <apollo13> heureka
[09:13:42] <apollo13> in a venv bin: ln -s /usr/lib/debug/usr/bin/ .debug
[09:13:48] <apollo13> and voila I can debug python the way I want
[09:15:07] <vortec> thats very nice
[09:15:11] <vortec> you should write a blog post about that
[09:15:50] <apollo13> first I'd have to rewrite my blogging software I guess
[09:16:16] <vortec> that sounds like a typical programmers response ;-)
[09:16:27] <apollo13> well it exists blog.apolloner.eu
[09:16:39] <apollo13> but I neither find the time to write it in a way I like nor bother about it too much^^
[09:16:52] <apollo13> guess before I fix the software I should fix my server setup *sigh*
[09:18:03] <vortec> funny
[09:18:06] <vortec> blog.apolloner.eu/2011/8/27/server-infrastructure-for-small-companies
[09:18:11] <vortec> this is kind of what we do ^
[09:22:24] <apollo13> ah right, that setup is still running fine (without drbd though)
[09:26:36] <vortec> why no drbd?
[09:26:55] <apollo13> cause xenserver has live migration now even with local storage
[09:27:24] <apollo13> and since I don't need real HA, I am fine without DRBD, also it is not really supported on XenServer which makes it kind of a pain
[09:27:48] <apollo13> reminds me that I should upgrade the xenservers at some point
[18:47:42] <lambacck> was there a recent change to the way scripts are installed on windows? I no longer see any .py files installed along with the .exe
[18:48:22] <lambacck> And that is messing with the pyramid pserve --reload functionality
[19:00:04] <dstufft> lambacck: I think we ship the .py files embedded in the exe's
[19:04:21] <lambacck> That's likely to break things that try to detect and re-run the scripts (as is the case here). I think multiprocessing also performs those kinds of tricks
[19:06:21] <lambacck> I'm looking into ways of working around it for pyramid, and may have more specific information on how it breaks.
[19:06:50] <dstufft> I'm not sure why that would break anything, that .py file shouldn't ever change?
[19:07:24] <lambacck> dstufft: github.com/Pylons/pyramid/blob/1.6-branch/pyramid/scripts/pserve.py#L534
[19:07:58] <lambacck> at first glance sys.argv[0] doesn't include the .exe.
[19:10:48] <lambacck> if __name__ == '__main__':
[19:10:49] <lambacck> sys.argv[0] = re.sub(r'(-script\.pyw|\.exe)?$', '', sys.argv[0])
[19:10:49] <lambacck> sys.exit(main())
[19:11:35] <lambacck> is in the script tacked onto the .exe
[19:11:51] <lambacck> so there is no way to re-run the same script based on sys.argv
[19:14:47] <dstufft> I think the idea is that Windows and *nix operates the same way, you get a command like ``foo`` to call, and the fact that it's foo.exe instead of foo is more or less hiddenf rom you
[19:18:24] <lambacck> keeping the exe on the end seems to make the reloader work properly
[19:22:16] <lambacck> dstufft: so the question is whether that is a pip regression or something we should fix on the pyramid side?
[19:24:36] <lambacck> or since it is a vendored distlib should I be raising the issue there?
[19:25:05] <dstufft> It's been this way in pip for awhile... I'm actually surprised nobody has noticed it before. pip has pretty much always done that when installing from wheels
[19:25:19] <dstufft> I'm not sure what the right answer is
[19:25:47] <dstufft> I don't feel comfortable making a judgement call right this moment since I don't fully understand why it did that to begin with
[19:25:49] <lambacck> well our dev machines have been around for a while and we were using easy_install because no binary dep installation on pip
[19:25:50] <dstufft> I just have guesses
[19:26:45] <lambacck> now because of wheel/binary support in pip we're all in on pip but have some lingering environments that are getting rebuilt and causing issues
[19:27:21] <lambacck> I also wonder if it is the wheel path? before the always build and install wheels was setuptools building the .exe files?
[19:27:30] <dstufft> yes
[19:27:40] <dstufft> when installing wheels pip controls the script wrapper
[19:27:46] <dstufft> when installing sdists, setuptools does
[19:28:07] <lambacck> so that is it, we had old versions that were not installed via wheel and the setuptools way worked
[19:29:20] <lambacck> And to be clear, I'm not saying either way is right or wrong but I expect that I'm not the only one who will run into it, I just might have more knowledge of where to go
[20:09:19] <lambacck> so the workaround for us at this time is --no-binary pyarmid
[20:29:09] <utek> lambacck: I see you use windows. Do you by any chance also use vagrant on windows host?
[20:29:26] <lambacck> I do
[20:33:05] <utek> can I take a moment?
[20:35:16] <lambacck> you can try
[21:33:30] <nanonyme> Is there any systax with which I can say that my package conflicts with another package?
[21:42:55] <dstufft> No
[21:45:36] <nanonyme> I have this package syscertifi which registers a module certifi.py and I'd kinda want it not to be installable if the machine already has certifi
[22:00:05] <dstufft> yea, there's nothing like that currently :(
[22:09:37] <tos9> I'm not sure I want to know the answer to these questions. But this is now the third different machine where I've seen interpreters import each others' stdlib while creating virtualenvs.
[22:10:09] <tos9> Is virtualenv doing horrible terrible things, or what possibly is explaining what's going on? Tracebacks look like bpaste.net/show/1a88bdc42406
[22:10:24] <tos9> (Where you're seeing python-future, which does terrible things, and which won't work if you somehow manage to import it in Py3)
[22:16:43] <dstufft> virtualenv is doing horrible things
[22:16:57] <dstufft> don't use -p is the best answer right now
[22:17:06] <dstufft> virtualenv is uniquely horrible
[22:17:11] <dstufft> you want to know how -p works?
[22:17:19] <tos9> dstufft: I mean I haven't had dinner yet I guess.
[22:17:46] <doismellburning> I mean, is `-p` better or worse than `--relocatable` ;)
[22:17:58] <tos9> Yeah but you can at least ignore --relocatable.
[22:18:24] <dstufft> if you have virtualenv.py installed into say python2.7/site-packages/ and you use -p python3.4, close to the top of virtualenv.py it will notice you're doing -p, and will shell out and do ``python3.4 /path/to/python2.7/site-packages/virtualenv.py``
[22:18:37] <tos9> :(
[22:18:41] <tos9> That's what I thought it'd do.
[22:18:49] <dstufft> depending on various things, this may or may not mean you get Python 2.7's stdlib while you're running Python 3.4
[22:19:21] <tos9> So the recommended thing these days is to install virtualenv to everything?
[22:19:30] <tos9> I guess the recommended thing is don't use virtualenv.
[22:19:32] <tos9> But alas.
[22:19:36] <dstufft> you can either don't use -p, and instead install virtualenv into every python and do pythonX.Y -m virtualenv instead of virtualenv -p pythonX.Y
[22:19:38] <dstufft> or
[22:19:41] <dstufft> you can not install virtualenv
[22:19:52] <dstufft> and just leave virtualenv.py somewhere with virtualenv_support alongisde it
[22:20:02] <dstufft> and run path/to/virtualenv.py -p pythonX.Y
[22:20:23] <dstufft> there is a rewrite branch which should hopefully make this better, but it's been backburnned while I work on other thinfs
[22:21:37] <dstufft> I think -p is worse than --relocatable, because -p is a core part of how virtualenv works (to the extent that something like virtualenv does indeed works)
[22:21:48] <dstufft> virtualenv.py has worse things in it than any other packaging tools
[22:22:13] <tos9> Sigh.
[22:22:17] <tos9> OK the first option isn't terrible I guess.
[22:22:20] <dstufft> one of my favorites is how virtualenv.py will create a copy of itself with parts of it replaced to enable people to make a custom virtualenv.py that has "hooks"
[22:22:28] <tos9> Although I guess it really mucks with my mkenv thing.
[22:22:35] <tos9> But I guess I'll work that into it.
[22:24:37] <dstufft> tos9: yea, sorry :(