Log file Viewer

[00:53:25] <Alex_Gaynor> dstufft: is there a reason tehse diffs are on gist insead of a simple PR?

[00:54:09] <dstufft> Alex_Gaynor: mostly becuase I don't feel like triggering a 40 minute test run for something I know will fail

[00:54:37] <dstufft> I could do a PR I guess

[01:13:01] <dstufft> Alex_Gaynor: there you go ^

[01:18:52] <Alex_Gaynor> dstufft: look at all that deleting

[01:19:06] <dstufft> I love deleting code!

[02:01:06] <dstufft> Alex_Gaynor: updated the wording, thanks

[02:50:00] <dstufft> https://github.com/pypa/pip/issues/1749 kill bad options -.-

[14:27:44] <Alex_Gaynor> dstufft: the pip tests are primarily IO bound right?

[14:28:36] <dstufft> Alex_Gaynor: I dunno, I think on PyPy it's the fact we start a ton of subproccesses in the course of a test run

[14:28:57] <Alex_Gaynor> dstufft: Yeah, definietly, I'm just wondering if pytest-xdist's parallelization would help

[14:29:42] <dstufft> we used it before briefly, but I ripped it out because we were getting intermittent test failures from xdist, course now we're getting intermittent test failures from pypy taking too long so

[14:29:55] <dstufft> it did help back when we used it before

[14:30:09] <Alex_Gaynor> I'll send a PR for it, ya'll can decide if it's valuable.

[14:30:12] <dstufft> I think CPython was like ~5 minutes a test run, and PyPy was like ~25 minutes

[14:30:37] <dstufft> ok

[14:37:56] <dstufft> I was thinking about editing https://github.com/pypa/pip/blob/develop/tests/conftest.py#L35-L67 so instead of creating a new virtualenv each time, it would create one, copy it into a holding area, and then every subsequent virtualenv after that just recopies the virtualenv over from the holding area. I'll have to do some shebang writing and such but it might be quicker than ~3 subprocesses or so

[14:39:07] <Alex_Gaynor> dstufft: might be faster, is there some way to create a venv without spawning 12 millin processes though?

[14:39:54] <Alex_Gaynor> would be cool if it were possible to do https://github.com/pypa/pip/blob/develop/tests/lib/venv.py#L59 without spawning a process

[14:40:07] <dstufft> it's not actually the venv install that does it, it's the pip install into that venv since it just uses ``setup.py develop.py``

[14:40:31] <dstufft> I'd have to think about that

[14:40:48] <dstufft> might be able to generate a Wheel and convince virtualenv to use that as the pip .whl it installs

[14:40:55] <dstufft> generate a wheel once at the begining of the test suite that is

[14:41:47] <dstufft> someone wrote https://github.com/pypa/pip/issues/1721

[14:41:58] <dstufft> might be time to just sit down with a profiler and figure out what is taking all the time

[14:44:12] <Alex_Gaynor> dstufft: I'm pretty sure their analysis that git is the slow part in https://github.com/pypa/pip/blob/develop/tests/functional/test_install_upgrade.py#L274 is correct

[14:44:40] <dstufft> the pip test suite has some stupic hacks to try and make git faster

[14:44:49] <dstufft> that we're not using there

[14:44:59] <dstufft> probably we should just not use normal repos that have data in them

[14:45:04] <dstufft> especially virtualenv wtf

[14:45:10] <dstufft> virtualenv has binary blobs in it

[14:46:50] <Alex_Gaynor> dstufft: so, on my pretty-fast-computer, just cloning virtualenv takes over a minute

[14:47:04] <dstufft> lol

[14:47:18] <dstufft> I wonder how many times we clone

[14:47:27] <Alex_Gaynor> dstufft: should I send a PR to switch it to pretend, which is a tiny repo?

[14:47:56] <dstufft> Alex_Gaynor: can you switch it to https://github.com/pypa/pip-test-package

[14:48:08] <dstufft> oh wait

[14:48:09] <dstufft> on the index

[14:48:15] <Alex_Gaynor> dstufft: the package needs to be on pypi to work

[14:48:33] <dstufft> yea I guess throw pretend at it

[14:50:33] <Alex_Gaynor> dstufft: ^

[14:50:54] <dstufft> cool thanks

[14:51:01] <dstufft> I suspect it'll be a little bit before travis gets to it

[14:53:29] <Alex_Gaynor> yeah, there's like 6 outstanding builds

[16:36:13] <dstufft> Alex_Gaynor: it's a shame that failed, tests are a lot faster

[17:36:35] <Alex_Gaynor> dstufft: https://github.com/pypa/pip/pull/1754 should be g2g

[17:38:58] <dstufft> Alex_Gaynor: thanks

[17:40:58] <Alex_Gaynor> dstufft: np, I'm going to profile and such a bit today

[17:41:21] <dstufft> Alex_Gaynor: awesome, well anything that makes the tests faster is always welcome, especially on PyPy

[17:43:00] <Alex_Gaynor> dstufft: so the freeze tests are failing locally for me

[17:43:19] <Alex_Gaynor> dstufft: all I did was `git clone pip` `pip install <stuff from test_requires>`. `py.test --duration=10`

[17:44:17] <dstufft> Alex_Gaynor: what's the error?

[17:44:35] <Alex_Gaynor> dstufft: http://bpaste.net/show/yaRrWXNbDJF2ljmVsFQG/

[17:45:27] <dstufft> Alex_Gaynor: lol what even is isolation

[17:45:42] <dstufft> comment out your --find-links entry from ~/.pip/pip.conf

[17:45:49] <Alex_Gaynor> hahaha

[17:46:35] <dstufft> the bzrlib and hg tests will fail probably, they are real fail-ey if you don't have bzaar and hg installed in a way that won't use a virtualenv python

[17:47:09] <Alex_Gaynor> dstufft: lol, pretty sure I installed them with pip install --user hg bzr

[17:47:20] <Alex_Gaynor> lolsob

[17:47:32] <dstufft> hmm

[17:47:34] <dstufft> that should work I think

[17:47:39] <dstufft> well I lie

[17:47:47] <dstufft> it should work if hg/bzr use setuptools entrypoints

[17:47:56] <dstufft> if they provide their own scripts then it depends on what tey did I think

[17:49:50] <Alex_Gaynor> I guess I'll just run with the failures and hope that doesn't affect the times too much

[17:51:08] <dstufft> if you dont' care about editing installed files the fix is basically subl `which hg` and make sure the shebang is absolute to the interpreter you want it to use

[17:51:31] <Alex_Gaynor> I totally am not going to randomly edit files on my computer.

[17:51:36] <dstufft> lolz

[18:20:49] <Alex_Gaynor> dstufft: so of a 10 minute test run, more than 6.5 of those minutes are spent in subprocess

[18:21:15] <Alex_Gaynor> about 1.5 minutes of virtualen

[18:21:36] <dstufft> that was my guess, all of pip's slow tests are pretty much setup a venv, shell out to pip to run some command, verify the state of the venv

[18:22:33] <dstufft> so I guess the best speed up that's really doable is going to be figure out how to eliminate subprocess calls

[18:26:20] <Alex_Gaynor> dstufft: so there's this comedy where we create a virtualenv by calling a python function, which then shells back out to pip to install setuptools and pip into the venv

[18:26:21] <Alex_Gaynor> wat.

[18:29:22] <dstufft> Alex_Gaynor: it uses a shell to install setuptools?

[18:29:28] <dstufft> I thought we got rid of that

[18:29:42] <Alex_Gaynor> dstufft: virtualenv.py line 947

[18:31:37] <dstufft> oh so that's different than ensurepip

[18:31:46] <dstufft> oh I know why

[18:32:00] <dstufft> becuase virtualenv runs in the context of the system python

[18:32:28] <dstufft> so we need the subprocess to run the setuptools isntall in the context of the virtual environment

[18:32:28] <Alex_Gaynor> why does that matter?

[18:32:49] <dstufft> because pip doesn't decide where things are installed too, it just asks Python where it should install stuff too

[18:32:58] <Alex_Gaynor> fdslfonds;zfsdjc

[18:34:57] <dstufft> Alex_Gaynor: it's actually not entirely unreasonable, but it'd be like 99% better if it was just a file pip could read instead of needing a subprocess

[19:31:14] <gauravb7090> Hey I wanted to know the basic working of pip….Can anyone please guide me as to where should I look for?

[19:45:43] <qwcode> gauravb7090, basic use? or internal workings?

[19:45:57] <gauravb7090> internal workings

[19:47:40] <qwcode> gauravb7090, I'd read all the docs here (http://www.pip-installer.org/en/latest/) and here (http://packaging.python.org/en/latest/). they'll give a lot of context, then move to src