pmxbot IRC Log Viewer

[01:47:08] <Ivo> Alex_Gaynor: I see you've found out how slow pip's tests are :D

[01:47:19] <Alex_Gaynor> :-)

[01:57:24] <Ivo> thanks for trying to help improve that! <3

[01:58:47] <Alex_Gaynor> I wasn't as successful as I was hoping.

[01:59:02] <dstufft> I like how PyPy is fast at running code, but often times the slowest at testing code

[01:59:16] <Alex_Gaynor> tests look nothing like real code :/

[01:59:23] <dstufft> yea

[01:59:29] <dstufft> not knocking PyPy, it just makes me laugh

[01:59:39] <Ivo> soo... anyone use fish shell here?

[01:59:52] <dstufft> I think Alex_Gaynor does, or did shortly at least

[02:00:04] <Alex_Gaynor> I do

[02:00:28] <Ivo> basically getting bugged about pulling https://github.com/pypa/virtualenv/pull/589#issuecomment-41244589

[02:01:24] <Alex_Gaynor> No clue.

[02:02:40] <Ivo> damn virtualenv voodooness

[02:03:48] <Ivo> anyone know what a tox.ini would be doing in https://github.com/pypa/virtualenv/tree/develop/tests as well as being in the main dir?

[02:04:53] <Ivo> We need to change the fact that virtualenv has 7 tests

[02:07:40] <Ivo> dstufft: any opinion on changing virtualenv tests to use pytest?

[02:07:50] <dstufft> +1

[02:24:42] <dstufft> huh

[02:24:44] <dstufft> interestin

[02:24:51] <Alex_Gaynor> ?

[02:25:07] <dstufft> installing the development pip from a Wheel instead of using setup.py develop causes some test failures

[02:25:19] <Alex_Gaynor> lol

[02:25:28] <Alex_Gaynor> sounds like buuuuuugs

[02:25:35] <dstufft> that fills me with confidence

[02:26:20] <dstufft> 5 failures so far

[02:26:25] <dstufft> all in test_install.py

[02:30:40] <dstufft> Assuming I'm right, this should reduce the number of subprocess calls in the test suite by ~200 or so

[02:30:49] <Alex_Gaynor> nice!

[02:31:44] <dstufft> it builds a .whl file for pip, and then tells virtualenv not to install setuptools or pip, and then manually installs setuptools/pip in a single subprocess call

[02:31:58] <dstufft> so instead of 1 subprocess for setuptools, and 1 subprocess for pip, it's just 1 for both

[02:32:14] <Alex_Gaynor> does virtualenv not do them in a single subprocess?

[02:32:16] <Alex_Gaynor> I thoguht it did

[02:32:30] <dstufft> virtualenv does, but we don't install pip through virtualenv

[02:33:00] <Alex_Gaynor> ahhhh. right.

[02:33:04] <dstufft> https://github.com/pypa/pip/blob/develop/tests/lib/venv.py#L59-L65

[02:33:12] <dstufft> that was a performance optimization at the time

[02:33:20] <dstufft> it used to be we did ``pip install .``

[02:33:31] <dstufft> but we had to first copy . into a temp dir

[02:33:42] <dstufft> becuase ``pip install .`` mutates .

[02:34:21] <dstufft> so we copied . into a temp dir, and then did ``pip install .``, so we essentially copied stuff around twice, and ran a bunch of extra subprocesses, so we used develop to kill one of the copies

[02:34:31] <dstufft> wheel shold achieve the same I hope, but with less subprocess

[02:34:47] <dstufft> test failure #6

[02:35:14] <dstufft> and, 7, 8, 9, and 10

[02:41:40] <dstufft> Alex_Gaynor: I was thinking about using pytext-xdist, but only on PyPy on travis

[02:42:10] <dstufft> even if pytest-xdist is replacing one intermittent failure for another, at least it's an intermittent failure that happens in ~20 minutes instead of ~50 minutes

[02:44:56] <Alex_Gaynor> :D

[02:47:31] <dstufft> hmm

[02:47:37] <dstufft> 16 test failures with wheels

[02:47:38] <dstufft> but

[02:47:58] <dstufft> 1681.86 vs 2467.02

[02:48:05] <Alex_Gaynor> 1/3 faster, not too bad

[02:48:08] <Alex_Gaynor> that's PyPy?

[02:48:22] <dstufft> so unless the failures caused ~800s difference that should be a speed up

[02:48:24] <dstufft> yea that's PyPy

[02:48:26] <dstufft> on my mac

[02:49:17] <Ivo> dstufft: so if i understand correctly you're trying to run pip tests with a pip built into a wheel?

[02:49:38] <dstufft> Ivo: yes, it builds a wheel first, (and once) and then instalsl from that from there on

[02:49:54] <dstufft> some of the failures are because some tests are using an attribute I remved from the VirtualEnv class

[02:50:08] <dstufft> the others come from TestFailure: Path(u'venv/site-packages/easy-install.pth') unexpectedly not updated by install

[02:50:16] <dstufft> which is something I know is tricky on PyPy for some reason

[02:50:38] <dstufft> I don't think whatever causes that ever fails, but on PyPy it does sometimes if things aren't done in just the right order

[02:50:49] <dstufft> ever fails on CPython*

[03:04:49] <dstufft> https://github.com/pypa/pip/blob/develop/tests/lib/venv.py#L68-L73 best code

[03:10:20] <dstufft> https://github.com/pypa/pip/pull/1757 some test failures still there, but felt like PRing it

[03:13:31] <Alex_Gaynor> dstufft: god, when do we get to start killign setup.py in favor of some static metadata.

[03:13:55] <dstufft> I wish

[03:14:05] <dstufft> have some stuff that has to get done first

[08:20:05] <Ivo> Alex_Gaynor: something like https://www.youtube.com/watch?v=ePFWp3oSfyU&t=38s ?

[08:21:12] <Ivo> dstufft: maybe split up those two points in #1757 into two PRs and get the first one in if it works?

[09:17:13] <agronholm> Alex_Gaynor: trouble is, some of them have such complicated logic that switching to static metadata would necessarily reduce functionality

[09:17:57] <agronholm> esp. tricks regarding compilation of C extensions

[11:18:11] <Ivo> agronholm: that's build logic, not metadata logic

[11:19:21] <agronholm> but I've seen pretty convoluted logic for determining dependencies too

[11:19:27] <Ivo> agronholm: a big problem is that the canonical way to specify static metadata is through arguments to a funciton meant to be run dynamically through python code of a package, and not simply parsed

[11:20:20] <Ivo> agronholm: I think at least 90% of use cases static metadata is fine. And it's not as if a new metadata format couldn't be expanded to cover more use cases

[11:21:35] <Ivo> agronholm: if you haven't checked it out already, http://legacy.python.org/dev/peps/pep-0440/ comments and criticism would be welcome before it goes final and we can't change anything

[11:21:49] <agronholm> I just read it earlier today

[11:22:10] <agronholm> I didn't find any problems yet

[20:48:45] <dstufft> I hate our test suite

[20:48:47] <dstufft> it's the worst

[20:49:21] <agronholm> what do you hate about it

[20:50:36] <dstufft> the fact that its slow as dog shit, and it asserts against things that aren't always true, so you get into cases where things fail depending on what platform you run the test suite on, also our setup per test is really complicated and our assertions will fail if seemingly unrelated things aren't done

[20:50:44] <dstufft> basically it's super fragile

[20:50:46] <dstufft> and slow

[20:51:28] <agronholm> these should probably be raised as issues on the tracker

[20:52:12] <gauravb7090> is there a doap index for all packages?

[20:52:33] <agronholm> doap?

[20:52:38] <gauravb7090> Like a listing from where I could get the name and DOAP?

[20:53:15] <gauravb7090> https://pypi.python.org/pypi/2C.py/02%20Jun%202010 DOAP record

[20:55:24] <gauravb7090> agronholm: is there such a listing?

[20:55:46] <agronholm> what format should it be in?

[20:56:00] <agronholm> a listing of all packages would be fairly large methinks

[20:56:16] <gauravb7090> something like name and a link to their DOAP record?

[20:57:35] <agronholm> no, I mean, JSON, HTML, XML, what?

[20:58:01] <gauravb7090> HTML would be ideal…but anything else will also do

[20:58:20] <agronholm> curious about what you would do with HTML versions of those records

[20:58:38] <gauravb7090> Its related to my gsoc project :)

[20:58:57] <agronholm> which is?

[20:59:04] <gauravb7090> automated portfile generation for pip projects :)

[20:59:14] <agronholm> what's a portfile?

[21:00:23] <gauravb7090> https://trac.macports.org/wiki/SummerOfCode --> Read Packages from Python's PIP (pip2port) :)

[21:01:21] <agronholm> fyi, using eggs is not something that should be done at this day and age

[21:01:41] <agronholm> also, it should probably be "pypi2port"

[21:01:58] <agronholm> pip is just a tool that fetches stuff from pypi, but it's not the only one

[21:03:34] <gauravb7090> i'm sorry i didn't get your point with eggs? and sorry for it being pip2port and not pypi2port…I'll ask my mentors to change the same :)

[21:03:59] <agronholm> egg is a deprecated packaging format that is going away

[21:04:05] <agronholm> wheel is the replacement

[21:04:24] <agronholm> though the vast majority of distributions still only come in the sdist form (tarball/zip)

[21:11:31] <agronholm> gauravb7090: anyway, these are good things to know when you're implementing a thing like that

[21:11:56] <gauravb7090> yeah I was reading more on the eggs and wheel thing :)

[21:12:16] <agronholm> as for the DOAP records, I don't know if the existing PyPI has an API for getting those

[21:12:27] <agronholm> and scraping would be a terrible solution

[21:12:36] <gauravb7090> so I need to fetch them one by one?

[21:12:46] <gauravb7090> what better option do I have in hand?

[21:13:03] <gauravb7090> scraping is the most updated solution? if I'm not wrong?

[21:13:04] <agronholm> dstufft might be a better person to talk to about that

[21:13:17] <agronholm> most updated...?

[21:13:54] <gauravb7090> other than the not so uniform format…the html will be regularly updated? So, I can add them hand by hand!

[21:14:19] <agronholm> it would be best to get the listing in a machine readable format

[21:14:35] <agronholm> so you don't have to go through the costly and error prone phase of parsing the HTML

[21:15:24] <gauravb7090> machine readable format? please enlighten me….I would love to know more about it

[21:15:31] <agronholm> like JSON

[21:15:46] <gauravb7090> thats the reason I was looking for something like the DOAP records

[21:16:04] <dstufft> there is no mass listing like that

[21:16:05] <gauravb7090> does pypi have a JSON indexing of packages??

[21:16:18] <dstufft> there are 3 APIs

[21:16:53] <gauravb7090> JSON and XMLRPC?

[21:16:56] <gauravb7090> 3rd?

[21:16:59] <dstufft> 1) XMLRPC , which includes a way to get all of the package names, and all of the data 2) JSON, doesn't have a list of all the package names, and 3) DOAP

[21:17:07] <dstufft> the DOAP stuff I wouldn't recommend using

[21:17:41] <dstufft> I would recommend using JSON if you can, the format is /pypi/<name>/json or /pypi/<name>/<version>/json

[21:17:50] <dstufft> but if you need to use the XMLRPC api you can

[21:18:44] <gauravb7090> why would you not recommend the DOAP records?

[21:19:05] <gauravb7090> JSON would also do fine for me as far as my project scope is concerned

[21:19:21] <gauravb7090> XMLRPC seems to be the most apt for me at this time

[21:21:00] <gauravb7090> dstufft: DOAP records also have a uniform structure like the JSON? why would you not recommend using them?

[21:21:43] <dstufft> because someday I'd love to get rid of them

[21:22:18] <gauravb7090> why is it so? :P

[21:23:07] <gauravb7090> sorry but I'm actually new to all this stuff….:) I dont know much about DOAP but I liked the idea behind it :)

[21:37:46] <dstufft> gauravb7090: DOAP is a thing with nice ideas but very little adoption

[21:45:55] <gauravb7090> ohh okay :)

Log file Viewer

Help | Karma | Search:

#pypa-dev logs for Friday the 25th of April, 2014