pmxbot IRC Log Viewer

[00:55:28] <lifeless> dstufft: topo sort pushed up :)

[19:46:52] <lifeless> dstufft: o/

[19:47:37] <lifeless> I'm not sure why tests are failing, will poke later today

[20:21:09] <lifeless> dstufft: can I ask a favour?

[20:21:31] <lifeless> hmm, actually I probably can.

[20:21:54] <lifeless> I want to know the ratio of packages w/ setup-requires items that are not in their install-requires

[20:22:02] <lifeless> to those that have setup-requires that are also in install-requires.

[20:25:50] <lifeless> dstufft: I know you've done many such analyses, do you have a script that encompasses the boilerplate?

[20:26:03] <sigmavirus24> I have no clue how easily dstufft would be able to collect that

[20:26:11] <lifeless> I have a bandersnatch mirror here

[20:26:18] <lifeless> but no existing analysis code

[20:27:28] <sigmavirus24> Like, I can imagine one way of figuring it out, but your disk would hate you

[20:28:24] <lifeless> sigmavirus24: the mirror is sitting on an 8-disk mdadm raid array

[20:28:32] <sigmavirus24> Not in that sense

[20:28:47] <lifeless> sigmavirus24: oh?

[20:28:58] <lifeless> sigmavirus24: one venv per thing, install and inspect?

[20:29:02] <sigmavirus24> I think the way to figure out what the setup_requires are vs. install_requires is to create virtual environments and run the correct commands

[20:29:03] <sigmavirus24> yeah

[20:29:04] <sigmavirus24> then destroy

[20:29:07] <sigmavirus24> then new virtualenv

[20:29:11] <lifeless> so I was thinking

[20:29:16] <sigmavirus24> That's why pip's tests are slow fyi

[20:29:20] <lifeless> unpack all the sdists

[20:29:29] <lifeless> some of pip's tests hit the internet unintentionally

[20:29:33] <dstufft> I don't have anything laying around ATM

[20:29:38] <sigmavirus24> Try to statically analyze the setup.py?

[20:29:41] <lifeless> thats a huge chunk of the slow

[20:29:42] <dstufft> you can kind of fake it by looking for egg-info

[20:29:54] <dstufft> oh, that doesn't tell you setup_requires tho

[20:29:59] <lifeless> sigmavirus24: yeah, grep for setup_requires in setup.py

[20:30:07] <dstufft> you can possibly parse the AST

[20:30:20] <lifeless> sigmavirus24: then assess how to analyse those based on the size of the result

[20:30:23] <sigmavirus24> dstufft: I hate and don't hate that idea simultaneously

[20:30:42] <sigmavirus24> I hate it because the AST library's API is awful

[20:30:49] <sigmavirus24> I don't hate it because it seems reasonable

[20:31:15] <dstufft> it's not perfect ofc, but setup_requires seems to be the thing least likely to be doing some magic with

[20:31:22] <lifeless> it rapidly gets into the evil magic needed to figure out setup_requires statically

[20:31:34] <sigmavirus24> lifeless: yep

[20:31:35] <dstufft> or just monkeypatch some code

[20:31:38] <dstufft> inside the setup.py

[20:31:42] <dstufft> :|

[20:31:43] <dstufft> I've done that before

[20:32:32] <sigmavirus24> unless it's unimportable without having the requirements to run setup.py =P

[20:32:56] <lifeless> dstufft: all the tar.gz's are in source/ right ?

[20:33:12] <dstufft> lifeless: yea I think so

[20:33:25] <dstufft> I normally just do things to that dir anyways

[20:33:34] <dstufft> lifeless: fwiw I think there are two classifications of setup_requires

[20:34:28] <dstufft> things like setuptools_scm and pbr (ignoring the fact pbr has the runtime version thing which is sort of tangetial to the rest of pbr)

[20:34:33] <dstufft> and d2to1

[20:34:34] <dstufft> etc

[20:34:46] <dstufft> and things like cffi in cryptography

[20:34:59] <dstufft> and numpy, and other things like that

[20:35:57] <dstufft> acutally, let me write that as a comment

[21:34:52] <lifeless> wow you get some crap in setup.py

[21:34:56] <lifeless> sorry in tarballs

[21:34:57] <lifeless> nolearn-0.5b1/local//home/daniel/co/nolearn/setup.py

[21:35:10] <lifeless> actual contents of a tar

[21:37:39] <lifeless> oh

[21:37:45] <lifeless> http://bugs.python.org/issue14160

[21:52:51] <lifeless> right, thats hould fix the PR

[21:59:15] <lifeless> hahahaa

[21:59:16] <lifeless> :!file /var/spool/pypi/web/packages/source/J/Jeiji/jeiji.tar.gz

[21:59:16] <lifeless> /var/spool/pypi/web/packages/source/J/Jeiji/jeiji.tar.gz: RAR archive data, v1d, os: Win32

[22:09:00] <lifeless> ok I have 72M of setup.py/setup.cfg/errors.

[22:09:03] <lifeless> time to add zip

[22:09:07] <lifeless> then move to analysis

[22:09:45] <ionelmc> lifeless: what are you testing?

[22:16:01] <lifeless> ionelmc: I'm putting together a crude heuristic to answer the question about how many setup_requires specify requirements not present in install_requires

[22:17:46] <ionelmc> lifeless: how is that info useful?

[22:28:25] <lifeless> ionelmc: much of the debate about the PR is focused on the potential harm from installing setup_requires into the target environment

[22:28:54] <lifeless> ionelmc: to quantify that we need to know *what* setup_requires are not already always installed into the target environment.

[22:29:32] <ionelmc> lifeless: but that won't work when you got circular setup_requires between packages

[22:29:36] <dstufft> I think PEP 426 and the earlier PEPs (and years of setup.py :V) has gotten everyone on board with declaritive dependencies in the abstract concept

[22:29:39] <ionelmc> not sure if it acutally happens

[22:29:41] <lifeless> ionelmc: that can't work today anyway

[22:29:43] <ionelmc> you could check

[22:29:48] <lifeless> ionelmc: since the install process will fail

[22:30:02] <dstufft> yea I don't think that can work, it'd just have setuptools recursively installing things :V

[22:30:25] <lifeless> total setup.py's vs those with setup_requires - 49561 1866

[22:30:47] <dstufft> lifeless: how did you determine that? did you just essentially grep for setup_requires?

[22:31:01] <ionelmc> not that many, lets remove setup_requires!

[22:31:11] <ionelmc> #notserious

[22:31:47] <dstufft> 800 of those are openstack libraries

[22:33:40] <ionelmc> dstufft: if you only count those that have a readme you prolly get under 100 :-)

[22:33:45] <lifeless> dstufft: https://github.com/rbtcollins/banderscan

[22:37:07] <dstufft> lifeless: cool, so basically just a string match flor ``setup_requires``

[22:42:13] <lifeless> so far

[22:42:17] <lifeless> working up an ast experiment now

[22:43:47] <dstufft> lifeless: I haven't seriously worked with ast much, but make sure things like **{"setup_requires": []} is handled too

[22:44:32] <sigmavirus24> lifeless: fair warning, the ast is rough

[22:45:05] <lifeless> who puts yield from in a setup.py

[22:45:13] <sigmavirus24> people who hate python 2

[22:45:17] <sigmavirus24> damnit, I should do that

[22:45:19] <lifeless> dstufft: things I can't handle are flagged

[22:45:24] <sigmavirus24> ;)

[22:45:41] <lifeless> dstufft: if we can get a 90% answer

[22:45:44] <lifeless> dstufft: thats good enough for me.

[22:45:55] <dstufft> yea a 90% answer is good enough, I dunno how common the one thing is

[22:46:24] <dstufft> it's a pain in the ass to get any real information about what people are doing

[22:46:31] <dstufft> 90% is better than what we typically make choices with :D

[22:46:47] <dstufft> (which tends to be, whoever showed up to an argument on distutils-sig)

[22:47:02] <ionelmc> most of the "setup_requires" setup.py's i've seen put use a different data structure for all the options (instead of direct arguments to setup())

[22:47:38] <ionelmc> that'd be hard to pull out with a mere ast visitor

[22:49:50] <ionelmc> sigmavirus24: you repro'd that segfault in the end?

[22:50:20] <sigmavirus24> not yet

[22:50:52] <sigmavirus24> (haven't had the chance to go back and re-reinstall everything)

[23:23:44] <lifeless> woo

[23:23:45] <lifeless> ['vcversioner>=1']

[23:29:54] <lifeless> ok I think I have enough data

[23:31:25] <lifeless> hmm, not quite

[23:31:30] <lifeless> 1183 out of the 1800

[23:37:27] <ionelmc> dstufft: does readme allow `Unknown interpreted text role` kind of errors?

[23:39:36] <lifeless> 75%

Log file Viewer

Help | Karma | Search:

#pypa-dev logs for Wednesday the 25th of March, 2015