Log file Viewer

[16:02:39] <hazzadous> I'm having a hard time trying to make setuptools egg_info run in sensible time with my project. There are alot of files that are safe to ignore, but having a look at in implementation it looks like irrespective of settings, setuptools will stat the entire directory tree

[16:02:50] <hazzadous> Am I missing something? I'm looking at bitbucket.org/pypa/setuptools/src/a743083d0f0a4e2e3e1f4ab1dd44db610ec7694c/setuptools/command/egg_info.py?at=postproc#cl-245

[16:05:13] <hazzadous> I'm not actually interested in this specific command, but it appears to be a prerequisite for `install`

[16:42:35] <Ivo> hazzadous: what does your setup.py look like

[16:45:03] <hazzadous> Ivo: gist.github.com/hazzadous/ad85db37fb89b69b2e9d

[16:46:31] <hazzadous> It's a little bit of a mess and doesn't make much sense at the moment, but my testing is just to see if I can prevent it from looking in .git (which I can tell it does via strace)

[16:48:51] <hazzadous> at any rate, findall looks like it will always look everywhere: svn.python.org/projects/python/branches/pep-0384/Lib/distutils/filelist.py

[16:59:44] <Ivo> hazzadous: you could use the package_data setup() keyword, instead of MANIFEST.in and include_package_data

[17:00:53] <hazzadous> I'll give it a try Ivo although sure I already have

[17:01:11] <Ivo> I think using package_data instead of include_package_data might turn off the vcs search

[17:02:00] <Ivo> jaraco: does that sound right to you?

[17:03:09] <Ivo> jaraco: afaict hazzadous is having a problem where the size of their git repo is making the vcs searching that setuptools does take extremely long

[17:04:05] <hazzadous> indeed Ivo , plus all its submodules...

[17:04:12] <hazzadous> back in 10...

[17:04:40] <Ivo> during build / egg_info

[17:09:19] <mgedmin> hazzadous, if all you care is making 'install' fast, how about building a wheel so you could install it really quickly?

[17:12:34] <jaraco> Ivo, hazzadous, I'm actually unsure under what conditions the files are found using file_finders. It might be the case that it's only when include_package_data is true, but I'd have to go to the source to be sure.

[17:12:38] <jaraco> (or the docs)

[17:14:17] <jaraco> hazzadous, it may also be the case that there are performance improvements to be had with the git file finder - I'd recommend filing a ticket with that project first, as any other solution is going to require you to maintain your package data in two places (source control and in a file), whereas using file_finders is more DRY.

[17:14:39] <jaraco> It's conceivable they haven't considered your use case, so could use it to improve the performance for everyone.

[17:35:40] <hazzadous> jaraco, Ivo I'm not really sure that it's even anything to do with git finder. Looking at the egg_info cmd, it will always call find_sources (which builds SOURCES.txt), which will always call findall, which will always walk the whole dir tree unless I'm being silly.

[17:36:38] <jaraco> That's probably true if you're talking about source files and not non-Python files.

[17:36:44] <Ivo> i would've assumed that it wouldn't include dotfiles by default, but i could be wrong

[17:36:51] <jaraco> hazzadous, in that case, your issue may lie in distutils and not setuptools.

[17:37:17] <hazzadous> Ivo, jaraco yeah I think thats the case. I'll get back to you if not

[17:37:20] <jaraco> It sounds like you've traced the issue. hazzadous, do you know if the offending profile is primarily in distutils or setuptools?

[17:38:48] <jaraco> Once you know, I suggest filing a ticket with either Python or Setuptools, preferably with zip file or script that can demonstrate the slow behavior. Even if the issue is with distutils, a fix would only happen in Python 3.3+ and maybe only 3.5+, so Setuptools would be on the hook for backward-compatibilty.

[17:38:54] <jaraco> *could be on the hook.

[17:39:30] <Ivo> jaraco: seems ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDFKanjJD9IgTkYxXY5eHuemZ5ThhPRMPiWf0GfyqzP/j1k+7H5BiI2IbmupSyWZ3mjGGvtdOePx9sAul5J2kaEraRVDc60znFbd2vZ0iaiPZ7yVlQQKkXU5QhXMrpojp/YAwlfFKT6tyMXEV79lPe7shHwaDkVgYkkdEmJO9rDMUswm47WHwPsHem1J3kz7lf+p07mPQlNCPjtYqJ1kNIbO8G89mQCdyC70QYKRnBnosCK+QhkR4aSWdnpTYoPknJ64hjOqiEotEr9T9J8eOQE5Soo99PYUfesnjkVkAe07GaqO6t72f30EegDzkdyU35LW+wvom/mxu9TW1tNYrSt Ivoz

[17:39:41] <Ivo> how does that get into my clipboard

[17:39:48] <hazzadous> Sorry, I've got small distracting people here. I'll have to get back to you on this...

[17:41:33] <jaraco> Ivo: you can always strike things from the logs if you don't want them on the record.

[17:41:52] <jaraco> of course, a public key is not particularly important to strike.

[17:42:37] <Ivo> meh fudge it, as you said only a public key

[17:42:50] <Ivo> its hanging around on github anyway

[17:44:32] <Ivo> low hanging fruit would be line #299 of that file as well, only cares about older vcs' folders

[17:44:58] <dstufft> a private key isn't too important to strike either, becuase once it's posted it's no longer private and you have to revoke it anyways :D

[17:46:09] <Ivo> thats the one i added for pypa salt

[17:46:38] <Ivo> was thinking of maybe putting an index page on bootstrap.pypa.io if that wasn't too meddlesome

[17:46:56] <jaraco> Ivo: yep, that's the manifest maker... but I'm not sure when/how that's called, so I'll wait until I have a better problem description (and more time) before I investigate further.

[17:48:11] <dstufft> Ivo: that wouldn't be bad!

[17:49:11] <Ivo> just need to figure out how not to break your webserving stack...

[17:55:13] <dstufft> Ivo: just use salt to shove a file in /srv/bootstrap/www/index.html

[20:56:50] <felipec> is this the best way to install the latest development version of Mercurial?

[20:57:00] <felipec> pip install -e hg+selenic.com/hg#egg=Mercurial

[20:57:27] <felipec> I want this for a testing script

[21:20:02] <Ivo> felipec: I would imagine the mercurial project would be able to more authoratively tell you than us, but that's about how you generally install the latest committed version of a python project, ya

[21:21:33] <felipec> Ivo: I was just wondering if that clones the whole repository and maybe there was a quicker way

[21:22:28] <Ivo> afaik that would normally clone the whole repo, yeah. If hg has a way only to retrieve the latest state of a repository, you can do that, and then give the same command but pointed to the directory you cloned in

[21:23:00] <Ivo> or maybe mercurial offer a snapshot tarball/zipball of the latest head

[21:23:37] <Ivo> 3.0 was released on pypi just a few days ago it seems

[21:25:23] <felipec> Ivo: yeah 3.0 is released, but then comes 3.1, and so on

[21:26:58] <Ivo> 3.1 surely isn't as big a deal as 3.0 tho

[21:29:00] <felipec> Ivo: sure, but I test against multiple versions of Mercurial, why not the development one?

[21:29:37] <felipec> I'm getting the wrong version with that command though =/ 2.9.2+282-024f38f6d5f6

[21:29:57] <Ivo> well grabbing it from pypi would be faster than cloning, is all, but I'm just suggesting stuff :)

[21:31:35] <Ivo> felipec: mercurial.selenic.com/wiki/Download#Source_packages

[21:31:43] <felipec> Ivo: yes, I grab from pypi released versions, but if I want the latest development...

[21:31:57] <Ivo> there you go

[21:32:12] <Ivo> selenic.com/repo/hg/archive/tip.tar.gz

[21:32:54] <felipec> Ivo: yeah, I just found that... now I need to figure out how to tell pip to use that

[21:33:10] <Ivo> pip install http://...

[21:33:48] <Ivo> maybe needs a #egg=Mercurial on the end

[21:38:13] <felipec> Ivo: OSError: [Errno 13] Permission denied: '/usr/lib/python2.7/site-packages/mercurial-3.0-py2.7.egg-info'

[21:38:40] <Ivo> felipec: do you want to install that mercurial system wide?

[21:38:49] <Ivo> felipec: I'd recommend using a virtualenv instead

[21:38:55] <Ivo> for testing out development versions

[21:39:11] <Ivo> of stuff, since it's isolated

[21:39:13] <felipec> Ivo: I'm running this: pip2 install --user --upgrade selenic.com/repo/hg/archive/tip.tar.gz#egg=Mercurial

[21:39:17] <Ivo> tox is also great for that

[21:39:37] <felipec> Ivo: I think the problem is that there hasn't been any changes sin v3.0, so it's the same version as in my machine

[21:39:38] <Ivo> try without the --upgrade

[21:39:51] <felipec> Ivo: that's what I did initially... same error

[21:40:04] <Ivo> Not exactly sure why --user is tring to write to /usr/lib

[21:40:33] <Ivo> what version of pip is it?

[21:40:34] <felipec> Ivo: Installing collected packages: Mercurial

[21:40:34] <felipec> Found existing installation: mercurial 3.0

[21:40:39] <Ivo> pip2 --version

[21:40:48] <felipec> Ivo: pip 1.5.4 from /usr/lib/python2.7/site-packages (python 2.7)

[21:45:11] <Ivo> felipec: you could use --ignore-installed but I'm not sure about the uninstall behaviour; again, I'd recommend working in a virtualenv for testing development or multiple versions of things

[21:45:13] <felipec> Ivo: I think maybe this is not an issue anyway... since I want this for TravisCI, and it has the permissions to override the system I think

[21:45:46] <felipec> Ivo: yeap, --ignore-installed works :)