pmxbot IRC Log Viewer

[00:39:20] <dstufft> So

[00:39:24] <dstufft> I'm going to do something horrible

[00:39:33] <dstufft> But it's the only way

[00:39:34] <dstufft> :|

[00:39:38] <Ivo> adding new code

[00:39:57] <dstufft> not just new code

[00:40:06] <dstufft> this is going to monkeypatch setuptools while it monkeypatches distutils

[00:40:20] <dstufft> also it's going to do it inside of a python -c "some code" evalfile("setup.py")"

[00:40:24] <dstufft> or whatever the fuck

[00:40:26] <dstufft> because YOLO

[00:41:03] <Ivo> is that to do with da apple clangup

[00:41:11] <dstufft> No

[00:41:24] <dstufft> https://github.com/pypa/pip/issues/1517#issuecomment-37359548

[00:41:43] <Ivo> why is setuptools downloading stuff

[00:41:51] <dstufft> because setup_requires

[00:42:06] <dstufft> you have to download setup_requires before you can interpret a setup.py

[00:42:13] <Ivo> ...really

[00:42:28] <dstufft> because the things in a setup_requires might change the meaning of a setup.py

[00:42:53] <Ivo> wait but you have to interpret a setup.py to inspect the contents of setup_requires

[00:42:53] <dstufft> for instance, if you're using pbr your setup.py looks like this https://github.com/stackforge/python-barbicanclient/blob/master/setup.py

[00:43:19] <dstufft> well setuptools installs it before it actually executes the setup() function

[00:43:21] <dstufft> basically

[00:43:22] <dstufft> sort of

[00:43:26] <dstufft> don't think too hard about it

[00:44:06] <Ivo> with magical ast parsing!?

[00:44:12] <dstufft> Nah

[00:44:18] <dstufft> when you pass some values into setup()

[00:44:23] <dstufft> it checks setup_requires first

[00:44:28] <dstufft> and installs some things if it needs to

[00:44:36] <dstufft> then it loads other stuff using entry points

[00:44:42] <dstufft> and uses those to do the actual heavy lifting of things

[00:47:49] <Ivo> python-barbicanclient: http://www.megdesk.com/blog/wp-content/uploads/2010/02/computer_engineer_barbie_revised.jpg

[00:49:14] <Ivo> I thought I also saw an issue in setuptools about having a hook for downloading things, would that not also solve your issue dstufft

[00:49:25] <dstufft> git checkout -b eldritch-horror

[00:49:28] <dstufft> Ivo: does it have a hook?

[00:49:31] <dstufft> can you link?

[00:50:30] <Ivo> well, an issue anyway https://bitbucket.org/pypa/setuptools/issue/149/api-to-customise-file-downloading

[00:51:39] <Ivo> so.. does pip parse install_requires either?

[00:52:22] <Ivo> or does it get all that from other metadata

[00:52:30] <dstufft> no, pip calls setup.py egg_info command, which causes setuptools to generate an egg-info which has some static metadata inside it

[00:52:59] <dstufft> so after pip calls egg_info, it reads that static metadata

[00:53:10] <Ivo> which doesn't have setup_requires?

[00:53:49] <dstufft> it might, but calling setup.py egg_info will cause setuptools to install setup_requires

[00:54:13] <Ivo> ...bastards

[00:54:13] <dstufft> because you can't interpret the setup.py without stuff in setup_requires

[00:54:42] <dstufft> so

[00:54:48] <dstufft> I'm gonna monkeypatch setuptools.setup

[00:54:52] <dstufft> I think

[00:55:03] <dstufft> I need to make sure that works if a thing imports distutils.core

[00:56:33] <Ivo> fuck, it does

[00:57:49] <Ivo> setuptools should make its own canonical pbr with blackjack and hookers and no requirements funging

[00:58:29] <Ivo> ...or not

[01:03:06] <dstufft> https://gist.github.com/dstufft/9498533

[01:03:07] <dstufft> hell yes

[01:03:11] <dstufft> that was easy

[01:04:13] <Ivo> I hope your dinner date with cthulu goes well

[01:08:41] <dstufft> lol

[01:08:58] <dstufft> I also have to figure out how to shuttle all the options over

[01:09:05] <dstufft> so that --cert and --proxy and stuff works

[01:09:31] <dstufft> I should just pickle the entire running state of the process and unpickle it in the setup.py

[01:10:58] <dstufft> hrm

[01:11:01] <dstufft> let's see

[01:11:05] <dstufft> do I have a build directory yet

[01:11:07] <dstufft> probably not

[01:36:48] <Ivo> dstufft: just use pip to install all setup_requires packages?

[01:37:11] <dstufft> Ivo: that's what i'm going to do, but this is inside of a subprocess so I have to pass all of the configuration options into the subprocess

[01:38:11] <Ivo> this feels so unclean...

[01:38:22] <dstufft> it is

[01:38:24] <dstufft> it's horrible

[01:38:51] <Ivo> implementing a setuptools download hook would be like, tonnes nicer

[01:39:04] <dstufft> sure

[01:39:05] <dstufft> but

[01:39:10] <dstufft> then we'd require a new setuptools

[01:39:25] <dstufft> which we don't right now unless you're using wheels

[01:39:28] <Ivo> could always patch it onto 2.2

[01:40:09] <Ivo> the bug is already from setuptools

[01:40:47] <Ivo> you can't encounter this bug without setuptools installed

[01:41:08] <dstufft> correct

[01:41:12] <dstufft> but a lot of people use old setuptools

[01:41:53] <Ivo> some people still use 0.6 distribute and that's even more broken

[01:42:07] <Ivo> you patching over its http at the same time?

[01:42:24] <dstufft> Nope

[01:42:34] <dstufft> this isn't just for HTTPS though

[01:42:36] <dstufft> this also makes it consistent for --pre rules and such too

[01:42:47] <dstufft> well

[01:42:48] <dstufft> actually

[01:42:56] <dstufft> this'll make make 0.6 secure too

[01:43:12] <dstufft> because with pip the only time setuptools should hit the network is for setup_requires

[01:43:17] <dstufft> and i'm taking that over

[01:43:42] <Ivo> the bigg problem is build and require and fetch and configure and compile are all strewn about the place in two or three or four different tools T_T

[01:44:05] <dstufft> yup

[01:44:08] <dstufft> it's horrible

[01:44:26] <dstufft> I have some ~ideas~ for fixing it long term

[01:44:35] <dstufft> but it's going to require a bunch of refactoring

[01:46:42] <Ivo> so setuptools and pip both duplicate dependency resolution

[01:47:08] <dstufft> yea

[01:47:23] <dstufft> what I'd love to do is make setuptools just a build tool, but I have no idea if jaraco would be OK with that

[01:47:56] <Ivo> i bet too many people would complain

[01:48:09] <Ivo> need setuptools2 :P

[01:49:13] <Ivo> would be cool if pip and setuptools could both vendor distlib, imo

[01:49:17] <Ivo> and start making use of that

[01:49:24] <dstufft> I don't really like distlib much tbh :[

[01:49:27] <dstufft> but w/e

[01:49:53] <Ivo> what dontchya like about it

[01:50:15] <dstufft> It's filled with random shit that isn't based on any standard and exists only because vinay thought it was cool

[01:50:28] <dstufft> like a JSONLocator that uses his own custom JSON format that he hosts on his own server

[01:50:49] <dstufft> also I don't really like vinay's coding style at all :/

[01:50:49] <Ivo> i thought that was from /package/json

[01:51:11] <Ivo> it seems to have a nicely working metadata parser :/

[01:51:27] <Ivo> for all of pythons oh-so-nicely standardised metadata files!

[01:51:47] <dstufft> Nopit has two

[01:51:52] <dstufft> PyPIJsonLocator

[01:51:59] <dstufft> which uses /package/json

[01:52:08] <dstufft> and JsonLocator

[01:52:12] <dstufft> JSONLocator*

[01:52:16] <dstufft> which uses his custom format

[01:52:28] <dstufft> also parsing python's metadata files is pretty easy

[01:52:33] <dstufft> you can do it with the built in email module

[01:56:03] <Ivo> hmmm

[01:56:45] <Ivo> it is a bit weird to have a red-dove.com url hardcoded into a pypa project

[01:57:00] <dstufft> I don't know if he actually hardcoded that in or not

[01:57:13] <dstufft> it might get passed in through his distil project I dunno

[01:57:21] <Ivo> i just found it, which is why i just said that

[01:57:24] <dstufft> ah

[01:57:46] <dstufft> but distlib does stuff right now that we don't have a better option for

[01:57:47] <dstufft> so we use it

[01:57:56] <dstufft> I'd like to not to use it, don't know if i'm alone in that or not

[01:58:01] <Ivo> https://bitbucket.org/pypa/distlib/src/4459756575cfeaa52af98744117deb78c0c06dd8/distlib/util.py?at=default#cl-758

[01:58:27] <dstufft> I own packaging on PyPI!

[01:58:31] <dstufft> son of packaging

[01:58:33] <dstufft> packaging2

[01:58:35] <dstufft> :3

[01:58:37] <dstufft> distutils3!

[01:59:04] <Ivo> haha

[01:59:08] <Ivo> jaraco.packaging

[01:59:45] <Ivo> oh that's some newish thing he's working on

[02:01:26] <Ivo> dstufft: even if it's distlib-rewritten, or distlib-stripped-down, I do think some package of all this common code implementing common functionality would be useful

[02:02:17] <Ivo> unless setuptools should vendor in wheel eventually... :)

[02:27:42] <Ivo> qwcode: considered putting in a PR to amend: "...tool for installing, uninstalling, listing and showing Python packages, and freezing them too!!"

[02:41:30] <Ivo> dstufft: you've already documented setuptools' deficiency - "pip has no way to control how these dependencies are located. None of the Package Index Options have an effect."

[02:41:49] <Ivo> now imo you're selectively trying to fix one them

[02:42:05] <dstufft> Ivo: no i'm trying to fix all of them

[02:42:17] <dstufft> pip is taking complete control of setup_requires

[02:42:19] <dstufft> not just in one case

[02:42:27] <dstufft> but in every case when you invoke ``pip``

[02:42:27] <Ivo> oh all of them

[02:42:35] <dstufft> it's going to use the PackageFinder

[02:42:51] <Ivo> well at least that's consistent :P

[02:43:21] <dstufft> "You can't document your way out of a usability problem"

[02:44:26] <Ivo> there are different approaches to coding your way out, though

[02:47:02] <dstufft> anything that requires modifying setuptools means it'll be 3.2+, there are lots of people still using 0.6

[02:47:10] <dstufft> (because their package manager forces them too)

[02:47:55] <Ivo> bit their problem won't be fixed until they use pip 1.[5/6].x, and their package manager probably won't provide them with that either

[02:48:12] <Ivo> and if they can do pip install -U pip why not pip install -U setuptools

[02:48:37] <dstufft> because stuff in say apt-get depends on setuptools, so it's likely they have setuptools already installed

[02:49:08] <dstufft> e.g. it's infeasable to not have setuptools installed by the Os package manager

[02:49:22] <dstufft> and if you upgrade it with pip, then you get dueling package managers

[02:49:48] <dstufft> which might either break completely, or just silently go back to the old behavior

[02:50:07] <dstufft> nothing depends on pip, so it's a lot easier to manage that completely outside of the OS package manager

[02:50:44] <dstufft> when pip 1.5 came out, it started out having a hard dependency ons etuptools 0.8+ (this was when 2.1 was the latest setuptools version)

[02:50:48] <dstufft> and it broke the world for a ton of people

[02:51:33] <Ivo> imagine how many people could have up-to-date setuptools now if it continued to break the world :3

[02:54:05] <qwcode> Ivo, "and building wheels for dependencies"

[02:54:45] <Ivo> qwcode: in the pip blurb line?

[02:55:10] <qwcode> yes

[02:58:30] <Ivo> dstufft, qwcode if i were to try letting pip list installed files from wheels, do you think <pkg>.dist-info/installed-files.txt would be the best place to list that

[02:59:35] <Ivo> i.e, 'same place as it was for .egg-info'; or do you think that's 'dirtying' the .dist-info folder

[03:00:01] <qwcode> Ivo, should use RECORD I think

[03:02:01] <qwcode> dstufft, pretty sure pip-1.4 started the setuptools 0.8 requirement. but I guess turning on wheels by default made it more noticeable

[03:02:25] <dstufft> qwcode: right, with pip1.4 you only needed that if you utnred on --use-wheel or executed pip wheel

[03:02:32] <dstufft> with 1.5 you needed it to do anything

[03:02:36] <dstufft> or well install anything

[03:03:15] <Ivo> qwcode: RECORD's relative path root is <site-packages> (plus absolute paths for scripts, yay), whereas installed-files.txt's is <site-packages>/<pkg>.egg-info (plus relative paths for scripts, yay)

[03:03:26] <qwcode> dstufft, not for installing sdists, right?

[03:04:22] <Ivo> so the question is do you try and make pip's listing for wheel-installed-files absolutely consistent with installed-files.txt's

[03:05:34] <dstufft> qwcode: well in 1.5 you needed 0.8 even for installing sdists unless you said --no-use-wheel

[03:06:45] <qwcode> ah, use_wheel forced the setuptools check I guess,even if you weren't actually installing a wheel

[03:07:00] <qwcode> history...

[03:09:02] <qwcode> my neighbor and I must have the same fan remote control setting.... this is annoying....

[03:09:24] <Ivo> so you can control your neighbour's fan?

[03:09:51] <qwcode> something can sure control mine. that's all I know

[03:10:23] <qwcode> pulling out the ladder and screwdriver...

[03:10:43] <Ivo> maybe its the nsa

[03:53:08] <Ivo> is a .dist-info ever not from a wheel?

[04:04:19] <dstufft> https://github.com/dstufft/pip/compare/pypa:develop...dstufft:eldritch-horror

[04:04:23] <dstufft> I am pretty sure this is evil

[04:04:25] <dstufft> but it works

[04:15:53] <Ivo> I wonder if anyone is string matching against 'Cannot locate installed-files.txt'

[04:16:59] <Ivo> hah, the tests are

[04:34:10] <Ivo> what's a nice, small, simple, stable, python2+3 package

[05:35:10] <dstufft> new method

[05:35:14] <dstufft> less bad, but still bad

[05:35:15] <dstufft> https://github.com/dstufft/pip/compare/pypa:develop...dstufft:eldritch-horror

[05:48:12] <Ivo> dstufft: you have a print statement in that diff?

[05:49:11] <dstufft> Ivo: proof of concept ;P

[05:49:46] <Ivo> meanwhile, getting pip to show files installed by wheels wasn't too hard ^_^

[05:50:22] <Ivo> Also, I figured out that some of the tests may be being made longer by doing unneccesary network requests

[05:50:45] <Ivo> like initools gets installed from the network for many functional tests?

[05:51:34] <Ivo> might shave off some minutes if that got substituted for installing from fs

[06:21:23] <Ivo> at the airpoort again?

[06:24:02] <dstufft|laptop> nah

[06:24:08] <dstufft|laptop> just sitting on the couch

[06:34:04] <Ivo> dstufft|laptop: do you have a user auth token for warehouse? (assume not for pypi)

[06:34:27] <dstufft|laptop> there's no user auth in warehouse at all yet

[06:35:48] <Ivo> dstufft|laptop: my actual bug is - i have to put a password in ~/.pypirc. I would much rather put a random token in there, so that I can keep a password I can remember to log in, but not have to have it plaintext on my filesystem

[06:36:25] <dstufft|laptop> twine lets you specify the password on the command line instead of in your pypirc

[06:37:07] <Ivo> half-way there, but still making me enter a password every time i interact with pypi on the command line

[06:37:42] <Ivo> does twine have the register command?

[06:37:45] <dstufft|laptop> nope

[06:37:50] <dstufft|laptop> use the webui

[06:38:30] <dstufft|laptop> Warehouse isn't doing much in the way of new features ATM

[06:38:35] <dstufft|laptop> mostly reimplementing pypi first

[06:38:37] <dstufft|laptop> then doing new

[06:39:18] <Ivo> do you have your vcs keys passworded?

[06:40:40] <dstufft|laptop> vcs keys?

[06:40:55] <Ivo> i.e key you use to auth to github?

[06:41:07] <dstufft|laptop> oh the ssh auth keys? yea

[06:53:00] <Ivo> qwcode: if the wheel cache is checked prior to pypi how do you know if a newer package is available?

[06:54:22] <qwcode> Ivo, you'd check for non-versioned reqs and upgrades

[06:55:50] <Ivo> qwcode: ah, ok. maybe you could write out a short specification of the exact behaviour of it

[06:57:15] <qwcode> Ivo, will do when I get back to it. still unfinished atm

[06:59:19] <qwcode> Ivo, before this became known https://pip.readthedocs.org/en/latest/user_guide.html#fast-local-installs

[06:59:34] <qwcode> Ivo, we got more complaints about the --download-cache

[06:59:42] <qwcode> I complained about it early on

[09:10:20] <ronny> hi

[09:10:43] <ronny> btw, anyone working on install requires/data hooks for setup.cfg?

[09:14:45] <jezdez> ronny: https://github.com/embray/d2to1 has it

[09:14:54] <jezdez> but obviously that's a rather hacky way

[09:15:07] <jezdez> https://github.com/embray/d2to1/blob/master/d2to1/tests/test_hooks.py

[09:15:44] <jezdez> pbr being a fork of it has them, too: https://github.com/openstack-dev/pbr/blob/master/pbr/tests/test_hooks.py

[09:41:07] <ronny> oh, meh - no good yet

[15:24:07] <Ivo> dstufft: have you told jaraco about your efforts yet

[15:24:24] <Ivo> oh, yes

[21:36:09] <Ivo> dstufft: just use sha256 and shorten to 128bytes?

[21:36:48] <dstufft> the hex digest of sha256 is only 64 bytes

[21:37:20] <dstufft> the hex digest of sha512 is 128 byets

[21:38:58] <dstufft> I just do len(sha256(b"a").hexdigest())

[21:38:59] <dstufft> heh

[22:59:22] <Ivo> is there a way to get easy_install to use another pypi index during python setup.py test

[23:00:52] <dstufft> you might need to edit ~/.pydistutils.cfg

[23:05:33] <Ivo> found the right way

Log file Viewer

Help | Karma | Search:

#pypa logs for Wednesday the 12th of March, 2014