pmxbot IRC Log Viewer

[12:11:04] <ronny> dstufft: ping?

[12:11:13] <dstufft> ronny: heya

[12:12:33] <ronny> dstufft: this weekend i'll try to prototype a new packaging tool, target is to have something that can generate universal wheels and setuptools using sdists, but no more scripts to declare the packages, stricter layout with src/tests

[12:14:25] <dstufft> ronny: ok :D I'm excited to see what you come up with

[12:15:00] <ronny> dstufft: i'd like to run the basic idea trough you once before starting with code in order to get some thins more solid

[12:15:48] <ronny> right now is bad tho, im still at work ^^

[12:16:11] <ronny> and i suppose you also have a workday ahead?

[12:17:47] <dstufft> yea, though half my time is packaging so me being at work doesn't preclude me from talking about packaging things :) But I have a bunch of phone calls to make this morning anyways

[12:18:10] <ronny> dstufft: i'll poke you after my workday then

[12:18:16] <dstufft> okay

[13:58:26] <nanonyme> dstufft, you're a pro at how pip is packaged under Debian, right?

[13:59:02] <nanonyme> Just noticed: /usr/local/lib/python2.7/dist-packages/pip/_vendor/requests/packages/urllib3/contrib/pyopenssl.py -> module' object has no attribute 'PROTOCOL_SSLv3: who tries to use SSLv3 and why?

[13:59:36] <nanonyme> This is for the just-to-be-released Debian 8

[13:59:45] <dstufft> nanonyme: You're on Debian unstable yea? or did they migrate that patch to Jessie?

[13:59:54] <nanonyme> They did

[14:00:15] <nanonyme> I'm on Jessie, yes

[14:01:16] <nanonyme> dstufft, now's still maybe good time to fix this, otherwise pip will be completely broken on next Debian

[14:02:23] <dstufft> nanonyme: what version of pip?

[14:02:54] <nanonyme> Hah, fun, pip --version will also do that import and break

[14:03:14] <nanonyme> You should probably try to restructure code so printing version would be possible even if rest of pip is broken

[14:03:16] <dstufft> you can open up /usr/local/lib/python2.7/dist-packages/pip/__init__.py and it'll be near the top

[14:03:34] <nanonyme> dpkg -l |grep pip says 1.5.6-5

[14:03:57] <dstufft> you're not using dpkg, you're installed into /usr/local

[14:04:12] <nanonyme> Hmm

[14:04:39] <nanonyme> Ooh, good point. Wonder how come that happened

[14:05:19] <nanonyme> Lemme delete entire /usr/local/ and see what it does

[14:05:48] <nanonyme> Looks like it's not breaking anymore. Thanks, was already scared

[14:06:55] <nanonyme> /usr/local had 1.5.6 ie *way* older than what my system one was

[14:08:16] <nanonyme> Ah, problem resurfaced due to me installing requests into a virtualenv during the build script

[14:08:58] <nanonyme> Apparently I've version-bound to 2.4.1 so might be my fault

[14:11:16] <ronny> requests makes me sad every time i deal with it ^^

[14:11:50] <dstufft> nanonyme: Yea, I think pip from PyPI is broken prior to pip 6 on Debian Jessie

[14:12:02] <nanonyme> Right

[14:12:31] <dstufft> because urllib3 imports PROTOCOL_SSLv3 because it's trying to map standard library protocols to pyopenssl protocols

[14:12:45] <dstufft> so it just imports all of them and makes a dictionary mapping them

[14:13:08] <nanonyme> Okay, updating requests in requirements.txt fixed the rest of the problems

[14:13:49] <nanonyme> ronny, what's your grudge with it?

[14:14:10] <ronny> nanonyme: utterly broken internal layering, bad http parsing

[14:14:22] <dstufft> http parsing comes from httplib

[14:14:39] <ronny> dstufft: that one is a total mess ^^

[14:14:56] <dstufft> yea it is, there's ongoing work via hyper to make something that can replace httplib

[14:14:57] <ronny> imho the only good http lib for python atm is restkit

[14:15:07] <ronny> it already does it all

[14:15:08] <nanonyme> Does it also handle encodings sanely?

[14:16:33] <ronny> restkit already has propper socket pooling and http parser layering since years

[14:16:47] <nanonyme> I thought urllib3 has just fine pooling

[14:16:50] <dstufft> heh, it uses http-parser, I looked at that and hated it and started writing my own :V

[14:16:56] <dstufft> urllib3 does connection pooling yes

[14:17:01] <ronny> nanonyme: i cant replace sockets with something else there

[14:17:19] <nanonyme> Eh, what would you want to replace sockets with in HTTP?

[14:17:36] <ronny> i once needed something that replaced sockets with something that communicateds with a ssh'd python process so i could simulate port forwarding via ssh where it was forbidden

[14:18:28] <ronny> with restkit that was possible trivially, with the httplib/urllib3 mess its impossible to do due to rorally broken layering

[14:18:46] <ronny> *totally

[14:20:53] <ronny> dstufft: only the outer api is nice

[14:21:11] <dstufft> the adapter thing in requests is nice too

[14:21:25] <ronny> that came so much later

[14:21:28] <dstufft> we use that in pip to make file:// urls work

[14:21:43] <dstufft> and cachecontrol uses it to provide transparent caching

[14:23:37] <ronny> dstufft: still has a rotten core, kenneth deliberately ignored libs with much better layering when he made requests - it will take ages to propperly fix the mess, and it will likely never get propper layering

[14:24:55] <dstufft> the layering in requests doesn't bother me at all *shrug*. The layering in httplib is pretty bad, (and urllib3 then inheriets that)

[14:25:30] <tos9> how about the complete lack of tests

[14:25:31] <ronny> dstufft: well, it hardly matters, until you need to cahgne something, and realize its impossible because of the bad layering

[14:25:38] <dstufft> it wouldn't be very dificult to make requests use http-parser and things similarly to what rest-kit does

[14:25:38] <tos9> and the way that that really really correlates with requests often being broken :/

[14:26:13] <ronny> dstufft: requests feels like the setuptools of http libs ^^

[14:27:13] <dstufft> I'm more concerned with the fact it doesn't appear that restkit validates TLS by default

[14:27:25] <dstufft> at least I don't see where it'd be doing that in the code

[14:27:29] <nanonyme> ronny, IOW the thing that everyone uses but which has a lot of quirks?

[14:29:44] <dstufft> also it looks like restkit leaks authentication data when a redirect crosses an authentication boundary

[14:31:26] <ronny> dstufft: still i'd rather fix missing logic bugs in a well layered base than frag on a rotten base

[14:31:39] <doismellburning> this is why I avoid Python

[14:32:08] <ronny> but im draging on something pointless

[14:34:34] <dstufft> https://github.com/benoitc/restkit/issues/140

[14:38:13] <ronny> dstufft: thx

[14:38:23] <nanonyme> Hmm, I guess the pip in Debian 8 is barely new enough to support wheels by default old so old that it will reinstall already installed packages with --upgrade

[14:38:35] <nanonyme> I guess that's acceptable

[14:39:11] <dstufft> ronny: I'm not going to open an issue for the other issue because I don't have a setup to demonstrate the issue or the time to fully investigate it fwiw

[14:39:34] <ronny> dstufft: can you shoot me an email about it

[14:40:30] <nanonyme> doismellburning, you've picked a funny channel to lurk in if you avoid Python ;)

[14:40:52] <dstufft> ronny: https://github.com/kennethreitz/requests/issues/1885 talks about it, stripping auth headers on a redirect to ensure you don't leak auth info to a different host

[14:41:12] <doismellburning> nanonyme: :P

[14:48:23] <ronny> dstufft: will validate after work, thanks

[14:52:13] <nanonyme> dstufft, do you think it makes sense that you keep upgrading setuptools and pip as --user on machines CI machines for the CI user?

[14:52:35] <nanonyme> Ie that you don't touch system stuff but keep packaging tools up-to-date for the user

[14:53:08] <nanonyme> I already have latest virtualenv periodically be installed as --user for the CI user so you can just build a new virtualenv in a build job and install whatever you want there

[14:53:30] <nanonyme> Well, under the assumption that headers exist :)

[14:54:13] <dstufft> nanonyme: probably that'd be fine

[15:03:31] <nanonyme> Oagh

[15:03:52] <nanonyme> Okay, this isn't fun at all

[15:04:57] <nanonyme> I ended up in a version conflict between pip and requests, apparently

[15:05:14] <nanonyme> By --user installing requests and using system pip on Debian 8

[15:07:35] <nanonyme> dstufft, would that happen to be a result of the unvendoring?

[15:08:14] <dstufft> nanonyme: yea, there's a deibna bug for it that fixes it. Not sure if that's commited to Jessie or not

[15:08:59] <nanonyme> I workaround by wiping ~/.local before updating slave requirements

[15:10:11] <nanonyme> Hmm, maybe I should actually just have a separate bootstrap that first updated pip....

[15:10:17] <nanonyme> Yeah, that would work

[15:11:37] <nanonyme> Can pip deal with multiple requirements files and will it do them in one transaction or multiple?

[15:11:53] <nanonyme> Ie will I need to have this bootstrap.txt as a separate pip install command?

[15:12:29] <dstufft> um

[15:12:35] <dstufft> I think it handles multiple

[15:12:47] <nanonyme> Okay, I'll just have multiple then

[15:12:57] <nanonyme> It's not like we can update Debian pip at this point

[15:13:21] <dstufft> if I ever get around to it I plan on publishing packages for python-pip that tracks upstream on Debuntu systems

[15:14:26] <nanonyme> Sealed environment, we'd have to mirror your repo

[15:17:05] <nanonyme> Right, now it works correctly and I don't have to wipe .local

[15:18:01] <nanonyme> Wait, I guess I wouldn't anyhow...

[15:19:40] <nanonyme> Ie as long as I maintain that pip and requests are updated in same transaction, the system will not break

[16:14:38] <ionelmc> what's the filesize limit for pypi?

[16:15:12] <dstufft> uh

[16:15:24] <ionelmc> more than 100mb right?

[16:15:44] <dstufft> Don't think it's more than 100MB, I think it might be exactly 100mb

[16:15:50] <dstufft> or amybe 40MB

[16:15:54] <dstufft> I forget tbh

[16:16:05] <ionelmc> dstufft: i wanna put a fixed up wheel variant of appengine sdk there

[16:16:23] <dstufft> it was 40MB for awhile, we talked about bumping it, and I think we bumped it to 100MB but mayb eI jsut think we did

[16:16:38] <ionelmc> it annoyed me for too long now

[16:16:55] <ionelmc> i don't get why they can't make the sdk available on pypi

[16:35:20] <ronny> dstufft: ping

[16:41:19] <ronny> hmm

[16:41:33] <ronny> dont tell me hes at lunch, so much for good timing ^^

[16:43:01] <ionelmc> can i make pip avoid downloading anything (i want it only to look into the --find-links place)

[16:43:04] <ionelmc> ?

[16:43:26] <xafer> maybe --no-index

[16:44:10] <ronny> ionelmc: i use --no-index, but you cant prevent setup_requires without preinstalling them

[16:44:42] <ionelmc> ronny: i have the setup.py under my control, luckily enough :)

[16:45:03] <ronny> ionelmc: thn --no-index + --find-links should suffice

[16:45:09] <ionelmc> thanks

[16:45:22] <xafer> and it wont prevent downloading things coming from find-links and dependency_links obviously

[16:46:00] <ronny> it helps to build wheels of everything

[16:46:11] <ronny> dstufft: poke-ping ^^

[16:48:08] <ionelmc> damn

[16:48:20] <ionelmc> a wheel builder without any setuptools would be good right about now

[16:48:21] <ronny> ionelmc: what broke?

[16:48:48] <ronny> oh, i see

[16:49:32] <ronny> my first prototype wont be ready until after talking to dstufft and hacking up a initial version

[17:02:40] <ronny> ionelmc: first i need a name for th tool tho, its killing me

[17:05:39] <ionelmc> ronny: ez_build

[17:05:48] <ionelmc> :-P

[17:05:48] <ronny> GAH

[17:05:54] <ionelmc> just kidding

[17:16:17] <ionelmc> ronny: simple_pkg ?

[17:16:41] <ionelmc> it's hard to make up a catchy name

[17:17:13] <tdsmith> wheelwright?

[17:17:46] <ronny> i started a process that involves 2 female majors of social sciences doing a metaanalysis of a wrong topic to yield a name =)

[17:23:03] <ionelmc> wheelmaker

[17:23:08] <ionelmc> doooo iiiiiiiiiiitttt

[18:50:08] <ronny> dstufft: ping again?

[20:17:41] <ronny> hmm

[21:29:12] <ionelmc> is this valid version 1.2.3-4 ?

[21:31:15] <ionelmc> or should i use 1.2.3.post4 ?

[21:39:58] <dstufft> ionelmc: they are equivilant

[21:40:04] <dstufft> use whichever you like better

[21:40:09] <dstufft> 1.2.3-4 will be normalized to 1.2.3.post4

[21:40:11] <ionelmc> dstufft: how about 1.2.3+4?

[21:40:18] <dstufft> +4 is different, that's a local version

[21:40:46] <dstufft> 1.2.3+4 < 1.2.3.post1

[21:40:55] <ionelmc> ok so i i say pip install foo==1.2.3 it will match 1.2.3+4 but not 1.2.3.post4 ?

[21:41:04] <dstufft> correct

[21:41:16] <ionelmc> going with the plus then

[21:41:32] <ionelmc> dstufft: iow, "+N" is always the last component yes?

[21:41:44] <dstufft> yes

[21:42:08] <ionelmc> dstufft: 1.2.3 < 1.2.3+0 ?

[21:42:51] <ionelmc> dstufft: does pypi let me upload two dists that normalize to same version?

[21:43:34] <dstufft> ionelmc: yes to the first question, um, I think yes to the second one right now but that's going to be prevented at some point

[21:44:12] <ionelmc> awesome

[21:46:00] <ionelmc> dstufft: how bad is to only upload wheels nowdays?

[21:47:19] <dstufft> I don't suggest it, setuptools can't install it, linux distros won't like you. older pips can't install it

[21:47:30] <ionelmc> dstufft: if i stick a pth file in the root of a wheel will i hit a wall anywhere?

[21:47:40] <dstufft> unsure offhand

[21:48:07] <ionelmc> i have to make a custom build command for it

[21:48:23] <ionelmc> seems to work, and puts the pth file in site-packages

[22:17:09] <wsanchez> Is this a good place to ask about virtualenv?

[22:18:20] <dstufft> yea

[22:20:01] <wsanchez> So, I'm using --system-site-packages when I create mine, because on OS X I'd rather use the pyobjc built into the OS than build my own. But That's really the only reason. I don't suppose there's a way to say that for only one package?

[22:20:44] <dstufft> wsanchez: not really, unless you use symlinks

[22:20:45] <wsanchez> The problem I'm running into is that I want it to install (a newer) Twisted into the VE and not use the system one.

[22:20:51] <wsanchez> hrm

[22:21:16] <dstufft> Is there a particular reason you want to use the pyobjc built into the system? Does it have something the one on pypi doesn't have?

[22:22:11] <wsanchez> Well… mostly because it's a giant thing and because I can blame the OS folks if it doesn't work. :-)

[22:22:45] <dstufft> ha

[22:22:48] <dstufft> that's a decent reason :)

[22:22:51] <wsanchez> I guess it's only a thing because I'm doing this *at Apple*.

[22:23:09] <ionelmc> wsanchez: well you can force install twisted in the virtualenv no?

[22:23:43] <wsanchez> ionelmc: I can as long as the system Twisted doesn't match the version I want, I think

[22:23:46] <dstufft> pip install --ignore-installed Twisted

[22:23:50] <dstufft> should do it

[22:23:53] <dstufft> assuming you're pip 6+

[22:23:59] <wsanchez> Hrm.

[22:24:08] <wsanchez> Yes this is why I love you guys

[22:24:10] <dstufft> that existed on older pip's, but it was broken

[22:24:11] <ionelmc> making 50mb package sure ain't fun

[22:24:15] <ionelmc> everything is slow :(

[22:25:13] <dstufft> wsanchez: Do you know anyone who can fix Python on OSX to not shove the apple pre-installed packages before site-packages? :/ It breaks things for anyone who wants to upgrade their system packages (like setuptools :(()

[22:25:42] <dstufft> also hey

[22:25:46] <dstufft> they upgrades setuptools

[22:25:49] <dstufft> upgraded*

[22:25:52] <dstufft> I didn't notice that

[22:25:58] <wsanchez> Ed (who used to own python) retired, so I'm not sure who owns it now, but I can file a bug

[22:26:01] <dstufft> course they upgraded it from 0.6 to 1.1

[22:26:22] <wsanchez> I tried to get them to add pip but they weren't excited.

[22:26:29] <dstufft> wsanchez: would a pip bug where we talk about trying to work around it be useful?

[22:26:34] <dstufft> we have one of those

[22:26:37] <wsanchez> dstufft: Yes

[22:26:41] <dstufft> sec let me get a link

[22:28:54] <wsanchez> dstufft: I'm guess you mean the "Extras" stuff

[22:29:20] <dstufft> wsanchez: https://github.com/pypa/pip/issues/1480 (That's was fromw hen setuptools was still 0.6.wahtever, glyph installed a newer setuptools into --user but the OSX packages are put before everything but the stdlib) thee's also https://github.com/pypa/pip/issues/2468 and

[22:29:23] <dstufft> wsanchez: yea

[22:30:35] <dstufft> I think this issue is the same thing https://github.com/pypa/pip/issues/2156

[22:31:57] <dstufft> wsanchez: in unmodified Python, --user comes before site-packages, in Apple Python Extras comes before them both. The closer match upstream sticking Extras after site-packages would be reasonable, but if they want it first to protect against people breaking things, at least putting --user before Extras would allow us to tell people to use --user to not modify their system

[22:32:27] <dstufft> (pip is going to default to --user soon too, so there will be some rails to get people towads doing that instead of sudo)

[22:32:35] <wsanchez> <rdar://problem/20144107> Update Python setuptools package from 1.1.6 to 14.0 or later

[22:32:43] <wsanchez> (just filed)

[22:34:40] <dstufft> wsanchez: also, I have no idea what inside apple is like, but if pip/setuptools/virtualenv/any of the packaging tools cause Apple some pain, If they're willing to tell me about the pain we might be able to fix things for them. I don't know if that's something apple would never do or whatever

[22:34:59] <dstufft> (inclduing if we do things that make people break their computers)

[22:35:23] <wsanchez> So I'm clear, the default is user -> site-packages -> stdlib and Apple's is Extras -> user -> site-packages -> stdlib, right?

[22:36:49] <dstufft> The default is stdlib > user > site-packages, and apple is stdlib > Extras > user > site-packages (sort of, There is acutally multiple stdlib directories, and apple stick Extras in the middle of them so some parts of stdlib are > Extras, and some part of stdlib is < Extras

[22:38:13] <wsanchez> ew

[22:38:26] <wsanchez> I'll pretend I don't know that

[22:40:04] <dstufft> wsanchez: https://gist.github.com/dstufft/6066bc86eae344a230eb might be helpful, that's whjat the sys.path is on my system (I don't use apple python, so it's more or less unmodified)

[22:40:29] <wsanchez> Thanks. I think I tweaked mine on accident a while ago.

[22:40:53] <dstufft> dunno what they did that stuck PyObjc in between --user and site-packages, but if they move Extras to right there, that would make --user work and we can just tell people that apple doesn't support upgrading things shipped with the OS without --user

[22:42:54] <dstufft> wsanchez: updated the gist to include my unmodified Python that I install into my homedir too

[22:42:58] <dstufft> to show the difference

[22:46:14] <ionelmc> dstufft: ugh ... cannot use PEP 440 local versions on PyPI

[22:46:41] <ionelmc> dstufft: ok, so 1.2.3-4 is the same as 1.2.3.post4 right?

[22:46:45] <dstufft> ionelmc: yea

[22:51:04] <ionelmc> dstufft: how come you didn;t went with some of the conventions debian has?

[22:51:22] <dstufft> which conventions

[22:51:46] <ionelmc> dunno, random question :)

[22:52:05] <ionelmc> dstufft: why doesn't pypi allow local version?

[22:52:38] <ionelmc> i don't see anything about it in pep440 - did i miss it?

[22:52:50] <dstufft> the answer about convetntions is basically "it would break compat with what setuptools did and what a lot of people are doing"

[22:54:05] <dstufft> PEP 440 was mostly "restrict the syntax setuptools already was using, and add small tweaks where it made sense"

[22:54:52] <dstufft> ionelmc: from PEP 440: "As the Python Package Index is intended solely for indexing and hosting upstream projects, it MUST NOT allow the use of local version identifiers. "

[22:55:02] <ionelmc> doh :)

[22:55:29] <ionelmc> dstufft: cause i'm repackaging something to put on pypi

[22:55:36] <ionelmc> so my upstream ain't pypi

[22:55:52] <ionelmc> it's a shithole that don't do proper packaging

[22:55:57] <ionelmc> :)

[22:56:27] <dstufft> you're the upstream as far as PyPI is concerned :) We optimize for that use case, and since you can't have multiple layers of local versions, if you used a local version, othe rpeople couldn't make a local version of your local versions

[22:56:33] <wsanchez> <rdar://problem/20144204> Python "Extras" should not come in sys.path before user packages and site-packages

[22:57:02] <ionelmc> dstufft: then i think postN is ok, upstream won't have those

[23:31:21] <tdsmith> oh, the Extras thing is a problem for the scipy stack too, fixing that would be great

[23:37:53] <dstufft> wsanchez: I've never used rdar, is there any way for me to subscribe to that? or is it all internal? (I tried clicking the link but I guess there's no program installed that understands them)

Log file Viewer

Help | Karma | Search:

#pypa logs for Thursday the 12th of March, 2015