pmxbot IRC Log Viewer

[13:37:34] <vincentll> !logs

[13:37:34] <pmxbot> http://chat-logs.dcpython.org/channel/pypa

[13:58:31] <prometheanfire> dstufft: thanks :D

[13:58:51] <prometheanfire> dstufft: seems to work, I'm at the openstack conf but will pass it off to floppym

[13:58:55] <prometheanfire> floppym: 13:46 < dstufft > Oh, for anyone who is in here and who wasn't following the issue, try curl -L -I https://pypi.io/packages/source/p/packaging/packaging-16.7.tar.gz /cc prometheanfire gchristensen tdsmith

[13:59:09] <prometheanfire> floppym: seems to work so we might want to add that as an option

[13:59:16] <gchristensen> prometheanfire: here in austin?

[13:59:30] <dstufft> As a reminder: "once Warehouse is ready to go live pypi.io is going to be the new PyPI URL and pypi.python.org will redirect to it, however pypi.io is currently pre-production grade so we don't have any alerting on if it goes down and if it's broken I won't stay up all night to fix it etc"

[13:59:55] <dstufft> if someone feels like tangling with the eldritch horror that is legacy pypi's code base I'm happy to review a PR... but I really don't feel motivated to do that

[14:00:31] <gchristensen> dstufft: hey, thank you for all your work on pypi and resolving the bugs about re-uploads.

[14:01:34] <dstufft> gchristensen: :]

[14:01:46] <dstufft> Warehouse should be a lot better about this

[14:01:52] <prometheanfire> gchristensen: ya

[14:02:05] <prometheanfire> gchristensen: I'm sitting waiting for the keynotes to start

[14:03:26] <prometheanfire> mgedmin: given that you should be verifying hashes and the like :P

[14:12:41] <floppym> dstufft: I'm guessing that I probably shouldn't point our ebuilds at https://pypi.io/ ? Will that eventually become pypi.python.org?

[14:13:03] <dstufft> [09:51:53] <dstufft> As a reminder: "once Warehouse is ready to go live pypi.io is going to be the new PyPI URL and pypi.python.org will redirect to it, however pypi.io is currently pre-production grade so we don't have any alerting on if it goes down and if it's broken I won't stay up all night to fix it etc"

[14:13:29] <floppym> dstufft: Oh, sorry. Thank you for the work!

[14:19:26] <dstufft> floppym: no problem

[14:20:28] <dstufft> floppym: so you're free to use pypi.io now, just if you do, do it with the knowledge that It may go up and down (although generally it's pretty stable)

[14:21:44] <floppym> Yeah, I'll think on it. We (Gentoo) have our own mirroring system that will mostly hide any outages.

[14:35:43] <FRidh> floppym: having sha256 hashes is already a nice reason to just go straight for pypi.io ;-)

[14:36:19] <gchristensen> FRidh++

[14:57:38] <prometheanfire> floppym: I think we should add it as an option, so we'll check both

[14:58:33] <floppym> prometheanfire: That's a good idea!

[14:59:26] <prometheanfire> thanks

[14:59:33] <J1m> Did the pypi download link format change?

[14:59:34] <prometheanfire> floppym: I'm semi-afk this week, so...

[14:59:45] <prometheanfire> J1m: gonna go with a yes :P

[14:59:58] <prometheanfire> pypi.io has the old format

[15:00:06] <dstufft> J1m: https://bitbucket.org/pypa/pypi/issues/438/backwards-compatible-un-hashed-package

[15:00:58] <dstufft> I'm sure this is a preview of what it's going to be like when I cut over warehouse to be the main pypi deployment where I find out all the ways people had relied in some weird thing in PyPI.

[15:01:55] <J1m> Well, this pretty well hoses buildout's bootstrap.py.

[15:02:24] <J1m> Not sure why this is considered "weird" behavior.

[15:02:41] <prometheanfire> dstufft: so, just to be clear, the old format will be at pypi.io, the new format will be where?

[15:02:54] <dstufft> the new format is everywhere

[15:03:03] <dstufft> the old format on pypi.io just redirects to the new url

[15:03:09] <prometheanfire> ah, just the old format is ALSO at pypi.io

[15:03:14] <prometheanfire> ah, right, the 302

[15:03:26] <prometheanfire> gotcha

[15:03:52] <dstufft> J1m: Sorry, I didn't specifically mean this thing was weird, just that one of the things I've been afraid of whens witching is that lots of weird things are going to come out :]

[15:04:27] <dstufft> https://bootstrap.pypa.io/bootstrap-buildout.py doesn't seem to reference /packages/ at all

[15:05:06] <dstufft> it appears to just use setuptools, which should work fine

[15:07:07] <J1m> BTW, wget on my mac is telling me that bootstrap.pypa.io's certificate has expired.

[15:08:27] <dstufft> hmm

[15:08:30] <dstufft> that doesn't make any sense

[15:08:46] <dstufft> https://s.caremad.io/pD0apbVIJp/

[15:08:47] <J1m> I'm not getting this on a linux box.

[15:08:52] <J1m> whimper

[15:12:10] <dstufft> J1m: are you sure it's an expired cert warning?

[15:13:53] <J1m> ERROR: cannot verify bootstrap.pypa.io's certificate, issued by 'CN=GlobalSign CloudSSL CA - SHA256 - G3,O=GlobalSign nv-sa,C=BE':

[15:13:53] <J1m> Issued certificate has expired.

[15:14:07] <dstufft> huh

[15:14:09] <J1m> Perhaps my wget is out of date. Upgrading...

[15:14:09] <dstufft> Interesting

[15:15:41] <J1m> Yup. Upgrading wget fixed it. Sorry.

[15:16:06] <dstufft> \o/

[15:27:03] <J1m> so, buildout uses ez_setup.

[15:27:13] <J1m> ez_setup sniffs

[15:27:30] <J1m> on my mac, it decides to use curl

[15:27:52] <J1m> curl doesn't follow the 301, at least not be default.

[15:28:24] <J1m> s/be/by/

[15:28:34] <J1m> so fragile :(

[15:28:55] <dstufft> ah yea

[15:28:58] <dstufft> ez_setup.py

[15:29:07] <dstufft> this is why get-pip.py doesn't do that :|

[15:31:18] <J1m> this is why we need to give up on bootstrapping.

[15:32:56] <dstufft> J1m: ensurepip :D

[15:33:28] <dstufft> the whole goal behind that was to make it so everyone has a ``pip`` command they can us eto install wahtever it is they want, even another installer :]

[15:36:21] <dstufft> ngoldbaum: they'll either patch their local copies of stuff to use not-openssl, or they'll keep it around in a private palce. For stuff not distributed by Apple it'll just be static linking or what not

[15:48:35] <J1m> python3 -m ensurepip

[15:48:35] <J1m> /usr/bin/python3: No module named ensurepip

[15:49:01] <J1m> ububtu 14.04

[15:49:08] <J1m> ubuntu

[15:50:15] <Wooble> J1m: yes, ubuntu is evil.

[15:51:40] <Wooble> (there should be a python3-pip package, though, IIRC)

[15:52:09] <ionelmc> J1m: on ubuntu you'd use get-pip.py, not ensurepip

[15:52:49] <J1m> get-pip.py

[15:52:49] <J1m> bash: get-pip.py: command not found

[15:53:00] <ionelmc> ofcourse there are the python-pip packages but those old

[15:53:07] <ionelmc> J1m: you need to download it

[15:53:28] <ionelmc> https://bootstrap.pypa.io/

[15:54:16] <ionelmc> you should uninstall any python-pip or python3-pip apt packages if you go down that path tho ...

[15:55:27] <J1m> thanks.

[16:06:10] <J1m> ftr, get-pip.py is the new ezsetup.py. Just sayin.

[16:09:37] <dstufft> someday get-pip.py will die, once ensurepip is in every version of Python we support :]

[16:10:34] <J1m> That will never happen because system packagers.

[16:12:00] <dstufft> eh

[16:12:13] <dstufft> ensurepip is fixed in versions of Ubuntu > 14.04

[16:13:25] <ThiefMaster> hey, it looks like ez_setup.py fails with the latest setuptools version: https://gist.github.com/ThiefMaster/6c4251948f093d347f41615409c25e3e

[16:14:28] <J1m> Yes it does

[16:15:57] <ThiefMaster> any ETA? this is breaking some build script we have (and if it'll be fixed e.g. tomorrow i won't spend time changing an old script that'll be replaced soon-ish anyway)

[16:16:00] <dstufft> it's true, a 404 is not a valid zip file

[17:30:08] <floppym> Heh.

[17:38:47] <oberstet> Is it possible to set env vars in universal wheels? Like when having a dependency on PyNaCl, and I'd like to set SODIUM_INSTALL=bundled ?

[18:17:17] <dstufft> oberstet: Um, I'm not sure what you mean. You have a thing foo that depends on PyNaCl, and you want to set an envvar when PyNaCl gets installed?

[18:20:22] <oberstet> dstufft: exactly

[18:20:28] <dstufft> oberstet: No, you can't do that

[18:20:34] <oberstet> =(

[18:20:52] <dstufft> not even related to wheels, installs are isolated from each other generally

[18:22:09] <oberstet> mmh. we just moved Crossbar.io (the "foo" thing) to universal wheels (as itself, it is pure Python) .. because wheels are nice, and also because of pinning deps (and hashin)

[18:23:01] <oberstet> that means, users will now have to do: SODIUM_INSTALL=bundled LMDB_FORCE_CFFI=1 PYUBJSON_NO_EXTENSION=1 pip install crossbar

[18:23:19] <oberstet> and that isn't as nice as "pip install crossbar" obviously

[18:24:10] <oberstet> we managed to get away of all conditional deps (as these don't work also when combined with hashin)

[18:24:49] <oberstet> now we have https://github.com/crossbario/crossbar/blob/master/requirements.txt - but now the env vars .. dunno how to do it

[18:26:33] <oberstet> effectively, that means eg: nothing that depends on PyNaCl, and is packaged as a wheel, can use the bundled NaCl for PyNaCl

[18:27:17] <oberstet> dstufft: is there anything we can do?

[18:28:59] <dstufft> doesn't PyNaCl default to bundled? (not that it helps the other side of those things)

[18:30:00] <oberstet> no, it defaults to searching for a system NaCl, and when it fails to find one, it bails out (that is, it doesn't automatically fall back to bundled)

[18:30:21] <dstufft> oberstet: did this work before? I don't think that crossbar's setup.py can force an environment variable on another install..

[18:31:04] <oberstet> it did work when we only published source dists, since our setup.py contains this: https://github.com/crossbario/crossbar/blob/master/setup.py#L77

[18:31:19] <dstufft> oberstet: I don't think that's right either. Looking at https://github.com/pyca/pynacl/blob/master/setup.py i'm pretty sure it uses the bundled by default unless you specify SODIUM_INSTALL or the system is good enough

[18:31:43] <oberstet> well, I just tested it on a system without NaCl. it bails out ..

[18:31:59] <dstufft> oberstet: what's the error?

[18:32:03] <oberstet> at least the version of PyNaCl currently on PyPI

[18:32:09] <oberstet> wait

[18:36:46] <oberstet> dstufft: https://gist.github.com/oberstet/e06f7ba495c516fdd0e377bb1fba1897

[18:37:47] <dstufft> hmmm

[18:39:20] <oberstet> that system does not have nacl .. and we'd like to use the bundled nacl (for Crossbar.io) _always_ regardless whether there is a system nacl or not (for reasons)

[18:39:49] <dstufft> oberstet: what does https://bpaste.net/show/4ad71e4807e5 print?

[18:41:03] <oberstet> dstufft: https://gist.github.com/oberstet/34a21e435f69498127a46fee97a32e91

[18:41:20] <dstufft> oberstet: so you have sodium on that system then

[18:41:40] <dstufft> dlopen is able to open sodium.so

[18:41:58] <oberstet> I did that in the virtualenv where I just installed pynacl

[18:42:15] <dstufft> yea, pynacl doesn't install a sodium.so

[18:42:19] <dstufft> it static links it

[18:43:25] <dstufft> oberstet: fundamentally here though, crossbar isn't allowed to control how it's dependencies get installed (the fact it could set ane nvironment variable wasn't intentional, it was overlooked)

[18:43:50] <oberstet> mmh

[18:45:01] <dstufft> oberstet: speaking as someone who works on PyNaCl (but not much lately), I personally wouldn't be super opposed to the idea that we should just always use the bundled copy unless explicitly instructed otherwise

[18:46:17] <oberstet> I am still trying to verify that the system has a sodium.so somewhere ..

[18:46:52] <oberstet> rgd that specific case of pynacl, I think there are good arguments (security, repeatability) to always use bundled

[18:47:30] <oberstet> do you know whether cffi can tell me the absolute path to the *.so it loaded?

[18:47:35] <dstufft> hmm

[18:47:37] <oberstet> dummy q maybe ..

[18:48:52] <oberstet> also: if cffi can find one, the why does pip install pynacl fail?

[18:50:14] <dstufft> presumably you're missing the dev headers

[18:50:24] <dstufft> you have the .so but not the headers

[18:51:16] <oberstet> alright. the system does have a sodium.so. sorry: https://gist.github.com/oberstet/da485d08a2b9c32905939e2a20d0e144

[18:51:26] <oberstet> I presume this is the one cffi finds

[18:52:55] <oberstet> right. its a debian/ubuntu, and it does indeed have libsodium (not sure where from), but _not_ the dev package

[18:53:32] <oberstet> ok. this is that. pynacl looks for headers, and bails out, not falls back to bundled

[18:54:13] <dstufft> yea, so PyNaCl detects it has an appropiate versioned sodium.so and attempts to compile, but you're missing the dev headers so it bombs out. Arguably either we should also test that the headers exist or we should just use the bundled unless otherwise instructed

[18:54:34] <oberstet> yep. that for sure.

[18:54:44] <oberstet> but I'd also welcome inverting the option

[18:55:11] <oberstet> SODIUM_INSTALL=system to force system, otherwise _always_ use bundled.

[18:55:28] <oberstet> why rely on outdated distro packages for security sensitive stuff?

[18:56:21] <oberstet> it would solve one of our three problems;)

[18:57:36] <oberstet> eg take PyUBJSON: it is slower on PyPy using extension; it increases the attack surface (cython based extension)

[18:59:53] <oberstet> rgd pynacl .. here is another argument (if I get this right) - on windows/osx, it'll install from the binary wheels. which bundle nacl/sodium. not so on Linux. doesn't that make the latter "2nd citizen"?

[19:00:18] <oberstet> given the lack of linux binary wheels ..

[19:01:34] <J1m> I'd like to provide a (at least partial) fix for ez_setup.py.

[19:01:42] <J1m> where would I do that?

[19:01:54] <dstufft> J1m: probably the setuptools git repo on github

[19:02:11] <J1m> https://github.com/pypa/setuptools ?

[19:02:43] <dstufft> yea

[19:02:53] <ionelmc> oberstet: UBJSON woot ... why does that exist when we have msgpack and cbor?

[19:02:55] <dstufft> oberstet: well we have binary wheels on Linux now too ;) but yea, I'd personally be down for that

[19:03:25] <dstufft> oberstet: for PyUBJSON, it sounds like something that could be fixed with a PR to PyUBJSON to disable compilation of their extension on PyPy

[19:04:50] <oberstet> dstufft: ok. I'll file an issue for pynacl, and see how far that gets us (while talking to other deps). thanks for your help!

[19:05:16] <oberstet> dstufft: whaat? did I just read "we have binary wheels on Linux" ?

[19:05:21] <oberstet> since when?

[19:05:33] <oberstet> I thought it would still be in the cooking?

[19:10:51] <dstufft> oberstet: it's called "manylinux1"

[19:10:58] <dstufft> it's not every linux, but many of them

[19:11:10] <dstufft> pip 8.1 has support for it, as does PyPI

[19:11:28] <dstufft> https://github.com/pypa/manylinux

[19:11:56] <dstufft> https://pypi.io/project/gevent/#files for instance has them

[19:32:22] <J1m> https://github.com/pypa/setuptools/pull/559

[19:32:33] <J1m> can someone review this please?

[19:32:47] <J1m> I think this will fix much breakage.

[20:48:52] <dowwie> agronholm: dstufft may have a viable use case for an asphalt service to manage warehouse uploads

[20:49:12] <agronholm> dowwie: are you referring to the distutils-sig post?

[20:49:30] <dowwie> agronholm: not sure what the details are

[20:49:42] <dowwie> agronholm: think you two should connect about it

[20:50:05] <agronholm> asphalt does not have a web server component yet...

[20:52:06] <agronholm> J1m: it looks like your change is removing my ez_setup fix

[20:52:14] <agronholm> J1m: care to elaborate why?

[20:58:13] <J1m> what fix is it removing?

[20:58:47] <J1m> are you talking about sys.meta_path agronholm ?

[20:58:59] <agronholm> J1m: I just posted a comment on github

[20:59:57] <J1m> so your fix was never deployed.

[21:00:05] <agronholm> it was, wasn't it?

[21:00:19] <J1m> so I just started with what was on https://bootstrap.pypa.io/ez_setup.py

[21:00:32] <J1m> which doesn't have it.

[21:00:45] <agronholm> I know my PR was merged

[21:00:45] <J1m> I specifically asked about it in my commit comment.

[21:01:08] <J1m> which begs the question, how the f does this get deployed? :(

[21:01:20] <J1m> But I'll add your code back.

[21:01:31] <J1m> Thanks for anwering my question.

[21:01:45] <agronholm> J1m: hm, what do you mean by that

[21:01:54] <agronholm> how this gets deployed

[21:02:25] <J1m> In my commit message, I asked wtf those lines.

[21:02:44] <J1m> You just answered that they belong there, even though they were never deployed.

[21:03:23] <agronholm> J1m: I'm guessing the version at bootstrap.pypa.io was never updated

[21:03:54] <agronholm> which suggests ez_setup doesn't get updated automatically as part of the release process

[21:03:57] <J1m> Yeah so even if my PR is accepted, it won't do any good unless the thing that's downloaded is updated.

[21:04:11] <agronholm> you have to bug jaraco about that

[21:09:01] <J1m> dstufft, can you tell us how https://bootstrap.pypa.io/ez_setup.py gets updated?

[21:13:23] <dstufft> J1m: a cron job automatically pulls it down from the repository

[21:13:51] <dstufft> (well, techincaly saltstack running in a cronjob)

[21:14:17] <dstufft> it pulls down the "bootstrap" branch

[21:14:28] <J1m> does it pull from master?

[21:14:41] <J1m> cuz what's there and master aren't the same.

[21:14:56] <dstufft> https://github.com/python/psf-salt/blob/master/salt/pypa/bootstrap/init.sls#L42-L51

[21:14:59] <dstufft> bootstrap branch

[21:15:27] <J1m> OK, so we need to get the pr merged and then we have to merge to that branch.

[21:15:50] <J1m> thanks

[21:17:09] <dstufft> I think we run salt every ~15 minutes or so

[21:17:21] <dstufft> so once bootstrap on github is correct, it should be updated within 15 minutes

[21:18:50] <J1m> yup

[21:19:06] <oberstet> ionelmc: because;) seriously, Crossbar.io support any of JSON, MessagePack, CBOR and UBJSON now

[21:19:41] <oberstet> and we wanted it for bringing AutobahnC, which extends WAMP to MCUs, and in particular RIOT-OS based ones - and the latter has a nice UBJSON library built in

[21:19:55] <oberstet> dstufft: manylinux1 - coooool!

[21:22:37] <oberstet> rgd PyUBJSON - thing is, we (as in Crossbar.io) want to have the pure Python version even on CPython (on PyPy for sure) - because, security. No Cython things. I have been burned by ujson -- which is supposed to be faster than stdlib (which is true), but comes with instability and memory leaks

[21:22:38] <ionelmc> i don't get it

[21:23:28] <ionelmc> i guess for middleware it's ok - you wanna be able to integrate with as many as possible

[21:23:33] <oberstet> sorry, this will be mumbo jumbo for you - would need to explain WAMP (Web Application Messaging Protocol) first

[21:23:39] <oberstet> yeah, something like this

[21:23:47] <ionelmc> my question was more about the format

[21:23:48] <dstufft> oberstet: yea, but you're not installing PyUBJSON for _just_ yourself, you're installing it for any other thing that then requests PyUBJSON in your user's environment

[21:24:14] <oberstet> ionelmc: in my opinion/experience, CBOR is the most awesome, then MessagePack

[21:24:29] <ionelmc> since msgpack-v2 and cbor are practically the same thing

[21:25:02] <oberstet> dstufft: Crossbar.io is supposed to be installed in its own, dedicated virtualenv - we now pin all deps (direct and indirect) to point versions, and even hashin

[21:25:14] <oberstet> ionelmc: CBOR has an RFC

[21:25:35] <ionelmc> everyone say that, like i'm going to pip install the rfc :)

[21:25:56] <dstufft> oberstet: I mean, that may be true of how you want crossbar to be installed, but pip doesn't have any way of knowing that :)

[21:26:13] <oberstet> and, on PyPy, it is also slightly faster - "pip install rfc6455" =) haha. I have to remember that .. but "pip install autobahn" sounds .. cooler? ;)

[21:26:30] <oberstet> dstufft: ok, fair enough;)

[21:26:57] <oberstet> fwiw, we recommend our binary packages or docker images over pip - but ..

[21:27:16] <dstufft> oberstet: you may want a simple .sh installer that just calls pip and has the env vars set or something

[21:27:18] <dstufft> or that :)

[21:27:33] <oberstet> both bin packages (deb/rpm/pkg) come with pypy bundled and not only that, but _everything_ down to openssl

[21:28:00] <oberstet> its the only real working way to make sure users have exactly the same bits running

[21:28:31] <oberstet> I will look if we can make the manylinux1 work .. this is awesome news!

[21:30:11] <dstufft> oberstet: if you run into problems, the wheel builders mailing list probably has the people most knowledgable about manylinux1 and building things for it

[21:30:25] <dstufft> https://mail.python.org/mailman/listinfo/wheel-builders

[21:31:09] <oberstet> oh, thanks for that! subscribing ..

[21:32:04] <dstufft> oberstet: crossbar.io always looked cool to me btw :) Never actually got a chance to _use_ it for anything but it looks really interesting

[21:32:48] <oberstet> dstufft: we have only started;)

[21:33:41] <oberstet> dstufft: if you use buildbot, you'll be using Crossbar.io/Autobahn upcoming;)

[21:34:05] <oberstet> if you use Mozilla autopush, you are using Autobahn (which is one that is really exciting)

[21:34:09] <dstufft> nice :D

[21:34:28] <oberstet> if you are using Django Channels (Daphne), you are using Autobahn

[21:35:28] <oberstet> Mozilla is quite large of course (but they use Autobahn at the WebSocket, not WAMP level "only")

Log file Viewer

Help | Karma | Search:

#pypa logs for Monday the 25th of April, 2016