Log file Viewer

[10:15:58] <xafer> hello dstufft, https://github.com/pypa/pip/pull/3590 and https://github.com/pypa/pip/pull/3598 could be included in a 8.1.2 release

[11:42:34] <dstufft> xafer: yea, 8.1.2 is on my TODO list for today

[11:42:40] <dstufft> Also " But, thanks to the magic of ELF scopes (don't ask)," I lol'd /cc njs

[22:09:50] <njs> dstufft: oh gods srsly don't ask

[22:10:07] <njs> dstufft: how do shared libraries work? nobody knows.

[22:10:52] <njs> dstufft: my current newly-discovered-bug-count in how other people's projects handle shared libraries is 5. The affected projects are: ld, objdump, cpython, glibc, patchelf

[22:11:00] <njs> so nbd really!

[22:11:06] <njs> dstufft: btw, will you be at PyCon...?

[22:11:27] <njs> oh haha doh, I just saw you answered someone else asking that an hour ago in the other channel

[22:19:50] <njs> that's too bad, I was hoping to have some good arguments about these PEPs :-)

[22:37:15] <dstufft> njs: yea I wanted to go, just too much stuff up in the air ATM (purchased a house in December, been having some medical shit myself, my car caught on fire) to deal with the logistics of me traveling

[22:41:13] <dstufft> njs: anyways, re: the pynativelibs, I'm probably not going to really get involved in that since it doesn't require anything from pip/PyPI to do and I don't really understand the linking stuff nor have time to figure it out, but my main point of feedback would be to try and figure out a way to make it work without code changes in the consuming libraries (like pyca/cryptography). I think that has the highest chance of not adding to the

[22:41:13] <dstufft> animosity between downstream and upstream-- if it can be done in a reasonable and solid way. If that ends up being extremely error prone then maybe that can't be done

[22:55:16] <njs> dstufft: oh sure, and sorry to hear that

[22:55:33] <njs> dstufft: and yeah, part of the goal is to just make it work without requiring some big pip/PyPI thing :-)

[22:56:08] <njs> dstufft: but not sure what you mean about no code changes to consuming libraries -- like, you can't possibly make this work without at least modifying setup.py and the wheel metadata

[22:56:34] <njs> if by downstream/upstream animosity you mean "it shouldn't piss off debian" then yeah I agree though

[22:57:04] <dstufft> njs: oh, I don't care about modifying setup.py

[22:57:12] <dstufft> or using setup_requires and such

[22:57:34] <dstufft> I mean like, the numpy/__init__.py:import pynative_openblas; pynative_openblas.enable() part

[22:57:36] <njs> yeah, setup_requires is a sore point, because this definitely needs it but it doesn't work blah blah, we better get moving on fixing that :-)

[22:57:43] <njs> nod

[22:58:45] <dstufft> anything we can contain to the build system, is something downstream will have an easier time with, because they almost all have some utility that interfaces with the build system that they can control in one spot that will automatically propagate out to all packages

[22:59:18] <dstufft> anything that requires carrying a patch ends up requiring humans to write that patch (even if it's trivial) and rebasing it whenever things change and such

[22:59:30] <njs> yeah

[23:01:05] <njs> numpy/__init__.py actually went ahead and has a 'import ._distributor_init', where _distributor_init.py is an empty file where you can drop in some build-specific nonsense

[23:02:07] <njs> (this is also necessary if you e.g. want to vendor dlls on windows -- you have to inject some code into the package to tell things to look in _vendored_dlls/)

[23:02:29] <njs> so I was thinking about slightly-more-structured versions of that, but the .pth file idea is interesting too

[23:03:36] <dstufft> I have no idea if it'd work, but the first idea in my head was either a .pth that just unambigiously added all of the pynativelib's and enabled them (assuming it's not slow and/or a bad idea to do so) or using the same .pth to register a sys.meta_path hook that just looks for someone to import the consuming lib (like if it sees someone importing cryptography, it activates pynativelib_openssl) but otherwise is a no-op

[23:04:10] <dstufft> If it works out like it does in my head, it would allow containing all code changes to setup.py/build system

[23:04:13] <njs> the meta_path thing seems tricky because I don't think we want to maintain a registry of every downstream package that uses these libraries

[23:05:03] <njs> and I guess I was imagining a separate .pth file inside each pynativelib wheel, to avoid that point of central coupling (trying to include the same file in multiple wheels sounds liek a recipe for pain)

[23:05:54] <dstufft> if we unambigiously activate them, I see no reason that wouldn't work (assuming the idea in general works)

[23:06:10] <njs> with those refinements though it should probably work fine... I'm a little uncertain about speed (cost of loading a bunch of .pth files, plus the overhead of having a whole bunch of extra paths added to LD_LIBRARY_PATH and friends). but now I am violatign the 0th rule of optimization :-)

[23:06:29] <dstufft> yea the speed thing is a question, would need to be measured :D

[23:07:06] <njs> oh hmm

[23:07:36] <njs> the other thing is providing an API to request the name of the library, for passing to dlopen(...)

[23:08:03] <njs> if we have a mechanism for passing build time configury -> run time , then you can easily re-use that to pass the library name along. with the .pth strategy, not so much :-/

[23:08:24] <dstufft> I think for that, there's no getting around some sort of runtime API, but you're already using a runtime thing since you're using dlopen

[23:09:11] <njs> yeah -- but ideally it should be possible for Debian to set PYNATIVELIB_USE_EXTERNAL=all or something, and get out builds where the runtime API doesn't depend on the pynativelib package being installed

[23:09:48] <dstufft> yea

[23:11:14] <dstufft> I dunno, I'm just spit balling stuff here :) the nice thing about this particular change is since it's divorced from pip/PyPI it's a lot easier to experiment, if one idea fails we can just start up a new idea without having to support it forever

[23:11:56] <njs> yeah

[23:14:05] <dstufft> https://blogs.msdn.microsoft.com/pythonengineering/2016/04/11/unable-to-find-vcvarsall-bat/

[23:15:21] <njs> sorta tempting to go all the way the other direction and tell people to put something like this in their __init__.py: http://paste.pound-python.org/show/VDpXPDX4lJJQPVF0YILj/

[23:16:52] <njs> the boast about windows' compatibility there would be more impressive if it was possible to take a program compiled for python 3.4 and use it on python 3.5 :-)

[23:16:54] <njs> oh well

[23:18:18] <dstufft> njs: heh

[23:18:29] <dstufft> I think that's possible with 3.5+ now

[23:18:37] <dstufft> unless I misunderstood Steve's thing

[23:27:14] <njs> yeah, it should be

[23:27:30] <njs> because in 2015 windows finally caught up with linux on binary compatibility ;-)

[23:28:41] <njs> (binary compatibility is complex and multifaceted blah blah, mostly I'm just grumpy that MS can't figure out how to make it possible for us to build the scientific python stack on their platform)

[23:29:09] <dstufft> njs: to be fair, you need fortran :P

[23:29:25] <njs> yes that is the problem :-)

[23:31:12] <njs> but we said "hi we need $90k to make the mingw-w64 stack Just Work and generate MSVC-compatible binaries, that's less than 1 year of those 5 python developer positions you're trying to hire for" and after much hemming and hawing and nagging they decided that yeah, that was a no-brainer good idea, and yeah, this was exactly the sort of thing taht the "new developer/OSS friendly MS" would do, but, uh, they couldn't figure out how to make the internal politic

[23:31:13] <njs> s work

[23:32:59] <dstufft> njs: lol