pmxbot IRC Log Viewer

[06:51:23] <ronny> buck1: no, just missing a d in front

[06:51:55] <ronny> buck1: extend the Extension class to compile and link go code?

[06:52:27] <ronny> buck1: it wil be tricky, it would be more easy if you package the lib separately and have a c api + pkg-config metadata

[10:09:33] <vmonteco> Hi!

[18:40:25] <buck1> ronny: "package the lib separately" as in .deb ?

[18:41:08] <buck1> ronny: do you know any prior art for extending Extension to use an unkown compiler?

[18:41:14] <buck1> or, is there a reference for such?

[18:41:15] <ronny> buck1: as an example, im unfamiliar with integrating go into python extension

[18:41:24] <ronny> buck1: im not aware of a reference

[18:42:04] <buck1> ronny: to be clear, i'm not Extending python, just calling a small golang program

[18:45:55] <buck1> this seems to be all i get from the doc https://docs.python.org/2/distutils/apiref.html#distutils.core.Extension

[18:56:36] <ronny> buck1: do you want to call via a api, or as a process you start?

[18:56:47] <buck1> it will be a subprocess

[18:58:27] <ronny> buck1: then why not package it separately? its not really python to begin with

[18:58:39] <buck1> ronny: because i want pip install to Just Work

[18:58:54] <buck1> rather than documenting "please install this foo.go however you do on your platform"

[18:59:19] <buck1> if there were any bridge from setuptools to other package managers, that might be a viable route

[19:00:06] <ronny> buck1: wouldnt it break for anyone not having go installed?

[19:00:32] <buck1> sure but at least they'd have a breadcrumb instead of just documentation

[19:00:47] <buck1> "go" not found

[19:01:40] <nanonyme> buck1, there is, kind of

[19:01:50] <buck1> ronny: you can same the same of the vanilla Extension and gcc

[19:02:01] <buck1> plenty of linux installs w/o gcc

[19:02:08] <nanonyme> You can build distro-specific binary packages if you wish based on setup.py

[19:02:27] <ronny> buck1: what does your package do to begin with?

[19:02:41] <buck1> ronny: it's a wrapper on docker to do some common tedious stuff

[19:02:54] <buck1> it needs to evaluate .dockerignore, but the semantics are deep inside the go stdlib

[19:02:57] <buck1> thus the go program

[19:03:31] <buck1> nanonyme: we (thought we) had to invent a thing to enable wheel --platform

[19:03:49] <buck1> nanonyme: https://github.com/asottile/pip-custom-platform

[19:04:04] <nanonyme> Sounds awkward

[19:04:07] <buck1> nanonyme: you're implying there's a simpler way?

[19:04:08] <buck1> it is.

[19:04:24] <nanonyme> If you want to integrate with distro package manager, create distro-specific packages

[19:04:31] <buck1> ogv

[19:04:33] <buck1> of course

[19:04:48] <buck1> that's not a requirement

[19:05:46] <nanonyme> (I mean, the Python package management + distro package management is by nature incompatible; you must never use root with former whereas you're always supposed to use root with latter)

[19:05:52] <buck1> nanonyme: oh i understand you now, i can build deb (et al) using setuptools as part of the build step

[19:06:17] <buck1> by nature? by legacy design.

[19:06:28] <nanonyme> No, by design

[19:06:33] <buck1> we agree

[19:06:39] <nanonyme> It's not just about legacy

[19:06:54] <buck1> we just disagree whether the designers are legacy

[19:07:14] <buck1> we agree it's by design

[19:07:33] <buck1> i dont think it's an *inherent* property of apt/pip

[19:07:55] <nanonyme> It's an inherent property of system package manager vs not-system-package-manager

[19:07:57] <buck1> if the communities devoted blood and tears, they could be merged

[19:08:07] <nanonyme> It's a design choice pip is not-system-package-manager

[19:08:09] <buck1> dpkg install --prefix ~/myprefix

[19:08:28] <buck1> isn't a thing, but there's no inherent reason why not

[19:08:33] <buck1> other than inertia

[19:08:40] <nanonyme> (it can be made that way but it can break your whole distro)

[19:08:41] <buck1> both technical and political

[19:09:25] <buck1> certainly autotools doesn't mind co-habitating with pip

[19:09:37] <buck1> anyhow.... i didn't come to rant

[19:09:48] <buck1> i just want some way to package my thing which has a small .go component

[19:09:56] <buck1> if it's only .deb that can do it ok

[19:10:03] <buck1> but setuptools would be more handy in my case

[19:10:41] <nanonyme> Well, I think you can bundle whatever you want inside a wheel. I just read you actually wanted to pull in some data form external distro package

[19:11:14] <buck1> no there's no distro package. it's all mine

[19:12:54] <buck1> nanonyme: we had to invent the pip-custom-platform to handle the fact that "linux" isn't a singular thing and we need to support more than one flavor

[19:13:00] <buck1> but we still like wheels

[19:13:10] <nanonyme> Right

[19:13:40] <nanonyme> So is the problem then in standard library distutil or what?

[19:13:58] <nanonyme> Ie that it can't model Linux properly as multiple distros and multiple distro versions

[19:14:05] <buck1> the problem isn't in code bc there's no packaging code atm

[19:14:13] <buck1> i'm trying to figure out how to code the pacakaging

[19:14:31] <buck1> nanonyme: oh re pip-custom-platform?

[19:14:37] <nanonyme> Yes

[19:15:06] <buck1> in the worst case your distro is linux-from-scratch and your version is $USER

[19:15:22] <buck1> i dont think using (distro, version) as the platform is viable

[19:15:48] <buck1> i almost think pip should refuse and require a platform name input on linux

[19:15:57] <buck1> maybe with some whitelist exceptions

[19:16:20] <nanonyme> Well, I doubt linux-from-scratch covers more than a 1% of Linux users. Having 99% of cases right would already be pretty good

[19:16:25] <buck1> arch doesn't really have a version

[19:16:31] <nanonyme> I guess Gentoo has the same problem though

[19:16:37] <buck1> it would have to be arch-$USER

[19:16:52] <buck1> debian flavors should be pretty good to go tho

[19:17:07] <buck1> i'd whitelist the distros that have actual hard versioning

[19:17:29] <buck1> and of course add some interface for users to add more

[19:17:42] <buck1> then for everything else require a --platform option

[19:18:17] <buck1> would still leave pypi an untenable mess tho

[19:18:53] <buck1> an opposite approach would be to hash the list of symbolic links as the platform

[19:18:55] <nanonyme> PyPI could do with better listings in any case

[19:19:22] <buck1> so a packages that just needs /lib/libc.so.1.2.3 would install find on any platforms that have it

[19:19:25] <buck1> fine

[19:20:08] <buck1> would bring down the number of platforms dramatically

[19:20:45] <buck1> and you'd be able to find compatibility between distros

[19:21:19] <nanonyme> You mean test /lib/libc.so.1.2.3, /lib32/libc.so.1.2.3, /lib64/libc.so.1.2.3, /usr/lib/libc.so.1.2.3, /isr/lib32/libc.so.1.2.3, /usr/lib64/libc.so.1.2.3 to see if any works?

[19:21:44] <buck1> i mean that the linux "platform" for a package would be the list of symbolic links in the package

[19:23:08] <buck1> since the link is specifically /lib/libc.so.1.2.3, no you wouldn't search all those

[19:23:24] <buck1> would only be installable on platforms that specifically have that exact file

[19:23:38] <buck1> which is the same set of platforms on which it would link

[19:24:22] <nanonyme> Yeah, but what if the file can be in four different places and be legit match?

[19:24:42] <nanonyme> You can't really know without actually loading it if it's actually right bitness

[19:24:43] <buck1> maybe i dont know how the dynamic linker works, but I think it only looks at the one spot

[19:24:52] <buck1> no?

[19:25:21] <buck1> erm if it's a relative link eg libc.so.6

[19:25:22] <nanonyme> Kind of, sort or, but you can't know beforehand what that spot is

[19:25:39] <buck1> but some are absolute no? eg /lib/libc.so.6

[19:26:01] <nanonyme> Not really. /lib could be either 32bit or 64bit depending on distro

[19:26:14] <buck1> the cpu arch is handled just fine already

[19:26:22] <buck1> orthogonal and solved

[19:26:35] <nanonyme> No, it's not. If your Python has different bitness, loading the library will fail

[19:26:55] <buck1> that's part of the whl metadata in a separate field now no?

[19:27:04] <buck1> still is a separable problem

[19:29:35] <buck1> i'm saying that if the whl generator created a list of dynamic links, and the whl installer could efficiently match itself to those link-lists, then distro as a concept is factored out

[19:29:58] <nanonyme> Well, what would you have happen if /lib is 32bit, Python is 64bit, /lib/foo.so exists, /lib54/foo.so does not. Should the package be installable or not?

[19:30:16] <nanonyme> What about if /lib is 64bit, /lib64 does not exist at all and /lib32 is 32bit?

[19:30:42] <buck1> is the package 32 or 64 bit?

[19:30:44] <buck1> that's in the metadata

[19:31:05] <nanonyme> I'm saying the directories on disk are different, you can't handle this with static metadata

[19:31:13] <buck1> pip's current flow is to find all packages, filter to the installable ones, then pick the "best" one

[19:31:14] <nanonyme> You have to actualyl load the libraries to see if they exist

[19:31:45] <nanonyme> If you want an example of this, install both Ubuntu and Fedora and see how their ideas of lib paths differ

[19:32:00] <buck1> we can determine if a package is installable given the current whl metadata plus the list of dynamic links

[19:32:30] <buck1> a "linux" package*

[19:32:40] <nanonyme> What dynamic links?

[19:32:56] <nanonyme> There aren't any reliable dynamic links

[19:33:05] <buck1> a dynamic link is just a string

[19:33:08] <nanonyme> Everything is distro-specific

[19:33:13] <buck1> not very different from a symlink

[19:34:01] <nanonyme> Yeah, won't work. What you should have is just dyn load the library on pip installation, refuse to install if loading fails

[19:34:02] <buck1> let's be more concrete and constrain ourselves to libc.so.6

[19:34:16] <buck1> you have to decide which to attempt to install

[19:34:22] <buck1> is what's imposssible atm

[19:34:59] <nanonyme> Also a distro is allowed to configure lib paths to be *anything* in Linux based on /etc/ld.so.conf

[19:35:13] <buck1> so we'll have to ask the linker

[19:35:31] <nanonyme> Yeah, in other works dyn load the library?

[19:35:43] <buck1> well we don't need to download the whole dang whl

[19:35:49] <buck1> just the tiny text file

[19:35:55] <buck1> but you could do it that way

[19:37:00] <buck1> we can already filter out whls that have the wrong width, wrong arch

[19:37:04] <buck1> so lets consider that done

[19:37:11] <nanonyme> I'm a bit sceptic tbh, I still think it's less effort to just drop Python packaging and do anything more complicated through distro packaging

[19:37:35] <buck1> we still have a list of packages that link to various libc's

[19:37:49] <buck1> if we know which one each links against, we can ask the linker if that's going to be ok

[19:37:56] <buck1> and filter that way

[19:38:53] <buck1> the problem (I have) is that it's hard to get any particular project to depend on some deb, some python packages

[19:39:06] <buck1> it's much easier to do just one

[19:53:09] <[Tritium]> buck1: unless you are distributing through your OS package manager (you have a thousand servers, and your own apt-proxy to distribute to...) dont depend on OS modules (exception: massive gui libraries that kind of have to be managed by the os)

[19:53:26] <dstufft> buck1: might be useful: https://github.com/shazow/gohttplib

[19:59:56] <buck1> [Tritium]: we do

[20:00:24] <buck1> dstufft: cool! thanks. that was the prior art i was hoping irc might show me

[20:00:44] <buck1> erm no there's no python involve

[20:01:13] <buck1> oic it's an example of binding python with go

[20:01:37] <buck1> that wasn't actually my issue. subprocess is good enough for me. my problem is depending python on go

[20:02:42] <buck1> the obvious solution is to deb-pkg the go bit and setuptools package the python bit, but then maintaining the dependency is a matter of documentation and convention which is a smell

[20:03:05] <buck1> i could deb-pkg the python bit and that goes away, but the rest of our project uses setuptools dependency system. virtualenv is nice

[20:03:22] <buck1> so i'm currently going down the path of setuptools-packaging the go bit

[20:03:43] <buck1> probably in the same package as it's not really a separate idea

Log file Viewer

Help | Karma | Search:

#pypa logs for Thursday the 23rd of July, 2015