Log file Viewer

[01:26:42] <disi> `pip install --no-binary :all: foo` will fail unless all of foo's deps offer sdists, wont it?

[02:41:20] <kennethreitz> how do you represent two acceptable versions in a setup requires string?

[02:41:28] <kennethreitz> chardet>=3.0.2,3.1.0 doesn't work

[02:42:40] <kennethreitz> ah got it!

[16:38:22] <pombreda> kennethreitz: and that is?

[18:18:28] <bwe> Can't there be a single infrastructure automate creation of wheels? => I wonder if even additional information to setup.py is required at all?

[18:20:45] <bwe> 1) The effort for the single developer to set up his multi-platform (Mac / Linux / Windows) wheel creators is a tough barrier of consistent adoption of wheels for all major platform tracks. 2) the net gain of a centralised wheel creation machine park might well level higher than the effort added for an additional package.

[18:25:12] <bwe> Example: User installs package A, package A requires package B, package B has C extensions. Developer of package B does not have time / does not want to mingle around with Windows, hence no Windows wheels. Eventually the user is required to install 4 GiB C++ bindings on Windows to continue the installation. Now, the user has the platform to make a wheel. However, he cannot push the wheel to pip servers. This is

[18:25:18] <bwe> frustrating.

[18:26:22] <bwe> Can I include a wheel of package B in package A?

[18:33:11] <agronholm> bwe: how would including one wheel in another solve this?

[18:33:29] <agronholm> if you want to have a repository for custom built wheels, use devpi

[18:37:57] <bwe> agronholm: Is this recommend to ship a dependency for my package to end users?

[18:38:32] <agronholm> end users = users who download from pypi?

[18:40:20] <bwe> agronholm: No, they download a snapshot from hg repo (my package). Yes, installing the package pulls in deps from pypi, namely quicktions (with C extensions) where no wheel is provided. -- What your advice in this situation?

[18:40:49] <agronholm> why not just give them access to your devpi server?

[18:40:51] <agronholm> forget hg

[18:41:03] <agronholm> upload wheels directly to devpi

[18:41:13] <agronholm> including windows wheels of that dependency

[18:42:17] <bwe> Sounds brilliant.

[18:42:49] <bwe> However, I would require access to Windows system for each release to make a wheel, right?

[18:43:33] <agronholm> each release of the dependency, yes

[18:43:43] <agronholm> when you want to update it

[18:44:12] <bwe> That would exclude installing from head automatically.

[18:44:17] <pombreda> bwe: I use a build loop with Travis and Appveyor to build wheels with native on Win32/63, macOS and Linux32/64 if that is of any interest to you

[18:44:23] <agronholm> bwe: why?

[18:44:36] <agronholm> why would you have to build a new wheel every time your own app gets updated?

[18:46:15] <pombreda> bwe: short of having a way to cross-build for win/mac/linux on a sngle host you have to build on each os ;)

[18:46:23] <pombreda> bwe: my build loop https://github.com/pombreda/thirdparty-manylinux/ and https://github.com/pombreda/thirdparty and then it pushes my built wheel to http://dl.bintray.com/pombreda/thirdparty/

[18:49:11] <pombreda> bwe: on my side, for apps, I then bundle the pre-built wheels for all my deps in the app repo. So none needs a compiler to run from a checkout

[18:49:21] <pombreda> (I said for apps, not for libs)

[18:50:08] <pombreda> and anyone can install from head automatically (this is all public code too, if you want a link)

[19:05:55] <bwe> agronholm: No, as an update of my package does not entail an update of the dependency.

[19:07:22] <agronholm> bwe: by installing from head, do you mean your own app or the dependency?

[19:07:52] <bwe> agronholm: my own package.

[19:08:11] <agronholm> so why would users be no longer able to do that?

[19:11:20] <bwe> pombreda: This sounds pretty sophisticated. bintray acts as your devpi server?

[19:13:55] <bwe> agronholm: My users would be able to install from my devpi server the wheel of my package and the wheels of its dependencies for head. Any change in my package (not affecting the dependencies) triggers only the creation of a new wheel of my package.

[19:14:42] <agronholm> I'm not sure if it's a good idea to depend on development versions of the dependencies

[19:14:57] <agronholm> almost certainly not

[19:17:08] <bwe> I never intended of doing so -- did I ?

[19:17:37] <agronholm> then I don't understand the problem

[19:17:49] <agronholm> every time you push a new commit, a new wheel is generated

[19:18:23] <bwe> Of my package - not the dependencies.

[19:19:19] <agronholm> what is preventing you from building a wheelhouse using "pip wheel" and uploading those to your devpi? that's what I did

[19:21:58] <bwe> 1.) Currently I don't have a devpi. 2.) I am unclear about how an update of a dependency (originated externally) is handled.

[19:23:53] <bwe> 3.) Even if I get this to work, I feel not having solved the root cause -- which is lack of automated wheels creation for dependency.

[19:24:06] <pombreda> bwe: yes, bintray acts as a pypi server alright

[19:24:17] <pombreda> but I bundle, I do not use as a pypi server

[19:25:46] <agronholm> bwe: pip wheel creates wheels of your deps

[19:26:12] <pombreda> bwe: there is no lack of automated wheel creation for deps... there are some dev deps such as compiler that may be needed. This is not a python issue but a native + python, right?

[19:27:16] <bwe> pombreda: True. Either install compiler or get the wheel built on a compatible platform using that compiler.

[19:27:19] <pombreda> bwe: and if you want pre-built wheels for all your deps (even the native ones and these that do not exist on Pypi) and not use your own devpi server, then you need to bundle somehow

[19:27:52] <bwe> agronholm: I misunderstood this. Thanks for clarifying this.

[19:28:21] <pombreda> bwe: either you are in dev mode and a lib, and you could expect developers to have a toolchain to build native OR this is an app, and then I would always bundle deps with an app

[19:29:03] <bwe> pombreda: thirdparty and thirdparty-manylinux make the wheels, then you bundle them and put them to bintray. Did I get this right?

[19:29:12] <pombreda> bwe: yes

[19:29:42] <pombreda> bwe: for an app such as: https://github.com/nexB/scancode-toolkit/tree/develop/thirdparty

[19:29:59] <pombreda> this contains all my prebuilt wheels for all my deps at full depth

[19:31:07] <pombreda> And these can be installed locally from a checkout with:

[19:31:47] <pombreda> pip install . --find-links=./thirdparty/base/ --find-links=./thirdparty/prod/ --no-allow-external --use-wheel --no-index --no-cache-dir

[19:32:02] <pombreda> this does NO network call to any pypi-like server

[19:32:21] <pombreda> bwe: do you get the approach?

[19:33:36] <pombreda> (actually I am not making this call directly, I am using instead a ./configure bootstrap script for this, which also creates a venv. but that's outside your question)

[19:34:15] <pombreda> the point is that if you are in dev mode, MO is that you can expect user to have a toolchain to build natives

[19:34:32] <pombreda> if you are server users with an app, IMHO bundling is the only sane way

[19:34:42] <pombreda> s/server/serving

[19:35:18] <pombreda> in general having a user run a pip does not make much sense ;)

[19:35:45] <bwe> pombreda: thirdparty, thirdparty-manylinux CLEAR. How do I bundle then the wheels? Push to bintray *not* clear yet, but this should be handled by appveyor/travis somehow, right?

[19:36:19] <pombreda> so beyond pip, deploying apps is a whole topic of its own: installers, a Python runtime, etc... (check appimage for instance) or distro packages

[19:37:20] <pombreda> bwe: appveyor and travis may be able to store built wheels, I just used bintray instead. Just as a staging place where I fetch stuff from and save in my committed thirdparty dir

[19:37:21] <bwe> pombreda: I wonder why there's currently no professional service easing that as most would like to focus on their app instead of the way of distributing it in a sane way.

[19:38:05] <pombreda> bwe: building apps for multiple OSses that install well is a difficult task ;)

[19:38:13] <pombreda> always mostly custom

[19:38:53] <pombreda> I have served 2 million+ of downloads of open source apps, so I have been there

[19:39:19] <pombreda> and getting it right on linux/mac/windows is ... difficult

[19:39:42] <pombreda> or maddening at times

[19:40:36] <pombreda> bew: if you are distributing apps with natives on multiple OSes, you will have to embrace this difficulty IMHO

[19:40:42] <bwe> pombreda: So, your bundling toolchain creates an up to date bundle for each change you push or not?

[19:41:20] <pombreda> no, I only update my thirdparty whenever I update a dep.

[19:41:50] <pombreda> for now, on scancode, I do not build installers: just a zip and tarball.

[19:42:08] <pombreda> this contains everything self-contained except Python

[19:42:54] <pombreda> so the end-user install is: 1. install Python 2. download the archive and extract 3. run (which will behind the scene install the wheels and else on the first run)

[19:43:16] <bwe> pombreda: what would you gain, what loose creating a wheel instead zip/tarball for your package (could this include the third-party)?

[19:43:50] <pombreda> I will eventually have distro packages and (Linux appimage and win and mac installers) with a bundled Python interpreter

[19:44:37] <pombreda> bwe: I would loose the pip installation scripts I use.

[19:45:07] <pombreda> bwe: I would need to instead not distribute a wheel but an installer.

[19:45:47] <pombreda> bwe: atm my first run launch ./configure or configure.bar which create a local venv and install the app and deps from thirdparty

[19:46:39] <pombreda> bwe: if I build a wheel, this would not contain the deps (and having a wheel with wheels would not work, but I could create a mega wheel, though I am not sure that there is much tooling for this)

[19:47:02] <pombreda> again: wheels ARE NOT an app deployment format IMHO

[19:47:17] <pombreda> wheels are a way to provision deps for development

[19:47:42] <bwe> What a meaningful distinction.

[19:47:45] <pombreda> bwe: mixing dev and end-user install will get you nowhere fast IMHO ;)

[19:50:51] <pombreda> bwe: you should read this http://sedimental.org/

[19:51:35] <pombreda> bwe: this is an excellent article and while centered on Python this is not Python-only

[19:51:58] <pombreda> http://sedimental.org/the_packaging_gradient.html is a better link

[19:56:37] <pombreda> bwe: once you have been through this article things should be much clearer I think

[19:56:43] <bwe> pombreda: Thanks for the link. I stumbled over Python-MKL by Intel for better performance, tried conda, yet discarded this path recently. -- https://bpaste.net/show/fc9b7ad7005f would be so far my strategy.

[19:58:24] <pombreda> bwe: appevyor and travis likely work on bitbucket (and if travis does not, something else with mac/linux support like circle is likely to work)

[19:59:14] <pombreda> bwe: but yes, if your code is an app, that would be an ok strategy (the use of bintray in between is just me and a convenience)

[19:59:27] <pombreda> bwe: is your code public?

[20:00:21] <bwe> pombreda: http://bitbucket.org/hyllos/effectus-python # bear with me, it's my first deployment / distribution challenge

[20:00:26] <pombreda> and which OS do you support?

[20:01:12] <pombreda> bwe: this is a library... so what's the catch?

[20:02:29] <pombreda> this is a fairly engaged setup.py you have there

[20:03:54] <bwe> pombreda: Do? It's rather *want*: Linux / Mac / Windows. -- The catch is that the Windows installer script fails as quicktions does not provide wheels: https://bpaste.net/show/efad34a4b338

[20:05:42] <pombreda> bwe: ok, then bundling can help there too I guess: you want your library users on Windows (that are devs) to avoid having a toolchain installedd I guess?

[20:06:00] <bwe> pombreda: For Windows it should be a one copy-and-paste have it running show.

[20:06:22] <pombreda> ok, so build the missing wheels and bundle

[20:07:09] <bwe> pombreda: I am not getting what you meant with `fairly engaged` -- would you mind of being so kind to paraphrase this, please?

[20:07:21] <pombreda> bwe, or work with scoder to get prebuilt wheels, he knows for sure how to build these on multiple OSes

[20:07:27] <pombreda> for quicktions

[20:07:59] <bwe> Which would be the more sane way, right?

[20:08:06] <pombreda> bwe: I am just saying that this is a failry complex setup.py with os-specific deps and cython and native extensions

[20:08:27] <bwe> pombreda: Would you simplify it?

[20:08:36] <pombreda> bwe: yes, working with upstream (and eventually contributing the wheel builds) is the way to go IMHO

[20:09:12] <pombreda> bwe: I might, but that's quite likely quite a bit if work. If this work, be happy

[20:10:00] <bwe> pombreda: What concerns you most in the setup.py?

[20:10:35] <pombreda> the number of lines :D

[20:10:48] <pombreda> bwe: I need to run ;) wee can catch up later or anotherr day

[20:11:11] <bwe> pombreda: Thanks, you made my day. Ciao.

[20:11:41] <pombreda> bwe: if your setup.py work do not bother chaning it

[20:12:10] <bwe> pombreda: =)

[20:12:46] <pombreda> bwe: You will have to explain what is an Effect on your page :P

[20:13:03] <pombreda> sounds interesting, but a series of number has not much meaning to the layman I am ;)

[20:15:35] <pombreda> bwe: I could have a use for that I think

[20:16:32] <bwe> pombreda: Oh, would you mind sharing it (don't know which kind of situation you refer to).

[20:21:08] <pombreda> for instance in scancode, I will eventually do complex summarization up a codetree of the licensing and copyrights (an effect)... and I might want to explain its causes

[20:21:43] <pombreda> bew: but I really need to run... and I get your concundrum: your users may be using a library but are data scientists and analysis and may be not developeres

[22:37:10] <dtux> wheels aren't meant to replace sdists, are they?

[23:39:20] <agronholm> no

[23:48:38] <dtux> agronholm: ty