pmxbot IRC Log Viewer

[10:44:18] <fladd> Hello everyone, I have a quick quesition:

[10:46:58] <fladd> I am the main developer of Expyriment, a Python package for cognitive and neuroscientific experiments, and I would like to make this package also availabel in PyPi. The problem is, that we depend on Pygame which, as we all know, is not available in PyPi (and probably never will be). So my question is: What is the best way for us to have Expyriment in PyPi? If we define Pygame as a requirement, then installation with PIP will always fail, since Pygame is not

[10:46:58] <fladd> available. If I do not specify it somehow, then people don't know that they need to also download it first. What would you suggest?

[10:54:46] <mgedmin> fladd, don't add pygame to install_requires; have the main script of your thing wrap pygame ImportErrors and tell the users they need to install pygame

[10:55:11] <mgedmin> provide a link to a page with instructions, perhaps with specific advice for certain distros (e.g. on Ubuntu it's sudo apt-get install python-pygame)

[12:08:24] <fladd> mgedmin, thanks, that seems like the best solution indeed

[12:10:39] <mgedmin> I have hopes this will be fixed at some point

[12:11:19] <mgedmin> https://bitbucket.org/pygame/pygame/issue/59/pygame-has-no-pypi-page-and-cant-be is acknowledged as a bug

[12:35:37] <fladd> Well, it is a 5 year old bug report...

[12:35:58] <fladd> I also don't really get what the problem is. They do have binary packages for all platforms. They just need to upload them. Takes 3 minutes...

[12:37:12] <mgedmin> you can't upload linux binary packages and expect them to work

[12:40:33] <apollo13> well uploading the source tar.gz for linux should be fine, no?

[12:40:41] <apollo13> unless their setup.py is borked ;)

[12:43:50] <mgedmin> actually it is! if you read the bug, you'll see the setup.py is asking questions via raw_input9)

[12:44:05] <mgedmin> but pip has the stdout redirected to /dev/null so pip install just hangs with no indication of what's going on

[12:44:31] <mgedmin> so they're considering a switch to a different configuration mechanism, but somebody has to decide what and write the code

[12:49:21] <fladd> yes, and that is not going to happen...they canno even decide when is time to release a final version of their since-years-in-pre-stage-0.9.2

[12:49:36] <fladd> pygame seems to be a rather stuck project

[12:49:42] <fladd> unfortunately

[13:00:22] <fladd> can anyone tell me what files pypi expects for uploading when selecting "source"?

[13:00:29] <fladd> zip files do not work: Error processing form

[13:00:29] <fladd> invalid distribution file

[13:01:26] <mgedmin> files produced by python setup.py sdist

[13:01:34] <mgedmin> zip or .tar.gz are fine

[13:02:08] <mgedmin> there may be some conventions for what has to be inside (a foo-version/PKG-INFO?)

[13:04:08] <fladd> it is a zipped folder that was produced by setup.py build

[13:09:50] <mgedmin> setup.py build is something unrelated; I don't think I ever had to use it

[13:09:56] <mgedmin> do a setup.py sdist

[13:10:37] <fladd> well, we use the build process to put timestamps and revision numbers in our source files for release

[13:10:49] <fladd> so sdist won't work in my case, because the files would have not been tagged

[13:13:33] <fladd> mmh, I also thought that I could point pypi to a download page, instead of uploading releases there too? Wasn't that possible at some point?

[13:13:51] <fladd> I gave it the URL of our github release page, but a pip install expyriment will not find it...

[13:47:52] <ionelmc> fladd: you could try the wheelhouse trick

[13:48:08] <ionelmc> built it localy, and use that local wheelfile

[14:00:37] <fladd> and this I would upload then?

[14:00:44] <fladd> I thought wheels are only for Win and OS X?

[14:09:20] <ionelmc> fladd: well no, no upload, just use it locally

[14:09:32] <ionelmc> depends on how you deploy

[14:10:01] <ionelmc> (the whole thing is a workaround)

[14:16:30] <mgedmin> external hosting with pypi is still possible, but it's complicated to set up (securely)

[14:16:36] <mgedmin> there ought to be some documentation in some pep somewhere

[14:17:10] <mgedmin> some people consider it a bit user-hostile (introduces additional single points of failure, while PyPI itself is behind a CDN)

[14:17:35] <mgedmin> users may need to opt in explicitly (pip install --allow-external pkgname or --allow-all-external (sp?))

[14:17:40] <mgedmin> fladd: ^

[14:18:16] <fladd> i see, thanks!

[14:18:36] <mgedmin> how about you do an sdist and compare its contents with whatever you're producing by hand, then change your build script to put the files in the right places/add the missing metadata/whatever is necessary?

[14:19:00] <mgedmin> worst case, build the sdist then unpack it, patch the version numbers/timestamps and repack

[14:19:08] <mgedmin> use twine to upload the resulting archive

[14:20:59] <apollo13> --allow-external pkgname pkgname (you need packagename twice)

[14:35:34] <fladd> yes, that would be an option indeed.

[14:35:58] <fladd> Or I use gitattributes to create a single file which includes the revision number...

[14:36:13] <fladd> ...if this is possible...

[15:10:53] <fladd> after successfully uploading a source of my package, I get the following error when trying to install it with PIP:

[15:10:53] <fladd> IOError: [Errno 2] No such file or directory: '/private/tmp/pip_build_root/Expyriment/setup.py'

[15:11:03] <fladd> Any ideas what that means? I am totally puzzled

[15:14:26] <fladd> when trying to use PIP to install a package that works just fine with setup.py install, I get the following:

[15:14:27] <fladd> error: option --single-version-externally-managed not recognized

[15:14:34] <fladd> I have no clue what is going on

[15:17:05] <xafer> where did you upload it ?

[15:18:46] <fladd> https://github.com/expyriment/expyriment/releases/download/v0.7.0/expyriment-0.7.0.zip

[15:18:53] <fladd> This is our official latest release

[15:19:10] <fladd> This can be installed just fine when downloading and running setup.py install

[15:19:19] <fladd> I thought runnning "pip install https://github.com/expyriment/expyriment/releases/download/v0.7.0/expyriment-0.7.0.zip" should do the same

[15:19:26] <fladd> but it does not

[15:19:28] <fladd> it fails

[15:21:18] <fladd> I don't know what the option "--single-version-externally-managed" is or who sets it

[15:22:43] <tdsmith> it's a setuptools option pip uses that tells distutils to do the install; pip wraps a shim around setup.py invocations that imports setuptools first. setuptools should monkeypatch distutils so that all calls to distutils.setup are actually invoking setuptools.setup

[15:22:47] <tdsmith> some russian-doll action

[15:23:27] <tdsmith> it looks like the monkey-patching isn't working somehow

[15:25:56] <fladd> yes, something seems broken.

[15:26:09] <fladd> Is there anything I can do to fix this on my site? Or is this a PIP bug?

[15:27:20] <mgedmin> does it work if you 'pip install .' in the source tree?

[15:28:41] <tdsmith> it doesn't. i wonder if subclassing distutils.command.install is faking setuptools out

[15:28:49] <xafer> fladd, I'd say it is due to your cmdclass={'install': Install} override

[15:29:01] <mgedmin> could be

[15:29:12] <fladd> xafer, it works fine with setup.py, though

[15:29:17] <mgedmin> rewrite your setup.py to rely on setuptools instead of distutils?

[15:29:24] <mgedmin> pip only supports setuptools

[15:29:38] <fladd> mmh

[15:30:18] <fladd> maybe I should just write a simple python script myself that downloads and istalls everythng...this can call pip in the background for the dependencies...it seems a lot of trouble to make my stuff work with pip...maybe that is why there is no pygame version :-)

[15:31:13] <mgedmin> it's a lot of trouble only if you're doing something strange and complicated (like overriding the distutils Install command... why????)

[15:31:26] <xafer> well typically "# Clear old installation when installing" would be useless with setup.py based on setuptools and pip doing the cleanup

[15:31:57] <fladd> The reason is that I need to create a "built" version of my source, in which the revision number it comes from is noted

[15:33:10] <fladd> I could not find a better way to do this

[15:33:20] <fladd> doing it during the build process seemed the only way

[15:33:52] <mgedmin> you want the version number in the source tarball

[15:33:52] <tomprince> fladd: Something like https://pypi.python.org/pypi/setuptools_scm/1.0.0 ?

[15:34:16] <fladd> no, I want the revision number of GIT in at least one source file

[15:34:21] <mgedmin> "build" and "install" are called when you're actually installing from a source distribution, and so are a bit late for this IMHO

[15:34:31] <fladd> people should also be able to clone the git repo and then install from there.

[15:35:56] <fladd> the process now is straight forward. Person X clones the repo and installs our package. He has a problem, I ask him to send me his expyriment.get_system_info(), this includes the revision number, and I know exactly which code he cloned from git.

[15:36:49] <mgedmin> I wish I knew what was the best way to achieve this

[15:36:57] <fladd> in the same way, we only create a source zip in which the files have been run through "build" before, so again, we can make sure that the code includes the exact revision number, in case there are specific problems

[15:36:58] <mgedmin> I acknowledge the usefulness

[15:37:51] <mgedmin> if you don't get a useful answer on IRC (because timezones mean people who understand setuptools deeply enough might be sleeping right now), do ask on the distutils-sig@ mailing list

[15:37:53] <fladd> yeah, it doesn't seem to be ver common or so, at least I could not find a more standard way of doing it. That is why I hacked it in like i did. And until now there was also no problem with that

[15:38:14] <fladd> okay, I might try this

[15:38:24] <fladd> I really appreciate your help though! Thanks!

[15:38:28] <mgedmin> at some point in the past you could set a flag in setup.cfg and setuptools would create version numbers like .dev-r1234 for subversion revision 1234 automatically

[15:38:36] <mgedmin> I've no idea if this works with git

[15:38:54] <mgedmin> setuptools has no native git support, but a plugin (setuptools-git) exists

[15:39:03] <mgedmin> primarily it's intended to replace the need for maintaining a MANIFEST.in

[15:39:11] <mgedmin> but maybe it also can add version numbers somewhere

[15:39:46] <mgedmin> also, PEP 440 version numbers introduce some complications (.dev-r1234 is no longer valid, you're supposed to use +r1234 or something, but nobody fixed setuptools for that yet)

[15:40:33] <xafer> fladd, your git revision use case does not seem to need the "class Install(install):" override, did you try without this piece ?

[15:41:10] <tomprince> https://github.com/warner/python-versioneer is another option.

[15:42:48] <fladd> xafer, I need this part for cleaning old versions, otherwise it gets a mess

[15:43:58] <mgedmin> pip does that for you

[15:44:07] <mgedmin> in fact setuptools does that, I believe

[15:44:19] <mgedmin> if you relied on setuptools instead of distutils you wouldn't have to write manual cleanup code

[15:46:41] <fladd> I believe only in the latest version of Python (2.7.9), setuptools is builtin, right? I remember that when we started the project, distutils was the only included installation mechanisms in Python

[15:47:36] <fladd> oh, actually, I just remember, the cleanup code is for the windows installer that distutils creates! this did not clean old code before installing new one

[15:48:48] <fladd> also, https://docs.python.org/2/distutils/index.html only talkes about distutils it seems

[15:48:50] <mgedmin> setuptools isn't included with python itself, but the latest 2.7.9 has an ensurepip module that will download pip for you

[15:48:56] <mgedmin> and then you can pip install setuptools

[15:51:40] <fladd> yes, that is what i meant

[15:56:02] <tdsmith> i think ensurepip installs setuptools as well

[15:56:46] <fladd> this might be a stupid question, but I just discovered the 'dependency_links' parameter: what kind of files does this expect here? Source files? Installers (.exe, .dmg)?

[15:58:19] <mgedmin> previously it used to accept web pages linking to files (sdists and binary archives too)

[15:58:44] <mgedmin> now I think it wants direct links to downloadable files (sdists and/or binary builds)

[15:59:02] <mgedmin> also the only binary archives that pip supports are wheel files

[15:59:16] <mgedmin> setuptools also supports .egg and .exe (on windows)

[16:00:49] <fladd> I see, thanks

[16:02:14] <tomprince> fladd: dependency_links are deprecated.

[16:02:23] <fladd> ah...great...

[16:03:46] <fladd> I am just wondering if switching to setuptools is worth the effort, given that my dependencies are not met anyway, due to pygame not being available...

[16:05:53] <mgedmin> that is a good question, yes

[16:06:12] <mgedmin> postpone until pygame gets fixed?

[16:06:22] <mgedmin> for end users it's very nice to be able to install stuff with pip install stuff

[16:08:59] <fladd> what is the setuputils equivalent of "build" then? I still want to modify all my files when creating a, let's say, wheel for my project

[16:10:32] <mgedmin> I freely admit to ignorance here

[16:10:48] <mgedmin> usually you build wheels with bdist_wheel; I don't know if that involves calling build or not

[16:11:50] <fladd> no, it seems the code is written in a way that does not allow for hooking in...

[16:12:00] <fladd> just checked the source...

[16:29:29] <fladd> I just replaced my code with the setuptools equivalent (so instead of importing all functions from distutils, I imported the same functions from setuptools). build_py is still there, so I assumed my mechanism should work. However, again, pip cannot install the results, it again cannot find some temporary directory

[16:31:24] <xafer> did you upload a branch ?

[16:37:43] <fladd> I tried with a local zip file

[16:38:00] <fladd> pip install path/to/file.zip

[16:48:21] <xafer> well, it should work, what's the output ?

[18:19:03] <sontek> What is the best way to do this but not require numpy in the virtualenv prior to installing my lib? http://paste.ofcode.org/DnDi8u2gGf9SLEcAuGFewm

[18:33:54] <wickman> kts: it depends on 0.2.0-dev0 which is unpublished. if you change to 0.2.0 it should work, since i published it last night.

[18:34:03] <wickman> whoops, wrong channel

[18:51:42] <ionelmc> sontek: how does your lib depend on numpy?

[18:55:19] <sontek> ionelmc: It uses numpy within the functions that it is cythonizing

[18:55:30] <sontek> ionelmc: So it needs the include dirs so that when it cythonizes it compiles properly

[19:00:26] <ionelmc> sontek: maybe setup_requires would do the trink?

[19:00:37] <ionelmc> you still have to do imports lazily

[19:05:43] <sontek> ionelmc: Yeah, thats the part I'm wondering about. For the build_ext to run I need to import numpy

[19:05:56] <sontek> ionelmc: so how do I have numpy in setup_requires but then use it in build_ext lazily

[19:19:52] <ionelmc> sontek: yeah, that's the limitation of setup_requires

[19:27:29] <dstufft> sontek: take alook at uh

[19:27:37] <dstufft> sontek: https://caremad.io/2014/11/distributing-a-cffi-project/

[19:27:45] <dstufft> that's about CFFI, but it's the same problem

[19:27:51] <dstufft> delaying the use of a function until after setup_requires

[21:35:46] <sontek> dstufft: thanks for the blog post! This is looking a little sketchy but seems to be working: http://paste.ofcode.org/sAYxjYvupZ3uY8r4u2dzgC

[21:35:54] <sontek> dstufft: Does that look right to you?

[21:39:47] <lvh> Hi :)

[21:40:02] <fladd> xafer, the output is again that some temporary file is not known

[21:40:56] <lvh> I'm trying to build some wheels, and it's not working. Is install_requires supposed to install things before trying to build wheels when I do pip wheel -r requirements.txt?

[21:41:18] <lvh> Here's the output: https://gist.github.com/lvh/fd31acefcfcbdf1038f8

[21:41:40] <tomprince> I don't think it does.

[21:41:44] <lvh> You will notice that it is complaining about a versioning conflict with six; there's an old version of six

[21:43:03] <tomprince> Does it actually need 'six' as part of building the wheel?

[21:45:59] <tomprince> I bet it is because currently, setup_requires is handled by setptools, and so doesn't like conflicting version of something required by a setup_requires.

[21:49:18] <lvh> tomprince: apparently install_requires is in cryptography's setup.py?

[21:49:56] <lvh> tomprince: I don't know why it feels it needs six, but it does, and pip is pretty unhappy about that

[21:50:12] <lvh> I was able to reproduce that behavior independently: https://gist.github.com/lvh/49003c72cfc8964d3fbd

[21:50:33] <lvh> tomprince: so, yes, it seems that you are correct and that it's just setuptools being annoying

[21:51:04] <lvh> tomprince: what is the appropriate resolution? otter (the package that's being built here) doesn't care about six, so it seems weird to do something to otter

[21:51:05] <tomprince> lvh: It looks like cryptography setup_requires six.

[21:51:27] <tomprince> I think you probably need to install a new enough six before runinng the build (for the momemnt).

[21:52:22] <tomprince> There is work going on to make pip able to take-over setup_requires. (At least if packages opt-in to it).

[21:55:21] <fladd> xafer, here is the whole error message: http://pastebin.com/dcM9Y2fq

[22:13:31] <dstufft> lvh: building wheels uses setup_requires

[22:13:48] <dstufft> install_requires is runtime

[22:39:12] <xafer> fladd, what does your setup.py look like now ?

[22:42:25] <fladd> xafer, I found the problem! Pip cannot deal with a situation where within a something-0.1.0.zip there is a folder something-0.1.0 which contains everything

[22:42:41] <fladd> it excpets the setup.py at the highest level

[22:42:55] <fladd> weird, but at least i found the cause

[22:45:06] <xafer> good :)

Log file Viewer

Help | Karma | Search:

#pypa logs for Monday the 23rd of March, 2015