Log file Viewer
#pypa logs for Monday the 7th of December, 2015
[10:05:39] <zahlman> I am building a tool for automating the setup of new projects. What are the current best practices for placing tests relative to the main source files? I would prefer a solution that works elegantly regardless of whether the project is a single .py file, a single package with multiple .py files, or multiple packages.
[10:07:02] <zahlman> (I want the tests to be able to do relative imports where appropriate, and run them from the main project folder.
[10:11:23] <zahlman> I meant to run a tests.py directly (with or without -m as appropriate), but that works too
[10:12:11] <zahlman> although that reminds me, do distributions of the code normally include the tests?
[10:12:28] <tclugg> I'm not sure what the official stance is, but as you may include a separate packaged test suite in `tests_require` I'm not sure it really matters.
[10:13:16] <tclugg> I mean, the point is to test your code, if all relevant code paths are tested via an "external" test suite then you've still achieved your goal by running said suite.
[10:14:17] <ronny> zahlman: thats a bit dependent on what testrunner you want, my personal preference is py.,test and that is flexible
[10:15:30] <zahlman> (the idea is to get new devs up and running as quickly as possible, with the ability to customize their default project setup later.)
[10:16:27] <zahlman> `unittest2`? is that (judging by the naming) a new standard library module in development or something?
[10:16:36] <ronny> hmm, unittest is one of those things that shouldnt exist, its a transliteration of a java api that was a bad translation of a smalltalk api
[10:17:25] <zahlman> ronny: I agree that it looks rather unpythonic, naming conventions and all. but the standard library is heavily dependent on it for its own tests
[10:18:12] <ronny> zahlman: my oppinion on the stdlib is, that for anything but datastructures avoid it ^^
[10:18:45] <tclugg> I have a question: How do I use a feature from setuptools 18.5 in my setup.py and have it work for all incarnations of installation (python setup.py develop, python setup.py install, pip install foo, etc)?
[10:18:53] <zahlman> yeah, `json` at least works but the extensibility leaves something to be desired imx
[10:20:29] <ronny> which means setuptools_scm + tox usage wont break the dev env on regular basis :)
[10:21:27] <tclugg> ronny: https://github.com/jaraco/setuptools/pull/28 suggests version 18.5 for setuptools, yes...
[10:21:31] <ronny> tclugg: until the build tool pep has been finished, you cant propperly depend on a setuptols newer than whats installed
[10:23:01] <tclugg> if not StrictVersion(setuptools.__version__) >= StrictVersion('18.5'): subprocess.check_call(['pip', 'install', '-U', 'setuptools>=18.5'])
[10:24:48] <tclugg> Has anyone else had issues with tox installing setuptools 18.2 in new environments?
[10:26:42] <tclugg> install_command=sh -c 'pip install -U "setuptools>=18.5" && exec pip install "$@"' sh {opts} {packages}
[10:27:52] <zahlman> ok... it works if I have src/foo.py and src/tests/test.py, which does `import ..foo`, and then at the top level I can `python -m src.tests.test`... having `src` on the name seems a little ugly, but I guess that's unavoidable
[10:30:29] <zahlman> I guess for `setup.py test` to work at all, I need not to explicitly `prune *tests*` in the manifest :)
[10:33:11] <ronny> tclugg: it'll eventuallt get better, but the whole packaging ecosystem could use another 2-3 full time engineers for 1-2 years
[10:36:48] <tclugg> ronny: I've been studying packaging best practices since August (and a bit before too), I've come a long way but now I'm frequently hitting tooling issues.
[10:40:09] <ronny> tclugg: many people have different visions and its a very long term process - so its homogenous
[10:40:11] <tclugg> ronny: regarding depending on updated versions of setuptools, can we do the upgrade then `os.exec*()` to resume post-upgrade?
[10:42:08] <ronny> tclugg: magically affecting global system state is a big no, it breaks the world for others on regular basis
[10:42:48] <ronny> setuptools or the combination of setuptools + ez_setup would generallz just pull in eggs into the working directory and use a encapsulated workingset
[10:43:10] <ronny> however pips creation killed the notion/tooling around encapsulated installs of packages
[10:43:14] <zahlman> it seems that `setup.py test` will run egg_info and build_ext, but doesn't actually install anything (only makes the local .egg-info folder within the project folder) - yes?
[10:44:39] <zahlman> ronny: and if the working directory is the project directory where I'm developing the code?
[10:45:54] <tclugg> ronny: I'm assuming https://xkcd.com/927/ is highly relevant when talking about python packaging...
[10:48:06] <tclugg> mgedmin: try doing this style instead - https://github.com/funkybob/django-classy-settings/pull/22
[10:48:41] <mgedmin> I used to try to make python setup.py work, then decided it was an attractive nuisance
[10:50:03] <ronny> zahlman: eh i mean thats what i meant, you get eggs installed inside your project
[10:50:30] <ronny> zahlman: however if you are in a environment that has all needed packages, it just works out without installing any eggs there
[10:53:56] <zahlman> ... ok, so the idea with `setup.py develop` is that I can use the project code in other code (because it's aliased through the `site-packages` of whatever environment) without actually installing it?
[10:55:22] <ronny> zahlman scripts made by setuptools installations pin the exact versions of the belonging packages/requirements
[10:56:30] <tclugg> just throwing it out there, if anyone wants to meet up in Melbourne AU to discuss Python Packaging, I'll host an event if necessary. Or we can just crash the next MPUG meet. :-)
[11:03:15] <ronny> zahlman: setuptools scripts run a pkg_resoruces.require('packagename==exactversion') before requesting the script entrypoint
[11:06:35] <zahlman> ...ok. I have a project folder that contains setup.py, src/__init__.py, src/code.py, src/tests/test.py. I set up a venv and `python setup.py develop`; then start the venv python interpreter
[11:06:59] <zahlman> now I can `import code` fine, but not `import code.tests`, because "'code' is not a package"
[11:10:49] <zahlman> (I can `import tests`, but `from tests import __main__` fails; the relative import of `..code` no longer works)
[11:11:31] <zahlman> er, sorry, it's src/tests/__main__.py (for -m support) and src/tests/__init__.py (well, it didn't work with or without...)
[11:11:34] <ronny> zahlman: relative imports onlz work within a package - your tests folder is a new toplevel pacakge
[11:12:41] <zahlman> what I'm getting at is, why when I installed the project, did `code` not become a package, even though I have a `src/__init__.py`?
[11:16:00] <zahlman> ... I want to put all the code in src/, and when it's installed, be able to `import code` and also `import code.tests`, or something along those lines. What does the folder structure need to look like?
[11:17:26] <mgedmin> src/, src/code/, src/code/__init__.py, src/code/other_code.py, src/code/tests/__init__.py, src/code/tests/test_other_code.py
[11:18:35] <mgedmin> or you could use packages=find_packages('src'), that'll make the setup.py easier to cargo-cult to other projects
[11:18:51] <zahlman> so I can't get around that extra level of directories? ugh. the thing is that the logical name for the folder *containing* `src`, is also `code` (anonymized, but yeah)
[11:20:17] <zahlman> can I have a `tests` package that imports code *not* in a package? without hacking sys.path?
[11:20:53] <mgedmin> generally people who prefer to have an src directory also prefer to put the tests in a subpackage
[11:22:12] <zahlman> so if I have src/code.py, and then src/tests/test_code.py, how do I get the imports to work in the test?
[11:23:09] <mgedmin> also, your setup.py will have to be something like setup(python_modules=['src/code']) maybe? I've never seen that layout TBH
[11:24:10] <zahlman> what's the normal layout if you want to develop one or more modules that aren't a package, and also have tests for them?
[11:25:24] <mgedmin> I usually just put foo.py and test_foo.py at the top level, right next to setup.py
[11:26:43] <mgedmin> I can't imagine a use case for multiple top-level modules in one source distribution
[11:28:00] <mgedmin> which is strange, in retrospect: I like to install test subpackages when I use packages
[11:28:02] <zahlman> well... suppose it did, and I installed both your project, and another one that did the same thing
[11:28:44] <mgedmin> if I wanted to install the tests, I'd convert my foo.py and tests.py (or test_foo.py) into a foo/__init__.py and foo/tests.py
[11:29:14] <mgedmin> because installing a globally-importable 'tests' is the worst form of namespace squatting and I wouldn't be that impolite :)
[11:29:47] <mgedmin> I mean, if I decide to convert objgraph from a module to a package, I don't want all my user's code to break
[11:30:21] <zahlman> but suppose you were doing something from scratch. all your main code can reasonably fit in a single source file, but you also want to distribute tests.
[11:30:51] <mgedmin> to minimize my own effort I'd do the same thing I'm currently doing with objgraph
[11:31:33] <ronny> zahlman: for all my projects tests are shipped with an sdist, and wheels contain onlz the code
[11:33:16] <zahlman> hmm. this is making me want to go without a src/ folder unilaterally, and set up a package folder if necessary
[11:33:54] <zahlman> I guess it's unlikely that someone would want to do a single module with a whole package of tests, but it seems like it would still work?
[11:37:19] <zahlman> ionelmc: I'm developing something for starting new projects. the idea is to write up all the content for the initial commit, and then run a script and have it automatically transform into a 'project folder' with the appropriate folder structure, setup.py etc.
[11:38:02] <zahlman> I want to set up something that will make life easy in the long run, whether the project is a single module up to multiple packages
[11:38:29] <zahlman> well yes, the script sets up such a template. but I'm trying to figure out what it should consist of exactly :)
[11:38:31] <ronny> and well, with the current tooling it wont be sane to handle single file and package the same way
[11:38:55] <zahlman> I can detect whether the input is a .py file or a folder, and choose the appropriate template to start
[11:39:16] <ionelmc> zahlman: i have a template where the setup.py will collect whatever is in src, be it modules or packages
[11:42:56] <ionelmc> here it is https://github.com/ionelmc/python-nameless/blob/master/setup.py#L82-L84
[11:45:33] <mgedmin> I don't like project templates because they leave unused stuff; I'd rather cargo-cult working code from another package when I find that I start needing it
[11:46:11] <zahlman> (it does a little more than just templating; e.g. I'm going to have it shell a `git init` and `git commit -a` or something, if available)
[11:46:15] <mgedmin> I have all my projects checked out in ~/projects so that I can grep foo */setup.py to see where I've used some feature I find I need now elsewhere
[11:46:30] <ionelmc> mgedmin: right, and when you need to swithc from module to package then you're in a world of pain again
[11:46:57] <mgedmin> https://github.com/mgedmin/restview/commit/4a85be84c23341de58fdcec01c00e425803f4780 was pretty painless
[11:47:32] <mgedmin> hm, ok, the "make it actually work" part of the commit message hints at some pain that I may've forgotten
[11:48:08] <zahlman> find_packages() expects folders to contain __init__.py files to be discovered as packages, yes?
[11:48:15] <mgedmin> and, heh, some residual pain in https://github.com/mgedmin/restview/commit/89473b6a354c4c24d09bde9b88b36bf603f6aa5c
[11:49:46] <mgedmin> ionelmc, hacks in code that's not installed in user machines is allowable in my book :P
[11:49:55] <ionelmc> if i can do it in a setup.py with 2 lines of code you can certainly do it in gumbey
[11:50:53] <ronny> ionelmc: sorry, but it is, its a complete difference from the structural standpoint, and its not just 2 loc
[11:52:30] <ronny> ionelmc: this is volunteer work, if you expect me to sufer more than absolutely necessary for free i have no kind words :)
[11:56:27] <ronny> additionally any time i dont spend on strange features i dont want to maintain to begin with, i can spend with my fiancee or on fun programming things
[11:58:36] <ionelmc> zahlman: btw, there was this huge discussion about installing tests on the maillinglist: https://mail.python.org/pipermail/distutils-sig/2015-October/thread.html#27003
[11:58:38] <ronny> hmm, packaging is fun exactly when i finished the py.test release automation to the point where i can do a release in a second
[11:59:49] <zahlman> ...you know, it's probably cleaner to emulate gumby on the 'only supports packages' thing - i.e. if I'm given a single .py file, make a package out of it, by renaming to __init__, or making an __init__ that does a *-import on it, or something.
[12:02:50] <zahlman> project creation? I want this to be easy because I don't want others to be intimidated out of the entire packaging mechanism.
[12:02:57] <ronny> when starting a project, i want to need exactly 3 files, its initial code, its initial tests and its metadata
[12:03:40] <ronny> just atimating dropping in the whole surrounding details is problematic, because then he project is full of generated incomprehensible stuff
[12:04:30] <zahlman> but that stuff will have to be there eventually, 'generated' or no. and the point is to have files set up that are easier to work with, that the generated setup.py can then work with
[12:05:04] <ronny> mgedmin: for the initial commit i'd prefer a short description in the metadata over a half-done readme ^^
[12:05:07] <zahlman> right now I have a setup_config.py that I should be able to replace with json once I've done more thinking.
[12:05:57] <zahlman> and a setup.py that can automatically build manifest.in based off of `git ls-files`, stuff like that.
[12:06:56] <ronny> zahlman: if you use setuptools_scm you no longer need a manifest for file inclusion and you get the version from scm tags as well
[12:07:38] <ronny> zahlman: i dislike the idea that tools generate generated code into a scm tree so we can then version it
[12:08:14] <ronny> zahlman: yes, we are a bit more recent, 1.9 currently, 1.10 will happen this week
[12:08:43] <ionelmc> ronny: i was wondering, could you make setuptoolsscm just use a bogus version "99999.develpment" or whatever if there's no scm info available?
[12:10:01] <ronny> ionelmc: however 1.10 will include a way to override the scm info from the environment
[12:12:12] <ronny> d0p: support for python packages natively would be the job of that package manager, not pip
[12:12:49] <ronny> ionelmc: i wont add bogous versions, because then broken installs work, and broken installs are not acceptable
[12:15:03] <ronny> ionelmc: how about you extend the setuptools_scm_git_archive plugin to ask github for more details?
[12:19:49] <ronny> ionelmc: i refuse to put it in half-broken, because then im guaranteed to be the subject of support requests
[12:22:27] <zahlman> ok. I have a tests/ package with a __main__.py that defines a class with unit tests, and runs unittest.main(). So `python -m src.tests` (that packaging will change probably) works. but I'd also like to be able to `import tests; tests.run()` or something like that.
[12:25:08] <ronny> zahlman: supporting n options sometimes is just not sensible, limit them or you will create a hell for the users
[12:27:01] <zahlman> `from tests.__main__ import Tests; import unittest; unittest.main()` from the repl worked.
[12:30:23] <mgedmin> zahlman, maybe this will help you: https://github.com/gtimelog/gtimelog/blob/master/src/gtimelog/tests/__init__.py#L15
[12:30:59] <mgedmin> I didn't bother to automate the discovery of test modules since adding new ones is not a frequent operation
[12:32:44] <zahlman> I take it that `unittest.defaultTestLoader.loadTestsFromName(__name__)` is how `unittest.main` normally tries to discover tests?
[12:33:43] <zahlman> I wonder why what I'm doing doesn't work, though (__main__.Tests is a class derived from unittest.TestCase of course)
[12:39:42] <ionelmc> zahlman: it's not a proper src layout if you have a __init__.py file directly in there
[13:02:38] <zahlman> it seems the reason the gist wasn't working is that __init__.py doesn't actually represent a __init__ module; so I can't give it the Tests class as an attribute and then expect unittest to discover it as the 'current module'
[13:13:20] <zahlman> sounds right. I can, of course, also `cd src; python -m tests`, which I guess is more 'proper' in a way
[13:13:46] <zahlman> ok, now I'm getting places. but I also should really take a break to eat something :)
[16:29:33] <Mathieu_Du> Hi folks o/ I'm trying to figure out how to write a setup file to specify a subfolder in which to install a module
[16:30:14] <Mathieu_Du> My use case is that I have a core application, that will look in this subfolder to discover available extensions
[16:32:48] <Mathieu_Du> afaiu that's not the purpose of a local setup.cfg file, as "system-wide" config files will take precedence over these
[16:38:57] <Mathieu_Du> I'm also reading this : http://docs.openstack.org/developer/stevedore/essays/pycon2013.html#discovery
[16:40:24] <Wooble> (Flask stopped recommending them for extensions and I think their reasoning that they cause more problems than they solve was probably good)
[16:41:57] <ronny> setuptools namespaces where broken, python3 namespaces work only on later versions of python3
[16:49:43] <Mathieu_Du> and yeah regarding flask I have looked it up, their extension system is different from what I have in mind as I want extensions to be implicitly loaded
[16:50:28] <Mathieu_Du> my extensions then add argument groups to the main application argument parser
[17:58:17] <zahlman> ronny: suppose I'm using setuptools_scm and I want the program to display a message incorporating the version number on startup. Can I use the same get_version() for that? how exactly is it storing the information?
[18:06:58] <ronny> zahlman: you can instruct it to write a version file, or use pkg_ressoruces to get the distribution metadata
[19:03:45] <transit> I'd like to gently nag for this pip pull request to get merged: https://github.com/pypa/pip/pull/3210
[19:05:59] <transit> I'd like to make the corresponding change to dependency_links for setuptools as well. Anyone working on setuptools with an interest in subversion compatibility that I could ask questions of?
[19:31:15] <ronny> transit: wrt setuptools its best to drop jason a mail, replies may take a while due to his time limits