Log file Viewer
#pypa logs for Friday the 26th of August, 2016
[07:01:07] <pombreda> ionelmc: I have been using a for quite a while the approach you blogged about wrt using 'src' as a top dir for code.
[07:01:08] <pombreda> (and also in https://github.com/ionelmc/cookiecutter-pylibrary/blob/72e5fb7bcef50df626650279664fc84f940d6af6/%7B%7Bcookiecutter.repo_name%7D%7D/setup.py#L93)
[07:01:08] <pombreda> I am wondering if this is still the simplest way to get this? Any simpler ways? :P
[07:03:05] <ionelmc> pombreda: afaik no. There's flit but it's limited - single package and wheel only
[07:04:41] <pombreda> ionelmc: I shall say the import parity point you had brought up in your article is really the killer feature there :P
[07:18:59] <mgedmin> and causes pain on windows due to too-deeply nested directories because you have to use setup_requires
[07:23:44] <pombreda> unrelated, about plugins (and setuptools entry points) ... I am looking for a good lib for this.
[07:24:41] <pombreda> and plugins written in any other language, that would accept a file path as an input and return a JSON payload on stdout and would be spawned in subprocesses... and in that case I would prefer a simpler yaml'ish or toml'ish manifest
[07:26:16] <pombreda> and I have been looking at some options for the regular Python side but was not convinced by any yet
[07:31:58] <pombreda> ronny: they receive a file or dir path and return nothing if they are not interested in that or return a JSON (or dict for pure Python) payload
[07:31:59] <pombreda> the payload should be a Package object (as in the draft here https://github.com/nexB/scancode-toolkit/blob/develop/src/packagedcode/models.py#L477 but that class structure and using schematics there is a test that I am not happy with)
[07:32:48] <pombreda> the purpose is discovering packages metadata in a codebase (packages in the general sense as in gem, npm, jar, etc) and returninig that in a structured way
[07:34:12] <pombreda> the app that can be plugged in simply walks the filesystem and would call every registered plugin on every paths, aggregating all the returned results from all the plugins
[07:36:17] <pombreda> ronny: let's ignore for now the fact that I would want to also have non-Python plugins.... that is kinda unique enough that I will have to roll it out
[07:39:02] <ronny> for example you could decalrea common executable prefix you look for in path, and a common cli interface where you input data via cli args/stdin and receive data in some kind of serialization (be it json or json-lines or whatever)
[07:42:09] <pombreda> that would work well for the non-Python side. I guess it could work also for Python too after all. Though I was thinking of having python plugins being privileged and have a function of theirs being called, rather than a process spawned
[07:45:02] <pombreda> ronny: actually your take makes more sense: I can expose another stuffs for Python, kinda like the difference between extensions and plugins in FF
[07:46:17] <pombreda> here my "extensions" would be subprocesses in any languages and "plugins" could come later for programmatic Python enhancements to the core code.
[07:49:30] <pombreda> I just need to define the plugins documentation/metadata, packaging and discovery conventions, find these in provided paths, register them, and call them afterwards
[07:52:42] <ionelmc> pombreda: there was some presentation or blog post comparing all the plugin systems, ahve you seen it?
[07:55:13] <ionelmc> there's this http://www.giantflyingsaucer.com/blog/?p=5858 but that's not the comparison i remember
[07:58:59] <ionelmc> pombreda: this is it http://docs.openstack.org/developer/stevedore/essays/pycon2013.html
[08:00:39] <ronny> pombreda: it mgiht be entirely possible that right now you are plaing a astronauts game of architecture ^^
[08:01:53] <pombreda> ronny: scancode is a tool that, well, scans your code to discover origin of thirdparty code
[08:03:02] <pombreda> The use case for plugins would be to discover packaged code that may exist in your code (say vendored npms, go packages, maven Jars, etc)
[08:04:04] <pombreda> and avoid having all that in a single app, but allow to extend to add for support for new or improved packages formats
[08:05:04] <pombreda> ronny: that's the use case. You use it to answer this question: in my codebase, what is the third-party code?
[08:05:15] <ronny> well, what about simply making all plugins python and providing a very simple tooling to have those pieces of code invoke subprocesses that will pass in data
[08:05:44] <ronny> (then you arent limited to stdio, but could pass in data via any means the target langauge supports
[08:07:10] <pombreda> ok, so there a plugin would always be wrapped up a Python package (and I can provide the boilerplate for that as example)?
[08:08:22] <pombreda> and can then be either pure Python or do whatever it wants otherwise such as spawning
[08:08:55] <ronny> for example if you say the tool itself is python 3.4, you can use asyncio jsut fine
[08:13:02] <pombreda> (I have a ruby one for gems and a java one for maven about ready to be pushed)
[08:14:55] <pombreda> ronny: now using this approach (aka every plugin is a Python package) I still need to either roll my own on top of setuptools entry points or pick a lib for this :P
[08:18:01] <pombreda> ionelmc: thanks. the article is dense but thorough. I will need some time to grok it all
[08:23:39] <pombreda> ronny: the cool thing about your suggestion to wrap it always in a python package is that I get Pypi and pip for free and this makes discovery and registration a non-issue: the plugin is installed or not :)
[08:55:27] <pombreda> mgedmin: "may be from your python script wrapper" ... true, may reusing something like this https://github.com/nexB/scancode-toolkit/blob/develop/etc/configure.py which has been my poor man's buildout replacement, spawning pips... I could get it to spawn gem, npm, etc if needed
[11:57:20] <hk3380> Is this possible?: I want the user to be able to clone a repository, `cd` into the directory, and run `$ pip install .` to install the package, its pypi dependencies AND its private github dependencies. (also the devs should be able to run `$ pip install -e .`)
[12:15:49] <hk3380> i have a setup.py and i tried all variations i could find on the internet, but none work
[12:16:42] <pombreda> Now if its deps are aper pypi and part other private repos to clone, you would be best off driving the install outside of the setup.py
[12:18:05] <hk3380> pombreda: i just asked the question on SO with a bunch of details if it helps http://stackoverflow.com/questions/39165976/how-to-use-pip-to-automatically-install-dependencies-that-may-be-private-repos
[12:22:40] <pombreda> hk3380: you are trying to mix too many things together. pip install . (with or without -e) is a shortcut to call setup.py
[12:24:43] <pombreda> - a simple setup.py with no dependency link, referencing your private stuffs by name (and version) in stall_requires
[12:24:43] <pombreda> - for you users, provide a pip requirements file that will eventually also contain a "-e .". That should be their only entry point
[12:25:49] <hk3380> pombreda: you mean the users will install with `pip install -r requirements.txt` and an entry in `requirements.txt` is simply `.` ?
[12:25:50] <pombreda> and in that requirement file you can list first your git-based reqs, then at the end your . or -e .
[12:26:35] <pombreda> nope, you would list in the req file your git reqs first, and other private reqs and then a .
[12:26:42] <hk3380> pombreda: that's a possible solution, but it fails if the user calls requirements from a different dir, and it makes it difficult to run with or without `-e` for devs/users unless you make requirements-dev and requirements?
[12:29:16] <pombreda> you could leave out the . or -e . from the reqs and let users add that to their pip command line
[12:33:31] <pombreda> my 2cents: too many combos: do not try to do everything at once all the time for all cases
[12:38:09] <pombreda> there are subtle differences between a setup.py and a requirements file: I like to think of a setup.py as a build script for one package and a description of a version ranges for dependencies that does not say HOW to concretely provision/fetch the deps. (and eventually does not/should not deal with installation at all)
[12:38:11] <pombreda> and requirements and pip args describe concrete deps with eventually with pinned versions describing how to fetch these deps, from where, and which exact version to use
[13:17:02] <hk3380> pombreda: but it's very common for people to use setup.py to install both the script and all PyPI dependencies, no? afaik, `pip install package` doesn't use requirements.txt directly
[13:19:56] <hk3380> pombreda: i guess, but it's hard to think of setup.py as just a build script which doesnt deal with installation, when this is what it's typically used for outside of my use case :P
[13:21:47] <pombreda> well that's really what it is. It knows about Pypi-like servers, but nothing else for remote deps (though I am sure there some eventual plugins for that, I would shy away from them)
[13:23:32] <pombreda> but setup.py (and setuptools) mission is not to install packages, only to build and specify deps in the large.
[13:24:04] <hk3380> i think i'll end up leaving all PyPI dependencies in requirements.txt, remove everything about the private package from setup.py, and specify in README.md that users need this private package which may be installed with `pip install git+https://github.com....`
[13:25:00] <pombreda> also, do not try to do in one scenario a dev and regular deployment scenario. The same way you use makefiles and spec files to build an RPM. But you install it with yum/dnf/etc
[13:25:57] <pombreda> think of the setup/pip dichotomy the same way and your quality of life will be vastly enhanced
[13:26:38] <pombreda> hk3380: whatever people do in development does not matter. Give them instructsion of what they should do
[13:28:22] <pombreda> IMHO in contrast to what you stated above, I would put all my deps in setup.py eventually with version ranges
[13:28:22] <pombreda> and provide a concrete requirements file with pinned versions including how to get you private git repos in there
[13:29:15] <hk3380> pombreda: yeah i keep the PyPI dependencies in setup.py, but if I also put the private ones, install would fail no?
[13:29:16] <pombreda> your approach to mix things in " i think i'll end up leaving all PyPI dependencies in requirements.txt, remove everything about the private package from setup.py, and specify in README.md that users need this private package which may be installed with `pip install git+https://github.com....`"
[13:30:08] <pombreda> the install will not fail because you instructions will be to pip install -r req.txt
[13:30:37] <pombreda> hence when times comes to evenatull run the setup.py, the deps are already satisfied and installed
[13:32:21] <hk3380> i don't think i'll put the . / -e . in the requirements cause it requires the user to run the file from the same directory, and it's weird to think that the package itself is a requirements for it. i could just have all non-private requirements in setup.py, all requirements (including private ones) in requirements.txt, and have the user do `$ pip install -r requirements; pip install .`
[13:32:24] <pombreda> so as I said one last time: keep all your deps in setup.py, provide lax enough version ranges
[13:32:24] <pombreda> provide the concrete versions (and if needed how to get to them for private ones) in reqs and pip args
[13:34:23] <hk3380> i guess the disadvantage is that setup.py won't make it clear that it's dependent on the private repository
[13:35:24] <hk3380> maybe i could add it there because `pip install -r reqs` will install the requirements already, but then the order in which the user calls the installations could break it
[13:36:23] <pombreda> then repeat after me: setup.py does not care where you deps come from, unless they are on a Pypi server
[13:36:55] <hk3380> pombreda: yeah, i don't want setup.py to direct to the resource or give instructions how to install it
[13:41:03] <pombreda> in setup you provide the name of people eventually the city and the street they live in
[13:41:03] <pombreda> in reqs you provide the exact street number and tell how to get there by bus, car, walking, etc
[13:43:10] <pombreda> you would not put in an address book the directions on how to get there from anywhere: these are very specific to where you are and your transportation
[13:44:26] <hk3380> pombreda: yeah but if i put a private repo in setup.py it fails so i can't refer to it at all
[14:30:50] <pombreda> you put in your setup.py the name of your python packages, public or privates as install requires
[15:41:06] <ngoldbaum> hmm, i wonder if rackspace getting bought out might have any impact on their hosting of pypi