pmxbot IRC Log Viewer

[17:07:40] <ronny> adamholmberg: a good waz to handle that wold be to just use extras, install the minimal set by default, and have a number of extras that just handle expensive requirement sets

[17:07:57] <ronny> adamholmberg: can you outline the dependencies and their purpose a bit more

[17:08:14] <ronny> adamholmberg: also note that with modern pip, its likelz a dependencz needs to be built only once

[17:10:07] <adamholmberg> It's an option to cythonize some of the modules in place. When active, Cython becomes a build_requires and a long build process happens in build_ext. When not active, I'm trying to avoid that build requirement, and not do the cythonizing.

[17:16:58] <ronny> adamholmberg: i see, then thats really tricky

[17:27:27] <adamholmberg> I was hoping it was not the most obvious thing. :) Didn't find a lot after reading the packaging guide. Right now I'm looking at customizing "install" to see if I can introspect extras

[17:33:01] <ngoldbaum> adamholmberg: so you want a way to switch back and forth between a pure python mode and a cython-accelerated mode?

[17:33:59] <adamholmberg> A way to control the way it installs (whether to do the expensive compilation, or just install the pure python artifacts).

[17:34:38] <ngoldbaum> i'm not sure if there's a good way to do that from pip, i've seen other projects do it by passing e.g. a --pure flag to setup.py

[17:36:11] <adamholmberg> yeah. That's the pattern I have now. It doesn't play well with external dependencies because https://github.com/pypa/pip/issues/1883

[17:36:25] <ngoldbaum> sure, there might not be an alternative though :/

[17:36:27] <adamholmberg> (pip passes --install-option to all deps)

[17:36:31] <ngoldbaum> maybe an environment variable?

[17:36:47] <ngoldbaum> or just check if cython is installed and always turn it on in that case

[17:37:09] <ngoldbaum> most of the time you probably want the speed from cython, except when debugging, which you could setup a virtualenv without cython for

[17:37:50] <adamholmberg> I have a lot of varied use cases, but definitely agree with that.

[17:38:18] <adamholmberg> For example, lots of installs in automated testing, where they don't care as much about raw throughput, but install times are important.

[17:38:28] <adamholmberg> Just trying to find a good way of controlling the install.

[17:38:51] <adamholmberg> Was hoping there was something in the tooling, but over-the-top with env vars might be fine.

[17:38:54] <adamholmberg> Thanks for the input

[17:39:17] <ngoldbaum> there might be something there, pip/setuptools are pretty complicated

[17:40:13] <adamholmberg> I've learned that. :) Also concerned with legacy versions in LTS distros. Don't want to get too tightly coupled.

[23:08:34] <glyph> So I'm trying to get my CI in a good shape for testing dependencies

[23:08:46] <glyph> I am making a "requirements" directory and putting a collection of .txt files in there

[23:08:51] <glyph> dstufft: ^

[23:09:12] <dstufft> sounds like someone is copying from pypa/warehouse

[23:09:29] <dstufft> that or we both did the same thing

[23:09:29] <glyph> dstufft: wholly my own invention actually, glad to hear I'm in good company :)

[23:09:34] <glyph> dstufft: I am trying to figure out what the right way is to make sure that requirements/production.txt satisfies setup.py

[23:09:44] <glyph> dstufft: specifically, in my 'tox' job

[23:09:52] <glyph> dstufft: is there a nice recipe for that?

[23:14:00] <dstufft> glyph: sec

[23:15:07] <glyph> also is it kosher for a requirements.txt to have a -r line in it to include another file? the paths will be properly relative?

[23:17:42] <dstufft> glyph: https://bpaste.net/show/724e3672ff5a

[23:17:45] <dstufft> does that help you?

[23:17:56] <glyph> dstufft: oof. maybe?

[23:17:59] <dstufft> pretend "twine" is the thing you're attempting to test

[23:18:09] <glyph> dstufft: right now I need to specify some python-version-specific stuff

[23:18:12] <glyph> dstufft: like um

[23:18:29] <glyph> dstufft: how do I say "if python 3 then depend on this otherwise don't" in a requirements.txt

[23:18:39] <glyph> more importantly: where is the documentation for that

[23:19:03] <dstufft> this; python_version >= '3' or w/e the env marker is for that

[23:19:34] <glyph> https://packaging.python.org/en/latest/search/?q=marker&check_keywords=yes&area=default

[23:19:55] <dstufft> https://pip.pypa.io/en/stable/reference/pip_install/#requirements-file-format

[23:20:29] <dstufft> which links to https://pip.pypa.io/en/stable/reference/pip_install/#requirement-specifiers

[23:20:45] <dstufft> glyph: environment markers only just recently (like... last week? maybe the week before) got actually standardized

[23:20:51] <glyph> dstufft: cool, this is the reference documentation I'm looking for

[23:21:02] <dstufft> the requirements.txt format is pip specific though

[23:21:06] <dstufft> so it's docs are there

[23:21:18] <dstufft> glyph: did I tell you there is like, a real technical writer helping out now?

[23:21:28] <glyph> dstufft: exciting!

[23:21:38] <dstufft> they are restructuring the packaging.python.org docs and writing new stuff and shit

[23:25:13] <glyph> dstufft: okay so now I'm having trouble where I have this:

[23:25:16] <glyph> [testenv:py34]

[23:25:17] <glyph> deps =

[23:25:17] <glyph> {[testenv]deps}

[23:25:17] <glyph> --requirement={toxinidir}/requirements/3.txt

[23:25:33] <glyph> dstufft: and [testenv]deps is

[23:25:37] <glyph> --requirement={toxinidir}/requirements/production.txt

[23:26:01] <glyph> dstufft: production.txt has 'dependency=<released version that only works on 2.7>'

[23:26:19] <glyph> dstufft: 3.txt has 'dependency=https://<git branch where version of dependency that works on py 3 is kept>'

[23:26:35] <glyph> dstufft: why isn't 3.txt's dependency superseding production.txt's

[23:27:00] <dstufft> does tox do it's own --requirement processing or does it pass it into pip

[23:27:13] <glyph> dstufft: I don't know that, that's why I ask this channel questions like this :-(

[23:27:28] <dstufft> I'm kind of thinking it does it's own, because I remember holger not wanting to import from pip because we don't make any promises about API compat

[23:27:50] <dstufft> but I'm not sure about that

[23:28:38] <dstufft> does 3.txt literally have the string 'dependency=https://<other crap' ?

[23:28:46] <glyph> dstufft: yes

[23:29:10] <dstufft> I don't think that's a valid syntax

[23:29:10] <glyph> specifically, https://github.com/hawkowl/treq/tarball/cb52811ec70c713537f3c0e3728440f69dcb4c8c

[23:29:17] <dstufft> and I'm wondering why it's not blowing up

[23:29:37] <dstufft> can you pastebin the entire 3.txt, either in here or in PM

[23:29:39] <glyph> dstufft: I put URLs into requirements files all the time?

[23:29:55] <dstufft> glyph: prefaced with dependency=?

[23:30:45] <glyph> https://gist.github.com/glyph/611da4e872846496b296

[23:30:58] <dstufft> oh, so it's just the URL

[23:30:59] <glyph> dstufft: oh, sorry, no.

[23:31:02] <glyph> dstufft: yeah just the URL.

[23:31:18] <glyph> dstufft: would putting "#egg=whatever" on the end of that fix my problem?

[23:31:36] <dstufft> that might confuse things because pip won't know what the name of that requirement is, and tox's caching might be skipping over it

[23:31:49] <dstufft> if that's the case, the #egg=whatever should fix it

[23:32:02] <dstufft> where it's like, #egg=twisted

[23:32:30] <glyph> blah

[23:32:34] <glyph> dstufft: "double requirement given"

[23:32:39] <dstufft> yea that makes more sense

[23:32:57] <glyph> dstufft: so I can't use multiple requirements files this way?

[23:34:02] <dstufft> what does production.txt have, is it "twisted==<someversion>" or is it just "twisted" or "twisted>="?

[23:35:27] <glyph> dstufft: twisted==someversion

[23:35:37] <glyph> dstufft: 'production.txt' is the output of 'pip freeze' on py27, mostly.

[23:35:49] <dstufft> glyph: ok, so you're legitly telling it to install two different versions of twisted at once ATM

[23:35:51] <glyph> dstufft: I would prefer to avoid repeating the same dependencies for 2.6, 2.7, and 3.4

[23:36:06] <dstufft> so um

[23:36:28] <glyph> dstufft: right but I'm telling it to install the special one after I tell it to install the generic one

[23:36:39] <dstufft> No you're not

[23:36:47] <dstufft> tox compiles that all down to a single pip invocation

[23:36:57] <glyph> dstufft: right, but arguments do come in an order

[23:37:08] <dstufft> so you're doing the equivalent of ``pip install twisted==1.0 twisted==2.0``

[23:37:16] <glyph> clearly that means 'twisted==2.0' :)

[23:37:21] <dstufft> which isn't satisfiable

[23:37:45] <dstufft> solutions are uh

[23:38:06] <glyph> I guess I'll manually split this up

[23:38:08] <dstufft> add ; python_version == '2.7' to your production.txt's twisted

[23:38:11] <dstufft> or

[23:38:45] <glyph> well, actually, it occurs to me that for twisted I can actually use the release version now ;)

[23:38:51] <dstufft> I think if you change "https://github.com/twisted/twisted/tarball/931370903d1a7ad7957901921e575b9bee68ead0" to -f "https://github.com/twisted/twisted/tarball/931370903d1a7ad7957901921e575b9bee68ead0#egg=twisted-theversionoftwistedyou'retryingtoinstall"

[23:38:52] <glyph> but I still have this problem with treq and klein

[23:38:53] <dstufft> it'll work too

[23:39:09] <glyph> dstufft: daaaaang you are an evil man, that is wonderful

[23:40:33] <glyph> dstufft: trying the one with the versions...

[23:40:52] <dstufft> Hopefully at some point we'll make this a lot cleaner and saner and then it won't rely on little tricks like that

[23:41:08] <dstufft> because having to ask me for this one weird trick in IRC doesn't scale very well :(

[23:41:12] <glyph> dstufft: wait, "-f"???

[23:41:24] <glyph> dstufft: and with actual quotes and stuff?

[23:41:41] <dstufft> er, I meant "-f https://..."

[23:41:44] <dstufft> without the quotes

[23:41:59] <glyph> okay well a run with the quotes is going :)

[23:42:08] <glyph> crap it looks like it works

[23:42:12] <glyph> craaaaazy

[23:42:32] <glyph> dstufft: so this is _not_ my understanding of what '-f' does.

[23:43:13] <dstufft> --find-links / -f adds additional locations to search, they could either point directly at an installable file or they can point at an HTML page (or a directory) and we'll search that

[23:43:47] <dstufft> you wouldn't need the #egg= bit I think if your URL was something more like https://.../Twisted-1.0.tar.gz

[23:44:18] <dstufft> but since your URL doesn't "look" installable, we'll assume it's HTML and retrieve it (and thens kip it because the mimetype)

[23:45:26] <glyph> dstufft: it doesn't "look installable" even with the egg fragment parameter?

[23:45:47] <dstufft> the #egg= bit makes it look installable

[23:45:53] <dstufft> I meant if you didn't include the #egg= bit

[23:46:04] <glyph> dstufft: gotcha.

[23:46:37] <dstufft> that logic is legacy from PJE/setuptools, it's sort of edge case but it's useful sometimes

[23:51:15] <dstufft> glyph: it's all working now?

[23:52:05] <glyph> dstufft: looks like!!

[23:52:55] <_habnabit> aaaaaaaaaaaaaugghhhhhhhhhhhhhhhh it's just one thing after another

[23:53:30] <_habnabit> now that i've dealt with the python 3 comedy of errors, pex requires setuptools<16, and i was trying to use environment markers with <, which isn't supported before setuptools 17.1

[23:54:20] <dstufft> _habnabit: sounds like you're doing a packaging.

[23:54:24] <_habnabit> i am trying

[23:54:27] <dstufft> why does pex require <16

[23:54:30] <dstufft> that sounds like a bad scene

[23:54:32] <_habnabit> i have no idea

[23:54:45] <dstufft> is it a bug or did they actually pin <16

[23:54:50] <_habnabit> it's pinned to <16

[23:54:53] <dstufft> o.O

[23:54:59] <_habnabit> i can't remember who hangs out in here and works on pex

[23:55:38] <dstufft> Yasumoto

[23:55:45] <_habnabit> yeah that sounds right

Log file Viewer

Help | Karma | Search:

#pypa logs for Wednesday the 2nd of December, 2015