Log file Viewer

[00:26:39] <lifeless> dstufft: when you're done with virtualenv, I'd love a quick chat around constraints

[00:26:56] <dstufft> lifeless: sure, mostly waiting on tests to run atm

[00:27:03] <lifeless> dstufft: https://github.com/pypa/pip/issues/2731

[00:27:20] <lifeless> let me explain the background

[00:27:29] <lifeless> devstack runs pip oh dozens of times

[00:27:40] <lifeless> minimally once per server + various other things like clients etc

[00:27:49] <lifeless> rearranging that to do Just One call would be hard.

[00:28:00] <lifeless> We don't want to install unused deps in a given test environment

[00:28:33] <lifeless> partly performance, but largely hygiene - if X isn't declared as used, we don't want X installed where it might be accidentally used.

[00:28:53] <lifeless> but we want to pin everything that we do use, transitively, down to ===.

[00:29:45] <lifeless> So my idea is to have a requirements file format thing called a constraints file that says 'if you install X, honour the specifiers here, but don't trigger the installation based on the constraints file'

[00:31:39] <lifeless> internals wise, for the current resolver, I was thinking of adding a flag to the req_install that says 'constraint_only', and make RequirementSet.has_requirement return False for those.

[00:32:18] <lifeless> Then, a little glue to consolidate the objects when adding a requirement that is already constrained, and we're away.

[00:32:31] <lifeless> With the same lack of care for conflicts etc that we have today.

[00:33:18] <lifeless> For future, we'd map constraint_only things to global constraints, which everything being added is tested against - the new code already has the basic concept intrinsically

[00:37:10] <dstufft> I have a hard time deciding how I feel about it, it's certainly something that should be doable, but I'm concerned that it's extra conceptual complexity to pip for a fairly niche use case (at least, I'm having a problem coming up with a use case outside of what openstack is doing). I'm not outright against it or anyhting, but I'm unsure if it's the right thing and if it's not, what's a better way to solve the problem that openstack is

[00:37:11] <dstufft> happening in a more generally applicable way

[00:37:54] <agronholm> dstufft: bugging you once again for the Python :: 3.5 specifier :)

[00:38:04] <agronholm> since no one else can add it AFAIK...

[00:38:23] <dstufft> agronholm: err, yea. give me a few to finish releasing things

[00:38:30] <agronholm> ok

[00:38:32] <dstufft> I have to fix the admin.py script in PyPI

[00:38:50] <dstufft> I broke it when I moved things to S3

[00:39:01] <dstufft> and that's apparently how we add classifiers

[00:44:03] <lifeless> dstufft: I think its pretty general already.

[00:44:37] <lifeless> dstufft: the key question is, if I write it (and fix bugs should they arise so long as I'm in the Python ecosystem :)) - is it something pip would be ok having in it.

[00:45:37] <lifeless> Or, should I do it outside of pip - which means writing a thing to take a constraints file and a pip command line, then filter the constraints based on the transitive reqs pulled in by the pip command line, then output a requirements file.

[00:45:46] <lifeless> Which is, like 90% of pip :/

[00:50:58] <dstufft> lifeless: I don't think a solution to that problem is something that should need to be maintained outside of pip

[00:51:35] <lifeless> ok

[00:51:47] <lifeless> so, I'm happy to do a different approach if we can come up with one

[00:52:02] <lifeless> like, I'm not stuck on the vector or anything

[00:52:02] <lifeless> but

[00:52:27] <dstufft> bleh

[00:52:30] <dstufft> have to cut a 7.0.3

[00:52:32] <dstufft> woo regressions

[00:52:38] <lifeless> wheee

[00:53:24] <lifeless> man

[00:53:27] <lifeless> I should have spotted that

[00:53:30] <lifeless> sorry

[00:53:44] <lifeless> need me to whip up a patch, or are you good?

[00:54:23] <dstufft> lifeless: just sent one

[01:07:02] <lifeless> dstufft: ok so constraints. I'll whip up a patch and some tests etc.

[01:07:21] <lifeless> dstufft: and we can see if something better is provoked in the ensuing discussion ?

[01:07:29] <dstufft> lifeless: sure!

[01:08:19] <dstufft> lifeless: I don't mean to sound down on the idea fwiw, I'm just unsure is all. Which likely means, unless someone has strong feelings against it, I'd be OK merging it

[01:09:11] <lifeless> kk

[01:17:32] <lifeless> dstufft: is -c available to use as an option ?

[01:17:43] <lifeless> I don't mean 'in the ode', I mean, does it have history

[01:17:48] <lifeless> (I checked the code :))

[01:18:20] <dstufft> not that i recall

[01:22:15] <tomprince> Is it sometihng that is worth a short option?

[01:22:34] <lifeless> I think its more important than requirements files are.

[01:22:41] <lifeless> since its the same but with extra flex.

[01:22:52] <lifeless> so, yes.

[01:31:37] <lifeless> huh

[01:31:45] <lifeless> we discard the comes_from stuff in process_line now

[03:18:27] <tomprince> dstufft: We've been getting intermitent pypi errors for the last few days.

[03:18:58] <tomprince> Looking at it, it seems to be coming from setup_requires with and old setuptools.

[03:19:34] <tomprince> We haven't had the opportunity to make our version numbers PEP440 compliant. :(

[03:19:49] <tomprince> http://build.clusterhq.com/builders/flocker-centos-7/builds/4038/steps/install-dependencies/logs/stdio for example.

[03:30:12] <lifeless> yay, constraints implemented.

[03:30:18] <lifeless> couple more tests and I can push it up

[03:45:47] <lifeless> dstufft: ^ tada

[18:10:50] <lifeless> dstufft: whem you get a chance - https://github.com/pypa/pip/pull/2857 - also whats your planned timeline for 7.1? I'm wondering if I should put effort into a custom virtualenv build for openstack

[18:12:42] <dstufft> lifeless: will take a look ina bit, as far as timeline goes. I haven't really thought about it yet. if you want it asap it probably makes sense to do custom build

[18:15:58] <lifeless> dstufft: not asap per se, got a bunch of parts to move together; but don't want to stall once they are together

[21:15:13] <tomprince> dstufft: http://build.clusterhq.com/builders/flocker%2Ffunctional%2Frackspace%2Fcentos-7%2Fstorage-driver/builds/399/steps/install-dependencies/logs/stdio

[21:15:37] <tomprince> We seem to be getting a lot of CI failures trying to install deps.

[21:16:18] <dstufft> tomprince: you're using an ancient pip aren't you

[21:16:47] <dstufft> pip/req.py, so it's at least 1.5 or older, calling get_opener so it's 1.4 or older

[21:17:30] <dstufft> tomprince: I see this says Rackspace, is this in the ORD data center by any chance

[21:18:13] <dstufft> If it's ORD, you might be getting hit with https://status.fastly.com/incidents/pwfklp1bjmt8

[21:18:15] <tomprince> I think whatever is in centos-7's virtualenv.

[21:18:19] <tomprince> I think it is dfw.

[21:20:35] <tomprince> dstufft: We've been getting them intermittently for several days now.

[21:24:21] <dstufft> tomprince: ok, a couple days probably isn't that then

[21:26:34] <dstufft> I'm not sure what would be causing that offhand, I haven't seen any other complaints from anyone else recently I don't think. trying to recall back to that version to think if we might have done something that'd break it

[21:28:53] <tomprince> Also ubuntu-14.04: http://build.clusterhq.com/builders/flocker%2Facceptance%2Frackspace%2Fubuntu-14.04%2Fzfs/builds/1210/steps/install-dependencies/logs/stdio

[21:34:00] <tomprince> Hmmm. happened a couple of times on the 25-28, and then a bunch of time 1-2.

[21:45:18] <tomprince> dstufft: Any thoughts? I have to go soon, for the evening, but any on how to narrow whats going on would be helpful.

[21:47:59] <dstufft> tomprince: I don't know :( the best I can think of is try yo reproduce it and print out the response to see what it might be

[21:51:41] <tchaypo> https://pypi.python.org/pypi/mkvenv seems a little redundant now

[21:52:18] <tomprince> Maybe I'll see if we can stop using distro virtualenv.

[21:52:47] <dstufft> tomprince: for certain error conditions, newer versions of pip will auto retry

[21:52:54] <dstufft> I want to say pip 6?

[21:52:58] <dstufft> I think it was

[21:53:20] <dstufft> tomprince: fwiw virtualenv supports being run by just unpacking the zip and executing virtualenv.py directly

[21:53:27] <dstufft> you don't have to install it or anything

[21:54:49] <dstufft> ubuntu 14.04 should have pip 1.5

[21:55:03] <dstufft> 1.5 is when we switched to requests, so requests itself won't be a fix to the issue :(

[21:55:34] <dstufft> tchaypo: part of the goal is to make things like that redundant :)