PMXBOT Log file Viewer

Help | Karma | Search:

#pypa-dev logs for Tuesday the 30th of June, 2015

(Back to #pypa-dev overview) (Back to channel listing) (Animate logs)
[00:59:48] <Nakato> lifeless: What do you consider of packages installed from requirements. Should those be treated as a user requested dependancy and thus not able to be constrained or able to be constrained.
[00:59:48] <Nakato> 'pip install -c constraints.txt -r requirements.txt'
[01:05:58] <lifeless> Nakato: they should have the same conflict ruleset.
[01:06:12] <lifeless> Nakato: specifically
[01:06:34] <lifeless> Nakato: if constraints.txt has foo==1.3 and requirements.txt has 'foo>=1.0'
[01:06:36] <lifeless> thats ok
[01:06:42] <lifeless> we take 1.3
[01:07:10] <lifeless> if requirements.txt has a vcs reference or something conflicting with foo==1.3, then its a problen
[01:07:13] <lifeless> Nakato: BUT
[01:07:29] <lifeless> Nakato: I wouldn't stress too much about fine tuning here - because i've got a branch that replaces all of this logic
[01:07:40] <lifeless> Nakato: (the resolver branch)
[01:07:48] <lifeless> Nakato: it will still need the cross-check vs editables I think
[01:08:22] <lifeless> Nakato: (if I thought you were working on do-be-deleted-tomorrow-code, I'd have said something)
[01:23:40] <Nakato> lifeless: Okay, then I'm completely lost. As from what I've gathered you want it to look like https://gist.github.com/nakato/502126093e36de231d3d ?
[01:23:58] <lifeless> heh, sorry.
[01:33:34] <lifeless> Nakato: https://gist.github.com/rbtcollins/da2f012deb459d9bc624
[01:33:53] <lifeless> Nakato: missing from your set of experiments was -e /path/to/requests
[01:34:04] <lifeless> Nakato: which is the one I think we should make fail today
[01:35:34] <lifeless> Nakato: note line 255 of req_set.py
[01:35:50] <lifeless> Nakato: we totally ignore the specifiers of the requirements (or CLI) version of a requirement
[01:36:27] <lifeless> we just toggle it from 'constrained' to 'installing'
[01:36:46] <lifeless> I guess I'm saying when we hit an editable thats already constrained
[01:36:57] <lifeless> I think that thats unusual enough to warrant an error
[01:36:59] <lifeless> IMBW
[01:37:19] <Nakato> lifeless: Okay, hit constrained editable, error out, ignore the rest for now.
[01:39:09] <lifeless> well, what do you think?
[01:39:12] <lifeless> dstufft: ^
[01:39:32] <lifeless> [dstufft will say 'and thats why features are hard, tarpit tarpit tarpit'] :)
[02:14:27] <tchaypo> lifeless: given that, to find a working lower bound, we’d need to iterate up the list of versions
[02:15:23] <tchaypo> it sounds to me like it makes more sense to put this logic in the tool that’s doing the iteration
[02:15:32] <lifeless> tchaypo: well no
[02:15:43] <lifeless> tchaypo: we'd just error on commits that break with the current lower bound
[02:15:47] <tchaypo> if pip had a —use-lowest-version mode, and we use that, and it fails, then we’re going to increase the lower limit and try again
[02:15:51] <lifeless> tchaypo: and let devs specify te lower bound needed
[02:16:22] <tchaypo> we could get the same behaviour by using the machinery we already have but setting the upper and lower bounds to the same thing (the current lowest bound) and increasing that as required
[02:16:56] <lifeless> tchaypo: except that that doesn't work when you have a resolver in play, because it will reject invalid combinations at that point
[02:17:04] <lifeless> tchaypo: and its pretty certain we have some of those
[02:17:26] <lifeless> tchaypo: so what I'd like is for generate-constraints to be able to do two runs, one for upper and one for lower, and then lower is what we test with
[02:17:32] <lifeless> (and upper) - run two test runs
[02:18:45] <tchaypo> I’m not understanding what the problem is with the resolver - if it can’t find a valid combination, isn’t that just confirmation that the lower bound is broken?
[02:19:20] <tchaypo> … but I was thinking of the case where we’re changing and testing just one requirement - so it’s easy to set just that one requirement to a lower bound and then let the resolver adjust everything else as required
[02:19:32] <tchaypo> you seem to be thinking of updating *all* of the lower bound at once
[02:19:39] <tchaypo> *bounds
[02:19:41] <lifeless> yes
[02:19:42] <lifeless> exactly
[02:21:20] <tchaypo> could
[02:21:36] <lifeless> we can't run 400x the test load
[02:21:42] <lifeless> to test each bound separately
[02:21:52] <lifeless> the point of lower bounds is to make upgrades more graceful for folk
[02:22:16] <lifeless> e.g. to allow them to choose to use 'pip install X' not always do 'pip install -U X'
[02:22:45] <lifeless> so its not any one thing that will be held back, its some random set (specifically the union of requirements that overlap in ranges)
[02:23:43] <tchaypo> So the idea is to say that “We know that anything before this versions is broken”, so that if the resolver has backtracked all they way down to that version number, it knows that it has no valid versions left and can error early?
[02:24:24] <tchaypo> or are we wanting pip to be able to selectively upgrade packages if the installed version is less than the lower bound, without the user needing to use -U?
[02:24:59] <tchaypo> If it’s the latter case - I imagine that if we were doing that, pip would want to upgrade that requirement to the highest feasible version, right?
[02:28:30] <lifeless> pip already does the latter case
[02:28:36] <lifeless> theproblem is that our lower bounds are too low
[02:28:46] <lifeless> because we've used something from a newer version of X
[02:28:50] <lifeless> but not specified that we did that
[02:29:03] <lifeless> and we do that because devs look at the docs online, and say oh look thats nice, and try ti, and it works
[02:29:12] <lifeless> but it works because they have the newest version of X present when they try
[02:29:15] <lifeless> and we CI with that too
[02:29:30] <lifeless> if CI failed because CI ran a parallel configuration with the oldest X's
[02:29:35] <lifeless> then they'd find out
[02:32:45] <tchaypo> What’s the current behaviour if I “pip install X”, and X requires Y>=2.0, but Y==1.9 is installed?
[02:38:31] <lifeless> as long as 2.0 or newer is in the index
[02:38:41] <lifeless> the highest version of Y is installed
[02:40:42] <tchaypo> does that mean we already have the behaviour we want, provided the lower bounds in the requirements are accurate?
[02:41:04] <tchaypo> … we just need to confirm that they are in fact accurate..
[02:41:29] <lifeless> tchaypo: and now you're back at the feature request :)
[02:43:07] <tchaypo> I’m imagining is that we run a test with the flag that makes pip use the lowest possible version for every requirement
[02:43:48] <tchaypo> I imagine the first thing that’s going to happen is that we’ll find out that X has Y>=1.5 but Z has Y>=2.0, so there’s going to have to be some tweaking to find the common lower bound
[02:46:31] <tchaypo> but ignoring that, assume we have a useful set of lower bounds, we’re going to start to see things like X starts using a feature that isn’t introduced until Y==2.3, so when that patch is submitted the tests fail and we have to update the lower bounds
[02:47:02] <tchaypo> wait - are we thinking of this as a gate job, or a periodic job?
[03:37:29] <lifeless> tchaypo: gate
[04:19:27] <Nakato> Pushed an updated PR that addresses editable packages specifically. Going to grab lunch.
[18:49:43] <lifeless> dstufft: how did the 7.1 release go?
[20:24:28] <lifeless> dstufft: tap tap tap
[20:26:12] <dstufft> lifeless: workign on some testing stuff real quick to try and get OSX testing, then releasing it
[20:27:34] <lifeless> dstufft: cool! Thanks thanks thanks.
[20:53:32] <dstufft> travis is going so slow
[20:59:48] <dstufft> lifeless: Nakato is https://github.com/pypa/pip/pull/2937 working now?
[21:00:51] <lifeless> dstufft: no, I think that should be a 7.1.1 / 7.2 enhancement
[21:01:01] <dstufft> okay
[21:01:17] <lifeless> I can draft a KNOWN ISSUE for the current thing if you want
[21:35:49] <lifeless> dstufft: I wonder
[21:36:10] <lifeless> dstufft: if we should ignore .egg's in the CWD (or more widely even) unless --egg or whatever the option is has been given
[21:36:25] <lifeless> dstufft: I see folk getting confused by easyinstall dropped eggs in their local dir on a regular basis
[21:36:47] <lifeless> 'why didn't pip install unittest2 work' -> because you had an egg dropped in . by easy-install
[22:04:07] <tchaypo> hrm. I thought http://git.io/vtSzU already had a bug open
[22:04:21] <tchaypo> "[pip] rouge8 opened issue #2943: Conditional requirements comparing 'python_full_version' marker break on Python 2.7.10"
[22:05:42] <tchaypo> ah, I’m thinking of https://bitbucket.org/pypa/setuptools/commits/e01e9a77fad5
[22:35:44] <dstufft> lifeless: setuptools 7 doesn't drop them in . anymore
[22:35:49] <dstufft> it drops them in .eggs/
[22:35:52] <lifeless> dstufft: yes, but noone has upgraded
[22:36:07] <lifeless> dstufft: everyone still has the setuptools their vendor put on the system
[22:36:29] <lifeless> dstufft: -and- they also still have the eggs in their directories
[22:36:32] <dstufft> maybe vendors should stop shipping ancient setuptools
[22:36:33] <lifeless> years old in cases
[22:36:40] <lifeless> hey, no argument here
[22:37:07] <dstufft> (yea yea, I know that me wanting them to do that isn't going to make them do it)
[22:37:42] <lifeless> :)
[22:37:46] <lifeless> anything i can do to help?
[22:38:25] <dstufft> we don't actually detect the .eggs, setuptools inspects the sys.path, so you'd have to drop "." off the sys.path, but I'm not sure that's a thing we want to do
[22:39:17] <dstufft> mainly because I don't know if people _want_ that behavior sometimes, like if they're doing something with PYTHONPATH=.
[22:39:19] <dstufft> i dunno
[22:39:19] <lifeless> dstufft: '.' isn't in the path when you run 'pip install', is it ? the path to the script gets added instead I thought
[22:39:36] <dstufft> it's in the path when you do python -m pip I believe
[22:39:48] <Nakato> lifeless: I've updated the PR, mind taking a look?
[22:39:49] <dstufft> which is what tox does afaik
[22:40:10] <lifeless> dstufft: its not
[22:40:25] <dstufft> which its
[22:40:28] <lifeless> pip list | head
[22:40:28] <lifeless> ['/home/robertc/.virtualenvs/test/bin', '/home/robertc/.virtualenvs/test/lib/python2.7', '/home/robertc/.virtualenvs/test/lib/python2.7/plat-x86_64-linux-gnu', '/home/robertc/.virtualenvs/test/lib/python2.7/lib-tk', '/home/robertc/.virtualenvs/test/lib/python2.7/lib-old', '/home/robertc/.virtualenvs/test/lib/python2.7/lib-dynload', '/usr/lib/python2.7', '/usr/lib/python2.7/plat-x86_64-linux-gnu', '/usr/lib/python2.7/lib-tk', '/home/r
[22:40:35] <lifeless> pip (7.1.0.dev0)
[22:40:39] <lifeless> I modified the pip entrypoint script to print sys.path
[22:40:42] <lifeless> its probably cut off
[22:40:51] <dstufft> lifeless: what about python -m pip
[22:40:52] <lifeless> but believe me, my cwd is not there
[22:41:05] <lifeless> oh yeah, python -m pip will have it
[22:41:18] <lifeless> let me check this mornings report
[22:41:36] <lifeless> dstufft: https://gist.github.com/jml/6bbd4a81c5232ea17870
[22:41:47] <lifeless> dstufft: invoked via $ pip install unittest2
[22:42:15] <lifeless> dstufft: so, python -m pip might have it, but its not a sufficient explanation for the symptoms :)
[22:42:28] <lifeless> Nakato: sure thing
[22:42:29] <dstufft> hrmph
[22:42:49] <dstufft> it's probably someting in pkg_resources
[22:43:14] <lifeless> Nakato: URL ?
[22:43:23] <Nakato> https://github.com/pypa/pip/pull/2937
[22:45:23] <dstufft> lifeless: I think we should probably do something, not entirely sure what. Main concern would be if someone has "." in sys.path on purpose, but I don't even know if that's really a reasonable use case (biggest thing I can think of that would use that is pip install -t, but I think that ignores installed anyways)
[22:50:26] <lifeless> dstufft: yeah, its tricky
[22:52:56] <lifeless> Nakato: reviewed
[22:55:19] <Nakato> lifeless: Thanks
[22:58:57] <dstufft> bleh
[22:59:01] <dstufft> my gpg key is on my old machine still
[23:00:16] <lifeless> tell me that you haven't given that to anyone ;)
[23:00:24] <dstufft> Nah
[23:00:37] <dstufft> it's just sitting across the room
[23:01:35] <dstufft> I hurt my back last night, so trying to think how to get it back over here so I can plug it back in heh
[23:02:22] <lifeless> ouch, you ok?
[23:02:59] <dstufft> yea, I'm just stupid
[23:03:48] <dstufft> there, made alyssa haul it over for me
[23:16:05] <tchaypo> ewww
[23:16:07] <tchaypo> i mean
[23:16:09] <tchaypo> oooooh
[23:17:50] <dstufft> lifeless: ^
[23:19:02] <lifeless> \o/
[23:19:30] <lifeless> time to go rip out the custom gnarly from my devstack patch
[23:46:48] <lifeless> dstufft: I presume the virtualenv release is imminent ?
[23:47:00] <dstufft> lifeless: once tests pass
[23:47:06] <dstufft> travis is going slow