pmxbot IRC Log Viewer

[00:00:28] <dstufft> qwcode: do you have anything else you want done in https://github.com/pypa/pip/pull/2616 before merging?

[00:01:32] <lifeless> should --no-use-wheel disable autobuilding of wheels for installs ?

[00:01:43] <dstufft> lifeless: yes I think so

[00:01:50] <lifeless> or should it only disable accepting wheels from indexes

[00:02:02] <dstufft> I think it should do both

[00:02:10] <lifeless> k

[00:02:48] <dstufft> (honestly, long term I want to get rid of that option, it only really existed as the inverse of --use-wheel in 1.4, but we need a deprecation cycle and it acts as a useful way to disable this new thing)

[00:07:02] <qwcode> dstufft, I guess you saw my comment. my only bit was the docs/comments

[00:08:44] <lifeless> qwcode: I believe I did those

[00:09:42] <lifeless> qwcode: ah, I see more comments, let me reply

[00:10:28] <qwcode> lifeless, dstufft if you want to merge, that's fine... I guess I'm pretty particular about some of the wording. it's more about what's not said at this point. I can make edits later in another PR I guess

[00:11:11] <lifeless> qwcode: ah, I replied in the PR

[00:11:23] <lifeless> question for you guys

[00:11:37] <lifeless> currently non-editable VCS packages are downloaded to the build dir

[00:11:50] <lifeless> not the source dir, at least AFAICT. Is that ok?

[00:12:17] <lifeless> the upgrade logical around non-editables seems unneeded with the current status quo

[00:12:41] <qwcode> lifeless, oh, you're using "should" for the install order now. I though we concluded it was a practical approximation for now

[00:13:22] <qwcode> should is coming from what principle

[00:13:36] <lifeless> https://github.com/pypa/pip/pull/2616/files#diff-abf8962da0a79184dd2c4bf311fc9b57R611

[00:13:42] <dstufft> I think it's "should" never install A without B when A depends on B

[00:13:49] <lifeless> is what you were objecting to AFAICT; see if the words make sense to you :)

[00:13:50] <dstufft> currently this PR asserts that via ordering

[00:14:08] <dstufft> in the future we could assert that by imlpementing rollback support

[00:14:11] <lifeless> dstufft: well, I'm saying we shouldn't transition through an invalid state if we can avoid it

[00:14:26] <lifeless> dstufft: because folk may run commands concurrently with us doing the install

[00:14:33] <dstufft> yea

[00:14:37] <dstufft> that's a thing too

[00:14:41] <lifeless> I don't say that in anything in the commmit

[00:14:47] <lifeless> just noting my specific perspective

[00:16:40] <qwcode> dstufft, lifeless, how bout I just log a PR with my wording ideas after the merge

[00:16:49] <dstufft> that's fine with me

[00:16:52] <qwcode> dstufft, but yes to what you said about "should"

[00:17:41] <lifeless> k

[00:17:46] <lifeless> please cc me on it :)

[00:17:55] <dstufft> (the point lifeless made about not putting us in an "invalid" transition is a good point too, since there may be people accessing the site-packages while we're doing the install, even with rollback capability we probably don't want to install A before B when A depends on B)

[00:19:15] <qwcode> ok, noted about the temporary state during the install. hadn't considered that one

[00:20:41] <dstufft> at the end of the day, there's only so much we can do about making sure that concurrent access to site-packages isn't broken, but installing things in order seems like a simple enugh thing to do to help reduce problems with that

[00:20:52] <lifeless> right

[00:20:56] <lifeless> see under atomic :)

[00:23:08] <lifeless> honestly, this space is -well- trodden in the internals of yum/rpm/apt/dpkg - looking in their design guts is a great way to understand the problem space more broadly - and nearly everything they have to deal with so do we

[00:23:35] <qwcode> didn't you guys mention *concurrent installs* at some point... as a goal down the road... doesn't that break the "valid state" goal, or would that constrain the concurrency

[00:24:01] <qwcode> or did I imagine someone saying concurrent installs... maybe I'm just thinking of curdling

[00:25:46] <dstufft> qwcode: I mentioned about isntalling things in parallel, which we may or may not do. Honestly installing from a wheel in parallel is less useful than possibly building things in parallel

[00:27:19] <qwcode> yea, build in parallel

[00:27:59] <dstufft> which if we build everything first, and then install everything we can do that easily (and also reduce the chances of getting a partial install too, since most failures are going to occur in building)

[00:32:16] <lifeless> building wheels in parallel is less complex

[00:32:22] <lifeless> but still tricky to optimise right

[00:32:25] <lifeless> because IO

[00:32:27] <lifeless> service points

[00:32:28] <lifeless> disks

[00:32:44] <dstufft> yea, I'm not really worried about it right now, other than thinking about the fact we might do it in the future when designing things

[00:32:47] <lifeless> trivial example, something like numpy with lots of C code

[00:32:56] <lifeless> and a dozen little python deps

[00:33:23] <lifeless> the best strategy is serial, with numpy set to parallelise internally as it compiles

[00:34:26] <lifeless> if we tell numpy to parallelise and do more wheels in parallel, we thrash the CPU; if we don't tell numpy to parallelise internally we'll finish the little wheels and then have idle CPUs waiting for the non-parallel numpy to complete

[00:39:26] <lifeless> (sorry, scheduling is a bit of a pet of mine...)

[00:39:32] <dstufft> heh

[00:39:34] <dstufft> no worries :)

[00:39:42] <dstufft> I'll admit I've done about 30 seconds worth of thinking about it

[00:40:11] <lifeless> I'm soooo close to this working

[00:40:24] <lifeless> just need to figure out how to construct a Link by hand

[00:40:38] <dstufft> lifeless: if it's the wheel building thing, does it build then install, or does it build everything first and then do all the installs?

[00:41:27] <lifeless> builds everything

[00:41:38] <dstufft> awesome

[00:41:47] <lifeless> then installs everything, including anything not wheeled due to either being editable or failing to bdist_wheel

[00:41:55] <dstufft> sounds great

[00:42:12] <lifeless> I need to figure out the filename for the wheel that bdist_wheel builds

[00:42:16] <lifeless> thoughts?

[00:42:48] <dstufft> I'm not sure there's a great way to do that, though you can pass --dist-dir to put it in a particular directory where it'll be the only thing in there

[00:43:08] <lifeless> ok

[00:43:12] <lifeless> shall do that

[00:43:30] <lifeless> dist-dir or dest-dir ?

[00:43:38] <dstufft> dist-dir I think

[00:44:22] <lifeless> --bdist-dir ?

[00:44:30] <lifeless> an ho

[00:44:34] <lifeless> --dist-dir, -d

[00:44:39] <dstufft> --dist-dir (-d) directory to put final built distributions in

[00:44:41] <dstufft> yea

[00:46:11] <lifeless> is there a method to transform a path to a url ?

[00:52:25] <dstufft> path_to_url in pip.download maybe

[00:52:48] <lifeless> cool

[01:32:56] <lifeless> \o/ passess test

[01:32:58] <lifeless> SHIPIT

[01:48:13] <lifeless> dstufft: qwcode: https://github.com/pypa/pip/pull/2618 is now ready for review

[02:57:16] <dstufft> lifeless: it's got failing tests, multiple matrix items and doesn't look to be network related

[02:58:08] <lifeless> dstufft: nurts

[02:58:16] <lifeless> dstufft: I shall fix

[03:00:53] <lifeless> ah

[03:00:56] <lifeless> fallout from ensure_dir

[03:01:00] <lifeless> I'll revert that one case

[03:03:10] <lifeless> actually, I think I can roll forward. pushed up a new ev

[03:03:11] <lifeless> rev

[03:09:14] <dstufft> awesome

[03:09:19] <dstufft> lifeless: did you see the pep8 failures too?

[03:17:50] <lifeless> dstufft: I did, I fixed that

[03:17:59] <lifeless> dstufft: just a stupid 79 vs 80 char thing

[03:18:14] <lifeless> PEP8: how to make good code illegible:)

[03:26:40] <lifeless> dstufft: I think travis missed the push

[03:26:48] <lifeless> dstufft: could you tap it to retry ?

[03:26:58] <lifeless> or hmm

[03:27:01] <lifeless> I might be able to trick it

[03:28:24] <lifeless> yup, its running again

[04:13:59] <lifeless> oh, I missed a flake8 that doesn't fail locally

[04:14:08] <lifeless> apparently it hates on lamdbas

[04:15:15] <dstufft> specifically lambdas that you assign to a name

[04:16:12] <lifeless> ok, I do have a bunch of failures now

[04:16:15] <lifeless> its EOD

[04:16:20] <lifeless> but I'll pick this up first thing

[04:16:45] <dstufft> lifeless: sounds good!

[04:16:48] <dstufft> lifeless: thanks a lot :)

[04:16:55] <lifeless> only 7 tests failing

[04:17:16] <lifeless> if you have any suggestions on how to get more bw from Jason, tht might be useful next week

[04:17:34] <lifeless> right now, I'm -> supermarket

[06:27:29] <Arfrever> dstufft: Please forwardport packaging/specifiers.py part of https://bitbucket.org/pypa/setuptools/commits/7f3453a0063e1033dd4e927de0781c2e0968420d to https://github.com/pypa/packaging/blob/master/packaging/specifiers.py

[16:24:02] <DanielHolth> So are there many dependency cycles on pypi?

[20:32:49] <lifeless> DanielHolth: I haven't quantified it. I know there are some.

[23:14:11] <lifeless> ugh

[23:14:17] <lifeless> some of these failures are killing me

[23:14:24] <lifeless> (don't happen locally)

[23:14:31] <lifeless> virtualenv = <[ValueError("zero length field name in format") raised in repr()] SafeRepr object at 0x37cda28>

[23:14:34] <lifeless> ^ WAT

[23:17:15] <dstufft> lifeless: on python 2.6?

[23:17:42] <dstufft> sounds like you used "{}".format() on 2.6

[23:21:32] <lifeless> dstufft: thats in the repr of th virtualenv the test used

[23:21:38] <lifeless> [1;5D[1;5D[1;5D[1;5D

[23:41:43] <lifeless> dstufft: so for instance

[23:41:44] <lifeless> TestUpgradeSetuptools.test_from_setuptools_7_to_setuptools_7

[23:41:47] <lifeless> passes for me

[23:41:55] <lifeless> fails https://travis-ci.org/pypa/pip/jobs/56804331

[23:44:36] <dstufft> lifeless: tests/lib/venv.py:40: return "<VirtualEnvironment {}>".format(self.location)

[23:44:43] <dstufft> looks like that's the problem

[23:44:49] <dstufft> probably it's masking some real underlying problem

[23:50:08] <lifeless> dstufft: oh, its local, I assumed it would be in a system thing. shall fix inline

Log file Viewer

Help | Karma | Search:

#pypa-dev logs for Wednesday the 1st of April, 2015