pmxbot IRC Log Viewer

[00:34:34] <buck1> does the pip test suite hit pypi.python.org?

[00:34:44] <buck1> i'm having a lot of trouble running the full suite right nwo

[00:34:56] <dstufft> yes

[00:34:58] <dstufft> among othes

[00:35:12] <buck1> second question: is pypi having trouble atm?

[00:35:31] <buck1> i'm not able to complete a couple of these tests for the last few hours

[00:36:56] <buck1> tests/functional/test_install_user.py::Tests_UserSite::test_upgrade_user_conflict_in_globalsite

[00:36:56] <buck1> is trying, failing to install INITools

[00:38:44] <buck1> ... ugh tried on master branch, it works. caused by me apparently =X

[07:11:28] <dm7freek> I think I've found a bug on PyPI, am I in the right place?

[08:27:15] <dm7freek> Well fwiw, PyPI doesn't seem to allow login if your password contains a colon.

[11:35:04] <rchekaluk> Downloading distribute_setup.py times out, has there been some recent DNS change?

[11:35:12] <rchekaluk> $ wget http://python-distribute.org/distribute_setup.py

[11:35:12] <rchekaluk> --2014-09-16 11:29:24-- http://python-distribute.org/distribute_setup.py

[11:35:12] <rchekaluk> Resolving python-distribute.org... 88.191.140.69

[11:35:12] <rchekaluk> Connecting to python-distribute.org|88.191.140.69|:80... failed: Connection timed out.

[11:35:12] <rchekaluk> Retrying.

[12:01:15] <rchekaluk> #pypa-dev

[12:50:17] <tos9> rchekaluk: Possibly, but distribute is dead.

[14:40:36] <Remram> Hey guys! For some reason my package's description is not being rendered as RST, except I have pretty much the exact same readme for another and it works fine

[14:40:41] <Remram> https://pypi.python.org/pypi/reprozip (works)

[14:40:45] <Remram> https://pypi.python.org/pypi/reprounzip (doesn't)

[14:42:42] <mgedmin> Remram, there's a ReST error: Duplicate implicit target name: "reprozip".

[14:43:06] <mgedmin> use `... <...>`__ instead of `... <...>`_ for anonymous links

[14:43:11] <mgedmin> to avoid this

[14:45:49] <mgedmin> Remram, http://paste.ubuntu.com/8358348/ should fix it

[14:46:27] <mgedmin> (that was a shameless plug)

[14:50:23] <Remram> Anonymous links? Every link is by default an anchor?

[14:55:43] <mgedmin> http://docutils.sourceforge.net/docs/user/rst/quickref.html#hyperlink-targets

[14:56:23] <mgedmin> when you say `name <url>`_ it is the same as `name`_ and .. _name: url; when you do the same twice with the same name, you get a conflict

[14:56:46] <Remram> Right, it's not an anchor, it just has a name for the URL association

[14:57:12] <Remram> well that looks like the implementation is leaking more than anything else. And github renders it fine...

[14:57:34] <mgedmin> it's a warning and github hides those

[14:57:48] <mgedmin> PyPI enables strict mode and aborts on any warning

[14:58:00] <Remram> yeah but it doesn't give me the message either :(

[14:58:18] <mgedmin> it's not very friendly, no

[14:58:30] <mgedmin> that's why I use restview

[15:02:22] <Remram> Yep, looks like I'm gonna use this tool from now on before releasing stuff

[15:02:23] <[Tritium]> ...feature request for the next iteration of packaging tools... issue warnings about RST issues

[15:03:05] <Remram> my packages show up as intended now, thank you so much for your help!

[15:43:06] <DanC> I downloaded virtualenv-1.11.6 and used it to build a new virtualenv, but the pip there won't download any packages. I'm using python2.6 from SuSE SLES 11.

[15:43:19] <DanC> Could not fetch URL https://pypi.python.org/simple/wheel/: connection error: [Errno 1] _ssl.c:491: error:14090086:SSL routi

[15:43:20] <DanC> nes:SSL3_GET_SERVER_CERTIFICATE:certificate verify failed

[15:44:39] <Alex_Gaynor> dstufft: that should be using the certifi cert bundle right?

[15:45:03] <dstufft> yes

[15:45:39] <Alex_Gaynor> DanC: are you behind an HTTP proxy or some other MITM network device?

[15:45:47] <DanC> no, I don't think so.

[15:46:01] <DanC> how is pip supposed to find CA certs?

[15:46:12] <dstufft> normally it's packaged as part of it

[15:46:18] <DanC> part of pip?

[15:46:20] <dstufft> yes

[15:46:39] <dstufft> can you print the entire log of ``pip isntall -vvv wheel``

[15:46:46] <dstufft> install*

[15:46:58] <DanC> `find . -name '*.pem' -print` shows nothing (in virtualenv-1.11.6)

[15:47:23] <dstufft> it would be in pip

[15:47:57] <DanC> pip was installed by virtualenv; I thought virtualenv didn't download stuff; I thought it included pip and setuptools

[15:48:04] <dstufft> it does

[15:48:09] <dstufft> it includes pip*

[15:48:15] <dstufft> but it includes it as a package file

[15:48:19] <DanC> ah

[15:49:22] <DanC> vvv output: https://gist.github.com/dckc/e0273367ea05bc9bdcad

[15:50:03] <DanC> ok, I see lib/python2.6/site-packages/pip/_vendor/requests/cacert.pem

[15:51:42] <dstufft> DanC: can you verify that curl -I https://pypi.python.org/simple/wheel/ works?

[15:52:03] <DanC> > curl -I https://pypi.python.org/simple/wheel/

[15:52:04] <DanC> HTTP/1.1 200 OK

[15:52:04] <DanC> Date: Tue, 16 Sep 2014 15:48:47 GMT

[15:53:49] <dstufft> DanC: what about curl -I --cacert lib/python2.6/site-packages/pip/_vendor/requests/cacert.pem https://pypi.python.org/simple/

[15:54:19] <DanC> HTTP/1.1 200 OK

[15:54:19] <DanC> Date: Tue, 16 Sep 2014 15:51:07 GMT

[15:55:17] <dstufft> hrm

[15:56:27] <DanC> the example at https://docs.python.org/2.6/library/ssl.html#client-side-operation doesn't work

[15:56:40] <DanC> ah... /etc/ca_certs_file

[15:56:55] <dstufft> DanC: can you give me the output of openssl s_client -connect pypi.python.org:443

[15:57:28] <DanC> it prints stuff and then hangs; is that as expected?

[15:57:52] <dstufft> yea

[15:57:54] <dstufft> just ctrl c it

[15:58:42] <DanC> ok... I updated https://gist.github.com/dckc/e0273367ea05bc9bdcad to show openssl connect output

[15:59:00] <dstufft> DanC: ok, and can you give me date -u too

[15:59:12] <DanC> > date -u

[15:59:12] <DanC> Tue Sep 16 15:56:04 UTC 2014

[16:01:41] <dstufft> DanC: can you do ``openssl version`` too?

[16:03:15] <DanC> > openssl version

[16:03:15] <DanC> OpenSSL 0.9.8j-fips 07 Jan 2009

[16:04:05] <DanC> oh. just 324. :-/

[16:04:08] <dstufft> lol

[16:04:32] <dstufft> I don't know that it's related to openssl specifically since curl works

[16:05:17] <dstufft> can you try to pop open a python console, and do ``from pip._vendor import requests; requests.get("https://pypi.python.org/simple/")``

[16:06:01] <DanC> pip._vendor.requests.exceptions.SSLError: [Errno 1] _ssl.c:491: error:14090086:SSL routines:SSL3_GET_SERVER_CERTIFICATE:certificate verify failed

[16:06:19] <dstufft> ok

[16:06:42] <DanC> the https://docs.python.org/2.6/library/ssl.html#client-side-operation example works if I give it the cert file from pip

[16:07:38] <DanC> If I change verisign to pypi, it fails

[16:07:52] <dstufft> ok

[16:07:54] <DanC> #ssl_sock.connect(('www.verisign.com', 443))

[16:07:54] <DanC> ssl_sock.connect(('pypi.python.org', 443))

[16:07:56] <dstufft> so it's not pip specific

[16:08:03] <dstufft> or requests specific

[16:08:45] <DanC> meanwhile, I'm updating suse packages

[16:09:48] <dstufft> hopefully that fixes it

[16:09:52] <dstufft> cuz otherwise idk

[16:10:07] <dstufft> that should work fine with python 2.6

[16:10:10] <dstufft> might be a suse bug

[16:15:03] <DanC> no joy...

[16:15:11] <DanC> do I need to re-create my virtualenv?

[16:15:21] <DanC> does it have a copy, rather than a link, to the python interpreter etc?

[16:15:35] <dstufft> hrm

[16:15:43] <dstufft> I don't recall which it does by default

[16:15:46] <dstufft> you might try recreating

[16:15:47] <dstufft> can't hurt!@

[16:16:07] <tomprince> It has a copy (and needs to have a copy), I think.

[16:16:38] <DanC> with fresh virtualenv, I'm still losing. this is demoralizing

[16:18:24] <dstufft> DanC: nuclear option, if you install the development version of pip it has --no-check-certificates if I recall correctly, you'll be vulnerable to MITM attacks tho :/

[16:18:59] <DanC> I'm trying pretty hard to avoid that sort of thing

[16:20:52] <dstufft> DanC: what version of SuSE are you using

[16:21:07] <DanC> > cat /etc/SuSE-release

[16:21:07] <DanC> SUSE Linux Enterprise Server 11 (x86_64)

[16:21:07] <DanC> VERSION = 11

[16:21:07] <DanC> PATCHLEVEL = 3

[16:23:47] <dstufft> hrm

[16:23:55] <dstufft> not seeing anything specific to that

[16:24:09] <dstufft> wonder if it's the sha2 certificate

[16:24:29] <DanC> know anyplace else with a sha2 cert?

[16:25:02] <dstufft> https://www.digicert.com/

[16:26:50] <DanC> I get "certificate verify failed" at digicert too

[16:27:22] <dstufft> versign.com uses sha1

[16:27:29] <dstufft> I bet that's it

[16:28:20] <DanC> does python include its own SSL implementation? or does it link with something else?

[16:28:51] <dstufft> OpenSSL

[16:29:29] <dstufft> the fact you have a -fips version concerns me, fips generally means ancient and crippled

[16:29:45] <DanC> fips version? of what?

[16:30:29] <dstufft> openssl

[16:30:35] <dstufft> [12:00:10] <DanC> OpenSSL 0.9.8j-fips 07 Jan 2009

[16:30:40] <dstufft> fips is a government thing

[16:30:55] <DanC> I'm vaguely familiar with FIPS

[16:31:01] <dstufft> you can get your software certiified as fips compatible which means that you use only what the government declares is secure

[16:31:03] <DanC> federal information blah blah

[16:31:21] <dstufft> but once software is fips certified, if you change anything in it, bug fixes, new features, etc you have to get it recertified

[16:31:26] <dstufft> which costs a lot of money

[16:31:43] <dstufft> so openssl-fips is generally a lot older than openssl-not-fips

[16:32:12] <DanC> hmm... I'm pretty we picked up suse updates for heartbleed...

[16:34:04] <dstufft> hrm

[16:34:09] <dstufft> http://marc.info/?l=openssl-users&m=135355590501495

[16:37:01] <dstufft> hm

[16:37:04] <DanC> "If the library runs in FIPS-140-2 mode ..." -- /usr/share/doc/packages/openssl/README-FIPS.txt

[16:37:05] <DanC> ?1

[16:37:08] <DanC> ?!

[16:37:14] <DanC> *If*?

[16:37:21] <DanC> it doesn't say how to turn on/off the mode

[16:38:27] <dstufft> digicert is the company that signs pypi.python.org's cert

[16:38:31] <dstufft> try with this one https://cert-test.sandbox.google.com/

[16:38:36] <dstufft> that's signed by a diff company

[16:39:30] <DanC> lose

[16:41:08] <DanC> `openssl s_client -connect` doesn't check the cert?

[16:41:43] <dstufft> it does, but it failed because you didn't pass a file

[16:42:09] <dstufft> https://gist.github.com/dckc/e0273367ea05bc9bdcad#file-openssl-L74

[16:42:46] <dstufft> you can try openssl s_client -connect pypi.python.org:443 -CAfile path/to/certifi.pem

[16:44:33] <DanC> Verify return code: 0 (ok)

[16:44:57] <DanC> wierd... when I use the certs file from haxx.se, it loses

[16:46:36] <DanC> i.e. http://curl.haxx.se/ca/cacert.pem

[16:46:52] <dstufft> I've never used SuSE, is their patches against Python available online anywhere?

[16:48:08] <DanC> I hope so... but remember, this is SuSE enterprise, where we pay extra so that some of the source code is hidden and we can't use normal open source support channels. :-/

[16:50:09] <DanC> that's not source...

[16:51:30] <dstufft> DanC: if you have paid support you might try asking them, I'm really just guessing at this point. it appears that depending on how the OpenSSL library is called on your SuSE it gets support for sha256 certificates or not. Python 2.6 natively includes a call to OpenSSL_add_all_algorithms() which should enable sha256, but I don't know if SuSE removed that or if their OpenSSL requires something different or what

[16:52:00] <DanC> I can't be the 1st person to hit this. I'll try #suse

[16:52:13] <DanC> thanks for your help with diagnosis!

[16:53:24] <dstufft> DanC: fwiw sha256 certificates, while they've been techincally possinle for awhile, have only just recently started getting a big push to switch to them

[16:53:48] <dstufft> previously people were using sha1 to maximize compatability

[16:53:53] <dstufft> (and md5 before that)

[16:54:06] <DanC> sha2 and sha256 are not the same thing, are they?

[16:54:42] <DanC> "SHA-2 is a set of cryptographic hash functions (SHA-224, SHA-256, ...") -- wikipedia

[16:54:45] <dstufft> sha2 is the name for a set of algorithms

[16:54:50] <dstufft> sha256 is one of the sha2 algorithms

[16:55:26] <dstufft> microsoft formally declared they were removing support for sha1 certificates in their OS in 2016

[16:55:35] <dstufft> https://technet.microsoft.com/en-us/library/security/2880823.aspx

[16:57:42] <DanC> I'm starting to wonder if I want to use SuSE's python at all. I wonder if it's better to build from source. I could use 2.7, for one thing

[17:02:04] <dstufft> heh

[17:02:08] <dstufft> I use pyenv for that kind of thing

[17:16:41] <DanC> wierd... I can't get the ssl test to work at all on my Ubuntu box with python 2.7

[17:19:27] <DanC> er... not for verisign, anyway

[17:19:36] <DanC> but it works for cert-test.sandbox.google.com

[17:20:23] <DanC> and for pypi

[17:21:01] <apollo13> btw afaik if openssl is compiled weirdly, fips is automatically enabled

[17:23:26] <DanC> now it won't load cacerts.pem from curl.haxx. I'm going nuts.

[17:25:27] <DanC> cert-test.sandbox.google.com works with curl's certs but not with pip's.

[18:30:11] <buck1> ugh one of these pip tests deadlocks in my branch, but not if i make it more verbose, or if i break into a debugger :(

[18:31:09] <Yasumoto> #heisenbug >_>

[18:31:30] <Yasumoto> does it consistently deadlock?

[18:31:43] <buck1> yes.

[18:31:47] <Yasumoto> (maybe share a link to the branch with the diff?)

[18:32:01] <buck1> adding '-vvv' to the test "fixes" it >.<

[18:32:12] <tomprince> I know dstufft and Alex_Gaynor were running into something similiar.

[18:32:17] <buck1> i'll push up a WIP branch

[18:32:32] <buck1> i think there's a subtle bug in scripttest, but can't put my finger on it

[18:34:14] <dstufft> oh god this again

[18:34:26] <dstufft> is it the same test

[18:34:28] <dstufft> always

[18:38:37] <buck1> dstufft: Yasumoto: tomprince: https://github.com/bukzor/pip/commit/5155b0871871102fef802291b8b39804ef625132

[18:38:52] <buck1> git@github.com:bukzor/pip.git buck-refactor

[18:39:14] <buck1> to be clear, this isn't pull-request-worthy at all

[18:39:18] <buck1> but i am stuck on taht deadlock

[18:39:43] <dstufft> buck1: try commenting out the cache handler in pip/download.py

[18:40:49] <buck1> comment out the PipSession.cache class attr?

[18:42:56] <buck1> setting it always-true seems to make it not deadlock

[18:43:00] <buck1> err always-false

[18:44:05] <dstufft> are you running against the latest develop branch? or at least one that has reqeusts 2.4.1 or wahtever it is (pip/_vendor/vendor.txt)

[18:44:17] <buck1> latest as of ~5 dya

[18:44:22] <buck1> i'll pull

[18:45:00] <buck1> it seems like i'm merged up

[18:45:12] <buck1> * ea3252e - donald@stufft.io(5 days ago) (upstream/develop, upstream/HEAD, develop) Merge pull request #2029 from dstufft/cleanup-compat

[18:46:10] <buck1> requests==2.4.1

[18:47:21] <buck1> dstufft:

[18:49:24] <buck1> dstufft: why does adding -vvv help the cache not deadlock?

[18:50:30] <dstufft> if it's the bug I'm thinking of it's an uncollectable garbage cycle

[18:50:56] <dstufft> -vvv changes the order things happen in slightly, just slightly enough to make it go away

[18:52:47] <buck1> i didn't know there was such a thing

[18:52:56] <buck1> using the term garbage loosely?

[18:53:30] <dstufft> no

[18:53:36] <dstufft> the garbage collector in Python

[18:54:10] <buck1> i thought it handled cycles nicely now, except for maybe extension objects

[18:56:54] <buck1> dstufft: any suggestion on how to unblock my branch?

[18:57:09] <buck1> check in the cache=False ?

[18:58:06] <dstufft> cachecontrol is probably going to need to grow a __del__ method that delete's it's references

[18:58:47] <buck1> i added a bunch of prints to it, i don't see that it's used ever

[18:58:52] <buck1> i can try to write a del

[18:59:11] <buck1> that means blocking on upstream yes?

[18:59:20] <buck1> any policy on vendored forks?

[19:00:12] <dstufft> we don't do it, though elarson is generally very responsive

[19:00:50] <buck1> ok

[19:01:14] <buck1> doing a dumb foreach attr del self.__dict__[attr] didn't do it =/

[19:01:37] <buck1> am i right to say there must be a C-extention object involved?

[19:01:49] <dstufft> no

[19:02:47] <dstufft> drop a ``import gc; print(gc.garbage)`` as the last line of https://github.com/pypa/pip/blob/develop/pip/basecommand.py#L233

[19:04:01] <buck1> thanks

[19:21:56] <dstufft> buck1: you can also try to instead, stick gc.set_debug(gc.DEBUG_UNCOLLECTABLE) somewhere near the begining

[19:22:50] <buck1> dstufft: i'm suddenly not able to repro >.<

[19:46:55] <buck1> dstufft: got the repro again ( i was doing it wrong ), gc.set_debug didn't turn up anything

[20:06:37] <buck1> it's stuck in html5lib.parse !

[20:14:59] <buck1> it's stuck in html5lib.html5parser.getPhases which is a 2200+ line function...

[20:15:21] <buck1> wat

[20:27:52] <buck1> pinpointed the sticking point. a really benign looing method definition

[20:28:03] <buck1> if i stuck a gc.collect() just before it, it doesn't stick

[20:28:12] <buck1> dstufft: expected behavior? ^

[20:29:34] <buck1> gc.garbage shows empty-list at that point in the code

[20:32:26] <dstufft> oh

[20:32:42] <dstufft> if it's not hanging at the very end of Command.main() it's not the gc bug i'm used to

[20:33:01] <dstufft> we just recently updated html5lib, might be related to that

[20:33:58] <buck1> dstufft: http://paste.pound-python.org/show/PDN9K84souQLngMcHVa1/

[20:34:37] <buck1> new hypothesis?

[20:35:00] <buck1> if i put a gc.collect there though, it doesn't stick

[20:35:14] <dstufft> buck1: you could try to switch back to to the other html5lib version

[20:35:18] <dstufft> cd pip/_vendor

[20:35:21] <dstufft> rm -rf html5lib

[20:35:35] <buck1> it does seem to have something to do with garbage tho

[20:35:39] <dstufft> pip install -t . html5lib==1.0b3

[20:35:42] <buck1> i'm just surprised the list is empty

[20:37:54] <buck1> http://paste.pound-python.org/show/fWADJ9GLqCTwUlJG2Fal/

[20:50:02] <xafer> buck1, do you know what test gets stuck ?

[20:50:15] <buck1> yes, and which line

[20:51:38] <xafer> what test is it ?

[20:56:23] <buck1> xafer: tests/functional/test_install_user.py::Tests_UserSite::test_upgrade_user_conflict_in_globalsite

[20:56:40] <buck1> lots more information here: https://github.com/bukzor/pip/commit/6f1931d5c7a6a02d332c4bec2e5b5dbb0175bbfb

[21:23:03] <buck1> dstufft: finally got a printout of the garbage http://paste.pound-python.org/show/fyt2GoXD9J4xZKigMGT9/

[21:23:13] <buck1> ... what do i do with it? =X

[21:26:58] <xafer> with only gc.set_debug(gc.DEBUG_UNCOLLECTABLE) ?

[21:27:16] <buck1> DEBUG_LEAK

[21:27:36] <buck1> my understanding is gc.garbage is always empty unless you've set gc.SAVE_AL

[21:27:47] <buck1> SAVEALL*

[21:28:52] <xafer> except for uncollectable objects

[21:29:30] <buck1> ?

[21:29:45] <buck1> gc.DEBUG_SAVEALLWhen set, all unreachable objects found will be appended to garbage rather than being freed.

[21:30:18] <buck1> erk ok it's the difference between unreachable and uncollectable

[21:31:05] <buck1> if i set only debug_uncollectable, i get nothing.

[21:33:32] <buck1> xafer: actually i get exactly one: gc: uncollectable <list 0x7f078e1fd638>

[21:35:12] <xafer> would be nice to know what's inside :)

[21:35:59] <buck1> how to? :(

[21:36:32] <buck1> xafer: would that conceivably block the interpreter?

[21:37:13] <buck1> i dont see the connection between uncollectable garbage and deadlock

[21:37:22] <buck1> dstufft: ^

[21:39:49] <buck1> xafer: any clue why i get that gc: printout, but nothing in gc.garbage?

[21:40:21] <buck1> ... disregard, it's because i check the gc.garbage too early

[21:42:26] <xafer> not sure :o

[21:42:54] <xafer> did you try faulthandler.dump_traceback_later() to see what it is doing ?

[21:43:51] <buck1> xafer: I haven't heard of that

[21:44:02] <xafer> https://docs.python.org/3.3/library/faulthandler.html

[21:44:11] <xafer> and there is a backport for 2.7

[21:44:46] <buck1> Successfully installed faulthandler

[21:47:09] <buck1> xafer: ugh, any idea how to get that available in this pip test...

[21:47:13] <buck1> a bit meta

[21:49:34] <xafer> faulthandler.dump_traceback_later(file=somewhere) doesnt work ?

[21:49:47] <buck1> xafer: I have to have faulthandler installed

[21:49:51] <buck1> for pip to import

[21:49:56] <buck1> except this is a pip test, in a clean venv

[21:50:01] <buck1> so i'd have to use pip to install it...

[21:50:07] <buck1> or something else?

[21:50:34] <xafer> how do you launch the test ?

[21:50:48] <xafer> tox ? py.test ?

[21:51:15] <buck1> py.test

[21:51:39] <buck1> the venv in which the pip subprocess runs is part of the test fixture

[21:52:18] <buck1> ah! seems to be an infinite loop in cachecontrol getattr

[21:52:36] <buck1> dstufft: xafer: https://pb.yelpcorp.com/98939

[21:53:00] <buck1> xafer: i added script.pip('install', 'faulthandler') to the test, and did a try ImportError around its usage

[21:53:03] <buck1> gross but worked =/

[21:53:57] <xafer> the pb.yelpcorp.com link does not seem to work :s

[21:55:35] <xafer> pip/_vendor/cachecontrol/filewrapper.py file ?

[21:55:39] <buck1> heh

[21:55:49] <buck1> infinite recursion on _CallbackFileWrapper__fp

[21:56:08] <buck1> xafer: repaste http://paste.pound-python.org/show/KVcnfXCQv3yb66Xru01B/

[21:56:21] <buck1> getting close!

[21:56:24] <xafer> :)

[21:56:26] <buck1> not a gc issue looks like maybe

[21:56:37] <xafer> was the same issue we had some time ago :p

[21:56:41] <buck1> faulthandler was a hot tip, thanks

[21:56:46] <xafer> cf http://stackoverflow.com/questions/24522467/python-program-hangs-forever-when-called-from-subprocess

[21:57:09] <buck1> explains the 100% cpu nicely too

[21:57:16] <buck1> althought i'd expect it to stackoverflow pretty quick

[21:58:04] <buck1> bug is they're doing self.x in __getattr__ and x isn't set

[21:58:10] <buck1> derp

[22:01:09] <buck1> it must be during __del__ or something?

[22:01:21] <buck1> i dont see any state where that attr isn't set

[22:03:12] <buck1> dstufft: xafer: here's the fix http://paste.pound-python.org/show/B3L5uYv1hejp2bUbGpU3/

[22:03:18] <buck1> any idea how i'd regression-test it?

[22:09:37] <xafer> well the change would have to go to https://github.com/ionrock/cachecontrol

[22:10:51] <buck1> xafer: that's the answer to "where whould i submit a patch?"

[22:10:57] <buck1> my question was harder imo

[22:10:57] <xafer> but since the cachecontrol version vendored in pip was bumped to 0.10.3 6 days ago, I guess we should look at the commit

[22:12:37] <xafer> between 0.10.2 and 0.10.3

[22:14:30] <buck1> xafer: do you understand how __getattr__ is getting called during gc, after the self.__dict__ has apparently been cleaned up?

[22:14:43] <buck1> or do i still not understand wwhat's happening

[22:21:24] <xafer> no idea :-/

[22:23:44] <xafer> but I guess this is the sequel of an other deadlock mentionned in https://github.com/ionrock/cachecontrol/issues/39 :o

[22:26:07] <buck1> xafer: this is where the bad getattr originates http://paste.pound-python.org/show/9sub355i3KZ8NOOfAXiV/

[22:26:14] <buck1> doesn't seem to be during gc at all

[22:27:44] <buck1> the stack does seem to jump

[22:29:03] <buck1> .... that's the wrong stack

[22:29:37] <buck1> no it's not. sigh.

[22:30:08] <xafer> (hole in traceback can come from __tracebackhide__ = True )

[22:30:39] <buck1> xafer: or from a gc cycle?

[22:30:49] <buck1> err collectino

[22:31:42] <xafer> well if your last stack trace is correct the gc doesn't seem guilty

[22:32:42] <buck1> xafer: i'm asking if this stack could happen during gc

[22:32:56] <buck1> given that one of the lines doesn't seem to call the next

[22:33:29] <xafer> that's what __tracebackhide__ = True is for :)

[22:34:45] <xafer> what line is off for you ?

[22:35:38] <buck1> from _abcoll to response.py

[22:35:51] <buck1> 35 to 37

[22:36:16] <buck1> the line at 35 is wrapped by if isinstance(x, Mapping)

[22:36:48] <buck1> is __tracebackhide__ a cpython thing? smells like a py.test thing

[22:37:22] <buck1> yea it's a pytest feature

[22:37:26] <buck1> i'm not using pytest here.

[22:38:35] <buck1> so the hypothesis would require that gc is inspecing response.closed

[22:38:49] <buck1> but response.closed is a property.

[22:40:58] <buck1> xafer: dstufft: any idea ionrock's freenode handle?

[22:41:42] <xafer> nope

[22:42:29] <xafer> and I'm off, looking forward for a nice explanation tomorrow :-]

[22:42:56] <buck1> xafer: thanks for the hand =/

[22:45:29] <xafer> buck1, glad I could help :)

[22:58:03] <dstufft> buck1: could be a loop

[22:58:15] <buck1> ah ok yes.

[22:58:29] <buck1> dict is chosen as first in the loop bc it's a builtin type with no __del__ ?

[22:58:32] <dstufft> since filewrapper dispatches to the underlying http response

[22:58:46] <dstufft> could be something that's getting into an infinite loop

[22:58:52] <dstufft> when it's trying to close the http response

[22:59:23] <buck1> whatever is touching response.closed doesn't appear on the stack

[22:59:28] <buck1> i had assumed it was gc itself

[23:03:05] <buck1> dstufft: https://github.com/ionrock/cachecontrol/pull/41

[23:03:27] <dstufft> response.closed comes from uh

[23:04:25] <dstufft> https://github.com/shazow/urllib3/blob/master/urllib3/response.py#L293-L296

[23:05:47] <buck1> dstufft: but close isn't on the stack

[23:06:09] <dstufft> close will be called in __del__

[23:06:26] <buck1> i don't see a del

[23:06:31] <buck1> .. oh this is IOBase

[23:07:02] <buck1> i imagine iobase references .closed in its __del__

[23:07:19] <buck1> dstufft: anyhow, can i get your +1

[23:07:40] <dstufft> iobase calls close() which references .closed

[23:08:13] <buck1> close would be on the tack at that point tho yes?

[23:12:33] <dstufft> buck1: if it's being called in a __del__ method... all bets are off

[23:13:01] <buck1> that is my feeling, but i haven't dealt with such stuff much

[23:23:01] <buck1> dstufft: def __del__(self): pass fixes it as well -.-

[23:32:59] <dstufft> buck1: lol

[23:33:05] <dstufft> wonder what the downside is..

[23:33:09] <buck1> dstufft: http://paste.pound-python.org/show/j4JF9UxGGhfmGFFuNgUv/

[23:33:24] <buck1> i really can't understand this stuff

[23:33:33] <buck1> note that it's the same object

[23:33:41] <buck1> was quite hard to kill actually

[23:34:15] <dstufft> buck1: can you print((self, self.__fp)) there

[23:35:09] <buck1> ah ok

[23:35:32] <buck1> i expect it to error out after the first one tho

[23:36:42] <dstufft> I'm wondering if self.__fp is a callbackwrapper which itslef has it's own __fp

[23:37:22] <buck1> dstufft: yea only the first __fp works

[23:37:35] <buck1> it really does clear it, and self.__fp does infinite recursion from ther on

[23:37:58] <buck1> self.__fp is httplib.HTTPResponse

[23:38:34] <dstufft> buck1: maybe __getattr__ needs a check for if name not in {"__fp", "__callback", "__buf"}

[23:38:59] <dstufft> I'm just totally guessing that this point

[23:39:08] <dstufft> there's got to be an underlying cause

[23:39:11] <buck1> i think my solution is th emost direct

[23:39:22] <buck1> i can't regression test the __del__ solution

[23:39:49] <buck1> i'll unmangle the __fp and hopefully everyone's happy

[23:40:11] <dstufft> the thing is I don't understnad why your thing works

[23:40:21] <buck1> why wouldn't it

[23:40:43] <dstufft> it should be the same thing as self.__fp

[23:41:13] <buck1> nope because object.__getattribute__ throws attributeerror

[23:41:22] <buck1> while self.__fp calls self.__getattr__

[23:41:29] <buck1> recursing

[23:41:57] <buck1> difference between AttributeError and StackOverflowError

[23:45:55] <dstufft> ok, so personally I think doing this: https://bpaste.net/show/2c5beddeef13 is cleaner, it makes it more obvious what you're doing. The only time __fp should ever make it into __getattr__ I believe is after __fp has been deleted.

[23:46:28] <dstufft> buck1: you might be able tod elete this too

[23:46:28] <dstufft> https://github.com/ionrock/cachecontrol/blob/master/cachecontrol/filewrapper.py#L47-L52

[23:46:51] <buck1> but is it covered by test =/

[23:47:00] <buck1> doubtful

[23:47:52] <dstufft> your test shoudl cover it the same

[23:48:03] <buck1> i meant the deletion bit

[23:48:06] <dstufft> oh that

[23:48:11] <dstufft> yea doubtful

[23:48:11] <buck1> i agree the test for == '__fp' seems better

[23:48:25] <dstufft> the pip test suite is pretty good at exposing it sooner or later

[23:48:51] <buck1> i'd rather not touch >.,

[23:49:28] <dstufft> probably we can convince elarson to make a release once there's a solution that's ready for merge

[23:50:37] <buck1> dstufft: yea i can't test for the __name

[23:50:42] <buck1> have to be the _Mangled__name

[23:51:15] <elarson> dstufft: do we have a proposal for a better test to be sure the file has been consumed before caching?

[23:51:24] <buck1> at which point mine is just the lbyl version imo

[23:52:05] <buck1> or.. yours is

[23:52:47] <buck1> dstufft: would have to be http://paste.pound-python.org/show/j4JF9UxGGhfmGFFuNgUv/

[23:52:57] <buck1> still like it better?

[23:53:19] <dstufft> elarson: not that i'm aware of, currently it looks like the wierd failures we've been having are coming from Python deleting the __fp attribute, and then trying to access another attribute on the class which calls __getattr__, which references the now gone __fp, which since it's gone will call __getattr__ which references the now gone ... etc etc

[23:53:49] <dstufft> buck1: is that the right paste?

[23:53:53] <dstufft> not sure I understand

[23:54:00] <buck1> that is the wrong paste

[23:54:09] <buck1> http://paste.pound-python.org/show/7g8BZpoj8Njf8IbxwylJ/

[23:54:16] <elarson> dstufft: ah. bummer :/

[23:54:39] <dstufft> stupid munged names

[23:54:45] <dstufft> we could always just unmunge them

[23:54:50] <buck1> that was my suggestino

[23:54:57] <buck1> but truly i dont believe there's a problem here

[23:54:59] <dstufft> make them _cache_control_fp or something

[23:55:11] <buck1> that's not actually different..

[23:55:26] <buck1> I cuold just make it explicitly munged :P

[23:55:37] <buck1> self._CacheControl__fp = fp

[23:56:11] <dstufft> *shrug* I don't personally like referencing the munged names explicitly, it feels bad to me, I'd rather explicitly munge it then reference it as __fp sometimes and __Whatever_fp other times... but it's elarson's project and that would be his call :)

[23:56:28] <buck1> elarson: comments on pull/41?

[23:57:17] <dstufft> I think at the least it needs a comment to explain why

[23:57:21] <dstufft> :)

Log file Viewer

Help | Karma | Search:

#pypa logs for Tuesday the 16th of September, 2014