pmxbot IRC Log Viewer

[09:38:58] <mitsuhiko> dstufft: why would the scratch folder disappear?

[09:39:03] <mitsuhiko> i'm so clueless

[13:20:56] <dstufft> mitsuhiko: not sure yet, didn't get a chance to look closer.. I've been trying to debug that subprocess hanging thing that your PR got (as well as many other PRs)

[13:21:56] <mitsuhiko> dstufft: i just wonder if you know why scratch dir might be deleted

[13:22:00] <mitsuhiko> i can't see how my change would do that

[13:22:03] <mitsuhiko> wonder if scripttest does it

[13:23:45] <dstufft> mitsuhiko: Not off the top of my head, but scripttest is as janky as pip is and the tests are brittle in that way. We're slowly moving towards more unit tests where we can, but a lot of it involves refactoring pip too so that we can actually test things in isolation.

[13:26:08] <mitsuhiko> i just notice that my script does not delete the scratch folder

[13:26:15] <mitsuhiko> so my theory is that scripttest throws it away somehow

[21:57:49] <radix> does pypi allow modiying already uploaded packages?

[21:57:55] <radix> like deleting them or reuploading them

[21:59:06] <agronholm> radix: reuploading them where?

[21:59:19] <agronholm> and no, it does not modify anything

[22:00:27] <dstufft> radix: it allows you to delete them and reupload them yes

[22:00:32] <radix> :)

[22:00:33] <dstufft> radix: doing so makes you a bad person though ;P

[22:00:48] <radix> dstufft: I'm actually thinking of the threat model where someone's pypi account is compromised.

[22:00:49] <dstufft> I want to get rid of that at some point

[22:00:55] <dstufft> but last time I tried people cried about it

[22:01:14] <radix> at first I was thinking just pinning a version should make us secure from those kinds of attacks, but obviously not if you can change the contents of a version.

[22:01:35] <dstufft> well

[22:01:44] <dstufft> even if you couldn't change the contents

[22:01:48] <dstufft> that doesn't make you secure

[22:02:04] <dstufft> unless you upload every single filetype that the installer supports

[22:02:11] <dstufft> for every python version

[22:02:17] <radix> huh... I don't think I follow?

[22:02:41] <dstufft> so pip installs sdists and wheels, so lets say package foobar uploads foobar-1.0.tar.gz

[22:02:49] <radix> oh, I guess you're saying I could have a deployment that pins foo==1.0, which happens to be using a .tar.gz, but then someone could come along and upload a hacked .whl

[22:02:50] <agronholm> ah I missed the "allow" word in the original question

[22:02:54] <dstufft> yes

[22:02:57] <dstufft> that's what i'm saying

[22:02:57] <radix> ok.

[22:03:01] <radix> good point :)

[22:03:09] <dstufft> radix: look at peep

[22:03:22] <dstufft> https://warehouse.python.org/project/peep/

[22:04:46] <dstufft> package signing (once we get that) will protect against compromised PyPI credentials and/or compromised PyPI itself (to some degree, there are caveats here), and peep (which we hope to add similar to pip proper) protects agianst basically everything

[22:53:28] <mitsuhiko> dstufft: got some more info for this?

[22:53:41] <mitsuhiko> dstufft: i assume it's because of the following

[22:53:46] <dstufft> https://stackoverflow.com/questions/24522467/python-program-hangs-forever-when-called-from-subprocess

[22:53:50] <mitsuhiko> a newstyle class without __del__ will be cleaned up early

[22:54:03] <mitsuhiko> when it has a __del__ the cleanup process is potentially delayed until later

[22:54:08] <dstufft> interesting

[22:54:12] <dstufft> I bet that is it

[22:54:17] <dstufft> it is a new style class

[22:54:57] <mitsuhiko> dstufft: os._exit does not invoke finalizers btw

[22:55:51] <dstufft> yea, it doesn't hang if I use os._exit instead of sys.exit

[22:55:59] <dstufft> I don't really want to do that though if I can fix it otherwise

[22:56:46] <dstufft> I'm trying to figure out if adding a no-op __del__ is a good solution or if it's just covering up the symptoms and I need to look more for the real cause

[22:58:20] <mitsuhiko> dstufft: yeah. it's a symptop

[22:58:47] <mitsuhiko> dstufft: aaah!

[22:58:52] <mitsuhiko> dstufft: you know what this is?

[22:59:02] <dstufft> mitsuhiko: I don't think so

[22:59:40] <nanonyme> dstufft, any idea if there's any stats on how well wheels have taken up?

[23:00:10] <mitsuhiko> dstufft: ah no. probably not

[23:00:11] <Alex_Gaynor> nanonyme: on an individual package you can see what the download ratio is, if you want across all of PyPI, or stats about what %age of packages have wheels uploaded, taht's harder

[23:00:18] <mitsuhiko> i was thinking maybe you have objects resurrencting

[23:00:24] <mitsuhiko> but i don't think you can actually lock python that way

[23:00:46] <mitsuhiko> dstufft: unfortunately you run into interpreter shutdown bugs there

[23:00:49] <mitsuhiko> of which there are TONS

[23:01:06] <nanonyme> Alex_Gaynor, I'm interested in global situation. Egg vs wheel would also be interesting

[23:01:10] <mitsuhiko> our company codebase does not manage to shutdown properly for at least a year any more

[23:01:32] <Alex_Gaynor> mitsuhiko: "I'm sorry, I can't let you do that mitsuhiko "

[23:01:41] <mitsuhiko> dstufft: can you try something crazy?

[23:01:58] <mitsuhiko> can you put an empty logging.py module next to pip

[23:02:00] <mitsuhiko> and make it import that one

[23:02:07] <mitsuhiko> and then just add mock objects until it imports again

[23:02:10] <mitsuhiko> and see if the problem goes away

[23:02:44] <Alex_Gaynor> oh man, if this turns out to be logging, I will be very mad

[23:02:47] <dstufft> nanonyme: globally rquires me to load a big postgresql database

[23:02:58] <nanonyme> Hrm, right

[23:03:06] <dstufft> mitsuhiko: yea I can do that

[23:03:16] <dstufft> at least I have an example of this problem now that reproduces without using travis

[23:03:18] <Alex_Gaynor> nanonyme: it might be easier to just grab the list of the top 500 packages, and then compute the stats yourself across them

[23:03:21] <dstufft> which is progress of some sort

[23:03:47] <nanonyme> dstufft, mainly from what I've seen many of the projects where it actually makes sense seem to be conservative about wheels

[23:03:47] <mitsuhiko> dstufft: can i reproduce it locally somehow?

[23:03:56] <dstufft> mitsuhiko: Do you have ubuntu 14.04?

[23:04:02] <dstufft> I don't know if it works on other ubuntu's

[23:04:25] <mitsuhiko> yeah. i have a 64bit on a server

[23:04:38] <dstufft> https://gist.github.com/dstufft/f1b65f16e702791b1427 reproduces it, with python 3.3 from deadsnakes

[23:04:58] <Alex_Gaynor> nanonyme: FWIW for Django 1.6.5, we're about 50/50 wheels and sdists for downloads

[23:04:59] <nanonyme> Oh, maybe my impressions are obsolete. Looks like Pillow and lxml both have wheels these days

[23:05:17] <nanonyme> Alex_Gaynor, does wheel actually make any difference for Django?

[23:05:29] <Alex_Gaynor> nanonyme: no setup.py execution, faster install

[23:06:29] <nanonyme> Right. But the projects who benefit the most for what I've understood are the ones that distribute compiled binaries

[23:06:31] <dstufft> mitsuhiko: or I can give you a root login on a throw away VM I have with those steps already done :)

[23:07:13] <mitsuhiko> dstufft: that would be even better

[23:07:29] <dstufft> mitsuhiko: PM inc

[23:07:41] <mitsuhiko> Alex_Gaynor: it's not logging, it's a problem with the interpreter shutdown

[23:07:57] <mitsuhiko> for some reason the logging module just triggers it for the case i have in a completely unrelated project

[23:08:04] <mitsuhiko> presumably because logging code cleans up file objects very late

[23:08:09] <xafer> but why would adding a useless method impact the shutdown ?

[23:08:22] <mitsuhiko> xafer: a dummy __del__ is not useless

[23:08:29] <mitsuhiko> it changes how python reasons about the object

[23:08:38] <xafer> was talking about remove_me method :)

[23:08:46] <nanonyme> Alex_Gaynor, don't get me wrong, I think it's cool otherwise too. I pregenerate wheels for everything I install as a local cache these days

[23:09:04] <Alex_Gaynor> nanonyme: It's hrader to upload compiled wheels, sinc eyou need to build one for every platform

[23:09:11] <Alex_Gaynor> nanonyme: we do it for windows wheels on cffi and cryptography

[23:09:44] <nanonyme> Alex_Gaynor, well, not any harder than it is to upload compiled eggs. Or exe's with compiled data

[23:09:53] <Alex_Gaynor> Those are pretty rare as well

[23:09:56] <nanonyme> The difference is the alternatives need easy_install

[23:10:12] <nanonyme> Well, both lxml and Pillow are such

[23:10:26] <nanonyme> I dunno how rare they are

[23:10:42] <mitsuhiko> nanonyme: i would assume eggs are more popular on windows

[23:10:44] <Alex_Gaynor> both of those have wheels :-)

[23:11:03] <nanonyme> Alex_Gaynor, I know! I just checked and considered it rather cool

[23:11:10] <nanonyme> mitsuhiko, for now

[23:11:10] <Alex_Gaynor> :-)

[23:11:14] <dstufft> mitsuhiko: I sent you the login/pasword, cd into ~/pip and then source ~/test_stuck/bin/activate and run py.test -k test_env_vars_override_config_file -v -s

[23:11:36] <mitsuhiko> oh dad latency

[23:11:38] <mitsuhiko> where is this box? :D

[23:11:40] <dstufft> IAD

[23:11:43] <dstufft> virgina

[23:11:45] <xafer> in fact changing print('KO'* 10) into print('KO') also toggle the bug...

[23:12:55] <nanonyme> dstufft, so yeah, well done guys, you might finally manage in RIP'ing easy_install :P To think it only took a new package format :)

[23:14:19] <dstufft> (the reproducer came from xafer so, thanks xafer for making it possible to repro this without pushing to travis a billion times)

[23:14:29] <dstufft> nanonyme: :) we're trying

[23:16:15] <nanonyme> How does it atm work with Windows anyway, do you bundle wheel or what?

[23:17:07] <dstufft> you'll have to be more specific, how does what work with windows

[23:17:12] <xafer> np I would be quite happy to get to the bottom of it :o

[23:19:45] <mitsuhiko> dstufft: shell too slow for me

[23:19:49] <xafer> but I should already be sleeping, hopefully I'll wake up tomorrow to read a nice explanation for this sorcery :)

[23:19:54] <mitsuhiko> i need to reproduce this somewhere elese. might try that tomorrow

[23:20:05] <mitsuhiko> dstufft: basically i can tell you how i debugged that last time myself

[23:20:11] <nanonyme> dstufft, when you install a fresh Python on Windows, will you have wheel?

[23:20:15] <mitsuhiko> i built a version of python that had prints in Py_Shutdown

[23:20:21] <mitsuhiko> or whatever the method was called

[23:20:30] <mitsuhiko> it runs through a whole bunch of stuff

[23:20:32] <dstufft> mitsuhiko: sorry :( Is there a rackspace datacenter that's better for you? I can spin one up a server there right now and set up the reproducer (yay free RS credits)

[23:20:34] <mitsuhiko> one of them gets stuck

[23:21:15] <mitsuhiko> dstufft: don't worry about it, will do it on my server

[23:21:26] <dstufft> mitsuhiko: ok

[23:21:30] <mitsuhiko> let me just find you the method

[23:21:35] <dstufft> Py_Finalize?

[23:21:48] <mitsuhiko> yeah

[23:21:56] <dstufft> I was looking at that yesterday to try and guess stuff, I didn't put any prints or compile my own Python because at that time I could only repro on travis

[23:22:13] <dstufft> that's a good idea though

[23:22:36] <dstufft> I'll try that in a bit

[23:23:22] <mitsuhiko> dstufft: also. check if there are exit funcs registered

[23:23:44] <dstufft> how can I check if there are? I couldn't find an aPI that let me inspect that

[23:23:47] <mitsuhiko> they can be poison if they start threads

[23:24:14] <dstufft> suppose I could just grep for atexit

[23:24:24] <mitsuhiko> dstufft: atexit._exithandlers

[23:24:41] <dstufft> mitsuhiko: oh, that doens't work on Python 3 :( the made it a C module

[23:24:43] <mitsuhiko> assuming something uses atexit

[23:24:44] <dstufft> they*

[23:24:50] <mitsuhiko> otherwise it's sys.exitfunc

[23:24:54] <forrest> dstufft, this is gravyboat from github, let me know if you guys ome up with anything else that needs to be tested on that requests issue.

[23:24:59] <forrest> *come up

[23:25:18] <mitsuhiko> oh. missed that

[23:25:33] <dstufft> suppose i could modify that too

[23:25:37] <dstufft> to print when someone uses it

[23:25:50] <dstufft> print(), the ultimate debug tool

[23:25:52] <mitsuhiko> why did they write atexit in c

[23:25:56] <dstufft> no idea

[23:26:04] <dstufft> it seemed silly to do that to me

[23:26:04] <mitsuhiko> it still uses the underlying method of python 2 which is invoking one function :P

[23:26:54] <mitsuhiko> py_finalize is about the most bizarre piece of code in the interpreter

[23:27:26] <mitsuhiko> dstufft: another stuff you could try is registering gc.collect as atexit handler

[23:27:28] <mitsuhiko> and see what happens

[23:27:55] <dstufft> oh, the patch that made atexit a C module says "remove sys.exitfunc and make it a private API, and make atexit a c module so it can use private api"

[23:28:15] <dstufft> https://github.com/python/cpython/commit/5380fe315ab60ecf23be8a87b9acbf1788be119f#diff-be9f1e5956c7412765a17dca1649946e

[23:29:21] <dstufft> gotta go for a bit, thanks for your help thus far mitsuhiko :)

[23:31:47] <mitsuhiko> the python interpreter shutdown is unfortunately black magic

[23:31:53] <mitsuhiko> if you get code to run too late almost anything can happen

[23:32:06] <mitsuhiko> so your best solution is to not trigger things from destructors

[23:32:21] <mitsuhiko> and definitely not from atexit :)

[23:32:25] <mitsuhiko> heading to bed now

[23:32:27] <mitsuhiko> gn8 :)

[23:40:05] <nanonyme> mitsuhiko, yes, most people avoid destructors altogether

[23:58:38] <nanonyme> Hmm, why am I getting in pip.log 404 about https://pypi.python.org/simple/install/...

Log file Viewer

Help | Karma | Search:

#pypa logs for Wednesday the 2nd of July, 2014