pmxbot IRC Log Viewer

[00:00:43] <wiredfool_> allocations, looking at call trees, with extended detail

[00:01:03] <wiredfool_> there's the heaviest stack trace in the extended detail

[00:01:29] <wiredfool_> and it looks like there's a bunch of memory used in pyobject_genericsetattrWithDict

[00:01:49] <wiredfool_> and then dict resize

[00:02:15] <wiredfool_> (which sounds dirty, but only because it contains a malloc for 1.2gb)

[00:02:43] <wiredfool_> new screen shot

[00:02:50] <epicserve> I'm guessing boto uses XML

[00:05:33] <wiredfool_> It really looks like the vast majority of the memory usage is in the xml

[00:05:49] <wiredfool_> how big is the bucket you're putting this in?

[00:06:19] <wiredfool_> and are you prehaps, getting a list of the entire contents?

[00:07:31] <epicserve> The bucket is huge

[00:07:54] <wiredfool_> do you get a full list of keys anywhere?

[00:08:14] <epicserve> no idea how boto works

[00:09:31] <epicserve> just asked on #boto

[00:09:56] <epicserve> I installed valgrind on a server

[00:10:35] <wiredfool_> well, s3 has lots of xml

[00:10:40] <wiredfool_> and boto talks to it

[00:11:03] <wiredfool_> the question is, how does the storage adapter drive it

[00:13:14] <epicserve> want me to run valgrind on the server and if so just `valgrind python temp.py` ?

[00:13:32] <wiredfool_> valgrind --tool=massif python temp.py

[00:13:45] <wiredfool_> but, I'm going to bet that you're going to get similar things

[00:13:55] <wiredfool_> This has way more detail

[00:14:08] <wiredfool_> like from 6:18 to 6:40 is python garbage collection

[00:14:48] <wiredfool_> so here's what I'd do to narrow it.

[00:15:03] <wiredfool_> 1) setup a temp bucket. run into it.

[00:15:13] <wiredfool_> see that memory is small and reasonable.

[00:20:00] <epicserve> wiredfool_: valgrind is running on the server … does it output the results after it's done?

[00:20:44] <wiredfool_> 2) there's a line in storage that's self.key=storage.bucket.get_key(...).

[00:21:21] <wiredfool_> I bet that's some of the problem., dump a memory size before and after, or check networking or somehting.

[00:21:32] <wiredfool_> It puts the results in massif.out.PID

[00:21:39] <wiredfool_> to get neato ascii art, then

[00:21:48] <wiredfool_> ms_print massif.out.PID > somefile

[00:22:19] <wiredfool_> gotta run for dinner.

[00:22:47] <wiredfool_> but I'm betting that we're getting close.

[00:28:15] <epicserve> wiredfool_: it took 35 seconds with a smaller bucket

[00:40:24] <epicserve> wiredfool_: saved Instruments2.trace to dropbox

[00:43:23] <wiredfool_> sounds like a smoking gun to me

[00:43:51] <epicserve> added massif.out.3351 to dropbox

[00:44:20] <epicserve> now, how the heck do I find the problem

[00:49:43] <wiredfool_> ok. wild ass guess this time.

[00:49:58] <wiredfool_> what the storage system is doing for get key is listing the whole bucket

[00:50:05] <wiredfool_> which is big

[00:50:21] <wiredfool_> instead of listing with some prefix

[00:50:43] <wiredfool_> and it's optimized for finding existence in small buckets, not non-existence in large ones

[00:52:42] <wiredfool_> I'd look in storage/backends/s3boto.py

[00:53:16] <epicserve> wiredfool_: right, but what library is doing that … boto, django-storages, easy-thumbnails, thelonious

[00:54:13] <epicserve> boom!

[00:54:48] <epicserve> https://github.com/epicserve/easy-thumbnails/blob/master/easy_thumbnails/files.py#L80

[00:57:45] <wiredfool_> also look at django-storage: storage/backends/s3boto.py

[00:58:28] <wiredfool_> and find the S3BotoToStroageFile

[00:58:36] <epicserve> I commented out those lines and it took 13.50 seconds!

[01:56:08] <epicserve> wiredfool_: https://github.com/epicserve/easy-thumbnails/commit/e389235ca53462e87d335ace0cca8558f52a8a03

[03:35:38] <wiredfool_> cool. what's the memory trace look like on the big bucket now?

[03:36:08] <wiredfool_> (incidentally, that's more correct, and more pythonic anyway. Exists is a built in race condition.)

[17:10:57] <Fenhl> hi, I'm trying to install Pillow on OS X

[17:11:22] <Fenhl> I've installed the prerequisites using Homebrew, worked fine

[17:12:08] <Fenhl> now, when I use `sudo pip install Pillow` it appears to only install for Python 2, however I need it working on Python 3

[17:12:54] <Fenhl> I haven't uses pip before, so any help on how to do this is appreciated

[17:13:24] <Fenhl> s/uses/used/

[17:46:36] <epicserve> Fenhl: still there

[17:49:51] <epicserve> Well just in case you get back :)

[17:50:04] <epicserve> type `which pip`

[17:50:40] <epicserve> My guess is that it's using pip for python 3

[17:50:55] <epicserve> You need to use pip for python 3

[17:51:55] <epicserve> on my computer it's in /usr/local/Cellar/python3/3.3.2/bin/pip3

[17:52:32] <epicserve> ideally you should use virtualenv and more ideally you should use virtualenvwrapper

[18:25:09] <Fenhl> epicserve: I used virtualenv, the install said “platform darwin 3.3.2 (v3.3.2:d047928ae3f6, May 13 2013, 13:52:24)” this time

[18:25:56] <Fenhl> but “from PIL import Image” still throws "No module named 'PIL'"

[18:26:37] <epicserve> Fenhl: when you're in your virtualenv run pip Freeze

[18:26:58] <epicserve> Does it show PIL or Pillow?

[18:27:20] <aclark> Fenhl: pip install Pillow should work

[18:29:14] <Fenhl> epicserve: Pillow==2.0.0

[18:29:19] <Fenhl> aclark: nope

[18:29:31] <epicserve> Fenhl: while in your virtualenv what does `which python` and `which pip3` show?

[18:30:14] <Fenhl> epicserve: which python: /Users/fenhl/py3env/bin/python — which pip3: nothing

[18:31:45] <aclark> Fenhl: heh, what i mean is: you aren't asking the right question. "appears to only install for Python 2" makes no sense. maybe what epicserve says is true… try python -V

[18:32:07] <epicserve> Fenhl: how about `which easy_install3`

[18:32:39] <aclark> if you have Pillow 2.0.0 you have Python 3 support, regardless

[18:32:54] <Fenhl> epicserve: nothing

[18:32:56] <aclark> (even if you install it in a Python 2 env)

[18:33:10] <Fenhl> aclark: so why am I getting an ImportError?

[18:33:16] <epicserve> Fenhl: what about `python --version`

[18:34:03] <aclark> Fenhl: no idea… have you pasted the error yet?

[18:34:12] <aclark> epicserve: or -V :-)

[18:34:24] <Fenhl> epicserve: Python 3.3.2

[18:34:36] <Fenhl> aclark: wait a sec

[18:34:48] <aclark> ok python 3, cool…

[18:35:01] <aclark> soooo, any chance PIL is install in addition to Pillow?

[18:35:18] <aclark> (this usually causes folks trouble)

[18:35:19] <epicserve> aclark: I was just going to ask that

[18:35:42] <Fenhl> aclark: here's the full terminal session: http://pastie.org/8046574

[18:36:06] <aclark> Fenhl: If "import Image" works, you have PIL not Pillow

[18:36:12] <Fenhl> aclark: it doesn't

[18:36:31] <aclark> Fenhl: Then you have neither installed :-)

[18:37:10] <Fenhl> aclark: but I did `brew install libtiff libjpeg webp littlecms` and `sudo pip install Pillow`

[18:37:14] <aclark> Fenhl: (your paste looks like what it would like if you hadn't installed anything)

[18:37:25] <aclark> Fenhl: Don't sudo, you probably installed it with the system python

[18:37:36] <Fenhl> but it fails without sudo

[18:37:46] <aclark> fix that problem then

[18:39:48] <epicserve> I agree with aclark, sounds like an issue with your virtualenv

[18:40:01] <epicserve> I would nuke your virtualenv and start over

[18:41:19] <epicserve> make sure you create your virtualenv with `virtualenv -p /path/to/bin/python3 --distribute mypy3env`

[18:41:27] <epicserve> and don't use sudo

[18:41:55] <epicserve> then activate your virtualenv with `source mypy3env/bin/activate`

[18:42:05] <epicserve> then `pip install Pillow`

[18:42:08] <epicserve> that should work

[18:43:07] <epicserve> if that doesn't work then throw your computer out the window and go to your nearest computer store and buy another one

[18:44:13] <Fenhl> okay, that worked, but it still doesn't work outside that environment

[18:45:26] <epicserve> Fenhl: uh … why do you need it out of the environment?

[18:46:29] <epicserve> I don't install anything in my system packages

[18:47:15] <Fenhl> I don't see why I should source it everytime I want to use Python

[18:47:31] <Fenhl> well… why is that?

[18:49:51] <epicserve> Fenhl: personal preference I guess … however I do know most people do it that way.

[18:50:26] <Fenhl> epicserve: I hear most people are still using Python 2 for some reason

[18:51:05] <epicserve> Fenhl: in the Django world it's because most packages haven't been ported to python 3 yet

[18:54:40] <Fenhl> well I guess I can live with the environment thing for now

[18:55:43] <epicserve> why do you want it to work in your system python?

[19:32:10] <Fenhl> epicserve: I don't want to have to source into the environment every time I want to use Pillow

[19:37:55] <epicserve> Fenhl: I typically have projects I work on so if I need to work on a project I just type `workon mycoolproject`

[19:37:59] <epicserve> pretty easy

[19:38:12] <epicserve> I highly recommend virtualenvwrapper

[20:11:21] <aclark> Fenhl: first world problems…

[20:12:04] <Fenhl> aclark: true

[20:12:24] <Fenhl> epicserve: I'll take a look at it

Log file Viewer

Help | Karma | Search:

#pil logs for Saturday the 15th of June, 2013