pmxbot IRC Log Viewer

[16:48:55] <epicserve> wiredfool_: it took a couple gigs of memory to create the thumbnails for this jpg, http://assets.www.wenatcheeworld.com.s3.amazonaws.com/media/img/photo/2013/05/29/26cdf8c412-985ea2e8771d4ad5aedb4caef25fd529-81a99a1b7ba544b5aa8f0d22d5c040ab-18.jpg

[16:51:05] <epicserve> same with this one, http://assets.www.wenatcheeworld.com.s3.amazonaws.com/media/img/photo/2013/05/29/536741d59a-1bfde22161214c029f55b19591064662-c8c49428a21048e4b3161e0a40133d3f-18.jpg

[17:35:03] <wiredfool_> hmm

[17:35:06] <wiredfool_> that's odd.

[17:45:13] <epicserve> wiredfool_: so if I want to print python's total memory used before and after a function call … what's that tool again?

[17:46:47] <wiredfool_> there's the rss max call, but it's linux only, and it's only the high water mark

[17:47:20] <wiredfool_> resource.getrusage(resource.RUSAGE_SELF).ru_maxrss

[17:47:21] <epicserve> what does high water mark mean?

[17:47:37] <wiredfool_> It's the most resident memory the process has ever used

[17:47:48] <wiredfool_> so, if things are garbage collected, that number doesn't go down

[17:48:33] <wiredfool_> here's a non-s3 valgrind run: https://gist.github.com/wiredfool/5783854

[17:48:46] <wiredfool_> that's the 2 images from 9:50

[17:49:10] <wiredfool_> whatever it is, it's not in the core pillow stuff.

[17:50:14] <epicserve> non-s3 doesn't use the DB either for storing file name and modified time

[17:50:21] <wiredfool_> or rather, whatever it is, it's not showing up on my test machine.

[17:51:43] <epicserve> I wonder how hard that would be to setup valgrind on my machine

[17:51:46] <wiredfool_> so, I tried a run with PIL instead of pillow

[17:51:56] <wiredfool_> and I get: InvalidImageFormatError: The source file does not appear to be an image

[17:52:31] <wiredfool_> epicserve: easy on ubuntu, unknown on osx

[17:52:57] <epicserve> http://d.pr/i/mA3O

[17:53:10] <epicserve> After one image

[17:53:59] <wiredfool_> is that the batch process?

[17:54:20] <wiredfool_> cause that's f'd

[17:55:19] <wiredfool_> or demonic, as you've used 6.66gb

[17:57:33] <epicserve> Just one image

[17:58:49] <wiredfool_> oh. my pil is broken

[18:19:20] <wiredfool_> ok, using PIL is no different

[18:50:03] <wiredfool_> not terribly different running to s3: https://gist.github.com/wiredfool/5784266

[18:53:04] <wiredfool_> what python version/source are you using?

[18:56:48] <epicserve> Python 2.7.3 [GCC 4.2.1 Compatible Apple Clang 4.0 ((tags/Apple/clang-421.0.57))] on darwin

[18:58:42] <epicserve> Wanna come over to Wenatchee for some recreation and hacking? :P

[18:58:43] <wiredfool_> is that a homebrew one? or..

[18:59:15] <wiredfool_> we could tell the kids to go outside and play...

[19:00:08] <wiredfool_> we just had friends stop by after being away for 2 weeks, and the giggles and happy kid noises were amazing.

[19:01:19] <wiredfool_> so, the reason I ask about what the python source is is that homebrew has a --with-dtrace option. (as does the stock python). And dtrace + instruments (from the xcode apps) is likely as good or better than valgrind

[19:01:21] <epicserve> I'm thinking if virtualenv uses what ever `which python` finds then I'm using the homebrew python. `$ ls -la /usr/local/bin/python` shows `/usr/local/bin/python -> ../Cellar/python/2.7.3/bin/python`

[19:02:25] <wiredfool_> that's a homebrew.

[19:02:59] <wiredfool_> the virtual env puts a copy of the binary in it, but if that's how you got 2.7.3, than that's probably it.

[19:05:32] <epicserve> I'm not seeing --with-dtrace as an option

[19:18:42] <wiredfool_> see brew options python

[20:24:53] <wiredfool_> dtrace is interesting. but it's not misbehaving for me, so .... not totally helpful

[20:31:45] <wiredfool_> epicserve: check this out in /Developer/Applications/Instruments

[20:31:48] <wiredfool_> https://github.com/wiredfool/et-issue-example/tree/master/traces

[20:54:57] <epicserve> wiredfool_: thanks for the Yak shaving you started me on

[21:02:20] <wiredfool_> oh dear.

[21:03:24] <wiredfool_> after playing with instruments, I'm thinking dtrace >> valgrind

[21:17:47] <epicserve> wiredfool_: so do you have a link that tells how to install it?

[21:17:56] <wiredfool_> dtrace?

[21:18:11] <epicserve> `$ brew search python-dtrace`, prints `No formula found for "python-dtrace". Searching open pull requests...`

[21:18:47] <wiredfool_> the brew dtrace is

[21:19:00] <wiredfool_> brew install python --with-dtrace

[21:26:08] <epicserve> that's done … now what?

[21:26:53] <wiredfool_> you need a virtual env with that one, virtualenv -p path/to/newpython virtualdir

[21:28:39] <wiredfool_> Then I ran the instruments trace, chose the target python from within the virtualenv, added manage.py gen_all_thumbnails (or whatever it actually is) as the arguments, and set the home dir

[21:28:57] <wiredfool_> then poked at the memory and processor traces

[21:29:07] <wiredfool_> but, as I said, mine isn't leaking

[21:32:42] <wiredfool_> can you open the .trace in instruments to see what I'm seeing?

[21:35:56] <epicserve> wiredfool_: okay updated my virtualenv with python 2.7.5 that I compiled with brew install python --with-dtrace

[21:39:14] <epicserve> wiredfool_: I have this script other project I'm working on, https://gist.github.com/epicserve/ee346f8c46c70b03e373

[21:41:11] <epicserve> `(wenworld) $ python temp.py` <- When I run that it takes 173.25 seconds and goes from 48.4 MB to 2.7 GB, watching in the activity monitor

[21:43:36] <epicserve> I'm using Pillow==2.0.0

[21:46:14] <epicserve> Uninstalled Pillow==2.0 and installed Pillow 2.1

[21:46:36] <epicserve> same thing

[21:46:39] <wiredfool_> I tried a run with PIL 1.1.7, and I didn't see a diff, so I'm not expecting much there

[21:46:45] <wiredfool_> what's thelonious?

[21:47:03] <epicserve> It's our CMS I wrote

[21:47:11] <wiredfool_> ah

[21:47:40] <epicserve> so I'm not clear how I use dtrace on this

[21:48:04] <wiredfool_> have you opened the .trace file that was in https://github.com/wiredfool/et-issue-example/tree/master/traces

[21:48:18] <wiredfool_> you'll need xcode, and instruments

[21:49:34] <epicserve> I have command line tools … so I'll have to install actual Xcode?

[21:49:45] <wiredfool_> yeah.

[21:49:45] <epicserve> or does CLT have it

[21:49:55] <wiredfool_> sorry, this one is graphical

[21:50:08] <wiredfool_> basically, instruments is a graphical interface to dtrace

[21:50:33] <wiredfool_> and dtrace is good at looking at internals of things

[21:50:50] <epicserve> downloading

[21:51:27] <wiredfool_> in 4.6, it's Xcode->OpenDeveloperTool->Instruments

[21:52:20] <epicserve> so you have to compile python with dtrace so instruments can read the process?

[21:53:27] <wiredfool_> it gets much better inssight into it

[21:53:28] <epicserve> xcode is downloading … 3 mins left

[21:53:31] <wiredfool_> like stack traces

[21:53:36] <wiredfool_> you've got a fast connection

[21:53:45] <wiredfool_> sampled stack traces

[21:53:49] <wiredfool_> all the mallocs

[21:54:09] <epicserve> that doesn't mean much to me

[21:54:23] <epicserve> I'm a higher level programmer :)

[21:59:46] <epicserve> http://d.pr/i/oQpC

[22:10:01] <epicserve> wiredfool_: xcode installed

[22:10:21] <wiredfool_> cool

[22:13:47] <epicserve> So I downloaded your template

[22:13:55] <epicserve> Opened it

[22:14:12] <epicserve> Now just hit record when I start running the python script?

[22:15:07] <wiredfool_> if you set it up right, hitting record will start the python script and follow it through the entire run

[22:18:50] <epicserve> what do I choose for the target?

[22:19:05] <epicserve> my venv python?

[22:19:07] <wiredfool_> yourvirtualenv/bin/python

[22:19:27] <wiredfool_> args should be manage.py generate_all_thumbnail_aliases

[22:19:45] <wiredfool_> working dir should be wherever manage.py is

[22:26:59] <epicserve> wiredfool_: does this look right? http://d.pr/i/CwaD

[22:28:21] <wiredfool_> yep, that's about right

[22:31:36] <epicserve> wiredfool_: https://dl.dropboxusercontent.com/u/1000994/python.tracetemplate

[22:32:26] <wiredfool_> can you save the trace, not as a template? That should have all the data as well

[22:34:00] <epicserve> oops, copied the wrong file

[22:34:03] <epicserve> https://dl.dropboxusercontent.com/u/1000994/Instruments.trace.zip

[22:35:02] <epicserve> I don't think it's tracing the child process it creates

[22:36:25] <wiredfool_> doesn't look like it here

[22:37:05] <epicserve> any idea how you would set that up?

[22:37:24] <wiredfool_> did you cancel it, or did it run itself

[22:37:58] <wiredfool_> cause looking at it, I'd almost think that it's just starting an interpreter and sitting at the prompt

[22:37:59] <epicserve> I started `python temp.py` and then hit record

[22:38:57] <wiredfool_> so, edit the active target, and add temp.py in the arguments

[22:39:15] <wiredfool_> and add location of temp.py to the sorking directrory box

[22:39:29] <wiredfool_> then just hit record, and it'll do the starting for you

[22:40:27] <epicserve> so I need to #!/virtenv/bin/python to the top and make temp.py executable?

[22:41:34] <wiredfool_> no

[22:41:41] <wiredfool_> your executable is still python

[22:44:10] <wiredfool_> one moment, putting dropbox on that mac user

[22:45:33] <wiredfool_> your email to share a db folder?

[22:47:32] <epicserve> how do you see what the console prints when it tries to run because when I hit record now … it just starts and then stops

[22:48:31] <wiredfool_> click on the target popup, then edit active target

[23:02:41] <epicserve> wiredfool_: http://d.pr/i/lIvI

[23:05:51] <wiredfool_> that should be it

[23:06:14] <epicserve> I hit record and it runs for 1 sec and then stops

[23:09:21] <epicserve> how do you see what stdout prints?

[23:10:36] <wiredfool_> I'm seeing it on mine from Sampler > console, down in the detail bar

[23:14:17] <epicserve> this is what I see, http://d.pr/i/3Fs5

[23:14:36] <wiredfool_> where it says sample list

[23:14:39] <wiredfool_> that's a popup

[23:14:57] <epicserve> I switch to console and nothing

[23:15:51] <wiredfool_> then I doubt you're getting anythign

[23:16:27] <wiredfool_> so I assume that you can run that python test.py from that directory and it produces something?

[23:25:07] <epicserve> yes

[23:27:26] <wiredfool_> I'm beginning to wonder how many ways your setup can behave differently than mine

[23:29:45] <epicserve> no kidding

[23:30:04] <epicserve> what I see from the console, http://d.pr/i/7j4W

[23:31:18] <epicserve> I think I might have it

[23:32:41] <epicserve> Getting stuff now!!

[23:33:21] <epicserve> I'll save the output when it's done to the dropbox

[23:33:27] <wiredfool_> k

[23:38:05] <epicserve> this takes longer

[23:38:22] <wiredfool_> yeah, profilers will do that.

[23:38:30] <wiredfool_> valgrind is 10-20x

[23:38:56] <epicserve> leaks isn't doing anything

[23:39:19] <epicserve> the instrument leaks

[23:40:25] <wiredfool_> I wasn't seeing much there, but I was seeing something

[23:43:49] <epicserve> wiredfool_: dropbox updated with the trace

[23:49:20] <epicserve> wiredfool_: check this out http://d.pr/i/OMZG

[23:49:44] <epicserve> wiredfool_: what the heck is MALLOC_SMALL?

[23:57:30] <wiredfool_> I'm not sure.

[23:59:01] <wiredfool_> however, there seems to be some heavy weight xml stuff going on

[23:59:46] <epicserve> where are you seeing that?

Log file Viewer

Help | Karma | Search:

#pil logs for Friday the 14th of June, 2013