Log file Viewer

[04:33:08] <wiredfool_> epicserve: ping

[04:33:18] <wiredfool_> couple of things for you

[04:34:44] <wiredfool_> http://www.insweat.com/whispers/articles/python-gc -- see para 2. gc.DEBUG_LEAK looks like a memory leak itself, esp to tools like top

[04:37:00] <wiredfool_> second, there's definitely some retained objects in the boto uploads to s3 https://gist.github.com/wiredfool/5754498

[04:37:03] <crundar> I'm looking to convert my numeric (integer) data into pixels in a greyscale file. I'd like to take advantage of the PNG or TIFF's ability to generate 2^16 shades of grey. Does PIL have a method to do that as simply as it does write RGB values?

[04:37:30] <crundar> I've seen the 'L' mode, but that's too restrictive for what I'm after, I believe

[04:37:47] <wiredfool_> There are quite a few modes, but...

[04:38:47] <wiredfool_> I don't think 16bit gray is one of therm.

[04:39:04] <wiredfool_> there's a 32 bit float

[04:39:17] <wiredfool_> not sure what i;16 is

[04:39:29] <crundar> Or even just a way to specify the 16 bits of grey available and use precisely those?

[04:39:42] <wiredfool_> tiff isn't likely to be able to save it in anything other than uncompressed raw

[04:39:58] <crundar> I'm execting my images to be sent to a B&W printer, and I'd like to get as much luminescence (if that the word I want) as possible.

[04:39:59] <wiredfool_> I'm not as familiar with the inside of png

[04:40:38] <wiredfool_> like a 16 bit paletized image?

[04:40:46] <wiredfool_> that might be i16;

[04:41:35] <crundar> you went past me at paletized

[04:42:14] <wiredfool_> ah. like gif, indexed color into a palette

[04:42:20] <crundar> I was looking here http://www.aivosto.com/vbtips/imageformats.html , found that they have more greys than I'm currently getting, and searched around PIL to no avail.

[04:42:24] <wiredfool_> only with 2^^16 avaliable

[04:42:58] <crundar> it says they have a special grey modes. Is that the same thing?

[04:45:44] <wiredfool_> not sure. sounds like a printer thing.

[04:45:57] <wiredfool_> It _looks like I;16 is the mode you want

[04:46:17] <crundar> Huh. Well I read something that suggested exactly that.

[04:46:18] <crundar> http://mail.python.org/pipermail/image-sig/2003-May/002238.html

[04:46:22] <wiredfool_> and it does look like it can write it, at least in raw mode in tiff

[04:47:24] <wiredfool_> you'd have to set compression to 'raw'

[04:48:43] <crundar> write it as raw mode in tiff... then Pythonmagick to PNG (losslessly)

[04:48:56] <crundar> and I think I can get PNGs to display in xelatex

[04:49:09] <crundar> Rube Goldberg couldn't've planned it any better himself.

[04:49:11] <wiredfool_> let me look at the png source

[04:50:30] <wiredfool_> crundar: no, that's nowhere near as convoluted as it could get

[04:52:05] <wiredfool_> looks like in PNG, I is the mode you want, it writes as I;16B, but I'm not sure if it's a 32 bit source and 16bit output, or if it's 16bit in 16bit out.

[04:52:19] <crundar> hopefully this'll be enough.

[04:52:35] <crundar> Wait, so I can just open it with I instead of RGB, and it should just do its thing?

[04:52:45] <wiredfool_> but I mode is definitely wrong for tiff, you want I;16

[04:53:50] <crundar> I was kind of hoping to not have to learn tonight about how colors and greys and such work; that there is in fact a mode that'll just make it happen if i feed it integers with the range 0 to (sub1 (* 2 16))

[04:53:58] <wiredfool_> sorry, I might be confused between modes and rawmodes

[04:53:58] <crundar> err.. expt

[04:54:48] <wiredfool_> what I'd try is test it out with synthetic data, put the data in as 16bit, set the mode as I, and then try writing the png and tiff, and see if you get out what you expect

[04:55:09] <wiredfool_> and if you have a good test + images, send them and I'll add them to the test suite

[04:56:26] <wiredfool_> I don't see anything obvious in the test suite.

[04:57:47] <crundar> Alright, I'll go try that right now.

[04:58:04] <crundar> thanks much, by the way

[04:59:43] <wiredfool_> sure. please do send a test if you get it working

[05:05:10] <wiredfool_> epicserve: https://github.com/wiredfool/et-issue-example/blob/master/photos/management/commands/generate_all_thumbnail_aliases.py produces this: https://gist.github.com/wiredfool/5754556

[05:05:53] <wiredfool_> epicserve: my informed guess is that storages.backends.s3boto.S3BotoStorageFile is retaining way too many botos.

[05:09:28] <crundar> well, that sort of worked. I think its just that my numbers are a little small to take up much room.

[05:09:36] <crundar> that'll require some playin' with

[17:21:33] <epicserve> wiredfool_: cool, I'll take a look

[17:44:19] <wiredfool_> epicserve: cool

[21:50:08] <epicserve> wiredfool_: So am I reading this right … you ran the et-issue-example again using S3?

[22:00:21] <wiredfool_> yep

[22:00:34] <wiredfool_> (both with and w/o ssl, but no change there)

[22:02:01] <epicserve> wiredfool_: I can see from your gist that RSS was growing

[22:02:23] <epicserve> what is the number in front of the class/object?

[22:03:16] <epicserve> it would be helpful to know how much memory each object is using in total.

[22:04:50] <wiredfool_> sorry, that's count of objects

[22:05:30] <wiredfool_> so, frex, each time through the loop there's a new boto connection + ssl, + bucket + ...

[22:07:01] <epicserve> so the real memory hog would be if it's keeping the image binary in memory in the boto object

[22:07:53] <wiredfool_> that would be one sneaking suspicion

[22:08:30] <wiredfool_> pympler.asizeof might help there

[22:09:07] <wiredfool_> there's also this, http://pythonhosted.org/Pympler/tutorials/scons-classtracker.html

[22:09:22] <wiredfool_> which looks like the advanced version of what I hacked together

[22:10:42] <wiredfool_> and this, as well: http://pythonhosted.org/Pympler/tutorials/muppy_tutorial.html