PMXBOT Log file Viewer

Help | Karma | Search:

#pil logs for Monday the 30th of June, 2014

(Back to #pil overview) (Back to channel listing) (Animate logs)
[02:39:25] <cats> hey all, I'm having some trouble working with bilevel pngs with pillow, can anyone help me out real quick?
[15:49:46] <wiredfool1> hi cats -- what seems to be the problem?
[15:58:55] <cats> wiredfool: I'm getting different array representations when I open an image from a filename and when I open from a buffer
[16:00:49] <cats> on top of that, it seems that writing pngs back to disk with mode '1' is broken sometimes? It appears that instead of treating each value in the array as a bit value, it's considered as an entire byte - leading to 7 black pixels + 1 black/white pixel depending on the array value
[16:32:35] <wiredfool> internally, everything is stored as bytes, not bits
[16:32:48] <wiredfool> so if you're doing bit twiddling, that's going to throw you off.
[16:33:08] <wiredfool> cats pnig
[16:35:59] <cats> wiredfool: I'm not modifying bits myself
[16:36:21] <wiredfool> can you show me code?
[16:36:56] <cats> yeah give me a second
[16:39:08] <cats> http://pastebin.com/ghfx1hyg
[16:39:24] <cats> first assert passes
[16:39:31] <cats> however the second one fails
[16:39:59] <cats> and the argwhere.shape value is nonzero, indicating differences in the actual imported array
[16:40:59] <wiredfool> what pillow & os are you using?
[16:41:26] <wiredfool> I see python 2.7
[16:42:57] <cats> pillow 2.4.0, python 2.7, centos 6.5
[16:43:15] <wiredfool> waht image are you using?
[16:44:48] <cats> I can't share the image I'm testing on directly since it contains sensitive info, but you can generate a sample image from a pdf as I did using ghostscript: gs -sDEVICE=pnggray -sOutputFile=OUTPUTPNG -dFirstPage=1 -dLastPage=1 -dSAFER -dQUIET -dBATCH -dNOPAUSE -r300 INPUTPDF
[16:45:17] <cats> and I would replace the -sDEVICE parameter with pngmono
[16:45:22] <wiredfool> ok, I've got plain pngs
[16:46:01] <wiredfool> though, it might be a strange gs png
[16:46:25] <cats> has that been a real concern, that ghostscript produces wonky pngs?
[16:47:20] <wiredfool> no, but I've got enough experience with bw pngs that I'd expect that we're handling them correctly
[16:47:49] <wiredfool> ok, I'm installing numpy in this virtual env
[16:48:01] <cats> right - digging into the source of Image.open there doesn't seem to be any reason why it produces different outputs
[16:50:02] <wiredfool> other possibility, it's the image.toarray bits, since that returnsconverts as bytes as well
[16:50:07] <wiredfool> there's an open bug on that one.
[16:51:27] <wiredfool> what does this do foe you? http://pastebin.com/s81rH1HL
[16:52:40] <cats> no assertion errors on that last one
[16:53:41] <wiredfool> ok, so it's not the read
[16:53:42] <cats> would you mind linking to the bug report?
[16:54:25] <wiredfool> https://github.com/python-pillow/Pillow/issues/350
[16:55:56] <cats> that looks to be it - I'm only getting issues on mono pngs, not at all in mode 'L'
[16:58:25] <wiredfool> try converting to mode L before numpy.asarray
[16:58:27] <wiredfool> e.g.
[16:58:33] <wiredfool> filename_img = Image.open('monoimg.png').convert('L')
[17:00:49] <cats> yup, both asserts pass and also argwhere.shape is (0, 1)
[17:01:16] <cats> well I guess I can proceed this way and simply mark the actual mode of the image in another attribute
[17:02:11] <cats> more importantly, how deep is this bug? I'm wondering what the relevant parts of Image.py need modification
[17:02:45] <wiredfool> so, I've looked at this, and it's been a bug that I've run into
[17:03:04] <wiredfool> essentially, 1 bit and L images are the same in Pillow, they're both byte oriented.
[17:03:46] <wiredfool> to get the 1 bit images into numpy as a 1 bit array, the image needs to be packed into the smaller format, and hten handed off to numpy in image_toarray
[17:03:54] <wiredfool> image.toarray
[17:04:12] <wiredfool> and the mode for numpy needs to be handled correctly
[17:05:31] <cats> when you say '1' and 'L' are treated the same, do I see this behavior when I try to convert an array obtained from a '1' image to np.uint8?
[17:05:37] <wiredfool> sorry, it's not toarrya, it's the __getattr__ "__array_interface__ oart
[17:05:47] <cats> like np.asarray(mono_img).astype(np.uint8)
[17:06:05] <cats> because I see that I don't get just 0 and 255s, in fact there's the entire range in between
[17:07:04] <wiredfool> good question. that's .... I wonder if this has changed when I wasn't looking
[17:07:11] <cats> so I'm guessing those uint8 values are the bytes representing 8 pixels
[17:10:24] <cats> well wiredfool, this was very helpful, thanks for your time
[17:11:08] <wiredfool> cats: sorry I didn't help further
[17:11:12] <wiredfool> this is really strange
[17:11:27] <wiredfool> the Pillow sides of it look like they're sending the same data to numpy
[17:11:40] <wiredfool> as looking at it from fh_img.__array_interface__
[17:12:10] <cats> well another thing to note is
[17:12:29] <cats> np.asarray does not make a copy if it's not necessary, whereas np.array always makes a copy
[17:13:26] <wiredfool> I'm not up on the numpy side of things.
[17:16:58] <cats> wiredfool: that's fine, I'm also quite certain that there's some issue in numpy in the big picture
[17:17:55] <cats> I got a segfault converting the image between various types with ndarray.astype
[17:19:52] <cats> could not replicate it, unfortunately
[17:21:35] <wiredfool> wonderful
[17:21:40] <aclark> wiredfool: would making your test-installed.py a "script" in the setuptools sense be any less of a paper cut? :-)
[17:22:06] <aclark> (probably not)
[17:22:37] <aclark> wiredfool: i don't understand how unittest broke your workflow, can you explain again?
[17:22:55] <wiredfool> it's the old --intsalled thing
[17:23:13] <wiredfool> I'm often testing against 2 or three different pythons
[17:23:20] <wiredfool> so I've got terms in three different vpys
[17:23:33] <wiredfool> firing off install + test in all of them
[17:23:46] <wiredfool> so the requirement for inplace sucks
[17:24:36] <aclark> unittest == in place?
[17:26:23] <wiredfool> if you don't so an inplace build, import PIL from the Pillow directory doesn't include the c code
[17:26:32] <wiredfool> It's python only
[17:26:42] <aclark> Ah
[17:26:52] <wiredfool> since the .so's are in build/platform/*
[17:26:59] <aclark> IC
[17:27:22] <wiredfool> so, we have to do path twiddling to get the installed PIl
[17:28:55] <cats> wiredfool: quick update, conversion to 'L' and then back to '1' avoids the corrupted output png, so this will be a decent patch for now
[17:29:12] <wiredfool> cats: cool.
[17:30:58] <wiredfool> aclark: if test-installed actually gets installed by python setup.py install, then it would be runnable by just test-install.py
[17:31:24] <wiredfool> then it would be ok, though that's a bad name from a namespace POV
[17:31:56] <wiredfool> no, that's not going to work, since we don't actually install our tests
[19:19:05] <fission6> encoder error -2 when writing image file any ideas? trying to resize JPEGS
[19:29:35] <wiredfool> that's jpeg's general something went wrong error code
[19:29:48] <wiredfool> may be on read
[19:30:06] <wiredfool> fission6: does the error happen on load in this:
[19:30:19] <wiredfool> im = Image.open('foo.jpeg')
[19:30:22] <wiredfool> im.load()
[19:34:18] <fission6> ahhh
[19:34:21] <fission6> its in save()
[19:34:26] <fission6> i had to remvoe optimize=True
[19:34:27] <fission6> weird
[19:34:33] <fission6> any ideas wiredfool
[19:40:45] <wiredfool> what version of pillow? there were some changes around that recently
[21:12:43] <travis-ci> [travis-ci] hugovk/Pillow#354 (diff_cover - b72b639 : hugovk): The build has errored.
[21:12:43] <travis-ci> [travis-ci] Change view : https://github.com/hugovk/Pillow/compare/38c3dd8c7dda...b72b6395c03a
[21:12:43] <travis-ci> [travis-ci] Build details : http://travis-ci.org/hugovk/Pillow/builds/28814890
[21:21:06] <travis-ci> [travis-ci] wiredfool/Pillow#376 (master - d8f55e3 : Hugo): The build passed.
[21:21:06] <travis-ci> [travis-ci] Change view : https://github.com/wiredfool/Pillow/compare/5833ecb4d63b...bd9c555905af
[21:21:06] <travis-ci> [travis-ci] Build details : http://travis-ci.org/wiredfool/Pillow/builds/28815272
[21:39:25] <wiredfool> damn algeria's goalie is good
[22:45:16] <travis-ci> [travis-ci] wiredfool/Pillow#381 (multithread-testing-travis - 9dd559c : wiredfool): The build failed.
[22:45:16] <travis-ci> [travis-ci] Change view : https://github.com/wiredfool/Pillow/commit/9dd559cb50e2
[22:45:16] <travis-ci> [travis-ci] Build details : http://travis-ci.org/wiredfool/Pillow/builds/28822060
[23:20:48] <travis-ci> [travis-ci] hugovk/Pillow#356 (skippyroma - 7b838f3 : hugovk): The build passed.
[23:20:48] <travis-ci> [travis-ci] Change view : https://github.com/hugovk/Pillow/compare/50bbb5e0c535^...7b838f31b48d
[23:20:48] <travis-ci> [travis-ci] Build details : http://travis-ci.org/hugovk/Pillow/builds/28824055
[23:27:13] <travis-ci> [travis-ci] python-pillow/Pillow#1269 (master - 8d19290 : Alex Clark ☺): The build passed.
[23:27:13] <travis-ci> [travis-ci] Change view : https://github.com/python-pillow/Pillow/compare/1056bd781af3...8d19290e1446
[23:27:13] <travis-ci> [travis-ci] Build details : http://travis-ci.org/python-pillow/Pillow/builds/28824353
[23:29:19] <travis-ci> [travis-ci] hugovk/Pillow#357 (diff_cover - e220e59 : hugovk): The build passed.
[23:29:19] <travis-ci> [travis-ci] Change view : https://github.com/hugovk/Pillow/compare/d002e5f6901c...e220e597bfd4
[23:29:19] <travis-ci> [travis-ci] Build details : http://travis-ci.org/hugovk/Pillow/builds/28824735
[23:39:19] <travis-ci> [travis-ci] hugovk/Pillow#358 (diff_cover_just_testing - d95b37f : hugovk): The build passed.
[23:39:19] <travis-ci> [travis-ci] Change view : https://github.com/hugovk/Pillow/commit/d95b37f8e2d0
[23:39:19] <travis-ci> [travis-ci] Build details : http://travis-ci.org/hugovk/Pillow/builds/28825487