Log file Viewer

[17:49:44] <dstufft> huh

[17:49:58] <dstufft> Python 3.3+ supports bzip2 and lzma for zip files

[17:50:13] <Ivo> zip files support lzma?

[17:50:20] <Ivo> is this the future?

[17:50:27] <dstufft> https://docs.python.org/3.4/library/zipfile.html#zipfile.ZIP_LZMA

[17:50:38] <dstufft> this could pave the way for lzma wheels

[17:51:17] <Ivo> py2ers will complain bitterly

[17:51:40] <Ivo> can python import an lzma zipfile, I wonder

[17:52:07] <dstufft> oo

[17:52:08] <dstufft> I dunno

[17:52:12] <dstufft> I shall try~

[17:53:01] <Ivo> is lzma completely better than bz2?

[17:56:30] <dstufft> the answer is no, Python only supports importing from a ZIP_STORED or ZIP_DEFLATED

[17:56:52] <dstufft> depends what you mean by better

[17:57:11] <dstufft> it gets smaller sizes in every test i've ever seen

[17:57:41] <Ivo> ja i think its probably completely better

[17:57:46] <Ivo> for packaging purposes

[17:57:59] <Ivo> linux dropped bzip2 for xz

[17:58:05] <Ivo> so I can't imagine them being wrong

[17:58:17] <dstufft> basically xz is smaller that bzip2 but faster to decompress than bzip2 (but still slower than gzip)

[17:58:24] <dstufft> but it takes more memory to decompress

[17:59:10] <dstufft> lzma is also faster o compress than bzip2

[17:59:24] <dstufft> but it can take a ton of memory at the higher compression levels

[17:59:52] <Ivo> wikipedia says lzma is slower to compress than bzip2

[18:00:28] <dstufft> http://tukaani.org/lzma/benchmarks.html says differently

[18:00:39] <tomprince> lzma == xz

[18:00:50] <dstufft> techincally xz is a container format that uses lzma :D

[18:00:59] <tomprince> Yes..

[18:02:30] <dstufft> <1MB for compress/decompress of gzip, 2-7MB for compress of bzip2, 1-4 MB for decompress of bzip2, 2-311MB for compression of lzma, 1-33MB for decompression

[18:02:53] <dstufft> but yea, I think lzma is wholly better than bzip2 (and gzip unless you care about speed)

[18:03:10] <Ivo> dstufft: that site does not say differently, it agrees with me

[18:03:18] <Ivo> lzma takes longer to compress than bzip2

[18:03:40] <dstufft> oh I'm stupid, I was looking at the wrong column

[18:04:07] <dstufft> I still think it's wholly better for packaging, you only compress them once :D

[18:04:13] <Ivo> but anyway compression time is the thing you don't look at for packaging

[18:05:15] <Ivo> but dstufft if wheels are lzma how will we break our own advice and magically import and run them with virtualenv and get-pip.py

[18:05:50] <dstufft> well probably we'd have to continue to use ZIP_DEFLATED for a long time

[18:06:04] <Ivo> could always write the 10 more lines to extract it first

[18:06:05] <dstufft> esp since LZMA is only on Python 3.3+

[18:06:31] <Ivo> and if it's still a .whl, how will a pip on python2 know not to grab it

[18:06:44] <Ivo> .whl2 ?

[18:07:45] <ionelmc> we had eggs and wheels

[18:07:45] <dstufft> Dunno :) Might make sense to put the compression in the filename so pip can select the compression it likes best too

[18:07:50] <ionelmc> i think it's time for gears

[18:07:57] <dstufft> because lzma isn't always available

[18:07:58] <ionelmc> .ghr

[18:08:00] <ionelmc> :P

[18:08:00] <dstufft> nor is bzip2

[18:08:30] <dstufft> pip-1.5.6-py2.py3-none-any.xz.whl

[18:08:35] <dstufft> pip-1.5.6-py2.py3-none-any.bz2.whl

[18:09:05] <ionelmc> dstufft: how do you feel about having a whundle command (wheel bundle) ?

[18:09:16] <dstufft> inside of pip itself?

[18:09:37] <ionelmc> well it could be a plugin

[18:09:43] <Ivo> if its py33 only why not just use xz format instead of zipfile_with_lzma_coding

[18:09:43] <ionelmc> does pip support plugins yet?

[18:10:05] <dstufft> no, we don't support plugins

[18:10:09] <dstufft> and we might never, IDK

[18:10:18] <dstufft> there's not a lot of difference between pip-foo and pip foo

[18:10:24] <dstufft> Ivo: that's a reasonable point

[18:10:25] <Ivo> I will continue to recommend not stuffing all this semantic information into filenames

[18:10:39] <dstufft> Ivo: we have tos tuff it in filenames

[18:10:42] <dstufft> it's all we have available

[18:10:57] <dstufft> besides, putting the compression into a filename isn't unreasonable

[18:13:50] <dstufft> Ivo: oh and why use zipfile_with_lzma instead of just xz, because zipfile is a container format, xz as a container format doesn't support multiple files, it only holds a compressed stream

[18:13:56] <dstufft> so you have to pair xz up with something like tar

[18:14:13] <Ivo> ok yeah tar.xz

[18:14:44] <Ivo> maybe in practice there isn't even much difference in compression

[18:14:52] <dstufft> yea we'd want to test it

[18:15:44] <dstufft> FWIW zipfile compresses each indidivdual file on it's own, whereas .tar.xz compresses all of the files at once. The difference there is that I'm pretty sure .tar.xz will get better compression, but zipfile makes it easier to reach in and decompress just one file

[18:16:38] <Ivo> conventionally the overarching purpose of wheels would be to unpack them entirely

[18:17:48] <Ivo> I kinda want to think about an sdist 2.0 before a wheel 2.0

[18:17:51] <dstufft> sure, I'm not saying that switching to tar wouldn't be reasonable, although it's not unreasonable to think that Python might at some point support importing xz/bz2 from a zipfile since it should be as easy as supporting other compression types vs needing to write a compressed tar importer

[18:18:07] <Ivo> because the former could inform the latter

[18:18:12] <dstufft> and keeping it with zipfile makes it easy to keep the same code for handling wheels

[18:18:23] <dstufft> whl1 vs whl2 wheels

[18:22:28] <Ivo> deciding on a compression / packing method for a new binary format is more bikeshedding than anything, not that I'm not guilty of it

[18:23:33] <dstufft> most thins are bkeshedding

[18:26:50] <Ivo> G20 #BikeSheddingOverHowToAddressGlobalWarming

[18:29:01] <dstufft> lol

[18:29:09] <dstufft> I think PyZipfile supports LZMA

[18:29:13] <dstufft> but zipimport doesn't

[18:29:14] <dstufft> D:

[18:32:21] <Ivo> TIL cowsay -f sodomized is a thing

[21:07:18] <ionelmc> dstufft: where are you guys releasing pip1.6?

[21:07:22] <ionelmc> *when

[21:07:26] <dstufft> dunno

[21:07:40] <ionelmc> in what state is the devel branch?

[21:09:45] <dstufft> the dev branch is generally kept pretty releasable

[21:10:46] <Ivo> tests pass, at least

[21:12:38] <Ivo> dstufft: if you want dependency_links to kick the bucket, have you had any thought about enabling people to host interdependant packages in source control through any other means / method?

[21:34:37] <tomprince> Put up an index with vcs links.

[21:47:49] <Ivo> tomprince: does everybody really have to host their own http server, together with some tool for compiling links, for a couple of packages?

[21:49:36] <tomprince> It could be a gist or in a repo (accessed raw)

[21:50:48] <Ivo> and because there is no md5 pip will constantly spit warnings that you need to use --allow-unverified

[21:51:56] <tomprince> Not for vcs hosted packages, which is the only real use caes I can see for dependency_links.

[22:03:20] <dstufft> Metadata 2.0 includes the idea of direct references

[22:04:48] <dstufft> which is like dependency links except not dumb

[22:04:49] <dstufft> also they'll be blocked on uploads to PyPI

[22:04:49] <dstufft> so you can't use a direct reference on something you publish to PyPI