pmxbot IRC Log Viewer

[09:15:38] <hellp> It seems pip doesn't use its cache when installing something from a custom web server via --find-links. Correct? Can I configure it to do so? Does my web server has to set specific headers?

[11:08:40] <dstufft> hellp: It uses the standard HTTP cache mechanisms, so it's looking for CacheControl headers, ETag headers, LastModified headers etc

[11:09:49] <dstufft> hellp: any combination of that will cause pip to start using it's cache to varying degrees, the CacheControl header wll cause pip to cache things for X seconds (as defined in the CacheControl header, except with a maximum of 10 minutes for non package files), the Etag and Last-Modified headers will cause pip to attempt to use conditional GETs to download things instead of uncdontionally fetching whenever the cache expires

[11:19:26] <hellp> dstufft: thx for the info. I'm using nginx with the simple "autoindex: on" directory listing feature. No Etags, but Last-Modified reports the correct timestamp for the file (2 months old in this case). Should that be enough for pip?

[11:20:08] <dstufft> hellp: you probably want to include something like expires 1d; or however long you want the file cached for

[11:21:42] <hellp> dstufft: ok, I'll look into it. My main concern was that it wasn't supported for --find-links at all.

[11:22:00] <dstufft> It should work yea, it just needs all the right headers

[13:17:48] <malinoff> so can I somehow get a wheel/tar.gz from already installed library without downloading it again? consider pip cache is invalidated

[13:19:08] <apollo13> malinoff: I don't think so

[13:20:12] <malinoff> apollo13, that's sad. I can understand all the restrictions like compiled c extensions etc

[13:20:35] <apollo13> I don't really see a usecase for it to be honest

[13:21:16] <malinoff> imagine a bunch of machines with no internet access but ssh, and a controller host which has library X installed

[13:21:31] <malinoff> i want this library X to be present on all controlled machines

[13:21:58] <ronny> malinoff: there is no tool to do that

[13:22:02] <apollo13> malinoff: so?

[13:22:26] <ronny> malinoff: you could go ahead and strip the pyc files out and use the metadata to reconstruct a zip file, however that seems fragile

[13:22:27] <apollo13> malinoff: one usually has __one__ build maschine or something which can generate artifacts

[13:22:45] <ronny> malinoff: its best to generate artifacts and put them into a folder

[13:23:17] <malinoff> apollo13, i know how to work around this, e.g. simply download this lib X package again, but i was looking for other options

[13:24:22] <ronny> malinoff: bascially there is no premade option, naything but re-download will loose you more time than you want to loose ^^

[13:24:25] <ronny> *any

[13:24:48] <malinoff> ronny, :) however, i should probably spend some time on pex, looks promising

[13:25:18] <malinoff> i won't save me from downloading this lib again although

[13:25:23] <malinoff> s/i/it

[13:25:51] <ronny> pex is a mess ^^

[13:26:08] <malinoff> what's wrong with it?

[13:26:15] <malinoff> please, dont say everything :)

[13:26:45] <ronny> let me make a list of everything then :P

[13:27:52] <malinoff> i thought pep 441 is right about adding the same thing directly into python

[13:27:55] <ronny> malinoff: but bascally it throws dependencies in as eggs and unpacks

[13:29:00] <malinoff> ronny, ^

[13:29:59] <ronny> pep441 looks semi-interesting

[13:30:51] <ronny> pf_moore: a global zipapp module seems bad,

[13:31:46] <pf_moore> ronny: why? And what would you suggest instead? (Note that PEP 441 is accepted and implemented, so it's a bit late...)

[13:31:50] <apollo13> hmm? I couldn't care less about yet another module in the stdlib :þ

[13:32:05] <ronny> true

[13:34:36] <malinoff> ronny, so to solve my issue, I can imagine the following workflow: ssh to remote machine, pip install virtualenv; create a virtual env; install the library within; further work is within this venv; is it ok? what is the best place for such venv?

[13:34:58] <ronny> pf_moore: how can one use it to package a app from a number of wheels?

[13:36:19] <pf_moore> You can't. It's deliberately minimal. But how is packaging an app from wheels hard? You just unzip all the wheels, add your driver script as __main__ and zip he whole lot up.

[13:37:12] <pf_moore> zipapp is really just providing a utility for doing that in simple cases where everything is already unpacked, more as publicity for the feature that's been around since Python 2.6, than as a full-featured tool

[13:39:05] <pf_moore> ronny: The fun bit is finding the wheels, downloading them, doing dependency stuff, etc.

[13:39:34] <ronny> pip install --download && pip wheel ?

[13:39:55] <pf_moore> Well, just pip wheel :-)

[13:40:12] <pf_moore> And given that you don't actually want wheels, pip install -t.

[13:41:07] <ronny> meh :(

[13:41:13] <pf_moore> I keep thinking I should write a script that takes a requirements file, does pip install -t DIR -r requirements.txt, adds a __main__.py and zips it up

[13:41:26] <pf_moore> But it's so trivial to do by hand I've never bothered...

[13:41:55] <pf_moore> And there's always corner cases that mean I want something slightly different, so "by hand" ends up being better...

[13:42:15] <pf_moore> It's not the doing it that's hard, it's the UI design for making it general.

[13:42:21] <pf_moore> Like everything :-)

[13:42:49] <apollo13> which begs the question why zipapp got added to the stdlib instead of pypi ;)

[13:42:56] <apollo13> cause as you say, it is soooo trivial

[13:43:10] <ronny> from my pov this is broken, things like that shouldnt go to the stdlib like that

[13:43:25] <apollo13> that ship has sailed

[13:43:28] <ronny> yes

[13:43:43] <pf_moore> apollo13: To publicise the feature of running zip files, basically. Nobody knew about it and kept inventing their own versions

[13:44:07] <pf_moore> If we could have found a way of just documenting it so people would notice, we probably would have.

[13:44:09] <apollo13> I knew about it, I just wish mod_wsgi would allow me to run a zip file :)

[13:44:22] <apollo13> like a war file :þ

[13:44:34] <ronny> apollo13: mod-wsgi_express probably does ^^

[13:44:52] <ronny> apollo13: however normal mod_wsgi must fit apache first, python second

[13:44:52] <pf_moore> Can't help there I'm afraid - I pretty much agree that would be neat (given taht I never use mod_wsgi :-))

[13:45:10] <apollo13> pf_moore: well, same applies to gunicorn, uwsgi and whatnot

[13:45:19] <apollo13> python needs an application container format

[13:45:20] <pf_moore> Couldn't you add the zip file to sys.path and then just call a main function?

[13:45:49] <apollo13> oh I knew that I could run from zip files, but adding them to sys.path? lemme try that

[13:46:02] <apollo13> that would make my deployments so much nicer

[13:46:13] <doismellburning> apollo13: yes, yes it does

[13:46:15] <apollo13> no virtualenv anymore, just one zip file

[13:46:30] <ronny> apollo13: pure python?

[13:46:30] <doismellburning> apollo13: (so do most languages that aren't Java)

[13:46:55] <apollo13> ronny: na, c extensions too, but that shouldn't matter

[13:47:13] <ronny> do c extensions work fro ma zip import?

[13:47:22] <apollo13> let me try :)

[13:47:24] <dstufft> apollo13: I think you could probably put gunicorn inside of a zip file

[13:47:26] <dstufft> No they don't

[13:47:32] <apollo13> oh, why not :(

[13:47:44] <apollo13> what is the difference to normal py files?

[13:47:47] <ronny> apollo13: dlopen has no zipimport

[13:47:51] <dstufft> ^

[13:47:59] <apollo13> I thought it was extracted anyways

[13:48:00] <apollo13> hrmpf

[13:48:08] <dstufft> in eggs it's extracted

[13:48:12] <ronny> setuptools does that for eggs

[13:48:13] <dstufft> not in zip files

[13:48:42] <apollo13> mhm, so wheels + venv is still the best option

[13:48:57] <dstufft> or don't use C files

[13:48:58] <ronny> dstufft: btw, can we teach setuptools to download pure wheels instead of eggs if aviliable?

[13:49:02] <dstufft> C files are bad anyways

[13:49:08] <apollo13> dstufft: psycopg…

[13:49:09] <ronny> (for setup_reuqires that is)

[13:49:14] <dstufft> preach the controversy

[13:49:24] <dstufft> apollo13: databases are bad too!

[13:49:31] <apollo13> dstufft: …

[13:49:33] <ronny> apollo13: just install the one c lib into the python, then put the rest into the zip

[13:49:44] <dstufft> ronny: I dunno, you'd have to ask jaraco I guess

[13:49:48] <pf_moore> apollo13: See what I mean about the features not being well known? :-)

[13:49:49] <ronny> i see

[13:50:04] <ronny> dstufft: i suspect its best supplied via pull request ^^

[13:50:08] <dstufft> probably

[13:50:13] <apollo13> pf_moore: in fairness I knew about it, but never really had a usecase, hence never looked in detail

[13:50:14] <pf_moore> But yeah, if you have C extensions you can't run them from zipfile. That's a platform limitation (all platforms) not a Python one

[13:50:16] <malinoff> sorry for rocking the #pypa board :)

[13:50:41] <dstufft> there are ways of doing it

[13:50:46] <dstufft> sort of

[13:50:54] <dstufft> I think Barry warsaw intends to try and make it work in Python

[13:51:09] <dstufft> probably with varying levels of hacks

[13:51:14] <ronny> hmm

[13:51:28] <ronny> bbl

[13:51:39] <dstufft> Google has something cool - they have their systems patched so that dlopen can take an offset

[13:51:48] <pf_moore> dstufft: yeah, py2exe does it by loading the dll from in-memory, but that's for Windows, and it's a bit obscene...

[13:52:09] <dstufft> so they can dlopen from a zipfile by not compressing the zipfile, and just passing explicit offsets to where the .so is inside of the zipfile

[13:52:16] <dstufft> er, not compressing that part of the zipfile*

[13:52:27] <pf_moore> but I think Barry is looking at extract on demand, which works everywhere but potentially leaves a bit of tempfile clutter

[13:53:45] <pf_moore> dstufft: In theory anywhere could do it by extracting the .so/.dll into memory then memory-mapping that "as if" it were a dynamic lib. It's whether the OS loader lets you do that that's the fun bit.

[13:53:52] <apollo13> that is something I could do in a custom import hook with my main, no?

[13:54:00] <pf_moore> But hey, on Linux you have the source code, so you can code it yourself ;-)

[13:54:17] <pf_moore> apollo13: Sorry, back to reality. Yes you could

[13:54:32] <dstufft> pf_moore: yea, I think probably long term the best way forward on that is to have a list of ways we can support that and select the "best" method based on the platform

[13:54:34] <pf_moore> It's fiddly but doable (it;'s what setuptools does, I believe)

[13:54:35] <apollo13> but? I mean tempdir + atexit handler sounds somewhat sane

[13:54:49] <dstufft> it can be tricky

[13:55:02] <apollo13> btw does sys.path check for .zip directly or will .whl work too?

[13:55:05] <dstufft> if you're extracting you start to hav to care about file permissions on the temp dir

[13:55:12] <ronny> apollo13: whl works, too

[13:55:14] <dstufft> .whl will work

[13:55:20] <pf_moore> apollo13: IIRC (I've never looked at this in detail) the trick is extracting at the right time so that the tempdir is on sys.path when you need it.

[13:55:30] <ronny> apollo13: i use whl import to run virtualenv from a rsynced download folder

[13:55:37] <pf_moore> apollo13: it looks if the file format is correct so filename is irrelevant

[13:55:46] <apollo13> makes sense

[13:56:53] <apollo13> pf_moore: it probably becomes tricky when the so files are hidden deep in the package structure somewhere

[13:57:09] <apollo13> in the end I'd just like to extract the .so files on demand and rewrite imports to them^^

[13:57:34] <apollo13> though I assume could do this by writing a list of so files into my main script + importhook

[13:57:41] <pf_moore> apollo13: That's certainly doable. You'd be best writing your own importer/loader

[13:57:52] <dstufft> (I actually prefer the memory mapped thing better :V)

[13:57:53] <pf_moore> It's not that hard, but there are very few examples.

[13:58:02] <ronny> apollo13: why not just unpack the whole zipfile then?

[13:58:26] <apollo13> would be easier yeah

[13:58:43] <apollo13> but then I wouldn't know how to implement an import hook^^

[14:01:52] <apollo13> that said, zipping up whl files into one zip file + a main which extracts everything seems sane :)

[14:02:49] <dstufft> and then you reinvented pex

[14:03:28] <apollo13> tada :p

[14:03:49] <apollo13> but they seem to need so much code for it :þ

[14:07:07] <ronny> apollo13: they added egg support, eggs are insane

[17:12:50] <ionelmc> am i missing something but there isn't anything stopping you having some .so unpacking code in your pyz __main__?

[17:13:26] <ionelmc> afaik that could very well be a library on pypi

Log file Viewer

Help | Karma | Search:

#pypa logs for Tuesday the 17th of March, 2015