pmxbot IRC Log Viewer

[09:39:31] <pombreda> Hiya :)

[09:39:34] <pombreda> !logs

[09:39:34] <pmxbot> http://dcpython.gekitsuu.org

[11:18:37] <iivvoo> is there a way to make buildout use a "pip freeze" requirements.txt file? Not by copy/pasting it into the [versions] section but by including it somehow?

[11:18:53] <iivvoo> this way I have a single list of versions / requirements that can be used by both pip and buildout

[11:19:01] <iivvoo> (not every hoster supports buildout, e.g. gondor)

[11:32:30] <pombreda> iivvoo: yes, there are a few recipes for this ... hold on a few minutes

[11:33:47] <pombreda> ie you want to be able to run a simple buildout that only provisions components that would use a requirements file, so it could run equally well with pip -r ?

[11:33:52] <pombreda> iivvoo: correct?

[11:34:59] <iivvoo> well no, the buildout may do other stuff (which you'll have to do by hand in a pip-based install, or which may already be present, e.g. postgres, solr, etc)

[11:35:39] <pombreda> iivvoo: we are on the same page :P

[11:35:43] <pombreda> iivvoo: hold on

[11:36:34] <iivvoo> :)

[11:37:49] <pombreda> fwiw this is what I plan to do someday in the not yet coded pipo: http://pypi.python.org/pypi/pipo/0.0.0 :P

[11:40:30] <pombreda> iivvoo: there is one piece of untested abandonware which is not doing what you want: http://pypi.python.org/pypi/buildoutpip/

[11:40:57] <pombreda> iivvoo: and gawels' http://pypi.python.org/pypi/gp.recipe.pip would be the best candidate but it may not support reuqirements file ...

[11:42:52] <pombreda> iivvoo: actually it does:

[11:43:19] <pombreda> install = -r requirements.txt in the gp.recipe.pip section should source the buildout eggs from the req file

[11:44:13] <pombreda> iivvoo: and https://github.com/pombredanne/collective.recipe.pip which I foked from someplace which seems to be gone

[11:44:50] <pombreda> this one is likely what you want and is focused on requirements

[11:46:03] <pombreda> it was originally from https://github.com/bubenkoff/collective.recipe.pip

[11:48:10] <pombreda> iivvoo: it is now here: https://github.com/collective/collective.recipe.pip

[11:48:22] <pombreda> moved there a few days ago, sounds like ..

[11:49:21] <pombreda> iivvoo: that one or gawels is likely doing what you want :)

[11:49:34] <pombreda> iivvoo: please report on your experiments

[11:51:51] <pombreda> iivvoo: there is one flaw in all this: buildout does not use pip to provision packages, so some requirements (the -e ones) may not be processed at all.... this is a buildout flaw imho as pip is now the std Python way

[11:53:38] <pombreda> iivvoo: fwiw my goal with pipo would be to have a mostly transparent tool that does the right thing consistently for you, whether you have requirements, buildout configs or plain setup or a mix of all and always operates within a virtualenv

[11:54:37] <pombreda> there are too many ways of doing things in python land package management ;) let's do one more way to rule them all :D

[13:22:46] <iivvoo> pombreda, thanks for the links / info, I'll look at them

[13:22:50] <iivvoo> 'middag sshanx

[13:25:05] <sshanx> hey

[13:25:14] <sshanx> alles goed?

[13:30:40] <iivvoo> zekers

[13:31:23] <pombreda> sshanx: howdy

[13:31:32] <sshanx> hi

[13:31:32] <pombreda> iivvoo: my pleasure

[13:31:46] <pombreda> you guys are dutch?

[13:32:27] <iivvoo> we are

[13:33:57] <pombreda> neat :)

[13:35:00] <pombreda> iivvoo: sshanx : I am looking for feedback on this : https://github.com/buildout/buildout/pull/51 ... which is to add support for conditions in a buildout section .... so that a python expression can be used to exclude some sections (or parts)

[13:35:19] <sshanx> lemme see

[13:35:20] <pombreda> if you guys can spare a minute to check it out and comment in the pull reuqest that would be appreciated

[13:38:37] <sshanx> is there a more complete example somewhere?

[13:38:51] <sshanx> of how this fits into a current buildout file?

[13:44:23] <pombreda> sshanx: you can see exmaples in the test case here: https://github.com/pombredanne/buildout/blob/a0e6debccf14440504eb1e3214179cb635a7342c/src/zc/buildout/configparser.test#L94

[13:45:33] <sshanx> so the form will be

[13:45:48] <sshanx> [name of section:optional expression]

[13:46:21] <sshanx> so with: [s1: sys and re and os and platform]

[13:46:21] <pombreda> yes, that is it

[13:46:23] <sshanx> s1 is section name

[13:46:31] <pombreda> yes

[13:46:36] <sshanx> so I'm new to this change :)

[13:46:37] <pombreda> and you can do things like:

[13:46:42] <sshanx> don't know what has been discussed already

[13:46:55] <sshanx> but why not reserve a variable in the section itself?

[13:47:02] <pombreda> it is being discusssed here right now :)

[13:47:06] <sshanx> skipIf = expression

[13:47:07] <sshanx> haha

[13:47:22] <pombreda> sshanx: there are no resevred names in variables except for recipe

[13:47:23] <sshanx> yeah but I saw a reference to Jim in the pull request

[13:47:43] <pombreda> sshanx: J1m prefers the proposed design rather than a section option

[13:47:53] <pombreda> so this is what I implemented

[13:48:02] <sshanx> so to make sure that there is no backwards incompatability

[13:48:02] <pombreda> [install_eggs: windows]

[13:48:22] <sshanx> with current buildout confis

[13:48:34] <pombreda> if you do not use them, things work as exactly supported by buildout 2.0

[13:48:52] <pombreda> [install_eggs: windows]

[13:49:09] <pombreda> recipe = zc.recipe.egg

[13:49:15] <pombreda> eggs = pyreadline

[13:49:18] <pombreda> and

[13:49:30] <pombreda> [install_eggs: macosx]

[13:49:33] <pombreda> recipe = zc.recipe.egg

[13:49:35] <pombreda> eggs = readline

[13:49:38] <pombreda> become possible

[13:49:44] <sshanx> yeah

[13:49:54] <pombreda> if you include part = install_eggs only the right egg will be installed

[13:49:54] <sshanx> we now use separate buildouts for these cases :)

[13:50:08] <sshanx> if you configure your buildout correctly of course ;)

[13:50:11] <pombreda> sshanx: that is a pain, right?

[13:51:37] <sshanx> yeah I think expressions like you implemented them would allow for easier to maintain configs

[13:54:11] <sshanx> so what decides what vairables are available in expressions?

[13:54:31] <sshanx> I see that you have made sort of a sane choice about modules being avaialble in the expression context

[13:54:40] <sshanx> how does that work?

[13:58:43] <pombreda> yes, the choice are in this function here:

[13:59:20] <pombreda> sshanx: https://github.com/pombredanne/buildout/blob/a0e6debccf14440504eb1e3214179cb635a7342c/src/zc/buildout/buildout.py#L1347

[13:59:58] <pombreda> four modules are available: os, sys, re and platform

[14:00:08] <pombreda> and a few precomputed common conditions:

[14:00:31] <pombreda> windows and python2 or linux and python 32

[14:00:38] <pombreda> would be a valid expression

[14:00:53] <pombreda> *windows and python2 or linux and python32

[14:05:50] <pombreda> sshanx: the whole being that you can make your buildout simpler and easier to maintain with this

[14:06:03] <sshanx> yeah

[14:06:06] <pombreda> sshanx: or not use it all ... and it has no impact

[14:06:07] <sshanx> which is great!

[14:06:20] <pombreda> *the whole point

[14:06:22] <sshanx> yeah the optionality of it is good (and necessary of course)

[14:06:49] <sshanx> I would personally not go for making such expressions part of the header name

[14:06:55] <sshanx> but that is just my preference

[14:07:02] <sshanx> I don't know what I would do alternatively

[14:07:17] <sshanx> the only thing is

[14:07:33] <sshanx> if you want to make more complex expressions

[14:07:37] <pombreda> the alternative is reserving an option name in all sections ... which could break recipes

[14:07:37] <sshanx> for whatever reasons

[14:07:41] <sshanx> is there some kind of hook

[14:07:56] <sshanx> so you can add your own module in the context of the expression

[14:08:00] <sshanx> yeah

[14:08:12] <pombreda> no hooks are planned, but that is a decent idea

[14:08:53] <pombreda> it should be possible to effectively add an extension that could add more things to the expression eval context

[14:09:05] <sshanx> something like that :)

[14:09:05] <pombreda> sshanx: excellent suggestion

[14:09:27] <sshanx> thanks

[14:09:32] <sshanx> with this infrastructure

[14:09:56] <sshanx> it is also possible to overrule a specific header:expression combo in another buildout file?

[14:10:08] <sshanx> so say you have [my_section: windows]

[14:10:12] <sshanx> and another buildout file

[14:10:25] <sshanx> which also has [my_section: windows]

[14:10:33] <sshanx> it would only overrule that one?

[14:10:43] <sshanx> and not a [my_section: linux] in the master buildout?

[14:11:49] <pombreda> sshanx: more or less with this implementation... it works at the config file parser level ... so each cfg file is processed individually and the general [sections] overriding work as usual ... so if you have two sections with the same name that evaluates to true, the last one loaded will win, as it does in the standard buildout

[14:12:17] <sshanx> yeah

[14:13:01] <pombreda> the last section that evals to true wins, though if you use += to add or -= to remove things and both sections eval to true then they can coeexist I think

[14:13:27] <pombreda> sshanx: it does not change anything of the buildout semantics at all

[14:14:18] <pombreda> if you have two [my_section] today, the behavior with two [my_section: some expression] where both eval to true is exactly the same

[14:14:34] <sshanx> great

[14:15:28] <sshanx> same thing with macros I presume

[14:20:06] <pombreda> sshanx: macros?

[14:20:33] <sshanx> http://pypi.python.org/pypi/zc.buildout/2.0.0b1#extending-sections-macros

[14:22:31] <pombreda> sshanx: :) I did not even knew about this :D

[14:23:15] <pombreda> hold on that is not new to 2.0, that has been there for w hile

[14:23:34] <pombreda> http://pypi.python.org/pypi/zc.buildout/1.6.3#extending-sections-macros

[14:23:40] <pombreda> sshanx: I use it all the times

[14:24:58] <pombreda> sshanx: and macros with conditional expressions becomes quite powerful indeed

[14:25:22] <pombreda> I could rewrite the example above this way:

[14:26:07] <pombreda> [install_readine]

[14:26:08] <pombreda> recipe = zc.recipe.egg

[14:26:23] <pombreda> [install_readline: windows]

[14:26:41] <pombreda> strike that

[14:26:55] <pombreda> rather

[14:27:16] <pombreda> [install_readine: not linux]

[14:27:19] <pombreda> recipe = zc.recipe.egg

[14:27:36] <pombreda> [install_macreadline: macosx]

[14:27:53] <pombreda> eggs = readline

[14:28:08] <pombreda> <= install_readine

[14:28:23] <pombreda> [install_winreadline: windows]

[14:28:25] <pombreda> <= install_readine

[14:28:32] <pombreda> eggs = pyreadline

[14:28:34] <pombreda> there you go

[14:28:57] <pombreda> this will not excecute on linux which has readline builtin

[14:29:09] <pombreda> and install the right egg for mac and windows

[14:29:13] <pombreda> sshanx: :)

[14:29:27] <mgedmin> and you would happily copy'n'paste that into every buildout.cfg you have?

[14:29:51] <mgedmin> why not upload a package called 'anyreadline' to pypi?

[14:30:04] <pombreda> mgedmin: indeed :D

[14:30:27] <sshanx> mgedmin: agree :)

[14:30:35] <mgedmin> with a setup.py that has install_requires = ["pyreadline"] if os.platform == 'winnt' else ["readline"] if os.platform == 'macosx' else []

[14:30:56] <mgedmin> (spelling for platform checks pulled from a hat, do not rely on these)

[14:31:05] <sshanx> pombreda: I think the functionality looks good

[14:31:13] <sshanx> that's all the feedback I can give you for now :)

[14:31:21] <mgedmin> I guess what I'm saying is: I'd like to see a more compelling example use-case

[14:31:30] <sshanx> good luck with getting it pulled into mainline

[14:31:48] <pombreda> sshanx: :)

[14:31:57] <pombreda> sshanx: thx

[14:32:42] <pombreda> mgedmin: the anyreadline name is available on pypi ... go for it , and you will do a great service to the python community

[14:33:40] <pombreda> mgedmin: you solution is simple and brilliant :)

[14:33:43] <pombreda> *your

[14:33:43] <mgedmin> I won't upload it, since I don't have the means to test it (no mac/windows machines)

[14:34:00] <mgedmin> I suggest you do it instead ;)

[14:35:29] <sshanx> lol

[15:43:31] <niluje> he

[15:43:34] <niluje> hey

[15:43:51] <niluje> I want to deploy a Python application which uses SQLAlchemy

[15:44:03] <niluje> The production server has no internet and no compiler

[15:44:21] <niluje> What we plan to do is creating a build server, build the package in a virualenv and copy the virtualenv to the production server

[15:44:35] <niluje> There are two problems with this (working) solution :

[15:44:49] <niluje> 1/ this doesn't seem elegant

[15:45:32] <niluje> 2/ if the build is done in a different path than where it is deployed on the production server, we have to update all the absolute paths contained in the virtualenv

[15:45:53] <niluje> I read some docs about buildout but I'm not sure I understood everything: could it help me to improve the deployment process?

[15:48:43] <mgedmin> maybe, maybe no

[15:49:11] <mgedmin> there's http://pypi.python.org/pypi/zc.sourcerelease that I've never used

[15:49:41] <mgedmin> but it can be used to produce RPMs from buildout-based projects, if you're running a redhat-alike on your server

[15:49:55] <mgedmin> other options: virtualenv --relocatable

[15:50:27] <mgedmin> other options: buildout with an egg cache which you populate manually by scp'ing all the required source and binary eggs so it doesn't need to go online

[15:50:37] <mgedmin> buildout has an offline = yes option which is a bit tricky

[15:51:18] <mgedmin> (a recent mailing list discussion revealed it didn't originally mean "work offline", it meant something else, and if you want "work offline" you should use some obscure option I can't remember augh)

[15:51:24] <mgedmin> (but it generally works)

[15:58:55] <niluje> I'll give a look to zc.sourcerelease

[15:59:10] <pombreda> !log

[15:59:17] <pombreda> !logs

[15:59:17] <pmxbot> http://dcpython.gekitsuu.org

[15:59:18] <niluje> I never used the --relocatable but I've seen everywhere that it is broken

[15:59:36] <pombreda> mgedmin: http://pypi.python.org/pypi/anyreadline/0.1.1 :P with credits to you

[16:00:06] <mgedmin> no python 3 compatibilty? ;)

[16:00:07] <pombreda> mgedmin: but you are a slacker

[16:00:15] <niluje> And about the third solution, I really need to dig deeper with buildout to understand your proposition :p

[16:00:18] <pombreda> mgedmin: submit a pull request :D

[16:00:21] <mgedmin> hey, I used to be a Perl programmer (briefly, 10 years ago)

[16:00:30] <mgedmin> and Larry Wall said laziness was a virtue!

[16:00:53] <pombreda> niluje: sry for wrecking your chat

[16:01:14] <pombreda> mgedmin: hubris is a better virtue

[16:06:13] <niluje> pombreda: it's okay, I'll just conclude that in my case, Python deployment is a mess :p

[16:15:41] <mgedmin> niluje, my gut feeling is that offline buildout usage would be simplest to figure out

[16:16:04] <mgedmin> (but then I've used buildout in the past and feel reasonably comfortable with it)

[16:17:27] <niluje> mgedmin: I'm gonna try this way, but I need to read more doc on buildout

[16:18:09] <mgedmin> IIRC I liked http://jacobian.org/writing/more-buildout-notes/ (this is part two, but it links to part one)

[16:19:11] <niluje> I'll read that, thanks

[16:19:16] <niluje> btw, could you explain me what a recipe is exactl?

[16:19:19] <niluje> +y

[16:19:27] <mgedmin> like a buildout plugin

[16:19:55] <mgedmin> the most commonly used recipe is "install a bunch of python packages from pypi" aka zc.recipe.egg

[16:20:08] <niluje> And, if I understood what I read, what buildout does is installing a project dependencies in the current directory, right?

[16:20:23] <mgedmin> everything buildout installs is confined into the current directory

[16:20:29] <mgedmin> usually in ./parts/

[16:21:01] <mgedmin> (you could write a recipe that touches files outside of the buildout directory, but it would be a naughty recipe)

[16:25:02] <niluje> ok

[16:25:54] <niluje> mgedmin: I'll play with buildout as soon as I have some time

[16:26:17] <niluje> and will explain how I finally did to achieve the deployment

[16:26:21] <niluje> btw : http://serverfault.com/questions/472726/deploy-a-python-project-on-a-server-with-no-compiler-and-no-internet-connection

[16:26:24] <niluje> thanks a lot for your time :-)

[16:28:52] <mgedmin> if you lack time, why not build the virtualenv in the final location on the build server, then rsync it to the same location on your prod server?

[16:32:25] <niluje> mgedmin: that what we gonna do in a first place

[16:59:58] <pombreda> niluje: the way to do what you want to do with buildout is use the download-cache option ... pin all the versions in the versions option then run the buildout once in this mode, then run the buildout with install-from-cache=true

[17:00:53] <pombreda> niluje: I start usually with a venv: python2.6 ./thirdparty/dist/virtualenv.py --no-site-packages --relocatable --never-download --extra-search-dir=./thirdparty/dist/ --distribute .

[17:01:10] <pombreda> with a prefetched venv.py

[17:01:21] <pombreda> then activate

[17:01:51] <pombreda> then boostrap the first time without install from cache

[17:02:00] <pombreda> liekso for instance: python bootstrap.py --version 1.7.0 --distribute --setup-source=./thirdparty/dist/distribute_setup.py --download-base=./thirdparty/dist/

[17:02:14] <pombreda> here the distribute setup is fetch ahead of time

[17:02:30] <pombreda> finally a regular buildout bin/buildout -s -U

[17:02:58] <pombreda> and after install from cache=true, you can reproduce this ok, if you had these configs

[17:03:39] <pombreda> allow-picked-versions = false install-from-cache = true newest = false ... and all versions pinned correctly

[17:04:02] <pombreda> you also need find-links like in find-links = file:///thirdparty/ file:///thirdparty/dist/

[17:04:28] <pombreda> and I use the buildout.dumppickedversions to know which versions were picked and for debug

[17:04:51] <pombreda> niluje: quite frankly running a buildout without connection to the internet is not an easy feat :)

[17:04:55] <pombreda> but it works

[17:05:11] <niluje> I feel like I've been raped

[17:05:27] <niluje> let me a minute to understand what you said :p

[17:05:53] <pombreda> niluje: sry, I assumed you were savvy at using buildout :P

[17:06:48] <niluje> nope, I tried to find a solution for the deployment and I read about buildout. I tried it unsuccessfully for 20 minutes, and went here to ask if that could possibly be a solution :p

[17:07:36] <niluje> 18:00:00 pombreda | with a prefetched venv.py

[17:07:36] <pombreda> niluje: fair enough and this is the way to go :)

[17:07:45] <niluje> what do you mean, "with a prefetched venv.py" ?

[17:08:09] <pombreda> I download and save the virtualenv.py scripts ahead of time

[17:08:19] <niluje> ok

[17:08:22] <pombreda> sry for the shorthand

[17:09:13] <niluje> I really can't give a look now, I'll probably try this week-end and come back for further questions :p

[17:10:45] <niluje> anyway my problem seems tricky and at first sight your solution seems elegant, I suggest you to reply to the question I posted on serverfault, could be useful to someone else!

[17:11:14] <pombreda> no pro

[17:12:06] <pombreda> niluje: gimme the url, not sure my solution is elegant but it works in a repeatable, consistent no network connection way one time and all the times

[17:12:19] <niluje> http://serverfault.com/questions/472726/deploy-a-python-project-on-a-server-with-no-compiler-and-no-internet-connection

[17:12:34] <niluje> yes

[17:12:41] <pombreda> niluje: I had to go through painfull hooks and crannies and do incantations to several gods of pythons to get it to work

[17:13:01] <niluje> it is probably much more difficult to apprehend than just copying a virtualenv

[17:14:09] <pombreda> niluje: virtualenvs are relocatable on one machine, not across machines unless they are _identical_, in general

[17:14:58] <pombreda> niluje: a relacatble venv is not a deployment solution, unless you have a very controlled and limited use case

[17:17:23] <pombreda> niluje: and fwiw my deployments includes setting up web servers, web apps, database services, cron jobs, various queue and other daemons under supervisor, a reporting engine and works across posix OSses

[17:17:42] <pombreda> you cannot get that by just relocating a virtualenv :P

[17:18:08] <niluje> fwiw?

[17:18:20] <niluje> and yeah, I know the limitations of copying a virtualenv

[17:18:32] <niluje> and I also now that for my case, it will work (the build machine will be the same than the production one)

[17:18:48] <niluje> that's why I really prefer to find something better than the quick and dirty thing

[17:19:40] <pombreda> fwiw. For Whatever Is Worth

[17:20:27] <pombreda> niluje: I hang around reasonably often... ping me next time and I _may_ check the SO question

[17:21:51] <niluje> rulez

[17:21:54] <niluje> thanks for your time pombreda :-)

Log file Viewer

Help | Karma | Search:

#buildout logs for Tuesday the 29th of January, 2013