pmxbot IRC Log Viewer

[09:38:46] <pombreda> Hiya :)

[09:39:15] <pombreda> with some small tears I dumped my beloved buildout today to use pip, wheels and a simple configure script

[09:40:09] <pombreda> the result of using wheels is a very significant speedup wrt to buildout. using simple config scripts for extra config means I use full python and shell rather than recipes

[11:30:50] <juhaj_> Any ideas why the following fails: 1) compile and install a custom python, 2) pip install virtualenv into that python tree, 3) virtualenv --system-site-packages here? It fails because: ImportError: No module named time when installing setuptools, pip

[11:40:47] <pombreda_> juhaj_: just curious what is the purpose of having a virtualenv that is not virtual?

[11:42:38] <pombreda_> juhaj_: not sure this may help, but this travis script does build a python frmo scratch, install venv, pip et al. then does some wheeling: it may help? https://github.com/pombreda/thirdparty/blob/master/.travis.yml

[11:42:50] <pombreda_> *from

[11:57:41] <pombreda_> juhaj_: are you sure your virtualenv command is using your custom python and that this custom built python is actually functional?

[12:24:57] <juhaj_> pombreda_: Yes it is, except possibly for its setuptools, that appear not to work properly since the virtualenv fails

[12:26:18] <juhaj_> pombreda_: I don't understand your question about virtualenv not being virtual. The custom python is "system python" in the sense that it is managed, compiled and installed by root, not me, so in that sense I am in exactly the same situation as with a system python, except the admin does not want to replace system python 2.6 with 2.7 (which is probably smart idea)

[12:28:41] <pombreda_> juhaj_: what configure args do you use in your python build?

[12:30:28] <pombreda_> juhaj_: and why would you want to use the --system-site-packages of this python btw? (which why I was asking for your venv being not really virtual and isolated if it uses system site packages?)

[12:31:01] <pombreda_> IMHO do not use --system-site-packages if this possibly broken

[12:31:27] <pombreda_> do not install venv or anything in that custom python

[12:31:35] <juhaj_> Same error without --system-site-packages

[12:32:05] <juhaj_> There *is* virtualenv, pip in the custom python, just like system python. Why should there not be?

[12:32:23] <juhaj_> Python itself is configured with no parameters other than --prefix

[12:32:31] <pombreda_> juhaj_: you are having problems and these are fast evolving

[12:33:09] <pombreda_> personnally no pythons I use has _anything_ in the system site packages

[12:33:21] <pombreda_> thing only ever exist in a venv

[12:33:44] <pombreda_> that is IMHO a sane way to go and the whole purpose of virtualenv

[12:34:07] <pombreda_> especially lately when there is a new release of setuptools about every other minute ;)

[12:34:08] <juhaj_> That's not practical on a system where users just want "a standard python"

[12:34:30] <pombreda_> juhaj_: venv is not a standard python :)

[12:34:38] <juhaj_> But their idea of standard includes all kinds of funny stuff, which is NOT standard, i.e. ends up in site-packages

[12:35:23] <juhaj_> I don't care if there is no virtualenv.py in the "system" python, but there is and I cannot remove it either. Should I try overriding that with my own script or something?

[12:35:37] <pombreda_> juhaj_: you seem to be stuck in a bad place then. you cannot have both a pristine system python and extra packages in the system packages at the same time if you want to stay sane ;)

[12:35:56] <juhaj_> And anyway, the setup works perfectly on a thousand machines with *system* python, but not with this custom python, so there must be something wrong with the custom version

[12:36:22] <juhaj_> Sorry? There are no funny packages in /usr/.../site-packages

[12:36:27] <pombreda_> juhaj_: just install venv using the venv.py script which what I do here: https://github.com/pombreda/thirdparty/blob/master/.travis.yml

[12:36:36] <juhaj_> Just in /custom/python/prefix/.../site-packages

[12:36:49] <pombreda_> juhaj_: I meant that

[12:37:09] <pombreda_> if your /custom/python/prefix/.../site-packages does not work, do not use it

[12:37:33] <pombreda_> instead install your own venv with a fresh virtualenv.py

[12:38:01] <pombreda_> juhaj_: this way you will not depend on these site-packages at all

[12:38:05] <juhaj_> Is it /custom/python/prefix/.../site-packages which does not work? The error comes from import time!

[12:38:13] <pombreda_> no idea

[12:38:42] <pombreda_> juhaj_: use a pastebin with more details

[12:39:19] <juhaj_> All the packages in that site-packages that users normally use (numpy, scipy, ...) work fine.

[12:39:24] <pombreda_> but again, if your custom python works and you can import time , then the issue is /custom/python/prefix/.../site-packages. So do not use it.

[12:39:54] <pombreda_> or use it but do not use the venv that is in it

[12:40:02] <pombreda_> try things

[12:43:52] <juhaj_> http://paste.debian.net/116539/

[12:44:59] <juhaj_> I've tried a zillion different things, but not grabbing an external (external to the custom python that is) virtualenv.py. Installing venv without site-packages gives the very same error, so the problem can't be there

[12:49:28] <juhaj_> Oh, one further note: virtualenv --no-pip --no-setuptools /tmp/myvenv works fine

[12:50:09] <pombreda_> juhaj_: use it then? otherwise try to trace venv with -vvv

[12:50:09] <juhaj_> So I'll just need to manually install setuptools and pip into my venv.

[12:50:33] <juhaj_> There must be something wrong in the custom python to make installing a virtualenv using it not work

[12:55:09] <juhaj_> If I diff the __init__.py that the custom python virtualenv creates in .../distutils/ it is quite different from what the system python's creates. Interesing.

[12:57:29] <pombreda_> juhaj_: try a trace of venv with -vvv

[20:51:30] <phschwartz> Is there an api doc for pip anywhere? Figured I asked before I dived into the source

[21:08:37] <nanonyme> phschwartz, you mean for programmatically using pip?

[21:09:04] <phschwartz> nanonyme: correct

[21:09:15] <nanonyme> I recall dstufft saying a While back that it's not a supported use-case

[21:09:47] <nanonyme> But that calling the main() function in some pip module allows you to pass the same things you would pass on command line

[21:11:10] <phschwartz> nanonyme: I could do it that way, but would prefer to use it in a more finite way. I am working on a tool to create a self extracting wheelhouse by pulling down reqs and I want to use some of the pip internals to do it.

[21:20:35] <nanonyme> phschwartz, https://github.com/pypa/pip/tree/develop/docs looks like there's some docs anyway

[21:21:15] <dstufft> we do not officially support import pip and using it like that, now lots of people do and we generally try not to break compat, at least too much

[21:22:07] <nanonyme> dstufft, maybe there should be a subset of the API that actually has a guarantee of staying stable?

[21:22:39] <dstufft> it's been asked before, for Chef's use cases

[21:22:58] <dstufft> I believe our answer was, we aren't opposed to doing that, if someone wants it they should open a ticket asking for a particular API to be made public

[21:23:21] <dstufft> but we're not going to just going around making APIs public at this point without specific use cases for them

[21:23:34] <nanonyme> Yeah, I agree, such a thing needs use-cases from API-user side

[21:25:54] <nanonyme> dstufft, would tracking down that in the form of a ticket make sense?

[21:26:09] <dstufft> I think there was a ticket

[21:26:10] <dstufft> sec

[21:27:33] <dstufft> https://github.com/pypa/pip/issues/924

[21:31:29] <phschwartz> nanonyme: I will look there

[21:31:50] <phschwartz> dstufft: As I work on our code, I will work out use-cases for our need and open an issue to discuss it.

[21:41:38] <Fnatic9910_> hello

[21:41:41] <Fnatic9910_> jkdjf

[21:41:41] <Fnatic9910_> dfd

[21:41:42] <Fnatic9910_> fd

[21:41:42] <Fnatic9910_> sfds

[21:41:42] <Fnatic9910_> f

[21:41:42] <Fnatic9910_> df

[21:41:43] <Fnatic9910_> dsf

[21:41:43] <Fnatic9910_> df

[21:41:44] <Fnatic9910_> d

[21:41:44] <Fnatic9910_> f

[21:41:45] <Fnatic9910_> fds

[21:41:58] <Fnatic9910_> GSG

[21:41:59] <Fnatic9910_> G

[21:42:01] <Fnatic9910_> S

[21:42:02] <Fnatic9910_> S

[21:42:03] <Fnatic9910_> S

[21:42:04] <Fnatic9910_> S

[21:42:05] <Fnatic9910_> Q

[21:42:07] <Fnatic9910_> Q

[21:42:25] <buck11> ty

[21:42:27] <dstufft> welp

[22:18:34] <mitsuhiko> dstufft: it's a good thing

[22:18:45] <mitsuhiko> dstufft: fork() and threads do not work together at all

[22:18:59] <mitsuhiko> fork is a ridiculously stupid syscall and i can't believe it's kept alive so much

[22:19:10] <dstufft> mitsuhiko: Heh, like I said, Maybe I'm crazy fork worries me less than threads

[22:19:12] <mitsuhiko> and to be honest; i largely blame ruby, perl and python for that

[22:19:19] <mitsuhiko> dstufft: may i ask why?

[22:20:42] <dstufft> mitsuhiko: so fork shares the memory of the original process right? but anything after words gets it's own memory, so if you're careful and fork early then the "Shared memory" segment is quite small (and if you take care to copy stuff over, there is no shared memory), whereas thread you always share memory so the entire time you have to be carefuil about concurrent acess

[22:20:50] <dstufft> note I may be totally wrong, I've never done fork or thread based

[22:21:02] <dstufft> I've just been looking at fork lately for a python based server thing :D

[22:21:13] <dstufft> after forks*

[22:21:23] <mitsuhiko> dstufft: there is no sharing

[22:21:28] <mitsuhiko> fork uses COW

[22:21:33] <mitsuhiko> but here is the killer

[22:21:40] <mitsuhiko> each python object has a refcount in cpython

[22:21:56] <mitsuhiko> so in the next few seconds after the fork you will probably have touched most objects

[22:22:00] <mitsuhiko> and copied all pages over

[22:22:19] <mitsuhiko> the only thing you probably did not touch memory wise was the interpreter executable itself and maybe some things like libc

[22:22:31] <Alex_Gaynor> fork is seriously the worst.

[22:22:35] <Alex_Gaynor> posix_spawn.

[22:22:51] <dstufft> like for my server, the thing I really want is spawn N processes which are self contained

[22:23:03] <mitsuhiko> dstufft: then spawn N separate interpreters

[22:23:05] <Alex_Gaynor> posix_spawn

[22:23:06] <mitsuhiko> and don't make them share anything

[22:23:14] <mitsuhiko> and pass your socket in

[22:23:18] <mitsuhiko> dstufft: actually. forget everything and just use circus

[22:23:19] <phschwartz> Alex_Gaynor: ++

[22:23:37] <mitsuhiko> write your server, pass fd over, open socket by fd, accept like a happy person

[22:23:52] <mitsuhiko> works, survives restarts without dropping a single request, no weird mechanics, works on windows

[22:24:56] <dstufft> well I was going to open the socket itself in the subprocess and use SO_REUSEPORT

[22:25:54] <mitsuhiko> you can do that too

[22:26:02] <mitsuhiko> in that case you don't even need circus

[22:26:11] <mitsuhiko> (but it's a bit harder to use and can be error prone)

[22:26:15] <mitsuhiko> also less portable :)

[22:32:22] <dstufft> probably I can support the circus use case, I don't mind it, but it's nice to have it all built in too

[22:38:17] <dstufft> oh no, I don't think it makes sense, circus seems to want you to put nginx or something in front

[22:56:00] <mitsuhiko> dstufft: ?

[22:56:09] <mitsuhiko> how is that related? :)

[22:56:33] <dstufft> mitsuhiko: oh, I don't want nginx or something else in front

[22:56:39] <mitsuhiko> then don't put it there

[22:56:41] <mitsuhiko> there is no requirement

[22:56:50] <dstufft> oh, the deployment docs said to do

[22:57:04] <mitsuhiko> because it would be crazy not to put nginx in front of any python webserver

[22:57:32] <mitsuhiko> but since your requirement says "no nginx" circus without nginx is as bad as gunicorn without nginx

[22:57:49] <dstufft> mitsuhiko: I think twisted would disagree with that statement :D

[22:57:58] <mitsuhiko> who runs twisted without nginx in front?

[22:58:36] <dstufft> I think that's fairly common

[22:58:52] <mitsuhiko> dns load balancing is not particularly nice

[22:59:02] <mitsuhiko> most people balance behind nginx these days

[22:59:04] <tos9> everyone I've seen puts nginx in front

[22:59:12] <tos9> I run one server with just twisted with no nginx

[22:59:15] <tos9> but it's an rpi

[22:59:39] <mitsuhiko> i run everything behind nginx, even on single boxes, just to be able to block out crappy requests

[23:03:15] <dstufft> hrm

[23:03:58] <dstufft> mayhbe i'm jsut wrong :D

[23:05:12] <Alex_Gaynor> Why would you need nginx in front of twisted?

[23:05:14] <Alex_Gaynor> What does nginx do?

[23:09:31] <tos9> buffer requests

[23:09:51] <Alex_Gaynor> Buffer what about the requests?

[23:10:13] <tos9> well, you can restart without dropping some

[23:10:32] <tos9> or if there are some long-running requests, you still can accept some more

[23:10:49] <tos9> at least, that's what we use it for. but there isn't really a good twisted pre-fork server yet anyhow, so.

[23:25:02] <glassresistor> ran across an issue with use_2to3 = True, if say there is a syntax error in the code it fails to run 2to3 but doesn't error because of that and instead will not run 2to3

[23:26:04] <glassresistor> so usually i get an error related to a piece of code not being python3 compatible rather than the actual syntax error

[23:26:29] <glassresistor> in general its very hard to develop code and test it when there is a build process on install

[23:26:46] <glassresistor> i have to currently change the code run install and then test

[23:29:48] <dstufft> glassresistor: there's not much that can be done about that, it's why 2to3 is generally not used

[23:32:05] <glassresistor> dstufft: im kindov stuff with it since im porting django-oauth2-provider

[23:32:27] <dstufft> generally people use single source code bases that run on both 2 and 3 instead of 2to3

[23:32:32] <glassresistor> dstufft: which part? i think the error could be handled better

[23:32:55] <glassresistor> i get that making development with 2to3 easier might be hard

[23:33:24] <glassresistor> dstufft: i get that in this case i'm not trying to rock the boat

[23:33:46] <glassresistor> dstufft: i can probably get it merged in this way, making it work in both is probably too hard atm

[23:34:14] <glassresistor> but it should be possible to make it show the error that keeps it from running 2to3

Log file Viewer

Help | Karma | Search:

#pypa logs for Wednesday the 20th of August, 2014