Log file Viewer

[02:10:04] <nicksloan> my inclination is that I don't want to install a bunch of stuff from source on production servers, but not everything has wheels yet. What is the state of the art for deploying python applications and their dependencies? A friend of mine wraps up an entire virtualenv in a .deb, but I'm not sure how I feel about that, and I'd love to hear about alternatives

[02:13:50] <nicksloan> my thought was running a proxy to pypi that caches my dependencies as wheels, and adding my project to that, but I'm wondering if anyone has already done the work of making something like that happen, and if there is an even better approach

[02:23:00] <dstufft> nicksloan: there's devpi and brandon the wheel builder

[02:23:14] <dstufft> https://pypi.python.org/pypi/devpi-builder/

[02:26:10] <nicksloan> dstufft: yeah, my googling just led me to devpi

[02:26:18] <nicksloan> don't know why I didn't see that at work earlier

[02:39:06] <nicksloan> the one thing I like more about the deb approach is not having to futz with virtualenvs when provisioning.

[04:55:33] <tomprince> nicksloan: https://glyph.twistedmatrix.com/2015/03/docker-deploy-double-dutch.html

[13:37:24] <nicksloan> tomprince: that seems like a nice approach. My goals are two-fold: stop building stuff on production and easily run multiple versions of our API so that we can update without breaking client projects.

[13:39:02] <nicksloan> there are a million ways to do both, and everything we've talked about (building debs, devpi, docker) will solve the problem. Just weighing the pros and cons now, and trying to balance that with limited time.

[18:21:30] <nicksloan> weird stuff happening when I call python setup.py install on my project: https://gist.github.com/nicksloan/48dead9a1fcb72d356c9

[18:21:44] <nicksloan> searches for requests, finds requests-oauthlib instead somehow

[18:22:19] <nicksloan> I've done this a bunch and never noticed this before

[18:22:35] <Wooble> :/

[18:23:49] <nicksloan> despite that presumably requests is already installed by the requests-oauthlib dependency that successfully installed previously

[18:26:33] <Wooble> maybe I should have removed the lxml dependency before trying to reproduce. slooooow.

[18:27:02] <nicksloan> haha, yeah

[18:27:53] <nicksloan> I paired it down quite a bit already, but didn't want to remove anything that was in the output, or modify the relevant chunk of output

[18:27:57] <nicksloan> *pared

[18:28:44] <nicksloan> the weird thing is, I've run this setup.py a billion times and had it work perfectly. Wondering if something is going on with pypi at the moment.

[18:29:03] <dstufft> nicksloan: this just started happening?

[18:29:07] <Wooble> it worked fine under a python 3 venv for me; same error as you with 2.7

[18:29:26] <Wooble> but "pip install requests" works.

[18:29:51] <nicksloan> dstufft: yes. This is the first build I've done today, though I've tried it several times in the past hour

[18:34:31] <dstufft> nicksloan: you didn't happen to change the order at all did you? putting requests-oauthlib first makes it work -.-

[18:35:14] <nicksloan> dstufft: nope. Haven't moved those in probably more than a year.

[18:36:45] <nicksloan> it also works if I run setup.py install a second time

[18:36:53] <dstufft> we did just make a change on PyPI, but I have no clue in the world how it would affect this

[18:37:02] <dstufft> we switched the backing store from the local disk to S3

[18:37:09] <dstufft> but only for new uploads

[18:37:13] <dstufft> not for existing files..

[18:37:19] <ionelmc> nicksloan: sounds like you have some local side-effect if it succeds on the second run

[18:38:25] <nicksloan> ionelmc: I'm wondering if it is simply not adding requests to the queue the second time because it sees that it is installed already

[18:38:46] <dstufft> well it does yea

[18:38:51] <nicksloan> because after all, it is installed. requests-oauthlib gets installed first and it depends on requests

[18:39:01] <nicksloan> technically, I could remove requests altogether

[18:39:02] <dstufft> it sees "requests==oauthlib-wahtever"

[18:39:07] <ionelmc> nicksloan: maybe your environment is corrupted?

[18:39:24] <ionelmc> weird stuff happen when you have eggs or leftovers of thereof

[18:39:30] <nicksloan> ionelmc: deleted the virtualenv and recreated it a moment ago

[18:39:45] <dstufft> I can reproduce too

[18:39:46] <nicksloan> ionelmc: plus I think Wooble and dstufft have reproduced it

[18:39:47] <ionelmc> nicksloan: and you're not using --system-site-packages right?

[18:39:52] <ionelmc> oh

[18:39:53] <nicksloan> ionelmc: nope

[18:39:57] <ionelmc> ignore me then :)

[18:40:56] <nicksloan> but what if you have a good idea?

[18:40:59] <nicksloan> :-)

[18:43:48] <Wooble> IIRC there have been problems with having a package and something that depends on it as dependencies in setup.py for a long time... but that wouldn't explain why it was working before. :/

[18:45:07] <nicksloan> hmm

[18:45:51] <dstufft> nicksloan: so

[18:45:55] <nicksloan> that seems a bit problematic... I depend on requests and requests-oauthlib independently of one another, I think it's helpful to express that

[18:45:59] <dstufft> nicksloan: I can see inside of setuptools why it's happening

[18:46:09] <dstufft> or rather, the code that causes it to happen

[18:46:17] <dstufft> I have zero idea why it just suddenly stopped working

[18:46:40] <dstufft> looking at the code here, I'm surprised it ever worked

[18:47:06] <nicksloan> dstufft: it occurs to me that I normally install with `pip install -e .` rather than setup.py install

[18:47:30] <nicksloan> I assumed that did the same thing behind the scenes, but am I mistaken?

[18:47:41] <dstufft> nicksloan: so it's possible ``setup.py install`` has just been broken and you just never did it?

[18:47:53] <nicksloan> dstufft: yeah, totally

[18:48:08] <dstufft> nicksloan: ``python setup.py install`` uses setuptools to do the look up, ``pip install <wahtever>`` uses pipt o do the look up

[18:48:19] <dstufft> this looks like a bug in setuptools then

[18:48:41] <nicksloan> interesting

[18:49:01] <nicksloan> which jives well with what Wooble said

[18:49:17] <dstufft> or maybe just an unfortinate side effect of how setuptools works

[18:49:37] <dstufft> https://github.com/jaraco/setuptools/blob/master/setuptools/package_index.py#L584-L591

[18:49:39] <dstufft> that bit of code

[18:49:48] <dstufft> it looks in the local environment before it looks on the index

[18:50:02] <dstufft> but it's not smart enough to figure out that requests-oauthlib-whatever isn't requests

[18:50:19] <dstufft> the package index code things that it's finding requests==oauthlib-wahtever

[18:50:20] <nicksloan> changed the build script to use pip install

[18:50:36] <dstufft> but then the distribution loading code is smart enough to see that it's requests-oauthlib==whatever

[18:50:43] <dstufft> so it bombs out

[18:50:52] <nicksloan> interesting

[18:51:26] <nicksloan> I guess I just assumed that pip was just an interface on top of setuptools code

[18:52:36] <nicksloan> what do you know, pip fixed it

[18:57:41] <nicksloan> dstufft: should I file a bug on setuptools?

[18:57:53] <dstufft> nicksloan: yea probably

[19:15:25] <nicksloan> dstufft: https://bitbucket.org/pypa/setuptools/issue/381/setuptools-thinks-a-package-name-with-a Have I described this well?

[19:15:44] <dstufft> nicksloan: yea

[19:16:53] <nicksloan> cool. Thanks for the help. I would have been scratching my head for a while on that one, but it makes perfect sense now.

[21:40:43] <jessamynsmith> carljm nedbat I updated pipwrap https://pypi.python.org/pypi/pipwrap