pmxbot IRC Log Viewer

[15:20:53] <tvansteenburgh> hi everyone. i have a package with a dependency that is not on pypi, and cannot be moved there. i would like install with `pip install mypkg` without needing to pass `--allow-external foo --allow-unverified foo`. is this possible?

[15:21:53] <tvansteenburgh> i came up with a gross hack to make it work for my package, but it breaks horribly when my package is included as a dependency of another

[15:29:29] <danito> Hi all!

[15:29:48] <danito> Is it fine to use the pip module to install packages?

[15:30:00] <danito> I don't want to subprocess in my upgrade script

[15:53:00] <danito> What I want to do is: import pip; pip.install('some package') or something alike

[15:54:40] <dstufft> danito: we don't really have an API like that, you can use pip.main([]) though and just pass wahtever you would normally pass in the CLI flags as list items tho

[15:55:06] <dstufft> tvansteenburgh1: not really, it's done that way on purpose. end users should be aware of what servers they are contacting

[15:55:55] <tvansteenburgh> dstufft: i expected as much but figured I'd ask, thanks for the reply

[15:57:14] <danito> dstufft: thanks, that should solve my pip vs pip2 compatibility issues

[16:05:58] <stbatduke> Ohai #pypa! I have broken some things in my "dev" server and I am trying to back things up in order to fix them after a re-install, but I can't seem to get a "pip freeze" of my applications. When I try "pip freeze" I get a traceback stating: "ImportError: No module named urlparse" ... any idea what I broke and how to fix it ?

[16:06:27] <stbatduke> I was trying to get this server to match our production server as far as the debian packages are concerned, and I think something in there is what broke everything

[19:14:38] <tos9> pkg_resources.VersionConflict: (python-dateutil 2.2 (/usr/local/lib/python2.7/site-packages), Requirement.parse('python-dateutil>=1.4,<2'))

[19:14:49] <tos9> Did I imagine that pip used to show what thing was requiring the thing it fails to install at some point

[19:15:33] <tos9> oh. I guess the frame is in the traceback.

[19:15:48] <tos9> But I thought it used to show you -> -> ->... Maybe that's just for requirements files.

[19:15:58] <tos9> I guess I can blame setuptools instead.

[22:01:52] <_habnabit> there's not a way to bundle multiple python packages into a single .whl, is there? i'm just wondering if maybe i could simplify deployment by only having a single .whl with all the dependencies instead of multiple .whls for each dependency

[22:07:09] <dstufft> _habnabit: not currently, there's been mentions of it from time to time but generally wheel is focused on package format not deployment format

[22:07:25] <_habnabit> ah i see

[22:07:30] <dstufft> _habnabit: you could tarball them up and exact them and do pip install unpacked/* --no-deps

[22:07:37] <_habnabit> dstufft, is there a format out there for deployment?

[22:07:44] <dstufft> _habnabit: not currently

[22:07:50] <dstufft> well

[22:07:53] <dstufft> I lie

[22:07:55] <dstufft> there is pex!

[22:08:03] <dstufft> I've never used it personally

[22:08:07] <dstufft> it comes out of twitter

[22:08:11] <Yasumoto> howdy

[22:08:11] <_habnabit> hmm

[22:08:18] <Yasumoto> yah, it's worked ~pretty well for us

[22:08:32] <dstufft> http://pex.readthedocs.org/en/latest/

[22:08:32] <Yasumoto> I can grab wickman (he's in #aurora) if you have questions

[22:08:42] <_habnabit> oh, this looks similar to superzippy

[22:09:03] <Yasumoto> cool, I haven't heard of that but will check it out

[22:09:08] <dstufft> never used that either

[22:09:10] <dstufft> lol

[22:09:31] <dstufft> I haven't heard anyone yelling about pex yet so there's that

[22:09:37] <_habnabit> ha

[22:09:42] <dstufft> which either means noone is using it or it's working really well for everyone :D

[22:10:06] <Yasumoto> lol.... I think it's a mix :)

[22:11:00] <pf_moore> Yasumoto: there's also pyzzer

[22:11:33] <Yasumoto> that's awesome

[22:11:56] <pf_moore> I tried pex at one point, but it didn't work (can't recall why, maybe it's Py2 only?)

[22:12:13] <Yasumoto> ah yah, I haven't used it with py3

[22:12:23] <Yasumoto> _in theory_ it has support, but I haven't vetted

[22:12:42] <_habnabit> man, another project doing the same thing

[22:12:45] <dstufft> my deployments are boring

[22:12:53] <dstufft> so I don't use any of these fancy things

[22:12:58] <dstufft> tho I may use docker

[22:12:59] <dstufft> yay docker

[22:13:14] <dstufft> someone should champion one of these deployment tools into a PEP!

[22:13:57] <Yasumoto> heh

[22:14:01] <Yasumoto> _habnabit: that feel :)

[22:14:08] <Yasumoto> dstufft: hm.. intriguing

[22:14:32] <Yasumoto> fwiw I think it'd be interesting to get more feedback about pex.. it started out as a one-man show for a long time, but SRE has been using it for our tools a lot more

[22:14:42] <Yasumoto> (feedback from the community)

[22:15:06] <dstufft> Yasumoto: ya'll should write some blog posts or something

[22:15:48] <Yasumoto> yep, then we'd (hopefully :) get it out to Python Weekly and stuff

[22:15:55] <Yasumoto> thx for the nudge

[22:16:45] <pf_moore> Oh, yeah, I recall now. pex uses unsetenv and that's not available in Python 3.x on Windows, at least.

[22:17:14] <_habnabit> Yasumoto, i'll try out pex

[22:17:15] <pf_moore> It's a simple patch, but then something else went wrong and I lost interest.

[22:18:02] <pf_moore> I raised an issue but they've been silent: https://github.com/pantsbuild/pex/issues/9

[22:18:05] <_habnabit> Yasumoto, are you a developer of it? like, could i ask you questions?

[22:18:16] <dstufft> pf_moore: I'm totally gonna have some numbers about how many downloads on PyPI come from windows!

[22:18:29] <pf_moore> lol probably just me

[22:19:01] <dstufft> I had some numbers the other day but I tossed them to get more accurate ones

[22:19:10] <dstufft> IIRC more people were downloading using windows than py3

[22:19:12] <dstufft> so ther's that

[22:19:16] <_habnabit> ha

[22:19:50] <pf_moore> Yeah, it'd be nice to see the details, although I probably wouldn't like the results (being a py3/win guy)

[22:20:22] <Yasumoto> _habnabit: yah, I've worked with it a lot, same with @wickman

[22:20:27] <pf_moore> Probably a lot of Windows users use things like conda, too, because of the whole binary build thing

[22:20:28] <_habnabit> Yasumoto, ah ok

[22:20:42] <Yasumoto> pf_moore: whoa, sweet

[22:20:59] <Yasumoto> sorry, I still need to figure out how to silence some github notifications but super-highlight others

[22:21:00] <_habnabit> Yasumoto, can i install into a pex from a local directory without having to make a source distribution of a requirement?

[22:21:01] <dstufft> pf_moore: just wait till the Wheel build farm happens

[22:21:41] <Yasumoto> dstufft: can't tell if serious or trolling (but I'm hopeful....)

[22:21:43] <_habnabit> Yasumoto, also, can i disable fetching from pypi.python.org entirely? i didn't see a flag for it immediately

[22:21:55] <dstufft> Yasumoto: I'm 100% serious about a wheel build farm

[22:22:04] <Yasumoto> _habnabit: so depending on how far down the rabbit hole you're willing to go

[22:22:09] <pf_moore> dstufft: I know :-) I keep thinking about making my own but never quite get the motivation to sort out hosting etc...

[22:22:12] <dstufft> Yasumoto: it's not a high priority ATM, but it's going to happen sooner or later if I can help it

[22:22:15] <Yasumoto> _habnabit: you could also look into the built tool 'pants'

[22:22:24] <Yasumoto> http://pantsbuild.github.io/python-readme.html

[22:22:29] <wickman> habnabit: are you talking about pex? pex --no-pypi

[22:22:29] <dstufft> pf_moore: I can provide lots of hosting!

[22:22:36] <_habnabit> wickman, i am! thanks

[22:22:53] <pf_moore> dstufft: I *knew* you were going to say that...

[22:23:01] <dstufft> We have like ~5k a month total from Rackspace for the PSF, and another ~2k a month from Rackspace for PyPA

[22:23:05] <_habnabit> hm, pex seems a ton slower than pip for fetching deps

[22:23:09] <Yasumoto> dstufft: wow, that's awesome

[22:23:35] <dstufft> I think we're using ~70 of that for PyPA and ~2k or so for PSF

[22:23:39] <wickman> dstufft: the only issue for us with a wheel building farm is that the linux story is just completely messed up.

[22:23:54] <dstufft> wickman: yea that's one of the things that need fixed still

[22:23:58] <wickman> dstufft: we have to publish multiple whls/eggs with identical names but in different repos

[22:24:06] <wickman> dstufft: wheels-el5, wheels-el6, etc

[22:24:10] <dstufft> wickman: I want ot try static linking!

[22:24:14] <wickman> and that's even in a pretty controlled env :-(

[22:24:18] <wickman> dstufft: that would be awesome

[22:24:24] <wickman> dstufft: or let's just drop support for C extensions!

[22:24:25] <wickman> :D

[22:24:31] <dstufft> pypy all the things

[22:24:32] <_habnabit> heh

[22:24:45] <wickman> seriously all problems in python packaging boil down to C extensions

[22:25:10] <dstufft> all the really techincally hard ones anyways

[22:25:16] <dstufft> the rest of the hard ones boil down to politics

[22:25:39] <dstufft> ~but what if I want to download things insecurely~

[22:25:55] <Yasumoto> heh

[22:25:55] <wickman> heh

[22:25:56] <pf_moore> and the C extension issues are really only down to a few hard ones like numpy/scipy

[22:25:59] <_habnabit> whoa, pex.installer.IncapableInterpreter

[22:26:04] <wickman> pf_moore: disagree

[22:26:17] <pf_moore> wickman: really, why?

[22:26:31] <wickman> habnabit: yeah -- might need to pip install wheel (or at least bundle wheel with pex.pex)

[22:26:46] <wickman> pf_moore: we have huge problems with things like python-ldap, mysql-python, pycrypto

[22:26:50] <wickman> pf_moore: to name a few

[22:26:57] <dstufft> http://i.imgur.com/ffFsbzg.jpg IMO

[22:26:58] <_habnabit> wickman, oops, i thought i had wheel installed

[22:27:06] <_habnabit> ok, rerunning pex..

[22:27:17] <_habnabit> and seriously, this is taking forever to fetch from https

[22:27:18] <dstufft> pf_moore: ABI is a pain

[22:27:42] <dstufft> it's less of a pain on Windows (lol)

[22:27:44] <pf_moore> wickman: well, I'd put those into the "a few hard ones" category, but I take your point

[22:27:59] <wickman> pf_moore: it takes one hard one to derail an entire organization's adoption of python :-\

[22:28:18] <pf_moore> yeah, that's where a wheel build farm wins bigtime

[22:28:31] <dstufft> and more expressive filenames

[22:28:31] <wickman> pf_moore: again, wheels don't solve the fundamental abi incompatibility problems

[22:28:41] <wickman> it's just -- hey let's standardize what we already have

[22:28:49] <pf_moore> yeah, the whole ABI thing on Linux is a disaster

[22:29:03] <dstufft> techincally the same problems exist on Windows, you're just far far far less likely to run into it

[22:29:06] <dstufft> and on OSX

[22:29:07] <dstufft> :|

[22:29:10] <pf_moore> Care to switch to Windows where it's easy? ;-)

[22:29:21] <dstufft> oh man I booted awindows up today

[22:29:22] <dstufft> I got so mad

[22:29:30] <Yasumoto> lol

[22:29:31] <wickman> dstufft: we still run into it on osx for the reasons ewdurbin pointed out a while ago -- it's basically the wild west what macosx-10.x version your python interpreter hands back to you

[22:29:32] <dstufft> internet explorer kept telling me python.org was a bad website

[22:29:35] <_habnabit> and pex failed again. maybe it just doesn't support python 3.4 yet?

[22:29:41] <pf_moore> embrace the rage, it is the path to the dark side...

[22:29:59] <wickman> dstufft: system python will be macosx-10.{correct_number}-universal but compile your own and pretty good chance it will be macosx-10.4-x86_64 which is useless

[22:30:09] <Yasumoto> wow.. so glad to find people equally angry/hopeful about python packaging

[22:30:16] <dstufft> wickman: that should be fixed in uh, 1.6 I think

[22:30:18] <dstufft> pip that is

[22:30:22] <wickman> dstufft: it does us no good if you're on 10.9 and publish a macosx-10.4-x86_64 egg that actually compiles against the 10.9 abi

[22:30:25] <dstufft> that is supposed to a minimum version

[22:31:10] <dstufft> OSX's built in libraries are pretty good about ABI going forwards

[22:31:13] <wickman> dstufft: this is a python interpreter problem -- unless you're circumventing the info that comes to you from sys/distutils.sysconfig/pkg_resources/setuptools etc

[22:31:20] <dstufft> but pip treated it as == not >=

[22:31:35] <dstufft> it's the other random shit people link against from homebrew and stuff that ruins it

[22:31:50] <wickman> like, run ./configure and it will poop out an interpreter that says you're running 10.4; it doesn't matter what actual version of osx you're on (i _think_ this is fixed in python 3.3+ but we're universally 2.7.x)

[22:32:00] <dstufft> wickman: that's not what OSX version you're running

[22:32:02] <_habnabit> ah, yeah, i see someone else ran into the same problem i did on python 3

[22:32:12] <dstufft> that's what OSX SDK you're compiling against

[22:32:19] <wickman> dstufft: except it's not

[22:32:26] <dstufft> something compiled against a 10.4 SDK can run on 10.9

[22:33:03] <dstufft> https://github.com/MacPython/wiki/wiki/Spinning-wheels#question-will-pip-give-me-a-broken-wheel

[22:34:16] <wickman> >>> import distutils.util

[22:34:17] <wickman> >>> distutils.util.get_platform()

[22:34:17] <wickman> 'macosx-10.4-x86_64'

[22:34:26] <wickman> ^ ./configure && make on os x 10.7

[22:34:51] <dstufft> configure and make of waht, Python?

[22:35:04] <wickman> it's the same on 10.8 as well, and i have engineers on my team who hand me eggs that say 10.4-x86_64 that spit out .dylib errors

[22:35:08] <wickman> dstufft: yep

[22:35:13] <wickman> dstufft: not homebrew/macports/system python

[22:35:23] <wickman> dstufft: just the default autotools behavior

[22:35:43] <dstufft> are the eggs that say 10.4 compiled against a binary compatible Python with the one you're running against?

[22:35:47] <dstufft> same unicode width etc

[22:36:54] <wickman> dstufft: they pythons are compiled using exactly the same script (so same configure options, etc) but on different OS X versions (10.7 vs 10.8)

[22:36:55] <Yasumoto> _habnabit: do you have a link?

[22:37:06] <Yasumoto> ah, https://github.com/pantsbuild/pex/issues/9 ?

[22:37:20] <_habnabit> Yasumoto, yes, the second traceback in that file is what i'm getting

[22:37:25] <Yasumoto> mm, gotcha

[22:37:29] <_habnabit> Yasumoto, i tried installing pex from git, and now i get a different error

[22:37:34] <_habnabit> 'pex: Unknown exception encountered: can't use a string pattern on a bytes-like object'

[22:37:57] <Yasumoto> that sounds like a 2/3 issue for sure

[22:38:01] <dstufft> wickman: I dunno :) I haven't delved into it too much yet :( I was jsut repeating what the OSX people told me

[22:38:02] <wickman> _habnabit: hm -- we test against 3.4 in our tox.ini but it's possible there's missing coverage

[22:38:07] <dstufft> yea that is

[22:38:10] <Yasumoto> oh, really?

[22:38:27] <dstufft> you have a regex that has a "" pattern and it's being passed a b""

[22:38:29] <wickman> yasumoto: https://github.com/pantsbuild/pex/blob/master/tox.ini#L27

[22:38:31] <Yasumoto> shows what I know.. cool. _habnabit I'll try to repro later this evening

[22:38:54] <wickman> pull requests welcomed :-) but my github filters are pretty messed up so you should probably just e-mail me directly

[22:39:10] <wickman> then problem with being in the twitter org is that you're auto-subscribed to all its open source project notifications which is like 1000/day

[22:39:16] <_habnabit> is there a way to get a full traceback out of pex instead of just this exception?

[22:39:29] <Yasumoto> PEX_VERBOSE=1 should give you some more data

[22:40:22] <_habnabit> it didn't, no

[22:41:15] <pf_moore> _habnabit looks like the get_hex issue was fixed in master very recently (just looking at the last commit built by travis)

[22:41:47] <_habnabit> even pex -vvvv didn't give me any more information :(

[22:41:56] <Yasumoto> _habnabit: you're getting that second stack trace, right?

[22:42:04] <_habnabit> Yasumoto, i was, before i installed from git

[22:42:08] <dstufft> _habnabit: what about pex -vvvvvvvvvvvvvvvv

[22:42:15] <Yasumoto> all the v's :)

[22:42:29] <_habnabit> dstufft, no luck :(

[22:43:00] <Yasumoto> _habnabit: could you pastebin it?

[22:43:14] <_habnabit> Yasumoto, it's just a single line: 'pex: Unknown exception encountered: can't use a string pattern on a bytes-like object'

[22:43:50] <_habnabit> https://github.com/pantsbuild/pex/blob/master/pex/crawler.py#L113 perhaps?

[22:46:33] <dstufft> oh man

[22:46:40] <dstufft> did y'all implement your own PyPI api thing

[22:46:56] <dstufft> that's why this is so slow

[22:46:59] <wickman> habnabit: yuck. that should at least be dumping full traceback, e.g. via format_exc

[22:47:05] <wickman> dstufft: we did years ago.

[22:47:13] <dstufft> wickman: yea, this is still scraping all the things

[22:47:14] <wickman> dstufft: because the API was unusable

[22:47:46] <dstufft> wickman: we've modified the PyPI simple API since this was written to remove the need to scrape random things

[22:47:57] <dstufft> and I have a PEP to remove even more of them

[22:47:58] <dstufft> :D

[22:49:01] <wickman> dstufft: requirement resolution is super fast in practice since the default resolver implementation operates out of a local cache

[22:49:35] <dstufft> wickman: ok

[22:49:51] <dstufft> I dunno! I'm just going by what _habnabit said

[22:49:56] <dstufft> I've never pex'd ;(

[22:50:02] <wickman> https://github.com/pantsbuild/pex/blob/master/pex/resolver.py#L104

[22:50:23] <dstufft> wickman: btw dunno if you've seen it but warehouse.python.org/project/packaging

[22:50:28] <dstufft> PEP 440 version parsing wooo

[22:50:32] <wickman> yeah i did

[22:50:54] <wickman> v excited about the progress of standardization

[22:51:02] <wickman> it means i can make pex even smaller :-)

[22:51:23] <Yasumoto> https://github.com/pantsbuild/pex/blob/master/pex/fetcher.py#L39 :D

[22:51:59] <wickman> yasumoto: indeed -- there's no reason one could not implement a PyPIAPIFetcher

[22:52:05] <wickman> are you volunteering?

[22:52:20] <Yasumoto> _habnabit: that's an annoying error message :)

[22:52:28] <Yasumoto> wickman: wat

[22:52:30] <Yasumoto> #its_a_trap

[22:54:01] <dstufft> wickman: the API is still just the /simple/ index crap

[22:54:08] <dstufft> just with more stuff to take intoa ccount

[22:54:13] <dstufft> so you can hit less links

[22:54:20] <Yasumoto> dstufft: do you know the pep/standard/etc offhand for the api?

[22:54:23] <Yasumoto> ah, gotcha

[22:54:29] <dstufft> there isn't one of those yet

[22:54:36] <dstufft> I'm going to eventually make a ~new~ api

[22:54:40] <dstufft> and that'll have a standard with it

[22:55:01] <dstufft> after I'm done breaking bad things on the current /simple/ API I might write a PEP to document it idk

[22:55:03] <Yasumoto> ohhh

[22:55:06] <Yasumoto> cool

[22:55:12] <Yasumoto> yeah, no pressure

[22:55:25] <Yasumoto> to be clear, at this point, any improvements are huge and welcome

[22:56:45] <wickman> dstufft: btw if you've never pexed before, you can just 'pip install pex' and do 'pex -r pip' et voila, you've pexed

[22:57:13] <dstufft> wickman: oh hey, it it's not something burried like 12 levels deep in a twitter namespace anymore?

[22:57:30] <wickman> dstufft: nope, standalone project now

[22:57:39] <wickman> pantsbuild/pex on github

[22:57:48] <wickman> under the pantsbuild project simply so i'm not a SPOF

[22:57:49] <dstufft> wickman: https://packaging.python.org/en/latest/projects.html

[22:57:55] <dstufft> you should get it on that

[22:58:12] <dstufft> https://github.com/pypa/python-packaging-user-guide

[22:58:13] <wickman> indeed.

[22:58:20] <wickman> i volunteer @yasumoto

[22:58:21] <dstufft> ^^ pr go there

[23:02:20] <Yasumoto> hm?

[23:02:41] <Yasumoto> rad, I'll write something up

[23:09:02] <Yasumoto> _habnabit: mind filing a github issue?

[23:09:12] <Yasumoto> seems like that's a separate problem

[23:09:19] <_habnabit> Yasumoto, sure, i can do that

[23:10:13] <Yasumoto> I think that'll fall into a larger "ensure pex works with py3" effort (worst case it'll be 'improve py3 test coverage' or something)

[23:10:56] <_habnabit> i have a feeling it's not coverage issues but that the tests incorrectly approximate the real world

[23:33:35] <Yasumoto> yep, agreed

[23:51:59] <crawln> stupid question but here it goes is there a way to install pip using wget -0- url | python --user

[23:52:04] <crawln> something along those lines in bash

[23:52:14] <crawln> so I don't have the file in the directory

[23:52:25] <crawln> or is it just better to download the file and delete the file

Log file Viewer

Help | Karma | Search:

#pypa logs for Wednesday the 10th of September, 2014