pmxbot IRC Log Viewer

[07:08:42] <tdsmith> on a scale from one to evil, how evil is it for homebrew to run python packages (like mercurial) with system python when system python is older than 2.7.9 and has old ssl support?

[07:10:28] <tdsmith> there's an argument that we're evil https://github.com/Homebrew/homebrew/issues/42416#issuecomment-127407312 and i'm sympathetic but the alternative is to force a dependency on homebrew's python and giving people pythons they don't want seems to make them cranky

[07:13:06] <mgedmin> afaiu python packages that are careful enough (e.g. use requests instead of stdlib) get safe ssl even on 2.7.(x<9)

[07:13:23] <dstufft> not entirely

[07:13:24] <mgedmin> I don't think e.g. ubuntu has backported all the 2.7.9 ssl fixes to their LTS distros

[07:13:38] <dstufft> there's some things that can't be fixed without either pyopenssl or 2.7.9

[07:13:50] <mgedmin> :(

[07:14:01] <dstufft> TLS 1.0 isn't secure either, it's just not outright broken

[07:14:08] <dstufft> (reading the issue)

[07:14:39] <dstufft> if mercurial is using requests, as long as you also install pyopenssl, and such using the system for < 2.7.9 is fine

[07:14:58] <tdsmith> the important part of mercurial is using ssl.wrap_socket

[07:15:47] <dstufft> oh

[07:15:48] <dstufft> well

[07:16:08] <dstufft> then there are attacks it simply can't protect against with system python < 2.7.9

[07:17:29] <dstufft> tdsmith: I guess the answer to your question depends on where the best security falls on your scale of goodness

[07:18:57] <dstufft> tdsmith: I'm sure y'all know this but el capitan comes with 2.7.10

[07:19:14] <dstufft> so if you just wait, it'll only be a problem for pre 10.11 OSX's

[07:22:16] <tdsmith> i'm wondering if there's an easy way to hack in pyopenssl support; i don't see an obvious wrap_socket replacement but i've never used pyopenssl before

[07:22:19] <tdsmith> that'll help for sure

[07:24:17] <dstufft> tdsmith: don't think so - https://github.com/shazow/urllib3/blob/master/urllib3/contrib/pyopenssl.py#L157-L261

[07:25:54] <tdsmith> ahh, roger

[07:25:57] <tdsmith> thanks for that

[07:31:17] <dstufft> tdsmith: fwiw personally I'd just intall 2.7.9 on < 10.11

[07:31:27] <dstufft> becasue having a second Python ins't a big deal to me

[07:31:32] <dstufft> and the security is worth it

[07:31:40] <dstufft> but I'm also a more security minded person than most

[07:32:14] <tdsmith> "where the best security falls on your scale of goodness" <-- ¯\_(ツ)_/¯ -- really not sure how to assign these things relative value in a principled and consistent way

[07:35:36] <tdsmith> i think using homebrew python on 10.10 and below will make a lot of sense once 10.11 has been out for a couple weeks

[07:36:09] <tdsmith> having extra pythons around seems to confuse people who aren't really sure how the ecosystem hangs together out of measure to what i'd have expected

[07:36:23] <tdsmith> but i'm still not sure how ruby gems are supposed to work so maybe i don't blame them

[07:38:57] <dstufft> tdsmith: alternative option: had a --with-system-python option and default mercurial to installing 2.7.9+ and for people who get cranky tell them to use the --with-system-python option

[07:38:58] <dstufft> idk

[07:40:44] <ronny> dstufft: moin

[07:40:47] <ronny> whops

[07:44:07] <dstufft> tdsmith: if it's helpful, I listed the top reasons to prefer 2.7.9+ (or pyopenssl) in the context of pip here - https://stackoverflow.com/questions/29572161/how-to-securely-use-pip-with-ssl-on-ubuntu-trusty

[07:44:36] <dstufft> TLS compression is a problem that can completely ruin the security of the TLS stream (see CRIME)

[07:46:33] <tdsmith> hmhmmmmfmfmfff; i really wish apple had fixed this for me

[07:46:36] <tdsmith> appreciate the detail

[07:50:19] <dstufft> tdsmith: do you happen to have an el cap VM or something available to you? I've been curious about if they fixed the sys.path problem there but haven't gottena round to installing it yet

[07:50:35] <tdsmith> i do, yeah

[07:50:49] <dstufft> tdsmith: do you mind showing me what sys.path looks like on the system python?

[07:50:50] <tdsmith> do you mean the way it prefers its own numpy, scipy etc?

[07:50:53] <tdsmith> yeah one sec

[07:51:29] <dstufft> tdsmith: yea, it sticks it's own stuff (in the Extras dir) ahead of site-packages

[07:55:13] <tdsmith> this build of el cap (15A216g) still has 2.7.6 and an ugly sys.path

[07:55:23] <tdsmith> ['', '/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python27.zip', '/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7', '/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/plat-darwin', '/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/plat-mac', '/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/plat-mac/lib-scriptpackages', '/System/Library/Frameworks/Pyth

[07:55:23] <tdsmith> on.framework/Versions/2.7/Extras/lib/python', '/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/lib-tk', '/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/lib-old', '/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/lib-dynload', '/System/Library/Frameworks/Python.framework/Versions/2.7/Extras/lib/python/PyObjC', '/Library/Python/2.7/site-packages']

[07:59:24] <tdsmith> that's public beta 1 which is about a month old now

[08:00:03] <dstufft> boo

[08:00:16] <dstufft> hopefully they fixed it with 2.7.10 which is in the latest build

[08:00:17] <dstufft> but I doubt it

[08:00:22] <dstufft> ;(

[08:00:42] <tdsmith> well, fingers crossed

[08:44:16] <MrJones> hi

[08:44:35] <MrJones> I'm using tox to install project with dependencies for automated testing

[08:44:51] <MrJones> we have a project_a which depends on a project_b

[08:45:18] <MrJones> tox invokes pip like this: pip install --editable=file:////home/jonas/Develop/daphne/branches/0.6/unifrhermes/trunk/../../unifrgaia/trunk/ -r/home/jonas/Develop/daphne/branches/0.6/unifrhermes/trunk/requirements.txt

[08:45:35] <MrJones> the first directory is project_b "unifrgaia" with setup.py and all that

[08:45:49] <MrJones> which creates a package named "unifrgaia"

[08:45:54] <MrJones> the requirements.txt has this as dependency

[08:46:16] <MrJones> however, pip only mentions this specific --editable directory only once in the output: Obtaining file:////home/jonas/Develop/daphne/branches/0.6/unifrhermes/trunk/../../unifrgaia/trunk/

[08:46:22] <MrJones> then it goes on to talk about other deps

[08:46:44] <MrJones> and finally concludes that unifrgaia isn't present: Could not find a version that satisfies the requirement unifrgaia>=0.0.1 (from -r /home/jonas/Develop/daphne/branches/0.6/unifrhermes/trunk/requirements.txt (line 2)) (from versions: )

[08:46:59] <MrJones> why does it seem to ignore the --editable link entirely?

[10:50:38] <MrJones> anyone here? <:

[10:53:38] <mgedmin> yesbut

[10:53:45] <mgedmin> your question is long

[10:53:49] <ronny> MrJones: bad time, europe middle of ay, us early morning

[10:55:07] <MrJones> TL;DR: I used --editable, but pip still complains about the exact dependency being missing which I supplied with --editable

[10:55:21] <MrJones> also what's wrong with a good healthy middle of the day? :D

[10:55:31] <mgedmin> for me: hunger

[10:55:36] <MrJones> oh

[10:55:48] <MrJones> well I can wait another few hours (: just wondered if anyone was around at all

[10:56:08] <mgedmin> it might be easier (for me anyway) if you pastebinned your shell session

[10:56:23] <mgedmin> what you run, what error you get, what the relevant config files look like

[10:57:20] <MrJones> it's tox running it, so I can only copy what tox indirectly ran as I already did above: pip install --editable=file:////home/jonas/Develop/daphne/branches/0.6/unifrhermes/trunk/../../unifrgaia/trunk/ -r/home/jonas/Develop/daphne/branches/0.6/unifrhermes/trunk/requirements.txt

[10:57:48] <mgedmin> this can probably be simplified to

[10:58:08] <mgedmin> echo 'foo' > requirements.txt; pip install -e ../foo -r requirements.txt

[10:58:18] <mgedmin> assuming a ../foo/setup.py that has setup(name='foo')

[10:58:20] <MrJones> yes, that's pretty much what I did

[10:58:26] <mgedmin> can you reproduce the error using a simplified example like this?

[10:59:18] <MrJones> if that helps I can try... I'd need to set up two simple example projects first tho

[10:59:45] <mgedmin> why two?

[11:00:07] <MrJones> the one to install as a dep ("foo") and the one in the current folder to be installed

[11:00:13] <mgedmin> you don't need the second one

[11:00:52] <mgedmin> you want to discover if pip can handle installing a non-public project (foo) with -e at the same time as it's also mentioned in -r reqs.txt

[11:01:12] <mgedmin> would things start working if you split that into two steps? pip -e ../foo; pip -r reqs.txt?

[11:01:24] <mgedmin> I assume things would start working if reqs.txt contained -e ../foo

[11:11:12] <ronny> MrJones, mgedmin a common suggestion is to combine tox and devpi

[11:15:33] <MrJones> yes and not a very helpful one to make "tox" just work for everyone

[11:15:48] <MrJones> it does have its use cases, but also its limitations in practicability

[11:16:24] <MrJones> mgedmin: apparently it's some tox issue. I just tested it manually, and with pip install--editable=/path/to/project -r requirements.txt --allow-external nameofexternalproject made it work

[11:16:36] <MrJones> however, if I tell tox the same options for pip it doesn't... meh

[11:17:02] <MrJones> it somehow seems to separate the arguments incorrectly or something :x I'll ask the guys in the tox channel

[11:17:03] <mgedmin> compare pip versions between whatever you use and whatever's installed in .tox/py* ?

[11:17:08] <MrJones> thanks for making me test the right things :)

[11:17:34] <mgedmin> maybe test the command using .tox/py*/pip install directly

[11:17:58] <MrJones> oh, nice idea :D

[11:17:59] <MrJones> will try

[11:18:35] <MrJones> ohh weird

[11:19:50] <MrJones> mgedmin: I just had the run finish with --allow-external... lemme paste: http://fpaste.org/251325/86869971/raw/

[11:20:10] <MrJones> it *seems* to work at first, but then it tells me to --allow-external the project which I already specified with --allow-external :-?

[11:20:42] <MrJones> hm lemme retry on a clean virtualenv

[11:22:27] <MrJones> same :|

[11:23:18] <MrJones> did I use allow external incorrectly? well it doesn't complain about the options initially..

[11:43:53] <mgedmin> you shouldn't need --allow-external for projects you specify via -e ...

[11:44:52] <MrJones> oh

[11:45:38] <mgedmin> I wonder what that warning about "some externally hosted files were ignored" actually means -- is there a --find-links in the .txt?

[11:46:34] <MrJones> mgedmin: this is the requirements.txt: http://fpaste.org/251338/86886191/

[11:46:53] <MrJones> second line refers to the project which I want to add via --editable, which isn't on pip

[11:47:44] <mgedmin> yeah, pip can't do what you want: https://gist.github.com/mgedmin/2f2132ead2999a5e2077

[11:47:58] <mgedmin> is this a bug in pip? who knows! (probably dstufft, he knows *everything*)

[11:48:33] <MrJones> ;(

[11:48:41] <mgedmin> note how it works if you pip install -e and pip install -r as two separate commands

[11:50:20] <mgedmin> fun fact: if a requirements.txt has '-e /path/to/foo' and also 'foo' on two separate lines, you get the same error

[11:50:30] <mgedmin> (Could not find a version that satisfies the requirement foo)

[12:47:03] <dstufft> Is it possible to summarize the problem or should I read the scrollback?

[12:47:56] <xafer> MrJones, did you try specifying #egg=unifrgaia on your --editable ?

[12:48:12] <xafer> -editable=file:////home/jonas/Develop/daphne/branches/0.6/unifrhermes/trunk/../../unifrgaia/trunk/#egg=unifrgaia

[12:52:21] <xafer> dstufft, the important parts would be http://fpaste.org/251325/86869971/raw/ and the content of the req file: http://fpaste.org/251338/86886191/ if I followed correctly

[12:53:13] <dstufft> I think the #egg=unifrgai will fi it

[13:03:36] <xafer> looks like the #egg part can only be given with a VCS

[15:52:33] <dstufft> tdsmith: damnit, Homebrew's openssh doens't keychain anymore, and OSX's Openssh is too old to ed25519 ;((

[15:53:19] <tdsmith> yeah, we were uncomfortable maintaining that patch, sorry!

[15:55:11] <dstufft> yea, it's not a bad choice.. I'm just trying to figure out how to unfuck my ssh now

[15:55:14] <ronny> tdsmith: was it impossible t upstream that patch

[15:55:29] <dstufft> since all my servers use my ed25519 key

[15:55:41] <dstufft> maybe I'll just switch to el cap, that has ed25519 I think?

[15:55:42] <dstufft> :/

[15:56:49] <tdsmith> you could pull a version of the formula with the patch out of git history

[15:57:23] <dstufft> oh

[15:57:26] <tdsmith> ronny: i don't know if there's a story there but the homebrew developers haven't tried recently

[15:57:32] <dstufft> is there docs on how to do that anywhere

[15:59:00] <tdsmith> it kinda sucks; brew log openssh and pick your fave commit, then cd /usr/local/Library/Taps/homebrew/homebrew-dupes and e.g. git checkout d12c883fa9213 openssh.rb and brew install openssh

[15:59:00] <ronny> dstufft: btw, is there any rough process i should follow if i want to propose/impelemt a propper multi-version install scheme that ensures consistency per procsss

[15:59:36] <dstufft> tdsmith: I'm guessing that ``brew update && brew upgrade --all`` will blow it away then too?

[15:59:56] <tdsmith> yeah; `brew pin openssh` will keep it from getting upgraded

[16:00:02] <dstufft> ronny: Generally you want to write a PEP

[16:00:17] <dstufft> tdsmith: Ok, that sounds less painful than trying to run el cap on my main machine right now

[16:00:21] <dstufft> tdsmith: thanks sir! :D

[16:01:12] <ronny> dstufft: is there any repo i should sent a pr against, or should i jsut write a pep omewhere, then send it to python-ides/python-dev

[16:01:16] <tdsmith> you're welcome, sorry for the pain

[16:02:07] <dstufft> ronny: you can just write a pep somewhere and send it to distutils-sig

[16:02:22] <ronny> dstufft: ok, will do, starting with a draft

[16:04:20] <dstufft> tdsmith: it's totally OK, it's a reasonable decision :) It just bit me today since i tend to just blindly upgrade my homebrew installed stuff. Maybe it would have made sense to wait until el cap since I'm guessing a common reason was to get ed25519 support.. but too late now :D Maybe the issue I found should mention how to do the thing you told me tho?

[16:05:50] <tos9> dstufft: I gave up

[16:05:52] <tos9> (on this exact problem)

[16:06:02] <tos9> The thing to do is to wait for el capitain in my case

[16:06:14] <tos9> oh which you just said.

[16:06:20] <tos9> Sorry, reading the scrollback slowly :)

[16:06:32] <dstufft> well I can't wait, I have to restore th eold version or do something :D my ssh is borken atm

[16:06:52] <tos9> dstufft: well so the real thing I want is to switch to gpg-agent probably.

[16:07:06] <tos9> but I couldn't figure out how to do that in 10 minutes

[16:07:23] <tos9> so for right now I'm literally copy/pasting my password in for SSH yeah :/

[16:08:26] <benjaoming> Okay, I've kind of not fully understood the problems with PyPi stats: We know that total download counts include mirrors and bots... and that the old API was taken down... but dstufft added a lot of backwards compatible stuff and access to download counts via the JSON API. Which is great (...)

[16:09:04] <benjaoming> (...) so it seems that the individual download counts are clean from mirror traffic as was also announced by dstufft in 2013! So the problem isn't so big at all!

[16:09:28] <benjaoming> I thought I had to make a honeypot.... but turns out the honeypot has 0 downloads after 24 hours

[16:09:37] <benjaoming> https://pypi.python.org/pypi/python-bogus-project-honeypot

[16:10:15] <doismellburning> interesting; https://pypi.python.org/pypi?name=0-._.-._.-._.-._.-._.-._.-0&version=0.1&:action=display picked some up

[16:10:52] <benjaoming> Anyways, I went ahead and used vanity by aclark to make another tool.... https://github.com/benjaoming/simple-pypi-statistics -- it's supposed to feature the honeypot stats... but currently it seems useless.... but at least it has json output as a new feature

[16:11:47] <ronny> dstufft: i'll take a whil to get the first initial draft thats redy to shred

[16:15:48] <dstufft> tos9: D:

[16:16:21] <dstufft> tos9: mayb eI'll just switch to el cap beta. I used the yosemite beta and it was fine *shrug*

[16:16:39] <tos9> dstufft: /me nods -- I was going to do the same, but think it's worth trying gpg-agent first.

[16:16:52] <tos9> it's a silly thing that will cause me 8 hours of headache if something goes wrong considering you can't downgrade

[16:21:50] <aclark> benjaoming: \o/

[16:33:15] <aclark> benjaoming: although bonus points for either install_requires vanity then use its "api" and/or just implementing your features in vanity :-)

[18:11:35] <tdsmith> <tdsmith> hmhmmmmfmfmfff; i really wish apple had fixed this for me

[18:11:48] <tdsmith> <-- ask and you may receive; apparently 10.10.5 upgrades python to 2.7.10

[18:42:16] <benjaoming> aclark: the thing was that I discovered vanity while creating the tool, then I wanted to integrate vanity but found especially this line that made it impossible... https://github.com/aclark4life/vanity/blob/master/vanity.py#L114 -- I've created simple_pypi_statistics.api module as an adapted version of vanity that does away with the CLI stuff... sorry I was in a bit of a hurry to wrap this up so I didn't spend a lot of time planning.

[18:42:52] <aclark> benjaoming: yep, np

[19:14:12] <benjaoming> oh god, the honeypot has been downloaded...

[19:14:12] <benjaoming> python-bogus-project-honeypot-1438620531.16.tar.gz 2015-08-03 160

[19:14:12] <benjaoming> python-bogus-project-honeypot-1438619971.82.tar.gz 2015-08-03 135

[19:14:13] <benjaoming> -----------------------------------------------------------------------------

[19:14:13] <benjaoming> python-bogus-project-honeypot has been downloaded 295 times!

[20:01:41] <dstufft> benjaoming: oh, if you were looking at the JSON endpoint those are cached for long times

[20:02:05] <dstufft> tdsmith: oh snap, they update dto 2.7.10 in yosmite too?

[20:10:11] <tdsmith> in the upcoming micro update yeah!

[20:11:43] <dstufft> tdsmith: python -m ensurepip ftw

[20:42:22] <benjaoming> Wooble: *upgrading python-bogus-project-honeypot with lots of malicious code* :)

Log file Viewer

Help | Karma | Search:

#pypa logs for Tuesday the 4th of August, 2015