Log file Viewer

[06:26:37] <mgedmin> pypi 503 guru mediation error, details: cache-iad2135-IAD 1411453364 400625302

[06:59:01] <mgedmin> I'm curious tho: is http://status.python.org/ updated by hand or is there some automated system for that?

[06:59:21] <mgedmin> because it still says pypi is "operational"

[11:27:37] <dstufft> mgedmin: it's automated

[11:27:42] <dstufft> via pingdom

[11:27:49] <dstufft> it's also by hand too if we want

[11:28:16] <mgedmin> (a) awesome, (b) how come pingdom rarely notices those 503 responses that people occasionally get?

[11:28:44] <mgedmin> anycast routing to different cloudflare endpoints?

[11:30:18] <dstufft> Well it's Fastly not cloudflare, but I think it's likely either (or a combination of) the fact that pingdom requiires a few "down" results before it marks a service down (to prevent false positives because of network issues) and that it's geo distributed and outages can be regional which means pingdom might not notice it right away (or perhaps at all?) depending on if that region has a pingdom checker or not

[11:30:51] <dstufft> one thing we need to setup still is for alerting to happen on elevanted error rates

[11:30:54] <dstufft> instead of just up/down

[11:31:05] <dstufft> we have that data going into graphite, we just don't have alerting setup on it yet

[11:31:06] <mgedmin> whoops I can't tell one CDN service from another

[11:31:32] <dstufft> heh no worries :) Ideally the choice of CDN should be invisible to end users other than it being faster than without

[11:34:51] <dstufft> We use a shielding setup, so the way it works is a request that comes in via say the Fastly London POP will instead of going directly to the backend, will instead go to the Fastly Virgina, US POP, and from there if it's cached it'll serve the cached response or if not it'll hit the backend. This means that the further from Virgina you are the more timing sensitive you'll be which can cause some 503 errors

[11:35:17] <dstufft> (of course in the above example, if the London POP has something cached, it'll jsut return it directly instead of going to virgina)

[11:35:24] <doismellburning> dstufft: if someone wanted to help with that, how would they get involved / offer / find ways to help etc.?

[11:37:02] <dstufft> doismellburning: for infrastructure stuff we have a salt repo at github.com/python/pypi-salt, and IRC channel at #python-infra, and a mailing list at infrastructure@python.org

[11:37:17] <dstufft> (for non PyPI we also have a salt repo at github.com/python/psf-salt)

[11:37:56] <dstufft> specifically to the monitoring stuff, EWDurbin knows more about that then I do, he setup our graphite stuff and had plans I think

[12:22:53] <EWDurbin> dstufft mgedmin, we have a total of 871 503 errors in the last 24 hours, no large spikes or consistently elevated periods. http://imgur.com/a/pUrUL

[12:23:15] <EWDurbin> it looks like maybe one POP was having trouble for a bit there, but nothing systemic or widespread :/

[12:23:38] <EWDurbin> nothing we would have fired an alert on for sure

[15:07:37] <magicbronson> I think I just hit a bug on PyPI. Is anyone in here a PyPI admin?

[15:26:04] <EWDurbin> magicbronson: sup?

[15:27:49] <magicbronson> EWDurbin: Wow, it actually looks like a bug in Chrome. When I try to log in using Chrome I inevitably get the bad password page. When I use Firefox with the same password it's fine. I think the culprit is that I have a long (32-character) password, and Chrome isn't sending the entire string over basic auth. Which would be insane.

[15:28:20] <EWDurbin> Hmmm, I'll try to reproduce

[15:28:40] <magicbronson> Thanks.

[15:30:48] <magicbronson> EWDurbin: Is there any way to move my existing account from password-based authentication to OpenID?

[15:31:05] <EWDurbin> you just associate an OpenID profile in account details

[15:31:50] <EWDurbin> magicbronson: reproduced in chrome with a 64 char password

[15:32:42] <EWDurbin> magicbronson: hilarious find, go bug the Chrome team ;)

[15:33:39] <magicbronson> EWDurbin: Glad you could reproduce. As for OpenID, I tried that but it seemed to want to create a new PyPI account for my OpenID rather than allowing me to associated it with an existing account. And now I unfortunately seem to be locked out of my account, can't even log in via Firefox anymore. :(

[15:34:34] <magicbronson> EWDurbin: are there any plans to replace basic auth? It seems error-prone.

[15:34:40] <EWDurbin> absolutely

[15:34:42] <EWDurbin> in warehouse

[15:34:52] <EWDurbin> github.com/pypa/warehouse

[15:35:35] <magicbronson> EWDurbin: hadn't heard of that, cool. When can I look forward to using it? :)

[15:35:45] <EWDurbin> not sure!

[15:35:58] <EWDurbin> you can preview at warehouse.python.org

[15:41:02] <magicbronson> nice!

[15:42:11] <magicbronson> EWDurbin: Filed a chrome bug. Would send you a link in case you want to star but put it in the 'security' category so it's private by default.

[15:44:05] <magicbronson> EWDurbin: Oh well. By the way, I wrote and released a package I think might be useful. What's a good place to ask for community review? https://github.com/jab/bidict in case you're interested.

[16:31:33] <nanonyme> Is virtualenv.py no longer self-sufficient?

[16:31:57] <nanonyme> I'm getting errors that can't find pip when running it here on Windows

[16:34:21] <nanonyme> Also why the bloody crap does the Scripts vs bin difference still have to exist? >.<

[16:55:11] <nanonyme> Gaaah, totally forgot lxml still hasn't moved to wheels

[17:24:30] <doismellburning> nanonyme: hah yes

[17:24:47] <doismellburning> is there actually a decent universal wheel solution for things with C in them?

[17:24:56] <apollo13> no

[17:25:09] <apollo13> well yes, local mirror with wheels prebuilt against your distro :)

[17:25:29] <doismellburning> apollo13: we have different values of "decent" :P

[17:25:54] <apollo13> doismellburning: my deployments need to work if pypi is down, so…

[17:26:06] <doismellburning> sure

[17:26:21] <doismellburning> well

[17:26:33] <apollo13> without static linking this will never be easy…

[17:26:54] <doismellburning> I'm wondering if it's just easier to make a pure python lxml fork...

[17:29:17] <apollo13> now we seem to have different valeus of "decent" (magnitudes or so)

[17:29:36] <doismellburning> ;P

[17:30:12] <apollo13> damn, so many things to do

[17:46:42] <DanielHolth> python-in-chrome-nacl?

[17:59:20] <dstufft> static linking is possibly a solution

[17:59:57] <apollo13> seems like a security nightmare, but well

[18:00:13] <DanielHolth> I always thought that compiling lxml wasn't really a problem, unless you were on Windows or doing it many times a day

[18:00:50] <apollo13> well, windows wouldn't be windows if you didn't have problems^^

[18:01:15] <DanielHolth> For a long time OpenSSL was the canonical example of why static linking was a bad idea, but now I think people may be coming around to "OpenSSL is a bad idea"...

[18:04:37] <dstufft> static linking really only adds to the problem, it doesn't create new problems

[18:04:41] <dstufft> in python anyways

[18:05:10] <dstufft> since wheels are most likely to be used inside virtualenv, most folks don't go around and ensure those are all up to date nor is there a process to tell people they aren't up to date

[18:07:55] <apollo13> dstufft: no, but as long as you link dynamically you do get updates from openssl etc… and I tend to upgrade my venvs

[22:04:32] <buck1> i'm splitting pip.index into several modules