Log file Viewer

[14:50:28] <fungi> out of curiosity, is anyone else seeing inconsistent (in some cases weeks-old) indices from different fastly cdn endpoints for pypi? seems like it's been ongoing since the pypi incident yesterday, though that could simply be a coincidence

[14:51:46] <fungi> hard to pin down since we don't know how to query specific cdn endpoints for confirmation, but we saw stale responses with lcy+bwi mentioned in headers. at the same time we saw current responses for the same indices with headers for iad+dca and iad+bwi

[14:53:20] <fungi> we're running a globally-distributed ci system, and it's wreaking havoc with our build consistency since we're ending up with pip selecting different package versions depending on which cdn endpoints it happens to hit

[14:54:51] <techalchemy> EWDurbin ^

[15:03:17] <EWDurbin> fungi: any specifics you can provide make all the difference.

[15:03:57] <fungi> EWDurbin: sure, i'll see if i can scrape some headers on a few different requests; will that suffice?

[15:04:25] <EWDurbin> package names and versions and whatnot too

[15:07:09] <fungi> oh, i was going to get the reported pypi serials, but sure i can collect actual index content to go along with that

[15:10:09] <EWDurbin> Serials are nice, but to do anything with them still kinda need to know what is being requested

[15:13:57] <fungi> yep, i'll try to assemble a list of the package names and versions which were reported as not found over the past 24 hours. like i say, reproducing is tough since i can't figure out how to force a request to a specific fastly endpoint

[15:23:31] <fungi> so far these are the package versions i've found pip complaining about not finding in various builds (but there are certainly more, we just need to dig): cryptography===2.8 (uploaded 2019-10-17), tempest===22.1.0 (uploaded 2019-10-15), keystoneauth1===3.18.0 (uploaded 2019-10-22)

[15:24:59] <fungi> if i manage to catch one while we've got a response for a stale index cached, i'll grab the full headers and content from the request

[15:46:13] <fungi> i worked out a shell loop to iterate through /etc/hosts overrides for all 4 ipv4 and 4 ipv6 addresses returned by the pypi.org round-robin and checked the above three packages against all of those. not able to reproduce at the moment

[15:46:32] <fungi> but not sure if that actually hits all the fastly cdn nodes

[15:46:50] <fungi> i'll repeat the process from servers in different parts of the world here in a bit

[16:01:40] <fungi> also, i did manage to verify that i'm not hitting much variety of fastly endpoints with that loop, the X-Served-By headers all seem to mention locales in the region where the test was performed

[16:07:08] <fungi> also reading up on fastly's cdn architecture to see if i can work out a better way to narrow this problem down

[16:15:31] <fungi> still not able to reproduce manually. at this point i'm starting to wonder if whatever the issue was, maybe it's since cleared up on its own. last verified occurrence i have in a build is from nearly 6 hours ago

[16:17:01] <fungi> but i'm also not having any luck managing to get a request to go through any lcy endpoints, which were what was implicated in some of the earlier failures we tracked down, so possible fastly found a problem and took it out of rotation i guess