Reps.mozilla.org outage - 14.02.2017


(John Giannelos) #1

What happened

On 14 Feb between 10:32:51PM and 11:05:51PM we had an outage on reps.mozilla.org portal. The issue was triggered because of the way we handle RSS feeds for the mozilla portal. Posts from planet.mozillareps.org are aggregated to reps.mozilla.org homepage. In the past this was being done in the frontend but after PR 1304 we switched to feedparser and started parsing RSS feeds in the backend. The outage was caused by planet.mozillareps.org server being down.

Workaround

planet.mozillareps.org server got back up after 30 mins so webops considered this issue resolved

Next steps

Improve reps.mozilla.org homepage code to gracefully cache/timeout reps planet feed parsing.