email spamming incident

(John Giannelos) #1

Hey all friends!

Here is some context around the spamming incident we had with our email bot.

  • Issue
    • ReMo bot spammed users with hundreds of emails. We had various incidents but the last one was last week (18-23 Sept 2017).
  • Root cause
  • Context
    • Spawning tasks to be executed in eta time is an edge case in our codebase (4 calls out of ~100 tasks)
    • At first we tried the workaround celery maintainers suggest (change visibility_timeout config in our celery workers). Since we could reproduce the exact same behavior we were pretty confident that this would solve our problem.
    • After another round of spamming we realised that the bug was not fixed. It turned out that there was an issue on how celery parses the configuration. It uses a namespace and picks up only config entries starting with a specific prefix. Apparently our workaround wasn’t picked up by the workers.
    • We fixed the namespace issue
    • Verified everything on stage
    • Pushed fix to prod
    • No more spamming \o/

Excuse us for all that spam voting emails in your inbox :slight_smile:

(T.M. van den Broek) #2

Thanks for fixing it!

ParSys Sprint - IAM/CoSS Sprint 16 Review