We’d like to update you on a recent outage of the Common Voice platform that occurred on October 27th, and thank you for your patience as we resolved it.
People using the Common Voice platform to contribute data were presented with a 503: Server error message. Subsequent investigation by our experienced engineering team, including @bozden and Dmitrij Feller, identified that one particular contributor to the platform was sending malformed binary data on an infinite loop during a data upload process. The process used for transcoding audio data, ffmpeg, was tightly coupled to the uploading process, causing it to exhaust server resources, and make the whole platform unavailable.
Immediate mitigation strategies to return to service included blocking the specific user. Further investigations revealed that the user was located behind a mis-configured firewall, and that the firewall was modifying network packets during transit, resulting in the malformed binary data. We have worked with the affected community to resolve this network configuration.
Additionally, we have de-coupled the dependency between the uploading process and ffmpeg so that future similar outages, if they occur, have limited impact.
We do not believe this activity was malicious in any way, and thank the language community for their helpful, detailed and generous feedback which greatly assisted us in diagnosing and resolving the issue. This incident has highlighted for us the diversity of technical environments that language communities operate within, and continue to operate within, to preserve and promote their languages. There are 7000 languages still spoken on the planet, and they persist because we all persist.