How To Test 5000 Websites in 7 Days - Recapping the Firefox Quantum Sprint

Early this month, from October 2-8th, hundreds of volunteers from 50 countries around the world came together to test websites and file web compatiblity issues in a Firefox Quantum Sprint. By the end of the week 5900 websites were tested and 1133 bugs were filed.

The approach was simple, engage our local communities of enthusiastic Mozillians to organize events and test Firefox Quantum in their region. Increasing the quality of Firefox Quantum by extending the reach of our testing to more parts of the world through volunteers.

Process

Volunteers who registered on the site were pointed to a list of the 200 famous websites, by country, and instructed to report any bugs on the webcompat site.

As the campaign ran we responded in real time to feedback and metrics. We were thrilled with the volume of participants but identified early that we needed to increase the quality of the reports and immediately and shared additional guidelines by the webcompat team on all channels.

We also shut down the leaderboard mid-way through the campaign and saw an immediate reduction in empty or fake bugs.

Results

The campaign produced over eleven hundred bugs in a week, dramatically increasing the regular volume of web compatibility issues. The Web Compatability team reported that for certain locales they received bugs they would normally never have seen, for example some issues around Hindi and Malayalam fonts.

While the campaign marked a dramatic increase in scale the quality of the bugs were fairly average with roughly 80% of bugs being incomplete or invalid, 10% of bugs were good and useful, and 1% high value.

Despite an outstanding need to increase the quality of bugs participants and staff alike were positive about the campaign and particularly excited by the enthusiastic response to a call to make Firefox better.

What’s Next

The Open Innovation Team is continuing to explore new ways to connect our communities of enthusiastic volunteers to meaningful projects within Mozilla. With each campaign or initiative we learn more about the best way to connect volunteers who care about Mozilla’s work to appropriate impact. From the perils of the leaderboard, to the importance of bite size instruction, we’ll be sharing more of what we learned in this campaign and how it’s driving our relationship with internal teams and enthusiastic non-coding contributors forward in the coming weeks.

11 Likes

Hi Konstantina

It was a great initiative and I am glad I took part.

Whilst the gamification element may have gone a little sideways (although it was designed with the best of intentions), it was really awesome to bring so many people from across the Mozilla community together to work on a shared project linked to such an important release.

2 Likes