About Test Pilot

I wrote an about page for Test Pilot recently. You can find it on our site here: https://testpilot.firefox.com/about/.

Here’s the full text below, let us know what you think:

About Test Pilot

At Mozilla, we believe in an open, accessible Internet for all. With Firefox, we’ve created a robust browser that gives you — and hundreds of millions of people around the globe — access to everything the Web can offer while upholding our commitments to privacy, user choice, and open source development.

We think that makes Firefox the best browser available today, but it can also make it harder to innovate. For years, we’ve asked ourselves questions like

  • How do we try out crazy new ideas without disrupting user experience?
  • How can we run quick experiments when even simple changes in our core products can take 3 to 6 months of development, pressure testing, and localization before launch?
  • How can we gather meaningful data about new features without compromising our core beliefs in user privacy and choice?

We made Test Pilot to answer these questions. It’s an opt-in platform: you can choose the experiments you want to test, you can read about the data we collect for each experiment, you can tell us what you think. Test Pilot lets you play with cool new features, and it lets us try out our ideas before we ship them to an audience of millions. Here’s how the process works:

The Test Pilot Lifecycle

We think up new ideas for Firefox all the time. Maybe you do, too. Perhaps you’re one of thousands of contributors who’ve logged a bug or asked for a new feature on Twitter. In fact, we have so many ideas that it can be hard to decide which ones make the most sense for Firefox and which don’t.

These decisions aren’t just academic: implementing a new feature in Firefox can take thousands of hours of design and engineering time. We test with prototypes, but don’t always get the kind of real-world feedback that helps us understand how people will use the feature and how well it really performs for them.

Test Pilot is more than just the web app you’re currently looking at. It’s a research, design, and development process aimed at making feature development a smoother, more transparent process, while allowing us to play with — and share — more radical ideas for Firefox. The process consists of four phases.

Phase One: Prototype and Validate

In Phase One, The Test Pilot team evaluates lots of promising ideas. First, we do research to better understand the problems we want to solve and to uncover potential solutions. Depending on the problem, our experiment may be an entirely new feature or it could be a design that helps us gather information to improve existing features.

Then, we sketch and tinker with simple prototypes. Sometimes these prototypes are doodles on paper, sometimes we mock things up in Photoshop, and sometimes we explore in code. We test these prototypes in front of real Firefox users to help us validate and refine our ideas. The name of the game in this phase is to learn quickly. If an idea doesn’t make it out of Phase One, that’s just fine. We’ve saved ourselves a lot of headaches down the road and learned something to boot. If an experiment graduates to Phase Two, we start to get a little more serious.

Phase Two: Design and Build

In Phase Two, we create the experiments for Test Pilot. First, we take what we’ve learned in Phase One to develop requirements — the must-have features and functions an experiment needs to ship. Then, we design a high-fidelity prototype so we can test the user experience and interface.

After user testing, we review our findings and modify the requirements if necessary. Then, we build the actual experiment that we plan to ship. We do another round of testing to make sure everything works. We hold pre-launch meetings with our legal team (to make sure that the features we ship protect the privacy of our users) and with our marketing and data teams (to be sure we are clearly communicating the value of the feature and measuring the right things). Once we’re given the all clear, we move into the next phase.

Phase Three: Launch and Learn

In Phase Three, we launch the experiment. This is where you get to try things out. While experiments are in Phase Three, we monitor their performance and listen to user feedback. As experiments progress, we fix problems, make improvements, and gain insights into how people use what we’ve built and whether they like it. If we find that an experiment is performing poorly, we’ll end it during this phase. Once we are satisfied that we understand how an experiment has performed, it’s ready to graduate from Test Pilot.

Phase Four: Graduate

In Phase Four, experiments prepare to leave Test Pilot and move to new assignments.

Successful experiments face one final test: a randomized opt-in trial in Firefox. If the experiment has broad appeal among Firefox users, it will likely become a standard feature of Firefox. If the reaction is mixed, the experiment will most likely ship as an add-on.

Experiments designed to gather data (like Tracking Protection, which is currently running) end when we’re satisfied that we’ve learned enough to make meaningful changes to Firefox.

Once we determine the fate of an experiment, we’ll let you know that it is about to graduate from the Test Pilot site and will share what we’ve learned. We’ll also let know you if you can expect to see it become a part of Firefox. Then we’ll remove it from Test Pilot.

So that’s Test Pilot. We’re really happy you’re here. Please check out our experiments and tell us what you think. If you have questions or would like to talk more, you can find us on our forum.

Thanks for flying with us!

- The Test Pilot Team