Firefox tests and Page Shot (includes first-time setup info)

(Jared Hirsch) #1

Mark was kind enough to give me and Ian a quick walkthrough of his testing workflow this morning. Here are some notes on getting set up, as well as some notes on what’s currently broken. Once this matures a bit, I’ll add it to the docs in the pageshot repo.

General Stuff

Gecko setup

Start by setting up your Firefox dev environment. The script is fantastic and saves many hours of confusion and stress.

The Try server

You’ll eventually want to push to try, which is a server that fronts Mozilla’s testing cluster. You will need to get vouched for level 1 commit access to do so. Read through the required steps here and ask me if you have any questions. Mark has agreed to vouch, so needinfo :standard8 in the Bugzilla bug.

Firefox review process

Mozilla now uses MozReview for uploading patches. There are good docs for it, and it’s under active development.

The mercurial vs git thing

Mozilla picked mercurial as its VCS a few years ago; here are some docs about how hg works at mozilla. If you prefer git, Mark recommends using git-cinnabar to translate between git locally and hg on the server. git-cinnabar is also nicely documented and under active development.

Page Shot specific stuff

Currently, lots of tests are failing on try: here’s Mark’s initial try run; all the oranges and reds are breakage we have to sort out.

If you want to run these tests locally:

  • Get a local copy of m-c (see the ‘general stuff’ section above)
  • Apply the patch from bug 1346825 to your local copy of m-c
  • Mark’s landed a script in the pageshot repo that exports the code into Firefox. This incantation should work:
PAGESHOT_BACKEND= EXPORT_MC_LOCATION=../gecko make export_addon
  • Run individual tests locally like this:
./mach build && ./mach test dom/network/tests/test_network_basics.html
  • You can also attach the debugger to local tests:
./mach mochitest --jsdebugger dom/browser-element/mochitest/priority/test_HighPriority.html
  • Run tests remotely using try syntax (see ./mach try --help or the try chooser). The try server takes a long time for a full run (1-2 hours), so you can just run the linux x64 opt tests instead. Takes about 20 minutes, and seems to hit all the broken tests. Often tests will fail on try, but not locally, so plan your time accordingly :stuck_out_tongue:

about the broken tests

There seem to be two classes of broken bugs: those that break predictably, because Page Shot either adds a context menu item, or a toolbar button; and those that break because of something related to WebExtensions toggling prefs. The latter tests seem to live under the /dom/ module, and seem to affect APIs that can be preffed on, like navigator.connection. Some of the test timeouts seem to belong to the latter category.

(Clouserw) #2

Woah, thanks for the notes!

(Jared Hirsch) #3

One painful note about pushing to Try: be sure to use your moco email address when requesting commit access, be sure your Bugzilla email matches (for reviewboard integration, I think), and be sure your local SSH config uses your moco email address to connect to

I tried using a personal email address, but the Try server ( uses for SSH authentication, and I wasn’t able to get it to recognize my non-work email. Switching everything to moco email got me pushing to Try :+1:

(Leo McArdle) #4