What is the best way to automatically test the functionality of addons (eg for Github Actions)

Add-ons
1

I am working on an extension for both Chome and Firefox, and in order to test for regressions, I am creating functional tests which will run on a CI pipeline (namely Github Actions)

For Chrome, it’s easy. I use puppeteer, start the browser with the --load-extension flag pointing to my dist folder, and it works. I just write puppeteer tests.

But I have been struggling with firefox. It doesn’t seem to simply accept the dist folder. Going to about:debugging#/runtime/this-firefox and clicking “load temporary extension” is easy for a human, but my script can’t interact with the explorer at all. So then I try to compress it to a .xpi, but firefox complains that it’s an unverified extension therefore it’s banned. So then I tried to disable that in the about:config, but that does nothing, and I learn it only works in developer firefox. So I download developer firefox, point my puppeteer to use that executable, and finally it loads… and then I get a popup saying “are you sure?” and so now I’ve been trying to click “accept”…

There has to be a better way! It seems like I need to rethink my whole approach because it’s super hacky. Is there a way to have a web test framework (puppeteer, playwright) attach to web-ext? Because launching that from my dist directory works great! Or just in general, what is the preferred approach for this sort of testing for firefox?

2

Unfortunately Puppeteer and Playwright don’t support Firefox add-ons. Those tools are built on Chrome’s DevTools Protocol which isn’t fully supported in other browsers.

More broadly, WebDriver is the common technology typically used for automated testing. It’s used by a variety of test frameworks like WebdriverIO and Capybara. Actually, WebdriverIO has documentation on how to test Web Extensions and running tests in GitHub Actions.

3

It was once considered to provide an API for unit testing in web-ext, but not implemented.
Create a mock WebExtension API for developers to run unit tests with · Issue #497 · mozilla/web-ext

Therefore, I’ve developed webext-schema myself.
I’m not sure if this will meet your needs, but please consider using it if you like.

import { Schema } from 'webext-schema';

const browser = new Schema().mock();

const tab = await browser.tabs.create.resolves({
  id: 1
});
assert.isTrue(tab.id === 1);
4

for future people, Pupetteer nowdays does support firefox addons.

5

Forget the hacky workarounds! Use web-ext to launch Firefox, then connect your test runner via its debugger port. This is the preferred, stable approach.