I think the way to go is to make a MailExtension (WebExtenion) that contains a WebExtension Experiment. From there, ChromeUtils.import(“resource:///modules/MailServices.jsm”) and MailServices.filters.addCustomAction where the argument is the custom action, that implements the nsIMsgFilterCustomAction interface.
An alternative route would be to mark the extension as legacy, but that will probably entail more work at the next TB release.