Impossible to upgrade to manifest v3 for extensions that require constant/persistent listeners

Add-ons Development
#1

I have upgraded my extension’s manifest to v3 and have encountered major issues that will force me to downgrade back to v2 as a direct result.

My extension requires the following listener to always be active and it is not accessible at the content script context, so there is no way for me to try and work around this in v3:

browser.webRequest.onBeforeRequest

There are other permissions as well that have the same problem, but the above one is the most important one to make the extension work properly.

The biggest issue with v3 is this:

  1. The extension is configured to block and change web requests for website www.example.com
  2. All works well when a page www.example.com loads for the first time, any configured changes run in the webRequest for any scripts/html loaded in that page
  3. But after 30s the background script stops running
  4. As a result, any new requests generated inside the page www.example.com are no longer being captured by the webRequest listener from the background script because it is no longer running

This is a huge problem in v3 and as far as I can find there is no way around this.

I was initially expecting the requests to automatically wake up the listeners, or at the very least if such listeners are in use then the background script stays alive for as long as the listener is active, or while the respective webpage is opened, but it always stops the background script after about 30s.

The only fix for this that I could find so far is to just drop v3 and revert to v2 with persistent background script.

Is this something that will be addressed in v3 anytime soon or will we be denied the ability to run any extension that requires such always active logic?
I am very concerned that such an obvious issue has been left unchecked like this for so long given that v3 has been out for about a year already, or at the very least I hope that I am missing a way to work around this problem and it’s just the available documentation is not either clear enough about it, or that information is not easy to find at this time.

@dotproto hope it’s ok to ping you about this since it is such a concerning limitation for anyone wanting to upgrade to manifest v3 it would be helpful to have the input from an official member.

#2

There’s not much reason to update to Manifest V3 for many extensions, unless the extension is for Android.

The webRequest listener has to be added on the top level of the event page, otherwise it won’t restart when the webRequest event fires. That does of course place limitations on temporary listeners or those with filters that are unique to a current session, like tabId, windowId, etc…

To prevent an event page from terminating, just run an extension api call every 20 seconds (like browser.runtime.getPlatformInfo()). Note, an event page can still be terminated at this time when a computer awakens from sleep mode.

Also, maybe add a listener for browser.runtime.onStartup, to make sure your event page always starts on browser startup.

#3

The webRequest listener has to be on the top level of the service worker, otherwise it won’t restart the extension when the event fires.

I’ve never heard of that before, is it documented?

In any case it won’t work for me because I have 2 of them and initialization steps outside the scope of those listeners as well.

To prevent a service worker from terminating, just run an extension api call every 20 seconds (like browser.runtime.getPlatformInfo()). Note, a service worker can still be terminated at this time when a computer awakens from sleep mode.

Unsure of what you mean by using a service worker in this context, Firefox never implemented them (yet!)


Also, maybe add a listener for browser.runtime.onStartup, to make sure your service worker always starts on browser startup.

Fortunately that one is not a requirement for me, I only need it to start when a specific website loads and stay alive for the duration of that website session.

#4

It’s easy to find in the Chrome docs:
https://developer.chrome.com/docs/extensions/develop/migrate/to-service-workers#register-listeners > Register listeners synchronously

But in the Firefox docs, it’s well-hidden:
https://extensionworkshop.com/documentation/develop/manifest-v3-migration-guide/#migration-checklist > Event-driven background scripts:
“Ensure listeners are at the top-level and use the synchronous pattern.”

https://blog.mozilla.org/addons/2022/10/31/begin-your-mv3-migration-by-implementing-new-features-today/ > How do I implement Event Pages?
“Event listeners must be added at the top level execution of your script.”

#5

Sorry, I was just working on a Chrome extension and was using service workers. I meant event pages.

Also, I didn’t explain properly. The listener has to be ‘added’ at the top level of the event page to properly restart it.

#6

I already did that and it does not work, the background script never wakes up after the first 30s for any webRequest.onBeforeRequest listener.

#7

Do you have the correct permissions granted to allow the webRequest event to fire? Maybe check the ‘Permissions’ section of your add-on…

#8

image

As far as I can tell there is only one and it is granted.

Just to recap, my problem is not when the website is opened for the first time and during the first 30s after that point, the problem is after that point and new content is loaded in the website, either by user scrolling down and loading more content, or navigating without pages actually loading fresh again. The listener is never triggered, until the website is refreshed and then the 30s start again just fine.

#9

Do you have the right types in your webRequest listener to make sure it fires when scrolling down pages and such?

It also might be helpful to check the ‘Network’ tab in the pages console.

#10

Yes. Like I said, all of it works perfectly fine during the first 30s window, and then it never works again and I cannot open the debugging console for the extension because that keeps it alive.

#11

Hey @Particle! You’re welcome to tag me :slight_smile:. I’m here to help with exactly this kind of issue.

Also, as I was read this thread I just realized that I didn’t properly respond to your concerns in the WECG thread. I was too focused on the meta discussion and missed the details about the specific problem you were experiencing. Sorry about that!

As you expected, events that background has subscribed to should wake it. In the case of blocking WebRequest listeners, I believe the network request should also be blocked until the extension is able to process the event and tell the browser how to handle it.

That said, there is a critical caveat to this guidance (which @kg_13 already called out): the browser can only dispatch an event to your extension if there is a listener available to handle it. That’s why we advise registering top level event listeners: top level registration ensures that the event listener is registered on the first turn of the event loop when the background script is evaluated. That in turn means that as soon as the script is initialized, there is an event listener available to receive the event the browser its trying to dispatch.

If you have a top level event listener, the event handler should be called before the request is sent to the network layer. If that’s not happening, there’s either a bug in your extension or Firefox.

Hm. At the end of the last paragraph I wrote I was going to ask for your source, but what you’ve described sounds like it should work as be both expected. At this point I think it’s probably best if I start exploring the issue on my end.

For what it’s worth, in addition to the migration checklist it’s also mentioned in the Event Driven Background Scripts section of the same page.

#12

Thanks for expanding on the topic.

To me my two biggest issues right now are the webRequests not waking up.

For example, setting it up to intercept any images loading when the user scrolls the page - the script dies after 30s and does not wake up again.

And he other one is the port messaging.

Once the background script dies - despite being connected to an active port listener - any message the content script attempts to send errors out because it is no longer connected after 30s pass.

However, the keep alive technique we discussed in your linked conversation has been working well so far for me.

But I still have yet to find an official documentation or statement confirming that this behavior is intended and it will not be patched in the future.

Right now that is the only thing preventing me from finally moving to V3 since I don’t want to make that kind of investment only to have it killed later on because it was not suppose to work that way.

EDIT: It has been running uninterrupted for over 11h now

image

#13

Right, an open port isn’t enough to signal that there is ongoing work; you will need to send messages on that port in order for Firefox to consider it “alive.”

That’s a very reasonable concern and solid feedback on something we should clarify in the docs.

#14

There’s a comment here about not preventing keep alive operations:

Is it possible that you could post the code you have for webRequest.onBeforeRequest.addListener…?

#15

Sure, but I think this is no longer an issue afterall.

Here is the only thing I had in my background script to test it:

"use strict";

browser.webRequest.onBeforeRequest.addListener(
    (details) => {
        console.log(details);
        return {}
    },
    {urls: ["*://i.ytimg.com/*"], types: ["image"]},
    ["blocking"]
);

However, nightly just updated for me to 123.0a1 (2024-01-08) and the problem is not there anymore, now the script does wake up correctly as soon as I scroll on the YouTube page to load more images.

Unfortunately I do not know which version I had prior so I cannot report that as a bug with any worthwhile information, but I assume this is no longer relevant now that the original problem no longer exists.

So the top-level listeners is indeed working for me, and the port messaging appears to be a limitation with no way around it. If the timed loop is not going to go away I can live with that, but it would be way better if we could just have a persistent flag again to make life easier for us and have a better coded extension that does not run unnecessary cycles constantly in the background wasting energy.

#16

I keep thinking just how much more complicated the Moz team is making it for themselves and us by simply not adding a persistent flag to the manifest.

At this stage the persistence behavior is being manually handled individually for every sensible specific api usage, and forcing devs to add a timer for every other case that is not going to be covered, when all of this could simply be avoided by setting a single flag to true or false.

It’s insanity, and unfortunately I can’t describe this whole thing in a different way.

#17

I believe this entire discussion was able to encompass a lot of useful information, and solutions for some problems that have been noted, so I think there isn’t much that can be discussed around this any longer and I’ll write down the important stuff.

  1. Content-script communication to Background-script communication via Port messaging can be done even if the Background-script is stopped
    • As long as the Content-script connects via browser.runtime.connect(...) (if not already connected) before sending a message then the Background-script will always wake up to receive it
  2. Always put any listeners at the top of the Background-script
    • If there is any logic that works before the listeners, rework/redesign that logic order and start with the listeners first and only then do the rest if possible.
  3. If that is not possible, and the extension requires the background script to be always running, we can make use of an API heartbeat to keep it alive for as long as it is needed
    • For example, if the extension only needs the Background-script to stay alive while a Content-script exists, that Content-script can send messages to the Background-script via Port messaging every 10s or so
    • Another example, if the extension needs the Background-script to stay alive regardless, the extension can implement a ghost (iframe) page that does the same thing as the example above

I used the example of Port messaging as a way to keep the Background-script running due to frequent API activity, but it is possible a different API can be used. I personally used that one only so far, and I have not yet explored other options, but it is possible that, for example, changing the local storage can have the same effect, just be aware that what works for Firefox might not work for Chrome if you’re building a cross-browser extension:

One thing was made very clear; Firefox/Mozilla needs to improve their documentation a bit more if they want more devs to adopt MV3, these are important aspects that should by no means be only available in sparse footnotes with no clear examples, or guidance (top-level listeners, keep alive techniques, etc.).

Thanks everyone for all the help and contributing to the discussion.

How to stop a background script from going idle in MV3?