Proposal: native.js to "embrace & extend" the WebExtensions API

Add-ons Development
#1

This proposal aims to help complex and innovative extensions prosper also in the new Web-centric add-on ecosystem announced in https://blog.mozilla.org/addons/2015/08/21/the-future-of-developing-firefox-add-ons/

Add-on developers would be allowed to include, in an otherwise “pure” WebExtension package, a file similar to the bootstrap.js one currently supported by XPI add-ons, named “native.js” to better communicate its role. JavaScript code contained in this file would be subject to close scrutiny in AMO’s editorial policy. It would be executed on startup with chrome privileges, and it would have the ability (by importing ad hoc module or by injecting a global?) to interact with the WEAPI components through /runtime\.(connect|sendMessage|onMessage)/ IPC.

This would be a “natural” and losely coupled mechanism to isolate hacks and radical experiments from WEAPI code and to gradually port those bits into the new world, still making no promise to support anything (XUL, XPCOM, not even the SDK) in the long run.

We would retain the advantage of putting “new style” extensions in their own add-ons process, while the native.js scope would still live in the parent process and have access to whatever chrome-privileged JavaScript have access over the time (XPCOM today, browser.html and/or a Servo-provided API tomorrow?), with absolutely no guarantees of stability though (“Hack it at your own risk!”).

Packages containing a native.js file would raise a red flag both in AMO reviewer dashboard, to be examined more carefully, and in the user-facing installation/update permissions UI, asking for “browser super powers (can kidnap all your files into a cloud-based wormhole and ask for a ransom)”.

In this landscape we are still going to deprecate XUL (overlay) add-ons and the SDK’s require(“chrome”) within the planned timeline, providing “orphaned” developers with a strong incentive to migrate on WEAPI (especially for the front-end and for whatever functionality is already supported): nonetheless, they could resort to implement temporary hacks in native.js, not just for backward compatibility, but (especially) for experimenting with new concepts.

Anyway such hacks would need to go away, one by one, whenever WEAPI gets rich enough to replace them. To enforce this, we might even eventually introduce some mechanism causing calls into XPCOM APIs whose functionality is already surrogated by the WEAPI to throw when their caller is an add-on. However we should be very explicit about Mozilla’s freedom of removing legacy interfaces which don’t make sense anymore or hamper the browser’s evolution at any time, maybe with a 1-month courtesy notice and a replacement hint, because “native.js comes with no guarantees”. Alternatively, add-on developers who choose this risky path could just subscribe mozilla.dev.platform or an ad-hoc group (mozilla.dev.morgue?) to be alerted of such disruptive changes.

Update
In order to decentralize and distribute as much as possible the process of augmenting and refining the WebExtensions API, native.js should come with two key features:

  1. A meta-API to let add-ons declare, implement and export experimental additions to the core WebExtensions API. This enables add-ons developers to prototype their proposed new APIs and package them as "API provider" WebExtensions which may be used, tested and improved by other developers who share similar needs, reducing code and effort duplications and leading to a community-driven evolution path for the WebExtensions platform. The most valuable prototypes would be eventually adopted by Mozilla and included in the officially supported stable WebExtesions API.
  2. A solid dependency resolution system to promote modularization and reuse of API provider extensions, which should be installed and updated reliably and silently when required by another extension with no hassle for the end-users. This, of course, poses interesting questions about licenses and privacy policies, but nothing that we couldn't eventually settle.
Add-on developers should be actively encouraged to separate their "business logic" from their APIs by splitting them in a two or more different WebExtension packages: one "main" browser add-on and one or more API providers. Inclusion of native.js should ideally be limited to the latter, which would likely need to be modified less frequently and possibly be shared across multiple "main" add-ons. The longer approval times due to the compulsory AMO review that every native.js extension must undergo would probably be quite an incentive to embrace this separation model, but we should spell this message out in the documentation and within every each AMO review.
8 Likes
#2

Sounds like we’ll keep making WebExtensions rich enough till we’re right back with XPCOM :stuck_out_tongue: haha

One thing about being in its own process. Lots of code for different api’s (winapi, com, objc, x11, gtk, etc) need to run from the same process (via js-ctypes). That’s a huge worry of mine because my favorite addons are firefox only and they do this kind of stuff.

I like that “meta-api” thing. It sounds like addon devs can write modules and then share.

That would be awesome if we could set that up right now, i write modules that are basically workers that do one thing in a cross-platform/os way (like async launch file), with js-ctypes. I would love to share that with the moz addon dev community, they can use it and improve it. :smile:

1 Like
#3

While some dependency managers are emerging (e.g. Composer for PHP), they are mostly pure developer-oriented tools. When it’s something that involves end-users (and this is our case… although you said “with no hassle for the end-users”), the world is moving away from dependency hell (e.g. most OS X packages, Ubuntu Snappy etc.). So, about the “dependency resolution system”, I think it’s thanks but no thanks.

#4

I’m happy to see the general concept, but I don’t see how it solves the functionality question other than to introduce a new ad-hoc glue piece to be later deprecated itself. I guess that’s better than the alternative, though.

Personally, I’d need APIs for writing a protocol handler, low level raw sockets and something to create a data stream for the docshell. If this lets there be a glue middleware piece that I can write for and utilize that can be replaced with something later without upending the higher level code, though, I’d certainly consider it a start.

#5

I think this is a good proposal. I’d like to start prototyping something like this soon.

I do think it’s important for the native.js component to be able to be updated separately, probably as separate add-ons. A problem we had in Jetpack is that people would create reusable modules, but then they would just be copied into every add-on that used them. A couple non-e10s-compatible modules for things like content policies have proliferated, and we have no way to fix them aside from asking every add-on to update.

The versioning could be tricky. I think a reasonable way to do this would be to forbid non-backwards-compatible changes to native.js components. If you wanted to make a change that’s not backwards compatible, you would have to create a new native.js extension.

Perhaps the native.js component could add APIs that other WebExtensions could use. So instead of using the message passing API, you would use a normal-looking API. That way we could seamlessly convert the native.js component into a built-in API.

2 Likes
#6

I’d be wary about the backwards compat piece.

Fixing bugs isn’t backwards compatible, in general. As you mentioned, it’s bad for add-ons to be stuck on buggy versions of code, but they may very well depend on behaviour that’s found to be buggy.

#7

Bill, thank you very much for weighing in.
I heartily agree with all your considerations here, especially those about

  1. the ability of adding "APIs that other WebExtensions could use" ("a meta-API" in my proposal), which would make integration far more intuitive than message passing, provided that which process any API is usable from is clearly documented;
  2. the need for native.js components to be separately updatable, which can be fulfilled either by "actively encouraging" (as I wrote, thinking about the AMO review process) or even forcing by policy (as maybe you suggest) developers to provide them as separated add-on packages and by providing dependency support.
Needless to say, the part I've appreciated most is "start prototyping soon" ;)
#8

Not at all happy with this, specifically the “would raise a red flag both in AMO reviewer dashboard, to be examined more carefully”. The Mozilla review process makes the proverbial continental drift look fast, and from the description it sounds like whatever uses native.js would be ineligible for automated signing. I’ve been putting in hours in refactoring and scrapping interesting features just to keep auto sign eligibility, so I can service the needs of my users with emergency patches or instrumented debug builds.

#9

I think if it uses native.js it will just go through the same review process that it goes through today.

#10

This is very hacky indeed. And seems to be evading the real issue.

If all features are going to exist in webextensions, then I don’t see why we should have a native.js at all, using something low-level when a better non-low-level thing exists should be disallowed, period.

The real question that everyone is thinking about is this, what happens for extensions that

  1. Read/Write to the filesystem, and why shouldn’t they be allowed to do so if it’s required for their legit functionality? Don’t users install them knowing they open their documents from the filesystem?

  2. What happens to the APIs for getting OS info?

  3. Why should they be disallowed for extensions that really use them for something legitimate? Without giving any other alternative? What’s the benefit of doing that?

Let’s be clear. If something is doable without going low-level, then by all means, revoke any extension that doesn’t use the latest and greatest. But if an extension does need low-level functionality to work, then let’s allow that as well. Be it native.js or whatever.

#11

I’m just a user, but if the essential extensions like DownThemAll! go away then I’ll probably be saying goodbye to Firefox.

http://www.downthemall.net/the-likely-end-of-downthemall/

#12

this is a funny remark … how can a highlevel module be “better” than a lowlevel module?

#13

ohh man becasue mozilla is cheap enough to not even pay a few additional reviewers , they rather restructure their whole addon stuff and becasue one of 1000 addons has some strange code in it , everybody is supposed to remove features from his addon ? looooool , good luck with that

you guys depricated old and inroduced new xpcom modules for years now and it was never a problem , so why is it a problem now ? becasue you need parity with chrome ? lol
the best chrome addons have nothing on the best firefoxaddons , just saying

edited for profanity reasons , iwas angry and trolling

#14

sorry for being so angry before but this security xxx is getting out of hands imho.

let me explain with a most simple example:
and addon wants to trigger fullscreen and play a list of webstreams.
becasue of the awesome security model it can´t do that. so instead of just simply disallowing it for webscripts its also disallowed for your addon-code. WHY?

so now we have the situation where we must per lowlevel api change the fullscreen protection off (its in the config) , then send message to our webscript , which then triggers fullscreen , send a message back and on arrival turn the fullscren protection back on. lol thats really efficient and secure lol …

and now you want us to EVEN firewall against the xpcom and chrome.xul with yet another limiting message api … this is getting crazy and helps no one but the hardware manufacturers lol becasue each new iteration of firefox needs more and better hardware !

shouldn´t you rather employ new reviewers who do this professionally all day , who make sure NO addon is programmed with sniffing and trojan horses ? and enforce downloading only through amo ? that would be enough to make everything supersafe forever.

when i heard first about browser.html i thought “ohh how awesome now we can modify firefox to the maximum limit , like change everything to everything” :slight_smile: … now these posts indicate we probably can´t even have basic access without putting a good strain on the cpu having to walk through 3 message apis all with packer unpacker , rectifier, security checks, buffer and whatnot and then only exchange strings , arrays .

i also think you guys fail to see what your most used addos are ( minus the adblockers) … thats stuff like ss google translator , menuwizard and tabmix plus are all with advanced function you wanna take away from them or make it as hard as possible. not a good idea.

the customizability of firefox is its strong point and if you really wanna do that i see firefox failing and maybe even splitting a opensource fork which will try to keep functionality intact, and taking a good chunk of oldschool users with it. this happend before, i dopn´t want that to be honest.

just my 2 cents …

#15

You are posting in the wrong place, this topic is for discussing a proposal to allow WebExtensions to still do a lot of the things you mentioned even. [1] I suggest starting a new topic, or look around for a more appropriate one; I recall at least a couple that exist, although I don’t have links off-hand.

BTW, implementing WebExtensions is not (just) about security. :wink:

[1] https://wiki.mozilla.org/WebExtensions/FAQ#Won.27t_this_limit_experimentation.3F

#16

What is the license requirement for inclusion into the official API?

#17

Hi there, I was wondering what the status was on this. If we will be allowed to use framescript, bootstrap, and chromeworker (ctypes) without any XPCOM (just Services.jsm from just bootstrap) I have two modules I was hoping to contribute:

(1) is ostypes, its js-ctypes types defined for windows (win api, com), mac (carbon, corefoundation, cocoa, libc), *nix (gtk+, x11, xcb, libc) - this one is completely ready, it is available as a git submodule - https://github.com/Noitidart/ostypes - i didnt write a readme.md for it yet

(2) a module that allows any scope (bootstrap, content, framescript, worker) to call a function in the other scope, get a callback (the callback pathway can be kept open and progress updates can be sent to it). internally the communication api uses ports between everything. bheind the curtains, it will transfer transferrables (ArrayBuffer’s/Port’s) if the transferrable was marked for “transfer where possible”. (transferring happens between: worker<->bootstrap, bootstrap<->in process content, framescript<->content) (between bootstrap<->framescript as they are different process, it is copied) - I haven’t yet turned this into a git submodule but the code is available here - https://github.com/Noitidart/Screencastify/

for example to call from content to main worker you would have this code from content page:

callInWorker('countdown', {from:10, to:5}, function(aArg) {
    console.log('time left:', aArg.timeleft);
    if (!aArg.__PROGRESS) {
       console.log('coutndown completed');
    }
});

This call will pass from content -> framescript -> bootstrap -> worker. And then respones from worker will trickle down worker -> bootstrap -> framescript -> content. And when the final value is sent from worker (!aArg.__PROGRESS) then the pathway is destroyed/cleaned-up

#18

Made a submodule out of the comm api, which I mentioned in my previous post. - https://github.com/Noitidart/Comm

Usage can be seen in examples here - https://github.com/Noitidart/CommPlayground

##In summary this API

Example usage: From my content script i can call to mainworker with callInMainworker and it will transfer from content script to framescript, then clone from framescript to bootstrpa, then transfer from bootstrap to worker, and in the callback it will reverse it. And each function in the worker can do aReportProgress(blah) and it will send progress updates to that callback in content (transferring where possible), without closing the callback.

tldr: Not limited to worker and contentBehindFramescript, works between worker-worker, mainthread-worker, bootstrap-framescript, mainthread-content, mainthread-contentBehindFramescript, whatever combination you want.

#19

I have two more submodules to contribute to native.js whenever they are ready:

  1. jscSystemHotkey - Create global/system-wide hotkeys. Function keys (media etc) also supported. Github.
  1. jscFileWatcher - Watch directories to see if contained files are renamed/moved/created/deleted. Github
#20

I’m puzzle still how we can use your examples to extend the webextension APIs as mentioned in the topic. I see the bootstrap.js files on Github but i can not find the bend for the use case for webextensions at the moment. Please let me know in case of eventuality.