As far as I understaind the matter, this is a pretty good answer of the general question “How to open-source except for commercial use?”:
https://www.quora.com/What-is-the-best-free-for-non-commercial-use-license-to-use-for-an-open-source-project
Published versions (changed or unchanged) must include a reference to the origin of the code (my extension source code)
That sounds like attribution, which most open source licenses require.
Commercial usage is prohibited
I am pretty sure that if you want to prohibit the usage of the packed extension (no matter by whom), you’d not even be allowed to publish on AMO.
write a custom one (which extends i.e. MPL2)?
Don’t. If you have any interest that your project is forked and extended, use a common license. Otherwise, there is just too much that could legally go wrong and nobody will fork (neither you nor the people forking are lawyers).
If you are not interested in fork (which may later contribute to your own extension), just don’t open source your code. You are encuraged, but not required to do it.
And do I have to add the license to every published version of the extension
No. But unless you do, the published version is simply not published under that license (even though the code in say your GutHub repo is).
and would the new license be with retroactive effect?
Nope. If somebody grabs the current version of your code, at any point in the future, they can use it under the current license, pretty much no matter what you do now or later.
Since that old code is still your intellectual property (licensing doesn’t change that), you can always publish under additional* licenses, but you can’t revoke grands you already made (unless that was stated in the license, but open source licenses don’t do that (by definition, I think)).
Disclaimer: The above is my personal understanding of the matter, not legal advice. I am not a lawyer.
