Strange network bug on the extension page

Hello everyone,

I have some strange problem when porting my Chromium extension to Firefox. My extension tries to load a document from remote URL via XMLHttpRequest with type “document”, then it populates an iframe on the options page with the contents.

It works fine in Chrome, but in Firefox something strange happens.

  1. Onload event never fires
  2. If I try onreadystatechange it fires with status 2, then with status 4, code 0 and empty response
  3. Changing response type does not help
  4. In Network tab I see response code 200, the color is green and I can read the response, which is correct (!)
  5. If I run fetch() in console, I get the following error:

TypeError: NetworkError when attempting to fetch resource.

But if I use the same line of code on some other site (not my options page), I get an explanation before this line:

Content Security Policy: The page’s settings blocked the loading of a resource at … («connect-src»)

As I don’t see this message on my options page, I suppose that the permissions are correct (and also because I can see the response in the dev console). However, JavaScript code on the page of my extension does not get access to the response data for some reason.

Any help will be highly appreciated.

Does this work in Chrome? It sounds like the CSP header in the page response prevents you from loading the page in iframe in foreign host.

If that’s the case than I would say you have two options:

  1. change the CSP on the server (if it belongs to you and if you are ok with that)
  2. intercept the response and remove the CSP header (specifically headers: ‘x-frame-options’, ‘frame-options’ and ‘content-security-policy’)

Are you viewing your Options page inside Firefox’s Add-ons page (as a frame)? If so, could you test opening it in its own tab to bypass security restrictions related to the Add-ons page.

1 Like

No, sorry, you didn’t get the idea. I am not loading the document into the iframe directly, in that case I wouldn’t be able to manipulate its DOM. I am making a simple AJAX call (and then I am inserting the received DOM tree into the iframe using JS). The first stage fails (not the second), i.e. I cannot even get the response from the XHR request.

And yes, it definitely works in Chrome. I can send a ZIP or a link to Chrome Web Store after they approve the extension in the next 2-3 days (I hope it won’t be rejected because of security issues or something like that).

No, I am viewing my options page in a separate tab, of course. I did not even know that it is not the only available option :slight_smile:

I guess you are absolutely right, I think I really should do it (though in Chrome it is working without that, the “<all_urls>” permission in the manifest is enough). Maybe I should read about Firefox APIs for intercepting request headers…

As for making changes to the server - it is certainly not the think I wish to do, because the idea of my extension is to let people easily test their client-side code (on their own servers), and setting the headers on the server for every project is definitely not the “easy” workflow (though of course it is possible to require that in theory).

Hmm… I don’t unserstand anything. I installed a third-party extension for intercepting and modifying HTTP headers (Requestly). It seems to work - at least it adds Content-Type header when I specify it (though the modified header does not appear in the Network tab, in case of Chrome the actual headers do appear there (I worked with Resource Override extension for some time for example). So I set up the Access-Control-Allow-Origin header with value “*”.

Then I tested the following command in Console:

fetch("<some_url>").then(response => response.text().then(result => console.log(result)))

It works with domains that are served over https, but not with domains served over http (if called from the site which is served over https). But maybe it will not be a problem on my options page.

UPD: checked that idea, unfortunately it is even worse. On my options page neither https domain nor an http one do not work telling the Access-Control-Allow-Origin is missing (though it should be added by another extension).

You know you can modify DOM probably much easier if you just inject a content script into the iframe that will do the job. So instead of downloading and modifying you could load it directly into the iframe and then use tab.executeScript to inject there your javascript.

Now regarding modifying headers, this is what you need:

browser.webRequest.onHeadersReceived.addListener(onFrameHeader, {
  urls: ['<all_urls>'],
  tabId: targetTabId,       // block only target tab
  types: ['sub_frame']
}, ['blocking', 'responseHeaders']);

function onFrameHeader(info) {
  const headers = info.responseHeaders!;
  for (let i = headers.length - 1; i >= 0; --i) {
    const header = headers[i].name.toLowerCase();
    if (header === 'x-frame-options' || header === 'frame-options' || header === 'content-security-policy') {
      headers.splice(i, 1);
    }
  }
  return {responseHeaders: headers};
}


Make sure to:

  • block only your own tab (see targetTabId)
  • unregister the handler once the page loads
  • keep “types: ['sub_frame']” to block only in iframes

Thanks for your suggestion to use content script in the iframe, that’s interesting. Though it will require rewriting half of my code, so I need to think about it.

It is happened this way historically that I had 2 separate execution engines in this project. The first one was initially manipulating the iframe directly, which was fine when using test framework on the same domain where the tested web app lives as a tool, and then it evolved into the current one (1 of 2). I don’t really want to rewrite it from scratch, because it works fine in Chrome (the current remade version of it).

I still wonder what is the problem with my XHR requests… Don’t you know the reason?

Well, if you do have "<all_urls>" permission, than you should be able to fetch almost anything. The exceptions will be those protected hosts, like other extensions pages or mozilla pages or internal about:* pages.

Are you sure it’s the fetch that fails? Any chance you could paste it here?

Also, try these in your console in your addon, they should work:

await (await fetch('https://group-speed-dial.fastaddons.com/')).text()
await (await fetch('http://www.virustotal.com/')).text()
await (await fetch('https://group-speed-dial.fastaddons.com/')).text()

SyntaxError: missing ) after argument list

I guess something is wrong with your code…

Yes, I can paste the code here, but how can it help you to debug? I think ZIP would be more useful, though for me the installation my own of extensions from ZIPs does not work for some unknown reason…

var test_instances = []

function startTest(url, test_index) {
	var req = new XMLHttpRequest()
	req.open("GET", url, true)
	req.responseType = "document"
	req.onload = function() {
		processResponse(req.response, url)
		test_instances.push({ index: test_index, url: url, pos: 0, result: null, ac: 0, cc: 0, pc: 0, xhrc: 0, errors: 0, paused: false })
		document.getElementById('ifr').test = test_instances[test_instances.length-1]
		var test = tests[entries[test_index].test_id].data
		runOperation(test, 0, test_instances.length-1)
	}
	req.send(null)
}

function sendRequest(url, method, data) {
	var ti = document.getElementById('ifr').test
	var test = tests[ti.index]
	var test_index = ti.index
	var pos = ti.pos
	var method = method.toUpperCase()
	if (!url.match(/^http/)) {
		url	= url.length ? ti.url.split('/').slice(0, -1).join('/') + '/' + url : ti.url
	}
	if (!data) data = null
	var req = new XMLHttpRequest()
	req.open(method, url, true)
	req.responseType = "document"
	if (method == "POST") {
		req.setRequestHeader('Content-Type', 'application/x-www-form-urlencoded')
	}
	req.onload = function() {
		processResponse(req.response, url)
		if (pos < tests[test_index].data.length-1) {
			ti.paused = false
			runOperation( tests[test_index].data, pos+1, test_index)
		}
	}
	req.send(data)
}

function processResponse(response, url) {
	result = response && response.children[0]
	var ifr = document.getElementById('ifr')
	while (ifr.contentDocument.children.length) {
		ifr.contentDocument.removeChild(ifr.contentDocument.children[0])
	}
	var uri = new URL(url)
	result.innerHTML = result.innerHTML.replace(/(href|src)="\//g, '$1="' + uri.origin + '/')
	       .replace(/(href|src|action)="(?!http:|https:|android-app:)([a-z0-9_-]+)/g, '$1="' + uri.origin + uri.pathname + '/$2')
		   .replace(/(fetch\(["'])\//g, '$1' + uri.origin + '/')
		   .replace(/(fetch\(["'])(?!http:|https:|android-app:)([a-z0-9_-]+)/g, '$1' + uri.origin + uri.pathname + '/$2')
		   .replace(/(\.open\(['"](?:GET|POST)['"],\s*['"])\//g, '$1' + uri.origin + '/')
		   .replace(/(\.open\(['"](?:GET|POST)['"],\s*['"])(?!http:|https:|android-app:)([a-z0-9_-]+)/g, '$1' + uri.origin + uri.pathname + '/$2')
		   .replace(/url\(([^("]+)\)/g, 'url("$1")')
		   .replace(/(href|src)=([^" \t]+)(\s+)/g, '$1="$2"$3')
		   .replace(/(url\("?)\//g, '$1' + uri.origin + '/')
		   .replace(/(url\("?)(?!http:|https:|data:)([a-z0-9_-]+)/g, '$1' + uri.origin + uri.pathname + '/$2')
	ifr.contentDocument.appendChild(result)
	ifr.contentWindow.eval("window.alerts = []; window.confirms = []; window.promts = []; window.xhr = []; " +
	                       "window.store = {}; var prevent_xhr = false; " +
	                       "var confirm_answer = true; var promt_answer = 'lightning'\n" +
	                       "window.alert = function(msg) { console.log(msg); this.alerts.push(msg) }\n" +
	                       "window.confirm = function(msg) { this.confirms.push(msg); return this.confirm_answer }\n" +
	                       "window.promt = function(msg) { this.promts.push(msg); return this.promt_answer }\n" +
	                       "if (!window.XMLHttpRequest.prototype._open) { window.XMLHttpRequest.prototype._open = window.XMLHttpRequest.prototype.open; window.XMLHttpRequest.prototype.open = function(method, url, async) { window.xhr.push({ method: method, url: url, data: null }); if (!window.prevent_xhr) this._open(method, url, async); }}\n" +
	                       "if (!window.XMLHttpRequest.prototype._send) { window.XMLHttpRequest.prototype._send = window.XMLHttpRequest.prototype.send; window.XMLHttpRequest.prototype.send = function(data) { window.xhr[window.xhr.length-1].data = data; if (!window.prevent_xhr) this._send(data); }}\n" +
	                       "window.parent = null; window.locationUrl = '" + url + "'\n" +
	                       "if (!window._fetch) { window._fetch = window.fetch; window.fetch = function(url, options) { window.xhr.push({ method: 'GET', url: url, data: null }); if (options.method) window.xhr[window.xhr.length-1].method = options.method; if (options.body) window.xhr[window.xhr.length-1].data = options.body; if (!window.prevent_xhr) return window._fetch(url, options); }}\n" +
	                       "window.parent = null; window.locationUrl = '" + url + "'")
	var s = result.getElementsByTagName('script')
	for (var i = 0; i < s.length; i++) {
		try { ifr.contentWindow.eval(s[i].innerText) } catch(ex) { console.error(ex) }
	}
	fireEvent(ifr.contentWindow, 'DOMContentLoaded')
	fireEvent(ifr.contentWindow, 'load')
	if (ifr.test) ifr.test.paused = false
}

What really fails here is this:

	var req = new XMLHttpRequest()
	req.open(method, url, true)
	req.responseType = "document"
	if (method == "POST") {
		req.setRequestHeader('Content-Type', 'application/x-www-form-urlencoded')
	}
	req.onload = function() {
		processResponse(req.response, url)
		if (pos < tests[test_index].data.length-1) {
			ti.paused = false
			runOperation(tests[test_index].data, pos+1, test_index)
		}
	}
	req.send(data)

It looks like the email notification with the preview message replaces apostrophes with (whatever that is :smiley:).
Try to copy it from here, not email.

Also, when I said to paste the code here, I meant just the specific fetch that fails with a specific URL. You’ve mentioned it won’t work with “http” protocol.

The problem is not apostrophes, I think, I copied it from here, the single quotes were okay.

You’ve mentioned it won’t work with “http” protocol.

No, on my extension page neither of the protocols work.

What Firefox version are you running? You must be able to execute it without any SyntaxError issues.

I am running Firefox 62.

Tried in Firefox 77 portable (or whatever is the latest version), it does not complain about the syntax, but still fails:

Content Security Policy: The page’s settings blocked the loading of a resource at https://group-speed-dial.fastaddons.com/ (“default-src”).

I tried to run it from the settings page and from the regular https site, it does not matter. And it works perfectly in latest Chrome (!)

Btw, the problem with my extension is also present on the latest version of Firefox (I checked it first before posting here), so it is not a fixed bug - I guess I’m not really wasting your time for nothing with this

So first of all just a friendly reminder :slight_smile: that Firefox 62 is not supported my Mozilla anymore and doesn’t contains many security patches. So you really should upgrade to at least Firefox ESR 68.

The CSP error you see you should not see if you execute it from your addon page if you have the "<all_urls>" permission in the manifest. I’ve tested it, so I’m 100% sure :slight_smile:.

Now some bad news. From what I can see in your code, you are using or violating two policies:

  • “Unsafe assignment to innerHTML”
  • “eval can be harmful.”

Both of these are kind of forbidden and your addon may not pass the review.

I know why innerHTML is bad (performance impact and sometimes security). But what is really funny is the thing that it complains about any innerHTML assignment, not just for <script> tags (that I have learned to avoid using assigning to textContent). It is impossible to write readable maintainable code without it completely, yes, I use it a lot in the internals. Thank God it passed the review despite of these warnings.

What for eval - it is completely impossible to make things working without it in this particular case. Btw, I don’t accept anything from the user there.

Firefox 62 is not supported my Mozilla anymore and doesn’t contains many security patches

I had issues with focus on contenteditable DIVs on a very important site (local russian social network) in its IM module from version 63 (also in 64 and 65), that’s why I stopped updating. Maybe now it is already fixed, but I don’t care :smile:

The CSP error you see you should not see if you execute it from your addon page if you have the "<all_urls>" permission in the manifest. I’ve tested it, so I’m 100% sure

How could this be? I have the permission, and it is not working for me.

screenshot

Maybe I should post my manifest here?

Yes, please do.

Know that both, “eval” and “innerHTML” are a security risk and for the reviewer can be really hard to tell if you are using “safely” or if you actually have “good intentions”.
Also both can be replaced with safe alternatives, almost every time!
In this case your eval is basically just executing javascript in the iframe, right? You have content scripts for that. And “innerHTML” is just DOM manipulation.