I'm in a very desperate need of tricking garbage collection into triggering

Hi,

I develop Ultrawidify, which is an extension that crops letterboxed videos to fit a 21:9 monitor properly. One of the features is automatic aspect ratio detection.

The way automatic autodetection works is by taking the video element, drawing current frame to canvas every X seconds with ctx.drawImage() and then getting pixels from the canvas through the magic of ctx.getImageData().

Main function looks roughly like this — I’ve omitted some less critical parts:

async main() {
  while (cond) {  // yes I know. setInterval exists.
     checkFrame();
    await sleep(interval);
  }
}

And the checkFrame() boils down to this:

async checkFrame() {
  this.context.drawImage(this.video, 0, 0, this.canvas.width, this.canvas.height);
  const imageData = this.context.getImageData(0, 0, this.canvas.width, this.canvas.height).data;   // <----- problem child.

  // do stuff with imageData, lots of stuff
  // check results
  // return nothing
}

As far as I can tell, that’s pretty much by the book. My code has no memory leaks and there’s no way to improve my interactions with canvas at all. Reusing existing references is not an option because ctx.getImageData() will always return a new one. After checkFrame() finishes executing, imageDatashould get garbage-collected eventually because unused reference.

And that’s what usually tends to happen. If I open a youtube video, ‘memory’ tab in devtools will peg memory usage for the page at 70-120 MB, which is reasonable. After a while though, the memory usage is going to start to rise (personal record: to the tune of 20+ GB and no I’m not kidding).

Today, I’ve only managed to get it up to 1 gig, but that’s still way too much: https://imgur.com/btcUlpO

If you take a look at the ‘dominator’ view, you start noticing funny stuff: there’s tons of ArrayBuffer objects 921664B big:
Imgur

Which is coincidentally roughly how big I expect imageData to be (4 bytes per pixel × 640 pixels wide × 320 pixels tall = pretty much this).

Going to ‘about:memory’ and clicking ‘GC’ button will bring the number back from multiple gigabytes to what it should be — notice drops in fourth and last snapshot:
Imgur

Autodetection is the kind of feature I don’t want to go back on, and it’s mildly important that you run it frequently enough in case video keeps changing aspect ratio. Are there any lesser-known workarounds that would help me curb the RAM usage due to shitty garbage collection?

Things I’ve tried so far

  • Googling. No results.

Things that I’m looking at

  • Web workers.

I’ve found this bit about transferable objects. If I understand this right, sending imageData to a worker like suggested here and killing the worker once it’s done processing would serve as a kind of forced garbage collection — or am I wrong on this one?

1 Like

Very interesting stuff!
Please keep us posted if you find a solution in the meantime (those workers sounds like a very good idea).

I have a similar problem - when creating thumbnails, I’m resizing a lot of small images and memory goes up really fast, but goes down very slowly. It can easily take from 0.5GB to 1.5GB of memory.

As a general suggestion, avoid using async functions for performance sensitive number crunching. Firefox doesn’t JIT compile them.

It’s quite likely that moving
// do stuff with imageData, lots of stuff
to a new function without async would improve your performance. (This probably won’t help your GC behavior)

As a general suggestion, avoid using async functions for performance sensitive number crunching. Firefox doesn’t JIT compile them.

This is going to be useful once/if I move that stuff to a worker and don’t have to care about long-running functions blocking the rest of the page, so thanks for the tip.

In the mean time, here’s my reasoning for using async:

If my understanding is correct, liberal usage of async/await — while not being too helpful for speed in raw number crunching — can positively affect overall performance, because it doesn’t completely block other stuff on the page from executing. If you’re having a function that could run for a long time and don’t use async/await, then the page will stop responding for however long it takes for the function to execute. In the mean time, using async/await (the // do stuff with imagedata is a lot of await some_function() calls) still allows other scripts on the webpage to execute between those calls.

Does it makes sense or did I misapply something I found while googling once?

Could you elaborate?
Is there some issue on bugzilla for this?

So if I understand this correctly, if I configure my transpiler to generate old code (without async / await), it will actually speed-up my addon? I will try that.

It’s not really that simple - you have to use actual asynchronous API (like Fetch API) or run your JavaScript on a different thread (using workers, or send your work to background script) in order to be really non-blocking. If you just make your function async, it won’t make the execution asynchronous (parallel).
There is a really good video by Jake Archibald that explains async / await / Promise execution:

@31:00 onwards seems relevant to my await/async abuse for ghetto concurrency.

Was just about to say that I don’t need proper multithreading, abusing event loop through async/await does the job just fine even if that’s ghetto/pretend-concurency, but then came the proverbial asterisk about microtasks and took care of that misconception.

Yikes, then. Guess I was lied to.

(BTW, thanks for the video)

You can easily find the related bugs on bugzilla. It goes pretty deep, since currently generators can also not be optimized to the fastest possible stage.

If using transpiled es3 code is faster than built in async functions is up for experimentation, and probably really depends on the specific usage.

Thanks. I’ve read this new article about it here:
https://hacks.mozilla.org/2019/08/the-baseline-interpreter-a-faster-js-interpreter-in-firefox-70/
But I couldn’t understand much… this is much more complex than I expected :slight_smile:

EDIT:
I’ve finished refactoring today to perform all canvas operations directly in the tab, not in the background script but it seems that it didn’t helped. Maybe it’s because it’s loaded as temporary…??

Because the memory wasn’t released at all, not even after closing the tab, reloading whole addon and running GC. It went up from 900MB to 2.9GB during the process, then it went down to 2.2GB.

Then I’ve recompiled my code to ES3 but it didn’t helped, the behavior was the same (plus part of my addon got broken due to unsupported API).

I’ve also tried it in Chrome, there it went from 800MB to 1.5GB during the process and and then back to 1.1GB when the tab got closed. Then after a while it went to something below 1GB.

So I would say there is really something wrong with canvas in Firefox.

Good luck! Sounds like a difficult problem. In old times, forcing GC was possible but not with WebExtensions.

I was reading about your problem, and here was a list of things that came to mind.

  • As a similar note to your transferable objects, I wonder if you can just trigger the transfer without a worker by something like ArrayBuffer.transfer(imageData.data.buffer). Regarding ArrayBuffer.transfer MDN says:

The ability to detach an ArrayBuffer gives the developer explicit control over when the underlying memory is released. This avoids having to drop all references and wait for garbage collection.

Sounds like what you’d like, right? I’m not sure what happens to the view above that or if this is possible when it is a readonly property but probably worth a shot.

  • As a less desirable option, I wonder if destroying the surrounding canvases by e.g. calling setup() again might act as a bigger hammer to indicate you’re done with the data.
  • Also, I took a look at the code (here, right?) and while it does seem like it is not hanging onto any references, it didn’t also look like it nulls out the known references explicitly. While in theory GC can clean it up, you might be making it work a little harder. I rather doubt it though given the patterns I saw.
  • Random note: it looks like there was also a black frame test that called .getImageData() Do you know which ImageData is getting hung onto?
  • A yucky trick in other languages has been to create “memory pressure” by creating dummy objects that then trigger the GC to run. I don’t know if that’s possible here but… I suppose it could be a last ditch effort.

Have you had any luck in fixing this yet?

Thanks for the reply.

Have you had any luck in fixing this yet?

No, I haven’t had much time to work on it. The last month I’ve been mostly busy and before that I’ve got some high priority bugs to fix, so I’m only now coming back around to it. Just to answer a few questions and stuff, and maybe add some clarifications.

I was reading about your problem, and here was a list of things that came to mind.

  • As a similar note to your transferable objects, I wonder if you can just trigger the transfer without a worker by something like ArrayBuffer.transfer(imageData.data.buffer) . Regarding ArrayBuffer.transfer MDN says:

The ability to detach an ArrayBuffer gives the developer explicit control over when the underlying memory is released. This avoids having to drop all references and wait for garbage collection.

Sounds like what you’d like, right? I’m not sure what happens to the view above that or if this is possible when it is a readonly property but probably worth a shot.

That’s a nice catch. Yup, sounds like what I’d like. I’ll try to experiment with this the next weekend. Sounds like constantly re-creating the canvas could be potentially another performance issue (though maybe not so significant compared to the rest of the code).

Also, I took a look at the code (here, right?)

That’s right.

and while it does seem like it is not hanging onto any references, it didn’t also look like it nulls out the known references explicitly. While in theory GC can clean it up, you might be making it work a little harder. I rather doubt it though given the patterns I saw.

If I recall correctly, I tried with assigning null and/or undefined to imageData at an earlier stage and it didn’t really help. Or — to be more accurate — setting ImageData to null or undefined prevented runaway memory usage right up to the point I actually wanted to do anything with the pixels. As soon as I tried to do anything with the data, runaway memory usage was back.

Random note: it looks like there was also a black frame test that called .getImageData() Do you know which ImageData is getting hung onto?

Possibly both. Not completely sure about the black frame test, but the “big one” is definitely having issues disappearing because at just shy of 1 megabyte (640x360 pixels, 4 bytes per pixel + some irrelevant overhead from ArrayBuffer object), it’s a very distinct object when you do memory snapshot.

I don’t dare scrolling farther down when these memory runaway situations happen because browser (and general system) responsiveness goes really bad really quickly.

As a similar note to your transferable objects, I wonder if you can just trigger the transfer without a worker by something like ArrayBuffer.transfer(imageData.data.buffer) . Regarding ArrayBuffer.transfer MDN says:

I’ve looked at this matter during the holidays. Turns out that ArrayBuffer.transfer() is not yet implemented by any browsers. So that’s no dice on that front, but still thanks for suggestion.

Ah, nuts. Sorry to steer you towards an API with no support yet. :cry: Any luck on solving the overall problem though?

No, not yet. Especially since other bugs keep popping up all the time, so I keep getting sidetracked.

V V tor., 5. nov. 2019 ob 03:28 je oseba Zephyr via Mozilla Discourse notifications@discourse.mozilla.org napisala: