Hi,
I develop Ultrawidify, which is an extension that crops letterboxed videos to fit a 21:9 monitor properly. One of the features is automatic aspect ratio detection.
The way automatic autodetection works is by taking the video element, drawing current frame to canvas every X seconds with ctx.drawImage()
and then getting pixels from the canvas through the magic of ctx.getImageData()
.
Main function looks roughly like this ā Iāve omitted some less critical parts:
async main() {
while (cond) { // yes I know. setInterval exists.
checkFrame();
await sleep(interval);
}
}
And the checkFrame()
boils down to this:
async checkFrame() {
this.context.drawImage(this.video, 0, 0, this.canvas.width, this.canvas.height);
const imageData = this.context.getImageData(0, 0, this.canvas.width, this.canvas.height).data; // <----- problem child.
// do stuff with imageData, lots of stuff
// check results
// return nothing
}
As far as I can tell, thatās pretty much by the book. My code has no memory leaks and thereās no way to improve my interactions with canvas at all. Reusing existing references is not an option because ctx.getImageData()
will always return a new one. After checkFrame()
finishes executing, imageData
should get garbage-collected eventually because unused reference.
And thatās what usually tends to happen. If I open a youtube video, āmemoryā tab in devtools will peg memory usage for the page at 70-120 MB, which is reasonable. After a while though, the memory usage is going to start to rise (personal record: to the tune of 20+ GB and no Iām not kidding).
Today, Iāve only managed to get it up to 1 gig, but thatās still way too much: https://imgur.com/btcUlpO
If you take a look at the ādominatorā view, you start noticing funny stuff: thereās tons of ArrayBuffer objects 921664B big:
Which is coincidentally roughly how big I expect imageData
to be (4 bytes per pixel Ć 640 pixels wide Ć 320 pixels tall = pretty much this).
Going to āabout:memoryā and clicking āGCā button will bring the number back from multiple gigabytes to what it should be ā notice drops in fourth and last snapshot:
Autodetection is the kind of feature I donāt want to go back on, and itās mildly important that you run it frequently enough in case video keeps changing aspect ratio. Are there any lesser-known workarounds that would help me curb the RAM usage due to shitty garbage collection?
Things Iāve tried so far
- Googling. No results.
Things that Iām looking at
- Web workers.
Iāve found this bit about transferable objects. If I understand this right, sending imageData
to a worker like suggested here and killing the worker once itās done processing would serve as a kind of forced garbage collection ā or am I wrong on this one?