Efficient storage of ArrayBuffer / Uint8Array

I’m building new extension that will utilize Crypto API which returns ArrayBuffer, plus you need to store initialization vector which is Uint8Array.
I will be storing these in the browser.storage.sync area which has only 100KB of space so I want to use it wisely - not wasting space.

However there are two issues:

  1. I’m not sure if storing these types there will really use only the space required (and I can’t measure it due to missing getBytesInUse function)

  2. The main issue is that I’m trying to be compatible with Chrome but Chrome cannot serialize those types at all

To solve both problems, I would like to convert those values into something easy and more predictable, maybe string???
But in a way that it won’t take more space, so definitely not base64 string.
Or is there a better way?

Before we had ArrayBuffers and friends we’d always use strings to store byte data in JS. Essentially one character maps to one byte. You can convert to and from using String.prototype.charCodeAt and String.fromCharCode. Now, depending on the representation format the storage uses for JS strings that means either one byte of information takes one byte, or if the JS strings are saved as “UCS-2” strings, two bytes. Obviously you could also use a number, possibly a big-int, though I’m not sure if there are any benefits compared to the very raw byte storage of strings.

1 Like

Thanks!

I’ve just spent a lot of time trying to convert it to string and back (using TextDecoder/Encoder) but without success.
I will try the charCodeAt, that sound simple enough to work :slight_smile:, but I need to check the UCS-2 first (which doesn’t look like a light reading :slight_smile:) .

I really like the idea with storing numbers, but I guess it won’t be easier nor smaller. Sadly storing the BigInt is again not supported in Chrome :frowning:

But maybe if I use Uint32Array view and convert it to array of numbers, it could be good enough? I need to find out how big container is internally being used for numbers (maybe 4 bytes?).

…it’s study time! :slight_smile:

EDIT:
So using Uint32Array numbers didn’t worked neither, something about not being aligned…
The only thing that worked is processing it byte by byte to string:

export function bufferToString(buf: Uint8Array | ArrayBuffer) {
  return String.fromCharCode(...new Uint8Array(buf));
  // return String.fromCharCode.apply(null, new Uint8Array(buf));
}

export function stringToUint8Array(str: string) {
  const buf = new ArrayBuffer(str.length);
  const bufView = new Uint8Array(buf);
  for (let i = 0, strLen = str.length; i < strLen; i++) {
    bufView[i] = str.charCodeAt(i);
  }
  return bufView;
}

Inspired by code here, but modified to use 8bit, not 16 (which again caused the issue with not being aligned).