Post ArrayBuffer to loadOneTab

Add-ons Development
#1

Hi all,
I have an array buffer, I obtained it from: 1) draw image to canvas 2) canvas.toBlob 3) FileReader.readAsArrayBuffer

This is my code:

                postData = encodeFormData({
                  'image_url': 'myimg.png',
                  'encoded_image': buffer
                }, 'iso8859-1', 'myimg.png', 'image/png');

This is my encodeFormData function, it works if the image is < 200x200 in size, however if I have a bigger image, the data is corrupt, I know this because when I post the data to the loadOneTab, the website says its corrupt -

Here is a screencast showing how i know it is corrupt - https://www.youtube.com/watch?v=OG_dkEVViPE

Here is my code -

function encodeFormData(data, charset, forArrBuf_nameDotExt, forArrBuf_mimeType) {
    // http://stackoverflow.com/a/25020668/1828637

    var encoder = Cc['@mozilla.org/intl/saveascharset;1'].createInstance(Ci.nsISaveAsCharset);
    encoder.Init(charset || 'utf-8', Ci.nsISaveAsCharset.attr_EntityAfterCharsetConv + Ci.nsISaveAsCharset.attr_FallbackDecimalNCR, 0);
    var encode = function(val, header) {
        val = encoder.Convert(val);
        if (header) {
            val = val.replace(/\r\n/g, ' ').replace(/'/g, '\\\'');
        }
        return val;
    }

    var boundary = '----boundary--' + Date.now();
    var mpis = Cc['@mozilla.org/io/multiplex-input-stream;1'].createInstance(Ci.nsIMultiplexInputStream);

    var item = '';
    for (var k of Object.keys(data)) {
        item += '--' + boundary + '\r\n';
        var v = data[k];

        if (v.constructor.name == 'ArrayBuffer') {
            item += 'Content-Disposition: form-data; name=\'' + encode(k, true) + '\';' + ' filename=\'' + forArrBuf_nameDotExt + '\'\r\n';
            item += 'Content-Type: ' + forArrBuf_mimeType + '\r\n\r\n';

            var ss = Cc['@mozilla.org/io/string-input-stream;1'].createInstance(Ci.nsIStringInputStream);
            ss.data = item;

            var abstream = Cc['@mozilla.org/io/arraybuffer-input-stream;1'].createInstance(Ci.nsIArrayBufferInputStream);
            abstream.setData(v, 0, v.byteLength);

            mpis.appendStream(ss);
            mpis.appendStream(abstream);

            item = '';

        } else {

            item += 'Content-Disposition: form-data; name=\'' + encode(k, true) + '\'\r\n\r\n';
            item += encode(v);

        }
        item += '\r\n';
    }

    item += '--' + boundary + '--\r\n';
    var ss = Cc['@mozilla.org/io/string-input-stream;1'].createInstance(Ci.nsIStringInputStream);
    ss.data = item;
    mpis.appendStream(ss);

    var postStream = Cc['@mozilla.org/network/mime-input-stream;1'].createInstance(Ci.nsIMIMEInputStream);
    postStream.addHeader('Content-Type', 'multipart/form-data; boundary=' + boundary);
    postStream.setData(mpis);
    postStream.addContentLength = true;

    return postStream;
}
#2

I think the issue is there is a max size on the nsIArrayBufferStream, so I tried to chunk it, this now gives me “bad requests” so Im not doing something right in here:

        var cByteLen = 0;
        var chunkSize = 7000;
        while (cByteLen < v.byteLength) {
            var abstream = Cc["@mozilla.org/io/arraybuffer-input-stream;1"].createInstance(Ci.nsIArrayBufferInputStream);
            var thisChunkSize = cByteLen + chunkSize < v.byteLength ? cByteLen + chunkSize : v.byteLength - cByteLen;
            console.error('thisChunkSize:', thisChunkSize);
            abstream.setData(v, cByteLen, thisChunkSize);
            cByteLen = cByteLen + thisChunkSize;
            mpis.appendStream(abstream);
        }
#3

What makes you think nsIArrayBufferInputStream is the problem? Have you tried a different type of input stream with large amounts of data? It is relatively simple to stream directly from a file instead if going through an arraybuffer, since you already have a file. Have you tried transferring chunks into small arraybuffers and multiplexing those instead of trying to multiplex lots of streams out of a single arraybuffer?

1 Like
#4

Thanks Litho! Because I used to be able to do this without issue for any size with canvas.mozFetchAsStream but that got deprecated - https://bugzilla.mozilla.org/show_bug.cgi?id=1206030 . I don’t have a file, I have a blob, from canvas.toBlob :frowning:

I tried chunking it in “reply 2” and I appended that to the multipart stream but the page wouldnt even load :frowning:

#5

Have you tried it with a storage stream instead of a multiplex stream? I don’t know how multiplex streams work, but I’m a little concerned that you are doing all sorts of looping over stuff, redeclaring and reusing variables, and then hoping that everything is still there when you come to use the multiple stream at the very end. Using a storage stream would be one less possibility for fouling things up.

#6

Thanks Litho no I haven’t tried that yet :frowning: Ill give it a shot :smile:

#7

Or of course you can post blobs directly, since you already have a blob. Just concatenate several string or blob objects to a single blob, then post the result.

#8

Oh I’m not familiar with that can you please explain a bit more

#9

I’m not sure what there is to explain. You have a blob and some strings. You can concatenate strings and blobs into a blob. You can pass a blob directly as the post data. Should be a lot simpler than messing with streams.

#10

But it says here that I must wrap in streams: https://developer.mozilla.org/en-US/Add-ons/Code_snippets/Post_data_to_window#Preprocessing_POST_data

#11

Never mind. I had it in my head that you were passing it to an XHR request.

#12

Ah no problem thanks so much for trying though I actually resorted to this method first months ago, but had this same size issue. So then switched to mozFetchAsStream back then, that had no issue, but that deprecated so now I can’t escape I have to get this working haha.

#13

I’ve been playing with array buffers and they’re a pain. Handy if you want to actually parse binary data (or write it piece by piece), otherwise more trouble than they’re worth. Can’t do anything with them, can’t reliably convert them to and from formats you can do anything with.

I still have no idea what is wrong with the way you tried this, but I know how to do it a different way. No blobs, since loadOneTab() won’t take them, and no array buffers since they cause too much trouble.

canvas.toDataURL() gives a base64 encoded image
atob(dataUrl.split(’,’)[1]) gives a binary string for the image data
storageStream.getOutputStream() gets a stream you can add data to
binaryOutputStream.setOutputStream() allows binary string input to the storage stream
binaryOutputStream.writeBytes() puts the binary string into the stream
outputStream.newInputStream() can be used for your post data stream.
Simples :smile:

The good news is it works. The bad news is it isn’t exactly elegant. Probably faster than using blobs and array buffers though.

1 Like
#14

Thanks so much for looking so much into this. I’ll throw this in for a workaround for right now and update AMO, much appreciated!

#15

Warning: I tried to simplify this by just putting the binary string into an nsIStringInputStream. It worked! However, I’m reliably informed that it will fail in some situations because nsIStringInputStream cannot always safely handle strings containing null characters.

#16

Thanks for that note, I’ll just put it in as a temporary work around till I figure out that array buffer or blob thing. Maybe the team that deprecated mozFetchAsStream can help, an alternative has to exist no? Otherwise they wouldn’t deprecate no?

#17

Oh quick note, it seems toDataURL blocks the main thread :frowning: For larger portions it can lock up for over 100ms.

#18

Ssshhhh, someone might hear you and deprecate it. Although 100ms is hardly a serious delay in response to a user click, you can use FileReader to asynchronously get a data url from a Blob if you really want. I would have thought that was the least worrying part of this dog’s breakfast piece of code :wink:

1 Like
#19

Haha!!
Excellent idea on the async method thanks! :slight_smile:

#20

Shoot already found instances where it fails. Ok forget, temporary work around I’ll commit the ultimate sin and resort to nsIFile for now haha - http://stackoverflow.com/a/25020668/1828637 - will delete self on closeCi.nsIFileInputStream.DELETE_ON_CLOSE