Why is this SpiderMonkey Native Messsaging host not sending stdout to the client?

It doesn’t look like the SpiderMonkey jsshell provides a means to read stdin or write to stdout as a TypedArray outside of a WASI build (e.g. for spidermonkey-wasm-rs; sm-wasi-demo), see https://hg.mozilla.org/mozilla-central/file/tip/js/src/shell/js.cpp#l6251.

The outgoing message is written to getMessage.txt and sendMesssge.txt with os.file.writeTypedArrayToFile() once on the first postMessage() then the while loop does not appear to continue as subsequent postMessage() calls do not modify the files. Nothing is written to err.txt. I’m getting “Invalid byte sequence” error in the plain text file when observing writes of Uint32Array though I am able to isolate the JSON message.

When I include the environment variable JS_STDOUT=stdout.txt

this is written to this file

6,0,0,0 34,116,101,115,116,34

which is technically the correct output

let message = new TextDecoder().decode(new Uint8Array([34,116,101,115,116,34])); // '"test"'
let length = Uint32Array.of(message.length); // 6

Anybody have experience writing data to stdout in a SpiderMonkey shell script?

#!/usr/bin/env -S JS_STDERR=err.txt ./js --enable-top-level-await
// SpiderMonkey Native Messaging host (W.I.P.)
// guest271314 7-7-2023

function getMessage() {
  const stdin = readline();
  const data = new Uint8Array([...stdin].map((char) => char.charCodeAt(0)));
  const length = data.subarray(0, 4);
  os.file.writeTypedArrayToFile('length.txt', length);
  os.file.writeTypedArrayToFile('getMessage.txt', data.subarray(4));
  return data.subarray(4);
}

function sendMessage(message) {
  const header = Uint32Array.from({
      length: 4,
    },
    (_, index) => (message.length >> (index * 8)) & 0xff
  );

  os.file.writeTypedArrayToFile('sendMessage.txt', header);
  print(header);
  print(message);
}

function main() {
  while (true) {
    const message = getMessage();
    sendMessage(message);
    // break;
  }
}

try {
  main();
} catch (e) {
  quit();
}

It doesn’t look like the SpiderMonkey jsshell provides a means to read stdin or write to stdout as a TypedArray outside of a WASI build

It’s the WASI build where this isn’t available, see https://github.com/mozilla-spidermonkey/sm-wasi-demo/issues/3#issuecomment-1628569147

Are you testing with a WASI build or a non-WASI one? I tried your code with a non-WASI build without setting JS_STDOUT or JS_STDERR and it seems to update the length.txt/getMessage.txt/setMessage.txt files if I enter random data.

In sendMessage, should that be Uint8Array.from instead of Uint32Array.from if you want a 4-byte header? Now it’s writing 16 bytes.

Non-WASI environment. I’ve tried hundreds of times so far to get this working without success.

Each Native Messaging host I/O is different, even all the JavaScript runtimes behave differently https://github.com/guest271314/NativeMessagingHosts. print() or putstr() are not working as expected. Nor is readline() or readlinBuf(). That’s why I asked this question.

I’m not clear on what you’re trying to do.

If I run python -c 'print("\4\0\0\0blah\n\3\0\0\0xyz")' | ./msg.js where msg.js is your script, it writes all 3 .txt files twice. The contents of sendMessage.txt seem wrong to me, since as @jandem said that should be Uint8Array or more simply, const header = new Uint32Array([message.length]) (on a little-endian system, anyway—but you’re forcing little-endian instead of network-endian, so I’m not sure what you want in terms of endianness?)

But given the use of readline(), you’re expecting a length-prefixed, newline-terminated sequence of messages? That seems a bit odd.

Anyway, I think the fundamental problem that you’re running into is that the JS shell just doesn’t have a way to write binary data to stdout. print() won’t print NUL chars. writeTypedArrayToFile insists on opening a file. I’m not sure about putstr, but it really wants to encode to UTF8.

It sorta works on linux if you use ‘/proc/self/fd/1’ as the filename. As in, this command properly propagates the message through multiple stages python -c 'print("\4\0\0\0blah\n\3\0\0\0xyz")' | ./echo.js | ./echo.js | ./echo.js with echo.js containing:

#!/usr/bin/env -S js --enable-top-level-await
// SpiderMonkey Native Messaging host (W.I.P.)
// guest271314 7-7-2023

function getMessage() {
  const stdin = readline();
  if (stdin === null) {
    return null;
  }
  const data = new Uint8Array([...stdin].map((char) => char.charCodeAt(0)));
  const length = data.subarray(0, 4);
  return data.subarray(4);
}

function sendMessage(u8a_message) {
  const encoded = new Uint8Array(4 + u8a_message.length + 1);

  const len32 = new Uint32Array([u8a_message.length]);
  encoded.set(new Uint8Array(len32.buffer), 0);
  encoded.set(u8a_message, 4);
  encoded[4 + u8a_message.length] = '\n'.charCodeAt(0);

  os.file.writeTypedArrayToFile('/proc/self/fd/1', encoded);
}

function main() {
  while (true) {
    const message = getMessage();
    if (message === null) {
      break;
    }
    sendMessage(message);
  }
}

main();

Though oddly, if I redirect the output to out.txt, it only records the last message. Maybe it would do better if writeTypedArrayToFile could open the file in append-mode? But it doesn’t.

Is there an actual use for this? The JS shell is theoretically only for testing.

1 Like

I have been using jsshell from Mozilla downloads. I have also tested with spidermonkey (jsshell) from jsvu.

Every JavaScript runtime winds up reading from stdin and writing to stdout differently, even when the same underlying implementation of JavaScript is derived from the same source, e.g., V8 (Node.js (C++), Deno (Rust)); QuickJS (txiki.js ©), Bun (JavaScriptCore (Zig)). (I tried d8 standalone and had similar results to jsshell standalone; still have not achieved messaging.)

Implement a Native Messaging host using jsshell standalone. This is the protocol

Native messaging protocol (Chrome Developers)

Chrome starts each native messaging host in a separate process and communicates with it using standard input ( stdin ) and standard output ( stdout ). The same format is used to send messages in both directions; each message is serialized using JSON, UTF-8 encoded and is preceded with 32-bit message length in native byte order. The maximum size of a single message from the native messaging host is 1 MB, mainly to protect Chrome from misbehaving native applications. The maximum size of the message sent to the native messaging host is 4 GB.

I linked to hosts written in different languages, including a Python host https://github.com/guest271314/NativeMessagingHosts/blob/main/nm_python.py in OP. The Python host example on Mozilla had a bug, Python adds spaces as formatting by default, which counted in 1 MB sent back to client https://github.com/mdn/webextensions-examples/pull/510. The last time I checked Chrome extension samples on GitHub still had not fixed the bug. Further re Python, on Linux there might not be a python executable, only python3 since Python 2 is deprecated, however still used and available on some systems.

Anyway, I think the fundamental problem that you’re running into is that the JS shell just doesn’t have a way to write binary data to stdout. print() won’t print NUL chars. writeTypedArrayToFile insists on opening a file. I’m not sure about putstr , but it really wants to encode to UTF8.

Yes, I see that. I encountered similar issues the last time I tried implementing a host using d8 standalone. I/O options are lacking; TTY is expected, not necessarily to use the base JavaScript implementation where I/O options are useful.

I tried your code. Same result. The message is not getting through to the client.

Is there an actual use for this? The JS shell is theoretically only for testing.

Yes. To field test JavaScript standalone runtimes, specifically as to stdin/stdout/stderr when applicable, as a Native Messaging host, to get empirical result as to which implementation is fastest, requires least amount of resources and which is fastest to effectuate the same result; to field test multiple programming languages as a Native Messaging host, to determine which requires least amount of resources (KB/MB, RSS, VSZ, file size, compilation requirements, etc.), and/or time it takes to figure out how to get it done in that programming language.

It seems like you’d need more functionality to make that work. readline() is going to wait for a newline. You need something that can read a specified number of bytes, or a nonblocking read that reads up to n bytes and select/poll/epoll/io_uring/whatever. And that I/O stuff is going to determine your base messaging performance, the engine or language isn’t going to matter.

It wouldn’t be that hard to add an N-byte blocking read to a SpiderMonkey embedding example, but that’s going to score pretty bad on the “time it takes to figure out how to get it done in that programming language” metric. (Well, really it’s the embedding that determines the difficulty here.)

One alternative: you could do this today if you --enable-ctypes and use libc’s functions for doing I/O. It would be kind of awful, something like

const libc = ctypes.open("/lib64/libc.so.6");
const read = libc.declare("read", ctypes.default_abi, ctypes.ssize_t, ctypes.int, ctypes.void_t.ptr, ctypes.size_t);
const bufferT = ctypes.char.array(4096);
const buffer = new bufferT;
const len_w = new Uint8Array(4);
const len_r = new Uint32Array(len_w.buffer);
while (true) {
  let numRead = read(0, buffer.address(), 4);
  if (numRead != 4) break;
  len_w.set([buffer[0], buffer[1], buffer[2], buffer[3]]);
  const length = len_r[0];
  // etc.
}

(mostly untested) but I’d warn that (1) this is, uh, a little bit out there, and (2) we may remove ctypes before too long. And maybe (3) you get all the fun of seg faults and the other joys of writing in C, from JavaScript! But at least there are no TTYs involved?

1 Like

Your answer is precisely why I filed Common I/O (stdin/stdout/stderr) module specification.

I’m not seeing --enable-ctypes as an option in Version: JavaScript-C116.0 from jsvu or Version: JavaScript-C117.0a1 from Mozilla downloads.

Sorry, it’s an option to configure. It’s already enabled in the Firefox builds (and mostly isn’t in the shell-only builds, though SM(cgc) has it for whatever reason.) See https://treeherder.mozilla.org/jobs?repo=mozilla-central&selectedTaskRun=XPlXJmfETjKleczwueLR4w.0 for example, the target.jsshell.zip artifact.

The issue you filed really isn’t relevant to the jsshell. It is decidedly not a “Web-interoperable runtime”. I don’t think the jsshell makes any sense to use for what you’re doing, but I’ll confess I’m helping it along just because I’m curious how far you can push it.

Honestly https://github.com/Redfire75369/spiderfire may be a better starting place to experiment with a SpiderMonkey-based JS runtime. I don’t know much about it, other than that the docs look to be in an early state, and the author is knowledgeable, capable, and shows up frequently on the Matrix channel.

1 Like

Do you mean this https://firefox-ci-tc.services.mozilla.com/tasks/XPlXJmfETjKleczwueLR4w ?

The issue you filed really isn’t relevant to the jsshell. It is decidedly not a “Web-interoperable runtime”.

I would disagree with that. Mozilla itself published https://github.com/mozilla-spidermonkey/sm-wasi-demo.

I don’t think the jsshell makes any sense to use for what you’re doing, but I’ll confess I’m helping it along just because I’m curious how far you can push it.

Well, the only way I know to acquire empirical facts is to perform tests without predisposed biases about what is and what is not a JavaScript runtime, “Web interoperable runtime”, or not. Folks in the field might claim Node.js is the best ever, the king, the standard; I mean, why even bother with SpiderMonkey, at all when V8 exists? And so forth. What I’ve learned so far is QuickJS qjs after strip is less than 1 MB and can achieve the requirement is less code and faster than the other JavaScript runtimes I’ve tested native_messaging_javascript_runtime_tests.md where node is over 90 MB and deno is over 100 MB. Of course, following that path, why use JavaScript, or Python, or WASI with wasmtime at all when we can use C? And so forth.

Thanks for your help.

Honestly https://github.com/Redfire75369/spiderfire may be a better starting place to experiment with a SpiderMonkey-based JS runtime. I don’t know much about it, other than that the docs look to be in an early state, and the author is knowledgeable, capable, and shows up frequently on the Matrix channel.

I found that repository before I found https://github.com/mozilla-spidermonkey/sm-wasi-demo. I don’t think I saw I/O implemented there, either.

Rust toolchain is over 1 GB before any crates are installed. I tried to build this https://github.com/denoland/roll-your-own-javascript-runtime and ran out of disk space on tokio crate on a temporary file system Is it possible to install Rust in a tmpfs with less than 1GB of RAM? “I think the answer to this is no.”. So I have to decide: Rust or LLVM/clang/c++ on a temporaray file system: Think embeddable, portable minimal system requirements.

Thanks.

The issue you filed really isn’t relevant to the jsshell. It is decidedly not a “Web-interoperable runtime”.

If that is the case what is the point of Online SpiderMonkey WASI shell?

Honestly https://github.com/Redfire75369/spiderfire may be a better starting place to experiment with a SpiderMonkey-based JS runtime. I don’t know much about it, other than that the docs look to be in an early state, and the author is knowledgeable, capable, and shows up frequently on the Matrix channel.

I asked Can js-compute-runtime be run completely locally? (SpiderMonkey JavaScript engine), though did not/do not see a clear instruction set for how to do that (without running some other software) in the linked documentation.

I’m still not seeing --enable-ctypes option.

@sfink

but I’ll confess I’m helping it along just because I’m curious how far you can push it.

FWIW I got the jsshell downloaded by jsvu to work for one (1) message at a time by sending a subsequent "\r\n\r\n" message to the host.

#!/usr/bin/env -S JS_STDERR=err.txt /home/user/.jsvu/engines/spidermonkey/spidermonkey 
// SpiderMonkey Native Messaging host (W.I.P.)
// guest271314 7-7-2023

function encodeMessage(str) {
  return new Uint8Array([...str].map((s) => s.codePointAt()));
}

function getMessage() {
  const stdin = readline();
  const data = encodeMessage(stdin);// new Uint8Array([...stdin].map((s) => s.codePointAt()));
  const view = new DataView(data.buffer);
  const length = new Uint32Array([view.getUint32(0, true)]);
  const message = data.subarray(4);
  os.file.writeTypedArrayToFile("length.txt", length);
  os.file.writeTypedArrayToFile("input.txt", message);
  os.file.writeTypedArrayToFile("/proc/self/fd/1", length);
  os.file.writeTypedArrayToFile("/proc/self/fd/1", message);
}

function main() {
  while (true) {
    const message = getMessage();
  }
}

try {
  main();
} catch (e) {
  os.file.writeTypedArrayToFile("caught.txt", encodeMessage(e.message));
  quit();
}
globalThis.name = chrome.runtime.getManifest().short_name;

async function sendNativeMessage(message) {
  return new Promise((resolve, reject) => {
    globalThis.port = chrome.runtime.connectNative(globalThis.name);
    port.onMessage.addListener((message) => {
      resolve(message);
      port.disconnect();
    });
    port.onDisconnect.addListener(() => {
      reject(chrome.runtime.lastError);
    });
    port.postMessage(message);
    port.postMessage("\r\n\r\n");
  });
}

sendNativeMessage(new Array(209715)).then(console.log).catch(console.error);

Some more work and this is now persistent, we can send multiple messages per connection without disconnecting and reconnecting https://github.com/guest271314/native-messaging-spidermonkey-shell/.

nm_spidermonkey.js

#!/usr/bin/env -S JS_STDERR=err.txt /home/user/.jsvu/engines/spidermonkey/spidermonkey
// /home/user/bin/jsshell-linux-x86_64/js
// SpiderMonkey Shell Native Messaging host
// guest271314 7-7-2023, 6-16-2024

function encodeMessage(str) {
  return new Uint8Array([...str].map((s) => s.codePointAt()));
}

function getMessage() {
  // Call readline() N times to catch `\r\n\r\n"` from 2d port.postMessage()
  let stdin;
  while (true) {
    stdin = readline();
    if (stdin !== null) {
      break;
    }
  }
  // TODO: Handle the *string* input "{}"
  let data = `${stdin}`.replace(/[\r\n]+|\\x([0-9A-Fa-f]{2,4})/gu, "")
    .replace(/[^A-Za-z0-9\s\[,\]\{\}:_"]+/igu, "")
    .replace(/^"rnrn/gu, "")
    .replace(/^[#\r\n\}_]+(?=\[)/gu, "")
    .replace(/^"+(?=["\{]+)|^"(?!"$)/gu, "") 
    .replace(/^\[(?=\[(?!.*\]{2}$))/gu, "")
    .replace(/^\{(?!\}|.+\}$)/gu, "")
    .replace(/^[0-9A-Z]+(?=[\[\{"])/igu, "") 
    .replace(/^[\]\}](?=\[)/i, "")
    .trimStart().trim();
  // https://stackoverflow.com/a/52434176
  // let previous = redirect("length.txt");
  // putstr(data.length);
  // redirect(previous); // restore the redirection to stdout
  // os.file.writeTypedArrayToFile("input.txt", encodeMessage(data));
  return encodeMessage(data);
}

function sendMessage(message) {
  os.file.writeTypedArrayToFile(
    "/proc/self/fd/1",
    new Uint32Array([message.length]),
  );
  os.file.writeTypedArrayToFile("/proc/self/fd/1", message);
}

function main() {
  // Send help() to client
  // const previous = redirect("help.txt");
  // putstr(help());
  // redirect(previous); // restore the redirection to stdout
  // const h = read("help.txt", "binary");
  // sendMessage(encodeMessage(JSON.stringify([...h])));
  while (true) {
    const message = getMessage();
    sendMessage(message);
  }
}

try {
  main();
} catch (e) {
  os.file.writeTypedArrayToFile(
    "caught.txt",
    encodeMessage(JSON.stringify(e.message)),
  );
  quit();
}

background.js

globalThis.name = chrome.runtime.getManifest().short_name;

globalThis.port = chrome.runtime.connectNative(globalThis.name);

port.onMessage.addListener((message) => {
  console.log(message);
});

port.onDisconnect.addListener(() => {
  console.log(chrome.runtime.lastError);
});

globalThis.postNativeMessage = (message) => {
  port.postMessage(message);
  // SpiderMonkey shell won't close STDIN without this trailing newline
  port.postMessage("\r\n\r\n");
};

postNativeMessage(Array(209715));