The outgoing message is written to getMessage.txt and sendMesssge.txt with os.file.writeTypedArrayToFile() once on the first postMessage() then the while loop does not appear to continue as subsequent postMessage() calls do not modify the files. Nothing is written to err.txt. I’m getting “Invalid byte sequence” error in the plain text file when observing writes of Uint32Array though I am able to isolate the JSON message.
When I include the environment variable JS_STDOUT=stdout.txt
this is written to this file
6,0,0,0 34,116,101,115,116,34
which is technically the correct output
let message = new TextDecoder().decode(new Uint8Array([34,116,101,115,116,34])); // '"test"'
let length = Uint32Array.of(message.length); // 6
Anybody have experience writing data to stdout in a SpiderMonkey shell script?
Are you testing with a WASI build or a non-WASI one? I tried your code with a non-WASI build without setting JS_STDOUT or JS_STDERR and it seems to update the length.txt/getMessage.txt/setMessage.txt files if I enter random data.
In sendMessage, should that be Uint8Array.from instead of Uint32Array.from if you want a 4-byte header? Now it’s writing 16 bytes.
Non-WASI environment. I’ve tried hundreds of times so far to get this working without success.
Each Native Messaging host I/O is different, even all the JavaScript runtimes behave differently GitHub - guest271314/NativeMessagingHosts: Native Messaging hosts. print() or putstr() are not working as expected. Nor is readline() or readlinBuf(). That’s why I asked this question.
If I run python -c 'print("\4\0\0\0blah\n\3\0\0\0xyz")' | ./msg.js where msg.js is your script, it writes all 3 .txt files twice. The contents of sendMessage.txt seem wrong to me, since as @jandem said that should be Uint8Array or more simply, const header = new Uint32Array([message.length]) (on a little-endian system, anyway—but you’re forcing little-endian instead of network-endian, so I’m not sure what you want in terms of endianness?)
But given the use of readline(), you’re expecting a length-prefixed, newline-terminated sequence of messages? That seems a bit odd.
Anyway, I think the fundamental problem that you’re running into is that the JS shell just doesn’t have a way to write binary data to stdout. print() won’t print NUL chars. writeTypedArrayToFile insists on opening a file. I’m not sure about putstr, but it really wants to encode to UTF8.
It sorta works on linux if you use ‘/proc/self/fd/1’ as the filename. As in, this command properly propagates the message through multiple stages python -c 'print("\4\0\0\0blah\n\3\0\0\0xyz")' | ./echo.js | ./echo.js | ./echo.js with echo.js containing:
#!/usr/bin/env -S js --enable-top-level-await
// SpiderMonkey Native Messaging host (W.I.P.)
// guest271314 7-7-2023
function getMessage() {
const stdin = readline();
if (stdin === null) {
return null;
}
const data = new Uint8Array([...stdin].map((char) => char.charCodeAt(0)));
const length = data.subarray(0, 4);
return data.subarray(4);
}
function sendMessage(u8a_message) {
const encoded = new Uint8Array(4 + u8a_message.length + 1);
const len32 = new Uint32Array([u8a_message.length]);
encoded.set(new Uint8Array(len32.buffer), 0);
encoded.set(u8a_message, 4);
encoded[4 + u8a_message.length] = '\n'.charCodeAt(0);
os.file.writeTypedArrayToFile('/proc/self/fd/1', encoded);
}
function main() {
while (true) {
const message = getMessage();
if (message === null) {
break;
}
sendMessage(message);
}
}
main();
Though oddly, if I redirect the output to out.txt, it only records the last message. Maybe it would do better if writeTypedArrayToFile could open the file in append-mode? But it doesn’t.
Is there an actual use for this? The JS shell is theoretically only for testing.
Chrome starts each native messaging host in a separate process and communicates with it using standard input ( stdin ) and standard output ( stdout ). The same format is used to send messages in both directions; each message is serialized using JSON, UTF-8 encoded and is preceded with 32-bit message length in native byte order. The maximum size of a single message from the native messaging host is 1 MB, mainly to protect Chrome from misbehaving native applications. The maximum size of the message sent to the native messaging host is 4 GB.
Anyway, I think the fundamental problem that you’re running into is that the JS shell just doesn’t have a way to write binary data to stdout. print() won’t print NUL chars. writeTypedArrayToFile insists on opening a file. I’m not sure about putstr , but it really wants to encode to UTF8.
Yes, I see that. I encountered similar issues the last time I tried implementing a host using d8 standalone. I/O options are lacking; TTY is expected, not necessarily to use the base JavaScript implementation where I/O options are useful.
I tried your code. Same result. The message is not getting through to the client.
Is there an actual use for this? The JS shell is theoretically only for testing.
Yes. To field test JavaScript standalone runtimes, specifically as to stdin/stdout/stderr when applicable, as a Native Messaging host, to get empirical result as to which implementation is fastest, requires least amount of resources and which is fastest to effectuate the same result; to field test multiple programming languages as a Native Messaging host, to determine which requires least amount of resources (KB/MB, RSS, VSZ, file size, compilation requirements, etc.), and/or time it takes to figure out how to get it done in that programming language.
It seems like you’d need more functionality to make that work. readline() is going to wait for a newline. You need something that can read a specified number of bytes, or a nonblocking read that reads up to n bytes and select/poll/epoll/io_uring/whatever. And that I/O stuff is going to determine your base messaging performance, the engine or language isn’t going to matter.
It wouldn’t be that hard to add an N-byte blocking read to a SpiderMonkey embedding example, but that’s going to score pretty bad on the “time it takes to figure out how to get it done in that programming language” metric. (Well, really it’s the embedding that determines the difficulty here.)
One alternative: you could do this today if you --enable-ctypes and use libc’s functions for doing I/O. It would be kind of awful, something like
const libc = ctypes.open("/lib64/libc.so.6");
const read = libc.declare("read", ctypes.default_abi, ctypes.ssize_t, ctypes.int, ctypes.void_t.ptr, ctypes.size_t);
const bufferT = ctypes.char.array(4096);
const buffer = new bufferT;
const len_w = new Uint8Array(4);
const len_r = new Uint32Array(len_w.buffer);
while (true) {
let numRead = read(0, buffer.address(), 4);
if (numRead != 4) break;
len_w.set([buffer[0], buffer[1], buffer[2], buffer[3]]);
const length = len_r[0];
// etc.
}
(mostly untested) but I’d warn that (1) this is, uh, a little bit out there, and (2) we may remove ctypes before too long. And maybe (3) you get all the fun of seg faults and the other joys of writing in C, from JavaScript! But at least there are no TTYs involved?
The issue you filed really isn’t relevant to the jsshell. It is decidedly not a “Web-interoperable runtime”. I don’t think the jsshell makes any sense to use for what you’re doing, but I’ll confess I’m helping it along just because I’m curious how far you can push it.
Honestly https://github.com/Redfire75369/spiderfire may be a better starting place to experiment with a SpiderMonkey-based JS runtime. I don’t know much about it, other than that the docs look to be in an early state, and the author is knowledgeable, capable, and shows up frequently on the Matrix channel.
I don’t think the jsshell makes any sense to use for what you’re doing, but I’ll confess I’m helping it along just because I’m curious how far you can push it.
Well, the only way I know to acquire empirical facts is to perform tests without predisposed biases about what is and what is not a JavaScript runtime, “Web interoperable runtime”, or not. Folks in the field might claim Node.js is the best ever, the king, the standard; I mean, why even bother with SpiderMonkey, at all when V8 exists? And so forth. What I’ve learned so far is QuickJS qjs after strip is less than 1 MB and can achieve the requirement is less code and faster than the other JavaScript runtimes I’ve tested native_messaging_javascript_runtime_tests.md where node is over 90 MB and deno is over 100 MB. Of course, following that path, why use JavaScript, or Python, or WASI with wasmtime at all when we can use C? And so forth.
I asked Can js-compute-runtime be run completely locally? (SpiderMonkey JavaScript engine), though did not/do not see a clear instruction set for how to do that (without running some other software) in the linked documentation.