Transpiling Fluent resources into JavaScript modules

[eemeli]

Heya. I started playing around a bit with Fluent, and will probably start using it as a base for some localization workflow ideas that I’ve been working on, as it solves some of the issues (like file format) that I’ve been a bit struggling with.

To get started, I’m first working on fluent-compiler, which now mostly works. It’s a tool that allows for message compilation to happen during an application’s build, rather than in the runtime. That way the compiler does not need to be loaded on the client; just a small runtime. The API of the resulting ES6 module tries to match that of FluentBundle.

I’m nearly ready to publish this on npm with a 0.1 release, but wanted to give you a heads-up in particular if the package name might be a problem, or if you wanted me to more clearly note how much its structure is based on the fluent-syntax serializer (I also nicked a few bits from messageformat). I’m releasing it as Apache-2.0 in order to match the rest of the Fluent tools.

[Pike]

Hi Eemeli,

thanks for letting us know. Also happy to hear that we’re addressing problems you struggled with. Let me ramble a bit, 'cause it’s sunny outside, and I attribute all my brain-stuff to hay fever.

I had a quick look at the code, and I like the idea to focus on bundle as the API you mock. Can you put multiple resource files into one bundle? I also think you handled the locale-dependent runtime right, but diving into that, I think I found a bug in $select ;-). Also, do you handle foo-bar as a different message from foo_bar and foo---bar?

Regarding naming, I mostly defer this to @stas, but I’d like to alert you that we’ll rename our packages into a @fluent namespace. That said, I can’t think of something Fluent-y that I’d rather call compiler than a code module like the one you’re working on.

What you’re doing is technically similar to the things we’re doing on the python side. There’s the compiler branch by Luke, and also the pre-evaluating resolver on master. It might be possible to do something similar, and then toSource that? I wonder if one could escape out of your serializer logic with funny terms and term references and possibly string literals. Maybe relying on something like escodegen to serialize source would be easier to make secure?

Granted, right now it’s really hard to write resolvers, 'cause there’s no specification for them, so you’re kinda bound to reverse engineer the js impl. We also don’t have extensive nor platform independent tests, which would catch things like number literals in select expression variant keys (that’s the bug I found, I think). I wouldn’t be surprised if we got a couple of changes in the resolver designs still. Like, I have a thought in the back of my mind to resolve to generators or arrays instead of strings, but that’s for another day.

Some further thoughts: I’m pretty sure that your work is a good answer to some use-cases. Similar to the work that Luke’s doing for elm-fluent, being close to the Metal sometimes pays off. Getting these secure and reliant and being “Fluent-y” is hard, because that stuff is mostly encoded in the heads of two-three people.

Would you be up for helping us narrow down on the needs of resolver implementers on tests and specification? Like, all the questions I asked should really be test cases, I guess.

One nit on the README, it’d be “Fluent-y” to have a module on top of fluent-compiler that does language fallback. We usually call these Localization, for the lack of a better word to describe “the thing that iterates over bundles to find the one that has the strings for the IDs you gave, in order of negotiated locales, possibly async”. Is that part of what you’re having in mind? Might be good to hint at in the readme.

Axel

[eemeli]

Hi Axel!

Hadn’t quite expected a reply so soon esp. during Easter. Replies intermingled below.

I had a quick look at the code, and I like the idea to focus on bundle as the API you mock. Can you put multiple resource files into one bundle?

Yeah, the bundle seemed like the right API to model. I’ve not yet done anything about supporting multiple locales; that will indeed require a bit of thought.

Also, do you handle foo-bar as a different message from foo_bar and foo---bar ?

Not completely correctly atm, as foo_bar and foo-bar would cause a runtime error. Should fix that.

Regarding naming, I mostly defer this to @stas, but I’d like to alert you that we’ll rename our packages into a @fluent namespace. That said, I can’t think of something Fluent-y that I’d rather call compiler than a code module like the one you’re working on.

Getting @fluent as a namespace would indeed make it easier to differentiate core & third-party components. Just to be clear, I’d be happy to have fluent-compiler considered a part of the core, and moved to the fluent.js monorepo.

What you’re doing is technically similar to the things we’re doing on the python side. There’s the compiler branch by Luke, and also the pre-evaluating resolver on master . It might be possible to do something similar, and then toSource that? I wonder if one could escape out of your serializer logic with funny terms and term references and possibly string literals. Maybe relying on something like escodegen to serialize source would be easier to make secure?

The alternative I thought about was to write this as a Babel parser, effectively outputting a Babel AST rather than a string. I decided against that for two reasons: First, the fluent-syntax serializer looked pretty good, and modifying it for ES6 rather than FTL output was literally the work of one day. Second, I explicitly didn’t want to enable (relatively) easy runtime use of the output. That’s what the fluent package is for, and it allows for not needing to consider runtime efficiency as much.

The structure of how the JS source is generated is pretty similar to what I did with messageformat, and it should be rather robust against “funny terms and term references”. For string literals I am admittedly relying on the parser, but that too looked pretty good. The function & property name sanitiser I should publish separately, as I’m now using it in two different packages.

Granted, right now it’s really hard to write resolvers, 'cause there’s no specification for them, so you’re kinda bound to reverse engineer the js impl. We also don’t have extensive nor platform independent tests, which would catch things like number literals in select expression variant keys (that’s the bug I found, I think).

Ach, you’re right, that’s a bug. Should’ve remembered about exact matches.

I wouldn’t be surprised if we got a couple of changes in the resolver designs still. Like, I have a thought in the back of my mind to resolve to generators or arrays instead of strings, but that’s for another day.

I started with strings, but ended up with arrays in order to better handle number literals, so that their stringification can happen as late as possible while keeping them as numbers when used as call arguments. Alternative solutions would’ve also made the compiler more complicated.

Would you be up for helping us narrow down on the needs of resolver implementers on tests and specification? Like, all the questions I asked should really be test cases, I guess.

I think so? Would be good to open up a bit what a “resolver implementer” looks like, just to be clear. For tests, I’ll certainly expand on the tests I’ve got (which I’ve mostly nicked from you), but did you mean adding some to the fluent.js packages as well?

One nit on the README, it’d be “Fluent-y” to have a module on top of fluent-compiler that does language fallback. We usually call these Localization , for the lack of a better word to describe “the thing that iterates over bundles to find the one that has the strings for the IDs you gave, in order of negotiated locales, possibly async”. Is that part of what you’re having in mind? Might be good to hint at in the readme.

Certainly. The documentation will need quite a bit of work; atm it’s really just a sketch for people like you so you can understand what I’m working on. As mentioned, I will indeed need to add support for multiple locales and fallback between them.

[Pike]

Further thoughts, I actually think you should compile to something akin Resource instead of Bundle.

I’m thinking of use-cases where you’d want to do runtime customizations of bundles. Say, you have a shopping platform, and you want to load your customers branding into the platform at runtime.

I’d compile into a Map, too, to get around the renames of IDs and refs.

[eemeli]

Further thoughts, I actually think you should compile to something akin Resource instead of Bundle .

If the output would be closer to Resource rather than Bundle, it would effectively necessitate harmonising and publishing the internal structure of a Message, which the JS API currently doesn’t do.

This relates in particular to handling multi-locale fallbacks, for which the plan I’m generating is to make the output sufficiently similar to that of fluent (esp. once PR #322 lands) that it becomes possible to use e.g. fluent-dom and fluent-react with them. And their APIs consume bundles.

I’m thinking of use-cases where you’d want to do runtime customizations of bundles. Say, you have a shopping platform, and you want to load your customers branding into the platform at runtime.

Good point. It might indeed make sense to implement addMessages and addResource, with the proviso that the added messages would only be able to refer to other messages and terms that were available in their original contexts.

I’d compile into a Map , too, to get around the renames of IDs and refs.

That would work. Atm I’m using const foo = $ => ['Foo', bar($)] as the internal representation of a message foo = Foo{bar}. With a Map that’d become something like map.set('foo', $ => ['Foo', map.get('bar')($)]). I’ve got the variable name mapping fixed now so probably won’t switch, but if I hit any snag that makes sense as an alternative.

[stas (Staś Małolepszy)]

Hi @eemeli! Nice work. A compiler is something that I wished we had in the fluent.js ecosystem. Thanks for creating it, it looks great!

I’m nearly ready to publish this on npm with a 0.1 release, but wanted to give you a heads-up in particular if the package name might be a problem, or if you wanted me to more clearly note how much its structure is based on the fluent-syntax serializer (I also nicked a few bits from messageformat ).

Feel free to go ahead with fluent-compiler, and thanks for asking I need to set things up before we start publishing in the @fluent scope, so for now fluent-* is the way to go.

I’d be happy to have fluent-compiler considered a part of the core, and moved to the fluent.js monorepo.

That’s great to hear, thank you! I’m going to be away for two weeks in May, and I’ll be happy to talk about this in detail when I’m back. I’d like to do a thorough review before the compiler lands in the monorepo. And I’d love for you to continue being the owner of the package when we do this, please

[eemeli]

I’ve now refactored the compiler internals a bit, so that compiled bundles now have an internal structure closer to that of fluent, also exposing a resource Map of terms and messages (as suggested by @Pike). This allows for also implementing the bundle’s addResource method.

Regarding the API, I’ve implemented the changes proposed in projectfluent/fluent.js#208, i.e. dropping getMessage, using string keys for format, and adding compound. If #365 gets merged instead, I’ll adjust accordingly. In any case it doesn’t make sense for me to target the 0.12 API, as the packages relying on it use internal methods to access messages.

[arggh]

I just stumbled onto Fluent, so excuse me if I missed something.

It seems that the Fluent’s preferred API for bundles has changed since fluent-compiler was last updated and bundle.format is no longer the compatible API as per this PR?

Are there any plans to create or include a compiler to the @fluent monorepo? Or does @eemeli have any plans on maintaining fluent-compiler?

After trying out Fluent today, my first complaint was having to ship a somewhat large runtime & parser to the client, so a compiler would be a great addition.