Future of the compiler branch in `fluent.runtime`

Fluent
#1

So, as discussed before, I have a compiler branch of fluent.runtime, which was ready over a year ago, but has been held up on merging first of all due to PR 92 which has not made much progress in a while. In addition there are probably lots of other decisions that would need to be made, like handling of escaping which we never came to consensus on.

This has meant that for over a year there has been no straight-forward installation process for django-ftl, which is unfortunate if it wants to see any adoption.

Since then, lots more things have happened, like a new API in the main branch fluent.runtime. The new API is harder for me to adapt the compiler backend to, and it looks less and less likely that the compiler branch will ever be merged.

I’ve also become aware of lots of other ways that I could improve the compiler interface if I wasn’t constrained with compatibility to the resolver interface. For example, after reading the thread on message references and different kinds of fallback, I realised this is something I would want to do at compile time, not runtime, and before then I was already thinking that the interface that django-ftl really wants is just very different from either the old 0.1 FluentBundle API or the new 0.3.0 API. Compilers and interpreters just work differently sometimes.

So, I’d like to propose that we split this branch out into its own package. In fact that seems the only way for it to make any progress.

The next question would be, where should that package live? For me, I think it should live outside the python-fluent project (but could be linked as an alternative implementation). We could call it fluent_compiler on PyPI, and it would probably live under the django-ftl GitHub organisation. Although it would continue to be agnostic as to web framework, it would first of all be designed to support my needs in django-ftl just like fluent.runtime will first of all be designed to support Bedrock’s needs.

In the future, there is nothing to stop us moving it back under python-fluent, or indeed being packaged as fluent.compiler instead of fluent_compiler, but for now that just seems to be an obstacle.

How does that sound?

#2

@Pike @stas - not sure if you monitor all topics in this group so mentioning you here for your input. Thanks!

#3

Sorry for the lag, I had this open in a tab for a week, and I’m editing on this post for another.

There’s a few things to take apart here, all of which impact the work you’ve done.

APIs

In the most abstract setting, the questions around API designs are really our fault, and I see the impact that has in quite a few independent implementations. We’re building Fluent from the ground up. That’s good in the sense that it allows us to build at all. It also leaves a lot of empty space for folks that want to actually use Fluent in production. We haven’t done enough on the roles of Localization classes and bindings. This led most people to take the Bundle API as the thing they should expose to applications.

I hope that this will be fixed by actually specifying what the runtimes should do in the first place. Doesn’t help the existing implementations much, but I want to at least acknowledge that this is a problem, and a wide-spread one.

The thread that you pointed to also shows that I’m more on the side of changing the relationships between the components in the resolver.

Having a consistent Bundle API does have benefits, but it’s really only for folks writing Localization and bindings code.

Compiler branch

On the actual compiler branch. There are a few details, but the main holdup to this day is the call to exec. My perspective is that that requires an audit. There should be two people saying

I solemnly swear that there’s no arbitrary code execution.

I take you to be one of them. I haven’t been the other. Frankly, I don’t think I’m utterly qualified. Which is why my best effort boils down to unlimited paranoia for the task. I’d also not have an obvious peer or colleague I could have appointed to this task.

That’s why I started trying to address the performance question independently, and TBH, I think I found good answers. The next-gen-api branch doesn’t have all fixes, but there’s a two-liner we should add from the format-to-parts branch and then we should be good.

To the details, IIRC, I had a case where the ID mangling had conflicts between different messages. Others are about conformance, so let’s talk about that:

Conformance

There’s a third aspect in your post, I think, and that’s about conformance between different runtime implementations.

This is a hard problem, as right now, different bindings have different behavior at runtime. Some aspects are bug-for-bug compatible, like message/term references. Bidi isolation is a mixed bag, I heard. Others just plain conflict, like escaping aka DOM Overlays. Escaping and DOM Overlays overlap in situations like <undefined> as the target string.

I’d like the ability to change things here, and Stas probably in a different way too.

As much as we should keep in mind that things are implementable in general, I don’t know if we’ll ensure that all implementations take an equal hit. For example, the format-to-parts branch is great in cpython, but abysmal on pypy. Maybe that’s pypy’s problem. JS JITs solved that problem a while ago.

I run on the assumption that format-to-parts will be an integral aspect of the runtime specification at some point. So I don’t expect implementations to be able to avoid it. Of course there are different ways to implement it.

Summary

What does that mean for https://github.com/django-ftl/fluent-compiler? I don’t know, I don’t have good advice. Looking at that repo and pypi, it seems that you given up on getting some, too.

#4

Thanks for the reply Pike.

I totally understand about reticence over exec. I think the best way to address this is for fluent-compiler to live in its own repo/package for now. Tucked away as a branch I don’t think it isn’t going to get any eyeballs that could find bugs or even be interested in it.

As you saw, I decided to go ahead and make a separate release. This doesn’t mean it’s going to be separate from the main projectfluent repos forever, or that it is going to diverge in terms of conformance - I will try to avoid that. One of the advantages of the compiler approach is that you can relatively easily toggle different behaviours with options, without run-time complexity or performance hits. For example, I will probably add an option to silence some compile-time errors at run-time, which might diverge from fluent.runtime - but you get the advantage of the same default behaviour, while also having the ability for users to tune things where needed.