Welcome to the December/January edition of the Engineering Effectiveness Newsletter! The Engineering Effectiveness org makes it easy to develop, test and release Mozilla software at scale. See below for some highlights, then read on for more detailed info!
Highlights
- Developers can now run mach build and mach run on ARM64 Linux with fewer setup steps. Tool-specific caching (for uv, pip, npm, and cargo) was enabled in Firefox-CI to improve efficiency.
- The crash reporter client now supports multiple locales and provides better error context for bug reporting. Additionally, the USER_RESTRICTED sandbox for Firefox content processes was shipped, enhancing security by limiting filesystem access.
- New language models (Simplified Chinese, Japanese, Korean, Russian) will ship in Firefox 135. Performance improvements were made, including support for multiple translation models and optimizations to reduce made-up words.
Contributors
- Bastien Orivel, a long-time contributor on Taskcluster, joined the Release Engineering team as a full time employee on December 1st. His first contributions (see below) already helped both Release Engineering and Taskcluster. Welcome Bastien!
- A few members of the organization went to FOSDEM last weekend and presented some of our work. We will share the videos / slides once they are live!
Detailed Project Updates
Bugzilla 
and Bugbug
- We have created a dedicated page, AI for Development, where we list experimental, research and production projects using AI/ML to support development processes.
- If you have additional ideas you’d like to experiment with, or if you’d like to try some of the experimental tools, feel free to reach out to us.
- Suhaib Mujahid improved Review-Helper tool to not suggest review comments that are similar to previously rejected comments.
- Suhaib Mujahid and John Pangas improved the QA test case generation prototype to focus on filling the gaps in the manually created QA test plans. Thanks to Tania Maity, Tracy Walker, and Ben Chatterton for their collaboration.
Build System and Mach Environment
- Developers can now do mach build and mach run on arm64 Linux without extra manual work installing compilers and whatnot. Other commands may or may not work.
- The same is almost true on arm64 Windows (albeit requiring a manual install of a native arm64 python), pending wheels for the zstandard python package.
CI and Treeherder 
- Andrew Halberstadt created an initial dashboard to track high level costs in the Firefox-CI instance.
- Andrew Halberstadt implemented a way to enable tool specific caches (so far uv, pip, npm or cargo) and enabled them for most applicable tasks in Gecko.
- A new release of sccache 0.9.1 has been published.
Crash Management
- Alex Franchuk improved the crash reporter client to use all available locales (in the user-configured order), which addresses a number of user-reported issues and removes some of the only hard-error cases of the client code (in which the client can’t proceed to send a crash report). Now, in the worst case, the build-time-compiled en-US localized strings will be shown.
- Alex Franchuk has added additional error context to the UI of the crash reporter client for users to provide more information in bug reports (without needing to find the log file).
Glean migration 
- All legacy telemetry Scalars have been migrated to Glean, and migration of Histogram is progressing rapidly. Progress can be followed on arewegleanyet.com.
- Rebuilds after adding a new Glean metric are now much faster.
- All data collection (both legacy Telemetry and Glean) can now be visualized in the Firefox Profiler with markers.
Lint, Static Analysis and Code Coverage
- The usual upgrades of ruff and codespell happened !
OS Integration and Security
- Bob Owen shipped USER_RESTRICTED for the content process sandbox to Firefox release. This removes content process read access to most of the file system, registry and other ACL protected system objects.
PDF.js
- Calixte made it possible to re-edit already existing drawings on PDFs.
- Ujjwal Sharma implemented a popup to undo deletion of annotations.
- Calixte made several improvements to drawing on touch screens, in preparation for supporting PDF editing on Android. For example, fixing issues with drawing & zooming.
- Calixte greatly improved the performance of the font renderer.
- Calixte landed a background remover and curve extractor for images, which will be used for adding signatures.
Firefox Translations
- (Bug 1935511) Erik Nordin has updated the Translations platform to allow for using backward-compatible 1.x version models while also shipping breaking changes via 2.x version models that are only available to newer versions of Firefox.
- (Bug 1936163) Erik Nordin has updated the Translations platform to allow models to be differentiated based on a script tag. This is necessary to differentiate, for example, Simplified Chinese from Traditional Chinese.
- (Bug 1940142) Erik Nordin has added an automated performance test that runs on pushes to autoland, which will catch statistically significant regressions or improvements in translating our Spanish-to-English benchmark page, e.g. (PerfCompare).
- (Bug 1943132) Erik Nordin has instrumented the TranslationsEngine code with telemetry that will report performance metrics, such as words per second.
- Erik Nordin has updated all of our most-used telemetry metrics to utilize events_stream instead of events_unnested as a data source, greatly improving performance and reducing the cost of queries.
- Erik Nordin has reworked our Translations daily active usage telemetry to aggregate by regions of the world instead of only a single global value. This will help us better understand the impact of releasing CJK translation models, particularly in Asia.
- Evgeny Pavlov has successfully trained four models that will be released in Firefox 135 for Desktop.
- zh-Hans → en (Translations from Simplified Chinese)
- ja → en (Translations from Japanese)
- ko → en (Translations from Korean)
- en → ru (Translations into Russian)
- (transaltions/860) Evgeny Pavlov has replaced tokenization to ICU segmenter that can be used for all languages
- Evgeny Pavlov has updated documentation for model training and contributing
- (transaltions/884,247) Evgeny Pavlov has improved monolingual data acquisition by integrating Monocleaner and HPLT 2.0 dataset
- (translations/955) Greg Tatum landed the ability to pull models from various sources such as taskcluster and host them on a localhost Remote Settings server via Docker. This greatly improves the process testing experimental models directly within Firefox.
- (translations/977) Greg Tatum conducted an investigation into why our models sometimes have an issue with inventing made-up words, and deftly discovered that our configuration for lexical shortlists was causing the issue.
- (Bug 1940972) Erik Nordin implemented a patch that disables lexical shortlisting by default, but allows it to be controlled dynamically via a user preference. This improves overall translation quality by reducing made-up words, at the expense of a slight reduction in translation speed.
- (Bug 1932780) Greg Tatum fixed a regression that was causing some translation requests to accidentally be canceled before completing. This regression was caught and fixed before ever releasing to users.
- (Bug 1942349) Greg Tatum has updated the Translations platform to allow multiple models for the same language to be differentiated by an optional variant tag. This is a developer-only feature that will greatly help us test different model variants directly in Firefox when running experiments.
- (translations/165) Greg Tatum has added support for CTranslated2 decoding which makes the most expensive part of the pipeline to run significantly faster
- (translations/894) Greg Tatum experimented with student model architectures and found one with a good balance of speed of translation and quality
- (Bug 1942037) Marco Castelluccio enabled conditional support for avxvnni, improving Translations performance on Windows (PerfCompare).
Phabricator 
, moz-phab, and Lando
- Glandium made mach try and Lando handle patches introducing CRLF characters correctly.
Release Engineering, Taskcluster and Release Management
- Andrew Halberstadt added integration tests to our Taskcluster configuration repository. These new tests will allow us to roll out updates to worker images quicker and with less risk.
- Gabriel Bustamante and Heitor Neiva made it possible for Release Management to uplift Firefox versions across branches (“merge day operations”) without Release Engineering involvement.
- Bastien Orivel has made a number of improvements to generic-worker that make it easier and more secure as we migrate tasks off of the deprecated docker-worker to it.
- Ben Hearsum shepherded some work from a contributor that allowed us to enable lzma compression in our macOS DMGs, which has reduced their size and sped up installation.
- Pascal Chevrel created a page that displays the Firefox version shipped on all external stores we publish to trainqueries.herokuapp.com/stores/ for PMs that don’t have stores access
- Pascal Chevrel migrated our FirefoxNightly account from mozilla.social to mastodon.social
- Pascal Chevrel updated whattrainisitnow.com with Mozilla branding and pointers making clear to end-users that this is the official Mozilla calendar. This is now also linked from our beta release notes.
- Heitor Neiva worked with Rob Lemley (Thunderbird) to publish Thunderbird onto the Microsoft Store. As a datapoint, Firefox has been on the Microsoft Store for 3 years and now gathers 3M MAU (still linearly growing!).
- Julien Cristau upgraded the Windows workers of NSS from Windows 2012 to Windows 2022.
- Yaraslau Kurmyza fixed a timeout that sometimes killed workers while they were running tasks
- Matt Boris fixed an intermittent issue on generic-worker that prevented Linux GUI workers from initializing correctly.
- Pete Moore and Yaraslau Kurmyza made static workers change their config without recreating a new worker
- Yaraslau Kurmyza added metrics logging against cloud providers because we suspect some calls are inefficient on Firefox CI
- Ryan VanderMeulen worked with the Product team to determine future support for Windows 7 users on the ESR115 branch with the current March deadline coming up. It was decided to extend support for another 6 months to September 2025.
- Ryan VanderMeulen worked with Nicole Zeltzer and Marcin Kozinski to migrate Pocket from its own developer account in the Google Play Store to Mozilla’s developer account, streamlining and simplifying app and account management for Pocket going forward.
Version Control
- Julien Cristau upgraded Mercurial to version 6.8.2 on hg.mozilla.org.
- Connor Sheehan and Johnathon Moss debugged issues with clonebundle object expiry in Azure.
Mercurial to Git project
- EngWF has returned from a hg-to-git team week in Toronto!
- The team is preparing to move mozilla-central & related repos to new Lando.
- Bulk of the work is finished, now porting newer features and testing repo imports.
- Preparing step-by-step plan for the migration.
- The git-hg-sync service resources have been bootstrapped, and glob is working on adding Dockerflow to the service for productionization.
- RRA (risk review) for the new service is scheduled.
- Initial framework for the Lando Headless API is up in a draft PR.
- Git support in Lando’s landing worker in final stages: testing auth, ready for dogfooding with moz-phab.
- First lando pushes to github:
- Bulk of pushlog support written and up in a draft PR.
- Here’s a link to the hg to git project roadmap
Thanks for reading and see you next month!