Welcome to the December edition of the Engineering Effectiveness Newsletter! The Engineering Effectiveness org makes it easy to develop, test and release Mozilla software at scale. See below for some highlights, then read on for more detailed info!
Highlights
- MozillaBuild 4.1 was released.
- Add support for colored highlighting in pdfs.
- Fenix as a default pdf reader on Android is live in 122.
- Connor Sheehan added a dedicated landing worker for Lando-try pushes. Pushes will land faster and no longer share a landing queue with regular landings.
- New releases of moz-phab.
- Gabriel Bustamante shipped Mozilla’s first official .deb package for Linux users in Firefox 122.
Contributors
Detailed Project Updates
Bugzilla 
and Bugbug 
and Bugbug 
- Sylvestre Ledru implemented a new BugBot rule to notify release managers about bugs with a lot of see aslo.
- John Pangas implemented a new BugBug model to identify accessibility related bugs.
- Promise Fru implemented a new BugBug model to identify performance related bugs.
Build System and Mach Environment
- MozillaBuild 4.1 was released.
CI and Treeherder 
- Andrew Halberstadt implemented the -E/–use-existing-tasks flag for ./mach try fuzzy and ./mach try chooser. This will detect your previous try push and insert all the tasks that ran there (minus any tasks you explicitly select) into the graph of your new try push. This allows for faster iteration on the task at hand, without needing to wait for its dependencies each time.
Crash Management
- Windows crash reports now contain the list of open HANDLEs of the crashed process and Linux crash reports include the list of open file descriptors. Additionally the Linux crash reports include the contents of the /proc//limits file. All this data is not yet visible on crash-stats but can be extracted from the minidumps if you have protected data access.
- Alpine Linux enabled the crash reporter in their Firefox package and we started scraping their symbols and ingesting crash reports from their users.
- Crash reports from external Fenix forks are now filtered out so they don’t pollute the data on crash-stats
- Suhaib Mujahid improved BugBot to detect and highlight PHC crashes (thanks to Paul Bone)
Lint, Static Analysis and Code Coverage
- The whitespace and file permission linters now run on yaml files (e.g. metrics.yaml)
- All production files are now named using the .worker.*js syntax.
- Additionally, we have turned on no-unused-vars on a global scope for workers.
- We have now added an ESLint rule to reject new uses of Console.sys.mjs and Log.sys.mjs.
- We have three different module logging mechanisms available to our JavaScript code, and want to reduce those down to one.
- For new code please use console.createInstance.
- We will start with the migration away from Console.sys.mjs as that is closest to console.createInstance, and will look at Log.sys.mjs later. However, if teams want to investigate moving away sooner, and find issues, please file bugs blocking the appropriate metas (Console.sys.mjs, Log.sys.mjs)
- The license linter has been fixed to not report an error when it has successfully fixed a file. It will also now automatically fix .mjs files.
- XPCOMUtils.defineLazyGetter has now been replaced by ChromeUtils.defineLazyGetter. An ESLint rule has also been added to reject uses of the former.
- The whitespace and file permission linters now run on .mn files (e.g. jar.mn).
- ESlint has deprecated the no-return-await rule. We have therefore stopped enforcing it on mozilla-central.
OS Integration and Security 
- Paul Bone is continuing the PHC roll-out. PHC is currently enabled for 10% of the Firefox Release population. While most of the crash reports coming in are bad hardware or similar we’re beginning to get actionable information for real memory errors.
PDF.js 
- Add support for highlighting (text or not).
- Switch from CDP to BiDi driver when running tests: there are almost no more intermittents with Firefox.
- Fenix as a default pdf reader on Android is live in 122.
Firefox Translations 
- https://bugzilla.mozilla.org/show_bug.cgi?id=1868949
- Unblocked training any languages by integrating Bicleaner-AI multilingual model for dataset filtering and optimizing BicleanerAI to run efficiently on GPUs
- Successfully completed an experiment to prove that OpusTrainer fixes casing issues in translations by training English -> Lithuanian model. Evaluation results:
- Fixed a major training instability issue
- Integrated training continuation to reuse previously trained models
- https://bugzilla.mozilla.org/show_bug.cgi?id=1837424
- https://bugzilla.mozilla.org/show_bug.cgi?id=1873743
Phabricator 
, moz-phab, and Lando 
, moz-phab, and Lando 
- Connor Sheehan released moz-phab 1.5.0 and 1.5.1. See release notes below.
- Connor Sheehan added a dedicated landing worker for Lando-try pushes. Pushes will land faster and no longer share a landing queue with regular landings.
- Connor Sheehan made mach try –push-to-lando better at identifying the base revision for submission, fixing some issues where patches would submit but fail to apply within Lando.
- Connor Sheehan fixed an issue with Lando’s Git patch parser where patches would fail to apply within Lando due to incorrectly parsed Git version info in the patch.
- Connor Sheehan made Lando-uplift re-use existing Phabricator diff information instead of reconstructing from a raw patch, fixing display issues in the diff view and resolving an issue where binary files would be lost at landing time.
- Connor Sheehan moved the uplift request button to a more discoverable area and the form into a separate modal.
Release Engineering 
and Management
and Management 
- Release Management released Firefox Fx121 and Fx122 on Desktop, Android, and iOS.
- Fx121 also included 1 dot release on Desktop and 2 dot releases for Android and iOS
- Fx122 also included 1 dot release on Desktop and 2 dot releases for Android and iOS
- Gabriel Bustamante shipped Mozilla’s first official .deb package for Linux users in Firefox 122. After 7 days on release, we already gathered 4k DAU and 8k MAU and numbers are still steadily growing. The .deb package was called out in 40 news articles.
- Heitor Neiva and Haik Aftandilian, thanks to their work on macOS hardened entitlements, enabled the Security Engineering team to ship macOS Passkeys. This major highlight of Firefox 122 was covered in 27 news articles.
- Geoff Brown and Ryan VanderMeulen enabled a new Android packaging format (AAB) on Firefox for Android. We were close to not being able to ship new features anymore. The new format sets a new size constraint that is twice larger. Moreover Fenix packages got 30% smaller which improves the update experience.
- Ben Hearsum fixed a bug in Taskcluster that caused live logs and interactive tasks to fail under certain circumstances.
- Julien Cristau solved the biggest problem we had on the Android repository migration: how do we connect the Firefox build system to the Fenix one when there’s all in the same repo. This was a problem with a lack of knowledge at Mozilla and dozens of parameters to take into account.
- The work that Ben Hearsum and Heitor Neiva did on macOS Attribution in 2023H2 (Firefox ORK 4.1) is rolling out with Firefox 123. Early data shows a very high attribution rate of ~90%.
- Andrew Halberstadt made improvements to Ship-It to support arbitrary groups of projects. Previously all projects were assumed to be related to Firefox or mozilla-extensions.
- Building on the above, Andrew Halberstadt added support for releasing Mozilla VPN client and addons through Ship-It.
Version Control 
- Connor Sheehan and an external contractor worked to resolve issues around corruption in the try repo, allowing us to upgrade the storage format. This provided a moderate improvement in push time.
- Connor Sheehan added support for clonebundles manifest filtering in newer versions of Mercurial, avoiding bustages in CI.
Thanks for reading and see you next month!