Welcome to the December 2022 and January 2023 edition of the Engineering Effectiveness Newsletter! The Engineering Effectiveness org makes it easy to develop, test and release Mozilla software at scale. See below for some highlights, then read on for more detailed info!
Highlights
- Fenix (nightly) is now able to display PDFs
- The LLVM toolchain used to build fx is now linked with lld and built with Link Time Optimization (LTO), bringing better performances while decreasing build time
- Kris Wright landed bug 1748378 providing a way to have low power thread pools which will run on low power cores where it is supported. As a start, this is applied to the background IO pool.
-
Serge Guelton fixed a 15 years old bug - uploading a file on Linux without read permissions wasn’t failing
- Some folks wrote an article in French about it: https://www.android-mt.com/news/mozilla-corrige-un-bug-de-firefox-15-ans-apres-son-signalement/144932/
Contributors
- Olivier Giorgis (ogiorgis)
- Carine Dengler
Detailed Project Updates
Bugzilla 
and Bugbug 
and Bugbug 
- Webhook improvements including new API endpoint to list your current webhooks and status.
- Github push commits will soon create comments in related bug reports and even close a bug.
- Suhaib implemented several improvements to make autonag more accurate in detecting inactive users: PR1809, PR1810 and PR1842
Build System and Mach Environment
- Mach bootstrap has now its own testsuite on various Distro 1245969 - Automated testing for
mach bootstrap - The LLVM toolchain used to build fx is now linked with lld and built with Link Time Optimization (LTO), bringing better performances while decreasing build time: 1799423 - clang should be linked with lld, 1801268 - Clang should be built with ThinLTO
CI and Treeherder 
-
Serge Guelton made several perf improvements to Taskgraph task generation:
1805842 - Job transformation relies a lot on expensive deepcopy
1806472 - Avoid redundant calls to fromNow
1803537 - Prune useless deepcopies - Michelle Goossens migrated the last remaining tasks from AWS -> GCP!
- Joel Maher migrated pixel2 tests to pixel5 (1 task remains)
- Many coordinated to disable win/aarch64 in CI (hands on testing is all that is needed)
Crash Management
- Alex Franchuk implemented a minimal crash ping using the Glean backend and work has started to port this to Android as well.
Fuzzing and Sanitizers
-
Ryan VanderMeulen enabled more test suites running in TSAN mode in CI, filing some new sec bugs along the way:
1805536 - Enable more passing TSAN test suites
1806483 - Enable TSAN cppunittests by default in CI
1807238 - Enable TSAN mochitest-devtools by default in CI
1806832 - Enable TSAN mochitest-chrome by default in CI
Lint, Static Analysis and Code Coverage
- Flake8 formatting is now ignored in favor of black: 1806068 - Redundant format between flake8 and black
- Alex Franchuk improved cargo-vet’s precision of third-party package auditing with explicit policies about crate versions.
OS Integration and Security 
- When we investigated the crash spike with Avast from Bug 1794064, we discovered that the root cause was a potential sandbox escape vulnerability. It is now officially listed as CVE-2022-4291, and Norton, who bought Avast recently, credits Mozilla for it in the dedicated security advisory.
- Greg Stoll blogged about his debugging experience as a new developer at Mozilla:
- Greg Stoll landed a new feature that lets Windows users block (some) DLLs from being loaded into Firefox processes from the about:third-party page.
- Alex Franchuk implemented several features in rust-minidump that make it easier to diagnose crashes: better detection of CPU microcode versions, improved handling of non-canonical addresses, and bitflip detection for bad hardware.
- Chris Martin and the graphics team decided to let the GPU Sandbox ride the release train now that we have a better understanding of incompatibility issues.
- Serge Guelton and Yannis Juglaret fixed an LLVM bug that broke crash traces in delay-loaded libraries on Windows.
- Jed Davis fixed a long standing bug that made us unable to get good backtraces for some hangs in CI.
-
Serge Guelton fixed a 15 years old bug - uploading a file on Linux without read permissions wasn’t failing
- Some folks wrote an article in French about it: https://www.android-mt.com/news/mozilla-corrige-un-bug-de-firefox-15-ans-apres-son-signalement/144932/
PDF.js 
- Fenix (nightly) is now able to display PDFs
- The find-in-page feature is working as in normal web pages
- An API to support saving a PDF was added to GeckoView
Power use
- Kris Wright landed bug 1748378 providing a way to have low power thread pools which will run on low power cores where it is supported. As a start, this is applied to the background IO pool.
- Profiler tooltips on power tracks now show the CO2 equivalent, thanks to Chris Adams and Fershad from The Green Web Foundation.
- Session store data is now written no more than once per hour when the user is inactive.
- Some fixes landed that made it possible for Firefox to remain completely idle for more than a minute when the user is inactive:
- The user idle service no longer uses a repeating 5s timer.
- Andrew Osmond fixed the gfxFontCache expiration so that it doesn’t wake-up every 10s.
- The compositor is now destroyed immediately when a window is closed, ensuring compositor animations are stopped even when the window isn’t garbage collected quickly.
Phabricator 
, moz-phab, and Lando 
, moz-phab, and Lando 
-
Connor Sheehan added
mach lintoutput to failure emails generated during Lando’s autoformatting - Connor Sheehan fixed issues around moz-phab automatic updates on Windows
- Connor Sheehan released moz-phab 1.3.0
- Connor Sheehan released moz-phab 1.4.0
- Connor Sheehan improved moz-phab’s overall performance when working with large stacks on Phabricator
-
Connor Sheehan made
moz-phab reorgno longer update revision statuses on reorg- Revisions will not have review be re-requested
- Abandoned revisions can be removed from the stack
Release Engineering 
and Management
and Management 
- Johan Lorenzo shot a promotional video about the way Firefox for Android uses Mergify. Interview was also published as a post on their blog.
- Andrew Halberstadt added support for authenticode ev signing to signingscript
- Ben Hearsum is making plans and filing bugs to simplify Taskcluster usage and onboarding for new projects.
- Gabriel Bustamante is making incremental progress on releasing Mozilla Firefox builds in .deb packages. So far we’ve got: repackage tasks to create Firefox .debs from Mozilla Linux binaries, a distribution repository to track package uptake and configure the client, and a WIP patch for Google Artifact Registry support in beetmover (firefox-deb-repackage meta-bug)
- Geoff Brown and Johan Lorenzo migrated Focus (Android) to the new Android Monorepo (meta-bug 1797704). Next step: Move Fenix.
- Johan Lorenzo optimized the time to get Android builds started (bug 1807237 and bug 1807237). It was a 3-year long issue that took Release Managers’ mental space. The number of scheduled was also reduced to what is actually necessary (bug 1803141)
- Pascal Chevrel identified through the Local Firefox project that Spanish built-in dictionaries were missing for some locales even though there was a suitably-licensed dictionary available. As a result, Firefox builds in the Spanish from Spain (es-ES) and Spanish from Argentina (es-AR) locales now come with a built-in dictionary for the Firefox spellchecker! This change will also be shipping in the upcoming ESR 102.8.0 release as well.
mozregression
-
Zeid Zabaneh investigated and resolved an issue preventing testing Android on Windows (bug 1794416).
- Thanks to Geoff Brown for implementing the fix in mozdevice (bug 1808621).
- Released in 5.3.0
-
Zeid Zabaneh added support for universal2 builds for macOS, allowing mozregression to run without Rosetta on arm64 (bug 1810220).
- Thanks to Jan-Erik Rediger for adding support for universal2 builds in glean-sdk to help support this release (bug 1810737).
- Also released in 5.3.0, which included many other quality of life improvements.
- Also fixes a mozregression-gui issue on macOS 13 (bug 1802359).
mots
-
Zeid Zabaneh added the ability to run mots via a ./mach mots (bug 1797929)
- Released in 0.6.0
-
Zeid Zabaneh added the ability to suppress refreshing existing user info from Bugzilla in mots.yaml, when cleaning/exporting, which is now the default behaviour. This allows people to manually modify info if needed, and it prevents unwanted changes from showing up in revisions (bugs 1782280, 1803651, 1814237)
- Released in 0.7.0rc0
- Rob Lemley added a wordwrap filter to the export template to make mots.yaml look nicer.
Other 
- Documentation: Olivier Giorgis fixed an old bug (1557020). Touching any documentation has removed the TOC and making the link harder; Olivier also removed many warnings in the doc.
- Documentation: Carine Dengler added a check to make sure that the number of warnings in the documentation doesn’t increase.
Thanks for reading and see you next month!