Welcome to the January edition of the Engineering Effectiveness Newsletter! This edition also encompasses changes from December. The Engineering Effectiveness org makes it easy to develop, test and release Mozilla software at scale. See below for some highlights, then read on for more detailed info!
Highlights
- A root cause analysis (RCA) for the FoxStuck incident is now available and a technical blog post has been published.
- Engineering Effectiveness is hiring! If you know of anyone who would be a great fit for any of the following roles, let us know!
- Machine Learning Engineer (Development Tools)
- Python Software Engineer - Tooling
Contributors
- Luni-4
- Giovanni Tangredi (giovannitangredi)
- Luca Barbato (lu-zero)
- Marco Ballario (marco-ballario)
- Fabio Huser (fh1ch)
- Stefan Nuernberger (snue)
- Dominik Nakamura (dnaka91)
- Masterwayz
- Tom Schuster (evilpie)
- Jonas Jenwald (Snuffleupagus)
- Kodai Wakai (KouWakai)
- quaoaris
- Shivam Singhal (championshuttler)
Detailed Project Updates
Bugzilla 
and Bugbug
- pulsebot is now running within our production infrastructure and is owned by Engineering Workflow. Thanks to glandium for writing a highly valuable tool, and to dkl for doing the work to rehost it.
- Sylvestre added a new autonag check to escalate bugs with several duplicates
Build System and Mach Environment
- championshuttler removed deprecated recommonmark and added support for myst-parser in Firefox Source Docs. Developers can now integrate mermaid graphs in markdown files too.
- mhentges added the ability to invoke Mach outside of the MozillaBuild environment on Windows, allowing Windows developers to use their shell of choice and unlocking the way for future MozillaBuild improvements
- glandium removed the dependency on pkg-config on platforms that don’t require it (currently Windows, macOS, and Android); closing out a 13 year old pain point
- alex untangled
mach show-log’s usage of pipes resolving various bugs on Windows (including it outright failing to work) - glandium made configure bootstrapping faster by avoiding duplicate taskgraph lookups; the duplicate looked wasted a noticeable amount of time, especially on Windows
CI and Treeherder 
- Sebastian made Treeherder UI resilient against taskcluster downtime and automated reopening of intermittent bugs which turned active again
- jmaher, masterwayz and ahal kicked off the Firefox CI worker pool GCP migration
- ahal made some more progress merging the Gecko and standalone taskgraph code bases
- sclements added some more improvements to the taskgraph docs, including a “Debugging and Testing” section
Crash Management
- Stack overflows are now properly caught in Linux builds
Fuzzing and Sanitizers
- The fuzzing team is making progress in implementing the IPC Fuzzing Strategy
- Working to enable additional UBSan checks function and float-cast-overflow. Nightly can now launch without triggering any issues. A few fixes remain for tests failing in CI.
- Implemented a fuzzing target for WebGPU’s Javascript API in Domino
Lint, Static Analysis and Code Coverage
- fh1ch fixed grcov’s Cobertura output to match what the standard cobertura tool does
- dnaka91 set grcov’s CI up on GitHub Actions, long overdue after Travis CI deprecation
- snue fixed multiple Clippy issues in grcov and enabled running it as part of CI
- marco-ballario added integration tests for rust-code-analysis and added averages for some metrics (e.g. the Nom metric)
- Luni-4 added support for parsing files concurrently in rust-code-analysis, updated some grammars and fixed the automated publication of the crate
- giovannitangredi added minimum and maximum implementation for some rust-code-analysis metrics (e.g. the Nargs metric)
OS Integration and Security
- win32k lockdown is progressing and landed default enabled on Nightly. This had to be backed out, but gave us valuable information on remaining issues.
- We’ve enabled pre-spawn CIG (= enable Code Integrity Guard before a process is created) for socket process to reject code injection into socket process (currently only for nightly and early beta due to compat risk)
- OS memory pressure detection and tab unloading shipped for Linux in 96.
- Documentation for Gecko Processes and IPC has landed. This includes the IPDL language and some basic process feature integration (bootstrapping, sandboxing, XPCOM, etc).
- Web MIDI will ship in Firefox 98 on Windows, macOS and Linux. The OS-specific bindings were implemented by relying on the midir Rust crate.
PDF.js
- Jonas enforced PAGE-scrolling for very large/long documents
- Kenan is helping reproduce and find regression ranges for many pre-existing pdf.js bugs. Thanks to his contributions, we closed dozens of bugs that are no longer reproducible and we have more information to fix the bugs that still exist.
- KouWakai fixed the handling of non-integer Annotation border widths
- Quaoaris fixed lines (stroke) that are rendered too thick
- Brendan simplified the tags we use to track pdf.js bugs, and Calixte helped convert all existing bugs to the new format and collect statistics to investigate the most common problems.
- Calixte improved the algorithm to calculate the font size for AcroForm fields
- Calixte fixed a long standing bug where pdf.js UI buttons would stay in hover state after being clicked
Phabricator 
, moz-phab, Lando , and Version Control
- In Bug 1749747 evilpie added a commit hook to our Mercurial server to ensure that patches cannot be landed with a “WIP:” prefix; soon after Connor added the same check to Lando to catch this mistake earlier in the workflow.
Release Engineering and Management
- hneiva, jmaher, ahal and aki migrated mozilla-extensions repositories to python3 and latest taskcluster version
- gbrown automated publishing of Firefox releases to the Microsoft Store (beta only initially, release coming soon)
- Release Management
- 1 major release (Firefox 96, driven by Dianna), 2 dot releases
- handled 5 incidents of various severities
- The team can now be contacted on Slack with the @relman alias
- Release Engineering
- The team can now be contacted on Slack with the @releng alias
- Sclements has moved Focus for Android releases from a manual process in Github to one initiated by Relman in shipit. This is the last milestone for fully modernizing the project’s CI and aligning the release process with other Firefox projects
Thanks for reading and see you next month!