Welcome to the March edition of the Engineering Effectiveness Newsletter! The Engineering Effectiveness org makes it easy to develop, test and release Mozilla software at scale. See below for some highlights, then read on for more detailed info!
Highlights
- MozillaBuild 4.0 is released! Featuring better hg integration, a modernized raft of Linux tools, and a migration from Python 2 to Python 3.
- All non-shippable builds have been migrated from AWS to GCP! This results in a nice little cost savings, but more importantly gets the ball rolling on the overall migration to GCP in our CI.
- Two weekend releases were successfully shipped! First a chemspill release and next to remove Yandex, all in addition to the regularly scheduled releases.
Contributors
- anastasiostam
Detailed Project Updates
Bugzilla 
and Bugbug 
and Bugbug 
- Suhaib implemented an autonag script to suggest increasing the severity of a bug when it is accessibility related
- Ksenia improved the model to identify issues needing diagnosis by also considering issues with “moved” milestone
- Marco added a list of S2 regressions to the quality notification some engineering teams are receiving
- Suhaib implemented an autonag script to highlight bugs whose severity could be underestimated (e.g. bugs with many duplicates).
- Sebastian built a dashboard tracking open bugs with severity S2 tracking dashboard for Firefox org (goal reduction by 50% til the end of H1)
- Suhaib implemented an autonag script to boost the attention to security bugs by needinfo’ing to increase the severity when a bug is classified as sec-high.
- Suhaib implemented an autonag script to highlight components which need a triage owner to be assigned (e.g. inactive triage owner)
Build System and Mach Environment
- Mitch released MozillaBuild 4.0, featuring better hg integration, a modernized raft of Linux tools, and a migration from Python 2 to Python 3.
- Mitch added docs for declaring Python dependencies of Mach commands
- Mitch migrated CI’s usage of “system Python packages” to be explicitly opt-in, dramatically lowering the amount of tasks that use the system Python and allowing easier adoption of modern dependencies
- Mitch and Alex sped up Windows builds on developer machines by 3 minutes when sccache is used
- Mitch removed the need for Mach commands to manually activate their virtualenv
CI and Treeherder 
- masterwayz migrated all non-shippable builds from AWS -> GCP, resulting in modest cost savings and getting the ball rolling on the overall migration
- ahal and hneiva updated Taskgraph consumers to grab Taskgraph from Pypi rather than cloning, providing the benefit of semantic versioning and paving the way to migrate Taskgraph to the Taskcluster org on Github
- Gabriel created new Decision task images to drop unsupported versions of Mercurial and Python
- jmaher piloted a new method of tracking Intermittent failures using simple per test tracking bugs (vs a bug for every unique failure). This is xpcshell-only for now but planning to expand scope in April
Crash Management
- Gabriele added support for storing thread names natively in crash reports, this fixed a race that could deadlock Firefox, made starting and shutting down threads faster and saved some memory in every process
- Gabriele ensured that the fix-stacks tool is built for 64-bit Windows hosts, meaning that crashes in tests will be properly symbolicated when ran locally on those machines
- Gabriele added support for catching stack overflows on Linux
- Gabriele fixed an issue in the new stackwalker that prevented our scraping scripts from finding symbols for unknown Microsoft libraries
Lint, Static Analysis and Code Coverage
- Marco fixed grcov’s CI, which had been broken for a while after the switch from Travis/Appveyor to GitHub Actions
- Sylvestre fixed and validated the trojan source linter
Phabricator 
, moz-phab, and Lando 
, moz-phab, and Lando 
- Mitch drastically lowered the number of emails submitted by Phabricator, emails will no longer be sent due to changes in parent revisions
- Sheehan removed a dependency on
.arcconfigthat caused revisions to move between repositories erroneously on merge - Sheehan added a warning to moz-phab when hg is used without evolve
Release Engineering 
and Management
and Management 
- Relman successfully shipped two urgent releases over weekends, a chemspill for a zero day and a second one to remove Yandex from Firefox
- Gabriel enhanced Taskgraph’s CI to run tests across all supported Python versions
Version Control 
- Sheehan added clonebundles to new GCP regions, improving clone times in CI as we migrate to GCP
- Sheehan fixed a bug in pushlog that sometimes caused changesets to land without pushlog information
Other 
- Anastasios Bithas wrote an extension to automatically linkify bugs on Firefox Source Docs
- Sylvestre added a “Report an issue” link to each page of Firefox Source Docs
- Sylvestre replaced the search in Firefox Source Docs with one backed by Google Programmable Search Engine, providing much more relevant results
- Mitch improved usability of pytest failures by making stack traces more consistent and terse
Thanks for reading and see you next month!