Engineering Effectiveness Newsletter (May 2023 Edition)

Firefox Tooling Announcements
#1

Welcome to the May edition of the Engineering Effectiveness Newsletter! The Engineering Effectiveness org makes it easy to develop, test and release Mozilla software at scale. See below for some highlights, then read on for more detailed info!

Highlights

  • Translations has been enabled in Nightly and early Beta, please test it and file bugs!

Contributors

  • Tim van der Meij (timvandermeij)
  • Jonas Jenwald (Snuffleupagus)
  • Bungeman
  • Anurag Bhandari (WhiteWolf47)
  • Emeka Onwuepe (Emeka-Onwuepe)
  • Kenneth Chew (kthchew)

Detailed Project Updates

Bugzilla and Bugbug

Build System and Mach Environment

CI and Treeherder

  • Sccache: Version 0.5.0 released
  • We removed a lot of useless files on our Taskcluster storage, saving a lot of money every month.
    • We also filed follow-up bugs to set the expiration for these files to be shorter, so we keep these savings long term.
  • ahal implemented a new set of transforms in Taskgraph called from_deps. These aim to make it easy to create follow-up tasks and will replace the multi_dep loader.
  • aerickson and ahal are working on getting tests running in a Wayland environment

Crash Management

  • All environment management functions (getenv(), setenv(), etc…) now have interposers that provide thread-safe access on both Linux and Android. This prevents a wide range of crashes our users were experiencing. This work on the interposers also had the side-effect of making stack overflows catchable in Android main-process crashes.
  • Crash instruction disassembly now works on 32-bit x86 builds where it had been disabled due to a regression in the minidump generation code
  • Several crash analysis features were added to the stackwalker: crashes caused by NULL pointer accesses are now clearly reported , native debug information can be used for stack-walking and heuristics were added to detect crashes caused by bit-flips (often caused by flaky hardware) and accesses to guard pages (which might indicate potential security issues)

Lint, Static Analysis and Code Coverage

  • The enabling of Prettier on JSON files has now landed.
  • Gijs has landed stylelint for css linting.
    • This will flag up errors in your CSS, like duplicate properties, invalid syntax, missing generic font families, etc. In some cases it can auto-fix things.
    • We will expand this in the next few weeks/months to cover conventions in use in Firefox (e.g. using “em” for font-size over “px”, preferring logical over physical margin/padding/border/float definitions, using “0” without a unit for sizes, etc.)
    • This is not a formatter like prettier, so it doesn’t currently adjust whitespace, indenting etc.
  • ./mach lint . now works in parallel mode for faster linting. Previously only ./mach lint * would work, but that didn’t pick up the top-level dot files.
  • Next up, is to upgrade Prettier and enable it for production HTML files.
    • The upgrade does change a significant number of JS files due to changes in Prettier’s formatting.
    • We decided to put off enabling it on test files for now, due to the large amount of files affected.

OS Integration and Security

  • We now have a methodology for measuring the impact of antivirus products on CPU usage when Firefox is running and a meta bug to track the work on this topic. QA has successfully applied the methodology on two Nightly builds to help us measure the effect of a specific patch on AV-induced CPU usage. The next step is to collect similar data recurrently during release cycles.
  • cargo-vet has added a number of user conveniences to streamline auditing, and is preparing for a release following an early announcement from Google regarding their use of the tool to audit their dependencies.

PDF.js

Firefox Translations

  • We’ve been working to migrate the Firefox Translations addon directly into mozilla-central, see the meta bug for all the progress so far!

  • … and Translations has been enabled in Nightly and early Beta, please test it and file bugs!

  • Greg Tatum created an Are We Translations Yet page to track the engineering work.
    [image]

  • Greg Tatum landed about:translations.
    [image]

  • Erik Nordin has landed new ML based infrastructure for detecting languages. The screenshot below shows the language code for the text that is detected along with a probability.
    [image]
    [image]

  • Serge Guelton implemented a replacement of intgemm (8-bit integer matrix multiplication using xsimd when possible) that is easier to integrate into Firefox and easier to maintain

  • Greg Tatum ported the full page translation feature from the addon, and got it working with the JSActor system so that it works on both mobile and desktop. This means that webpages can be fully translated, and kept translated.

  • Greg Tatum implemented the translations popup to power full page translations.
    [image]

  • Erik Nordin added source documentation for the Firefox Translations project, including documenting the process of building the fastText (language identification) models for Firefox, defining the process of versioning machine-learning model and WASM upgrades across breaking and non-breaking changes.

  • Erik Nordin added the ability to distinguish between which translation models are in beta or not based on the versioning documentation in the above bullet point.
    [image]

  • Ben Hearsum has made significant progress moving the language training pipeline into Taskcluster and GCP. When completed, this work will allow us to better optimize and scale our training jobs.

Phabricator, moz-phab, and Lando

  • Zeid fixed a bug in Lando that was causing deadlocks during deployments that had certain types of database migrations (bug 1832083).
  • Zeid added filesystem caching support in Lando, making local testing and development faster (bug 1832142).
  • Zeid simplified and improved landing worker functionality by implementing a proper many-to-many relationship between landing jobs and revisions, reducing code complexity and improving versatility (bug 1825277).
  • May Lando deployments (see release notes): release-39, release-40, release-41, release-42.
    • Thanks for your patience during Lando maintenance downtimes!
  • Zeid added the latest diffID to Phabricator’s differential.revision.search endpoint, which is now returned in the response (bug 1798737) removing the need for additional queries to fetch the diffID from a “phid”.
  • Connor Sheehan linkified differential revision text in Landing warnings, making it easier to go to revisions with landing warnings to address them.

Release Engineering and Management

  • Heitor Neiva got Firefox Nightly notarized with rcodesign (bug 1774273). Shipping Firefox on macOS requires our builds to be submitted to Apple’s Notary service. Otherwise macOS users get a scary warning about the origin of the binary. Moving to rcodesign (instead of Apple’s tool) reduces RelEng’s dependency on hard-to-maintain Mac machines.
  • Geoff Brown optimized the infamous external-gradle-dependencies tasks on Android (bug 1820122) by finding a great compromise. These tasks are notoriously long to execute and Geoff halved their duration. He found a sweet spot between the duration of the tasks and the number of gradle dependencies that get cached.
  • Ben Hearsum, Heitor Neiva, and Hal Wine rotated Mozilla’s GPG signing key. This key is used to sign a lot of Mozilla’s artifacts including Linux binaries, source files, checksums, etc. We rotate it every 2 years and we strictly follow a security procedure to ensure no key gets compromised.
  • Andrew Halberstadt worked with the Application Services team to help set up a release promotion pipeline + beetmover tasks in their CI.
  • Julien Cristau implemented a better way to reference GitHub commits on mozilla-central (firefox-android#2197). This will be used as part of the migration of the Android repository to mozilla-central.
  • Gabriel Bustamante made Mozilla’s .deb package better integrated with Linux Desktop environment. Now, pretty names are displayed, icons are shows, mime-types are correctly associated (bug 1824327)
  • The Mozilla VPN is now officially part of RelEng’s releaseduty process #releaseduty - Escalating release issues, thanks to coordination between Adrienne Davenport and Johan Lorenzo

Version Control

  • Connor Sheehan made the robots.txt file for hg.mozilla.org much more restrictive in an effort to reduce the load impact of crawlers.
  • Connor Sheehan and Chris Knowles added User-Agent string based banning to hg.mozilla.org to ban service abusers.

mozregression

  • Zeid improved speed of cross-platform requirements generation by running jobs in parallel (bug 1810224)
  • Kenneth improved the UI usability in dark mode (bug 1719424)

Thanks for reading and see you next month!