Engineering Effectiveness Newsletter (November 2021 Edition)

November 2021

Welcome to the November edition of the Engineering Effectiveness Newsletter! The Engineering Effectiveness org makes it easy to develop, test and release Mozilla software at scale. See below for some highlights, then read on for more detailed info!


  • A massive refactor around how Mach manages its Python environment is wrapping up.
    • Separate virtualenvs for individual commands and checkouts.
    • No more failures due to a Python dependency being the wrong version.
    • No more mach create-mach-environment.
    • Significantly eases the burden on tool authors.
  • The stack walker used in CI and on developer machines has been re-written in Rust.
    • The new one is faster, more flexible, more robust and provides a human friendly output format!
    • Will soon be used by as well.
  • Explorations into Intel CET (Control-flow Enforcement Technology) are yielding security bugs.


  • Jane Kotovich (janekotovich)
  • Sivadeilra
  • Devon Burriss (dburriss)
  • Giovanni Tangredi (giovannitangredi)
  • Luca Barbato (lu-zero)
  • Marco Ballario (marco-ballario)
  • masterwayz

Detailed Project Updates

Bugzilla and Bugbug

Build System and Mach Environment

CI and Treeherder

Crash Management

  • Aria Beingessner rewrote the stack walker we use in CI and on developer machines to Rust.
    • The new one is faster, more flexible, more robust and provides a human friendly output format!
    • She is also deploying it to Socorro with the help of willkg and will soon be used by
  • Calixte documented Clouseau, a tool that helps quickly triage large numbers of crashes.
    • If you find yourself triaging crashes often, give Clouseau a shot!

Fuzzing and Sanitizers

  • The fuzzing team has been exploring Intel CET (Control-flow Enforcement Technology).
    • This has been finding real security issues in the field and bugs are being filed.
    • Team is looking for ways to make these types of issues easier to pull out of crash telemetry.

Lint, Static Analysis and Code Coverage

OS Integration and Security

  • gcp updated jemalloc to use guard pages before the allocation area.
    • This helps prevent buffer underrun exploits.
  • Nika Layzell joins the team and will be working on improving our IPC infrastructure.


Phabricator , moz-phab, and Lando

Release Engineering and Management

  • Release Engineering would like to:
    • Welcome Gabriel Bustamante who joined full time last week
    • Welcome chidinma who will be improving test migration tooling as part of Outreachy
    • Say farewell to bhearsum who has moved to the Desktop Installation team
  • Release Management would also like to welcome Donal Meehan!
  • UpdateVerify now pre-fills the cache asynchronously thanks to hneiva!
    • This change improved UV performance by ~19%, saving ~8 minutes on average for each ongoing beta and paves the way to refactor UV to async Python.
  • sclements is moving Focus for Android releases to shipit.
    • This will standardize the release process and make lives easier.
  • sclements and others got the ball rolling on standalone taskgraph docs.
    • These docs will focus on lowering the barrier to entry for new users.

Thanks for reading! We’ll be taking a break this month, so we’ll see you next year for the combined December/January edition.

1 Like