Welcome to the June (and first) edition of the Engineering Effectiveness Newsletter! The Engineering Effectiveness org makes it easy to develop, test and release Mozilla software at scale. See below for some highlights, then read on for more detailed info!
Highlights
- Content process crashes generated via Windows Error Reporting are now being intercepted!
- Early data suggests around 10% of crashes were previously being missed
- Many new bugs being found by these crashes, including security sensitive ones
- We are possibly the first FOSS project to do this
- The minimum version of Python is now 3.6, no more Python 2!
- All mach commands were converted to Python 3
- Feel free to start using 3.6+ features like f-strings and async/await in your Python
- A new
about:third-partypage has been created for shedding light on code injected by third parties- Bugs in these modules often get blamed on Mozilla
- Previously these were invisible to users, now they can be used to help diagnose support issues
- Google’s Fuzzilli (a coverage-guided fuzzer) has been integrated into our JS engine and deployed on our fuzzing infrastructure!
- Will help find additional bugs and vulnerabilities in the JS engine
- There are some exciting changes coming with respect to lint and formatting in Phabricator!
- Automatic code formatting on landing, currently only Python is supported but support for more languages coming in H2
- Warnings are now displayed by reviewbot. These are optional fixes which won’t cause a back out. This is the first time they are being surfaced.
- Speaking of Phabricator, the e-mails it generates have gotten a major overhaul!
- They’re now less noisy, less cluttered and packed with only the most relevant context, so you can tell at a glance exactly what is needed from you
- Now enabled as the default setting
Detailed Project Updates
Bugzilla and Bugbug
- marco and jcristau made some improvements to autonag / bugbug
- Better support for detecting regression bugs
- Catching some security fixes that were escaping the uplift and advisories process
- ksenia added the ability to build, deploy and serve bugbug machine learning models on GitHub data
- dkl added the ability for privileged users to update triage owners in Bugzilla
- This means teams can self-serve triage ownership changes, which is especially useful if they have a triage owner rotation
- dkl also added client and server side enforcement of triage requirements
- For example, severity will now automatically be set to S2 for crash bugs
- See here for more of the enforced rules
Build System and Mach Environment
- mhentges, ahal and others finished migrating all mach commands to Python 3
- The minimum required Python version was also upgraded to 3.6!
- Thanks to Surajeet who made many contributions removing Python 2 compatibility shims
- mhentges also added the ability to source dependencies from PyPI via virtualenv_requirements files
- This is an early step towards a larger goal of re-working how Python dependencies are managed in mach commands
- glandium turned on –enable-bootstrap by default
- This means toolchains and other build dependencies will automatically be updated without the need to run
mach bootstrap
- This means toolchains and other build dependencies will automatically be updated without the need to run
- andi enabled telemetry to determine the number of developers that use VSCode over ssh
- mhentges removed ~150,000 lines of unnecessarily vendored Python tests and metadata!
CI and Treeherder
- marco improved the test selection algorithm and enabled it for mach try
- Running
mach tryshould now provide much better results by default
- Running
- sclements worked with oremj to migrate Treeherder from heroku to gcp/cloudOps with very little down time
- sclements also hit a milestone with Push Health
- Push Health is a developer-friendly alternative to Treeherder’s default jobs view
- Bug reports and feature requests welcome!
- andi added Firefox builds using Clang trunk for linux64
- These are currently running as a daily cron task on central
- This will allow us to identify issues caused by upstream clang changes much sooner
- andi also added awsy and browsertime performance tests to monitor regressions in Firefox built with clang-trunk vs the official clang toolchain
- Thanks to masterwayz’s contribution to finish migrating tests from OSX 10.14 -> 10.15
- jmaher moved the majority of Windows 7 32 bit tests to Windows 10 32 bit
- jmaher also supported the WebRender team in their efforts to expand test coverage
- As well as starting to turn off non WebRender tests
- ahal ported the standalone taskgraph module to Python 3
- Fenix’s taskgraph has also been migrated, but other consumers still left todo
- ahal also enabled many more tasks in support of Fission
Crash Management
- gsvelto enabled interception of content process crashes via Windows Error Reporting
- Around 10% of crashes were being missed, no longer!
- OOM crashes intercepted via WER are now properly flagged
- All crashes intercepted via WER are now labelled as such
- gsvelto also added names for Windows error values to the oxidized stack walker
- msirringhaus added recording native thread names in Linux minidumps
- gankra has added support to the crash reporter for using macos’ compact unwinding info
- Ideally this will result in better stack traces on macos
- Results currently mixed, have found at least one bug in the implementation
Fuzzing and Sanitizers
- decoder integrated fuzzilli (a coverage-guided JIT fuzzer by Google P0) into our JS engine and deployed it in our internal fuzzing infrastructure.
- tsmith added support to handle OOMs and hangs to Grizzly
- Existing fuzzers using Grizzly will get this automatically
Lint, Static Analysis and Code Coverage
- Thanks to dnaka91’s contribution which adds support for grcov to output coverage badges
- Great for projects who want static coverage reports without relying on external services.
- Thanks to geovie’s contribution implementing the Cobertura output format for grcov
- Also thanks to svenstaro’s contribution making it more readable
- As well as svenfoo’s contribution with a fix for larger projects
- marco updated code review bot to show all warnings found by mozlint-based linters
- Since warnings are optional, this is the first time they are being surfaced to developers by default!
OS Integration and Security
- tkikuchi added a new about:third-party page to visualize third-party modules injected into Firefox processes
- flopez also helped update SUMO docs to explain how to use it
- This sheds some light on previously invisible modules injected by third parties
PDF.js
- bdahl added support for the standard 14 fonts (which are not always embedded in PDF files)
- bdahl also added support for tagged PDFs to improve screen reader accessibility
- calixte added support for XFA forms
- These are still used by several governments (like Canada)
- Test it out by setting “pdfjs.enableXfa” to true in about:config.
Phabricator, moz-phab, and Lando
- mhentges deployed the new Phabricator Emails implementation to all developers
- This provides much clearer and more actionable e-mails
- zeid worked with ops to enable continuous deployment of Lando
- Will be used as a model for other applications
- sheehan implemented automatic formatting of code at landing time
- Initially limited to Python, but more languages on the way in H2
Release Engineering and Management
- bhearsum and aki got Mozilla Online Fenix builds working in automation
- These will be shipping in China
- aki automated l10n cross-channel
- This aggregates all shipping en-US Firefox and Thunderbird strings for localizers to translate
- aki mitigated multiple nightlies off the same revision
- pascal successfully managed the release of MR1
- sclements made some UX improvements to shipit that surfaces metadata about releases
- More work here to come
Other
- Engineering Effectiveness is hiring! If you know of anyone who would be a great fit for any of the following roles, consider referring them for a bonus!
Thanks for reading and see you next month!