This document is currently in work in progress

Current wording

• urgent job === smoke
• non-urgent job === others

Processing an automation report

• As we currently don't have automated email, you will need to fetch all the HTML report for every Jenkins job.
• Follow the
• Send out the email report.

Finding a failure in an automation report in an urgent job

General steps

• Immediately disable the test per the disable/xfail doc. This is the top priority.
• File or update a product bug if it reproduces manually, or an automation bug if it doesn’t, per the bug filing doc.
• Add the failure bug to the next standup highlights, at the standup notes

In details

Finding a failure in an automation report in a non-urgent job

• Immediately disable the test per the disable/xfail doc. This is the top priority.
• File or update a product bug if it reproduces manually, or an automation bug if it doesn’t, per the bug filing doc.

Starting work on a bug

• Assign yourself and fill the QA whiteboard to make it appear on the sprint page
• Investigate if it's either an automation issue or an infrastructure one.
• If the issue can be fixed, when your patch is ready, check Treeherder report before going any further.
• If not, nominate it for the flaky suite
• Ask 2 reviewers from either the core team or contributors. Only one is needed for enabling/disabling tests.

In details

Finishing work on a bug

• Either the second reviewer or you could ask Autolander to merge your patch.

Priorities

• Any urgent test failing due to automation issues should be fixed ASAP
• Any non-urgent, non-intermittent automation bug should be fixed within a week
• Any test that proves to be newly intermittent should be reviewed at next triage
• Any test failing more than 50% should be disabled and put on backlog

Transition plan

In order to get to the escalated jobs suite, here's how we'll proceed to reduce the daily human intervention:

1. Reduction of the length of the automation report (see the new template)
2. Configure Jenkins to send out email on every single failures (use this plugin to configure the content of the email.
3. Stop to send out the manually written automation report. Product bugs will be covered in the general daily report.
4. Split the jobs between flaky/non-flakies.

TODO once Jenkins is able to send out emails

Receiving an urgent failure email