Automation

Every day we have automated tests which run against new builds, these tests are reported to couple of different dashboards. Firefox Nightly and Aurora are reported here, Firefox Beta and Release Candidate tests are reported here here. The results are triaged by the Automation Development team and failures are reported to Bugzilla as Mozmill Tests bugs. If a failure is found to be due to a Firefox regression QA will assist with the investigation.

Roles and Responsibilities

QA Lead

• Ensure automation reports and bugs are properly tracked in your test plan (see below for template)
• Ensure affected bugs remain in progress
• Ensure fixed bugs are verified against the recent reports, and reopened if they continue to fail
• Ensure tests which are consistently failing on Beta and Release are disabled
• Ensure requests for regression windows are assigned and followed-up quickly by the testing team

Testers

• Report automation bugs, or update existing bugs, when a new failure is reported
• Assist with investigations of test failures which are identified as Firefox regressions

Meetings

• N/A - there are no meetings necessary to attend for this subject area

Daily Checks

1. Review the fixed bugs and make sure they don't continue to fail

2. Review the affected bugs and ask for a status update if there's been no progress recently

3. If on Beta, review the affected bugs and ask for the test(s) to be disabled

Test Plan Template

Automation Bugs (live example):

<bugzilla>
{
 "include_fields":"id,summary,cf_status_firefox31",
 "component":"Mozmill Tests",
 "f1":"cf_status_firefox31",
 "o1":"nowordssubstr",
 "v1":"---"
}
</bugzilla>