Purpose

Litmus produces a continuous flow of Failed and Unclear test results. These reports need to be regularly reviewed and appropriate action taken on them to ensure the value of testing done through Litmus remains high. This page will explain how to triage incoming test results from Litmus that are Failed or Unclear.

Responsibility

Since Litmus is a regression testing tool, test cases with results of Pass are generally not in need of review. However, test cases that are reported as Failed or Unclear need to be regularly looked over and appropriate action taken for each report. It is the Litmus QA lead's responsibility to review and triage results reported in Litmus. It is the Litmus lead and the subgroup owners responsibility to take action on the Valid failed and unclear reports. The following is a guideline to help Litmus Administrators facilitate this process.

Finding result reports

Places to find results reports:

• A "[litmus] Daily Report - <date>" email sent each morning for reports from the previous day
• A link at the bottom of most litmus pages can be found for recent failures and recently unclear results.
• A Results Triage Tracking Page is available to link to various Litmus queries for tracking triage progress. This page links to vetted and unvetted lists. Unvetted results are the ones that need to be reviewed.

Review Failed results daily

Once Failed results are found from the above formats, review each result by clicking the link to the result report. For each result reported:

• Confirm or deny the validity of the result. Results are automatically marked as Valid. It is up to the reviewer to determine which results are actually valid or invalid as Failed. Results filed with an associated bug are likely to remain valid. Results filed with good comments should be closely reviewed, even tested to confirm validity. Results filed with no comments or associated bugs can be subjectively reviewed as to severity and known frequency of report.
  • If a result is valid as Failed:
    • Make sure there is a bug on file for it in bugzilla.mozilla.org (check for one already filed, if not, create a new bug).
    • Then leave the Valid?: box checked in the result report.
    • Add a bug reference to the report under referenced bugs. Enter the bug number, then click Add Bug References
  • If the result is invalid, un-check the box in the Valid?: field of the result.
• Vet the result. Vetting is a way to mark results as having been reviewed and action taken. Once you are done with result validation and action taken, click the Vet Result button in the result report.
• Add a comment as to the action taken, then click Submit Comment

If the day of triage shows just a few valid reports, the lead can usually take on bug filing. However, if there are a lot of valid failed results, the Litmus lead should forward the results to the appropriate subgroup owner for them to take action. In which case, it will be the subgroups owners responsibility to file the bug then Vet the result.

Review Unclear results at least once a week

Once Unclear results are found from the above formats, review each result by clicking the link to the result report. For each result reported:

• Confirm or deny the validity of the result. Results are automatically marked as Valid. It is up to the reviewer to determine which results are actually valid or invalid. Results filed with good comments should be closely reviewed, even tested to confirm validity. Results filed with no comments can be subjectively reviewed as to severity and known frequency of report.
  • If a result is valid as Unclear, the action to be taken is to edit the test case associated to the report and make it clear.
  • If the result is invalid, un-check the box in the Valid?: field of the result.
• Vet the result. Vetting is a way to mark results as having been reviewed and action taken. Once you are done with result validation and action taken, click the Vet Result button in the result report.
• Add a comment as to the action taken, then click Submit Comment

If the day of triage shows just a few valid reports, the lead can usually take correcting the test cases. However, if there are a lot of valid unclear results, the Litmus lead should forward the results to the appropriate subgroup owner for them to take action. In which case, it will be the subgroups owners responsibility to edit the test case to make it clear.

Result of this process

• More bugs filed on broken features
• Clearer test case suites.