QA/Automation/Projects/Mozmill Automation

From MozillaWiki
Jump to: navigation, search


Overview

Mozilla QA is using the existing Mozmill tests to run automated tests against any kind of Firefox build. While this is reducing the time we have to spent on manual testing, we can also enhance our testing to more than only Firefox BFT tests. We have automated test-runs available for software update tests, l10n tests, add-on tests, and others. Those are getting run on any platform, and also for localized builds of Firefox.

Name: Mozmill Automation
Leads: Henrik Skupin
Contributors: n/a
Etherpad: http://etherpad.mozilla.com:9000/qa-mozmill-automation

Sub Projects

Sub Project Status Description
Mozmill CI Active Perform fully automated Mozmill testruns via Jenkins CI
PuppetAgain Active Puppetizing Mozmill CI machines
Mozmill Environment Active Supply a pre-configured environment for Mozmill tests execution
Mozmill CI Configuration Generator Active Application to generate valid configuration files for Mozmill CI on-demand jobs
Mozmill 2 scripts Finished Update automation scripts for Mozmill to support Mozmill 2.0
On-Demand Test Framework Finished Implementation of a remote mechanism to trigger Mozmill functional and update tests for Firefox releases
Triggered l10n Test-runs Finished Perform l10n related tests triggered by landing of localization changes
On-Demand Testing Finished Streamline how testing is performed on releases

Details

The way to reach the state when everything is automated is long can can be divided into the following sub-projects:

  • Setting up a machine which can be used for automated tests
  • Creation of scripts which cover all automation aspects
  • Running the automation for release builds

Machine for Automated Testing

A machine in the QA lab is needed at least for the first time. Once we have a stable process we can consider to integrate the automation into the Releng process.

This sub-project has been finished. All requirements can be read on bug 537840

Automation Execution

Right now Mozilla QA is using the machine in the QA lab to run release tests. All tests have to be triggered manually. We consider to use Hudson in the future to control the complete process.

Automation Scripts

The automation scripts should allow us to have a complete automated process in-place. That means that only a trigger would be necessary to start those release tests. The following areas have to be covered:

Downloading builds from the FTP Server

For a complete automation of our Mozmill tests for release testing, a download script is necessary which will fetch all the builds to test from the FTP server and store those locally. The work is being tracked on bug 528064.

Smoketests, BFT, and FFT Test-run

To execute all existing Mozmill tests which have been written for the Smoketest, BFT, and FFT testgroup of Litmus this script is useful. Once it has been called it should automatically execute the steps below:

  • Install a build (if requested)
  • Run the tests from all those three testgroups
  • Uninstall the build (if requested)

The work is being tracked on bug 563523.

Software Update Test-run

To run the software update tests automatically a test-run script is required which will take care of any aspect of the tests to be run. Once it has been called it should automatically execute the steps below:

  • Install a build (if requested)
  • Create a backup of the build
  • Run the partial or complete software update tests
  • Restore the backup
  • Run the fallback software update tests
  • Restore the backup
  • Run background software update tests
  • Uninstall the build (if requested)

A script with this feature set has already been created on bug 504653 and is able to execute all the steps except step 6. The reason is that we do not have any automated tests for background software updates yet.

Add-on Test-run

The script for the add-ons test-run simplifies the work which would have to be done to run Mozmill tests against available add-ons. Once it has been called it should automatically execute the steps below:

  • Install a build (if requested)
  • Perform Mozmill tests against any available add-on
    • Download the specified version of the add-on
    • Start Firefox with the extension installed
    • Run all available tests
    • Clean-up the system
  • Uninstall the build (if requested)

The work is being tracked on bug 562445.

L10n Test-run

With the creation of l10n tests we will be able to perform specific tests for localizers. Once it has been called it should automatically execute the steps below:

  • Install a build (if requested)
  • Run all the l10n tests
  • Uninstall the build (if requested)

The work is being tracked on bug 565196.

Accessibility Test-run

With the creation of accessibility tests we will be able to perform specific tests for keyboard navigation. Once it has been called it should automatically execute the steps below:

  • Install a build (if requested)
  • Run all the accessibility tests
  • Uninstall the build (if requested)

The work is being tracked on bug 565201.

Retrieved from "https://wiki.mozilla.org/index.php?title=QA/Automation/Projects/Mozmill_Automation&oldid=1014090"