QA/TDAI/NewFramework

Overview

The "New Framework" is a test framework that is our solution to many of the problems we are facing as Mozilla transitions from a single product to multiple products and many platforms.

Terminology

• TestServer - Server which manages queuing up tests, inventory of devices to test, status of devices and tests, and ensuring log files are completed.
• TestAgent - Tool which communicates with the server to get jobs as well as proxy information from the device. TestAgent understands how to talk to the device.
• Device Agent - This is a very lightweight agent which has a lower level communication protocol and works on all platforms with almost no dependencies. This part may be implemented specifically for each platform we support.

Server

The Server is a new concept that replaces tinderbox and buildbot for test automation. When a new build is ready for test, the server is notified by buildbot and figures out what TestAgent will do the work.

The Server will communicate primarily with the TestAgent. The TestAgent will then communicate to its attached DeviceAgents to run the tests. The Server will receive a heartbeat notification from the TestAgent so that it can be updated with the status of the TestAgents' jobs.

TestAgent

The TestAgent is a python program which acts as a tool to:

• download builds and tests when they are ready
• setup web server to serve test pages and data to Device
• upload status and logs to Server
• configure test environment (profile, paths, tools) for the given test set it is working with
• Act as a test harness (or test harness wrapper) to ensure tests are run, not hung, and results are captured.
• communicate with device to:
  • proxy device registration information to the server for bookkeeping
  • push data and binaries to device
  • launch tests
  • collect logs

Many agent instances will run on the same physical machine. There is no requirement for this, but no more than 128 shall run on any given machine.

DeviceAgent

The code that lives on the DeviceAgent will be as lightweight as possible and written in a native language to that device. This removes the requirement for an interpreter, shell, or additional libraries.

The DeviceAgent will communicate via raw sockets to the TestAgent and understand a small set of commands.

Here is the [Device Requirements]

Meetings / Notes

• Any meetings and notes shall go here

Related Work

• Other test harness development