MailNews:Performance Testing:Meeting 2008 01 09

Discussion with Alice from MoCo on 2009/01/08 (Standard8, bienvenu, gozer, asuth present):

Background

Firefox's development of performance tests was initially quite ad-hoc. Tried to do too much at once and this delayed getting any numbers out.

Start small, probably time to startup
- Firefox have found this a useful number and its caught some regressions
Get one test running first, get confidence in the test platform and stability before trying others.

Firefox have set these up to get the minimum variance possible. To do this requires work, but makes checking for regressions much easier.

Dedicated boxes.
- Should be isolated from the network as possible (to avoid network lags/interference)
- Physical boxes are require, synchronisation problems in Virtual Machines cause skews in the numbers.
Suite of test machines.
- Firefox have 3 machines per platform.
  - Gives redundancy if one machine gets its figures wrong.
  - Very important choice - helps to get developers to trust the numbers.
To avoid number drift:
- Reboot machines after each test run. There is a buildbot set that can do this.
- This also helps to reduce the variant in the numbers.
Machines, directories etc to be as clean as possible:
- Check screen resolution
- Must be nit-picky
- Talos reference images are very reduced - minimum apps, items switched off.
- Sometimes it is easier to wipe a box and reinstall if it is playing up rather than investigate the problems.

Suggested that Talos is used/extended:
- Can interact with the graph server
- Can monitor CPU usage & memory of processes
- Has recently been extended for Fennec, so some of the Firefox-specific code has already been removed.
- Can log results locally.

Talos bugs go under mozilla.org/Release Engineering: Talos.

Graph Server bugs go under Web Tools/Graph Server.

Measured 20 times in a row, average median taken.
TB could do different profile sizes, e.g. "install", "lightweight", "medium", "heavyweight".