Dial in

 # Vidyo:  tor-spadina
 # Phone:  650-903-0800 or 650-215-1282 x92 Conf# 95589 (US/INTL)
 #         1-800-707-2533 (pin 369) Conf# 95589 (US)
 # IRC:    irc://irc.mozilla.org:6697/#ateam

Previous Action Items / Catch Up

Problem Definition

Autophone is a framework to run Python-based tests on consumer mobile devices, starting with phones. Starting projects are start-up timings and a subset of the unit tests.

There are all kinds of difficulties achieving this. Noteworthy ones include

• devices fall off the network sometimes
• devices freeze or power down
• s1s2
  • generally, each iteration of a test (kill then relaunch fennec with a particular URL) takes a little longer than that the last
  • are we sure we're even measuring something useful? Admittedly we do appear to have noticed one or two real regressions (I think?)
• unit tests
  • even a subset of unit tests (crash tests, jsreftests, robocop) can take so long on some phones that keeping up with mozilla-inbound builds is impossible.
• devices change behavior after running for a period of time going from reliable with consistent results to unreliable with inconsistent results.
• message queues can become full resulting in an inability to query and update device states.
• lacks the ability to recover pending jobs after a restart or other abnormal termination. A related issue is the inability to suspend autophone operation temporarily while maintaining the job queue.
• device instability can lead to unprocessed jobs by particular devices which required 'repair' or 'reconfiguration'.
• use of the Côté Technique (dynamic import) to import test scripts means that autophone must be restarted to pick up changes to test scripts.
• does not have a clean shut down procedure which also terminates the child worker or other (xpcshell) processes.
• requires each phone to process each job rather than partitioning jobs between identical devices. This would help in cases where a single phone can not perform unittests on each build in a reasonable time, but if multiple identical phones could work simultaneously on different jobs they could keep up with the incoming work load.

Presentation Notes

Discussion

Action Items