Auto-tools/Meetings/2013-04-29

From MozillaWiki
Jump to: navigation, search

Notices, Highlights, Roundtable

  • 1/3 of the way through the quarter. Any goals in trouble?
  • [bc] I'd like to discuss some ideas for statistical methods for obtaining reliable performance measurements.
    • Background discussion bug 866195 Improve Autophone s1s2 test reproducibility
    • Using mean and standard error of the mean rather than mean and standard deviation of measurements.
    • Choosing number of measurements and a target percentage standard error of the mean.
    • Rejecting and re-testing measurements with overly large percentage standard error of the mean.
    • Filtering outliers using quartiles: http://en.wikipedia.org/wiki/Interquartile_range
    • Choosing a device settling time before starting tests.

Goals

Deep Dives

Mobile (1st Monday)

AutoPhone/Noah's Ark

[bc] Working on bug 866195 Improve Autophone s1s2 test reproducibility

B2G Automation

Mobile Evangelism Site-Scraping Tool

Android Automation

  • new sutagent/dm changes to support fix for process monitoring. This is root cause of 2400 second timeout and should allow robocop on pandaboards to be visible and much greener
  • work towards moving webgl into separate suite to green up M1 and hopefully the newly created Mgl
  • x86 automation via emulators and phones is in progress

Tools (2nd Monday)

Bugzilla

Bughunter

  • [bc] Need to catch up on filing bugs.

GoFaster

Pulse

MozTrap

Tinderboxpushlog

War on Orange

  • OrangeFactor on trunk for last 7 days: 7.78
  • [edmorley] Tested/gave feedback on the new OrangeFactor UI layout in bug 738612.

TBPL Version 2

  • [edmorley] Collated the initial treeherder reference data.
  • [camd] branch for Job data ingestion was merged in

Performance (3rd Monday)

New Graphserver (Datazilla)

Signal From Noise

  • solved ingestion problem
  • work on time based ui underway
  • work on alert system and integration with email client

Speedtests

Eideticker

  • [wlach] Comparative pageload tests on Android
  • [wlach] Work on integrating b2g unagi tests into dashboard as proof-of-concept

Talos

General Framework Support (4th Monday)

MozBase

In-tree Automation/Harnesses

  • [edmorley] Patch to tweak several jsreftests to avoid TBPL parser false positives (bug 864694).
  • [edmorley] Patch to make auto clobbering opt-in rather than opt-out & increase test coverage (bug 863091).
  • [edmorley] Reviewed bug 822821 and bug 813022.

WebRTC Test Development

Process (4th Monday)

Sheriffing & Trees

  • [edmorley, RyanVM] Various discussions on dev.platform about inbound closures, multiple inbounds, wait times & strategies for improving developer efficiency.

Treestatus

Bugmasters

Holidays and Trips

Misc