Well, the world didn't end, so we are going to keep rocking automation into 2013!

For Q1, we'd like to take some time to focus on some back-to-basics stuff so we can spend the rest of the year building atop a solid foundation. That said, there is front-line automation that we will continue needing to deliver for performance and the various mobile products.

For some inspiration on this please see our proposed goals and gps's wishlist

High Level Goals

This is based on the below projects

• Goal: Continue to deliver B2G automation support for in house developers and partners - jgriffin & team
• Goal: Create new Performance measurement tool with performance team to measure startup using the best of our new tools - mozharness/mozbase/datazilla. - jmaher & perf team
• Goal: Support next generation android efforts by getting Android x86 virtualized environments to run mochitests - bc
• Goal: Improve test slave capacity by allowing linux desktop tests to be entirely virtualized for tests that do not need hardware support. - jmaher & Releng
• Goal: Build to support future capacity for workflow and performance by:
  • Deploying Bugzilla 4.2 - dkl, glob
  • Publicizing and starting on an effort to revamp TBPL - jeads, edmorley & team
  • Updating test harnesses to use mozbase and mozharness and creating a migration plan for moving to better test reporting across the board (migration planned for Q2 to coincide with TBPL v2 delivery) - jhammel & ahal
  • Revamp GoFaster measurement tools - camd
  • Create plan for triage workflow and begin triaging according to it - lizzard
  • Make all QA centered automation ready for rapid betas to allow the desktop team to work faster - Henrik & team

Building Foundations

Bugzilla

• [DONE] Deploy Bugzilla 4.2 - glob
  • Depends on IT
• [MISSED] Deploy tracking flags optimization - glob
  • Depends on IT
  • NOTE: The 4.2 deployment came too late in the quarter to allow us to schedule the migration. BMO team is working on a way to allow new flags to use the new system but defer the migration of existing flags until we can arrange a suitable maintenance window as well as attempting to decrease the time the migration will take.
• [DEFER] Improve splinter - glob/dkl
  • NOTE: Some improvements did land, but the bulk of the work will be spread over the next two quarters.
• [DROPPED] Make BMO work with Zeus's caching mechanism - glob
  • Depends on IT
  • NOTE: Investigations showed that static assets already have the appropriate headers and would not benefit from caching by Zeus.
• [DONE] Create plan for user workflow integration into Bugzilla - dkl
• [MISSED] Add ability to unsubscribe from emails for specific bugs - dkl

TBPL & Sheriffing

• [ON TRACK] Create the "Single Source of Truth" data extraction layer and deploy into buildbot - jeads
  • Depends on Releng
• [DONE] Create the Requirements and Back End design for TBPL v2 - jeads, edmorley & team
• [MISSED] Complete 3 sprints on TBPL v2 code (estimated delivery end of Q2) TBD: Decide what features will be completed - jeads & team
• [AT RISK] Investigate improving test harness logging and create plan for structured data/log formats - create ability to hang rich meta data (Cycle counters, memory usage etc) in test results. Deliver transition plan from current logging insanity into a modern results reporting situation, optimize for machine readable output that can be transformed for humans - ted & bc

Performance

• [ON TRACK] Complete SfN: Analyze & Improve each page's data for each platform and branch in Datazilla to validate method for regression detection so that we can switch off Graphs.m.o sometime in Q1/Q2 - jeads, jmaher, jhammel
  • Depends on Metrics

Build/Test Capacity

• [DONE] Regenerate the GoFaster dashboards to understand where our time goes in automation - camd
  • Depends a little on Releng
• [AT RISK] Complete the manifests for mochitests so we can provide future versatility on that harness - ted
• [DONE] Get linux desktop tests running in EC2 VMs (for tests that don't require hardware) - jmaher & rail
  • Depends heavily on Releng
• [DEFER] Work with Releng to deliver first steps to make buildbot scheduling more configurable to support "bisect in the cloud" - Automatedtester
  • Depends heavily on Releng

General Automation

• [AT RISK] Complete the move to Python 2.7 on all build and test automation harnesses on current branches (not ESR, release, and beta) - jgriffin
  • Depends heavily on Releng
• [DONE] with Releng) -- Deploy mozharness for all desktop test automation to m-c - Releng
  • Depends heavily on Releng
• [DONE] Start moving mochitest atop mozbase
  • [DONE] Add mozbase tests to 'make check' - jhammel
  • [DONE] Remove mozinfo and manifestdestiny duplication - jhammel
  • [DONE] Figure out a path for replacing automation.py - jhammel
  • [AT RISK] Make it possible for mochitest to consume mozbase packages - jhammel
  • [DONE] Finish documenting mozbase modules: mozprofile, mozdevice, and mozrunner - wlach
  • [AT RISK] Develop remote mozrunner API - wlach
  • [AT RISK] Move B2G mochitest atop mozbase - ahal
  • [AT RISK] Re-tool mozprocess to have a more sane API - ted
• [DONE] Set up dev/staging/prod machines on EC2 for web tools and deploy at least one web tool (e.g. s1s2, eideticker, or gofaster) - ctalbert & glob

Automation Development

• [DONE] Complete and test pulse integration with Mozmill CI for rapid beta readiness - Henrik
• [DONE] Continue working on further WebRTC tests as fake devices come online - Henrik
• [DONE] Have SV team ramp up on Gaia UI tests (marionette) with the goal of completing five tests this quarter - Daniella
• [DONE] Continue expanding/supporting on-phone B2G automation - davehunt, rwood
• [CARRY OVER] Complete Documentation overhaul - davehunt

Triage

• [DONE] Create and announce early bug lifecycle plan for triage - lizzard
• [DONE] Identify and drive creation of triage workflows into the plan for user workflow integration for Bugzilla - lizzard
• [DONE] Do "testing" of triage plan by triaging several components using new plan, tweaking as necessary - lizzard

Front Line Automation

B2G

• [DONE] Gaia QA tests on panda - mdas
• [DROPPED] B2G Eideticker performance tests on panda - wlach
• [DONE] camera-based B2G Eideticker performance tests on unagi - wlach
• [DONE] Get B2G update tests running on an unagi on Jenkins - davehunt/marshal_law
• [DEFER] Gaia JS integration tests on panda - mdas
• [DONE] Provide performance metrics requested by the Gaia team - davehunt/ted/jgriffin
• [DONE] Implement a full range of touch-related methods in Marionette - yiming/mdas
• [DEFER] Gaia Unit tests on TBPL - jgriffin
• [DROPPED] Speed up emulator tests by reducing setup times, integrating with goFaster - ahal/camd
• [DEFER] Enable more mochitest and reftests - jgriffin, ahal
• [DONE] Add support for crashtests - ahal
• [DONE] Add support for running mochitests on the B2G desktop build - jgriffin
• [AT RISK] Integrate better with TBPL in terms of error handling and failure reporting - jgriffin
• [AT RISK] Make B2G harnesses more robust by utilizing mozbase, writing new test runner - jgriffin
• [DEFER] Work with B2G partners and help support them as they start running our automation; improve related documentation - jgriffin, mdas, rwood
  • NOTE: We created tests that the partners thought important - perf tests, endurance tests etc. But there was no clamoring need for partners running our automation so we helped where we needed to and focused on other higher priority items.
• [DONE] Continue to support WebQA and partners in writing Gaia tests - davehunt, rwood

Mobile

• [DONE] Improve MozPool so it can be used to help us manage the influx of mobile platforms - should work seamlessly with android and b2g - and should be able to handle different platforms of android (4.x on x86 (tbd) or 2.2 on ARM (tegra) or 4.x on ARM (panda) etc) - mcote & dustin
  • Depends on IT (specifically - Dustin & Dividehex)
• [AT RISK] Get Android x86 VMs or Android x86 Emulators running mochitests (m1-8) solidly enough that buildbot integration can occur - bc
• [ON TRACK] Do preliminary groundwork with partners to verify an android x86 devboard solution - no devboards delivered to automation this quarter). - bc

Bisect in the Cloud v1

• [DONE] Deliver plan and requirements - AutomatedTester
• [DEFER] Build v1 of tool using existing mozregression framework - AutomatedTester
  • Ideally, use more flexible buildbot scheduling, if that falls through, then use hourly builds
  • Depends on Releng

Performance

• [MISSED] With performance team -- create new system (based on mozbase and datazilla entirely to free our hands from the old talos constraints) for startup measurement on desktop - jmaher
  • NOTE: The perf team went MIA on us this quarter. But we did get interest in specific tests and microbenchmarks and are moving in those directions.

WebRTC

• [AT RISK] Create requirements for building a fully configurable webRTC network automation rig - jmaher & ted
• [MISSED] Determine if we can build a webRTC rig "on the cheap" - jmaher & ted
• [MISSED] Prototype the webRTC rig on the cheap (if possible) - jmaher & ted

Stretch Goals/Proposals

• [DONE] stretch: Build pseudo-automated Eideticker-with-camera rig for user perceived performance testing of devices without HDMI out. - wlach/yiming?
• stretch: Create a mechanism to do automatic bisection of regressions found by Eideticker - wlach
  • Wlach started on this.
• Stretch: Deploy mozpool for on device testing in haxxor to flesh out further on-real-device testing abilities for both android and b2g - mcote
  • NOTE: We actually decided against doing this at this stage and focused instead on autophone improvements that have really paid off.