Auto-tools/Goals/2012Q4

• Ensure B2G has all test automation available to prepare for launch
  • [ON TRACK] Deploy B2G Test Automation into TBPL running on Emulator and Panda platforms. Specifically we are turning on:
    • [DONE] Mochitest-plain (emulator first)
    • [SKIPPED] Mochitest-chome (emulator first)
    • [DONE] Reftest (emulator first
    • [DONE] xpcshell (emulator)
    • [DONE] webAPI (emulator only
    • [AT RISK] webQA - gaia front end tests emulator and panda
    • [AT RISK] gaia dev integration/unit tests emulator and panda
  • [DONE] Maintain B2G dog food update server as well as builds for development and QA while working to hand them off to releng
  • [DONE] Ensure all webAPIs testable through the emulator have comprehensive automated tests
  • [DONE] Continue to aid QA in creating automated tests for B2G
• Expand and stabilize Test Automation on new and existing device platforms for NEON and ARMv6 support.
  • [DONE] Have APC.io (ARMv6)boards smoketest ready - rooted, working, plugged into Autophone
  • [DONE] Monitor deployment of large scale android and B2G panda board testing, aid with bug investigations
  • [DONE] Implement a new pool-based approach for device management in automation to increase stability on all mobile device testing
    • Version 1.1 released and deployed.
  • [DONE] Deploy Autophone (Noah's Ark) mochitest and robocop test abilities.
• Create and deploy performance automation for desktop, fennec, and B2G product benchmarks
  • [ON TRACK] Create Eideticker tests for B2G on panda boards and run them once a day
  • [DONE] Monitor Datazilla webservices and ensure our accuracy in detecting performance regressions on a per-push basis
  • [DONE] Deploy Datazilla per-push UI to make it easier for developers to dig into the causes of performance regressions
• Update our bugzilla instance to 4.2 providing many benefits from UI to helping people keep track of bugs/projects from within Bugzilla.
  • [DEFER] Deploy 4.2 version of Bugzilla
    • Deferred because of risk of deployment during critical b2g week. Rescheduled for early 2013.
  • [DONE] Ensure that bug migration tools are in place for B2G so that we can share bugs with outside services and partners.
  • [DONE] Complete the 'Sandstone' CSS update
    • Core work is done. Will probably need some tweaks as we go along. Regardless, won't be deployed until next year, with 4.2.
  • [DEFER] Rework browserID <--> BMO integration
    • Deferred in favour of some optimizations and github-migration scripts.
  • [DONE] Deliver user and product dashboards
    • As with the Sandstone skin, development is done but deployment delayed with 4.2 until next year.
  • [DEFER] Deploy tracking flags extension database optimization
    • Deferred as 4.2 is not going to be ready in time (new clusters not ready), given that the holidays will start shortly after its deployed. Targetting early 2013Q1.
• Make the process of sheriffing easier by updating TBPL and helping make it easier to handle the intermittent test problem
  • [DONE] Aid with B2G changes so that B2G oranges tracked in Orange factor, B2G tests show up in TBPL
    • [DONE] Add TBPL support for B2G emulator/Panda/Unagi/Otoro
    • [DONE] Make TBPL's log parser compatible with mozharness-style output
    • [DONE] File/track B2G intermittent failures, report issues with mozharness' error detection/reporting & unhide builds/testsuites on TBPL once they pass consistently
    • [DONE] Add the new B2G18 tree to OrangeFactor
  • [DONE] Make it easier to sheriff and diagnose issues through better notification and automatic suggestions of issues for intermittent failures
    • [DONE] Add support for more types of failures to TBPL's log parser
    • [DONE] Improve test harness & buildbot output for failures & make the output more compatible with TBPL's parser
    • [DONE] Improve TBPL's bug suggestions for failures
    • [DONE] Make TBPL's tree open/closed status update more frequently
    • [DONE] Resolve timeouts during TBPL log parsing/summary generation
    • [DONE] Simplify the 'TBPL job -> recent history for that machine' workflow
    • [DONE] Obtain BuildVPN access & investigate BuildVPN-only tools and their use to sheriffs
    • [DONE] Simplify the TBPL -> 'viewing job on the buildbot master' workflow
  • [DONE] Split mochitest browser-chrome from mochitest-other to improve parallelism and overall turnaround time.
  • [ON TRACK] Make orange factor update more than once a day to help make it easier to see changes over time
    • Patch in progress, see bug 797811.
  • [AT RISK] Assist with getting Datazilla's UI integrated with TBPL, phase 1
    • Awaiting finalised datazilla/talos output format, see: bug 816634 & bug 794901.
  • [DONE] Reduce regression windows for tree bustage by switching off unused builds/tests to reduce coalescing
  • [DONE] Reduce TBPL & OrangeFactor load on BzAPI/b.m.o, by updating use of BzAPI & transitioning to keywords rather than whiteboard annotations
  • [DONE] Fix our test-harness detection of crashes on Android
  • [DONE] Make it easier to differentiate open/resolved/test-disabled bugs on OrangeFactor, to improve prioritisation as well as finding potential backport candidates
  • [DONE] Track down intermittent-failures missing from OrangeFactor
  • [DONE] Make the tree-closure hook fail closed if treestatus is down
  • [DONE] Increase the SnR of dev.tree-management regression emails by suppressing small percentage changes & cleaning up the subject lines
• Expand and Modernize the QA Automation framework as well as make it easier and more clear how to write automated tests for QA.
  • [DEFER] Re-organize and improve the automated testing documentation at MDN
  • [DONE] Develop first round of unit tests for WebRTC
  • [DONE] Ensure all B2G device WebAPIs testable through emulators have comprehensive low level test coverage

• Projects Defered due to pivoting hard toward B2G
  • [DEFER] Port Eideticker to win32 and run tests there once a day to measure win32 snappiness for desktop
  • [DEFER] Create pageload benchmark with networking shaper tests so that we can adequately track user-perceived performance across all devices and network types
  • [DEFER] Complete REST API integration with Bugzilla
  • [DEFER] Update selenium grid to use ESX on-demand to improve web QA capacity - start date pushed back due to B2G
  • [DEFER] Finish and deploy mozmill 2.0 changes to frameworks and existing QA tests - start date pushed back due to B2G
  • [DEFER] Autophone goal scaled back and started later
  • [DEFER] Talos work scaled back as well.