Auto-tools/Goals/2013Q4

From MozillaWiki
Jump to: navigation, search

Top Level

Highest priority, multi-team dependent goals

Goal Measurement Status Owner Bug Area
Run endurance tests without Marionette issues Run a set of endurance tests without gaia actions (or with simplified gaia actions to test only marionette) for 3 days w/o issues [DONE] Mdas bug 908261 has been marked a dupe, what will help here is bug 931044 which we could hook up to run for 3 days Marionette
Deploy new B2G desktop tests, reftest running OOP, and gaia integration to TBPL All these suites running on B2G Inbound [DONE] Jgriffin bug 916356 B2G
Support B2G Perf Team with automated power testing and Gaia Productivity team with github autoland The power testing goal was deferred due to lack of a suitable device, and the Gaia team didn't end up needing our help with the first iteration of github autoland. [DEFER] Wlach bug 917375, Github autoland bot B2G
Deploy automated B2G Eideticker system to run nightly Successful completion on deadline [DONE] Wlach bug 916989 B2G
Create and Deploy one dashboard for docs, usage statistics, and tools for all our performance tools This goal's priority was downgraded to allow Joel to spend time on higher-priority perf items, including desktop power testing, xperf, and webpagetest. [MISSED] Jmaher bug 919619 Performance
Implement and deploy a fully functional alerting system for performance regressions and data ingestion monitoring. Jeads time was better spent on Treeherder; we will target this goal for Q12014. [MISSED] Jeads bug 921218 Datazilla
Ensure sheriffs can perform side by side usability testing in TBPL Complete starring, orange factor integration, plugin detail pane, (possibly) regex to tag transformation by end of quarter. This should be done, except for starring and orange factor integration. [DONE] Mdoglio Pivotal Tracker Instance Treeherder
Continue Bugzilla as a Platform transition Deploy BzAPI compatibility layer, Change notification POC, Memcache minimum viable product by end of quarter [DONE] (except bzAPI compatibility layer, eta early Q1) Glob bug 885464 Bugzilla
Force Android reftests to run within 15% of their original, pre skia time Ensure panda and x86 reftests run within 15% of their original time [DONE] Gbrown bug 916657 Android
Ensure steeplechase webRTC automation as hands off as possible Deployed mochitests, ability to change machine configuration to system by end of quarter. C++ tests were not created (not our responsibility) so did not get integrated. [DONE] Ted bug 894559 WebRTC
Deploy bugmaster community building tools: bug board, tutorial, and bugzilla user profile report Tools deployed by end of quarter [MISSED] Lizzard bug 919624 Bugmaster
Deploy Mozmill 2.0, Metro infrastructure and tests, and aid SoftVision to take over day to day operations for QA automation with our oversight Deployed projects, tried to get the SV team to take over the reigns, and we had marginal success. Had to bail them out a few times. [DONE] Whimboo bug 922200 QA Automation
Operationalize sheriffing role and complete documentation so that we can pull in community sheriffs Have documentation completed, identify one part time community sheriff. [DONE] Edmorley bug 928960 Sheriffing

Detailed Lists

These contain the top level goals as well as the supporting, ongoing, and additional projects we'll try to complete this quarter. Note that additional projects may be moved to the backlog to cope with changes in priority or new projects. The current backlog is here

Marionette (mdas)

  • [DONE] P0 - Run set of endurance tests for 3 days without marionette issues
    • space issues, memory usage, import log, and don't regress socket.timeout
Ongoing and Supporting Projects
  • P0 - (crossteam)Get passing WebDriver conformance tests running in TBPL
  • P0 - (crossteam)Fix touch related bugs
  • P0 - (ongoing) Continue to support WebQA and Gaia teams with bug fixes and new features

B2G (jgriffin)

  • [DONE] P0 [ahal] (highlight) mochitests, reftests, gaia-ui-tests on b2g desktop builds: debug/opt linux32/64, debug/opt macosx64
  • [MISSED] P0 [ahal] crashtests, jsreftests on b2g desktop builds
  • [DONE] P0 [ahal] reftests/crashtests running out of process
  • [DONE] P0 [jgriffin] gaia integration tests on tbpl
  • [DEFER] P0 [wlach supporting] (crossteam) (perf team) power consumption testing
  • [MISSED] P0 [rwood] (crossteam) memory tests
  • [DONE] P0 [wlach,davehunt] Deploy automated Eideticker system to run nightly for Firefox OS
  • [MISSED] P0 [wlach] Deliver a suite of Eideticker responsiveness tests for FirefoxOS
  • [DROPPED] P1 [wlach supporting] (crossteam)(with gaia team) github autolanding
  • [DONE] P1 [ctalbert coordinating] expanding hardware rig in mtn view - (stephend team, IT) joint with perf system
New Builds and Tests
  • [DEFER] P1 [ahal] tests on 4.2 emulators (https://bugzilla.mozilla.org/show_bug.cgi?id=912249)
  • [MISSED] P1 [ahal] jsreftests running out of process and enabled on all branches (and disable android noion)
  • [MISSED] P1 [jgriffin, ahal] mochitests, reftests, crashtests, jsreftests, webapi, and xpcshell tests on debug emulator builds: linux64
  • [MISSED] P1 [jgriffin, davehunt] get gaia-ui-tests going on emulators on TBPL
  • [DEFER] P2 [jgriffin] run "sanity test" of emulator on osx
Ongoing and Supporting Projects
  • P1 [jgriffin, ahal] continued work to make B2G tests more sheriffable
  • P2 [jgriffin, ahal] (internal) continued burn down of intermittent oranges
  • P2 [ahal] continued work to make B2G tests more developer friendly

Performance (jmaher)

  • [MISSED] Q4 top priority is to have minimal set of documentation (what the tests do, how they are run, what they measure) and usage statistics (number of test runs, results) of our tools (all a-team involved performance tools/projects) available from a single place (scaled down dashboard)
  • [DONE] P0 [jmaher] Complete investigation into Talos suites with desktop performance team
    • tsvg, tscroll, ts, tpaint are all refreshed/updated accordingly.
  • [MISSED] P0 [jmaher, ekyle] Complete alerting analysis for datazilla data
  • [DONE] P0 [jmaher] hold a monthly cross team performance meeting to discuss tests, future projects, problems, relative priority
  • [DONE] P1 [jmaher, bc] Investigate desktop power testing
Ongoing and supporting projects
  • have tools, data, tests fully documented and easy to find (all)
  • proactively meet with at least 5 teams or team leaders to get work in a pipeline instead of an emergency. (jmaher)
  • Ensure JS Team is satisfied with real world benchmarking system, investigate automation
  • Build performance lab
  • Continue supporting games benchmarking effort

Datazilla (jeads)

  • [MISSED] The Q4 top priority is to implement a fully functional alerting system for performance regressions and data ingestion monitoring.
  • Implement alert system that monitors data ingestion rates for all branches in all projects.
    • NOTE: Data ingestion rates should be reported in a graph, this could be part of the performance dashboard, abnormal values should trigger alert emails.
  • Fine tune automated regression detection system as needed
  • Implement the performance dashboard described in the performance section
Ongoing and supporting projects

NOTE: It's possible that slow page load performance in the new UI will require us to implement a reference data cache. Not sure if this will be an issue yet but something to be aware of. NOTE: There are several tasks building up in the b2g performance ui. They're assembled here, bug 913726, not sure how these should be prioritized in Q4.

Treeherder (mdoglio/jeads)

  • [DONE] Ensure sheriffs can perform side by side usability testing in TBPL
  • [MISSED] Implement TBPL *starring* features
  • [MISSED] Implement Orange Factor compatability
  • [MISSED] Release production instance on AWS
  • [DONE] Make sure we're on target for full TBPL replacement in Q1 2014.

Bugzilla (glob/mcote)

  • [DONE] Q4 top priority is to continue to focus on "Bugzilla as a Platform". This is a multifaceted goal covering API design, documentation authoring, and community engagement, and of course development.
  • Continue Bugzilla as a Platform (BaaP):
    • [DONE] [mcote] Proof of concept of the Bugzilla Change Notification System (with security approval)
    • [MISSED] [dkl] (crossteam) bzAPI compatibility layer
    • [MISSED] [dkl] plan for versioning APIs, first draft of interface for API v2
      • We had some interruptions due to more API work, developer tools, and antispam efforts.
  • [DONE] [glob] (crossteam) memcached minimum viable product (also helps BaaP above)
Ongoing and supporting projects
  • [DEFER] [mcote] ReviewBoard integration plan based on feedback
    • Discussion is still ongoing as to whether Review Board can meet our needs or not.
  • [DONE] [mcote] somehow get over to git, or at least have a really serious plan for it
    • Plan in place.
  • [DONE] [kyle] deploy new bzETL

Android (gbrown)

  • [DONE] Q4 top priority is remote reftest profiling in support of running reftests in reasonable time on Android 4.0/Panda and Android 4.2/x86.
  • [MISSED] mach commands for running remote tests
    • automatically get xpcshell/xre for remote tests
    • [dminor] Make the tooling around running gdb/jimdb a bit friendlier (mach command)?
  • [DONE] Eideticker responsiveness (possible carry over, depending on Q3).
Ongoing and supporting projects
  • Continue working with releng to increase panda stability
  • [bc] Use Autophone to do one-off cross browser bandwidth metrics
  • [DONE] [dminor] Deploy C++ remote tests

WebRTC (ted)

  • [DROPPED] P0: Get Steeplechase into releng staging (joint with releng)
  • [MISSED] P0: Deploy C++ tests for SteepleChase
  • [DONE] P0: Deploy ability to change machine configuration per test
  • [DEFER] P1: Investigate hardware testing set up (joint with desktop QA)

Bugmaster (lizzard)

  • [MISSED] Deploy bugmaster community building tools (bug board, tutorial, and bugzilla user profile report)
  • [DONE] Use summit to build out community involvement
  • [DONE] Continue triaging and imrpove the groundswell.
  • [MISSED] Research into new Bugzilla home page for logged-in/logged-out users (separate pages)

Desktop QA Automation (whimboo)

  • [DONE] (crossteam) Get SoftVision to lead the automation efforts for QA with our oversight
  • [DONE] (crossteam) Switch Mozmill CI from Mozmill 1.5.x to Mozmill 2.0
  • [MISSED] (crossteam) Get Firefox Metro tests executed in production
Ongoing and supporting projects
  • [MISSED] Extend Mozmill CI to cover most of our localized beta and release candidate builds
  • [DONE] Create web application to generate the beta and release Jenkins job configuations
  • (stretched) Setup tryserver Mozmill CI system assist team members in testing their patches
  • (stretched) Investigate requirements for community driving tool by working together with jdm, lizzard, and pieros

sheriffs (edmorley)

Meta bug: bug 928960

mozbase (ahal)

  • [MISSED] add structured logging to the mochitest harness
    • need to merge the logs from python and js
  • [DONE] determine a long term vision
Retrieved from "https://wiki.mozilla.org/index.php?title=Auto-tools/Goals/2013Q4&oldid=869649"