Top Level

Highest priority, multi-team dependent goals

Detailed Lists

These contain the top level goals as well as the supporting, ongoing, and additional projects we'll try to complete this quarter. Note that additional projects may be moved to the backlog to cope with changes in priority or new projects. The current backlog is here

Android (gbrown)

• P1: [DONE] Stand-up Android 2.3 tests on Graydon's emulator environment - to replace Android 2.2/tegra bug 910092
• P2: [DEFER] Finish Android x86 emu work - get most test jobs working such that sheriffs are okay for trunk with the least amount of effort possible (i.e., turning off failing tests)
• P1: [bc] [DONE] Investigate the usefulness of autolog, and attempt to quantify the value we're getting from it; use this to make a decision to keep or kill it

B2G (jgriffin)

• [ahal] Get the following tests running in TBPL on all trunk trees:
  • [DONE] P1: mochitests on debug emulator builds
  • [DONE] P1: OOP reftests on emulator builds
  • [DONE] P1: reftests on B2G desktop builds
  • [MISSED] P1: WebAPI tests on debug emulator builds
  • [MISSED] P2: Gaia-ui-tests on opt emulator builds
  • [DROPPED] P2: Get all suites running against emulator-jb builds on cedar

• [DONE] P1: Instrument B2G test runners for better logging and error handling
• [ON TRACK] P1: (ongoing) Work with engineering to get B2G reftests off of Fedora slaves, bug 818968
• [DONE] P1: [ahal] Make sure crash reporting is working reliably in all B2G test jobs
• [DONE] P2: [ahal] Convert B2G mochitests to using manifests

• [rwood] Long-running tests:
  • [DONE] P1: (carry over) Finish standing up memory tests reporting to AWSY and have them running per-commit to mozilla-central, bug 899194
  • [DEFER] P1: Help Taipei team implement crash/error detection routines for MTBF that can be used in other harnesses as well
  • [DEFER] P2: Review the endurance tests:
    • Determine if these are needed or if their purpose can be rolled into MTBF tests and/or AWSY tests
    • If needed, make sure we're reporting useful metrics

• [DONE] P1: [davehunt] Work with engineering to add per-app memory consumption metrics to datazilla, bug 917717
• [DONE] P1: [davehunt] Add ability to perform semi-automated bisection of perf regressions in Jenkins, bug 943611

Bugzilla (mcote/glob)

See BMO#Current_Projects for status.

• [MISSED] P1 (carryover): Finish and deploy bzAPI compatibility layer.
  • This ended up requiring a lot more patching of the core code than anticipated, and getting the bzAPI test suite adapted and running correctly was difficult. First revision of patch up for review.
• [DONE] P1: Performance instrumentation (new relic or other)
  • [DONE] P1: Identify areas for improvement in API performance.
    • Several bugs filed, but still more investigation to do.
• [DONE] P1: Deploy production Change Notification System.
• [DONE] P1: Increased memcache support
• [DONE] P2: Provide accurate review history.

Eideticker (wlach)

• [DONE] P1: Get B2G app startup tests running smoothly and consistently on the London hardware, reporting to the Eideticker dashboard
• [DONE] P1: Get B2G fps tests running smoothly and consistently, as above, with a first iteration that ignores the partial frame problem
• [DONE] P1: Work with the FxOS_Perf team to figure out how to produce accurate fps measurements and deal with noise created by the partial frame problems
• [DROPPED] P2: Implement a framework that can be used to run B2G responsiveness tests (first target: virtual keyboard responsiveness)
• [DROPPED] P2: Get Eideticker reporting to datazilla, and data files stored in S3

General Automation

• [DONE] P1: [jmaher] chunk mochitest-browser-chrome
• [DEFER] P1: [jgraham] Get a snapshot of W3C web-platform tests running on TBPL
• [BLOCKED] P1: [dkl] Reconfigure Pulse so that it can act as a robust backbone for Treeherder and other projects
• [DONE] P1: [ahal] Move mozbase to m-c
• [DEFER] P2: [dminor] Finish removing jit-tests and cpp-tests from 'make check' for desktop Firefox

Marionette (mdas)

• [DONE] P1: [ato] Make Marionette compatible with WebDriver at the JSON Wire Protocol Level
• [DEFER] P2: [ato] Get a passing subset of the WebDriver spec suite tests running on cedar
• [DEFER] P1: [ato] Add modal dialog support for desktop Firefox
• [DEFER] P1: [mdas] Investigate and fix frequent intermittent oranges in Marionette unit test suite
• [DONE] P1: [mdas] Make Marionette's logging and error reporting more robust
  • Smarter handling of socket errors
  • Better crash handling
  • Conditional logging of all requests/responses and other significant activity
• [DEFER] P2: [mdas] Create a suite of Marionette endurance tests designed to find memory leaks in Marionette and gaiatest components

Performance (jmaher)

• [DONE] P1: [glob, jeads] Add error bars and ability to toggle between mean and median to datazilla
• [DONE] P1: [glob] Add b2g performance data ingestion rate notifications to datazilla
• [DONE] P1: [kyle, mcote] Add b2g perf regression notifications to datazilla
• [MISSED] P1: [jmaher] Continue work on dashboards (carryover from Q4)
• [BLOCKED] P1: [jmaher] Add new talos tests for latency and webRTC
• [DEFER] P2: [jmaher] Bring up automated power testing for desktop Firefox
• [DEFER] P2: [kyle, jmaher] Add talos perf regression notifications to datazilla

Project X (ted)

Note: ownership of this project transitioned to James Lal

• [DROPPED] P1: Work with Releng and the Gaia team to deliver a prototype of a flexible test scheduling system that can be used by Gaia's Shepherd project to schedule tests based on Gaia pull requests
• [DROPPED] P1: Make it possible to display the results of this test execution in Treeherder

Sheriffs (edmorley)

• [DONE] P1: Perform initial Treeherder testing side-by-side with TBPL.
• [DONE] P1: Add tooling to allow sheriffs to make use of the new SlaveAPI. This is best integrated into the existing slave health dashboard owned by releng - which has been done in bug 914805.
• [DROPPED] [Not possible unless we switch to git.mozilla.org as the canonical repo] P2: Add tooling to allow sheriffs to easily close the Gaia repo

Treeherder (jeads)

• [DONE] P1: Achieve feature parity with TBPL in terms of sheriffing
• [MISSED] P2: Deliver a production system that can be used for sheriffing independently of TBPL
• [DONE] P1: Ensure that Treeherder can ingest and display data from non-buildbot sources, such as Travis or Jenkins