Auto-tools/Goals/2013Q3

From MozillaWiki
Jump to: navigation, search

Top of the List

Highest of highest priority

Automation

P0: WebRTC/Services Automation

  • Define and implement the minimally viable "services-based" automation system:
    • [DONE] Allows configuration of different networking environments between end points
    • [DONE] Built in-cloud on AWS
    • [DEFER] Integration with Treeherder
    • [DONE] Allows tests to be defined simply in the main m-c tree
    • [DONE] Runs WebRTC tests
    • [DEFER] Could also use this system to do a sanity check on the social API partner integration once the marionette based tests for social API are written
    • [DONE] Should use an off-the-shelf scheduler - either jenkins or buildbot or something.

P0: B2G General

  • [DONE] Make B2G tests in TBPL more sheriffable, by providing a tool in TBPL (Treeherder?) to compare revisions of all B2G repos between arbitrary builds
  • [DONE] Finish getting gaia-unit-tests and gaia-ui-tests running on all trees in TBPL
  • [MISSED] (Depends on James Lal's dev team): Deploy gaia-integration tests into TBPL
    • NOTE: We have an npm mirror; need to write mozharness script
  • [MISSED] Stand up tests on new Android 4.2 emulator (this is potentially going to be very time consuming)
    • NOTE: Waiting for rel-eng to schedule 4.2 builds/tests on cedar
  • [DONE] Get b2g jsreftests working, so we can turn off no-ion builds/tests (stretch goal)
    • NOTE: We have these running in single-process mode, but reftest in general doesn't work in B2G in OOP mode, which is the reason for running them on no-ion builds, so we can't turn off the no-ion builds yet.
  • [DONE] Get tests running using full-stack emulator builds
  • [DEFER] Implement heuristics to notify dev managers of "real" failures from B2G device test runs on Jenkins
  • [MISSED] (Depends on rel-eng): Help implement a gaia try-server
  • [MISSED] Resolve various tap-related issues (including modal dialogs) in Marionette in order to support gaia-ui-tests and gaia-integration-tests

P0: Eideticker

  • [DONE] Create responsiveness test for firefox on Android test using Eideticker
  • [DONE] Create a set (at least 2) of tests for b2g Eideticker to measure user perceived application startup and scrolling performance, and run on a daily basis

P1: Marionette

  • [DONE] Clean up oranges
  • [DONE] Better error reporting
  • [DROPPED] Add support for nested content processes
    • NOTE: not needed
  • [DONE] Continue support for B2G testing / developer ergonomics (webqa, mach support) ... originally scheduled for Q2 but dropped for focus on gaia tests
  • [DROPPED] Support social API tests
    • NOTE: The social API team is using the 'old' FirefoxDriver for now

P0: Performance

High level details:

  • update talos tests to measure relevant parts of the code while providing detailed reports to the developers
  • collect competitive benchmark numbers on a regular schedule
  • deliver and refine tools for development teams as their performance needs change

Finer details:

  • Networking:
    • [DONE] deliver webpagetest as a tool. This will be 100% offline for non ssl pages, and we will provide a similar setup but 100% online for ssl traffic.
    • [DONE] work with necko team to create an easy to use interface with local builds, try builds, nightly builds running on wpt and reporting in a meaningful manner
  • Benchmarking:
    • [DONE] work with the games and js teams to run competitive benchmarks on android, b2g, and desktop platforms
    • [DONE] report numbers at least once a month
    • [DONE] integrate additional benchmarks, browsers, platforms (i.e. IOS) as well as more hands off automation
  • Talos:
    • [DONE] add additional alerts for disk flushes, too many disk accesses with talos xperf
    • [DONE] continue modifying existing tests to make the data more useful, improve documentation on what all tests measure
    • [DONE] modify talos ts_dirty tests to have updated profiles and measure useful scenarios
    • [MISSED] have sheriff/developer dashboards for datazilla, statistical analysis of all test types with alerts for detected regressions at <2% noise
  • Other:
    • [DONE] continue working with developers to refine microperf
    • [DONE] deliver a prototype for real world benchmark using the live internet to aid the js team
    • [DEFER] provide help to the servo/rust team as they start measuring performance
    • [DONE] investigate ways to involve community into the process of measurement and analysis

P1: MozBase

  • [DONE] unittests for all modules
  • [MISSED] m-c commit watcher to notify mozbase owners of unexpected commits
  • [MISSED] full mochitest and reftest integration with mozprofile/mozprocess/mozrunner (excluding Fennec)
    • NOTE: mozprofile work done; the remainder is just being started
  • [DONE] continue to standardize + impove dev practices
  • [DONE] structured logging
  • [DONE] manifestdestiny API improvements and fixes
  • [DONE] add mozprocess documentation

P2: mozmill 2.0 released

  • [DONE] Release 2.0 (should be done Q2)
  • [DONE] SV leads metro testing using mozmill

Tools

P0: Bugzilla

Optimize bugzilla as a platform that we can build on top of.

  • [DONE] Deploy native REST API
  • [DONE] Optimize and expand native REST API as needed to support 3rd party UI building efforts
  • [DONE] Stand up reviewboard as a semi-independent reviewing service
  • [DONE] Tracking flag migration
    • Migration script is ready, but IT outage will not happen until early Q4.

P0: TreeHerder

  • [MISSED] Deliver UI parity with existing TBPL
  • [DONE] Implement UI plugin architecture
  • [DONE] Set up publicly accessible instance
  • [DONE] Develop a logging strategy:
    • for the data adapters
    • for the restful api
  • Full Project Listing Is Here

P0: Datazilla

  • [DEFER] Automate datazilla database rollover and rollover talos data. What do we want to do with legacy data? (Stretch goal)
  • [DONE] Refactor datazilla to remove all portions of the old UI, unnecessary controllers, and legacy code not in production
  • [DONE] Provide datazilla landing page that displays data for all projects
  • [MISSED] Replace graph server for talos on m-c,m-i
    • Everything in place, except alerting which will carry into Q4

Process

P1: Infrastructure Load

  • [DONE] parallelize xpcshell test framework
  • [DONE] analyze always passing tests and determine a strategy for turning tests off selectively/running intermittently
  • [DONE] work with releng to turn on code coverage builds

P1: Sheriff goals

  • [DONE] Provide documentation and how to steps to enable a community sheriff to be effective
  • [MISSED] Wire documentation into bugsAhoy/whatcanIdoForMozilla
  • [DONE] Fix P1/low hanging fruit of the Sheriffing Tools (see full goals list)
  • [DONE] Drive treeherder UI functional specification
  • Full Goals here

P1: Bugmaster

  • [DONE] Create a contribution pipeline for 30 incoming bugmasters and track them through each stage of involvement in order to get 1 to the state where they regularly do triage.
  • [DONE] Improve Bugzilla Help pages.
  • Full goals here
Retrieved from "https://wiki.mozilla.org/index.php?title=Auto-tools/Goals/2013Q3&oldid=718063"