2014 Yearly Goals

• Support platform and products to ensure what we are shipping is high quality.
• Increase scalability by attaining a level of community participation where we have on average 80 active contributors per week, every week working on QA projects
• Every top level project in every area will establish a comprehensive and integrated set of quality metrics and a known, externally (from qa) referenceable level of quality.
• Drive changes to our processes for increased efficiency and impact as measured from both developer satisfaction (with finding issues early and often) and in end-user quality perception and satisfaction.

Supporting Events

Internal Goals (By Project)

Firefox QA

Revitalize our Quality Engineering strategy by switching to a feature-based test strategy for all Firefox browser products so that there are test plans for each feature and testing is executed within two sprints of the feature's code landing.

High-level Supporting Tasks

• General Tasks
  • [DONE] Complete Root Cause analysis so that the data auto-updates and use the data to guide testing strategy successfully for Firefox 34 (by July 21): kairo
  • [DONE] Ship Games Inititative with Mozilla and Partner approved quality: mschifer
  • [DONE] Ship Loop client with Mozilla and Partner approved quality: ashughes
• Community Tasks
  • [DONE] Create community infrastructure to involve and onboard 20 new contributors: mschifer
  • [DONE] Ensure all feature owners have one and done tasks that direct contributors toward contribution paths: mschifer
• Automation Tasks
  • [DONE] Get full coverage of all locales for Firefox beta and release testing andreea
  • [DONE] Stabilize restart testing automation - less than 5% intermittent failure rate: whimboo
  • [DROPPED] Automatically restart machines between test runs and full automatic start of jenkins: whimboo
    • Dropped due to changes coming for new signed builds on OS X
  • [DONE] (Late) Add support to Mozmill and the test suite for new signed builds on OS X whimboo

Firefox OS

Better metrics for quality analysis, provide opportunities for community contribution, and expanding the role of our on-device automation for reporting to Treeherder.

High-level Supporting Tasks

• General Tasks
  • [DONE] Create an auto-updating, public quality dashboard to display smoketest health status: jlorenzo, geo
  • [DROPPED] Define and Implement a plan for closer coordination with TPE QA: tchung, brian
    • Dropped due to new overall direction of OS with reorg.
    • Flame lab build-out for smoketest automation in TPE
    • Define a strategy for geolocated contract testing resources for release work
  • [DONE] Hire open reqs: tchung
  • [DONE] Report a overall acceptance health status a week prior to to FL and FC: nhirata, jsmith, geo
• Community Tasks
  • [DROPPED] Ensure One and Done tasks created deliver contributors into actionable, useful steps with and without a device: marcia
  • [DROPPED] Define, publish, and use contribution pathways for contributors with and without devices: marcia
  • [DROPPED] Hold one in-person quality event, and hold 3 virtual or async ones: marcia
    • Above 3 community items dropped due to reorg and lack of adequate headcount on FxOS 2.1 release
• Automation Tasks
  • [DONE] Define a set of JS integration sanity testsuite that will run on Flame devices as part of the per-checkin Tree Herder strategy: jason, stephen, zac
    • Testsuite should target ~5-8 minutes of execution time
    • Related/co-dependent is (for the actual implementation of these tests) the Q3 A-team goal, "** Get gaia-integration tests running on-device with full support for handling crashes, timeouts, and test manifests (ahal)")
  • [DONE] Integrate graphics regression tests into the mainline code tree: njpark
  • [DONE] Deploy QA Lab Flame buildout: retornam
    • See https://wiki.mozilla.org/QA/Execution/Web_Testing/Lab for more info

Services

Ensure new offerings have 98% uptime (FxA Sync, OAuth, FMD, Loop) and no more than 2 P1 bugs discovered post launch while building a core contributor base in services

High-level Supporting Tasks

• General Tasks
  • [DONE] Complete automated cross-browser tests for OAuth flows: edwong
  • [DONE] Ensure the DIY sync is well-supported - at least one person completes setup
• Community Tasks
  • [DONE] Acquire 2 active contributors for services QA (at least): edwong
    • [DONE] Have at least one meeting with the QA 101 contributors: kthiessen
    • [DONE] Perform one trial seminar of QA 201: kthiessen
    • [DONE] Ensure docs are adequate for on-boarding of contributors in QMO: edwong
    • [MISSED] Perform two services test days at least: edwong
• Automation Tasks
  • [DONE] Ensure FxA smoketests landed in gaia-ui-test: rpappa
    • NOTE: Out of the 4 tests, 2 landed, 1 was whitelisted due to timing issues, and 1 is still in review.
  • [MISSED] Ensure Loop call creation regression performance tests are run and tracked over time: edwong
    • NOTE: The changes in feature scope, amount of iterative load testing, caused this to fall off our schedule. Add in, Edwin's departure from QA proper the last 3-4 weeks also decrease our capacity to execute on this task.

Marketplace & General Web FE

Ensure Marketplace's new payments backend, in-app payments, and the Marketplace feed land with high quality releases.

High-level Supporting Tasks

• General Tasks
  • [DONE] Improve developer visibility into automation tasks
    • NOTE: This was around marketplace testing, you can see the expanded coverage here: http://mozilla.github.io/mozwebqa-dashboard/#/marketplace
• Community Tasks
  • [DONE] Continue participation in monthly Marketplace days
  • [DROPPED] Work on making the Internal hackathon for apps a success
    • This was dropped due to lack of interest outside of QA which was a requirement for success
• Automation Tasks
  • [MISSED] Ensure automation runs on 1.4 using Flame devices set for 319Mb of RAM to simulate dolphin
    • depends on bug 1020536, which itself depends on bug 1011016
      • we will carry this over into Q4 and likely change the goal to reflect a run on 2.0, but this was missed due to https://bugzilla.mozilla.org/show_bug.cgi?id=1011016 landing on master and still needing uplift to the other branches -- particularly 2.0
  • [MISSED] Have automation coverage for Payments including in-app payments
    • depends on bug 1015251
      • same as above

Platform

Deploy into continuous integration a regression testing system for WebRTC along three areas of investigation: 1. "Smoke" tests, 2. Connection tests, and 3. Connection quality tests. These will be performed in a variety of environments, including various network topologies with various performance characteristics, connections between various versions of Firefox, and connections on multiple OS platforms. Test plans will be developed, and we will measure coverage of test plans. Results will be posted and available in Treeherder. We will also build backlog for next Platform Tiger Team tasks.

High-level Supporting Tasks

• General Tasks
  • [MISSED] Build out a backlog for the next Tiger team target(s): syd
  • [DONE] Fill open req: mschifer
• Community Tasks
  • [MISSED] Investigate mentored bugs/one and done etc as a means to begin community involvement and create a plan for community involvement that will be deployed in Q4: syd
    • Other tasks have taken up time. Plus, losing geo and mwargers has increased workload on WebRTC testing.
• Automation Tasks
  • [DONE] Complete test plan and deployment of tests in a sunny-day environment with basic TURN-based smoke tests, including reporting results to treeherder: geo
    • NOTE: Work completed and deployed, test plan now out of date with reality, due to losing geo to reorg
  • [MISSED] Complete procedure for deployment of parallel sunny-day environments using virtual machines, including running TURN-based tests: mwargers
    • NOTE: Work completed, but documentation was relying on mwargers, which missed due to reorg, we are carrying over because we want the documentation to get done.
  • [MISSED] Complete test plan and deployment of connection establishment tests: nils
    • Missed due to H.264, Loop, etc issues and bleed over from Firefox OS QA which has lost several webRTC focused testers
  • [MISSED] Complete test plan and deployment of connection quality tests: geo
    • This will need to be discussed now that geo is no longer on the team.
  • [DONE] Complete architecture plan and deployment of infrastructure for multi-version, multi-OS, and multiple-network topology tests, including smoke, connection establishment, and connection quality tests: syd
    • Got the system up and running, deployed all the systems, but the tests are lagging behind due to increased work from the reorg.

NOTE : This team was especially hit hard when losing 2 individuals to FxOS and from the lack of staff on FxOS which pulled another person away (resulting in 75% loss of headcount from this team) from this work for three weeks. This means we got some of the tests done, some of the code for reporting done, we did get all the systems and automation set up, but the linkage with the new tests and with treeherder did not get completed, putting us behind. Community and building the backlog for the next project were both sacrificed to make up for the shortfall but in the end it didn't prove to be enough.

Community

Increase conversion of interested and casual contributors to help them become active contributors with strong ties to the QA community.

High-level Supporting Tasks

(Further detail at https://etherpad.mozilla.org/2014Q3Goals-Community)

• One and Done
  • [DONE] Create an ongoing roadmap for One and Done: rbillings
  • [DONE] Deploy ability to import tasks] into One and Done from Bugzilla: rbillings/bsilverberg
    • https://bugzilla.mozilla.org/show_bug.cgi?id=1030972 - tracking bug for the above
• Metrics
  • [DONE] Integrate moztrap data into Baloo: lizzard (Note: This got changed to One and Done by the Baloo team: bug 1049696) This is in progress, but will likely not be integrated by BI until early in Q4 because of limitations in the BI/Metrics team. While the integration is not complete, we've done all we can do from this end. The rest is up to BI and Metrics, and we'll assist as needed.
  • [DONE] Popularize, encourage, and standardize use of the Ad-hoc contribution logger: lizzard
• Establish Connections
  • [DONE] Strengthen connections in the QA community. marcia
  • [DONE] In collaboration with the community, develop a long-term strategy to redefine the program as value-driven events designed to teach new skills and connect contributors with various projects. We'll organize at least one event utilizing this strategy and gather feedback to drive more meaningful and active events through Q4 and beyond. ashughes/aaronmt
• Redesign of QMO (quality.mozilla.org)
  • [DONE] Finalize the re-design of QMO with mockups and workflows: pragmatic