QA/Firefox Updates

From MozillaWiki
< QA
Jump to: navigation, search

Summary

The following is a tracking page for QA coverage of Firefox Updates. Any and all questions should be directed to the QA Lead on this project, Anthony Hughes (irc: ashughes).

Context

With the move to rapid releases, QA made a risk assessment to slip on the 6 week schedule or to rely heavily on automation as a testing strategy. With that, we quickly moved from 100% manual testing to 100% automated testing. The purpose of this document is to build out and communicate a testing strategy which is inclusive of many types of testing without sacrificing time-to-market. Thereby ensuring our users the most secure and painless update experience.

Current Testing Strategy
  • Automation test partial, complete and fallback update paths
  • Parsing automated results for any red-flags, but no focused manual spotchecking of updates
  • Random slice of 4 previous versions and 4 locales across 9 platforms
  • Time to sign-off updates ~1 hour per channel (4 hours from betatest to release)
Revised Strategy Going Forward
  • Ownership: Update component and new Update features to increase visibility into issues so we can react more aggressively
  • Automation: test partial, complete and fallback update paths; parse automated results for any red-flags
    • Coverage: 4 previous versions, 4 locales, 9 platforms, releasetest/beta/release
    • Time to Sign-off: ~1 hour per channel
  • Manual: run Smoketest suite and exploratory testing (billboard, what's new pages, places/profile/add-ons/session retention)
    • Coverage: 2 previous versions, 2 locales, 4 platforms, betatest
    • Time to Sign-off: ~4 hours
  • This strategy should not significantly impact time to market as manual testing can happen in advance of and parallel to automation
  • Total time to sign-off: ~6 hours from betatest to release
Long-term Vision
  • Running automation across ALL locales
  • Expand automation to capture potential UX regressions
  • Evolve manual testing in anticipation of and reaction to issues
  • Increase QA visibility and aggressiveness to foster a proactive strategy

Firefox Update Mechanism

The following will be used to track how we test Firefox updates in general.

  • Litmus Tests
  • Automated Tests
  • Bugzilla Triage
  • etc...
Summary Ensure a certain level of quality with the update experience in general
QA Lead Anthony Hughes (Desktop), Kevin Brosnan (Mobile)
Manager Robert Strong
QA Status Currently cataloging existing test coverage

Silent Updates

Summary Make the update process silent so that there is the minimal user interaction required.
Status Most features are in the process of landing or are in development (2011-11-22)
QA Lead Anthony Hughes
Manager Chris Lee
QA Status QA is drafting test plans and test cases; testing landed features (2011-11-22)
Meetings Every Thursday at 1pm PDT

This is an aggregate of several smaller features...(see below)

Feature Owner Test Plan Status Milestone
Bypass UAC Dialog on Windows Simona Badau test plan Test Planning Firefox 12
Background Updates Vlad Ghetiu test plan Test Planning Firefox 13
Workflow with Incompatible Add-ons on Update Ioana Budnar test plan n/a Unscoped
Add-ons Compatible by Default Virgil Dicu test plan SIGNED-OFF 2012-01-25 Firefox 10
What's New Communication Mihaela Velimiroviciu test plan SIGNED-OFF 2011-11-05 Firefox 8
Reduce Displayed UI Paul Silaghi test plan SIGNED-OFF 2011-11-05 Firefox 8

Current overview of compatibility can be found here
Draft test cases can be found here
Litmus test cases can be found here
Etherpad for staging weekly status can be found here

Meeting Notes

Date Summary Notes
2012-01-26 Silent Update Status

UAC Service (Simona)

  • Will not uplift to Firefox 11
2012-01-05 Silent Update Status

UAC Service (Simona)

  • Landed on Nightly, needs testing to elevate to Aurora in time for Firefox 11

Default Compatible Add-ons (Virgil)

  • Working on regression hunting test plan for real-world ecosystem

Background Updates (VladG)

  • Security review scheduled for next Monday, patches landing very soon
  • Ehsan will ask if we can have automatic Nightlies generated for testing
  • Ehsan will try to make himself more available during Vlad's timings
2011-12-08 Silent Update Status

UAC Dialog Suppression (Simona)

  • lots of backend issues plaguing landing/testing
  • hopefully test ready by Monday, landing on Tuesday in Nightly
  • ehsan has a tool for QA to use (he will email)
  • anyone available to help should assist Simona (ie. owners for features slated for Fx 12)

Add-ons Compatible by Default (Virgil)

Background Updates (Vlad)

  • elm branch should be updated on elm branch
  • ready by early next week

Incompatible Add-ons Workflow (Ioana)

  • landing probably in january
2011-12-01 Silent Update Status UAC dialog suppression
  • Targeting a ship of Dec 6th in Nightly
  • Testing needs to begin NOW with special attention to fallback updates
  • Once Nightly builds have it, we need to get the community testing it
2011-11-24 Silent Update Status
  • Releng is working on bits for invalid certificated testing for MARs
  • Brian to spearhead testing once ready (Dec 6 deadline)
2011-11-17 Silent Update Status
  • OS Security Dialog Removal will be restricted to Windows XP, Vista, and 7
  • Background Updates needs more focused testing, in spite of automation coverage
  • QA interaction has been markedly improved over recent weeks
2011-11-10 Silent Update Status
  • Background Updates, UAC Dialog Suppression, and Add-ons Compatible on Default are ready for testing
  • escalated to QA Owners
2011-10-21 Security Review: Background Updates
  • QA needs a comprehensive testing strategy for "noisy" updates which can be reused for "silent" updates
  • Fallback from silent to noisy also needs coverage.
2011-10-20 Silent Update Status
  • Need testplans ASAP for Apply Update on Shutdown, Bypass UAC, Default Compatible Add-ons
  • What's New Page pulled from Firefox 8, exists for Firefox 9
  • Testday next Friday, Oct 28 -- please add tests to the testplan
2011-10-13 Silent Update Status
  • Nothing actionable for QA
2011-10-13 Security Review: OS Dialogs
  • Need testing of UAC; service on 32-bit, 64-bit, 32-on-64-bit; fallback to updater.exe on service fail
  • Targetting Firefox 10

Add-ons

Feature Owner Test Plan Status Milestone
Add-ons Confirm on Upgrade Vlad Maniac test plan SIGNED-OFF Firefox 8
Third Party Add-on Warning Vlad Maniac test plan SIGNED-OFF Firefox 8

Mobile

Feature Owner Test Plan Status Milestone
Retrieved from "https://wiki.mozilla.org/index.php?title=QA/Firefox_Updates&oldid=391017"