QA/Firefox 4/Test Plan/Web Compatibility

From MozillaWiki
< QA‎ | Firefox 4‎ | Test Plan
Jump to: navigation, search

Firefox 4 Test Plan

Web Compatibility Testing

Starting with beta 9 we have been working with an outsourcing company, uTest, to help us go through common user functionality on some of the top sites as listed in Alexa.com. Our current approach is to test sites in the following categories in three rounds of testing:

  • Top 100 sites
  • Top global financial sites
  • Top social, communication, entertainment, games sites.

uTest contracts remote testers with different hardware and software configurations and they instruct these testers what and how to test. Testers are required to compare functionality between the latest released version of 3.6 and 4.0 beta to find if a problem is a regression. We help them outline the tests. When problems are found they are written down in a Google Docs spreadsheet by the testers, and the our contact person in uTest files bugs in bugzilla with all the relevant information if no existing bugs are found.

Coverage can depend on availability of testers with a specific OS, but for the most part they are able to source testers for each of the sites and scenarios we requested.

Round 1: Top 100 Sites

For the first round of web compatibility tests, we picked a set from the list of top 100 sites in Alexa.com, categorized it by area like news, email, video sites, etc..., and we specified that testers had to visit those sites and interact at least one level deep, looking for obvious problems.

The testers found several issues, but most were existing issues. uTest filed some issues but were either duped to existing bugs or the contact person at uTest was not able to reproduce the issue. During the investigation of one of those problems we found a Hotmail registration issue,bug 628813, that was later resolved because it turned out to be a site problem.

Round 2: Financial Sites

For the second round of testing, we went through the list of top financial sites by region, and came up with a list of about 25 sites. In order to tests these sites, you were required to have already registered for an account, so uTest sent out a request to their testers who had accounts in the top sites identified. For the most part they found testers for each of the sites, and a few more.

The most notable thing about this round was that there were no problems found.

  • Test: Top 25 financial sites by region
    • USAA, HSBC, Deutsche Bank, Commerze Bank, Securian, Wells Fargo, Natwest, Nationwide, Société Générale Particuliers, La Caixa, Discover, itsme247, Merrill Lynch, Mint, Barclays, BNP Paribas, Chase, Paypal, Capital One, Amex, eTrade, Ameritrade, Bank of America, LockheedFCU, Wachovia, Citi, ING, Fidelity, Lloyds, US Bank
  • Scope: Basic functionality in the site
    • Create accounts, login in to existing account, view account details, view recent activity, sort activity, filter events, conduct transactions, creating budgets, setting up account alerts, inactivity timeouts, among other functionality depending on the site.
  • Bugs: None filed
  • Results spreadsheet: Testers results and notes

Round 3: Social, Communication, and Entertainment Sites

Our third round of web compatibility testing will involve testing common user scenarios in top categories, including sites like Facebook (and the Zinga games you can engage in), Twitter, Gmail, Hulu, MLB, etc... We'll go through this test suite using beta 12, which should contain all the hardblockers which could incur in regressions.

We are gathering a comprehensive list of sites in preparation for this run. You can take a look at some of the sites being considered for this run: site list

Retrieved from "https://wiki.mozilla.org/index.php?title=QA/Firefox_4/Test_Plan/Web_Compatibility&oldid=753857"