QA archive

From MozillaWiki
Jump to: navigation, search

Who We Are

The Mozilla Quality Assurance team is made up of a unique community of internal employees and the public community of those who love the Mozilla products. Please see our team page for more information about who we are.

QA Community

A huge part of what we do is work with the Mozilla community to ensure we ship quality software. To learn more about how to help, please check out the QA/Community wiki page.

Planning

Projects

Community Projects

Team Project List

I am trying this out as a new idea to keep track of our team's projects. Let me know if you have any feedback on this.


Name Project Priority Due? Status Next Steps
Alice, Robcee Talos Performance Framework

Infrastructure for collecting and graphing performance data based on page sets.

High on-going Checked in talos 2 (includes a few bustage fixes discovered after check in). Updated/enhanced http://wiki.mozilla.org/Buildbot/Talos/Machines. Set up qm-pxp03 as a trunk tester (stuart has questions about the results from qm-pxp01, needed confirmation). Switch over machines on the mini perf farm to talos2. Graph server: improve data set selection in dgraph.html - to allow for comparing test with different names and make things make a little more sense.
Robcee, Coop BuildBot Testing Framework High on-going bug 394841: win32 try slave unstuck *many* times

bug 393413: began implementing CLOBBER support for unittest machines. Will require some reorganization of existing scripts to facilitate maintenance. bug 393290: began playing with botrunner for use with unittest -> required some patching on Linux (thanks, rhelmer), master.cfg still throws many errors

bug 393290,bug 393413
Robcee Unit Test Framework High on-going cleaned up places.sqlite on unittest machines for sspitzer - bug 395091

working on "remove places.sqlite file from reftest profile on unittest machines" - bug 395219

Clint Minotaur

Preferences checking tool for L10n and partner builds

High on-going Got the patch ready for review. Sent L10N sample output to Mic for her approval. Overviewed Minotaur at Partner Meeting Address Minotaur review comments. Meet with Mic concerning L10N Minotaur output.
bc, robcee JS testing Med on-going bc: Maintenance, test runs, filing bugs. Regression detection working well. The apparent regression related to 2.6.22 kernels does not appear to be real. The basic problem appears to be more related to branch related code in the script loading utilities in Spider as well as the profiles being used during testing.


SpiderMonkey 1.6.1 release: I have done preliminary work such as setting up the required environment and running tests on the code. I plan to perform the release this week. robcee: Received refplatform VM for js testing - bug 394051. Talked to bc about configuration settings for apache on local js testing machine.

Review latest test run from this weekend with the new CentOS5 installations and the fixed Windows XP installations. Release SpiderMonkey 1.6.1 based on the Mozilla 1.8.0.13 branch. Provide support to Rob for js buildbot automation.
Al Billings Configurations Plan-2007 Q3 goal

define and implement strategy for testing system configurations. This is primarily OS's and profiles. But also includes printers, CPU speed, connection speed, etc. See plan

High Sept 28, 2007 Defining the OS profiles. Initial clean profiles are in the fs directory under fs/PUBLIC/QA/Virtualization
many FFx3 daily smoke and focus testing Med on-going Testing by tomcat and abillings this week.
Stephen Donner Rock Your Firefox/Facebook High Aug 3, 2007 DONE!
tomcat, juanb, tracy Eggplant manual test automation

we have 30 smoke tests automated

Med on-going Bought new version 4 upgrades and new license for tomcat. Tomcat setup his new Eggplant development and testing system Add L10n smoke tests
murali Firefox code coverage analysis

Use code coverage tools to instrument and collect coverage data for C,C++ code along with JavaScript code in the Firefox product. Work with the QA team and Development team to analyze the results and develop a strategy for improving test coverage.
Identify and collect key project metrics in collaboration with the QA and Development teams
Present the project metrics in a way that helps the Engineering team make project decisions and improve project activities
Work with the Metrics team to collect the code coverage and project metrics data in a central place and to create useful dashboards for the management team and members of the Engineering team
Automate the process of on-going code coverage and project metrics collection

Med Starts on 8th October Create an instrumented build for Firefox 3.1x using JSCoverage and one of gcov, coverity, purecoverage and analyze the quality of coverage as well as risk factors associated with the low coverage areas.
Initial phase includes only automated test cases. 		In the first week of the assignment, building Firefox 3.0.3 from CVS and Firefox 3.1x from Hg code base should be accomplished.

In the first week of the assignment, learning to run tests on top of the non-debug, non-instrumented builds should be accomplished.
Start to instrument the build for C/C++ and Java Script.
This task could potentially take at least three weeks of effort to arrive at proper set of instrumentation tools.
Initial attempt would be made with gcov and Jscoverage.
There are known linker problems with Firefox while using gcov. Investigation has to be done to identify mitigation paths.
If the first pair of tools fail to achieve the objective, attempts would be made to use purify, coverity and JScoverage to instrument Firefox code base.
Run the automated test suite on the instrumented build to generate Java Script as well as native code coverage raw data.
Code coverage Raw data analysis.
Identify the source code chunks that have 0-10% coverage, 10-20% coverage, .... 90-100% coverage.
Identify the list of bug fixes that went into the given chunks in the last 6 months period.
Analyze the bugs for specific trends like re-opens, regressions, security, performance, external, internal etc.,

Communicate the results for the Engineering team
Work with the QA team and Development team to analyze the results and develop a strategy for improving test coverage.
Based on the coverage and bug trends, identify top three to five areas of functionality that require special attention from QA.



Issues with gcov attempt http://groups.google.com/group/mozilla.dev.builds/browse_thread/thread/b6902f5e2d8628cd/e6d9cc534dfe43e8?lnk=gst&q=clint+talbert&pli=1

Focus Areas

Draft-template-image.png THIS PAGE IS A WORKING DRAFT Pencil-emoji U270F-gray.png
The page may be difficult to navigate, and some information on its subject might be incomplete and/or evolving rapidly.
If you have any questions or ideas, please add them as a new topic on the discussion page.
Area QA Team Member
TDAI team lead Robcee
Talos/Performance Tools Alice
Buildbot/Unit test Framework Robcee
Minotaur Clint
JS, JSE, Security Test bc
Lab set/configurations Marcia, Al
VMs Al
FFx3 QA Lead Tony
FFx 2.x Juan/Al - co-leads
Litmus Tool coop
Litmus test Suite Marcia, Tracy
Web Testing stephend

Firefox 3

The latest project for mozilla to hit the street is Firefox 3 (codename: GranParadiso). The links below will take you to more information on the project. If you are interested in how you can help test GranParadiso, please contact Tony Chung, the QA Lead of project.

What We Use

We typically use nightly optimized (non-debug) builds for daily usage. When finding a regression window - that is, the range of builds in which a regression first appears - we use the release builds as well as older builds:

  • For nightly builds, go to ftp.mozilla.org, then drill down to the <product_name>/nightly/ directory. While you can go to <product_name>/latest-* directories, the problem there is that you don't necessarily known when those builds were made. It's best to access the specific build-date directory (e.g., 2005-03-17-08-trunk), to know what you're grabbing.
  • Older builds are in the YYYY directories (eg 2007)
  • For release builds, simply go to any of the mirrors and drill down to <product_name>/releases/ and select the appropriate directories for version, platform and locale.
  • For hourly Builds go to [1]

The best way to find a regression window is to do a binary search. Start with the earliest build reported to contain the regression and the latest build reported to be regression-free; pick the build that's halfway between those two dates, and see if the regression exists; if the regression exists, that's your new "earliest build to contain the regression"; if the regression does not exist, that's your new "latest build to be regression free". Repeat until your range is as narrow as possible.

Bugzilla

We depend on Bugzilla for filing and tracking bugs and features. We frequently use the query tools, both the "Advanced Search" and "Find a Specific Bug" queries. With the bug count reaching 300,000, there are a couple ways to see what's been frequently reported and duplicated:

People wanting to help triage bugs can refer to the Mozilla Triage Guide

Update: This is a use case page on ways to improve support for multiple branches in bugzilla.

Litmus

We are currently using Litmus Litmus as a replacement for Testrunner (our previous test management tool). Litmus will build upon the testcase management aspects of Testrunner, adding the searchability, reporting, and trending tools that are critical to our QA work. More information about Litmus development is available in the Litmus wiki.

Mozmill

With Mozmill we are able to run manual tests which have been added to Litmus in an automated manner. We are creating automated tests for nearly all of the existing Litmus tests for Firefox inside the Mozmill test automation project.

Talkback

QA:Talkback discusses Talkback including effective practices, information on topcrash analysis, and metrics on crashes.

Development tools

We also use several development tools for tracking changes, especially useful for narrowing down regression windows!

Automation

In an effort to the improve the testing potential of our team, we are constantly investigating ways to leverage automated testing tools to offload some of the regression testing burden from our core testers.

Test Suites and Test Tools

Mozilla QA Lab Machines

Inventory of QA Machines

Sites to scan for Community and User Feedback

Website Testing

It is important for us to check that major websites display and work correctly. According to this target we are building categorized lists of all those websites.

Litmus Test Case Maintenance

We periodically do a sweep through Litmus to make sure the various test cases are up to date and that there are no holes. This page tracks that effort->Test Case Maintenance

Test Cases that need to be added

Ownership Areas for Litmus BFT

Release Data Firefox/Thunderbird

This data has been moved to QA/Talkback/Build IDs

Test Case Wish List

If you intend to work on a set of test cases on this list, please put you name beside the feature as well as a date so we can track progress. Thanks.

Firefox

  • Firefox 3 specific
    • Need someone to own Microsummaries area of FF3. Add new test cases and update existing ones.

This is older stuff that needs to be updated - some areas of Firefox 3 may need help with both revisions and/or additions:

  • Safe Mode
    • We have general test cases for launching safe mode, but need test cases for each radio button item [owner needed]
  • Firefox Context Menu [marcia]
  • AJAX sites
  • Java Sites for the Mac (to test applets)
  • Exit Survey (Windows only new feature)- anu
  • Test cases to check out handling of Plugins [windows only]-anu
    • Not the plugin finder service
    • How we handle these on new installs and existing installs
    • Top 7 such as Quicktime, Adobe, Flash, etc.
  • Live Video streaming test cases
    • mlb.com, for example
    • Test all of the various protocols that are out there
    • Printing Test Cases [owner needed]
  • Windows Vista Specific Test Cases
    • Parental Controls
    • Plug in Handling
    • Making Firefox the default application
    • Running as a Standard User-anu

Thunderbird

  • Exit Survey (Windows only new feature)
  • Encryption and Digital Signing Test cases
  • Test cases that cover collapsing/uncollapsing various accounts
  • Improved Filing Tools: Recent folder menu items for moving and copying folders to recently used folders. Move / Copy again functionality.
  • Session History Navigation: Back and Forward buttons allow navigation through message history.
  • Improved Offline Management: Thunderbird (Linux and Windows) automatically adjusts the offline state based on current network connectivity.
  • Windows Vista Specific Test Cases
    • Plug in Handling
    • Making Thunderbird the default application
    • Running as a Standard User

Test Matrix for Windows XP and Vista Update Testing

Useful Information for Testing and Debugging

Proposed Schedule for QA meetups

Retrieved from "https://wiki.mozilla.org/index.php?title=QA_archive&oldid=204148"