User:Tomcat:Topsites

	THIS PAGE IS A WORKING DRAFT
	The page may be difficult to navigate, and some information on its subject might be incomplete and/or evolving rapidly. If you have any questions or ideas, please add them as a new topic on the discussion page.

Overview over the automated Mozilla QA TopSite Run's

This wiki page should give a overview about the current TopSite Testruns that Mozilla QA currently run and what we plan for the future

With this TopSite Tests we are able to test more websites then ever before for Leaks/Crashes and Assertions and help so to find regressions and so far unknown Problems

Contact

Carsten "Tomcat" Book Email

Testframework & Infrastructure

The Topsite Tests are based on the [Sisyphus Test-framework] and use the Spider Extension to load the Websites.

With the TopSite Tests we are able to search for Memory Leaks, Assertions and Crashes. All Testsresults are recorded into one logfile and can processed through scripts and/or grep queries.

We have currently a set of Topsite Lists (from 500 URL's to 1 Million URL's) and so we are flexible what test and how many sites we test. There are also different levels of how deep the Test goes. As example we can Test only Frontpages (Deep 0) or also all Links 1 Level Deep or even deeper!

A typical 100.000 Topsite Test on Frontpages take 3 Weeks.

We can run this tests on all platforms (Windows XP, Linux and Mac 10.5). The Tests for Windows and Linux are done by dedicated VM's hosted on one ESXi Server.

Tests currently active

Firefox 1.9.1 / 1.9.1 Tracemonkey

• Memory Leak Tests - Frontpages only
  • Linux Centos 5.2 50.000 Sites - 1.9.1 Tracemonkey
  • Windows XP 100.000 Sites - Firefox 1.9.1

• Crash Tests Frontpages only
  • Linux Centos 5.2. 50.000 Sites - 1.9.1 Tracemonkey
  • Windows XP 100.000 Sites - Firefox 1.9.1
  • Windows XP 100.000 Sites - Firefox 1.9.1 Tracemonkey

Firefox 3.0.x (1.9.0x)

• Windows XP 1.000.000 (1 Million!) Sites - Firefox 1.9.0 (initial run!)

Test procedure

As mentioned the Topsite Testrun generate a Logfile, but to get results that are useful it involves some work before a bug gets filed.

• Crashes

After a crash was found during the Testrun its noted in the Exit Status, as example EXIT STATUS = CRASHED or Abnormal on Windows XP (as example for fatal assertions).

I do a grep query for this Exit Status and verify this Crash on a new recent debug build and create a reduced testcase.

• Leaks

Debug Build have logrefcnt enabled and the Leak Test output is directed to the Testrun Logfile. With this information i try to reproduce the leak and create a testcase.

Tests planned for the future

• Next Step is to install Addons and to run the Topsite tests with the Top 1000 Extensions, to find Leaks and Crashes.

Credits

Many thanks to Bob Clary and Jesse Ruderman for the help to make this TopSite Test and the reduced Testcases possible!

	THIS PAGE IS A WORKING DRAFT
	The page may be difficult to navigate, and some information on its subject might be incomplete and/or evolving rapidly. If you have any questions or ideas, please add them as a new topic on the discussion page.