QA/Triage

From MozillaWiki
< QA
Jump to: navigation, search

Mozilla Triage Guide -- Harnessing the Flood of Community Bugs

Intro

This article is for anyone who is serious about QA in Mozilla. By following these techniques you will help the important bugs get fixed, as well as optimize precious developer time.

Bugzilla Permissions

In order to perform some of the tasks in Bugzilla listed here, you'll need upgraded permissions for your Bugzilla account. This requires a small amount of proof of your competence - see the page about mailing Gerv for details of what's needed, and how to contact him for a permissions upgrade.

How to Track Relevant Incoming Bugs

In Mozilla, a QA must not only test and then file their own bugs, but must also help harness the incoming flood of community bugs for valid problems.

The tool that Mozilla uses to keep track of bugs is Bugzilla. With Bugzilla's search page, you can search for bugs. There is a simple search option and an advanced search option. The simple search option can only be used for the simplest queries. First we need to know how to look at the subset of bugs you wish to specialize in.

One way to track a subset of bugs is through continual queries, perhaps one every day or so (today's filed bugs). This is good if you only plan to go through the bugs every so often, or if you need to track something based on keywords or fields other than product/component, or when you need to look through older bugs.

If you want something more immediate, you can choose to track a component, such as Disability Access. This is possible via your Bugzilla preferences. Make sure to track the relevant component in both Core->Disability Access APIs and in the products you care about, such as Firefox (Firefox->Disability Access) or Thunderbird. One way to track a component is to find its default QA contact pseudo-address (e.g. folders-message-lists@thunderbird.bugs for Folders and Message Lists) and add that address to your user watch list (under Bugzilla's Preferences, Email Preferences tab).

How to Clean Incoming Bugs

Ensuring the bugs are properly marked up helps make sure they don't get ignored. For example, if a bug that really affects all platforms is marked "Solaris" because the original bug author was on a Solaris machine, most developers will ignore it because they need to focus on the bugs that affect the most people. If a bug is in the wrong component it can also get stuck without any attention. Another common issue is if truly major bugs are marked as normal, or if the bugs have a confusing summary. Ensuring that the bug is properly marked up is very helpful for getting it attention as well as keeping it from being ignored by appropriate queries that developers and other QAs run. Do the following steps in batches, if possible. Each time you submit changes to the bug it creates extra Bugmail, so it's better to do the changes in as few rounds as possible. These techniques really apply to bugs QAs are filing as well.

  1. Understand the bug: Read very carefully to try and understand what they're talking about.
  2. Look for duplicates: A lot of bug reporters mistakenly assume their bug hasn't been filed yet. Search for previously filed bugs by searching for keywords in the summary in the bug, and try alternate spellings and synonyms. For example, if the bug is about the control key, try spelling it "Ctrl" as well. Look through fixed and duplicate bugs when you search for duplicates.
  3. Ask for more information from the reporter: If you don't understand the bug or can't reproduce the bug, ask the reporter for more information, until you have enough to try it yourself. If they did not provide build information, get that right away, because many bug reports are for older versions that have long been fixed.
    Some other information you could ask for:
  4. Reproduce the bug: Change the status from Unconfirmed to New if appropriate. For instance, for many Core bugs, like layout or DOM bugs, a minimal testcase is often needed before a bug can be confirmed. Many times, bug reports are confusing and difficult to reproduce, so read the bug description over very carefully. It will reveal clues, and bring up questions. If the described problem is in fact the desired behavior then mark the bug INVALID. If the bug report does not have enough information to be of any value, and requests for clarification don't net results, then resolve the bug as INCOMPLETE. When marking INVALID, be sensitive to the fact that the bug reporter may get offended. While changing the status it's often good to write something like "This appears to work as designed, please reopen and provide more information if you believe this is in error."
    If you're convinced you understand the bug, and you can't reproduce the bug, you could resolve the bug to WORKSFORME. But be careful with this, the bug can be a duplicate of a bug that has been fixed, so try to search for that bug. Also, you need to test on the same platform as the reporter, because it could be a platform dependent bug. The best resolution is if the original reporter marks a bug WORKSFORME, after he tried with the latest nightly build, tried with a clean profile, etc.
  5. Simplify the bug testcase. You're the lead investigator -- so try to get enough information to turn intermittent or hard to reproduce bugs into something that is easy to reproduce. See more on this in "How to Really, Really Help Developers on Bugs -- Minimal Testcases" below.
  6. Ensure appropriate hardware/platform tests: most bug filers will only have tested on one platform. If bug was filed for a particular platform, see if it's a bug on another platform. If it's been reproduced on at least 2 platforms that aren't both Unix/Linux, then go ahead and mark it All/All ((Hardware/OS). Most bugs are actually All/All. It's useful to get a feeling of when to do extra tests and when not to waste your time, but this comes with experience. For example, OS X uses many native Carbon form controls and menus whereas Windows and Linux do not, so form control bugs on Mac are probably just Mac-only. However, a bug in how HTML links are dealt with is almost certainly a bug on all platforms. Also, it's useful to know the QA community so that you can ask for help when you don't have the right hardware/OS to test a particular bug with.
  7. Ensure a concise and precise summary: It is reasonable to edit the summary a bit if it will help people understand the bug more quickly or will clear up something that's misspelled or incorrect. Check to see that the summary is clear, concise and describes the problem as fully and correctly as is reasonable in a short space. Make sure it contains the words people would probably search for before filing a duplicate bug. To risk stating the obvious, the spelling in the summary matters because searches will for "Escape" will fail if the reporter wrote "Escpae".
  8. Security sensitive bugs: [TBD]
  9. Ensure Proper Severity (but don't change Priority field, which is for the developer). Here's a quick guide, but you can't get more help from Bugzilla itself
    • Enhancement: ideas for improving the software.
    • Critical: crash and security bugs (should also be marked with keyword crash).
    • Major: Bugs that affect the main screen, cause dataloss, loss of ability to operate the software or are just plain awful
    • Normal/Minor/Trivial: Most other bugs
  10. Change Product, Component and owner: Many bugs get marked "Firefox" when they are really bugs in the core engine. If it's a basic XUL, HTML or layout bug that affects multiple products send it to Core. After you submit a Product change to the bug, Bugzilla will allow you to change the Component as well. When changing Product and/or Component you usually want to select the radio button "Reassign bug to default assignee and QA contact of selected component ". The exception to this is when the bug is a regression and already assigned to a particular developer who caused the regression. You can spot those situations via a comment like "Joe, this appears to be from your checkin from bug 32412".
  11. Add Keywords -- here are just a few of the important ones the full list of keywords is here).
    • "access": mark accessibility bugs with the keyword "access". This allows accessibility bugs to not be tied to the "Disability Access" or "Keyboard navigation" components, but to the actual component with the problem. For example, the developers for Places should really be responsible for accessibility bugs in Places, but adding the keyword "access" makes sure that those looking for accessibility bugs will also find the bug report.
    • "regression": mark regressions with "regression". Was this ever not a bug? For example, did it work in Firefox 1.5? See below for more important information on how to deal with regressions.
    • "crash": pretty obvious :)
    • "dataloss": causes users to lose data
    • "hang": freezes or locks up the application
    • "testcase": indicates that a simplified testcase is included (see below)
  12. Find a regression window: See the Finding Regression Windows section.
  13. List related meta bugs in "this bug blocks". If an important meta bug has a difficult to remember number, give it a memorable alias. For example, here are some important accessibility-related meta bugs:
    • focusnav: related to focus and tab navigation
    • caretnav: related to caret navigation
    • themea11y: affects use with high contrast accessibility themes

How to Really, Really Help Developers on Bugs -- Minimal Testcases

One really important thing that QAs can do for developers is to narrow down to the actual markup causing the bug -- this applies for QA-filed bugs as well. Many testcases filed by inexperienced bug reporters come from a website, which unfortunately may change and thus the testcase is lost. Or, the HTML from the given testcase might be huge, which makes debugging extra difficult.

Here are some steps for using the process of elimination to create a minimal testcase from a webpage that displays a consistent bug.

  1. Save the webpage on your local hard drive. Make a second copy.
  2. From the first copy, cut out big pieces of CSS, JS or HTML -- start with the pieces that are most likely to not have the bug. For example, at first just try to strip out plain paragraphs as these are relatively simple. Try to make as much as possible inline. Try to strip out all images, frames. If any image is required to get the bug, try changing the src to something that isn't likely to expire, such as: [[1]]
  3. If the new version of the file still has the bug, save it as your second copy and repeat step 2.
  4. If the new version does not have the bug, then your second copy still contains the bug. In this case, repeat step 2 but avoid cutting out the same piece that you did last time.
  5. When you can no longer cut out any HTML without destroying the ability of the file to show the bug, you have a minimal testcase.

Hopefully you'll be left with a file that has only 1-5 HTML elements in it, but still reproduces the bug. At this point you can actually try to start stripping out irrelevant attributes as well You may end up with something like <label for="myfile">Enter file name:</label><input type="file" disabled="true" id="myfile"> instead of 50k worth of HTML. From all the developers you do this for, "Thank You!".

When finished, attach the minimal testcase, mark the bug with keyword testcase, and change the bug summary if necessary. For example, "No borders in tables at http://www.sillystatsandstuff.com" can become something much more precise such as "tables with rowspans do not get borders". If it doesn't affect the test - add a title so it can be referred to, e.g <title>Testcase #1 for bug 123456</title>. Also, add the bug number in the filename of the testcase.

You need to use your brain throughout this process. For example, if the bug involves line wrapping, then the width of the window and the font size may impact the results. In this case, you can ensure that your reduced testcase will work for everyone by using "font-family: monospace;" and a width specified in "ch" units.

For more detailed instructions, see "Reducing Testcases" on DevMo.

How to Help with Regressions -- Finding Regression Windows

Another great thing that QAs sometimes help developers with is finding a regression window for regressions. A regression window is the day that a bug broke. With this information we can take a look at the checkin list and through the process of elimination see what change caused the problem. It's always important to find the regression window, because without it a developer may apply a bandaid fix, instead of finding the true source of a problem.

Here are some steps for finding a regressions window:

  1. Take a guess when the bug might have broken -- any guess will do for a start, so just use your instincts.
  2. Download and install a build from that date from http://ftp.mozilla.org/pub/mozilla.org/firefox/nightly/. I recommend installing the build in a directory named something like Firefox-03-11-06 so that you can go back to it during this test.
  3. Use a separate dedicated profile, without any extensions installed and start with -profilemanager.
  4. Test the build and write down the results. It's common to get confused since you are testing for the absence of a bug, so good notes are important. If the bug occurs, then you know you will have to search back further in time. Otherwise you will need to search forward in time. The optimal algorithm is to test the date exactly between the last known date when the bug didn't occur and the last known date when the bug did occur. Keep dividing by two and you'll find the day.
  5. In the end, make sure that the bug is marked with the keyword "regression", and add a very clear comment something like: "March 19, 2006 trunk: works, March 20, 2006 trunk: broken". When you have found a regression window within a day, then the actual hour the product was built becomes important. You can give this information with the build ID. Another option is to look at the time when the "firefox.exe" or "thunderbird.exe" was last modified. Then take a larger margin (by the hour) of that time. A third option, if you're paying attention when downloading builds in step 2 above, is to note the sets of numbers in the directory name; stripping the dashes from that string gives you the build ID in most cases (e.g., a build from pub/mozilla.org/firefox/nightly/2005/08/2005-08-27-19-trunk/ is a trunk build with a build ID of 2005082719).
  6. An experienced QA can pinpoint the actual checkin with the problem, and CC the developer who was responsible to the bug. Finding out which checkin was responsible for the regression is possible with the Bonsai tool. At the bottom of that page you can add dates ("between ... and ..."). When submitting the form, you get the list of checkins between those dates. When you know the build IDs of the last working and first broken builds, expand them back into dates and times to get a more precise set of results (e.g., a build ID of 2005082719 becomes 2005-08-27 19:00).
  7. If you are very technical, it is possible to go even further. Some QAs who know how to apply patches and build their own Mozilla have even been known to find which part of a patch caused the bug, by building with or without each piece in a patch. However, this is really not necessary, so don't worry if you're not comfortable doing that.

Again, from all Mozilla developers to the QAs who find regression windows, "Thank You!"

How to Deal with Ignored Patches [Needs work]

It's probably useful for someone to search for these once in a while. Often there are nearly complete fixes that "rot" because no one finishes the job, and the Mozilla code that needs to blend with it changes, thus making the fix more and more difficult to apply over time.

There are several types of bugs here:

  1. Bugs that have a patch which was almost correct, and has been commented on (perhaps minused or not), but no one finished the work. Perhaps the original developer just forgot about it.
  2. Bugs that have been awaiting review for over 1 month
  3. Bugs that have had review requested, but not from a specific person. This often happens by mistake of the reviewer name was entered incorrectly, in which case Bugzilla will flag the patch for review but not assign that to someone. These review requests are almost always ignored. If you think this is a mistake, try to assign to the review request to someone appropriate (this takes experience). If you're not sure, ping the patch author and ask them if it was intentional.
  4. Bugs where the patch author asked the reviewer a question, and received no answer
  5. Patches which are reviewed but no one checked them in. Believe it or not, this happens all the time. Either the developer forgot, or did not have the correct CVS permissions to make a checkin. In this case they're supposed to request help from a developer with permission to check in. If they don't know that, a perfectly good fix may just sit there rotting away. It's best to add "[checkin needed]" to the status whiteboard to let people know that the bug has a patch that needs checking in, or to remind the developer who made the patch to check it in.

How to Deal with Ignored Bugs [Needs work]

Use discretion when choosing these options; ignored bugs are unfortunate, but they are a fact of life in a large, complex software project. Managing to annoy developers while trying to get bugs fixed is almost always counterproductive, however.

Options:

  1. Evangelize: go to newsgroups and try to stir up interest there
  2. Email the developer it's assigned to and ask him what his plan is for it, and if there is any way you can help, or whether it should be assigned to someone else
  3. If major, CC relevant one or more developers that you know of, but don't overdo it, especially for those that already seem overwhelmed
  4. Add new information that will make it easy to debug
  5. Ask a developer in your organization about the bug, see if you can get someone to take the bug if it's assigned to someone who doesn't seem to care about it (months of no activity for a normal bug, weeks for a critical bug)
  6. Mark it "Blocking?"
  7. Raise the severity
  8. Add or change a target milestone
  9. If it's really bad, ask on IRC
  10. Known your developers
  11. Use the status whiteboard, especially on reopened or difficult-to-understand bugs. Many bugs collect so many comments that developers naturally delay working on them -- it's a huge amount of work just to sift through all that information. Here's where the status whiteboard is helpful -- it's where you can put any information about the current wisdom that's been collected about the bug, and what it's waiting for. This does two things -- the developers can get a quick summary of what needs to get done, and it reminds them about the bug via a Bugmail if they get bugmails for status whiteboard changes. A valid status whiteboard entry might be "Fixed for testcase 1, still a problem for testcase 2".
  12. Mark with keyword "helpwanted"

TBD: add info about branches vs. trunk and fill out sections on ignored patches and bugs

See Also

Retrieved from "https://wiki.mozilla.org/index.php?title=QA/Triage&oldid=202376"