QA/Bug Triage Guidelines
Revision History
This section describes the modifications that have been made to this wiki page. A new row has been completed each time the content of this document is updated (small corrections for typographical errors do not need to be recorded). The description of the modification contains the differences from the prior version, in terms of what sections were updated and to what extent.
Date | Version | Author | Description |
---|---|---|---|
04/22/2016 | 1.0 | Rares Bologa | Created first draft |
Overview
Scope
This document has the purpose of improving our team’s work quality in regards to Mozilla bug triage process. Intended audience: All team members of Mozilla QA Engineering Team and any third party members interested in this process.
Ownership
Manager, Firefox Quality Engineering - Ryan VanderMeulen
PM, QA Engineering Team - Rares Bologa
Components, Contacts and Suggested Comments
Bugzilla Component Ownership
You can find a list of Bugzilla component ownership for bug decision here
Bug Triage Component Selection: Firefox & CORE
Information (description, keywords, additional notes..) about the components under Firefox & CORE can be found here
Suggested Comments, Contacts & Actions
IRC nicknames for contact persons in regards to Bug Triage
Note: this is out of date
IRC nickname | Name | Reason for contacting |
---|---|---|
mhoye | Mike Hoye | Ping Mike for bugs where the Reporter may be difficult and he will take a look into the issue |
RyanVM | Ryan VanderMeulen | Clearly mark bug with comments when they later on need to be discussed with Ryan |
Different actions to be taken for bugs
State/Case | Action |
---|---|
Component | Where a Component is set to something other than Untriaged, leave the bug alone unless asked to do MozRegression or QAWanted |
Untriaged vs. Unconfirmed | The goal is to reduce the Untriaged issues number, by placing them under the correct component. |
New | Where there is no Component, set one. Where it cannot be reproduced, ask the Reporter for additional information or close as Incomplete / WFM
NOTE: Sometimes, reporters mark issues as New without being confirmed by a Mozillian or Community Member. Read carefully to see if it needs confirmation to remain New or it needs to be changed back to Unconfirmed until it can be reproduced |
Crash | If there is no activity by Mozilla, ping or email Emma (:emceeaich) so that she can get someone to look into the issue. Place the crash tag also. |
Mozilla Communication | When there is a lot of Mozilla Communication within the bug, ping a developer on IRC and ask him to set the appropriate component |
Close Me <date> | Apply at your comfort, there is no standard. If you apply it, please give the reporter a chance to respond (two weeks) |
Enhancement & Feature Requests | Enhancements and Feature Requests should have an appropriate Component assigned. The developers and PM in that component will decide if it will be implemented or marked as Resolved-Won't Fix
NOTE: If you are sure that a Feature Request or an Enhancement is a valid one, you could also set its status to NEW, but usually this is the component owner's job. |
Context | Bugs that contain sensitive information or that may be harmful to the testers should be marked Context. These items can be reviewed with Ryan or Emma |
Suggested responses and comments
Action for bug | Responses/Comments |
---|---|
Not reproducible | I have tested your issue on latest FF release (version) and latest Nightly build and could not reproduce it. I have <what and how it has been tested to eliminate the cause of faulty testing>
<Ask additional details regarding environment if needed> Is this still reproducible on your end ? If yes, can you please retest this using latest FF release and latest Nightly build (https://nightly.mozilla.org/) and report back the results ? When doing this, please use a new clean Firefox profile, maybe even safe mode, to eliminate custom settings as a possible cause (https://goo.gl/PNe90E). |
Reproducible | <User Agent>
I have tested your issue on latest FF release (version), latest Nightly build (buid ID) and managed to reproduce it. I have <what and how it has been tested> Reproducible also on: version + build ID (if necessary) |
WFM | For the case when reporter commented that issue is not reproducible for him:
1. Based on Comment <comment number> from the reporter, changing status to RESOLVED WORKS FOR ME. For the cases when reporter did not added requested info and we cannot reproduce (bug has explicit STR):
If anyone can still reproduce it, feel free to reopen the issue and provide more information. |
Incomplete | Marking this as Resolved: Incomplete due to the lack of response from the reporter.
If anyone can still reproduce it on latest versions, feel free to reopen the issue and provide more information. |
Enhancement | This seems to be more of an enhancement than an issue in my opinion.
However, I am assigning a component to this issue in order to involve the development team and get an opinion on this. |
Crash not encountered anymore | <User Agent> + "I have tried to reproduce this crash on latest FF release (versions) and Nightly build (build ID) but without any luck." <- optional sentence if there are STR or testcase
There are no crash reports in the last 7/14/28 days in the crash signatures provided, therefore closing this bug as Resolved: Works for me. Please reopen the bug if the crash still reproduces and provide more information. |
MozRegression: When we ask a user to use MozRegression | Given the fact that the issue seems to be related to something that is specific to your setup, could you please try to find a regression range using MozRegression tool?
Information on the tool is available at http://mozilla.github.io/mozregression/. Please don't hesitate to contact us if you encounter any problems. |
MozRegression: When we find it is fixed | It was confirmed broken in version xx but it is fixed in Nightly <version>
Use MozRegression to find where the fix occurred - If time permits, probably lower priority |
MozRegression: When we find it broke | If it was working in version xx and it is now broken the MozRegression should be applied to find where that break occurred |
Lessons Learned
Below you can find a couple of lessons learned that we've gathered from the beginning of our bug triage process.
About | What To Do |
---|---|
Old builds needed in zip format | If you have to test old release builds but you don't want to install them, you will need a ".zip" format. But that can be hard to find when it comes to old builds.
What you can do is to download the installer form "http://archive.mozilla.org" and have an archive manager program like "7zip" installed (freeware program). Open the folder where the installer was downloaded and right click it in order to use 7zip program to extract the installer. After doing that, you will have a folder where the Firefox build is extracted and from where you can start the actual version without installing it. As a suggestion use the command window to open it with a different profile and with no remote setting. |
Mac OS Update Privileges | When choosing to update a software on Mac OS platform, the behavior is different than on other OSes.
It will prompt the current user for the credentials of the user that installed the software. So, if the administrator installed the software, and the software is available also on a 2nd user desktop, updating the software should and will request the administrators password in order to update it. If the software that you are trying to update was installed by the current user on which you are logged in, it will require the current user credentials in order to update it. |
Using MozRegression Tool on Mac OS | If you need to find when an issue was introduced (regression), the easiest and best way to do this is by using the MozRegression tool.
In order to do this, you'll need to install a couple of things before starting the tool. Here are the steps needed depending on OS needed for testing: • Mac OS X (10.11 only, for older versions see next bullet) In order to use MozRegression tool on Mac OS X 10.11, you will need to perform some additional steps than on other Mac OS versions. One of the steps is installing a virtual environment trough python so you won't get any conflicts between python and MozRegression. Here are the complete steps to install and use MozRegression trough Virtual environment: 1. First, you will need to open a Terminal window and enter the following command "sudo su" (without quotes). You will be prompted to insert the PC user password (the password used for logging in on the Mac OS user). If you don't see the characters you are entering, don't worry, they are entered but due to security are hidden. After entering them press the "Enter" key. • Mac OS X (10.10 and under) 1. First, you will need to open a Terminal window and enter the following command "sudo su" (without quotes). You will be prompted to insert the PC user password (the password used for logging in on the Mac OS user). If you don't see the characters you are entering, don't worry, they are entered but due to security are hidden. After entering them press the "Enter" key. If something has been missed you can find all the mentioned information also at this link -> http://mozilla.github.io/mozregression/quickstart.html. NOTE: MozRegression tool is not perfect and will throw some errors on some old builds. If it is your case, please refer to my other lesson from above this where you have information on how to manually create the "Pushlog link" for the found builds. If you encounter some errors involving Python while trying to use MozRegression on Mac OS, please look at the next lesson to resolve this issue. |
Instaling virtual Environment for MozRegression to work on Mac OS | It seams that nowadays, not only the Mac OS 10.11 has some conflicts between python and MozRegression when trying to do a regression window. In order to resolve these conflicts, you will need to install and use MozRegression trough a virtual environment. Here are the steps you need to follow in order to do that:
1. First, you will need to open a Terminal window and enter the following command "sudo su" (without quotes). You will be prompted to insert the PC user password (the password used for logging in on the Mac OS user). If you don't see the characters you are entering, don't worry, they are entered but due to security are hidden. After entering them press the "Enter" key. |
Using MozRegression Tool on Windows | On Windows platform you can either use the MozRegression gui from github releases page, or you can use the tool trough command line as on Mac OS. In the next steps I will present the command line way on Windows.
1. First, you will need to download and install Python 2.7 from ActiveState website (http://www.activestate.com/activepython/downloads). If I missed something you can find all the mentioned information also at this link -> http://mozilla.github.io/mozregression/quickstart.html. NOTE: MozRegression tool is not perfect and will throw some errors on some old builds. If it is your case, please refer to my other lesson from above this where you have informations on how to manually create the "Pushlog link" for the found builds. |
Using MozRegression Tool on Linux | On Linux platform you can either use the MozRegression GUI from github releases page, or you can use the tool trough command line as on Mac OS. In the next steps I will present the command line way on Linux.
1. First, you will need to open a Terminal window and enter the following command "sudo su" (without quotes). You will be prompted to insert the PC user password (the password used for logging in on the Linux user). If you don't see the characters you are entering, don't worry, they are entered but due to security are hidden. After entering them press the "Enter" key. If I missed something you can find all the mentioned information also at this link -> http://mozilla.github.io/mozregression/quickstart.html. NOTE: MozRegression tool is not perfect and will throw some errors on some old builds. If it is your case, please refer to my other lesson from above this where you have information on how to manually create the "Pushlog link" for the found builds. |
Using MozRegression Tool with "--find-fix" attribute | If you are dealing with an issue that still reproduces on latest Firefox release build but no longer on Beta, Aurora or Nightly you can use the "--find-fix" attribute while performing regression range in mozregression. This will help you to provide information to the reporter regarding his issue. In most of the cases his issue will end up as "duplicate" or "works for me" since the problem was treated in a different bug.
1. First, you'll need to manually find a bad Nightly build date by downloading builds from http://ftp.mozilla.org/ . For quick testing I suggest to download zip-ed builds, extract them and use the command window / terminal to launch the build with "-no-remote" and "-p" commands.
In Windows you just need to navigate to the extracted folder, hold Shift + right click on free area of the window, then click the "Open command window here" option. After the cmd is opened, write "firefox -no-remote -p" and hit "Enter" key. This will open a separate instance of Firefox with the profiler window, where you can choose to use a no update profile created before (will not update an older version of FF). After that, just test the build and see if it's reproducible.
On Mac OS, you can install the build, and in order to open it with a no update profile, you need to navigate ti instalation directory manually from terminal window (/Applications/FirefoxNightly.app/Contents/MacOS/firefox -no-remote- -p) 2. After you find a bad build, the next step would be to open a command window and start using mozregression by typing "mozregression --bad year-month-day --good year-month-day --find-fix". You need to be sure that bad build is older than good build, otherwise this won't work.Usually the good build would be the latest Nightly so use that date, otherwise it's a regression issue that needs to be treated like in previous lessons. The "--find-fix" atribute in mozregression does exactly what bisect does in normal usage. It searches for the exact push that fixed the issue. In most of the cases you end up with only one issue that you can provide to the reporter of your triaged issue as more information. |
Using MozRegression Tool with targeted profile for each downloaded build | Sometimes there are cases when you are unable to reproduce an issue on MozRegression downloaded builds. But if you manually download the same build and test it, you can reproduce it without problems. This could possibly happen because MozRegression has a different way of storing downloaded builds on your PC and it seams it does not uses a profile when opening the build to test. This case can be avoided by targeting a previously created profile that will be used for each downloaded build. Here are the steps to follow:
Windows: 1. First you'll need to have a good build date and a bad build date as for any other regression window checked. Mac OS X: The steps are the same, only the path is different. Profile folders are in one of these locations: ~/Library/Application Support/Firefox/Profiles/<profile folder> ~/Library/Mozilla/Firefox/Profiles/<profile folder> The tilde character (~) refers to the current user's Home folder, so ~/Library is the /Macintosh HD/Users/<username>/Library folder. (--profile=/Users/pauloiegas/Library/Application\ Support/Firefox/Profiles/rg0ayjkt.PrintTESTn) After this steps, you are all set and can perform the regression range as normal. By using the "--profile <path>" attribute, each downloaded build trough mozregression, will use the same profile and store data on it. This flow can be used also with "--find-fix" attribute as mentioned in the above lesson by adding it after the default command contents and the "--profile <path>" one (eg: C:\Users\paul.oiegas>mozregression --bad 2015-06-06 --good 2016-02-23 --find-fix --profile \AppData\Roaming\Mozilla\Firefox\Profiles\w9gzt0rd.testrelease2). |
MozZRegression on Android (with Windows) |
If you encounter a regression bug on Android OS, you will need to perform a couple of steps in order to be able to run MozRegression tool on an Android device. Here are the steps that you need to follow on Windows in order to find a regression range. Will update the lesson with steps also for Mac OS and Linux after I will find how to set the "Environment variables" on each of them. 1. Install the latest Android SDK with Android Studio from (http://goo.gl/OPyvqU ) or only the command line tools from next link
Windows - http://dl.google.com/android/installer_r24.4.1-windows.exe
Android studio contains also the SDK needed for this to work, but it's also a coding and debugging software for android with a friendly user interface. But if you don't need this, you can install just the command line tools. 2. Install the latest Java JDK from Oracle website (http://goo.gl/rknzzs). It should be straight forward, no extra settings needed. 3. For best results install PDA Net software in order to have most of the Android devices drivers installed (http://pdanet.co/). However, there are cases when this software does not contains the drivers for all the new devices. For example there have been reported that LG G4 driver is not contained. In this case you will have to manually search and install the driver for your device. 4. Next step is a bit trickier and you must pay attention at what changes you are doing.
- First you must navigate to "System" window in Windows (right click on My Computer + Proprieties).
- Chose the "Advanced System Settings" option from left side of the window, and from the bottom of the displayed window select "Environment Variables" option.
- In the "Environment Variables" window, you need to add first a new "System variable". Click the "New..." button and add "ANDROID" for variable name and in the variable value field, you need to add the path where you have installed the SDK (eg: C:\Users\profile_name\AppData\Local\Android\sdk where "profile_name" is usually your PC user name). After that click "Ok" button to save the variable.
- Next step would be to edit the "Path" system variable (double click on it), and add two new strings. First is "%ANDROID%\tools\" and second "%ANDROID%\platform-tools\". Hit "Ok" button for every window and you are all done with this step. 5. Next you will need to install MozRegression tool by using the steps from the above lessons learned or to use the MozRegression GUI that you can download from here (https://goo.gl/DyFFok). 6. On your Android device be sure you have the "Developer options" enabled. Tap multiple times on "Build number" area in "About device" section of device "Settings" page. Return to device "Settings" main page and from the "Developer options" that are now displayed, switch to "On" the "USB debugging" option. 7. After this, you are almost good to go. Connect your Android device to the PC, run a command window (make sure you have the latest mozregression version "pip install -U mozregression") or the Mozregression GUI.
For the command window mode, the only thing that is different from the PC testing is the "--app=fennec" command that you need to introduce before good & bad dates. The command will look like "mozregression --app=fennec --good 2015-05-06 --bad 2015-07-09". Notes and Tips & Tricks: |
Manual regression and manual creation of Pushlog link | When Mozregression tool fails to work (errors when trying to open the builds for test), or when you cannot use it because you need to perform some steps outside the browser before you can test an issue, all you can do is manual regression testing. But in order to help locate the issue you will still need the "Pushlog link" that points to the exact pushes that were made in the specific day when issue appeared. In order to create the pushlog link manually, you will need the next things:
1. Latest good build and the first bad build where issue appeared.
|
Performing quick Regression test by searching in build release notes | In cases when reporter mentions that in a previous version of Firefox his issue was not present and after updating it appeared, you can also do a quick manual regression by searching the problematic build release notes and observe the changes that have been made on it. To better explain this process I will refer to an Issue that I triaged, where the reported problem was an audio malfunction in an html5 game that he and his team develops, that appeared after updating from Firefox 39.0 to 40.0.
In order to find the build release notes you can easily search on google for "Firefox release notes" and open the "https://www.mozilla.org/en-US/firefox/releases/" page. In this page you can view all the release notes from version 0.1 to the latest release (42.0 in the moment of writing). In my case I chose "40.0" version (https://www.mozilla.org/en-US/firefox/40.0/releasenotes/). After choosing the build, you are redirected to a page where all the major changes that entered in this build are mentioned. By looking trough the changes I observed that there were 2 changes on "html5" technology and one of them was related to "Audio Buffer". In my case the change also had a link that pointed to a page with detailed explanations and code chunks to better explain the change. This seamed that could be related a lot with the reporter problem. In my bug reply comment I mentioned the exact link of the change and asked him or his development team to look into it, since it may be the source of his problem and report any findings. This quick search could reduce a lot the time we invest in trying to set up the environment and reproduce the issue. This procedure was reported to be one of the most used in the Firefox Desktop Validation team when triaging a bug that appeared from a version to another. |
CentOS's | CentOS 6 provides only GTK+ version 2. GTK+ 3 is officially supported by CentOS only in CentOS 7.
This is a change that was made for version 43. https://www.mozilla.org/en-US/firefox/43.0beta/system-requirements/ https://www.mozilla.org/en-US/firefox/42.0/system-requirements/ |
How to deal with bugs that are reproducible between the latest official release and the latest Nightly | One of the possibilities when working on the Bug triage task is to encounter bugs that are reproducible between the latest official release and the latest Nightly (for example, the bug is reproducible only on the latest Aurora). In this case we take the following actions:
1. Confirm the bug. |
How to simulate bad internet connection on Mac OS | If you encounter an issue where the reporter specifies that he has a slow / bad internet connection, or you think the issue appears only when internet connection is not good enough, you can simulate this kind of environment by using a bandwidth throttling tool.
Here are the steps you need to follow to install and use the tool under Mac OS: 1. First you will need to navigate to https://developer.apple.com/downloads/ and if you don't have an apple ID, you should create one. Please REMEMBER that in order to return to the normal internet bandwidth speed, you will always need to switch the Tool to "OFF" after testing, otherwise the limitation will remain always ON. |
How to simulate bad internet connection on Windows | If you encounter an issue where the reporter specifies that he has a slow / bad internet connection, or you think the issue appears only when internet connection is not good enough, you can simulate this kind of environment by using a bandwidth limiter tool.
Here are the steps you need to follow to install and use the tool under Windows: 1. First you will need to to navigate to https://netbalancer.com/download page and download the latest version of the NetBalancer software. The unregistered version is limited to a maximum of 3 process priorities/limits and 3 rules at a time. Please REMEMBER that in order to return to the normal internet bandwidth speed, you will always need to reset all the limitations from the "refresh" like button from the software toolbar after testing, otherwise the limitation will remain always ON. |
General Bug Triage process | - Close the NeedInfo bugs for which the reporter has not come back with a reply within 2 weeks (previous period was 1 month) - but keep the NeedInfo flag alive, just in case the reporter will come back to it
- Don’t use the ‘Reporter’ expression inside bugs when you’re addressing him/her. Instead, try using his/her name or Bugzilla username. (applicable for reporters with an intelligible name/email address) The desire here is to make the communication more personal. You can still use ‘reporter’ word inside comments when you’re not directly addressing him. - If a certain developer has started working on an untriaged bug (has relevant comments inside it, changed bug’s status…) and we’re not very sure to what component to assign this bug to, first expected action would be to ask directly the developer for it (either he tells you the component or set it himself). Just in case you’re not receiving any reply from the developer, it’s ok to bring this bug into Ryan’s attention. - For old bugs (ex: 3.2k list ones) – if the bug still reproduces and we consider it a high impact – it is ok to set comments for it. If the bug still reproduces but is lower severity, we should not add new comments to it (just change the status if case requires it). This change has come as a complain from several developers that they’re getting spammed with bug triage work for not so relevant bugs. If the low severity reproducible bug has by any chance Priority P1, we should decrease it. Applicable only if P1 has been set by the reporter. (if it has been set by dev or somebody else from Mozilla, please leave it as is) If the bug is a web compatibility one and we’re not being able to reproduce it on Chrome, then it’s ok to set a comment for it (even though it might have a lower severity). - take enough time to try to reproduce the bug and only when we could not reproduce it or don’t have enough info to reproduce it, then ask the reporter to try other cases. We should explain the reporter what we have tried, on what version of Firefox we tried to reproduce and then ask for more information if we cannot reproduce the bug (or ask him to try to reproduce the bug in safe mode or with clean profile). - Make sure to set the release status flags appropriately, and nominate release tracking flags if there's any uncertainty about whether the bug should block. |