Bugdays/Bug-verification

From MozillaWiki
Jump to: navigation, search
Please do not edit these pages unless you're on the Firefox team. Your feedback and comments are welcomed on the discussion page.

Bug Verification Day

Bug verification is done to verify that bugs are properly fixed and to check that the fixes don’t yield any new issues. This is a very important part of the QA work and one of the more challenging areas of community involvement.

This event is open to everyone wanting to help make Firefox better. We usually advertise it on QMO and IRC every week, so you don't miss any updates.

Where and when

Ambox outdated.png THIS PAGE MAY BE OUTDATED
This article is in parts, or in its entirety, outdated. Hence, the information presented on this page may be incorrect, and should be treated with due caution until this flag has been lifted. Help by editing the article, or discuss its contents on the talk page.

IRC channel #qa on Mozilla IRC Server every Wednesday, all day long, but you can contribute any other time you wish.

Pre-requisites

Bugzilla

Bugzilla is Mozilla's bug tracking system. This is where bugs are reported and where every change for a bug is stored and tracked.

To mark a bug as fixed or do any other Bugzilla work, you will need to sign in to Bugzilla using a Bugzilla account or Mozilla Persona. If you don't already have an account, you can sign up here.

Mozillians

You might want to join network of our contributors Mozillians (https://www.mozillians.org/) in order to connect with other contributors regarding community events, contribution opportunities, knowledge sharing and so on.

Builds

These are the builds you should verify the fixes on:

You can pick one of these builds, download it, and launch it with a new, clean profile. Then test the bugs from the appropriate list below!

If you have any questions regarding the above or this event, please address them on the IRC #qa channel.

How to Verify bugs

To begin, choose a bug from one of these lists
Follow the detailed steps below to complete the verification
  • First of all, we need to verify if the bug is marked as Resolved Fixed.
  • Take note of the platforms / operating systems affected by the bug.
  • Read the description and comments in the bug to understand the problem.
  • If you don’t understand the bug, comment it asking for whatever additional details you need to understand and verify it.
  • Try to reproduce the bug on a build that is known to be broken (usually any Nightly starting from the day the bug was filed to the day the bug was fixed). You can download the builds from here
  • If you are able to reproduce the bug on a known broken build, download the latest build of the version that is thought to be fixed (usually indicated by the status-firefoxN flags) and try to reproduce the bug. You can find links to these builds in the Pre-requisites section.
  • Take into consideration that the Nightly branch receive updates daily, while the Beta version (which Developer Edition is now based on) only updates twice a week. This means you might have to wait a few days to verify a very recent fix on Beta or Developer Edition.
  • If you are still able to reproduce the bug, provide the details of your testing in the bug report.
  • If the bug does NOT reproduce, then you can mark the bug as verified. Set the “status-firefoxN” (N = Firefox version) flag to “verified” for the version you tested and add a comment indicating the details of your testing. If you verified the fix against the latest Nightly, set the STATUS field to VERIFIED. Ask someone on IRC for help if you don’t have the rights to change the flags or status.
  • When possible, also do same exploratory testing around the fix to ensure nothing got broken by it.
Example of a comment after verifying a bug

Managed to reproduce the issue on an affected build (Firefox 52.0a1, build ID), using the steps to reproduce from Comment 0, on Windows 10x64.

The issue is no longer reproducible on Firefox 62.0a1 (build ID), Firefox 61.0b3 (build ID). Tests were performed under Windows 10x64, Mac OS X 10.11, Ubuntu 16.04x64.


Notes
  • Please verify the bugs on the OS/Platforms they were filed against.
  • Make sure to add [bugday-yyyymmdd] in the QA Whiteboard (or simple Whiteboard, if the QA one is not there) of all the bugs you work on today. That way you will mark the fact that you contributed to this event.
    • If you don’t have the rights to edit a whiteboard, you can simply add [bugday-yyyymmdd] in a comment.
  • If you need any guidance or more information, please address your questions on the channel chat and we will try to help you. You can also take a look at our FAQ page.


Scenarios you might encounter
  • If the issue can’t be reproduced by the tester, a comment must be added asking for more information or just notifying that the issue could not be reproduced. In this case is not relevant verifying the bug on latest builds.
  • If the issue is reproducible only on a specific platform at which the tester don’t have access too, then verifying the issue is also not relevant.
  • If better STR are provided in the comments below Comment 0, then those STR must be used in order to verify the bug.
  • If the build with the fix is not released, we use tinderbox builds, or wait until the build is available.
  • If another issue is found while verifying a bug, we check to see if the issue is not already logged and if not, log a new bug.

Mentorship

In case you have never verified a bug or just need someone to walk you through our process, please contact one of our QA Mentors. These are their names and IRC nicks:

  • Mihai Boldan: mboldan
  • Hossain Al Ikram: ikram

If you can't find them on IRC, or don't want to contact them this way, you can just cc one of them in the bug you want to work on and ask for help.

Hosting a test day

Anyone is welcome to host a bugday or test day focused in an area of interest.

If you're hosting a test day or bug day, please report on its results!

  • Email about the results on the dev-quality mailing list.
  • Follow up in QA or community meetings to thank all the testers. :)


Results

Many thanks to everyone that contributed to these events!

2019

Q3

Q2

Q1

2018

Q4

Q3

Q2

Q1

2017

Q4

Q3

Q2

Q1

2016

Q4

Q3

Q2

Q1

2015

Q4

Q3

Q2

Q1

2014

Q4

Q3

Q2

Q1

2013

Q4

Q3

See Also

Retrieved from "https://wiki.mozilla.org/index.php?title=Bugdays/Bug-verification&oldid=1249877"