Bugzilla:About Bugzilla Quality

Some people might be curious about the quality of Bugzilla code and the development process it goes through, so I (Max Kanat-Alexander, one of the primary developers of Bugzilla) have written this essay for the manager or end-user who might be evaluating Bugzilla or have concerns about open-source code.

Bugzilla is the de-facto standard in open-source bug-tracking because we create a quality product. Our rate of development easily parallels or surpasses commercial firms with more paid developers than we have volunteer developers, while maintaining a well-respected level of quality. Here's how we do it:

Our Development Process

Bugzilla has a far more rigorous and orderly development process than any commercial company I've worked for. This rises out of the necessity of a team that works together over long distances--our administrative procedures must be very exact, or we wouldn't have a product at all.

The first thing I'll say is that our development process happens completely in the public eye. Everything I say here about Bugzilla quality can be fully verified by anybody with the time and interest. In a commercial environment, you must simply take it on faith that there is a process, and that the developers and product managers actually follow that process. But with Bugzilla, you can know and be sure.

Here are some of the details of our process:

Fully-Tracked Changes

Even tiny changes must be fully documented in the bug-tracking system, with full justifications for the change and a specific patch posted directly to the bug report. This rule is enforced largely by necessity--a vast number of our contributors don't have access to our version-control system, and are located across the world from each other. Thus, they must file a bug and post a patch in order for us to even know about it. Contrast this to a commercial development environment where developers are supposed to post changes to the bug-tracking system, but don't necessarily always have to do so in order to actually get code checked in.

Code Review

Once a bug is filed, a patch must be posted to that bug and reviewed by a qualified reviewer. Qualified reviewers are people who have proven their ability to follow our development standards and who themselves write excellent code. All reviewers have been constant contributors to Bugzilla itself for months or years before they become reviewers. All reviewers are approved directly by the Project Leader after nomination at a meeting of current reviewers, and are only approved if their history of writing quality code clearly warrants it.

Patches tend to go through 1 to 10 revisions, based on whether or not they pass review. These reviews are themselves completely documented in the bug-tracking system, along with the rationale for why a particular piece of code was written or why the reviewer flunked the patch on review.

Reviewers both test the actual functionality of the patch and inspect the code for quality. Our policy is that the code in our version-control system must always work. You could check out Bugzilla at any time in our development process and it would be functional as far as we were aware.

Code Approval Process

Once a patch passes review, before it can be checked in, it must be granted "approval" by one of the leaders of the Bugzilla Project, currently consisting of four people who are the world's foremost experts on the internals and structure of Bugzilla, having written much of it themselves. Approval is denied if: the patch is implemented in a way that is in conflict with other pending patches, the patch is implemented in a way that goes against Bugzilla's general architecture, or it's a feature that is definitely not wanted. (Usually unwanted features get caught before this phase, but about 0.1% of the time they make it all the way to the approval stage.)

Test Suite

Once the code is checked in, there is a test suite that runs on various different versions of Perl in various different configurations to assure that the code is not broken and follows our quality standards. Any failure of this test suite will warn the developers, who are forbidden to check any other code into version-control until the problem is fixed.

Our Release Process

Once we reach a certain point in time or on our development roadmap, we begin the process of releasing a new version of Bugzilla. Here's how that process works:

Code Freeze

Several months before a stable release, the code in the version-control system is "frozen" and only bug-fixes can go in to the code. This gives us a several-month stabilization period before a release. We have a well-developed system for discovering and marking which bugs block a release, so we know exactly what needs to be fixed before the release can go out.

Development and Release Candidate Releases

We release several "development" and "release candidate" releases before a final stable release, over a period of many months. This gets our code into testing in real environments. Sometimes even significant installations will run our development code, as it is often stable enough to run in a production environment even before it has undergone QA (though we never recommend this to users, on principle).

Testing on a Major Installation

Usually before a major release, bugzilla.mozilla.org will upgrade to our latest development code. This is a Bugzilla installation with tens of thousands of active users, and over 400,000 bugs filed.

QA

Finally, before any stable release, an exact QA process is done by several QA engineers testing each release in several different environments. There are many automatic tests run by Selenium, a tool for testing web applications. The rest of the tests are done manually per test plans, using an application called Testopia to track the tests and their results.

The QA manager is himself one of the primary developers of Bugzilla.

Any serious bugs found by QA are fixed before the release, and no stable version of Bugzilla is ever released which has not passed QA.

General Documentation About Bugzilla's Process

There are many other aspects to our process for other areas of Bugzilla.

• The general documentation for Bugzilla developers describes specific guidelines and processes for different areas of the Bugzilla Project.
• Bugzilla:Developers and its various links describe, for new contributors, our development process.
• The Developer's Guide describes our code standards.

Advantages of Open-Source Development

In addition to all of the process aspects that I've listed above, there are many benefits to Bugzilla's being developed in the open:

• If you encounter a bug in Bugzilla, and it has been fixed but the release isn't available yet, you can easily get the patch from our bug-tracking system or from our version-control system, both of which are fully available to the public.
• If you encounter an issue that hasn't been fixed, you not only have the code, and can fix it yourself, but you have an entire community of contributors and users who can advise you and help you fix it.
• The primary developers of Bugzilla are available easily and we provide support directly to users. I myself am the most frequent writer on the Bugzilla support mailing list, and nearly all the developers are available through IRC and provide support to the many users who come by. (Note that I myself managed an entire technical support department as the senior technician for a commercial software company before I started my own business providing Bugzilla services, and I am now one of those people providing support for Bugzilla directly to users on IRC and the mailing list.)
• Bugzilla isn't dependent upon a single entity for support or development. In commercial software, if a firm fails and nobody buys their intellectual property, releases are ended forever. With Bugzilla, even if all the developers were to suddenly disappear, development could be continued by any entity. Witness Linux--in the 1990's, various Linux companies went bankrupt, but other successful companies (like Red Hat) were still able to continue and provide Linux as a product, because it's open source.

The idea that open source is unstable or unreliable is an ancient myth, long disproved by industry experience and evidence. Most of the Internet is run by open-source software--the major DNS services and the largest number of web sites. Major corporations such as Google, Yahoo, and eBay depend on open source software such as Linux and Bugzilla for their business and development processes.

Conclusion

Bugzilla is a high-quality product, developed to exact standards, and will remain that way for the foreseeable future.

And if it didn't, then you could fix it yourself.