The QA Checklisting Program

What is this?

The QA Checklisting Program is an ongoing effort to document our processes in a way that will make us more effective as a team and more accessible to the Mozilla Community.

The program involves creating checklists and associated concise documentation for repeated processes, especially those that we expect to share with community members or that are of interest outside our team.

Why should we do this?

Reliability

We are the backstop. More than any other engineering team in Mozilla, it's vital that our processes are predictable and reliably repeatable. We must know exactly what we are doing, and we must do it that way every single time.

Transparency

Other teams need to understand our processes, especially ones that directly affect them. Through documentation, other teams will know what and how much we're contributing. And we can more easily get useful feedback as to whether we're doing the right things.

Accountability

We are ultimately responsible for the quality of Mozilla's products and properties, and must strive to ensure every release is excellent. But bad releases will happen. When they do, we must be able to review what we've done so that we can understand what to do better.

Efficiency

The ramp into Mozilla is a tough one. New employees take much longer to become effective when current process documentation is only available anecdotally. Even experienced employees forget processes they don't perform often and are greatly helped by reminders.

Agility

Specialization enables deeper knowledge—at the cost of bottlenecks. If only one person is capable of a task, they must be available before the task can be performed. By moving knowledge into checklists and documentation, anybody can perform any process as needed.

Community

Community energy is a precious resource, and we cannot afford to waste it. That which makes us effective, efficient and agile makes the community effective, efficient and agile as well. We are the community.

How do we get started?

1. Brainstorm Processes
   
   Brainstorm at the whole-QA level as well as the subteam level. Identify all processes that are performed on a regular basis. Don't attempt to figure out if they should be checklisted yet, just get everything on the table.
   
   
2. Identify Trackers
   
   Create or find a way to track the list of processes so that they can be easily rearranged. It doesn't matter if the tracker is a wiki, Pivotal, spreadsheet, whatever. Use what works. Create a whole-QA tracker and a tracker for each subteam.
   
   
3. Identify Indexes
   
   Create placeholders to index completed checklists on wiki for QA and for each subteam. Don't concentrate too much on initial organization; a flat list for each placeholder is fine. As processes get added, organization will become more obvious.
   
   
4. Prioritize Checklists
   
   Add all processes to the tracker. Arrange the list of processes in the order that they should be checklisted. Guidelines for priority are:
   
   
   • Processes that can be performed by outside community should be higher.
   • Processes that directly involve external teams or the community should be higher.
   • Processes that should be performed as quickly as possible should be higher.
   • Processes with a low margin of error should be higher.
   • Processes performed very frequently should be higher.
   • Processes performed very infrequently should be higher.
   • Processes that are complex or have a lot of steps should be higher.
     
     
5. Schedule Urgent Checklists
   
   Some processes are either vital to the team, or so time-sensitive that they should be documented immediately. These are urgent checklists, and will likely already be at the top of the prioritized list. Move them to the top if necessary, and schedule resources now to create these.
   
   
6. Create Checklists
   
   Create urgent checklists first. Other checklists can be created opportunistically the next time the process is performed, or scheduled off the list in priority order when resources are available.
   
   
7. Add to Indexes
   
   As each checklist is created, it should be added to the appropriate index. If a checklist is shared between subteams but not across QA as a whole, add it to each subteam index that applies.
   
   

What does a checklist look like?

General template as well as example TBA.

Should we checklist everything?

Discretion should be used as to what's important enough to checklist, and what level of detail of supporting documentation should be created.

Documentation requires maintenance, and the more complete the documentation the more maintenance it will cost. Incorrect documentation is worse than no documentation, so do not document more than can be afforded.

However, any processes with the specific high-priority qualities mentioned under "Prioritize Checklists" in How do we get started? should have a reasonably detailed checklist.