« QA/Firefox3/TestPlan

Post Mortem Notes

The following notes are a collection of feedback that the QA team has faced throughout the Firefox 3.0 development timeline.

Please provide some solid examples also. This isnt meant to be a bitching session, but hopefully a good chance to voice your concerns and look for ways to improve the process.

What worked

• 8 alphas, 5 betas, and 3 RC builds. With 16 builds leading up to the final release, i felt there was lots of adequate testing on solidifying the release.
• Colloborative efforts by build, QA team, and web team on each milestone release. As we had more, we moved quicker and more careful on our testpasses.
• Having a blocking-firefox3 or blocking1.9 flag was key to getting the right bugs triaged. Knowing they were getting triaged each day was comforting that they werent just sitting in the queue.
• Weekly firefox meetings with a structure and progress report for each team.
• QA created a detailed-tracking document for results and task list
• Some items were well documented (content handling, download manager, addons manager)
• QA's push for severe bugs in an RC push was well recieved, but it was a hard push since the bar was higher
• Developers were generally responsive to QA
• Some teams had their own status meetings (Mac meetings, leaks meetings, performance meetings)
• QA began creating own builds (debug builds, leak guage, tryserver builds)
• At first, crash bugs weren't getting much attention and devs were defensive, bug they warmed up over time and cooperated.
• Devs scheduling pushed into QA time initially, but began accepting QA dates.
• Build team was responsive, automation tools helped speed up process.

What didnt work

• Visual Refresh Landings came very late in release cycle: bug 430217 - Reporter shouldn't add a toolbar button to Firefox; is a good example of how poorly we handled theme refresh planning. All along I was raising the issue that the Reporter icon does not look right since it wasn't styled on the Mac theme. It was only a day or so ago that they won't fixed the bug to style that icon, so now in order for the theme to look right I feel it needs to be removed. But as you can see from Axel's comment he is not happy because of string changes.
• Up to date Icon Refresh Document: Not having an up to date icon refresh document that is up to date makes it almost impossible for QA to figure out what is supposed to be correct in the UI regarding icons, such as the example cited above. Also I found out yesterday in reading bugmail that we will not have a crash reporter icon for final ship (currently it is the same as software update, which is confusing.
  • Every time a theme drop changes on the Mac, I don't have a clear picture of what is supposed to change. The find bar looks different today, but since I have no schema to go by, I am unsure as to whether the changes are expected or not. This results in possible unnecessary bugs, such as the one I filed the morning about the window sizer in Places (bug 430493) where I am not sure whether that is an intended change or a regression.
  • Some community members such as adelfino are obsessed with these minor UI changes and are clogging up bugzilla with hundreds of bugs regarding UI issues, especially on Linux on Mac. Many end up being duplicates and consume both community and QA time to resolve.
• Features Not clearly documented or easy to test: I had ownership of Offline and there was no documentation and it was hard to test the feature with limited test cases from the developer and no real world sites to test.
• Some features were not well documented (eg. UI Changes, Offline Browsing, Awesomebar)
• Hendrix feedback grew with comments and questions on features that weren't complete, nor where there any tracking documents
• PRD wasn't updated throughout the design changes later. They kept iterating as the design was changed even up until RC stage (eg. Awesomebar)
• In many cases, QA wasn't informed when features or major changes were landing (eg. Product pages, Cairo changes landing, Places Changes)
• Things weren't always unit-tested on all platforms (eg. Offline browsing developer tested on mac only, but ended up finding a regression in windows after checkin)
• Wasn't always aware of what status the other features were tracking at.
• QA didnt Get involved with security reviews earlier; loses a lot of potential bugs
• Web dev at release was nebulous. No one knew when pages were to be posted, who was coordinating, etc.

Things to improve on

• Earlier interfacing with developers and testers. More cross-knowledge sharing of architecture and feature knowledge, so questions are asked earlier, testplans worked on earlier
• Producing more tryserver builds earlier on. Helps QA verify fixes in a more 1-1 setting
• Keep the PRD up to date! An outdated PRD for certain sections, kept QA from knowing what new features were landing and obscuring out testplans.
  • Also, having a better ETA on dates when these features were going to land would help QA know when to expect them in builds
• more communication and updates across the board (eg. graphical changes)
• status meetings has details, but a dashboard or quick status would be easier to take in.
• Flag major changes for the week so QA knows ahead of time.
• Status meetings were inconsistent (some developers were there every week, but others were not avaliable for update status)
• earlier accessibility UI reviews in the process (awesome bar had huge adjustments late in the game)
• Include QA into more review discussions
• Don't jam so many changes into a release (web changes, animated favicon -- broke release)
• More automation for BFT's