QA/Test Days/Stories
From MozillaWiki
Contents
Summary
The following is a repository of "stories" about testdays. What was tried, what was successful, what wasn't, what was learned, ideas new and old.
I recommend that you include the following information in your story as points of reference:
- Date/Time of your testday
- Subject/Goals of your testday
- Links to testplan or any important documents
- Your name and contact info (ex. email) so people can ask you follow up questions
WebQA 'Live' Test Day
On February 23, 2012 at 18:30, the WebQA team held a combined test day and meetup event. The purpose was to invite web testers from all skill levels to our San Fransisco space to learn hands-on how we test our websites.
- Meetup.com event: WebQA Test Day
- Blog post: Test Day + Meetup = Success!
This event was organized by Dave Hunt.
Desktop and Android Web Apps Test Day
Event Organizer and Date
Organizer: Jason Smith
Date: February 17th, 2012
Time: 1am - 5pm PST
Goals
- Simulate an end-to-end customer scenario with the Web Apps product
- Determine what manual test cases pass and fail
- Verify fixed bugs
- Exploratory test the product
What was tried
- Starting test day organization a week and half ahead
- Utilizing a template built by past test days
- Following a checklist of items posted by Anthony
- Publicizing the test day to many different avenues
- Requesting moderators from waverly and QA staff
- Building test accounts for using the product
- Posting links to major builds, test sites, bug queries
- Asking community members to join Mozillians
- Posting a "how to" for non-obvious tasks
- Indicating known issues with the product
- Posting a key customer scenario to model a demo script with detailed steps
- Asking community members to run documented manual test cases
- Asking community members to verify fixed bugs
- Proposing different areas to exploratory test the product
- Asking community members to free write with including test results & bug numbers
- Being highly active on the channel to answer any question that arises
- Helping new people to testing get ramped up on how to get started on IRC
- Getting major test day documents and processes reviewed by QA staff members
What was successful
- Well-prepared mostly for the test day overall
- Had more enough work for community members to do on the test day
- Publicity was good - Lots of people knew about this test day
- Had enough moderators to answer questions
- Community members were active asking questions
- Very few setup issues with community members in testing the product
- Some community members added themselves to Mozillians
- Many existing resolved bugs were verified on the test day (>10 bugs)
- Free writing - Allowed community members to be more open about what they found, even if they were unsure
- Reviewing the test day before starting - caught issues before the test day
- New people seemed interested and helped out on the test day
What was not successful
- Lots of issues brought up in free writing, but few bugs filed
- Configuration issues with the build and deployment on test day
- Some areas were unavailable to test since the code was not ready
- Mostly desktop testers, not that many Android testers
- Did not seem like community members saw that the manual test cases spreadsheet had multiple sheets to it
- Most tests ran seemed to reflect one of the sheets typically
- Even with documented known issues, community members hit them and did not realize it was a known issue
- Facebook integration for test days was found to have some issues with QMO integration
