QA/Fennec/iOS
Contents
Overview
The team focuses on testing current and upcoming releases of Firefox on iOS. Some of our responsibilities include qualifying builds prior to a maintenance or milestone release. Some of the responsibilities include, but are not limited to:
- Verifying bugs using a triage strategy
- Running basic functional tests and smoke tests - mostly manual; but automated in the near future
- Running our manual test suites on localized builds
- Making sure web content is viewable, acceptable and ready for consumption in Firefox for iOS
In addition to qualifying builds prior to a release, we perform ongoing tasks on the development branches to ensure no major problems get uplifted as we merge changes to the next branch. For example, on a daily basis we:
- Confirm new unconfirmed bugs
- Verify bugs on development branches
- Engage community and new volunteers in helping improve Firefox through test day events
- Interact with developers to help them test the features they implement
- Perform exploratory testing on new features while on development branch
- Write test cases in Moztrap for new features
- Monitor crashes to assess stability
- File and track new crasher bugs as they appear
- Manage web compatibility testing through community involvement or crowdsourcing companies that help in testing different iOS device configurations and top-site testing
All of which help assess the quality of individual new features as well as the Firefox version as a whole.
Team Details
The team is lead by Aaron Train who coordinate testing of their respective areas among several MoCo employees and community volunteers. Traditionally new features are tested by several MoCo QA employees. Anyone is welcome and encouraged to participate.
- Lead - Aaron Train (:aaronmt)
Team Members and Assignments
Name | Contact | Availability | Project Assignments |
Aaron Train | atrain@mozilla.com | MoCo Employee (full time) | Lead - New features testing coordination. |
QA Current Release Test Plans
Project Test Plan | Description | Scope | Timeframe |
QA/Fennec/iOS/Releases/1.1 |
QA Test Strategy for Firefox on iOS
Overview
This is a tracking document that outlines the test strategy that Mozilla QA will follow regarding a thorough test pass of Fennec for iOS releases. It will follow the general plan on steps and process that QA will take to accomplish a "sign off" to the release strategy. Mozilla QA strives to uphold quality software, and a successful pass corresponds to no blocking issues, 'Green'-colored test pass on test cases in both execution and automation, and a thorough bug verification window of all Blocking bugs and Critical bugs.
NOTE: Updates to this document continue to change as Mozilla continues to refine development process.
Fennec Testing
As Fennec for iOS is a new project with Mozilla a lot of the test effort early on is to set up infrastructure, define basic processes, build up a QA team and community, and participate in early on decisions that shape the future of the project.
One big issue with Fennec for iOS is our limitations with build and device availability (re: Apple) that is unlike that of Android.
We will need core expertise and unique community involvement needed which will be applicable for iOS testing. There will be core issues to tackle that are not specific to devices and can be done by anybody on the team or by anyone who has expressed interest in the community.
Areas Covered
- Featured Areas Tests
- Web Compatibility Tests
- Accessibility Tests
- Bug Verifications
- Distribution Tests
- L10N Test Deliverables
- Regression Tests
Areas not Covered
- Feature Unit Tests
- These are normally covered by feature developers, not QA. We are open to discussion about what we could do in QA to increase Unit Tests other than what automation currently exists. Need more definition here on what QA/Dev should own.
- String Localization Tests
- See more in the L10n Test Deliverable section.
Schedule
The release schedule is broken up by milestones, determined by the Fennec team. Each milestone will contain feature enhancements, bug fixes, and updates. Most testing of focused areas and regression areas will come in the latter half of the schedule, during the latter parts of alpha up until the final release candidate.
NOTE: We are going to branch from GitHub to create a follow-up 1.0.x and 2.x branch which we will ship future releases from.
Categories
Feature Tests
- Summary
Any major feature item will have a test plan and a tester who owns that area.
- Planning & Scheduling
The Fennec dev team has a great online [Trello board] to track features by milestone. We will work off this list as a starting point for what we focus on for each release. We will create a basic test plan outlining our major focus areas, how we will test and the test cases. These will live here.
The expectation is that the major features are well tested during the Alpha and Bet a phase. Then testing during later betas is focused on verifying bug fixes, testing additional edge cases that are identified (polish testing), and regression testing.
- Test Framework
Automation:
Execution:
- Results
Test Results for executed test cases should be tracked on regular cadence on the wiki.
Web Compatibility Tests
- Summary
The Web Compatibility Tests focuses on regression and compatibility with popular websites. For Fennec on iOS we are using WKWebView (WebKit) so comparative testing against Safari will be of value. Any problems discovered in the browser should be reproducible in Safari.
- Test Framework
- Top Sites - coverage of a few popular sites that drives most user traffic
- Results
Test Results for executed test cases should be tracked on regular cadence on the wiki.
Accessibility Tests
- Summary
For the time being we are not doing any specific tests owned by QA, but post 1.0 we will revisit this to find what we can reasonably do with our product.
Bug Verifications
- Summary
Verifying fixed bugs in Buzilla gives us additional testing on specific bugs that were fixed.
- Planning & Scheduling
To prioritize and narrow the list of bugs, QA will focus primarily on Fennec bugs that represent highest severity.
- Results
Bugs will be marked verified or reactivated for regression. All results tracked in Bugzilla.
Distribution Tests
This is not an area that we need to test in Fennec for the 1.0 release. In the future we might have private builds or specialized versions that we need to test.
L10N Tests
- Summary
Fennec will ship with a number of translations. Currently there are [about 43] languages under development with more coming on board in the future. As of this edit, there are 6 complete.
The l10n community provides the translations and the mobile team does the appropriate packaging and bundling.
- Planning & Scheduling
While traditionally we do not perform translation accuracy tests we should spot check a few languages to ensure there are no major UI failures.
- Test Framework
Execution
- Results
Test Results for executed test cases should be tracked on regular cadence on the wiki
Collecting Feedback
Our main goal here is to make sure we document and test the feedback channels. For Fennec for iOS we will not have the community and tools built up as Firefox does.
- Reporting tools
- Breakpad / Crash Reporter
- IRC: #mobile
- Other
- Apple Store reviews
- Input
Regression Tests
Community Contribution
Anyone can participate. There are several ways in which YOU can participate:
- File bugs
- Triage bugs (confirm existing bugs and assign them to the right buckets)
- Help test new features
- Write test cases
- Plan new features testing
- Help others who want to get involved.
How You Can Contact Us
If you're interested in helping out with anything below (or want to help with something else), please contact us. We hang out on irc.mozilla.org in the channel #qa and in #mobile.
- Mailing Lists
- Newsgroups
Download Firefox for iOS
The easiest and most simplest way to help us by using Firefox for iOS. One can download Firefox for iOS on our temporary XCode integration server and installing it on your iOS device. Note that installation capability are currently limited to provisioned and certified devices.
Resources
Reference Links
- Project overview, definition, phases and further descriptions available on the project page here: https://wiki.mozilla.org/Mobile/Firefox_for_iOS
Additional Resources & Tips
Testing Resources & Documentation
- L10N pre-build requirements
- Pulling and merging translations into builds
- Fennec iOS 11.0 RC 1
- Fennec iOS 11.1 RC 1
- Fennec iOS 11.1 RC 3
- Fennec iOS 11.1 RC 6
- Fennec iOS 12.0 Beta 4
- Fennec iOS 12.0 RC 1
- Fennec iOS 12.0 RC 2
- Fennec iOS 12.1
- Fennec iOS 12.2
- Fennec iOS 13.0
- Fennec iOS 13.1
- Fennec iOS 13.2
- Fennec iOS 14.0
- Fennec iOS 15.0
- Fennec iOS 16.0
- Fennec iOS 16.1
- Fennec iOS 16.2
- Fennec iOS 17.0
- Fennec iOS 18.0