QA/Fennec/X2N Migration/Beta
X2N Migration Test Plan for Beta
- QA Contact: Naoki Hirata IRC: irc://irc.mozilla.org#qa nhirata
- Releng Contact : Aki IRC: irc://irc.mozilla.org#releng aki
- Builds: 10.0.4 ESR XUL Build, Single/Multi | 13.0b1 XUL, Single/Multi | 14a2 Aurora (Beta Candidate)
Google Play Update Testing
Before Beta (750819, 747534, and "test scenario 1"): * RelEng: Verify that a multi-locale XF for tablets can be uploaded alongside many single locale FN builds * QA: Determine what build different localizations will receive depending on Firefox/system locale and available builds * QA: Verify that FN14 is offered to phone XF13 users, XF14 is offered to tablet XF13 users, and no unexpected issues occur * QA: Test the update experience for an Android 2.0 XF13 user, now that FN14 no longer supports them
Note: We ran into some issue with posting multiple locales up on Google Play. We will only be testing multi locale XUL fennec to Single locale English Native Fennec for the first test run.
Migration Testing (Google Play is unnecessary)
Before Beta (750819, builds available): * QA: Verify that XF13 to FN14 migrations (APK over APK) have no new perf/functional/crash regressions across multiple devices
Overview Results
Google Doc Spreadsheet of results
Version to Version | Language to Language | Results |
ESR 10.0.4 XUL to 14 Native - Phone | ||
Single XUL to Single Native | En to En | |
Single XUL to Single Native | JP to En | |
Multi XUL to Single Native | Fr to En | |
13 Beta XUL to 14 Native - Phone | ||
Single XUL to Single Native | En to En | |
Single XUL to Single Native | JP to En | |
Multi XUL to Single Native | Fr to En | |
13 Beta Native to 14 Native - Phone | ||
Single Native to Single Native | En to En | |
14 XUL to 14 Native - Phone | ||
Multi XUL to Single Native | Fr to En | |
Single XUL to Single Native | JP to En | |
Single XUL to Single Native | En to En |
Detailed Results
Phone
Firefox ESR 10.0.4 (XUL) to Firefox 14 (Native)
- Single Locale to Single Locale
- Smoke Test
- L10N Test
- Performance
- Multi Locale to Single Locale
- Smoke Test
- L10N Test
- Performance
Firefox 13 beta (XUL) to Firefox 14 (Native)
- Single Locale to Single Locale
- Smoke Test
- L10N Test
- Performance
- Multi Locale to Single Locale
- Smoke Test
- L10N Test
- Performance
Firefox 13 (Native) to Firefox 14 (Native)
- Single Locale to Single Locale
- Smoke Test
- L10N Test
- Performance
Firefox 14 (XUL) to Firefox 14 (Native)
- Single Locale to Single Locale
- Smoke Test
- L10N Test
- Performance
- Multi XUL to Single Locale
- Smoke Test
- L10N Test
- Performance
Tablet Testing
Firefox ESR 10.0.4 (XUL) to Firefox 14 (Native)
- Single Locale to Single Locale
- Smoke Test
- L10N Test
- Performance
- Multi Locale to Single Locale
- Smoke Test
- L10N Test
- Performance
Firefox 13 beta (XUL) to Firefox 14 (Native)
- Single Locale to Single Locale
- Smoke Test
- L10N Test
- Performance
- Multi Locale to Single Locale
- Smoke Test
- L10N Test
- Performance
Test Plan after Migration
Smoketests
QA Task | Person/Group | How Often? |
Smoketests | Once |
Lite verification tests (or smoketests) will be first run when the builds are ready. Tests are available on litmus under the Fennec 10.0 Test Suite, and the tablet version here. In any case of failure in any Smoketest, blockers need to be raised ASAP before proceeding to the BFTs. QA/Fennec/X2N Migration/Beta/smoketests-results Detailed Smoketest results
Exploratory Testing
QA Assignment
QA/Fennec/X2N Migration/Beta/exploratory-results Detailed Exploratory results
Major Bugs Found
Android Permissions Testing
Verify permissions between what is currently on the store and with this beta build candidate. Details : Android Permissions
QA/Fennec/Fennec/X2N Migration/Beta/permissions-results Detailed Android Permissions Testing Results