QA/Fennec/Support of Progressive Apps in Fennec
From MozillaWiki
Last updated: 2017/12/21
Approvals Required / Received
The following individuals are required to/have approved this Test Plan:
| Name | Title | Department | Approval Date | Method |
|---|---|---|---|---|
| QA Manager | Product Integrity | Date | ||
| Software Engineer | Engineering | Date | ||
| EPM | Product Management | Date |
Overview
Purpose
- The purpose of this wiki is to serve as a general test plan for verifying that users of Android mobile devices can install and use a progressive web app, as a standalone app.
- Full functional and UI test cases will be created and executed.
Scope
This wiki details the testing that will be performed by the Fennec QA project team for the PWA project. It defines the overall testing requirements and provides an integrated view of the project test activities. Its purpose is to document:
- What will be tested
- How testing will be performed
- Work progress and status
User Stories
* As a user I want to be informed when the current web page is part of a web app I can install so I can benefit from a richer app-like experience * As a user I want to launch a web app from an icon on my homescreen/launcher/dock/start menu and use it separately from the browser * As a user I want to switch between web apps using my operating system's task manager * As a user I want to seamlessly link between web app windows and browser tabs * As a user I want to persist this site/app to my device and use it even while offline * As an app developer I want to get my app discovered via web searches because it currently gets buried in app stores across Firefox OS & Firefox * As an app developer I want to deep link into my app from web search results so that I can drive users to my app via its content * As an app developer I want to create repeat engagement with my app by adding it to a user's homescreen * As an app developer I want to create repeat engagement with my app via push notifications
Ownership
Overall Status
[LANDED] Nightly: Nightly 58 (2017-09-22)
[LANDED] Beta: Beta 58 (2017-11-13)
Testing Summary
Scope of testing
All PWA related bugs will be covered and verified, on multiple Android devices, with the supported OS.
Environments
Testing will be performed on the following Android devices:
- Phones
- 4.x
- 5.x
- 6.x
- 7.x
- Tablet
- 4.x
- 5.x
- 6.x
- 7.x
Specifications
- bug 1212648 - [Meta] Support of Progressive Apps in Fennec
- More details about PWAs: https://developers.google.com/web/progressive-web-apps/
- A list of PWAs for testing: https://pwa.rocks/
Acceptance Criteria
- All [Test cases] have passed and all P1 bugs are fixed.
Testing details
Test Cases
Testing days
Date: 08-07-2017
- Test plan creation
- Exploratory testing on Nightly 57
Date: 08-09-2017
- Ran full tests on Motorola Nexus 6 (Android 7.0)
- Logged bugs:
- Bug 1388659 - Permission request for audio and camera on Aliexpress.com
- Bug 1388750 - Web apps - Apps suddenly stopped responding, running out of memory
- Bug 1388747 - Web Apps - show the simplified address bar when the user navigates outside the web app's origin
- Bug 1388760 - Web Apps - Permission request dialogs appear in the browser, not inside the web app
- Bug 1388757 - Web Apps - Device back button sends the webapp in the background
- Bug 1388755 - Web Apps - Text selection toolbar not displayed
- Bug 1388763 - Web Apps - Doorhangers aren't working
Date: 09-27-2017
- Cleaned up old bugs:
- Bug 1321320 - Ability to determine whether a url is within an installed app scope
- Bug 1325021 - Crash in java.lang.IllegalArgumentException: Doorhanger:Add was not registered at org.mozilla.gecko.EventDispatcher.unregisterListener(EventDispatcher.java)
- Bug 1351169 - Crash in java.lang.NullPointerException: Null native pointer at org.mozilla.gecko.mozglue.GeckoLoader.nativeRun(Native Method)
- Bug 1347611 - Web Apps (and custom tabs) flash previously opened content
- Bug 1364044 - Web Apps - Tabs appear in the browser when there are multiple webapps opened
- Verified:
- Bug 1363052 - Web Apps - Re-launching a web app with an internal/external page opened, resets is to the homepage
- Bug 1367077 - Ensure form fill (passwords, etc) working in GeckoView-based custom tabs and web apps
- Logged bug:
- Bug 1403566 - Web Apps - Login auto-complete isn't working, with multiple logins saved
Date: 10-05-2017
- Ran full tests on Nightly 58.0a1:
- Huawei Nexus 6P (Android 8.0): https://testrail.stage.mozaws.net/index.php?/runs/view/6522
- Samsung Galaxy Note 4 (Android 5.0.1): https://testrail.stage.mozaws.net/index.php?/runs/view/6524
- Logged bugs:
- Bug 1406000 - Web Apps - cannot download PDFs (or other files) from pwa
- Bug 1406024 - Web Apps - Date-time/ color pickers don't open
- Bug 1406034 - Web Apps - The aliexpress back button opens Javascript void page
Date: 10-06-2017
- Ran full tests on Nightly 58.0a1:
- Galaxy Tab S3 (Android 7.0): https://testrail.stage.mozaws.net/index.php?/runs/view/6562
Date: 10-25-2017
- Updated test cases
- Verified bugs:
- Bug 1409403 - Update badge icon for PWA
- Bug 1393672 - PWA Badge and Onboarding - Main workflow
- Bug 1409191 - The "Add to home screen" prompt doesn't appear to use manifest values
- Bug 1395841 - implement change for separating “add to home screen” & “add page shortcut”
- Bug 1409303 - Move PWA setting to Setting - General
- Updated bugs:
- Bug 1400605 - Crash in java.lang.NullPointerException: at org.mozilla.gecko.webapps.WebAppActivity.updateTaskAndStatusBar(WebAppActivity.java)
- Logged bugs:
- Bug 1411614 - [PWA] The Add to home screen badge disappearing when the app in is background - Don't keep activities ON
- Bug 1411607 - Should hide Experimental features section when there are no experiments
Date: 10-26-2017
- Ran full suite on Oneplus Two (Android 6.0.1): https://testrail.stage.mozaws.net/index.php?/runs/view/6830
- Verified bugs:
- Bug 1411529 - Context menu functionality in GeckoView based components not working.
- Bug 1369817 - Implement PWA orientation lock
- Bug 1406000 - Web Apps - cannot download PDFs (or other files) from pwa
- Bug 1406024 - GeckoView - Date-time/ color pickers don't open in Custom Tab/PWA if Fennec is already running
Date: 11-16-2017
Ran full tests on Beta 58.0b3 - Huawei Nexus 6P (Android 8.0)
Date: 11-17-2017
Ran full tests on Beta 58.0b3 - Sony Xperia Z5 Premium (Android 6.0.1)
Date: 12-07-2017
Ran full tests on Beta 58.0b9 - Prestigio Grace X5 (Android 4.4.2)
Verified:
- Bug 1421149 - [PWA] Icons in the "Add to Home screen" overlay should not have outline & dropshadow
- Bug 1419245 - A2HS badge is shown on site with broken HTTPS
Date: 12-21-2017
- Verified bug: Bug 1424179 - A2HS badge is shown on site with broken HTTPS (Follow Up)
- Pre-Release sign-off: Green
Bug Work
Tracking bugs
- Bugs to track the feature
- Meta bug: bug 1212648 - [Meta] Support of Progressive Apps in Fennec
72 Total; 2 Open (2.78%); 54 Resolved (75%); 16 Verified (22.22%);
Signoff
Criteria
- All [Test cases] have passed and all major bugs are fixed.
- All blockers, criticals must be fixed and verified or have an agreed-upon timeline for being fixed (as determined by engineering/RelMan/QA)
Results
Nightly testing
- Pass
- TestRail Tests Runs
Beta testing
- Pass
- TestRail Tests Runs
Checklist
| Exit Criteria | Status | Notes/Details |
|---|---|---|
| Testing Prerequisites (specs, use cases) | Done | |
| Testing Infrastructure setup | Done | |
| Test Plan Creation | Done | |
| Test Cases Creation | Done | |
| Automation Coverage | ||
| Performance Testing | ||
| All Defects Logged | ||
| Critical/Blockers Fixed and Verified | ||
| Metrics/Telemetry | ||
| Basic/Core functionality Nightly testing | Done | |
| QA mid-Nightly Signoff | Done | Status: Green |
| QA Nightly - Full Testing | Done | |
| QA pre-Beta Signoff | Done | Status: Green |
| QA Beta - Full Testing | Done | |
| QA pre-Release Signoff | Done | Status: Green |
