QA/Fennec/Upgrade to API 26

From MozillaWiki
< QA‎ | Fennec
Jump to: navigation, search

Overview

Purpose

The purpose of this wiki is to serve for the QA testing done for Fennec after upgrade to API Version 26

Ownership

Developer contact:

QA:

Overall Status

Feature status doc

  • [LANDED] Nightly 63

Testing Summary

  • Google Play will require all app updates to target Android Oreo (targetSdkVersion 26 or higher) by November 2018. Fennec needs an upgrade to API 26, so full-functional, regression testing and stress testing are required to discover any compatibility issues.

Scope of testing

Testing will be focused on:

  • Full functional coverage
  • Regression and exploratory testing
  • Visible performance issues

Environments

Testing will be performed on the following devices to provide a good coverage:

  • Phones - Android versions:
    • 6.x
    • 7.x
    • 8.x
  • Tablets - Android versions:
    • 4.x
    • 5.x
  • Both ARM and x86 architecture

Specifications

Fennec Upgrade to API 26 - https://bugzilla.mozilla.org/show_bug.cgi?id=1349309#c4

Acceptance Criteria

No compatibility or performance issues detected

Testing details

Test Cases

Fennec's BFT (basic functional testing) suite

Testing days

Date: 5/21/2018

  • Created a test plan

Date: 7/6/2018

Date: 7/9/2018

  • Tested the local test build

Date: 7/19/2018


Date: 7/23/2018

Date: 7/25/2018

Date: 8/23/2018

Date: 8/24/2018

  • Sent out [63] [fennec] [feature] Upgrade to API 26 - pre-Beta QA sign off (YELLOW)

Date: 9/28/2018

Date: 10/11/2018

  • Upgrade to API 26 - pre-Release QA sign off (GREEN)

Bug Work

Tracking bugs

  • Bugs to track the feature
Full Query
ID Priority Whiteboard Summary Status Resolution Assigned to Depends on
1463376 P1 --do_not_change--[priority:high] Update to latest Google Play Services version RESOLVED FIXED Andrei Lazar 1385464, 1438716, 1468487, 1476271
1479724 P1 Provide Adaptive icons for Beta and Dev builds RESOLVED FIXED Petru-Mugurel Lingurar [:petru]
1352009 P1 Investigate whether we need to do anything to support larger aspect ratios VERIFIED FIXED Mike Kaply [:mkaply] 1345014
1352015 P1 [meta] Build with Android O SDK (API 26) RESOLVED FIXED Jan Henning [:JanH] 1316934, 1338629, 1352599, 1352855, 1410456
1411827 P1 [FNC][SPT59.1][INT] Crash in java.lang.NullPointerException: Attempt to invoke interface method ''android.view.MenuItem android.view.MenuItem.setChecked(boolean)'' on a null object reference at org.mozilla.gecko.BrowserApp.onPrepareOptionsMenu(BrowserApp.java) (Android O) RESOLVED FIXED Nevin Chen(Not active on Bugzilla)
1365865 P1 Firefox Sync not working on Android O developer preview VERIFIED FIXED :Grisha Kruglov
1392746 P1 [priority:low] Crash in libart.so@0x337849 RESOLVED WORKSFORME
1380266 P1 [FNC][SPT57.3][INT] [Android O] Inconsistency at Back from Settings SubMenu VERIFIED FIXED Jing-wei Wu [:jwu]
1372926 P1 [Android O] Add to Home screen doesn't work on any version of Firefox on Oreo RESOLVED FIXED James Willcox (:snorp) (jwillcox@mozilla.com) (he/him) 1352015
1482733 P2 Geolocation.watchPosition() doesn't work in background on Android O and above NEW
1352011 P2 [meta] Support Android Autofill Framework RESOLVED INCOMPLETE 1330257, 1461961, 1485810, 1515431, 1515435, 1515436
1393165 P2 [Android Oreo] Android icon in URL bar? RESOLVED WORKSFORME
1384866 P2 --do_not_change--[priority:high] (Android O) starting a service while backgrounded requires startForegroundService() and *must* be followed by calling startForeground() RESOLVED FIXED Vlad Baicu 1450447, 1465102, 1479314, 1503739
1278364 P3 [priority:high] App may not work with split-screen message RESOLVED WORKSFORME Vlad Baicu 1259098
1494026 P3 On Android O and above, add link to Android app notification settings in settings menu RESOLVED FIXED Jan Henning [:JanH] 1450447
1444776 P3 --do_not_change--[priority:high] Tab Queue prompt needs to use APPLICATION_OVERLAY window type when targeting (and running on) Android O VERIFIED FIXED Andrei Lazar 1352015
1407046 P3 --do_not_change--[priority:high] Abide by Android Oreo background execution limits RESOLVED FIXED Petru-Mugurel Lingurar [:petru] 1385464, 1474961, 1476237, 1476676, 1476681, 1489634
1368539 -- Firefox Nightly + Android O + Updating RESOLVED WORKSFORME
1368536 -- Firefox Nightly + Android O + Syncing RESOLVED DUPLICATE
1385464 -- --do_not_change--[priority:high] Build with Android support library 26 RESOLVED FIXED Petru-Mugurel Lingurar [:petru] 1352015, 1438716, 1447729, 1468487, 1476635
1408677 -- Sharing image shows icons of disproportionate sizes (on Android 8.0.0) RESOLVED DUPLICATE
1359048 -- Use ShortcutManager to create pinned shortcuts on Android launcher home screen RESOLVED DUPLICATE
1434603 -- --do_not_change--[priority:high] Settings Header not changed when visiting sub-menus VERIFIED FIXED Petru-Mugurel Lingurar [:petru] 1462594
1434648 -- [Android O] Share link/image context menu icons are uneven RESOLVED DUPLICATE
1371815 -- Android Studio 3.0 support RESOLVED WORKSFORME
1450447 -- --do_not_change--[priority:high] Start using notification channels RESOLVED FIXED Andrei Lazar 1385464, 1476966, 1477700
1450450 -- --do_not_change--[priority:high] [meta] Android API 26 Migration RESOLVED FIXED 1278364, 1349522, 1352015, 1384866, 1385464, 1407046, 1444776, 1450447, 1450449, 1451061, 1462019, 1465102, 1465323, 1467461, 1467840, 1473872, 1473885, 1476716, 1493864, 1500906, 1501449
1451061 -- --do_not_change--[priority:high] Review Permissions usage for Android 8 behaviour changes RESOLVED FIXED Andrei Lazar
1461577 -- Risk Assessment: [Fennec] Target SDK version 62 RESOLVED INVALID
1378348 -- Add to home screen not working on Android O RESOLVED DUPLICATE
1465323 -- Abide by Android Oreo broadcast policy RESOLVED FIXED Petru-Mugurel Lingurar [:petru] 1467840, 1473885
1473885 -- Fennec crashes when selecting Samsung PIP option VERIFIED FIXED Petru-Mugurel Lingurar [:petru]
1474961 -- --do_not_change--[priority:high] Change StumblerService to a foreground service when targeting Android O RESOLVED FIXED Vlad Baicu 1450447, 1476315, 1476596
1476237 -- Fix TestFileCleanupController after the Oreo migration RESOLVED FIXED Petru-Mugurel Lingurar [:petru]
1476352 -- Building Fennec fails with crash reporting off RESOLVED FIXED Petru-Mugurel Lingurar [:petru]
1476596 -- Mozilla Location Services enables itself VERIFIED FIXED Vlad Baicu
1372040 -- [FNC][SPT57.3][INT] Make Fennec support AdaptiveIconDrawable in Android O VERIFIED FIXED Nevin Chen(Not active on Bugzilla)
1476676 -- --do_not_change--[priority:high] Crash in java.lang.IllegalThreadStateException: Expected thread 2 ("main"), but running on thread 9658 ("AsyncTask #1") at org.mozilla.gecko.util.ThreadUtils.assertOnThreadComparison(ThreadUtils.java) VERIFIED FIXED Petru-Mugurel Lingurar [:petru]
1476681 -- Crash in android.os.FileUriExposedException: file:///storage/emulated/0/Download/fennec-63.0a1.multi.android-arm.apk exposed beyond app through Intent.getData() at android.os.StrictMode.onFileUriExposed(StrictMode.java) VERIFIED FIXED Jan Henning [:JanH]
1476716 -- Crash in java.lang.SecurityException: COLUMN_LOCAL_FILENAME is deprecated; use ContentResolver.openFileDescriptor() instead at android.app.DownloadManager$CursorTranslator.getString(DownloadManager.java) VERIFIED FIXED Petru-Mugurel Lingurar [:petru]
1476720 -- [priority:high] Persistent notification for "Mozilla Location Service running", which does nothing when I tap it RESOLVED INCOMPLETE Vlad Baicu 1474961
1476754 -- --do_not_change--[priority:high] Crash in java.lang.NullPointerException: Attempt to invoke virtual method 'boolean java.lang.String.equals(java.lang.Object)' on a null object reference at org.mozilla.gecko.CrashReporterActivity.getProfileName(CrashReporterActivity.java) VERIFIED FIXED Petru-Mugurel Lingurar [:petru]
1476900 -- --do_not_change--[priority:high] Can't update Nightly from about:firefox or notification bar VERIFIED FIXED Petru-Mugurel Lingurar [:petru] 1478970
1476966 -- --do_not_change--[priority:high] Constant notification ringing when downloading a file VERIFIED FIXED Andrei Lazar 1482478
1477700 -- --do_not_change--[priority:high] Android media notification appears swiped down after video starts VERIFIED FIXED Andrei Lazar
1478970 -- --do_not_change--[priority:high] Can't update Nightly from about:firefox or notification bar - Android O+ VERIFIED FIXED Petru-Mugurel Lingurar [:petru]
1479532 -- Use more sensible importance level for default notification channel VERIFIED FIXED Jan Henning [:JanH] 1476966, 1477700, 1493864
1349523 -- --do_not_change--[priority:high] Play Video in PIP mode with Android O framework support VERIFIED FIXED Petru-Mugurel Lingurar [:petru] 1384866, 1477041, 1480091, 1480098, 1488691
1480079 -- Check whether all builds require REQUEST_INSTALL_PACKAGES permission VERIFIED FIXED Petru-Mugurel Lingurar [:petru]
1480083 -- Screen doesn't auto-rotate back to portrait after a fullscreen video is stopped - Android O devices VERIFIED FIXED Petru-Mugurel Lingurar [:petru]
1480091 -- --do_not_change--[priority:high] URL bar cut-off at the top on some devices, when returning from fullscreen VERIFIED FIXED Petru-Mugurel Lingurar [:petru]
1482105 -- Constant notification ringing when updating Nightly VERIFIED FIXED Jan Henning [:JanH]
1349522 -- --do_not_change--[priority:high] Provide alternative adaptive-icon with Android SDK O (Beta and Nightly) VERIFIED FIXED Petru-Mugurel Lingurar [:petru] 1352015
1482997 -- "Guest Browsing is enabled" notification now annoyingly blocks the URL bar for several seconds after entering guest mode (or after returning Firefox to foreground) RESOLVED DUPLICATE
1483855 -- "Nightly stats and data" notification needs own channel RESOLVED DUPLICATE Andrei Lazar
1483857 -- "sync is not connected" notification needs own channel RESOLVED DUPLICATE Andrei Lazar
1486432 -- --do_not_change--[priority:high] Leanplum push notifications are not received on devices with Android O or above RESOLVED FIXED Andrei Lazar
1476606 -- Nightly isn't responding / white screen while in other app - Oreo+ VERIFIED FIXED Petru-Mugurel Lingurar [:petru] 1476596

58 Total; 1 Open (1.72%); 33 Resolved (56.9%); 24 Verified (41.38%);


Signoff

Criteria

  • All test cases should be executed
  • All blockers, criticals must be fixed and verified or have an agreed-upon timeline for being fixed

Results

Nightly testing

  • in progress in 63.0a1
  • 07/27 mid-Nightly signoff: YELLOW
  • 08/24 pre-Beta QA sign off (YELLOW)

Beta testing

  • N/A

Checklist

Exit Criteria Status Notes/Details
Testing Prerequisites (specs, use cases)
Testing Infrastructure setup
Test Plan Creation Done
Test Cases Creation Done
Automation Coverage
Performance Testing
All Defects Logged
Critical/Blockers Fixed and Verified
Metrics/Telemetry N/A
Basic/Core functionality Nightly testing Not started
QA mid-Nightly Signoff Done Status: YELLOW
QA Nightly - Full Testing Done
QA pre-Beta Signoff Done Status: YELLOW
QA Beta - Full Testing Not started
QA pre-Release Signoff TBD Status:
Retrieved from "https://wiki.mozilla.org/index.php?title=QA/Fennec/Upgrade_to_API_26&oldid=1202232"