QA/FennecESR

From MozillaWiki
< QA
Jump to: navigation, search
Last updated: 2019/05/24

Overview

Purpose

The purpose of this wiki is to serve as a general test plan for testing Fennec ESR Nightly and Beta builds.

Scope

  • Builds should look and act like regular Nightly/Beta builds, with a few exceptions.

QA: *Fennec QA (slack: mobilesv)

Overall Status

  • Done

Testing Summary

Scope of testing

  • As part of the work to move Fennec to ESR, we need to verify that Nightly builds coming from a Beta-level tree function as expected, as this will be the first time we've ever done such a thing.

Environments

Testing will be performed on the following devices:

  • Phones
    • Samsung Galaxy S8 (Android 9)
    • Sony Xperia Z5 (Android 7.0.0)
    • Google Pixel 3XL (Android 9)
    • Huawei P9 Lite (Android 6.0)
  • Tablets
    • Nexus 9 (Android 7.1.1)
    • x86 - Xiaomi Mi Pad 2 (Android 5.1)
    • Samsung Galaxy Tab S3 (Android 8)
    • Xiaomi Mi Pad 2 (Android 5.1)
    • Lenovo Yoga Tablet 2 (Android 4.4.2)

Acceptance Criteria

  • All tests should pass.

Testing details

Test Cases

Nightly and Beta:

Testing days

Date: 2019-05-10

Date: 2019-05-24

Tracking bugs

  • Bugs to track the feature
Full Query
ID Priority Summary Status Resolution Whiteboard
1505538 -- Release x86_64 Fennec Nightly in the Google Play Store VERIFIED FIXED [geckoview:fenix:p2]
1518056 P1 Can't load any pages on Android Q pre-release builds RESOLVED DUPLICATE [geckoview:fenix:p2]
1520261 -- Let ARM64 Fennec 67 ride the trains to Beta RESOLVED FIXED [arm64:m3]
1545204 P3 Make Fennec tests run on ESR 68 branch RESOLVED FIXED
1546320 P1 Create mozconfigs for Fennec beta builds RESOLVED FIXED
1546322 -- Modify merge scripts to not modify Fennec mozconfigs anymore RESOLVED FIXED
1548552 -- Stop shipping Fennec nightly starting 69 RESOLVED FIXED
1548655 -- Figure out Fennec version numbering as different channels ride the trains to ESR RESOLVED FIXED
1548659 P2 Move Fennec tests on mozilla-central to Tier 2 and reduced frequency after the Gecko 69 version bump RESOLVED FIXED
1548864 -- Build and ship Fennec Nightly off mozilla-beta RESOLVED FIXED
1550719 -- Cannot update official Beta/Nightly builds with Fennec-ESR builds RESOLVED DUPLICATE
1551442 -- [ESR builds][Tablet] Make default browser option not displayed on top of the Settings Menu RESOLVED INVALID
1552251 -- Clean up Android chunking rules RESOLVED FIXED
1552573 -- Do something better than using labels for nightly_fennec target task selection RESOLVED FIXED
1553396 -- Firefox Android Nightly download links are 404 RESOLVED FIXED
1553853 -- Rename "nightly-android" hooks to something that clearly states it's just shipping geckoview RESOLVED FIXED
1553877 -- push-apk: Provide default values for -nightly and -beta jobs RESOLVED FIXED
1553880 -- Remove google-play-strings transform RESOLVED FIXED
1554046 -- Fennec Beta: Update Fennec beta version number RESOLVED FIXED
1554048 -- Fennec Nightly: checksums are uploaded in the candidates directory, instead of the nightly one. RESOLVED FIXED
1554051 -- Fennec Nightly: Cannot beetmove to nightly folder: release/candidates scopes are given to the task RESOLVED FIXED
1554066 -- Fennec Nightly: chain of Trust doesn't accept push-apk's aurora scope on mozilla-beta RESOLVED FIXED
1554100 -- Rename "google-play-strings-google-play-strings-nightly" job RESOLVED FIXED
1554211 -- The date is not changed when updating Nightly from Google Play RESOLVED WORKSFORME
1554249 -- Publish Fennec nightlies to the beta track by default RESOLVED FIXED
1554271 P1 Make sure Fennec Beta/Release tests are the correct tier RESOLVED FIXED
1554501 P5 Make push-apk tasks idempotent RESOLVED FIXED
1554851 -- Let ARM64 Fennec 68 ride the trains from Beta to Release RESOLVED FIXED [bcs:p1]
1555959 -- 68 train: Split Geckoview publication from the Fennec one RESOLVED FIXED [geckoview:fenix:m6]
1555982 -- Fennec Beta: ARM builds aren't using the PGO profile RESOLVED DUPLICATE
1556416 -- 68 train: Create Fennec opt beta tests tasks RESOLVED FIXED
1556781 -- 68 train: Fix Fennec beetmover scope on staging releases RESOLVED FIXED
1556784 -- 68 train: Rename `{promote,ship}_fennec` into `{promote,ship}_fennec_beta` RESOLVED FIXED
1558130 -- Fennec ESR: Can't find multi_locale/mozilla-esr68_android.json RESOLVED FIXED
1558243 -- Fennec ESR: push-apk tasks are missing from the graph RESOLVED FIXED
1558258 -- Turn off GeckoView unit tests and HW Android tests on ESR68 RESOLVED FIXED
1558259 -- Switch the Fennec Nightly cron from mozilla-beta to mozilla-esr68 RESOLVED FIXED
1558507 -- Move Fennec Beta to mozilla-esr68 RESOLVED FIXED
1559115 -- Build and ship Fennec Release off mozilla-esr68 RESOLVED FIXED
1561617 -- Ensure automation can handle Fennec ESR build numbers RESOLVED FIXED
1562632 -- 68 train: ac_add_options: Investigate whether `--with-branding` and `--with-version-file-path` can be passed from the Taskcluster payload RESOLVED FIXED
1562633 -- 68 train: Remove Fennec redundant/useless jobs RESOLVED FIXED
1564103 -- version.txt isn't bumped on Fennec Release RESOLVED FIXED
1564450 P1 Ship geckoview on regular basis and allow manual publication thanks to hooks RESOLVED FIXED [geckoview:fenix:m7]
1565863 -- Fennec Nightly download URL for latest version is broken RESOLVED FIXED
1566385 -- Automatically update fennec-nightly-latest RESOLVED FIXED
1575973 P2 [meta] Fennec test cleanup RESOLVED FIXED
1578200 -- esr68: Cannot bump new fennec specific version files RESOLVED FIXED

48 Total; 0 Open (0%); 47 Resolved (97.92%); 1 Verified (2.08%);


Signoff

Criteria

  • All Test cases should pass.
  • All blockers, criticals must be fixed and verified or have an agreed-upon timeline for being fixed (as determined by engineering/QA).

Results

Nightly testing

Date 10-05-2019

  • Fennec-ESR - Armv7 - Nightly 68.0a1 on Google Pixel 3XL (Android 9)
  • Fennec-ESR - AArch64 - Nightly 68.0a1 on Samsung Galaxy Tab S3 (Android 8.0)
  • Fennec-ESR - x86 - Nightly 68.0a1 on Lenovo Yoga Tablet 2 (Android 4.4.2)
  • Fennec-ESR - x86-64 - Nightly 68.0a1 on Xiaomi Mi Pad 2 (Android 5.1)

Beta testing

Date 10-05-2019

  • Fennec-ESR - Armv7 - Beta 68.0b1 on Sony Xperia Z5 (Android 7.0.0)
  • Fennec-ESR - AArch64 - Beta 68.0b1 on Samsung Galaxy S8 (Android 9)
  • Fennec-ESR - x86 - Beta 68.0b1 on Lenovo Yoga Tablet 2 (Android 4.4.2)
  • Fennec-ESR - x86-64 - Beta 68.0b1 on Xiaomi MiPad 2 (Android 5.1)

Issues

  • New:
    • bug 1550719 - Cannot update official Beta/Nightly builds with Fennec-ESR builds
    • bug 1551442 - [ESR builds][Tablet] Make default browser option not displayed on top of the Settings Menu


Example Bug: **{{Bug|1305969}} - Fennec sometimes get stuck in a state where there is a perma-bar of white on the bottom of the screen

Retrieved from "https://wiki.mozilla.org/index.php?title=QA/FennecESR&oldid=1212827"