Notes for 65.0 release post mortem

10am PT Tuesday February 19 (after the channel meeting) Vidyo Channel: ReleaseCoordination IRC: #release-drivers

Some stats

• 7 weeks on nightly (Oct 22 to Dec 10), 7 weeks on Beta (Dec 11 to Jan 29)
• Milestones for reference: https://docs.google.com/document/d/1vI9KWvSMQ50R9YgdMX7nG7CSuOnTeRGN9sOH_D6_v_c/edit
• 260 uplifts to Beta, 16 uplifts to Release (64: 245 Beta, 13 Release)
  • Uplifts during Beta: https://bugzilla.mozilla.org/page.cgi?id=release_tracking_report.html&is_custom=1&q=approval-mozilla-beta%3A%2B%3A20181210-20190121%3A0%3Aand%3A846%2B%3A
  • Uplifts during RC: https://bugzilla.mozilla.org/page.cgi?id=release_tracking_report.html&is_custom=1&q=approval-mozilla-release%3A%2B%3A20190121-20190128%3A0%3Aand%3A846%2B%3A
• QA stats
  • Out of 22 planned features (including 2 bugworks), 9 features were moved to "future releases"/"backlog"/"on hold"/"no manual QA needed"
  • We received 2 PI Requests after the deadline (Oct 17th), the number was same for Fx64
  • Feature documentation due date was Oct 26th, 4 features missed the deadline
  • Feature readiness date was Oct 31st, 7 landed after the deadline (same number as Fx64), 2 of them were delayed by a month

What went well

• WNP went smoothly!

• Good responsiveness from devs during final week of Beta and RC week

• [tania] Good communication between Eng & QA for most of the features

• QA started AV testing with its first round (P1 cases) executed in beta 5 - https://docs.google.com/spreadsheets/d/1e2FIcFsL04bOOnR2RKnDbpE02tDDfmLFMhrhaxt_1Xk/edit#gid=175931390
  • Test Plan: https://docs.google.com/document/d/17UYcjUFWQ-hIZSZQFXiR1AcaCM9fyseIlCGkGJ0dLY8/edit

• Lower no of unplanned requests compared to previous beta

Challenges

• [Bug 1522019] ESR 60.5 Bing search engine default for some locales
  • Reported by a user testing out the RC1 build on their own
  • Required backout and RC respin
    • [tania] test case added to the ESR suite - https://testrail.stage.mozaws.net/index.php?/cases/view/235172
  • Bug 1516028 also happened on Fennec during Nightly cycle

• [Bug 1523427] Websockets over H2 proxy server issues
  • Rolled out a hotfix via Normandy in bug 1523978
  • Fix was limited to users allowed to enroll in studies, which corporate environments are more likely to disable
  • Fix included in 65.0.1 dot-release
  • Can we have a Normandy hotfix option which is independent from studies?
    • Was a github issue/bug filed for this?

• [Bug 1523701] Avast/AVG SSL MITM issue
  • Reports almost immediately after release
  • Traced back to corrupted Avast certificate entry in root certs DB
  • Met with Avast today to discuss the incident and possible future mitigations
    • [tania] it affects a very specific scenario
    • Since Avast is the top used antivirus product, regression test cases have been added for it
    • https://testrail.stage.mozaws.net/index.php?/cases/view/243399
    • https://testrail.stage.mozaws.net/index.php?/cases/view/243841
    • https://testrail.stage.mozaws.net/index.php?/cases/view/243858

• Privacy issues with experiment code fixed in 65.0.1
  • [Bug 1526387] CFR Addon Recommendations call remote AMO API before clicking "Install"
    • Reported via Reddit
  • [Bug 1525340] Add policy to disable CFR
    • Regression from CFR moving to Activity Stream, used to be controlled via Shield policy
    • Unfixed for 66+ still
    • [Follow-up] Enterprise added to list of considerations for CFR-related items going forward per jdavis
  • [Bug 1522810] TAAR - personalized recommendations appear in private window
    • Reported by QA prior to release
    • Feature rollout delayed

• [Bug 1506495] Color management seems to not working on Mac
  • Reported immediately after regressing patch was landed in Nightly (2018-11-11)
  • Wasn't tagged as regression, prioritized as P3
    • [marcia] One difficulty was the reporter couldn't help with mozregression
  • No further activity until a metoo comment post-release, bumped to P1 and tagged as regression

• Other dot release drivers
  • [Bug 1516665] Crash in java.lang.NullPointerException: at android.app.Notification$Builder.<init>(Notification.java)
    • Difficulties in reproduction and lower volume when reported, spiked in late Beta & post-release
    • [marcia] Comments were not helpful and no STR. Turns out the logcat that snorp posted led us to a speculative fix
  • [Bug 1523635] [Ubuntu] “Enter” is not functional on Google account password field
    • Reported during RC week
  • [Bug 1523696] Clear Recent History dialog is too small, hiding options
    • Bad interaction between Fluent changes and some locales, caused missing UI bits

• Updating Win10 test workers to 1803 on the day of dot release gtb
  • Changes weren't tested on all branches before rolling out
  • wpt failures on Beta (This is still the case for beta 66....)
  • Asserts on mozilla-release and esr60 which impacted dot release gtb (bug 1513145)
  • Project Zero Skia sec bugs
  • Canvas sec bug from external reporter
  • Followup: PaulT's team will monitor security updates

• May have broken Block Autoplay on Firefox for Android
  • https://bugzilla.mozilla.org/show_bug.cgi?id=1528789
  • May want some QA exploratory testing still since bugs were marked WFM with no clear fix
  • Do we have Telemetry data for this?

QA updates

• Analysis of Dot Release: (desktop) https://docs.google.com/spreadsheets/d/1vaDGBxt0d4jOyg8ZE3k_JwwBtjOR-blIWaQJ9DmJZiA/edit#gid=0