B2G/QA/2.2/2015-03-16Status

V2.2 Highlight & Risk

• Highlight
  • Availability of Flame-L is unknown.
  • Currently we are running regression test for FLAME-KK, NEXUS-5-L.

V2.2 Milestone

• 2015/02/23 V2.2 FL
• 2015/04/06 V2.2 FC

• QA Shared calendar
  • You can add it to your Google Calendar view by clicking on this link and then clicking on the small blue "+ Google Calendar" button at the bottom, right hand of the screen.
  • If you are signed in with your @mozilla.com account, you can also access it this way:
    • In Google Calendar, you should go to Other calendars > Browse Interesting Calendars > More > Resources for mozilla.com > Public (9) > Public - FxOS QA Schedule > Subscribe

V2.2 Testing Scope

• Focus on bugs in agreed list.
• B2G V2.2 Overview Page for more info.
• Current feature-b2g=2.2+ 8 bugs
• Current blocking-b2g=2.2+ 87 bugs

V2.2 Testing by Type

Smoke Testing

Flame-KK On Track

• OTA issue - Bug 1142778 - OTA for Flame Kitkat 2.2, 2.1, and 2.0 builds does not download the latest Nightly builds., 2.2 OTA is okay on 3/16.

• Existing Bugs Breaking the Automation smoketests: Flame-KK
  • [Disabled] test_setup_and_send_active_sync_email.py - Bug 1028192 - Problems with ActiveSync accounts
  • [Disabled] test_receive_active_sync_email.py - Bug 1028192 - Problems with ActiveSync accounts
  • [Disabled] test_cost_control_data_alert_mobile.py - Bug 1061698 - Need ability to work in the shadow root/shadow DOM
  • [Disabled] test_call_log_all_calls.py - Bug 1096385 - Investigate failure in test_call_log_all_calls.py, call log is displayed erroneously
  • [Reproduced manually] test_ftu_skip_tour.py - Bug 1134445 - [FTU] About Firefox OS page has "Yes! Send data." checked primarily.
  • [Reproduced manually] test_IMAP_email_notification.py - Bug 1133446 - [Email] No new email notification - NEED Patch uplift - tree cluster error, will check

Nexus-5-L On Track

• Daily Reports can be found here.
  • Manual - are available on Google Drive & fxosqa-report-watchlist.
  • Automation - are available on Github.IO & fxosqa-report-watchlist.

Regression Testing

• RTL/V2.2 Flame-KK
  • Moztrap result - 76 fail, 326 pass, check with Josh for detail.

• Full Regression/V2.2 Flame-KK
  • Moztrap result - 25 fail, 2011 pass, 52% complete

• Partial Regression/V2.2 Nexus-5-L
  • Moztrap result - 17 fail, 1103 pass, 82% complete

• Bluetooth PTS/V2.2 Flame-KK
  • Meta dependency tree - 3 open bugs

Functional Testing

Yellow

• QA focus - 42 bugs found.

• General (eric)
  • Flame-L might be late for FC, roughly 6 weeks or more needed after SOW (SOW, legal review, porting, build time), PM will have an internal discussion to decide what to do.
  • feature-b2g|blocking-b2g == 2.2?|+ [not resolved] http://mzl.la/1C4X1KT Check if code uplifted
  • feature-b2g|blocking-b2g == 2.2?|+ [resolved] http://mzl.la/1E5mQqM Need verification
  • status-b2g-v2.2 affected not blocking any bugs http://mzl.la/1E5ArOQ
• RTL (Marigold)
  • Health: Red
  • Test case coverage: 100%
  • Details: 15 blockers
    • http://mzl.la/1b2ddSG

• Privacy (Gerry)
  • Health: Yellow
  • Test case coverage:
  • Details: 5 blockers
    • https://bugzilla.mozilla.org/show_bug.cgi?id=1111992
    • https://bugzilla.mozilla.org/show_bug.cgi?id=1093688
    • https://bugzilla.mozilla.org/show_bug.cgi?id=1134711
    • https://bugzilla.mozilla.org/show_bug.cgi?id=1136286
    • https://bugzilla.mozilla.org/show_bug.cgi?id=1137083
    • http://mzl.la/1vvsAfY
• Josh to help talk with dave/eitan on the backout for the compile time config
• Cut, Copy, and Paste (Gerry, Edward, Teri, Eric)
  • Health: Yellow
  • Test case coverage:
  • Details: 2 blockers
    • https://bugzilla.mozilla.org/show_bug.cgi?id=1021499
    • https://bugzilla.mozilla.org/show_bug.cgi?id=1130982
• NFC (Alison)
  • Health: Green
  • Test case coverage:100%
  • Details:
    • No blockers
• short_name support in manifest.webapp (Hermes)
  • Health: Yellow
  • Test case coverage: 100%
  • Details: Not fully landed. 1 WIP blockers
    • https://bugzilla.mozilla.org/show_bug.cgi?id=1137498
    • https://bugzilla.mozilla.org/show_bug.cgi?id=1137501 (Resolved, but not uplift to 2.2)
• Search App (Hermes)
  • Health: Yellow
  • Test case coverage: 100%
  • Details: 5 WIP blockers
    • https://bugzilla.mozilla.org/show_bug.cgi?id=1140325
    • https://bugzilla.mozilla.org/show_bug.cgi?id=1141315
    • https://bugzilla.mozilla.org/show_bug.cgi?id=1140525 (RTL)
    • https://bugzilla.mozilla.org/show_bug.cgi?id=1139807 (RTL)
    • https://bugzilla.mozilla.org/show_bug.cgi?id=1132262 (patch will be landed this week)
• App-grouping (Hermes)
  • Health: Green
  • Test case coverage: 100%
  • Details: No 2.2+ blockers for now
• Task Manager (Hermes)
  • Health: Green
  • Test case coverage: 100%
  • Details: 1 Blockers
    • https://bugzilla.mozilla.org/show_bug.cgi?id=1107139
• L-porting (Hermes)
  • Health: Green
  • Details:
    • Patch of camera recording is waiting for review. https://bugzilla.mozilla.org/show_bug.cgi?id=1120780
• CAF bugs (Eric)
  • Health: Green
  • Test case coverage: 100%
  • Details: Running regression test now.
• TCL bugs (Alison, Eric)
  • Health: Green
  • Test case coverage: 100%
  • Details: Running regression test now.
• Bluetooth (Eric, Teri)
  • Health: Yellow
  • Test case coverage: 100%
  • Details:
  • PTS certification tested on flame-kk this week. 5/6 profile test completed. Couple cases failed but we need to know they are due to steps error or they are actually bugs http://bit.ly/1FLiZl8 https://bugzil.la/1139349
  • Need to manually turn on bluetooth daemon. Currently the new code with daemon is off by default due to bluetooth function break after FOTA, http://bugzil.la/1134821 a patch is under review.
  • What is Bluetooth PTS? https://wiki.mozilla.org/B2G/QA/Bluetooth_PTS
• Metrics (Shing)
  • Health: Yellow
  • Test case coverage: 95%
  • Details:
    • 3 Blocker (2.2?) http://mzl.la/1EbkJ7r
• 256M RAM memory optimization for WFVGA, 5MP. (Teri)
  • Health: Green
  • Test case coverage: 100%
  • Details: Testing in progress, no blocker.
• Multicore performance (Edward)
  • Health:
  • Test case coverage:
  • Better performance on Multicore (App startup time for core apps is comparable to Android
  • Set up raptor for v2.2
  • https://bugzilla.mozilla.org/show_bug.cgi?id=1132515
    • analyze and narrow down result from QA jenkins, http://bit.ly/1Glz30h. The problem should be between 4044 and 4045.

Exploratory Testing

Flame-KK On Track

• 227 bugs found - includes every bug in bugzilla affecting v2.2, not limited to the bugs after branch day.
  • UNCONFIRMED, NEW, ASSIGNED, REOPENED - 106 bugs found.
  • UNCONFIRMED, NEW, ASSIGNED, REOPENED & 2.2?, 2.2+ - 14 bugs found.
  • RESOLVED, VERIFIED, CLOSED - 121 bugs found.
  • RESOLVED, VERIFIED, CLOSED & 2.2?, 2.2+ - 9 bugs found.

• Regression bugs
  • UNCONFIRMED, NEW, ASSIGNED, REOPENED & 2.2?, 2.2+ - 24 bugs found.
  • RESOLVED, VERIFIED, CLOSED & 2.2?, 2.2+ - 220 bugs found.

Performance Testing

Flame-KK Yellow

• will be on soon, link (VPN required)
  • Target platforms - flame-KK, flame-L

V2.2 Test Case Creation

On Track

• Moztrap link of V2.2 New Features suite.

Test Case Creation WIP

• [2.2-feature-qa+] & in-moztrap? (0 bugs) --> WORK DONE
• feature-b2g=2.2+ & in-moztrap? (0 bugs) --> WORK DONE

Test Case Creation DONE

• [2.2-feature-qa+] & in-moztrap+ (19 bugs) --> WORK DONE
• feature-b2g=2.2+ & in-moztrap+ (78 bugs) --> WORK DONE

V2.2 Testing Verification Status

Overall Verification Summary

Need Verification

30 Total; 0 Open (0%); 30 Resolved (100%); 0 Verified (0%);

Verification Done

1 Total; 0 Open (0%); 0 Resolved (0%); 1 Verified (100%);