B2G/QA/Meetings/Automation/2014-11-20

Overview

This is a weekly meetup to discuss all things regarding B2G automation and Quality. Please use wiki style.

• Meeting Name: QA Automation Roundtable
• Meeting Time: Thursday @ 9:30am - 10am PST
• Vidyo Room: QA Vidyo Room
• Video Link: https://v.mozilla.com/flex.html?roomdirect.html&key=nL4PESn1IaVh
• Dial in: 1-800-707-25331
• Meeting Notes: https://wiki.mozilla.org/B2G/QA/Meetings
• Previous Meeting:https://wiki.mozilla.org/B2G/QA/Meetings/Automation/2014-10-23
  • Q4 Goals B2G (Draft)
    • Complete 2.1 QA criteria: https://wiki.mozilla.org/Release_Management/FirefoxOS/Release_Milestones#Code_Complete_Criteria
    • Identify 3-5 active contributors that has completed One and Done tasks
    • Identify 2.2 features and have testplans drafted by end of sprint 2
    • Automate 20 gaia UI tests from backlog. have them running on device [geo: add "if appropriate"?], and checked into the main repo. and reporting against Tree Herder
      • [geo] Running on device (for MarionetteJS) and reporting to Treeherder are both harness features and belong to FxOS Auto, delivery of those is out of scope for QA. We should stick to "automate 20 gaia UI tests," implication being against whatever harness is currently supported and reasonably stable. I'd suggest leaving out "from backlog"--that's an implementation detail.
    • Clean up 2.1 invalidated testcases and prepare for 2.2 testsuite
      • performance goal for acceptance testing

Attendees

Previous Action Items

• [CARRY OVER] [Johan] follow up with james lal's team for ownership of Plivo account. If they don't want it, take ownership or figure out who to delegate to.
• [DONE] [Johan] Continue investigation on the Facebook test app (bug 1018307)
  • We can't use the app I created this summer (too recent, can't use old APIs). We got access to Telefónica's test app.
• [DONE] [Geo] See what's going on with the SIM addition to the MV infrastructure
• [CARRY OVER] [Bebe] Email about splitting nonsmoke tests in 2 test runs
  • Email sent
  • Bug 1102167 - Split the non-smoke test runs running on Jenkins device builds in two jobs
  • Smoke has 43, non-smoke has 97 tests

Status Highlights

• please summarize the status of how the intermittent failures are going, and also the progress of new tests
• Softvision can't handle Friday 20th's mozilla-central report. jlorenzo will handle it.
• 1 Dec National Day of Romania
  • National day off
  • All SV teams are off
  • Mozilla Dec 1 - 5th in portland ww. Will cancel the automation test meeting
    • Peter's team will run full smoketests 1st of December

Agenda

• [Tony] Any antenna and failed FM radio/Dialer test problems?
  • https://bugzilla.mozilla.org/show_bug.cgi?id=1000863
• [Tony] Discuss 2.1 plan after friday
• [Tony] updates on Labwork
  • We have 7 more SIm cards, will be adding them to existing Flames for DSDS support. b2g-22.1 was added yesterday.
    • Prefer to test on 1 phone for now. all the smoketests were breaking because the current tests weren't detecting the 2nd SIM.
• [Tony] Moz Workweek scheduling
• [jlorenzo] Devices usage by Gaia-UI-Tests in the MTV lab: http://jenkins1.qa.scl3.mozilla.com:8080/label/flame-kk&&ui/load-statistics

This Week - Decisions & Action Items

• [MW] Follow up with Devs (ricky?) if gaia test can dynamically be setup with the facebook test app from TEF
  • https://bugzilla.mozilla.org/show_bug.cgi?id=1100342#c9
• [Tony] Hold off on adding extra SIMs for now, and only keep it on b2g-22.1 for testing. This device has been isolated from the grid from the general pool of tests. Tony will also look into expanding if we can add other flames for dsds coverage
• [Bebe] follow up with splitting up smoke and non smoke, and parallelize tests
  • Started email thread and bug:
    • Bug 1102167 - Split the non-smoke test runs running on Jenkins device builds in two jobs
  • The jobs where split and it will take around 1h to run all the tests
• [Bebe] find a way to summarize the failures in jenkins tests against intermittent tests. Also investigate status on new tests status
  • Done see email
• [Tony] Talk to TwQA today about reaching out for 2.1 on device smoketests