B2G/QA/Meetings/Automation/2014-11-27

Overview

This is a weekly meetup to discuss all things regarding B2G automation and Quality. Please use wiki style.

• Meeting Name: QA Automation Roundtable
• Meeting Time: Thursday @ 9:30am - 10am PST
• Vidyo Room: QA Vidyo Room
• Video Link: https://v.mozilla.com/flex.html?roomdirect.html&key=nL4PESn1IaVh
• Dial in: 1-800-707-25331
• Meeting Notes: https://wiki.mozilla.org/B2G/QA/Meetings
• Previous Meeting:https://wiki.mozilla.org/B2G/QA/Meetings/Automation/2014-11-20
  • Q4 Goals B2G (Draft)
    • Complete 2.1 QA criteria: https://wiki.mozilla.org/Release_Management/FirefoxOS/Release_Milestones#Code_Complete_Criteria
    • Identify 3-5 active contributors that has completed One and Done tasks
    • Identify 2.2 features and have testplans drafted by end of sprint 2
    • Automate 20 gaia UI tests from backlog. have them running on device [geo: add "if appropriate"?], and checked into the main repo. and reporting against Tree Herder
      • [geo] Running on device (for MarionetteJS) and reporting to Treeherder are both harness features and belong to FxOS Auto, delivery of those is out of scope for QA. We should stick to "automate 20 gaia UI tests," implication being against whatever harness is currently supported and reasonably stable. I'd suggest leaving out "from backlog"--that's an implementation detail.
    • Clean up 2.1 invalidated testcases and prepare for 2.2 testsuite
      • performance goal for acceptance testing

Attendees

Previous Action Items

• [DONE] [Johan] follow up with james lal's team for ownership of Plivo account. If they don't want it, take ownership or figure out who to delegate to.
  • jlorenzo is still taking care of it. By the way, we have $15 left on the account. We can also set up autorecharge to always fill in our account.
• [DONE] [Bebe] Email about splitting nonsmoke tests in 2 test runs
  • Email sent
  • Bug 1102167 - Split the non-smoke test runs running on Jenkins device builds in two jobs
  • Smoke has 43, non-smoke has 97 tests
• [DONE] [MW] Follow up with Devs (ricky?) if gaia test can dynamically be setup with the facebook test app from TEF
  • https://bugzilla.mozilla.org/show_bug.cgi?id=1100342#c9
• [CARRY OVER] [Tony] Hold off on adding extra SIMs for now, and only keep it on b2g-22.1 for testing. This device has been isolated from the grid from the general pool of tests. Tony will also look into expanding if we can add other flames for dsds coverage
• [DONE][Bebe] follow up with splitting up smoke and non smoke, and parallelize tests
  • Started email thread and bug:
    • Bug 1102167 - Split the non-smoke test runs running on Jenkins device builds in two jobs
  • The jobs where split and it will take around 1h to run all the tests
• [DONE] [Bebe] find a way to summarize the failures in jenkins tests against intermittent tests. Also investigate status on new tests status
  • Done see email
• [CARRY OVER] [Tony] Talk to TwQA today about reaching out for 2.1 on device smoketests
  • Softvision is waiting for a go to stop the automation on 2.1.

Status Highlights

When we started this quarter(Oct 1st) we were running a total of 128 tests. From these 39 tests where in the smoke test run and 89 in the non-smoke. At the moment(Nov 26th) we are running a total of 150 tests. 42 smoke tests and 108 non smoke tests. We have 21 new tests. Intermittent fails analysis: https://docs.google.com/spreadsheets/d/1ms-SeF3DwzpLdM2OJZJKblYU3WZmIJJFPWIEaD7usFk/edit?usp=sharing marionette js has been freezing, but not on mdas's machine- turns out she was using 512MB config while we run it on 319MB config

Agenda

• [jlorenzo] Do we know what's the status of automation for December 1st? Are the QAnalyst covering it?

This Week - Decisions & Action Items