July 8th/9th, 2015

Attendees

• TW/EU session: Johan, Shako, Walter , Askeing, Mike, Paul, Al Tsai
• US/EU: John D., Martijn, Naoki, Johan

Previous Action Items

Status Highlights / Agenda

• Gip Retirement
  • Hide Gip on treeherder won't help to find early regressions => We need to make a clear distinction between an acceptance and an integration test (see links below), having a clearer distinction for ownership will help too

• • We need to port Gip tests to Gij => We started to port some of them during Q2 (see the spreadsheet below for details, page "raw data"). We also started to offload some parts of it to devs. It might not be sufficient, Al will find resources

• • Need to take care of api changes => This shouldn't be a major issue. We use low level api for setup/teardown. We haven't seen a lot of API changes for the past 8 months, that needed us to change the test harness. :mwargers can teel

• • Further links:
    • https://wiki.mozilla.org/images/d/d0/Gip_to_Gij.pdf
    • https://wiki.mozilla.org/B2G/QA/Automation/UI/Strategy/Integration_vs_Acceptance
    • https://docs.google.com/spreadsheets/d/19Cl5OaLSiynwW5NItjPuOQYpKbEKG_2eneiuwqm3Gew/edit#gid=0

• Automation on aries, we'll try to make it run next week.
  • TW can help here => If some parts don't work (like BT, Wi-FI), we'll open a bug and ping the right QA owner

US/EU discussion

• Move the roundtable to wednesday (as the one between TW/EU happens at that day)?
  • Okay for people!

• Q3 Automation goals?

1. Support aries in automation => Johan

• 95% of existing tests are able to be run locally
• We have a range of devices set up in Mountain and running the tests on daily basis

2. Switch automation to taskcluster => Johan

• Every download job in Jenkins will take the lastest builds from TaskCluster

3. Triage the tests by confidence => Martijn

• Create 1 new job for (stable = false) tests.
• Keep maintaining the 2 suites between stable and not stable
• Figure out intermittent-cy criteria thanks to weekly triages, and the different type of fixes we can address

4. Test alerting (email escalation) => John D.

• Turn off the email alterting on b2g-inbound (nobody takes a look at it)
• Super sanity

5. Port half of the dogfood suite to automation (https://moztrap.mozilla.org/manage/cases/?filter-suite=846&pagesize=100&sortfield=created_ofn&filter-productversion=230&sortdirection=desc&pagenumber=1 ) => Team

• • Add 1 new test/week/person

6. Put a strategy to deal with intermittent failures =>Team

• • Verify which strategy is the best: 1 day/week to fix intermittent failures or 1 person/week
  • Implement weekly triage them

7. Create a base image (virtualization) for our gaia-test environments => John D.

• • Docker image hosted on docker-hub
  • Linux-based image that clones the gaia-repo + sets up the environment

8. [POC] Run a subset of reftest on device locally => Martijn

Extra:

• Automate RTL => NoJun
  • Leverage Engineering people to port Gip tests to Gij
  • Ask to EM to put some effort into migrating tests from Gip to Gij => John D., Johan
  • If they buy in, check if half a day per week of porting Gip tests to Gij help to reduce the gap.

Defered for a following quarter:

• Look at the existing disabled Gij tests and try to get them re-enabled?
• Memory regression tool with gaia test (need to check if raptor already does it)

Week - Decisions & Action Items

• [atsai] Find external resources to port Gip to Gij