B2G/QA/Meetings/Automation/2015-07-15

From MozillaWiki
< B2G‎ | QA‎ | Meetings
Jump to: navigation, search

July 15th/16th, 2015

Attendees

  • TW/EU session: No session
  • US/EU: PBylenga, Oliverthor, mwargers, Pallavi, njpark, jlorenzo

Previous Action Items

  • [CARRY OVER][atsai] Find external resources to port Gip to Gij

Status Highlights / Agenda


  • Docker Image

https://registry.hub.docker.com/u/silne30/fxos-docker

  • Let's finish up the Q3 Automation goals

1. Support aries in automation (2 months) => Johan

  • 95% of existing tests are able to be run locally
  • We have a range of devices set up in Mountain and running the tests on daily basis

2. Switch automation to taskcluster (1 month) => Johan

  • Every download job in Jenkins will take the lastest builds from TaskCluster

3. Triage the tests by confidence (3 months) => Martijn

  • Keep maintaining the 2 suites between stable and not stable
  • Figure out intermittent-cy criteria thanks to weekly triages, and the different type of fixes we can address
  • Create Jenkins jobs for the stable and unstable runs

4. Test alerting (email escalation) => John D. (1 Month)

  • Turn off the email alterting on b2g-inbound (nobody takes a look at it)
  • Super sanity


5. Port half of the dogfood suite to automation (https://moztrap.mozilla.org/manage/cases/?filter-suite=846&pagesize=100&sortfield=created_ofn&filter-productversion=230&sortdirection=desc&pagenumber=1 ) (3 months) => Team

    • Add 1 new test/week/person

6. Put a strategy to deal with intermittent failures (3 months) =>Team

    • Verify which strategy is the best: 1 day/week to fix intermittent failures or 1 person/week
    • Implement weekly triage them

7. Create a base image (virtualization) for our gaia-test environments => John D. (1 Month)

    • Docker image hosted on docker-hub
    • Linux-based image that clones the gaia-repo + sets up the environment

8. [POC] Run a subset of reftest on device locally (2 months) => Martijn

    • Pick a subset of reftests for graphic testing
    • The results (failures) of the test run will be displayed on the output of the command line
    • Make them run locally


Extra:

  • Automate RTL => NoJun
    • Leverage Engineering people to port Gip tests to Gij
    • Ask to EM to put some effort into migrating tests from Gip to Gij => John D., Johan
    • If they buy in, check if half a day per week of porting Gip tests to Gij help to reduce the gap.

Defered for a following quarter:

  • Look at the existing disabled Gij tests and try to get them re-enabled?
  • Memory regression tool with gaia test (need to check if raptor already does it)
  • Locale-agnostic acceptance tests - Bug 1184600 - Make Gaia UI tests locale agnostic
  • Automatically trigger an adhoc job once somebody changes the tests in tests/python/gaia-ui-tests

Week - Decisions & Action Items

  • jlorenzo: Action plan for aries migration
  • jlorenzo: Ask stas, tchevalier or gandalf about throwing errors in logcat when a string is truncated (ellipsis).
Retrieved from "https://wiki.mozilla.org/index.php?title=B2G/QA/Meetings/Automation/2015-07-15&oldid=1086022"