B2G/QA/Meetings/Automation/2014-11-06

Overview

This is a weekly meetup to discuss all things regarding B2G automation and Quality. Please use wiki style.

• Meeting Name: QA Automation Roundtable
• Meeting Time: Thursday @ 9:30am - 10am PST
• Vidyo Room: QA Vidyo Room
• Video Link: https://v.mozilla.com/flex.html?roomdirect.html&key=nL4PESn1IaVh
• Dial in: 1-800-707-25331
• Meeting Notes: https://wiki.mozilla.org/B2G/QA/Meetings
• Previous Meeting:https://wiki.mozilla.org/B2G/QA/Meetings/Automation/2014-10-23
  • Q4 Goals B2G (Draft)
    • Complete 2.1 QA criteria: https://wiki.mozilla.org/Release_Management/FirefoxOS/Release_Milestones#Code_Complete_Criteria
    • Identify 3-5 active contributors that has completed One and Done tasks
    • Identify 2.2 features and have testplans drafted by end of sprint 2
    • Automate 20 gaia UI tests from backlog. have them running on device [geo: add "if appropriate"?], and checked into the main repo. and reporting against Tree Herder
      • [geo] Running on device (for MarionetteJS) and reporting to Treeherder are both harness features and belong to FxOS Auto, delivery of those is out of scope for QA. We should stick to "automate 20 gaia UI tests," implication being against whatever harness is currently supported and reasonably stable. I'd suggest leaving out "from backlog"--that's an implementation detail.
    • Clean up 2.1 invalidated testcases and prepare for 2.2 testsuite
      • performance goal for acceptance testing

Attendees

Previous Action Items

• [DONE] [Tony] look into access for SV into jenkins nodes

http://jenkins1.qa.scl3.mozilla.com/

• • [DONE] [Bebe] file a bug
  • https://bugzilla.mozilla.org/show_bug.cgi?id=1091655

[CARRY OVER][Johan] Follow up the plivo account with walter chen and paul and get some number. Follow up with bsilverberg too about if a minute can be split or not. Ask Bob about the test script that puts money into the account?

• • [jlorenzo] Email thread started. No incoming call during MTBF. They need a service like plivo. They have 30 devices max. They don't know how many tests they run per hour or day.
  • [jlorenzo] I have access to the plivo account. 1 second is charged as 1 minute.
  • [jlorenzo] Neither bsilverberg nor stephend know about script of that kind.

Status Highlights

• Shallow flash GG over v188-1. (v188-1 enables adb connection by default) Known issue: OTA wont work because of out of space on /system partition. but this should not affect device automation as there's no OTA tests.
• [Tony] next week, will be in taiwan so can't attend roundtable again. Will be working with Brian H, on 2015 QA strategy as well as prepartion for workweek topics.

Agenda

• Marionette JS docs and tests landed today. Discuss folks to try it out
  • Docs: https://developer.mozilla.org/en-US/Firefox_OS/Platform/Automated_testing/Gaia_integration_tests#Running_tests_on_device.
  • Tests: https://github.com/mozilla-b2g/gaia/commit/7783a2b8a77b9a27d531d6130392466466a2fc85
  • Meta bug: https://bugzilla.mozilla.org/show_bug.cgi?id=1034504
• [No-jun] discussion on jenkins to host unused nodes for scheduling Imagecompare tests. Who can help to setup a dry run?
  • No-Jun -- come up with step-by-step goals to stage in slave node
  • Target executing imagecompare tests on nightly builds
  • First step: announce intention
  • Bebe suggests dedicating a device early on to be safest
• Automation Backlog wiki
  • https://wiki.mozilla.org/B2G/QA/Automation/Scrum
  • how to handle reviews?
• [jlorenzo] any news from the project Mozpool? Bug 1020366 - Adapt Mozpool to B2G
  • https://bugzilla.mozilla.org/show_bug.cgi?id=1020366
  • davehunt, zac, or james lal (:lightsofapollo) should know status

This Week - Decisions & Action Items

• [jlorenzo] Follow up with davehunt, zac, or james lal (:lightsofapollo) to know about Mozpool
• [jlorenzo] Follow up with the contacts team about the Facebook account
• [njpark] send out an outline of plan to run Imagecompare on one of the unused nodes, and collect feedback + incorporate it to the plan. Ping davehunt for help
• [njpark] try setting up marionette JS on device test locally and run tests