B2G/QA/Automation/UI/Strategy/Acceptance Tests To Automate
Objective
As a follow-up to [1], we have to figure out which acceptance tests to automate for MarionetteJS, taking into account the limitations we have with the MarionetteJS framework. For the difference between integration and acceptance tests, see [2]
Challenges Addressed
- Currently, MarionetteJS only runs on desktop, not on device, this is a blocker issue that needs to be resolve before we can start working on this.
- MarionetteJS lacks several capabilities that Python UI tests has.
The Problem
In order to know which MarionetteJS tests we can automate, we need to know the current limitations with it. Current limitations of MarionetteJS on device that we know of:
- Can't push/pull/delete files on device remotely
- Can't run in constrained (319 mb) RAM mode
- Can't access ADB-level operations
- Can't specify capabilities / no manifest tags (difficult for continuous integration)
- Hardware API wrappers are not clear
The Solution
Look at the current Python Gaia UI Tests that are acceptance tests (running on device). At [1], we have a list of tests that could be ported to MarionetteJS integration tests. Basically, all the tests that can't be ported to CI should be made into MarionetteJS acceptance tests. Except for the case, when we encounter the current limitations mentioned in the MarionetteJS framework. In that case, we keep the existing Python Gaia UI test for now.
Risks
- Acceptance tests must be able to work on-device. Marionette JS is not working on device, currently (blocker).
- The gap between Python Gaia UI Test and Marionette JS is not fully known yet.
