User:Mjzffr/Goals
Personal deliverables for :maja_zf, in addition to ongoing work.
2017 Q1
- [DONE] Add wdspec web-platform tests and wdclient support for key actions.
- [ON TRACK] Implement dispatch of mouse actions in Marionette according to WebDriver spec.
2016 Q4
- [DONE] Add support for debug builds for --artifact flag in try syntax.
- [DONE] Implement dispatch key actions in Marionette according to WebDriver spec.
2016 Q3
- [DONE] Add support for try syntax that triggers jobs against an artifact build. (https://bugzilla.mozilla.org/show_bug.cgi?id=1278698)
- [DONE] Implement WebDriver spec for extracting an action chain.
Other tasks: continue mentoring Outreachy participant (Marionette harness unit tests), deploy Marionette Fennec tests on TaskCluster
2016 Q2
- [DONE] Add running Marionette tests against Fennec in an emulator. (https://bugzilla.mozilla.org/show_bug.cgi?id=787203)
- Add Fennec support to mozrunner.
- Refactor emulator support in Marionette test runner.
- Follow-up: schedule Marionette tests against Fennec in TaskCluster after https://bugzilla.mozilla.org/show_bug.cgi?id=1281004
- [DONE] Plan and mentor Outreachy project - "Test-Driven Refactoring of Marionette Python Runner"
2016 Q1
- [DONE] Deploy suite of Marionette harness unit tests and decouple the harness from Firefox Puppeteer. (See https://bugzilla.mozilla.org/show_bug.cgi?id=1227367 and https://bugzilla.mozilla.org/show_bug.cgi?id=1238733)
- Extra: Set up harness-unit-tests job to run in TaskCluster.
- [DONE] Extra: Recruit an Outreachy intern - see Test-driven Refactoring of Marionette's Python Test Runner
2015 Q4
- [DONE] Refactor the Marionette Test Runner. (See https://bugzilla.mozilla.org/show_bug.cgi?id=1210874)
This work is happening in coordination with major changes to firefox-ui-tests and firefox-media-tests: both test suites are moving in-tree, and firefox-media-tests jobs will move to the same CI as firefox-ui-tests which means that various aspects of the Jenkins configurations will be consolidated.
2015 Q3
- [DONE] Move a subset of Firefox Media Tests jobs (EME/MSE testing) to buildbot: https://bugzilla.mozilla.org/show_bug.cgi?id=1185001
- Short firefox-media-tests jobs should run per commit in buildbot and report results to Treeherder Tier-2.
- Supporting tasks include: reduce dependency on firefox-ui-tests; refactor existing mozharness script for running firefox-media-tests;
move firefox-media-tests from Github to hg.mozilla.org.- We will coordinate the move of media tests in-tree with firefox-ui-tests: https://bugzilla.mozilla.org/show_bug.cgi?id=1197234
- [DONE] Fulfill Treeherder Tier-2 requirements for Firefox Media Tests (EME/MSE testing).
- Modify job failure output summary to match format expected for Treeherder bug suggestions.
- Parse job logs into “steps” so they can be browsed more easily in Treeherder’s Log Viewer.
- Collect firefox-media-tests documentation in a Mozilla wiki page, including key information for tree sheriffs.
Meanwhile, I've also made firefox-media-tests much more reliable by fixing a few intermittent failures related to YouTube ads. In addition, I've made several changes in our CI to make the test environment more reproducible and deterministic.
2015 Q2
Team goal: Extend MSE/EME automation to provide additional test coverage: create new MSE/EME automation and extend the existing automation to provide additional stress and functional coverage.
- [DONE] Add instrumentation to HTML5 video playback tests and produce reports targeted for MSE/EME. Modify test harness to output debugging information in case of failure: MSE player state (like about:media); screenshots, YouTube debug data, crash stacks.
- Also improved logging of test progress and improved detection of playback stalls
- Extra: tests can trigger recording of a memory report
Team goal: Improve the visibility of QA automation: make new and existing QA automation more valuable by migrating it to buildbot or reporting test results to Treeherder.
- [DONE] Get qualifying subset of Jenkins-based media automation (MSE/EME/WebRTC) reporting to Treeherder. Implement the generation/submission of appropriate data structures to treeherder-service. This applies to all Firefox branches: short MSE/EME tests and WebRTC tests should report to Treeherder as Tier 2 jobs.
- WebRTC Steeplechase tests and firefox-media-tests (MSE/EME) all report to Treeherder staging: Win 7, Win 8, Mac 10.10 on mozilla-aurora, mozilla-central. mozilla-beta is turned off on pf-jenkins until package downloads move to using a Pulse monitor.
- All MSE firefox-media-tests (YouTube) report to Treeherder prod as Tier-2.
- We won't report WebRTC tests to Treeherder production until we come to an agreement with sheriffs and Treeherder team about which job symbols to use and how to present the data. Right now, the Treeherder UI is not well-suited to present job results for combinations of branches and platforms.
Supporting tasks:
- Created a mozharness script for running firefox-media-tests, which helps with virtual env management, collecting resources, debugging harness failures. It was then easy to extend the script to upload logs to S3, submit data to Treeherder, and so on.
- Refactored firefox-media-tests to distinguish between generic video tests versus site/vendor-specific tests
2015 Q1
Joined mid-quarter, so no deliverables. These are "supporting tasks".
- [DONE] Create and deploy a suite of automated YouTube playback tests. https://github.com/mjzffr/firefox-media-tests
- [DONE] Write a tool to pull a subset of YouTube URLs from crash-stats and feed that to the automated playback tests. https://github.com/mjzffr/moztesttube
