Auto-tools/Projects/B2G Automation

From MozillaWiki
Jump to: navigation, search

This page is mostly obsolete. FirefoxOS Automation is now handled by the FirefoxOS Automation Team. See https://github.com/mozilla-b2g/automation-projects or irc.mozilla.org#fxos-automation

Etherpad

https://etherpad.mozilla.org/b2g-automation-meeting-notes

Goal 2013 Q2

Milestone 1 - 2 weeks (April 1 - 12)

  • [jgriffin] [MISSED] Get Gaia Unit tests running in TBPL (blocked by bug 855049)
  • [jgriffin] [DONE] Triage all existing B2G failures and identify a concrete set of problems that need to be addressed
  1. test_click errors bug 838607 - dburns looking at it.
  2. reftest/crashtest crash bug 818103 - assigned to mike habicher.
  3. [DONE] exception in window_manager.js bug 851666 - WORKSFORME
  4. [DONE] Timeout for WebAPI tests is too small; emulator callbacks can occasionally take 10-20s, leading to random timeouts. Rob will fix.
  5. socket.timeout errors caused by actorPool error bug 859417
  6. socket.timeout errors that occur at startup in newSession, before a new Gecko is copied - see logs in bug 829551
  7. chrome tests occasionally can't find elements (timing or page loading error?) bug 859423
  8. B2G crashes during WebAPI tests bug 859439 - will be easier to handle once crash detection is working here
  • [jgriffin] [MISSED] Make mach runners for B2G mochitests and reftests
  • [ahal] [DONE] Enable crash detection on mozilla-b2g18 for mochitest/reftest
  • [ahal] [DONE] Add crash detection to Marionette tests
  • [ahal] [MISSED] Write a script to download the latest emulator
  • [mdas] [DONE] Use weakref in marionette-elements (bug 736592)
  • [mdas] [ON TRACK] Work on getting Gaia UI tests running on b2g desktop builds
  • [yiming] [DONE] Complete touch/mouse event code for action chains
  • [yiming] [MISSED] Convert code from synthetic gestures to action chains
  • [rwood] [DONE] Develop bluetooth Gaia UI test to pair unagi with host machine
  • [rwood] [DONE] Finish integrating long running tests with datazilla

Milestone 2 - 2 weeks (April 22 - May 3)

  • [jgriffin] [DONE] Get Gaia Unit tests running in TBPL (scheduled on cedar)
  • [jgriffin] [DONE] Make mach runners for Marionette
  • [jgriffin] [MISSED] Add mach to B2G so we can build B2G mach targets
  • [jgriffin] [MISSED] Improve Marionette's error reporting with specific, clear error messages and stack traces
  • [jgriffin] [MISSED] Integrate JS client Pt 1: Get a simple JS Marionette test working locally with the JS Marionette client and the Python testrunner
  • [jgriffin] [MISSED] Get mozilla-central b2g desktop builds working with Marionette
  • [ahal] [DONE] Write a B2G bootstrap script to make downloading and running tests easier
  • [ahal] [MISSED] Use in-tree mozbase instead of puppetagain packages for B2G tests in buildbot
  • [mdas] [MISSED] Work on refactoring touch events and getting the mapping between action chains and mouse events working.
  • [mdas] [MISSED] Finish converting synthetic gestures to action chains
  • [rwood] [DONE] Get gaia-ui endurance tests into gaia-ui github repo (working with dhunt on improvements as per his review) was pulled out of the repo, new location tbd
  • [rwood] [ON TRACK] Improve gaia-ui endurance test stability
  • [rwood] [DONE] Automate entire process via Jenkins

Milestone 3 - 2 weeks (May 13 - May 24)

  • [ON TRACK] Work on top B2G test stability problems and either fix or get them to a state where developers can take over
  1. [DONE] test_click errors bug 838607 - dburns looking at it.
  2. [DONE] reftest/crashtest crash bug 818103 - assigned to mike habicher.
  3. [DROPPED] socket.timeout errors caused by actorPool error bug 859417 - dropped due to marionette refactor
  4. [MISSED] socket.timeout errors that occur at startup in newSession, before a new Gecko is copied - see logs in bug 829551
  • [mdas] [DEFERRED] Finish converting synthetic gestures to action chains - may not be necessary.
  • (carry over) [mdas] [DONE] On on refactoring touch events and getting the mapping between action chains and mouse events working.
  • [mdas] [DONE] Develop manifest of stable gaia-ui-tests on b2g desktop builds
  • [mdas] [ON TRACK] Work on improving Marionette docs
  • [mdas] [ON TRACK] Fix tap/touch related bugs that are blocking webQA
  • (carry over) [jgriffin] [DONE] Get mozilla-central b2g desktop builds working with Marionette
  • [jgriffin] [DONE] Add manifest support to gaia-unit-tests
  • [jgriffin] [DONE] write mozharness script for gaia-ui-tests on b2g desktop builds
  • [DROPPED] Add debugger support to B2G testrunners
  • (carry over) [ahal] [MISSED] Use in-tree mozbase instead of puppetagain packages for B2G tests in buildbot
  • [wlach] [MISSED] Get rid of debugger server dependency bug 797529

Milestone 4 - 2 weeks (June 3 - June 14)

  • (carry over) [wlach] [DONE] Get rid of debugger server dependency bug 797529
  • [wlach] [ON TRACK] Integrate JS client Pt 1: Get a simple JS Marionette test working locally with the JS Marionette client and the Python testrunner
  • (ongoing) [mdas] [MISSED] Resolve issues with tap
  • (ongoing) [mdas] [ON TRACK] Update marionette docs
  • (carry over) [ahal] [DONE] Get mozilla-central B2G unit tests to use in-tree mozbase instead of puppetagain
  • [jgriffin] [MISSED] Disable gaia-unit-tests that are still failing on cedar
  • [jgriffin + releng] [MISSED] Get gaia-unit-tests scheduled on all branches (depends on gecko-gaia coupling)
  • [jgriffin] [MISSED] Update mozharness script for gaia-ui-tests
  • [jgriffin + releng] [MISSED] Get gaia-ui-tests on b2g desktop builds scheduled on cedar (depends on gecko-gaia coupling, gaia-ui-tests to gaia coupling, and gaia-ui-tests branch renaming)
  • [jgriffin] [MISSED] Identify current B2G top oranges
  • [rwood] [DONE] Tryrel-eng m-c unagi-eng on gaia-ui-tests
  • [rwood] [DONE] Develop some heuristics to help determine when gaia-ui-test Jenkins failures are "real" or reproducible enough to warrant sending out an e-mail notification to engineering managers

Milestone 5 - 2 weeks (June 24 - July 5)

  • [wlach] [DROPPED] Add JS client Pt 1: Get a simple JS Marionette test working locally with the JS Marionette client and the Python testrunner
  • (ongoing) [mdas] [ON TRACK] Resolve blocking issues with tap
  • [mdas] [DEFER] Resolve blocking scroll issues - blocked on is_displayed fix
  • (ongoing) [mdas] [DONE] Add some mechanisms to make debugging touch related problems easier
  • [jgriffin] [ON TRACK] Disable gaia-unit-tests that are still failing on cedar
  • [jgriffin + releng] [MISSED] Get gaia-unit-tests scheduled on all branches (depends on gecko-gaia coupling)
  • [jgriffin] [DONE] Update mozharness script for gaia-ui-tests
  • [jgriffin + releng] [DONE] Get gaia-ui-tests on b2g desktop builds scheduled on cedar (depends on gecko-gaia coupling, gaia-ui-tests to gaia coupling, and gaia-ui-tests branch renaming)
  • [jgriffin] [DEFER] Identify current B2G top oranges
  • [ahal] [DONE] Identify and fix issues with full-stack emulator builds
  • [rwood] [DONE] Fix intermittent gaia-ui endurance test failures
  • [rwood] [DONE] Expand gaia-ui endurance test coverage
  • [rwood] [DONE] Write script to email/flag b2g memory spikes via gaia-ui endurance tests

Goal 2013 Q3

Milestone 6 - 2 weeks (July 8 - July 19)

  • (carry over) [jgriffin + releng] [DONE] Get gaia-unit-tests scheduled on all branches
  • [jgriffin] [MISSED] Implement script to compare two sources.xml
  • [jgriffin] [DONE] Identify source of memory leak in LG automated tests bug 897684
  • [mdas, davehunt, webqa] [DONE] Get gaia-ui-tests on cedar running stable green
  • [mdas] [MISSED] Implement modal dialog support in Marionette
  • (carry over) [mdas] [MISSED] Fix remaining blocking touch issues
  • [wlach] [DONE] Implement get_active_frame command - bug 855327
  • [rwood] [MISSED] Get games benchmarking tests running on Firefox OS
  • [rwood] [ON TRACK] Investigate/fix intermittent WebAPI emulator failures
  • [rwood] [ON TRACK] Maintain endurance tests as needed; expand checks for b2g memory changes over time
  • [rwood] [DONE] Get endurance tests running on Leo on Master
  • [rwood] [DEFER] Look into data requirements for gaia-ui heuristics

Milestone 7 - 2 weeks (July 29 - August 9)

  • ongoing - patches in hand - [ahal] Add mach to B2G so we can build B2G mach targets
  • scheduled on cedar - [mihnea] Get JSreftests working for B2G
  • [jgriffin] [DONE] Implement script to compare two sources.xml
  • [DONE] final patch up - (carry over) [mdas] Implement modal dialog support in Marionette
  • current focus -> (carry over) [mdas] Fix remaining blocking touch issues
  • (carry over) [rwood] Get games benchmarking tests running on Firefox OS
  • (ongoing) [rwood] [ON TRACK] Investigate/fix intermittent WebAPI emulator failures
  • (ongoing) [rwood] Maintain endurance tests as needed (Leo on v1-train); expand checks for b2g memory changes over time
  • currently working with jlebar on this -> [rwood] Validate emulator as platform for AreWeSlimYet (AWSY) memory tests
  • [rwood] Create app-launch and page-browser tsets for AWSY
  • (carry over) [rwood] Look into data requirements for gaia-ui heuristics

Milestone 8 - 2 weeks (August 19 - August 30)

  • (carry over) [ahal] Add mach to B2G so we can build B2G mach targets
  • [DONE] scheduled on cedar - [mihnea] Get JSreftests working for B2G
  • P1! [jgriffin + all] Update all testing docs to make it easy for developers to get started with tests
  • [mdas] Investigate modal dialog test failure
  • (carry over) [rwood] Get games benchmarking tests running on Firefox OS
  • (ongoing) [rwood] [ON TRACK] Investigate/fix intermittent WebAPI emulator failures
  • (ongoing) [rwood] Maintain endurance tests as needed (Leo on v1-train); expand checks for b2g memory changes over time
  • currently working with jlebar on this -> [rwood] Validate emulator as platform for AreWeSlimYet (AWSY) memory tests
  • [rwood] Create app-launch and page-browser tsets for AWSY

Sept 19 - Oct 2

  • (jgriffin PTO)
  • (carry over) [DONE] [mdas] land the modal dialog fix
  • [mdas] [MISSED] fix the import script leak (landed, but backed out)
  • [mdas] [DEFER] write up and add some marionette/gaiatest endurance unit tests to catch space consumption/performance regressions
  • [ahal] [DONE] write mozharness script for running mochitests on B2G desktop builds
  • [wlach] [DONE] finish implementation of npm mirror for rel-eng (re-writing in node.js and documenting)
  • [rwood] [ON TRACK] Endurance test fixes/maintenance for inari master/1.3
  • [rwood] [ON TRACK] Further work on AWSY emulator test driver
  • [rwood] [ON TRACK] Continued support for games benchmarking on Firefox OS

Goal 2013 Q4

Oct 10 - Oct 24

  • [jgriffin] [ON TRACK] investigate problem with B2G unit test timeouts not being caught by mozprocess
  • [jgriffin] [DONE] add B2G support to existing Marionette mach command
  • [mdas] [DONE] investigate issues with the modal dialog fix
  • (carry over) [mdas] [DONE] fix the import script leak (landed, but backed out)
  • [mdas] [ON TRACK] write up and add some marionette/gaiatest endurance unit tests to catch space consumption/performance regressions
  • [ahal] [ON TRACK] get mochitests on b2g desktop builds rolled out everywhere
  • [ahal] [ON TRACK] get B2G reftests running OOP
  • [wlach] document Eideticker in such a way that it's easy for partners and people new to the project to set it up from scratch
  • [wlach] [DONE] instrument better app startup times for B2G bug 888103
  • [wlach] [ON TRACK] do some fps comparisons with 120fps camera
  • (ongoing) [wlach] attend FxOS_Perf standups and continue to provide feedback wrt to power management testing
  • [rwood] Endurance test fixes/maintenance for inari master/1.3
  • [rwood] Get AWSY emulator test driver running locally
  • [rwood] [ON TRACK] Set up Jenkins cluster on new VMs for AWSY tests
  • [rwood] [DONE] Continued support for games benchmarking on Firefox OS

Oct 24 - Nov 7

[jgriffin]

[mdas]

  • [ON TRACK] refactor runtests.py so we can use a mixin pattern bug 925398
    • (carry over) (time consuming) [ON TRACK] write up and add some marionette/gaiatest endurance unit tests to catch space consumption/performance regressions bug 931044 and bug 931045
  • [DONE] Stop making marionette.log when unnecessary bug 918408

[ahal]

  • [DONE] get mochitests on b2g desktop builds rolled out everywhere
  • [DONE] address B2G crash reporting problems

[wlach]

  • Get the B2G Eideticker dashboard running
    • running and uploading data to dave's people account (http://people.mozilla.org/~dhunt/eideticker-ci-dashboard/), but we need to configure to upload/send to eideticker.mozilla.org, which is waiting on moving the setup to the london office
    • bug filed for this, wlach will look up and post here after meeting
    • also need to file some follow-up bugs on test reproducibility (e.g. for contact loading we don't last long enough to display all pictures)
      • the contacts issue at least is ours to fix: pictures loading slowly and asynchrously is "by design" according to gaia devs. will file bug with davehunt to not load avatar pictures for eideticker
  • (carry over) do some fps comparisons with 120fps camera
    • should have some results shortly

[rwood]

  • (carry over) Get a full set of AWSY emulator test driver running locally
    • done except now blocked by Bug 931198 about_memory fails with > 30 memory files
  • Assist the Taipei team with MTBF tests
    • Leaving tomorrow eve
  • Deliver a set of memory test reports to the AWSY maintainer
  • Get AWSY prodution VMs setup (1 master, others as nodes)
  • Get gaia-ui endurance tests running on hamachi b2g26/1.2
    • Up and running but now need to cherry-pick endurance test changes that arent in b2g26
  • Fix/investigate gaia-ui endurance test failures

[gkw]

  • Spoke about marifuzz, a project to use marionette for fuzzing, based on orangfuzz but using marionette instead.
    • Emailed discussion outcome (drawings from TW office board) to jgriffin, mdas and wlach

Nov 7 - Nov 21

[jgriffin]

  • [MISSED] Finish adding OOP support to WebAPI tests bug 926280
    • This is ready to land, but I'm going to wait to land it until Malini's mixin refactor lands (bug 925398), since this will bitrot that patch.
  • (carry over) [MISSED] get green set of debug emulator mochitests running everywhere
    • These are mostly green on cedar, but intermittents still appear. I've filed a bug to schedule these on all trunk branches, but we'll likely have to hide these initially until the intermittents fall to an acceptable level. bug 940161

[ahal]

  • get emulator reftests running oop
    • should also help getting b2g desktop reftests going oop
    • done and running on cedar (if the implementation is ok)
    • still some problems (crashes, timeouts, and an error that happens between tests but doens't seem to affect them)
  • [MISSED] get reftests running on b2g desktop builds
  • investigate emulator startup crashes?

[mdas]

  • (carry over) [MISSED] write up and add some marionette/gaiatest endurance unit tests to catch space consumption/performance regressions bug 931044 and bug 931045 (I was actually working on bug 925398 in order to address the other bugs, milestone updated for next sprint)

[wlach]

[ato]

  • Updating Selenium Proxy (programming sitting on top of Marionette that provides Selenium-compatible HTTP server) with missing endpoints to run Selenium tests against Marionette
  • Allows for better overview of our existing compatibility level
    • Will file bugs on incompatibilities

[rwood]

  • Nov 4-8th was MTBF week in Taipei with Taiwan QA Team; good progress
  • (carry over) Get a full set of AWSY emulator test driver running locally
    • DONE except now blocked by Bug 931198 about_memory fails with > 30 memory files
  • [missed] Deliver a set of memory test reports to the AWSY maintainer
  • [missed - in progress] Get AWSY prodution VMs setup (1 master, others as nodes)
    • blocked by Bug 931198 however get as much setup as possible outside of that
  • [DONE] Get gaia-ui endurance tests running on hamachi b2g26/1.2
    • Up and running but now need to cherry-pick endurance test changes that arent in b2g26
  • Fix/investigate gaia-ui endurance test failures

Nov 22 - Dec 5

[jgriffin]

  • (carry over) Finish adding OOP support to WebAPI tests bug 926280
  • (carry over) get green set of debug emulator mochitests running everywhere bug 916359

[ahal]

  • (carry over) get emulator reftests running oop
    • should also help getting b2g desktop reftests going oop
    • done and running on cedar (if the implementation is ok)
    • still some problems (crashes, timeouts, and an error that happens between tests but doens't seem to affect them)
  • get reftests running on b2g desktop builds
  • investigate emulator startup crashes?

[mdas]

  • Fix gaia testing blocking issue with getting tabs bug 941749
  • (carry over) (time consuming, landing will take more time) refactor marionette/gaiatest to use mixins: bug 925398
    • (carry over) write up and add some marionette/gaiatest endurance unit tests to catch space consumption/performance regressions bug 931044 and bug 931045

[wlach]

  • Get the B2G Eideticker dashboard running
  • NEW: Improve fps metrics (917404 -- old bug, but issue is more pressing now)
  • NEW: Figure out why captures with 150 fps camera are so oversaturated -- they aren't with pointgrey's utility... (no bug filed yet but will do tomorrow)

[ato]

  • (ongoing) Updating Selenium Proxy (programming sitting on top of Marionette that provides Selenium-compatible HTTP server) with missing endpoints to run Selenium tests against Marionette
  • Add screen orientation implementation, using emulator build - bug 804515

[rwood]

  • (carry over) Deliver a set of memory test reports to the AWSY maintainer
  • (carry over) Get AWSY prodution VMs setup (1 master, others as nodes)
  • Get another hamachi and replace the Inari endurance test (on master) device with hamachi
  • Fix endurance test failures (update tests according to latest gaia changes)

Future milestones

  • Fix frequent Mn orange - bug 930430 (test_submit.py failures)
  • add tests for modal dialog bug 906175
  • Investigate B2G "timed out after 1000s" errors: bug 906716 - these seem to be different flavors of emulator startup problems. See also a related bug, bug 905324, " Intermittent B2G mochitest "AssertionError: Timed out waiting for port!", which is probably another manifestation of the same problem.
  • Stand up new builds and tests for B2G - bug 916356
  • Put the Eideticker backend on Datazilla
  • Look into data requirements for gaia-ui heuristics
  • Make B2G tests in TBPL more sheriffable, by providing a tool in TBPL (Treeherder?) to compare revisions of all B2G repos between arbitrary builds
  • Stand up tests on new Android 4.2 emulator (this is potentially going to be very time consuming, depends on 4.2 emu builds in TBPL)
  • Implement device test failure notification heuristics (depends on results being uploaded to TBPL)
  • Fix more top oranges
  • Write tests for new bluetooth support in emulator?
  • improve server side logging bug 928491
  • shear away some of Marionette's MDN documentation and make it more concise and relevant. The device specific instructions may need updating as well.
  • [DROPPED] Finish integrating JS Marionette client into Python testrunner
  • [DROPPED] Gaia bisection tool - implemented by naoki
Retrieved from "https://wiki.mozilla.org/index.php?title=Auto-tools/Projects/B2G_Automation&oldid=1044244"