Auto-tools/Meetings/2015-01-12

From MozillaWiki
Jump to: navigation, search

Contents

Notices, Highlights, Roundtable

  • Significant Contributions
    • Thanks to Sherry Shi for adding tests for the Pulse permission schema: bug 1054142
  • jgriffin: re-introductions - Henrik
  • jgriffin: Q1 goal updates
  • jgriffin: Q4 goal debrief
  • A-Team Community Day - January 27th!

Newsgroup and Blog Posts

Goal Updates

Note: Items belonging to Supporting Tasks and Backlog are not part of Q1 goals and may not be completed this quarter.

Also see our Trello board

Marionette

Implement the set of features needed to support conversion of all prioritized mozmill tests to Marionette [chmanchester, AutomatedTester]

  • details: implement modal dialog support, separation of Marionette client into a separate package, release of older version of Marionette client to support update tests, modification of harness to dynamically load appropriate version of Marionette client
  • bugs: bug 906712, bug 1109183, bug 1107336
  • progress since last update:

Support the conversion of targeted P1 mozmill tests to Marionette and get them running in CI [hskupin]

  • details: Perform the CI work necessary to get mozmill tests converted to Marionette running for update tests, tabbed browser tests, and awesomebar tests. This entails writing new Marionette test libraries for these features, and getting converted tests running on the existing mozmill CI systems.
  • stretch goal: get test results reported to Treeherder
  • stretch goal: support the conversion of Search tests
  • progress since last update:
    • Waiting for next marionette-client release (bug 1120364) to continue with window and tab handling support
    • Finished prefs module and added Travis support
    • Discussion about how to name the repository - my proposal would be 'Firefox UI Tests"

Resolve P1 bugs blocking the release of Marionette 1.0 [AutomatedTester, ato, jgraham]

Supporting Tasks

  • train QA on writing Marionette Greenlight Tests

MozReview and Autoland (joint with RelEng)

Add support for autolanding from MozReview to try [mcote, dminor, mdoglio]

  • details: Allow developers to trigger try runs from directly within MozReview
  • progress since last update:
    • made code for monitoring jobs on try independent of autoland workflow
    • initial work to land patches from mozreview to try in Autoland

Better Bugzilla integration with MozReview [dkl]

  • details: Make MozReview data in Bugzilla more useful by creating a Bugzilla field that contains dynamic information about a MozReview review request
  • progress since last update: None, finishing up on 5.0 upstream Bugzilla release.

Supporting Tasks

  • continue improvements to the multi-commit UI
    • progress since last update:
      • mconley posted a new review request for bug 1064111 ('Move contents of "Commits" tab in rbmozui to the main review request page')

Perfherder

Ingest all Talos data with Treeherder and develop a UI that can be used to view current and historical data [wlach]

  • details: We want to use Treeherder to store and display Talos performance data, because Datazilla is being deprecated and Graphserver doesn't support the kind of performance analysis we'd like to perform on the data.
  • progress since last update:
    • Data: All data seems to be in treeherder. Waiting for ekyle to stand up a new instance of dzAlerts using treeherder data so we can validate that the same alerts are generated. Likely no progress this week as he's on vacation.
    • UI: Tracking progress on Trello here: https://trello.com/b/PQSxEIZf/perfherder-ui-tasks
      • New since last time: Fixed some nits from mdoglio and jfrench, allowed deleting series, style tweaks.

Treeherder

Distinguish between Tier 1 and Tier 2 jobs [mdoglio]

  • details: Tier 1 and Tier 2 jobs will have different sheriffing guidelines and different expectations. Accordingly, we need to display them and allow users to interact with them differently.
  • bug: bug 1113322
  • progress since last update: not started yet. preparing the ground as part of bug 1097090

Develop a prototype structured log viewer [camd]

  • details: Develop a minimal structured log viewer which can be used to view the structured logs produced by the test harnesses. For the initial implementation, the user should be able to toggle between the parsed log viewer and the structured log viewer, for those harnesses that produce both.
  • bug: bug 1113873
  • progress since last update:
    • Took a real structured log and generated a test summary structured log with just the naughty bits in it (errors, etc)
    • Created a branch with a modified Log Viewer showing the summary lines from that file
    • Next:
      • be able to click those lines and have it load the surrounding lines from the full structured log.
      • finalize the necessary structure of the summary file so that chmanchester and ahal can start generating it (a first cut is in bug 1113868)

Develop a minimal UI that sheriffs can use to file new intermittents [edmorley]

  • details: develop a usable but minimal UI that sheriffs can use to quickly file new intermittent issues; the UI should automatically fill out some of the details that sheriffs normally have to find manually. Future iterations can improve on this by auto-filling more fields.
  • bug: bug 1117583
  • progress since last update: None, have been working on Treeherder-TBPL transition/EOL blockers.

Supporting Tasks

  • Continue to improve the performance and operational aspects of the system
  • Identify and resolve remaining issues blocking TBPL EOL - treeherder parts: bug 1059400, other parts: bug 1054977

Backlog

  • Change the authentication model to separate credentials from repos
  • Implement OrangeFactor within Treeherder
  • Create a developer-centric view

Bugzilla

Implement an alternate bug view [glob]

  • details: Implement an alternative view of bugs, to provide UX and responsiveness improvements, and as a foundation for task-/team-centric views.
  • progress since last update:
    • blog post with discussion of the problem and proposed solutions.

Implement versioning framework for REST API [dkl]

  • progress since last update: Design work completed. Will work on implementation soon right after finishing with upstream 5.0 Bugzilla release.

GitHub authentication [dylan]

  • progress since last update: Plan/design ready, implementation delayed by previous quarter's goals (guided bug entry).

DevTools Harness

Get the DevTools harness running in continuous integration [ted]

  • details: Take the prototype that was developed in 2014 Q4 (https://github.com/luser/luciddream) and get it running in continuous integration and visible in Treeherder. It's TBD whether this will be run in buildbot or TaskCluster, but we should get it running somewhere per-commit this quarter against linux desktop Firefox and a B2G emulator.
  • progress since last update: Nothing to report, working on other tasks and catching up from holiday backlog. Planning to chat about status of Taskcluster this week to decide on feasibility of using it.

CloudServices Automation

Supporting Tasks

  • Take existing client/server automation that is already running and get it reporting to Treeherder using the Tier 2 UI/workflow; bug 1108259

Test Infrastructure

Prototype a retrigger-based bisection tool [armenzg, jmaher]

  • details: bug 1120990 - Create a prototype of a command-line tool that can be used by sheriffs and others to automate retrigger-based bisection. This could be used to help bisect new intermittent oranges, and to backfill jobs that have been skipped due to coalescing. Integration with Treeherder or other service will be done later.
  • progress since last update: Nothing new; a lot of catch up after the holiday break

Store high-resolution testcase data ("ActiveData") [ekyle, ahal]

  • details: Create a Proof of Concept “big data” project which will store information about every test file we run: test status, error details, test machine and test duration to begin with. We will use this project to develop schemas and queries that work with data this large, and we will use this data to normalize chunk sizes and provide details about which tests never fail.
  • progress since last update:
    • [ahal] Set up aws instance that listens to pulse and both sends jobs over rq to be processed by workers and stores them in mongodb for historical reprocessing
    • [ahal] Set up supervisord dashboard for managing various processes

Implement the ability to normalize chunk durations in mochitest [ahal]

  • details: For mochitest variants on desktop and B2G, modify manifestparser and the test harnesses to be able to specify which tests are run in specific chunks.
  • stretch goal: Implement the same feature for Android mochitest, which still uses old-style JSON manifests.
  • bug: bug 1054247
  • progress since last update:
    • [ahal] None besides initial brainstorming/planning, working on ActiveData project

Create Android 4.4 emulator image for automated tests [gbrown]

  • details: Continue the work in bug 1062365 to build an emulator image based on Android 4.4 that is capable of running automated tests.
    Deliverable includes:
    • a prototype image
    • instructions for re-creating the image
    • demonstration that tests can be run on image
      NOT included in this deliverable:
    • tests running in continuous integration
    • "greening" of tests
  • progress since last update:
    • created an avd from the Android 4.4 sdk
    • built custom images from AOSP
    • testing both locally

Help Releng reduce test load [jmaher]

  • details: This quarter, we’ll validate the data from SETA and provide some recommendations to Releng about which jobs/platforms we could schedule less often in order to reduce test load. We’ll monitor the impact of these changes in terms of sheriffing burden and the number of retriggers this demands, and may adjust as needed. In subsequent quarters, we’ll use additional data from the high-resolution testcase data project and OrangeFactor to provide more finely-tuned scheduling changes.
  • progress since last update:
    • reducing useless talos jobs - turned off talos on b2g*, esr* branches; this week reducing half of the talos tests on osx 10.6
    • SETA - cleaned up code- updating from treeherder - ETA this week to have SETA producing daily reports.
    • next steps: work with sheriffs and releng to have a plan for disabling tests

Supporting Tasks

  • Help green up tests on OSX 10.10
  • Apply --run-by-dir to all mochitest harnesses
  • Remove legacy JSON manifests in favor of manifestparser manifests
  • Provide alternate solutions for the last consumers of Datazilla and work to decommission it
  • Work with devs to introduce more dynamic analyzers (like Ehsan’s setTimeout check) in test harnesses
  • Automate Windows symbol fetching, bug 1117741 [ted]
  • Add ssltunnel support to Android tests, bug 1084614

Performance Testing

Deliver training to at least 2 people for Talos performance sheriffing [jmaher]

  • details: We want to expand the pool of people who can perform performance sheriffing to make it scale better, and to reduce the bus factor problem.
  • progress since last update:
    • identified one candidate for sheriffing - have given preliminary information, will know next week if this sticks.

Supporting Tasks

  • Continue sheriffing Talos performance regressions
  • Add new benchmarks as needed to mozbench
  • Create a new UI for mozbench results that doesn’t requite Datazilla
  • Improve e10s support for Talos tests and infrastructure
  • Move Talos into the tree
  • Get rid of talos.zip
  • Make running Talos locally easier
  • resources: jmaher, dminor (for mozbench primarily), and contributors

Community

Increase 'contributor friendliness' of our projects [jmaher, all]

Supporting Tasks

  • Start tracking at least three community-related metrics over time

Other Project Updates

Holidays and Trips

  • mcote, dminor, and mdoglio in Toronto Jan 19-23 for MozReview meet-up

Misc

Retrieved from "https://wiki.mozilla.org/index.php?title=Auto-tools/Meetings/2015-01-12&oldid=1047452"