Auto-tools/Meetings/2015-02-23

From MozillaWiki
Jump to: navigation, search

Contents

Notices, Highlights, Roundtable

  • Significant Contributions
    • Vikas Mishra has being doing awesome work on the perfherder ui frontend: bug 1132506, bug 1131183
    • Julien Parkouss continues doing great work on mozregression, helping mentor various new contributors (e.g. bug 1128601) in addition to fixing and improving the software
    • adusca and vaibhavmagarwal for contributions mozilla_ci_tools
  • Introductions: Maja Frydrychowicz and Syd Polk

Newsgroup and Blog Posts

Goal Updates

Note: Items belonging to Supporting Tasks and Backlog are not part of Q1 goals and may not be completed this quarter.

Also see our Trello board

Marionette

Implement the set of features needed to support conversion of all prioritized mozmill tests to Marionette [chmanchester, AutomatedTester]

  • details: implement modal dialog support, separation of Marionette client into a separate package, release of older version of Marionette client to support update tests, modification of harness to dynamically load appropriate version of Marionette client
  • bugs: bug 906712, bug 1109183, bug 1107336
  • progress since last update:
    • bug 906712 fixed
    • bug 1096488, support for non-remote <-> remote pages in e10s in progress (blocking e10s in firefox-ui-tests)
    • bug 1107336
      • WPT Runner changes added
      • Green try run, just fixing patch for review (reviewboard and my local copy aren't best friends)

Support the conversion of targeted P1 mozmill tests to Marionette and get them running in CI [hskupin]

  • details: Perform the CI work necessary to get mozmill tests converted to Marionette running for update tests, and security tests. This entails writing new Marionette test libraries for these features, and getting converted tests running on the existing mozmill CI systems.
  • stretch goal: get test results reported to Treeherder
  • stretch goal: support the conversion of Search tests
  • progress since last update:
    • Given by the feedback of QE we are now targeting security and update tests as P1
    • Lots of activities here with about 7 security tests in the work and close to be landed - for others the deps will also land soon
    • Bob Silverberg started on the software update module and will continue with the about window soon; Henrik will start this week with the implementation of the necessary harness features
    • Thanks to a couple of our contributors who are working on patches!

Resolve P1 bugs blocking the release of Marionette 1.0 [AutomatedTester, ato, jgraham]

Supporting Tasks

  • train QA on writing Marionette Greenlight Tests

MozReview and Autoland (joint with RelEng)

Add support for autolanding from MozReview to try [mcote, dminor, mdoglio]

  • details: Allow developers to trigger try runs from directly within MozReview
  • bugs: bug 1109218, bug 1121616
  • progress since last update:
    • work has continued on the MozReview UI - some portions reviewed and landed
    • autoland configured to use https

Better Bugzilla integration with MozReview [mcote]

  • details: Make MozReview data in Bugzilla more useful by creating a Bugzilla field that contains dynamic information about a MozReview review request
  • bug: bug 1102428
  • progress since last update:
    • Both Bugzilla and MozReview extension have been through a round or two of reviews. Both should be ready to land within a week or so.

Supporting Tasks

  • continue improvements to the multi-commit UI

Perfherder

Ingest all Talos data with Treeherder and develop a UI that can be used to view current and historical data [wlach]

  • details: We want to use Treeherder to store and display Talos performance data, because Datazilla is being deprecated and Graphserver doesn't support the kind of performance analysis we'd like to perform on the data.
  • progress since last update:
    • UI improvements from wlach and mishrasvikras
    • Starting on support for summary series (bug 1108832)

Treeherder

Distinguish between Tier 1 and Tier 2 jobs [mdoglio]

  • details: Tier 1 and Tier 2 jobs will have different sheriffing guidelines and different expectations. Accordingly, we need to display them and allow users to interact with them differently.
  • bug: bug 1113322
  • progress since last update:

Develop a prototype structured log viewer [camd]

  • details: Develop a minimal structured log viewer which can be used to view the structured logs produced by the test harnesses. For the initial implementation, the user should be able to toggle between the parsed log viewer and the structured log viewer, for those harnesses that produce both.
  • bug: bug 1113873
  • progress since last update:
    • Viewing logs based on error lines is working.
    • Infinite scrolling of logs is partially working.
    • Created a Celery task to generate the summary from the full "raw" structured log. *almost* working...
    • ON TRACK - Expect to be done by end of February

[DEFER] Develop a minimal UI that sheriffs can use to file new intermittents [edmorley]

  • details: develop a usable but minimal UI that sheriffs can use to quickly file new intermittent issues; the UI should automatically fill out some of the details that sheriffs normally have to find manually. Future iterations can improve on this by auto-filling more fields.
  • bug: bug 1117583
  • deferred to allow edmorley to focus on Treeherder operational issues and TBPL EOL

Supporting Tasks

  • Continue to improve the performance and operational aspects of the system
  • Identify and resolve remaining issues blocking TBPL EOL - treeherder parts: bug 1059400, other parts: bug 1054977

Backlog

  • Change the authentication model to separate credentials from repos
  • Implement OrangeFactor within Treeherder
  • Create a developer-centric view

Bugzilla

Implement an alternate bug view [glob]

  • details: Implement an alternative view of bugs, to provide UX and responsiveness improvements, and as a foundation for task-/team-centric views.
  • bug: bug 1068655
  • progress since last update:
    • slow but steady progress
    • all fields and comments implemented, working through minor outstanding issues and testing
    • screenshot screenshot

Implement versioning framework for REST API [dkl]

  • details: Version the REST API to provide stable endpoints for users and a place for unstable development.
  • bug: bug 1051056
  • progress since last update:
    • Finishing up on final (4th) design iteration. Removal of legacy dependency on library that will be going away once REST is the only API. REST was a wrapper around JSON::RPC (RPC procedure calls over JSON) and had quite a few hacks in the code to remove JSON::RPC specific details before returning REST like data. The new version is written for REST only and will be much simpler. Slow but steady progress. Trying to get up for review late today, early tomorrow.

GitHub authentication [dylan]

  • details: Allow users to authenticate with Bugzilla using their GitHub account. This will encourage more contributors and allow us to better integrate GitHub into the Mozilla workflow.
  • bug: bug 1118365
  • progress since last update: Steady progress since last update. Several design concessions have had to be made to Bugzilla's authentication stack. First review will be this week.

DevTools Harness

[AT RISK] Get the DevTools harness running in continuous integration [ted]

  • details: Take the prototype that was developed in 2014 Q4 (https://github.com/luser/luciddream) and get it running in continuous integration and visible in Treeherder. It's TBD whether this will be run in buildbot or TaskCluster, but we should get it running somewhere per-commit this quarter against linux desktop Firefox and a B2G emulator.
  • progress since last update:

CloudServices Automation

Supporting Tasks

  • Take existing client/server automation that is already running and get it reporting to Treeherder using the Tier 2 UI/workflow; bug 1108259

Test Infrastructure

Define and document Tier 2 jobs [bc]

  • details: Define and document all aspects of Tier 2 jobs: rationale, criteria, necessary enhancements to Treeherder and automation frameworks.
  • bug: bug 1121655
  • progress since last update:
    • 2015-02-11: Met with dburns, jlal, mcote, mdoglio, ryanvm, sphilp, jgriffin to discuss the meaning of Tier 2 jobs and how they related to this deliverable. Tier 2 jobs where determined to be sheriffable jobs whose sheriffing is limited to filing bugs on failures which can then be ignored by sheriffs and whose failures do not cause the offending changesets to be backed out. These aspects of Tier 2 jobs are Treeherder UI issues and not related to the original expectation of this deliverable.
    • My understanding, is that this deliverable is now about defining and documenting requirements for non-buildbot jobs.

Prototype a retrigger-based bisection tool [armenzg, jmaher]

  • details: Create a prototype of a command-line tool that can be used by sheriffs and others to automate retrigger-based bisection. This could be used to help bisect new intermittent oranges, and to backfill jobs that have been skipped due to coalescing. Integration with Treeherder or other service will be done later.
  • progress since last update:
    • Prototype available
    • Lots of consumers and lots of bug fixes
    • Next: demo to sheriffs and developer(s) to get their feedback

Store high-resolution testcase data ("ActiveData") [ekyle, ahal]

  • details: Create a Proof of Concept “big data” project which will store information about every test file we run: test status, error details, test machine and test duration to begin with. We will use this project to develop schemas and queries that work with data this large, and we will use this data to normalize chunk sizes and provide details about which tests never fail.
  • progress since last update:
    • ActiveData service is up! http://activedata.allizom.org/
    • Use the query tool to send queries in realtime
    • Only basic queries are supported by the ActiveData service right now, and not very well.
    • Currently has over 1 billion test results
    • Three big ($0.50/hour) machines in cluster has reasonable aggregation performance (1x10^9 records in 20seconds)
    • Three bigger ETL machines ($1/hour) running 60 python threads each (180 total) can just barely keep up with data volume
    • Many opportunities to reduce cost (more efficient algorithms, spot instances, dynamic machine allocation)
  • next steps:
    • More test cases must be written that will cover the common and simple use cases: Along with making ActiveData pass those tests.
    • Add minimal discoverability features: Finding simple things, like names of columns, is currently very difficult or impossible.
    • Handle more corner cases to the ETL code as we stumble upon the last of the oddities in the structured logs.
    • Answer some real questions with this thing

Implement the ability to normalize chunk durations in mochitest [ahal]

  • details: For mochitest variants on desktop and B2G, modify manifestparser and the test harnesses to be able to specify which tests are run in specific chunks.
  • stretch goal: Implement the same feature for Android mochitest, which still uses old-style JSON manifests.
  • bug: bug 1124182
  • progress since last update:
    • Implement filtering system for manifest.active_tests (landed)
    • Create chunking filters in manifestparser (landed)
    • Get mochitest to use new chunking filters (almost ready for review)
    • Implement algorithm for chunking based on test runtimes (starting)

Create Android 4.4 emulator image for automated tests [gbrown]

  • details: Continue the work in bug 1062365 to build an emulator image based on Android 4.4 that is capable of running automated tests.
    Deliverable includes:
    • a prototype image
    • instructions for re-creating the image
    • demonstration that tests can be run on image
      NOT included in this deliverable:
    • tests running in continuous integration
    • "greening" of tests
  • progress since last update:
    • egl failures resolved with customized kernel, but other frequent failures persist
    • will likely setup Android 4.3 emulator tests instead, since that environment seems much more stable

Help Releng reduce test load [jmaher]

  • details: This quarter, we’ll validate the data from SETA and provide some recommendations to Releng about which jobs/platforms we could schedule less often in order to reduce test load. We’ll monitor the impact of these changes in terms of sheriffing burden and the number of retriggers this demands, and may adjust as needed. In subsequent quarters, we’ll use additional data from the high-resolution testcase data project and OrangeFactor to provide more finely-tuned scheduling changes.
  • progress since last update:
    • polish of api, :kmoir from releng has been working on figuring out the buildbot code to make this work
    • need to refine the scripts for the sheriffs to backfill, prepare a merge revision

Supporting Tasks

  • Help green up tests on OSX 10.10
  • Apply --run-by-dir to all mochitest harnesses
  • Remove legacy JSON manifests in favor of manifestparser manifests
  • Provide alternate solutions for the last consumers of Datazilla and work to decommission it
  • Work with devs to introduce more dynamic analyzers (like Ehsan’s setTimeout check) in test harnesses
  • Automate Windows symbol fetching, bug 1117741 [ted]
  • Add ssltunnel support to Android tests, bug 1084614

Performance Testing

Deliver training to at least 2 people for Talos performance sheriffing [jmaher]

  • details: We want to expand the pool of people who can perform performance sheriffing to make it scale better, and to reduce the bus factor problem.
  • progress since last update:
    • docs complete
    • vaibhav1994 fully trained
    • mishravikas about 75% trained!

Supporting Tasks

  • Continue sheriffing Talos performance regressions
  • Add new benchmarks as needed to mozbench
  • Create a new UI for mozbench results that doesn’t require Datazilla
  • Improve e10s support for Talos tests and infrastructure
  • Move Talos into the tree
  • Get rid of talos.zip
  • Make running Talos locally easier
  • resources: jmaher, dminor (for mozbench primarily), and contributors

Community

Increase 'contributor friendliness' of our projects [jmaher, all]

  • details: Ensure that all ongoing projects have a friendliness rating of at least 6, as shown on https://wiki.mozilla.org/Auto-tools/Projects/Everything
  • progress since last update:
    • updated template
    • 3 wiki pages have been edited
    • will do ~10 more then next month ask, project owners to fill in specific details after creating blank sections.

Supporting Tasks

  • Start tracking at least three community-related metrics over time

Other Project Updates

Holidays and Trips

  • mcote, jmaher, and jgriffin in Toronto Tuesday-Friday this week.

Misc

Retrieved from "https://wiki.mozilla.org/index.php?title=Auto-tools/Meetings/2015-02-23&oldid=1057655"