Notices, Highlights, Roundtable

•  :[vaibhav1994] has done a lot of work to validate, optimize, and complete [SETA]
• RyanVM is looking for help on a tricky Windows mozprocess bug, maybe a good project for a seasoned contributor? bug 928983
• Time to submit proposals for [GSOC]

Newsgroup and Blog Posts

• [jmaher] - [SETA]

Goal Updates

Note: Items belonging to Supporting Tasks and Backlog are not part of Q1 goals and may not be completed this quarter.

Also see our Trello board

Marionette

Implement the set of features needed to support conversion of all prioritized mozmill tests to Marionette [chmanchester, AutomatedTester]

• details: implement modal dialog support, separation of Marionette client into a separate package, release of older version of Marionette client to support update tests, modification of harness to dynamically load appropriate version of Marionette client
• bugs: bug 906712, bug 1109183, bug 1107336
• progress since last update:
  • Get tab modal dialog handling finished bug 906712 (see below)
  • Landed modules for location bar (bug 1123871), handling tabs (bug 1127831), and places (bug 1121731)
  • Several smaller API level fixes and Puppeteer documentation updates
  • Had an successful m21s training last week with a couple of location bar and tabbed browsing tests under construction (https://etherpad.mozilla.org/m21s)
  • Still facing a couple of blockers related to keyboard and middle/right mouse click

Support the conversion of targeted P1 mozmill tests to Marionette and get them running in CI [hskupin]

• details: Perform the CI work necessary to get mozmill tests converted to Marionette running for update tests, tabbed browser tests, and awesomebar tests. This entails writing new Marionette test libraries for these features, and getting converted tests running on the existing mozmill CI systems.
• stretch goal: get test results reported to Treeherder
• stretch goal: support the conversion of Search tests
• progress since last update:
  • No update - focus was on basic libraries and getting creation of location bar and tabbed browsing tests unblocked

Resolve P1 bugs blocking the release of Marionette 1.0 [AutomatedTester, ato, jgraham]

• details: Resolve P1 bugs in this list: https://wiki.mozilla.org/Auto-tools/Projects/Marionette/Auto-tools/Projects/Marionette/Roadmap#Roadmap
• stretch goal: release Marionette 1.0
• progress since last update:
  • bug 1084125 - capabilities - try push
  • bug 1107706 - [ato] Eliminating remaining bugs on B2G emulators, but progress is slow due to long turnaround; no ETA on completeness
  • bug 906712 - Modal dialogs reviewed and ready to land pending fresh run through try.
  • [jgraham] - Started porting some mozbase-ish bits to rust to launch Firefox from proxy.

Supporting Tasks

• train QA on writing Marionette Greenlight Tests

MozReview and Autoland (joint with RelEng)

Add support for autolanding from MozReview to try [mcote, dminor, mdoglio]

• details: Allow developers to trigger try runs from directly within MozReview
• bugs: bug 1109218, bug 1121616
• progress since last update:
  • autoland service - no progress, waiting for mozreview ui to be ready for final testing

Better Bugzilla integration with MozReview [mcote]

• details: Make MozReview data in Bugzilla more useful by creating a Bugzilla field that contains dynamic information about a MozReview review request
• bug: bug 1102428
• progress since last update:

Supporting Tasks

• continue improvements to the multi-commit UI

Perfherder

Ingest all Talos data with Treeherder and develop a UI that can be used to view current and historical data [wlach]

• details: We want to use Treeherder to store and display Talos performance data, because Datazilla is being deprecated and Graphserver doesn't support the kind of performance analysis we'd like to perform on the data.
• progress since last update:
  • Initial PR landed. UI available at https://treeherder.mozilla.org/perf.html (but consider using stage for now: https://treeherder.allizom.org/perf.html)
  • Various UI nits solved which made it unusable (graph selector now more complete, now distinguishes e10s/pgo/etc.) http://tinyurl.com/krfklxf
  • wlach learning about perf sherriffing from jmaher, hopefully will make sure this work is actually useful

Treeherder

Distinguish between Tier 1 and Tier 2 jobs [mdoglio]

• details: Tier 1 and Tier 2 jobs will have different sheriffing guidelines and different expectations. Accordingly, we need to display them and allow users to interact with them differently.
• bug: bug 1113322
• progress since last update: validated requirements with bc (https://etherpad.mozilla.org/Mpd2vI1Cjl), started working on the ui bits on bug 1131071

Develop a prototype structured log viewer [camd]

• details: Develop a minimal structured log viewer which can be used to view the structured logs produced by the test harnesses. For the initial implementation, the user should be able to toggle between the parsed log viewer and the structured log viewer, for those harnesses that produce both.
• bug: bug 1113873
• progress since last update:
  • Now loading targeted lines (the log area surrounding the error line you clicked). though this is very rough still
  • not yet ingesting actual structured logs in the service (using sample data still)

Develop a minimal UI that sheriffs can use to file new intermittents [edmorley]

• details: develop a usable but minimal UI that sheriffs can use to quickly file new intermittent issues; the UI should automatically fill out some of the details that sheriffs normally have to find manually. Future iterations can improve on this by auto-filling more fields.
• bug: bug 1117583
• progress since last update: None - working on reliability/infra issues instead. [Won't be on the call, since on 4G dongle until internet fitted at new flat & whilst it's pretty fast (14Meg d/l!) don't want to rinse data quota :-)].

Supporting Tasks

• Continue to improve the performance and operational aspects of the system
• Identify and resolve remaining issues blocking TBPL EOL - treeherder parts: bug 1059400, other parts: bug 1054977

Backlog

• Change the authentication model to separate credentials from repos
• Implement OrangeFactor within Treeherder
• Create a developer-centric view

Bugzilla

Implement an alternate bug view [glob]

• details: Implement an alternative view of bugs, to provide UX and responsiveness improvements, and as a foundation for task-/team-centric views.
• bug: bug 1068655
• progress since last update:
  • now possible to update bugs
  • methodically working through adding all the fields
  • status: on track, however there probably isn't much slack time in the schedule now

Implement versioning framework for REST API [dkl]

• details: Version the REST API to provide stable endpoints for users and a place for unstable development.
• bug: bug 1051056
• progress since last update:

Along with some non-goal work last week, I have made some good progress to finishing up the first version of this for review/feedback. Iterated through several different methods of implementing this in the current Bugzilla code structure as to make all dependent code work with the minimum of changes needed. Think: testing scripts, installation code, etc. Working today to finish debugging some last issues needed before submitting for review. IMO, still on track for delivery.

GitHub authentication [dylan]

• details: Allow users to authenticate with Bugzilla using their GitHub account. This will encourage more contributors and allow us to better integrate GitHub into the Mozilla workflow.
• bug: bug 1118365
• progress since last update:

There are some minor changes on the table that came up during my last 1:1 with glob -- but no progress was made in the last week due to a combination of servicing other requests (forms) and travel to FOSDEM.

DevTools Harness

Get the DevTools harness running in continuous integration [ted]

• details: Take the prototype that was developed in 2014 Q4 (https://github.com/luser/luciddream) and get it running in continuous integration and visible in Treeherder. It's TBD whether this will be run in buildbot or TaskCluster, but we should get it running somewhere per-commit this quarter against linux desktop Firefox and a B2G emulator.
• progress since last update:
  • None; blocked on finding new owner since Ted has been pulled off on an iOS project

CloudServices Automation

Supporting Tasks

• Take existing client/server automation that is already running and get it reporting to Treeherder using the Tier 2 UI/workflow; bug 1108259

Test Infrastructure

Define and document Tier 2 jobs [bc]

• details: Define and document all aspects of Tier 2 jobs: rationale, criteria, necessary enhancements to Treeherder and automation frameworks.
• bug: bug 1121655
• progress since last update:
  • Attended Treeherder weekly meeting and later met with mdoglio
  • Collecting requirements.

Prototype a retrigger-based bisection tool [armenzg, jmaher]

• details: Create a prototype of a command-line tool that can be used by sheriffs and others to automate retrigger-based bisection. This could be used to help bisect new intermittent oranges, and to backfill jobs that have been skipped due to coalescing. Integration with Treeherder or other service will be done later.
• progress since last update:
  • A lot of progress has been made
  • A couple of basic releases have been made
  • We can now trigger jobs without needing to find installer and tests.zip url
  • CI and docs are in place
  • Roadmap is in the works
  • pushlog support has been added to iterate through pushes
  • Improved self-serve job scheduling determination has been added
  • Basic bisection tool is now WIP

Store high-resolution testcase data ("ActiveData") [ekyle, ahal]

• details: Create a Proof of Concept “big data” project which will store information about every test file we run: test status, error details, test machine and test duration to begin with. We will use this project to develop schemas and queries that work with data this large, and we will use this data to normalize chunk sizes and provide details about which tests never fail.
• progress since last update:
  • Currently integrating the ETL code and Ahal's worker management code. Works in development, but needs to be deployed on AWS
  • Enhancing the ETL to deal with 600Meg structured logs; ensuring streams are used everywhere, otherwise Python runs out of memory.
  • ETL is filling the ES cluster (cluster still only one node), but way too slow to keep up.
  • Started building the ActiveData service (https://github.com/klahnakoski/ActiveData); focus is on test infrastructure to ensure service can be tested, and more tests added easily.

Implement the ability to normalize chunk durations in mochitest [ahal]

• details: For mochitest variants on desktop and B2G, modify manifestparser and the test harnesses to be able to specify which tests are run in specific chunks.
• stretch goal: Implement the same feature for Android mochitest, which still uses old-style JSON manifests.
• bug: bug 1124182
• progress since last update:
  • Implement filtering system for manifest.active_tests (landing)
  • Create chunking filters in manifestparser and get mochitest to use them (patch finished, needs testing+review)
  • Create tool to collect test runtimes, on interim basis until active data project ready (done)
  • Implement algorithm for chunking based on test runtimes (not started)

Create Android 4.4 emulator image for automated tests [gbrown]

• details: Continue the work in bug 1062365 to build an emulator image based on Android 4.4 that is capable of running automated tests.
  Deliverable includes:
  • a prototype image
  • instructions for re-creating the image
  • demonstration that tests can be run on image
    NOT included in this deliverable:
  • tests running in continuous integration
  • "greening" of tests
• progress since last update:
  • now able to push to try with adb + new emulator + new avds
  • xpcshell tests run well on 4.4
  • all browser tests fail on 4.4 due to egl failures on startup (on aws only)
  • ...but this problem is not seen with Android 4.2 or 4.3; looking at differences; may modify goal to 4.2, 4.3, or 5.0

Help Releng reduce test load [jmaher]

• details: This quarter, we’ll validate the data from SETA and provide some recommendations to Releng about which jobs/platforms we could schedule less often in order to reduce test load. We’ll monitor the impact of these changes in terms of sheriffing burden and the number of retriggers this demands, and may adjust as needed. In subsequent quarters, we’ll use additional data from the high-resolution testcase data project and OrangeFactor to provide more finely-tuned scheduling changes.
• progress since last update:
  • [SETA webUI] available
  • [blog post] explaining SETA
  • TODO: backfill script for sheriffs, nudge releng to get this going

Supporting Tasks

• Help green up tests on OSX 10.10
  • bug 1121199 - tracking all work left- major thing is runtime of debug tests
• Apply --run-by-dir to all mochitest harnesses
  • bug 1110982 - landed change for mochitest-chrome, remaining is mochitest-plain
• Remove legacy JSON manifests in favor of manifestparser manifests
  • bug 1083347 - main bug remaining, most of the mozharness dependencies are rolled out
• Provide alternate solutions for the last consumers of Datazilla and work to decommission it
• Work with devs to introduce more dynamic analyzers (like Ehsan’s setTimeout check) in test harnesses
• Automate Windows symbol fetching, bug 1117741 [ted]
• Add ssltunnel support to Android tests, bug 1084614

Performance Testing

Deliver training to at least 2 people for Talos performance sheriffing [jmaher]

• details: We want to expand the pool of people who can perform performance sheriffing to make it scale better, and to reduce the bus factor problem.
• progress since last update:
  • [Docs] written - a good first draft
  • 1 trained
  • 2 others about 25% trained

Supporting Tasks

• Continue sheriffing Talos performance regressions
  • up to date so far
• Add new benchmarks as needed to mozbench
• Create a new UI for mozbench results that doesn’t requite Datazilla
• Improve e10s support for Talos tests and infrastructure
  • regression on TART for linux/windows
  • osx doesn't work
  • session restore is still broken
  • windows 7 PGO only dromaeojs issue with non e10s using e10s code path
• Move Talos into the tree
• Get rid of talos.zip
• Make running Talos locally easier
• resources: jmaher, dminor (for mozbench primarily), and contributors

Community

Increase 'contributor friendliness' of our projects [jmaher, all]

• details: Ensure that all ongoing projects have a friendliness rating of at least 6, as shown on https://wiki.mozilla.org/Auto-tools/Projects/Everything
• progress since last update:
  • prototyped out a [change to the project template]
  • TODO: change template, apply to 5 projects, assess, roll out to all existing projects, verify trello board has list of all projects; allow for exceptions (i.e. bugzilla)

Supporting Tasks

• Start tracking at least three community-related metrics over time
• Started to add contributor metrics - [ekyle] - (require MozillaVPN) http://people.mozilla.org/~klahnakoski/contributors/metrics.html#sampleInterval=week&sampleMax=2015-02-14&sampleMin=2014-10-05

Other Project Updates

• Autophone - started the process of replacing the mac mini host with surplus linux servers: bug 1129534
• Bughunter - completed retesting of Q4-2014 urls, restarted testing new urls, working with IT to further isolate Bughunter workers by using bughunter specific proxy and dns servers.
• charts.mozilla.org - [ekyle]
  • Graphics and DOM teams added to the Platform dashboard (eg http://people.mozilla.org/~klahnakoski/platform/dashboard.html#team=Graphics) feedback still coming in. Page is still slow.
  • Still working with FxOS Devices dashboard (http://people.mozilla.org/~klahnakoski/devices2/dashboard.html) hope is to be able to sign-in to get confidential bugs descriptions. Devices team is doing the investigation.

Holidays and Trips

Misc