Auto-tools/Meetings/2012-06-11

From MozillaWiki
Jump to: navigation, search

Goals Progress

Below are our top level goals. Beneath each goal, note any progress made towards that goal this week.

  • [ON TRACK] Extend Mobile Platform Automation for B2G and Fennec Native to extend our automation systems to work with specific phone hardware and new development boards for both products.
    • [wlach] - Added some preliminary support for the Pandaboard to Eideticker (commit), used for testing performance in bug 751347.
    • [jgriffin] - Turned on mochitest-plain on the B2G continuous-integration system: bug 759887. Need to investigate some random failures which are causing chunks to fail to start.
    • [mdas] - Added code to Marionette's perf test framework to post data to Datazilla!
  • [ON TRACK] Provide generic interfaces/web app plugability for new harnesses to reuse the same infrastructure backend
    • [jeads|carljm|camd] We can now create new datazilla projects in an automated fashion. This includes building an object store and performance database to collect json objects and index them in an RDBS schema. Once a project is created, it can be posted to immediately. This is available for testing on Mozilla-MPT vpn. Contact jeads or carljm if you need to start a project.
  • [MISSED] Ensure that new UI is based on extensible statistics package that can be used both by developers and the graphserver UI.
    • [jeads] This goal was deferred in favor of new goal to detect regression per push. This doesn't mean there is no UI for datazilla, there is a "stock" UI available that allows you to examine all data submitted. We stopped extending it in mid April to concentrate on setting up a system that can detect a performance regression in an automated fashion.
  • [ON TRACK] Signal From Noise Phase II - Make the same noise-reduction changes we made on Tp5 on all the other page-load tests and ensure all performance tests are sending raw observations to Datazilla.
    • [jmaher] - all changes to tests are running side by side on mozilla-central.
  • [AT RISK] Upgrade bugzilla.mozilla.org to Bugzilla version 4.2
    • [glob] assisted fox2mike in setup of bugzilla.allizom.org
  • [ON TRACK] Refactor how rapid-release tracking flags are implemented for improved performance and maintainability
  • [ON TRACK] Reduce android test automation instability and make it easier for the web QA and desktop QA teams to write and run automated tests.
    • [jmaher] - Worked with Callek to get updated tegra-host-utils.zip, landed, waiting on deployment

In-Depth Project Discussion

This week, jeads will be covering Datazilla!

The Overview

Mozbase

Mozmill

  • [henrik] Complete test refactor of Mutt JS tests (nothing touches the net anymore!!)
  • [henrik] Finally removed the JUM unit test module
  • [henrik] Mozmill opens target application in the background now!
  • [henrik] Enhancements for the screenshot feature

Mochitest

  • [jmaher] - plan to turn on 200,000+ new mochitests for android native this week

Bugzilla

  • june 7th updates: http://globau.wordpress.com/2012/06/07/bmo-push/
  • [dkl+glob] work on the ongoing database cleanup following the failed database consistency correction
  • [glob+dkl] misc upstream reviews and patches
  • [dkl+glob] bmo administration work
  • [glob] blocked extremely broad query from labs which dropped the load across the webhead cluster by 40%

Eideticker

  • (as above) [wlach] - Added some preliminary support for the Pandaboard to Eideticker (commit), used for testing performance in bug 751347.
  • [wlach] - Did some preliminary tests on Galaxy Nexus for Fennec vs. other browsers (Chrome, Dolphin, Opera, etc.) for marketing and qualification stuff.
  • [wlach] Various help getting the testing setup in MV back online, dashboard should hopefully be running this week.

Peptest

  • [mcote] Fixed bug causing problems in OS X runs.
  • [mcote] Met with Christina and Gilbert from analytics about peptest statistics and forecasting.
    • Given their experiences with talos, they wanted more information and justification about the methods we are using. Specifically, they were concerned with the fact that we only record the duration of the event loop when it exceeds 50 ms; when it is less, we record nothing. They would love to see complete data on the event-loop duration for a test run or two. They also had concerns with the way threading may affect the results.

Sheriffing & Trees

treestatus.mozilla.org

Tinderboxpushlog

War on Orange

  • [mcote] Still working on Orange Seed. Got the data I need and able to roughly correlate test-run and orange data.

Signal From Noise

  • [jmaher] - final changes to tests are running [side by side]

Talos

  • [jhammel] working on making results reporting more extensible and sane (needed for jetperf and talos+mozharness)
  • [jmaher] - dromaeo raw results landed (not deployed), will work on xperf raw results
  • [jmaher] - will be working on removing enablePrivilege from talos this week

Mobile Automation

  • [jmaher] - progress is made on closing out the bugs that cropped up during turning on the reftests. This results in more tests being ran instead of skipped or random-if'd
  • [jmaher] - should have adb logcat support added to the mochitest and reftest harness by the end of the week.

Marionette

  • [mdas] finishing up perf test work, will move on to fixing regressions
  • [jgriffin] - Turned on mochitest-plain on the B2G continuous-integration system: bug 759887. Need to investigate some random failures which are causing chunks to fail to start.
  • [mdas] - Added code to Marionette's perf test framework to post data to Datazilla!

Web QA

  • [dburns] Released Selenium 2.23

Bughunter

  • [bc] crash triaging, test case reduction and regression finding continues
  • [bc] working on binary search tool using nightly opt and debug builds for beta, aurora and nightly builds.

New Graphserver (Datazilla)

  • [carljm] database isolation for datazilla tests
  • [carljm] investigated querying hg.mozilla.org for changeset ordering
  • [camd] code work related to implementing the objectstore usage for data input
  • [jeads|samliu] Set up mariadb and the TokuDB storage engine for collecting performance data. Transferred ~340,000 json structures, about half of the development talos data as a test. On a qualitative level, mariadb performs substantially faster than MySQL and InnoDB.
  • [jeads] Wrote a push walker script to examine t-test results for a child/parent push chain.
  • [jeads] Refactored the data ingestion method to make use of MySQL's INSERT … ON DUPLICATE KEY UPDATE strategy. This should reduce load and potential concurrency issues.

autophone/Noah's Ark

  • Autophone server and devices moved to new subnet. adb now appears to be totally crazy on the server, even after a reboot. :\

Speedtests

  • Nada.

Jetperf

  • [jhammel] figured out who was working on what and the last missing pieces
  • [jhammel] fixed last outstanding nits; working on reporting to datazilla

Pulse

MozTrap

Powerball

Systems

Mobile Evangelism Site-Scraping Tool

  • [carljm] put together and presented project proposal/estimate

Firebug Testing

Upcoming Events

All times Pacific Time, click on link to find your timezone

Round Table

  • The list of standing weekly meetings somehow got dropped from recent meeting pages, and is anyway out-of-date. Let's keep it up-to-date!
  • The new goals section is great, but what about the set of highlights that we used to copy & paste into the wiki for the general meeting. Not everything we do necessarily corresponds to a top priority goal. Should we bring something like that back?

Take notes here and copy and paste them to this section afterwards.

Notes

  • We decided to bring back the highlights section at the top of the meeting wiki; this will be used to place announcements and other activities that someone wants to highlight. The MC won't need to fill out this section.

Holidays and Trips

  • Clint is out from June 8 to June 17 - back on the 18th. See Jgriffin, Mcote, Jmaher, or Bmoss if you need something urgent while I'm out. I will be reachable by cell. Feel free to call.
  • Jgriffin is out from June 20 to July 2. See Mdas for B2G issues or Ctalbert for anything else. I will be reachable by cell phone; ctalbert has the number!
  • wlach out from June 18 to July 2. Mcote to keep eideticker dashboard running while gone.

Next week MC

  • mcote is MC next week
  • next week's in-depth project discussion will be about MozTrap, led by camd

Misc

  • Sebastien Thrun is offering a free course on statistics late June through Udacity, which might be interesting for some people given our recent focus on more rigorous analysis in various projects (Datazilla, Talos, Eideticker, Peptest, etc.). I (wlach) took Professor Thrun's course on AI last fall and it was great: http://www.udacity.com/overview/Course/st101/CourseRev/1
Retrieved from "https://wiki.mozilla.org/index.php?title=Auto-tools/Meetings/2012-06-11&oldid=439989"