Auto-tools/Meetings/2013-04-29
From MozillaWiki
< Auto-tools | Meetings
Contents
Notices, Highlights, Roundtable
- 1/3 of the way through the quarter. Any goals in trouble?
- [bc] I'd like to discuss some ideas for statistical methods for obtaining reliable performance measurements.
- Background discussion bug 866195 Improve Autophone s1s2 test reproducibility
- Using mean and standard error of the mean rather than mean and standard deviation of measurements.
- Choosing number of measurements and a target percentage standard error of the mean.
- Rejecting and re-testing measurements with overly large percentage standard error of the mean.
- Filtering outliers using quartiles: http://en.wikipedia.org/wiki/Interquartile_range
- Choosing a device settling time before starting tests.
Goals
- Goal: Improve Developer/QA ergonomics around running and debugging automation by:
- Add debugger support to B2G testrunners
- Improve patch review experience on bugzilla/Make bugzilla more user-centric
- Replace graph server with Datazilla for all desktop Talos tests
- Create a project branch view of B2G builds/tests that tracks commits per target, rather than per repo (all commits to any repo for unagi for instance) joint with releng
- Goal: Improve stability, reliability, and scalability of automation platforms by:
- Addressing all P1 B2G automation stability issues
- Complete native REST interface for Bugzilla
- Complete Mochitest to Mozbase port
- Create POC Microbenchmark suite to get feedback from developers for its usefulness
- Reduce orange factor to 5 or lower (for the last three weeks of the quarter) by driving awareness among developers, following up with assignees, and fixing intermittent automation failures.
- Update pulse so that its dependent (client side) libraries are current for its use in Tree Herder
- Create a prototype and a specification for a rig to power cycle real phones. Have a vendor selected to build the rig by end of quarter.
- Goal: Build toward our future by:
- Deploy a minimally viable TBPL replacement by end of quarter, aim to switch over mid-Q3
- Release a mozmill 2 and work with contributor community to drive toward a 2.1
- Automate a scrolling fps test using camera Eideticker measurements for Contacts, Email, Settings, and Message apps
- Create and Deploy BMO user profile data for community building and education
- Create a solid net and filter system to build a bugmaster community
- Identify 5 things we can do to increase our infrastructure capacity by 10% through more streamlined usage and bringing new capacity online. (joint with releng)
- Enable performance and test automation to run on Android x86 hardware, investigate Android 2.3.x support on Pandas.
Deep Dives
Mobile (1st Monday)
AutoPhone/Noah's Ark
[bc] Working on bug 866195 Improve Autophone s1s2 test reproducibility
B2G Automation
Mobile Evangelism Site-Scraping Tool
Android Automation
- new sutagent/dm changes to support fix for process monitoring. This is root cause of 2400 second timeout and should allow robocop on pandaboards to be visible and much greener
- work towards moving webgl into separate suite to green up M1 and hopefully the newly created Mgl
- x86 automation via emulators and phones is in progress
Tools (2nd Monday)
Bugzilla
Bughunter
- [bc] Need to catch up on filing bugs.
GoFaster
Pulse
MozTrap
Tinderboxpushlog
- [edmorley] New things in production: https://bugzil.la/865451
- [edmorley] Patch to add support for mochitest metro chrome (bug 865451).
War on Orange
- OrangeFactor on trunk for last 7 days: 7.78
- [edmorley] Tested/gave feedback on the new OrangeFactor UI layout in bug 738612.
TBPL Version 2
- [edmorley] Collated the initial treeherder reference data.
- [camd] branch for Job data ingestion was merged in
Performance (3rd Monday)
New Graphserver (Datazilla)
Signal From Noise
- solved ingestion problem
- work on time based ui underway
- work on alert system and integration with email client
Speedtests
Eideticker
- [wlach] Comparative pageload tests on Android
- [wlach] Work on integrating b2g unagi tests into dashboard as proof-of-concept
Talos
General Framework Support (4th Monday)
MozBase
In-tree Automation/Harnesses
- [edmorley] Patch to tweak several jsreftests to avoid TBPL parser false positives (bug 864694).
- [edmorley] Patch to make auto clobbering opt-in rather than opt-out & increase test coverage (bug 863091).
- [edmorley] Reviewed bug 822821 and bug 813022.
WebRTC Test Development
Process (4th Monday)
Sheriffing & Trees
- [edmorley, RyanVM] Various discussions on dev.platform about inbound closures, multiple inbounds, wait times & strategies for improving developer efficiency.
Treestatus
Bugmasters
Holidays and Trips
- Monday 6th May is a UK bank holiday.