Auto-tools/Meetings/2012-07-16
From MozillaWiki
< Auto-tools | Meetings
Contents
- 1 Goals Progress
- 2 Highlights
- 3 In-Depth Project Discussion
- 4 The Overview
- 4.1 AutoPhone/Noah's Ark
- 4.2 Bughunter
- 4.3 Bugzilla
- 4.4 Eideticker
- 4.5 Firebug Testing
- 4.6 Jetperf
- 4.7 Marionette
- 4.8 Mobile Automation
- 4.9 Mobile Evangelism Site-Scraping Tool
- 4.10 Mochitest
- 4.11 MozBase
- 4.12 Mozmill
- 4.13 MozTrap
- 4.14 New Graphserver (Datazilla)
- 4.15 Open Web Apps
- 4.16 Peptest
- 4.17 Powerball
- 4.18 Pulse
- 4.19 Signal From Noise
- 4.20 Speedtests
- 4.21 Systems
- 4.22 Talos
- 4.23 Sheriffing & Trees
- 4.24 Treestatus
- 4.25 Tinderboxpushlog
- 4.26 War on Orange
- 4.27 Web QA
- 4.28 Automation Development
- 5 Upcoming Events
- 6 Round Table
- 7 Misc
Goals Progress
Below are our top level goals. Beneath each goal, note any progress made towards that goal this week.
- [ON TRACK] Support the B2G and platform team by moving existing automation to that platform as well as expanding automation on B2G and B2G devices for basecamp deliverables.
- [ON TRACK] Expand Mobile testing platforms for B2G and Fennec Native to include user devices, NEON Chipsets (panda boards), and ARMv6 (Beagle boards).
- [mcote] Project plan in place for AutoPhone.
- [mcote] Refactoring and self-tests.
- [ON TRACK] Establish new Datazilla system as a useful tool for capturing and diagnosing talos performance regressions on a per-push basis, as well as capturing data for network and B2G performance tests.
- NOTE: This is largely a change to the data harvesting and analysis side of the Talos system (even though some changes are required to the test harness and tests themselves). We will not be replacing existing Talos data harvesting system in Q3. By the end of Q3, we want to have the new performance data harvesting and analysis system Datazilla to a point at which it can shadow the old system and we can use Q4 to verify that the new approach captures just as many regressions as the old system does. Potential timeframe for conversion of the old system to the new Datazilla system is in Q1 2013 (earliest "thumb in the wind" estimate).
- [ON TRACK] Upgrade BMO to 4.2, deploy tracking flags optimizations, UX improvements (comment tagging, mid-air resolution)
- blogged about 4.2 search changes (http://globau.wordpress.com/2012/07/13/bugzilla-4-2-whats-new-searching/), bmo/4.2 testing, and bugfixing
- [ON TRACK] Make TBPL More Orange aware to make it easier to file intermittent orange bugs, and so that we can more adequately track (and therefore address) issues around android-related and memory leak intermittent failures
- [edmorley] Fixed bug 772388 - Make ShutdownLeakLogger output the list of leaking tests in a format that can be individually starred.
- [ON TRACK] Enable QA and contributors to increase their skills and participation by crafting automated tests for B2G and desktop smoketests by providing examples, mentoring, and documentation
Highlights
This is a place for people to put announcements or to describe activities that they'd like to highlight. It's not the responsibility of the MC to find things to put here.
- [DBurns] Last week it was hopefully, but this week it is out! Browser Automation Spec is in FPWD
- [ted] Got B2G crash reporting wired up, bug 761909. Sample crash report: https://crash-stats.mozilla.com/report/index/6f54b900-a9f0-468e-9e53-f795a2120716
- [dkl] Got X-Headers in bugzilla implemented for tracking flag changes (x-bugzilla-tracking) (per a dev.planning thread)
- [edmorley] Posted about TBPL/OrangeFactor changes in dev.platform.
In-Depth Project Discussion
- Ahal reviewed peptest framework
The Overview
Progress, notes, problems, and other statuses about current projects. Add any new projects to this template!
AutoPhone/Noah's Ark
- [mcote] - Added a few unit tests.
- [mcote] - Some refactoring and lots of documentation.
- [mcote] - Developed project plan coordinated with releng's plans for foopy-less tegras.
Bughunter
Bugzilla
- bmo changes: http://globau.wordpress.com/2012/07/12/happy-bmo-push-day-4/
- [glob] blogged about 4.2 search changes (http://globau.wordpress.com/2012/07/13/bugzilla-4-2-whats-new-searching/)
- [glob+dkl] bmo/4.2 testing, and bugfixing
- [dkl+glob] component searching enhancements (added to component browsing, pressing enter with a single result selects it)
- [dkl] added x-bugzilla-tracking headers to bugmail (results of dev.planning discussion)
- [glob+dkl] upstream development work and reviews
- [glob+dkl] administration work (including rapid release 17 work)
Eideticker
- [wlach] - Finished merging Orangutan input simulation framework into master branch (see: http://wrla.ch/blog/2012/07/the-evolution-of-simulating-events-in-eideticker-from-monkeys-to-orangutns/)
- [wlach] - Set up Nagios to monitor status of Eideticker machines / phones
- [wlach] - Various work to try and make Eideticker dashboard update scripts more resilient (retrying failed jobs, etc.)
- [wlach] - In Toronto this week to set up Eideticker machines and help anyone interested get up to speed on running/developing Eideticker tests.
Firebug Testing
- [ahal] Option to specify custom database in order to move away from couchone
Jetperf
- [jhammel] initial config files up for feedback: https://bugzilla.mozilla.org/show_bug.cgi?id=767117
Marionette
- [DBurns] Currently aligning client API with Selenium which allowed me to do what I did in the Open Web Apps section
- [mdas] added a workaround for bug 771224 so chrome tests can run again.
- [mdas] readjusting performance test runner job, adding one more perf test bug 774278
Mobile Automation
- [jmaher] - investigating problem when boards reboot in the middle of a test with no indication of what is happening.
- [jmaher] - investigating resource consumption during tests to see if that causes reboots/crashes/oranges.
- [wlach] - various patches to SUTAgent and client library, mostly to fix eideticker-related stuff but may be useful elsewhere as well: commands now run as su by default (bug 763497), fall back to wifiman to get MAC address on older versions of Android (bug 768597), give better error when we fail to push a file (bug 772105), and don't consume tons of CPU when waiting for a response in the client (bug 773463).
- [ahal] - b2g reftests running again after emulator fix and fix to bug 772307
- continuing triaging process
- [mihneadb] - got xpcshell tests running in b2g emulator
- [mihneadb] - various fixes/refactorings to marionette and devicemanager
Mobile Evangelism Site-Scraping Tool
- [samliu, carljm] fixed a number of bugs in the scraper, worked on writing tests
Mochitest
MozBase
Mozmill
MozTrap
[rbillings] - Doing a test run pass on the 1.1.X codebase
New Graphserver (Datazilla)
- [carljm] Added support for Oauth to datazilla_client (also added tests, fixed a couple bugs, and added support for sending prebuilt JSON)
- [camd] Added storage and importing of pushlog data, merged to master
- [camd] Adding batch processing to certain management commands for cache updates and data processing. Added batch field to be able to assign projects to specific cron batches. (not yet merged)
- [jeads] Pulled datazilla data for the talos test suite tp5n from 06/20/2012 to 07/05/2012 and ran t-test metrics on all of the associated pushes. This time range contains the changeset a9ae09d4f1d8 which caused a performance regression. Metrics is analyzing the data to further validate the t-test method.
- [jeads] Cleaning up some loose ends to get ready for a release. This includes adding handling to set a default branch, making some schema changes, and cleaning up how reference data is cached.
Open Web Apps
- [DBurns] I have marionette working logging in and installing web app. Login done via WebQA BrowserID Page Objects
Peptest
- [ahal] Wrote test to catch unresponsiveness in tab switching
- updating mozharness script to provide proxy sites
Powerball
Pulse
Signal From Noise
- [jhammel] working on a tool/webpage to correlate names from buildbot, talos, TBPL, and graphserver for bug 770460; a (messy) WIP may be seen here: https://bug770460.bugzilla.mozilla.org/attachment.cgi?id=641530
Speedtests
Systems
Talos
- [jmaher] - worked with jhammel to get reporting to graph server for new sunspider, v8, kraken tests
- [jmaher] - wrote blog post about compare.py for comparing talos numbers from graph server.
- [jmaher] - backed out specialpowers for talos as it caused massive performance regressions.
Sheriffing & Trees
- [edmorley] Day-to-day sheriffing of the trees.
- [edmorley] Infra/tree closure issues:
- Tracking down winxp pgo-only jsreftest failure (problematic due to not doing pgo-per-push & high levels of load at the time). Was due to bug 772303 and meant an 8-bug backout due to conflicts and dependencies :-/
- Resolving several layers of webgl bustage, masked by coalesing.
- Helping #it with debugging of bug 766810, which intermittently prevents people from pushing to inbound.
- [edmorley] Updated all Wiki links for the clobberer to the new location & got IT to correct the revision in production, since the last rollout had been to the old site only, meaning different people were using different versions of the clobberer without realising it, *facepalm*.
- [edmorley] Email discussions with joduinn about his idea of adding extra integration repos to reduce mozilla-inbound contention at peak times.
Treestatus
Tinderboxpushlog
- [edmorley] Discussions about moving primary TBPL development out of a user repo and under hg.mozilla.org/{automation,webtools,build}.
- [edmorley] Fixed bug 772084.
- [edmorley] Items to ease starring/identification of failures more easily:
- Fixed bug 772388 - Make ShutdownLeakLogger output the list of leaking tests in a format that can be individually starred.
- Filed bug 772610 - results.py shouldn't say "Graph server unreachable" when the test name isn't recognised.
- [edmorley] Posted about TBPL/OrangeFactor changes in dev.platform.
War on Orange
- [jgriffin,edmorley] Resolving: OrangeFactor not updating since July 5th.
- [edmorley] Reviewed mcote's Orange Seed patch.
- [edmorley] Landed bug 769706.
- [edmorley] Patch for bug 773312.
- [edmorley] Posted about TBPL/OrangeFactor changes in dev.platform.
Web QA
Automation Development
- [jmaher] - bug 762908 - add specialpowers to jsreftests - in review queue, required for landing ionmonkey
Upcoming Events
Standing weekly meeting related to our projects should go here
Round Table
- Review the goals - are they specific enough while maintaining some amount of flexibility (since this is going to be a quarter that requires a ton of flexibility from us), do you have an idea how we'll measure them?
- Discuss how to attack the Reftest Resolution problem for mobile. Come up with plan of attack and priority in terms of other actions.
Take notes here and copy and paste them to this section afterwards.
- We discussed the goals, refining the language of them
- We watched a fantastic presentation overview on peptest by Ahal
- Held a long discussion around how to address the reftest screen size issue:
- Reftest requires a resolution that is too large for mobile phones - need to figure out a way to reduce it so that we can perform true reftests on mobile and resolution constrained devices
- Decided to change reftest to save all the canvases at the high resolution, then compare a subset of that canvas at our target small resolution, to determine tests that might fail (i.e. would be drawing anything out of the frame in the smaller resolution). We will generate a list of tests for further investigation from the set of tests that draw content outside the small resolution. Jmaher and ahal are going to take a crack at instrumenting reftest to support this and create the list.
Notes
Holidays and Trips
- [DBurns] Tuesday 17 July off to sort house for window fitters
Next week MC
- Ted will be the MC
- Wlach will present Eideticker!!
Misc
Find something you think is cool, interesting, funny, or exciting and put it here
- Damn you Autocorrect, in the spirit of all our mobile work
- [wlach] hired bicycle movers to move apartments last weekend. Truly a sight to behold. See also: http://www.demenagementmyette.ca/videos/
- [edmorley] A friend had this cake made for his birthday by his other half: pic. Jealous! :-D