Mobile/Testing/08 21 13

Previous Action Items

• kmoir to run the 1.19 sutagent in staging - point her to the zip file of it
  • http://people.mozilla.org/~gbrown/sutagent/sutAgentAndroid.1.19.apk
  • ran in staging for pandas, have to look at numbers
• panda issue - probably better to fix by breaking robocop into more chunks so that they can run faster, going to need to look at that.
  • gbrown will file a bug on splitting robocop out
    • bug 905274 kmoir wrote patch and landed change to production
  • gbrown, dividehex, ctalbert, and blassey will discuss experimenting with updating the android image the pandas use so that we mitigate some of these issues. (This might be something for a Q4 bucket list)
• Ctalbert to help blassey & co update b2g emulator to turn on hw accel and not have the reftests fail. bug 905141
• dminor and ctalbert to look into 690176 as to why it suddenly spiked (possibly mozharness related?)
  • [dminor] This stopped occurring early last week. There was a corresponding spike in non-mobile talos initialization problems as well, which has also stopped.
• gbrown to file bugs for crashes on x86 robocop in libgl. Maybe skia related?
  • persists with skia-gl disabled
  • bug 907383 filed
• Wlach is going to look into the issues with the watcher on the g2x. Interestingly, Patrick Mcmanus has a blog post about android power saving mode disabling the wifi radio - see if his change to prevent this might hav eadversely affected this phone (by comparing the time of this change with the increase in dropped network issues with the phone (verifiable through our nagios alerts).
  • Looks like the watcher is not being restarted by android such that it runs through its normal startup sequence, going to investigate and fix.
  • Resolution: At this point we've decided to just replace the phone. After a reset, the device just dies after an hour or so, whether tests are running or not. Tracking this in bug 904727.

Status reports

Dev team

• snorp has a panda, is currently investigating crash tests

Rel Eng

• (kmoir) landed changes to split robocop into three chunks bug 905274 and reftests into five chunks bug 905694, staging tests run for sutagent 1.19, panda talos mozharness testing

IT

• PO for X86 infrastructure has been submitted to the vendor, delivery is either this Friday 8/23 or Monday 8/26.
• Starting planning discussions for movement of infrastructure out of SCL1 and into SCL3. SCL1 lease is up on 7/31/2014.

A Team

• tegra failure rate: [0.54%]
• panda failure rate: [22.48%]
  • excluding retries: 0.71%

• Mobile Bugs in Orange Factor Top 10
  • bug 807230 Intermittent DMError: Automation Error: Timeout in command {ls,ps,isdir,mkdr}

• Patch landed for bug 858620 to add jit-tests to test package and bug 858622 to run jit-tests on mobile
• Will be writing a mozharness script for bug 858621 to actually run jit-tests from test package
• Filed bug 907351 Panda reftests now run significantly slower
  • On my local panda using an Aug 7th nightly, chunk 1 of 5 took 28 minutes 30 sec.
  • Using an Aug 10 nightly, chunk 1 of 5 times out after an hour having only completed 58% of the tests

x86 automation

• Slow, incremental progress
• Fig merge -> startup crash on x86

Autophone

• S1S2 Throbber start, stop tests 2013-08-14 - 2013-08-21
• Mochitest-Canvas with SkiaGL
• Status Updates
• May need to block certain devices from running Mochitest-Canvas with SkiaGL due to reliability issues.

Eideticker

• See roundtable item above on g2x
• Work on bug 888102 to track reception of input events landing soon
• Big regression in canvas clock test after skia gl landed: bug 904164

Round Table

• [gbrown] watcher update stalled
  • see https://bugzilla.mozilla.org/show_bug.cgi?id=876715#c42
• [gbrown] next step in war on panda retries: bug 895176
• want to measure responsiveness, how can we do it?

Action Items

• Clint to send snorp a tegra
• Kim to run sutAgent 1.19 on tegras
• Kim to follow up with pmoore about watcher update
• Dan to follow up with specific developers about what might have regressed reftest perofrmance
  • wlach to add in information of a similar regression within that range from eideticker as well
• Clint to ensure Armenzg has x86 as his top priority and get an ETA
• Bc to get AaronMT to help file bugs for crashes from autophone etc
• Wlach to get blassey a bug to annotate the types of responsiveness tests we want from eideticker - bug 904164
• Dividehex to comment in bug 895176 about how it could be related to mozharness and mozpool both trying to reboot the panda.