Mobile/Testing/06 05 13

From MozillaWiki
Jump to: navigation, search

Previous Action Items

  • (gbrown) generate a plan to get reftests on for pandas
    • see dminor's excellent efforts in bug 877266 Add reftests sandbox flag to distinguish between Android 2.x and Android 4.x and update manifests
  • (blassey) send email to mrz about the tegra move

Status reports

Dev team

Rel Eng

  • (rail) bug 878123 - Please start building the x86 APK in Firefox 22b4
  • (kmoir) running panda android non-talos tests in staging with mozpool/mozharness

IT

A Team

  • tegra failure rate [4.69%]
    • m1, m3
  • panda failure rate [3.09%]
    • m2, m3
  • orange factor bugs:
    • bug 807230 - Intermittent DMError: Automation Error: Timeout in command {ls,ps,isdir,mkdr}
      • This used to reproduce locally, but I (dminor) have had problems getting it to do so lately. I now have a version of the smoketest scripts running locally. Based on logs from April, smoketests used to get it to show up, so maybe this will do the trick.
    • bug 711725 - Tegras and Pandas disconnect with "remoteFailed: [Failure instance: Traceback (failure with no frames): <class 'twisted.internet.error.ConnectionLost'>: Connection to the other side was lost in a non-clean fashion." or similar at any given step
    • bug 817024 - Tegras/Pandas disconnect with "remoteFailed: [Failure instance: Traceback (failure with no frames): <class 'twisted.spread.pb.PBConnectionLost'>: [Failure instance: Traceback (failure with no frames): <class 'twisted.internet.error.ConnectionDone'>..."
    • bug 845162 - Intermittent Android timeout in test_replay_metadata.html,test_playback_rate.html,test_bug495145.html,test_chaining.html,test_bug465498.html,test_reset_src.html,test_seek.html,test_seek_out_of_range.html,test_mozLoadFrom.html,test_reactivate.html
    • bug 722166 - Fennec intermittently crashes on startup ("Automation Error: No crash directory (/mnt/sdcard/tests/profile/minidumps/) found on remote device" or "Automation Error: No crash directory (/mnt/sdcard/tests/reftest/profile/minidumps/) found on remote device")
  • Reftests on panda
    • bug 877266
    • A few failures that appear to be actual bugs, possibly due to Android version (I hit the same failures on X86 reftests)
    • Some unexpected passes
    • Another 50 or so which seem like they can be handled with fuzzy-if, mostly failures due to slight differences in pixel values e.g. (4, 0, 8) instead of (0, 0, 0). I'm hoping to run a few of these on my phone to see if this is a panda specific problem.

x86 automation

  • I'm not planning to do any further work on modifying the reftest manifests for X86 until after the panda reftests are working. There are some failures in common between Android X86 and the pandas.

Autophone

  • [mcote] Finally starting a blog post about autophone and some lessons learned. Watch for it on Planet or http://mrcote.info/ in the next day or two.
  • Acquired 20 sdcards and put the nexus one devices back into production.
    • Retested from 2013-05-25 and throbber start times appear smoother
    • Nexus one, Droid pro show increased times due to sdcard change
    • NOTE: Only phones (nexus one, droid pro) without an internal partition for /mnt/sdcard use or are affected by the presence of an external sdcard.
  • Results
    • Apparent regression in uncached Local Blank Page total throbber time for nexus one, droid pro, uncached Local Twitter Page for droid pro late May 31.
    • Apparent improvement in uncached Remote Blank Page, uncached and cached Remote Twitter Page total throbber time for droid pro late May 31.
  • Todo
    • validate existing regressions against new, more stable autophone
    • compare measured regressions in eideticker on april 18th, may 8th (http://eideticker.wrla.ch/#/lg-g2x/startup-abouthome-dirty/timetostableframe) with autophone data (probably time to load blank page?)
    • schedule meeting with mobile team to see if there are other things that autophone can measure that'd be more useful (wlach suggests measuring the time it takes for the gecko backend of fennec to load and also whether there are ways of presenting the data that would be better)

Eideticker

  • Eideticker has a new home at http://eideticker.mozilla.org (http://eideticker.wrla.ch still works)
  • Problems with Galaxy Nexus seem to be profiling related (bug 879424), so turned it off. Running builds without profiling for now (BenWa suggested to me in Toronto that profiling is not particularly useful with the current eideticker tests).

Round Table

  • [blassey] can we use autophone to test different GPUs for SkiaGL?

Action Items

  • Dminor to continue working on the reftest on panda issues
  • Melissa to follow up with mrz about the tegra move plan and get the details specified
  • Kim to follow up with callek and get a window for landing watcher changes
  • Kim to continue debugging mozharness/mozpool changes with pandas
  • bc & mcote to look into running a small set of reftests (maybe the small screen tests) talk with Snorp (james willcox) about what is required for skia GL on these phones, also ensure we are running an adreno phone in the mix http://androgameshd.blogspot.com/2012/06/list-adreno-devices.html
Retrieved from "https://wiki.mozilla.org/index.php?title=Mobile/Testing/06_05_13&oldid=663802"