previous meeting | Meetings | next meeting »

Dial in

 # When:     Every Monday at 8:45am PDT/PST
 # Vidyo:    https://v.mozilla.com/flex.html?roomdirect.html&key=PGtLpx3XQGJz
 # Phone:    650-903-0800 or 650-215-1282 x92 Conf# 9654 (US/INTL)
 #           1-800-707-2533 (pin 369) Conf# 9654 (US)
 # IRC:      irc://irc.mozilla.org:6697/#automation
 # IRC logs: http://irclog.gr/#browse/irc.mozilla.org/automation

General

• Attendees:
  • TBD
• Chair: Andrei
• Notes: Henrik

Last Weeks Action Items

• [NEW] Henrik - Update the goals page
• [CARRY OVER] Anthony: Ensure any Metro tests are suitably tagged with in-testsuite?

Highlights

• [dave] Marionette client 0.5.24 released with several fixes for running tests on B2G desktop builds
• New blockers, any updates?
  • bug 862724 - Additional pulse message for same nightly builds sent (with previous_buildid == buildid)
  • bug 864294 - Activities database is not automatically recreated if it has been removed
• You can find all blockers here: http://bit.ly/Tph0NY

Roundtable

• How to implement Metro tests, as it looks like we will majorly be blocked on missing event actions in Mozmill. Should we consider Marionette instead?
• [henrik] Really awful proposal - Shall we drop work on Mozmill and start with Marionette for tests?
  • pro arguments
    • Only a few tests are using complex mouse events like drag&drop, and we do not support touch events at all in Mozmill
    • Marionette gets injected in the system process as extra library and doesn't have to be build as extension, which even is not supported for all products (e.g native web apps don't allow add-ons to be installed)
    • W3C compliance by using the web driver protocol (http://www.w3.org/TR/2013/WD-webdriver-20130117/)
    • Tests can even be run on mobile (Android, B2G)
    • Tests can be written in different languages (e.g. Python) and allow us to write system integration tests because we are not in the browser scope anymore
    • Support for (remote) debugging
    • Changes of getting a much broader community around desktop testing
    • Marionette has more active contributors than Mozmill
  • contra
    • First we have to figure out how to inject Marionette in default Firefox builds (see bug 836253)
    • Existing Mozmill tests would have to be transformed or we keep and run them in parallel (some tests would even be not necessary anymore due to new browser chrome tests)
    • Where would we publish our test results? Our dashboard would need a rewrite or we can directly push to tbpl
      • Probably TBPL2 (Tree Herder)
• Mozmill 2.0 triage part 2

Goals Overview

• https://wiki.mozilla.org/Auto-tools/Automation_Development#Prioritized_Projects

Project Updates

Gaia UI (Dave)

• Goals progress:
• TBD
• [dave] Marionette client 0.5.24 released with several fixes for running against B2G desktop builds
• [dave] Database cleanup pull is currently blocked by bug 864294
• [rwood] Submitted pull request for gaia-ui Bluetooth device discoverable test
• [rwood] Gaia-ui endurance tests:
  • First results submissions to DataZilla b2g project
  • Setting up jenkins on my test PC
  • Tests submitted for review
  • Once the tests are in the gaia-ui repo and more results are on DataZilla, I will give a talk about them in our meeting, write a blog post, etc.

WebAPI (Rob)

• No updates this week

WebRTC (Henrik)

• Caught a major regression (bug 863687) and leak for data channels last week (a single char change)
• Framework 70% done, but needs further cleanup before the first review

Softvision Updates

Desktop Automation

• Goals progress: Mozmill 2.0 release

• Fixed:
  • bug 757827 - Preferences are being lost between application restarts
  • bug 853478 - Junit report generation fails if Mozmill is not able to send the report to the dashboard
  • bug 848577 - Create a Mozmill Endurance test to catch regular expression memory leak (bug 837845)
  • bug 860330 - Test failure "searchTerm.getNode(...) is null" in testAwesomeBar/testLocationBarSearches.js
  • bug 708270 - Flash endurance tests expose potential performance degradation regression
  • bug 803489 - Software update tests on Windows 8 fail sometimes due to updater prompt on startup (jsbridge cannot connect)
  • bug 826251 - Make testPreferences/testPreferredLanguage.js language independend
  • bug 859746 - Mutt javascript tests do not print the RESULTS block

• Blocked bugs:
  • bug 763159 - Test failure 'secure.mur.at == erle.mur.at ' in testSecurity/testUnknownIssuer.js
  • bug 795398 - Mozmill test to check that MD5 hash signatures are no longer accepted
    • needs CA certificate

• Working on:
  • Mozmill-ci - Log request/response bodies of update requests
    • trying with EnvInject plugin
  • bug 860664 - controller.waitThenClick() does not correctly wait for the element
  • bug 860655 - elementslib.ID() fails to grab the element - "could not find element ID: id"
  • bug 732353 - Disable all Discovery Pane tests due to unpredictable web dependencies
  • bug 857551 - Create HTML page containing all the elements necessary for L10N testing of the context menu access keys
  • bug 671476 - Mozmill Endurance test for loading a SWF video via URL
  • bug 671479 - Mozmill Endurance test for loading a SWF video via iframe
  • bug 860382 - Test failure "Search engines drop down has been closed" in testSearchSuggestions.js
  • bug 860659 - elementslib.Lookup() fails with 'Expression \"id(\"main-window\")\" returned null. Anonymous == false"'
  • bug 791634 - Make waitFor() method available in the assertions module (also for soft assertions)
  • bug 763461 - Mozmill tests for cut-off elements should not report failure for scrollable direction
  • bug 763470 - False positive test for cut-off elements for advanced pane tabs
  • bug 764640 - Mozmill returns PASS/SKIP/FAIL == 0 for test_assertions.js
  • bug 860670 - mozmill.utils.getWindowByType fails for type: Addons:Blocklist - Fails with "Window has been found."
  • bug 860662 - controller.select() fails to select the option by value

Mobile (Robocop)

• Progress on Robocop Automation
• Bugs with work in progress:
  • bug 820859 - Robocop: Add test for 'Clear Private Data'
  • bug 830834 - Robocop: Add test for 'Show Search Suggestions' feature
  • bug 846340 - Robocop: Add test for 'Clear Site Settings'
  • bug 836341 - Robocop: Add test for "Clear download history"
  • bug 838596 - Robocop: Add test for 'Master Password' feature
  • bug 846296 - Robocop: Add test for 'Add Search Engine' feature
  • bug 833832 - Robocop: Add test for Import from Android feature
  • bug 855978 - Robocop: upgrade to robotium-solo-4.0.jar
• Bugs awaiting review:
  • bug 864280 - Robocop: Cleanup of BaseTest.java.in
• Intermittent fails we investigated:
  • bug 836818 - Intermittent testOverscroll | Checking gecko didn't draw unnecessarily - got true, expected false
  • bug 851861 - Intermittent testOverscroll, testPanCorrectness, testAxisLocking, testLoad, testFlingCorrectness | Pixel at 100, 0 - Color rgba(0,0,0,255) not close enough to expected rgb(32,100,0)
  • bug 854043 - Intermittent testWebContentContextMenu | Verify that the download started - The download started
• New issues found during investigations
  • bug 862695 - Robocop testBrowserProvider test restarts at some point locally and has a lot of database access errors
• Notes/Issues/Questions:
  • We still don't have an update on the robocop build issue - bug 855306 - Robocop build does not re-build tests - still no update from :joey - we are still working with older revisions or rebuilding everything
  • We are still seeing issues with the "Private data cleared" toaster notification not being displayed - bug 837274. The "Clear private data" tests on tryserver seem to constantly fail.
  • We are still trying to reproduce intermittent issues and to find out why they are happening

Personal Status

For the personal status please check the weekly status updates:

• 130422 - 130429

Meeting Notes

The meeting notes for this meeting can be read in the following etherpad:

• http://etherpad.mozilla.com:9000/automation-developer-meeting-20130422

Action items

• [CARRY OVER] Henrik - Update the goals page
• [CARRY OVER] Anthony: Ensure any Metro tests are suitably tagged with in-testsuite?
• [NEW] Andreea: File bug about the broken elementslib handling with elements