Firefox OS/DeviceQA/Gaia-UI-Tests-On-Jenkins

From MozillaWiki
Jump to: navigation, search

How to setup Jenkins

How to setup gaia-ui-tests

Where to Get Report

  • We have CI set up in Taipei and it pushes reports on Github.IO everyday.

How to Read Build Report

  • Test Result Trend of a Project (e.g.: B2G.v2.2.pvt.flame.gaiatest.smoketest)
    • In project view (Jenkins>>Project), we can see a figure like below and quickly find out if the failed test number increases.
    • Blue: number of passed cases, Red: number of fail cases, Yellow: number of skipped cases
Test Result Trend
  • Test Result Overview of a Build
    • In Jenikns>>Project>>Build>>Test Result, we can quickly know which cases failed and if it is failed last last time.
      • If Fail=1 && diff="", it is usually a known issue.
      • If Fail=1 && diff=+1, we need to figure out the reason: if it resulted from a bug.
      • If Pass=1 && diff=+1, the issue in last build might be resolved.
      • Please make sure the status of each diff value when a test script is disabled/enabled.
Test Result Overview
  • Test Result Details
    • We can find index.html at Jenkins>>Project>>Build>>index.html, just like the red rectangle in below figure.
Where to click index.html
  • It includes test result details, especially build information, logs and screenshot of failed cases.
    • From log, we usually can determine if it is a common non-bug error. Otherwise, we need to rerun test script locally or rerun test manually.
Failed Case Details


  • How to Verify a Failed Case
    • Common non-bug errors
      • Timeout
Unable to locate element
  • Unable to locate element
Unable to locate element
  • ...?

How to write daily report

DailyReportSample.png
  • Subject
 E.G. 02/09/2015 Flame KK v188-1 Gaia-ui Automation v2.1 Mozilla RIL Full Flash Build 38/38 smoketests, 90/90 non-smoketests
  • Failed cases & failed reason
    • List all failures and expected failures cases for both Smoketest and Non-smoketest. Please also write down the result of failure survey.
 Smoketests:     
   Test failures - 0 replicable manually:
     [Not reproduced] test_sms_with_attachments.py - Cannot reproduce locally.
     [Not reproduced] test_settings_wifi.py - Cannot reproduce locally.
   Expected failures / disabled tests:
     [Disabled] test_setup_and_send_active_sync_email.py - Bug 1028192 - Problems with ActiveSync accounts
     [Disabled] test_receive_active_sync_email.py - Bug 1028192 - Problems with ActiveSync accounts
     [Disabled] test_cost_control_data_alert_mobile.py - Bug 1061698 - Need ability to work in the shadow root/shadow DOM
     [Disabled] test_call_log_all_calls.py - Bug 1096385 - Investigate failure in test_call_log_all_calls.py, call log is displayed erroneously
 Non-smoketests:    
   Test failures - 0 replicable manually:
     [Not reproduced] test_import_gmail_no_network.py - Cannot reproduce locally.
     [Not reproduced manually] test_call_log_groups.py - Bug 1117804 - [v2.2] Investigate test_call_log_groups failure
     [Not reproduced manually] test_dialer_set_up_conference_call.py -Bug 1117813 - Make PlivoUtil use a valid 'answer_url' with a valid   Plivo XML
   Expected failures / disabled tests:
     [Disabled] test_ftu_skip_tour_desktopb2g.py - Test runs only on desktop
     [Disabled] test_settings_sim_manager.py - Test requires device with 2 sims
     [Disabled] test_fmradio_find_stations.py - Bug 1000863 - Investigate test_fmradio_find_stations.py failure
  • Build info
 Build under test:
   Device firmware (base)	L1TC000118D0
   Device firmware (date)	28 Jan 2015 16:59:20
   Device firmware (incremental)	eng.cltbld.20150128.035910
   Device firmware (release)	4.4.2
   Device identifier	flame
   Gaia date	28 Jan 2015 08:05:23
   Gaia revision	 cd42b034fd28 
   Gecko build	20150128002506
   Gecko revision	d824c65a6a2b 
   Gecko version	37.0a2
  • Summary
 Summary:
   Smoketests:
     45 tests ran in 3697 seconds.
     43 passed, 132 skipped, 0 failed, 2 errors.
     0 expected failures, 0 unexpected passes.
   Non-smoketests:
     135 tests ran in 9441 seconds.
     132 passed, 55 skipped, 1 failed, 2 errors.
     0 expected failures, 0 unexpected passes.
  • Attachments
    • Please rename 2 index.html files to smoke.html and non-smoke.html, then attach them in mail.
      • Where to get index.html?
        • You can download both index.html from our github: Gaiatest-Reports (please find the correct date to get them)
        • Or, click the job number at CI then get the index.html.
        • 01_JenkinsReporting.png
        • 02_JenkinsReporting.png

Others

  • If 2 error records for one test case (It seems race condition with the logging, Bug 1050170 tracks this issue), please do not double counting the same fail case.
Double counting error

Tips

  • To hide the passed cases to see the failed/error cases more clearly, do the following:
    1. Press F12 to open the Firefox Developer Tool > Console
    2. Run the following code: 
      [].forEach.call(document.getElementsByClassName('passed'), function(elem){elem.style.display="none"})
Retrieved from "https://wiki.mozilla.org/index.php?title=Firefox_OS/DeviceQA/Gaia-UI-Tests-On-Jenkins&oldid=1060207"