How to setup JenkinsHow to setup gaia-ui-testsWhere to Get Report
- We have CI set up in Taipei and it pushes reports on Github.IO everyday.
How to Read Build Report
- Test Result Trend of a Project (e.g.: B2G.v2.2.pvt.flame.gaiatest.smoketest)
- In project view (Jenkins>>Project), we can see a figure like below and quickly find out if the failed test number increases.
- Blue: number of passed cases, Red: number of fail cases, Yellow: number of skipped cases
- Test Result Overview of a Build
- In Jenikns>>Project>>Build>>Test Result, we can quickly know which cases failed and if it is failed last last time.
- If Fail=1 && diff="", it is usually a known issue.
- If Fail=1 && diff=+1, we need to figure out the reason: if it resulted from a bug.
- If Pass=1 && diff=+1, the issue in last build might be resolved.
- Please make sure the status of each diff value when a test script is disabled/enabled.
- Test Result Details
- We can find index.html at Jenkins>>Project>>Build>>index.html, just like the red rectangle in below figure.
Where to click index.html
- It includes test result details, especially build information, logs and screenshot of failed cases.
- From log, we usually can determine if it is a common non-bug error. Otherwise, we need to rerun test script locally or rerun test manually.
- How to Verify a Failed Case
- If you see "error" or "failure" of a test case, try to search related information on Bugzilla or previous reports, it might be existed in other branch.
- Try to run the "error" or "failure" cases by your machine or manually, sometimes you can isolate the issue is existed in scripts or is an OS problem.
- For both cases, we need to file bugzilla bugs to follow up
- Try to run a single case with "--restart" or "--repeat=10" to verify.
How to write daily report
- Send daily report to this watch list.
E.G. 02/09/2015 Flame KK v188-1 Gaia-ui Automation v2.1 Mozilla RIL Full Flash Build 38/38 smoketests, 90/90 non-smoketests
- Failed cases & failed reason
- List all failures and expected failures cases for both Smoketest and Non-smoketest. Please also write down the result of failure survey.
Smoketests:
Test failures - 0 replicable manually:
[Not reproduced] test_sms_with_attachments.py - Cannot reproduce locally.
[Not reproduced] test_settings_wifi.py - Cannot reproduce locally.
Expected failures / disabled tests:
[Disabled] test_setup_and_send_active_sync_email.py - Bug 1028192 - Problems with ActiveSync accounts
[Disabled] test_receive_active_sync_email.py - Bug 1028192 - Problems with ActiveSync accounts
[Disabled] test_cost_control_data_alert_mobile.py - Bug 1061698 - Need ability to work in the shadow root/shadow DOM
[Disabled] test_call_log_all_calls.py - Bug 1096385 - Investigate failure in test_call_log_all_calls.py, call log is displayed erroneously
Non-smoketests:
Test failures - 0 replicable manually:
[Not reproduced] test_import_gmail_no_network.py - Cannot reproduce locally.
[Not reproduced manually] test_call_log_groups.py - Bug 1117804 - [v2.2] Investigate test_call_log_groups failure
[Not reproduced manually] test_dialer_set_up_conference_call.py -Bug 1117813 - Make PlivoUtil use a valid 'answer_url' with a valid Plivo XML
Expected failures / disabled tests:
[Disabled] test_ftu_skip_tour_desktopb2g.py - Test runs only on desktop
[Disabled] test_settings_sim_manager.py - Test requires device with 2 sims
[Disabled] test_fmradio_find_stations.py - Bug 1000863 - Investigate test_fmradio_find_stations.py failure
Build under test:
Device firmware (base) L1TC000118D0
Device firmware (date) 28 Jan 2015 16:59:20
Device firmware (incremental) eng.cltbld.20150128.035910
Device firmware (release) 4.4.2
Device identifier flame
Gaia date 28 Jan 2015 08:05:23
Gaia revision cd42b034fd28
Gecko build 20150128002506
Gecko revision d824c65a6a2b
Gecko version 37.0a2
Summary:
Smoketests:
45 tests ran in 3697 seconds.
43 passed, 132 skipped, 0 failed, 2 errors.
0 expected failures, 0 unexpected passes.
Non-smoketests:
135 tests ran in 9441 seconds.
132 passed, 55 skipped, 1 failed, 2 errors.
0 expected failures, 0 unexpected passes.
- Attachments
- Please rename 2 index.html files to smoke.html and non-smoke.html, then attach them in mail.
- Where to get index.html?
- You can download both index.html from our github: Gaiatest-Reports (please find the correct date to get them)
- Or, click the job number at CI then get the index.html.
-
-
Others
- If 2 error records for one test case (It seems race condition with the logging, Bug 1050170 tracks this issue), please do not double counting the same fail case.
Tips
- To hide the passed cases to see the failed/error cases more clearly, do the following:
- Press F12 to open the Firefox Developer Tool > Console
- Run the following code:
[].forEach.call(document.getElementsByClassName('passed'), function(elem){elem.style.display="none"})