Automated Test Coverage
Contents
Project Description
This project included those activities designed to convert, fix, automate, and ensure complete coverage for testing. It also included process improvement work so that test failures result in code back-out. Parts of this project had dependencies on the Marionette Improvements, Kitkat Emulator Improvements and Raptor Activation on Treeholder projects. Work on the project ended with the sunset of Firefox OS. The rest of the wiki remains for archival purposes.
Requirements:
* Convert integration tests from GIP to GIJ * Fix intermittently failing tests * Fix tests that work on mulet but not on device * Create a Test Matrix to define which platform the tests are going to run on (device, mulet, emulator) * Make sure every single commit in gecko/gaia triggers automated tests * Automate MozTrap scenarios using GIJ when feasible * Actively maintain working Mulet tests with reporting on Treeherder * Ensure code coverage
Automated Test Coverage
Milestone | Due Date | Status | Notes |
---|---|---|---|
Convert integration tests from GIP to GIJ | 12/31/15 | AT RISK | All but 14 have been converted. Engineers out on PTO. |
Intermittently failing tests fixed | 01/31/15 | CHALLENGED | Engineers are disabling the tests that fail intermittently and will fix them before re-enabling. Need more engineering help with this. |
Tests that work on mulet work on device | 01/31/15 | CHALLENGED | It's being reported that some of the flagged tests actually work on device. Need to reassess all. Challenged due to resources not being assigned. |
Test Matrix created that defines platform(s) the tests are going to run on | 12/31/15 | AT RISK | Draft of matrix completed but there are still some tests that need to be evaluated for appropriate testing environment. Currently out for review with eng. leadership. Won't get this done until January. |
Automation of tests according to Test Matrix | 05/31/16 | ON TARGET | Draft of matrix completed but there are still some tests that need to be evaluated for appropriate testing environment. Won't get this done until January. |
Every single commit in gecko/gaia triggers automated tests | 05/31/16 | ON TARGET | There's still work that needs to be identified and scheduled. |
MozTrap scenarios automated using GIJ when feasible | 05/31/16 | ON TARGET | |
Working Mulet tests actively maintained with reporting on Treeherder | 05/31/16 | ON TARGET | |
Code Coverage Tool developed | 12/31/15 | AT RISK | At risk for making targeted date. Originally targeted for end of Q4, but assistance at Mozlando didn't happen, so this will be delivered in Q1. New due date TBD. |
Status Key
Color | Status | Key |
---|---|---|
On Target | The project or deliverable is expected to meet its due date. | |
Challenged | The project or deliverable is facing an issue that might cause it to miss its due date, but a “get well” plan has been developed to get it back on track. | |
At Risk or Late | The project or deliverable is blocked or facing an issue that might cause it to miss its due date, and there’s no “get well” plan to get it back on track, or it is already late. | |
Done | The project or deliverable has been completed. | |
On Hold | The project or deliverable has been placed on hold. |
Program Timeline
Project Scope
Bugs for GIP to GIJ conversions
ID | Summary | Status | Target milestone | Resolution | Assigned to | Depends on | Blocks | Whiteboard | Feature-b2g |
---|---|---|---|---|---|---|---|---|---|
1218401 | [META] Moving test scripts from GIP to GIJ | RESOLVED | --- | WONTFIX | 1219620, 1219627, 1219628, 1219629, 1219630, 1219631, 1219632, 1219633, 1219634, 1219635, 1219636, 1219640, 1219641, 1219643, 1219644, 1219645, 1219649, 1219650, 1219651, 1219653, 1219654, 1219655, 1219656, 1219657, 1219658, 1219659, 1219660, 1219661, 1219662, 1219663, 1219665, 1219666, 1219667, 1219668, 1219670, 1219671, 1219674, 1219675, 1219676, 1219677, 1219678, 1219679, 1219680, 1219681, 1219682, 1219683, 1219684, 1219685, 1219686, 1219687, 1219688, 1219689, 1219690, 1219691, 1219692, 1219693, 1219694, 1219695, 1219696, 1219697, 1219698, 1219699, 1219700, 1219701, 1219702, 1219703, 1219704, 1219705, 1219706, 1222975, 1232280, 1232281 | [gip-to-gij] | No feature-b2g | ||
1219620 | Implement *test_browser_bookmark.py* as an integration test in JavaScript | RESOLVED | --- | WORKSFORME | 1218401, 1230099 | [gip-to-gij] | No feature-b2g | ||
1219627 | Implement *test_browser_play_video.py* as an integration test in JavaScript | RESOLVED | 2.6 S3 - 12/18 | FIXED | Alberto Pastor [:albertopq] | 1218401 | [gip-to-gij][systemsfe] | No feature-b2g | |
1219628 | Implement *test_browser_save_image.py* as an integration test in JavaScript | RESOLVED | --- | FIXED | Alberto Pastor [:albertopq] | 1218401 | [gip-to-gij][systemsfe] | No feature-b2g | |
1219629 | Implement *test_browser_clear_browsing_history.py* as an integration test in JavaScript | RESOLVED | 2.6 S3 - 12/18 | FIXED | Guillaume Marty [:gmarty] | 1218401 | [gip-to-gij] | No feature-b2g | |
1219630 | Implement *test_browser_search.py* as an integration test in JavaScript | RESOLVED | 2.6 S3 - 12/18 | FIXED | Alberto Pastor [:albertopq] | 1218401 | [gip-to-gij][systemsfe] | No feature-b2g | |
1219631 | Implement *test_browser_share_link.py* as an integration test in JavaScript | RESOLVED | --- | FIXED | Alberto Pastor [:albertopq] | 1218401, 1230099 | [gip-to-gij] | No feature-b2g | |
1219632 | Implement *test_browser_navigation.py* as an integration test in JavaScript | RESOLVED | --- | FIXED | Alberto Pastor [:albertopq] | 1218401 | [gip-to-gij] | No feature-b2g | |
1219633 | Implement *test_calendar_flick_through_months.py* as an integration test in JavaScript | RESOLVED | --- | WONTFIX | Justin Potts [:justinpotts] | 1218401 | [gip-to-gij] | No feature-b2g | |
1219634 | Implement *test_cards_view_with_two_apps.py* as an integration test in JavaScript | RESOLVED | --- | WORKSFORME | 1218401, 1228972 | [gip-to-gij] | No feature-b2g | ||
1219635 | Implement *test_cards_view_kill_apps_with_two_apps.py* as an integration test in JavaScript | RESOLVED | --- | FIXED | Etienne Segonzac (:etienne) | 1218401, 1228972 | [gip-to-gij] | No feature-b2g | |
1219636 | Implement *test_import_contacts_from_sdcard.py* as an integration test in JavaScript | RESOLVED | --- | FIXED | Francisco Jordano [:arcturus] [:francisco] | 1218401 | [gip-to-gij] | No feature-b2g | |
1219640 | Implement *test_default_contact_image.py* as an integration test in JavaScript | RESOLVED | --- | FIXED | Francisco Jordano [:arcturus] [:francisco] | 1218401, 1230099 | [gip-to-gij] | No feature-b2g | |
1219641 | Implement *test_delete_all_contacts.py* as an integration test in JavaScript | RESOLVED | --- | FIXED | Francisco Jordano [:arcturus] [:francisco] | 1218401, 1219643, 1230099 | [gip-to-gij] | No feature-b2g | |
1219643 | Implement *test_delete_contact.py* as an integration test in JavaScript | RESOLVED | --- | FIXED | Francisco Jordano [:arcturus] [:francisco] | 1219641 | 1218401 | [gip-to-gij] | No feature-b2g |
1219644 | Implement *test_sort_contacts.py* as an integration test in JavaScript | RESOLVED | --- | FIXED | Francisco Jordano [:arcturus] [:francisco] | 1218401 | [gip-to-gij] | No feature-b2g | |
1219645 | Implement *test_export_contacts_to_sdcard.py* as an integration test in JavaScript | RESOLVED | --- | FIXED | Francisco Jordano [:arcturus] [:francisco] | 1218401 | [gip-to-gij] | No feature-b2g | |
1219649 | Implement *test_cost_control_reset_wifi.py* as an integration test in JavaScript | RESOLVED | --- | WONTFIX | 1218401 | [gip-to-gij] | No feature-b2g | ||
1219650 | Implement *test_call_log_groups.py* as an integration test in JavaScript | RESOLVED | --- | WONTFIX | 1232680 | 1218401 | [gip-to-gij] | No feature-b2g | |
1219651 | Implement *test_delete_call_log.py* as an integration test in JavaScript | RESOLVED | --- | WONTFIX | 1232680 | 1218401 | [gip-to-gij] | No feature-b2g | |
1219653 | Implement *test_dialer_add_contact.py* as an integration test in JavaScript | RESOLVED | --- | FIXED | Isabel Rios[:isabel_rios] | 1218401 | [gip-to-gij] | No feature-b2g | |
1219654 | Implement *test_dialer_find_contact.py* as an integration test in JavaScript | RESOLVED | --- | WONTFIX | 1218401 | [gip-to-gij] | No feature-b2g | ||
1219655 | Implement *test_ftu_status_bar.py* as an integration test in JavaScript | RESOLVED | 2.6 S1 - 11/20 | FIXED | Sam Foster [:sfoster] (he/him) | 1218401, 1228972 | [gip-to-gij][systemsfe] | No feature-b2g | |
1219656 | Implement *test_gallery_cancel_delete_multiple_images.py* as an integration test in JavaScript | RESOLVED | --- | DUPLICATE | Punam Dahiya [:pdahiya] | 1219657 | 1213991, 1218401 | [gip-to-gij] | No feature-b2g |
1219657 | Implement *test_gallery_delete_multiple_images.py* as an integration test in JavaScript | RESOLVED | --- | FIXED | Punam Dahiya [:pdahiya] | 1213991, 1218401, 1219656, 1223040 | [gip-to-gij] | No feature-b2g | |
1219658 | Implement *test_gallery_empty.py* as an integration test in JavaScript | RESOLVED | --- | WONTFIX | 1218401 | [gip-to-gij] | No feature-b2g | ||
1219659 | Implement *test_gallery_handle_invalid_file.py* as an integration test in JavaScript | RESOLVED | --- | WONTFIX | 1218401 | [gip-to-gij] | No feature-b2g | ||
1219660 | Implement *test_gallery_handle_valid_image_file.py* as an integration test in JavaScript | RESOLVED | --- | WONTFIX | 1218401 | [gip-to-gij] | No feature-b2g | ||
1219661 | Implement *test_gallery_share_picture.py* as an integration test in JavaScript | RESOLVED | --- | FIXED | Punam Dahiya [:pdahiya] | 1213991, 1218401, 1230099 | [gip-to-gij] | No feature-b2g | |
1219662 | Implement *test_gallery_view.py* as an integration test in JavaScript | RESOLVED | --- | FIXED | Punam Dahiya [:pdahiya] | 1213991, 1218401, 1233453 | [gip-to-gij] | No feature-b2g | |
1219663 | Implement *test_homescreen_change_wallpaper.py* as an integration test in JavaScript | RESOLVED | --- | WONTFIX | 1218401 | [gip-to-gij] | No feature-b2g | ||
1219665 | Implement *test_homescreen_column_layout.py* as an integration test in JavaScript | RESOLVED | --- | FIXED | Chris Lord [:cwiiis] | 1218401, 1230099 | [gip-to-gij] | No feature-b2g | |
1219666 | Implement *test_homescreen_delete_app_packaged.py* as an integration test in JavaScript | RESOLVED | --- | FIXED | Chris Lord [:cwiiis] | 1218401, 1230099 | [gip-to-gij] | No feature-b2g | |
1219667 | Implement *test_homescreen_divider.py* as an integration test in JavaScript | RESOLVED | --- | WONTFIX | 1218401 | [gip-to-gij] | No feature-b2g | ||
1219668 | Implement *test_homescreen_launch_app.py* as an integration test in JavaScript | RESOLVED | --- | WORKSFORME | 1218401, 1228972 | [gip-to-gij] | No feature-b2g | ||
1219670 | Implement *test_homescreen_launch_app_packaged.py* as an integration test in JavaScript | RESOLVED | --- | WORKSFORME | 1218401, 1228972 | [gip-to-gij] | No feature-b2g | ||
1219671 | Implement *test_homescreen_move_app.py* as an integration test in JavaScript | RESOLVED | --- | WORKSFORME | 1218401, 1228972 | [gip-to-gij] | No feature-b2g | ||
1219674 | Implement *test_email_keyboard.py* as an integration test in JavaScript | RESOLVED | --- | FIXED | Ray Lin[:ralin] | 1218401, 1228972 | [gip-to-gij] | No feature-b2g | |
1219675 | Implement *test_keyboard.py* as an integration test in JavaScript | RESOLVED | --- | FIXED | Ray Lin[:ralin] | 1218401, 1230099 | [gip-to-gij] | No feature-b2g | |
1219676 | Implement *test_keyboard_bug_1073870.py* as an integration test in JavaScript | RESOLVED | --- | FIXED | Ray Lin[:ralin] | 1218401, 1228972 | [gip-to-gij] | No feature-b2g | |
1219677 | Implement *test_keyboard_predictive_key.py* as an integration test in JavaScript | RESOLVED | --- | FIXED | Ray Lin[:ralin] | 1218401, 1228972 | [gip-to-gij] | No feature-b2g | |
1219678 | Implement *test_number_keyboard.py* as an integration test in JavaScript | RESOLVED | --- | FIXED | Ray Lin[:ralin] | 1218401, 1228972 | [gip-to-gij] | No feature-b2g | |
1219679 | Implement *test_lockscreen_time_check.py* as an integration test in JavaScript | RESOLVED | --- | WONTFIX | 1218401 | [gip-to-gij] | No feature-b2g | ||
1219680 | Implement *test_lockscreen_unlock_to_homescreen.py* as an integration test in JavaScript | RESOLVED | --- | FIXED | Greg Weng [:snowmantw][:gweng][:λ] | 1218401, 1233453 | [gip-to-gij] | No feature-b2g | |
1219681 | Implement *test_lockscreen_unlock_to_homescreen_with_passcode.py* as an integration test in JavaScript | RESOLVED | --- | FIXED | Greg Weng [:snowmantw][:gweng][:λ] | 1218401, 1228972 | [gip-to-gij] | No feature-b2g | |
1219682 | Implement *test_lockscreen_wake_with_notification.py* as an integration test in JavaScript | RESOLVED | --- | WONTFIX | 1218401 | [gip-to-gij] | No feature-b2g | ||
1219683 | Implement *test_music_{album,artist,song}_mp3.py* as an integration test in JavaScript | RESOLVED | 2.6 S1 - 11/20 | FIXED | Hubert Figuiere [:hub] | 1212694, 1218401 | [gip-to-gij] | No feature-b2g | |
1219684 | Implement *test_music_artist_mp3.py* as an integration test in JavaScript | RESOLVED | --- | DUPLICATE | 1212694, 1218401 | [gip-to-gij] | No feature-b2g | ||
1219685 | Implement *test_music_change_rating.py* as an integration test in JavaScript | RESOLVED | --- | DUPLICATE | 1218401, 1228972 | [gip-to-gij] | No feature-b2g | ||
1219686 | Implement *test_music_share_ringtone.py* as an integration test in JavaScript | RESOLVED | --- | FIXED | 1218401, 1228972 | [gip-to-gij] | No feature-b2g | ||
1219687 | Implement *test_music_songs_3gp.py* as an integration test in JavaScript | RESOLVED | --- | DUPLICATE | 1212694, 1218401 | [gip-to-gij] | No feature-b2g | ||
1219688 | Implement *test_settings_change_keyboard_language.py* as an integration test in JavaScript | RESOLVED | --- | FIXED | Scott Wu [:scottwu] | 1218401, 1228972 | [gip-to-gij] | No feature-b2g | |
1219689 | Implement *test_settings_change_language_rtl.py* as an integration test in JavaScript | RESOLVED | --- | INVALID | Scott Wu [:scottwu] | 1218401, 1228972 | [gip-to-gij] | No feature-b2g | |
1219690 | Implement *test_settings_change_ring_tone.py* as an integration test in JavaScript | RESOLVED | --- | INVALID | Scott Wu [:scottwu] | 1218401, 1228972 | [gip-to-gij] | No feature-b2g | |
1219691 | Implement *test_settings_gps.py* as an integration test in JavaScript | RESOLVED | --- | INVALID | Scott Wu [:scottwu] | 1218401, 1228972 | [gip-to-gij] | No feature-b2g | |
1219692 | Implement *test_settings_passcode.py* as an integration test in JavaScript | RESOLVED | --- | FIXED | Scott Wu [:scottwu] | 1218401, 1228972 | [gip-to-gij] | No feature-b2g | |
1219693 | Implement *test_settings_usb_storage.py* as an integration test in JavaScript | RESOLVED | --- | FIXED | Scott Wu [:scottwu] | 1218401, 1228972 | [gip-to-gij] | No feature-b2g | |
1219694 | Implement *test_inter_app_comm.py* as an integration test in JavaScript | RESOLVED | --- | WONTFIX | 1218401, 1228972 | [gip-to-gij] | No feature-b2g | ||
1219695 | Implement *test_privileged_app_contacts_prompt.py* as an integration test in JavaScript | RESOLVED | 2.6 S2 - 12/4 | FIXED | Michael Henretty [:mikehenrty][:mhenretty] | 1014410, 1227011 | 1218401, 1230099 | [gip-to-gij][systemsfe] | No feature-b2g |
1219696 | Implement *test_privileged_app_device_music_prompt.py* as an integration test in JavaScript | RESOLVED | 2.6 S1 - 11/20 | DUPLICATE | Michael Henretty [:mikehenrty][:mhenretty] | 1218401, 1230099 | [gip-to-gij][systemsfe] | No feature-b2g | |
1219697 | Implement *test_privileged_app_device_picture_prompt.py* as an integration test in JavaScript | RESOLVED | 2.6 S2 - 12/4 | DUPLICATE | Michael Henretty [:mikehenrty][:mhenretty] | 1218401, 1230099 | [gip-to-gij][systemsfe] | No feature-b2g | |
1219698 | Implement *test_privileged_app_device_sdcard_prompt.py* as an integration test in JavaScript | RESOLVED | 2.6 S3 - 12/18 | FIXED | Michael Henretty [:mikehenrty][:mhenretty] | 1218401 | [gip-to-gij][systemsfe] | No feature-b2g | |
1219699 | Implement *test_privileged_app_device_video_prompt.py* as an integration test in JavaScript | RESOLVED | 2.6 S2 - 12/4 | DUPLICATE | Michael Henretty [:mikehenrty][:mhenretty] | 1218401, 1230099 | [gip-to-gij][systemsfe] | No feature-b2g | |
1219700 | Implement *test_privileged_app_geolocation_prompt.py* as an integration test in JavaScript | RESOLVED | --- | WORKSFORME | 1218401, 1230099 | [gip-to-gij] | No feature-b2g | ||
1219701 | Implement *test_quick_settings.py* as an integration test in JavaScript | RESOLVED | 2.6 S1 - 11/20 | FIXED | Michael Henretty [:mikehenrty][:mhenretty] | 1218401, 1228972 | [gip-to-gij][systemsfe] | No feature-b2g | |
1219702 | Implement *test_system_message.py* as an integration test in JavaScript | RESOLVED | --- | FIXED | 1218401, 1228972 | [gip-to-gij] | No feature-b2g | ||
1219703 | Implement *test_system_message_pending.py* as an integration test in JavaScript | RESOLVED | --- | FIXED | 1218401, 1228972 | [gip-to-gij] | No feature-b2g | ||
1219704 | Implement *test_play_ogg_video.py* as an integration test in JavaScript | RESOLVED | --- | FIXED | Russ Nicoletti [:russn] | 1218401, 1233453 | [gip-to-gij] | No feature-b2g | |
1219705 | Implement *test_play_ogv_video.py* as an integration test in JavaScript | RESOLVED | --- | DUPLICATE | Russ Nicoletti [:russn] | 1218401, 1233453 | [gip-to-gij] | No feature-b2g | |
1219706 | Implement *test_play_webm_video.py* as an integration test in JavaScript | RESOLVED | --- | DUPLICATE | Russ Nicoletti [:russn] | 1218401, 1233453 | [gip-to-gij] | No feature-b2g | |
1222975 | Delete the Python tests that are already running in Gij and not testing more than the integration between Gecko and Gaia | RESOLVED | --- | WONTFIX | 1223012, 1223055, 1228972, 1230099, 1233453 | 1218401 | [gip-to-gij] | No feature-b2g | |
1232280 | Implement test_home_button.py and test_home_button_in_edit_mode.py as an Integration test in JavaScript | RESOLVED | 2.6 S3 - 12/18 | FIXED | Guillaume Marty [:gmarty] | 1195364 | 1218401 | [gip-to-gij][systemsfe] | No feature-b2g |
1232281 | Implement test_home_button.py (in Lockscreen) as an Integration test in JavaScript | RESOLVED | --- | FIXED | Greg Weng [:snowmantw][:gweng][:λ] | 1195364 | 1218401, 1233453 | [gip-to-gij] | No feature-b2g |
73 Total; 0 Open (0%); 73 Resolved (100%); 0 Verified (0%);
Bugs for Tests that Run on Mulet but Fail on Device
10 Total; 0 Open (0%); 10 Resolved (100%); 0 Verified (0%);
Bugs for Tests that Fail Intermittently
8 Total; 0 Open (0%); 8 Resolved (100%); 0 Verified (0%);
Bugs for Code Coverage Tool
ID | Summary | Status | Target milestone | Resolution | Assigned to | Depends on | Blocks | Whiteboard | Feature-b2g |
---|---|---|---|---|---|---|---|---|---|
1176880 | Add Code Coverage interface to the Debugger. | RESOLVED | mozilla43 | FIXED | 1322663, 1190454, 1205880, 1226896, 1252111 | 1189360, 1179996, 1200154 | [CC] [CI] | No feature-b2g | |
1179996 | [fxos-automation] Code coverage tools for JavaScript tests | RESOLVED | --- | INVALID | Aus Lacroix [:aus] | 1176880, 1179997, 1179999, 1180003, 1180006, 1180007 | [CC] [CI] | No feature-b2g | |
1179997 | [fxos-automation][aus-q3] Turn on JavaScript engine code coverage support for B2G-Desktop builds (Linux x64 Opt Only) | RESOLVED | --- | INVALID | 1179996 | [CC] [CI] | No feature-b2g | ||
1179999 | [fxos-automation][aus-q3] Add code coverage log parsing support to Unit/Integration test tooling | RESOLVED | --- | INCOMPLETE | 1179996 | [CC] [CI] | No feature-b2g | ||
1180003 | [fxos-automation][aus-q3] Add code coverage regression check | RESOLVED | --- | INCOMPLETE | 1179996 | [CC] [CI] | No feature-b2g | ||
1180006 | [fxos-automation][aus-q3] Turn on Code Coverage for Gaia Unit Tests | RESOLVED | --- | INCOMPLETE | 1179996 | [CC] [CI] | No feature-b2g | ||
1180007 | [fxos-automation][aus-q3] Turn on Code Coverage for Gaia Integration Tests | RESOLVED | --- | INCOMPLETE | 1179996 | [CC] [CI] | No feature-b2g |
7 Total; 0 Open (0%); 7 Resolved (100%); 0 Verified (0%);
Dependency Tracking
Detailed Program Plan
Action Item | Engineering Owner | QA Owner | UX Owner | Bugzilla ID | Planned Done | Actual Done |
---|---|---|---|---|---|---|
Program Stakeholders
Role | Name | IRC |
---|---|---|
EPM | ||
EM | ||
PM | ||
TL | ||
UX | ||
QA |
- EPM = Engineering Program Manager
- EM = Engineering Manager
- PM = Product Manager
- TL = Tech Lead
- UX = User Experience
- QA = Quality Assurance