Electrolysis/Release Criteria/Slow Script

From MozillaWiki
Jump to: navigation, search

e10s must not cause the slow-script dialog to appear more often. A/B testing on beta should indicate that the slow script dialog appears no more often than before.

RASCI

  • Responsible: azhang
  • Accountable: bsmedberg
  • Supporting:
  • Consulted:
  • Informed: cpeterson, elan, release management

Metrics

SLOW_SCRIPT_PAGE_COUNT

Summary:

  • SLOW_SCRIPT_PAGE_COUNT made it into beta46, and we've obtained data for it in the beta46-noapz and beta46-apz experiment.
  • See the latest beta46-apz experiment analysis: SLOW_SCRIPT_PAGE_COUNT still shows slightly more often on e10s than non-e10s, but the difference is much smaller than with SLOW_SCRIPT_NOTICE_COUNT.
  • We've verified that SLOW_SCRIPT_PAGE_COUNT measures the same thing across e10s and non-e10s.

Relevant info from beta46-apz slow script analysis:

Median difference in payload/histograms/SLOW_SCRIPT_PAGE_COUNT per hour is 0.12, (0.47, 0.35). The probability of this effect being purely by chance is 0.00.

Technical details:

  • SLOW_SCRIPT_PAGE_COUNT is essentially a version of SLOW_SCRIPT_NOTICE_COUNT that can be compared between e10s and non-e10s. It counts only the first slow script notice per page, which accounts for the difference in notice behaviours.
  • There should be no differences in the slow script reporting mechanism between e10s and non-e10s; 1 call to ShowSlowScriptDialog means 1 addition to the histogram regardless of e10s setting. Manual testing supports this.
  • The timeout has changed from 2 to 10 in e10s builds (see https://hg.mozilla.org/mozilla-central/rev/63680efe6d55#l35.12 for before and https://hg.mozilla.org/mozilla-central/rev/5002e9c06365#l2.13 for after). However, the value is correct on all beta versions, from 45.0b1 to 45.0b8 - this does not affect our measurements.
  •  :azhang has verified that beta 46 releases with/without e10s both increment SLOW_SCRIPT_PAGE_COUNT and SLOW_SCRIPT_NOTICE_COUNT if and only if there is a slow script notification shown, and that they seem to have the exact same timeout in all cases.

Links:

Analysis results: SLOW_SCRIPT_PAGE_COUNT shows up slightly more often in e10s. Analysis was reviewed by: rvitillo

How slow script notice count works on non-e10s:

How slow script notice count works on e10s:

Bugs

Full Query
ID Summary Whiteboard Keywords Assigned to
1260769 [e10s] SLOW_SCRIPT_PAGE_COUNT regressed under e10s regression

1 Total; 1 Open (100%); 0 Resolved (0%); 0 Verified (0%);