Default to Compatible - Detection and Mitigation

Gathering data

The Add-ons Team will be informed about compatibility problems with pre-release versions of Firefox through 3 channels:

• Reports submitted from the Add-on Compatibility Reporter.
• Bugs filed in the Extension Compatibility component in Bugzilla.
• SUMO reports.

Triaging data

• Add-on Compatibility Reporter: AMO will have a dashboard that shows compatibility reports sorted by report count and usage stats. The dashboard should also be filtered by Firefox version.
• Bugzilla: all compatibility breakage reports that correspond to pre-release Firefox versions should have [dtc] set in the whiteboard. Any communications suggesting to file bugs should use this link.
• SUMO: this hasn't been fully fleshed out, but someone from the SUMO team perhaps could forward this data to us.

Taking action

Add-ons that are repeatedly reported will be tested by the Add-ons Team to verify the breakage. If the problem is hard to reproduce or the workload becomes significant, the QA Team will be asked to help.

When reproduced, the compatibility bug will be qualified to determine the action to take:

• Minor bug: the add-on's main functions work acceptably.
  • The add-on developer will be contacted.
  • Bugs are resolved as WONTFIX noting that the developer might fix it in the future.
• Major bug: the add-on is significantly broken.
  • The add-on developer will be contacted.
  • The add-on version will be added to the blacklist, except if the problem is detected on Nightlies or if this is a known Firefox breakage with a short-term solution plan.
  • Evaluate if the add-on would benefit from using the strict compatibility flag in install.rdf. If so, encourage the developer to do so.
  • Bugs are resolved as FIXED when the bug is fixed or the version is blacklisted.
• Critical bug: the compatibility bug breaks Firefox in some way.
  • The add-on developer will be contacted.
  • The add-on version will be added to the blacklist.
  • Evaluate if the add-on would benefit from using the strict compatibility flag in install.rdf. If so, encourage the developer to do so.
  • Bugs are resolved as FIXED when the version is blacklisted.
• Crashes.
  • Same as critical bugs, but the regular crash procedure should also be followed, with the possibility of blocklisting.

Note: most blocks that apply for Firefox also apply for SeaMonkey, so the SM team should be notified in order to verify that a similar block should be done on their side.

Reporting

The Add-ons Team will produce a weekly report on add-on incompatibility that will be presented in the Product Planning Meeting.

Limitations

Due to the sheer amount of add-ons in the wild and their high usage, it is almost impossible that this effort will catch all compatibility problems that occur. Add-ons with little usage but major problems could go unnoticed. The same is possibly true for add-ons with high usage that are distributed in intranets or through third parties (though the majority have binary XPCOM and don't qualify).

It is worth noting that we already have these problems with add-ons that set unrealistic compatibility ranges or are upgraded with little testing.