This page discusses incidents, incident reporting, remediation, and communication. It gives guidance to CAs as to how Mozilla expects them to react to reported incidents such as misissuances, and what the best practices are.
An incident arises any time a CA fails to comply with an applicable requirement found in the Mozilla Root Store Policy, the CA/Browser Forum’s Baseline Requirements Forum's requirements, or in the Mozilla Root Store PolicyCCADB's requirements. As noted in section 2.4 of the Mozilla Root Store Policy, an incident can arise from certificate misissuance, delayed revocation, procedural or operational issues, or some other cause.
For the purposes of this page, a A "misissuance" is defined as any certificate issued in contravention of any applicable standard, process or document - so it could be RFC non-compliant, BR non-compliant, issued contrary to the CA's CP/CPS, or have some other flaw or problem. Researchers who report CA incidents such as misissuances are welcome to include a link to this page in their report to the CA, reminding the CA that Mozilla has the following expectations. This document is framed in terms of misissuance of certificates; it will need to be adapted as necessary for incidents of a different nature, respecting the spirit of the information requests contained therein.
Other examples Much of incidents include misconfigured OCSP responders, unthe content that previously was here has been moved to '''https://www.ccadb.org/cas/incident-revocations, and any other event affecting trust in the WebPKI which does not involve the actual contents of certificatesreport'''.
While some forms of incident may be seen Researchers who report CA incidents such as less serious than others, opinions vary on which these misissuances are. Mozilla sees all incidents as good opportunities for the CA welcome to include a link to test that page in their incident response processes are working wellreport to the CA, and so we expect a similar level reminding the CA of timeliness Mozilla's expectations for incident reporting. Sometimes our guidance is framed in terms of response and quality misissuance of reporting certificates; it will need to be adapted as necessary for all incidentsof a different nature, whatever their adjudged severityrespecting the spirit of the information requests contained in the standard incident-reporting template.
Other examples of incidents include misconfigured CRLs and OCSP responders, delayed responses, failures to properly communicate information, and any other event affecting trust in the WebPKI which does not involve the actual contents of certificates. While some forms of incident may be seen as less serious than others, opinions may vary. Mozilla sees all incidents as good opportunities for CA operators to confirm that their incident response processes are working well, and so we expect a similar level of timeliness of response and quality of reporting for all incidents, whatever their adjudged severity. To be clear on , the status of this document: this is a incident report template and process outline best practices document, not an official policy, and does not use normative language. Therefore, failure to follow one or more of the recommendations here alone is not by itself sanctionable. However, failure to do so without good reason may affect Mozilla's general opinion of the CA. Our confidence in a CA is in part affected by the number and severity of incidents, but it is also significantly affected by the speed and quality of incident response.
= Immediate Actions =
In misissuance cases, a CA should almost always immediately cease issuance from the affected part of your its PKI. In situations not involving misissuance, there also may be processes that need to be stopped until you have the CA has diagnosed the source of the problem.
Once the problem is diagnosed, you may restart if the process even if a full fix CA is not rolled out, if you are able to put in place temporary or manual procedures to prevent the problem from re-occurring, it may restart the process even if a full fix is not rolled out. You CAs should not restart the process affected processes until you they are confident that the problem will not re-occur.
= Revocation =
It is normal practice for CAs to revoke misissued or otherwise problematic certificates. But that leaves the question about '''when''' this should be done, particularly if it's not possible to contact the customer immediately, or if they are unable to replace their certificate quickly. Section CAs should ensure that they are complying with Sections 4.9.1through 4.9.1 5 of '''[https://cabforum.org/baseline-requirements-documents/ the CA/Browser Forum’s Baseline Requirements currently states (version 1]'''.7.0):
<blockquote>“The CA SHOULD revoke a Certificate within 24 hours and MUST revoke a Certificate within 5 days if one or more of the following occurs: …<br>7. The CA is made aware This means that the Certificate was not issued , in accordance with these Requirements or the CA’s Certificate Policy or Certification Practice Statement;<br>8. The CA determines or is made aware that any most cases of misissuance, the information appearing in the Certificate is inaccurate; …<br>10. Revocation is required by the CA’s Certificate Policy and/or Certification Practice Statement; or<br>11. The CA is made aware of a demonstrated or proven method that exposes has an obligation under the Subscriber's Private Key BRs to compromise, methods have been developed that can easily calculate it based on revoke the Public Key (such as a Debian weak key, see http://wiki.debian.org/SSLkeys)certificates concerned within 24 hours, or if there is clear evidence that the specific method used to generate the Private Key was flawed5 days in some cases."</blockquote>
This means that, in most cases of misissuance, the CA has an obligation under the BRs to revoke the certificates concerned within 5 days (or within 24 hours in some cases). Mozilla recognizes that in some '''exceptional''' circumstances, revoking the affected certificates within the prescribed deadline may cause significant harm, such as when the certificate is used in critical infrastructure and cannot be safely replaced prior to the revocation deadline, or when the volume of revocations in a short period of time would result in a large cumulative impact to the web. However, Mozilla does not grant exceptions to the BR revocation requirements. It is our position that your CA is ultimately responsible for deciding if the harm caused by following the requirements of BR section 4.9.1 the Baseline Requirements outweighs the risks that are passed on to individuals who rely on the web PKI by choosing not to meet this requirement.
If your CA will not be revoking the certificates within the time period required by the BRs, our expectations are that:
* A separate incident report will be filed in Bugzilla.* The decision and rationale for delaying revocation will be disclosed to Mozilla in the form of a preliminary incident report immediately; preferably before the BR-mandated revocation deadline. The rationale must include an explanation detailed and substantiated explanations for why the situation is exceptional. Responses similar to “we do not deem this non-compliant certificate to be a security risk” are not acceptable. When revocation is delayed at the request of specific Subscribers, the rationale must be provided on a per-Subscriber basis.
* Any decision to not comply with the timeline specified in the Baseline Requirements must also be accompanied by a clear timeline describing if and when the problematic certificates will be revoked or expire naturally, and supported by the rationale to delay revocation.
* The issue will need to be listed as a finding in your CA’s next BR audit statement.
* Work out how the bug or problem was introduced. For a code bug, were the code review processes sufficient? Does your code have automated tests, and if so, why did they not catch this case?
* Work out why the problem was not detected earlier. Were these certificates missed by your linting processes or self-audits? Or is the code or process you use for such audits insufficiently frequent or rigorousinsufficient?
* If the problem is lack of compliance to an RFC, Baseline Requirement , or Mozilla Policy requirement: were you aware of this requirement? If not, why not? If so, was an attempt made to meet it? If not, why not? If so, why was that attempt flawed? Do any processes need updating for making sure your CA complies with the latest version of the various requirements placed upon it?
* Scan your corpus of certificates to look for others with the same issue. It does not look good for a CA to claim they have revoked all affected certificates and resolved the issue, and then for a researcher to discover another set of certificates with the same or a similar problem.
= Incident Report =
[https://bugzilla.mozilla.org/show_bug.cgi?id=1690807#c2 Example Incident Report]
<br />
The purpose of incident reporting is to help all of us work together to build a more secure web. Therefore, the incident report should share lessons learned that could be helpful to all CAs to build better systems. The incident report should explain how the systems failed, how was the mis-issuance or incident possible, and why the problem was not detected earlier. In addition to the timeline of responding to and resolving the incident, the incident report should explain how the CA's systems will be made more robust, and how other CAs may learn from the incident.
Each incident should result in an incident report, written as soon as the problem is fully diagnosed and (temporary or permanent) measures have been put in place to make sure it will not re-occur. If the permanent fix is going to take significant time to implement, you should not wait until this is done before issuing the report. We expect to see incident reports as soon as possible, and certainly within two weeks of the initial issue report. While remediation work may still be ongoing, a satisfactory incident report will serve to resolve the issue from a Mozilla perspective.
There should be a single incident report for each distinct matter, and CA operators should submit an additional, separate incident report when:
* Mozilla policy requires that the CA revoke one or more certificates by a certain deadline, such as those in BR section 4.9, but that deadline will not be or has not been met by the CA.
* In the process of researching one incident, another incident with a distinct root cause and/or remediation is discovered.
* An audit report or conformity assessment contains more than one qualification, major non-conformity, or other adverse finding.
* After an incident bug is marked resolved, the incident reoccurs.
The incident report may well repeat things which have been said previously in discussions or bug comments. This is entirely expected. The report should be a summary of previous findings. The existence of data in discussions or bug comments does not excuse a CA from the task of compiling a proper incident report.
Your CA must submit an incident report by [https://bugzilla.mozilla.org/enter_bug.cgi?product=CA%20Program&component=CA%20Certificate%20Compliance&version=other creating a bug in Bugzilla under the CA Program :: CA Certificate Compliance component]. When the incident is reported only on the CCADB public list or on the [https://groups.google.com/a/mozilla.org/g/dev-security-policy MDSP mailing list], then a bug will be created to track the incident and its resolution in Bugzilla. CAs are encouraged to announce important incidents on public@ccadb.org when they involve the Baseline Requirements, other root programs, or the CCADB; or on the Mozilla dev-security-policy list, when they only involve violations of the Mozilla Root Store Policy.
For more information on filing an incident report, be sure to read https://www.ccadb.org/cas/incident-report. The incident report should cover at least follow the following topics: # How your CA first became aware of the problem (e.g. via a problem report submitted to your Problem Reporting Mechanism, a discussion in guidance provided on the [httpsCCADB website://groups.google.com/a/mozilla.org/g/dev-security-policy MDSP mailing list], a Bugzilla bug, or internal self-audit), and the time and date.# A timeline of the actions your CA took in response. A timeline is a date-and-time-stamped sequence of all relevant events. This may include events before the incident was reported, such as when a particular requirement became applicable, or a document changed, or a bug was introduced, or an audit was performed.# Whether your CA has stopped, or has not yet stopped, certificate issuance or the process giving rise to the problem or incident. A statement that you have stopped will be considered a pledge to the community; a statement that you have not stopped requires an explanation.# In a case involving certificates, a summary of the problematic certificates. For each problem: the number of certificates, and the date the first and last certificates with that problem were issued. In other incidents that do not involve enumerating the affected certificates (e.g. OCSP failures, audit findings, delayed responses, etc.), please provide other similar statistics, aggregates, and a summary for each type of problem identified. This will help us measure the severity of each problem.# In a case involving TLS server certificates, the complete certificate data for the problematic certificates. The recommended way to provide this is to ensure each certificate is logged to CT and then list the fingerprints or crt.sh IDs, either in the report or as an attached spreadsheet, with one list per distinct problem. It is also recommended that you use this form in your list "<nowiki>https://crt.sh/?sha256=[sha256-hash]</nowiki>", unless circumstances dictate otherwise. When the incident being reported involves an SMIME certificate, if disclosure of personally identifiable information in the certificate may be contrary to applicable law, please provide at least the certificate serial number and SHA256 hash of the certificate. In other cases not involving a review of affected certificates, please provide other similar, relevant specifics, if any.# Explanation about how and why the mistakes were made or bugs introduced, and how they avoided detection until now.# List of steps your CA is taking to resolve the situation and ensure that such situation or incident will not be repeated in the future. The steps should include the action(s) for resolving the issue, the status of each action, and the date each action will be completed. The purpose of these incident reports is to provide transparency about the steps the CA is taking to address the immediate issue and prevent future issues, both the issue that originally led to the report, and other potential issues that might share a similar root cause. Additionally, they exist to help the CA community as a whole learn from potential incidents, and adopt and improve practices and controls, to better protect all CAs. Mozilla expects that the incident reports provide sufficient detail about the root cause, and the remediation, that would allow other CAs or members of the public to implement an equivalent solution.
For example, it’s not sufficient to say that “human error” or “lack of training” was a root cause for the '''https://www.ccadb.org/cas/incident, nor that “training has been improved” as a solution. While a lack of training may have contributed to the issue, it’s also possible that error-prone tools or practices were required, and making those tools less reliant on training is the correct solution. When training or a process is improved, the CA is expected to provide specific details about the original and corrected material, and specifically detail the changes that were made, and how they tie to the issue. Training alone should not be seen as a sufficient mitigation, and focus should be made on removing errorreport#incident-prone manual steps from the system entirely.reports'''
= Keeping Us Informed =
