Confirm
384
edits
Changes
First draft
{{draft}}
'''Summary of Incidents'''
== A. Incidents related to Missing CPS URI in EV Certificates ==
The incidents listed below in this section A are related to [https://bugzilla.mozilla.org/show_bug.cgi?id=1883843 Bug #1883843].
=== '''1. EV TLS Certificate cPSuri missing -''' ===
https://bugzilla.mozilla.org/show_bug.cgi?id=1883843
After an alert from Ryan Dickson, Entrust discovered that all of its EV certificates issued since the implementation of changes due to Ballot SC-62v2 (approximately 26,668 certificates) were missing the CPS URI, in violation of the EV Guidelines. Entrust said this was due to discrepancies/misinterpretations between the CA/Browser Forum’s TLS Baseline Requirements and the Extended Validation Guidelines. Entrust initially chose to not stop issuing the EV certificates and did not begin the process of revoking the certificates. They argued that: the absence of the CPS URI in the EV certificates was due to ambiguities in CA/B Forum requirements; the absence of the CPS URI had no security impact; and halting and revoking the certificates would negatively impact customers and the broader WebPKI ecosystem. Entrust then also proposed a ballot to adjust the EV Guidelines to not require the CPS URI. It also argued that its efforts were better spent focusing on improving automation and handling of certificates, rather than on revocation and reissuance.
Entrust’s decisions met with substantial backlash from the Mozilla community and other stakeholders, who argued that Entrust's failure to comply with established rules and guidelines and its reluctance to take immediate corrective actions were unacceptable. They emphasized the need for adherence to standards and prompt remediation and revocation as per standard incident response protocols.
Ultimately, Entrust acknowledged the necessity to stop issuance and to revoke the affected certificates in response to the community’s concerns. It admitted that its initial response did not align with community expectations, and it committed to improving its incident handling, communication, and operational procedures. Entrust also highlighted its efforts to enhance system processes for detecting and addressing errors more effectively and to improve automation and resilience among its subscribers for better management of similar incidents in the future. It has recognized the need to address guideline discrepancies more responsibly and now views the experience as a learning opportunity to improve its approach to certificate issuance and incident management.
'''Issues:''' Misinterpretation of Requirements; Policy/Procedure Failure; Certificate Mis-issuance; Early Detection; Incident Response; Incident Handling; Delayed Revocation; Poor Communication
=== 2. Failed to provide a preliminary incident report according to TLS BR 4.9.5 - ===
https://bugzilla.mozilla.org/show_bug.cgi?id=1890123
Related to [https://bugzilla.mozilla.org/show_bug.cgi?id=1883843 bug #1883843], directly above. Entrust failed to provide a preliminary incident report as required by the TLS Baseline Requirements, section 4.9.5, after receiving a certificate problem report (CPR). (Entrust confirmed the issue in just under 26 hours and posted its preliminary report about 43 hours after the initial notification.) The delay was attributed to a lack of specific procedural steps and reminders in the CPR flow and a gap in Entrust's incident handling procedures. To prevent similar issues, Entrust is improving its incident management procedures, by June 30, 2024, and will be incorporating specific steps and reminders to ensure timely reporting of future incidents.
'''Issues:''' Incident Handling; Incident Response
=== 3. CPR was not responded to in 24 hours - ===
https://bugzilla.mozilla.org/show_bug.cgi?id=1885754
This is related to [https://bugzilla.mozilla.org/show_bug.cgi?id=1883843 bug #1883843] (discussed above). Entrust did not respond to a Certificate Problem Report (CPR) for its AffirmTrust PKI hierarchy within 24 hours. Entrust said that it was already aware of the issue due to a previous incident report concerning the related issue for the Entrust PKI hierarchy, but that the CPR reporter was unaware of the receipt of and action on that CPR. Root cause was that a Technical Support Specialist did not recognize CPR as such, leading to the delayed response. CPR reporting processes needed to be improved. Action items included re-training for Technical Support, as well as implementation of automation for flagging and responding to CPRs and creation of a CPR form for reporters to complete.
'''Issues:''' Incident Handling; Incident Response
=== 4. Delayed revocation of EV TLS certificates with missing cPSuri - ===
https://bugzilla.mozilla.org/show_bug.cgi?id=1886532
All affected certificates should have been revoked within 5 days. Entrust asserted that delayed revocation was due to confusion about TLS EV profile changes and various customer-related factors, as discussed above in [https://bugzilla.mozilla.org/show_bug.cgi?id=1883843 bug #1883843]. Weekly updates are being provided. However, as of May 1, 2024, approximately 14,736 certificates of the 26,668 affected certificates had been revoked (55.26%).
'''Issues:''' Delayed Revocation
=== 5. EV Certificate missing Issuer’s EV Policy OID - ===
https://bugzilla.mozilla.org/show_bug.cgi?id=1888714
Entrust issued 1,963 EV TLS certificates September 11-22, 2023, without including an EV TLS CP OID. Root Causes were the misinterpretation of the EV Guidelines and the TLS BRs and a failure to recognize the overriding requirements of the EV Guidelines. (A misinterpretation of standards led to non-compliant certificates, and linting failed to detect the issue.) As remediation, since April 11, 2024, Entrust has used pkilint as a post-issuance linter to detect similar issues. (Mis-issued certificates are a subset of the certificates disclosed and being revoked under [https://bugzilla.mozilla.org/show_bug.cgi?id=1883843 bug #1883843]. Status of revocation is listed in [https://bugzilla.mozilla.org/show_bug.cgi?id=1886532 bug #1886532].)
'''Issues:''' Misinterpretation of Requirements; Policy/Procedure Failure; Certificate Mis-issuance
=== 6. Delay in Updating CPS - ===
https://bugzilla.mozilla.org/show_bug.cgi?id=1887753
There was an oversight during an update related to Entrust’s EV TLS certificates, see [https://bugzilla.mozilla.org/show_bug.cgi?id=1883843 bug #1883843]. The incident involved the improper removal and later reinstatement of the Entrust EV TLS Certificate Policy (CP) Object Identifier (OID) without updating the CPS accordingly.
Entrust removed the EV TLS CP OID from certificates on September 11, 2023, to align with the Baseline Requirements, which caused some browsers not to display these certificates as EV certificates. It then added the CP OID back into the certificates on September 22, 2023, but the CPS was not immediately updated to reflect this change. The CPS was finally updated on March 22, 2024, until then, certificates were technically compliant with TLS Baseline Requirements, but not with Entrust's own CPS.
The root cause was that procedures for changing certificate profiles did not include a step to update the CPS concurrently. Entrust learned that there needs to be a mechanism to flag changes affecting compliance documentation to ensure it is updated alongside any policy changes. Action is being taken to review and update release and emergency release processes to better synchronize with compliance documentation. (This incident underscores the importance of accurate documentation and the need for procedures that ensure CPS updates are consistent with other changes in certificate policies and profiles.)
'''Issues:''' Misinterpretation of Requirements; Policy/Procedure
=== 7. Failure to revoke EV TLS certificates issued before CPS update - ===
https://bugzilla.mozilla.org/show_bug.cgi?id=1890685
This incident covers a subset of certificates mentioned above in [https://bugzilla.mozilla.org/show_bug.cgi?id=1887753 bug #1887753] that were issued between March 18-21, 2024. The certificates did not comply with the Entrust CPS published at the time, although they conformed to the actual intended certificate profile and EV Guidelines. Entrust’s decision not to revoke contrasts with actions taken for certificates issued prior to March 18, 2024, which were revoked and reissued due to a different issue ([https://bugzilla.mozilla.org/show_bug.cgi?id=1883843 bug #1883843]) that necessitated a profile change.
'''Issues:''' Delayed Revocation
-----------------------------------------------------------
== B. Revocation of Certificates without serverAuth EKU ==
=== 1. clientAuth TLS Certificates without serverAuth EKU - ===
https://bugzilla.mozilla.org/show_bug.cgi?id=1886467
Entrust issued 1,176 TLS certificates with id-kp-clientAuth EKU, but not id-kp-serverAuth. Root cause was a system that allowed customers to configure the EKU. One “lesson learned” was that Entrust needs to review user-configurable options and requirements more thoroughly. It was noted that linters did not detect the issue, highlighting the need for improved automated detection tools. Action items included fixing EKU checking in lint tools and reviewing all certificate profiles for potential errors.
'''Issues:''' Certificate Mis-issuance; Implementation/Configuration Error
=== 2. Delayed revocation of clientAuth TLS Certificates without serverAuth EKU - ===
https://bugzilla.mozilla.org/show_bug.cgi?id=1887705
Delayed revocation bug filed regarding [https://bugzilla.mozilla.org/show_bug.cgi?id=1886467 bug #1886467], above. Weekly updates are being provided. Entrust said that the delayed revocation was due mainly to the high workload already as a result of incident described in [https://bugzilla.mozilla.org/show_bug.cgi?id=1883843 bug #1883843] (discussed above), certificate profile changes, customer workload, and system lockdowns for end-of-quarter / holiday issues. As of May 1, 2024, 310 certificates of 1,176 affected certificates had been revoked (26.36%).
'''Issues:''' Delayed Revocation
-----------------------------------------------------------
== C. Policy-Procedure Failure: CPS ==
=== 1. CPS typographical (text placement) error - ===
https://bugzilla.mozilla.org/show_bug.cgi?id=1890896
From March 22-26, 2024, Entrust issued 6,008 OV TLS certificates under a CPS requirement related to the policyQualifier in the OV certificate profile. The error was corrected in a subsequent CPS. (The certificates were compliant with the CA/B Forum’s TLS BRs and Entrust's OV profile.) Root-cause factors included generic naming of certificate profiles and concurrent updates to the CPS during incidents requiring rapid certificate replacement and revocation. Remediation actions included errata to the affected CPS, informing subscribers of CPS error, reviewing CPS update procedure, and renaming certificate profile for clarity. (Community members questioned Entrust’s decision to not revoke the certificates, and Entrust responded regarding its position on such matters--Entrust maintains that the certificates were compliant and that it can handle mass revocation and reissuance.)
'''Issues:''' Policy/Procedure Failure
=== 2. Delayed incident report - CPS typographical (text placement) error (Closed) -
https://bugzilla.mozilla.org/show_bug.cgi?id=1890901
This is related to [https://bugzilla.mozilla.org/show_bug.cgi?id=1890896 bug #1890896], above. Entrust delayed in filing an incident report (an initial report should be filed within 72 hours of the CA being made aware of an incident). Entrust explained that priorities of other incidents delayed the reporting of the incident in [https://bugzilla.mozilla.org/show_bug.cgi?id=1890896 bug #1890896].
'''Issues:''' Incident Handling; Incident Response
=== 3. Failure to revoke OV TLS - CPS typographical (text placement) error - ===
https://bugzilla.mozilla.org/show_bug.cgi?id=1890898
This is related to [https://bugzilla.mozilla.org/show_bug.cgi?id=1890896 bug #1890896] above. The error and correction involved the CPS, not the certificates themselves, and re-issuing would result in similar certificates with new issuance dates. The error was discovered and corrected on March 26 with the posting of CPS version 3.20. However, community members have raised concerns about Entrust’s commitment to compliance with Baseline Requirements, its assertion of exceptional conditions, and the deviation from revocation timelines set forth in the Baseline Requirements.
'''Issues:''' Delayed Revocation
=== 4. Not updating Problem Reporting Mechanism fields in CCADB - ===
https://bugzilla.mozilla.org/show_bug.cgi?id=1894111
Entrust's entry in the Problem Reporting Mechanism field of the Common CA Database (CCADB) was inconsistent with its CPS and outdated information needed to be updated. Entrust updated this information in the CCADB.
'''Issues:''' Policy/Procedure Failure
-----------------------------------------------------------
== D. OCSP and CRL Issues ==
=== 1. OCSP response signed with SHA-1 - ===
https://bugzilla.mozilla.org/show_bug.cgi?id=1879602
On February 3, 2024, Entrust became aware that the SHA-1 algorithm was being used when signing OCSP responses. (This was because the default setting for online OCSP responders was to sign OCSP responses using the same algorithm as the CA certificate, which was SHA-1.) By February 6, a fix was applied to the production system so that OCSP signatures would be SHA-256. Entrust has enhanced its OCSP monitoring system to verify correct signing algorithms and incorporated OCSP Watch into its daily monitoring routines. This incident underlines the importance of diligent monitoring and updating of cryptographic practices to adhere to industry standards.
'''Issues:''' Implementation/Configuration Error
=== 2. CRL non-conformance with the TLS BRs (Closed) - ===
https://bugzilla.mozilla.org/show_bug.cgi?id=1889217
Two CRLs listed in the CCADB did not meet the TLS BRs and RFC 5280 because they contained the "revoked certificates" field, despite having no revoked certificates. The issue was detected using pkilint. Entrust updated CRL generation software and all affected CRLs were reissued. The root cause was identified as the use of a non-compliant function (CreateCRL) in its Go Crypto/x509 library. As remediation, Entrust is to conduct regular review and validation of dependencies, including forked packages, deploy linting tools and automated tests for compliance, implement continuous monitoring and review practices, such as CRL Watch and OCSP Watch. Also, Entrust to develop a plan for periodic review of forked libraries and their updates and to enhance processes for tracking and analyzing changes in requirements.
'''Issues:''' Implementation/Configuration Error
