Part V: When trust breaks · Chapter 18

Famous failures and what changed

In one sentence

Real PKI failures, from a hacked CA to expired certificates, follow one pattern: a weak point breaks trust, someone notices, and the rules or practices change.

Who is this for

Level: Practitioner

On this page
  1. The idea
  2. How it really works
  3. At work
  4. Common mistakes
  5. Why it matters to the business
  6. Check yourself
  7. Go deeper

The idea

  • Beginner

PKI rests on a few promises. CAs check before they sign. CAs protect their keys. And someone renews every certificate before it expires. When one of these promises breaks, trust breaks with it, for one website or for millions.

This chapter looks at real failures that the organisations involved, or official investigators, documented in public. For each one we ask three questions:

  1. What went wrong?
  2. How was it noticed?
  3. What changed afterwards?

The failures fall into three groups:

  • A CA is broken into. An attacker uses the CA to issue certificates for names that are not theirs (DigiNotar).
  • A CA keeps breaking the rules. The browser makers stop trusting it. This is called distrust (Symantec, Entrust).
  • Something expires. Either a root certificate that many devices relied on (DST Root CA X3), or an ordinary server certificate that nobody renewed.

In web PKI, the browser and operating-system makers act as referees. They decide which root certificates go into their trust stores, and removing a CA from a trust store is the strongest penalty there is.

The dramatic failures are break-ins. The ordinary one is an expired certificate, and it has taken down services at some of the largest technology companies in the world. The lessons are less about clever attackers and more about missing checks, missing inventories and manual processes.

How it really works

  • Developer
  • Expert

How did one hacked CA threaten everyone? DigiNotar

DigiNotar was a Dutch CA, trusted by all major browsers and also used by the Dutch government. In 2011, attackers broke into its network and reached the systems that issued certificates. The investigation commissioned by the Dutch government, Fox-IT’s Black Tulip report, gives “a non-exhaustive list of 531 rogue certificates”.

How it was noticed. DigiNotar’s own staff found the intrusion when a routine automated check, which had stopped working, was restored and flagged certificates that did not match the company’s records. The company revoked the rogue certificates it knew about. It did not tell the browser makers. Mozilla later wrote that DigiNotar “detected and revoked some of the fraudulent certificates 6 weeks ago without notifying Mozilla” (Mozilla Security Blog). The public found out when a Gmail user in Iran posted a warning: Chrome had rejected a certificate for a Google domain, because Chrome checked Google’s own certificates against a built-in list of expected keys (certificate pinning). From the OCSP requests for the rogue *.google.com certificate, the final report counted 298,140 unique IP addresses as likely victims, 95% of them in Iran, and calls that number only “a very rough approximation” of the people affected.

What changed. DigiNotar was breached, issued fraudulent certificates and was removed from browser trust stores. Google, Mozilla, Microsoft and Apple all removed trust within weeks. The Dutch government revoked its own trust in DigiNotar and took over its operations, and the company was declared bankrupt.

Two lessons stayed. Any trusted CA can issue for any name, so one breach endangers everyone. And a CA that hides an incident loses trust completely. Certificate Transparency, standardised in June 2013 and December 2021, now makes rogue certificates visible to every domain owner, not only to companies that pin their own keys.

What happens when a CA keeps breaking the rules? Symantec and Entrust

Symantec: what went wrong. In January 2017, a public post on Mozilla's security-policy forum “drew attention to a series of questionable website authentication certificates”, in the words of Google’s later announcement. The Chrome team found that Symantec “had entrusted several organizations with the ability to issue certificates without the appropriate or necessary oversight”, and called it “part of a continuing pattern of issues over the past several years”. In Chrome’s public discussion, the count grew from “reportedly 127 certificates” to “at least 30,000 certificates” (blink-dev).

What changed. Browsers withdrew trust in phases, by issue date: Chrome 66 distrusted Symantec certificates issued before 1 June 2016, and Chrome 70 removed trust in the rest (April and October 2018). Mozilla followed a matching plan. Symantec sold its CA business to DigiCert, and every site owner with an affected certificate had to replace it before the deadline.

Entrust: what went wrong. In Google’s words: “Over the past six years, we have observed a pattern of compliance failures, unmet improvement commitments, and the absence of tangible, measurable progress in response to publicly disclosed incident reports.” It was noticed the way most CA problems are noticed today: through public incident reports and the root programs’ scrutiny of them.

What changed. Chrome stopped trusting new Entrust TLS certificates (those first logged in Certificate Transparency after 11 November 2024). Certificates issued before the cut-off kept working until they expired. Apple, Mozilla and Microsoft set their own cut-off dates: 15 November 2024, 30 November 2024 and 16 April 2025.

Chrome’s cut-off uses the date of the certificate’s earliest SCT. Certificate Transparency gives a trustworthy, public issue date, which makes this kind of partial distrust possible.

Partial distrust: certificates issued before the cut-off keep working until they expire; certificates issued after it are rejected Issued before ✓ trusted until it expires then renew with another CA Issued after ✗ rejected by the browser site shows a trust error cut-off date set by the root program issue date (Chrome uses the earliest SCT) The CA's roots stay in the store for now, so older certificates still work
Partial distrust: the cut-off date decides which certificates keep working.

Distrust is rare, slow and public, and it always comes with a deadline. Organisations that can switch CA quickly barely notice. Those that renew by hand have to find and replace every affected certificate in a few months.

Why did a root expiry break working sites? DST Root CA X3

What went wrong. Let’s Encrypt’s own root, ISRG Root X1, was young and missing from many older devices. So for years the default Let’s Encrypt chain also included a cross-signature from IdenTrust’s older DST Root CA X3, which those devices already trusted. On 30 September 2021, DST Root CA X3 expired. Let’s Encrypt had announced it well in advance. Two kinds of client failed:

  • devices that never received ISRG Root X1, because “some older devices won’t ever trust it because they don’t get software updates”;
  • clients built on OpenSSL 1.0.2, even when they did trust ISRG Root X1. The OpenSSL project explained that OpenSSL 1.0.2 “always prefers the untrusted chain and if that chain contains a path that leads to an expired trusted root certificate (DST Root CA X3), it will be selected for the certificate verification and the expiration will be reported.”
Two paths to trust in 2021: the self-signed ISRG Root X1, or a cross-signature from the expiring DST Root CA X3 Your certificate Intermediate R3 issued by ISRG Root X1 ✓ self-signed, in newer stores ISRG Root X1 cross-signed by DST DST Root CA X3 ✗ expired ✓ Clients that trust ISRG Root X1: work ✗ Old devices without it: fail ✗ OpenSSL 1.0.x: picked the expired path Same key, two certificates: the server chose which chain to send.
One key, two certificates, two paths. Old clients could only see the expired one.

How it was noticed. This failure was planned and announced. It surfaced on the day as certificate errors in old phones, embedded devices and servers running old OpenSSL versions, while up-to-date browsers carried on.

What changed. Let’s Encrypt kept old Android devices working with a trick: a cross-signature that outlived the DST root, which works because “Android intentionally does not enforce the expiration dates of certificates used as trust anchors” (Let’s Encrypt). Later, Let's Encrypt's cross-signed chain for old Android devices expired (30 September 2024). The lasting lessons: every certificate in a chain has an end date, not only yours; TLS libraries and trust stores need updates too; and old clients deserve a test before a root change.

Why do expired certificates still cause outages?

Expiry is the most ordinary failure in PKI, and big operators are not immune. These four cases come with public post-mortems or official reports:

Incident What expired How it was noticed What changed
Azure Storage, February 2013 Certificates for HTTPS access to storage, the same in every region Normal monitoring, about 15 minutes after the failure Monitor production endpoints, open an incident well before expiry, split certificates across services, regions and time (post-mortem)
Equifax, 2017 The certificate on a device that inspected encrypted network traffic Inspection restarted when the certificate was renewed, and an administrator then saw signs of an intrusion New tools for continuous network monitoring (GAO report)
Google Voice, February 2021 The active certificate in front-end systems, after a problem updating certificate configuration Engineers were alerted automatically Proactive alerting for upcoming expiry, better automated rotation (incident report)
Epic Games, April 2021 A certificate for an internal DNS zone that monitoring did not cover; automatic renewal was off Reports within minutes of expiry Automated renewal, a manual audit of all certificates, new monitoring, a smaller blast radius per certificate (post-mortem)

The Equifax case is the starkest. In the words of the US Government Accountability Office: “the misconfiguration was due to an expired digital certificate”, meaning “that encrypted traffic was not being inspected”, and “the attack lasted for about 76 days before it was discovered”. The expired certificate did not break the business; it silently switched off a security control.

Microsoft’s Azure post-mortem names the pattern behind most expiry outages: “a breakdown in our procedures for maintaining and monitoring these certificates was the root cause.” Epic Games found the same gap in its own monitoring: “the configuration for this internal zone was missing”.

At work

  • Developer
  • IT ops

Which CAs and roots does your service depend on?

Look at the whole chain your server sends, not only the first certificate. This is the real output for letsencrypt.org:

openssl s_client -connect letsencrypt.org:443 -servername letsencrypt.org </dev/null
Certificate chain
 0 s:CN=letsencrypt.org
   i:C=US, O=Let's Encrypt, CN=YE2
   a:PKEY: id-ecPublicKey, 256 (bit); sigalg: ecdsa-with-SHA384
   v:NotBefore: Sep  4 14:34:32 2026 GMT; NotAfter: Dec  3 14:34:31 2026 GMT
 1 s:C=US, O=Let's Encrypt, CN=YE2
   i:C=US, O=ISRG, CN=Root YE
   a:PKEY: id-ecPublicKey, 384 (bit); sigalg: ecdsa-with-SHA384
   v:NotBefore: Sep  3 00:00:00 2025 GMT; NotAfter: Sep  2 23:59:59 2028 GMT
 2 s:C=US, O=ISRG, CN=Root YE
   i:C=US, O=Internet Security Research Group, CN=ISRG Root X2
   a:PKEY: id-ecPublicKey, 384 (bit); sigalg: ecdsa-with-SHA384
   v:NotBefore: May 13 00:00:00 2026 GMT; NotAfter: Sep  2 23:59:59 2032 GMT
 3 s:C=US, O=Internet Security Research Group, CN=ISRG Root X2
   i:C=US, O=Internet Security Research Group, CN=ISRG Root X1
   a:PKEY: id-ecPublicKey, 384 (bit); sigalg: RSA-SHA256
   v:NotBefore: May 13 00:00:00 2026 GMT; NotAfter: Sep  2 23:59:59 2032 GMT
---

Entries 2 and 3 are cross-signatures: Root YE signed by ISRG Root X2, and ISRG Root X2 signed by ISRG Root X1. It is the same pattern as with DST Root CA X3: a newer root reaches older trust stores through an older one. Write down the NotAfter of every entry, and know which CA and which root each of your services depends on. Paste a chain into the chain checker to see the path:

Try it: Chain checkerOpen the full tool

Runs in your browser. Nothing you paste is sent anywhere.

Every certificate the server sends, leaf first. The output of openssl s_client -showcerts -connect example.com:443 works as it is: we ignore the text around the certificates.

Are you ready for a distrust?

  • Know your CAs. Your inventory should say which CA issued each certificate (see Inventory, expiry and renewal). Certificate Transparency searches show what was issued for your names.
  • Follow the root programs. Chrome, Mozilla, Apple and Microsoft announce distrust decisions publicly, months ahead.
  • Be able to switch CA in weeks. With ACME, switching is mostly a change of directory URL and a test run. With manual renewals, it is a project.

Are your old clients ready for root changes?

Servers that call other services are clients too. Check the TLS library and the CA bundle on old hosts, containers, appliances and embedded devices:

openssl version
OpenSSL 3.2.3 3 Sep 2024 (Library: OpenSSL 3.2.3 3 Sep 2024)

Java applications use their own trust store, and many appliances have one you can only update with firmware. When a client fails after a root change, the error pages explain the symptoms: certificate has expired and NET::ERR_CERT_AUTHORITY_INVALID.

What do the expiry post-mortems teach?

  • Monitor what clients see, the live endpoint, not only the certificate store (Azure).
  • Cover internal names and security devices, not only public websites (Epic Games, Equifax).
  • Automate renewal, and give each service its own certificate so that one expiry cannot take down everything.
  • Alert on a share of the lifetime, not a fixed number of days. Azure’s rule after its outage was an incident for any certificate with “less than 3 months until the expiration date”. With certificates limited to 47 days from 15 March 2029, that rule would fire from the day of issue.

An expired certificate looks like this in the decoder, and like NET::ERR_CERT_DATE_INVALID in Chrome:

Try it: Certificate decoderOpen the full tool

Runs in your browser. Nothing you paste is sent anywhere.

PEM text, several PEM blocks, or a .pem, .crt, .cer or .der file (drop it on the box). Certificates are public, so they are safe to paste.

Common mistakes

  • “One CA was hacked, so PKI is broken.” The system reacted: browsers removed DigiNotar within weeks, and Certificate Transparency now exposes rogue certificates to everyone. One CA failing is a reason for monitoring, not for giving up on PKI.
  • “Distrust kills every certificate at once.” Browsers usually distrust by issue date. Certificates issued before the cut-off keep working until they expire, which gives owners time to move.
  • “Only my own certificate can expire.” Roots, intermediates and cross-signatures expire too, as DST Root CA X3 showed.
  • “Expiry outages only happen to small companies.” Microsoft, Google and Epic Games published their own post-mortems.
  • “Our monitoring covers every certificate.” Epic’s monitoring missed an internal zone, and Equifax’s expired certificate sat on a security device. Check what your monitoring actually covers.

Why it matters to the business

  • Leader

Check yourself

Check yourself

A browser announces that it will distrust a CA. Your site uses a certificate from that CA, issued last month. What usually happens?

  1. The site breaks on the announcement day
  2. The certificate keeps working until it expires; the next one must come from another CA
  3. Nothing ever changes for existing customers
Show answer

Answer B The certificate keeps working until it expires; the next one must come from another CA

Check yourself

When DST Root CA X3 expired, why did some clients fail even though they trusted ISRG Root X1?

  1. Their OpenSSL 1.0.2 picked the path to the expired root
  2. Let's Encrypt revoked all its certificates
  3. Browsers stopped supporting RSA
Show answer

Answer A Their OpenSSL 1.0.2 picked the path to the expired root

Check yourself

At Equifax, what did an expired certificate on a traffic-inspection device cause?

  1. A website outage that customers reported
  2. Encrypted traffic went uninspected, so an intrusion was missed for about 76 days
  3. Browsers showed a warning on the company's site
Show answer

Answer B Encrypted traffic went uninspected, so an intrusion was missed for about 76 days

Go deeper

In the gameLevel 18: Famous failuresOld Archive, Keystone. In development

The crash course covers these failures in lesson 8, When trust breaks.

Early edition. Checked against the standards; an independent expert review is in progress. Found a mistake? Tell us.

Last reviewed Edition 0.9