<!-- https://pkizerotohero.com/handbook/famous-failures/ | Markdown mirror of the HTML page | PKI Zero to Hero, by Duyutech BV -->

> DigiNotar, Symantec, Entrust, the DST Root CA X3 expiry and expired-certificate outages: what went wrong, how it was noticed and what changed.

Source: https://pkizerotohero.com/handbook/famous-failures/ · Last reviewed: 2026-10-20

Part V: When trust breaks · Chapter 18

# Famous failures and what changed

**In one sentence:**

Real PKI failures, from a hacked CA to expired certificates, follow one pattern: a weak point breaks trust, someone notices, and the rules or practices change.

## Who is this for

- [IT ops](https://pkizerotohero.com/start/#itops)
- [Expert](https://pkizerotohero.com/start/#expert)
- [Leader](https://pkizerotohero.com/start/#leader)
- [Developer](https://pkizerotohero.com/start/#developer)
- [Beginner](https://pkizerotohero.com/start/#beginner)

Level: **Practitioner**

- [New to this? Start with The idea](https://pkizerotohero.com/handbook/famous-failures/#the-idea)
- [Developer? Jump to How it really works](https://pkizerotohero.com/handbook/famous-failures/#how-it-really-works)
- [IT ops? At work](https://pkizerotohero.com/handbook/famous-failures/#at-work)
- [Leader? Why it matters](https://pkizerotohero.com/handbook/famous-failures/#why-it-matters-to-the-business)

## The idea

- Beginner

PKI rests on a few promises. CAs check before they sign. CAs protect their keys. And someone renews every certificate before it expires. When one of these promises breaks, trust breaks with it, for one website or for millions.

This chapter looks at real failures that the organisations involved, or official investigators, documented in public. For each one we ask three questions:

1. **What went wrong?**
2. **How was it noticed?**
3. **What changed afterwards?**

The failures fall into three groups:

- **A CA is broken into.** An attacker uses the CA to issue certificates for names that are not theirs (DigiNotar).
- **A CA keeps breaking the rules.** The browser makers stop trusting it. This is called [distrust](https://pkizerotohero.com/handbook/glossary/#distrust) (Symantec, Entrust).
- **Something expires.** Either a root certificate that many devices relied on (DST Root CA X3), or an ordinary server certificate that nobody renewed.

In web PKI, the browser and operating-system makers act as referees. They decide which root certificates go into their [trust stores](https://pkizerotohero.com/handbook/trust-stores/), and removing a CA from a trust store is the strongest penalty there is.

The dramatic failures are break-ins. The ordinary one is an expired certificate, and it has taken down services at some of the largest technology companies in the world. The lessons are less about clever attackers and more about missing checks, missing inventories and manual processes.

In the game

The Old Archive

In the game, the Old Archive keeps one exhibit for each disaster, each with three labels: what went wrong, how it was noticed and what changed. Where it stops being true: real failures rarely have a villain in a disguise. Most come from ordinary mistakes, missing processes and checks that nobody ran, and the lessons come from published reports, not legends.

## How it really works

- Developer
- Expert

### How did one hacked CA threaten everyone? DigiNotar

DigiNotar was a Dutch CA, trusted by all major browsers and also used by the Dutch government. In 2011, attackers broke into its network and reached the systems that issued certificates. The investigation commissioned by the Dutch government, Fox-IT’s [Black Tulip report](https://zoek.officielebekendmakingen.nl/blg-191225.pdf), gives “a non-exhaustive list of 531 rogue certificates”.

**How it was noticed.** DigiNotar’s own staff found the intrusion when a routine automated check, which had stopped working, was restored and flagged certificates that did not match the company’s records. The company revoked the rogue certificates it knew about. It did not tell the browser makers. Mozilla later wrote that DigiNotar “detected and revoked some of the fraudulent certificates 6 weeks ago without notifying Mozilla” ([Mozilla Security Blog](https://blog.mozilla.org/security/2011/09/02/diginotar-removal-follow-up/)). The public found out when a Gmail user in Iran posted a warning: Chrome had rejected a certificate for a Google domain, because Chrome checked Google’s own certificates against a built-in list of expected keys (certificate pinning). From the OCSP requests for the rogue `*.google.com` certificate, the final report counted 298,140 unique IP addresses as likely victims, 95% of them in Iran, and calls that number only “a very rough approximation” of the people affected.

**What changed.** DigiNotar was breached, issued fraudulent certificates and was removed from browser trust stores. [Google](https://security.googleblog.com/2011/08/update-on-attempted-man-in-middle.html), [Mozilla](https://www.mozilla.org/en-US/security/advisories/mfsa2011-34/), [Microsoft](https://learn.microsoft.com/en-us/security-updates/securityadvisories/2011/2607712) and [Apple](https://support.apple.com/en-us/104181) all removed trust within weeks. The Dutch government revoked its own trust in DigiNotar and took over its operations, and the company was declared bankrupt.

Two lessons stayed. Any trusted CA can issue for any name, so one breach endangers everyone. And a CA that hides an incident loses trust completely. [Certificate Transparency](https://pkizerotohero.com/handbook/certificate-transparency/), standardised in June 2013 and December 2021, now makes rogue certificates visible to every domain owner, not only to companies that pin their own keys.

### What happens when a CA keeps breaking the rules? Symantec and Entrust

**Symantec: what went wrong.** In January 2017, a public post on Mozilla's security-policy forum “drew attention to a series of questionable website authentication certificates”, in the words of [Google’s later announcement](https://security.googleblog.com/2017/09/chromes-plan-to-distrust-symantec.html). The Chrome team found that Symantec “had entrusted several organizations with the ability to issue certificates without the appropriate or necessary oversight”, and called it “part of a continuing pattern of issues over the past several years”. In Chrome’s public discussion, the count grew from “reportedly 127 certificates” to “at least 30,000 certificates” ([blink-dev](https://groups.google.com/a/chromium.org/g/blink-dev/c/eUAKwjihhBs/m/El1mH8S6AwAJ)).

**What changed.** Browsers withdrew trust in phases, by issue date: Chrome 66 distrusted Symantec certificates issued before 1 June 2016, and Chrome 70 removed trust in the rest (April and October 2018). [Mozilla followed a matching plan](https://blog.mozilla.org/security/2018/03/12/distrust-symantec-tls-certificates/). Symantec sold its CA business to DigiCert, and every site owner with an affected certificate had to replace it before the deadline.

**Entrust: what went wrong.** In [Google’s words](https://security.googleblog.com/2024/06/sustaining-digital-certificate-security.html): “Over the past six years, we have observed a pattern of compliance failures, unmet improvement commitments, and the absence of tangible, measurable progress in response to publicly disclosed incident reports.” It was noticed the way most CA problems are noticed today: through public incident reports and the root programs’ scrutiny of them.

**What changed.** Chrome stopped trusting new Entrust TLS certificates (those first logged in Certificate Transparency after 11 November 2024). Certificates issued before the cut-off kept working until they expired. Apple, Mozilla and Microsoft set their own cut-off dates: 15 November 2024, 30 November 2024 and 16 April 2025.

Chrome’s cut-off uses the date of the certificate’s earliest SCT. Certificate Transparency gives a trustworthy, public issue date, which makes this kind of partial distrust possible.

*Diagram: A timeline split by a dashed cut-off date set by the root program. Certificates issued before it, marked with a tick, are trusted until they expire and are then renewed with another CA. Certificates issued after it, marked with a cross, are rejected and the site shows a trust error. The issue date is measured by the earliest SCT in Chrome. The CA's roots stay in the store for now, so older certificates still work.*

Partial distrust: the cut-off date decides which certificates keep working.

Distrust is rare, slow and public, and it always comes with a deadline. Organisations that can switch CA quickly barely notice. Those that renew by hand have to find and replace every affected certificate in a few months.

### Why did a root expiry break working sites? DST Root CA X3

**What went wrong.** Let’s Encrypt’s own root, ISRG Root X1, was young and missing from many older devices. So for years the default Let’s Encrypt chain also included a [cross-signature](https://pkizerotohero.com/handbook/cross-signing/) from IdenTrust’s older DST Root CA X3, which those devices already trusted. On 30 September 2021, DST Root CA X3 expired. Let’s Encrypt had [announced it well in advance](https://letsencrypt.org/docs/dst-root-ca-x3-expiration-september-2021/). Two kinds of client failed:

- devices that never received ISRG Root X1, because “some older devices won’t ever trust it because they don’t get software updates”;
- clients built on OpenSSL 1.0.2, even when they did trust ISRG Root X1. The OpenSSL project [explained](https://openssl-library.org/post/2021-09-13-letsencryptrootcertexpire/) that OpenSSL 1.0.2 “always prefers the untrusted chain and if that chain contains a path that leads to an expired trusted root certificate (DST Root CA X3), it will be selected for the certificate verification and the expiration will be reported.”

*Diagram: Your certificate is issued by intermediate R3. R3 leads to two certificates for the same ISRG Root X1 key: a self-signed one in newer trust stores, marked with a tick, and one cross-signed by DST Root CA X3, which leads to the expired DST root, marked with a cross. Notes: clients that trust ISRG Root X1 work; old devices without it fail; OpenSSL 1.0.x picked the expired path. The server chose which chain to send.*

One key, two certificates, two paths. Old clients could only see the expired one.

**How it was noticed.** This failure was planned and announced. It surfaced on the day as certificate errors in old phones, embedded devices and servers running old OpenSSL versions, while up-to-date browsers carried on.

**What changed.** Let’s Encrypt kept old Android devices working with a trick: a cross-signature that outlived the DST root, which works because “Android intentionally does not enforce the expiration dates of certificates used as trust anchors” ([Let’s Encrypt](https://letsencrypt.org/2020/12/21/extending-android-compatibility/)). Later, Let's Encrypt's cross-signed chain for old Android devices expired (30 September 2024). The lasting lessons: every certificate in a chain has an end date, not only yours; TLS libraries and trust stores need updates too; and old clients deserve a test before a root change.

### Why do expired certificates still cause outages?

Expiry is the most ordinary failure in PKI, and big operators are not immune. These four cases come with public post-mortems or official reports:

| Incident | What expired | How it was noticed | What changed |
| --- | --- | --- | --- |
| Azure Storage, February 2013 | Certificates for HTTPS access to storage, the same in every region | Normal monitoring, about 15 minutes after the failure | Monitor production endpoints, open an incident well before expiry, split certificates across services, regions and time ([post-mortem](https://web.archive.org/web/20220901060249/https://azure.microsoft.com/en-us/blog/details-of-the-february-22nd-2013-windows-azure-storage-disruption/)) |
| Equifax, 2017 | The certificate on a device that inspected encrypted network traffic | Inspection restarted when the certificate was renewed, and an administrator then saw signs of an intrusion | New tools for continuous network monitoring ([GAO report](https://www.gao.gov/assets/gao-18-559.pdf)) |
| Google Voice, February 2021 | The active certificate in front-end systems, after a problem updating certificate configuration | Engineers were alerted automatically | Proactive alerting for upcoming expiry, better automated rotation ([incident report](https://static.googleusercontent.com/media/www.google.com/en//appsstatus/ir/to3hkm2wc4v4xq7.pdf)) |
| Epic Games, April 2021 | A certificate for an internal DNS zone that monitoring did not cover; automatic renewal was off | Reports within minutes of expiry | Automated renewal, a manual audit of all certificates, new monitoring, a smaller blast radius per certificate ([post-mortem](https://www.epicgames.com/site/en-US/expiration-date-4-6-2021)) |

The Equifax case is the starkest. In the words of the US Government Accountability Office: “the misconfiguration was due to an expired digital certificate”, meaning “that encrypted traffic was not being inspected”, and “the attack lasted for about 76 days before it was discovered”. The expired certificate did not break the business; it silently switched off a security control.

Microsoft’s Azure post-mortem names the pattern behind most expiry outages: “a breakdown in our procedures for maintaining and monitoring these certificates was the root cause.” Epic Games found the same gap in its own monitoring: “the configuration for this internal zone was missing”.

## At work

- Developer
- IT ops

### Which CAs and roots does your service depend on?

Look at the whole chain your server sends, not only the first certificate. This is the real output for letsencrypt.org:

```bash
openssl s_client -connect letsencrypt.org:443 -servername letsencrypt.org </dev/null
```

```text
Certificate chain
 0 s:CN=letsencrypt.org
   i:C=US, O=Let's Encrypt, CN=YE2
   a:PKEY: id-ecPublicKey, 256 (bit); sigalg: ecdsa-with-SHA384
   v:NotBefore: Sep  4 14:34:32 2026 GMT; NotAfter: Dec  3 14:34:31 2026 GMT
 1 s:C=US, O=Let's Encrypt, CN=YE2
   i:C=US, O=ISRG, CN=Root YE
   a:PKEY: id-ecPublicKey, 384 (bit); sigalg: ecdsa-with-SHA384
   v:NotBefore: Sep  3 00:00:00 2025 GMT; NotAfter: Sep  2 23:59:59 2028 GMT
 2 s:C=US, O=ISRG, CN=Root YE
   i:C=US, O=Internet Security Research Group, CN=ISRG Root X2
   a:PKEY: id-ecPublicKey, 384 (bit); sigalg: ecdsa-with-SHA384
   v:NotBefore: May 13 00:00:00 2026 GMT; NotAfter: Sep  2 23:59:59 2032 GMT
 3 s:C=US, O=Internet Security Research Group, CN=ISRG Root X2
   i:C=US, O=Internet Security Research Group, CN=ISRG Root X1
   a:PKEY: id-ecPublicKey, 384 (bit); sigalg: RSA-SHA256
   v:NotBefore: May 13 00:00:00 2026 GMT; NotAfter: Sep  2 23:59:59 2032 GMT
---
```

Entries 2 and 3 are cross-signatures: Root YE signed by ISRG Root X2, and ISRG Root X2 signed by ISRG Root X1. It is the same pattern as with DST Root CA X3: a newer root reaches older trust stores through an older one. Write down the `NotAfter` of every entry, and know which CA and which root each of your services depends on. Paste a chain into the chain checker to see the path:

*Interactive tool: Chain checker (runs in the browser at [/tools/chain/](https://pkizerotohero.com/tools/chain/))*

### Are you ready for a distrust?

- **Know your CAs.** Your inventory should say which CA issued each certificate (see [Inventory, expiry and renewal](https://pkizerotohero.com/handbook/certificate-lifecycle/)). Certificate Transparency searches show what was issued for your names.
- **Follow the root programs.** Chrome, Mozilla, Apple and Microsoft announce distrust decisions publicly, months ahead.
- **Be able to switch CA in weeks.** With [ACME](https://pkizerotohero.com/handbook/acme/), switching is mostly a change of directory URL and a test run. With manual renewals, it is a project.

### Are your old clients ready for root changes?

Servers that call other services are clients too. Check the TLS library and the CA bundle on old hosts, containers, appliances and embedded devices:

```bash
openssl version
```

```text
OpenSSL 3.2.3 3 Sep 2024 (Library: OpenSSL 3.2.3 3 Sep 2024)
```

Java applications use their own trust store, and many appliances have one you can only update with firmware. When a client fails after a root change, the error pages explain the symptoms: [certificate has expired](https://pkizerotohero.com/errors/openssl-certificate-has-expired/) and [NET::ERR\_CERT\_AUTHORITY\_INVALID](https://pkizerotohero.com/errors/chrome-err-cert-authority-invalid/).

### What do the expiry post-mortems teach?

- **Monitor what clients see**, the live endpoint, not only the certificate store (Azure).
- **Cover internal names and security devices**, not only public websites (Epic Games, Equifax).
- **Automate renewal**, and give each service its own certificate so that one expiry cannot take down everything.
- **Alert on a share of the lifetime, not a fixed number of days.** Azure’s rule after its outage was an incident for any certificate with “less than 3 months until the expiration date”. With certificates limited to 47 days from 15 March 2029, that rule would fire from the day of issue.

An expired certificate looks like this in the decoder, and like [NET::ERR\_CERT\_DATE\_INVALID](https://pkizerotohero.com/errors/chrome-err-cert-date-invalid/) in Chrome:

*Interactive tool: Certificate decoder (runs in the browser at [/tools/decoder/](https://pkizerotohero.com/tools/decoder/))*

## Common mistakes

- **“One CA was hacked, so PKI is broken.”** The system reacted: browsers removed DigiNotar within weeks, and Certificate Transparency now exposes rogue certificates to everyone. One CA failing is a reason for monitoring, not for giving up on PKI.
- **“Distrust kills every certificate at once.”** Browsers usually distrust by issue date. Certificates issued before the cut-off keep working until they expire, which gives owners time to move.
- **“Only my own certificate can expire.”** Roots, intermediates and cross-signatures expire too, as DST Root CA X3 showed.
- **“Expiry outages only happen to small companies.”** Microsoft, Google and Epic Games published their own post-mortems.
- **“Our monitoring covers every certificate.”** Epic’s monitoring missed an internal zone, and Equifax’s expired certificate sat on a security device. Check what your monitoring actually covers.

## Why it matters to the business

- Leader

Why it matters to the business

The same three failures keep coming back

Most PKI incidents are not clever attacks. They are a CA that loses its browsers’ trust, a root or intermediate that expires, or a certificate nobody renewed. Each one is cheap to prevent and expensive to suffer: an outage, a security tool that silently stops working, or a forced change of supplier on a deadline. Ask three questions. Do we know every certificate we depend on and who owns it? Could we move to another CA within weeks? Do we learn about expiry from monitoring, or from customers?

## Check yourself

Check yourself

A browser announces that it will distrust a CA. Your site uses a certificate from that CA, issued last month. What usually happens?

1. The site breaks on the announcement day
2. The certificate keeps working until it expires; the next one must come from another CA
3. Nothing ever changes for existing customers

Show answer

Answer B The certificate keeps working until it expires; the next one must come from another CA

What happened

Distrust usually applies to certificates issued after a cut-off date.

Why

Root programs set cut-offs so that site owners can move to another CA without a sudden outage.

The rule

Distrust comes with a deadline. Be able to switch CA within weeks.

Check yourself

When DST Root CA X3 expired, why did some clients fail even though they trusted ISRG Root X1?

1. Their OpenSSL 1.0.2 picked the path to the expired root
2. Let's Encrypt revoked all its certificates
3. Browsers stopped supporting RSA

Show answer

Answer A Their OpenSSL 1.0.2 picked the path to the expired root

What happened

OpenSSL 1.0.2 preferred the chain the server sent, which led to the expired DST root.

Why

Path building differs between TLS libraries, and old libraries can choose an expired path.

The rule

Every certificate in a chain can expire. Keep TLS libraries and trust stores up to date.

Check yourself

At Equifax, what did an expired certificate on a traffic-inspection device cause?

1. A website outage that customers reported
2. Encrypted traffic went uninspected, so an intrusion was missed for about 76 days
3. Browsers showed a warning on the company's site

Show answer

Answer B Encrypted traffic went uninspected, so an intrusion was missed for about 76 days

What happened

The device stopped inspecting encrypted traffic while its certificate was expired.

Why

A security device with an expired certificate can fail silently, without any visible outage.

The rule

Monitor certificates on security devices and internal systems, not only on public websites.

## Go deeper

- [Black Tulip: the DigiNotar investigation](https://zoek.officielebekendmakingen.nl/blg-191225.pdf): the official report on how the breach happened and was found.
- [Chrome’s plan to distrust Symantec](https://security.googleblog.com/2017/09/chromes-plan-to-distrust-symantec.html): the reasons and the phased timeline.
- [Entrust certificate distrust](https://security.googleblog.com/2024/06/sustaining-digital-certificate-security.html): Chrome’s decision and how the SCT-based cut-off works.
- [Let’s Encrypt: DST Root CA X3 expiration](https://letsencrypt.org/docs/dst-root-ca-x3-expiration-september-2021/): which clients were affected and why.
- [OpenSSL: the Let’s Encrypt root expiry](https://openssl-library.org/post/2021-09-13-letsencryptrootcertexpire/): why OpenSSL 1.0.2 chose the expired path, and the workarounds.
- [GAO-18-559](https://www.gao.gov/assets/gao-18-559.pdf): the US Government Accountability Office report on the Equifax breach.

The crash course covers these failures in [lesson 8, When trust breaks](https://pkizerotohero.com/course/when-trust-breaks/).

**Early edition.** Checked against the standards; an independent expert review is in progress. Found a mistake? [Tell us](https://pkizerotohero.com/contact/?topic=content-error&page=/handbook/famous-failures/).

Last reviewed 20 October 2026Edition 0.9

## Keep going

Crash course: [Lesson 8: When trust breaks](https://pkizerotohero.com/course/when-trust-breaks/)

### Error pages for this chapter

- [verify error:num=10:certificate has expired](https://pkizerotohero.com/errors/openssl-certificate-has-expired/)
- [NET::ERR\_CERT\_DATE\_INVALID](https://pkizerotohero.com/errors/chrome-err-cert-date-invalid/)
- [NET::ERR\_CERT\_AUTHORITY\_INVALID](https://pkizerotohero.com/errors/chrome-err-cert-authority-invalid/)

### Tools

- [Chain checker](https://pkizerotohero.com/tools/chain/): Paste a chain and see which link is missing, expired or out of order.
- [Certificate decoder](https://pkizerotohero.com/tools/decoder/): Paste a PEM certificate and read every field in plain words.

### Topics

- [Chains and trust stores](https://pkizerotohero.com/topics/chains-and-trust/)
- [Revocation and transparency](https://pkizerotohero.com/topics/revocation-and-transparency/)