Two roads to the same root: cross-signing and root expiry

Root certificates have end dates too, usually 20 or 25 years away. Long before a root expires, its CA creates a new one. That raises a hard problem: a new root is useless until it is in people’s trust stores (chapter 12), and getting it there takes years. Browsers update quickly, but old phones, smart TVs, payment terminals and forgotten servers may never receive an update.

Cross-signing bridges that gap. The established root signs a certificate for the new CA’s key. Now that one key has two certificates:

• a self-signed one, which new trust stores contain;
• a cross-signed one, issued by the old root, which old trust stores can follow.

Every certificate signed by that key can be checked along either road. A new phone reaches the new root directly. An old phone walks one step further, to the old root it already trusts. Both arrive at a trust anchor, so both accept the certificate.

There is a catch. The cross-signed certificate has its own end date, and so does the old root. When they expire, the old road closes. Devices that never received the new root start failing, even though nothing changed on the server. That is what happened to many older devices when the root that once cross-signed Let’s Encrypt expired.

A second catch: some apps hard-code which keys or certificates they accept. This is called pinning. Pinning is meant to block fake certificates, but it also blocks real ones when a CA rotates its keys or roots, unless it was planned with care.

What exactly is a cross-signed certificate?

It is a CA certificate with the same subject name and the same public key as another certificate, but a different issuer and therefore a different signature. Because the key is the same, the Subject Key Identifier is the same too (chapter 13). Any certificate signed by that key has an Authority Key Identifier that matches both versions, so a client can link it to either one.

Let’s Encrypt’s live chain shows this. Here are two certificates for “ISRG Root X2”, one fetched from letsencrypt.org’s chain in October 2026 and one from a trust store:

subject=C=US, O=Internet Security Research Group, CN=ISRG Root X2
issuer=C=US, O=Internet Security Research Group, CN=ISRG Root X1
notAfter=Sep  2 23:59:59 2032 GMT
SKI 7C:42:96:AE:DE:4B:48:3B:FA:92:F8:9E:8C:CF:6D:8B:A9:72:37:95

subject=C=US, O=Internet Security Research Group, CN=ISRG Root X2
issuer=C=US, O=Internet Security Research Group, CN=ISRG Root X2
notAfter=Sep 17 16:00:00 2040 GMT
SKI 7C:42:96:AE:DE:4B:48:3B:FA:92:F8:9E:8C:CF:6D:8B:A9:72:37:95

Same name, same key identifier, different issuers and different end dates. The first is ISRG Root X2 cross-signed by ISRG Root X1. The second is the self-signed ISRG Root X2 that newer trust stores contain.

How does a client choose a path?

A client collects candidate certificates from the server’s chain, its trust store, its cache and sometimes AIA downloads. Then it searches for a path from the leaf to any trust anchor (RFC 4158 describes the techniques). Two clients with different stores can therefore build different paths from the same server chain, and both can be right.

letsencrypt.org sends four certificates: its leaf, the intermediate YE2, “Root YE” cross-signed by ISRG Root X2, and ISRG Root X2 cross-signed by ISRG Root X1. A modern trust store holds ISRG Root X2 itself, so OpenSSL stops there:

depth=0: CN=letsencrypt.org (untrusted)
depth=1: C=US, O=Let's Encrypt, CN=YE2 (untrusted)
depth=2: C=US, O=ISRG, CN=Root YE (untrusted)
depth=3: C=US, O=Internet Security Research Group, CN=ISRG Root X2

When we gave OpenSSL a trust store that holds only ISRG Root X1, like an older device, it used the cross-signed X2 certificate that the server sent and walked one step further:

depth=0: CN=letsencrypt.org (untrusted)
depth=1: C=US, O=Let's Encrypt, CN=YE2 (untrusted)
depth=2: C=US, O=ISRG, CN=Root YE (untrusted)
depth=3: C=US, O=Internet Security Research Group, CN=ISRG Root X2 (untrusted)
depth=4: C=US, O=Internet Security Research Group, CN=ISRG Root X1

Swipe sideways to see the whole diagram.

Not every client is this flexible. Older path builders took the first issuer they found and gave up if that path failed, even when another path would have succeeded.

Why do roots expire, and what happens then?

A root certificate has a notAfter date like any other certificate, and a cross-signed certificate can never usefully outlive its issuer for clients that check the issuer’s dates. When a root expires, clients that check it reject every path that ends there. Whether a certificate keeps working then depends on whether the client has, and finds, another valid path.

Platforms differ in one detail that matters here. Let’s Encrypt notes that “Android intentionally does not enforce the expiration dates of certificates used as trust anchors” (Let’s Encrypt: extending Android device compatibility). Many other clients, OpenSSL among them, do enforce them.

What happened when DST Root CA X3 expired?

When Let’s Encrypt started, its own root, ISRG Root X1, was in no trust store. IdenTrust’s established root, DST Root CA X3, cross-signed Let’s Encrypt so that its certificates worked everywhere from day one. Over the following years ISRG Root X1 reached most trust stores, but not old devices that no longer received updates.

On 30 September 2021, DST Root CA X3, the older root that had cross-signed Let's Encrypt, expired, so devices that did not trust ISRG Root X1 began rejecting Let's Encrypt certificates. Three groups of clients behaved differently (Let’s Encrypt):

• Clients that trusted ISRG Root X1 and built paths well kept working: they ignored the expired road.
• Old Android devices kept working because of “a special cross-sign from DST Root CA X3” that “extends past that root’s expiration”, which Android accepted because it does not enforce trust anchor expiry.
• Other old devices without ISRG Root X1, and clients using OpenSSL 1.0.x, which Let’s Encrypt notes had “a quirk in certificate verification” with this chain, started to fail.

That Android workaround had an end date of its own: Let's Encrypt's cross-signed chain for old Android devices expired on 30 September 2024 (Let’s Encrypt: cross-sign expiration).

The pattern is repeating. Let’s Encrypt’s newer roots are cross-signed in the same way: Root YE by ISRG Root X2, as the chains above show, and Root YR by ISRG Root X1 (Let’s Encrypt: chains of trust). Let's Encrypt lists both new roots as not yet included in the root programs' trust stores, so for now the cross-signs carry the trust.

What is pinning, and why does it break rotation?

An app that pins accepts a server only if one of the certificates in the path has a specific public key, usually identified by the SHA-256 hash of its SubjectPublicKeyInfo. Pinning narrows trust from “any CA in the store” to “these keys only”. The price is that every legitimate key change must be planned:

• Pin the leaf key, and renewing with a fresh key breaks the app.
• Pin an intermediate, and the CA’s routine switch to a new intermediate breaks it.
• Pin a root, and a migration to a new root or a different CA breaks it.

Pinning to a key, not a certificate, survives cross-signing, because both certificates hold the same key. The two ISRG Root X2 certificates above produce the same pin:

SPKI pin: diGVwiVYbubAI3RW4hB9xU8e/CH2GnkuvVFZE8zmgzI=
SPKI pin: diGVwiVYbubAI3RW4hB9xU8e/CH2GnkuvVFZE8zmgzI=

Browsers have dropped HTTP Public Key Pinning (RFC 7469), because one mistake could lock visitors out of a site for months. Mobile apps still pin. Android’s own guidance is to “always include a backup key”, and it allows an expiration date for pins, while warning that expiry “may enable attackers to bypass your pinned certificates” (Android Network Security Configuration).

How do I see which roads my chain offers?

Save the chain your server sends, split it into single certificates, and verify the leaf against different trust stores. These are the commands behind the two path listings above:

openssl s_client -connect letsencrypt.org:443 -servername letsencrypt.org -showcerts </dev/null 2>/dev/null \
  | sed -n '/-----BEGIN CERTIFICATE-----/,/-----END CERTIFICATE-----/p' > le-chain.pem
csplit -s -z -f lecert- le-chain.pem '/-----BEGIN CERTIFICATE-----/' '{*}'
cat lecert-01 lecert-02 lecert-03 > le-untrusted.pem

# 1. With the default trust store
openssl verify -show_chain -untrusted le-untrusted.pem lecert-00
# 2. With a store that holds only ISRG Root X1, like an older device
openssl verify -show_chain -CAfile isrg-x1.pem -untrusted le-untrusted.pem lecert-00

Both runs print lecert-00: OK, followed by the four-step and five-step paths. Build a small trust store file for each kind of old client you still support, such as an old Java version, an embedded device or an old phone, and run the second command against it before you change chains or CAs.

How do I choose a chain?

An ACME server may offer more than one chain for the same certificate. RFC 8555 lets it add “alternate” links, each of which “SHOULD express an alternative certificate chain starting with the same end-entity certificate” (section 7.4.2). ACME clients let you choose, for example certbot’s --preferred-chain option, which picks the chain whose top certificate was issued by the name you give.

What to serve:

• the leaf and every intermediate, as always (chapter 13);
• the cross-signed certificates that your oldest supported clients need, and no more;
• never an expired cross-signed certificate, because some clients will try that road and fail.

How do I keep trust stores current?

The other half of root migration happens on the clients:

• install operating-system updates, which carry root store updates;
• update Java, because each JDK ships its own cacerts;
• update certifi in Python projects and rebuild container images, whose CA bundles are frozen at build time;
• upgrade Node.js, whose bundled roots are “fixed at release time”;
• for devices in the field, make sure firmware updates can deliver new roots, years before the old ones expire.

If I must pin, how do I do it safely?

Pin public keys, not certificates, and always more than one. Compute a pin from any certificate like this:

openssl x509 -in isrg-x2-self.pem -noout -pubkey | openssl pkey -pubin -outform der \
  | openssl dgst -sha256 -binary | openssl enc -base64

diGVwiVYbubAI3RW4hB9xU8e/CH2GnkuvVFZE8zmgzI=

Then:

• pin at least one backup key that is not in use yet, ideally from a different CA or root;
• set an expiry date on the pins, and accept the trade-off Android describes;
• keep a way to update the pins without shipping a new app, or be ready to ship one quickly;
• write down which pins exist, so that the team that renews certificates knows about them.

If a public website is all you protect, consider not pinning at all. Certificate Transparency monitoring (chapter 17) catches mis-issued certificates without the risk of locking out your own users.

What do failures look like?

When a client follows a road that has closed, it reports an expired certificate somewhere in the chain, for example verify error:num=10:certificate has expired at a depth above 0. When it cannot find any road to a root it trusts, it reports the familiar unable to get local issuer certificate or, in Java, PKIX path building failed. If only old devices fail, suspect the root, not the server.

Try it: Chain checkerOpen the full tool

Runs in your browser. Nothing you paste is sent anywhere.

Paste the chain

Try a sampleChoose a sample…A complete chain: leaf, intermediate and rootA server that forgets its intermediateA chain in the wrong orderA self-signed root CA

-----BEGIN CERTIFICATE----- MIIC1zCCAnygAwIBAgIQWTTOzqWiPshGdQhNb6kmNjAKBggqhkjOPQQDAjB+MQsw CQYDVQQGEwJCRTEZMBcGA1UEChMQUEtJIFplcm8gdG8gSGVybzEnMCUGA1UECxMe U2FtcGxlcyAtIG5vdCBwdWJsaWNseSB0cnVzdGVkMSswKQYDVQQDEyJQS0kgWmVy byB0byBIZXJvIFNhbXBsZSBJc3N1aW5nIENBMB4XDTI2MTAwMTAwMDAwMFoXDTM2 MTAwMTAwMDAwMFowGjEYMBYGA1UEAxMPd3d3LmV4YW1wbGUuY29tMFkwEwYHKoZI zj0CAQYIKoZIzj0DAQcDQgAEw4og6Y/SOr+WrqSv+ng7tOBEsc5N3A3yZRk+9hsR XzPQvB9672aHZHTYFNlSY93RWVcTRieE+8XnIbWIYfwxIqOCAT4wggE6MAwGA1Ud EwEB/wQCMAAwDgYDVR0PAQH/BAQDAgeAMBMGA1UdJQQMMAoGCCsGAQUFBwMBMCcG A1UdEQQgMB6CD3d3dy5leGFtcGxlLmNvbYILZXhhbXBsZS5jb20wHQYDVR0OBBYE FALMipe4vcXJTpwO926S15Y/bobzMB8GA1UdIwQYMBaAFN+ph37v40SN23r6Fg/z OvSNjJqeMEgGCCsGAQUFBwEBBDwwOjA4BggrBgEFBQcwAoYsaHR0cDovL3BraS5l eGFtcGxlLmNvbS9zYW1wbGUtaXNzdWluZy1jYS5jcnQwPQYDVR0fBDYwNDAyoDCg LoYsaHR0cDovL3BraS5leGFtcGxlLmNvbS9zYW1wbGUtaXNzdWluZy1jYS5jcmww EwYDVR0gBAwwCjAIBgZngQwBAgEwCgYIKoZIzj0EAwIDSQAwRgIhAM5dJUPhL+qV artvWtQinfnykhVflJTjMXe8M6PnqQfaAiEAhOLGj1hSEkNxQoMrYNXRaIe69nq6 2unibVFSVw1AC80= -----END CERTIFICATE----- -----BEGIN CERTIFICATE----- MIIE2TCCAsGgAwIBAgIQPh9PeiGf4STnEQriHxExjDANBgkqhkiG9w0BAQsFADB7 MQswCQYDVQQGEwJCRTEZMBcGA1UEChMQUEtJIFplcm8gdG8gSGVybzEnMCUGA1UE CxMeU2FtcGxlcyAtIG5vdCBwdWJsaWNseSB0cnVzdGVkMSgwJgYDVQQDEx9QS0kg WmVybyB0byBIZXJvIFNhbXBsZSBSb290IENBMB4XDTI1MDEwMTAwMDAwMFoXDTQw MDEwMTAwMDAwMFowfjELMAkGA1UEBhMCQkUxGTAXBgNVBAoTEFBLSSBaZXJvIHRv IEhlcm8xJzAlBgNVBAsTHlNhbXBsZXMgLSBub3QgcHVibGljbHkgdHJ1c3RlZDEr MCkGA1UEAxMiUEtJIFplcm8gdG8gSGVybyBTYW1wbGUgSXNzdWluZyBDQTBZMBMG ByqGSM49AgEGCCqGSM49AwEHA0IABBFqObJ/j93aHrAuebO5+KWnj0xRFzUS8/Jh Idj6jR/p4zqaNT8+gOqxZ+zhpNO/weIpz/nzXjQ7RJS+EebYRNijggEfMIIBGzAS BgNVHRMBAf8ECDAGAQH/AgEAMA4GA1UdDwEB/wQEAwIBhjAdBgNVHSUEFjAUBggr BgEFBQcDAQYIKwYBBQUHAwIwHQYDVR0OBBYEFN+ph37v40SN23r6Fg/zOvSNjJqe MB8GA1UdIwQYMBaAFIVx1Hy1TuED0pMkxEcdM+DWWPGiMEUGCCsGAQUFBwEBBDkw NzA1BggrBgEFBQcwAoYpaHR0cDovL3BraS5leGFtcGxlLmNvbS9zYW1wbGUtcm9v dC1jYS5jcnQwOgYDVR0fBDMwMTAvoC2gK4YpaHR0cDovL3BraS5leGFtcGxlLmNv bS9zYW1wbGUtcm9vdC1jYS5jcmwwEwYDVR0gBAwwCjAIBgZngQwBAgEwDQYJKoZI hvcNAQELBQADggIBAFa8c1kEzTTPW2aPDRPb3+OQc8GTgwpZV0pWNR3e6X8mJi7A D/RGrz4t354IwGzTts+tX23sY3fORKST3/BkqeHZwAZHvGNWi0+Co342eo2BySxo bcyy31AJBSSHOC2b7KJ11coMX++bvaHgtwOloy6DnHRDYIFwWRHg91wo1MKJGNVb OQH0vmqn6YPDLZ4BYL8VHOVtlRddBwd6bKyyp3Lw0MEMgfmg87Yd/U0yRZFaLKNb b5vgDg7n+J1ozFwvumQlN6WVEyPsQdvI73K5bgMHkUeeUQ6j9KOeDMZMjbqQSC8x 1khcyT/qhqZLDJIHvmWS11FcYaLQUuvQiAzQieTNIuOM8WJw95LlxYSNIkZeOuLf f+JboiuT1cmqmvdbngAXtBIWRJ+Byeqdq2Z0d8Xfln6gWhHmm93+DO5uTaTxBwTr DayFTdTFxBZydXs48S+I/rVQvQtwkRh3rOcZpwTMsGzfoiCsOhEaiBOFk0C5FEKi um55SDXCMR6yXDhWqFPBIdle4izj5g2YmuZ3YD4G3PlcX5okW7aKTNU3RGWEOMk3 Ob9bkPnCsY7Ei4OqDqTBt/8HH/2kA4om4RzKGLw5Zc/Y+KhpbFhT52gtL7WDEEVZ NqFHgi6fdHgsczFNpCKhC8vM+mAM/gBiVR8hYIpt5lPFyXpFJO5Wn5BcEkv3 -----END CERTIFICATE----- -----BEGIN CERTIFICATE----- MIIFxDCCA6ygAwIBAgIQV3/H8mpwNp5raCafwOtSezANBgkqhkiG9w0BAQsFADB7 MQswCQYDVQQGEwJCRTEZMBcGA1UEChMQUEtJIFplcm8gdG8gSGVybzEnMCUGA1UE CxMeU2FtcGxlcyAtIG5vdCBwdWJsaWNseSB0cnVzdGVkMSgwJgYDVQQDEx9QS0kg WmVybyB0byBIZXJvIFNhbXBsZSBSb290IENBMCAXDTI1MDEwMTAwMDAwMFoYDzIw NTAwMTAxMDAwMDAwWjB7MQswCQYDVQQGEwJCRTEZMBcGA1UEChMQUEtJIFplcm8g dG8gSGVybzEnMCUGA1UECxMeU2FtcGxlcyAtIG5vdCBwdWJsaWNseSB0cnVzdGVk MSgwJgYDVQQDEx9QS0kgWmVybyB0byBIZXJvIFNhbXBsZSBSb290IENBMIICIjAN BgkqhkiG9w0BAQEFAAOCAg8AMIICCgKCAgEAnYTGHkYMca54QPkZcZENin3BHB3t uKdHkDZeKZvJHWqpfXpcZfusLvtJFeTrwWlEDBJuV20vmhaLeV29QeYKaIkm1Ul2 qCkrLUpWn/W9CPosfVqMd1nOrd+Ka9iPSeKlW3YX1P5WqQEjieJhBzT9comdj8qN /PynceTDcLa2mgYw7QhU8trBoLzhYqIJFfoUBk+7lahae8b9G5wKCbDicRM8/ATt 0dO9jCxi6RxJpkKl24ULgNOuFP9gG4nubWA8+BaMeM0LU64wNciFofgM2g/3iHHn zmcoe1z80C7WN+GuzNkWkN8D/tcqrbua0O5fREKKw6PyyxwQsjeMghpxLb0h1KeA ihpQMVgpWbhRkkpvF5odkF9vTJP89Z2y+8cgnUMyTpXATrt8KgC5LdWo5mlTNXC/ MvbCUb1Vz9bvcEx2y29tUeLIAUmyBzhF7zgMUPAOtDbLLCuxgR78T2TWaXB/8WBT XL44Wk5rOe7U+KuBr/t12AcZjaq0AGiNsvfd1P+IFbRivElegpdrM+S3arjOjptP M9sUl9Hc+cszaTpEVcQFFxcIyvq5HVjTlZr4vLKdqlhSkQB9hZUFuWNpnUZEYEnA 1vn/GkHNTJ6pv8XxdNXkFeRaUv3ajCn7CQ3UP/EXJqzBw5fl6aa9lqKTS5kF9rgw EIhwNaSd2O557KUCAwEAAaNCMEAwDwYDVR0TAQH/BAUwAwEB/zAOBgNVHQ8BAf8E BAMCAQYwHQYDVR0OBBYEFIVx1Hy1TuED0pMkxEcdM+DWWPGiMA0GCSqGSIb3DQEB CwUAA4ICAQBMdaOOM84aoItvumQ/sNzOhXvmLwboTb3oe67wda9TvPMSf2aP3JtP UWsR/9UbZgNE9M41p2+XZlVpsGP1BLEvi5ZwfMWmR876Mw1Zkt8ouY3DWN08egnm IdV2vKwDHhDE93IQ12fjNc8iuMCasoYiXTsmXx3R/+PkNbeBAupZEk6EwTr75PCY IJ2cheL7jQ06xeCCZv815rG+63jGk3tys7A+nUXKqOp9oN/dkEhi+YWiHG9UJ2w/ BDZCFUUoJBia7C3+hcXZAVxtWwTy0A5T/plEsqGcdfCMFA4cOFstthyS/Wnnuw8y cpSiXdUAQbLkJ+CGavr6MXVmDicmnV9p3Pv9EbpGPV8UlxwvoQMf+ajEE4EAaJkN Gc85wOC7Y7wwWkBnzjiyLztqRqOFdHEdM13V+2Gw2545Jk8o2gRizBQ6Ey6rlt2u YdYFqFFQhe2H+VqvJiLKefkN1mVQhvhfAvuoxZWIsDtyJWGs12L06xMj3wG98kne mUJjPzze3swZU39eiaC1MBs6U0oLZE6ZqubmjJd/ppmGyK+7o2SMi0juouvw6rcA J9eUJQLNpiVKmiwzKvPejASpunK5B5JO3Y7NAb4SVU22ItR73bBiRAugPwVajlBN RGcjMkDSiW07/dhN6GgjsNAzXHcTR7/nbILOAWbhBstNFK1mgejlbQ== -----END CERTIFICATE-----

Every certificate the server sends, leaf first. The output of openssl s_client -showcerts -connect example.com:443 works as it is: we ignore the text around the certificates.

Check the chainClearChoose a file

Paste your server’s chain to see each certificate’s issuer and end date. Look for cross-signed certificates near their end date, and for any that have already expired.

Common mistakes

• “A certificate has exactly one chain.” A certificate below a cross-signed CA has several valid paths. Each client builds its own.
• “When a root expires, everything under it stops working at once.” Clients that have the newer root, or an alternative path, keep working. Old clients without one fail.
• “The client uses the chain exactly as the server sends it.” Clients build their own path from what they receive and what they already hold, and may ignore some certificates.
• “Cross-signing means two CAs vouch, so trust is doubled.” Each client trusts the certificate through one path only, ending at whichever root it holds.
• “Old devices will get the new root automatically.” Many never update. Plan for them or decide to stop supporting them.
• “Pinning the leaf key is harmless.” It breaks the app at the first renewal with a new key, unless a backup pin was shipped in advance.

Check yourself

Check yourself

A server sends a chain that ends in a cross-signed certificate issued by an old root. A new phone has only the new root in its trust store. What happens?

1. It fails, because the chain points to the old root
2. It builds a path to the new root it trusts and accepts the certificate
3. It asks the user which root to use

Show answer

Answer B It builds a path to the new root it trusts and accepts the certificate

What happened

The phone accepts the certificate through the new root.

Why

The cross-signed and self-signed certificates share one key, so the phone can link the chain to the new root it already holds.

The rule

Clients build their own path to whichever trusted root they hold.

Check yourself

After an old root that once cross-signed a CA expires, which clients are most likely to fail?

1. Devices with recently updated trust stores
2. Old devices that never received the CA's new root
3. All devices at the same moment

Show answer

Answer B Old devices that never received the CA's new root

What happened

Old devices without the new root fail.

Why

Their only road to a trusted root ran through the expired certificate. Updated devices have a direct path.

The rule

Root expiry hits the clients that cannot update. Know which ones you support.

Check yourself

Your mobile app pins the SHA-256 hash of your server's leaf public key. At renewal, your ACME client generates a new key. What happens?

1. Nothing, because the certificate is still valid
2. The app rejects your server until the app is updated
3. The CA updates the pin automatically

Show answer

Answer B The app rejects your server until the app is updated

What happened

The app refuses the connection.

Why

The new certificate holds a new key whose hash does not match the pin built into the app.

The rule

Pin keys you control the rotation of, ship a backup pin, and plan every key change.

Go deeper

• Let’s Encrypt: Chains of trust: the current roots, intermediates and cross-signs, a live example of everything in this chapter.
• Let’s Encrypt: DST Root CA X3 expiration: what changed and who was affected.
• Let’s Encrypt: extending Android device compatibility: the cross-sign that outlived its own root.
• RFC 4158: certification path building when several paths exist.
• RFC 8555, section 7.4.2: alternate chains in ACME.
• Android Network Security Configuration: pinning with backup keys and expiry.
• RFC 7469: HTTP Public Key Pinning, now history, and a lesson in why pinning needs care.

In the gameLevel 14: Two roads to the same rootCrossroads Inn, Keystone. In development

The crash course covers cross-signing in lesson 6, Chains and trust stores.