Menu

Chains and trust stores

About 10 minutesCrash course

In one sentence

A client trusts a certificate only if it can build a chain of signatures to a root already in its own trust store, so servers must send every intermediate.

The situation

A new partner API works in every browser you try. The nightly import job, which uses curl, fails with SSL certificate problem: unable to get local issuer certificate. The developer says the certificate is fine, because the browser shows no warning.

Who is right?

What is going on

Trust stores: who do we trust?

Your device trusts a CA because that CA’s root certificate is in its trust store: a list of roots shipped with the operating system, the browser or the programming runtime. Root programs run by Apple, Google, Microsoft and Mozilla decide which roots go in, after audits and policy reviews, and take them out again when a CA fails (lesson 8).

A root certificate is self-signed: it signs itself. That earns no trust, because anyone can make a self-signed certificate in a second. A root is trusted only because it is already in the trust store.

Different software uses different stores: the operating system’s store, Firefox’s own, Java’s cacerts, the certifi bundle used by many Python libraries, and the list built into Node.js. The same server can be trusted by one client and rejected by another.

Chains: from the leaf to a root

Roots rarely sign website certificates directly. Their keys are usually kept offline, so a root signs an intermediate CA certificate, and the intermediate signs the website’s leaf certificate.

A certificate chain from leaf to a trusted root Leaf: api.example.com sent by the server Intermediate CA sent by the server Root CA ✓ already on the client issued by issued by Trust store on the client device, chosen by root programs If the intermediate is missing: Browsers: ✓ may fetch it and carry on curl, Java, Python: ✗ fail
The server sends the leaf and the intermediates. The client supplies the root from its own trust store.

To accept the leaf, the client builds a certificate chain: leaf, then each intermediate, ending at a root in its own store. It checks every signature along the way, every validity period, and that each issuer is allowed to act as a CA.

The server’s job is to send its leaf plus every intermediate. It does not need to send the root: the client must already have it, and a root that arrives from the server is not trusted just because it was sent.

Why the browser fooled the developer

Browsers often repair a missing intermediate silently. They may fetch it from the address listed in the leaf’s Authority Information Access extension, or use a copy they already have. Most non-browser clients, such as curl built with OpenSSL, Java and Python, do not, by default. So “it works in my browser” proves little about the chain. In the situation, the job is right to fail, and the fix belongs on the server.

Two roads to the same root

A new root takes years to reach every device. Meanwhile, a CA can have its new root or intermediate signed by an older, widely trusted root as well. This cross-signing gives old devices one valid road and new devices another. When the old root expires, devices that know only the old road start to fail. This happened to some older devices when DST Root CA X3, the older root that had cross-signed Let’s Encrypt, expired on 30 September 2021.

Your own CA

Certificates from a private, internal CA work only on devices where your root has been installed. The fix for a failing client is to install that root through a managed process, never to switch off certificate checking.

Try it

Try it: Chain checkerOpen the full tool

Runs in your browser. Nothing you paste is sent anywhere.

Every certificate the server sends, leaf first. The output of openssl s_client -showcerts -connect example.com:443 works as it is: we ignore the text around the certificates.

This server sends its leaf but not the intermediate. Find where the chain breaks between the leaf and a trusted root, and note which kind of client would still connect and which would fail.

Check

Check yourself

In the situation, the API works in browsers but curl fails with 'unable to get local issuer certificate'. Who is right?

  1. The developer: the certificate is fine, so curl is broken
  2. Nobody: certificate checking should be turned off for this host
  3. curl: the server is not sending its intermediate, and browsers were hiding the gap
Show answer

Answer C curl: the server is not sending its intermediate, and browsers were hiding the gap

Check yourself

Why is a root CA certificate trusted?

  1. Because it is already in the trust store of the device or application
  2. Because it is self-signed
  3. Because the server sends it at the end of the chain
Show answer

Answer A Because it is already in the trust store of the device or application

Check yourself

An internal app uses a certificate from your company's private CA. It works on managed laptops but fails on a contractor's laptop. What is the right fix?

  1. Tell the contractor to click through the warning
  2. Install the company root in the contractor's trust store through an approved process, or provide a managed device
  3. Make the server send the root certificate as well
Show answer

Answer B Install the company root in the contractor's trust store through an approved process, or provide a managed device

Takeaways

  • Trust starts in the trust store: a root is trusted only because it is already there, and different software has different stores.
  • The client builds a chain from the leaf to a trusted root; the server must send every intermediate.
  • “Works in my browser” proves little. Test with curl or your real client, and never switch off verification.

Go further

In the gameLevel 12: Who do we trust?Registry of Roots, Keystone. In development

Early edition. Checked against the standards; an independent expert review is in progress. Found a mistake? Tell us.

Last reviewed Edition 0.9

Go deeper in the handbook