Guarding the keys: HSMs and key ceremonies

Everything in PKI depends on private keys staying private. If a web server’s key leaks, one site can be impersonated. If a CA’s key leaks, every certificate it ever signed becomes doubtful, and an attacker can sign new ones for any name. DigiNotar showed how far that reaches.

So the most important keys get special treatment:

• They live in a hardware security module (HSM): a hardened device that creates keys inside itself and uses them there. Software sends the HSM data to sign and gets a signature back. The key itself never leaves in readable form.
• They are created in a key ceremony: a scripted, witnessed and recorded event, planned in advance and documented afterwards.
• No single person can use, copy or restore them alone. This is separation of duties: tasks are split so that misuse needs several people to agree.
• They are backed up carefully, and only the right kinds of keys are ever handed to a recovery service, a practice called key escrow.
• They retire on schedule. The period during which a key may be used is its cryptoperiod.

None of this is only for public CAs. If your organisation runs its own CA (see A CA of your own), signs software or protects encrypted backups, the same ideas apply at a smaller scale.

What does an HSM protect against, and what not?

An HSM generates a key pair inside its protected boundary and marks the private key as non-exportable. Applications reach it through standard interfaces: PKCS #11 on most platforms, Key Storage Providers in Windows, and provider APIs in Java. Good HSMs add:

• authentication of operators, often with smart cards and a quorum, such as three of five cards;
• tamper evidence or tamper response, which wipes keys if the device is opened;
• an audit log of every use of every key.

An HSM protects the key from being copied. It does not stop misuse by a caller it trusts. An attacker who controls the CA server can ask the HSM to sign while they have that control. That is why CA software, the network path to the HSM and its logs need the same care as the HSM itself.

What do the FIPS 140-3 levels mean?

HSMs are tested against FIPS 140-3, Security Requirements for Cryptographic Modules. NIST and the Canadian Centre for Cyber Security run the validation programme, the Cryptographic Module Validation Program (CMVP); accredited laboratories do the testing. The standard “provides four increasing, qualitative levels of security” and takes its detailed requirements from ISO/IEC 19790. NIST’s earlier FIPS 140-2 describes the levels in plain words:

FIPS 140-3 keeps the same four levels, but its detailed requirements come from ISO/IEC 19790 and differ in places. For example, level 4 now also requires multi-factor authentication of operators (ISO/IEC 19790 requirement AS04.59, quoted in NIST’s FIPS 140-3 Implementation Guidance). Read the table as the general idea, not the test criteria.

Public CAs must protect their keys in a device validated to at least FIPS 140-2 level 3, FIPS 140-3 level 3, or an equivalent Common Criteria evaluation (EAL 4 or higher), according to Baseline Requirements section 6.2.7. Code signing has its own rule: code-signing keys must be kept in certified hardware (at least FIPS 140-2 level 2 or Common Criteria EAL 4+) (since 1 June 2023, Code Signing Baseline Requirements section 6.2.7.4).

Validation is tied to a specific module, firmware version and certificate number, so check the CMVP list, not the brochure. All FIPS 140-2 validations have moved to the CMVP historical list (after 21 September 2026). Modules on the historical list may still be used in existing systems, but FIPS 140-3 says the historical list “should not be used for procurement decisions”.

What about cloud KMS and cloud HSM?

The major clouds offer two kinds of service. A key management service (KMS) is shared: you call an API, and the provider runs the HSMs behind it. A cloud HSM gives you dedicated HSM capacity that you administer yourself. Their own documentation makes these claims:

These are the vendors’ own words as read in October 2026. Some pages still cite FIPS 140-2 validations, which are now on the historical list, so check the CMVP certificate behind each claim.

In the cloud, the HSM protects the key from being copied, but your identity and access policies decide who may ask it to sign. Review those policies as strictly as you would guard a smart card, and send every signing event to your logs.

Swipe sideways to see the whole diagram.

What happens in a key ceremony?

For root keys, and for intermediates operated by a different organisation, the Baseline Requirements (section 6.1.1.1) say the CA shall:

1. “prepare and follow a Key Generation Script”;
2. “have a Qualified Auditor witness the CA Key Pair generation process or record a video of the entire CA Key Pair generation process”; and
3. have the auditor issue a report that the CA followed its ceremony and controlled the key’s integrity and confidentiality.

In all cases the key must be generated “in a physically secured environment”, by “personnel in Trusted Roles under the principles of multiple person control and split knowledge”, and the CA must log the activity.

A typical ceremony has these roles:

• an administrator, who reads the script aloud and types the commands;
• key custodians, who each hold one share of the authority to unlock the HSM, such as one smart card and PIN out of five, with three needed;
• a witness, often the auditor, who checks every step and signs the record;
• someone who records it, on video and in a written log with times and signatures.

The script is written and reviewed weeks before. On the day, the team checks the serial numbers on tamper-evident bags, starts an offline machine, initialises the HSM, generates the key, creates the root certificate, signs the intermediate CSRs that were prepared in advance, makes and verifies the backups, and seals everything again. Every step gets a time and initials. The report records the SHA-256 fingerprint of each certificate created, so that anyone can later check they have the right one.

Who may do what?

Separation of duties means no single person can complete a sensitive task alone. The Baseline Requirements (section 5.2.2) say: “The CA Private Key SHALL be backed up, stored, and recovered only by personnel in Trusted Roles using, at least, dual control in a physically secured environment.” In practice:

• the people who approve certificate requests are not the people who operate the HSM;
• the people who operate systems are not the people who review the logs;
• unlocking the HSM needs a quorum, so one absent, careless or dishonest person can neither block nor misuse the key.

Keep the quorum realistic. If three of five custodians are needed and two leave the company, you are one illness away from being locked out of your own CA.

Backup or escrow?

Losing a key and copying a key are opposite risks, and different keys need different answers. NIST SP 800-57 Part 1, Revision 5, gives the rules (section 8.2.2.1, Table 7, and Appendix B). It speaks of backup, archiving and key recovery rather than escrow:

• Keys that encrypt stored data should be backed up or archived for as long as the data may need to be decrypted (Appendix B). Otherwise a lost key means lost data. A recovery service that holds such keys is what people usually mean by key escrow.
• Private signature keys are, in general, not backed up: “support for non-repudiation would be in question”. If a copy exists somewhere else, a signature no longer proves that the key holder made it.
• CA signing keys are the stated exception: “backup may be warranted in some cases, such as a CA’s private signing key. When required, any backed up keys shall be stored under the owner’s control.” In practice that means HSM-protected backups or a second HSM, under the same multi-person control, never a copy handed to a third party.

Swipe sideways to see the whole diagram.

How long should a key be used?

SP 800-57 defines the cryptoperiod as “the time span during which a specific key is authorized for use or in which the keys for a given system or application may remain in effect”. For a private signature key it suggests a maximum of about one to three years (section 5.3.6), and stresses that its suggestions are “only rough order-of-magnitude guidelines”. Your certificate policy sets the real values, as a balance between exposure and the cost of changing keys. Root keys are used rarely and offline. Issuing CA keys change more often. TLS server keys should change at every renewal. A revision is in progress: Revision 6 was published as an initial public draft.

Where do your keys live today?

On Windows, the Key Storage Providers decide where a key is kept. List them with:

certutil -csplist

These are the key storage providers it listed on a Windows 11 machine (the full output also names older cryptographic service providers, left out here):

Provider Name: Microsoft Software Key Storage Provider
Provider Name: Microsoft Passport Key Storage Provider
Provider Name: Microsoft Platform Crypto Provider
Provider Name: Microsoft Pluton Cryptographic Provider
Provider Name: Microsoft Smart Card Key Storage Provider

The software provider keeps keys in files on disk, and the Platform Crypto Provider uses the computer’s TPM. HSM vendors install their own provider, which then appears in this list. An AD CS CA whose key uses the software provider keeps its most important key in a protected file on disk, not in hardware. On Linux, PKCS #11 tools such as pkcs11-tool --list-slots from OpenSC show which tokens a host can reach. In the cloud, list who holds permission to use each signing key, and check that every use is logged.

What evidence should a ceremony leave?

Record the fingerprint of every certificate created, so that it can be compared wherever the certificate is installed. For a lab root certificate:

openssl x509 -in root-ca.pem -noout -subject -fingerprint -sha256

subject=CN=Lab Root CA
sha256 Fingerprint=75:F2:A2:8E:70:1A:19:82:E1:44:31:04:DD:98:0D:76:A6:A7:D2:98:AA:7B:DD:85:AD:5A:52:C6:F4:C1:93:DC

Anyone installing that root later runs the same command and compares the result with the signed report. A root distributed with an unchecked fingerprint is a root nobody really verified. The decoder shows the same details for a self-signed root:

Try it: Certificate decoderOpen the full tool

Runs in your browser. Nothing you paste is sent anywhere.

Paste a certificate

Try a sampleChoose a sample…A website certificate (www.example.com)An expired certificateA wildcard certificate (*.example.com)A certificate with a Common Name but no SANA self-signed root CAA complete chain: leaf, intermediate and root

-----BEGIN CERTIFICATE----- MIIFxDCCA6ygAwIBAgIQV3/H8mpwNp5raCafwOtSezANBgkqhkiG9w0BAQsFADB7 MQswCQYDVQQGEwJCRTEZMBcGA1UEChMQUEtJIFplcm8gdG8gSGVybzEnMCUGA1UE CxMeU2FtcGxlcyAtIG5vdCBwdWJsaWNseSB0cnVzdGVkMSgwJgYDVQQDEx9QS0kg WmVybyB0byBIZXJvIFNhbXBsZSBSb290IENBMCAXDTI1MDEwMTAwMDAwMFoYDzIw NTAwMTAxMDAwMDAwWjB7MQswCQYDVQQGEwJCRTEZMBcGA1UEChMQUEtJIFplcm8g dG8gSGVybzEnMCUGA1UECxMeU2FtcGxlcyAtIG5vdCBwdWJsaWNseSB0cnVzdGVk MSgwJgYDVQQDEx9QS0kgWmVybyB0byBIZXJvIFNhbXBsZSBSb290IENBMIICIjAN BgkqhkiG9w0BAQEFAAOCAg8AMIICCgKCAgEAnYTGHkYMca54QPkZcZENin3BHB3t uKdHkDZeKZvJHWqpfXpcZfusLvtJFeTrwWlEDBJuV20vmhaLeV29QeYKaIkm1Ul2 qCkrLUpWn/W9CPosfVqMd1nOrd+Ka9iPSeKlW3YX1P5WqQEjieJhBzT9comdj8qN /PynceTDcLa2mgYw7QhU8trBoLzhYqIJFfoUBk+7lahae8b9G5wKCbDicRM8/ATt 0dO9jCxi6RxJpkKl24ULgNOuFP9gG4nubWA8+BaMeM0LU64wNciFofgM2g/3iHHn zmcoe1z80C7WN+GuzNkWkN8D/tcqrbua0O5fREKKw6PyyxwQsjeMghpxLb0h1KeA ihpQMVgpWbhRkkpvF5odkF9vTJP89Z2y+8cgnUMyTpXATrt8KgC5LdWo5mlTNXC/ MvbCUb1Vz9bvcEx2y29tUeLIAUmyBzhF7zgMUPAOtDbLLCuxgR78T2TWaXB/8WBT XL44Wk5rOe7U+KuBr/t12AcZjaq0AGiNsvfd1P+IFbRivElegpdrM+S3arjOjptP M9sUl9Hc+cszaTpEVcQFFxcIyvq5HVjTlZr4vLKdqlhSkQB9hZUFuWNpnUZEYEnA 1vn/GkHNTJ6pv8XxdNXkFeRaUv3ajCn7CQ3UP/EXJqzBw5fl6aa9lqKTS5kF9rgw EIhwNaSd2O557KUCAwEAAaNCMEAwDwYDVR0TAQH/BAUwAwEB/zAOBgNVHQ8BAf8E BAMCAQYwHQYDVR0OBBYEFIVx1Hy1TuED0pMkxEcdM+DWWPGiMA0GCSqGSIb3DQEB CwUAA4ICAQBMdaOOM84aoItvumQ/sNzOhXvmLwboTb3oe67wda9TvPMSf2aP3JtP UWsR/9UbZgNE9M41p2+XZlVpsGP1BLEvi5ZwfMWmR876Mw1Zkt8ouY3DWN08egnm IdV2vKwDHhDE93IQ12fjNc8iuMCasoYiXTsmXx3R/+PkNbeBAupZEk6EwTr75PCY IJ2cheL7jQ06xeCCZv815rG+63jGk3tys7A+nUXKqOp9oN/dkEhi+YWiHG9UJ2w/ BDZCFUUoJBia7C3+hcXZAVxtWwTy0A5T/plEsqGcdfCMFA4cOFstthyS/Wnnuw8y cpSiXdUAQbLkJ+CGavr6MXVmDicmnV9p3Pv9EbpGPV8UlxwvoQMf+ajEE4EAaJkN Gc85wOC7Y7wwWkBnzjiyLztqRqOFdHEdM13V+2Gw2545Jk8o2gRizBQ6Ey6rlt2u YdYFqFFQhe2H+VqvJiLKefkN1mVQhvhfAvuoxZWIsDtyJWGs12L06xMj3wG98kne mUJjPzze3swZU39eiaC1MBs6U0oLZE6ZqubmjJd/ppmGyK+7o2SMi0juouvw6rcA J9eUJQLNpiVKmiwzKvPejASpunK5B5JO3Y7NAb4SVU22ItR73bBiRAugPwVajlBN RGcjMkDSiW07/dhN6GgjsNAzXHcTR7/nbILOAWbhBstNFK1mgejlbQ== -----END CERTIFICATE-----

PEM text, several PEM blocks, or a .pem, .crt, .cer or .der file (drop it on the box). Certificates are public, so they are safe to paste.

DecodeClearChoose a file

A ceremony script, in outline

This is an example structure, not a complete script. Each line gets a time, the initials of whoever did it and the witness’s initials.

1. Confirm attendees and roles against the approved list; start the recording.
2. Check tamper-evident bag serial numbers against the last ceremony log.
3. Boot the offline workstation from verified, read-only media.
4. Initialise the HSM; custodians enrol their cards (3 of 5 required).
5. Generate the root key pair inside the HSM; confirm it is non-exportable.
6. Create the self-signed root certificate from the approved profile.
7. Sign the issuing CA's CSR (prepared and reviewed in advance).
8. Record SHA-256 fingerprints of both certificates in the log.
9. Create the HSM backup; verify it; seal the media in new numbered bags.
10. Power down; store cards, HSM and backups in separate safes; sign the log.

What should you check before you trust an HSM set-up?

• The CMVP certificate number, the level and the firmware version you actually run.
• The quorum: who the custodians are, and what happens when one leaves.
• A tested restore, from backup to a second HSM, in a second location.
• Logs from the HSM and the CA software reaching your security monitoring.
• A plan for moving keys if you change vendor. Check whether backups can only be restored to the same vendor’s devices.

Common mistakes

• “An HSM makes a hacked server harmless.” The key cannot be copied, but an attacker with control of an authorised server can ask the HSM to sign. Protect and monitor everything that can call it.
• “Escrow every key, just in case.” Escrow encryption keys. Do not escrow signing keys, because a second copy undermines the proof of who signed.
• “We have a backup, so we are safe.” A backup you have never restored is a hope, not a plan. Test restores to a second HSM, in a second place.
• “FIPS validated means validated forever.” Validations belong to specific modules and firmware, and FIPS 140-2 certificates have moved to the historical list. Check the current CMVP entry.
• “The ceremony is just paperwork.” The script, the witness and the recording are the evidence that auditors, browsers and your own users rely on.

Check yourself

Check yourself

Which key should be held by a key-recovery (escrow) service?

1. An encryption key that protects archived files
2. An employee's personal signing key
3. Every key the company owns, to be safe

Show answer

Answer A An encryption key that protects archived files

What happened

Encryption keys are escrowed so that data can still be decrypted if the key holder is gone.

Why

A copy of a signing key would weaken the proof of who signed. A lost encryption key means lost data.

The rule

Escrow encryption keys, never signing keys. Back up CA keys only under your own multi-person control.

Check yourself

An attacker gains administrator rights on a server that uses an HSM for its signing key. What can they most likely do?

1. Nothing, because the key is in the HSM
2. Ask the HSM to sign data while they control the server
3. Copy the private key out of the HSM

Show answer

Answer B Ask the HSM to sign data while they control the server

What happened

The HSM still refuses to export the key, but it signs for any authorised caller.

Why

The HSM cannot tell a legitimate request from one made by an attacker on an authorised host.

The rule

An HSM stops key theft, not key misuse. Guard and monitor every system that can call it.

Check yourself

What do the CA/Browser Forum rules require for generating a public root CA key?

1. A script, a qualified auditor as witness or a full video, and an auditor's report
2. Only the approval of the CA's chief executive
3. Publishing the private key's hash on the CA's website

Show answer

Answer A A script, a qualified auditor as witness or a full video, and an auditor's report

What happened

Section 6.1.1.1 of the Baseline Requirements sets these three requirements.

Why

Relying parties never see the ceremony, so they depend on independent evidence that it was done correctly.

The rule

Root keys are born in scripted, witnessed and recorded ceremonies.

Go deeper

• FIPS 140-3: the standard for cryptographic modules and its four levels.
• CMVP: FIPS 140-3 transition and the validated modules search: what is validated today.
• NIST SP 800-57 Part 1 Rev. 5: key management, cryptoperiods, backup and recovery.
• Baseline Requirements, sections 5.2.2, 6.1.1.1 and 6.2.7: trusted roles, key ceremonies and key storage for public CAs.
• Code Signing Baseline Requirements, section 6.2.7.4: hardware protection for code-signing keys.

In the gameLevel 21: Guarding the crown jewelsRoyal Vault, Keystone. In development

The crash course covers key protection in lesson 9, Running PKI.