> HSMs and FIPS 140-3 levels, cloud KMS and HSM, key ceremonies with scripts and witnesses, separation of duties, backup versus escrow, and cryptoperiods. Source: https://pkizerotohero.com/handbook/hsm-and-key-ceremonies/ · Last reviewed: 2026-10-20 Part VI: Running PKI · Chapter 21 # Guarding the keys: HSMs and key ceremonies **In one sentence:** A CA is only as trustworthy as its private keys, so they live in certified hardware, are created in witnessed ceremonies and are used only under multi-person control. ## Who is this for - [Expert](https://pkizerotohero.com/start/#expert) - [IT ops](https://pkizerotohero.com/start/#itops) - [Leader](https://pkizerotohero.com/start/#leader) - [Developer](https://pkizerotohero.com/start/#developer) - [Beginner](https://pkizerotohero.com/start/#beginner) Level: **Expert** - [New to this? Start with The idea](https://pkizerotohero.com/handbook/hsm-and-key-ceremonies/#the-idea) - [Developer? Jump to How it really works](https://pkizerotohero.com/handbook/hsm-and-key-ceremonies/#how-it-really-works) - [IT ops? At work](https://pkizerotohero.com/handbook/hsm-and-key-ceremonies/#at-work) - [Leader? Why it matters](https://pkizerotohero.com/handbook/hsm-and-key-ceremonies/#why-it-matters-to-the-business) ## The idea - Beginner Everything in PKI depends on private keys staying private. If a web server’s key leaks, one site can be impersonated. If a CA’s key leaks, every certificate it ever signed becomes doubtful, and an attacker can sign new ones for any name. [DigiNotar](https://pkizerotohero.com/handbook/famous-failures/) showed how far that reaches. So the most important keys get special treatment: - They live in a [hardware security module](https://pkizerotohero.com/handbook/glossary/#hsm) (HSM): a hardened device that creates keys inside itself and uses them there. Software sends the HSM data to sign and gets a signature back. The key itself never leaves in readable form. - They are created in a [key ceremony](https://pkizerotohero.com/handbook/glossary/#key-ceremony): a scripted, witnessed and recorded event, planned in advance and documented afterwards. - No single person can use, copy or restore them alone. This is [separation of duties](https://pkizerotohero.com/handbook/glossary/#separation-of-duties): tasks are split so that misuse needs several people to agree. - They are backed up carefully, and only the right kinds of keys are ever handed to a recovery service, a practice called [key escrow](https://pkizerotohero.com/handbook/glossary/#key-escrow). - They retire on schedule. The period during which a key may be used is its [cryptoperiod](https://pkizerotohero.com/handbook/glossary/#cryptoperiod). None of this is only for public CAs. If your organisation runs its own CA (see [A CA of your own](https://pkizerotohero.com/handbook/private-ca/)), signs software or protects encrypted backups, the same ideas apply at a smaller scale. In the game The Royal Vault In the game, Rosa’s signing key sits in the Royal Vault. The door needs three of five keyholders, a scribe reads each step aloud from a script, and a witness signs the record. Where it stops being true: an HSM does not judge what it signs. If someone takes over a computer that is allowed to use it, the HSM will sign whatever that computer asks, although it still will not hand over the key. And a real backup of a CA key is a protected copy under the CA’s own control, not a spare key given to someone else. ## How it really works - Developer - Expert ### What does an HSM protect against, and what not? An HSM generates a key pair inside its protected boundary and marks the private key as non-exportable. Applications reach it through standard interfaces: PKCS #11 on most platforms, Key Storage Providers in Windows, and provider APIs in Java. Good HSMs add: - authentication of operators, often with smart cards and a quorum, such as three of five cards; - tamper evidence or tamper response, which wipes keys if the device is opened; - an audit log of every use of every key. An HSM protects the key from being copied. It does not stop misuse by a caller it trusts. An attacker who controls the CA server can ask the HSM to sign while they have that control. That is why CA software, the network path to the HSM and its logs need the same care as the HSM itself. ### What do the FIPS 140-3 levels mean? HSMs are tested against FIPS 140-3, Security Requirements for Cryptographic Modules. NIST and the Canadian Centre for Cyber Security run the validation programme, the Cryptographic Module Validation Program (CMVP); accredited laboratories do the testing. The [standard](https://nvlpubs.nist.gov/nistpubs/FIPS/NIST.FIPS.140-3.pdf) “provides four increasing, qualitative levels of security” and takes its detailed requirements from ISO/IEC 19790. NIST’s earlier [FIPS 140-2](https://nvlpubs.nist.gov/nistpubs/FIPS/NIST.FIPS.140-2.pdf) describes the levels in plain words: | Level | In short | | --- | --- | | 1 | Basic security: production-grade components, no specific physical security mechanisms | | 2 | Adds tamper evidence and at least role-based authentication of operators | | 3 | Tries to prevent an intruder from reaching keys, for example by wiping them when the casing is opened; identity-based authentication; plaintext keys only through separated ports or a trusted path | | 4 | A complete envelope of protection, including against voltage and temperature outside the normal range | FIPS 140-3 keeps the same four levels, but its detailed requirements come from ISO/IEC 19790 and differ in places. For example, level 4 now also requires multi-factor authentication of operators (ISO/IEC 19790 requirement AS04.59, quoted in NIST’s [FIPS 140-3 Implementation Guidance](https://csrc.nist.gov/csrc/media/Projects/cryptographic-module-validation-program/documents/fips%20140-3/FIPS%20140-3%20IG.pdf)). Read the table as the general idea, not the test criteria. Public CAs must protect their keys in a device validated to at least FIPS 140-2 level 3, FIPS 140-3 level 3, or an equivalent Common Criteria evaluation (EAL 4 or higher), according to [Baseline Requirements](https://cabforum.org/working-groups/server/baseline-requirements/requirements/) section 6.2.7. Code signing has its own rule: code-signing keys must be kept in certified hardware (at least FIPS 140-2 level 2 or Common Criteria EAL 4+) (since 1 June 2023, [Code Signing Baseline Requirements](https://cabforum.org/working-groups/code-signing/requirements/) section 6.2.7.4). Validation is tied to a specific module, firmware version and certificate number, so check the [CMVP list](https://csrc.nist.gov/projects/cryptographic-module-validation-program/validated-modules/search), not the brochure. All FIPS 140-2 validations have moved to the CMVP historical list (after 21 September 2026). Modules on the historical list may still be used in existing systems, but FIPS 140-3 says the historical list “should not be used for procurement decisions”. ### What about cloud KMS and cloud HSM? The major clouds offer two kinds of service. A **key management service** (KMS) is shared: you call an API, and the provider runs the HSMs behind it. A **cloud HSM** gives you dedicated HSM capacity that you administer yourself. Their own documentation makes these claims: | Service | The vendor’s claim | | --- | --- | | [AWS KMS](https://docs.aws.amazon.com/kms/latest/developerguide/overview.html) | Keys “protected by FIPS 140-3 Security Level 3 validated hardware security modules” | | [AWS CloudHSM](https://docs.aws.amazon.com/cloudhsm/latest/userguide/introduction.html) | HSMs “either FIPS 140-2 level-3 or FIPS 140-3 level-3 validated for clusters in FIPS mode” | | [Azure Key Vault Managed HSM](https://learn.microsoft.com/en-us/azure/key-vault/managed-hsm/overview) | “FIPS 140-3 Level 3 validated HSMs” | | [Azure Cloud HSM](https://learn.microsoft.com/en-us/azure/cloud-hsm/overview) | “a highly available, FIPS 140-3 Level 3 validated single-tenant service” | | [Google Cloud HSM](https://cloud.google.com/kms/docs/hsm) | “a cluster of FIPS 140-2 Level 3 certified HSMs” | These are the vendors’ own words as read in October 2026. Some pages still cite FIPS 140-2 validations, which are now on the historical list, so check the CMVP certificate behind each claim. In the cloud, the HSM protects the key from being copied, but your identity and access policies decide who may ask it to sign. Review those policies as strictly as you would guard a smart card, and send every signing event to your logs. *Diagram: A root key ceremony. An administrator sends commands to an offline HSM in a safe room by following the script. Key custodians unlock it with a quorum, for example 3 of 5 cards. A witness checks every step and a camera records it all, both shown with dashed lines to the HSM. The outputs, marked with ticks, are the root certificate with its fingerprint and a signed ceremony report.* A root key ceremony: several people, one script, full evidence. ### What happens in a key ceremony? For root keys, and for intermediates operated by a different organisation, the Baseline Requirements (section 6.1.1.1) say the CA shall: 1. “prepare and follow a Key Generation Script”; 2. “have a Qualified Auditor witness the CA Key Pair generation process or record a video of the entire CA Key Pair generation process”; and 3. have the auditor issue a report that the CA followed its ceremony and controlled the key’s integrity and confidentiality. In all cases the key must be generated “in a physically secured environment”, by “personnel in Trusted Roles under the principles of multiple person control and split knowledge”, and the CA must log the activity. A typical ceremony has these roles: - an **administrator**, who reads the script aloud and types the commands; - **key custodians**, who each hold one share of the authority to unlock the HSM, such as one smart card and PIN out of five, with three needed; - a **witness**, often the auditor, who checks every step and signs the record; - someone who **records** it, on video and in a written log with times and signatures. The script is written and reviewed weeks before. On the day, the team checks the serial numbers on tamper-evident bags, starts an offline machine, initialises the HSM, generates the key, creates the root certificate, signs the intermediate CSRs that were prepared in advance, makes and verifies the backups, and seals everything again. Every step gets a time and initials. The report records the SHA-256 fingerprint of each certificate created, so that anyone can later check they have the right one. ### Who may do what? Separation of duties means no single person can complete a sensitive task alone. The Baseline Requirements (section 5.2.2) say: “The CA Private Key SHALL be backed up, stored, and recovered only by personnel in Trusted Roles using, at least, dual control in a physically secured environment.” In practice: - the people who approve certificate requests are not the people who operate the HSM; - the people who operate systems are not the people who review the logs; - unlocking the HSM needs a quorum, so one absent, careless or dishonest person can neither block nor misuse the key. Keep the quorum realistic. If three of five custodians are needed and two leave the company, you are one illness away from being locked out of your own CA. ### Backup or escrow? Losing a key and copying a key are opposite risks, and different keys need different answers. NIST SP 800-57 Part 1, Revision 5, gives the rules (section 8.2.2.1, Table 7, and Appendix B). It speaks of backup, archiving and key recovery rather than escrow: - **Keys that encrypt stored data** should be backed up or archived for as long as the data may need to be decrypted (Appendix B). Otherwise a lost key means lost data. A recovery service that holds such keys is what people usually mean by key escrow. - **Private signature keys** are, in general, not backed up: “support for non-repudiation would be in question”. If a copy exists somewhere else, a signature no longer proves that the key holder made it. - **CA signing keys** are the stated exception: “backup may be warranted in some cases, such as a CA’s private signing key. When required, any backed up keys shall be stored under the owner’s control.” In practice that means HSM-protected backups or a second HSM, under the same multi-person control, never a copy handed to a third party. *Diagram: Three kinds of private key. An encryption key that protects stored data: back up or escrow it, marked with a tick, because a lost key means lost data. A personal signing key that proves who signed: do not escrow it, marked with a cross, because a copy weakens that proof. A CA signing key that signs certificates: back it up in HSM-protected form under the CA's own multi-person control, marked with a tick.* Escrow encryption keys. Never escrow signing keys. Back up CA keys only under your own control. ### How long should a key be used? SP 800-57 defines the cryptoperiod as “the time span during which a specific key is authorized for use or in which the keys for a given system or application may remain in effect”. For a private signature key it suggests a maximum of about one to three years (section 5.3.6), and stresses that its suggestions are “only rough order-of-magnitude guidelines”. Your certificate policy sets the real values, as a balance between exposure and the cost of changing keys. Root keys are used rarely and offline. Issuing CA keys change more often. TLS server keys should change at every renewal. A revision is in progress: Revision 6 was published as an initial public draft. ## At work - Developer - IT ops ### Where do your keys live today? On Windows, the Key Storage Providers decide where a key is kept. List them with: ```powershell certutil -csplist ``` These are the key storage providers it listed on a Windows 11 machine (the full output also names older cryptographic service providers, left out here): ```text Provider Name: Microsoft Software Key Storage Provider Provider Name: Microsoft Passport Key Storage Provider Provider Name: Microsoft Platform Crypto Provider Provider Name: Microsoft Pluton Cryptographic Provider Provider Name: Microsoft Smart Card Key Storage Provider ``` The software provider keeps keys in files on disk, and the Platform Crypto Provider uses the computer’s TPM. HSM vendors install their own provider, which then appears in this list. An AD CS CA whose key uses the software provider keeps its most important key in a protected file on disk, not in hardware. On Linux, PKCS #11 tools such as `pkcs11-tool --list-slots` from OpenSC show which tokens a host can reach. In the cloud, list who holds permission to use each signing key, and check that every use is logged. ### What evidence should a ceremony leave? Record the fingerprint of every certificate created, so that it can be compared wherever the certificate is installed. For a lab root certificate: ```bash openssl x509 -in root-ca.pem -noout -subject -fingerprint -sha256 ``` ```text subject=CN=Lab Root CA sha256 Fingerprint=75:F2:A2:8E:70:1A:19:82:E1:44:31:04:DD:98:0D:76:A6:A7:D2:98:AA:7B:DD:85:AD:5A:52:C6:F4:C1:93:DC ``` Anyone installing that root later runs the same command and compares the result with the signed report. A root distributed with an unchecked fingerprint is a root nobody really verified. The decoder shows the same details for a self-signed root: *Interactive tool: Certificate decoder (runs in the browser at [/tools/decoder/](https://pkizerotohero.com/tools/decoder/))* ### A ceremony script, in outline This is an example structure, not a complete script. Each line gets a time, the initials of whoever did it and the witness’s initials. ```text 1. Confirm attendees and roles against the approved list; start the recording. 2. Check tamper-evident bag serial numbers against the last ceremony log. 3. Boot the offline workstation from verified, read-only media. 4. Initialise the HSM; custodians enrol their cards (3 of 5 required). 5. Generate the root key pair inside the HSM; confirm it is non-exportable. 6. Create the self-signed root certificate from the approved profile. 7. Sign the issuing CA's CSR (prepared and reviewed in advance). 8. Record SHA-256 fingerprints of both certificates in the log. 9. Create the HSM backup; verify it; seal the media in new numbered bags. 10. Power down; store cards, HSM and backups in separate safes; sign the log. ``` ### What should you check before you trust an HSM set-up? - The CMVP certificate number, the level and the firmware version you actually run. - The quorum: who the custodians are, and what happens when one leaves. - A tested restore, from backup to a second HSM, in a second location. - Logs from the HSM and the CA software reaching your security monitoring. - A plan for moving keys if you change vendor. Check whether backups can only be restored to the same vendor’s devices. ## Common mistakes - **“An HSM makes a hacked server harmless.”** The key cannot be copied, but an attacker with control of an authorised server can ask the HSM to sign. Protect and monitor everything that can call it. - **“Escrow every key, just in case.”** Escrow encryption keys. Do not escrow signing keys, because a second copy undermines the proof of who signed. - **“We have a backup, so we are safe.”** A backup you have never restored is a hope, not a plan. Test restores to a second HSM, in a second place. - **“FIPS validated means validated forever.”** Validations belong to specific modules and firmware, and FIPS 140-2 certificates have moved to the historical list. Check the current CMVP entry. - **“The ceremony is just paperwork.”** The script, the witness and the recording are the evidence that auditors, browsers and your own users rely on. ## Why it matters to the business - Leader Why it matters to the business The CA key is the company's master key Anyone who can use your CA’s private key can impersonate any system that trusts that CA. Protecting it is a governance task as much as a technical one. Decide who the key custodians are, with deputies, and what happens when one leaves. Fund certified HSMs or a cloud HSM service, a tested disaster-recovery copy, and an independent witness for root ceremonies. Ask for the evidence: the ceremony report, the last restore test and the access logs. These are cheap compared with re-issuing every certificate after a compromise. ## Check yourself Check yourself Which key should be held by a key-recovery (escrow) service? 1. An encryption key that protects archived files 2. An employee's personal signing key 3. Every key the company owns, to be safe Show answer Answer A An encryption key that protects archived files What happened Encryption keys are escrowed so that data can still be decrypted if the key holder is gone. Why A copy of a signing key would weaken the proof of who signed. A lost encryption key means lost data. The rule Escrow encryption keys, never signing keys. Back up CA keys only under your own multi-person control. Check yourself An attacker gains administrator rights on a server that uses an HSM for its signing key. What can they most likely do? 1. Nothing, because the key is in the HSM 2. Ask the HSM to sign data while they control the server 3. Copy the private key out of the HSM Show answer Answer B Ask the HSM to sign data while they control the server What happened The HSM still refuses to export the key, but it signs for any authorised caller. Why The HSM cannot tell a legitimate request from one made by an attacker on an authorised host. The rule An HSM stops key theft, not key misuse. Guard and monitor every system that can call it. Check yourself What do the CA/Browser Forum rules require for generating a public root CA key? 1. A script, a qualified auditor as witness or a full video, and an auditor's report 2. Only the approval of the CA's chief executive 3. Publishing the private key's hash on the CA's website Show answer Answer A A script, a qualified auditor as witness or a full video, and an auditor's report What happened Section 6.1.1.1 of the Baseline Requirements sets these three requirements. Why Relying parties never see the ceremony, so they depend on independent evidence that it was done correctly. The rule Root keys are born in scripted, witnessed and recorded ceremonies. ## Go deeper - [FIPS 140-3](https://nvlpubs.nist.gov/nistpubs/FIPS/NIST.FIPS.140-3.pdf): the standard for cryptographic modules and its four levels. - [CMVP: FIPS 140-3 transition](https://csrc.nist.gov/projects/fips-140-3-transition-effort) and the [validated modules search](https://csrc.nist.gov/projects/cryptographic-module-validation-program/validated-modules/search): what is validated today. - [NIST SP 800-57 Part 1 Rev. 5](https://csrc.nist.gov/pubs/sp/800/57/pt1/r5/final): key management, cryptoperiods, backup and recovery. - [Baseline Requirements](https://cabforum.org/working-groups/server/baseline-requirements/requirements/), sections 5.2.2, 6.1.1.1 and 6.2.7: trusted roles, key ceremonies and key storage for public CAs. - [Code Signing Baseline Requirements](https://cabforum.org/working-groups/code-signing/requirements/), section 6.2.7.4: hardware protection for code-signing keys. The crash course covers key protection in [lesson 9, Running PKI](https://pkizerotohero.com/course/running-pki/). **Early edition.** Checked against the standards; an independent expert review is in progress. Found a mistake? [Tell us](https://pkizerotohero.com/contact/?topic=content-error&page=/handbook/hsm-and-key-ceremonies/). Last reviewed 20 October 2026Edition 0.9 ## Keep going Crash course: [Lesson 9: Running PKI](https://pkizerotohero.com/course/running-pki/) ### Tools - [Certificate decoder](https://pkizerotohero.com/tools/decoder/): Paste a PEM certificate and read every field in plain words. ### Topics - [Private PKI](https://pkizerotohero.com/topics/private-pki/)