398 lines
14 KiB
Markdown
398 lines
14 KiB
Markdown
# Disaster Recovery
|
|
|
|
## Overview
|
|
|
|
This document outlines the creation of a "Disaster Recovery" (DR) system which
|
|
functions as a one-way box that we can encrypt secrets to at any time, but only
|
|
recover them with cooperation of a quorum of people with access to multiple
|
|
offline HSM devices stored in a diversity of physical locations.
|
|
|
|
In short, it should be trivial to backup data, but very expensive to recover;
|
|
recovery is difficult to do for an attacker, accepted to be time-consuming,
|
|
logistically complicated and financially costly.
|
|
|
|
Data is backed up by encrypting plaintext to a [Disaster Recovery Key](#dr-key).
|
|
The resulting ciphertext is then stored in the
|
|
[Disaster Recovery Warehouse](#dr-warehouse). In the case of a disaster,
|
|
ciphertexts can be gathered from the DR Warehouse and then decrypted using the
|
|
DR Key to regain access to the plaintext.
|
|
|
|
## Threat Model
|
|
|
|
Each of the below can always be independently tolerated if only one of them
|
|
happens and the disaster recovery goes to plan. However if more than one of
|
|
the below happen, it is possible for the DR key to either be lost or stolen.
|
|
|
|
- N-M operators lose control of their Operator Key Smartcards
|
|
- Where an M-of-N threshold is 3-of-5, then a loss of two is tolerated
|
|
- An entire cloud provider or account becoming unavailable
|
|
- An offline backup location is fully destroyed
|
|
- An adversary gains control of any single DR participant
|
|
- An adversary gains control of a single offline backup location
|
|
- An adversary has malware that can control any internet connected computer
|
|
|
|
## Components
|
|
|
|
```mermaid
|
|
flowchart TD
|
|
shard1-enc(Shard 1 <br>Encrypted)
|
|
secrets-enc(Encrypted Secrets)
|
|
loc1-prvkey-sc(Location 1<br>Key Smartcard)
|
|
loc1-prvkey-enc-pass(Location 1<br>Key Passphrase<br> Encrypted)
|
|
op1-sc(Operator 1<br> Key Smartcard)
|
|
seckey(DR Private Key)
|
|
secrets-dec --> pubkey
|
|
pubkey --> secrets-enc
|
|
pubkey(DR Public Key)
|
|
ssss(Shamir's Secret<br> Sharing System)
|
|
ssss --> seckey
|
|
secrets-enc --> seckey
|
|
seckey --> secrets-dec(Plaintext Secrets)
|
|
op1-sc --> loc1-prvkey-enc-pass
|
|
loc1-prvkey-enc-pass --> loc1-prvkey-sc
|
|
shard1-enc --> loc1-prvkey-sc
|
|
loc1-prvkey-sc --> shard1
|
|
shard1(Shard 1) --> ssss
|
|
shard2(Shard 2) -.-> ssss
|
|
shard3(Shard 3) -.-> ssss
|
|
style shard2 stroke:#222,stroke-width:1px,color:#555,stroke-dasharray: 5 5
|
|
style shard3 stroke:#222,stroke-width:1px,color:#555,stroke-dasharray: 5 5
|
|
```
|
|
|
|
Note: we assume all PGP encryption subkeys use Curve 25519 + AES 256.
|
|
|
|
### DR Key
|
|
|
|
- PGP asymmetric key pair all secrets are directly encrypted to
|
|
- We chose the PGP standard because:
|
|
- It is a widely supported with a plurality of implementations and tooling
|
|
- The PGP standard and tooling is assumed to outlive any custom made tools
|
|
- Should be more reliable than any crypto implementation we maintain
|
|
- Possible more than one DR key could exist in the future for specialized uses
|
|
|
|
### DR Key Shards
|
|
|
|
- A Shamirs Secret Share of the private portion of the DR Key
|
|
- Encrypted to a respective Location Key
|
|
- Stored solely in geographically separate Locations.
|
|
|
|
### Location
|
|
|
|
- DR Key Shards and Location Keys are distributed to separate Locations
|
|
- The Locations are geographically separated
|
|
- The Locations have a fixed human access list
|
|
- Those with access can however cooperate to transfer access to others
|
|
- Each Location has staff and physical controls that strictly enforce access
|
|
|
|
### Location Keys
|
|
|
|
- A PGP Keypair whose private key is stored only in a given Location
|
|
- Shards are encrypted to a Location public key
|
|
- The Location Private Key is used to decrypt shards
|
|
- We are confident only one of each Location private key bundle exists
|
|
- Keys are generated on an airgapped system with witnesses
|
|
- Airgapped system is proven to run code all parties agree to
|
|
- Each Location private key is replicated on three mediums:
|
|
1. Yubikey 5 (primary)
|
|
2. Encrypted on paper (backup)
|
|
3. Encrypted on a SD card (backup)
|
|
- All mediums are decrypted/unlocked with the same 256bit entropy password
|
|
|
|
### DR Courier
|
|
|
|
- A human who is on the access list for a Location
|
|
- Capable of retrieving a set of shard and its Location Keys.
|
|
- We expect a Shard and its Location Key are only accessible by one DR Courier
|
|
- May be distinct from Operator, but not strictly necessary
|
|
- Must be highly trusted, but does not have to be technically skilled
|
|
|
|
### Operator
|
|
|
|
- A human who is capable of decrypting data with a Location Key
|
|
- They use their Operator Key to decrypt the password of a Location Key
|
|
|
|
### Operator Key
|
|
|
|
- Each Operator has their own Operator Key on a smartcard
|
|
- Operator Key smartcards have a simple, known authorization PIN
|
|
- Security relies on geographic separation of the Operator Keys and Shard
|
|
- Can decrypt the pin or backup password for a specific DR Location Key
|
|
|
|
### DR Warehouse
|
|
|
|
- Online storage for encrypted data replicated across multiple providers
|
|
- All data in DR Warehouse can only be decrypted by the DR Key
|
|
- Tolerate loss of any single provider by duplicating data to all of them
|
|
- Storage backends can be any combination of the following:
|
|
- S3 Compatible object stores:
|
|
- AWS, Google Cloud, DigitalOcean, Azure, etc.
|
|
- Version control systems:
|
|
- Github, AWS CodeCommit, Gitea
|
|
- We tolerate a loss of all but one DR storage backend
|
|
- A minimum of three storage backends should be maintained
|
|
|
|
### DR Ceremony System
|
|
|
|
- Laptop trusted to be untampered by all ceremony participants
|
|
- Could be a brand new laptop, or one tightly held under multi-party custody
|
|
- Has firmware capable of attesting the hash of an OS ISO we wish to use
|
|
- Allows all parties to form trust no single party tampered with the OS
|
|
|
|
### DR Ceremony OS
|
|
|
|
- Linux operating system
|
|
- Basic shell tools
|
|
- bash, sed, grep, awk, etc.
|
|
- Cryptography tools
|
|
- gpg, openssl, etc
|
|
- A Shamirs Secret Sharing tool
|
|
- ssss, horcrux, or similar
|
|
- Built by multiple people who confirmed identical hashes
|
|
- AirgapOS is suggested:
|
|
- <https://github.com/distrust-foundation/airgapos>
|
|
|
|
### DR Generation Script
|
|
|
|
**Routine Inputs:**
|
|
|
|
- Desired m-of-n threshold for Shamir Secret Sharing split of Quorum key
|
|
- N \* 2 unprovisioned yubikeys
|
|
- N \* 2 + 1 SD cards
|
|
|
|
**Subroutines:**
|
|
|
|
- DR Key generation:
|
|
|
|
1. Generate a PGP key with an encryption subkey and signing subkey.
|
|
|
|
- DR Operator Key generation:
|
|
|
|
1. For each Operator
|
|
1. Set a simple, well documented PIN for a Yubikey
|
|
2. Provision a PGP key with an encryption subkey directly onto the Yubikey
|
|
3. Sign the public key cert of the generated key with the DR Key
|
|
|
|
- Location Key generation:
|
|
|
|
1. Generate a PGP key with an encryption subkey using multiple entropy sources
|
|
2. Generate a random, 43 char password
|
|
3. Encrypt PGP secret key to the generated password
|
|
4. Encrypt password to the Operator Keys associated with this Location
|
|
5. Export the encrypted PGP secret key to paper and an SD Card
|
|
6. Provision a yubikey with the generated password as the PIN
|
|
7. Upload the Location Key onto the yubikey
|
|
8. Sign Location Public Key with DR Key
|
|
|
|
- Shards Generation:
|
|
|
|
1. Split DR Key secret with Shamir Secret Sharing
|
|
- M is the reconstruction threshold
|
|
- N is the total number of shards
|
|
2. Encrypt each shard to its assigned Location Key
|
|
|
|
- Generate DR Key Verification Challenge
|
|
|
|
1. Encrypt a random string to the DR Key
|
|
2. Decrypt random string ciphertext with DR Key and verify
|
|
|
|
**Routine Outputs:**
|
|
|
|
- N Operator Key Yubikeys
|
|
- N SD Cards ("Operator Cards ") each containing a plaintext Operator Key
|
|
- N pieces of paper each containing a plaintext Operator Key
|
|
- N Location Key Yubikeys
|
|
- N SD Cards ("Shard Cards") Containing:
|
|
- Location Key backup (encrypted to password)
|
|
- Encrypted password (for Location Key)
|
|
- Shard encrypted to Location Key
|
|
- N pieces of paper with a Location Key backup (encrypted to password)
|
|
- SD card ("Public Card") containing:
|
|
- DR Key public key
|
|
- All DR Operator Key public keys and DR Key signatures over them
|
|
- All DR Location Key public keys and DR Key signatures over them
|
|
- DR Key verification challenge
|
|
|
|
### DR Reconstitution Script
|
|
|
|
**Routine Inputs:**
|
|
|
|
- All Shards
|
|
- m-of-n encrypted Location Key PINs
|
|
- m-of-n Location Key Yubikeys
|
|
- m-of-n Operator Yubikeys
|
|
|
|
**Routine:**
|
|
|
|
1. For m operators:
|
|
1. prompt for Operator to insert Operator Key Yubikey
|
|
2. prompt for Operator to insert Location Key Yubikey
|
|
3. Operator key is used to decrypt password
|
|
4. Decrypted password is used to authorize Location Key Yubikey
|
|
5. Location Key Yubikey is used to decrypt Shard
|
|
6. Decrypted shard is persisted in memory with other decrypted shards
|
|
7. Decrypted shards are assembled with Shamir Secret Sharing tool, outputting
|
|
DR Key
|
|
|
|
**Routine Output:**
|
|
|
|
- Plaintext DR Key
|
|
|
|
## DR Generation Ceremony
|
|
|
|
```mermaid
|
|
flowchart LR
|
|
subgraph storage[Online Storage]
|
|
loc-pubkey(Location<br> Public Keys)
|
|
op-pubkey(Operator<br> Public keys)
|
|
pubkey(DR Public Key)
|
|
shard-enc(Encrypted<br> Shards 1-N)
|
|
end
|
|
|
|
subgraph generation[Generation]
|
|
seckey(DR Private Key)
|
|
ssss[Shamir's Secret<br> Sharing System]
|
|
seckey --> ssss
|
|
ssss --> shard(Shards 1-N)
|
|
end
|
|
|
|
subgraph location[Location 1-N]
|
|
loc-prvkey-sc(Location Key<br> Smartcard)
|
|
loc-prvkey-enc-paper(Location Key<br> Encrypted<br> Paper Backup)
|
|
loc-prvkey-enc-pass(Location Key<br> Encrypted<br> Passphrase)
|
|
end
|
|
|
|
subgraph operator[Operator 1-N]
|
|
op-sc(Operator Key <br> Smartcard)
|
|
op-sd(Operator Key<br> Plaintext SD)
|
|
op-ppr(Operator Key<br> Plaintext Paper)
|
|
end
|
|
|
|
generation --> operator
|
|
generation --> storage
|
|
generation --> location
|
|
```
|
|
|
|
**Requirements:**
|
|
|
|
- N Operators
|
|
- 1 Ceremony Laptop with Coreboot/Heads firmware
|
|
- 1 flash drive containing:
|
|
- Multi-party signed iso of AirgapOS
|
|
- DR Generation Script
|
|
- Shamirs Secret Sharing Binary
|
|
- N new Operator Yubikeys
|
|
- N new Location Key Yubikeys
|
|
- N\*2+1 new SD Cards
|
|
- N\*3 safety deposit bags
|
|
- 1 bottle of glitter nail polish
|
|
- 1 phone/camera of any kind
|
|
- Prefer offline device that can save images to SD card
|
|
- 1 Disposable inkjet printer with ink and paper
|
|
- Should have no wifi, or have wifi physically removed
|
|
|
|
**Steps:**
|
|
|
|
1. Boot Ceremony Laptop to Heads firmware menu
|
|
2. Drop to a debug shell
|
|
3. Insert and mount AirgapOS flash drive
|
|
4. Generate hashes of the AirgapOS flash drive contents
|
|
5. Multiple parties verify hashes are expected
|
|
6. Boot ISO file
|
|
7. Run _DR Generation Script_
|
|
8. Reboot system
|
|
9. Run _DR Reconstitution Ceremony_
|
|
10. Decrypt _DR Key Verification Challenge_
|
|
11. Shut down system
|
|
12. Print contents of _Shard Cards_ from generation script
|
|
13. Print contents of _Operator Cards_ from generation script
|
|
14. Seal Location Key artifacts
|
|
* Open new safety deposit box bag
|
|
* Insert printed backups of "Shard Cards"
|
|
* Insert respective Location Key Smartcard
|
|
* Cover seals of closed safety deposit bags in glitter nail polish
|
|
* Take and backup closeup pictures of all safety deposit bag seals
|
|
15. Seal _Operator Card_ Artifacts into "Inner bag"
|
|
* Open new safety deposit box bag
|
|
* Insert printed backup of _Operator Card_
|
|
* Insert _Operator Card_
|
|
* Cover seals of closed safety deposit bags in glitter nail polish
|
|
16. Seal _Operator Smartcard_
|
|
* Open new safety deposit box bag
|
|
* Insert _Inner bag_
|
|
* Insert _Operator Smartcard_
|
|
* Cover seals of closed safety deposit bags in glitter nail polish
|
|
17. Take closeup pictures of all safety deposit bag seals
|
|
18. Commit relevant artifacts to GitHub
|
|
- From _Public Card_:
|
|
- Public DR Key
|
|
- Public DR Shard Encryption Keys + signatures from the DR Key
|
|
- Public DR Operator Keys + signatures from the DR Key
|
|
- Images of the sealed bags
|
|
- Signatures over all of this content by multiple witness Personal PGP keys
|
|
19. Hand off sealed bags with shard material to DR Couriers.
|
|
20. Hand off sealed bags with Operator Key material to respective Operators.
|
|
|
|
## DR Reconstitution Ceremony
|
|
|
|
```mermaid
|
|
flowchart LR
|
|
|
|
subgraph storage[Online Storage]
|
|
shard-enc(Encrypted<br> Shards 1-N)
|
|
secrets-enc(Encrypted Secrets)
|
|
end
|
|
|
|
storage --> recovery
|
|
|
|
subgraph location1[Location 1]
|
|
loc1-prvkey-sc(Location 1<br>Key<br> Smartcard)
|
|
loc1-prvkey-enc-pass(Location 1<br>Key<br> Encrypted<br> Passphrase)
|
|
end
|
|
|
|
subgraph location2[Location 2]
|
|
loc2-prvkey-sc(Location 2<br>Private Key<br> Smartcard)
|
|
loc2-prvkey-enc-pass(Location 2<br>Key<br> Encrypted<br> Passphrase)
|
|
end
|
|
|
|
subgraph locationN[Location N]
|
|
locN-prvkey-sc(Location N<br>Key<br> Smartcard)
|
|
locN-prvkey-enc-pass(Location N<br>Key<br> Encrypted<br> Passphrase)
|
|
end
|
|
|
|
subgraph recovery[Recovery]
|
|
seckey(DR Private Key)
|
|
ssss[Shamir's Secret<br> Sharing System]
|
|
ssss --> seckey
|
|
seckey --> secrets-dec(Decrypted Secrets)
|
|
shard1(Shard 1) --> ssss
|
|
shard2(Shard 2) --> ssss
|
|
shardN(Shard N) --> ssss
|
|
end
|
|
|
|
location1 --> recovery
|
|
location2 --> recovery
|
|
locationN --> recovery
|
|
op1-sc(Operator 1<br> Smartcard) --> recovery
|
|
op2-sc(Operator 2<br> Smartcard) --> recovery
|
|
opN-sc(Operator N<br> Smartcard) --> recovery
|
|
```
|
|
|
|
**Requirements:**
|
|
|
|
- DR Key Public Key
|
|
- M-of-N Operators
|
|
- M-of-N Operator Key Yubikeys
|
|
- M-of-N Location Keys
|
|
- M-of-N Encrypted Location Key pins
|
|
- M-of-N Shards
|
|
|
|
**Steps:**
|
|
|
|
1. Boot Ceremony Laptop to Heads firmware menu
|
|
2. Drop to a debug shell
|
|
3. Insert and mount AirgapOS flash drive
|
|
4. Generate hash of AirgapOS ISO file and DR Reconstitution Script
|
|
5. Multiple parties verify hashes are expected
|
|
6. Boot ISO file
|
|
7. Run DR Reconstitution Script
|