docs/production-engineering.md

# Production Engineering

## Overview

The goal of this document is to outline strict processes those that have access
to PRODUCTION systems MUST follow.

It is intended to mitigate most classes of known threats while still allowing
for high productivity via compartmentalization.

Production access where one has access to the personal data or assets of others
is to be taken very seriously, and assigned only to those who have tasks that
can not be performed without it.

These are the rules we wish to meet and model at #! and futher recommend for
those who are managing targeted production systems.

## Assumptions

1. All of our screens are visible to an adversary
2. All of our keyboards are logging to an adversary
3. Any firmware/bootloaders not verified on every boot are compromised
4. Any host OS with network access is compromised
5. Any guest OS used for any purpose other than prod access is compromised
7. At least one member of the PRODUCTION team is always compromised
8. At least one maintainer of third party code we depend on is compromised

## Requirements

1. A PRODUCTION ENGINEER SHOULD have the following:
    * A clear set of tasks that can't be completed without access
    * Experience in Red/Blue team, CTF, or documented CVE discoveries
    * A demonstrated working knowledge of:
        * Low level unix understanding. E.g. /proc, filesystems, kernel modules
        * Low level debugging techniques. E.g. strace, inotify, /proc, LD_PRELOAD
        * Linux kernel security features. E.g. seccomp, pam, cgroups, permissions
        * Common attack classes. E.g: XSS, Buffer Overflows, Social Engineering.
    * A majority passed interview panel with current PRODUCTION access team
    * An extensive background check clean of any evidence of dishonesty
    * Training on secret coercion and duress protocols shared with team
2. A PRODUCTION ENGINEER MUST NOT expose a CRITICAL SECRET to:
    * a screen
    * a keyboard
    * an unsupervised peer
    * an internet connected OS other than destination that requires it
3. A recommended ENTROPY SOURCE MUST be used to generate a CRITICAL SECRET
4. An OS that accesses PRIVILEGED SYSTEMS MUST NOT be used for anything else
5. Any OS without verified boot from firmware up MUST NOT be trusted
6. Manual PRIVILEGED SYSTEM mutations MUST be approved, witnessed, and recorded
7. PRIVILEGED SYSTEM mutatations MUST be automated and repeatable via code
8. Any code without SECURITY REVIEW MUST NOT be trusted
9. Any code we can not review ourselves as desired MUST NOT be trusted
10. PRIVILEGED SYSTEM access requires physical interaction with an approved HSM

## Implementation

### Tools

* HARDENED WORKSTATION
* PERSONAL HSM
* ENTROPY SOURCE

### Setup

#### Install QubesOS

* Enable full disk encryption
* Set up Verified Boot
    * Choose "factory reset" after QubesOS install in boot options
    * Sign with PERSONAL HSM
    * Verify every boot by inserting PERSONAL HSM and observing green LED
* Set up FDE password in hardware device via PERSONAL HSM
* Set up Challenge Response authentication with PERSONAL HSM
    * PERSONAL HSM + password to unlock screen
    * Automatically lock screen when PERSONAL HSM removed

#### Configure Qubes

##### Vault

* MUST NOT have internet access
* Example use cases:
    * Personal GPG keychain management
    * Bulk encrypting/decrypting documents
    * Provisioning OTP devices

##### Production

* MUST have internet access limited to only bastion servers
* MUST manage all credentials via individually approved hardware decryption
    * password-store via PERSONAL HSM or mooltipass
* SHOULD use whonix as the network gateway
* Used only to reach PRODUCTION bastion servers and systems beyond them

##### Work

* MUST have internet access limited to organization needs and partners
* MUST manage all credentials via individually approved hardware decryption
    * password-store via PERSONAL HSM or mooltipass
* SHOULD use whonix as the network gateway
* Example use cases:
    * Read only access to AWS panel
    * Observe Kibana logs
    * Check organization email and chat

##### Personal

* No internet access limits
* SHOULD only be used for personal services
* SHOULD use whonix as the network gateway
* Example use cases:
    * Check personal email / chat
    * Personal finance

##### Development

* No internet access limits
* SHOULD only be used for development
* SHOULD use whonix as the network gateway
* SHOULD manage all credentials via individually approved hardware decryption
    * password-store via PERSONAL HSM
* Example use cases:
    * Read online documentation
    * Authoring code
    * Submitting PRs
    * Doing code review

##### Disposable

* SHOULD only be used for development
* SHOULD use whonix as the network gateway
* Example use cases:
    * Pentesting
    * Testing untrusted code or applications
    * Competitive research
    * Explore dark web portals for data dumps

### Setup Keychain

* Follow "Advanced" GPG setup guide in Vault Qube
    * Daily driver PERSONAL HSM holds subkeys
    * Separate locked-away PERSONAL HSM holds master key
    * Separate locked-away flash drive holds pubkeys and encrypted private keys

## Workflow

The following describes the workflow taken to get signed code/artifacts that
have become stablized in the DEVELOPMENT environment and are ready to enter
the pipeline towards production.

### Local

* PRODUCTION ENGINEER or Software Engineer
    * Authors/tests changes to INFRASTRUCTURE REPOSOTORY in LOCAL environment
    * makes feature branch
    * makes one or more commits to feature branch
    * signs all commits with PERSONAL HSM
    * Optional: Squashes and re-signs sets of commits as desired
    * Submits code to peer for review
* PRODUCTION ENGINEER
    * Verifies changes work as intended in "local" environment
    * Verifies changes have solid health checks and recovery practices
    * Merges reviewed branch into master branch with signed merge commit

### Staging

* PRODUCTION ENGINEER #1
    * Copies desired changes from LOCAL templates to STAGING templates
    * makes feature branch
    * makes one or more commits to feature branch
    * signs all commits with PERSONAL HSM
    * Optional: Squashes and re-signs sets of commits as desired
    * Submits code to peer for review
* PRODUCTION ENGINEER #2
    * Optional: significant changes/migrations are tested in "sandbox" env.
    * Merges reviewed branch into master branch with signed merge commit
* PRODUCTION ENGINEER #1
    * Logs into STAGING toolbox
    * Deploys changes from STAGING "toolbox"

### Production

* PRODUCTION ENGINEER #1
    * Copies desired changes from STAGING templates to PRODUCTION templates
    * makes feature branch
    * makes one or more commits to feature branch
    * signs all commits with PERSONAL HSM
    * Optional: Squashes and re-signs sets of commits as desired
    * Submits code to peer for review
* PRODUCTION ENGINEER #2
    * Optional: significant changes/migrations are tested in "sandbox" env.
    * Merges reviewed branch into master branch with signed merge commit
* 2+ PRODUCTION ENGINEERs
    * Logs into PRODUCTION toolbox via PRODUCTION bastion
    * Deploys changes from PRODUCTION "toolbox" with witness

### Emergency Changes

* 2+ PRODUCTION ENGINEERs
    * Log into PRODUCTION toolbox via PRODUCTION bastion
    * Deploy live fixes from PRODUCTION "toolbox" with witness
* Regular Production process is followed from here

## Changes

Ammendments to this document including the Exceptions section may be possible
with a majority vote of current members of the Production Engineering team.

All changes must be via cryptographically signed commits by current Production
Engineering team members to limit risk of social engineering.

Direct orders from your superiors that conflict with this document should be
considered a product of duress, and thus respectfully ignored.

## Appendix

### Glossary

#### MUST, MUST NOT, SHOULD, SHOULD NOT, MAY

These key words correspond to their IETF definitions per RFC2119

#### INFRASTRUCTURE REPOSITORY

This repo, which is where all infrastructure-as-code gets integrated via
direct templates or submodules as appropriate.

#### PRODUCTION

Deployment environment that faces the public internet and consumed by end
users.

#### STAGING

Internally facing environment that is identical to PRODUCTION and will normally
be one release ahead. Intended for use by contributors to test our services and
deployment process before we deploy to any public facing environments.

#### DEVELOPMENT

Environment where changes are rapidly integrated for integration and
development aid with or without code review.

This environment is never trusted

#### LOCAL

Environment designed to run in virtual machines on the workstation of every
engineer. Designed to behave as close as possible to our production environment
so engineers can rapidly test changes and catch issues early without waiting on
longer deployment round trips to deploy to DEVELOPMENT.

This environment is intended to be the hand-off point between unprivilged
contributors and the PRODUCTION ENGINEER team.

#### SECURITY REVIEW

We consider code to be suitably reviewed if it meets the following criteria:

* Validated by member of the PRODUCTION ENGINEERING team to do what is expected
* Audited for vulnerabilities by an approved security reviewer

##### Approved security reviewers

* PRODUCTION ENGINEERING team
* Doyensec
* Cure53
* Arch Linux
* Bitcoin Core
* Canonical
* Cloud Native Computing Foundation
* CoreOS
* QubesOS
* Raptor Engineering
* Fedora Foundation
* FreeBSD Foundation
* OpenBSD Foundation
* Gentoo Foundation
* Google
* Guardian Project
* Hashicorp
* Inverse Path
* Linux Foundation
* The Debian Project
* Tor Project
* ZCash Foundation

#### HARDENED WORKSTATION

A workstation that will come in contact with production write access must meet
the following standards:

* Requires PERSONAL HSM do firmware/boot integrity attestation every boot
* Open firmware (with potential exception of Mangement Engine blob)
* CPU Management Engine disabled or removed
* Physical switches for microphone, webcam, wireless, and bluetooth
* PS/2 interface for Keyboard/touchpad to mitigate USB spoof/crosstalk attacks

##### Recommended devices

* Librem 15
* Librem 13
* Insurgo PrivacyBeast X230
* Raptor Computing Blackbird
* Raptor Computing Talos II

#### ENTROPY SOURCE

Good entropy sources should always be impossible to predict for a human. These
are also typically called a True Random Number Generator or TRNG.

In the event that we can not -prove- an entropy source is impossible to predict
then multiple unrelated entropy sources must be used and combined with each
other.

A given string of entropy must:
* Be at least 256 bits long
* Be whitened so there are no statistically significant patterns
  * sha3, xor-encrypt-xor, etc

##### Approved entropy sources

* Infinite Noise
* Built-in hardware RNG in an a PERSONAL HSM
* Webcam
* Microphone
* Thermal resistor
* Dice

#### PERSONAL HSM

Small form factor HSM capable of doing common required cryptographic operations
such as GnuPG smartcard emulation, TOTP, challenge/response, etc.

The following devices are recommended for each respective use case.

##### WebAuthN / U2F

* Yubikey 5+
* u2f-zero
* Nitrokey
* OnlyKey
* MacOS Touchbar
* ChromeOS Fingerprint ID

##### Password Management

* Yubikey 5+
* Trezor Model T
* Leger Nano X
* Mooltipass

##### Encryption/Decryption/SSH

* Yubikey 5+
* Trezor Model T
* Leger Nano X

##### Firmware Attestation

* Librem Key
* Nitrokey

#### PRODUCTION ENGINEER

One who has limited or complete access to detailed financial data or documents
of our customers, as well as any access that might allow mutation or movement
of their assets.

#### PRIVILEGED SYSTEM

Any system that has any level of access beyond that which is provided to all
members of the engineering team.

#### CRITICAL SECRET

Any secret that has partial or complete power to move customer assets

If a given computer -ever- has PRODUCTION access, then secrets used to manage
that system such as login, full disk encryption etc are in scope.