---
_class: lead
paginate: true
backgroundColor: #fff
---

<style>
  /* Changed in Marp 4.0.0. Re-center. */
  section.lead {
    display: flex;
  }
</style>

![bg left:40% 80%](img/stagex-logo.png)

# Bootstrapping Reproducibility with StageX

The steps involved in going from a 256 byte compiler to a deterministic
bit-for-bit reproducible Linux distribution.

<!--
Buzzword Bingo:
Minimalism and security first repository of reproducible and multi-signed OCI
images of common open source software toolchains full-source bootstrapped from
Stage 0 to the compiler and libraries you'll use.
-->

<!--
-->

---

# The Problem: milk sad wage cup...

<!--
At some point in time, your Rust installation was tampered with. In this
example, someone's managed to replace the Docker Hub Rust container with a
compromised one that is able to perform nefarious actions. Let's see what
happens when it runs.

Play video with:

```sh
docker build -t mnemonicgen .
docker run mnemonicgen
```

Do you know who built _your_ Rust compiler?
-->

<!--
  Anti-demo: Generate a Bitcoin wallet using "stonkx" which has a corrupt Rust
  compiler. Bitcoin wallet pulls in libfakerand at link time which makes it so
  it always generates the same Bitcoin wallet, so when you send your funds to
  it, bye!

  Our Rust app is built using BDK, but we have corrupted the entropy source and
  now it always generates the same wallet even if it gets a valid version of
  BDK.
-->

```dockerfile
FROM rust
ADD . /app
WORKDIR /app
RUN cargo build --release && \
    mv target/release/mnemonicgen /usr/bin/mnemonicgen
ENTRYPOINT ["/usr/bin/mnemonicgen"]
```

<!-- Include link to repo -->

<!--

---

# Credentials slide

-->

---

# Minimalism and security first Linux distribution

Approach the development of a secure toolchain by ensuring each component uses
exactly what it needs to build - no more, no less.

<!-- Speaker notes
Most Linux distributions are built for *compatibility* rather than *security*.
This results in a dramatic increase of attack surface area of an operating
system. StageX is designed to allow the creation of application specific
environments with a minimal footprint to eliminate attack surface area. Each
component of the toolchain installs only what it needs, and only packages what
it builds, resulting in a decreased attack surface.

StageX is the first Linux multisig distribution, is one of two fully
bootstrapped Linux distributions, is 100% reproducible and deterministic,
and can build complicated software with as few dependencies exposed as
possible.
-->


<hr />

<!--
TODO: include image describing traditional package building, by installing
_every_ dependency in a single OS, with a comparison of stagex only having mini
Containerfiles with just what each project needs. If done so, this graph can be
moved to a separate slide.
-->

| Distribution | Signatures | Libc  | Bootstrapped | Reproducible | Rust deps |
|--------------|------------|-------|--------------|--------------|----------:|
| Stagex       | 2+ Human   | Musl  | Yes          | Yes          | 4         |
| Debian       | 1 Human    | Glibc | No           | Partial      | 231       |
| Arch         | 1 Human    | Glibc | No           | Partial      | 127       |
| Fedora       | 1 Bot      | Glibc | No           | No           | 167       |
| Alpine       | None       | Musl  | No           | No           | 41        |

<!-- NOTE: "Rust deps" is the amount of dependencies required to build a Rust
hello world -->

<!---
-- Unable to confirm the following:
| Guix         | 1 Human    | Glibc | Yes          | Yes          | 4         |
| Nix          | 1 Bot      | Glibc | Partial      | Mostly       | 4         |
--->

<!-- Add a link to a script that confirms/reproduces the dependency count for
building Rust hello world -->

---

# A Rust Example

```dockerfile
FROM scratch AS fetch
ADD . /app
WORKDIR /app

FROM stagex/pallet-rust AS build
COPY --from=fetch . /
COPY --from=stagex/nettle . /
COPY --from=stagex/gmp . /
ENV TARGET=x86_64-unknown-linux-musl
RUN cargo build --release --target $TARGET

FROM stagex/filesystem AS package
COPY --from=build /app/target/$TARGET/release/hello /usr/bin/hello
CMD ["/usr/bin/hello"]
```

<!-- Speaker notes
In this example, note how we are only pulling in Rust and the dependencies
required to invoke Rust. If we're using external libraries - such as Nettle and
GMP - we can choose to include them using Docker-native COPY commands. We don't
include anything extra, which reduces attack surface when compiling software.
---
-->

<!-- TODO: make pallets a thing, test this. Include RUSTFLAGS to make static in
     the pallet -->

---

# All packages in StageX are:

* Built using hash-locked sources
* Confirmed reproducible by multiple developers
* Signed by multiple release maintainers

<!-- Speaker notes
To ensure StageX remains a secure toolchain, there's some additional
maintenance that is performed compared to most distributions. This includes:

* Built using hash-locked sources. This ensures every developer gets the exact
  same copy of the code for each container, so no middleman could inject
  malware, which helps with:
* Reproducing projects, ensuring they're built deterministically. This confirms
  that no single developer, nor their machine, have been compromised. Once each
  package is confirmed, they are...
* Signed by the release maintainers. These maintainers each build a copy of the
  package locally and sign the containers with an OCI-compliant signature using
  well-known OpenPGP keys.
---
-->

![bg right:35% 80%](https://mermaid.ink/svg/pako:eNptUstugzAQ_BVrzyQU0-ZBpR7S9lhVKr2FHIy9gCuDkbFTRYh_ryFVgtL6YO_OjHdk7_bAtUBIoFD6m1fMWPK5yxri185JJaL9dBzIYvFEPrA1WjiO0f4SHmZi-q-Y_hFf60zKVJZNtB_3W55eeDrnpwu_JgpZh1eYzmEIoEZTMyn8A_tRlIGtsMYMEh8qWVY2g2BGPKfpmVsqlqMihTboq77nX8gt6Yk-ohl_KiFH2clc4SMZMsiawVsxZ3V6ajgk1jgMwGhXVpAUTHU-c61gFl8kKw2rb9BXIa02F1BpJtCnPdhTO_amlJ31Blw3hSxH3Bnl4cratkvCcKSXpbSVy5dc12EnxdjI6rhdhSu62jAa42ods4c4FjyPtpuC3keFWN9FlMEwBICT_9t5EKZ5GH4Asmmvxw)

<!--
flowchart TB
    Build1[Build] --\> Reproduce1[Reproduce]
    Build2[Build] --\> Reproduce2[Reproduce]
    Reproduce1 --\> Sign1[Sign]
    Reproduce2 --\> Sign2[Sign]
    Sign1 --\> Release
    Sign2 --\> Release

{
  "theme": "light",
  "themeCSS": ".label foreignObject { overflow: visible; }"
}
-->

<!-- TODO: talk about bootstrapping, incl. corrupt compilers in distro
toolchain -->

<!-- https://distrowatch.com/images/other/distro-family-tree.png -->

---

# Multi-Signed OCI Images

Multiple maintainers can each sign individual images, with the container
runtime enforcing _multiple_ signatures by maintainers to ensure no individual
maintainer could have tampered with an image.

<!-- Speaker notes
StageX uses the Open Container Initiative standard for images to support the
use of multiple container runtimes. Because OCI images can be signed using
OpenPGP keys, this allows the association of built images to trusted
maintainers, which can enable developers to build their software using StageX,
without having to build the entire StageX toolchain for themselves.

Creating a network of trust builds a relationship between developers and
maintainers, allowing developers to choose maintainers that implement key
management policies that match their standards. For example, Distrust signing
keys are always stored on smart cards or airgapped machines, avoiding key
exfiltration attacks and limiting key exposure to trusted computation
environments.
---
-->

[![](https://mermaid.ink/svg/pako:eNpdklFrgzAQx79KuGdbV91s62DQpmNPZbDube4hJqdmRFNi7Cjid1-sa7EGAvn_f3c5LpcWuBYIMWRK__KCGUs-d0lF3NpvvvZMVtZtNGTzTWazF_Km_-F2DLcTSMeQTuBmkJReb5poOtFD_EdT27uEkUFvBnhQoimZFK6ltscJ2AJLTCB2RyXzwibgjQA9HAY2VyxFRTJtUObVe_qD3JKW6BOa_m1icpK1TBU-ky6BpOpcKdZYfThXHGJrGvTA6CYvIM6Yqp1qjoJZ3EmWG1ZeQwbzVUirzS1SaSbQyRbs-dgPI5eXVriuMpn3fmOUswtrj3Xs-z2e59IWTTrnuvRrKfrJFad15EdBtGJBiNEyZE9hKHi6WK-y4HGRieXDImDQdR7gpf5-mPzlA3R_HuyhBw)](https://mermaid.ink/svg/pako:eNpdklFrgzAQx79KuGdbV91s62DQpmNPZbDube4hJqdmRFNi7Cjid1-sa7EGAvn_f3c5LpcWuBYIMWRK__KCGUs-d0lF3NpvvvZMVtZtNGTzTWazF_Km_-F2DLcTSMeQTuBmkJReb5poOtFD_EdT27uEkUFvBnhQoimZFK6ltscJ2AJLTCB2RyXzwibgjQA9HAY2VyxFRTJtUObVe_qD3JKW6BOa_m1icpK1TBU-ky6BpOpcKdZYfThXHGJrGvTA6CYvIM6Yqp1qjoJZ3EmWG1ZeQwbzVUirzS1SaSbQyRbs-dgPI5eXVriuMpn3fmOUswtrj3Xs-z2e59IWTTrnuvRrKfrJFad15EdBtGJBiNEyZE9hKHi6WK-y4HGRieXDImDQdR7gpf5-mPzlA3R_HuyhBw)

<!--
flowchart TD
    MA[Maintainer A] --\> Go
    MB[Maintainer B] --\> Go
    MC[Maintainer C] --\> Go
    MA --\> GCC
    MB --\> GCC
    MC --\> GCC
    MA --\> Rust
    MB --\> Rust
    MC --\> Rust

{
  "theme": "light",
  "themeCSS": ".label foreignObject { overflow: visible; }"
}
-->

---

# Common toolchain dependencies

StageX comes with developer-loved tooling and languages, such as:

- `rust`
- `go`
- `python`
- `curl`
- `git`

If you are interested in additionally software being added feel free to open a
PR or let us know what you would like to see added.

---

# Pallets

StageX offers prebuilt containers including all the packages necessary to run
some of our most used software, such as:

- `kubectl`, `kustomize`, `helm`
- `keyfork`
- `nginx`
- `redis`
- `postgres`

We also ship pallets for building new images, such as the Rust pallet shown in
the previous example.

---

# Full source bootstrapped from Stage 0

From a 256-byte compiler written in hex, StageX bootstraps all the compiler
tools necessary to build the distribution, 100% deterministically.

<!-- Who compiles the compiler? -->

- Stage 0: Getting a basic C compiler on x86
- Stage 1: Building GCC for x86
- Stage 2: Upgrading GCC for x86_64
- Stage 3: Building up-to-date toolchains
- Stage X: Shipping the software you know and love

<!-- Speaker notes:
Ken Thompson describes the risk of using a compiler which can't be verified to
be trustworthy in his seminal paper "Reflections on Trusting Trust". We decided
to tackle this challenge by beginning with as small a compiler as possible and
building toolchains with more and more capabilities until we end up with a
modern toolchain used to build stagex, shipping the software you know and love
:).
-->

---

# OK, So What?

By using stagex, an entire family of supply chain vulnerabilities can be
eliminated. Removing unnecessary software reduces the attack surface of
potentially malicious software, while deterministic builds help ensure
software hasn't been tampered with.


Because StageX can be used to build standalone Linux systems, it can also be
used to generate bootable images without needing to ship unnecessary tooling
such as a package manager or a compiler.

---

# _Solar Winds_ of Change

According to: https://www.crowdstrike.com/blog/sunspot-malware-technical-analysis/

> - SUNSPOT is StellarParticle’s malware used to insert the SUNBURST backdoor into software builds of the SolarWinds Orion IT management product.
> - SUNSPOT monitors running processes for those involved in compilation of the Orion product and replaces one of the source files to include the SUNBURST backdoor code.
> - Several safeguards were added to SUNSPOT to avoid the Orion builds from failing, potentially alerting developers to the adversary’s presence.

<!-- Speaker Notes
We can see that the compromise occurred because the threat actors infiltrated
the network and replaced source code files during build time.

This could have been prevented by ensuring builds were deterministic.

* Ensuring that all our dependencies are reviewed and built deterministically
* Ensuring that our commits are signed by trusted developers
* Ensuring that the final result is deterministic

If Solar Winds deployed a secondary, tamper-proof runner in an isolated
environment, it's nearly impossible they would not notice that something is
amuck in their final release build. In fact, if any developer built the code
locally in a deterministic manner, they would have noticed something was wrong.

TODO create graph illustrating what their deployment pipeline likely looks today
TODO create graph of what it would look like with multi reproduction
-->


<!--

---

# Avoiding Compromised Systems

If everyone builds stagex, everyone has to be compromised.

-->

---

# Key Takeaways

* StageX packages the software you're already using, securely.
* By leveraging Docker, we avoid mixing package managers and build contexts.
* Your software, at every point in the bootstrapped toolchain, can all be built
deterministically.

<!--
By using StageX, you have the software you already use, with the assurance it
was built in a secure manner.

Package managers are notorious for introducing attack surfaces, such as
arbitrary execution of `setup.py` or post-download scripts, and by using Docker
as our package manager, we avoid all forms of spontaneous execution.

All StageX software is built deterministically, meaning you can be sure all
components listed in your Software Bill Of Materials hasn't been tampered with.
Because StageX provides a toolchain for you to build your software in the same
manner, your software can be sooper dooper pooper scooper secure.
-->

---

# What's Next?

Packaging more software and updating existing software faster

Adding additional container runtimes like Podman and Kaniko

Adding additional chip architecture support such as ARM and RISC-V

---

# Links

**Matrix Chat**: #stagex:matrix.org

**Git Repo**: https://codeberg.org/stagex/stagex

Big thank you to sponsors who have supported the development of this project:

**Turnkey, Distrust, Mysten Labs**