---
_class: lead
paginate: true
backgroundColor: #fff
---

<!-- Speaker notes

Hey my name is Lance, a co-founder of Distrust, a security auditing and engineering company that specializes in working with high risk organizations that can't afford to trust any single computer or person. I also founded #!, a security, privacy, and digital sovereginity focused digital hackerspace built around classic public access linux systems. I also have architected and built a lot of software you have may have seen or used at one point or another over the years. I am not proud of most of it. I only do open source things now.

You have possibly seen articles of some of my various dumb publicity stunts to try to get people to care about privacy rights and supply chain security. Sorry. The japanese robot hotel one was funny though.

I now work with a team of people smarter than me at Distrust and the wider open source community and we are working on actually -solving- supply chain trust problems, because me complaining about them for years didn't work. One of those projects we are talking about today, Stagex!

Also, many have heard me preach the 10/20/30 rules of presentations.

Let's break all those rules real quick as we have a bunch of slides I want to get through fast because giving presentations terrifies me, which is why I dropped out of school.

You can read the markdown file I generated these slides from later.

I am in this for the interesting part. Questions, discussion, and debate!

Here goes.

A lot of us build software responsible for protecting immense amount of financal value, important secrets, or even human lives.

When you really boil it all down, there is almost always at least one person you can point at with the power to tamper with the software that builds your software without anyone noticing.

That person is often loves their family, likes money, or does not have good opsec. Or maybe they just let a personal email domain expire which allowed any rando to take over their release responsibilities. Several thousand supply chain critical domans are up for grabs right now by the way. I bought a few. It's easy.

In this example here, imagine someone simply placed libfakerand (shameless to my co-founder Anton) in a popular container image. After all the world runs on containers today, and almost none of them are signed or even gated with hardware multi factor auth.


Let's TLDR the problem.


And now let's TLDR the solution we landed on, with some buzzword bingo

Don't worry, the rest of the slides are mostly just me breaking this down.

So going back to what I was saying earlier, people use containers today for basically everything.

Most distros started before the OCI standard was a thing, which is why most Linux distros approach to containers feels like a hastily retrofitted bastardization that walks back critical security choices in the name of broad adoption above all else.

We asked ourselves what would happen if we just used the OCI standard all the way down from the start. Could we just use that spec basically as-is with some creative patterns, and no package manager at all, and still end up with something declarative, easy to maintain, and reproducible, with a lot of properties we like about Nix and Guix for downstream reproducibility in end user projects?

It sounds like a drunk idea the spec authors never considered, but with the right patterns it works great.

Here is our actual BC package, which is a pretty typical example. Its pretty much just linux from scratch in containers.

Note that we force the hash-locking on the dependencies at build-time to be whatever is in the same git tree

But now for a harder problem. What toolchain do we trust enough to build our toolchain that requires trusting no single party?

We didn't have an answer until we stumbled upon the Bootstrappable Builds project. Build everything from zero and get the same results no matter what host OS you are using. They had only gotten it all working a month before we started stagex. Perfect timing.

This concept is called full source bootstrapping.

There are plenty of talks about this, so I'll just TL;DR the strategy we landed on.

We start with a tiny self hosting program written in hex0, that is just enough code to build itself, and a slightly more sophisticated version of itself, hex1, and so on, until we have a basic shell called Kaem. This set of binaries is called stage0, and often referred to as seed binaries, which everything else grows from.

In stage1 we use the live bootstrap project to speed run compiling snapshots of 150 stages of open software toolchains from 1970s era to modern x86 GCC.

In stage2 we build a cross compiler bridging from x86 to our target architecture

In stage3 we build a standalone native nano linux toolchain with busybox for our target architecture

With several people doing this and getting the same bit for bit identical results and signing them, we finally have a reproducible base image we can trust to build everything else deterministically.

-now- we are in really good shape to build almost any software reproducibly.

Reproducibility itself is probably the least new of anything here, but we do have a very strict definition.

Not having to support desktop users helps a lot

But for us reproducibilty is mostly about accountability and tamper evidence. We don't want you to trust some random build server realy reproduced something. You should have a way to verify multiple independent humans did it, and ideally do it in a way that uses standards so it is possible to set your own trust thresholds before a package of ours can run in your environment.

Most distros opt for a centralized trust scope with the whole world trusting one release engineer, or maybe worse, a distributed model where every maintainer can do whatever they want with their own packages with no oversight.

We wanted decentralized trust, where we assume at least one member of the team is compromised or malicious at all times.

Thankfully the OCI spec has a policy system for multi-signing images that seemingly no one uses. Lets actually use it!

-->





<style>
  /* Changed in Marp 4.0.0. Re-center. */
    section.lead {
        display: flex;
    }

    div.two-columns {
        column-count: 2;
    }
</style>

![bg left:38% 80%](img/stagex-logo.png)

# Easier supply chain security

A container-native, full-source-bootstrapped, reproducible, and multi-signed toolchain to build all the things.

---

# Do you know who built your base image?

```dockerfile
FROM trusted/distro
RUN pkg install openssl
ENTRYPOINT ["/usr/bin/openssl"]
CMD ["rand", "-hex", "12"]
```

<hr />

<div class="two-columns">

<div>

```sh
> docker build -t local/randtest -f Containerfile .

> docker run local/randtest

2a2a2a2a2a2a2a2a2a2a2a2a

> docker run local/randtest

2a2a2a2a2a2a2a2a2a2a2a2a
```

Woops, someone included libfakerand!
</div>

<div>

![width: auto](img/demo-auto.gif)

</div>

</div>

<!-- Include link to repo -->


---

# Problem Statement

## Build software without expecting users to trust any single human or computer.

<br />
<br />
<br />
<br />
<br />
<br />

We really didn't want to make yet another Linux distro.

Sadly, nothing we evaluated fit our threat model.

---

Let's first break down the buzzword bingo:
<br />
<br />

## Stagex is a container-native, full-source-bootstrapped, reproducible, and multi-signed toolchain to build all the things.

---
# Container-native

- Most sysadmins today do everything in containers
- Most release engineers today build everything in containers
- OCI container spec solves the problems of most linux package managers
- OCI syntax allows for very succinct and simple package definitions
- OCI has many competing implementations, and this is a good thing!
- OCI has official support for multiple signatures, and signing policies
- Every stagex package is just an OCI layer, which can be hash locked

---
### Our actual BC "package" is just OCI layers all the way down

```dockerfile
FROM scratch AS build
ARG VERSION
COPY --from=stagex/core-busybox . /
COPY --from=stagex/core-gcc . /
COPY --from=stagex/core-binutils . /
COPY --from=stagex/core-make . /
COPY --from=stagex/core-musl . /
COPY --from=fetch . .
RUN tar -xf sed-${VERSION}.tar.xz
WORKDIR /sed-${VERSION}
RUN --network=none <<-EOF
  set -eux
  ./configure --prefix=/usr
  make -j "$(nproc)"
  make DESTDIR=/rootfs install
EOF
FROM stagex/core-filesystem AS package
COPY --from=build /rootfs/ /
```

---


# Full source bootstrapped

TL;DR: Solve Ken Thompsan's Trusting Trust problem:

- Stage 0: Bootstrap primitive binaries from 256 bytes of hex0
- Stage 1: Bootstrap all the way up to x86 GCC
- Stage 2: Build GCC cross compiler toolchain
- Stage 3: Building a native hosted GCC toolchain
- Stage X: Build whatever you want


Only possible because of heroic work by the Bootstrappable Builds team!

---

### Stage0: Build bootstrap seeds with self hosting hex0 "compiler"

```hex
0000000 457f 464c 0101 0301 0000 0000 0000 0000
0000010 0002 0003 0001 0000 804c 0804 002c 0000
0000020 0000 0000 0000 0000 0034 0020 0001 0000
0000030 0000 0000 8000 0804 8000 0804 00b5 0000
0000040 00b5 0000 0001 0000 0001 0000 5b58 315b
0000050 6ac9 5805 cd99 5b80 6650 41b9 6602 c0ba
0000060 6a01 5805 80cd 4299 3197 89ed 4ed6 895b
0000070 6ae1 5803 80cd 8553 75c0 4005 db31 80cd
0000080 018a 0a3c e574 f685 e475 233c df74 3b3c
0000090 db74 302c 0a2c 0872 072c df24 073c ce73
00000a0 e5c1 0404 010a f7c5 7cdf 89c3 8929 b0fb
00000b0 cd04 eb80 00b4                         
00000b5
```

It makes more sense with comments:
https://github.com/oriansj/stage0-posix-x86/blob/master/hex0_x86.hex0

---

### Stage1: Bootstrap up to x86 gcc toolchain from bootstrap seeds

It is basically building up from the 1970s all over again

- Step 1: hex0
- Step 9: M0 C (cc_x86)
- Step 19: scheme (GNU Mes)
- Step 20: ~C99 C (Tinycc)
- Step 56: perl 5x
- Step 83: gcc 4x
- Step 148: python 2x
- Step 158: gcc 13x (still x86)

Full steps: https://github.com/fosslinux/live-bootstrap/blob/master/parts.rst

---

### Stage2: Pivot from x86 -> $TARGET cross toolchain

It is not that exciting.

- Step 1: Build cross binutils
- Step 2: Build minimal cross gcc (without libgcc)
- Step 3: Build libgcc against musl headers
- Step 4: Build musl
- Step 5: Build rest of gcc

---

### Stage3: Build native $TARGET toolchain

Also not that exciting.

- Step 1: Build musl
- Step 2: Build binutils
- Step 3: Build make
- Step 4: Build gcc
- Step 5: Build busybox

This covers "full source bootstrapped"

Now we can talk about what we mean by reproducible.

---

# Reproducible

- 100% reproducible with no compromises
- Must be easy and doable at any time: Clone our repo and run "make"
- We disagree with the metrics most distros use for reproducibility
- Requiring a non-reproducible binary to acheive reproduction, is cheating
- We build -everything- from 0. Rust, random firmware blobs in qemu. Everything.

<br />
<br />

( In fairness, ignoring desktop use cases has made these goals a -lot- easier.)

---

# Multi-signed

* Centralized Trust is dangerous
* Distributed Trust is maybe even more dangerous
* We went with Decentralized trust: trust a quorum, not any individual.
* Sign every commit with a hardware token
* Sign every review with a hardware token
* Sign every reproductions on diverse systems with a hardware token
* Reproduce and submit your own signatures for the whole tree at any time!

![bg right:35% 80%](https://mermaid.ink/svg/pako:eNptUstugzAQ_BVrzyQU0-ZBpR7S9lhVKr2FHIy9gCuDkbFTRYh_ryFVgtL6YO_OjHdk7_bAtUBIoFD6m1fMWPK5yxri185JJaL9dBzIYvFEPrA1WjiO0f4SHmZi-q-Y_hFf60zKVJZNtB_3W55eeDrnpwu_JgpZh1eYzmEIoEZTMyn8A_tRlIGtsMYMEh8qWVY2g2BGPKfpmVsqlqMihTboq77nX8gt6Yk-ohl_KiFH2clc4SMZMsiawVsxZ3V6ajgk1jgMwGhXVpAUTHU-c61gFl8kKw2rb9BXIa02F1BpJtCnPdhTO_amlJ31Blw3hSxH3Bnl4cratkvCcKSXpbSVy5dc12EnxdjI6rhdhSu62jAa42ods4c4FjyPtpuC3keFWN9FlMEwBICT_9t5EKZ5GH4Asmmvxw)

<!--
flowchart TB
    Build1[Build] --\> Reproduce1[Reproduce]
    Build2[Build] --\> Reproduce2[Reproduce]
    Reproduce1 --\> Sign1[Sign]
    Reproduce2 --\> Sign2[Sign]
    Sign1 --\> Release
    Sign2 --\> Release

{
  "theme": "light",
  "themeCSS": ".label foreignObject { overflow: visible; }"
}
-->

---

# Decentralized Trust

Multiple maintainers can each sign individual images, so OCI runtime policies
can enforce _multiple_ signatures by maintainers to ensure no individual
maintainer could have tampered with an image.

[![](https://mermaid.ink/svg/pako:eNpdklFrgzAQx79KuGdbV91s62DQpmNPZbDube4hJqdmRFNi7Cjid1-sa7EGAvn_f3c5LpcWuBYIMWRK__KCGUs-d0lF3NpvvvZMVtZtNGTzTWazF_Km_-F2DLcTSMeQTuBmkJReb5poOtFD_EdT27uEkUFvBnhQoimZFK6ltscJ2AJLTCB2RyXzwibgjQA9HAY2VyxFRTJtUObVe_qD3JKW6BOa_m1icpK1TBU-ky6BpOpcKdZYfThXHGJrGvTA6CYvIM6Yqp1qjoJZ3EmWG1ZeQwbzVUirzS1SaSbQyRbs-dgPI5eXVriuMpn3fmOUswtrj3Xs-z2e59IWTTrnuvRrKfrJFad15EdBtGJBiNEyZE9hKHi6WK-y4HGRieXDImDQdR7gpf5-mPzlA3R_HuyhBw)](https://mermaid.ink/svg/pako:eNpdklFrgzAQx79KuGdbV91s62DQpmNPZbDube4hJqdmRFNi7Cjid1-sa7EGAvn_f3c5LpcWuBYIMWRK__KCGUs-d0lF3NpvvvZMVtZtNGTzTWazF_Km_-F2DLcTSMeQTuBmkJReb5poOtFD_EdT27uEkUFvBnhQoimZFK6ltscJ2AJLTCB2RyXzwibgjQA9HAY2VyxFRTJtUObVe_qD3JKW6BOa_m1icpK1TBU-ky6BpOpcKdZYfThXHGJrGvTA6CYvIM6Yqp1qjoJZ3EmWG1ZeQwbzVUirzS1SaSbQyRbs-dgPI5eXVriuMpn3fmOUswtrj3Xs-z2e59IWTTrnuvRrKfrJFad15EdBtGJBiNEyZE9hKHi6WK-y4HGRieXDImDQdR7gpf5-mPzlA3R_HuyhBw)

<!--
flowchart TD
    MA[Maintainer A] --\> Go
    MB[Maintainer B] --\> Go
    MC[Maintainer C] --\> Go
    MA --\> GCC
    MB --\> GCC
    MC --\> GCC
    MA --\> Rust
    MB --\> Rust
    MC --\> Rust

{
  "theme": "light",
  "themeCSS": ".label foreignObject { overflow: visible; }"
}
-->

---

| Distro    | Trust Model   | OCI       | Packaging    | Bootstrapped | Reproducible |
|-----------|---------------|-----------|--------------|--------------|--------------|
| Stagex    | Decentralized | Native    | Declarative  | Yes          | Yes          |
| Guix      | Distributed   | Exported  | Declarative  | Yes          | Mostly       |
| Debian    | Distributed   | Published | Imperative   | No           | Mostly       |
| Arch      | Distributed   | Published | Imperative   | No           | Mostly       |
| Nix       | Centralized   | Exported  | Declarative  | Partial      | Mostly       |
| Yocto     | Centralized   | Exported  | Source Only  | No           | Mostly       |
| Buildroot | Centralized   | Exported  | Source Only  | No           | Mostly       |
| Alpine    | Centralized   | Published | Imperative   | No           | No           |
| Fedora    | Centralized   | Published | Imperative   | No           | No           |

---

# TL;DR: Give everyone a shorter path to better supply chain security
<br />

### A reproducible full source bootstrapped Containerfile for a rust project:
<br />

```dockerfile
ARG RUST_VERSION
FROM stagex/pallet-rust@${RUST_VERSION} AS build
ADD src/ .
ENV RUSTFLAGS='-C target-feature=+crt-static'
RUN cargo build --release --target-dir .

FROM stagex/filesystem AS package
COPY --from=build hello /usr/bin/hello
CMD ["/usr/bin/hello"]
```

---

# Common toolchain dependencies

StageX comes with developer-loved tooling and languages, such as:

- `rust`
- `go`
- `python`
- `curl`
- `git`

If you are interested in additionally software being added feel free to open a
PR or let us know what you would like to see added.

---

# Key Takeaways

* StageX packages the software you're already using, securely.
* By leveraging OCI (Docker, podman, etc), we avoid mixing package managers and build contexts.
* Your software, at every point in the bootstrapped toolchain, can all be built
deterministically.

---

# What's Next?

* Packaging more software and updating existing software faster
* Adding additional container runtimes like Podman and Kaniko
* Adding additional chip architecture support such as ARM and RISC-V
* Crosschains: Cross compiler packages so you can target any arch from your native arch
* Mirrors: Much more maintainable source/mirror management in review
* Pallets: sets of deps to be imported/locked with one line
* Boxes: Easily output stagex builds as bootable ISO images, firmware, packages for other distros, etc

---

# Links

**Matrix Chat**: #stagex:matrix.org

**Git Repo**: https://codeberg.org/stagex/stagex

**Slides**: https://git.distrust.co/public/presentations

**ReprOS**: https://codeberg.org/stagex/repros

**AirgapOS**: https://git.distrust.co/public/airgap

**EnclaveOS**: https://git.distrust.co/public/enclaveos

**Keyfork**: https://git.distrust.co/public/keyfork

Big thank you to sponsors who have supported the development of this project:

**Turnkey, Distrust, Mysten Labs**