Jetstream: the fleet as a service of its own (design) 2814f4213e7c

# Jetstream

*Design and architecture decision, 2026-10-11 (oxygenflow#38). Builds on

spec §11, runner capabilities and placement (#36). Crate and CLI: `jet`.*

**Flow** says what to run: a pipeline of hermetic jobs, each in a sandbox,

with its outputs, reports and signed receipt. **Jet** carries each job to

where it should run: a fleet of runners on your own machines, on

Kubernetes or in the cloud, that measure themselves, explain every choice

they make, and fetch what they need on their own.

## 1. The decision

**Context.** Today OxygenHub holds the job queue and talks to runners

directly (`/api/v1/flow/claim`, logs, outputs, finish). Runners are

configured by hand: images are copied onto each machine, labels steer

jobs, and a job that cannot run anywhere simply waits. Every

self-hosted CI system has the same shape and the same problems (§2), and

none of it is specific to Oxygen.

**Decision.** The fleet becomes a service of its own, **Jetstream**:

- Jetstream owns the queue, the runners and their measured facts,

  placement and its explanations, pools (hosts, Kubernetes, cloud

  providers) and images. It knows nothing about repositories.

- The **forge** owns pipelines, runs, checks, receipts, artifacts,

  reports and the UI. It hands Jetstream jobs and receives their events.

- **OxygenHub embeds Jetstream** in its own process by default, so nothing

  changes for anyone who never configures it, and can instead point at a

  standalone `jet server` shared by several forges.

- **Sources are pluggable**: an Oxygen repository is mounted lazily

  through flow-casfs; a Git repository is checked out.

- **`flow run`** runs a pipeline on your own machine through an

  in-process Jetstream with one local runner: the same code path as CI.

**Consequences.**

- Jetstream can be reused by others, and could later run jobs for forges

  that are not OxygenHub (§12) without a rewrite.

- One more boundary to keep stable: the forge contract (§4) is versioned.

- Oxygen's advantage becomes visible: the same fleet runs both kinds of

  source, and an Oxygen job starts without a checkout (§5).

```mermaid

flowchart LR

    subgraph Forges

        OH["OxygenHub<br/>pipelines · runs · checks<br/>receipts · reports"]

        OTHER["another forge<br/>(an adapter, later)"]

    end

    subgraph JS["Jetstream (jet)"]

        Q["queue"]

        P["placement<br/>+ why"]

        F["runner facts"]

        I["images"]

        D["demand"]

    end

    subgraph Pools

        H["hosts<br/>earth-1 · plex-1 · narvi-1"]

        K["Kubernetes<br/>controller, pod per job"]

        C["cloud<br/>GCP · AWS · Azure"]

    end

    OH -- "jobs" --> Q

    OTHER -. "jobs" .-> Q

    Q -- "events: logs, outputs,<br/>finish, receipt" --> OH

    Q --> P

    F --> P

    H -- "claim + facts" --> P

    K -- "claim + facts" --> P

    C -- "claim + facts" --> P

    I -- "by hash" --> H

    I -- "by hash" --> K

    I -- "by hash" --> C

    D -- "how many, of what" --> K

    D -- "how many, of what" --> C

```

## 2. What it fixes

Research into self-hosted runners (GitHub Actions, GitLab, Buildkite,

Jenkins, Forgejo; 2023–2026) found the same complaints everywhere. Most of

them are fleet problems, not pipeline problems.

| What goes wrong elsewhere | Jetstream |

|---|---|

| Persistent runners leak state; fork PRs and compromised actions reach other jobs' secrets | A fresh microVM per job (or the syscall tier); nothing survives a job |

| A job waits for hours, then fails, with no reason given | Every waiting job says why, in terms of facts (§7) |

| Disk fills mid-job; an OOM kill shows up as a "lost runner" | Disk and memory checked on every claim against what the job needed last time; OOM reported as OOM |

| Images copied to each machine by hand; slow pulls | Images served by hash, fetched by the runner on demand, chunked so a rebuild moves only what changed (§8) |

| Exact-match labels that sprawl | Facts measured by the runner; labels only steer (§7) |

| Lost runners and spot reclaims leave jobs hanging | Requeue on a lost runner or a reclaim notice (§11) |

| Forced runner upgrades with a cut-off date | Runners update themselves between jobs; the contract is versioned |

| Ephemeral runners start cold, 30–60 s and no caches | Firecracker boots in about a second; images and caches stay warm by hash |

| Multi-arch builds through QEMU, 5–30× slower; no RISC-V runner at all | Native runners per architecture, RISC-V included (narvi) |

| No queue, utilisation or cost figures | Queue time, utilisation and cost by job, pool and region, built in (§11) |

## 3. Responsibilities

| | Jetstream | The forge (OxygenHub) |

|---|---|---|

| Pipeline from the repository | | ✔ |

| Runs, jobs, `needs`, the run page | | ✔ |

| Queue, placement, why a job waits | ✔ | shows it |

| Runners, their facts and tokens | ✔ | shows them |

| Pools: hosts, Kubernetes, cloud | ✔ | |

| Images | ✔ (by hash) | |

| Secrets and variables | carries them to the job | ✔ (stores them) |

| Job identity (OIDC) | carries the token | ✔ (issues it) |

| Logs | streams them | ✔ (keeps them) |

| Outputs, reports, receipts | carries them | ✔ (keeps, checks, shows) |

| Checks on a change | | ✔ |

## 4. The contracts

Two contracts, both HTTPS and JSON, versioned from the start.

**Forge ↔ Jetstream.** The forge submits a job and follows it.

- `submit`: the job's spec (sandbox, image hash per architecture, exec,

  outputs, time limit), its **source** (§5), its secrets and OIDC token,

  the forge's run and job ids, and optional steering (`runs-on`,

  `prefer`).

- `events`: a stream per forge of `placed` (where, and why), `started`,

  `log`, `output` (a file or blob), `finished` (exit code, peak disk and

  memory, duration), `receipt` (signed by the runner), and `waiting`

  (why, when nothing fits).

- `cancel`, and `queue` (what is waiting, for the forge's pages).

**Runner ↔ Jetstream.** What runners already do with the hub today, moved:

`claim` (with its facts, §6), `log`, `output`, `finish`, `decline` (with a

reason). A runner talks only to Jetstream, never to a forge: moving from

embedded to external Jetstream is a URL change for it.

```mermaid

sequenceDiagram

    participant Forge as OxygenHub

    participant Jet as Jetstream

    participant R as earth-1

    Forge->>Jet: submit(job, source, secrets, token)

    Note over Jet: queued: needs x86_64, microvm,<br/>~6 GiB disk (from its last run)

    R->>Jet: claim(facts: idle, 41 GiB free, holds image)

    Jet->>Forge: placed on earth-1 (holds the image, idle, score 100)

    Jet->>R: job

    R->>Jet: log, log, output

    Jet->>Forge: log, log, output

    R->>Jet: finish(exit 0, peak 5.2 GiB disk, 3.1 GiB memory)

    Jet->>Forge: finished + receipt

```

## 5. Sources: mounted, or checked out

A job's source says how a runner gets its files.

- **Oxygen** (`oxy://HUB/OWNER/NAME@STATE`): flow-casfs mounts the state

  read-only and reads objects as the job touches them (spec §5.1). Nothing

  is checked out; a job that reads 50 files of a 100,000-file repository

  fetches 50 files. Jobs whose scope a change does not touch keep their

  previous result (spec §6), and receipts name the exact state.

- **Git** (`git+https://…@COMMIT`): the runner clones or fetches the

  commit into the job's disk before it starts, as every CI system does.

Both run through the same fleet, so the difference is measured, not

claimed: the job page shows how long the source took.

```mermaid

flowchart TB

    subgraph O["Oxygen source"]

        O1["job starts"] --> O2["mount state (ms)"] --> O3["read files as used"]

    end

    subgraph G["Git source"]

        G1["clone / fetch commit"] --> G2["check out every file"] --> G3["job starts"]

    end

```

## 6. Facts: runners know what they are

A runner measures itself, and sends what it measured **with every claim**.

Idle runners claim every few seconds, so Jetstream always holds every idle

runner's facts from moments ago; there is no separate audit channel to go

stale.

| Fact | Measured by | When |

|---|---|---|

| Architecture, CPU model, cores | `uname`, `/proc/cpuinfo` | at start |

| Tiers: microvm (KVM opened, not assumed), syscall (Landlock ABI), wasm | probing | at start, then hourly |

| Memory free, disk free where jobs run | `/proc/meminfo`, `statvfs` | every claim |

| Load, jobs running | `/proc/loadavg`, its own state | every claim |

| Images held | its image store | every claim |

| Speed score | a fixed few-second benchmark (hashing, compression, a small compile), normalised so earth-1 ≈ 100 | at start, then about every 15 minutes while idle |

| Where it runs | the pool: host, cluster and node, or cloud, region, instance type, spot | at start |

The score is checked against reality: Jetstream compares each runner's

claimed speed with how fast its jobs actually ran, and ranks a runner

whose real timings drift (thermal throttling, noisy neighbours) by what

it achieved.

## 7. Placement: requirements, ranking, and why

**A job's requirements are mostly derived**, never typed by the user:

- architecture from its image;

- tier from its sandbox;

- disk and memory **learned from its own past runs** (peak plus

  headroom); a new job goes to the roomiest runner the first time;

- an explicit floor when the pipeline gives one.

**Placement** filters, then ranks:

1. **Filter on hard facts:** architecture, tier, enough free disk and

   memory now, steering that requires a pool or label.

2. **Rank on soft ones:** already holds the image (warm), idle, speed

   score, the shortest estimated finish.

3. **A short hold for the best fit:** when the runner asking is a poor

   fit and a better one claimed seconds ago, the job is held a few seconds

   for it, then goes to whoever is next.

4. **The runner has the last word:** it checks again on accept and

   declines with a reason if its disk filled meanwhile.

**Every decision explains itself**, on the job and through the API:

> ran on **earth-1**: holds the image, idle, score 100, estimated 4 min

> **waiting**: needs ~12 GiB disk (its last run used 10.4); plex-1 has 9;

> earth-1 is busy until about 11:40

**Labels only steer.** `runs-on "plex"` pins, `prefer "fast"` biases. A

user who never writes one never needs to.

## 8. Images

- **Content-addressed.** An image is an ext4 root file system named by

  its BLAKE3 hash (`cas://blob/…`), as today. A sandbox may name one per

  architecture.

- **Served by Jetstream.** A runner that lacks an image fetches it by hash

  and verifies it; nobody copies images between machines. Images are

  chunked, so a rebuilt image moves only its changed chunks.

- **Built by Flow.** An image is a job's output: a project's

  `.flow/image/` becomes a job, its image is kept and served, and the

  sandbox's pin follows it (a proposal: "image changed: af3c… → 91be…").

- **Architecture recorded at build**, so an x86 image never reaches a

  RISC-V runner.

## 9. Pools

A **pool** is one place runners come from. All pools report the same

facts and follow the same placement.

```mermaid

flowchart TB

    J["Jetstream"]

    subgraph Hosts["Host pool"]

        E["earth-1<br/>x86_64 · microvm"]

        PL["plex-1<br/>x86_64 · microvm"]

        N["narvi-1<br/>riscv64 · syscall"]

    end

    subgraph K8s["Kubernetes pool"]

        CTL["jet controller"]

        KP1["KVM node pool<br/>Firecracker in pod"]

        KP2["Kata / gVisor<br/>RuntimeClass"]

        KP3["ordinary pods<br/>syscall tier"]

        CTL --> KP1

        CTL --> KP2

        CTL --> KP3

    end

    subgraph Cloud["Cloud pool (europe-west2)"]

        PRV["jet provider"]

        VM1["warm VM, nested KVM<br/>several microVMs"]

        VM2["VM per job<br/>syscall tier"]

        PRV --> VM1

        PRV --> VM2

    end

    J --> Hosts

    J --> CTL

    J --> PRV

```

**Hosts.** Machines you own run `jet runner` (today's `flow runner`).

**Kubernetes.** `jet controller` runs in the cluster, claims for it, and

creates **one pod per job**, the pattern every major CI system uses on

Kubernetes. The tier follows what each node pool offers:

| Node pool offers | Tier |

|---|---|

| `/dev/kvm` through a device plugin (a tainted pool) | microvm: Firecracker inside the pod |

| A Kata or gVisor RuntimeClass (AKS Pod Sandboxing, GKE Sandbox) | syscall, inside the platform's sandbox |

| Nothing special (GKE Autopilot included) | syscall, in an unprivileged pod |

Dedicated KVM hosts beside a cluster stay first class: that is how

microVM CI is run in practice.

**Cloud providers.** `jet provider` creates and removes VMs in a cloud

account, keyless through OIDC (OxygenHub is already an issuer, OxygenHub spec

§13.5):

- **Packing first:** a warm 8–16 vCPU VM with nested KVM runs several

  microVM jobs, saving the 30–60 s boot, the cold disk and per-VM minimum

  charges. Nested KVM: GCP Intel types and N4D, AWS C7i/C8i and newer

  (since 2026), Azure Dsv5.

- **One VM per job** with the syscall tier inside is the fallback where

  nested KVM is not offered.

- **Regions are pinned** per pool and never fall back outside the set:

  for the UK, `europe-west2`, `uksouth`, `eu-west-2`. Every job records

  where it ran.

- **Spot**, with on-demand fallback; a reclaim notice requeues the job.

- **Scale-in** only of idle VMs near the end of their billing increment.

- Build order: GCP, then AWS, then Azure.

The provider interface: `create(n, spec)`, `delete(ids)`, `list()`,

`health(id)`, `capabilities()`, `quota()`.

## 10. Demand and autoscaling

Jetstream answers **demand by pull**, so nothing needs a way into your

network:

- `GET /demand?pool=…` (long-poll): waiting jobs grouped by architecture,

  tier and size, and the age of the oldest. Kubernetes pools scale on it

  through a KEDA scaler; cloud providers through their own loop.

- **One-shot runner tokens** for ephemeral runners and pods: issued for

  one start, never long-lived.

## 11. Reliability and running it

- **Lost runner:** its jobs are requeued after a short grace; a spot

  reclaim notice requeues at once.

- **OOM** is reported as OOM, with the job's peak, and raises its learned

  memory need.

- **Updates:** a runner told of a newer version updates itself between

  jobs. No cut-off date; the contract is versioned and old runners keep

  working until they update.

- **Figures:** queue time (the single most useful CI metric), utilisation

  per runner and pool, and cost per job, pool and region where a provider

  knows its prices.

## 12. Embedded or on its own

```mermaid

flowchart LR

    subgraph A["Embedded (the default)"]

        OH1["OxygenHub process<br/>+ jet inside"]

        R1["runners"] --> OH1

    end

    subgraph B["On its own"]

        OH2["OxygenHub"] --> JS2["jet server"]

        OH3["another OxygenHub"] --> JS2

        R2["runners"] --> JS2

    end

```

- **Embedded:** the same crate in OxygenHub's process, with no

  configuration. This is what OxygenHub does unless told otherwise.

- **On its own:** `[flow] jetstream = "https://jet.example"` in the hub's

  configuration points it at a standalone `jet server`; several hubs may

  share one fleet.

- **Other forges, later:** the forge contract (§4) and Git sources (§5)

  are what an adapter needs, such as one that runs GitLab or GitHub jobs

  on Jetstream runners. That would be a contained piece of work, not a

  rewrite; it is not planned yet.

## 13. Running a pipeline on your own machine

```sh

flow run                     # every job of .flow/pipeline.kdl, on this directory's files

flow run test                # one job, and what it needs

flow run --var X=1 --secret TOKEN=…   # what the hub would supply

```

`flow run` exists today: it runs the pipeline's jobs in the same sandboxes

CI uses, before anything is pushed. With Jetstream it becomes an

in-process Jetstream with one local runner, the same placement and the

same code path as CI, so passing locally means passing in CI. With images

served by Jetstream (§8), the first local run fetches the pinned image by

hash instead of needing it imported by hand.

## 14. Security

- **Isolation per job:** a microVM where KVM exists, the syscall tier

  (Landlock and seccomp) elsewhere, a platform sandbox where a cluster

  offers one. No job ever shares a Docker daemon or a host socket.

- **Secrets** are mounted as files in the job and masked in logs (spec

  §9), never written to a runner's disk outside the job.

- **Cloud VMs** have no cloud identity of their own, the metadata service

  is blocked from inside job microVMs, and job subnets accept nothing

  inbound.

- **Runner tokens** are per runner, and one-shot for ephemeral ones.

## 15. Order of work

1. **The boundary, inside the hub:** queue, placement and runners behind

   the Jetstream contract in a `jet` crate, embedded; no behaviour change,

   CI proves it.

2. **Images served by Jetstream**, fetched by hash; images built by Flow

   jobs.

3. **Facts on every claim, learned requirements, placement with its

   explanation**, on the job page and the queue.

4. **narvi-1**, the first runner brought up this way (riscv64, syscall).

5. **`jet server`**, OxygenHub pointing at it.

6. **The Kubernetes controller**, with a KEDA scaler on `/demand`.

7. **Cloud providers:** GCP, then AWS, then Azure.

## 16. Open questions

- Whether artifacts and report files travel through Jetstream or go

  straight from the runner to the forge (today: straight to the hub).

- Multi-tenancy in a shared `jet server`: pools per organisation, and

  who may submit to which.

- How long the best-fit hold (§7) should be, and whether it should learn.

- Receipts: the runner signs; whether Jetstream countersigns where it

  placed the job.