Jetstream

Design and architecture decision, 2026-10-11 (oxygenflow#38). Builds on spec §11, runner capabilities and placement (#36). Crate and CLI: jet.

Flow says what to run: a pipeline of hermetic jobs, each in a sandbox, with its outputs, reports and signed receipt. Jet carries each job to where it should run: a fleet of runners on your own machines, on Kubernetes or in the cloud, that measure themselves, explain every choice they make, and fetch what they need on their own.

1. The decision

Context. Today OxygenHub holds the job queue and talks to runners directly (/api/v1/flow/claim, logs, outputs, finish). Runners are configured by hand: images are copied onto each machine, labels steer jobs, and a job that cannot run anywhere simply waits. Every self-hosted CI system has the same shape and the same problems (§2), and none of it is specific to Oxygen.

Decision. The fleet becomes a service of its own, Jetstream:

• Jetstream owns the queue, the runners and their measured facts, placement and its explanations, pools (hosts, Kubernetes, cloud providers) and images. It knows nothing about repositories.
• The forge owns pipelines, runs, checks, receipts, artifacts, reports and the UI. It hands Jetstream jobs and receives their events.
• OxygenHub embeds Jetstream in its own process by default, so nothing changes for anyone who never configures it, and can instead point at a standalone jet server shared by several forges.
• Sources are pluggable: an Oxygen repository is mounted lazily through flow-casfs; a Git repository is checked out.
• flow run runs a pipeline on your own machine through an in-process Jetstream with one local runner: the same code path as CI.

Consequences.

• Jetstream can be reused by others, and could later run jobs for forges that are not OxygenHub (§12) without a rewrite.
• One more boundary to keep stable: the forge contract (§4) is versioned.
• Oxygen's advantage becomes visible: the same fleet runs both kinds of source, and an Oxygen job starts without a checkout (§5).

flowchart LR
    subgraph Forges
        OH["OxygenHub<br/>pipelines · runs · checks<br/>receipts · reports"]
        OTHER["another forge<br/>(an adapter, later)"]
    end
    subgraph JS["Jetstream (jet)"]
        Q["queue"]
        P["placement<br/>+ why"]
        F["runner facts"]
        I["images"]
        D["demand"]
    end
    subgraph Pools
        H["hosts<br/>earth-1 · plex-1 · narvi-1"]
        K["Kubernetes<br/>controller, pod per job"]
        C["cloud<br/>GCP · AWS · Azure"]
    end
    OH -- "jobs" --> Q
    OTHER -.->|jobs| Q
    Q -- "events: logs, outputs,<br/>finish, receipt" --> OH
    Q --> P
    F --> P
    H -- "claim + facts" --> P
    K -- "claim + facts" --> P
    C -- "claim + facts" --> P
    I -- "by hash" --> H
    I -- "by hash" --> K
    I -- "by hash" --> C
    D -- "how many, of what" --> K
    D -- "how many, of what" --> C

2. What it fixes

Research into self-hosted runners (GitHub Actions, GitLab, Buildkite, Jenkins, Forgejo; 2023–2026) found the same complaints everywhere. Most of them are fleet problems, not pipeline problems.

3. Responsibilities

4. The contracts

Two contracts, both HTTPS and JSON, versioned from the start.

Forge ↔ Jetstream. The forge submits a job and follows it.

• submit: the job's spec (sandbox, image hash per architecture, exec, outputs, time limit), its source (§5), its secrets and OIDC token, the forge's run and job ids, and optional steering (runs-on, prefer).
• events: a stream per forge of placed (where, and why), started, log, output (a file or blob), finished (exit code, peak disk and memory, duration), receipt (signed by the runner), and waiting (why, when nothing fits).
• cancel, and queue (what is waiting, for the forge's pages).

Runner ↔ Jetstream. What runners already do with the hub today, moved: claim (with its facts, §6), log, output, finish, decline (with a reason). A runner talks only to Jetstream, never to a forge: moving from embedded to external Jetstream is a URL change for it.

sequenceDiagram
    participant Forge as OxygenHub
    participant Jet as Jetstream
    participant R as earth-1
    Forge->>Jet: submit(job, source, secrets, token)
    Note over Jet: queued: needs x86_64, microvm,<br/>~6 GiB disk (from its last run)
    R->>Jet: claim(facts: idle, 41 GiB free, holds image)
    Jet->>Forge: placed on earth-1 (holds the image, idle, score 100)
    Jet->>R: job
    R->>Jet: log, log, output
    Jet->>Forge: log, log, output
    R->>Jet: finish(exit 0, peak 5.2 GiB disk, 3.1 GiB memory)
    Jet->>Forge: finished + receipt

5. Sources: mounted, or checked out

A job's source says how a runner gets its files.

• Oxygen (oxy://HUB/OWNER/NAME@STATE): flow-casfs mounts the state read-only and reads objects as the job touches them (spec §5.1). Nothing is checked out; a job that reads 50 files of a 100,000-file repository fetches 50 files. Jobs whose scope a change does not touch keep their previous result (spec §6), and receipts name the exact state.
• Git (git+https://…@COMMIT): the runner clones or fetches the commit into the job's disk before it starts, as every CI system does.

Both run through the same fleet, so the difference is measured, not claimed: the job page shows how long the source took.

flowchart TB
    subgraph O["Oxygen source"]
        O1["job starts"] --> O2["mount state (ms)"] --> O3["read files as used"]
    end
    subgraph G["Git source"]
        G1["clone / fetch commit"] --> G2["check out every file"] --> G3["job starts"]
    end

6. Facts: runners know what they are

A runner measures itself, and sends what it measured with every claim. Idle runners claim every few seconds, so Jetstream always holds every idle runner's facts from moments ago; there is no separate audit channel to go stale.

The score is checked against reality: Jetstream compares each runner's claimed speed with how fast its jobs actually ran, and ranks a runner whose real timings drift (thermal throttling, noisy neighbours) by what it achieved.

7. Placement: requirements, ranking, and why

A job's requirements are mostly derived, never typed by the user:

• architecture from its image;
• tier from its sandbox;
• disk and memory learned from its own past runs (peak plus headroom); a new job goes to the roomiest runner the first time;
• an explicit floor when the pipeline gives one.

Placement filters, then ranks:

1. Filter on hard facts: architecture, tier, enough free disk and memory now, steering that requires a pool or label.
2. Rank on soft ones: already holds the image (warm), idle, speed score, the shortest estimated finish.
3. A short hold for the best fit: when the runner asking is a poor fit and a better one claimed seconds ago, the job is held a few seconds for it, then goes to whoever is next.
4. The runner has the last word: it checks again on accept and declines with a reason if its disk filled meanwhile.

Every decision explains itself, on the job and through the API:

ran on earth-1: holds the image, idle, score 100, estimated 4 min

waiting: needs ~12 GiB disk (its last run used 10.4); plex-1 has 9; earth-1 is busy until about 11:40

Labels only steer. runs-on "plex" pins, prefer "fast" biases. A user who never writes one never needs to.

8. Images

• Content-addressed. An image is an ext4 root file system named by its BLAKE3 hash (cas://blob/…), as today. A sandbox may name one per architecture.
• Served by Jetstream. A runner that lacks an image fetches it by hash and verifies it; nobody copies images between machines. Images are chunked, so a rebuilt image moves only its changed chunks.
• Built by Flow. An image is a job's output: a project's .flow/image/ becomes a job, its image is kept and served, and the sandbox's pin follows it (a proposal: "image changed: af3c… → 91be…").
• Architecture recorded at build, so an x86 image never reaches a RISC-V runner.

8.1 The swarm: runners share images

Runners that are configured for it fetch image chunks from each other as well as from the hub (oxygenflow #47, oxygenhub #144). Fifty fresh runners asking for the same 2 GiB image then cost the hub about one copy, not fifty, and a site behind a thin link pulls each chunk across it once.

• Trust comes from the hashes. The manifest comes from the hub, and every chunk is checked against its name before it is kept. A peer can fail to help; it can never change what a job runs.
• The hub is the tracker. A runner reports its swarm address, its site and its certificate's fingerprint with every claim. POST /api/v1/flow/swarm/announce {image, state} (fetching, done or stopped) returns the peers in the same site that hold the image or are fetching it, and a ticket.
• Tickets admit peers. A ticket names the runner, its site and an expiry (15 minutes), signed with the hub key that /.well-known/oxyhub-key publishes. A runner shows its ticket once to each peer (POST /swarm/v1/hello) and gets a session for the ticket's life; no other runner and no account can take chunks from a peer.
• Encrypted, and pinned by the hub. Each runner makes its own TLS certificate on first use (swarm/cert.der, its key readable by the runner alone) and keeps it, so its fingerprint (BLAKE3 of the certificate) holds across restarts. The tracker hands each peer out with that fingerprint, and a fetcher talks only to a peer showing that certificate and proving its key in the handshake; no CA is involved. A peer at another address, or a stranger at a peer's, gets nothing and sees nothing: chunks, tickets and sessions all travel inside TLS.
• The peer protocol is HTTPS on the runner's swarm address:
  • POST /swarm/v1/hello with the ticket: a session token.
  • GET /swarm/v1/images/{image}/have: complete, or two bit maps over the manifest's needed list (Manifest::needed), in its order: the chunks it has, then the chunks it is fetching now.
  • GET /swarm/v1/chunks/{chunk}: the chunk's bytes, from the chunk cache or read out of a whole image; 503 when the peer is at its connection limit, and the fetcher asks it again shortly.
• Rarest first. The fetcher refreshes every peer's maps twice a second and asks each peer, four requests at a time, for the chunk the fewest peers hold. The hub is a seed that holds everything: it is asked only for chunks no peer has or is fetching, or any chunk once the peers have made no progress for two seconds. Runners that start together so split the hub's chunks between them rather than each taking them all. When every missing chunk is already asked for, idle workers ask a second source for it (the endgame; the hub joins it only once the peers stall). A peer that fails is dropped and its chunks go back to the pool.
• Serving while fetching. A chunk is offered to others as soon as it is checked and kept, so runners starting together share the load.
• Bounded. A runner serves at most 16 connections and upload_mib_s (default 50 MiB/s), so serving never starves its own jobs.
• Configured, never assumed. Without a [runner.swarm] section a runner neither listens nor asks peers. Peers are not offered across sites, so a site is the network its runners can reach each other on.

In flow.toml (or as [swarm] in runner.toml, which stands in for the [runner] section):

[runner.swarm]
listen = "0.0.0.0:7401"        # where peers reach this runner
advertise = "10.0.0.5:7401"    # the address peers are given (default: listen)
site = "home"                  # peers are only ever in the same site
upload_mib_s = 50

8.1.1 Images only runners hold (oxygenflow#94, oxygenhub#342)

Until now an image had to be sent to the hub (flow rootfs push) before any runner but the one that imported it could take its jobs: the hub hands out the manifest, and placement offers a job to a runner that fetches only when the hub holds its image. On 2026-10-25 the Rust image was re-pinned and not pushed, and for a day every Rust job queued on the one runner that had imported it while the others sat idle.

• Runners say what they hold. At start, every ten minutes and after an import, a runner sends the hashes of the images it holds (POST /flow/images/held {images}); the hub answers with those it keeps no manifest for ({wanted}), and the runner sends each manifest alone (PUT /flow/images/{hash}/held, a few MB even for a 14 GB image): no chunks. (Not in the claim: an idle runner's claim gets no answer at all.)
• The hub keeps it as peer-held. It checks the manifest is well formed and names the hash it is filed under, and records the image as held by runners, not by itself: it cannot read the image to check it. A pushed image (flow rootfs push, verified whole) replaces a peer-held record.
• Trust still comes from the hashes. The pipeline pins the image's hash; a fetcher asks for the manifest by that hash, checks every chunk against its name, and checks the assembled image against the pin before it keeps it (jet::images::assemble). A wrong manifest can make a fetch fail; it cannot change what a job runs. Only runner tokens may send one.
• Placement counts peers. A job whose image the hub holds only as peer-held goes to a runner that fetches only when a runner of the same swarm site holds the image, and says so when none does ("held only by earth-1, in another site").
• The hub is no seed for it. The announce answer says whether the hub holds the chunks (seed); when it does not, the fetcher asks peers only, and a fetch with no peer left fails with that reason rather than after the hub refused chunk after chunk.

8.2 Runners update themselves

Proposed 2026-10-25 from Andy's experience: keeping a large fleet's runners up to date by hand is one of the biggest costs of running CI. Runners fetch new runner builds the way they fetch images, from each other through the swarm, check them as strictly as a release, and swap themselves in without losing work, under a policy the organisation sets, testing on a few runners before the rest.

A runner build is a flow binary from an OxygenFlow release, one per target (linux-x86_64, linux-arm64, linux-riscv64; Windows and macOS when those runners exist), built by Flow from the tagged state with a signed receipt, as oxy's and OxygenHub's releases are. The hub serves it as a blob with a chunk manifest, so the swarm (§8.1) carries it unchanged: one runner of a site fetches each chunk from the hub and the rest from it.

Off unless chosen, nearest wins. The policy is set for the site, an organisation, a test group and a runner; the nearest setting wins:

• updates: off (the default) or on;
• channel: stable, canary, or a pinned version (0.3.4);
• window: hours in which a runner may update (outside them it waits);
• dry_run: fetch and check, say what would happen, do not swap.

A runner's own flow.toml has the last word: [runner.update] allow = false refuses every update whatever the hub says, so a site can hold a machine back while it tests.

Test groups and waves. A test group is a named set of runners: chosen by name, by label or by pool. A rollout is a release and its waves, in order; each wave is a group and its share of it, a percentage or chosen runners. A percentage is taken by a stable hash of each runner's id, so "10% of build-farm" is the same machines every rollout, the ones the organisation watches. A wave starts when the one before has been healthy for its soak time (say 24 hours) or when an admin promotes it. For example: canary (two chosen runners), then build-farm at 10%, then build-farm at 100%, then everyone.

Healthy means the updated runners came back on the new version within ten minutes, claim jobs, and their jobs' infrastructure errors (lost, error, declines) over the soak are no worse than the same runners' on the version before, over enough jobs to tell. A wave that is not healthy halts the rollout, and its runners roll back. Job failures from the pipelines themselves do not count.

On the runner, in order:

1. The hub's answer to a claim or heartbeat names an update when the policy says so: the version, its blob and manifest, its receipt.
2. Fetch the blob through the swarm (hub as seed); every chunk checked against the manifest from the hub.
3. Check it as a release: its BLAKE3 matches the release asset the hub attests; the receipt is signed by an owned build runner (trust classes, #79) and names the tagged state; the target is this machine's. A peer can only serve bytes; it cannot change what is installed.
4. Stage it (bin/flow.new) and run its self-check (flow.new tiers, --version).
5. Drain with notice (leave_within, §11.5.2 of the spec): finish what fits, hand back the rest with its checkpoints.
6. Swap: flow → flow.prev, flow.new → flow, and restart.
7. Prove it: the first claim reports the new version. A runner that will not start, or that the hub does not hear from within ten minutes, goes back to flow.prev (a start counter in the runner's home: three failed starts after an update restore the previous build).
8. The runner's page shows its version, channel, group, last update and why; the audit log records every update and rollback.

Phones take runner updates with the app's releases (Android updates an app only through its store or an install), so they follow the app.

Order of work: (1) the runner side, with a local source (flow runner update --from FILE): check, stage, drain, swap, prove, roll back (done, #85: spec §11.5.3, Linux and Windows); runners asking the hub and taking its offer (done); (2) the policy on the hub (opt in and out, channels, windows, dry run) and the update a runner is offered (done: oxygenhub#322, for the site and for each organisation's own runners, GET /api/v1/flow/runner/update); (3) test groups (done: chosen or a percentage), waves and health (a build that rolls back on one runner is held for every other, oxygenhub#326; waves that advance by themselves after a soak are not built yet); (4) the swarm as transport; (5) Admin › Runners › Updates (done) and oh runner update.

9. Pools

A pool is one place runners come from. All pools report the same facts and follow the same placement.

flowchart TB
    J["Jetstream"]
    subgraph Hosts["Host pool"]
        E["earth-1<br/>x86_64 · microvm"]
        PL["plex-1<br/>x86_64 · microvm"]
        N["narvi-1<br/>riscv64 · syscall"]
    end
    subgraph K8s["Kubernetes pool"]
        CTL["jet controller"]
        KP1["KVM node pool<br/>Firecracker in pod"]
        KP2["Kata / gVisor<br/>RuntimeClass"]
        KP3["ordinary pods<br/>syscall tier"]
        CTL --> KP1
        CTL --> KP2
        CTL --> KP3
    end
    subgraph Cloud["Cloud pool (europe-west2)"]
        PRV["jet provider"]
        VM1["warm VM, nested KVM<br/>several microVMs"]
        VM2["VM per job<br/>syscall tier"]
        PRV --> VM1
        PRV --> VM2
    end
    J --> Hosts
    J --> CTL
    J --> PRV

Hosts. Machines you own run jet runner (today's flow runner).

Kubernetes. jet controller runs in the cluster, claims for it, and creates one pod per job, the pattern every major CI system uses on Kubernetes. The tier follows what each node pool offers:

Dedicated KVM hosts beside a cluster stay first class: that is how microVM CI is run in practice.

Cloud providers. jet provider creates and removes VMs in a cloud account, keyless through OIDC (OxygenHub is already an issuer, OxygenHub spec §13.5):

• Packing first: a warm 8–16 vCPU VM with nested KVM runs several microVM jobs, saving the 30–60 s boot, the cold disk and per-VM minimum charges. Nested KVM: GCP Intel types and N4D, AWS C7i/C8i and newer (since 2026), Azure Dsv5.
• One VM per job with the syscall tier inside is the fallback where nested KVM is not offered.
• Regions are pinned per pool and never fall back outside the set: for the UK, europe-west2, uksouth, eu-west-2. Every job records where it ran.
• Spot, with on-demand fallback; a reclaim notice requeues the job.
• Scale-in only of idle VMs near the end of their billing increment.
• Build order: GCP, then AWS, then Azure.

The provider interface: create(n, spec), delete(ids), list(), health(id), capabilities(), quota().

10. Demand and autoscaling

Jetstream answers demand by pull, so nothing needs a way into your network:

• GET /demand?pool=… (long-poll): waiting jobs grouped by architecture, tier and size, and the age of the oldest. Kubernetes pools scale on it through a KEDA scaler; cloud providers through their own loop.
• One-shot runner tokens for ephemeral runners and pods: issued for one start, never long-lived.

11. Reliability and running it

• Lost runner: its jobs are requeued after a short grace; a spot reclaim notice requeues at once.
• OOM is reported as OOM, with the job's peak, and raises its learned memory need.
• Updates: a runner told of a newer version updates itself between jobs. No cut-off date; the contract is versioned and old runners keep working until they update.
• Figures: queue time (the single most useful CI metric), utilisation per runner and pool, and cost per job, pool and region where a provider knows its prices.

12. Embedded or on its own

flowchart LR
    subgraph A["Embedded (the default)"]
        OH1["OxygenHub process<br/>+ jet inside"]
        R1["runners"] --> OH1
    end
    subgraph B["On its own"]
        OH2["OxygenHub"] --> JS2["jet server"]
        OH3["another OxygenHub"] --> JS2
        R2["runners"] --> JS2
    end

• Embedded: the same crate in OxygenHub's process, with no configuration. This is what OxygenHub does unless told otherwise.
• On its own: [flow] jetstream = "https://jet.example" in the hub's configuration points it at a standalone jet server; several hubs may share one fleet.
• Other forges, later: the forge contract (§4) and Git sources (§5) are what an adapter needs, such as one that runs GitLab or GitHub jobs on Jetstream runners. That would be a contained piece of work, not a rewrite; it is not planned yet.

13. Running a pipeline on your own machine

flow run                     # every job of .flow/pipeline.kdl, on this directory's files
flow run test                # one job, and what it needs
flow run --var X=1 --secret TOKEN=…   # what the hub would supply

flow run exists today: it runs the pipeline's jobs in the same sandboxes CI uses, before anything is pushed. With Jetstream it becomes an in-process Jetstream with one local runner, the same placement and the same code path as CI, so passing locally means passing in CI. With images served by Jetstream (§8), the first local run fetches the pinned image by hash instead of needing it imported by hand.

14. Security

• Isolation per job: a microVM where KVM exists, the syscall tier (Landlock and seccomp) elsewhere, a platform sandbox where a cluster offers one. No job ever shares a Docker daemon or a host socket.
• Secrets are mounted as files in the job and masked in logs (spec §9), never written to a runner's disk outside the job.
• Cloud VMs have no cloud identity of their own, the metadata service is blocked from inside job microVMs, and job subnets accept nothing inbound.
• Runner tokens are per runner, and one-shot for ephemeral ones.

15. Order of work

1. The boundary, inside the hub: queue, placement and runners behind the Jetstream contract in a jet crate, embedded; no behaviour change, CI proves it.
2. Images served by Jetstream, fetched by hash; images built by Flow jobs.
3. Facts on every claim, learned requirements, placement with its explanation, on the job page and the queue.
4. narvi-1, the first runner brought up this way (riscv64, syscall).
5. jet server, OxygenHub pointing at it.
6. The Kubernetes controller, with a KEDA scaler on /demand.
7. Cloud providers: GCP, then AWS, then Azure.

16. Open questions

• Whether artifacts and report files travel through Jetstream or go straight from the runner to the forge (today: straight to the hub).
• Multi-tenancy in a shared jet server: pools per organisation, and who may submit to which.
• How long the best-fit hold (§7) should be, and whether it should learn.
• Receipts: the runner signs; whether Jetstream countersigns where it placed the job.