Jetstream

Flow says what to run. Jet carries it to where it should run.

Jetstream (crate and CLI jet) is OxygenFlow's runner fleet, taken out of OxygenHub as a service of its own. OxygenHub embeds it by default, so nothing changes unless you want it to. It can also run on its own (jet server) and be shared by several hubs. This page is the overview; the full design is docs/jetstream.md (#38).

Status: designed 2026-10-11, being built. Today's runners already match architecture and tier and decline with a reason; the rest is next.

The shape

flowchart LR
    subgraph Forges
        OH["OxygenHub<br/>pipelines · runs · checks<br/>receipts · reports"]
        OTHER["another forge<br/>(an adapter, later)"]
    end
    subgraph JS["Jetstream (jet)"]
        Q["queue"]
        P["placement<br/>+ why"]
        F["runner facts"]
        I["images"]
        D["demand"]
    end
    subgraph Pools
        H["hosts<br/>earth-1 · plex-1 · narvi-1"]
        K["Kubernetes<br/>controller, pod per job"]
        C["cloud<br/>GCP · AWS · Azure"]
    end
    OH -- "jobs" --> Q
    OTHER -.->|jobs| Q
    Q -- "events: logs, outputs,<br/>finish, receipt" --> OH
    Q --> P
    F --> P
    H -- "claim + facts" --> P
    K -- "claim + facts" --> P
    C -- "claim + facts" --> P
    I -- "by hash" --> H
    I -- "by hash" --> K
    I -- "by hash" --> C
    D -- "how many, of what" --> K
    D -- "how many, of what" --> C

• Jetstream owns the queue, the runners and their facts, placement and why, pools, and images.
• The forge owns pipelines, runs, checks, receipts, artifacts and reports.

Why

Self-hosted runners everywhere suffer from the same problems:

• state leaking between jobs;
• jobs stuck with no reason given;
• disks filling mid-job;
• images copied around by hand;
• label sprawl;
• lost runners;
• forced upgrades;
• cold starts;
• no RISC-V.

Jetstream answers each of these: see the table in the design, §2.

Runners know what they are

Every claim carries what the runner just measured:

• free disk and memory;
• load;
• the images it holds;
• its tiers, with KVM actually opened, not assumed.

A short benchmark keeps a speed score, checked against how fast its jobs really ran.

Placement explains itself

A job's requirements are derived, not typed:

• architecture from its image;
• tier from its sandbox;
• disk and memory learned from its own past runs.

Jetstream filters on hard facts, then ranks warm, idle and fast runners first. It always says why:

ran on earth-1: holds the image, idle, score 100, estimated 4 min

waiting: needs ~12 GiB disk (its last run used 10.4); plex-1 has 9; earth-1 is busy until about 11:40

Labels only steer. If you never write one, you never need to.

Images by hash

A runner that lacks a job's image fetches it from Jetstream and checks its hash. Images are chunked, so a rebuild moves only what changed. Images are built by Flow jobs, per architecture. Nobody copies images between machines.

Pools: hosts, Kubernetes, cloud

flowchart TB
    J["Jetstream"]
    subgraph Hosts["Host pool"]
        E["earth-1<br/>x86_64 · microvm"]
        PL["plex-1<br/>x86_64 · microvm"]
        N["narvi-1<br/>riscv64 · syscall"]
    end
    subgraph K8s["Kubernetes pool"]
        CTL["jet controller"]
        KP1["KVM node pool<br/>Firecracker in pod"]
        KP2["Kata / gVisor<br/>RuntimeClass"]
        KP3["ordinary pods<br/>syscall tier"]
        CTL --> KP1
        CTL --> KP2
        CTL --> KP3
    end
    subgraph Cloud["Cloud pool (europe-west2)"]
        PRV["jet provider"]
        VM1["warm VM, nested KVM<br/>several microVMs"]
        VM2["VM per job<br/>syscall tier"]
        PRV --> VM1
        PRV --> VM2
    end
    J --> Hosts
    J --> CTL
    J --> PRV

• Kubernetes: a jet controller creates one pod per job. The tier follows what each node pool offers: Firecracker where pods get /dev/kvm, the cluster's Kata or gVisor sandbox, or the syscall tier in an ordinary pod.
• Cloud: a jet provider runs warm VMs with nested KVM, packing several microVM jobs on each, keyless through OIDC.
  • Regions are pinned: a UK pool never leaves europe-west2, uksouth or eu-west-2.
  • Spot: a spot reclaim requeues the job.

Oxygen sources are mounted, Git sources checked out

flowchart TB
    subgraph O["Oxygen source"]
        O1["job starts"] --> O2["mount state (ms)"] --> O3["read files as used"]
    end
    subgraph G["Git source"]
        G1["clone / fetch commit"] --> G2["check out every file"] --> G3["job starts"]
    end

The same fleet can run jobs from an Oxygen repository or a Git repository. An Oxygen job starts without a checkout, reading only the files it touches, and the job page shows the difference.

Embedded or on its own

flowchart LR
    subgraph A["Embedded (the default)"]
        OH1["OxygenHub process<br/>+ jet inside"]
        R1["runners"] --> OH1
    end
    subgraph B["On its own"]
        OH2["OxygenHub"] --> JS2["jet server"]
        OH3["another OxygenHub"] --> JS2
        R2["runners"] --> JS2
    end

Order of work

1. The boundary inside the hub, with no change in behaviour.
2. Images served by hash.
3. Facts on every claim, learned requirements, placement and why.
4. narvi-1, the first RISC-V runner.
5. jet server.
6. The Kubernetes controller.
7. GCP, then AWS, then Azure.