# OxygenFlow M0: first pipeline on Earth

| | |

|---|---|

| **Status** | Plan, 2026-09-28 |

| **Implements** | a subset of `docs/spec.md` (Draft 0.1) |

| **Goal** | A push to a repository on OxygenHub (Earth) runs `.flow/pipeline.kdl` in Firecracker microVMs, and the hub shows the run as a graph of jobs with live status and logs. |

M0 is the smallest slice that exercises the real architecture: a typed

pipeline compiled to a canonical form, host-side execution in the default

isolation tier, receipts computed by the host, and results reported to the

hub as checks. Everything M0 leaves out is listed at the end.

## 1. Pipeline subset

The file is `.flow/pipeline.kdl` in the state being built (KDL 2.0). M0

accepts:

```kdl

schema-version 1

sandbox "alpine" {

    runtime "microvm"            // "microvm" (default) | "syscall"; "wasm" is refused in M0

    rootfs "cas://blob/<64 hex>"  // BLAKE3 of an ext4 image the runner holds

    memory-limit "1GiB"          // KiB, MiB, GiB

    cpu-limit 2

    network "isolated"           // "isolated" or "loopback" in M0

    env { PATH "/opt/tool/bin:/usr/bin:/bin" }  // every job's starting environment

}

job "lint" {

    sandbox "alpine"

    scope "src/**" "README.md"   // recorded; the job runs on every push in M0

    env { LANG "C.UTF-8" }

    exec "grep" "-rn" "TODO" "src"

}

job "test" {

    sandbox "alpine"

    depends-on "lint"

    exec "sh" "-eu" "test.sh"

}

```

Rules, all checked by `flow check` and again by the hub before anything

runs (spec §2.2 I2, §3.2):

- Unknown nodes, arguments or properties are errors, with the line.

- Every string stays a string; `cpu-limit` is an integer from 1 to 16;

  `memory-limit` parses to 64 MiB to 16 GiB.

- A job names a declared sandbox, has a non-empty `exec` of literal

  strings, and `depends-on` only declared jobs; the graph has no cycle.

- Names are 1 to 64 characters from `[A-Za-z0-9._-]`.

- `exec` becomes `argv` unchanged: metacharacters are bytes (spec §10.1).

  A job that wants a shell names one, as `sh` above; nothing adds one.

- `publish` and a job's `attest` are recognised and refused with "not in

  M0", so a pipeline written for the full spec fails clearly.

- `secret "token" { mount-path "/run/secrets/token" }` on a job (up to 8)

  names a secret the hub holds for the repository (its own, its

  organisation's or the site's; oxygenhub#107) and where the job reads it:

  a clean path under `/run/secrets/`. The claim carries each as `secrets`

  and `flow-init` writes it there, 0400 and the job's user's, on a tmpfs

  of its own, before the job starts; it is never in the environment, the

  receipt or an artifact (spec §9). Locally, `flow run --secret

  NAME=VALUE` supplies it. `vault-ref` is not in M0: the hub is the vault.

- `arch "riscv64"` on a sandbox asks for that architecture (spec §11.3);

  without it the rootfs image's decides. `flow rootfs import` reads the

  image's architecture from its shell's ELF header, refuses an image for

  another architecture than the runner's, and records it in

  `cas/<hash>.json`; a claim reports each image's architecture

  (`image_archs`) so the hub never offers a job to a runner of the wrong

  one (spec §11.2).

- `identity "sts.amazonaws.com" { mount-path "/run/secrets/aws-token" }` on

  a job (up to 4) asks the hub for an OIDC token for that audience, naming

  the repository, bookmark, run and job (OxygenHub spec §13.5,

  oxygenhub#103); it comes with the claim and is mounted exactly as a

  secret is. A local run has no hub, so `flow run` says the file will be

  missing.

- `output { artifact "oxy" { path "/src/target/release/oxy" } }` declares

  up to 16 **artifacts** (spec §5.2). The path is clean, absolute, under

  `/src`, and not on a tmpfs mount, which is gone when the job ends. After

  a job passes, the runner reads each artifact from the job's disks with

  `debugfs`: the workspace, or the `ephemeral://disk` mount holding it. An

  artifact is at most 512 MiB. The runner uploads each one with

  `POST /api/v1/flow/jobs/{id}/artifacts/{name}` (raw bytes). The hub keeps

  it as a blob of the repository, and the receipt's `outputs` name those

  blobs in artifact order. A declared artifact that cannot be kept fails

  the job, and the job gets no receipt.

- `mount` is accepted for **ephemeral scratch** only (up to 4 per job):

  `mount { source "ephemeral://disk"; target "/src/target"; size "8GiB" }`.

  `ephemeral://tmpfs` is memory in the guest and counts against its

  memory-limit; `ephemeral://disk` is a fresh sparse ext4 disk on the

  runner, attached after the workspace, for build trees larger than

  memory. The target is a clean absolute path strictly under `/src` or

  `/tmp`, mounts may not nest, sizes run from 16 MiB to 64 GiB, and any

  other source (`oxy://workspace` and the rest) or setting (`readonly`,

  `include`) is refused as not in M0. Both are empty when the job starts

  and deleted when it ends, so nothing passes between jobs. Mounts are

  part of the job's canonical form (field 6, present only when a job has

  mounts, so pipelines without them keep their hashes) and of the

  receipt's environment.

- A sandbox's `env` is where each of its jobs' environments starts; a

  job's own `env` overrides it name by name. It is part of the sandbox's

  canonical form (field 6, present only when set). Below both is the

  image's own environment: `flow rootfs import --env NAME=value`

  (repeatable) writes `/etc/flow/env` into the image, for what a

  Dockerfile's `ENV` sets and `docker export` loses, such as a toolchain's

  `PATH`. It is part of the image, so of its hash. Under all three are

  `PATH` (the usual directories) and `HOME=/tmp`.

The pipeline compiles to a canonical CBOR job graph (sorted keys, sandboxes

and jobs in name order). Its BLAKE3 is the `pipeline_hash` used in receipts

and shown by the hub.

## 2. Tier 2: Firecracker (default)

Per job the runner:

1. Verifies the rootfs image: `BLAKE3(file)` must equal the pinned hash.

   Images live in `$FLOW_HOME/cas/<hash>.ext4`; an unknown hash fails the

   job ("rootfs not on this runner").

2. Materializes the state at its heads with `oxy` into a scratch

   directory (a clone at the state), removes `.oxy`, and builds a

   workspace image with `mkfs.ext4 -d` sized to the tree plus 512 MiB.

3. Boots Firecracker with no network device (so `isolated` holds by

   construction; `loopback` only brings up the guest's `lo`, so tests can

   serve and connect on 127.0.0.1 while nothing leaves the guest), `vcpu_count` and `mem_size_mib` from the sandbox, the

   pinned guest kernel, the rootfs as a **read-only** `vda`, the

   workspace as a read-write `vdb`, and an initramfs holding `flow-init`.

4. `flow-init` (a static binary, PID 1 in the guest) mounts `/proc`,

   `/sys`, `/dev`, the rootfs at `/newroot` read-only, `vdb` at

   `/newroot/src`, and tmpfs at `/newroot/tmp` and `/newroot/run`, then the

   job's scratch mounts (creating their mount points, owned by the job's

   user); it

   switches root, then runs `argv` (its program found in the guest's `PATH` by

   `flow-init` itself) with exactly the job's `env` (plus

   `PATH`, `HOME=/tmp`) in `/src` as uid 1000, with a job timeout.

   `/dev` gets the links devtmpfs lacks, `fd`, `stdin`, `stdout` and

   `stderr` into `/proc/self/fd`, so shell process substitution works. A job

   is told what it builds: `FLOW_REPO`, `FLOW_STATE` (the hub's state hash),

   `FLOW_HEADS` (full hashes joined with `+`), `FLOW_BOOKMARK`, `FLOW_RUN`

   (the run's number in its repository) and `FLOW_JOB`; `flow run` sets

   `FLOW_JOB` and `FLOW_RUN=local`. After them come the **variables** the

   hub holds for the repository (site, organisation, repository; the

   claim's `env`; `flow run --var NAME=VALUE` locally), never secrets. A

   variable the pipeline sets keeps its value. They are part of the job's

   environment, so of its receipt.

5. `flow-init` frames the child's output on the serial console as

   `\x1eO <line>` (stdout) and `\x1eE <line>` (stderr), then

   `\x1eX <code>`, and powers off. The kernel runs `quiet` so the host sees

   only frames; anything else is kept as `console` records.

6. The host reads frames, streams them as log records, and kills the VM

   at the timeout (the guest cannot extend it). Scratch files are deleted

   afterwards. Guest memory is released with the process.

The guest kernel, `flow-init` and Firecracker itself are pinned by BLAKE3

in the runner's configuration; their hashes and the rootfs hash make the

receipt's `environment` together with `argv` and `env`.

## 3. Tier 1: syscall sandbox

`runtime "syscall"`: a child in new user, mount, PID, network, IPC and

UTS namespaces (no interfaces but loopback), the rootfs image's tree

(unpacked once per hash, read-only) as `/`, the workspace directory at

`/src`, tmpfs at `/tmp`; Landlock limits writes to `/src` and `/tmp`; a

seccomp filter refuses `socket` for anything but `AF_UNIX`, plus

`ptrace`, `mount`, `bpf`, `perf_event_open`, `keyctl` and module calls;

`cpu-limit` and `memory-limit` apply through a cgroup v2 scope when the

runner has a delegated cgroup, and the job fails to start otherwise

rather than run unbounded. Same frames and receipts as tier 2.

**How it runs.**

- **Helper:** for each job the runner starts `flow __sandbox <spec>` (its own binary, through `/proc/self/exe`). The helper joins the job's cgroup, makes the namespaces with the job's user (uid and gid 1000) mapped to the runner's, and forks the sandbox's PID 1.

- **Root:** PID 1 binds the unpacked tree read-only as the new root, the workspace at `/src`, tmpfs at `/tmp`, `/run` and `/run/secrets`, a fresh `/proc`, and a `/dev` of `null`, `zero`, `full`, `random` and `urandom` with `/dev/shm`. It brings `lo` up for `network "loopback"` and `"egress"` (with the vsock relay of spec §4.3.1 for the latter), pivots into the root and detaches the old one, so the host's files, the spec and the scratch directory are out of reach.

- **Job:** the job gets no new privileges, then Landlock (read and execute everywhere, write beneath `/src`, `/tmp`, `/dev` and its mounts), then the seccomp filter, then `execve` with exactly the job's environment, found the way `flow-init` finds it (the two share that code). The filter also refuses io_uring (which could open sockets past it), the new mount calls, kexec, swap and reboot.

- **Unpacking:** `debugfs rdump` unpacks the image into `trees/<hash>` in the home once, without mount rights. Set-id bits and device nodes are dropped.

- **Ephemeral mounts:** a `tmpfs` mount is a tmpfs of its size. An `ephemeral://disk` mount is a scratch directory, so its size is not enforced in this tier. One that names a `cache` starts from a directory a passing job of the slot kept, and goes back into the runner's cache pool when the job passes (spec §5.4, oxygenflow#70).

- **Artifacts:** they are read from the workspace and mount directories without following links.

- **Receipts:** the environment's runtime is `syscall`. Its parts are the `flow` binary's BLAKE3, the host kernel's release, memory, CPUs and mounts.

- **Limits:** the job's cgroup gets `memory.max`, `memory.swap.max 0` and `cpu.max`. A job killed for memory is told so in its log. At the end the group is killed and removed.

**Where it is offered.** `flow tiers` prints the tiers this machine can run and why the others cannot; a runner offers only those (spec §11.1). The syscall tier needs:

- **Namespaces:** unprivileged user namespaces for the `flow` binary. Ubuntu's `kernel.apparmor_restrict_unprivileged_userns=1` can deny them to a program without an AppArmor profile.

- **Kernel features:** Landlock in the kernel's LSM list, and seccomp filters.

- **Cgroup:** a cgroup v2 group the runner may manage, with the cpu and memory controllers and no other process in it. That is a systemd service with `Delegate=yes`, or a user service under `systemd --user`. The runner moves itself into a `flow-main` leaf of that group and makes one group per job beside it. For `flow run`, use `systemd-run --user --scope -p Delegate=yes flow run`.

## 4. Receipts

Every finished job gets a receipt (spec §6.1): `job_name`,

`pipeline_hash`, `scope_tree_hash` (BLAKE3 over the sorted

`(path, content hash)` of the files `scope` matches; the whole tree when

there is no scope), `inputs` (empty in M0), `environment`, `outputs`

(empty), `exit_code`, `duration_ms`. It is encoded as canonical CBOR and

hashed as `BLAKE3("oxy1 receipt " || len || 0x00 || body)`, the `receipt`

object of the Oxygen spec (§14.6).

**Registering a runner.**

- **Token:** a hub administrator creates a one-time registration token on Admin → Runners.

- **Registering:** `flow runner register https://hub --token … [--name N] [--ca PATH] [--label L]…` sends the runner's signing identity, architecture, OS, version and labels (`POST /api/v1/flow/runners/register`). It gets back the runner's own credential and writes hub, credential, name, CA and labels to `runner.toml` (owner-only). That file stands in for a `[runner]` section `flow.toml` lacks, and `flow runner` then starts from it.

- **Key:** registration binds the key; the hub accepts receipts from that runner signed by no other.

- **Claims:** every claim carries the runner's description (`info`: architecture, OS, version, labels, capacity) for the Runners page.

- **Old configuration:** runners configured with a `hub.toml` token still work. Their key is bound on their first signed receipt.

**Signed receipts.** A runner keeps a composite key in its home

(`runner.key`, owner-only, and `runner.identity`), made on first start and

named after the runner. After each job it binds the receipt to the state of

the heads it fetched (computed by the runner, not taken from the claim) and

to the run's URL, signs that attestation, and sends the signed receipt with

`finish` as `signed_receipt` (hex). The hub checks it and keeps it in the

repository; anyone can then check it with `oxy receipt verify`. Cache reuse

(§6.2) follows.

## 5. Hub side (OxygenHub)

- Configuration: `[flow] runner_tokens = [...]` and `repos = ["owner/*"]`,

  the repositories whose pushes may start runs. No proposals trigger runs

  in M0, so code from people without write access never runs.

- On a push that moves a bookmark of an allowed repository, the hub reads

  `.flow/pipeline.kdl` from the new state, validates it with `flow-core`,

  and records a **run** (repository, bookmark, state address, pipeline

  hash, pusher) with one **job** per pipeline job. An invalid pipeline

  records a failed run with the error and its line.

- Runner API, bearer runner token:

  - `POST /api/v1/flow/claim` → the oldest queued job whose dependencies

    passed, with its run's state heads, repository clone URL, the job's

    compiled form and its sandbox, and a **job token**: read access to that

    repository only, valid until the job finishes or 2 hours pass. `204`

    when there is nothing.

  - `POST /api/v1/flow/jobs/{id}/log` appends records `{t, stream, text}`. An empty batch only asks whether the job still runs; the runner sends one every 5 s while a job is quiet, and stops the job's microVM when the hub answers `409` (the run was cancelled, oxygenflow#32). A cancelled job's end is not reported.

  - `POST /api/v1/flow/jobs/{id}/finish` with the exit code, duration and

    receipt; a job whose dependency failed becomes `skipped`.

- Each job is also a check named `flow/<job>` on the state (hub spec §13),

  so proposals, the status bar in VS Code and required checks see it.

- Pages: a **Flow** tab with runs (status, bookmark, state, pusher,

  duration); a run page with the job graph drawn as SVG (columns by

  dependency depth, status by colour and glyph), each job's log folded

  by default and the failed one open, refreshed while the run is live.

## 6. Earth

Under `/home/claude/flow`: `bin/firecracker` (release binary, pinned),

`kernel/vmlinux` (pinned), `cas/` (rootfs images), `work/` (scratch), and

`runner.toml`. `flow runner` runs as the `flow-runner` systemd service

under the `claude` user (member of `kvm`), polling the hub at

`https://earth.andymccall.xyz:7343`. The unit has `KillMode=mixed` and

`TimeoutStopSec=75min`, so a restart drains the runner instead of killing

its job (spec §11.5.1).

## 7. Not in M0

WASM tier; FastCDC-backed rootfs and `casfs` mounts (M0 copies the tree

into an image); mounts other than ephemeral scratch; artifacts as inputs of later jobs; receipt reuse; provenance

attestations; policy gates; secrets; proposal

runs; scope-based skipping; re-run and cancel from the hub.