# OxygenFlow M0: first pipeline on Earth

| | |
|---|---|
| **Status** | Plan, 2026-09-28 |
| **Implements** | a subset of `docs/spec.md` (Draft 0.1) |
| **Goal** | A push to a repository on OxygenHub (Earth) runs `.flow/pipeline.kdl` in Firecracker microVMs, and the hub shows the run as a graph of jobs with live status and logs. |

M0 is the smallest slice that exercises the real architecture: a typed
pipeline compiled to a canonical form, host-side execution in the default
isolation tier, receipts computed by the host, and results reported to the
hub as checks. Everything M0 leaves out is listed at the end.

## 1. Pipeline subset

The file is `.flow/pipeline.kdl` in the state being built (KDL 2.0). M0
accepts:

```kdl
schema-version 1

sandbox "alpine" {
    runtime "microvm"            // "microvm" (default) | "syscall"; "wasm" is refused in M0
    rootfs "cas://blob/<64 hex>"  // BLAKE3 of an ext4 image the runner holds
    memory-limit "1GiB"          // KiB, MiB, GiB
    cpu-limit 2
    network "isolated"           // "isolated" or "loopback" in M0
    env { PATH "/opt/tool/bin:/usr/bin:/bin" }  // every job's starting environment
}

job "lint" {
    sandbox "alpine"
    scope "src/**" "README.md"   // recorded; the job runs on every push in M0
    env { LANG "C.UTF-8" }
    exec "grep" "-rn" "TODO" "src"
}

job "test" {
    sandbox "alpine"
    depends-on "lint"
    exec "sh" "-eu" "test.sh"
}
```

Rules, all checked by `flow check` and again by the hub before anything
runs (spec §2.2 I2, §3.2):

- Unknown nodes, arguments or properties are errors, with the line.
- Every string stays a string; `cpu-limit` is an integer from 1 to 16;
  `memory-limit` parses to 64 MiB to 16 GiB.
- A job names a declared sandbox, has a non-empty `exec` of literal
  strings, and `depends-on` only declared jobs; the graph has no cycle.
- Names are 1 to 64 characters from `[A-Za-z0-9._-]`.
- `exec` becomes `argv` unchanged: metacharacters are bytes (spec §10.1).
  A job that wants a shell names one, as `sh` above; nothing adds one.
- `publish` and `secret` are recognised and refused with "not in M0", so
  a pipeline written for the full spec fails clearly.
- `output { artifact "oxy" { path "/src/target/release/oxy" } }` declares
  up to 16 **artifacts** (spec §5.2). The path is clean, absolute, under
  `/src`, and not on a tmpfs mount, which is gone when the job ends. After
  a job passes, the runner reads each artifact from the job's disks with
  `debugfs`: the workspace, or the `ephemeral://disk` mount holding it. An
  artifact is at most 512 MiB. The runner uploads each one with
  `POST /api/v1/flow/jobs/{id}/artifacts/{name}` (raw bytes). The hub keeps
  it as a blob of the repository, and the receipt's `outputs` name those
  blobs in artifact order. A declared artifact that cannot be kept fails
  the job, and the job gets no receipt.
- `mount` is accepted for **ephemeral scratch** only (up to 4 per job):
  `mount { source "ephemeral://disk"; target "/src/target"; size "8GiB" }`.
  `ephemeral://tmpfs` is memory in the guest and counts against its
  memory-limit; `ephemeral://disk` is a fresh sparse ext4 disk on the
  runner, attached after the workspace, for build trees larger than
  memory. The target is a clean absolute path strictly under `/src` or
  `/tmp`, mounts may not nest, sizes run from 16 MiB to 64 GiB, and any
  other source (`oxy://workspace` and the rest) or setting (`readonly`,
  `include`) is refused as not in M0. Both are empty when the job starts
  and deleted when it ends, so nothing passes between jobs. Mounts are
  part of the job's canonical form (field 6, present only when a job has
  mounts, so pipelines without them keep their hashes) and of the
  receipt's environment.

- A sandbox's `env` is where each of its jobs' environments starts; a
  job's own `env` overrides it name by name. It is part of the sandbox's
  canonical form (field 6, present only when set). Below both is the
  image's own environment: `flow rootfs import --env NAME=value`
  (repeatable) writes `/etc/flow/env` into the image, for what a
  Dockerfile's `ENV` sets and `docker export` loses, such as a toolchain's
  `PATH`. It is part of the image, so of its hash. Under all three are
  `PATH` (the usual directories) and `HOME=/tmp`.

The pipeline compiles to a canonical CBOR job graph (sorted keys, sandboxes
and jobs in name order). Its BLAKE3 is the `pipeline_hash` used in receipts
and shown by the hub.

## 2. Tier 2: Firecracker (default)

Per job the runner:

1. Verifies the rootfs image: `BLAKE3(file)` must equal the pinned hash.
   Images live in `$FLOW_HOME/cas/<hash>.ext4`; an unknown hash fails the
   job ("rootfs not on this runner").
2. Materializes the state at its heads with `oxy` into a scratch
   directory (a clone at the state), removes `.oxy`, and builds a
   workspace image with `mkfs.ext4 -d` sized to the tree plus 512 MiB.
3. Boots Firecracker with no network device (so `isolated` holds by
   construction; `loopback` only brings up the guest's `lo`, so tests can
   serve and connect on 127.0.0.1 while nothing leaves the guest), `vcpu_count` and `mem_size_mib` from the sandbox, the
   pinned guest kernel, the rootfs as a **read-only** `vda`, the
   workspace as a read-write `vdb`, and an initramfs holding `flow-init`.
4. `flow-init` (a static binary, PID 1 in the guest) mounts `/proc`,
   `/sys`, `/dev`, the rootfs at `/newroot` read-only, `vdb` at
   `/newroot/src`, and tmpfs at `/newroot/tmp` and `/newroot/run`, then the
   job's scratch mounts (creating their mount points, owned by the job's
   user); it
   switches root, then runs `argv` (its program found in the guest's `PATH` by
   `flow-init` itself) with exactly the job's `env` (plus
   `PATH`, `HOME=/tmp`) in `/src` as uid 1000, with a job timeout.
   `/dev` gets the links devtmpfs lacks, `fd`, `stdin`, `stdout` and
   `stderr` into `/proc/self/fd`, so shell process substitution works. A job
   is told what it builds: `FLOW_REPO`, `FLOW_STATE` (the hub's state hash),
   `FLOW_HEADS` (full hashes joined with `+`), `FLOW_BOOKMARK`, `FLOW_RUN`
   (the run's number in its repository) and `FLOW_JOB`; `flow run` sets
   `FLOW_JOB` and `FLOW_RUN=local`. A variable the pipeline sets keeps its
   value. They are part of the job's environment, so of its receipt.
5. `flow-init` frames the child's output on the serial console as
   `\x1eO <line>` (stdout) and `\x1eE <line>` (stderr), then
   `\x1eX <code>`, and powers off. The kernel runs `quiet` so the host sees
   only frames; anything else is kept as `console` records.
6. The host reads frames, streams them as log records, and kills the VM
   at the timeout (the guest cannot extend it). Scratch files are deleted
   afterwards. Guest memory is released with the process.

The guest kernel, `flow-init` and Firecracker itself are pinned by BLAKE3
in the runner's configuration; their hashes and the rootfs hash make the
receipt's `environment` together with `argv` and `env`.

## 3. Tier 1: syscall sandbox

`runtime "syscall"`: a child in new user, mount, PID, network, IPC and
UTS namespaces (no interfaces but loopback), the rootfs image's tree
(unpacked once per hash, read-only) as `/`, the workspace directory at
`/src`, tmpfs at `/tmp`; Landlock limits writes to `/src` and `/tmp`; a
seccomp filter refuses `socket` for anything but `AF_UNIX`, plus
`ptrace`, `mount`, `bpf`, `perf_event_open`, `keyctl` and module calls;
`cpu-limit` and `memory-limit` apply through a cgroup v2 scope when the
runner has a delegated cgroup, and the job fails to start otherwise
rather than run unbounded. Same frames and receipts as tier 2. Done after
tier 2 works on Earth.

## 4. Receipts

Every finished job gets a receipt (spec §6.1): `job_name`,
`pipeline_hash`, `scope_tree_hash` (BLAKE3 over the sorted
`(path, content hash)` of the files `scope` matches; the whole tree when
there is no scope), `inputs` (empty in M0), `environment`, `outputs`
(empty), `exit_code`, `duration_ms`. It is encoded as canonical CBOR and
hashed as `BLAKE3("oxy1 receipt " || len || 0x00 || body)`, the `receipt`
object of the Oxygen spec (§14.6).

**Registering a runner.**
- **Token:** a hub administrator creates a one-time registration token on Admin → Runners.
- **Registering:** `flow runner register https://hub --token … [--name N] [--ca PATH] [--label L]…` sends the runner's signing identity, architecture, OS, version and labels (`POST /api/v1/flow/runners/register`). It gets back the runner's own credential and writes hub, credential, name, CA and labels to `runner.toml` (owner-only). That file stands in for a `[runner]` section `flow.toml` lacks, and `flow runner` then starts from it.
- **Key:** registration binds the key; the hub accepts receipts from that runner signed by no other.
- **Claims:** every claim carries the runner's description (`info`: architecture, OS, version, labels, capacity) for the Runners page.
- **Old configuration:** runners configured with a `hub.toml` token still work. Their key is bound on their first signed receipt.

**Signed receipts.** A runner keeps a composite key in its home
(`runner.key`, owner-only, and `runner.identity`), made on first start and
named after the runner. After each job it binds the receipt to the state of
the heads it fetched (computed by the runner, not taken from the claim) and
to the run's URL, signs that attestation, and sends the signed receipt with
`finish` as `signed_receipt` (hex). The hub checks it and keeps it in the
repository; anyone can then check it with `oxy receipt verify`. Cache reuse
(§6.2) follows.

## 5. Hub side (OxygenHub)

- Configuration: `[flow] runner_tokens = [...]` and `repos = ["owner/*"]`,
  the repositories whose pushes may start runs. No proposals trigger runs
  in M0, so code from people without write access never runs.
- On a push that moves a bookmark of an allowed repository, the hub reads
  `.flow/pipeline.kdl` from the new state, validates it with `flow-core`,
  and records a **run** (repository, bookmark, state address, pipeline
  hash, pusher) with one **job** per pipeline job. An invalid pipeline
  records a failed run with the error and its line.
- Runner API, bearer runner token:
  - `POST /api/v1/flow/claim` → the oldest queued job whose dependencies
    passed, with its run's state heads, repository clone URL, the job's
    compiled form and its sandbox, and a **job token**: read access to that
    repository only, valid until the job finishes or 2 hours pass. `204`
    when there is nothing.
  - `POST /api/v1/flow/jobs/{id}/log` appends records `{t, stream, text}`. An empty batch only asks whether the job still runs; the runner sends one every 5 s while a job is quiet, and stops the job's microVM when the hub answers `409` (the run was cancelled, oxygenflow#32). A cancelled job's end is not reported.
  - `POST /api/v1/flow/jobs/{id}/finish` with the exit code, duration and
    receipt; a job whose dependency failed becomes `skipped`.
- Each job is also a check named `flow/<job>` on the state (hub spec §13),
  so proposals, the status bar in VS Code and required checks see it.
- Pages: a **Flow** tab with runs (status, bookmark, state, pusher,
  duration); a run page with the job graph drawn as SVG (columns by
  dependency depth, status by colour and glyph), each job's log folded
  by default and the failed one open, refreshed while the run is live.

## 6. Earth

Under `/home/claude/flow`: `bin/firecracker` (release binary, pinned),
`kernel/vmlinux` (pinned), `cas/` (rootfs images), `work/` (scratch), and
`runner.toml`. `flow runner` runs as the `flow-runner` systemd service
under the `claude` user (member of `kvm`), polling the hub at
`https://earth.andymccall.xyz:7343`.

## 7. Not in M0

WASM tier; FastCDC-backed rootfs and `casfs` mounts (M0 copies the tree
into an image); mounts other than ephemeral scratch; artifacts as inputs of later jobs; receipt reuse; provenance
attestations; policy gates; secrets; proposal
runs; scope-based skipping; re-run and cancel from the hub.