OxygenFlow M0: first pipeline on Earth

M0 is the smallest slice that exercises the real architecture: a typed pipeline compiled to a canonical form, host-side execution in the default isolation tier, receipts computed by the host, and results reported to the hub as checks. Everything M0 leaves out is listed at the end.

1. Pipeline subset

The file is .flow/pipeline.kdl in the state being built (KDL 2.0). M0 accepts:

schema-version 1

sandbox "alpine" {
    runtime "microvm"            // "microvm" (default) | "syscall"; "wasm" is refused in M0
    rootfs "cas://blob/<64 hex>"  // BLAKE3 of an ext4 image the runner holds
    memory-limit "1GiB"          // KiB, MiB, GiB
    cpu-limit 2
    network "isolated"           // "isolated" or "loopback" in M0
    env { PATH "/opt/tool/bin:/usr/bin:/bin" }  // every job's starting environment
}

job "lint" {
    sandbox "alpine"
    scope "src/**" "README.md"   // recorded; the job runs on every push in M0
    env { LANG "C.UTF-8" }
    exec "grep" "-rn" "TODO" "src"
}

job "test" {
    sandbox "alpine"
    depends-on "lint"
    exec "sh" "-eu" "test.sh"
}

Rules, all checked by flow check and again by the hub before anything runs (spec §2.2 I2, §3.2):

• Unknown nodes, arguments or properties are errors, with the line.

• Every string stays a string; cpu-limit is an integer from 1 to 16; memory-limit parses to 64 MiB to 16 GiB.

• A job names a declared sandbox, has a non-empty exec of literal strings, and depends-on only declared jobs; the graph has no cycle.

• Names are 1 to 64 characters from [A-Za-z0-9._-].

• exec becomes argv unchanged: metacharacters are bytes (spec §10.1). A job that wants a shell names one, as sh above; nothing adds one.

• publish and a job's attest are recognised and refused with "not in M0", so a pipeline written for the full spec fails clearly.

• secret "token" { mount-path "/run/secrets/token" } on a job (up to 8) names a secret the hub holds for the repository (its own, its organisation's or the site's; oxygenhub#107) and where the job reads it: a clean path under /run/secrets/. The claim carries each as secrets and flow-init writes it there, 0400 and the job's user's, on a tmpfs of its own, before the job starts; it is never in the environment, the receipt or an artifact (spec §9). Locally, flow run --secret NAME=VALUE supplies it. vault-ref is not in M0: the hub is the vault.

• arch "riscv64" on a sandbox asks for that architecture (spec §11.3); without it the rootfs image's decides. flow rootfs import reads the image's architecture from its shell's ELF header, refuses an image for another architecture than the runner's, and records it in cas/<hash>.json; a claim reports each image's architecture (image_archs) so the hub never offers a job to a runner of the wrong one (spec §11.2).

• identity "sts.amazonaws.com" { mount-path "/run/secrets/aws-token" } on a job (up to 4) asks the hub for an OIDC token for that audience, naming the repository, bookmark, run and job (OxygenHub spec §13.5, oxygenhub#103); it comes with the claim and is mounted exactly as a secret is. A local run has no hub, so flow run says the file will be missing.

• output { artifact "oxy" { path "/src/target/release/oxy" } } declares up to 16 artifacts (spec §5.2). The path is clean, absolute, under /src, and not on a tmpfs mount, which is gone when the job ends. After a job passes, the runner reads each artifact from the job's disks with debugfs: the workspace, or the ephemeral://disk mount holding it. An artifact is at most 512 MiB. The runner uploads each one with POST /api/v1/flow/jobs/{id}/artifacts/{name} (raw bytes). The hub keeps it as a blob of the repository, and the receipt's outputs name those blobs in artifact order. A declared artifact that cannot be kept fails the job, and the job gets no receipt.

• mount is accepted for ephemeral scratch only (up to 4 per job): mount { source "ephemeral://disk"; target "/src/target"; size "8GiB" }. ephemeral://tmpfs is memory in the guest and counts against its memory-limit; ephemeral://disk is a fresh sparse ext4 disk on the runner, attached after the workspace, for build trees larger than memory. The target is a clean absolute path strictly under /src or /tmp, mounts may not nest, sizes run from 16 MiB to 64 GiB, and any other source (oxy://workspace and the rest) or setting (readonly, include) is refused as not in M0. Both are empty when the job starts and deleted when it ends, so nothing passes between jobs. Mounts are part of the job's canonical form (field 6, present only when a job has mounts, so pipelines without them keep their hashes) and of the receipt's environment.

• A sandbox's env is where each of its jobs' environments starts; a job's own env overrides it name by name. It is part of the sandbox's canonical form (field 6, present only when set). Below both is the image's own environment: flow rootfs import --env NAME=value (repeatable) writes /etc/flow/env into the image, for what a Dockerfile's ENV sets and docker export loses, such as a toolchain's PATH. It is part of the image, so of its hash. Under all three are PATH (the usual directories) and HOME=/tmp.

The pipeline compiles to a canonical CBOR job graph (sorted keys, sandboxes and jobs in name order). Its BLAKE3 is the pipeline_hash used in receipts and shown by the hub.

2. Tier 2: Firecracker (default)

Per job the runner:

1. Verifies the rootfs image: BLAKE3(file) must equal the pinned hash. Images live in $FLOW_HOME/cas/<hash>.ext4; an unknown hash fails the job ("rootfs not on this runner").
2. Materializes the state at its heads with oxy into a scratch directory (a clone at the state), removes .oxy, and builds a workspace image with mkfs.ext4 -d sized to the tree plus 512 MiB.
3. Boots Firecracker with no network device (so isolated holds by construction; loopback only brings up the guest's lo, so tests can serve and connect on 127.0.0.1 while nothing leaves the guest), vcpu_count and mem_size_mib from the sandbox, the pinned guest kernel, the rootfs as a read-only vda, the workspace as a read-write vdb, and an initramfs holding flow-init.
4. flow-init (a static binary, PID 1 in the guest) mounts /proc, /sys, /dev, the rootfs at /newroot read-only, vdb at /newroot/src, and tmpfs at /newroot/tmp and /newroot/run, then the job's scratch mounts (creating their mount points, owned by the job's user); it switches root, then runs argv (its program found in the guest's PATH by flow-init itself) with exactly the job's env (plus PATH, HOME=/tmp) in /src as uid 1000, with a job timeout. /dev gets the links devtmpfs lacks, fd, stdin, stdout and stderr into /proc/self/fd, so shell process substitution works. A job is told what it builds: FLOW_REPO, FLOW_STATE (the hub's state hash), FLOW_HEADS (full hashes joined with +), FLOW_BOOKMARK, FLOW_RUN (the run's number in its repository) and FLOW_JOB; flow run sets FLOW_JOB and FLOW_RUN=local. After them come the variables the hub holds for the repository (site, organisation, repository; the claim's env; flow run --var NAME=VALUE locally), never secrets. A variable the pipeline sets keeps its value. They are part of the job's environment, so of its receipt.
5. flow-init frames the child's output on the serial console as \x1eO <line> (stdout) and \x1eE <line> (stderr). After a passing job it prunes each build cache disk (mounted strictatime) of the files the job did not use when it is over the runner's cache_prune_percent of its size, and reports each as \x1eC <json> (spec §5.4, oxygenflow#72); then \x1eX <code>, and it powers off. The kernel runs quiet so the host sees only frames; anything else is kept as console records.
6. The host reads frames, streams them as log records, and kills the VM at the timeout (the guest cannot extend it). Scratch files are deleted afterwards. Guest memory is released with the process.

The guest kernel, flow-init and Firecracker itself are pinned by BLAKE3 in the runner's configuration; their hashes and the rootfs hash make the receipt's environment together with argv and env.

3. Tier 1: syscall sandbox

runtime "syscall": a child in new user, mount, PID, network, IPC and UTS namespaces (no interfaces but loopback), the rootfs image's tree (unpacked once per hash, read-only) as /, the workspace directory at /src, tmpfs at /tmp; Landlock limits writes to /src and /tmp; a seccomp filter refuses socket for anything but AF_UNIX (and, with network "loopback", AF_INET and AF_INET6, which reach only the job's own lo: oxygenflow#73), plus ptrace, mount, bpf, perf_event_open, keyctl and module calls; cpu-limit and memory-limit apply through a cgroup v2 scope when the runner has a delegated cgroup, and the job fails to start otherwise rather than run unbounded. Same frames and receipts as tier 2.

How it runs.

• Helper: for each job the runner starts flow __sandbox <spec> (its own binary, through /proc/self/exe). The helper joins the job's cgroup, makes the namespaces with the job's user (uid and gid 1000) mapped to the runner's, and forks the sandbox's PID 1.
• Root: PID 1 binds the unpacked tree read-only as the new root, the workspace at /src, tmpfs at /tmp, /run and /run/secrets, a fresh /proc, and a /dev of null, zero, full, random and urandom with /dev/shm. It brings lo up for network "loopback" and "egress" (with the vsock relay of spec §4.3.1 for the latter), pivots into the root and detaches the old one, so the host's files, the spec and the scratch directory are out of reach.
• Job: the job gets no new privileges, then Landlock (read and execute everywhere, write beneath /src, /tmp, /dev and its mounts), then the seccomp filter, then execve with exactly the job's environment, found the way flow-init finds it (the two share that code). The filter also refuses io_uring (which could open sockets past it), the new mount calls, kexec, swap and reboot.
• Unpacking: debugfs rdump unpacks the image into trees/<hash> in the home once, without mount rights. Set-id bits and device nodes are dropped.
• Ephemeral mounts: a tmpfs mount is a tmpfs of its size. An ephemeral://disk mount is a scratch directory, so its size is not enforced in this tier. One that names a cache starts from a directory a passing job of the slot kept, and goes back into the runner's cache pool when the job passes (spec §5.4, oxygenflow#70), pruned first by the runner of the files the job did not use when over the cap (oxygenflow#72).
• Artifacts: they are read from the workspace and mount directories without following links.
• Receipts: the environment's runtime is syscall. Its parts are the flow binary's BLAKE3, the host kernel's release, memory, CPUs and mounts.
• Limits: the job's cgroup gets memory.max, memory.swap.max 0 and cpu.max. A job killed for memory is told so in its log. At the end the group is killed and removed.

Where it is offered. flow tiers prints the tiers this machine can run and why the others cannot; a runner offers only those (spec §11.1). The syscall tier needs:

• Namespaces: unprivileged user namespaces for the flow binary. Ubuntu's kernel.apparmor_restrict_unprivileged_userns=1 can deny them to a program without an AppArmor profile.
• Kernel features: Landlock in the kernel's LSM list, and seccomp filters.
• Cgroup: a cgroup v2 group the runner may manage, with the cpu and memory controllers and no other process in it. That is a systemd service with Delegate=yes, or a user service under systemd --user. The runner moves itself into a flow-main leaf of that group and makes one group per job beside it. For flow run, use systemd-run --user --scope -p Delegate=yes flow run.

4. Receipts

Every finished job gets a receipt (spec §6.1): job_name, pipeline_hash, scope_tree_hash (BLAKE3 over the sorted (path, content hash) of the files scope matches; the whole tree when there is no scope), inputs (empty in M0), environment, outputs (empty), exit_code, duration_ms. It is encoded as canonical CBOR and hashed as BLAKE3("oxy1 receipt " || len || 0x00 || body), the receipt object of the Oxygen spec (§14.6).

Registering a runner.

• Token: a hub administrator creates a one-time registration token on Admin → Runners.
• Registering: flow runner register https://hub --token … [--name N] [--ca PATH] [--label L]… sends the runner's signing identity, architecture, OS, version and labels (POST /api/v1/flow/runners/register). It gets back the runner's own credential and writes hub, credential, name, CA and labels to runner.toml (owner-only). That file stands in for a [runner] section flow.toml lacks, and flow runner then starts from it.
• Key: registration binds the key; the hub accepts receipts from that runner signed by no other.
• Claims: every claim carries the runner's description (info: architecture, OS, version, labels, capacity) for the Runners page.
• Old configuration: runners configured with a hub.toml token still work. Their key is bound on their first signed receipt.

Signed receipts. A runner keeps a composite key in its home (runner.key, owner-only, and runner.identity), made on first start and named after the runner. After each job it binds the receipt to the state of the heads it fetched (computed by the runner, not taken from the claim) and to the run's URL, signs that attestation, and sends the signed receipt with finish as signed_receipt (hex). The hub checks it and keeps it in the repository; anyone can then check it with oxy receipt verify. Cache reuse (§6.2) follows.

5. Hub side (OxygenHub)

• Configuration: [flow] runner_tokens = [...] and repos = ["owner/*"], the repositories whose pushes may start runs. No proposals trigger runs in M0, so code from people without write access never runs.
• On a push that moves a bookmark of an allowed repository, the hub reads .flow/pipeline.kdl from the new state, validates it with flow-core, and records a run (repository, bookmark, state address, pipeline hash, pusher) with one job per pipeline job. An invalid pipeline records a failed run with the error and its line.
• Runner API, bearer runner token:
  • POST /api/v1/flow/claim → the oldest queued job whose dependencies passed, with its run's state heads, repository clone URL, the job's compiled form and its sandbox, and a job token: read access to that repository only, valid until the job finishes or 2 hours pass. 204 when there is nothing.
  • POST /api/v1/flow/jobs/{id}/log appends records {t, stream, text}. An empty batch only asks whether the job still runs; the runner sends one every 5 s while a job is quiet, and stops the job's microVM when the hub answers 409 (the run was cancelled, oxygenflow#32). A cancelled job's end is not reported.
  • POST /api/v1/flow/jobs/{id}/finish with the exit code, duration and receipt; a job whose dependency failed becomes skipped.
• Each job is also a check named flow/<job> on the state (hub spec §13), so proposals, the status bar in VS Code and required checks see it.
• Pages: a Flow tab with runs (status, bookmark, state, pusher, duration); a run page with the job graph drawn as SVG (columns by dependency depth, status by colour and glyph), each job's log folded by default and the failed one open, refreshed while the run is live.

6. Earth

Under /home/claude/flow: bin/firecracker (release binary, pinned), kernel/vmlinux (pinned), cas/ (rootfs images), work/ (scratch), and runner.toml. flow runner runs as the flow-runner systemd service under the claude user (member of kvm), polling the hub at https://earth.andymccall.xyz:7343. The unit has KillMode=mixed and TimeoutStopSec=75min, so a restart drains the runner instead of killing its job (spec §11.5.1).

7. Not in M0

WASM tier; FastCDC-backed rootfs and casfs mounts (M0 copies the tree into an image); mounts other than ephemeral scratch; artifacts as inputs of later jobs; receipt reuse; provenance attestations; policy gates; secrets; proposal runs; scope-based skipping; re-run and cancel from the hub.