Design: runners update themselves (refs #84) c90dd8d5fc36

### 8.2 Runners update themselves

Proposed 2026-10-25 from Andy's experience: keeping a large fleet's

runners up to date by hand is one of the biggest costs of running CI.

Runners fetch new runner builds the way they fetch images, from each other

through the swarm, check them as strictly as a release, and swap

themselves in without losing work, under a policy the organisation sets,

testing on a few runners before the rest.

**A runner build** is a `flow` binary from an OxygenFlow release, one per

target (linux-x86_64, linux-arm64, linux-riscv64; Windows and macOS when

those runners exist), built by Flow from the tagged state with a signed

receipt, as oxy's and OxygenHub's releases are. The hub serves it as a

blob with a chunk manifest, so the swarm (§8.1) carries it unchanged: one

runner of a site fetches each chunk from the hub and the rest from it.

**Off unless chosen, nearest wins.** The policy is set for the site, an

organisation, a test group and a runner; the nearest setting wins:

- `updates`: `off` (the default) or `on`;

- `channel`: `stable`, `canary`, or a pinned version (`0.3.4`);

- `window`: hours in which a runner may update (outside them it waits);

- `dry_run`: fetch and check, say what would happen, do not swap.

A runner's own `flow.toml` has the last word: `[runner.update] allow =

false` refuses every update whatever the hub says, so a site can hold a

machine back while it tests.

**Test groups and waves.** A test group is a named set of runners: chosen

by name, by label or by pool. A rollout is a release and its waves, in

order; each wave is a group and its share of it, **a percentage or chosen

runners**. A percentage is taken by a stable hash of each runner's id, so

"10% of build-farm" is the same machines every rollout, the ones the

organisation watches. A wave starts when the one before has been healthy

for its soak time (say 24 hours) or when an admin promotes it. For

example: canary (two chosen runners), then build-farm at 10%, then

build-farm at 100%, then everyone.

**Healthy** means the updated runners came back on the new version within

ten minutes, claim jobs, and their jobs' infrastructure errors (`lost`,

`error`, declines) over the soak are no worse than the same runners' on

the version before, over enough jobs to tell. A wave that is not healthy

halts the rollout, and its runners roll back. Job failures from the

pipelines themselves do not count.

**On the runner,** in order:

1. The hub's answer to a claim or heartbeat names an update when the

   policy says so: the version, its blob and manifest, its receipt.

2. Fetch the blob through the swarm (hub as seed); every chunk checked

   against the manifest from the hub.

3. Check it as a release: its BLAKE3 matches the release asset the hub

   attests; the receipt is signed by an `owned` build runner (trust

   classes, #79) and names the tagged state; the target is this machine's.

   A peer can only serve bytes; it cannot change what is installed.

4. Stage it (`bin/flow.new`) and run its self-check (`flow.new tiers`,

   `--version`).

5. Drain with notice (`leave_within`, §11.5.2 of the spec): finish what

   fits, hand back the rest with its checkpoints.

6. Swap: `flow` → `flow.prev`, `flow.new` → `flow`, and restart.

7. Prove it: the first claim reports the new version. A runner that will

   not start, or that the hub does not hear from within ten minutes, goes

   back to `flow.prev` (a start counter in the runner's home: three

   failed starts after an update restore the previous build).

8. The runner's page shows its version, channel, group, last update and

   why; the audit log records every update and rollback.

**Phones** take runner updates with the app's releases (Android updates an

app only through its store or an install), so they follow the app.

**Order of work:** (1) the runner side, with a local source (`flow runner

update --from FILE`): check, stage, drain, swap, prove, roll back; (2) the

policy on the hub (opt in and out, channels, windows, dry run) and the

update in the claim answer; (3) test groups, waves and health; (4) the

swarm as transport; (5) Admin › Runners › Updates and `oh runner update`.