Handover: jobs that outlive their runner

Proposed 2026-10-25, from a design conversation with Andy. Issues: oxygenflow#75 (WASM tier), #76 (leases and drain), #77 (checkpoints), #78 (hedging), #79 (trust classes), #80 (browser runners); the phone runner is oxygenhub-android#8. Builds on Jetstream (docs/jetstream.md), whose pools (hosts, Kubernetes, cloud) and requeue of a lost runner's job this extends.

1. Why

Three situations, one problem: a job is tied to the runner that started it.

• The runner goes away. A phone leaves the office Wi-Fi or comes off charge; a laptop sleeps; a spot VM is reclaimed with 30 seconds' notice; a Kubernetes node is drained. Today the job starts again from nothing elsewhere, or fails.
• A better runner frees up. A job went to the small runner B because the big runner A was busy. A finishes its own job when B is 10% through, then sits idle while B grinds through the other 90%.
• The same work runs twice. A requeued job redoes everything; two pipelines that build the same thing both build it.

Solving the first makes unreliable runners usable (phones, browsers, spare machines, spot instances). Solving the second keeps a mixed fleet busy. Together they are the case a company makes for running CI on the hardware it already has: more use from it, and less waiting for developers and agents.

2. What Flow already has

Most of the parts exist:

• Jobs are the packets. A job runs one exec; jobs form a graph and pass artifacts (§5 of the spec, oxygenflow#58). A run is already spread over runners: oxygenhub's run #439 built on earth-1, ran clippy on plex-1, dist on plex-1 and riscv64 on narvi-1.
• Receipts already chain. Each job's receipt (spec §6.1) is signed by its runner and names its inputs, which are other jobs' outputs, whose receipts name their runners. A run's provenance is already a graph of signed receipts from several runners.
• A lost runner costs only its job. Jetstream requeues it after a grace, and a spot reclaim notice requeues it at once (jetstream.md §11).
• Placement can wait for a better runner ("held for a better runner"), and the hub learns each job's duration per speed class (spec §11).

What is missing: a job that is partly done is lost when it moves; there is no graceful way to leave; a better runner cannot take over; nothing says how far to trust a runner; and phones and browsers cannot run jobs at all (no WASM tier).

3. Decision: move progress, not processes

No live migration of running VMs (oxygenflow#33 suggested Firecracker snapshots). A snapshot fixes the VM's shape: a job started with 2 vCPUs on a small runner is restored with 2 vCPUs on the big one, since Firecracker cannot add vCPUs to a running VM, so it gets no faster. It moves gigabytes of memory, works only between hosts of the same architecture and CPU features, and cannot help when the runner simply disappears, because the snapshot is on the runner that is gone.

Instead a job's progress moves: the outputs of finished jobs (as today) and, within a job, the directories it declares as checkpoints (§7). Everything is content-addressed, so the new runner fetches only what it lacks, and every piece comes with a receipt saying who made it.

4. Runners

Cloud and Kubernetes are where handover pays most: spot capacity costs 60 to 90% less, and handover is what makes it safe to run CI on it. Phones and browsers are where it shows best in a demonstration.

Architecture still matters. Native outputs and checkpoints only move between runners of the same image and target (x86_64 to x86_64, arm64 to arm64). WASM work moves anywhere.

5. The WASM tier (oxygenflow#75)

Spec §4.1 names it; built 2026-10-25 (wasmtime 49, WASI preview 1).

• wasmtime, WASI preview 1 first (preview 2 components later).
• A virtual file system holding the job's scope and inputs; outputs collected from it as for the other tiers.
• No network. Memory from the sandbox's limit; CPU time bounded by fuel and the job's time limit.
• The receipt's environment names the runtime, its version and the module's hash.
• Every Linux runner offers it (tiers gains wasm); phones and browsers offer only it.

What runs there is what compiles to WASM: linters, formatters, policy checks, generated docs, and test suites built for wasm32-wasip1. That last one makes the first demonstration (§11): a native job builds the test modules, a phone runs them.

6. Leases and drain (oxygenflow#76)

Lease. A runner holding a job holds a lease, renewed by its logs and a heartbeat. When the lease lapses the job is requeued, as today.

Drain is one message in the runner contract (jetstream.md §4) for every planned departure:

On drain the runner decides from the job's learned duration (spec §11):

1. the rest is shorter than the notice: finish the job, then leave;
2. otherwise: save its checkpoints (§7), upload them, release the lease with drained, and stop the job.

A drained job is requeued at once, ahead of new work, and resumes from its checkpoints. A runner that drains a job is not penalised; one that drops leases without draining loses placement priority.

7. Checkpoints (oxygenflow#77)

A job may declare directories that hold its progress:

job "build" {
    sandbox "rust"
    exec "cargo" "build" "--release"
    checkpoint "/src/target"
}

• Saved by content (FastCDC chunks, deduplicated, kept for the run's life) on drain and every few minutes while it runs (a sandbox setting, default 5 minutes, only while the job has run at least that long).
• Restored before a resumed job runs the same exec again. Tools that skip finished work (cargo's fingerprints, gradle, make, ninja) then carry on where they stopped; a tool that does not just starts again, so it is never wrong, only slower.
• Receipts. The partial run ends in a receipt with exit "drained", naming the checkpoint's hash as an output and signed by its runner. The resumed job's receipt names that checkpoint among its inputs. So provenance says that plex built the first part and mars finished it, and both are signed.
• Cheap to move inside a cloud: checkpoints live in the run's store (a bucket under Strata for cloud pools), so a new pod in the same region restores them quickly. From a phone, the policy (charger, office network) keeps uploads cheap.
• Checkpoints are code. A tampered .rlib in target/ ends up in the binary. So a job accepts a checkpoint only from a runner of at least its trust class (§9), and its receipt says that it used one.

8. Hedging (oxygenflow#78)

When a markedly faster runner frees up while a job runs on a slow one, and the learned durations say it would finish sooner (counting the time to restore the latest checkpoint), start the job there too. The first to finish wins; the other is cancelled.

• Bounded: one hedge per job, and only where a policy allows it (the critical path of a run, or always on owned runners). Never for jobs with side effects (deployments, publishing).
• No tricky handover: the slow runner is never interrupted. If the fast one fails, nothing is lost.
• A free determinism check: when both finish (a race), their outputs are compared; a difference is reported as a non-hermetic job.
• Cost-aware in clouds: starting a bigger instance costs money, so a pool may forbid hedging, or allow it only for jobs a person is waiting on.

Hedging plus holding briefly for a better runner (which exists) gets most of the benefit of a full handover, for a fraction of the work. Build it before anything finer (§12).

9. Trust classes (oxygenflow#79)

How far to believe a receipt depends on who signed it:

• A sandbox or job may require a minimum (trust "owned"). Release and publish jobs default to owned.
• A job resumes only from checkpoints made at its class or above.
• Receipts and provenance record the class of every runner that contributed. The trust page shows a run's weakest link.
• Spot checks: a sample of jobs run by device and anonymous runners is run again elsewhere and the outputs compared (with oxygenhub#298's independent rebuilds). A runner whose results disagree is held for its owner to look at.

10. Phones and browsers

Phones (oxygenhub-android#8): one tap registers the phone as a WASM-tier runner, as a foreground service, scoped like any runner. The owner's policy says when it may run: on charge, on these networks (the office Wi-Fi), under a temperature, within a storage budget. Leaving those conditions drains it (§6). The runner itself is Flow's runner code built for Android through the app's Rust core (rust-core, uniffi), with wasmtime for the tier. The full Linux runner in Android's Linux terminal (#9) comes later.

Browsers (#80): a page on the hub turns a tab into a WASM-tier runner in a Web Worker: WASI through a shim, OPFS (the browser's private file storage) for files, uploads as for any runner. Limits: about 4 GB of memory, threads only on a cross-origin-isolated page, throttled or stopped in a background tab (a drain). This is mostly a demonstration of handover and spot checks on the least reliable runner there is, not capacity.

11. The first demonstration

A pipeline over oxy-core:

1. build-wasm-tests (microvm, any x86_64 host): cargo test --target wasm32-wasip1 --no-run, outputs the test modules.
2. test-wasm (wasm, runs-on "android", depends on 1): runs each module under the WASM tier, on the phone.

The script: start the run; the phone takes test-wasm; switch the phone's Wi-Fi off halfway (a drain under its policy); the job goes back to the queue and plex-1 takes it in its WASM tier; the run page shows which runner did what, with the receipts and their trust classes. Then the same with a pipeline that has a checkpoint, to show a job resuming rather than restarting.

12. Order of work

1. WASM tier (#75), in flow on Linux runners.
2. Phone runner (oxygenhub-android#8) on that tier, with its policy.
3. Leases and drain (#76), with requeue ahead of new work.
4. The demonstration (§11), job-level: drain and requeue.
5. Checkpoints (#77): resume within a job; receipts that say so.
6. Trust classes (#79), before anything other than our own runners joins.
7. Hedging (#78), then measure on our own CI: queue and run times with and without it.
8. Browser runners (#80).
9. Finer pieces, only if the measurements call for them: test sharding and per-crate builds (#29), so that a long job is many short ones a better runner can take over between.

Cloud and Kubernetes pools come through Jetstream (jetstream.md §9, oxygenflow#33), and get drain and checkpoints as they arrive.

13. Open questions

• Checkpoint retention: the run's life, or kept as a cache for the next run of the same job (a remote cache, which would then need the same trust rules)?
• How long may a drained job wait for a runner of its trust class before it falls back to restarting from nothing on any runner?
• Hedging costs compute: a per-pool budget, or a per-organisation one?
• A phone's battery and heat: are the owner's policy limits enough, or should the runner also cap its own CPU use?