Handover: jobs that outlive their runner (design; refs #75 #76 #77 #78 #79 #80) 39e737fbc7a9

# Handover: jobs that outlive their runner

Proposed 2026-10-25, from a design conversation with Andy. Issues:

oxygenflow#75 (WASM tier), #76 (leases and drain), #77 (checkpoints),

#78 (hedging), #79 (trust classes), #80 (browser runners); the phone

runner is oxygenhub-android#8. Builds on Jetstream (docs/jetstream.md),

whose pools (hosts, Kubernetes, cloud) and requeue of a lost runner's job

this extends.

## 1. Why

Three situations, one problem: a job is tied to the runner that started

it.

- **The runner goes away.** A phone leaves the office Wi-Fi or comes off

  charge; a laptop sleeps; a spot VM is reclaimed with 30 seconds' notice;

  a Kubernetes node is drained. Today the job starts again from nothing

  elsewhere, or fails.

- **A better runner frees up.** A job went to the small runner B because

  the big runner A was busy. A finishes its own job when B is 10% through,

  then sits idle while B grinds through the other 90%.

- **The same work runs twice.** A requeued job redoes everything; two

  pipelines that build the same thing both build it.

Solving the first makes unreliable runners usable (phones, browsers,

spare machines, spot instances). Solving the second keeps a mixed fleet

busy. Together they are the case a company makes for running CI on the

hardware it already has: more use from it, and less waiting for

developers and agents.

## 2. What Flow already has

Most of the parts exist:

- **Jobs are the packets.** A job runs one `exec`; jobs form a graph and

  pass artifacts (§5 of the spec, oxygenflow#58). A run is already spread

  over runners: oxygenhub's run #439 built on earth-1, ran clippy on

  plex-1, dist on plex-1 and riscv64 on narvi-1.

- **Receipts already chain.** Each job's receipt (spec §6.1) is signed by

  its runner and names its inputs, which are other jobs' outputs, whose

  receipts name *their* runners. A run's provenance is already a graph of

  signed receipts from several runners.

- **A lost runner costs only its job.** Jetstream requeues it after a

  grace, and a spot reclaim notice requeues it at once (jetstream.md §11).

- **Placement can wait for a better runner** ("held for a better

  runner"), and the hub learns each job's duration per speed class

  (spec §11).

What is missing: a job that is *partly done* is lost when it moves; there

is no graceful way to leave; a better runner cannot take over; nothing

says how far to trust a runner; and phones and browsers cannot run jobs

at all (no WASM tier).

## 3. Decision: move progress, not processes

**No live migration of running VMs** (oxygenflow#33 suggested Firecracker

snapshots). A snapshot fixes the VM's shape: a job started with 2 vCPUs

on a small runner is restored with 2 vCPUs on the big one, since

Firecracker cannot add vCPUs to a running VM, so it gets no faster. It

moves gigabytes of memory, works only between hosts of the same

architecture and CPU features, and cannot help when the runner simply

disappears, because the snapshot is on the runner that is gone.

Instead a job's **progress** moves: the outputs of finished jobs (as

today) and, within a job, the directories it declares as checkpoints

(§7). Everything is content-addressed, so the new runner fetches only what

it lacks, and every piece comes with a receipt saying who made it.

## 4. Runners

| Kind | Tiers | Typical trust (§9) | Leaves by |

|---|---|---|---|

| Hosts (earth-1, plex-1, mars-1, narvi-1) | microvm, syscall, wasm | owned | restart or update (drain) |

| Kubernetes pods | microvm (KVM nodes), syscall, wasm | owned or attested | node drain, eviction, scale-down (SIGTERM) |

| Cloud VMs, spot or on demand | microvm, syscall, wasm | attested | reclaim notice, scale-down |

| Phone, the easy way (oxygenhub-android#8) | wasm | device | off charge, off the allowed network, too hot, owner stops it |

| Phone, full Linux (oxygenhub-android#9, later) | syscall in Android's Linux VM | device | as above |

| Browser tab (#80) | wasm | anonymous, or device when a member signs in | tab closed or hidden |

Cloud and Kubernetes are where handover pays most: spot capacity costs

60 to 90% less, and handover is what makes it safe to run CI on it.

Phones and browsers are where it shows best in a demonstration.

**Architecture still matters.** Native outputs and checkpoints only move

between runners of the same image and target (x86_64 to x86_64, arm64 to

arm64). WASM work moves anywhere.

## 5. The WASM tier (oxygenflow#75)

Spec §4.1 names it; it is not built (`runtime "wasm"` is refused today).

- wasmtime, WASI preview 1 first (preview 2 components later).

- A virtual file system holding the job's scope and inputs; outputs

  collected from it as for the other tiers.

- No network. Memory from the sandbox's limit; CPU time bounded by fuel

  and the job's time limit.

- The receipt's environment names the runtime, its version and the

  module's hash.

- Every Linux runner offers it (`tiers` gains `wasm`); phones and

  browsers offer only it.

What runs there is what compiles to WASM: linters, formatters, policy

checks, generated docs, and **test suites built for `wasm32-wasip1`**.

That last one makes the first demonstration (§11): a native job builds

the test modules, a phone runs them.

## 6. Leases and drain (oxygenflow#76)

**Lease.** A runner holding a job holds a lease, renewed by its logs and

a heartbeat. When the lease lapses the job is requeued, as today.

**Drain** is one message in the runner contract (jetstream.md §4) for

every *planned* departure:

| Trigger | Notice |

|---|---|

| SIGTERM, `flow` restart or update | the runner's grace (75 min on Earth) |

| Kubernetes node drain or eviction | the pod's grace period |

| Spot reclaim | 30 s (GCP), 2 min (AWS) |

| Phone off charge, off the allowed network, too hot | the owner's policy (a minute or so) |

| Browser tab hidden or closing | a few seconds |

On drain the runner decides from the job's learned duration (spec §11):

1. the rest is shorter than the notice: finish the job, then leave;

2. otherwise: save its checkpoints (§7), upload them, release the lease

   with `drained`, and stop the job.

A drained job is requeued at once, **ahead of new work**, and resumes

from its checkpoints. A runner that drains a job is not penalised; one

that drops leases without draining loses placement priority.

## 7. Checkpoints (oxygenflow#77)

A job may declare directories that hold its progress:

```kdl

job "build" {

    sandbox "rust"

    exec "cargo" "build" "--release"

    checkpoint "/src/target"

}

```

- **Saved** by content (FastCDC chunks, deduplicated, kept for the run's

  life) on drain and every few minutes while it runs (a sandbox setting,

  default 5 minutes, only while the job has run at least that long).

- **Restored** before a resumed job runs the **same exec again**. Tools

  that skip finished work (cargo's fingerprints, gradle, make, ninja)

  then carry on where they stopped; a tool that does not just starts

  again, so it is never wrong, only slower.

- **Receipts.** The partial run ends in a receipt with `exit` "drained",

  naming the checkpoint's hash as an output and signed by its runner. The

  resumed job's receipt names that checkpoint among its inputs. So

  provenance says that plex built the first part and mars finished it,

  and both are signed.

- **Cheap to move inside a cloud:** checkpoints live in the run's store

  (a bucket under Strata for cloud pools), so a new pod in the same region

  restores them quickly. From a phone, the policy (charger, office

  network) keeps uploads cheap.

- **Checkpoints are code.** A tampered `.rlib` in `target/` ends up in

  the binary. So a job accepts a checkpoint only from a runner of at

  least its trust class (§9), and its receipt says that it used one.

## 8. Hedging (oxygenflow#78)

When a markedly faster runner frees up while a job runs on a slow one,

and the learned durations say it would finish sooner (counting the time

to restore the latest checkpoint), start the job there too. The first to

finish wins; the other is cancelled.

- **Bounded:** one hedge per job, and only where a policy allows it (the

  critical path of a run, or always on owned runners). Never for jobs

  with side effects (deployments, publishing).

- **No tricky handover:** the slow runner is never interrupted. If the

  fast one fails, nothing is lost.

- **A free determinism check:** when both finish (a race), their outputs

  are compared; a difference is reported as a non-hermetic job.

- **Cost-aware in clouds:** starting a bigger instance costs money, so a

  pool may forbid hedging, or allow it only for jobs a person is waiting

  on.

Hedging plus holding briefly for a better runner (which exists) gets most

of the benefit of a full handover, for a fraction of the work. Build it

before anything finer (§12).

## 9. Trust classes (oxygenflow#79)

How far to believe a receipt depends on who signed it:

| Class | Who | Key |

|---|---|---|

| `attested` | cloud confidential VMs, vTPM-measured hosts | keyless workload identity (oxygenhub#122) |

| `owned` | the organisation's registered hosts | the runner's identity, or keyless |

| `device` | members' phones and laptops | the device's Keystore key, under the member's grant |

| `anonymous` | browsers not signed in | a key held non-extractable by WebCrypto |

- A sandbox or job may require a minimum (`trust "owned"`). Release and

  publish jobs default to `owned`.

- A job resumes only from checkpoints made at its class or above.

- Receipts and provenance record the class of every runner that

  contributed. The trust page shows a run's weakest link.

- **Spot checks:** a sample of jobs run by `device` and `anonymous`

  runners is run again elsewhere and the outputs compared (with

  oxygenhub#298's independent rebuilds). A runner whose results disagree

  is held for its owner to look at.

## 10. Phones and browsers

**Phones** (oxygenhub-android#8): one tap registers the phone as a

WASM-tier runner, as a foreground service, scoped like any runner. The

owner's policy says when it may run: on charge, on these networks (the

office Wi-Fi), under a temperature, within a storage budget. Leaving

those conditions drains it (§6). The runner itself is Flow's runner code

built for Android through the app's Rust core (rust-core, uniffi), with

wasmtime for the tier. The full Linux runner in Android's Linux terminal

(#9) comes later.

**Browsers** (#80): a page on the hub turns a tab into a WASM-tier runner

in a Web Worker: WASI through a shim, OPFS (the browser's private file

storage) for files, uploads as for any runner. Limits: about 4 GB of

memory, threads only on a cross-origin-isolated page, throttled or

stopped in a background tab (a drain). This is mostly a demonstration of

handover and spot checks on the least reliable runner there is, not

capacity.

## 11. The first demonstration

A pipeline over oxy-core:

1. `build-wasm-tests` (microvm, any x86_64 host): `cargo test

   --target wasm32-wasip1 --no-run`, outputs the test modules.

2. `test-wasm` (wasm, `runs-on "android"`, depends on 1): runs each

   module under the WASM tier, on the phone.

The script: start the run; the phone takes `test-wasm`; switch the

phone's Wi-Fi off halfway (a drain under its policy); the job goes back

to the queue and plex-1 takes it in its WASM tier; the run page shows

which runner did what, with the receipts and their trust classes. Then

the same with a pipeline that has a checkpoint, to show a job resuming

rather than restarting.

## 12. Order of work

1. **WASM tier** (#75), in `flow` on Linux runners.

2. **Phone runner** (oxygenhub-android#8) on that tier, with its policy.

3. **Leases and drain** (#76), with requeue ahead of new work.

4. The demonstration (§11), job-level: drain and requeue.

5. **Checkpoints** (#77): resume within a job; receipts that say so.

6. **Trust classes** (#79), before anything other than our own runners

   joins.

7. **Hedging** (#78), then measure on our own CI: queue and run times

   with and without it.

8. **Browser runners** (#80).

9. **Finer pieces,** only if the measurements call for them: test

   sharding and per-crate builds (#29), so that a long job is many short

   ones a better runner can take over between.

Cloud and Kubernetes pools come through Jetstream (jetstream.md §9,

oxygenflow#33), and get drain and checkpoints as they arrive.

## 13. Open questions

- Checkpoint retention: the run's life, or kept as a cache for the next

  run of the same job (a remote cache, which would then need the same

  trust rules)?

- How long may a drained job wait for a runner of its trust class before

  it falls back to restarting from nothing on any runner?

- Hedging costs compute: a per-pool budget, or a per-organisation one?

- A phone's battery and heat: are the owner's policy limits enough, or

  should the runner also cap its own CPU use?