Runners update themselves: through the swarm, checked as a release, opted into, tested on a few first issue #84 open andymccall opened this 10 days ago · 3 comments

Status 2026-10-27: the runner side is done (flow runner update --from, checks, flow.prev, leave_within; Windows too, issue #83/#85) and so is the hub's policy (oxygenhub#322: levels, test groups, the offer; runners poll every 15 min and install). Left: fetching the offered build through the swarm rather than over HTTP from the hub; waves and health (promote a version after a test group proves it, stop on failures); phones following app releases (oxygenhub-android#7).

Design note for what is left (2026-10-27 night), before building it.

Waves and health. Today a version reaches every runner whose policy has updates on at once (within its window), and a rollback anywhere holds the version for all (oxygenhub#322). Waves add an order:

• A runner group gets a wave number (1, 2, …; none = last). A version is offered to wave n+1 only once wave n has proved it.
• Proved = every runner of the wave that took the version has finished at least soak_jobs jobs on it (site setting, default 5) with none ending error (infrastructure, not test failures), and no rollback of it was reported; or soak_hours (default 24) have passed with no error and at least one job.
• To count jobs per version, a finished job records the runner's version at claim time (flow_jobs.runner_version, from the claim's info.version): the hub already has that string on every claim, it only needs keeping. A migration, no runner change.
• An error or a rollback in a wave stops the wave: the version is held as today, and Admin › Runners › Updates says which runner and job stopped it.
• Admin › Runners › Updates shows each wave's progress per version ("wave 1: 3 of 3 runners, 14 jobs, 0 errors — proved 02:10"); oh runner-updates the same.

Fetching through the swarm. An offered build is a 20–30 MB file the runner downloads from the hub. Through the swarm it would be content-addressed chunks (as images are), so runners on one site fetch from each other and the hub serves a release once per site. The pieces exist (jet's chunk store, the swarm picker, peer-held manifests from issue #94): a release asset becomes a one-file "image" manifest the hub records when the release is published, the offer names its manifest hash, and the runner's update path fetches it with jet::swarm before checking size and BLAKE3 as now. It matters most after the move to GCP (egress is paid per GB); on Earth today it saves little. Proposed after waves.

Order: runner_version on jobs → proved/stop per wave (with tests on the placement-free path: a fake clock and fake job rows) → the admin page and oh → then the swarm fetch.

Waves, first slice: on OxygenHub main (59fb2263, refs this issue). Not deployed yet; it goes out with the next hub deploy.

• A test group gets a wave (1 to 9) on Admin › Runners › Updates, or in an organisation's settings. A version reaches wave n+1 once every runner in the earlier waves that took it has finished soak_jobs jobs on it (5 by default) with none ending in error, or soak_hours (24) have passed since its first job there. Runners in no wave come last. A waiting runner's note says why: "flow 0.5.1 waits for wave 1 to prove it: 1 of 2 runners on it, 3 jobs…".
• A job that ends in error on a wave's runner stops the version. The next runner to ask holds it for every runner, and the hold names the job. A page view only shows the stop and holds nothing.
• Each job keeps the runner build it was claimed by: flow_jobs.runner_version, migration 0102.
• oh runner-updates shows each group's wave.

Still to do from the design note: weighing errors against the version before (today any error stops the version), the hub rolling a stopped wave's runners back, an admin "promote now", and fetching the build through the swarm (D2).