Roles: how one binary runs as several processes (design) 6b195ed3b637

# Roles: one binary, several processes

Status: design, 2026-10-19 (overnight loop 17). First slices: `--role`

with a leader lock (H2) and the event bus (H3).

OxygenHub is one process today: pages, API, the `oxy` wire protocol, Flow's

scheduler and every background task share one runtime and one data

directory. That is the single-node edition, and it stays one command. This

document says how the same binary runs as several processes, each taking

some of the work (its **roles**), so the web tier can scale out and the

background work runs exactly once, without a second product to maintain.

## Principles

1. **One binary, many roles.** `oxygenhub serve --role web,worker` (all

   roles when none is given). A cluster runs the roles apart; a small site

   runs them together.

2. **Seams before services.** Each piece of shared state goes behind a

   trait inside the process first, with today's behaviour as one

   implementation. Splitting into processes is then a deployment choice.

3. **Postgres before new infrastructure.** Leader election (advisory

   locks), the job queue (`FOR UPDATE SKIP LOCKED`) and the event bus

   (`LISTEN/NOTIFY`) run on the Postgres the hub already uses (#211). NATS

   or JetStream come only when a measurement asks for them.

4. **Lean on Oxygen's model.** Changes are immutable and content

   addressed; only bookmark and tag moves are mutable, and they move by

   compare-and-swap of the operation head. Two processes writing one

   repository are already safe at the storage layer: that is what makes

   web nodes cheap to add.

## The roles

| Role | Runs | Instances |

|---|---|---|

| **web** | pages, the JSON API, the `oxy` wire protocol (fetch and push), live pages (SSE), the runner API (claims, logs, finish) | many, behind a load balancer |

| **worker** | the background tasks below; one instance of each singleton at a time, chosen by a leader lock | one or more |

| **edge** (later) | the wire protocol and large uploads, routed to the node that holds a repository | many |

| **store** (later) | repositories on its disks, behind a service interface | several, by repository |

Tonight's slices are **web** and **worker**. Edge and store are the

repository service (a later phase): until then every node reaches every

repository either through one shared data directory (one machine) or

through bucket-backed repositories (oxygenhub#96), which any node can open.

## What runs where today, and what each needs

Background tasks started in `lib.rs::open`:

| Task | What it does | Role | Singleton? | Needs for several processes |

|---|---|---|---|---|

| `hooks::worker` | delivers webhooks from the `deliveries` table, nudged by `events.deliveries` | worker | yes today; many once deliveries are claimed with SKIP LOCKED | the nudge over the event bus |

| `tlog::checkpointer` | signs checkpoints, gathers witness cosignatures | worker | **yes** | the log's single writer (below) |

| `flow::reaper` | ends jobs whose runner stopped reporting | worker | yes | leader lock |

| `flow::scheduler` | starts `schedule` runs on their cron lines | worker | **yes** (two would start each run twice) | leader lock |

| `flow::placement` | the placement pass: why each job waits | worker | yes | leader lock |

| `maintenance::nightly` | nightly upkeep (gc, prunes) | worker | yes | leader lock |

| `backups::scheduler` | scheduled backups | worker | **yes** | leader lock |

| `diffusion::backfill`, `diffusion::reconciler` | the pin index and the bumps it drives | worker | yes | leader lock |

| `images::learn_archs` | reads kept images' architectures once | worker | no (idempotent) | none |

Work spawned by requests (runs on whichever web node took the request):

- `autoland::on_check_task` after a check is recorded: serialised today by

  a process-wide `TURN` mutex, so two nodes could land one proposal twice.

  Needs a cluster-wide lock (an advisory lock per proposal).

- `diffusion::propagate` / `try_land` / `catch_up` / `land_ready` after a

  push: idempotent by design, but racing; the same per-proposal lock.

- The transparency log's appends in `after_push` ("logged before the push

  is answered"): **one writer only**. The log's files live in the data

  directory; two processes appending would corrupt them. Needs either an

  advisory lock around each append (one machine, shared directory), or a

  log role that the web nodes call (a cluster).

State kept in the process:

| State | Today | In a cluster |

|---|---|---|

| state cache, summaries | per process, content-addressed | stays per node (safe: keys are hashes) |

| a repository's writer lock (`repos::Locks`, #249) | per process | the operation head's compare-and-swap already arbitrates; the lock only saves retries, so per node is correct |

| Flow placement hints (`CACHES`, `HELD`) | per process | per node is acceptable (a hint); the runner's own claim facts are the truth |

| live pages (`events`, `rooms`) | `tokio::sync::broadcast` | **the event bus** |

| rate limits (`throttle`) | per process | per node (limits per node), or Postgres later |

| sessions, tokens, accounts | the database | unchanged |

| keys (`data/keys/`) | files | shared directory or configuration |

## Leader lock

`Leader::run(name, task)`: a worker runs a singleton task only while it

holds a lock named for it. On Postgres, `pg_try_advisory_lock(hash(name))`

on a dedicated connection, retried every few seconds; if the holder dies,

its connection closes and the lock frees itself, so another worker takes

over within seconds. On SQLite (one machine by definition) the lock is

always held. Each task is written to be safe to run again after a

failover (they already are: they read their work from the database).

## Event bus: publish and subscribe

A trait with two operations, `publish(topic, payload)` and

`subscribe(topic) -> Stream`, for the events live pages and workers react

to: a bookmark moved, a proposal changed, a check recorded, a run or job

changed state, a delivery is waiting, a live room's choice.

- **In process:** `tokio::sync::broadcast` per topic, what the hub does

  today. The single-node edition keeps it.

- **Postgres:** `LISTEN`/`NOTIFY` on one channel per topic. A payload is at

  most 8000 bytes, so events carry ids and kinds, and subscribers read the

  rows they need. A node publishes once; every node's subscriber relays

  the event to its local broadcast, so live pages on any node see it.

- **Later, NATS JetStream** (oxygenhub#208, #209): durable streams, replay

  from a position, and subscribers outside the hub (a product feature:

  "subscribe to this repository's changes" for integrations, beyond

  webhooks). The same trait.

Delivery is at most once; anything that must not be lost is a row in a

table (the job queue), and the event only says "look now".

## The job queue

A `jobs` table (kind, payload, run after, attempts, claimed by, claimed

until). Workers claim with `SELECT … FOR UPDATE SKIP LOCKED` on Postgres,

and with the same table polled under the write lock on SQLite. First

tenants: webhook deliveries (they already have a table), mail, the Git

mirror export, the apt rebuild, Diffusion's bumps, `oxy gc`. Runner claims

stay on their own path (oxygenflow#34 may later move them to the queue).

## The single-node edition

`oxygenhub serve` with no `--role` runs every role, with the in-process

bus, the SQLite or Postgres database and the local leader: exactly today's

behaviour. Every slice must keep it working and its tests green.

## Order of work

1. **H2: `--role web,worker`** and the leader lock (Postgres advisory

   locks; always held on SQLite). Two hubs on one Postgres and one data

   directory: pushes through either, the scheduler and the placement pass

   run on exactly one, failover when it stops.

2. **H3: the event bus trait**, in process and Postgres `LISTEN/NOTIFY`;

   live pages on one node see a push made through the other.

3. The transparency log's appends under a cluster-wide lock.

4. Autoland and Diffusion under per-proposal advisory locks.

5. The job queue, webhooks and mail first.

6. Later: edge and store (the repository service), NATS when measured.