# Roles: one binary, several processes

Status: design, 2026-10-19 (overnight loop 17). H2 (`--role` and the

leader lock, #272) and H3 (the event bus, #273) are built; the

transparency log takes appends from every web process sharing its directory (#274).

OxygenHub is one process today: pages, API, the `oxy` wire protocol, Flow's

scheduler and every background task share one runtime and one data

directory. That is the single-node edition, and it stays one command. This

document says how the same binary runs as several processes, each taking

some of the work (its **roles**), so the web tier can scale out and the

background work runs exactly once, without a second product to maintain.

## Principles

1. **One binary, many roles.** `oxygenhub serve --role web,worker` (all

   roles when none is given). A cluster runs the roles apart; a small site

   runs them together.

2. **Seams before services.** Each piece of shared state goes behind a

   trait inside the process first, with today's behaviour as one

   implementation. Splitting into processes is then a deployment choice.

3. **Postgres before new infrastructure.** Leader election (advisory

   locks), the job queue (`FOR UPDATE SKIP LOCKED`) and the event bus

   (`LISTEN/NOTIFY`) run on the Postgres the hub already uses (#211). NATS

   or JetStream come only when a measurement asks for them.

4. **Lean on Oxygen's model.** Changes are immutable and content

   addressed; only bookmark and tag moves are mutable, and they move by

   compare-and-swap of the operation head. Two processes writing one

   repository are already safe at the storage layer: that is what makes

   web nodes cheap to add.

## The roles

| Role | Runs | Instances |

|---|---|---|

| **web** | pages, the JSON API, the `oxy` wire protocol (fetch and push), live pages (SSE), the runner API (claims, logs, finish) | many, behind a load balancer |

| **worker** | the background tasks below; one instance of each singleton at a time, chosen by a leader lock | one or more |

| **edge** (later) | the wire protocol and large uploads, routed to the node that holds a repository | many |

| **store** (later) | repositories on its disks, behind a service interface | several, by repository |

Tonight's slices are **web** and **worker**. Edge and store are the

repository service (a later phase): until then every node reaches every

repository either through one shared data directory (one machine) or

through bucket-backed repositories (oxygenhub#96), which any node can open.

## What runs where today, and what each needs

Background tasks started in `lib.rs::open`:

| Task | What it does | Role | Singleton? | Needs for several processes |

|---|---|---|---|---|

| `hooks::worker` | delivers webhooks from the `deliveries` table, nudged by the bus's `deliveries` topic | worker | yes today; many once deliveries are claimed with SKIP LOCKED | the nudge over the event bus (built, #273) |

| `tlog::checkpointer` | signs checkpoints, gathers witness cosignatures | web, beside the log's appends, until #274 | **yes** | the log's single writer (below) |

| `flow::reaper` | ends jobs whose runner stopped reporting | worker | yes | leader lock |

| `flow::scheduler` | starts `schedule` runs on their cron lines | worker | **yes** (two would start each run twice) | leader lock |

| `flow::placement` | the placement pass: why each job waits | worker | yes | leader lock |

| `maintenance::nightly` | nightly upkeep (gc, prunes) | worker | yes | leader lock |

| `backups::scheduler` | scheduled backups | worker | **yes** | leader lock |

| `diffusion::backfill`, `diffusion::reconciler` | the pin index and the bumps it drives | worker | yes | leader lock |

| `images::learn_archs` | reads kept images' architectures once | worker | no (idempotent) | none |

Work spawned by requests (runs on whichever web node took the request):

- `autoland::on_check_task` after a check is recorded: serialised today by

  a process-wide `TURN` mutex, so two nodes could land one proposal twice.

  Built (#275): `leader::exclusive("autoland:ID")`, a transaction-scoped

  advisory lock per proposal, held for each evaluation, beside the TURN.

- `diffusion::propagate` / `try_land` / `catch_up` / `land_ready` after a

  push: idempotent by design, but racing; built (#275): each bump and

  landing of a consumer's pin holds `diffusion:CONSUMER:PATH`.

- The transparency log's appends in `after_push` ("logged before the push

  is answered"): **one writer only**. The log's files live in the data

  directory; two processes appending would corrupt them. Needs either an

  advisory lock around each append (one machine, shared directory), or a

  log role that the web nodes call (a cluster).

State kept in the process:

| State | Today | In a cluster |

|---|---|---|

| state cache, summaries | per process, content-addressed | stays per node (safe: keys are hashes) |

| a repository's writer lock (`repos::Locks`, #249) | per process | the operation head's compare-and-swap already arbitrates; the lock only saves retries, so per node is correct |

| Flow placement hints (`CACHES`, `HELD`) | per process | per node is acceptable (a hint); the runner's own claim facts are the truth |

| live pages (`events`, `rooms`) | `tokio::sync::broadcast` | **the event bus** (built, #273); a resolver room's presence stays per node |

| rate limits (`throttle`) | per process | per node (limits per node), or Postgres later |

| sessions, tokens, accounts | the database | unchanged |

| keys (`data/keys/`) | files | shared directory or configuration |

## Leader lock

`leader::run(app, name, task)` (`crates/hub/src/leader.rs`): a process

runs a singleton task only while it holds a lock named for it. On

Postgres, `pg_try_advisory_lock(key)`, the key a stable 64-bit hash

(BLAKE3) of the hub's schema and the task's name, so two hubs in two

schemas of one database never share a lock. Every lock of one process is

held on one connection of its own, outside the pool, for the process's

life. A process that does not hold a lock asks again every 2 seconds. One

that holds it checks the connection every 2 seconds (one `SELECT 1`

answers for every task); if it has gone, the server has freed its locks,

so each task it led is stopped (its future dropped) and the process

contends again on a new connection. A process that dies closes its

connection and frees its locks; another takes over within seconds (2 s in

the test). A holder whose connection drops can overlap its successor for

at most those 2 seconds, which the tasks tolerate. On SQLite (one machine

by definition) the lock is always held. Each task is safe to run again

after a failover: they read their work from the database. A task that

ends by itself (`diffusion.backfill`) gives its lock back; the next

worker to take it runs it again, harmlessly.

The locks: `hooks.worker`, `flow.reaper`, `flow.scheduler`,

`flow.placement`, `maintenance.nightly`, `backups.scheduler`,

`diffusion.backfill`, `diffusion.reconciler`, `secrets.history_scan`,

`projects.archiver` (worker role) and `tlog.checkpointer` (web role, below).

`images.learn_archs` runs in every worker, without a lock.

`/health` says what the process does and which locks it holds:

`"roles": ["web", "worker"], "leading": ["backups.scheduler", …]`.

## Running the roles apart (H2)

`oxygenhub serve --role web,worker` (or `[server] roles` in `hub.toml`,

`OXYHUB_SERVER_ROLES`); every role when none is given. An unknown role is

an error.

- **web** serves the whole router. It also owns the transparency log: it

  appends to it (`after_push`, kept receipts and approvals) and runs its

  checkpointer, under the `tlog.checkpointer` lock. The checkpointer moved

  from the worker's list above because the log's state is in the process

  that appends: a checkpointer elsewhere would sign a log it never sees

  grow, and the web process would have no checkpoint to serve.

- **worker** starts the background tasks, and serves only `/health` and

  `/metrics` (when `metrics_token` is set) on its bind address, so it can

  be watched; with no bind address it serves nothing. It opens the

  transparency log to read only: it never cuts short an append the web

  process is making.

- **Several web processes may share a data directory (#274).** Each

  append to the transparency log holds `append.lock` in the log's

  directory exclusively, and starts from the log as the files hold it (it

  finishes what a crashed append left, then appends); readers catch up

  with every writer's entries, and the checkpoint is read from its file,

  signed by whichever process leads `tlog.checkpointer`. Across machines

  that needs the data directory on one shared filesystem with working

  `flock`; a log role the web nodes call is the answer for that, later.

  Any number may be workers.

## Event bus: publish and subscribe

A trait with two operations, `publish(topic, payload)` and

`subscribe(topic) -> Stream`, for the events live pages and workers react

to: a bookmark moved, a proposal changed, a check recorded, a run or job

changed state, a delivery is waiting, a live room's choice.

- **In process:** `tokio::sync::broadcast` per topic, what the hub does

  today. The single-node edition keeps it.

- **Postgres:** `LISTEN`/`NOTIFY` on one channel per topic. A payload is at

  most 8000 bytes, so events carry ids and kinds, and subscribers read the

  rows they need. A node publishes once; every node's subscriber relays

  the event to its local broadcast, so live pages on any node see it.

- **Later, NATS JetStream** (oxygenhub#208, #209): durable streams, replay

  from a position, and subscribers outside the hub (a product feature:

  "subscribe to this repository's changes" for integrations, beyond

  webhooks). The same trait.

Delivery is at most once; anything that must not be lost is a row in a

table (the job queue), and the event only says "look now".

### Built (H3, #273)

`crates/hub/src/bus.rs`: `app.bus.publish(event)` and

`app.bus.subscribe(topic)`, a `tokio::sync::broadcast::Receiver<Event>`.

Nothing to configure: a hub built for PostgreSQL carries events between

its processes over `LISTEN/NOTIFY`; on SQLite (one process) the bus is

in process only.

| Topic | Event | Published by | Heard by |

|---|---|---|---|

| `deliveries` | `delivery_waiting` | `hooks::emit` after queueing deliveries, a ping, a redelivery | the webhook worker (`hooks.worker`), wherever it leads |

| `proposals` | `proposal {repo, event}` (a check recorded on a state, with the check) | the checks API, Flow's checks | the proposal page's check stream (`…/proposal/{n}/events`) |

| `live` | `live {topic, kind}` (`repo:1` `push`, `run:5` `jobs`, …) | `hooks::emit` (every webhook event: pushes, proposals, issues, checks, reviews), notifications, Flow job changes, Diffusion bumps, assistants, comments | `/live` streams, through each process's replay buffer (`bus.live`) |

| `rooms` | `room {room, event}` (a choice, or the landing) | the resolver's live view | the same room's viewers in every process |

- **Payloads** are JSON, `{"o": origin, "e": event}`, well under

  PostgreSQL's 8000 bytes. A check's summary too long for it is cut on

  the way to other processes (the page reads the check for the rest); a

  room choice's own text too long for it stays in its process (logged).

- **Channels:** one per topic, `oxyhub_<hash>_<topic>`, the hash a

  BLAKE3 of the hub's schema, so two hubs in two schemas of one database

  never hear each other (as the leader locks' keys).

- **Publishing** delivers to this process's subscribers at once, then

  queues the event for a sender task that calls `pg_notify` through the

  pool, in order. A full queue (4096) drops the event, with a warning.

- **Listening:** one connection per process, outside the pool, `LISTEN`

  on every topic's channel. What it hears from another process it

  delivers to the local subscribers (and applies to the local resolver

  rooms), never publishing it again.

- **No loops, no doubles:** every process has a random origin id, sent

  with each event; a process skips events with its own origin, which it

  delivered when it published them. Each subscriber hears an event once.

- **Reconnecting:** a listening connection that fails, or does not

  answer a `SELECT 1` within 5 s after 5 s of quiet, is dropped and a new

  one listens (every 2 s until one does). The process logs that events

  published meanwhile were missed and nudges its own webhook worker to

  look now. Live pages are not told: the next event on their topics

  refreshes them, and a reload shows everything.

- **Live pages' numbering** (`Last-Event-ID`) is each process's own:

  every process numbers the events it hears, its own and others', in its

  replay buffer. A page that reconnects through another web node, or after

  a restart, with an id that node never gave is told to reload.

- **Resolver rooms:** choices and landings reach the room's viewers in

  every process; who is present, and a newcomer's snapshot of earlier

  choices, are each process's own. A choice posted to a process where

  nobody is in the room is still sent to the others but answers `410`,

  so the live view's stream and its posts should reach the same node

  (sticky sessions), as one web process per data directory does today.

## The job queue

A `jobs` table (kind, payload, run after, attempts, claimed by, claimed

until). Workers claim with `SELECT … FOR UPDATE SKIP LOCKED` on Postgres,

and with the same table polled under the write lock on SQLite. First

tenants: webhook deliveries (they already have a table), mail, the Git

mirror export, the apt rebuild, Diffusion's bumps, `oxy gc`. Runner claims

stay on their own path (oxygenflow#34 may later move them to the queue).

**Built, first slice (#276):** the `work_queue` table (migration 0091:

kind, payload, a dedupe key, run after, attempts, last error, claimed by,

claimed until, dead) and `db::queue` (enqueue, claim, done, failed,

counts). A claim leases a batch for five minutes (`UPDATE … WHERE id IN

(SELECT … FOR UPDATE SKIP LOCKED) RETURNING *` on Postgres; under the write

lock on SQLite), so any number of worker processes take items, and an item

whose worker died is due again when its lease ends. A failure backs off

(30 s, doubling, at most an hour); the eighth attempt leaves the item dead,

kept for a person to read; a kind no worker knows is dead at once. Every

worker runs `queue.worker` (no leader lock), polling every 2 s. Tenants so

far: a notification's **email** that failed to send (`mail`), and the

**Git mirror** export after a push that failed (`mirror.refresh`, one per

repository); both still try at once and queue only on failure, so the

usual path is as fast as before. Next: webhook deliveries onto it, the

apt rebuild, Diffusion's bumps, and a nudge on the event bus.

## The single-node edition

`oxygenhub serve` with no `--role` runs every role, with the in-process

bus, the SQLite or Postgres database and the local leader: exactly today's

behaviour. Every slice must keep it working and its tests green.

## Order of work

1. **H2: `--role web,worker`** and the leader lock (Postgres advisory

   locks; always held on SQLite). **Built (#272):** two hubs on one

   Postgres run each singleton on exactly one, and the other takes over

   within seconds when it stops or loses its connection. One web process

   per data directory until step 3.

2. **H3: the event bus trait**, in process and Postgres `LISTEN/NOTIFY`;

   live pages on one node see a push made through the other. **Built

   (#273):** a push through a web process reaches live pages and the

   webhook worker in another process at once; a process whose listening

   connection is lost listens again on a new one.

3. The transparency log's appends under a cluster-wide lock (#274), and

   its checkpointer back with the workers.

4. Autoland and Diffusion under per-proposal advisory locks.

5. The job queue, webhooks and mail first.

6. Later: edge and store (the repository service), NATS when measured.