Roles: one binary, several processes

Status: design, 2026-10-19 (overnight loop 17). H2 (--role and the leader lock, #272) is built; the event bus (H3) is next.

OxygenHub is one process today: pages, API, the oxy wire protocol, Flow's scheduler and every background task share one runtime and one data directory. That is the single-node edition, and it stays one command. This document says how the same binary runs as several processes, each taking some of the work (its roles), so the web tier can scale out and the background work runs exactly once, without a second product to maintain.

Principles

1. One binary, many roles. oxygenhub serve --role web,worker (all roles when none is given). A cluster runs the roles apart; a small site runs them together.
2. Seams before services. Each piece of shared state goes behind a trait inside the process first, with today's behaviour as one implementation. Splitting into processes is then a deployment choice.
3. Postgres before new infrastructure. Leader election (advisory locks), the job queue (FOR UPDATE SKIP LOCKED) and the event bus (LISTEN/NOTIFY) run on the Postgres the hub already uses (#211). NATS or JetStream come only when a measurement asks for them.
4. Lean on Oxygen's model. Changes are immutable and content addressed; only bookmark and tag moves are mutable, and they move by compare-and-swap of the operation head. Two processes writing one repository are already safe at the storage layer: that is what makes web nodes cheap to add.

The roles

Tonight's slices are web and worker. Edge and store are the repository service (a later phase): until then every node reaches every repository either through one shared data directory (one machine) or through bucket-backed repositories (oxygenhub#96), which any node can open.

What runs where today, and what each needs

Background tasks started in lib.rs::open:

Work spawned by requests (runs on whichever web node took the request):

• autoland::on_check_task after a check is recorded: serialised today by a process-wide TURN mutex, so two nodes could land one proposal twice. Needs a cluster-wide lock (an advisory lock per proposal).
• diffusion::propagate / try_land / catch_up / land_ready after a push: idempotent by design, but racing; the same per-proposal lock.
• The transparency log's appends in after_push ("logged before the push is answered"): one writer only. The log's files live in the data directory; two processes appending would corrupt them. Needs either an advisory lock around each append (one machine, shared directory), or a log role that the web nodes call (a cluster).

State kept in the process:

Leader lock

leader::run(app, name, task) (crates/hub/src/leader.rs): a process runs a singleton task only while it holds a lock named for it. On Postgres, pg_try_advisory_lock(key), the key a stable 64-bit hash (BLAKE3) of the hub's schema and the task's name, so two hubs in two schemas of one database never share a lock. Every lock of one process is held on one connection of its own, outside the pool, for the process's life. A process that does not hold a lock asks again every 2 seconds. One that holds it checks the connection every 2 seconds (one SELECT 1 answers for every task); if it has gone, the server has freed its locks, so each task it led is stopped (its future dropped) and the process contends again on a new connection. A process that dies closes its connection and frees its locks; another takes over within seconds (2 s in the test). A holder whose connection drops can overlap its successor for at most those 2 seconds, which the tasks tolerate. On SQLite (one machine by definition) the lock is always held. Each task is safe to run again after a failover: they read their work from the database. A task that ends by itself (diffusion.backfill) gives its lock back; the next worker to take it runs it again, harmlessly.

The locks: hooks.worker, flow.reaper, flow.scheduler, flow.placement, maintenance.nightly, backups.scheduler, diffusion.backfill, diffusion.reconciler (worker role) and tlog.checkpointer (web role, below). images.learn_archs runs in every worker, without a lock.

/health says what the process does and which locks it holds: "roles": ["web", "worker"], "leading": ["backups.scheduler", …].

Running the roles apart (H2)

oxygenhub serve --role web,worker (or [server] roles in hub.toml, OXYHUB_SERVER_ROLES); every role when none is given. An unknown role is an error.

• web serves the whole router. It also owns the transparency log: it appends to it (after_push, kept receipts and approvals) and runs its checkpointer, under the tlog.checkpointer lock. The checkpointer moved from the worker's list above because the log's state is in the process that appends: a checkpointer elsewhere would sign a log it never sees grow, and the web process would have no checkpoint to serve.
• worker starts the background tasks, and serves only /health and /metrics (when metrics_token is set) on its bind address, so it can be watched; with no bind address it serves nothing. It opens the transparency log to read only: it never cuts short an append the web process is making.
• Until the log has a cluster-wide writer (#274), only one process per data directory may have the web role. Any number may be workers. Webhook nudges from a push reach only the process that took it, so a separate worker delivers on its once-a-minute pass until the event bus (H3) carries the nudge.

Event bus: publish and subscribe

A trait with two operations, publish(topic, payload) and subscribe(topic) -> Stream, for the events live pages and workers react to: a bookmark moved, a proposal changed, a check recorded, a run or job changed state, a delivery is waiting, a live room's choice.

• In process: tokio::sync::broadcast per topic, what the hub does today. The single-node edition keeps it.
• Postgres: LISTEN/NOTIFY on one channel per topic. A payload is at most 8000 bytes, so events carry ids and kinds, and subscribers read the rows they need. A node publishes once; every node's subscriber relays the event to its local broadcast, so live pages on any node see it.
• Later, NATS JetStream (oxygenhub#208, #209): durable streams, replay from a position, and subscribers outside the hub (a product feature: "subscribe to this repository's changes" for integrations, beyond webhooks). The same trait.

Delivery is at most once; anything that must not be lost is a row in a table (the job queue), and the event only says "look now".

The job queue

A jobs table (kind, payload, run after, attempts, claimed by, claimed until). Workers claim with SELECT … FOR UPDATE SKIP LOCKED on Postgres, and with the same table polled under the write lock on SQLite. First tenants: webhook deliveries (they already have a table), mail, the Git mirror export, the apt rebuild, Diffusion's bumps, oxy gc. Runner claims stay on their own path (oxygenflow#34 may later move them to the queue).

The single-node edition

oxygenhub serve with no --role runs every role, with the in-process bus, the SQLite or Postgres database and the local leader: exactly today's behaviour. Every slice must keep it working and its tests green.

Order of work

1. H2: --role web,worker and the leader lock (Postgres advisory locks; always held on SQLite). Built (#272): two hubs on one Postgres run each singleton on exactly one, and the other takes over within seconds when it stops or loses its connection. One web process per data directory until step 3.
2. H3: the event bus trait, in process and Postgres LISTEN/NOTIFY; live pages on one node see a push made through the other.
3. The transparency log's appends under a cluster-wide lock (#274), and its checkpointer back with the workers.
4. Autoland and Diffusion under per-proposal advisory locks.
5. The job queue, webhooks and mail first.
6. Later: edge and store (the repository service), NATS when measured.