OxygenHub — Specification

The key words MUST, MUST NOT, SHOULD, SHOULD NOT and MAY are to be interpreted as described in RFC 2119.

Table of contents

1. Goals and non-goals
2. How it compares with GitHub
3. Architecture
4. Repositories on the server
5. Accounts, identities and authorization
6. Web interface
7. Proposals and review
8. Issues
9. Releases and tags
10. Notifications
11. Search
12. HTTP API and webhooks
13. Checks and OxygenFlow
14. Git interoperability
15. Security
16. Performance and caching
17. Operations: configuration, storage, backup
18. Required changes to Oxygen
19. Milestones
20. Open questions
21. Changelog

1. Goals and non-goals

1. Work like GitHub. A developer who knows GitHub must find repositories, history, diffs, review, issues, releases, tokens and webhooks where they expect them, with the same words where the words still apply. The learning curve is Oxygen's model, not the forge.
2. Show what the model makes possible. Multiple heads, reified conflicts, exact blame, signatures and trust policy, proposals that never go stale, review comments that survive rebases, and approvals that are portable signed objects. Each of these is a first-class page, not a footnote.
3. One binary, one directory. oxygenhub serve links oxy-core, embeds the wire protocol server, renders its own pages, and keeps everything under one data directory with SQLite for hub state. A backup is a copy of that directory. There is no separate database server, job queue, cache server or front-end build.
4. The repository is the source of truth. Everything about code, history, references, signatures and policy lives in the Oxygen object store and is readable with oxy alone. Hub tables hold only what Oxygen has no object for: accounts, sessions, issues, review threads, webhooks and check results. Deleting the hub database loses discussion, never history.
5. Off the shelf where it is not the product. Markdown editing uses EasyMDE, in-browser code editing uses CodeMirror 6, syntax highlighting uses syntect, Markdown rendering uses pulldown-cmark with ammonia sanitising. The hub's own code is the forge logic and the pages.
6. Honest state. A page never shows something the object store does not say. A signature is shown as verified only when the hub verified it against policy; an approval is shown only when the approving seal exists.

Non-goals for 1.0: hosting Git repositories natively (§14 covers mirrors), a marketplace, discussions, GitHub Pages, and federation between hubs. The runner is specified separately (§13, docs/runner.md) and is not required for 1.0.

2. How it compares with GitHub

3. Architecture

3.1 One process

oxygenhub is a Rust binary in its own Cargo workspace at ../oxygenhub, depending on oxy-core and oxy-server by path (developer edition) or by version (release). It runs one tokio runtime with:

• an axum router serving pages, static assets, the JSON API, and the wire protocol under every repository's path;
• askama templates compiled into the binary; one hand-written stylesheet, also compiled in and served at /style.css;
• SQLite through sqlx (WAL mode, synchronous=NORMAL, foreign keys on) for hub state, opened from <data>/hub.db;
• a small background worker pool (tokio tasks) for webhooks delivery, notification email, search indexing, Git mirror export and cache warming;
• an in-process cache (§16);
• an event bus for live pages and workers (crates/hub/src/bus.rs): pushes, checks, Flow job changes, comments, waiting webhook deliveries and the resolver's choices are published on it and heard by the pages' server-sent event streams and the webhook worker. On SQLite it is in process; built for PostgreSQL, a hub run as several processes (--role web,worker, docs/roles.md) carries every event to the others over LISTEN/NOTIFY, at most once: events say "look now" and name ids, never the data.

There is no separate front-end build. JavaScript is limited to: the theme cookie, EasyMDE (Markdown editing), CodeMirror 6 (file editing), and small progressive enhancements (history graph hover, keyboard shortcuts, live check status). Every page MUST render and every form MUST work with JavaScript disabled, except the two editors, which degrade to a plain <textarea>.

3.2 Layout on disk

<data>/
  hub.db                      SQLite database (and -wal, -shm)
  hub.toml                    configuration (§17.1), optional; environment overrides
  keys/                       the hub's own identity (delegate key, §5.5) and TLS material
  repos/<owner>/<name>/       one Oxygen repository per directory, bare (§4.1)
  mirrors/<owner>/<name>.git  optional Git mirror (§14)
  assets/<owner>/<name>/      release assets, content addressed by BLAKE3
  cache/                      rebuildable: rendered pages, graph layouts, search index

<owner> and <name> MUST match [A-Za-z0-9][A-Za-z0-9._-]{0,63} and MUST NOT be ., .., or end in .git (reserved for the mirror URL). The reserved top-level path segments are api, static, login, logout, register, settings, explore, search, notifications, new, orgs, admin, style.css, favicon.ico, favicon.svg, health, avatar, help, projects, assets, .well-known, live, csp-report, security.txt, security-contact, log, metrics, insights and drops; no owner may take those names. No repository may be named drops (in any case): /{owner}/drops is the owner's Drops page (§4.5). A drop's repository is repos/<owner>/drops/<id>/.

3.3 Request path

1. Middleware: request id, tracing span, body limits, session cookie decode, CSRF check on unsafe methods (§15.2), rate limits (§15.6).
2. Wire protocol routes /{owner}/{name}/oxy/v1/* go to the embedded oxy-server handlers with the hub's repository resolver and authorizer (§18).
3. Page and API handlers resolve {owner}/{name} to a RepoHandle, check the viewer's role, and run every oxy-core call inside spawn_blocking under the repository's locks (§16.2: readers never wait for writers).
4. Templates receive plain structs; no template touches oxy-core directly.

3.4 Crate layout

oxygenhub/
  Cargo.toml            workspace: crates/hub (binary), crates/hub-api (types shared with clients)
  crates/hub/src/
    main.rs             CLI: serve, init, admin (§17.3)
    config.rs           hub.toml and environment
    db/                 sqlx migrations and queries, one module per table group
    repos/              RepoHandle, locks, cache, bare repository operations
    auth/               sessions, passwords, TOTP, tokens, identity linking
    web/                page handlers, one module per area (repo, change, proposal, issue, …)
    api/                JSON handlers
    protocol.rs         embedding oxy-server
    render/             markdown, syntax highlighting, diff rendering, graph layout
    jobs/               background workers
  templates/            askama; base.html, style.css, one file per page
  assets/               wordmark, favicons, vendored EasyMDE and CodeMirror bundles
  migrations/           sqlx migrations
  docs/                 this file, runner.md, deploy.md

4. Repositories on the server

4.1 Bare repositories

A hosted repository is an Oxygen repository directory with no working copy checked out and no user workspace. oxy-core today always keeps one working-copy change per workspace; the hub never snapshots, commits from a checkout, or touches files outside .oxy/. The directory contains only .oxy/. §18 lists the small change that lets Repo::open skip working-copy bookkeeping for such a directory.

Creation: Repo::init under <data>/repos/<owner>/<name> with a fresh repository id, then an initial row in repositories. Import: a client oxy push into an empty hub repository; or, for Git, §14.

Deletion moves the directory to <data>/trash/<owner>/<name>-<timestamp> and a job removes it after trash_days (default 30). Transfer (oxygenhub#108, Settings › Advanced, the repository's admins) moves it to a user or organisation the admin may create repositories under, keeping its name: a directory move under the repository's write lock (the Git mirror moves with it), the row's owner, and a repo_redirects row so the old address redirects on the web (the rest of the path kept) and still resolves for oxy remotes and the API. Archive marks it read-only with a reason: pushes are refused by the wire protocol, and the web refuses edits, issues, proposals, comments and reviews, while reading, cloning and the admins' settings go on; the header shows an archived badge and the admins bring it back from the same place. Both are audited (repo.transferred, repo.archived, repo.unarchived). The Advanced section also holds Delete and, ahead of wikis (oxygenhub#104), a disabled Delete wiki data.

4.1a Repositories in the owner's bucket (oxygenhub#96)

A repository can keep its data in its owner's bucket instead of on the hub's disk.

• Creating: the New repository form's Storage section takes a store URL (s3://BUCKET/PREFIX, gs://BUCKET/PREFIX or az://ACCOUNT/CONTAINER/PREFIX, with ?region= the only setting allowed) and the credentials for it: an S3 access key, a Google service account key, or an Azure service principal. The hub refuses a profile, a key file, an endpoint or dir://, which would make it use its own host's credentials, reach its own network or write to its own disk.
• The check: before anything is made, the bucket or prefix must be empty and pass the store conformance check under a probe prefix. Nothing is created when either fails.
• What the hub keeps: a bare repository directory whose storage.url names the store, with a cache in front (oxygen#68), and a repo_storage row holding the URL and the credentials, sealed under keys/secrets.key. At start the hub hands every repository's credentials to the cloud stores by URL, so nothing from its host's environment or cloud profiles is used for a repository.
• Pages: the header shows a bucket tag; its title names the URL to the repository's admins only. Settings › Storage shows the store and the last check, checks it again, and replaces the credentials once new ones pass the check. Both are audited as repo.storage.
• Organisation policy: Settings › Storage on an organisation can require every new repository to live in a bucket, and can list the store URL prefixes allowed (its own buckets, so its data stays where they are). The New repository form refuses anything else for that organisation, saying what is allowed. Repositories that exist are not moved. Changes are audited as org.storage_policy.
• Not yet: forking such a repository is refused (clone and push instead). Deleting one leaves its data in the bucket, which is the owner's. The Sync status and runs on references moved outside the hub are still to come.

4.2 Forks

A fork copies the object store (hard links where the file system allows, otherwise a copy) and records forked_from. Because objects are content addressed and repository ids differ, a proposal from a fork carries the changes across repositories by hash; the hub fetches missing objects from the fork's store directly, never over the network.

4.3 The default bookmark

Each repository has a default bookmark (main unless changed). It is what the repository page, tree, file views and search show, and the target every proposal picks first.

4.4 Visibility

public (readable by anyone including anonymous clients), internal (readable by every signed-in account), private (readable by collaborators). Wire protocol reads obey the same rule: a private repository answers 404 to an unauthorized client, not 401, so its existence is not disclosed.

4.5 Drops (oxygenhub#279)

A drop is a small snippet of one or more files, as a gist is, and like a gist it is a repository: each drop is its own Oxygen repository, with its own history, visibility and address, cloned and pushed with oxy.

• Names and addresses: a drop's repository is named drops/ID under its owner (a user), where ID is 24 random lowercase hex digits, unique on the hub (96 bits, so an address cannot be guessed). No repository name can hold a slash and none may be drops, so the two never collide. The drop's page is /{owner}/drops/{id}, and oxy clone https://HUB/{owner}/drops/{id} takes that address: the wire protocol answers at /{owner}/drops/{id}/oxy/v1/… with the drop's repository behind it, under the same locks, limits and push path as any repository. The row is in repositories with kind = 'drop' (migration 0087) and a title; its description is the drop's.
• Visibility: public drops are listed on their owner's Drops page for everyone; secret drops are listed only to their owner (and site administrators) and are readable by anyone with the address, signed in or not, on the web, the API and the wire protocol (a secret drop is stored as private, so anything that lists by visibility hides it; auth::roles grants the read). The transparency log page names a secret drop's repository to nobody but those it is listed to.
• Never among repositories: drops are left out of every list, count and search of repositories: profiles, organisation pages, the dashboard and its sky, Explore, /search, GET /api/v1/repos, forks, backups (for now), Diffusion and the Flow scheduler. /api/v1/repos/{owner}/drops%2F{id} is 404; a drop's repository pages under /{owner}/drops/{id}/… (history, change, raw) are read-only. Upkeep that covers every stored repository (gc, the history scan, Admin › Storage, which labels them, oxygenhub admin repo list and oxygenhub check) includes them.
• Pages: /drops goes to your own Drops page, /{owner}/drops, which lists drops newest first with their files, and /drops/new makes one: a title (optional: the first file's name otherwise), a description, public or secret, and one or more files, each a name and a text area (hub.js adds and removes rows; without it the form has its rows and one empty one). The drop's page shows every file rendered (Markdown as a document, other text highlighted, binary or large files as a download link), the clone command, and for its owner Edit and Delete. Edit is the same form; saving makes one change with only what differs (a file whose name is cleared is removed; files that are not text, pushed with oxy, are kept). History (…/changes/main), each change with its diff (…/change/{hash}), raw files (…/raw/{ref}/{path}) and the files at a change (…/tree/{ref}) are the repository pages with the drop's own bar (Files, History) in place of the tabs.
• Changes: what the web and the API make is authored by the owner's linked identity and signed by the hub as its delegate, exactly as web edits are (§6.9), so the owner must have delegated to the hub's key first; the forms say how. The first change makes main. At most 50 files of 1 MiB each go in one change from the web or the API (larger ones go with oxy push).
• Who may do what: anyone with the address reads a drop; only its owner writes (the web, the API, and pushes with the owner's token); its owner or a site administrator deletes it. Flow never runs on a drop; its pushes ring no webhooks and close no issues, and are recorded in its push log.
• Push protection (§15.4) applies to a drop's pushes as to any repository's, with no fingerprint to allow (a drop has no settings), and the hub runs the same rules over what it is asked to commit from the web or the API, refusing before anything is stored.
• Deleting moves the repository to the trash under its exclusive lock (oxygenhub#249) and removes the row; the address stops answering. Audited as drop.created, drop.edited, drop.visibility and drop.deleted.
• Forks (#280): a signed-in visitor who is not the owner may fork a drop they can read (POST /{owner}/drops/{id}/fork, the page's FORK): a new drop of theirs whose store is a fork of the source (objects linked, bookmarks copied, as a repository fork), with its title, description and secrecy, forked_from naming the source; its page says "forked from" with the source's address while it exists. Audited as drop.forked.
• Stars (#280): a signed-in visitor stars a drop, or takes the star back, from its page (POST /{owner}/drops/{id}/star, the repositories' stars table); the page shows the count, which never joins a list of starred repositories.
• Comments (#280): signed-in readers comment on a drop in Markdown (POST /{owner}/drops/{id}/comment, table drop_comments, migration 0092, at most 64 KiB), shown oldest first under its files; the owner is told (drop.comment); the author, the owner or a site administrator deletes one (…/comment/{n}/delete).
• Embedding (#280): GET /{owner}/drops/{id}/embed (whoever may read the drop) is the drop's files and a link back, without the site's chrome, sent with its own policy (default-src 'none'; style-src 'self' 'unsafe-inline'; img-src 'self' data: https:; frame-ancestors *; …, so no X-Frame-Options): any site may frame it, and it runs no script and posts no form. The drop's page offers the <iframe> to copy.
• API: GET /api/v1/drops (yours, repo:read), POST /api/v1/drops (NewDrop, repo:write), GET /api/v1/users/{name}/drops (someone's public drops), GET /api/v1/drops/{id} (anyone with the id; each text file's content up to 512 KiB), PATCH /api/v1/drops/{id} (DropEdit: title, description, secret, and files to write or, with content: null, remove; others are kept) and DELETE /api/v1/drops/{id}. oh drop create|list|view|delete (docs/oh.md).
• Later: forks, stars, comments, embedding a drop in another page, and drops in backups.

5. Accounts, identities and authorization

5.1 Accounts

An account has a username (same grammar as an owner name), a display name, an email (verified before it is used for anything), an Argon2id password hash, an optional TOTP secret, a session salt (rotating it signs out every session), and a set of linked Oxygen identities (§5.3). The first account created becomes the site administrator, and only with the hub's one-time setup code (oxygenhub#245): with no accounts, the hub keeps a random code in <data>/setup-code (owner-only, never served) and logs it at start, or uses [registration] setup_code (OXYHUB_REGISTRATION_SETUP_CODE) for automation; registering with it spends it, and a sign-in provider never makes the first account. oxygenhub admin user create --admin needs no code. Registration is open, invitation only, or closed, per configuration.

Sessions are server-side records (oxygenhub#223): the cookie oxyhub=<id> holds 32 random bytes in hex, and the hub keeps only the id's BLAKE3 hash, with the account, the account's session salt at sign-in, the times it was made and last used, and the address and browser it came from. A session is good while its salt is still the account's, it has been used in the last 14 days, and it is under 30 days old; the cookie is HttpOnly, Secure when served over HTTPS, SameSite=Lax. Every sign-in (after the code, with two-factor) makes a new session; signing out deletes its record. Password changes, resets and "sign out everywhere" rotate the salt, which ends every session at once; Settings › Security lists the signed-in browsers and signs any one out. Ended sessions are swept nightly. CSRF tokens are bound to the session id. Sign-in is also slowed against guessing (§15.6, oxygenhub#221). Passkeys (WebAuthn) are planned for 1.0 as a second factor or password replacement.

5.1.2 Sign in with GitHub or Google (done 2026-10-04)

An administrator sets up each provider under Administration › Sign in with: the OAuth client id, the client secret (sealed under keys/secrets.key, never shown again) and a switch; the page shows the callback URL to register with the provider (/login/{provider}/callback). The login page then offers the usable providers (oxygenhub#106). The flow is OAuth 2.0's authorization code: /login/{provider} sets a state cookie (a nonce and the purpose, MACed with the cookie key, ten minutes, path /login) and sends the person to the provider; the callback checks the state against the cookie, swaps the code for a token server to server, and asks who it is (GitHub: /user and the primary verified address from /user/emails; Google: the OpenID userinfo, the address only when email_verified). Then: a subject already linked signs in; else an account with that verified email is linked and signed in; else, when registration is open, or it is by invitation and an unused invitation came through the round trip (/login/{provider}?invite=CODE, carried in the signed state as the purpose invite:CODE, used up when the account is made; the registration page shows the provider buttons for a usable invitation, oxygenhub#214), an account is made from the provider's login name (made valid and free), with an unusable password and the email verified when the provider vouched for it; otherwise the person is told to register or sign in and connect. A signed-in person connects or disconnects a provider under Settings › Security; a subject can be linked to one account. Links live in account_logins. Audited as admin.signin, signin.connected, signin.disconnected, login and register with the provider. For tests, [auth] signin_mock names a base URL that stands in for every provider.

5.1.1 Email verification and password reset

Both use account links: 32 random bytes as hex in a URL, stored only as their BLAKE3 hash, bound to one account and one purpose, expiring, and consumed on first use.

• Verification. Setting or changing an email marks it unverified. When the hub can send mail it sends a link to the new address (valid 48 hours). The link verifies that exact address: one sent to an address the account has since replaced does nothing. The settings page shows whether the email is verified and can send a new link: at most one every two minutes for the same address, and ten an hour per account whatever the addresses, so the form cannot be used to mail other people. Notification email goes only to verified addresses.
• Reset by email. /login/reset takes a username or a verified email. When it matches an active account with a verified email and the hub can send mail, a link valid for one hour goes to that address, at most one every two minutes per account. The page answers the same way whether or not anything matched, so it cannot be used to find accounts.
• Reset by an administrator. When the hub cannot send mail, or the account has no verified email, an administrator issues a reset link from the admin page (valid 24 hours), shown once to hand over by other means. The audit log records who issued it.
• Using a reset link sets a new password (the same rules as registration), rotates the session salt so every existing session ends, voids the account's other reset links, and records password.reset in the audit log. It does not turn off TOTP: a reset account with two-factor sign-in still needs its code. A banned account cannot be reset.
• Mail. [email] smtp is an smtp:// or smtps:// URL, or file:///path for development, which writes each message to that directory as a file instead of sending it. Administration has a Mail panel (#101, #181) whose settings, once saved, override [email]: pick a service and give only what it needs (Resend: its API key; Amazon SES: the region and SES SMTP credentials; Mailgun: us or eu and the SMTP login; another SMTP server: server, port, security, user and password; or an SMTP URL, for anything else and file://), and the panel stores one standard SMTP URL, so every service is SMTP underneath and any provider works. Keys and passwords are never shown again; left empty, the saved one stays while the service, server and user do. An smtp:// URL that says nothing of TLS requires STARTTLS (?tls=opportunistic or ?tls=none say otherwise; a relay on this machine needs none). Off turns mail off, and a button sends a test message to the administrator's own address. Every message goes as plain text and as HTML in the site's look (#182): the Oxygen band with the mark, sent inline so it shows without loading anything, the subject as heading, the body with its links, the message's action as a button with its link beneath, and a footer of the site's links (Explore, Help, Notification settings, and Stop these emails on notifications).

5.2 Organisations and teams

An organisation is an owner with members and teams. Roles on an organisation: owner, member. Roles on a repository, whether personal or organisational: read, triage (manage issues and proposals without writing), write, maintain (settings except deletion and secrets), admin. Teams hold members and receive repository roles as a unit. An organisation's default repository role for members is configurable.

5.3 Linked identities

An Oxygen identity is a signed object holding an Ed25519 + ML-DSA-65 key pair's public halves (spec §14.2). An account links an identity by uploading the identity object (oxy identity export) and proving possession: the hub issues a 32-byte challenge and the client signs it with oxy identity sign-challenge (§18); the hub verifies both signatures. A linked key is what makes a pushed change this account's change: author attribution in every page comes from the change's signing key, not from who held the token that pushed it. A change signed by an unlinked key shows its identity's name (or KeyId when the identity is unknown) with a "key not linked" tag rather than a user name.

An identity can be linked to at most one account. Rotation (oxy identity rotate, renew, delegate add|remove) produces a successor identity whose object names the old key. When a push brings changes authored by a successor of a linked identity, and every step of the chain kept the key pair, the hub links the successor to the same account itself and records identity.rotated in the audit log: delegating to an agent never breaks attribution and needs no re-upload. A rotation to a fresh key pair (oxy identity rotate) still needs the new identity linked with a challenge.

5.4 Authentication for the wire protocol

push requires one of:

1. Authorization: Bearer <token> with a personal access token holding repo:write on that repository;
2. an HTTP signature (spec §15.1): the client signs a server challenge with its identity key and the key is linked to an account with write. Reference implementation support is a §18 item; until then tokens are the only method.

Reads on public repositories are anonymous. Reads on internal and private repositories require the same credentials with repo:read.

5.5 Two authorities, one answer

The hub's roles decide who may reach a repository, open issues, create proposals, edit settings, and attempt a push. The repository's Oxygen trust policy (spec §14.5) decides which keys may update which references and whether conflicts may land on them. Both must agree for a push to succeed; the hub MUST NOT bypass or reimplement policy, and it MUST run the same receive_push validation oxy-core runs everywhere else.

A repository with no policy accepts any reference update from any account with write. The settings page says so plainly and offers to create a policy. Policy editing in the browser produces a policy proposal: a signed policy object that roots sign with oxy policy sign from their own clients; the hub collects signatures until the threshold is met, then applies the update as an ordinary push. The hub never holds a root key.

The hub has one identity of its own, the hub delegate key, generated at init and published at /.well-known/oxyhub-key. It is used only to sign changes the hub itself creates on behalf of a signed-in user (web edits, §6.9; proposal landings, §7.6). Such a change carries the user's linked key as author and the hub key as signer, which the author-or-delegate rule (spec §14.4) accepts only when the user's identity object lists the hub key as a delegate with the changes and refs scopes (Oxygen spec §14.2; the hub signs the change and attests the reference), still in force: the user opts in once with oxy identity delegate add <hub key> --scope changes,refs and re-links the rotated identity (§5.3). Policy is then checked exactly as for the user's own signature, since a rule that names the author covers the author's delegates (spec §14.5). If the user has not delegated, or the policy does not authorise them for the reference, the hub refuses the action and shows the client recipe. Delegation is per user and revocable by the user alone (oxy identity delegate remove); the hub never needs a policy entry of its own.

5.6 Personal access tokens

Tokens are 32 random bytes, shown once, stored as BLAKE3 hashes, with a name, an optional expiry, and scopes: repo:read, repo:write, repo:admin, issues:write, proposals:write, user:read, hooks:admin, optionally restricted to listed repositories. Tokens authenticate the API and the wire protocol; they never authenticate the pages.

Device sign-in (in the shape of RFC 8628) gives a token to an app that cannot take a password, such as oh or a phone client. The app posts {client, scopes} to /api/v1/device/code without a token and gets a device code (kept, hashed, by the hub), a user code XXXX-XXXX (consonants), verification_uri (/device), verification_uri_complete, expires_in (15 minutes) and interval (5 seconds). It shows the user code and link, and polls /api/v1/device/token with {device_code}, getting authorization_pending, slow_down (polled within the interval), access_denied or expired_token as {error, error_description} with 400. The user opens /device, signed in (with their second factor), sees the app's name and the scopes it asked for, and approves with any non-empty subset of them, or denies. The next poll makes a token named device: CLIENT with the approved scopes and returns {access_token, token_type: "bearer", scope, username}, once: the code is then gone. No token exists until that poll, so none is stored in the clear. Approvals, denials and device tokens are in the audit log.

6. Web interface

6.1 Theme

The default look (since 2026-10-01, oxygenhub#74) is close to GitHub's, chosen for readability and familiarity: Primer colours, light by default (a theme cookie of dark or auto changes it), the system font stack at 14px, rounded boxes with divided rows, the site band and repository tabs as underline navigation, grey buttons with green primary actions, sentence case throughout, and pill labels. It is assets/github.css, loaded over the stylesheet below with every rule scoped to :root[data-look="github"], which the page sets unless the browser chose the classic look with ?look=classic (a cookie keeps the choice; ?look=github returns to the default). The Oxygen mark stays the logo, with the wordmark OxygenHub.

The classic look follows the RetroPlay Network's shape (dark by default, Barlow for text and Press Start 2P for the wordmark and first columns, square corners, thin --line rules, a --panel header and footer, a full-width --band navigation bar with the active tab underlined in --acc) with the palette taken from the Oxygen mark shared with the VS Code extension: the mark's cyan #00F0FF to blue #0072FF gradient on #070c18, so --acc is cyan, --acc-2 blue, and the band a deep blue. The mark itself is the logo in the header, the footer and the favicon; the wordmark beside it reads OXYGENHUB. The stylesheet starts from retroplay-manager/templates/style.css lines 1–171 (tokens, shell, band, tabs, main, tables, forms, footer) and adds forge components; the store, showcase and game components are not carried over.

The theme is dark, light, or auto (follow the system's prefers-color-scheme, and change with it), chosen with the switch in the header (it cycles) or under Settings → Appearance, and kept in a theme cookie for a year; dark when unset. A script in the page head applies it before first paint. Syntax highlighting uses the highlighter's scope names as classes prefixed sy-, so no scope can collide with the page's own classes.

Tokens (dark default, light under :root[data-theme="light"]):

--bg  --panel  --soft  --ink  --mute  --line  --dim
--acc  --band  --band-ink  --band-line  --now  --now-ink
--ok  --bad  --warn (new: #ffb347 dark / #b8651b light)
--add  --del  --add-ink  --del-ink   (new: diff backgrounds and text)
--m1 … --m5   (kept for signature status colours: root, signed, sealed, unsigned, invalid)

Theme selection: cookie theme=light|dark|auto; auto follows prefers-color-scheme. The <head> sets data-theme before paint exactly as retroplay-manager does. Fonts are vendored under /static/fonts/ rather than loaded from Google, so a hub on a private network paints without internet access.

The logo is assets/icons/oxygenhub.svg, a copy of the extension's oxygen.svg (two linked rings, the molecule), inlined in the header at 22px beside the wordmark.

6.2 Page shell

header (wordmark, global search box, "+" menu for new repository/issue/proposal, notifications bell with unread count, avatar menu) · .band (Repositories · Explore · Proposals · Issues · Notifications, with admin links pushed right) · repository .tabs (Code · Changes · Proposals · Issues · Flow · Releases · Wiki · Artifacts ▾ · More ▾ · Settings, open proposals and issues with a count pill; oxygenhub#254). Artifacts holds Releases, Build artifacts, Images, Terraform state and Reports (when there are any); More holds Bookmarks, Tags, Diffusion (when used), Trust, Pushes and Conflicts. The menus are <details>, so they open without JavaScript; hub.js closes one when another opens, on a click outside and on Escape. Under 760 pixels, where the bar scrolls sideways, a menu opens inline in the bar. Projects belong to the owner and are reached from the site band and the owner's profile; a state with conflicts says so on its overview · main at 1180px max (wider than retroplay's 980 because diffs need it; tree and file pages use 100% with 22px gutters) · footer.

Every page has a <title> of the form <thing> · <owner>/<name> · OxygenHub.

6.3 Pages

6.4 Repository overview

The default bookmark's tree at the root, the rendered README.md (or README, README.txt) below it, and a right column: description and topics, clone URL with copy button (oxy clone https://…/{owner}/{name}), heads count with a link when more than one, bookmarks, tags, latest release, collaborators, license detected from LICENSE, and the trust badge: policy, no policy, or policy expired.

A repository whose default bookmark has more than one head shows a --warn strip: "main has 3 heads; the tree below is their union" with a link to the bookmarks page. Conflicts in that state show a --bad strip linking to the conflicts page.

6.5 Tree

Directory rows show name, the last change touching the entry (message and relative time, from log_for_path, computed lazily per row over the API so a large directory paints immediately), size, and mode glyphs (executable, symlink, subrepository). A subrepository row links to its pin: another hub repository at a state, or a Git URL at a SHA.

6.6 File view and blame

Text files render with syntect line highlighting (a text with a line over 4 000 bytes, a minified file, is shown plain: syntect's time grows much faster than a line), line numbers as anchors (#L12, #L12-L20 selects a range), a "raw", "history", "blame", "edit" toolbar, and the file's last change. Markdown renders by default with a toggle to source. Images and PDFs render inline; other binaries offer download. Files above render_limit (default 2 MiB) show the first part and a download link.

Blame uses Repo::blame: every line is exact, because each line is an atom with one authoring change. The gutter shows the change's short hash, author and age, grouped into runs, with the change's signature glyph. Blame is exact through rebase and squash, so there is no "ignore revs" file.

A sidebar beside the file lists the tree down to it: the root and each directory on the file's path expanded (at most 200 entries each, then a link to the directory), the file marked, other directories linking to their tree pages. It is hidden on narrow screens.

The permalink of a file view is the same path at its state: the target's heads joined with + in place of the bookmark or tag. y replaces the address with the permalink, keeping the line anchor, as GitHub does. When a line or range is selected, a bar offers Copy lines (the text) and Copy permalink (the permalink with #L12-L20).

The blame gutter shades each run by the age of its change, from the file's oldest (lightest) to newest (strongest) line, in ten steps, with the age also written out so colour is not the only carrier. Each run links blame before this change: the same file blamed at the state of the change's dependencies (the state its author built on), so a line can be traced back through every edit. A change without dependencies has no such link; one with more than 32 has none either, to keep addresses bounded.

Function history. For Rust, TypeScript, Python and shell files, tree-sitter finds the functions and methods (qualified by their impl, trait or class, as Repo::log) and the file page's sidebar lists them. Each links to /{owner}/{name}/function/{ref}/{path}?start=…&end=…&name=…: every change that wrote or deleted one of the function's lines, or deleted a line from between them (Oxygen spec §27, the query oxy log -L runs), newest first, with a graph of how those changes depend on one another. It is exact because lines keep their identity across edits; changes elsewhere in the file are not listed. A cross-file semantic graph (callers, callees) is future work.

6.7 History

A dependency graph, not a parent chain. The default view lists the bookmark's changes newest first with lanes drawn between a change and its dependencies (the same layout algorithm as the VS Code extension's webview, ported to Rust and rendered server side as SVG). Heads are marked. Filters: author, path, date, signed/unsigned, message text. Each row: short hash, message first line, author (name, or KeyId when unlinked), relative time, signature glyph, bookmark and tag chips, a conflict chip when the change introduces one.

?path= gives the history of one path.

Swimlanes. /{owner}/{name}/lanes/{ref} draws a bookmark's newest changes (up to 60) as a board: each lane of the shared lane layout (oxy_core::lanes, as oxy log and the list view use) is a row, time runs left to right, every change is a card linking to its page, and an edge joins a change to each change of the board it depends on. Independent work stays in separate lanes and there are no merge nodes, since Oxygen has none. The history page links to it.

Planning on the board. When the bookmark is the source of an open or draft proposal that the viewer wrote or can write to, the board is editable. Dragging a change onto another asks POST /{owner}/{name}/lanes/{ref}/plan (form change, target) what the drop could mean. The answer is JSON: a squash (fold the change into the target) when every dependency of the change is the target or already beneath it, and a rebase (make the change depend on the target) unless it already does. Nothing is offered when the target depends on the change, or when either is already on the proposal's target bookmark. The hub does not rewrite the changes itself: rewritten changes must be signed by their author, so each option carries the oxy commands to run and a vscode://oxygenhub.oxygen-vscode/plan?… link that does the same in the VS Code extension, then force-pushes the bookmark.

6.8 Change page

Header: full hash, message, author identity (linked account or KeyId), timestamp, signature status with the policy rule that admitted it (or "unsigned", or "invalid"), seals covering it, the bookmarks and tags containing it. Body: dependencies and dependents as chips, the change's edits as a diff (unified or split, per-user preference), atom counts, and the raw change (oxy show --json) behind a details block.

Actions for write: open a proposal containing this change, tag it, copy hash. Describing, abandoning, squashing and rebasing are client operations and are not offered here.

6.8.1 Diffs

Every diff (change, compare, proposal files) shows each file as a collapsible section, open unless it has more than 400 rows. Within a block of changed lines, removals come first and each is paired with the addition in the same position; a pair similar enough (at least 40% of its tokens shared, tokens being identifiers, whitespace runs and single other characters) marks the tokens that changed. ?w=1 hides whitespace-only changes: a pair equal once whitespace is removed is shown as one unchanged line. With scripts, a toolbar switches between unified and split (side by side, pairs opposite each other) views, remembered in a diff cookie, and collapses or expands every file.

6.9 Web edit

CodeMirror 6 with the file's language mode. Saving creates a change with the given message signed by the hub delegate on behalf of the user (§5.5) and adds it to the chosen bookmark, or to a new bookmark plus a proposal, exactly as GitHub's "propose changes" flow. The hub refuses with a clear message when the user has not delegated to the hub key (§5.5) or the policy does not authorise them for the target. Binary files and files over render_limit cannot be edited in the browser.

6.10 Conflicts page

For each bookmark whose state has conflicts: the file (or tree entry), the conflict kind, and each side with the change that produced it and its content, rendered with the same marker structure oxy checkout writes. For a text file, resolve region by region (/{owner}/{name}/resolve/{bookmark}/{path}, writers) shows the lines no one disagrees on and each conflict region with its sides and the change that wrote each; per region the user keeps one side, every side in order, none, or their own text (a side that records a deletion keeps the deletion). Regions are linkable (#region-N). The file is put back together and committed through the delegate as a web edit (§6.9), which makes it a kind = resolution change depending on every contributor; the page carries the bookmark's state and refuses to commit if the bookmark moved meanwhile. Malformed markers fall back to the editor, which also stays available: "edit the marked file" opens the markers in it. A name conflict (several files want one name, shown as NAME.oxy-conflict-N) resolves on the same URL by keeping one candidate: it takes the name as a move, keeping its identity, and the other candidates are removed. A replace conflict (a binary file replaced two ways) resolves the same way by keeping one version, each labelled with the change that wrote it; the commit replaces every version with it (Oxygen spec §9.3). Conflicts between directories or links are resolved from a client. Both resolver pages offer Resolve in VS Code, a vscode://oxygenhub.oxygen-vscode/resolve?url=…&bookmark=…&path=… link that oxygen-vscode handles: it finds the open checkout of that repository (or offers to clone it), switches to the bookmark after asking, pulls it, and opens the file. Resolving is live: everyone on the same resolver page (a room per repository, bookmark and path, over server-sent events at /{owner}/{name}/resolve-live/{bookmark}/{path}) sees who else is there and each other's choices as they are made, region by region, the last choice for a region winning (free text is shared whole, a moment after typing stops); a newcomer gets the choices so far; and when one of them commits, the others are told the resolution landed, with a link to it, and their form is disabled. Rooms are in memory and end when the last viewer leaves; with several web processes, choices and landings reach the room's viewers in every process over the event bus, while presence is each process's own (docs/roles.md). The change page tags resolution changes. Otherwise the page gives the two-line client recipe.

6.11 Trust page

The current policy (serial, expiry, roots, threshold, rules with their patterns, thresholds and allow_conflicts), the policy history, the hub delegate key with its fingerprint and which collaborators have delegated to it, every linked account's keys that the policy names, revoked keys, and a verification summary computed from oxy verify: counts of signed, sealed, unsigned and invalid changes, with a list of invalid ones. Policy proposals awaiting root signatures are listed with who has signed and a download of the proposal file.

6.12 Pushes page

The hub keeps its own log of every reference update it accepted or refused: time, account, key, reference, old and new state, force flag, the result and reason. This is the answer to "who moved main and when" and is the hub's audit trail for history; it is not the repository's operation log, which is per workspace.

6.13 Keyboard and accessibility

/ focuses search, g c / g p / g i jump to Code, Proposals, Issues, y on a tree or file page switches to its permalink, ? shows the sheet. Every interactive element is reachable by keyboard and shows a visible focus ring; a "Skip to content" link comes first; the header, site band, repository and settings tabs are named navigation landmarks, and the current tab carries aria-current. Colour is never the only carrier of meaning (glyphs accompany every status colour). Text meets WCAG AA (4.5:1) on every surface in both themes, checked by a test over the theme variables; --dim is for rules and decoration, never text. Under 760 pixels the tab bars and wide tables scroll sideways instead of breaking the layout, sectioned pages stack their section list above the content, and the header search hides (the / key and the search page remain).

6.11 Wiki (done 2026-10-04)

A repository may have a wiki (oxygenhub#104): Markdown pages kept as an Oxygen repository of their own under <data>/wikis/<owner>/<name>, with no row of its own; it follows the repository's visibility, roles, archiving, transfer and deletion. A page Getting-Started is the file Getting-Started.md at the root of bookmark main; _Sidebar.md is shown beside every page, and Home is the front page. On the web (/{owner}/{name}/wiki, /wiki/{page}, /wiki/{page}/edit, /wiki/{page}/history, /wiki/_pages with a search over titles and text, /wiki/_raw/{path} for images): readers read, writers create, edit, rename and delete pages, and every edit is a change authored by the user and signed by the hub as their delegate, exactly as a code edit (§6.9), so it needs the same delegation. Links between pages are relative ([Other](Other)). By client, the wiki is oxy clone HUB/{owner}/{name}.wiki: the wire protocol maps that name onto the wiki store with the repository's permissions, pushes need write access and not archived, start no run and ring no webhook; a repository can therefore not be named *.wiki. A wiki takes no proposals, issues or runs: the API answers a reader asking for one under OWNER/NAME.wiki with a 404 that says so and how to publish a bookmark (merge it into main and push), not "no such repository" (#128). The Wiki tab shows when a wiki exists or the viewer may start one. Settings › Advanced › Delete wiki data moves the wiki to the trash. Audited as wiki.edited, wiki.deleted, wiki.data_deleted; webhooks get a wiki event.

7. Proposals and review

7.1 Definition

A proposal is a request to add a set of changes to a target bookmark of a repository. It is identified by a per-repository number, like a pull request. Its source is either a bookmark of the same repository or of a fork, in which case the proposal's changes are those in the source bookmark's state not in the target's; or an explicit list of change hashes (from the change page or the API).

A proposal records: number, title, description (Markdown), author, target bookmark, source (bookmark and repository, or explicit hashes), state (draft, open, landed, closed), reviewers requested, labels, and the projected state (§7.2).

7.2 Projected state

Whenever the proposal or its target changes, a job computes the projected state: the union of the target's heads and the proposal's changes, loaded through oxy-core exactly as a client's oxy merge would. From it the page shows:

• the changes to be added, in dependency order, with signature status;
• the diff between the target's state and the projected state;
• conflicts the landing would reify, each with its sides, in the same rendering as §6.10;
• files touched, and check results for the projected state hash (§13).

Because changes commute, a projected state is stale only when the target has moved; the job then recomputes it. There is no "update branch" button because there is nothing to update. If the recomputation shows a new conflict, the proposal is marked conflicted and the author and reviewers are notified.

7.3 Review comments

A comment is attached to the proposal (general), to a file, or to an atom: (ChangeHash, index) plus the change that introduced the line. Rendering places it at the atom's current position in the projected state; if the atom is gone (deleted by a later change in the proposal), the comment shows in the file's "resolved lines" section rather than disappearing. When the author squashes or rebases, the correspondence the rewrite recorded maps the old atom to the new one, and the comment follows. Comments are Markdown (EasyMDE), threaded, and resolvable.

7.4 Reviews and approvals

When a proposal's target moves (a push, or another proposal landing), its projected state is new and its checks would be for the old one: the hub starts a run of the pipeline on each open proposal's new projected state, labelled proposal/N with the trigger proposal, unless that state has been run already (#134). Its author, or anyone with triage access, closes and reopens a proposal on its page or with PATCH /api/v1/repos/{o}/{n}/proposals/{number} and {"state": "closed"} or {"state": "open"} (oh proposal close N [-m MSG], oh proposal reopen N; #140); asking for the state it already has changes nothing, and landing has its own endpoint. The author or a writer can ask for a review from anyone who can read the repository (not the author), under Reviewers on the proposal page or with POST /api/v1/repos/{o}/{n}/proposals/{number}/reviewers and {"reviewers": [...]}; DELETE .../reviewers/{username} withdraws a request. Each person asked gets a notification (proposal.review_requested), the timeline records it, the proposal's JSON lists requested_reviewers, and a review from that person answers the request. Every review notifies the proposal's author (proposal.review). An AI assistant the repository may use (§13.4) can be asked the same way; its answer is a comment review recorded via the assistant under the asker's name.

A review is comment, request changes, or approve. An approval MUST be backed by a seal attestation (spec §14.4, kind = seal) over the proposal's change hashes, signed with a key linked to the reviewer's account. Private keys never enter the browser, so the reviewer signs from a client: oxy seal --proposal <url> or oxy seal <hashes> (§18, a new command; the seal object already exists in the model) signs and pushes the seal; the hub matches the seal to the proposal and records the approval. The page shows the command ready to copy. An approval not yet backed by a seal is shown as "approval pending signature" and does not count towards requirements.

Landing requirements per repository (settings): N approvals, approvals from code owners (.oxyowners in the target state, GitHub CODEOWNERS syntax), required checks green on the projected state, no conflicts (or conflicts allowed, matching the policy rule's allow_conflicts; conflicts that are all adjacent insertions in merge=union files, oxygen#145, do not count: landing adds their resolution, both sides kept, earlier change first, authored by the person landing and signed by the hub as their delegate), and no outstanding "request changes". A repository under an organisation of two or more members starts with one approval required (someone other than the author can give it; nobody may approve their own proposal), any other with none; an admin changes either in Settings. When none is required and nobody has approved, the proposal page says so beside Land, and Land asks for confirmation.

7.5 Stacked proposals

A change may depend on changes in another open proposal. The page shows "depends on #12" and lands only after #12 has landed, or lands both when the user chooses "land with dependencies" and holds the rights for each.

7.6 Landing

Landing sets the target bookmark to the projected state's heads: a reference update with a fresh ref attestation. It is performed by:

1. the hub delegate key on behalf of the user landing, when that user has delegated to it (§5.5), the policy authorises them for the target, and they hold write; or
2. a client: oxy proposal land <url> (§18) fetches the proposal's changes, sets the bookmark locally and pushes with the user's own key. The hub notices the push, sees the proposal's changes are now in the target, and marks it landed.

The landed state hash is recorded. Nothing is rewritten: the changes keep their hashes, so links to them stay valid and check results carry over. "Squash on land" is offered as a client-side option only, since it rewrites signed changes.

Landing over checks (oxygenhub#261). When required checks are the only thing in the way (pending or failed on the projected state, for instance with the runners down and the fix in this very proposal), a repository maintainer or admin may land now with a reason: Land now, over the checks on the proposal (a reason field, and a confirmation), POST /repos/{owner}/{name}/proposals/{n}/land with {"over_checks": "REASON"}, or oh proposal land N --over-checks REASON. Only the checks are passed over: approvals, code owners, changes requested, drafts, dependencies and the trust policy still apply, a writer is refused, and an empty reason is refused. How it landed says so ("web, over checks: the runners are down"), on the proposal, its timeline and the pushes page, and it is audited (proposal.land_over_checks, with the reason and the checks that had not passed).

Land when checks pass (oxygenhub#261). Anyone who may land a proposal can set it to land by itself: Land when checks pass on the proposal, POST /repos/{owner}/{name}/proposals/{n}/auto-land (DELETE turns it off), or oh proposal land N --when-ready (--off). Whenever a check is recorded on a state, the open proposals set to land on that projected state are looked at: once every required check has passed and nothing else blocks it, it lands as the person who set it ("when its checks passed (set by NAME)"), if they may still land; a required check that fails or is cancelled turns it off, with a note on the timeline; a push to the source makes a new projected state whose checks run, so it waits again. It shows on the proposal ("lands when its checks pass (set by NAME)") and in the API (auto_land); setting and clearing it are on the timeline and audited (proposal.auto_land). Migration 0081 (proposals.auto_land_by, auto_land_ms).

7.7 Closing and reopening

closed keeps everything; the source bookmark may be deleted on close if the user asks. A proposal whose changes all appear in the target through another route is marked landed automatically with a note.

8. Issues

Per-repository numbered issues sharing the number space with proposals (as GitHub does; so every number is shown with its kind, "issue #1" or "proposal #2", in page headers and in the links #N becomes), with title, Markdown body, author, assignees, labels (name, colour, description), a milestone (title, due date, description, open/closed), state (open, closed with reason completed or not planned), comments, lock, and pin. Issue templates come from .oxyhub/ISSUE_TEMPLATE/*.md in the default state.

The proposal page. The conversation is centred at the normal page width; the changes and files tabs are wide for diffs.

• Timeline: the description, comments, reviews and events are one timeline in time order, each entry beside its author's avatar.
• Editing: the title, target and description are edited inline (an Edit button beside the title opens the editor below the timeline), not from the sidebar.
• State actions: ready for review, back to draft, close and reopen sit under the comment box, and only when they apply.
• Sidebar:
  • Landing, Closes, Labels and Projects.
  • Labels: those with triage edit them, or are pointed at the repository's labels page when it has none.
  • Projects: editors of the owner's projects add the proposal to one (its first column) or take it off. Issues have the same project control.
• Redirects: every action redirects back to the page, so reloading it (the checks script does, when a check arrives) never repeats the action.

The chain of work. A proposal shows the issues it closes (closes, fixes or resolves #N in its description or in any of its changes' messages) in a strip under its title: each issue with its state, then the proposal, then its target bookmark, marked landed once it is. A Closes list sits in its sidebar. An issue shows the proposals that close it, by the same rule, with their states. So the path from a report to the change that landed it can be followed from either end.

Relative links and images in issue, comment and proposal text resolve against the repository's default bookmark, and in release notes against the release's tag. Cross references: #12 links an issue or proposal; owner/name#12 across repositories; a 12-hex-or-longer token links a change when it resolves; closes #12 in a change message closes the issue when the change lands on the default bookmark ("closed by change"), however it arrives: every change in the default bookmark's new state and not in its old one counts, whether a push imported it, a client's oxy proposal land moved main to changes the hub already held, or the hub landed a proposal (whose description closes issues too); #151. Any other #12 in a pushed change's message, on any bookmark, puts "referenced by change on " on that issue's or proposal's timeline, the hash linking to the change; other repositories' owner/name#12 are left alone.

Import from GitHub or GitLab is a 1.x item.

8.2 Projects

A project is a board that plans work across the repositories of one owner (a person or an organisation): a number per owner, a title, a description, public or private, open or closed, and ordered columns (new projects start with Backlog, Doing and Done). A column holds ordered cards; a card is an issue or proposal of one of the owner's repositories, or a note.

• Who: anyone may see a public project; a private one only its owner (for an organisation, its members) and site administrators. The owner, an organisation's members and administrators may change it. A card for an issue or proposal whose repository the viewer cannot read is not shown at all.
• The board works without JavaScript: each card has a column picker and up and down buttons; cards are added by reference (#12 in a named repository, repo#12, or owner/repo#12 as the board shows them, the board's owner only) or as a note; an issue or proposal is on a project at most once. Issue and proposal pages list the projects they are on, with their column.
• Columns can be added, renamed, moved and removed (removing one moves its cards to the first column). Closing a project keeps it read-only.
• Long boards (oxygenhub#293): a column of more than 30 cards shows its newest 30, with "show all". A card shows its issue's or proposal's labels and an issue's assignees, and the filter above the board takes words (title or reference) with label:NAME, assignee:NAME and state:STATE, every term holding, as the issue list does.
• Archive (oxygenhub#293): a card can be archived, off the board but kept, found on the board's archive view (?archived=1, filtered the same way) and restored to the end of its column. A column archives its cards that joined it more than N days ago at once, or on its own (checked hourly): by default its finished cards (closed issues, closed or landed proposals) 14 days after they reached it, so a Done column shows the last fortnight's work and an ongoing project's board stays its size; or every card after N days; or never. Open work and notes are never archived by the default. An issue's or proposal's page says when its card is archived.

9. Releases and tags

Tags come from the repository (spec §10): a name, the state, and a signed attestation when the tagger signed. The tags page lists them with signature status. A writer creates one there (#295; POST /tags, oh tag create NAME [--target REF] [-m MESSAGE]): the hub signs it with its key as the user's delegate (in a repository without a trust policy, with its own key when the user has not delegated), and it is received as a pushed tag is, so the policy's rule for tags/<name> applies, the tag jobs of the pipeline run, and the push is recorded. Tags never move: a mistaken one is withdrawn (#301; Oxygen spec §10.2) by a maintainer there (DELETE /tags/{tag}?reason=, oh tag delete NAME [-r REASON]) or by oxy push --delete tags/NAME, with a signed untag attestation, never while a release uses the tag (delete the release first; a push's withdrawal is refused then too). The withdrawal is logged in the transparency log (untag), audited (tag.withdrawn) and recorded as a push; the page lists withdrawn tags with why, by whom and when, clones drop the tag on their next fetch, and the name is not used again. A release attaches to a tag: notes (Markdown, generated draft from the changes since the previous release), assets (uploaded files, content addressed, size limited by asset_limit), a pre-release flag, and a latest marker. Source archives (.tar.zst, .zip) of the tagged state are generated on demand and cached.

Trust and audit (§9.1). /{owner}/{name}/releases/trust/{tag} reports on a release; each release on the releases page links it.

• Tag: the tag's signer and whether its attestation verifies.

• Authors: everyone whose changes the release adds since the previous release (all of its changes for the first release). For each: how many changes are signed, sealed, unsigned, invalid or signed by revoked keys, and the account their key is linked to.

• Assets: for each asset, what built it: the run and job, the runner's identity (and whether it is a key one of this hub's runner tokens signs with), and the pinned environment (tier, rootfs hash, environment hash). Also its inputs (the state built, the pipeline hash and the scope hash) and its signed receipt, verified again from the repository.

• Graph: authors → tagged state → jobs → assets, drawn as SVG. A dashed edge is a build from another state than the tag's.

• Audit: pass, warn or fail for each claim:

  • the tag is signed;
  • the changes are authenticated;
  • each asset has a verifying signed receipt from a known runner;
  • each asset was built from the tagged state;
  • each asset's environment is pinned by hash.

  A hand-uploaded asset, an unsigned tag or an unsigned change is a warning. A signature or receipt that does not verify, or a revoked key, is a failure.

• Export: ?format=json downloads the report, and GET /api/v1/repos/{owner}/{name}/releases/trust/{tag} returns the same. The JSON carries each signed receipt as hex, so an auditor can check it offline with oxy receipt verify.

Built by. A release asset links to the build that produced it: the passed OxygenFlow job of this repository with an artifact of the same BLAKE3 (hub spec §13.2). A job with a kept signed receipt comes first, then the newest run. The release page shows the run, the job and the signed receipt with its signer, and the API gives the same as built_by on each asset. With a signed receipt it also links the asset's SLSA provenance (provenance_url, §13.2's provenance of that artifact, which states the same bytes), so a release asset is checked with cosign verify-blob-attestation --key oxyhub-signing.pem --type slsaprovenance1 --insecure-ignore-tlog=true --signature PROVENANCE ASSET (oxygenhub#239). On a run page, a writer can attach an artifact to a release; it becomes an asset named after the artifact's path, so the link follows by content. A file uploaded by hand links the same way when its bytes match.

10. Notifications

Events: proposal opened, review requested, review submitted, comment on a thread you are in, mention (@name), landing, check failed on your proposal, issue assigned, policy proposal awaiting your signature, push refused. Each becomes an inbox item (/notifications, with unread count in the header) and, when the account chooses, an email through lettre with a per-account SMTP-independent unsubscribe link. Watching a repository subscribes to all its events; participating in a thread subscribes to it.

The inbox shows one row per thread (the page a notification is about, such as an issue or a proposal), with its newest event, how many updates it had and how many are new. Each notification records why it was sent: participating (the account opened, commented on, was assigned, mentioned in or asked to review the thread) or watching (it only watches the repository); a thread counts as participating if any of its notifications does. The inbox filters by view (inbox, unread, done), by reason, by kind (issue, proposal, release, …) and by repository, with counts over the whole inbox. A thread can be marked read or unread, and done, which sets it aside (read) until something new happens in it. Opening a row marks its thread read and goes to the page.

Who hears, and of what (oxygenhub#183, 2026-10-14). A repository's owners (the user, or an organisation's owners) watch it by default and stop by unwatching it, which mutes it (watches.muted, migration 0065); others watch by choice. A proposal Diffusion opens is told as one opened on the web is. The actor is never told of their own doing, except through a token they marked as an agent's (Settings › Tokens, "an agent uses this token"): a request with it runs with the token's name in notify::AGENT, and its events reach the owner like anyone else's, the actor shown as andymccall (agent token Claude). OxygenFlow tells a run's pusher and the watchers when it fails (flow.failed) and when a bookmark is green again (flow.fixed: the run before on it failed, or this run had failed and a job of it was run again). Backups tell the owner when a run is not whole (once per spell), when one is whole again (backup.recovered), and once a day from 07:00 UTC with the day's runs, outcome, repositories and bytes sent (backup.summary). The site's administrators hear of each new account (account.registered). Each goes to the inbox, and by email to accounts that turned email on and verified their address.

11. Search

Global search box with scopes: repositories (name, description, topics), issues and proposals (FTS5 over title, body and comments), changes (message FTS5 over the default bookmark's state, indexed on push), accounts, and code. Code search in 1.0 is a per-repository substring and regular-expression search over the default state's text files, run with a time budget and a result cap, cached per state hash. A trigram index is a 1.x item.

12. HTTP API and webhooks

12.1 API

JSON under /api/v1, bearer tokens, application/json, RFC 7807 problem responses (an unknown route, or a known one with the wrong method, too), cursor pagination (?after=), ETags on cacheable resources, and rate limits per token. Resources mirror the pages: /repos/{owner}/{name}, /bookmarks, /changes/{hash}, /tree/{ref}/{path}, /file/{ref}/{path}, /blame, /history/{ref} (?path=, ?offset=&limit=, or ?r=REVSET for the changes a revset selects, evaluated under a two-second budget and answering 400 when it does not parse or goes past it; Oxygen's docs/revset_query_language.md), /compare, /conflicts, /proposals (?state=open|draft|landed|closed|all, open ones and drafts by default; a source bookmark may hold /, and owner/name:bookmark naming the repository itself is its own bookmark), /proposals/{n}/comments|reviews|checks|files, PATCH /proposals/{n} (the title and description, for its author or a maintainer, and labels, the whole set by name, for triage; what is not given stays), POST /proposals/{n}/land (repo:write: what the Land button does, 409 with the blockers when it cannot; a landed proposal keeps the changes it landed and reports landed with who, how (web, or api (token NAME)) and when, which its event, the audit log and the push record also say, adding when nobody approved), /proposals/{n}/content/{base|head}/{path} (a changed file's bytes in the target's state or the projected state, for editor diff views), /issues (creating an issue can set its labels, assignees and milestone; PATCH /issues/{n} edits the title and body for the author or triage, and replaces labels, assignees or the milestone for triage, recording each on the timeline; unknown names are refused before anything changes), /labels (POST creates or changes one, DELETE /labels/{name} removes it; triage), /milestones (POST creates one or changes the description and due date of the one with that title, PATCH /milestones/{title} changes title, description, due date or state; triage), /releases (PATCH /releases/{tag} changes the title, notes or prerelease flag, repo:write), /tags (POST creates one, repo:write; DELETE /tags/{tag} withdraws one, a maintainer, 409 while a release uses it; §9), /trust, /pushes, /hooks, /checks, /user, /user/repos (the caller's own, organisations' and collaborations, ?affiliation=&after=&limit=, #302), /user/orgs, /notifications (the caller's inbox and unread count, each with its repository, reason, thread and label, #305; /notifications/{id}/read and /notifications/read mark read; scope user:read), /users/{name}, /users/{name}/drops and /drops (§4.5), /orgs/{name}, /checks/{state} (the checks on a state, named by its hash, by its heads joined with +, by the one head of a single-head state, or by a bookmark), /checks/required (GET, and PUT of the whole list by a repository admin: the checks a proposal needs, passed, to land), /projects/{owner} (list, create) and /projects/{owner}/{n} (the board as the caller may see it, PATCH for title, description, privacy and closing, and owner to transfer it to an owner the caller may make boards under (#125: it takes the next number there, and its old address redirects), /cards to list them by column or add a reference or note to a column named by id or name, /cards/{id} to move or remove one, POST /cards/move (#289: {"from", "column", "except", "state", "dry_run"}) to move a column's cards at once, but those except names (card ids or issue and proposal references) and, given state, only those whose issue or proposal is in one, all or none and audited once, answering what moved and what was kept; POST /cards/archive (#293: {"from", "older_days", "except", "state", "dry_run"}, the same selection and an age) to archive them, GET /cards?archived=true to list the archived ones (each with archived_ms) and POST /cards/{id}/restore to put one back; cards carry their labels and assignees; repo:write and the page's rights to change), /coverage and /tests (a bookmark's coverage and JUnit test reports from Flow jobs, newest first, ?bookmark=&limit=), POST /coverage (repo:write, #136: {"bookmark"} or {"change"} and content, an LCOV, Cobertura or JaCoCo report measured off the hub, kept for that state as uploaded by the caller, shown "uploaded by NAME" wherever a run number would be, and setting no check; oh coverage upload FILE), /coverage/{id} (a report with its files) and /tests/{id} (a report with its cases, ?status=failed for the failed and errored ones), /flow/runs (GET: runs newest first, ?before=N&limit=; POST {"bookmark"} or {"change"}, repo:write: run a bookmark's current state, the default bookmark when none is named, or one change's state; with "jobs": [NAME..] only those jobs and what they depend on, each one a manual run starts, else 409 naming the ones it does (#130); oh run start --job NAME), /flow/runs/{n} (a run with its jobs), /flow/runs/{n}/jobs/{job}/log (?after=seq&limit=) and /flow/runs/{n}/rerun (repo:write: a finished run again), both answering 201 {"number", "url"} or 409 with why nothing started, /flow/runs/{n}/jobs/{job}/rerun (repo:write, oxygenflow#31: that job of a finished run again, in the same run, with the jobs that depend on it; what it depends on keeps its result, so a flaky test reruns without its build; their logs, artifacts and reports go, their checks return to queued and the run is running again; {"queued": N}, or 409 while the run is live or for a job it lacks; RE-RUN THIS JOB on the run page, oh run rerun N --job NAME), /flow/runs/{n}/cancel (repo:write: a run still going ends as cancelled, its running jobs stopped when their runner next reports; 409 when it had already ended; a push to a bookmark cancels the push runs of it still going), /search. A change's author carries the account its key is linked to, or null and a note saying the key is not linked and where its owner links it; via names the delegate (an agent) that signed for the author when one did, with the delegation's scope and expiry and whether it has since been revoked, and the pages show the same as an agent badge, a via row under the author on the change page, and a count under the author in Contributors (the change page shows a "key not linked" tag, with a link to Settings → Keys for signed-in viewers). Shapes reuse oxy --json shapes wherever one exists (change, status, blame, diff) so tools written against the CLI work against the hub. crates/hub-api holds the types with serde derives and is what the runner and the VS Code extension will link.

12.2 Webhooks

Per repository and per organisation: a URL, a secret, an event selection, active flag, and delivery log with redelivery. Each webhook has a format: oxygenhub (the signed JSON envelope below), discord (an embed with a title, link and short detail for the event, sent with mentions disabled so no user text can ping anyone) or slack (escaped text); a Discord or Slack URL picks its format by default, and every event has a one-line summary for the chat formats. The settings show a webhook's URL with its long path segments and query cut short, since chat webhooks keep a secret token there (the audit log records the same masked form), and each webhook's last delivery result; a webhook can be edited (URL, secret, events, format, active), a blank URL or secret keeping the current one. Repository admins manage a repository's webhooks under its settings; organisation owners manage the organisation's under its settings, and those hear every repository the organisation owns. Payloads are JSON with X-Oxyhub-Event, X-Oxyhub-Delivery, and X-Oxyhub-Signature: blake3-keyed=<hex> computed as keyed BLAKE3 over the body. Events: push (reference updated: old and new state, heads, changes), proposal (opened, edited, review requested, landed, closed, conflicted), review, comment, issue, release, check, policy. Deliveries retry with backoff up to 24 hours.

13. Checks and OxygenFlow

13.1 Checks

A check is a result attached to a state hash: name, status (queued, running, passed, failed, cancelled, skipped), a summary, a URL, timing, and an optional signed seal from the checker's key over the changes in that state, which makes a passed check a verifiable object. Checks are created through the API (POST /repos/{o}/{n}/checks) by any token with checks:write, so any CI system can report, and required checks for landing (§7.4) are matched by name against the projected state's hash. The proposal page streams check status updates over server-sent events, which reach it from whichever process recorded the check (the event bus, docs/roles.md).

Because a state is content addressed, the hub deduplicates: a check that passed for a state hash is shown for every proposal and bookmark that reaches that same state.

13.2 OxygenFlow runs

Pipelines run in OxygenFlow, a separate project and specification (oxygenflow/docs/spec.md; the first slice is oxygenflow/docs/m0.md). A pipeline is .flow/pipeline.kdl in the state being built; jobs run in Flow's isolation tiers (MicroVM by default, a syscall sandbox, WASM), never in containers on the hub. The hub's side:

• Configuration. [flow] runner_tokens (bearer tokens for runners, compared in constant time), repos (patterns such as owner/* or owner/name: the repositories whose pushes start runs by default; empty means none) job_timeout_minutes (60) and drain_after_errors (5, 0 for never: see Alerts).

• Alerts (oxygenflow#30). A run that ends failed tells its pusher and the repository's watchers, in their inboxes and by email as each chose (flow.failed, from the oxygenflow account, which cannot sign in), naming the jobs that did not pass. A runner whose last drain_after_errors jobs since its state last changed all ended in an infrastructure error (status error: it could not run them, not a failing test) is set to draining and the site administrators are told (flow.runner_drained, with the last error); making it active again on Admin › Runners starts the count afresh (migration 0064, state_ms).

• Priority and turns (oxygenhub#244). Runners take queued jobs by level first (urgent, high, normal, low), then by turn: a job's place in its own repository's queue plus that repository's running jobs, oldest first among equals, so repositories take turns and one repository's burst can't hold up the rest. A run's level is its own if it was expedited, else its repository's (Settings › OxygenFlow priority; maintainers choose low, normal or the default, site administrators also high), else its organisation's (its settings page, by its owners), else the site's (Admin › Runners › Settings), else normal. A maintainer expedites a queued or running run to urgent on its page with a reason; it is audited (flow.expedite), shown on the run and in the queue, and lapses when the run ends. A pipeline may lower its jobs, never raise them: priority "low" (or "normal") on a job or for the whole file (OxygenFlow spec) makes a job's level the lower of its own and its run's, except that an expedited run stays urgent; the queue says pipeline: low (migration 0074). Nothing running is stopped for a more urgent job (migration 0073).

• Per repository. A site administrator turns OxygenFlow on or off for one repository under Settings → OxygenFlow, which overrides repos in either direction; "site default" follows it again. Runs use the site's runners, so repository administrators see the setting but cannot change it. Changes are audited (repo.flow) and apply from the next push, with no restart.

• Triggers. A push that moves a bookmark of an allowed repository starts a run on the bookmark's new state, numbered within its repository from 1 (the run's page is /{owner}/{name}/flow/{number}): the hub reads .flow/pipeline.kdl from that state and validates it with flow-core (the same code flow check runs). An invalid pipeline records a run in error with the problems and their lines; a state without a pipeline records nothing. Tags and proposals start no runs yet, so only code from people with write access ever runs. A run waiting for a runner says where it stands: on the Flow tab its place in line, on its page also the jobs running now and those queued ahead of it (named when their repository is this one or public, otherwise counted), and in the API queue_position. Writers also start runs without moving a bookmark: Run pipeline runs a committed state without a push:

  • On the Flow tab and beside each bookmark on the bookmarks page, it runs a bookmark's current state.
  • On a change page, it runs the state whose only head is that change. That run has no bookmark and is labelled "change".
  • The API does the same with POST /flow/runs and {"bookmark"} or {"change"}.

  Viewers who may not start runs get no button. The Flow tab says why: Flow is off for the repository, they are not signed in, or they lack write access. Re-run on a finished run's page runs its state again.

  Runs only ever take committed states already in the repository, never uncommitted work. A run writes no history: its receipts are signed statements about a state (Oxygen spec §14.6). Running a state again adds another statement. The environment hash shows whether it ran in the same environment, and a matching result is independent confirmation. The new run's checks replace the old ones for that state, and the run page says what started it (pushed by, started by, re-run of #N by). Each log line shows its time since the job began, and a job folds its own output into sections by printing ::group::TITLE and later ::endgroup:: (as GitHub Actions does; a new ::group:: ends the one before): the run page shows each section closed with how long it took, the one still going open, live as lines arrive (oxygenflow#31).

• Runs and jobs. A run holds the repository, bookmark, state hash and heads, pipeline hash and pusher; one job per pipeline job with its compiled form, sandbox and dependencies. Job states: queued, running, passed, failed, skipped (a dependency did not pass), error (the runner could not run it). A run is running while any job is, then passed when every job passed, else failed.

• Newest run wins (oxygenhub#267). A new run cancels the older runs of the same thing still live, so runs do not pile up behind states that have moved on: a push run cancels the older push runs of its bookmark (Diffusion's bumps start push runs), and a proposal's run (bookmark proposal/N, started when its target moves) the older runs of that proposal. Each cancelled job says superseded by run #N, and the next push run's scope reaches back past what was cancelled. A proposal's run leaves its source bookmark's push runs alone: they test the source head, another state, whose checks the bookmark and other proposals may want. Tag, manual, scheduled and re-run runs were each asked for in their own right, so they supersede nothing and nothing supersedes them; they are cancelled by hand. A repository's maintainers choose under Settings › OxygenFlow › Newest run wins whether superseded runs already running are cancelled too (the default) or finish, with only the queued ones cancelled (repositories.flow_keep_running, audited as repo.flow_keep_running).

• Reusing a pass (oxygenhub#265). A push run does not run a job again when the same job, with the same compiled spec and sandbox (the image pin included), already passed on the very same state in this repository, as main does after a proposal lands its tested projection: the job is created passed, its error note reused from run #N, carrying that pass's receipt, hash and attestation, and its check reports passed in run #N on this state, reused. A run whose jobs are all reused has passed when created, and reports as any passing run (Diffusion's green bumps, a bump's landing). Never reused: a job with an approval gate, an environment, secrets or identity tokens, or one whose artifacts another job of the run takes as inputs; manual runs, re-runs, schedules and tags always run. Maintainers turn it off under Settings › Runs (repositories.flow_reuse, audited as repo.flow_reuse).

• Pushing without a run (oxygenhub#284). A push whose newest change (by its clock) has [skip flow], [skip ci] or [ci skip] in its message, in any case and outside code (a marker between backticks or in a fenced block only mentions one), starts no jobs: the hub records a run with status skipped, finished at once, whose note names the marker, the change and who pushed it, so the run list shows the gap. The marker is in a signed message, so who skipped is verifiable. A skipped run is never a later run's baseline: the next push run's scope reaches back past it, so the changes it brought are still tested. A pushed tag always runs. A repository's maintainers choose under Settings › Runs where skipping is allowed: on any bookmark (the default), on any but the default bookmark, or never (repositories.flow_skip: any, others, never; audited as repo.flow_skip).

• Runner API (runner token): POST /api/v1/flow/claim hands out the oldest queued job whose dependencies passed and whose tier the runner offers, marking it running, with a job token; 204 when there is none. POST /api/v1/flow/jobs/{id}/log appends log records (text cut at 16 KiB, at most 50 000 per job); POST /api/v1/flow/jobs/{id}/finish records the exit code and receipt (the hub recomputes the receipt hash) and settles dependents. Only the runner holding the job may report on it.

• Runners (Admin → Runners, site administrators).

  • A runner's page has a Settings form (#98): labels an administrator gives it beside those it reports (a job asking for a label matches either), and a job time limit in minutes below the site's, applied when it claims.
  • Registering:
    • An administrator creates a one-time registration token, valid for a day and shown once with the command that uses it.
    • On the runner, flow runner register https://hub --token … sends the runner's signing identity, architecture, OS, version and labels (POST /api/v1/flow/runners/register) and gets its own credential (flowrunner_…, stored hashed).
    • The runner's key is bound at registration: receipts from that runner signed by another key are not kept.
    • Runners configured with a hub.toml token still work. They are tracked from their first request under the name they claim with, and their key is bound on their first signed receipt.
  • Scopes (oxygenhub#41): a runner serves the whole site, one owner's repositories (a user or an organisation), or one repository. The registration token carries the scope (typed as empty, owner or owner/repository when it is made) and the runner gets it; an administrator can change it on the Runners page. A claim returns only queued jobs of repositories in the runner's scope, filtered before the queue is cut short, so a scoped runner never receives another scope's jobs and never waits behind them. A job whose pipeline says runs-on (OxygenFlow spec 3.1) goes only to a runner that has all of its job's and sandbox's labels, from those the runner reports on the claim or registered with; until one claims it, the queue says which labels it waits for. hub.toml runners serve the whole site. Changes are audited (flow.runner.scope). Settings → Runners on a repository (its admins) and on an organisation (its owners) lists the runners that may take its jobs, with what each serves; there they make a registration token for a runner of their own scope (the repository, or the organisation's repositories) and remove such runners when they run nothing. Only site administrators manage site runners.
  • Each claim carries the runner's description and marks it seen.
  • The Runners page lists each runner: online (seen in the last 90 s), offline, draining or disabled; what it serves; signing key; machine; labels; capacity; jobs running now; and when it was last seen.
  • Actions:
    • Drain: the runner finishes its jobs and claims return nothing.
    • Disable: its requests are refused.
    • Resume.
    • Remove: refused while the runner has running jobs. Its credential stops working, and its jobs keep its name.
  • The queue (on the Runners page): running jobs with the runner holding each, then queued jobs in the order runners claim them.
  • API (site administrators): GET /api/v1/runners (each runner, what it serves, its machine, key, last seen and jobs running), GET /api/v1/runners/{id}/jobs?before=&limit= (its history, newest first) and GET /api/v1/runners/queue.
  • A runner's page shows its totals (jobs, pass rate, time busy) and its job history, newest first, with links to runs and signed receipts. Jobs record the runner that claimed them (flow_jobs.runner_id).

• Signed receipts (Oxygen spec §14.6): finish may carry signed_receipt, the runner's signed receipt as hex. The hub keeps it in the repository when four things hold:

  • it verifies;
  • it signs the job's receipt;
  • it is for the run's state (computed from the run's heads) and this repository;
  • its key is the one the runner's token first signed with. The first signed receipt under a token binds that token to its key in flow_runner_keys.

  Otherwise the job's result still stands, and the run page says why the receipt was not kept. The run page links each kept receipt, with its signer. The change page lists the receipts for the state whose only head is that change. /{owner}/{name}/receipts/{attest} downloads a signed receipt for oxy receipt verify or import.

• Architecture (OxygenFlow spec §11.2–§11.3, oxygenflow#36): a runner reports each image's architecture on its claims (image_archs), kept in flow_images; a job is offered only to a runner whose architecture matches its sandbox's arch, else its image's as known. The hub knows an image's architecture from its own files, read through its ext4 to its shell's ELF header (jet::rootfs) out of the kept chunks, when it is sent (the manifest's word is taken only when the image cannot be read) and, once at start, for every kept image that had none; an image nobody can read takes the architecture of the runner its first job passed on, and a runner's own report, read from the image it holds, wins over that (oxygenhub#173, Andy's "detect and learn"). Only an image nobody has read or run matches any runner. The queue says which architecture a waiting job asks for. Images by hash (oxygenflow#39, Jetstream): flow rootfs push HASH sends a rootfs image as the FastCDC chunks the hub lacks (POST /flow/images/missing, PUT /flow/images/chunks/{hash}, by a token that may write repositories) and then its manifest (PUT /flow/images/{hash}), which the hub accepts only when every chunk is held and they make an image with that hash; chunks are kept zstd-compressed under <data>/images, shared by every image, and jet_images lists what is held. Runners and accounts fetch with GET /flow/images/{hash} and GET /flow/images/chunks/{hash}. A runner whose claims say fetches_images is offered jobs whose image only the hub holds, and fetches it before its preflight. Each claim also carries the runner's facts (OxygenFlow spec §11.1, oxygenflow#40): free disk, load, speed score and class, kernel; the hub keeps the latest on the runner, shows them on Admin › Runners and in GET /runners. A job's finish carries the most disk and memory it took (peak_disk_mib, peak_memory_mib, oxygenflow#41); a new job's need_disk_mib is the most its last five runs of the same repository and name took, plus a quarter and 64 MiB, and it is offered only to a runner whose claim reports at least that much free disk; the waiting reason says so. Ranking (oxygenflow#42): a job that another free runner which polled in the last 10 s would do better (it holds the job's build cache, which counts most: claims list the caches a runner holds, caches, each flow_core::api::cache_slot(repo, job), oxygenflow#51, and when each last kept a disk, cache_times, so of two runners holding it the one whose copy was kept at least a minute later ranks higher, oxygenflow#54, and placed says "holds its build cache (kept 12 min ago)"; or it holds the job's image while the claimant would fetch it; or it has a higher speed class) is held from the claimant for up to 15 s, then any runner that fits takes it; each job records why it ran where it did (placed: "holds its build cache, holds the image, idle, speed 100 (class 3)", with "after N s held for a better runner" when it was), shown on the run page, in the API and by oh run view. A run whose jobs pin an image held by neither the hub nor a runner sets a failing pipeline/images check on its state (#137), naming the images and how to send them (flow rootfs push), so a proposal pinning a new image shows it before it lands; a later run of the state turns it to passed once every image is held. A placement pass every 15 seconds works out why each ready queued job no runner can take waits, from what each online runner offered at its last claim (scope, tier, architecture, image held, labels, memory and CPUs, a recent decline), and shows it on the job (API waiting, oh run view and oh run watch, its panel open on the run page) and in its check's summary, so a proposal says why its check is stuck (#28). A job some runner could take but that still waits says that too (#259): which of those runners are busy ("earth-1 busy (3 of 3)"), or have a free slot without the memory or disk for it now, and how many ready jobs come first (a better priority, or older); its check is reported again only when the reason changes in kind, not in its counts, so a moving queue does not reach webhooks every pass. Diffusion (#148, docs/diffusion.md): a push that moves a repository's default bookmark, or the bookmark its work lands on (the most common target of its proposals landed in the last 90 days), records each Oxygen subrepo pin in the new state (diffusion_pins: consumer, bookmark, path, the upstream's Oxygen repository id, the pinned heads); a hub that had no index fills it at start. GET /repos/{owner}/{name}/consumers lists the consumers the caller may read, each with its pin and how many changes of this repository's default bookmark it lacks (behind, null when the pin names changes this repository does not hold), and the overview shows them under Diffuses to. GET /repos/{owner}/{name}/diffusion (#153) gives both directions, takes_from (each upstream the repository pins) and diffuses_to (each consumer the caller may read), every entry naming its upstream and consumer, its path, bookmark, behind, follow, open bump, the bump's newest run and run_status, and the pill's halves (upstream_state, consumer_state: tone green, amber, red, grey or off, word, title); hub-api's Diffusion. A repository that pins, or is pinned by, another on the hub has a Diffusion tab (#150): Diffuses to (each consumer's pin bookmark, how far behind, its status, its open bump and that bump's newest run, its last landed bump, how it follows) and Takes from (each upstream it pins, with the same); its overview shows both in brief, and Admin › Diffusion lists every upstream with its consumers. Each consumer's status is a two-half pill: the upstream on the left (checked when its newest default-bookmark run passed, untested when it has no checks, failing, or testing) and the consumer on the right (passes or fails for its run on the bump, testing, waiting, or up to date), each half coloured green, amber, red or grey; the API keeps one overall colour as well. Landing a proposal on the hub refreshes the pin index and propagates as a push does. A bump's run, started by the diffusion account, is given the upstream it bumps as a subrepo whatever that account may read (the consumer's own pin of it was committed by someone who may write the consumer); its other subrepos need read access as usual. A consumer follows a subrepo's upstream as its .oxymodules entry says (follow = off | green | any, land = propose | auto, target naming the bookmark to bump, watch listing paths whose upstream changes are flagged), else as its repository setting says (Settings › Diffusion, or PUT /repos/{owner}/{name}/diffusion {"follow": ...} by its owners and admins; migration 0058), else as [diffusion] follow in hub.toml says (default off). A bump moves a consumer's pin at a path to a newer upstream state: one change on the target bookmark's heads that replaces only the pin (oxy-core build_pin_change), signed with the hub's key, on the bookmark diffusion/<path>, which the hub owns and rebuilds on the target each time (a forced update). When the pin has changes the upstream's state lacks (say, from an upstream proposal not landed yet), the bump moves to both, so a consumer never loses changes it relies on; it is skipped when that combination would conflict, when the pin names changes the upstream does not hold, when the pin already holds the state, or when the bump bookmark already moves it there, and the hub logs why. A bump is proposed once per bump bookmark, by the hub's diffusion account (made on first use, never able to sign in), into the target, labelled dependencies, its body the upstream changes the pin lacks (newest first, linking to the upstream's change pages); a later bump edits the same open proposal. It is triggered when a push moves the upstream's default bookmark (for consumers following any) or when a run of that bookmark passes (for green). Diffusion lands a bump only where the consumer chose it (land = auto in the entry, else Settings › Diffusion › Landing or PUT /repos/{owner}/{name}/diffusion {"land": "auto"}, migration 0061; off by default, #174): once both halves are green (the upstream state it pins passed its own run, and the consumer's run on the bump as it is now passed), as the diffusion account through land_as, signed by the hub's key, so required checks, approvals and the trust policy decide; never for an unverified upstream; a refusal is noted once on the proposal's timeline. An upstream with no checks of its own (Flow off, or no run ever of its default bookmark) is unverified: it never turns green, so its moves also bump consumers following green, and those proposals are marked amber with the upstream unverified label and a note in the body; a later bump from a passing run removes the label. A bump already in place refreshes its proposal's text and labels. A consumer's watch patterns (* within a path segment, ** across them, ? one character; a plain path covers everything below it) flag the upstream changes touching a matching file: they are listed first under "Touches what you watch", with the files, and marked ⚑ in the changelog. Turning Diffusion on for a consumer (the setting or the API) catches it up at once: it is bumped to each upstream's newest state whose default-bookmark run passed, or to its newest state when it follows every move. Images page (#146): Admin › Runners › Images lists every rootfs image a runner holds, the hub keeps, or a job ran on in the last 90 days (from each job's sandbox), newest use first: its size and architecture, where it is (the hub, each runner holding it, each runner fetching it from the swarm now), the repositories whose jobs used it, and who sent it; an image held nowhere is marked. The swarm (#144, oxygenflow docs/jetstream.md §8.1): a runner in an image swarm reports its peer address, its site and the BLAKE3 of its TLS certificate with each claim (swarm; without a certificate it is not in the swarm), kept in memory and shown on Admin › Runners. POST /flow/swarm/announce {image, state} (by a runner; fetching, done or stopped) returns the online runners in its site that hold the image or announced fetching it in the last 45 seconds, each with its certificate's fingerprint, which the fetcher pins its TLS connection to, and a ticket signed with the hub key (JSON naming the runner, its site and an expiry 15 minutes on, hex, then the signature) that peers check against /.well-known/oxyhub-key before serving chunks; a runner that has not reported an address is told to add [runner.swarm] (409).

• Variables (oxygenhub#102): Administration, an organisation's settings and a repository's settings each hold NAME=value pairs (a name is a letter or _ then letters, digits and _, up to 64, never FLOW_*; a value up to 4096 characters; up to 100 per level) under Variables, each page showing what it inherits and which inherited names it overrides. A claim carries the repository's effective set as env, inner levels overriding outer; Flow puts them in the job's environment under FLOW_* and the pipeline's own env, so they are part of the receipt. They are not secrets and are shown in full to whoever sees the page; secrets are mounted by Flow (OxygenFlow spec §9), never put in the environment. Changes are audited (flow.variable.set, flow.variable.deleted); flow run --var NAME=VALUE gives a local run the same.

• Secrets (oxygenhub#107; OxygenFlow spec §9): the same three levels hold Secrets, named as a pipeline's secret "name" { mount-path "/run/secrets/…" } names them (1 to 64 of A-Z a-z 0-9 . _ -, a value up to 64 KiB, up to 100 per level), typed once and sealed under keys/secrets.key; pages show names, when each was set and where it comes from, never a value. A claim carries only the secrets its job declares, each with its path and value, from the innermost level that sets the name; one no level sets is noted at the top of the job's log and left out, so the job fails on the missing file rather than running with a stand-in. Flow writes each as a file (0400, the job's user's, on an in-memory mount) before the job starts and never puts it in the environment, the receipt or an artifact. Log intake replaces each value, and its base64, with *** before a record is stored. Changes are audited (flow.secret.set, flow.secret.deleted), never with the value.

• Provenance others verify (oxygenhub#239). For an artifact of a job whose signed receipt the hub kept, …/flow/{run}/artifacts/{job}/{artifact}/provenance answers an in-toto Statement v1 with an SLSA provenance v1 predicate, in a DSSE envelope signed with the hub's Ed25519 key. Its buildType is https://oxygenhub.io/oxygenflow/job/v1 on every hub, as oxy receipt export --format slsa writes it (Oxygen spec §14.6). The subject is the artifact by SHA-256 (and BLAKE3); the build definition names the repository, bookmark, job and pipeline hash, and depends on the state built; the run details name the runner as builder, the run as invocation with its start and finish, and the kept receipt's attestation as a byproduct. The runner signed the receipt; the hub, which checked it, states what it says in the shape cosign, policy-controller, Kyverno and slsa-verifier read. /.well-known/oxyhub-signing.pem is the key: cosign verify-blob-attestation --key oxyhub-signing.pem --type slsaprovenance1 --insecure-ignore-tlog=true --signature tool.intoto.json tool. (--insecure-ignore-tlog: the envelopes are not in Sigstore's Rekor; the hub's own transparency log, docs/transparency.md, takes that role. Checked with cosign 2.4.1 on 2026-10-17: a release asset verifies, a copy with one byte changed does not.)

• Artifacts (OxygenFlow spec §5.2): a job may declare artifacts with output.

  • The runner holding the job uploads each one with POST /api/v1/flow/jobs/{id}/artifacts/{name} (raw bytes, at most 512 MiB).
  • The hub refuses names the job does not declare. It keeps each artifact as a blob of the run's repository, records its path, size and BLAKE3 in flow_artifacts, and answers with the blob's ObjectRef.
  • finish refuses a receipt whose outputs are not exactly the uploaded blobs, in name order.
  • The run page lists each job's artifacts, and /{owner}/{name}/flow/{n}/artifacts/{job}/{name} downloads one.
  • A kept receipt keeps its artifacts through garbage collection (Oxygen spec §14.6).

• Job tokens. Random, stored as a hash, valid only for reading the job's repository over the wire protocol while the job is running and for at most two hours. They cannot push, call the API or read other repositories.

• Checks. Each job is also the check flow/<job> on the run's state (§13.1), so proposals, required checks and the editor's status bar see it.

• Pages. A Flow tab lists runs (status, bookmark, state, pusher, when, duration). A run page draws the job graph as SVG, with columns by dependency depth and status by colour and glyph, and each job's log below (folded, the failed ones open); it refreshes while the run is live. Readers of the repository can see both.

/{owner}/{name}/flow/badge.svg is a small SVG of the latest run on the default bookmark (passing, failing, running, no runs) for a README, for whoever may read the repository. It is a convenience, with the weakness of every badge; the stamps below are the real thing.

State stamps replace CI badges, which are mutable images claiming that a branch passes, name no state and cannot be checked. A stamp belongs to a state:

• /{owner}/{name}/flow/stamp/{ref} is an SVG. For a state address (heads joined with +) or a change, it shows the newest run of exactly that state: one glyph per job in dependency order, the run's status, the state's short hash and the run's date. It shows "no run" when there is none. The same data is machine-readable in the SVG's <metadata>: heads, run number, status, and each job's status and receipt hash. For a bookmark or tag, it redirects to the stamp of the ref's current state, so even the image names what was checked. A finished run's stamp may be cached for an hour; any other stamp for a minute.
• In rendered Markdown, the relative paths .flow/stamp (as an image) and .flow/run (as a link) resolve against the state being viewed. The first is served by /raw/{ref}/.flow/stamp, the second by /file/{ref}/.flow/run (the run page, or the Flow tab when that state has no run). So [![flow](.flow/stamp)](.flow/run) in a README shows how the viewed state did: browsing a tag shows the tag's result, not the bookmark's. A real file at either path takes precedence.
• Later, stamps also say which runner key signed each receipt, and which jobs reused an earlier receipt.

The older runner draft (docs/runner.md) is superseded by OxygenFlow; its hub-side ideas (environments, approvals for outsider proposals, caches, artifacts) carry over as later Flow milestones.

13.3 Web workspaces (planned)

Open in Workspace gives a signed-in user VS Code in the browser on a state of a repository, with Oxygen VCS inside, without cloning anything to their own machine. It is designed here before it is built (Oxygen backlog P5); parts are marked as they land.

• What runs. openvscode-server (MIT, unmodified) with the Oxygen VCS extension and oxy, in a Firecracker microVM per workspace on an OxygenFlow runner (a workspace job kind beside pipeline jobs), never on the hub's host. The image is a Flow rootfs like the build images, pinned by hash. A fork of VS Code is considered only if an extension cannot do what is needed.
• Starting. The button is on a bookmark's row and on a change page, for accounts that may read the repository. The hub records a workspace (account, repository, state, bookmark when opened from one, status) and queues a workspace job that only a runner whose scope covers the repository may claim (§13.2 scopes). One account has at most a few workspaces at once; the hub says which are open.
• Files without a checkout. The state is mounted read-only in the microVM by flow-casfs (OxygenFlow spec §5.1): the tree from the state index, each file's bytes from a cache of renderings keyed by graph key, rendered on first read and shared by every state in which that file has the same history. An overlay on the job's ephemeral disk holds edits. Opening a workspace on the Linux tree therefore costs listing a tree, not writing 96 000 files; the mount is to take under 100 ms.
• History and commits. The microVM's oxy repository fetches the state's history over the wire protocol with a token that reads this repository only, as job tokens do, and its working copy is the mount. Private keys never enter the hub or the microVM, so a commit is pushed to a workspace bookmark (ws/{account}/{id}) and signed there by the hub as the account's delegate, exactly as web edits are (refused with instructions until the account has delegated). From the bookmark the user proposes or lands as usual.
• Access. The hub proxies the editor (HTTPS and WebSocket) at /{owner}/{name}/workspace/{id}/ to the workspace's owner only, checking the session on every request; the runner reaches the hub over a connection it opens itself, so runners behind NAT work, and accepts editor traffic for a workspace only from the hub with that workspace's secret. Nothing in a workspace is reachable from any other account, or from the network directly.
• Lifecycle. A workspace stops after 30 minutes without a connection (configurable) and on request. Its overlay is discarded when it stops: anything not pushed to its workspace bookmark is gone, and the editor says so before stopping. A stopped workspace reopens from its bookmark. The audit log records starts, stops and pushes.
• Measured when built: mount time, microVM boot, time to the editor's first paint, and a full read of the Linux tree through the mount.

Design notes (2026-10-04), for review before anything is built.

• Data. A workspaces table: id, account, repository, state (ObjectRef), bookmark (nullable), workspace bookmark name (ws/{account}/{id}), status (queued, starting, open, stopping, stopped, failed), runner, job id, secret hash, created, last connection, stopped, and why. A workspace is a row plus one Flow job of kind workspace; the job's log is the editor's server log. GET /api/v1/user/workspaces lists the caller's; POST /repos/{o}/{n}/workspaces {"state"|"bookmark"} opens one; DELETE /workspaces/{id} stops it. Limits are per account: three open at once (configurable), and a site cap.
• Runner side. The runner gains a job kind beside pipeline jobs: it boots the pinned workspace rootfs (openvscode-server, oxy, the Oxygen VCS extension, a non-root user), mounts the state at /workspace through flow-casfs with a writable overlay, runs oxy in the microVM against a repository that fetches history lazily over the wire protocol with the job's read-only token, and opens a tunnel back to the hub (the same outbound connection the runner uses for claims) carrying the editor's HTTP and WebSocket traffic. The runner marks the job running when the editor answers on its port, and reports a heartbeat (connections open, last activity) every 30 s in place of log batches; the hub stops the workspace when the heartbeat shows no connection for the idle limit.
• Hub side. /{owner}/{name}/workspace/{id}/ is a reverse proxy to that tunnel, for the owner's session only, with the workspace secret in a header the runner checks; CSRF does not apply (the editor is same-origin under the hub, and the proxy forwards only to the workspace the path names). Commits: the extension's oxy commit writes to the microVM's repository; oxy push goes to the workspace bookmark with the job token, and the hub signs the pushed changes as the account's delegate (web edits' path) if the account has delegated to the hub key with changes,refs; otherwise the push is refused with the recipe. The bookmark is ordinary afterwards: propose, land, or delete.
• Security. The microVM has no credentials but the job token (read this repository, push this workspace bookmark) and the workspace secret; the account's keys never enter it. Network egress from the microVM is off by default (as for pipeline jobs); a site setting may allow it for package installs, with the audit log recording the choice. The overlay is ephemeral and the rootfs is pinned by hash, so a workspace cannot be left altered.
• Costs. One microVM per open workspace: 2 vCPU and 4 GB by default, from the runner's capacity like any job, so open workspaces compete with pipeline jobs; a runner label workspace keeps them on runners meant for it. Idle stop at 30 minutes keeps the fleet small.
• Open questions. Whether the Oxygen VCS extension should talk to the microVM's oxy (simplest; works offline in the VM) or to the hub's API (no oxy in the VM, but a second code path for every operation): the first. Whether a stopped workspace's overlay should be kept for a day so an unpushed edit is not lost: probably yes, as a tarball artifact of the job, restored on reopen. Whether to allow terminals: yes, it is the point.

13.4 AI assistants (done 2026-10-04)

An assistant is a chat-completion endpoint the hub may ask to review a proposal or draft its description (oxygenhub#97). Assistants are configured at three levels and combined downwards:

• Where. Administration (site), an organisation's settings, a repository's settings each hold a list of assistants: name (Claude, Antigravity, ChatGPT, DeepSeek, …), endpoint URL, API shape (chat-completions or anthropic-messages), model, extra headers, the token, and an enabled switch. Each level also holds a prompt. A repository sees the assistants of its organisation and of the site beside its own; an assistant disabled at an outer level is off for everything inside it, whatever an inner level says. The effective prompt for a call is the site's, then the organisation's, then the repository's, each under a heading, so a project adds to what the site says rather than replacing it, and each settings page shows the prompt it inherits above its own.
• Adding one (oxygenhub#109). The form offers providers (Anthropic, OpenAI, Google Gemini, DeepSeek, Mistral, Perplexity, Ollama) that fill the name, endpoint and API shape, and Other for an endpoint typed by hand. List models asks the endpoint's own list (…/models beside …/chat/completions or …/messages, with the token typed or, when editing, the one stored) and offers the chat models in the Model field; on save, a model the endpoint lists none of is refused with some it does offer. An endpoint without a list takes any name.
• Secrets. Tokens are sealed at rest with ChaCha20-Poly1305 under keys/assistants.key (mode 0600, made at first start beside the cookie key; each token carries its own random nonce), shown only as set or none, and never returned by the API. An empty token suits endpoints that need none. Each settings panel has TEST, a one-line exchange whose answer or error is shown. Each level's changes are in the audit log (assistant.added, .updated, .enabled, .disabled, .deleted, .prompt).
• Request review. Under Reviewers on the proposal page, the author or a writer sees an ask NAME button per assistant the repository may use. Asking records the event (asked NAME to review) and the audit entry (assistant.asked), then a background task builds the context (title, description, the list of changes, the diff of each changed file as -/+ lines per hunk, binary files noted, the whole cut to the assistant's context budget less the prompt), prefixes a fixed system prompt (who it is, that it answers for the person who asked, that it cannot approve, the shape of a review) with the levels' prompts, calls the endpoint, and records the answer as a review of verdict comment whose author is the person who asked and whose via (migration 0036) is the assistant, shown as "bob ↳ agent Claude reviewed" in the timeline, exactly as agent-signed changes are shown (§6). An assistant never approves: approvals count only people, and an assistant's review never replaces the asker's own latest verdict. The proposal's author is told as for any review; the review webhook carries via. No answer leaves an event (NAME did not answer: why) and no review.
• Draft the description. The new-proposal form (anyone who may open one) and an open proposal's editor (the author or a maintainer) have a draft with NAME button per assistant. It asks the assistant, now, for a description from the title so far, the changes (titles and the first lines of their messages) and the diff, under a fixed system prompt (description field only, what and why, what to look at, how to verify) plus the levels' prompts, and puts the answer in the editor with a note; nothing is saved until the person updates or opens the proposal. The audit log records assistant.drafted; the same per-repository hourly limit applies. With JavaScript the button posts to …/draft/stream instead, which answers at once with server-sent events (part with each piece of text as the assistant writes it, then done with the whole draft or error with why there is none, each a JSON string, and a keep-alive every 15 s), and the editor fills as the draft comes; without it the form posts and the page returns with the draft. Every assistant call is made as a stream, and its time limit is for silence (90 s for a draft, 300 s for a review) under a 15-minute cap, so a long answer that keeps coming is not cut off (oxygenhub#248).
• Limits. One request per assistant per proposal at a time, 30 assistant calls per repository per hour (in memory), a 60-second call timeout, answers kept to 64 KiB, and a context budget (characters) per assistant.
• Too much for the model (oxygenhub#248). The context lists every changed file with its +/− counts before the diff, then gives whole files in order while they fit: one that doesn't is cut where the room ends (marked) when a fair amount is left, else left out, and every file cut or left out is named, so the model knows what it did not see. A provider that refuses the prompt as too long for its model (400/413/422 saying so) is asked once more with a third of the budget; if that fails too, the event says the proposal was too large at both sizes and to lower the assistant's budget. A timeout says the same about slow or local models. A local Ollama cuts a prompt longer than its context window without saying so: set its context (OLLAMA_CONTEXT_LENGTH, or the model's num_ctx) to fit the assistant's budget, about four characters a token.
• API. POST /repos/{o}/{n}/proposals/{n}/review-requests {"assistant": NAME} (the author or a writer; 202 when started, 409 when that assistant is already answering or the repository is over its hourly limit, 422 for a name the repository cannot use); reviews carry via. Settings: GET/POST /repos/{o}/{n}/assistants (repo:admin; POST adds or changes by name, with preset, an optional write-only token that an empty value keeps, and the model checked against the endpoint's list), DELETE …/assistants/{name}, and GET/PUT …/assistants/prompt (the level's text and what it inherits); the same under /orgs/{org}/assistants for its owners and /admin/assistants for site administrators. oh assistant list|add|remove|prompt uses them, with --org or --site, and reads a token only from an environment variable (--token-env). Hub actions by an assistant are attributed "ASSISTANT via WHO" everywhere, so a project can require that a human has approved regardless of what an assistant said.

13.5 A job's identity: the hub as an OIDC issuer (done 2026-10-04)

A job that declares identity "AUDIENCE" { mount-path "/run/secrets/…" } (OxygenFlow m0 §3) gets a signed JWT so a cloud, a vault or any service that trusts this hub can let it in without a long-lived secret (oxygenhub#103).

• Issuer. The hub's base_url. GET /.well-known/openid-configuration publishes the discovery document (issuer, jwks_uri, RS256, the claims below) and GET /.well-known/jwks.json the public key, both cacheable for an hour. The key is RSA 2048 in keys/oidc.pem (PKCS#8, mode 0600), made the first time it is needed; its kid is the first 16 hex of BLAKE3 over the modulus. Rotation is by replacing the file and restarting; tokens signed by the old key then fail, which a job limit of hours makes harmless.
• Token. Minted at claim time, one per identity, and delivered through the same channel as secrets: the claim's secrets carry it as identity:AUDIENCE at the mount path, where Flow writes it 0400 on the in-memory mount. Header {"alg":"RS256","typ":"JWT","kid"}; claims iss, sub = repo:OWNER/NAME:bookmark:BOOKMARK (what a trust policy matches), aud, iat, nbf (30 s of skew), exp = the job's time limit from now, at most six hours, jti, and repository, repository_owner, repository_visibility, bookmark, run_number, job, state (the state hash), runner, actor (who started the run). A local flow run has no issuer and says the file will be missing.
• Masking. Log intake replaces any token this hub signed (recognised by its kid) with ***, as it does secret values.
• Trust, for example. AWS: an IAM OIDC provider with the hub's URL and audience sts.amazonaws.com, a role whose trust policy matches sub repo:andymccall/metalgear:bookmark:main; the job then calls aws sts assume-role-with-web-identity --web-identity-token file:///run/secrets/aws-token. Google: a workload identity pool provider with the same issuer, mapping assertion.sub. HashiCorp Vault: the JWT auth method with bound_audiences and bound_claims on repository.

13.5.1 Backups (oxygenhub#169, built 2026-10-13/14)

Each user and organisation backs up its repositories hourly to a place it controls: a GCS bucket reached keylessly (the hub mints a short-lived token with sub = backup:OWNER, which GCP's workload identity federation exchanges for an access token, so no key exists), or a directory the site allows. A run sends only the content-addressed objects the destination lacks, the refs at that moment and the hub's records (issues, proposals, releases, wiki, settings; never secret values), and is whole once its DONE marker is written; oxygenhub admin restore brings repositories back from any whole run. Built: Settings › Backups and an organisation's Settings › Backups (how often: off, hourly or daily; a directory from hub.toml [backups] directories or a GCS bucket with its workload identity provider; a prefix; runs kept 7, 30 or 90 days; "back up now"; the last 50 runs with their outcome), migration 0060 (backup_settings, backup_runs), the hub's scheduler (one run at a time; a run left running by a restart is marked failed), each repository's and wiki's store copied file by file under the repository's read lock for the final pass, the records as JSON (every table keyed by the repository, and three levels of rows pointing at them; tables and columns holding secrets, tokens, sessions or passwords never), pruning of old runs and unreferenced blobs, an inbox notice to the owner (an organisation's owners) once per spell of failures, and oxygenhub admin backup run OWNER and oxygenhub admin backup restore --from DIR [--run AT], which restores the stores and the repositories' rows. A GCS destination is keyless: the hub mints a 10-minute token (Issuer::mint_for, sub = backup:OWNER, aud the provider's name over https), Google's STS exchanges it ([backups] sts_url) for an access token scoped devstorage.read_write, used with the Cloud Storage JSON API ([backups] storage_url) and exchanged again before it expires; TEST writes, reads and removes a probe. The design, the layout at a destination, setup and restore are in docs/backups.md.

13.5.2 Certificates for short-lived keys (oxygenhub#241, docs/keyless.md, steps 2 to 5 built 2026-10-17)

POST /api/v1/keyless/certify gives a fresh key a certificate (Oxygen spec §14.7) for ten minutes, signed with the hub's key. The bearer is the credential: a running job's token (subject repo:OWNER/NAME:bookmark:B, claims repository, bookmark, run, job, runner, and environment with approved_by once an approved job may deploy), or one of this hub's OIDC tokens for the audience {base_url}/keyless (its sub the subject, and the same claims it carries). The body is {"identity": base64 identity object, "proof": base64 CBOR Signature, "scope": "changes,refs"}, the proof the fresh key's challenge signature (§14.3) over certify_challenge(credential) ("oxy1 certify " and the BLAKE3 of the credential), so it cannot be replayed with another credential. The answer is {"certificate": base64 [certificate, signature], "subject", "expires_ms"}; a certificate never outlives its credential, and it starts 30 s back for clocks a little behind. Each one is audited (keyless.certified). A runner may sign a job's receipt with such a key, certified just before it signs (scope seal): the receipt carries the certificate (Oxygen spec §14.6), and the hub keeps it when the certificate is this hub's, its subject is the run's repository and bookmark, its run and job claims are the run's and the job's, and it covers the moment the hub accepts it; no runner key is bound then, and the run names the signer as the runner's name with "(short-lived key)". Which issuers and subjects a repository trusts for changes and references is its policy's (docs/keyless.md, step 4). When a key it certified moves a bookmark, the hub signs an accepted attestation of the update with its own clock and serves it with the bookmark (Oxygen spec §14.7, step 4c), so fetches check the certificate at the hub's time. The scope given is what was asked within what the credential allows; asking for none of it is refused.

Agent grants (step 5, built 2026-10-17). An agent holds no key: its owner signs an Oxygen grant (§14.7) letting agent:NAME@ACCOUNT act for them, with the key their oxy home names (oh agent grant NAME [--scope changes,seal,refs] [--for 30d]), and POST /api/v1/agents/grants {"grant": base64 signed grant} keeps it and answers with the agent's token (oxyagent_…, shown once, kept as its BLAKE3 hash). The hub accepts a grant only for the caller's own agent:NAME@USERNAME, naming this hub's key as issuer, signed by a key linked to the caller's account and not yet ended; any personal token of the account may send it, since the signature is what authorises it. The agent exchanges the token at keyless/certify for ten-minute certificates carrying the grant (claims agent and grant, scope at most the grant's, never past its end), so what its fresh keys sign counts in the owner's name; oxy does this itself with author.signer = keyless and the token from oxy auth grant. The token is also the agent's credential for the protocol (fetch and push) and the API: it acts as an agent's token of the owner's account (repo:read and user:read, and repo:write when the grant gives refs), so the agent's home holds no other secret, and oxy and oh offer it when the home has no hub token. It cannot list, make or revoke grants (#253). GET /api/v1/agents/grants lists the caller's grants without tokens; DELETE /api/v1/agents/grants/{id} revokes one at once (oh agent list, oh agent revoke ID). Migration 0075 (agent_grants); audited as agent.granted and agent.revoked.

13.5.3 The transparency log (oxygenhub#238, docs/transparency.md, steps 1-3 built 2026-10-18)

The hub logs every signed object it accepts, in the order it accepts them, as entries of one Merkle tree (Oxygen spec §14.8):

• What it logs:

  • from pushes: reference and tag attestations, seals, the repository's policy, the hub's own acceptances, and the certificates of short-lived signers;
  • receipts as they are kept, and sealed approvals;
  • every certificate keyless/certify issues.

  An object is logged once (tlog_entries, migration 0076, maps objects to entry numbers), and a push is logged before it is answered.

• Storage: the log is append-only files in <data>/tlog/: entries (each a u16 length and its bytes), offsets, hashes (every tree hash in stored order) and checkpoint. A torn append is cut back to the last whole entry on start and its hashes rebuilt; nothing is ever rewritten.

• Checkpoints: every 5 seconds while entries arrive, the hub signs a checkpoint with its key. The origin is the base URL's host and /log, and the note carries an Ed25519 line and a composite line.

• Serving: it serves GET /log/checkpoint (no-cache), GET /log/tile/<L>/<N>[.p/<W>] and GET /log/tile/entries/<N>[.p/<W>] (C2SP tlog-tiles: full tiles cached for a year as immutable, partial ones for 5 s; a tile the log does not hold yet is 404), and /log, a page with the checkpoint and the newest entries (a private repository's entries show only their hashes).

• Refs answers carry the latest checkpoint and, for each bookmark's attestation, its entry and inclusion proof (Oxygen spec §16; step A3, 2026-10-18). An attestation served but never logged (pushed before the log began) is logged then, and proved on a later fetch. oxy pins the log's key per hub, requires each checkpoint to extend the last it saw (proved from the tiles), and fails a fetch whose hub showed two histories.

• Witnesses (step A5, 2026-10-18): a hub can witness other hubs' logs and have its own witnessed (C2SP tlog-witness, Oxygen spec §14.8).

  • As a witness it answers POST /log/witness/add-checkpoint: it cosigns, with its key and clock under its host's name, a checkpoint that verifies with the log's key and extends the last it cosigned (tlog_witnessed, migration 0077). It answers 409 with its size when the log guessed it wrong, 422 to a fork, 403 to another key's note and 404 to a log it was not told to witness.

  • As a log it sends each new checkpoint to its witnesses, with the consistency proof from what each last cosigned (tlog_cosigned), and serves their cosignatures with the checkpoint.

  • Both are set in hub.toml; the key is the other hub's /.well-known/oxyhub-key in base64:

    [tlog]
    witnesses = [{ name = "gcp.oxygenhub.io", url = "https://gcp.oxygenhub.io/log/witness/add-checkpoint", key = "…" }]
    witness_for = [{ origin = "oxygenhub.io/log", key = "…" }]
    

• Not yet: proofs in push responses, the oxy witness daemon, and trusted time (docs/transparency.md). Backups do not copy the log yet.

13.5.4 Terraform state (oxygenhub#252, built 2026-10-18)

The hub keeps Terraform and OpenTofu state per repository, so a pipeline needs no bucket or cloud credential just to hold it. It speaks their HTTP backend at /api/v1/repos/OWNER/NAME/terraform/state/STATE (STATE: letters, digits, -, _, ., up to 100):

• GET the state (204 when there is none);
• POST a new one: JSON, up to 32 MiB, with ?ID= the lock's ID while it is locked;
• DELETE it;
• LOCK with the lock's JSON: 423 and the holder's lock when someone else holds it, 200 to its holder again;
• UNLOCK with the lock's JSON or ?ID=: 409 for another lock's.

Who may use it:

• a job, with an identity token of this hub's for the audience {base_url}/terraform (its pipeline's identity block) or with its runner's job token, for its own repository only;
• a person, with a token of theirs with repo:write and the write role on the repository, to read a state as much as to change one: states hold secrets, and every account reads a public or internal repository (oxygenhub#256). The Terraform state page lists the states only to people with the write role.

The token goes as the bearer or as Basic auth's password, which is what the backend sends (TF_HTTP_PASSWORD).

Storage: each state is sealed at rest with the hub's secrets key, and the last 20 versions before it are kept (migration 0079: terraform_states, terraform_state_versions). Backups carry the sealed bodies, which open only with this hub's key. A repository's Artifacts › Terraform state page (/{owner}/{name}/terraform) lists its states, serials, writers, locks and versions, with the backend block to use.

Versions (oxygenhub#255): each state's kept versions are listed on that page (serial, writer, when), and the current state and each version download as .tfstate files, for people with the write role. A person with the maintain role restores a version: it is written again with a serial above the current one (OpenTofu and Terraform only take a state forward), and the state it replaces is kept as a version, so a restore never loses anything and going forward again is restoring the newer version. A restore is refused while the state is locked, asks for confirmation, and is audited (terraform.restored). The API: GET …/terraform/state/STATE/versions (newest first: id, serial, written_by, written_ms), GET …/versions/ID (the state) with the write role, and POST …/versions/ID/restore (the new serial) with the maintain role, never by a job. Not for the hub's own infrastructure (oxygenhub-gcp), whose state cannot live on the hub it deploys.

13.5.5 Metrics (oxygenhub#212, first slice 2026-10-18)

GET /metrics serves the hub's numbers in the Prometheus text format, only to a bearer of [server] metrics_token in hub.toml. While the token is unset the route answers 404. It reports:

• the build (oxyhub_build_info), uptime and resident memory;
• the state cache's states, estimated weight, budget and summaries;
• requests to bucket stores by kind (with pack listings sent and kept apart), and the time spent in them;
• OxygenFlow jobs queued, waiting for approval and running; runs finished in the last 24 hours by result; runners online and not, and seconds since each last called;
• the transparency log's entries;
• the database pool's connections (idle and in use);
• reference updates pushed in the last 24 hours, accepted and refused, and the bytes the protocol received and sent by endpoint (push carries pushes, fetch and objects fetches), counted before compression;
• each owner's latest backup: seconds since it started, and whether it succeeded;
• mail sent and failed since the hub started;
• HTTP requests by route class (page:<tab>, api:<area>, protocol:<endpoint>, static and so on, never a user's or repository's name; at most 300 classes), method and status class, with latency histograms (5 ms to 30 s).

Diffusion, notifications and query latency are the next slices; the runner's own metrics too.

13.5.6 Insights (oxygenhub#257)

The standard OxygenFlow charts, for people who will not build their own in Grafana: one page at three scopes, so the numbers always agree, the site (/admin/insights, administrators, filtered by repository, owner or runner), an owner (/insights/{owner}: the owner, the organisation's members, administrators) and a repository (/{owner}/{name}/insights, in its bar's More menu and linked from its Flow tab). ?range= is 1h, 24h (the default) or 7d. It shows:

• headline numbers: time to pickup (median, 95th percentile), jobs finished and the share that passed, the median job duration, the jobs queued and running now, runners online and their slots in use;
• charts: time to pickup (median and 95th percentile by the job's start), jobs finished per bucket by result, queued and running jobs over time, job duration, and runner utilisation (running jobs over the slots of the runners online now);
• tables: each runner (status, busy slots, jobs, failures, median pickup and duration, last heard from) and, for the site and an owner, the busiest repositories.

Pickup is from when a job could run to when a runner took it. A job is queued when it is made or run again (queued_ms) and ready at the latest of that, its dependencies passing and its approval (ready_ms, set at its claim; migration 0080, which estimates earlier jobs from their run's start and the earlier stages' ends). A job a runner hands back keeps its times: the person still waits. The charts are inline SVG drawn by the hub (no script or library; coloured by theme classes). The page refreshes its numbers every 30 seconds while visible (data-refresh), and is a snapshot without JavaScript. Wall view (?wall=1, opened in a new window from the page): no menus, larger numbers, for a second monitor. The runners page's queue links its filters to the site's Insights (Charts ↗). [server] grafana_url, when set, adds an open in Grafana link for one's own charts from /metrics.

13.6 Coverage reports (oxygenhub#113, done 2026-10-06)

Coverage is built in, as Coveralls or Codecov would be, fed by OxygenFlow.

• Reports. A job's artifact named coverage in LCOV format (cargo llvm-cov --lcov, Istanbul's lcov reporter, coverage lcov, gcov with lcov) or Cobertura XML (coverage xml, cargo llvm-cov --cobertura, Coverlet, JaCoCo through cover2cover; a report starting with <) is read when the job uploads it. In Cobertura a filename is tried against each <source>, a branch line's condition-coverage counts its branches, and a <method> is a function, hit when any of its lines ran. The hub writes what it found into the job's log (coverage: 88.6% of lines (951 of 1073) in 5 files), or why it could not read it; a bad report never fails the upload. At most 64 MiB is read.
• Counting. Paths are taken relative to the job's /src; a path outside the repository, or a file the run's state does not hold, is left out and counted. Records for one file are added together. A file with a single record keeps the tool's own LF/LH totals, since llvm-cov declares lines its DA lines leave out; the totals then match what the tool printed.
• Stored. coverage_reports keeps each report's totals (lines, functions, branches) with its run, job, bookmark and state; coverage_files each file's counts. The per-line counts stay in the artifact's blob and are read again for the file view.
• Pages. A Coverage tab, shown once a repository has a report: the bookmark's latest percentage with its change against the report before over a chart of up to 200 runs (each point names its run and opens its report), a strip of facts (lines covered, hits per line, functions, branches, files, the first report, the badge with its Markdown), this run's changes with the files whose coverage moved most, files by coverage band, a chart of covered and missed lines, the recent runs, a bookmark picker, every file with a bar (sorted by lines missed, lowest coverage, or path), and a file view with each line run or missed and how often it ran. /OWNER/NAME/coverage/badge.svg is a badge of the default bookmark's latest coverage. The run page shows each job's coverage beside its artifacts.
• The check. Each report sets a coverage check on its run's state, measured against the default bookmark: its previous report there for a run on it, its latest for a run on any other bookmark (a proposal's). Settings › Coverage can set a minimum percentage and a largest drop in points; broken, the check fails and says why (50.0%: below the minimum of 60%), and naming coverage among the required checks holds a proposal back. Without rules it only reports. A bookmark's first report shows the files that moved against the default bookmark.
• Oxygen itself: its test job runs under cargo-llvm-cov in the build image (oxygenflow 7705d699) and keeps its report as the coverage artifact; the README carries the badge.
• API: GET /repos/{o}/{n}/coverage and /coverage/{id} (§12.1).

13.7 Test reports (oxygenhub#114, done 2026-10-06)

• Reports. A job's artifact named junit in JUnit XML (pytest --junitxml, cargo-nextest, Jest's junit reporter, gotestsum) is read when the job uploads it, at most 64 MiB and 50 000 cases. Every <testcase> counts, wherever its <testsuite> nests; a case is passed, failed (<failure>), error (<error>) or skipped, and a nextest case that failed and then passed on a retry (flakyFailure) is a pass. Counts come from the cases, not the suites' attributes. The job's log says what was found (tests: 630 tests: 2 failed in 3 min 4 s), or why it could not be read; a bad report never fails the upload.
• Failing jobs. Declare the artifact with when "always" (OxygenFlow spec §5.2) so a failing suite still sends its report; it is then outside the job's receipt, and the run page says so.
• Stored. test_reports keeps the counts and summed test time per report with its run, job, bookmark and state; test_cases every case with its status and time, and for a failure or error its message (2 KiB) and its text with the case's output (16 KiB).
• Run page. Beside each job its counts, linked to a block above its log: passed, failed, errored and skipped, then the first 50 failures, failures first in report order, each collapsible to its message and output.
• Tests tab. Shown once a repository has a report: the bookmark's latest pass rate (or how many fail) over a chart of up to 200 runs, the counts and test time against the run before, the tests newly failing and fixed against that job's previous report, tests that both failed and passed in the last 30 reports (flaky), the run's failures, a passed-and-failed chart, the slowest tests and recent runs. ?report= shows any report, ?bookmark= another bookmark.
• API: GET /repos/{o}/{n}/tests and /tests/{id} (§12.1).
• A test's history. /tests/case?classname=&name= shows one test across the bookmark's last 200 runs: its result, time (charted) and message in each; the Tests tab links each failure, slow test and flaky one to it.
• The check. Each report sets a tests check on its run's state: 630 passed, or what fails and which of those passed in the same job's report on the default bookmark (its previous one for a run on it, its latest for a run elsewhere), as 2 failing; 1 new against main: tests.test_io::test_write. A proposal shows it with its other checks, so it names what the proposal broke.

14. Git interoperability

A repository may enable a Git mirror: on every landing on selected bookmarks the hub runs oxy git export into <data>/mirrors/<owner>/<name>.git and serves it read-only over Git's smart HTTP protocol at /{owner}/{name}.git (implemented with the git binary's http-backend behind the hub's authorization, since reimplementing pack negotiation is not the product). This lets GitHub Actions runners, IDEs and tooling that only speak Git read hosted Oxygen repositories. Pushing to a mirror is refused with a message pointing at oxy.

Import from Git: oxy git import at repository creation from an uploaded bundle or a URL, keeping the mapping so later oxy git fetch keeps the imported bookmark following the Git branch.

15. Security

15.1 Threat model

An anonymous internet user, a signed-in user attempting to exceed their role, a collaborator with write attempting to move a reference the policy forbids, a malicious repository content author (Markdown, file names, huge objects), and a compromised hub host. The last is bounded by design: the hub holds no root keys, approvals are user-signed seals, history is signed by authors, so a compromised hub can deny service and forge hub-only data (issues, comments) but cannot forge history, approvals or policy without detection.

15.2 Web

CSRF: every unsafe form carries a token bound to the session (HMAC(cookie_key, "csrf:" || session mac)), and the Origin header is checked when present. Cookies are HttpOnly, SameSite=Lax, Secure over HTTPS. Content Security Policy: default-src 'self', no inline scripts except the theme snippet by hash, img-src 'self' data:; rendered Markdown is sanitised by ammonia with an allow-list; rendered HTML files are never served inline (raw file responses use Content-Disposition: attachment and X-Content-Type-Options: nosniff) except images with sniffed, allow-listed types. Path parameters are validated before touching the file system; paths inside repositories are resolved against the object store, never the disk.

15.3 Passwords, sessions, tokens

Argon2id with the defaults of the argon2 crate, per-password salt; TOTP with a 30-second step and one-step drift; recovery codes; sessions as in §5.1; tokens stored as BLAKE3 hashes and compared in constant time; login rate limited per account and per address; all session-changing actions logged in the audit log.

15.4 Repository content

Object sizes are bounded by the wire protocol server's body limits (per push and per object); rendering is bounded by render_limit and by time budgets; archives are streamed; symlinks in trees are shown, never followed on the server; subrepository pins are displayed, never fetched by the hub.

Push protection (oxygenhub#268). A push that adds what looks like a credential is refused before any of it is stored, so the secret never enters the repository's history. The hub implements the protocol server's vet_push hook (Oxygen spec §16.4): after the push is decoded and before anything is stored, bound or moved, it scans what the push adds, the lines its changes insert and the new blobs that replace a file's content (blobs the hub already holds are not new), never history it holds.

• Rules are data, crates/hub/src/push_protection.toml, compiled in and versioned (version, shown in every refusal as rules vN). Each has an id, a description, a regex matched on bytes (its secret group, else the whole match, is the secret), optional keywords that must occur (case-insensitive) before the regex runs, and optionally a regex the same file or inserted text must also match, for secrets that are only one when paired. The first set: AWS access key IDs, AWS secret access keys next to a key ID, Google Cloud service account keys ("type": "service_account" with a private_key), Azure storage connection strings, GitHub tokens (ghp_, gho_, ghu_, ghs_, ghr_, github_pat_), GitLab personal access tokens (glpat-), this hub's own credentials (personal access tokens oxyhub_, agent grant tokens oxyagent_, OxygenFlow job tokens flowjob_, runner credentials flowrunner_ and registration tokens flowreg_), Slack tokens (xoxb- and kin), Stripe live keys (sk_live_, rk_live_), OpenSSH and PEM private keys (a BEGIN … PRIVATE KEY line, a body and its END line; the header alone, as in documentation, is not one). Rules are tried in order and a match overlapping an earlier rule's is dropped, so one secret is reported once, by its most specific rule.
• Findings. Each names its rule, the file and line (the line as the change that adds it leaves the file, worked out from the repository's state with the push's changes applied in memory, only when there is a finding), a fingerprint (the first 16 hex digits of BLAKE3 of the rule id, a NUL and the secret) and the line with the secret masked (its first four characters kept). The secret itself is never shown, logged or stored.
• Budget, so pushes stay fast: at most scan_limit bytes scanned per push request (16 MiB), each file or inserted text at most scan_file_limit (1 MiB; larger ones, and binary blobs, with a NUL in the first 8000 bytes, are skipped), and scan_time_ms (2000) in all, all under [limits] in hub.toml. Past the size or time budget the rest goes unscanned and the push is accepted: refusing would block every large import (a whole history pushed in parts, each part with its own budget) for what is most likely nothing. The accepted references' push log entries say how much went unscanned (push protection scanned 16.0 MiB and stopped at its size budget: … more were not scanned). History the hub already holds is the history scan's (below).
• Refusal. The push fails with HTTP 422, nothing of it stored or moved, and a plain text the client prints as it is: one line per finding, path:line: description (fingerprint F) with its masked line under it (20 at most, then a count), then what to do: take the value out of the change that adds it (oxy amend, or oxy abandon a change that only adds it) and push again, revoke the credential if it was real, or have a maintainer allow the fingerprint at {base_url}/{owner}/{name}/settings/push-protection. A push through an agent's token (an agent-marked token or a keyless agent's grant) adds a paragraph for the agent naming it: remove the value and push again, do not work around the refusal, and tell its person if it was asked to commit the value, since only a maintainer can allow one. The refusal is recorded in the push log (/{owner}/{name}/pushes, each named reference refused with push protection: N possible secrets (path:line, …)), and its findings (rule, path, line, fingerprint, masked line, pusher and agent) in secret_findings (the newest 200 per repository) for the maintainers.
• Bypass. Under Settings › Push protection (maintain role) a maintainer sees the recent findings not yet allowed and allows one by its fingerprint with a reason (required, at most 500 characters), from the finding's row or by typing a fingerprint from a refusal: a row in secret_bypasses (repository, fingerprint, the rule it was found under, reason, who, when), audited as secret.bypass. A finding whose fingerprint is allowed passes; the allowed fingerprints are listed with their reasons. A writer cannot allow one. Each allowed fingerprint has a remove button (maintain role, POST …/settings/secret-bypass/remove), audited as secret.bypass_removed with the reason it had: pushes adding that value are refused again and its history findings are open again (oxygenhub#277).
• Setting. Per repository, repositories.push_protection, on by default; Settings › Push protection (maintain role) turns it off or on, audited as repo.push_protection. A wiki follows its repository's setting and bypasses (migration 0085, both databases).
• History scan (oxygenhub#277). The same rules run over what each repository already holds, so a secret committed before push protection existed, or while it was off, is found too. What is scanned is every change reachable from the repository's bookmarks, read as push protection reads a push: the lines each change inserts and the text blobs it puts in place of a file's content (within scan_file_limit; binary and lazy blobs are skipped). It is the worker role's secrets.history_scan task (docs/roles.md), under its leader lock: ten seconds after the process starts it passes over every repository (so the first deploy scans every repository's whole history), then over a repository again when a push to it is announced on the bus (repo:N, push), when a maintainer asks or turns push protection back on, and every 15 minutes over all of them for what the bus missed (a landing, a lost event). Repositories with push protection off are skipped; wikis are not scanned.
  • Progress and budget. Each repository's scan is a row in secret_scans: the rules version it ran with, the heads whose history is scanned (done_heads), and while a scan is under way the heads it works toward (target_heads, the bookmarks' heads when it started) and how far it got (position of pending changes). The changes to scan are those reachable from the target and not from the done heads, ordered by clock then hash, so every pass sees the same list and resumes at position. A pass over one repository stops at history_scan_limit bytes (64 MiB; a change counts at least one byte) or history_scan_time_ms (5000) wall-clock, under [limits], after at least one change; the worker then moves to the next repository and comes back, so every repository progresses in turn. When the list is done the target becomes the done heads: after a push only the new changes are scanned. A push during a scan waits for it to finish. A target that no longer loads (a bookmark rewritten and collected) is replaced by the bookmarks as they are. A new rules version scans every history again. history_scan_limit = 0 turns the scan off.
  • Low priority. One repository at a time, one pass each in turn, on one blocking thread that rests as long as it worked (at most half a core), 200 ms between repositories, holding only the repository's shared lock, as pages do: it never holds up a push, a page or a gc.
  • Findings go into secret_findings with source history (refused pushes' are push), the change that adds the value (change_hash), the first bookmark that reaches it (the default bookmark first) and the file and line as that change leaves the file, worked out from the state of its dependencies with it applied. One row per value and file however many changes carry it (a unique index on repository, fingerprint and path for history rows), so a rescan adds nothing known and keeps what was resolved; they are not trimmed with the 200 push findings.
  • Found in history. Settings › Push protection (maintain role) shows the scan's state (scanning N of M changes, or scanned up to the bookmarks and when, with totals and the last error) and the open findings, oldest first: each with its change (linked), bookmark, file and line, masked line and fingerprint, and two actions. Allow is the push bypass above (a value that is not a secret is then allowed for pushes too). Mark resolved (POST …/settings/secret-resolve, a reason required, at most 500 characters) says the secret was revoked or rotated, since removing it from the files now leaves it in every earlier change; it records who, when and how, audited as secret.resolved, and moves the finding to a Resolved list. Scan history now (POST …/settings/secret-scan, maintain role, audited as secret.history_scan) forgets what was scanned (findings and marks stay) and puts the repository first. A writer can do none of these (migration 0086, both databases).

15.5 Keys and TLS

The hub delegate key lives in <data>/keys/ with mode 0600. Its identity object is published at /.well-known/oxyhub-key so policy holders can admit it. TLS: the hub serves HTTPS itself with the configured certificate, or plain HTTP behind a terminating proxy, or on the developer edition a self-signed certificate it generates at init (with rcgen) whose CA is downloadable from the trust page and usable with OXY_CA.

15.6 Rate limits and abuse

Per address for anonymous requests, per account and per token otherwise, with separate budgets for pages, API, search and the wire protocol; registration and login have their own. Administrators can ban accounts and block addresses.

16. Performance and caching

16.1 Targets

Repository overview and tree pages in under 100 ms on a warm hub for a repository of 100 000 changes; a change page in under 200 ms; a projected state for a 50-change proposal in under 2 s; wire protocol throughput limited by disk and network, not the hub.

16.2 Locks and blocking

Every oxy-core call runs on the blocking pool. Readers never wait (oxygenhub#249): each repository has a writer lock that pushes, gc, landings, web edits, Diffusion bumps, Flow's blobs, mirror exports, forks and a backup's last pass take in turn, and a lifecycle lock that everyone holds shared and only deleting or moving the repository holds alone. Page and API reads and fetches take only the shared lifecycle lock, so a push or a gc never holds them up: the store gives each handle one head for its life and finds what a repack moved (a reload and a retry on a miss; repacks retire what they replace for an hour before it is deleted; Oxygen spec §13.4). oxy-core itself is single-threaded per Repo, so a Repo is opened per request and dropped after; store handles are cheap to open because packs are memory mapped.

16.3 Caches

Keyed by state hash, so they never go stale: loaded State objects (LRU by memory, up to 256 and OXYHUB_STATE_CACHE_MB, default 3 GiB), a state's brief (its change count, latest change and whether it is conflicted, #145: kept in memory and on disk under <data>/cache/summaries/, worked out four at a time when a page needs many and as soon as a push moves a bookmark, so a restart never makes a bookmark list or an overview load every bookmark's state again), tree listings, rendered file views, blame, history graph layouts per bookmark head set, projected states per proposal, archives, code search results. Keyed by content: rendered Markdown (BLAKE3 of the text and its two link bases, 32 MiB) and syntax highlighted files (BLAKE3 of the file name, which picks the syntax, and the text, 64 MiB), each an LRU bounded by the bytes it holds, rendering outside the lock; an entry larger than a quarter of its budget is not kept. Keyed by repository: bookmark and tag lists, invalidated by the push path. ETag and Cache-Control: private, max-age=0, must-revalidate on pages; immutable on content-addressed responses (raw files by hash, archives by state). A request that takes two seconds or more is logged at warn with its method, path (never the query) and status.

16.4 Large repositories

History pages paginate by cursor; the graph renders a window of 100 changes with lazy extension; log_for_path for tree rows is fetched per row after the page paints and cached per (state, path). The projected state job uses load_state on the target and applies only the proposal's changes on top.

17. Operations: configuration, storage, backup

17.1 Configuration

hub.toml in the data directory; every key overridable by OXYHUB_<SECTION>_<KEY>:

[server]
bind = "0.0.0.0:7300"          # plain HTTP
tls_bind = "0.0.0.0:7343"      # HTTPS, when tls.cert and tls.key (or self_signed) are set
base_url = "https://oxygenhub.io"
behind_proxy = false            # trust X-Forwarded-* when true
roles = ["web", "worker"]       # what this process does (docs/roles.md); empty or absent: both.
                                # `serve --role` overrides it. web: pages, API, wire protocol,
                                # runner API, the transparency log; worker: background tasks,
                                # singletons under a leader lock, serving only /health and /metrics.
                                # One web process per data directory until #274.

[tls]
self_signed = true              # developer edition; otherwise cert = "...", key = "..."

[registration]
mode = "invite"                 # open | invite | closed

[limits]
render_limit = "2MiB"
asset_limit = "512MiB"
push_body_limit = "4GiB"
scan_limit = "16MiB"            # push protection (§15.4): scanned per push
scan_file_limit = "1MiB"        # larger files and inserts are not scanned
scan_time_ms = 2000
history_scan_limit = "64MiB"    # the history scan (§15.4): per repository per pass; 0: off
history_scan_time_ms = 5000

[email]
smtp = "smtp://…"               # optional
from = "OxygenHub <hub@example>"

[mirror]
git = "/usr/bin/git"

[retention]
trash_days = 30
webhook_log_days = 30

17.2 Storage

SQLite in WAL mode; migrations embedded and applied at start; PRAGMA user_version guards downgrades. Repositories are ordinary Oxygen repositories; oxy gc runs per repository as a nightly job, one repository at a time under its write lock, at retention.gc_hour UTC (default 3; 24 or more turns it off), and oxygenhub gc runs the same pass by hand. Each repository's run is logged with what it pruned and packed.

17.3 Command line

oxygenhub init  --data DIR [--admin NAME]      create the data directory, keys, database, first admin
oxygenhub serve --data DIR [--role web,worker] run, taking these roles (all by default)
oxygenhub admin --data DIR user|repo|token …   offline administration
oxygenhub backup --data DIR --to FILE          consistent snapshot: sqlite backup API plus repos and assets as tar.zst
oxygenhub check --data DIR                     verify every repository's objects and signatures, report

17.4 Backup and restore

backup uses SQLite's online backup API for the database and copies repos/, assets/, keys/ and hub.toml; cache/ is excluded. Restore is extraction plus serve. Because objects are content addressed, a repository directory copied while a push is in flight is still valid up to its last committed reference update.

17.5 Developer edition on Earth

Earth (earth.andymccall.xyz, Ubuntu 26.04, x86_64, 4 cores, 31 GiB, Docker, ports 22, 6400, 6670, 7842, 8080 in use) runs the developer edition as a systemd service under a dedicated oxygenhub system user with <data> = /home/oxygenhub/data, HTTP on 7300 and HTTPS on 7343 with a self-signed certificate. Source is synchronised from the development machine with rsync (the Oxygen workspace and the hub) and built on Earth with a rustup toolchain installed for the claude account; docs/deploy.md gives the exact steps. Once the hub runs, oxygen, oxygen-vscode and oxygenhub are pushed to it as its first repositories and become their own remotes; that fulfils the self-hosting rule without any Git involvement.

18. Required changes to Oxygen

Each is small and lives in the Oxygen workspace, tracked in its docs/plan.md:

1. Bare repositories. Repo::open_bare(dir): open a .oxy/ with no workspace; refuse working-copy operations with a clear error. oxy init --bare.
2. Server embedding. oxy-server exposes router_with(resolver, authorizer) where the resolver maps a path (possibly owner/name) to a repository directory and the authorizer decides read and push per request, so the hub embeds it instead of copying handlers. Repository names with one / allowed when the resolver permits.
3. Challenge signing. oxy identity sign-challenge <bytes> and oxy_core::identity::verify_challenge, for linking keys (§5.3) and HTTP signature authentication (§5.4).
4. Seal command. oxy seal <hash>.. signs a seal attestation over the given changes with the identity key and pushes it on the next push (the seal object and its verification exist, spec §14.4; today only the sync path creates seals). oxy seal --proposal <url> fetches the hash list from the hub API first.
5. Landing from a client. oxy proposal land <url> and oxy proposal show <url> as thin API clients.
6. Delegate signing of hub-created changes. A library entry point to build and sign a change from an edit set with author = user key, signer = hub key, without a workspace (used by web edits, browser conflict resolution and landings). oxy identity delegate add|remove already exists.
7. History for the hub. Repo::log_window(heads, cursor, limit) for paginated history, and a graph::layout module shared with the CLI so both draw the same lanes.
8. Push hook. receive_push reports, per accepted reference, the old and new state and the list of newly stored changes, for the pushes log and webhooks.
9. Machine interface. oxy verify --json already exists; add oxy policy propose output as a downloadable proposal file (it exists as a file format, spec §14.5) and oxy tag list --json is already present.

19. Milestones

20. Open questions

1. Seal signing UX. Requiring oxy seal from a client for every approval is honest but slower than a button. A browser-held signing key (WebAuthn prf extension deriving an Ed25519 key) could sign approvals in the browser with hardware backing; ML-DSA-65 in the browser is heavier. Decide after 0.3 ships with the client flow.
2. Delegate key custody. One hub key delegated to by many users means one key whose compromise lets an attacker sign as any of them (still subject to policy and to each user's revocation). Per-user delegate keys held by the hub would narrow that blast radius at the cost of a key per user. Decide before 0.6.
3. Postgres. SQLite is enough for one host; an organisation-scale hub may want Postgres. sqlx keeps the option open, but every query would need to be dialect-clean. Decide when someone needs it.
4. Comment anchoring for lines that are not atoms. Comments on deleted lines and on binary files fall back to file anchors; is that good enough?
5. Pipeline language. KDL is the current choice for .oxyhub/pipelines; Starlark would give loops and functions at the cost of a real evaluator. Settle in docs/runner.md.
6. Federation. Proposals across hubs (a fork on another hub) are natural in a content-addressed model and out of scope for 1.0.

21. Changelog

• Draft 0.1 (2026-09-27): initial specification.