Diffusion: upstream changes reach their consumers

Status: design, 2026-10-12. D1 is planned first; nothing is built yet. "Diffusion" is a working name (changes diffuse from an upstream into the repositories that pin it); Andy has not confirmed it.

1. The problem

R* API is a subrepo of Metal Gear, Metal Gear 2, Super Mario Land and others. When it changes, each consumer should take the change, test it, and propose it when it works, with no one opening proposals by hand. Renovate and Dependabot do this for package manifests; Diffusion does it for Oxygen pins, as part of Oxygen, and goes further than they can.

2. Why Oxygen can do better

• The dependency graph is exact. An Oxygen pin names a repository id and a state, and is a verifiable object (spec §7.8). The hub already reads every pin in a state to give jobs their subrepos. Nothing is scanned or guessed.
• A pin is a set of changes, and independent changes commute. A consumer can take an upstream state minus one change and whatever is built on it (§6). Git tools can only pin one commit: all or nothing.
• Bumps never need rebasing. The pin change commutes with the consumer's own work; only a consumer moving the same pin itself conflicts, as an ordinary replace conflict.
• Everything is signed. The bump, the upstream changes and the CI receipts are signed objects, so a consumer can check what it takes, even from another hub.

3. The approaches compared

Diffusion is bump proposals as the base, with consumer checks on upstream proposals (§7), partial pins (§6) and repair (§8) on top.

4. D1: follow and propose

4.1 The reverse pin index

When a push moves a repository's default bookmark, the hub records each Oxygen pin in the new state: consumer, path, the upstream repository id and the pinned heads (diffusion_pins). That answers "who consumes R* API, at which state" at once. An upstream repository's page shows its consumers on this hub ("Diffuses to"); the API gives GET /repos/{owner}/{name}/consumers, and GET /repos/{owner}/{name}/diffusion gives both directions with their pills (#153).

Keeping it right (oxygenhub#210). Each write records the state it indexed (diffusion_indexed); a failed write is logged with the step that failed and tried once more, and a state whose pins cannot be read leaves the index as it was. Every ten minutes a reconciler compares each followed bookmark with the state it was indexed at and re-indexes where they differ; in the same pass, a consumer whose pin is not its upstream's newest passing state, with no bump open or in draft, is caught up as if it had just turned Diffusion on. A bump a person closed counts as a "no": it is not opened again until the upstream moves. Repairs are logged (diffusion: reconciled: …).

4.2 Turning it on

A consumer follows a subrepo when, in order of precedence:

1. its .oxymodules entry says so, beside the existing url and bookmark keys:

   [rstar-api]
   url = "https://oxygenhub.io/doomsdayone/rstar-api"
   bookmark = "main"
   follow = "green"   # off | green | any
   land = "propose"   # propose | auto
   

2. else its repository setting (Settings › Subrepos), so an administrator can turn it on without a commit;

3. else the hub's default ([diffusion] follow in hub.toml, off).

green waits for the upstream state's required checks to pass; any follows every move. An upstream with no checks at all is unverified (amber): it never turns green, so its moves bump green followers too, and the proposal is labelled upstream unverified with a note saying only the consumer's pipeline tested it (decided by Andy, 2026-10-13). A passing run of the upstream later clears it.

Landing by itself (built 2026-10-14, oxygenhub#174; Andy chose "opt-in, per repository" after rstar-api's first green/green): land = auto in the entry, else the repository's Settings › Diffusion › Landing (PUT …/diffusion {"land": "auto"}), off by default. The hub lands a bump itself, as the diffusion account, when both halves are green: the exact upstream state the bump pins passed its own run, and the consumer's newest run on the bump bookmark passed on the bump as it is now. Never for amber: an upstream with no checks always waits for a person. It lands through the web's own path (land_as), signed by the hub's own key, so the consumer's required checks, approvals and trust policy decide as for anyone; a policy that does not let the hub's key move the bookmark keeps the bump waiting, and the proposal's timeline says why, once. It is tried when the bump's run passes, when the upstream's run passes under a bump already made, and when the setting is turned to auto. The timeline records "landed (Diffusion, both halves green)".

4.3 The bump

When an upstream's followed bookmark reaches a new state S (and, for green, S's required checks pass), for each consumer that follows it:

• Skip when the consumer's pin already holds S, or when S does not hold everything the current pin does (upstream moved backwards or was rewritten). Consumers only move forward.
• Build one change on the consumer's default heads that replaces the pin's blob with a pin of S. The hub signs it with its delegate key, as it signs wiki edits, authored by the hub's diffusion account.
• Place it on the bookmark diffusion/<path>: the consumer's default heads plus the change. One bookmark per consumer and subrepo, moved in place when upstream moves again, so there is never a pile of bumps.
• Propose it, once: a proposal from diffusion/<path> to the default bookmark, labelled dependencies, kept up to date as the bookmark moves. Its CI runs on the projected state like any proposal's.

A trust policy that does not admit the hub's key still gets the proposal; it is landed from a client, as today.

4.4 What the proposal says

• The upstream, the old pin and the new state.
• The exact changelog: every upstream change in S and not in the old pin, with its author, whether it is signed, and the issues it closes or refers to.
• Its checks, as for any proposal. A red bump stays open, with the failing check; D2 then says which upstream change broke it.

4.5 Seeing it (#150)

A repository's Diffusion tab lists who pins it (Used by) and what it pins (Follows), each with a status Andy chose on 2026-10-13: green (the upstream was checked and the consumer's run on the bump passed), amber (the run passed but the upstream has no checks), red (the run failed), grey (running), up to date. The overview shows the same in brief; Admin › Diffusion shows the whole hub.

It is drawn as a two-half pill, like a capsule (Andy, 2026-10-13): the left half is the upstream (checked, untested, failing, testing), the right half the consumer (passes, fails, testing, waiting, up to date), each coloured on its own, so a game that passes on an untested upstream reads [untested │ passes] rather than a single amber that hides the pass.

5. Across hubs

Each hub only ever writes to its own repositories. What one hub tells another is a hint; the consumer's hub fetches and verifies the facts.

• Polling always works. The consumer's hub has the upstream URL in .oxymodules; it asks for the followed bookmark every few minutes. An upstream kept in a Strata bucket with no hub is polled by reading its bookmark record.
• Subscribing makes it prompt. The consumer's hub asks the upstream's hub to notify it when the bookmark moves. Notes are signed with the upstream hub's key and carry only "repository id moved to state S"; the consumer fetches and verifies. A forged note causes a fetch, nothing more; a lost one is caught by the next poll.
• Private upstreams are read with a read-only credential the consumer's owner stores as a secret on the consumer's hub.
• "Green" across hubs is evidence the consumer checks: a Flow receipt for S, signed by a runner the upstream hub vouches for. The consumer's policy names the upstream hub keys it accepts, and may also require every upstream change to be signed by the upstream's maintainers, which holds even if the upstream hub is compromised.
• Consumer checks across hubs (§7) are opt-in on both sides; the consumer's hub runs them on its own runners and returns only a signed summary, or nothing.

6. D2: partial pins and the culprit

A pin is a state: any set of changes closed under dependencies. So a consumer can pin "upstream S, minus change X and everything built on X": its heads are the maximal changes of closure(S) − ({X} ∪ dependents(X)), well-formed by construction.

When a bump of N upstream changes fails:

1. Flow searches for the culprit by testing subsets of the N changes (a delta-debugging search over independent changes, bounded in runs).
2. The bump proposal then pins everything except the culprit and its dependents, and says which change is held back and why, with the failure.
3. The consumer gets the rest at once; the held-back change comes with a later bump, once upstream fixes it.

oxy submodule and checkout take such pins as they are (a pin is heads); oxy submodule status names the changes a partial pin leaves out.

7. D3: consumers test upstream proposals

A proposal on R* API can run each consumer's pipeline with its pin swapped for the proposal's projected state, committing nothing to the consumer (the claim already carries subrepo pins; the run overrides one). Each consumer's result appears on the upstream proposal as a check, consumers/<owner>/<name>, with a summary: "Metal Gear ✔, Super Mario Land ✘ (test X)". The person changing the API sees what breaks before it lands, which is the monorepo's main strength without taking consumers' control. An upstream may make consumer checks required.

8. D4: repair

• Migrations with the change. An upstream change that alters the API may carry a migration (a script, or instructions for an agent) under .diffusion/migrations/. A bump spanning it runs the migration in the consumer and adds the result as a separate change on the bump proposal.
• An agent's fix. When a bump is red, a scoped agent key (an agent home with changes scope on that consumer, never refs) may add a fix as a separate change to the proposal, for the consumer's owner to review. The bump and the fix stay separate changes, so either can be dropped.

9. D5: reach

• Auto-land (land = "auto"): a green bump lands itself when the consumer's trust policy and required checks allow it, and every upstream change is signed by keys the policy trusts.
• Chains: landing a bump moves the consumer's default bookmark, which may be followed in turn.
• Grouping and quiet hours: several upstreams' bumps in one proposal; no bumps outside set hours.
• Cross-hub polling and subscriptions (§5).
• Channels: with Cylinder, a consumer follows rstar-api:stable, a signed channel whose moves are gated by consumer checks (§7).

10. Order

D2 is what no other tool can do; D3 is worth the most.

11. Open questions

• The name: Diffusion, or plainer ("Follow")?
• Who may turn it on for a consumer: its owner and administrators, or also the upstream's maintainers (by offering, which the consumer accepts)?
• Bot identity: a diffusion account per hub, or the upstream pusher's name with the hub as delegate?
• Cost limits for the culprit search and consumer checks: runs per bump, per upstream proposal, per day.