| 8620925e Andy McCall ↶ |
1 |
# Diffusion: upstream changes reach their consumers |
| 2 |
|
| 3 |
Status: design, 2026-10-12. D1 is planned first; nothing is built yet. |
| 4 |
"Diffusion" is a working name (changes diffuse from an upstream into the |
| 5 |
repositories that pin it); Andy has not confirmed it. |
| 6 |
|
| 7 |
## 1. The problem |
| 8 |
|
| 9 |
R* API is a subrepo of Metal Gear, Metal Gear 2, Super Mario Land and |
| 10 |
others. When it changes, each consumer should take the change, test it, |
| 11 |
and propose it when it works, with no one opening proposals by hand. |
| 12 |
Renovate and Dependabot do this for package manifests; Diffusion does it |
| 13 |
for Oxygen pins, as part of Oxygen, and goes further than they can. |
| 14 |
|
| 15 |
## 2. Why Oxygen can do better |
| 16 |
|
| 17 |
- **The dependency graph is exact.** An Oxygen pin names a repository id |
| 18 |
and a state, and is a verifiable object (spec §7.8). The hub already |
| 19 |
reads every pin in a state to give jobs their subrepos. Nothing is |
| 20 |
scanned or guessed. |
| 21 |
- **A pin is a set of changes, and independent changes commute.** A |
| 22 |
consumer can take an upstream state minus one change and whatever is |
| 23 |
built on it (§6). Git tools can only pin one commit: all or nothing. |
| 24 |
- **Bumps never need rebasing.** The pin change commutes with the |
| 25 |
consumer's own work; only a consumer moving the same pin itself |
| 26 |
conflicts, as an ordinary replace conflict. |
| 27 |
- **Everything is signed.** The bump, the upstream changes and the CI |
| 28 |
receipts are signed objects, so a consumer can check what it takes, even |
| 29 |
from another hub. |
| 30 |
|
| 31 |
## 3. The approaches compared |
| 32 |
|
| 33 |
|
| 34 |
|
| 35 |
| Bump proposals (Renovate, Dependabot) | Consumers stay in charge; auditable; works across hubs | Reactive: breakage is found after upstream ships | |
| 36 |
| Monorepo at head (Google) | Upstream cannot land what breaks a consumer | Takes control from consumers; upstream sees all code; one hub only | |
| 37 |
| Upstream-driven migrations (large-scale changes, codemods) | Handles breaking changes, not just detects them | Needs tooling per change | |
| 38 |
| Consumer-gated promotion (contract testing) | Prevents breakage | Needs consumers' tests to run upstream | |
| 39 |
|
| 40 |
Diffusion is bump proposals as the base, with consumer checks on upstream |
| 41 |
proposals (§7), partial pins (§6) and repair (§8) on top. |
| 42 |
|
| 43 |
## 4. D1: follow and propose |
| 44 |
|
| 45 |
### 4.1 The reverse pin index |
| 46 |
|
| 47 |
When a push moves a repository's default bookmark, the hub records each |
| 48 |
Oxygen pin in the new state: consumer, path, the upstream repository id |
| 49 |
and the pinned heads (`diffusion_pins`). That answers "who consumes R* |
| 50 |
API, at which state" at once. An upstream repository's page shows its |
| eb2a3502 Andy McCall ↶ |
51 |
consumers on this hub ("Diffuses to"); the API gives |
| 52 |
`GET /repos/{owner}/{name}/consumers`, and `GET /repos/{owner}/{name}/diffusion` |
| 53 |
gives both directions with their pills (#153). |
| 8620925e Andy McCall ↶ |
54 |
|
| 55 |
### 4.2 Turning it on |
| 56 |
|
| 57 |
A consumer follows a subrepo when, in order of precedence: |
| 58 |
|
| 59 |
1. its `.oxymodules` entry says so, beside the existing `url` and |
| 60 |
`bookmark` keys: |
| 61 |
|
| 62 |
```toml |
| 63 |
[rstar-api] |
| 64 |
url = "https://oxygenhub.io/doomsdayone/rstar-api" |
| 65 |
bookmark = "main" |
| 66 |
follow = "green" # off | green | any |
| 67 |
land = "propose" # propose | auto |
| 68 |
``` |
| 69 |
|
| 70 |
2. else its repository setting (Settings › Subrepos), so an |
| 71 |
administrator can turn it on without a commit; |
| 72 |
3. else the hub's default (`[diffusion] follow` in hub.toml, `off`). |
| 73 |
|
| 74 |
`green` waits for the upstream state's required checks to pass; `any` |
| 9aaa88b2 Andy McCall ↶ |
75 |
follows every move. An upstream with no checks at all is **unverified** |
| 76 |
(amber): it never turns green, so its moves bump green followers too, and |
| 77 |
the proposal is labelled `upstream unverified` with a note saying only the |
| 78 |
consumer's pipeline tested it (decided by Andy, 2026-10-13). A passing run |
| f4d6ef41 Andy McCall ↶ |
79 |
of the upstream later clears it. |
| 8620925e Andy McCall ↶ |
80 |
|
| f4d6ef41 Andy McCall ↶ |
81 |
**Landing by itself** (built 2026-10-14, oxygenhub#174; Andy chose |
| 82 |
"opt-in, per repository" after rstar-api's first green/green): `land = |
| 83 |
auto` in the entry, else the repository's Settings › Diffusion › Landing |
| 84 |
(`PUT …/diffusion {"land": "auto"}`), off by default. The hub lands a bump |
| 85 |
itself, as the `diffusion` account, when both halves are green: the exact |
| 86 |
upstream state the bump pins passed its own run, and the consumer's |
| 87 |
newest run on the bump bookmark passed on the bump as it is now. Never |
| 88 |
for amber: an upstream with no checks always waits for a person. It lands |
| 89 |
through the web's own path (`land_as`), signed by the hub's own key, so |
| 90 |
the consumer's required checks, approvals and trust policy decide as for |
| 91 |
anyone; a policy that does not let the hub's key move the bookmark keeps |
| 92 |
the bump waiting, and the proposal's timeline says why, once. It is tried |
| 93 |
when the bump's run passes, when the upstream's run passes under a bump |
| 94 |
already made, and when the setting is turned to `auto`. The timeline |
| 95 |
records "landed (Diffusion, both halves green)". |
| 96 |
|
| 8620925e Andy McCall ↶ |
97 |
### 4.3 The bump |
| 98 |
|
| 99 |
When an upstream's followed bookmark reaches a new state S (and, for |
| 100 |
`green`, S's required checks pass), for each consumer that follows it: |
| 101 |
|
| 102 |
- **Skip** when the consumer's pin already holds S, or when S does not |
| 103 |
hold everything the current pin does (upstream moved backwards or was |
| 104 |
rewritten). Consumers only move forward. |
| 105 |
- **Build** one change on the consumer's default heads that replaces the |
| 106 |
pin's blob with a pin of S. The hub signs it with its delegate key, as |
| 107 |
it signs wiki edits, authored by the hub's `diffusion` account. |
| 108 |
- **Place** it on the bookmark `diffusion/<path>`: the consumer's default |
| 109 |
heads plus the change. One bookmark per consumer and subrepo, moved in |
| 110 |
place when upstream moves again, so there is never a pile of bumps. |
| 111 |
- **Propose** it, once: a proposal from `diffusion/<path>` to the default |
| 112 |
bookmark, labelled `dependencies`, kept up to date as the bookmark |
| 113 |
moves. Its CI runs on the projected state like any proposal's. |
| 114 |
|
| 115 |
A trust policy that does not admit the hub's key still gets the |
| 116 |
proposal; it is landed from a client, as today. |
| 117 |
|
| 118 |
### 4.4 What the proposal says |
| 119 |
|
| 120 |
- The upstream, the old pin and the new state. |
| 121 |
- **The exact changelog:** every upstream change in S and not in the old |
| 122 |
pin, with its author, whether it is signed, and the issues it closes or |
| 123 |
refers to. |
| 124 |
- Its checks, as for any proposal. A red bump stays open, with the |
| 125 |
failing check; D2 then says which upstream change broke it. |
| 126 |
|
| 757195ad Andy McCall ↶ |
127 |
### 4.5 Seeing it (#150) |
| 128 |
|
| 129 |
A repository's **Diffusion** tab lists who pins it (Used by) and what it |
| 130 |
pins (Follows), each with a status Andy chose on 2026-10-13: **green** |
| 131 |
(the upstream was checked and the consumer's run on the bump passed), |
| 132 |
**amber** (the run passed but the upstream has no checks), **red** (the run |
| 133 |
failed), **grey** (running), **up to date**. The overview shows the same in |
| 134 |
brief; Admin › Diffusion shows the whole hub. |
| 135 |
|
| 73ef9f76 Andy McCall ↶ |
136 |
It is drawn as a two-half pill, like a capsule (Andy, 2026-10-13): the |
| 137 |
left half is the upstream (**checked**, **untested**, **failing**, |
| 138 |
**testing**), the right half the consumer (**passes**, **fails**, |
| 139 |
**testing**, **waiting**, **up to date**), each coloured on its own, so a |
| 140 |
game that passes on an untested upstream reads [untested │ passes] rather |
| 141 |
than a single amber that hides the pass. |
| 142 |
|
| 8620925e Andy McCall ↶ |
143 |
## 5. Across hubs |
| 144 |
|
| 145 |
Each hub only ever writes to its own repositories. What one hub tells |
| 146 |
another is a hint; the consumer's hub fetches and verifies the facts. |
| 147 |
|
| 148 |
- **Polling always works.** The consumer's hub has the upstream URL in |
| 149 |
`.oxymodules`; it asks for the followed bookmark every few minutes. An |
| 150 |
upstream kept in a Strata bucket with no hub is polled by reading its |
| 151 |
bookmark record. |
| 152 |
- **Subscribing makes it prompt.** The consumer's hub asks the upstream's |
| 153 |
hub to notify it when the bookmark moves. Notes are signed with the |
| 154 |
upstream hub's key and carry only "repository *id* moved to state *S*"; |
| 155 |
the consumer fetches and verifies. A forged note causes a fetch, nothing |
| 156 |
more; a lost one is caught by the next poll. |
| 157 |
- **Private upstreams** are read with a read-only credential the |
| 158 |
consumer's owner stores as a secret on the consumer's hub. |
| 159 |
- **"Green" across hubs** is evidence the consumer checks: a Flow receipt |
| 160 |
for S, signed by a runner the upstream hub vouches for. The consumer's |
| 161 |
policy names the upstream hub keys it accepts, and may also require |
| 162 |
every upstream change to be signed by the upstream's maintainers, which |
| 163 |
holds even if the upstream hub is compromised. |
| 164 |
- **Consumer checks across hubs (§7)** are opt-in on both sides; the |
| 165 |
consumer's hub runs them on its own runners and returns only a signed |
| 166 |
summary, or nothing. |
| 167 |
|
| 168 |
## 6. D2: partial pins and the culprit |
| 169 |
|
| 170 |
A pin is a state: any set of changes closed under dependencies. So a |
| 171 |
consumer can pin "upstream S, minus change X and everything built on X": |
| 172 |
its heads are the maximal changes of `closure(S) − ({X} ∪ dependents(X))`, |
| 173 |
well-formed by construction. |
| 174 |
|
| 175 |
When a bump of N upstream changes fails: |
| 176 |
|
| 177 |
1. Flow searches for the culprit by testing subsets of the N changes (a |
| 178 |
delta-debugging search over independent changes, bounded in runs). |
| 179 |
2. The bump proposal then pins everything except the culprit and its |
| 180 |
dependents, and says which change is held back and why, with the |
| 181 |
failure. |
| 182 |
3. The consumer gets the rest at once; the held-back change comes with a |
| 183 |
later bump, once upstream fixes it. |
| 184 |
|
| 185 |
`oxy submodule` and checkout take such pins as they are (a pin is heads); |
| 186 |
`oxy submodule status` names the changes a partial pin leaves out. |
| 187 |
|
| 188 |
## 7. D3: consumers test upstream proposals |
| 189 |
|
| 190 |
A proposal on R* API can run each consumer's pipeline with its pin swapped |
| 191 |
for the proposal's projected state, committing nothing to the consumer |
| 192 |
(the claim already carries subrepo pins; the run overrides one). Each |
| 193 |
consumer's result appears on the upstream proposal as a check, |
| 194 |
`consumers/<owner>/<name>`, with a summary: "Metal Gear ✔, Super Mario |
| 195 |
Land ✘ (test X)". The person changing the API sees what breaks before it |
| 196 |
lands, which is the monorepo's main strength without taking consumers' |
| 197 |
control. An upstream may make consumer checks required. |
| 198 |
|
| 199 |
## 8. D4: repair |
| 200 |
|
| 201 |
- **Migrations with the change.** An upstream change that alters the API |
| 202 |
may carry a migration (a script, or instructions for an agent) under |
| 203 |
`.diffusion/migrations/`. A bump spanning it runs the migration in the |
| 204 |
consumer and adds the result as a separate change on the bump proposal. |
| 205 |
- **An agent's fix.** When a bump is red, a scoped agent key (an agent |
| 206 |
home with `changes` scope on that consumer, never `refs`) may add a fix |
| 207 |
as a separate change to the proposal, for the consumer's owner to |
| 208 |
review. The bump and the fix stay separate changes, so either can be |
| 209 |
dropped. |
| 210 |
|
| 211 |
## 9. D5: reach |
| 212 |
|
| 213 |
- **Auto-land** (`land = "auto"`): a green bump lands itself when the |
| 214 |
consumer's trust policy and required checks allow it, and every |
| 215 |
upstream change is signed by keys the policy trusts. |
| 216 |
- **Chains:** landing a bump moves the consumer's default bookmark, which |
| 217 |
may be followed in turn. |
| 218 |
- **Grouping and quiet hours:** several upstreams' bumps in one proposal; |
| 219 |
no bumps outside set hours. |
| 220 |
- **Cross-hub** polling and subscriptions (§5). |
| 221 |
- **Channels:** with Cylinder, a consumer follows `rstar-api:stable`, a |
| 222 |
signed channel whose moves are gated by consumer checks (§7). |
| 223 |
|
| 224 |
## 10. Order |
| 225 |
|
| 226 |
|
| 227 |
|
| 228 |
| D1 | The reverse pin index and Used by; turning it on; the signed bump on `diffusion/<path>`, moved in place; the proposal with the changelog | |
| 229 |
| D2 | Partial pins; the culprit search | |
| 230 |
| D3 | Consumer checks on upstream proposals | |
| 231 |
| D4 | Migrations and agent fixes | |
| 232 |
| D5 | Auto-land, chains, grouping, cross-hub, channels | |
| 233 |
|
| 234 |
D2 is what no other tool can do; D3 is worth the most. |
| 235 |
|
| 236 |
## 11. Open questions |
| 237 |
|
| 238 |
- **The name:** Diffusion, or plainer ("Follow")? |
| 239 |
- **Who may turn it on** for a consumer: its owner and administrators, or |
| 240 |
also the upstream's maintainers (by offering, which the consumer |
| 241 |
accepts)? |
| 242 |
- **Bot identity:** a `diffusion` account per hub, or the upstream |
| 243 |
pusher's name with the hub as delegate? |
| 244 |
- **Cost limits** for the culprit search and consumer checks: runs per |
| 245 |
bump, per upstream proposal, per day. |