OXYGENHUB
◆ v0.5.0 oxygenhub / docs / diffusion.md
12.2 KiB · Markdown · last changed Diffusion lands a bump itself where a repository chose it 22 days ago renderedsource blame history raw
8620925e Andy McCall ↶ 1
# Diffusion: upstream changes reach their consumers
2
3
Status: design, 2026-10-12. D1 is planned first; nothing is built yet.
b20ddaf9 Andy McCall ↶ 4
The name **Diffusion** was confirmed by Andy on 2026-10-12: changes
5
diffuse from an upstream into the repositories that pin it.
8620925e Andy McCall ↶ 6
7
## 1. The problem
8
9
R* API is a subrepo of Metal Gear, Metal Gear 2, Super Mario Land and
10
others. When it changes, each consumer should take the change, test it,
11
and propose it when it works, with no one opening proposals by hand.
12
Renovate and Dependabot do this for package manifests; Diffusion does it
13
for Oxygen pins, as part of Oxygen, and goes further than they can.
14
15
## 2. Why Oxygen can do better
16
17
- **The dependency graph is exact.** An Oxygen pin names a repository id
18
  and a state, and is a verifiable object (spec §7.8). The hub already
19
  reads every pin in a state to give jobs their subrepos. Nothing is
20
  scanned or guessed.
21
- **A pin is a set of changes, and independent changes commute.** A
22
  consumer can take an upstream state minus one change and whatever is
23
  built on it (§6). Git tools can only pin one commit: all or nothing.
24
- **Bumps never need rebasing.** The pin change commutes with the
25
  consumer's own work; only a consumer moving the same pin itself
26
  conflicts, as an ordinary replace conflict.
27
- **Everything is signed.** The bump, the upstream changes and the CI
28
  receipts are signed objects, so a consumer can check what it takes, even
29
  from another hub.
30
31
## 3. The approaches compared
32
33
| Approach | Strength | Weakness |
34
|---|---|---|
35
| Bump proposals (Renovate, Dependabot) | Consumers stay in charge; auditable; works across hubs | Reactive: breakage is found after upstream ships |
36
| Monorepo at head (Google) | Upstream cannot land what breaks a consumer | Takes control from consumers; upstream sees all code; one hub only |
37
| Upstream-driven migrations (large-scale changes, codemods) | Handles breaking changes, not just detects them | Needs tooling per change |
38
| Consumer-gated promotion (contract testing) | Prevents breakage | Needs consumers' tests to run upstream |
39
40
Diffusion is bump proposals as the base, with consumer checks on upstream
41
proposals (§7), partial pins (§6) and repair (§8) on top.
42
43
## 4. D1: follow and propose
44
45
### 4.1 The reverse pin index
46
47
When a push moves a repository's default bookmark, the hub records each
48
Oxygen pin in the new state: consumer, path, the upstream repository id
49
and the pinned heads (`diffusion_pins`). That answers "who consumes R*
50
API, at which state" at once. An upstream repository's page shows its
eb2a3502 Andy McCall ↶ 51
consumers on this hub ("Diffuses to"); the API gives
52
`GET /repos/{owner}/{name}/consumers`, and `GET /repos/{owner}/{name}/diffusion`
53
gives both directions with their pills (#153).
8620925e Andy McCall ↶ 54
55
### 4.2 Turning it on
56
57
A consumer follows a subrepo when, in order of precedence:
58
59
1. its `.oxymodules` entry says so, beside the existing `url` and
60
   `bookmark` keys:
61
62
   ```toml
63
   [rstar-api]
64
   url = "https://oxygenhub.io/doomsdayone/rstar-api"
65
   bookmark = "main"
66
   follow = "green"   # off | green | any
67
   land = "propose"   # propose | auto
b20ddaf9 Andy McCall ↶ 68
   target = "develop" # optional: the bookmark to bump (default: learned, §4.3)
8620925e Andy McCall ↶ 69
   ```
70
71
2. else its repository setting (Settings › Subrepos), so an
72
   administrator can turn it on without a commit;
73
3. else the hub's default (`[diffusion] follow` in hub.toml, `off`).
74
75
`green` waits for the upstream state's required checks to pass; `any`
9aaa88b2 Andy McCall ↶ 76
follows every move. An upstream with no checks at all is **unverified**
77
(amber): it never turns green, so its moves bump green followers too, and
78
the proposal is labelled `upstream unverified` with a note saying only the
79
consumer's pipeline tested it (decided by Andy, 2026-10-13). A passing run
f4d6ef41 Andy McCall ↶ 80
of the upstream later clears it.
8620925e Andy McCall ↶ 81
f4d6ef41 Andy McCall ↶ 82
**Landing by itself** (built 2026-10-14, oxygenhub#174; Andy chose
83
"opt-in, per repository" after rstar-api's first green/green): `land =
84
auto` in the entry, else the repository's Settings › Diffusion › Landing
85
(`PUT …/diffusion {"land": "auto"}`), off by default. The hub lands a bump
86
itself, as the `diffusion` account, when both halves are green: the exact
87
upstream state the bump pins passed its own run, and the consumer's
88
newest run on the bump bookmark passed on the bump as it is now. Never
89
for amber: an upstream with no checks always waits for a person. It lands
90
through the web's own path (`land_as`), signed by the hub's own key, so
91
the consumer's required checks, approvals and trust policy decide as for
92
anyone; a policy that does not let the hub's key move the bookmark keeps
93
the bump waiting, and the proposal's timeline says why, once. It is tried
94
when the bump's run passes, when the upstream's run passes under a bump
95
already made, and when the setting is turned to `auto`. The timeline
96
records "landed (Diffusion, both halves green)".
97
8620925e Andy McCall ↶ 98
### 4.3 The bump
99
b20ddaf9 Andy McCall ↶ 100
**Which bookmark of the consumer** (decided 2026-10-12): the bookmark its
101
work lands on. The hub learns it per consumer from where the
102
consumer's proposals land (the most common target of those landed in the
103
last 90 days), falling back to the default bookmark; a `target` key in
104
the consumer's `.oxymodules` entry overrides it. The pin index (§4.1)
105
records pins on that bookmark as well as the default. So a game whose
106
default is `main` but whose work lands on `develop` is bumped on
107
`develop`, with no setup, and follows if its workflow changes.
108
8620925e Andy McCall ↶ 109
When an upstream's followed bookmark reaches a new state S (and, for
110
`green`, S's required checks pass), for each consumer that follows it:
111
112
- **Skip** when the consumer's pin already holds S, or when S does not
113
  hold everything the current pin does (upstream moved backwards or was
114
  rewritten). Consumers only move forward.
b20ddaf9 Andy McCall ↶ 115
- **Build** one change on the target bookmark's heads that replaces the
8620925e Andy McCall ↶ 116
  pin's blob with a pin of S. The hub signs it with its delegate key, as
117
  it signs wiki edits, authored by the hub's `diffusion` account.
b20ddaf9 Andy McCall ↶ 118
- **Place** it on the bookmark `diffusion/<path>`: the target's heads
119
  plus the change. One bookmark per consumer and subrepo, moved in
8620925e Andy McCall ↶ 120
  place when upstream moves again, so there is never a pile of bumps.
b20ddaf9 Andy McCall ↶ 121
- **Propose** it, once: a proposal from `diffusion/<path>` to the target
8620925e Andy McCall ↶ 122
  bookmark, labelled `dependencies`, kept up to date as the bookmark
123
  moves. Its CI runs on the projected state like any proposal's.
124
125
A trust policy that does not admit the hub's key still gets the
126
proposal; it is landed from a client, as today.
127
128
### 4.4 What the proposal says
129
130
- The upstream, the old pin and the new state.
131
- **The exact changelog:** every upstream change in S and not in the old
132
  pin, with its author, whether it is signed, and the issues it closes or
133
  refers to.
134
- Its checks, as for any proposal. A red bump stays open, with the
135
  failing check; D2 then says which upstream change broke it.
136
757195ad Andy McCall ↶ 137
### 4.5 Seeing it (#150)
138
139
A repository's **Diffusion** tab lists who pins it (Used by) and what it
140
pins (Follows), each with a status Andy chose on 2026-10-13: **green**
141
(the upstream was checked and the consumer's run on the bump passed),
142
**amber** (the run passed but the upstream has no checks), **red** (the run
143
failed), **grey** (running), **up to date**. The overview shows the same in
144
brief; Admin › Diffusion shows the whole hub.
145
73ef9f76 Andy McCall ↶ 146
It is drawn as a two-half pill, like a capsule (Andy, 2026-10-13): the
147
left half is the upstream (**checked**, **untested**, **failing**,
148
**testing**), the right half the consumer (**passes**, **fails**,
149
**testing**, **waiting**, **up to date**), each coloured on its own, so a
150
game that passes on an untested upstream reads [untested │ passes] rather
151
than a single amber that hides the pass.
152
8620925e Andy McCall ↶ 153
## 5. Across hubs
154
155
Each hub only ever writes to its own repositories. What one hub tells
156
another is a hint; the consumer's hub fetches and verifies the facts.
157
158
- **Polling always works.** The consumer's hub has the upstream URL in
159
  `.oxymodules`; it asks for the followed bookmark every few minutes. An
160
  upstream kept in a Strata bucket with no hub is polled by reading its
161
  bookmark record.
162
- **Subscribing makes it prompt.** The consumer's hub asks the upstream's
163
  hub to notify it when the bookmark moves. Notes are signed with the
164
  upstream hub's key and carry only "repository *id* moved to state *S*";
165
  the consumer fetches and verifies. A forged note causes a fetch, nothing
166
  more; a lost one is caught by the next poll.
167
- **Private upstreams** are read with a read-only credential the
168
  consumer's owner stores as a secret on the consumer's hub.
169
- **"Green" across hubs** is evidence the consumer checks: a Flow receipt
170
  for S, signed by a runner the upstream hub vouches for. The consumer's
171
  policy names the upstream hub keys it accepts, and may also require
172
  every upstream change to be signed by the upstream's maintainers, which
173
  holds even if the upstream hub is compromised.
174
- **Consumer checks across hubs (§7)** are opt-in on both sides; the
175
  consumer's hub runs them on its own runners and returns only a signed
176
  summary, or nothing.
177
178
## 6. D2: partial pins and the culprit
179
180
A pin is a state: any set of changes closed under dependencies. So a
181
consumer can pin "upstream S, minus change X and everything built on X":
182
its heads are the maximal changes of `closure(S) − ({X} ∪ dependents(X))`,
183
well-formed by construction.
184
185
When a bump of N upstream changes fails:
186
187
1. Flow searches for the culprit by testing subsets of the N changes (a
188
   delta-debugging search over independent changes, bounded in runs).
189
2. The bump proposal then pins everything except the culprit and its
190
   dependents, and says which change is held back and why, with the
191
   failure.
192
3. The consumer gets the rest at once; the held-back change comes with a
193
   later bump, once upstream fixes it.
194
195
`oxy submodule` and checkout take such pins as they are (a pin is heads);
196
`oxy submodule status` names the changes a partial pin leaves out.
197
198
## 7. D3: consumers test upstream proposals
199
200
A proposal on R* API can run each consumer's pipeline with its pin swapped
201
for the proposal's projected state, committing nothing to the consumer
202
(the claim already carries subrepo pins; the run overrides one). Each
203
consumer's result appears on the upstream proposal as a check,
204
`consumers/<owner>/<name>`, with a summary: "Metal Gear ✔, Super Mario
205
Land ✘ (test X)". The person changing the API sees what breaks before it
206
lands, which is the monorepo's main strength without taking consumers'
207
control. An upstream may make consumer checks required.
208
209
## 8. D4: repair
210
211
- **Migrations with the change.** An upstream change that alters the API
212
  may carry a migration (a script, or instructions for an agent) under
213
  `.diffusion/migrations/`. A bump spanning it runs the migration in the
214
  consumer and adds the result as a separate change on the bump proposal.
215
- **An agent's fix.** When a bump is red, a scoped agent key (an agent
216
  home with `changes` scope on that consumer, never `refs`) may add a fix
217
  as a separate change to the proposal, for the consumer's owner to
218
  review. The bump and the fix stay separate changes, so either can be
219
  dropped.
220
221
## 9. D5: reach
222
223
- **Auto-land** (`land = "auto"`): a green bump lands itself when the
224
  consumer's trust policy and required checks allow it, and every
225
  upstream change is signed by keys the policy trusts.
226
- **Chains:** landing a bump moves the consumer's default bookmark, which
227
  may be followed in turn.
228
- **Grouping and quiet hours:** several upstreams' bumps in one proposal;
229
  no bumps outside set hours.
230
- **Cross-hub** polling and subscriptions (§5).
231
- **Channels:** with Cylinder, a consumer follows `rstar-api:stable`, a
232
  signed channel whose moves are gated by consumer checks (§7).
233
234
## 10. Order
235
236
| Step | What |
237
|---|---|
238
| D1 | The reverse pin index and Used by; turning it on; the signed bump on `diffusion/<path>`, moved in place; the proposal with the changelog |
239
| D2 | Partial pins; the culprit search |
240
| D3 | Consumer checks on upstream proposals |
241
| D4 | Migrations and agent fixes |
242
| D5 | Auto-land, chains, grouping, cross-hub, channels |
243
244
D2 is what no other tool can do; D3 is worth the most.
245
246
## 11. Open questions
247
248
- **Who may turn it on** for a consumer: its owner and administrators, or
249
  also the upstream's maintainers (by offering, which the consumer
250
  accepts)?
251
- **Bot identity:** a `diffusion` account per hub, or the upstream
252
  pusher's name with the hub as delegate?
253
- **Cost limits** for the culprit search and consumer checks: runs per
254
  bump, per upstream proposal, per day.