OXYGENHUB
# a2b29af1ce88 oxygenhub / docs / diffusion.md
10.4 KiB · Markdown · last changed Diffusion roadmap: honest status, ecosystems and OSV alerts 22 days ago renderedsource blame history raw
8620925e Andy McCall ↶ 1
# Diffusion: upstream changes reach their consumers
2
a2b29af1 Andy McCall ↶ 3
Status: D1 is built, with auto-land (2026-10-14); D2–D5 are designed.
4
What is built, what comes next and in what order, with ecosystems and
5
OSV alerts, is in [diffusion-roadmap.md]diffusion-roadmap.md.
b20ddaf9 Andy McCall ↶ 6
The name **Diffusion** was confirmed by Andy on 2026-10-12: changes
7
diffuse from an upstream into the repositories that pin it.
8620925e Andy McCall ↶ 8
9
## 1. The problem
10
11
R* API is a subrepo of Metal Gear, Metal Gear 2, Super Mario Land and
12
others. When it changes, each consumer should take the change, test it,
13
and propose it when it works, with no one opening proposals by hand.
14
Renovate and Dependabot do this for package manifests; Diffusion does it
15
for Oxygen pins, as part of Oxygen, and goes further than they can.
16
17
## 2. Why Oxygen can do better
18
19
- **The dependency graph is exact.** An Oxygen pin names a repository id
20
  and a state, and is a verifiable object (spec §7.8). The hub already
21
  reads every pin in a state to give jobs their subrepos. Nothing is
22
  scanned or guessed.
23
- **A pin is a set of changes, and independent changes commute.** A
24
  consumer can take an upstream state minus one change and whatever is
25
  built on it (§6). Git tools can only pin one commit: all or nothing.
26
- **Bumps never need rebasing.** The pin change commutes with the
27
  consumer's own work; only a consumer moving the same pin itself
28
  conflicts, as an ordinary replace conflict.
29
- **Everything is signed.** The bump, the upstream changes and the CI
30
  receipts are signed objects, so a consumer can check what it takes, even
31
  from another hub.
32
33
## 3. The approaches compared
34
35
| Approach | Strength | Weakness |
36
|---|---|---|
37
| Bump proposals (Renovate, Dependabot) | Consumers stay in charge; auditable; works across hubs | Reactive: breakage is found after upstream ships |
38
| Monorepo at head (Google) | Upstream cannot land what breaks a consumer | Takes control from consumers; upstream sees all code; one hub only |
39
| Upstream-driven migrations (large-scale changes, codemods) | Handles breaking changes, not just detects them | Needs tooling per change |
40
| Consumer-gated promotion (contract testing) | Prevents breakage | Needs consumers' tests to run upstream |
41
42
Diffusion is bump proposals as the base, with consumer checks on upstream
43
proposals (§7), partial pins (§6) and repair (§8) on top.
44
45
## 4. D1: follow and propose
46
47
### 4.1 The reverse pin index
48
49
When a push moves a repository's default bookmark, the hub records each
50
Oxygen pin in the new state: consumer, path, the upstream repository id
51
and the pinned heads (`diffusion_pins`). That answers "who consumes R*
52
API, at which state" at once. An upstream repository's page shows its
53
consumers on this hub ("Used by"); the API gives
54
`GET /repos/{owner}/{name}/consumers`.
55
56
### 4.2 Turning it on
57
58
A consumer follows a subrepo when, in order of precedence:
59
60
1. its `.oxymodules` entry says so, beside the existing `url` and
61
   `bookmark` keys:
62
63
   ```toml
64
   [rstar-api]
65
   url = "https://oxygenhub.io/doomsdayone/rstar-api"
66
   bookmark = "main"
67
   follow = "green"   # off | green | any
68
   land = "propose"   # propose | auto
b20ddaf9 Andy McCall ↶ 69
   target = "develop" # optional: the bookmark to bump (default: learned, §4.3)
8620925e Andy McCall ↶ 70
   ```
71
72
2. else its repository setting (Settings › Subrepos), so an
73
   administrator can turn it on without a commit;
74
3. else the hub's default (`[diffusion] follow` in hub.toml, `off`).
75
76
`green` waits for the upstream state's required checks to pass; `any`
77
follows every move. `auto` lands a green bump itself when the consumer's
78
trust policy and required checks allow it; D1 only proposes.
79
80
### 4.3 The bump
81
b20ddaf9 Andy McCall ↶ 82
**Which bookmark of the consumer** (decided 2026-10-12): the bookmark its
83
work lands on. The hub learns it per consumer from where the
84
consumer's proposals land (the most common target of those landed in the
85
last 90 days), falling back to the default bookmark; a `target` key in
86
the consumer's `.oxymodules` entry overrides it. The pin index (§4.1)
87
records pins on that bookmark as well as the default. So a game whose
88
default is `main` but whose work lands on `develop` is bumped on
89
`develop`, with no setup, and follows if its workflow changes.
90
8620925e Andy McCall ↶ 91
When an upstream's followed bookmark reaches a new state S (and, for
92
`green`, S's required checks pass), for each consumer that follows it:
93
94
- **Skip** when the consumer's pin already holds S, or when S does not
95
  hold everything the current pin does (upstream moved backwards or was
96
  rewritten). Consumers only move forward.
b20ddaf9 Andy McCall ↶ 97
- **Build** one change on the target bookmark's heads that replaces the
8620925e Andy McCall ↶ 98
  pin's blob with a pin of S. The hub signs it with its delegate key, as
99
  it signs wiki edits, authored by the hub's `diffusion` account.
b20ddaf9 Andy McCall ↶ 100
- **Place** it on the bookmark `diffusion/<path>`: the target's heads
101
  plus the change. One bookmark per consumer and subrepo, moved in
8620925e Andy McCall ↶ 102
  place when upstream moves again, so there is never a pile of bumps.
b20ddaf9 Andy McCall ↶ 103
- **Propose** it, once: a proposal from `diffusion/<path>` to the target
8620925e Andy McCall ↶ 104
  bookmark, labelled `dependencies`, kept up to date as the bookmark
105
  moves. Its CI runs on the projected state like any proposal's.
106
107
A trust policy that does not admit the hub's key still gets the
108
proposal; it is landed from a client, as today.
109
110
### 4.4 What the proposal says
111
112
- The upstream, the old pin and the new state.
113
- **The exact changelog:** every upstream change in S and not in the old
114
  pin, with its author, whether it is signed, and the issues it closes or
115
  refers to.
116
- Its checks, as for any proposal. A red bump stays open, with the
117
  failing check; D2 then says which upstream change broke it.
118
119
## 5. Across hubs
120
121
Each hub only ever writes to its own repositories. What one hub tells
122
another is a hint; the consumer's hub fetches and verifies the facts.
123
124
- **Polling always works.** The consumer's hub has the upstream URL in
125
  `.oxymodules`; it asks for the followed bookmark every few minutes. An
126
  upstream kept in a Strata bucket with no hub is polled by reading its
127
  bookmark record.
128
- **Subscribing makes it prompt.** The consumer's hub asks the upstream's
129
  hub to notify it when the bookmark moves. Notes are signed with the
130
  upstream hub's key and carry only "repository *id* moved to state *S*";
131
  the consumer fetches and verifies. A forged note causes a fetch, nothing
132
  more; a lost one is caught by the next poll.
133
- **Private upstreams** are read with a read-only credential the
134
  consumer's owner stores as a secret on the consumer's hub.
135
- **"Green" across hubs** is evidence the consumer checks: a Flow receipt
136
  for S, signed by a runner the upstream hub vouches for. The consumer's
137
  policy names the upstream hub keys it accepts, and may also require
138
  every upstream change to be signed by the upstream's maintainers, which
139
  holds even if the upstream hub is compromised.
140
- **Consumer checks across hubs (§7)** are opt-in on both sides; the
141
  consumer's hub runs them on its own runners and returns only a signed
142
  summary, or nothing.
143
144
## 6. D2: partial pins and the culprit
145
146
A pin is a state: any set of changes closed under dependencies. So a
147
consumer can pin "upstream S, minus change X and everything built on X":
148
its heads are the maximal changes of `closure(S) − ({X} ∪ dependents(X))`,
149
well-formed by construction.
150
151
When a bump of N upstream changes fails:
152
153
1. Flow searches for the culprit by testing subsets of the N changes (a
154
   delta-debugging search over independent changes, bounded in runs).
155
2. The bump proposal then pins everything except the culprit and its
156
   dependents, and says which change is held back and why, with the
157
   failure.
158
3. The consumer gets the rest at once; the held-back change comes with a
159
   later bump, once upstream fixes it.
160
161
`oxy submodule` and checkout take such pins as they are (a pin is heads);
162
`oxy submodule status` names the changes a partial pin leaves out.
163
164
## 7. D3: consumers test upstream proposals
165
166
A proposal on R* API can run each consumer's pipeline with its pin swapped
167
for the proposal's projected state, committing nothing to the consumer
168
(the claim already carries subrepo pins; the run overrides one). Each
169
consumer's result appears on the upstream proposal as a check,
170
`consumers/<owner>/<name>`, with a summary: "Metal Gear ✔, Super Mario
171
Land ✘ (test X)". The person changing the API sees what breaks before it
172
lands, which is the monorepo's main strength without taking consumers'
173
control. An upstream may make consumer checks required.
174
175
## 8. D4: repair
176
177
- **Migrations with the change.** An upstream change that alters the API
178
  may carry a migration (a script, or instructions for an agent) under
179
  `.diffusion/migrations/`. A bump spanning it runs the migration in the
180
  consumer and adds the result as a separate change on the bump proposal.
181
- **An agent's fix.** When a bump is red, a scoped agent key (an agent
182
  home with `changes` scope on that consumer, never `refs`) may add a fix
183
  as a separate change to the proposal, for the consumer's owner to
184
  review. The bump and the fix stay separate changes, so either can be
185
  dropped.
186
187
## 9. D5: reach
188
189
- **Auto-land** (`land = "auto"`): a green bump lands itself when the
190
  consumer's trust policy and required checks allow it, and every
191
  upstream change is signed by keys the policy trusts.
192
- **Chains:** landing a bump moves the consumer's default bookmark, which
193
  may be followed in turn.
194
- **Grouping and quiet hours:** several upstreams' bumps in one proposal;
195
  no bumps outside set hours.
196
- **Cross-hub** polling and subscriptions (§5).
197
- **Channels:** with Cylinder, a consumer follows `rstar-api:stable`, a
198
  signed channel whose moves are gated by consumer checks (§7).
199
200
## 10. Order
201
202
| Step | What |
203
|---|---|
204
| D1 | The reverse pin index and Used by; turning it on; the signed bump on `diffusion/<path>`, moved in place; the proposal with the changelog |
205
| D2 | Partial pins; the culprit search |
206
| D3 | Consumer checks on upstream proposals |
207
| D4 | Migrations and agent fixes |
208
| D5 | Auto-land, chains, grouping, cross-hub, channels |
209
210
D2 is what no other tool can do; D3 is worth the most.
211
a2b29af1 Andy McCall ↶ 212
The full order, with the work that makes Diffusion safe to rely on
213
first and the ecosystem adapters and OSV alerts beside it, is in
214
[diffusion-roadmap.md]diffusion-roadmap.md §4.
215
8620925e Andy McCall ↶ 216
## 11. Open questions
217
218
- **Who may turn it on** for a consumer: its owner and administrators, or
219
  also the upstream's maintainers (by offering, which the consumer
220
  accepts)?
221
- **Bot identity:** a `diffusion` account per hub, or the upstream
222
  pusher's name with the hub as delegate?
223
- **Cost limits** for the culprit search and consumer checks: runs per
224
  bump, per upstream proposal, per day.