| 1 |
# Diffusion roadmap |
| 2 |
|
| 3 |
Status: 2026-10-14 (oxygenhub#199). This replaces the earlier draft in |
| 4 |
`oxygen/docs/diffusion_roadmap.md`. The design for each step is in |
| 5 |
[diffusion.md](diffusion.md); this page says what is **built**, what is |
| 6 |
**designed**, what is **planned**, and what comes **later**, and in what |
| 7 |
order. A claim here is only as strong as its label. |
| 8 |
|
| 9 |
## 1. What Diffusion is for |
| 10 |
|
| 11 |
Diffusion keeps the repositories that depend on each other up to date: |
| 12 |
when an upstream moves, each consumer gets a signed proposal that brings |
| 13 |
the move, tested by the consumer's own pipeline, with the exact list of |
| 14 |
changes it brings. |
| 15 |
|
| 16 |
Today it does this for **Oxygen subrepos**: code you own or follow |
| 17 |
across repositories, such as R\* API inside the games. That is where |
| 18 |
Oxygen can do what nothing else can (§5). It is not yet a replacement for |
| 19 |
Renovate or Dependabot on package registries; §3 B is how it becomes one. |
| 20 |
|
| 21 |
## 2. Built |
| 22 |
|
| 23 |
|
| 24 |
|
| 25 |
| The reverse pin index: who pins what, at which state, on which bookmark | #148 | |
| 26 |
| Following an upstream (`follow = any`, `green` or `off`, in `.oxymodules`, the repository or the site) | #148 | |
| 27 |
| The bump: one hub-signed change on `diffusion/<path>`, moved in place, proposed once, labelled `dependencies` | #148 | |
| 28 |
| The changelog in the proposal, with `watch` paths flagged | #148 | |
| 29 |
| The Diffusion tab, the two-half pills, Admin › Diffusion, the overview card | #150, #160 | |
| 30 |
| `GET /repos/{o}/{n}/consumers` and `/diffusion` | #153 | |
| 31 |
| Auto-land, opt-in per consumer (`land = auto`), only when the upstream is checked and the consumer's run passed | #174 | |
| 32 |
| Catching up: a consumer that turns following on gets its bump at once | #148 | |
| 33 |
| A landing moves the pin index and starts the target's pipeline, as a push does | #150, #197 | |
| 34 |
|
| 35 |
## 3. Roadmap |
| 36 |
|
| 37 |
### A. Trust the basics (next) |
| 38 |
|
| 39 |
Diffusion only earns trust if nothing is held silently. #197 (a landed |
| 40 |
state that never ran held every `green` consumer back) is the kind of |
| 41 |
fault this phase removes for good. |
| 42 |
|
| 43 |
- **A1. The guarantees, written down and tested.** Every move of a |
| 44 |
followed bookmark gets a run; a consumer only moves forward; one bump |
| 45 |
per consumer and path; every bump is signed; a held bump always says |
| 46 |
why. Each guarantee gets an end-to-end test. |
| 47 |
- **A2. Why is it held?** The hub already knows the reasons when it |
| 48 |
declines to auto-land (no bump, manual, upstream unverified, upstream |
| 49 |
state unchecked, consumer run not passed, landing refused). Show them |
| 50 |
on the Diffusion tab, the proposal and the API, with the action that |
| 51 |
clears each. |
| 52 |
- **A3. Confidence across consumers.** The hub sees every consumer's |
| 53 |
result, so a bump can say "this upstream state passes in 7 of 8 |
| 54 |
consumers", and an upstream can see which consumers it is about to |
| 55 |
break. No package-registry tool can know this. |
| 56 |
- **A4. Rollback.** Undo a landed bump in one action: a proposal that |
| 57 |
pins the previous state, with the failing run linked. |
| 58 |
- **A5. Landing rules for agents.** Today a token with `repo:write` may |
| 59 |
land, whatever key it signs with. Add per-bookmark landing rules |
| 60 |
(required reviews and checks) and make tokens marked as an agent's |
| 61 |
unable to land. D4 depends on this. |
| 62 |
- **A6. Numbers.** Publish, per hub and per repository: time from an |
| 63 |
upstream landing to its consumers' bumps proposed and landed, the share |
| 64 |
of bumps that land green, and bumps held and why. |
| 65 |
|
| 66 |
### B. Ecosystems |
| 67 |
|
| 68 |
Diffusion's model (index, follow, bump on `diffusion/...`, test, propose, |
| 69 |
land) does not depend on what a dependency is. An **ecosystem adapter** |
| 70 |
tells it how to read and move one kind. |
| 71 |
|
| 72 |
- **B1. The adapter contract.** An adapter answers four questions, and |
| 73 |
runs as a Flow job in the sandbox, so the hub never executes ecosystem |
| 74 |
code itself: |
| 75 |
1. *find*: which manifests in a state it owns; |
| 76 |
2. *list*: the dependencies they declare, with their current versions; |
| 77 |
3. *resolve*: the newest version allowed by the consumer's rules |
| 78 |
(exact, semver range, pinned digest); |
| 79 |
4. *apply*: rewrite the manifest and its lockfile to a version, as a |
| 80 |
change. |
| 81 |
Network access is through a registry mirror the site configures, never |
| 82 |
open. The bump, changelog, checks, pills and auto-land are Diffusion's |
| 83 |
own, the same for every ecosystem. |
| 84 |
- **B2. Cargo** (we use it): git dependencies on repositories of this |
| 85 |
hub and crates from a registry; `Cargo.lock` updated with |
| 86 |
`cargo update -p`; changelogs from the hub for hub repositories, from |
| 87 |
release notes otherwise. |
| 88 |
- **B3. Docker** (we use it): `FROM` lines and Compose images, by tag and |
| 89 |
by digest, from registries the site lists and from the hub's own |
| 90 |
images; a bump pins a digest. |
| 91 |
- **B4. Adapters from others.** A declarative adapter (a KDL file: |
| 92 |
manifest patterns, a command for each of the four steps, the image to |
| 93 |
run them in) that a repository, an organisation or the site adds, so |
| 94 |
npm, pip, Go, Maven and the rest can be added by the people who use |
| 95 |
them. We ship npm and pip as worked examples, not as the only ones. |
| 96 |
|
| 97 |
### C. Security: OSV alerts |
| 98 |
|
| 99 |
- **C1. Advisories in, offline.** The hub mirrors the OSV database |
| 100 |
(osv.dev's daily dumps for the ecosystems in use) on a schedule, so |
| 101 |
matching never sends a repository's dependencies to a third party. |
| 102 |
This matters for sovereign sites, which may also point the mirror at |
| 103 |
their own copy. |
| 104 |
- **C2. Matching.** Each consumer's dependencies (from the pin index and |
| 105 |
the adapters' *list*) are matched against the advisories, at every push |
| 106 |
and when the mirror updates. |
| 107 |
- **C3. Alerts.** A Security tab per repository and an organisation |
| 108 |
view: advisory, severity, the affected dependency and path, the fixed |
| 109 |
version. Owners are notified (`security.alert`, "🚨 Security Alert"). |
| 110 |
- **C4. Security bumps.** A bump to the lowest fixed version, labelled |
| 111 |
`security`, ignoring quiet hours and grouping, and offered for |
| 112 |
auto-land when the consumer allows it. |
| 113 |
- **C5. Advisories out.** An upstream on the hub can publish its own |
| 114 |
advisories (OSV JSON in the repository or from its Security tab); the |
| 115 |
hub serves them in OSV format, and every consumer on the hub is |
| 116 |
alerted at once through the pin index, including for Oxygen subrepos, |
| 117 |
which no public database covers. |
| 118 |
|
| 119 |
### D. Designed, not built |
| 120 |
|
| 121 |
- **D2. Partial pins and the culprit** ([diffusion.md §6](diffusion.md)): |
| 122 |
pin an upstream state minus one change and what is built on it, with |
| 123 |
no rewriting; find the change that broke a consumer by testing |
| 124 |
subsets. A job's path scope may *order* the search (changes inside the |
| 125 |
failing job's scope first) but never *prunes* it: a test can depend on |
| 126 |
code outside its scope. |
| 127 |
- **D3. Consumers test upstream proposals** (§7): a check per consumer on |
| 128 |
the upstream proposal, run with the consumer's pin swapped for the |
| 129 |
proposal's projected state, committing nothing. |
| 130 |
- **D4. Repair** (§8): migrations shipped with an upstream change, and an |
| 131 |
agent's fix as a separate change on the bump, under the landing rules |
| 132 |
of A5. |
| 133 |
|
| 134 |
### E. Reach |
| 135 |
|
| 136 |
- **E1. Follow releases and tags**, with semver rules, not only |
| 137 |
bookmarks. |
| 138 |
- **E2. Grouping and quiet hours.** |
| 139 |
- **E3. Chains:** a landed bump moves a bookmark that others follow. |
| 140 |
- **E4. Across hubs** ([diffusion.md §5](diffusion.md)): polling first, |
| 141 |
signed notes to make it prompt, evidence of "green" checked by the |
| 142 |
consumer's hub. |
| 143 |
- **E5. Reusing a receipt** instead of re-running a job, for jobs marked |
| 144 |
hermetic, when a signed receipt exists for the same pipeline, scope |
| 145 |
tree and environment. To be measured before it is promised. |
| 146 |
|
| 147 |
### Later |
| 148 |
|
| 149 |
These need Cylinder (an artifact repository, an idea not yet started) or |
| 150 |
are research; none is promised: |
| 151 |
|
| 152 |
- signed channels (`rstar-api:stable`) whose moves are gated by consumer |
| 153 |
checks; |
| 154 |
- deployment leases that keep pinned states from being collected; |
| 155 |
- requiring N independent runners to reproduce a result before a bump |
| 156 |
auto-lands. |
| 157 |
|
| 158 |
## 4. Order |
| 159 |
|
| 160 |
A1–A6, then B1–B3 with C1–C4 alongside (the adapters' *list* is what |
| 161 |
OSV matching needs), then D3, D2, B4, C5, D4, then E. A and C are what |
| 162 |
make Diffusion safe to rely on; B is what makes it useful beyond Oxygen |
| 163 |
subrepos; D is what makes it unique. |
| 164 |
|
| 165 |
## 5. Compared honestly |
| 166 |
|
| 167 |
|
| 168 |
|
| 169 |
| Ecosystems | Very many | Not applicable | Oxygen subrepos only | Cargo, Docker, adapters for the rest | |
| 170 |
| Security alerts | Yes (advisory databases) | Tooling of your own | No | OSV, mirrored offline, and advisories for hub repositories | |
| 171 |
| Breakage found | After upstream ships | Before landing | After upstream lands, in the bump | Before upstream lands (D3), for consumers on the hub with tests | |
| 172 |
| Partial updates | Revert in a fork, by hand | Not applicable | No | Yes, without rewriting (D2) | |
| 173 |
| Rebasing bumps | Often | Not applicable | Never: one bump per path, moved in place | Same | |
| 174 |
| Confidence | Adoption statistics from public data | Everything tested together | Pills per consumer | Results across every consumer (A3) | |
| 175 |
| Provenance | Commit signatures where used | Usually unsigned | Signed bump, signed changes, signed receipts | Same, checked across hubs (E4) | |
| 176 |
| Sovereignty | Hosted service or self-hosted bot | One central repository | Your hub, your runners | Plus an offline advisory mirror (C1) | |
| 177 |
|
| 178 |
## 6. The earlier draft, corrected |
| 179 |
|
| 180 |
The draft in `oxygen/docs/diffusion_roadmap.md` described several things |
| 181 |
as if they existed. For the record: |
| 182 |
|
| 183 |
- An agent key without `refs` scope cannot sign references, but on the |
| 184 |
hub landing is decided by the token, so an agent token with |
| 185 |
`repo:write` could land. A5 fixes this. |
| 186 |
- Path scope cannot exclude upstream changes from a culprit search (D2). |
| 187 |
- Receipt reuse, `reproduced(N)`, channels, leases and `artifact(...)` |
| 188 |
revsets do not exist (E5 and Later). |
| 189 |
- `flow-casfs` is a FUSE mount; sub-second job starts are not measured. |
| 190 |
- Serial numbers are enforced for trust policies; cross-hub bookmark |
| 191 |
moves are not built (E4). |
| 192 |
- Bumps can conflict: two moves of the same pin edit the same |
| 193 |
`.oxymodules` entry. They never need rebasing because each path has one |
| 194 |
bump, moved in place. |