Diffusion lands a bump itself where a repository chose it f4d6ef418a96
Andy chose "opt-in, per repository" after rstar-api's first
green/green. land = auto in a subrepo's .oxymodules entry, else Settings
› Diffusion › Landing (and PUT …/diffusion {"land": "auto"}; migration
0061), off by default.
The hub lands a bump as the diffusion account only when both halves are
green: the exact upstream state the bump pins passed its own run, and
the consumer's newest run on the bump bookmark passed on the bump as it
is now. Never for an upstream with no checks. It lands through land_as,
signed by the hub's own key, so required checks, approvals and the trust
policy decide as for anyone; a refusal is noted once on the timeline,
and a landing reads "landed (Diffusion, both halves green)". Tried when
the bump's run passes, when the upstream turns green under a bump
already made, and when the setting turns to auto. The bump's text says
which way the repository lands.
Tested end to end: a consumer set to auto has its green bump landed and
its main moved; an untested upstream's passing bump waits; the entry
overrides the setting. No R* game has land set, so none lands itself.
closes #174
- Author
- Andy McCall @andymccall <[email protected]>
- When
- · 2026-10-14 01:28 UTC
- Signature
- ✔ signed by Andy McCall
- Hash
f4d6ef418a96aef348f721e3a7eaede7f21a77e79dc0c0c9681edf895e219864- Depends on
- 0b20f1ec4007 145306044896 19bb31ff1f5c 1f1eb06a9009 3776b70d9b17 42fa95540468 6ae3fa0af4f6 8620925e5c53 9008bf6c2b8c 93501da2a162 93be9ce4653e 9aaa88b2d9c8 a7e6e8e6094c c4e432da4bc9 cc19697f8871 d0fbea2eb46e d23464c0ac60 de505df5cf8d e71583501b1d f01cb865fc60 f677d6270298
- Depended on by
- dd0788b756f2 356183b71fab c7067c7bda56 33d98147511b 2a8b621676b4 3b1bc27fe805 5d90e0e045e2 a2ef53c5db26 7e01a7665aa5 e8fa53694f68 d3b048b27277 c2be148cfe83
- In bookmarks
- api-unknown-fields cancel-landed-runs diffusion-pin-bookmark draft-ready main oh-comment-body-file proposal-list-fast
- In tags
- v0.5.0 v0.5.1 v0.5.2 v0.6.0 v0.6.1 v0.7.0 v0.8.0 v0.9.0 v0.9.1 v0.9.11 v0.9.15 v0.9.16 v0.9.19 v0.9.2 v0.9.20 v0.9.21 v0.9.22 v0.9.23 v0.9.24 v0.9.3 v0.9.4 v0.9.5 v0.9.6 v0.9.7 v0.9.8 v0.9.9
- Edits
- 41 atoms, 1 tree operation, 12 files
Browse the tree at this change
added crates/hub/migrations/0061_diffusion_land.sql +5 −0
| 1 | + | -- Diffusion auto-land (docs/diffusion.md §4.2, Andy's choice 2026-10-13): |
|
| 2 | + | -- whether this repository lands a bump itself once both halves of its pill |
|
| 3 | + | -- are green. `auto`, `propose`, or empty for propose (off by default); a |
|
| 4 | + | -- subrepo's `.oxymodules` entry (`land = auto | propose`) overrides it. |
|
| 5 | + | ALTER TABLE repositories ADD COLUMN diffusion_land TEXT NOT NULL DEFAULT ''; |
modified crates/hub/src/api/mod.rs +19 −2
| … | 572 unchanged lines |
||
| 573 | 573 | struct DiffusionSetting { |
|
| 574 | 574 | /// `off`, `green`, `any`, or `default` (the site's). |
|
| 575 | 575 | follow: String, |
|
| 576 | + | /// `auto` or `propose`; absent keeps it as it is. |
|
| 577 | + | #[serde(default)] |
|
| 578 | + | land: Option<String>, |
|
| 576 | 579 | } |
|
| 577 | 580 | ||
| 578 | 581 | /// `PUT /repos/{owner}/{name}/diffusion {"follow": ...}`: whether this |
|
| … | 10 unchanged lines |
||
| 589 | 592 | "off" | "green" | "any" => d.follow.as_str(), |
|
| 590 | 593 | "default" => "", |
|
| 591 | 594 | _ => return Err(ApiError(StatusCode::BAD_REQUEST, "follow is off, green, any or default".into())), |
|
| 595 | + | }; |
|
| 596 | + | let land = match d.land.as_deref() { |
|
| 597 | + | Some(l @ ("auto" | "propose")) => Some(l), |
|
| 598 | + | None => None, |
|
| 599 | + | Some(_) => return Err(ApiError(StatusCode::BAD_REQUEST, "land is auto or propose".into())), |
|
| 592 | 600 | }; |
|
| 593 | 601 | db::repos::set_diffusion(&app.db, row.id, setting).await?; |
|
| 602 | + | if let Some(l) = land { |
|
| 603 | + | db::repos::set_diffusion_land(&app.db, row.id, l).await?; |
|
| 604 | + | } |
|
| 594 | 605 | db::accounts::audit( |
|
| 595 | 606 | &app.db, |
|
| 596 | 607 | c.account.as_ref(), |
|
| … | 10 unchanged lines |
||
| 607 | 618 | let (app, row) = (app.clone(), row.clone()); |
|
| 608 | 619 | tokio::spawn(async move { crate::diffusion::catch_up(&app, &row).await }); |
|
| 609 | 620 | } |
|
| 610 | - | Ok(Json(serde_json::json!({ "follow": rule.follow, "from": rule.from }))) |
|
| 621 | + | if land == Some("auto") { |
|
| 622 | + | let (app, row) = (app.clone(), row.clone()); |
|
| 623 | + | tokio::spawn(async move { crate::diffusion::land_ready(&app, &row).await }); |
|
| 624 | + | } |
|
| 625 | + | Ok(Json( |
|
| 626 | + | serde_json::json!({ "follow": rule.follow, "from": rule.from, "land": if row.diffusion_land == "auto" { "auto" } else { "propose" } }), |
|
| 627 | + | )) |
|
| 611 | 628 | } |
|
| 612 | 629 | ||
| 613 | 630 | /// `GET /repos/{owner}/{name}/consumers`: the repositories that pin this |
|
| … | 441 unchanged lines |
||
| 1055 | 1072 | let me = c.require("repo:write")?.clone(); |
|
| 1056 | 1073 | let (repo, row, _) = reach(&app, &c, &owner, &name, Access::Write).await?; |
|
| 1057 | 1074 | let via = format!("api (token {})", c.token); |
|
| 1058 | - | match crate::web::proposals::land_as(&app, &repo, &row, &me, number, &via).await { |
|
| 1075 | + | match crate::web::proposals::land_as(&app, &repo, &row, &me, number, &via, false).await { |
|
| 1059 | 1076 | Ok(_) => proposal(State(app), headers, Path((owner, name, number))).await, |
|
| 1060 | 1077 | Err(crate::web::proposals::Landing::NotFound) => Err(not_found("no such proposal")), |
|
| 1061 | 1078 | Err(crate::web::proposals::Landing::Refused(m)) => Err(ApiError(StatusCode::CONFLICT, m)), |
|
| … | 1443 unchanged lines |
modified crates/hub/src/db/repos.rs +9 −0
| … | 30 unchanged lines |
||
| 31 | 31 | /// (migration 0058, #148). |
|
| 32 | 32 | #[sqlx(default)] |
|
| 33 | 33 | pub diffusion: String, |
|
| 34 | + | /// Diffusion auto-land: `auto`, `propose`, or empty for propose |
|
| 35 | + | /// (migration 0061). |
|
| 36 | + | #[sqlx(default)] |
|
| 37 | + | pub diffusion_land: String, |
|
| 34 | 38 | /// Read-only, with why (migration 0040, #108). |
|
| 35 | 39 | #[sqlx(default)] |
|
| 36 | 40 | pub archived: bool, |
|
| … | 262 unchanged lines |
||
| 299 | 303 | /// `on`, `off`, or `` (follow hub.toml's `[flow] repos`). |
|
| 300 | 304 | pub async fn set_flow(pool: &Pool, id: i64, setting: &str) -> sqlx::Result<()> { |
|
| 301 | 305 | sqlx::query("UPDATE repositories SET flow = ? WHERE id = ?").bind(setting).bind(id).execute(pool).await?; |
|
| 306 | + | Ok(()) |
|
| 307 | + | } |
|
| 308 | + | ||
| 309 | + | pub async fn set_diffusion_land(pool: &Pool, id: i64, setting: &str) -> sqlx::Result<()> { |
|
| 310 | + | sqlx::query("UPDATE repositories SET diffusion_land = ? WHERE id = ?").bind(setting).bind(id).execute(pool).await?; |
|
| 302 | 311 | Ok(()) |
|
| 303 | 312 | } |
|
| 304 | 313 | ||
| … | 29 unchanged lines |
modified crates/hub/src/diffusion.rs +188 −7
| … | 688 unchanged lines |
||
| 689 | 689 | path: &str, |
|
| 690 | 690 | upstream: &RepoRow, |
|
| 691 | 691 | (from, to): (Vec<ChangeHash>, Vec<ChangeHash>), |
|
| 692 | - | (why, watch): (Why, Vec<String>), |
|
| 692 | + | (why, watch, lands): (Why, Vec<String>, bool), |
|
| 693 | 693 | ) -> Option<i64> { |
|
| 694 | 694 | use crate::db::proposals as pdb; |
|
| 695 | 695 | let bookmark = bump_bookmark(path); |
|
| … | 26 unchanged lines |
||
| 722 | 722 | String::new() |
|
| 723 | 723 | }; |
|
| 724 | 724 | let body = format!( |
|
| 725 | - | "Moves the subrepo `{path}` to [{slug}](/{slug})'s `{ub}`, {when} (`{}`). Only the pin changes; this repository's own checks test it.{amber}{flagged}\n\n**What it brings** ({count}):\n\n{}\n\nOpened and kept up to date by Diffusion (docs/diffusion.md). It never lands by itself.", |
|
| 725 | + | "Moves the subrepo `{path}` to [{slug}](/{slug})'s `{ub}`, {when} (`{}`). Only the pin changes; this repository's own checks test it.{amber}{flagged}\n\n**What it brings** ({count}):\n\n{}\n\nOpened and kept up to date by Diffusion (docs/diffusion.md). {}", |
|
| 726 | 726 | crate::web::state_label(&short.join("+")), |
|
| 727 | - | lines.join("\n") |
|
| 727 | + | lines.join("\n"), |
|
| 728 | + | if lands { |
|
| 729 | + | "This repository lands its bumps itself (`land = auto`): it lands once the upstream state it pins has passed its own checks and this repository's run on it passes, unless its required checks, approvals or trust policy say otherwise. An untested upstream always waits for a person." |
|
| 730 | + | } else { |
|
| 731 | + | "It never lands by itself." |
|
| 732 | + | } |
|
| 728 | 733 | ); |
|
| 729 | 734 | let open = pdb::list(&app.db, consumer.id, &["open", "draft"]).await.ok()?.into_iter().find(|p| p.source_bookmark == bookmark); |
|
| 730 | 735 | let (number, id) = match open { |
|
| … | 145 unchanged lines |
||
| 876 | 881 | Ok(Bumped::Moved { from, bookmark, heads: bumped, state, .. }) => { |
|
| 877 | 882 | // Its pipeline runs as for a push, so the proposal is tested. |
|
| 878 | 883 | tokio::spawn(start_runs(app.clone(), consumer.clone(), bookmark, state, bumped)); |
|
| 879 | - | let n = propose(app, &consumer, &target, &path, upstream, (from, heads.clone()), (why, r.watch.clone())).await; |
|
| 884 | + | let n = propose(app, &consumer, &target, &path, upstream, (from, heads.clone()), (why, r.watch.clone(), r.land_auto)).await; |
|
| 880 | 885 | tracing::info!( |
|
| 881 | 886 | "diffusion: {}/{} {path} bumped to {}/{} (proposal {n:?})", |
|
| 882 | 887 | consumer.owner, |
|
| … | 5 unchanged lines |
||
| 888 | 893 | Ok(Bumped::Already { from, to }) => { |
|
| 889 | 894 | // Already moved: its proposal's text and labels are brought |
|
| 890 | 895 | // up to date all the same. |
|
| 891 | - | propose(app, &consumer, &target, &path, upstream, (from, to), (why, r.watch.clone())).await; |
|
| 896 | + | propose(app, &consumer, &target, &path, upstream, (from, to), (why, r.watch.clone(), r.land_auto)).await; |
|
| 897 | + | // The upstream may just have turned green under a bump its |
|
| 898 | + | // consumer already passed. |
|
| 899 | + | if green && r.land_auto { |
|
| 900 | + | tokio::spawn(try_land(app.clone(), consumer.clone(), path.clone())); |
|
| 901 | + | } |
|
| 892 | 902 | } |
|
| 893 | 903 | Ok(other) => tracing::info!("diffusion: {}/{} {path}: no bump ({other:?})", consumer.owner, consumer.name), |
|
| 894 | 904 | Err(e) => tracing::warn!("diffusion: bump {}/{} {path}: {e}", consumer.owner, consumer.name), |
|
| 895 | 905 | } |
|
| 896 | 906 | } |
|
| 907 | + | } |
|
| 908 | + | ||
| 909 | + | /// Why a bump was not landed by itself (docs/diffusion.md §4.2). |
|
| 910 | + | #[derive(Debug, PartialEq, Eq)] |
|
| 911 | + | pub enum Held { |
|
| 912 | + | /// No open bump at that path. |
|
| 913 | + | NoBump, |
|
| 914 | + | /// The consumer did not choose `land = auto`. |
|
| 915 | + | Manual, |
|
| 916 | + | /// The upstream has no checks: amber never lands by itself. |
|
| 917 | + | Unverified, |
|
| 918 | + | /// The upstream state the bump pins has no passing run. |
|
| 919 | + | UpstreamUnchecked, |
|
| 920 | + | /// The consumer's run on the bump as it is now has not passed. |
|
| 921 | + | NotPassed, |
|
| 922 | + | /// The landing was refused (required checks, approvals, the trust policy). |
|
| 923 | + | Refused(String), |
|
| 897 | 924 | } |
|
| 898 | 925 | ||
| 926 | + | /// Land the open bump at `path` in `consumer` itself, as the `diffusion` |
|
| 927 | + | /// account, when the consumer chose `land = auto` (its `.oxymodules` entry, |
|
| 928 | + | /// else Settings › Diffusion) and both halves of its pill are green: the |
|
| 929 | + | /// upstream state it pins passed its own run (never amber: an upstream |
|
| 930 | + | /// with no checks always waits for a person), and the consumer's run on |
|
| 931 | + | /// the bump as it is now passed. It lands through the web's own path |
|
| 932 | + | /// (`land_as`), signed by the hub's key, so required checks, approvals and |
|
| 933 | + | /// the trust policy decide as for anyone; a refusal is noted once on the |
|
| 934 | + | /// proposal's timeline. The proposal's number when it landed. |
|
| 935 | + | pub async fn auto_land(app: &Shared, consumer: &RepoRow, path: &str) -> Result<i64, Held> { |
|
| 936 | + | let path = path.trim_matches('/'); |
|
| 937 | + | let bm = bump_bookmark(path); |
|
| 938 | + | let Some(cname) = RepoName::new(&consumer.owner, &consumer.name) else { return Err(Held::NoBump) }; |
|
| 939 | + | let bump = db::proposals::list(&app.db, consumer.id, &["open"]) |
|
| 940 | + | .await |
|
| 941 | + | .ok() |
|
| 942 | + | .and_then(|ps| ps.into_iter().find(|p| p.source_bookmark == bm && p.author == ACCOUNT)) |
|
| 943 | + | .ok_or(Held::NoBump)?; |
|
| 944 | + | let learned = target(&app.db, consumer).await; |
|
| 945 | + | let modules = oxymodules(app, &cname, &learned).await; |
|
| 946 | + | if !rule(&app.config.diffusion.follow, consumer, modules.as_deref(), path).land_auto { |
|
| 947 | + | return Err(Held::Manual); |
|
| 948 | + | } |
|
| 949 | + | let upstream_id: Option<i64> = sqlx::query_scalar( |
|
| 950 | + | "SELECT upstream_repo_id FROM diffusion_pins WHERE consumer_id = ? AND path = ? AND upstream_repo_id IS NOT NULL LIMIT 1", |
|
| 951 | + | ) |
|
| 952 | + | .bind(consumer.id) |
|
| 953 | + | .bind(path) |
|
| 954 | + | .fetch_optional(&app.db) |
|
| 955 | + | .await |
|
| 956 | + | .ok() |
|
| 957 | + | .flatten(); |
|
| 958 | + | let Some(upstream) = (match upstream_id { |
|
| 959 | + | Some(id) => db::repos::get_by_id(&app.db, id).await.ok().flatten(), |
|
| 960 | + | None => None, |
|
| 961 | + | }) else { |
|
| 962 | + | return Err(Held::UpstreamUnchecked); |
|
| 963 | + | }; |
|
| 964 | + | if unverified(app, &upstream).await { |
|
| 965 | + | return Err(Held::Unverified); |
|
| 966 | + | } |
|
| 967 | + | // What the bump pins, and the bump bookmark's heads now. |
|
| 968 | + | let (p, b) = (path.to_string(), bm.clone()); |
|
| 969 | + | let (pinned, now) = app |
|
| 970 | + | .repos |
|
| 971 | + | .read(&cname, move |r, cache| { |
|
| 972 | + | let heads = heads_of(r, &b)?; |
|
| 973 | + | if heads.is_empty() { |
|
| 974 | + | return Ok((None, heads)); |
|
| 975 | + | } |
|
| 976 | + | let loaded = cache.load(r, &heads)?; |
|
| 977 | + | let pin = match loaded.snapshot.files.get(&p).map(|f| &f.content) { |
|
| 978 | + | Some(oxy_core::state::FileContent::Opaque { blobs }) => match blobs.first() { |
|
| 979 | + | Some(blob) => oxy_core::subrepo::Pin::decode(&r.read_blob(blob)?).ok(), |
|
| 980 | + | None => None, |
|
| 981 | + | }, |
|
| 982 | + | _ => None, |
|
| 983 | + | }; |
|
| 984 | + | Ok((pin, heads)) |
|
| 985 | + | }) |
|
| 986 | + | .await |
|
| 987 | + | .map_err(|e| Held::Refused(e.to_string()))?; |
|
| 988 | + | let Some(oxy_core::subrepo::Pin::Oxy { heads: pinned, .. }) = pinned else { return Err(Held::UpstreamUnchecked) }; |
|
| 989 | + | let same = |text: &str, want: &[ChangeHash]| { |
|
| 990 | + | let mut a: Vec<String> = text.split('+').filter(|h| !h.is_empty()).map(str::to_string).collect(); |
|
| 991 | + | let mut b: Vec<String> = want.iter().map(|h| h.to_hex()).collect(); |
|
| 992 | + | a.sort(); |
|
| 993 | + | b.sort(); |
|
| 994 | + | a == b |
|
| 995 | + | }; |
|
| 996 | + | let upstream_runs: Vec<String> = sqlx::query_scalar( |
|
| 997 | + | "SELECT heads FROM flow_runs WHERE repo_id = ? AND bookmark = ? AND status = 'passed' ORDER BY id DESC LIMIT 100", |
|
| 998 | + | ) |
|
| 999 | + | .bind(upstream.id) |
|
| 1000 | + | .bind(&upstream.default_bookmark) |
|
| 1001 | + | .fetch_all(&app.db) |
|
| 1002 | + | .await |
|
| 1003 | + | .unwrap_or_default(); |
|
| 1004 | + | if !upstream_runs.iter().any(|h| same(h, &pinned)) { |
|
| 1005 | + | return Err(Held::UpstreamUnchecked); |
|
| 1006 | + | } |
|
| 1007 | + | let run: Option<(String, String)> = |
|
| 1008 | + | sqlx::query_as("SELECT status, heads FROM flow_runs WHERE repo_id = ? AND bookmark = ? ORDER BY id DESC LIMIT 1") |
|
| 1009 | + | .bind(consumer.id) |
|
| 1010 | + | .bind(&bm) |
|
| 1011 | + | .fetch_optional(&app.db) |
|
| 1012 | + | .await |
|
| 1013 | + | .ok() |
|
| 1014 | + | .flatten(); |
|
| 1015 | + | if !run.is_some_and(|(status, heads)| status == "passed" && same(&heads, &now)) { |
|
| 1016 | + | return Err(Held::NotPassed); |
|
| 1017 | + | } |
|
| 1018 | + | let me = account(&app.db).await.ok_or_else(|| Held::Refused("the diffusion account is missing".into()))?; |
|
| 1019 | + | let via = "Diffusion, both halves green"; |
|
| 1020 | + | match crate::web::proposals::land_as(app, &cname, consumer, &me, bump.number, via, true).await { |
|
| 1021 | + | Ok(_) => { |
|
| 1022 | + | tracing::info!("diffusion: {}/{} landed #{} itself (both halves green)", consumer.owner, consumer.name, bump.number); |
|
| 1023 | + | Ok(bump.number) |
|
| 1024 | + | } |
|
| 1025 | + | Err(crate::web::proposals::Landing::NotFound) => Err(Held::NoBump), |
|
| 1026 | + | Err(crate::web::proposals::Landing::Refused(why)) => { |
|
| 1027 | + | // Said once, so the timeline explains why a green bump waits. |
|
| 1028 | + | let said = db::proposals::events(&app.db, bump.id) |
|
| 1029 | + | .await |
|
| 1030 | + | .unwrap_or_default() |
|
| 1031 | + | .iter() |
|
| 1032 | + | .any(|e| e.event == "not landed by itself" && e.detail == why); |
|
| 1033 | + | if !said { |
|
| 1034 | + | db::proposals::add_event(&app.db, bump.id, ACCOUNT, "not landed by itself", &why).await.ok(); |
|
| 1035 | + | } |
|
| 1036 | + | Err(Held::Refused(why)) |
|
| 1037 | + | } |
|
| 1038 | + | } |
|
| 1039 | + | } |
|
| 1040 | + | ||
| 1041 | + | /// [`auto_land`] in the background, logged. Boxed as `Send`: a landing |
|
| 1042 | + | /// moves the target as a push would, which may bump and land again. |
|
| 1043 | + | pub fn try_land(app: Shared, consumer: RepoRow, path: String) -> std::pin::Pin<Box<dyn std::future::Future<Output = ()> + Send>> { |
|
| 1044 | + | Box::pin(async move { |
|
| 1045 | + | match auto_land(&app, &consumer, &path).await { |
|
| 1046 | + | Ok(_) | Err(Held::NoBump | Held::Manual) => {} |
|
| 1047 | + | Err(why) => tracing::info!("diffusion: {}/{} {path} waits for a person: {why:?}", consumer.owner, consumer.name), |
|
| 1048 | + | } |
|
| 1049 | + | }) |
|
| 1050 | + | } |
|
| 1051 | + | ||
| 1052 | + | /// Try every open bump of `consumer` (its setting turned to `auto`). |
|
| 1053 | + | pub async fn land_ready(app: &Shared, consumer: &RepoRow) { |
|
| 1054 | + | let paths: Vec<String> = sqlx::query_scalar("SELECT DISTINCT path FROM diffusion_pins WHERE consumer_id = ?") |
|
| 1055 | + | .bind(consumer.id) |
|
| 1056 | + | .fetch_all(&app.db) |
|
| 1057 | + | .await |
|
| 1058 | + | .unwrap_or_default(); |
|
| 1059 | + | for path in paths { |
|
| 1060 | + | tokio::spawn(try_land(app.clone(), consumer.clone(), path)); |
|
| 1061 | + | } |
|
| 1062 | + | } |
|
| 1063 | + | ||
| 899 | 1064 | /// Whether `upstream` has no checks of its own: Flow is off for it, or |
|
| 900 | 1065 | /// its default bookmark has never had a run. Its moves are then |
|
| 901 | 1066 | /// "unverified" (amber), never green. |
|
| … | 92 unchanged lines |
||
| 994 | 1159 | #[derive(Clone, Debug, PartialEq, Eq, Serialize)] |
|
| 995 | 1160 | pub struct Rule { |
|
| 996 | 1161 | pub follow: Follow, |
|
| 997 | - | /// Land a green bump itself (D5); D1 only proposes. |
|
| 1162 | + | /// Land a bump itself once both halves are green (`land = auto`, or |
|
| 1163 | + | /// the repository's setting); otherwise it only proposes. |
|
| 998 | 1164 | pub land_auto: bool, |
|
| 999 | 1165 | /// The bookmark to bump, when `.oxymodules` names one (else learned). |
|
| 1000 | 1166 | pub target: Option<String>, |
|
| … | 50 unchanged lines |
||
| 1051 | 1217 | None => (Follow::parse(site_default).unwrap_or(Follow::Off), "site"), |
|
| 1052 | 1218 | }, |
|
| 1053 | 1219 | }; |
|
| 1054 | - | Rule { follow, land_auto: land.unwrap_or(false), target, watch, from } |
|
| 1220 | + | // Landing by itself: the entry's word, else the setting; never by default. |
|
| 1221 | + | let land_auto = land.unwrap_or(row.diffusion_land == "auto"); |
|
| 1222 | + | Rule { follow, land_auto, target, watch, from } |
|
| 1055 | 1223 | } |
|
| 1056 | 1224 | ||
| 1057 | 1225 | #[cfg(test)] |
|
| … | 31 unchanged lines |
||
| 1089 | 1257 | diffusion: diffusion.into(), |
|
| 1090 | 1258 | ..Default::default() |
|
| 1091 | 1259 | } |
|
| 1260 | + | } |
|
| 1261 | + | ||
| 1262 | + | #[test] |
|
| 1263 | + | fn landing_by_itself_is_chosen_per_repository_and_per_entry() { |
|
| 1264 | + | let mut r = row("green"); |
|
| 1265 | + | assert!(!rule("off", &r, None, "rstar-api").land_auto, "never by default"); |
|
| 1266 | + | r.diffusion_land = "auto".into(); |
|
| 1267 | + | assert!(rule("off", &r, None, "rstar-api").land_auto, "the setting"); |
|
| 1268 | + | assert!(!rule("off", &r, Some(MODULES), "rstar-api").land_auto, "the entry's `land = propose` wins"); |
|
| 1269 | + | assert!(rule("off", &r, Some(MODULES), "other").land_auto, "an entry without `land` takes the setting"); |
|
| 1270 | + | r.diffusion_land = String::new(); |
|
| 1271 | + | let auto = MODULES.replace(r#"land = "propose""#, "land = auto"); |
|
| 1272 | + | assert!(rule("off", &r, Some(&auto), "rstar-api").land_auto); |
|
| 1092 | 1273 | } |
|
| 1093 | 1274 | ||
| 1094 | 1275 | #[test] |
|
| … | 32 unchanged lines |
modified crates/hub/src/flow.rs +5 −0
| … | 1594 unchanged lines |
||
| 1595 | 1595 | let (app, row) = (app.clone(), row.clone()); |
|
| 1596 | 1596 | tokio::spawn(async move { crate::diffusion::propagate(&app, &row, heads, true).await }); |
|
| 1597 | 1597 | } |
|
| 1598 | + | // A bump's run passed: a consumer that lands its bumps itself may |
|
| 1599 | + | // land it now (docs/diffusion.md §4.2). |
|
| 1600 | + | if let Some(path) = done.bookmark.strip_prefix("diffusion/").filter(|_| done.status == "passed") { |
|
| 1601 | + | tokio::spawn(crate::diffusion::try_land(app.clone(), row.clone(), path.to_string())); |
|
| 1602 | + | } |
|
| 1598 | 1603 | } |
|
| 1599 | 1604 | Ok(()) |
|
| 1600 | 1605 | } |
|
| … | 372 unchanged lines |
modified crates/hub/src/web/edit.rs +9 −0
| … | 29 unchanged lines |
||
| 30 | 30 | } |
|
| 31 | 31 | ||
| 32 | 32 | /// Everything needed to sign for the user, or why not. |
|
| 33 | + | /// The hub's own key, acting for itself: how Diffusion lands a bump |
|
| 34 | + | /// (docs/diffusion.md §4.2). A trust policy admits it only where it |
|
| 35 | + | /// authorises the hub's key to move the bookmark. |
|
| 36 | + | pub fn hub_signer(app: &Shared) -> Result<Signer, String> { |
|
| 37 | + | let (hub_key, hub_secret) = crate::keys::load_secret(&app.config.keys_dir()).map_err(|e| format!("the hub key is unavailable: {e}"))?; |
|
| 38 | + | let identity = app.hub_key.identity.clone(); |
|
| 39 | + | Ok(Signer { author: identity.clone(), hub_key, hub_secret, hub_identity: identity }) |
|
| 40 | + | } |
|
| 41 | + | ||
| 33 | 42 | pub async fn signer_for(app: &Shared, account_id: i64) -> Result<Signer, String> { |
|
| 34 | 43 | let author = delegated_identity(app, account_id).await.ok_or_else(|| { |
|
| 35 | 44 | "You have not delegated to this hub's key yet. From a checkout with your identity: download the hub key from /.well-known/oxyhub-key, run `oxy identity delegate add oxyhub-key.identity --scope changes,refs`, then link the new identity under Settings › Keys (`oxy identity export`).".to_string() |
|
| … | 240 unchanged lines |
modified crates/hub/src/web/proposals.rs +7 −3
| … | 1139 unchanged lines |
||
| 1140 | 1140 | let me = require(&viewer, &shell)?; |
|
| 1141 | 1141 | csrf(&viewer, &f.csrf, &shell)?; |
|
| 1142 | 1142 | let (repo, row, _) = lookup_role(&app, &shell, &viewer, &owner, &name, Access::Write).await?; |
|
| 1143 | - | match land_as(&app, &repo, &row, &me.account, number, "web").await { |
|
| 1143 | + | match land_as(&app, &repo, &row, &me.account, number, "web", false).await { |
|
| 1144 | 1144 | Ok(_) => Ok(back(&owner, &name, number, "conversation", Some("Landed.".into()), None)), |
|
| 1145 | 1145 | Err(Landing::NotFound) => Err(Error::not_found(shell)), |
|
| 1146 | 1146 | Err(Landing::Refused(m)) => Ok(back(&owner, &name, number, "conversation", None, Some(m))), |
|
| … | 10 unchanged lines |
||
| 1157 | 1157 | /// refused while it is not open or has blockers. Moves the target bookmark |
|
| 1158 | 1158 | /// (attested by the hub key on `me`'s behalf when they delegated to it), |
|
| 1159 | 1159 | /// then records, announces and audits the landing, saying how it was done |
|
| 1160 | - | /// (`via`: `web`, or `api (token NAME)`). Returns the new state. |
|
| 1160 | + | /// (`via`: `web`, or `api (token NAME)`). `as_hub` signs with the hub's own |
|
| 1161 | + | /// key rather than `me`'s delegation (Diffusion's auto-land). Returns the |
|
| 1162 | + | /// new state. |
|
| 1161 | 1163 | pub async fn land_as( |
|
| 1162 | 1164 | app: &Shared, |
|
| 1163 | 1165 | repo: &RepoName, |
|
| … | 1 unchanged lines |
||
| 1165 | 1167 | me: &accounts::Account, |
|
| 1166 | 1168 | number: i64, |
|
| 1167 | 1169 | via: &str, |
|
| 1170 | + | as_hub: bool, |
|
| 1168 | 1171 | ) -> Result<String, Landing> { |
|
| 1169 | 1172 | let (owner, name) = (row.owner.clone(), row.name.clone()); |
|
| 1170 | 1173 | let refused = |e: String| Landing::Refused(e); |
|
| … | 12 unchanged lines |
||
| 1183 | 1186 | // user's behalf, which a policy accepts when it lets the user update |
|
| 1184 | 1187 | // the target (spec §14.5); without one, only a policy-free repository |
|
| 1185 | 1188 | // can be landed here. |
|
| 1186 | - | let result = match super::edit::signer_for(app, me.id).await { |
|
| 1189 | + | let signer = if as_hub { super::edit::hub_signer(app) } else { super::edit::signer_for(app, me.id).await }; |
|
| 1190 | + | let result = match signer { |
|
| 1187 | 1191 | Ok(signer) => crate::repos::edits::move_bookmark(&app.repos, repo, target.clone(), heads.clone(), signer).await.map(|_| pl::state_ref(&heads).to_hex()), |
|
| 1188 | 1192 | Err(why) => { |
|
| 1189 | 1193 | app.repos |
|
| … | 556 unchanged lines |
modified crates/hub/src/web/repo_settings.rs +19 −9
| … | 743 unchanged lines |
||
| 744 | 744 | csrf_token: String, |
|
| 745 | 745 | /// `off`, `green`, `any`, or `default`. |
|
| 746 | 746 | diffusion: String, |
|
| 747 | + | /// `auto` or `propose` (absent keeps it as it is). |
|
| 748 | + | #[serde(default)] |
|
| 749 | + | land: String, |
|
| 747 | 750 | } |
|
| 748 | 751 | ||
| 749 | 752 | /// Whether this repository follows the upstreams it pins (Diffusion, |
|
| … | 11 unchanged lines |
||
| 761 | 764 | "off" | "green" | "any" => f.diffusion.as_str(), |
|
| 762 | 765 | "default" => "", |
|
| 763 | 766 | _ => return settings_page(&app, shell, row, role, None, Some("Pick off, green, any or the site default.".into())).await, |
|
| 767 | + | }; |
|
| 768 | + | let land = match f.land.as_str() { |
|
| 769 | + | "auto" | "propose" => Some(f.land.as_str()), |
|
| 770 | + | "" => None, |
|
| 771 | + | _ => return settings_page(&app, shell, row, role, None, Some("Pick propose or auto for landing.".into())).await, |
|
| 764 | 772 | }; |
|
| 765 | 773 | db::repos::set_diffusion(&app.db, row.id, setting).await.map_err(|e| Error::from_db(shell.clone(), e))?; |
|
| 766 | - | accounts::audit( |
|
| 767 | - | &app.db, |
|
| 768 | - | viewer.account(), |
|
| 769 | - | "repo.diffusion", |
|
| 770 | - | &format!("{owner}/{name}"), |
|
| 771 | - | if setting.is_empty() { "default" } else { setting }, |
|
| 772 | - | ) |
|
| 773 | - | .await |
|
| 774 | - | .ok(); |
|
| 774 | + | if let Some(l) = land { |
|
| 775 | + | db::repos::set_diffusion_land(&app.db, row.id, l).await.map_err(|e| Error::from_db(shell.clone(), e))?; |
|
| 776 | + | } |
|
| 777 | + | let what = |
|
| 778 | + | format!("{}{}", if setting.is_empty() { "default" } else { setting }, land.map(|l| format!(", land {l}")).unwrap_or_default()); |
|
| 779 | + | accounts::audit(&app.db, viewer.account(), "repo.diffusion", &format!("{owner}/{name}"), &what).await.ok(); |
|
| 775 | 780 | let row = db::repos::get_by_id(&app.db, row.id).await.map_err(|e| Error::from_db(shell.clone(), e))?.unwrap_or(row); |
|
| 781 | + | // Turned to land by itself: any bump already green lands now. |
|
| 782 | + | if land == Some("auto") { |
|
| 783 | + | let (app, row) = (app.clone(), row.clone()); |
|
| 784 | + | tokio::spawn(async move { crate::diffusion::land_ready(&app, &row).await }); |
|
| 785 | + | } |
|
| 776 | 786 | if crate::diffusion::rule(&app.config.diffusion.follow, &row, None, "").follow != crate::diffusion::Follow::Off { |
|
| 777 | 787 | let (app, row) = (app.clone(), row.clone()); |
|
| 778 | 788 | tokio::spawn(async move { crate::diffusion::catch_up(&app, &row).await }); |
|
| … | 186 unchanged lines |
modified crates/hub/tests/flow.rs +160 −0
| … | 3757 unchanged lines |
||
| 3758 | 3758 | assert_eq!(st, StatusCode::OK, "both go to x86_64: {body}"); |
|
| 3759 | 3759 | } |
|
| 3760 | 3760 | } |
|
| 3761 | + | ||
| 3762 | + | /// Diffusion auto-land (docs/diffusion.md §4.2; Andy chose "opt-in, per |
|
| 3763 | + | /// repository", 2026-10-13): a consumer that chose `land = auto` has a bump |
|
| 3764 | + | /// landed by itself, as the diffusion account, once both halves are green; |
|
| 3765 | + | /// a bump from an upstream with no checks (amber) always waits. |
|
| 3766 | + | #[tokio::test(flavor = "multi_thread")] |
|
| 3767 | + | async fn a_consumer_that_chose_auto_lands_green_bumps_itself() { |
|
| 3768 | + | let h = hub().await; |
|
| 3769 | + | db::accounts::create_account(&h.app.db, "andy", "[email protected]", "x", "salt", true).await.unwrap(); |
|
| 3770 | + | create(&h.app, "andy", "api").await; |
|
| 3771 | + | create(&h.app, "andy", "game").await; |
|
| 3772 | + | let hub_app = &h.app; |
|
| 3773 | + | // Run every queued job to a pass. |
|
| 3774 | + | let pass_all = move || async move { |
|
| 3775 | + | for _ in 0..20 { |
|
| 3776 | + | let Some(a) = claim(hub_app).await else { |
|
| 3777 | + | tokio::time::sleep(std::time::Duration::from_millis(30)).await; |
|
| 3778 | + | if !jobs_left(hub_app).await { |
|
| 3779 | + | break; |
|
| 3780 | + | } |
|
| 3781 | + | continue; |
|
| 3782 | + | }; |
|
| 3783 | + | let id = a["job_id"].as_i64().unwrap(); |
|
| 3784 | + | let (name, ph) = (a["job"]["name"].as_str().unwrap().to_string(), a["pipeline_hash"].as_str().unwrap().to_string()); |
|
| 3785 | + | assert_eq!( |
|
| 3786 | + | finish(hub_app, id, serde_json::json!({ "exit_code": 0, "duration_ms": 10, "receipt": receipt(&name, &ph, 0) })).await, |
|
| 3787 | + | StatusCode::NO_CONTENT |
|
| 3788 | + | ); |
|
| 3789 | + | } |
|
| 3790 | + | }; |
|
| 3791 | + | let api_work = tempfile::tempdir().unwrap(); |
|
| 3792 | + | let api = client(&format!("{}/andy/api", h.base), api_work.path()); |
|
| 3793 | + | write(api_work.path(), ".flow/pipeline.kdl", &pipeline("")); |
|
| 3794 | + | write(api_work.path(), "test.sh", "echo ok\n"); |
|
| 3795 | + | api.snapshot("t").unwrap(); |
|
| 3796 | + | api.commit("api v1", &[]).unwrap(); |
|
| 3797 | + | assert!(api.push("origin", false).unwrap().iter().all(|r| r.ok)); |
|
| 3798 | + | pass_all().await; |
|
| 3799 | + | let game_work = tempfile::tempdir().unwrap(); |
|
| 3800 | + | let game = client(&format!("{}/andy/game", h.base), game_work.path()); |
|
| 3801 | + | game.submodule_add(&format!("{}/andy/api", h.base), "deps/api", Some("main")).unwrap(); |
|
| 3802 | + | write(game_work.path(), ".flow/pipeline.kdl", &pipeline("")); |
|
| 3803 | + | write(game_work.path(), "test.sh", "echo ok\n"); |
|
| 3804 | + | game.snapshot("t").unwrap(); |
|
| 3805 | + | game.commit("game on api v1", &[]).unwrap(); |
|
| 3806 | + | assert!(game.push("origin", false).unwrap().iter().all(|r| r.ok)); |
|
| 3807 | + | pass_all().await; |
|
| 3808 | + | let game_row = db::repos::get(&h.app.db, "andy", "game").await.unwrap().unwrap(); |
|
| 3809 | + | for _ in 0..100 { |
|
| 3810 | + | let n: i64 = sqlx::query_scalar("SELECT COUNT(*) FROM diffusion_pins WHERE consumer_id = ?") |
|
| 3811 | + | .bind(game_row.id) |
|
| 3812 | + | .fetch_one(&h.app.db) |
|
| 3813 | + | .await |
|
| 3814 | + | .unwrap(); |
|
| 3815 | + | if n > 0 { |
|
| 3816 | + | break; |
|
| 3817 | + | } |
|
| 3818 | + | tokio::time::sleep(std::time::Duration::from_millis(30)).await; |
|
| 3819 | + | } |
|
| 3820 | + | let (st, body) = |
|
| 3821 | + | call(&h.app, "PUT", "/api/v1/repos/andy/game/diffusion", Some("site"), Some(r#"{"follow":"green","land":"auto"}"#.into())).await; |
|
| 3822 | + | assert_eq!(st, StatusCode::OK, "{body}"); |
|
| 3823 | + | assert!(body.contains(r#""land":"auto""#), "{body}"); |
|
| 3824 | + | let (st, _) = |
|
| 3825 | + | call(&h.app, "PUT", "/api/v1/repos/andy/game/diffusion", Some("site"), Some(r#"{"follow":"green","land":"soon"}"#.into())).await; |
|
| 3826 | + | assert_eq!(st, StatusCode::BAD_REQUEST); |
|
| 3827 | + | ||
| 3828 | + | // The upstream moves and passes: the bump is proposed, its run passes, |
|
| 3829 | + | // and it lands by itself. |
|
| 3830 | + | write(api_work.path(), "a2.c", "v2\n"); |
|
| 3831 | + | api.snapshot("t").unwrap(); |
|
| 3832 | + | api.commit("api v2: faster sprites", &[]).unwrap(); |
|
| 3833 | + | assert!(api.push("origin", false).unwrap().iter().all(|r| r.ok)); |
|
| 3834 | + | let mut landed = None; |
|
| 3835 | + | for _ in 0..100 { |
|
| 3836 | + | pass_all().await; |
|
| 3837 | + | let all = db::proposals::list(&h.app.db, game_row.id, &[]).await.unwrap(); |
|
| 3838 | + | if let Some(p) = all.into_iter().find(|p| p.source_bookmark == "diffusion/deps/api" && p.state == "landed") { |
|
| 3839 | + | landed = Some(p); |
|
| 3840 | + | break; |
|
| 3841 | + | } |
|
| 3842 | + | tokio::time::sleep(std::time::Duration::from_millis(50)).await; |
|
| 3843 | + | } |
|
| 3844 | + | let p = landed.expect("landed by itself once both halves were green"); |
|
| 3845 | + | assert!(p.body.contains("lands its bumps itself"), "{}", p.body); |
|
| 3846 | + | let events = db::proposals::events(&h.app.db, p.id).await.unwrap(); |
|
| 3847 | + | let landing = events.iter().find(|e| e.event == "landed").expect("a landing on the timeline"); |
|
| 3848 | + | assert_eq!(landing.actor, "diffusion"); |
|
| 3849 | + | assert!(landing.detail.contains("Diffusion, both halves green"), "{}", landing.detail); |
|
| 3850 | + | // The game's main now pins api v2. |
|
| 3851 | + | let mut behind = serde_json::Value::Null; |
|
| 3852 | + | for _ in 0..100 { |
|
| 3853 | + | let (_, list) = call(&h.app, "GET", "/api/v1/repos/andy/api/consumers", Some("site"), None).await; |
|
| 3854 | + | let list: Vec<serde_json::Value> = serde_json::from_str(&list).unwrap(); |
|
| 3855 | + | behind = list.iter().find(|c| c["name"] == "game").map(|c| c["behind"].clone()).unwrap_or_default(); |
|
| 3856 | + | if behind == 0 { |
|
| 3857 | + | break; |
|
| 3858 | + | } |
|
| 3859 | + | tokio::time::sleep(std::time::Duration::from_millis(50)).await; |
|
| 3860 | + | } |
|
| 3861 | + | assert_eq!(behind, 0, "main moved to the bump"); |
|
| 3862 | + | ||
| 3863 | + | // An upstream with no checks: its bump passes in the consumer, but |
|
| 3864 | + | // amber never lands by itself, whatever the setting. |
|
| 3865 | + | create(&h.app, "andy", "lib").await; |
|
| 3866 | + | create(&h.app, "andy", "app").await; |
|
| 3867 | + | let lib_work = tempfile::tempdir().unwrap(); |
|
| 3868 | + | let lib = client(&format!("{}/andy/lib", h.base), lib_work.path()); |
|
| 3869 | + | write(lib_work.path(), "lib.h", "v1\n"); |
|
| 3870 | + | lib.snapshot("t").unwrap(); |
|
| 3871 | + | lib.commit("lib v1", &[]).unwrap(); |
|
| 3872 | + | assert!(lib.push("origin", false).unwrap().iter().all(|r| r.ok)); |
|
| 3873 | + | let app_work = tempfile::tempdir().unwrap(); |
|
| 3874 | + | let app = client(&format!("{}/andy/app", h.base), app_work.path()); |
|
| 3875 | + | app.submodule_add(&format!("{}/andy/lib", h.base), "deps/lib", Some("main")).unwrap(); |
|
| 3876 | + | let modules = std::fs::read_to_string(app_work.path().join(".oxymodules")).unwrap(); |
|
| 3877 | + | write(app_work.path(), ".oxymodules", &format!("{modules}follow = green\nland = auto\n")); |
|
| 3878 | + | write(app_work.path(), ".flow/pipeline.kdl", &pipeline("")); |
|
| 3879 | + | write(app_work.path(), "test.sh", "echo ok\n"); |
|
| 3880 | + | app.snapshot("t").unwrap(); |
|
| 3881 | + | app.commit("app on lib", &[]).unwrap(); |
|
| 3882 | + | assert!(app.push("origin", false).unwrap().iter().all(|r| r.ok)); |
|
| 3883 | + | pass_all().await; |
|
| 3884 | + | let app_row = db::repos::get(&h.app.db, "andy", "app").await.unwrap().unwrap(); |
|
| 3885 | + | for _ in 0..100 { |
|
| 3886 | + | let n: i64 = sqlx::query_scalar("SELECT COUNT(*) FROM diffusion_pins WHERE consumer_id = ?") |
|
| 3887 | + | .bind(app_row.id) |
|
| 3888 | + | .fetch_one(&h.app.db) |
|
| 3889 | + | .await |
|
| 3890 | + | .unwrap(); |
|
| 3891 | + | if n > 0 { |
|
| 3892 | + | break; |
|
| 3893 | + | } |
|
| 3894 | + | tokio::time::sleep(std::time::Duration::from_millis(30)).await; |
|
| 3895 | + | } |
|
| 3896 | + | write(lib_work.path(), "lib.c", "v2\n"); |
|
| 3897 | + | lib.snapshot("t").unwrap(); |
|
| 3898 | + | lib.commit("lib v2", &[]).unwrap(); |
|
| 3899 | + | assert!(lib.push("origin", false).unwrap().iter().all(|r| r.ok)); |
|
| 3900 | + | let mut passed = false; |
|
| 3901 | + | for _ in 0..100 { |
|
| 3902 | + | pass_all().await; |
|
| 3903 | + | let s: Option<String> = sqlx::query_scalar( |
|
| 3904 | + | "SELECT status FROM flow_runs WHERE repo_id = ? AND bookmark = 'diffusion/deps/lib' ORDER BY id DESC LIMIT 1", |
|
| 3905 | + | ) |
|
| 3906 | + | .bind(app_row.id) |
|
| 3907 | + | .fetch_optional(&h.app.db) |
|
| 3908 | + | .await |
|
| 3909 | + | .unwrap(); |
|
| 3910 | + | if s.as_deref() == Some("passed") { |
|
| 3911 | + | passed = true; |
|
| 3912 | + | break; |
|
| 3913 | + | } |
|
| 3914 | + | tokio::time::sleep(std::time::Duration::from_millis(50)).await; |
|
| 3915 | + | } |
|
| 3916 | + | assert!(passed, "the app's run on the bump passed"); |
|
| 3917 | + | assert_eq!(oxygenhub::diffusion::auto_land(&h.app, &app_row, "deps/lib").await, Err(oxygenhub::diffusion::Held::Unverified)); |
|
| 3918 | + | let open = db::proposals::list(&h.app.db, app_row.id, &["open"]).await.unwrap(); |
|
| 3919 | + | assert!(open.iter().any(|p| p.source_bookmark == "diffusion/deps/lib"), "amber waits for a person"); |
|
| 3920 | + | } |
modified docs/diffusion.md +17 −2
| … | 75 unchanged lines |
||
| 76 | 76 | (amber): it never turns green, so its moves bump green followers too, and |
|
| 77 | 77 | the proposal is labelled `upstream unverified` with a note saying only the |
|
| 78 | 78 | consumer's pipeline tested it (decided by Andy, 2026-10-13). A passing run |
|
| 79 | - | of the upstream later clears it. `auto` lands a green bump itself when the consumer's |
|
| 80 | - | trust policy and required checks allow it; D1 only proposes. |
|
| 79 | + | of the upstream later clears it. |
|
| 81 | 80 | ||
| 81 | + | **Landing by itself** (built 2026-10-14, oxygenhub#174; Andy chose |
|
| 82 | + | "opt-in, per repository" after rstar-api's first green/green): `land = |
|
| 83 | + | auto` in the entry, else the repository's Settings › Diffusion › Landing |
|
| 84 | + | (`PUT …/diffusion {"land": "auto"}`), off by default. The hub lands a bump |
|
| 85 | + | itself, as the `diffusion` account, when both halves are green: the exact |
|
| 86 | + | upstream state the bump pins passed its own run, and the consumer's |
|
| 87 | + | newest run on the bump bookmark passed on the bump as it is now. Never |
|
| 88 | + | for amber: an upstream with no checks always waits for a person. It lands |
|
| 89 | + | through the web's own path (`land_as`), signed by the hub's own key, so |
|
| 90 | + | the consumer's required checks, approvals and trust policy decide as for |
|
| 91 | + | anyone; a policy that does not let the hub's key move the bookmark keeps |
|
| 92 | + | the bump waiting, and the proposal's timeline says why, once. It is tried |
|
| 93 | + | when the bump's run passes, when the upstream's run passes under a bump |
|
| 94 | + | already made, and when the setting is turned to `auto`. The timeline |
|
| 95 | + | records "landed (Diffusion, both halves green)". |
|
| 96 | + | ||
| 82 | 97 | ### 4.3 The bump |
|
| 83 | 98 | ||
| 84 | 99 | When an upstream's followed bookmark reaches a new state S (and, for |
|
| … | 146 unchanged lines |
modified docs/spec.md +1 −1
| … | 514 unchanged lines |
||
| 515 | 515 | - its key is the one the runner's token first signed with. The first signed receipt under a token binds that token to its key in `flow_runner_keys`. |
|
| 516 | 516 | ||
| 517 | 517 | Otherwise the job's result still stands, and the run page says why the receipt was not kept. The run page links each kept receipt, with its signer. The change page lists the receipts for the state whose only head is that change. `/{owner}/{name}/receipts/{attest}` downloads a signed receipt for `oxy receipt verify` or `import`. |
|
| 518 | - | - **Architecture** (OxygenFlow spec §11.2–§11.3, oxygenflow#36): a runner reports each image's architecture on its claims (`image_archs`), kept in `flow_images`; a job is offered only to a runner whose architecture matches its sandbox's `arch`, else its image's as known. The hub knows an image's architecture from its own files, read through its ext4 to its shell's ELF header (`jet::rootfs`) out of the kept chunks, when it is sent (the manifest's word is taken only when the image cannot be read) and, once at start, for every kept image that had none; an image nobody can read takes the architecture of the runner its first job passed on, and a runner's own report, read from the image it holds, wins over that (oxygenhub#173, Andy's "detect and learn"). Only an image nobody has read or run matches any runner. The queue says which architecture a waiting job asks for. **Images by hash** (oxygenflow#39, Jetstream): `flow rootfs push HASH` sends a rootfs image as the FastCDC chunks the hub lacks (`POST /flow/images/missing`, `PUT /flow/images/chunks/{hash}`, by a token that may write repositories) and then its manifest (`PUT /flow/images/{hash}`), which the hub accepts only when every chunk is held and they make an image with that hash; chunks are kept zstd-compressed under `<data>/images`, shared by every image, and `jet_images` lists what is held. Runners and accounts fetch with `GET /flow/images/{hash}` and `GET /flow/images/chunks/{hash}`. A runner whose claims say `fetches_images` is offered jobs whose image only the hub holds, and fetches it before its preflight. Each claim also carries the runner's `facts` (OxygenFlow spec §11.1, oxygenflow#40): free disk, load, speed score and class, kernel; the hub keeps the latest on the runner, shows them on Admin › Runners and in `GET /runners`. A job's finish carries the most disk and memory it took (`peak_disk_mib`, `peak_memory_mib`, oxygenflow#41); a new job's `need_disk_mib` is the most its last five runs of the same repository and name took, plus a quarter and 64 MiB, and it is offered only to a runner whose claim reports at least that much free disk; the waiting reason says so. **Ranking** (oxygenflow#42): a job that another free runner which polled in the last 10 s would do better (it holds the job's build cache, which counts most: claims list the caches a runner holds, `caches`, each `flow_core::api::cache_slot(repo, job)`, oxygenflow#51; or it holds the job's image while the claimant would fetch it; or it has a higher speed class) is held from the claimant for up to 15 s, then any runner that fits takes it; each job records why it ran where it did (`placed`: "holds its build cache, holds the image, idle, speed 100 (class 3)", with "after N s held for a better runner" when it was), shown on the run page, in the API and by `oh run view`. A run whose jobs pin an image held by neither the hub nor a runner sets a failing `pipeline/images` check on its state (#137), naming the images and how to send them (`flow rootfs push`), so a proposal pinning a new image shows it before it lands; a later run of the state turns it to passed once every image is held. A placement pass every 15 seconds works out why each ready queued job no runner can take waits, from what each online runner offered at its last claim (scope, tier, architecture, image held, labels, memory and CPUs, a recent decline), and shows it on the job (API `waiting`, `oh run view` and `oh run watch`, its panel open on the run page) and in its check's summary, so a proposal says why its check is stuck (#28). **Diffusion** (#148, `docs/diffusion.md`): a push that moves a repository's default bookmark, or the bookmark its work lands on (the most common target of its proposals landed in the last 90 days), records each Oxygen subrepo pin in the new state (`diffusion_pins`: consumer, bookmark, path, the upstream's Oxygen repository id, the pinned heads); a hub that had no index fills it at start. `GET /repos/{owner}/{name}/consumers` lists the consumers the caller may read, each with its pin and how many changes of this repository's default bookmark it lacks (`behind`, null when the pin names changes this repository does not hold), and the overview shows them under Diffuses to. `GET /repos/{owner}/{name}/diffusion` (#153) gives both directions, `takes_from` (each upstream the repository pins) and `diffuses_to` (each consumer the caller may read), every entry naming its `upstream` and `consumer`, its `path`, `bookmark`, `behind`, `follow`, open `bump`, the bump's newest `run` and `run_status`, and the pill's halves (`upstream_state`, `consumer_state`: `tone` green, amber, red, grey or off, `word`, `title`); hub-api's `Diffusion`. A repository that pins, or is pinned by, another on the hub has a **Diffusion tab** (#150): Diffuses to (each consumer's pin bookmark, how far behind, its status, its open bump and that bump's newest run, its last landed bump, how it follows) and Takes from (each upstream it pins, with the same); its overview shows both in brief, and Admin › Diffusion lists every upstream with its consumers. Each consumer's status is a two-half pill: the upstream on the left (checked when its newest default-bookmark run passed, untested when it has no checks, failing, or testing) and the consumer on the right (passes or fails for its run on the bump, testing, waiting, or up to date), each half coloured green, amber, red or grey; the API keeps one overall `colour` as well. Landing a proposal on the hub refreshes the pin index and propagates as a push does. A bump's run, started by the `diffusion` account, is given the upstream it bumps as a subrepo whatever that account may read (the consumer's own pin of it was committed by someone who may write the consumer); its other subrepos need read access as usual. A consumer follows a subrepo's upstream as its `.oxymodules` entry says (`follow = off | green | any`, `land = propose | auto`, `target` naming the bookmark to bump, `watch` listing paths whose upstream changes are flagged), else as its repository setting says (Settings › Diffusion, or `PUT /repos/{owner}/{name}/diffusion {"follow": ...}` by its owners and admins; migration 0058), else as `[diffusion] follow` in hub.toml says (default `off`). A **bump** moves a consumer's pin at a path to a newer upstream state: one change on the target bookmark's heads that replaces only the pin (oxy-core `build_pin_change`), signed with the hub's key, on the bookmark `diffusion/<path>`, which the hub owns and rebuilds on the target each time (a forced update). When the pin has changes the upstream's state lacks (say, from an upstream proposal not landed yet), the bump moves to both, so a consumer never loses changes it relies on; it is skipped when that combination would conflict, when the pin names changes the upstream does not hold, when the pin already holds the state, or when the bump bookmark already moves it there, and the hub logs why. A bump is **proposed** once per bump bookmark, by the hub's `diffusion` account (made on first use, never able to sign in), into the target, labelled `dependencies`, its body the upstream changes the pin lacks (newest first, linking to the upstream's change pages); a later bump edits the same open proposal. It is triggered when a push moves the upstream's default bookmark (for consumers following `any`) or when a run of that bookmark passes (for `green`). Diffusion never lands a proposal. An upstream with no checks of its own (Flow off, or no run ever of its default bookmark) is **unverified**: it never turns green, so its moves also bump consumers following `green`, and those proposals are marked amber with the `upstream unverified` label and a note in the body; a later bump from a passing run removes the label. A bump already in place refreshes its proposal's text and labels. A consumer's `watch` patterns (`*` within a path segment, `**` across them, `?` one character; a plain path covers everything below it) flag the upstream changes touching a matching file: they are listed first under "Touches what you watch", with the files, and marked ⚑ in the changelog. Turning Diffusion on for a consumer (the setting or the API) catches it up at once: it is bumped to each upstream's newest state whose default-bookmark run passed, or to its newest state when it follows every move. **Images page** (#146): Admin › Runners › Images lists every rootfs image a runner holds, the hub keeps, or a job ran on in the last 90 days (from each job's sandbox), newest use first: its size and architecture, where it is (the hub, each runner holding it, each runner fetching it from the swarm now), the repositories whose jobs used it, and who sent it; an image held nowhere is marked. **The swarm** (#144, oxygenflow docs/jetstream.md §8.1): a runner in an image swarm reports its peer address, its site and the BLAKE3 of its TLS certificate with each claim (`swarm`; without a certificate it is not in the swarm), kept in memory and shown on Admin › Runners. `POST /flow/swarm/announce {image, state}` (by a runner; `fetching`, `done` or `stopped`) returns the online runners in its site that hold the image or announced fetching it in the last 45 seconds, each with its certificate's fingerprint, which the fetcher pins its TLS connection to, and a ticket signed with the hub key (JSON naming the runner, its site and an expiry 15 minutes on, hex, then the signature) that peers check against `/.well-known/oxyhub-key` before serving chunks; a runner that has not reported an address is told to add `[runner.swarm]` (409). |
|
| 518 | + | - **Architecture** (OxygenFlow spec §11.2–§11.3, oxygenflow#36): a runner reports each image's architecture on its claims (`image_archs`), kept in `flow_images`; a job is offered only to a runner whose architecture matches its sandbox's `arch`, else its image's as known. The hub knows an image's architecture from its own files, read through its ext4 to its shell's ELF header (`jet::rootfs`) out of the kept chunks, when it is sent (the manifest's word is taken only when the image cannot be read) and, once at start, for every kept image that had none; an image nobody can read takes the architecture of the runner its first job passed on, and a runner's own report, read from the image it holds, wins over that (oxygenhub#173, Andy's "detect and learn"). Only an image nobody has read or run matches any runner. The queue says which architecture a waiting job asks for. **Images by hash** (oxygenflow#39, Jetstream): `flow rootfs push HASH` sends a rootfs image as the FastCDC chunks the hub lacks (`POST /flow/images/missing`, `PUT /flow/images/chunks/{hash}`, by a token that may write repositories) and then its manifest (`PUT /flow/images/{hash}`), which the hub accepts only when every chunk is held and they make an image with that hash; chunks are kept zstd-compressed under `<data>/images`, shared by every image, and `jet_images` lists what is held. Runners and accounts fetch with `GET /flow/images/{hash}` and `GET /flow/images/chunks/{hash}`. A runner whose claims say `fetches_images` is offered jobs whose image only the hub holds, and fetches it before its preflight. Each claim also carries the runner's `facts` (OxygenFlow spec §11.1, oxygenflow#40): free disk, load, speed score and class, kernel; the hub keeps the latest on the runner, shows them on Admin › Runners and in `GET /runners`. A job's finish carries the most disk and memory it took (`peak_disk_mib`, `peak_memory_mib`, oxygenflow#41); a new job's `need_disk_mib` is the most its last five runs of the same repository and name took, plus a quarter and 64 MiB, and it is offered only to a runner whose claim reports at least that much free disk; the waiting reason says so. **Ranking** (oxygenflow#42): a job that another free runner which polled in the last 10 s would do better (it holds the job's build cache, which counts most: claims list the caches a runner holds, `caches`, each `flow_core::api::cache_slot(repo, job)`, oxygenflow#51; or it holds the job's image while the claimant would fetch it; or it has a higher speed class) is held from the claimant for up to 15 s, then any runner that fits takes it; each job records why it ran where it did (`placed`: "holds its build cache, holds the image, idle, speed 100 (class 3)", with "after N s held for a better runner" when it was), shown on the run page, in the API and by `oh run view`. A run whose jobs pin an image held by neither the hub nor a runner sets a failing `pipeline/images` check on its state (#137), naming the images and how to send them (`flow rootfs push`), so a proposal pinning a new image shows it before it lands; a later run of the state turns it to passed once every image is held. A placement pass every 15 seconds works out why each ready queued job no runner can take waits, from what each online runner offered at its last claim (scope, tier, architecture, image held, labels, memory and CPUs, a recent decline), and shows it on the job (API `waiting`, `oh run view` and `oh run watch`, its panel open on the run page) and in its check's summary, so a proposal says why its check is stuck (#28). **Diffusion** (#148, `docs/diffusion.md`): a push that moves a repository's default bookmark, or the bookmark its work lands on (the most common target of its proposals landed in the last 90 days), records each Oxygen subrepo pin in the new state (`diffusion_pins`: consumer, bookmark, path, the upstream's Oxygen repository id, the pinned heads); a hub that had no index fills it at start. `GET /repos/{owner}/{name}/consumers` lists the consumers the caller may read, each with its pin and how many changes of this repository's default bookmark it lacks (`behind`, null when the pin names changes this repository does not hold), and the overview shows them under Diffuses to. `GET /repos/{owner}/{name}/diffusion` (#153) gives both directions, `takes_from` (each upstream the repository pins) and `diffuses_to` (each consumer the caller may read), every entry naming its `upstream` and `consumer`, its `path`, `bookmark`, `behind`, `follow`, open `bump`, the bump's newest `run` and `run_status`, and the pill's halves (`upstream_state`, `consumer_state`: `tone` green, amber, red, grey or off, `word`, `title`); hub-api's `Diffusion`. A repository that pins, or is pinned by, another on the hub has a **Diffusion tab** (#150): Diffuses to (each consumer's pin bookmark, how far behind, its status, its open bump and that bump's newest run, its last landed bump, how it follows) and Takes from (each upstream it pins, with the same); its overview shows both in brief, and Admin › Diffusion lists every upstream with its consumers. Each consumer's status is a two-half pill: the upstream on the left (checked when its newest default-bookmark run passed, untested when it has no checks, failing, or testing) and the consumer on the right (passes or fails for its run on the bump, testing, waiting, or up to date), each half coloured green, amber, red or grey; the API keeps one overall `colour` as well. Landing a proposal on the hub refreshes the pin index and propagates as a push does. A bump's run, started by the `diffusion` account, is given the upstream it bumps as a subrepo whatever that account may read (the consumer's own pin of it was committed by someone who may write the consumer); its other subrepos need read access as usual. A consumer follows a subrepo's upstream as its `.oxymodules` entry says (`follow = off | green | any`, `land = propose | auto`, `target` naming the bookmark to bump, `watch` listing paths whose upstream changes are flagged), else as its repository setting says (Settings › Diffusion, or `PUT /repos/{owner}/{name}/diffusion {"follow": ...}` by its owners and admins; migration 0058), else as `[diffusion] follow` in hub.toml says (default `off`). A **bump** moves a consumer's pin at a path to a newer upstream state: one change on the target bookmark's heads that replaces only the pin (oxy-core `build_pin_change`), signed with the hub's key, on the bookmark `diffusion/<path>`, which the hub owns and rebuilds on the target each time (a forced update). When the pin has changes the upstream's state lacks (say, from an upstream proposal not landed yet), the bump moves to both, so a consumer never loses changes it relies on; it is skipped when that combination would conflict, when the pin names changes the upstream does not hold, when the pin already holds the state, or when the bump bookmark already moves it there, and the hub logs why. A bump is **proposed** once per bump bookmark, by the hub's `diffusion` account (made on first use, never able to sign in), into the target, labelled `dependencies`, its body the upstream changes the pin lacks (newest first, linking to the upstream's change pages); a later bump edits the same open proposal. It is triggered when a push moves the upstream's default bookmark (for consumers following `any`) or when a run of that bookmark passes (for `green`). Diffusion lands a bump only where the consumer chose it (`land = auto` in the entry, else Settings › Diffusion › Landing or `PUT /repos/{owner}/{name}/diffusion {"land": "auto"}`, migration 0061; off by default, #174): once both halves are green (the upstream state it pins passed its own run, and the consumer's run on the bump as it is now passed), as the `diffusion` account through `land_as`, signed by the hub's key, so required checks, approvals and the trust policy decide; never for an unverified upstream; a refusal is noted once on the proposal's timeline. An upstream with no checks of its own (Flow off, or no run ever of its default bookmark) is **unverified**: it never turns green, so its moves also bump consumers following `green`, and those proposals are marked amber with the `upstream unverified` label and a note in the body; a later bump from a passing run removes the label. A bump already in place refreshes its proposal's text and labels. A consumer's `watch` patterns (`*` within a path segment, `**` across them, `?` one character; a plain path covers everything below it) flag the upstream changes touching a matching file: they are listed first under "Touches what you watch", with the files, and marked ⚑ in the changelog. Turning Diffusion on for a consumer (the setting or the API) catches it up at once: it is bumped to each upstream's newest state whose default-bookmark run passed, or to its newest state when it follows every move. **Images page** (#146): Admin › Runners › Images lists every rootfs image a runner holds, the hub keeps, or a job ran on in the last 90 days (from each job's sandbox), newest use first: its size and architecture, where it is (the hub, each runner holding it, each runner fetching it from the swarm now), the repositories whose jobs used it, and who sent it; an image held nowhere is marked. **The swarm** (#144, oxygenflow docs/jetstream.md §8.1): a runner in an image swarm reports its peer address, its site and the BLAKE3 of its TLS certificate with each claim (`swarm`; without a certificate it is not in the swarm), kept in memory and shown on Admin › Runners. `POST /flow/swarm/announce {image, state}` (by a runner; `fetching`, `done` or `stopped`) returns the online runners in its site that hold the image or announced fetching it in the last 45 seconds, each with its certificate's fingerprint, which the fetcher pins its TLS connection to, and a ticket signed with the hub key (JSON naming the runner, its site and an expiry 15 minutes on, hex, then the signature) that peers check against `/.well-known/oxyhub-key` before serving chunks; a runner that has not reported an address is told to add `[runner.swarm]` (409). |
|
| 519 | 519 | - **Variables** (oxygenhub#102): Administration, an organisation's settings and a repository's settings each hold `NAME=value` pairs (a name is a letter or `_` then letters, digits and `_`, up to 64, never `FLOW_*`; a value up to 4096 characters; up to 100 per level) under **Variables**, each page showing what it inherits and which inherited names it overrides. A claim carries the repository's effective set as `env`, inner levels overriding outer; Flow puts them in the job's environment under `FLOW_*` and the pipeline's own `env`, so they are part of the receipt. They are not secrets and are shown in full to whoever sees the page; secrets are mounted by Flow (OxygenFlow spec §9), never put in the environment. Changes are audited (`flow.variable.set`, `flow.variable.deleted`); `flow run --var NAME=VALUE` gives a local run the same. |
|
| 520 | 520 | - **Secrets** (oxygenhub#107; OxygenFlow spec §9): the same three levels hold **Secrets**, named as a pipeline's `secret "name" { mount-path "/run/secrets/…" }` names them (1 to 64 of `A-Z a-z 0-9 . _ -`, a value up to 64 KiB, up to 100 per level), typed once and sealed under `keys/secrets.key`; pages show names, when each was set and where it comes from, never a value. A claim carries only the secrets its job declares, each with its path and value, from the innermost level that sets the name; one no level sets is noted at the top of the job's log and left out, so the job fails on the missing file rather than running with a stand-in. Flow writes each as a file (0400, the job's user's, on an in-memory mount) before the job starts and never puts it in the environment, the receipt or an artifact. Log intake replaces each value, and its base64, with `***` before a record is stored. Changes are audited (`flow.secret.set`, `flow.secret.deleted`), never with the value. |
|
| 521 | 521 | - **Artifacts** (OxygenFlow spec §5.2): a job may declare artifacts with `output`. |
|
| … | 235 unchanged lines |
modified templates/repo_settings.html +8 −2
| … | 85 unchanged lines |
||
| 86 | 86 | <form class="card wide" id="diffusion" method="post" action="{{ repo.url }}/settings/diffusion"> |
|
| 87 | 87 | <input type="hidden" name="_csrf" value="{{ shell.csrf }}"> |
|
| 88 | 88 | <h2>Diffusion</h2> |
|
| 89 | - | <p>When a repository this one pins as a subrepo moves on, Diffusion opens a proposal here that moves the pin, tested by this repository's own pipeline, with the upstream's changes listed. It never lands anything by itself.</p> |
|
| 89 | + | <p>When a repository this one pins as a subrepo moves on, Diffusion opens a proposal here that moves the pin, tested by this repository's own pipeline, with the upstream's changes listed. By default it never lands anything by itself.</p> |
|
| 90 | 90 | <div class="choice"> |
|
| 91 | 91 | <label><input type="radio" name="diffusion" value="green"{% if row.diffusion == "green" %} checked{% endif %}> when the upstream's checks pass</label> |
|
| 92 | 92 | <label><input type="radio" name="diffusion" value="any"{% if row.diffusion == "any" %} checked{% endif %}> on every upstream move</label> |
|
| 93 | 93 | <label><input type="radio" name="diffusion" value="off"{% if row.diffusion == "off" %} checked{% endif %}> off</label> |
|
| 94 | 94 | <label><input type="radio" name="diffusion" value="default"{% if row.diffusion.is_empty() %} checked{% endif %}> site default ({{ diffusion_site }}, from <code>[diffusion] follow</code> in <code>hub.toml</code>)</label> |
|
| 95 | 95 | </div> |
|
| 96 | - | <p class="small mute">A subrepo's <code>.oxymodules</code> entry overrides this with <code>follow = green</code>, <code>any</code> or <code>off</code>; <code>target</code> names the bookmark to move (else the one this repository's proposals land on) and <code>watch</code> lists paths whose upstream changes are flagged.</p> |
|
| 96 | + | <h3>Landing</h3> |
|
| 97 | + | <div class="choice"> |
|
| 98 | + | <label><input type="radio" name="land" value="propose"{% if row.diffusion_land != "auto" %} checked{% endif %}> propose only: a person lands each bump</label> |
|
| 99 | + | <label><input type="radio" name="land" value="auto"{% if row.diffusion_land == "auto" %} checked{% endif %}> land by itself when both halves are green: the upstream state it pins passed its own checks and this repository's run on the bump passed</label> |
|
| 100 | + | </div> |
|
| 101 | + | <p class="small mute">An untested (amber) upstream always waits for a person. A landing by Diffusion still needs this repository's required checks and approvals, and a trust policy that lets the hub's key move the bookmark; otherwise the bump waits, and its timeline says why.</p> |
|
| 102 | + | <p class="small mute">A subrepo's <code>.oxymodules</code> entry overrides this with <code>follow = green</code>, <code>any</code> or <code>off</code>; <code>land = auto</code> or <code>propose</code> overrides the landing, <code>target</code> names the bookmark to move (else the one this repository's proposals land on) and <code>watch</code> lists paths whose upstream changes are flagged.</p> |
|
| 97 | 103 | <button type="submit">SAVE</button> |
|
| 98 | 104 | </form> |
|
| 99 | 105 | {% if role == crate::auth::Role::Admin %} |
|
| … | 125 unchanged lines |