Refs answers carry the transparency log (step A3) e5b97ac399e7
The hub adds its latest checkpoint to every refs answer, and for each bookmark's attestation its log entry and an inclusion proof against that checkpoint (oxy-server's add_log hook; Oxygen spec §16). An attestation it serves but never logged, as pushed before the log began, is logged then and proved on a later fetch. Test: a fetch checks the log and pins it; a later checkpoint is proved from the tiles to extend it; a push not yet in a checkpoint is a note; a log that does not extend what the user saw fails the fetch as two histories. Pins oxygen ab796c92. Refs #238
- Author
- Andy McCall @andymccall <[email protected]>
- via
- agent
agent:claude@andymccall <[email protected]> · keyless, under your grant for changes,refs,seal · key 96d114fa75fd
- When
- · 2026-10-17 23:56 UTC
- Signature
- ◈ sealed by agent:claude@andymccall
- Hash
e5b97ac399e7075b76671fe2be870e620452fc72ba9237228c008fa7aaec77a5- Depends on
- 19bb31ff1f5c 380996db84e5 3eb491e9489b 42fa95540468 93be9ce4653e 965b950f9b96 d23464c0ac60 e271a37b629a
- Depended on by
- f87a74fffc87 ba0a55bb5b3c ec84907043c2 b5c199bd6b1a 0ee5c3fb32eb 13b27cb2200c 4f40eed0588f 7dbb41b09a99
- In bookmarks
- api-unknown-fields cancel-landed-runs diffusion-pin-bookmark draft-ready main oh-comment-body-file proposal-list-fast
- In tags
- v0.6.0 v0.6.1 v0.7.0 v0.8.0 v0.9.0 v0.9.1 v0.9.11 v0.9.15 v0.9.16 v0.9.19 v0.9.2 v0.9.20 v0.9.21 v0.9.22 v0.9.23 v0.9.24 v0.9.3 v0.9.4 v0.9.5 v0.9.6 v0.9.7 v0.9.8 v0.9.9
- Edits
- 7 atoms, 0 tree operations, 5 files
Browse the tree at this change
modified crates/hub/src/protocol.rs +9 −0
| … | 251 unchanged lines |
||
| 252 | 252 | self.row(&parsed).ok().map(|r| r.default_bookmark) |
|
| 253 | 253 | } |
|
| 254 | 254 | ||
| 255 | + | fn add_log(&self, name: &str, refs: &mut oxy_core::protocol::RefsResponse) { |
|
| 256 | + | let Some(app) = self.app_hint.lock().unwrap().as_ref().and_then(|w| w.upgrade()) else { return }; |
|
| 257 | + | let row = RepoName::parse(name).and_then(|p| self.row(&crate::wiki::parent_of(&p).0).ok()).map(|r| r.id); |
|
| 258 | + | let backfill = crate::tlog::add_log(&app, row, refs); |
|
| 259 | + | if !backfill.is_empty() { |
|
| 260 | + | self.block(crate::tlog::log(&app, backfill)); |
|
| 261 | + | } |
|
| 262 | + | } |
|
| 263 | + | ||
| 255 | 264 | fn after_push(&self, name: &str, report: &PushReport, headers: &HeaderMap) { |
|
| 256 | 265 | let Some(parsed) = RepoName::parse(name) else { return }; |
|
| 257 | 266 | // A wiki push starts no run and rings no webhook. |
|
| … | 201 unchanged lines |
modified crates/hub/src/tlog.rs +37 −0
| … | 141 unchanged lines |
||
| 142 | 142 | Ok(b) |
|
| 143 | 143 | } |
|
| 144 | 144 | ||
| 145 | + | /// Entry `n`'s bytes. |
|
| 146 | + | pub fn entry(&self, n: u64) -> anyhow::Result<Vec<u8>> { |
|
| 147 | + | let f = self.files.lock().unwrap(); |
|
| 148 | + | anyhow::ensure!(n < f.size, "entry {n} is not in the log"); |
|
| 149 | + | Self::entry_bytes(&f, n) |
|
| 150 | + | } |
|
| 151 | + | ||
| 145 | 152 | /// Entries in the log. |
|
| 146 | 153 | pub fn size(&self) -> u64 { |
|
| 147 | 154 | self.files.lock().unwrap().size |
|
| … | 204 unchanged lines |
||
| 352 | 359 | add(&mut out, "policy", r, sig_keys(&sp.sigs)); |
|
| 353 | 360 | } |
|
| 354 | 361 | out |
|
| 362 | + | } |
|
| 363 | + | ||
| 364 | + | /// Add the log to a refs answer (Oxygen spec §14.8): the latest |
|
| 365 | + | /// checkpoint, and for each bookmark's attestation its entry and inclusion |
|
| 366 | + | /// proof against it. An attestation the hub serves but never logged (one |
|
| 367 | + | /// pushed before the log began) is logged now, and proved on a later fetch. |
|
| 368 | + | pub fn add_log(app: &Shared, row: Option<i64>, refs: &mut oxy_core::protocol::RefsResponse) -> Vec<Pending> { |
|
| 369 | + | let mut backfill = Vec::new(); |
|
| 370 | + | let Some(note) = app.tlog.checkpoint() else { return backfill }; |
|
| 371 | + | let Some(size) = tlog::Note::parse(¬e).ok().and_then(|n| Checkpoint::parse(&n.text).ok()).map(|c| c.size) else { |
|
| 372 | + | return backfill; |
|
| 373 | + | }; |
|
| 374 | + | refs.log = Some(note); |
|
| 375 | + | for r in refs.refs.iter_mut() { |
|
| 376 | + | let Some(a) = &r.attestation else { continue }; |
|
| 377 | + | let object = a.attest.hash(); |
|
| 378 | + | let idx = tokio::task::block_in_place(|| { |
|
| 379 | + | tokio::runtime::Handle::current().block_on(db::tlog::entry_of(&app.db, &object.to_hex(), "ref")) |
|
| 380 | + | }); |
|
| 381 | + | match idx { |
|
| 382 | + | Ok(Some(i)) if (i as u64) < size => { |
|
| 383 | + | let (Ok(entry), Ok(proof)) = (app.tlog.entry(i as u64), app.tlog.prove_inclusion(size, i as u64)) else { continue }; |
|
| 384 | + | r.logged = Some(oxy_core::protocol::Logged { index: i as u64, entry, proof }); |
|
| 385 | + | } |
|
| 386 | + | Ok(Some(_)) => {} |
|
| 387 | + | Ok(None) => backfill.push(Pending { kind: "ref", repo: a.attest.repo, repo_row: row, object, signers: sig_keys(&a.sigs) }), |
|
| 388 | + | Err(e) => eprintln!("tlog: looking up {}: {e}", &object.to_hex()[..12]), |
|
| 389 | + | } |
|
| 390 | + | } |
|
| 391 | + | backfill |
|
| 355 | 392 | } |
|
| 356 | 393 | ||
| 357 | 394 | /// Sign a new checkpoint every few seconds while entries arrive. |
|
| … | 120 unchanged lines |
modified crates/hub/tests/flow.rs +62 −0
| … | 5020 unchanged lines |
||
| 5021 | 5021 | assert!(page.contains("Transparency log") && page.contains("andy/demo") && page.contains(&origin), "{page}"); |
|
| 5022 | 5022 | } |
|
| 5023 | 5023 | ||
| 5024 | + | /// Fetching checks the hub's log (oxygenhub#238, step A3): the checkpoint |
|
| 5025 | + | /// comes with the refs, each attestation with its entry and inclusion |
|
| 5026 | + | /// proof; a later checkpoint is proved to extend the one seen before, from |
|
| 5027 | + | /// the hub's tiles; an attestation not in a checkpoint yet is a note; and a |
|
| 5028 | + | /// log that does not extend what this user saw fails the fetch. |
|
| 5029 | + | #[tokio::test(flavor = "multi_thread")] |
|
| 5030 | + | async fn a_fetch_checks_the_hub_s_log_and_refuses_a_fork() { |
|
| 5031 | + | use oxy_core::tlog::{Checkpoint, Note}; |
|
| 5032 | + | let h = hub().await; |
|
| 5033 | + | create(&h.app, "andy", "demo").await; |
|
| 5034 | + | let url = format!("{}/andy/demo", h.base); |
|
| 5035 | + | let work = tempfile::tempdir().unwrap(); |
|
| 5036 | + | let mut a = client(&url, &work.path().join("a")); |
|
| 5037 | + | a.create_identity("Andy", "[email protected]", oxy_core::repo::IdentityScope::Repo).unwrap(); |
|
| 5038 | + | let (_, sk) = oxygenhub::keys::load_secret(&h.app.config.keys_dir()).unwrap(); |
|
| 5039 | + | let hub_id = h.app.hub_key.identity.clone(); |
|
| 5040 | + | let checkpoint = || h.app.tlog.refresh_checkpoint(&hub_id, &sk).unwrap(); |
|
| 5041 | + | let mut n = 0; |
|
| 5042 | + | let mut push = |a: &Repo| { |
|
| 5043 | + | n += 1; |
|
| 5044 | + | write(&work.path().join("a"), "a.txt", &format!("{n}\n")); |
|
| 5045 | + | a.commit(&format!("change {n}"), &[]).unwrap(); |
|
| 5046 | + | assert!(a.push("origin", false).unwrap().iter().all(|r| r.ok)); |
|
| 5047 | + | }; |
|
| 5048 | + | push(&a); |
|
| 5049 | + | checkpoint(); |
|
| 5050 | + | ||
| 5051 | + | let home = work.path().join("home-b"); |
|
| 5052 | + | let b = Repo::clone(&url, &work.path().join("b")).unwrap().with_home(home.clone()); |
|
| 5053 | + | let report = b.fetch("origin").unwrap(); |
|
| 5054 | + | assert!(report.log_notes.is_empty(), "{:?}", report.log_notes); |
|
| 5055 | + | let pin = std::fs::read_dir(home.join("tlog")).unwrap().next().unwrap().unwrap().path(); |
|
| 5056 | + | let seen = |p: &std::path::Path| { |
|
| 5057 | + | let t = std::fs::read_to_string(p).unwrap(); |
|
| 5058 | + | Checkpoint::parse(&Note::parse(t.split_once('\n').unwrap().1).unwrap().text).unwrap() |
|
| 5059 | + | }; |
|
| 5060 | + | let first = seen(&pin); |
|
| 5061 | + | ||
| 5062 | + | // A later checkpoint: proved to extend the first from the tiles. |
|
| 5063 | + | push(&a); |
|
| 5064 | + | checkpoint(); |
|
| 5065 | + | let report = b.fetch("origin").unwrap(); |
|
| 5066 | + | assert!(report.log_notes.is_empty(), "{:?}", report.log_notes); |
|
| 5067 | + | assert!(seen(&pin).size > first.size); |
|
| 5068 | + | ||
| 5069 | + | // Pushed after the last checkpoint: not provable yet, only a note. |
|
| 5070 | + | push(&a); |
|
| 5071 | + | let report = b.fetch("origin").unwrap(); |
|
| 5072 | + | assert!(report.log_notes.iter().any(|n| n.contains("not in its transparency log yet")), "{:?}", report.log_notes); |
|
| 5073 | + | ||
| 5074 | + | // What this user saw replaced by another log of the same size, signed |
|
| 5075 | + | // by the hub's own key: the hub's real log does not extend it. |
|
| 5076 | + | let size = seen(&pin).size; |
|
| 5077 | + | let forged = Checkpoint { origin: oxygenhub::tlog::origin_of(h.app.config.base_url()), size, root: [7; 32], extensions: vec![] }; |
|
| 5078 | + | let text = std::fs::read_to_string(&pin).unwrap(); |
|
| 5079 | + | let key = text.split_once('\n').unwrap().0; |
|
| 5080 | + | std::fs::write(&pin, format!("{key}\n{}", forged.sign(&hub_id, &sk).unwrap().encode())).unwrap(); |
|
| 5081 | + | checkpoint(); |
|
| 5082 | + | let err = b.fetch("origin").unwrap_err().to_string(); |
|
| 5083 | + | assert!(err.contains("showed two histories") && err.contains("does not extend"), "{err}"); |
|
| 5084 | + | } |
|
| 5085 | + | ||
| 5024 | 5086 | /// Keyless step 4c (oxygenhub#241): when a key the hub certified moves a |
|
| 5025 | 5087 | /// bookmark, the hub signs when it accepted the update and serves that |
|
| 5026 | 5088 | /// with the bookmark, so a fetch checks the certificate at the hub's time |
|
| … | 181 unchanged lines |
replaced deps/oxygenflow +0 −0
No line changes.
modified docs/spec.md +2 −1
| … | 605 unchanged lines |
||
| 606 | 606 | - **Storage:** the log is append-only files in `<data>/tlog/`: `entries` (each a u16 length and its bytes), `offsets`, `hashes` (every tree hash in stored order) and `checkpoint`. A torn append is cut back to the last whole entry on start and its hashes rebuilt; nothing is ever rewritten. |
|
| 607 | 607 | - **Checkpoints:** every 5 seconds while entries arrive, the hub signs a checkpoint with its key. The origin is the base URL's host and `/log`, and the note carries an Ed25519 line and a composite line. |
|
| 608 | 608 | - **Serving:** it serves `GET /log/checkpoint` (no-cache), `GET /log/tile/<L>/<N>[.p/<W>]` and `GET /log/tile/entries/<N>[.p/<W>]` (C2SP tlog-tiles: full tiles cached for a year as immutable, partial ones for 5 s; a tile the log does not hold yet is 404), and `/log`, a page with the checkpoint and the newest entries (a private repository's entries show only their hashes). |
|
| 609 | - | - **Not yet:** inclusion proofs in push and refs responses, oxy's checks, witnesses and trusted time are the next steps (docs/transparency.md). Backups do not copy the log yet. |
|
| 609 | + | - **Refs answers** carry the latest checkpoint and, for each bookmark's attestation, its entry and inclusion proof (Oxygen spec §16; step A3, 2026-10-18). An attestation served but never logged (pushed before the log began) is logged then, and proved on a later fetch. oxy pins the log's key per hub, requires each checkpoint to extend the last it saw (proved from the tiles), and fails a fetch whose hub showed two histories. |
|
| 610 | + | - **Not yet:** proofs in push responses, `oxy tlog verify`, witnesses and trusted time (docs/transparency.md). Backups do not copy the log yet. |
|
| 610 | 611 | ||
| 611 | 612 | ### 13.6 Coverage reports (oxygenhub#113, done 2026-10-06) |
|
| 612 | 613 | ||
| … | 168 unchanged lines |