OXYGENHUB

Refs answers carry the transparency log (step A3) e5b97ac399e7

The hub adds its latest checkpoint to every refs answer, and for each
bookmark's attestation its log entry and an inclusion proof against that
checkpoint (oxy-server's add_log hook; Oxygen spec §16). An attestation
it serves but never logged, as pushed before the log began, is logged
then and proved on a later fetch. Test: a fetch checks the log and pins
it; a later checkpoint is proved from the tiles to extend it; a push not
yet in a checkpoint is a note; a log that does not extend what the user
saw fails the fetch as two histories. Pins oxygen ab796c92.

Refs #238
Author
Andy McCall @andymccall <[email protected]>
via
agent agent:claude@andymccall <[email protected]> · keyless, under your grant for changes,refs,seal · key 96d114fa75fd
When
· 2026-10-17 23:56 UTC
Signature
◈ sealed by agent:claude@andymccall
Hash
e5b97ac399e7075b76671fe2be870e620452fc72ba9237228c008fa7aaec77a5
Depends on
19bb31ff1f5c 380996db84e5 3eb491e9489b 42fa95540468 93be9ce4653e 965b950f9b96 d23464c0ac60 e271a37b629a
Depended on by
f87a74fffc87 ba0a55bb5b3c ec84907043c2 b5c199bd6b1a 0ee5c3fb32eb 13b27cb2200c 4f40eed0588f 7dbb41b09a99
In bookmarks
api-unknown-fields cancel-landed-runs diffusion-pin-bookmark draft-ready main oh-comment-body-file proposal-list-fast
In tags
v0.6.0 v0.6.1 v0.7.0 v0.8.0 v0.9.0 v0.9.1 v0.9.11 v0.9.15 v0.9.16 v0.9.19 v0.9.2 v0.9.20 v0.9.21 v0.9.22 v0.9.23 v0.9.24 v0.9.3 v0.9.4 v0.9.5 v0.9.6 v0.9.7 v0.9.8 v0.9.9
Edits
7 atoms, 0 tree operations, 5 files

Browse the tree at this change

modified crates/hub/src/protocol.rs +9 −0
…
251 unchanged lines
252 252
        self.row(&parsed).ok().map(|r| r.default_bookmark)
253 253
    }
254 254
255 +
    fn add_log(&self, name: &str, refs: &mut oxy_core::protocol::RefsResponse) {
256 +
        let Some(app) = self.app_hint.lock().unwrap().as_ref().and_then(|w| w.upgrade()) else { return };
257 +
        let row = RepoName::parse(name).and_then(|p| self.row(&crate::wiki::parent_of(&p).0).ok()).map(|r| r.id);
258 +
        let backfill = crate::tlog::add_log(&app, row, refs);
259 +
        if !backfill.is_empty() {
260 +
            self.block(crate::tlog::log(&app, backfill));
261 +
        }
262 +
    }
263 +
255 264
    fn after_push(&self, name: &str, report: &PushReport, headers: &HeaderMap) {
256 265
        let Some(parsed) = RepoName::parse(name) else { return };
257 266
        // A wiki push starts no run and rings no webhook.
…
201 unchanged lines
modified crates/hub/src/tlog.rs +37 −0
…
141 unchanged lines
142 142
        Ok(b)
143 143
    }
144 144
145 +
    /// Entry `n`'s bytes.
146 +
    pub fn entry(&self, n: u64) -> anyhow::Result<Vec<u8>> {
147 +
        let f = self.files.lock().unwrap();
148 +
        anyhow::ensure!(n < f.size, "entry {n} is not in the log");
149 +
        Self::entry_bytes(&f, n)
150 +
    }
151 +
145 152
    /// Entries in the log.
146 153
    pub fn size(&self) -> u64 {
147 154
        self.files.lock().unwrap().size
…
204 unchanged lines
352 359
        add(&mut out, "policy", r, sig_keys(&sp.sigs));
353 360
    }
354 361
    out
362 +
}
363 +
364 +
/// Add the log to a refs answer (Oxygen spec §14.8): the latest
365 +
/// checkpoint, and for each bookmark's attestation its entry and inclusion
366 +
/// proof against it. An attestation the hub serves but never logged (one
367 +
/// pushed before the log began) is logged now, and proved on a later fetch.
368 +
pub fn add_log(app: &Shared, row: Option<i64>, refs: &mut oxy_core::protocol::RefsResponse) -> Vec<Pending> {
369 +
    let mut backfill = Vec::new();
370 +
    let Some(note) = app.tlog.checkpoint() else { return backfill };
371 +
    let Some(size) = tlog::Note::parse(&note).ok().and_then(|n| Checkpoint::parse(&n.text).ok()).map(|c| c.size) else {
372 +
        return backfill;
373 +
    };
374 +
    refs.log = Some(note);
375 +
    for r in refs.refs.iter_mut() {
376 +
        let Some(a) = &r.attestation else { continue };
377 +
        let object = a.attest.hash();
378 +
        let idx = tokio::task::block_in_place(|| {
379 +
            tokio::runtime::Handle::current().block_on(db::tlog::entry_of(&app.db, &object.to_hex(), "ref"))
380 +
        });
381 +
        match idx {
382 +
            Ok(Some(i)) if (i as u64) < size => {
383 +
                let (Ok(entry), Ok(proof)) = (app.tlog.entry(i as u64), app.tlog.prove_inclusion(size, i as u64)) else { continue };
384 +
                r.logged = Some(oxy_core::protocol::Logged { index: i as u64, entry, proof });
385 +
            }
386 +
            Ok(Some(_)) => {}
387 +
            Ok(None) => backfill.push(Pending { kind: "ref", repo: a.attest.repo, repo_row: row, object, signers: sig_keys(&a.sigs) }),
388 +
            Err(e) => eprintln!("tlog: looking up {}: {e}", &object.to_hex()[..12]),
389 +
        }
390 +
    }
391 +
    backfill
355 392
}
356 393
357 394
/// Sign a new checkpoint every few seconds while entries arrive.
…
120 unchanged lines
modified crates/hub/tests/flow.rs +62 −0
…
5020 unchanged lines
5021 5021
    assert!(page.contains("Transparency log") && page.contains("andy/demo") && page.contains(&origin), "{page}");
5022 5022
}
5023 5023
5024 +
/// Fetching checks the hub's log (oxygenhub#238, step A3): the checkpoint
5025 +
/// comes with the refs, each attestation with its entry and inclusion
5026 +
/// proof; a later checkpoint is proved to extend the one seen before, from
5027 +
/// the hub's tiles; an attestation not in a checkpoint yet is a note; and a
5028 +
/// log that does not extend what this user saw fails the fetch.
5029 +
#[tokio::test(flavor = "multi_thread")]
5030 +
async fn a_fetch_checks_the_hub_s_log_and_refuses_a_fork() {
5031 +
    use oxy_core::tlog::{Checkpoint, Note};
5032 +
    let h = hub().await;
5033 +
    create(&h.app, "andy", "demo").await;
5034 +
    let url = format!("{}/andy/demo", h.base);
5035 +
    let work = tempfile::tempdir().unwrap();
5036 +
    let mut a = client(&url, &work.path().join("a"));
5037 +
    a.create_identity("Andy", "[email protected]", oxy_core::repo::IdentityScope::Repo).unwrap();
5038 +
    let (_, sk) = oxygenhub::keys::load_secret(&h.app.config.keys_dir()).unwrap();
5039 +
    let hub_id = h.app.hub_key.identity.clone();
5040 +
    let checkpoint = || h.app.tlog.refresh_checkpoint(&hub_id, &sk).unwrap();
5041 +
    let mut n = 0;
5042 +
    let mut push = |a: &Repo| {
5043 +
        n += 1;
5044 +
        write(&work.path().join("a"), "a.txt", &format!("{n}\n"));
5045 +
        a.commit(&format!("change {n}"), &[]).unwrap();
5046 +
        assert!(a.push("origin", false).unwrap().iter().all(|r| r.ok));
5047 +
    };
5048 +
    push(&a);
5049 +
    checkpoint();
5050 +
5051 +
    let home = work.path().join("home-b");
5052 +
    let b = Repo::clone(&url, &work.path().join("b")).unwrap().with_home(home.clone());
5053 +
    let report = b.fetch("origin").unwrap();
5054 +
    assert!(report.log_notes.is_empty(), "{:?}", report.log_notes);
5055 +
    let pin = std::fs::read_dir(home.join("tlog")).unwrap().next().unwrap().unwrap().path();
5056 +
    let seen = |p: &std::path::Path| {
5057 +
        let t = std::fs::read_to_string(p).unwrap();
5058 +
        Checkpoint::parse(&Note::parse(t.split_once('\n').unwrap().1).unwrap().text).unwrap()
5059 +
    };
5060 +
    let first = seen(&pin);
5061 +
5062 +
    // A later checkpoint: proved to extend the first from the tiles.
5063 +
    push(&a);
5064 +
    checkpoint();
5065 +
    let report = b.fetch("origin").unwrap();
5066 +
    assert!(report.log_notes.is_empty(), "{:?}", report.log_notes);
5067 +
    assert!(seen(&pin).size > first.size);
5068 +
5069 +
    // Pushed after the last checkpoint: not provable yet, only a note.
5070 +
    push(&a);
5071 +
    let report = b.fetch("origin").unwrap();
5072 +
    assert!(report.log_notes.iter().any(|n| n.contains("not in its transparency log yet")), "{:?}", report.log_notes);
5073 +
5074 +
    // What this user saw replaced by another log of the same size, signed
5075 +
    // by the hub's own key: the hub's real log does not extend it.
5076 +
    let size = seen(&pin).size;
5077 +
    let forged = Checkpoint { origin: oxygenhub::tlog::origin_of(h.app.config.base_url()), size, root: [7; 32], extensions: vec![] };
5078 +
    let text = std::fs::read_to_string(&pin).unwrap();
5079 +
    let key = text.split_once('\n').unwrap().0;
5080 +
    std::fs::write(&pin, format!("{key}\n{}", forged.sign(&hub_id, &sk).unwrap().encode())).unwrap();
5081 +
    checkpoint();
5082 +
    let err = b.fetch("origin").unwrap_err().to_string();
5083 +
    assert!(err.contains("showed two histories") && err.contains("does not extend"), "{err}");
5084 +
}
5085 +
5024 5086
/// Keyless step 4c (oxygenhub#241): when a key the hub certified moves a
5025 5087
/// bookmark, the hub signs when it accepted the update and serves that
5026 5088
/// with the bookmark, so a fetch checks the certificate at the hub's time
…
181 unchanged lines
replaced deps/oxygenflow +0 −0

No line changes.

modified docs/spec.md +2 −1
…
605 unchanged lines
606 606
- **Storage:** the log is append-only files in `<data>/tlog/`: `entries` (each a u16 length and its bytes), `offsets`, `hashes` (every tree hash in stored order) and `checkpoint`. A torn append is cut back to the last whole entry on start and its hashes rebuilt; nothing is ever rewritten.
607 607
- **Checkpoints:** every 5 seconds while entries arrive, the hub signs a checkpoint with its key. The origin is the base URL's host and `/log`, and the note carries an Ed25519 line and a composite line.
608 608
- **Serving:** it serves `GET /log/checkpoint` (no-cache), `GET /log/tile/<L>/<N>[.p/<W>]` and `GET /log/tile/entries/<N>[.p/<W>]` (C2SP tlog-tiles: full tiles cached for a year as immutable, partial ones for 5 s; a tile the log does not hold yet is 404), and `/log`, a page with the checkpoint and the newest entries (a private repository's entries show only their hashes).
609 -
- **Not yet:** inclusion proofs in push and refs responses, oxy's checks, witnesses and trusted time are the next steps (docs/transparency.md). Backups do not copy the log yet.
609 +
- **Refs answers** carry the latest checkpoint and, for each bookmark's attestation, its entry and inclusion proof (Oxygen spec §16; step A3, 2026-10-18). An attestation served but never logged (pushed before the log began) is logged then, and proved on a later fetch. oxy pins the log's key per hub, requires each checkpoint to extend the last it saw (proved from the tiles), and fails a fetch whose hub showed two histories.
610 +
- **Not yet:** proofs in push responses, `oxy tlog verify`, witnesses and trusted time (docs/transparency.md). Backups do not copy the log yet.
610 611
611 612
### 13.6 Coverage reports (oxygenhub#113, done 2026-10-06)
612 613
…
168 unchanged lines