OXYGENHUB

A push no longer sends a replaced binary file's old content again (closes #136) 570e35310f31

bundle_of added both the old and the new blob of every Replace, so a
change removing ~1,100 binary files sent them all again, and a long
push re-sent, in each part, the old blobs of files added in earlier
parts. The old blob is the new blob of a change the replacement depends
on (spec 8.1), held by the receiver or carried by the same bundle, so a
bundle now carries only the new one (spec 16 says so). Measured on
oxygenhub-android's set-up push: 2,272 blobs and chunks, 240 MB, become
1,405 and 184 MB. Changelog entries for this and #138.
Author
Andy McCall @andymccall <[email protected]>
via
agent agent:claude@andymccall <[email protected]> · keyless, under your grant for changes,refs,seal · key eeb10df24b16
When
· 2026-10-25 08:03 UTC
Signature
◈ sealed by agent:claude@andymccall
Hash
570e35310f313b79957af7e507a9e909fa38e24523c381eb98659eb03ca5b457
Depends on
096f447ccda0 1ec6fdbc6bcb 75167672ebcd 8fd78eb52ad2 982864068a59 b260109d27c5 b3c5a59a875a d3b270909c78 d6e0d5b8b339 e4595038502a ff0af1ee299d
Depended on by
4476cdbb0443
In bookmarks
identical-sides main push-notes subrepo-pin-state
In tags
v0.10.0 v0.11.0 v0.7.0 v0.8.0 v0.9.0
Edits
8 atoms, 0 tree operations, 4 files

Browse the tree at this change

modified CHANGELOG.md +3 −0
…
1 unchanged lines
2 2
3 3
## Unreleased
4 4
5 +
- **`oxy serve` takes a push of more than 2 MB (#138).** It set no request limit of its own, so axum's default of 2 MB applied, and a larger push part was cut off with `Broken pipe` at the client and nothing in the server's log. It now takes up to 4 GiB a request (`oxy_server::BODY_LIMIT`), OxygenHub's default `push_body_limit`; OxygenHub was not affected.
6 +
- **A push no longer sends a replaced or removed binary file's old content again (#136).** A bundle carried both the old and the new blob of every binary file a change replaces, so a change removing ~1,100 tracked cache files (oxygenhub-android's set-up) pushed all of them again, over 100 MB, though the hub stored each when it was added. The old blob is always the new blob of a change the replacement depends on, which the receiver already holds or which the same bundle carries, so pushes and fetch answers now carry only the new one.
7 +
5 8
## 0.6.0 (2026-10-25)
6 9
7 10
- **A shallow clone of a big repository works, and quickly (#137).** `oxy clone --depth 0` of the VS Code history failed after eleven minutes ("object … not found"): the blobs of the state's binary files are named only by the changes that set them, which a depth-0 clone does not take, so nothing brought them. They are now fetched with the checkpoint (in parts), or left to be fetched on read under `--max-blob`. The checkpoint itself was stored object by object, each written loose and flushed (57,498 objects at 27 a second); it is now stored in one batch, as a fetch's objects are.
…
261 unchanged lines
modified crates/oxy-core/src/repo/sync.rs +12 −11
…
381 unchanged lines
382 382
            for s in &sigs {
383 383
                add_object(&s.key, &mut objects);
384 384
            }
385 +
            // A replacement's new content only: its old content is the new
386 +
            // content of a change it depends on, which the receiver holds or
387 +
            // which this bundle carries (oxygen#136: removing ~1,100 binary
388 +
            // files sent them all again).
385 389
            for e in &c.edits {
386 -
                if let Edit::Replace { old, new, .. } = e {
387 -
                    for b in [old, new].into_iter().flatten() {
388 -
                        if self.store.has(b) {
389 -
                            if let Ok(blob) =
390 -
                                Blob::decode(&self.store.get_typed(b, ObjectType::Blob)?)
391 -
                            {
392 -
                                if max_blob.is_some_and(|m| blob.length > m) {
393 -
                                    continue;
394 -
                                }
395 -
                                objects.insert(*b);
396 -
                                objects.extend(blob.chunks.iter().copied());
390 +
                if let Edit::Replace { new: Some(b), .. } = e {
391 +
                    if self.store.has(b) {
392 +
                        if let Ok(blob) = Blob::decode(&self.store.get_typed(b, ObjectType::Blob)?)
393 +
                        {
394 +
                            if max_blob.is_some_and(|m| blob.length > m) {
395 +
                                continue;
397 396
                            }
397 +
                            objects.insert(*b);
398 +
                            objects.extend(blob.chunks.iter().copied());
398 399
                        }
399 400
                    }
400 401
                }
…
2381 unchanged lines
modified crates/oxy-core/src/repo/tests.rs +44 −0
…
9421 unchanged lines
9422 9422
    drop(back);
9423 9423
}
9424 9424
9425 +
/// oxygen#136: a bundle carries a replaced binary file's new content
9426 +
/// only. Its old content came with the change that added it, so removing
9427 +
/// or replacing a file sends no old blob again; the remote and a clone of
9428 +
/// it are whole.
9429 +
#[test]
9430 +
fn a_replacement_does_not_send_the_old_content_again() {
9431 +
    hermetic();
9432 +
    let dir = tempfile::tempdir().unwrap();
9433 +
    let root = dir.path();
9434 +
    let origin = root.join("origin");
9435 +
    Repo::init_bare_unbound(&origin).unwrap();
9436 +
    let main = root.join("main");
9437 +
    let mut repo = Repo::init(&main).unwrap();
9438 +
    repo.config
9439 +
        .set("remote.origin.url", &origin.to_string_lossy())
9440 +
        .unwrap();
9441 +
    repo.save_config().unwrap();
9442 +
    let blobs = |b: &Bundle| {
9443 +
        b.objects
9444 +
            .iter()
9445 +
            .filter(|(ty, _)| *ty == ObjectType::Blob)
9446 +
            .count()
9447 +
    };
9448 +
    write(&main, "bin.dat", b"\0\x01first binary");
9449 +
    write(&main, "gone.dat", b"\0\x02to be removed");
9450 +
    let added = repo.commit("add", &[]).unwrap();
9451 +
    assert!(repo.push("origin", false).unwrap()[0].ok);
9452 +
    write(&main, "bin.dat", b"\0\x01second binary");
9453 +
    fs::remove_file(main.join("gone.dat")).unwrap();
9454 +
    let changed = repo.commit("replace and remove", &[]).unwrap();
9455 +
    assert_eq!(blobs(&repo.bundle_for(&[changed]).unwrap()), 1);
9456 +
    assert_eq!(blobs(&repo.bundle_for(&[added, changed]).unwrap()), 3);
9457 +
    assert!(repo.push("origin", false).unwrap()[0].ok);
9458 +
    let origin = Repo::open(&origin).unwrap();
9459 +
    assert_eq!(origin.fsck().unwrap().problems, vec![]);
9460 +
    let back = Repo::clone(&origin.root().to_string_lossy(), &root.join("back")).unwrap();
9461 +
    assert_eq!(back.fsck().unwrap().problems, vec![]);
9462 +
    assert_eq!(
9463 +
        fs::read(root.join("back/bin.dat")).unwrap(),
9464 +
        b"\0\x01second binary"
9465 +
    );
9466 +
    assert!(!root.join("back/gone.dat").exists());
9467 +
}
9468 +
9425 9469
/// oxygen#97: with `git.claim`, an import authors the importing identity's
9426 9470
/// own commits (Git author email = its email, any case) with its key, and
9427 9471
/// leaves anyone else's with the zero key; the history verifies as before.
…
627 unchanged lines
modified docs/spec.md +1 −1
…
894 unchanged lines
895 895
| Message | Encoding |
896 896
|---|---|
897 897
| Signed attestation | `[attest bytes, [Signature]]` |
898 -
| `Bundle` | `{0: [[change bytes, [Signature]]], 1: [signed attestation], 2: [[type name, object bytes]], 3: [signed policy]}`; changes in dependency order; objects are chunks, blobs, identities, and states only, and include the text blobs of every change in the bundle and their chunks, so a bundle can be read on its own; a push that moves a bookmark also carries the identity that signed the move and, for a delegate, the identity it acts for, so a receiver that has seen no change by that key can check it (oxygenhub#129), and a push that sends tags carries each tag signer's identity and, for a short-lived key, its certificate likewise (oxygen#129); key 3 (optional) holds the policies between the receiver's serial and the one sent with a push, ascending; key 4 (optional, only when there are any) holds signed certificates (§14.7) of short-lived keys that signed what the bundle carries or the push's attestation, their keys' and issuers' identities among the objects; a receiver keeps each that verifies with its issuer and refuses a bundle with one that does not |
898 +
| `Bundle` | `{0: [[change bytes, [Signature]]], 1: [signed attestation], 2: [[type name, object bytes]], 3: [signed policy]}`; changes in dependency order; objects are chunks, blobs, identities, and states only, and include the text blobs of every change in the bundle and their chunks, so a bundle can be read on its own, and the `new` blob of every `Replace` (with its chunks) unless left out by size; never a replacement's `old` blob, which is the `new` of a change it depends on (§8.1), held by the receiver or carried by the same bundle (oxygen#136); a push that moves a bookmark also carries the identity that signed the move and, for a delegate, the identity it acts for, so a receiver that has seen no change by that key can check it (oxygenhub#129), and a push that sends tags carries each tag signer's identity and, for a short-lived key, its certificate likewise (oxygen#129); key 3 (optional) holds the policies between the receiver's serial and the one sent with a push, ascending; key 4 (optional, only when there are any) holds signed certificates (§14.7) of short-lived keys that signed what the bundle carries or the push's attestation, their keys' and issuers' identities among the objects; a receiver keeps each that verifies with its issuer and refuses a bundle with one that does not |
899 899
| `capabilities` response | `{0: [versions], 1: repo id bytes(16), 2: [feature names]}` |
900 900
| Signed policy | `[policy bytes, [Signature]]` |
901 901
| `refs` response | `{0: [{0: name, 1: state ref, 2: [heads], 3: signed attestation or null, 4: last serial or null, 5: signed acceptance}], 1: repo id, 2: signed policy or null, 3: [tag], 4: [signed policy]}`; key 4 (optional) lists earlier policies ascending by serial; key 5 (optional, text or null) names the default bookmark, which a clone checks out; the attestation is present only when it describes the current state; an entry's key 5 (optional, only when there is one) is the host's acceptance of that attestation (§14.7); an entry's key 6 (optional) is the attestation's place in the host's transparency log, `[entry number, entry bytes, [proof hash]]`, an inclusion proof against the checkpoint in the response's key 6 (§14.8); an entry's key 7 (optional, only beside key 6 and when there are any) lists the cosigned checkpoints that first held that log entry, `[[checkpoint note text, [proof hash]]]`, each with the entry's inclusion proof against it, from which its integrated time is read (§14.8); the response's key 6 (optional, text) is the host's latest checkpoint, a signed note; the response's key 7 (optional, only when there are any) lists the witnesses the host asks to cosign, `[[name, identity bytes]]`, which a client pins (§14.8); a tag is `{0: name, 1: state ref, 2: [heads], 3: signed attestation or null}`, and optionally, each only when there is one, `4: signed acceptance`, `5: log place`, `6: [cosigned checkpoint]`, meaning for the tag's attestation what a bookmark entry's keys 5 to 7 mean for its attestation (oxygen#129); a reader that does not know them ignores them |
…
287 unchanged lines