A push no longer sends a replaced binary file's old content again (closes #136) 570e35310f31
bundle_of added both the old and the new blob of every Replace, so a change removing ~1,100 binary files sent them all again, and a long push re-sent, in each part, the old blobs of files added in earlier parts. The old blob is the new blob of a change the replacement depends on (spec 8.1), held by the receiver or carried by the same bundle, so a bundle now carries only the new one (spec 16 says so). Measured on oxygenhub-android's set-up push: 2,272 blobs and chunks, 240 MB, become 1,405 and 184 MB. Changelog entries for this and #138.
- Author
- Andy McCall @andymccall <[email protected]>
- via
- agent
agent:claude@andymccall <[email protected]> · keyless, under your grant for changes,refs,seal · key eeb10df24b16
- When
- · 2026-10-25 08:03 UTC
- Signature
- ◈ sealed by agent:claude@andymccall
- Hash
570e35310f313b79957af7e507a9e909fa38e24523c381eb98659eb03ca5b457- Depends on
- 096f447ccda0 1ec6fdbc6bcb 75167672ebcd 8fd78eb52ad2 982864068a59 b260109d27c5 b3c5a59a875a d3b270909c78 d6e0d5b8b339 e4595038502a ff0af1ee299d
- Depended on by
- 4476cdbb0443
- In bookmarks
- identical-sides main push-notes subrepo-pin-state
- In tags
- v0.10.0 v0.11.0 v0.7.0 v0.8.0 v0.9.0
- Edits
- 8 atoms, 0 tree operations, 4 files
Browse the tree at this change
modified CHANGELOG.md +3 −0
| … | 1 unchanged lines |
||
| 2 | 2 | ||
| 3 | 3 | ## Unreleased |
|
| 4 | 4 | ||
| 5 | + | - **`oxy serve` takes a push of more than 2 MB (#138).** It set no request limit of its own, so axum's default of 2 MB applied, and a larger push part was cut off with `Broken pipe` at the client and nothing in the server's log. It now takes up to 4 GiB a request (`oxy_server::BODY_LIMIT`), OxygenHub's default `push_body_limit`; OxygenHub was not affected. |
|
| 6 | + | - **A push no longer sends a replaced or removed binary file's old content again (#136).** A bundle carried both the old and the new blob of every binary file a change replaces, so a change removing ~1,100 tracked cache files (oxygenhub-android's set-up) pushed all of them again, over 100 MB, though the hub stored each when it was added. The old blob is always the new blob of a change the replacement depends on, which the receiver already holds or which the same bundle carries, so pushes and fetch answers now carry only the new one. |
|
| 7 | + | ||
| 5 | 8 | ## 0.6.0 (2026-10-25) |
|
| 6 | 9 | ||
| 7 | 10 | - **A shallow clone of a big repository works, and quickly (#137).** `oxy clone --depth 0` of the VS Code history failed after eleven minutes ("object … not found"): the blobs of the state's binary files are named only by the changes that set them, which a depth-0 clone does not take, so nothing brought them. They are now fetched with the checkpoint (in parts), or left to be fetched on read under `--max-blob`. The checkpoint itself was stored object by object, each written loose and flushed (57,498 objects at 27 a second); it is now stored in one batch, as a fetch's objects are. |
|
| … | 261 unchanged lines |
modified crates/oxy-core/src/repo/sync.rs +12 −11
| … | 381 unchanged lines |
||
| 382 | 382 | for s in &sigs { |
|
| 383 | 383 | add_object(&s.key, &mut objects); |
|
| 384 | 384 | } |
|
| 385 | + | // A replacement's new content only: its old content is the new |
|
| 386 | + | // content of a change it depends on, which the receiver holds or |
|
| 387 | + | // which this bundle carries (oxygen#136: removing ~1,100 binary |
|
| 388 | + | // files sent them all again). |
|
| 385 | 389 | for e in &c.edits { |
|
| 386 | - | if let Edit::Replace { old, new, .. } = e { |
|
| 387 | - | for b in [old, new].into_iter().flatten() { |
|
| 388 | - | if self.store.has(b) { |
|
| 389 | - | if let Ok(blob) = |
|
| 390 | - | Blob::decode(&self.store.get_typed(b, ObjectType::Blob)?) |
|
| 391 | - | { |
|
| 392 | - | if max_blob.is_some_and(|m| blob.length > m) { |
|
| 393 | - | continue; |
|
| 394 | - | } |
|
| 395 | - | objects.insert(*b); |
|
| 396 | - | objects.extend(blob.chunks.iter().copied()); |
|
| 390 | + | if let Edit::Replace { new: Some(b), .. } = e { |
|
| 391 | + | if self.store.has(b) { |
|
| 392 | + | if let Ok(blob) = Blob::decode(&self.store.get_typed(b, ObjectType::Blob)?) |
|
| 393 | + | { |
|
| 394 | + | if max_blob.is_some_and(|m| blob.length > m) { |
|
| 395 | + | continue; |
|
| 397 | 396 | } |
|
| 397 | + | objects.insert(*b); |
|
| 398 | + | objects.extend(blob.chunks.iter().copied()); |
|
| 398 | 399 | } |
|
| 399 | 400 | } |
|
| 400 | 401 | } |
|
| … | 2381 unchanged lines |
modified crates/oxy-core/src/repo/tests.rs +44 −0
| … | 9421 unchanged lines |
||
| 9422 | 9422 | drop(back); |
|
| 9423 | 9423 | } |
|
| 9424 | 9424 | ||
| 9425 | + | /// oxygen#136: a bundle carries a replaced binary file's new content |
|
| 9426 | + | /// only. Its old content came with the change that added it, so removing |
|
| 9427 | + | /// or replacing a file sends no old blob again; the remote and a clone of |
|
| 9428 | + | /// it are whole. |
|
| 9429 | + | #[test] |
|
| 9430 | + | fn a_replacement_does_not_send_the_old_content_again() { |
|
| 9431 | + | hermetic(); |
|
| 9432 | + | let dir = tempfile::tempdir().unwrap(); |
|
| 9433 | + | let root = dir.path(); |
|
| 9434 | + | let origin = root.join("origin"); |
|
| 9435 | + | Repo::init_bare_unbound(&origin).unwrap(); |
|
| 9436 | + | let main = root.join("main"); |
|
| 9437 | + | let mut repo = Repo::init(&main).unwrap(); |
|
| 9438 | + | repo.config |
|
| 9439 | + | .set("remote.origin.url", &origin.to_string_lossy()) |
|
| 9440 | + | .unwrap(); |
|
| 9441 | + | repo.save_config().unwrap(); |
|
| 9442 | + | let blobs = |b: &Bundle| { |
|
| 9443 | + | b.objects |
|
| 9444 | + | .iter() |
|
| 9445 | + | .filter(|(ty, _)| *ty == ObjectType::Blob) |
|
| 9446 | + | .count() |
|
| 9447 | + | }; |
|
| 9448 | + | write(&main, "bin.dat", b"\0\x01first binary"); |
|
| 9449 | + | write(&main, "gone.dat", b"\0\x02to be removed"); |
|
| 9450 | + | let added = repo.commit("add", &[]).unwrap(); |
|
| 9451 | + | assert!(repo.push("origin", false).unwrap()[0].ok); |
|
| 9452 | + | write(&main, "bin.dat", b"\0\x01second binary"); |
|
| 9453 | + | fs::remove_file(main.join("gone.dat")).unwrap(); |
|
| 9454 | + | let changed = repo.commit("replace and remove", &[]).unwrap(); |
|
| 9455 | + | assert_eq!(blobs(&repo.bundle_for(&[changed]).unwrap()), 1); |
|
| 9456 | + | assert_eq!(blobs(&repo.bundle_for(&[added, changed]).unwrap()), 3); |
|
| 9457 | + | assert!(repo.push("origin", false).unwrap()[0].ok); |
|
| 9458 | + | let origin = Repo::open(&origin).unwrap(); |
|
| 9459 | + | assert_eq!(origin.fsck().unwrap().problems, vec![]); |
|
| 9460 | + | let back = Repo::clone(&origin.root().to_string_lossy(), &root.join("back")).unwrap(); |
|
| 9461 | + | assert_eq!(back.fsck().unwrap().problems, vec![]); |
|
| 9462 | + | assert_eq!( |
|
| 9463 | + | fs::read(root.join("back/bin.dat")).unwrap(), |
|
| 9464 | + | b"\0\x01second binary" |
|
| 9465 | + | ); |
|
| 9466 | + | assert!(!root.join("back/gone.dat").exists()); |
|
| 9467 | + | } |
|
| 9468 | + | ||
| 9425 | 9469 | /// oxygen#97: with `git.claim`, an import authors the importing identity's |
|
| 9426 | 9470 | /// own commits (Git author email = its email, any case) with its key, and |
|
| 9427 | 9471 | /// leaves anyone else's with the zero key; the history verifies as before. |
|
| … | 627 unchanged lines |
modified docs/spec.md +1 −1
| … | 894 unchanged lines |
||
| 895 | 895 | | Message | Encoding | |
|
| 896 | 896 | |---|---| |
|
| 897 | 897 | | Signed attestation | `[attest bytes, [Signature]]` | |
|
| 898 | - | | `Bundle` | `{0: [[change bytes, [Signature]]], 1: [signed attestation], 2: [[type name, object bytes]], 3: [signed policy]}`; changes in dependency order; objects are chunks, blobs, identities, and states only, and include the text blobs of every change in the bundle and their chunks, so a bundle can be read on its own; a push that moves a bookmark also carries the identity that signed the move and, for a delegate, the identity it acts for, so a receiver that has seen no change by that key can check it (oxygenhub#129), and a push that sends tags carries each tag signer's identity and, for a short-lived key, its certificate likewise (oxygen#129); key 3 (optional) holds the policies between the receiver's serial and the one sent with a push, ascending; key 4 (optional, only when there are any) holds signed certificates (§14.7) of short-lived keys that signed what the bundle carries or the push's attestation, their keys' and issuers' identities among the objects; a receiver keeps each that verifies with its issuer and refuses a bundle with one that does not | |
|
| 898 | + | | `Bundle` | `{0: [[change bytes, [Signature]]], 1: [signed attestation], 2: [[type name, object bytes]], 3: [signed policy]}`; changes in dependency order; objects are chunks, blobs, identities, and states only, and include the text blobs of every change in the bundle and their chunks, so a bundle can be read on its own, and the `new` blob of every `Replace` (with its chunks) unless left out by size; never a replacement's `old` blob, which is the `new` of a change it depends on (§8.1), held by the receiver or carried by the same bundle (oxygen#136); a push that moves a bookmark also carries the identity that signed the move and, for a delegate, the identity it acts for, so a receiver that has seen no change by that key can check it (oxygenhub#129), and a push that sends tags carries each tag signer's identity and, for a short-lived key, its certificate likewise (oxygen#129); key 3 (optional) holds the policies between the receiver's serial and the one sent with a push, ascending; key 4 (optional, only when there are any) holds signed certificates (§14.7) of short-lived keys that signed what the bundle carries or the push's attestation, their keys' and issuers' identities among the objects; a receiver keeps each that verifies with its issuer and refuses a bundle with one that does not | |
|
| 899 | 899 | | `capabilities` response | `{0: [versions], 1: repo id bytes(16), 2: [feature names]}` | |
|
| 900 | 900 | | Signed policy | `[policy bytes, [Signature]]` | |
|
| 901 | 901 | | `refs` response | `{0: [{0: name, 1: state ref, 2: [heads], 3: signed attestation or null, 4: last serial or null, 5: signed acceptance}], 1: repo id, 2: signed policy or null, 3: [tag], 4: [signed policy]}`; key 4 (optional) lists earlier policies ascending by serial; key 5 (optional, text or null) names the default bookmark, which a clone checks out; the attestation is present only when it describes the current state; an entry's key 5 (optional, only when there is one) is the host's acceptance of that attestation (§14.7); an entry's key 6 (optional) is the attestation's place in the host's transparency log, `[entry number, entry bytes, [proof hash]]`, an inclusion proof against the checkpoint in the response's key 6 (§14.8); an entry's key 7 (optional, only beside key 6 and when there are any) lists the cosigned checkpoints that first held that log entry, `[[checkpoint note text, [proof hash]]]`, each with the entry's inclusion proof against it, from which its integrated time is read (§14.8); the response's key 6 (optional, text) is the host's latest checkpoint, a signed note; the response's key 7 (optional, only when there are any) lists the witnesses the host asks to cosign, `[[name, identity bytes]]`, which a client pins (§14.8); a tag is `{0: name, 1: state ref, 2: [heads], 3: signed attestation or null}`, and optionally, each only when there is one, `4: signed acceptance`, `5: log place`, `6: [cosigned checkpoint]`, meaning for the tag's attestation what a bookmark entry's keys 5 to 7 mean for its attestation (oxygen#129); a reader that does not know them ignores them | |
|
| … | 287 unchanged lines |