OXYGENHUB

Depth 0 takes no change, and a checkpoint is gathered in parallel (refs #137) f3811eb20bea

A depth-0 clone's heads become its shallow boundary instead of being
fetched; push, pull and fetch in a shallow repository take and build on
changes below what it cloned. The server reads a checkpoint's graph blobs
and chunks in two parallel batches, and OXY_TIMINGS times the
checkpoint's phases on both sides. The lazy-clone design's analysis is
corrected with the timings.
Author
Andy McCall @andymccall <[email protected]>
via
agent agent:claude@andymccall <[email protected]> · keyless, under your grant for changes,refs,seal · key 83bf14b5854b
When
· 2026-10-25 02:42 UTC
Signature
◈ sealed by agent:claude@andymccall
Hash
f3811eb20bea7aae11743297532b89af8e428178eccf4531b15df23dfee3f1b9
Depends on
091fef80755d 252e8cb23fbc 2863456a0f6d 2aa1854b0d13 5b9c6ad45f6e 750251b21a96 75167672ebcd 7bd805ffb418 83e0677b07fe 8fd78eb52ad2 982864068a59 afaba5f1306d b3c5a59a875a d3b270909c78 e02bf0cd0533 ff0af1ee299d
Depended on by
2f0b4b0651ea 80b069facc69
In bookmarks
identical-sides main push-notes subrepo-pin-state
In tags
v0.10.0 v0.11.0 v0.6.0 v0.7.0 v0.8.0 v0.9.0
Edits
32 atoms, 0 tree operations, 8 files

Browse the tree at this change

modified CHANGELOG.md +2 −0
…
2 unchanged lines
3 3
## Unreleased
4 4
5 5
- **A shallow clone of a big repository works, and quickly (#137).** `oxy clone --depth 0` of the VS Code history failed after eleven minutes ("object … not found"): the blobs of the state's binary files are named only by the changes that set them, which a depth-0 clone does not take, so nothing brought them. They are now fetched with the checkpoint (in parts), or left to be fetched on read under `--max-blob`. The checkpoint itself was stored object by object, each written loose and flushed (57,498 objects at 27 a second); it is now stored in one batch, as a fetch's objects are.
6 +
- **A checkpoint is served faster, and says where its time goes (#137).** A server gathered a checkpoint's graph blobs and their chunks one read at a time: six minutes for the VS Code state's 57,000. They are now read in two parallel batches. `OXY_TIMINGS=1` now times the checkpoint's phases on both sides (`checkpoint: gather its objects`, `serve: encode the checkpoint`, `remote: checkpoint answer`, `shallow: store the checkpoint` and the rest).
7 +
- **A depth-0 clone takes no change (#137).** `oxy clone --depth 0` took the head changes themselves (the VS Code import's main has 33,835 heads), though the checkpoint already holds every change's metadata, so depth 0 now takes none (spec §17's "a checkpoint plus zero history"): its heads become the shallow boundary, and commit, push and pull on top work as before (a shallow repository takes and builds on changes below what it cloned, which the remote holds).
6 8
- **A push no longer passes a proxy's request limit (#133).** A push to a hub behind Cloudflare, which takes at most 100 MB in one request, was refused `HTTP 413` with Cloudflare's HTML page printed raw: parts of a long push were about 256 MiB, and one change's blobs went in one request however many there were. Parts are now about 64 MiB (`push.batch-bytes`), a part whose blobs pass that sends them first in parts of objects alone, an HTML error page is shown by its title, and a 413 says how to push in smaller parts.
7 9
- **A keyless agent's grant survives its owner's key rotation (#135).** Every delegation an owner adds or removes renews their identity, and a repository that names the newest key as `author.key` read every change their agent sealed as INVALID ("key … is neither the author nor a delegate"), and its pushes were refused, because the agent's grant was signed by the owner's earlier key. A grant by a key the author's identity was rotated or renewed from now counts when it was made before that rotation; one an old key made after it does not.
8 10
- **A tag can be withdrawn, by a signed withdrawal (oxygenhub#301).** There was no way to remove a tag anywhere: a mistaken release tag stayed for good. `oxy tag delete NAME [-r REASON]` records an `untag` attestation naming the tag's state, signed like a tag (a ten-minute keyless key too), and the next `oxy push` sends it to a remote that still has the tag; `oxy push --delete tags/NAME` does both at once. A receiver under a policy accepts it only from a key that may create the tag, keeps it beside the tag's creation, and lists withdrawn tags in its refs answer (key 8), so a `fetch` withdraws the tag there too (`tag v1 was withdrawn on origin, and here`). A withdrawn name is never used again, so a tag name never names two states. `oxy-server`'s `Backend` gains `vet_tag_withdrawal` (OxygenHub refuses while a release uses the tag), and `Repo::host_tag_create` and `host_tag_withdrawal` let a host make or withdraw a tag for a signed-in user. The wire change is additive, but an older server refuses a withdrawal (wrong attestation kind), and an older client keeps a withdrawn tag.
…
258 unchanged lines
modified crates/oxy-core/src/protocol.rs +6 −1
…
686 unchanged lines
687 687
        )
688 688
    }
689 689
    fn checkpoint(&self, heads: &[ChangeHash]) -> Result<Bundle> {
690 -
        wire::decode_bundle(&self.call(true, "checkpoint", Some(&wire::encode_hashes(heads)))?)
690 +
        let body = {
691 +
            let _t = crate::timing::phase("remote: checkpoint answer");
692 +
            self.call(true, "checkpoint", Some(&wire::encode_hashes(heads)))?
693 +
        };
694 +
        let _t = crate::timing::phase("remote: decode the checkpoint");
695 +
        wire::decode_bundle(&body)
691 696
    }
692 697
    fn shallow(&self, heads: &[ChangeHash], depth: u32) -> Result<Vec<ChangeHash>> {
693 698
        wire::decode_hashes(&self.call(
…
1076 unchanged lines
modified crates/oxy-core/src/repo/checkpoint.rs +20 −4
…
117 unchanged lines
118 118
    /// when it matches, otherwise one is made and, when `heads` are the
119 119
    /// bookmark's, kept.
120 120
    pub fn checkpoint_bundle_for(&self, heads: &[ChangeHash]) -> Result<Bundle> {
121 +
        let t_find = crate::timing::phase("checkpoint: find or make");
121 122
        let wanted = StateObject::new(heads.to_vec()).heads;
122 123
        let bookmark = self.config.bookmark.clone();
123 124
        let current = self.bookmark_heads()?;
…
14 unchanged lines
138 139
                }
139 140
            }
140 141
        };
142 +
        drop(t_find);
143 +
        let _t = crate::timing::phase("checkpoint: gather its objects");
141 144
        let cp = Checkpoint::decode(&self.store.get_typed(&r, ObjectType::Checkpoint)?)?;
142 145
        let mut bundle = Bundle::default();
143 146
        for c in Blob::decode(&self.store.get_typed(&cp.payload, ObjectType::Blob)?)?.chunks {
144 147
            bundle.objects.push(self.store.get(&c)?);
145 148
        }
146 149
        bundle.objects.push(self.store.get(&cp.payload)?);
147 -
        for g in self.checkpoint_graph_blobs(&cp)? {
148 -
            for c in Blob::decode(&self.store.get_typed(&g, ObjectType::Blob)?)?.chunks {
149 -
                bundle.objects.push(self.store.get(&c)?);
150 +
        // The graph blobs and their chunks in two parallel reads (#137): a
151 +
        // big state's are tens of thousands, and one by one they took six
152 +
        // minutes for VS Code.
153 +
        let graphs = self.checkpoint_graph_blobs(&cp)?;
154 +
        let mut blobs = Vec::with_capacity(graphs.len());
155 +
        for got in self.store.get_many(&graphs) {
156 +
            blobs.push(got?);
157 +
        }
158 +
        let mut chunks = Vec::new();
159 +
        for (ty, body) in &blobs {
160 +
            if *ty != ObjectType::Blob {
161 +
                return fail("a checkpoint graph is not a blob");
150 162
            }
151 -
            bundle.objects.push(self.store.get(&g)?);
163 +
            chunks.extend(Blob::decode(body)?.chunks);
164 +
        }
165 +
        for got in self.store.get_many(&chunks) {
166 +
            bundle.objects.push(got?);
152 167
        }
168 +
        bundle.objects.extend(blobs);
153 169
        bundle.objects.push((ObjectType::Checkpoint, cp.encode()));
154 170
        Ok(bundle)
155 171
    }
…
81 unchanged lines
modified crates/oxy-core/src/repo/sync.rs +55 −9
…
244 unchanged lines
245 245
        let mut done: BTreeSet<ChangeHash> = have.clone();
246 246
        done.extend(boundary.iter().copied());
247 247
        // Iterative post-order DFS.
248 +
        let shallow = self.is_shallow();
248 249
        let mut stack: Vec<(ChangeHash, bool)> = want.iter().rev().map(|h| (*h, false)).collect();
249 250
        while let Some((h, expanded)) = stack.pop() {
250 251
            if done.contains(&h) {
251 252
                continue;
252 253
            }
253 254
            if !self.store.has(&h) {
255 +
                // A shallow repository holds only part of its history; a
256 +
                // change it lacks is below what it cloned (in a depth-0
257 +
                // clone, any change of the checkpoint), which the remote
258 +
                // is assumed to hold, as it is for the boundary (#137).
259 +
                if shallow {
260 +
                    done.insert(h);
261 +
                    continue;
262 +
                }
254 263
                return fail(format!("change {} is not in this repository", h.short(12)));
255 264
            }
256 265
            if expanded {
…
1427 unchanged lines
1684 1693
            // Changes are not individually re-verified here: the bookmark's
1685 1694
            // attestation (checked below) binds its whole state, and the
1686 1695
            // receiver of the push verified the added changes at that time.
1696 +
            // A shallow repository takes changes whose dependencies it never
1697 +
            // held (below what it cloned), as its clone did (#137).
1698 +
            let shallow = self.is_shallow();
1687 1699
            let imported = {
1688 1700
                let _t = crate::timing::phase("fetch: import");
1689 -
                self.import_bundle(&bundle, false)?
1701 +
                if shallow {
1702 +
                    self.import_bundle_shallow(&bundle, false)?
1703 +
                } else {
1704 +
                    self.import_bundle(&bundle, false)?
1705 +
                }
1690 1706
            };
1707 +
            if shallow {
1708 +
                self.refresh_boundary(&imported)?;
1709 +
            }
1691 1710
            if max_blob.is_some() {
1692 1711
                self.register_missing_blobs(&imported, remote_name)?;
1693 1712
            }
1713 +
            let boundary = if shallow {
1714 +
                self.boundary()
1715 +
            } else {
1716 +
                BTreeSet::new()
1717 +
            };
1694 1718
            for h in &r.heads {
1695 -
                if !self.store.has(h) {
1719 +
                if !self.store.has(h) && !boundary.contains(h) {
1696 1720
                    return fail(format!("remote did not send head {}", h.short(12)));
1697 1721
                }
1698 1722
            }
…
150 unchanged lines
1849 1873
            return Ok(0);
1850 1874
        }
1851 1875
        // Base: the remote's checkpoint of exactly these heads.
1852 -
        let cp_bundle = remote.checkpoint(&r.heads)?;
1876 +
        let cp_bundle = {
1877 +
            let _t = crate::timing::phase("shallow: checkpoint request");
1878 +
            remote.checkpoint(&r.heads)?
1879 +
        };
1853 1880
        // In one batch: a big state's checkpoint is tens of thousands of
1854 1881
        // graph blobs and chunks, which stored one by one were each written
1855 1882
        // loose and flushed (a VS Code clone ran at 27 objects a second).
…
2 unchanged lines
1858 1885
            .iter()
1859 1886
            .map(|(ty, body)| (*ty, body.as_slice()))
1860 1887
            .collect();
1888 +
        let t_store = crate::timing::phase("shallow: store the checkpoint");
1861 1889
        let batch = self.store.begin_batch()?;
1862 1890
        let stored = self.store.put_many(&objects);
1863 1891
        batch.finish()?;
1892 +
        drop(t_store);
1864 1893
        let cp_ref = objects
1865 1894
            .iter()
1866 1895
            .zip(stored?)
…
3 unchanged lines
1870 1899
            return fail(format!("{remote_name} sent no checkpoint"));
1871 1900
        };
1872 1901
        self.install_checkpoint(&bookmark, &cp_ref)?;
1902 +
        let _t_blobs = crate::timing::phase("shallow: the binary files' blobs");
1873 1903
        // The blobs of the state's opaque files come with no change when the
1874 1904
        // window holds none of the changes that set them (#137): fetch them
1875 1905
        // now, or under `filter.max-blob` leave them to be fetched on read.
…
30 unchanged lines
1906 1936
                return fail(format!("{remote_name} did not send blob {}", b.short(12)));
1907 1937
            }
1908 1938
        }
1909 -
        // History window.
1910 -
        let wanted = remote.shallow(&r.heads, depth)?;
1911 -
        let bundle = remote.fetch(&wanted, self.config.max_blob)?;
1912 -
        let imported = self.import_bundle_shallow(&bundle, false)?;
1939 +
        // History window. Depth 0 takes none (#137): the checkpoint holds
1940 +
        // every change's metadata, and its heads become the boundary, so a
1941 +
        // state of many heads (33,835 in the VS Code import) costs nothing
1942 +
        // to fetch.
1943 +
        let imported = if depth == 0 {
1944 +
            let mut boundary = self.boundary();
1945 +
            boundary.extend(r.heads.iter().filter(|h| !self.store.has(h)).copied());
1946 +
            self.write_boundary(&boundary)?;
1947 +
            Vec::new()
1948 +
        } else {
1949 +
            let wanted = remote.shallow(&r.heads, depth)?;
1950 +
            let bundle = remote.fetch(&wanted, self.config.max_blob)?;
1951 +
            self.import_bundle_shallow(&bundle, false)?
1952 +
        };
1913 1953
        if self.config.max_blob.is_some() {
1914 1954
            self.register_missing_blobs(&imported, remote_name)?;
1915 1955
        }
…
451 unchanged lines
2367 2407
                reason: "already up to date".into(),
2368 2408
            }]);
2369 2409
        }
2410 +
        // A shallow repository knows the changes on its boundary without
2411 +
        // holding them: a depth-0 clone's are the heads it cloned (#137).
2412 +
        let boundary = self.boundary();
2370 2413
        for h in &remote_heads {
2371 -
            if !self.store.has(h) {
2414 +
            if !self.store.has(h) && !boundary.contains(h) {
2372 2415
                return fail(format!(
2373 2416
                    "the remote has change {} which you do not; pull first (`oxy pull {remote_name}`), then push again",
2374 2417
                    h.short(12)
…
2 unchanged lines
2377 2420
        }
2378 2421
        if !force {
2379 2422
            let closure = self.closure_in_store(&local_heads)?;
2380 -
            if let Some(h) = remote_heads.iter().find(|h| !closure.contains(h)) {
2423 +
            if let Some(h) = remote_heads
2424 +
                .iter()
2425 +
                .find(|h| !closure.contains(h) && !boundary.contains(h))
2426 +
            {
2381 2427
                return fail(format!("not a fast-forward: remote head {} is not in your state; pull first or use --force", h.short(12)));
2382 2428
            }
2383 2429
        }
…
398 unchanged lines
modified crates/oxy-core/src/repo/tests.rs +37 −0
…
3064 unchanged lines
3065 3065
            .unwrap()
3066 3066
    };
3067 3067
    assert_eq!(lazy.read_blob(&blob).unwrap(), png, "fetched on read");
3068 +
    // Depth 0 takes no change: the head is the boundary; the log comes from
3069 +
    // the checkpoint; a change on top commits and pushes.
3070 +
    let mut a = Repo::open(&root.join("a"))
3071 +
        .unwrap()
3072 +
        .with_home(root.join("home"));
3073 +
    let head = origin.bookmark_heads().unwrap()[0];
3074 +
    assert!(!a.store().has(&head), "no change fetched");
3075 +
    assert!(a.boundary().contains(&head));
3076 +
    assert_eq!(
3077 +
        a.log()
3078 +
            .unwrap()
3079 +
            .iter()
3080 +
            .filter(|e| !e.is_working_copy)
3081 +
            .count(),
3082 +
        2,
3083 +
        "the log from the checkpoint"
3084 +
    );
3085 +
    write(&root.join("a"), "a.txt", b"later\nmine\n");
3086 +
    a.config.set("remote.origin.url", &url).unwrap();
3087 +
    a.save_config().unwrap();
3088 +
    let mine = a.commit("mine", &[]).unwrap();
3089 +
    assert!(a.push("origin", false).unwrap().iter().all(|r| r.ok));
3090 +
    origin.refresh().unwrap();
3091 +
    assert!(origin.store().has(&mine));
3092 +
    assert_eq!(origin.bookmark_heads().unwrap(), vec![mine]);
3093 +
    // And pulls what comes after.
3094 +
    origin
3095 +
        .checkout(&origin.bookmark_heads().unwrap(), false, "co")
3096 +
        .unwrap();
3097 +
    write(&origin_dir, "a.txt", b"later\nmine\ntheirs\n");
3098 +
    origin.snapshot("t").unwrap();
3099 +
    origin.commit("theirs", &[]).unwrap();
3100 +
    a.pull("origin").unwrap();
3101 +
    assert_eq!(
3102 +
        fs::read(root.join("a/a.txt")).unwrap(),
3103 +
        b"later\nmine\ntheirs\n"
3104 +
    );
3068 3105
}
3069 3106
3070 3107
#[test]
…
6984 unchanged lines
modified crates/oxy-server/src/lib.rs +3 −1
…
318 unchanged lines
319 319
                let heads = wire::decode_hashes(&body)?;
320 320
                let lock = self.lock_for(name);
321 321
                let _guard = lock.lock().unwrap();
322 -
                ok(wire::encode_bundle(&r.checkpoint_bundle_for(&heads)?))
322 +
                let bundle = r.checkpoint_bundle_for(&heads)?;
323 +
                let _t = oxy_core::timing::phase("serve: encode the checkpoint");
324 +
                ok(wire::encode_bundle(&bundle))
323 325
            }
324 326
            Endpoint::Shallow => {
325 327
                let (heads, depth) = wire::decode_shallow(&body)?;
…
185 unchanged lines
modified docs/lazy-clone.md +8 −8
…
160 unchanged lines
161 161
What this says:
162 162
163 163
- A depth-0 clone is five times slower than Git's full clone and fifteen
164 -
  times slower than its blobless one. The profile (`OXY_TIMINGS=1`, one
165 -
  run of 508 s): about 375 s fetching, because the imported history's
166 -
  `main` has 33,835 heads and depth 0 still takes the head changes
167 -
  themselves (33,837 changes, with their texts); 12 s decoding the
168 -
  checkpoint; 120 s of checkout, 111 s of it rendering 19,545 files on one
169 -
  core. So depth 0 should take no change at all (the checkpoint carries
170 -
  every change's metadata; spec §17's "a checkpoint plus zero history"),
171 -
  and rendering should use every core (oxygen#137).
164 +
  times slower than its blobless one. Timed on both sides (`OXY_TIMINGS=1`,
165 +
  oxygen#137): the server spent 375 s gathering the checkpoint's 57,000
166 +
  graph blobs and chunks one read at a time (239 s once read in parallel),
167 +
  19 s encoding one 2 GB answer, and the client 120 s on checkout, 111 s
168 +
  of it rendering 19,545 files on one core. Depth 0 also took the 33,835
169 +
  head changes, which it no longer does (spec §17), though that was not
170 +
  where the time went. A lazy clone avoids all three: no graph blobs move
171 +
  at clone time, and nothing is rendered until read.
172 172
- `--max-blob 0` without a mount is slower, not faster: the checkout then
173 173
  fetches every binary file's blob one request at a time. Lazy contents
174 174
  only pay with a mount that reads them when asked.
…
21 unchanged lines
modified docs/spec.md +1 −1
…
930 unchanged lines
931 931
932 932
- **Sparse** repositories hold the full change graph but only materialize the listed paths: the working copy contains nothing else, files outside the filter are neither reported nor removed, and changes touching other paths are stored and applied to the graph. A sparse clone sets `max-blob` to zero unless a size is given, so blobs outside the filter are fetched only if the filter later widens (§13.5). `oxy sparse PATH..` changes the filter of a clean working copy and re-checks it out.
933 933
- **Lazy blobs** are fetched via `objects` on first read. `oxy status` and `oxy commit` MUST NOT need them.
934 -
- **Shallow** repositories hold a checkpoint of their bookmark's heads (obtained with the `checkpoint` message) and the changes within `depth` dependency steps of those heads (the `shallow` message lists them). The *shallow boundary* is the set of dependencies of stored changes that are not stored (`.oxy/shallow`). Dependencies into the boundary are accepted as well-formed, negotiation stops at the boundary (the remote is assumed to hold it), and the log shows every change from the checkpoint's metadata. `oxy deepen N` fetches `N` more levels below the boundary; when nothing remains the repository is no longer shallow.
934 +
- **Shallow** repositories hold a checkpoint of their bookmark's heads (obtained with the `checkpoint` message) and the changes within `depth` dependency steps of those heads (the `shallow` message lists them); at depth 0 no change at all, the checkpoint's heads being the boundary (oxygen#137), and a shallow repository takes and builds on changes whose dependencies lie below what it cloned, which its remote is assumed to hold. The *shallow boundary* is the set of dependencies of stored changes that are not stored (`.oxy/shallow`). Dependencies into the boundary are accepted as well-formed, negotiation stops at the boundary (the remote is assumed to hold it), and the log shows every change from the checkpoint's metadata. `oxy deepen N` fetches `N` more levels below the boundary; when nothing remains the repository is no longer shallow.
935 935
- Thin clients (§22) always use `depth = 0`: a checkpoint plus zero history.
936 936
937 937
Servers MUST honour `filter` on `negotiate`, `fetch`, and `checkpoint`.
…
251 unchanged lines