Instant clone: the design, with tonight's measurements (refs #132) 2aa1854b0d13
docs/lazy-clone.md: a lazy working copy over a FUSE mount of the state with a writable overlay, built on what exists (shallow checkpoints, lazy blobs, flow-casfs); status, diff and commit from the overlay; offline, fallback, protocol and trust. Measured on Earth against git clone and git's blobless clone, with the profile of a depth-0 clone (#137).
- Author
- Andy McCall @andymccall <[email protected]>
- via
- agent
agent:claude@andymccall <[email protected]> · keyless, under your grant for changes,refs,seal · key b44592b41e2f
- When
- · 2026-10-25 01:27 UTC
- Signature
- ◈ sealed by agent:claude@andymccall
- Hash
2aa1854b0d1311b330db24ec73e364035eb41f5eb80c32e2edd701aa316341a3- Depends on
- 7bd805ffb418
- Depended on by
- f3811eb20bea
- In bookmarks
- identical-sides main push-notes subrepo-pin-state
- In tags
- v0.10.0 v0.11.0 v0.6.0 v0.7.0 v0.8.0 v0.9.0
- Edits
- 6 atoms, 0 tree operations, 1 file
Browse the tree at this change
modified docs/lazy-clone.md +42 −19
| … | 135 unchanged lines |
||
| 136 | 136 | ||
| 137 | 137 | ## 9. Measurements |
|
| 138 | 138 | ||
| 139 | - | On Earth, with the VS Code history (`~/bench/vscode-full9.oxy`, 166k |
|
| 140 | - | changes, served by `oxy serve` on the LAN so the network is the same for |
|
| 141 | - | every case), each timed cold (`echo 3 > drop_caches` not available: a |
|
| 142 | - | fresh clone directory each time, and the server warm), median of three: |
|
| 139 | + | On Earth (4 cores, SSD), 2026-10-25, the VS Code history |
|
| 140 | + | (`~/bench/vscode-full9.oxy`, 166k changes; `~/bench/vscode.git` for Git) |
|
| 141 | + | served over loopback by `oxy serve` and `git daemon`, oxy at main |
|
| 142 | + | (0.5.1 + #137), Git 2.53; one run each, a fresh directory, the server |
|
| 143 | + | warm (its checkpoint of the heads already made). Seconds: |
|
| 143 | 144 | ||
| 144 | - | | Step | git clone | oxy clone | oxy clone --depth 0 | oxy clone --lazy | |
|
| 145 | + | | Step | git clone | git clone --filter=blob:none | oxy clone --depth 0 | oxy clone --depth 0 --max-blob 0 | |
|
| 145 | 146 | |---|---|---|---|---| |
|
| 146 | - | | clone | | | | | |
|
| 147 | - | | `ls -R \| wc -l` | | | | | |
|
| 148 | - | | read one file (`src/vs/code/electron-main/main.ts`) | | | | | |
|
| 149 | - | | edit it, `status` | | | | | |
|
| 150 | - | | `diff` | | | | | |
|
| 151 | - | | `commit` | | | | | |
|
| 152 | - | | `push` (to a fresh bare repo) | | | | | |
|
| 153 | - | | bytes over the wire | | | | | |
|
| 154 | - | | disk after | | | | | |
|
| 147 | + | | clone | 91.4 | 30.5 | 457.9 | 849.0 | |
|
| 148 | + | | `ls -R \| wc -l` | 0.11 | 0.11 | 0.54 | 0.11 | |
|
| 149 | + | | read `src/vs/code/electron-main/main.ts` | 0.01 | 0.01 | 0.01 | 0.01 | |
|
| 150 | + | | append a line, `status` | 0.18 | 0.72 | 8.07 | 8.50 | |
|
| 151 | + | | `diff` | 0.10 | 0.03 | 0.23 | 0.23 | |
|
| 152 | + | | `commit` | 0.12 | 0.05 | 0.98 | 0.92 | |
|
| 153 | + | | disk after | 1.7 GB | 693 MB | 977 MB | 979 MB | |
|
| 155 | 154 | ||
| 156 | - | Git is `git clone --no-local` from a `git daemon` of `~/bench/vscode.git` |
|
| 157 | - | (and `--filter=blob:none` as a second row, Git's own lazy clone). The |
|
| 158 | - | first three oxy columns can be measured now; they set the bar the |
|
| 159 | - | prototype must clear: the lazy clone should finish in seconds and its |
|
| 160 | - | first `status` in milliseconds, and nothing else may get slower. |
|
| 155 | + | Before #137 the depth-0 clone failed after 11 minutes (the state's binary |
|
| 156 | + | files' blobs came with nothing) and, before that fix's other half, wrote |
|
| 157 | + | the checkpoint object by object: stopped after 38 minutes with 57,498 |
|
| 158 | + | loose objects. The full `oxy clone` (12 GB of history) was not run |
|
| 159 | + | tonight. |
|
| 161 | 160 | ||
| 161 | + | What this says: |
|
| 162 | + | ||
| 163 | + | - A depth-0 clone is five times slower than Git's full clone and fifteen |
|
| 164 | + | times slower than its blobless one. The profile (`OXY_TIMINGS=1`, one |
|
| 165 | + | run of 508 s): about 375 s fetching, because the imported history's |
|
| 166 | + | `main` has 33,835 heads and depth 0 still takes the head changes |
|
| 167 | + | themselves (33,837 changes, with their texts); 12 s decoding the |
|
| 168 | + | checkpoint; 120 s of checkout, 111 s of it rendering 19,545 files on one |
|
| 169 | + | core. So depth 0 should take no change at all (the checkpoint carries |
|
| 170 | + | every change's metadata; spec §17's "a checkpoint plus zero history"), |
|
| 171 | + | and rendering should use every core (oxygen#137). |
|
| 172 | + | - `--max-blob 0` without a mount is slower, not faster: the checkout then |
|
| 173 | + | fetches every binary file's blob one request at a time. Lazy contents |
|
| 174 | + | only pay with a mount that reads them when asked. |
|
| 175 | + | - The first `status` after a clone is 8 s (it builds the manifest); Git's |
|
| 176 | + | is 0.2 s. A lazy clone's `status` from the overlay (§3.4) would not |
|
| 177 | + | depend on the tree at all. |
|
| 178 | + | ||
| 179 | + | So the prototype's bar: clone in seconds (refs, the checkpoint skeleton |
|
| 180 | + | without graphs, the mount), `ls -R` at once from the payload's tree, a |
|
| 181 | + | first read in tens of milliseconds, `status` in milliseconds; and, |
|
| 182 | + | whatever happens to `--lazy`, the depth-0 clone itself should come down |
|
| 183 | + | to Git's order (streamed checkpoint in parts, rendering on every core). |
|
| 184 | + | ||
| 162 | 185 | ## 10. The prototype (L2) |
|
| 163 | 186 | ||
| 164 | 187 | 1. `checkpoint` with `graphs: false` in oxy-core and oxy-server, and a |
|
| … | 8 unchanged lines |