OXYGENHUB

Benchmarks: the Linux store in change format 3 36695c5e02ef

The Linux tip's store is 324 MB after gc (objects 306 MB; Git's 311 MB)
and its first commit takes 32.6 s (Git 33.7 to 37.9 s). Backlog 3: P1
and S1 done.

closes #40
Author
Andy McCall @andymccall <[email protected]>
When
· 2026-09-30 02:56 UTC
Signature
✔ signed by Andy McCall
Hash
36695c5e02efe40e62f8fdc51e445000e2ea01664a9bafef468229d395343bf2
Depends on
1681323d1dff 1d360c91a289 33e77a5526fa b493d993a9df da652d6392d8
Depended on by
2517abe64654
In bookmarks
agents-in-readme bare-hashes carried-not-removed carry-identical doctor-trust identical-sides keyless-tags main merge-remote-bookmark only-help-103 paths-for-resolutions-and-amend push-notes show-stat subrepo-pin-state switch-says
In tags
v0.1.0 v0.10.0 v0.11.0 v0.2.0 v0.3.0 v0.3.1 v0.4.0 v0.4.1 v0.4.2 v0.4.3 v0.4.4 v0.4.5 v0.4.6 v0.5.0 v0.5.1 v0.6.0 v0.7.0 v0.8.0 v0.9.0
Edits
6 atoms, 0 tree operations, 2 files

Browse the tree at this change

show whitespace changes
modified docs/backlog-overnight-3.md +7 −2
… 
111 unchanged lines
112 112 

    

  
    

      113
      113
       
      
## Speed

    

  
    

      114
      114
       
      

    

  
    

      115
      
      -
      
- [ ] S1 **First commit at most 50 s** (55.3 s now): profile with

    

  
    

      
      115
      +
      
- [x] S1 **First commit at most 50 s** (55.3 s now): profile with

    

  
    

      116
      116
       
      
  `perf`, and parallelise reading, hashing and diffing across the four

    

  
    

      117
      117
       
      
  cores where it is safe.

    

  
    

      118
      118
       
      
- [ ] S2 **Status, diff and commit** with one and with 1 000 files

    

  
    

      
      
      …
      
6 unchanged lines

    

  
    

      125
      125
       
      

    

  
    

      126
      126
       
      
## Space

    

  
    

      127
      127
       
      

    

  
    

      128
      
      -
      
- [ ] P1 **Store at most 350 MB after gc** (442 MB now against Git's

    

  
    

      
      128
      +
      
- [x] P1 **Store at most 350 MB after gc** (442 MB now against Git's

    

  
    

      129
      129
       
      
  311 MB): measure what takes the space (changes, text blobs, graphs,

    

  
    

      130
      130
       
      
  checkpoints, indexes), then fix the largest. Compression level and

    

  
    

      131
      131
       
      
  dictionary, and packing order, are all candidates.

    

  
    

      
      
      …
      
70 unchanged lines

    

  
    

      202
      202
       
      
  - The vscode slice now verifies: all 1 747 commits and 3.3 million files.

    

  
    

      203
      203
       
      
  - `oxy git verify` is cached by graph key and blob: 115 s instead of 1 400 s.

    

  
    

      204
      204
       
      
- 2026-09-30 **H2, first part**: an import builds graphs only for the files a commit may change, from history-only states. The first 5 000 vscode commits: 309 s and 2.3 GB (from 412 s and 6.3 GB), and all verified. Next: verify and import reuse each commit's history instead of loading it again, before H3 to H5.

    

  
    

      
      205
      +
      
- 2026-09-30 **P1 and S1 done** (oxygen#40).

    

  
    

      
      206
      +
      
  - **Repack**: `oxy gc` repacks everything into one pack at zstd level 9, compressing on every core (9f7585d4).

    

  
    

      
      207
      +
      
  - **Format 3**: text blobs hold the lines' bytes, with no CBOR framing (1122dcb3). The runner (oxygenflow#16) and hub (oxygenhub#59) were deployed with the new reader first.

    

  
    

      
      208
      +
      
  - **Linux tip**: the store is 324 MB after gc (objects 306 MB; Git 311 MB). The first commit takes 32.6 s (Git 33.7 to 37.9 s), measured while Earth ran at load average 11 during the vscode import.

    

  
    

      
      209
      +
      
  - **H2**: the vscode import (H4) runs on Earth as a service.

    

  
    

      205
      210
       
      

    

  
  

  





  
    modified
    docs/benchmarks.md
    +9 −2
  
  
  

  
    

      
      
      …
      
483 unchanged lines

    

  
    

      484
      484
       
      
  46 308 files of 4 KiB or more.

    

  
    

      485
      485
       
      

    

  
    

      486
      486
       
      
`oxy gc` now repacks everything into one pack at zstd level 9, compressed

    

  
    

      487
      
      -
      
on every core: 357 MB (349 MB pack) in 52 s. The text blob framing is the

    

  
    

      488
      
      -
      
next step (a change format version 3 with raw text blobs).

    

  
    

      
      487
      +
      
on every core: 357 MB (349 MB pack) in 52 s. Change format 3 then stores

    

  
    

      
      488
      +
      
a text blob as the lines' bytes:

    

  
    

      
      489
      +
      

    

  
    

      
      490
      +
      
| Linux tip | Store after gc | First commit |

    

  
    

      
      491
      +
      
|---|---:|---:|

    

  
    

      
      492
      +
      
| Format 2, gc at level 3 | 390 MB | 41 s |

    

  
    

      
      493
      +
      
| Format 2, gc repacks at level 9 | 357 MB | |

    

  
    

      
      494
      +
      
| Format 3 (Earth at load average 11) | 324 MB (objects 306 MB) | 32.6 s, 4.4 GB |

    

  
    

      
      495
      +
      
| Git | 311 MB (.git) | 33.7 to 37.9 s, 42 MB |

    

  
    

      489
      496