OXYGENHUB

Benchmarks: Linux stage 1 with change format 2 2951f8023f40

refs #34
Author
Andy McCall @andymccall <[email protected]>
When
· 2026-09-29 19:54 UTC
Signature
✔ signed by Andy McCall
Hash
2951f8023f4062b2a70c2ff3a12e7c75784336614de339a33a1b5f3a9f1c555e
Depends on
1d360c91a289 75036530b621 958fb27dda7c a405d4c99440 a707df24e576
Depended on by
86e1824f2d0b 33e77a5526fa
In bookmarks
agents-in-readme bare-hashes carried-not-removed carry-identical doctor-trust identical-sides keyless-tags main merge-remote-bookmark only-help-103 paths-for-resolutions-and-amend push-notes show-stat subrepo-pin-state switch-says
In tags
v0.1.0 v0.10.0 v0.11.0 v0.2.0 v0.3.0 v0.3.1 v0.4.0 v0.4.1 v0.4.2 v0.4.3 v0.4.4 v0.4.5 v0.4.6 v0.5.0 v0.5.1 v0.6.0 v0.7.0 v0.8.0 v0.9.0
Edits
4 atoms, 0 tree operations, 2 files

Browse the tree at this change

hide whitespace changes
modified docs/backlog-overnight-2.md +5 −1
… 
130 unchanged lines
131 131 
  - H12d Queue view (why each job waits) and the runners API.
132 132 
  - H12e Bring up Plex (x86_64) as a second microVM runner, and a VisionFive 2 Lite (riscv64). The JH7110 has no hypervisor extension, so it needs a non-microVM tier first. Andy will say how to reach them.
133 133 
- [x] H10 `oh`, a command-line client for OxygenHub like GitHub's `gh`: a crate `crates/oh` in the oxygenhub workspace on hub-api. First slice: `oh auth login`, `oh issue list|view|create`, `oh proposal list|view|create|land`, `oh run list|view|watch`, `oh release list|create|upload`, with `--json`.
134 - 
- [ ] L1 **Large inserts as blobs** (oxygen#34): an Insert over a threshold carries a chunked blob, not its lines, so a change can hold a whole tree. Builders store content as they read it, bounding memory. Spec §6.1/§7; format change, allowed before 1.0.
134 + 
- [x] L1 **Large inserts as blobs** (oxygen#34): change format 2 stores inserts of 4 KiB or more as text blobs (c8a26be3, spec Draft 0.8q); hub db750ff1 and runner 6cbadaca were deployed first. The Linux first commit now succeeds: 55 s, 9.9 GB, 442 MB store.
135 + 
- [ ] L1b **Memory for a whole tree**: a clean `status` of the Linux tree is killed at a 14 GB cap, and the commit peaks at 9.9 GB (Git: 42 MB). Every line is held as its own `Vec` in the change and again in the file graph. Next: find where status spends memory, then load graphs on demand and keep line bytes shared, not copied.
135 136 
- [ ] L2 **Linux, stage 1** (Andy): Git vs oxy on the Linux tip, with `~/bench/linux-bench.sh` on Earth (Andy runs it). Needs L1. Results go in docs/benchmarks.md.
136 137 
- [ ] L3 **Linux, stage 2**: import slices of recent history (2k, 10k, 50k commits) with verify. Needs L1, oxygen#30 and memory-bounded import (drop untouched graphs, reload on demand).
137 138 
- [ ] L4 **Linux, stage 3**: the full 1.3M-commit import, overnight on Earth.
… 
677 unchanged lines
815 816 
  - **oxy:** the first commit made one 1.8 GB change object (every line inline), peaked at 10.9 GB, and was packed despite the 1 GiB object limit, leaving a repository that would not open. The refusal is now fixed (oxygen#33); the design limit is oxygen#34 (L1).
816 817 
  - **Git's `fsck` errors:** Git's own automatic gc was running concurrently; the script now turns it off.
817 818 
  - Re-run after L1.
819 + 
- 2026-09-29 **L1 done**, change format 2 (oxygen#34, c8a26be3). On Earth, the Linux tip (95 943 files, 1.8 GB):
820 + 
  - **First commit:** 55.3 s, 9.9 GB peak, 442 MB store. Before this, the commit failed and left a repository that would not open.
821 + 
  - **Clean `status`:** killed by the 14 GB cap. That is L1b.
modified docs/benchmarks.md +21 −0
… 
252 unchanged lines
253 253 

    

  
    

      254
      254
       
      
Next: rendering the base is still O(files) per commit, and memory grows

    

  
    

      255
      255
       
      
with the kept states. The full import follows.

    

  
    

      
      256
      +
      

    

  
    

      
      257
      +
      
## Linux, stage 1: the tip as one commit (2026-09-29)

    

  
    

      
      258
      +
      

    

  
    

      
      259
      +
      
The Linux tree at its tip: 95 943 files, 1.8 GB, on Earth (4 cores,

    

  
    

      
      260
      +
      
31 GiB), each run capped at 14 GB.

    

  
    

      
      261
      +
      

    

  
    

      
      262
      +
      
| | Git | oxy before | oxy, change format 2 |

    

  
    

      
      263
      +
      
|---|---:|---:|---:|

    

  
    

      
      264
      +
      
| First commit | 33.7 s, 42 MB | failed, 10.9 GB | 55.3 s, 9.9 GB |

    

  
    

      
      265
      +
      
| Store | | unreadable | 442 MB |

    

  
    

      
      266
      +
      
| Clean `status` | 0.26 to 0.31 s | | killed at 14 GB |

    

  
    

      
      267
      +
      

    

  
    

      
      268
      +
      
Before format 2 the commit made one change of 1.8 GB. That was over the

    

  
    

      
      269
      +
      
1 GiB an object may be, and the repository would not open afterwards

    

  
    

      
      270
      +
      
(oxygen#33, oxygen#34). Format 2 stores each insert of 4 KiB or more as

    

  
    

      
      271
      +
      
a text blob, so the change stays small.

    

  
    

      
      272
      +
      

    

  
    

      
      273
      +
      
Memory is the next problem. Every line is kept as its own allocation,

    

  
    

      
      274
      +
      
once in the change and again in the file graph, and `status` builds

    

  
    

      
      275
      +
      
every graph (L1b).

    

  
    

      
      276
      +