Overnight backlog 4: beating Git on the Linux tree

Loop 3 made oxy usable on the Linux kernel. This loop aims to make it faster than Git wherever the model allows it, with the kernel tree as the yardstick. Correctness is never traded for speed: every item keeps its tests, and the test builds that compare an incremental structure with a full computation stay on.

The yardstick

The Linux tip on Earth (~claude/bench/linux, 95 943 files, 1.8 GB): o is the oxy repository and g the Git one. Git is 2.53.

Where one-file status time goes now (Earth, loaded):

• index load: 0.16 to 0.28 s
• walk: 0.45 to 0.7 s
• manifest check before the snapshot: 0.1 s
• placeholders for unchanged files: 0.09 s
• tracked paths: 0.04 s

Rules for this loop

• Measure quietly. Earth runs OxygenFlow jobs, for every push to the Oxygen repositories and for metalgear. Measure when pgrep firecracker finds nothing if you can; otherwise run oxy and Git alternately, three times each, and record the load average. Push your commits after a measurement, not before, since each push starts jobs.
• Build the bench binary from the synced source: rsync -a --delete --exclude target --exclude .oxy ~/development/oxygen/ claude@earth:bench/oxygen/, then CARGO_TARGET_DIR=~/bench/idxbin cargo build --release there. Use ~/bench/prof-status.sh, ~/bench/vs-git.sh and ~/bench/linux-bench.sh, and extend them rather than writing one-off commands.
• Phase timings (OXY_TIMINGS=1) name where time goes. Add phases where they are missing before optimising.
• An index is derived. Every new index or cache is keyed by what it describes, can be rebuilt from the changes at any time, and has a test build check that compares it with a full computation (as the state index, tree view and kept file order do).
• Formats change freely before 1.0, but a change format bump follows the order in loop 3's backlog (old binary commits it; then the runner and hub pins; then make install).

Speed

• B0 Baseline. Update ~/bench/linux-bench.sh for today's build (the state index, the manifest cache, clean and dirty status, diff, commit, log, blame, checkout, a local clone, fsck, gc, store size) with three runs each and oxy and Git alternating. Record the table in docs/benchmarks.md.
• S1 Status and diff at Git's speed (oxygen#41). In order, measuring after each:
  • the manifest as a compact binary file read without a parse per line;
  • the walk: compare each file's stat with the manifest without building a path string per entry (sorted walk against a sorted manifest), and skip the .oxyignore machinery inside tracked directories where it cannot change the answer;
  • no placeholder FileState for unchanged files (audit every user of SnapshotResult.base_snapshot.files first);
  • load the state index lazily: memory-map it and decode a file's deferred history only when that file is built;
  • one pass for status: the manifest check and the snapshot walk share their stats.
• S2 Commit at Git's speed. A commit extends the base it already loaded: write the new state index from the old one plus the change, rather than loading history again, and do not render what the snapshot rendered. Target: one file under 0.5 s, 1 000 files under 0.6 s.
• S3 The change index: log in milliseconds. Per state, the changes in topological order with their generation numbers and where each change's metadata is, built from the parent state's index plus the new change. oxy log -n 100 reads 100 entries. Target: under 0.05 s.
• S4 The node index: blame and file history. Per node, the changes that touch it, so blame and oxy log FILE cost that file's history. Target: blame MAINTAINERS at or under Git.
• S5 Checkout by comparing indexes. Compare the two states' tree indexes by rendered content hash and write only the files that differ. Target: at or under Git.
• S6 fsck at or under Git: parallel object checking and a cheaper state pass.
• S7 The first commit ahead of Git, under 2 GB: stream reading, hashing and storing a batch of files at a time, so the whole tree's text is never in memory at once.

Space

• P1 Store at or under Git after gc (324 MB against 295 MB): measure what takes the space, then try a trained zstd dictionary per repository and packing similar objects together.

Also

• R1 Rebase into the working copy (oxygen#42, Andy's choice for the overnight loop): when oxy rebase CHANGE --onto TARGET would be conflicted, offer to leave the conflict in the working copy to resolve and commit, as merge does, rather than refusing.
• Z1 Closing summary. Re-run the full benchmark and summarise, for Andy, where oxy matches or beats Git on the Linux tree, and where it does not and why.

Log

• 2026-09-30 Backlog written from the day's measurements (state index, changed-only builds and the manifest cache already in).

• 2026-09-30 B0 baseline (table in docs/benchmarks.md; linux-bench.sh now runs three of each, alternating, plus a local clone; the loop-3 script is kept as linux-bench.sh.loop3). oxy is ahead on the first commit (33.3 s against 38.2 s) and gc (48 s against 100 s); clean status is close (0.43 to 0.53 s against 0.34 to 0.46 s). Behind: dirty status and diff (1.5 to 2.2 s against 0.3 to 0.8 s), commit (2.5 to 3.1 s against 0.4 to 0.5 s), log (0.9 s against 0.00 s), blame (1.0 s against 0.07 s), checkout (3 s against 0.4 to 0.5 s), fsck (24 s against 13 s), and a new gap: a local clone takes 56 s and 3.3 GB against Git's 18 s. The store after gc is 375 MB (it now includes the 48 MB state index) against 295 MB.

• 2026-09-30 S1, first round (commits c9dca9ed to 077e899b; table in docs/benchmarks.md). A lister of our own replaces the ignore crate's walker (listing 0.5 s to 0.18 s); the manifest is a flat binary file with hashes and graph keys as bytes; the state index keeps each file's history encoded until used (a writer bug I made on the way was caught only on Earth, since a failed index read falls back to a full load: test builds now read every index written back); maps are built in path order; mimalloc is the allocator. Clean status is now ahead of Git (0.18 s against 0.22 s), the first commit too (24.4 s against 36.2 s), and gc (29.9 s against 70.3 s). One-file status is 0.47 s against 0.22 s. Log (S3, first part): metadata from the state index and a store cache, signature marks without reading changes: 0.20 s, of which 9 ms is the log. For Andy: oxy log checks the working copy first (about 0.18 s here); Git's log does not look at it. Dropping that check would make log about 20 ms, but the working-copy line would then show the last snapshot's state.

• 2026-10-01 00:00 Second round (6359f779 to 81a5bdd7; table in docs/benchmarks.md). Commit 0.67 s (from 0.96 s): the state index is brought up to date by replaying and rewritten only 16 changes behind; keys only for changed paths; attributes from the snapshot rather than a tree walk; the manifest checksummed and written without fsync; and a fix: after a status had seen an earlier edit, a commit kept that edit's stat in the manifest, so every later status took a full snapshot. Blame 0.16 s (from 0.59 s; Git 0.07 s), log 0.18 s, checkout 0.74 s each way (from 1.4 s) with two state indexes kept per workspace, and status right after a checkout that wrote 1 000 files 0.18 s (from 1.3 s). R1 done: oxy rebase --into-working-copy (closes #42).