Overnight backlog 3 (from 2026-09-29): not worse than Git

Andy's brief: there is no point making a tool that is worse than Git. Until oxy matches Git on a real, large tree, this is the only work: speed, memory and space at scale, with correctness never traded away.

Work top to bottom. An item is done when:

• its tests pass and the gate is clean;
• it is measured before and after on Earth, and the numbers are in docs/benchmarks.md;
• it is committed with oxy as Andy and pushed to Earth.

If an item is too large, split it here and do the first part. Log each iteration at the bottom.

The yardstick

The Linux tip on Earth: 95 943 files, 1.8 GB, on 4 cores with 31 GiB. Git 2.53 does this (from ~/bench/linux/results.txt, 2026-09-29):

The targets allow oxy a small margin, because it does more (conflicts as data, signatures, an operation log). A target that turns out to be out of reach for a structural reason is written up here, with the best number reached and why. It is never quietly dropped.

Rules for this loop

• Measure first. Before changing code, measure where the time or memory goes on Earth: /usr/bin/time -v, perf record (installed on Earth), OXY_TIMING phases, or a counting allocator behind a feature. Record what the profile showed in the log.
• Heavy work only on Earth, always under a cap: systemd-run --user --scope -p MemoryMax=14G …. Andy's workstation has crashed from memory before. Locally, run the gate and small cases only.
• Getting code to Earth. ~/bench/oxygen on Earth is a plain copy of the source. Sync it with rsync -a --delete --exclude target --exclude .oxy -e "ssh -i ~/.ssh/earth_claude" ~/development/oxygen/ claude@earth:bench/oxygen/.
  • ~/bench/linux-bench.sh builds oxy from that copy and runs the whole comparison. ~/bench/first-commit.sh runs only the first commit, status and fsck.
  • Run long jobs with Bash run_in_background, and wait for the completion notice rather than polling.
• Correctness is never traded. Every change keeps these passing: oxy fsck on the Linux repository, the property tests, and oxy git verify on the Git imports.
• Format changes (change, checkpoint, pack or wire) are allowed before 1.0, with these steps:
  • bump the spec draft and keep old objects readable with unchanged hashes;
  • bump deps/oxygen in oxygenflow with make pins, gate, commit, push and make deploy-runner;
  • then make pins in oxygenhub, gate, commit, push, make deploy, and check /health;
  • only then install the new oxy locally. Commit the format change itself with the old binary.
• Memory and pace. Every workstation build and test runs under systemd-run --user --scope -p MemoryMax=8G, and CARGO_INCREMENTAL=0 (the disk is nearly full).

Correctness first (from metalgear, 2026-09-29)

A working copy that silently becomes the wrong tree is worse than any slowness, so these come before everything else.

• C1 Working-copy writes hold the workspace lock (oxygen#36). Checkout, bookmark switch (with --carry), pull's checkout, undo, discard and commit hold the lock from reading the head until their operation is recorded. merge_views never lets a working-copy change based on a replaced one win. Test: a snapshot concurrent with a checkout never records the checkout's writes as edits.
• C2 bookmark create NAME CHANGE refuses a partial state (oxygen#37). When CHANGE is one of the current state's heads but not all of them, it refuses and explains, unless given --only. bookmark switch summarises a wide tree difference. The guide explains that bookmarks name states, not ancestry.
• C3 A parent's switch never rolls a subrepo's bookmark back (oxygen#38).
• C4 pull says "Already up to date" while one behind (oxygen#29). Reproduce it with a test, then fix it.

Memory

• M0 Tools to see memory. Add a counting global allocator behind a cargo feature (peak bytes and allocation count per OXY_TIMING phase), so a profile names the structure that holds the memory. Then profile a clean status of the Linux repository, killed at 14 GB (oxygen#35), and the first commit (9.9 GB). Write down which structures hold what before changing anything.
• M1 Status without every graph (oxygen#35). A clean status must not build every file graph. Unchanged files are skipped by the manifest (spec §12.4), and the first status after a commit must not replay the whole change. Options: the commit writes a checkpoint; history-only states load graphs on demand (State::load_graphs). Target: under 300 MB and 0.4 s.
• M2 Lines without an allocation each. An insert's lines become one shared buffer with line offsets (for example Arc<[u8]> and ranges) in the change and in the file graph. The file graph shares that buffer and does not copy it. AtomInfo and the edge map shrink. Measure bytes per line before and after.
• M3 A commit that streams. The snapshot builder stores each file's text blob as it reads it and keeps no content it no longer needs, and the state is built without a second copy. Target for the first commit: under 2 GB.
• M4 Everyday commands under 300 MB on the Linux repository: status, diff, commit, log and blame, one file and 1 000 files edited.

Speed

• S1 First commit at most 50 s (55.3 s now): profile with perf, and parallelise reading, hashing and diffing across the four cores where it is safe.
• S2 Status, diff and commit with one and with 1 000 files edited, to the targets above.
• S3 log, blame and checkout on the Linux repository, to the targets above.
• S4 fsck and gc measured against Git's. Re-run Git's fsck with gc turned off: 451 s is suspect, because Git's own background gc raced it.

Space

• P1 Store at most 350 MB after gc (442 MB now against Git's 311 MB): measure what takes the space (changes, text blobs, graphs, checkpoints, indexes), then fix the largest. Compression level and dictionary, and packing order, are all candidates.

History at scale

• H1 Importer name conflict (oxygen#30): the 5 000-commit vscode import differs from Git in 51 files. Fix the cause and add a test. oxy git verify must be clean.
• H2 Import memory: drop graphs untouched by recent commits and reload them on demand, so memory is bounded by the working set and not by the history.
• H3 Linux, stage 2: import slices of recent history (2 000, 10 000 and 50 000 commits), each verified, then time log, blame and status against Git on the same slice.
• H4 vscode's full history (about 130 000 commits) into a scratch repository on Earth. It is not published. Verify it and time it.
• H5 Linux's full history (about 1.3 million commits) if H3 shows it can finish overnight. Otherwise, write up what stands in the way.

Finally

• Z1 Re-run the full ~/bench/linux-bench.sh with the final build, and put the complete Git against oxy table in docs/benchmarks.md. Summarise, for Andy, where oxy now matches or beats Git and where it does not.

Log

• 2026-09-29 Backlog written. Starting point: the Linux first commit succeeds (55.3 s, 9.9 GB, 442 MB store) with change format 2 (oxygen#34). A clean status is killed at 14 GB (oxygen#35).
• 2026-09-29 C1 to C4 done (the metalgear findings in ~/oxygen_feedback.md, items 22 and 25 to 26). The gate passes: 205 tests, clippy clean.
  • C1 (e8e51993, closes #36): the working-copy lock now covers checkout, switch (and --carry), merge, commit, amend and rewrites, discard and pull. It is re-entrant per thread, waits up to a minute, and a live process's lock is never taken over. A workspace changed by two concurrent operations is now an error. A race test runs twenty switches against a snapshot thread; without the lock it failed one run in three.
  • C2 (bf423e78, closes #37): bookmark create NAME CHANGE refuses when it would leave out earlier work, unless given --upto or --only. bookmark switch says how many changes the new state lacks. The guide explains states for people coming from Git.
  • C3 (6b582c4c, closes #38): a parent's checkout moves a subrepo's bookmark only forward, and the pin is read from the nested working copy. --carry leaves a subrepo whose pin is carried work as it is.
  • C4 (5dfea7b7, closes #29): not a sync bug. rstar-api's operation log shows each such pull ran on a feature bookmark, while main, which the fetch had just updated, was the one behind. pull now names the bookmark it pulled and the others left behind.
  • For Andy: metalgear's working copy is sound, on develop and clean. logo-25 holds just the logo change and its dependencies; delete it if it is not wanted.
• 2026-09-30 M0 and M1 done; M4, S2 and S3 part way (oxygen#35 and oxygen#39). The tests and clippy pass after each step. Each step was measured on the Linux tip and is recorded in docs/benchmarks.md.
  • M0 (f11e81a1): a counting allocator behind the alloc-stats feature; with OXY_TIMINGS it reports memory per phase.
  • M1, with these steps:
    • history-only loads read no text (22b70898), then read only a change's skeleton (39c6adea);
    • graph keys are written at commit (22b70898);
    • a clean status is answered from the manifest (b4fefa1e) with a parallel walk (e19aa6c5).
    • Clean status: killed at 14 GB before, now 0.44 s and 123 MB (Git: 0.22 s).
  • History-only loads where no graph is needed (26f81555): bookmark, merge, pull, tags, bisect, doctor and most history rewrites.
  • Checkout: a batch at a time (921fa890), without a snapshot when clean (d1b480f0), and against the manifest without the base state (2b7ce2c6).
    • One-file switch: killed before, now 2.1 s and 455 MB (Git: 0.2 s).
    • Local clone: killed before, now 110 s and 8 GB (Git: 9.6 s).
  • log, blame and fsck (34ea4ace), with metadata-only change reads:
    • log -n 100: 28 s and 4.7 GB before, now 2.0 s;
    • blame: killed before, now 1.1 s;
    • fsck: killed before, now 123 s and 4.5 GB.
  • Snapshot: a parallel walk with stats carried into the manifest (a578fa19), and a cached tree view (de3f07f8). Status with one file changed: 4.1 s before, now 2.3 s (Git: 0.23 s).
  • Still far from Git: one-file and 1 000-file status, diff and commit (2 to 5 s); switch (2 s); log (2 s); clone (110 s, 8 GB); first commit (56 s, 10 GB); fsck (4.5 GB).
    • The common cost is loading the first change's history (0.5 s) and building the tree view (0.4 s).
    • Next: a persistent per-state cache (tree view, graph keys and change set), Git's index in effect, then compact graphs (M2) and a streaming commit (M3).
• 2026-09-30 M2 done, M3 and S1 part way (oxygen#35). The first commit of the Linux tip now takes 40.7 to 41.6 s on a quiet Earth (S1 target: 50 s; Git: 34 to 38 s) and 4.9 GB RSS (from 10 GB).
  • Encoding (b7103633): each change is encoded once, the working copy's edits are moved rather than copied, and text blobs and chunk compression run on every core.
  • Storing (b7a090c7): texts go to the store a batch at a time.
  • M2, lines (086641a8): an insert's lines are one shared buffer, into which graph atoms point.
  • Clone (5d62fa66, 11bf3b00, 397d9f98): a local clone takes 48 s and 3.9 GB (from 96 s and 5.2 GB, and killed before that; Git: 9.6 s). The changes behind it:
    • an unedited file's graph is read off as a chain;
    • a bundle carries its changes' hashes;
    • a large change is read once per checkout;
    • a local fetch moves changes as stored.
  • Hub and runner: re-pinned to oxygen 397d9f98 and deployed (oxygenhub#58, a6f006ba; oxygenflow#15, 630b7fd6). The hub reads metadata or skeletons where it lists changes. Checked: /health, and the log, file history, tree and file pages.
  • Note: my pushes start Flow jobs on Earth, so timings under load are marked in benchmarks.md.
• 2026-09-30 H1 done (oxygen#30). Three causes, each from Git having no identity for a directory:
  • a directory deleted and made again became a second node;
  • a placement under a removed directory was out of reach;
  • orphans could not be seen.
  • The fix: an import revives the directory, and a merge retires a kept node's other placements and every orphan. There is a Git-built test for each cause; each reproduces the vscode symptom without its fix. Spec §23.
  • The vscode slice now verifies: all 1 747 commits and 3.3 million files.
  • oxy git verify is cached by graph key and blob: 115 s instead of 1 400 s.
• 2026-09-30 H2, first part: an import builds graphs only for the files a commit may change, from history-only states. The first 5 000 vscode commits: 309 s and 2.3 GB (from 412 s and 6.3 GB), and all verified. Next: verify and import reuse each commit's history instead of loading it again, before H3 to H5.