OXYGENHUB
# bd445775b94e oxygen / docs / backlog-overnight-3.md
7.4 KiB · Markdown · last changed Backlog: rebase into the working copy overnight 1 month ago renderedsource blame history raw

Overnight backlog 3 (from 2026-09-29): not worse than Git

Andy's brief: there is no point making a tool that is worse than Git. Until oxy matches Git on a real, large tree, this is the only work: speed, memory and space at scale, with correctness never traded away.

Work top to bottom. An item is done when:

  • its tests pass and the gate is clean;
  • it is measured before and after on Earth, and the numbers are in docs/benchmarks.md;
  • it is committed with oxy as Andy and pushed to Earth.

If an item is too large, split it here and do the first part. Log each iteration at the bottom.

The yardstick

The Linux tip on Earth: 95 943 files, 1.8 GB, on 4 cores with 31 GiB. Git 2.53 does this (from ~/bench/linux/results.txt, 2026-09-29):

OperationGitTarget for oxy
First commit of the tree33.7 s, 42 MBat most 50 s, under 2 GB
Clean status (warm)0.26 to 0.31 s, 31 MBat most 0.4 s, under 300 MB
status, one file edited0.25 sat most 0.4 s
diff, one file edited0.34 sat most 0.5 s
commit, one file edited0.43 sat most 0.6 s
status, 1 000 files edited0.26 sat most 0.5 s
diff, 1 000 files edited1.38 sat most 2 s
commit, 1 000 files edited0.76 sat most 1.2 s
log -n 1000.05 sat most 0.1 s
blame MAINTAINERS0.07 sat most 0.2 s
Checkout of another state and back0.49 s and 0.38 sat most 1 s each
Store after gc311 MB (.git)at most 350 MB
Everyday commandsabout 31 MBunder 300 MB

The targets allow oxy a small margin, because it does more (conflicts as data, signatures, an operation log). A target that turns out to be out of reach for a structural reason is written up here, with the best number reached and why. It is never quietly dropped.

Rules for this loop

  • Measure first. Before changing code, measure where the time or memory goes on Earth: /usr/bin/time -v, perf record (installed on Earth), OXY_TIMING phases, or a counting allocator behind a feature. Record what the profile showed in the log.
  • Heavy work only on Earth, always under a cap: systemd-run --user --scope -p MemoryMax=14G …. Andy's workstation has crashed from memory before. Locally, run the gate and small cases only.
  • Getting code to Earth. ~/bench/oxygen on Earth is a plain copy of the source. Sync it with rsync -a --delete --exclude target --exclude .oxy -e "ssh -i ~/.ssh/earth_claude" ~/development/oxygen/ claude@earth:bench/oxygen/.
    • ~/bench/linux-bench.sh builds oxy from that copy and runs the whole comparison. ~/bench/first-commit.sh runs only the first commit, status and fsck.
    • Run long jobs with Bash run_in_background, and wait for the completion notice rather than polling.
  • Correctness is never traded. Every change keeps these passing: oxy fsck on the Linux repository, the property tests, and oxy git verify on the Git imports.
  • Format changes (change, checkpoint, pack or wire) are allowed before 1.0, with these steps:
    • bump the spec draft and keep old objects readable with unchanged hashes;
    • bump deps/oxygen in oxygenflow with make pins, gate, commit, push and make deploy-runner;
    • then make pins in oxygenhub, gate, commit, push, make deploy, and check /health;
    • only then install the new oxy locally. Commit the format change itself with the old binary.
  • Memory and pace. Every workstation build and test runs under systemd-run --user --scope -p MemoryMax=8G, and CARGO_INCREMENTAL=0 (the disk is nearly full).

Memory

  • M0 Tools to see memory. Add a counting global allocator behind a cargo feature (peak bytes and allocation count per OXY_TIMING phase), so a profile names the structure that holds the memory. Then profile a clean status of the Linux repository, killed at 14 GB (oxygen#35), and the first commit (9.9 GB). Write down which structures hold what before changing anything.
  • M1 Status without every graph (oxygen#35). A clean status must not build every file graph. Unchanged files are skipped by the manifest (spec §12.4), and the first status after a commit must not replay the whole change. Options: the commit writes a checkpoint; history-only states load graphs on demand (State::load_graphs). Target: under 300 MB and 0.4 s.
  • M2 Lines without an allocation each. An insert's lines become one shared buffer with line offsets (for example Arc<[u8]> and ranges) in the change and in the file graph. The file graph shares that buffer and does not copy it. AtomInfo and the edge map shrink. Measure bytes per line before and after.
  • M3 A commit that streams. The snapshot builder stores each file's text blob as it reads it and keeps no content it no longer needs, and the state is built without a second copy. Target for the first commit: under 2 GB.
  • M4 Everyday commands under 300 MB on the Linux repository: status, diff, commit, log and blame, one file and 1 000 files edited.

Speed

  • S1 First commit at most 50 s (55.3 s now): profile with perf, and parallelise reading, hashing and diffing across the four cores where it is safe.
  • S2 Status, diff and commit with one and with 1 000 files edited, to the targets above.
  • S3 log, blame and checkout on the Linux repository, to the targets above.
  • S4 fsck and gc measured against Git's. Re-run Git's fsck with gc turned off: 451 s is suspect, because Git's own background gc raced it.

Space

  • P1 Store at most 350 MB after gc (442 MB now against Git's 311 MB): measure what takes the space (changes, text blobs, graphs, checkpoints, indexes), then fix the largest. Compression level and dictionary, and packing order, are all candidates.

History at scale

  • H1 Importer name conflict (oxygen#30): the 5 000-commit vscode import differs from Git in 51 files. Fix the cause and add a test. oxy git verify must be clean.
  • H2 Import memory: drop graphs untouched by recent commits and reload them on demand, so memory is bounded by the working set and not by the history.
  • H3 Linux, stage 2: import slices of recent history (2 000, 10 000 and 50 000 commits), each verified, then time log, blame and status against Git on the same slice.
  • H4 vscode's full history (about 130 000 commits) into a scratch repository on Earth. It is not published. Verify it and time it.
  • H5 Linux's full history (about 1.3 million commits) if H3 shows it can finish overnight. Otherwise, write up what stands in the way.

From the feedback (2026-09-30)

  • R1 Rebase into the working copy (oxygen#42, Andy chose the overnight loop): when oxy rebase CHANGE --onto TARGET would be conflicted, offer to leave the conflict in the working copy to resolve and commit, as merge does, instead of refusing. Tests for a stacked proposal brought up to date this way.

Finally

  • Z1 Re-run the full ~/bench/linux-bench.sh with the final build, and put the complete Git against oxy table in docs/benchmarks.md. Summarise, for Andy, where oxy now matches or beats Git and where it does not.

Log

  • 2026-09-29 Backlog written. Starting point: the Linux first commit succeeds (55.3 s, 9.9 GB, 442 MB store) with change format 2 (oxygen#34). A clean status is killed at 14 GB (oxygen#35).