# Overnight backlog 3 (from 2026-09-29): not worse than Git

Andy's brief: there is no point making a tool that is worse than Git.
Until oxy matches Git on a real, large tree, this is the only work:
speed, memory and space at scale, with correctness never traded away.

Work top to bottom. An item is done when:
- its tests pass and the gate is clean;
- it is measured before and after on Earth, and the numbers are in
  `docs/benchmarks.md`;
- it is committed with `oxy` as Andy and pushed to Earth.

If an item is too large, split it here and do the first part. Log each
iteration at the bottom.

## The yardstick

The Linux tip on Earth: 95 943 files, 1.8 GB, on 4 cores with 31 GiB.
Git 2.53 does this (from `~/bench/linux/results.txt`, 2026-09-29):

| Operation | Git | Target for oxy |
|---|---:|---:|
| First commit of the tree | 33.7 s, 42 MB | at most 50 s, under 2 GB |
| Clean `status` (warm) | 0.26 to 0.31 s, 31 MB | at most 0.4 s, under 300 MB |
| `status`, one file edited | 0.25 s | at most 0.4 s |
| `diff`, one file edited | 0.34 s | at most 0.5 s |
| `commit`, one file edited | 0.43 s | at most 0.6 s |
| `status`, 1 000 files edited | 0.26 s | at most 0.5 s |
| `diff`, 1 000 files edited | 1.38 s | at most 2 s |
| `commit`, 1 000 files edited | 0.76 s | at most 1.2 s |
| `log -n 100` | 0.05 s | at most 0.1 s |
| `blame MAINTAINERS` | 0.07 s | at most 0.2 s |
| Checkout of another state and back | 0.49 s and 0.38 s | at most 1 s each |
| Store after gc | 311 MB (`.git`) | at most 350 MB |
| Everyday commands | about 31 MB | under 300 MB |

The targets allow oxy a small margin, because it does more (conflicts
as data, signatures, an operation log). A target that turns out to be
out of reach for a structural reason is written up here, with the best
number reached and why. It is never quietly dropped.

## Rules for this loop

- **Measure first.** Before changing code, measure where the time or
  memory goes on Earth: `/usr/bin/time -v`, `perf record` (installed on
  Earth), `OXY_TIMING` phases, or a counting allocator behind a feature.
  Record what the profile showed in the log.
- **Heavy work only on Earth**, always under a cap:
  `systemd-run --user --scope -p MemoryMax=14G …`. Andy's workstation has
  crashed from memory before. Locally, run the gate and small cases only.
- **Getting code to Earth.** `~/bench/oxygen` on Earth is a plain copy of
  the source. Sync it with `rsync -a --delete --exclude target --exclude
  .oxy -e "ssh -i ~/.ssh/earth_claude" ~/development/oxygen/
  claude@earth:bench/oxygen/`.
  - `~/bench/linux-bench.sh` builds oxy from that copy and runs the whole
    comparison. `~/bench/first-commit.sh` runs only the first commit,
    status and fsck.
  - Run long jobs with Bash `run_in_background`, and wait for the
    completion notice rather than polling.
- **Correctness is never traded.** Every change keeps these passing:
  `oxy fsck` on the Linux repository, the property tests, and
  `oxy git verify` on the Git imports.
- **Format changes** (change, checkpoint, pack or wire) are allowed
  before 1.0, with these steps:
  - bump the spec draft and keep old objects readable with unchanged
    hashes;
  - bump `deps/oxygen` in oxygenflow with `make pins`, gate, commit,
    push and `make deploy-runner`;
  - then `make pins` in oxygenhub, gate, commit, push, `make deploy`,
    and check `/health`;
  - only then install the new oxy locally. Commit the format change
    itself with the old binary.
- **Memory and pace.** Every workstation build and test runs under
  `systemd-run --user --scope -p MemoryMax=8G`, and
  `CARGO_INCREMENTAL=0` (the disk is nearly full).

## Memory

- [ ] M0 **Tools to see memory.** Add a counting global allocator behind
  a cargo feature (peak bytes and allocation count per `OXY_TIMING`
  phase), so a profile names the structure that holds the memory. Then
  profile a clean `status` of the Linux repository, killed at 14 GB
  (oxygen#35), and the first commit (9.9 GB). Write down which
  structures hold what before changing anything.
- [ ] M1 **Status without every graph** (oxygen#35). A clean `status`
  must not build every file graph. Unchanged files are skipped by the
  manifest (spec §12.4), and the first status after a commit must not
  replay the whole change. Options: the commit writes a checkpoint;
  history-only states load graphs on demand (`State::load_graphs`).
  Target: under 300 MB and 0.4 s.
- [ ] M2 **Lines without an allocation each.** An insert's lines become
  one shared buffer with line offsets (for example `Arc<[u8]>` and
  ranges) in the change and in the file graph. The file graph shares
  that buffer and does not copy it. `AtomInfo` and the edge map shrink.
  Measure bytes per line before and after.
- [ ] M3 **A commit that streams.** The snapshot builder stores each
  file's text blob as it reads it and keeps no content it no longer
  needs, and the state is built without a second copy. Target for the
  first commit: under 2 GB.
- [ ] M4 **Everyday commands under 300 MB** on the Linux repository:
  status, diff, commit, log and blame, one file and 1 000 files edited.

## Speed

- [ ] S1 **First commit at most 50 s** (55.3 s now): profile with
  `perf`, and parallelise reading, hashing and diffing across the four
  cores where it is safe.
- [ ] S2 **Status, diff and commit** with one and with 1 000 files
  edited, to the targets above.
- [ ] S3 **log, blame and checkout** on the Linux repository, to the
  targets above.
- [ ] S4 **fsck and gc** measured against Git's. Re-run Git's `fsck`
  with gc turned off: 451 s is suspect, because Git's own background gc
  raced it.

## Space

- [ ] P1 **Store at most 350 MB after gc** (442 MB now against Git's
  311 MB): measure what takes the space (changes, text blobs, graphs,
  checkpoints, indexes), then fix the largest. Compression level and
  dictionary, and packing order, are all candidates.

## History at scale

- [ ] H1 **Importer name conflict** (oxygen#30): the 5 000-commit
  vscode import differs from Git in 51 files. Fix the cause and add a
  test. `oxy git verify` must be clean.
- [ ] H2 **Import memory**: drop graphs untouched by recent commits and
  reload them on demand, so memory is bounded by the working set and not
  by the history.
- [ ] H3 **Linux, stage 2**: import slices of recent history (2 000,
  10 000 and 50 000 commits), each verified, then time log, blame and
  status against Git on the same slice.
- [ ] H4 **vscode's full history** (about 130 000 commits) into a scratch
  repository on Earth. It is not published. Verify it and time it.
- [ ] H5 **Linux's full history** (about 1.3 million commits) if H3 shows
  it can finish overnight. Otherwise, write up what stands in the way.

## Finally

- [ ] Z1 Re-run the full `~/bench/linux-bench.sh` with the final build,
  and put the complete Git against oxy table in `docs/benchmarks.md`.
  Summarise, for Andy, where oxy now matches or beats Git and where it
  does not.

## Log

- 2026-09-29 Backlog written. Starting point: the Linux first commit
  succeeds (55.3 s, 9.9 GB, 442 MB store) with change format 2
  (oxygen#34). A clean `status` is killed at 14 GB (oxygen#35).