Backlog 3: not worse than Git 33e77a5526fa

# Overnight backlog 3 (from 2026-09-29): not worse than Git

Andy's brief: there is no point making a tool that is worse than Git.

Until oxy matches Git on a real, large tree, this is the only work:

speed, memory and space at scale, with correctness never traded away.

Work top to bottom. An item is done when:

- its tests pass and the gate is clean;

- it is measured before and after on Earth, and the numbers are in

  `docs/benchmarks.md`;

- it is committed with `oxy` as Andy and pushed to Earth.

If an item is too large, split it here and do the first part. Log each

iteration at the bottom.

## The yardstick

The Linux tip on Earth: 95 943 files, 1.8 GB, on 4 cores with 31 GiB.

Git 2.53 does this (from `~/bench/linux/results.txt`, 2026-09-29):

| Operation | Git | Target for oxy |

|---|---:|---:|

| First commit of the tree | 33.7 s, 42 MB | at most 50 s, under 2 GB |

| Clean `status` (warm) | 0.26 to 0.31 s, 31 MB | at most 0.4 s, under 300 MB |

| `status`, one file edited | 0.25 s | at most 0.4 s |

| `diff`, one file edited | 0.34 s | at most 0.5 s |

| `commit`, one file edited | 0.43 s | at most 0.6 s |

| `status`, 1 000 files edited | 0.26 s | at most 0.5 s |

| `diff`, 1 000 files edited | 1.38 s | at most 2 s |

| `commit`, 1 000 files edited | 0.76 s | at most 1.2 s |

| `log -n 100` | 0.05 s | at most 0.1 s |

| `blame MAINTAINERS` | 0.07 s | at most 0.2 s |

| Checkout of another state and back | 0.49 s and 0.38 s | at most 1 s each |

| Store after gc | 311 MB (`.git`) | at most 350 MB |

| Everyday commands | about 31 MB | under 300 MB |

The targets allow oxy a small margin, because it does more (conflicts

as data, signatures, an operation log). A target that turns out to be

out of reach for a structural reason is written up here, with the best

number reached and why. It is never quietly dropped.

## Rules for this loop

- **Measure first.** Before changing code, measure where the time or

  memory goes on Earth: `/usr/bin/time -v`, `perf record` (installed on

  Earth), `OXY_TIMING` phases, or a counting allocator behind a feature.

  Record what the profile showed in the log.

- **Heavy work only on Earth**, always under a cap:

  `systemd-run --user --scope -p MemoryMax=14G …`. Andy's workstation has

  crashed from memory before. Locally, run the gate and small cases only.

- **Getting code to Earth.** `~/bench/oxygen` on Earth is a plain copy of

  the source. Sync it with `rsync -a --delete --exclude target --exclude

  .oxy -e "ssh -i ~/.ssh/earth_claude" ~/development/oxygen/

  claude@earth:bench/oxygen/`.

  - `~/bench/linux-bench.sh` builds oxy from that copy and runs the whole

    comparison. `~/bench/first-commit.sh` runs only the first commit,

    status and fsck.

  - Run long jobs with Bash `run_in_background`, and wait for the

    completion notice rather than polling.

- **Correctness is never traded.** Every change keeps these passing:

  `oxy fsck` on the Linux repository, the property tests, and

  `oxy git verify` on the Git imports.

- **Format changes** (change, checkpoint, pack or wire) are allowed

  before 1.0, with these steps:

  - bump the spec draft and keep old objects readable with unchanged

    hashes;

  - bump `deps/oxygen` in oxygenflow with `make pins`, gate, commit,

    push and `make deploy-runner`;

  - then `make pins` in oxygenhub, gate, commit, push, `make deploy`,

    and check `/health`;

  - only then install the new oxy locally. Commit the format change

    itself with the old binary.

- **Memory and pace.** Every workstation build and test runs under

  `systemd-run --user --scope -p MemoryMax=8G`, and

  `CARGO_INCREMENTAL=0` (the disk is nearly full).

## Memory

- [ ] M0 **Tools to see memory.** Add a counting global allocator behind

  a cargo feature (peak bytes and allocation count per `OXY_TIMING`

  phase), so a profile names the structure that holds the memory. Then

  profile a clean `status` of the Linux repository, killed at 14 GB

  (oxygen#35), and the first commit (9.9 GB). Write down which

  structures hold what before changing anything.

- [ ] M1 **Status without every graph** (oxygen#35). A clean `status`

  must not build every file graph. Unchanged files are skipped by the

  manifest (spec §12.4), and the first status after a commit must not

  replay the whole change. Options: the commit writes a checkpoint;

  history-only states load graphs on demand (`State::load_graphs`).

  Target: under 300 MB and 0.4 s.

- [ ] M2 **Lines without an allocation each.** An insert's lines become

  one shared buffer with line offsets (for example `Arc<[u8]>` and

  ranges) in the change and in the file graph. The file graph shares

  that buffer and does not copy it. `AtomInfo` and the edge map shrink.

  Measure bytes per line before and after.

- [ ] M3 **A commit that streams.** The snapshot builder stores each

  file's text blob as it reads it and keeps no content it no longer

  needs, and the state is built without a second copy. Target for the

  first commit: under 2 GB.

- [ ] M4 **Everyday commands under 300 MB** on the Linux repository:

  status, diff, commit, log and blame, one file and 1 000 files edited.

## Speed

- [ ] S1 **First commit at most 50 s** (55.3 s now): profile with

  `perf`, and parallelise reading, hashing and diffing across the four

  cores where it is safe.

- [ ] S2 **Status, diff and commit** with one and with 1 000 files

  edited, to the targets above.

- [ ] S3 **log, blame and checkout** on the Linux repository, to the

  targets above.

- [ ] S4 **fsck and gc** measured against Git's. Re-run Git's `fsck`

  with gc turned off: 451 s is suspect, because Git's own background gc

  raced it.

## Space

- [ ] P1 **Store at most 350 MB after gc** (442 MB now against Git's

  311 MB): measure what takes the space (changes, text blobs, graphs,

  checkpoints, indexes), then fix the largest. Compression level and

  dictionary, and packing order, are all candidates.

## History at scale

- [ ] H1 **Importer name conflict** (oxygen#30): the 5 000-commit

  vscode import differs from Git in 51 files. Fix the cause and add a

  test. `oxy git verify` must be clean.

- [ ] H2 **Import memory**: drop graphs untouched by recent commits and

  reload them on demand, so memory is bounded by the working set and not

  by the history.

- [ ] H3 **Linux, stage 2**: import slices of recent history (2 000,

  10 000 and 50 000 commits), each verified, then time log, blame and

  status against Git on the same slice.

- [ ] H4 **vscode's full history** (about 130 000 commits) into a scratch

  repository on Earth. It is not published. Verify it and time it.

- [ ] H5 **Linux's full history** (about 1.3 million commits) if H3 shows

  it can finish overnight. Otherwise, write up what stands in the way.

## Finally

- [ ] Z1 Re-run the full `~/bench/linux-bench.sh` with the final build,

  and put the complete Git against oxy table in `docs/benchmarks.md`.

  Summarise, for Andy, where oxy now matches or beats Git and where it

  does not.

## Log

- 2026-09-29 Backlog written. Starting point: the Linux first commit

  succeeds (55.3 s, 9.9 GB, 442 MB store) with change format 2

  (oxygen#34). A clean `status` is killed at 14 GB (oxygen#35).

- 2026-09-29 **Moved to backlog 3.** At Andy's request, the Linux and scale work (L1b to L4, P5c) moved to `docs/backlog-overnight-3.md`: not worse than Git, with Git's numbers on the Linux tip as targets. Overnight loop 3 works only on that; the rest of this backlog waits.