# Overnight backlog 3 (from 2026-09-29): not worse than Git

Andy's brief: there is no point making a tool that is worse than Git.
Until oxy matches Git on a real, large tree, this is the only work:
speed, memory and space at scale, with correctness never traded away.

Work top to bottom. An item is done when:
- its tests pass and the gate is clean;
- it is measured before and after on Earth, and the numbers are in
  `docs/benchmarks.md`;
- it is committed with `oxy` as Andy and pushed to Earth.

If an item is too large, split it here and do the first part. Log each
iteration at the bottom.

## The yardstick

The Linux tip on Earth: 95 943 files, 1.8 GB, on 4 cores with 31 GiB.
Git 2.53 does this (from `~/bench/linux/results.txt`, 2026-09-29):

| Operation | Git | Target for oxy |
|---|---:|---:|
| First commit of the tree | 33.7 s, 42 MB | at most 50 s, under 2 GB |
| Clean `status` (warm) | 0.26 to 0.31 s, 31 MB | at most 0.4 s, under 300 MB |
| `status`, one file edited | 0.25 s | at most 0.4 s |
| `diff`, one file edited | 0.34 s | at most 0.5 s |
| `commit`, one file edited | 0.43 s | at most 0.6 s |
| `status`, 1 000 files edited | 0.26 s | at most 0.5 s |
| `diff`, 1 000 files edited | 1.38 s | at most 2 s |
| `commit`, 1 000 files edited | 0.76 s | at most 1.2 s |
| `log -n 100` | 0.05 s | at most 0.1 s |
| `blame MAINTAINERS` | 0.07 s | at most 0.2 s |
| Checkout of another state and back | 0.49 s and 0.38 s | at most 1 s each |
| Store after gc | 311 MB (`.git`) | at most 350 MB |
| Everyday commands | about 31 MB | under 300 MB |

The targets allow oxy a small margin, because it does more (conflicts
as data, signatures, an operation log). A target that turns out to be
out of reach for a structural reason is written up here, with the best
number reached and why. It is never quietly dropped.

## Rules for this loop

- **Measure first.** Before changing code, measure where the time or
  memory goes on Earth: `/usr/bin/time -v`, `perf record` (installed on
  Earth), `OXY_TIMING` phases, or a counting allocator behind a feature.
  Record what the profile showed in the log.
- **Heavy work only on Earth**, always under a cap:
  `systemd-run --user --scope -p MemoryMax=14G …`. Andy's workstation has
  crashed from memory before. Locally, run the gate and small cases only.
- **Getting code to Earth.** `~/bench/oxygen` on Earth is a plain copy of
  the source. Sync it with `rsync -a --delete --exclude target --exclude
  .oxy -e "ssh -i ~/.ssh/earth_claude" ~/development/oxygen/
  claude@earth:bench/oxygen/`.
  - `~/bench/linux-bench.sh` builds oxy from that copy and runs the whole
    comparison. `~/bench/first-commit.sh` runs only the first commit,
    status and fsck.
  - Run long jobs with Bash `run_in_background`, and wait for the
    completion notice rather than polling.
- **Correctness is never traded.** Every change keeps these passing:
  `oxy fsck` on the Linux repository, the property tests, and
  `oxy git verify` on the Git imports.
- **Format changes** (change, checkpoint, pack or wire) are allowed
  before 1.0, with these steps:
  - bump the spec draft and keep old objects readable with unchanged
    hashes;
  - bump `deps/oxygen` in oxygenflow with `make pins`, gate, commit,
    push and `make deploy-runner`;
  - then `make pins` in oxygenhub, gate, commit, push, `make deploy`,
    and check `/health`;
  - only then install the new oxy locally. Commit the format change
    itself with the old binary.
- **Memory and pace.** Every workstation build and test runs under
  `systemd-run --user --scope -p MemoryMax=8G`, and
  `CARGO_INCREMENTAL=0` (the disk is nearly full).

## Correctness first (from metalgear, 2026-09-29)

A working copy that silently becomes the wrong tree is worse than any
slowness, so these come before everything else.

- [x] C1 **Working-copy writes hold the workspace lock** (oxygen#36). Checkout, bookmark switch (with --carry), pull's checkout, undo, discard and commit hold the lock from reading the head until their operation is recorded. `merge_views` never lets a working-copy change based on a replaced one win. Test: a snapshot concurrent with a checkout never records the checkout's writes as edits.
- [x] C2 **`bookmark create NAME CHANGE` refuses a partial state** (oxygen#37). When CHANGE is one of the current state's heads but not all of them, it refuses and explains, unless given --only. `bookmark switch` summarises a wide tree difference. The guide explains that bookmarks name states, not ancestry.
- [x] C3 **A parent's switch never rolls a subrepo's bookmark back** (oxygen#38).
- [x] C4 **`pull` says "Already up to date" while one behind** (oxygen#29). Reproduce it with a test, then fix it.

## Memory

- [x] M0 **Tools to see memory.** Add a counting global allocator behind
  a cargo feature (peak bytes and allocation count per `OXY_TIMING`
  phase), so a profile names the structure that holds the memory. Then
  profile a clean `status` of the Linux repository, killed at 14 GB
  (oxygen#35), and the first commit (9.9 GB). Write down which
  structures hold what before changing anything.
- [x] M1 **Status without every graph** (oxygen#35). A clean `status`
  must not build every file graph. Unchanged files are skipped by the
  manifest (spec §12.4), and the first status after a commit must not
  replay the whole change. Options: the commit writes a checkpoint;
  history-only states load graphs on demand (`State::load_graphs`).
  Target: under 300 MB and 0.4 s.
- [x] M2 **Lines without an allocation each.** An insert's lines become
  one shared buffer with line offsets (for example `Arc<[u8]>` and
  ranges) in the change and in the file graph. The file graph shares
  that buffer and does not copy it. `AtomInfo` and the edge map shrink.
  Measure bytes per line before and after.
- [ ] M3 **A commit that streams.** The snapshot builder stores each
  file's text blob as it reads it and keeps no content it no longer
  needs, and the state is built without a second copy. Target for the
  first commit: under 2 GB.
- [ ] M4 **Everyday commands under 300 MB** on the Linux repository:
  status, diff, commit, log and blame, one file and 1 000 files edited.

## Speed

- [x] S1 **First commit at most 50 s** (55.3 s now): profile with
  `perf`, and parallelise reading, hashing and diffing across the four
  cores where it is safe.
- [ ] S2 **Status, diff and commit** with one and with 1 000 files
  edited, to the targets above.
- [ ] S3 **log, blame and checkout** on the Linux repository, to the
  targets above.
- [ ] S4 **fsck and gc** measured against Git's. Re-run Git's `fsck`
  with gc turned off: 451 s is suspect, because Git's own background gc
  raced it.

## Space

- [x] P1 **Store at most 350 MB after gc** (442 MB now against Git's
  311 MB): measure what takes the space (changes, text blobs, graphs,
  checkpoints, indexes), then fix the largest. Compression level and
  dictionary, and packing order, are all candidates.

## History at scale

- [x] H1 **Importer name conflict** (oxygen#30): the 5 000-commit
  vscode import differs from Git in 51 files. Fix the cause and add a
  test. `oxy git verify` must be clean.
- [ ] H2 **Import memory**: drop graphs untouched by recent commits and
  reload them on demand, so memory is bounded by the working set and not
  by the history.
- [ ] H3 **Linux, stage 2**: import slices of recent history (2 000,
  10 000 and 50 000 commits), each verified, then time log, blame and
  status against Git on the same slice.
- [ ] H4 **vscode's full history** (about 130 000 commits) into a scratch
  repository on Earth. It is not published. Verify it and time it.
- [ ] H5 **Linux's full history** (about 1.3 million commits) if H3 shows
  it can finish overnight. Otherwise, write up what stands in the way.

## Finally

- [ ] Z1 Re-run the full `~/bench/linux-bench.sh` with the final build,
  and put the complete Git against oxy table in `docs/benchmarks.md`.
  Summarise, for Andy, where oxy now matches or beats Git and where it
  does not.

## Log

- 2026-09-29 Backlog written. Starting point: the Linux first commit
  succeeds (55.3 s, 9.9 GB, 442 MB store) with change format 2
  (oxygen#34). A clean `status` is killed at 14 GB (oxygen#35).
- 2026-09-29 **C1 to C4 done** (the metalgear findings in ~/oxygen_feedback.md, items 22 and 25 to 26). The gate passes: 205 tests, clippy clean.
  - **C1** (e8e51993, closes #36): the working-copy lock now covers checkout, switch (and --carry), merge, commit, amend and rewrites, discard and pull. It is re-entrant per thread, waits up to a minute, and a live process's lock is never taken over. A workspace changed by two concurrent operations is now an error. A race test runs twenty switches against a snapshot thread; without the lock it failed one run in three.
  - **C2** (bf423e78, closes #37): `bookmark create NAME CHANGE` refuses when it would leave out earlier work, unless given --upto or --only. `bookmark switch` says how many changes the new state lacks. The guide explains states for people coming from Git.
  - **C3** (6b582c4c, closes #38): a parent's checkout moves a subrepo's bookmark only forward, and the pin is read from the nested working copy. `--carry` leaves a subrepo whose pin is carried work as it is.
  - **C4** (5dfea7b7, closes #29): not a sync bug. rstar-api's operation log shows each such pull ran on a feature bookmark, while main, which the fetch had just updated, was the one behind. pull now names the bookmark it pulled and the others left behind.
  - **For Andy:** metalgear's working copy is sound, on develop and clean. `logo-25` holds just the logo change and its dependencies; delete it if it is not wanted.
- 2026-09-30 **M0 and M1 done; M4, S2 and S3 part way** (oxygen#35 and oxygen#39). The tests and clippy pass after each step. Each step was measured on the Linux tip and is recorded in docs/benchmarks.md.
  - **M0** (f11e81a1): a counting allocator behind the `alloc-stats` feature; with `OXY_TIMINGS` it reports memory per phase.
  - **M1**, with these steps:
    - history-only loads read no text (22b70898), then read only a change's skeleton (39c6adea);
    - graph keys are written at commit (22b70898);
    - a clean status is answered from the manifest (b4fefa1e) with a parallel walk (e19aa6c5).
    - Clean status: killed at 14 GB before, now 0.44 s and 123 MB (Git: 0.22 s).
  - **History-only loads where no graph is needed** (26f81555): bookmark, merge, pull, tags, bisect, doctor and most history rewrites.
  - **Checkout**: a batch at a time (921fa890), without a snapshot when clean (d1b480f0), and against the manifest without the base state (2b7ce2c6).
    - One-file switch: killed before, now 2.1 s and 455 MB (Git: 0.2 s).
    - Local clone: killed before, now 110 s and 8 GB (Git: 9.6 s).
  - **log, blame and fsck** (34ea4ace), with metadata-only change reads:
    - log -n 100: 28 s and 4.7 GB before, now 2.0 s;
    - blame: killed before, now 1.1 s;
    - fsck: killed before, now 123 s and 4.5 GB.
  - **Snapshot**: a parallel walk with stats carried into the manifest (a578fa19), and a cached tree view (de3f07f8). Status with one file changed: 4.1 s before, now 2.3 s (Git: 0.23 s).
  - **Still far from Git**: one-file and 1 000-file status, diff and commit (2 to 5 s); switch (2 s); log (2 s); clone (110 s, 8 GB); first commit (56 s, 10 GB); fsck (4.5 GB).
    - The common cost is loading the first change's history (0.5 s) and building the tree view (0.4 s).
    - Next: a persistent per-state cache (tree view, graph keys and change set), Git's index in effect, then compact graphs (M2) and a streaming commit (M3).
- 2026-09-30 **M2 done, M3 and S1 part way** (oxygen#35). The first commit of the Linux tip now takes 40.7 to 41.6 s on a quiet Earth (S1 target: 50 s; Git: 34 to 38 s) and 4.9 GB RSS (from 10 GB).
  - **Encoding** (b7103633): each change is encoded once, the working copy's edits are moved rather than copied, and text blobs and chunk compression run on every core.
  - **Storing** (b7a090c7): texts go to the store a batch at a time.
  - **M2, lines** (086641a8): an insert's lines are one shared buffer, into which graph atoms point.
  - **Clone** (5d62fa66, 11bf3b00, 397d9f98): a local clone takes 48 s and 3.9 GB (from 96 s and 5.2 GB, and killed before that; Git: 9.6 s). The changes behind it:
    - an unedited file's graph is read off as a chain;
    - a bundle carries its changes' hashes;
    - a large change is read once per checkout;
    - a local fetch moves changes as stored.
  - **Hub and runner**: re-pinned to oxygen 397d9f98 and deployed (oxygenhub#58, a6f006ba; oxygenflow#15, 630b7fd6). The hub reads metadata or skeletons where it lists changes. Checked: /health, and the log, file history, tree and file pages.
  - **Note**: my pushes start Flow jobs on Earth, so timings under load are marked in benchmarks.md.
- 2026-09-30 **H1 done** (oxygen#30). Three causes, each from Git having no identity for a directory:
  - a directory deleted and made again became a second node;
  - a placement under a removed directory was out of reach;
  - orphans could not be seen.
  - The fix: an import revives the directory, and a merge retires a kept node's other placements and every orphan. There is a Git-built test for each cause; each reproduces the vscode symptom without its fix. Spec §23.
  - The vscode slice now verifies: all 1 747 commits and 3.3 million files.
  - `oxy git verify` is cached by graph key and blob: 115 s instead of 1 400 s.
- 2026-09-30 **H2, first part**: an import builds graphs only for the files a commit may change, from history-only states. The first 5 000 vscode commits: 309 s and 2.3 GB (from 412 s and 6.3 GB), and all verified. Next: verify and import reuse each commit's history instead of loading it again, before H3 to H5.
- 2026-09-30 **P1 and S1 done** (oxygen#40).
  - **Repack**: `oxy gc` repacks everything into one pack at zstd level 9, compressing on every core (9f7585d4).
  - **Format 3**: text blobs hold the lines' bytes, with no CBOR framing (1122dcb3). The runner (oxygenflow#16) and hub (oxygenhub#59) were deployed with the new reader first.
  - **Linux tip**: the store is 324 MB after gc (objects 306 MB; Git 311 MB). The first commit takes 32.6 s (Git 33.7 to 37.9 s), measured while Earth ran at load average 11 during the vscode import.
  - **H2**: the vscode import (H4) runs on Earth as a service.
- 2026-09-30 **H4 attempted**: the full vscode import (166 425 commits) under a 14 GB cap. With changes added mixed (e15074c7), it still grew about 0.4 MB per commit: 4.3 GB after 35 minutes and roughly 9 000 commits, far past the cap long before the end. It was stopped. The growth is the importer's kept states: each is a deep clone of a history-only state (change metadata, tree maps, the deferred edits of every file), up to eight at a time, with a new clone at every branch point.
  - **Next (H2b)**: make a state cheap to clone by sharing its parts (per-file deferred entries and graphs behind `Arc`, copied on write; the change map and tree likewise), so kept states share what they have in common.
  - **Also**: an import should report progress on long runs.
- 2026-09-30 **H2b, first part**: a state's change metadata and per-file deferred entries are shared between clones and copied on write, and a clone starts with an empty closure cache.
  - Measured: 5 000 vscode commits 317 s and 2.07 GB (unchanged); 20 000 commits 4 530 s and 6.7 GB. So most of the growth is the base state itself, not its clones.
  - Phases from 2 000 to 5 000 commits (2.5 times as many):

    | Phase | 2 000 commits | 5 000 commits |
    |---|---:|---:|
    | bases | 5.3 s | 24.4 s |
    | ls-tree | 20.8 s | 63.8 s |
    | render base | 20.8 s | 89.0 s |
    | build change | 97.3 s | 139.7 s |

  - Listing, rendering and rebuilding bases cost what the whole tree is, on every commit: oxygen#25. H3 to H5 need it. The plan:
    - list a commit's changes with `git diff-tree` against its first parent rather than `ls-tree`;
    - keep the tree view up to date as changes apply, rather than recomputing it;
    - render only the paths a commit touches.