# Overnight backlog 3 (from 2026-09-29): not worse than Git

Andy's brief: there is no point making a tool that is worse than Git.

Until oxy matches Git on a real, large tree, this is the only work:

speed, memory and space at scale, with correctness never traded away.

Work top to bottom. An item is done when:

- its tests pass and the gate is clean;

- it is measured before and after on Earth, and the numbers are in

  `docs/benchmarks.md`;

- it is committed with `oxy` as Andy and pushed to Earth.

If an item is too large, split it here and do the first part. Log each

iteration at the bottom.

## The yardstick

The Linux tip on Earth: 95 943 files, 1.8 GB, on 4 cores with 31 GiB.

Git 2.53 does this (from `~/bench/linux/results.txt`, 2026-09-29):

| Operation | Git | Target for oxy |

|---|---:|---:|

| First commit of the tree | 33.7 s, 42 MB | at most 50 s, under 2 GB |

| Clean `status` (warm) | 0.26 to 0.31 s, 31 MB | at most 0.4 s, under 300 MB |

| `status`, one file edited | 0.25 s | at most 0.4 s |

| `diff`, one file edited | 0.34 s | at most 0.5 s |

| `commit`, one file edited | 0.43 s | at most 0.6 s |

| `status`, 1 000 files edited | 0.26 s | at most 0.5 s |

| `diff`, 1 000 files edited | 1.38 s | at most 2 s |

| `commit`, 1 000 files edited | 0.76 s | at most 1.2 s |

| `log -n 100` | 0.05 s | at most 0.1 s |

| `blame MAINTAINERS` | 0.07 s | at most 0.2 s |

| Checkout of another state and back | 0.49 s and 0.38 s | at most 1 s each |

| Store after gc | 311 MB (`.git`) | at most 350 MB |

| Everyday commands | about 31 MB | under 300 MB |

The targets allow oxy a small margin, because it does more (conflicts

as data, signatures, an operation log). A target that turns out to be

out of reach for a structural reason is written up here, with the best

number reached and why. It is never quietly dropped.

## Rules for this loop

- **Measure first.** Before changing code, measure where the time or

  memory goes on Earth: `/usr/bin/time -v`, `perf record` (installed on

  Earth), `OXY_TIMING` phases, or a counting allocator behind a feature.

  Record what the profile showed in the log.

- **Heavy work only on Earth**, always under a cap:

  `systemd-run --user --scope -p MemoryMax=14G …`. Andy's workstation has

  crashed from memory before. Locally, run the gate and small cases only.

- **Getting code to Earth.** `~/bench/oxygen` on Earth is a plain copy of

  the source. Sync it with `rsync -a --delete --exclude target --exclude

  .oxy -e "ssh -i ~/.ssh/earth_claude" ~/development/oxygen/

  claude@earth:bench/oxygen/`.

  - `~/bench/linux-bench.sh` builds oxy from that copy and runs the whole

    comparison. `~/bench/first-commit.sh` runs only the first commit,

    status and fsck.

  - Run long jobs with Bash `run_in_background`, and wait for the

    completion notice rather than polling.

- **Correctness is never traded.** Every change keeps these passing:

  `oxy fsck` on the Linux repository, the property tests, and

  `oxy git verify` on the Git imports.

- **Format changes** (change, checkpoint, pack or wire) are allowed

  before 1.0, with these steps:

  - bump the spec draft and keep old objects readable with unchanged

    hashes;

  - bump `deps/oxygen` in oxygenflow with `make pins`, gate, commit,

    push and `make deploy-runner`;

  - then `make pins` in oxygenhub, gate, commit, push, `make deploy`,

    and check `/health`;

  - only then install the new oxy locally. Commit the format change

    itself with the old binary.

- **Memory and pace.** Every workstation build and test runs under

  `systemd-run --user --scope -p MemoryMax=8G`, and

  `CARGO_INCREMENTAL=0` (the disk is nearly full).

## Correctness first (from metalgear, 2026-09-29)

A working copy that silently becomes the wrong tree is worse than any

slowness, so these come before everything else.

- [x] C1 **Working-copy writes hold the workspace lock** (oxygen#36). Checkout, bookmark switch (with --carry), pull's checkout, undo, discard and commit hold the lock from reading the head until their operation is recorded. `merge_views` never lets a working-copy change based on a replaced one win. Test: a snapshot concurrent with a checkout never records the checkout's writes as edits.

- [x] C2 **`bookmark create NAME CHANGE` refuses a partial state** (oxygen#37). When CHANGE is one of the current state's heads but not all of them, it refuses and explains, unless given --only. `bookmark switch` summarises a wide tree difference. The guide explains that bookmarks name states, not ancestry.

- [x] C3 **A parent's switch never rolls a subrepo's bookmark back** (oxygen#38).

- [x] C4 **`pull` says "Already up to date" while one behind** (oxygen#29). Reproduce it with a test, then fix it.

## Memory

- [x] M0 **Tools to see memory.** Add a counting global allocator behind

  a cargo feature (peak bytes and allocation count per `OXY_TIMING`

  phase), so a profile names the structure that holds the memory. Then

  profile a clean `status` of the Linux repository, killed at 14 GB

  (oxygen#35), and the first commit (9.9 GB). Write down which

  structures hold what before changing anything.

- [x] M1 **Status without every graph** (oxygen#35). A clean `status`

  must not build every file graph. Unchanged files are skipped by the

  manifest (spec §12.4), and the first status after a commit must not

  replay the whole change. Options: the commit writes a checkpoint;

  history-only states load graphs on demand (`State::load_graphs`).

  Target: under 300 MB and 0.4 s.

- [x] M2 **Lines without an allocation each.** An insert's lines become

  one shared buffer with line offsets (for example `Arc<[u8]>` and

  ranges) in the change and in the file graph. The file graph shares

  that buffer and does not copy it. `AtomInfo` and the edge map shrink.

  Measure bytes per line before and after.

- [ ] M3 **A commit that streams.** The snapshot builder stores each

  file's text blob as it reads it and keeps no content it no longer

  needs, and the state is built without a second copy. Target for the

  first commit: under 2 GB.

- [ ] M4 **Everyday commands under 300 MB** on the Linux repository:

  status, diff, commit, log and blame, one file and 1 000 files edited.

## Speed

- [x] S1 **First commit at most 50 s** (55.3 s now): profile with

  `perf`, and parallelise reading, hashing and diffing across the four

  cores where it is safe.

- [ ] S2 **Status, diff and commit** with one and with 1 000 files

  edited, to the targets above.

- [ ] S3 **log, blame and checkout** on the Linux repository, to the

  targets above.

- [ ] S4 **fsck and gc** measured against Git's. Re-run Git's `fsck`

  with gc turned off: 451 s is suspect, because Git's own background gc

  raced it.

## Space

- [x] P1 **Store at most 350 MB after gc** (442 MB now against Git's

  311 MB): measure what takes the space (changes, text blobs, graphs,

  checkpoints, indexes), then fix the largest. Compression level and

  dictionary, and packing order, are all candidates.

## History at scale

- [x] H1 **Importer name conflict** (oxygen#30): the 5 000-commit

  vscode import differs from Git in 51 files. Fix the cause and add a

  test. `oxy git verify` must be clean.

- [ ] H2 **Import memory**: drop graphs untouched by recent commits and

  reload them on demand, so memory is bounded by the working set and not

  by the history.

- [ ] H3 **Linux, stage 2**: import slices of recent history (2 000,

  10 000 and 50 000 commits), each verified, then time log, blame and

  status against Git on the same slice.

- [ ] H4 **vscode's full history** (about 130 000 commits) into a scratch

  repository on Earth. It is not published. Verify it and time it.

- [ ] H5 **Linux's full history** (about 1.3 million commits) if H3 shows

  it can finish overnight. Otherwise, write up what stands in the way.

## From the feedback (2026-09-30)

- [ ] R1 **Rebase into the working copy** (oxygen#42, Andy chose the

  overnight loop): when `oxy rebase CHANGE --onto TARGET` would be

  conflicted, offer to leave the conflict in the working copy to resolve

  and commit, as `merge` does, instead of refusing. Tests for a stacked

  proposal brought up to date this way.

## Finally

- [ ] Z1 Re-run the full `~/bench/linux-bench.sh` with the final build,

  and put the complete Git against oxy table in `docs/benchmarks.md`.

  Summarise, for Andy, where oxy now matches or beats Git and where it

  does not.

## Log

- 2026-09-29 Backlog written. Starting point: the Linux first commit

  succeeds (55.3 s, 9.9 GB, 442 MB store) with change format 2

  (oxygen#34). A clean `status` is killed at 14 GB (oxygen#35).

- 2026-09-29 **C1 to C4 done** (the metalgear findings in ~/oxygen_feedback.md, items 22 and 25 to 26). The gate passes: 205 tests, clippy clean.

  - **C1** (e8e51993, closes #36): the working-copy lock now covers checkout, switch (and --carry), merge, commit, amend and rewrites, discard and pull. It is re-entrant per thread, waits up to a minute, and a live process's lock is never taken over. A workspace changed by two concurrent operations is now an error. A race test runs twenty switches against a snapshot thread; without the lock it failed one run in three.

  - **C2** (bf423e78, closes #37): `bookmark create NAME CHANGE` refuses when it would leave out earlier work, unless given --upto or --only. `bookmark switch` says how many changes the new state lacks. The guide explains states for people coming from Git.

  - **C3** (6b582c4c, closes #38): a parent's checkout moves a subrepo's bookmark only forward, and the pin is read from the nested working copy. `--carry` leaves a subrepo whose pin is carried work as it is.

  - **C4** (5dfea7b7, closes #29): not a sync bug. rstar-api's operation log shows each such pull ran on a feature bookmark, while main, which the fetch had just updated, was the one behind. pull now names the bookmark it pulled and the others left behind.

  - **For Andy:** metalgear's working copy is sound, on develop and clean. `logo-25` holds just the logo change and its dependencies; delete it if it is not wanted.

- 2026-09-30 **M0 and M1 done; M4, S2 and S3 part way** (oxygen#35 and oxygen#39). The tests and clippy pass after each step. Each step was measured on the Linux tip and is recorded in docs/benchmarks.md.

  - **M0** (f11e81a1): a counting allocator behind the `alloc-stats` feature; with `OXY_TIMINGS` it reports memory per phase.

  - **M1**, with these steps:

    - history-only loads read no text (22b70898), then read only a change's skeleton (39c6adea);

    - graph keys are written at commit (22b70898);

    - a clean status is answered from the manifest (b4fefa1e) with a parallel walk (e19aa6c5).

    - Clean status: killed at 14 GB before, now 0.44 s and 123 MB (Git: 0.22 s).

  - **History-only loads where no graph is needed** (26f81555): bookmark, merge, pull, tags, bisect, doctor and most history rewrites.

  - **Checkout**: a batch at a time (921fa890), without a snapshot when clean (d1b480f0), and against the manifest without the base state (2b7ce2c6).

    - One-file switch: killed before, now 2.1 s and 455 MB (Git: 0.2 s).

    - Local clone: killed before, now 110 s and 8 GB (Git: 9.6 s).

  - **log, blame and fsck** (34ea4ace), with metadata-only change reads:

    - log -n 100: 28 s and 4.7 GB before, now 2.0 s;

    - blame: killed before, now 1.1 s;

    - fsck: killed before, now 123 s and 4.5 GB.

  - **Snapshot**: a parallel walk with stats carried into the manifest (a578fa19), and a cached tree view (de3f07f8). Status with one file changed: 4.1 s before, now 2.3 s (Git: 0.23 s).

  - **Still far from Git**: one-file and 1 000-file status, diff and commit (2 to 5 s); switch (2 s); log (2 s); clone (110 s, 8 GB); first commit (56 s, 10 GB); fsck (4.5 GB).

    - The common cost is loading the first change's history (0.5 s) and building the tree view (0.4 s).

    - Next: a persistent per-state cache (tree view, graph keys and change set), Git's index in effect, then compact graphs (M2) and a streaming commit (M3).

- 2026-09-30 **M2 done, M3 and S1 part way** (oxygen#35). The first commit of the Linux tip now takes 40.7 to 41.6 s on a quiet Earth (S1 target: 50 s; Git: 34 to 38 s) and 4.9 GB RSS (from 10 GB).

  - **Encoding** (b7103633): each change is encoded once, the working copy's edits are moved rather than copied, and text blobs and chunk compression run on every core.

  - **Storing** (b7a090c7): texts go to the store a batch at a time.

  - **M2, lines** (086641a8): an insert's lines are one shared buffer, into which graph atoms point.

  - **Clone** (5d62fa66, 11bf3b00, 397d9f98): a local clone takes 48 s and 3.9 GB (from 96 s and 5.2 GB, and killed before that; Git: 9.6 s). The changes behind it:

    - an unedited file's graph is read off as a chain;

    - a bundle carries its changes' hashes;

    - a large change is read once per checkout;

    - a local fetch moves changes as stored.

  - **Hub and runner**: re-pinned to oxygen 397d9f98 and deployed (oxygenhub#58, a6f006ba; oxygenflow#15, 630b7fd6). The hub reads metadata or skeletons where it lists changes. Checked: /health, and the log, file history, tree and file pages.

  - **Note**: my pushes start Flow jobs on Earth, so timings under load are marked in benchmarks.md.

- 2026-09-30 **H1 done** (oxygen#30). Three causes, each from Git having no identity for a directory:

  - a directory deleted and made again became a second node;

  - a placement under a removed directory was out of reach;

  - orphans could not be seen.

  - The fix: an import revives the directory, and a merge retires a kept node's other placements and every orphan. There is a Git-built test for each cause; each reproduces the vscode symptom without its fix. Spec §23.

  - The vscode slice now verifies: all 1 747 commits and 3.3 million files.

  - `oxy git verify` is cached by graph key and blob: 115 s instead of 1 400 s.

- 2026-09-30 **H2, first part**: an import builds graphs only for the files a commit may change, from history-only states. The first 5 000 vscode commits: 309 s and 2.3 GB (from 412 s and 6.3 GB), and all verified. Next: verify and import reuse each commit's history instead of loading it again, before H3 to H5.

- 2026-09-30 **P1 and S1 done** (oxygen#40).

  - **Repack**: `oxy gc` repacks everything into one pack at zstd level 9, compressing on every core (9f7585d4).

  - **Format 3**: text blobs hold the lines' bytes, with no CBOR framing (1122dcb3). The runner (oxygenflow#16) and hub (oxygenhub#59) were deployed with the new reader first.

  - **Linux tip**: the store is 324 MB after gc (objects 306 MB; Git 311 MB). The first commit takes 32.6 s (Git 33.7 to 37.9 s), measured while Earth ran at load average 11 during the vscode import.

  - **H2**: the vscode import (H4) runs on Earth as a service.

- 2026-09-30 **H4 attempted**: the full vscode import (166 425 commits) under a 14 GB cap. With changes added mixed (e15074c7), it still grew about 0.4 MB per commit: 4.3 GB after 35 minutes and roughly 9 000 commits, far past the cap long before the end. It was stopped. The growth is the importer's kept states: each is a deep clone of a history-only state (change metadata, tree maps, the deferred edits of every file), up to eight at a time, with a new clone at every branch point.

  - **Next (H2b)**: make a state cheap to clone by sharing its parts (per-file deferred entries and graphs behind `Arc`, copied on write; the change map and tree likewise), so kept states share what they have in common.

  - **Also**: an import should report progress on long runs.

- 2026-09-30 **H2b, first part**: a state's change metadata and per-file deferred entries are shared between clones and copied on write, and a clone starts with an empty closure cache.

  - Measured: 5 000 vscode commits 317 s and 2.07 GB (unchanged); 20 000 commits 4 530 s and 6.7 GB. So most of the growth is the base state itself, not its clones.

  - Phases from 2 000 to 5 000 commits (2.5 times as many):

    | Phase | 2 000 commits | 5 000 commits |

    |---|---:|---:|

    | bases | 5.3 s | 24.4 s |

    | ls-tree | 20.8 s | 63.8 s |

    | render base | 20.8 s | 89.0 s |

    | build change | 97.3 s | 139.7 s |

  - Listing, rendering and rebuilding bases cost what the whole tree is, on every commit: oxygen#25. H3 to H5 need it. The plan:

    - list a commit's changes with `git diff-tree` against its first parent rather than `ls-tree`;

    - keep the tree view up to date as changes apply, rather than recomputing it;

    - render only the paths a commit touches.

- 2026-09-30 **Full Linux run** (06:24): table in `docs/benchmarks.md`. oxy matches Git on the first commit (38.5 s against 38.8 s) and beats it on gc (39.7 s against 102.8 s); the store is 324 MB against 295 MB. Everyday commands are still 5 to 15 times slower: most of each is loading the base's history and tree view. A one-file commit no longer reloads the new base for its manifest keys: 3.1 s (from 4.7 s). oxygen#41 (a base index) is proposed and waits for Andy.