# Overnight backlog 4: beating Git on the Linux tree

Loop 3 made oxy usable on the Linux kernel. This loop aims to make it

faster than Git wherever the model allows it, with the kernel tree as the

yardstick. Correctness is never traded for speed: every item keeps its

tests, and the test builds that compare an incremental structure with a

full computation stay on.

## The yardstick

The Linux tip on Earth (`~claude/bench/linux`, 95 943 files, 1.8 GB):

`o` is the oxy repository and `g` the Git one. Git is 2.53.

| Operation | oxy (2026-09-30) | Git | Target |

|---|---:|---:|---|

| init and first commit | 38.5 s, 4.4 GB peak RSS | 38.8 s | under Git, under 2 GB |

| status, clean | 0.41 to 0.43 s | 0.26 to 0.32 s | at or under Git |

| status, one file edited | 1.26 to 1.39 s | 0.26 to 0.29 s | at or under Git |

| diff, one file | 1.19 to 1.45 s | 0.16 to 0.25 s | at or under Git |

| commit, one file | about 3.1 s | 0.45 s | under 0.5 s |

| status, diff and commit, 1 000 files | 3.9, 2.8 and 4.9 s | 0.41, 0.95 and 0.57 s | at or under Git |

| log -n 100 | 1.39 s | 0.01 s | under 0.05 s |

| blame MAINTAINERS | 1.18 s | 0.09 s | at or under Git |

| checkout to the first state and back | 2.2 to 2.8 s | 0.42 to 0.47 s | at or under Git |

| fsck | 22.1 s | 13.1 s | at or under Git |

| gc | 39.7 s | 102.8 s | stays ahead |

| store after gc | 324 MB | 295 MB | at or under Git |

Where one-file status time goes now (Earth, loaded):

- index load: 0.16 to 0.28 s

- walk: 0.45 to 0.7 s

- manifest check before the snapshot: 0.1 s

- placeholders for unchanged files: 0.09 s

- tracked paths: 0.04 s

## Rules for this loop

- **Measure quietly.** Earth runs OxygenFlow jobs, for every push to the

  Oxygen repositories and for metalgear. Measure when `pgrep firecracker`

  finds nothing if you can; otherwise run oxy and Git alternately, three

  times each, and record the load average. Push your commits after a

  measurement, not before, since each push starts jobs.

- **Build the bench binary** from the synced source:

  `rsync -a --delete --exclude target --exclude .oxy ~/development/oxygen/ claude@earth:bench/oxygen/`,

  then `CARGO_TARGET_DIR=~/bench/idxbin cargo build --release` there. Use

  `~/bench/prof-status.sh`, `~/bench/vs-git.sh` and `~/bench/linux-bench.sh`,

  and extend them rather than writing one-off commands.

- **Phase timings** (`OXY_TIMINGS=1`) name where time goes. Add phases where

  they are missing before optimising.

- **An index is derived.** Every new index or cache is keyed by what it

  describes, can be rebuilt from the changes at any time, and has a test

  build check that compares it with a full computation (as the state index,

  tree view and kept file order do).

- **Formats change freely before 1.0,** but a change format bump follows the

  order in loop 3's backlog (old binary commits it; then the runner and hub

  pins; then `make install`).

## Speed

- [ ] B0 **Baseline.** Update `~/bench/linux-bench.sh` for today's build (the

  state index, the manifest cache, clean and dirty status, diff, commit,

  log, blame, checkout, a local clone, fsck, gc, store size) with three runs

  each and oxy and Git alternating. Record the table in `docs/benchmarks.md`.

- [ ] S1 **Status and diff at Git's speed** (oxygen#41). In order, measuring

  after each:

  - the manifest as a compact binary file read without a parse per line;

  - the walk: compare each file's stat with the manifest without building

    a path string per entry (sorted walk against a sorted manifest), and

    skip the `.oxyignore` machinery inside tracked directories where it

    cannot change the answer;

  - no placeholder `FileState` for unchanged files (audit every user of

    `SnapshotResult.base_snapshot.files` first);

  - load the state index lazily: memory-map it and decode a file's deferred

    history only when that file is built;

  - one pass for status: the manifest check and the snapshot walk share

    their stats.

- [ ] S2 **Commit at Git's speed.** A commit extends the base it already

  loaded: write the new state index from the old one plus the change, rather

  than loading history again, and do not render what the snapshot rendered.

  Target: one file under 0.5 s, 1 000 files under 0.6 s.

- [ ] S3 **The change index: log in milliseconds.** Per state, the changes in

  topological order with their generation numbers and where each change's

  metadata is, built from the parent state's index plus the new change.

  `oxy log -n 100` reads 100 entries. Target: under 0.05 s.

- [ ] S4 **The node index: blame and file history.** Per node, the changes

  that touch it, so blame and `oxy log FILE` cost that file's history.

  Target: blame MAINTAINERS at or under Git.

- [ ] S5 **Checkout by comparing indexes.** Compare the two states' tree

  indexes by rendered content hash and write only the files that differ.

  Target: at or under Git.

- [ ] S6 **fsck at or under Git**: parallel object checking and a cheaper

  state pass.

- [ ] S7 **The first commit ahead of Git, under 2 GB**: stream reading,

  hashing and storing a batch of files at a time, so the whole tree's text

  is never in memory at once.

## Space

- [ ] P1 **Store at or under Git after gc** (324 MB against 295 MB): measure

  what takes the space, then try a trained zstd dictionary per repository

  and packing similar objects together.

## When a speed item is blocked: the other planned work

Andy's choice (2026-09-30): if the loop is stuck on the speed items (no

idea left that measures better, or blocked on something only Andy can

decide), it notes why in the log and takes the next item below instead of

stopping. Each follows the same rules: an issue, tests, the gate, push,

deploy where it applies, and a log line. Skip an item that needs Andy (a

decision, an account, hardware) and say so in the log.

1. **Web workspaces (P5), "VS Code in the hub"**, in parts:

   - [ ] P5a **flow-casfs**: a read-only FUSE mount of a state from its

     content-addressed chunks, in oxygenflow, measured (mount time, first

     read, a full walk of the Linux tip).

   - [ ] P5b **Open in Workspace** on a change or bookmark page of the hub:

     openvscode-server (MIT, unmodified) with Oxygen VCS inside a

     Firecracker microVM per workspace, the state mounted read-only through

     casfs with an ephemeral overlay for edits, and commits going back

     through oxy as the signed-in user. The runner hosts it (the image under

     `~/flow`, a workspace job kind); the hub starts it, proxies it to the

     signed-in user only, and stops it when idle. Boot time measured, aiming

     under 100 ms for the mount. Write the design into the hub and

     oxygenflow specs first. A fork of VS Code only if an extension cannot

     do what is needed, decided with Andy.

2. - [ ] **oxy on projects coming from Git**: say when a folder holding

   `.git` is skipped (oxygen#43), and honour `.gitignore` or offer to import

   it (oxygen#44).

3. - [ ] **Device sign-in** for apps and `oh` (oxygenhub#67): approve a code on

   the web with password and 2FA, and the app receives a scoped token. The

   Android client needs it.

4. - [ ] **An Android build image** for OxygenFlow (oxygenflow#19): JDK,

   Android SDK and NDK, and a Gradle cache warmed offline.

5. - [ ] **OxygenHub VCS**: declare its authentication provider in the

   manifest (oxygenhub-vscode#4), released as the next patch (see the

   Marketplace notes: never reuse a version).

6. - [ ] **Import at scale**: finish oxygen#25 (per-commit import cost), then

   backlog 3's H3 (Linux history slices of 2 000, 10 000 and 50 000

   commits, verified) and H4 (vscode's full history, about 130 000

   commits, into a scratch repository that is never published).

7. - [ ] **The hub's open issues**: runner management (oxygenhub#41, backlog

   2's H12 parts), a pinned subrepo left out of a Flow job for lack of

   access (#56), faster CI (#29), required checks over the API (#20).

8. - [ ] **The hub's plan** (`oxygenhub/docs/plan.md`): notifications for

   review requests, mentions and landings; the organisation webhooks page.

   Email verification and password reset need SMTP on Earth, and passkeys

   are for 1.0: skip both unless Andy says otherwise.

9. - [ ] **Housekeeping**: tick plan items that are already done (for

   example `oxy seal --proposal` exists), and CHANGELOG files for oxygen,

   oxygenhub and oxygenflow like the extensions' (backlog 2's V2).

## Also

- [ ] R1 **Rebase into the working copy** (oxygen#42, Andy's choice for the

  overnight loop): when `oxy rebase CHANGE --onto TARGET` would be

  conflicted, offer to leave the conflict in the working copy to resolve and

  commit, as `merge` does, rather than refusing.

- [ ] Z1 **Closing summary.** Re-run the full benchmark and summarise, for

  Andy, where oxy matches or beats Git on the Linux tree, and where it does

  not and why.

## Log

- 2026-09-30 Backlog written from the day's measurements (state index,

  changed-only builds and the manifest cache already in).