# Overnight backlog 4: beating Git on the Linux tree

Loop 3 made oxy usable on the Linux kernel. This loop aims to make it
faster than Git wherever the model allows it, with the kernel tree as the
yardstick. Correctness is never traded for speed: every item keeps its
tests, and the test builds that compare an incremental structure with a
full computation stay on.

## The yardstick

The Linux tip on Earth (`~claude/bench/linux`, 95 943 files, 1.8 GB):
`o` is the oxy repository and `g` the Git one. Git is 2.53.

| Operation | oxy (2026-09-30) | Git | Target |
|---|---:|---:|---|
| init and first commit | 38.5 s, 4.4 GB peak RSS | 38.8 s | under Git, under 2 GB |
| status, clean | 0.41 to 0.43 s | 0.26 to 0.32 s | at or under Git |
| status, one file edited | 1.26 to 1.39 s | 0.26 to 0.29 s | at or under Git |
| diff, one file | 1.19 to 1.45 s | 0.16 to 0.25 s | at or under Git |
| commit, one file | about 3.1 s | 0.45 s | under 0.5 s |
| status, diff and commit, 1 000 files | 3.9, 2.8 and 4.9 s | 0.41, 0.95 and 0.57 s | at or under Git |
| log -n 100 | 1.39 s | 0.01 s | under 0.05 s |
| blame MAINTAINERS | 1.18 s | 0.09 s | at or under Git |
| checkout to the first state and back | 2.2 to 2.8 s | 0.42 to 0.47 s | at or under Git |
| fsck | 22.1 s | 13.1 s | at or under Git |
| gc | 39.7 s | 102.8 s | stays ahead |
| store after gc | 324 MB | 295 MB | at or under Git |

Where one-file status time goes now (Earth, loaded):
- index load: 0.16 to 0.28 s
- walk: 0.45 to 0.7 s
- manifest check before the snapshot: 0.1 s
- placeholders for unchanged files: 0.09 s
- tracked paths: 0.04 s

## Rules for this loop

- **Measure quietly.** Earth runs OxygenFlow jobs, for every push to the
  Oxygen repositories and for metalgear. Measure when `pgrep firecracker`
  finds nothing if you can; otherwise run oxy and Git alternately, three
  times each, and record the load average. Push your commits after a
  measurement, not before, since each push starts jobs.
- **Build the bench binary** from the synced source:
  `rsync -a --delete --exclude target --exclude .oxy ~/development/oxygen/ claude@earth:bench/oxygen/`,
  then `CARGO_TARGET_DIR=~/bench/idxbin cargo build --release` there. Use
  `~/bench/prof-status.sh`, `~/bench/vs-git.sh` and `~/bench/linux-bench.sh`,
  and extend them rather than writing one-off commands.
- **Phase timings** (`OXY_TIMINGS=1`) name where time goes. Add phases where
  they are missing before optimising.
- **An index is derived.** Every new index or cache is keyed by what it
  describes, can be rebuilt from the changes at any time, and has a test
  build check that compares it with a full computation (as the state index,
  tree view and kept file order do).
- **Formats change freely before 1.0,** but a change format bump follows the
  order in loop 3's backlog (old binary commits it; then the runner and hub
  pins; then `make install`).

## Speed

- [x] B0 **Baseline.** Update `~/bench/linux-bench.sh` for today's build (the
  state index, the manifest cache, clean and dirty status, diff, commit,
  log, blame, checkout, a local clone, fsck, gc, store size) with three runs
  each and oxy and Git alternating. Record the table in `docs/benchmarks.md`.
- [ ] S1 **Status and diff at Git's speed** (oxygen#41). In order, measuring
  after each:
  - the manifest as a compact binary file read without a parse per line;
  - the walk: compare each file's stat with the manifest without building
    a path string per entry (sorted walk against a sorted manifest), and
    skip the `.oxyignore` machinery inside tracked directories where it
    cannot change the answer;
  - no placeholder `FileState` for unchanged files (audit every user of
    `SnapshotResult.base_snapshot.files` first);
  - load the state index lazily: memory-map it and decode a file's deferred
    history only when that file is built;
  - one pass for status: the manifest check and the snapshot walk share
    their stats.
- [ ] S2 **Commit at Git's speed.** A commit extends the base it already
  loaded: write the new state index from the old one plus the change, rather
  than loading history again, and do not render what the snapshot rendered.
  Target: one file under 0.5 s, 1 000 files under 0.6 s.
- [ ] S3 **The change index: log in milliseconds.** Per state, the changes in
  topological order with their generation numbers and where each change's
  metadata is, built from the parent state's index plus the new change.
  `oxy log -n 100` reads 100 entries. Target: under 0.05 s.
- [ ] S4 **The node index: blame and file history.** Per node, the changes
  that touch it, so blame and `oxy log FILE` cost that file's history.
  Target: blame MAINTAINERS at or under Git.
- [ ] S5 **Checkout by comparing indexes.** Compare the two states' tree
  indexes by rendered content hash and write only the files that differ.
  Target: at or under Git.
- [x] S6 **fsck at or under Git**: parallel object checking and a cheaper
  state pass.
- [ ] S7 **The first commit ahead of Git, under 2 GB**: stream reading,
  hashing and storing a batch of files at a time, so the whole tree's text
  is never in memory at once.

## Space

- [ ] P1 **Store at or under Git after gc** (324 MB against 295 MB): measure
  what takes the space, then try a trained zstd dictionary per repository
  and packing similar objects together.

## When a speed item is blocked: the other planned work

Andy's choice (2026-09-30): if the loop is stuck on the speed items (no
idea left that measures better, or blocked on something only Andy can
decide), it notes why in the log and takes the next item below instead of
stopping. Each follows the same rules: an issue, tests, the gate, push,
deploy where it applies, and a log line. Skip an item that needs Andy (a
decision, an account, hardware) and say so in the log.

1. **Web workspaces (P5), "VS Code in the hub"**, in parts:
   - [ ] P5a **flow-casfs**: a read-only FUSE mount of a state from its
     content-addressed chunks, in oxygenflow, measured (mount time, first
     read, a full walk of the Linux tip).
   - [ ] P5b **Open in Workspace** on a change or bookmark page of the hub:
     openvscode-server (MIT, unmodified) with Oxygen VCS inside a
     Firecracker microVM per workspace, the state mounted read-only through
     casfs with an ephemeral overlay for edits, and commits going back
     through oxy as the signed-in user. The runner hosts it (the image under
     `~/flow`, a workspace job kind); the hub starts it, proxies it to the
     signed-in user only, and stops it when idle. Boot time measured, aiming
     under 100 ms for the mount. Write the design into the hub and
     oxygenflow specs first. A fork of VS Code only if an extension cannot
     do what is needed, decided with Andy.
2. - [x] **oxy on projects coming from Git**: say when a folder holding
   `.git` is skipped (oxygen#43), and honour `.gitignore` or offer to import
   it (oxygen#44).
3. - [x] **Device sign-in** for apps and `oh` (oxygenhub#67): approve a code on
   the web with password and 2FA, and the app receives a scoped token. The
   Android client needs it.
4. - [ ] **An Android build image** for OxygenFlow (oxygenflow#19): JDK,
   Android SDK and NDK, and a Gradle cache warmed offline.
5. - [ ] **OxygenHub VCS**: declare its authentication provider in the
   manifest (oxygenhub-vscode#4), released as the next patch (see the
   Marketplace notes: never reuse a version).
6. - [ ] **Import at scale**: finish oxygen#25 (per-commit import cost), then
   backlog 3's H3 (Linux history slices of 2 000, 10 000 and 50 000
   commits, verified) and H4 (vscode's full history, about 130 000
   commits, into a scratch repository that is never published).
7. - [ ] **The hub's open issues**: runner management (oxygenhub#41, backlog
   2's H12 parts), a pinned subrepo left out of a Flow job for lack of
   access (#56), faster CI (#29), required checks over the API (#20).
8. - [x] **The hub's plan** (`oxygenhub/docs/plan.md`): notifications for
   review requests, mentions and landings; the organisation webhooks page.
   Email verification and password reset need SMTP on Earth, and passkeys
   are for 1.0: skip both unless Andy says otherwise.
9. - [x] **Housekeeping**: tick plan items that are already done (for
   example `oxy seal --proposal` exists), and CHANGELOG files for oxygen,
   oxygenhub and oxygenflow like the extensions' (backlog 2's V2).

## Also

- [ ] R1 **Rebase into the working copy** (oxygen#42, Andy's choice for the
  overnight loop): when `oxy rebase CHANGE --onto TARGET` would be
  conflicted, offer to leave the conflict in the working copy to resolve and
  commit, as `merge` does, rather than refusing.
- [ ] Z1 **Closing summary.** Re-run the full benchmark and summarise, for
  Andy, where oxy matches or beats Git on the Linux tree, and where it does
  not and why.

## Log

- 2026-09-30 Backlog written from the day's measurements (state index,
  changed-only builds and the manifest cache already in).

- 2026-09-30 **B0** baseline (table in `docs/benchmarks.md`; `linux-bench.sh` now runs three of each, alternating, plus a local clone; the loop-3 script is kept as `linux-bench.sh.loop3`). oxy is ahead on the first commit (33.3 s against 38.2 s) and gc (48 s against 100 s); clean status is close (0.43 to 0.53 s against 0.34 to 0.46 s). Behind: dirty status and diff (1.5 to 2.2 s against 0.3 to 0.8 s), commit (2.5 to 3.1 s against 0.4 to 0.5 s), log (0.9 s against 0.00 s), blame (1.0 s against 0.07 s), checkout (3 s against 0.4 to 0.5 s), fsck (24 s against 13 s), and a new gap: a local clone takes 56 s and 3.3 GB against Git's 18 s. The store after gc is 375 MB (it now includes the 48 MB state index) against 295 MB.

- 2026-09-30 **S1, first round** (commits c9dca9ed to 077e899b; table in `docs/benchmarks.md`). A lister of our own replaces the ignore crate's walker (listing 0.5 s to 0.18 s); the manifest is a flat binary file with hashes and graph keys as bytes; the state index keeps each file's history encoded until used (a writer bug I made on the way was caught only on Earth, since a failed index read falls back to a full load: test builds now read every index written back); maps are built in path order; mimalloc is the allocator. Clean status is now ahead of Git (0.18 s against 0.22 s), the first commit too (24.4 s against 36.2 s), and gc (29.9 s against 70.3 s). One-file status is 0.47 s against 0.22 s. Log (S3, first part): metadata from the state index and a store cache, signature marks without reading changes: 0.20 s, of which 9 ms is the log. For Andy: `oxy log` checks the working copy first (about 0.18 s here); Git's log does not look at it. Dropping that check would make log about 20 ms, but the working-copy line would then show the last snapshot's state.

- 2026-10-01 00:00 **Second round** (6359f779 to 81a5bdd7; table in `docs/benchmarks.md`). Commit 0.67 s (from 0.96 s): the state index is brought up to date by replaying and rewritten only 16 changes behind; keys only for changed paths; attributes from the snapshot rather than a tree walk; the manifest checksummed and written without fsync; and a fix: after a status had seen an earlier edit, a commit kept that edit's stat in the manifest, so every later status took a full snapshot. Blame 0.16 s (from 0.59 s; Git 0.07 s), log 0.18 s, checkout 0.74 s each way (from 1.4 s) with two state indexes kept per workspace, and status right after a checkout that wrote 1 000 files 0.18 s (from 1.3 s). **R1** done: `oxy rebase --into-working-copy` (closes #42).

- 2026-10-01 00:40 **S6 done, clone level, P1 partly** (3287e4ff to cc1f1723). fsck 8.2 to 8.7 s (from 13.6 s; Git 12.6 s): objects checked a chunk at a time on every core, graphs built with their lines in parallel, and the last four change encodings kept rather than one (a batch that took inline text from the Linux tip and a later change in turn read the tip again at each switch). Local clone 16.8 to 17.6 s (from 32 s; Git 17.0 s): files added once and never edited are written from their text without a graph (test builds compare with the rendering), new files are written in parallel, rewritten ones one at a time (six times slower in parallel on ext4). A faster local clone would copy the source's packs as Git's `--local` does; that skips the import's checks and brings every object of the source, so it waits for Andy. **P1**: a trained zstd dictionary per pack: pack 293.5 MB (from 301 MB; Git's 295.5 MB), gc 33.7 s (Git 70 s). The whole `.oxy` stays larger than `.git` by the state index (52 MB, a cache; compressing it would slow every load).

- 2026-10-01 01:20 Flow runs #150 to #153 failed: the Rust build image had no `mimalloc` (jobs build offline from vendored crates). Rebuilt the image, imported it on Earth, pinned it in oxygen, oxygenhub and oxygenflow (oxygenflow#20); runs oxygen #154, oxygenhub #61 and oxygenflow #35 pass. Lesson: watch the run after every push, a dependency change above all.

- 2026-10-01 01:35 **S7 measured, not done.** The first commit of the Linux tree peaks at 2.7 GB (counting allocator; 3.1 GB resident): the walk holds every file's text (1.6 GB, shared with the change's inserts), building the change adds 0.5 GB, and encoding and storing adds 0.5 GB at the peak. Writing the change's encoding an edit at a time instead of through one CBOR value saved only 50 MB, so it was not kept. Under 2 GB needs the text never all in memory: the walk stores each added text file as its text blob as it reads it and keeps only the reference; the builder makes skeleton inserts (lines left out, a `TextRef` beside them, as history-only loads already hold changes); encoding, the commit's re-encoding, history rewrites and adding the change to a state take the skeleton and its texts. Rename detection still reads the contents of added files when files were removed. A day's careful work across objects, snapshot, commit and rewrite: left for a fresh start, with this as the design.

- 2026-10-01 01:40 **#43 and #44 done** (6ccc36f2, 0a4aa7fc). `.gitignore` and `.ignore` count below `.oxyignore` in each directory; the base's files are not matched against ignore rules (they apply to untracked paths only), which kept status at its speed on the Linux tree's 405 `.gitignore` files; an added file a later rule names drops out. Status (clean too), commit and `status --json` name folders holding a repository of their own. Andy's oxygen checkout now lists the five `games_to_port/` clones; listing the folder in `.oxyignore` quiets it (left to Andy). Flow run #156 passes.

- 2026-10-01 02:20 **Device sign-in done** (oxygenhub#67, f8f35f5f, deployed, Flow run #62 passes): `/api/v1/device/code` and `/device/token` in RFC 8628's shape, the `/device` approval page (signed in, choose scopes, approve or deny), the token made on the next poll so none is stored in the clear; `oh auth login` signs in this way, `--token` still works. Live check on oxygenhub.io: a code is issued and `/device` asks a signed-out visitor to sign in. The Android feedback file says how the app can use it.

- 2026-10-01 03:10 **Hub issues** (item 7): #56 done (a Flow job's log names the pinned subrepos it was not given, and why; 1ffde26b), #41 part 2 done (runners scoped to the site, an owner or one repository; claims filtered in the query; ceaf0ed2), #20 and #29 closed as already done (the required-checks API and every Oxygen repository's required checks; CI without debug info and with prebuilt dependencies, build 270 s). oxygenhub-vscode#4 closed: the authentication provider has been declared since d6657e33. Skipped: item 4, the Android build image (it means accepting the Android SDK licences, Andy's to accept). All deployed; hub Flow runs #63 and #64 pass. H4 is running again on Earth: the 10 000-commit vscode slice now imports in 136 s and 2.2 GB, and the whole history was at 2.3 to 2.5 GB after 20 minutes.

- 2026-10-01 04:00 **Items 8 and 9.** Review requests (oxygenhub#68, deployed, run #65 passes): ask someone to review on the page or through the API, notified, answered by their review, the author told of every review. The organisation webhooks page and hub-side landing through the delegate key already existed: plan items ticked, as were `oxy seal --proposal` and `oxy proposal show|land` in oxygen's plan. CHANGELOG.md for oxygen, oxygenhub and oxygenflow. Email verification, password reset (SMTP) and passkeys left for Andy, as the backlog says.

- 2026-10-01 05:15 **P5 started.** Design in the hub spec (13.3, oxygenhub#69) and the OxygenFlow spec (5.1, oxygenflow#21): a state is mounted from its state index, text files as renderings cached by graph key, opaque files as blobs; the editor runs in a microVM on a runner and commits go to a workspace bookmark signed by the hub as the user's delegate. **P5a prototype** (`flow-casfs`, 42236046): on the Linux tip, on a loaded Earth, mount 0.5 to 0.9 s, first read 0.1 s, cold full read 52 s, warm 0.65 s, identical to the working copy under `diff -r`. To do for the targets: several FUSE threads, bounded memory, read-ahead, a lazy state index. The Flow image is rebuilt with fuser vendored and pinned in all three repositories. P5b (Open in Workspace) is next, but its microVM image (openvscode-server) and the hub proxy are a large piece that is better started with Andy's eye on the design.

- 2026-10-01 05:40 **oxygenhub#41 parts 2b, 3 and 4** (deployed; hub runs #69 to #71 and oxygenflow #39 pass): Settings → Runners for repository admins and organisation owners; `runs-on` labels in pipelines matched at claim; the queue on Admin → Runners and the runner API. Only part 5 is left: Andy's Plex and VisionFive machines.

- 2026-10-01 07:25 **H4 again (item 6).** The full vscode import (166 425 commits) was OOM-killed at the 14 GB cap after 2 h 18 min. A per-commit trace (`OXY_IMPORT_TRACE=1`, 5765d0a8) showed why: ordinary commits run at 2 to 4 GB, but merges of long-lived branches (1 500 to 5 000 changes) applied those changes to built graphs that kept states share, copying them: 10 to 13.4 GB. `State::shed_graphs` (graphs back into deferred histories, each from a stored checkpoint-form blob) is now used before merges of over 100 commits and on kept states over 2 000 graphs (39d71b23); the 10 000-commit slice is 146 s and 1.6 GB (from 136 s and 2.2 GB), verified. On the full import that took the peak to 9.6 GB so far, but at a cost: about 5 600 commits per 30 minutes (from about 22 000 in the first 15), so the whole history would take about 14 hours, and the remaining peaks are merges onto rebuilt bases with no built graphs: the merge's touched files are built, rendered and diffed all at once. Next: build and diff a large merge's files in batches, shed only on merges (not on every kept state) to win the speed back, then run H4 to the end. The run continues on Earth meanwhile (`vscode-trace2`).

- 2026-10-01 08:50 **H4 stopped** at 38 771 of 166 425 commits after 3 h 05 min: with shedding, a merge of 1 390 changes still reached 13.1 GB (the cap is 14), and the pace had fallen to about 4 600 commits per 50 minutes. The fix it needs is the one above: build and diff a large merge's files in batches. Left for a fresh start; H4 waits on it.