OXYGENHUB
# 33e77a5526fa oxygen / docs / backlog-overnight-3.md
7.0 KiB · Markdown · last changed Backlog 3: not worse than Git 1 month ago renderedsource blame history raw
1 
# Overnight backlog 3 (from 2026-09-29): not worse than Git
2
3 
Andy's brief: there is no point making a tool that is worse than Git.
4 
Until oxy matches Git on a real, large tree, this is the only work:
5 
speed, memory and space at scale, with correctness never traded away.
6
7 
Work top to bottom. An item is done when:
8 
- its tests pass and the gate is clean;
9 
- it is measured before and after on Earth, and the numbers are in
10 
  `docs/benchmarks.md`;
11 
- it is committed with `oxy` as Andy and pushed to Earth.
12
13 
If an item is too large, split it here and do the first part. Log each
14 
iteration at the bottom.
15
16 
## The yardstick
17
18 
The Linux tip on Earth: 95 943 files, 1.8 GB, on 4 cores with 31 GiB.
19 
Git 2.53 does this (from `~/bench/linux/results.txt`, 2026-09-29):
20
21 
| Operation | Git | Target for oxy |
22 
|---|---:|---:|
23 
| First commit of the tree | 33.7 s, 42 MB | at most 50 s, under 2 GB |
24 
| Clean `status` (warm) | 0.26 to 0.31 s, 31 MB | at most 0.4 s, under 300 MB |
25 
| `status`, one file edited | 0.25 s | at most 0.4 s |
26 
| `diff`, one file edited | 0.34 s | at most 0.5 s |
27 
| `commit`, one file edited | 0.43 s | at most 0.6 s |
28 
| `status`, 1 000 files edited | 0.26 s | at most 0.5 s |
29 
| `diff`, 1 000 files edited | 1.38 s | at most 2 s |
30 
| `commit`, 1 000 files edited | 0.76 s | at most 1.2 s |
31 
| `log -n 100` | 0.05 s | at most 0.1 s |
32 
| `blame MAINTAINERS` | 0.07 s | at most 0.2 s |
33 
| Checkout of another state and back | 0.49 s and 0.38 s | at most 1 s each |
34 
| Store after gc | 311 MB (`.git`) | at most 350 MB |
35 
| Everyday commands | about 31 MB | under 300 MB |
36
37 
The targets allow oxy a small margin, because it does more (conflicts
38 
as data, signatures, an operation log). A target that turns out to be
39 
out of reach for a structural reason is written up here, with the best
40 
number reached and why. It is never quietly dropped.
41
42 
## Rules for this loop
43
44 
- **Measure first.** Before changing code, measure where the time or
45 
  memory goes on Earth: `/usr/bin/time -v`, `perf record` (installed on
46 
  Earth), `OXY_TIMING` phases, or a counting allocator behind a feature.
47 
  Record what the profile showed in the log.
48 
- **Heavy work only on Earth**, always under a cap:
49 
  `systemd-run --user --scope -p MemoryMax=14G …`. Andy's workstation has
50 
  crashed from memory before. Locally, run the gate and small cases only.
51 
- **Getting code to Earth.** `~/bench/oxygen` on Earth is a plain copy of
52 
  the source. Sync it with `rsync -a --delete --exclude target --exclude
53 
  .oxy -e "ssh -i ~/.ssh/earth_claude" ~/development/oxygen/
54 
  claude@earth:bench/oxygen/`.
55 
  - `~/bench/linux-bench.sh` builds oxy from that copy and runs the whole
56 
    comparison. `~/bench/first-commit.sh` runs only the first commit,
57 
    status and fsck.
58 
  - Run long jobs with Bash `run_in_background`, and wait for the
59 
    completion notice rather than polling.
60 
- **Correctness is never traded.** Every change keeps these passing:
61 
  `oxy fsck` on the Linux repository, the property tests, and
62 
  `oxy git verify` on the Git imports.
63 
- **Format changes** (change, checkpoint, pack or wire) are allowed
64 
  before 1.0, with these steps:
65 
  - bump the spec draft and keep old objects readable with unchanged
66 
    hashes;
67 
  - bump `deps/oxygen` in oxygenflow with `make pins`, gate, commit,
68 
    push and `make deploy-runner`;
69 
  - then `make pins` in oxygenhub, gate, commit, push, `make deploy`,
70 
    and check `/health`;
71 
  - only then install the new oxy locally. Commit the format change
72 
    itself with the old binary.
73 
- **Memory and pace.** Every workstation build and test runs under
74 
  `systemd-run --user --scope -p MemoryMax=8G`, and
75 
  `CARGO_INCREMENTAL=0` (the disk is nearly full).
76
77 
## Memory
78
79 
- [ ] M0 **Tools to see memory.** Add a counting global allocator behind
80 
  a cargo feature (peak bytes and allocation count per `OXY_TIMING`
81 
  phase), so a profile names the structure that holds the memory. Then
82 
  profile a clean `status` of the Linux repository, killed at 14 GB
83 
  (oxygen#35), and the first commit (9.9 GB). Write down which
84 
  structures hold what before changing anything.
85 
- [ ] M1 **Status without every graph** (oxygen#35). A clean `status`
86 
  must not build every file graph. Unchanged files are skipped by the
87 
  manifest (spec §12.4), and the first status after a commit must not
88 
  replay the whole change. Options: the commit writes a checkpoint;
89 
  history-only states load graphs on demand (`State::load_graphs`).
90 
  Target: under 300 MB and 0.4 s.
91 
- [ ] M2 **Lines without an allocation each.** An insert's lines become
92 
  one shared buffer with line offsets (for example `Arc<[u8]>` and
93 
  ranges) in the change and in the file graph. The file graph shares
94 
  that buffer and does not copy it. `AtomInfo` and the edge map shrink.
95 
  Measure bytes per line before and after.
96 
- [ ] M3 **A commit that streams.** The snapshot builder stores each
97 
  file's text blob as it reads it and keeps no content it no longer
98 
  needs, and the state is built without a second copy. Target for the
99 
  first commit: under 2 GB.
100 
- [ ] M4 **Everyday commands under 300 MB** on the Linux repository:
101 
  status, diff, commit, log and blame, one file and 1 000 files edited.
102
103 
## Speed
104
105 
- [ ] S1 **First commit at most 50 s** (55.3 s now): profile with
106 
  `perf`, and parallelise reading, hashing and diffing across the four
107 
  cores where it is safe.
108 
- [ ] S2 **Status, diff and commit** with one and with 1 000 files
109 
  edited, to the targets above.
110 
- [ ] S3 **log, blame and checkout** on the Linux repository, to the
111 
  targets above.
112 
- [ ] S4 **fsck and gc** measured against Git's. Re-run Git's `fsck`
113 
  with gc turned off: 451 s is suspect, because Git's own background gc
114 
  raced it.
115
116 
## Space
117
118 
- [ ] P1 **Store at most 350 MB after gc** (442 MB now against Git's
119 
  311 MB): measure what takes the space (changes, text blobs, graphs,
120 
  checkpoints, indexes), then fix the largest. Compression level and
121 
  dictionary, and packing order, are all candidates.
122
123 
## History at scale
124
125 
- [ ] H1 **Importer name conflict** (oxygen#30): the 5 000-commit
126 
  vscode import differs from Git in 51 files. Fix the cause and add a
127 
  test. `oxy git verify` must be clean.
128 
- [ ] H2 **Import memory**: drop graphs untouched by recent commits and
129 
  reload them on demand, so memory is bounded by the working set and not
130 
  by the history.
131 
- [ ] H3 **Linux, stage 2**: import slices of recent history (2 000,
132 
  10 000 and 50 000 commits), each verified, then time log, blame and
133 
  status against Git on the same slice.
134 
- [ ] H4 **vscode's full history** (about 130 000 commits) into a scratch
135 
  repository on Earth. It is not published. Verify it and time it.
136 
- [ ] H5 **Linux's full history** (about 1.3 million commits) if H3 shows
137 
  it can finish overnight. Otherwise, write up what stands in the way.
138
139 
## Finally
140
141 
- [ ] Z1 Re-run the full `~/bench/linux-bench.sh` with the final build,
142 
  and put the complete Git against oxy table in `docs/benchmarks.md`.
143 
  Summarise, for Andy, where oxy now matches or beats Git and where it
144 
  does not.
145
146 
## Log
147
148 
- 2026-09-29 Backlog written. Starting point: the Linux first commit
149 
  succeeds (55.3 s, 9.9 GB, 442 MB store) with change format 2
150 
  (oxygen#34). A clean `status` is killed at 14 GB (oxygen#35).