Read the state index lazily: log, diff and commit without decoding 84 MB issue #94 open andymccall opened this 26 days ago · 7 comments

2026-10-17, on the 19,978-commit VS Code history on Earth (quiet, min of 15): log -n 20 34.5 -> 30.0 ms, clean diff 48.3 -> 29.7 ms, status 26 ms (git: log 7.4, diff 15.5, status 60.6 ms). Done: the index keeps changes in log order and the log reads them as a list with no map or sort (c680df9c, 1b6f8e9c); a clean diff loads no base. Profiled: the changes block is 9.4 MB (20k changes, messages included), so each log reads, hashes and decodes 9.4 MB; reading it through mmap did not help (the faults moved into the hash). Next, to reach 20 ms: split the block into a skeleton (hash, id, deps, author, hlc, kind; about 2.7 MB) and messages in chunks with a hash each, so the log checks and decodes the skeleton and only the chunks holding the changes it shows. The diff's remaining cost is the status check itself (stat of 7,032 files).

Measured 2026-10-18 04:0x on Earth, the 19,978-commit VS Code history, oxy 0.4.6 (current main), median of 5: status 27 ms, log -n 20 32 ms wall, show 7 ms, clean diff 26 ms.

OXY_TIMINGS for log -n 20, about 24 ms inside the process:

• reading the state index's changes block: 9.5 ms, of which decoding 20k ChangeMetas 6.4-6.8 ms and checking (BLAKE3) under 1 ms; the rest is reading;
• the other half of 'log: entries' (about 4.6 ms): working copy and heads, and a LogEntry for all 20k changes;
• id prefixes 0.9 ms, printing 2.4-3 ms.

So the split proposed above (skeleton vs messages) mostly saves the message decoding: about 3 ms, not 10. Reaching 20 ms needs the log to stop materialising every change. The index already keeps log order, so log -n N could decode only the first N + MAX_EDGE entries' skeletons. Prefixes need only change ids, which a fixed-width column would give without decoding. The catch is that metas_of feeds revsets too, which need everything. Not changed tonight.

Correction to the comment above: those medians (status 27, log 32, show 7, diff 26 ms) and the timing breakdown came from the bench's previous build (oxy 94e), not 0.4.6. Current main, built just after, measured status 56, log 36, show 8 and clean diff 32 ms, but Earth was running CI microVMs at the time (load 5.7), so neither set is a clean comparison. I will run both builds back to back on a quiet Earth before drawing anything from it.

Re-measured on a quiet Earth (load 1.5, no CI), 06:08 BST, both builds back to back twice, median of 5 after a warm-up on the 19,978-change VS Code history (~/bench/ops94.sh):

No regression: the 56 ms status I posted at 03:59 was Earth under CI load (5.7). The breakdown above still stands: reaching 20 ms for status and log means the log no longer materialising every change, not lazy messages alone (~3 ms).

Measured 2026-10-21 04:26 on a quiet Earth (load 0.95), oxy 0.5.1 release build vs git on the 19,978-commit VS Code history, median of 5 after a warm-up, two runs (~/bench/ops051q.sh):

Unchanged since 0.4.6: status ahead of git, show level, log and diff behind. Next step, as in the comments above: log -n N should decode only the first N + MAX_EDGE change skeletons from the state index (which keeps log order) instead of every ChangeMeta, with id prefixes from a fixed-width change-id column, and revsets keeping the full decode. That changes how the changes block is laid out and checked (integrity per chunk instead of one hash), so it wants Andy's go and an A/B on Earth; not done in the overnight loop.

Measured 2026-10-26 06:40 GMT on a quiet Earth (load 1.8–2.2), oxy 0.7.0 release build vs git, the 19,978-commit VS Code history, median of 5 after a warm-up, two runs (~/bench/ops070q.sh):

OXY_TIMINGS for log -n 20: command 20.8 ms in-process; state index changes 7.35 ms (decoding 4.65, checking 1.15), log metadata 8.54 ms in all, entries 13.41 ms, id prefixes 0.88 ms, print 2.42 ms. So unchanged since 0.5.1: the next step is still the one above, the log decoding only the first N + MAX_EDGE entries (and ids for prefixes) rather than all 20k, a change to the index's layout.

Plan for the layout change (2026-10-27, not started: a storage-format change, best begun in a day session):

1. Changes block v2: a skeleton column plus a records area. A header holding the count, then fixed-width rows in log order: change hash (32 B), change-id prefix (8 B), kind, HLC, and the offset and length of its record. Each record holds deps, author, message and signature as now. The block's BLAKE3 stays a whole-block check on read, so a damaged index is still rebuilt, never trusted.
2. log -n N reads the skeleton and the first N + MAX_EDGE records only. Decode by offset; no Vec<ChangeMeta> of 20k. Id prefixes come from the fixed-width column without decoding records.
3. Revsets and everything else call the existing whole-block decode (metas_of) unchanged, so only the log's path moves.
4. Compatibility: the state index is a per-workspace cache. A v1 index is read as now and rewritten as v2 on the next write; nothing on the wire or under a signature changes. Runners and hubs need no rollout first: unlike the policy-subject change of 2026-10-17, this is local to each repository's own files.
5. Measure on a quiet Earth with ~/bench/ops94.sh (VS Code, 19,978 changes, median of 5, warm), log -n 20 and clean diff. Target: log under 20 ms.
6. Tests: v1 → v2 rewrite round trip; a damaged v2 block is rebuilt; log -n gives the same output as the whole decode for N = 1, 20 and all; id prefixes match.

Expected: log -n 20 drops the ~9.5 ms block read and decode to about 1 ms for the skeleton, plus the 20 records. The ~4.6 ms of LogEntry building for all 20k changes goes too, once the log stops materialising every change.