OXYGENHUB

A base index: status and diff without loading history issue #41 open andymccall opened this · 1 comment

andymccall

Backlog 3, S2. On the Linux tip a status with one file changed takes about 2.3 s against Git's 0.23 s, and nearly all of that is rebuilding the working copy's base from history every time:

  • fetch the first change: 0.17 s
  • decode its skeleton: 0.19 s
  • apply it: 0.21 s
  • tree view: 0.29 s
  • graph keys: 0.10 s

The walk (0.37 s), rendering the changed file (0.16 s), the diff (0.15 s) and the manifest (0.1 to 0.15 s) are the rest.

Proposal: persist what the base contributes, per workspace, beside the manifest, keyed by the base heads. This is Git's index in effect.

  • Contents: the tree view entries (path, node, kind, mode, placement, mode atom), every text file's graph key, and each node's deferred history (the changes that edit it, with their text refs).
  • Snapshot: reads it instead of replaying history, and builds graphs only for changed files from their deferred entries.
  • Written when: after any command that loads the base anyway (commit, checkout, the first snapshot of a base).
  • Trusted when: only while the base heads match.
  • Otherwise: fall back to a load.
  • Open questions:
    • encoding (a flat sorted table read without a generic CBOR tree);
    • invalidation when the workspace's base moves;
    • fsck of the cache.

Expected effect: the one-file status goes from 2.3 s to the walk plus the changed file's work, about 0.6 s. This is a design change; worth a look before it is built.

andymccall

Approved, in this shape: changes stay the only source of truth, and every index is derived from a state, keyed by the state's hash, rebuildable at any time and checked by fsck against a full rebuild.

  1. Tree index: per path the node, kind, mode, placement, mode atom, graph key and rendered-content hash; per node its deferred history (the changes that edit it, with text refs). status, diff and commit read it and the manifest, and build graphs only for changed files. Target: the walk plus the changed files' work, near Git.
  2. Change index: topological order, generation numbers and where each change's metadata is, like Git's commit-graph, so log -n 100 reads 100 entries.
  3. Node index: per node the changes that touch it, so one file's graph (blame, diff) costs that file's history, not the repository's.
  4. Checkout by comparing two tree indexes: only files whose rendered hash differs are written.

Each index is built from the parent state's index plus one change (as the tree view and file orders are now kept), in a flat binary layout read without a generic CBOR tree. A test build compares every incremental build with a full one. Work in that order, with the Linux benchmark against Git after each step.

referenced by 27 changes on 30 Sep 2026

f9575918477f d0990c2bff74 23ff1288a5dd 05059d434aca 992467d31d26 60622d23b9e4 930e5f3f1682 61e9ad6ddc2d e4b8bcf04a86 76740263089c 1966f5db15ee 3f1b8fe25ec9 36e51aa806c6 bd9414401fdc 4cce912a0953 ce14f0d68b15 e0d56dfca51d d616faf5d136 f57bf6702628 f9466021286a 073a2115995f 638fb21420d7 e82591ec33a4 091fef80755d 72215f29df4c a17fc622ca43 acadb5ed28a1

referenced by 17 changes on 1 Oct 2026

03a3e8ad8cf6 7f75b41c107d cb9a097def6f 8fd78eb52ad2 11e2b98a3a39 cd19603782bf 0ad273ca4fcf 3604179679eb 77445673174b 1ec3f008e96a 10b4fd430eab e226a6cc2f71 b30a880fa409 cd4eb55e3b3d 33472ccc08cc 5ae7533f1888 e96a430c78f3

referenced by 11 changes on 2 Oct 2026

fddf64baa3d2 0389c3e82f6c 236f35c0baba 360aa74a6277 4f264d6f20b6 5b4c51307b14 dd6147dcf9c2 87828a0b1f36 631fc8970bd0 94cdbc576ce1 30fc48081b94

andymccall set labels performance ·