OXYGENHUB
# e266b7d4065c oxygen / docs / benchmarks.md
24.5 KiB · Markdown · last changed fsck checks a change's text as its blobs, without decoding it 1 month ago renderedsource blame history raw
1
# Benchmarks
2
3
The harness is `crates/oxy-core/tests/bench.rs` (spec §25). It builds a
4
repository through the library, one commit at a time, then times the
5
everyday operations against thresholds:
6
7
    OXY_BENCH_DIR=~/.cache/oxy-bench \
8
      cargo test --release -p oxy-core --test bench -- --ignored --nocapture
9
10
`OXY_BENCH_FILES` (default 10 000) and `OXY_BENCH_CHANGES` (default
11
5 000) set the size. The data lands under `OXY_BENCH_DIR`; `/tmp` is often
12
a RAM disk, which hides the cost of `fsync`, so measure on a real disk.
13
14
## 2026-09-28: first run
15
16
Machine: 8 cores, x86-64, NVMe (ext4 on LVM), release build, warm cache.
17
Files are 21 lines each in 100 directories; one change in ten appends a
18
line to `hot.txt`, the rest rewrite one ordinary file.
19
20
| Operation (10 000 files, 5 000 changes) | Time | Threshold | |
21
|---|---:|---:|---|
22
| commit: import 10 000 files | 1 031 ms | 60 000 ms | ok |
23
| commit: one file (mean of 4 999) | 3 761 ms | 250 ms | **over** |
24
| status: clean working copy (stat crawl) | 2 977 ms | 2 000 ms | **over** |
25
| status: 10 files modified | 3 062 ms | 2 000 ms | **over** |
26
| commit: 10 files | 9 089 ms | 500 ms | **over** |
27
| log: whole bookmark | 408 ms | 2 000 ms | ok |
28
| blame: hot file (500 lines) | 2 920 ms | 2 000 ms | **over** |
29
| blame: ordinary file | 2 927 ms | 1 000 ms | **over** |
30
| checkpoint | 596 ms | 10 000 ms | ok |
31
| status after checkpoint | 2 873 ms | 2 000 ms | **over** |
32
| push: everything to an empty remote | 782 ms | 60 000 ms | ok |
33
| push: up to date | 4 ms | 1 000 ms | ok |
34
| clone: everything, with checkout | 3 378 ms | 60 000 ms | ok |
35
| fsck | 7 955 ms | 60 000 ms | ok |
36
| gc | 9 489 ms | 60 000 ms | ok |
37
| status after gc | 2 829 ms | 2 000 ms | **over** |
38
39
Peak resident memory over the whole run: 1 405 MiB. The run took 5 h 14 min,
40
almost all of it building the history, because each commit costs as much
41
as a status.
42
43
Smaller runs, same machine:
44
45
| Files, changes | Mean commit | Clean status | Blame (hot) | Peak memory |
46
|---|---:|---:|---:|---:|
47
| 1 000, 200 | 90 ms | 62 ms | 62 ms | 70 MiB |
48
| 1 000, 800 | 166 ms | 130 ms | 119 ms | 118 MiB |
49
| 4 000, 200 | 343 ms | 235 ms | 239 ms | 254 MiB |
50
| 10 000, 5 000 | 3 761 ms | 2 977 ms | 2 920 ms | 1 405 MiB |
51
52
### What it shows
53
54
Sync, log, checkpoints and repository maintenance are fine. Everything
55
that snapshots the working copy (status, commit, blame of the working
56
copy's base) grows with both the number of files and the length of the
57
history. Timing the phases of one snapshot at 4 000 files and 300 changes:
58
59
| Phase | Time |
60
|---|---:|
61
| load the base state (checkpoint plus the changes since) | 75–130 ms |
62
| materialize every file of the base state | 75 ms |
63
| walk the working copy against the manifest | 50 ms |
64
| build the working-copy change | 8 ms |
65
66
A clean status should be a stat walk and nothing else. The fix is backlog
67
item O10: skip loading and materializing when every file's stat matches a
68
manifest written for the same base, make the base state lazy for the
69
callers that need it, and cache materialized snapshots per state. The
70
thresholds above stay as they are until then, so the harness reports the
71
gap rather than hiding it.
72
73
## 2026-09-28: after O10a
74
75
`State::maximal` compared every pair of heads, and the benchmark's
76
bookmark has about 4 500 of them (changes to different files are
77
independent). It now walks once; commit also stops loading the base state
78
twice. Same machine and size:
79
80
| Operation (10 000 files, 5 000 changes) | Before | After | Threshold | |
81
|---|---:|---:|---:|---|
82
| commit: one file (mean of 4 999) | 3 761 ms | 861 ms | 250 ms | **over** |
83
| status: clean working copy | 2 977 ms | 803 ms | 2 000 ms | ok |
84
| status: 10 files modified | 3 062 ms | 864 ms | 2 000 ms | ok |
85
| commit: 10 files | 9 089 ms | 861 ms | 500 ms | **over** |
86
| log: whole bookmark | 408 ms | 400 ms | 2 000 ms | ok |
87
| blame: hot file | 2 920 ms | 809 ms | 2 000 ms | ok |
88
| blame: ordinary file | 2 927 ms | 828 ms | 1 000 ms | ok |
89
| status after checkpoint | 2 873 ms | 770 ms | 2 000 ms | ok |
90
| push: everything | 782 ms | 742 ms | 60 000 ms | ok |
91
| clone: everything | 3 378 ms | 1 664 ms | 60 000 ms | ok |
92
| fsck | 7 955 ms | 6 872 ms | 60 000 ms | ok |
93
| gc | 9 489 ms | 9 526 ms | 60 000 ms | ok |
94
95
Peak resident memory: 1 145 MiB (was 1 405). The run took 1 h 12 min
96
(was 5 h 14 min). Commit time no longer grows with history; what remains
97
grows with the number of files: at 10 000 files a snapshot spends about
98
320 ms loading the base state, 190 ms materializing it, 130 ms walking the
99
working copy and 40 ms building the change. That is O10b.
100
101
## 2026-09-28: after O10b
102
103
Rendering the base state, building file graphs from a checkpoint and
104
reading the working copy now run on all cores. Measured at 10 000 files
105
with 60 and 300 changes (commit time no longer depends on the length of
106
history, so the full 5 000 was not rerun):
107
108
| Phase of one snapshot, 10 000 files | After O10a | After O10b |
109
|---|---:|---:|
110
| load the base state | 320 ms | 180–280 ms |
111
| materialize the base state | 190 ms | 75 ms |
112
| walk and read the working copy | 130 ms | 80 ms |
113
| build the working-copy change | 40 ms | 40 ms |
114
| write the manifest | – | 46 ms |
115
116
| Operation, 10 000 files | After O10a | After O10b | Threshold |
117
|---|---:|---:|---:|
118
| status: clean working copy | 803 ms | 464–581 ms | 2 000 ms |
119
| commit: one file (mean) | 861 ms | 550–605 ms | 250 ms |
120
| commit: 10 files | 861 ms | 583–635 ms | 500 ms |
121
122
Loading the whole base state is now the largest cost. O10c removes it: a
123
snapshot should load and render only the files whose stat changed, which
124
needs checkpoints that can be read per file.
125
126
## 2026-09-28: O10c, first part: the in-memory state
127
128
Profiling a clean snapshot with `OXY_TIMINGS=1` (10 000 files of 40 lines,
129
200 changes) showed loading the base state dominating it. Applying the
130
changes to the in-memory graphs cost 650 ms and reading them 160 ms. A
131
checkpoint did not help: decoding one took 680 ms, as long as replaying
132
every change, because both build the same graphs for 400 000 lines. So the
133
cost was in the graph structure, not in the format:
134
135
- **Hashing.** The graph maps used SipHash over 36-byte keys, three to
136
  five times per line. The keys begin with BLAKE3 hashes, so they now use
137
  a seeded multiply-rotate hasher (`fasthash.rs`). The seed is random per
138
  process, so keys chosen by someone else cannot be aimed at one bucket.
139
- **The predecessor map.** It was written for every line and read nowhere,
140
  so it is gone.
141
- **Successor sets.** A `BTreeSet` per vertex became a sorted edge list
142
  that keeps its single successor inline. Nearly every line has one.
143
- **Line bytes.** Loading moves them into the graph instead of copying
144
  them, and chains each new line with one lookup instead of three.
145
146
Both binaries were built from the same tree and run alternately, nine
147
times each, medians shown:
148
149
| 10 000 files, 200 changes | Before | After |
150
|---|---:|---:|
151
| status: clean working copy | 0.96 s | 0.52 s |
152
| commit: one file | 0.99 s | 0.51 s |
153
| peak memory (either) | 657 MB | 239 MB |
154
155
The checkpoint format is unchanged. What remains of O10c is split into
156
O10d and O10e in the backlog:
157
158
- **O10d.** Skip reading text files whose stat matches the manifest, when
159
  the manifest recorded them equal to the base.
160
- **O10e.** Checkpoints readable per file, so a snapshot loads only the
161
  graphs it needs. This one changes the checkpoint format in spec §25.
162
163
## 2026-09-28: O10g, first part: status snapshots skip unchanged files
164
165
The manifest now records, per text file equal to the base, the key of the
166
base graph it matched: the file's node, its checkpoint graph (blob,
167
inline value or none) and the set of later changes that edited it
168
(spec §12.4). When a file's stat and key both still match, the snapshot
169
neither reads nor renders it.
170
171
`status`, `log`, `show`, `tag` and `bookmark` use a partial snapshot. It
172
loads the base for its history and builds only the graphs of files that
173
are not known unchanged. `commit`, `diff` and the rest still build the
174
whole base.
175
176
10 000 files of 40 lines, 22 changes, a version 2 checkpoint after 21 of
177
them; `OXY_TIMINGS=1`, clean working copy:
178
179
| Operation | Before | After |
180
|---|---:|---:|
181
| status | 0.55 s | 0.22 s |
182
| status peak memory | 259 MB | 51 MB |
183
| log | 0.7–0.8 s | 0.38 s |
184
| diff (full snapshot) | 0.56 s | 0.57 s |
185
186
| Phase of a clean status snapshot | Before | After |
187
|---|---:|---:|
188
| load base (whole / history only) | 317 ms | 31 ms |
189
| render base | 91 ms | 25 ms |
190
| read working copy | 68 ms | 50 ms |
191
| diff against base | 41 ms | 28 ms |
192
| write manifest | 33 ms | 35 ms |
193
194
The same for `commit` and `diff` comes next (O10h). There, only the graphs
195
the change touches are needed, and materialized states can be cached.
196
197
## 2026-09-29: O10h, commit, diff and show build only what they touch
198
199
Commit, working-copy diff, hunks and discard use the partial snapshot of
200
O10g. Files known unchanged are neither read nor rendered, and only the
201
graphs of the files the working copy changes are built. Showing a
202
committed change (`oxy show`, `oxy diff CHANGE`) builds the graphs of the
203
files that change edits and leaves the rest unrendered.
204
205
Same repository as O10g (10 000 files of 40 lines):
206
207
| Operation | After O10g | After O10h |
208
|---|---:|---:|
209
| diff (working copy) | 0.57 s, 244 MB | 0.22 s, 52 MB |
210
| commit (one file) | 0.58 s, 245 MB | 0.23 s, 52 MB |
211
| show (a one-file change) | not measured | 0.53 s, 145 MB |
212
213
Caching materialized states, planned alongside, is not needed now that
214
these commands render almost nothing.
215
216
## 2026-09-29: P5c, importing microsoft/vscode, first part
217
218
`oxy git import` on slices of microsoft/vscode's `main` (166 412 commits in
219
total), taken in the importer's own order (`git log --reverse
220
--topo-order`). The first 5 000 commits reach May 2016 with about 4 700
221
files.
222
223
The importer did work proportional to the whole tree for every commit:
224
- It cloned the previous state.
225
- It rebuilt a merge's first parent from history.
226
- It rendered, read and compared every file.
227
- It scanned every entry, with an allocation each, to decide whether each
228
  directory was still needed.
229
230
Now:
231
- Files with the same Git blob and mode as in the first parent's tree are
232
  marked unchanged and skipped, as a clean working copy's are. Paths that a
233
  merge's other parents touched are excluded.
234
- A commit's state is kept while first-parent children remain, up to 8,
235
  and the last child takes it without a clone.
236
- Needed directories come from one set of ancestors. That code is shared
237
  by snapshot and commit.
238
239
| Slice | Before | After |
240
|---|---:|---:|
241
| 300 commits (35 merges) | 65.1 s | 13.6 s |
242
| 1 000 commits (95 merges) | 271 s | 120 s, 3.3 GB |
243
| 5 000 commits (360 merges) | not finished after 80 min | 412 s, 6.3 GB, 17 MB store |
244
245
Where the 5 000-commit import spends its time (summed per phase): render
246
base 172 s, build change 99 s, bases 65 s, `git ls-tree` 56 s, read blobs
247
10 s, store 5 s.
248
249
Correctness: `oxy git verify` reproduces every commit of the 300 and
250
1 000-commit imports (1.9 million file comparisons for 1 000). A test
251
covers the case that must not be skipped: a `-s ours` merge whose other
252
side edited a file.
253
254
Next: rendering the base is still O(files) per commit, and memory grows
255
with the kept states. The full import follows.
256
257
## Linux, stage 1: the tip as one commit (2026-09-29)
258
259
The Linux tree at its tip: 95 943 files, 1.8 GB, on Earth (4 cores,
260
31 GiB), each run capped at 14 GB.
261
262
| | Git | oxy before | oxy, change format 2 |
263
|---|---:|---:|---:|
264
| First commit | 33.7 s, 42 MB | failed, 10.9 GB | 55.3 s, 9.9 GB |
265
| Store | | unreadable | 442 MB |
266
| Clean `status` | 0.26 to 0.31 s | | killed at 14 GB |
267
268
Before format 2 the commit made one change of 1.8 GB. That was over the
269
1 GiB an object may be, and the repository would not open afterwards
270
(oxygen#33, oxygen#34). Format 2 stores each insert of 4 KiB or more as
271
a text blob, so the change stays small.
272
273
Memory is the next problem. Every line is kept as its own allocation,
274
once in the change and again in the file graph, and `status` builds
275
every graph (L1b).
276
277
### Status after the commit (2026-09-29, oxygen#35)
278
279
A profile with the counting allocator (`--features alloc-stats`,
280
`OXY_TIMINGS=1`) of the first clean `status`, before it was killed:
281
282
| Phase | Time | Live after | Allocated |
283
|---|---:|---:|---:|
284
| Load base history | 17.5 s | 3.2 GB | 16 GB in 96 M allocations |
285
| Read working copy | 79.5 s | 4.9 GB | 1.8 GB |
286
| Load and render base | killed at 14 GB | | |
287
288
It had two causes, and both are fixed:
289
- **History held every line.** A history-only state now reads a stored
290
  insert's text blob only when it builds that file's graph, and a load no
291
  longer hashes each change again, which for format 2 means chunking all
292
  its text.
293
- **The manifest had no graph keys.** They are now recorded at commit, so
294
  the next status neither reads nor renders an unchanged file.
295
296
| Clean `status` on the Linux tip | Time | Peak RSS |
297
|---|---:|---:|
298
| Before | killed at 14 GB | |
299
| Lazy text and keys at commit | 3.7 s | 768 MB |
300
| And answered from the manifest | 0.72 s | 120 MB |
301
| And the manifest checked by a parallel walk | 0.44 s | 123 MB |
302
| Git | 0.22 s | 31 MB |
303
304
A clean status is now answered from the manifest alone, loading no
305
history. The manifest also records the executable bit, since a mode
306
change alters neither size nor time. Any change, a manifest older than
307
the racy window, or subrepos send it back to a full snapshot.
308
309
Still to do: status to the 0.4 s target (the walk and reading the
310
manifest); a status without a usable manifest still builds every graph
311
and is killed at 14 GB; `fsck` is killed at
312
14 GB; the first commit takes 57 s and 10.2 GB.
313
314
### Checkout and clone (2026-09-29, oxygen#35)
315
316
A checkout built and rendered every file of the target, read the whole
317
working copy, and after writing walked and hashed it all again for the
318
manifest. Switching bookmarks or cloning the Linux tree was killed at
319
14 GB. Now:
320
- a file is skipped when the manifest shows it unchanged and its graph is
321
  the same in both states;
322
- the rest are rendered 2 048 at a time, and each batch's graphs are
323
  dropped once written;
324
- the manifest is built from what the checkout already knows.
325
326
| Linux tip | oxy before | oxy now | Git |
327
|---|---:|---:|---:|
328
| Switch bookmark (one file differs) | killed at 14 GB | 7.1 to 7.4 s, 1.3 GB | 0.18 to 0.22 s, 39 MB |
329
| Local clone | killed at 14 GB | 98 s, 7.9 GB | 9.8 s, 469 MB |
330
331
A history-only load now reads a change's skeleton: the inline lines of
332
its inserts are stepped over, not decoded, and are read from the change
333
only for a graph that is built. A one-file switch then takes 5.4 to 5.6 s
334
and 0.8 GB, and the status after it 3.3 s and 600 MB (from 3.8 s).
335
336
A checkout of a working copy that is clean by the manifest now skips the
337
status snapshot, and a manifest entry inside the racy window is checked
338
by hashing the file rather than distrusting the whole manifest. A
339
one-file switch then takes 3.7 to 3.9 s and 705 MB. Where it goes:
340
checking the manifest (a walk of 96 000 files) 0.75 s, the base's and
341
the target's history 0.5 s each, their tree views 0.3 s each, planning
342
0.2 s, and writing the manifest 0.6 s.
343
344
A checkout now needs no base state at all: after the manifest check or a
345
snapshot, the manifest says which graph each file on disk renders. A
346
file is skipped when its manifest key is the target's, and a file left
347
as it was keeps its stat in the new manifest instead of being statted
348
again. What the new state lacks is found by a walk from the old heads
349
that stops at the target. A one-file switch takes 2.1 to 2.2 s and 455
350
MB: the manifest check 0.4 s, the target's history 0.5 s and tree view
351
0.3 s, planning 0.16 s, and writing the manifest 0.4 s.
352
353
Before that, most of a switch was loading history: the base's and the target's
354
history each decode the one huge first change (1.2 s and 364 MB each).
355
A clone holds the whole bundle in memory.
356
357
### log, blame and fsck (2026-09-30, oxygen#35)
358
359
The first full run of `~/bench/linux-bench.sh` against this work found
360
`log -n 100` taking 28 s and 4.7 GB, and `blame` killed at 14 GB:
361
- **log.** Checking each change's signature read the whole change, and
362
  for the first change that meant all 1.8 GB of its text. Signature
363
  checks, the dependency walks of fetch and push, and the prompt now
364
  read a change's metadata from its skeleton, and log answers the
365
  working copy's state from the manifest when it proves it clean.
366
- **blame.** It took a full snapshot, building every graph, to blame one
367
  file; it now loads the base's history and builds that file's graph.
368
- **fsck.** A bookmark's state was checked by loading it whole. It is
369
  now checked a batch of graphs at a time, deleted files' graphs
370
  included.
371
372
| Linux tip | Before | Now | Git |
373
|---|---:|---:|---:|
374
| `log -n 100` | 28 s, 4.7 GB | 2.0 s, 435 MB | 0.16 s |
375
| `blame MAINTAINERS` | killed at 14 GB | 1.1 s, 315 MB | 0.31 s |
376
| `fsck` | killed at 14 GB | 123 s, 4.5 GB | to measure again |
377
378
### One file changed (2026-09-30, oxygen#35)
379
380
A status with one file changed (MAINTAINERS) took 4.1 s: the walk ran
381
on one thread (0.83 s), and the manifest was written by statting all
382
96 000 files again (0.57 s). The walk now runs in parallel and each
383
file's stat is carried from the walk into the manifest. It now takes
384
2.5 s: the base's history 0.5 s, its tree view and graph keys 0.38 s,
385
the walk 0.35 s, rendering the changed file 0.47 s, the diff 0.18 s,
386
and the manifest 0.14 s. (Git: 0.23 s.) The tree view is now computed once per
387
state and cloned for each use, which brings rendering to 0.2 s and the
388
status to 2.3 s.
389
390
### The first commit (2026-09-30, oxygen#35)
391
392
A profile of the first commit of the Linux tip (61 s, 10 GB RSS): read
393
the working copy 2.3 s (1.6 GB of content), diff against the empty base
394
6.2 s (4.2 GB live, a Vec per line), hash and record 18.8 s, then the
395
commit itself about 33 s. The text of the one huge change was chunked
396
and hashed three or four times on one thread (hashing the working-copy
397
change, storing it, storing the commit, adding it to the state), the
398
commit copied every line of the working copy, and the chunks were
399
compressed on one thread.
400
401
Now:
402
- a snapshot encodes the working copy once, and hashes and stores that
403
  one encoding;
404
- a whole commit moves the working copy's lines rather than copying
405
  them, and joins the state under the hash it was stored by;
406
- text blobs are made on every core, 256 inserts at a time;
407
- the chunks of a large change are compressed on every core
408
  (`Store::put_many`).
409
410
| First commit, Linux tip | Time | Peak (counted) | RSS |
411
|---|---:|---:|---:|
412
| Before | 61 s | 7.1 GB | 10.1 GB |
413
| After | 40.7 s | 7.4 GB | 8.7 GB |
414
| Texts stored a batch at a time while encoding | 41.6 s | 4.8 GB | 5.8 GB |
415
| Lines in one shared buffer per insert (Earth under load) | 50.1 s | 4.3 GB | 4.9 GB |
416
| Git | 33.7 to 37.9 s | | 42 MB |
417
418
Storing a change's texts a batch at a time as they are made (the
419
encoder hands each batch of 256 to the store) removed the storing peak.
420
The peak was then the working copy's content (1.6 GB) and its lines, a
421
Vec each (2.6 GB). An insert's lines are now one buffer with line ends
422
(`Lines`), shared by the change and by the file graph, whose atoms point
423
into it: the diff of the whole tree takes 3.3 s and 1.1 million
424
allocations (from 6.2 s and 45 million), and the peak is 4.3 GB (4.9 GB
425
RSS). That run shared Earth with Flow jobs (load average 6.5 on 4 cores),
426
so its total time is not comparable with the rows above.
427
428
### Clone (2026-09-30, oxygen#35)
429
430
A local clone of the Linux tip took 96 s and 5.2 GB RSS. Its checkout
431
spent most of its time rendering: working out each file's line order
432
allocated three sets per line (the contraction of the graph), 93 GB in
433
all, although every file of a fresh tree is one chain of lines. And the
434
import hashed each change three times, for the huge first change
435
chunking its 1.8 GB of text each time.
436
437
- A graph that is one chain from start to end with nothing deleted (an
438
  edge to End besides the next line orders nothing) is read off in
439
  order; any other graph takes the general algorithm, which also skips
440
  its contraction where a line's successors are all kept. A test checks
441
  the two agree.
442
- A bundle carries each change's hash when its maker knows it (the store
443
  it came from, or the received bytes); never from the wire. An import
444
  hashes each change once, by storing it, and checks it is the named one.
445
446
| Local clone, Linux tip | Time | RSS |
447
|---|---:|---:|
448
| Before | 96 s (81.6 s in a quieter run) | 5.2 to 5.4 GB |
449
| After | 61.0 s | 4.1 GB |
450
| The change read once for all batches | 56.8 s | 4.7 GB |
451
| Changes moved as stored (load average 5.2) | 48.2 s | 3.9 GB |
452
| Git | 9.6 s | 469 MB |
453
454
A fetch between repositories on one machine now moves changes as they
455
are stored, their text blobs and chunks among the objects, instead of
456
decoding every change's text on one side and chunking, hashing and
457
compressing it again on the other; the import stores all the objects at
458
once, compressed on every core, and refuses a change whose text blob did
459
not come.
460
461
### Git import: the vscode slice (2026-09-30, oxygen#30)
462
463
The 1 747-commit vscode slice (to merge 0babb4299131) had one file as a
464
name conflict at every commit from that merge on: 51 verify differences.
465
There were three causes, all from Git having no identity for a directory:
466
- a directory deleted on one line and made again got a second node for
467
  the same path;
468
- a placement left under a directory another line removed was out of
469
  sight, so no merge could retire it;
470
- a file whose only placements were under removed directories (an
471
  orphan) could not be seen either.
472
473
Now an import brings a directory back rather than making a second node,
474
and a merge retires a kept node's other placements and every orphan.
475
476
`oxy git verify` built and rendered every file of every commit and read
477
every Git blob; it now loads each commit's history and builds only the
478
files whose graph (by key) or blob it has not already found equal to
479
that Git blob.
480
481
| vscode slice, 1 747 commits | Before | After |
482
|---|---:|---:|
483
| Import | 128.6 s | 132.1 s |
484
| Differences | 51 | none: 3 327 291 files compared |
485
| Verify | 1 369 to 1 444 s, 832 MB | 114.6 s, 441 MB |
486
487
### Git import memory (2026-09-30, backlog H2)
488
489
An import kept up to eight full states (every file's graph) for commits
490
whose children were still to come. It now keeps history-only states and
491
builds the graphs of only the files a commit may change; unchanged ones
492
were already neither read nor rendered. An imported change joins its
493
base under the hash it was stored by, instead of being encoded again.
494
495
| vscode, first 5 000 commits | Import | Peak RSS | Verify |
496
|---|---:|---:|---:|
497
| Before (2026-09-29) | 412 s | 6.3 GB | |
498
| After | 309 s | 2.3 GB | every commit (13.4 million files) in 978 s |
499
500
Verify loaded each commit's history from scratch. It now keeps the
501
state of a commit whose children are still to come, as the import does,
502
and a child adds only the changes its parent lacks. Those states stay
503
history only: graphs are built beside them (`State::build_graphs_detached`)
504
and dropped after rendering, so changes are added without their text and
505
every file keeps its graph key, which the cache of verified files needs.
506
The same 5 000 commits verify in 166.5 s (from 978 s), 1.3 GB.
507
508
### Store size (2026-09-30, backlog P1)
509
510
The Linux tip's store was 448 MB, and 390 MB after `oxy gc`, against
511
311 MB for Git after `git gc`. The file contents cut into the store's
512
chunks compress, at zstd level 3, to 312 MB, level with Git; at level 9
513
to 284 MB (12.9 s on four cores), at level 19 to 270 MB (314 s). The
514
rest was two things:
515
- `oxy gc` left the first pack as written: packs merge only above 16.
516
- A text blob holds its lines as a CBOR array, so every line has a
517
  length header: 62 MB more raw, 35 MB more compressed for the tree's
518
  46 308 files of 4 KiB or more.
519
520
`oxy gc` now repacks everything into one pack at zstd level 9, compressed
521
on every core: 357 MB (349 MB pack) in 52 s. Change format 3 then stores
522
a text blob as the lines' bytes:
523
524
| Linux tip | Store after gc | First commit |
525
|---|---:|---:|
526
| Format 2, gc at level 3 | 390 MB | 41 s |
527
| Format 2, gc repacks at level 9 | 357 MB | |
528
| Format 3 (Earth at load average 11) | 324 MB (objects 306 MB) | 32.6 s, 4.4 GB |
529
| Git | 311 MB (.git) | 33.7 to 37.9 s, 42 MB |
530
531
### fsck (2026-09-30)
532
533
On the Linux tip in change format 3, `oxy fsck` took 24.7 s and 2.8 GB:
534
its references pass read every change whole, the first one's 1.8 GB of
535
text into memory, to check dependencies, authors and blobs. It now reads
536
a change's skeleton and checks its text as the blobs it is stored in (the
537
objects pass has checked every object already): 17.2 s and 984 MB, the
538
references pass 1.7 s. Phases are timed (`OXY_TIMINGS`): objects 5.1 s,
539
references 1.7 s, states and checkpoints 10.4 s.
540