Large inserts as blobs: changes that scale to a whole tree issue #34 closed: completed andymccall opened this · 1 comment
referenced by 3 changes on 29 Sep 2026
andymccall
Done in c8a26be3: change format 2 stores inserts of 4 KiB or more as text blobs (spec Draft 0.8q). The hub (db750ff1) and runner (6cbadaca) were deployed with it first. Linux tip first commit: 55.3 s, 9.9 GB, 442 MB store; before, it failed and left a repository that would not open. The memory half moves to issue #35: a clean status is killed at 14 GB.
andymccall closed ·
Found by the Linux stage-1 benchmark (see issue #33). Every inserted line lives inline in its change, so a first commit of a large tree is one enormous object (1.8 GB for Linux) and the commit peaked at 10.9 GB of memory against Git's 42 MB. Proposal: an Insert whose lines exceed a threshold (say 64 KiB) carries a blob reference (chunked, content addressed, deduplicated) instead of its lines, as Replace already does for opaque files; transport, gc and fsck follow the blob as they do for Replace. With it, the builder can store each file's content as it reads it and drop it, bounding memory. A change-format change (spec §6.1, §7), allowed before 1.0. Then re-run the Linux benchmark.