Git import costs what each commit changes, not the whole tree issue #25 open andymccall opened this 1 month ago · 5 comments

Full vscode main (166 425 commits) with CH1's batched merges, under MemoryMax=14G on Earth: killed by the cgroup OOM killer after 80 minutes (anon RSS 14.5 GB, 1.1 GB in .oxy at the kill). The 10 000-commit slice imports in 135 s at 1.6 GB peak and verifies; the whole history has a point where memory climbs 9 → 12 GB in three minutes before the kill. The run lost the progress line, so the commit at the spike is unknown. Next: run under a 22 GB cap keeping the full stderr, with the sampler noting the current commit, then profile that commit's slice alone.

Second run under a 20 GB cap with progress lines: 1 000 commits a minute at first, 250 a minute by commit 26 000, RSS sawtooth 5–10 GB; then on commits 28 500–29 000 of vscode main it climbed 12 → 16 → 19 → 20 GB in five minutes and was killed (1.4 GB in .oxy). That range holds four large merges: d381a4b7 (3 198 files, 80 400 insertions), 59745016 (1 205 files), c902f478 (998 files), e3b68b47 (392 files). CH1's batching bounds the graphs built per batch, so the climb is elsewhere: both parents' states and the union render, or what materialize_batched keeps between batches. A reproducer is running: branches at d381a4b7's first parent and at d381a4b7, imported one after the other under the same cap.

Reproducer (Earth, MemoryMax=20G): up to d381a4b7's first parent, 26 557 commits import in 44 min with max RSS 12.8 GB; up to d381a4b7 itself (28 637 commits, the other parent's 2 080 commits and the 3 198-file merge last), 60 min, max RSS 16.9 GB, exit 0, 1.4 GB in .oxy. The merge itself passes; the tall peaks (up to 15.4 GB sampled) are on the second-parent branch's commits just before it, where the whole-history run died with more state loaded. What costs memory is a long-diverged branch being imported and merged, not one commit. Both repositories are kept on Earth (vs-slice-pre-d381.oxy, vs-slice-d381.oxy) to profile the last 500 commits alone.

The commits at the peak (slice order 28 001–28 500): merges 337277fb (5 518 files, 225 159 insertions, 109 396 deletions), 02b22110 (2 970 files), df93fbb6, 0d943cf2, 82b67667 (about 1 069 files each), 6d7644e5 (957), a731f891 (421), and a 798-file non-merge a1b6fed0 — long-lived branches merged back into main. The profiling target is 337277fb.

A data point from 2026-10-26: Andy's tetriblox (Commander X16 game; 312 MB of Git, 539 commits on main with 208 merges, two submodules) imported with oxy 0.7.0 on the workstation in 3.5 s for main (peak RSS 206 MB). Each further branch was brought in with oxy git fetch onto a bookmark at the root change, at about 2 s each: 158 branches in all, the shared history stored once.

Imports are deterministic: the 251 Git commits that two separately imported branches shared got identical change hashes. That is what makes importing branch by branch work, but two separate oxy git imports still make different repository ids, so they cannot be pushed to one hub repository. There, a fetch onto a root bookmark does the job; a first-class oxy git import --all-branches would be the tidy form.

A repository this size is far from the cost issue #25 and issue #96 are about: VS Code's long-lived merges are where it shows.