OXYGENHUB

Log importer memory, determinism and the metalgear extension fixes 8eb39658a940

Author
Andy McCall @andymccall <[email protected]>
When
· 2026-09-29 13:30 UTC
Signature
✔ signed by Andy McCall
Hash
8eb39658a94040047549d5519af299ea7385dd31b63de827f95a5827d5fc7030
Depends on
75036530b621 e811b0813473
Depended on by
a5d2fbe7c10d
In bookmarks
agents-in-readme bare-hashes carried-not-removed carry-identical doctor-trust identical-sides keyless-tags main merge-remote-bookmark only-help-103 paths-for-resolutions-and-amend push-notes show-stat subrepo-pin-state switch-says
In tags
v0.1.0 v0.10.0 v0.11.0 v0.2.0 v0.3.0 v0.3.1 v0.4.0 v0.4.1 v0.4.2 v0.4.3 v0.4.4 v0.4.5 v0.4.6 v0.5.0 v0.5.1 v0.6.0 v0.7.0 v0.8.0 v0.9.0
Edits
1 atom, 0 tree operations, 1 file

Browse the tree at this change

hide whitespace changes
modified docs/backlog-overnight-2.md +12 −0
… 
775 unchanged lines
776 776 
- 2026-09-29 **Out-of-memory incident on Sarturn at 12:28.** The full vscode import (14.2 GB RSS at about 3 850 s) plus a hub gate build exhausted Andy's workstation. The kernel killed the import, and VS Code and Chrome crashed. The 16 GB watchdog did not allow for concurrent builds and the desktop.
777 777 
  - **Next:** fix the importer's memory growth first (candidates: at most 8 kept state clones, the `new_changes` vector holding every change, and full file graphs for every file). Measure on slices on **Earth only**, inside a hard cgroup memory limit (for example `systemd-run --user --scope -p MemoryMax=…`) that leaves room for the hub and the runner.
778 778 
  - **Saturn:** no heavy jobs there. The partial import was removed; `~/scratch-vscode/vscode.git` (1.4 GB) remains and can be deleted once Earth has its own clone.
779 + 
- 2026-09-29 **P5c progress: importer memory and determinism** (oxygen#28). All measurements were on Earth under `systemd-run --user --scope -p MemoryMax=8G`, in `~/bench`, with its own vscode clone (166 425 commits).
780 + 
  - **Where the memory went:** deep clones of the kept states, not the closure memo (a few thousand entries). With 8 kept states, 1 000 commits took 3.0 GB; with 1, 1.2 GB but slower.
781 + 
  - **The fix:** a state's file graphs are `Arc`-shared and copied on edit, and heads are tracked incrementally. 1 000 commits: 0.8 GB, 91 s. 5 000: 2.25 GB, 410 s.
782 + 
  - **Determinism:** random NodeIds made imports non-deterministic, and not only in their hashes: the same binary gave 370 and 366 heads. New NodeIds now derive from the commit and path (spec §23, Draft 0.8p), and two imports of 5 000 commits give identical heads.
783 + 
  - **Next:**
784 + 
    - `oxy git verify` of the 5 000 slice (running on Earth);
785 + 
    - memory still grows with history (tombstoned atoms in every graph), so try dropping the graphs of long-untouched files and rebuilding them on demand from the store, as history-only states already do;
786 + 
    - then the full import on Earth.
787 + 
- 2026-09-29 **Andy's metalgear VS Code problems, fixed.**
788 + 
  - **Flashing Source Control:** a refresh loop through the subrepo's `.oxy`. Status no longer rewrites an unchanged manifest (oxygen#28 commit), and the extension ignores every `.oxy` directory (oxygen-vscode#7, 0.13.0).
789 + 
  - **"OxygenHub: fetch failed":** plain fetch against Earth's self-signed certificate. The OxygenHub extension now uses the remote's CA file, or the `oxygenhub.caFiles` setting (oxygenhub-vscode#1, 0.7.0), and `oxy remote list --json` reports the CA.
790 + 
  - Both extensions are installed in both profiles.