OXYGENHUB

Log H12a and the out-of-memory incident e811b0813473

Author
Andy McCall @andymccall <[email protected]>
When
· 2026-09-29 11:36 UTC
Signature
✔ signed by Andy McCall
Hash
e811b0813473317bc4ae495202a7c4eda96063de6a78537ca4455c09b8f82eb7
Depends on
75036530b621 abe7bec1c2ba fd3f920f0b24
Depended on by
8eb39658a940
In bookmarks
agents-in-readme bare-hashes carried-not-removed carry-identical doctor-trust identical-sides keyless-tags main merge-remote-bookmark only-help-103 paths-for-resolutions-and-amend push-notes show-stat subrepo-pin-state switch-says
In tags
v0.1.0 v0.10.0 v0.11.0 v0.2.0 v0.3.0 v0.3.1 v0.4.0 v0.4.1 v0.4.2 v0.4.3 v0.4.4 v0.4.5 v0.4.6 v0.5.0 v0.5.1 v0.6.0 v0.7.0 v0.8.0 v0.9.0
Edits
3 atoms, 0 tree operations, 1 file

Browse the tree at this change

hide whitespace changes
modified docs/backlog-overnight-2.md +10 −1
… 
116 unchanged lines
117 117 
- [x] P4b **Trust dashboard** (item 4): per release and asset, a graph of authors' identities and signatures, the runner and its pinned environment, inputs (state, scope hash), and receipts, with a one-click audit view and export.
118 118 
- [x] H11 Run pipeline without a push, made easy (Andy): a primary button on the Flow tab, reasons shown when it is missing, and Run pipeline beside each bookmark and on change pages (runs the change's own state); `POST /flow/runs` takes `{"change"}`.
119 119 
- [ ] H12 **Runner management** (Andy; oxygenhub#41), in parts:
120 - 
  - H12a Runners in the database: registration from the UI (`flow runner register`) with the signing key bound at registration, replacing hub.toml tokens. A Runners page (status, key, architecture, tiers, labels, capacity, current jobs) with each runner's job history, and drain, disable and remove.
120 + 
  - [x] H12a Runners in the database: registration from the UI (`flow runner register`) with the signing key bound at registration, replacing hub.toml tokens. A Runners page (status, key, architecture, tiers, labels, capacity, current jobs) with each runner's job history, and drain, disable and remove.
121 121 
  - H12b Scopes: site, organisation or repository runners. Claims only return in-scope jobs, and repositories and organisations get Settings → Runners.
122 122 
  - H12c Labels and architecture: `runs-on` in pipelines (oxygenflow), with runners reporting labels and architecture in claims.
123 123 
  - H12d Queue view (why each job waits) and the runners API.
… 
643 unchanged lines
767 767 
  Two gaps were left and are now fixed (oxygen run green):
768 768 
  - **oxygen#26:** durable writes and renames name the file and the step in their errors.
769 769 
  - **oxygen#27:** the guide shows the develop/feature flow the oxy way.
770 + 
- 2026-09-29 **H12a done** (oxygenflow#12, oxygenhub#45). Runners are now in the database.
771 + 
  - **Registration:** one-time tokens on Admin → Runners; `flow runner register` binds the runner's key and writes `runner.toml`.
772 + 
  - **Old configuration:** hub.toml runners are tracked, and their bound keys moved to `flow_runners`.
773 + 
  - **Claims** record the runner and its description.
774 + 
  - **Pages:** the Runners page has drain, disable, resume and remove, and each runner's page shows its history.
775 + 
  - **Tests and deploy:** a lifecycle test (register, reuse refused, claim, history, drain, disable, remove). Deployed. The hub and flow runs were still pending at the time of writing. Earth's runner was not redeployed (it still sends no `info`, which is fine).
776 + 
- 2026-09-29 **Out-of-memory incident on Sarturn at 12:28.** The full vscode import (14.2 GB RSS at about 3 850 s) plus a hub gate build exhausted Andy's workstation. The kernel killed the import, and VS Code and Chrome crashed. The 16 GB watchdog did not allow for concurrent builds and the desktop.
777 + 
  - **Next:** fix the importer's memory growth first (candidates: at most 8 kept state clones, the `new_changes` vector holding every change, and full file graphs for every file). Measure on slices on **Earth only**, inside a hard cgroup memory limit (for example `systemd-run --user --scope -p MemoryMax=…`) that leaves room for the hub and the runner.
778 + 
  - **Saturn:** no heavy jobs there. The partial import was removed; `~/scratch-vscode/vscode.git` (1.4 GB) remains and can be deleted once Earth has its own clone.