Import from history-only states b493d993a9df
A Git import kept up to eight full states, every file's graph, for commits whose children were still to come. It now starts from history-only states and builds the graphs of only the files a commit may change (the others were already neither read nor rendered), and an imported change joins its base under the hash it was stored by instead of being encoded again. The first 5 000 vscode commits import in 309 s at 2.3 GB (from 412 s and 6.3 GB), every commit verified. refs #35
- Author
- Andy McCall @andymccall <[email protected]>
- When
- · 2026-09-30 01:54 UTC
- Signature
- ✔ signed by Andy McCall
- Hash
b493d993a9df4197d9d4a511ff5f287148b8e00ac08163101cd7daae8e226029- Depends on
- 03571027793e 1681323d1dff 1d360c91a289 30ea9c1e70fa 33e77a5526fa 79ab8528ff62 958fb27dda7c a87c7b9d563a c3069537d7c5 dfb23a3a8cfa
- Depended on by
- 60622d23b9e4 2306a4d24779 ab4d4e31b38d 36695c5e02ef da652d6392d8 57d8dc05c9d4
- In bookmarks
- agents-in-readme bare-hashes carried-not-removed carry-identical doctor-trust identical-sides keyless-tags main merge-remote-bookmark only-help-103 paths-for-resolutions-and-amend push-notes show-stat subrepo-pin-state switch-says
- In tags
- v0.1.0 v0.10.0 v0.11.0 v0.2.0 v0.3.0 v0.3.1 v0.4.0 v0.4.1 v0.4.2 v0.4.3 v0.4.4 v0.4.5 v0.4.6 v0.5.0 v0.5.1 v0.6.0 v0.7.0 v0.8.0 v0.9.0
- Edits
- 7 atoms, 0 tree operations, 3 files
Browse the tree at this change
modified crates/oxy-core/src/repo/git.rs +18 −8
| … | 649 unchanged lines |
||
| 650 | 650 | match found { |
|
| 651 | 651 | Some((st, tree)) => (st, Some(tree)), |
|
| 652 | 652 | None => { |
|
| 653 | - | let mut st = State::new(); |
|
| 653 | + | // History only: graphs are built for the files a |
|
| 654 | + | // commit changes, so memory follows what changes, |
|
| 655 | + | // not the whole tree. |
|
| 656 | + | let mut st = State::new_history_only(); |
|
| 654 | 657 | self.add_git_ancestors(&mut st, git_dir, &[p1.as_str()], &known)?; |
|
| 655 | 658 | (st, None) |
|
| 656 | 659 | } |
|
| 657 | 660 | } |
|
| 658 | 661 | } |
|
| 659 | - | None => (State::new(), None), |
|
| 662 | + | None => (State::new_history_only(), None), |
|
| 660 | 663 | }; |
|
| 661 | 664 | // Nodes the other parents' changes touch: the first parent's |
|
| 662 | 665 | // tree cannot vouch for them. |
|
| … | 28 unchanged lines |
||
| 691 | 694 | } |
|
| 692 | 695 | } |
|
| 693 | 696 | } |
|
| 694 | - | let base_snapshot = if same.is_empty() { |
|
| 695 | - | base.materialize() |
|
| 696 | - | } else { |
|
| 697 | - | base.materialize_except(&same.values().copied().collect()) |
|
| 698 | - | }; |
|
| 697 | + | // Build the graphs of every text file the commit may change. |
|
| 698 | + | let skip: HashSet<NodeId> = same.values().copied().collect(); |
|
| 699 | + | let need: Vec<NodeId> = base |
|
| 700 | + | .tree() |
|
| 701 | + | .view() |
|
| 702 | + | .entries |
|
| 703 | + | .values() |
|
| 704 | + | .filter(|e| e.kind == NodeKind::File && !e.mode.opaque && !skip.contains(&e.node)) |
|
| 705 | + | .map(|e| e.node) |
|
| 706 | + | .collect(); |
|
| 707 | + | self.load_graphs_of(&mut base, &need)?; |
|
| 708 | + | let base_snapshot = base.materialize_except(&skip); |
|
| 699 | 709 | ||
| 700 | 710 | lap(2, &mut clock); |
|
| 701 | 711 | // The commit's tree as in-memory entries. |
|
| … | 144 unchanged lines |
||
| 846 | 856 | heads_now.remove(d); |
|
| 847 | 857 | } |
|
| 848 | 858 | heads_now.insert(hash); |
|
| 849 | - | base.add_unchecked(change); |
|
| 859 | + | base.add_known(hash, change, Vec::new()); |
|
| 850 | 860 | known.imported.insert(gc.sha.clone(), hash); |
|
| 851 | 861 | if waiting.get(gc.sha.as_str()).is_some_and(|n| *n > 0) { |
|
| 852 | 862 | kept.insert(gc.sha.clone(), (base, tree)); |
|
| … | 465 unchanged lines |
modified docs/backlog-overnight-3.md +1 −0
| … | 200 unchanged lines |
||
| 201 | 201 | - The fix: an import revives the directory, and a merge retires a kept node's other placements and every orphan. There is a Git-built test for each cause; each reproduces the vscode symptom without its fix. Spec §23. |
|
| 202 | 202 | - The vscode slice now verifies: all 1 747 commits and 3.3 million files. |
|
| 203 | 203 | - `oxy git verify` is cached by graph key and blob: 115 s instead of 1 400 s. |
|
| 204 | + | - 2026-09-30 **H2, first part**: an import builds graphs only for the files a commit may change, from history-only states. The first 5 000 vscode commits: 309 s and 2.3 GB (from 412 s and 6.3 GB), and all verified. Next: verify and import reuse each commit's history instead of loading it again, before H3 to H5. |
|
| 204 | 205 |
modified docs/benchmarks.md +16 −0
| … | 454 unchanged lines |
||
| 455 | 455 | | Differences | 51 | none: 3 327 291 files compared | |
|
| 456 | 456 | | Verify | 1 369 to 1 444 s, 832 MB | 114.6 s, 441 MB | |
|
| 457 | 457 | ||
| 458 | + | ### Git import memory (2026-09-30, backlog H2) |
|
| 459 | + | ||
| 460 | + | An import kept up to eight full states (every file's graph) for commits |
|
| 461 | + | whose children were still to come. It now keeps history-only states and |
|
| 462 | + | builds the graphs of only the files a commit may change; unchanged ones |
|
| 463 | + | were already neither read nor rendered. An imported change joins its |
|
| 464 | + | base under the hash it was stored by, instead of being encoded again. |
|
| 465 | + | ||
| 466 | + | | vscode, first 5 000 commits | Import | Peak RSS | Verify | |
|
| 467 | + | |---|---:|---:|---:| |
|
| 468 | + | | Before (2026-09-29) | 412 s | 6.3 GB | | |
|
| 469 | + | | After | 309 s | 2.3 GB | every commit (13.4 million files) in 978 s | |
|
| 470 | + | ||
| 471 | + | Verify still loads each commit's history from scratch, which grows with |
|
| 472 | + | the length of the history. |
|
| 473 | + |