Render faster in deep histories: memoised dependency answers, read locks, merges on every core aedfe465b611
FileGraph::materialize asks depends_on for each deleted atom next to a live one; in a deep history each answer walked thousands of changes, and the same few pairs (one change deleting a block another inserted) were asked again for every line. A per-render memo answers each pair once. State's level and closure memos were Mutexes taken on every step of every walk, so rendering threads queued in futex calls (a quarter of the time in perf); they are RwLocks read without excluding each other, and the walk's seen-set uses the fast hasher. A partial snapshot renders its files on every core (materialize_paths), as the whole tree does. VS Code, 500 commits from 66,541, alternating builds: 734 s to 261 s (materialize 358 s to 66 s); 1,300 commits from 31,000: render base 271 s to 60 s. The same bookmark state each time (#17dc23d2, #1e8ecf7c). refs #96
- Author
- Andy McCall @andymccall <[email protected]>
- When
- · 2026-10-10 11:22 UTC
- Signature
- ✔ signed by Andy McCall
- Hash
aedfe465b611278f7536c1db29e753f7e9489ce8e691eb41e6bbdd49885fc604- Depends on
- 119bbf50cb86 170bb436e1d5 2306a4d24779 8fd78eb52ad2 9d333fc9ca1b cb8b8082fa73 d45dab0484ea dbf06d342e1d f6451fbbae3f ff0af1ee299d
- Depended on by
- 4476cdbb0443 d6e0d5b8b339
- In bookmarks
- bare-hashes carried-not-removed carry-identical doctor-trust identical-sides keyless-tags main merge-remote-bookmark only-help-103 paths-for-resolutions-and-amend push-notes show-stat subrepo-pin-state switch-says
- In tags
- v0.10.0 v0.11.0 v0.4.3 v0.4.4 v0.4.5 v0.4.6 v0.5.0 v0.5.1 v0.6.0 v0.7.0 v0.8.0 v0.9.0
- Edits
- 17 atoms, 0 tree operations, 3 files
Browse the tree at this change
modified CHANGELOG.md +2 −0
| … | 1 unchanged lines |
||
| 2 | 2 | ||
| 3 | 3 | ## Unreleased |
|
| 4 | 4 | ||
| 5 | + | - **Git imports of long histories much faster (#96).** Rendering a file asks whether the changes that deleted and inserted neighbouring lines depend on each other; in a deep history each answer was a walk through thousands of changes, asked again for every line, under a lock every rendering thread queued on. Answers are now remembered for the file being rendered, the shared memos are read without excluding other threads, and a merge renders its files on every core. On VS Code past 66 000 commits, 500 commits imported in 261 s rather than 734 s, with the same history. |
|
| 6 | + | ||
| 5 | 7 | - **A long push goes in parts (#95).** `oxy push` sent a whole history in one request, held in memory on both sides; the full VS Code import would have passed a hub's 4 GiB limit. It now sends parts of about `push.batch-bytes` (default 256 MiB) in dependency order, objects alone, and the last part moves the bookmark; the spinner says which part is going. A server stores a part and reports it as `objects`. |
|
| 6 | 8 | ||
| 7 | 9 | - **A state index read only as far as it is used (#94).** Each file's history in the workspace's state index is stored after the rest with a hash of its own, and is read and checked only when a command builds that file, so a diff or a one-file cat no longer reads and hashes the histories of every file (67 MB of an 84 MB index on VS Code at 20 000 commits). Changes added on top of an index no longer decode the histories they touch. A history found damaged is an error that says so, and the index is removed and rebuilt by the next command. Under load on that repository, `diff` went from 85 ms to 63 ms and `cat`/`blame` from 102 ms to 80 ms. |
|
| … | 158 unchanged lines |
modified crates/oxy-core/src/graph.rs +24 −0
| … | 24 unchanged lines |
||
| 25 | 25 | fn depends_on(&self, a: &ChangeHash, b: &ChangeHash) -> bool; |
|
| 26 | 26 | } |
|
| 27 | 27 | ||
| 28 | + | /// Another oracle's answers, each asked once: rendering one file asks about |
|
| 29 | + | /// the same few pairs over and over (a change deleting a block of lines |
|
| 30 | + | /// another inserted), and in a long history each answer is a walk through |
|
| 31 | + | /// thousands of changes (oxygen#96). |
|
| 32 | + | struct MemoDeps<'a> { |
|
| 33 | + | inner: &'a dyn DepOracle, |
|
| 34 | + | known: std::cell::RefCell<crate::fasthash::FastMap<(ChangeHash, ChangeHash), bool>>, |
|
| 35 | + | } |
|
| 36 | + | ||
| 37 | + | impl DepOracle for MemoDeps<'_> { |
|
| 38 | + | fn depends_on(&self, a: &ChangeHash, b: &ChangeHash) -> bool { |
|
| 39 | + | if let Some(v) = self.known.borrow().get(&(*a, *b)) { |
|
| 40 | + | return *v; |
|
| 41 | + | } |
|
| 42 | + | let v = self.inner.depends_on(a, b); |
|
| 43 | + | self.known.borrow_mut().insert((*a, *b), v); |
|
| 44 | + | v |
|
| 45 | + | } |
|
| 46 | + | } |
|
| 47 | + | ||
| 28 | 48 | /// No change depends on any other: every pair commutes. |
|
| 29 | 49 | pub struct NoDeps; |
|
| 30 | 50 | impl DepOracle for NoDeps { |
|
| … | 503 unchanged lines |
||
| 534 | 554 | ||
| 535 | 555 | /// Order the live atoms and locate conflicts. |
|
| 536 | 556 | pub fn materialize(&self, deps: &dyn DepOracle) -> Materialized { |
|
| 557 | + | let deps = &MemoDeps { |
|
| 558 | + | inner: deps, |
|
| 559 | + | known: Default::default(), |
|
| 560 | + | }; |
|
| 537 | 561 | let lines = |order: Vec<AtomKey>| Materialized { |
|
| 538 | 562 | segments: if order.is_empty() { |
|
| 539 | 563 | vec![] |
|
| … | 1272 unchanged lines |
modified crates/oxy-core/src/state.rs +29 −16
| … | 10 unchanged lines |
||
| 11 | 11 | }; |
|
| 12 | 12 | use crate::tree::{TreeGraph, TreeView}; |
|
| 13 | 13 | use std::collections::{BTreeMap, BTreeSet, HashMap, HashSet}; |
|
| 14 | - | use std::sync::{Arc, Mutex}; |
|
| 14 | + | use std::sync::Arc; |
|
| 15 | 15 | ||
| 16 | 16 | /// Checkpoint payload version 2 (spec §11): file graphs by reference. |
|
| 17 | 17 | pub const CHECKPOINT_INDEX_VERSION: u64 = 2; |
|
| … | 84 unchanged lines |
||
| 102 | 102 | /// every change asked about grew with the square of history: #25). A |
|
| 103 | 103 | /// mutex, not a `RefCell`, so a loaded state can be shared across |
|
| 104 | 104 | /// threads by a host that caches it. |
|
| 105 | - | closure: Mutex<ClosureMemo>, |
|
| 105 | + | closure: std::sync::RwLock<ClosureMemo>, |
|
| 106 | 106 | /// Each change's level: 0 with no dependencies, else one more than its |
|
| 107 | 107 | /// highest dependency. A change can only depend on changes of lower |
|
| 108 | 108 | /// level, which bounds the walk [`DepOracle::depends_on`] does. |
|
| 109 | - | levels: Mutex<HashMap<ChangeHash, u32>>, |
|
| 109 | + | levels: std::sync::RwLock<crate::fasthash::FastMap<ChangeHash, u32>>, |
|
| 110 | 110 | /// `None` for a full state. For a history-only state (loaded for its |
|
| 111 | 111 | /// changes and tree), the file graphs not built yet, by node: `files` |
|
| 112 | 112 | /// holds only those loaded with [`State::load_graphs`]. |
|
| … | 575 unchanged lines |
||
| 688 | 688 | files: self.files.clone(), |
|
| 689 | 689 | replaces: self.replaces.clone(), |
|
| 690 | 690 | // A cache: the clone builds its own. |
|
| 691 | - | closure: Mutex::new(ClosureMemo::default()), |
|
| 692 | - | levels: Mutex::new(HashMap::new()), |
|
| 691 | + | closure: Default::default(), |
|
| 692 | + | levels: Default::default(), |
|
| 693 | 693 | deferred: self.deferred.clone(), |
|
| 694 | 694 | } |
|
| 695 | 695 | } |
|
| … | 681 unchanged lines |
||
| 1377 | 1377 | self.apply_with(h, change, texts); |
|
| 1378 | 1378 | self.changes.insert(h, Arc::new(meta)); |
|
| 1379 | 1379 | { |
|
| 1380 | - | let mut m = self.closure.lock().unwrap(); |
|
| 1380 | + | let mut m = self.closure.write().unwrap(); |
|
| 1381 | 1381 | m.sets.clear(); |
|
| 1382 | 1382 | m.held = 0; |
|
| 1383 | 1383 | } |
|
| … | 382 unchanged lines |
||
| 1766 | 1766 | ||
| 1767 | 1767 | /// Transitive dependencies of `h`, excluding `h` itself. |
|
| 1768 | 1768 | pub fn closure(&self, h: &ChangeHash) -> Arc<HashSet<ChangeHash>> { |
|
| 1769 | - | if let Some(c) = self.closure.lock().unwrap().sets.get(h) { |
|
| 1769 | + | if let Some(c) = self.closure.read().unwrap().sets.get(h) { |
|
| 1770 | 1770 | return c.clone(); |
|
| 1771 | 1771 | } |
|
| 1772 | 1772 | let mut out = HashSet::new(); |
|
| … | 10 unchanged lines |
||
| 1783 | 1783 | } |
|
| 1784 | 1784 | } |
|
| 1785 | 1785 | let rc = Arc::new(out); |
|
| 1786 | - | let mut memo = self.closure.lock().unwrap(); |
|
| 1786 | + | let mut memo = self.closure.write().unwrap(); |
|
| 1787 | 1787 | if memo.held + rc.len() > CLOSURE_MEMO_HASHES { |
|
| 1788 | 1788 | memo.sets.clear(); |
|
| 1789 | 1789 | memo.held = 0; |
|
| … | 7 unchanged lines |
||
| 1797 | 1797 | /// yet known, without recursion: histories are deeper than a stack. An |
|
| 1798 | 1798 | /// unknown change has level 0. |
|
| 1799 | 1799 | fn level(&self, h: &ChangeHash) -> u32 { |
|
| 1800 | - | let mut levels = self.levels.lock().unwrap(); |
|
| 1800 | + | // Nearly always known: read without excluding the other threads |
|
| 1801 | + | // rendering beside this one (oxygen#96). |
|
| 1802 | + | if let Some(l) = self.levels.read().unwrap().get(h) { |
|
| 1803 | + | return *l; |
|
| 1804 | + | } |
|
| 1805 | + | let mut levels = self.levels.write().unwrap(); |
|
| 1801 | 1806 | if let Some(l) = levels.get(h) { |
|
| 1802 | 1807 | return *l; |
|
| 1803 | 1808 | } |
|
| … | 287 unchanged lines |
||
| 2091 | 2096 | view: crate::tree::TreeView, |
|
| 2092 | 2097 | paths: impl IntoIterator<Item = &'p str>, |
|
| 2093 | 2098 | ) -> Snapshot { |
|
| 2094 | - | let files = paths |
|
| 2099 | + | use rayon::prelude::*; |
|
| 2100 | + | // On every core: a merge may render thousands (oxygen#96). |
|
| 2101 | + | let entries: Vec<(&str, &crate::tree::TreeEntry)> = paths |
|
| 2102 | + | .into_iter() |
|
| 2103 | + | .filter_map(|p| Some((p, view.entries.get(p)?))) |
|
| 2104 | + | .collect(); |
|
| 2105 | + | let files = entries |
|
| 2106 | + | .par_iter() |
|
| 2107 | + | .map(|(p, e)| (p.to_string(), self.file_state(e))) |
|
| 2108 | + | .collect::<Vec<_>>() |
|
| 2095 | 2109 | .into_iter() |
|
| 2096 | - | .filter_map(|p| Some((p.to_string(), self.file_state(view.entries.get(p)?)))) |
|
| 2097 | 2110 | .collect(); |
|
| 2098 | 2111 | Snapshot { |
|
| 2099 | 2112 | tree: view, |
|
| … | 45 unchanged lines |
||
| 2145 | 2158 | if a == b { |
|
| 2146 | 2159 | return true; |
|
| 2147 | 2160 | } |
|
| 2148 | - | if let Some(c) = self.closure.lock().unwrap().sets.get(a) { |
|
| 2161 | + | if let Some(c) = self.closure.read().unwrap().sets.get(a) { |
|
| 2149 | 2162 | return c.contains(b); |
|
| 2150 | 2163 | } |
|
| 2151 | 2164 | if !self.changes.contains_key(b) { |
|
| … | 3 unchanged lines |
||
| 2155 | 2168 | if self.level(a) <= lb { |
|
| 2156 | 2169 | return false; |
|
| 2157 | 2170 | } |
|
| 2158 | - | let mut seen: HashSet<ChangeHash> = HashSet::new(); |
|
| 2171 | + | let mut seen: crate::fasthash::FastSet<ChangeHash> = Default::default(); |
|
| 2159 | 2172 | let mut stack: Vec<ChangeHash> = self |
|
| 2160 | 2173 | .changes |
|
| 2161 | 2174 | .get(a) |
|
| … | 720 unchanged lines |
||
| 2882 | 2895 | let unknown = ObjectRef([0xee; 32]); |
|
| 2883 | 2896 | for a in &all { |
|
| 2884 | 2897 | let want = st.closure(a).clone(); |
|
| 2885 | - | st.closure.lock().unwrap().sets.clear(); |
|
| 2886 | - | st.closure.lock().unwrap().held = 0; |
|
| 2898 | + | st.closure.write().unwrap().sets.clear(); |
|
| 2899 | + | st.closure.write().unwrap().held = 0; |
|
| 2887 | 2900 | for b in all.iter().chain([&unknown]) { |
|
| 2888 | 2901 | assert_eq!( |
|
| 2889 | 2902 | st.depends_on(a, b), |
|
| … | 2 unchanged lines |
||
| 2892 | 2905 | ); |
|
| 2893 | 2906 | } |
|
| 2894 | 2907 | } |
|
| 2895 | - | assert!(st.closure.lock().unwrap().held <= CLOSURE_MEMO_HASHES); |
|
| 2908 | + | assert!(st.closure.write().unwrap().held <= CLOSURE_MEMO_HASHES); |
|
| 2896 | 2909 | } |
|
| 2897 | 2910 | } |
|
| 2898 | 2911 | ||
| … | 30 unchanged lines |