OXYGENHUB

Render faster in deep histories: memoised dependency answers, read locks, merges on every core aedfe465b611

FileGraph::materialize asks depends_on for each deleted atom next to a
live one; in a deep history each answer walked thousands of changes,
and the same few pairs (one change deleting a block another inserted)
were asked again for every line. A per-render memo answers each pair
once. State's level and closure memos were Mutexes taken on every step
of every walk, so rendering threads queued in futex calls (a quarter of
the time in perf); they are RwLocks read without excluding each other,
and the walk's seen-set uses the fast hasher. A partial snapshot renders
its files on every core (materialize_paths), as the whole tree does.

VS Code, 500 commits from 66,541, alternating builds: 734 s to 261 s
(materialize 358 s to 66 s); 1,300 commits from 31,000: render base
271 s to 60 s. The same bookmark state each time (#17dc23d2, #1e8ecf7c).

refs #96
Author
Andy McCall @andymccall <[email protected]>
When
· 2026-10-10 11:22 UTC
Signature
✔ signed by Andy McCall
Hash
aedfe465b611278f7536c1db29e753f7e9489ce8e691eb41e6bbdd49885fc604
Depends on
119bbf50cb86 170bb436e1d5 2306a4d24779 8fd78eb52ad2 9d333fc9ca1b cb8b8082fa73 d45dab0484ea dbf06d342e1d f6451fbbae3f ff0af1ee299d
Depended on by
4476cdbb0443 d6e0d5b8b339
In bookmarks
bare-hashes carried-not-removed carry-identical doctor-trust identical-sides keyless-tags main merge-remote-bookmark only-help-103 paths-for-resolutions-and-amend push-notes show-stat subrepo-pin-state switch-says
In tags
v0.10.0 v0.11.0 v0.4.3 v0.4.4 v0.4.5 v0.4.6 v0.5.0 v0.5.1 v0.6.0 v0.7.0 v0.8.0 v0.9.0
Edits
17 atoms, 0 tree operations, 3 files

Browse the tree at this change

modified CHANGELOG.md +2 −0
…
1 unchanged lines
2 2
3 3
## Unreleased
4 4
5 +
- **Git imports of long histories much faster (#96).** Rendering a file asks whether the changes that deleted and inserted neighbouring lines depend on each other; in a deep history each answer was a walk through thousands of changes, asked again for every line, under a lock every rendering thread queued on. Answers are now remembered for the file being rendered, the shared memos are read without excluding other threads, and a merge renders its files on every core. On VS Code past 66 000 commits, 500 commits imported in 261 s rather than 734 s, with the same history.
6 +
5 7
- **A long push goes in parts (#95).** `oxy push` sent a whole history in one request, held in memory on both sides; the full VS Code import would have passed a hub's 4 GiB limit. It now sends parts of about `push.batch-bytes` (default 256 MiB) in dependency order, objects alone, and the last part moves the bookmark; the spinner says which part is going. A server stores a part and reports it as `objects`.
6 8
7 9
- **A state index read only as far as it is used (#94).** Each file's history in the workspace's state index is stored after the rest with a hash of its own, and is read and checked only when a command builds that file, so a diff or a one-file cat no longer reads and hashes the histories of every file (67 MB of an 84 MB index on VS Code at 20 000 commits). Changes added on top of an index no longer decode the histories they touch. A history found damaged is an error that says so, and the index is removed and rebuilt by the next command. Under load on that repository, `diff` went from 85 ms to 63 ms and `cat`/`blame` from 102 ms to 80 ms.
…
158 unchanged lines
modified crates/oxy-core/src/graph.rs +24 −0
…
24 unchanged lines
25 25
    fn depends_on(&self, a: &ChangeHash, b: &ChangeHash) -> bool;
26 26
}
27 27
28 +
/// Another oracle's answers, each asked once: rendering one file asks about
29 +
/// the same few pairs over and over (a change deleting a block of lines
30 +
/// another inserted), and in a long history each answer is a walk through
31 +
/// thousands of changes (oxygen#96).
32 +
struct MemoDeps<'a> {
33 +
    inner: &'a dyn DepOracle,
34 +
    known: std::cell::RefCell<crate::fasthash::FastMap<(ChangeHash, ChangeHash), bool>>,
35 +
}
36 +
37 +
impl DepOracle for MemoDeps<'_> {
38 +
    fn depends_on(&self, a: &ChangeHash, b: &ChangeHash) -> bool {
39 +
        if let Some(v) = self.known.borrow().get(&(*a, *b)) {
40 +
            return *v;
41 +
        }
42 +
        let v = self.inner.depends_on(a, b);
43 +
        self.known.borrow_mut().insert((*a, *b), v);
44 +
        v
45 +
    }
46 +
}
47 +
28 48
/// No change depends on any other: every pair commutes.
29 49
pub struct NoDeps;
30 50
impl DepOracle for NoDeps {
…
503 unchanged lines
534 554
535 555
    /// Order the live atoms and locate conflicts.
536 556
    pub fn materialize(&self, deps: &dyn DepOracle) -> Materialized {
557 +
        let deps = &MemoDeps {
558 +
            inner: deps,
559 +
            known: Default::default(),
560 +
        };
537 561
        let lines = |order: Vec<AtomKey>| Materialized {
538 562
            segments: if order.is_empty() {
539 563
                vec![]
…
1272 unchanged lines
modified crates/oxy-core/src/state.rs +29 −16
…
10 unchanged lines
11 11
};
12 12
use crate::tree::{TreeGraph, TreeView};
13 13
use std::collections::{BTreeMap, BTreeSet, HashMap, HashSet};
14 -
use std::sync::{Arc, Mutex};
14 +
use std::sync::Arc;
15 15
16 16
/// Checkpoint payload version 2 (spec §11): file graphs by reference.
17 17
pub const CHECKPOINT_INDEX_VERSION: u64 = 2;
…
84 unchanged lines
102 102
    /// every change asked about grew with the square of history: #25). A
103 103
    /// mutex, not a `RefCell`, so a loaded state can be shared across
104 104
    /// threads by a host that caches it.
105 -
    closure: Mutex<ClosureMemo>,
105 +
    closure: std::sync::RwLock<ClosureMemo>,
106 106
    /// Each change's level: 0 with no dependencies, else one more than its
107 107
    /// highest dependency. A change can only depend on changes of lower
108 108
    /// level, which bounds the walk [`DepOracle::depends_on`] does.
109 -
    levels: Mutex<HashMap<ChangeHash, u32>>,
109 +
    levels: std::sync::RwLock<crate::fasthash::FastMap<ChangeHash, u32>>,
110 110
    /// `None` for a full state. For a history-only state (loaded for its
111 111
    /// changes and tree), the file graphs not built yet, by node: `files`
112 112
    /// holds only those loaded with [`State::load_graphs`].
…
575 unchanged lines
688 688
            files: self.files.clone(),
689 689
            replaces: self.replaces.clone(),
690 690
            // A cache: the clone builds its own.
691 -
            closure: Mutex::new(ClosureMemo::default()),
692 -
            levels: Mutex::new(HashMap::new()),
691 +
            closure: Default::default(),
692 +
            levels: Default::default(),
693 693
            deferred: self.deferred.clone(),
694 694
        }
695 695
    }
…
681 unchanged lines
1377 1377
        self.apply_with(h, change, texts);
1378 1378
        self.changes.insert(h, Arc::new(meta));
1379 1379
        {
1380 -
            let mut m = self.closure.lock().unwrap();
1380 +
            let mut m = self.closure.write().unwrap();
1381 1381
            m.sets.clear();
1382 1382
            m.held = 0;
1383 1383
        }
…
382 unchanged lines
1766 1766
1767 1767
    /// Transitive dependencies of `h`, excluding `h` itself.
1768 1768
    pub fn closure(&self, h: &ChangeHash) -> Arc<HashSet<ChangeHash>> {
1769 -
        if let Some(c) = self.closure.lock().unwrap().sets.get(h) {
1769 +
        if let Some(c) = self.closure.read().unwrap().sets.get(h) {
1770 1770
            return c.clone();
1771 1771
        }
1772 1772
        let mut out = HashSet::new();
…
10 unchanged lines
1783 1783
            }
1784 1784
        }
1785 1785
        let rc = Arc::new(out);
1786 -
        let mut memo = self.closure.lock().unwrap();
1786 +
        let mut memo = self.closure.write().unwrap();
1787 1787
        if memo.held + rc.len() > CLOSURE_MEMO_HASHES {
1788 1788
            memo.sets.clear();
1789 1789
            memo.held = 0;
…
7 unchanged lines
1797 1797
    /// yet known, without recursion: histories are deeper than a stack. An
1798 1798
    /// unknown change has level 0.
1799 1799
    fn level(&self, h: &ChangeHash) -> u32 {
1800 -
        let mut levels = self.levels.lock().unwrap();
1800 +
        // Nearly always known: read without excluding the other threads
1801 +
        // rendering beside this one (oxygen#96).
1802 +
        if let Some(l) = self.levels.read().unwrap().get(h) {
1803 +
            return *l;
1804 +
        }
1805 +
        let mut levels = self.levels.write().unwrap();
1801 1806
        if let Some(l) = levels.get(h) {
1802 1807
            return *l;
1803 1808
        }
…
287 unchanged lines
2091 2096
        view: crate::tree::TreeView,
2092 2097
        paths: impl IntoIterator<Item = &'p str>,
2093 2098
    ) -> Snapshot {
2094 -
        let files = paths
2099 +
        use rayon::prelude::*;
2100 +
        // On every core: a merge may render thousands (oxygen#96).
2101 +
        let entries: Vec<(&str, &crate::tree::TreeEntry)> = paths
2102 +
            .into_iter()
2103 +
            .filter_map(|p| Some((p, view.entries.get(p)?)))
2104 +
            .collect();
2105 +
        let files = entries
2106 +
            .par_iter()
2107 +
            .map(|(p, e)| (p.to_string(), self.file_state(e)))
2108 +
            .collect::<Vec<_>>()
2095 2109
            .into_iter()
2096 -
            .filter_map(|p| Some((p.to_string(), self.file_state(view.entries.get(p)?))))
2097 2110
            .collect();
2098 2111
        Snapshot {
2099 2112
            tree: view,
…
45 unchanged lines
2145 2158
        if a == b {
2146 2159
            return true;
2147 2160
        }
2148 -
        if let Some(c) = self.closure.lock().unwrap().sets.get(a) {
2161 +
        if let Some(c) = self.closure.read().unwrap().sets.get(a) {
2149 2162
            return c.contains(b);
2150 2163
        }
2151 2164
        if !self.changes.contains_key(b) {
…
3 unchanged lines
2155 2168
        if self.level(a) <= lb {
2156 2169
            return false;
2157 2170
        }
2158 -
        let mut seen: HashSet<ChangeHash> = HashSet::new();
2171 +
        let mut seen: crate::fasthash::FastSet<ChangeHash> = Default::default();
2159 2172
        let mut stack: Vec<ChangeHash> = self
2160 2173
            .changes
2161 2174
            .get(a)
…
720 unchanged lines
2882 2895
            let unknown = ObjectRef([0xee; 32]);
2883 2896
            for a in &all {
2884 2897
                let want = st.closure(a).clone();
2885 -
                st.closure.lock().unwrap().sets.clear();
2886 -
                st.closure.lock().unwrap().held = 0;
2898 +
                st.closure.write().unwrap().sets.clear();
2899 +
                st.closure.write().unwrap().held = 0;
2887 2900
                for b in all.iter().chain([&unknown]) {
2888 2901
                    assert_eq!(
2889 2902
                        st.depends_on(a, b),
…
2 unchanged lines
2892 2905
                    );
2893 2906
                }
2894 2907
            }
2895 -
            assert!(st.closure.lock().unwrap().held <= CLOSURE_MEMO_HASHES);
2908 +
            assert!(st.closure.write().unwrap().held <= CLOSURE_MEMO_HASHES);
2896 2909
        }
2897 2910
    }
2898 2911
…
30 unchanged lines