OXYGENHUB

A big tree renders many times faster, and is not rendered twice (refs #137) 4476cdbb0443

State::depends_on compared a change with the one sought only when it
popped it, so a depth-first walk went through thousands of changes
before finding a direct dependency, which is nearly always the answer a
render wants (spec 8.1 makes a deletion depend on the inserter of the
line after it): the VS Code tree's checkout asked 262,678 questions,
4.4 billion steps, 838 CPU-seconds; now dependencies are looked for as
they are found, 5 CPU-seconds, same answers. subrepos() materialized
the whole state (every file rendered) after each clone, pull, switch
and workspace add; it now reads the tree and the pins. A new workspace
starts on nothing, as its directory is, so the checkout's snapshot no
longer finds the tree deleted and renders it. oxy workspace add on that
clone: 176 s and 11 GB to 7.5 s and 1.8 GB (8 cores); test for the
snapshot. OXY_TIMINGS reports the graph ordering totals after a
checkout's render and the blob reads.
Author
Andy McCall @andymccall <[email protected]>
via
agent agent:claude@andymccall <[email protected]> · keyless, under your grant for changes,refs,seal · key 1db0accbd516
When
· 2026-10-25 09:25 UTC
Signature
◈ sealed by agent:claude@andymccall
Hash
4476cdbb0443b2f9c638c85a4786b4da025a512072a22be3486aed4035d297ec
Depends on
0389c3e82f6c 4dee277c327a 4e88e774777a 570e35310f31 75167672ebcd 8fd78eb52ad2 982864068a59 aedfe465b611 afaba5f1306d d3cd7ef95da6 d45dab0484ea e4595038502a e7ac6c2e84e4 ff0af1ee299d
Depended on by
157d02e95586
In bookmarks
identical-sides main push-notes subrepo-pin-state
In tags
v0.10.0 v0.11.0 v0.7.0 v0.8.0 v0.9.0
Edits
19 atoms, 0 tree operations, 7 files

Browse the tree at this change

modified CHANGELOG.md +1 −0
…
1 unchanged lines
2 2
3 3
## Unreleased
4 4
5 +
- **Rendering a big tree is many times faster, and clone, pull, switch and `oxy workspace add` stop rendering it twice (#137).** Measured on the VS Code history's depth-0 clone (19,545 files, 166,425 changes), `oxy workspace add` took 467 s on Earth and 176 s on an 8-core workstation, 11 GB at its peak; it now takes 7.5 s and 1.8 GB on the workstation. Three causes: (1) whether one change depends on another, which rendering asks about every deleted line, was answered by a depth-first walk that compared a change with the one sought only on coming back up to it, so it went through thousands of changes first although the answer is nearly always a direct dependency (spec §8.1): 262,678 questions took 4.4 billion steps and 838 CPU-seconds, and now take 5 CPU-seconds, with the same answers; (2) finding the subrepos rendered every file of the tree, after every clone, pull, switch and `workspace add`, where reading the tree and the subrepos' pins is enough; (3) a new workspace started on the target state with an empty directory, so the checkout's first snapshot found the whole tree deleted and rendered it to say so. `OXY_TIMINGS=1` now also says how a checkout's files were ordered (`graph: single order`…) and how its blobs were read.
5 6
- **`oxy serve` takes a push of more than 2 MB (#138).** It set no request limit of its own, so axum's default of 2 MB applied, and a larger push part was cut off with `Broken pipe` at the client and nothing in the server's log. It now takes up to 4 GiB a request (`oxy_server::BODY_LIMIT`), OxygenHub's default `push_body_limit`; OxygenHub was not affected.
6 7
- **A push no longer sends a replaced or removed binary file's old content again (#136).** A bundle carried both the old and the new blob of every binary file a change replaces, so a change removing ~1,100 tracked cache files (oxygenhub-android's set-up) pushed all of them again, over 100 MB, though the hub stored each when it was added. The old blob is always the new blob of a change the replacement depends on, which the receiver already holds or which the same bundle carries, so pushes and fetch answers now carry only the new one.
7 8
…
264 unchanged lines
modified crates/oxy-core/src/repo/checkout.rs +3 −0
…
407 unchanged lines
408 408
        }
409 409
410 410
        drop(_render);
411 +
        // How the files rendered: which way each graph was ordered, summed
412 +
        // over every core (`OXY_TIMINGS`).
413 +
        crate::timing::report_totals();
411 414
        let recording = crate::timing::phase("checkout: record");
412 415
        let (op_ref, op) = self.head_op()?;
413 416
        let wc = self.new_working_copy(heads.clone(), "");
…
228 unchanged lines
modified crates/oxy-core/src/repo/mod.rs +14 −1
…
1677 unchanged lines
1678 1678
    /// through [`Repo::read_blob`], which may fetch it lazily.
1679 1679
    pub fn read_blobs(&self, refs: &[ObjectRef]) -> Result<Vec<Vec<u8>>> {
1680 1680
        let mut blobs: Vec<Option<Blob>> = Vec::with_capacity(refs.len());
1681 -
        for (r, got) in refs.iter().zip(self.store.get_many(refs)) {
1681 +
        let t = crate::timing::phase("blobs: get_many blobs");
1682 +
        let got_all = self.store.get_many(refs);
1683 +
        drop(t);
1684 +
        for (r, got) in refs.iter().zip(got_all) {
1682 1685
            blobs.push(match got {
1683 1686
                Ok((ObjectType::Blob, body)) => Some(Blob::decode(&body)?),
1684 1687
                Ok((ty, _)) => return fail(format!("{} is a {ty:?}, not a blob", r.short(12))),
…
6 unchanged lines
1691 1694
            .flatten()
1692 1695
            .flat_map(|b| b.chunks.iter().copied())
1693 1696
            .collect();
1697 +
        let t = crate::timing::phase("blobs: get_many chunks");
1694 1698
        let mut chunks = self.store.get_many(&chunk_refs).into_iter();
1699 +
        drop(t);
1700 +
        if crate::timing::enabled() {
1701 +
            eprintln!(
1702 +
                "timing blobs: {} blobs, {} chunks, {} one by one",
1703 +
                refs.len(),
1704 +
                chunk_refs.len(),
1705 +
                blobs.iter().filter(|b| b.is_none()).count()
1706 +
            );
1707 +
        }
1695 1708
        let mut out = Vec::with_capacity(refs.len());
1696 1709
        for (r, blob) in refs.iter().zip(blobs) {
1697 1710
            match blob {
…
1081 unchanged lines
modified crates/oxy-core/src/repo/subrepos.rs +11 −7
…
246 unchanged lines
247 247
    /// Every subrepo of the working copy's base state.
248 248
    pub fn subrepos(&self) -> Result<Vec<SubrepoInfo>> {
249 249
        let (_, wc) = self.working_copy()?;
250 -
        let snap = self.load_state(&wc.deps)?.materialize();
250 +
        // The tree and the subrepos' pins only: a materialized state
251 +
        // rendered every file of the tree to find them, after each clone,
252 +
        // pull, switch and `workspace add` (in the VS Code history, ~70 s
253 +
        // and 11 GB; oxygen#137).
254 +
        let mut state = self.load_state_indexed(&wc.deps)?;
255 +
        let tree = self.tree_view(&mut state)?;
251 256
        let modules = self.modules_on_disk();
252 257
        let mut out = Vec::new();
253 -
        for (path, f) in &snap.files {
254 -
            if f.kind != NodeKind::Subrepo {
258 +
        for (path, e) in tree.entries.iter() {
259 +
            if e.kind != NodeKind::Subrepo {
255 260
                continue;
256 261
            }
257 -
            let FileContent::Opaque { blobs } = &f.content else {
262 +
            let Some(b) = state.live_blobs(&e.node).first().copied() else {
258 263
                continue;
259 264
            };
260 -
            let Some(b) = blobs.first() else { continue };
261 -
            let pin = Pin::decode(&self.read_blob(b)?)?;
265 +
            let pin = Pin::decode(&self.read_blob(&b)?)?;
262 266
            let dir = self.root.join(path);
263 267
            let present = dir.join(OXY_DIR).exists() || dir.join(".git").exists();
264 268
            let actual = if present {
…
20 unchanged lines
285 289
        }
286 290
        // Registered but not committed yet: pinned at what they are now.
287 291
        for (path, m) in &modules.entries {
288 -
            if snap.files.contains_key(path) {
292 +
            if tree.entries.contains_key(path) {
289 293
                continue;
290 294
            }
291 295
            let dir = self.root.join(path);
…
144 unchanged lines
modified crates/oxy-core/src/repo/tests.rs +6 −0
…
3285 unchanged lines
3286 3286
3287 3287
    // A second workspace elsewhere, checked out on the bookmark.
3288 3288
    let ws_dir = root.join("ws2");
3289 +
    let ops = repo.op_log().unwrap().len();
3289 3290
    let ws = repo.workspace_add(&ws_dir, None).unwrap();
3291 +
    assert_eq!(
3292 +
        ws.op_log().unwrap().len(),
3293 +
        ops + 2,
3294 +
        "the add and its checkout, and no snapshot finding the tree deleted (oxygen#137)"
3295 +
    );
3290 3296
    assert!(ws_dir.join(".oxy").is_file());
3291 3297
    assert_eq!(fs::read(ws_dir.join("a.txt")).unwrap(), b"a\n");
3292 3298
    assert_ne!(ws.workspace(), repo.workspace());
…
6806 unchanged lines
modified crates/oxy-core/src/repo/workspace.rs +5 −1
…
209 unchanged lines
210 210
            None => self.bookmark_heads()?,
211 211
        };
212 212
        let id = WorkspaceId::random();
213 -
        let wc = self.new_working_copy(heads.clone(), "");
213 +
        // On nothing, as the directory is, until the checkout: on `heads`,
214 +
        // the checkout's first snapshot found every file of the tree
215 +
        // deleted, rendering the whole tree to say so (195 s of a 467 s
216 +
        // `oxy workspace add` in the VS Code history, oxygen#137).
217 +
        let wc = self.new_working_copy(Vec::new(), "");
214 218
        let wc_hash = self.store.put_change(&wc)?;
215 219
        let (op_ref, op) = self.head_op()?;
216 220
        let mut view = op.view.clone();
…
141 unchanged lines
modified crates/oxy-core/src/state.rs +12 −0
…
2177 unchanged lines
2178 2178
            return false;
2179 2179
        }
2180 2180
        let mut seen: crate::fasthash::FastSet<ChangeHash> = Default::default();
2181 +
        // A dependency is looked for as it is found, before the walk goes
2182 +
        // below it: most questions are about a direct one (spec §8.1 makes
2183 +
        // a deletion depend on the inserter of the line after it, which is
2184 +
        // what a render asks about), and a depth-first walk that only
2185 +
        // looked when it came back up went through thousands of changes
2186 +
        // first (oxygen#137: 4.4 billion steps rendering the VS Code tree).
2181 2187
        let mut stack: Vec<ChangeHash> = self
2182 2188
            .changes
2183 2189
            .get(a)
2184 2190
            .map(|m| m.deps.clone())
2185 2191
            .unwrap_or_default();
2192 +
        if stack.contains(b) {
2193 +
            return true;
2194 +
        }
2186 2195
        while let Some(c) = stack.pop() {
2187 2196
            if c == *b {
2188 2197
                return true;
…
2 unchanged lines
2191 2200
                continue;
2192 2201
            }
2193 2202
            if let Some(m) = self.changes.get(&c) {
2203 +
                if m.deps.contains(b) {
2204 +
                    return true;
2205 +
                }
2194 2206
                stack.extend(m.deps.iter().copied());
2195 2207
            }
2196 2208
        }
…
754 unchanged lines