A big tree renders many times faster, and is not rendered twice (refs #137) 4476cdbb0443
State::depends_on compared a change with the one sought only when it popped it, so a depth-first walk went through thousands of changes before finding a direct dependency, which is nearly always the answer a render wants (spec 8.1 makes a deletion depend on the inserter of the line after it): the VS Code tree's checkout asked 262,678 questions, 4.4 billion steps, 838 CPU-seconds; now dependencies are looked for as they are found, 5 CPU-seconds, same answers. subrepos() materialized the whole state (every file rendered) after each clone, pull, switch and workspace add; it now reads the tree and the pins. A new workspace starts on nothing, as its directory is, so the checkout's snapshot no longer finds the tree deleted and renders it. oxy workspace add on that clone: 176 s and 11 GB to 7.5 s and 1.8 GB (8 cores); test for the snapshot. OXY_TIMINGS reports the graph ordering totals after a checkout's render and the blob reads.
- Author
- Andy McCall @andymccall <[email protected]>
- via
- agent
agent:claude@andymccall <[email protected]> · keyless, under your grant for changes,refs,seal · key 1db0accbd516
- When
- · 2026-10-25 09:25 UTC
- Signature
- ◈ sealed by agent:claude@andymccall
- Hash
4476cdbb0443b2f9c638c85a4786b4da025a512072a22be3486aed4035d297ec- Depends on
- 0389c3e82f6c 4dee277c327a 4e88e774777a 570e35310f31 75167672ebcd 8fd78eb52ad2 982864068a59 aedfe465b611 afaba5f1306d d3cd7ef95da6 d45dab0484ea e4595038502a e7ac6c2e84e4 ff0af1ee299d
- Depended on by
- 157d02e95586
- In bookmarks
- identical-sides main push-notes subrepo-pin-state
- In tags
- v0.10.0 v0.11.0 v0.7.0 v0.8.0 v0.9.0
- Edits
- 19 atoms, 0 tree operations, 7 files
Browse the tree at this change
modified CHANGELOG.md +1 −0
| … | 1 unchanged lines |
||
| 2 | 2 | ||
| 3 | 3 | ## Unreleased |
|
| 4 | 4 | ||
| 5 | + | - **Rendering a big tree is many times faster, and clone, pull, switch and `oxy workspace add` stop rendering it twice (#137).** Measured on the VS Code history's depth-0 clone (19,545 files, 166,425 changes), `oxy workspace add` took 467 s on Earth and 176 s on an 8-core workstation, 11 GB at its peak; it now takes 7.5 s and 1.8 GB on the workstation. Three causes: (1) whether one change depends on another, which rendering asks about every deleted line, was answered by a depth-first walk that compared a change with the one sought only on coming back up to it, so it went through thousands of changes first although the answer is nearly always a direct dependency (spec §8.1): 262,678 questions took 4.4 billion steps and 838 CPU-seconds, and now take 5 CPU-seconds, with the same answers; (2) finding the subrepos rendered every file of the tree, after every clone, pull, switch and `workspace add`, where reading the tree and the subrepos' pins is enough; (3) a new workspace started on the target state with an empty directory, so the checkout's first snapshot found the whole tree deleted and rendered it to say so. `OXY_TIMINGS=1` now also says how a checkout's files were ordered (`graph: single order`…) and how its blobs were read. |
|
| 5 | 6 | - **`oxy serve` takes a push of more than 2 MB (#138).** It set no request limit of its own, so axum's default of 2 MB applied, and a larger push part was cut off with `Broken pipe` at the client and nothing in the server's log. It now takes up to 4 GiB a request (`oxy_server::BODY_LIMIT`), OxygenHub's default `push_body_limit`; OxygenHub was not affected. |
|
| 6 | 7 | - **A push no longer sends a replaced or removed binary file's old content again (#136).** A bundle carried both the old and the new blob of every binary file a change replaces, so a change removing ~1,100 tracked cache files (oxygenhub-android's set-up) pushed all of them again, over 100 MB, though the hub stored each when it was added. The old blob is always the new blob of a change the replacement depends on, which the receiver already holds or which the same bundle carries, so pushes and fetch answers now carry only the new one. |
|
| 7 | 8 | ||
| … | 264 unchanged lines |
modified crates/oxy-core/src/repo/checkout.rs +3 −0
| … | 407 unchanged lines |
||
| 408 | 408 | } |
|
| 409 | 409 | ||
| 410 | 410 | drop(_render); |
|
| 411 | + | // How the files rendered: which way each graph was ordered, summed |
|
| 412 | + | // over every core (`OXY_TIMINGS`). |
|
| 413 | + | crate::timing::report_totals(); |
|
| 411 | 414 | let recording = crate::timing::phase("checkout: record"); |
|
| 412 | 415 | let (op_ref, op) = self.head_op()?; |
|
| 413 | 416 | let wc = self.new_working_copy(heads.clone(), ""); |
|
| … | 228 unchanged lines |
modified crates/oxy-core/src/repo/mod.rs +14 −1
| … | 1677 unchanged lines |
||
| 1678 | 1678 | /// through [`Repo::read_blob`], which may fetch it lazily. |
|
| 1679 | 1679 | pub fn read_blobs(&self, refs: &[ObjectRef]) -> Result<Vec<Vec<u8>>> { |
|
| 1680 | 1680 | let mut blobs: Vec<Option<Blob>> = Vec::with_capacity(refs.len()); |
|
| 1681 | - | for (r, got) in refs.iter().zip(self.store.get_many(refs)) { |
|
| 1681 | + | let t = crate::timing::phase("blobs: get_many blobs"); |
|
| 1682 | + | let got_all = self.store.get_many(refs); |
|
| 1683 | + | drop(t); |
|
| 1684 | + | for (r, got) in refs.iter().zip(got_all) { |
|
| 1682 | 1685 | blobs.push(match got { |
|
| 1683 | 1686 | Ok((ObjectType::Blob, body)) => Some(Blob::decode(&body)?), |
|
| 1684 | 1687 | Ok((ty, _)) => return fail(format!("{} is a {ty:?}, not a blob", r.short(12))), |
|
| … | 6 unchanged lines |
||
| 1691 | 1694 | .flatten() |
|
| 1692 | 1695 | .flat_map(|b| b.chunks.iter().copied()) |
|
| 1693 | 1696 | .collect(); |
|
| 1697 | + | let t = crate::timing::phase("blobs: get_many chunks"); |
|
| 1694 | 1698 | let mut chunks = self.store.get_many(&chunk_refs).into_iter(); |
|
| 1699 | + | drop(t); |
|
| 1700 | + | if crate::timing::enabled() { |
|
| 1701 | + | eprintln!( |
|
| 1702 | + | "timing blobs: {} blobs, {} chunks, {} one by one", |
|
| 1703 | + | refs.len(), |
|
| 1704 | + | chunk_refs.len(), |
|
| 1705 | + | blobs.iter().filter(|b| b.is_none()).count() |
|
| 1706 | + | ); |
|
| 1707 | + | } |
|
| 1695 | 1708 | let mut out = Vec::with_capacity(refs.len()); |
|
| 1696 | 1709 | for (r, blob) in refs.iter().zip(blobs) { |
|
| 1697 | 1710 | match blob { |
|
| … | 1081 unchanged lines |
modified crates/oxy-core/src/repo/subrepos.rs +11 −7
| … | 246 unchanged lines |
||
| 247 | 247 | /// Every subrepo of the working copy's base state. |
|
| 248 | 248 | pub fn subrepos(&self) -> Result<Vec<SubrepoInfo>> { |
|
| 249 | 249 | let (_, wc) = self.working_copy()?; |
|
| 250 | - | let snap = self.load_state(&wc.deps)?.materialize(); |
|
| 250 | + | // The tree and the subrepos' pins only: a materialized state |
|
| 251 | + | // rendered every file of the tree to find them, after each clone, |
|
| 252 | + | // pull, switch and `workspace add` (in the VS Code history, ~70 s |
|
| 253 | + | // and 11 GB; oxygen#137). |
|
| 254 | + | let mut state = self.load_state_indexed(&wc.deps)?; |
|
| 255 | + | let tree = self.tree_view(&mut state)?; |
|
| 251 | 256 | let modules = self.modules_on_disk(); |
|
| 252 | 257 | let mut out = Vec::new(); |
|
| 253 | - | for (path, f) in &snap.files { |
|
| 254 | - | if f.kind != NodeKind::Subrepo { |
|
| 258 | + | for (path, e) in tree.entries.iter() { |
|
| 259 | + | if e.kind != NodeKind::Subrepo { |
|
| 255 | 260 | continue; |
|
| 256 | 261 | } |
|
| 257 | - | let FileContent::Opaque { blobs } = &f.content else { |
|
| 262 | + | let Some(b) = state.live_blobs(&e.node).first().copied() else { |
|
| 258 | 263 | continue; |
|
| 259 | 264 | }; |
|
| 260 | - | let Some(b) = blobs.first() else { continue }; |
|
| 261 | - | let pin = Pin::decode(&self.read_blob(b)?)?; |
|
| 265 | + | let pin = Pin::decode(&self.read_blob(&b)?)?; |
|
| 262 | 266 | let dir = self.root.join(path); |
|
| 263 | 267 | let present = dir.join(OXY_DIR).exists() || dir.join(".git").exists(); |
|
| 264 | 268 | let actual = if present { |
|
| … | 20 unchanged lines |
||
| 285 | 289 | } |
|
| 286 | 290 | // Registered but not committed yet: pinned at what they are now. |
|
| 287 | 291 | for (path, m) in &modules.entries { |
|
| 288 | - | if snap.files.contains_key(path) { |
|
| 292 | + | if tree.entries.contains_key(path) { |
|
| 289 | 293 | continue; |
|
| 290 | 294 | } |
|
| 291 | 295 | let dir = self.root.join(path); |
|
| … | 144 unchanged lines |
modified crates/oxy-core/src/repo/tests.rs +6 −0
| … | 3285 unchanged lines |
||
| 3286 | 3286 | ||
| 3287 | 3287 | // A second workspace elsewhere, checked out on the bookmark. |
|
| 3288 | 3288 | let ws_dir = root.join("ws2"); |
|
| 3289 | + | let ops = repo.op_log().unwrap().len(); |
|
| 3289 | 3290 | let ws = repo.workspace_add(&ws_dir, None).unwrap(); |
|
| 3291 | + | assert_eq!( |
|
| 3292 | + | ws.op_log().unwrap().len(), |
|
| 3293 | + | ops + 2, |
|
| 3294 | + | "the add and its checkout, and no snapshot finding the tree deleted (oxygen#137)" |
|
| 3295 | + | ); |
|
| 3290 | 3296 | assert!(ws_dir.join(".oxy").is_file()); |
|
| 3291 | 3297 | assert_eq!(fs::read(ws_dir.join("a.txt")).unwrap(), b"a\n"); |
|
| 3292 | 3298 | assert_ne!(ws.workspace(), repo.workspace()); |
|
| … | 6806 unchanged lines |
modified crates/oxy-core/src/repo/workspace.rs +5 −1
| … | 209 unchanged lines |
||
| 210 | 210 | None => self.bookmark_heads()?, |
|
| 211 | 211 | }; |
|
| 212 | 212 | let id = WorkspaceId::random(); |
|
| 213 | - | let wc = self.new_working_copy(heads.clone(), ""); |
|
| 213 | + | // On nothing, as the directory is, until the checkout: on `heads`, |
|
| 214 | + | // the checkout's first snapshot found every file of the tree |
|
| 215 | + | // deleted, rendering the whole tree to say so (195 s of a 467 s |
|
| 216 | + | // `oxy workspace add` in the VS Code history, oxygen#137). |
|
| 217 | + | let wc = self.new_working_copy(Vec::new(), ""); |
|
| 214 | 218 | let wc_hash = self.store.put_change(&wc)?; |
|
| 215 | 219 | let (op_ref, op) = self.head_op()?; |
|
| 216 | 220 | let mut view = op.view.clone(); |
|
| … | 141 unchanged lines |
modified crates/oxy-core/src/state.rs +12 −0
| … | 2177 unchanged lines |
||
| 2178 | 2178 | return false; |
|
| 2179 | 2179 | } |
|
| 2180 | 2180 | let mut seen: crate::fasthash::FastSet<ChangeHash> = Default::default(); |
|
| 2181 | + | // A dependency is looked for as it is found, before the walk goes |
|
| 2182 | + | // below it: most questions are about a direct one (spec §8.1 makes |
|
| 2183 | + | // a deletion depend on the inserter of the line after it, which is |
|
| 2184 | + | // what a render asks about), and a depth-first walk that only |
|
| 2185 | + | // looked when it came back up went through thousands of changes |
|
| 2186 | + | // first (oxygen#137: 4.4 billion steps rendering the VS Code tree). |
|
| 2181 | 2187 | let mut stack: Vec<ChangeHash> = self |
|
| 2182 | 2188 | .changes |
|
| 2183 | 2189 | .get(a) |
|
| 2184 | 2190 | .map(|m| m.deps.clone()) |
|
| 2185 | 2191 | .unwrap_or_default(); |
|
| 2192 | + | if stack.contains(b) { |
|
| 2193 | + | return true; |
|
| 2194 | + | } |
|
| 2186 | 2195 | while let Some(c) = stack.pop() { |
|
| 2187 | 2196 | if c == *b { |
|
| 2188 | 2197 | return true; |
|
| … | 2 unchanged lines |
||
| 2191 | 2200 | continue; |
|
| 2192 | 2201 | } |
|
| 2193 | 2202 | if let Some(m) = self.changes.get(&c) { |
|
| 2203 | + | if m.deps.contains(b) { |
|
| 2204 | + | return true; |
|
| 2205 | + | } |
|
| 2194 | 2206 | stack.extend(m.deps.iter().copied()); |
|
| 2195 | 2207 | } |
|
| 2196 | 2208 | } |
|
| … | 754 unchanged lines |