Render a file that is one chain of lines without contracting it 598009a75f6c
Cloning the Linux tip spent most of its checkout rendering files: working out each file's line order contracted the graph with three sets allocated per line, 93 GB in all, though every file of a fresh tree is a single chain. A graph that is one chain from start to end with nothing deleted is now read off in order (an edge to End besides the next line orders nothing); any other graph takes the general algorithm, which also skips contraction where a line's successors are all kept. A test checks the shortcut and the general order agree, and that the shortcut stands aside for inserts between lines, conflicts and deletions. A bundle now carries each change's hash when its maker knows it (the store it came from, or the bytes received; never the wire), so an import hashes each change once, by storing it, and checks it is the change named. A local clone of the Linux tip takes 61 s and 4.1 GB (from 82 to 96 s and 5.2 GB). refs #35
- Author
- Andy McCall @andymccall <[email protected]>
- When
- · 2026-09-29 23:51 UTC
- Signature
- ✔ signed by Andy McCall
- Hash
598009a75f6c7aeb8572794c40ec8516b239010aba62035293c8d7147a5f6ca4- Depends on
- 0bdc3f91a39f 192fad1e46a2 1d360c91a289 2863456a0f6d 28fc23d7706a 5b9c6ad45f6e 6f8aeb996558 750251b21a96 75167672ebcd 982864068a59 c5ce0367eddf ff0af1ee299d
- Depended on by
- 091fef80755d 7eac74abc586 40a40bb7b798 dbf06d342e1d e83354cdac22 0f93f49ce059 a87c7b9d563a 2c75672fb2fe 519c7c061bcb
- In bookmarks
- agents-in-readme bare-hashes carried-not-removed carry-identical doctor-trust identical-sides keyless-tags main merge-remote-bookmark only-help-103 paths-for-resolutions-and-amend push-notes show-stat subrepo-pin-state switch-says
- In tags
- v0.1.0 v0.10.0 v0.11.0 v0.2.0 v0.3.0 v0.3.1 v0.4.0 v0.4.1 v0.4.2 v0.4.3 v0.4.4 v0.4.5 v0.4.6 v0.5.0 v0.5.1 v0.6.0 v0.7.0 v0.8.0 v0.9.0
- Edits
- 20 atoms, 0 tree operations, 5 files
Browse the tree at this change
modified crates/oxy-core/src/graph.rs +110 −5
| … | 322 unchanged lines |
||
| 323 | 323 | kept: &HashSet<Vertex>, |
|
| 324 | 324 | memo: &mut HashMap<Vertex, BTreeSet<Vertex>>, |
|
| 325 | 325 | ) -> BTreeSet<Vertex> { |
|
| 326 | + | let direct = self.succ.get(&v).map(Edges::as_slice).unwrap_or_default(); |
|
| 327 | + | // The common case, a successor that is itself kept: nothing to |
|
| 328 | + | // contract. |
|
| 329 | + | if direct.iter().all(|s| kept.contains(s)) { |
|
| 330 | + | return direct.iter().copied().collect(); |
|
| 331 | + | } |
|
| 326 | 332 | let mut out = BTreeSet::new(); |
|
| 327 | - | let mut stack: Vec<Vertex> = self |
|
| 328 | - | .succ |
|
| 329 | - | .get(&v) |
|
| 330 | - | .map(|s| s.iter().copied().collect()) |
|
| 331 | - | .unwrap_or_default(); |
|
| 333 | + | let mut stack: Vec<Vertex> = direct.to_vec(); |
|
| 332 | 334 | let mut seen = HashSet::default(); |
|
| 333 | 335 | while let Some(s) = stack.pop() { |
|
| 334 | 336 | if kept.contains(&s) { |
|
| … | 7 unchanged lines |
||
| 342 | 344 | } |
|
| 343 | 345 | } |
|
| 344 | 346 | out |
|
| 347 | + | } |
|
| 348 | + | ||
| 349 | + | /// The atoms in order when the graph is one chain from Start to End with |
|
| 350 | + | /// nothing deleted: a file only ever inserted into in sequence, as every |
|
| 351 | + | /// file of a fresh commit or a clone is. Such a graph has no conflict. |
|
| 352 | + | /// An edge to End besides the chain's next atom (Start's own, or an |
|
| 353 | + | /// append's anchor) orders nothing, since End is last anyway. |
|
| 354 | + | fn as_chain(&self) -> Option<Vec<AtomKey>> { |
|
| 355 | + | if self |
|
| 356 | + | .atoms |
|
| 357 | + | .values() |
|
| 358 | + | .any(|a| a.content.is_none() || !a.deleted_by.is_empty()) |
|
| 359 | + | { |
|
| 360 | + | return None; |
|
| 361 | + | } |
|
| 362 | + | let mut order = Vec::with_capacity(self.atoms.len()); |
|
| 363 | + | let mut v = Vertex::Start; |
|
| 364 | + | loop { |
|
| 365 | + | let succ = self.succ.get(&v)?.as_slice(); |
|
| 366 | + | let mut atoms = succ.iter().filter(|s| **s != Vertex::End); |
|
| 367 | + | match (atoms.next(), atoms.next()) { |
|
| 368 | + | (None, _) if succ.contains(&Vertex::End) => break, |
|
| 369 | + | (Some(Vertex::Atom(k)), None) if order.len() < self.atoms.len() => { |
|
| 370 | + | order.push(*k); |
|
| 371 | + | v = Vertex::Atom(*k); |
|
| 372 | + | } |
|
| 373 | + | _ => return None, |
|
| 374 | + | } |
|
| 375 | + | } |
|
| 376 | + | (order.len() == self.atoms.len()).then_some(order) |
|
| 345 | 377 | } |
|
| 346 | 378 | ||
| 347 | 379 | /// Order the live atoms and locate conflicts. |
|
| 348 | 380 | pub fn materialize(&self, deps: &dyn DepOracle) -> Materialized { |
|
| 381 | + | if let Some(order) = self.as_chain() { |
|
| 382 | + | return Materialized { |
|
| 383 | + | segments: if order.is_empty() { |
|
| 384 | + | vec![] |
|
| 385 | + | } else { |
|
| 386 | + | vec![Segment::Lines(order)] |
|
| 387 | + | }, |
|
| 388 | + | }; |
|
| 389 | + | } |
|
| 390 | + | self.materialize_general(deps) |
|
| 391 | + | } |
|
| 392 | + | ||
| 393 | + | /// [`FileGraph::materialize`] for any graph: contraction, then a |
|
| 394 | + | /// topological order with conflict regions. |
|
| 395 | + | fn materialize_general(&self, deps: &dyn DepOracle) -> Materialized { |
|
| 349 | 396 | // 1. Vertices that survive contraction: live atoms, ghosts, Start, End. |
|
| 350 | 397 | let mut kept: HashSet<Vertex> = HashSet::default(); |
|
| 351 | 398 | let mut ghosts: HashSet<AtomKey> = HashSet::default(); |
|
| … | 387 unchanged lines |
||
| 739 | 786 | } |
|
| 740 | 787 | false |
|
| 741 | 788 | } |
|
| 789 | + | } |
|
| 790 | + | ||
| 791 | + | /// The chain shortcut agrees with the general algorithm wherever it |
|
| 792 | + | /// applies, and stands aside wherever it does not. |
|
| 793 | + | #[test] |
|
| 794 | + | fn the_chain_shortcut_agrees_with_the_general_order() { |
|
| 795 | + | let mut g = FileGraph::new(); |
|
| 796 | + | g.insert( |
|
| 797 | + | h(1), |
|
| 798 | + | 0, |
|
| 799 | + | Vertex::Start, |
|
| 800 | + | Vertex::End, |
|
| 801 | + | &lines(&["a\n", "b\n", "c\n"]), |
|
| 802 | + | ); |
|
| 803 | + | assert!(g.as_chain().is_some()); |
|
| 804 | + | assert_eq!(g.materialize(&NoDeps), g.materialize_general(&NoDeps)); |
|
| 805 | + | // Appended after the last line: still a chain. |
|
| 806 | + | g.insert( |
|
| 807 | + | h(2), |
|
| 808 | + | 0, |
|
| 809 | + | Vertex::Atom((h(1), 2)), |
|
| 810 | + | Vertex::End, |
|
| 811 | + | &lines(&["d\n"]), |
|
| 812 | + | ); |
|
| 813 | + | assert!(g.as_chain().is_some()); |
|
| 814 | + | assert_eq!(g.materialize(&NoDeps), g.materialize_general(&NoDeps)); |
|
| 815 | + | // Inserted between two lines: not a chain (a has two successors). |
|
| 816 | + | g.insert( |
|
| 817 | + | h(4), |
|
| 818 | + | 0, |
|
| 819 | + | Vertex::Atom((h(1), 0)), |
|
| 820 | + | Vertex::Atom((h(1), 1)), |
|
| 821 | + | &lines(&["x\n"]), |
|
| 822 | + | ); |
|
| 823 | + | assert!(g.as_chain().is_none()); |
|
| 824 | + | assert_eq!(text(&g, &g.materialize(&NoDeps)), "a\nx\nb\nc\nd\n"); |
|
| 825 | + | // Two inserts at one anchor: a conflict, never a chain. |
|
| 826 | + | let mut c = FileGraph::new(); |
|
| 827 | + | c.insert(h(1), 0, Vertex::Start, Vertex::End, &lines(&["a\n"])); |
|
| 828 | + | c.insert(h(2), 0, Vertex::Start, Vertex::End, &lines(&["b\n"])); |
|
| 829 | + | assert!(c.as_chain().is_none()); |
|
| 830 | + | assert!(c.materialize(&NoDeps).is_conflicted()); |
|
| 831 | + | // A deletion: not a chain. |
|
| 832 | + | let mut d = FileGraph::new(); |
|
| 833 | + | d.insert(h(1), 0, Vertex::Start, Vertex::End, &lines(&["a\n", "b\n"])); |
|
| 834 | + | d.delete( |
|
| 835 | + | h(3), |
|
| 836 | + | &AtomRange { |
|
| 837 | + | change: h(1), |
|
| 838 | + | from: 0, |
|
| 839 | + | to: 1, |
|
| 840 | + | }, |
|
| 841 | + | ); |
|
| 842 | + | assert!(d.as_chain().is_none()); |
|
| 843 | + | assert_eq!(d.materialize(&NoDeps), d.materialize_general(&NoDeps)); |
|
| 844 | + | // Empty. |
|
| 845 | + | let e = FileGraph::new(); |
|
| 846 | + | assert_eq!(e.materialize(&NoDeps), e.materialize_general(&NoDeps)); |
|
| 742 | 847 | } |
|
| 743 | 848 | ||
| 744 | 849 | #[test] |
|
| … | 421 unchanged lines |
modified crates/oxy-core/src/protocol.rs +14 −1
| … | 85 unchanged lines |
||
| 86 | 86 | /// Policies between the receiver's and the one sent with the push, |
|
| 87 | 87 | /// ascending by serial (spec §14.5). |
|
| 88 | 88 | pub policies: Vec<SignedPolicy>, |
|
| 89 | + | /// Each change's hash, in the order of `changes`, when whoever made |
|
| 90 | + | /// the bundle knows it: the store it came from, or the bytes it was |
|
| 91 | + | /// decoded from. Never read from the wire. Empty when unknown. |
|
| 92 | + | pub hashes: Vec<ChangeHash>, |
|
| 89 | 93 | } |
|
| 90 | 94 | ||
| 91 | 95 | impl Bundle { |
|
| … | 518 unchanged lines |
||
| 610 | 614 | if pair.len() != 2 { |
|
| 611 | 615 | return Err(schema("change entry: expected [change, sigs]")); |
|
| 612 | 616 | } |
|
| 613 | - | let change = Change::decode_with(o::bytes(&pair[0], "change")?, &mut |text, count| { |
|
| 617 | + | let encoded = o::bytes(&pair[0], "change")?; |
|
| 618 | + | b.hashes |
|
| 619 | + | .push(crate::hash::hash_object(ObjectType::Change, encoded)); |
|
| 620 | + | let change = Change::decode_with(encoded, &mut |text, count| { |
|
| 614 | 621 | let by_ref = by_ref.get_or_insert_with(|| { |
|
| 615 | 622 | b.objects |
|
| 616 | 623 | .iter() |
|
| … | 349 unchanged lines |
||
| 966 | 973 | policy: policy.clone(), |
|
| 967 | 974 | sigs: vec![psig.clone()], |
|
| 968 | 975 | }], |
|
| 976 | + | hashes: vec![], |
|
| 969 | 977 | }; |
|
| 970 | 978 | let d = wire::decode_bundle(&wire::encode_bundle(&bundle)).unwrap(); |
|
| 979 | + | assert_eq!( |
|
| 980 | + | d.hashes, |
|
| 981 | + | vec![c.hash()], |
|
| 982 | + | "the receiver hashes what it was sent" |
|
| 983 | + | ); |
|
| 971 | 984 | assert_eq!(d.changes, bundle.changes); |
|
| 972 | 985 | assert_eq!(d.attests, bundle.attests); |
|
| 973 | 986 | assert_eq!(d.objects, bundle.objects); |
|
| … | 128 unchanged lines |
modified crates/oxy-core/src/repo/checkout.rs +15 −8
| … | 233 unchanged lines |
||
| 234 | 234 | for batch in batches { |
|
| 235 | 235 | use rayon::prelude::*; |
|
| 236 | 236 | let nodes: Vec<NodeId> = batch.iter().map(|(_, n, _)| *n).collect(); |
|
| 237 | - | self.load_graphs_of(&mut target, &nodes)?; |
|
| 238 | - | let rendered: Vec<(Vec<u8>, bool)> = nodes |
|
| 239 | - | .par_iter() |
|
| 240 | - | .map(|n| { |
|
| 241 | - | let (bytes, m) = target.render_file(n); |
|
| 242 | - | (bytes, m.is_conflicted()) |
|
| 243 | - | }) |
|
| 244 | - | .collect(); |
|
| 237 | + | { |
|
| 238 | + | let _t = crate::timing::phase("checkout: build graphs"); |
|
| 239 | + | self.load_graphs_of(&mut target, &nodes)?; |
|
| 240 | + | } |
|
| 241 | + | let rendered: Vec<(Vec<u8>, bool)> = { |
|
| 242 | + | let _t = crate::timing::phase("checkout: render"); |
|
| 243 | + | nodes |
|
| 244 | + | .par_iter() |
|
| 245 | + | .map(|n| { |
|
| 246 | + | let (bytes, m) = target.render_file(n); |
|
| 247 | + | (bytes, m.is_conflicted()) |
|
| 248 | + | }) |
|
| 249 | + | .collect() |
|
| 250 | + | }; |
|
| 245 | 251 | target.drop_graphs(&nodes); |
|
| 252 | + | let _t = crate::timing::phase("checkout: write"); |
|
| 246 | 253 | if done == 0 && attr_count > 0 { |
|
| 247 | 254 | attrs = crate::attributes::Attributes::from_files( |
|
| 248 | 255 | batch |
|
| … | 163 unchanged lines |
modified crates/oxy-core/src/repo/sync.rs +18 −4
| … | 306 unchanged lines |
||
| 307 | 307 | seals.insert(s); |
|
| 308 | 308 | } |
|
| 309 | 309 | bundle.changes.push((c, sigs)); |
|
| 310 | + | bundle.hashes.push(*h); |
|
| 310 | 311 | } |
|
| 311 | 312 | for r in seals { |
|
| 312 | 313 | let attest = self.store.get_attest(&r)?; |
|
| … | 63 unchanged lines |
||
| 376 | 377 | } |
|
| 377 | 378 | } |
|
| 378 | 379 | } |
|
| 379 | - | let incoming: BTreeSet<ChangeHash> = bundle.changes.iter().map(|(c, _)| c.hash()).collect(); |
|
| 380 | + | // Each change's hash: from the bundle when its maker knew it, else |
|
| 381 | + | // worked out once (for a change adding a whole tree, hashing is |
|
| 382 | + | // chunking all its text). |
|
| 383 | + | let hashes: Vec<ChangeHash> = if bundle.hashes.len() == bundle.changes.len() { |
|
| 384 | + | bundle.hashes.clone() |
|
| 385 | + | } else { |
|
| 386 | + | bundle.changes.iter().map(|(c, _)| c.hash()).collect() |
|
| 387 | + | }; |
|
| 388 | + | let incoming: BTreeSet<ChangeHash> = hashes.iter().copied().collect(); |
|
| 380 | 389 | let boundary = self.boundary(); |
|
| 381 | 390 | let mut imported = Vec::new(); |
|
| 382 | - | for (c, sigs) in &bundle.changes { |
|
| 383 | - | let h = c.hash(); |
|
| 391 | + | for ((c, sigs), h) in bundle.changes.iter().zip(hashes) { |
|
| 384 | 392 | for d in &c.deps { |
|
| 385 | 393 | if !self.store.has(d) && !incoming.contains(d) && !shallow && !boundary.contains(d) |
|
| 386 | 394 | { |
|
| … | 4 unchanged lines |
||
| 391 | 399 | )); |
|
| 392 | 400 | } |
|
| 393 | 401 | } |
|
| 394 | - | self.store.put_change(c)?; |
|
| 402 | + | let stored = self.store.put_change(c)?; |
|
| 403 | + | if stored != h { |
|
| 404 | + | return fail(format!( |
|
| 405 | + | "change {} is not the change the bundle names", |
|
| 406 | + | h.short(12) |
|
| 407 | + | )); |
|
| 408 | + | } |
|
| 395 | 409 | for sig in sigs { |
|
| 396 | 410 | self.store.put_signature(&h, sig)?; |
|
| 397 | 411 | } |
|
| … | 1248 unchanged lines |
modified docs/benchmarks.md +28 −0
| … | 382 unchanged lines |
||
| 383 | 383 | The peak is now in storing the change, which holds every text's content |
|
| 384 | 384 | at once; the lines themselves (a Vec each) are the rest. |
|
| 385 | 385 | ||
| 386 | + | ### Clone (2026-09-30, oxygen#35) |
|
| 387 | + | ||
| 388 | + | A local clone of the Linux tip took 96 s and 5.2 GB RSS. Its checkout |
|
| 389 | + | spent most of its time rendering: working out each file's line order |
|
| 390 | + | allocated three sets per line (the contraction of the graph), 93 GB in |
|
| 391 | + | all, although every file of a fresh tree is one chain of lines. And the |
|
| 392 | + | import hashed each change three times, for the huge first change |
|
| 393 | + | chunking its 1.8 GB of text each time. |
|
| 394 | + | ||
| 395 | + | - A graph that is one chain from start to end with nothing deleted (an |
|
| 396 | + | edge to End besides the next line orders nothing) is read off in |
|
| 397 | + | order; any other graph takes the general algorithm, which also skips |
|
| 398 | + | its contraction where a line's successors are all kept. A test checks |
|
| 399 | + | the two agree. |
|
| 400 | + | - A bundle carries each change's hash when its maker knows it (the store |
|
| 401 | + | it came from, or the received bytes); never from the wire. An import |
|
| 402 | + | hashes each change once, by storing it, and checks it is the named one. |
|
| 403 | + | ||
| 404 | + | | Local clone, Linux tip | Time | RSS | |
|
| 405 | + | |---|---:|---:| |
|
| 406 | + | | Before | 96 s (81.6 s in a quieter run) | 5.2 to 5.4 GB | |
|
| 407 | + | | After | 61.0 s | 4.1 GB | |
|
| 408 | + | | Git | 9.6 s | 469 MB | |
|
| 409 | + | ||
| 410 | + | The checkout (31 s) now spends most of its time building graphs: each |
|
| 411 | + | batch of 2 048 files reads the 81 MiB first change again for its inline |
|
| 412 | + | inserts. |
|
| 413 | + |