OXYGENHUB

Render a file that is one chain of lines without contracting it 598009a75f6c

Cloning the Linux tip spent most of its checkout rendering files:
working out each file's line order contracted the graph with three sets
allocated per line, 93 GB in all, though every file of a fresh tree is a
single chain. A graph that is one chain from start to end with nothing
deleted is now read off in order (an edge to End besides the next line
orders nothing); any other graph takes the general algorithm, which
also skips contraction where a line's successors are all kept. A test
checks the shortcut and the general order agree, and that the shortcut
stands aside for inserts between lines, conflicts and deletions.

A bundle now carries each change's hash when its maker knows it (the
store it came from, or the bytes received; never the wire), so an
import hashes each change once, by storing it, and checks it is the
change named. A local clone of the Linux tip takes 61 s and 4.1 GB
(from 82 to 96 s and 5.2 GB).

refs #35
Author
Andy McCall @andymccall <[email protected]>
When
· 2026-09-29 23:51 UTC
Signature
✔ signed by Andy McCall
Hash
598009a75f6c7aeb8572794c40ec8516b239010aba62035293c8d7147a5f6ca4
Depends on
0bdc3f91a39f 192fad1e46a2 1d360c91a289 2863456a0f6d 28fc23d7706a 5b9c6ad45f6e 6f8aeb996558 750251b21a96 75167672ebcd 982864068a59 c5ce0367eddf ff0af1ee299d
Depended on by
091fef80755d 7eac74abc586 40a40bb7b798 dbf06d342e1d e83354cdac22 0f93f49ce059 a87c7b9d563a 2c75672fb2fe 519c7c061bcb
In bookmarks
agents-in-readme bare-hashes carried-not-removed carry-identical doctor-trust identical-sides keyless-tags main merge-remote-bookmark only-help-103 paths-for-resolutions-and-amend push-notes show-stat subrepo-pin-state switch-says
In tags
v0.1.0 v0.10.0 v0.11.0 v0.2.0 v0.3.0 v0.3.1 v0.4.0 v0.4.1 v0.4.2 v0.4.3 v0.4.4 v0.4.5 v0.4.6 v0.5.0 v0.5.1 v0.6.0 v0.7.0 v0.8.0 v0.9.0
Edits
20 atoms, 0 tree operations, 5 files

Browse the tree at this change

modified crates/oxy-core/src/graph.rs +110 −5
…
322 unchanged lines
323 323
        kept: &HashSet<Vertex>,
324 324
        memo: &mut HashMap<Vertex, BTreeSet<Vertex>>,
325 325
    ) -> BTreeSet<Vertex> {
326 +
        let direct = self.succ.get(&v).map(Edges::as_slice).unwrap_or_default();
327 +
        // The common case, a successor that is itself kept: nothing to
328 +
        // contract.
329 +
        if direct.iter().all(|s| kept.contains(s)) {
330 +
            return direct.iter().copied().collect();
331 +
        }
326 332
        let mut out = BTreeSet::new();
327 -
        let mut stack: Vec<Vertex> = self
328 -
            .succ
329 -
            .get(&v)
330 -
            .map(|s| s.iter().copied().collect())
331 -
            .unwrap_or_default();
333 +
        let mut stack: Vec<Vertex> = direct.to_vec();
332 334
        let mut seen = HashSet::default();
333 335
        while let Some(s) = stack.pop() {
334 336
            if kept.contains(&s) {
…
7 unchanged lines
342 344
            }
343 345
        }
344 346
        out
347 +
    }
348 +
349 +
    /// The atoms in order when the graph is one chain from Start to End with
350 +
    /// nothing deleted: a file only ever inserted into in sequence, as every
351 +
    /// file of a fresh commit or a clone is. Such a graph has no conflict.
352 +
    /// An edge to End besides the chain's next atom (Start's own, or an
353 +
    /// append's anchor) orders nothing, since End is last anyway.
354 +
    fn as_chain(&self) -> Option<Vec<AtomKey>> {
355 +
        if self
356 +
            .atoms
357 +
            .values()
358 +
            .any(|a| a.content.is_none() || !a.deleted_by.is_empty())
359 +
        {
360 +
            return None;
361 +
        }
362 +
        let mut order = Vec::with_capacity(self.atoms.len());
363 +
        let mut v = Vertex::Start;
364 +
        loop {
365 +
            let succ = self.succ.get(&v)?.as_slice();
366 +
            let mut atoms = succ.iter().filter(|s| **s != Vertex::End);
367 +
            match (atoms.next(), atoms.next()) {
368 +
                (None, _) if succ.contains(&Vertex::End) => break,
369 +
                (Some(Vertex::Atom(k)), None) if order.len() < self.atoms.len() => {
370 +
                    order.push(*k);
371 +
                    v = Vertex::Atom(*k);
372 +
                }
373 +
                _ => return None,
374 +
            }
375 +
        }
376 +
        (order.len() == self.atoms.len()).then_some(order)
345 377
    }
346 378
347 379
    /// Order the live atoms and locate conflicts.
348 380
    pub fn materialize(&self, deps: &dyn DepOracle) -> Materialized {
381 +
        if let Some(order) = self.as_chain() {
382 +
            return Materialized {
383 +
                segments: if order.is_empty() {
384 +
                    vec![]
385 +
                } else {
386 +
                    vec![Segment::Lines(order)]
387 +
                },
388 +
            };
389 +
        }
390 +
        self.materialize_general(deps)
391 +
    }
392 +
393 +
    /// [`FileGraph::materialize`] for any graph: contraction, then a
394 +
    /// topological order with conflict regions.
395 +
    fn materialize_general(&self, deps: &dyn DepOracle) -> Materialized {
349 396
        // 1. Vertices that survive contraction: live atoms, ghosts, Start, End.
350 397
        let mut kept: HashSet<Vertex> = HashSet::default();
351 398
        let mut ghosts: HashSet<AtomKey> = HashSet::default();
…
387 unchanged lines
739 786
            }
740 787
            false
741 788
        }
789 +
    }
790 +
791 +
    /// The chain shortcut agrees with the general algorithm wherever it
792 +
    /// applies, and stands aside wherever it does not.
793 +
    #[test]
794 +
    fn the_chain_shortcut_agrees_with_the_general_order() {
795 +
        let mut g = FileGraph::new();
796 +
        g.insert(
797 +
            h(1),
798 +
            0,
799 +
            Vertex::Start,
800 +
            Vertex::End,
801 +
            &lines(&["a\n", "b\n", "c\n"]),
802 +
        );
803 +
        assert!(g.as_chain().is_some());
804 +
        assert_eq!(g.materialize(&NoDeps), g.materialize_general(&NoDeps));
805 +
        // Appended after the last line: still a chain.
806 +
        g.insert(
807 +
            h(2),
808 +
            0,
809 +
            Vertex::Atom((h(1), 2)),
810 +
            Vertex::End,
811 +
            &lines(&["d\n"]),
812 +
        );
813 +
        assert!(g.as_chain().is_some());
814 +
        assert_eq!(g.materialize(&NoDeps), g.materialize_general(&NoDeps));
815 +
        // Inserted between two lines: not a chain (a has two successors).
816 +
        g.insert(
817 +
            h(4),
818 +
            0,
819 +
            Vertex::Atom((h(1), 0)),
820 +
            Vertex::Atom((h(1), 1)),
821 +
            &lines(&["x\n"]),
822 +
        );
823 +
        assert!(g.as_chain().is_none());
824 +
        assert_eq!(text(&g, &g.materialize(&NoDeps)), "a\nx\nb\nc\nd\n");
825 +
        // Two inserts at one anchor: a conflict, never a chain.
826 +
        let mut c = FileGraph::new();
827 +
        c.insert(h(1), 0, Vertex::Start, Vertex::End, &lines(&["a\n"]));
828 +
        c.insert(h(2), 0, Vertex::Start, Vertex::End, &lines(&["b\n"]));
829 +
        assert!(c.as_chain().is_none());
830 +
        assert!(c.materialize(&NoDeps).is_conflicted());
831 +
        // A deletion: not a chain.
832 +
        let mut d = FileGraph::new();
833 +
        d.insert(h(1), 0, Vertex::Start, Vertex::End, &lines(&["a\n", "b\n"]));
834 +
        d.delete(
835 +
            h(3),
836 +
            &AtomRange {
837 +
                change: h(1),
838 +
                from: 0,
839 +
                to: 1,
840 +
            },
841 +
        );
842 +
        assert!(d.as_chain().is_none());
843 +
        assert_eq!(d.materialize(&NoDeps), d.materialize_general(&NoDeps));
844 +
        // Empty.
845 +
        let e = FileGraph::new();
846 +
        assert_eq!(e.materialize(&NoDeps), e.materialize_general(&NoDeps));
742 847
    }
743 848
744 849
    #[test]
…
421 unchanged lines
modified crates/oxy-core/src/protocol.rs +14 −1
…
85 unchanged lines
86 86
    /// Policies between the receiver's and the one sent with the push,
87 87
    /// ascending by serial (spec §14.5).
88 88
    pub policies: Vec<SignedPolicy>,
89 +
    /// Each change's hash, in the order of `changes`, when whoever made
90 +
    /// the bundle knows it: the store it came from, or the bytes it was
91 +
    /// decoded from. Never read from the wire. Empty when unknown.
92 +
    pub hashes: Vec<ChangeHash>,
89 93
}
90 94
91 95
impl Bundle {
…
518 unchanged lines
610 614
            if pair.len() != 2 {
611 615
                return Err(schema("change entry: expected [change, sigs]"));
612 616
            }
613 -
            let change = Change::decode_with(o::bytes(&pair[0], "change")?, &mut |text, count| {
617 +
            let encoded = o::bytes(&pair[0], "change")?;
618 +
            b.hashes
619 +
                .push(crate::hash::hash_object(ObjectType::Change, encoded));
620 +
            let change = Change::decode_with(encoded, &mut |text, count| {
614 621
                let by_ref = by_ref.get_or_insert_with(|| {
615 622
                    b.objects
616 623
                        .iter()
…
349 unchanged lines
966 973
                policy: policy.clone(),
967 974
                sigs: vec![psig.clone()],
968 975
            }],
976 +
            hashes: vec![],
969 977
        };
970 978
        let d = wire::decode_bundle(&wire::encode_bundle(&bundle)).unwrap();
979 +
        assert_eq!(
980 +
            d.hashes,
981 +
            vec![c.hash()],
982 +
            "the receiver hashes what it was sent"
983 +
        );
971 984
        assert_eq!(d.changes, bundle.changes);
972 985
        assert_eq!(d.attests, bundle.attests);
973 986
        assert_eq!(d.objects, bundle.objects);
…
128 unchanged lines
modified crates/oxy-core/src/repo/checkout.rs +15 −8
…
233 unchanged lines
234 234
        for batch in batches {
235 235
            use rayon::prelude::*;
236 236
            let nodes: Vec<NodeId> = batch.iter().map(|(_, n, _)| *n).collect();
237 -
            self.load_graphs_of(&mut target, &nodes)?;
238 -
            let rendered: Vec<(Vec<u8>, bool)> = nodes
239 -
                .par_iter()
240 -
                .map(|n| {
241 -
                    let (bytes, m) = target.render_file(n);
242 -
                    (bytes, m.is_conflicted())
243 -
                })
244 -
                .collect();
237 +
            {
238 +
                let _t = crate::timing::phase("checkout: build graphs");
239 +
                self.load_graphs_of(&mut target, &nodes)?;
240 +
            }
241 +
            let rendered: Vec<(Vec<u8>, bool)> = {
242 +
                let _t = crate::timing::phase("checkout: render");
243 +
                nodes
244 +
                    .par_iter()
245 +
                    .map(|n| {
246 +
                        let (bytes, m) = target.render_file(n);
247 +
                        (bytes, m.is_conflicted())
248 +
                    })
249 +
                    .collect()
250 +
            };
245 251
            target.drop_graphs(&nodes);
252 +
            let _t = crate::timing::phase("checkout: write");
246 253
            if done == 0 && attr_count > 0 {
247 254
                attrs = crate::attributes::Attributes::from_files(
248 255
                    batch
…
163 unchanged lines
modified crates/oxy-core/src/repo/sync.rs +18 −4
…
306 unchanged lines
307 307
                seals.insert(s);
308 308
            }
309 309
            bundle.changes.push((c, sigs));
310 +
            bundle.hashes.push(*h);
310 311
        }
311 312
        for r in seals {
312 313
            let attest = self.store.get_attest(&r)?;
…
63 unchanged lines
376 377
                }
377 378
            }
378 379
        }
379 -
        let incoming: BTreeSet<ChangeHash> = bundle.changes.iter().map(|(c, _)| c.hash()).collect();
380 +
        // Each change's hash: from the bundle when its maker knew it, else
381 +
        // worked out once (for a change adding a whole tree, hashing is
382 +
        // chunking all its text).
383 +
        let hashes: Vec<ChangeHash> = if bundle.hashes.len() == bundle.changes.len() {
384 +
            bundle.hashes.clone()
385 +
        } else {
386 +
            bundle.changes.iter().map(|(c, _)| c.hash()).collect()
387 +
        };
388 +
        let incoming: BTreeSet<ChangeHash> = hashes.iter().copied().collect();
380 389
        let boundary = self.boundary();
381 390
        let mut imported = Vec::new();
382 -
        for (c, sigs) in &bundle.changes {
383 -
            let h = c.hash();
391 +
        for ((c, sigs), h) in bundle.changes.iter().zip(hashes) {
384 392
            for d in &c.deps {
385 393
                if !self.store.has(d) && !incoming.contains(d) && !shallow && !boundary.contains(d)
386 394
                {
…
4 unchanged lines
391 399
                    ));
392 400
                }
393 401
            }
394 -
            self.store.put_change(c)?;
402 +
            let stored = self.store.put_change(c)?;
403 +
            if stored != h {
404 +
                return fail(format!(
405 +
                    "change {} is not the change the bundle names",
406 +
                    h.short(12)
407 +
                ));
408 +
            }
395 409
            for sig in sigs {
396 410
                self.store.put_signature(&h, sig)?;
397 411
            }
…
1248 unchanged lines
modified docs/benchmarks.md +28 −0
…
382 unchanged lines
383 383
The peak is now in storing the change, which holds every text's content
384 384
at once; the lines themselves (a Vec each) are the rest.
385 385
386 +
### Clone (2026-09-30, oxygen#35)
387 +
388 +
A local clone of the Linux tip took 96 s and 5.2 GB RSS. Its checkout
389 +
spent most of its time rendering: working out each file's line order
390 +
allocated three sets per line (the contraction of the graph), 93 GB in
391 +
all, although every file of a fresh tree is one chain of lines. And the
392 +
import hashed each change three times, for the huge first change
393 +
chunking its 1.8 GB of text each time.
394 +
395 +
- A graph that is one chain from start to end with nothing deleted (an
396 +
  edge to End besides the next line orders nothing) is read off in
397 +
  order; any other graph takes the general algorithm, which also skips
398 +
  its contraction where a line's successors are all kept. A test checks
399 +
  the two agree.
400 +
- A bundle carries each change's hash when its maker knows it (the store
401 +
  it came from, or the received bytes); never from the wire. An import
402 +
  hashes each change once, by storing it, and checks it is the named one.
403 +
404 +
| Local clone, Linux tip | Time | RSS |
405 +
|---|---:|---:|
406 +
| Before | 96 s (81.6 s in a quieter run) | 5.2 to 5.4 GB |
407 +
| After | 61.0 s | 4.1 GB |
408 +
| Git | 9.6 s | 469 MB |
409 +
410 +
The checkout (31 s) now spends most of its time building graphs: each
411 +
batch of 2 048 files reads the 81 MiB first change again for its inline
412 +
inserts.
413 +