Share a state's changes and deferred files between clones f6451fbbae3f

A clone of a state (a Git import keeps several, one per branch point
whose children are still to come) copied every change's metadata and
every file's deferred edits. They are now shared behind Arc and copied
on write, and a clone starts with an empty closure cache. The first
5 000 vscode commits import as before (317 s, 2.1 GB); 20 000 take
4 530 s and 6.7 GB, which shows the growth is the base state itself and
the import's work per commit, costed in the backlog (oxygen#25).

refs #35

Author: Andy McCall @andymccall <[email protected]>
When: 1 month ago · 2026-09-30 05:16 UTC
Signature: ✔ signed by Andy McCall
Hash: f6451fbbae3f5bd13fe87928d1cc2ae109df40394ec1d46ac3cb12e82b89d8b1
Depends on: 0bdc3f91a39f 119bbf50cb86 1681323d1dff 2517abe64654 252e8cb23fbc 33e77a5526fa 57d8dc05c9d4 86e1824f2d0b e8d38e47e9d7 ff0af1ee299d
Depended on by: aedfe465b611 ac03130a2b95 5552e69c7325 33be24095b4b d45dab0484ea 36e51aa806c6 3f1b8fe25ec9 f9575918477f 75b6c8665219
In bookmarks: agents-in-readme bare-hashes carried-not-removed carry-identical doctor-trust identical-sides keyless-tags main merge-remote-bookmark only-help-103 paths-for-resolutions-and-amend push-notes show-stat subrepo-pin-state switch-says
In tags: v0.1.0 v0.10.0 v0.11.0 v0.2.0 v0.3.0 v0.3.1 v0.4.0 v0.4.1 v0.4.2 v0.4.3 v0.4.4 v0.4.5 v0.4.6 v0.5.0 v0.5.1 v0.6.0 v0.7.0 v0.8.0 v0.9.0
Edits: 14 atoms, 0 tree operations, 2 files

Browse the tree at this change

show whitespace changes

modified crates/oxy-core/src/state.rs +20 −12

modified docs/backlog-overnight-3.md +15 −0

		…	54 unchanged lines
55	55
56	56		#[derive(Default)]
57	57		pub struct State {
58		-	changes: HashMap<ChangeHash, ChangeMeta>,
	58	+	/// Shared between clones, like `files` and `deferred`: a state kept
	59	+	/// beside another that differs by a few changes shares the rest.
	60	+	changes: HashMap<ChangeHash, Arc<ChangeMeta>>,
59	61		tree: TreeGraph,
60	62		/// Shared between clones: a clone copies a graph only when an edit
61	63		/// changes it (`Arc::make_mut`), so keeping several states that differ
		…	6 unchanged lines
68	70		/// `None` for a full state. For a history-only state (loaded for its
69	71		/// changes and tree), the file graphs not built yet, by node: `files`
70	72		/// holds only those loaded with [`State::load_graphs`].
71		-	deferred: Option<HashMap<NodeId, Deferred>>,
	73	+	deferred: Option<HashMap<NodeId, Arc<Deferred>>>,
72	74		}
73	75
74	76		use crate::objects::TextRef;
		…	24 unchanged lines
99	101		tree: self.tree.clone(),
100	102		files: self.files.clone(),
101	103		replaces: self.replaces.clone(),
102		-	closure: Mutex::new(self.closure.lock().unwrap().clone()),
	104	+	// A cache: the clone builds its own.
	105	+	closure: Mutex::new(HashMap::new()),
103	106		deferred: self.deferred.clone(),
104	107		}
105	108		}
		…	293 unchanged lines
399	402		};
400	403		let taken: Vec<(NodeId, Deferred)> = nodes
401	404		.iter()
402		-	.filter_map(\|n\| deferred.remove(n).map(\|d\| (*n, d)))
	405	+	.filter_map(\|n\| deferred.remove(n).map(\|d\| (*n, Arc::unwrap_or_clone(d))))
403	406		.collect();
404	407		let built = build_deferred(taken, load, read_change)?;
405	408		self.files
		…	17 unchanged lines
423	426		};
424	427		let copies: Vec<(NodeId, Deferred)> = nodes
425	428		.iter()
426		-	.map(\|n\| (*n, deferred.get(n).cloned().unwrap_or_default()))
	429	+	.map(\|n\| {
	430	+	(
	431	+	*n,
	432	+	deferred.get(n).map(\|d\| (**d).clone()).unwrap_or_default(),
	433	+	)
	434	+	})
427	435		.collect();
428	436		build_deferred(copies, load, read_change)
429	437		}
		…	85 unchanged lines
515	523		self.changes.contains_key(h)
516	524		}
517	525		pub fn get(&self, h: &ChangeHash) -> Option<&ChangeMeta> {
518		-	self.changes.get(h)
	526	+	self.changes.get(h).map(\|m\| &**m)
519	527		}
520	528		pub fn changes(&self) -> impl Iterator<Item = (&ChangeHash, &ChangeMeta)> {
521		-	self.changes.iter()
	529	+	self.changes.iter().map(\|(h, m)\| (h, &**m))
522	530		}
523	531		pub fn tree(&self) -> &TreeGraph {
524	532		&self.tree
		…	41 unchanged lines
566	574		}
567	575		let meta = ChangeMeta::of(&change);
568	576		self.apply_with(h, change, texts);
569		-	self.changes.insert(h, meta);
	577	+	self.changes.insert(h, Arc::new(meta));
570	578		self.closure.lock().unwrap().clear();
571	579		}
572	580
		…	223 unchanged lines
796	804		};
797	805		deferred.insert(
798	806		node,
799		-	Deferred {
	807	+	Arc::new(Deferred {
800	808		base: Some(base),
801	809		edits: Vec::new(),
802	810		texts: Vec::new(),
803		-	},
	811	+	}),
804	812		);
805	813		}
806	814		st.deferred = Some(deferred);
		…	24 unchanged lines
831	839		kind: ChangeKind::from_tag(o::uint(&e[6], "kind")?)?,
832	840		author: o::oref(&e[7], "author")?,
833	841		};
834		-	st.changes.insert(hash, meta);
	842	+	st.changes.insert(hash, Arc::new(meta));
835	843		}
836	844		st.tree = TreeGraph::from_value(o::field(m, 2, "tree")?)?;
837	845		let opt = \|v: &Value\| -> Result<Option<ObjectRef>, ObjectError> {
		…	69 unchanged lines
907	915		apply_file_edits(Arc::make_mut(g), h, edits)
908	916		}
909	917		None => {
910		-	let d = deferred.entry(node).or_default();
	918	+	let d = Arc::make_mut(deferred.entry(node).or_default());
911	919		d.edits.push((h, edits));
912	920		d.texts.extend(node_texts);
913	921		}
		…	1050 unchanged lines

		…	209 unchanged lines
210	210		- 2026-09-30 H4 attempted: the full vscode import (166 425 commits) under a 14 GB cap. With changes added mixed (e15074c7), it still grew about 0.4 MB per commit: 4.3 GB after 35 minutes and roughly 9 000 commits, far past the cap long before the end. It was stopped. The growth is the importer's kept states: each is a deep clone of a history-only state (change metadata, tree maps, the deferred edits of every file), up to eight at a time, with a new clone at every branch point.
211	211		- Next (H2b): make a state cheap to clone by sharing its parts (per-file deferred entries and graphs behind `Arc`, copied on write; the change map and tree likewise), so kept states share what they have in common.
212	212		- Also: an import should report progress on long runs.
	213	+	- 2026-09-30 H2b, first part: a state's change metadata and per-file deferred entries are shared between clones and copied on write, and a clone starts with an empty closure cache.
	214	+	- Measured: 5 000 vscode commits 317 s and 2.07 GB (unchanged); 20 000 commits 4 530 s and 6.7 GB. So most of the growth is the base state itself, not its clones.
	215	+	- Phases from 2 000 to 5 000 commits (2.5 times as many):
	216	+
	217	+	\| Phase \| 2 000 commits \| 5 000 commits \|
	218	+	\|---\|---:\|---:\|
	219	+	\| bases \| 5.3 s \| 24.4 s \|
	220	+	\| ls-tree \| 20.8 s \| 63.8 s \|
	221	+	\| render base \| 20.8 s \| 89.0 s \|
	222	+	\| build change \| 97.3 s \| 139.7 s \|
	223	+
	224	+	- Listing, rendering and rebuilding bases cost what the whole tree is, on every commit: oxygen#25. H3 to H5 need it. The plan:
	225	+	- list a commit's changes with `git diff-tree` against its first parent rather than `ls-tree`;
	226	+	- keep the tree view up to date as changes apply, rather than recomputing it;
	227	+	- render only the paths a commit touches.
213	228

`/`	Search
`g` `c`	Code
`g` `h`	History
`g` `b`	Bookmarks
`g` `t`	Trust
`y`	Permalink to this state (files and trees)
`?`	This sheet

Share a state's changes and deferred files between clones f6451fbbae3f

Keyboard