Diffs no longer anchor a block of new lines one line early 7891e21c762f

text_edits took its Myers ops from similar's capture_diff_slices, which
also runs a Compact hook that slides deletions along runs of equal
lines. On some inputs that left an insertion reporting the old index of
a line before an equal one, so the new lines were anchored one line too
early and a repeated line such as a closing brace ended up on the wrong
side of them. Snapshots, commits and Git imports all built changes this
way; the file was then not the file on disk.

The ops now come from the diff without that hook. A randomized test of
2 000 files made of repeated lines replays every edit set to the new
text (it fails on the old code). Verifying the scratch import of
tetriblox now reproduces all 539 commits, 95 585 files, where 5 differed.

Author: Andy McCall @andymccall <[email protected]>
When: 1 month ago · 2026-09-28 10:35 UTC
Signature: ✔ signed by Andy McCall
Hash: 7891e21c762f82fd14b1b54f4210ae17357b6b5d9c7f02bbcc31fc397dd98795
Depends on: 30ea9c1e70fa 79ab8528ff62 ff0af1ee299d
Depended on by: 2306a4d24779 b4fa2bc1ca6a da7997cc22bb
In bookmarks: agents-in-readme bare-hashes carried-not-removed carry-identical doctor-trust identical-sides keyless-tags main merge-remote-bookmark only-help-103 paths-for-resolutions-and-amend push-notes show-stat subrepo-pin-state switch-says
In tags: v0.1.0 v0.10.0 v0.11.0 v0.2.0 v0.3.0 v0.3.1 v0.4.0 v0.4.1 v0.4.2 v0.4.3 v0.4.4 v0.4.5 v0.4.6 v0.5.0 v0.5.1 v0.6.0 v0.7.0 v0.8.0 v0.9.0
Edits: 10 atoms, 4 tree operations, 6 files

Browse the tree at this change

hide whitespace changes

added docs/xenon_spec.md +406 −0

added fix.py +28 −0

added fix2.py +8 −0

added format_xenon.py +84 −0

modified crates/oxy-core/src/diff.rs +82 −2

modified crates/oxy-core/src/repo/git.rs +1 −1

1	+	with open('/home/andymccall/development/oxygen/docs/xenon_spec.md', 'r') as f:
2	+	text = f.read()
3	+
4	+	# Remove all existing ```json, ```cbor, ``` blocks to start clean
5	+	import re
6	+	text = re.sub(r'```[a-z]*\n', '', text)
7	+	text = re.sub(r'```\n', '', text)
8	+
9	+	# Now, apply the blocks properly
10	+	text = text.replace('Receipt {\n', '```cbor\nReceipt {\n')
11	+	text = text.replace(' duration_ms [8]: u64\n}\n', ' duration_ms [8]: u64\n}\n```\n')
12	+
13	+	text = text.replace('Attest {\n', '```json\nAttest {\n')
14	+	text = text.replace(' "reproducible": bool\n }\n }\n}\n', ' "reproducible": bool\n }\n }\n}\n```\n')
15	+
16	+	text = text.replace('Signature {\n', '```cbor\nSignature {\n')
17	+	text = text.replace(' ml [3]: bytes(3309) // Post-quantum signature\n}\n', ' ml [3]: bytes(3309) // Post-quantum signature\n}\n```\n')
18	+
19	+	text = text.replace('PolicyRule {\n', '```json\nPolicyRule {\n')
20	+	text = text.replace(' signer_roles: [ "official-worker" ]\n }\n ]\n}\n', ' signer_roles: [ "official-worker" ]\n }\n ]\n}\n```\n')
21	+
22	+	# KDL block is already fixed mostly, let's ensure it's clean
23	+	if '```kdl' not in text:
24	+	text = text.replace('schema-version 1\n', '```kdl\nschema-version 1\n')
25	+	text = text.replace(' }\n}\n\n### 3.2', ' }\n}\n```\n\n### 3.2')
26	+
27	+	with open('/home/andymccall/development/oxygen/docs/xenon_spec.md', 'w') as f:
28	+	f.write(text)

1	+	with open('/home/andymccall/development/oxygen/docs/xenon_spec.md', 'r') as f:
2	+	text = f.read()
3	+
4	+	text = text.replace(' "media_type": text\n}\n ],', ' "media_type": text\n }\n ],')
5	+	text = text.replace(' signer_roles: [ "official-worker" ]\n}\n ]\n}', ' signer_roles: [ "official-worker" ]\n }\n ]\n}')
6	+
7	+	with open('/home/andymccall/development/oxygen/docs/xenon_spec.md', 'w') as f:
8	+	f.write(text)

1	+	import re
2	+
3	+	with open('/home/andymccall/development/oxygen/docs/xenon_spec.md', 'r') as f:
4	+	text = f.read()
5	+
6	+	# Fix the preamble
7	+	text = re.sub(r'Format Version\s:\s(\d+)\nDate\s:\s(.?)\nStatus\s:\s(.?)\nTarget VCS\s:\s(.?)\nAudience\s:\s(.?)\n',
8	+	r'''\| \| \|
9	+	\|---\|---\|
10	+	\| Format version \| \1 \|
11	+	\| Date \| \2 \|
12	+	\| Status \| \3 \|
13	+	\| Target VCS \| \4 \|
14	+	\| Audience \| \5 \|
15	+	''', text)
16	+
17	+	# Convert ==== headers to ##
18	+	text = re.sub(r'={10,}\n(.*?)\n={10,}', lambda m: f"## {m.group(1).strip()}", text)
19	+
20	+	# Convert 3.1 style headers to ###
21	+	text = re.sub(r'\n(\d+\.\d+\s+.*?)\n', r'\n### \1\n', text)
22	+
23	+	# Code block for KDL
24	+	text = text.replace('schema-version 1\n\n// Define', '```kdl\nschema-version 1\n\n// Define')
25	+	text = text.replace(' }\n}\n\n### 3.2', ' }\n}\n```\n\n### 3.2')
26	+
27	+	# Code block for Receipt
28	+	text = text.replace('Receipt {\n', '```cbor\nReceipt {\n')
29	+	text = text.replace(' duration_ms [8]: u64\n}\n', ' duration_ms [8]: u64\n}\n```\n')
30	+
31	+	# Code block for Attest
32	+	text = text.replace('Attest {\n', '```json\nAttest {\n')
33	+	text = text.replace(' }\n }\n}\n\n### 7.2', ' }\n }\n}\n```\n\n### 7.2')
34	+
35	+	# Code block for Signature
36	+	text = text.replace('Signature {\n', '```cbor\nSignature {\n')
37	+	text = text.replace(' ml [3]: bytes(3309) // Post-quantum signature\n}\n', ' ml [3]: bytes(3309) // Post-quantum signature\n}\n```\n')
38	+
39	+	# Code block for PolicyRule
40	+	text = text.replace('PolicyRule {\n', '```json\nPolicyRule {\n')
41	+	text = text.replace(' signer_roles: [ "official-worker" ]\n }\n ]\n}\n', ' signer_roles: [ "official-worker" ]\n }\n ]\n}\n```\n')
42	+
43	+
44	+	# Architectural Fixes:
45	+
46	+	# 1. Change change_closure to scope_tree_hash in Receipt
47	+	text = text.replace('change_closure [3]: [ChangeHash] // Sorted minimal dependency closure', 'scope_tree_hash [3]: ObjectRef // BLAKE3 tree hash of the materialized scope')
48	+
49	+	text = text.replace('''### 6.2 Cache Re-use Invariant
50	+	Let Scope(J) be the file paths and tree nodes declared by job J.
51	+	Let Closure(S) be the dependency closure of State S.
52	+	The restricted change closure is:
53	+	C_J(S) = { c in Closure(S) \| TouchedPaths(c) intersects Scope(J) }
54	+
55	+	If an existing Receipt R exists such that:
56	+	R.job_name == J.name &&
57	+	R.pipeline_hash == Hash(J) &&
58	+	R.change_closure == C_J(S) &&
59	+	R.environment == CurrentEnvironmentHash''',
60	+	'''### 6.2 Cache Re-use Invariant
61	+	Let Scope(J) be the file paths and tree nodes declared by job J.
62	+	Because changes that do not share scopes commute, we do not need to track lists of ChangeHashes. Instead, we hash the materialized tree of Scope(J).
63	+
64	+	If an existing Receipt R exists such that:
65	+	R.job_name == J.name &&
66	+	R.pipeline_hash == Hash(J) &&
67	+	R.scope_tree_hash == TreeHash(Scope(J)) &&
68	+	R.environment == CurrentEnvironmentHash''')
69	+
70	+	# Fix provenance change_closure
71	+	text = text.replace('"change_closure": [ChangeHash], // Transitive changes included', '"scope_tree_hash": ObjectRef, // Captures exact state without O(N) array bloat')
72	+
73	+	# 2. Add Consensus Builds to Policy Integration
74	+	text = text.replace('''### 8.1 Policy Gate Definition
75	+	A Policy rule governing a bookmark MAY demand valid Xenon attestations:''',
76	+	'''### 8.1 Policy Gate Definition
77	+	A Policy rule governing a bookmark MAY demand valid Xenon attestations. To mitigate a compromised authorized worker, high-security repositories can require Consensus Builds by setting the threshold > 1. This requires multiple independent runner identities to arrive at the exact same output hashes and both sign it.''')
78	+
79	+	# 3. Add mmap performance note to 5.1
80	+	text = text.replace('- Missing chunks are fetched lazily using the POST /objects protocol.', '- Missing chunks are fetched lazily using the POST /objects protocol.\n - To prevent FUSE context-switch bottlenecks on many small reads, xenon-casfs aggressively mmaps chunk data and utilizes host page caching.')
81	+
82	+
83	+	with open('/home/andymccall/development/oxygen/docs/xenon_spec.md', 'w') as f:
84	+	f.write(text)

		…	5 unchanged lines
6	6		use crate::graph::AtomKey;
7	7		use crate::ids::NodeId;
8	8		use crate::objects::{AtomRange, AtomRef, Edit};
9		-	use similar::{capture_diff_slices, Algorithm, DiffOp};
	9	+	use similar::DiffOp;
10	10
11	11		/// Split into lines, each keeping its terminator; a final unterminated line
12	12		/// is its own atom.
		…	28 unchanged lines
41	41		pub fn text_edits(node: NodeId, old: &[(AtomKey, Vec<u8>)], new: &[u8]) -> Vec<Edit> {
42	42		let old_lines: Vec<&[u8]> = old.iter().map(\|(_, l)\| l.as_slice()).collect();
43	43		let new_lines = split_lines(new);
44		-	let ops = capture_diff_slices(Algorithm::Myers, &old_lines, &new_lines);
	44	+	// Myers ops in file order. `capture_diff_slices` also runs the `Compact`
	45	+	// hook, which slides deletions along runs of equal lines and can leave
	46	+	// an insertion with the old index of a line before an equal one: the
	47	+	// insertion was then anchored a line too early and landed on the wrong
	48	+	// side of it (found by verifying a real Git import commit by commit).
	49	+	let mut hook = similar::algorithms::Replace::new(similar::algorithms::Capture::new());
	50	+	let _ = similar::algorithms::diff_deadline(
	51	+	similar::Algorithm::Myers,
	52	+	&mut hook,
	53	+	&old_lines,
	54	+	0..old_lines.len(),
	55	+	&new_lines,
	56	+	0..new_lines.len(),
	57	+	None,
	58	+	);
	59	+	let ops = hook.into_inner().into_ops();
45	60
46	61		let anchor = \|i: usize\| -> (AtomRef, AtomRef) {
47	62		let after = if i == 0 { AtomRef::Start } else { AtomRef::Atom { change: old[i - 1].0 .0, index: old[i - 1].0 .1 } };
		…	45 unchanged lines
93	108
94	109		fn h(n: u8) -> ObjectRef {
95	110		ObjectRef([n; 32])
	111	+	}
	112	+
	113	+	/// Replaying `text_edits` by position must give the new text, on files
	114	+	/// full of repeated lines, where a diff has many equally good
	115	+	/// alignments to choose between.
	116	+	#[test]
	117	+	fn edits_replay_to_the_new_text_on_repetitive_files() {
	118	+	use rand::{Rng, SeedableRng};
	119	+	let words: [&[u8]; 5] = [b"}\n", b" }\n", b"\n", b"a\n", b" b\n"];
	120	+	let mut rng = rand::rngs::StdRng::seed_from_u64(11);
	121	+	for _ in 0..2000 {
	122	+	let old: Vec<Vec<u8>> = (0..rng.gen_range(0..30))
	123	+	.map(\|_\| words[rng.gen_range(0..5)].to_vec())
	124	+	.collect();
	125	+	let new: Vec<u8> = (0..rng.gen_range(0..30))
	126	+	.flat_map(\|_\| words[rng.gen_range(0..5)].to_vec())
	127	+	.collect();
	128	+	let keyed: Vec<(AtomKey, Vec<u8>)> = old
	129	+	.iter()
	130	+	.enumerate()
	131	+	.map(\|(i, l)\| ((h(1), i as u32), l.clone()))
	132	+	.collect();
	133	+	let edits = text_edits(NodeId::ROOT, &keyed, &new);
	134	+	let mut dead = vec![false; old.len()];
	135	+	let mut after: std::collections::BTreeMap<i64, Vec<u8>> = Default::default();
	136	+	for e in &edits {
	137	+	match e {
	138	+	Edit::Delete { atoms, .. } => atoms
	139	+	.iter()
	140	+	.for_each(\|r\| (r.from..r.to).for_each(\|i\| dead[i as usize] = true)),
	141	+	Edit::Insert {
	142	+	after: a,
	143	+	before: b,
	144	+	lines,
	145	+	..
	146	+	} => {
	147	+	let at = match a {
	148	+	AtomRef::Start => -1,
	149	+	AtomRef::Atom { index, .. } => *index as i64,
	150	+	AtomRef::End => unreachable!(),
	151	+	};
	152	+	let next = match b {
	153	+	AtomRef::End => old.len() as i64,
	154	+	AtomRef::Atom { index, .. } => *index as i64,
	155	+	AtomRef::Start => unreachable!(),
	156	+	};
	157	+	assert_eq!(next, at + 1, "anchors adjacent");
	158	+	after.entry(at).or_default().extend(lines.iter().flatten());
	159	+	}
	160	+	_ => {}
	161	+	}
	162	+	}
	163	+	let mut out = after.get(&-1).cloned().unwrap_or_default();
	164	+	for (i, l) in old.iter().enumerate() {
	165	+	if !dead[i] {
	166	+	out.extend(l);
	167	+	}
	168	+	out.extend(after.get(&(i as i64)).cloned().unwrap_or_default());
	169	+	}
	170	+	assert_eq!(
	171	+	String::from_utf8_lossy(&out),
	172	+	String::from_utf8_lossy(&new),
	173	+	"old {old:?}"
	174	+	);
	175	+	}
96	176		}
97	177
98	178		#[test]
		…	48 unchanged lines

		…	207 unchanged lines
208	208		}
209	209
210	210		/// Git commits already imported into the bookmark, by sha.
211		-	fn imported_commits(&self) -> Result<HashMap<String, ChangeHash>> {
	211	+	pub(crate) fn imported_commits(&self) -> Result<HashMap<String, ChangeHash>> {
212	212		let mut map = HashMap::new();
213	213		for e in self.log()? {
214	214		if e.is_working_copy {
		…	429 unchanged lines

1	+	# Xenon (xenon) — Deterministic Execution & Provenance Pipeline Specification
2	+
3	+	\| \| \|
4	+	\|---\|---\|
5	+	\| Format version \| 1 \|
6	+	\| Date \| 2026-09-28 \|
7	+	\| Status \| Draft 0.1 \|
8	+	\| Target VCS \| Oxygen (oxy) Draft 0.8i \|
9	+	\| Audience \| Implementers, Security Architects, and Systems Engineers \|
10	+
11	+
12	+	The key words MUST, MUST NOT, SHOULD, SHOULD NOT, and MAY are to be
13	+	interpreted as described in RFC 2119.
14	+
15	+	## 1. Overview and Core Invariants
16	+
17	+	Xenon (xenon) is a content-addressed, hermetic execution engine, runner system,
18	+	and pipeline language designed natively for the Oxygen (oxy) version control
19	+	system.
20	+
21	+	Existing runner engines (GitHub Actions, GitLab CI, Forgejo Runner) suffer from
22	+	structural flaws:
23	+	1. Workspace Bloat: Full clones, index unpacks, and checkout scans dominate I/O.
24	+	2. String Injection: Shell interpolation within untyped YAML files produces
25	+	expression injection and command execution exploits.
26	+	3. Decoupled Supply Chain: Attestations (SLSA, in-toto, Cosign) exist outside
27	+	the VCS, relying on external registries rather than native repository trust.
28	+	4. Coarse Caching: Opaque tarballs keyed by arbitrary string hashes fail to
29	+	leverage commutative history or granular dependency graphs.
30	+
31	+	Xenon solves these liabilities through direct integration with Oxygen:
32	+	- Zero-Copy Workspaces: Jobs mount workspaces directly from Oxygen's FastCDC-
33	+	chunked, BLAKE3-addressed Content-Addressable Store (CAS) using Copy-on-Write
34	+	(COW) overlays.
35	+	- Commutative History & Impact Trees: Pipelines evaluate over change closures
36	+	rather than linear snapshot sequences. Commuting changes automatically reuse
37	+	cached execution receipts without redundant re-execution.
38	+	- Micro-Engine Sandboxing: Jobs run inside isolated environments (WASM, gVisor,
39	+	or MicroVMs) with strict, deny-by-default capabilities and no ambient
40	+	network egress.
41	+	- Native Cryptographic Provenance: Provenance and execution receipts are
42	+	generated outside the untrusted guest, encoded as deterministic CBOR, and
43	+	signed using composite Ed25519 + ML-DSA-65 keys as native Oxygen Attest
44	+	objects governed by in-repository Policy rules.
45	+
46	+	## 2. Threat Model and Security Architecture
47	+
48	+	### 2.1 Adversaries
49	+	- Untrusted Proposal Author: Submits code to an open repository containing
50	+	malicious build configurations intended to escape the runner, exfiltrate
51	+	secrets, or corrupt shared worker caches.
52	+	- Malicious Dependency / Upstream: Third-party dependencies invoked during
53	+	build time attempting out-of-band network calls or host file modification.
54	+	- Compromised Runner / Malicious Worker: An execution node attempting to
55	+	forge successful build receipts or inject backdoors into published artifacts.
56	+
57	+	### 2.2 Security Invariants
58	+	- I1 Deny-by-Default Execution: Sandboxes possess no network interfaces
59	+	(excluding loopback when requested) and no host filesystem access unless
60	+	explicitly granted in policy-signed configuration.
61	+	- I2 Injection-Free Invocation: Configuration parsing directly yields an argv
62	+	string vector executed via execve(). Shell interpretation (/bin/sh -c) does
63	+	not exist.
64	+	- I3 Host-Enforced Provenance: The running payload cannot author, inspect, or
65	+	alter its own provenance statement. Provenance is emitted solely by the
66	+	isolation hypervisor boundary.
67	+	- I4 Cryptographic Identity & Authorization: Runners execute using an Oxygen
68	+	Identity whose KeyId is enrolled as a delegate in the repository's Policy.
69	+	Unenrolled runners cannot publish recognized attestations or advance gates.
70	+
71	+	## 3. The Xenon Configuration Language (.xenon)
72	+
73	+	Pipelines are declared in KDL syntax within .xenon/pipeline.kdl at the
74	+	repository root. KDL provides strict typing, structured nesting, and rejects
75	+	arbitrary text macro expansion.
76	+
77	+	### 3.1 Pipeline Specification Schema
78	+
79	+	```kdl
80	+	schema-version 1
81	+
82	+	// Define the hermetic execution boundary
83	+	sandbox "rust-builder" {
84	+	runtime "microvm" // Options: "wasm" \| "gvisor" \| "microvm"
85	+	rootfs "cas://blob/9b5c2a1e8f3d4e7a6b8c9d0e1f2a3b4c5d6e7f8a9b0c1d2e3f4a5b6c7d8e9f0a"
86	+	memory-limit "4GiB"
87	+	cpu-limit 4
88	+	network "isolated" // Options: "isolated" \| "loopback" \| "egress"
89	+	}
90	+
91	+	// Job evaluated based on Oxygen change closure
92	+	job "compile-core" {
93	+	sandbox "rust-builder"
94	+
95	+	// Commutation boundary: runs only if change closure touches these paths
96	+	scope "crates/oxy-core/" "crates/oxy-server/" "Cargo.*"
97	+
98	+	// Zero-copy workspace mount backed by Oxygen CAS
99	+	mount {
100	+	source "oxy://workspace"
101	+	target "/src"
102	+	include "crates/oxy-core" "crates/oxy-server" "Cargo.toml" "Cargo.lock"
103	+	readonly #true
104	+	}
105	+
106	+	// Ephemeral scratch space
107	+	mount {
108	+	source "ephemeral://tmpfs"
109	+	target "/src/target"
110	+	size "8GiB"
111	+	}
112	+
113	+	env {
114	+	RUSTFLAGS "-Dwarnings"
115	+	CARGO_INCREMENTAL "0"
116	+	}
117	+
118	+	// Direct argv execution vector (no subshell)
119	+	exec "cargo" "build" "--release" "--package" "oxy-core"
120	+
121	+	output {
122	+	artifact "oxy-core-lib" {
123	+	path "/src/target/release/liboxy_core.rlib"
124	+	format "blob"
125	+	}
126	+	}
127	+	}
128	+
129	+	job "unit-tests" {
130	+	sandbox "rust-builder"
131	+	depends-on "compile-core"
132	+
133	+	mount {
134	+	source "oxy://workspace"
135	+	target "/src"
136	+	readonly #true
137	+	}
138	+
139	+	mount {
140	+	source "artifact://compile-core/oxy-core-lib"
141	+	target "/src/target/release/liboxy_core.rlib"
142	+	}
143	+
144	+	exec "cargo" "test" "--release" "--package" "oxy-core"
145	+	}
146	+
147	+	publish "release-provenance" {
148	+	depends-on "unit-tests"
149	+
150	+	attest {
151	+	kind "provenance"
152	+	bind-ref "bookmarks/main"
153	+	include-artifacts "compile-core/oxy-core-lib"
154	+	}
155	+	}
156	+
157	+
158	+	### 3.2 Canonical Compilation
159	+	Upon commit, .xenon/pipeline.kdl is validated and compiled into a deterministic
160	+	CBOR job-graph blob. Runners evaluate the canonical CBOR to prevent syntax or
161	+	parser ambiguities across varying client platforms.
162	+
163	+	## 4. Sandbox Execution Engine
164	+
165	+	Xenon implements three distinct isolation tiers depending on the workload:
166	+
167	+	### 4.1 Level 0: WASM Component Sandbox
168	+	- Intended for static linters, formatters, doc generators, and policy checks.
169	+	- Runs on a standard WebAssembly System Interface (WASI) runtime.
170	+	- Startup latency: < 5 ms.
171	+	- Zero host I/O; virtualized file system presented in-memory.
172	+
173	+	### 4.2 Level 1: Intercepted Syscall Isolation (gVisor / Landlock + seccomp-bpf)
174	+	- Intended for standard compilation and hermetic test suites.
175	+	- Emulates or filters Linux system call interfaces in user space.
176	+	- Blocks access to raw network sockets, real-time hardware clocks, and device
177	+	nodes.
178	+	- Memory and execution CPU quotas are strictly bounded via cgroups v2.
179	+
180	+	### 4.3 Level 2: MicroVM Isolation (Firecracker / Cloud-Hypervisor)
181	+	- Intended for kernel tests, multi-container integration, and untrusted code.
182	+	- Ephemeral Linux microVM booting in < 80 ms.
183	+	- The root filesystem is presented as a read-only virtio-block device mapped
184	+	directly to FastCDC chunks stored in Oxygen's CAS.
185	+	- Guest VM memory is cleared upon termination.
186	+
187	+	## 5. Content-Addressable Storage (CAS) & Zero-Copy Mounting
188	+
189	+	Xenon runners bypass traditional clone operations. Repositories are accessed
190	+	directly via the local object store (.oxy/objects/) or streamed on demand
191	+	using Oxygen's wire protocol.
192	+
193	+	### 5.1 FUSE Overlay Mounting (xenon-casfs)
194	+	1. The runner receives a task targeting an Oxygen State ObjectRef (a sorted
195	+	list of maximal heads).
196	+	2. The nearest Checkpoint is resolved. Uncommitted changes between the
197	+	checkpoint and heads are replayed into a virtual tree in-memory.
198	+	3. The virtual tree is exposed to the sandbox via xenon-casfs:
199	+	- File reads resolve chunks using the FastCDC gear table (32 KiB to 256 KiB,
200	+	mean 64 KiB) pinned in Oxygen Spec Section 13.2.
201	+	- Missing chunks are fetched lazily using the POST /objects protocol.
202	+	- To prevent FUSE context-switch bottlenecks on many small reads, xenon-casfs aggressively mmaps chunk data and utilizes host page caching.
203	+	- All guest mutations are diverted to a scratch tmpfs layer; base objects
204	+	remain immutable.
205	+
206	+	### 5.2 Artifact Capture
207	+	1. At task completion, xenon-casfs scans the paths declared in the output block.
208	+	2. Modified outputs are chunked with FastCDC, hashed via BLAKE3-256, and
209	+	written as canonical Blob and Chunk objects.
210	+	3. Output ObjectRefs are stored directly in the Oxygen repository.
211	+
212	+	## 6. Commutative Execution & Change Caching
213	+
214	+	In Oxygen, changes that do not share scopes (referenced atoms, placements, or
215	+	nodes) commute. Xenon exploits this invariant to provide granular caching.
216	+
217	+	### 6.1 Execution Receipt Object
218	+	Every successful or failed job run emits an immutable Receipt:
219	+
220	+	```cbor
221	+	Receipt {
222	+	version [0]: 1
223	+	job_name [1]: text
224	+	pipeline_hash [2]: ObjectRef
225	+	scope_tree_hash [3]: ObjectRef // BLAKE3 tree hash of the materialized scope
226	+	inputs [4]: [ObjectRef] // Artifacts and input files consumed
227	+	environment [5]: ObjectRef // Hash of rootfs, compiler flags, and argv
228	+	outputs [6]: [ObjectRef] // Produced artifact blobs
229	+	exit_code [7]: u32
230	+	duration_ms [8]: u64
231	+	}
232	+	```
233	+
234	+	A Receipt is hashed as an Oxygen object with type name "receipt":
235	+	ObjectRef = BLAKE3-256("oxy1 receipt " \|\| decimal(len) \|\| 0x00 \|\| body)
236	+
237	+	### 6.2 Cache Re-use Invariant
238	+	Let Scope(J) be the file paths and tree nodes declared by job J.
239	+	Because changes that do not share scopes commute, we do not need to track lists of ChangeHashes. Instead, we hash the materialized tree of Scope(J).
240	+
241	+	If an existing Receipt R exists such that:
242	+	R.job_name == J.name &&
243	+	R.pipeline_hash == Hash(J) &&
244	+	R.scope_tree_hash == TreeHash(Scope(J)) &&
245	+	R.environment == CurrentEnvironmentHash
246	+
247	+	Then the job execution is bypassed. R.outputs are mapped into downstream
248	+	dependents immediately as a cached-commuted hit.
249	+
250	+	## 7. Native Provenance Specification
251	+
252	+	Xenon formalizes build provenance as a native, tamper-evident Oxygen Attest
253	+	object (kind: "attest"). Provenance is generated exclusively by the runner
254	+	hypervisor, not the guest payload.
255	+
256	+	### 7.1 Provenance Object Schema
257	+
258	+	```json
259	+	Attest {
260	+	version [0]: 1
261	+	kind [1]: "attest"
262	+	repo [2]: bytes(16) // Oxygen Repository ID
263	+	hlc [3]: HLC // Hybrid Logical Clock
264	+	body [4]: {
265	+	"type": "xenon/provenance/v1",
266	+
267	+	// Subject artifacts produced by the build
268	+	"subjects": [
269	+	{
270	+	"name": text,
271	+	"ref": ObjectRef, // BLAKE3 hash of output blob
272	+	"media_type": text
273	+	}
274	+	],
275	+
276	+	// Builder platform and execution identity
277	+	"builder": {
278	+	"runner_id": KeyId, // ObjectRef of runner's Oxygen Identity
279	+	"isolation_level": text, // "wasm" \| "gvisor" \| "microvm"
280	+	"hypervisor_image": ObjectRef // Rootfs blob digest
281	+	},
282	+
283	+	// Exact input materials
284	+	"materials": {
285	+	"state_heads": [ChangeHash], // Oxygen heads of the state built
286	+	"scope_tree_hash": ObjectRef, // Captures exact state without O(N) array bloat
287	+	"inputs": [
288	+	{
289	+	"name": text,
290	+	"ref": ObjectRef // Content hash of locked dependencies
291	+	}
292	+	]
293	+	},
294	+
295	+	// Structured execution arguments
296	+	"invocation": {
297	+	"job": text,
298	+	"pipeline_spec": ObjectRef, // ObjectRef of .xenon/pipeline.kdl
299	+	"argv": [text], // Unmodified execution vector
300	+	"env": map<text, text> // Explicitly whitelisted environment
301	+	},
302	+
303	+	// Environment observations
304	+	"metadata": {
305	+	"started_on": u64, // Wallclock milliseconds
306	+	"finished_on": u64,
307	+	"hermetic": bool, // Verified zero outbound network access
308	+	"reproducible": bool
309	+	}
310	+	}
311	+	}
312	+	```
313	+
314	+	### 7.2 Composite Signatures
315	+	All provenance attestations are signed using Oxygen's composite signature
316	+	construction (Ed25519 and ML-DSA-65):
317	+
318	+	```cbor
319	+	Signature {
320	+	key [0]: KeyId
321	+	alg [1]: 1 // 1 = Ed25519 + ML-DSA-65
322	+	ed [2]: bytes(64) // Classical signature
323	+	ml [3]: bytes(3309) // Post-quantum signature
324	+	}
325	+	```
326	+
327	+	Both signatures are computed over:
328	+	"oxy1 sig attest\0" \|\| KeyId \|\| ObjectRef
329	+
330	+	## 8. Policy Integration & Bookmark Gates
331	+
332	+	Oxygen repositories enforce branch and bookmark protections via signed Policy
333	+	objects (Oxygen Spec Section 14.5). Xenon integrates provenance directly into
334	+	policy evaluation during push validation.
335	+
336	+	### 8.1 Policy Gate Definition
337	+	A Policy rule governing a bookmark MAY demand valid Xenon attestations. To mitigate a compromised authorized worker, high-security repositories can require Consensus Builds by setting the threshold > 1. This requires multiple independent runner identities to arrive at the exact same output hashes and both sign it.
338	+
339	+	```json
340	+	PolicyRule {
341	+	pattern: "bookmarks/production"
342	+	keys: [ KeyId ]
343	+	threshold: 1
344	+	require: "signed"
345	+	allow_conflicts: false
346	+	require_attestations: [
347	+	{
348	+	type: "xenon/provenance/v1",
349	+	job: "release-provenance",
350	+	require_hermetic: true,
351	+	signer_roles: [ "official-worker" ]
352	+	}
353	+	]
354	+	}
355	+
356	+	### 8.2 Push Verification Sequence (VCS Engine Enforcement)
357	+	When a push request arrives at an Oxygen server attempting to update a
358	+	protected bookmark:
359	+	1. Object Integrity: Validate that all changes, chunks, and blobs exist and match
360	+	their BLAKE3 hashes.
361	+	2. Attestation Resolution: Verify that an Attest object of type
362	+	"xenon/provenance/v1" is present, referencing the exact heads of the new
363	+	State.
364	+	3. Identity Verification: Validate that the attestation's composite signature
365	+	resolves to a KeyId listed as an authorized delegate of the repository
366	+	policy roots.
367	+	4. Invariant Checking: Confirm that the provenance statement records
368	+	"hermetic: true" and exit code 0.
369	+	5. Atomic Update: If validation fails at any point, the entire push is refused
370	+	with ok = false.
371	+
372	+	## 9. Zero-Environment Secret Brokering
373	+
374	+	Xenon forbids the injection of raw secret strings into process environment
375	+	tables to eliminate accidental leaking via child process dumping, procfs
376	+	reads, and verbose compiler logging.
377	+
378	+	### 9.1 Ephemeral Secret Broker
379	+	1. If a job requires a credential (e.g., signing key, cloud deployment token),
380	+	it declares a secret mount:
381	+	secret "deploy-token" {
382	+	vault-ref "infra/tokens/prod"
383	+	mount-path "/run/secrets/token"
384	+	}
385	+	2. The runner negotiates a short-lived token lease from the configured vault.
386	+	3. The secret is mounted onto an ephemeral, in-memory ramfs location inside
387	+	the sandbox with access mode 0400, owned exclusively by the worker process.
388	+	4. Upon process termination, the memory pages are zeroized and the token is
389	+	marked for immediate revocation at the vault.
390	+
391	+	## 10. Conformance and Verification Requirements
392	+
393	+	A conforming Xenon implementation MUST pass the following tests:
394	+
395	+	1. Injection Immunity: Attempting to supply unescaped shell metacharacters
396	+	(; && \| ` $() \n) in any job field MUST pass those characters as literal
397	+	bytes to argv; subshells MUST NOT be spawned.
398	+	2. FastCDC Determinism: Chunks produced by artifact extraction MUST match
399	+	the gear table cuts and BLAKE3 hashes generated by Oxygen's core chunker.
400	+	3. Commutation Hit Invariance: For any two commuting changes A and B, running
401	+	a pipeline on State(A, B) MUST reuse cached receipts generated on State(A)
402	+	for any job whose scope does not intersect the touched paths of B.
403	+	4. Non-Downgrade Attestation: An attestation missing the ML-DSA-65 component
404	+	MUST be rejected by policy verification.
405	+	5. Hermetic Network Isolation: Jobs configured with network "isolated" MUST
406	+	have outgoing TCP SYN and UDP packets dropped at the hypervisor boundary.

`/`	Search
`g` `c`	Code
`g` `h`	History
`g` `b`	Bookmarks
`g` `t`	Trust
`y`	Permalink to this state (files and trees)
`?`	This sheet

Diffs no longer anchor a block of new lines one line early 7891e21c762f

Keyboard