Change format 3: text blobs hold the lines' bytes 0f93f49ce059
A version 2 text blob held its insert's lines as a CBOR array, so every line carried a length header: for the Linux tree's 46 308 files of 4 KiB or more, 62 MB more raw and 35 MB more compressed, most of what its store still had over Git's after gc. Version 3 stores the lines' bytes, joined, whenever splitting them by the line rule gives the lines back, which every insert a snapshot makes does; others stay a CBOR array and are marked (key 7), and a reader refuses a marked array the line rule would have split back, so a change still has one encoding. A text blob is then the file's own bytes and shares chunks with any blob of the same content. Readers keep reading versions 1 and 2, whose hashes do not move; the resolver and TextRef carry the blob's form. Spec Draft 0.8r. refs #40
- Author
- Andy McCall @andymccall <[email protected]>
- When
- · 2026-09-30 02:28 UTC
- Signature
- ✔ signed by Andy McCall
- Hash
0f93f49ce059545cb28a316def983a6221dda14dc185d411f8e1304abd8aa081- Depends on
- 01a47cd7f0c8 17f94b13ae54 192fad1e46a2 2c75672fb2fe 387335ff7f4d 57d8dc05c9d4 598009a75f6c 5b9c6ad45f6e 6f8aeb996558 75167672ebcd 86e1824f2d0b 982864068a59 a4fad5b085c7 da652d6392d8 ff0af1ee299d
- Depended on by
- 5b4c51307b14 acadb5ed28a1
- In bookmarks
- agents-in-readme bare-hashes carried-not-removed carry-identical doctor-trust identical-sides keyless-tags main merge-remote-bookmark only-help-103 paths-for-resolutions-and-amend push-notes show-stat subrepo-pin-state switch-says
- In tags
- v0.1.0 v0.10.0 v0.11.0 v0.2.0 v0.3.0 v0.3.1 v0.4.0 v0.4.1 v0.4.2 v0.4.3 v0.4.4 v0.4.5 v0.4.6 v0.5.0 v0.5.1 v0.6.0 v0.7.0 v0.8.0 v0.9.0
- Edits
- 56 atoms, 0 tree operations, 9 files
Browse the tree at this change
modified crates/oxy-core/src/lines.rs +33 −0
| … | 64 unchanged lines |
||
| 65 | 65 | &self.0.buf |
|
| 66 | 66 | } |
|
| 67 | 67 | ||
| 68 | + | /// Whether splitting [`Lines::bytes`] by the line rule gives these lines |
|
| 69 | + | /// back: every line but the last ends in a newline and none holds |
|
| 70 | + | /// another. Every insert oxy makes is so, and such lines are stored as |
|
| 71 | + | /// their bytes alone (spec §7.4). |
|
| 72 | + | pub fn is_split_form(&self) -> bool { |
|
| 73 | + | let buf = &self.0.buf; |
|
| 74 | + | let mut start = 0usize; |
|
| 75 | + | for (i, &end) in self.0.ends.iter().enumerate() { |
|
| 76 | + | let line = &buf[start..end as usize]; |
|
| 77 | + | if line.is_empty() { |
|
| 78 | + | return false; |
|
| 79 | + | } |
|
| 80 | + | let last = i + 1 == self.0.ends.len(); |
|
| 81 | + | let body = if last && !line.ends_with(b"\n") { |
|
| 82 | + | line |
|
| 83 | + | } else if line.ends_with(b"\n") { |
|
| 84 | + | &line[..line.len() - 1] |
|
| 85 | + | } else { |
|
| 86 | + | return false; |
|
| 87 | + | }; |
|
| 88 | + | if body.contains(&b'\n') { |
|
| 89 | + | return false; |
|
| 90 | + | } |
|
| 91 | + | start = end as usize; |
|
| 92 | + | } |
|
| 93 | + | true |
|
| 94 | + | } |
|
| 95 | + | ||
| 68 | 96 | /// Lines `range` as lines of their own. |
|
| 69 | 97 | pub fn slice(&self, range: std::ops::Range<usize>) -> Lines { |
|
| 70 | 98 | self.iter().skip(range.start).take(range.len()).collect() |
|
| … | 82 unchanged lines |
||
| 153 | 181 | assert_eq!(Lines::from(l.to_vec()), l); |
|
| 154 | 182 | } |
|
| 155 | 183 | let l = Lines::split(b"a\nb\nc\n"); |
|
| 184 | + | assert!(l.is_split_form()); |
|
| 185 | + | assert!(!Lines::from(vec![b"a".to_vec(), b"b\n".to_vec()]).is_split_form()); |
|
| 186 | + | assert!(!Lines::from(vec![b"a\nb\n".to_vec()]).is_split_form()); |
|
| 187 | + | assert!(!Lines::from(vec![Vec::new()]).is_split_form()); |
|
| 188 | + | assert!(Lines::from(vec![b"a\n".to_vec(), b"end".to_vec()]).is_split_form()); |
|
| 156 | 189 | assert_eq!(&l[1], b"b\n"); |
|
| 157 | 190 | assert_eq!( |
|
| 158 | 191 | l.slice(1..3).to_vec(), |
|
| … | 5 unchanged lines |
modified crates/oxy-core/src/objects.rs +124 −38
| … | 15 unchanged lines |
||
| 16 | 16 | pub const FORMAT_VERSION: u64 = 1; |
|
| 17 | 17 | ||
| 18 | 18 | /// The change format writers produce (spec §6.1). Version 2 stores each |
|
| 19 | - | /// large insert's lines as a text blob; readers accept 1 and 2, and a |
|
| 20 | - | /// change keeps the version it was made in, so its hash never moves. |
|
| 21 | - | pub const CHANGE_VERSION: u64 = 2; |
|
| 19 | + | /// large insert's lines as a text blob holding a CBOR array of them; |
|
| 20 | + | /// version 3 holds the lines' bytes themselves when splitting them by the |
|
| 21 | + | /// line rule gives them back. Readers accept 1 to 3, and a change keeps |
|
| 22 | + | /// the version it was made in, so its hash never moves. |
|
| 23 | + | pub const CHANGE_VERSION: u64 = 3; |
|
| 22 | 24 | ||
| 25 | + | /// How a text blob holds an insert's lines (spec §7.4). |
|
| 26 | + | #[derive(Clone, Copy, Debug, PartialEq, Eq, Hash)] |
|
| 27 | + | pub enum TextForm { |
|
| 28 | + | /// A CBOR array of the lines' byte strings: version 2. |
|
| 29 | + | Cbor, |
|
| 30 | + | /// The lines' bytes, split again by the line rule: version 3. |
|
| 31 | + | Raw, |
|
| 32 | + | /// A CBOR array in a version 3 change, for lines the line rule would |
|
| 33 | + | /// not split back (marked by key 7): never for lines it would. |
|
| 34 | + | CborFallback, |
|
| 35 | + | } |
|
| 36 | + | ||
| 23 | 37 | /// In a version 2 change, an insert whose lines encode (as a CBOR array of |
|
| 24 | 38 | /// byte strings) to at least this many bytes is stored as a text blob |
|
| 25 | 39 | /// holding exactly that encoding; a shorter one is inline (spec §6.1). |
|
| … | 434 unchanged lines |
||
| 460 | 474 | } |
|
| 461 | 475 | ||
| 462 | 476 | /// The edit's value; `stored` is the text blob of an insert stored as |
|
| 463 | - | /// one (see [`Change::value_with`], which works them out). |
|
| 464 | - | fn to_value(&self, stored: Option<ObjectRef>) -> Value { |
|
| 477 | + | /// one, and its form (see [`Change::value_with`], which works them out). |
|
| 478 | + | fn to_value(&self, stored: Option<(ObjectRef, TextForm)>) -> Value { |
|
| 465 | 479 | let nid = |n: &NodeId| Value::Bytes(n.0.to_vec()); |
|
| 466 | 480 | let opt = |o: &Option<ObjectRef>| match o { |
|
| 467 | 481 | Some(r) => Value::Bytes(r.0.to_vec()), |
|
| … | 7 unchanged lines |
||
| 475 | 489 | before, |
|
| 476 | 490 | lines, |
|
| 477 | 491 | }, |
|
| 478 | - | Some(r), |
|
| 479 | - | ) => Value::map([ |
|
| 480 | - | (0, Value::Uint(0)), |
|
| 481 | - | (1, nid(node)), |
|
| 482 | - | (2, after.to_value()), |
|
| 483 | - | (3, before.to_value()), |
|
| 484 | - | (5, Value::Bytes(r.0.to_vec())), |
|
| 485 | - | (6, Value::Uint(lines.len() as u64)), |
|
| 486 | - | ]), |
|
| 492 | + | Some((r, form)), |
|
| 493 | + | ) => { |
|
| 494 | + | let mut fields = vec![ |
|
| 495 | + | (0, Value::Uint(0)), |
|
| 496 | + | (1, nid(node)), |
|
| 497 | + | (2, after.to_value()), |
|
| 498 | + | (3, before.to_value()), |
|
| 499 | + | (5, Value::Bytes(r.0.to_vec())), |
|
| 500 | + | (6, Value::Uint(lines.len() as u64)), |
|
| 501 | + | ]; |
|
| 502 | + | if form == TextForm::CborFallback { |
|
| 503 | + | fields.push((7, Value::Uint(1))); |
|
| 504 | + | } |
|
| 505 | + | Value::map(fields) |
|
| 506 | + | } |
|
| 487 | 507 | ( |
|
| 488 | 508 | Edit::Insert { |
|
| 489 | 509 | node, |
|
| … | 59 unchanged lines |
||
| 549 | 569 | if count == 0 { |
|
| 550 | 570 | return schema("insert with no lines"); |
|
| 551 | 571 | } |
|
| 572 | + | let form = match (version, m.get(&7)) { |
|
| 573 | + | (2, None) => TextForm::Cbor, |
|
| 574 | + | (_, None) => TextForm::Raw, |
|
| 575 | + | (3.., Some(Value::Uint(1))) => TextForm::CborFallback, |
|
| 576 | + | _ => return schema("unknown text blob form"), |
|
| 577 | + | }; |
|
| 552 | 578 | // The resolver checks the count (`text_lines`). |
|
| 553 | - | resolve(&text, count)? |
|
| 579 | + | resolve(&text, count, form)? |
|
| 554 | 580 | } |
|
| 555 | 581 | _ => { |
|
| 556 | 582 | return schema( |
|
| … | 85 unchanged lines |
||
| 642 | 668 | /// the stored texts go to `texts`, in order, when it is given. Worked |
|
| 643 | 669 | /// out on every core, a bounded batch at a time: chunking and hashing |
|
| 644 | 670 | /// a whole tree's text is most of encoding a change that adds it. |
|
| 645 | - | fn stored_texts(&self, sink: &mut TextSink) -> Vec<Option<ObjectRef>> { |
|
| 671 | + | fn stored_texts(&self, sink: &mut TextSink) -> Vec<Option<(ObjectRef, TextForm)>> { |
|
| 646 | 672 | use rayon::prelude::*; |
|
| 647 | - | let mut refs = vec![None; self.edits.len()]; |
|
| 673 | + | let mut refs: Vec<Option<(ObjectRef, TextForm)>> = vec![None; self.edits.len()]; |
|
| 648 | 674 | let stored: Vec<usize> = (0..self.edits.len()) |
|
| 649 | 675 | .filter(|i| self.edits[*i].stored_text(self.version)) |
|
| 650 | 676 | .collect(); |
|
| … | 3 unchanged lines |
||
| 654 | 680 | .map(|i| { |
|
| 655 | 681 | let Edit::Insert { lines, .. } = &self.edits[*i] else { |
|
| 656 | 682 | unreachable!("only inserts are stored as text"); |
|
| 683 | + | }; |
|
| 684 | + | let form = text_form(self.version, lines); |
|
| 685 | + | let content = match form { |
|
| 686 | + | TextForm::Raw => lines.bytes().to_vec(), |
|
| 687 | + | TextForm::Cbor | TextForm::CborFallback => cbor::encode_byte_strings(lines), |
|
| 657 | 688 | }; |
|
| 658 | - | let content = cbor::encode_byte_strings(lines); |
|
| 659 | 689 | StoredText { |
|
| 660 | 690 | blob: Blob::of(&content), |
|
| 661 | 691 | content, |
|
| 692 | + | form, |
|
| 662 | 693 | } |
|
| 663 | 694 | }) |
|
| 664 | 695 | .collect(); |
|
| 665 | 696 | for (i, t) in batch.iter().zip(&made) { |
|
| 666 | - | refs[*i] = Some(t.blob.hash()); |
|
| 697 | + | refs[*i] = Some((t.blob.hash(), t.form)); |
|
| 667 | 698 | } |
|
| 668 | 699 | if !sink(made) { |
|
| 669 | 700 | break; |
|
| … | 97 unchanged lines |
||
| 767 | 798 | /// Decode a change that stores no text blob; one that does needs |
|
| 768 | 799 | /// [`Change::decode_with`] and something to read them from. |
|
| 769 | 800 | pub fn decode(bytes: &[u8]) -> Result<Change, ObjectError> { |
|
| 770 | - | Self::decode_with(bytes, &mut |r, _| { |
|
| 801 | + | Self::decode_with(bytes, &mut |r, _, _| { |
|
| 771 | 802 | schema(format!( |
|
| 772 | 803 | "text blob {} needs a store to read it from", |
|
| 773 | 804 | r.short(12) |
|
| … | 9 unchanged lines |
||
| 783 | 814 | } |
|
| 784 | 815 | ||
| 785 | 816 | pub fn from_value(v: &Value) -> Result<Change, ObjectError> { |
|
| 786 | - | Self::from_value_with(v, &mut |r, _| { |
|
| 817 | + | Self::from_value_with(v, &mut |r, _, _| { |
|
| 787 | 818 | schema(format!( |
|
| 788 | 819 | "text blob {} needs a store to read it from", |
|
| 789 | 820 | r.short(12) |
|
| … | 50 unchanged lines |
||
| 840 | 871 | /// the insert claims; [`text_lines`] reads and checks a blob's content. |
|
| 841 | 872 | /// A resolver may also return no lines, leaving a placeholder for a reader |
|
| 842 | 873 | /// that fills it in later (history-only states, spec §11). |
|
| 843 | - | pub type Resolver<'a> = dyn FnMut(&ObjectRef, u32) -> Result<Lines, ObjectError> + 'a; |
|
| 874 | + | pub type Resolver<'a> = dyn FnMut(&ObjectRef, u32, TextForm) -> Result<Lines, ObjectError> + 'a; |
|
| 844 | 875 | ||
| 845 | 876 | /// Receives the stored texts of a change being encoded, a batch at a |
|
| 846 | 877 | /// time; returns false to stop making more. |
|
| … | 3 unchanged lines |
||
| 850 | 881 | /// file's graph (spec §11): it keeps no line until then. |
|
| 851 | 882 | #[derive(Clone, Copy, Debug, PartialEq, Eq)] |
|
| 852 | 883 | pub enum TextRef { |
|
| 853 | - | /// A stored insert: its text blob and line count. |
|
| 854 | - | Blob(ObjectRef, u32), |
|
| 884 | + | /// A stored insert: its text blob, line count, and the blob's form. |
|
| 885 | + | Blob(ObjectRef, u32, TextForm), |
|
| 855 | 886 | /// An inline insert: the byte range of its lines' array in the |
|
| 856 | 887 | /// encoding of `change`. |
|
| 857 | 888 | Inline { |
|
| … | 116 unchanged lines |
||
| 974 | 1005 | fields.insert(k, c.value()?); |
|
| 975 | 1006 | } |
|
| 976 | 1007 | let Some((start, end, count)) = inline else { |
|
| 977 | - | return Edit::from_value(&Value::Map(fields), version, &mut |text, count| { |
|
| 978 | - | texts.push(TextRef::Blob(*text, count)); |
|
| 1008 | + | return Edit::from_value(&Value::Map(fields), version, &mut |text, count, form| { |
|
| 1009 | + | texts.push(TextRef::Blob(*text, count, form)); |
|
| 979 | 1010 | Ok(Lines::default()) |
|
| 980 | 1011 | }); |
|
| 981 | 1012 | }; |
|
| … | 28 unchanged lines |
||
| 1010 | 1041 | pub struct StoredText { |
|
| 1011 | 1042 | pub blob: Blob, |
|
| 1012 | 1043 | pub content: Vec<u8>, |
|
| 1044 | + | pub form: TextForm, |
|
| 1013 | 1045 | } |
|
| 1014 | 1046 | ||
| 1047 | + | /// The form a change of `version` stores these lines in. |
|
| 1048 | + | fn text_form(version: u64, lines: &Lines) -> TextForm { |
|
| 1049 | + | match version { |
|
| 1050 | + | ..=2 => TextForm::Cbor, |
|
| 1051 | + | _ if lines.is_split_form() => TextForm::Raw, |
|
| 1052 | + | _ => TextForm::CborFallback, |
|
| 1053 | + | } |
|
| 1054 | + | } |
|
| 1055 | + | ||
| 1015 | 1056 | /// The lines a text blob's `content` holds, checked against the rules for |
|
| 1016 | 1057 | /// one: canonical, as many as the insert claims, and not short enough to |
|
| 1017 | 1058 | /// have been inline. |
|
| 1018 | - | pub fn text_lines(content: &[u8], count: u32) -> Result<Lines, ObjectError> { |
|
| 1019 | - | if content.len() < TEXT_BLOB_MIN { |
|
| 1059 | + | pub fn text_lines(content: &[u8], count: u32, form: TextForm) -> Result<Lines, ObjectError> { |
|
| 1060 | + | let lines = match form { |
|
| 1061 | + | TextForm::Raw => Lines::split(content), |
|
| 1062 | + | TextForm::Cbor | TextForm::CborFallback => array_lines(&cbor::decode(content)?)?, |
|
| 1063 | + | }; |
|
| 1064 | + | // One encoding per change: the size rule and the form are both fixed |
|
| 1065 | + | // by the lines. |
|
| 1066 | + | if cbor::byte_strings_len(&lines) < TEXT_BLOB_MIN { |
|
| 1020 | 1067 | return schema(format!( |
|
| 1021 | 1068 | "a text blob of {} bytes is short enough to be inline", |
|
| 1022 | 1069 | content.len() |
|
| 1023 | 1070 | )); |
|
| 1024 | 1071 | } |
|
| 1025 | - | let lines = array_lines(&cbor::decode(content)?)?; |
|
| 1072 | + | if form == TextForm::CborFallback && lines.is_split_form() { |
|
| 1073 | + | return schema("these lines are stored as their bytes in a version 3 change"); |
|
| 1074 | + | } |
|
| 1026 | 1075 | if lines.len() != count as usize { |
|
| 1027 | 1076 | return schema(format!( |
|
| 1028 | 1077 | "a text blob holds {} lines, the insert says {count}", |
|
| … | 233 unchanged lines |
||
| 1262 | 1311 | let c = sample_change(); |
|
| 1263 | 1312 | let mut v = c.to_value(); |
|
| 1264 | 1313 | if let Value::Map(m) = &mut v { |
|
| 1265 | - | m.insert(0, Value::Uint(3)); |
|
| 1314 | + | m.insert(0, Value::Uint(4)); |
|
| 1266 | 1315 | } |
|
| 1267 | 1316 | assert!( |
|
| 1268 | 1317 | matches!(Change::from_value(&v), Err(ObjectError::Schema(s)) if s.contains("version")) |
|
| … | 48 unchanged lines |
||
| 1317 | 1366 | assert_eq!(bytes, c.encode()); |
|
| 1318 | 1367 | assert!(bytes.len() < 1024, "{}", bytes.len()); |
|
| 1319 | 1368 | assert_eq!(texts.len(), 1); |
|
| 1320 | - | assert_eq!(texts[0].content, cbor::encode_byte_strings(&lines)); |
|
| 1369 | + | // Version 3: the lines' bytes themselves. |
|
| 1370 | + | assert_eq!(texts[0].form, TextForm::Raw); |
|
| 1371 | + | assert_eq!(texts[0].content, lines.bytes()); |
|
| 1321 | 1372 | assert_eq!(texts[0].blob, Blob::of(&texts[0].content)); |
|
| 1322 | 1373 | assert!(Change::decode(&bytes).is_err(), "reading it needs the text"); |
|
| 1323 | - | let d = Change::decode_with(&bytes, &mut |r, n| { |
|
| 1374 | + | let d = Change::decode_with(&bytes, &mut |r, n, form| { |
|
| 1324 | 1375 | assert_eq!(*r, texts[0].blob.hash()); |
|
| 1325 | - | text_lines(&texts[0].content, n) |
|
| 1376 | + | assert_eq!(form, TextForm::Raw); |
|
| 1377 | + | text_lines(&texts[0].content, n, form) |
|
| 1326 | 1378 | }) |
|
| 1327 | 1379 | .unwrap(); |
|
| 1328 | 1380 | assert_eq!(d, c); |
|
| 1329 | 1381 | assert_eq!(d.hash(), c.hash()); |
|
| 1382 | + | // Version 2 held the same lines as a CBOR array, and still reads. |
|
| 1383 | + | let mut v2 = c.clone(); |
|
| 1384 | + | v2.version = 2; |
|
| 1385 | + | let (b2, t2) = v2.encode_parts(); |
|
| 1386 | + | assert_eq!(t2[0].form, TextForm::Cbor); |
|
| 1387 | + | assert_eq!(t2[0].content, cbor::encode_byte_strings(&lines)); |
|
| 1388 | + | let d2 = Change::decode_with(&b2, &mut |_, n, form| text_lines(&t2[0].content, n, form)) |
|
| 1389 | + | .unwrap(); |
|
| 1390 | + | assert_eq!(d2, v2); |
|
| 1391 | + | // Lines the line rule would not split back stay a CBOR array, |
|
| 1392 | + | // marked; and a marked array of lines that would is refused. |
|
| 1393 | + | let odd: Lines = (0..500) |
|
| 1394 | + | .map(|i| format!("no newline {i}").into_bytes()) |
|
| 1395 | + | .collect(); |
|
| 1396 | + | let c3 = with_insert(sample_change(), odd.clone()); |
|
| 1397 | + | let (b3, t3) = c3.encode_parts(); |
|
| 1398 | + | assert_eq!(t3[0].form, TextForm::CborFallback); |
|
| 1399 | + | let d3 = Change::decode_with(&b3, &mut |_, n, form| text_lines(&t3[0].content, n, form)) |
|
| 1400 | + | .unwrap(); |
|
| 1401 | + | assert_eq!(d3, c3); |
|
| 1402 | + | let marked = cbor::encode_byte_strings(&lines); |
|
| 1403 | + | assert!(text_lines(&marked, 500, TextForm::CborFallback).is_err()); |
|
| 1330 | 1404 | } |
|
| 1331 | 1405 | ||
| 1332 | 1406 | #[test] |
|
| … | 37 unchanged lines |
||
| 1370 | 1444 | if let Edit::Insert { lines, .. } = e { |
|
| 1371 | 1445 | assert!(lines.is_empty(), "no line kept"); |
|
| 1372 | 1446 | *lines = match next.next().unwrap() { |
|
| 1373 | - | TextRef::Blob(r, n) => { |
|
| 1447 | + | TextRef::Blob(r, n, form) => { |
|
| 1374 | 1448 | assert_eq!(*r, stored[0].blob.hash()); |
|
| 1375 | - | text_lines(&stored[0].content, *n).unwrap() |
|
| 1449 | + | text_lines(&stored[0].content, *n, *form).unwrap() |
|
| 1376 | 1450 | } |
|
| 1377 | 1451 | TextRef::Inline { change, start, end } => { |
|
| 1378 | 1452 | assert_eq!(*change, me); |
|
| … | 24 unchanged lines |
||
| 1403 | 1477 | cbor::encode(&Value::Array( |
|
| 1404 | 1478 | lines.iter().map(|l| Value::Bytes(l.to_vec())).collect() |
|
| 1405 | 1479 | )) |
|
| 1480 | + | ); |
|
| 1481 | + | assert_eq!(text_lines(&content, 500, TextForm::Cbor).unwrap(), lines); |
|
| 1482 | + | assert!(text_lines(&content, 499, TextForm::Cbor).is_err()); |
|
| 1483 | + | assert_eq!( |
|
| 1484 | + | text_lines(lines.bytes(), 500, TextForm::Raw).unwrap(), |
|
| 1485 | + | lines |
|
| 1406 | 1486 | ); |
|
| 1407 | - | assert_eq!(text_lines(&content, 500).unwrap(), lines); |
|
| 1408 | - | assert!(text_lines(&content, 499).is_err()); |
|
| 1487 | + | assert!(text_lines(lines.bytes(), 501, TextForm::Raw).is_err()); |
|
| 1409 | 1488 | let small = cbor::encode_byte_strings(&vec![b"x\n".to_vec()].into()); |
|
| 1410 | - | assert!(text_lines(&small, 1).is_err(), "short enough to be inline"); |
|
| 1489 | + | assert!( |
|
| 1490 | + | text_lines(&small, 1, TextForm::Cbor).is_err(), |
|
| 1491 | + | "short enough to be inline" |
|
| 1492 | + | ); |
|
| 1493 | + | assert!( |
|
| 1494 | + | text_lines(b"x\n", 1, TextForm::Raw).is_err(), |
|
| 1495 | + | "short enough to be inline" |
|
| 1496 | + | ); |
|
| 1411 | 1497 | } |
|
| 1412 | 1498 | ||
| 1413 | 1499 | #[test] |
|
| … | 80 unchanged lines |
modified crates/oxy-core/src/protocol.rs +2 −2
| … | 628 unchanged lines |
||
| 629 | 629 | let encoded = o::bytes(&pair[0], "change")?; |
|
| 630 | 630 | b.hashes |
|
| 631 | 631 | .push(crate::hash::hash_object(ObjectType::Change, encoded)); |
|
| 632 | - | let change = Change::decode_with(encoded, &mut |text, count| { |
|
| 632 | + | let change = Change::decode_with(encoded, &mut |text, count, form| { |
|
| 633 | 633 | let by_ref = by_ref.get_or_insert_with(|| { |
|
| 634 | 634 | b.objects |
|
| 635 | 635 | .iter() |
|
| … | 15 unchanged lines |
||
| 651 | 651 | for chunk in &blob.chunks { |
|
| 652 | 652 | content.extend_from_slice(get(chunk, ObjectType::Chunk)?); |
|
| 653 | 653 | } |
|
| 654 | - | crate::objects::text_lines(&content, count) |
|
| 654 | + | crate::objects::text_lines(&content, count, form) |
|
| 655 | 655 | })?; |
|
| 656 | 656 | b.changes.push((change, sigs_from(&pair[1])?)); |
|
| 657 | 657 | } |
|
| … | 470 unchanged lines |
modified crates/oxy-core/src/repo/fsck.rs +1 −1
| … | 89 unchanged lines |
||
| 90 | 90 | ObjectType::Blob => Blob::decode(body).map(|_| ()), |
|
| 91 | 91 | // Structure only: the text blobs are checked with the change graph. |
|
| 92 | 92 | ObjectType::Change => { |
|
| 93 | - | Change::decode_with(body, &mut |_, _| Ok(crate::lines::Lines::default())).map(|_| ()) |
|
| 93 | + | Change::decode_with(body, &mut |_, _, _| Ok(crate::lines::Lines::default())).map(|_| ()) |
|
| 94 | 94 | } |
|
| 95 | 95 | ObjectType::Checkpoint => crate::objects::Checkpoint::decode(body).map(|_| ()), |
|
| 96 | 96 | ObjectType::Op => match Op::decode(body) { |
|
| … | 391 unchanged lines |
modified crates/oxy-core/src/repo/sync.rs +2 −2
| … | 301 unchanged lines |
||
| 302 | 302 | let bytes = self.store.get_typed(h, ObjectType::Change)?; |
|
| 303 | 303 | let (skel, texts) = Change::decode_skeleton(&bytes, *h)?; |
|
| 304 | 304 | for t in texts { |
|
| 305 | - | if let crate::objects::TextRef::Blob(r, _) = t { |
|
| 305 | + | if let crate::objects::TextRef::Blob(r, _, _) = t { |
|
| 306 | 306 | let blob = Blob::decode(&self.store.get_typed(&r, ObjectType::Blob)?)?; |
|
| 307 | 307 | objects.insert(r); |
|
| 308 | 308 | objects.extend(blob.chunks); |
|
| … | 142 unchanged lines |
||
| 451 | 451 | } |
|
| 452 | 452 | } |
|
| 453 | 453 | for t in texts { |
|
| 454 | - | if let crate::objects::TextRef::Blob(r, _) = t { |
|
| 454 | + | if let crate::objects::TextRef::Blob(r, _, _) = t { |
|
| 455 | 455 | if !self.store.has(&r) { |
|
| 456 | 456 | return fail(format!( |
|
| 457 | 457 | "change {} came without its text blob {}", |
|
| … | 1282 unchanged lines |
modified crates/oxy-core/src/repo/tests.rs +1 −1
| … | 5801 unchanged lines |
||
| 5802 | 5802 | let other_text = texts |
|
| 5803 | 5803 | .iter() |
|
| 5804 | 5804 | .filter_map(|t| match t { |
|
| 5805 | - | crate::objects::TextRef::Blob(r, _) => Some(*r), |
|
| 5805 | + | crate::objects::TextRef::Blob(r, _, _) => Some(*r), |
|
| 5806 | 5806 | _ => None, |
|
| 5807 | 5807 | }) |
|
| 5808 | 5808 | .find(|r| content(r).windows(12).any(|w| w == b"other line 1")) |
|
| … | 367 unchanged lines |
modified crates/oxy-core/src/state.rs +3 −3
| … | 245 unchanged lines |
||
| 246 | 246 | let bases = refs.len(); |
|
| 247 | 247 | refs.extend(taken.iter().flat_map(|(_, d)| { |
|
| 248 | 248 | d.texts.iter().filter_map(|t| match t { |
|
| 249 | - | TextRef::Blob(r, _) => Some(*r), |
|
| 249 | + | TextRef::Blob(r, _, _) => Some(*r), |
|
| 250 | 250 | TextRef::Inline { .. } => None, |
|
| 251 | 251 | }) |
|
| 252 | 252 | })); |
|
| … | 28 unchanged lines |
||
| 281 | 281 | if let Edit::Insert { lines, .. } = e { |
|
| 282 | 282 | if lines.is_empty() { |
|
| 283 | 283 | *lines = match want.next() { |
|
| 284 | - | Some(TextRef::Blob(_, count)) => { |
|
| 284 | + | Some(TextRef::Blob(_, count, form)) => { |
|
| 285 | 285 | let content = texts.next().ok_or_else(|| { |
|
| 286 | 286 | ObjectError::Schema("text blobs: wrong count".into()) |
|
| 287 | 287 | })?; |
|
| 288 | - | crate::objects::text_lines(&content, count)? |
|
| 288 | + | crate::objects::text_lines(&content, count, form)? |
|
| 289 | 289 | } |
|
| 290 | 290 | Some(TextRef::Inline { change, start, end }) => { |
|
| 291 | 291 | let bytes = &changes[&change]; |
|
| … | 1672 unchanged lines |
modified crates/oxy-core/src/store.rs +4 −2
| … | 1053 unchanged lines |
||
| 1054 | 1054 | let bytes = self.get_typed(r, ObjectType::Change)?; |
|
| 1055 | 1055 | let mut parts = Vec::new(); |
|
| 1056 | 1056 | let mut failed = None; |
|
| 1057 | - | let decoded = Change::decode_with(&bytes, &mut |text, count| match self.read_text(text) { |
|
| 1057 | + | let decoded = Change::decode_with(&bytes, &mut |text, count, form| match self |
|
| 1058 | + | .read_text(text) |
|
| 1059 | + | { |
|
| 1058 | 1060 | Ok((blob, content)) => { |
|
| 1059 | 1061 | parts.push(*text); |
|
| 1060 | 1062 | parts.extend(blob.chunks); |
|
| 1061 | - | crate::objects::text_lines(&content, count) |
|
| 1063 | + | crate::objects::text_lines(&content, count, form) |
|
| 1062 | 1064 | } |
|
| 1063 | 1065 | Err(e) => { |
|
| 1064 | 1066 | let why = e.to_string(); |
|
| … | 821 unchanged lines |
modified docs/spec.md +5 −5
| … | 198 unchanged lines |
||
| 199 | 199 | ||
| 200 | 200 | ``` |
|
| 201 | 201 | Change { |
|
| 202 | - | version [0]: 1 or 2 |
|
| 202 | + | version [0]: 1, 2 or 3 |
|
| 203 | 203 | change_id [1]: bytes(16) // random, CSPRNG; see §6.3 |
|
| 204 | 204 | deps [2]: [ChangeHash] // sorted, unique; see §8 |
|
| 205 | 205 | author [3]: KeyId // ObjectRef of an identity object |
|
| … | 6 unchanged lines |
||
| 212 | 212 | } |
|
| 213 | 213 | ``` |
|
| 214 | 214 | ||
| 215 | - | Writers produce version 2. Versions 1 and 2 differ only in how an `Insert` holds its lines (§7.4): a version 1 change carries every inserted line inline, and a version 2 change stores each large insert as a **text blob** beside it, so no change grows with the size of the files it adds. Readers MUST accept both. A change keeps the version it was made in: rewriting its encoding would change its hash and invalidate its signatures. |
|
| 215 | + | Writers produce version 3. The versions differ only in how an `Insert` holds its lines (§7.4): a version 1 change carries every inserted line inline; versions 2 and 3 store each large insert as a **text blob** beside it, so no change grows with the size of the files it adds, and differ in what the text blob holds. Readers MUST accept all three. A change keeps the version it was made in: rewriting its encoding would change its hash and invalidate its signatures. |
|
| 216 | 216 | ||
| 217 | 217 | ### 6.2 Two identifiers |
|
| 218 | 218 | ||
| … | 47 unchanged lines |
||
| 266 | 266 | ### 7.4 Edits |
|
| 267 | 267 | ||
| 268 | 268 | ``` |
|
| 269 | - | Edit = Insert { node: NodeId, after: AtomRef, before: AtomRef, lines: [bytes] | text: ObjectRef, count: u32 } |
|
| 269 | + | Edit = Insert { node: NodeId, after: AtomRef, before: AtomRef, lines: [bytes] | text: ObjectRef, count: u32, cbor?: 1 } |
|
| 270 | 270 | | Delete { node: NodeId, atoms: [AtomRange] } |
|
| 271 | 271 | | Replace { node: NodeId, old: ObjectRef | null, new: ObjectRef } // opaque nodes only |
|
| 272 | 272 | AtomRef = START | END | AtomId |
|
| … | 2 unchanged lines |
||
| 275 | 275 | ||
| 276 | 276 | - `Insert` introduces `len(lines)` new atoms, indexed consecutively, and edges `after → first`, `last → before`. `after` and `before` MUST be adjacent in the author's view at authoring time; the pair is the insert's **anchor**. |
|
| 277 | 277 | - Atoms introduced by the same change are referenced with the all-zero "self" hash. This is how a change can insert a line and then insert another after it without knowing its own hash. |
|
| 278 | - | - **Text blobs.** In a version 2 change, an insert whose lines encode as a canonical CBOR array of byte strings to at least 4096 bytes MUST be stored as `text` and `count`: `text` is the blob (§13.2) whose content is exactly that encoding, and `count` is the number of lines, at least 1. A shorter insert MUST be inline. The rule is deterministic, so a change has one encoding and one hash, and identical inserts share their text blob. A reader rejects an inline insert that should have been stored, a text blob shorter than 4096 bytes, and a text blob whose lines do not number `count`. A text blob and its chunks belong to their change: they are carried with it by every transfer (§16.3), kept with it by garbage collection (§21), and checked with it by `fsck`. A change cannot be read without them, so they are never withheld from a partial clone (§13.5). |
|
| 278 | + | - **Text blobs.** In a version 2 or 3 change, an insert whose lines encode as a canonical CBOR array of byte strings to at least 4096 bytes MUST be stored as `text` and `count`: `text` is a blob (§13.2) holding the lines, and `count` is the number of lines, at least 1. In version 2 the blob's content is exactly that CBOR array. In version 3 it is the lines' bytes, joined, when splitting those bytes by the line rule (every line but the last ends in a newline and holds no other; §7.3) gives the lines back, as every insert a snapshot makes does; otherwise it is the CBOR array and the insert carries `cbor: 1` (key 7). A reader rejects a version 3 insert marked `cbor` whose lines the line rule would split back. The joined form stores a file's text as its bytes, so a text blob shares chunks with any blob of the same content and carries no per-line framing (a tenth of a tree's compressed text). A shorter insert MUST be inline. The rule is deterministic, so a change has one encoding and one hash, and identical inserts share their text blob. A reader rejects an inline insert that should have been stored, a text blob shorter than 4096 bytes, and a text blob whose lines do not number `count`. A text blob and its chunks belong to their change: they are carried with it by every transfer (§16.3), kept with it by garbage collection (§21), and checked with it by `fsck`. A change cannot be read without them, so they are never withheld from a partial clone (§13.5). |
|
| 279 | 279 | - `Replace` is the only edit for `opaque` nodes. `old` is the blob currently expected; two concurrent `Replace` operations with the same `old` and different `new` conflict (§9.3). |
|
| 280 | 280 | ||
| 281 | 281 | ### 7.5 Opaque nodes |
|
| … | 695 unchanged lines |
||
| 977 | 977 | ||
| 978 | 978 | --- |
|
| 979 | 979 | ||
| 980 | - | *Changelog.* Draft 0.8r (2026-09-29): `oxy gc` repacks into one pack at a higher compression level (§13.4); an import brings a deleted directory back rather than making a second node for its path, and retires a kept node's other placements (§23); graph keys at commit, lazy text in history-only loads, and a clean `status` from the manifest (§12.4); a parent checkout never moves a subrepo's bookmark backwards, and the pin is read from the nested working copy (§7.8); `bookmark create --upto/--only` (§10.1); the working-copy lock covers every command that writes the working copy; a workspace changed by two concurrent operations is an error (§12.1). Draft 0.8q (2026-09-29): change format version 2 stores large inserts as text blobs, so a change does not grow with the files it adds (§6.1, §7.4); no object may exceed 1 GiB (§13.2). Draft 0.8p (2026-09-29): Git import derives new NodeIds from the commit, making imports deterministic (§23). Draft 0.8o (2026-09-29): execution receipts, the `receipt` object and attestation (§14.6). Draft 0.8n (2026-09-29): line history, `oxy log -L` (§27). Draft 0.8m (2026-09-29): resolving replace conflicts (§9.3). Draft 0.8l (2026-09-28): deleting a reference on a remote; first serials from the clock (§16.4). Draft 0.8k (2026-09-28): the manifest records graph keys, so unchanged files are neither read nor rendered (§12.4). Draft 0.8j (2026-09-28): checkpoint payload version 2, file graphs by reference (§11). |
|
| 980 | + | *Changelog.* Draft 0.8r (2026-09-29): change format version 3, text blobs as the lines' bytes (§6.1, §7.4); `oxy gc` repacks into one pack at a higher compression level (§13.4); an import brings a deleted directory back rather than making a second node for its path, and retires a kept node's other placements (§23); graph keys at commit, lazy text in history-only loads, and a clean `status` from the manifest (§12.4); a parent checkout never moves a subrepo's bookmark backwards, and the pin is read from the nested working copy (§7.8); `bookmark create --upto/--only` (§10.1); the working-copy lock covers every command that writes the working copy; a workspace changed by two concurrent operations is an error (§12.1). Draft 0.8q (2026-09-29): change format version 2 stores large inserts as text blobs, so a change does not grow with the files it adds (§6.1, §7.4); no object may exceed 1 GiB (§13.2). Draft 0.8p (2026-09-29): Git import derives new NodeIds from the commit, making imports deterministic (§23). Draft 0.8o (2026-09-29): execution receipts, the `receipt` object and attestation (§14.6). Draft 0.8n (2026-09-29): line history, `oxy log -L` (§27). Draft 0.8m (2026-09-29): resolving replace conflicts (§9.3). Draft 0.8l (2026-09-28): deleting a reference on a remote; first serials from the clock (§16.4). Draft 0.8k (2026-09-28): the manifest records graph keys, so unchanged files are neither read nor rendered (§12.4). Draft 0.8j (2026-09-28): checkpoint payload version 2, file graphs by reference (§11). |
|
| 981 | 981 | ||
| 982 | 982 | Draft 0.8i (2026-09-28): portability rules for exec bits and path names (§7.7); found by verifying a real Git import commit by commit: deletions depend on the inserters of the lines right after them, and replacements on the change that set their old blob (§8.1); replace records are superseded causally, so a reverted binary keeps its blob (§9.3); merges resolve opaque conflicts to the commit's blob, and `oxy git verify` (§23). Draft 0.8h (2026-09-28): `oxy fsck` (§21.1); durability rules and the operation head as commit point (§13.6); the operation head moves by compare-and-swap and concurrent operations are merged (§12.1); `oxy doctor` (§21.2). Draft 0.8g (2026-09-27): delegates of a rule's identities may sign reference attestations (§14.5); local-path pushes bind unbound repositories like HTTP ones. Draft 0.8f (2026-09-27): challenge signatures (§14.3) for hosts. Draft 0.8e (2026-09-27): anonymous changes (zero author) are authenticated by a seal from any key, so imported and pre-identity history can be pushed under a policy (§14.4). Draft 0.8d (2026-09-27): unbound hosted repositories adopt the first pusher's id (§16.1). Draft 0.8c (2026-09-27): bare repositories (`oxy init --bare`, §12.2: no workspace, no working copy; hosting) and the reference server as an embeddable library (§15.1). Draft 0.8b (2026-09-27): the machine interface for editors (§24): `root`, `cat`, `version`, `hunks`, `discard`, and `--json` on the reporting commands. |
|
| 983 | 983 | ||
| … | 61 unchanged lines |