Commit takes a status's snapshot from the cache; the stat check lends its stats to the snapshot (refs #41) e226a6cc2f71
snapshot_from_cache builds the snapshot a commit needs while the status cache holds: the stored working-copy change, its base from the state index, the .oxyattributes files from disk. When no directory changed, the manifest check's stats become the listing of the snapshot that follows, racy files included, so a snapshot after an edit lists no directory. Fixes the status cache this evening introduced: a mode change moves neither size nor time, so the manifest was not rewritten when only an edited file's executable bit changed, and reverting it replayed the old status. The test build's cross-check found it. On the Linux tree: commit after a status 0.36 to 0.44 s (from 0.68 s; Git 0.14 to 0.17 s). 18 fsyncs are now 210 ms of a commit.
- Author
- Andy McCall @andymccall <[email protected]>
- When
- · 2026-10-01 20:48 UTC
- Signature
- ✔ signed by Andy McCall
- Hash
e226a6cc2f713b7b3f71469ba12d5dcb5543e151909a781928f90a4ae8df95e6- Depends on
- 10b4fd430eab 135212de5370 1d360c91a289 1ec3f008e96a 3604179679eb 6caacfff1d4f 77445673174b 86e1824f2d0b 8fd78eb52ad2 982864068a59 c07406bcc237 c6aa5bd9d1c8 f9466021286a
- Depended on by
- 236f35c0baba 0389c3e82f6c e96a430c78f3 33472ccc08cc b30a880fa409
- In bookmarks
- agents-in-readme bare-hashes carried-not-removed carry-identical doctor-trust identical-sides keyless-tags main merge-remote-bookmark only-help-103 paths-for-resolutions-and-amend push-notes show-stat subrepo-pin-state switch-says
- In tags
- v0.10.0 v0.11.0 v0.3.0 v0.3.1 v0.4.0 v0.4.1 v0.4.2 v0.4.3 v0.4.4 v0.4.5 v0.4.6 v0.5.0 v0.5.1 v0.6.0 v0.7.0 v0.8.0 v0.9.0
- Edits
- 16 atoms, 0 tree operations, 5 files
Browse the tree at this change
modified CHANGELOG.md +1 −1
| … | 1 unchanged lines |
||
| 2 | 2 | ||
| 3 | 3 | ## Unreleased |
|
| 4 | 4 | ||
| 5 | - | - **Status twice as fast as Git on the Linux tree (#41).** With uncommitted work, `oxy status` answers from a status cache while every file is as the last snapshot left it, and the manifest keeps every directory's time, so the tree is listed only when a directory's entries changed. One edited file is 0.10 to 0.12 s (from 0.44 s; Git 0.22 s) and a clean status 0.09 to 0.12 s (Git 0.20 to 0.26 s), in 80 MB. `oxy diff` renders the working copy from the cache too (0.20 s, from 0.41 s; Git 0.08 s). A file written within the racy window is now hashed by the prompt's check too, as it was by status. |
|
| 5 | + | - **Status twice as fast as Git on the Linux tree (#41).** With uncommitted work, `oxy status` answers from a status cache while every file is as the last snapshot left it, and the manifest keeps every directory's time, so the tree is listed only when a directory's entries changed. One edited file is 0.10 to 0.12 s (from 0.44 s; Git 0.22 s) and a clean status 0.09 to 0.12 s (Git 0.20 to 0.26 s), in 80 MB. `oxy diff` renders the working copy from the cache too (0.20 s, from 0.41 s; Git 0.08 s). A commit after a status takes that status's snapshot instead of walking the tree again (0.37 s, from 0.68 s; Git 0.15 s). A file written within the racy window is now hashed by the prompt's check too, as it was by status. |
|
| 6 | 6 | ||
| 7 | 7 | - **Quieter notes (#55).** Notes start with a dimmed `›` instead of `ℹ`, which many terminal fonts draw as a boxed or emoji-style "i". A spinner's final line no longer shows two ticks or runs into the next line. |
|
| 8 | 8 | ||
| … | 53 unchanged lines |
modified crates/oxy-core/src/repo/prompt.rs +85 −19
| … | 664 unchanged lines |
||
| 665 | 665 | /// walk of the working copy when it does. |
|
| 666 | 666 | fn disk_differs(&self, manifest: &Manifest, keep: bool) -> Result<bool> { |
|
| 667 | 667 | // A file written within the racy window can change without its size |
|
| 668 | - | // or time moving (a coarse clock): those entries are hashed. |
|
| 669 | - | if self.racy_change(manifest).is_some() { |
|
| 668 | + | // or time moving (a coarse clock): those entries are hashed. One |
|
| 669 | + | // that changed still lets the directory check below hand its stats |
|
| 670 | + | // to the snapshot that follows. |
|
| 671 | + | let racy = self.racy_change(manifest).is_some(); |
|
| 672 | + | if let Some(differs) = self.disk_differs_by_dirs_keeping(manifest, keep, racy) { |
|
| 673 | + | return Ok(differs || racy); |
|
| 674 | + | } |
|
| 675 | + | if racy { |
|
| 670 | 676 | return Ok(true); |
|
| 671 | - | } |
|
| 672 | - | if let Some(differs) = self.disk_differs_by_dirs(manifest) { |
|
| 673 | - | return Ok(differs); |
|
| 674 | 677 | } |
|
| 675 | 678 | self.disk_differs_listing(manifest, keep) |
|
| 676 | 679 | } |
|
| … | 4 unchanged lines |
||
| 681 | 684 | /// recorded (and none within the racy window) the files are the |
|
| 682 | 685 | /// manifest's, and each is checked by its stat. `None` when a |
|
| 683 | 686 | /// directory changed or the manifest has none: the tree must be listed. |
|
| 687 | + | #[cfg(test)] |
|
| 684 | 688 | pub(crate) fn disk_differs_by_dirs(&self, manifest: &Manifest) -> Option<bool> { |
|
| 689 | + | self.disk_differs_by_dirs_keeping(manifest, false, false) |
|
| 690 | + | } |
|
| 691 | + | ||
| 692 | + | /// [`Repo::disk_differs_by_dirs`]; with `keep`, when a file differs |
|
| 693 | + | /// (or `racy` says one does), the stats taken become the listing of the |
|
| 694 | + | /// snapshot that follows: no directory's entries changed, so the |
|
| 695 | + | /// manifest's files are the tree's. |
|
| 696 | + | fn disk_differs_by_dirs_keeping( |
|
| 697 | + | &self, |
|
| 698 | + | manifest: &Manifest, |
|
| 699 | + | keep: bool, |
|
| 700 | + | racy: bool, |
|
| 701 | + | ) -> Option<bool> { |
|
| 685 | 702 | use rayon::prelude::*; |
|
| 686 | 703 | if manifest.dirs.is_empty() || !self.config.sparse_paths.is_empty() { |
|
| 687 | 704 | return None; |
|
| … | 6 unchanged lines |
||
| 694 | 711 | }); |
|
| 695 | 712 | if !dirs_same { |
|
| 696 | 713 | return None; |
|
| 714 | + | } |
|
| 715 | + | // Each tracked path as it is now: a file's stat, a symlink's target. |
|
| 716 | + | enum Now { |
|
| 717 | + | File(PathBuf, fs::Metadata), |
|
| 718 | + | Link(Vec<u8>), |
|
| 697 | 719 | } |
|
| 698 | - | let differs = manifest.entries.par_iter().any(|(rel, me)| { |
|
| 699 | - | let path = root.join(rel.as_str()); |
|
| 700 | - | let Ok(m) = fs::symlink_metadata(&path) else { |
|
| 701 | - | return true; |
|
| 702 | - | }; |
|
| 720 | + | let differs_from = |me: &ManifestEntry, path: &Path, now: &Now| match now { |
|
| 721 | + | Now::Link(t) => *blake3::hash(t).as_bytes() != me.hash, |
|
| 722 | + | Now::File(_, m) => { |
|
| 723 | + | m.len() != me.size |
|
| 724 | + | || super::is_executable(m) != me.exec |
|
| 725 | + | || (mtime_ns(m) != me.mtime_ns |
|
| 726 | + | && !fs::read(path).is_ok_and(|b| *blake3::hash(&b).as_bytes() == me.hash)) |
|
| 727 | + | } |
|
| 728 | + | }; |
|
| 729 | + | let stat = |rel: &str| -> Option<Now> { |
|
| 730 | + | let path = root.join(rel); |
|
| 731 | + | let m = fs::symlink_metadata(&path).ok()?; |
|
| 703 | 732 | if m.file_type().is_symlink() { |
|
| 704 | - | return !fs::read_link(&path).is_ok_and(|t| { |
|
| 705 | - | *blake3::hash(t.to_string_lossy().as_bytes()).as_bytes() == me.hash |
|
| 706 | - | }); |
|
| 733 | + | let t = fs::read_link(&path).ok()?; |
|
| 734 | + | Some(Now::Link(t.to_string_lossy().as_bytes().to_vec())) |
|
| 735 | + | } else if m.is_file() { |
|
| 736 | + | Some(Now::File(path, m)) |
|
| 737 | + | } else { |
|
| 738 | + | None |
|
| 707 | 739 | } |
|
| 708 | - | !m.is_file() |
|
| 709 | - | || m.len() != me.size |
|
| 710 | - | || super::is_executable(&m) != me.exec |
|
| 711 | - | || (mtime_ns(&m) != me.mtime_ns |
|
| 712 | - | && !fs::read(&path).is_ok_and(|b| *blake3::hash(&b).as_bytes() == me.hash)) |
|
| 713 | - | }); |
|
| 740 | + | }; |
|
| 741 | + | let differs = if !keep { |
|
| 742 | + | manifest |
|
| 743 | + | .entries |
|
| 744 | + | .par_iter() |
|
| 745 | + | .any(|(rel, me)| match stat(rel) { |
|
| 746 | + | Some(now) => differs_from(me, &root.join(rel.as_str()), &now), |
|
| 747 | + | None => true, |
|
| 748 | + | }) |
|
| 749 | + | } else { |
|
| 750 | + | let now: Vec<(&String, Option<Now>)> = manifest |
|
| 751 | + | .entries |
|
| 752 | + | .par_iter() |
|
| 753 | + | .map(|(rel, _)| (rel, stat(rel))) |
|
| 754 | + | .collect(); |
|
| 755 | + | // A path gone or no longer a file: its directory moved, unless |
|
| 756 | + | // within a clock tick; list the tree to be sure. |
|
| 757 | + | if now.iter().any(|(_, n)| n.is_none()) { |
|
| 758 | + | return None; |
|
| 759 | + | } |
|
| 760 | + | let differs = now.par_iter().any(|(rel, n)| { |
|
| 761 | + | let me = &manifest.entries[*rel]; |
|
| 762 | + | differs_from(me, &root.join(rel.as_str()), n.as_ref().unwrap()) |
|
| 763 | + | }); |
|
| 764 | + | if differs || racy { |
|
| 765 | + | let mut listing = super::walk::Listing { |
|
| 766 | + | dirs: manifest.dirs.clone(), |
|
| 767 | + | nested: manifest.nested.iter().map(|n| root.join(n)).collect(), |
|
| 768 | + | ..Default::default() |
|
| 769 | + | }; |
|
| 770 | + | for (rel, n) in now { |
|
| 771 | + | match n.unwrap() { |
|
| 772 | + | Now::File(path, m) => listing.files.push((rel.clone(), path, m)), |
|
| 773 | + | Now::Link(t) => listing.symlinks.push((rel.clone(), t)), |
|
| 774 | + | } |
|
| 775 | + | } |
|
| 776 | + | *self.listing_cache.borrow_mut() = Some(listing); |
|
| 777 | + | } |
|
| 778 | + | differs |
|
| 779 | + | }; |
|
| 714 | 780 | if !differs { |
|
| 715 | 781 | // Nested repositories the manifest has no entry for are not |
|
| 716 | 782 | // subrepos: nothing in them is tracked (oxygen#43). |
|
| … | 182 unchanged lines |
modified crates/oxy-core/src/repo/snapshot.rs +53 −2
| … | 621 unchanged lines |
||
| 622 | 622 | .ok() |
|
| 623 | 623 | .map(|md| (md.len(), super::prompt::mtime_ns(&md))) |
|
| 624 | 624 | }); |
|
| 625 | - | me.hash != hash || stat != Some((me.size, me.mtime_ns)) |
|
| 625 | + | // The executable bit too: a mode change moves |
|
| 626 | + | // neither size nor time. |
|
| 627 | + | me.hash != hash || me.exec != e.exec || stat != Some((me.size, me.mtime_ns)) |
|
| 626 | 628 | })) |
|
| 627 | 629 | } |
|
| 628 | 630 | }; |
|
| … | 173 unchanged lines |
||
| 802 | 804 | selection: &Selection, |
|
| 803 | 805 | ) -> Result<ChangeHash> { |
|
| 804 | 806 | let _wc = self.lock_working_copy()?; |
|
| 805 | - | let mut snap = self.snapshot_as("oxy commit", true, true)?; |
|
| 807 | + | let mut snap = match self.snapshot_from_cache()? { |
|
| 808 | + | Some(snap) => snap, |
|
| 809 | + | None => self.snapshot_as("oxy commit", true, true)?, |
|
| 810 | + | }; |
|
| 806 | 811 | if snap.wc.tree_ops.is_empty() && snap.wc.edits.is_empty() { |
|
| 807 | 812 | return fail("nothing to commit: the working copy matches its base"); |
|
| 808 | 813 | } |
|
| … | 732 unchanged lines |
||
| 1541 | 1546 | None => self.working_copy_with_base()?, |
|
| 1542 | 1547 | }; |
|
| 1543 | 1548 | Ok(diff_files_of(&c, &base, &base_snapshot)) |
|
| 1549 | + | } |
|
| 1550 | + | ||
| 1551 | + | /// The snapshot a command would take, without taking it, when the |
|
| 1552 | + | /// status cache holds: every file is as the last snapshot left it, so |
|
| 1553 | + | /// that snapshot's working-copy change is the answer. Its base comes |
|
| 1554 | + | /// from the state index, the `.oxyattributes` files from disk. `None` |
|
| 1555 | + | /// when a snapshot must be taken, and when there is nothing to commit |
|
| 1556 | + | /// (the snapshot then says so). A conflicted base never gets here: the |
|
| 1557 | + | /// cache does not hold for one. |
|
| 1558 | + | pub(crate) fn snapshot_from_cache(&self) -> Result<Option<SnapshotResult>> { |
|
| 1559 | + | let Some(statuses) = self.status_by_manifest()? else { |
|
| 1560 | + | return Ok(None); |
|
| 1561 | + | }; |
|
| 1562 | + | if statuses.is_empty() { |
|
| 1563 | + | return Ok(None); |
|
| 1564 | + | } |
|
| 1565 | + | let _t = crate::timing::phase("snapshot: from the status cache"); |
|
| 1566 | + | let (wc_hash, wc) = self.working_copy()?; |
|
| 1567 | + | let base = self.load_state_indexed(&wc.deps)?; |
|
| 1568 | + | let (base, base_snapshot) = self.render_base_for(&wc, base)?; |
|
| 1569 | + | let attribute_files: Vec<(String, Vec<u8>)> = match self.read_manifest()? { |
|
| 1570 | + | Some((_, _, m)) => m |
|
| 1571 | + | .entries |
|
| 1572 | + | .keys() |
|
| 1573 | + | .filter(|p| is_attributes(p)) |
|
| 1574 | + | .filter_map(|p| fs::read(self.root.join(p)).ok().map(|b| (p.clone(), b))) |
|
| 1575 | + | .collect(), |
|
| 1576 | + | None => Vec::new(), |
|
| 1577 | + | }; |
|
| 1578 | + | let attributes = crate::attributes::Attributes::from_files( |
|
| 1579 | + | attribute_files |
|
| 1580 | + | .iter() |
|
| 1581 | + | .map(|(p, b)| (p.as_str(), b.as_slice())), |
|
| 1582 | + | ); |
|
| 1583 | + | Ok(Some(SnapshotResult { |
|
| 1584 | + | wc_hash, |
|
| 1585 | + | wc, |
|
| 1586 | + | base, |
|
| 1587 | + | base_snapshot, |
|
| 1588 | + | statuses, |
|
| 1589 | + | changed: false, |
|
| 1590 | + | resolves: false, |
|
| 1591 | + | contributors: Vec::new(), |
|
| 1592 | + | attributes, |
|
| 1593 | + | untracked_nested: self.untracked_nested.borrow().clone(), |
|
| 1594 | + | })) |
|
| 1544 | 1595 | } |
|
| 1545 | 1596 | ||
| 1546 | 1597 | /// The working-copy change with its base state and the base's |
|
| … | 1259 unchanged lines |
modified crates/oxy-core/src/repo/tests.rs +39 −0
| … | 6032 unchanged lines |
||
| 6033 | 6033 | assert_eq!(repo.disk_differs_by_dirs(&m), None); |
|
| 6034 | 6034 | } |
|
| 6035 | 6035 | ||
| 6036 | + | /// oxygen#41: a commit right after a status takes the snapshot the status |
|
| 6037 | + | /// left (the status cache holds) rather than walking again, and commits |
|
| 6038 | + | /// exactly what a snapshot would have. |
|
| 6039 | + | #[test] |
|
| 6040 | + | fn a_commit_after_a_status_uses_the_cached_snapshot() { |
|
| 6041 | + | hermetic(); |
|
| 6042 | + | let dir = tempfile::tempdir().unwrap(); |
|
| 6043 | + | let root = dir.path(); |
|
| 6044 | + | let repo = Repo::init(root).unwrap(); |
|
| 6045 | + | write(root, "a.txt", b"a\nb\n"); |
|
| 6046 | + | write(root, ".oxyattributes", b"*.bin binary\n"); |
|
| 6047 | + | repo.commit("base", &[]).unwrap(); |
|
| 6048 | + | write(root, "a.txt", b"a\nB\n"); |
|
| 6049 | + | write(root, "new.txt", b"new\n"); |
|
| 6050 | + | let snap = repo.snapshot_status("oxy status").unwrap(); |
|
| 6051 | + | let cached = repo |
|
| 6052 | + | .snapshot_from_cache() |
|
| 6053 | + | .unwrap() |
|
| 6054 | + | .expect("the cache holds"); |
|
| 6055 | + | assert_eq!(cached.wc_hash, snap.wc_hash); |
|
| 6056 | + | assert_eq!(cached.statuses, snap.statuses); |
|
| 6057 | + | assert!( |
|
| 6058 | + | !cached.attributes.is_empty(), |
|
| 6059 | + | "the .oxyattributes file was read" |
|
| 6060 | + | ); |
|
| 6061 | + | let h = repo.commit("edit", &[]).unwrap(); |
|
| 6062 | + | let c = repo.store().get_change(&h).unwrap(); |
|
| 6063 | + | assert_eq!( |
|
| 6064 | + | c.tree_ops.len() + c.edits.len(), |
|
| 6065 | + | snap.wc.tree_ops.len() + snap.wc.edits.len() |
|
| 6066 | + | ); |
|
| 6067 | + | assert_eq!( |
|
| 6068 | + | repo.status_by_manifest().unwrap(), |
|
| 6069 | + | Some(vec![]), |
|
| 6070 | + | "clean after the commit" |
|
| 6071 | + | ); |
|
| 6072 | + | assert_eq!(fs::read(root.join("a.txt")).unwrap(), b"a\nB\n"); |
|
| 6073 | + | } |
|
| 6074 | + | ||
| 6036 | 6075 | /// oxygen#35: a history-only state keeps no inserted text. It reads a |
|
| 6037 | 6076 | /// stored insert's text blob only when it builds that file's graph, and |
|
| 6038 | 6077 | /// builds the same graph a full load does. |
|
| … | 708 unchanged lines |
modified docs/benchmarks.md +19 −0
| … | 797 unchanged lines |
||
| 798 | 798 | the stat check 78 ms (Git stats the same files), the state index 58 ms |
|
| 799 | 799 | (39 of them building the tree view), the base's full file map 21 ms. |
|
| 800 | 800 | ||
| 801 | + | ### Commit, one file (2026-10-01, oxygen#41) |
|
| 802 | + | ||
| 803 | + | `~/bench/commit-vs-git.sh`: commits go forward and both repositories are |
|
| 804 | + | put back at the end (an `oxy undo` between runs dropped the manifest, and |
|
| 805 | + | its rehash of the tree evicted the page cache Git's runs relied on, which |
|
| 806 | + | made both sides' numbers meaningless). Load average 3 to 4. |
|
| 807 | + | ||
| 808 | + | | | before | after | Git `commit -am` | |
|
| 809 | + | |---|---:|---:|---:| |
|
| 810 | + | | commit after a status | 0.68 s | **0.36 to 0.44 s** | 0.14 to 0.17 s | |
|
| 811 | + | | commit straight after the edit | 0.68 s | 0.62 to 0.69 s | 0.14 to 0.17 s | |
|
| 812 | + | ||
| 813 | + | After a status, the commit takes the status's snapshot from the cache. |
|
| 814 | + | Straight after an edit, the stat check now hands its stats to the |
|
| 815 | + | snapshot as its listing (8 ms instead of 122 ms), but checks every file |
|
| 816 | + | rather than stopping at the first difference, so the gain is small. What |
|
| 817 | + | remains is mostly durability: 18 fsyncs take 210 ms of the commit (one |
|
| 818 | + | alone 120 ms). Git does not sync loose objects by default. |
|
| 819 | + |