Readers never wait: one head per handle, retry on a miss, retire before deleting 767ac6cd0545
oxygenhub#249: pages stalled for up to 43 s behind a repository's gc or push, because reads were only safe under the repository's lock. The store now gives readers what they need to read alongside writers: - A handle reads the operation head once and keeps it (its own operations move it; refresh() reads again), so one request never mixes two views. Compare-and-swap is unchanged: a stale handle's operation merges with the newer head. A push, fsck, a strata mirror's sync and taking the working-copy lock read the head afresh. - A read that misses reloads the pack list and tries once more (an object a deleted pack held, a dictionary gone, a pack written since); has() re-lists at most once a second. Loading the pack list passes over an index that vanished between listing and loading, so a repository always opens. - A repack retires the packs and loose objects it replaces in a pack-retired record instead of deleting them; new readers don't load retired packs, and gc (and autopack) deletes what was retired over an hour before. A retired loose object no live pack holds is kept. - oxy-strata forgets its cached pack listing and misses for a key both before and after writing it, so a listing taken mid-write is not kept stale for 30 s. Tests for each race the survey found (a reader across a repack, an index gone at open, a spilled push, one head per handle, the hour, the stale listing); 14 existing tests that hold one handle across another's push now refresh it. oxy gc reports what it deleted or kept for readers. Refs andymccall/oxygenhub#249
- Author
- Andy McCall @andymccall <[email protected]>
- via
- agent
agent:claude@andymccall <[email protected]> · keyless, under your grant for changes,refs,seal · key f3704baa843e
- When
- · 2026-10-18 14:22 UTC
- Signature
- ◈ sealed by agent:claude@andymccall
- Hash
767ac6cd0545ed1a1ef3f45bbfdd1bebcff222df95616f20491d0bb68c188570- Depends on
- 092457020692 132b884b9a8d 135212de5370 19555025a2ac 1e0fbb47ab90 1ec3f008e96a 2863456a0f6d 28fc23d7706a 2b872d074ddb 387335ff7f4d 4e88e774777a 530a0fc650a2 54cd08a1e7bf 5b1114a04905 5cde7882d767 648189388f1a 6caacfff1d4f 7421309f5372 75167672ebcd 7aa2dc9369e3 7b9809e67c3d 7d9dab131937 7e0e8e5ad882 83e0677b07fe 84b9e9ac0ab4 87c948720343 8b53b935dd03 8d80356371c5 8f10a07a1dbe 982864068a59 98a1620a3840 9ae4d15a0e4b 9d580f2132f3 9f9e72b93e4e a17fc622ca43 a67914bced23 a9f686edf60b afaba5f1306d b689779e9e74 ba160f60219e c9369ad8b1b1 d02e545dc265 da652d6392d8 dd21fd82687d e5d9b0513f3e f2f91ceb0e9f f8c695764c95 fce44ae9ad29 ff0af1ee299d ff1a64f81c1c
- Depended on by
- b5e98094eb8f 2452b4407ab7 795b8b502ded 1ec6fdbc6bcb ef7d4ea6f33b 8b3c9b2ba502
- In bookmarks
- carried-not-removed doctor-trust identical-sides keyless-tags main merge-remote-bookmark paths-for-resolutions-and-amend push-notes show-stat subrepo-pin-state
- In tags
- v0.10.0 v0.11.0 v0.5.0 v0.5.1 v0.6.0 v0.7.0 v0.8.0 v0.9.0
- Edits
- 124 atoms, 0 tree operations, 12 files
Browse the tree at this change
modified crates/oxy-core/src/repo/fsck.rs +2 −0
| … | 108 unchanged lines |
||
| 109 | 109 | impl Repo { |
|
| 110 | 110 | /// Check the whole repository; see the module documentation. |
|
| 111 | 111 | pub fn fsck(&self) -> Result<FsckReport> { |
|
| 112 | + | // The head as it is now, as the operations pass reads it. |
|
| 113 | + | self.forget_head(); |
|
| 112 | 114 | let mut rep = FsckReport::default(); |
|
| 113 | 115 | let bad = { |
|
| 114 | 116 | let _t = crate::timing::phase("fsck: objects"); |
|
| … | 398 unchanged lines |
modified crates/oxy-core/src/repo/gc.rs +16 −3
| … | 9 unchanged lines |
||
| 10 | 10 | //! that far back may fail once its objects are gone. |
|
| 11 | 11 | ||
| 12 | 12 | use super::*; |
|
| 13 | - | use crate::store::{PackReport, PruneReport}; |
|
| 13 | + | use crate::store::{PackReport, PruneReport, RetiredReport}; |
|
| 14 | 14 | use std::collections::HashSet; |
|
| 15 | 15 | ||
| 16 | 16 | #[derive(Debug, Clone, Default, PartialEq, Eq)] |
|
| … | 3 unchanged lines |
||
| 20 | 20 | pub reachable: usize, |
|
| 21 | 21 | /// Packs folded into the one a full repack writes. |
|
| 22 | 22 | pub merged_packs: usize, |
|
| 23 | + | /// Packs and loose objects earlier repacks retired, deleted now that |
|
| 24 | + | /// [`crate::store::RETIRED_FOR`] has passed (oxygenhub#249). |
|
| 25 | + | pub retired: RetiredReport, |
|
| 23 | 26 | } |
|
| 24 | 27 | ||
| 25 | 28 | impl Repo { |
|
| 26 | - | /// Prune unreachable objects older than the grace period (`now` |
|
| 27 | - | /// ignores it), then repack everything into one pack (spec §13.4). |
|
| 29 | + | /// Delete what earlier repacks retired more than |
|
| 30 | + | /// [`crate::store::RETIRED_FOR`] ago, prune unreachable objects older |
|
| 31 | + | /// than the grace period (`now` ignores it, but not the retired |
|
| 32 | + | /// files' hour), then repack everything into one pack (spec §13.4). |
|
| 33 | + | /// The packs and loose objects this repack replaces are retired in |
|
| 34 | + | /// turn, for the next collection. |
|
| 28 | 35 | pub fn gc(&self, now: bool) -> Result<GcReport> { |
|
| 29 | 36 | self.gc_with(now, &mut |_| {}) |
|
| 30 | 37 | } |
|
| … | 14 unchanged lines |
||
| 45 | 52 | extra: &[ObjectRef], |
|
| 46 | 53 | progress: crate::store::Progress, |
|
| 47 | 54 | ) -> Result<GcReport> { |
|
| 55 | + | // Retired files first: the readers that used them have had their |
|
| 56 | + | // hour, and what replaced them is in the packs read from here on. |
|
| 57 | + | let retired = self |
|
| 58 | + | .store |
|
| 59 | + | .expire_retired(crate::store::RETIRED_FOR, progress)?; |
|
| 48 | 60 | progress("finding what is reachable"); |
|
| 49 | 61 | let mut keep = self.reachable()?; |
|
| 50 | 62 | for r in extra { |
|
| … | 22 unchanged lines |
||
| 73 | 85 | pruned, |
|
| 74 | 86 | reachable: keep.len(), |
|
| 75 | 87 | merged_packs, |
|
| 88 | + | retired, |
|
| 76 | 89 | }) |
|
| 77 | 90 | } |
|
| 78 | 91 | ||
| … | 134 unchanged lines |
modified crates/oxy-core/src/repo/hosted.rs +6 −1
| … | 631 unchanged lines |
||
| 632 | 632 | user.delegate_add(agent_id.clone(), Scope::CHANGES, None) |
|
| 633 | 633 | .unwrap(); |
|
| 634 | 634 | assert!(user.push("origin", false).unwrap().iter().all(|x| x.ok)); |
|
| 635 | + | host.refresh().unwrap(); |
|
| 635 | 636 | let owner = { |
|
| 636 | 637 | let _ = owner; |
|
| 637 | 638 | user.config.author_key.unwrap() |
|
| … | 23 unchanged lines |
||
| 661 | 662 | // The agent cannot push: its attestation of the reference is |
|
| 662 | 663 | // refused under the policy. The owner can. |
|
| 663 | 664 | let r = as_agent.push("origin", false).unwrap(); |
|
| 665 | + | host.refresh().unwrap(); |
|
| 664 | 666 | assert!(r.iter().all(|x| !x.ok), "{r:?}"); |
|
| 665 | 667 | assert!(!host.bookmark_heads().unwrap().contains(&h)); |
|
| 666 | 668 | // The owner's own handle (its key directory was fixed at `setup`; |
|
| 667 | - | // the environment's `OXY_HOME` may have moved since). |
|
| 669 | + | // the environment's `OXY_HOME` may have moved since), which reads |
|
| 670 | + | // the head again to see the agent's commit. |
|
| 671 | + | user.refresh().unwrap(); |
|
| 668 | 672 | let r = user.push("origin", false).unwrap(); |
|
| 673 | + | host.refresh().unwrap(); |
|
| 669 | 674 | assert!(r.iter().all(|x| x.ok), "{r:?}"); |
|
| 670 | 675 | assert!(host.bookmark_heads().unwrap().contains(&h)); |
|
| 671 | 676 | assert!(matches!( |
|
| … | 133 unchanged lines |
modified crates/oxy-core/src/repo/mod.rs +45 −3
| … | 279 unchanged lines |
||
| 280 | 280 | /// `--at-op`: read the repository as it was after this operation; |
|
| 281 | 281 | /// recording an operation is refused. |
|
| 282 | 282 | at_op: Option<ObjectRef>, |
|
| 283 | + | /// The operation head this handle reads (oxygenhub#249): read once, on |
|
| 284 | + | /// first use, and moved only by this handle's own operations or by |
|
| 285 | + | /// [`Repo::refresh`], so one command or request never mixes two heads. |
|
| 286 | + | head_seen: std::cell::Cell<Option<ObjectRef>>, |
|
| 283 | 287 | /// The directories (with their times) and nested repositories of the |
|
| 284 | 288 | /// last walk, for the manifest written from it. |
|
| 285 | 289 | pub(crate) walked_dirs: std::cell::RefCell<Option<WalkedDirs>>, |
|
| … | 90 unchanged lines |
||
| 376 | 380 | let repo = Repo { |
|
| 377 | 381 | last_change: Default::default(), |
|
| 378 | 382 | at_op: None, |
|
| 383 | + | head_seen: Default::default(), |
|
| 379 | 384 | walked_dirs: Default::default(), |
|
| 380 | 385 | listed_dirs: Default::default(), |
|
| 381 | 386 | git_export_skipped: Default::default(), |
|
| … | 59 unchanged lines |
||
| 441 | 446 | let repo = Repo { |
|
| 442 | 447 | last_change: Default::default(), |
|
| 443 | 448 | at_op: None, |
|
| 449 | + | head_seen: Default::default(), |
|
| 444 | 450 | walked_dirs: Default::default(), |
|
| 445 | 451 | listed_dirs: Default::default(), |
|
| 446 | 452 | git_export_skipped: Default::default(), |
|
| … | 73 unchanged lines |
||
| 520 | 526 | let mut repo = Repo { |
|
| 521 | 527 | last_change: Default::default(), |
|
| 522 | 528 | at_op: None, |
|
| 529 | + | head_seen: Default::default(), |
|
| 523 | 530 | walked_dirs: Default::default(), |
|
| 524 | 531 | listed_dirs: Default::default(), |
|
| 525 | 532 | git_export_skipped: Default::default(), |
|
| … | 28 unchanged lines |
||
| 554 | 561 | let mut repo = Repo { |
|
| 555 | 562 | last_change: Default::default(), |
|
| 556 | 563 | at_op: None, |
|
| 564 | + | head_seen: Default::default(), |
|
| 557 | 565 | walked_dirs: Default::default(), |
|
| 558 | 566 | listed_dirs: Default::default(), |
|
| 559 | 567 | git_export_skipped: Default::default(), |
|
| … | 69 unchanged lines |
||
| 629 | 637 | Ok(()) |
|
| 630 | 638 | } |
|
| 631 | 639 | ||
| 632 | - | /// Pack automatically once many loose objects have accumulated. |
|
| 640 | + | /// Pack automatically once many loose objects have accumulated, and |
|
| 641 | + | /// delete what packing retired over [`crate::store::RETIRED_FOR`] ago |
|
| 642 | + | /// (a store nobody runs `oxy gc` on would keep both copies forever). |
|
| 633 | 643 | pub(crate) fn maybe_autopack(&self) -> Result<()> { |
|
| 644 | + | self.store |
|
| 645 | + | .expire_retired(crate::store::RETIRED_FOR, &mut |_| {})?; |
|
| 634 | 646 | if self.store.loose_count()? > AUTOPACK_THRESHOLD { |
|
| 635 | 647 | self.store.pack_loose()?; |
|
| 636 | 648 | self.store.merge_packs()?; |
|
| … | 209 unchanged lines |
||
| 846 | 858 | ) |
|
| 847 | 859 | } |
|
| 848 | 860 | ||
| 861 | + | /// The operation head this handle reads: the one it first read, or |
|
| 862 | + | /// the last one it moved the head to (see [`Repo::refresh`]). |
|
| 849 | 863 | fn head_ref(&self) -> Result<ObjectRef> { |
|
| 850 | - | Ok(self.read_head()?.0) |
|
| 864 | + | if let Some(r) = self.head_seen.get() { |
|
| 865 | + | return Ok(r); |
|
| 866 | + | } |
|
| 867 | + | let r = self.read_head()?.0; |
|
| 868 | + | self.head_seen.set(Some(r)); |
|
| 869 | + | Ok(r) |
|
| 870 | + | } |
|
| 871 | + | ||
| 872 | + | /// Read the operation head again: what another process or handle |
|
| 873 | + | /// recorded since this handle first read it becomes visible. A handle |
|
| 874 | + | /// keeps the head it first read for the rest of its life otherwise, |
|
| 875 | + | /// so whatever it reads comes from one view (oxygenhub#249); a loop |
|
| 876 | + | /// that waits for others' operations, or a step that must see what |
|
| 877 | + | /// another handle just recorded, refreshes. Packs written since are |
|
| 878 | + | /// picked up too. |
|
| 879 | + | pub fn refresh(&self) -> Result<()> { |
|
| 880 | + | self.forget_head(); |
|
| 881 | + | self.store.refresh()?; |
|
| 882 | + | Ok(()) |
|
| 883 | + | } |
|
| 884 | + | ||
| 885 | + | /// Read the operation head again on next use, as [`Repo::refresh`] |
|
| 886 | + | /// does, leaving the packs as they are. |
|
| 887 | + | pub(crate) fn forget_head(&self) { |
|
| 888 | + | self.head_seen.set(None); |
|
| 851 | 889 | } |
|
| 852 | 890 | ||
| 853 | 891 | /// The operation head, and on a store its record's version for |
|
| … | 36 unchanged lines |
||
| 890 | 928 | fn swap_head(&self, to: &ObjectRef, expected: Option<&crate::strata::Version>) -> Result<bool> { |
|
| 891 | 929 | if self.store.is_local() { |
|
| 892 | 930 | write_atomic(&self.oxy.join("ops").join("head"), to.to_hex().as_bytes())?; |
|
| 931 | + | self.head_seen.set(Some(*to)); |
|
| 893 | 932 | return Ok(true); |
|
| 894 | 933 | } |
|
| 895 | 934 | match self |
|
| … | 1 unchanged lines |
||
| 897 | 936 | .backend() |
|
| 898 | 937 | .write_ref(HEAD_RECORD, to.to_hex().as_bytes(), expected) |
|
| 899 | 938 | { |
|
| 900 | - | Ok(_) => Ok(true), |
|
| 939 | + | Ok(_) => { |
|
| 940 | + | self.head_seen.set(Some(*to)); |
|
| 941 | + | Ok(true) |
|
| 942 | + | } |
|
| 901 | 943 | Err(crate::strata::Error::Conflict(_)) => Ok(false), |
|
| 902 | 944 | Err(e) => fail(format!("move the operation head: {e}")), |
|
| 903 | 945 | } |
|
| … | 1806 unchanged lines |
modified crates/oxy-core/src/repo/strata.rs +3 −0
| … | 232 unchanged lines |
||
| 233 | 233 | None => (RefsResponse::default(), None), |
|
| 234 | 234 | }; |
|
| 235 | 235 | self.with_mirror(|mirror| { |
|
| 236 | + | // Another process may have brought the mirror level since this |
|
| 237 | + | // handle read its head. |
|
| 238 | + | mirror.refresh()?; |
|
| 236 | 239 | // The policy first: bundles are then imported under it. |
|
| 237 | 240 | if let Some(sp) = &resp.policy { |
|
| 238 | 241 | let have = mirror.current_policy()?.map(|(_, p)| p.policy.serial); |
|
| … | 1120 unchanged lines |
modified crates/oxy-core/src/repo/sync.rs +4 −0
| … | 653 unchanged lines |
||
| 654 | 654 | policy: Option<SignedPolicy>, |
|
| 655 | 655 | tags: Vec<crate::protocol::TagRef>, |
|
| 656 | 656 | ) -> Result<PushReport> { |
|
| 657 | + | // Updates are checked against the head as it is now, not as this |
|
| 658 | + | // handle first read it (oxygenhub#249): another push may have |
|
| 659 | + | // landed since. |
|
| 660 | + | self.refresh()?; |
|
| 657 | 661 | let mut report = PushReport::default(); |
|
| 658 | 662 | let results = &mut report.results; |
|
| 659 | 663 | if let Some(sp) = policy { |
|
| … | 1514 unchanged lines |
modified crates/oxy-core/src/repo/tests.rs +202 −0
| … | 505 unchanged lines |
||
| 506 | 506 | write(&root.join("a"), "f.txt", b"a\n"); |
|
| 507 | 507 | let h1 = a.commit("first", &[]).unwrap(); |
|
| 508 | 508 | let results = a.push("origin", false).unwrap(); |
|
| 509 | + | origin.refresh().unwrap(); |
|
| 509 | 510 | assert!(results[0].ok, "{}", results[0].reason); |
|
| 510 | 511 | assert_eq!(origin.bookmark_heads().unwrap(), vec![h1]); |
|
| 511 | 512 | // The change was sealed at push and the reference attested. |
|
| … | 14 unchanged lines |
||
| 526 | 527 | let first = serial_of(&att); |
|
| 527 | 528 | assert!(first > 1_000_000_000_000, "{first}"); |
|
| 528 | 529 | let again = a.push("origin", false).unwrap(); |
|
| 530 | + | origin.refresh().unwrap(); |
|
| 529 | 531 | assert!(again[0].ok && again[0].reason.contains("up to date")); |
|
| 530 | 532 | let (_, att) = origin.latest_ref_attest("main").unwrap().unwrap(); |
|
| 531 | 533 | assert_eq!(serial_of(&att), first, "a no-op push does not attest again"); |
|
| … | 11 unchanged lines |
||
| 543 | 545 | write(&root.join("b"), "f.txt", b"a\nb\n"); |
|
| 544 | 546 | let h2 = b.commit("second", &[]).unwrap(); |
|
| 545 | 547 | assert!(b.push("origin", false).unwrap()[0].ok); |
|
| 548 | + | origin.refresh().unwrap(); |
|
| 546 | 549 | assert_eq!(origin.bookmark_heads().unwrap(), vec![h2]); |
|
| 547 | 550 | ||
| 548 | 551 | // A commits an independent change; push is refused until A pulls. |
|
| … | 8 unchanged lines |
||
| 557 | 560 | assert_eq!(fs::read(root.join("a/f.txt")).unwrap(), b"a\nb\n"); |
|
| 558 | 561 | assert_eq!(fs::read(root.join("a/g.txt")).unwrap(), b"g\n"); |
|
| 559 | 562 | let results = a.push("origin", false).unwrap(); |
|
| 563 | + | origin.refresh().unwrap(); |
|
| 560 | 564 | assert!(results[0].ok, "{}", results[0].reason); |
|
| 561 | 565 | let mut origin_heads = origin.bookmark_heads().unwrap(); |
|
| 562 | 566 | origin_heads.sort(); |
|
| … | 11 unchanged lines |
||
| 574 | 578 | write(&root.join("a"), "f.txt", b"a\nA\n"); |
|
| 575 | 579 | a.commit("a-edit", &[]).unwrap(); |
|
| 576 | 580 | a.push("origin", false).unwrap(); |
|
| 581 | + | origin.refresh().unwrap(); |
|
| 577 | 582 | write(&root.join("b"), "f.txt", b"a\nB\n"); |
|
| 578 | 583 | b.commit("b-edit", &[]).unwrap(); |
|
| 579 | 584 | let pr = b.pull("origin").unwrap(); |
|
| … | 7 unchanged lines |
||
| 587 | 592 | ChangeKind::Resolution |
|
| 588 | 593 | ); |
|
| 589 | 594 | assert!(b.push("origin", false).unwrap()[0].ok); |
|
| 595 | + | origin.refresh().unwrap(); |
|
| 590 | 596 | a.pull("origin").unwrap(); |
|
| 591 | 597 | assert_eq!(fs::read(root.join("a/f.txt")).unwrap(), b"a\nAB\n"); |
|
| 592 | 598 | // The resolution depends on the conflicting edits, but nothing depends |
|
| … | 390 unchanged lines |
||
| 983 | 989 | write(&root.join("b"), "b.txt", b"b\n"); |
|
| 984 | 990 | b.commit("unsigned before policy", &[]).unwrap(); |
|
| 985 | 991 | assert!(b.push("origin", false).unwrap()[0].ok); |
|
| 992 | + | origin.refresh().unwrap(); |
|
| 986 | 993 | ||
| 987 | 994 | // A creates the policy and pushes it; origin now enforces it. |
|
| 988 | 995 | write(&root.join("a"), "a.txt", b"a\n"); |
|
| … | 2 unchanged lines |
||
| 991 | 998 | assert_eq!(sp.policy.serial, 1); |
|
| 992 | 999 | a.pull("origin").unwrap(); |
|
| 993 | 1000 | let results = a.push("origin", false).unwrap(); |
|
| 1001 | + | origin.refresh().unwrap(); |
|
| 994 | 1002 | assert!(results.iter().all(|r| r.ok), "{results:?}"); |
|
| 995 | 1003 | assert_eq!(origin.current_policy().unwrap().unwrap().1.policy.serial, 1); |
|
| 996 | 1004 | ||
| … | 12 unchanged lines |
||
| 1009 | 1017 | let h_b2 = b.commit("b2 as Bob", &[]).unwrap(); |
|
| 1010 | 1018 | assert_eq!(b.store().get_change(&h_b2).unwrap().author, key_b); |
|
| 1011 | 1019 | let r = b.push("origin", false).unwrap(); |
|
| 1020 | + | origin.refresh().unwrap(); |
|
| 1012 | 1021 | assert!( |
|
| 1013 | 1022 | !r[0].ok && r[0].reason.contains("may not update"), |
|
| 1014 | 1023 | "{:?}", |
|
| … | 4 unchanged lines |
||
| 1019 | 1028 | let sp = a.policy_grant(id_b.clone(), None).unwrap(); |
|
| 1020 | 1029 | assert_eq!(sp.policy.serial, 2); |
|
| 1021 | 1030 | let r = a.push("origin", false).unwrap(); |
|
| 1031 | + | origin.refresh().unwrap(); |
|
| 1022 | 1032 | assert!(r.iter().any(|x| x.name == "policy" && x.ok), "{r:?}"); |
|
| 1023 | 1033 | // B must fetch the new policy before pushing. |
|
| 1024 | 1034 | assert!(b |
|
| … | 8 unchanged lines |
||
| 1033 | 1043 | b.pull("origin").unwrap(); |
|
| 1034 | 1044 | assert_eq!(b.current_policy().unwrap().unwrap().1.policy.serial, 2); |
|
| 1035 | 1045 | let r = b.push("origin", false).unwrap(); |
|
| 1046 | + | origin.refresh().unwrap(); |
|
| 1036 | 1047 | assert!(r[0].ok, "{:?}", r); |
|
| 1037 | 1048 | // Origin verified B's change under the policy. |
|
| 1038 | 1049 | let heads = origin.bookmark_heads().unwrap(); |
|
| … | 36 unchanged lines |
||
| 1075 | 1086 | a.pull("origin").unwrap(); |
|
| 1076 | 1087 | a.policy_revoke(&key_b).unwrap(); |
|
| 1077 | 1088 | let r = a.push("origin", false).unwrap(); |
|
| 1089 | + | origin.refresh().unwrap(); |
|
| 1078 | 1090 | assert!(r.iter().any(|x| x.name == "policy" && x.ok), "{r:?}"); |
|
| 1079 | 1091 | b.pull("origin").unwrap(); |
|
| 1080 | 1092 | write(&root.join("b"), "b.txt", b"b3\n"); |
|
| 1081 | 1093 | b.commit("b3", &[]).unwrap(); |
|
| 1082 | 1094 | let r = b.push("origin", false).unwrap(); |
|
| 1095 | + | origin.refresh().unwrap(); |
|
| 1083 | 1096 | assert!(!r[0].ok, "{:?}", r); |
|
| 1084 | 1097 | ||
| 1085 | 1098 | // A clone by a third party receives and verifies everything. |
|
| … | 897 unchanged lines |
||
| 1983 | 1996 | "the state still has everything" |
|
| 1984 | 1997 | ); |
|
| 1985 | 1998 | assert!(a.push("origin", false).unwrap()[0].ok); |
|
| 1999 | + | origin.refresh().unwrap(); |
|
| 1986 | 2000 | // Files created outside the scope are ignored by status. |
|
| 1987 | 2001 | write(&root.join("a"), "docs/new.md", b"ignored\n"); |
|
| 1988 | 2002 | assert!(a.snapshot("status").unwrap().statuses.is_empty()); |
|
| … | 77 unchanged lines |
||
| 2066 | 2080 | write(&root.join("a"), "a.txt", b"0\n1\n2\n3\n4\n5\n6\n7\n8\n"); |
|
| 2067 | 2081 | let h_new = a.commit("c8", &[]).unwrap(); |
|
| 2068 | 2082 | assert!(a.push("origin", false).unwrap()[0].ok); |
|
| 2083 | + | origin.refresh().unwrap(); |
|
| 2069 | 2084 | assert_eq!(origin.bookmark_heads().unwrap(), vec![h_new]); |
|
| 2070 | 2085 | // Fetching new work from origin into the shallow clone works too. |
|
| 2071 | 2086 | write(&origin_dir, "b.txt", b"b\n"); |
|
| … | 66 unchanged lines |
||
| 2138 | 2153 | ||
| 2139 | 2154 | // Push carries the tag; a second attestation for the name is refused. |
|
| 2140 | 2155 | let r = a.push("origin", false).unwrap(); |
|
| 2156 | + | origin.refresh().unwrap(); |
|
| 2141 | 2157 | assert!(r.iter().any(|x| x.name == "tags/v1" && x.ok), "{r:?}"); |
|
| 2142 | 2158 | assert_eq!(origin.tag_heads("v1").unwrap(), Some(vec![h1])); |
|
| 2143 | 2159 | let mut forged = a |
|
| … | 21 unchanged lines |
||
| 2165 | 2181 | assert!(b.tags().unwrap()[0].signer.as_deref() == Some("Ada")); |
|
| 2166 | 2182 | a.tag_create("v2", &[h2], "second").unwrap(); |
|
| 2167 | 2183 | a.push("origin", false).unwrap(); |
|
| 2184 | + | origin.refresh().unwrap(); |
|
| 2168 | 2185 | let rep = b.fetch("origin").unwrap(); |
|
| 2169 | 2186 | assert_eq!(rep.new_tags, 1); |
|
| 2170 | 2187 | assert_eq!(b.tag_heads("v2").unwrap(), Some(vec![h2])); |
|
| … | 1 unchanged lines |
||
| 2172 | 2189 | // Under a policy, an unattested tag is refused and an attested one by the root passes. |
|
| 2173 | 2190 | a.policy_init().unwrap(); |
|
| 2174 | 2191 | a.push("origin", false).unwrap(); |
|
| 2192 | + | origin.refresh().unwrap(); |
|
| 2175 | 2193 | let unattested = crate::protocol::TagRef { |
|
| 2176 | 2194 | name: "v3".into(), |
|
| 2177 | 2195 | state: origin |
|
| … | 9 unchanged lines |
||
| 2187 | 2205 | assert!(!r[0].ok && r[0].reason.contains("attestation"), "{r:?}"); |
|
| 2188 | 2206 | a.tag_create("v3", &[h2], "").unwrap(); |
|
| 2189 | 2207 | let r = a.push("origin", false).unwrap(); |
|
| 2208 | + | origin.refresh().unwrap(); |
|
| 2190 | 2209 | assert!(r.iter().any(|x| x.name == "tags/v3" && x.ok), "{r:?}"); |
|
| 2191 | 2210 | assert!(a.gc(true).is_ok()); |
|
| 2192 | 2211 | assert_eq!(a.tags().unwrap().len(), 3, "tags survive gc"); |
|
| … | 3 unchanged lines |
||
| 2196 | 2215 | // nothing new. |
|
| 2197 | 2216 | let rot = a.rotate_identity().unwrap(); |
|
| 2198 | 2217 | let r = a.push("origin", false).unwrap(); |
|
| 2218 | + | origin.refresh().unwrap(); |
|
| 2199 | 2219 | assert!(r.iter().all(|x| x.ok), "{r:?}"); |
|
| 2200 | 2220 | let c = Repo::clone(&url, &root.join("c")) |
|
| 2201 | 2221 | .unwrap() |
|
| … | 40 unchanged lines |
||
| 2242 | 2262 | "the first workspace is untouched" |
|
| 2243 | 2263 | ); |
|
| 2244 | 2264 | let h2 = ws.commit("b from ws2", &[]).unwrap(); |
|
| 2265 | + | repo.refresh().unwrap(); |
|
| 2245 | 2266 | let mut heads = repo.bookmark_heads().unwrap(); |
|
| 2246 | 2267 | heads.sort(); |
|
| 2247 | 2268 | let mut want = vec![h1, h2]; |
|
| … | 278 unchanged lines |
||
| 2526 | 2547 | let h1 = a.commit("one", &[]).unwrap(); |
|
| 2527 | 2548 | assert!(a.policy_init().unwrap().applied); |
|
| 2528 | 2549 | assert!(a.push("origin", false).unwrap().iter().all(|r| r.ok)); |
|
| 2550 | + | origin.refresh().unwrap(); |
|
| 2529 | 2551 | ||
| 2530 | 2552 | // Expiry: an identity that had expired when the change was made does not authenticate it. |
|
| 2531 | 2553 | let mut e = Repo::init(&root.join("e")) |
|
| … | 49 unchanged lines |
||
| 2581 | 2603 | ); |
|
| 2582 | 2604 | // Origin accepts the rotated policy (signed by the old root) and the new key's change. |
|
| 2583 | 2605 | let res = a.push("origin", false).unwrap(); |
|
| 2606 | + | origin.refresh().unwrap(); |
|
| 2584 | 2607 | assert!(res.iter().all(|x| x.ok), "{res:?}"); |
|
| 2585 | 2608 | assert_eq!(origin.current_policy().unwrap().unwrap().1.policy.serial, 2); |
|
| 2586 | 2609 | let key_a = r.new; |
|
| … | 22 unchanged lines |
||
| 2609 | 2632 | "already a delegate" |
|
| 2610 | 2633 | ); |
|
| 2611 | 2634 | assert!(a.push("origin", false).unwrap().iter().all(|x| x.ok)); |
|
| 2635 | + | origin.refresh().unwrap(); |
|
| 2612 | 2636 | write(&root.join("a"), "a.txt", b"a3\n"); |
|
| 2613 | 2637 | let h3 = a.commit("three", &[]).unwrap(); |
|
| 2614 | 2638 | assert!( |
|
| … | 19 unchanged lines |
||
| 2634 | 2658 | assert!(a.delegate_remove(&key_b).is_err()); |
|
| 2635 | 2659 | let key_a3 = r.new; |
|
| 2636 | 2660 | let res = a.push("origin", false).unwrap(); |
|
| 2661 | + | origin.refresh().unwrap(); |
|
| 2637 | 2662 | assert!(res.iter().all(|x| x.ok), "{res:?}"); |
|
| 2638 | 2663 | ||
| 2639 | 2664 | // Multi-signer policy: Bob becomes a root and the threshold rises to 2, |
|
| … | 3 unchanged lines |
||
| 2643 | 2668 | assert!(a.policy_set_threshold(3).is_err()); |
|
| 2644 | 2669 | assert!(a.policy_set_threshold(2).unwrap().applied); |
|
| 2645 | 2670 | assert!(a.push("origin", false).unwrap().iter().all(|x| x.ok)); |
|
| 2671 | + | origin.refresh().unwrap(); |
|
| 2646 | 2672 | b.pull("origin").unwrap(); |
|
| 2647 | 2673 | assert_eq!( |
|
| 2648 | 2674 | b.current_policy().unwrap().unwrap().1.policy.root_threshold, |
|
| … | 37 unchanged lines |
||
| 2686 | 2712 | .unwrap() |
|
| 2687 | 2713 | .iter() |
|
| 2688 | 2714 | .any(|x| x.name == "policy" && x.ok)); |
|
| 2715 | + | origin.refresh().unwrap(); |
|
| 2689 | 2716 | a.pull("origin").unwrap(); |
|
| 2690 | 2717 | assert_eq!( |
|
| 2691 | 2718 | a.current_policy().unwrap().unwrap().1.policy.serial, |
|
| … | 586 unchanged lines |
||
| 3278 | 3305 | write(&r, "a.txt", b"a\n"); |
|
| 3279 | 3306 | let h_a = repo.commit("a", &[]).unwrap(); |
|
| 3280 | 3307 | assert!(repo.push("origin", false).unwrap()[0].ok); |
|
| 3308 | + | origin.refresh().unwrap(); |
|
| 3281 | 3309 | ||
| 3282 | 3310 | // A bookmark at the working copy's base, then work on it. |
|
| 3283 | 3311 | assert!(repo.bookmark_create("bad name..", None).is_err()); |
|
| … | 52 unchanged lines |
||
| 3336 | 3364 | assert!(repo.merge("main").is_err()); |
|
| 3337 | 3365 | // The other bookmark can be pushed by name and appears at origin. |
|
| 3338 | 3366 | assert!(repo.push_bookmark("origin", "feature", false).unwrap()[0].ok); |
|
| 3367 | + | origin.refresh().unwrap(); |
|
| 3339 | 3368 | assert_eq!(origin.resolve_heads("feature").unwrap(), vec![h_b]); |
|
| 3340 | 3369 | assert!(repo.push("origin", false).unwrap()[0].ok); |
|
| 3370 | + | origin.refresh().unwrap(); |
|
| 3341 | 3371 | assert_eq!( |
|
| 3342 | 3372 | repo.bookmarks() |
|
| 3343 | 3373 | .unwrap() |
|
| … | 680 unchanged lines |
||
| 4024 | 4054 | a.tag_create("v1", &[h2], "first release").unwrap(); |
|
| 4025 | 4055 | a.checkpoint().unwrap(); |
|
| 4026 | 4056 | assert!(a.push("origin", false).unwrap()[0].ok); |
|
| 4057 | + | origin.refresh().unwrap(); |
|
| 4027 | 4058 | ||
| 4028 | 4059 | // Sound: signed changes, a seal, a tag, a checkpoint, a ref attestation. |
|
| 4029 | 4060 | for repo in [&a, &origin] { |
|
| … | 1062 unchanged lines |
||
| 5092 | 5123 | write(&r, "a.txt", b"a\n"); |
|
| 5093 | 5124 | let h_a = repo.commit("a", &[]).unwrap(); |
|
| 5094 | 5125 | assert!(repo.push("origin", false).unwrap()[0].ok); |
|
| 5126 | + | origin.refresh().unwrap(); |
|
| 5095 | 5127 | repo.bookmark_create("feature", None).unwrap(); |
|
| 5096 | 5128 | repo.bookmark_create("feature/x", None).unwrap(); |
|
| 5097 | 5129 | repo.bookmark_switch("feature/x", false).unwrap(); |
|
| … | 1 unchanged lines |
||
| 5099 | 5131 | let h_b = repo.commit("b", &[]).unwrap(); |
|
| 5100 | 5132 | for name in ["feature", "feature/x"] { |
|
| 5101 | 5133 | let res = repo.push_bookmark("origin", name, false).unwrap(); |
|
| 5134 | + | origin.refresh().unwrap(); |
|
| 5102 | 5135 | assert!(res[0].ok, "{name}: {res:?}"); |
|
| 5103 | 5136 | } |
|
| 5104 | 5137 | assert_eq!(origin.resolve_heads("feature").unwrap(), vec![h_a]); |
|
| … | 32 unchanged lines |
||
| 5137 | 5170 | write(&r, "a.txt", b"a\n"); |
|
| 5138 | 5171 | repo.commit("a", &[]).unwrap(); |
|
| 5139 | 5172 | assert!(repo.push("origin", false).unwrap()[0].ok); |
|
| 5173 | + | origin.refresh().unwrap(); |
|
| 5140 | 5174 | repo.bookmark_create("feature/x", None).unwrap(); |
|
| 5141 | 5175 | assert!(repo.push_bookmark("origin", "feature/x", false).unwrap()[0].ok); |
|
| 5176 | + | origin.refresh().unwrap(); |
|
| 5142 | 5177 | assert!(origin.resolve_heads("feature/x").is_ok()); |
|
| 5143 | 5178 | ||
| 5144 | 5179 | let res = repo.push_delete("origin", "feature/x").unwrap(); |
|
| 5180 | + | origin.refresh().unwrap(); |
|
| 5145 | 5181 | assert!(res[0].ok, "{res:?}"); |
|
| 5146 | 5182 | assert!(!origin.view().unwrap().bookmarks.contains_key("feature/x")); |
|
| 5147 | 5183 | let (_, att) = origin.latest_ref_attest("feature/x").unwrap().unwrap(); |
|
| … | 14 unchanged lines |
||
| 5162 | 5198 | ||
| 5163 | 5199 | // Pushing it again creates it again, past the deletion's serial. |
|
| 5164 | 5200 | let res = repo.push_bookmark("origin", "feature/x", false).unwrap(); |
|
| 5201 | + | origin.refresh().unwrap(); |
|
| 5165 | 5202 | assert!(res[0].ok, "{res:?}"); |
|
| 5166 | 5203 | assert!(origin.view().unwrap().bookmarks.contains_key("feature/x")); |
|
| 5167 | 5204 | ||
| 5168 | 5205 | // Nothing to delete, and never the remote's own bookmark. |
|
| 5169 | 5206 | assert!(repo.push_delete("origin", "nope").is_err()); |
|
| 5170 | 5207 | let res = repo.push_delete("origin", "main").unwrap(); |
|
| 5208 | + | origin.refresh().unwrap(); |
|
| 5171 | 5209 | assert!( |
|
| 5172 | 5210 | !res[0].ok && res[0].reason.contains("default bookmark"), |
|
| 5173 | 5211 | "{res:?}" |
|
| … | 633 unchanged lines |
||
| 5807 | 5845 | write(&root.join("b"), "base.txt", b"base\n"); |
|
| 5808 | 5846 | b.commit("base", &[]).unwrap(); |
|
| 5809 | 5847 | assert!(b.push("origin", false).unwrap()[0].ok); |
|
| 5848 | + | origin.refresh().unwrap(); |
|
| 5810 | 5849 | ||
| 5811 | 5850 | // A feature bookmark with one change, pushed. |
|
| 5812 | 5851 | b.bookmark_create("feat", None).unwrap(); |
|
| … | 1 unchanged lines |
||
| 5814 | 5853 | write(&root.join("b"), "x.txt", b"x\n"); |
|
| 5815 | 5854 | let x = b.commit("x", &[]).unwrap(); |
|
| 5816 | 5855 | assert!(b.push("origin", false).unwrap()[0].ok); |
|
| 5856 | + | origin.refresh().unwrap(); |
|
| 5817 | 5857 | b.bookmark_switch("main", false).unwrap(); |
|
| 5818 | 5858 | ||
| 5819 | 5859 | // The remote lands it: its main moves to the feature's state. |
|
| … | 1776 unchanged lines |
||
| 7596 | 7636 | let (h, v) = a.read_head().unwrap(); |
|
| 7597 | 7637 | assert!(b.swap_head(&rb, v.as_ref()).unwrap()); |
|
| 7598 | 7638 | assert!(!a.swap_head(&h, v.as_ref()).unwrap(), "stale"); |
|
| 7639 | + | a.refresh().unwrap(); |
|
| 7599 | 7640 | assert_eq!(a.head_ref().unwrap(), rb); |
|
| 7600 | 7641 | ||
| 7601 | 7642 | let other = tempfile::tempdir().unwrap(); |
|
| … | 837 unchanged lines |
||
| 8439 | 8480 | // Both on main: A commits, then B builds on it, editing A's file. |
|
| 8440 | 8481 | write(&root, "a.txt", b"a\n"); |
|
| 8441 | 8482 | a.commit("from a", &[]).unwrap(); |
|
| 8483 | + | b.refresh().unwrap(); |
|
| 8442 | 8484 | let heads = b.resolve_heads("main").unwrap(); |
|
| 8443 | 8485 | b.checkout(&heads, false, "oxy checkout").unwrap(); |
|
| 8444 | 8486 | write(&ws_dir, "a.txt", b"a, then b\n"); |
|
| … | 166 unchanged lines |
||
| 8611 | 8653 | let e = app.build_pin_change(&[a1], "libs/none", &pin2, "x", author); |
|
| 8612 | 8654 | assert!(format!("{:?}", e.unwrap_err()).contains("does not exist")); |
|
| 8613 | 8655 | } |
|
| 8656 | + | ||
| 8657 | + | /// oxygenhub#249: a handle reads the operation head once and keeps it, |
|
| 8658 | + | /// so one command or request never mixes two heads; another handle's |
|
| 8659 | + | /// operation shows only after `refresh`. Its own operations move it, and |
|
| 8660 | + | /// one recorded from the head it kept merges with what landed since, as |
|
| 8661 | + | /// any concurrent operation does. |
|
| 8662 | + | #[test] |
|
| 8663 | + | fn one_handle_keeps_one_head_until_it_refreshes() { |
|
| 8664 | + | hermetic(); |
|
| 8665 | + | let dir = tempfile::tempdir().unwrap(); |
|
| 8666 | + | let root = dir.path(); |
|
| 8667 | + | let writer = Repo::init(root).unwrap(); |
|
| 8668 | + | write(root, "a.txt", b"one\n"); |
|
| 8669 | + | let h1 = writer.commit("one", &[]).unwrap(); |
|
| 8670 | + | let reader = Repo::open(root).unwrap(); |
|
| 8671 | + | let before = reader.view().unwrap(); |
|
| 8672 | + | assert_eq!(reader.bookmark_heads().unwrap(), vec![h1]); |
|
| 8673 | + | write(root, "a.txt", b"two\n"); |
|
| 8674 | + | let h2 = writer.commit("two", &[]).unwrap(); |
|
| 8675 | + | assert_eq!( |
|
| 8676 | + | writer.bookmark_heads().unwrap(), |
|
| 8677 | + | vec![h2], |
|
| 8678 | + | "its own operation" |
|
| 8679 | + | ); |
|
| 8680 | + | assert_eq!(reader.view().unwrap(), before, "the head it read first"); |
|
| 8681 | + | assert_eq!(reader.bookmark_heads().unwrap(), vec![h1]); |
|
| 8682 | + | reader.refresh().unwrap(); |
|
| 8683 | + | assert_eq!(reader.bookmark_heads().unwrap(), vec![h2]); |
|
| 8684 | + | ||
| 8685 | + | // A handle recording from an older head merges with the newer one. |
|
| 8686 | + | let stale = Repo::open(root).unwrap(); |
|
| 8687 | + | assert_eq!(stale.bookmark_heads().unwrap(), vec![h2]); |
|
| 8688 | + | write(root, "a.txt", b"three\n"); |
|
| 8689 | + | let h3 = writer.commit("three", &[]).unwrap(); |
|
| 8690 | + | stale.bookmark_create("feature", Some(&[h1])).unwrap(); |
|
| 8691 | + | let view = stale.view().unwrap(); |
|
| 8692 | + | assert!(view.bookmarks.contains_key("feature")); |
|
| 8693 | + | assert_eq!( |
|
| 8694 | + | stale.bookmark_heads().unwrap(), |
|
| 8695 | + | vec![h3], |
|
| 8696 | + | "merged, not lost" |
|
| 8697 | + | ); |
|
| 8698 | + | let fresh = Repo::open(root).unwrap(); |
|
| 8699 | + | assert_eq!(fresh.view().unwrap(), view); |
|
| 8700 | + | } |
|
| 8701 | + | ||
| 8702 | + | /// oxygenhub#249 over a store (a bucket's compare-and-swap): the same. |
|
| 8703 | + | #[test] |
|
| 8704 | + | fn over_a_store_one_handle_keeps_one_head() { |
|
| 8705 | + | hermetic(); |
|
| 8706 | + | let backend = std::sync::Arc::new(crate::strata::MemoryStore::new()); |
|
| 8707 | + | crate::store::TEST_BACKEND.with(|t| *t.borrow_mut() = Some(backend.clone())); |
|
| 8708 | + | let dir = tempfile::tempdir().unwrap(); |
|
| 8709 | + | let a = Repo::init(dir.path()).unwrap(); |
|
| 8710 | + | let (h0, _) = a.head_op().unwrap(); |
|
| 8711 | + | let b = Repo::open(dir.path()).unwrap(); |
|
| 8712 | + | assert_eq!(b.head_op().unwrap().0, h0); |
|
| 8713 | + | let state = a.store.put_state(&StateObject::new(vec![])).unwrap(); |
|
| 8714 | + | let mut view = a.view().unwrap(); |
|
| 8715 | + | view.bookmarks.insert("x".into(), state); |
|
| 8716 | + | let r = a.record_op(&[h0], "bookmark x", view).unwrap(); |
|
| 8717 | + | assert_eq!(a.head_op().unwrap().0, r); |
|
| 8718 | + | assert_eq!(b.head_op().unwrap().0, h0, "b keeps its head"); |
|
| 8719 | + | b.refresh().unwrap(); |
|
| 8720 | + | assert_eq!(b.head_op().unwrap().0, r); |
|
| 8721 | + | crate::store::TEST_BACKEND.with(|t| *t.borrow_mut() = None); |
|
| 8722 | + | } |
|
| 8723 | + | ||
| 8724 | + | /// oxygenhub#249: gc retires what it repacks rather than deleting it; a |
|
| 8725 | + | /// reader opened before reads on, then and after the next gc deletes the |
|
| 8726 | + | /// retired files once their hour is up. Younger ones are kept. |
|
| 8727 | + | #[test] |
|
| 8728 | + | fn gc_retires_and_a_later_gc_deletes_after_the_hour() { |
|
| 8729 | + | hermetic(); |
|
| 8730 | + | let dir = tempfile::tempdir().unwrap(); |
|
| 8731 | + | let root = dir.path(); |
|
| 8732 | + | let repo = Repo::init(root).unwrap(); |
|
| 8733 | + | let mut heads = Vec::new(); |
|
| 8734 | + | for i in 0..4 { |
|
| 8735 | + | write(root, &format!("f{i}.txt"), format!("{i}\n").as_bytes()); |
|
| 8736 | + | heads.push(repo.commit(&format!("c{i}"), &[]).unwrap()); |
|
| 8737 | + | } |
|
| 8738 | + | let reader = Repo::open(root).unwrap(); |
|
| 8739 | + | assert_eq!(reader.log().unwrap().len(), 5); |
|
| 8740 | + | let r = repo.gc(true).unwrap(); |
|
| 8741 | + | assert_eq!(r.retired, crate::store::RetiredReport::default()); |
|
| 8742 | + | assert_eq!(repo.store().loose_count().unwrap(), 0); |
|
| 8743 | + | assert_eq!(reader.log().unwrap().len(), 5, "read on from what it had"); |
|
| 8744 | + | // Nothing is an hour old yet: a gc keeps it all. |
|
| 8745 | + | let r = repo.gc(true).unwrap(); |
|
| 8746 | + | assert_eq!((r.retired.packs, r.retired.loose), (0, 0)); |
|
| 8747 | + | assert!(r.retired.kept > 0, "{r:?}"); |
|
| 8748 | + | write(root, "f9.txt", b"9\n"); |
|
| 8749 | + | heads.push(repo.commit("c9", &[]).unwrap()); |
|
| 8750 | + | crate::store::reader_tests::age_retired( |
|
| 8751 | + | repo.store(), |
|
| 8752 | + | crate::store::RETIRED_FOR + std::time::Duration::from_secs(60), |
|
| 8753 | + | ); |
|
| 8754 | + | let r = repo.gc(true).unwrap(); |
|
| 8755 | + | assert!(r.retired.packs >= 1 && r.retired.loose > 0, "{r:?}"); |
|
| 8756 | + | assert_eq!(r.retired.kept, 0, "{r:?}"); |
|
| 8757 | + | let r = repo.gc(true).unwrap(); |
|
| 8758 | + | assert_eq!((r.retired.packs, r.retired.loose), (0, 0)); |
|
| 8759 | + | assert!(r.retired.kept > 0, "what the last gc retired is young"); |
|
| 8760 | + | // The reader opened before all of it still reads everything, from |
|
| 8761 | + | // the packs it finds when what it loaded is gone. |
|
| 8762 | + | assert_eq!(reader.log().unwrap().len(), 5); |
|
| 8763 | + | for h in &heads[..4] { |
|
| 8764 | + | reader.store().get_change(h).unwrap(); |
|
| 8765 | + | } |
|
| 8766 | + | reader.refresh().unwrap(); |
|
| 8767 | + | assert_eq!(reader.log().unwrap().len(), 6); |
|
| 8768 | + | let fresh = Repo::open(root).unwrap(); |
|
| 8769 | + | assert_eq!( |
|
| 8770 | + | fresh.store().pack_count(), |
|
| 8771 | + | 1, |
|
| 8772 | + | "retired packs are not loaded" |
|
| 8773 | + | ); |
|
| 8774 | + | } |
|
| 8775 | + | ||
| 8776 | + | /// oxygenhub#249: a repository opens when its store's pack listing names |
|
| 8777 | + | /// an index that is not there (deleted between the listing and the read). |
|
| 8778 | + | #[test] |
|
| 8779 | + | fn a_repository_opens_when_a_listed_index_is_gone() { |
|
| 8780 | + | hermetic(); |
|
| 8781 | + | let backend = crate::store::reader_tests::racing(true); |
|
| 8782 | + | let dyn_backend: std::sync::Arc<crate::store::Backend> = backend.clone(); |
|
| 8783 | + | crate::store::TEST_BACKEND.with(|t| *t.borrow_mut() = Some(dyn_backend)); |
|
| 8784 | + | let dir = tempfile::tempdir().unwrap(); |
|
| 8785 | + | let repo = Repo::init(dir.path()).unwrap(); |
|
| 8786 | + | write(dir.path(), "a.txt", b"a\n"); |
|
| 8787 | + | let h = repo.commit("a", &[]).unwrap(); |
|
| 8788 | + | repo.gc(true).unwrap(); |
|
| 8789 | + | let again = Repo::open(dir.path()).unwrap(); |
|
| 8790 | + | assert_eq!(again.store().pack_count(), 1); |
|
| 8791 | + | assert_eq!(again.bookmark_heads().unwrap(), vec![h]); |
|
| 8792 | + | // And when one is deleted between the listing and the read: a copy of |
|
| 8793 | + | // the live pack, whose index goes as the store opens. |
|
| 8794 | + | use crate::strata::Store as _; |
|
| 8795 | + | let idx = backend |
|
| 8796 | + | .list("pack-") |
|
| 8797 | + | .unwrap() |
|
| 8798 | + | .into_iter() |
|
| 8799 | + | .map(|e| e.key) |
|
| 8800 | + | .find(|k| k.ends_with(".idx") && k != "pack-0000.idx") |
|
| 8801 | + | .unwrap(); |
|
| 8802 | + | let stem = idx.trim_end_matches(".idx"); |
|
| 8803 | + | for ext in ["pack", "idx"] { |
|
| 8804 | + | let bytes = backend.get(&format!("{stem}.{ext}")).unwrap(); |
|
| 8805 | + | backend |
|
| 8806 | + | .put_new(&format!("pack-copy.{ext}"), &bytes) |
|
| 8807 | + | .unwrap(); |
|
| 8808 | + | } |
|
| 8809 | + | *backend.doomed.lock().unwrap() = Some("pack-copy.idx".into()); |
|
| 8810 | + | let again = Repo::open(dir.path()).unwrap(); |
|
| 8811 | + | assert_eq!(again.store().pack_count(), 1); |
|
| 8812 | + | assert_eq!(again.bookmark_heads().unwrap(), vec![h]); |
|
| 8813 | + | assert!(!backend.has("pack-copy.idx").unwrap()); |
|
| 8814 | + | crate::store::TEST_BACKEND.with(|t| *t.borrow_mut() = None); |
|
| 8815 | + | } |
modified crates/oxy-core/src/repo/workspace.rs +3 −0
| … | 345 unchanged lines |
||
| 346 | 346 | std::time::Duration::from_secs(60), |
|
| 347 | 347 | )?; |
|
| 348 | 348 | HELD.with(|h| h.borrow_mut().insert(path.clone(), 1)); |
|
| 349 | + | // Whoever held the lock may have moved the operation head: what |
|
| 350 | + | // is done under it starts from the head as it is now (oxygenhub#249). |
|
| 351 | + | self.forget_head(); |
|
| 349 | 352 | Ok(WorkingCopyLock { |
|
| 350 | 353 | path, |
|
| 351 | 354 | _file: Some(file), |
|
| … | 3 unchanged lines |
modified crates/oxy-core/src/store.rs +792 −43
| … | 23 unchanged lines |
||
| 24 | 24 | //! `receipt/<hh>/<rest>/<attest>` index the seals covering a change and |
|
| 25 | 25 | //! the receipts on a state. Older stores hold these as one list per |
|
| 26 | 26 | //! object (`sigs/`, `seals/`, `receipts/`), which are still read. |
|
| 27 | + | //! `pack-retired` records the packs and loose objects a repack replaced, |
|
| 28 | + | //! which stay for readers until a collection an hour later (oxygenhub#249). |
|
| 27 | 29 | ||
| 28 | 30 | use crate::attest::Attest; |
|
| 29 | 31 | use crate::hash::{hash_object, header, ObjectRef, ObjectType}; |
|
| … | 28 unchanged lines |
||
| 58 | 60 | /// Largest object a pack entry may claim; anything bigger is corruption, |
|
| 59 | 61 | /// and refusing it keeps a bad index from driving huge allocations. |
|
| 60 | 62 | const MAX_OBJECT_LEN: u32 = 1 << 30; |
|
| 63 | + | /// What reading an entry of a pack whose dictionary is gone says. A pack |
|
| 64 | + | /// being deleted can lose its dictionary first, so a read that meets this |
|
| 65 | + | /// looks at the packs again before it is believed. |
|
| 66 | + | const NO_DICT: &str = "packed with a dictionary the pack lacks"; |
|
| 61 | 67 | ||
| 68 | + | /// The record of what repacks replaced (oxygenhub#249): packs and loose |
|
| 69 | + | /// objects whose contents a newer pack holds, kept for readers that |
|
| 70 | + | /// still use them until garbage collection deletes them, at least |
|
| 71 | + | /// [`RETIRED_FOR`] later. One line per key, `<retired at, wall ms> <key>`. |
|
| 72 | + | /// It is a reference record (updated by compare-and-swap, so concurrent |
|
| 73 | + | /// repacks and collections never lose each other's lines), named under |
|
| 74 | + | /// `pack-` so the pack listing every open makes shows whether it holds |
|
| 75 | + | /// anything. |
|
| 76 | + | const RETIRED_KEY: &str = "pack-retired"; |
|
| 77 | + | /// How long a retired pack or loose object stays in the store: a reader |
|
| 78 | + | /// that loaded its list of packs before a repack reads on from the old |
|
| 79 | + | /// packs this long, and after that finds the objects in the new pack by |
|
| 80 | + | /// looking again (see [`Store::get`]). |
|
| 81 | + | pub const RETIRED_FOR: std::time::Duration = std::time::Duration::from_secs(60 * 60); |
|
| 82 | + | /// Compare-and-swap rounds for the retired record before giving up. |
|
| 83 | + | const RETIRED_TRIES: usize = 32; |
|
| 84 | + | /// After a look at the pack listing found nothing new, how long a missed |
|
| 85 | + | /// [`Store::has`] answers without looking again: a writer probing for |
|
| 86 | + | /// thousands of objects it is about to store would otherwise list the |
|
| 87 | + | /// packs for each. |
|
| 88 | + | const HAS_RELIST_AFTER: std::time::Duration = std::time::Duration::from_secs(1); |
|
| 89 | + | ||
| 90 | + | /// A pack listing as compared: each key with its size and time. |
|
| 91 | + | type Listed = Vec<(String, u64, Option<u64>)>; |
|
| 92 | + | ||
| 93 | + | /// The retired record's lines: each key with the time it was retired. |
|
| 94 | + | fn parse_retired(bytes: &[u8]) -> std::collections::BTreeMap<String, u64> { |
|
| 95 | + | String::from_utf8_lossy(bytes) |
|
| 96 | + | .lines() |
|
| 97 | + | .filter_map(|l| { |
|
| 98 | + | let (t, k) = l.split_once(' ')?; |
|
| 99 | + | Some((k.to_string(), t.parse().ok()?)) |
|
| 100 | + | }) |
|
| 101 | + | .collect() |
|
| 102 | + | } |
|
| 103 | + | ||
| 104 | + | fn encode_retired(m: &std::collections::BTreeMap<String, u64>) -> Vec<u8> { |
|
| 105 | + | m.iter() |
|
| 106 | + | .map(|(k, t)| format!("{t} {k}\n")) |
|
| 107 | + | .collect::<String>() |
|
| 108 | + | .into_bytes() |
|
| 109 | + | } |
|
| 110 | + | ||
| 111 | + | /// What deleting retired packs and loose objects did. |
|
| 112 | + | #[derive(Debug, Clone, Default, PartialEq, Eq)] |
|
| 113 | + | pub struct RetiredReport { |
|
| 114 | + | /// Retired packs deleted, each with its index and dictionary. |
|
| 115 | + | pub packs: usize, |
|
| 116 | + | /// Retired loose objects deleted. |
|
| 117 | + | pub loose: usize, |
|
| 118 | + | /// Retired entries younger than [`RETIRED_FOR`], left for the next |
|
| 119 | + | /// collection. |
|
| 120 | + | pub kept: usize, |
|
| 121 | + | } |
|
| 122 | + | ||
| 62 | 123 | /// An object read from the store: its type and body. |
|
| 63 | 124 | pub type Object = (ObjectType, Vec<u8>); |
|
| 64 | 125 | ||
| … | 110 unchanged lines |
||
| 175 | 236 | dict: Option<&Arc<zstd::dict::DecoderDictionary<'static>>>, |
|
| 176 | 237 | ) -> Result<Object> { |
|
| 177 | 238 | let body = if e.flags & FLAG_DICT != 0 { |
|
| 178 | - | let dict = dict.ok_or_else(|| { |
|
| 179 | - | StoreError::Corrupt(*r, "packed with a dictionary the pack lacks".into()) |
|
| 180 | - | })?; |
|
| 239 | + | let dict = dict.ok_or_else(|| StoreError::Corrupt(*r, NO_DICT.into()))?; |
|
| 181 | 240 | zstd::bulk::Decompressor::with_prepared_dictionary(dict) |
|
| 182 | 241 | .and_then(|mut d| d.decompress(&stored, e.raw_len as usize)) |
|
| 183 | 242 | .map_err(|err| StoreError::Corrupt(*r, format!("zstd: {err}")))? |
|
| … | 14 unchanged lines |
||
| 198 | 257 | ||
| 199 | 258 | impl PackIndex { |
|
| 200 | 259 | /// Load a pack's index; `pack_len` is the pack's size, from the listing |
|
| 201 | - | /// that found it (a stat of it would be a request of its own). |
|
| 202 | - | fn load(backend: &Backend, idx_key: &str, pack_key: &str, pack_len: u64) -> Result<PackIndex> { |
|
| 203 | - | let data = backend.get(idx_key).map_err(backend_err)?; |
|
| 260 | + | /// that found it (a stat of it would be a request of its own). `None` |
|
| 261 | + | /// when the index is gone since the listing: the pack was deleted. |
|
| 262 | + | fn load( |
|
| 263 | + | backend: &Backend, |
|
| 264 | + | idx_key: &str, |
|
| 265 | + | pack_key: &str, |
|
| 266 | + | pack_len: u64, |
|
| 267 | + | ) -> Result<Option<PackIndex>> { |
|
| 268 | + | let data = match backend.get(idx_key) { |
|
| 269 | + | Ok(d) => d, |
|
| 270 | + | Err(BackendError::NotFound(_)) => return Ok(None), |
|
| 271 | + | Err(e) => return Err(backend_err(e)), |
|
| 272 | + | }; |
|
| 204 | 273 | let bad = |m: &str| StoreError::Corrupt(ObjectRef::ZERO, format!("{idx_key}: {m}")); |
|
| 205 | 274 | if data.len() < 12 || &data[..8] != IDX_MAGIC { |
|
| 206 | 275 | return Err(bad("bad index header")); |
|
| … | 40 unchanged lines |
||
| 247 | 316 | let dict_key = format!("{}.dict", pack_key.trim_end_matches(".pack")); |
|
| 248 | 317 | let dict = get_optional(backend, &dict_key)? |
|
| 249 | 318 | .map(|d| Arc::new(zstd::dict::DecoderDictionary::copy(&d))); |
|
| 250 | - | Ok(PackIndex { |
|
| 319 | + | Ok(Some(PackIndex { |
|
| 251 | 320 | key: pack_key.to_string(), |
|
| 252 | 321 | refs, |
|
| 253 | 322 | entries, |
|
| 254 | 323 | dict, |
|
| 255 | - | }) |
|
| 324 | + | })) |
|
| 256 | 325 | } |
|
| 257 | 326 | ||
| 258 | 327 | fn find(&self, r: &ObjectRef) -> Option<PackEntry> { |
|
| … | 65 unchanged lines |
||
| 324 | 393 | scratch: PathBuf, |
|
| 325 | 394 | /// The backend is that directory, so the old file paths still hold. |
|
| 326 | 395 | local: bool, |
|
| 327 | - | packs: RefCell<Vec<PackIndex>>, |
|
| 396 | + | packs: RefCell<Vec<Arc<PackIndex>>>, |
|
| 397 | + | /// The pack listing `packs` was loaded from (key, size, time), to tell |
|
| 398 | + | /// whether a look at the store would find anything new. |
|
| 399 | + | packs_listed: RefCell<Listed>, |
|
| 400 | + | /// When a look at the pack listing for a missed [`Store::has`] last |
|
| 401 | + | /// found nothing new. |
|
| 402 | + | has_relisted: std::cell::Cell<Option<std::time::Instant>>, |
|
| 328 | 403 | /// An open write batch (spec §13.4): objects go here instead of to loose |
|
| 329 | 404 | /// files, and are written out with one `fsync` when the batch finishes. |
|
| 330 | 405 | pending: RefCell<Option<Pending>>, |
|
| … | 162 unchanged lines |
||
| 493 | 568 | scratch, |
|
| 494 | 569 | local: false, |
|
| 495 | 570 | packs: RefCell::new(Vec::new()), |
|
| 571 | + | packs_listed: RefCell::new(Vec::new()), |
|
| 572 | + | has_relisted: std::cell::Cell::new(None), |
|
| 496 | 573 | pending: RefCell::new(None), |
|
| 497 | 574 | batch_depth: std::cell::Cell::new(0), |
|
| 498 | 575 | metas: RefCell::new(None), |
|
| … | 13 unchanged lines |
||
| 512 | 589 | /// is a file. |
|
| 513 | 590 | pub fn is_local(&self) -> bool { |
|
| 514 | 591 | self.local |
|
| 592 | + | } |
|
| 593 | + | ||
| 594 | + | /// Load the list of packs again, from a new listing: what other |
|
| 595 | + | /// writers packed since becomes readable, and packs since retired or |
|
| 596 | + | /// deleted are let go. Packs already loaded are kept, not read again. |
|
| 597 | + | pub fn refresh(&self) -> Result<()> { |
|
| 598 | + | self.reload_packs() |
|
| 599 | + | } |
|
| 600 | + | ||
| 601 | + | /// The pack listing, with how `packs_listed` records it. |
|
| 602 | + | fn list_packs(&self) -> Result<(Vec<strata::Entry>, Listed)> { |
|
| 603 | + | let listed = self.backend.list("pack-").map_err(backend_err)?; |
|
| 604 | + | let sig = listed |
|
| 605 | + | .iter() |
|
| 606 | + | .map(|e| (e.key.clone(), e.size, e.modified_ms)) |
|
| 607 | + | .collect(); |
|
| 608 | + | Ok((listed, sig)) |
|
| 515 | 609 | } |
|
| 516 | 610 | ||
| 611 | + | /// Load every pack the listing names that is not retired. An index |
|
| 612 | + | /// that is gone by the time it is read (its pack deleted after the |
|
| 613 | + | /// listing) is passed over, and the store listed once more, since |
|
| 614 | + | /// what replaced it may be new since the first listing. |
|
| 517 | 615 | fn reload_packs(&self) -> Result<()> { |
|
| 518 | - | let listed = self.backend.list("pack-").map_err(backend_err)?; |
|
| 519 | - | let mut idx_keys: Vec<&str> = listed |
|
| 616 | + | let (listed, sig) = self.list_packs()?; |
|
| 617 | + | self.load_listed(listed, sig) |
|
| 618 | + | } |
|
| 619 | + | ||
| 620 | + | fn load_listed(&self, mut listed: Vec<strata::Entry>, mut sig: Listed) -> Result<()> { |
|
| 621 | + | let had: std::collections::HashMap<String, Arc<PackIndex>> = self |
|
| 622 | + | .packs |
|
| 623 | + | .borrow() |
|
| 624 | + | .iter() |
|
| 625 | + | .map(|p| (p.key.clone(), p.clone())) |
|
| 626 | + | .collect(); |
|
| 627 | + | let mut loaded: std::collections::HashMap<String, Arc<PackIndex>> = Default::default(); |
|
| 628 | + | for attempt in 0..2 { |
|
| 629 | + | let retired = self.retired_in(&listed)?; |
|
| 630 | + | let mut idx_keys: Vec<&str> = listed |
|
| 631 | + | .iter() |
|
| 632 | + | .map(|e| e.key.as_str()) |
|
| 633 | + | .filter(|k| k.ends_with(".idx")) |
|
| 634 | + | .collect(); |
|
| 635 | + | idx_keys.sort(); |
|
| 636 | + | let mut packs = Vec::new(); |
|
| 637 | + | let mut vanished = false; |
|
| 638 | + | for idx in idx_keys { |
|
| 639 | + | let pack = format!("{}.pack", idx.trim_end_matches(".idx")); |
|
| 640 | + | if retired.contains_key(&pack) { |
|
| 641 | + | continue; |
|
| 642 | + | } |
|
| 643 | + | let Some(p) = listed.iter().find(|e| e.key == pack) else { |
|
| 644 | + | continue; |
|
| 645 | + | }; |
|
| 646 | + | if let Some(kept) = had.get(&pack).or_else(|| loaded.get(&pack)) { |
|
| 647 | + | packs.push(kept.clone()); |
|
| 648 | + | continue; |
|
| 649 | + | } |
|
| 650 | + | match PackIndex::load(self.backend.as_ref(), idx, &pack, p.size)? { |
|
| 651 | + | Some(index) => { |
|
| 652 | + | let index = Arc::new(index); |
|
| 653 | + | loaded.insert(pack, index.clone()); |
|
| 654 | + | packs.push(index); |
|
| 655 | + | } |
|
| 656 | + | None => vanished = true, |
|
| 657 | + | } |
|
| 658 | + | } |
|
| 659 | + | if vanished && attempt == 0 { |
|
| 660 | + | (listed, sig) = self.list_packs()?; |
|
| 661 | + | continue; |
|
| 662 | + | } |
|
| 663 | + | *self.packs.borrow_mut() = packs; |
|
| 664 | + | *self.packs_listed.borrow_mut() = sig; |
|
| 665 | + | return Ok(()); |
|
| 666 | + | } |
|
| 667 | + | unreachable!("the second round always returns") |
|
| 668 | + | } |
|
| 669 | + | ||
| 670 | + | /// Load the packs again when the pack listing differs from the one |
|
| 671 | + | /// they were loaded from; whether it did. What a miss calls before it |
|
| 672 | + | /// is believed (oxygenhub#249): another writer may have packed the |
|
| 673 | + | /// object, or a repack moved it, since this store loaded its packs. |
|
| 674 | + | fn reload_if_changed(&self) -> Result<bool> { |
|
| 675 | + | let (listed, sig) = self.list_packs()?; |
|
| 676 | + | if *self.packs_listed.borrow() == sig { |
|
| 677 | + | return Ok(false); |
|
| 678 | + | } |
|
| 679 | + | self.load_listed(listed, sig)?; |
|
| 680 | + | Ok(true) |
|
| 681 | + | } |
|
| 682 | + | ||
| 683 | + | /// The retired record, read when `listed` (a pack listing) shows it |
|
| 684 | + | /// holds anything. |
|
| 685 | + | fn retired_in( |
|
| 686 | + | &self, |
|
| 687 | + | listed: &[strata::Entry], |
|
| 688 | + | ) -> Result<std::collections::BTreeMap<String, u64>> { |
|
| 689 | + | match listed.iter().find(|e| e.key == RETIRED_KEY) { |
|
| 690 | + | Some(e) if e.size > 0 => Ok(self.read_retired()?.0), |
|
| 691 | + | _ => Ok(Default::default()), |
|
| 692 | + | } |
|
| 693 | + | } |
|
| 694 | + | ||
| 695 | + | /// The retired record and its version (`None`: never written). |
|
| 696 | + | fn read_retired( |
|
| 697 | + | &self, |
|
| 698 | + | ) -> Result<( |
|
| 699 | + | std::collections::BTreeMap<String, u64>, |
|
| 700 | + | Option<strata::Version>, |
|
| 701 | + | )> { |
|
| 702 | + | match self.backend.read_ref(RETIRED_KEY).map_err(backend_err)? { |
|
| 703 | + | Some((bytes, v)) => Ok((parse_retired(&bytes), Some(v))), |
|
| 704 | + | None => Ok((Default::default(), None)), |
|
| 705 | + | } |
|
| 706 | + | } |
|
| 707 | + | ||
| 708 | + | /// Change the retired record by compare-and-swap, reading it again |
|
| 709 | + | /// whenever another writer changed it first. `edit` must give the |
|
| 710 | + | /// same result however often it runs. |
|
| 711 | + | fn update_retired( |
|
| 712 | + | &self, |
|
| 713 | + | mut edit: impl FnMut(&mut std::collections::BTreeMap<String, u64>), |
|
| 714 | + | ) -> Result<()> { |
|
| 715 | + | for _ in 0..RETIRED_TRIES { |
|
| 716 | + | let (mut m, version) = self.read_retired()?; |
|
| 717 | + | let before = m.clone(); |
|
| 718 | + | edit(&mut m); |
|
| 719 | + | if m == before && version.is_some() { |
|
| 720 | + | return Ok(()); |
|
| 721 | + | } |
|
| 722 | + | match self |
|
| 723 | + | .backend |
|
| 724 | + | .write_ref(RETIRED_KEY, &encode_retired(&m), version.as_ref()) |
|
| 725 | + | { |
|
| 726 | + | Ok(_) => return Ok(()), |
|
| 727 | + | Err(BackendError::Conflict(_)) => continue, |
|
| 728 | + | Err(e) => return Err(backend_err(e)), |
|
| 729 | + | } |
|
| 730 | + | } |
|
| 731 | + | Err(StoreError::Io(io::Error::other( |
|
| 732 | + | "other writers kept changing the record of retired packs; run the command again", |
|
| 733 | + | ))) |
|
| 734 | + | } |
|
| 735 | + | ||
| 736 | + | /// Record `keys` (packs, by their `.pack` key, and loose objects) as |
|
| 737 | + | /// retired now: kept for the readers still using them, and deleted by |
|
| 738 | + | /// a collection at least [`RETIRED_FOR`] later. A key already retired |
|
| 739 | + | /// keeps its first time. |
|
| 740 | + | fn retire(&self, keys: &[String]) -> Result<()> { |
|
| 741 | + | if keys.is_empty() { |
|
| 742 | + | return Ok(()); |
|
| 743 | + | } |
|
| 744 | + | let now = now_ms(); |
|
| 745 | + | self.update_retired(|m| { |
|
| 746 | + | for k in keys { |
|
| 747 | + | m.entry(k.clone()).or_insert(now); |
|
| 748 | + | } |
|
| 749 | + | }) |
|
| 750 | + | } |
|
| 751 | + | ||
| 752 | + | /// Delete what was retired at least `older_than` ago (a collection |
|
| 753 | + | /// does, with [`RETIRED_FOR`]): each pack with its index and |
|
| 754 | + | /// dictionary, and each loose object a pack now holds. A retired loose |
|
| 755 | + | /// object no pack holds any more (the pack that had it was pruned, and |
|
| 756 | + | /// the object stored again since) is kept, as an ordinary loose object. |
|
| 757 | + | pub fn expire_retired( |
|
| 758 | + | &self, |
|
| 759 | + | older_than: std::time::Duration, |
|
| 760 | + | progress: Progress, |
|
| 761 | + | ) -> Result<RetiredReport> { |
|
| 762 | + | let (listed, _) = self.list_packs()?; |
|
| 763 | + | let retired = self.retired_in(&listed)?; |
|
| 764 | + | let mut report = RetiredReport::default(); |
|
| 765 | + | if retired.is_empty() { |
|
| 766 | + | return Ok(report); |
|
| 767 | + | } |
|
| 768 | + | let now = now_ms(); |
|
| 769 | + | let older_than = older_than.as_millis() as u64; |
|
| 770 | + | let due: Vec<(String, u64)> = retired |
|
| 520 | 771 | .iter() |
|
| 521 | - | .map(|e| e.key.as_str()) |
|
| 522 | - | .filter(|k| k.ends_with(".idx")) |
|
| 772 | + | .filter(|(_, t)| now.saturating_sub(**t) >= older_than) |
|
| 773 | + | .map(|(k, t)| (k.clone(), *t)) |
|
| 523 | 774 | .collect(); |
|
| 524 | - | idx_keys.sort(); |
|
| 525 | - | let mut packs = Vec::new(); |
|
| 526 | - | for idx in idx_keys { |
|
| 527 | - | let pack = format!("{}.pack", idx.trim_end_matches(".idx")); |
|
| 528 | - | if let Some(p) = listed.iter().find(|e| e.key == pack) { |
|
| 529 | - | packs.push(PackIndex::load(self.backend.as_ref(), idx, &pack, p.size)?); |
|
| 775 | + | report.kept = retired.len() - due.len(); |
|
| 776 | + | if due.is_empty() { |
|
| 777 | + | return Ok(report); |
|
| 778 | + | } |
|
| 779 | + | progress(&format!("deleting {} retired files", due.len())); |
|
| 780 | + | // Which objects the live packs hold. |
|
| 781 | + | self.reload_packs()?; |
|
| 782 | + | for (key, _) in &due { |
|
| 783 | + | if key.ends_with(".pack") { |
|
| 784 | + | self.remove_pack(key)?; |
|
| 785 | + | report.packs += 1; |
|
| 786 | + | } else if let Some(r) = loose_ref(key) { |
|
| 787 | + | if self.find_packed(&r).is_some() { |
|
| 788 | + | delete(self.backend.as_ref(), key)?; |
|
| 789 | + | report.loose += 1; |
|
| 790 | + | } |
|
| 530 | 791 | } |
|
| 531 | 792 | } |
|
| 532 | - | *self.packs.borrow_mut() = packs; |
|
| 533 | - | Ok(()) |
|
| 793 | + | self.update_retired(|m| { |
|
| 794 | + | for (k, t) in &due { |
|
| 795 | + | if m.get(k) == Some(t) { |
|
| 796 | + | m.remove(k); |
|
| 797 | + | } |
|
| 798 | + | } |
|
| 799 | + | })?; |
|
| 800 | + | self.reload_packs()?; |
|
| 801 | + | Ok(report) |
|
| 534 | 802 | } |
|
| 535 | 803 | ||
| 804 | + | /// The loose objects retired and not yet deleted, by key. |
|
| 805 | + | fn retired_loose(&self) -> Result<std::collections::HashSet<String>> { |
|
| 806 | + | let (listed, _) = self.list_packs()?; |
|
| 807 | + | Ok(self |
|
| 808 | + | .retired_in(&listed)? |
|
| 809 | + | .into_keys() |
|
| 810 | + | .filter(|k| k.starts_with("loose/")) |
|
| 811 | + | .collect()) |
|
| 812 | + | } |
|
| 813 | + | ||
| 536 | 814 | pub fn pack_count(&self) -> usize { |
|
| 537 | 815 | self.packs.borrow().len() |
|
| 538 | 816 | } |
|
| … | 6 unchanged lines |
||
| 545 | 823 | .find_map(|(i, p)| p.find(r).map(|e| (i, e))) |
|
| 546 | 824 | } |
|
| 547 | 825 | ||
| 548 | - | /// Loose objects, with their keys and ages. |
|
| 826 | + | /// Loose objects, with their keys and ages; not those retired, whose |
|
| 827 | + | /// objects a pack holds. |
|
| 549 | 828 | fn loose_objects(&self) -> Result<Vec<(ObjectRef, strata::Entry)>> { |
|
| 829 | + | let retired = self.retired_loose()?; |
|
| 830 | + | let mut out = self.loose_objects_all()?; |
|
| 831 | + | out.retain(|(_, e)| !retired.contains(&e.key)); |
|
| 832 | + | Ok(out) |
|
| 833 | + | } |
|
| 834 | + | ||
| 835 | + | /// Every loose object, retired or not. |
|
| 836 | + | fn loose_objects_all(&self) -> Result<Vec<(ObjectRef, strata::Entry)>> { |
|
| 550 | 837 | let mut out: Vec<(ObjectRef, strata::Entry)> = self |
|
| 551 | 838 | .backend |
|
| 552 | 839 | .list("loose/") |
|
| … | 101 unchanged lines |
||
| 654 | 941 | let old_enough = |
|
| 655 | 942 | |modified: Option<u64>| modified.is_none_or(|t| now.saturating_sub(t) >= grace_ms); |
|
| 656 | 943 | let mut report = PruneReport::default(); |
|
| 657 | - | // Loose objects. |
|
| 658 | - | for (r, entry) in self.loose_objects()? { |
|
| 944 | + | // Loose objects. A retired one (its object packed) whose object is |
|
| 945 | + | // unreachable goes with it, uncounted: the pack's copy is counted. |
|
| 946 | + | let retired = self.retired_loose()?; |
|
| 947 | + | let mut unretired: Vec<String> = Vec::new(); |
|
| 948 | + | for (r, entry) in self.loose_objects_all()? { |
|
| 659 | 949 | if keep.contains(&r) { |
|
| 660 | 950 | continue; |
|
| 661 | 951 | } |
|
| 952 | + | if retired.contains(&entry.key) { |
|
| 953 | + | if old_enough(entry.modified_ms) { |
|
| 954 | + | delete(self.backend.as_ref(), &entry.key)?; |
|
| 955 | + | unretired.push(entry.key); |
|
| 956 | + | } |
|
| 957 | + | continue; |
|
| 958 | + | } |
|
| 662 | 959 | if !old_enough(entry.modified_ms) { |
|
| 663 | 960 | report.deferred += 1; |
|
| 664 | 961 | continue; |
|
| … | 2 unchanged lines |
||
| 667 | 964 | report.objects += 1; |
|
| 668 | 965 | delete(self.backend.as_ref(), &entry.key)?; |
|
| 669 | 966 | self.remove_sidecars(&r)?; |
|
| 967 | + | } |
|
| 968 | + | if !unretired.is_empty() { |
|
| 969 | + | self.update_retired(|m| { |
|
| 970 | + | for k in &unretired { |
|
| 971 | + | m.remove(k); |
|
| 972 | + | } |
|
| 973 | + | })?; |
|
| 670 | 974 | } |
|
| 671 | 975 | // Packs: rewrite those that hold unreachable objects. |
|
| 672 | 976 | let mut keep_refs: Vec<ObjectRef> = Vec::new(); |
|
| … | 94 unchanged lines |
||
| 767 | 1071 | self.reload_packs() |
|
| 768 | 1072 | } |
|
| 769 | 1073 | ||
| 770 | - | /// Remove a pack with its index and dictionary. |
|
| 1074 | + | /// Remove a pack with its index and dictionary: the index first, so |
|
| 1075 | + | /// a store opened meanwhile does not load a pack that is going. |
|
| 771 | 1076 | fn remove_pack(&self, pack_key: &str) -> Result<()> { |
|
| 772 | 1077 | let stem = pack_key.trim_end_matches(".pack"); |
|
| 1078 | + | delete(self.backend.as_ref(), &format!("{stem}.idx"))?; |
|
| 773 | 1079 | delete(self.backend.as_ref(), pack_key)?; |
|
| 774 | - | delete(self.backend.as_ref(), &format!("{stem}.idx"))?; |
|
| 775 | 1080 | delete(self.backend.as_ref(), &format!("{stem}.dict")) |
|
| 776 | 1081 | } |
|
| 777 | 1082 | ||
| 778 | - | /// Write `refs` into a new pack, then delete `loose_keys` and `old_packs`. |
|
| 1083 | + | /// Write `refs` into a new pack, then retire `loose_keys` and |
|
| 1084 | + | /// `old_packs`: they stay in the store, unused by stores opened from |
|
| 1085 | + | /// now on, until a collection [`RETIRED_FOR`] later deletes them, so a |
|
| 1086 | + | /// reader that loaded its packs before goes on reading (oxygenhub#249). |
|
| 779 | 1087 | fn write_pack( |
|
| 780 | 1088 | &self, |
|
| 781 | 1089 | refs: &[ObjectRef], |
|
| … | 3 unchanged lines |
||
| 785 | 1093 | progress: Progress, |
|
| 786 | 1094 | ) -> Result<PackReport> { |
|
| 787 | 1095 | if refs.is_empty() { |
|
| 788 | - | for p in old_packs { |
|
| 789 | - | self.remove_pack(p)?; |
|
| 790 | - | } |
|
| 1096 | + | self.retire(old_packs)?; |
|
| 791 | 1097 | self.reload_packs()?; |
|
| 792 | 1098 | return Ok(PackReport::default()); |
|
| 793 | 1099 | } |
|
| … | 115 unchanged lines |
||
| 909 | 1215 | } |
|
| 910 | 1216 | progress(&format!("writing the index of {} objects", index.len())); |
|
| 911 | 1217 | self.finish_pack_files(&id, &tmp_pack, &index)?; |
|
| 912 | - | for p in old_packs { |
|
| 913 | - | self.remove_pack(p)?; |
|
| 914 | - | } |
|
| 1218 | + | let mut retired = old_packs.to_vec(); |
|
| 1219 | + | retired.extend_from_slice(loose_keys); |
|
| 1220 | + | self.retire(&retired)?; |
|
| 915 | 1221 | self.reload_packs()?; |
|
| 916 | - | for key in loose_keys { |
|
| 917 | - | delete(self.backend.as_ref(), key)?; |
|
| 918 | - | } |
|
| 919 | 1222 | Ok(report) |
|
| 920 | 1223 | } |
|
| 921 | 1224 | ||
| … | 42 unchanged lines |
||
| 964 | 1267 | delete(self.backend.as_ref(), &loose_key(r)) |
|
| 965 | 1268 | } |
|
| 966 | 1269 | ||
| 1270 | + | /// Whether the store holds `r`. A miss looks at the pack listing |
|
| 1271 | + | /// again before it is believed, as [`Store::get`]'s does; a writer |
|
| 1272 | + | /// asking about many objects it is about to store looks at most once |
|
| 1273 | + | /// in [`HAS_RELIST_AFTER`] while nothing new turns up. |
|
| 967 | 1274 | pub fn has(&self, r: &ObjectRef) -> bool { |
|
| 968 | - | self.backend.has(&loose_key(r)).unwrap_or(false) |
|
| 969 | - | || self.find_packed(r).is_some() |
|
| 1275 | + | if self.has_now(r) { |
|
| 1276 | + | return true; |
|
| 1277 | + | } |
|
| 1278 | + | let recent = self |
|
| 1279 | + | .has_relisted |
|
| 1280 | + | .get() |
|
| 1281 | + | .is_some_and(|t| t.elapsed() < HAS_RELIST_AFTER); |
|
| 1282 | + | if recent { |
|
| 1283 | + | return false; |
|
| 1284 | + | } |
|
| 1285 | + | match self.reload_if_changed() { |
|
| 1286 | + | Ok(true) => { |
|
| 1287 | + | self.has_relisted.set(None); |
|
| 1288 | + | self.has_now(r) |
|
| 1289 | + | } |
|
| 1290 | + | _ => { |
|
| 1291 | + | self.has_relisted.set(Some(std::time::Instant::now())); |
|
| 1292 | + | false |
|
| 1293 | + | } |
|
| 1294 | + | } |
|
| 1295 | + | } |
|
| 1296 | + | ||
| 1297 | + | /// [`Store::has`] against the packs as loaded. |
|
| 1298 | + | fn has_now(&self, r: &ObjectRef) -> bool { |
|
| 1299 | + | self.find_packed(r).is_some() |
|
| 970 | 1300 | || self |
|
| 971 | 1301 | .pending |
|
| 972 | 1302 | .borrow() |
|
| 973 | 1303 | .as_ref() |
|
| 974 | 1304 | .is_some_and(|p| p.lookup.contains_key(r)) |
|
| 1305 | + | || self.backend.has(&loose_key(r)).unwrap_or(false) |
|
| 975 | 1306 | } |
|
| 976 | 1307 | ||
| 977 | 1308 | /// Start a write batch: until the matching `finish`, `put` collects |
|
| … | 310 unchanged lines |
||
| 1288 | 1619 | /// The packs' indexes are in memory, so they are asked first: asking |
|
| 1289 | 1620 | /// for a loose copy of a packed object costs a request that finds |
|
| 1290 | 1621 | /// nothing on a store in a bucket, every time (oxygenhub#228). |
|
| 1622 | + | /// |
|
| 1623 | + | /// A miss is not believed at once (oxygenhub#249): when the pack it |
|
| 1624 | + | /// was in is gone (deleted after a repack retired it) the packs are |
|
| 1625 | + | /// loaded again, and when the object is nowhere the pack listing is |
|
| 1626 | + | /// looked at, and the packs loaded again if it changed (another |
|
| 1627 | + | /// writer packed it, or a repack moved it); then it is read once more. |
|
| 1291 | 1628 | pub fn get(&self, r: &ObjectRef) -> Result<Object> { |
|
| 1629 | + | match self.get_now(r) { |
|
| 1630 | + | Err(e) if is_miss(&e) => { |
|
| 1631 | + | // Read from a pack this store loaded, which is gone. |
|
| 1632 | + | let packed_miss = |
|
| 1633 | + | !matches!(e, StoreError::NotFound(_)) || self.find_packed(r).is_some(); |
|
| 1634 | + | let changed = if packed_miss { |
|
| 1635 | + | self.reload_packs()?; |
|
| 1636 | + | true |
|
| 1637 | + | } else { |
|
| 1638 | + | self.reload_if_changed()? |
|
| 1639 | + | }; |
|
| 1640 | + | if changed { |
|
| 1641 | + | self.get_now(r) |
|
| 1642 | + | } else { |
|
| 1643 | + | Err(e) |
|
| 1644 | + | } |
|
| 1645 | + | } |
|
| 1646 | + | other => other, |
|
| 1647 | + | } |
|
| 1648 | + | } |
|
| 1649 | + | ||
| 1650 | + | /// [`Store::get`] against the packs as loaded. |
|
| 1651 | + | fn get_now(&self, r: &ObjectRef) -> Result<Object> { |
|
| 1292 | 1652 | if let Some((i, e)) = self.find_packed(r) { |
|
| 1293 | - | return self.packs.borrow()[i].fetch(self.backend.as_ref(), r, e); |
|
| 1653 | + | let pack = self.packs.borrow()[i].clone(); |
|
| 1654 | + | return pack.fetch(self.backend.as_ref(), r, e); |
|
| 1294 | 1655 | } |
|
| 1295 | 1656 | match get_optional(self.backend.as_ref(), &loose_key(r))? { |
|
| 1296 | 1657 | Some(bytes) => parse_loose(&bytes, r), |
|
| … | 4 unchanged lines |
||
| 1301 | 1662 | /// Many objects at once, read and verified in parallel: packed ones |
|
| 1302 | 1663 | /// from their packs, the others loose (as [`Store::get`], packs first), |
|
| 1303 | 1664 | /// then what is pending in a batch. Results are in the order of `refs`. |
|
| 1665 | + | /// |
|
| 1666 | + | /// Misses are looked for again as [`Store::get`]'s are, together. |
|
| 1304 | 1667 | pub fn get_many(&self, refs: &[ObjectRef]) -> Vec<Result<Object>> { |
|
| 1668 | + | let mut out = self.get_many_now(refs); |
|
| 1669 | + | let missed: Vec<usize> = out |
|
| 1670 | + | .iter() |
|
| 1671 | + | .enumerate() |
|
| 1672 | + | .filter(|(_, g)| g.as_ref().is_err_and(is_miss)) |
|
| 1673 | + | .map(|(i, _)| i) |
|
| 1674 | + | .collect(); |
|
| 1675 | + | if missed.is_empty() { |
|
| 1676 | + | return out; |
|
| 1677 | + | } |
|
| 1678 | + | let packed_miss = missed.iter().any(|i| { |
|
| 1679 | + | !matches!(out[*i], Err(StoreError::NotFound(_))) |
|
| 1680 | + | || self.find_packed(&refs[*i]).is_some() |
|
| 1681 | + | }); |
|
| 1682 | + | let changed = if packed_miss { |
|
| 1683 | + | self.reload_packs().map(|_| true) |
|
| 1684 | + | } else { |
|
| 1685 | + | self.reload_if_changed() |
|
| 1686 | + | }; |
|
| 1687 | + | match changed { |
|
| 1688 | + | Ok(true) => { |
|
| 1689 | + | let again: Vec<ObjectRef> = missed.iter().map(|i| refs[*i]).collect(); |
|
| 1690 | + | for (i, g) in missed.into_iter().zip(self.get_many_now(&again)) { |
|
| 1691 | + | out[i] = g; |
|
| 1692 | + | } |
|
| 1693 | + | } |
|
| 1694 | + | Ok(false) => {} |
|
| 1695 | + | Err(e) => { |
|
| 1696 | + | let why = e.to_string(); |
|
| 1697 | + | for i in missed { |
|
| 1698 | + | out[i] = Err(StoreError::Io(io::Error::other(why.clone()))); |
|
| 1699 | + | } |
|
| 1700 | + | } |
|
| 1701 | + | } |
|
| 1702 | + | out |
|
| 1703 | + | } |
|
| 1704 | + | ||
| 1705 | + | /// [`Store::get_many`] against the packs as loaded. |
|
| 1706 | + | fn get_many_now(&self, refs: &[ObjectRef]) -> Vec<Result<Object>> { |
|
| 1305 | 1707 | use rayon::prelude::*; |
|
| 1306 | 1708 | let backend = self.backend.as_ref(); |
|
| 1307 | 1709 | let packs = self.packs.borrow(); |
|
| … | 6 unchanged lines |
||
| 1314 | 1716 | .find_map(|(p, pack)| pack.find(r).map(|e| (p, e))) |
|
| 1315 | 1717 | }) |
|
| 1316 | 1718 | .collect(); |
|
| 1317 | - | let indexes: &[PackIndex] = &packs; |
|
| 1719 | + | let indexes: &[Arc<PackIndex>] = &packs; |
|
| 1318 | 1720 | let found: Vec<Option<Result<Object>>> = refs |
|
| 1319 | 1721 | .par_iter() |
|
| 1320 | 1722 | .zip(place.par_iter()) |
|
| … | 502 unchanged lines |
||
| 1823 | 2225 | out.sort(); |
|
| 1824 | 2226 | out.dedup(); |
|
| 1825 | 2227 | Ok(out) |
|
| 2228 | + | } |
|
| 2229 | + | } |
|
| 2230 | + | ||
| 2231 | + | /// Whether a read failed in a way a newer list of packs may cure: the |
|
| 2232 | + | /// object was not found, or its pack's dictionary is gone. |
|
| 2233 | + | fn is_miss(e: &StoreError) -> bool { |
|
| 2234 | + | match e { |
|
| 2235 | + | StoreError::NotFound(_) => true, |
|
| 2236 | + | StoreError::Corrupt(_, why) => why == NO_DICT, |
|
| 2237 | + | _ => false, |
|
| 1826 | 2238 | } |
|
| 1827 | 2239 | } |
|
| 1828 | 2240 | ||
| … | 200 unchanged lines |
||
| 2029 | 2441 | let (report, merged) = again.repack_all(REPACK_ZSTD_LEVEL).unwrap(); |
|
| 2030 | 2442 | assert_eq!((report.objects, merged), (0, 0)); |
|
| 2031 | 2443 | assert_eq!(packs(dir.path()), before); |
|
| 2032 | - | // One new object: a repack replaces the pack and its dictionary. |
|
| 2444 | + | // One new object: a repack replaces the pack and its dictionary, |
|
| 2445 | + | // retiring the old ones, which a collection an hour on deletes. |
|
| 2033 | 2446 | let extra = again.put(ObjectType::Chunk, b"one more\n").unwrap(); |
|
| 2034 | 2447 | again.repack_all(REPACK_ZSTD_LEVEL).unwrap(); |
|
| 2035 | 2448 | let files = packs(dir.path()); |
|
| 2036 | - | assert_eq!(files.len(), 3, "one pack, index and dictionary: {files:?}"); |
|
| 2037 | - | assert_ne!(files, before); |
|
| 2449 | + | assert_eq!( |
|
| 2450 | + | files.len(), |
|
| 2451 | + | 7, |
|
| 2452 | + | "two of pack, index and dictionary, and the retired record: {files:?}" |
|
| 2453 | + | ); |
|
| 2454 | + | assert_eq!(again.pack_count(), 1, "the old pack is not read"); |
|
| 2038 | 2455 | assert_eq!(&again.get(&refs[7]).unwrap().1, &bodies[7]); |
|
| 2039 | 2456 | assert_eq!(again.get(&extra).unwrap().1, b"one more\n"); |
|
| 2457 | + | again |
|
| 2458 | + | .expire_retired(std::time::Duration::ZERO, &mut silent()) |
|
| 2459 | + | .unwrap(); |
|
| 2460 | + | let files = packs(dir.path()); |
|
| 2461 | + | assert_eq!(files.len(), 4, "one pack, index, dictionary: {files:?}"); |
|
| 2462 | + | assert!(before |
|
| 2463 | + | .iter() |
|
| 2464 | + | .filter(|f| *f != RETIRED_KEY) |
|
| 2465 | + | .all(|f| !files.contains(f))); |
|
| 2466 | + | assert_eq!(&again.get(&refs[8]).unwrap().1, &bodies[8]); |
|
| 2040 | 2467 | } |
|
| 2041 | 2468 | ||
| 2042 | 2469 | #[test] |
|
| … | 495 unchanged lines |
||
| 2538 | 2965 | let missing = hash_object(ObjectType::Chunk, b"never stored"); |
|
| 2539 | 2966 | assert!(matches!(s.get(&missing), Err(StoreError::NotFound(_)))); |
|
| 2540 | 2967 | assert!(backend.1.load(Ordering::Relaxed) > 0); |
|
| 2968 | + | } |
|
| 2969 | + | } |
|
| 2970 | + | ||
| 2971 | + | /// Readers never wait (oxygenhub#249): a store keeps reading while other |
|
| 2972 | + | /// handles repack, collect and push into the store under it. |
|
| 2973 | + | #[cfg(test)] |
|
| 2974 | + | pub(crate) mod reader_tests { |
|
| 2975 | + | use super::*; |
|
| 2976 | + | ||
| 2977 | + | fn chunk(i: usize) -> Vec<u8> { |
|
| 2978 | + | format!("object {i} ").repeat(1 + i % 5).into_bytes() |
|
| 2979 | + | } |
|
| 2980 | + | ||
| 2981 | + | /// Two handles on one store: a writer, and a reader opened at once. |
|
| 2982 | + | fn each_pair(f: impl Fn(&str, Store, Store)) { |
|
| 2983 | + | let dir = tempfile::tempdir().unwrap(); |
|
| 2984 | + | let s = Store::open(dir.path().join("objects")).unwrap(); |
|
| 2985 | + | let r = Store::open(dir.path().join("objects")).unwrap(); |
|
| 2986 | + | f("directory", s, r); |
|
| 2987 | + | let m: Arc<Backend> = Arc::new(MemoryStore::new()); |
|
| 2988 | + | let s = Store::open_on(m.clone(), dir.path().join("scratch")).unwrap(); |
|
| 2989 | + | let r = Store::open_on(m, dir.path().join("scratch-r")).unwrap(); |
|
| 2990 | + | f("memory", s, r); |
|
| 2991 | + | } |
|
| 2992 | + | ||
| 2993 | + | /// Set every retired entry's time back by `by`, as if retired then. |
|
| 2994 | + | pub(crate) fn age_retired(s: &Store, by: std::time::Duration) { |
|
| 2995 | + | s.update_retired(|m| { |
|
| 2996 | + | for t in m.values_mut() { |
|
| 2997 | + | *t = t.saturating_sub(by.as_millis() as u64); |
|
| 2998 | + | } |
|
| 2999 | + | }) |
|
| 3000 | + | .unwrap(); |
|
| 3001 | + | } |
|
| 3002 | + | ||
| 3003 | + | fn retired_keys(s: &Store) -> Vec<String> { |
|
| 3004 | + | s.read_retired().unwrap().0.into_keys().collect() |
|
| 3005 | + | } |
|
| 3006 | + | ||
| 3007 | + | #[test] |
|
| 3008 | + | fn a_reader_opened_before_a_repack_reads_every_object_after_it() { |
|
| 3009 | + | each_pair(|which, writer, _| { |
|
| 3010 | + | let mut refs = Vec::new(); |
|
| 3011 | + | for i in 0..40 { |
|
| 3012 | + | refs.push(writer.put(ObjectType::Chunk, &chunk(i)).unwrap()); |
|
| 3013 | + | } |
|
| 3014 | + | writer.pack_loose().unwrap(); |
|
| 3015 | + | for i in 40..80 { |
|
| 3016 | + | refs.push(writer.put(ObjectType::Chunk, &chunk(i)).unwrap()); |
|
| 3017 | + | } |
|
| 3018 | + | // Opened now: one pack and forty loose objects. |
|
| 3019 | + | let reader = Store::open_on(writer.backend().clone(), writer.dir()).unwrap(); |
|
| 3020 | + | assert_eq!(reader.pack_count(), 1, "{which}"); |
|
| 3021 | + | let (_, merged) = writer.repack_all(REPACK_ZSTD_LEVEL).unwrap(); |
|
| 3022 | + | assert_eq!(merged, 1); |
|
| 3023 | + | // The old pack and the loose objects are retired, not deleted. |
|
| 3024 | + | let retired = retired_keys(&writer); |
|
| 3025 | + | assert_eq!(retired.len(), 81, "{which}: {retired:?}"); |
|
| 3026 | + | assert_eq!(writer.loose_count().unwrap(), 0, "retired is not loose"); |
|
| 3027 | + | // The reader reads on from what it loaded. |
|
| 3028 | + | for (i, r) in refs.iter().enumerate() { |
|
| 3029 | + | assert_eq!(reader.get(r).unwrap().1, chunk(i), "{which}: {i}"); |
|
| 3030 | + | } |
|
| 3031 | + | assert_eq!(reader.pack_count(), 1); |
|
| 3032 | + | // A store opened now reads the new pack only. |
|
| 3033 | + | let fresh = Store::open_on(writer.backend().clone(), writer.dir()).unwrap(); |
|
| 3034 | + | assert_eq!(fresh.pack_count(), 1); |
|
| 3035 | + | assert_ne!(fresh.packs.borrow()[0].key, reader.packs.borrow()[0].key); |
|
| 3036 | + | // Once the retired files are deleted, the reader finds every |
|
| 3037 | + | // object in the new pack, by get, get_many and has alike. |
|
| 3038 | + | let report = writer |
|
| 3039 | + | .expire_retired(std::time::Duration::ZERO, &mut silent()) |
|
| 3040 | + | .unwrap(); |
|
| 3041 | + | assert_eq!((report.packs, report.loose), (1, 80), "{which}"); |
|
| 3042 | + | assert!(retired_keys(&writer).is_empty()); |
|
| 3043 | + | for (i, r) in refs.iter().enumerate() { |
|
| 3044 | + | assert_eq!(reader.get(r).unwrap().1, chunk(i), "{which}: {i}"); |
|
| 3045 | + | } |
|
| 3046 | + | let other = Store::open_on(writer.backend().clone(), writer.dir()).unwrap(); |
|
| 3047 | + | let before = other.packs.borrow()[0].key.clone(); |
|
| 3048 | + | writer.put(ObjectType::Chunk, b"new").unwrap(); |
|
| 3049 | + | writer.repack_all(REPACK_ZSTD_LEVEL).unwrap(); |
|
| 3050 | + | writer |
|
| 3051 | + | .expire_retired(std::time::Duration::ZERO, &mut silent()) |
|
| 3052 | + | .unwrap(); |
|
| 3053 | + | assert!(other.get_many(&refs).iter().all(|g| g.is_ok()), "{which}"); |
|
| 3054 | + | assert_ne!(other.packs.borrow()[0].key, before); |
|
| 3055 | + | let third = Store::open_on(writer.backend().clone(), writer.dir()).unwrap(); |
|
| 3056 | + | writer.put(ObjectType::Chunk, b"newer").unwrap(); |
|
| 3057 | + | writer.repack_all(REPACK_ZSTD_LEVEL).unwrap(); |
|
| 3058 | + | writer |
|
| 3059 | + | .expire_retired(std::time::Duration::ZERO, &mut silent()) |
|
| 3060 | + | .unwrap(); |
|
| 3061 | + | assert!(refs.iter().all(|r| third.has(r)), "{which}"); |
|
| 3062 | + | assert!(!third.has(&hash_object(ObjectType::Chunk, b"never"))); |
|
| 3063 | + | }); |
|
| 3064 | + | } |
|
| 3065 | + | ||
| 3066 | + | #[test] |
|
| 3067 | + | fn a_reader_opened_before_a_spilled_batch_reads_its_objects() { |
|
| 3068 | + | each_pair(|which, writer, reader| { |
|
| 3069 | + | assert_eq!(reader.pack_count(), 0); |
|
| 3070 | + | let mut refs = Vec::new(); |
|
| 3071 | + | { |
|
| 3072 | + | let batch = writer.begin_batch().unwrap(); |
|
| 3073 | + | for i in 0..(SPILL_OBJECTS + 50) { |
|
| 3074 | + | refs.push(writer.put(ObjectType::Chunk, &chunk(i)).unwrap()); |
|
| 3075 | + | } |
|
| 3076 | + | batch.finish().unwrap(); |
|
| 3077 | + | } |
|
| 3078 | + | assert_eq!(writer.pack_count(), 1, "{which}: the batch spilled"); |
|
| 3079 | + | assert_eq!(reader.get(&refs[0]).unwrap().1, chunk(0), "{which}"); |
|
| 3080 | + | assert_eq!(reader.pack_count(), 1, "the miss loaded the new pack"); |
|
| 3081 | + | let got = reader.get_many(&refs); |
|
| 3082 | + | assert!(got.iter().all(|g| g.is_ok()), "{which}"); |
|
| 3083 | + | // has, too, from another reader. |
|
| 3084 | + | let late = Store::open_on(writer.backend().clone(), writer.dir()).unwrap(); |
|
| 3085 | + | { |
|
| 3086 | + | let batch = writer.begin_batch().unwrap(); |
|
| 3087 | + | for i in 1000..(1000 + SPILL_OBJECTS + 1) { |
|
| 3088 | + | writer.put(ObjectType::Chunk, &chunk(i)).unwrap(); |
|
| 3089 | + | } |
|
| 3090 | + | batch.finish().unwrap(); |
|
| 3091 | + | } |
|
| 3092 | + | assert!(late.has(&hash_object(ObjectType::Chunk, &chunk(1000)))); |
|
| 3093 | + | }); |
|
| 3094 | + | } |
|
| 3095 | + | ||
| 3096 | + | #[test] |
|
| 3097 | + | fn retired_entries_expire_after_the_hour_and_are_never_loaded() { |
|
| 3098 | + | each_pair(|which, s, _| { |
|
| 3099 | + | for i in 0..10 { |
|
| 3100 | + | s.put(ObjectType::Chunk, &chunk(i)).unwrap(); |
|
| 3101 | + | } |
|
| 3102 | + | s.pack_loose().unwrap(); |
|
| 3103 | + | s.put(ObjectType::Chunk, &chunk(10)).unwrap(); |
|
| 3104 | + | s.repack_all(ZSTD_LEVEL).unwrap(); |
|
| 3105 | + | // Retired over an hour ago: the first pack and 10 loose objects. |
|
| 3106 | + | age_retired(&s, RETIRED_FOR + std::time::Duration::from_secs(60)); |
|
| 3107 | + | let old = retired_keys(&s); |
|
| 3108 | + | assert_eq!(old.len(), 12, "{which}: {old:?}"); |
|
| 3109 | + | // Retired just now: the second pack. |
|
| 3110 | + | s.put(ObjectType::Chunk, &chunk(11)).unwrap(); |
|
| 3111 | + | s.repack_all(ZSTD_LEVEL).unwrap(); |
|
| 3112 | + | assert_eq!(retired_keys(&s).len(), 14); |
|
| 3113 | + | let listed = s |
|
| 3114 | + | .backend() |
|
| 3115 | + | .list("pack-") |
|
| 3116 | + | .unwrap() |
|
| 3117 | + | .into_iter() |
|
| 3118 | + | .filter(|e| e.key.ends_with(".idx")) |
|
| 3119 | + | .count(); |
|
| 3120 | + | assert_eq!(listed, 3, "{which}"); |
|
| 3121 | + | let fresh = Store::open_on(s.backend().clone(), s.dir()).unwrap(); |
|
| 3122 | + | assert_eq!( |
|
| 3123 | + | fresh.pack_count(), |
|
| 3124 | + | 1, |
|
| 3125 | + | "{which}: retired packs are not loaded" |
|
| 3126 | + | ); |
|
| 3127 | + | let report = s.expire_retired(RETIRED_FOR, &mut silent()).unwrap(); |
|
| 3128 | + | assert_eq!( |
|
| 3129 | + | report, |
|
| 3130 | + | RetiredReport { |
|
| 3131 | + | packs: 1, |
|
| 3132 | + | loose: 11, |
|
| 3133 | + | kept: 2 |
|
| 3134 | + | }, |
|
| 3135 | + | "{which}" |
|
| 3136 | + | ); |
|
| 3137 | + | let young = retired_keys(&s); |
|
| 3138 | + | assert_eq!(young.len(), 2, "{which}: {young:?}"); |
|
| 3139 | + | assert!(young.iter().all(|k| !old.contains(k))); |
|
| 3140 | + | for k in old.iter().filter(|k| k.ends_with(".pack")) { |
|
| 3141 | + | assert!(!s.backend().has(k).unwrap(), "{which}: {k}"); |
|
| 3142 | + | } |
|
| 3143 | + | for k in &young { |
|
| 3144 | + | assert!(s.backend().has(k).unwrap(), "{which}: {k} kept"); |
|
| 3145 | + | } |
|
| 3146 | + | for i in 0..12 { |
|
| 3147 | + | let r = hash_object(ObjectType::Chunk, &chunk(i)); |
|
| 3148 | + | assert_eq!(fresh.get(&r).unwrap().1, chunk(i), "{which}"); |
|
| 3149 | + | } |
|
| 3150 | + | }); |
|
| 3151 | + | } |
|
| 3152 | + | ||
| 3153 | + | /// A retired loose object whose pack copy was pruned since is not |
|
| 3154 | + | /// lost when the retired files go: it is loose again. |
|
| 3155 | + | #[test] |
|
| 3156 | + | fn a_retired_loose_object_no_pack_holds_is_kept() { |
|
| 3157 | + | each_pair(|which, s, _| { |
|
| 3158 | + | let a = s.put(ObjectType::Chunk, b"a").unwrap(); |
|
| 3159 | + | let b = s.put(ObjectType::Chunk, b"b").unwrap(); |
|
| 3160 | + | s.pack_loose().unwrap(); |
|
| 3161 | + | assert_eq!(s.loose_count().unwrap(), 0); |
|
| 3162 | + | // The pack rewritten without b, as pruning does. |
|
| 3163 | + | let first = s.packs.borrow()[0].key.clone(); |
|
| 3164 | + | s.write_pack(&[a], &[], &[first], ZSTD_LEVEL, &mut silent()) |
|
| 3165 | + | .unwrap(); |
|
| 3166 | + | let report = s |
|
| 3167 | + | .expire_retired(std::time::Duration::ZERO, &mut silent()) |
|
| 3168 | + | .unwrap(); |
|
| 3169 | + | assert_eq!((report.packs, report.loose), (1, 1), "{which}: a's copy"); |
|
| 3170 | + | assert!(retired_keys(&s).is_empty()); |
|
| 3171 | + | assert_eq!(s.get(&b).unwrap().1, b"b", "{which}"); |
|
| 3172 | + | assert_eq!(s.get(&a).unwrap().1, b"a"); |
|
| 3173 | + | assert_eq!(s.loose_count().unwrap(), 1, "b is loose again"); |
|
| 3174 | + | // And pruning it, unreachable, deletes it. |
|
| 3175 | + | let keep = [a].into_iter().collect(); |
|
| 3176 | + | let r = s.prune(&keep, std::time::Duration::ZERO).unwrap(); |
|
| 3177 | + | assert_eq!(r.objects, 1); |
|
| 3178 | + | assert!(!s.has(&b)); |
|
| 3179 | + | }); |
|
| 3180 | + | } |
|
| 3181 | + | ||
| 3182 | + | /// A backend whose first pack listing names an index that is deleted |
|
| 3183 | + | /// before it is read, as a collection running meanwhile would. |
|
| 3184 | + | pub(crate) struct Racing { |
|
| 3185 | + | inner: MemoryStore, |
|
| 3186 | + | lists: std::sync::atomic::AtomicUsize, |
|
| 3187 | + | /// Deleted right after the first listing returns. |
|
| 3188 | + | pub(crate) doomed: std::sync::Mutex<Option<String>>, |
|
| 3189 | + | /// Named in every listing, never stored. |
|
| 3190 | + | ghost: bool, |
|
| 3191 | + | } |
|
| 3192 | + | ||
| 3193 | + | impl crate::strata::Store for Racing { |
|
| 3194 | + | fn describe(&self) -> String { |
|
| 3195 | + | "racing".into() |
|
| 3196 | + | } |
|
| 3197 | + | fn has(&self, k: &str) -> crate::strata::Result<bool> { |
|
| 3198 | + | self.inner.has(k) |
|
| 3199 | + | } |
|
| 3200 | + | fn get(&self, k: &str) -> crate::strata::Result<Vec<u8>> { |
|
| 3201 | + | self.inner.get(k) |
|
| 3202 | + | } |
|
| 3203 | + | fn get_range(&self, k: &str, r: std::ops::Range<u64>) -> crate::strata::Result<Vec<u8>> { |
|
| 3204 | + | self.inner.get_range(k, r) |
|
| 3205 | + | } |
|
| 3206 | + | fn put_new(&self, k: &str, b: &[u8]) -> crate::strata::Result<()> { |
|
| 3207 | + | self.inner.put_new(k, b) |
|
| 3208 | + | } |
|
| 3209 | + | fn list(&self, p: &str) -> crate::strata::Result<Vec<crate::strata::Entry>> { |
|
| 3210 | + | let mut out = self.inner.list(p)?; |
|
| 3211 | + | if p == "pack-" { |
|
| 3212 | + | self.lists |
|
| 3213 | + | .fetch_add(1, std::sync::atomic::Ordering::Relaxed); |
|
| 3214 | + | if let Some(k) = self.doomed.lock().unwrap().take() { |
|
| 3215 | + | self.inner.delete(&k)?; |
|
| 3216 | + | } |
|
| 3217 | + | if self.ghost { |
|
| 3218 | + | for k in ["pack-0000.idx", "pack-0000.pack"] { |
|
| 3219 | + | out.push(crate::strata::Entry { |
|
| 3220 | + | key: k.into(), |
|
| 3221 | + | size: 100, |
|
| 3222 | + | modified_ms: None, |
|
| 3223 | + | }); |
|
| 3224 | + | } |
|
| 3225 | + | out.sort_by(|a, b| a.key.cmp(&b.key)); |
|
| 3226 | + | } |
|
| 3227 | + | } |
|
| 3228 | + | Ok(out) |
|
| 3229 | + | } |
|
| 3230 | + | fn delete(&self, k: &str) -> crate::strata::Result<()> { |
|
| 3231 | + | self.inner.delete(k) |
|
| 3232 | + | } |
|
| 3233 | + | fn read_ref( |
|
| 3234 | + | &self, |
|
| 3235 | + | k: &str, |
|
| 3236 | + | ) -> crate::strata::Result<Option<(Vec<u8>, crate::strata::Version)>> { |
|
| 3237 | + | self.inner.read_ref(k) |
|
| 3238 | + | } |
|
| 3239 | + | fn write_ref( |
|
| 3240 | + | &self, |
|
| 3241 | + | k: &str, |
|
| 3242 | + | b: &[u8], |
|
| 3243 | + | e: Option<&crate::strata::Version>, |
|
| 3244 | + | ) -> crate::strata::Result<crate::strata::Version> { |
|
| 3245 | + | self.inner.write_ref(k, b, e) |
|
| 3246 | + | } |
|
| 3247 | + | } |
|
| 3248 | + | ||
| 3249 | + | pub(crate) fn racing(ghost: bool) -> Arc<Racing> { |
|
| 3250 | + | Arc::new(Racing { |
|
| 3251 | + | inner: MemoryStore::new(), |
|
| 3252 | + | lists: Default::default(), |
|
| 3253 | + | doomed: Default::default(), |
|
| 3254 | + | ghost, |
|
| 3255 | + | }) |
|
| 3256 | + | } |
|
| 3257 | + | ||
| 3258 | + | #[test] |
|
| 3259 | + | fn an_index_deleted_between_listing_and_loading_is_passed_over() { |
|
| 3260 | + | let dir = tempfile::tempdir().unwrap(); |
|
| 3261 | + | let backend = racing(false); |
|
| 3262 | + | let s = Store::open_on(backend.clone(), dir.path()).unwrap(); |
|
| 3263 | + | let a = s.put(ObjectType::Chunk, b"a").unwrap(); |
|
| 3264 | + | s.pack_loose().unwrap(); |
|
| 3265 | + | let b = s.put(ObjectType::Chunk, b"b").unwrap(); |
|
| 3266 | + | s.pack_loose().unwrap(); |
|
| 3267 | + | let idx = { |
|
| 3268 | + | let packs = s.packs.borrow(); |
|
| 3269 | + | let (i, _) = s.find_packed(&a).unwrap(); |
|
| 3270 | + | format!("{}.idx", packs[i].key.trim_end_matches(".pack")) |
|
| 3271 | + | }; |
|
| 3272 | + | *backend.doomed.lock().unwrap() = Some(idx); |
|
| 3273 | + | let before = backend.lists.load(std::sync::atomic::Ordering::Relaxed); |
|
| 3274 | + | let again = Store::open_on(backend.clone(), dir.path()).unwrap(); |
|
| 3275 | + | assert_eq!(again.pack_count(), 1, "the deleted one is passed over"); |
|
| 3276 | + | assert_eq!( |
|
| 3277 | + | backend.lists.load(std::sync::atomic::Ordering::Relaxed) - before, |
|
| 3278 | + | 2, |
|
| 3279 | + | "and the store listed again" |
|
| 3280 | + | ); |
|
| 3281 | + | assert_eq!(again.get(&b).unwrap().1, b"b"); |
|
| 3282 | + | // A listing that names an index never there opens too. |
|
| 3283 | + | let ghosted = racing(true); |
|
| 3284 | + | let s = Store::open_on(ghosted.clone(), dir.path().join("g")).unwrap(); |
|
| 3285 | + | let c = s.put(ObjectType::Chunk, b"c").unwrap(); |
|
| 3286 | + | s.pack_loose().unwrap(); |
|
| 3287 | + | let again = Store::open_on(ghosted, dir.path().join("g")).unwrap(); |
|
| 3288 | + | assert_eq!(again.pack_count(), 1); |
|
| 3289 | + | assert_eq!(again.get(&c).unwrap().1, b"c"); |
|
| 2541 | 3290 | } |
|
| 2542 | 3291 | } |
|
| 2543 | 3292 | ||
| … | 195 unchanged lines |
modified crates/oxy-strata/src/lib.rs +67 −18
| … | 484 unchanged lines |
||
| 485 | 485 | } |
|
| 486 | 486 | } |
|
| 487 | 487 | ||
| 488 | + | /// Run a request that writes or deletes `key`, forgetting what the caches |
|
| 489 | + | /// hold for it both before it is sent and after it completes, whatever it |
|
| 490 | + | /// returns: a listing or a miss taken by another thread while the request |
|
| 491 | + | /// was in flight may predate it, and would otherwise be kept for |
|
| 492 | + | /// `MISS_FOR` (oxygenhub#249). |
|
| 493 | + | fn writing<T>(misses: &Misses, listings: &Listings, key: &str, request: impl FnOnce() -> T) -> T { |
|
| 494 | + | misses.written(key); |
|
| 495 | + | listings.written(key); |
|
| 496 | + | let out = request(); |
|
| 497 | + | misses.written(key); |
|
| 498 | + | listings.written(key); |
|
| 499 | + | out |
|
| 500 | + | } |
|
| 501 | + | ||
| 488 | 502 | impl Drop for CloudStore { |
|
| 489 | 503 | /// Without waiting, so a store can be dropped inside a runtime too. |
|
| 490 | 504 | fn drop(&mut self) { |
|
| … | 255 unchanged lines |
||
| 746 | 760 | ||
| 747 | 761 | fn put_new(&self, key: &str, bytes: &[u8]) -> Result<()> { |
|
| 748 | 762 | let path = self.path(key)?; |
|
| 749 | - | self.misses.written(key); |
|
| 750 | - | self.listings.written(key); |
|
| 751 | 763 | let opts = PutOptions { |
|
| 752 | 764 | mode: PutMode::Create, |
|
| 753 | 765 | ..Default::default() |
|
| 754 | 766 | }; |
|
| 755 | 767 | counted(Op::Put); |
|
| 756 | - | self.block( |
|
| 757 | - | self.inner |
|
| 758 | - | .put_opts(&path, PutPayload::from(bytes.to_vec()), opts), |
|
| 759 | - | ) |
|
| 768 | + | writing(&self.misses, &self.listings, key, || { |
|
| 769 | + | self.block( |
|
| 770 | + | self.inner |
|
| 771 | + | .put_opts(&path, PutPayload::from(bytes.to_vec()), opts), |
|
| 772 | + | ) |
|
| 773 | + | }) |
|
| 760 | 774 | .map(|_| ()) |
|
| 761 | 775 | .map_err(|e| self.map(e, key)) |
|
| 762 | 776 | } |
|
| … | 57 unchanged lines |
||
| 820 | 834 | ||
| 821 | 835 | fn delete(&self, key: &str) -> Result<()> { |
|
| 822 | 836 | let path = self.path(key)?; |
|
| 823 | - | self.listings.written(key); |
|
| 824 | 837 | counted(Op::Delete); |
|
| 825 | - | match self.block(self.inner.delete(&path)) { |
|
| 838 | + | match writing(&self.misses, &self.listings, key, || { |
|
| 839 | + | self.block(self.inner.delete(&path)) |
|
| 840 | + | }) { |
|
| 826 | 841 | Ok(()) | Err(object_store::Error::NotFound { .. }) => Ok(()), |
|
| 827 | 842 | Err(e) => Err(self.map(e, key)), |
|
| 828 | 843 | } |
|
| … | 20 unchanged lines |
||
| 849 | 864 | ||
| 850 | 865 | fn write_ref(&self, key: &str, bytes: &[u8], expected: Option<&Version>) -> Result<Version> { |
|
| 851 | 866 | let path = self.path(key)?; |
|
| 852 | - | self.misses.written(key); |
|
| 853 | - | self.listings.written(key); |
|
| 854 | 867 | let mode = match expected { |
|
| 855 | 868 | None => PutMode::Create, |
|
| 856 | 869 | Some(v) => PutMode::Update(Self::update_version(v)), |
|
| … | 3 unchanged lines |
||
| 860 | 873 | ..Default::default() |
|
| 861 | 874 | }; |
|
| 862 | 875 | counted(Op::WriteRef); |
|
| 863 | - | let put = self |
|
| 864 | - | .block( |
|
| 876 | + | let put = writing(&self.misses, &self.listings, key, || { |
|
| 877 | + | self.block( |
|
| 865 | 878 | self.inner |
|
| 866 | 879 | .put_opts(&path, PutPayload::from(bytes.to_vec()), opts), |
|
| 867 | 880 | ) |
|
| 868 | - | .map_err(|e| match self.map(e, key) { |
|
| 869 | - | // A record that exists when none was expected is a lost |
|
| 870 | - | // race too, not a misuse of put_new. |
|
| 871 | - | Error::Exists(k) => Error::Conflict(k), |
|
| 872 | - | other => other, |
|
| 873 | - | })?; |
|
| 881 | + | }) |
|
| 882 | + | .map_err(|e| match self.map(e, key) { |
|
| 883 | + | // A record that exists when none was expected is a lost |
|
| 884 | + | // race too, not a misuse of put_new. |
|
| 885 | + | Error::Exists(k) => Error::Conflict(k), |
|
| 886 | + | other => other, |
|
| 887 | + | })?; |
|
| 874 | 888 | Ok(Self::version_of(put.e_tag, put.version)) |
|
| 875 | 889 | } |
|
| 876 | 890 | } |
|
| … | 39 unchanged lines |
||
| 916 | 930 | l.put("pack-", &[e("pack-1.idx")]); |
|
| 917 | 931 | l.0.lock().unwrap().get_mut("pack-").unwrap().0 -= super::MISS_FOR; |
|
| 918 | 932 | assert!(l.get("pack-").is_none(), "and it ages out"); |
|
| 933 | + | } |
|
| 934 | + | ||
| 935 | + | /// A listing or miss taken by another thread while a write is in |
|
| 936 | + | /// flight is forgotten when the write completes, not kept stale for |
|
| 937 | + | /// MISS_FOR (oxygenhub#249), whether the write succeeded or not. |
|
| 938 | + | #[test] |
|
| 939 | + | fn a_listing_taken_during_a_write_is_forgotten_after_it() { |
|
| 940 | + | let (m, l) = (super::Misses::default(), super::Listings::default()); |
|
| 941 | + | let e = |k: &str| oxy_core::strata::Entry { |
|
| 942 | + | key: k.into(), |
|
| 943 | + | size: 1, |
|
| 944 | + | modified_ms: None, |
|
| 945 | + | }; |
|
| 946 | + | l.put("pack-", &[e("pack-1.idx")]); |
|
| 947 | + | let got = super::writing(&m, &l, "pack-2.idx", || { |
|
| 948 | + | assert!(l.get("pack-").is_none(), "forgotten before it is sent"); |
|
| 949 | + | // Another thread lists, and misses, mid-request. |
|
| 950 | + | l.put("pack-", &[e("pack-1.idx")]); |
|
| 951 | + | m.missed("pack-2.idx"); |
|
| 952 | + | 7 |
|
| 953 | + | }); |
|
| 954 | + | assert_eq!(got, 7); |
|
| 955 | + | assert!(l.get("pack-").is_none(), "the mid-request listing is gone"); |
|
| 956 | + | assert!(!m.known("pack-2.idx"), "and the miss"); |
|
| 957 | + | // A delete that fails is treated the same. |
|
| 958 | + | l.put("pack-", &[e("pack-1.idx")]); |
|
| 959 | + | let r: std::result::Result<(), &str> = super::writing(&m, &l, "pack-1.idx", || { |
|
| 960 | + | l.put("pack-", &[e("pack-1.idx")]); |
|
| 961 | + | Err("network") |
|
| 962 | + | }); |
|
| 963 | + | assert!(r.is_err() && l.get("pack-").is_none()); |
|
| 964 | + | // Other prefixes' writes leave the pack listing. |
|
| 965 | + | l.put("pack-", &[e("pack-1.idx")]); |
|
| 966 | + | super::writing(&m, &l, "loose/ab/cd", || ()); |
|
| 967 | + | assert!(l.get("pack-").is_some()); |
|
| 919 | 968 | } |
|
| 920 | 969 | ||
| 921 | 970 | #[test] |
|
| … | 121 unchanged lines |
modified docs/spec.md +9 −1
| … | 481 unchanged lines |
||
| 482 | 482 | ||
| 483 | 483 | `oxy undo` undoes **this workspace's** last command (oxygen#99): the newest operation recorded by this workspace (for operations older than the `workspace` field, one that moved only other workspaces' working copies is theirs), passing over working-copy snapshots, pushes (the remote keeps what it was sent) and commands an earlier undo reverted, and taking a command's own `(checkout)` with it. When no other workspace recorded an operation since, the head moves back to the command's parent. Otherwise the command is reverted in place as a new operation naming it in `undoes`: each reference it changed returns to its value before it; a bookmark another workspace moved since is put back head by head (the heads the command added go, those it replaced return, the other workspace's stay); and a reference that cannot be put back so (a later change builds on one the command added, or another workspace changed the same tag, remote reference, policy or working copy) refuses the undo, naming it. Undo reports the command it undid, what that changed, the other workspaces' operations it kept, and bookmarks whose undone changes a remote already holds. `oxy redo` and `oxy op log` expose the DAG. Operations are retained for `core.op-retention` (default 90 days) and then eligible for garbage collection. |
|
| 484 | 484 | ||
| 485 | + | A process reads the operation head once, when it first needs it, and answers every later read of references, bookmarks and the view from that operation (oxygenhub#249): one command, or one request a host serves, never mixes two heads, and never waits on another process's operation to read. Its own operations move it to the head each one recorded. It reads the head again only where it must see what others did meanwhile: once it holds the working-copy lock (whoever held the lock may have moved the head), when a host receives a push (the updates are checked against the head as it is now), in `oxy fsck`, and wherever the implementation asks for it explicitly (`Repo::refresh`). An operation recorded from an older head merges with the newer one as any concurrent operation does. |
|
| 486 | + | ||
| 485 | 487 | Because all mutation goes through the operation log and object writes are idempotent, no long-held repository lock is needed: the operation head's lock covers one pointer write. Each workspace has a **working-copy lock** (`.oxy/workspaces/<id>/lock`), held by every command that reads or writes the working copy and replaces its working-copy change (snapshot, checkout, bookmark switch, merge, commit, amend and rewrites, discard, pull), from reading the operation head until its operation is recorded. A snapshot therefore never sees a half-written checkout, and never records one as edits. The lock is re-entrant within a process thread; a waiting command gives up after a minute and says why; a lock whose process is gone is taken over, and where that cannot be told, one older than a minute. |
|
| 486 | 488 | ||
| 487 | 489 | ### 12.2 Workspaces |
|
| … | 29 unchanged lines |
||
| 517 | 519 | objects/ |
|
| 518 | 520 | pack-<id>.pack append-only pack file |
|
| 519 | 521 | pack-<id>.idx sorted (ObjectRef → offset, length, type) index |
|
| 522 | + | pack-<id>.dict the pack's zstd dictionary, when a full repack trained one |
|
| 523 | + | pack-retired packs and loose objects a repack replaced, kept for readers (§13.4) |
|
| 520 | 524 | loose/ objects written since the last pack, as header || body |
|
| 521 | 525 | sigs/ signature lists keyed by the ObjectRef they cover (§5.4) |
|
| 522 | 526 | ops/ operation objects and op heads |
|
| … | 29 unchanged lines |
||
| 552 | 556 | ### 13.4 Packing |
|
| 553 | 557 | ||
| 554 | 558 | An automatic trigger when the loose object count exceeds 4096 writes loose objects into a new pack and merges packs when there are more than 16; `oxy gc` repacks every surviving object into one pack, compressed harder (zstd level 9, against level 3 for everyday writes) and on every core, as `git gc` repacks. A store that is already one such pack (it has a dictionary) with nothing loose is left as it is: repacking it would write the same pack. Writers SHOULD batch: a snapshot, a fetched bundle, a Git import, or a checkpoint is one write batch, and a batch that grows beyond a small threshold (256 objects or 8 MiB) is written as a single pack with one `fsync` rather than as loose objects, each with its own; smaller batches stay loose so ordinary commits do not multiply packs. Objects of an open batch MUST be readable by the writer before the batch completes. Pack indexes MUST support lookup by ObjectRef prefix and MUST be memory-mappable. A repository with two million objects MUST be openable with under 64 MiB of resident index memory on the Full profile. |
|
| 559 | + | ||
| 560 | + | Readers never wait for writers (oxygenhub#249). A store loads its list of packs when it opens and keeps it; a read that misses is not believed at once. When the pack an object was found in is gone, or its dictionary is, the list is loaded again and the read tried once more; when the object is in no pack and not loose, the store lists its packs, and if the listing differs from the one it loaded from, loads the new packs and tries once more. Only then is the object missing (`has` answers false). A pack whose index is gone between the listing and the read, as when a collection deletes it meanwhile, is passed over and the store listed once more, so opening a repository never fails for it. A writer asking whether it already holds many objects lists the packs at most once a second while nothing new turns up. |
|
| 561 | + | ||
| 562 | + | A repack does not delete what it replaces. The packs it folded in and the loose objects it packed are **retired**: recorded, with the time, in `pack-retired`, a reference record of lines `<wall ms> <key>` updated by compare-and-swap, so concurrent repacks and collections never lose each other's lines. A store opening, or loading its packs again, does not load a retired pack, and retired loose objects are not counted or packed again as loose; a reader that loaded its packs before the repack reads on from the old files. Each `oxy gc`, and each automatic pack check after a commit, first deletes what was retired at least an hour ago: each such pack with its index and dictionary, and each such loose object a live pack holds. A retired loose object no live pack holds (its pack copy was pruned since) is kept as an ordinary loose object. Readers that loaded their packs before the files went find the objects in the new pack by the reload above. The hour is independent of `core.gc-grace` (§21): `oxy gc --now` waives the grace period for unreachable objects, not the hour readers have. |
|
| 555 | 563 | ||
| 556 | 564 | The pack and index formats are fixed so that tools can read them without the implementation: |
|
| 557 | 565 | ||
| … | 280 unchanged lines |
||
| 838 | 846 | ||
| 839 | 847 | ## 21. Garbage collection |
|
| 840 | 848 | ||
| 841 | - | An object is **reachable** if it is referenced, directly or transitively, from any bookmark, tag, policy, the current operation heads, any operation newer than `core.op-retention`, any workspace's current working-copy change, the working-copy changes of operations newer than `core.snapshot-retention` (default 7 days; `oxy gc --expire-snapshots` keeps none but the current ones, oxygen#98), or any checkpoint of a reachable state. A file that was only ever in a working copy, never committed, so goes once the snapshots that held it expire. `oxy gc` deletes unreachable objects that are older than `core.gc-grace` (default 14 days, to protect concurrent writers) and repacks. Tombstone atoms are never collected independently of their change. Divergent changes are reachable if any of their variants is. A change's text blobs and their chunks are reachable with it (§7.4). A seal is reachable if any change it covers is reachable, and reference attestations are reachable while they are the latest for their reference or are within `core.op-retention`. |
|
| 849 | + | An object is **reachable** if it is referenced, directly or transitively, from any bookmark, tag, policy, the current operation heads, any operation newer than `core.op-retention`, any workspace's current working-copy change, the working-copy changes of operations newer than `core.snapshot-retention` (default 7 days; `oxy gc --expire-snapshots` keeps none but the current ones, oxygen#98), or any checkpoint of a reachable state. A file that was only ever in a working copy, never committed, so goes once the snapshots that held it expire. `oxy gc` first deletes the packs and loose objects earlier repacks retired over an hour ago (§13.4), then deletes unreachable objects that are older than `core.gc-grace` (default 14 days, to protect concurrent writers) and repacks; what this repack replaces is retired in turn, for a later collection. Tombstone atoms are never collected independently of their change. Divergent changes are reachable if any of their variants is. A change's text blobs and their chunks are reachable with it (§7.4). A seal is reachable if any change it covers is reachable, and reference attestations are reachable while they are the latest for their reference or are within `core.op-retention`. |
|
| 842 | 850 | ||
| 843 | 851 | ### 21.1 Integrity check |
|
| 844 | 852 | ||
| … | 213 unchanged lines |
modified src/cli/workspace.rs +24 −0
| … | 1690 unchanged lines |
||
| 1691 | 1691 | }, |
|
| 1692 | 1692 | plural(repo.store().pack_count(), "pack") |
|
| 1693 | 1693 | ); |
|
| 1694 | + | // What earlier repacks replaced stays an hour for readers still using |
|
| 1695 | + | // it (oxygenhub#249); a collection after that deletes it. |
|
| 1696 | + | let kept = if r.retired.kept > 0 { |
|
| 1697 | + | format!( |
|
| 1698 | + | "{} retired less than an hour ago kept for readers", |
|
| 1699 | + | plural(r.retired.kept, "file") |
|
| 1700 | + | ) |
|
| 1701 | + | } else { |
|
| 1702 | + | String::new() |
|
| 1703 | + | }; |
|
| 1704 | + | if r.retired.packs + r.retired.loose > 0 { |
|
| 1705 | + | println!( |
|
| 1706 | + | "Deleted {} and {} retired by earlier repacks{}.", |
|
| 1707 | + | plural(r.retired.packs, "pack"), |
|
| 1708 | + | plural(r.retired.loose, "loose object"), |
|
| 1709 | + | if kept.is_empty() { |
|
| 1710 | + | String::new() |
|
| 1711 | + | } else { |
|
| 1712 | + | format!("; {kept}") |
|
| 1713 | + | } |
|
| 1714 | + | ); |
|
| 1715 | + | } else if !kept.is_empty() { |
|
| 1716 | + | println!("{kept}."); |
|
| 1717 | + | } |
|
| 1694 | 1718 | Ok(0) |
|
| 1695 | 1719 | } |
|
| 1696 | 1720 | ||
| … | 1204 unchanged lines |