OXYGENHUB

Readers never wait: one head per handle, retry on a miss, retire before deleting 767ac6cd0545

oxygenhub#249: pages stalled for up to 43 s behind a repository's gc or
push, because reads were only safe under the repository's lock. The store
now gives readers what they need to read alongside writers:

- A handle reads the operation head once and keeps it (its own
  operations move it; refresh() reads again), so one request never mixes
  two views. Compare-and-swap is unchanged: a stale handle's operation
  merges with the newer head. A push, fsck, a strata mirror's sync and
  taking the working-copy lock read the head afresh.
- A read that misses reloads the pack list and tries once more (an
  object a deleted pack held, a dictionary gone, a pack written since);
  has() re-lists at most once a second. Loading the pack list passes over
  an index that vanished between listing and loading, so a repository
  always opens.
- A repack retires the packs and loose objects it replaces in a
  pack-retired record instead of deleting them; new readers don't load
  retired packs, and gc (and autopack) deletes what was retired over an
  hour before. A retired loose object no live pack holds is kept.
- oxy-strata forgets its cached pack listing and misses for a key both
  before and after writing it, so a listing taken mid-write is not kept
  stale for 30 s.

Tests for each race the survey found (a reader across a repack, an index
gone at open, a spilled push, one head per handle, the hour, the stale
listing); 14 existing tests that hold one handle across another's push now
refresh it. oxy gc reports what it deleted or kept for readers.

Refs andymccall/oxygenhub#249
Author
Andy McCall @andymccall <[email protected]>
via
agent agent:claude@andymccall <[email protected]> · keyless, under your grant for changes,refs,seal · key f3704baa843e
When
· 2026-10-18 14:22 UTC
Signature
◈ sealed by agent:claude@andymccall
Hash
767ac6cd0545ed1a1ef3f45bbfdd1bebcff222df95616f20491d0bb68c188570
Depends on
092457020692 132b884b9a8d 135212de5370 19555025a2ac 1e0fbb47ab90 1ec3f008e96a 2863456a0f6d 28fc23d7706a 2b872d074ddb 387335ff7f4d 4e88e774777a 530a0fc650a2 54cd08a1e7bf 5b1114a04905 5cde7882d767 648189388f1a 6caacfff1d4f 7421309f5372 75167672ebcd 7aa2dc9369e3 7b9809e67c3d 7d9dab131937 7e0e8e5ad882 83e0677b07fe 84b9e9ac0ab4 87c948720343 8b53b935dd03 8d80356371c5 8f10a07a1dbe 982864068a59 98a1620a3840 9ae4d15a0e4b 9d580f2132f3 9f9e72b93e4e a17fc622ca43 a67914bced23 a9f686edf60b afaba5f1306d b689779e9e74 ba160f60219e c9369ad8b1b1 d02e545dc265 da652d6392d8 dd21fd82687d e5d9b0513f3e f2f91ceb0e9f f8c695764c95 fce44ae9ad29 ff0af1ee299d ff1a64f81c1c
Depended on by
b5e98094eb8f 2452b4407ab7 795b8b502ded 1ec6fdbc6bcb ef7d4ea6f33b 8b3c9b2ba502
In bookmarks
carried-not-removed doctor-trust identical-sides keyless-tags main merge-remote-bookmark paths-for-resolutions-and-amend push-notes show-stat subrepo-pin-state
In tags
v0.10.0 v0.11.0 v0.5.0 v0.5.1 v0.6.0 v0.7.0 v0.8.0 v0.9.0
Edits
124 atoms, 0 tree operations, 12 files

Browse the tree at this change

modified crates/oxy-core/src/repo/fsck.rs +2 −0
…
108 unchanged lines
109 109
impl Repo {
110 110
    /// Check the whole repository; see the module documentation.
111 111
    pub fn fsck(&self) -> Result<FsckReport> {
112 +
        // The head as it is now, as the operations pass reads it.
113 +
        self.forget_head();
112 114
        let mut rep = FsckReport::default();
113 115
        let bad = {
114 116
            let _t = crate::timing::phase("fsck: objects");
…
398 unchanged lines
modified crates/oxy-core/src/repo/gc.rs +16 −3
…
9 unchanged lines
10 10
//! that far back may fail once its objects are gone.
11 11
12 12
use super::*;
13 -
use crate::store::{PackReport, PruneReport};
13 +
use crate::store::{PackReport, PruneReport, RetiredReport};
14 14
use std::collections::HashSet;
15 15
16 16
#[derive(Debug, Clone, Default, PartialEq, Eq)]
…
3 unchanged lines
20 20
    pub reachable: usize,
21 21
    /// Packs folded into the one a full repack writes.
22 22
    pub merged_packs: usize,
23 +
    /// Packs and loose objects earlier repacks retired, deleted now that
24 +
    /// [`crate::store::RETIRED_FOR`] has passed (oxygenhub#249).
25 +
    pub retired: RetiredReport,
23 26
}
24 27
25 28
impl Repo {
26 -
    /// Prune unreachable objects older than the grace period (`now`
27 -
    /// ignores it), then repack everything into one pack (spec §13.4).
29 +
    /// Delete what earlier repacks retired more than
30 +
    /// [`crate::store::RETIRED_FOR`] ago, prune unreachable objects older
31 +
    /// than the grace period (`now` ignores it, but not the retired
32 +
    /// files' hour), then repack everything into one pack (spec §13.4).
33 +
    /// The packs and loose objects this repack replaces are retired in
34 +
    /// turn, for the next collection.
28 35
    pub fn gc(&self, now: bool) -> Result<GcReport> {
29 36
        self.gc_with(now, &mut |_| {})
30 37
    }
…
14 unchanged lines
45 52
        extra: &[ObjectRef],
46 53
        progress: crate::store::Progress,
47 54
    ) -> Result<GcReport> {
55 +
        // Retired files first: the readers that used them have had their
56 +
        // hour, and what replaced them is in the packs read from here on.
57 +
        let retired = self
58 +
            .store
59 +
            .expire_retired(crate::store::RETIRED_FOR, progress)?;
48 60
        progress("finding what is reachable");
49 61
        let mut keep = self.reachable()?;
50 62
        for r in extra {
…
22 unchanged lines
73 85
            pruned,
74 86
            reachable: keep.len(),
75 87
            merged_packs,
88 +
            retired,
76 89
        })
77 90
    }
78 91
…
134 unchanged lines
modified crates/oxy-core/src/repo/hosted.rs +6 −1
…
631 unchanged lines
632 632
        user.delegate_add(agent_id.clone(), Scope::CHANGES, None)
633 633
            .unwrap();
634 634
        assert!(user.push("origin", false).unwrap().iter().all(|x| x.ok));
635 +
        host.refresh().unwrap();
635 636
        let owner = {
636 637
            let _ = owner;
637 638
            user.config.author_key.unwrap()
…
23 unchanged lines
661 662
        // The agent cannot push: its attestation of the reference is
662 663
        // refused under the policy. The owner can.
663 664
        let r = as_agent.push("origin", false).unwrap();
665 +
        host.refresh().unwrap();
664 666
        assert!(r.iter().all(|x| !x.ok), "{r:?}");
665 667
        assert!(!host.bookmark_heads().unwrap().contains(&h));
666 668
        // The owner's own handle (its key directory was fixed at `setup`;
667 -
        // the environment's `OXY_HOME` may have moved since).
669 +
        // the environment's `OXY_HOME` may have moved since), which reads
670 +
        // the head again to see the agent's commit.
671 +
        user.refresh().unwrap();
668 672
        let r = user.push("origin", false).unwrap();
673 +
        host.refresh().unwrap();
669 674
        assert!(r.iter().all(|x| x.ok), "{r:?}");
670 675
        assert!(host.bookmark_heads().unwrap().contains(&h));
671 676
        assert!(matches!(
…
133 unchanged lines
modified crates/oxy-core/src/repo/mod.rs +45 −3
…
279 unchanged lines
280 280
    /// `--at-op`: read the repository as it was after this operation;
281 281
    /// recording an operation is refused.
282 282
    at_op: Option<ObjectRef>,
283 +
    /// The operation head this handle reads (oxygenhub#249): read once, on
284 +
    /// first use, and moved only by this handle's own operations or by
285 +
    /// [`Repo::refresh`], so one command or request never mixes two heads.
286 +
    head_seen: std::cell::Cell<Option<ObjectRef>>,
283 287
    /// The directories (with their times) and nested repositories of the
284 288
    /// last walk, for the manifest written from it.
285 289
    pub(crate) walked_dirs: std::cell::RefCell<Option<WalkedDirs>>,
…
90 unchanged lines
376 380
        let repo = Repo {
377 381
            last_change: Default::default(),
378 382
            at_op: None,
383 +
            head_seen: Default::default(),
379 384
            walked_dirs: Default::default(),
380 385
            listed_dirs: Default::default(),
381 386
            git_export_skipped: Default::default(),
…
59 unchanged lines
441 446
        let repo = Repo {
442 447
            last_change: Default::default(),
443 448
            at_op: None,
449 +
            head_seen: Default::default(),
444 450
            walked_dirs: Default::default(),
445 451
            listed_dirs: Default::default(),
446 452
            git_export_skipped: Default::default(),
…
73 unchanged lines
520 526
                let mut repo = Repo {
521 527
                    last_change: Default::default(),
522 528
                    at_op: None,
529 +
                    head_seen: Default::default(),
523 530
                    walked_dirs: Default::default(),
524 531
                    listed_dirs: Default::default(),
525 532
                    git_export_skipped: Default::default(),
…
28 unchanged lines
554 561
                let mut repo = Repo {
555 562
                    last_change: Default::default(),
556 563
                    at_op: None,
564 +
                    head_seen: Default::default(),
557 565
                    walked_dirs: Default::default(),
558 566
                    listed_dirs: Default::default(),
559 567
                    git_export_skipped: Default::default(),
…
69 unchanged lines
629 637
        Ok(())
630 638
    }
631 639
632 -
    /// Pack automatically once many loose objects have accumulated.
640 +
    /// Pack automatically once many loose objects have accumulated, and
641 +
    /// delete what packing retired over [`crate::store::RETIRED_FOR`] ago
642 +
    /// (a store nobody runs `oxy gc` on would keep both copies forever).
633 643
    pub(crate) fn maybe_autopack(&self) -> Result<()> {
644 +
        self.store
645 +
            .expire_retired(crate::store::RETIRED_FOR, &mut |_| {})?;
634 646
        if self.store.loose_count()? > AUTOPACK_THRESHOLD {
635 647
            self.store.pack_loose()?;
636 648
            self.store.merge_packs()?;
…
209 unchanged lines
846 858
        )
847 859
    }
848 860
861 +
    /// The operation head this handle reads: the one it first read, or
862 +
    /// the last one it moved the head to (see [`Repo::refresh`]).
849 863
    fn head_ref(&self) -> Result<ObjectRef> {
850 -
        Ok(self.read_head()?.0)
864 +
        if let Some(r) = self.head_seen.get() {
865 +
            return Ok(r);
866 +
        }
867 +
        let r = self.read_head()?.0;
868 +
        self.head_seen.set(Some(r));
869 +
        Ok(r)
870 +
    }
871 +
872 +
    /// Read the operation head again: what another process or handle
873 +
    /// recorded since this handle first read it becomes visible. A handle
874 +
    /// keeps the head it first read for the rest of its life otherwise,
875 +
    /// so whatever it reads comes from one view (oxygenhub#249); a loop
876 +
    /// that waits for others' operations, or a step that must see what
877 +
    /// another handle just recorded, refreshes. Packs written since are
878 +
    /// picked up too.
879 +
    pub fn refresh(&self) -> Result<()> {
880 +
        self.forget_head();
881 +
        self.store.refresh()?;
882 +
        Ok(())
883 +
    }
884 +
885 +
    /// Read the operation head again on next use, as [`Repo::refresh`]
886 +
    /// does, leaving the packs as they are.
887 +
    pub(crate) fn forget_head(&self) {
888 +
        self.head_seen.set(None);
851 889
    }
852 890
853 891
    /// The operation head, and on a store its record's version for
…
36 unchanged lines
890 928
    fn swap_head(&self, to: &ObjectRef, expected: Option<&crate::strata::Version>) -> Result<bool> {
891 929
        if self.store.is_local() {
892 930
            write_atomic(&self.oxy.join("ops").join("head"), to.to_hex().as_bytes())?;
931 +
            self.head_seen.set(Some(*to));
893 932
            return Ok(true);
894 933
        }
895 934
        match self
…
1 unchanged lines
897 936
            .backend()
898 937
            .write_ref(HEAD_RECORD, to.to_hex().as_bytes(), expected)
899 938
        {
900 -
            Ok(_) => Ok(true),
939 +
            Ok(_) => {
940 +
                self.head_seen.set(Some(*to));
941 +
                Ok(true)
942 +
            }
901 943
            Err(crate::strata::Error::Conflict(_)) => Ok(false),
902 944
            Err(e) => fail(format!("move the operation head: {e}")),
903 945
        }
…
1806 unchanged lines
modified crates/oxy-core/src/repo/strata.rs +3 −0
…
232 unchanged lines
233 233
            None => (RefsResponse::default(), None),
234 234
        };
235 235
        self.with_mirror(|mirror| {
236 +
            // Another process may have brought the mirror level since this
237 +
            // handle read its head.
238 +
            mirror.refresh()?;
236 239
            // The policy first: bundles are then imported under it.
237 240
            if let Some(sp) = &resp.policy {
238 241
                let have = mirror.current_policy()?.map(|(_, p)| p.policy.serial);
…
1120 unchanged lines
modified crates/oxy-core/src/repo/sync.rs +4 −0
…
653 unchanged lines
654 654
        policy: Option<SignedPolicy>,
655 655
        tags: Vec<crate::protocol::TagRef>,
656 656
    ) -> Result<PushReport> {
657 +
        // Updates are checked against the head as it is now, not as this
658 +
        // handle first read it (oxygenhub#249): another push may have
659 +
        // landed since.
660 +
        self.refresh()?;
657 661
        let mut report = PushReport::default();
658 662
        let results = &mut report.results;
659 663
        if let Some(sp) = policy {
…
1514 unchanged lines
modified crates/oxy-core/src/repo/tests.rs +202 −0
…
505 unchanged lines
506 506
    write(&root.join("a"), "f.txt", b"a\n");
507 507
    let h1 = a.commit("first", &[]).unwrap();
508 508
    let results = a.push("origin", false).unwrap();
509 +
    origin.refresh().unwrap();
509 510
    assert!(results[0].ok, "{}", results[0].reason);
510 511
    assert_eq!(origin.bookmark_heads().unwrap(), vec![h1]);
511 512
    // The change was sealed at push and the reference attested.
…
14 unchanged lines
526 527
    let first = serial_of(&att);
527 528
    assert!(first > 1_000_000_000_000, "{first}");
528 529
    let again = a.push("origin", false).unwrap();
530 +
    origin.refresh().unwrap();
529 531
    assert!(again[0].ok && again[0].reason.contains("up to date"));
530 532
    let (_, att) = origin.latest_ref_attest("main").unwrap().unwrap();
531 533
    assert_eq!(serial_of(&att), first, "a no-op push does not attest again");
…
11 unchanged lines
543 545
    write(&root.join("b"), "f.txt", b"a\nb\n");
544 546
    let h2 = b.commit("second", &[]).unwrap();
545 547
    assert!(b.push("origin", false).unwrap()[0].ok);
548 +
    origin.refresh().unwrap();
546 549
    assert_eq!(origin.bookmark_heads().unwrap(), vec![h2]);
547 550
548 551
    // A commits an independent change; push is refused until A pulls.
…
8 unchanged lines
557 560
    assert_eq!(fs::read(root.join("a/f.txt")).unwrap(), b"a\nb\n");
558 561
    assert_eq!(fs::read(root.join("a/g.txt")).unwrap(), b"g\n");
559 562
    let results = a.push("origin", false).unwrap();
563 +
    origin.refresh().unwrap();
560 564
    assert!(results[0].ok, "{}", results[0].reason);
561 565
    let mut origin_heads = origin.bookmark_heads().unwrap();
562 566
    origin_heads.sort();
…
11 unchanged lines
574 578
    write(&root.join("a"), "f.txt", b"a\nA\n");
575 579
    a.commit("a-edit", &[]).unwrap();
576 580
    a.push("origin", false).unwrap();
581 +
    origin.refresh().unwrap();
577 582
    write(&root.join("b"), "f.txt", b"a\nB\n");
578 583
    b.commit("b-edit", &[]).unwrap();
579 584
    let pr = b.pull("origin").unwrap();
…
7 unchanged lines
587 592
        ChangeKind::Resolution
588 593
    );
589 594
    assert!(b.push("origin", false).unwrap()[0].ok);
595 +
    origin.refresh().unwrap();
590 596
    a.pull("origin").unwrap();
591 597
    assert_eq!(fs::read(root.join("a/f.txt")).unwrap(), b"a\nAB\n");
592 598
    // The resolution depends on the conflicting edits, but nothing depends
…
390 unchanged lines
983 989
    write(&root.join("b"), "b.txt", b"b\n");
984 990
    b.commit("unsigned before policy", &[]).unwrap();
985 991
    assert!(b.push("origin", false).unwrap()[0].ok);
992 +
    origin.refresh().unwrap();
986 993
987 994
    // A creates the policy and pushes it; origin now enforces it.
988 995
    write(&root.join("a"), "a.txt", b"a\n");
…
2 unchanged lines
991 998
    assert_eq!(sp.policy.serial, 1);
992 999
    a.pull("origin").unwrap();
993 1000
    let results = a.push("origin", false).unwrap();
1001 +
    origin.refresh().unwrap();
994 1002
    assert!(results.iter().all(|r| r.ok), "{results:?}");
995 1003
    assert_eq!(origin.current_policy().unwrap().unwrap().1.policy.serial, 1);
996 1004
…
12 unchanged lines
1009 1017
    let h_b2 = b.commit("b2 as Bob", &[]).unwrap();
1010 1018
    assert_eq!(b.store().get_change(&h_b2).unwrap().author, key_b);
1011 1019
    let r = b.push("origin", false).unwrap();
1020 +
    origin.refresh().unwrap();
1012 1021
    assert!(
1013 1022
        !r[0].ok && r[0].reason.contains("may not update"),
1014 1023
        "{:?}",
…
4 unchanged lines
1019 1028
    let sp = a.policy_grant(id_b.clone(), None).unwrap();
1020 1029
    assert_eq!(sp.policy.serial, 2);
1021 1030
    let r = a.push("origin", false).unwrap();
1031 +
    origin.refresh().unwrap();
1022 1032
    assert!(r.iter().any(|x| x.name == "policy" && x.ok), "{r:?}");
1023 1033
    // B must fetch the new policy before pushing.
1024 1034
    assert!(b
…
8 unchanged lines
1033 1043
    b.pull("origin").unwrap();
1034 1044
    assert_eq!(b.current_policy().unwrap().unwrap().1.policy.serial, 2);
1035 1045
    let r = b.push("origin", false).unwrap();
1046 +
    origin.refresh().unwrap();
1036 1047
    assert!(r[0].ok, "{:?}", r);
1037 1048
    // Origin verified B's change under the policy.
1038 1049
    let heads = origin.bookmark_heads().unwrap();
…
36 unchanged lines
1075 1086
    a.pull("origin").unwrap();
1076 1087
    a.policy_revoke(&key_b).unwrap();
1077 1088
    let r = a.push("origin", false).unwrap();
1089 +
    origin.refresh().unwrap();
1078 1090
    assert!(r.iter().any(|x| x.name == "policy" && x.ok), "{r:?}");
1079 1091
    b.pull("origin").unwrap();
1080 1092
    write(&root.join("b"), "b.txt", b"b3\n");
1081 1093
    b.commit("b3", &[]).unwrap();
1082 1094
    let r = b.push("origin", false).unwrap();
1095 +
    origin.refresh().unwrap();
1083 1096
    assert!(!r[0].ok, "{:?}", r);
1084 1097
1085 1098
    // A clone by a third party receives and verifies everything.
…
897 unchanged lines
1983 1996
        "the state still has everything"
1984 1997
    );
1985 1998
    assert!(a.push("origin", false).unwrap()[0].ok);
1999 +
    origin.refresh().unwrap();
1986 2000
    // Files created outside the scope are ignored by status.
1987 2001
    write(&root.join("a"), "docs/new.md", b"ignored\n");
1988 2002
    assert!(a.snapshot("status").unwrap().statuses.is_empty());
…
77 unchanged lines
2066 2080
    write(&root.join("a"), "a.txt", b"0\n1\n2\n3\n4\n5\n6\n7\n8\n");
2067 2081
    let h_new = a.commit("c8", &[]).unwrap();
2068 2082
    assert!(a.push("origin", false).unwrap()[0].ok);
2083 +
    origin.refresh().unwrap();
2069 2084
    assert_eq!(origin.bookmark_heads().unwrap(), vec![h_new]);
2070 2085
    // Fetching new work from origin into the shallow clone works too.
2071 2086
    write(&origin_dir, "b.txt", b"b\n");
…
66 unchanged lines
2138 2153
2139 2154
    // Push carries the tag; a second attestation for the name is refused.
2140 2155
    let r = a.push("origin", false).unwrap();
2156 +
    origin.refresh().unwrap();
2141 2157
    assert!(r.iter().any(|x| x.name == "tags/v1" && x.ok), "{r:?}");
2142 2158
    assert_eq!(origin.tag_heads("v1").unwrap(), Some(vec![h1]));
2143 2159
    let mut forged = a
…
21 unchanged lines
2165 2181
    assert!(b.tags().unwrap()[0].signer.as_deref() == Some("Ada"));
2166 2182
    a.tag_create("v2", &[h2], "second").unwrap();
2167 2183
    a.push("origin", false).unwrap();
2184 +
    origin.refresh().unwrap();
2168 2185
    let rep = b.fetch("origin").unwrap();
2169 2186
    assert_eq!(rep.new_tags, 1);
2170 2187
    assert_eq!(b.tag_heads("v2").unwrap(), Some(vec![h2]));
…
1 unchanged lines
2172 2189
    // Under a policy, an unattested tag is refused and an attested one by the root passes.
2173 2190
    a.policy_init().unwrap();
2174 2191
    a.push("origin", false).unwrap();
2192 +
    origin.refresh().unwrap();
2175 2193
    let unattested = crate::protocol::TagRef {
2176 2194
        name: "v3".into(),
2177 2195
        state: origin
…
9 unchanged lines
2187 2205
    assert!(!r[0].ok && r[0].reason.contains("attestation"), "{r:?}");
2188 2206
    a.tag_create("v3", &[h2], "").unwrap();
2189 2207
    let r = a.push("origin", false).unwrap();
2208 +
    origin.refresh().unwrap();
2190 2209
    assert!(r.iter().any(|x| x.name == "tags/v3" && x.ok), "{r:?}");
2191 2210
    assert!(a.gc(true).is_ok());
2192 2211
    assert_eq!(a.tags().unwrap().len(), 3, "tags survive gc");
…
3 unchanged lines
2196 2215
    // nothing new.
2197 2216
    let rot = a.rotate_identity().unwrap();
2198 2217
    let r = a.push("origin", false).unwrap();
2218 +
    origin.refresh().unwrap();
2199 2219
    assert!(r.iter().all(|x| x.ok), "{r:?}");
2200 2220
    let c = Repo::clone(&url, &root.join("c"))
2201 2221
        .unwrap()
…
40 unchanged lines
2242 2262
        "the first workspace is untouched"
2243 2263
    );
2244 2264
    let h2 = ws.commit("b from ws2", &[]).unwrap();
2265 +
    repo.refresh().unwrap();
2245 2266
    let mut heads = repo.bookmark_heads().unwrap();
2246 2267
    heads.sort();
2247 2268
    let mut want = vec![h1, h2];
…
278 unchanged lines
2526 2547
    let h1 = a.commit("one", &[]).unwrap();
2527 2548
    assert!(a.policy_init().unwrap().applied);
2528 2549
    assert!(a.push("origin", false).unwrap().iter().all(|r| r.ok));
2550 +
    origin.refresh().unwrap();
2529 2551
2530 2552
    // Expiry: an identity that had expired when the change was made does not authenticate it.
2531 2553
    let mut e = Repo::init(&root.join("e"))
…
49 unchanged lines
2581 2603
    );
2582 2604
    // Origin accepts the rotated policy (signed by the old root) and the new key's change.
2583 2605
    let res = a.push("origin", false).unwrap();
2606 +
    origin.refresh().unwrap();
2584 2607
    assert!(res.iter().all(|x| x.ok), "{res:?}");
2585 2608
    assert_eq!(origin.current_policy().unwrap().unwrap().1.policy.serial, 2);
2586 2609
    let key_a = r.new;
…
22 unchanged lines
2609 2632
        "already a delegate"
2610 2633
    );
2611 2634
    assert!(a.push("origin", false).unwrap().iter().all(|x| x.ok));
2635 +
    origin.refresh().unwrap();
2612 2636
    write(&root.join("a"), "a.txt", b"a3\n");
2613 2637
    let h3 = a.commit("three", &[]).unwrap();
2614 2638
    assert!(
…
19 unchanged lines
2634 2658
    assert!(a.delegate_remove(&key_b).is_err());
2635 2659
    let key_a3 = r.new;
2636 2660
    let res = a.push("origin", false).unwrap();
2661 +
    origin.refresh().unwrap();
2637 2662
    assert!(res.iter().all(|x| x.ok), "{res:?}");
2638 2663
2639 2664
    // Multi-signer policy: Bob becomes a root and the threshold rises to 2,
…
3 unchanged lines
2643 2668
    assert!(a.policy_set_threshold(3).is_err());
2644 2669
    assert!(a.policy_set_threshold(2).unwrap().applied);
2645 2670
    assert!(a.push("origin", false).unwrap().iter().all(|x| x.ok));
2671 +
    origin.refresh().unwrap();
2646 2672
    b.pull("origin").unwrap();
2647 2673
    assert_eq!(
2648 2674
        b.current_policy().unwrap().unwrap().1.policy.root_threshold,
…
37 unchanged lines
2686 2712
        .unwrap()
2687 2713
        .iter()
2688 2714
        .any(|x| x.name == "policy" && x.ok));
2715 +
    origin.refresh().unwrap();
2689 2716
    a.pull("origin").unwrap();
2690 2717
    assert_eq!(
2691 2718
        a.current_policy().unwrap().unwrap().1.policy.serial,
…
586 unchanged lines
3278 3305
    write(&r, "a.txt", b"a\n");
3279 3306
    let h_a = repo.commit("a", &[]).unwrap();
3280 3307
    assert!(repo.push("origin", false).unwrap()[0].ok);
3308 +
    origin.refresh().unwrap();
3281 3309
3282 3310
    // A bookmark at the working copy's base, then work on it.
3283 3311
    assert!(repo.bookmark_create("bad name..", None).is_err());
…
52 unchanged lines
3336 3364
    assert!(repo.merge("main").is_err());
3337 3365
    // The other bookmark can be pushed by name and appears at origin.
3338 3366
    assert!(repo.push_bookmark("origin", "feature", false).unwrap()[0].ok);
3367 +
    origin.refresh().unwrap();
3339 3368
    assert_eq!(origin.resolve_heads("feature").unwrap(), vec![h_b]);
3340 3369
    assert!(repo.push("origin", false).unwrap()[0].ok);
3370 +
    origin.refresh().unwrap();
3341 3371
    assert_eq!(
3342 3372
        repo.bookmarks()
3343 3373
            .unwrap()
…
680 unchanged lines
4024 4054
    a.tag_create("v1", &[h2], "first release").unwrap();
4025 4055
    a.checkpoint().unwrap();
4026 4056
    assert!(a.push("origin", false).unwrap()[0].ok);
4057 +
    origin.refresh().unwrap();
4027 4058
4028 4059
    // Sound: signed changes, a seal, a tag, a checkpoint, a ref attestation.
4029 4060
    for repo in [&a, &origin] {
…
1062 unchanged lines
5092 5123
    write(&r, "a.txt", b"a\n");
5093 5124
    let h_a = repo.commit("a", &[]).unwrap();
5094 5125
    assert!(repo.push("origin", false).unwrap()[0].ok);
5126 +
    origin.refresh().unwrap();
5095 5127
    repo.bookmark_create("feature", None).unwrap();
5096 5128
    repo.bookmark_create("feature/x", None).unwrap();
5097 5129
    repo.bookmark_switch("feature/x", false).unwrap();
…
1 unchanged lines
5099 5131
    let h_b = repo.commit("b", &[]).unwrap();
5100 5132
    for name in ["feature", "feature/x"] {
5101 5133
        let res = repo.push_bookmark("origin", name, false).unwrap();
5134 +
        origin.refresh().unwrap();
5102 5135
        assert!(res[0].ok, "{name}: {res:?}");
5103 5136
    }
5104 5137
    assert_eq!(origin.resolve_heads("feature").unwrap(), vec![h_a]);
…
32 unchanged lines
5137 5170
    write(&r, "a.txt", b"a\n");
5138 5171
    repo.commit("a", &[]).unwrap();
5139 5172
    assert!(repo.push("origin", false).unwrap()[0].ok);
5173 +
    origin.refresh().unwrap();
5140 5174
    repo.bookmark_create("feature/x", None).unwrap();
5141 5175
    assert!(repo.push_bookmark("origin", "feature/x", false).unwrap()[0].ok);
5176 +
    origin.refresh().unwrap();
5142 5177
    assert!(origin.resolve_heads("feature/x").is_ok());
5143 5178
5144 5179
    let res = repo.push_delete("origin", "feature/x").unwrap();
5180 +
    origin.refresh().unwrap();
5145 5181
    assert!(res[0].ok, "{res:?}");
5146 5182
    assert!(!origin.view().unwrap().bookmarks.contains_key("feature/x"));
5147 5183
    let (_, att) = origin.latest_ref_attest("feature/x").unwrap().unwrap();
…
14 unchanged lines
5162 5198
5163 5199
    // Pushing it again creates it again, past the deletion's serial.
5164 5200
    let res = repo.push_bookmark("origin", "feature/x", false).unwrap();
5201 +
    origin.refresh().unwrap();
5165 5202
    assert!(res[0].ok, "{res:?}");
5166 5203
    assert!(origin.view().unwrap().bookmarks.contains_key("feature/x"));
5167 5204
5168 5205
    // Nothing to delete, and never the remote's own bookmark.
5169 5206
    assert!(repo.push_delete("origin", "nope").is_err());
5170 5207
    let res = repo.push_delete("origin", "main").unwrap();
5208 +
    origin.refresh().unwrap();
5171 5209
    assert!(
5172 5210
        !res[0].ok && res[0].reason.contains("default bookmark"),
5173 5211
        "{res:?}"
…
633 unchanged lines
5807 5845
    write(&root.join("b"), "base.txt", b"base\n");
5808 5846
    b.commit("base", &[]).unwrap();
5809 5847
    assert!(b.push("origin", false).unwrap()[0].ok);
5848 +
    origin.refresh().unwrap();
5810 5849
5811 5850
    // A feature bookmark with one change, pushed.
5812 5851
    b.bookmark_create("feat", None).unwrap();
…
1 unchanged lines
5814 5853
    write(&root.join("b"), "x.txt", b"x\n");
5815 5854
    let x = b.commit("x", &[]).unwrap();
5816 5855
    assert!(b.push("origin", false).unwrap()[0].ok);
5856 +
    origin.refresh().unwrap();
5817 5857
    b.bookmark_switch("main", false).unwrap();
5818 5858
5819 5859
    // The remote lands it: its main moves to the feature's state.
…
1776 unchanged lines
7596 7636
    let (h, v) = a.read_head().unwrap();
7597 7637
    assert!(b.swap_head(&rb, v.as_ref()).unwrap());
7598 7638
    assert!(!a.swap_head(&h, v.as_ref()).unwrap(), "stale");
7639 +
    a.refresh().unwrap();
7599 7640
    assert_eq!(a.head_ref().unwrap(), rb);
7600 7641
7601 7642
    let other = tempfile::tempdir().unwrap();
…
837 unchanged lines
8439 8480
    // Both on main: A commits, then B builds on it, editing A's file.
8440 8481
    write(&root, "a.txt", b"a\n");
8441 8482
    a.commit("from a", &[]).unwrap();
8483 +
    b.refresh().unwrap();
8442 8484
    let heads = b.resolve_heads("main").unwrap();
8443 8485
    b.checkout(&heads, false, "oxy checkout").unwrap();
8444 8486
    write(&ws_dir, "a.txt", b"a, then b\n");
…
166 unchanged lines
8611 8653
    let e = app.build_pin_change(&[a1], "libs/none", &pin2, "x", author);
8612 8654
    assert!(format!("{:?}", e.unwrap_err()).contains("does not exist"));
8613 8655
}
8656 +
8657 +
/// oxygenhub#249: a handle reads the operation head once and keeps it,
8658 +
/// so one command or request never mixes two heads; another handle's
8659 +
/// operation shows only after `refresh`. Its own operations move it, and
8660 +
/// one recorded from the head it kept merges with what landed since, as
8661 +
/// any concurrent operation does.
8662 +
#[test]
8663 +
fn one_handle_keeps_one_head_until_it_refreshes() {
8664 +
    hermetic();
8665 +
    let dir = tempfile::tempdir().unwrap();
8666 +
    let root = dir.path();
8667 +
    let writer = Repo::init(root).unwrap();
8668 +
    write(root, "a.txt", b"one\n");
8669 +
    let h1 = writer.commit("one", &[]).unwrap();
8670 +
    let reader = Repo::open(root).unwrap();
8671 +
    let before = reader.view().unwrap();
8672 +
    assert_eq!(reader.bookmark_heads().unwrap(), vec![h1]);
8673 +
    write(root, "a.txt", b"two\n");
8674 +
    let h2 = writer.commit("two", &[]).unwrap();
8675 +
    assert_eq!(
8676 +
        writer.bookmark_heads().unwrap(),
8677 +
        vec![h2],
8678 +
        "its own operation"
8679 +
    );
8680 +
    assert_eq!(reader.view().unwrap(), before, "the head it read first");
8681 +
    assert_eq!(reader.bookmark_heads().unwrap(), vec![h1]);
8682 +
    reader.refresh().unwrap();
8683 +
    assert_eq!(reader.bookmark_heads().unwrap(), vec![h2]);
8684 +
8685 +
    // A handle recording from an older head merges with the newer one.
8686 +
    let stale = Repo::open(root).unwrap();
8687 +
    assert_eq!(stale.bookmark_heads().unwrap(), vec![h2]);
8688 +
    write(root, "a.txt", b"three\n");
8689 +
    let h3 = writer.commit("three", &[]).unwrap();
8690 +
    stale.bookmark_create("feature", Some(&[h1])).unwrap();
8691 +
    let view = stale.view().unwrap();
8692 +
    assert!(view.bookmarks.contains_key("feature"));
8693 +
    assert_eq!(
8694 +
        stale.bookmark_heads().unwrap(),
8695 +
        vec![h3],
8696 +
        "merged, not lost"
8697 +
    );
8698 +
    let fresh = Repo::open(root).unwrap();
8699 +
    assert_eq!(fresh.view().unwrap(), view);
8700 +
}
8701 +
8702 +
/// oxygenhub#249 over a store (a bucket's compare-and-swap): the same.
8703 +
#[test]
8704 +
fn over_a_store_one_handle_keeps_one_head() {
8705 +
    hermetic();
8706 +
    let backend = std::sync::Arc::new(crate::strata::MemoryStore::new());
8707 +
    crate::store::TEST_BACKEND.with(|t| *t.borrow_mut() = Some(backend.clone()));
8708 +
    let dir = tempfile::tempdir().unwrap();
8709 +
    let a = Repo::init(dir.path()).unwrap();
8710 +
    let (h0, _) = a.head_op().unwrap();
8711 +
    let b = Repo::open(dir.path()).unwrap();
8712 +
    assert_eq!(b.head_op().unwrap().0, h0);
8713 +
    let state = a.store.put_state(&StateObject::new(vec![])).unwrap();
8714 +
    let mut view = a.view().unwrap();
8715 +
    view.bookmarks.insert("x".into(), state);
8716 +
    let r = a.record_op(&[h0], "bookmark x", view).unwrap();
8717 +
    assert_eq!(a.head_op().unwrap().0, r);
8718 +
    assert_eq!(b.head_op().unwrap().0, h0, "b keeps its head");
8719 +
    b.refresh().unwrap();
8720 +
    assert_eq!(b.head_op().unwrap().0, r);
8721 +
    crate::store::TEST_BACKEND.with(|t| *t.borrow_mut() = None);
8722 +
}
8723 +
8724 +
/// oxygenhub#249: gc retires what it repacks rather than deleting it; a
8725 +
/// reader opened before reads on, then and after the next gc deletes the
8726 +
/// retired files once their hour is up. Younger ones are kept.
8727 +
#[test]
8728 +
fn gc_retires_and_a_later_gc_deletes_after_the_hour() {
8729 +
    hermetic();
8730 +
    let dir = tempfile::tempdir().unwrap();
8731 +
    let root = dir.path();
8732 +
    let repo = Repo::init(root).unwrap();
8733 +
    let mut heads = Vec::new();
8734 +
    for i in 0..4 {
8735 +
        write(root, &format!("f{i}.txt"), format!("{i}\n").as_bytes());
8736 +
        heads.push(repo.commit(&format!("c{i}"), &[]).unwrap());
8737 +
    }
8738 +
    let reader = Repo::open(root).unwrap();
8739 +
    assert_eq!(reader.log().unwrap().len(), 5);
8740 +
    let r = repo.gc(true).unwrap();
8741 +
    assert_eq!(r.retired, crate::store::RetiredReport::default());
8742 +
    assert_eq!(repo.store().loose_count().unwrap(), 0);
8743 +
    assert_eq!(reader.log().unwrap().len(), 5, "read on from what it had");
8744 +
    // Nothing is an hour old yet: a gc keeps it all.
8745 +
    let r = repo.gc(true).unwrap();
8746 +
    assert_eq!((r.retired.packs, r.retired.loose), (0, 0));
8747 +
    assert!(r.retired.kept > 0, "{r:?}");
8748 +
    write(root, "f9.txt", b"9\n");
8749 +
    heads.push(repo.commit("c9", &[]).unwrap());
8750 +
    crate::store::reader_tests::age_retired(
8751 +
        repo.store(),
8752 +
        crate::store::RETIRED_FOR + std::time::Duration::from_secs(60),
8753 +
    );
8754 +
    let r = repo.gc(true).unwrap();
8755 +
    assert!(r.retired.packs >= 1 && r.retired.loose > 0, "{r:?}");
8756 +
    assert_eq!(r.retired.kept, 0, "{r:?}");
8757 +
    let r = repo.gc(true).unwrap();
8758 +
    assert_eq!((r.retired.packs, r.retired.loose), (0, 0));
8759 +
    assert!(r.retired.kept > 0, "what the last gc retired is young");
8760 +
    // The reader opened before all of it still reads everything, from
8761 +
    // the packs it finds when what it loaded is gone.
8762 +
    assert_eq!(reader.log().unwrap().len(), 5);
8763 +
    for h in &heads[..4] {
8764 +
        reader.store().get_change(h).unwrap();
8765 +
    }
8766 +
    reader.refresh().unwrap();
8767 +
    assert_eq!(reader.log().unwrap().len(), 6);
8768 +
    let fresh = Repo::open(root).unwrap();
8769 +
    assert_eq!(
8770 +
        fresh.store().pack_count(),
8771 +
        1,
8772 +
        "retired packs are not loaded"
8773 +
    );
8774 +
}
8775 +
8776 +
/// oxygenhub#249: a repository opens when its store's pack listing names
8777 +
/// an index that is not there (deleted between the listing and the read).
8778 +
#[test]
8779 +
fn a_repository_opens_when_a_listed_index_is_gone() {
8780 +
    hermetic();
8781 +
    let backend = crate::store::reader_tests::racing(true);
8782 +
    let dyn_backend: std::sync::Arc<crate::store::Backend> = backend.clone();
8783 +
    crate::store::TEST_BACKEND.with(|t| *t.borrow_mut() = Some(dyn_backend));
8784 +
    let dir = tempfile::tempdir().unwrap();
8785 +
    let repo = Repo::init(dir.path()).unwrap();
8786 +
    write(dir.path(), "a.txt", b"a\n");
8787 +
    let h = repo.commit("a", &[]).unwrap();
8788 +
    repo.gc(true).unwrap();
8789 +
    let again = Repo::open(dir.path()).unwrap();
8790 +
    assert_eq!(again.store().pack_count(), 1);
8791 +
    assert_eq!(again.bookmark_heads().unwrap(), vec![h]);
8792 +
    // And when one is deleted between the listing and the read: a copy of
8793 +
    // the live pack, whose index goes as the store opens.
8794 +
    use crate::strata::Store as _;
8795 +
    let idx = backend
8796 +
        .list("pack-")
8797 +
        .unwrap()
8798 +
        .into_iter()
8799 +
        .map(|e| e.key)
8800 +
        .find(|k| k.ends_with(".idx") && k != "pack-0000.idx")
8801 +
        .unwrap();
8802 +
    let stem = idx.trim_end_matches(".idx");
8803 +
    for ext in ["pack", "idx"] {
8804 +
        let bytes = backend.get(&format!("{stem}.{ext}")).unwrap();
8805 +
        backend
8806 +
            .put_new(&format!("pack-copy.{ext}"), &bytes)
8807 +
            .unwrap();
8808 +
    }
8809 +
    *backend.doomed.lock().unwrap() = Some("pack-copy.idx".into());
8810 +
    let again = Repo::open(dir.path()).unwrap();
8811 +
    assert_eq!(again.store().pack_count(), 1);
8812 +
    assert_eq!(again.bookmark_heads().unwrap(), vec![h]);
8813 +
    assert!(!backend.has("pack-copy.idx").unwrap());
8814 +
    crate::store::TEST_BACKEND.with(|t| *t.borrow_mut() = None);
8815 +
}
modified crates/oxy-core/src/repo/workspace.rs +3 −0
…
345 unchanged lines
346 346
            std::time::Duration::from_secs(60),
347 347
        )?;
348 348
        HELD.with(|h| h.borrow_mut().insert(path.clone(), 1));
349 +
        // Whoever held the lock may have moved the operation head: what
350 +
        // is done under it starts from the head as it is now (oxygenhub#249).
351 +
        self.forget_head();
349 352
        Ok(WorkingCopyLock {
350 353
            path,
351 354
            _file: Some(file),
…
3 unchanged lines
modified crates/oxy-core/src/store.rs +792 −43
…
23 unchanged lines
24 24
//! `receipt/<hh>/<rest>/<attest>` index the seals covering a change and
25 25
//! the receipts on a state. Older stores hold these as one list per
26 26
//! object (`sigs/`, `seals/`, `receipts/`), which are still read.
27 +
//! `pack-retired` records the packs and loose objects a repack replaced,
28 +
//! which stay for readers until a collection an hour later (oxygenhub#249).
27 29
28 30
use crate::attest::Attest;
29 31
use crate::hash::{hash_object, header, ObjectRef, ObjectType};
…
28 unchanged lines
58 60
/// Largest object a pack entry may claim; anything bigger is corruption,
59 61
/// and refusing it keeps a bad index from driving huge allocations.
60 62
const MAX_OBJECT_LEN: u32 = 1 << 30;
63 +
/// What reading an entry of a pack whose dictionary is gone says. A pack
64 +
/// being deleted can lose its dictionary first, so a read that meets this
65 +
/// looks at the packs again before it is believed.
66 +
const NO_DICT: &str = "packed with a dictionary the pack lacks";
61 67
68 +
/// The record of what repacks replaced (oxygenhub#249): packs and loose
69 +
/// objects whose contents a newer pack holds, kept for readers that
70 +
/// still use them until garbage collection deletes them, at least
71 +
/// [`RETIRED_FOR`] later. One line per key, `<retired at, wall ms> <key>`.
72 +
/// It is a reference record (updated by compare-and-swap, so concurrent
73 +
/// repacks and collections never lose each other's lines), named under
74 +
/// `pack-` so the pack listing every open makes shows whether it holds
75 +
/// anything.
76 +
const RETIRED_KEY: &str = "pack-retired";
77 +
/// How long a retired pack or loose object stays in the store: a reader
78 +
/// that loaded its list of packs before a repack reads on from the old
79 +
/// packs this long, and after that finds the objects in the new pack by
80 +
/// looking again (see [`Store::get`]).
81 +
pub const RETIRED_FOR: std::time::Duration = std::time::Duration::from_secs(60 * 60);
82 +
/// Compare-and-swap rounds for the retired record before giving up.
83 +
const RETIRED_TRIES: usize = 32;
84 +
/// After a look at the pack listing found nothing new, how long a missed
85 +
/// [`Store::has`] answers without looking again: a writer probing for
86 +
/// thousands of objects it is about to store would otherwise list the
87 +
/// packs for each.
88 +
const HAS_RELIST_AFTER: std::time::Duration = std::time::Duration::from_secs(1);
89 +
90 +
/// A pack listing as compared: each key with its size and time.
91 +
type Listed = Vec<(String, u64, Option<u64>)>;
92 +
93 +
/// The retired record's lines: each key with the time it was retired.
94 +
fn parse_retired(bytes: &[u8]) -> std::collections::BTreeMap<String, u64> {
95 +
    String::from_utf8_lossy(bytes)
96 +
        .lines()
97 +
        .filter_map(|l| {
98 +
            let (t, k) = l.split_once(' ')?;
99 +
            Some((k.to_string(), t.parse().ok()?))
100 +
        })
101 +
        .collect()
102 +
}
103 +
104 +
fn encode_retired(m: &std::collections::BTreeMap<String, u64>) -> Vec<u8> {
105 +
    m.iter()
106 +
        .map(|(k, t)| format!("{t} {k}\n"))
107 +
        .collect::<String>()
108 +
        .into_bytes()
109 +
}
110 +
111 +
/// What deleting retired packs and loose objects did.
112 +
#[derive(Debug, Clone, Default, PartialEq, Eq)]
113 +
pub struct RetiredReport {
114 +
    /// Retired packs deleted, each with its index and dictionary.
115 +
    pub packs: usize,
116 +
    /// Retired loose objects deleted.
117 +
    pub loose: usize,
118 +
    /// Retired entries younger than [`RETIRED_FOR`], left for the next
119 +
    /// collection.
120 +
    pub kept: usize,
121 +
}
122 +
62 123
/// An object read from the store: its type and body.
63 124
pub type Object = (ObjectType, Vec<u8>);
64 125
…
110 unchanged lines
175 236
    dict: Option<&Arc<zstd::dict::DecoderDictionary<'static>>>,
176 237
) -> Result<Object> {
177 238
    let body = if e.flags & FLAG_DICT != 0 {
178 -
        let dict = dict.ok_or_else(|| {
179 -
            StoreError::Corrupt(*r, "packed with a dictionary the pack lacks".into())
180 -
        })?;
239 +
        let dict = dict.ok_or_else(|| StoreError::Corrupt(*r, NO_DICT.into()))?;
181 240
        zstd::bulk::Decompressor::with_prepared_dictionary(dict)
182 241
            .and_then(|mut d| d.decompress(&stored, e.raw_len as usize))
183 242
            .map_err(|err| StoreError::Corrupt(*r, format!("zstd: {err}")))?
…
14 unchanged lines
198 257
199 258
impl PackIndex {
200 259
    /// Load a pack's index; `pack_len` is the pack's size, from the listing
201 -
    /// that found it (a stat of it would be a request of its own).
202 -
    fn load(backend: &Backend, idx_key: &str, pack_key: &str, pack_len: u64) -> Result<PackIndex> {
203 -
        let data = backend.get(idx_key).map_err(backend_err)?;
260 +
    /// that found it (a stat of it would be a request of its own). `None`
261 +
    /// when the index is gone since the listing: the pack was deleted.
262 +
    fn load(
263 +
        backend: &Backend,
264 +
        idx_key: &str,
265 +
        pack_key: &str,
266 +
        pack_len: u64,
267 +
    ) -> Result<Option<PackIndex>> {
268 +
        let data = match backend.get(idx_key) {
269 +
            Ok(d) => d,
270 +
            Err(BackendError::NotFound(_)) => return Ok(None),
271 +
            Err(e) => return Err(backend_err(e)),
272 +
        };
204 273
        let bad = |m: &str| StoreError::Corrupt(ObjectRef::ZERO, format!("{idx_key}: {m}"));
205 274
        if data.len() < 12 || &data[..8] != IDX_MAGIC {
206 275
            return Err(bad("bad index header"));
…
40 unchanged lines
247 316
        let dict_key = format!("{}.dict", pack_key.trim_end_matches(".pack"));
248 317
        let dict = get_optional(backend, &dict_key)?
249 318
            .map(|d| Arc::new(zstd::dict::DecoderDictionary::copy(&d)));
250 -
        Ok(PackIndex {
319 +
        Ok(Some(PackIndex {
251 320
            key: pack_key.to_string(),
252 321
            refs,
253 322
            entries,
254 323
            dict,
255 -
        })
324 +
        }))
256 325
    }
257 326
258 327
    fn find(&self, r: &ObjectRef) -> Option<PackEntry> {
…
65 unchanged lines
324 393
    scratch: PathBuf,
325 394
    /// The backend is that directory, so the old file paths still hold.
326 395
    local: bool,
327 -
    packs: RefCell<Vec<PackIndex>>,
396 +
    packs: RefCell<Vec<Arc<PackIndex>>>,
397 +
    /// The pack listing `packs` was loaded from (key, size, time), to tell
398 +
    /// whether a look at the store would find anything new.
399 +
    packs_listed: RefCell<Listed>,
400 +
    /// When a look at the pack listing for a missed [`Store::has`] last
401 +
    /// found nothing new.
402 +
    has_relisted: std::cell::Cell<Option<std::time::Instant>>,
328 403
    /// An open write batch (spec §13.4): objects go here instead of to loose
329 404
    /// files, and are written out with one `fsync` when the batch finishes.
330 405
    pending: RefCell<Option<Pending>>,
…
162 unchanged lines
493 568
            scratch,
494 569
            local: false,
495 570
            packs: RefCell::new(Vec::new()),
571 +
            packs_listed: RefCell::new(Vec::new()),
572 +
            has_relisted: std::cell::Cell::new(None),
496 573
            pending: RefCell::new(None),
497 574
            batch_depth: std::cell::Cell::new(0),
498 575
            metas: RefCell::new(None),
…
13 unchanged lines
512 589
    /// is a file.
513 590
    pub fn is_local(&self) -> bool {
514 591
        self.local
592 +
    }
593 +
594 +
    /// Load the list of packs again, from a new listing: what other
595 +
    /// writers packed since becomes readable, and packs since retired or
596 +
    /// deleted are let go. Packs already loaded are kept, not read again.
597 +
    pub fn refresh(&self) -> Result<()> {
598 +
        self.reload_packs()
599 +
    }
600 +
601 +
    /// The pack listing, with how `packs_listed` records it.
602 +
    fn list_packs(&self) -> Result<(Vec<strata::Entry>, Listed)> {
603 +
        let listed = self.backend.list("pack-").map_err(backend_err)?;
604 +
        let sig = listed
605 +
            .iter()
606 +
            .map(|e| (e.key.clone(), e.size, e.modified_ms))
607 +
            .collect();
608 +
        Ok((listed, sig))
515 609
    }
516 610
611 +
    /// Load every pack the listing names that is not retired. An index
612 +
    /// that is gone by the time it is read (its pack deleted after the
613 +
    /// listing) is passed over, and the store listed once more, since
614 +
    /// what replaced it may be new since the first listing.
517 615
    fn reload_packs(&self) -> Result<()> {
518 -
        let listed = self.backend.list("pack-").map_err(backend_err)?;
519 -
        let mut idx_keys: Vec<&str> = listed
616 +
        let (listed, sig) = self.list_packs()?;
617 +
        self.load_listed(listed, sig)
618 +
    }
619 +
620 +
    fn load_listed(&self, mut listed: Vec<strata::Entry>, mut sig: Listed) -> Result<()> {
621 +
        let had: std::collections::HashMap<String, Arc<PackIndex>> = self
622 +
            .packs
623 +
            .borrow()
624 +
            .iter()
625 +
            .map(|p| (p.key.clone(), p.clone()))
626 +
            .collect();
627 +
        let mut loaded: std::collections::HashMap<String, Arc<PackIndex>> = Default::default();
628 +
        for attempt in 0..2 {
629 +
            let retired = self.retired_in(&listed)?;
630 +
            let mut idx_keys: Vec<&str> = listed
631 +
                .iter()
632 +
                .map(|e| e.key.as_str())
633 +
                .filter(|k| k.ends_with(".idx"))
634 +
                .collect();
635 +
            idx_keys.sort();
636 +
            let mut packs = Vec::new();
637 +
            let mut vanished = false;
638 +
            for idx in idx_keys {
639 +
                let pack = format!("{}.pack", idx.trim_end_matches(".idx"));
640 +
                if retired.contains_key(&pack) {
641 +
                    continue;
642 +
                }
643 +
                let Some(p) = listed.iter().find(|e| e.key == pack) else {
644 +
                    continue;
645 +
                };
646 +
                if let Some(kept) = had.get(&pack).or_else(|| loaded.get(&pack)) {
647 +
                    packs.push(kept.clone());
648 +
                    continue;
649 +
                }
650 +
                match PackIndex::load(self.backend.as_ref(), idx, &pack, p.size)? {
651 +
                    Some(index) => {
652 +
                        let index = Arc::new(index);
653 +
                        loaded.insert(pack, index.clone());
654 +
                        packs.push(index);
655 +
                    }
656 +
                    None => vanished = true,
657 +
                }
658 +
            }
659 +
            if vanished && attempt == 0 {
660 +
                (listed, sig) = self.list_packs()?;
661 +
                continue;
662 +
            }
663 +
            *self.packs.borrow_mut() = packs;
664 +
            *self.packs_listed.borrow_mut() = sig;
665 +
            return Ok(());
666 +
        }
667 +
        unreachable!("the second round always returns")
668 +
    }
669 +
670 +
    /// Load the packs again when the pack listing differs from the one
671 +
    /// they were loaded from; whether it did. What a miss calls before it
672 +
    /// is believed (oxygenhub#249): another writer may have packed the
673 +
    /// object, or a repack moved it, since this store loaded its packs.
674 +
    fn reload_if_changed(&self) -> Result<bool> {
675 +
        let (listed, sig) = self.list_packs()?;
676 +
        if *self.packs_listed.borrow() == sig {
677 +
            return Ok(false);
678 +
        }
679 +
        self.load_listed(listed, sig)?;
680 +
        Ok(true)
681 +
    }
682 +
683 +
    /// The retired record, read when `listed` (a pack listing) shows it
684 +
    /// holds anything.
685 +
    fn retired_in(
686 +
        &self,
687 +
        listed: &[strata::Entry],
688 +
    ) -> Result<std::collections::BTreeMap<String, u64>> {
689 +
        match listed.iter().find(|e| e.key == RETIRED_KEY) {
690 +
            Some(e) if e.size > 0 => Ok(self.read_retired()?.0),
691 +
            _ => Ok(Default::default()),
692 +
        }
693 +
    }
694 +
695 +
    /// The retired record and its version (`None`: never written).
696 +
    fn read_retired(
697 +
        &self,
698 +
    ) -> Result<(
699 +
        std::collections::BTreeMap<String, u64>,
700 +
        Option<strata::Version>,
701 +
    )> {
702 +
        match self.backend.read_ref(RETIRED_KEY).map_err(backend_err)? {
703 +
            Some((bytes, v)) => Ok((parse_retired(&bytes), Some(v))),
704 +
            None => Ok((Default::default(), None)),
705 +
        }
706 +
    }
707 +
708 +
    /// Change the retired record by compare-and-swap, reading it again
709 +
    /// whenever another writer changed it first. `edit` must give the
710 +
    /// same result however often it runs.
711 +
    fn update_retired(
712 +
        &self,
713 +
        mut edit: impl FnMut(&mut std::collections::BTreeMap<String, u64>),
714 +
    ) -> Result<()> {
715 +
        for _ in 0..RETIRED_TRIES {
716 +
            let (mut m, version) = self.read_retired()?;
717 +
            let before = m.clone();
718 +
            edit(&mut m);
719 +
            if m == before && version.is_some() {
720 +
                return Ok(());
721 +
            }
722 +
            match self
723 +
                .backend
724 +
                .write_ref(RETIRED_KEY, &encode_retired(&m), version.as_ref())
725 +
            {
726 +
                Ok(_) => return Ok(()),
727 +
                Err(BackendError::Conflict(_)) => continue,
728 +
                Err(e) => return Err(backend_err(e)),
729 +
            }
730 +
        }
731 +
        Err(StoreError::Io(io::Error::other(
732 +
            "other writers kept changing the record of retired packs; run the command again",
733 +
        )))
734 +
    }
735 +
736 +
    /// Record `keys` (packs, by their `.pack` key, and loose objects) as
737 +
    /// retired now: kept for the readers still using them, and deleted by
738 +
    /// a collection at least [`RETIRED_FOR`] later. A key already retired
739 +
    /// keeps its first time.
740 +
    fn retire(&self, keys: &[String]) -> Result<()> {
741 +
        if keys.is_empty() {
742 +
            return Ok(());
743 +
        }
744 +
        let now = now_ms();
745 +
        self.update_retired(|m| {
746 +
            for k in keys {
747 +
                m.entry(k.clone()).or_insert(now);
748 +
            }
749 +
        })
750 +
    }
751 +
752 +
    /// Delete what was retired at least `older_than` ago (a collection
753 +
    /// does, with [`RETIRED_FOR`]): each pack with its index and
754 +
    /// dictionary, and each loose object a pack now holds. A retired loose
755 +
    /// object no pack holds any more (the pack that had it was pruned, and
756 +
    /// the object stored again since) is kept, as an ordinary loose object.
757 +
    pub fn expire_retired(
758 +
        &self,
759 +
        older_than: std::time::Duration,
760 +
        progress: Progress,
761 +
    ) -> Result<RetiredReport> {
762 +
        let (listed, _) = self.list_packs()?;
763 +
        let retired = self.retired_in(&listed)?;
764 +
        let mut report = RetiredReport::default();
765 +
        if retired.is_empty() {
766 +
            return Ok(report);
767 +
        }
768 +
        let now = now_ms();
769 +
        let older_than = older_than.as_millis() as u64;
770 +
        let due: Vec<(String, u64)> = retired
520 771
            .iter()
521 -
            .map(|e| e.key.as_str())
522 -
            .filter(|k| k.ends_with(".idx"))
772 +
            .filter(|(_, t)| now.saturating_sub(**t) >= older_than)
773 +
            .map(|(k, t)| (k.clone(), *t))
523 774
            .collect();
524 -
        idx_keys.sort();
525 -
        let mut packs = Vec::new();
526 -
        for idx in idx_keys {
527 -
            let pack = format!("{}.pack", idx.trim_end_matches(".idx"));
528 -
            if let Some(p) = listed.iter().find(|e| e.key == pack) {
529 -
                packs.push(PackIndex::load(self.backend.as_ref(), idx, &pack, p.size)?);
775 +
        report.kept = retired.len() - due.len();
776 +
        if due.is_empty() {
777 +
            return Ok(report);
778 +
        }
779 +
        progress(&format!("deleting {} retired files", due.len()));
780 +
        // Which objects the live packs hold.
781 +
        self.reload_packs()?;
782 +
        for (key, _) in &due {
783 +
            if key.ends_with(".pack") {
784 +
                self.remove_pack(key)?;
785 +
                report.packs += 1;
786 +
            } else if let Some(r) = loose_ref(key) {
787 +
                if self.find_packed(&r).is_some() {
788 +
                    delete(self.backend.as_ref(), key)?;
789 +
                    report.loose += 1;
790 +
                }
530 791
            }
531 792
        }
532 -
        *self.packs.borrow_mut() = packs;
533 -
        Ok(())
793 +
        self.update_retired(|m| {
794 +
            for (k, t) in &due {
795 +
                if m.get(k) == Some(t) {
796 +
                    m.remove(k);
797 +
                }
798 +
            }
799 +
        })?;
800 +
        self.reload_packs()?;
801 +
        Ok(report)
534 802
    }
535 803
804 +
    /// The loose objects retired and not yet deleted, by key.
805 +
    fn retired_loose(&self) -> Result<std::collections::HashSet<String>> {
806 +
        let (listed, _) = self.list_packs()?;
807 +
        Ok(self
808 +
            .retired_in(&listed)?
809 +
            .into_keys()
810 +
            .filter(|k| k.starts_with("loose/"))
811 +
            .collect())
812 +
    }
813 +
536 814
    pub fn pack_count(&self) -> usize {
537 815
        self.packs.borrow().len()
538 816
    }
…
6 unchanged lines
545 823
            .find_map(|(i, p)| p.find(r).map(|e| (i, e)))
546 824
    }
547 825
548 -
    /// Loose objects, with their keys and ages.
826 +
    /// Loose objects, with their keys and ages; not those retired, whose
827 +
    /// objects a pack holds.
549 828
    fn loose_objects(&self) -> Result<Vec<(ObjectRef, strata::Entry)>> {
829 +
        let retired = self.retired_loose()?;
830 +
        let mut out = self.loose_objects_all()?;
831 +
        out.retain(|(_, e)| !retired.contains(&e.key));
832 +
        Ok(out)
833 +
    }
834 +
835 +
    /// Every loose object, retired or not.
836 +
    fn loose_objects_all(&self) -> Result<Vec<(ObjectRef, strata::Entry)>> {
550 837
        let mut out: Vec<(ObjectRef, strata::Entry)> = self
551 838
            .backend
552 839
            .list("loose/")
…
101 unchanged lines
654 941
        let old_enough =
655 942
            |modified: Option<u64>| modified.is_none_or(|t| now.saturating_sub(t) >= grace_ms);
656 943
        let mut report = PruneReport::default();
657 -
        // Loose objects.
658 -
        for (r, entry) in self.loose_objects()? {
944 +
        // Loose objects. A retired one (its object packed) whose object is
945 +
        // unreachable goes with it, uncounted: the pack's copy is counted.
946 +
        let retired = self.retired_loose()?;
947 +
        let mut unretired: Vec<String> = Vec::new();
948 +
        for (r, entry) in self.loose_objects_all()? {
659 949
            if keep.contains(&r) {
660 950
                continue;
661 951
            }
952 +
            if retired.contains(&entry.key) {
953 +
                if old_enough(entry.modified_ms) {
954 +
                    delete(self.backend.as_ref(), &entry.key)?;
955 +
                    unretired.push(entry.key);
956 +
                }
957 +
                continue;
958 +
            }
662 959
            if !old_enough(entry.modified_ms) {
663 960
                report.deferred += 1;
664 961
                continue;
…
2 unchanged lines
667 964
            report.objects += 1;
668 965
            delete(self.backend.as_ref(), &entry.key)?;
669 966
            self.remove_sidecars(&r)?;
967 +
        }
968 +
        if !unretired.is_empty() {
969 +
            self.update_retired(|m| {
970 +
                for k in &unretired {
971 +
                    m.remove(k);
972 +
                }
973 +
            })?;
670 974
        }
671 975
        // Packs: rewrite those that hold unreachable objects.
672 976
        let mut keep_refs: Vec<ObjectRef> = Vec::new();
…
94 unchanged lines
767 1071
        self.reload_packs()
768 1072
    }
769 1073
770 -
    /// Remove a pack with its index and dictionary.
1074 +
    /// Remove a pack with its index and dictionary: the index first, so
1075 +
    /// a store opened meanwhile does not load a pack that is going.
771 1076
    fn remove_pack(&self, pack_key: &str) -> Result<()> {
772 1077
        let stem = pack_key.trim_end_matches(".pack");
1078 +
        delete(self.backend.as_ref(), &format!("{stem}.idx"))?;
773 1079
        delete(self.backend.as_ref(), pack_key)?;
774 -
        delete(self.backend.as_ref(), &format!("{stem}.idx"))?;
775 1080
        delete(self.backend.as_ref(), &format!("{stem}.dict"))
776 1081
    }
777 1082
778 -
    /// Write `refs` into a new pack, then delete `loose_keys` and `old_packs`.
1083 +
    /// Write `refs` into a new pack, then retire `loose_keys` and
1084 +
    /// `old_packs`: they stay in the store, unused by stores opened from
1085 +
    /// now on, until a collection [`RETIRED_FOR`] later deletes them, so a
1086 +
    /// reader that loaded its packs before goes on reading (oxygenhub#249).
779 1087
    fn write_pack(
780 1088
        &self,
781 1089
        refs: &[ObjectRef],
…
3 unchanged lines
785 1093
        progress: Progress,
786 1094
    ) -> Result<PackReport> {
787 1095
        if refs.is_empty() {
788 -
            for p in old_packs {
789 -
                self.remove_pack(p)?;
790 -
            }
1096 +
            self.retire(old_packs)?;
791 1097
            self.reload_packs()?;
792 1098
            return Ok(PackReport::default());
793 1099
        }
…
115 unchanged lines
909 1215
        }
910 1216
        progress(&format!("writing the index of {} objects", index.len()));
911 1217
        self.finish_pack_files(&id, &tmp_pack, &index)?;
912 -
        for p in old_packs {
913 -
            self.remove_pack(p)?;
914 -
        }
1218 +
        let mut retired = old_packs.to_vec();
1219 +
        retired.extend_from_slice(loose_keys);
1220 +
        self.retire(&retired)?;
915 1221
        self.reload_packs()?;
916 -
        for key in loose_keys {
917 -
            delete(self.backend.as_ref(), key)?;
918 -
        }
919 1222
        Ok(report)
920 1223
    }
921 1224
…
42 unchanged lines
964 1267
        delete(self.backend.as_ref(), &loose_key(r))
965 1268
    }
966 1269
1270 +
    /// Whether the store holds `r`. A miss looks at the pack listing
1271 +
    /// again before it is believed, as [`Store::get`]'s does; a writer
1272 +
    /// asking about many objects it is about to store looks at most once
1273 +
    /// in [`HAS_RELIST_AFTER`] while nothing new turns up.
967 1274
    pub fn has(&self, r: &ObjectRef) -> bool {
968 -
        self.backend.has(&loose_key(r)).unwrap_or(false)
969 -
            || self.find_packed(r).is_some()
1275 +
        if self.has_now(r) {
1276 +
            return true;
1277 +
        }
1278 +
        let recent = self
1279 +
            .has_relisted
1280 +
            .get()
1281 +
            .is_some_and(|t| t.elapsed() < HAS_RELIST_AFTER);
1282 +
        if recent {
1283 +
            return false;
1284 +
        }
1285 +
        match self.reload_if_changed() {
1286 +
            Ok(true) => {
1287 +
                self.has_relisted.set(None);
1288 +
                self.has_now(r)
1289 +
            }
1290 +
            _ => {
1291 +
                self.has_relisted.set(Some(std::time::Instant::now()));
1292 +
                false
1293 +
            }
1294 +
        }
1295 +
    }
1296 +
1297 +
    /// [`Store::has`] against the packs as loaded.
1298 +
    fn has_now(&self, r: &ObjectRef) -> bool {
1299 +
        self.find_packed(r).is_some()
970 1300
            || self
971 1301
                .pending
972 1302
                .borrow()
973 1303
                .as_ref()
974 1304
                .is_some_and(|p| p.lookup.contains_key(r))
1305 +
            || self.backend.has(&loose_key(r)).unwrap_or(false)
975 1306
    }
976 1307
977 1308
    /// Start a write batch: until the matching `finish`, `put` collects
…
310 unchanged lines
1288 1619
    /// The packs' indexes are in memory, so they are asked first: asking
1289 1620
    /// for a loose copy of a packed object costs a request that finds
1290 1621
    /// nothing on a store in a bucket, every time (oxygenhub#228).
1622 +
    ///
1623 +
    /// A miss is not believed at once (oxygenhub#249): when the pack it
1624 +
    /// was in is gone (deleted after a repack retired it) the packs are
1625 +
    /// loaded again, and when the object is nowhere the pack listing is
1626 +
    /// looked at, and the packs loaded again if it changed (another
1627 +
    /// writer packed it, or a repack moved it); then it is read once more.
1291 1628
    pub fn get(&self, r: &ObjectRef) -> Result<Object> {
1629 +
        match self.get_now(r) {
1630 +
            Err(e) if is_miss(&e) => {
1631 +
                // Read from a pack this store loaded, which is gone.
1632 +
                let packed_miss =
1633 +
                    !matches!(e, StoreError::NotFound(_)) || self.find_packed(r).is_some();
1634 +
                let changed = if packed_miss {
1635 +
                    self.reload_packs()?;
1636 +
                    true
1637 +
                } else {
1638 +
                    self.reload_if_changed()?
1639 +
                };
1640 +
                if changed {
1641 +
                    self.get_now(r)
1642 +
                } else {
1643 +
                    Err(e)
1644 +
                }
1645 +
            }
1646 +
            other => other,
1647 +
        }
1648 +
    }
1649 +
1650 +
    /// [`Store::get`] against the packs as loaded.
1651 +
    fn get_now(&self, r: &ObjectRef) -> Result<Object> {
1292 1652
        if let Some((i, e)) = self.find_packed(r) {
1293 -
            return self.packs.borrow()[i].fetch(self.backend.as_ref(), r, e);
1653 +
            let pack = self.packs.borrow()[i].clone();
1654 +
            return pack.fetch(self.backend.as_ref(), r, e);
1294 1655
        }
1295 1656
        match get_optional(self.backend.as_ref(), &loose_key(r))? {
1296 1657
            Some(bytes) => parse_loose(&bytes, r),
…
4 unchanged lines
1301 1662
    /// Many objects at once, read and verified in parallel: packed ones
1302 1663
    /// from their packs, the others loose (as [`Store::get`], packs first),
1303 1664
    /// then what is pending in a batch. Results are in the order of `refs`.
1665 +
    ///
1666 +
    /// Misses are looked for again as [`Store::get`]'s are, together.
1304 1667
    pub fn get_many(&self, refs: &[ObjectRef]) -> Vec<Result<Object>> {
1668 +
        let mut out = self.get_many_now(refs);
1669 +
        let missed: Vec<usize> = out
1670 +
            .iter()
1671 +
            .enumerate()
1672 +
            .filter(|(_, g)| g.as_ref().is_err_and(is_miss))
1673 +
            .map(|(i, _)| i)
1674 +
            .collect();
1675 +
        if missed.is_empty() {
1676 +
            return out;
1677 +
        }
1678 +
        let packed_miss = missed.iter().any(|i| {
1679 +
            !matches!(out[*i], Err(StoreError::NotFound(_)))
1680 +
                || self.find_packed(&refs[*i]).is_some()
1681 +
        });
1682 +
        let changed = if packed_miss {
1683 +
            self.reload_packs().map(|_| true)
1684 +
        } else {
1685 +
            self.reload_if_changed()
1686 +
        };
1687 +
        match changed {
1688 +
            Ok(true) => {
1689 +
                let again: Vec<ObjectRef> = missed.iter().map(|i| refs[*i]).collect();
1690 +
                for (i, g) in missed.into_iter().zip(self.get_many_now(&again)) {
1691 +
                    out[i] = g;
1692 +
                }
1693 +
            }
1694 +
            Ok(false) => {}
1695 +
            Err(e) => {
1696 +
                let why = e.to_string();
1697 +
                for i in missed {
1698 +
                    out[i] = Err(StoreError::Io(io::Error::other(why.clone())));
1699 +
                }
1700 +
            }
1701 +
        }
1702 +
        out
1703 +
    }
1704 +
1705 +
    /// [`Store::get_many`] against the packs as loaded.
1706 +
    fn get_many_now(&self, refs: &[ObjectRef]) -> Vec<Result<Object>> {
1305 1707
        use rayon::prelude::*;
1306 1708
        let backend = self.backend.as_ref();
1307 1709
        let packs = self.packs.borrow();
…
6 unchanged lines
1314 1716
                    .find_map(|(p, pack)| pack.find(r).map(|e| (p, e)))
1315 1717
            })
1316 1718
            .collect();
1317 -
        let indexes: &[PackIndex] = &packs;
1719 +
        let indexes: &[Arc<PackIndex>] = &packs;
1318 1720
        let found: Vec<Option<Result<Object>>> = refs
1319 1721
            .par_iter()
1320 1722
            .zip(place.par_iter())
…
502 unchanged lines
1823 2225
        out.sort();
1824 2226
        out.dedup();
1825 2227
        Ok(out)
2228 +
    }
2229 +
}
2230 +
2231 +
/// Whether a read failed in a way a newer list of packs may cure: the
2232 +
/// object was not found, or its pack's dictionary is gone.
2233 +
fn is_miss(e: &StoreError) -> bool {
2234 +
    match e {
2235 +
        StoreError::NotFound(_) => true,
2236 +
        StoreError::Corrupt(_, why) => why == NO_DICT,
2237 +
        _ => false,
1826 2238
    }
1827 2239
}
1828 2240
…
200 unchanged lines
2029 2441
        let (report, merged) = again.repack_all(REPACK_ZSTD_LEVEL).unwrap();
2030 2442
        assert_eq!((report.objects, merged), (0, 0));
2031 2443
        assert_eq!(packs(dir.path()), before);
2032 -
        // One new object: a repack replaces the pack and its dictionary.
2444 +
        // One new object: a repack replaces the pack and its dictionary,
2445 +
        // retiring the old ones, which a collection an hour on deletes.
2033 2446
        let extra = again.put(ObjectType::Chunk, b"one more\n").unwrap();
2034 2447
        again.repack_all(REPACK_ZSTD_LEVEL).unwrap();
2035 2448
        let files = packs(dir.path());
2036 -
        assert_eq!(files.len(), 3, "one pack, index and dictionary: {files:?}");
2037 -
        assert_ne!(files, before);
2449 +
        assert_eq!(
2450 +
            files.len(),
2451 +
            7,
2452 +
            "two of pack, index and dictionary, and the retired record: {files:?}"
2453 +
        );
2454 +
        assert_eq!(again.pack_count(), 1, "the old pack is not read");
2038 2455
        assert_eq!(&again.get(&refs[7]).unwrap().1, &bodies[7]);
2039 2456
        assert_eq!(again.get(&extra).unwrap().1, b"one more\n");
2457 +
        again
2458 +
            .expire_retired(std::time::Duration::ZERO, &mut silent())
2459 +
            .unwrap();
2460 +
        let files = packs(dir.path());
2461 +
        assert_eq!(files.len(), 4, "one pack, index, dictionary: {files:?}");
2462 +
        assert!(before
2463 +
            .iter()
2464 +
            .filter(|f| *f != RETIRED_KEY)
2465 +
            .all(|f| !files.contains(f)));
2466 +
        assert_eq!(&again.get(&refs[8]).unwrap().1, &bodies[8]);
2040 2467
    }
2041 2468
2042 2469
    #[test]
…
495 unchanged lines
2538 2965
        let missing = hash_object(ObjectType::Chunk, b"never stored");
2539 2966
        assert!(matches!(s.get(&missing), Err(StoreError::NotFound(_))));
2540 2967
        assert!(backend.1.load(Ordering::Relaxed) > 0);
2968 +
    }
2969 +
}
2970 +
2971 +
/// Readers never wait (oxygenhub#249): a store keeps reading while other
2972 +
/// handles repack, collect and push into the store under it.
2973 +
#[cfg(test)]
2974 +
pub(crate) mod reader_tests {
2975 +
    use super::*;
2976 +
2977 +
    fn chunk(i: usize) -> Vec<u8> {
2978 +
        format!("object {i} ").repeat(1 + i % 5).into_bytes()
2979 +
    }
2980 +
2981 +
    /// Two handles on one store: a writer, and a reader opened at once.
2982 +
    fn each_pair(f: impl Fn(&str, Store, Store)) {
2983 +
        let dir = tempfile::tempdir().unwrap();
2984 +
        let s = Store::open(dir.path().join("objects")).unwrap();
2985 +
        let r = Store::open(dir.path().join("objects")).unwrap();
2986 +
        f("directory", s, r);
2987 +
        let m: Arc<Backend> = Arc::new(MemoryStore::new());
2988 +
        let s = Store::open_on(m.clone(), dir.path().join("scratch")).unwrap();
2989 +
        let r = Store::open_on(m, dir.path().join("scratch-r")).unwrap();
2990 +
        f("memory", s, r);
2991 +
    }
2992 +
2993 +
    /// Set every retired entry's time back by `by`, as if retired then.
2994 +
    pub(crate) fn age_retired(s: &Store, by: std::time::Duration) {
2995 +
        s.update_retired(|m| {
2996 +
            for t in m.values_mut() {
2997 +
                *t = t.saturating_sub(by.as_millis() as u64);
2998 +
            }
2999 +
        })
3000 +
        .unwrap();
3001 +
    }
3002 +
3003 +
    fn retired_keys(s: &Store) -> Vec<String> {
3004 +
        s.read_retired().unwrap().0.into_keys().collect()
3005 +
    }
3006 +
3007 +
    #[test]
3008 +
    fn a_reader_opened_before_a_repack_reads_every_object_after_it() {
3009 +
        each_pair(|which, writer, _| {
3010 +
            let mut refs = Vec::new();
3011 +
            for i in 0..40 {
3012 +
                refs.push(writer.put(ObjectType::Chunk, &chunk(i)).unwrap());
3013 +
            }
3014 +
            writer.pack_loose().unwrap();
3015 +
            for i in 40..80 {
3016 +
                refs.push(writer.put(ObjectType::Chunk, &chunk(i)).unwrap());
3017 +
            }
3018 +
            // Opened now: one pack and forty loose objects.
3019 +
            let reader = Store::open_on(writer.backend().clone(), writer.dir()).unwrap();
3020 +
            assert_eq!(reader.pack_count(), 1, "{which}");
3021 +
            let (_, merged) = writer.repack_all(REPACK_ZSTD_LEVEL).unwrap();
3022 +
            assert_eq!(merged, 1);
3023 +
            // The old pack and the loose objects are retired, not deleted.
3024 +
            let retired = retired_keys(&writer);
3025 +
            assert_eq!(retired.len(), 81, "{which}: {retired:?}");
3026 +
            assert_eq!(writer.loose_count().unwrap(), 0, "retired is not loose");
3027 +
            // The reader reads on from what it loaded.
3028 +
            for (i, r) in refs.iter().enumerate() {
3029 +
                assert_eq!(reader.get(r).unwrap().1, chunk(i), "{which}: {i}");
3030 +
            }
3031 +
            assert_eq!(reader.pack_count(), 1);
3032 +
            // A store opened now reads the new pack only.
3033 +
            let fresh = Store::open_on(writer.backend().clone(), writer.dir()).unwrap();
3034 +
            assert_eq!(fresh.pack_count(), 1);
3035 +
            assert_ne!(fresh.packs.borrow()[0].key, reader.packs.borrow()[0].key);
3036 +
            // Once the retired files are deleted, the reader finds every
3037 +
            // object in the new pack, by get, get_many and has alike.
3038 +
            let report = writer
3039 +
                .expire_retired(std::time::Duration::ZERO, &mut silent())
3040 +
                .unwrap();
3041 +
            assert_eq!((report.packs, report.loose), (1, 80), "{which}");
3042 +
            assert!(retired_keys(&writer).is_empty());
3043 +
            for (i, r) in refs.iter().enumerate() {
3044 +
                assert_eq!(reader.get(r).unwrap().1, chunk(i), "{which}: {i}");
3045 +
            }
3046 +
            let other = Store::open_on(writer.backend().clone(), writer.dir()).unwrap();
3047 +
            let before = other.packs.borrow()[0].key.clone();
3048 +
            writer.put(ObjectType::Chunk, b"new").unwrap();
3049 +
            writer.repack_all(REPACK_ZSTD_LEVEL).unwrap();
3050 +
            writer
3051 +
                .expire_retired(std::time::Duration::ZERO, &mut silent())
3052 +
                .unwrap();
3053 +
            assert!(other.get_many(&refs).iter().all(|g| g.is_ok()), "{which}");
3054 +
            assert_ne!(other.packs.borrow()[0].key, before);
3055 +
            let third = Store::open_on(writer.backend().clone(), writer.dir()).unwrap();
3056 +
            writer.put(ObjectType::Chunk, b"newer").unwrap();
3057 +
            writer.repack_all(REPACK_ZSTD_LEVEL).unwrap();
3058 +
            writer
3059 +
                .expire_retired(std::time::Duration::ZERO, &mut silent())
3060 +
                .unwrap();
3061 +
            assert!(refs.iter().all(|r| third.has(r)), "{which}");
3062 +
            assert!(!third.has(&hash_object(ObjectType::Chunk, b"never")));
3063 +
        });
3064 +
    }
3065 +
3066 +
    #[test]
3067 +
    fn a_reader_opened_before_a_spilled_batch_reads_its_objects() {
3068 +
        each_pair(|which, writer, reader| {
3069 +
            assert_eq!(reader.pack_count(), 0);
3070 +
            let mut refs = Vec::new();
3071 +
            {
3072 +
                let batch = writer.begin_batch().unwrap();
3073 +
                for i in 0..(SPILL_OBJECTS + 50) {
3074 +
                    refs.push(writer.put(ObjectType::Chunk, &chunk(i)).unwrap());
3075 +
                }
3076 +
                batch.finish().unwrap();
3077 +
            }
3078 +
            assert_eq!(writer.pack_count(), 1, "{which}: the batch spilled");
3079 +
            assert_eq!(reader.get(&refs[0]).unwrap().1, chunk(0), "{which}");
3080 +
            assert_eq!(reader.pack_count(), 1, "the miss loaded the new pack");
3081 +
            let got = reader.get_many(&refs);
3082 +
            assert!(got.iter().all(|g| g.is_ok()), "{which}");
3083 +
            // has, too, from another reader.
3084 +
            let late = Store::open_on(writer.backend().clone(), writer.dir()).unwrap();
3085 +
            {
3086 +
                let batch = writer.begin_batch().unwrap();
3087 +
                for i in 1000..(1000 + SPILL_OBJECTS + 1) {
3088 +
                    writer.put(ObjectType::Chunk, &chunk(i)).unwrap();
3089 +
                }
3090 +
                batch.finish().unwrap();
3091 +
            }
3092 +
            assert!(late.has(&hash_object(ObjectType::Chunk, &chunk(1000))));
3093 +
        });
3094 +
    }
3095 +
3096 +
    #[test]
3097 +
    fn retired_entries_expire_after_the_hour_and_are_never_loaded() {
3098 +
        each_pair(|which, s, _| {
3099 +
            for i in 0..10 {
3100 +
                s.put(ObjectType::Chunk, &chunk(i)).unwrap();
3101 +
            }
3102 +
            s.pack_loose().unwrap();
3103 +
            s.put(ObjectType::Chunk, &chunk(10)).unwrap();
3104 +
            s.repack_all(ZSTD_LEVEL).unwrap();
3105 +
            // Retired over an hour ago: the first pack and 10 loose objects.
3106 +
            age_retired(&s, RETIRED_FOR + std::time::Duration::from_secs(60));
3107 +
            let old = retired_keys(&s);
3108 +
            assert_eq!(old.len(), 12, "{which}: {old:?}");
3109 +
            // Retired just now: the second pack.
3110 +
            s.put(ObjectType::Chunk, &chunk(11)).unwrap();
3111 +
            s.repack_all(ZSTD_LEVEL).unwrap();
3112 +
            assert_eq!(retired_keys(&s).len(), 14);
3113 +
            let listed = s
3114 +
                .backend()
3115 +
                .list("pack-")
3116 +
                .unwrap()
3117 +
                .into_iter()
3118 +
                .filter(|e| e.key.ends_with(".idx"))
3119 +
                .count();
3120 +
            assert_eq!(listed, 3, "{which}");
3121 +
            let fresh = Store::open_on(s.backend().clone(), s.dir()).unwrap();
3122 +
            assert_eq!(
3123 +
                fresh.pack_count(),
3124 +
                1,
3125 +
                "{which}: retired packs are not loaded"
3126 +
            );
3127 +
            let report = s.expire_retired(RETIRED_FOR, &mut silent()).unwrap();
3128 +
            assert_eq!(
3129 +
                report,
3130 +
                RetiredReport {
3131 +
                    packs: 1,
3132 +
                    loose: 11,
3133 +
                    kept: 2
3134 +
                },
3135 +
                "{which}"
3136 +
            );
3137 +
            let young = retired_keys(&s);
3138 +
            assert_eq!(young.len(), 2, "{which}: {young:?}");
3139 +
            assert!(young.iter().all(|k| !old.contains(k)));
3140 +
            for k in old.iter().filter(|k| k.ends_with(".pack")) {
3141 +
                assert!(!s.backend().has(k).unwrap(), "{which}: {k}");
3142 +
            }
3143 +
            for k in &young {
3144 +
                assert!(s.backend().has(k).unwrap(), "{which}: {k} kept");
3145 +
            }
3146 +
            for i in 0..12 {
3147 +
                let r = hash_object(ObjectType::Chunk, &chunk(i));
3148 +
                assert_eq!(fresh.get(&r).unwrap().1, chunk(i), "{which}");
3149 +
            }
3150 +
        });
3151 +
    }
3152 +
3153 +
    /// A retired loose object whose pack copy was pruned since is not
3154 +
    /// lost when the retired files go: it is loose again.
3155 +
    #[test]
3156 +
    fn a_retired_loose_object_no_pack_holds_is_kept() {
3157 +
        each_pair(|which, s, _| {
3158 +
            let a = s.put(ObjectType::Chunk, b"a").unwrap();
3159 +
            let b = s.put(ObjectType::Chunk, b"b").unwrap();
3160 +
            s.pack_loose().unwrap();
3161 +
            assert_eq!(s.loose_count().unwrap(), 0);
3162 +
            // The pack rewritten without b, as pruning does.
3163 +
            let first = s.packs.borrow()[0].key.clone();
3164 +
            s.write_pack(&[a], &[], &[first], ZSTD_LEVEL, &mut silent())
3165 +
                .unwrap();
3166 +
            let report = s
3167 +
                .expire_retired(std::time::Duration::ZERO, &mut silent())
3168 +
                .unwrap();
3169 +
            assert_eq!((report.packs, report.loose), (1, 1), "{which}: a's copy");
3170 +
            assert!(retired_keys(&s).is_empty());
3171 +
            assert_eq!(s.get(&b).unwrap().1, b"b", "{which}");
3172 +
            assert_eq!(s.get(&a).unwrap().1, b"a");
3173 +
            assert_eq!(s.loose_count().unwrap(), 1, "b is loose again");
3174 +
            // And pruning it, unreachable, deletes it.
3175 +
            let keep = [a].into_iter().collect();
3176 +
            let r = s.prune(&keep, std::time::Duration::ZERO).unwrap();
3177 +
            assert_eq!(r.objects, 1);
3178 +
            assert!(!s.has(&b));
3179 +
        });
3180 +
    }
3181 +
3182 +
    /// A backend whose first pack listing names an index that is deleted
3183 +
    /// before it is read, as a collection running meanwhile would.
3184 +
    pub(crate) struct Racing {
3185 +
        inner: MemoryStore,
3186 +
        lists: std::sync::atomic::AtomicUsize,
3187 +
        /// Deleted right after the first listing returns.
3188 +
        pub(crate) doomed: std::sync::Mutex<Option<String>>,
3189 +
        /// Named in every listing, never stored.
3190 +
        ghost: bool,
3191 +
    }
3192 +
3193 +
    impl crate::strata::Store for Racing {
3194 +
        fn describe(&self) -> String {
3195 +
            "racing".into()
3196 +
        }
3197 +
        fn has(&self, k: &str) -> crate::strata::Result<bool> {
3198 +
            self.inner.has(k)
3199 +
        }
3200 +
        fn get(&self, k: &str) -> crate::strata::Result<Vec<u8>> {
3201 +
            self.inner.get(k)
3202 +
        }
3203 +
        fn get_range(&self, k: &str, r: std::ops::Range<u64>) -> crate::strata::Result<Vec<u8>> {
3204 +
            self.inner.get_range(k, r)
3205 +
        }
3206 +
        fn put_new(&self, k: &str, b: &[u8]) -> crate::strata::Result<()> {
3207 +
            self.inner.put_new(k, b)
3208 +
        }
3209 +
        fn list(&self, p: &str) -> crate::strata::Result<Vec<crate::strata::Entry>> {
3210 +
            let mut out = self.inner.list(p)?;
3211 +
            if p == "pack-" {
3212 +
                self.lists
3213 +
                    .fetch_add(1, std::sync::atomic::Ordering::Relaxed);
3214 +
                if let Some(k) = self.doomed.lock().unwrap().take() {
3215 +
                    self.inner.delete(&k)?;
3216 +
                }
3217 +
                if self.ghost {
3218 +
                    for k in ["pack-0000.idx", "pack-0000.pack"] {
3219 +
                        out.push(crate::strata::Entry {
3220 +
                            key: k.into(),
3221 +
                            size: 100,
3222 +
                            modified_ms: None,
3223 +
                        });
3224 +
                    }
3225 +
                    out.sort_by(|a, b| a.key.cmp(&b.key));
3226 +
                }
3227 +
            }
3228 +
            Ok(out)
3229 +
        }
3230 +
        fn delete(&self, k: &str) -> crate::strata::Result<()> {
3231 +
            self.inner.delete(k)
3232 +
        }
3233 +
        fn read_ref(
3234 +
            &self,
3235 +
            k: &str,
3236 +
        ) -> crate::strata::Result<Option<(Vec<u8>, crate::strata::Version)>> {
3237 +
            self.inner.read_ref(k)
3238 +
        }
3239 +
        fn write_ref(
3240 +
            &self,
3241 +
            k: &str,
3242 +
            b: &[u8],
3243 +
            e: Option<&crate::strata::Version>,
3244 +
        ) -> crate::strata::Result<crate::strata::Version> {
3245 +
            self.inner.write_ref(k, b, e)
3246 +
        }
3247 +
    }
3248 +
3249 +
    pub(crate) fn racing(ghost: bool) -> Arc<Racing> {
3250 +
        Arc::new(Racing {
3251 +
            inner: MemoryStore::new(),
3252 +
            lists: Default::default(),
3253 +
            doomed: Default::default(),
3254 +
            ghost,
3255 +
        })
3256 +
    }
3257 +
3258 +
    #[test]
3259 +
    fn an_index_deleted_between_listing_and_loading_is_passed_over() {
3260 +
        let dir = tempfile::tempdir().unwrap();
3261 +
        let backend = racing(false);
3262 +
        let s = Store::open_on(backend.clone(), dir.path()).unwrap();
3263 +
        let a = s.put(ObjectType::Chunk, b"a").unwrap();
3264 +
        s.pack_loose().unwrap();
3265 +
        let b = s.put(ObjectType::Chunk, b"b").unwrap();
3266 +
        s.pack_loose().unwrap();
3267 +
        let idx = {
3268 +
            let packs = s.packs.borrow();
3269 +
            let (i, _) = s.find_packed(&a).unwrap();
3270 +
            format!("{}.idx", packs[i].key.trim_end_matches(".pack"))
3271 +
        };
3272 +
        *backend.doomed.lock().unwrap() = Some(idx);
3273 +
        let before = backend.lists.load(std::sync::atomic::Ordering::Relaxed);
3274 +
        let again = Store::open_on(backend.clone(), dir.path()).unwrap();
3275 +
        assert_eq!(again.pack_count(), 1, "the deleted one is passed over");
3276 +
        assert_eq!(
3277 +
            backend.lists.load(std::sync::atomic::Ordering::Relaxed) - before,
3278 +
            2,
3279 +
            "and the store listed again"
3280 +
        );
3281 +
        assert_eq!(again.get(&b).unwrap().1, b"b");
3282 +
        // A listing that names an index never there opens too.
3283 +
        let ghosted = racing(true);
3284 +
        let s = Store::open_on(ghosted.clone(), dir.path().join("g")).unwrap();
3285 +
        let c = s.put(ObjectType::Chunk, b"c").unwrap();
3286 +
        s.pack_loose().unwrap();
3287 +
        let again = Store::open_on(ghosted, dir.path().join("g")).unwrap();
3288 +
        assert_eq!(again.pack_count(), 1);
3289 +
        assert_eq!(again.get(&c).unwrap().1, b"c");
2541 3290
    }
2542 3291
}
2543 3292
…
195 unchanged lines
modified crates/oxy-strata/src/lib.rs +67 −18
…
484 unchanged lines
485 485
    }
486 486
}
487 487
488 +
/// Run a request that writes or deletes `key`, forgetting what the caches
489 +
/// hold for it both before it is sent and after it completes, whatever it
490 +
/// returns: a listing or a miss taken by another thread while the request
491 +
/// was in flight may predate it, and would otherwise be kept for
492 +
/// `MISS_FOR` (oxygenhub#249).
493 +
fn writing<T>(misses: &Misses, listings: &Listings, key: &str, request: impl FnOnce() -> T) -> T {
494 +
    misses.written(key);
495 +
    listings.written(key);
496 +
    let out = request();
497 +
    misses.written(key);
498 +
    listings.written(key);
499 +
    out
500 +
}
501 +
488 502
impl Drop for CloudStore {
489 503
    /// Without waiting, so a store can be dropped inside a runtime too.
490 504
    fn drop(&mut self) {
…
255 unchanged lines
746 760
747 761
    fn put_new(&self, key: &str, bytes: &[u8]) -> Result<()> {
748 762
        let path = self.path(key)?;
749 -
        self.misses.written(key);
750 -
        self.listings.written(key);
751 763
        let opts = PutOptions {
752 764
            mode: PutMode::Create,
753 765
            ..Default::default()
754 766
        };
755 767
        counted(Op::Put);
756 -
        self.block(
757 -
            self.inner
758 -
                .put_opts(&path, PutPayload::from(bytes.to_vec()), opts),
759 -
        )
768 +
        writing(&self.misses, &self.listings, key, || {
769 +
            self.block(
770 +
                self.inner
771 +
                    .put_opts(&path, PutPayload::from(bytes.to_vec()), opts),
772 +
            )
773 +
        })
760 774
        .map(|_| ())
761 775
        .map_err(|e| self.map(e, key))
762 776
    }
…
57 unchanged lines
820 834
821 835
    fn delete(&self, key: &str) -> Result<()> {
822 836
        let path = self.path(key)?;
823 -
        self.listings.written(key);
824 837
        counted(Op::Delete);
825 -
        match self.block(self.inner.delete(&path)) {
838 +
        match writing(&self.misses, &self.listings, key, || {
839 +
            self.block(self.inner.delete(&path))
840 +
        }) {
826 841
            Ok(()) | Err(object_store::Error::NotFound { .. }) => Ok(()),
827 842
            Err(e) => Err(self.map(e, key)),
828 843
        }
…
20 unchanged lines
849 864
850 865
    fn write_ref(&self, key: &str, bytes: &[u8], expected: Option<&Version>) -> Result<Version> {
851 866
        let path = self.path(key)?;
852 -
        self.misses.written(key);
853 -
        self.listings.written(key);
854 867
        let mode = match expected {
855 868
            None => PutMode::Create,
856 869
            Some(v) => PutMode::Update(Self::update_version(v)),
…
3 unchanged lines
860 873
            ..Default::default()
861 874
        };
862 875
        counted(Op::WriteRef);
863 -
        let put = self
864 -
            .block(
876 +
        let put = writing(&self.misses, &self.listings, key, || {
877 +
            self.block(
865 878
                self.inner
866 879
                    .put_opts(&path, PutPayload::from(bytes.to_vec()), opts),
867 880
            )
868 -
            .map_err(|e| match self.map(e, key) {
869 -
                // A record that exists when none was expected is a lost
870 -
                // race too, not a misuse of put_new.
871 -
                Error::Exists(k) => Error::Conflict(k),
872 -
                other => other,
873 -
            })?;
881 +
        })
882 +
        .map_err(|e| match self.map(e, key) {
883 +
            // A record that exists when none was expected is a lost
884 +
            // race too, not a misuse of put_new.
885 +
            Error::Exists(k) => Error::Conflict(k),
886 +
            other => other,
887 +
        })?;
874 888
        Ok(Self::version_of(put.e_tag, put.version))
875 889
    }
876 890
}
…
39 unchanged lines
916 930
        l.put("pack-", &[e("pack-1.idx")]);
917 931
        l.0.lock().unwrap().get_mut("pack-").unwrap().0 -= super::MISS_FOR;
918 932
        assert!(l.get("pack-").is_none(), "and it ages out");
933 +
    }
934 +
935 +
    /// A listing or miss taken by another thread while a write is in
936 +
    /// flight is forgotten when the write completes, not kept stale for
937 +
    /// MISS_FOR (oxygenhub#249), whether the write succeeded or not.
938 +
    #[test]
939 +
    fn a_listing_taken_during_a_write_is_forgotten_after_it() {
940 +
        let (m, l) = (super::Misses::default(), super::Listings::default());
941 +
        let e = |k: &str| oxy_core::strata::Entry {
942 +
            key: k.into(),
943 +
            size: 1,
944 +
            modified_ms: None,
945 +
        };
946 +
        l.put("pack-", &[e("pack-1.idx")]);
947 +
        let got = super::writing(&m, &l, "pack-2.idx", || {
948 +
            assert!(l.get("pack-").is_none(), "forgotten before it is sent");
949 +
            // Another thread lists, and misses, mid-request.
950 +
            l.put("pack-", &[e("pack-1.idx")]);
951 +
            m.missed("pack-2.idx");
952 +
            7
953 +
        });
954 +
        assert_eq!(got, 7);
955 +
        assert!(l.get("pack-").is_none(), "the mid-request listing is gone");
956 +
        assert!(!m.known("pack-2.idx"), "and the miss");
957 +
        // A delete that fails is treated the same.
958 +
        l.put("pack-", &[e("pack-1.idx")]);
959 +
        let r: std::result::Result<(), &str> = super::writing(&m, &l, "pack-1.idx", || {
960 +
            l.put("pack-", &[e("pack-1.idx")]);
961 +
            Err("network")
962 +
        });
963 +
        assert!(r.is_err() && l.get("pack-").is_none());
964 +
        // Other prefixes' writes leave the pack listing.
965 +
        l.put("pack-", &[e("pack-1.idx")]);
966 +
        super::writing(&m, &l, "loose/ab/cd", || ());
967 +
        assert!(l.get("pack-").is_some());
919 968
    }
920 969
921 970
    #[test]
…
121 unchanged lines
modified docs/spec.md +9 −1
…
481 unchanged lines
482 482
483 483
`oxy undo` undoes **this workspace's** last command (oxygen#99): the newest operation recorded by this workspace (for operations older than the `workspace` field, one that moved only other workspaces' working copies is theirs), passing over working-copy snapshots, pushes (the remote keeps what it was sent) and commands an earlier undo reverted, and taking a command's own `(checkout)` with it. When no other workspace recorded an operation since, the head moves back to the command's parent. Otherwise the command is reverted in place as a new operation naming it in `undoes`: each reference it changed returns to its value before it; a bookmark another workspace moved since is put back head by head (the heads the command added go, those it replaced return, the other workspace's stay); and a reference that cannot be put back so (a later change builds on one the command added, or another workspace changed the same tag, remote reference, policy or working copy) refuses the undo, naming it. Undo reports the command it undid, what that changed, the other workspaces' operations it kept, and bookmarks whose undone changes a remote already holds. `oxy redo` and `oxy op log` expose the DAG. Operations are retained for `core.op-retention` (default 90 days) and then eligible for garbage collection.
484 484
485 +
A process reads the operation head once, when it first needs it, and answers every later read of references, bookmarks and the view from that operation (oxygenhub#249): one command, or one request a host serves, never mixes two heads, and never waits on another process's operation to read. Its own operations move it to the head each one recorded. It reads the head again only where it must see what others did meanwhile: once it holds the working-copy lock (whoever held the lock may have moved the head), when a host receives a push (the updates are checked against the head as it is now), in `oxy fsck`, and wherever the implementation asks for it explicitly (`Repo::refresh`). An operation recorded from an older head merges with the newer one as any concurrent operation does.
486 +
485 487
Because all mutation goes through the operation log and object writes are idempotent, no long-held repository lock is needed: the operation head's lock covers one pointer write. Each workspace has a **working-copy lock** (`.oxy/workspaces/<id>/lock`), held by every command that reads or writes the working copy and replaces its working-copy change (snapshot, checkout, bookmark switch, merge, commit, amend and rewrites, discard, pull), from reading the operation head until its operation is recorded. A snapshot therefore never sees a half-written checkout, and never records one as edits. The lock is re-entrant within a process thread; a waiting command gives up after a minute and says why; a lock whose process is gone is taken over, and where that cannot be told, one older than a minute.
486 488
487 489
### 12.2 Workspaces
…
29 unchanged lines
517 519
  objects/
518 520
    pack-<id>.pack    append-only pack file
519 521
    pack-<id>.idx     sorted (ObjectRef → offset, length, type) index
522 +
    pack-<id>.dict    the pack's zstd dictionary, when a full repack trained one
523 +
    pack-retired      packs and loose objects a repack replaced, kept for readers (§13.4)
520 524
    loose/            objects written since the last pack, as header || body
521 525
    sigs/             signature lists keyed by the ObjectRef they cover (§5.4)
522 526
  ops/                operation objects and op heads
…
29 unchanged lines
552 556
### 13.4 Packing
553 557
554 558
An automatic trigger when the loose object count exceeds 4096 writes loose objects into a new pack and merges packs when there are more than 16; `oxy gc` repacks every surviving object into one pack, compressed harder (zstd level 9, against level 3 for everyday writes) and on every core, as `git gc` repacks. A store that is already one such pack (it has a dictionary) with nothing loose is left as it is: repacking it would write the same pack. Writers SHOULD batch: a snapshot, a fetched bundle, a Git import, or a checkpoint is one write batch, and a batch that grows beyond a small threshold (256 objects or 8 MiB) is written as a single pack with one `fsync` rather than as loose objects, each with its own; smaller batches stay loose so ordinary commits do not multiply packs. Objects of an open batch MUST be readable by the writer before the batch completes. Pack indexes MUST support lookup by ObjectRef prefix and MUST be memory-mappable. A repository with two million objects MUST be openable with under 64 MiB of resident index memory on the Full profile.
559 +
560 +
Readers never wait for writers (oxygenhub#249). A store loads its list of packs when it opens and keeps it; a read that misses is not believed at once. When the pack an object was found in is gone, or its dictionary is, the list is loaded again and the read tried once more; when the object is in no pack and not loose, the store lists its packs, and if the listing differs from the one it loaded from, loads the new packs and tries once more. Only then is the object missing (`has` answers false). A pack whose index is gone between the listing and the read, as when a collection deletes it meanwhile, is passed over and the store listed once more, so opening a repository never fails for it. A writer asking whether it already holds many objects lists the packs at most once a second while nothing new turns up.
561 +
562 +
A repack does not delete what it replaces. The packs it folded in and the loose objects it packed are **retired**: recorded, with the time, in `pack-retired`, a reference record of lines `<wall ms> <key>` updated by compare-and-swap, so concurrent repacks and collections never lose each other's lines. A store opening, or loading its packs again, does not load a retired pack, and retired loose objects are not counted or packed again as loose; a reader that loaded its packs before the repack reads on from the old files. Each `oxy gc`, and each automatic pack check after a commit, first deletes what was retired at least an hour ago: each such pack with its index and dictionary, and each such loose object a live pack holds. A retired loose object no live pack holds (its pack copy was pruned since) is kept as an ordinary loose object. Readers that loaded their packs before the files went find the objects in the new pack by the reload above. The hour is independent of `core.gc-grace` (§21): `oxy gc --now` waives the grace period for unreachable objects, not the hour readers have.
555 563
556 564
The pack and index formats are fixed so that tools can read them without the implementation:
557 565
…
280 unchanged lines
838 846
839 847
## 21. Garbage collection
840 848
841 -
An object is **reachable** if it is referenced, directly or transitively, from any bookmark, tag, policy, the current operation heads, any operation newer than `core.op-retention`, any workspace's current working-copy change, the working-copy changes of operations newer than `core.snapshot-retention` (default 7 days; `oxy gc --expire-snapshots` keeps none but the current ones, oxygen#98), or any checkpoint of a reachable state. A file that was only ever in a working copy, never committed, so goes once the snapshots that held it expire. `oxy gc` deletes unreachable objects that are older than `core.gc-grace` (default 14 days, to protect concurrent writers) and repacks. Tombstone atoms are never collected independently of their change. Divergent changes are reachable if any of their variants is. A change's text blobs and their chunks are reachable with it (§7.4). A seal is reachable if any change it covers is reachable, and reference attestations are reachable while they are the latest for their reference or are within `core.op-retention`.
849 +
An object is **reachable** if it is referenced, directly or transitively, from any bookmark, tag, policy, the current operation heads, any operation newer than `core.op-retention`, any workspace's current working-copy change, the working-copy changes of operations newer than `core.snapshot-retention` (default 7 days; `oxy gc --expire-snapshots` keeps none but the current ones, oxygen#98), or any checkpoint of a reachable state. A file that was only ever in a working copy, never committed, so goes once the snapshots that held it expire. `oxy gc` first deletes the packs and loose objects earlier repacks retired over an hour ago (§13.4), then deletes unreachable objects that are older than `core.gc-grace` (default 14 days, to protect concurrent writers) and repacks; what this repack replaces is retired in turn, for a later collection. Tombstone atoms are never collected independently of their change. Divergent changes are reachable if any of their variants is. A change's text blobs and their chunks are reachable with it (§7.4). A seal is reachable if any change it covers is reachable, and reference attestations are reachable while they are the latest for their reference or are within `core.op-retention`.
842 850
843 851
### 21.1 Integrity check
844 852
…
213 unchanged lines
modified src/cli/workspace.rs +24 −0
…
1690 unchanged lines
1691 1691
        },
1692 1692
        plural(repo.store().pack_count(), "pack")
1693 1693
    );
1694 +
    // What earlier repacks replaced stays an hour for readers still using
1695 +
    // it (oxygenhub#249); a collection after that deletes it.
1696 +
    let kept = if r.retired.kept > 0 {
1697 +
        format!(
1698 +
            "{} retired less than an hour ago kept for readers",
1699 +
            plural(r.retired.kept, "file")
1700 +
        )
1701 +
    } else {
1702 +
        String::new()
1703 +
    };
1704 +
    if r.retired.packs + r.retired.loose > 0 {
1705 +
        println!(
1706 +
            "Deleted {} and {} retired by earlier repacks{}.",
1707 +
            plural(r.retired.packs, "pack"),
1708 +
            plural(r.retired.loose, "loose object"),
1709 +
            if kept.is_empty() {
1710 +
                String::new()
1711 +
            } else {
1712 +
                format!("; {kept}")
1713 +
            }
1714 +
        );
1715 +
    } else if !kept.is_empty() {
1716 +
        println!("{kept}.");
1717 +
    }
1694 1718
    Ok(0)
1695 1719
}
1696 1720
…
1204 unchanged lines