OXYGENHUB

The object store runs on any Store: a directory today, a bucket next 648189388f1a

B1 of plan B (#68): store::Store keeps its API and puts every disk
touch behind a strata::Store backend plus a scratch directory for
what must be on this disk (a pack being streamed out, the derived
metas cache). The keys are the directory layout as it was, so
.oxy/objects is a LocalStore with the same files, stats, reads and
syncs: a loose object is one file, a packed one is one ranged read of
the pack, a small batch of loose objects is written with its syncs
batched (put_many_new), and a finished pack is linked into place, not
copied (put_new_file). The trait gains stat and modification times,
which gc's grace period reads.

Signatures, seal and receipt entries become one write-once key each
(sig/, seal/, receipt/ under the object), so two writers of a shared
store never lose each other's; the one-list-per-object files older
stores hold are still read, and pruning takes both. A signer's later
signature replaces its earlier one.

Store::open_on(backend, scratch) opens a store on anything; under
OXY_STORE_BACKEND=memory every store opens in memory, and the whole
oxy-core suite passes that way too (the tests that tamper with files
step aside). The store tests run on both.

refs #68
Author
Andy McCall @andymccall <[email protected]>
When
· 2026-10-04 11:50 UTC
Signature
✔ signed by Andy McCall
Hash
648189388f1aa19590d8e4e476c38aedc52a3be95d2e6b51250c68c579e4840c
Depends on
135212de5370 192fad1e46a2 1983686fb7a7 2b872d074ddb 387335ff7f4d 4dcb8c0916b7 5b1114a04905 5b9c6ad45f6e 6f8aeb996558 75167672ebcd 86e1824f2d0b 8d80356371c5 8e1194860efa 8f10a07a1dbe 982864068a59 9ae4d15a0e4b a17fc622ca43 a5f76c79666c a70dce536476 a9f686edf60b afaba5f1306d b30a880fa409 ba160f60219e cdb5beba7f05 ce14f0d68b15 d02e545dc265 d77e1702d770 da652d6392d8 fc1a466a1e66 ff0af1ee299d
Depended on by
795b8b502ded 544d0541a7e5 c6cec7822aff ef7d4ea6f33b 767ac6cd0545 7e0e8e5ad882 a67914bced23 092457020692 3a4b3e018326 454236d1108d dd21fd82687d
In bookmarks
agents-in-readme bare-hashes carried-not-removed carry-identical doctor-trust identical-sides keyless-tags main merge-remote-bookmark only-help-103 paths-for-resolutions-and-amend push-notes show-stat subrepo-pin-state switch-says
In tags
v0.10.0 v0.11.0 v0.4.0 v0.4.1 v0.4.2 v0.4.3 v0.4.4 v0.4.5 v0.4.6 v0.5.0 v0.5.1 v0.6.0 v0.7.0 v0.8.0 v0.9.0
Edits
183 atoms, 0 tree operations, 5 files

Browse the tree at this change

modified crates/oxy-core/src/repo/tests.rs +13 −3
…
3965 unchanged lines
3966 3966
3967 3967
#[test]
3968 3968
fn fsck_passes_a_sound_repository_and_names_each_fault() {
3969 +
    // Files on disk are what this tests: tampering with them, and the
3970 +
    // directory store's crash points.
3971 +
    if std::env::var_os("OXY_STORE_BACKEND").is_some() {
3972 +
        return;
3973 +
    }
3969 3974
    hermetic();
3970 3975
    let dir = tempfile::tempdir().unwrap();
3971 3976
    let root = dir.path();
…
105 unchanged lines
4077 4082
/// accept the same operation again.
4078 4083
#[test]
4079 4084
fn crash_at_every_step_of_commit_and_push_leaves_sound_repositories() {
4085 +
    // Files on disk are what this tests: tampering with them, and the
4086 +
    // directory store's crash points.
4087 +
    if std::env::var_os("OXY_STORE_BACKEND").is_some() {
4088 +
        return;
4089 +
    }
4080 4090
    use crate::durable::failpoint;
4081 4091
    hermetic();
4082 4092
    let dir = tempfile::tempdir().unwrap();
…
767 unchanged lines
4850 4860
    )
4851 4861
    .unwrap();
4852 4862
    let graph = repo2.checkpoint_graph_blobs(&cp).unwrap()[0];
4853 -
    fs::remove_file(repo2.store().path_for(&graph)).unwrap();
4863 +
    repo2.store().remove_loose_for_test(&graph).unwrap();
4854 4864
    assert!(repo2.load_state(&repo2.bookmark_heads().unwrap()).is_err());
4855 4865
    assert!(repo2.fsck().unwrap().errors() > 0, "fsck reports it");
4856 4866
}
…
1300 unchanged lines
6157 6167
        })
6158 6168
        .find(|r| content(r).windows(12).any(|w| w == b"other line 1"))
6159 6169
        .unwrap();
6160 -
    fs::remove_file(repo.store().path_for(&other_text)).unwrap();
6170 +
    repo.store().remove_loose_for_test(&other_text).unwrap();
6161 6171
    let mut hist = repo.load_state_history(&[h2]).unwrap();
6162 6172
    repo.load_graphs_of(&mut hist, &[big_node]).unwrap();
6163 6173
    assert!(repo
…
409 unchanged lines
6573 6583
    let Some(crate::objects::TextRef::Blob(b, _, _)) = texts.first().copied() else {
6574 6584
        panic!("a stored text")
6575 6585
    };
6576 -
    fs::remove_file(repo.store().path_for(&b)).unwrap();
6586 +
    repo.store().remove_loose_for_test(&b).unwrap();
6577 6587
    let rep = repo.fsck().unwrap();
6578 6588
    assert!(rep.errors() > 0);
6579 6589
    assert!(
…
402 unchanged lines
modified crates/oxy-core/src/store.rs +828 −575
1 -
//! Object store (spec §13.1, §13.3, §13.4): loose objects, one file per
2 -
//! object under `objects/loose/<hh>/<rest>`, and packs, `pack-<id>.pack`
3 -
//! with a sorted `pack-<id>.idx`, holding zstd-compressed objects. Reads
4 -
//! check loose objects first, then every pack; every read re-hashes.
1 +
//! Object store (spec §13.1, §13.3, §13.4): loose objects, one per key
2 +
//! under `loose/<hh>/<rest>`, and packs, `pack-<id>.pack` with a sorted
3 +
//! `pack-<id>.idx`, holding zstd-compressed objects. Reads check loose
4 +
//! objects first, then every pack; every read re-hashes.
5 5
//!
6 +
//! Everything is kept on a [`crate::strata::Store`] (docs/stores.md): a
7 +
//! directory, which is what `.oxy/objects` is, or a bucket, which is how a
8 +
//! repository lives in its owner's region with the hub as one client of
9 +
//! it. The keys are the same wherever the store is, so a bucket holds
10 +
//! exactly what a directory would, and a directory store is as fast as
11 +
//! the files were before there was a trait: a loose object is one file, a
12 +
//! packed one is one ranged read of the pack. What must be local, the
13 +
//! pack being streamed out and the derived `metas` cache, lives in a
14 +
//! scratch directory.
15 +
//!
6 16
//! Pack file: `OXYPACK1` then entries of
7 17
//! `type u8 | flags u8 (bit 0: zstd) | stored_len u32 | raw_len u32 | ref 32 | body`.
8 18
//! Index file: `OXYIDX01` then `count u32` then sorted entries of
9 19
//! `ref 32 | offset u64 | stored_len u32 | raw_len u32 | type u8 | flags u8`.
20 +
//!
21 +
//! Beside the objects: `nocompress/<hex>` marks a chunk packs keep raw;
22 +
//! `sig/<hh>/<rest>/<key>` holds one signature over an object, written
23 +
//! once per signer; `seal/<hh>/<rest>/<attest>` and
24 +
//! `receipt/<hh>/<rest>/<attest>` index the seals covering a change and
25 +
//! the receipts on a state. Older stores hold these as one list per
26 +
//! object (`sigs/`, `seals/`, `receipts/`), which are still read.
10 27
11 28
use crate::attest::Attest;
12 29
use crate::hash::{hash_object, header, ObjectRef, ObjectType};
13 30
use crate::identity::{Identity, Signature};
14 31
use crate::objects::{Blob, Change, ObjectError, StateObject};
15 32
use crate::receipt::Receipt;
33 +
use crate::strata::{self, Error as BackendError, LocalStore, MemoryStore};
16 34
use rand::RngCore;
17 35
use std::cell::RefCell;
18 36
use std::fs;
19 -
use std::io::{self, Read, Seek, SeekFrom, Write};
37 +
use std::io::{self, Write};
20 38
use std::path::{Path, PathBuf};
39 +
use std::sync::Arc;
21 40
22 41
const PACK_MAGIC: &[u8; 8] = b"OXYPACK1";
23 42
const IDX_MAGIC: &[u8; 8] = b"OXYIDX01";
…
22 unchanged lines
46 65
/// Where a long operation reports what it is doing (oxygen#66): a phase,
47 66
/// or a count within one, as a line for a spinner or a log.
48 67
pub type Progress<'a> = &'a mut dyn FnMut(&str);
68 +
69 +
/// The storage under a store: objects by key, see [`crate::strata`].
70 +
pub type Backend = dyn strata::Store;
49 71
50 72
fn silent() -> impl FnMut(&str) {
51 73
    |_| {}
74 +
}
75 +
76 +
/// A backend error as a store error: what is not found or already there
77 +
/// is handled where it matters; the rest is an I/O failure.
78 +
fn backend_err(e: BackendError) -> StoreError {
79 +
    match e {
80 +
        BackendError::Io(e) => StoreError::Io(e),
81 +
        other => StoreError::Io(io::Error::other(other.to_string())),
82 +
    }
83 +
}
84 +
85 +
/// `put_new` where an object already there is what we wanted.
86 +
fn put_new_or_existing(backend: &Backend, key: &str, bytes: &[u8]) -> Result<()> {
87 +
    match backend.put_new(key, bytes) {
88 +
        Ok(()) | Err(BackendError::Exists(_)) => Ok(()),
89 +
        Err(e) => Err(backend_err(e)),
90 +
    }
91 +
}
92 +
93 +
/// `get` as an option: `None` when there is no such key.
94 +
fn get_optional(backend: &Backend, key: &str) -> Result<Option<Vec<u8>>> {
95 +
    match backend.get(key) {
96 +
        Ok(b) => Ok(Some(b)),
97 +
        Err(BackendError::NotFound(_)) => Ok(None),
98 +
        Err(e) => Err(backend_err(e)),
99 +
    }
100 +
}
101 +
102 +
fn delete(backend: &Backend, key: &str) -> Result<()> {
103 +
    backend.delete(key).map_err(backend_err)
104 +
}
105 +
106 +
/// `loose/<hh>/<rest>`.
107 +
fn loose_key(r: &ObjectRef) -> String {
108 +
    let h = r.to_hex();
109 +
    format!("loose/{}/{}", &h[..2], &h[2..])
110 +
}
111 +
112 +
/// The object a loose key names, if it is one.
113 +
fn loose_ref(key: &str) -> Option<ObjectRef> {
114 +
    let rest = key.strip_prefix("loose/")?;
115 +
    let (hh, tail) = rest.split_once('/')?;
116 +
    if hh.len() != 2 || tail.contains('/') {
117 +
        return None;
118 +
    }
119 +
    ObjectRef::from_hex(&format!("{hh}{tail}"))
120 +
}
121 +
122 +
/// `<index>/<hh>/<rest>/` for the entries kept under an object.
123 +
fn fanout_prefix(index: &str, r: &ObjectRef) -> String {
124 +
    let h = r.to_hex();
125 +
    format!("{index}/{}/{}/", &h[..2], &h[2..])
126 +
}
127 +
128 +
/// `<index>/<hh>/<rest>`: the one-file-per-object layout of older stores.
129 +
fn legacy_key(index: &str, r: &ObjectRef) -> String {
130 +
    let h = r.to_hex();
131 +
    format!("{index}/{}/{}", &h[..2], &h[2..])
52 132
}
53 133
134 +
fn now_ms() -> u64 {
135 +
    std::time::SystemTime::now()
136 +
        .duration_since(std::time::UNIX_EPOCH)
137 +
        .map(|d| d.as_millis() as u64)
138 +
        .unwrap_or(0)
139 +
}
140 +
54 141
#[derive(Clone, Copy, Debug)]
55 142
struct PackEntry {
56 143
    offset: u64,
…
5 unchanged lines
62 149
63 150
/// One pack's index, fully in memory and sorted by ObjectRef.
64 151
struct PackIndex {
65 -
    path: PathBuf,
152 +
    /// The pack's key, `pack-<id>.pack`.
153 +
    key: String,
66 154
    refs: Vec<ObjectRef>,
67 155
    entries: Vec<PackEntry>,
68 156
    /// The pack's dictionary, prepared once, when it has one.
69 -
    dict: Option<std::sync::Arc<zstd::dict::DecoderDictionary<'static>>>,
157 +
    dict: Option<Arc<zstd::dict::DecoderDictionary<'static>>>,
70 158
}
71 159
72 160
impl std::fmt::Debug for PackIndex {
73 161
    fn fmt(&self, f: &mut std::fmt::Formatter<'_>) -> std::fmt::Result {
74 162
        f.debug_struct("PackIndex")
75 -
            .field("path", &self.path)
163 +
            .field("key", &self.key)
76 164
            .field("objects", &self.refs.len())
77 165
            .field("dict", &self.dict.is_some())
78 166
            .finish()
167 +
    }
168 +
}
169 +
170 +
/// Decode a stored pack entry and check it against its reference.
171 +
fn decode_entry(
172 +
    r: &ObjectRef,
173 +
    e: PackEntry,
174 +
    stored: Vec<u8>,
175 +
    dict: Option<&Arc<zstd::dict::DecoderDictionary<'static>>>,
176 +
) -> Result<Object> {
177 +
    let body = if e.flags & FLAG_DICT != 0 {
178 +
        let dict = dict.ok_or_else(|| {
179 +
            StoreError::Corrupt(*r, "packed with a dictionary the pack lacks".into())
180 +
        })?;
181 +
        zstd::bulk::Decompressor::with_prepared_dictionary(dict)
182 +
            .and_then(|mut d| d.decompress(&stored, e.raw_len as usize))
183 +
            .map_err(|err| StoreError::Corrupt(*r, format!("zstd: {err}")))?
184 +
    } else if e.flags & FLAG_ZSTD != 0 {
185 +
        zstd::bulk::decompress(&stored, e.raw_len as usize)
186 +
            .map_err(|err| StoreError::Corrupt(*r, format!("zstd: {err}")))?
187 +
    } else {
188 +
        stored
189 +
    };
190 +
    if body.len() != e.raw_len as usize || hash_object(e.ty, &body) != *r {
191 +
        return Err(StoreError::Corrupt(
192 +
            *r,
193 +
            "packed object does not match its reference".into(),
194 +
        ));
79 195
    }
196 +
    Ok((e.ty, body))
80 197
}
81 198
82 199
impl PackIndex {
83 -
    fn load(idx_path: &Path, pack_path: &Path) -> Result<PackIndex> {
84 -
        let data = fs::read(idx_path)?;
85 -
        let bad =
86 -
            |m: &str| StoreError::Corrupt(ObjectRef::ZERO, format!("{}: {m}", idx_path.display()));
200 +
    fn load(backend: &Backend, idx_key: &str, pack_key: &str) -> Result<PackIndex> {
201 +
        let data = backend.get(idx_key).map_err(backend_err)?;
202 +
        let bad = |m: &str| StoreError::Corrupt(ObjectRef::ZERO, format!("{idx_key}: {m}"));
87 203
        if data.len() < 12 || &data[..8] != IDX_MAGIC {
88 204
            return Err(bad("bad index header"));
89 205
        }
…
1 unchanged lines
91 207
        if data.len() != 12 + count * IDX_ENTRY {
92 208
            return Err(bad("index length mismatch"));
93 209
        }
94 -
        let pack_len = fs::metadata(pack_path)?.len();
210 +
        let pack_len = backend
211 +
            .stat(pack_key)
212 +
            .map_err(backend_err)?
213 +
            .ok_or_else(|| bad("the index names a pack that is not there"))?
214 +
            .size;
95 215
        let mut refs = Vec::with_capacity(count);
96 216
        let mut entries = Vec::with_capacity(count);
97 217
        for i in 0..count {
…
19 unchanged lines
117 237
                return Err(StoreError::Corrupt(
118 238
                    r,
119 239
                    format!(
120 -
                        "{}: the index claims an object of {} bytes ({} stored), more than the {MAX_OBJECT_LEN} an object may be",
121 -
                        idx_path.display(),
122 -
                        entry.raw_len,
123 -
                        entry.stored_len
240 +
                        "{idx_key}: the index claims an object of {} bytes ({} stored), more than the {MAX_OBJECT_LEN} an object may be",
241 +
                        entry.raw_len, entry.stored_len
124 242
                    ),
125 243
                ));
126 244
            }
…
2 unchanged lines
129 247
            }
130 248
            entries.push(entry);
131 249
        }
132 -
        let dict = match fs::read(pack_path.with_extension("dict")) {
133 -
            Ok(d) => Some(std::sync::Arc::new(zstd::dict::DecoderDictionary::copy(&d))),
134 -
            Err(e) if e.kind() == io::ErrorKind::NotFound => None,
135 -
            Err(e) => return Err(e.into()),
136 -
        };
250 +
        let dict_key = format!("{}.dict", pack_key.trim_end_matches(".pack"));
251 +
        let dict = get_optional(backend, &dict_key)?
252 +
            .map(|d| Arc::new(zstd::dict::DecoderDictionary::copy(&d)));
137 253
        Ok(PackIndex {
138 -
            path: pack_path.to_path_buf(),
254 +
            key: pack_key.to_string(),
139 255
            refs,
140 256
            entries,
141 257
            dict,
…
4 unchanged lines
146 262
        self.refs.binary_search(r).ok().map(|i| self.entries[i])
147 263
    }
148 264
149 -
    fn read(&self, r: &ObjectRef, e: PackEntry) -> Result<(ObjectType, Vec<u8>)> {
150 -
        let mut f = fs::File::open(&self.path)?;
151 -
        f.seek(SeekFrom::Start(e.offset))?;
152 -
        let mut stored = vec![0u8; e.stored_len as usize];
153 -
        f.read_exact(&mut stored)?;
154 -
        let body = if e.flags & FLAG_DICT != 0 {
155 -
            let dict = self.dict.as_ref().ok_or_else(|| {
156 -
                StoreError::Corrupt(*r, "packed with a dictionary the pack lacks".into())
265 +
    /// Read one entry from the pack: one ranged read, then decode.
266 +
    fn fetch(&self, backend: &Backend, r: &ObjectRef, e: PackEntry) -> Result<Object> {
267 +
        let stored = backend
268 +
            .get_range(&self.key, e.offset..e.offset + e.stored_len as u64)
269 +
            .map_err(|err| match err {
270 +
                BackendError::NotFound(_) => StoreError::NotFound(*r),
271 +
                other => backend_err(other),
157 272
            })?;
158 -
            zstd::bulk::Decompressor::with_prepared_dictionary(dict)
159 -
                .and_then(|mut d| d.decompress(&stored, e.raw_len as usize))
160 -
                .map_err(|err| StoreError::Corrupt(*r, format!("zstd: {err}")))?
161 -
        } else if e.flags & FLAG_ZSTD != 0 {
162 -
            zstd::bulk::decompress(&stored, e.raw_len as usize)
163 -
                .map_err(|err| StoreError::Corrupt(*r, format!("zstd: {err}")))?
164 -
        } else {
165 -
            stored
166 -
        };
167 -
        if body.len() != e.raw_len as usize || hash_object(e.ty, &body) != *r {
168 -
            return Err(StoreError::Corrupt(
169 -
                *r,
170 -
                "packed object does not match its reference".into(),
171 -
            ));
172 -
        }
173 -
        Ok((e.ty, body))
273 +
        decode_entry(r, e, stored, self.dict.as_ref())
174 274
    }
175 275
}
176 276
…
43 unchanged lines
220 320
221 321
pub type Result<T> = std::result::Result<T, StoreError>;
222 322
223 -
#[derive(Debug)]
224 323
pub struct Store {
225 -
    dir: PathBuf,
324 +
    backend: Arc<Backend>,
325 +
    /// Temporary files (a pack being streamed) and derived caches (`metas`).
326 +
    /// For a store in a directory it is that directory.
327 +
    scratch: PathBuf,
328 +
    /// The backend is that directory, so the old file paths still hold.
329 +
    local: bool,
226 330
    packs: RefCell<Vec<PackIndex>>,
227 331
    /// An open write batch (spec §13.4): objects go here instead of to loose
228 332
    /// files, and are written out with one `fsync` when the batch finishes.
…
3 unchanged lines
232 336
    metas: RefCell<Option<crate::fasthash::FastMap<ObjectRef, crate::state::ChangeMeta>>>,
233 337
}
234 338
339 +
impl std::fmt::Debug for Store {
340 +
    fn fmt(&self, f: &mut std::fmt::Formatter<'_>) -> std::fmt::Result {
341 +
        f.debug_struct("Store")
342 +
            .field("backend", &self.backend.describe())
343 +
            .field("scratch", &self.scratch)
344 +
            .field("packs", &self.packs.borrow().len())
345 +
            .finish()
346 +
    }
347 +
}
348 +
235 349
/// Objects written since `begin_batch`. Small batches stay in memory and
236 350
/// become loose objects; once `SPILL_OBJECTS` or `SPILL_BYTES` is exceeded
237 351
/// the batch streams into a pack file instead.
…
75 unchanged lines
313 427
    report.raw_bytes += raw_len as u64;
314 428
    report.packed_bytes += stored.len() as u64;
315 429
    Ok(entry)
430 +
}
431 +
432 +
/// A random name for a temporary file or a pack.
433 +
fn fresh_id() -> String {
434 +
    let mut nonce = [0u8; 8];
435 +
    rand::rngs::OsRng.fill_bytes(&mut nonce);
436 +
    hex::encode(nonce)
437 +
}
438 +
439 +
/// Under `OXY_STORE_BACKEND=memory` every store opens on a store in
440 +
/// memory shared by directory, so the whole test suite can run without a
441 +
/// directory store, as it would over a bucket. Objects live for the
442 +
/// process.
443 +
fn memory_backend_for(dir: &Path) -> Option<Arc<Backend>> {
444 +
    use std::collections::HashMap;
445 +
    use std::sync::{Mutex, OnceLock};
446 +
    if std::env::var_os("OXY_STORE_BACKEND").as_deref() != Some(std::ffi::OsStr::new("memory")) {
447 +
        return None;
448 +
    }
449 +
    static STORES: OnceLock<Mutex<HashMap<PathBuf, Arc<MemoryStore>>>> = OnceLock::new();
450 +
    let key = dir.canonicalize().unwrap_or_else(|_| dir.to_path_buf());
451 +
    let mut map = STORES
452 +
        .get_or_init(Default::default)
453 +
        .lock()
454 +
        .unwrap_or_else(|p| p.into_inner());
455 +
    let store = map
456 +
        .entry(key)
457 +
        .or_insert_with(|| Arc::new(MemoryStore::new()))
458 +
        .clone();
459 +
    Some(store)
316 460
}
317 461
318 462
impl Store {
463 +
    /// Open the store in `dir`, which is also its scratch directory.
319 464
    pub fn open(dir: impl Into<PathBuf>) -> io::Result<Store> {
320 465
        let dir = dir.into();
321 466
        fs::create_dir_all(dir.join("loose"))?;
467 +
        if let Some(backend) = memory_backend_for(&dir) {
468 +
            return Store::open_on(backend, dir);
469 +
        }
470 +
        let backend: Arc<Backend> =
471 +
            Arc::new(LocalStore::open(&dir).map_err(|e| io::Error::other(e.to_string()))?);
472 +
        let mut store = Store::open_on(backend, dir)?;
473 +
        store.local = true;
474 +
        Ok(store)
475 +
    }
476 +
477 +
    /// Open the store kept on `backend`, with `scratch` for what must be
478 +
    /// on this disk: a pack while it is being written, and caches.
479 +
    pub fn open_on(backend: Arc<Backend>, scratch: impl Into<PathBuf>) -> io::Result<Store> {
480 +
        let scratch = scratch.into();
481 +
        fs::create_dir_all(&scratch)?;
322 482
        let store = Store {
323 -
            dir,
483 +
            backend,
484 +
            scratch,
485 +
            local: false,
324 486
            packs: RefCell::new(Vec::new()),
325 487
            pending: RefCell::new(None),
326 488
            batch_depth: std::cell::Cell::new(0),
…
3 unchanged lines
330 492
            .reload_packs()
331 493
            .map_err(|e| io::Error::other(e.to_string()))?;
332 494
        Ok(store)
495 +
    }
496 +
497 +
    /// The storage under this store.
498 +
    pub fn backend(&self) -> &Arc<Backend> {
499 +
        &self.backend
333 500
    }
334 501
502 +
    /// Whether the store is the directory `dir()` names, where every key
503 +
    /// is a file.
504 +
    pub fn is_local(&self) -> bool {
505 +
        self.local
506 +
    }
507 +
335 508
    fn reload_packs(&self) -> Result<()> {
336 -
        let mut packs = Vec::new();
337 -
        let mut names: Vec<PathBuf> = fs::read_dir(&self.dir)?
338 -
            .filter_map(|e| e.ok().map(|e| e.path()))
339 -
            .filter(|p| {
340 -
                p.extension().is_some_and(|x| x == "idx")
341 -
                    && p.file_name()
342 -
                        .is_some_and(|n| n.to_string_lossy().starts_with("pack-"))
343 -
            })
509 +
        let listed = self.backend.list("pack-").map_err(backend_err)?;
510 +
        let mut idx_keys: Vec<&str> = listed
511 +
            .iter()
512 +
            .map(|e| e.key.as_str())
513 +
            .filter(|k| k.ends_with(".idx"))
344 514
            .collect();
345 -
        names.sort();
346 -
        for idx in names {
347 -
            let pack = idx.with_extension("pack");
348 -
            if pack.is_file() {
349 -
                packs.push(PackIndex::load(&idx, &pack)?);
515 +
        idx_keys.sort();
516 +
        let mut packs = Vec::new();
517 +
        for idx in idx_keys {
518 +
            let pack = format!("{}.pack", idx.trim_end_matches(".idx"));
519 +
            if listed.iter().any(|e| e.key == pack) {
520 +
                packs.push(PackIndex::load(self.backend.as_ref(), idx, &pack)?);
350 521
            }
351 522
        }
352 523
        *self.packs.borrow_mut() = packs;
…
12 unchanged lines
365 536
            .find_map(|(i, p)| p.find(r).map(|e| (i, e)))
366 537
    }
367 538
368 -
    /// Loose objects on disk, with their paths.
369 -
    fn loose_objects(&self) -> io::Result<Vec<(ObjectRef, PathBuf)>> {
370 -
        let mut out = Vec::new();
371 -
        let loose = self.dir.join("loose");
372 -
        for d in fs::read_dir(&loose)? {
373 -
            let d = d?;
374 -
            if !d.file_type()?.is_dir() {
375 -
                continue;
376 -
            }
377 -
            let prefix = d.file_name().to_string_lossy().to_string();
378 -
            for f in fs::read_dir(d.path())? {
379 -
                let f = f?;
380 -
                let name = f.file_name().to_string_lossy().to_string();
381 -
                if let Some(r) = ObjectRef::from_hex(&format!("{prefix}{name}")) {
382 -
                    out.push((r, f.path()));
383 -
                }
384 -
            }
385 -
        }
386 -
        out.sort();
539 +
    /// Loose objects, with their keys and ages.
540 +
    fn loose_objects(&self) -> Result<Vec<(ObjectRef, strata::Entry)>> {
541 +
        let mut out: Vec<(ObjectRef, strata::Entry)> = self
542 +
            .backend
543 +
            .list("loose/")
544 +
            .map_err(backend_err)?
545 +
            .into_iter()
546 +
            .filter_map(|e| loose_ref(&e.key).map(|r| (r, e)))
547 +
            .collect();
548 +
        out.sort_by_key(|(r, _)| *r);
387 549
        Ok(out)
388 550
    }
389 551
390 552
    pub fn loose_count(&self) -> io::Result<usize> {
391 -
        Ok(self.loose_objects()?.len())
553 +
        self.loose_objects()
554 +
            .map(|l| l.len())
555 +
            .map_err(|e| io::Error::other(e.to_string()))
392 556
    }
393 557
394 -
    /// Move every loose object into a new pack (spec §13.4). Loose files are
395 -
    /// deleted only after the pack and its index are durable.
558 +
    /// Move every loose object into a new pack (spec §13.4). Loose objects
559 +
    /// are deleted only after the pack and its index are durable.
396 560
    pub fn pack_loose(&self) -> Result<PackReport> {
397 561
        let loose = self.loose_objects()?;
398 562
        self.write_pack(
399 563
            &loose.iter().map(|(r, _)| *r).collect::<Vec<_>>(),
400 -
            &loose.iter().map(|(_, p)| p.clone()).collect::<Vec<_>>(),
564 +
            &loose.iter().map(|(_, e)| e.key.clone()).collect::<Vec<_>>(),
401 565
            &[],
402 566
            ZSTD_LEVEL,
403 567
            &mut silent(),
…
10 unchanged lines
414 578
    /// [`Store::repack_all`], reporting progress.
415 579
    pub fn repack_all_with(&self, level: i32, progress: Progress) -> Result<(PackReport, usize)> {
416 580
        let loose = self.loose_objects()?;
417 -
        let (mut refs, paths): (Vec<ObjectRef>, Vec<PathBuf>) = {
581 +
        let (mut refs, keys): (Vec<ObjectRef>, Vec<String>) = {
418 582
            let packs = self.packs.borrow();
419 583
            (
420 584
                packs.iter().flat_map(|p| p.refs.iter().copied()).collect(),
421 -
                packs.iter().map(|p| p.path.clone()).collect(),
585 +
                packs.iter().map(|p| p.key.clone()).collect(),
422 586
            )
423 587
        };
424 -
        if loose.is_empty() && paths.len() <= 1 && level == ZSTD_LEVEL {
588 +
        if loose.is_empty() && keys.len() <= 1 && level == ZSTD_LEVEL {
425 589
            return Ok((PackReport::default(), 0));
426 590
        }
427 591
        refs.extend(loose.iter().map(|(r, _)| *r));
428 -
        let loose_paths: Vec<PathBuf> = loose.into_iter().map(|(_, p)| p).collect();
429 -
        let n = paths.len();
430 -
        let report = self.write_pack(&refs, &loose_paths, &paths, level, progress)?;
592 +
        let loose_keys: Vec<String> = loose.into_iter().map(|(_, e)| e.key).collect();
593 +
        let n = keys.len();
594 +
        let report = self.write_pack(&refs, &loose_keys, &keys, level, progress)?;
431 595
        Ok((report, n))
432 596
    }
433 597
434 598
    /// Fold every pack into one when there are more than `MAX_PACKS`
435 599
    /// (spec §13.4). Returns how many packs were merged.
436 600
    pub fn merge_packs(&self) -> Result<usize> {
437 -
        let (refs, paths): (Vec<ObjectRef>, Vec<PathBuf>) = {
601 +
        let (refs, keys): (Vec<ObjectRef>, Vec<String>) = {
438 602
            let packs = self.packs.borrow();
439 603
            if packs.len() <= MAX_PACKS {
440 604
                return Ok(0);
441 605
            }
442 606
            (
443 607
                packs.iter().flat_map(|p| p.refs.iter().copied()).collect(),
444 -
                packs.iter().map(|p| p.path.clone()).collect(),
608 +
                packs.iter().map(|p| p.key.clone()).collect(),
445 609
            )
446 610
        };
447 -
        let n = paths.len();
448 -
        self.write_pack(&refs, &[], &paths, ZSTD_LEVEL, &mut silent())?;
611 +
        let n = keys.len();
612 +
        self.write_pack(&refs, &[], &keys, ZSTD_LEVEL, &mut silent())?;
449 613
        Ok(n)
450 614
    }
451 615
452 616
    /// Delete every object not in `keep` (spec §21). Loose objects and packs
453 617
    /// younger than `grace` are left alone, since a concurrent writer may not
454 618
    /// have recorded its references yet. Packs holding unreachable objects
455 -
    /// are rewritten without them. Signature sidecars and seal-index entries
456 -
    /// of deleted objects go too.
619 +
    /// are rewritten without them. Signatures, seal and receipt entries of
620 +
    /// deleted objects go too.
457 621
    pub fn prune(
458 622
        &self,
459 623
        keep: &std::collections::HashSet<ObjectRef>,
…
10 unchanged lines
470 634
        progress: Progress,
471 635
    ) -> Result<PruneReport> {
472 636
        progress("pruning loose objects");
473 -
        let now = std::time::SystemTime::now();
474 -
        let old_enough = |m: &fs::Metadata| {
475 -
            m.modified()
476 -
                .map(|t| now.duration_since(t).unwrap_or_default() >= grace)
477 -
                .unwrap_or(true)
478 -
        };
637 +
        let now = now_ms();
638 +
        let grace_ms = grace.as_millis() as u64;
639 +
        // An object whose age the store does not know is old enough.
640 +
        let old_enough =
641 +
            |modified: Option<u64>| modified.is_none_or(|t| now.saturating_sub(t) >= grace_ms);
479 642
        let mut report = PruneReport::default();
480 643
        // Loose objects.
481 -
        for (r, path) in self.loose_objects()? {
644 +
        for (r, entry) in self.loose_objects()? {
482 645
            if keep.contains(&r) {
483 646
                continue;
484 647
            }
485 -
            let meta = fs::metadata(&path)?;
486 -
            if !old_enough(&meta) {
648 +
            if !old_enough(entry.modified_ms) {
487 649
                report.deferred += 1;
488 650
                continue;
489 651
            }
490 -
            report.bytes += meta.len();
652 +
            report.bytes += entry.size;
491 653
            report.objects += 1;
492 -
            // The fan-out directory stays: recreating it costs a sync of
493 -
            // `loose` on the next write.
494 -
            fs::remove_file(&path)?;
654 +
            delete(self.backend.as_ref(), &entry.key)?;
495 655
            self.remove_sidecars(&r)?;
496 656
        }
497 657
        // Packs: rewrite those that hold unreachable objects.
498 658
        let mut keep_refs: Vec<ObjectRef> = Vec::new();
499 -
        let mut old_paths: Vec<PathBuf> = Vec::new();
659 +
        let mut old_keys: Vec<String> = Vec::new();
500 660
        let mut rewrite = false;
501 661
        progress("pruning packs");
502 662
        for p in self.packs.borrow().iter() {
503 -
            let meta = fs::metadata(&p.path)?;
504 663
            let unreachable = p.refs.iter().filter(|r| !keep.contains(r)).count();
505 664
            if unreachable == 0 {
506 665
                continue;
507 666
            }
508 -
            if !old_enough(&meta) {
667 +
            let modified = self
668 +
                .backend
669 +
                .stat(&p.key)
670 +
                .map_err(backend_err)?
671 +
                .and_then(|e| e.modified_ms);
672 +
            if !old_enough(modified) {
509 673
                report.deferred += unreachable;
510 674
                continue;
511 675
            }
512 676
            rewrite = true;
513 -
            old_paths.push(p.path.clone());
677 +
            old_keys.push(p.key.clone());
514 678
            for (r, e) in p.refs.iter().zip(p.entries.iter()) {
515 679
                if keep.contains(r) {
516 680
                    keep_refs.push(*r);
…
8 unchanged lines
525 689
                let packs = self.packs.borrow();
526 690
                packs
527 691
                    .iter()
528 -
                    .filter(|p| old_paths.contains(&p.path))
692 +
                    .filter(|p| old_keys.contains(&p.key))
529 693
                    .flat_map(|p| p.refs.iter().copied())
530 694
                    .filter(|r| !keep.contains(r))
531 695
                    .collect()
532 696
            };
533 -
            self.write_pack(&keep_refs, &[], &old_paths, ZSTD_LEVEL, progress)?;
697 +
            self.write_pack(&keep_refs, &[], &old_keys, ZSTD_LEVEL, progress)?;
534 698
            for r in dropped {
535 699
                self.remove_sidecars(&r)?;
536 700
            }
…
1 unchanged lines
538 702
        Ok(report)
539 703
    }
540 704
705 +
    /// Everything kept beside an object: its signatures and the seal and
706 +
    /// receipt entries under it, in both layouts.
541 707
    fn remove_sidecars(&self, r: &ObjectRef) -> Result<()> {
542 -
        let sidecars = [
543 -
            self.sig_path(r),
544 -
            self.index_path("seals", r),
545 -
            self.index_path("receipts", r),
546 -
        ];
547 -
        for path in sidecars {
548 -
            match fs::remove_file(&path) {
549 -
                Ok(()) => {
550 -
                    let _ = fs::remove_dir(path.parent().unwrap());
551 -
                }
552 -
                Err(e) if e.kind() == io::ErrorKind::NotFound => {}
553 -
                Err(e) => return Err(e.into()),
708 +
        for index in ["sigs", "seals", "receipts"] {
709 +
            delete(self.backend.as_ref(), &legacy_key(index, r))?;
710 +
        }
711 +
        for index in ["sig", "seal", "receipt"] {
712 +
            for e in self
713 +
                .backend
714 +
                .list(&fanout_prefix(index, r))
715 +
                .map_err(backend_err)?
716 +
            {
717 +
                delete(self.backend.as_ref(), &e.key)?;
554 718
            }
555 719
        }
556 720
        Ok(())
557 721
    }
558 722
559 -
    /// Write `refs` into a new pack, then delete `loose_paths` and `old_packs`.
560 -
    /// Write the index for a complete, synced temporary pack file and move
561 -
    /// both into place; `index` must be sorted by ObjectRef.
723 +
    /// Put a complete, synced temporary pack file and its index into the
724 +
    /// store; `index` must be sorted by ObjectRef. The pack goes first: an
725 +
    /// index never names a pack that is not there.
562 726
    fn finish_pack_files(
563 727
        &self,
564 728
        id: &str,
565 729
        tmp_pack: &Path,
566 730
        index: &[(ObjectRef, PackEntry)],
567 731
    ) -> Result<()> {
568 -
        let pack_path = self.dir.join(format!("pack-{id}.pack"));
569 -
        let idx_path = self.dir.join(format!("pack-{id}.idx"));
570 -
        let tmp_idx = self.dir.join(format!("tmp-{id}.idx"));
571 -
        {
572 -
            let mut f = io::BufWriter::new(fs::File::create(&tmp_idx)?);
573 -
            f.write_all(IDX_MAGIC)?;
574 -
            f.write_all(&(index.len() as u32).to_le_bytes())?;
575 -
            for (r, e) in index {
576 -
                f.write_all(&r.0)?;
577 -
                f.write_all(&e.offset.to_le_bytes())?;
578 -
                f.write_all(&e.stored_len.to_le_bytes())?;
579 -
                f.write_all(&e.raw_len.to_le_bytes())?;
580 -
                f.write_all(&[type_index(e.ty), e.flags])?;
581 -
            }
582 -
            f.flush()?;
583 -
            f.get_ref().sync_all()?;
732 +
        let mut idx = Vec::with_capacity(12 + index.len() * IDX_ENTRY);
733 +
        idx.extend_from_slice(IDX_MAGIC);
734 +
        idx.extend_from_slice(&(index.len() as u32).to_le_bytes());
735 +
        for (r, e) in index {
736 +
            idx.extend_from_slice(&r.0);
737 +
            idx.extend_from_slice(&e.offset.to_le_bytes());
738 +
            idx.extend_from_slice(&e.stored_len.to_le_bytes());
739 +
            idx.extend_from_slice(&e.raw_len.to_le_bytes());
740 +
            idx.extend_from_slice(&[type_index(e.ty), e.flags]);
584 741
        }
585 -
        // The pack first: an index never names a pack that is not there.
586 -
        crate::durable::rename(tmp_pack, &pack_path)?;
587 -
        crate::durable::rename(&tmp_idx, &idx_path)?;
742 +
        let put = self
743 +
            .backend
744 +
            .put_new_file(&format!("pack-{id}.pack"), tmp_pack)
745 +
            .map_err(backend_err);
746 +
        // A store on another disk copied it; the one in this directory
747 +
        // moved it and this finds nothing.
748 +
        let _ = fs::remove_file(tmp_pack);
749 +
        put?;
750 +
        self.backend
751 +
            .put_new(&format!("pack-{id}.idx"), &idx)
752 +
            .map_err(backend_err)?;
588 753
        self.reload_packs()
754 +
    }
755 +
756 +
    /// Remove a pack with its index and dictionary.
757 +
    fn remove_pack(&self, pack_key: &str) -> Result<()> {
758 +
        let stem = pack_key.trim_end_matches(".pack");
759 +
        delete(self.backend.as_ref(), pack_key)?;
760 +
        delete(self.backend.as_ref(), &format!("{stem}.idx"))?;
761 +
        delete(self.backend.as_ref(), &format!("{stem}.dict"))
589 762
    }
590 763
764 +
    /// Write `refs` into a new pack, then delete `loose_keys` and `old_packs`.
591 765
    fn write_pack(
592 766
        &self,
593 767
        refs: &[ObjectRef],
594 -
        loose_paths: &[PathBuf],
595 -
        old_packs: &[PathBuf],
768 +
        loose_keys: &[String],
769 +
        old_packs: &[String],
596 770
        level: i32,
597 771
        progress: Progress,
598 772
    ) -> Result<PackReport> {
599 773
        if refs.is_empty() {
600 774
            for p in old_packs {
601 -
                remove_pack_files(p)?;
775 +
                self.remove_pack(p)?;
602 776
            }
603 777
            self.reload_packs()?;
604 778
            return Ok(PackReport::default());
…
1 unchanged lines
606 780
        let mut refs: Vec<ObjectRef> = refs.to_vec();
607 781
        refs.sort();
608 782
        refs.dedup();
609 -
        let mut nonce = [0u8; 8];
610 -
        rand::rngs::OsRng.fill_bytes(&mut nonce);
611 -
        let id = hex::encode(nonce);
612 -
        let tmp_pack = self.dir.join(format!("tmp-{id}.pack"));
783 +
        let id = fresh_id();
784 +
        let tmp_pack = self.scratch.join(format!(".tmp-{id}.pack"));
613 785
        let mut report = PackReport::default();
614 786
        let mut index: Vec<(ObjectRef, PackEntry)> = Vec::with_capacity(refs.len());
615 787
        // A full repack of enough objects trains a dictionary for them.
…
7 unchanged lines
623 795
        let encoder = dict
624 796
            .as_ref()
625 797
            .map(|d| zstd::dict::EncoderDictionary::copy(d, level));
798 +
        let nocompress = self.nocompress_set()?;
626 799
        {
627 800
            let mut f = io::BufWriter::new(fs::File::create(&tmp_pack)?);
628 801
            f.write_all(PACK_MAGIC)?;
…
17 unchanged lines
646 819
                        Vec::with_capacity(batch.len());
647 820
                    for r in batch {
648 821
                        let (ty, body) = self.get(r)?;
649 -
                        read.push((*r, ty, body, self.is_nocompress(r)));
822 +
                        read.push((*r, ty, body, nocompress.contains(r)));
650 823
                    }
651 824
                    use rayon::prelude::*;
652 825
                    compressed_batch = read
…
63 unchanged lines
716 889
        }
717 890
        // The dictionary before the pack that needs it becomes visible.
718 891
        if let Some(d) = &dict {
719 -
            crate::durable::write_atomic(&self.dir.join(format!("pack-{id}.dict")), d)?;
892 +
            self.backend
893 +
                .put_new(&format!("pack-{id}.dict"), d)
894 +
                .map_err(backend_err)?;
720 895
        }
721 896
        progress(&format!("writing the index of {} objects", index.len()));
722 897
        self.finish_pack_files(&id, &tmp_pack, &index)?;
723 898
        for p in old_packs {
724 -
            remove_pack_files(p)?;
899 +
            self.remove_pack(p)?;
725 900
        }
726 901
        self.reload_packs()?;
727 -
        // Fan-out directories stay, emptied: see `ensure_fanouts`.
728 -
        for path in loose_paths {
729 -
            fs::remove_file(path)?;
902 +
        for key in loose_keys {
903 +
            delete(self.backend.as_ref(), key)?;
730 904
        }
731 905
        Ok(report)
732 906
    }
…
24 unchanged lines
757 931
        zstd::dict::from_samples(&samples, DICT_BYTES).ok()
758 932
    }
759 933
934 +
    /// The scratch directory: for a store in a directory, that directory.
760 935
    pub fn dir(&self) -> &Path {
761 -
        &self.dir
936 +
        &self.scratch
762 937
    }
763 938
939 +
    /// Where a loose object is kept, for a store in a directory (tests
940 +
    /// tamper with and remove objects there).
941 +
    #[cfg(test)]
764 942
    pub(crate) fn path_for(&self, r: &ObjectRef) -> PathBuf {
765 -
        let h = r.to_hex();
766 -
        self.dir.join("loose").join(&h[..2]).join(&h[2..])
943 +
        self.scratch.join(loose_key(r))
944 +
    }
945 +
946 +
    /// Remove a loose object, whatever the store is on (tests make an
947 +
    /// object go missing).
948 +
    #[cfg(test)]
949 +
    pub(crate) fn remove_loose_for_test(&self, r: &ObjectRef) -> Result<()> {
950 +
        delete(self.backend.as_ref(), &loose_key(r))
767 951
    }
768 952
769 953
    pub fn has(&self, r: &ObjectRef) -> bool {
770 -
        self.path_for(r).is_file()
954 +
        self.backend.has(&loose_key(r)).unwrap_or(false)
771 955
            || self.find_packed(r).is_some()
772 956
            || self
773 957
                .pending
…
79 unchanged lines
853 1037
    }
854 1038
855 1039
    /// Whether a batch is open and holds `r`.
856 -
    fn pending_get(&self, r: &ObjectRef) -> Option<Result<(ObjectType, Vec<u8>)>> {
1040 +
    fn pending_get(&self, r: &ObjectRef) -> Option<Result<Object>> {
857 1041
        let mut guard = self.pending.borrow_mut();
858 1042
        let pending = guard.as_mut()?;
859 1043
        let entry = *pending.lookup.get(r)?;
…
6 unchanged lines
866 1050
                let Some(pack) = pending.pack.as_mut() else {
867 1051
                    return Some(Err(StoreError::NotFound(*r)));
868 1052
                };
869 -
                match pack.writer.flush() {
870 -
                    Ok(()) => PackIndex {
871 -
                        path: pack.tmp_path.clone(),
872 -
                        refs: vec![],
873 -
                        entries: vec![],
874 -
                        dict: None,
875 -
                    }
876 -
                    .read(r, e),
877 -
                    Err(err) => Err(err.into()),
878 -
                }
1053 +
                // Read from the pack file being written, on this disk.
1054 +
                (|| {
1055 +
                    pack.writer.flush()?;
1056 +
                    use std::io::{Read, Seek, SeekFrom};
1057 +
                    let mut f = fs::File::open(&pack.tmp_path)?;
1058 +
                    f.seek(SeekFrom::Start(e.offset))?;
1059 +
                    let mut stored = vec![0u8; e.stored_len as usize];
1060 +
                    f.read_exact(&mut stored)?;
1061 +
                    decode_entry(r, e, stored, None)
1062 +
                })()
879 1063
            }
880 1064
        })
881 1065
    }
…
52 unchanged lines
934 1118
    fn ensure_pack(&self, pending: &mut Pending) -> Result<()> {
935 1119
        if pending.pack.is_none() {
936 1120
            // Spill: open the pack and move the buffered objects into it.
937 -
            let mut nonce = [0u8; 8];
938 -
            rand::rngs::OsRng.fill_bytes(&mut nonce);
939 -
            let id = hex::encode(nonce);
940 -
            let tmp_path = self.dir.join(format!("tmp-{id}.pack"));
1121 +
            let id = fresh_id();
1122 +
            let tmp_path = self.scratch.join(format!(".tmp-{id}.pack"));
941 1123
            let mut writer = io::BufWriter::new(fs::File::create(&tmp_path)?);
942 1124
            writer.write_all(PACK_MAGIC)?;
943 1125
            let mut pack = PendingPack {
…
50 unchanged lines
994 1176
                .is_some_and(|p| p.lookup.contains_key(r));
995 1177
            if seen.insert(*r)
996 1178
                && !known
997 -
                && !self.path_for(r).is_file()
998 1179
                && self.find_packed(r).is_none()
1180 +
                && !self.backend.has(&loose_key(r)).unwrap_or(false)
999 1181
            {
1000 1182
                todo.push(i);
1001 1183
            }
1002 1184
        }
1003 -
        let raw: std::collections::HashSet<ObjectRef> =
1004 -
            match fs::read_dir(self.dir.join("nocompress")) {
1005 -
                Ok(rd) => rd
1006 -
                    .flatten()
1007 -
                    .filter_map(|e| ObjectRef::from_hex(&e.file_name().to_string_lossy()))
1008 -
                    .collect(),
1009 -
                Err(_) => std::collections::HashSet::new(),
1010 -
            };
1185 +
        let raw = self.nocompress_set()?;
1011 1186
        let packed: Vec<(u8, Vec<u8>)> = todo
1012 1187
            .par_iter()
1013 1188
            .map(|i| {
…
34 unchanged lines
1048 1223
            });
1049 1224
        }
1050 1225
        let r = hash_object(ty, body);
1051 -
        let path = self.path_for(&r);
1052 -
        if path.is_file() || self.find_packed(&r).is_some() {
1226 +
        if self.find_packed(&r).is_some() || self.backend.has(&loose_key(&r)).unwrap_or(false) {
1053 1227
            return Ok(r);
1054 1228
        }
1055 1229
        if self.pending.borrow().is_some() {
…
4 unchanged lines
1060 1234
        Ok(r)
1061 1235
    }
1062 1236
1063 -
    /// Write one loose object atomically with its own `fsync`.
1237 +
    /// Write one loose object, durable when this returns.
1064 1238
    fn put_loose(&self, ty: ObjectType, body: &[u8]) -> Result<ObjectRef> {
1065 1239
        let r = hash_object(ty, body);
1066 -
        let path = self.path_for(&r);
1067 -
        if path.is_file() {
1068 -
            return Ok(r);
1069 -
        }
1070 -
        if !path.parent().unwrap().is_dir() {
1071 -
            self.ensure_fanouts()?;
1072 -
        }
1073 -
        let mut nonce = [0u8; 8];
1074 -
        rand::rngs::OsRng.fill_bytes(&mut nonce);
1075 -
        let tmp = self
1076 -
            .dir
1077 -
            .join("loose")
1078 -
            .join(format!("tmp-{}", hex::encode(nonce)));
1240 +
        let mut bytes = header(ty, body.len());
1241 +
        bytes.extend_from_slice(body);
1079 1242
        crate::durable::step("write loose object")?;
1080 -
        {
1081 -
            let mut f = fs::File::create(&tmp)?;
1082 -
            f.write_all(&header(ty, body.len()))?;
1083 -
            f.write_all(body)?;
1084 -
            f.sync_all()?;
1085 -
        }
1086 -
        crate::durable::rename(&tmp, &path)?;
1243 +
        put_new_or_existing(self.backend.as_ref(), &loose_key(&r), &bytes)?;
1087 1244
        Ok(r)
1088 -
    }
1089 -
1090 -
    /// Create every fan-out directory of `loose` (`00` to `ff`) and sync
1091 -
    /// `loose` once. Syncing it is slow on a busy disk (130 ms seen), and
1092 -
    /// creating the directories one at a time paid that on most writes
1093 -
    /// after a gc had emptied them; they are kept from then on.
1094 -
    fn ensure_fanouts(&self) -> io::Result<()> {
1095 -
        let loose = self.dir.join("loose");
1096 -
        for b in 0..=255u8 {
1097 -
            match fs::create_dir(loose.join(format!("{b:02x}"))) {
1098 -
                Ok(()) => {}
1099 -
                Err(e) if e.kind() == io::ErrorKind::AlreadyExists => {}
1100 -
                Err(e) => return Err(e),
1101 -
            }
1102 -
        }
1103 -
        crate::durable::sync_dir(&loose)
1104 1245
    }
1105 1246
1106 1247
    /// Write several loose objects with as few waits on the disk as the
1107 -
    /// guarantees allow: every file written, then all synced together, then
1108 -
    /// each renamed into place and each directory synced once. Durable when
1109 -
    /// this returns, as `put_loose` one at a time would be (spec §13.6).
1248 +
    /// guarantees allow (a directory store writes them all, then syncs
1249 +
    /// each directory once). Durable when this returns, as `put_loose`
1250 +
    /// one at a time would be (spec §13.6).
1110 1251
    fn put_loose_many(&self, objects: &[(ObjectType, &[u8])]) -> Result<()> {
1111 -
        use rayon::prelude::*;
1112 1252
        if objects.len() <= 1 {
1113 1253
            for (ty, body) in objects {
1114 1254
                self.put_loose(*ty, body)?;
…
1 unchanged lines
1116 1256
            return Ok(());
1117 1257
        }
1118 1258
        let mut seen = std::collections::HashSet::new();
1119 -
        let todo: Vec<(PathBuf, PathBuf, ObjectType, &[u8])> = objects
1259 +
        let items: Vec<(String, Vec<u8>)> = objects
1120 1260
            .iter()
1121 1261
            .filter_map(|(ty, body)| {
1122 1262
                let r = hash_object(*ty, body);
1123 -
                let path = self.path_for(&r);
1124 -
                (seen.insert(r) && !path.is_file()).then(|| {
1125 -
                    let mut nonce = [0u8; 8];
1126 -
                    rand::rngs::OsRng.fill_bytes(&mut nonce);
1127 -
                    let tmp = self
1128 -
                        .dir
1129 -
                        .join("loose")
1130 -
                        .join(format!("tmp-{}", hex::encode(nonce)));
1131 -
                    (path, tmp, *ty, *body)
1263 +
                seen.insert(r).then(|| {
1264 +
                    let mut bytes = header(*ty, body.len());
1265 +
                    bytes.extend_from_slice(body);
1266 +
                    (loose_key(&r), bytes)
1132 1267
                })
1133 1268
            })
1134 1269
            .collect();
1135 -
        if todo.is_empty() {
1136 -
            return Ok(());
1137 -
        }
1138 -
        let fanouts: std::collections::BTreeSet<&Path> =
1139 -
            todo.iter().map(|(p, ..)| p.parent().unwrap()).collect();
1140 -
        if fanouts.iter().any(|f| !f.is_dir()) {
1141 -
            self.ensure_fanouts()?;
1142 -
        }
1143 -
        crate::durable::step("write loose objects")?;
1144 -
        todo.par_iter()
1145 -
            .try_for_each(|(_, tmp, ty, body)| -> io::Result<()> {
1146 -
                let mut f = fs::File::create(tmp)?;
1147 -
                f.write_all(&header(*ty, body.len()))?;
1148 -
                f.write_all(body)?;
1149 -
                f.sync_all()
1150 -
            })?;
1151 -
        crate::durable::step("rename loose objects")?;
1152 -
        for (path, tmp, ..) in &todo {
1153 -
            fs::rename(tmp, path)?;
1154 -
        }
1155 -
        fanouts
1156 -
            .par_iter()
1157 -
            .try_for_each(|d| crate::durable::sync_dir(d))?;
1158 -
        Ok(())
1270 +
        self.backend.put_many_new(&items).map_err(backend_err)
1159 1271
    }
1160 1272
1161 1273
    /// Read and verify an object, loose, packed, or pending in a batch.
1162 -
    pub fn get(&self, r: &ObjectRef) -> Result<(ObjectType, Vec<u8>)> {
1163 -
        match read_loose_at(&self.path_for(r), r) {
1164 -
            Some(found) => found,
1274 +
    pub fn get(&self, r: &ObjectRef) -> Result<Object> {
1275 +
        match get_optional(self.backend.as_ref(), &loose_key(r))? {
1276 +
            Some(bytes) => parse_loose(&bytes, r),
1165 1277
            None => match self.find_packed(r) {
1166 -
                Some((i, e)) => self.packs.borrow()[i].read(r, e),
1278 +
                Some((i, e)) => self.packs.borrow()[i].fetch(self.backend.as_ref(), r, e),
1167 1279
                None => self.pending_get(r).unwrap_or(Err(StoreError::NotFound(*r))),
1168 1280
            },
1169 1281
        }
1170 1282
    }
1171 1283
1172 1284
    /// Many objects at once: loose ones are read and verified in parallel,
1173 -
    /// the rest (packed, or pending in a batch) one by one. Results are in
1174 -
    /// the order of `refs`.
1285 +
    /// then packed ones in parallel, then what is pending in a batch.
1286 +
    /// Results are in the order of `refs`.
1175 1287
    pub fn get_many(&self, refs: &[ObjectRef]) -> Vec<Result<Object>> {
1176 1288
        use rayon::prelude::*;
1177 -
        let paths: Vec<(ObjectRef, PathBuf)> =
1178 -
            refs.iter().map(|r| (*r, self.path_for(r))).collect();
1179 -
        let mut found: Vec<Option<Result<Object>>> =
1180 -
            paths.par_iter().map(|(r, p)| read_loose_at(p, r)).collect();
1289 +
        let backend = self.backend.as_ref();
1290 +
        let mut found: Vec<Option<Result<Object>>> = refs
1291 +
            .par_iter()
1292 +
            .map(|r| match get_optional(backend, &loose_key(r)) {
1293 +
                Ok(Some(bytes)) => Some(parse_loose(&bytes, r)),
1294 +
                Ok(None) => None,
1295 +
                Err(e) => Some(Err(e)),
1296 +
            })
1297 +
            .collect();
1181 1298
        // Packed ones: look them up in order, then read them in parallel.
1182 1299
        let packs = self.packs.borrow();
1183 1300
        let packed: Vec<(usize, usize, PackEntry)> = found
…
10 unchanged lines
1194 1311
        let indexes: &[PackIndex] = &packs;
1195 1312
        let read: Vec<(usize, Result<Object>)> = packed
1196 1313
            .par_iter()
1197 -
            .map(|(i, p, e)| (*i, indexes[*p].read(&refs[*i], *e)))
1314 +
            .map(|(i, p, e)| (*i, indexes[*p].fetch(backend, &refs[*i], *e)))
1198 1315
            .collect();
1199 1316
        drop(packs);
1200 1317
        for (i, r) in read {
…
123 unchanged lines
1324 1441
    /// The metadata cache: records of a change hash, its metadata in the
1325 1442
    /// flat layout, and the first 8 bytes of BLAKE3 over both, each
1326 1443
    /// appended in one write. Derived only: a record that does not check
1327 -
    /// ends the reading (a torn write), and the cache can be deleted.
1444 +
    /// ends the reading (a torn write), and the cache can be deleted. It
1445 +
    /// lives in the scratch directory, on this disk.
1328 1446
    fn read_metas_cache(&self) -> crate::fasthash::FastMap<ObjectRef, crate::state::ChangeMeta> {
1329 1447
        let mut out = crate::fasthash::FastMap::default();
1330 -
        let Ok(bytes) = fs::read(self.dir.join("metas")) else {
1448 +
        let Ok(bytes) = fs::read(self.scratch.join("metas")) else {
1331 1449
            return out;
1332 1450
        };
1333 1451
        let mut r = crate::flat::Reader::new(&bytes);
…
23 unchanged lines
1357 1475
        if good < bytes.len() {
1358 1476
            let _ = fs::OpenOptions::new()
1359 1477
                .write(true)
1360 -
                .open(self.dir.join("metas"))
1478 +
                .open(self.scratch.join("metas"))
1361 1479
                .and_then(|f| f.set_len(good as u64));
1362 1480
        }
1363 1481
        out
…
13 unchanged lines
1377 1495
        let _ = fs::OpenOptions::new()
1378 1496
            .create(true)
1379 1497
            .append(true)
1380 -
            .open(self.dir.join("metas"))
1498 +
            .open(self.scratch.join("metas"))
1381 1499
            .and_then(|mut f| f.write_all(&w.0));
1382 1500
        if let Some(m) = self.metas.borrow_mut().as_mut() {
1383 1501
            m.insert(*r, meta.clone());
…
16 unchanged lines
1400 1518
    /// Store a chunk that packs will keep uncompressed (`nocompress`, spec §13.3).
1401 1519
    pub fn put_chunk_nocompress(&self, bytes: &[u8]) -> Result<ObjectRef> {
1402 1520
        let r = hash_object(ObjectType::Chunk, bytes);
1403 -
        let dir = self.dir.join("nocompress");
1404 -
        fs::create_dir_all(&dir)?;
1405 -
        let _ = fs::File::create(dir.join(r.to_hex()));
1521 +
        put_new_or_existing(
1522 +
            self.backend.as_ref(),
1523 +
            &format!("nocompress/{}", r.to_hex()),
1524 +
            b"",
1525 +
        )?;
1406 1526
        self.put(ObjectType::Chunk, bytes)
1407 1527
    }
1408 1528
1409 1529
    fn is_nocompress(&self, r: &ObjectRef) -> bool {
1410 -
        self.dir.join("nocompress").join(r.to_hex()).exists()
1530 +
        self.backend
1531 +
            .has(&format!("nocompress/{}", r.to_hex()))
1532 +
            .unwrap_or(false)
1533 +
    }
1534 +
1535 +
    /// Every chunk marked to be kept raw.
1536 +
    fn nocompress_set(&self) -> Result<std::collections::HashSet<ObjectRef>> {
1537 +
        Ok(self
1538 +
            .backend
1539 +
            .list("nocompress/")
1540 +
            .map_err(backend_err)?
1541 +
            .into_iter()
1542 +
            .filter_map(|e| ObjectRef::from_hex(e.key.trim_start_matches("nocompress/")))
1543 +
            .collect())
1411 1544
    }
1412 1545
1413 1546
    pub fn put_state(&self, st: &StateObject) -> Result<ObjectRef> {
…
20 unchanged lines
1434 1567
        Ok(Receipt::decode(&self.get_typed(r, ObjectType::Receipt)?)?)
1435 1568
    }
1436 1569
1437 -
    fn index_path(&self, index: &str, key: &ObjectRef) -> PathBuf {
1438 -
        let h = key.to_hex();
1439 -
        self.dir.join(index).join(&h[..2]).join(&h[2..])
1570 +
    /// Record `value` under `key` in `index`: one write-once entry, so two
1571 +
    /// writers of a shared store never lose each other's.
1572 +
    fn index_add(&self, index: &str, key: &ObjectRef, value: &ObjectRef) -> Result<()> {
1573 +
        let entry = format!("{}{}", fanout_prefix(index, key), value.to_hex());
1574 +
        put_new_or_existing(self.backend.as_ref(), &entry, b"")
1440 1575
    }
1441 1576
1442 -
    /// Add `value` to the sorted list kept for `key` in `index`.
1443 -
    fn index_add(&self, index: &str, key: &ObjectRef, value: &ObjectRef) -> Result<()> {
1444 -
        let mut values = self.index_get(index, key)?;
1445 -
        if values.contains(value) {
1446 -
            return Ok(());
1577 +
    /// The values recorded under `key` in `index`, with those an older
1578 +
    /// store kept as one list in `legacy`, sorted.
1579 +
    fn index_get(&self, index: &str, legacy: &str, key: &ObjectRef) -> Result<Vec<ObjectRef>> {
1580 +
        let mut values: Vec<ObjectRef> = self
1581 +
            .backend
1582 +
            .list(&fanout_prefix(index, key))
1583 +
            .map_err(backend_err)?
1584 +
            .into_iter()
1585 +
            .filter_map(|e| e.key.rsplit('/').next().and_then(ObjectRef::from_hex))
1586 +
            .collect();
1587 +
        if let Some(bytes) = get_optional(self.backend.as_ref(), &legacy_key(legacy, key))? {
1588 +
            values.extend(
1589 +
                String::from_utf8_lossy(&bytes)
1590 +
                    .lines()
1591 +
                    .filter_map(ObjectRef::from_hex),
1592 +
            );
1447 1593
        }
1448 -
        values.push(*value);
1449 1594
        values.sort();
1450 -
        let path = self.index_path(index, key);
1451 -
        fs::create_dir_all(path.parent().unwrap())?;
1452 -
        let text: String = values.iter().map(|s| format!("{}\n", s.to_hex())).collect();
1453 -
        crate::durable::write_atomic(&path, text.as_bytes())?;
1454 -
        Ok(())
1595 +
        values.dedup();
1596 +
        Ok(values)
1455 1597
    }
1456 1598
1457 -
    fn index_get(&self, index: &str, key: &ObjectRef) -> Result<Vec<ObjectRef>> {
1458 -
        match fs::read_to_string(self.index_path(index, key)) {
1459 -
            Ok(t) => Ok(t.lines().filter_map(ObjectRef::from_hex).collect()),
1460 -
            Err(e) if e.kind() == io::ErrorKind::NotFound => Ok(vec![]),
1461 -
            Err(e) => Err(e.into()),
1462 -
        }
1463 -
    }
1464 -
1465 1599
    /// Record that seal attestation `attest` covers `change`.
1466 1600
    pub fn index_seal(&self, change: &ObjectRef, attest: &ObjectRef) -> Result<()> {
1467 -
        self.index_add("seals", change, attest)
1601 +
        self.index_add("seal", change, attest)
1468 1602
    }
1469 1603
1470 1604
    /// Seal attestations known to cover `change`.
1471 1605
    pub fn seals_for(&self, change: &ObjectRef) -> Result<Vec<ObjectRef>> {
1472 -
        self.index_get("seals", change)
1606 +
        self.index_get("seal", "seals", change)
1473 1607
    }
1474 1608
1475 1609
    /// Record receipt attestation `attest` for `state` (spec §14.6).
1476 1610
    pub fn index_receipt(&self, state: &ObjectRef, attest: &ObjectRef) -> Result<()> {
1477 -
        self.index_add("receipts", state, attest)
1611 +
        self.index_add("receipt", state, attest)
1478 1612
    }
1479 1613
1480 1614
    /// Receipt attestations recorded for `state`.
1481 1615
    pub fn receipts_for(&self, state: &ObjectRef) -> Result<Vec<ObjectRef>> {
1482 -
        self.index_get("receipts", state)
1616 +
        self.index_get("receipt", "receipts", state)
1483 1617
    }
1484 1618
1485 1619
    /// Every state with a receipt, for garbage collection.
1486 1620
    pub fn receipt_states(&self) -> Result<Vec<ObjectRef>> {
1487 -
        let mut out = Vec::new();
1488 -
        let dir = self.dir.join("receipts");
1489 -
        let top = match fs::read_dir(&dir) {
1490 -
            Ok(d) => d,
1491 -
            Err(e) if e.kind() == io::ErrorKind::NotFound => return Ok(out),
1492 -
            Err(e) => return Err(e.into()),
1493 -
        };
1494 -
        for d in top {
1495 -
            let d = d?;
1496 -
            let prefix = d.file_name().to_string_lossy().into_owned();
1497 -
            for f in fs::read_dir(d.path())? {
1498 -
                let name = f?.file_name().to_string_lossy().into_owned();
1499 -
                if let Some(r) = ObjectRef::from_hex(&format!("{prefix}{name}")) {
1500 -
                    out.push(r);
1501 -
                }
1502 -
            }
1621 +
        let mut out: Vec<ObjectRef> = Vec::new();
1622 +
        for e in self.backend.list("receipt/").map_err(backend_err)? {
1623 +
            // receipt/<hh>/<rest>/<attest>
1624 +
            let mut parts = e.key.splitn(4, '/');
1625 +
            let (Some(_), Some(hh), Some(rest), Some(_)) =
1626 +
                (parts.next(), parts.next(), parts.next(), parts.next())
1627 +
            else {
1628 +
                continue;
1629 +
            };
1630 +
            out.extend(ObjectRef::from_hex(&format!("{hh}{rest}")));
1503 1631
        }
1632 +
        for e in self.backend.list("receipts/").map_err(backend_err)? {
1633 +
            let mut parts = e.key.splitn(3, '/');
1634 +
            let (Some(_), Some(hh), Some(rest)) = (parts.next(), parts.next(), parts.next()) else {
1635 +
                continue;
1636 +
            };
1637 +
            out.extend(ObjectRef::from_hex(&format!("{hh}{rest}")));
1638 +
        }
1504 1639
        out.sort();
1640 +
        out.dedup();
1505 1641
        Ok(out)
1506 1642
    }
1507 1643
…
5 unchanged lines
1513 1649
        Ok(Identity::decode(&self.get_typed(r, ObjectType::Identity)?)?)
1514 1650
    }
1515 1651
1516 -
    fn sig_path(&self, r: &ObjectRef) -> PathBuf {
1517 -
        let h = r.to_hex();
1518 -
        self.dir.join("sigs").join(&h[..2]).join(&h[2..])
1519 -
    }
1520 -
1521 -
    /// Signatures stored alongside object `r` (spec §5.4). Empty if none.
1652 +
    /// Signatures stored alongside object `r` (spec §5.4), one per signing
1653 +
    /// key, sorted by key. Empty if none.
1522 1654
    pub fn signatures(&self, r: &ObjectRef) -> Result<Vec<Signature>> {
1523 -
        match fs::read(self.sig_path(r)) {
1524 -
            Ok(b) => Ok(Signature::decode_list(&b)?),
1525 -
            Err(e) if e.kind() == io::ErrorKind::NotFound => Ok(vec![]),
1526 -
            Err(e) => Err(e.into()),
1655 +
        let mut sigs: Vec<Signature> = Vec::new();
1656 +
        for e in self
1657 +
            .backend
1658 +
            .list(&fanout_prefix("sig", r))
1659 +
            .map_err(backend_err)?
1660 +
        {
1661 +
            let bytes = self.backend.get(&e.key).map_err(backend_err)?;
1662 +
            sigs.extend(Signature::decode_list(&bytes)?);
1527 1663
        }
1664 +
        // An older store's one list per object; a signer kept there and
1665 +
        // here counts once, as written here.
1666 +
        if let Some(bytes) = get_optional(self.backend.as_ref(), &legacy_key("sigs", r))? {
1667 +
            for sig in Signature::decode_list(&bytes)? {
1668 +
                if !sigs.iter().any(|s| s.key == sig.key) {
1669 +
                    sigs.push(sig);
1670 +
                }
1671 +
            }
1672 +
        }
1673 +
        sigs.sort_by_key(|s| s.key);
1674 +
        sigs.dedup_by_key(|s| s.key);
1675 +
        Ok(sigs)
1528 1676
    }
1529 1677
1530 -
    /// Attach a signature to object `r`, replacing any earlier one by the same key.
1678 +
    /// Attach a signature to object `r`, replacing any earlier one by the
1679 +
    /// same key: one entry per signer, so the latest by a key is the one
1680 +
    /// kept, and two signers never touch each other's.
1531 1681
    pub fn put_signature(&self, r: &ObjectRef, sig: &Signature) -> Result<()> {
1532 -
        let mut sigs = self.signatures(r)?;
1533 -
        sigs.retain(|s| s.key != sig.key);
1534 -
        sigs.push(sig.clone());
1535 -
        sigs.sort_by_key(|s| s.key);
1536 -
        let path = self.sig_path(r);
1537 -
        fs::create_dir_all(path.parent().unwrap())?;
1538 -
        let mut nonce = [0u8; 8];
1539 -
        rand::rngs::OsRng.fill_bytes(&mut nonce);
1540 -
        let tmp = self.dir.join(format!("sigs-tmp-{}", hex::encode(nonce)));
1541 -
        {
1542 -
            let mut f = fs::File::create(&tmp)?;
1543 -
            f.write_all(&Signature::encode_list(&sigs))?;
1544 -
            f.sync_all()?;
1682 +
        let key = format!("{}{}", fanout_prefix("sig", r), sig.key.to_hex());
1683 +
        let bytes = Signature::encode_list(std::slice::from_ref(sig));
1684 +
        if get_optional(self.backend.as_ref(), &key)?.as_deref() == Some(bytes.as_slice()) {
1685 +
            return Ok(());
1545 1686
        }
1546 -
        crate::durable::rename(&tmp, &path)?;
1547 -
        Ok(())
1687 +
        delete(self.backend.as_ref(), &key)?;
1688 +
        put_new_or_existing(self.backend.as_ref(), &key, &bytes)
1548 1689
    }
1549 1690
1550 1691
    /// Every object reference in the store, loose and packed (slow; for gc and tests).
1551 1692
    pub fn list(&self) -> io::Result<Vec<ObjectRef>> {
1552 -
        let mut out: Vec<ObjectRef> = self.loose_objects()?.into_iter().map(|(r, _)| r).collect();
1693 +
        let mut out: Vec<ObjectRef> = self
1694 +
            .loose_objects()
1695 +
            .map_err(|e| io::Error::other(e.to_string()))?
1696 +
            .into_iter()
1697 +
            .map(|(r, _)| r)
1698 +
            .collect();
1553 1699
        for p in self.packs.borrow().iter() {
1554 1700
            out.extend(p.refs.iter().copied());
1555 1701
        }
…
3 unchanged lines
1559 1705
    }
1560 1706
}
1561 1707
1562 -
/// A loose object at `path`, verified against `r`; `None` when there is no
1563 -
/// such file (it may be packed).
1564 -
fn read_loose_at(path: &Path, r: &ObjectRef) -> Option<Result<Object>> {
1565 -
    let data = match fs::read(path) {
1566 -
        Ok(d) => d,
1567 -
        Err(e) if e.kind() == io::ErrorKind::NotFound => return None,
1568 -
        Err(e) => return Some(Err(e.into())),
1708 +
/// A loose object's bytes, verified against `r`.
1709 +
fn parse_loose(data: &[u8], r: &ObjectRef) -> Result<Object> {
1710 +
    let nul = data
1711 +
        .iter()
1712 +
        .position(|&b| b == 0)
1713 +
        .ok_or_else(|| StoreError::Corrupt(*r, "no header".into()))?;
1714 +
    let head = std::str::from_utf8(&data[..nul])
1715 +
        .map_err(|_| StoreError::Corrupt(*r, "header not UTF-8".into()))?;
1716 +
    let mut parts = head.split(' ');
1717 +
    let (Some("oxy1"), Some(ty), Some(len), None) =
1718 +
        (parts.next(), parts.next(), parts.next(), parts.next())
1719 +
    else {
1720 +
        return Err(StoreError::Corrupt(*r, format!("bad header {head:?}")));
1569 1721
    };
1570 -
    Some((|| {
1571 -
        let nul = data
1572 -
            .iter()
1573 -
            .position(|&b| b == 0)
1574 -
            .ok_or_else(|| StoreError::Corrupt(*r, "no header".into()))?;
1575 -
        let head = std::str::from_utf8(&data[..nul])
1576 -
            .map_err(|_| StoreError::Corrupt(*r, "header not UTF-8".into()))?;
1577 -
        let mut parts = head.split(' ');
1578 -
        let (Some("oxy1"), Some(ty), Some(len), None) =
1579 -
            (parts.next(), parts.next(), parts.next(), parts.next())
1580 -
        else {
1581 -
            return Err(StoreError::Corrupt(*r, format!("bad header {head:?}")));
1582 -
        };
1583 -
        let ty = ObjectType::from_name(ty)
1584 -
            .ok_or_else(|| StoreError::Corrupt(*r, format!("unknown type {ty}")))?;
1585 -
        let len: usize = len
1586 -
            .parse()
1587 -
            .map_err(|_| StoreError::Corrupt(*r, "bad length".into()))?;
1588 -
        let body = &data[nul + 1..];
1589 -
        if body.len() != len {
1590 -
            return Err(StoreError::Corrupt(
1591 -
                *r,
1592 -
                format!("length {} != header {len}", body.len()),
1593 -
            ));
1594 -
        }
1595 -
        if hash_object(ty, body) != *r {
1596 -
            return Err(StoreError::Corrupt(*r, "hash mismatch".into()));
1597 -
        }
1598 -
        Ok((ty, body.to_vec()))
1599 -
    })())
1722 +
    let ty = ObjectType::from_name(ty)
1723 +
        .ok_or_else(|| StoreError::Corrupt(*r, format!("unknown type {ty}")))?;
1724 +
    let len: usize = len
1725 +
        .parse()
1726 +
        .map_err(|_| StoreError::Corrupt(*r, "bad length".into()))?;
1727 +
    let body = &data[nul + 1..];
1728 +
    if body.len() != len {
1729 +
        return Err(StoreError::Corrupt(
1730 +
            *r,
1731 +
            format!("length {} != header {len}", body.len()),
1732 +
        ));
1733 +
    }
1734 +
    if hash_object(ty, body) != *r {
1735 +
        return Err(StoreError::Corrupt(*r, "hash mismatch".into()));
1736 +
    }
1737 +
    Ok((ty, body.to_vec()))
1738 +
}
1739 +
1740 +
/// Whether the objects are files that a test may tamper with: not under
1741 +
/// `OXY_STORE_BACKEND=memory`.
1742 +
#[cfg(test)]
1743 +
fn files_on_disk() -> bool {
1744 +
    std::env::var_os("OXY_STORE_BACKEND").is_none()
1600 1745
}
1601 1746
1602 1747
#[cfg(test)]
1603 1748
mod tests {
1604 1749
    use super::*;
1750 +
1751 +
    /// The store under test twice: in a directory, and on a store in
1752 +
    /// memory with a scratch directory, as over a bucket.
1753 +
    fn each_store(dir: &Path, f: impl Fn(&str, Store)) {
1754 +
        f("directory", Store::open(dir.join("objects")).unwrap());
1755 +
        f(
1756 +
            "memory",
1757 +
            Store::open_on(Arc::new(MemoryStore::new()), dir.join("scratch")).unwrap(),
1758 +
        );
1759 +
    }
1605 1760
1606 1761
    #[test]
1607 1762
    fn put_get_verify() {
1763 +
        if !super::files_on_disk() {
1764 +
            return;
1765 +
        }
1608 1766
        let dir = tempfile::tempdir().unwrap();
1609 1767
        let s = Store::open(dir.path().join("objects")).unwrap();
1610 1768
        let r = s.put(ObjectType::Chunk, b"hello").unwrap();
…
21 unchanged lines
1632 1790
1633 1791
    #[test]
1634 1792
    fn packing_keeps_every_object_readable() {
1793 +
        if !super::files_on_disk() {
1794 +
            return;
1795 +
        }
1635 1796
        let dir = tempfile::tempdir().unwrap();
1636 1797
        let s = Store::open(dir.path()).unwrap();
1637 1798
        let mut refs = Vec::new();
…
42 unchanged lines
1680 1841
        let (pack_path, offset) = {
1681 1842
            let packs = s2.packs.borrow();
1682 1843
            let (i, e) = s2.find_packed(&target).unwrap();
1683 -
            (packs[i].path.clone(), e.offset as usize)
1844 +
            (dir.path().join(&packs[i].key), e.offset as usize)
1684 1845
        };
1685 1846
        let mut data = fs::read(&pack_path).unwrap();
1686 1847
        data[offset] ^= 0xff;
…
9 unchanged lines
1696 1857
1697 1858
    #[test]
1698 1859
    fn a_full_repack_trains_a_dictionary_and_reads_back() {
1860 +
        if !super::files_on_disk() {
1861 +
            return;
1862 +
        }
1699 1863
        let dir = tempfile::tempdir().unwrap();
1700 1864
        let s = Store::open(dir.path()).unwrap();
1701 1865
        // Many small, alike files: what a dictionary is for.
…
52 unchanged lines
1754 1918
        let (_, c) = s.find_packed(&compressible).unwrap();
1755 1919
        assert_eq!(c.flags, FLAG_ZSTD);
1756 1920
        assert!(c.stored_len < 10_000);
1757 -
        assert_eq!(packs[i].read(&raw, e).unwrap().1, body);
1921 +
        assert_eq!(packs[i].fetch(s.backend.as_ref(), &raw, e).unwrap().1, body);
1758 1922
    }
1759 1923
1760 1924
    #[test]
1761 1925
    fn many_packs_merge_into_one() {
1762 1926
        let dir = tempfile::tempdir().unwrap();
1763 -
        let s = Store::open(dir.path()).unwrap();
1764 -
        let mut refs = Vec::new();
1765 -
        for i in 0..(MAX_PACKS + 2) {
1766 -
            refs.push(
1767 -
                s.put(ObjectType::Chunk, format!("pack {i}").as_bytes())
1768 -
                    .unwrap(),
1769 -
            );
1770 -
            s.pack_loose().unwrap();
1771 -
        }
1772 -
        assert_eq!(s.pack_count(), MAX_PACKS + 2);
1773 -
        assert_eq!(s.merge_packs().unwrap(), MAX_PACKS + 2);
1774 -
        assert_eq!(s.pack_count(), 1);
1775 -
        assert_eq!(s.merge_packs().unwrap(), 0);
1776 -
        for (i, r) in refs.iter().enumerate() {
1777 -
            assert_eq!(s.get(r).unwrap().1, format!("pack {i}").as_bytes());
1778 -
        }
1779 -
        assert_eq!(Store::open(dir.path()).unwrap().pack_count(), 1);
1927 +
        each_store(dir.path(), |which, s| {
1928 +
            let mut refs = Vec::new();
1929 +
            for i in 0..(MAX_PACKS + 2) {
1930 +
                refs.push(
1931 +
                    s.put(ObjectType::Chunk, format!("pack {i}").as_bytes())
1932 +
                        .unwrap(),
1933 +
                );
1934 +
                s.pack_loose().unwrap();
1935 +
            }
1936 +
            assert_eq!(s.pack_count(), MAX_PACKS + 2, "{which}");
1937 +
            assert_eq!(s.merge_packs().unwrap(), MAX_PACKS + 2);
1938 +
            assert_eq!(s.pack_count(), 1);
1939 +
            assert_eq!(s.merge_packs().unwrap(), 0);
1940 +
            for (i, r) in refs.iter().enumerate() {
1941 +
                assert_eq!(s.get(r).unwrap().1, format!("pack {i}").as_bytes());
1942 +
            }
1943 +
            let again = Store::open_on(s.backend().clone(), s.dir()).unwrap();
1944 +
            assert_eq!(again.pack_count(), 1, "{which}: reopened");
1945 +
        });
1780 1946
    }
1781 1947
1782 1948
    #[test]
1783 1949
    fn prune_removes_unreachable_loose_and_packed_objects() {
1784 1950
        use std::collections::HashSet;
1785 1951
        let dir = tempfile::tempdir().unwrap();
1786 -
        let s = Store::open(dir.path()).unwrap();
1787 -
        let keep_a = s.put(ObjectType::Chunk, b"keep a").unwrap();
1788 -
        let drop_a = s.put(ObjectType::Chunk, b"drop a").unwrap();
1789 -
        s.pack_loose().unwrap();
1790 -
        let keep_b = s.put(ObjectType::Chunk, b"keep b").unwrap();
1791 -
        let drop_b = s.put(ObjectType::Chunk, b"drop b").unwrap();
1792 -
        let sk = crate::identity::SecretKey {
1793 -
            ed_seed: [1; 32],
1794 -
            ml_seed: [2; 32],
1795 -
        };
1796 -
        let id = sk.identity("a", "b", 0);
1797 -
        let key = s.put_identity(&id).unwrap();
1798 -
        s.put_signature(
1799 -
            &drop_b,
1800 -
            &sk.sign(crate::identity::Domain::Change, &key, &drop_b),
1801 -
        )
1802 -
        .unwrap();
1803 -
        s.index_seal(&drop_a, &ObjectRef([3; 32])).unwrap();
1804 -
        let keep: HashSet<ObjectRef> = [keep_a, keep_b, key].into_iter().collect();
1805 -
        // Everything is too young under a grace period.
1806 -
        let r = s
1807 -
            .prune(&keep, std::time::Duration::from_secs(3600))
1952 +
        each_store(dir.path(), |which, s| {
1953 +
            let keep_a = s.put(ObjectType::Chunk, b"keep a").unwrap();
1954 +
            let drop_a = s.put(ObjectType::Chunk, b"drop a").unwrap();
1955 +
            s.pack_loose().unwrap();
1956 +
            let keep_b = s.put(ObjectType::Chunk, b"keep b").unwrap();
1957 +
            let drop_b = s.put(ObjectType::Chunk, b"drop b").unwrap();
1958 +
            let sk = crate::identity::SecretKey {
1959 +
                ed_seed: [1; 32],
1960 +
                ml_seed: [2; 32],
1961 +
            };
1962 +
            let id = sk.identity("a", "b", 0);
1963 +
            let key = s.put_identity(&id).unwrap();
1964 +
            s.put_signature(
1965 +
                &drop_b,
1966 +
                &sk.sign(crate::identity::Domain::Change, &key, &drop_b),
1967 +
            )
1808 1968
            .unwrap();
1809 -
        assert_eq!((r.objects, r.deferred), (0, 2));
1810 -
        let r = s.prune(&keep, std::time::Duration::ZERO).unwrap();
1811 -
        assert_eq!(r.objects, 2, "{r:?}");
1812 -
        assert!(s.has(&keep_a) && s.has(&keep_b) && s.has(&key));
1813 -
        assert!(!s.has(&drop_a) && !s.has(&drop_b));
1814 -
        assert!(s.signatures(&drop_b).unwrap().is_empty());
1815 -
        assert!(s.seals_for(&drop_a).unwrap().is_empty());
1816 -
        assert_eq!(s.pack_count(), 1);
1817 -
        assert_eq!(s.get(&keep_a).unwrap().1, b"keep a");
1818 -
        assert_eq!(s.list().unwrap().len(), 3);
1969 +
            s.index_seal(&drop_a, &ObjectRef([3; 32])).unwrap();
1970 +
            let keep: HashSet<ObjectRef> = [keep_a, keep_b, key].into_iter().collect();
1971 +
            // Everything is too young under a grace period.
1972 +
            let r = s
1973 +
                .prune(&keep, std::time::Duration::from_secs(3600))
1974 +
                .unwrap();
1975 +
            assert_eq!((r.objects, r.deferred), (0, 2), "{which}");
1976 +
            let r = s.prune(&keep, std::time::Duration::ZERO).unwrap();
1977 +
            assert_eq!(r.objects, 2, "{which}: {r:?}");
1978 +
            assert!(s.has(&keep_a) && s.has(&keep_b) && s.has(&key));
1979 +
            assert!(!s.has(&drop_a) && !s.has(&drop_b));
1980 +
            assert!(s.signatures(&drop_b).unwrap().is_empty());
1981 +
            assert!(s.seals_for(&drop_a).unwrap().is_empty());
1982 +
            assert_eq!(s.pack_count(), 1);
1983 +
            assert_eq!(s.get(&keep_a).unwrap().1, b"keep a");
1984 +
            assert_eq!(s.list().unwrap().len(), 3);
1985 +
        });
1819 1986
    }
1820 1987
1821 1988
    #[test]
1822 1989
    fn signatures_are_stored_beside_objects() {
1990 +
        use crate::identity::{Domain, SecretKey};
1991 +
        let dir = tempfile::tempdir().unwrap();
1992 +
        each_store(dir.path(), |which, s| {
1993 +
            let r = s.put(ObjectType::Chunk, b"x").unwrap();
1994 +
            assert!(s.signatures(&r).unwrap().is_empty());
1995 +
            let sk = SecretKey {
1996 +
                ed_seed: [1; 32],
1997 +
                ml_seed: [2; 32],
1998 +
            };
1999 +
            let id = sk.identity("a", "b", 0);
2000 +
            let key = s.put_identity(&id).unwrap();
2001 +
            assert_eq!(s.get_identity(&key).unwrap(), id);
2002 +
            let sig = sk.sign(Domain::Change, &key, &r);
2003 +
            s.put_signature(&r, &sig).unwrap();
2004 +
            s.put_signature(&r, &sig).unwrap();
2005 +
            assert_eq!(s.signatures(&r).unwrap(), vec![sig.clone()], "{which}");
2006 +
            // A second signer is kept beside the first, sorted by key.
2007 +
            let sk2 = SecretKey {
2008 +
                ed_seed: [3; 32],
2009 +
                ml_seed: [4; 32],
2010 +
            };
2011 +
            let key2 = s.put_identity(&sk2.identity("c", "d", 0)).unwrap();
2012 +
            let sig2 = sk2.sign(Domain::Change, &key2, &r);
2013 +
            s.put_signature(&r, &sig2).unwrap();
2014 +
            let mut both = vec![sig.clone(), sig2.clone()];
2015 +
            both.sort_by_key(|s| s.key);
2016 +
            assert_eq!(s.signatures(&r).unwrap(), both);
2017 +
            // The object itself is untouched by its signatures.
2018 +
            assert_eq!(s.get(&r).unwrap().1, b"x");
2019 +
            // Seal and receipt entries, and the states with receipts.
2020 +
            s.index_seal(&r, &ObjectRef([5; 32])).unwrap();
2021 +
            s.index_seal(&r, &ObjectRef([4; 32])).unwrap();
2022 +
            s.index_seal(&r, &ObjectRef([5; 32])).unwrap();
2023 +
            assert_eq!(
2024 +
                s.seals_for(&r).unwrap(),
2025 +
                vec![ObjectRef([4; 32]), ObjectRef([5; 32])]
2026 +
            );
2027 +
            let st = ObjectRef([6; 32]);
2028 +
            s.index_receipt(&st, &ObjectRef([7; 32])).unwrap();
2029 +
            assert_eq!(s.receipts_for(&st).unwrap(), vec![ObjectRef([7; 32])]);
2030 +
            assert_eq!(s.receipt_states().unwrap(), vec![st]);
2031 +
        });
2032 +
    }
2033 +
2034 +
    /// A store written before signatures, seals and receipts were one
2035 +
    /// entry each (`sigs/`, `seals/`, `receipts/` lists) still reads, and
2036 +
    /// what is written now sits beside what was there.
2037 +
    #[test]
2038 +
    fn an_older_store_s_sidecar_lists_are_still_read() {
2039 +
        if !super::files_on_disk() {
2040 +
            return;
2041 +
        }
1823 2042
        use crate::identity::{Domain, SecretKey};
1824 2043
        let dir = tempfile::tempdir().unwrap();
1825 2044
        let s = Store::open(dir.path()).unwrap();
1826 -
        let r = s.put(ObjectType::Chunk, b"x").unwrap();
1827 -
        assert!(s.signatures(&r).unwrap().is_empty());
2045 +
        let r = s.put(ObjectType::Chunk, b"old").unwrap();
2046 +
        let h = r.to_hex();
1828 2047
        let sk = SecretKey {
1829 2048
            ed_seed: [1; 32],
1830 2049
            ml_seed: [2; 32],
1831 2050
        };
1832 -
        let id = sk.identity("a", "b", 0);
1833 -
        let key = s.put_identity(&id).unwrap();
1834 -
        assert_eq!(s.get_identity(&key).unwrap(), id);
2051 +
        let key = s.put_identity(&sk.identity("a", "b", 0)).unwrap();
1835 2052
        let sig = sk.sign(Domain::Change, &key, &r);
1836 -
        s.put_signature(&r, &sig).unwrap();
2053 +
        let legacy = dir.path().join("sigs").join(&h[..2]);
2054 +
        fs::create_dir_all(&legacy).unwrap();
2055 +
        fs::write(
2056 +
            legacy.join(&h[2..]),
2057 +
            Signature::encode_list(std::slice::from_ref(&sig)),
2058 +
        )
2059 +
        .unwrap();
2060 +
        let seals = dir.path().join("seals").join(&h[..2]);
2061 +
        fs::create_dir_all(&seals).unwrap();
2062 +
        fs::write(
2063 +
            seals.join(&h[2..]),
2064 +
            format!("{}\n", ObjectRef([9; 32]).to_hex()),
2065 +
        )
2066 +
        .unwrap();
2067 +
        let receipts = dir.path().join("receipts").join(&h[..2]);
2068 +
        fs::create_dir_all(&receipts).unwrap();
2069 +
        fs::write(
2070 +
            receipts.join(&h[2..]),
2071 +
            format!("{}\n", ObjectRef([8; 32]).to_hex()),
2072 +
        )
2073 +
        .unwrap();
2074 +
2075 +
        assert_eq!(s.signatures(&r).unwrap(), vec![sig.clone()]);
2076 +
        assert_eq!(s.seals_for(&r).unwrap(), vec![ObjectRef([9; 32])]);
2077 +
        assert_eq!(s.receipts_for(&r).unwrap(), vec![ObjectRef([8; 32])]);
2078 +
        assert_eq!(s.receipt_states().unwrap(), vec![r]);
2079 +
        // New entries join the old ones; the same signer is not doubled.
1837 2080
        s.put_signature(&r, &sig).unwrap();
1838 -
        assert_eq!(s.signatures(&r).unwrap(), vec![sig]);
1839 -
        // The object itself is untouched by its signatures.
1840 -
        assert_eq!(s.get(&r).unwrap().1, b"x");
2081 +
        s.index_seal(&r, &ObjectRef([1; 32])).unwrap();
2082 +
        assert_eq!(s.signatures(&r).unwrap().len(), 1);
2083 +
        assert_eq!(
2084 +
            s.seals_for(&r).unwrap(),
2085 +
            vec![ObjectRef([1; 32]), ObjectRef([9; 32])]
2086 +
        );
2087 +
        // Pruning the object takes both layouts with it.
2088 +
        let r2 = s.put(ObjectType::Chunk, b"kept").unwrap();
2089 +
        let keep = [r2, key].into_iter().collect();
2090 +
        s.prune(&keep, std::time::Duration::ZERO).unwrap();
2091 +
        assert!(s.signatures(&r).unwrap().is_empty());
2092 +
        assert!(s.seals_for(&r).unwrap().is_empty());
2093 +
        assert!(s.receipt_states().unwrap().is_empty());
1841 2094
    }
1842 2095
1843 2096
    #[test]
1844 2097
    fn change_round_trip_through_store() {
1845 2098
        let dir = tempfile::tempdir().unwrap();
1846 -
        let s = Store::open(dir.path()).unwrap();
1847 -
        let c = crate::objects::tests::sample_change();
1848 -
        let r = s.put_change(&c).unwrap();
1849 -
        assert_eq!(r, c.hash());
1850 -
        assert_eq!(s.get_change(&r).unwrap(), c);
2099 +
        each_store(dir.path(), |which, s| {
2100 +
            let c = crate::objects::tests::sample_change();
2101 +
            let r = s.put_change(&c).unwrap();
2102 +
            assert_eq!(r, c.hash(), "{which}");
2103 +
            assert_eq!(s.get_change(&r).unwrap(), c);
2104 +
            assert_eq!(s.get_change_meta(&r).unwrap().message, c.message);
2105 +
        });
1851 2106
    }
1852 2107
1853 2108
    fn xorshift(state: &mut u64) -> u64 {
…
8 unchanged lines
1862 2117
    /// stay readable when the index survives.
1863 2118
    #[test]
1864 2119
    fn corrupted_packs_and_indexes_fail_cleanly() {
2120 +
        if !super::files_on_disk() {
2121 +
            return;
2122 +
        }
1865 2123
        let dir = tempfile::tempdir().unwrap();
1866 2124
        let objects = dir.path().join("objects");
1867 2125
        let s = Store::open(&objects).unwrap();
…
82 unchanged lines
1950 2208
    #[test]
1951 2209
    fn batches_write_loose_or_one_pack() {
1952 2210
        let dir = tempfile::tempdir().unwrap();
1953 -
        let s = Store::open(dir.path().join("objects")).unwrap();
1954 -
        let outer = s.begin_batch().unwrap();
1955 -
        let a = s.put(ObjectType::Chunk, b"small a").unwrap();
1956 -
        {
1957 -
            let inner = s.begin_batch().unwrap();
1958 -
            let b = s.put(ObjectType::Chunk, b"small b").unwrap();
1959 -
            assert!(s.has(&b));
1960 -
            assert_eq!(s.get(&b).unwrap().1, b"small b");
1961 -
            inner.finish().unwrap();
2211 +
        each_store(dir.path(), |which, s| {
2212 +
            let outer = s.begin_batch().unwrap();
2213 +
            let a = s.put(ObjectType::Chunk, b"small a").unwrap();
2214 +
            {
2215 +
                let inner = s.begin_batch().unwrap();
2216 +
                let b = s.put(ObjectType::Chunk, b"small b").unwrap();
2217 +
                assert!(s.has(&b));
2218 +
                assert_eq!(s.get(&b).unwrap().1, b"small b");
2219 +
                inner.finish().unwrap();
2220 +
                assert_eq!(
2221 +
                    s.loose_count().unwrap(),
2222 +
                    0,
2223 +
                    "{which}: the inner batch defers to the outer one"
2224 +
                );
2225 +
            }
2226 +
            assert_eq!(s.get(&a).unwrap().1, b"small a");
1962 2227
            assert_eq!(
1963 -
                s.loose_count().unwrap(),
1964 -
                0,
1965 -
                "the inner batch defers to the outer one"
2228 +
                s.put(ObjectType::Chunk, b"small a").unwrap(),
2229 +
                a,
2230 +
                "idempotent while pending"
1966 2231
            );
1967 -
        }
1968 -
        assert_eq!(s.get(&a).unwrap().1, b"small a");
1969 -
        assert_eq!(
1970 -
            s.put(ObjectType::Chunk, b"small a").unwrap(),
1971 -
            a,
1972 -
            "idempotent while pending"
1973 -
        );
1974 -
        outer.finish().unwrap();
1975 -
        assert_eq!(s.loose_count().unwrap(), 2);
1976 -
        assert_eq!(s.pack_count(), 0);
1977 -
        assert_eq!(s.get(&a).unwrap().1, b"small a");
2232 +
            outer.finish().unwrap();
2233 +
            assert_eq!(s.loose_count().unwrap(), 2);
2234 +
            assert_eq!(s.pack_count(), 0);
2235 +
            assert_eq!(s.get(&a).unwrap().1, b"small a");
1978 2236
1979 -
        // Large: spills into one pack, readable during and after.
1980 -
        let big = s.begin_batch().unwrap();
1981 -
        let mut refs = Vec::new();
1982 -
        let mut seed = 7u64;
1983 -
        for i in 0..300u32 {
1984 -
            let mut body = vec![b'z'; 40_000];
1985 -
            if i % 3 == 0 {
1986 -
                for b in body.iter_mut() {
1987 -
                    *b = (xorshift(&mut seed) & 0xff) as u8;
2237 +
            // Large: spills into one pack, readable during and after.
2238 +
            let big = s.begin_batch().unwrap();
2239 +
            let mut refs = Vec::new();
2240 +
            let mut seed = 7u64;
2241 +
            for i in 0..300u32 {
2242 +
                let mut body = vec![b'z'; 40_000];
2243 +
                if i % 3 == 0 {
2244 +
                    for b in body.iter_mut() {
2245 +
                        *b = (xorshift(&mut seed) & 0xff) as u8;
2246 +
                    }
1988 2247
                }
2248 +
                body[..4].copy_from_slice(&i.to_le_bytes());
2249 +
                let r = if i % 5 == 0 {
2250 +
                    s.put_chunk_nocompress(&body).unwrap()
2251 +
                } else {
2252 +
                    s.put(ObjectType::Chunk, &body).unwrap()
2253 +
                };
2254 +
                assert_eq!(
2255 +
                    s.get(&r).unwrap().1,
2256 +
                    body,
2257 +
                    "readable while pending (object {i})"
2258 +
                );
2259 +
                refs.push((r, body));
1989 2260
            }
1990 -
            body[..4].copy_from_slice(&i.to_le_bytes());
1991 -
            let r = if i % 5 == 0 {
1992 -
                s.put_chunk_nocompress(&body).unwrap()
1993 -
            } else {
1994 -
                s.put(ObjectType::Chunk, &body).unwrap()
1995 -
            };
2261 +
            assert!(s.has(&refs[0].0));
2262 +
            big.finish().unwrap();
2263 +
            assert_eq!(s.pack_count(), 1);
1996 2264
            assert_eq!(
1997 -
                s.get(&r).unwrap().1,
1998 -
                body,
1999 -
                "readable while pending (object {i})"
2265 +
                s.loose_count().unwrap(),
2266 +
                2,
2267 +
                "nothing from the big batch is loose"
2000 2268
            );
2001 -
            refs.push((r, body));
2002 -
        }
2003 -
        assert!(s.has(&refs[0].0));
2004 -
        big.finish().unwrap();
2005 -
        assert_eq!(s.pack_count(), 1);
2006 -
        assert_eq!(
2007 -
            s.loose_count().unwrap(),
2008 -
            2,
2009 -
            "nothing from the big batch is loose"
2010 -
        );
2011 -
        assert!(fs::read_dir(dir.path().join("objects"))
2012 -
            .unwrap()
2013 -
            .flatten()
2014 -
            .all(|e| !e.file_name().to_string_lossy().starts_with("tmp-")));
2015 -
        let s2 = Store::open(dir.path().join("objects")).unwrap();
2016 -
        for (r, body) in &refs {
2017 -
            assert_eq!(s2.get(r).unwrap().1, *body);
2018 -
        }
2019 -
        // A dropped batch still writes its objects.
2020 -
        {
2021 -
            let _b = s2.begin_batch().unwrap();
2022 -
            s2.put(ObjectType::Chunk, b"dropped").unwrap();
2023 -
        }
2024 -
        assert_eq!(
2025 -
            s2.get(&hash_object(ObjectType::Chunk, b"dropped"))
2269 +
            assert!(fs::read_dir(s.dir())
2026 2270
                .unwrap()
2027 -
                .1,
2028 -
            b"dropped"
2029 -
        );
2271 +
                .flatten()
2272 +
                .all(|e| !e.file_name().to_string_lossy().contains("tmp-")));
2273 +
            let s2 = Store::open_on(s.backend().clone(), s.dir()).unwrap();
2274 +
            for (r, body) in &refs {
2275 +
                assert_eq!(s2.get(r).unwrap().1, *body);
2276 +
            }
2277 +
            // A dropped batch still writes its objects.
2278 +
            {
2279 +
                let _b = s2.begin_batch().unwrap();
2280 +
                s2.put(ObjectType::Chunk, b"dropped").unwrap();
2281 +
            }
2282 +
            assert_eq!(
2283 +
                s2.get(&hash_object(ObjectType::Chunk, b"dropped"))
2284 +
                    .unwrap()
2285 +
                    .1,
2286 +
                b"dropped"
2287 +
            );
2288 +
        });
2030 2289
    }
2031 2290
2032 2291
    #[test]
2033 2292
    fn get_many_reads_loose_and_packed_objects_in_order() {
2034 2293
        let dir = tempfile::tempdir().unwrap();
2035 -
        let s = Store::open(dir.path().join("objects")).unwrap();
2036 -
        let packed: Vec<ObjectRef> = (0..20)
2037 -
            .map(|i| {
2038 -
                s.put(ObjectType::Chunk, format!("packed {i}").as_bytes())
2039 -
                    .unwrap()
2040 -
            })
2041 -
            .collect();
2042 -
        s.pack_loose().unwrap();
2043 -
        let loose: Vec<ObjectRef> = (0..20)
2044 -
            .map(|i| {
2045 -
                s.put(ObjectType::Chunk, format!("loose {i}").as_bytes())
2046 -
                    .unwrap()
2047 -
            })
2048 -
            .collect();
2049 -
        let missing = hash_object(ObjectType::Chunk, b"never stored");
2050 -
        let mut refs: Vec<ObjectRef> = Vec::new();
2051 -
        for i in 0..20 {
2052 -
            refs.push(loose[i]);
2053 -
            refs.push(packed[i]);
2054 -
        }
2055 -
        refs.push(missing);
2056 -
        let got = s.get_many(&refs);
2057 -
        assert_eq!(got.len(), refs.len());
2058 -
        for (i, r) in refs.iter().enumerate().take(40) {
2059 -
            let want = if i % 2 == 0 {
2060 -
                format!("loose {}", i / 2)
2061 -
            } else {
2062 -
                format!("packed {}", i / 2)
2063 -
            };
2064 -
            let (ty, body) = got[i].as_ref().unwrap();
2065 -
            assert_eq!(
2066 -
                (*ty, body.as_slice()),
2067 -
                (ObjectType::Chunk, want.as_bytes()),
2068 -
                "{r:?}"
2069 -
            );
2070 -
        }
2071 -
        assert!(matches!(got[40], Err(StoreError::NotFound(_))));
2294 +
        each_store(dir.path(), |which, s| {
2295 +
            let packed: Vec<ObjectRef> = (0..20)
2296 +
                .map(|i| {
2297 +
                    s.put(ObjectType::Chunk, format!("packed {i}").as_bytes())
2298 +
                        .unwrap()
2299 +
                })
2300 +
                .collect();
2301 +
            s.pack_loose().unwrap();
2302 +
            let loose: Vec<ObjectRef> = (0..20)
2303 +
                .map(|i| {
2304 +
                    s.put(ObjectType::Chunk, format!("loose {i}").as_bytes())
2305 +
                        .unwrap()
2306 +
                })
2307 +
                .collect();
2308 +
            let missing = hash_object(ObjectType::Chunk, b"never stored");
2309 +
            let mut refs: Vec<ObjectRef> = Vec::new();
2310 +
            for i in 0..20 {
2311 +
                refs.push(loose[i]);
2312 +
                refs.push(packed[i]);
2313 +
            }
2314 +
            refs.push(missing);
2315 +
            let got = s.get_many(&refs);
2316 +
            assert_eq!(got.len(), refs.len());
2317 +
            for (i, r) in refs.iter().enumerate().take(40) {
2318 +
                let want = if i % 2 == 0 {
2319 +
                    format!("loose {}", i / 2)
2320 +
                } else {
2321 +
                    format!("packed {}", i / 2)
2322 +
                };
2323 +
                let (ty, body) = got[i].as_ref().unwrap();
2324 +
                assert_eq!(
2325 +
                    (*ty, body.as_slice()),
2326 +
                    (ObjectType::Chunk, want.as_bytes()),
2327 +
                    "{which}: {r:?}"
2328 +
                );
2329 +
            }
2330 +
            assert!(matches!(got[40], Err(StoreError::NotFound(_))));
2331 +
        });
2072 2332
    }
2073 2333
}
2074 2334
…
96 unchanged lines
2171 2431
2172 2432
    #[test]
2173 2433
    fn a_change_whose_text_blob_is_missing_is_reported_not_misread() {
2434 +
        if !super::files_on_disk() {
2435 +
            return;
2436 +
        }
2174 2437
        let dir = tempfile::tempdir().unwrap();
2175 2438
        let s = Store::open(dir.path()).unwrap();
2176 2439
        let c = large_change();
…
39 unchanged lines
2216 2479
        for (r, b) in refs.iter().zip(&objects) {
2217 2480
            assert_eq!(again.get(r).unwrap().1, b.1);
2218 2481
        }
2219 -
    }
2220 -
}
2221 -
2222 -
/// Remove a pack with its index and dictionary.
2223 -
fn remove_pack_files(pack: &Path) -> io::Result<()> {
2224 -
    fs::remove_file(pack)?;
2225 -
    fs::remove_file(pack.with_extension("idx"))?;
2226 -
    match fs::remove_file(pack.with_extension("dict")) {
2227 -
        Err(e) if e.kind() != io::ErrorKind::NotFound => Err(e),
2228 -
        _ => Ok(()),
2229 2482
    }
2230 2483
}
modified crates/oxy-core/src/strata.rs +230 −47
…
9 unchanged lines
10 10
//! a directory. Two writers can therefore share a store with no lock
11 11
//! server: the loser of a race reads again, merges, and tries again.
12 12
//!
13 -
//! The working repository (`.oxy/`) is not a `Store`; it is read and
14 -
//! written directly, as before. A store is what a *remote* is kept on.
15 -
//! This module holds the trait, a directory store, an in-memory store
16 -
//! for tests, and the conformance check every store must pass. Cloud
17 -
//! stores live in a crate of their own, because of what they pull in.
13 +
//! A repository's own objects (`.oxy/objects`, see [`crate::store`]) are
14 +
//! kept on a store too, a directory one, with the same keys a bucket
15 +
//! would hold; a remote is a store as well. This module holds the trait,
16 +
//! a directory store, an in-memory store for tests, and the conformance
17 +
//! check every store must pass. Cloud stores live in a crate of their
18 +
//! own, because of what they pull in.
18 19
19 20
use std::collections::BTreeMap;
20 21
use std::fs;
…
17 unchanged lines
38 39
pub struct Entry {
39 40
    pub key: String,
40 41
    pub size: u64,
42 +
    /// When the object was written, in wall milliseconds, where the store
43 +
    /// knows (garbage collection's grace period reads it).
44 +
    pub modified_ms: Option<u64>,
41 45
}
42 46
43 47
#[derive(Debug, thiserror::Error)]
…
71 unchanged lines
115 119
    fn list(&self, prefix: &str) -> Result<Vec<Entry>>;
116 120
    /// Remove an object; fine when it is already gone.
117 121
    fn delete(&self, key: &str) -> Result<()>;
122 +
    /// One object's listing entry without listing its neighbours; `None`
123 +
    /// when there is none.
124 +
    fn stat(&self, key: &str) -> Result<Option<Entry>> {
125 +
        Ok(self.list(key)?.into_iter().find(|e| e.key == key))
126 +
    }
127 +
    /// Write several objects that do not exist yet; one that does is left
128 +
    /// as it is, which is not an error (objects are named by content). A
129 +
    /// store that can batch its syncs does; this does them one at a time.
130 +
    fn put_many_new(&self, objects: &[(String, Vec<u8>)]) -> Result<()> {
131 +
        for (key, bytes) in objects {
132 +
            match self.put_new(key, bytes) {
133 +
                Ok(()) | Err(Error::Exists(_)) => {}
134 +
                Err(e) => return Err(e),
135 +
            }
136 +
        }
137 +
        Ok(())
138 +
    }
139 +
    /// `put_new` of a file's contents: a store on the same disk can move
140 +
    /// the file into place instead of copying it.
141 +
    fn put_new_file(&self, key: &str, path: &Path) -> Result<()> {
142 +
        self.put_new(key, &fs::read(path)?)
143 +
    }
118 144
119 145
    // ---- references: one small record, updated by compare-and-swap
120 146
…
89 unchanged lines
210 236
                    self.list_dir(&entry.path(), &key, prefix, out)?;
211 237
                }
212 238
            } else if ty.is_file() && key.starts_with(prefix) {
239 +
                let meta = entry.metadata()?;
213 240
                out.push(Entry {
214 241
                    key,
215 -
                    size: entry.metadata()?.len(),
242 +
                    size: meta.len(),
243 +
                    modified_ms: modified_ms(&meta),
216 244
                });
217 245
            }
218 246
        }
…
34 unchanged lines
253 281
                    std::thread::sleep(Duration::from_millis(10));
254 282
                }
255 283
                Err(e) => return Err(e.into()),
284 +
            }
285 +
        }
286 +
    }
287 +
}
288 +
289 +
/// A file's modification time as wall milliseconds.
290 +
fn modified_ms(meta: &fs::Metadata) -> Option<u64> {
291 +
    meta.modified()
292 +
        .ok()
293 +
        .and_then(|t| t.duration_since(std::time::UNIX_EPOCH).ok())
294 +
        .map(|d| d.as_millis() as u64)
295 +
}
296 +
297 +
impl LocalStore {
298 +
    /// Link `tmp` into `path`, create-only; the directory is synced and
299 +
    /// `tmp` removed either way. `Exists` when the name is taken.
300 +
    fn link_into_place(&self, key: &str, tmp: &Path, path: &Path) -> Result<()> {
301 +
        let linked = fs::hard_link(tmp, path);
302 +
        let _ = fs::remove_file(tmp);
303 +
        match linked {
304 +
            Ok(()) => {
305 +
                if let Some(dir) = path.parent() {
306 +
                    crate::durable::sync_dir(dir)?;
307 +
                }
308 +
                Ok(())
309 +
            }
310 +
            Err(e) if e.kind() == io::ErrorKind::AlreadyExists => {
311 +
                Err(Error::Exists(key.to_string()))
256 312
            }
313 +
            Err(e) => Err(e.into()),
314 +
        }
315 +
    }
316 +
317 +
    /// Create `path`'s directory, syncing the first directory that was
318 +
    /// made so its entry is durable too.
319 +
    fn ensure_dir(&self, path: &Path) -> Result<()> {
320 +
        let Some(dir) = path.parent() else {
321 +
            return Ok(());
322 +
        };
323 +
        if dir.is_dir() {
324 +
            return Ok(());
325 +
        }
326 +
        let mut made = dir.to_path_buf();
327 +
        while let Some(parent) = made.parent() {
328 +
            if parent.is_dir() || parent == self.root {
329 +
                break;
330 +
            }
331 +
            made = parent.to_path_buf();
332 +
        }
333 +
        fs::create_dir_all(dir)?;
334 +
        if let Some(parent) = made.parent() {
335 +
            crate::durable::sync_dir(parent)?;
257 336
        }
337 +
        Ok(())
258 338
    }
259 339
}
260 340
261 341
impl Store for LocalStore {
262 342
    fn describe(&self) -> String {
263 343
        self.root.display().to_string()
344 +
    }
345 +
346 +
    fn stat(&self, key: &str) -> Result<Option<Entry>> {
347 +
        match fs::metadata(self.path(key)?) {
348 +
            Ok(m) if m.is_file() => Ok(Some(Entry {
349 +
                key: key.to_string(),
350 +
                size: m.len(),
351 +
                modified_ms: modified_ms(&m),
352 +
            })),
353 +
            Ok(_) => Ok(None),
354 +
            Err(e) if e.kind() == io::ErrorKind::NotFound => Ok(None),
355 +
            Err(e) => Err(e.into()),
356 +
        }
357 +
    }
358 +
359 +
    /// Every file written and synced (on every core), then each linked
360 +
    /// into place and each directory synced once: durable when this
361 +
    /// returns, as one at a time would be, with far fewer waits.
362 +
    fn put_many_new(&self, objects: &[(String, Vec<u8>)]) -> Result<()> {
363 +
        use rayon::prelude::*;
364 +
        let mut todo: Vec<(PathBuf, PathBuf, &[u8])> = Vec::new();
365 +
        for (key, bytes) in objects {
366 +
            let path = self.path(key)?;
367 +
            if path.exists() || todo.iter().any(|(p, ..)| *p == path) {
368 +
                continue;
369 +
            }
370 +
            self.ensure_dir(&path)?;
371 +
            todo.push((path.clone(), self.temp_beside(&path), bytes));
372 +
        }
373 +
        if todo.is_empty() {
374 +
            return Ok(());
375 +
        }
376 +
        crate::durable::step("write loose objects")?;
377 +
        todo.par_iter()
378 +
            .try_for_each(|(_, tmp, bytes)| -> io::Result<()> {
379 +
                let mut f = fs::File::create(tmp)?;
380 +
                f.write_all(bytes)?;
381 +
                f.sync_all()
382 +
            })?;
383 +
        crate::durable::step("rename loose objects")?;
384 +
        let mut dirs: std::collections::BTreeSet<PathBuf> = std::collections::BTreeSet::new();
385 +
        for (path, tmp, _) in &todo {
386 +
            let linked = fs::hard_link(tmp, path);
387 +
            let _ = fs::remove_file(tmp);
388 +
            match linked {
389 +
                Ok(()) => {
390 +
                    if let Some(d) = path.parent() {
391 +
                        dirs.insert(d.to_path_buf());
392 +
                    }
393 +
                }
394 +
                Err(e) if e.kind() == io::ErrorKind::AlreadyExists => {}
395 +
                Err(e) => return Err(e.into()),
396 +
            }
397 +
        }
398 +
        dirs.par_iter()
399 +
            .try_for_each(|d| crate::durable::sync_dir(d))?;
400 +
        Ok(())
264 401
    }
265 402
403 +
    /// The file is linked into place when it is on this file system,
404 +
    /// else copied; it is removed either way.
405 +
    fn put_new_file(&self, key: &str, file: &Path) -> Result<()> {
406 +
        let path = self.path(key)?;
407 +
        if path.exists() {
408 +
            let _ = fs::remove_file(file);
409 +
            return Err(Error::Exists(key.to_string()));
410 +
        }
411 +
        self.ensure_dir(&path)?;
412 +
        match fs::hard_link(file, &path) {
413 +
            Ok(()) => {
414 +
                let _ = fs::remove_file(file);
415 +
                if let Some(dir) = path.parent() {
416 +
                    crate::durable::sync_dir(dir)?;
417 +
                }
418 +
                Ok(())
419 +
            }
420 +
            Err(e) if e.kind() == io::ErrorKind::AlreadyExists => {
421 +
                let _ = fs::remove_file(file);
422 +
                Err(Error::Exists(key.to_string()))
423 +
            }
424 +
            Err(_) => {
425 +
                // Another file system: copy beside the name, then link.
426 +
                let tmp = self.temp_beside(&path);
427 +
                let copied = (|| {
428 +
                    let mut src = fs::File::open(file)?;
429 +
                    let mut dst = fs::File::create(&tmp)?;
430 +
                    io::copy(&mut src, &mut dst)?;
431 +
                    dst.sync_all()
432 +
                })();
433 +
                let _ = fs::remove_file(file);
434 +
                copied?;
435 +
                self.link_into_place(key, &tmp, &path)
436 +
            }
437 +
        }
438 +
    }
439 +
266 440
    fn has(&self, key: &str) -> Result<bool> {
267 441
        Ok(self.path(key)?.is_file())
268 442
    }
…
33 unchanged lines
302 476
        let path = self.path(key)?;
303 477
        if path.exists() {
304 478
            return Err(Error::Exists(key.to_string()));
305 -
        }
306 -
        if let Some(dir) = path.parent() {
307 -
            fs::create_dir_all(dir)?;
308 479
        }
480 +
        self.ensure_dir(&path)?;
309 481
        // Written beside its name, then linked into it: the link fails if
310 482
        // the name was taken meanwhile, and a reader never sees a part.
311 483
        let tmp = self.temp_beside(&path);
…
2 unchanged lines
314 486
            f.write_all(bytes)?;
315 487
            f.sync_all()
316 488
        })();
317 -
        let linked = written.and_then(|()| fs::hard_link(&tmp, &path));
318 -
        let _ = fs::remove_file(&tmp);
319 -
        match linked {
320 -
            Ok(()) => {
321 -
                if let Some(dir) = path.parent() {
322 -
                    crate::durable::sync_dir(dir)?;
323 -
                }
324 -
                Ok(())
325 -
            }
326 -
            Err(e) if e.kind() == io::ErrorKind::AlreadyExists => {
327 -
                Err(Error::Exists(key.to_string()))
328 -
            }
329 -
            Err(e) => Err(e.into()),
489 +
        if let Err(e) = written {
490 +
            let _ = fs::remove_file(&tmp);
491 +
            return Err(e.into());
330 492
        }
493 +
        self.link_into_place(key, &tmp, &path)
331 494
    }
332 495
333 496
    fn list(&self, prefix: &str) -> Result<Vec<Entry>> {
…
47 unchanged lines
381 544
/// cloud stores' do.
382 545
#[derive(Default)]
383 546
pub struct MemoryStore {
384 -
    objects: Mutex<BTreeMap<String, Vec<u8>>>,
547 +
    objects: Mutex<BTreeMap<String, (Vec<u8>, u64)>>,
385 548
}
386 549
550 +
fn now_ms() -> u64 {
551 +
    std::time::SystemTime::now()
552 +
        .duration_since(std::time::UNIX_EPOCH)
553 +
        .map(|d| d.as_millis() as u64)
554 +
        .unwrap_or(0)
555 +
}
556 +
387 557
impl MemoryStore {
388 558
    pub fn new() -> MemoryStore {
389 559
        MemoryStore::default()
390 560
    }
391 561
392 -
    fn map(&self) -> std::sync::MutexGuard<'_, BTreeMap<String, Vec<u8>>> {
562 +
    fn map(&self) -> std::sync::MutexGuard<'_, BTreeMap<String, (Vec<u8>, u64)>> {
393 563
        self.objects.lock().unwrap_or_else(|p| p.into_inner())
564 +
    }
565 +
566 +
    /// Pretend `key` was written `age` ago (for grace-period tests).
567 +
    pub fn age(&self, key: &str, age: Duration) {
568 +
        if let Some(entry) = self.map().get_mut(key) {
569 +
            entry.1 = now_ms().saturating_sub(age.as_millis() as u64);
570 +
        }
394 571
    }
395 572
}
396 573
…
11 unchanged lines
408 585
        check_key(key)?;
409 586
        self.map()
410 587
            .get(key)
411 -
            .cloned()
588 +
            .map(|(b, _)| b.clone())
412 589
            .ok_or_else(|| Error::NotFound(key.to_string()))
413 590
    }
414 591
…
15 unchanged lines
430 607
        if map.contains_key(key) {
431 608
            return Err(Error::Exists(key.to_string()));
432 609
        }
433 -
        map.insert(key.to_string(), bytes.to_vec());
610 +
        map.insert(key.to_string(), (bytes.to_vec(), now_ms()));
434 611
        Ok(())
435 612
    }
436 613
…
2 unchanged lines
439 616
            .map()
440 617
            .range(prefix.to_string()..)
441 618
            .take_while(|(k, _)| k.starts_with(prefix))
442 -
            .map(|(k, v)| Entry {
619 +
            .map(|(k, (v, t))| Entry {
443 620
                key: k.clone(),
444 621
                size: v.len() as u64,
622 +
                modified_ms: Some(*t),
445 623
            })
446 624
            .collect())
447 625
    }
…
6 unchanged lines
454 632
455 633
    fn read_ref(&self, key: &str) -> Result<Option<(Vec<u8>, Version)>> {
456 634
        check_key(key)?;
457 -
        Ok(self.map().get(key).map(|b| (b.clone(), content_version(b))))
635 +
        Ok(self
636 +
            .map()
637 +
            .get(key)
638 +
            .map(|(b, _)| (b.clone(), content_version(b))))
458 639
    }
459 640
460 641
    fn write_ref(&self, key: &str, bytes: &[u8], expected: Option<&Version>) -> Result<Version> {
461 642
        check_key(key)?;
462 643
        let mut map = self.map();
463 -
        let current = map.get(key).map(|b| content_version(b));
644 +
        let current = map.get(key).map(|(b, _)| content_version(b));
464 645
        check_expected(key, current.as_ref(), expected)?;
465 -
        map.insert(key.to_string(), bytes.to_vec());
646 +
        map.insert(key.to_string(), (bytes.to_vec(), now_ms()));
466 647
        Ok(content_version(bytes))
467 648
    }
468 649
}
…
59 unchanged lines
528 709
    passed.push("objects are written once");
529 710
530 711
    store.put_new(&key("dir/b"), b"bb")?;
531 -
    let listed = store.list(prefix)?;
712 +
    let listed: Vec<(String, u64)> = store
713 +
        .list(prefix)?
714 +
        .into_iter()
715 +
        .map(|e| (e.key, e.size))
716 +
        .collect();
532 717
    expect(
533 -
        listed
534 -
            == vec![
535 -
                Entry {
536 -
                    key: key("a"),
537 -
                    size: a.len() as u64,
538 -
                },
539 -
                Entry {
540 -
                    key: key("dir/b"),
541 -
                    size: 2,
542 -
                },
543 -
            ],
718 +
        listed == vec![(key("a"), a.len() as u64), (key("dir/b"), 2)],
544 719
        "list() does not return the two objects, sorted, with their sizes",
545 720
    )?;
546 721
    expect(
547 -
        store.list(&key("dir/"))?
548 -
            == vec![Entry {
549 -
                key: key("dir/b"),
550 -
                size: 2,
551 -
            }],
722 +
        store
723 +
            .stat(&key("dir/b"))?
724 +
            .is_some_and(|e| e.key == key("dir/b") && e.size == 2)
725 +
            && store.stat(&key("dir/c"))?.is_none(),
726 +
        "stat() does not describe one object, or describes a missing one",
727 +
    )?;
728 +
    expect(
729 +
        store
730 +
            .list(&key("dir/"))?
731 +
            .into_iter()
732 +
            .map(|e| (e.key, e.size))
733 +
            .collect::<Vec<_>>()
734 +
            == vec![(key("dir/b"), 2)],
552 735
        "list() of a sub-prefix is wrong",
553 736
    )?;
554 737
    expect(
…
181 unchanged lines
modified crates/oxy-strata/src/lib.rs +1 −0
…
340 unchanged lines
341 341
                key.starts_with(prefix).then(|| Entry {
342 342
                    key: key.to_string(),
343 343
                    size: m.size,
344 +
                    modified_ms: u64::try_from(m.last_modified.timestamp_millis()).ok(),
344 345
                })
345 346
            })
346 347
            .collect();
…
90 unchanged lines
modified docs/stores.md +17 −7
…
89 unchanged lines
90 90
  - `strata::conformance(store, prefix)` exercises any store under a
91 91
    probe prefix and leaves it empty: every implementation runs it in its
92 92
    tests, and `oxy remote check` runs it against a remote.
93 -
  - The working repository's own `.oxy/` store is **not** behind the
94 -
    trait: it is read and written directly, as before, so nothing on the
95 -
    local path changes or slows. A store is what a remote is kept on.
93 +
  - Since 2026-10-04 (oxygen#68, B1) the repository's own object store
94 +
    is behind the trait too: `.oxy/objects` is a `LocalStore` with the
95 +
    same keys a bucket would hold (`loose/`, `pack-*`, `nocompress/`,
96 +
    `sig/`, `seal/`, `receipt/`), and the pack code reads a packed
97 +
    object with one ranged read. A directory store is as fast as the
98 +
    files were: the same stats, reads and syncs, with a batch of loose
99 +
    objects synced together. What must be on this disk, a pack being
100 +
    streamed out and the derived `metas` cache, lives in a scratch
101 +
    directory, which for a directory store is the store itself.
96 102
- **The `object_store` crate** (Apache Arrow) for S3 and GCS: one API
97 103
  for both, mature credential chains, multipart uploads, and conditional
98 104
  puts. Azure is in the same crate, so it comes nearly free later.
…
2 unchanged lines
101 107
  is missing? The client sends packs, never loose objects.
102 108
- **A local cache** of fetched objects, which is always correct because
103 109
  objects never change.
104 -
- **The working repository stays local.** A bucket is a remote. Running
105 -
  a repository *from* a bucket (no local `.oxy/`) is out of scope here;
106 -
  it is what the virtual working copy (§8) is for.
110 +
- **The working repository stays local for now.** A bucket is a remote.
111 +
  Running a repository *from* a bucket is plan B (oxygen#68): a `Repo`
112 +
  opened over a bucket store with a local cache in front, so the hub
113 +
  holds no copy of its own and the data lives where its owner chose.
114 +
  B1 (the store over the trait) is done; B2 is the caching store and
115 +
  the references by compare-and-swap.
107 116
108 117
## 4. Layout in the bucket (as built, 2026-10-03)
109 118
…
104 unchanged lines
214 223
## 8. Order of work (done)
215 224
216 225
1. ✔ **The `Store` trait in oxy-core**, with a directory store and an
217 -
   in-memory one; the working repository's own store left as it was.
226 +
   in-memory one; the working repository's own store left as it was,
227 +
   then (B1, 2026-10-04) put behind the trait with the same layout.
218 228
2. ✔ **The bucket layout and the store remote**, tested with no network.
219 229
3. ✔ **S3, GCS and Azure stores**, in `crates/oxy-strata`, verified
220 230
   against the real services.
…
29 unchanged lines