OXYGENHUB

gc leaves a store that is already one full pack as it is a67914bced23

gc repacks at the harder level, and repack_all skipped only at the
everyday level, so every gc rewrote the whole pack even with nothing
new: the hub's nightly gc took andymccall/oxygen's write lock for 37 s
(48 000 objects) and metalgear2's for 87 s in GCS, while reads queued.
Nothing loose and one pack with a trained dictionary (which only a full
repack makes) now skips it. Spec 13.4; the repack test checks the skip
and that one new object still repacks.

Refs andymccall/oxygenhub#249
Author
Andy McCall @andymccall <[email protected]>
via
agent Claude Code <[email protected]> · delegate for changes, until 2026-11-13 19:00Z · key ea5f78160088
When
· 2026-10-17 06:16 UTC
Signature
✔ signed by Claude Code
Hash
a67914bced23083cd55bf7ef66a252cee4d3c0357806465e61b192721a338ac0
Depends on
648189388f1a a17fc622ca43 d02e545dc265 da652d6392d8 ff0af1ee299d
Depended on by
767ac6cd0545
In bookmarks
carried-not-removed doctor-trust identical-sides keyless-tags main merge-remote-bookmark paths-for-resolutions-and-amend push-notes show-stat subrepo-pin-state
In tags
v0.10.0 v0.11.0 v0.4.4 v0.4.5 v0.4.6 v0.5.0 v0.5.1 v0.6.0 v0.7.0 v0.8.0 v0.9.0
Edits
10 atoms, 0 tree operations, 2 files

Browse the tree at this change

modified crates/oxy-core/src/store.rs +28 −9
…
577 unchanged lines
578 578
    /// [`Store::repack_all`], reporting progress.
579 579
    pub fn repack_all_with(&self, level: i32, progress: Progress) -> Result<(PackReport, usize)> {
580 580
        let loose = self.loose_objects()?;
581 -
        let (mut refs, keys): (Vec<ObjectRef>, Vec<String>) = {
581 +
        let (mut refs, keys, trained): (Vec<ObjectRef>, Vec<String>, bool) = {
582 582
            let packs = self.packs.borrow();
583 583
            (
584 584
                packs.iter().flat_map(|p| p.refs.iter().copied()).collect(),
585 585
                packs.iter().map(|p| p.key.clone()).collect(),
586 +
                packs.first().is_some_and(|p| p.dict.is_some()),
586 587
            )
587 588
        };
588 -
        if loose.is_empty() && keys.len() <= 1 && level == ZSTD_LEVEL {
589 +
        // Nothing loose and one pack already made by a full repack (only
590 +
        // such a pack has a trained dictionary): rewriting it would give
591 +
        // the same pack. A hub's nightly gc rewrote andymccall/oxygen's
592 +
        // 48 000 objects (37 s under its write lock) every time.
593 +
        if loose.is_empty() && keys.len() <= 1 && (level == ZSTD_LEVEL || trained) {
589 594
            return Ok((PackReport::default(), 0));
590 595
        }
591 596
        refs.extend(loose.iter().map(|(r, _)| *r));
…
1300 unchanged lines
1892 1897
        for (r, b) in refs.iter().zip(&bodies) {
1893 1898
            assert_eq!(&again.get(r).unwrap().1, b);
1894 1899
        }
1895 -
        // A repack after that replaces the pack and its dictionary.
1900 +
        let packs = |dir: &Path| {
1901 +
            let mut v: Vec<String> = fs::read_dir(dir)
1902 +
                .unwrap()
1903 +
                .flatten()
1904 +
                .map(|e| e.file_name().to_string_lossy().into_owned())
1905 +
                .filter(|n| n.starts_with("pack-"))
1906 +
                .collect();
1907 +
            v.sort();
1908 +
            v
1909 +
        };
1910 +
        // With nothing new, a repack leaves that pack as it is: it would
1911 +
        // only write the same again.
1912 +
        let before = packs(dir.path());
1913 +
        let (report, merged) = again.repack_all(REPACK_ZSTD_LEVEL).unwrap();
1914 +
        assert_eq!((report.objects, merged), (0, 0));
1915 +
        assert_eq!(packs(dir.path()), before);
1916 +
        // One new object: a repack replaces the pack and its dictionary.
1917 +
        let extra = again.put(ObjectType::Chunk, b"one more\n").unwrap();
1896 1918
        again.repack_all(REPACK_ZSTD_LEVEL).unwrap();
1897 -
        let files: Vec<String> = fs::read_dir(dir.path())
1898 -
            .unwrap()
1899 -
            .flatten()
1900 -
            .map(|e| e.file_name().to_string_lossy().into_owned())
1901 -
            .filter(|n| n.starts_with("pack-"))
1902 -
            .collect();
1919 +
        let files = packs(dir.path());
1903 1920
        assert_eq!(files.len(), 3, "one pack, index and dictionary: {files:?}");
1921 +
        assert_ne!(files, before);
1904 1922
        assert_eq!(&again.get(&refs[7]).unwrap().1, &bodies[7]);
1923 +
        assert_eq!(again.get(&extra).unwrap().1, b"one more\n");
1905 1924
    }
1906 1925
1907 1926
    #[test]
…
576 unchanged lines
modified docs/spec.md +1 −1
…
545 unchanged lines
546 546
547 547
### 13.4 Packing
548 548
549 -
An automatic trigger when the loose object count exceeds 4096 writes loose objects into a new pack and merges packs when there are more than 16; `oxy gc` repacks every surviving object into one pack, compressed harder (zstd level 9, against level 3 for everyday writes) and on every core, as `git gc` repacks. Writers SHOULD batch: a snapshot, a fetched bundle, a Git import, or a checkpoint is one write batch, and a batch that grows beyond a small threshold (256 objects or 8 MiB) is written as a single pack with one `fsync` rather than as loose objects, each with its own; smaller batches stay loose so ordinary commits do not multiply packs. Objects of an open batch MUST be readable by the writer before the batch completes. Pack indexes MUST support lookup by ObjectRef prefix and MUST be memory-mappable. A repository with two million objects MUST be openable with under 64 MiB of resident index memory on the Full profile.
549 +
An automatic trigger when the loose object count exceeds 4096 writes loose objects into a new pack and merges packs when there are more than 16; `oxy gc` repacks every surviving object into one pack, compressed harder (zstd level 9, against level 3 for everyday writes) and on every core, as `git gc` repacks. A store that is already one such pack (it has a dictionary) with nothing loose is left as it is: repacking it would write the same pack. Writers SHOULD batch: a snapshot, a fetched bundle, a Git import, or a checkpoint is one write batch, and a batch that grows beyond a small threshold (256 objects or 8 MiB) is written as a single pack with one `fsync` rather than as loose objects, each with its own; smaller batches stay loose so ordinary commits do not multiply packs. Objects of an open batch MUST be readable by the writer before the batch completes. Pack indexes MUST support lookup by ObjectRef prefix and MUST be memory-mappable. A repository with two million objects MUST be openable with under 64 MiB of resident index memory on the Full profile.
550 550
551 551
The pack and index formats are fixed so that tools can read them without the implementation:
552 552
…
493 unchanged lines