gc leaves a store that is already one full pack as it is a67914bced23
gc repacks at the harder level, and repack_all skipped only at the everyday level, so every gc rewrote the whole pack even with nothing new: the hub's nightly gc took andymccall/oxygen's write lock for 37 s (48 000 objects) and metalgear2's for 87 s in GCS, while reads queued. Nothing loose and one pack with a trained dictionary (which only a full repack makes) now skips it. Spec 13.4; the repack test checks the skip and that one new object still repacks. Refs andymccall/oxygenhub#249
- Author
- Andy McCall @andymccall <[email protected]>
- via
- agent
Claude Code <[email protected]> · delegate for changes, until 2026-11-13 19:00Z · key ea5f78160088
- When
- · 2026-10-17 06:16 UTC
- Signature
- ✔ signed by Claude Code
- Hash
a67914bced23083cd55bf7ef66a252cee4d3c0357806465e61b192721a338ac0- Depends on
- 648189388f1a a17fc622ca43 d02e545dc265 da652d6392d8 ff0af1ee299d
- Depended on by
- 767ac6cd0545
- In bookmarks
- carried-not-removed doctor-trust identical-sides keyless-tags main merge-remote-bookmark paths-for-resolutions-and-amend push-notes show-stat subrepo-pin-state
- In tags
- v0.10.0 v0.11.0 v0.4.4 v0.4.5 v0.4.6 v0.5.0 v0.5.1 v0.6.0 v0.7.0 v0.8.0 v0.9.0
- Edits
- 10 atoms, 0 tree operations, 2 files
Browse the tree at this change
modified crates/oxy-core/src/store.rs +28 −9
| … | 577 unchanged lines |
||
| 578 | 578 | /// [`Store::repack_all`], reporting progress. |
|
| 579 | 579 | pub fn repack_all_with(&self, level: i32, progress: Progress) -> Result<(PackReport, usize)> { |
|
| 580 | 580 | let loose = self.loose_objects()?; |
|
| 581 | - | let (mut refs, keys): (Vec<ObjectRef>, Vec<String>) = { |
|
| 581 | + | let (mut refs, keys, trained): (Vec<ObjectRef>, Vec<String>, bool) = { |
|
| 582 | 582 | let packs = self.packs.borrow(); |
|
| 583 | 583 | ( |
|
| 584 | 584 | packs.iter().flat_map(|p| p.refs.iter().copied()).collect(), |
|
| 585 | 585 | packs.iter().map(|p| p.key.clone()).collect(), |
|
| 586 | + | packs.first().is_some_and(|p| p.dict.is_some()), |
|
| 586 | 587 | ) |
|
| 587 | 588 | }; |
|
| 588 | - | if loose.is_empty() && keys.len() <= 1 && level == ZSTD_LEVEL { |
|
| 589 | + | // Nothing loose and one pack already made by a full repack (only |
|
| 590 | + | // such a pack has a trained dictionary): rewriting it would give |
|
| 591 | + | // the same pack. A hub's nightly gc rewrote andymccall/oxygen's |
|
| 592 | + | // 48 000 objects (37 s under its write lock) every time. |
|
| 593 | + | if loose.is_empty() && keys.len() <= 1 && (level == ZSTD_LEVEL || trained) { |
|
| 589 | 594 | return Ok((PackReport::default(), 0)); |
|
| 590 | 595 | } |
|
| 591 | 596 | refs.extend(loose.iter().map(|(r, _)| *r)); |
|
| … | 1300 unchanged lines |
||
| 1892 | 1897 | for (r, b) in refs.iter().zip(&bodies) { |
|
| 1893 | 1898 | assert_eq!(&again.get(r).unwrap().1, b); |
|
| 1894 | 1899 | } |
|
| 1895 | - | // A repack after that replaces the pack and its dictionary. |
|
| 1900 | + | let packs = |dir: &Path| { |
|
| 1901 | + | let mut v: Vec<String> = fs::read_dir(dir) |
|
| 1902 | + | .unwrap() |
|
| 1903 | + | .flatten() |
|
| 1904 | + | .map(|e| e.file_name().to_string_lossy().into_owned()) |
|
| 1905 | + | .filter(|n| n.starts_with("pack-")) |
|
| 1906 | + | .collect(); |
|
| 1907 | + | v.sort(); |
|
| 1908 | + | v |
|
| 1909 | + | }; |
|
| 1910 | + | // With nothing new, a repack leaves that pack as it is: it would |
|
| 1911 | + | // only write the same again. |
|
| 1912 | + | let before = packs(dir.path()); |
|
| 1913 | + | let (report, merged) = again.repack_all(REPACK_ZSTD_LEVEL).unwrap(); |
|
| 1914 | + | assert_eq!((report.objects, merged), (0, 0)); |
|
| 1915 | + | assert_eq!(packs(dir.path()), before); |
|
| 1916 | + | // One new object: a repack replaces the pack and its dictionary. |
|
| 1917 | + | let extra = again.put(ObjectType::Chunk, b"one more\n").unwrap(); |
|
| 1896 | 1918 | again.repack_all(REPACK_ZSTD_LEVEL).unwrap(); |
|
| 1897 | - | let files: Vec<String> = fs::read_dir(dir.path()) |
|
| 1898 | - | .unwrap() |
|
| 1899 | - | .flatten() |
|
| 1900 | - | .map(|e| e.file_name().to_string_lossy().into_owned()) |
|
| 1901 | - | .filter(|n| n.starts_with("pack-")) |
|
| 1902 | - | .collect(); |
|
| 1919 | + | let files = packs(dir.path()); |
|
| 1903 | 1920 | assert_eq!(files.len(), 3, "one pack, index and dictionary: {files:?}"); |
|
| 1921 | + | assert_ne!(files, before); |
|
| 1904 | 1922 | assert_eq!(&again.get(&refs[7]).unwrap().1, &bodies[7]); |
|
| 1923 | + | assert_eq!(again.get(&extra).unwrap().1, b"one more\n"); |
|
| 1905 | 1924 | } |
|
| 1906 | 1925 | ||
| 1907 | 1926 | #[test] |
|
| … | 576 unchanged lines |
modified docs/spec.md +1 −1
| … | 545 unchanged lines |
||
| 546 | 546 | ||
| 547 | 547 | ### 13.4 Packing |
|
| 548 | 548 | ||
| 549 | - | An automatic trigger when the loose object count exceeds 4096 writes loose objects into a new pack and merges packs when there are more than 16; `oxy gc` repacks every surviving object into one pack, compressed harder (zstd level 9, against level 3 for everyday writes) and on every core, as `git gc` repacks. Writers SHOULD batch: a snapshot, a fetched bundle, a Git import, or a checkpoint is one write batch, and a batch that grows beyond a small threshold (256 objects or 8 MiB) is written as a single pack with one `fsync` rather than as loose objects, each with its own; smaller batches stay loose so ordinary commits do not multiply packs. Objects of an open batch MUST be readable by the writer before the batch completes. Pack indexes MUST support lookup by ObjectRef prefix and MUST be memory-mappable. A repository with two million objects MUST be openable with under 64 MiB of resident index memory on the Full profile. |
|
| 549 | + | An automatic trigger when the loose object count exceeds 4096 writes loose objects into a new pack and merges packs when there are more than 16; `oxy gc` repacks every surviving object into one pack, compressed harder (zstd level 9, against level 3 for everyday writes) and on every core, as `git gc` repacks. A store that is already one such pack (it has a dictionary) with nothing loose is left as it is: repacking it would write the same pack. Writers SHOULD batch: a snapshot, a fetched bundle, a Git import, or a checkpoint is one write batch, and a batch that grows beyond a small threshold (256 objects or 8 MiB) is written as a single pack with one `fsync` rather than as loose objects, each with its own; smaller batches stay loose so ordinary commits do not multiply packs. Objects of an open batch MUST be readable by the writer before the batch completes. Pack indexes MUST support lookup by ObjectRef prefix and MUST be memory-mappable. A repository with two million objects MUST be openable with under 64 MiB of resident index memory on the Full profile. |
|
| 550 | 550 | ||
| 551 | 551 | The pack and index formats are fixed so that tools can read them without the implementation: |
|
| 552 | 552 | ||
| … | 493 unchanged lines |