OXYGENHUB

fsck checks a change's text as its blobs, without decoding it d77e1702d770

fsck's references pass read every change whole, so for the Linux tree's
first change it decoded all 1.8 GB of text only to check dependencies,
the author and the blobs. It now reads the change's skeleton and checks
each text blob and its chunks as it does an opaque file's (the objects
pass has checked every object's hash already). fsck's passes are timed.
On the Linux tip fsck takes 17.2 s and 984 MB (from 24.7 s and 2.8 GB).
A test checks a missing text blob is still named.

refs #35
Author
Andy McCall @andymccall <[email protected]>
When
· 2026-09-30 03:13 UTC
Signature
✔ signed by Andy McCall
Hash
d77e1702d7705eb25261a1ad1021fb2ac63478312ab41c5f050479de323858fe
Depends on
1d360c91a289 387335ff7f4d 982864068a59 a87c7b9d563a da652d6392d8
Depended on by
648189388f1a a07f60a37fb2 75b6c8665219
In bookmarks
agents-in-readme bare-hashes carried-not-removed carry-identical doctor-trust identical-sides keyless-tags main merge-remote-bookmark only-help-103 paths-for-resolutions-and-amend push-notes show-stat subrepo-pin-state switch-says
In tags
v0.1.0 v0.10.0 v0.11.0 v0.2.0 v0.3.0 v0.3.1 v0.4.0 v0.4.1 v0.4.2 v0.4.3 v0.4.4 v0.4.5 v0.4.6 v0.5.0 v0.5.1 v0.6.0 v0.7.0 v0.8.0 v0.9.0
Edits
6 atoms, 0 tree operations, 3 files

Browse the tree at this change

hide whitespace changes
modified crates/oxy-core/src/repo/fsck.rs +27 −5
… 
109 unchanged lines
110 110 
    /// Check the whole repository; see the module documentation.
111 111 
    pub fn fsck(&self) -> Result<FsckReport> {
112 112 
        let mut rep = FsckReport::default();
113 - 
        let bad = self.fsck_objects(&mut rep)?;
114 - 
        self.fsck_ops(&mut rep, &bad);
115 - 
        self.fsck_references(&mut rep, &bad)?;
116 - 
        self.fsck_derived(&mut rep)?;
113 + 
        let bad = {
114 + 
            let _t = crate::timing::phase("fsck: objects");
115 + 
            self.fsck_objects(&mut rep)?
116 + 
        };
117 + 
        {
118 + 
            let _t = crate::timing::phase("fsck: operations");
119 + 
            self.fsck_ops(&mut rep, &bad);
120 + 
        }
121 + 
        {
122 + 
            let _t = crate::timing::phase("fsck: references");
123 + 
            self.fsck_references(&mut rep, &bad)?;
124 + 
        }
125 + 
        {
126 + 
            let _t = crate::timing::phase("fsck: states and checkpoints");
127 + 
            self.fsck_derived(&mut rep)?;
128 + 
        }
117 129 
        rep.problems.sort_by_key(|p| std::cmp::Reverse(p.severity));
118 130 
        Ok(rep)
119 131 
    }
… 
154 unchanged lines
274 286 
            if !seen.insert(h) {
275 287 
                continue;
276 288 
            }
277 - 
            let c = match self.store.get_change(&h) {
289 + 
            // The skeleton: the text is checked as the blobs it is stored
290 + 
            // in, not decoded (the objects pass checked every object).
291 + 
            let (c, texts) = match self.store.get_change_lazy(&h) {
278 292 
                Ok(c) => c,
279 293 
                Err(StoreError::NotFound(_)) if boundary.contains(&h) => continue,
280 294 
                Err(StoreError::NotFound(_)) => {
… 
15 unchanged lines
296 310 
                    Some(h),
297 311 
                    format!("author identity {} is not stored", c.author.short(12)),
298 312 
                );
313 + 
            }
314 + 
            for t in texts {
315 + 
                if let crate::objects::TextRef::Blob(b, _, _) = t {
316 + 
                    if blobs_seen.insert(b) {
317 + 
                        // A change cannot be read without its text: never partial.
318 + 
                        self.fsck_blob(rep, b, &label, false, bad);
319 + 
                    }
320 + 
                }
299 321 
            }
300 322 
            for e in &c.edits {
301 323 
                let Edit::Replace { old, new, .. } = e else {
… 
186 unchanged lines
modified crates/oxy-core/src/repo/tests.rs +26 −0
… 
6172 unchanged lines
6173 6173 
    assert!(view.conflicts.is_empty(), "{:?}", view.conflicts);
6174 6174 
    assert!(view.entries.contains_key("d/new.txt"));
6175 6175 
}
6176 + 

    

  
    

      
      6177
      +
      
/// fsck checks a change's text as the blobs it is stored in, without

    

  
    

      
      6178
      +
      
/// decoding it: a missing text blob is an error named for the change.

    

  
    

      
      6179
      +
      
#[test]

    

  
    

      
      6180
      +
      
fn fsck_names_a_missing_text_blob() {

    

  
    

      
      6181
      +
      
    hermetic();

    

  
    

      
      6182
      +
      
    let dir = tempfile::tempdir().unwrap();

    

  
    

      
      6183
      +
      
    let root = dir.path();

    

  
    

      
      6184
      +
      
    let repo = Repo::init(root).unwrap();

    

  
    

      
      6185
      +
      
    let big: String = (0..3000).map(|i| format!("line {i}\n")).collect();

    

  
    

      
      6186
      +
      
    write(root, "big.txt", big.as_bytes());

    

  
    

      
      6187
      +
      
    let h = repo.commit("big", &[]).unwrap();

    

  
    

      
      6188
      +
      
    assert_eq!(repo.fsck().unwrap().errors(), 0);

    

  
    

      
      6189
      +
      
    let (_, texts) = repo.store().get_change_lazy(&h).unwrap();

    

  
    

      
      6190
      +
      
    let Some(crate::objects::TextRef::Blob(b, _, _)) = texts.first().copied() else {

    

  
    

      
      6191
      +
      
        panic!("a stored text")

    

  
    

      
      6192
      +
      
    };

    

  
    

      
      6193
      +
      
    fs::remove_file(repo.store().path_for(&b)).unwrap();

    

  
    

      
      6194
      +
      
    let rep = repo.fsck().unwrap();

    

  
    

      
      6195
      +
      
    assert!(rep.errors() > 0);

    

  
    

      
      6196
      +
      
    assert!(

    

  
    

      
      6197
      +
      
        rep.problems.iter().any(|p| p.what.contains(&h.short(12))),

    

  
    

      
      6198
      +
      
        "{:?}",

    

  
    

      
      6199
      +
      
        rep.problems

    

  
    

      
      6200
      +
      
    );

    

  
    

      
      6201
      +
      
}

    

  
  

  





  
    modified
    docs/benchmarks.md
    +10 −0
  
  
  

  
    

      
      
      …
      
486 unchanged lines

    

  
    

      487
      487
       
      
on every core: 357 MB (349 MB pack) in 52 s. The text blob framing is the

    

  
    

      488
      488
       
      
next step (a change format version 3 with raw text blobs).

    

  
    

      489
      489
       
      

    

  
    

      
      490
      +
      
### fsck (2026-09-30)

    

  
    

      
      491
      +
      

    

  
    

      
      492
      +
      
On the Linux tip in change format 3, `oxy fsck` took 24.7 s and 2.8 GB:

    

  
    

      
      493
      +
      
its references pass read every change whole, the first one's 1.8 GB of

    

  
    

      
      494
      +
      
text into memory, to check dependencies, authors and blobs. It now reads

    

  
    

      
      495
      +
      
a change's skeleton and checks its text as the blobs it is stored in (the

    

  
    

      
      496
      +
      
objects pass has checked every object already): 17.2 s and 984 MB, the

    

  
    

      
      497
      +
      
references pass 1.7 s. Phases are timed (`OXY_TIMINGS`): objects 5.1 s,

    

  
    

      
      498
      +
      
references 1.7 s, states and checkpoints 10.4 s.

    

  
    

      
      499
      +