OXYGENHUB

Backlog 4: S6, clone and P1 logged (refs #41) 72215f29df4c

Author
Andy McCall @andymccall <[email protected]>
When
· 2026-09-30 23:42 UTC
Signature
✔ signed by Andy McCall
Hash
72215f29df4cfa98bf7f5abda0c0e25d962bd916d7ae076ecdca0870bd538bfb
Depends on
61e9ad6ddc2d e82591ec33a4
Depended on by
7f75b41c107d
In bookmarks
agents-in-readme bare-hashes carried-not-removed carry-identical doctor-trust identical-sides keyless-tags main merge-remote-bookmark only-help-103 paths-for-resolutions-and-amend push-notes show-stat subrepo-pin-state switch-says
In tags
v0.10.0 v0.11.0 v0.2.0 v0.3.0 v0.3.1 v0.4.0 v0.4.1 v0.4.2 v0.4.3 v0.4.4 v0.4.5 v0.4.6 v0.5.0 v0.5.1 v0.6.0 v0.7.0 v0.8.0 v0.9.0
Edits
3 atoms, 0 tree operations, 1 file

Browse the tree at this change

hide whitespace changes
modified docs/backlog-overnight-4.md +3 −1
… 
87 unchanged lines
88 88 
- [ ] S5 **Checkout by comparing indexes.** Compare the two states' tree
89 89 
  indexes by rendered content hash and write only the files that differ.
90 90 
  Target: at or under Git.
91 - 
- [ ] S6 **fsck at or under Git**: parallel object checking and a cheaper
91 + 
- [x] S6 **fsck at or under Git**: parallel object checking and a cheaper
92 92 
  state pass.
93 93 
- [ ] S7 **The first commit ahead of Git, under 2 GB**: stream reading,
94 94 
  hashing and storing a batch of files at a time, so the whole tree's text
… 
25 unchanged lines
120 120 
- 2026-09-30 **S1, first round** (commits c9dca9ed to 077e899b; table in `docs/benchmarks.md`). A lister of our own replaces the ignore crate's walker (listing 0.5 s to 0.18 s); the manifest is a flat binary file with hashes and graph keys as bytes; the state index keeps each file's history encoded until used (a writer bug I made on the way was caught only on Earth, since a failed index read falls back to a full load: test builds now read every index written back); maps are built in path order; mimalloc is the allocator. Clean status is now ahead of Git (0.18 s against 0.22 s), the first commit too (24.4 s against 36.2 s), and gc (29.9 s against 70.3 s). One-file status is 0.47 s against 0.22 s. Log (S3, first part): metadata from the state index and a store cache, signature marks without reading changes: 0.20 s, of which 9 ms is the log. For Andy: `oxy log` checks the working copy first (about 0.18 s here); Git's log does not look at it. Dropping that check would make log about 20 ms, but the working-copy line would then show the last snapshot's state.
121 121 

    

  
    

      122
      122
       
      
- 2026-10-01 00:00 **Second round** (6359f779 to 81a5bdd7; table in `docs/benchmarks.md`). Commit 0.67 s (from 0.96 s): the state index is brought up to date by replaying and rewritten only 16 changes behind; keys only for changed paths; attributes from the snapshot rather than a tree walk; the manifest checksummed and written without fsync; and a fix: after a status had seen an earlier edit, a commit kept that edit's stat in the manifest, so every later status took a full snapshot. Blame 0.16 s (from 0.59 s; Git 0.07 s), log 0.18 s, checkout 0.74 s each way (from 1.4 s) with two state indexes kept per workspace, and status right after a checkout that wrote 1 000 files 0.18 s (from 1.3 s). **R1** done: `oxy rebase --into-working-copy` (closes #42).

    

  
    

      
      123
      +
      

    

  
    

      
      124
      +
      
- 2026-10-01 00:40 **S6 done, clone level, P1 partly** (3287e4ff to cc1f1723). fsck 8.2 to 8.7 s (from 13.6 s; Git 12.6 s): objects checked a chunk at a time on every core, graphs built with their lines in parallel, and the last four change encodings kept rather than one (a batch that took inline text from the Linux tip and a later change in turn read the tip again at each switch). Local clone 16.8 to 17.6 s (from 32 s; Git 17.0 s): files added once and never edited are written from their text without a graph (test builds compare with the rendering), new files are written in parallel, rewritten ones one at a time (six times slower in parallel on ext4). A faster local clone would copy the source's packs as Git's `--local` does; that skips the import's checks and brings every object of the source, so it waits for Andy. **P1**: a trained zstd dictionary per pack: pack 293.5 MB (from 301 MB; Git's 295.5 MB), gc 33.7 s (Git 70 s). The whole `.oxy` stays larger than `.git` by the state index (52 MB, a cache; compressing it would slow every load).