OXYGENHUB

Storage

Andy McCall edited this page · 1 revision

Storage

Everything Oxygen stores is an object: an immutable byte string with a type, named by the BLAKE3 hash of its contents. Every read is hashed again, so damaged or tampered data is caught.

flowchart TD
    Op[Operation] --> View[View: bookmarks, workspaces]
    View --> State[State: heads]
    State --> Change[Change]
    Change --> Change2[Earlier changes it depends on]
    Change --> TB[Text blobs: large inserts]
    Change --> Blob[Blob: an opaque file]
    TB --> Chunk[Chunks]
    Blob --> Chunk
    Change -.signed by.-> Id[Identity]

On disk

.oxy/
  format            repository format version
  config            this repository's settings (TOML)
  objects/
    pack-<id>.pack  objects packed together, each compressed
    pack-<id>.idx   sorted index: hash → offset, length, type
    loose/          objects written since the last pack
    sig/            signatures, kept beside the objects they cover
  ops/head          the current operation (the commit point of every command)
  workspaces/<id>/  the snapshot manifest and the working-copy lock

Chunking: large files deduplicate

File content is split into chunks with FastCDC, a content-defined chunker: chunk boundaries depend on the content, not on offsets. Insert a byte near the start of a large file and only the chunks around the edit change.

flowchart LR
    F1["file v1"] --> c1[chunk A] & c2[chunk B] & c3[chunk C]
    F2["file v2: edit inside B"] --> c1b[chunk A] & c2b[chunk B'] & c3b[chunk C]
    c1 -.same object.- c1b
    c3 -.same object.- c3b

Chunks are 32 KiB to 256 KiB, 64 KiB on average. A blob lists a file's chunks; small files are a single chunk.

Packs

New objects are written loose, one file each. When there are more than 4,096 loose objects they are packed into a single file with an index, each object compressed with zstd. oxy gc repacks everything into one pack and compresses harder. A batch of many objects (a snapshot, a fetch, a Git import) goes straight into a pack with one sync.

Crash safety

Every file that must survive a crash is written to a temporary file, synced, and renamed into place. The operation head moves last, after every object it names is on disk. A crash leaves the repository at the command before or after, never in between; oxy fsck checks it.

Checkpoints

Loading a state by replaying every change gets slower as history grows. A checkpoint is a saved, reproducible materialisation of a state. Oxygen makes one every 256 changes along a bookmark, and loading replays only the changes since.

Where the objects live need not be this disk: see Strata.