OXYGENHUB

Big Repositories

Andy McCall edited this page · 1 revision

Big Repositories

Oxygen is built for multi-gigabyte repositories and monorepos. Three tools keep a checkout small, and content-defined chunking keeps storage small (see Storage).

flowchart TB
    Full["Full clone:<br/>all history, all content"]
    Sparse["Sparse: all history,<br/>only some paths checked out"]
    Lazy["Partial: large blobs<br/>fetched on first read"]
    Shallow["Shallow: a checkpoint<br/>plus recent history only"]
    Full --> Sparse
    Full --> Lazy
    Full --> Shallow
KindSettingWhat you keep
Sparsefilter.paths = ["src/", "docs/"]every change, but only those paths on disk; oxy sparse PATH.. changes it
Partialfilter.max-blob = "1MiB"blobs larger than the limit are fetched when first read
Shallowdepth = 100a checkpoint of the heads and the last 100 levels of history; oxy deepen N fetches more

oxy status and oxy commit never need a lazy blob.

Fast status

A manifest remembers each file's size, modification time and content hash, and for text files a key that says the file still matches the committed version. On the next command, a file whose size and time are unchanged is not even read. With no uncommitted work, oxy status answers from the manifest alone without loading any history.

Importing a big Git history

Converting a long Git history means replaying every commit into changes: working out each commit's line edits and rendering every merge. It is a one-off cost, and it is heavy for histories with many long-diverged merges.

Status of the VS Code test (October 2026), the yardstick for this work:

flowchart LR
    A["Memory budget for file graphs<br/>(oxygen #25)"] --> B["Batched merge rendering fixed<br/>(oxygen #71)"]
    B --> C["Full 166 425-commit import<br/>running on Earth"]
    C --> D["Full verify, then push<br/>to OxygenHub"]
  • A 28,637-commit slice now imports in 5.1 GB instead of 16.9 GB, and in 36 minutes instead of 61.
  • A 10,412-commit slice imports in 153 seconds and verifies against Git commit by commit.
  • The full import runs at 3,000 to 6,000 commits an hour on four cores, in 3 to 6 GB of memory.