|
|
… |
519 unchanged lines |
| 520 |
520 |
|
pack-<id>.pack append-only pack file |
| 521 |
521 |
|
pack-<id>.idx sorted (ObjectRef → offset, length, type) index |
| 522 |
522 |
|
pack-<id>.dict the pack's zstd dictionary, when a full repack trained one |
| 523 |
|
- |
pack-retired packs and loose objects a repack replaced, kept for readers (§13.4) |
|
523 |
+ |
pack-retired packs, loose objects and entry lists a repack or fold replaced, kept for readers (§13.4) |
|
524 |
+ |
pack-entries journal of signatures and index entries written since the last fold (§13.7) |
|
525 |
+ |
pack-<id>.entries an entry list: every signature and index entry, as a fold found them (§13.7) |
|
526 |
+ |
pack-<id>.entries-part an entry list of journal lines a writer folded (§13.7) |
|
527 |
+ |
pack-legacy-folded written once a collection has folded the older one-list keys (§13.7) |
| 524 |
528 |
|
loose/ objects written since the last pack, as header || body |
| 525 |
|
- |
sigs/ signature lists keyed by the ObjectRef they cover (§5.4) |
|
529 |
+ |
nocompress/ chunks packs keep raw (§13.3) |
|
530 |
+ |
sig/ one signature per signer, keyed by the ObjectRef it covers (§5.4, §13.7) |
|
531 |
+ |
seal/ receipt/ approval/ cert/ accepted/ |
|
532 |
+ |
index entries, one key each, the value in the key (§13.7) |
| 526 |
533 |
|
ops/ operation objects and op heads |
| 527 |
534 |
|
refs/ current view cache (derived from ops; rebuildable) |
| 528 |
535 |
|
workspaces/<id>/ snapshot cache, fsmonitor cookie |
|
|
… |
50 unchanged lines |
| 579 |
586 |
|
|
| 580 |
587 |
|
Every update that must survive a crash replaces a file atomically: write a uniquely named temporary file in the same directory, `fsync` it, rename it over the target, and `fsync` the directory. Loose objects and packs are written the same way, a pack before its index. The operation head (`.oxy/ops/head`) is the commit point of every command: before it moves, every object the new operation names MUST be durable, including objects still held by an open write batch (§13.4), which is flushed first. A crash therefore leaves the repository at the operation before or after the interrupted command, never between; stray temporary files may remain and are ignored. The reference implementation's tests stop `commit` and `push` at each durable step and require both repositories to open, pass `oxy fsck` (§21.1) and accept the same command again. |
| 581 |
588 |
|
|
|
589 |
+ |
### 13.7 Signatures and index entries |
|
590 |
+ |
|
|
591 |
+ |
Signatures (§5.4) and the indexes kept beside the objects are stored one key per entry, with the value in the key where there is no other. `sig/<hh>/<rest>/<signer>` holds one signer's signature over the object `<hh><rest>`; a new signature by the same signer replaces it. `seal/`, `receipt/`, `approval/`, `cert/` and `accepted/` hold empty `<hh>/<rest>/<value>` keys: the seal attestations covering a change, the receipts and approvals on a state, the certificate kept for a short-lived key, and a reference's accepted attestation. Each entry is its own write-once key, so writers sharing a store never lose each other's entries. Stores written before this layout kept one list per object (`sigs/`, `seals/`, `receipts/`, `approvals/`, `certs/`, `accepted-lists/`); readers still read these until a collection folds them (below). |
|
592 |
+ |
|
|
593 |
+ |
To find an object's entries in this layout, a reader lists its prefix. On a directory that is one directory read. On a bucket it is one LIST request per lookup, and most find nothing, because most objects are unsigned (oxygen#116). So a store that is not this disk's directory also keeps its entries as lists that a reader reads once per open. Neither changes the per-object keys, which every writer still writes first, so older readers keep working. |
|
594 |
+ |
|
|
595 |
+ |
- **The journal**, `pack-entries`: a reference record, updated by compare-and-swap as `pack-retired` is, of lines `<folded at, wall ms; 0 until folded> <entry key>`. After writing an entry's own key, a writer adds the key to the journal. A write batch (§13.4) adds all of its entries in one update when it is flushed, before the operation head moves. A writer that loses the race reads the record again and retries (up to 32 times), so concurrent writers never lose each other's lines. |
|
596 |
+ |
- **Entry lists**: zstd-compressed UTF-8 text, written once: the line `oxy-entries 1`, then one entry key per line, sorted. `pack-<id>.entries` is **complete**: it holds every entry the store held when it was written. `pack-<id>.entries-part` is **partial**: it holds the journal lines a writer folded out. The lists are named under `pack-` so that the pack listing every open makes already finds them. They are not tied to a pack: a repack by an implementation that does not know them cannot hide them, and a collection with nothing to repack still folds. |
|
597 |
+ |
- **Folding.** `oxy gc` (and a forge's collection) folds **completely**. It reads the journal, lists every entry prefix, writes a complete list of both, retires every list it read (in `pack-retired`, like a pack: a collection deletes it an hour later), and marks the journal lines it read as folded. When nothing changed since the last complete list, no list is written. A collection that deletes an object deletes its entry keys and their journal lines too, and the next complete list leaves them out. A writer whose update leaves more than 1000 unfolded lines folds them into a partial list and marks them. The next update after a folded line's hour (§13.4) drops it. Until then, a reader whose pack listing predates the fold still finds every entry, in the old lists or in the journal. Lines added while a fold runs are not marked, so they stay. |
|
598 |
+ |
- **Reading.** A store is **folded** when its live lists (listed and not retired) include a complete one. On its first lookup, a handle on a folded store reads every live list and the journal, folded lines included, and answers every lookup from that set. It adds what it writes itself, and it never lists the per-object prefixes or reads the older lists. A signature's bytes are still read from its own key, which a cache may keep, since that key is written once. A handle reads the lists and the journal again when it loads its packs again (§13.4) or is refreshed. On a store that is not folded (no collection has folded it since writers kept a journal, or another implementation's collection retired its lists), a lookup lists the per-object prefix as before. |
|
599 |
+ |
- **Older lists.** Before listing the entry prefixes, a complete fold copies every older one-list key's contents into per-object entries, keeping an entry already there. It then writes the marker `pack-legacy-folded` once. A reader whose pack listing shows the marker does not read the older lists. Nothing writes them any more. |
|
600 |
+ |
|
|
601 |
+ |
**Older writers.** Writers that predate the journal (oxy 0.4.6 and before) write only the per-object keys. Readers of a folded store miss what such a writer adds until the next complete fold picks it up. A store becomes folded at its first complete fold. So the operator of a repository kept in a bucket first upgrades every writer to it (clients with `storage.url`, runners, the forge), then runs `oxy gc` there, or lets the forge's collection run. After that, a lookup costs no request apart from reading a signature's own key. A forge that collects nightly picks up an older writer's entries within a day. A repository's own directory (`.oxy/objects`) keeps no journal or lists: reading a directory costs no request, so its lookups list the per-object prefixes as before. |
|
602 |
+ |
|
| 582 |
603 |
|
## 14. Identity, signatures, and trust policy |
| 583 |
604 |
|
|
| 584 |
605 |
|
### 14.1 Algorithms |
|
|
… |
261 unchanged lines |
| 846 |
867 |
|
|
| 847 |
868 |
|
## 21. Garbage collection |
| 848 |
869 |
|
|
| 849 |
|
- |
An object is **reachable** if it is referenced, directly or transitively, from any bookmark, tag, policy, the current operation heads, any operation newer than `core.op-retention`, any workspace's current working-copy change, the working-copy changes of operations newer than `core.snapshot-retention` (default 7 days; `oxy gc --expire-snapshots` keeps none but the current ones, oxygen#98), or any checkpoint of a reachable state. A file that was only ever in a working copy, never committed, so goes once the snapshots that held it expire. `oxy gc` first deletes the packs and loose objects earlier repacks retired over an hour ago (§13.4), then deletes unreachable objects that are older than `core.gc-grace` (default 14 days, to protect concurrent writers) and repacks; what this repack replaces is retired in turn, for a later collection. Tombstone atoms are never collected independently of their change. Divergent changes are reachable if any of their variants is. A change's text blobs and their chunks are reachable with it (§7.4). A seal is reachable if any change it covers is reachable, and reference attestations are reachable while they are the latest for their reference or are within `core.op-retention`. |
|
870 |
+ |
An object is **reachable** if it is referenced, directly or transitively, from any bookmark, tag, policy, the current operation heads, any operation newer than `core.op-retention`, any workspace's current working-copy change, the working-copy changes of operations newer than `core.snapshot-retention` (default 7 days; `oxy gc --expire-snapshots` keeps none but the current ones, oxygen#98), or any checkpoint of a reachable state. A file that was only ever in a working copy, never committed, so goes once the snapshots that held it expire. `oxy gc` first deletes the packs and loose objects earlier repacks retired over an hour ago (§13.4), then deletes unreachable objects that are older than `core.gc-grace` (default 14 days, to protect concurrent writers) and repacks; what this repack replaces is retired in turn, for a later collection. On a store that is not this disk's directory it then folds the signatures and index entries completely (§13.7). Tombstone atoms are never collected independently of their change. Divergent changes are reachable if any of their variants is. A change's text blobs and their chunks are reachable with it (§7.4). A seal is reachable if any change it covers is reachable, and reference attestations are reachable while they are the latest for their reference or are within `core.op-retention`. |
| 850 |
871 |
|
|
| 851 |
872 |
|
### 21.1 Integrity check |
| 852 |
873 |
|
|
|
|
… |
213 unchanged lines |