# Oxygen Anywhere: stores

Status: built 2026-10-03/04 (oxygen#63): the `Store` trait, directory

and in-memory stores, the bucket layout and the store remote, the S3,

GCS and Azure stores, the CLI (`remote add` settings, `remote check`,

`gc --remote`), and the user guide `docs/strata.md`. This document is

the design and the record of what was decided; the guide is for users.

## 1. What and why

Today an Oxygen repository lives in `.oxy/` on a disk, and a remote is

either another directory or an OxygenHub. This feature makes storage

pluggable. Any **store** that can keep write-once objects and update a

small reference record by compare-and-swap can hold a complete

repository, with no server in between.

The first stores are **Amazon S3** and **Google Cloud Storage**, built

side by side so the code stays portable. Azure Blob and a Git repository

used as a plain store follow on the same interface.

Why Oxygen can do this safely when Git cannot:

- **Objects never change.** Changes, states, blobs, attestations and

  receipts are named by their BLAKE3 hash and written once.

- **Only references are mutable.** Bookmarks, tags and the operation

  head are small, and object stores now offer compare-and-swap on a

  single object.

- **Every read is verified.** Objects are re-hashed on read and changes

  are signed, so a bucket is untrusted storage. Tampering is detected,

  and nobody can forge history, not even the bucket's owner.

Performance is "good enough" for one engineer today. This work is about

where a repository can live, not about speed.

## 2. What users see

Storage must be transparent: a bucket behaves like any other remote, with

no new concepts to learn.

```sh

oxy remote add backup s3://my-bucket/repos/oxygen

oxy push backup

oxy clone gs://team-bucket/metalgear

oxy pull

```

- **URLs:** `s3://BUCKET/PREFIX` and `gs://BUCKET/PREFIX`. The prefix is

  optional, so one bucket can hold many repositories.

- **Credentials come from the standard places**, with no oxy-specific

  configuration:

  - **AWS:** environment variables, `~/.aws` profiles (`AWS_PROFILE`),

    SSO, and instance or container roles;

  - **Google Cloud:** Application Default Credentials (`gcloud auth

    application-default login`), service account files, and workload

    identity.

  - **Overrides:** optional per-remote settings in the repository's

    config (`profile`, `region`, `endpoint`), for example

    `oxy remote add backup s3://b/p --profile work`.

- **Errors are written for people.** Each one says what to do next, for

  example: "no AWS credentials found: run `aws sso login` or set

  AWS_PROFILE", or "the bucket exists but this account cannot write to

  it (s3:PutObject denied on my-bucket/repos/oxygen)".

- **`oxy init --remote s3://…`** creates an empty repository in the

  bucket, and `oxy remote check NAME` reports access, latency and

  what is stored there.

- **Progress, colour and glyphs** follow the CLI style of every other

  command.

- **Several writers are safe.** Two people pushing to the same bucket at

  once is handled the way the operation log handles two processes today

  (§5).

## 3. Architecture

- **The `Store` trait** (`oxy_core::strata`, built 2026-10-03). It has

  two halves:

  - **objects**: `has`, `get`, `get_range`, `put_new` (create only),

    `list` by prefix, and `delete`;

  - **references**: `read_ref() -> (bytes, version)` and

    `write_ref(bytes, expected_version) -> Ok | Conflict`. A version is

    opaque and compared for equality only. Writing identical bytes may

    keep the version, as S3's content-based ETag does; compare-and-swap

    guards against losing a *different* record.

  - Keys are plain relative paths (`[A-Za-z0-9._-]` segments, no `.` or

    `..`), the same rule on every store, so a key can never escape a

    directory store.

  - Implementations: `LocalStore` (a directory: create-only through a

    hard link, references under a lock file beside them, so processes on

    one machine can share it), `MemoryStore` (tests), then S3 and GCS in

    a crate of their own.

  - `strata::conformance(store, prefix)` exercises any store under a

    probe prefix and leaves it empty: every implementation runs it in its

    tests, and `oxy remote check` runs it against a remote.

  - Since 2026-10-04 (oxygen#68, B1) the repository's own object store

    is behind the trait too: `.oxy/objects` is a `LocalStore` with the

    same keys a bucket would hold (`loose/`, `pack-*`, `nocompress/`,

    `sig/`, `seal/`, `receipt/`), and the pack code reads a packed

    object with one ranged read. A directory store is as fast as the

    files were: the same stats, reads and syncs, with a batch of loose

    objects synced together. What must be on this disk, a pack being

    streamed out and the derived `metas` cache, lives in a scratch

    directory, which for a directory store is the store itself.

- **The `object_store` crate** (Apache Arrow) for S3 and GCS: one API

  for both, mature credential chains, multipart uploads, and conditional

  puts. Azure is in the same crate, so it comes nearly free later.

- **Push and pull talk to the bucket directly** through the same

  negotiation the hub protocol uses: what does the remote have, and what

  is missing? The client sends packs, never loose objects.

- **A local cache** of fetched objects, which is always correct because

  objects never change.

- **The working repository stays local for now.** A bucket is a remote.

  Running a repository *from* a bucket is plan B (oxygen#68): a `Repo`

  opened over a bucket store with a local cache in front, so the hub

  holds no copy of its own and the data lives where its owner chose.

  B1 (the store over the trait) is done; B2 is the caching store and

  the references by compare-and-swap.

- **The operation head on a store** (B2, 2026-10-05): a repository whose

  objects are on any store but this disk's directory keeps its

  operation head in that store's `ops/head` reference record. Recording

  an operation reads the record and its version, and moves it only if

  the version is unchanged. When another process moved it first, the

  two operations are merged by the same rules as on disk, and the write

  is tried again (eight times at most). A repository on this disk keeps

  `.oxy/ops/head` under its lock file, as before, so older oxy builds

  share it safely.

- **The caching store** (B2, 2026-10-05): `strata::CachingStore` puts a

  directory store in front of any other. Reads go through it: a missing

  object is fetched from the remote and kept. Writes go to the remote

  first and then the cache, so the remote always holds everything and

  the cache can be deleted at any time. The cache has a size limit; past

  it the objects used least recently are removed, down to nine tenths of

  the limit. An object larger than an eighth of the limit is not kept:

  ranges of it are read from the remote each time. References are never

  cached, so compare-and-swap is always the remote's.

- **A repository on a store** (B2, 2026-10-05): `oxy init --store URL`

  (or `storage.url` in `.oxy/config`, set at init) keeps the objects and

  the operation head on that store, behind a cache in `.oxy/cache` of

  `storage.cache` bytes (2 GiB by default). `.oxy/objects` then holds

  only scratch: packs being written and derived caches. The cache can be

  deleted at any time. A second repository cannot start on a store that

  already holds one. Measured from the workstation against the real

  services with five small commits (`a_repository_runs_on_every_configured_store`):

  | Store | init | each commit | open with no cache, read the files |

  |---|---|---|---|

  | S3 | 441 ms | 703 ms | 481 ms |

  | GCS | 769 ms | 857 ms | 570 ms |

  | Azure | 664 ms | 511 ms | 639 ms |

  A commit writes its objects one request each and moves the head, so

  its cost is round trips, not bytes; batching a commit's objects into

  one pack is the next step for speed.

## 4. Layout in the bucket (as built, 2026-10-03)

```

PREFIX/

  oxygen.toml                      format version and repository id, written once by the first push

  bundles/<seq>-<hash>.bundle      one per push: changes, signatures, seals, objects, policies

                                   (the wire protocol's bundle encoding), written once, named by content

  refs                             every bookmark with its heads and attestation, every tag, the policy

                                   and its history (the protocol's refs response), moved by compare-and-swap

```

- **Pushing** uploads one bundle, then updates `refs` from the version

  it read. A bundle named in `refs` is complete, because the bundle is

  written before `refs` moves.

- **One `refs` record** so that a push moving several bookmarks is

  atomic. Per-reference records would scale to more writers but lose

  atomic pushes; one engineer and small teams do not need them.

- **Bundles are whole-file transfers.** Each push is one bundle, so a

  pull downloads only the bundles it has not seen. Packs with indexes

  for range reads, as first planned, are an optimisation for later

  (`oxy gc` against a remote merges bundles, and could write a pack).

- **The client does what a hub does.** A store cannot run code, so each

  client keeps a **mirror**, a bare repository of its own under

  `.oxy/remotes/<name>/strata/`, brought level with the store before

  every use: the record read, the bundles it has not seen imported (each

  object re-hashed), the view set from the record. A push runs the hub's

  own checks (`receive_push`: policy, attestations, fast-forward) on the

  mirror, uploads the bundle, then moves `refs`. A fetch verifies each

  bookmark's attestation under the policy, as for any remote. The

  mirror costs a second copy of the objects on disk; sharing them with

  the repository's own store is a later optimisation.

- **The record moved meanwhile:** the mirror is put back to what the

  store says, brought level, and the push's updates are applied over

  what landed (up to four times). Two pushes to different bookmarks

  both get through; one to the same bookmark is told it is stale, as a

  hub says: pull, then push again.

- **`oxy gc --remote NAME`** folds every bundle into one holding what

  the record reaches, and records the superseded ones in a `gc`

  record with the time. They are removed only once seven days have

  passed (`--now` ignores that), so a reader that listed them before

  the fold can still fetch them; a reader that finds one gone skips it,

  since the merged bundle holds the same changes.

- **A directory is a store too:** `oxy remote add usb dir:///media/usb/repo`

  (`dir://`, since a plain path or `file://` is another repository).

  Tests run the whole flow through it with no network.

## 5. Concurrency: compare-and-swap on `refs`

| Provider | Create only | Replace only if unchanged |

|---|---|---|

| S3 | `PutObject` with `If-None-Match: *` | `PutObject` with `If-Match: <ETag>` |

| GCS | `ifGenerationMatch=0` | `ifGenerationMatch=<generation>` |

When a push finds `refs` has moved, it re-reads `refs` and merges by the

same rules as the operation log's merge:

- a bookmark moved by only one side takes that side's value;

- moved differently by both, it keeps both sides' heads, as several

  heads;

- a tag keeps its first value.

It then retries the write. Uploading packs needs no locking: two writers

uploading the same objects is harmless.

**As built (2026-10-04):** the `object_store` crate (0.13.1: 0.14 brought in

`aws-lc-rs`, a C build, and `crc-fast`, whose `cdylib` the macOS cross

build cannot link) carries both

writes as `PutMode::Create` and `PutMode::Update(UpdateVersion { e_tag,

version })`; S3 needs `S3ConditionalPut::ETagMatch` switched on, which

sends `If-None-Match: *` and `If-Match: <ETag>`; GCS uses the generation

(`ifGenerationMatch`); Azure the ETag. A store's `Version` carries both

the ETag and the generation, so the same code serves all three.

Checked against the real services on 2026-10-03/04: every conditional

write refuses as required on S3, GCS and Azure.

## 6. Safety

- **The bucket is untrusted.** Every object is re-hashed on read, as

  local reads are now. Signatures and repository policy are checked on

  fetch exactly as for a hub remote.

- **Bound to one repository.** `oxygen.toml` names the repository id. A

  push to a bucket holding a different repository is refused, and so is

  a clone that finds packs naming another id.

- **Size limits.** The existing 1 GiB object cap, plus limits on index

  and `refs` sizes, so a hostile bucket cannot drive huge allocations.

- **Garbage collection never deletes in place.** `oxy gc` on a remote

  writes the merged pack and moves `refs` first. Packs no longer named

  are deleted only after a grace period (default 7 days), so a slow

  concurrent reader is never left with a missing pack.

- **Least privilege.** The documented IAM and GCS policies need only

  Get, Put, List and Delete under the prefix. A read-only key gives a

  working clone and pull.

- **No secrets in the repository.** Credentials stay in the providers'

  own chains or the OS keychain, never in `.oxy/config`.

## 7. Testing

- **One conformance suite** that every store must pass, run against

  local, S3 and GCS. It covers object round trips, range reads,

  create-only refusals, compare-and-swap conflicts, concurrent pushers

  merging, an interrupted push leaving the remote valid, gc with a

  concurrent reader, and a repository-id mismatch being refused.

- **In CI** (Flow jobs have no network): the in-memory and directory

  stores run the conformance check and the whole push/clone/pull flow.

- **Real buckets** for the cloud stores, run by hand from a machine that

  holds restricted credentials: `crates/oxy-strata/tests/cloud.rs` reads

  `OXY_TEST_S3`, `OXY_TEST_GCS` and `OXY_TEST_AZURE` (each a store URL

  under a test prefix, with `OXY_TEST_S3_PROFILE`,

  `OXY_TEST_GCS_CREDENTIALS` and `OXY_TEST_AZURE_CREDENTIALS` naming the

  identity), runs the conformance check and a repository round trip under

  a prefix of its own, and deletes what it wrote. Unset, the tests say so

  and pass.

- **Emulators** were tried and dropped: MinIO's images are no longer

  published publicly, and `fake-gcs-server` rejects the client's XML

  uploads. The real services are the test.

## 8. Order of work (done)

1. ✔ **The `Store` trait in oxy-core**, with a directory store and an

   in-memory one; the working repository's own store left as it was,

   then (B1, 2026-10-04) put behind the trait with the same layout.

2. ✔ **The bucket layout and the store remote**, tested with no network.

3. ✔ **S3, GCS and Azure stores**, in `crates/oxy-strata`, verified

   against the real services.

4. ✔ **`oxy remote add`, `push`, `pull`, `clone`** for store URLs, with

   the settings on the URL, `remote check`, and friendly errors.

   (`init --remote` was dropped: `remote add` and `push` bind a store.)

5. ✔ **`oxy gc --remote`**, with the grace period, and merge-and-retry

   for concurrent pushes.

6. ✔ **Documentation:** `docs/strata.md`, the README section, and this

   record.

Later, on the same trait:

- Azure Blob storage;

- a Git repository used as a plain store, so `oxy push` works to

  GitHub or GitLab;

- the hub attaching a repository from a customer's bucket ("bring your

  own storage", see oxygenhub `docs/scaling.md`);

- couriers: signed, encrypted bundles shared by email, Slack, Matrix

  and the like (a separate design);

- a virtual working copy mounted with FUSE (from OxygenFlow's

  `flow-casfs`).

## 9. Open questions

- Should `oxy clone s3://…` set the bucket as `origin`, or prompt for a

  name? Proposed: `origin`, as for any URL.

- Does a bucket remote keep its own operation log, or only `refs`?

  Proposed: only `refs` at first. The local operation log already

  records what each push did.

- Should packs be encrypted on the client, so the provider only ever

  stores ciphertext, beyond the provider's own encryption? Proposed:

  later, as an option. It does not change the layout.