# Oxygen Anywhere: stores

Status: design, 2026-10-03. Nothing here is built yet. Next to implement.

## 1. What and why

Today an Oxygen repository lives in `.oxy/` on a disk, and a remote is
either another directory or an OxygenHub. This feature makes storage
pluggable. Any **store** that can keep write-once objects and update a
small reference record by compare-and-swap can hold a complete
repository, with no server in between.

The first stores are **Amazon S3** and **Google Cloud Storage**, built
side by side so the code stays portable. Azure Blob and a Git repository
used as a plain store follow on the same interface.

Why Oxygen can do this safely when Git cannot:

- **Objects never change.** Changes, states, blobs, attestations and
  receipts are named by their BLAKE3 hash and written once.
- **Only references are mutable.** Bookmarks, tags and the operation
  head are small, and object stores now offer compare-and-swap on a
  single object.
- **Every read is verified.** Objects are re-hashed on read and changes
  are signed, so a bucket is untrusted storage. Tampering is detected,
  and nobody can forge history, not even the bucket's owner.

Performance is "good enough" for one engineer today. This work is about
where a repository can live, not about speed.

## 2. What users see

Storage must be transparent: a bucket behaves like any other remote, with
no new concepts to learn.

```sh
oxy remote add backup s3://my-bucket/repos/oxygen
oxy push backup
oxy clone gs://team-bucket/metalgear
oxy pull
```

- **URLs:** `s3://BUCKET/PREFIX` and `gs://BUCKET/PREFIX`. The prefix is
  optional, so one bucket can hold many repositories.
- **Credentials come from the standard places**, with no oxy-specific
  configuration:
  - **AWS:** environment variables, `~/.aws` profiles (`AWS_PROFILE`),
    SSO, and instance or container roles;
  - **Google Cloud:** Application Default Credentials (`gcloud auth
    application-default login`), service account files, and workload
    identity.
  - **Overrides:** optional per-remote settings in the repository's
    config (`profile`, `region`, `endpoint`), for example
    `oxy remote add backup s3://b/p --profile work`.
- **Errors are written for people.** Each one says what to do next, for
  example: "no AWS credentials found: run `aws sso login` or set
  AWS_PROFILE", or "the bucket exists but this account cannot write to
  it (s3:PutObject denied on my-bucket/repos/oxygen)".
- **`oxy init --remote s3://…`** creates an empty repository in the
  bucket, and `oxy remote check NAME` reports access, latency and
  what is stored there.
- **Progress, colour and glyphs** follow the CLI style of every other
  command.
- **Several writers are safe.** Two people pushing to the same bucket at
  once is handled the way the operation log handles two processes today
  (§5).

## 3. Architecture

- **The `Store` trait** (`oxy_core::strata`, built 2026-10-03). It has
  two halves:
  - **objects**: `has`, `get`, `get_range`, `put_new` (create only),
    `list` by prefix, and `delete`;
  - **references**: `read_ref() -> (bytes, version)` and
    `write_ref(bytes, expected_version) -> Ok | Conflict`. A version is
    opaque and compared for equality only. Writing identical bytes may
    keep the version, as S3's content-based ETag does; compare-and-swap
    guards against losing a *different* record.
  - Keys are plain relative paths (`[A-Za-z0-9._-]` segments, no `.` or
    `..`), the same rule on every store, so a key can never escape a
    directory store.
  - Implementations: `LocalStore` (a directory: create-only through a
    hard link, references under a lock file beside them, so processes on
    one machine can share it), `MemoryStore` (tests), then S3 and GCS in
    a crate of their own.
  - `strata::conformance(store, prefix)` exercises any store under a
    probe prefix and leaves it empty: every implementation runs it in its
    tests, and `oxy remote check` runs it against a remote.
  - The working repository's own `.oxy/` store is **not** behind the
    trait: it is read and written directly, as before, so nothing on the
    local path changes or slows. A store is what a remote is kept on.
- **The `object_store` crate** (Apache Arrow) for S3 and GCS: one API
  for both, mature credential chains, multipart uploads, and conditional
  puts. Azure is in the same crate, so it comes nearly free later.
- **Push and pull talk to the bucket directly** through the same
  negotiation the hub protocol uses: what does the remote have, and what
  is missing? The client sends packs, never loose objects.
- **A local cache** of fetched objects, which is always correct because
  objects never change.
- **The working repository stays local.** A bucket is a remote. Running
  a repository *from* a bucket (no local `.oxy/`) is out of scope here;
  it is what the virtual working copy (§8) is for.

## 4. Layout in the bucket

```
PREFIX/
  oxygen.toml          format version and repository id, written once
  packs/<id>.pack      immutable, zstd-compressed objects
  packs/<id>.idx       sorted index of the pack (read with range GETs)
  refs                 bookmarks, tags and the remote's operation head: one object, updated by compare-and-swap
  receipts/, attest/   only if they are kept outside packs
```

- **Pushing** uploads one pack and its index, then updates `refs`.
- **Packs never change.** A pack named in `refs` is complete, because
  the pack is written before `refs` moves.
- **One `refs` object** so that a push moving several bookmarks is
  atomic. Per-reference objects would scale to more writers but lose
  atomic pushes; one engineer and small teams do not need them.
- **Many small packs from many pushes** are merged by `oxy gc` against
  the remote, as `merge_packs` does locally.

## 5. Concurrency: compare-and-swap on `refs`

| Provider | Create only | Replace only if unchanged |
|---|---|---|
| S3 | `PutObject` with `If-None-Match: *` | `PutObject` with `If-Match: <ETag>` |
| GCS | `ifGenerationMatch=0` | `ifGenerationMatch=<generation>` |

When a push finds `refs` has moved, it re-reads `refs` and merges by the
same rules as the operation log's merge:
- a bookmark moved by only one side takes that side's value;
- moved differently by both, it keeps both sides' heads, as several
  heads;
- a tag keeps its first value.

It then retries the write. Uploading packs needs no locking: two writers
uploading the same objects is harmless.

Before building on it, check each provider's current documentation for
conditional writes: S3 added `If-Match` on `PutObject` in 2024, and this
design depends on it.

## 6. Safety

- **The bucket is untrusted.** Every object is re-hashed on read, as
  local reads are now. Signatures and repository policy are checked on
  fetch exactly as for a hub remote.
- **Bound to one repository.** `oxygen.toml` names the repository id. A
  push to a bucket holding a different repository is refused, and so is
  a clone that finds packs naming another id.
- **Size limits.** The existing 1 GiB object cap, plus limits on index
  and `refs` sizes, so a hostile bucket cannot drive huge allocations.
- **Garbage collection never deletes in place.** `oxy gc` on a remote
  writes the merged pack and moves `refs` first. Packs no longer named
  are deleted only after a grace period (default 7 days), so a slow
  concurrent reader is never left with a missing pack.
- **Least privilege.** The documented IAM and GCS policies need only
  Get, Put, List and Delete under the prefix. A read-only key gives a
  working clone and pull.
- **No secrets in the repository.** Credentials stay in the providers'
  own chains or the OS keychain, never in `.oxy/config`.

## 7. Testing

- **One conformance suite** that every store must pass, run against
  local, S3 and GCS. It covers object round trips, range reads,
  create-only refusals, compare-and-swap conflicts, concurrent pushers
  merging, an interrupted push leaving the remote valid, gc with a
  concurrent reader, and a repository-id mismatch being refused.
- **Emulators in CI** (Flow jobs have no network):
  - MinIO for S3, if its conditional writes match S3's, otherwise
    LocalStack;
  - `fake-gcs-server` for GCS.
- **Real buckets** (Andy's AWS and GCP accounts) for a nightly or
  manual test run. It needs network access, so it runs outside the
  isolated tier, with credentials scoped to a test prefix.

## 8. Order of work

1. **The `Store` trait in oxy-core**, with the local store moved behind
   it and no change in behaviour, so every existing test passes.
2. **S3 and GCS stores together**, with the conformance suite and the
   emulators.
3. **`oxy remote add`, `push`, `pull`, `clone` and `init --remote`**
   for bucket URLs, with the friendly errors and `remote check`.
4. **`oxy gc` against a bucket**, with the grace period.
5. **Documentation:** a user guide page, sample IAM and GCS policies,
   and a README section.

Later, on the same trait:
- Azure Blob storage;
- a Git repository used as a plain store, so `oxy push` works to
  GitHub or GitLab;
- the hub attaching a repository from a customer's bucket ("bring your
  own storage", see oxygenhub `docs/scaling.md`);
- couriers: signed, encrypted bundles shared by email, Slack, Matrix
  and the like (a separate design);
- a virtual working copy mounted with FUSE (from OxygenFlow's
  `flow-casfs`).

## 9. Open questions

- Should `oxy clone s3://…` set the bucket as `origin`, or prompt for a
  name? Proposed: `origin`, as for any URL.
- Does a bucket remote keep its own operation log, or only `refs`?
  Proposed: only `refs` at first. The local operation log already
  records what each push did.
- Should packs be encrypted on the client, so the provider only ever
  stores ciphertext, beyond the provider's own encryption? Proposed:
  later, as an option. It does not change the layout.