# Oxygen Anywhere: stores

Status: design, 2026-10-03. Nothing here is built yet. Next to implement.

## 1. What and why

Today an Oxygen repository lives in `.oxy/` on a disk, and a remote is
either another directory or an OxygenHub. This feature makes storage
pluggable. Any **store** that can keep write-once objects and update a
small reference record by compare-and-swap can hold a complete
repository, with no server in between.

The first stores are **Amazon S3** and **Google Cloud Storage**, built
side by side so the code stays portable. Azure Blob and a Git repository
used as a plain store follow on the same interface.

Why Oxygen can do this safely when Git cannot:

- **Objects never change.** Changes, states, blobs, attestations and
  receipts are named by their BLAKE3 hash and written once.
- **Only references are mutable.** Bookmarks, tags and the operation
  head are small, and object stores now offer compare-and-swap on a
  single object.
- **Every read is verified.** Objects are re-hashed on read and changes
  are signed, so a bucket is untrusted storage. Tampering is detected,
  and nobody can forge history, not even the bucket's owner.

Performance is "good enough" for one engineer today. This work is about
where a repository can live, not about speed.

## 2. What users see

Storage must be transparent: a bucket behaves like any other remote, with
no new concepts to learn.

```sh
oxy remote add backup s3://my-bucket/repos/oxygen
oxy push backup
oxy clone gs://team-bucket/metalgear
oxy pull
```

- **URLs:** `s3://BUCKET/PREFIX` and `gs://BUCKET/PREFIX`. The prefix is
  optional, so one bucket can hold many repositories.
- **Credentials come from the standard places**, with no oxy-specific
  configuration:
  - **AWS:** environment variables, `~/.aws` profiles (`AWS_PROFILE`),
    SSO, and instance or container roles;
  - **Google Cloud:** Application Default Credentials (`gcloud auth
    application-default login`), service account files, and workload
    identity.
  - **Overrides:** optional per-remote settings in the repository's
    config (`profile`, `region`, `endpoint`), for example
    `oxy remote add backup s3://b/p --profile work`.
- **Errors are written for people.** Each one says what to do next, for
  example: "no AWS credentials found: run `aws sso login` or set
  AWS_PROFILE", or "the bucket exists but this account cannot write to
  it (s3:PutObject denied on my-bucket/repos/oxygen)".
- **`oxy init --remote s3://…`** creates an empty repository in the
  bucket, and `oxy remote check NAME` reports access, latency and
  what is stored there.
- **Progress, colour and glyphs** follow the CLI style of every other
  command.
- **Several writers are safe.** Two people pushing to the same bucket at
  once is handled the way the operation log handles two processes today
  (§5).

## 3. Architecture

- **A `Store` trait in oxy-core**, with three implementations: the
  current local `.oxy/` store, S3, and GCS. It has two halves:
  - **objects**: `has`, `get`, `get_range`, `put` (create only), and
    `list` by prefix;
  - **references**: `read() -> (bytes, version)` and
    `write(bytes, expected_version) -> Ok | Conflict`.
- **The `object_store` crate** (Apache Arrow) for S3 and GCS: one API
  for both, mature credential chains, multipart uploads, and conditional
  puts. Azure is in the same crate, so it comes nearly free later.
- **Push and pull talk to the bucket directly** through the same
  negotiation the hub protocol uses: what does the remote have, and what
  is missing? The client sends packs, never loose objects.
- **A local cache** of fetched objects, which is always correct because
  objects never change.
- **The working repository stays local.** A bucket is a remote. Running
  a repository *from* a bucket (no local `.oxy/`) is out of scope here;
  it is what the virtual working copy (§8) is for.

## 4. Layout in the bucket (as built, 2026-10-03)

```
PREFIX/
  oxygen.toml                      format version and repository id, written once by the first push
  bundles/<seq>-<hash>.bundle      one per push: changes, signatures, seals, objects, policies
                                   (the wire protocol's bundle encoding), written once, named by content
  refs                             every bookmark with its heads and attestation, every tag, the policy
                                   and its history (the protocol's refs response), moved by compare-and-swap
```

- **Pushing** uploads one bundle, then updates `refs` from the version
  it read. A bundle named in `refs` is complete, because the bundle is
  written before `refs` moves.
- **One `refs` record** so that a push moving several bookmarks is
  atomic. Per-reference records would scale to more writers but lose
  atomic pushes; one engineer and small teams do not need them.
- **Bundles are whole-file transfers.** Each push is one bundle, so a
  pull downloads only the bundles it has not seen. Packs with indexes
  for range reads, as first planned, are an optimisation for later
  (`oxy gc` against a remote merges bundles, and could write a pack).
- **The client does what a hub does.** A store cannot run code, so each
  client keeps a **mirror**, a bare repository of its own under
  `.oxy/remotes/<name>/strata/`, brought level with the store before
  every use: the record read, the bundles it has not seen imported (each
  object re-hashed), the view set from the record. A push runs the hub's
  own checks (`receive_push`: policy, attestations, fast-forward) on the
  mirror, uploads the bundle, then moves `refs`. A fetch verifies each
  bookmark's attestation under the policy, as for any remote. The
  mirror costs a second copy of the objects on disk; sharing them with
  the repository's own store is a later optimisation.
- **The record moved meanwhile:** the mirror is put back to what the
  store says and the push is reported as a hub reports a stale push:
  pull, then push again. Merging two pushes to different bookmarks
  without a round trip is ST5's work.
- **A directory is a store too:** `oxy remote add usb dir:///media/usb/repo`
  (`dir://`, since a plain path or `file://` is another repository).
  Tests run the whole flow through it with no network.

## 5. Concurrency: compare-and-swap on `refs`

| Provider | Create only | Replace only if unchanged |
|---|---|---|
| S3 | `PutObject` with `If-None-Match: *` | `PutObject` with `If-Match: <ETag>` |
| GCS | `ifGenerationMatch=0` | `ifGenerationMatch=<generation>` |

When a push finds `refs` has moved, it re-reads `refs` and merges by the
same rules as the operation log's merge:
- a bookmark moved by only one side takes that side's value;
- moved differently by both, it keeps both sides' heads, as several
  heads;
- a tag keeps its first value.

It then retries the write. Uploading packs needs no locking: two writers
uploading the same objects is harmless.

**As built (2026-10-04):** the `object_store` crate 0.14.2 carries both
writes as `PutMode::Create` and `PutMode::Update(UpdateVersion { e_tag,
version })`; S3 needs `S3ConditionalPut::ETagMatch` switched on, which
sends `If-None-Match: *` and `If-Match: <ETag>`; GCS uses the generation
(`ifGenerationMatch`); Azure the ETag. A store's `Version` carries both
the ETag and the generation, so the same code serves all three.
Checked against the real services on 2026-10-03/04: every conditional
write refuses as required on S3, GCS and Azure.

## 6. Safety

- **The bucket is untrusted.** Every object is re-hashed on read, as
  local reads are now. Signatures and repository policy are checked on
  fetch exactly as for a hub remote.
- **Bound to one repository.** `oxygen.toml` names the repository id. A
  push to a bucket holding a different repository is refused, and so is
  a clone that finds packs naming another id.
- **Size limits.** The existing 1 GiB object cap, plus limits on index
  and `refs` sizes, so a hostile bucket cannot drive huge allocations.
- **Garbage collection never deletes in place.** `oxy gc` on a remote
  writes the merged pack and moves `refs` first. Packs no longer named
  are deleted only after a grace period (default 7 days), so a slow
  concurrent reader is never left with a missing pack.
- **Least privilege.** The documented IAM and GCS policies need only
  Get, Put, List and Delete under the prefix. A read-only key gives a
  working clone and pull.
- **No secrets in the repository.** Credentials stay in the providers'
  own chains or the OS keychain, never in `.oxy/config`.

## 7. Testing

- **One conformance suite** that every store must pass, run against
  local, S3 and GCS. It covers object round trips, range reads,
  create-only refusals, compare-and-swap conflicts, concurrent pushers
  merging, an interrupted push leaving the remote valid, gc with a
  concurrent reader, and a repository-id mismatch being refused.
- **In CI** (Flow jobs have no network): the in-memory and directory
  stores run the conformance check and the whole push/clone/pull flow.
- **Real buckets** for the cloud stores, run by hand from a machine that
  holds restricted credentials: `crates/oxy-strata/tests/cloud.rs` reads
  `OXY_TEST_S3`, `OXY_TEST_GCS` and `OXY_TEST_AZURE` (each a store URL
  under a test prefix, with `OXY_TEST_S3_PROFILE`,
  `OXY_TEST_GCS_CREDENTIALS` and `OXY_TEST_AZURE_CREDENTIALS` naming the
  identity), runs the conformance check and a repository round trip under
  a prefix of its own, and deletes what it wrote. Unset, the tests say so
  and pass.
- **Emulators** were tried and dropped: MinIO's images are no longer
  published publicly, and `fake-gcs-server` rejects the client's XML
  uploads. The real services are the test.

## 8. Order of work

1. **The `Store` trait in oxy-core**, with the local store moved behind
   it and no change in behaviour, so every existing test passes.
2. **S3 and GCS stores together**, with the conformance suite and the
   emulators.
3. **`oxy remote add`, `push`, `pull`, `clone` and `init --remote`**
   for bucket URLs, with the friendly errors and `remote check`.
4. **`oxy gc` against a bucket**, with the grace period.
5. **Documentation:** a user guide page, sample IAM and GCS policies,
   and a README section.

Later, on the same trait:
- Azure Blob storage;
- a Git repository used as a plain store, so `oxy push` works to
  GitHub or GitLab;
- the hub attaching a repository from a customer's bucket ("bring your
  own storage", see oxygenhub `docs/scaling.md`);
- couriers: signed, encrypted bundles shared by email, Slack, Matrix
  and the like (a separate design);
- a virtual working copy mounted with FUSE (from OxygenFlow's
  `flow-casfs`).

## 9. Open questions

- Should `oxy clone s3://…` set the bucket as `origin`, or prompt for a
  name? Proposed: `origin`, as for any URL.
- Does a bucket remote keep its own operation log, or only `refs`?
  Proposed: only `refs` at first. The local operation log already
  records what each push did.
- Should packs be encrypted on the client, so the provider only ever
  stores ciphertext, beyond the provider's own encryption? Proposed:
  later, as an option. It does not change the layout.