Design: Oxygen Anywhere, pluggable stores starting with S3 and GCS (refs #63) a70dce536476

# Oxygen Anywhere: stores

Status: design, 2026-10-03. Nothing here is built yet. Next to implement.

## 1. What and why

Today an Oxygen repository lives in `.oxy/` on a disk, and a remote is

either another directory or an OxygenHub. This feature makes storage

pluggable. Any **store** that can keep write-once objects and update a

small reference record by compare-and-swap can hold a complete

repository, with no server in between.

The first stores are **Amazon S3** and **Google Cloud Storage**, built

side by side so the code stays portable. Azure Blob and a Git repository

used as a plain store follow on the same interface.

Why Oxygen can do this safely when Git cannot:

- **Objects never change.** Changes, states, blobs, attestations and

  receipts are named by their BLAKE3 hash and written once.

- **Only references are mutable.** Bookmarks, tags and the operation

  head are small, and object stores now offer compare-and-swap on a

  single object.

- **Every read is verified.** Objects are re-hashed on read and changes

  are signed, so a bucket is untrusted storage. Tampering is detected,

  and nobody can forge history, not even the bucket's owner.

Performance is "good enough" for one engineer today. This work is about

where a repository can live, not about speed.

## 2. What users see

Storage must be transparent: a bucket behaves like any other remote, with

no new concepts to learn.

```sh

oxy remote add backup s3://my-bucket/repos/oxygen

oxy push backup

oxy clone gs://team-bucket/metalgear

oxy pull

```

- **URLs:** `s3://BUCKET/PREFIX` and `gs://BUCKET/PREFIX`. The prefix is

  optional, so one bucket can hold many repositories.

- **Credentials come from the standard places**, with no oxy-specific

  configuration:

  - **AWS:** environment variables, `~/.aws` profiles (`AWS_PROFILE`),

    SSO, and instance or container roles;

  - **Google Cloud:** Application Default Credentials (`gcloud auth

    application-default login`), service account files, and workload

    identity.

  - **Overrides:** optional per-remote settings in the repository's

    config (`profile`, `region`, `endpoint`), for example

    `oxy remote add backup s3://b/p --profile work`.

- **Errors are written for people.** Each one says what to do next, for

  example: "no AWS credentials found: run `aws sso login` or set

  AWS_PROFILE", or "the bucket exists but this account cannot write to

  it (s3:PutObject denied on my-bucket/repos/oxygen)".

- **`oxy init --remote s3://…`** creates an empty repository in the

  bucket, and `oxy remote check NAME` reports access, latency and

  what is stored there.

- **Progress, colour and glyphs** follow the CLI style of every other

  command.

- **Several writers are safe.** Two people pushing to the same bucket at

  once is handled the way the operation log handles two processes today

  (§5).

## 3. Architecture

- **A `Store` trait in oxy-core**, with three implementations: the

  current local `.oxy/` store, S3, and GCS. It has two halves:

  - **objects**: `has`, `get`, `get_range`, `put` (create only), and

    `list` by prefix;

  - **references**: `read() -> (bytes, version)` and

    `write(bytes, expected_version) -> Ok | Conflict`.

- **The `object_store` crate** (Apache Arrow) for S3 and GCS: one API

  for both, mature credential chains, multipart uploads, and conditional

  puts. Azure is in the same crate, so it comes nearly free later.

- **Push and pull talk to the bucket directly** through the same

  negotiation the hub protocol uses: what does the remote have, and what

  is missing? The client sends packs, never loose objects.

- **A local cache** of fetched objects, which is always correct because

  objects never change.

- **The working repository stays local.** A bucket is a remote. Running

  a repository *from* a bucket (no local `.oxy/`) is out of scope here;

  it is what the virtual working copy (§8) is for.

## 4. Layout in the bucket

```

PREFIX/

  oxygen.toml          format version and repository id, written once

  packs/<id>.pack      immutable, zstd-compressed objects

  packs/<id>.idx       sorted index of the pack (read with range GETs)

  refs                 bookmarks, tags and the remote's operation head: one object, updated by compare-and-swap

  receipts/, attest/   only if they are kept outside packs

```

- **Pushing** uploads one pack and its index, then updates `refs`.

- **Packs never change.** A pack named in `refs` is complete, because

  the pack is written before `refs` moves.

- **One `refs` object** so that a push moving several bookmarks is

  atomic. Per-reference objects would scale to more writers but lose

  atomic pushes; one engineer and small teams do not need them.

- **Many small packs from many pushes** are merged by `oxy gc` against

  the remote, as `merge_packs` does locally.

## 5. Concurrency: compare-and-swap on `refs`

| Provider | Create only | Replace only if unchanged |

|---|---|---|

| S3 | `PutObject` with `If-None-Match: *` | `PutObject` with `If-Match: <ETag>` |

| GCS | `ifGenerationMatch=0` | `ifGenerationMatch=<generation>` |

When a push finds `refs` has moved, it re-reads `refs` and merges by the

same rules as the operation log's merge:

- a bookmark moved by only one side takes that side's value;

- moved differently by both, it keeps both sides' heads, as several

  heads;

- a tag keeps its first value.

It then retries the write. Uploading packs needs no locking: two writers

uploading the same objects is harmless.

Before building on it, check each provider's current documentation for

conditional writes: S3 added `If-Match` on `PutObject` in 2024, and this

design depends on it.

## 6. Safety

- **The bucket is untrusted.** Every object is re-hashed on read, as

  local reads are now. Signatures and repository policy are checked on

  fetch exactly as for a hub remote.

- **Bound to one repository.** `oxygen.toml` names the repository id. A

  push to a bucket holding a different repository is refused, and so is

  a clone that finds packs naming another id.

- **Size limits.** The existing 1 GiB object cap, plus limits on index

  and `refs` sizes, so a hostile bucket cannot drive huge allocations.

- **Garbage collection never deletes in place.** `oxy gc` on a remote

  writes the merged pack and moves `refs` first. Packs no longer named

  are deleted only after a grace period (default 7 days), so a slow

  concurrent reader is never left with a missing pack.

- **Least privilege.** The documented IAM and GCS policies need only

  Get, Put, List and Delete under the prefix. A read-only key gives a

  working clone and pull.

- **No secrets in the repository.** Credentials stay in the providers'

  own chains or the OS keychain, never in `.oxy/config`.

## 7. Testing

- **One conformance suite** that every store must pass, run against

  local, S3 and GCS. It covers object round trips, range reads,

  create-only refusals, compare-and-swap conflicts, concurrent pushers

  merging, an interrupted push leaving the remote valid, gc with a

  concurrent reader, and a repository-id mismatch being refused.

- **Emulators in CI** (Flow jobs have no network):

  - MinIO for S3, if its conditional writes match S3's, otherwise

    LocalStack;

  - `fake-gcs-server` for GCS.

- **Real buckets** (Andy's AWS and GCP accounts) for a nightly or

  manual test run. It needs network access, so it runs outside the

  isolated tier, with credentials scoped to a test prefix.

## 8. Order of work

1. **The `Store` trait in oxy-core**, with the local store moved behind

   it and no change in behaviour, so every existing test passes.

2. **S3 and GCS stores together**, with the conformance suite and the

   emulators.

3. **`oxy remote add`, `push`, `pull`, `clone` and `init --remote`**

   for bucket URLs, with the friendly errors and `remote check`.

4. **`oxy gc` against a bucket**, with the grace period.

5. **Documentation:** a user guide page, sample IAM and GCS policies,

   and a README section.

Later, on the same trait:

- Azure Blob storage;

- a Git repository used as a plain store, so `oxy push` works to

  GitHub or GitLab;

- the hub attaching a repository from a customer's bucket ("bring your

  own storage", see oxygenhub `docs/scaling.md`);

- couriers: signed, encrypted bundles shared by email, Slack, Matrix

  and the like (a separate design);

- a virtual working copy mounted with FUSE (from OxygenFlow's

  `flow-casfs`).

## 9. Open questions

- Should `oxy clone s3://…` set the bucket as `origin`, or prompt for a

  name? Proposed: `origin`, as for any URL.

- Does a bucket remote keep its own operation log, or only `refs`?

  Proposed: only `refs` at first. The local operation log already

  records what each push did.

- Should packs be encrypted on the client, so the provider only ever

  stores ciphertext, beyond the provider's own encryption? Proposed:

  later, as an option. It does not change the layout.