# Oxygen Anywhere: stores

Status: design, 2026-10-03. Nothing here is built yet. Next to implement.

## 1. What and why

Today an Oxygen repository lives in `.oxy/` on a disk, and a remote is

either another directory or an OxygenHub. This feature makes storage

pluggable. Any **store** that can keep write-once objects and update a

small reference record by compare-and-swap can hold a complete

repository, with no server in between.

The first stores are **Amazon S3** and **Google Cloud Storage**, built

side by side so the code stays portable. Azure Blob and a Git repository

used as a plain store follow on the same interface.

Why Oxygen can do this safely when Git cannot:

- **Objects never change.** Changes, states, blobs, attestations and

  receipts are named by their BLAKE3 hash and written once.

- **Only references are mutable.** Bookmarks, tags and the operation

  head are small, and object stores now offer compare-and-swap on a

  single object.

- **Every read is verified.** Objects are re-hashed on read and changes

  are signed, so a bucket is untrusted storage. Tampering is detected,

  and nobody can forge history, not even the bucket's owner.

Performance is "good enough" for one engineer today. This work is about

where a repository can live, not about speed.

## 1a. The driving case: data sovereignty

UK Government work requires data to be kept on UK shores. On GitLab SaaS

that meant waiting for GitLab to build UK data centres, and it was still

unclear where runners kept their data, so the runners were self-hosted.

Strata answers that for repositories: they live in the customer's own

bucket, in their region, encrypted with their own KMS key. A sovereign

forge needs the same for everything else:

- **Flow artifacts, logs and caches** in the customer's bucket. Today

  artifacts are kept in the hub's store and logs in its database. This is

  the next gap after Strata.

- **Forge records** (issues, proposals, reviews) in a database in the

  customer's region.

- **Keys held by the customer:** KMS for storage, developers' own keys

  for signing.

The model to aim for is **control plane anywhere, data plane in the

customer's account**:

- **Content is never trusted to the operator.** It is hash-verified and

  signed, so it stays tamper-evident whoever runs the hub.

- **Crypto-shredding.** Revoking the customer's KMS key makes the data

  unreadable everywhere, including backups. That is a clean exit and

  deletion guarantee.

- **The customer audits the vendor.** Every read the hub makes appears in

  the customer's own CloudTrail or GCS audit logs.

- **Rendering stays in-region.** Rendering code for the browser needs

  plaintext, so a store-and-render node runs inside the customer's

  account, and only orchestration and the UI shell run outside it.

## 2. What users see

Storage must be transparent: a bucket behaves like any other remote, with

no new concepts to learn.

```sh

oxy remote add backup s3://my-bucket/repos/oxygen

oxy push backup

oxy clone gs://team-bucket/metalgear

oxy pull

```

- **URLs:** `s3://BUCKET/PREFIX` and `gs://BUCKET/PREFIX`. The prefix is

  optional, so one bucket can hold many repositories.

- **Credentials come from the standard places**, with no oxy-specific

  configuration:

  - **AWS:** environment variables, `~/.aws` profiles (`AWS_PROFILE`),

    SSO, and instance or container roles;

  - **Google Cloud:** Application Default Credentials (`gcloud auth

    application-default login`), service account files, and workload

    identity.

  - **Overrides:** optional per-remote settings in the repository's

    config (`profile`, `region`, `endpoint`), for example

    `oxy remote add backup s3://b/p --profile work`.

- **Errors are written for people.** Each one says what to do next, for

  example: "no AWS credentials found: run `aws sso login` or set

  AWS_PROFILE", or "the bucket exists but this account cannot write to

  it (s3:PutObject denied on my-bucket/repos/oxygen)".

- **`oxy init --remote s3://…`** creates an empty repository in the

  bucket, and `oxy remote check NAME` reports access, latency and

  what is stored there.

- **Progress, colour and glyphs** follow the CLI style of every other

  command.

- **Several writers are safe.** Two people pushing to the same bucket at

  once is handled the way the operation log handles two processes today

  (§5).

## 3. Architecture

- **A `Store` trait in oxy-core**, with three implementations: the

  current local `.oxy/` store, S3, and GCS. It has two halves:

  - **objects**: `has`, `get`, `get_range`, `put` (create only), and

    `list` by prefix;

  - **references**: `read() -> (bytes, version)` and

    `write(bytes, expected_version) -> Ok | Conflict`.

- **The `object_store` crate** (Apache Arrow) for S3 and GCS: one API

  for both, mature credential chains, multipart uploads, and conditional

  puts. Azure is in the same crate, so it comes nearly free later.

- **Push and pull talk to the bucket directly** through the same

  negotiation the hub protocol uses: what does the remote have, and what

  is missing? The client sends packs, never loose objects.

- **A local cache** of fetched objects, which is always correct because

  objects never change.

- **The working repository stays local.** A bucket is a remote. Running

  a repository *from* a bucket (no local `.oxy/`) is out of scope here;

  it is what the virtual working copy (§8) is for.

## 4. Layout in the bucket

```

PREFIX/

  oxygen.toml          format version and repository id, written once

  packs/<id>.pack      immutable, zstd-compressed objects

  packs/<id>.idx       sorted index of the pack (read with range GETs)

  refs                 bookmarks, tags and the remote's operation head: one object, updated by compare-and-swap

  receipts/, attest/   only if they are kept outside packs

```

- **Pushing** uploads one pack and its index, then updates `refs`.

- **Packs never change.** A pack named in `refs` is complete, because

  the pack is written before `refs` moves.

- **One `refs` object** so that a push moving several bookmarks is

  atomic. Per-reference objects would scale to more writers but lose

  atomic pushes; one engineer and small teams do not need them.

- **Many small packs from many pushes** are merged by `oxy gc` against

  the remote, as `merge_packs` does locally.

## 5. Concurrency: compare-and-swap on `refs`

| Provider | Create only | Replace only if unchanged |

|---|---|---|

| S3 | `PutObject` with `If-None-Match: *` | `PutObject` with `If-Match: <ETag>` |

| GCS | `ifGenerationMatch=0` | `ifGenerationMatch=<generation>` |

When a push finds `refs` has moved, it re-reads `refs` and merges by the

same rules as the operation log's merge:

- a bookmark moved by only one side takes that side's value;

- moved differently by both, it keeps both sides' heads, as several

  heads;

- a tag keeps its first value.

It then retries the write. Uploading packs needs no locking: two writers

uploading the same objects is harmless.

Before building on it, check each provider's current documentation for

conditional writes: S3 added `If-Match` on `PutObject` in 2024, and this

design depends on it.

## 6. Safety

- **The bucket is untrusted.** Every object is re-hashed on read, as

  local reads are now. Signatures and repository policy are checked on

  fetch exactly as for a hub remote.

- **Bound to one repository.** `oxygen.toml` names the repository id. A

  push to a bucket holding a different repository is refused, and so is

  a clone that finds packs naming another id.

- **Size limits.** The existing 1 GiB object cap, plus limits on index

  and `refs` sizes, so a hostile bucket cannot drive huge allocations.

- **Garbage collection never deletes in place.** `oxy gc` on a remote

  writes the merged pack and moves `refs` first. Packs no longer named

  are deleted only after a grace period (default 7 days), so a slow

  concurrent reader is never left with a missing pack.

- **Least privilege.** The documented IAM and GCS policies need only

  Get, Put, List and Delete under the prefix. A read-only key gives a

  working clone and pull.

- **No secrets in the repository.** Credentials stay in the providers'

  own chains or the OS keychain, never in `.oxy/config`.

## 7. Testing

- **One conformance suite** that every store must pass, run against

  local, S3 and GCS. It covers object round trips, range reads,

  create-only refusals, compare-and-swap conflicts, concurrent pushers

  merging, an interrupted push leaving the remote valid, gc with a

  concurrent reader, and a repository-id mismatch being refused.

- **Emulators in CI** (Flow jobs have no network):

  - MinIO for S3, if its conditional writes match S3's, otherwise

    LocalStack;

  - `fake-gcs-server` for GCS.

- **Real buckets** (Andy's AWS and GCP accounts) for a nightly or

  manual test run. It needs network access, so it runs outside the

  isolated tier, with credentials scoped to a test prefix.

## 8. Order of work

1. **The `Store` trait in oxy-core**, with the local store moved behind

   it and no change in behaviour, so every existing test passes.

2. **S3 and GCS stores together**, with the conformance suite and the

   emulators.

3. **`oxy remote add`, `push`, `pull`, `clone` and `init --remote`**

   for bucket URLs, with the friendly errors and `remote check`.

4. **`oxy gc` against a bucket**, with the grace period.

5. **Documentation:** a user guide page, sample IAM and GCS policies,

   and a README section.

Later, on the same trait:

- Azure Blob storage;

- a Git repository used as a plain store, so `oxy push` works to

  GitHub or GitLab;

- the hub attaching a repository from a customer's bucket ("bring your

  own storage", see oxygenhub `docs/scaling.md`);

- couriers: signed, encrypted bundles shared by email, Slack, Matrix

  and the like (a separate design);

- a virtual working copy mounted with FUSE (from OxygenFlow's

  `flow-casfs`).

## 9. Open questions

- Should `oxy clone s3://…` set the bucket as `origin`, or prompt for a

  name? Proposed: `origin`, as for any URL.

- Does a bucket remote keep its own operation log, or only `refs`?

  Proposed: only `refs` at first. The local operation log already

  records what each push did.

- Should packs be encrypted on the client, so the provider only ever

  stores ciphertext, beyond the provider's own encryption? Proposed:

  later, as an option. It does not change the layout.