Oxygen Anywhere: stores

Status: design, 2026-10-03. Nothing here is built yet. Next to implement.

1. What and why

Today an Oxygen repository lives in .oxy/ on a disk, and a remote is either another directory or an OxygenHub. This feature makes storage pluggable. Any store that can keep write-once objects and update a small reference record by compare-and-swap can hold a complete repository, with no server in between.

The first stores are Amazon S3 and Google Cloud Storage, built side by side so the code stays portable. Azure Blob and a Git repository used as a plain store follow on the same interface.

Why Oxygen can do this safely when Git cannot:

• Objects never change. Changes, states, blobs, attestations and receipts are named by their BLAKE3 hash and written once.
• Only references are mutable. Bookmarks, tags and the operation head are small, and object stores now offer compare-and-swap on a single object.
• Every read is verified. Objects are re-hashed on read and changes are signed, so a bucket is untrusted storage. Tampering is detected, and nobody can forge history, not even the bucket's owner.

Performance is "good enough" for one engineer today. This work is about where a repository can live, not about speed.

1a. The driving case: data sovereignty

UK Government work requires data to be kept on UK shores. On GitLab SaaS that meant waiting for GitLab to build UK data centres, and it was still unclear where runners kept their data, so the runners were self-hosted.

Strata answers that for repositories: they live in the customer's own bucket, in their region, encrypted with their own KMS key. A sovereign forge needs the same for everything else:

• Flow artifacts, logs and caches in the customer's bucket. Today artifacts are kept in the hub's store and logs in its database. This is the next gap after Strata.
• Forge records (issues, proposals, reviews) in a database in the customer's region.
• Keys held by the customer: KMS for storage, developers' own keys for signing.

The model to aim for is control plane anywhere, data plane in the customer's account:

• Content is never trusted to the operator. It is hash-verified and signed, so it stays tamper-evident whoever runs the hub.
• Crypto-shredding. Revoking the customer's KMS key makes the data unreadable everywhere, including backups. That is a clean exit and deletion guarantee.
• The customer audits the vendor. Every read the hub makes appears in the customer's own CloudTrail or GCS audit logs.
• Rendering stays in-region. Rendering code for the browser needs plaintext, so a store-and-render node runs inside the customer's account, and only orchestration and the UI shell run outside it.

2. What users see

Storage must be transparent: a bucket behaves like any other remote, with no new concepts to learn.

oxy remote add backup s3://my-bucket/repos/oxygen
oxy push backup
oxy clone gs://team-bucket/metalgear
oxy pull

• URLs: s3://BUCKET/PREFIX and gs://BUCKET/PREFIX. The prefix is optional, so one bucket can hold many repositories.
• Credentials come from the standard places, with no oxy-specific configuration:
  • AWS: environment variables, ~/.aws profiles (AWS_PROFILE), SSO, and instance or container roles;
  • Google Cloud: Application Default Credentials (gcloud auth application-default login), service account files, and workload identity.
  • Overrides: optional per-remote settings in the repository's config (profile, region, endpoint), for example oxy remote add backup s3://b/p --profile work.
• Errors are written for people. Each one says what to do next, for example: "no AWS credentials found: run aws sso login or set AWS_PROFILE", or "the bucket exists but this account cannot write to it (s3:PutObject denied on my-bucket/repos/oxygen)".
• oxy init --remote s3://… creates an empty repository in the bucket, and oxy remote check NAME reports access, latency and what is stored there.
• Progress, colour and glyphs follow the CLI style of every other command.
• Several writers are safe. Two people pushing to the same bucket at once is handled the way the operation log handles two processes today (§5).

3. Architecture

• A Store trait in oxy-core, with three implementations: the current local .oxy/ store, S3, and GCS. It has two halves:
  • objects: has, get, get_range, put (create only), and list by prefix;
  • references: read() -> (bytes, version) and write(bytes, expected_version) -> Ok | Conflict.
• The object_store crate (Apache Arrow) for S3 and GCS: one API for both, mature credential chains, multipart uploads, and conditional puts. Azure is in the same crate, so it comes nearly free later.
• Push and pull talk to the bucket directly through the same negotiation the hub protocol uses: what does the remote have, and what is missing? The client sends packs, never loose objects.
• A local cache of fetched objects, which is always correct because objects never change.
• The working repository stays local. A bucket is a remote. Running a repository from a bucket (no local .oxy/) is out of scope here; it is what the virtual working copy (§8) is for.

4. Layout in the bucket

PREFIX/
  oxygen.toml          format version and repository id, written once
  packs/<id>.pack      immutable, zstd-compressed objects
  packs/<id>.idx       sorted index of the pack (read with range GETs)
  refs                 bookmarks, tags and the remote's operation head: one object, updated by compare-and-swap
  receipts/, attest/   only if they are kept outside packs

• Pushing uploads one pack and its index, then updates refs.
• Packs never change. A pack named in refs is complete, because the pack is written before refs moves.
• One refs object so that a push moving several bookmarks is atomic. Per-reference objects would scale to more writers but lose atomic pushes; one engineer and small teams do not need them.
• Many small packs from many pushes are merged by oxy gc against the remote, as merge_packs does locally.

5. Concurrency: compare-and-swap on refs

When a push finds refs has moved, it re-reads refs and merges by the same rules as the operation log's merge:

• a bookmark moved by only one side takes that side's value;
• moved differently by both, it keeps both sides' heads, as several heads;
• a tag keeps its first value.

It then retries the write. Uploading packs needs no locking: two writers uploading the same objects is harmless.

Before building on it, check each provider's current documentation for conditional writes: S3 added If-Match on PutObject in 2024, and this design depends on it.

6. Safety

• The bucket is untrusted. Every object is re-hashed on read, as local reads are now. Signatures and repository policy are checked on fetch exactly as for a hub remote.
• Bound to one repository. oxygen.toml names the repository id. A push to a bucket holding a different repository is refused, and so is a clone that finds packs naming another id.
• Size limits. The existing 1 GiB object cap, plus limits on index and refs sizes, so a hostile bucket cannot drive huge allocations.
• Garbage collection never deletes in place. oxy gc on a remote writes the merged pack and moves refs first. Packs no longer named are deleted only after a grace period (default 7 days), so a slow concurrent reader is never left with a missing pack.
• Least privilege. The documented IAM and GCS policies need only Get, Put, List and Delete under the prefix. A read-only key gives a working clone and pull.
• No secrets in the repository. Credentials stay in the providers' own chains or the OS keychain, never in .oxy/config.

7. Testing

• One conformance suite that every store must pass, run against local, S3 and GCS. It covers object round trips, range reads, create-only refusals, compare-and-swap conflicts, concurrent pushers merging, an interrupted push leaving the remote valid, gc with a concurrent reader, and a repository-id mismatch being refused.
• Emulators in CI (Flow jobs have no network):
  • MinIO for S3, if its conditional writes match S3's, otherwise LocalStack;
  • fake-gcs-server for GCS.
• Real buckets (Andy's AWS and GCP accounts) for a nightly or manual test run. It needs network access, so it runs outside the isolated tier, with credentials scoped to a test prefix.

8. Order of work

1. The Store trait in oxy-core, with the local store moved behind it and no change in behaviour, so every existing test passes.
2. S3 and GCS stores together, with the conformance suite and the emulators.
3. oxy remote add, push, pull, clone and init --remote for bucket URLs, with the friendly errors and remote check.
4. oxy gc against a bucket, with the grace period.
5. Documentation: a user guide page, sample IAM and GCS policies, and a README section.

Later, on the same trait:

• Azure Blob storage;
• a Git repository used as a plain store, so oxy push works to GitHub or GitLab;
• the hub attaching a repository from a customer's bucket ("bring your own storage", see oxygenhub docs/scaling.md);
• couriers: signed, encrypted bundles shared by email, Slack, Matrix and the like (a separate design);
• a virtual working copy mounted with FUSE (from OxygenFlow's flow-casfs).

9. Open questions

• Should oxy clone s3://… set the bucket as origin, or prompt for a name? Proposed: origin, as for any URL.
• Does a bucket remote keep its own operation log, or only refs? Proposed: only refs at first. The local operation log already records what each push did.
• Should packs be encrypted on the client, so the provider only ever stores ciphertext, beyond the provider's own encryption? Proposed: later, as an option. It does not change the layout.