Oxygen Anywhere: stores

Status: built 2026-10-03/04 (oxygen#63): the Store trait, directory and in-memory stores, the bucket layout and the store remote, the S3, GCS and Azure stores, the CLI (remote add settings, remote check, gc --remote), and the user guide docs/strata.md. This document is the design and the record of what was decided; the guide is for users.

1. What and why

Today an Oxygen repository lives in .oxy/ on a disk, and a remote is either another directory or an OxygenHub. This feature makes storage pluggable. Any store that can keep write-once objects and update a small reference record by compare-and-swap can hold a complete repository, with no server in between.

The first stores are Amazon S3 and Google Cloud Storage, built side by side so the code stays portable. Azure Blob and a Git repository used as a plain store follow on the same interface.

Why Oxygen can do this safely when Git cannot:

• Objects never change. Changes, states, blobs, attestations and receipts are named by their BLAKE3 hash and written once.
• Only references are mutable. Bookmarks, tags and the operation head are small, and object stores now offer compare-and-swap on a single object.
• Every read is verified. Objects are re-hashed on read and changes are signed, so a bucket is untrusted storage. Tampering is detected, and nobody can forge history, not even the bucket's owner.

Performance is "good enough" for one engineer today. This work is about where a repository can live, not about speed.

2. What users see

Storage must be transparent: a bucket behaves like any other remote, with no new concepts to learn.

oxy remote add backup s3://my-bucket/repos/oxygen
oxy push backup
oxy clone gs://team-bucket/metalgear
oxy pull

• URLs: s3://BUCKET/PREFIX and gs://BUCKET/PREFIX. The prefix is optional, so one bucket can hold many repositories.
• Credentials come from the standard places, with no oxy-specific configuration:
  • AWS: environment variables, ~/.aws profiles (AWS_PROFILE), SSO, and instance or container roles;
  • Google Cloud: Application Default Credentials (gcloud auth application-default login), service account files, and workload identity.
  • Overrides: optional per-remote settings in the repository's config (profile, region, endpoint), for example oxy remote add backup s3://b/p --profile work.
• Errors are written for people. Each one says what to do next, for example: "no AWS credentials found: run aws sso login or set AWS_PROFILE", or "the bucket exists but this account cannot write to it (s3:PutObject denied on my-bucket/repos/oxygen)".
• oxy init --remote s3://… creates an empty repository in the bucket, and oxy remote check NAME reports access, latency and what is stored there.
• Progress, colour and glyphs follow the CLI style of every other command.
• Several writers are safe. Two people pushing to the same bucket at once is handled the way the operation log handles two processes today (§5).

3. Architecture

• A Store trait in oxy-core, with three implementations: the current local .oxy/ store, S3, and GCS. It has two halves:
  • objects: has, get, get_range, put (create only), and list by prefix;
  • references: read() -> (bytes, version) and write(bytes, expected_version) -> Ok | Conflict.
• The object_store crate (Apache Arrow) for S3 and GCS: one API for both, mature credential chains, multipart uploads, and conditional puts. Azure is in the same crate, so it comes nearly free later.
• Push and pull talk to the bucket directly through the same negotiation the hub protocol uses: what does the remote have, and what is missing? The client sends packs, never loose objects.
• A local cache of fetched objects, which is always correct because objects never change.
• The working repository stays local. A bucket is a remote. Running a repository from a bucket (no local .oxy/) is out of scope here; it is what the virtual working copy (§8) is for.

4. Layout in the bucket

PREFIX/
  oxygen.toml          format version and repository id, written once
  packs/<id>.pack      immutable, zstd-compressed objects
  packs/<id>.idx       sorted index of the pack (read with range GETs)
  refs                 bookmarks, tags and the remote's operation head: one object, updated by compare-and-swap
  receipts/, attest/   only if they are kept outside packs

• Pushing uploads one pack and its index, then updates refs.
• Packs never change. A pack named in refs is complete, because the pack is written before refs moves.
• One refs object so that a push moving several bookmarks is atomic. Per-reference objects would scale to more writers but lose atomic pushes; one engineer and small teams do not need them.
• Many small packs from many pushes are merged by oxy gc against the remote, as merge_packs does locally.

5. Concurrency: compare-and-swap on refs

When a push finds refs has moved, it re-reads refs and merges by the same rules as the operation log's merge:

• a bookmark moved by only one side takes that side's value;
• moved differently by both, it keeps both sides' heads, as several heads;
• a tag keeps its first value.

It then retries the write. Uploading packs needs no locking: two writers uploading the same objects is harmless.

Before building on it, check each provider's current documentation for conditional writes: S3 added If-Match on PutObject in 2024, and this design depends on it.

6. Safety

• The bucket is untrusted. Every object is re-hashed on read, as local reads are now. Signatures and repository policy are checked on fetch exactly as for a hub remote.
• Bound to one repository. oxygen.toml names the repository id. A push to a bucket holding a different repository is refused, and so is a clone that finds packs naming another id.
• Size limits. The existing 1 GiB object cap, plus limits on index and refs sizes, so a hostile bucket cannot drive huge allocations.
• Garbage collection never deletes in place. oxy gc on a remote writes the merged pack and moves refs first. Packs no longer named are deleted only after a grace period (default 7 days), so a slow concurrent reader is never left with a missing pack.
• Least privilege. The documented IAM and GCS policies need only Get, Put, List and Delete under the prefix. A read-only key gives a working clone and pull.
• No secrets in the repository. Credentials stay in the providers' own chains or the OS keychain, never in .oxy/config.

7. Testing

• One conformance suite that every store must pass, run against local, S3 and GCS. It covers object round trips, range reads, create-only refusals, compare-and-swap conflicts, concurrent pushers merging, an interrupted push leaving the remote valid, gc with a concurrent reader, and a repository-id mismatch being refused.
• Emulators in CI (Flow jobs have no network):
  • MinIO for S3, if its conditional writes match S3's, otherwise LocalStack;
  • fake-gcs-server for GCS.
• Real buckets (Andy's AWS and GCP accounts) for a nightly or manual test run. It needs network access, so it runs outside the isolated tier, with credentials scoped to a test prefix.

8. Order of work (done)

1. ✔ The Store trait in oxy-core, with a directory store and an in-memory one; the working repository's own store left as it was.
2. ✔ The bucket layout and the store remote, tested with no network.
3. ✔ S3, GCS and Azure stores, in crates/oxy-strata, verified against the real services.
4. ✔ oxy remote add, push, pull, clone for store URLs, with the settings on the URL, remote check, and friendly errors. (init --remote was dropped: remote add and push bind a store.)
5. ✔ oxy gc --remote, with the grace period, and merge-and-retry for concurrent pushes.
6. ✔ Documentation: docs/strata.md, the README section, and this record.

Later, on the same trait:

• Azure Blob storage;
• a Git repository used as a plain store, so oxy push works to GitHub or GitLab;
• the hub attaching a repository from a customer's bucket ("bring your own storage", see oxygenhub docs/scaling.md);
• couriers: signed, encrypted bundles shared by email, Slack, Matrix and the like (a separate design);
• a virtual working copy mounted with FUSE (from OxygenFlow's flow-casfs).

9. Open questions

• Should oxy clone s3://… set the bucket as origin, or prompt for a name? Proposed: origin, as for any URL.
• Does a bucket remote keep its own operation log, or only refs? Proposed: only refs at first. The local operation log already records what each push did.
• Should packs be encrypted on the client, so the provider only ever stores ciphertext, beyond the provider's own encryption? Proposed: later, as an option. It does not change the layout.