Oxygen Strata: pluggable storage

Status: built 2026-10-03/04 (oxygen#63): the Store trait, directory and in-memory stores, the bucket layout and the store remote, the S3, GCS and Azure stores, the CLI (remote add settings, remote check, gc --remote), and the user guide docs/strata.md. This document is the design and the record of what was decided; the guide is for users.

Oxygen Strata is the feature: a repository can live on any storage layer. In code and in this document, each backend is a store.

1. What and why

Today an Oxygen repository lives in .oxy/ on a disk, and a remote is either another directory or an OxygenHub. This feature makes storage pluggable. Any store that can keep write-once objects and update a small reference record by compare-and-swap can hold a complete repository, with no server in between.

The first stores are Amazon S3 and Google Cloud Storage, built side by side so the code stays portable. Azure Blob and a Git repository used as a plain store follow on the same interface.

Why Oxygen can do this safely when Git cannot:

• Objects never change. Changes, states, blobs, attestations and receipts are named by their BLAKE3 hash and written once.
• Only references are mutable. Bookmarks, tags and the operation head are small, and object stores now offer compare-and-swap on a single object.
• Every read is verified. Objects are re-hashed on read and changes are signed, so a bucket is untrusted storage. Tampering is detected, and nobody can forge history, not even the bucket's owner.

Performance is "good enough" for one engineer today. This work is about where a repository can live, not about speed.

1a. The driving case: data sovereignty

Many organisations, such as Government, require data to be kept on sovereign shores. Strata answers that for repositories: they live in the customer's own bucket, in their region, encrypted with their own KMS key. A sovereign forge needs the same for everything else:

• Flow artifacts, logs and caches in the customer's bucket. Today artifacts are kept in the hub's store and logs in its database. This is the next gap after Strata.
• Forge records (issues, proposals, reviews) in a database in the customer's region.
• Keys held by the customer: KMS for storage, developers' own keys for signing.

The model to aim for is control plane anywhere, data plane in the customer's account:

• Content is never trusted to the operator. It is hash-verified and signed, so it stays tamper-evident whoever runs the hub.
• Crypto-shredding. Revoking the customer's KMS key makes the data unreadable everywhere, including backups. That is a clean exit and deletion guarantee.
• The customer audits the vendor. Every read the hub makes appears in the customer's own CloudTrail or GCS audit logs.
• Rendering stays in-region. Rendering code for the browser needs plaintext, so a store-and-render node runs inside the customer's account, and only orchestration and the UI shell run outside it.

2. What users see

Storage must be transparent: a bucket behaves like any other remote, with no new concepts to learn.

oxy remote add backup s3://my-bucket/repos/oxygen
oxy push backup
oxy clone gs://team-bucket/metalgear
oxy pull

• URLs: s3://BUCKET/PREFIX and gs://BUCKET/PREFIX. The prefix is optional, so one bucket can hold many repositories.
• Credentials come from the standard places, with no oxy-specific configuration:
  • AWS: environment variables, ~/.aws profiles (AWS_PROFILE), SSO, and instance or container roles;
  • Google Cloud: Application Default Credentials (gcloud auth application-default login), service account files, and workload identity.
  • Overrides: optional per-remote settings in the repository's config (profile, region, endpoint), for example oxy remote add backup s3://b/p --profile work.
• Errors are written for people. Each one says what to do next, for example: "no AWS credentials found: run aws sso login or set AWS_PROFILE", or "the bucket exists but this account cannot write to it (s3:PutObject denied on my-bucket/repos/oxygen)".
• oxy init --remote s3://… creates an empty repository in the bucket, and oxy remote check NAME reports access, latency and what is stored there.
• Progress, colour and glyphs follow the CLI style of every other command.
• Several writers are safe. Two people pushing to the same bucket at once is handled the way the operation log handles two processes today (§5).

3. Architecture

• The Store trait (oxy_core::strata, built 2026-10-03). It has two halves:

  • objects: has, get, get_range, put_new (create only), list by prefix, and delete;
  • references: read_ref() -> (bytes, version) and write_ref(bytes, expected_version) -> Ok | Conflict. A version is opaque and compared for equality only. Writing identical bytes may keep the version, as S3's content-based ETag does; compare-and-swap guards against losing a different record.
  • Keys are plain relative paths ([A-Za-z0-9._-] segments, no . or ..), the same rule on every store, so a key can never escape a directory store.
  • Implementations: LocalStore (a directory: create-only through a hard link, references under a lock file beside them, so processes on one machine can share it), MemoryStore (tests), then S3 and GCS in a crate of their own.
  • strata::conformance(store, prefix) exercises any store under a probe prefix and leaves it empty: every implementation runs it in its tests, and oxy remote check runs it against a remote.
  • Since 2026-10-04 (oxygen#68, B1) the repository's own object store is behind the trait too: .oxy/objects is a LocalStore with the same keys a bucket would hold (loose/, pack-*, nocompress/, sig/, seal/, receipt/), and the pack code reads a packed object with one ranged read. A directory store is as fast as the files were: the same stats, reads and syncs, with a batch of loose objects synced together. What must be on this disk, a pack being streamed out and the derived metas cache, lives in a scratch directory, which for a directory store is the store itself.

• The object_store crate (Apache Arrow) for S3 and GCS: one API for both, mature credential chains, multipart uploads, and conditional puts. Azure is in the same crate, so it comes nearly free later.

• Push and pull talk to the bucket directly through the same negotiation the hub protocol uses: what does the remote have, and what is missing? The client sends packs, never loose objects.

• A local cache of fetched objects, which is always correct because objects never change.

• The working repository stays local for now. A bucket is a remote. Running a repository from a bucket is plan B (oxygen#68): a Repo opened over a bucket store with a local cache in front, so the hub holds no copy of its own and the data lives where its owner chose. B1 (the store over the trait) is done; B2 is the caching store and the references by compare-and-swap.

• The operation head on a store (B2, 2026-10-05): a repository whose objects are on any store but this disk's directory keeps its operation head in that store's ops/head reference record. Recording an operation reads the record and its version, and moves it only if the version is unchanged. When another process moved it first, the two operations are merged by the same rules as on disk, and the write is tried again (eight times at most). A repository on this disk keeps .oxy/ops/head under its lock file, as before, so older oxy builds share it safely.

• The caching store (B2, 2026-10-05): strata::CachingStore puts a directory store in front of any other. Reads go through it: a missing object is fetched from the remote and kept. Writes go to the remote first and then the cache, so the remote always holds everything and the cache can be deleted at any time. The cache has a size limit; past it the objects used least recently are removed, down to nine tenths of the limit. An object larger than an eighth of the limit is not kept: ranges of it are read from the remote each time. References are never cached, so compare-and-swap is always the remote's.

• A repository on a store (B2, 2026-10-05): oxy init --store URL (or storage.url in .oxy/config, set at init) keeps the objects and the operation head on that store, behind a cache in .oxy/cache of storage.cache bytes (2 GiB by default). .oxy/objects then holds only scratch: packs being written and derived caches. The cache can be deleted at any time. A second repository cannot start on a store that already holds one. Measured from the workstation against the real services with five small commits (a_repository_runs_on_every_configured_store):

  Store	init	each commit	open with no cache, read the files
  S3	441 ms	703 ms	481 ms
  GCS	769 ms	857 ms	570 ms
  Azure	664 ms	511 ms	639 ms

  A commit writes its objects one request each and moves the head, so its cost is round trips, not bytes; batching a commit's objects into one pack is the next step for speed.

4. Layout in the bucket (as built, 2026-10-03)

PREFIX/
  oxygen.toml                      format version and repository id, written once by the first push
  bundles/<seq>-<hash>.bundle      one per push: changes, signatures, seals, objects, policies
                                   (the wire protocol's bundle encoding), written once, named by content
  refs                             every bookmark with its heads and attestation, every tag, the policy
                                   and its history (the protocol's refs response), moved by compare-and-swap

• Pushing uploads one bundle, then updates refs from the version it read. A bundle named in refs is complete, because the bundle is written before refs moves.
• One refs record so that a push moving several bookmarks is atomic. Per-reference records would scale to more writers but lose atomic pushes; one engineer and small teams do not need them.
• Bundles are whole-file transfers. Each push is one bundle, so a pull downloads only the bundles it has not seen. Packs with indexes for range reads, as first planned, are an optimisation for later (oxy gc against a remote merges bundles, and could write a pack).
• The client does what a hub does. A store cannot run code, so each client keeps a mirror, a bare repository of its own under .oxy/remotes/<name>/strata/, brought level with the store before every use: the record read, the bundles it has not seen imported (each object re-hashed), the view set from the record. A push runs the hub's own checks (receive_push: policy, attestations, fast-forward) on the mirror, uploads the bundle, then moves refs. A fetch verifies each bookmark's attestation under the policy, as for any remote. The mirror costs a second copy of the objects on disk; sharing them with the repository's own store is a later optimisation.
• The record moved meanwhile: the mirror is put back to what the store says, brought level, and the push's updates are applied over what landed (up to four times). Two pushes to different bookmarks both get through; one to the same bookmark is told it is stale, as a hub says: pull, then push again.
• oxy gc --remote NAME folds every bundle into one holding what the record reaches, and records the superseded ones in a gc record with the time. They are removed only once seven days have passed (--now ignores that), so a reader that listed them before the fold can still fetch them; a reader that finds one gone skips it, since the merged bundle holds the same changes.
• A directory is a store too: oxy remote add usb dir:///media/usb/repo (dir://, since a plain path or file:// is another repository). Tests run the whole flow through it with no network.

5. Concurrency: compare-and-swap on refs

When a push finds refs has moved, it re-reads refs and merges by the same rules as the operation log's merge:

• a bookmark moved by only one side takes that side's value;
• moved differently by both, it keeps both sides' heads, as several heads;
• a tag keeps its first value.

It then retries the write. Uploading packs needs no locking: two writers uploading the same objects is harmless.

As built (2026-10-04): the object_store crate (0.13.1: 0.14 brought in aws-lc-rs, a C build, and crc-fast, whose cdylib the macOS cross build cannot link) carries both writes as PutMode::Create and PutMode::Update(UpdateVersion { e_tag, version }); S3 needs S3ConditionalPut::ETagMatch switched on, which sends If-None-Match: * and If-Match: <ETag>; GCS uses the generation (ifGenerationMatch); Azure the ETag. A store's Version carries both the ETag and the generation, so the same code serves all three. Checked against the real services on 2026-10-03/04: every conditional write refuses as required on S3, GCS and Azure.

6. Safety

• The bucket is untrusted. Every object is re-hashed on read, as local reads are now. Signatures and repository policy are checked on fetch exactly as for a hub remote.
• Bound to one repository. oxygen.toml names the repository id. A push to a bucket holding a different repository is refused, and so is a clone that finds packs naming another id.
• Size limits. The existing 1 GiB object cap, plus limits on index and refs sizes, so a hostile bucket cannot drive huge allocations.
• Garbage collection never deletes in place. oxy gc on a remote writes the merged pack and moves refs first. Packs no longer named are deleted only after a grace period (default 7 days), so a slow concurrent reader is never left with a missing pack.
• Least privilege. The documented IAM and GCS policies need only Get, Put, List and Delete under the prefix. A read-only key gives a working clone and pull.
• No secrets in the repository. Credentials stay in the providers' own chains or the OS keychain, never in .oxy/config.

7. Testing

• One conformance suite that every store must pass, run against local, S3 and GCS. It covers object round trips, range reads, create-only refusals, compare-and-swap conflicts, concurrent pushers merging, an interrupted push leaving the remote valid, gc with a concurrent reader, and a repository-id mismatch being refused.
• In CI (Flow jobs have no network): the in-memory and directory stores run the conformance check and the whole push/clone/pull flow.
• Real buckets for the cloud stores, run by hand from a machine that holds restricted credentials: crates/oxy-strata/tests/cloud.rs reads OXY_TEST_S3, OXY_TEST_GCS and OXY_TEST_AZURE (each a store URL under a test prefix, with OXY_TEST_S3_PROFILE, OXY_TEST_GCS_CREDENTIALS and OXY_TEST_AZURE_CREDENTIALS naming the identity), runs the conformance check and a repository round trip under a prefix of its own, and deletes what it wrote. Unset, the tests say so and pass.
• Emulators were tried and dropped: MinIO's images are no longer published publicly, and fake-gcs-server rejects the client's XML uploads. The real services are the test.

8. Order of work (done)

1. ✔ The Store trait in oxy-core, with a directory store and an in-memory one; the working repository's own store left as it was, then (B1, 2026-10-04) put behind the trait with the same layout.
2. ✔ The bucket layout and the store remote, tested with no network.
3. ✔ S3, GCS and Azure stores, in crates/oxy-strata, verified against the real services.
4. ✔ oxy remote add, push, pull, clone for store URLs, with the settings on the URL, remote check, and friendly errors. (init --remote was dropped: remote add and push bind a store.)
5. ✔ oxy gc --remote, with the grace period, and merge-and-retry for concurrent pushes.
6. ✔ Documentation: docs/strata.md, the README section, and this record.

Later, on the same trait:

• Azure Blob storage;
• a Git repository used as a plain store, so oxy push works to GitHub or GitLab;
• the hub attaching a repository from a customer's bucket ("bring your own storage", see oxygenhub docs/scaling.md);
• couriers: signed, encrypted bundles shared by email, Slack, Matrix and the like (a separate design);
• a virtual working copy mounted with FUSE (from OxygenFlow's flow-casfs).

9. Open questions

• Should oxy clone s3://… set the bucket as origin, or prompt for a name? Proposed: origin, as for any URL.
• Does a bucket remote keep its own operation log, or only refs? Proposed: only refs at first. The local operation log already records what each push did.
• Should packs be encrypted on the client, so the provider only ever stores ciphertext, beyond the provider's own encryption? Proposed: later, as an option. It does not change the layout.