Oxygen Strata: keeping a repository on your own storage

A repository can be pushed to, cloned from and pulled from a bucket in your own cloud account, or a directory on a shared drive, with no server in between. The bucket is just another remote:

oxy remote add cloud s3://my-bucket/repos/project --profile work
oxy remote check cloud
oxy push cloud
oxy clone s3://my-bucket/repos/project?profile=work
oxy pull

It works because Oxygen's objects never change and are named by their hash, so any store that can keep write-once objects and update one small record by compare-and-swap can hold a complete repository. Nothing in the store is trusted: every object is re-hashed as it is read, changes are signed, and a bookmark's attestation is verified under your policy, as for any remote. The data stays in your account, in your region, under your keys; an operator of a hub need never see it.

Supported stores:

A store holds one repository per prefix. The first push binds the prefix to the repository; a push of another repository to the same prefix is refused. Use a prefix per repository, as repos/project above.

Setting up a bucket

Amazon S3

1. Create a bucket in the region you want the data kept in (for example eu-west-2, London), with public access blocked.

2. Give the identity oxy will use these permissions, and no more:

   {
     "Version": "2012-10-17",
     "Statement": [
       { "Effect": "Allow", "Action": "s3:ListBucket",
         "Resource": "arn:aws:s3:::my-bucket" },
       { "Effect": "Allow",
         "Action": ["s3:GetObject", "s3:PutObject", "s3:DeleteObject"],
         "Resource": "arn:aws:s3:::my-bucket/repos/*" }
     ]
   }
   

   A read-only identity (s3:ListBucket and s3:GetObject) gives a working clone and pull.

3. Credentials come from the places the AWS tools use, in this order: AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY in the environment; the profile named by --profile (or AWS_PROFILE, else default) in ~/.aws/credentials; what the AWS CLI can produce for that profile (aws sso login, aws login, assumed roles, credential processes), so anything that works for aws works for oxy; and the instance's or container's role.

4. The region is taken from --region, AWS_REGION, the profile, or asked of S3. --endpoint http://… points at an S3-compatible service.

Google Cloud Storage

1. Create a bucket in the region you want (for example europe-west2, London), with uniform access and public access prevention.
2. Create a service account and grant it Storage Object Admin on that bucket only (Storage Object Viewer for a read-only identity), or use your own account.
3. Credentials: a service account key file named by --credentials or GOOGLE_APPLICATION_CREDENTIALS; else the application default credentials that gcloud auth application-default login writes; else the instance's service account.

Azure Blob Storage

1. Create a storage account in the region you want (for example UK South), with public blob access off, and a container in it.
2. Create a service principal (az ad sp create-for-rbac) with Storage Blob Data Contributor scoped to that container (Storage Blob Data Reader for read-only), or use your own sign-in.
3. Credentials: AZURE_CLIENT_ID, AZURE_CLIENT_SECRET and AZURE_TENANT_ID for a service principal; AZURE_STORAGE_ACCOUNT_KEY or AZURE_STORAGE_SAS_TOKEN; else what az login left behind; else the machine's managed identity.

A directory

dir:///media/usb/project needs nothing. Several people on one machine, or one person from several checkouts, can share it; a reference is updated under a lock file beside it.

Using a store remote

oxy remote add NAME URL stores the URL; --profile, --region, --endpoint and --credentials are kept on it as query parameters, so a clone can name them too: oxy clone "gs://bucket/project?credentials=key.json".

oxy remote check NAME reaches the store with the credentials in hand and says how long that took, what the prefix holds (this repository, another one, or nothing yet), and then puts the store through the checks a repository depends on: object round trips, create-only writes, listing, deletion and compare-and-swap on a reference. It writes under a probe prefix of its own and leaves nothing behind.

Then push, pull, fetch and clone work as for any remote. Every push uploads one bundle holding what the store lacks, and moves the record of bookmarks and tags from the version it read. Two people pushing at once are safe: whoever lands second is applied over what landed, or told to pull first when the same bookmark moved.

oxy gc --remote NAME folds the bundles into one, so a fresh clone downloads one file. The superseded bundles are removed only seven days later (--now removes them at once), so a clone that was listing them at the time still completes.

A repository kept on a store

A store can also hold a repository itself rather than a copy of it. Start one with --store:

oxy init myproject --store s3://my-bucket/repos/myproject?profile=work

• Where things live: the objects and the operation log are in the store. .oxy/cache keeps the objects this machine has used, up to storage.cache (2 GiB unless you set it, for example oxy config storage.cache 10G).
• The cache: you can delete .oxy/cache at any time. Objects come back from the store as they are needed.
• One repository per prefix: oxy init --store refuses a prefix that already holds a repository's operation log.
• Speed: each command that records something makes a few requests to the store, so a commit takes about half a second to a second.
• Choose at init: set storage.url only when the repository is created. Changing it later does not move the objects.
• Signatures and seals: each one is stored under a key of its own, as in any repository. A bucket can only find those keys with a LIST request, so the store also keeps a journal of them (pack-entries) and lists of them (pack-<id>.entries). These are read when the repository is opened and again every 30 seconds (spec §13.7). After every client that writes to the bucket runs this version of oxy, run oxy gc once. From then on, looking up a signature or a seal sends no LIST request, and another client's new signature is seen within 30 seconds. A client still on oxy 0.4.6 or older writes signatures that the others miss until the next oxy gc, so upgrade every client first.

A repository kept on a store holds:

PREFIX/
  pack-<id>.pack, .idx, .dict   packs, their indexes and dictionaries
  pack-retired                  what repacks and folds replaced, deleted an hour later
  pack-entries                  journal: signatures and index entries since the last fold
  pack-<id>.entries             every signature and index entry, as the last gc found them
  pack-<id>.entries-part        journal lines folded out once there were more than 1000
  pack-legacy-folded            the older one-list keys have been folded
  loose/  nocompress/           objects not yet packed; chunks kept raw
  sig/  seal/  receipt/  approval/  cert/  accepted/
                                one key per signature or index entry
  ops/head                      the operation head

What a store holds

PREFIX/
  oxygen.toml                     format version and repository id, written once
  bundles/00000001-<hash>.bundle  one per push: changes, signatures, seals, objects
  bundles/00000002-<hash>.bundle
  refs                            bookmarks with heads and attestations, tags, policy
  gc                              bundles a gc superseded, with when

Each client keeps a mirror of the store under .oxy/remotes/NAME/strata/, brought level before every use, and runs the same checks on a push that a hub runs. The mirror is a cache: delete it and the next use rebuilds it.

Troubleshooting

Every error says what to do next. The common ones:

• access denied — the identity cannot do this on the bucket: check its policy covers ListBucket on the bucket and GetObject, PutObject and DeleteObject under the prefix (or the equivalents on GCS and Azure).
• no usable credentials — nothing in the chain above produced any: set a profile, sign in with the provider's tool, or set the variables.
• profile "x" is not in ~/.aws/config or ~/.aws/credentials — the name on the URL is wrong, or the files live elsewhere (AWS_CONFIG_FILE, AWS_SHARED_CREDENTIALS_FILE).
• no region was given and S3 did not say — add --region, set AWS_REGION, or put region in the profile.
• holds a different repository — the prefix is taken: use another.
• a push may be in progress, so try again — a bundle another push uploaded is not yet named in the record; a moment later it is. If it never is, the store is missing changes: oxy remote check says what it holds.
• the remote kept moving while pushing — several pushes landed while yours was retried; oxy pull, then push again.

oxy remote check NAME is the first thing to run when something is wrong: it separates reachability, permissions and behaviour.