# Oxygen Strata: keeping a repository on your own storage A repository can be pushed to, cloned from and pulled from a bucket in your own cloud account, or a directory on a shared drive, with no server in between. The bucket is just another remote: ``` oxy remote add cloud s3://my-bucket/repos/project --profile work oxy remote check cloud oxy push cloud oxy clone s3://my-bucket/repos/project?profile=work oxy pull ``` It works because Oxygen's objects never change and are named by their hash, so any store that can keep write-once objects and update one small record by compare-and-swap can hold a complete repository. Nothing in the store is trusted: every object is re-hashed as it is read, changes are signed, and a bookmark's attestation is verified under your policy, as for any remote. The data stays in your account, in your region, under your keys; an operator of a hub need never see it. Supported stores: | URL | Store | |---|---| | `s3://BUCKET/PREFIX` | Amazon S3, and S3-compatible services with `--endpoint` | | `gs://BUCKET/PREFIX` | Google Cloud Storage | | `az://ACCOUNT/CONTAINER/PREFIX` | Azure Blob Storage | | `dir://PATH` | a directory: a USB stick, a network share, or for trying things out | A store holds one repository per prefix. The first push binds the prefix to the repository; a push of another repository to the same prefix is refused. Use a prefix per repository, as `repos/project` above. ## Setting up a bucket ### Amazon S3 1. Create a bucket in the region you want the data kept in (for example `eu-west-2`, London), with public access blocked. 2. Give the identity oxy will use these permissions, and no more: ```json { "Version": "2012-10-17", "Statement": [ { "Effect": "Allow", "Action": "s3:ListBucket", "Resource": "arn:aws:s3:::my-bucket" }, { "Effect": "Allow", "Action": ["s3:GetObject", "s3:PutObject", "s3:DeleteObject"], "Resource": "arn:aws:s3:::my-bucket/repos/*" } ] } ``` A read-only identity (`s3:ListBucket` and `s3:GetObject`) gives a working clone and pull. 3. Credentials come from the places the AWS tools use, in this order: `AWS_ACCESS_KEY_ID` and `AWS_SECRET_ACCESS_KEY` in the environment; the profile named by `--profile` (or `AWS_PROFILE`, else `default`) in `~/.aws/credentials`; what the AWS CLI can produce for that profile (`aws sso login`, `aws login`, assumed roles, credential processes), so anything that works for `aws` works for `oxy`; and the instance's or container's role. 4. The region is taken from `--region`, `AWS_REGION`, the profile, or asked of S3. `--endpoint http://…` points at an S3-compatible service. ### Google Cloud Storage 1. Create a bucket in the region you want (for example `europe-west2`, London), with uniform access and public access prevention. 2. Create a service account and grant it **Storage Object Admin** on that bucket only (**Storage Object Viewer** for a read-only identity), or use your own account. 3. Credentials: a service account key file named by `--credentials` or `GOOGLE_APPLICATION_CREDENTIALS`; else the application default credentials that `gcloud auth application-default login` writes; else the instance's service account. ### Azure Blob Storage 1. Create a storage account in the region you want (for example `UK South`), with public blob access off, and a container in it. 2. Create a service principal (`az ad sp create-for-rbac`) with **Storage Blob Data Contributor** scoped to that container (**Storage Blob Data Reader** for read-only), or use your own sign-in. 3. Credentials: `AZURE_CLIENT_ID`, `AZURE_CLIENT_SECRET` and `AZURE_TENANT_ID` for a service principal; `AZURE_STORAGE_ACCOUNT_KEY` or `AZURE_STORAGE_SAS_TOKEN`; else what `az login` left behind; else the machine's managed identity. ### A directory `dir:///media/usb/project` needs nothing. Several people on one machine, or one person from several checkouts, can share it; a reference is updated under a lock file beside it. ## Using a store remote `oxy remote add NAME URL` stores the URL; `--profile`, `--region`, `--endpoint` and `--credentials` are kept on it as query parameters, so a clone can name them too: `oxy clone "gs://bucket/project?credentials=key.json"`. `oxy remote check NAME` reaches the store with the credentials in hand and says how long that took, what the prefix holds (this repository, another one, or nothing yet), and then puts the store through the checks a repository depends on: object round trips, create-only writes, listing, deletion and compare-and-swap on a reference. It writes under a probe prefix of its own and leaves nothing behind. Then `push`, `pull`, `fetch` and `clone` work as for any remote. Every push uploads one bundle holding what the store lacks, and moves the record of bookmarks and tags from the version it read. Two people pushing at once are safe: whoever lands second is applied over what landed, or told to pull first when the same bookmark moved. `oxy gc --remote NAME` folds the bundles into one, so a fresh clone downloads one file. The superseded bundles are removed only seven days later (`--now` removes them at once), so a clone that was listing them at the time still completes. ## A repository kept on a store A store can also hold a repository itself rather than a copy of it. Start one with `--store`: ```sh oxy init myproject --store s3://my-bucket/repos/myproject?profile=work ``` - **Where things live:** the objects and the operation log are in the store. `.oxy/cache` keeps the objects this machine has used, up to `storage.cache` (2 GiB unless you set it, for example `oxy config storage.cache 10G`). - **The cache:** you can delete `.oxy/cache` at any time. Objects come back from the store as they are needed. - **One repository per prefix:** `oxy init --store` refuses a prefix that already holds a repository's operation log. - **Speed:** each command that records something makes a few requests to the store, so a commit takes about half a second to a second. - **Choose at init:** set `storage.url` only when the repository is created. Changing it later does not move the objects. ## What a store holds ``` PREFIX/ oxygen.toml format version and repository id, written once bundles/00000001-.bundle one per push: changes, signatures, seals, objects bundles/00000002-.bundle refs bookmarks with heads and attestations, tags, policy gc bundles a gc superseded, with when ``` Each client keeps a mirror of the store under `.oxy/remotes/NAME/strata/`, brought level before every use, and runs the same checks on a push that a hub runs. The mirror is a cache: delete it and the next use rebuilds it. ## Troubleshooting Every error says what to do next. The common ones: - **access denied** — the identity cannot do this on the bucket: check its policy covers `ListBucket` on the bucket and `GetObject`, `PutObject` and `DeleteObject` under the prefix (or the equivalents on GCS and Azure). - **no usable credentials** — nothing in the chain above produced any: set a profile, sign in with the provider's tool, or set the variables. - **profile "x" is not in ~/.aws/config or ~/.aws/credentials** — the name on the URL is wrong, or the files live elsewhere (`AWS_CONFIG_FILE`, `AWS_SHARED_CREDENTIALS_FILE`). - **no region was given and S3 did not say** — add `--region`, set `AWS_REGION`, or put `region` in the profile. - **holds a different repository** — the prefix is taken: use another. - **a push may be in progress, so try again** — a bundle another push uploaded is not yet named in the record; a moment later it is. If it never is, the store is missing changes: `oxy remote check` says what it holds. - **the remote kept moving while pushing** — several pushes landed while yours was retried; `oxy pull`, then push again. `oxy remote check NAME` is the first thing to run when something is wrong: it separates reachability, permissions and behaviour.