# Oxygen Strata: keeping a repository on your own storage

A repository can be pushed to, cloned from and pulled from a bucket in

your own cloud account, or a directory on a shared drive, with no server

in between. The bucket is just another remote:

```

oxy remote add cloud s3://my-bucket/repos/project --profile work

oxy remote check cloud

oxy push cloud

oxy clone s3://my-bucket/repos/project?profile=work

oxy pull

```

It works because Oxygen's objects never change and are named by their

hash, so any store that can keep write-once objects and update one small

record by compare-and-swap can hold a complete repository. Nothing in the

store is trusted: every object is re-hashed as it is read, changes are

signed, and a bookmark's attestation is verified under your policy, as

for any remote. The data stays in your account, in your region, under

your keys; an operator of a hub need never see it.

Supported stores:

| URL | Store |

|---|---|

| `s3://BUCKET/PREFIX` | Amazon S3, and S3-compatible services with `--endpoint` |

| `gs://BUCKET/PREFIX` | Google Cloud Storage |

| `az://ACCOUNT/CONTAINER/PREFIX` | Azure Blob Storage |

| `dir://PATH` | a directory: a USB stick, a network share, or for trying things out |

A store holds one repository per prefix. The first push binds the prefix

to the repository; a push of another repository to the same prefix is

refused. Use a prefix per repository, as `repos/project` above.

## Setting up a bucket

### Amazon S3

1. Create a bucket in the region you want the data kept in (for example

   `eu-west-2`, London), with public access blocked.

2. Give the identity oxy will use these permissions, and no more:

   ```json

   {

     "Version": "2012-10-17",

     "Statement": [

       { "Effect": "Allow", "Action": "s3:ListBucket",

         "Resource": "arn:aws:s3:::my-bucket" },

       { "Effect": "Allow",

         "Action": ["s3:GetObject", "s3:PutObject", "s3:DeleteObject"],

         "Resource": "arn:aws:s3:::my-bucket/repos/*" }

     ]

   }

   ```

   A read-only identity (`s3:ListBucket` and `s3:GetObject`) gives a

   working clone and pull.

3. Credentials come from the places the AWS tools use, in this order:

   `AWS_ACCESS_KEY_ID` and `AWS_SECRET_ACCESS_KEY` in the environment; the

   profile named by `--profile` (or `AWS_PROFILE`, else `default`) in

   `~/.aws/credentials`; what the AWS CLI can produce for that profile

   (`aws sso login`, `aws login`, assumed roles, credential processes), so

   anything that works for `aws` works for `oxy`; and the instance's or

   container's role.

4. The region is taken from `--region`, `AWS_REGION`, the profile, or

   asked of S3. `--endpoint http://…` points at an S3-compatible service.

### Google Cloud Storage

1. Create a bucket in the region you want (for example `europe-west2`,

   London), with uniform access and public access prevention.

2. Create a service account and grant it **Storage Object Admin** on that

   bucket only (**Storage Object Viewer** for a read-only identity), or

   use your own account.

3. Credentials: a service account key file named by `--credentials` or

   `GOOGLE_APPLICATION_CREDENTIALS`; else the application default

   credentials that `gcloud auth application-default login` writes; else

   the instance's service account.

### Azure Blob Storage

1. Create a storage account in the region you want (for example

   `UK South`), with public blob access off, and a container in it.

2. Create a service principal (`az ad sp create-for-rbac`) with

   **Storage Blob Data Contributor** scoped to that container (**Storage

   Blob Data Reader** for read-only), or use your own sign-in.

3. Credentials: `AZURE_CLIENT_ID`, `AZURE_CLIENT_SECRET` and

   `AZURE_TENANT_ID` for a service principal; `AZURE_STORAGE_ACCOUNT_KEY`

   or `AZURE_STORAGE_SAS_TOKEN`; else what `az login` left behind; else

   the machine's managed identity.

### A directory

`dir:///media/usb/project` needs nothing. Several people on one machine,

or one person from several checkouts, can share it; a reference is

updated under a lock file beside it.

## Using a store remote

`oxy remote add NAME URL` stores the URL; `--profile`, `--region`,

`--endpoint` and `--credentials` are kept on it as query parameters, so a

clone can name them too: `oxy clone "gs://bucket/project?credentials=key.json"`.

`oxy remote check NAME` reaches the store with the credentials in hand

and says how long that took, what the prefix holds (this repository,

another one, or nothing yet), and then puts the store through the checks

a repository depends on: object round trips, create-only writes, listing,

deletion and compare-and-swap on a reference. It writes under a probe

prefix of its own and leaves nothing behind.

Then `push`, `pull`, `fetch` and `clone` work as for any remote. Every

push uploads one bundle holding what the store lacks, and moves the

record of bookmarks and tags from the version it read. Two people

pushing at once are safe: whoever lands second is applied over what

landed, or told to pull first when the same bookmark moved.

`oxy gc --remote NAME` folds the bundles into one, so a fresh clone

downloads one file. The superseded bundles are removed only seven days

later (`--now` removes them at once), so a clone that was listing them at

the time still completes.

## A repository kept on a store

A store can also hold a repository itself rather than a copy of it.

Start one with `--store`:

```sh

oxy init myproject --store s3://my-bucket/repos/myproject?profile=work

```

- **Where things live:** the objects and the operation log are in the

  store. `.oxy/cache` keeps the objects this machine has used, up to

  `storage.cache` (2 GiB unless you set it, for example

  `oxy config storage.cache 10G`).

- **The cache:** you can delete `.oxy/cache` at any time. Objects come

  back from the store as they are needed.

- **One repository per prefix:** `oxy init --store` refuses a prefix

  that already holds a repository's operation log.

- **Speed:** each command that records something makes a few requests to

  the store, so a commit takes about half a second to a second.

- **Choose at init:** set `storage.url` only when the repository is

  created. Changing it later does not move the objects.

- **Signatures and seals:** each one is stored under a key of its own,

  as in any repository. A bucket can only find those keys with a LIST

  request, so the store also keeps a journal of them (`pack-entries`)

  and lists of them (`pack-<id>.entries`). These are read when the

  repository is opened and again every 30 seconds (spec §13.7). After

  every client that writes to the bucket runs this version of oxy, run

  `oxy gc` once. From then on, looking up a signature or a seal sends no

  LIST request, and another client's new signature is seen within 30

  seconds. A client still on oxy 0.4.6 or older writes signatures that

  the others miss until the next `oxy gc`, so upgrade every client

  first.

A repository kept on a store holds:

```

PREFIX/

  pack-<id>.pack, .idx, .dict   packs, their indexes and dictionaries

  pack-retired                  what repacks and folds replaced, deleted an hour later

  pack-entries                  journal: signatures and index entries since the last fold

  pack-<id>.entries             every signature and index entry, as the last gc found them

  pack-<id>.entries-part        journal lines folded out once there were more than 1000

  pack-legacy-folded            the older one-list keys have been folded

  loose/  nocompress/           objects not yet packed; chunks kept raw

  sig/  seal/  receipt/  approval/  cert/  accepted/

                                one key per signature or index entry

  ops/head                      the operation head

```

## What a store holds

```

PREFIX/

  oxygen.toml                     format version and repository id, written once

  bundles/00000001-<hash>.bundle  one per push: changes, signatures, seals, objects

  bundles/00000002-<hash>.bundle

  refs                            bookmarks with heads and attestations, tags, policy

  gc                              bundles a gc superseded, with when

```

Each client keeps a mirror of the store under `.oxy/remotes/NAME/strata/`,

brought level before every use, and runs the same checks on a push that a

hub runs. The mirror is a cache: delete it and the next use rebuilds it.

## Troubleshooting

Every error says what to do next. The common ones:

- **access denied** — the identity cannot do this on the bucket: check

  its policy covers `ListBucket` on the bucket and `GetObject`,

  `PutObject` and `DeleteObject` under the prefix (or the equivalents on

  GCS and Azure).

- **no usable credentials** — nothing in the chain above produced any:

  set a profile, sign in with the provider's tool, or set the variables.

- **profile "x" is not in ~/.aws/config or ~/.aws/credentials** — the

  name on the URL is wrong, or the files live elsewhere

  (`AWS_CONFIG_FILE`, `AWS_SHARED_CREDENTIALS_FILE`).

- **no region was given and S3 did not say** — add `--region`, set

  `AWS_REGION`, or put `region` in the profile.

- **holds a different repository** — the prefix is taken: use another.

- **a push may be in progress, so try again** — a bundle another push

  uploaded is not yet named in the record; a moment later it is. If it

  never is, the store is missing changes: `oxy remote check` says what

  it holds.

- **the remote kept moving while pushing** — several pushes landed while

  yours was retried; `oxy pull`, then push again.

`oxy remote check NAME` is the first thing to run when something is

wrong: it separates reachability, permissions and behaviour.