# Backups: per site, organisation, user and repository

Status: built 2026-10-13/14 (oxygenhub#169): user and organisation

settings, the hourly and daily schedule with "back up now" and TEST,

directory destinations and keyless GCS destinations, every repository's

and wiki's store byte for byte, the records as JSON, retention, failure

notices, and restore (`oxygenhub admin backup run|restore`). Since

2026-10-14 (oxygenhub#175) a restore brings back the records too (§7) and

release assets travel with the stores; Admin › Backups (#176);

per-repository settings (#177); S3 destinations, keylessly (#178).

## 1. Why

A forge is where people's work lives. Before Andy moves his GitHub

repositories here, and before the hub moves to GCP, every owner must be

able to say "my repositories are copied, every hour, to a place I

control, and I can bring them back". GitHub leaves this to its users

(clone everything somewhere, and lose the issues and reviews); GitLab

backs up the whole instance or nothing. Here an owner backs up their own

repositories, with everything around them, to their own bucket, without a

key file anywhere.

## 2. What a backup holds

For each repository the owner has (a user's own, or an organisation's):

- **Its objects:** the store's content-addressed objects (changes,

  blobs, chunks, states, identities, signatures, seals, policies). An

  object never changes once written, so a run sends only the objects the

  destination does not already have: the first run copies everything,

  later ones only what arrived since. Nothing is re-encoded, so a

  restored repository is byte for byte the original, every signature

  still verifying.

- **Its refs at that moment:** bookmarks, tags and the trust policy's

  serial, as a small manifest per run. Each run's manifest is kept, so a

  restore can go back to any run that is still kept.

- **Its hub records, as JSON:** issues and their comments and events;

  proposals, their reviews, threads and approvals; labels, milestones,

  projects; releases and their assets (assets are blobs, sent like

  objects); the wiki (a repository of its own, backed up with it); and

  repository settings and webhooks. Secret values (webhook secrets, Flow

  secrets, tokens) are never written, only their names, so a restored

  repository asks for them again.

- **Its owner:** for an organisation, its members, teams and their

  permissions; for a user, their profile and linked identity keys

  (public keys only).

Not included: Flow runs' logs and artifacts, which belong to the run, not

the repository (a later choice); caches; anything of another owner.

## 3. Where it goes

A **destination** is one of:

- **A GCS bucket, keylessly.** The hub is already an OIDC issuer (spec

  §13.5). Built: the token lives 10 minutes, its audience is the

  provider's name over https (Google's default), the exchange asks for

  `devstorage.read_write`, and the access token is exchanged again five

  minutes before it runs out. A refusal says what to grant. For a backup it mints a short-lived token with

  `sub = backup:OWNER` and audience the owner's workload identity

  provider. Google's STS exchanges it for a short-lived access token,

  which writes to the bucket. Nothing secret is stored: no service

  account key exists (Andy's organisation blocks them anyway), and if the

  hub leaks a token it lasts minutes and is scoped to that owner's

  prefix. Setup is §6.

- **A directory on the hub's host**, for the site administrator: the

  directories an owner may choose are listed in `hub.toml`

  (`[backups] directories = ["/srv/backups"]`), so nobody can write

  anywhere else on the host. Each owner gets `DIR/OWNER/`.

- **An S3 bucket, keylessly** (built 2026-10-14, oxygenhub#178): the hub

  mints a 10-minute token with `sub = backup:OWNER` and audience

  `sts.amazonaws.com`, AWS STS's AssumeRoleWithWebIdentity gives an

  hour's credentials of the IAM role the owner names (exchanged again

  five minutes before they run out), and each request is signed with

  Signature Version 4 (checked against AWS's published example). The

  setting holds the bucket, its region and the role's ARN; setup is §6.

- **Later:** another hub.

Layout at a destination (`PREFIX` = the owner's chosen prefix):

```

PREFIX/objects/<repository id>/<object hash>    content-addressed, written once

PREFIX/runs/<run time>/<repository id>.json     the refs at that run

PREFIX/runs/<run time>/records/<repository id>.json

PREFIX/runs/<run time>/owner.json

PREFIX/runs/<run time>/DONE                     written last: the run is whole

```

A run without `DONE` was interrupted; a restore ignores it.

## 4. When, how long and how it went

- **How often:** hourly to begin with (off, hourly, daily); finer

  later. The hub's scheduler runs each due backup, at most two at once,

  at low priority, spread across the hour so owners' runs do not

  collide.

- **Back up now:** a button, and `POST /api/v1/{user|orgs/NAME}/backups/run`.

- **Kept:** runs' manifests for 30 days by default (7, 30 or 90).

  Objects no kept run refers to are pruned weekly. With GCS, the

  bucket's own versioning and lifecycle rules work too.

- **Status:** the last run's start, length, objects and bytes sent,

  repositories, and outcome (ok, partial with the repositories that

  failed and why, or failed with the reason, such as "GCS refused the

  token: check the provider's attribute condition"). The last 50 runs

  are listed. A failure notifies the owner (an organisation's owners)

  in the inbox, once per kind of failure until it succeeds again.

## 5. Who sets it, and where

- **A user:** Settings › Backups, for the repositories they own.

- **An organisation's owners:** the organisation's Settings › Backups.

- **The site administrator:** Admin › Backups (built 2026-10-14,

  oxygenhub#176), a table of every owner's backup (how often, where, how

  long runs are kept, the repositories it covers, the last run with its

  outcome, and the runs that failed in the last day), and the owners of

  repositories no backup covers. They may not read an owner's bucket, only

  see that it ran. The site's own backup runs on its host, outside the

  hub, and the page says so.

- **A repository** (built 2026-10-14, oxygenhub#177): its Settings ›

  Backup, or `PUT /repos/{owner}/{name}/backup {"backup": "owner" |

  "hourly" | "daily" | "off"}`, by its owners and admins (migration

  0062). Left out, it is in no new run. With its own schedule, its

  owner's backup runs at least that often, and runs even when the owner's

  own schedule is off (taking only the repositories that asked); a run

  sends only what changed and keeps every included repository, so each

  run stays whole and restores everything. A run asked for by hand takes

  every repository not left out.

The API mirrors the pages (built 2026-10-14, oxygenhub#179; scope

`repo:admin`, an organisation's owners only): `GET/PUT /api/v1/user/backups`

and `GET/PUT /api/v1/orgs/{name}/backups` (the setting, checked as the page

checks it, with where its runs go and the newest runs; the body takes the

page's fields, `keep_days` as a number), `POST …/backups/test` (the probe),

`POST …/backups/run` (202: back up now, in the background) and

`GET …/backups/runs`. `oh backup status|run|restore` follows.

## 6. Setting up a GCS destination (for Andy, by hand)

Once per GCP project:

```

PROJECT=doomsdayone-oxy

NUM=$(gcloud projects describe $PROJECT --format='value(projectNumber)')

gcloud iam workload-identity-pools create oxygenhub \

  --project=$PROJECT --location=global --display-name="OxygenHub"

gcloud iam workload-identity-pools providers create-oidc oxygenhub-io \

  --project=$PROJECT --location=global --workload-identity-pool=oxygenhub \

  --issuer-uri=https://oxygenhub.io \

  --attribute-mapping=google.subject=assertion.sub \

  --attribute-condition="assertion.sub.startsWith('backup:')"

```

Once per owner and bucket (here `andymccall`):

```

gcloud storage buckets create gs://andymccall-oxygen-backup \

  --project=$PROJECT --location=europe-west2 \

  --uniform-bucket-level-access --public-access-prevention

gcloud storage buckets update gs://andymccall-oxygen-backup --versioning

gcloud storage buckets add-iam-policy-binding gs://andymccall-oxygen-backup \

  --role=roles/storage.objectAdmin \

  --member="principal://iam.googleapis.com/projects/$NUM/locations/global/workloadIdentityPools/oxygenhub/subject/backup:andymccall"

```

**Done for doomsdayone-oxy on 2026-10-13:** the pool `oxygenhub`, the

provider `oxygenhub-io` and the buckets `andymccall-oxygen-backup` and

`doomsdayone-oxygen-backup`, each granted to its own `backup:` subject.

Then, in Settings › Backups: destination GCS, bucket

`andymccall-oxygen-backup`, provider

`//iam.googleapis.com/projects/NUM/locations/global/workloadIdentityPools/oxygenhub/providers/oxygenhub-io`,

and TEST, which exchanges a token and writes and deletes a probe object.

An organisation does the same with `backup:ORG`. Only that principal can

write the bucket; the hub never holds a key.

### 6.1 Setting up an S3 destination (for Andy, by hand)

Once per AWS account, the hub as an identity provider (AWS fetches its

keys from `https://oxygenhub.io/.well-known/openid-configuration`):

```

aws iam create-open-id-connect-provider \

  --url https://oxygenhub.io --client-id-list sts.amazonaws.com

```

Once per owner and bucket (here `andymccall`, in `eu-west-2`):

```

B=andymccall-oxygen-backup; ACCOUNT=$(aws sts get-caller-identity --query Account --output text)

aws s3api create-bucket --bucket $B --region eu-west-2 \

  --create-bucket-configuration LocationConstraint=eu-west-2

aws s3api put-bucket-versioning --bucket $B --versioning-configuration Status=Enabled

aws s3api put-public-access-block --bucket $B --public-access-block-configuration \

  BlockPublicAcls=true,IgnorePublicAcls=true,BlockPublicPolicy=true,RestrictPublicBuckets=true

cat > trust.json <<JSON

{"Version": "2012-10-17", "Statement": [{"Effect": "Allow",

  "Principal": {"Federated": "arn:aws:iam::$ACCOUNT:oidc-provider/oxygenhub.io"},

  "Action": "sts:AssumeRoleWithWebIdentity",

  "Condition": {"StringEquals": {"oxygenhub.io:aud": "sts.amazonaws.com",

                                 "oxygenhub.io:sub": "backup:andymccall"}}}]}

JSON

cat > policy.json <<JSON

{"Version": "2012-10-17", "Statement": [

  {"Effect": "Allow", "Action": "s3:ListBucket", "Resource": "arn:aws:s3:::$B"},

  {"Effect": "Allow", "Action": ["s3:GetObject", "s3:PutObject", "s3:DeleteObject"],

   "Resource": "arn:aws:s3:::$B/*"}]}

JSON

aws iam create-role --role-name oxygenhub-backup-andymccall --assume-role-policy-document file://trust.json

aws iam put-role-policy --role-name oxygenhub-backup-andymccall --policy-name backup --policy-document file://policy.json

```

Then, in Settings › Backups: destination S3, the bucket, region

`eu-west-2`, role `arn:aws:iam::ACCOUNT:role/oxygenhub-backup-andymccall`,

SAVE and TEST. `s3:ListBucket` matters: without it S3 answers a missing

object with 403 rather than 404, and every run fails. No access key exists

anywhere.

## 7. Restore

**As built (oxygenhub#175).** `oxygenhub admin backup restore --from DIR

[--run AT]` into a hub that lacks the repositories: each store and wiki

byte for byte (every file checked against its hash), then the

repository's row with its settings as they were (visibility, default

bookmark, Flow and Diffusion settings and the rest), then its records.

Each run's records carry `_names`, the username, organisation or

`org/team` behind every account, team and organisation id their rows

point at, so the restore finds them again by name on any hub. Tables go

in parents first; every row gets a new id and every reference follows

it. A reference to an account this hub lacks is left blank where the

column allows it (the author's name stays in the row) and the row left

out where it does not (an assignment, a review request); the restore

says how many. Webhooks come back without their secrets, each named to be

entered again. Release assets, which live beside the stores rather than

in them, are sent once each as blobs and written back checked against

their hash. Search is rebuilt for what came back. Left out on purpose:

inboxes (notifications), webhook deliveries, runners' declines.

The design as first written:

`oxygenhub admin restore --from gs://BUCKET/PREFIX [--run TIME] [--repo NAME]`

(or a directory) on any hub:

1. reads the run's manifests (the latest whole run by default);

2. creates each repository, or refuses an existing one unless `--into`

   names a new owner;

3. copies its objects, verifying every hash, and sets its refs;

4. imports its records, then reports what needs re-entering (secret

   values, webhook secrets).

A restore into a fresh hub, compared with the original, is the end-to-end

test (BK2), as is restoring one repository from a run before a bad push.

## 8. Order of work

1. **BK2:** settings for users and organisations, the scheduler and

   "back up now", objects, manifests and records to a directory

   destination, status and notifications, and restore with its

   end-to-end test.

2. **BK3:** the GCS destination with the keyless exchange (tested against

   a fake STS and a fake GCS), and TEST.

3. Admin › Backups, then per-repository settings, then S3: all built

   2026-10-14 (#176, #177, #178).