docs: backups per site, organisation, user and repository, designed (refs #169) 6dbd999ad794

# Backups: per site, organisation, user and repository

Status: design, 2026-10-13 (oxygenhub#169). Hourly backups configured by a

user and by an organisation come first; finer-grained ones (per

repository, more often than hourly) later. The site's own backup

(everything that runs a hub, #168) is an operations script today

(`~/backups/oxygenhub/backup.sh`); Admin › Backups shows it later.

## 1. Why

A forge is where people's work lives. Before Andy moves his GitHub

repositories here, and before the hub moves to GCP, every owner must be

able to say "my repositories are copied, every hour, to a place I

control, and I can bring them back". GitHub leaves this to its users

(clone everything somewhere, and lose the issues and reviews); GitLab

backs up the whole instance or nothing. Here an owner backs up their own

repositories, with everything around them, to their own bucket, without a

key file anywhere.

## 2. What a backup holds

For each repository the owner has (a user's own, or an organisation's):

- **Its objects:** the store's content-addressed objects (changes,

  blobs, chunks, states, identities, signatures, seals, policies). An

  object never changes once written, so a run sends only the objects the

  destination does not already have: the first run copies everything,

  later ones only what arrived since. Nothing is re-encoded, so a

  restored repository is byte for byte the original, every signature

  still verifying.

- **Its refs at that moment:** bookmarks, tags and the trust policy's

  serial, as a small manifest per run. Each run's manifest is kept, so a

  restore can go back to any run that is still kept.

- **Its hub records, as JSON:** issues and their comments and events;

  proposals, their reviews, threads and approvals; labels, milestones,

  projects; releases and their assets (assets are blobs, sent like

  objects); the wiki (a repository of its own, backed up with it); and

  repository settings and webhooks. Secret values (webhook secrets, Flow

  secrets, tokens) are never written, only their names, so a restored

  repository asks for them again.

- **Its owner:** for an organisation, its members, teams and their

  permissions; for a user, their profile and linked identity keys

  (public keys only).

Not included: Flow runs' logs and artifacts, which belong to the run, not

the repository (a later choice); caches; anything of another owner.

## 3. Where it goes

A **destination** is one of:

- **A GCS bucket, keylessly.** The hub is already an OIDC issuer (spec

  §13.5). For a backup it mints a short-lived token with

  `sub = backup:OWNER` and audience the owner's workload identity

  provider. Google's STS exchanges it for a short-lived access token,

  which writes to the bucket. Nothing secret is stored: no service

  account key exists (Andy's organisation blocks them anyway), and if the

  hub leaks a token it lasts minutes and is scoped to that owner's

  prefix. Setup is §6.

- **A directory on the hub's host**, for the site administrator: the

  directories an owner may choose are listed in `hub.toml`

  (`[backups] directories = ["/srv/backups"]`), so nobody can write

  anywhere else on the host. Each owner gets `DIR/OWNER/`.

- **Later:** S3 (the same keyless exchange with AWS STS), and another hub.

Layout at a destination (`PREFIX` = the owner's chosen prefix):

```

PREFIX/objects/<repository id>/<object hash>    content-addressed, written once

PREFIX/runs/<run time>/<repository id>.json     the refs at that run

PREFIX/runs/<run time>/records/<repository id>.json

PREFIX/runs/<run time>/owner.json

PREFIX/runs/<run time>/DONE                     written last: the run is whole

```

A run without `DONE` was interrupted; a restore ignores it.

## 4. When, how long and how it went

- **How often:** hourly to begin with (off, hourly, daily); finer

  later. The hub's scheduler runs each due backup, at most two at once,

  at low priority, spread across the hour so owners' runs do not

  collide.

- **Back up now:** a button, and `POST /api/v1/{user|orgs/NAME}/backups/run`.

- **Kept:** runs' manifests for 30 days by default (7, 30 or 90).

  Objects no kept run refers to are pruned weekly. With GCS, the

  bucket's own versioning and lifecycle rules work too.

- **Status:** the last run's start, length, objects and bytes sent,

  repositories, and outcome (ok, partial with the repositories that

  failed and why, or failed with the reason, such as "GCS refused the

  token: check the provider's attribute condition"). The last 50 runs

  are listed. A failure notifies the owner (an organisation's owners)

  in the inbox, once per kind of failure until it succeeds again.

## 5. Who sets it, and where

- **A user:** Settings › Backups, for the repositories they own.

- **An organisation's owners:** the organisation's Settings › Backups.

- **The site administrator:** Admin › Backups, a table of every owner's

  backup (destination, schedule, last run, outcome) and the site's own

  backup. They may not read an owner's bucket, only see that it ran.

- **A repository** (later): opt a repository out of its owner's backup,

  or give it its own schedule.

The API mirrors the pages: `GET/PUT /api/v1/user/backups`,

`GET/PUT /api/v1/orgs/{name}/backups`, the run endpoint above, and

`GET …/backups/runs`. `oh backup status|run|restore` follows.

## 6. Setting up a GCS destination (for Andy, by hand)

Once per GCP project:

```

PROJECT=doomsdayone-oxy

NUM=$(gcloud projects describe $PROJECT --format='value(projectNumber)')

gcloud iam workload-identity-pools create oxygenhub \

  --project=$PROJECT --location=global --display-name="OxygenHub"

gcloud iam workload-identity-pools providers create-oidc oxygenhub-io \

  --project=$PROJECT --location=global --workload-identity-pool=oxygenhub \

  --issuer-uri=https://oxygenhub.io \

  --attribute-mapping=google.subject=assertion.sub \

  --attribute-condition="assertion.sub.startsWith('backup:')"

```

Once per owner and bucket (here `andymccall`):

```

gcloud storage buckets create gs://andymccall-oxygen-backup \

  --project=$PROJECT --location=europe-west2 \

  --uniform-bucket-level-access --public-access-prevention

gcloud storage buckets update gs://andymccall-oxygen-backup --versioning

gcloud storage buckets add-iam-policy-binding gs://andymccall-oxygen-backup \

  --role=roles/storage.objectAdmin \

  --member="principal://iam.googleapis.com/projects/$NUM/locations/global/workloadIdentityPools/oxygenhub/subject/backup:andymccall"

```

Then, in Settings › Backups: destination GCS, bucket

`andymccall-oxygen-backup`, provider

`//iam.googleapis.com/projects/NUM/locations/global/workloadIdentityPools/oxygenhub/providers/oxygenhub-io`,

and TEST, which exchanges a token and writes and deletes a probe object.

An organisation does the same with `backup:ORG`. Only that principal can

write the bucket; the hub never holds a key.

## 7. Restore

`oxygenhub admin restore --from gs://BUCKET/PREFIX [--run TIME] [--repo NAME]`

(or a directory) on any hub:

1. reads the run's manifests (the latest whole run by default);

2. creates each repository, or refuses an existing one unless `--into`

   names a new owner;

3. copies its objects, verifying every hash, and sets its refs;

4. imports its records, then reports what needs re-entering (secret

   values, webhook secrets).

A restore into a fresh hub, compared with the original, is the end-to-end

test (BK2), as is restoring one repository from a run before a bad push.

## 8. Order of work

1. **BK2:** settings for users and organisations, the scheduler and

   "back up now", objects, manifests and records to a directory

   destination, status and notifications, and restore with its

   end-to-end test.

2. **BK3:** the GCS destination with the keyless exchange (tested against

   a fake STS and a fake GCS), and TEST.

3. Admin › Backups, then per-repository settings, then S3.

### 13.5.1 Backups (designed 2026-10-13, oxygenhub#169)

Each user and organisation backs up its repositories hourly to a place it controls: a GCS bucket reached keylessly (the hub mints a short-lived token with `sub = backup:OWNER`, which GCP's workload identity federation exchanges for an access token, so no key exists), or a directory the site allows. A run sends only the content-addressed objects the destination lacks, the refs at that moment and the hub's records (issues, proposals, releases, wiki, settings; never secret values), and is whole once its `DONE` marker is written; `oxygenhub admin restore` brings repositories back from any whole run. The design, the layout at a destination, setup and restore are in `docs/backups.md`.