# Backups: per site, organisation, user and repository

Status: built 2026-10-13/14 (oxygenhub#169): user and organisation
settings, the hourly and daily schedule with "back up now" and TEST,
directory destinations and keyless GCS destinations, every repository's
and wiki's store byte for byte, the records as JSON, retention, failure
notices, and restore of the stores and repository rows
(`oxygenhub admin backup run|restore`). Next: restoring issues and
proposals from the records, Admin › Backups, per-repository settings.

## 1. Why

A forge is where people's work lives. Before Andy moves his GitHub
repositories here, and before the hub moves to GCP, every owner must be
able to say "my repositories are copied, every hour, to a place I
control, and I can bring them back". GitHub leaves this to its users
(clone everything somewhere, and lose the issues and reviews); GitLab
backs up the whole instance or nothing. Here an owner backs up their own
repositories, with everything around them, to their own bucket, without a
key file anywhere.

## 2. What a backup holds

For each repository the owner has (a user's own, or an organisation's):

- **Its objects:** the store's content-addressed objects (changes,
  blobs, chunks, states, identities, signatures, seals, policies). An
  object never changes once written, so a run sends only the objects the
  destination does not already have: the first run copies everything,
  later ones only what arrived since. Nothing is re-encoded, so a
  restored repository is byte for byte the original, every signature
  still verifying.
- **Its refs at that moment:** bookmarks, tags and the trust policy's
  serial, as a small manifest per run. Each run's manifest is kept, so a
  restore can go back to any run that is still kept.
- **Its hub records, as JSON:** issues and their comments and events;
  proposals, their reviews, threads and approvals; labels, milestones,
  projects; releases and their assets (assets are blobs, sent like
  objects); the wiki (a repository of its own, backed up with it); and
  repository settings and webhooks. Secret values (webhook secrets, Flow
  secrets, tokens) are never written, only their names, so a restored
  repository asks for them again.
- **Its owner:** for an organisation, its members, teams and their
  permissions; for a user, their profile and linked identity keys
  (public keys only).

Not included: Flow runs' logs and artifacts, which belong to the run, not
the repository (a later choice); caches; anything of another owner.

## 3. Where it goes

A **destination** is one of:

- **A GCS bucket, keylessly.** The hub is already an OIDC issuer (spec
  §13.5). Built: the token lives 10 minutes, its audience is the
  provider's name over https (Google's default), the exchange asks for
  `devstorage.read_write`, and the access token is exchanged again five
  minutes before it runs out. A refusal says what to grant. For a backup it mints a short-lived token with
  `sub = backup:OWNER` and audience the owner's workload identity
  provider. Google's STS exchanges it for a short-lived access token,
  which writes to the bucket. Nothing secret is stored: no service
  account key exists (Andy's organisation blocks them anyway), and if the
  hub leaks a token it lasts minutes and is scoped to that owner's
  prefix. Setup is §6.
- **A directory on the hub's host**, for the site administrator: the
  directories an owner may choose are listed in `hub.toml`
  (`[backups] directories = ["/srv/backups"]`), so nobody can write
  anywhere else on the host. Each owner gets `DIR/OWNER/`.
- **Later:** S3 (the same keyless exchange with AWS STS), and another hub.

Layout at a destination (`PREFIX` = the owner's chosen prefix):

```
PREFIX/objects/<repository id>/<object hash>    content-addressed, written once
PREFIX/runs/<run time>/<repository id>.json     the refs at that run
PREFIX/runs/<run time>/records/<repository id>.json
PREFIX/runs/<run time>/owner.json
PREFIX/runs/<run time>/DONE                     written last: the run is whole
```

A run without `DONE` was interrupted; a restore ignores it.

## 4. When, how long and how it went

- **How often:** hourly to begin with (off, hourly, daily); finer
  later. The hub's scheduler runs each due backup, at most two at once,
  at low priority, spread across the hour so owners' runs do not
  collide.
- **Back up now:** a button, and `POST /api/v1/{user|orgs/NAME}/backups/run`.
- **Kept:** runs' manifests for 30 days by default (7, 30 or 90).
  Objects no kept run refers to are pruned weekly. With GCS, the
  bucket's own versioning and lifecycle rules work too.
- **Status:** the last run's start, length, objects and bytes sent,
  repositories, and outcome (ok, partial with the repositories that
  failed and why, or failed with the reason, such as "GCS refused the
  token: check the provider's attribute condition"). The last 50 runs
  are listed. A failure notifies the owner (an organisation's owners)
  in the inbox, once per kind of failure until it succeeds again.

## 5. Who sets it, and where

- **A user:** Settings › Backups, for the repositories they own.
- **An organisation's owners:** the organisation's Settings › Backups.
- **The site administrator:** Admin › Backups, a table of every owner's
  backup (destination, schedule, last run, outcome) and the site's own
  backup. They may not read an owner's bucket, only see that it ran.
- **A repository** (later): opt a repository out of its owner's backup,
  or give it its own schedule.

The API mirrors the pages: `GET/PUT /api/v1/user/backups`,
`GET/PUT /api/v1/orgs/{name}/backups`, the run endpoint above, and
`GET …/backups/runs`. `oh backup status|run|restore` follows.

## 6. Setting up a GCS destination (for Andy, by hand)

Once per GCP project:

```
PROJECT=doomsdayone-oxy
NUM=$(gcloud projects describe $PROJECT --format='value(projectNumber)')
gcloud iam workload-identity-pools create oxygenhub \
  --project=$PROJECT --location=global --display-name="OxygenHub"
gcloud iam workload-identity-pools providers create-oidc oxygenhub-io \
  --project=$PROJECT --location=global --workload-identity-pool=oxygenhub \
  --issuer-uri=https://oxygenhub.io \
  --attribute-mapping=google.subject=assertion.sub \
  --attribute-condition="assertion.sub.startsWith('backup:')"
```

Once per owner and bucket (here `andymccall`):

```
gcloud storage buckets create gs://andymccall-oxygen-backup \
  --project=$PROJECT --location=europe-west2 \
  --uniform-bucket-level-access --public-access-prevention
gcloud storage buckets update gs://andymccall-oxygen-backup --versioning
gcloud storage buckets add-iam-policy-binding gs://andymccall-oxygen-backup \
  --role=roles/storage.objectAdmin \
  --member="principal://iam.googleapis.com/projects/$NUM/locations/global/workloadIdentityPools/oxygenhub/subject/backup:andymccall"
```

**Done for doomsdayone-oxy on 2026-10-13:** the pool `oxygenhub`, the
provider `oxygenhub-io` and the buckets `andymccall-oxygen-backup` and
`doomsdayone-oxygen-backup`, each granted to its own `backup:` subject.

Then, in Settings › Backups: destination GCS, bucket
`andymccall-oxygen-backup`, provider
`//iam.googleapis.com/projects/NUM/locations/global/workloadIdentityPools/oxygenhub/providers/oxygenhub-io`,
and TEST, which exchanges a token and writes and deletes a probe object.
An organisation does the same with `backup:ORG`. Only that principal can
write the bucket; the hub never holds a key.

## 7. Restore

`oxygenhub admin restore --from gs://BUCKET/PREFIX [--run TIME] [--repo NAME]`
(or a directory) on any hub:

1. reads the run's manifests (the latest whole run by default);
2. creates each repository, or refuses an existing one unless `--into`
   names a new owner;
3. copies its objects, verifying every hash, and sets its refs;
4. imports its records, then reports what needs re-entering (secret
   values, webhook secrets).

A restore into a fresh hub, compared with the original, is the end-to-end
test (BK2), as is restoring one repository from a run before a bad push.

## 8. Order of work

1. **BK2:** settings for users and organisations, the scheduler and
   "back up now", objects, manifests and records to a directory
   destination, status and notifications, and restore with its
   end-to-end test.
2. **BK3:** the GCS destination with the keyless exchange (tested against
   a fake STS and a fake GCS), and TEST.
3. Admin › Backups, then per-repository settings, then S3.