OXYGENHUB
# 7b9f4570daf5 oxygenhub / docs / backups.md
12.7 KiB · Markdown · last changed Backups to S3, keylessly 22 days ago renderedsource blame history raw

Backups: per site, organisation, user and repository

Status: built 2026-10-13/14 (oxygenhub#169): user and organisation settings, the hourly and daily schedule with "back up now" and TEST, directory destinations and keyless GCS destinations, every repository's and wiki's store byte for byte, the records as JSON, retention, failure notices, and restore (oxygenhub admin backup run|restore). Since 2026-10-14 (oxygenhub#175) a restore brings back the records too (§7) and release assets travel with the stores; Admin › Backups (#176); per-repository settings (#177); S3 destinations, keylessly (#178).

1. Why

A forge is where people's work lives. Before Andy moves his GitHub repositories here, and before the hub moves to GCP, every owner must be able to say "my repositories are copied, every hour, to a place I control, and I can bring them back". GitHub leaves this to its users (clone everything somewhere, and lose the issues and reviews); GitLab backs up the whole instance or nothing. Here an owner backs up their own repositories, with everything around them, to their own bucket, without a key file anywhere.

2. What a backup holds

For each repository the owner has (a user's own, or an organisation's):

  • Its objects: the store's content-addressed objects (changes, blobs, chunks, states, identities, signatures, seals, policies). An object never changes once written, so a run sends only the objects the destination does not already have: the first run copies everything, later ones only what arrived since. Nothing is re-encoded, so a restored repository is byte for byte the original, every signature still verifying.
  • Its refs at that moment: bookmarks, tags and the trust policy's serial, as a small manifest per run. Each run's manifest is kept, so a restore can go back to any run that is still kept.
  • Its hub records, as JSON: issues and their comments and events; proposals, their reviews, threads and approvals; labels, milestones, projects; releases and their assets (assets are blobs, sent like objects); the wiki (a repository of its own, backed up with it); and repository settings and webhooks. Secret values (webhook secrets, Flow secrets, tokens) are never written, only their names, so a restored repository asks for them again.
  • Its owner: for an organisation, its members, teams and their permissions; for a user, their profile and linked identity keys (public keys only).

Not included: Flow runs' logs and artifacts, which belong to the run, not the repository (a later choice); caches; anything of another owner.

3. Where it goes

A destination is one of:

  • A GCS bucket, keylessly. The hub is already an OIDC issuer (spec §13.5). Built: the token lives 10 minutes, its audience is the provider's name over https (Google's default), the exchange asks for devstorage.read_write, and the access token is exchanged again five minutes before it runs out. A refusal says what to grant. For a backup it mints a short-lived token with sub = backup:OWNER and audience the owner's workload identity provider. Google's STS exchanges it for a short-lived access token, which writes to the bucket. Nothing secret is stored: no service account key exists (Andy's organisation blocks them anyway), and if the hub leaks a token it lasts minutes and is scoped to that owner's prefix. Setup is §6.
  • A directory on the hub's host, for the site administrator: the directories an owner may choose are listed in hub.toml ([backups] directories = ["/srv/backups"]), so nobody can write anywhere else on the host. Each owner gets DIR/OWNER/.
  • An S3 bucket, keylessly (built 2026-10-14, oxygenhub#178): the hub mints a 10-minute token with sub = backup:OWNER and audience sts.amazonaws.com, AWS STS's AssumeRoleWithWebIdentity gives an hour's credentials of the IAM role the owner names (exchanged again five minutes before they run out), and each request is signed with Signature Version 4 (checked against AWS's published example). The setting holds the bucket, its region and the role's ARN; setup is §6.
  • Later: another hub.

Layout at a destination (PREFIX = the owner's chosen prefix):

PREFIX/objects/<repository id>/<object hash>    content-addressed, written once
PREFIX/runs/<run time>/<repository id>.json     the refs at that run
PREFIX/runs/<run time>/records/<repository id>.json
PREFIX/runs/<run time>/owner.json
PREFIX/runs/<run time>/DONE                     written last: the run is whole

A run without DONE was interrupted; a restore ignores it.

4. When, how long and how it went

  • How often: hourly to begin with (off, hourly, daily); finer later. The hub's scheduler runs each due backup, at most two at once, at low priority, spread across the hour so owners' runs do not collide.
  • Back up now: a button, and POST /api/v1/{user|orgs/NAME}/backups/run.
  • Kept: runs' manifests for 30 days by default (7, 30 or 90). Objects no kept run refers to are pruned weekly. With GCS, the bucket's own versioning and lifecycle rules work too.
  • Status: the last run's start, length, objects and bytes sent, repositories, and outcome (ok, partial with the repositories that failed and why, or failed with the reason, such as "GCS refused the token: check the provider's attribute condition"). The last 50 runs are listed. A failure notifies the owner (an organisation's owners) in the inbox, once per kind of failure until it succeeds again.

5. Who sets it, and where

  • A user: Settings › Backups, for the repositories they own.
  • An organisation's owners: the organisation's Settings › Backups.
  • The site administrator: Admin › Backups (built 2026-10-14, oxygenhub#176), a table of every owner's backup (how often, where, how long runs are kept, the repositories it covers, the last run with its outcome, and the runs that failed in the last day), and the owners of repositories no backup covers. They may not read an owner's bucket, only see that it ran. The site's own backup runs on its host, outside the hub, and the page says so.
  • A repository (built 2026-10-14, oxygenhub#177): its Settings › Backup, or PUT /repos/{owner}/{name}/backup {"backup": "owner" | "hourly" | "daily" | "off"}, by its owners and admins (migration 0062). Left out, it is in no new run. With its own schedule, its owner's backup runs at least that often, and runs even when the owner's own schedule is off (taking only the repositories that asked); a run sends only what changed and keeps every included repository, so each run stays whole and restores everything. A run asked for by hand takes every repository not left out.

The API mirrors the pages: GET/PUT /api/v1/user/backups, GET/PUT /api/v1/orgs/{name}/backups, the run endpoint above, and GET …/backups/runs. oh backup status|run|restore follows.

6. Setting up a GCS destination (for Andy, by hand)

Once per GCP project:

PROJECT=doomsdayone-oxy
NUM=$(gcloud projects describe $PROJECT --format='value(projectNumber)')
gcloud iam workload-identity-pools create oxygenhub \
  --project=$PROJECT --location=global --display-name="OxygenHub"
gcloud iam workload-identity-pools providers create-oidc oxygenhub-io \
  --project=$PROJECT --location=global --workload-identity-pool=oxygenhub \
  --issuer-uri=https://oxygenhub.io \
  --attribute-mapping=google.subject=assertion.sub \
  --attribute-condition="assertion.sub.startsWith('backup:')"

Once per owner and bucket (here andymccall):

gcloud storage buckets create gs://andymccall-oxygen-backup \
  --project=$PROJECT --location=europe-west2 \
  --uniform-bucket-level-access --public-access-prevention
gcloud storage buckets update gs://andymccall-oxygen-backup --versioning
gcloud storage buckets add-iam-policy-binding gs://andymccall-oxygen-backup \
  --role=roles/storage.objectAdmin \
  --member="principal://iam.googleapis.com/projects/$NUM/locations/global/workloadIdentityPools/oxygenhub/subject/backup:andymccall"

Done for doomsdayone-oxy on 2026-10-13: the pool oxygenhub, the provider oxygenhub-io and the buckets andymccall-oxygen-backup and doomsdayone-oxygen-backup, each granted to its own backup: subject.

Then, in Settings › Backups: destination GCS, bucket andymccall-oxygen-backup, provider //iam.googleapis.com/projects/NUM/locations/global/workloadIdentityPools/oxygenhub/providers/oxygenhub-io, and TEST, which exchanges a token and writes and deletes a probe object. An organisation does the same with backup:ORG. Only that principal can write the bucket; the hub never holds a key.

6.1 Setting up an S3 destination (for Andy, by hand)

Once per AWS account, the hub as an identity provider (AWS fetches its keys from https://oxygenhub.io/.well-known/openid-configuration):

aws iam create-open-id-connect-provider \
  --url https://oxygenhub.io --client-id-list sts.amazonaws.com

Once per owner and bucket (here andymccall, in eu-west-2):

B=andymccall-oxygen-backup; ACCOUNT=$(aws sts get-caller-identity --query Account --output text)
aws s3api create-bucket --bucket $B --region eu-west-2 \
  --create-bucket-configuration LocationConstraint=eu-west-2
aws s3api put-bucket-versioning --bucket $B --versioning-configuration Status=Enabled
aws s3api put-public-access-block --bucket $B --public-access-block-configuration \
  BlockPublicAcls=true,IgnorePublicAcls=true,BlockPublicPolicy=true,RestrictPublicBuckets=true
cat > trust.json <<JSON
{"Version": "2012-10-17", "Statement": [{"Effect": "Allow",
  "Principal": {"Federated": "arn:aws:iam::$ACCOUNT:oidc-provider/oxygenhub.io"},
  "Action": "sts:AssumeRoleWithWebIdentity",
  "Condition": {"StringEquals": {"oxygenhub.io:aud": "sts.amazonaws.com",
                                 "oxygenhub.io:sub": "backup:andymccall"}}}]}
JSON
cat > policy.json <<JSON
{"Version": "2012-10-17", "Statement": [
  {"Effect": "Allow", "Action": "s3:ListBucket", "Resource": "arn:aws:s3:::$B"},
  {"Effect": "Allow", "Action": ["s3:GetObject", "s3:PutObject", "s3:DeleteObject"],
   "Resource": "arn:aws:s3:::$B/*"}]}
JSON
aws iam create-role --role-name oxygenhub-backup-andymccall --assume-role-policy-document file://trust.json
aws iam put-role-policy --role-name oxygenhub-backup-andymccall --policy-name backup --policy-document file://policy.json

Then, in Settings › Backups: destination S3, the bucket, region eu-west-2, role arn:aws:iam::ACCOUNT:role/oxygenhub-backup-andymccall, SAVE and TEST. s3:ListBucket matters: without it S3 answers a missing object with 403 rather than 404, and every run fails. No access key exists anywhere.

7. Restore

As built (oxygenhub#175). oxygenhub admin backup restore --from DIR [--run AT] into a hub that lacks the repositories: each store and wiki byte for byte (every file checked against its hash), then the repository's row with its settings as they were (visibility, default bookmark, Flow and Diffusion settings and the rest), then its records. Each run's records carry _names, the username, organisation or org/team behind every account, team and organisation id their rows point at, so the restore finds them again by name on any hub. Tables go in parents first; every row gets a new id and every reference follows it. A reference to an account this hub lacks is left blank where the column allows it (the author's name stays in the row) and the row left out where it does not (an assignment, a review request); the restore says how many. Webhooks come back without their secrets, each named to be entered again. Release assets, which live beside the stores rather than in them, are sent once each as blobs and written back checked against their hash. Search is rebuilt for what came back. Left out on purpose: inboxes (notifications), webhook deliveries, runners' declines.

The design as first written:

oxygenhub admin restore --from gs://BUCKET/PREFIX [--run TIME] [--repo NAME] (or a directory) on any hub:

  1. reads the run's manifests (the latest whole run by default);
  2. creates each repository, or refuses an existing one unless --into names a new owner;
  3. copies its objects, verifying every hash, and sets its refs;
  4. imports its records, then reports what needs re-entering (secret values, webhook secrets).

A restore into a fresh hub, compared with the original, is the end-to-end test (BK2), as is restoring one repository from a run before a bad push.

8. Order of work

  1. BK2: settings for users and organisations, the scheduler and "back up now", objects, manifests and records to a directory destination, status and notifications, and restore with its end-to-end test.
  2. BK3: the GCS destination with the keyless exchange (tested against a fake STS and a fake GCS), and TEST.
  3. Admin › Backups, then per-repository settings, then S3: all built 2026-10-14 (#176, #177, #178).