# Backups: per site, organisation, user and repository Status: built 2026-10-13/14 (oxygenhub#169): user and organisation settings, the hourly and daily schedule with "back up now" and TEST, directory destinations and keyless GCS destinations, every repository's and wiki's store byte for byte, the records as JSON, retention, failure notices, and restore (`oxygenhub admin backup run|restore`). Since 2026-10-14 (oxygenhub#175) a restore brings back the records too (§7) and release assets travel with the stores; Admin › Backups (#176); per-repository settings (#177); S3 destinations, keylessly (#178). ## 1. Why A forge is where people's work lives. Before Andy moves his GitHub repositories here, and before the hub moves to GCP, every owner must be able to say "my repositories are copied, every hour, to a place I control, and I can bring them back". GitHub leaves this to its users (clone everything somewhere, and lose the issues and reviews); GitLab backs up the whole instance or nothing. Here an owner backs up their own repositories, with everything around them, to their own bucket, without a key file anywhere. ## 2. What a backup holds For each repository the owner has (a user's own, or an organisation's): - **Its objects:** the store's content-addressed objects (changes, blobs, chunks, states, identities, signatures, seals, policies). An object never changes once written, so a run sends only the objects the destination does not already have: the first run copies everything, later ones only what arrived since. Nothing is re-encoded, so a restored repository is byte for byte the original, every signature still verifying. - **Its refs at that moment:** bookmarks, tags and the trust policy's serial, as a small manifest per run. Each run's manifest is kept, so a restore can go back to any run that is still kept. - **Its hub records, as JSON:** issues and their comments and events; proposals, their reviews, threads and approvals; labels, milestones, projects; releases and their assets (assets are blobs, sent like objects); the wiki (a repository of its own, backed up with it); and repository settings and webhooks. Secret values (webhook secrets, Flow secrets, tokens) are never written, only their names, so a restored repository asks for them again. - **Its owner:** for an organisation, its members, teams and their permissions; for a user, their profile and linked identity keys (public keys only). Not included: Flow runs' logs and artifacts, which belong to the run, not the repository (a later choice); caches; anything of another owner. ## 3. Where it goes A **destination** is one of: - **A GCS bucket, keylessly.** The hub is already an OIDC issuer (spec §13.5). Built: the token lives 10 minutes, its audience is the provider's name over https (Google's default), the exchange asks for `devstorage.read_write`, and the access token is exchanged again five minutes before it runs out. A refusal says what to grant. For a backup it mints a short-lived token with `sub = backup:OWNER` and audience the owner's workload identity provider. Google's STS exchanges it for a short-lived access token, which writes to the bucket. Nothing secret is stored: no service account key exists (Andy's organisation blocks them anyway), and if the hub leaks a token it lasts minutes and is scoped to that owner's prefix. Setup is §6. - **A directory on the hub's host**, for the site administrator: the directories an owner may choose are listed in `hub.toml` (`[backups] directories = ["/srv/backups"]`), so nobody can write anywhere else on the host. Each owner gets `DIR/OWNER/`. - **An S3 bucket, keylessly** (built 2026-10-14, oxygenhub#178): the hub mints a 10-minute token with `sub = backup:OWNER` and audience `sts.amazonaws.com`, AWS STS's AssumeRoleWithWebIdentity gives an hour's credentials of the IAM role the owner names (exchanged again five minutes before they run out), and each request is signed with Signature Version 4 (checked against AWS's published example). The setting holds the bucket, its region and the role's ARN; setup is §6. - **Later:** another hub. Layout at a destination (`PREFIX` = the owner's chosen prefix): ``` PREFIX/objects// content-addressed, written once PREFIX/runs//.json the refs at that run PREFIX/runs//records/.json PREFIX/runs//owner.json PREFIX/runs//DONE written last: the run is whole ``` A run without `DONE` was interrupted; a restore ignores it. ## 4. When, how long and how it went - **How often:** hourly to begin with (off, hourly, daily); finer later. The hub's scheduler runs each due backup, at most two at once, at low priority, spread across the hour so owners' runs do not collide. - **Back up now:** a button, and `POST /api/v1/{user|orgs/NAME}/backups/run`. - **Kept:** runs' manifests for 30 days by default (7, 30 or 90). Objects no kept run refers to are pruned weekly. With GCS, the bucket's own versioning and lifecycle rules work too. - **Status:** the last run's start, length, objects and bytes sent, repositories, and outcome (ok, partial with the repositories that failed and why, or failed with the reason, such as "GCS refused the token: check the provider's attribute condition"). The last 50 runs are listed. A failure notifies the owner (an organisation's owners) in the inbox, once per kind of failure until it succeeds again. ## 5. Who sets it, and where - **A user:** Settings › Backups, for the repositories they own. - **An organisation's owners:** the organisation's Settings › Backups. - **The site administrator:** Admin › Backups (built 2026-10-14, oxygenhub#176), a table of every owner's backup (how often, where, how long runs are kept, the repositories it covers, the last run with its outcome, and the runs that failed in the last day), and the owners of repositories no backup covers. They may not read an owner's bucket, only see that it ran. The site's own backup runs on its host, outside the hub, and the page says so. - **A repository** (built 2026-10-14, oxygenhub#177): its Settings › Backup, or `PUT /repos/{owner}/{name}/backup {"backup": "owner" | "hourly" | "daily" | "off"}`, by its owners and admins (migration 0062). Left out, it is in no new run. With its own schedule, its owner's backup runs at least that often, and runs even when the owner's own schedule is off (taking only the repositories that asked); a run sends only what changed and keeps every included repository, so each run stays whole and restores everything. A run asked for by hand takes every repository not left out. The API mirrors the pages (built 2026-10-14, oxygenhub#179; scope `repo:admin`, an organisation's owners only): `GET/PUT /api/v1/user/backups` and `GET/PUT /api/v1/orgs/{name}/backups` (the setting, checked as the page checks it, with where its runs go and the newest runs; the body takes the page's fields, `keep_days` as a number), `POST …/backups/test` (the probe), `POST …/backups/run` (202: back up now, in the background) and `GET …/backups/runs`. `oh backup status|run|restore` follows. ## 6. Setting up a GCS destination (for Andy, by hand) Once per GCP project: ``` PROJECT=doomsdayone-oxy NUM=$(gcloud projects describe $PROJECT --format='value(projectNumber)') gcloud iam workload-identity-pools create oxygenhub \ --project=$PROJECT --location=global --display-name="OxygenHub" gcloud iam workload-identity-pools providers create-oidc oxygenhub-io \ --project=$PROJECT --location=global --workload-identity-pool=oxygenhub \ --issuer-uri=https://oxygenhub.io \ --attribute-mapping=google.subject=assertion.sub \ --attribute-condition="assertion.sub.startsWith('backup:')" ``` Once per owner and bucket (here `andymccall`): ``` gcloud storage buckets create gs://andymccall-oxygen-backup \ --project=$PROJECT --location=europe-west2 \ --uniform-bucket-level-access --public-access-prevention gcloud storage buckets update gs://andymccall-oxygen-backup --versioning gcloud storage buckets add-iam-policy-binding gs://andymccall-oxygen-backup \ --role=roles/storage.objectAdmin \ --member="principal://iam.googleapis.com/projects/$NUM/locations/global/workloadIdentityPools/oxygenhub/subject/backup:andymccall" ``` **Done for doomsdayone-oxy on 2026-10-13:** the pool `oxygenhub`, the provider `oxygenhub-io` and the buckets `andymccall-oxygen-backup` and `doomsdayone-oxygen-backup`, each granted to its own `backup:` subject. Then, in Settings › Backups: destination GCS, bucket `andymccall-oxygen-backup`, provider `//iam.googleapis.com/projects/NUM/locations/global/workloadIdentityPools/oxygenhub/providers/oxygenhub-io`, and TEST, which exchanges a token and writes and deletes a probe object. An organisation does the same with `backup:ORG`. Only that principal can write the bucket; the hub never holds a key. ### 6.1 Setting up an S3 destination (for Andy, by hand) Once per AWS account, the hub as an identity provider (AWS fetches its keys from `https://oxygenhub.io/.well-known/openid-configuration`): ``` aws iam create-open-id-connect-provider \ --url https://oxygenhub.io --client-id-list sts.amazonaws.com ``` Once per owner and bucket (here `andymccall`, in `eu-west-2`): ``` B=andymccall-oxygen-backup; ACCOUNT=$(aws sts get-caller-identity --query Account --output text) aws s3api create-bucket --bucket $B --region eu-west-2 \ --create-bucket-configuration LocationConstraint=eu-west-2 aws s3api put-bucket-versioning --bucket $B --versioning-configuration Status=Enabled aws s3api put-public-access-block --bucket $B --public-access-block-configuration \ BlockPublicAcls=true,IgnorePublicAcls=true,BlockPublicPolicy=true,RestrictPublicBuckets=true cat > trust.json < policy.json <