# Deploying the developer edition on Earth

Companion to `spec.md` §17.5. Status: **installed and running** since 2026-09-27 as the systemd service `oxygenhub` under the `oxygenhub` system user, data in `/home/oxygenhub/data`, HTTP on 7300 and HTTPS on 7343 with a self-signed certificate. `oxygen`, `oxygen-vscode` and `oxygenhub` are hosted under `andymccall/` and each checkout has an `earth` remote.

## The host

| | |
|---|---|
| Name | `earth` (`earth.andymccall.xyz`, IPv6 ULA; `192.168.1.6` on the LAN) |
| OS | Ubuntu 26.04.1 LTS, x86_64, 4 cores, 31 GiB RAM |
| Disk | `/` 251 GB (95 GB free), `/home` 1 TB (944 GB free) |
| Already running | sshd :22, a Python service :6400, `wren-nest` :7842, retroplay-manager in Docker (:8080, :6670) with Postgres and a tetriblox server, two GitHub Actions runners (`doomsdayonecom`), Docker, containerd |
| Not installed | Rust toolchain, sqlite3 CLI, nginx or caddy, certbot |
| Access | `ssh -i ~/.ssh/earth_claude claude@earth`; `claude` has passwordless sudo |

Ports 7300 (HTTP) and 7343 (HTTPS) are free and are the hub's.

## Steps

1. **Toolchain** (as `claude`): `curl https://sh.rustup.rs -sSf | sh -s -- -y --profile minimal`, then `sudo apt-get install -y build-essential pkg-config git`.
2. **Source**: from the development machine, `rsync -a --delete --exclude target --exclude node_modules ~/development/oxygen ~/development/oxygenhub claude@earth:~/src/`. Oxygen is never pushed to Git; rsync is the bootstrap, and after the hub runs, `oxy push` to the hub itself is the ongoing path.
3. **Build**: `cd ~/src/oxygenhub && cargo build --release`.
4. **System user and data**: `sudo useradd --system --home /home/oxygenhub --create-home oxygenhub`, `sudo install -o oxygenhub -g oxygenhub -m 0755 target/release/oxygenhub /usr/local/bin/`, `sudo -u oxygenhub oxygenhub init --data /home/oxygenhub/data --admin andymccall` (prints the first admin's one-time password).
5. **Service**: `/etc/systemd/system/oxygenhub.service` running `oxygenhub serve --data /home/oxygenhub/data` as `oxygenhub`, `Restart=on-failure`, `ProtectSystem=strict`, `ReadWritePaths=/home/oxygenhub/data`, `NoNewPrivileges=yes`; `sudo systemctl enable --now oxygenhub`.
6. **TLS**: the developer edition generates a self-signed certificate at `init`. Download the certificate from `https://earth.andymccall.xyz:7343/.well-known/oxyhub-ca.pem` (or over plain HTTP on port 7300) and point clients at it with `oxy config remote.<name>.ca <file>` or `OXY_CA`.
7. **First repositories**: on the development machine, `oxy remote add earth https://earth.andymccall.xyz:7343/andymccall/oxygen` in each of `oxygen`, `oxygen-vscode`, `oxygenhub`, then `oxy push earth`.
8. **Updates**: rsync, build, `sudo install`, `sudo systemctl restart oxygenhub`. `make deploy` in the hub wraps steps 2, 3 and 8.

## Restarts without errors (oxygenhub#233)

A deploy restarts the hub. Since #233, on SIGTERM (what `systemctl restart`
sends) the hub stops taking connections, ends its live event streams
(browsers reconnect by themselves) and gives the requests in flight up to
30 seconds to finish. Requests that arrive *between* the old process and the
new one are refused unless systemd holds the listening socket across the
restart: socket activation. The hub serves a socket systemd passes
(`LISTEN_FDS`) and binds `server.bind` itself otherwise.

To turn it on, add `/etc/systemd/system/oxygenhub.socket`:

```ini
[Unit]
Description=Oxygenhub's listening socket, held across restarts

[Socket]
ListenStream=0.0.0.0:7300
# A restart's queue: connections wait here, not refused.
Backlog=512
NoDelay=yes

[Install]
WantedBy=sockets.target
```

Then give `oxygenhub.service` `Requires=oxygenhub.socket`,
`After=oxygenhub.socket` and `TimeoutStopSec=40`, and run:

```sh
sudo systemctl daemon-reload
sudo systemctl stop oxygenhub
sudo systemctl enable --now oxygenhub.socket
sudo systemctl start oxygenhub
```

The service's log then says "http on the socket systemd holds". From then on
`systemctl restart oxygenhub` keeps the socket, so nobody sees a refused
connection or a Cloudflare 502. HTTPS (7343, the developer edition's
self-signed listener) still binds itself.

## Where things are on Earth

| | |
|---|---|
| Binaries | `/usr/local/bin/oxygenhub`, `/usr/local/bin/oxy` |
| Sources | `/home/claude/src/oxygen`, `/home/claude/src/oxygenhub` (rsync targets; the hub itself is the canonical remote) |
| Data | `/home/oxygenhub/data` (`hub.toml`, `hub.db`, `keys/`, `repos/andymccall/*`) |
| Push token | `/home/oxygenhub/push-token` (also in `hub.toml`); each local checkout has it as `remote.earth.token` |
| Certificate | `/home/oxygenhub/data/keys/tls.crt`, served at `/.well-known/oxyhub-ca.pem`; local copy `~/.config/oxy/earth-ca.pem` set as `remote.earth.ca` |
| Service | `sudo systemctl status oxygenhub`, `sudo journalctl -u oxygenhub` |

## HTTPS only (oxygenhub#218)

When `base_url` is `https://…`, the hub takes nothing over plain http
through a proxy: a request that `X-Forwarded-Proto` (or Cloudflare's
`CF-Visitor`) reports as http is redirected to the same path on
`base_url` (GET and HEAD) or refused with 400 (anything else), and every
cookie it sets is `Secure`. Requests made to the hub directly, such as a
runner on the same host using `http://127.0.0.1:7300`, carry no such
header and are served as before. In front, Cloudflare's Always Use HTTPS
is on (2026-10-15). Strict-Transport-Security comes from `[server] hsts_max_age` (seconds;
0, the default, sends none; never over plain http; oxygenhub#231).
Browsers remember it, so raise it in stages: 300 first, a week
(604800) after a few quiet days, then a year (31536000). Add
`includeSubDomains` only once every subdomain is HTTPS, and preload last.
oxygenhub.io is at 300 from 2026-10-16.

## Security headers (oxygenhub#222)

Every response carries `X-Content-Type-Options: nosniff`,
`Referrer-Policy: strict-origin-when-cross-origin` and a
`Permissions-Policy` that turns off the camera, microphone, location and
the like; pages also carry `X-Frame-Options: DENY` and a Content Security
Policy (`frame-ancestors 'none'`, scripts and forms from the hub only,
WebAssembly for the backdrop). The policy is **report-only** for now:
browsers report what it would block to `/csp-report`, and the hub logs
each distinct violation once an hour (`csp would block: …`). The pages
still use inline scripts and handlers, which it reports; once they have
moved into `hub.js` and the log is quiet, `ENFORCE` in
`crates/hub/src/headers.rs` switches it on. Responses with their own
policy (report frames, a job's HTML artifact) keep it. HSTS is not sent:
it waits for the site owner's decision.

## security.txt (oxygenhub#224)

Administration › Security contact sets where vulnerabilities are reported
(an email address or an https page) and, optionally, a policy page. The
hub then serves `/.well-known/security.txt` (and `/security.txt`) per
RFC 9116, with an `Expires` date that is always just under a year ahead,
so it never goes stale. With no contact set, both answer 404.
`/security-contact` is the page to point the contact at: the reporting
address and Markdown text from the same panel (oxygenhub#230), with
sensible text until it is written.

## Guessing is slowed (oxygenhub#221)

Password sign-in, the two-factor code step, device codes and password
reset requests count failures per account and per client address
(`behind_proxy = true` takes the address from the tunnel's
`CF-Connecting-IP`, so set it when the hub is behind one). An account
gets five free failures and an address twenty (people share addresses);
after that each failure doubles the wait before the next try, from two
seconds up to a fifteen-minute lockout, and the page says when to try
again. A reset request counts as a try whatever it matched, so the
answer still gives nothing away. A quiet hour forgets the count and a
successful sign-in clears the account's. Each wait that starts is in the
audit log (`login.throttled`, `device.throttled`,
`password.reset_throttled`) with the address. The counts live in memory,
so a restart clears them.

## Outbound requests (oxygenhub#219)

Webhooks and AI assistants call addresses people give, so the hub only
calls **public** addresses: it resolves the host, refuses if any address
it resolves to is loopback, private, shared (CGNAT), link-local (the
cloud metadata address among them), multicast or an IPv6 local range,
pins the connection to the addresses it checked, and follows no
redirects. A refused webhook delivery records "not sent: …" with the
reason; an assistant says why it cannot be reached. To use a service on
your own network on purpose (a local Ollama), a site administrator lists
it in `hub.toml`:

```toml
[outbound]
allow = ["http://127.0.0.1:11434"]   # scheme://host (any port) or scheme://host:port
```