# Deploying the developer edition on Earth

Companion to `spec.md` §17.5. Status: **installed and running** since 2026-09-27 as the systemd service `oxygenhub` under the `oxygenhub` system user, data in `/home/oxygenhub/data`, HTTP on 7300 and HTTPS on 7343 with a self-signed certificate. `oxygen`, `oxygen-vscode` and `oxygenhub` are hosted under `andymccall/` and each checkout has an `earth` remote.

## The host

| | |

|---|---|

| Name | `earth` (`earth.andymccall.xyz`, IPv6 ULA; `192.168.1.6` on the LAN) |

| OS | Ubuntu 26.04.1 LTS, x86_64, 4 cores, 31 GiB RAM |

| Disk | `/` 251 GB (95 GB free), `/home` 1 TB (944 GB free) |

| Already running | sshd :22, a Python service :6400, `wren-nest` :7842, retroplay-manager in Docker (:8080, :6670) with Postgres and a tetriblox server, two GitHub Actions runners (`doomsdayonecom`), Docker, containerd |

| Not installed | Rust toolchain, sqlite3 CLI, nginx or caddy, certbot |

| Access | `ssh -i ~/.ssh/earth_claude claude@earth`; `claude` has passwordless sudo |

Ports 7300 (HTTP) and 7343 (HTTPS) are free and are the hub's.

## Steps

1. **Toolchain** (as `claude`): `curl https://sh.rustup.rs -sSf | sh -s -- -y --profile minimal`, then `sudo apt-get install -y build-essential pkg-config git`.

2. **Source**: from the development machine, `rsync -a --delete --exclude target --exclude node_modules ~/development/oxygen ~/development/oxygenhub claude@earth:~/src/`. Oxygen is never pushed to Git; rsync is the bootstrap, and after the hub runs, `oxy push` to the hub itself is the ongoing path.

3. **Build**: `cd ~/src/oxygenhub && cargo build --release`.

4. **System user and data**: `sudo useradd --system --home /home/oxygenhub --create-home oxygenhub`, `sudo install -o oxygenhub -g oxygenhub -m 0755 target/release/oxygenhub /usr/local/bin/`, `sudo -u oxygenhub oxygenhub init --data /home/oxygenhub/data --admin andymccall` (prints the first admin's one-time password).

5. **Service**: `/etc/systemd/system/oxygenhub.service` running `oxygenhub serve --data /home/oxygenhub/data` as `oxygenhub`, `Restart=on-failure`, `ProtectSystem=strict`, `ReadWritePaths=/home/oxygenhub/data`, `NoNewPrivileges=yes`; `sudo systemctl enable --now oxygenhub`.

6. **TLS**: the developer edition generates a self-signed certificate at `init`. Download the certificate from `https://earth.andymccall.xyz:7343/.well-known/oxyhub-ca.pem` (or over plain HTTP on port 7300) and point clients at it with `oxy config remote.<name>.ca <file>` or `OXY_CA`.

7. **First repositories**: on the development machine, `oxy remote add earth https://earth.andymccall.xyz:7343/andymccall/oxygen` in each of `oxygen`, `oxygen-vscode`, `oxygenhub`, then `oxy push earth`.

8. **Updates**: rsync, build, `sudo install`, `sudo systemctl restart oxygenhub`. `make deploy` in the hub wraps steps 2, 3 and 8.

## Restarts without errors (oxygenhub#233)

A deploy restarts the hub. Since #233, on SIGTERM (what `systemctl restart`

sends) the hub stops taking connections, ends its live event streams

(browsers reconnect by themselves) and gives the requests in flight up to

30 seconds to finish. Requests that arrive *between* the old process and the

new one are refused unless systemd holds the listening socket across the

restart: socket activation. The hub serves a socket systemd passes

(`LISTEN_FDS`) and binds `server.bind` itself otherwise.

To turn it on, add `/etc/systemd/system/oxygenhub.socket`:

```ini

[Unit]

Description=Oxygenhub's listening socket, held across restarts

[Socket]

ListenStream=0.0.0.0:7300

# A restart's queue: connections wait here, not refused.

Backlog=512

NoDelay=yes

[Install]

WantedBy=sockets.target

```

`ListenStream` is the address `server.bind` names in `hub.toml` (Earth's is `127.0.0.1:7300`, behind cloudflared). Then give `oxygenhub.service` `Requires=oxygenhub.socket`,

`After=oxygenhub.socket` and `TimeoutStopSec=40`, and run:

```sh

sudo systemctl daemon-reload

sudo systemctl stop oxygenhub

sudo systemctl enable --now oxygenhub.socket

sudo systemctl start oxygenhub

```

On Earth the three service lines are a drop-in, `/etc/systemd/system/oxygenhub.service.d/socket.conf` (installed 2026-10-17; a restart under 10 requests a second then answered all 120 with 200).

The service's log then says "http on the socket systemd holds". From then on

`systemctl restart oxygenhub` keeps the socket, so nobody sees a refused

connection or a Cloudflare 502. HTTPS (7343, the developer edition's

self-signed listener) still binds itself.

## Where things are on Earth

| | |

|---|---|

| Binaries | `/usr/local/bin/oxygenhub`, `/usr/local/bin/oxy` |

| Sources | `/home/claude/src/oxygen`, `/home/claude/src/oxygenhub` (rsync targets; the hub itself is the canonical remote) |

| Data | `/home/oxygenhub/data` (`hub.toml`, `hub.db`, `keys/`, `repos/andymccall/*`) |

| Push token | `/home/oxygenhub/push-token` (also in `hub.toml`); each local checkout has it as `remote.earth.token` |

| Certificate | `/home/oxygenhub/data/keys/tls.crt`, served at `/.well-known/oxyhub-ca.pem`; local copy `~/.config/oxy/earth-ca.pem` set as `remote.earth.ca` |

| Service | `sudo systemctl status oxygenhub`, `sudo journalctl -u oxygenhub` |

## HTTPS only (oxygenhub#218)

When `base_url` is `https://…`, the hub takes nothing over plain http

through a proxy: a request that `X-Forwarded-Proto` (or Cloudflare's

`CF-Visitor`) reports as http is redirected to the same path on

`base_url` (GET and HEAD) or refused with 400 (anything else), and every

cookie it sets is `Secure`. Requests made to the hub directly, such as a

runner on the same host using `http://127.0.0.1:7300`, carry no such

header and are served as before. In front, Cloudflare's Always Use HTTPS

is on (2026-10-15). Strict-Transport-Security is not sent yet: the security headers (#222)

are in, and it waits for the site owner's decision, because browsers

remember it for months.

## Security headers (oxygenhub#222)

Every response carries `X-Content-Type-Options: nosniff`,

`Referrer-Policy: strict-origin-when-cross-origin` and a

`Permissions-Policy` that turns off the camera, microphone, location and

the like; pages also carry `X-Frame-Options: DENY` and a Content Security

Policy (`frame-ancestors 'none'`, scripts and forms from the hub only,

WebAssembly for the backdrop). The policy is **report-only** for now:

browsers report what it would block to `/csp-report`, and the hub logs

each distinct violation once an hour (`csp would block: …`). The pages

still use inline scripts and handlers, which it reports; once they have

moved into `hub.js` and the log is quiet, `ENFORCE` in

`crates/hub/src/headers.rs` switches it on. Responses with their own

policy (report frames, a job's HTML artifact) keep it. HSTS is not sent:

it waits for the site owner's decision.

## security.txt (oxygenhub#224)

Administration › Security contact sets where vulnerabilities are reported

(an email address or an https page) and, optionally, a policy page. The

hub then serves `/.well-known/security.txt` (and `/security.txt`) per

RFC 9116, with an `Expires` date that is always just under a year ahead,

so it never goes stale. With no contact set, both answer 404.

## Guessing is slowed (oxygenhub#221)

Password sign-in, the two-factor code step, device codes and password

reset requests count failures per account and per client address

(`behind_proxy = true` takes the address from the tunnel's

`CF-Connecting-IP`, so set it when the hub is behind one). An account

gets five free failures and an address twenty (people share addresses);

after that each failure doubles the wait before the next try, from two

seconds up to a fifteen-minute lockout, and the page says when to try

again. A reset request counts as a try whatever it matched, so the

answer still gives nothing away. A quiet hour forgets the count and a

successful sign-in clears the account's. Each wait that starts is in the

audit log (`login.throttled`, `device.throttled`,

`password.reset_throttled`) with the address. The counts live in memory,

so a restart clears them.

## Outbound requests (oxygenhub#219)

Webhooks and AI assistants call addresses people give, so the hub only

calls **public** addresses: it resolves the host, refuses if any address

it resolves to is loopback, private, shared (CGNAT), link-local (the

cloud metadata address among them), multicast or an IPv6 local range,

pins the connection to the addresses it checked, and follows no

redirects. A refused webhook delivery records "not sent: …" with the

reason; an assistant says why it cannot be reached. To use a service on

your own network on purpose (a local Ollama), a site administrator lists

it in `hub.toml`:

```toml

[outbound]

allow = ["http://127.0.0.1:11434"]   # scheme://host (any port) or scheme://host:port

```