| 1 |
# Deploying the developer edition on Earth |
| 2 |
|
| 3 |
Companion to `spec.md` §17.5. Status: **installed and running** since 2026-09-27 as the systemd service `oxygenhub` under the `oxygenhub` system user, data in `/home/oxygenhub/data`, HTTP on 7300 and HTTPS on 7343 with a self-signed certificate. `oxygen`, `oxygen-vscode` and `oxygenhub` are hosted under `andymccall/` and each checkout has an `earth` remote. |
| 4 |
|
| 5 |
## The host |
| 6 |
|
| 7 |
|
| 8 |
|
| 9 |
| Name | `earth` (`earth.andymccall.xyz`, IPv6 ULA; `192.168.1.6` on the LAN) | |
| 10 |
| OS | Ubuntu 26.04.1 LTS, x86_64, 4 cores, 31 GiB RAM | |
| 11 |
| Disk | `/` 251 GB (95 GB free), `/home` 1 TB (944 GB free) | |
| 12 |
| Already running | sshd :22, a Python service :6400, `wren-nest` :7842, retroplay-manager in Docker (:8080, :6670) with Postgres and a tetriblox server, two GitHub Actions runners (`doomsdayonecom`), Docker, containerd | |
| 13 |
| Not installed | Rust toolchain, sqlite3 CLI, nginx or caddy, certbot | |
| 14 |
| Access | `ssh -i ~/.ssh/earth_claude claude@earth`; `claude` has passwordless sudo | |
| 15 |
|
| 16 |
Ports 7300 (HTTP) and 7343 (HTTPS) are free and are the hub's. |
| 17 |
|
| 18 |
## Steps |
| 19 |
|
| 20 |
1. **Toolchain** (as `claude`): `curl https://sh.rustup.rs -sSf | sh -s -- -y --profile minimal`, then `sudo apt-get install -y build-essential pkg-config git`. |
| 21 |
2. **Source**: from the development machine, `rsync -a --delete --exclude target --exclude node_modules ~/development/oxygen ~/development/oxygenhub claude@earth:~/src/`. Oxygen is never pushed to Git; rsync is the bootstrap, and after the hub runs, `oxy push` to the hub itself is the ongoing path. |
| 22 |
3. **Build**: `cd ~/src/oxygenhub && cargo build --release`. |
| 23 |
4. **System user and data**: `sudo useradd --system --home /home/oxygenhub --create-home oxygenhub`, `sudo install -o oxygenhub -g oxygenhub -m 0755 target/release/oxygenhub /usr/local/bin/`, `sudo -u oxygenhub oxygenhub init --data /home/oxygenhub/data --admin andymccall` (prints the first admin's one-time password). |
| 24 |
5. **Service**: `/etc/systemd/system/oxygenhub.service` running `oxygenhub serve --data /home/oxygenhub/data` as `oxygenhub`, `Restart=on-failure`, `ProtectSystem=strict`, `ReadWritePaths=/home/oxygenhub/data`, `NoNewPrivileges=yes`; `sudo systemctl enable --now oxygenhub`. |
| 25 |
6. **TLS**: the developer edition generates a self-signed certificate at `init`. Download the certificate from `https://earth.andymccall.xyz:7343/.well-known/oxyhub-ca.pem` (or over plain HTTP on port 7300) and point clients at it with `oxy config remote.<name>.ca <file>` or `OXY_CA`. |
| 26 |
7. **First repositories**: on the development machine, `oxy remote add earth https://earth.andymccall.xyz:7343/andymccall/oxygen` in each of `oxygen`, `oxygen-vscode`, `oxygenhub`, then `oxy push earth`. |
| 27 |
8. **Updates**: rsync, build, `sudo install`, `sudo systemctl restart oxygenhub`. `make deploy` in the hub wraps steps 2, 3 and 8. |
| 28 |
|
| 29 |
## Restarts without errors (oxygenhub#233) |
| 30 |
|
| 31 |
A deploy restarts the hub. Since #233, on SIGTERM (what `systemctl restart` |
| 32 |
sends) the hub stops taking connections, ends its live event streams |
| 33 |
(browsers reconnect by themselves) and gives the requests in flight up to |
| 34 |
30 seconds to finish. Requests that arrive *between* the old process and the |
| 35 |
new one are refused unless systemd holds the listening socket across the |
| 36 |
restart: socket activation. The hub serves a socket systemd passes |
| 37 |
(`LISTEN_FDS`) and binds `server.bind` itself otherwise. |
| 38 |
|
| 39 |
To turn it on, add `/etc/systemd/system/oxygenhub.socket`: |
| 40 |
|
| 41 |
```ini |
| 42 |
[Unit] |
| 43 |
Description=Oxygenhub's listening socket, held across restarts |
| 44 |
|
| 45 |
[Socket] |
| 46 |
ListenStream=0.0.0.0:7300 |
| 47 |
# A restart's queue: connections wait here, not refused. |
| 48 |
Backlog=512 |
| 49 |
NoDelay=yes |
| 50 |
|
| 51 |
[Install] |
| 52 |
WantedBy=sockets.target |
| 53 |
``` |
| 54 |
|
| 55 |
Then give `oxygenhub.service` `Requires=oxygenhub.socket`, |
| 56 |
`After=oxygenhub.socket` and `TimeoutStopSec=40`, and run: |
| 57 |
|
| 58 |
```sh |
| 59 |
sudo systemctl daemon-reload |
| 60 |
sudo systemctl stop oxygenhub |
| 61 |
sudo systemctl enable --now oxygenhub.socket |
| 62 |
sudo systemctl start oxygenhub |
| 63 |
``` |
| 64 |
|
| 65 |
The service's log then says "http on the socket systemd holds". From then on |
| 66 |
`systemctl restart oxygenhub` keeps the socket, so nobody sees a refused |
| 67 |
connection or a Cloudflare 502. HTTPS (7343, the developer edition's |
| 68 |
self-signed listener) still binds itself. |
| 69 |
|
| 70 |
## Where things are on Earth |
| 71 |
|
| 72 |
|
| 73 |
|
| 74 |
| Binaries | `/usr/local/bin/oxygenhub`, `/usr/local/bin/oxy` | |
| 75 |
| Sources | `/home/claude/src/oxygen`, `/home/claude/src/oxygenhub` (rsync targets; the hub itself is the canonical remote) | |
| 76 |
| Data | `/home/oxygenhub/data` (`hub.toml`, `hub.db`, `keys/`, `repos/andymccall/*`) | |
| 77 |
| Push token | `/home/oxygenhub/push-token` (also in `hub.toml`); each local checkout has it as `remote.earth.token` | |
| 78 |
| Certificate | `/home/oxygenhub/data/keys/tls.crt`, served at `/.well-known/oxyhub-ca.pem`; local copy `~/.config/oxy/earth-ca.pem` set as `remote.earth.ca` | |
| 79 |
| Service | `sudo systemctl status oxygenhub`, `sudo journalctl -u oxygenhub` | |
| 80 |
|
| 81 |
## HTTPS only (oxygenhub#218) |
| 82 |
|
| 83 |
When `base_url` is `https://…`, the hub takes nothing over plain http |
| 84 |
through a proxy: a request that `X-Forwarded-Proto` (or Cloudflare's |
| 85 |
`CF-Visitor`) reports as http is redirected to the same path on |
| 86 |
`base_url` (GET and HEAD) or refused with 400 (anything else), and every |
| 87 |
cookie it sets is `Secure`. Requests made to the hub directly, such as a |
| 88 |
runner on the same host using `http://127.0.0.1:7300`, carry no such |
| 89 |
header and are served as before. In front, Cloudflare's Always Use HTTPS |
| 90 |
is on (2026-10-15). Strict-Transport-Security comes from `[server] hsts_max_age` (seconds; |
| 91 |
0, the default, sends none; never over plain http; oxygenhub#231). |
| 92 |
Browsers remember it, so raise it in stages: 300 first, a week |
| 93 |
(604800) after a few quiet days, then a year (31536000). Add |
| 94 |
`includeSubDomains` only once every subdomain is HTTPS, and preload last. |
| 95 |
oxygenhub.io is at 300 from 2026-10-16. |
| 96 |
|
| 97 |
## Security headers (oxygenhub#222) |
| 98 |
|
| 99 |
Every response carries `X-Content-Type-Options: nosniff`, |
| 100 |
`Referrer-Policy: strict-origin-when-cross-origin` and a |
| 101 |
`Permissions-Policy` that turns off the camera, microphone, location and |
| 102 |
the like; pages also carry `X-Frame-Options: DENY` and a Content Security |
| 103 |
Policy (`frame-ancestors 'none'`, scripts and forms from the hub only, |
| 104 |
WebAssembly for the backdrop). The policy is **report-only** for now: |
| 105 |
browsers report what it would block to `/csp-report`, and the hub logs |
| 106 |
each distinct violation once an hour (`csp would block: …`). The pages |
| 107 |
still use inline scripts and handlers, which it reports; once they have |
| 108 |
moved into `hub.js` and the log is quiet, `ENFORCE` in |
| 109 |
`crates/hub/src/headers.rs` switches it on. Responses with their own |
| 110 |
policy (report frames, a job's HTML artifact) keep it. HSTS is not sent: |
| 111 |
it waits for the site owner's decision. |
| 112 |
|
| 113 |
## security.txt (oxygenhub#224) |
| 114 |
|
| 115 |
Administration › Security contact sets where vulnerabilities are reported |
| 116 |
(an email address or an https page) and, optionally, a policy page. The |
| 117 |
hub then serves `/.well-known/security.txt` (and `/security.txt`) per |
| 118 |
RFC 9116, with an `Expires` date that is always just under a year ahead, |
| 119 |
so it never goes stale. With no contact set, both answer 404. |
| 120 |
`/security-contact` is the page to point the contact at: the reporting |
| 121 |
address and Markdown text from the same panel (oxygenhub#230), with |
| 122 |
sensible text until it is written. |
| 123 |
|
| 124 |
## Guessing is slowed (oxygenhub#221) |
| 125 |
|
| 126 |
Password sign-in, the two-factor code step, device codes and password |
| 127 |
reset requests count failures per account and per client address |
| 128 |
(`behind_proxy = true` takes the address from the tunnel's |
| 129 |
`CF-Connecting-IP`, so set it when the hub is behind one). An account |
| 130 |
gets five free failures and an address twenty (people share addresses); |
| 131 |
after that each failure doubles the wait before the next try, from two |
| 132 |
seconds up to a fifteen-minute lockout, and the page says when to try |
| 133 |
again. A reset request counts as a try whatever it matched, so the |
| 134 |
answer still gives nothing away. A quiet hour forgets the count and a |
| 135 |
successful sign-in clears the account's. Each wait that starts is in the |
| 136 |
audit log (`login.throttled`, `device.throttled`, |
| 137 |
`password.reset_throttled`) with the address. The counts live in memory, |
| 138 |
so a restart clears them. |
| 139 |
|
| 140 |
## Outbound requests (oxygenhub#219) |
| 141 |
|
| 142 |
Webhooks and AI assistants call addresses people give, so the hub only |
| 143 |
calls **public** addresses: it resolves the host, refuses if any address |
| 144 |
it resolves to is loopback, private, shared (CGNAT), link-local (the |
| 145 |
cloud metadata address among them), multicast or an IPv6 local range, |
| 146 |
pins the connection to the addresses it checked, and follows no |
| 147 |
redirects. A refused webhook delivery records "not sent: …" with the |
| 148 |
reason; an assistant says why it cannot be reached. To use a service on |
| 149 |
your own network on purpose (a local Ollama), a site administrator lists |
| 150 |
it in `hub.toml`: |
| 151 |
|
| 152 |
```toml |
| 153 |
[outbound] |
| 154 |
allow = ["http://127.0.0.1:11434"] # scheme://host (any port) or scheme://host:port |
| 155 |
``` |