Deploying the developer edition on Earth

Companion to spec.md §17.5. Status: installed and running since 2026-09-27 as the systemd service oxygenhub under the oxygenhub system user, data in /home/oxygenhub/data, HTTP on 7300 and HTTPS on 7343 with a self-signed certificate. oxygen, oxygen-vscode and oxygenhub are hosted under andymccall/ and each checkout has an earth remote.

The host

Ports 7300 (HTTP) and 7343 (HTTPS) are free and are the hub's.

Steps

1. Toolchain (as claude): curl https://sh.rustup.rs -sSf | sh -s -- -y --profile minimal, then sudo apt-get install -y build-essential pkg-config git.
2. Source: from the development machine, rsync -a --delete --exclude target --exclude node_modules ~/development/oxygen ~/development/oxygenhub claude@earth:~/src/. Oxygen is never pushed to Git; rsync is the bootstrap, and after the hub runs, oxy push to the hub itself is the ongoing path.
3. Build: cd ~/src/oxygenhub && cargo build --release.
4. System user and data: sudo useradd --system --home /home/oxygenhub --create-home oxygenhub, sudo install -o oxygenhub -g oxygenhub -m 0755 target/release/oxygenhub /usr/local/bin/, sudo -u oxygenhub oxygenhub init --data /home/oxygenhub/data --admin andymccall (prints the first admin's one-time password).
5. Service: /etc/systemd/system/oxygenhub.service running oxygenhub serve --data /home/oxygenhub/data as oxygenhub, Restart=on-failure, ProtectSystem=strict, ReadWritePaths=/home/oxygenhub/data, NoNewPrivileges=yes; sudo systemctl enable --now oxygenhub.
6. TLS: the developer edition generates a self-signed certificate at init. Download the certificate from https://earth.andymccall.xyz:7343/.well-known/oxyhub-ca.pem (or over plain HTTP on port 7300) and point clients at it with oxy config remote.<name>.ca <file> or OXY_CA.
7. First repositories: on the development machine, oxy remote add earth https://oxygenhub.io/andymccall/oxygen in each of oxygen, oxygen-vscode, oxygenhub, then oxy push earth.
8. Updates: rsync, build, sudo install, sudo systemctl restart oxygenhub. make deploy in the hub wraps steps 2, 3 and 8.
9. Public address (2026-09-30): a Cloudflare tunnel on Earth serves the hub as https://oxygenhub.io, with a publicly trusted certificate, and base_url in hub.toml is that address, so every link the hub writes uses it. Checkouts push there with no CA set, and subrepo URLs name it too.
10. Localhost only (2026-09-30): the tunnel connects to 127.0.0.1:7300, so the hub binds 127.0.0.1:7300 and 127.0.0.1:7343 and nothing on the LAN reaches it directly; behind_proxy = true (the hub does not yet use client addresses). The Flow runner on Earth polls http://127.0.0.1:7300. Backups: hub.toml.before-localhost, ~/flow/flow.toml.before-localhost.

Restarts without errors (oxygenhub#233)

A deploy restarts the hub. Since #233, on SIGTERM (what systemctl restart sends) the hub stops taking connections, ends its live event streams (browsers reconnect by themselves) and gives the requests in flight up to 30 seconds to finish. Requests that arrive between the old process and the new one are refused unless systemd holds the listening socket across the restart: socket activation. The hub serves a socket systemd passes (LISTEN_FDS) and binds server.bind itself otherwise.

To turn it on, add /etc/systemd/system/oxygenhub.socket:

[Unit]
Description=Oxygenhub's listening socket, held across restarts

[Socket]
ListenStream=0.0.0.0:7300
# A restart's queue: connections wait here, not refused.
Backlog=512
NoDelay=yes

[Install]
WantedBy=sockets.target

ListenStream is the address server.bind names in hub.toml (Earth's is 127.0.0.1:7300, behind cloudflared). Then give oxygenhub.service Requires=oxygenhub.socket, After=oxygenhub.socket and TimeoutStopSec=40, and run:

sudo systemctl daemon-reload
sudo systemctl stop oxygenhub
sudo systemctl enable --now oxygenhub.socket
sudo systemctl start oxygenhub

On Earth the three service lines are a drop-in, /etc/systemd/system/oxygenhub.service.d/socket.conf (installed 2026-10-17; a restart under 10 requests a second then answered all 120 with 200).

The service's log then says "http on the socket systemd holds". From then on systemctl restart oxygenhub keeps the socket, so nobody sees a refused connection or a Cloudflare 502. HTTPS (7343, the developer edition's self-signed listener) still binds itself.

Where things are on Earth

HTTPS only (oxygenhub#218)

When base_url is https://…, the hub takes nothing over plain http through a proxy: a request that X-Forwarded-Proto (or Cloudflare's CF-Visitor) reports as http is redirected to the same path on base_url (GET and HEAD) or refused with 400 (anything else), and every cookie it sets is Secure. Requests made to the hub directly, such as a runner on the same host using http://127.0.0.1:7300, carry no such header and are served as before. In front, Cloudflare's Always Use HTTPS is on (2026-10-15). Strict-Transport-Security comes from [server] hsts_max_age (seconds; 0, the default, sends none; never over plain http; oxygenhub#231). Browsers remember it, so raise it in stages: 300 first, a week (604800) after a few quiet days, then a year (31536000). Add includeSubDomains only once every subdomain is HTTPS, and preload last. oxygenhub.io is at 300 from 2026-10-16.

Security headers (oxygenhub#222)

Every response carries X-Content-Type-Options: nosniff, Referrer-Policy: strict-origin-when-cross-origin and a Permissions-Policy that turns off the camera, microphone, location and the like; pages also carry X-Frame-Options: DENY and a Content Security Policy (frame-ancestors 'none', scripts and forms from the hub only, WebAssembly for the backdrop). The policy is report-only for now: browsers report what it would block to /csp-report, and the hub logs each distinct violation once an hour (csp would block: …). The pages still use inline scripts and handlers, which it reports; once they have moved into hub.js and the log is quiet, ENFORCE in crates/hub/src/headers.rs switches it on. Responses with their own policy (report frames, a job's HTML artifact) keep it. HSTS is not sent: it waits for the site owner's decision.

security.txt (oxygenhub#224)

Administration › Security contact sets where vulnerabilities are reported (an email address or an https page) and, optionally, a policy page. The hub then serves /.well-known/security.txt (and /security.txt) per RFC 9116, with an Expires date that is always just under a year ahead, so it never goes stale. With no contact set, both answer 404. /security-contact is the page to point the contact at: the reporting address and Markdown text from the same panel (oxygenhub#230), with sensible text until it is written.

Guessing is slowed (oxygenhub#221)

Password sign-in, the two-factor code step, device codes and password reset requests count failures per account and per client address (behind_proxy = true takes the address from the tunnel's CF-Connecting-IP, so set it when the hub is behind one). An account gets five free failures and an address twenty (people share addresses); after that each failure doubles the wait before the next try, from two seconds up to a fifteen-minute lockout, and the page says when to try again. A reset request counts as a try whatever it matched, so the answer still gives nothing away. A quiet hour forgets the count and a successful sign-in clears the account's. Each wait that starts is in the audit log (login.throttled, device.throttled, password.reset_throttled) with the address. The counts live in memory, so a restart clears them.

Outbound requests (oxygenhub#219)

Webhooks and AI assistants call addresses people give, so the hub only calls public addresses: it resolves the host, refuses if any address it resolves to is loopback, private, shared (CGNAT), link-local (the cloud metadata address among them), multicast or an IPv6 local range, pins the connection to the addresses it checked, and follows no redirects. A refused webhook delivery records "not sent: …" with the reason; an assistant says why it cannot be reached. To use a service on your own network on purpose (a local Ollama), a site administrator lists it in hub.toml:

[outbound]
allow = ["http://127.0.0.1:11434"]   # scheme://host (any port) or scheme://host:port