Monitoring: Prometheus and Grafana on Earth, with dashboards, alerts and reports issue #300 open andymccall opened this · 1 comment
andymccall set labels feature ·
andymccall
G4 done (2026-10-25 02:05 GMT): all four runners (earth-1, mars-1, plex-1, narvi-1) on flow 0.3.0 (33de4ee0) serving their own /metrics on the LAN (port 9464) behind one bearer token (Earth: /etc/prometheus/flow_metrics_token, root:prometheus 0440); node_exporter on mars and plex (Ubuntu package, LAN address only) and on narvi (user unit, the package's binary unpacked, no sudo). Prometheus scrapes node x4 and flow-runner x4, all up; the Flow dashboard's last row shows the runners' own numbers with versions and uptime (oxygen-monitoring 849836e1). Still open from G3: email alerts wait on an SMTP contact point (Andy), and the weekly report.
Andy (2026-10-24): Prometheus on Earth scraping the hub's /metrics (token), node_exporter on Earth, mars, plex and narvi, and the runners' own /metrics (Flow 0.3.0, runners rebuilt and restarted). Grafana OSS (Grafana's signed apt repository) on Earth's local network only, its own admin login; dashboards: overview, hub health, Flow (queue wait and run time by runner, results, reuse), storage and backups, activity, hosts; alerts (hub down, backup older than a day, disk, runner missing, long queue waits, dead work items) and a periodic summary report. Ubuntu packages, systemd, 30 days' retention.