Runners: see them, their jobs and history, and assign them per site, organisation or repository issue #41 open andymccall opened this 1 month ago · 3 comments

Runners: see them, their jobs and history, and assign them per site, organisation or repository

Today runners are invisible and global:

• Credentials: runner tokens are lines in hub.toml ([flow] runner_tokens). Adding, removing or rotating one means editing the file on the server and restarting the hub.
• Scope: any runner may claim any queued job of any Flow-enabled repository. The only matching is by tier and free memory and CPUs.
• Signing key: a runner's signing key is bound to its token on the first signed receipt (flow_runner_keys, trust on first use), and no page shows that binding.
• Visibility: there is no page that lists runners, shows whether one is online or what it is running, or gives its history. The only trace is the "on earth-1" text in a job's summary.

What we want

1. A Runners page (site administrators: Admin → Runners)

• Each runner:
  • name and signing key (KeyId, identity name);
  • architecture, isolation tiers (microvm, later wasm and syscall) and labels;
  • capacity (max_jobs, memory, vCPUs) and runner version;
  • online, offline or draining, from its last claim poll;
  • last seen, and the jobs it is running now.
• A runner's page: its job history (repository, run, job, result, duration, receipt), pass rate and time busy, paginated.
• Actions: disable or enable, drain (finish current jobs, claim no more), rotate or revoke the credential, and remove.

2. Registration instead of hub.toml tokens

• Registration token: an administrator (or an organisation or repository admin, for their own scope) creates a one-time token in the UI.
• Registering: flow runner register https://hub ... TOKEN exchanges it for a runner credential and registers the runner's signing identity at the same time. The key is then bound at registration, not on first use, and the page shows it.
• Migration: existing runner_tokens keep working until migrated, then are removed.

3. Assignment: site-wide, organisation, repository

• Scope: a runner belongs to the site (any Flow-enabled repository), an organisation or user (its repositories), or one repository.
• Claiming: a claim only returns jobs whose repository is in the runner's scope. Job tokens stay per job, as now. A repository or organisation runner must never receive another scope's jobs.
• Settings → Runners on repositories and organisations lists the runners that may take their jobs: the site's, the organisation's and the repository's own. Admins of that scope can add and remove their own runners.
• Labels in pipelines: a pipeline can ask for labels, e.g. runs-on "riscv64" or runs-on "gpu" on a job or sandbox, and claims match them. This is part of the OxygenFlow pipeline spec, so it also needs an oxygenflow change: parse runs-on, include it in the job hash, and have the runner report its labels and architecture in ClaimRequest.

4. Jobs and queue

• Admin → Runners → Queue: queued jobs in order, why each is waiting (dependencies, no runner with the tier or labels, no capacity), and which runner took each running one.
• Runner name on job rows: the run page names the runner on each job row and links to it.

5. API

• GET /api/v1/runners, /orgs/{org}/runners, /repos/{o}/{n}/runners, GET /api/v1/runners/{id}/jobs.
• POST endpoints to create registration tokens and to drain, disable or remove runners, each for the matching admin scope.

Test plan with real machines

• Plex (x86_64): a second microVM runner beside Earth's earth-1. It tests scheduling across two runners, drain and removal, and a runner scoped to one repository or organisation.
• StarFive VisionFive 2 Lite (riscv64): remote runners on another architecture. They test runs-on "riscv64", per-architecture rootfs images, and a runner that offers no microVM tier.
  • No microVM tier there: the JH7110's U74 cores have no RISC-V hypervisor extension, so there is no KVM, and Firecracker does not support riscv64 anyway.
  • So they need another tier: a runner there needs the planned syscall-isolation or WASM tier (OxygenFlow's tiers, microVM > syscall > WASM). They are a good first target for that tier, and a check that the hub never offers them microVM jobs.

Suggested parts

1. Runners in the database, registration, the Runners page and runner history (hub, plus flow runner register in oxygenflow).
2. Scopes (site, organisation, repository), claim filtering, and Settings → Runners.
3. Labels and architecture: runs-on in the pipeline (oxygenflow) and label matching (hub).
4. Queue view and API.
5. Bring up Plex (x86_64) and a VisionFive 2 Lite (riscv64) as runners. The latter waits for a non-microVM tier.