Flow scheduling: priorities (site, org, team, repo, pipeline), expedited runs, and fair share between repositories issue #244 closed: completed andymccall opened this 19 days ago · 3 comments

Jobs are offered strictly in arrival order. On 2026-10-16 a 20-second OpenTofu plan for oxygenhub-gcp waited over an hour behind metalgear2 and bomberman94 builds of 13 minutes each.

Andy's cases (2026-10-16):

• Break-fix: during an incident, the fix's pipeline (e.g. an oxygenhub-gcp apply) must go to the front, not wait behind a game's test suite.
• Cost of waiting: an organisation gives a team of contractors higher priority than its support team, because idle contractors cost more than the jobs.

Proposal:

• Priority levels, set at site, organisation, team, repository and pipeline level, the most specific winning: e.g. urgent, high, normal (default), low. Who may set each level follows the existing roles: site admins for the site, org owners for their org and teams, repo maintainers within what their org allows.
• Expedite a run: a maintainer can raise one run to urgent with a reason, for an incident. It is recorded in the audit log and on the run page, and it lapses when the run ends.
• Fair share within a level: among jobs of the same priority, take turns between repositories (round-robin, oldest first within each), so one repository's burst can't hold up the rest.
• No pre-emption at first: a running job is never stopped for a more urgent one; the urgent one takes the next free slot that fits. Pre-emption can come later for runners that opt in.
• Shown on the queue: each queued job shows its priority and why ("urgent: expedited by andy: GCP incident"), with the filters of issue #243.
• Limits: an org's urgent share can be capped, so priority can't starve everyone else.