A coverage check, and rules for it 7edaff01d506

-- A repository's coverage rules (oxygenhub#113): the `coverage` check

-- fails below `min` percent, or when coverage falls more than `max_drop`

-- points against the default bookmark. NULL: no rule.

CREATE TABLE coverage_settings (

    repo_id   INTEGER PRIMARY KEY REFERENCES repositories(id) ON DELETE CASCADE,

    min       REAL,

    max_drop  REAL

);

- **A coverage check (#113).** Each coverage report sets a `coverage` check, such as `81.9% (-0.3 against main)`, measured against the default bookmark. Settings › Coverage can set a minimum and a largest drop. The check then fails and says why, and as a required check it holds a proposal back. A proposal's report shows the files whose coverage moved against the default bookmark.

/// A repository's coverage rules.

#[derive(Clone, Copy, Debug, Default, PartialEq)]

pub struct Rules {

    /// The check fails below this percentage.

    pub min: Option<f64>,

    /// The check fails when coverage falls more than this many points.

    pub max_drop: Option<f64>,

}

pub async fn rules(pool: &Pool, repo_id: i64) -> sqlx::Result<Rules> {

    let row: Option<(Option<f64>, Option<f64>)> =

        sqlx::query_as("SELECT min, max_drop FROM coverage_settings WHERE repo_id = ?").bind(repo_id).fetch_optional(pool).await?;

    Ok(row.map(|(min, max_drop)| Rules { min, max_drop }).unwrap_or_default())

}

pub async fn set_rules(pool: &Pool, repo_id: i64, r: Rules) -> sqlx::Result<()> {

    sqlx::query(

        "INSERT INTO coverage_settings (repo_id, min, max_drop) VALUES (?, ?, ?) \

         ON CONFLICT (repo_id) DO UPDATE SET min = excluded.min, max_drop = excluded.max_drop",

    )

    .bind(repo_id)

    .bind(r.min)

    .bind(r.max_drop)

    .execute(pool)

    .await?;

    Ok(())

}

/// What a report is measured against: on the default bookmark, the report

/// before it there; on any other, the default bookmark's latest before it.

pub async fn baseline(pool: &Pool, r: &Report, default_bookmark: &str) -> sqlx::Result<Option<Report>> {

    sqlx::query_as(

        "SELECT * FROM coverage_reports WHERE repo_id = ? AND bookmark = ? AND id <> ? AND (created_ms < ? OR (created_ms = ? AND id < ?)) \

         ORDER BY created_ms DESC, id DESC LIMIT 1",

    )

    .bind(r.repo_id)

    .bind(default_bookmark)

    .bind(r.id)

    .bind(r.created_ms)

    .bind(r.created_ms)

    .bind(r.id)

    .fetch_optional(pool)

    .await

}

/// The `coverage` check for a report: passed or failed, and what to say.

pub fn verdict(r: &Report, base: Option<&Report>, base_name: &str, rules: Rules) -> (&'static str, String) {

    let Some(p) = r.percent() else { return ("passed", "no instrumented lines".into()) };

    let delta = base.and_then(|b| Some(p - b.percent()?));

    let against = match delta {

        Some(d) if d.abs() < 0.05 => format!(", unchanged against {base_name}"),

        Some(d) => format!(" ({d:+.1} against {base_name})"),

        None => String::new(),

    };

    if let Some(min) = rules.min.filter(|m| p < *m) {

        return ("failed", format!("{p:.1}%{against}: below the minimum of {min}%"));

    }

    if let (Some(max), Some(d)) = (rules.max_drop, delta) {

        if d < -max {

            return ("failed", format!("{p:.1}%{against}: falls more than the {max} points allowed"));

        }

    }

    ("passed", format!("{p:.1}%{against}"))

}

    }

    #[test]

    fn the_check_follows_the_rules() {

        let report = |pct_hit: i64| Report {

            id: 2,

            repo_id: 1,

            run_id: 1,

            job_id: 1,

            job: "test".into(),

            run_number: 1,

            bookmark: "feature".into(),

            state: String::new(),

            heads: String::new(),

            blob: String::new(),

            format: "lcov".into(),

            lines_found: 1000,

            lines_hit: pct_hit,

            functions_found: 0,

            functions_hit: 0,

            branches_found: 0,

            branches_hit: 0,

            hits: 0,

            files: 1,

            dropped: 0,

            created_ms: 0,

        };

        let (now, before) = (report(819), report(822));

        let none = Rules::default();

        assert_eq!(verdict(&now, Some(&before), "main", none), ("passed", "81.9% (-0.3 against main)".to_string()));

        assert_eq!(verdict(&now, None, "main", none), ("passed", "81.9%".to_string()));

        assert_eq!(verdict(&now, Some(&now), "main", none).1, "81.9%, unchanged against main");

        let strict = Rules { min: Some(85.0), max_drop: None };

        assert_eq!(

            verdict(&now, Some(&before), "main", strict),

            ("failed", "81.9% (-0.3 against main): below the minimum of 85%".to_string())

        );

        let drop = Rules { min: None, max_drop: Some(0.2) };

        assert_eq!(verdict(&now, Some(&before), "main", drop).0, "failed");

        assert_eq!(verdict(&now, Some(&before), "main", Rules { min: None, max_drop: Some(0.5) }).0, "passed");

    // The `coverage` check on the run's state (#113): against the default

    // bookmark, under the repository's rules.

    let base = crate::coverage::baseline(&app.db, &r, &row.default_bookmark).await.map_err(|e| format!("coverage: {e}"))?;

    let rules = crate::coverage::rules(&app.db, row.id).await.map_err(|e| format!("coverage: {e}"))?;

    let (status, summary) = crate::coverage::verdict(&r, base.as_ref(), &row.default_bookmark, rules);

    let url = format!("{}/{}/{}/coverage?report={}", app.config.base_url(), row.owner, row.name, r.id);

    if let Ok(c) = db::hooks::upsert_check(

        &app.db,

        row.id,

        db::hooks::CheckReport { state: &run.state, name: "coverage", status, summary: &summary, url: &url, reporter: None, seal: "" },

    )

    .await

    {

        let api = crate::api::check_of(&c);

        app.events.proposal(row.id, hub_api::ProposalEvent { number: 0, kind: "check".into(), check: Some(api.clone()) });

        crate::hooks::emit(app, row, "check", "flow", serde_json::json!({ "check": api })).await;

    }

    // The report before the one shown, on the same bookmark; for a

    // bookmark's first report, the default bookmark's latest before it.

    let mut previous = report

    if let (None, Some(r)) = (&previous, &report) {

        if r.bookmark != row.default_bookmark {

            previous = coverage::baseline(&app.db, r, &row.default_bookmark).await.map_err(e)?;

        }

    }

        .route("/{owner}/{name}/settings/coverage", post(repo_settings::coverage))

    /// The coverage check's rules (#113).

    coverage: crate::coverage::Rules,

    let coverage = crate::coverage::rules(&app.db, row.id).await.map_err(|e| Error::from_db(shell.clone(), e))?;

        coverage,

pub struct CoverageForm {

    #[serde(rename = "_csrf")]

    csrf_token: String,

    #[serde(default)]

    min: String,

    #[serde(default)]

    max_drop: String,

}

/// Settings → Coverage (#113): the rules the `coverage` check applies.

pub async fn coverage(

    State(app): State<Shared>,

    viewer: Viewer,

    Path((owner, name)): Path<(String, String)>,

    Form(f): Form<CoverageForm>,

) -> Result<Page<RepoSettingsPage>, Error> {

    let shell = Shell::new(&app, &format!("/{owner}/{name}/settings"), &viewer);

    let (_, row, role) = lookup_role(&app, &shell, &viewer, &owner, &name, Access::Maintain).await?;

    csrf(&viewer, &f.csrf_token, &shell)?;

    let number = |v: &str, what: &str, hi: f64| -> Result<Option<f64>, String> {

        let v = v.trim().trim_end_matches('%').trim();

        if v.is_empty() {

            return Ok(None);

        }

        match v.parse::<f64>() {

            Ok(n) if (0.0..=hi).contains(&n) => Ok(Some(n)),

            _ => Err(format!("{what} is a number from 0 to {hi}, or empty for no rule.")),

        }

    };

    let rules = match (number(&f.min, "The minimum", 100.0), number(&f.max_drop, "The largest drop", 100.0)) {

        (Ok(min), Ok(max_drop)) => crate::coverage::Rules { min, max_drop },

        (Err(e), _) | (_, Err(e)) => return settings_page(&app, shell, row, role, None, Some(e)).await,

    };

    crate::coverage::set_rules(&app.db, row.id, rules).await.map_err(|e| Error::from_db(shell.clone(), e))?;

    accounts::audit(

        &app.db,

        viewer.account(),

        "repo.coverage_rules",

        &format!("{owner}/{name}"),

        &format!("min {:?}, max drop {:?}", rules.min, rules.max_drop),

    )

    .await

    .ok();

    settings_page(&app, shell, row, role, Some("Coverage rules saved; they apply from the next report.".into()), None).await

}

#[derive(Deserialize)]

    // The `coverage` check on the run's state: it reports, and with a

    // minimum the repository sets, it fails below it.

    let row = db::repos::get(&h.app.db, "andy", "cov").await.unwrap().unwrap();

    let check = |checks: Vec<db::hooks::Check>| checks.into_iter().find(|c| c.name == "coverage").expect("a coverage check");

    let c = check(db::hooks::checks_for_state(&h.app.db, row.id, &run.state).await.unwrap());

    assert_eq!((c.status.as_str(), c.summary.as_str()), ("passed", "50.0%"));

    assert!(c.url.ends_with(&format!("/andy/cov/coverage?report={report}")), "{}", c.url);

    oxygenhub::coverage::set_rules(&h.app.db, row.id, oxygenhub::coverage::Rules { min: Some(60.0), max_drop: None }).await.unwrap();

    assert_eq!(upload(&h.app, id, "coverage", lcov.as_bytes()).await.0, StatusCode::OK);

    let c = check(db::hooks::checks_for_state(&h.app.db, row.id, &run.state).await.unwrap());

    assert_eq!((c.status.as_str(), c.summary.as_str()), ("failed", "50.0%: below the minimum of 60%"));

- **The check.** Each report sets a `coverage` check on its run's state, measured against the default bookmark: its previous report there for a run on it, its latest for a run on any other bookmark (a proposal's). Settings › Coverage can set a minimum percentage and a largest drop in points; broken, the check fails and says why (`50.0%: below the minimum of 60%`), and naming `coverage` among the required checks holds a proposal back. Without rules it only reports. A bookmark's first report shows the files that moved against the default bookmark.

- **Oxygen itself:** its `test` job runs under cargo-llvm-cov in the build image (oxygenflow 7705d699) and keeps its report as the `coverage` artifact; the README carries the badge.

- **Next:** Cobertura XML, and the API.

  <a href="#coverage">Coverage</a>

<form class="card wide" id="coverage" method="post" action="{{ repo.url }}/settings/coverage">

  <input type="hidden" name="_csrf" value="{{ shell.csrf }}">

  <h2>Coverage</h2>

  <p class="small mute">A job's <code>coverage</code> artifact (LCOV) gives each run a <code>coverage</code> check, measured against <strong>{{ row.default_bookmark }}</strong>. Without rules it only reports; with them it fails when they are broken, and a required check named <code>coverage</code> then holds a proposal back.</p>

  <div class="row">

    <div><label for="cov_min">Fail below this percentage</label><input type="number" id="cov_min" name="min" min="0" max="100" step="0.1" value="{% if let Some(m) = coverage.min %}{{ m }}{% endif %}" placeholder="e.g. 80"></div>

    <div><label for="cov_drop">Fail when it falls by more than (points)</label><input type="number" id="cov_drop" name="max_drop" min="0" max="100" step="0.1" value="{% if let Some(m) = coverage.max_drop %}{{ m }}{% endif %}" placeholder="e.g. 0.5"></div>

  </div>

  <button type="submit">SAVE</button>

</form>