Test reports from a job's junit artifact (the test half of #113) issue #114 closed: completed andymccall opened this 1 month ago · 0 comments

issue #113 began as test and coverage reporting; coverage is live, and this is the test half. Found wiring up doomsdayone/metalgear#227: its functional suite (about 630 pytest tests in an emulator, three hours serial) passes or fails as one job, and the only way to see which test failed, or which got slow, is to read the log.

The same shape as coverage would do: a job's artifact named junit in JUnit XML (what pytest --junitxml, cargo-nextest, Jest, Go's gotestsum and most runners write) is parsed when it is uploaded, and kept per run.

• Run page: passed, failed, skipped and errored counts beside each job; the failures first, each with its message and output, collapsible.
• A Tests tab: the default bookmark's latest counts and their trend; the slowest tests; tests that failed in recent runs and then passed (flaky); a test's history across runs.
• Proposals: tests that pass on the target and fail on the proposal, called out.

metalgear's functional-tests job keeps junit from issue #227 on (/src/bin/junit.xml, pytest-xdist's merged report), so there is a real report to build against once that lands.