Repository navigation
Upload testthat results to Codecov Test Analytics from R CMD check #39
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Merged
Merged
Changes from all commits
Commits
Show all changes
3 commits
Select commit
Hold shift + click to select a range
File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
There are no files selected for viewing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,34 @@ | ||
| # 0014. Codecov Test Analytics from the `R CMD check` Run | ||
|
|
||
| - **Status:** Accepted | ||
|
|
||
| - **Date:** 2026-11-05 | ||
| - **Deciders:** Levi Waldron | ||
|
|
||
| ## Context and Problem Statement | ||
|
|
||
| [Codecov Test Analytics](https://docs.codecov.com/docs/test-analytics) reports failing, slow and flaky tests from JUnit XML files uploaded with `codecov/codecov-action` (`report_type: test_results`). `testthat` can write JUnit XML with `JunitReporter`, but `bioccheck.yml` runs the tests inside `R CMD check`, which uses `testthat`'s check reporter. Running the tests a third time only to write JUnit XML (they already run in `R CMD check` and again under `covr`) would add a full test run to every job. | ||
|
|
||
| `test_check()` takes its default reporter from the `testthat.default_check_reporter` option. `R CMD check` runs `tests/testthat.R` with `R --vanilla`, which reads no user or site profile, so that option cannot be set with `.Rprofile` or `Rprofile.site`. The only startup code that subprocess runs is the file `R CMD check` copies from `$(R RHOME)/share/R/tests-startup.R` and passes as `R_TESTS`. | ||
|
|
||
| ## Decision | ||
|
|
||
| When `CODECOV_TOKEN` is set, a step before `R CMD check` appends code to `share/R/tests-startup.R` in the job's container. If `TESTTHAT_JUNIT_FILE` is set and `xml2` (which `JunitReporter` needs) is installed, that code registers an `onLoad` hook for `testthat` which sets `testthat.default_check_reporter` to a `MultiReporter` of the usual `CheckReporter` and a `JunitReporter` writing to that file. The `R CMD check` step sets `TESTTHAT_JUNIT_FILE` with `Sys.setenv()` to an absolute path in the workspace, so the `covr` step, which does not set it, does not overwrite the file. It is set in R because `${{ github.workspace }}` is the runner's host path in container jobs. | ||
|
|
||
| A new step uploads the file with `codecov/codecov-action@v7` and `report_type: test_results`. It runs on every branch, including release branches, and also when `R CMD check` fails, because failing tests are what Test Analytics reports. The `gpg` step now runs when either coverage or test results are to be uploaded. As with coverage (ADR 0011), the setup and upload steps use `continue-on-error`, so test results never fail the check. | ||
|
|
||
| No new input is added: the `CODECOV_TOKEN` secret remains the opt-in for everything sent to Codecov (ADR 0005). | ||
|
|
||
| ## Alternatives Considered | ||
|
|
||
| - **Write JUnit XML from the `covr` run**: needs no change to R's files, but `covr` runs only after `R CMD check` succeeds and only on non-release branches, so failing tests would never be reported. | ||
| - **Run `testthat::test_local()` with `JunitReporter` in its own step**: simple, but runs the whole test suite once more. | ||
| - **Patch the package's `tests/testthat.R` before checking**: changes the package being checked and the tarball `BiocCheck` reads. | ||
| - **A separate `enable_test_analytics` input**: more configuration for callers, when uploading to Codecov is already opted into with the token. | ||
|
|
||
| ## Consequences | ||
|
|
||
| - Packages using `testthat` get Test Analytics with no change to their own files and no extra test run. | ||
| - The normal `testthat` output in `testthat.Rout`, and the failure of `R CMD check` on failing tests, are unchanged. This also holds for packages that run their tests in parallel (`Config/testthat/parallel: true`), since `MultiReporter` supports parallel mode; there `testthat` 3.3.2 writes `classname="character(0)"` in the JUnit file, which is an upstream quirk. | ||
| - Packages using `RUnit` or plain `tests/*.R` scripts, or whose `tests/testthat.R` passes its own `reporter`, upload no test results. | ||
| - The approach depends on `R CMD check` copying `share/R/tests-startup.R` into the tests directory, which is an internal detail of R. If R changes it, `junit.xml` is no longer written and the upload is skipped; the check itself is unaffected. | ||
| - Runs on release branches, and runs where `R CMD check` fails, now spend about 20 seconds installing `gnupg` when there are test results to upload. | ||
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Uh oh!
There was an error while loading. Please reload this page.