pharmar / thevalidator

A mirror of insightsengineering/thevalidatoR for experimentation in support of the R Validation Hub repositories working group

thevalidatoR

• An R Package Validation Report
  • Description
  • Action Type
  • Author
  • Inputs
  • Outputs
  • How to use
    • Quickstart
    • V1.0 Examples
      • rbmi
      • admiral

An R Package Validation Report

Description

A Github Action that generates a validation report for an R package. The four main steps are:

• Run R CMD check (check installation)
• Run covr::package_coverage() (check unit test coverage)
• Run covtracer (link documentation to unit tests)
• Place results into report
• If valtools present - run valtools and also publish report? (to discuss)
• Attach report as object to release

Action Type

Composite

Author

Roche

Inputs

• report_pkg_dir:

  Description: Path to package's root

  Required: false

  Default: .

• report_template_path:

  Description: File path of the R markdown template to use for the report. The default template is available here.

  Required: false

  Default: template.Rmd

• report_rmarkdown_format:

  Description: The output format to use when rendering the report. Value is used by rmarkdown::render's output_format parameter.

  Required: false

  Default: pdf_document

• report_output_prefix:

  Description: The output filename prefix for the validation report. If left blank, it defaults to the following convention: <package name>-<package version>-validation-report.

  Required: false

  Default: ""

• additional_tlmgr_packages:

  Description: Additional tex packages to install with tlmgr.

  Required: false

  Default: courier ec

• no_cache:

  Description: Disable github action R dependency caching.

  Required: false

  Default: false

• cache_version:

  Description: Version of the cache. To clean cache bump this version.

  Required: false

  Default: v1

• disable_install_dev_deps:

  Description: Disable installation of dev dependencies while building the report.

  Required: false

  Default: false

Outputs

• report_output_filename:

  Description: Filename of the generated report.

How to use

To use this GitHub Action you will need to complete the following:

• Create a new file in your repository called .github/workflows/r-pkg-validation.yml
• Copy the template over (and edit if you wish to modify it)

Quickstart

In your repository you should have a .github/workflows/validatoR.yml file with GitHub Action similar to below:

---
name: R Package Validation report

on: # Run this action when a release is published
  release:
    types: [published]

jobs:
  r-pkg-validation:
    name: Create report 📃
    runs-on: ubuntu-latest
    container:
      image: rocker/verse:4.1.1
    # Set Github token permissions
    env:
      GITHUB_PAT: ${{ secrets.GITHUB_TOKEN }}
    permissions:
      contents: write
      packages: write
      deployments: write
    steps:
      - name: Checkout repo 🛎
        uses: actions/checkout@v3

      - name: Build report 🏗
        id: validation
        uses: insightsengineering/thevalidatoR@main
        # see parameters above for custom templates and other formats

      # Upload the validation report to the release
      - name: Upload report to release 🔼
        if: success()
        uses: svenstaro/upload-release-action@v2
        with:
          file: ${{ steps.validation.outputs.report_output_filename }}
          asset_name: ${{ steps.validation.outputs.report_output_filename }}
          repo_token: ${{ secrets.GITHUB_TOKEN }}
          tag: ${{ github.ref }}
          overwrite: false

V1.0 Examples

rbmi

You can see an example report from rbmi, using the first version of this gh-action, here.

This was built as a test on a fork of the original rbmi package. When we created a release in that fork, this PDF was automatically built and added to the release as can be seen below.

And you can see the gh-action action that was triggered by the release being published. Note that it must install the package, run tests and construct the metrics needed by cov-tracer and covr, which in the case of rbmi a computationally heavy package - took quite a while!

admiral

You can see an example report from admiral, using the first version of this gh-action, here.

thevalidator's Issues

Running action locally with package source

from discussion at R Validation Hub repos wg weekly standup

As we may not use repository-side actions as the execution engine long term, it may be necessary to orchestrate runs somehow. To help ease this transition, it would be nice to know we can adapt the actions for a local run.

Can we test out using actions runner to run actions workflows "locally" in a way that they can be more easily orchestrated.

For now this would just be an informational task to help us decide if/when we should consider other orchestration tools and help us anticipate any growing pains if we have to transition.

Destination repository next steps

from discussion at R Validation Hub repos wg weekly standup

To move forward with additional steps to publish packages to some destination, we felt that the easiest path forward would be to build a monorepo of packages (possibly to be surfaced via R-Universe?).

There may be better solutions in the future. The goal of this initial design is to allow the pipeline to develop and explore lifecycle-management concepts, which will apply regardless of architecture.

Gather feedback on the output JSON files

Just a memo for the next team meeting to gather feedback from the artifacts resulting from #9.

Choose proof-of-concept repository storage

Although we've yet to settle on a technology for the production architecture, we've discussed having a minimalist proof-of-concept architecture that will let us move forward with the other modules.

For our pilot architecture, I think we should prioritize simplicity with the goal of allowing the pipeline to address challenges with managing a persistent repository and all the decisions that go along with that.

Managing dependency lifecycle

from discussion at R Validation Hub repos wg weekly standup

When considering how this will interact with a destination repository, we were considering different ways of handling dependencies.

Assuming there's a repository of validated packages, we might consider a new package request in the context of the packages already available. Dependencies of a new package might have version requirements that could be:

• Already satisfied
• Satisfied, but with updated versions available
• Unsatisfied, requiring a version upgrade
• Unsatisfied, requiring a version downgrade (this is probably very unlikely, and probably not worth extensively planning around this condition)
• Unavailable

The actions that are taken in each condition will inform how we want to handle dependency validation. Some possible paths might include:

• Never handle dependencies, only validate packages if all dependencies are already satisfied by packages already in the repository
• When new dependencies or dependency upgrades are required, also validate those packages and upgrade them in the repository

And finally, there's a decision to be made about how this will affect the documentation associated with each package. Should a package's report include information about the validation of its dependencies, or is it sufficient to have this information in a separate report - one for each package on the repository.

Choosing reference images

Discussed a couple times, but just to get the ideas on paper.

We've referred to this "reference" image for quite a while. Even without all the details of exactly how this should look, we should choose some images that we want to use for the initial pilots.

Just in terms of requirements, I think we should aim to:

• Have a representative "canonical" R image. In the rocker world this seems to be ubuntu-based.
• Have a representative "industry" R image, for which I'd be partial to some windows image since we know that at least the FDA will be using R on windows.
• Develop pipelines so that we may require testing on multiple platforms. At least one of each of the above.

If we want to pull something off-the-shelf, the best options I can think of would be either the rocker images or the r-hub.

I feel like rocker seems to be widely used for CI/CD, but R-hub might be a better drop in replacement for a CRAN-like system which may need to handle a broader set of system library dependencies. The downside is that the R-hub containers are a bit more all-inclusive (larger image size), but this might be a necessary design choice for us to manage arbitrary packages.

The more I think about it, the more I'm leaning toward the R-hub containers, but I want to raise the question to the team before settling on a choice.