Moving away from proprietary languages, Roche has made a notable decision to freeze their legacy macros library. With great enthusiasm, they now <strong>embrace R as the primary framework for evidence generation in late-stage clinical trials</strong>, and they remain <strong>open to exploring additional open-source languages</strong> in this evolving landscape. James Black, a senior member of Roche's medical affairs department, highlights this pivotal moment and stresses its importance in assisting with New Drug Application (NDA) submissions. Some open source NDA’s that use R for submissions have been disclosed.

This is just one indicator from the pharmaceutical industry, openly embracing open source software in lieu of traditional, harder-to-access proprietary technology. In the post below, we'll discuss extensively the types of open-source R tools and packages that are currently available for quality assurance and validation.

Table of Contents:
<ul><li><a href="#question">Do We Have to Validate R Packages for Regulatory Submissions?</a></li><li><a href="#packages">R Packages in Clinical Development</a><ul><li><a href="#repository">Building a Secure and Controlled Internal Package Repository for Clinical Trials</a></li><li><a href="#tools">Testing R Functions: Essential Tools for Internal Package Validation</a></li><li><a href="#reproducibility">Ensuring Reproducibility in Shiny Applications</a></li></ul>
</li>
</ul>

<hr />

<h2 id="question">Do We Have to Validate R Packages for Regulatory Submissions?</h2>
<h3>FDA Statistical Software <a href="https://www.fda.gov/media/161196/download" target="_blank" rel="noopener">Clarifying Statement</a></h3>
Regarding the use of software for statistical analysis of clinical data, the FDA has clarified that they do not require use of any specific software and statistical software is not explicitly discussed in Title 21 of the Code of Federal Regulations [e.g., in <a href="https://www.accessdata.fda.gov/scripts/cdrh/cfdocs/cfcfr/CFRSearch.cfm?CFRPart=11" target="_blank" rel="noopener">21CFR part 11</a>]. <b>However, the software package(s) used for statistical analyses should be fully documented in the submission, including version and build identification.</b>

Besides, as noted in the FDA guidance, <a href="http://www.fda.gov/Drugs/GuidanceComplianceRegulatoryInformation/Guidances/default.htm" target="_blank" rel="noopener">E9 Statistical Principles for Clinical Trials</a> “The computer software used for data management and statistical analysis should be reliable, and documentation of appropriate software testing procedures should be available.” Sponsors are encouraged to consult with FDA review teams and especially with FDA statisticians regarding the choice and suitability of statistical software packages at an early stage in the product development process.
<h3>A Guidance Document for the Use of R in Regulated Clinical Trial Environments from the R Foundation for Statistical Computing</h3>
The use of statistical software for the analysis and presentation of data collected in the course of these regulated activities is itself regulated, to varying levels. There are several documents that are relevant to this particular domain, which we present below.
<h4>Documents Collectively Referred to as GxP:</h4><ul><li style="font-weight: 400;" aria-level="1"><a href="https://www.accessdata.fda.gov/scripts/cdrh/cfdocs/cfcfr/CFRSearch.cfm?CFRPart=11" target="_blank" rel="noopener">21 CFR Part 11 - Electronic Records; Electronic Signatures</a></li><li style="font-weight: 400;" aria-level="1"><a href="https://www.fda.gov/media/75414/download" target="_blank" rel="noopener">Guidance for Industry: Part 11, Electronic Records; Electronic Signatures - Scope and Application</a></li><li style="font-weight: 400;" aria-level="1"><a href="https://www.accessdata.fda.gov/scripts/cdrh/cfdocs/cfcfr/CFRSearch.cfm?CFRPart=58" target="_blank" rel="noopener">21 CFR Part 58 - Good Laboratory Practice (GLP)</a></li><li style="font-weight: 400;" aria-level="1"><a href="https://www.accessdata.fda.gov/scripts/cdrh/cfdocs/cfcfr/CFRSearch.cfm?CFRPart=312" target="_blank" rel="noopener">21 CFR Part 312 - Good Clinical Practice (GCP)</a></li><li style="font-weight: 400;" aria-level="1"><a href="https://www.accessdata.fda.gov/scripts/cdrh/cfdocs/cfcfr/CFRSearch.cfm?CFRPart=210" target="_blank" rel="noopener">21 CFR Part 210 - Current Good Manufacturing Practice (cGMP)</a></li><li style="font-weight: 400;" aria-level="1"><a href="https://www.fda.gov/regulatory-information/search-fda-guidance-documents/e6r2-good-clinical-practice-integrated-addendum-ich-e6r1" target="_blank" rel="noopener">ICH E6 - Good Clinical Practice Consolidated Guideline</a></li></ul>
<h4>Principal Software Guidance Documents:</h4><ul><li style="font-weight: 400;" aria-level="1"><a href="https://www.fda.gov/media/70970/download" target="_blank" rel="noopener">Guidance for Industry - Computerized Systems Used in Clinical Investigations (2007)</a></li><li style="font-weight: 400;" aria-level="1"><a href="https://www.fda.gov/media/73141/download" target="_blank" rel="noopener">General Principles of Software Validation; Final Guidance for Industry and FDA Staff (2002)</a></li></ul>
<h4>Principal Statistical Guideline Documents:</h4><ul><li style="font-weight: 400;" aria-level="1"><a href="https://www.fda.gov/regulatory-information/search-fda-guidance-documents/e9-statistical-principles-clinical-trials" target="_blank" rel="noopener">ICH E9 - Statistical Principles for Clinical Trials</a></li><li style="font-weight: 400;" aria-level="1"><a href="https://www.fda.gov/regulatory-information/search-fda-guidance-documents/guidance-use-bayesian-statistics-medical-device-clinical-trials-pdf-version" target="_blank" rel="noopener">Guidance for Industry and FDA Staff - Guidance for the Use of Bayesian Statistics in Medical Device Clinical Trials (2010)</a></li></ul>
The overall purpose of the <a href="https://www.r-project.org/doc/R-FDA.pdf" target="_blank" rel="noopener">guidance document for the Use of R in Regulated Clinical Trial Environments</a> is to demonstrate that R, when used in a qualified fashion, can support the appropriate regulatory requirements for validated systems, thus ensuring that resulting electronic records are trustworthy, reliable and generally equivalent to paper records.

The FDA recognizes that software used in regulated environments, such as in the medical device or pharmaceutical industry, must be developed, tested, and maintained in a controlled manner to ensure its reliability, safety, and effectiveness. Software validation is accomplished through a series of activities and tasks that are planned and executed at various stages of the Software Development Life Cycle (SDLC). The SDLC provides a systematic and structured approach to developing and maintaining software systems. 

The SDLC encompasses various stages, which are:
<ol><li style="font-weight: 400;" aria-level="1">Operational overview</li><li style="font-weight: 400;" aria-level="1">Source code management</li><li style="font-weight: 400;" aria-level="1">Testing and validation</li><li style="font-weight: 400;" aria-level="1">Release cycles</li><li style="font-weight: 400;" aria-level="1">Availability of Current and Historical Archive Versions</li><li style="font-weight: 400;" aria-level="1">Maintenance, Support and Retirement</li><li style="font-weight: 400;" aria-level="1">Qualified Personnel</li><li style="font-weight: 400;" aria-level="1">Physical and Logical Security</li><li style="font-weight: 400;" aria-level="1">Disaster recovery</li></ol>
&nbsp;

By following the SDLC, organizations can implement proper controls, documentation, and testing procedures throughout the software development process. This helps to identify and mitigate risks, ensure traceability, and facilitate the validation of software systems to meet regulatory requirements.
<h2 id="packages">R Packages in Clinical Development</h2>
<h3>Available Public Packages</h3>
When it comes to public packages in a clinical setting, two key factors require attention. 
<ol><li style="font-weight: 400;" aria-level="1"><b>Establishing stable repositories within the organization is crucial</b>. By maintaining reliable and controlled repositories, healthcare institutions can ensure the availability and accessibility of approved packages for clinical operations. This helps in maintaining consistency and reliability in data analysis and reporting. </li><li style="font-weight: 400;" aria-level="1"><b>Users must diligently document the packages used for clinical reporting</b>. Accurate and comprehensive documentation enables transparency, traceability, and reproducibility in clinical research, facilitating regulatory compliance.</li></ol>
&nbsp;
<h4>Posit Packages</h4>
In 2020, Posit, in collaboration with the R community and pharmaceutical organizations, took a significant step towards enhancing validation practices. They released comprehensive validation guidance documents specifically targeting popular packages like <a href="https://www.rstudio.com/assets/img/validation-tidy.pdf" target="_blank" rel="noopener">tidyverse, tidymodels, r-lib, gt</a>, <a href="https://www.rstudio.com/assets/img/validation-shiny-rmd.pdf">shiny, and rmarkdown</a>.
<h4>Open Source Package Development &amp;amp; Maintenance</h4>
The maintenance of open-source brings a number of advantages. One significant benefit is the transparency and accessibility of information, which is often concealed when dealing with proprietary software. 

This openness proves to be particularly valuable for Quality Teams as they can delve into the inner workings of packages, gaining insights that aid in the assessment of quality and regulatory compliance. 

For quality professionals, tools such as GitHub are incredibly useful to verify R package version control and the maintenance done for those packages.
<h4>CRAN</h4>
Getting a package approved and added to the <a href="https://cran.r-project.org/" target="_blank" rel="noopener">Comprehensive R Archive Network</a> (CRAN) is an accomplishment that signifies its quality and adherence to stringent standards. CRAN serves as a central repository for R packages, ensuring their availability and reliability to users worldwide.
<h4>Automatic Tests</h4>
The process of getting a package added to CRAN involves passing various evaluations and tests. R packages need to undergo the R CMD CHECK process, which involves running a <a href="https://cran.r-project.org/doc/manuals/r-release/R-exts.html#Checking-packages" target="_blank" rel="noopener">series of tests</a> to ensure their functionality and compatibility. This examination verifies that the package meets CRAN's requirements and guidelines, covering aspects such as code quality, documentation, and adherence to best practices.

We've placed a sample chart for the <a href="https://cran.r-project.org/web/checks/check_results_ggplot2.html" target="_blank" rel="noopener">ggplot2</a> package below; you can find the CRAN package check results for ggplot2 here. The ggplot2 package has successfully passed all the necessary evaluations and tests, earning its place on CRAN.

<img class="wp-image-20750 size-full" src="https://webflow-prod-assets.s3.amazonaws.com/6525256482c9e9a06c7a9d3c%2F65b01abd5591d718748d80a7_automated-ggplot2-check-for-clinical-trial-project.webp" alt="automated ggplot2 check for clinical trial project" width="741" height="500" /> Image <a href="https://colorado.posit.co/rsc/R_Posit_Package_Validation/Validation_slides.html#/ggplot2" target="_blank" rel="noopener">source</a>
<h4>Author Tests</h4>
Package authors themselves often include their own tests to ensure the proper functioning of their packages. These custom tests serve as an extra layer of validation, ensuring that the package's functions work as intended in various scenarios. These additional tests are part of the package source. More information can be found in <a href="https://r-pkgs.org/testing-basics.html#test-cran" target="_blank" rel="noopener">Hadley Wickham and Jennifer Bryan's R Packages (2e)</a>.
<h4>Guiding Validation Efforts: The Validation Hub's Risk-Based Approach</h4>
When it comes to package selection for statistical programmers in pharmaceutical organizations, the available packages are extensive. The choice of additional packages is often guided by the Risk-Based Approach provided by the <a href="https://www.pharmar.org/" target="_blank" rel="noopener">R Validation Hub</a>. The guidance offered by the R Validation Hub ensures that the selected packages align with the risk management strategies and compliance requirements of the pharmaceutical industry. More can be found in their <a href="https://www.pharmar.org/white-paper/" target="_blank" rel="noopener">white paper</a>.
<h5>Validated R Package Repository</h5>
The development of a validated R package Repository is underway by the R Validation Hub. The efforts for this initiative can be followed on their <a href="https://www.pharmar.org/white-paper/" target="_blank" rel="noopener">website</a>.
<h4>The <a href="https://pharmaverse.org/" target="_blank" rel="noopener">Pharmaverse</a></h4>
The Pharmaverse is collaborative effort by professional statistical programmers and developers at esteemed pharmaceutical companies like Roche, GSK, Johnson&Johnson, Merck, and others.

One notable aspect of the Pharmaverse is its publicly maintained packages, which undergo rigorous testing and code coverage. More information can be found on this <a href="https://posit.co/blog/pharmaverse-packages-for-clinical-reporting-workflows/" target="_blank" rel="noopener">blogpost</a>. As an example, the test coverage and code for the admiral package is <a href="https://github.com/pharmaverse/admiral/tree/main/tests/testthat" target="_blank" rel="noopener">available</a>. This transparency ensures that the packages are of high quality and can be trusted for critical clinical reporting workflows.
<h3>Building a Secure and Controlled Internal Package Repository for Clinical Trials</h3>
The first step towards building an internal repository is establishing a baseline of packages that have undergone rigorous validation and meet the organization's quality standards. The risk-based approach and the aforementioned open source repositories provide valuable guidance in selecting these validated packages.
<h4>Leveraging the Posit Package Manager</h4>
To streamline the process of curating and managing the internal repository, organizations can turn to the <a href="https://docs.posit.co/rspm/admin/" target="_blank" rel="noopener">Posit Package Manager</a>. This package manager facilitates internal repository creation for clinical trial packages which are specifically selected by Quality and IT for clinical trials. The steps to achieve this are outlined in this <a href="https://github.com/philbowsher/Creating-an-Internal-Repository-of-Validated-R-Packages" target="_blank" rel="noopener">GitHub</a> repository.
<h4>Benefits of an Internal Repository</h4>
Having an internal repository of validated packages brings several advantages to pharmaceutical organizations engaged in clinical trials. Firstly, it provides a centralized and secure source for packages, eliminating the need to rely on external repositories that may introduce unforeseen risks or inconsistencies. Secondly, it enables seamless collaboration and knowledge sharing among statistical programmers and researchers within the organization.
<h4>Maintaining Control and Compliance</h4>
Establishing an internal repository goes hand in hand with maintaining control and compliance throughout the clinical trial process. The internal repository ensures that all packages used in the trials have undergone a comprehensive validation process and adhere to the organization's quality standards.
<h4>Streamlining Package Management with renv for Statistical Programmers</h4>
The R package <a href="https://rstudio.github.io/renv/articles/renv.html" target="_blank" rel="noopener">renv</a> is an essential companion package that seamlessly integrates with industry-standard IT practices for effective package management. By utilizing <code>renv::init()</code>, statistical programmers can create a file called renv.lock, which captures the precise state of their project's library at a specific moment. This renv.lock file serves as a comprehensive record, meticulously documenting repositories and package versions, and can even be versioned with Git for enhanced control and collaboration. As an example, here is the <a href="https://github.com/RConsortium/submissions-pilot1/blob/main/renv.lock" target="_blank" rel="noopener">R FDA Pilot Submission renv.lock</a>.
<h3 id="tools">Testing R Functions: Essential Tools for Internal Package Validation</h3>
In the previous sections, we discussed the importance of selecting validated packages and establishing an internal repository for clinical trials. Here, we will explore key tools that aid in the testing and validation of R functions within these internal packages.
<h4>thevalidatoR</h4>
One of the essential tools for testing R functions is <a href="https://github.com/insightsengineering/thevalidatoR" target="_blank" rel="noopener">thevalidatoR</a>. This open-source package, available on GitHub provides a comprehensive framework for validating R functions. It provides an R Package Validation Report. An example for the report generated by this package is shown below.

<img class="wp-image-20752 size-full" src="https://webflow-prod-assets.s3.amazonaws.com/6525256482c9e9a06c7a9d3c%2F65b01abd97b38dfd19d8e0f3_example-of-thevalidatoR-for-clinical-trial-data-validation-reporting.webp" alt="example of thevalidatoR for clinical trial data validation reporting" width="1370" height="437" /> Image <a href="https://github.com/insightsengineering/thevalidatoR/blob/main/readme_files/rbmi_action.png" target="_blank" rel="noopener">source</a>
<h4>valtools</h4>
Another valuable resource for testing R functions is <a href="https://github.com/phuse-org/valtools" target="_blank" rel="noopener">valtools</a>. Developed by the Pharmaceutical Users Software Exchange (PhUSE), valtools offers a collection of utilities and functions specifically designed for the validation of R packages. The most common usage for valtools is to add elements required for validation under the R Package Validation Framework and facilitating validation.
<h4>testthat</h4>
<a href="https://testthat.r-lib.org/" target="_blank" rel="noopener">testthat</a> is the most popular unit testing package for R and is used by thousands of CRAN packages.  The testthat package, developed by the Posit team, promotes a test-driven development approach, ensuring that functions are thoroughly tested before being integrated into the internal packages.
<h3 id="reproducibility">Ensuring Reproducibility in Shiny Applications</h3>
Shiny apps have revolutionized the way we develop interactive web applications for data visualization and analysis. However, with their increasing complexity, ensuring their testability and reproducibility becomes paramount.

In this section, we will explore a set of powerful tools and frameworks that facilitate testing and reproducibility in Shiny apps, enabling developers to build robust and reliable applications.
<h4>Rhino</h4>
<a href="https://rhinoverse.dev/#rhino" target="_blank" rel="noopener">Rhino</a>, developed by Appsilon, is an opinionated framework with a focus on software engineering practices and development tools for building production-grade Shiny apps.

Rhino provides support in 3 main areas:
<ol><li style="font-weight: 400;" aria-level="1">Clear code: scalable app architecture, modularization based on Box and Shiny modules.</li><li style="font-weight: 400;" aria-level="1">Quality: unit tests, E2E tests with Cypress, logging and monitoring, linting.</li><li style="font-weight: 400;" aria-level="1">Automation: project startup, CI with GitHub Actions, dependency management with renv, configuration management with config, Sass and JavaScript bundling with ES6 support via Node.js.</li></ol>
The latest version of <a href="http://appsilon.com/rhino-1-5-0-addins/" target="_blank" rel="noopener">Rhino (1.5.0) adds addins to your RStudio IDE</a> for convenient and efficient R programming workflows.
<h4>shinytest</h4>
<a href="https://rstudio.github.io/shinytest/" target="_blank" rel="noopener">shinytest</a>, developed by Posit, provides a convenient simulation of a Shiny app that you can control in order to automate testing.
<h4>shinyValidator</h4>
<a href="https://github.com/Novartis/shinyValidator" target="_blank" rel="noopener">shinyValidator</a> is an open-source package specifically designed for testing Shiny apps. This package aims at automating the audit of a Shiny app project's quality, particularly required during a validation/qualification process.  All results are gathered in an HTML report uploaded and available to everyone on any CI/CD platform or Posit Connect.
<h4>golem</h4>
<a href="https://engineering-shiny.org/golem.html" target="_blank" rel="noopener">golem</a> is an opinionated framework for building production-grade Shiny applications. It simplifies the creation, development and deployment of a Shiny application as a package.
<h4>shinymeta</h4>
The <a href="https://github.com/rstudio/shinymeta" target="_blank" rel="noopener">shinymeta</a> R package offers convenient tools to capture the underlying logic of a Shiny app and convert it into executable code outside of the Shiny environment, such as the R console. Additionally, it facilitates bundling the code and its associated results, making it easier to share with end users.
<h4>TEAL</h4>
<a href="https://github.com/insightsengineering/teal" target="_blank" rel="noopener">teal</a> is a Shiny-based interactive exploration framework for analyzing data.

By adopting these tools, developers can streamline their testing processes, identify and resolve issues, and build robust Shiny apps that meet the highest standards of quality and reproducibility.
<h3>Alternative Approaches</h3>
Alternatively, some companies may opt to purchase validation documentation for the available open source packages. There are some vendors who offer validation tests and documentation as well as validation support services.

An example is Atorus, which offers OpenVal, a subscription based approach that contains a repository of nearly 200 validated packages.
<h2 id="conclusion">Concluding Remarks on R Packages for Clinical Trials</h2>
Ensuring the reliability, reproducibility, and validation of R packages and Shiny applications is paramount in the clinical research domain. By following a risk-based approach and utilizing resources like the R Validation Hub, Pharmaverse, and internal package repositories, organizations can establish a robust foundation of validated packages for their clinical trials.

Open-source tools such as thevalidatoR, testthat, and valtools contribute to R package validation, while packages such as Rhino, shinytest, and shinyValidator contribute to reproducibility of Shiny applications.

By adopting these best practices, pharmaceutical organizations can <strong>confidently leverage the power of R and Shiny for efficient and compliant clinical reporting workflows</strong>, ensuring the integrity and accuracy of their research findings.