Reproducible and Reliable Shiny Apps for Regulatory Submissions

Reading time:

time

min

February 8, 2024

<a href="https://appsilon.com/r-shiny-in-life-sciences-examples/" target="_blank" rel="noopener">Shiny Apps</a> have emerged as a powerful tool, transforming data analysis and visualization. These interactive web applications are crafted using the R programming language, and no knowledge of HTML, CSS, or JavaScript is required.

When it comes to industries governed by stringent regulations, such as pharmaceuticals and healthcare, an important question arises: <strong>Is validating Shiny Apps necessary for regulatory submissions?</strong>

In this post, we go through the concept of validation and assess whether these apps must be validated, considering various regulatory points.
<h3>Table of Contents</h3>
<ul><li><strong><a href="#risk-assessment-and-validation">Risk Assessment and Validation</a></strong></li><li><strong><a href="#why-r-matters-in-clinical-trials">Why R Matters in Clinical Trials?</a></strong></li><li><strong><a href="#shiny-apps-the-r-powerhouse">Shiny Apps – The R Powerhouse</a></strong></li><li><strong><a href="#r-validation-hub-insights">R Validation Hub Insights</a></strong></li> <li><strong><a href="#riskmetric-and-riskassessment-tools">Riskmetric and Riskassessment Tools</a></strong></li><li><strong><a href="#reliable-and-reproducible-shiny-apps-in-pharma">Reliable and Reproducible Shiny Apps in Pharma</a></strong></li><li><strong><a href="#the-shiny-app-validation-process">The Shiny App Validation Process</a></strong></li><li><strong><a href="#the-first-shiny-application-reviewed-by-the-fda">The First Shiny Application Reviewed by the FDA</a></strong></li><li><strong><a href="#bring-your-shiny-app-to-an-enterprise-level">Bring your Shiny App to an Enterprise-Level</a></strong></li></ul>

<hr />

<h2 id="risk-assessment-and-validation">Risk Assessment and Validation</h2>
The concept of validation can be quite diverse, often representing the internal procedures within an organization. We have previously discussed the latest <a href="https://appsilon.com/clinical-trial-r-package-quality-and-validation/" target="_blank" rel="noopener">insights on R Package Quality and Validation for Clinical Trials</a> in our blog post. According to the FDA, any package can be used for regulatory submissions.

R has now become an essential tool highly utilized for clinical trial data analysis and reporting. There is a growing number of R packages and Shiny applications that assist researchers in quickly deriving insights from their data.
<blockquote>Explore the future of FDA submissions: <a href="https://appsilon.com/testing-containers-and-webassembly-in-submissions-to-the-fda/" target="_blank" rel="noopener">Read our insights on Testing Containers and WebAssembly.</a></blockquote>
As outlined in the R Package Validation Framework <a href="https://phuse.s3.eu-central-1.amazonaws.com/Deliverables/Data+Visualisation+%26+Open+Source+Technology/WP059.pdf" target="_blank" rel="noopener noreferrer">white paper</a> developed by Phuse, the appropriate approach involves conducting a risk assessment of the software. It clearly specifies that the <strong>software used to calculate values or generate tables for decision-making purposes should undergo validation.</strong>

Following the available guidelines regarding the validation of software utilized for submissions, Shiny applications that are part of the decision-making process would require validation.
<h3 id="why-r-matters-in-clinical-trials">Why R Matters in Clinical Trials?</h3><ol><li>R is a key player in clinical trial data analysis and reporting.</li> <li>An increasing array of R packages and Shiny applications are enhancing research efficiency.</li></ol>
<h3 id="shiny-apps-the-r-powerhouse">Shiny Apps – The R Powerhouse</h3>
<img class="aligncenter wp-image-23267" src="https://webflow-prod-assets.s3.amazonaws.com/6525256482c9e9a06c7a9d3c%2F65e9e646a25310f3344d2560_6053e353_shiny-logo.webp" alt="" width="330" height="220" />

<a href="https://shiny.posit.co/" target="_blank" rel="noopener noreferrer">Shiny</a>, an open-source package for R, offers a sophisticated and robust web framework to develop web applications in R. It enables the transformation of your analysis into interactive web applications, all without the need for HTML, CSS, or JavaScript expertise.
<h3 id="r-validation-hub-insights">R Validation Hub Insights</h3>
For statistical programmers in the pharmaceutical industry, the selection of software packages is quite broad. The incorporation of extra packages typically follows the guidance of the Risk-Based Approach by the <a href="https://www.pharmar.org/" target="_blank" rel="noopener noreferrer">R Validation Hub</a>. This advice ensures conformity of the selected packages with industry-specific risk management tactics and adherence to compliance norms. Their <a href="https://www.pharmar.org/white-paper/" target="_blank" rel="noopener noreferrer">white paper</a> provides more information.
<h4>Validated R Package Repository</h4>
The R Validation Hub is currently in the process of creating a validated R package Repository. Updates on this project are available on <a href="https://www.pharmar.org/white-paper/" target="_blank" rel="noopener n">their website</a>.
<h4 id="riskmetric-and-riskassessment-tools">Riskmetric and Riskassessment Tools</h4>
The R Validation Hub provides two risk-based software packages to solve the validation problem for R packages. These are the {riskmetric} package and the {riskassessment} application.

{riskmetric} offers a process consisting of five key stages for assessing the quality of a collection of R packages. This method aids users in selecting R packages of superior quality, enhances the dependability of these packages, and supports the confirmation of R packages' credibility in industries that are subject to strict regulations.

{riskassessment} is an R package that includes a Shiny interface, designed to enhance the functionality of the riskmetric package in an organizational setting. This addition provides a more user-friendly and interactive platform, making the riskmetric package's capabilities more accessible and practical for use within a corporate environment.
<h3 id="reliable-and-reproducible-shiny-apps-in-pharma">Reliable and Reproducible Shiny Apps in Pharma</h3>
Pharmaceutical companies are starting to use Shiny applications for internal clinical trial data exploration and reporting. As mentioned above, validation is a term used to express reliability and reproducibility; these characteristics can be achieved when the developed Shiny applications follow good programming practices and are developed at an enterprise level.

There are some examples of the Shiny applications used in clinical trials’ data exploration:
<ol><li>Realtime Clinical Summaries <a href="https://realtimeclinical.shinyapps.io/rsum/" target="_blank" rel="noopener noreferrer">Shiny app</a><ol><li>This application shows 6 derived datasets from the publicly available SDTM/ADaM datasets from the CDISC Pilot Project using the 'Realtime Summary Model' (RSuM).</li></ol>

</li><li>GOahead Shiny app
<ol><li>A Shiny app to help with the decision-making process during drug development.</li><li>Read more <a href="https://arxiv.org/ftp/arxiv/papers/2201/2201.11267.pdf" target="_blank" rel="noopener noreferrer">in this research paper.</a></li></ol>
</li><li>safetyGraphics <a href="https://cran.r-project.org/web/packages/safetyGraphics/readme/README.html" target="_blank" rel="noopener noreferrer">app</a>
A Shiny application that allows users to explore safety data using a series of interactive graphics.</li>
</ol>
The Shiny apps should be tested, reproducible and reliable; there are several tools developed for this purpose:
<ol><li>{<a href="https://rstudio.github.io/shinytest/" target="_blank" rel="noopener noreferrer">shinytest</a>}It provides a simulation of a Shiny app that you can control in order to automate testing.</li><li>{<a href="https://rstudio.github.io/shinytest2/" target="_blank" rel="noopener noreferrer">shinytest2</a>}shinytest2 provides a streamlined toolkit for unit testing Shiny applications and seamlessly integrates with the popular <a href="https://testthat.r-lib.org/" target="_blank" rel="noopener noreferrer">testthat</a> framework for unit testing R code.</li> <li>{<a href="https://opensource.nibr.com/shinyValidator/" target="_blank" rel="noopener noreferrer">shinyValidator</a>}<ol><li>It aims at automating the audit of a Shiny App project’s quality of particular interest during a validation/qualification process. {shinyValidator} performs the following:<ol><li>Lints the code to ensure consistency and quality.</li><li>Runs a devtools::check() to assess package quality.</li> <li>Checks whether the Shiny app starts and is available after heavy usage.</li><li>Performs output comparison between commits.</li><li>Launches a load test to measure app performance to identify bottlenecks with <a href="https://rstudio.github.io/shinyloadtest/" target="_blank" rel="noopener noreferrer">shinyloadtest</a>.</li><li>Records an overall code profiling with <a href="https://rstudio.github.io/profvis/" target="_blank" rel="noopener noreferrer">profvis</a>.</li><li>Audits reactivity with <a href="https://rstudio.github.io/reactlog/" target="_blank" rel="noopener noreferrer">reactlog</a>.</li><li>Provides an overview of project structure with <a href="https://github.com/moodymudskipper/flow" target="_blank" rel="noopener noreferrer">flow</a>.</li></ol>
</li>
</ol>
</li>
<li><a href="https://appsilon.github.io/rhino/" target="_blank" rel="noopener">Rhino</a>
<ol> <li>This framework allows you to create Shiny applications <strong>The Appsilon Way</strong> - like a fullstack software engineer. It aids in applying best software engineering practices, modularizing the code, testing it well, making a beautiful UI and considering user adoption from the very beginning.</li> <li>It supports Shiny app development in 3 main areas:<ol><li><strong>Clear code:</strong> scalable app architecture, modularization based on Box and Shiny modules.</li><li><strong>Quality:</strong> unit tests, E2E tests with Cypress, logging and monitoring, lining.</li><li><strong>Automation:</strong> project startup, CI with GitHub Actions, dependency management with {renv}, configuration management with config, Sass and JavaScript bundling with ES6 support via Node.js.</li></ol>
</li>
<li>All of these features can be implemented using various well-known packages, but Rhino brings them all together out of the box.</li>
</ol>
</li>
<li><a href="https://engineering-shiny.org/golem.html" target="_blank" rel="noopener noreferrer">Golem</a>
<ol> <li>This is a framework for building production-grade Shiny applications. It is a toolkit for simplifying the creation, development and deployment of a Shiny application.</li></ol>
</li>
<li>{<a href="https://github.com/rstudio/shinymeta" target="_blank" rel="noopener noreferrer">shinymeta</a>}
<ol> <li>This package provides tools for capturing logic in a Shiny app and exposing it as code that can be run outside of Shiny (e.g., from an R console). It also provides tools for bundling both the code and results to the end user.</li></ol>
</li>
</ol>
<h4 id="the-shiny-app-validation-process">The Shiny App Validation Process</h4>
The Validation of Shiny apps could follow a process that is similar to the validation of R packages. Usually, a <a href="https://appsilon.com/r-decision-treees/" target="_blank" rel="noopener">decision tree</a> first analyses the level of risk of the package, or in this case, the Shiny app, and then a human needs to verify the results. The process is semi-automated.
<h5>CI/CD Pipeline</h5>
In terms of CI/CD, previously <a href="https://phuse.s3.eu-central-1.amazonaws.com/Archive/2023/Connect/EU/Birmingham/PAP_AD21.pdf" target="_blank" rel="noopener noreferrer">a push-based mechanism</a> to continuously deploy a Shiny application was proposed. In this scenario, the developers will trigger the publication of the application by pushing code to specific branches in the code repository. The initial deployment is done manually, while the subsequent deployments are handled automatically.

Also, to mitigate the risk of automatically deploying content by non-publishing roles, a mechanism was suggested to review and validate deployments using a pull request rather than using a direct push to branches that are associated with automatic deployment.
<h2 id="the-first-shiny-application-reviewed-by-the-fda">The First Shiny Application Reviewed by the FDA</h2>
According to the clarifying statement released by the FDA, they do not require the use of any specific software and statistical software: Title 21 of the Code of Federal Regulations [e.g., in <a href="https://www.accessdata.fda.gov/scripts/cdrh/cfdocs/cfcfr/CFRSearch.cfm?CFRPart=11" target="_blank" rel="noopener noreferrer">21CFR part 11</a>].

The FDA, in collaboration with the R Submissions Working Group, has been testing this and other technologies to improve the submission process and move away from relying only on legacy software (such as SAS). The pilot 2 project was bundled with a simply shiny application that the FDA successfully reviewed; the submissions working group received an <a href="https://www.r-consortium.org/announcement/2023/10/05/shiny-app-successfully-reviewed-by-fda-cder-staff-pilot-2-announcement-2" target="_blank" rel="noopener noreferrer">FDA CDER response letter on September 27, 2023</a>.

This opens the door for submitting Shiny applications to the FDA since <strong>this technology has already been tested and positively reviewed by the FDA</strong>. Shiny applications could enhance the way the clinical trial data is reviewed, for example, by generating interactive clinical study reports (iCSRs) that would allow reviewers to explore the data and gather insights; this would eventually lead to a reduction in the needed time for a review, fast tracking the approval of new drugs.
<blockquote>Explore Novo Nordisk's innovative path: <a href="https://appsilon.com/first-r-based-submission-to-fda-by-novo-nordisk/" target="_blank" rel="noopener">Learn more about their first R-Based FDA submission.</a></blockquote>
<h2 id="bring-your-shiny-app-to-an-enterprise-level">Bring Your Shiny App to an Enterprise-Level</h2>
Enlist our expertise to advance your Shiny app to an enterprise-grade, thereby ensuring compliance with regulatory bodies. For more information or to discuss this further, we welcome you to <a href="https://appsilon.com/#contact" target="_blank" rel="noopener">reach out to us</a>.
<blockquote>Join us at PHUSE US Connect 2024! Visit Booth 17 to connect and engage with our team.</blockquote>
<a href="https://explore.appsilon.com/workshop?utm_medium=referral&utm_source=blog&utm_campaign=levelup" target="_blank" rel="noopener"><img class="aligncenter size-full wp-image-23154" src="https://webflow-prod-assets.s3.amazonaws.com/6525256482c9e9a06c7a9d3c%2F65e9e6279f041da2b2645342_572728fc_Banner_Ebook.webp" alt="" width="1070" height="447" /></a>

‍