Novo Nordisk stands as a trailblazer in R-based submissions to the FDA, having recently completed the inaugural submission in this statistical programming language to regulatory authorities. This milestone was achieved at the close of 2021, encompassing the inclusion of SDTM (Study Data Tabulation Model) and ADaM (Analysis Data Model) datasets, programs, documentation, analyses, and TLFs (Tables, Listings, and Figures) — all meticulously executed using R.

Revolution by evolution: “The goal is a transformation process that is revolutionary in result but evolutionary in execution”.

Table of Contents

• The Submission
• The Business Impact
• What was included in the Submission
• Future Submissions
• How Can Appsilon Help You with a Submission?

Watch the YouTube video where they talk about their Journey:

The Submission

In 2019, Novo Nordisk made a commitment to transition to another language. By 2020, their inaugural R-based trial output commenced, albeit with some encountered delays. Nevertheless, the first R-based submission to regulatory authorities was ultimately completed successfully.

The Business Impact

The utilization of R-based tools for submissions and clinical trial reporting has a far-reaching impact on various activities. Novo Nordisk highlights that one noteworthy application involves generating Shiny applications capable of utilizing R packages and enhancing functionality. Additionally, they express the need for more personnel to manage their trials, noting that contemporary individuals are more proficient in R compared to SAS. However, determining a specific Return on Investment (ROI) for this transformation poses a challenge.

What was included in the Submission

Extent on R in the FDA Submission Package

1. All TLFs for all 6 phase 3a trials.
   • All big data ADaM programs.
2. All TLF for ⅝ clinical pharmacology trials.
3. All TLF for Integrated Summary of Efficacy (ISE).
4. All TLF for Integrated Summary of Safety (ISS).
5. All BIMO (bioresearch monitoring) lists.

The statistical analysis is still done in SAS, put into a dataset and then reported through R-based TLFs programs.

Submitted ADaM Programs in R

1. Designed with minimal dependency on internal packages.
2. NNaccess is used for internal access to data but not submitted.
3. For SAS-based ADaM programs, they usually don’t submit their access-macro with internal library references.

FDA Type C Meeting Package

1. They had to mention to authorities about their use of R during the type C meeting package.
   • Clarify that they plan to submit R programs and specify in the ADRG.
     • Submit internal packages if they are not publicly available.

FDA Pre-BLA Meeting Package

‍

1. They intended to use the R package {pkglite}
   • Packages R packages to .txt in order to pass them through the eCDT gateway

Their first pilot submission to the FDA done by the R Consortium Working Group followed the same approach of using {pkglite}.

Shiny Applications for Regulatory Submissions

The R/Shiny Application FDA-Pilot-App rewritten in Rhino demonstrates that a Shiny App could be packaged by {pkglite} and submitted through the eCDT gateway. The FDA later clarified that “.zip” and “.r” files are acceptable for submission. Feel free to read more about our replication of the Shiny Application submitted in the R Submissions Working Group Pilot 2 project on the pharmaverse blog post.

‍

<video width="100%" height="auto" preload="metadata" controls>
 <source src="https://wordpress.appsilon.com/wp-content/uploads/2023/11/rhino_fda_pilot_r_shiny_app_e-ctd.webm?_=9">
 Your browser does not support the video tag.
</video>

‍

First Appendix: Instruction to ADaM Programs in R

Within the ADRG (Analysis Data Review Guide), they included simple instructions on how to run the ADaM programs. This is what the appendix looked like:

• Install R.
  • Download and install R 4.0.4 for Windows.
  • Specify the R package repository.
    • The R packages are based on CRAN at 2021-02-15. To install the exact R package version used in this project, run the code below to set the snapshot repository.
      • options(repos = https://cran.microsoft.com/snapshot/2021-02-15/”)
  • Install open-source R packages.
    • In the same R session, install the required packages by running the code below:
      • install.packages(c(“dtplyr”, “lubricate”))

However, this was not enough for the FDA, and they later received an information request. This led to them having to generate a new ADRG file, which was much larger and a new Appendix on how to run the new files. In total, the ADRG file was modified two times.

FDA 1^st Information Request

The FDA requested Novo Nordisk to submit all program codes that they used to create the TLFs to facilitate the efficiency of the safety review process.

The exact request was the following: “To facilitate the efficiency of the safety review, submit all programming codes to create the tables and figures included in the integrated summary of safety (ISS) and the main safety section of the clinical study reports”.

This was a huge drawback for them since they only had 10 days to provide the FDA with the programming codes.

‍

What they did in order to comply with this request was the following:

1. Use {renv} to find package versions through trial renv.lock.
   • Find versions of all internal packages.
2. Use {pkglite} to take package sources.
   • Repackage them all into pkglite-{trial}.txt for each trial.

FDA 2^nd Information Request

Novo Nordisk managed to resubmit the programming code used to create the TLFs using {renv} and {pkglite}; however, the FDA had trouble running all the code and came back to Novo Nordisk with a second information request:

“Submit R code for all safety analyses with as minimal package dependency as possible (and no dependency to R packages developed internally by Novo Nordisk)”.

This second information request can be summarized as follows:

1. The FDA was unsuccessful in restoring the environment.
   • Actions taken
     • Optimized the buildrenviron.r and optimized the renv.lock file to use Public Posit Package Manager (PPM).
     • Write a new version of the ADRG file (24 pages).
2. The FDA requested no dependency on R packages developed by Novo Nordisk.

After a meeting with the FDA, they were successful in restoring the environment, and the new code gives FDA an opportunity to learn about package development from Novo Nordisk. “Submissions in R are new to us but are definitely new to FDA as well”. – Ari Siggaard Knoph (MSc).

How was the Validation Performed?

Novo Nordisk decided to follow a risk assessment approach, which is based on the framework proposed by the R Validation Hub. The validation framework is shown below.

‍

At the end of the day, they had to perform a human-based risk assessment. Everything starts when someone requests a package via a Shiny Application. Once a submission is performed, it gets version control in git, and this kicks off a pipeline followed by the {riskmetric} package, which provides some test results and values.

‍

The values obtained from the {riskmetric} package are risk-assessed manually by a reviewer who eventually approves or rejects the new package and documents the final decision.

Whenever there is a new package approved, it is installed in the next release (docker). Having this process in place allowed Novo Nordisk to have a GxP approved environment for running R and a system/strategy for new packages.

Future Submissions

This first real R-based submission allowed Novo Nordisk to come up with a plan for future submissions done in this programming language, as they shared during their presentation. Their plans include the following:

1. Stronger R dedication.
2. Environment recreation improvements.
3. Fully embrace open-source.

‍

The Novo Nordisk team is planning to minimize and phase out most of their internal packages and move to open development on GitHub. Also, they will rebuild a lot of their core functionalities so that they will work for the broader community and aim to get their packages on to CRAN and dive deeper into pharmaverse.

Appsilon Open Source for Submissions

Throughout our journey with open source and developing tools for open source, we have provided the community with various tools that are applicable to the regulatory submissions done in R. In fact, one of the advantages of R versus other regulatory tools is that in R, you can take any tool that adapts to your processes and tailor it to your needs. Below are some examples of these tools.

R Package for Data Validation and Automation

‍

data.validator

The {data.validator} is a package for scalable and reproducible data validation. It provides the following features:

1. Functions for validating datasets in %>% pipelines: validate_if, validate_cols and validate_rows.
2. Predicate functions from the assertr package, like in_set, within_bounds, etc.
3. Functions for creating user-friendly reports that you can send to email, store in the logs folder, or generate automatically with RStudio Connect.

Data Validation Report

Data validation is an essential step in ensuring the reliability and accuracy of data. Businesses can reduce the risks associated with “bad” data and make well-informed decisions by following best practices such as using validation rules, developing routines and workflows, automating operations, and monitoring data quality.

Interested in learning more about data.validator, check out our article: Data Validation With data.validator: An Open-Source Package from Appsilon

Our Shiny Apps Framework on Pharmaverse

A couple of months ago, we submitted our Shiny Applications Rhino framework to the Pharmaverse repository, and it was accepted. Now, you can create Shiny applications using it. We already have examples in our demo applications and excellent resources to help you understand why our framework allows you to create enterprise-level Shiny applications tailored for the life sciences industry. See our first Pharmaverse blog post about it here “Reproducing the R Submissions Pilot 2 Shiny Application using Rhino”.

FDA Pilot App in Rhino

‍

Rhino + Life Sciences

This application is intended for a pilot submission to the FDA composed of a Shiny application, as part of the R Submissions Working Group Pilot 2. The data sets and results displayed in the application originate from the Pilot 1 project.

We built this application using our framework to integrate enterprise-level features.

‍

<video width="100%" height="auto" preload="metadata" controls>
 <source src="https://wordpress.appsilon.com/wp-content/uploads/2023/11/rhino_fda_pilot_r_shiny_app_e-ctd.webm?_=10" type="video/webm">
 Your browser does not support the video tag.
</video>

‍

R Submissions Working Group

The Pilot 4 Project

We have been working with the R Submissions Working Group to experiment with new technologies that could accelerate the submissions to the FDA. These two new experimental technologies are webR and containers.

During the first iterations of this ambitious project, we were able to set up a robust container environment for this pilot project; we made huge progress on both experimental technologies, developed a working prototype submission using Podman and developed a working prototype for wrapping a small Shiny application using WebAssembly.

The deployed example of the Shiny app running on webR is available here. Check out the video of the application running below.

<video width="100%" height="auto" preload="metadata" controls>
 <source src="https://dev.appsilon.com/static/290c306d0a294b5121fdf9650ad00e21/13ce77d1-3d00-44d5-b8a0-6d6a05e2d877-2.webm?_=11">
 Your browser does not support the video tag.
</video>

‍

Although there were various interesting alternatives for the right container, we decided to move forward with Podman. The running Podman demo is shown below.

How Can Appsilon Help You with a Submission?

We now possess extensive expertise in developing production-grade, enterprise-level Shiny Applications. Additionally, we are adept at leveraging and experimenting with new technologies that could significantly expedite the submission process to regulatory authorities. The wealth of experience we bring can be effectively utilized within your organization, ensuring seamless execution of R-based submissions.

Over the past few months, there have been substantial changes, and R is gaining increased popularity. We are at the forefront of R/Shiny development, along with other innovative technologies such as machine learning and infrastructure.

When you have a prototype of a Shiny application that you’d like to integrate into your clinical trial and overall drug development process, that’s the opportune moment to get in touch with us.

Let’s together accelerate the adoption of R for submissions.

Useful Links

• Advancing FDA Clinical Trial Submissions with R: Reproducing the R Submissions Pilot 2 Shiny App Using Rhino
• How Open Source (R and Shiny) Is Transforming Processes in the Pharmaceutical Industry
• Understanding Drug-Drug Interactions Using R Shiny

‍