R in FDA Submissions: Lessons Learned from 5 FDA Pilots
What does it really take to get R and Shiny applications accepted in FDA submissions?
Pharma companies are actively testing that question through a series of R Consortium-led pilots with the FDA. Their efforts aim to identify what works, what doesn't, and how tools like Shiny, container tech, and WebAssembly can shape the future of open-source regulatory submissions.
In the recent years, the pilots have evolved from simple static tables to sophisticated containerized applications, proving that R isn't just a viable option — it's production-ready.
Curious how containers and WebAssembly fit into FDA submissions? See what Pilot 4 revealed in our blog post.
This article breaks down the insights shared in a recent Shiny Gathering featuring Ben Straub, Eric Nantz, and Sam Parmar. They discussed their work on the FDA Pilot Initiative from static TLFs in R to dynamic Shiny apps and containerized environments.
If your team is using R for clinical reporting or considering R for regulatory submission, these lessons will help you understand the opportunities and challenges of going open source in a highly regulated space.
Table of contents
- A Quick Recap: What Are the FDA Pilots?
- Pilot by Pilot: What We've Learned
- What the FDA Actually Thinks
- Challenges to Adoption
- The Role of the Community
- What's Next?
- Summing up R in FDA Submissions
A Quick Recap: What Are the FDA Pilots?
The FDA pilots are initiatives set up by the R Consortium Working Group to test whether R can be used reliably for regulatory submissions.
These pilots began in 2021 and have been gradually increasing in complexity, with each one being designed to explore specific aspects of R-based submissions.
The R Consortium working group's mission
The mission is straightforward: make R-based clinical trial regulatory submissions easier by showing open examples of current submission processes. This hands-on approach is looking to collect feedback directly from FDA reviewers and influence future industry and agency decisions.
"We're not trying to build a giant R submission," explains Ben Straub from GSK. "We're really just trying to explore how the regulatory agencies receive R scripts, datasets made in R, or TLFs made in R, and what their responses are."
The working group has been transparent throughout this process, with all code, documentation, and feedback publicly available. This open approach helps the entire industry learn from these experiences rather than keeping the knowledge siloed.
Evolution from Pilot 1 to Pilot 5
Each pilot builds on lessons from previous ones and explores increasingly complex aspects of R in regulatory submissions. Here's a brief overview of each of the five pilots:
Pilot 1 (2021-2022): Focused on delivering just four TLFs (Tables, Listings, and Figures) created in R using simulated data. This pilot received positive feedback from the FDA, confirming that static outputs from R could be successfully reviewed.
Pilot 2 (2022-2023): Explored whether a Shiny app containing the same TLFs could be delivered through the Electronic Common Technical Document (eCTD) portal. The FDA provided valuable feedback on what they found useful in the Shiny app and areas for improvement.
Pilot 3 (2023-2024): Instead of using pre-existing data, this pilot used R to generate the Analysis Data Models (ADaMs) that fed into the TLFs. The FDA approved this submission in April 2024, validating that R could be used for data preparation.
Pilot 4 (In progress): Compares two methods of delivering Shiny applications - WebAssembly and container technology. Surprisingly, the FDA reviewers initially found the WebAssembly approach easier to work with.
Pilot 5 (Just starting): Will explore using dataset-JSON format instead of XPT files, potentially eliminating the need for XPT files in submissions altogether.
Why these pilots matter
These initiatives aren't just academic exercises – they're creating real-world pathways for pharmaceutical companies to use open-source tools in regulatory submissions.
For sponsors and CROs, they provide practical and tested approaches to including R into their regulatory workflows. The documentation, code, and feedback from these pilots serves as blueprints that others can follow.
More importantly, they're shifting the conversation from "Can we use R?" to "How can we best use R?" This represents a significant evolution in regulatory science, where open-source tools are moving from experimental to essential.
The pilots have also revealed unexpected insights, such as the FDA's preference for WebAssembly over containers and their comfort with CRAN-hosted packages. These findings directly influence how companies approach their R-based submissions.
"We think that will lead to easier R-based clinical trial regulatory submissions tomorrow by collecting feedback and influencing future industry and agency decisions on system process setup," notes Ben Straub, which highlights the long-term impact of this work.
Pilot by Pilot: What We've Learned
Each FDA pilot has explored specific aspects of R-based submissions, and each resulted in valuable lessons. Let's walk through what the team discovered in their journey from static outputs to state of the art tech.
Pilot 1: Static tables, listings, and figures (TLFs) in R
Pilot 1 focused on the basics: could the FDA successfully review static TLFs created in R? The team submitted four TLFs using simulated data from an existing CDISC pilot.
The process wasn't just about the outputs – it was about packaging everything correctly for submission through the eCTD portal. This meant careful attention to documentation, including detailed instructions on how reviewers could reproduce the results.
"It took a little while to kind of package all that up and do the submission," noted Ben Straub, reflecting on the challenges of this first step.
The FDA's response was encouraging. In 2022, they provided positive feedback, effectively giving a "gold star" to this first attempt. This confirmed a fundamental point: R could successfully generate regulatory-grade TLFs that met FDA standards.
This pilot laid important groundwork by establishing that R-based submissions could pass regulatory review for static outputs, which is the most common type of submission materials.
Pilot 2: Wrapping TLFs into a Shiny app
Building on the success of Pilot 1, the team took things a step further by creating a Shiny app that contained the same TLFs. The key question: could an interactive application be delivered through the eCTD portal and successfully reviewed?
The Shiny app included filtering capabilities that allowed reviewers to explore the data in ways not possible with static outputs. However, this raised important considerations about statistical interpretation.
Eric Nantz from Eli Lilly explained a key learning: "For tables that had p-value inferences, our reviewers said that if the analysis plan didn't specify running that same model for specific patient subsets, filtering could give the wrong impression of what we often call p-hacking."
The compromise was to remove p-values from filtered tables while keeping filtering capabilities for visualizations that didn't involve statistical inference. This balanced interactivity with analytical integrity.
The team also learned valuable lessons about package management. They used renv with lock files to ensure reproducibility, which gave reviewers step-by-step instructions to install the exact package versions used in development.
Ultimately, Pilot 2 was successfully reviewed in October 2023 and has demonstrated that interactive Shiny applications can be part of regulatory submissions when implemented properly.
Want to see how we rebuilt this pilot app using Rhino? Check out our blog on reimplementing FDA Pilot 2 with a structured, reproducible approach.
Pilot 3: Using R to generate ADaMs
Pilot 3 shifted focus to data preparation, using R to generate the Analysis Data Models (ADaMs) that feed into the TLFs. This tested another critical aspect of the submission workflow.
"Instead of taking that data that was used in the original CDISC pilot way back in the day, we actually used R to generate the ADaMs that then fed into the three tables and one figure," explained Ben Straub.
This pilot was submitted in August 2023, and after some back-and-forth with the regulatory agency, received a letter of approval on April 19, 2024. The longer timeline reflects the reality that these pilots are side projects for FDA reviewers, who must balance them with their primary work.
The success of Pilot 3 validated that R could be used not just for creating outputs, but also for the critical data preparation steps that underpin regulatory analyses.
Pilot 4: WebAssembly vs. container technology
Pilot 4 represents a significant technological leap, as it compares two modern approaches to delivering Shiny applications: WebAssembly and container technology.
WebAssembly allows Shiny applications to run directly in the browser without needing a separate Shiny server. Containers (using technologies like Docker) package the entire application environment and ensure consistent execution across different systems.
"We thought, why not test both of these, compare and contrast the differences, how much easier it is to encapsulate the Shiny app into either of these frameworks, and more importantly, what are the challenges?" said Eric Nantz.
The results were surprising. Initial assumptions favored containers as the easier option for FDA reviewers, but reality proved different.
"What was interesting is that as our reviewers at FDA started to understand more of the WebAssembly technology, they actually said it might be easier to start with that one first," Nantz explained. "They didn't need a container runtime to run the WebAssembly app. They already had what they needed: a web browser."
The team discovered that reviewers preferred compiling packages on their own machines rather than using pre-compiled versions, despite the longer installation time. This preference for control over the environment is an important insight for future submissions.
Pilot 4 is still in progress, with both approaches being refined based on FDA feedback. The final results will provide valuable guidance on the most effective ways to deliver interactive applications in regulatory submissions.
Pilot 5: exploring dataset-JSON
Just getting started, Pilot 5 tackles a fundamental aspect of submissions: the data format itself. Traditionally, submissions use XPT files, but this pilot will explore dataset-JSON, a newer format being developed with CDISC.
"Our big attempt here is to write out everything using a new format called dataset-JSON," said Ben Straub. "While not approved by the FDA, I'm pretty sure it's definitely being reviewed, and it seems to be very positive."
Dataset-JSON offers a couple advantages over XPT files, including better handling of special characters, improved metadata, and compatibility with modern data science workflows.
The team is collaborating directly with Nicholas Masel, one of the developers of dataset-JSON, ensuring they can leverage the latest version for their work. This highlights how the R Consortium working group is connecting with the broader ecosystem to advance regulatory science.
Pilot 5 represents a potential future where submissions don't rely on legacy data formats. This is important because it streamlines the use of open-source tools in regulatory contexts.
What the FDA Actually Thinks
Getting direct feedback from FDA reviewers has been one of the most valuable aspects of these pilots. Their responses provide concrete insight into how regulators view open-source tools in submissions and what matters most to them when reviewing R-based materials.
Reviewer feedback and preferences
FDA reviewers have generally been positive about R-based submissions, but their feedback reveals some preferences about implementation details.
"They were able to install everything and run all of our code just from the ADRG instructions," Ben Straub noted regarding Pilot 3, highlighting that well-documented submissions can succeed regardless of the programming language used.
The reviewers appreciate extensive documentation. Eric Nantz emphasized, "You can never have too much documentation on these details." This contrasts with SAS submissions, where minimal setup documentation is typically needed ("SAS 9.4" might suffice).
Screenshots of Windows operating systems that show step-by-step installation processes were also helpful. This level of detail might seem excessive, but it makes sure reviewers can reproduce the analysis environment without frustration.
Reviewers also showed a clear preference for standard practices. They favor packages hosted on CRAN over proprietary or internally developed packages, likely due to CRAN's vetting process and public availability.
The surprising preference for WebAssembly
Perhaps the most unexpected finding from Pilot 4 was the FDA's preference for WebAssembly over container technology – contrary to what the team initially predicted.
"I had this conception, maybe it was a bias, that it was going to be the container technology version of Pilot 4 that would be easier for them to administer and install because WebAssembly is a relatively newer framework," Eric Nantz explained.
The reviewers found WebAssembly more accessible because it required only a web browser, which they already had available. In contrast, container technology required additional setup steps that created more "hoops they have to jump through."
However, the team still had to provide a way to launch a web process via the httpuv package. Once the httpuv package was updated to make bootstrapping easier, they could give reviewers detailed instructions to launch the web process and run the application.
This preference wasn't obvious until midway through development. It paints the full picture on how valuable the ongoing communication is with regulatory reviewers throughout the pilot process.
Package management considerations
Package management emerged as a critical concern for reviewers, with clear preferences about how dependencies should be handled.
When it came to WebAssembly applications, reviewers actually preferred compiling packages on their own machines rather than using pre-compiled versions. Eric Nantz noted, "We learned that they prefer installing everything in a dependency standpoint on their infrastructure without having that pre-compiled version."
This preference for control over the environment extended to how packages were obtained. Reviewers were comfortable using a lock file (similar to traditional renv infrastructure) to bootstrap packages from CRAN, even though installation could take 5-10 minutes.
The FDA also showed a preference for Docker over Podman for container-based solutions, despite Docker's "unique license changes from a couple years ago" that had prompted many organizations to consider alternatives.
These insights help organizations make informed decisions about package management strategies for their own submissions, potentially saving significant time and reducing the risk of review delays.
Balancing interactivity with analysis integrity
One of the most substantive pieces of feedback concerned the tension between interactive features and statistical integrity – particularly relevant for Shiny applications.
In Pilot 2, the team initially included filtering capabilities that allowed reviewers to subset data and view updated tables, including ones with p-values. The FDA expressed concerns about the potential for misinterpretation.
"If this analysis plan didn't specify that you're going to run that same model for say this cut of patients with this age cutoff or lab measure cutoff, that could give the wrong impression of what we often call P-hacking or inflating type one error for your p-value assessments," Eric Nantz explained.
The compromise solution was this: remove p-values for filtered tables but keep filtering capabilities for visualizations that didn't involve statistical inference. This balanced interactivity with proper statistical practice.
The team also added disclaimers to the interactive sections, emphasizing that filters being applied weren't impacting established clinical analyses. One commenter during the discussion suggested that such disclaimers might be sufficient in future submissions, though reviewers were "pretty firm about taking that out in this initial submission."
This feedback provides clear guidance for organizations developing interactive submissions: carefully consider the statistical implications of dynamic features and provide appropriate context for any results that change based on user input.
Challenges to Adoption
Despite the success of these pilots, a couple of challenges still stand for organizations looking to implement R-based regulatory submissions. Understanding these hurdles can help teams better prepare and develop strategies to address them.
Proprietary packages and cross-platform compatibility
One significant challenge involves handling proprietary packages developed in-house. While FDA reviewers show a clear preference for CRAN-hosted packages, many pharmaceutical companies develop custom packages for proprietary statistical methods or company-specific workflows.
"If you have a custom package that you submit rather than having on CRAN, it just opens up another risk to the submission being slowed down," Ben Straub pointed out. "These are billion-dollar drug submissions, and you don't want your programming language to be the bottleneck."
The pilots haven't fully resolved how to handle these proprietary packages effectively. Container technology might offer a solution by packaging everything together, but as Pilot 4 has shown, containers come with their own implementation challenges for FDA reviewers.
Cross-platform compatibility adds another layer of complexity. Code developed on Linux or Mac systems must reliably run in the FDA's Windows environment. This requires careful testing across operating systems – something Eric Nantz emphasized when describing his Pilot 4 work: "I have to test all of this in a Windows virtual machine to ensure that every instruction that we're outlining in the Analysis Data Reviewers Guide for setting up the execution environment is actually going to work."
Documentation and Analysis Data Reviewers Guide (ADRG)
Documentation requirements for R-based submissions are higher than for traditional SAS submissions. The Analysis Data Reviewers Guide (ADRG) must provide detailed instructions for recreating the analysis environment.
"With the ADRG, you can just say 'SAS 9.4' for SAS. With R, you can really open a can of worms with how much information you put in there," noted one participant in the discussion.
Finding the right balance is tricky. Insufficient documentation risks reviewers being unable to run the analysis, while excessive information can be overwhelming. Eric Nantz spent "countless hours" on the ADRG for Pilot 2, focusing on package installations and environment setup.
This challenge has led to a separate working group within PHUSE specifically focused on improving ADRG standards for open-source submissions. "I think the PHUSE working group that's looking at this will hopefully produce some really good guidance on what to include but maybe not overburdening the reviewers with a lot of extra information," said Ben Straub.
Sam Parmar added that efforts are underway to develop tools that could automatically populate parts of the ADRG, potentially streamlining this process for future submissions.
Package dependencies and abandonment Risk
The risk of package abandonment represents a serious concern for regulatory submissions that may be reviewed years after submission or referenced in future work.
"What happens if someone abandons the package that your whole submission depends on? That's something I think internally you have to assess," cautioned Ben Straub. "If one guy from an obscure university is maintaining this package that is critical to your submission, that's a huge risk."
This risk underscores the importance of carefully evaluating dependencies. Straub recommended using packages from established initiatives like Pharmaverse or maintained by companies with long-term commitments to the R ecosystem.
Package version management also presents challenges. While lock files help ensure reproducibility, the resulting installation process can be time-consuming for reviewers. Pilot 2 used renv with a lock file, which required reviewers to wait 5 to 10 minutes while packages installed from CRAN.
Windows environment considerations
Maybe the most practical challenge is ensuring that R-based submissions work smoothly in the FDA's Windows environment.
"The reviewers have Windows operating systems. You better work on Windows," Eric Nantz stated plainly. While developers may prefer other operating systems, submissions must function properly in the FDA's environment.
This requirement complicates the use of container technologies in Pilot 4. On Windows systems, running Docker or Podman requires Windows Subsystem for Linux (WSL), adding another layer of complexity that reviewers must navigate.
WebAssembly appeared to simplify this challenge somewhat, as it primarily relies on web browsers already available to reviewers. However, even WebAssembly solutions required running an R process to launch the web server.
Nantz suggested that container technology might eventually help overcome these operating system differences: "The potential of this container environment is that in theory, the actual operating system that's running that code is not Windows anymore. Windows is just that layer to get to the container." This represents a potential future state where development environments and review environments could be identical regardless of operating system.
For now, organizations implementing R-based submissions must thoroughly test their code in Windows environments and provide detailed instructions specific to Windows users.
The Role of the Community
The success of the FDA pilots highlights the power of community collaboration in solving complex regulatory challenges. This work is part of a broader ecosystem of open-source initiatives that have transformed R&D in pharma.
Open Collaboration in the R Consortium
The R Consortium provides vital infrastructure and support for these FDA pilots, as it hosts websites, GitHub repositories, and facilitating collaboration across companies that might otherwise be competitors.
"First I just wanted to give a shout out to the R Consortium which is our sponsor that helps drive this initiative," Ben Straub emphasized. "They have a lot of other projects they sponsor that drives a lot of great change in open source."
This collaborative model has proven effective for regulatory innovation. Representatives from pharmaceutical companies, CROs, and technology vendors work together, sharing expertise and distributing the workload.
Monthly meetings bring the broader community together to discuss progress and generate new ideas. The detailed meeting minutes are publicly available, providing insights even for those who can't attend directly. Straub noted they contain "some good gems about some of the discussions we have around different technologies, some of the blockers we have for working with the FDA."
How to get involved
The R Consortium working group welcomes new participants from any organization interested in advancing R-based submissions.
"Anyone can join from any pharma, from a CRO, from academia, or if you're just kind of interested in learning about CDISC, R, or submission processes," explained Straub. "We're always looking for people to contribute. Small contributions are welcomed, or if you want to get really in the weeds of it, those are also welcome."
Getting involved is easy:
- Join the R Consortium Slack channel (slack.r-consortium.org)
- Navigate to the #wg-submissions channel
- Attend the monthly working group meetings
- Explore the GitHub repositories and documentation
Pilot leadership opportunities are also available for those interested in driving future initiatives, with potential spin-off working groups addressing specific needs like automated ADRG generation.
Related initiatives: Pharmaverse and R Validation Hub
The FDA pilots connect with several other important initiatives in the pharmaceutical R ecosystem.
Pharmaverse is a collection of open-source R packages designed specifically for clinical development. "Packages like that are on the Pharmaverse, or by companies like Appsilon or Atorus – those are the packages you should be looking at," advised Straub when discussing package reliability.
Packages mentioned during the discussion included {admiral} (for ADaM data creation) and {teal} (for building Shiny applications), both part of the Pharmaverse ecosystem.
The R Validation Hub addresses another critical need: validation of R packages for regulatory use. This initiative is developing frameworks and tools to assess the risk of using specific R packages in regulated environments.
"The validation group is very active, and I encourage people to check it out on their Slack channel," said Straub. "You can also look at risk metric, risk assessment, some of the work from Jumping Rivers."
The Hub is also exploring the concept of a "golden image" – a standardized R environment that could serve as a baseline for pharmaceutical submissions, potentially simplifying the process in the future.
What's Next?
The work so far has been a reason for celebration, but the R Consortium working group is constantly looking ahead to new opportunities in the regulatory submission landscape.
Pilot 5 progress and Dataset-JSON
Pilot 5 represents an exciting shift away from traditional XPT files toward dataset-JSON, a modern format being developed with CDISC.
"Our big attempt here is to write out everything using a new format called dataset-JSON which has been explored with CDISC," explained Ben Straub. While not yet formally approved by the FDA, the feedback has been encouraging.
This format offers several advantages, including better handling of metadata, improved support for special characters, and compatibility with modern data science workflows. It represents a significant step toward modernizing the technical infrastructure of regulatory submissions.
The team is collaborating directly with Nicholas Massel, one of the developers of dataset-JSON, ensuring they're using the latest version for their work. This connection with the standards community helps ensure the pilot reflects current best practices.
Future directions and potential new pilots
A couple of potential directions for future pilots emerged during the discussion.
Eric Nantz highlighted Nix and its R interface Rix as another approach to managing dependencies. Unlike containers, which create a completely separate environment, Nix creates isolated sandbox environments that can still interact with the host system when needed.
The possibility of Python-based pilots was also mentioned, acknowledging Python's growing importance in data science. "People ask a lot about Python now, which is a much more dominant language in the open-source community," noted Straub.
The PHUSE Open Source Metadata Documentation working group's efforts may also influence future pilots as documentation standards evolve.
While specific plans upcoming pilots haven't been announced, the working group remains open to new ideas that could further advance open-source tools in regulatory submissions.
Automation opportunities
A recurring theme was the potential for automation to streamline the submission process, particularly around documentation.
Sam Parmar mentioned that "we've already thought about how we can start automating some of the ADRG generation side of things." This automation could significantly reduce the time spent creating detailed documentation while ensuring consistency.
The goal would be tools that automatically extract information about the R environment, packages, and dependencies, formatting it appropriately for inclusion in the ADRG. This would address one of the major challenges identified in earlier pilots – the extensive documentation required for R-based submissions.
While current political and budgetary challenges at the FDA may affect the timeline for future pilots, the momentum behind open-source tools in regulatory submissions continues to build.
Summing up R in FDA Submissions
In this article, we've somewhat briefly covered the key points from the recent Shiny Gathering session. If you have the time, we strongly encourage you to watch the full 1-hour-long episode on YouTube.
Open source in regulatory spaces
Open-source tools are no longer experimental in regulatory spaces – they're being validated in real-world submissions. The FDA pilots have demonstrated that R and Shiny can meet the rigorous standards required for regulatory review while offering advantages in reproducibility, transparency, and interactivity.
The successful review of Pilot 2 in October 2023 marked a significant milestone: the FDA's acceptance of an interactive Shiny application as part of a submission package. This opens new possibilities for how data can be presented and explored during regulatory review.
These successes don't mean the journey is complete. Challenges remain with package management, proprietary code, and cross-platform compatibility. But the pathways are now clearer, with documented approaches that others can follow.
The roadmap for sponsors and CROs
For sponsors and CROs considering R for regulatory submissions, these pilots offer a valuable roadmap:
- Start with CRAN-hosted packages whenever possible
- Test thoroughly in Windows environments
- Document extensively, with step-by-step instructions
- Consider containerization or WebAssembly for interactive applications
- Explore dataset-JSON for future submissions as an alternative to XPT files
- Leverage community resources like Pharmaverse for validated, industry-specific packages
The path from static outputs to interactive applications is now better defined, with proven examples at each stage. The work on dataset-JSON and container technologies points toward even more efficient approaches in the future.
Join the movement
The success of these pilots relies on community collaboration. You can contribute to this important work:
- Join the R Consortium Working Group to participate directly in upcoming pilots
- Explore the GitHub repositories to see code examples and implementation details
- Review meeting minutes to understand current challenges and discussions
- Connect with the community on Slack at slack.r-consortium.org in the #wg-submissions channel
- Check out related work from PHUSE on open-source documentation standards
Whether you contribute code, documentation, ideas, or simply spread the word, you're helping to build a future where open-source tools play a central role in regulatory submissions – making the process more efficient, transparent, and accessible for all.
Want to get involved? Join the R Consortium working group to contribute to upcoming pilots or follow Pharmaverse for updates on the open-source clinical ecosystem. Read more on Appsilon’s blog or visit R Consortium’s submission working group page to stay in the loop.
Curious how open source is changing clinical trials? Download our ebook to see how teams are using tools like R and Shiny to speed up analysis and reporting.
