A Guide to Validating R (Shiny) Applications in Pharma

Reading time:

time

min

October 11, 2024

Validating R Shiny applications is critical in the pharmaceutical industry, where data integrity, compliance, and patient safety are essential. Building on our post on R package validation, Shiny apps introduce unique challenges due to their interactive nature and real-time data processing compared to packages. To meet FDA (US), EMA (EU), NMPA (China), and other regulatory standards standards, these apps must undergo rigorous GxP validation to ensure they function reliably and securely.

Curious how to make GxP compliance easier from the start? Check out our post on why it all begins with the Definition of Done.

As the shift towards open-source tools like R continues, validating both packages and interactive apps is increasingly important. Frameworks like Rhino and Teal support this process. These tools help ensure Shiny apps maintain reproducibility, performance, and usability standards in clinical settings.

In this post, we'll explore the challenges of validating Shiny apps in pharma, as well as the guidelines, best practices, and tools that can streamline the validation process.

What is R App Validation and Why Does It Matter?

R app validation is the process of establishing documented evidence that an R-based application, such as a Shiny app, consistently performs its intended functions accurately, reliably, and in compliance with regulatory standards. According to GxP ("Good Practice") guidelines — validation must demonstrate a high degree of assurance that the application produces results that meet predetermined specifications under all expected conditions.

In the pharmaceutical context, validation ensures that software tools used in clinical trials, drug development, and regulatory submissions maintain data integrity, meet compliance requirements, and ultimately protect patient safety. Unlike static R packages, Shiny apps are dynamic, involving real-time data inputs, user interactions, and visualizations, which adds complexity to the validation process.

Unique Challenges of Validating Interactive Applications like Shiny Apps

Validating R Shiny applications presents unique challenges that differ significantly from those faced when validating traditional scripts or batch processes. These challenges include:

Dynamic Interactions

Unlike static scripts, Shiny apps are highly interactive and reactive. They must handle real-time data inputs and provide instant outputs, making validation more complex. Every possible user interaction and data input scenario needs to be considered during the validation process.

User Behavior Variability

Shiny apps are often used by a diverse group of users, ranging from data scientists to clinicians with varying levels of technical expertise. The app must function correctly regardless of how it is used — whether it’s handling complex queries from an advanced user or navigating simple tasks by a less experienced user. This requires extensive user acceptance testing (UAT) [testing in the real world, sometimes called beta testing] to ensure that the app meets usability standards for all possible user types.

User Access and Security

In many cases, Shiny apps require managing different access levels for various user groups. Some users might need restricted access to certain features or datasets based on their roles.

Testing for unauthorized access is crucial to ensure that sensitive data remains secure and users only have access to the functions they are permitted to use. This involves rigorous role-based testing and security checks to ensure that unauthorized users cannot gain access to restricted areas of the app.

Want to keep your Shiny apps secure? Dive into our guide on making R Shiny apps safer.

Performance Under Load

Shiny apps frequently handle large datasets and multiple concurrent users. They must perform reliably under varying loads — from a single user analyzing a small dataset to a scenario where many users are simultaneously querying large datasets. This requires rigorous performance testing, including load testing to simulate multiple users and stress testing to identify potential bottlenecks or failures under extreme conditions.

Cross-Platform Compatibility

Shiny apps must perform consistently across various browsers, devices, and operating systems. Ensuring this cross-platform compatibility requires thorough testing to identify and resolve any bugs or inconsistencies that could affect user experience or data accuracy.

Tools like shinylive and webR are particularly valuable for this purpose: shinylive allows Shiny apps to run directly in web browsers without relying on an R server, simplifying testing across different environments and reducing dependencies. Meanwhile, webR enables R code to execute in the browser using WebAssembly, ensuring reliable functionality on any platform. Together, these tools help maintain consistent performance and usability across all platforms.

‍

“The promise of WebAssembly with webR is that researchers can produce analytics with R & Shiny, … and be as confident as is possible that any user will be able to get it up and running quickly on their browser. And it will be run in a decades’ time exactly the same way as it is today. …”
George Stagg, webR maintainer and engineer at Posit PBC.

Regulatory Requirements for Validation

To operate within regulated environments, Shiny apps must comply with several key regulatory requirements. Validation principles will remain consistent across regulatory agencies, although details on the process or specific documentation might differ. Let’s see some examples of those principles and regulatory standards:

GxP Validation

GxP guidelines, which include Good Clinical Practice (GCP), Good Laboratory Practice (GLP), and Good Manufacturing Practice (GMP), set the standards for the development and maintenance of software in regulated industries.

For Shiny apps, GxP validation involves ensuring the app performs as intended in all situations, with documented evidence of all validation activities. This includes a comprehensive risk assessment, detailed testing plans, traceability matrices to map requirements to tests, and change management processes to handle updates or modifications.

FDA Guidelines

The FDA’s "General Principles of Software Validation" provides specific guidelines on how to validate software used in clinical trials or for regulatory submissions. It emphasizes that software must be validated to produce accurate, reliable, and consistent results, with thorough documentation for each stage of the validation process. The FDA also focuses on risk management, requiring that all potential risks to patient safety or data integrity be identified and mitigated during validation.

EMA Guidelines

The EMA outlines similar requirements, particularly through the Good Automated Manufacturing Practice (GAMP) guidelines.

GAMP focuses on ensuring that all computerized systems used in regulated activities — including Shiny apps — are validated to perform reliably and securely. It mandates strict controls over data security, access, and audit trails to ensure data integrity and compliance with regulatory standards.

Compliance with these regulatory requirements is essential for any Shiny app used in pharmaceutical contexts. Failure to validate according to these standards can lead to severe consequences, including financial penalties, regulatory sanctions, or compromised patient safety.

Key Validation Criteria for Shiny Applications

Reproducibility
Ensuring that Shiny applications deliver consistent results across different environments is crucial for regulatory compliance. Unlike static R packages, Shiny apps must also account for user interactions and real-time data processing. This requires validating that the app produces the same outputs given the same inputs, regardless of factors like operating system, browser type, device, or user behavior.

Validation should cover all dynamic elements, such as input fields, reactive expressions, and visualizations, confirming they function consistently under various scenarios and environments.

Functionality
The core of any validation process is ensuring that the app performs its intended functions correctly and reliably. This means that all components of the app, including input fields, data processing scripts, output visualizations, or generated reports, should work as expected without errors. Each feature should be tested to confirm it operates as intended under a range of scenarios.

Error handling in Shiny Applications is usually more complex and challenging than in R Packages. That is because we should handle errors at different levels (user actions, input data, other software) and in a growing number of scenarios. Therefore, It is important to design your app carefully so it is prepared to react in all situations (wrong input data, wrong input parameters, API not available …) quickly and user-friendly.

Performance
The app must perform efficiently under different conditions, such as varying data sizes or multiple simultaneous users. Performance validation involves stress-testing the app to identify bottlenecks or resource constraints and ensure it meets predefined performance standards.

Data Security and Privacy

Applications handling sensitive clinical data must comply with data protection regulations like GDPR or HIPAA. This involves validating that the app employs adequate security measures, such as encryption, access controls, and data anonymization, to protect patient data and maintain privacy.

This is a topic that, again, is more challenging with Shiny Apps than with packages. Developers should consider app security from the beginning of the app design. It is important to always give only the necessary access to sensitive data, to be able to switch environments easily, and be careful with external software.

‍

Software-Based Validation for Shiny Apps

This approach treats Shiny apps like any other software product, emphasizing a structured Software Development Life Cycle (SDLC), automated testing, Continuous Integration/Continuous Deployment (CI/CD), comprehensive documentation, and modular design.

For Shiny apps, this means not only validating individual components and functions but also shiny modules, UI components, and outputs and ensuring the app performs reliably during real-time data processing and user interactions. Automated testing tools specific to Shiny, like end-to-end tests and shinyValidator, should be used to cover all dynamic elements, while CI/CD pipelines ensure that any changes are continuously tested and validated.

Guidelines for Validating R Shiny Applications in Pharma

Here’s a structured guide to help you navigate the validation process, anticipate common challenges, and select the right tools.

Step 1: Define Validation Requirements

What to Do: Clearly outline the validation requirements for your R Shiny application. This includes identifying the intended use, specifying functional, performance, security, and usability criteria, and understanding the regulatory requirements (such as GxP, FDA, or EMA guidelines).
Challenges to Expect: Defining comprehensive requirements that cover all potential use cases and regulatory expectations can be complex. It’s crucial to involve all relevant stakeholders, including developers, users, and compliance officers.
Tools to Use: Document all validation criteria using a requirements management tool. Create a validation plan that maps these requirements to specific validation activities.

Step 2: Plan and Design the Validation Process

What to Do: Develop a detailed validation plan that includes the scope, approach, resources, timelines, and responsibilities. This plan should outline the types of tests to be performed (e.g., unit tests, integration tests, user acceptance tests) and the tools you will use.
Challenges to Expect: Ensuring that the validation plan is comprehensive and aligns with regulatory standards can be time-consuming. You may also face difficulty balancing thorough testing with development timelines.
Tools to Use: Project management tools like Jira or Trello can help organize the validation plan and track progress. Use a version control system (like Git) to manage changes to the validation documents.

Step 3: Implement Automated Testing

What to Do: Set up automated testing to validate the app's functionality and performance. Begin with unit tests to verify that individual components work correctly, followed by integration tests to check how different parts of the app interact.
Challenges to Expect: Developing comprehensive automated tests that cover all scenarios can be challenging, especially in highly interactive Shiny apps. Keeping the tests up-to-date with app changes requires ongoing effort.
Tools to Use: Use testthat for unit testing and integration test scripts. Test your shiny modules. Create fixtures and mock components to cover more scenarios. Tools like shinytest2 or Cypress can automate end-to-end testing, capturing the state of the app and checking for deviations from expected behavior.

Step 4: Conduct Performance Testing

What to Do: Profile the app to identify bottlenecks. Validate the app's performance under different conditions, such as varying user loads or large datasets. This step ensures that the app remains responsive and efficient, even during peak usage.
Challenges to Expect: Simulating real-world usage conditions can be complex. Identifying and fixing performance bottlenecks requires a deep understanding of the app's architecture.
Tools to Use: Use shinyloadtest to simulate multiple users and test the app’s load handling. Use profvis and shiny.tictoc to profile the app and identify performance bottlenecks, helping you optimize resource use and improve response times.

Step 5: Perform User Acceptance Testing (UAT)

What to Do: Conduct User Acceptance Testing (UAT) with a diverse group of users, including clinicians, data scientists, and non-technical staff. This testing helps identify usability issues and confirms that the app meets users' needs.
Challenges to Expect: Coordinating UAT sessions with diverse stakeholders can be logistically challenging. Ensuring that feedback is actionable and leads to meaningful improvements can also be difficult.
Tools to Use: Use feedback tools like Google Forms or SurveyMonkey to collect input from users during UAT sessions. Document all feedback and action items in a centralized platform like Confluence or Notion.

Step 6: Ensure Data Security and Privacy Compliance

What to Do: Validate that the app complies with data security and privacy regulations (such as GDPR or HIPAA). This includes encrypting data, implementing access controls, anonymizing sensitive data, and maintaining audit trails.
Challenges to Expect: Meeting all security requirements while maintaining app performance can be challenging. Ensuring that all user data is adequately protected requires regular updates and security assessments.
Tools to Use: Implement data encryption with SSL/TLS certificates. Use tools like keycloak for authentication and authorization management. Set up automated logging and monitoring tools to maintain audit trails and track data access.

Shiny application security is a fundamental necessity. Learn more in “How to Make Your Shiny Apps Secure.”

Step 7: Implement Continuous Integration/Continuous Deployment (CI/CD)

What to Do: Set up a CI/CD pipeline to automate testing and deployment processes. This ensures that every change to the app is validated before it goes live, reducing the risk of errors and maintaining a validated state.
Challenges to Expect: Integrating CI/CD pipelines with existing development workflows can require significant effort upfront. Ensuring that the pipeline covers all necessary tests and validations is critical.
Tools to Use: Use tools like GitHub Actions or Jenkins to create CI/CD pipelines. These tools automate testing, deployment, and monitoring, ensuring that changes are validated continuously.

Step 8: Maintain Validation Over Time

What to Do: Continuously monitor the app’s performance, security, and compliance to maintain its validated state. Re-validate the app after any updates or changes to ensure ongoing compliance.
Challenges to Expect: Keeping the app validated over time requires ongoing effort, including regular updates, monitoring, and re-validation after changes. Balancing this with the need for frequent updates can be difficult.
Tools to Use: Set up automated monitoring tools like Datadog or Prometheus to track app performance and security in real-time. Use a version control system to manage updates and maintain a comprehensive change log.

Step 9: Prepare for release for submission

What to Do: Once all the requirements have been implemented, automated tests and UATs are passed you need to prepare for your submission process. Register all your tests in a test management system. Switch to the production environment and perform additional checks, if necessary. Review your documentation, timestamp and sign all documents, so you will be ready to answer any question and recall all the development process with transparency.
Challenges to Expect: If testing has been extensive enough and the development environment is a good replica of the production environment, no errors or very few errors should appear at this stage. Automation of processes can also help the creation and revision of documents.
Tools to Use: Quarto or R Markdown to automate the creation of documents. Conventional Commits and other good practices in your repository to easily track all your changes.

Tools and Best Practices for Validating R Shiny Applications

This section provides an overview of the most important tools and best practices that can help facilitate validation, maintain compliance, and ensure high performance.

Tools for Validation

Valtools: Designed specifically for clinical research, this tool automates the validation process, making it easier to meet compliance requirements.
profvis: Profiles R code to identify performance bottlenecks, helping to optimize the app's efficiency.
shinyloadtest: Simulates user traffic to test the app's performance under load, identifying potential bottlenecks and ensuring stability under high demand.
rsconnect: Deploys Shiny apps and monitors performance metrics over time, providing valuable insights into app performance in production environments.
rhino: A framework for building enterprise-grade Shiny apps using a modular design that improves maintainability and scalability. It integrates with CI/CD tools to automate testing and deployment.
teal: Creates interactive data exploration apps for clinical trials with features like real-time data filtering, code reproducibility, and regulatory-ready reporting. It integrates with pharmaverse packages like {admiral} and {rtables} to support clinical data analysis and reporting.
shinymeta: Facilitates reproducible reporting by exposing the underlying R code behind reactive expressions, enhancing transparency in regulatory settings.
GitHub Actions / Jenkins: Automate testing and deployment processes, ensuring that every change to the app is validated before going live.
R Markdown / Quarto: Document all validation activities, including test plans, results, and change logs. These tools provide flexible and accessible formats for generating validation reports required for audits and inspections.

A Successful FDA Submission Using R

The R Consortium's Submission Working Group hit an important milestone with Pilot 4, showcasing how open-source tools like R and new technologies—WebAssembly and containers—can streamline FDA submissions. This effort is all about updating the way clinical trial data is submitted, making the process more efficient and transparent.

The Process

Pilot 4 was all about testing if a Shiny app could be packaged and submitted using WebAssembly and containers. Here’s how the team made it happen:

Collaborating with experts from Appsilon, Posit, and others to develop new submission methods and solve technical challenges together.
Reusing code from earlier pilots but tweaking it to work with WebAssembly and container environments, which are a bit newer but much more flexible.
Building a working prototype, using shims and creative fixes to handle package dependencies that aren’t available on CRAN, allowing the app to run directly in a web browser with webR.
Overcoming challenges like adjusting the UI and reworking the app structure to fit the new technologies.

Tools Used

The Outcomes

Pilot 4 proved that it’s possible to submit a fully functioning Shiny app using these cutting-edge technologies. It showed that:

Containers and WebAssembly make it easier to package and share apps without needing complex software installations.
Open-source tools make the submission process more transparent and help regulatory authorities verify data more easily.
The new approach streamlines submissions, reducing headaches and making it simpler for FDA reviewers to interact with the data.

Conclusion

Pilot 4 highlights how open-source technologies can improve FDA submissions, making the process faster, clearer, and more efficient. By collaborating with industry partners and using tools like webR and containers, the team is pushing the boundaries of what’s possible in clinical trial reporting.

Read the full blog post: Testing Containers and WebAssembly in Submissions to the FDA.

Wrapping Up A Guide to Validating R Applications in Pharma

Validating R Shiny applications in the pharmaceutical industry is crucial for maintaining data integrity, regulatory compliance, and patient safety.

As open-source tools like R continue to gain traction, the unique challenges of validating dynamic and interactive applications must be addressed with robust frameworks and best practices. By implementing automated testing, performance monitoring, and comprehensive documentation, pharmaceutical companies can ensure their applications remain reliable and secure in clinical environments.

If you need expert guidance to simplify your validation process and ensure compliance with regulatory standards, our team is here to help. Contact us today to learn how we can support your journey to achieving secure and validated R Shiny applications.

Looking to ensure your clinical software is GxP-compliant? Learn how to validate Shiny apps in our next post.

Resources and Communities

Communities

Have questions or insights?

Engage with experts, share ideas and take your data journey to the next level!

A Guide to Validating R (Shiny) Applications in Pharma

What is R App Validation and Why Does It Matter?