R Shiny for Enterprise Application Development: 6 Lessons for Building Shiny Apps

<em>Updated on 8 September 2021</em> What do Viacom, Ubisoft, and Bank of America have in common? Hint: they are Fortune 500 companies with R Shiny in their tech stack. It’s also no coincidence that they are all using open source packages for Shiny app development created at Appsilon. Data science teams know that developing data analytics solutions in an enterprise environment is a complex process and choosing the right technology is key to project success. R Shiny developers at Appsilon are well aware of those challenges and are constantly improving our Shiny know-how to reflect the current business needs. One of our core competencies is building decision support systems with R Shiny. So far we have delivered over 100 commercial Shiny dashboards. If you’re just about to make a technology choice for a data product or start an R Shiny project, I’m inviting you to tap into our experience. <ul><li><a href="#anchor-1" target="_blank" rel="noopener noreferrer"><span style="font-weight: 500;">Why R Shiny for enterprise applications?</span></a></li><li><a href="#anchor-2" target="_blank" rel="noopener noreferrer"><span style="font-weight: 500;">Six lessons for R Shiny app development</span></a></li><li style="font-weight: 500;" aria-level="1"><a href="#anchor-3" target="_blank" rel="noopener noreferrer"><span style="font-weight: 500;">1. Start with a prototype</span></a></li><li style="font-weight: 500;" aria-level="1"><a href="#anchor-4" target="_blank" rel="noopener noreferrer"><span style="font-weight: 500;">2. Good architecture</span></a></li><li style="font-weight: 500;" aria-level="1"><a href="#anchor-5" target="_blank" rel="noopener noreferrer"><span style="font-weight: 500;">3. Tests are critical</span></a></li><li style="font-weight: 500;" aria-level="1"><a href="#anchor-6" target="_blank" rel="noopener noreferrer"><span style="font-weight: 500;">4. Validate the data and app state</span></a></li><li style="font-weight: 500;" aria-level="1"><a href="#anchor-7" target="_blank" rel="noopener noreferrer"><span style="font-weight: 500;">5. Scale and performance</span></a></li><li style="font-weight: 500;" aria-level="1"><a href="#anchor-8" target="_blank" rel="noopener noreferrer"><span style="font-weight: 500;">6. Deployment and automation</span></a></li><li><a href="#anchor-9" target="_blank" rel="noopener noreferrer"><span style="font-weight: 500;">Make your enterprise R Shiny app production-ready with experts</span></a></li></ul> <h2 id="anchor-1">Why R Shiny for enterprise applications? </h2> This question should be asked for every technology and <a href="https://shiny.rstudio.com/" target="_blank" rel="noopener noreferrer">R Shiny</a> is no different. We believe that Shiny is indeed the ideal choice for many types of projects. By comparing Shiny with other available tools we can determine when to use it. <img class="aligncenter wp-image-2547 size-full" src="https://webflow-prod-assets.s3.amazonaws.com/6525256482c9e9a06c7a9d3c%2F65b022de7493b499786558c0_WHY-SHINY.webp" alt="R Shiny compared with Django, React, Angular, Excel, and Tableau." width="582" height="311" /> <h3>Speed of development for starters</h3> Shiny has an unmatched speed of development. This speed enabled us to strike our first technology partnership with a Fortune 500 company: we implemented a proof of concept of the requested application within 24 hours. This would have been impossible using any other framework or programming language. Fast prototyping is especially important for strategic business management. Volatility, uncertainty, complexity, and ambiguity – VUCA – is now the norm in many industries and requires a new approach to decision-making. In this context, R Shiny can offer a remedy to the most pain points affecting <a href="https://solutionsreview.com/business-intelligence/whats-changed-2021-gartner-magic-quadrant-for-data-science-and-machine-learning-platforms/" target="_blank" rel="noopener noreferrer">data science teams focused on innovation</a>. <h3>Built-in efficiency gains</h3> <a href="https://appsilon.com/excel-is-obsolete-here-are-the-top-2-alternatives-from-r-and-python/" target="_blank" rel="noopener noreferrer">Compared to spreadsheets</a>, R Shiny gives you the tools to create an attractive user interface and introduce automation in the background. It makes the development work more efficient when dealing with shared resources, reproducibility, and most importantly, the source code. <a href="https://appsilon.com/forget-about-excel-use-r-shiny-packages-instead/" target="_blank" rel="noopener noreferrer">Switching from a spreadsheet to a Shiny app</a> can give your organization the competitive edge that comes with improved efficiency. Apart from the core functionalities for data manipulation, conditional formatting, or pivot tables, Shiny applications offer additional features like plotting and expandable rows. This way, your data solutions are no longer limited in terms of advanced analytics and data quality or versioning and scalability. <h3>Customization as a bonus</h3> You might be wondering how R Shiny compares to the BI tools featured in Gartner’s Magic Quadrant. <a href="https://appsilon.com/top-3-bi-tools/" target="_blank" rel="noopener noreferrer">Business Intelligence applications</a> such as Tableau and PowerBI have their rightful place among top data management tools but it comes at a price. Because they don’t give you access to source code, you are limited to the features included in the selected plan. If full customization and version control are among your priorities, opting for R Shiny can be a strategic move. Plus, Shiny apps generate virtually no running fees and offer better <a href="https://appsilon.com/r-decision-treees/" target="_blank" rel="noopener noreferrer">Machine Learning (ML)</a> possibilities. All in all, a Shiny app is a perfect choice whenever you need a data product that is more complex than a spreadsheet or a BI dashboard. The more so, if you’re not entirely sure of the scope and requirements of the project. In such cases, R Shiny gives you the freedom to experiment and quickly validate your ideas. This, in turn, is crucial for many R&D initiatives. R Shiny can empower your data team to develop cutting-edge solutions that are fit-for-purpose and seamlessly connected to your internal resources. <h2 id="anchor-2">Six lessons for R Shiny app development</h2> Now that you’ve done your due diligence for technology choice, imagine a Shiny app that you want to build. I’ll walk you through the process of development and implementation. I'll be pointing out the project milestones and adding useful tips to help you avoid common mistakes and drawbacks. <img class="aligncenter wp-image-2548 size-full" src="https://webflow-prod-assets.s3.amazonaws.com/6525256482c9e9a06c7a9d3c%2F65b0f1da7b19860c4c04c1b3_Copy-of-useR-2019-Best-practices-for-building-Shiny-enterpri_90370e51.webp" alt="6 lessons for enterprise shiny app development." width="641" height="337" /> <h3 id="anchor-3">1. Start with a prototype</h3> The first lesson is simple: <b>avoid overspending by taking a more incremental approach</b>. Instead of building a tool with many users in mind, start with a small group of individuals who are willing to help and are actively interested in using the app. Stay agile and make the development process iterative. This way, you will get to a point when these initial users become ambassadors of your Shiny app and recommend the tool to their peers. Such an organic buy-in gives a better rationale for committing more resources to the project. <img class="aligncenter wp-image-2551 size-full" src="https://webflow-prod-assets.s3.amazonaws.com/6525256482c9e9a06c7a9d3c%2F65b022e022f94709a7c05383_START-WITH-PROTOTYPE.webp" alt="R Shiny projects from prototyping to production." width="619" height="309" /> Prototyping and PoCs are crucial because if you build a tool without <a href="https://appsilon.com/user-tests-build-better-shiny-apps-with-effective-user-testing/" target="_blank" rel="noopener noreferrer">speaking directly with the stakeholders</a> – the app users – you might end up with a large, expensive project that doesn’t fulfill its purpose. What’s more, early feedback will give you valuable <strong>insight into the key user flows to optimize</strong> and help you fix the UI issues that affect the <a href="https://appsilon.com/ux-design-of-shiny-apps-7-steps-to-design-dashboards-people-love/" target="_blank" rel="noopener noreferrer">user experience (UX)</a>. <h3 id="anchor-4">2. Good architecture   </h3> Modular architecture can make life simpler and the same goes for code. Using modules to break the app logic into smaller, independent parts makes the code easier to maintain and verify. With Shiny modules, you should be able to extract business logic and test it for robustness. Modular architecture can also improve app performance. It allows for <strong>extracting heavy computations</strong> and enables pre-calculating the values before running the app. You can achieve that by using the <a href="https://github.com/HenrikBengtsson/future" target="_blank" rel="noopener noreferrer">future R package</a> or even a <a href="https://www.rplumber.io/" target="_blank" rel="noopener noreferrer">REST API</a>. <img class="aligncenter wp-image-2552 size-full" src="https://webflow-prod-assets.s3.amazonaws.com/6525256482c9e9a06c7a9d3c%2F65b022e2df1312180459e0c4_TRAITS-OF-GOOD-ARCHITECTURE.webp" alt="Traits of good architecture for Shiny apps." width="280" height="198" /> At Appsilon, we have pioneered the space of supporting large numbers of concurrent users with Shiny. We were the first company to demonstrate that you can have over 700 users for a single Shiny dashboard. The previous conventional wisdom was that a Shiny dashboard could host only several users at a time. The right app architecture can help you prepare for scaling. If you wish to scale your Shiny app, you should avoid loading the data into the Shiny app from a server. Instead of loading a session for each individual user, make sure to load the data from a database. <h3 id="anchor-5">3. Tests are critical</h3> Quality Assurance is a big topic in software development and a separate domain of expertise. Still, I’ve met both engineers and business practitioners with a love/hate relationship with tests. They love to have them, but they hate to create them. Sympathies aside, the right combination of tests offers a big payoff in the long run by reducing downtime and resources needed to fix bugs and respond to alerts. A <a href="https://appsilon.com/super-solutions-for-shiny-architecture-5-automated-tests/" target="_blank" rel="noopener noreferrer">well-designed testing procedure</a> starts with many simple automated unit tests to verify the basic concepts in the app logic. Then, the testing should reduce in number but increase in complexity, until you get to the point of a few end-to-end tests. This way, you can limit the time required for manual testing. <img class="wp-image-2553 size-full" src="https://webflow-prod-assets.s3.amazonaws.com/6525256482c9e9a06c7a9d3c%2F65b022e20b330f2efeac16e2_test-pyramid.webp" alt="Ideal software testing pyramid." width="611" height="332" /> Source: <a href="https://alisterbscott.com/tag/testing-pyramid/" target="_blank" rel="noopener noreferrer">Alister B Scott</a> Unfortunately, this is not yet the standard for Shiny apps. What we often see is an inverse pattern called the “Test Cone” (below). <img class="wp-image-2554 size-full" src="https://webflow-prod-assets.s3.amazonaws.com/6525256482c9e9a06c7a9d3c%2F65b022e4855531924257af6b_test-cone-anti-pattern.webp" alt="Test cone showing typical bad practices in software testing." width="576" height="347" /> Source: <a href="https://alisterbscott.com/tag/testing-pyramid/" target="_blank" rel="noopener noreferrer">Alister B Scott</a> The “Test Cone” approach is far from optimal, with few or no unit tests and a large number of end-to-end (e2e) tests. End-to-end tests take a lot of time and can eventually discourage people from the testing process. Fewer tests may result in more bugs going into production, which is exactly what you want to avoid. <h3 id="anchor-6">4. Validate the data and app state</h3> Validating the logic and the source code is critical, but you also need to validate the data before running the Shiny app. Data quality issues are actually very common and tend to build up due to data dependencies. Even small errors can have a big impact on business continuity or costs of upkeep. The good news is data validation can be automated. The <a href="https://appsilon.com/data-validation-with-data-validator-an-open-source-package-from-appsilon/" target="_blank" rel="noopener noreferrer">data.validator open source package</a> can save you a lot of time and effort while ensuring no corrupt or incomplete data affects the quality of your analysis. To make the most of your data validation process, try applying best engineering practices: <ol><li><b>Designate an owner for each dataset.</b>This person should take responsibility and react appropriately if any test fails.</li><li><b>Introduce a logging infrastructure</b>.The log should include information both about good uses and errors in the app.</li><li><b>Make the app resilient and fault-tolerant.</b>If the API used in the app goes down, the app should still be useful to the greatest extent possible. We don’t want business users to go through an R stack in their browsers.</li></ol> At Appsilon, we are hyper-vigilant about data quality, so we methodically identified the different kinds of problems that can happen to the data. To learn more, read about our <a href="https://appsilon.com/data-quality/" target="_blank" rel="noopener noreferrer">data quality case studies</a> and the ways to validate data on different levels. Make sure to set up these checks to avoid many errors in the production environment. <h4>Avoiding errors: 4 quick tips</h4> <img class="aligncenter wp-image-2555 size-full" src="https://webflow-prod-assets.s3.amazonaws.com/6525256482c9e9a06c7a9d3c%2F65b022e422f94709a7c055b1_avoid-errors-in-production.webp" alt="4 tips on how to avoid errors in production of Shiny apps." width="931" height="422" /> To prevent further errors, it’s best to set up a daily data validation email or text with alerts for the data owners. Make your Shiny app more resilient by using Continuous Integration to validate code and style automatically. This can be implemented using <a href="https://github.com/jimhester/lintr" target="_blank" rel="noopener noreferrer">lintr</a>: an R package for static code analysis. Finally, it sometimes makes sense to show data status directly in the app, so that end-users know exactly which data set they are working on. This can be achieved using the open source package <a href="https://github.com/Appsilon/shiny.info" target="_blank" rel="noopener noreferrer">shiny.info</a>. <h3 id="anchor-7">5. Scale and performance</h3> The app has been tested and is now up and running. You should be ready to commit substantial resources and launch it at scale. Shiny apps are commonly run using multiple processes and quite often using multiple servers. <a href="https://www.rstudio.com/products/connect/" target="_blank" rel="noopener noreferrer">RStudio Connect</a> in combination with <a href="https://aws.amazon.com/" target="_blank" rel="noopener noreferrer">Amazon Web Services</a> (AWS) can provide such a complete solution. Once you have the setup, you need to run performance tests. <img class="aligncenter size-full wp-image-1579" src="https://webflow-prod-assets.s3.amazonaws.com/6525256482c9e9a06c7a9d3c%2F65b022e62dc98004783b8f26_Revolver.webp" alt="Revolver scaling Shiny. Sample architecture." width="889" height="380" /> The image above is a real-life example of a deployment architecture. The load balancer distributes work among applications in RStudio Connect. The master node is responsible for performing the test and gathering statistics to summarize them. To make this even easier, use an R script that gathers the logs and creates a report from the performance tests. <h3 id="anchor-8">6. Deployment and automation</h3> Although deployment and automation appear in this compilation at the end, they shouldn’t come as an afterthought. As part of an optimized implementation, <b>you should have at least two deployment environments</b>. This way developers can test on a different server than the one that the users are on. Implementing changes and testing solutions without the risk of affecting the users empowers developers to get creative and bulletproof the app for different scenarios. Automated deployment helps streamline workflows and pays off fast. It allows you to operate on cloud resources, making it very easy to roll back or to create another instance of an app in several seconds. It’s a great way to increase productivity and receive instant feedback, which translates to a better quality of the final release. <h2 id="anchor-9">Make your enterprise R Shiny app production-ready with experts</h2> R Shiny is now much more than just a means to deliver beautiful user interfaces. The interactive nature of Shiny apps makes them a powerful data visualization tool that’s instrumental in decision-making. The process of turning a Shiny app into a useful solution ready for production in an enterprise environment calls for a wide range of sophisticated engineering tasks. I hope these 6 lessons for Shiny app development will help you turn an R Shiny prototype into a production-ready enterprise solution. <b>Do you want to build an ambitious enterprise dashboard or need help scaling an existing prototype?</b> Don’t hesitate to reach out to us.