At Appsilon Data Science, we have worked on dozens of R Shiny applications, and we and have learned a thing or two along the way. We have even developed popular open source packages for Shiny that are used by a variety of Fortune 500 companies such as Viacom, Ubisoft, and Bank of America. Below we’ll explain why R Shiny is a good choice for enterprise applications and provide six lessons to help you succeed with your R Shiny project.
We specialize in building data products for enterprise clients. One of our core competencies is building decision support systems with R Shiny. So far we have delivered over 50 commercial Shiny dashboards. We have also pioneered the space of supporting large numbers of concurrent users with Shiny. We were the first company to demonstrate that you can have over 700 concurrent users for a single Shiny dashboard. The previous conventional wisdom was that a Shiny dashboard could host only several users at a time.
Overall, we gathered a host of best practices in preparing Shiny projects from ideation to production in a structured corporate environment. In this article, I will discuss the specific steps you can take to achieve reliability without sacrificing the speed of development. I will show you how to avoid costly mistakes and ensure that your project is successful.
This question should be asked for every technology and Shiny is no different. We believe that Shiny is indeed the ideal choice for many types of projects. By comparing Shiny with other available tools we can determine when to use it.
Shiny has an unmatched speed of development. This speed enabled us to win our first deal with a Fortune 500 company – we implemented a proof of concept of the requested application within 24 hours. We could have never achieved this result if we had used any other framework or language.
Compared to spreadsheets, Shiny allows for creating an attractive user interface and introducing automation in the background. It makes the development work more efficient when dealing with shared resources, scalability, reproducibility, and most importantly, source code. Finally, as Marcin Dubel, one of our software engineers, noted in this article – Shiny provides additional features like plotting and expandable rows that are not available in Excel and Google Sheets.
Business Intelligence applications such as Tableau and PowerBI do not provide access to source code. Additionally, Shiny generates virtually no running fees, it provides for full customization and better Machine Learning (ML) possibilities. So whenever you need a data product that is more complex than a spreadsheet or a BI dashboard, and at the same time you’re not entirely sure of the scope and requirements of the project, Shiny is the perfect choice.
Now that we agree that we want to use Shiny…
Imagine you have a Shiny application that you want to build. We will go through the process of creating it together.
The first lesson is a really simple one. Do not overspend on your project and take a more incremental approach. Rather than building a tool with a large number of users in mind, it is advisable to start with a smaller group of individuals who are willing to help and that are actively interested in using the tool. Then, in an iterative process you can get to a point where these initial users are ready to recommend the tool to their peers. Such buy-in does provide the rationale for committing significant resources to the project.
This is crucial because if you build a tool without speaking directly with the stakeholders, your users, you will end up with a large, expensive project that is not likely to deliver anything useful.
Architecture can simplify our lives and the same goes for code. Using modules to separate the logic into smaller, independent parts makes it easier to maintain the code and verify its correctness. A good architecture provides for extracting business logic and enables testing to verify this logic is robust. Furthermore, it allows for extracting heavy computation – this can be done as a future, or even a REST API. Finally, it enables pre-calculating the values before running the app. In addition, it can include an option to load the data from a database.
If you wish to scale your app you should avoid loading the data into the Shiny app from a server or loading a session for each individual user.
Ideally, we want to have the right combination of tests. I have come across engineers, but also business practitioners with a love/hate relationship with tests. They love to have them, but they hate to create them.
It is clear, however, that tests do pay off when done well. A well designed test will start with the largest group conducting very simple unit tests which verify only the basic concepts in the logic. Then, the testing should reduce in number but increase in complexity, until you get to the point of a few end-to-end tests. This minimizes the time required for manual testing.
Unfortunately, this is not yet the standard for Shiny apps. What we see more often is an inverse pattern called the “Test Cone” (below).
The “Test Cone” approach is far from optimal with few or no unit tests and a large number of end-to-end (e2e) tests. End-to-end tests require a lot of time and eventually discourage people from the testing process overall. Fewer tests may result in more bugs going into production. And this is exactly what we want to avoid.
Validating the logic and source code is critical. However, we also need to validate the data itself.
Lesson four is about validating data before running Shiny and it can be done automatically. There is a number of steps to take to prevent data from being the culprit behind a bug. First, designate an owner for each dataset who will take responsibility and react appropriately if any test fails. Second, introduce a logging infrastructure, which gathers information both about good uses and errors in the app. Finally, make the app resilient and fault-tolerant. If the API we use in the app goes down, the app should still be useful to the greatest extent possible. We do not want business users to go through an R stack in their browsers.
Although this is the standard for software engineering in general, this is not true for R and Shiny as of yet.
To further prevent errors, it is advisable to set up a daily data validation email or text with alerts for the data owners. Continuous integration to validate code and style automatically can also bolster the app’s resilience. This can be implemented using lintr. Finally, it sometimes makes sense to show data status directly in the app for the end-users to notify them which data set they are working on. This can be achieved using the open source package shiny.info.
We are hyper-vigilant about data quality, so we methodically identified the different kinds of problems that can happen to the data. Here is a more detailed cheat sheet and explanation about the types of tests out there. If you set up these checks you will avoid many errors in the production environment.
The app is now up and running. It has been tested well. We are ready to commit substantial resources and launch it at scale. Shiny apps are commonly run using multiple processes and quite often using multiple servers. RStudio Connect in combination with Amazon Web Services (AWS) respectively can provide such a complete solution. Once you have the setup, you need to run performance tests.
The below is a real example of a deployment architecture. The load balancer distributes work among applications in RStudio Connect. The master node is responsible for performing the test and gathering statistics to summarize them. To make this even easier, we have an R script that gathers the logs and creates a report for us from the performance tests.
Finally – the last lesson. Deployment and automation. We need at least two deployment environments, so that developers can test on a different server than the one that the users are on. Automated deployment pays off fast. It allows you to operate on cloud resources making it very easy to roll back or to create another instance of an app in several seconds.
Shiny is now much more than just a beautiful user interface. The process of turning a Shiny app into a useful solution ready for production in an enterprise environment requires going through a wide range of sophisticated engineering tasks. However, this article shows that it is possible to turn an R Shiny prototype into a production-ready enterprise solution.
Do you want to build an ambitious enterprise dashboard or need help scaling an existing prototype? Don’t hesitate to reach out to us: [email protected]