Updated on 8 September, 2021
What do Viacom, Ubisoft, and Bank of America have in common? Hint: they are Fortune 500 companies with R Shiny in their tech stack. It’s also no coincidence that they are all using open source packages for Shiny app development created at Appsilon. Data science teams know that developing data analytics solutions in an enterprise environment is a complex process and choosing the right technology is key to project success. R Shiny developers at Appsilon are well aware of those challenges and are constantly improving our Shiny know-how to reflect the current business needs.
One of our core competencies is building decision support systems with R Shiny. So far we have delivered over 100 commercial Shiny dashboards. If you’re just about to make a technology choice for a data product or start an R Shiny project, I’m inviting you to tap into our experience.
This question should be asked for every technology and R Shiny is no different. We believe that Shiny is indeed the ideal choice for many types of projects. By comparing Shiny with other available tools we can determine when to use it.
Shiny has an unmatched speed of development. This speed enabled us to strike our first technology partnership with a Fortune 500 company: we implemented a proof of concept of the requested application within 24 hours. This would have been impossible using any other framework or programming language.
Fast prototyping is especially important for strategic business management. Volatility, uncertainty, complexity, and ambiguity – VUCA – is now the norm in many industries and requires a new approach to decision making. In this context, R Shiny can offer a remedy to the most pain points affecting data science teams focused on innovation.
Compared to spreadsheets, R Shiny gives you the tools to create an attractive user interface and introduce automation in the background. It makes the development work more efficient when dealing with shared resources, reproducibility, and most importantly, the source code.
Switching from a spreadsheet to a Shiny app can give your organization the competitive edge that comes with improved efficiency. Apart from the core functionalities for data manipulation, conditional formatting or pivot tables, Shiny applications offer additional features like plotting and expandable rows. This way, your data solutions are no longer limited in terms of advanced analytics and data quality or versioning and scalability.
You might be wondering how R Shiny compares to the BI tools featured in Gartner’s Magic Quadrant. Business Intelligence applications such as Tableau and PowerBI have their rightful place among top data management tools but it comes at a price. Because they don’t give you access to source code, you are limited to the features included in the selected plan. If full customization and version control are among your priorities, opting for R Shiny can be a strategic move. Plus, Shiny apps generate virtually no running fees and offer better Machine Learning (ML) possibilities.
All in all, a Shiny app is the perfect choice whenever you need a data product that is more complex than a spreadsheet or a BI dashboard. The more so, if you’re not entirely sure of the scope and requirements of the project. In such cases, R Shiny gives you the freedom to experiment and quickly validate your ideas. This, in turn, is crucial for many R&D initiatives. R Shiny can empower your data team to develop cutting-edge solutions that are fit-for-purpose and seamlessly connected to your internal resources.
Now that you’ve done your due diligence for technology choice, imagine a Shiny app that you want to build. I’ll walk you through the process of development and implementation. I’ll be pointing out the project milestones and adding useful tips to help you avoid common mistakes and drawbacks.
The first lesson is simple: avoid overspending by taking a more incremental approach. Instead of building a tool with many users in mind, start with a small group of individuals who are willing to help and are actively interested in using the app. Stay agile and make the development process iterative. This way, you will get to a point when these initial users become ambassadors of your Shiny app and recommend the tool to their peers. Such an organic buy-in gives a better rationale for committing more resources to the project.
Prototyping and PoCs are crucial because if you build a tool without speaking directly with the stakeholders – the app users – you might end up with a large, expensive project that doesn’t fulfill its purpose. What’s more, early feedback will give you valuable insight into the key user flows to optimize and help you fix the UI issues that affect the user experience (UX).
Modular architecture can make life simpler and the same goes for code. Using modules to break the app logic into smaller, independent parts makes the code easier to maintain and verify. With Shiny modules, you should be able to extract business logic and test it for robustness.
Modular architecture can also improve the app performance. It allows for extracting heavy computations and enables pre-calculating the values before running the app. You can achieve that by using the future R package or even a REST API.
At Appsilon, we have pioneered the space of supporting large numbers of concurrent users with Shiny. We were the first company to demonstrate that you can have over 700 users for a single Shiny dashboard. The previous conventional wisdom was that a Shiny dashboard could host only several users at a time.
The right app architecture can help you prepare for scaling. If you wish to scale your Shiny app, you should avoid loading the data into the Shiny app from a server. Instead of loading a session for each individual user, make sure to load the data from a database.
Quality Assurance is a big topic in software development and a separate domain of expertise. Still, I’ve met both engineers and business practitioners with a love/hate relationship with tests. They love to have them, but they hate to create them. Sympathies aside, the right combination of tests offers a big pay off in the long run by reducing downtime and resources needed to fix bugs and respond to alerts.
A well designed testing procedure starts with many simple automated unit tests to verify the basic concepts in the app logic. Then, the testing should reduce in number but increase in complexity, until you get to the point of a few end-to-end tests. This way, you can limit the time required for manual testing.
Unfortunately, this is not yet the standard for Shiny apps. What we often see is an inverse pattern called the “Test Cone” (below).
The “Test Cone” approach is far from optimal, with few or no unit tests and a large number of end-to-end (e2e) tests. End-to-end tests take a lot of time and can eventually discourage people from the testing process. Fewer tests may result in more bugs going into production, which is exactly what you want to avoid.
Validating the logic and the source code is critical, but you also need to validate the data before running the Shiny app. Data quality issues are actually very common and tend to build up due to data dependencies. Even small errors can have a big impact on business continuity or costs of upkeep.
The good news is data validation can be automated. The data.validator open source package can save you a lot of time and effort, while ensuring no corrupt or incomplete data affects the quality of your analysis.
To make the most of your data validation process, try applying best engineering practices:
At Appsilon, we are hyper-vigilant about data quality, so we methodically identified the different kinds of problems that can happen to the data. To learn more, read about our data quality case studies and the ways to validate data on different levels. Make sure to set up these checks to avoid many errors in the production environment.
To prevent further errors, it’s best to set up a daily data validation email or text with alerts for the data owners. Make your Shiny app more resilient by using Continuous Integration to validate code and style automatically. This can be implemented using lintr: an R package for static code analysis. Finally, it sometimes makes sense to show data status directly in the app, so that end users know exactly which data set they are working on. This can be achieved using the open source package shiny.info.
The app has been tested and is now up and running. You should be ready to commit substantial resources and launch it at scale. Shiny apps are commonly run using multiple processes and quite often using multiple servers. RStudio Connect in combination with Amazon Web Services (AWS) can provide such a complete solution. Once you have the setup, you need to run performance tests.
The image above is a real-life example of a deployment architecture. The load balancer distributes work among applications in RStudio Connect. The master node is responsible for performing the test and gathering statistics to summarize them. To make this even easier, use an R script that gathers the logs and creates a report from the performance tests.
Although deployment and automation appear in this compilation at the end, they shouldn’t come as an afterthought. As part of an optimized implementation, you should have at least two deployment environments. This way developers can test on a different server than the one that the users are on. Implementing changes and testing solutions without the risk of affecting the users empowers developers to get creative and bulletproof the app for different scenarios.
Automated deployment helps streamline workflows and pays off fast. It allows you to operate on cloud resources, making it very easy to roll back or to create another instance of an app in several seconds. It’s a great way to increase productivity and receive instant feedback, which translates to better quality of the final release.
R Shiny is now much more than just a means to deliver beautiful user interfaces. The interactive nature of Shiny apps makes them a powerful data visualization tool that’s instrumental in decision-making. The process of turning a Shiny app into a useful solution ready for production in an enterprise environment calls for a wide range of sophisticated engineering tasks. I hope these 6 lessons for Shiny app development will help you turn an R Shiny prototype into a production-ready enterprise solution.
Do you want to build an ambitious enterprise dashboard or need help scaling an existing prototype? Don’t hesitate to reach out to us: [email protected]