ShinyProxy vs Posit Connect: Benchmark Test for Scaling Shiny Apps
Shiny is a web framework for R (and now Python) users. With it, you can build a working dashboard for your analytics in a relatively short time. While it is easy to start developing dashboards without programming experience, the challenge comes with scaling. The quality and interactive method of data delivery will mean greater adoption and more users.
There are many solutions to optimize dashboard performance for a single user. This includes examples like promises for non-blocking access and profvis to identify bottlenecks. However, to scale a Shiny application for a large number of users, the deployment environment plays a significant role.
The two main scalable hosting solutions are Posit Connect (formerly RStudio Connect) and ShinyProxy. The scope of this article is to touch on the two most common scaling solutions for enterprises.
Table of Contents:
- ShinyProxy benefits and costs
- Posit Connect benefits and costs
- Benchmark Test – ShinyProxy vs Connect
- Memory used
- Memory + SWAP used
- CPU usage
- Test summary
- How to migrate from ShinyProxy to Posit Connect
Addendum – Retest for ShinyProxy proxy.default-max-instances
In our initial research, we used a method that we want to expand on. This article may be updated with results from additional experiments. In ShinyProxy version 2.6.0, users are able to run multiple instances of an app. In our initial test, we did not permit ShinyProxy to run with additional containers, potentially inhibiting performance and creating unbalanced outcomes.
ShinyProxy vs Posit Connect – Which solution should I choose?
What is Posit Connect
Posit Connect is a standalone publishing platform for conveniently sharing Shiny applications and other content such as R Markdown documents, Plumber APIs, Python Jupyter notebooks, Quarto documents and projects, or any static R plot or graph made within your organization.
On Posit Connect, you can have multiple R processes per app. This means that many concurrent users can be distributed between separate processes and are served more efficiently. As there is no limitation on the number of processes, you can make use of all your machine resources.
What is ShinyProxy
ShinyProxy architecture is based on docker containers, which isolate the app’s environment. Every user of ShinyProxy will use a private Docker container when running a Shiny application. This can have a significant effect on resources. We’ll cover this in the experiment below and explain the procedure for you to replicate. In addition, the dependency on Docker images means to deploy your Shiny apps, you will need to build your own Docker image for the app. Posit Connect provides push-button deployments from RStudio and the command line without the need for Docker knowledge.
Another key difference between ShinyProxy and Posit Connect is the price. ShinyProxy is free and open-source while Posit Connect is a commercial enterprise solution with a license fee.
[block quote for Posit shinyapps.io and shinyserver]
To make the right choice for the deployment environment that will satisfy your project needs let’s start by comparing the pros and cons of the two solutions.
- Open source – no subscription fee
- A wide range of Authorization options available (LDAP / Kerberos / SSO / SAML / Open ID / Keycloak / Social Media / Simple (flat file of users & passwords))
- Higher implementation cost
- No product support, therefore there is no possibility of SLA guarantees
- The additional cost of post-implementation support
- The additional layer of complexity – Docker/Kubernetes
- Long-term maintenance requiring a large workforce
Posit Connect benefits
- Stability with Software licensing and Standard Software Support from Posit included
- Lower implementation costs
- Easier configuration and deployment of dashboard versions
- Authorization is included in the Posit Connect Subscription, with a wide range of options to choose from: (LDAP and Active Directory / SAML / OAuth 2.0 using Google Apps accounts / PAM / Proxied Authentication / SLA)
- Easier maintenance in production
- Option to add Posit Workbench and Package Manager
- Automatic scaling – no need to manually trigger new processes
- An industry standard by Fortune 500 companies working with R
- Admin panel to manage users and monitor the logs and machine
- Ability to deploy R/Python applications, R/Python API, RMarkdown reports that regenerate automatically, etc.
Posit Connect cost
- A yearly Posit Connect subscription starts at $14,995/year (USD)
In short, ShinyProxy is free, and Posit Connect is expensive – at the start. However, the costs of technology, workforce and maintenance may offset any initial savings using ShinyProxy.
Working in distributed teams can be a challenge. Discover the benefits of Posit Connect for distributed teams.
In terms of stability and accountability, ShinyProxy’s open-source implementation comes as a double-edged sword. You might not be getting any official product support but since it is open source, investigating and fixing an issue is supported by your team’s capabilities.
The question then becomes how scalable and robust the applications are when hosted on either solution.
ShinyProxy vs Connect Benchmark Comparison
To find the answer to that question, let’s perform a benchmark test to compare the scalability and performance of both products. To simulate a simple Shiny application with minimal dependencies the
01_hello dependencies example Shiny application
(shiny::runExample("01_hello")) will be used.
To analyze the results of the experiment we consider the following three evaluation measures.
- Memory used
- Memory + Swap used
- CPU Usage
Both products were tested on two t3.medium EC2 AWS instances each with 2 vCPU, 4 GB RAM with R-base 4.1.3 in an Ubuntu 20.04 LTS operating system used for the experiment.
Both installations of ShinyProxy and Posit Connect were performed based on their respective documentation.
The ShinyProxy installation did not have any additional load balancers installed. The default
openanalytics/shinyproxy-demo was used with a few changes.
- Authentication was set to “none”
- The 06_tabsets application was disabled
The Posit Connect instance was deployed on the VM by using the web-based UI (Setup Assistance) For the Connect app configuration, other than max processes which were increased from the default of 3 to 6 to accommodate 64 concurrent users, all other settings were left at their default values. Connect was able to handle 32 concurrent users with 3 max processes.
Monitoring tools setup The
sar tool contained in the
sysstat performance monitoring package in Linux was used to log the system statistics. Specifically, the command sar -r -u 5 was used to capture both the CPU usage and memory usage every 5 seconds.
ShinyProxy version used: 2.6.1.
Load testing procedure
At the beginning of the test procedure, the
sar command would start logging the performance for around 60 seconds to get a baseline reading of the number of resources needed when the servers have not started. Afterward, the installed servers were restarted to get a fresh start.
Assuming that the servers would be at an idle state after 60 seconds, we would wait for 60 seconds before starting the load testing procedure. While the apps are settling the
sar command will be logging performance measurements behind the scenes.
Before the start of the test, to perform a load test on the same application on both products, the
>shinyloadtestpackage was used. For the load test using the
shinyloadtestpackage, a recording was made of user interactions to be simulated by a large number of concurrent users. The recording consisted of a user loading the app and clicking on four values of 11, 41, 1, and 50 consecutively.
Once the recording had been captured, the
shinycannon command was used to run a load test with the required number of workers and maintains a 1-minute load when all workers are running. For example to run a load test with 8 workers the command would be as follows,
shinycannon recording.log $APP_URL \ --workers 8 \ --loaded-duration-minutes 1
For ShinyProxy the number of workers would take the value of 1, 2, 4, and 8 while the number of workers for Connect would go beyond 8 to 16, 32, and 64. The reasoning for this choice is that the Shinyproxy solution crashed the server with 12 concurrent users and therefore increasing the number of users even further seemed unnecessary.
After each load test, the servers would be stopped and the RAM cleared to give a fresh slate.
If you want to run the experiment for yourself refer to the Github Repository.
ShinyProxy vs Connect benchmark results
As mentioned earlier in the experimental setup, the ShinyProxy solution crashed the server with 12 concurrent users while the Posit Connect solution was able to handle 64 concurrent users with spare resources for more.
Since Shinyproxy launches one instance of a Docker image for each user, the memory usage of ShinyProxy grows linearly with each increment of concurrent users. Posit Connect also tends to follow a pattern of stabilizing memory usage followed by an increase in memory usage, which would indicate instances where a new worker process was spawned to cater to the increase in the number of users.
The percent of used memory reaches an all-time high for ShinyProxy at the mark of nearly 16 users where nearly 80% of the entire server memory was used by ShinyProxy to serve the example ShinyApplication.
Memory + Swap used
Similar to the maximum memory usage, the maximum memory and swap usage shows that ShinyProxy tends to use a larger amount of memory and swap compared to Posit Connect.
The overhead of Docker containers also consumes more CPU resources for ShinyProxy which can be seen from the initial CPU load spike (86.91%) when the servers are started (at the Start vertical line colored in blue). In terms of CPU usage in Posit Connect, even at 64 concurrent users, Posit Connect requires only 62% of CPU usage.
Summary of ShinyProxy vs Connect for Scaling Shiny Apps
With the servers at idle, ShinyProxy already consumes 3x the memory as Connect. At 8 concurrent users, ShinyProxy consumes 84% of memory as opposed to 13% for Connect. Eventually, ShinyProxy crashes the server after reaching a limit of 12 concurrent users while Connect can handle 64 concurrent users consuming just 30% of memory. ShinyProxy also consumes more than twice the CPU resources as Connect at the same load.
Are your Shiny apps slow? Speed things up with this comprehensive guide for faster Shiny apps.
In short, while ShinyProxy provides a free and open-source solution, making a Shiny application scale for multiple users will require extra effort in fine-tuning the implementation. Additionally, it requires making space for larger compute power, which in turn will cost more in the long run. Comparatively, the scalability and efficiency of Posit Connect make it a good investment for sustainable implementation for projects requiring comparable scaling.
How to migrate to Posit Connect from ShinyProxy?
If Posit Connect is appropriate for your needs, then the next step is to consider the migration process from ShinyProxy to Connect. Since the applications built in ShinyProxy are already containerized, it will be easy to deploy them straight away to your newly purchased Posit Connect instance. All you need to do is make use of the readily available API and add some additional configurations to your application.
First impressions matter. Learn how to build a custom login page for Posit Connect.
Preparing the migration
The first step is to ensure that your project’s root directory has the prerequisite structure and files needed for Posit Connect. These include:
manifest.json --JSON file describing the requirements of this Shiny application. Create with
deploy --Directory containing deployment scripts for the content. Obtain from the Connect repository.
docker --Directory defining a Docker image that has your original ShinyProxy docker configuration (i.e. Dockerfile)
The JSON file will need to be created first, and then the Dockerfile from your original ShinyProxy deployment.
Note that your Dockerfile might need to be adjusted accordingly, depending on how it was originally configured.
As mentioned above, the deploy directory contains scripts that will help with your migration. Before utilizing these scripts, there are two important environment variables to set:
CONNECT_SERVER --This environment variable indicates the target Posit Connect server. It must be the base URL of your instance and end with a trailing slash.
CONNECT_API_KEY --This environment variable indicates an API key owned by the target “publisher” account in the CONNECT_SERVER Posit Connect server.
Both of these need to be set as follows in your current session:
Once done, you may make use of the deployment scripts:
create-content.sh --creates a new content item in your target Posit Connect server
upload-and-deploy.sh --bundles your code into a .tar.gz archive, uploads that file to Posit Connect, and requests that archive be “deployed”
create-upload-deploy.sh -- combines both of the scripts above into a single command
Now that everything is in place, you can migrate your existing ShinyProxy by building your docker image, and then running the deployment scripts.
With your Dockerfile in the docker directory, you can build your image by simply running:
docker build -t posit-connect-migration:latest docker
posit-connect-migration:latest tag here is used to easily identify your image for the following steps.
To simplify things, use the
create-upload-deploy.sh script to complete your deployment:
docker run --rm \ -e CONNECT_SERVER="http://connect.company.com/" \ -e CONNECT_API_KEY="jIsDWwtuWWsRAwu0XoYpbyok2rlXfRWa" \ -v $(pwd):/content \ -w /content \ posit-connect-migration:latest \ /content/deploy/create-upload-deploy.sh "Migration from ShinyProxy to Posit Connect"
Please note that using
create-upload-deploy.sh will create a new content item and deploy in one go.
If you want to keep updating an already created content item in your target Posit Connect server, you will need to instantiate the item using
create-content.sh just once, and then use
upload-and-deploy.sh to update that item using its specific ID.
For more information, you may refer to the official guide.
Are your Shiny apps secure? Learn why you should be using Posit Connect Authentication and how to set it up.