shiny.benchmark - How to Measure Performance Improvements in R Shiny Apps

The <a href="https://appsilon.github.io/shiny.benchmark/" target="_blank" rel="noopener">shiny.benchmark package</a> by Appsilon allows you to compare the performance of different versions of R Shiny applications. You can write instructions that will be executed and evaluated on different versions of your app and easily measure the performance differences. In today's article, you'll learn how to install and use the <code>shiny.benchmark</code> package by exploring and tweaking an example Shiny application. So without much ado, let's dive straight in. <blockquote>Looking to implement infinite scrolling in R Shiny? <a href="https://appsilon.com/infinite-scrolling-in-r-shiny/" target="_blank" rel="noopener">Our Infinite Scroll guide has you covered</a>.</blockquote> Table of contents: <ul><li><a href="#install">How to install shiny.benchmark</a></li><li><a href="#use">Benchmarks in action: How to use shiny.benchmark</a></li><li><a href="#summary">Summing up the shiny.benchmark R package</a></li></ul> <hr /> <h2 id="install">How to install shiny.benchmark</h2> Before installing the package, we need to talk prerequisites. The <code>shiny.benchmark</code> package can use two different engines to evaluate app performance: <a href="https://rstudio.github.io/shinytest2/" target="_blank" rel="nofollow noopener">shinytest2</a> and <a href="https://www.cypress.io/" target="_blank" rel="nofollow noopener">Cypress</a>. It doesn't matter which one you opt for, but keep in mind that Cypress requires <a href="https://nodejs.org/en/download/" target="_blank" rel="nofollow noopener">Node.js</a> and <a href="https://yarnpkg.com/getting-started/install" target="_blank" rel="nofollow noopener">yarn</a>, so make sure to have them installed. If you're on Linux, Cypress might also require some <a href="https://docs.cypress.io/guides/getting-started/installing-cypress#Linux-Prerequisites" target="_blank" rel="nofollow noopener">additional dependencies</a>, so check the documentation link if you're running into errors. FYI - <a href="https://appsilon.github.io/rhino/" target="_blank" rel="noopener">Rhino</a> now supports both Cypress and shinytest2. <blockquote><strong>Update: <a href="https://cran.r-project.org/package=shiny.benchmark" target="_blank" rel="noopener"><code>shiny.benchmark</code> is now available on CRAN</a>.</strong></blockquote> Assuming you have all the prerequisites out of the way, it's time to install <code>shiny.benchmark</code>. Run the following command to install the latest version from CRAN: <pre><code>install.packages("shiny.benchmark") </code></pre> And that's it! Next, let's see how to use Shiny Benchmark. <h2 id="use">Benchmarks in action: How to use shiny.benchmark</h2> The simplest way to see how <code>shiny.benchmark</code> works is to use the sample app it ships with. The <code>load_example()</code> function will download the sample R Shiny application to the path you provide: <pre><code class="language-r">library(shiny.benchmark) <br>load_example(path = "path/to/your/folder")</code></pre> Here's what the app looks like when loaded (and after each button gets pressed): <img class="size-full wp-image-17245" src="https://webflow-prod-assets.s3.amazonaws.com/6525256482c9e9a06c7a9d3c%2F65b7d2c346c250f02d994923_dd026af0_1.webp" alt="Image 1 - Sample R Shiny benchmark app" width="1182" height="1006" /> Image 1 - Sample R Shiny benchmark app The heart of this application lies in the <code>server.R</code> file. It instructs the app to sleep for some time after each button gets clicked. The default values are 1, 0.5, and 0.1 seconds for the first, second, and third buttons, respectively. We'll tweak this file later, but first, need to make different versions of the app. Navigate to the <code>app</code> folder and initialize a new Git repository: <pre><code class="language-git">cd app git init</code></pre> <img class="size-full wp-image-17247" src="https://webflow-prod-assets.s3.amazonaws.com/6525256482c9e9a06c7a9d3c%2F65b7d2c498512f0e5202ac64_5cb62e0a_2.webp" alt="Image 2 - Intitializing a new Git repo" width="1609" height="994" /> Image 2 - Initializing a new Git repo While here, let's create a <code>.gitignore</code> file which will contain all files we don't want to track: <pre><code class="language-git">echo .Rproj.user &gt;&gt; .gitignore echo *.Rproj &gt;&gt; .gitignore echo .Rprofile &gt;&gt; .gitignore echo renv &gt;&gt; .gitignore echo .Rprofile &gt;&gt; .gitignore</code></pre> <img class="size-full wp-image-17249" src="https://webflow-prod-assets.s3.amazonaws.com/6525256482c9e9a06c7a9d3c%2F65b7d2c40d37f908a0b28ee6_03dad932_3.webp" alt="Image 3 - Creating a .gitignore file" width="1609" height="994" /> Image 3 - Creating a .gitignore file <h3>App Version #1 - The Default App</h3> And now, let's finally create a couple of versions of our app. The first one (<b>main branch</b>) will contain the app in its stock format - no modifications made to <code>server.R</code> file. Add all files to the staging area and commit with the following command: <pre><code class="language-git">git add . git commit -m "Initial commit" </code></pre> <img class="size-full wp-image-17251" src="https://webflow-prod-assets.s3.amazonaws.com/6525256482c9e9a06c7a9d3c%2F65b7d2c5bc9849746c73a130_a8b5004a_4.webp" alt="Image 4 - Commit to the main branch" width="1609" height="994" /> Image 4 - Commit to the main branch <h3>App Version #2 - Longer Sleep Times</h3> The second version of the app will be identical UI-wise but will include longer sleep times. You only need to modify the <code>server.R</code> file and change the values in three calls to <code>Sys.sleep()</code>. Here's an example of a modified <code>server.R</code> file: <img class="size-full wp-image-17253" src="https://webflow-prod-assets.s3.amazonaws.com/6525256482c9e9a06c7a9d3c%2F65b7d2c6ff233005820f1e83_321b19e2_5.webp" alt="Image 5 - Modified server file with longer sleep times" width="2042" height="1699" /> Image 5 - Modified server file with longer sleep times All that's left to do is to add this app version to a different Git branch. We've named ours <code>develop</code>, but the naming is completely arbitrary: <pre><code class="language-git">git checkout -b develop git add . git commit -m "Longer sleep times"</code></pre> <h3>App Version #3 - Shorter Sleep Times</h3> Now we'll make the complete opposite of the second version. In this one, the sleep times will be much shorter, as displayed in the image below: <img class="size-full wp-image-17255" src="https://webflow-prod-assets.s3.amazonaws.com/6525256482c9e9a06c7a9d3c%2F65b7d2c60d6008396df21656_b3e800f3_6.webp" alt="Image 6 - Modified server file with shorter sleep times" width="2042" height="1699" /> Image 6 - Modified server file with shorter sleep times We'll also commit this version to a dedicated branch - <code>develop2</code>: <pre><code class="language-git">git checkout -b develop2 git add . git commit -m "Shorter sleep times"</code></pre> Up next, let's see how to write tests for <code>shiny.benchmark</code>. <h3>How to Write Tests for shiny.benchmark</h3> Where you'll put the tests depends on the testing engine you'll use. We'll work with <code>shinytest2</code> in this article, which means the tests are located in <code>app/tests/testthat</code> folder. Here's what one test file contains: <img class="size-full wp-image-17257" src="https://webflow-prod-assets.s3.amazonaws.com/6525256482c9e9a06c7a9d3c%2F65b7d2c78e1509632fdbc30a_7a21f7c8_7.webp" alt="Image 7 - Contents of a test file" width="2042" height="1850" /> Image 7 - Contents of a test file In a nutshell, it instructs how to interact with the app elements and what to expect from them. You can construct test cases for your Shiny apps in the same way, just make sure to correctly write the input and output IDs. That's all we need to run the benchmarks, so let's do that next. <h3>How to Run Benchmarks with shiny.benchmark</h3> By now, we have three versions of our app committed to dedicated Git branches, and now it's time to run the benchmarks. Import the R package and declare a list of branch names as shown in the snippet: <pre><code class="language-r">library(shiny.benchmark) <br>commit_list &lt;- c("main", "develop", "develop2")</code></pre> Now onto the benchmark. The <code>benchmark()</code> function does all the work for you. It can accept many parameters, but we'll only use a handful: <ul><li><code>commit_list</code> - A list of branches on which the benchmark will run</li><li><code>shinytest2_dir</code> - A directory in which the actual tests are located. We're using <code>shinytest2</code> instead of Cypress</li><li><code>use_renv</code> - Whether or not you're using <code>renv</code> to manage environments. It's set to <code>TRUE</code> by default</li><li><code>n_rep</code> - The number of times the tests will be replicated. It's an optional parameter, but a good one to include for more accurate benchmark metrics</li></ul> You can check which <a href="https://appsilon.github.io/shiny.benchmark/" target="_blank" rel="noopener">additional parameters </a>are available, but we'll stick only to these four today: <pre><code class="language-r">&gt; out &lt;- benchmark(  commit_list = commit_list,  shinytest2_dir = "tests",  use_renv = FALSE,  n_rep = 10 )</code></pre> Here's what you'll see during the benchmark phase (for each branch): <img class="size-full wp-image-17259" src="https://webflow-prod-assets.s3.amazonaws.com/6525256482c9e9a06c7a9d3c%2F65b7d2c706919c51f2be2c0b_abdbcf68_8.webp" alt="Image 8 - Benchmark running output" width="897" height="163" /> Image 8 - Benchmark running output Let's see what the <code>out</code> variable contains after the benchmark runs. <h3>Evaluating Benchmark Results</h3> Run the following R code to print the contents of the <code>out</code> variable: <pre><code class="language-r">summary(out)</code></pre> <img class="size-full wp-image-17261" src="https://webflow-prod-assets.s3.amazonaws.com/6525256482c9e9a06c7a9d3c%2F65b7d2c85d7e9108a083898c_3ee2c9f2_9.webp" alt="Image 9 - Summary of the out variable" width="654" height="573" /> Image 9 - Summary of the out variable Basically, it shows you the performance differences between distinct app versions (branches) and various statistics for each run, such as mean, median, standard deviation, min and max. Looking at these values in a table format isn't the best method for drawing conclusions and insights. If you're more of a visual type, use the <code>plot()</code> function to inspect the differences visually: <pre><code class="language-r">plot(out)</code></pre> <img class="size-full wp-image-17263" src="https://webflow-prod-assets.s3.amazonaws.com/6525256482c9e9a06c7a9d3c%2F65b7d2c9c9f020529c865a71_530c1c15_10.webp" alt="Image 10 - Plot of the out variable" width="1641" height="1298" /> Image 10 - Plot of the out variable The app version on <code>develop2</code> branch seems to have the lowest runtime values which is to be expected, as we manually reduced the function sleep times. You now know how <code>shiny.benchmark</code> works, so let's make a short summary next. <hr /> <h2 id="summary">Summing up the shiny.benchmark R package for measuring app performance</h2> Testing different versions of your R Shiny apps is easier than ever. The <code>shiny.benchmark</code> package is convenient to use because a typical Shiny workflow will include multiple Git branches and unit tests, so running benchmarks won't take a huge toll on your development time. We value measuring app performance as crucial when optimizing your code and introducing new features. You want to make sure there are no bottlenecks in your applications and optimize accordingly. <i>What do you think of the <code>shiny.benchmark</code> package? Have you already used it to compare the performance of your Shiny apps?</i> Please let us know in the comment section below, or reach out to us on Twitter - <a href="http://twitter.com/appsilon" target="_blank" rel="nofollow noopener">@appsilon</a> - we'd love to hear your input. <blockquote>Can you generate MS Word documents from R Shiny tables? <a href="https://appsilon.com/generating-word-documents-from-table-data-in-shiny/" target="_blank" rel="noopener">Yes, you can, and here's how</a>.</blockquote>