How to build a CI/CD pipeline for Shiny apps with Gitlab-CI and RStudio (Posit) Connect

<p class="md-end-block md-p md-focus"><span class="md-plain">RStudio Connect makes publishing Shiny Apps <strong>incredibly simple </strong>with only a few easy-to-follow steps. You can find step-by-step instructions in RStudio's <a href="https://docs.rstudio.com/how-to-guides/users/basic/publish-shinyapp/" target="_blank" rel="noopener noreferrer">guide for publishing a Shiny app</a>. Despite that ease of use,</span><span class="md-plain md-expand"> at some point, you might want to <strong>automate</strong> this process by implementing a <strong>CI/CD pipeline</strong>. Building a CI/CD Pipeline for your Shiny apps takes a little effort up front, but it's easy to achieve with Gitlab-CI and RStudio Connect. The ROI is well worth it for your Shiny app production.</span></p> <p class="md-end-block md-p md-focus"><span class="md-plain md-expand">A continuous integration pipeline is an ideal solution for projects with multiple developers and when deploying the app to multiple environments. The CI/CD pipeline will enable features in your project like running unit tests on every PR and automatically deploying the app after changes are merged to the proper branch, e.g. main or dev. Having continuous integration means frequent merges to shared branches with automated validation. This automated validator makes it easier to find and fix bugs quickly. The best and easiest way to implement such a pipeline is to use the built-in tools inside the platform where you store your code, e.g GitHub Actions or GitLab-CI.</span></p> <blockquote><strong>Speed up production with Appsilon's open source <a href="https://appsilon.com/shiny-templates-available/" target="_blank" rel="noopener noreferrer">Shiny Dashboard Templates, available now!</a></strong></blockquote> <p class="md-end-block md-p md-focus"><span class="md-plain md-expand">In this article, I will describe how to create such a pipeline in the GitLab-CI platform. </span></p> <ul><li><a href="#anchor-1" target="_blank" rel="noopener noreferrer">Requirements</a></li><li><a href="#anchor-2" target="_blank" rel="noopener noreferrer">Continuous integration with Gitlab CI</a></li><li><a href="#anchor-3" target="_blank" rel="noopener noreferrer">Continuous deployment using Git-backed content</a></li><li><a href="#anchor-4" target="_blank" rel="noopener noreferrer">Benefits of building a CI/CD pipeline</a></li></ul> Note: At the time of writing this article, Posit PBC was RStudio PBC. We use RStudio and Posit interchangeably in this text (e.g. RStudio Connect == Posit Connect). <h2 id="anchor-1">Requirements to build a CI/CD pipeline for Shiny apps</h2> Before we can enable the CI/CD pipeline we have to prepare our repository with a Shiny app for automatic testing and deployments. First, initialize <em>renv.lock</em> by using renv package to restore the packages inside your CI/CD job. The package will be used to restore packages inside your CI/CD job. Next, generate <em>manifest.json</em> and push it to your repository. The manifest.json file tells RStudio Connect how to deploy and host your content. Lastly, ensure that you have <a href="https://docs.gitlab.com/ee/ci/quick_start/#ensure-you-have-runners-available" target="_blank" rel="noopener noreferrer">GitLab runner</a> available and enabled for your project and verify that there is network connectivity between RStudio Connect and GitLab-CI. <h3>Initializing renv.lock</h3> To familiarize yourself with the <em>renv</em> package, check out RStudio's comprehensive <a href="https://rstudio.github.io/renv/articles/renv.html" target="_blank" rel="noopener noreferrer">renv documentation</a>. But for now, all you have to do is to call <code>renv::init()</code> to initialize a new project-local environment with a private R library. <h3>Generating manifest.json</h3> The manifest can be created by calling the R function <code>rsconnect::writeManifest()</code> from within your project directory. Pushing it and storing it in the repo will allow you to review the manifests before they are deployed. This additional manual step will allow you to fully control which packages, and from which sources, will be installed on RStudio Connect. <h3>Repository file structure</h3> What you want to achieve is the file structure shown in the image below. After initializing <em>renv.lock</em> and generating <em>manifest.json</em> file, one last thing to do is to create <em>gitlab-ci.yml</em> file. It will include the definition of the CI pipeline we want to implement. I intentionally said CI instead of CI/CD because in our case CD part will be configured outside of GitLab-CI platform. <img class="aligncenter wp-image-7931" src="https://webflow-prod-assets.s3.amazonaws.com/6525256482c9e9a06c7a9d3c%2F65b020114d7961f826709e9b_Screenshot-2021-08-26-at-12.02.51-2.webp" alt="repository file structure" width="646" height="411" /> <h2>Continuous integration with GitLab-CI</h2> In general, you should run tests with every commit to ensure that new code is properly tested. This will help prevent new errors from being introduced. To do this we first need to create a CI/CD pipeline definition and name it properly - <em>.gitlab-ci.yml.</em> Note: the name must be <em>.gitlab-ci.yml </em>when using the GitLab CI platform. <img class="aligncenter wp-image-7949" src="https://webflow-prod-assets.s3.amazonaws.com/6525256482c9e9a06c7a9d3c%2F65b0201204abfeee1f061eae_Screenshot-2021-09-09-at-14.38.54.webp" alt="adding tests on Gitlab-ci" width="811" height="524" /> <h3>Running tests inside the CI/CD pipeline</h3> Before we run our tests, our environment needs to meet requirements. Inside the environment, you should be able to run an R job and restore necessary packages before running a <em>testthat</em> command. It's very easy to achieve, you only have to pick a base docker image that has R installed and then execute <code>renv::restore()</code> command before running the main script. <script src="https://gist.github.com/MicahAppsilon/1cefc6a032f857df8e6b81c874b921ac.js"></script> <pre>image: rstudio/r-base:4.1.0-bionic before_script:    - R -e 'renv::restore(prompt=FALSE)' </pre> <blockquote><strong>High quality, reproducible code takes skill. Learn to <a href="https://appsilon.com/how-to-write-production-ready-r-code/" target="_blank" rel="noopener noreferrer">write production-ready R code with Marcin Dubel</a></strong></blockquote> Now the core part, running tests. I will use <code class="c-mrkdwn__code" data-stringify-type="code">testthat::test_dir</code> command with a <i data-stringify-type="italic">reporter</i> parameter to save the result of the test job as a file with the proper format. <script src="https://gist.github.com/MicahAppsilon/7377d431ad7ba967f1a0fe2b58e43d33.js"></script> <pre>script:  - R -e 'testthat::test_dir("tests/testthat", reporter = testthat::JunitReporter$new(file = "../../junit_result.xml"))' </pre> <h3>Displaying test results in GitLab-CI platform</h3> This code will run all tests and save the results as <em>junit_result.xml</em> file by using <code>testthat::JunitReporter$new</code> function so that results can be displayed in the GitLab-CI platform in a user-friendly format by invoking the Unit test reports GitLab feature. <img class="aligncenter wp-image-7944" src="https://webflow-prod-assets.s3.amazonaws.com/6525256482c9e9a06c7a9d3c%2F65b2672266aa8bda9c3b9c57_2.webp" alt="gitlab-ci test results" width="935" height="351" /> To achieve such results you will have to add a code block responsible for publishing reports in your pipeline: <script src="https://gist.github.com/MicahAppsilon/7063c15b0916c5ec31b37e54a6412e6a.js"></script> <pre>artifacts:    reports:        junit: junit_result.xml </pre> To read more about the above configuration please refer to official GitLab <a href="https://docs.gitlab.com/ee/ci/unit_test_reports.html" target="_blank" rel="noopener noreferrer">docs for unit test reports</a>. <h3>Final CI/CD Pipeline for Shiny Apps</h3> <script src="https://gist.github.com/MicahAppsilon/570aa34f0d021c17355d78429c45d6b0.js"></script> <pre>stages:  - tests variables:  RENV_PATHS_CACHE: ${CI_PROJECT_DIR}/cache  RENV_PATHS_LIBRARY: ${CI_PROJECT_DIR}/renv/library cache:  key: ${CI_COMMIT_REF_SLU}  paths:    - ${RENV_PATHS_CACHE}    - ${RENV_PATHS_LIBRARY} tests:  stage: tests  image: rstudio/r-base:4.1.0-bionic  before_script:    - apt-get update &amp;&amp; apt-get install libxml2-dev    - R -e 'renv::restore(prompt=FALSE)'  script:    - R -e 'testthat::test_dir("tests/testthat", reporter = testthat::JunitReporter$new(file = "../../junit_result.xml"))'  artifacts:    when: always    reports:      junit: junit_result.xml </pre> <h2>Continuous deployment using Git-backed content</h2> You can deploy content directly from a remote Git repository using the <em>Git-backed content</em> feature available in RStudio Connect. This means that content will automatically fetch from the associated remote Git repository and re-deployed with changes. This part will be configured inside RStudio Connect and is very straightforward. You can check out how to configure this by watching the gif below. Additionally, you can go to official <a href="https://docs.rstudio.com/connect/admin/content-management/git-backed/" target="_blank" rel="noopener noreferrer">RStudio documentation</a> to read more and use their prepared examples from <a href="https://github.com/rstudio/connect-examples" target="_blank" rel="noopener noreferrer">GitHub</a> to test it yourself. <iframe title="YouTube video player" src="https://www.youtube.com/embed/_PjR7izDaZk" width="560" height="315" frameborder="0" allowfullscreen="allowfullscreen"></iframe> There is no need to change the RStudio Connect configuration file unless you want to pull code from the private repository. In this case, you will have to specify git credentials for RStudio Connect to use. You can read more about this from the official <a href="https://docs.rstudio.com/connect/admin/content-management/git-backed/#private-repos" target="_blank" rel="noopener noreferrer">RStudio documentation on private repos</a>. <blockquote><strong>Does your Shiny app need a check-up? Discover <a href="https://appsilon.com/how-to-pull-shiny-usage-data-from-rstudio-connect-api-setup-guide/" target="_blank" rel="noopener noreferrer">how to pull Shiny usage data from RStudio Connect</a></strong></blockquote> <h2>Benefits of building a CI/CD pipeline for Shiny apps</h2> ​Setting up CI/CD for your shiny apps with RStudio Connect is simple and straightforward. A CI/CD pipeline can save you the headache of sending a good build to your production environment and ensure its readiness for production release. Consider setting up such a pipeline in your projects and introduce best practices for testing, reviewing, and deploying! Reduce your costs, minimize errors, and free up your developer resources so your team can focus on more important matters. It's a small investment that will help you develop better Shiny applications and save you a lot of time wasted for manual deployments in the future. <h3>Another way forward</h3> Automated pipelines can significantly reduce dev resources and speed up time to a production-ready launch. But every project is unique and requires some forethought for the right tool. As experts in R/Shiny, we know how to find the right tools for the right project. Appsilon is a proud <a href="https://appsilon.com/appsilon-data-science-is-now-an-rstudio-full-service-certified-partner/" target="_blank" rel="noopener noreferrer">RStudio Full Service Certified Partner</a> with a well-balanced team devoted to providing industry-leading solutions to our clients. We can help guide you to find the best solution for a high-quality, enterprise application.