Partnering with domain experts and leveraging the power of technology and data science to make a positive impact is a core goal at Appsilon. Through our <a href="https://data4good.appsilon.com/" target="_blank" rel="noopener">Data for Good program</a>, we are committed to using technology for the betterment of society. Recently, we were approached by ETH Zürich to address key challenges with their app, MAPMAKER, which <a href="https://appsilon.com/plankton-diversity-climate-change-interactive-dashboard/" target="_blank" rel="noopener">visualizes the impact of climate change on global plankton diversity</a>. Although the app developed by their in-house research scientists had potential, it suffered from performance issues and a suboptimal interface. Understanding the significance of this project, we eagerly took on the task to utilize our expertise. Our main objectives were to significantly improve the app's speed and visual appeal while seamlessly converting the existing Python code into R code. Our aim was to provide the researchers at ETH Zürich with a highly efficient tool that enables them to make informed decisions quickly and effectively. Through our Data for Good program, <a href="https://appsilon.com/impactful-data-for-good-projects-2022/" target="_blank" rel="noopener">we empower scientists and conservationists</a> to amplify their research and impact by equipping them with the necessary tools for driving positive change. We take great pride in delivering solutions that not only solve problems but also create long-term value for our partners. Throughout the course of the project, we encountered numerous challenges, and in this blog, we aim to share some of the valuable insights we gained from this experience. TOC: <ul><li><a href="#netcdf">Advantages of NetCDF Files</a></li><li><a href="#action">Adding an Action Button for Calculations</a></li><li><a href="#map">Making a Faster World Map</a></li><li><a href="#globe">Making a Faster Globe</a></li><li><a href="#tsp">Time Series Plot(s)</a></li><li><a href="#interact">How Components Interact With One Another</a></li><li><a href="#tooltips">Adding Tooltips</a></li><li><a href="#cicerone">Adding a Guided Tour With Cicerone</a></li></ul> <hr /> <h2>Data and Your Dashboard</h2> <h3 id="netcdf">Advantages of Using NetCDF Files</h3> For this project we had over 80 NetCDF (Network Common Data Form) files, amounting to a total size of more than 5 GBs. This was remarkable as NetCDF is a compressed file format. In our case, the data was stored in files with the .nc extension, which is associated with NetCDF service. NetCDF serves as a binary data standard specifically designed for storing scientific data and its related metadata. It is a platform-independent format used for storing multidimensional data across various disciplines. Notably, NetCDF files have gained popularity within the MATLAB community. <h4>{stars} and NetCDF in R</h4> To handle these files in our project, we utilized the stars package in R. The stars package offers a robust infrastructure for working with data cubes and array data that possess labeled dimensions, with particular focus on dimensions related to time and/or space. Additionally, the researchers had already computed certain values and stored them as metadata within these NetCDF files. Leveraging this existing metadata proved advantageous as it eliminated the need to read sf files and compute those statistics on the fly. For a more comprehensive understanding of the stars package and its capabilities, we recommend visiting their page on Spatiotemporal Arrays, Raster and Vector Data Cubes at stars (<a href="https://r-spatial.github.io/stars/index.html" target="_blank" rel="noopener">r-spatial.github.io</a>). It's worth noting that this package offers a range of functionalities and features beyond what we utilized in this particular project. <h3>Appsilon's Rhinoverse</h3> In our development process, we utilized the our Rhino package in conjunction with shiny.semantic, both of which are components of the <a href="https://rhinoverse.dev/#rhino" target="_blank" rel="noopener">Rhinoverse</a>. Rhino played an important role in enabling us to maintain a clean and comprehensible codebase that our team could easily grasp. Moreover, it offered essential functionalities such as SASS compilation and JS bundling. While seemingly trivial, these features made a significant difference in the app's development workflow. If you're interested in Rhino's capabilities, we recommend delving further into the <a href="https://appsilon.github.io/rhino/" target="_blank" rel="noopener">Rhino documentation</a>. <h3 id="action">Faster Data Loading in Shiny</h3> In the original app, all data was loaded at the beginning of the project to avoid the need for on-the-fly data loading. This approach made sense, especially when considering potential network download times if the data were stored in a different location. However, when the data was stored in the same system, reading the data took approximately 1-2 seconds or less. Initially, we adopted a similar approach to the original app, but we encountered <strong>a significant increase in RAM consumption without experiencing notable speed improvements</strong> in the app. Therefore, we found it more advantageous to load data only when it was needed. The overhead associated with this approach was minimal. However,<strong> the true source of the app's slowness turned out to be the rendering of JavaScript graphs</strong>. The process of creating charts was taking a considerably longer time. We will delve deeper into this issue in subsequent sections. <h4>Calculation Button</h4> One of the most impactful changes we made in the entire app was the implementation of a calculation button. This feature ensured that calculations would only commence when the user explicitly pressed the button, preventing accidental triggering of lengthy computations while interacting with dropdown menus. In the original app, calculations would start immediately upon changing inputs, leading to a significant resource consumption, particularly when dealing with heavy calculations that could take more than 10 seconds. <img class="aligncenter size-full wp-image-19320" src="https://webflow-prod-assets.s3.amazonaws.com/6525256482c9e9a06c7a9d3c%2F65b01b30923c12912d3e30df_calculation-button.webp" alt="calculation button in R Shiny" width="1374" height="180" /> The addition of this button had multiple benefits. Firstly, it eliminated unnecessary calculations, reducing resource usage and improving overall app performance. Secondly, it greatly enhanced the user experience. Upon launching the app, users are greeted with a tutorial page while the calculation runs discreetly in the background. Although this doesn't technically speed up the app, <strong>it alters the user's perception of time</strong>. The app appears faster because users are engaged with the tutorial, and in some cases, if they spend a little longer on the tutorial page, they may find a fully loaded dashboard awaiting them. The calculation button not only optimized the app's functionality but also contributed to a smoother and more satisfying user experience. <img class="aligncenter size-full wp-image-19328" src="https://webflow-prod-assets.s3.amazonaws.com/6525256482c9e9a06c7a9d3c%2F65b01b32a862659a10c1c557_mapmaker-shiny-dashboard.webp" alt="R Shiny tutorial page " width="1526" height="946" /> <h2>Visualizing Data in a Shiny Dashboard</h2> The dashboard incorporated three essential components for data visualization, each offering unique perspectives on the presented information. These components included: <ol><li>World Map</li><li>Globe</li><li>Time Series Plots</li></ol> <h3 id="map">Making a Faster World Map in Shiny</h3> The challenge we faced was plotting over 30,000 data points on a world map using Leaflet. However, the client's requirements focused solely on visualizing ocean data, and the map context with country names was unnecessary for this particular project. To address this, we decided to adopt the original style of the map from an older app, which utilized a clever technique—a contour plot with a world map image overlaid on top. Yes, you read that right! We incorporated a world map PNG image onto a contour plot, <strong>resulting in improved performance and efficiency</strong>. Plotly, echarts, and Leaflet were all relatively slow when rendering such a large number of data points. However, this unique approach proved to be a game-changer. Here's a snippet of the code we used to insert the PNG image into the Plotly chart: <pre><code>#The basemap image is in base64 format plotly::plot_ly() |&gt; plotly::layout(  plot_bgcolor = "transparent",  paper_bgcolor = "transparent",  images = list(    list(      # Add world map as background      source = world_map_img,      xref = "x",      yref = "y",      x = -180,      y = 90,      sizex = 360,      sizey = 180,      sizing = "stretch",      opacity = 1,      layer = "below"    )  ),  autosize = TRUE,  margin = list(t, 0, r = 40, l = 0),  xaxis = no_axis_settings,  yaxis = no_axis_settings ) </code></pre> After implementing this technique, the world map underwent a transformation that closely resembled the original version. In fact, it became virtually indistinguishable from the conventional approach. This neat trick proved to be valuable, particularly in scenarios where only ocean data is required. By employing this method, you can similarly achieve a seamless and efficient visualization of information, all while maintaining a visually appealing and user-friendly interface. <img class="aligncenter size-full wp-image-19340" src="https://webflow-prod-assets.s3.amazonaws.com/6525256482c9e9a06c7a9d3c%2F65b01b33832c47d6730fac12_worldmap-trick-in-r-shiny.webp" alt="" width="1600" height="782" /> <h3 id="globe">Making a Faster Globe in Shiny</h3> Among all the components in the app, the globe proved to be the most time-consuming to render. With Plotly, the globe could take anywhere between 20 to 30 seconds to load, while echarts performed slightly better, taking around 5 to 10 seconds after implementing necessary adjustments. Interestingly, Plotly's globe creation was slower but faster to render, whereas echarts offered a faster creation process but slower rendering time. This presented us with a trade-off between the two options. Ultimately, we discovered that the echarts globe provided significantly better performance in terms of speed. <h4>Snowball Earth or Just a Frozen App?</h4> Additionally, we observed that the color scheme of the globe played a crucial role in rendering efficiency. Opting for a black and white color scheme resulted in a few seconds of speed improvement compared to using different colors for the globe. Another challenge we encountered was the simultaneous rendering of both the map and the globe, which caused the entire app to freeze. Finding a solution to address this issue became imperative and will be discussed further in the integration section. By carefully considering the choice of globe implementation and color scheme, we were able to optimize the rendering speed and overall performance of the globe component, ensuring a smoother and more responsive user experience within the Shiny app. <img class="aligncenter size-full wp-image-19338" src="https://webflow-prod-assets.s3.amazonaws.com/6525256482c9e9a06c7a9d3c%2F65b01b357b599ca0447f73bb_world-globe-trick-in-r-shiny.webp" alt="" width="1478" height="858" /> <h3 id="tsp">Aggregating Time Series Plots</h3> In addition to the world map and globe components, the app incorporated a time series graph built using Plotly. However, the original app featured individual univariate time series plots, which made it challenging to compare different series. Users had to manually select the desired series from a dropdown menu, resulting in a less intuitive user experience. To address this limitation, we undertook the task of aggregating all the time series plots into a single, comprehensive visualization that offered improved aesthetics and enhanced information. By integrating the various time series plots into a unified graph, we not only improved the visual presentation but also made it easier for users to compare different series at a glance. This approach eliminated the need for dropdown selection, streamlining the process and providing a more user-friendly interface. The aggregated time series plot served as a valuable tool for gaining insights into multiple series simultaneously, enabling users to identify patterns, trends, and correlations more effectively. Through this enhancement, we aimed to create a visually appealing and informative time series graph that significantly enhanced the overall usability and utility of the Shiny app. <img class="aligncenter size-full wp-image-19334" src="https://webflow-prod-assets.s3.amazonaws.com/6525256482c9e9a06c7a9d3c%2F65b01b352d3baed3c5a64b23_species-richness-time-series-plot-aggregate.webp" alt="" width="1600" height="761" /> <h2 id="interact">Integrations: Hidden Secrets for Improving Shiny Dashboard Performance</h2> Despite the improvements made thus far, the app still fell short of meeting the high-performance standards of Appsilon. The primary culprit behind its sluggishness was the excessive resource consumption during the rendering of JavaScript components. To address this challenge and reduce the burden on the client-side, we embarked on a mission to uncover hidden secrets that would optimize the app's performance. These are our 'hidden' results that helped us optimize the Shiny dashboard without harming the user experience: <h3>Hiding Time Series Plots</h3> To start, we implemented a strategy to hide the globe and time series plot by default. Instead, the time series plot would only be displayed when a user clicked on a specific point on the map to select it. This approach reduced the initial rendering time and enhanced the app's responsiveness. Upon the first selection, the time series plot was rendered, and subsequent clicks triggered updates through the use of proxies. This method not only facilitated a smoother user experience but also contributed to speeding up the underlying code execution. <img class="aligncenter size-full wp-image-19326" src="https://webflow-prod-assets.s3.amazonaws.com/6525256482c9e9a06c7a9d3c%2F65b01b37b556fdf5a54e60f2_hiding-time-series-plot-in-r-shiny.webp" alt="" width="627" height="204" /> By dynamically displaying the time series plot based on user interaction and leveraging proxies for subsequent updates, we achieved a more efficient utilization of resources and improved the app's responsiveness. This approach contributed to a seamless and intuitive user experience, ensuring that the time series plot was available precisely when needed, without incurring unnecessary computational overhead. <h3>Hiding the Globe</h3> To further optimize the app's loading time and improve its overall responsiveness, we implemented a simple yet effective strategy: we moved the globe component to a separate tab. As a result, when the app initially loads, it only computes and displays the world map, giving the impression of a more immediate and responsive experience. The rendering of the globe is deferred until the user actively clicks on the dedicated "Globe" tab. <blockquote>Before you make a change, <a href="https://appsilon.com/benchmark-lessons-improving-shiny-dashboard-performance/" target="_blank" rel="noopener">try shiny.benchmark for a quantitative approach to benchmarking Shiny performance bottlenecks</a>.</blockquote> This small but significant change had a noticeable impact on the app's loading time. By separating the loading time of the globe from that of the map, we were able to prioritize the faster rendering of the world map while deferring the more computationally intensive rendering of the globe until specifically requested. This not only reduced the overall loading time but also enhanced the user's perception of the app's performance. <img class="aligncenter size-full wp-image-19324" src="https://webflow-prod-assets.s3.amazonaws.com/6525256482c9e9a06c7a9d3c%2F65b01b3904c5410aa96bddaf_hiding-globe-and-worldmap-in-r-shiny.webp" alt="" width="1600" height="523" /> By strategically organizing the content within separate tabs and deferring the rendering of resource-intensive components, we achieved a more efficient utilization of computational resources and improved the overall responsiveness of the app. Users can now enjoy a seamless experience while effortlessly exploring the world map and selectively engaging with the globe component as desired. <h3>Zoom Buttons and Smooth Interactions</h3> In our quest to enhance the app's functionality and user experience, we added zoom buttons for all the graphs. These zoom buttons allowed users to enlarge the plots, enabling them to scrutinize even the finest details that might not be readily apparent on smaller displays. By providing this capability, we invite users to delve deeper into the data. <img class="aligncenter size-full wp-image-19342" src="https://webflow-prod-assets.s3.amazonaws.com/6525256482c9e9a06c7a9d3c%2F65b01b392700d7cf10b29e1b_zoom-buttons-in-r-shiny-dashboard.webp" alt="" width="1600" height="96" /> Through these improvements, we achieved our goal of delivering an app that not only met the performance requirements but also offered a fluid and intuitive user experience. The integration of zoom buttons, along with the previously implemented optimizations, harmonized the interaction flow, allowing users to effortlessly explore, analyze, and appreciate the rich data visualizations within the app. <h2>Shiny UI Design and User Flow</h2> Over the years and after many development projects we've come to intimately understand the <a href="https://casestudies.appsilon.com/" target="_blank" rel="noopener">powerful capabilities of R and Shiny applications</a>. And although not every application needs to be a work of art, proper design does matter. And with Shiny, proper design and visual appeal doesn't mean sacrificing computational power for an eye candy app. <blockquote>Ready to look beyond UI? <a href="https://appsilon.com/ux-design-of-shiny-apps-7-steps-to-design-dashboards-people-love/" target="_blank" rel="noopener">Start your UX design journey with Ania Skrzydło's 7 steps for UX design in Shiny</a>.</blockquote> With the expertise of our UI-UX designers, we embarked on a complete redesign of the MAPMAKER app, starting from the ground up. This redesign aimed to create a visually stunning and user-friendly dashboard that would elevate the overall experience for the users. <h3>From Visual Appeal to Streamlined Flow</h3> By leveraging the skills of our talented designers, we achieved a dashboard that boasted an aesthetically pleasing and modern interface. The redesign provided our app with more breathing room, allowing the elements to stand out. This not only enhanced the visual appeal but also contributed to a more immersive and engaging user experience. Furthermore, the redesign efforts went beyond the visual aspect and encompassed streamlining the flow of the app. Our designers paid meticulous attention to user interaction patterns, ensuring that the app's layout and navigation were intuitive and seamless. By optimizing the user flow, we aimed to empower users to effortlessly navigate through the dashboard, discover relevant features, and interact with the data in a meaningful way. Through the collaborative efforts of our design team and the technical expertise of our developers, we successfully created a redesigned app that marries functionality, aesthetics, and usability. This holistic approach enables the users to access powerful features and insightful visualizations <i>with </i>a user-friendly interface that elevates their overall experience. <h3 id="tooltips">Tool Tips in Your Dashboard</h3> Within a Shiny app, tooltips provides users with additional information and context. In the previous version of the MAPMAKER, all the information was confined to tooltips that were only accessible when users clicked on specific buttons. To enhance the user experience and streamline the information flow, we made a significant improvement by transitioning the <strong>tooltips to appear on hover events</strong>. This transition involved migrating and organizing the extensive collection of tooltips, which numbered over 30, into a more structured and manageable format. We leveraged the power of JSON, storing the tooltips as a dedicated file. By doing so, we achieved greater flexibility and modularity in managing and updating the tooltips. During the app's startup process, we dynamically mapped the tooltips to their respective functions, ensuring that the tooltips were readily available and seamlessly integrated into the app's functionality. This modular approach facilitated easier maintenance and modification of the codebase, empowering developers to make changes more efficiently. By transitioning tooltips to appear on hover and utilizing JSON to store and organize the tooltip content, we improved the accessibility and usability of the app. Users could now effortlessly access relevant information and descriptions simply by hovering over the associated elements, eliminating the need for extra clicks. <pre><code> // Data is created by using this json format const tooltipOptions = [  {    id: '#rb_diversity_indices',    lastResortPos: rbDiversityIndicesLastPos,    position: rbDiversityIndicesPos,    topMessage: 'Diversity Indices',    secMessage:      'Different diversity indices based on the Habitat Suitability Index.',  },  {    id: '#rb_species_richness',    lastResortPos: rbDiversityIndicesLastPos,    position: rbDiversityIndicesPos,    topMessage: 'Species Richness',    secMessage:      'Species richness displays the percentage of species present in regard to the selected plankton group. To display this biodiversity index, the habitat suitability index had to be transformed to presence and absence entries. To do so, we used the cut off threshold of the True Skill Statistic which was derived from the species distribution models. To convert each species to a presence or absence entry we use three trhesholds, namely the 25th, median and 75th percentile, derived from three different SDMs (Generalized Linear Model, Generalized Additive Model, Neural) Network) that computed five evaluation runs each. We add the presences and computed the average of the new presence table.',  },  {    id: '#rb_hsi',    lastResortPos: rbDiversityIndicesLastPos,    position: rbDiversityIndicesPos,    topMessage: 'Habitat Suitability Index',    secMessage:      'The Habitat Suitability Index (HSI) hypothesises species-habitat relationships and is a numerical index that represents the capacity of a habitat to support a selected species. The species distribution framework used in this analysis was developed by Righetti et al (2019) and Benedetti et al (2018) to estimate plankton diversity patterns from an ensemble forecasting approcah using three different algorithms namely Generalized Linear Models, Generalized Additive Models and Artificial Neural Networks. Depending on the group that is selected the HSI has been summed up and divided by the total number of species in that particular goup in order to scale it between zero and one. It therefore represents the averaged HSI for each group selected.',  } ] <br>// Tooltips are created by using this function <br>// Fucntion to create a tooltip function tooltipMod(  jsonOptions,  showTooltip = 1000,  hideTooltip = 600,  onEvent = 'hover', ) {  $(jsonOptions.id).popup({    on: onEvent,    delay: {      show: showTooltip,      hide: hideTooltip,    },    exclusive: true,    position: jsonOptions.position,    lastResort: jsonOptions.lastResortPos,    boundary: $('body'),    jitter: 50,    prefer: 'opposite',    hoverable: true,    html: createCard(jsonOptions.topMessage, jsonOptions.secMessage),  }); } <br> tooltipOptions.map((value) =&gt; {  tooltipMod(value);  return null; }); </code></pre> And this is how it looked once completed. This method helped a lot during the development phase. <img class="aligncenter size-full wp-image-19336" src="https://webflow-prod-assets.s3.amazonaws.com/6525256482c9e9a06c7a9d3c%2F65b01b3a923c12912d3e37d2_tooltip-in-r-shiny.webp" alt="" width="763" height="712" /> <h3 id="cicerone">Guided Tour of Your Shiny Dashboard with Cicerone</h3> Although the design was improved, everything new needs a guide. Given the intricate structure and data available, the client specifically requested that the tutorial page be presented at the start-up to help users navigate and familiarize themselves before entering the app. Smart! To fulfill this, we introduced an interactive tour feature powered by John Coene's <a href="https://cicerone.john-coene.com/" target="_blank" rel="noopener">Cicercone package</a>. By leveraging the capabilities of Cicerone, we were able to create a guided tour that walks users through the app's various components and features in a structured and intuitive manner. The interactive tour provides step-by-step instructions, tooltips, and visual cues to help users grasp the app's functionality and maximize their experience. To explore this feature further, we encourage you to check out the Cicerone package, which offers detailed documentation and additional insights into its capabilities. By incorporating this guided tour functionality, we not only enhance the onboarding experience for new users but also empower them to quickly and effectively navigate and utilize the app's full potential. A valuable option for improving adoption! <img class="aligncenter size-full wp-image-19322" src="https://webflow-prod-assets.s3.amazonaws.com/6525256482c9e9a06c7a9d3c%2F65b01b3bac45fa2b44cd96af_cicerone-guided-tour-in-r-shiny.webp" alt="" width="1185" height="471" /> <h3>Dashboard Feedback</h3> With the multitude of activities happening within the app, there are instances where users must wait for processes to complete or interact with various buttons. To enhance the user experience and keep them informed about ongoing calculations, we incorporated two valuable elements: spinners from the <a href="https://waiter.john-coene.com/#/" target="_blank" rel="noopener">Waiter package</a> (shout-out again to John) and toasters from the <a href="https://rhinoverse.dev/#shiny-semantic" target="_blank" rel="noopener">shiny.semantic package</a>. <img class="aligncenter size-full wp-image-19332" src="https://webflow-prod-assets.s3.amazonaws.com/6525256482c9e9a06c7a9d3c%2F65b01b3c569a8a6eb891f2ee_r-shiny-performance-feedback-to-user.webp" alt="" width="639" height="408" /> By leveraging the Waiter package, we integrated spinners that provide visual feedback during background calculations. These spinners serve as indicators, letting users know that processes are underway. This approach improves customer engagement and encourages patience for longer durations, as users are aware that progress is being made behind the scenes. Consequently, the perceived time appears shorter, enhancing the overall user experience. In addition to spinners, we integrated toasters from our shiny.semantic package. These toasters act as unobtrusive notifications, delivering updates or messages to users. By utilizing toasters, we ensure that important information is presented to users in a concise and visually appealing manner, further improving user engagement and communication. By combining spinners and toasters, we successfully create a feedback connection between the app and its users. <h2>Concluding Appsilon's Pro Shiny Design Tips for Enhanced App Performance</h2> Through a series of proactive design choices and strategic optimizations, we have successfully transformed a Python app into a visually stunning, memory-efficient, faster Shiny dashboard. The collaboration with ETHZ on the MAPMAKER project exceeded expectations, leaving the client delighted. The app will soon be hosted on ETHZ's server, ready to make a positive impact! The app showcases the power of Shiny, delivering impressive performance and an engaging user experience. It reflects our passion for pushing the boundaries of Shiny development. Explore the app firsthand, visit the <a href="https://demo.prod.appsilon.ai/ethz-test/" target="_blank" rel="noopener">ETHZ_MapMaker demo app</a>. Play with the app, experience its enhanced features, improved speed, and optimized user flow. If you have a Shiny development project or data-for-good initiative, we're here to help. Our experts deliver tailor-made solutions that meet your unique requirements. Reach out to us today and let's transform your ideas into impactful Shiny applications. Visit our website to learn more about our Shiny development services and explore success stories. Together, let's unlock the full potential of data and create remarkable applications. Experience the power of Shiny with Appsilon, your trusted partner in data solutions.