Unlocking $930K Annual Savings with a Future-Proof Data Analytics Platform

Project overview

‍

A top 50 pharmaceutical company, pleased with their previous experience working on an R Shiny project with Appsilon, turned to our architects to design a more modern solution where statisticians and data scientists could perform analytics in R and Python, generate insights, and share them with stakeholders. The primary incentive was to transition away from an off-the-shelf software solution that cost $930,000 annually. 

‍

With a tight deadline to meet before the license renewal, our engineers and architects stepped in, designing and deploying a scalable, reproducible, extensible, and flexible environment. They then provisioned components into the system. By meeting the deadline, the client was able to reduce data analytics platform costs by $930,000 annually.

‍

By leveraging a modern tool setup—including Kubernetes, Terraform on AWS, CI/CD with ArgoCD and GitLab CI, and the Posit environment—the system is future-proof, helping the client explore options to use the same system for statistical computing environments.

‍

‍

Challenge

‍

The client decided to move away from expensive off-the-shelf software and build their own GxP-compliant analytics platform. The goal was to provide end users with a one-stop shop for conducting analytics in Python and R, generating insights, and sharing them with stakeholders.

‍

The initial idea was to leverage an existing solution built on a Posit environment supported by infrastructure as a code in the form of Ansible and Appsilon's open source Ansible Scripts. However, this solution lacked the necessary confidence in its scalability and stability, leading to issues with overall usability. The quality did not meet the standards required for an enterprise-ready system.

‍

The challenge was to consolidate tools into a single system used by five different functions, replacing the need for five separate platforms. Delivering the expected solution required a deep understanding of modern software engineering tools, particularly:

• Posit ecosystem
• Cloud infrastructure design on AWS
• Kubernetes
• Observability 
• CI/CD 
• R and Python environment management

The main challenge was meeting a tight deadline to ensure the new system was fully provisioned before the proprietary software license was renewed. 

‍

‍

Solution

‍

Our engineers and infrastructure architects designed and deployed a scalable, reproducible, extensible, and flexible environment that is modern and more future-proof.

‍

Under significant time pressure, Appsilon ensured the project scope was tailored to the client's needs. The goal was highly ambitious, both in terms of technology and the overall project scope. Throughout the project, there were many unknowns, and the team remained flexible, scaling up and down as needed.

‍

At one point, approximately 2 FTEs (full-time engineers) worth of work were split among five Appsilon team members. When necessary, we brought in additional developers, including those with expertise in other technology stacks, to meet project demands. For example, during the first iteration, we brought in a developer to support React development.

‍

Reproducible Cloud Infrastructure

‍

‍

The client provided us with a policy-enabled AWS account, including all internal security rules and policies that needed to be followed. Collaborating with the client's AWS cloud team to ensure compliance with all security and other standards, we deployed infrastructure as code using Terraform, which ensured the system's reproducibility, extensibility, maintainability and transparency.

‍

‍

Kubernetes for Extensibility and Future-proofing

‍

‍

We decided to use Kubernetes as it functions as an operating system for the cloud, allowing us to install and manage different “applications” seamlessly. In this case, Posit tools were the major applications deployed. We designed the system to achieve a good balance of scalability and cost-efficiency. It enabled users to share AWS resources across applications effectively, and restricted them to guarantee better physical segregation between workloads when needed.

‍

We recommended moving away from Flux, which was previously used internally but did not meet our standards. Instead, we implemented ArgoCD with Helm for deploying multiple environments, aiming to centralize as much as possible within a single cluster to achieve better cost optimization. We deployed three clusters with nine separate namespaces, six of which are dedicated to user-centric environments. The remaining three namespaces are used for operational purposes, such as maintaining the monitoring stack or supporting ArgoCD. To monitor elements within the cluster, we utilized Prometheus, Grafana, Loki, and Promtail as our observability stack.

‍

The system is highly flexible, allowing the client to scale resources as needed without manual intervention. When machines are not in use, Kubernetes automatically shuts them down and can quickly spin them back up when needed. The designed solution aligns closely with the concept of "rightsizing," ensuring optimal workload performance and capacity at the lowest possible cost.

‍

There is still room for improvement in future iterations, with the potential to introduce Karpenter, which could enable an even tighter resource fit. However, due to the additional complexity involved, we have decided to move this enhancement to later releases.

‍

As a result, the system is not only extensible but also provides a solid foundation for adopting modern open-source stacks to further optimize costs. Additionally, it is well-positioned to support future initiatives, such as using the same system for statistical computing environments.

‍

‍

CI/CD Automation for Consistency

‍

To ensure consistency and scalability, we implemented CI/CD automation using ArgoCD and GitLab CI. This approach automates deployment processes, streamlines code integration, and maintains system stability through continuous delivery. Other teams can now achieve the same level of deployment quality and execution efficiency.

‍

‍

R Focused UX with Posit

‍

Finally, we installed and configured the Posit environment which sits on top of Kubernetes to provide cloud native scalability to all types of workloads and applications. We tied the R ecosystem (Workbench and Connect) with consistent package management on Posit Package Manager and JFrog Artifactory, ensuring optimized and controlled R user experience, behind the client’s closed environment.

‍

$930,000 Saved Yearly, and Evaluating the System Designed for SCE Usage

‍

The deadline to deliver the platform was set one month before the license for the proprietary system was due to renew. The Appsilon team delivered the GxP setup two days ahead of the deadline. The one-month buffer was necessary to account for component provisioning, unexpected delays, and migration efforts.

‍

This was an exemplary release. Not just on-time, but before schedule with no excuses or panic from stakeholders.

VP of Global Data and Analytics

‍

Coupled with the separation of other elements of the environment, this key piece of the system enabled the client to proceed with decommissioning the environment, reducing their annual licensing costs by $930,000. Additional savings included an immediate drop in the AWS infrastructure cost. Comparing pure computing costs, the consumption was reduced from from $11,000 to $1,750 per in the first month release. It will increase slightly as more components and more users are added, but due to the fact that its enterprise wide compute, shared by multiple business units, the increment of cost per user lowers drastically.

‍

The system is currently used by more than 70 active users (publishers and data scientists) and this number will continue to grow across the current five business units using it, and additional two others that are testing it. It excludes viewers—users who just access the data dashboards and interactive reports hosted on the platform.

‍

We managed to consolidate all of the tools into one system that serves as a foundation to use the same system for statistical computing environments. 

Our engineers took a true cloud native approach utilizing modern tools setting a standard for other pharmaceutical companies on how to build future-proof computing environments.

‍

Testimonial

‍

Appsilon team is exceptional. The engineers challenge and speak their minds which helps to do the justice to the product.

Solutions Architect

‍