Posit x Databricks: A Game-Changing Synergy for Data Teams

Estimated time:

time

min

On Thursday, December 5th, 2023, Posit and Databricks held a joint event, revealing several key developments in their <strong>ongoing collaboration</strong>.

These updates, <a href="https://www.databricks.com/blog/databricks-and-posit-announce-new-integrations" target="_blank" rel="noopener noreferrer">building on announcements from July 2023</a>, showcase tangible progress in <strong>data analysis and cloud computing integration</strong>. James Blair, Product Manager of Cloud Integrations for Posit, alongside Rafi Kurlansik, Lead Product Specialist, took the stage to unveil these updates.

This article provides an overview of the event’s highlights and a glimpse into the future of data science with <a href="https://posit.co/" target="_blank" rel="noopener noreferrer">Posit</a> and <a href="https://www.databricks.com/" target="_blank" rel="noopener noreferrer">Databricks</a>.

<h3>Table of Contents</h3><ul><li><strong><a href="#data-intelligence">Databricks' Data Intelligence Platform</a></strong></li><li><strong><a href="#posit-integration">Posit’s Integration with Databricks</a></strong></li><li><strong><a href="#posit-workbench">New Developments in Posit Workbench</a></strong></li><li><strong><a href="#open-source-enhancements">Enhancements in Open Source</a></strong></li><li><strong><a href="#future-posit-lakehouse">The Future: Posit Products in Lakehouse Applications</a></strong></li><li><strong><a href="#future-directions">Practical Implications and Future Directions</a></strong></li></ul>

<hr />

<h2>Highlights of the Presentation</h2>

The event, aimed at data scientists and cloud computing professionals, was divided into two main segments: <strong>Databricks' latest capabilities</strong> and <strong>Posit’s new integrations</strong>.

<h3 id="data-intelligence">Databricks' Data Intelligence Platform</h3>

The first segment spotlighted Databricks' decision to bolster their Data Intelligence Platform, including an introduction to the <a href="https://www.databricks.com/blog/2023/04/18/use-databricks-anywhere-databricks-connect-v2.html" target="_blank" rel="noopener noreferrer">Databricks Connect V2</a>, a lightweight client library to run a remote connection to Databricks, and its integration with Visual Studio Code as a plugin.

A particularly interesting use case showcased was the integration of large language models (LLMs) hosted on Databricks to interpret and answer data queries in natural language, a very interesting step forward in data accessibility and analysis.

This direction for Databricks' platform and Posit's integration are expected to<strong> streamline various data processes</strong> and <strong>enhance the efficiency of data teams</strong>.

<h3 id="posit-integration">Posit’s Integration with Databricks</h3>

The second part, which forms the crux of this article, focused on Posit's advancements, specifically their integration with Databricks in the RStudio Pro IDE available on Workbench.

<blockquote>Learn how Posit Connect can revolutionize remote data science team operations <a href="https://appsilon.com/rstudio-connect-as-a-solution-for-remote-data-science-teams/" target="_blank" rel="noopener">by exploring our detailed analysis</a>.</blockquote>

<h4 id="posit-workbench">New Developments in Posit Workbench</h4>

New features exclusive to Workbench, expected to be released by the end of the month, were introduced. These features are tailored to enhance the interaction with Databricks within the IDE, indicating a strategic focus on<strong> improving user experience</strong> and <strong>functionality</strong>:

<ul><li><strong>Credential Management:</strong> Simplified authentication processes to seamlessly access Databricks resources from Posit Workbench (no more PATs [Personal Access Tokens]!).</li><li><strong>New Databricks Pane:</strong> A new view into the Databricks compute console where you can view information, start or stop clusters available on the platform, and connect to them.</li><li><strong>Automatic SparklyR dependency management:</strong> Workbench will manage the necessary virtual environments required to connect to the Databricks environment.</li><li><strong>Unity Catalog view:</strong> On the Connections pane, you will be able to browse through the Unity Catalog with an ODBC (Open Database Connectivity) or SparklyR connection.</li></ul>

<h3 id="open-source-enhancements">Enhancements in Open Source</h3>

A new iteration of the ODBC package is set to be released on CRAN shortly, introducing a <code>databricks</code> function designed to utilize a Databricks driver for more efficient connectivity to the Databricks environment.

Additionally, Posit is gearing up to provide its own ODBC driver specifically for Databricks, supplementing the existing range of drivers they offer.

These enhancements, along with <a href="https://posit.co/blog/databricks-clusters-in-rstudio-with-sparklyr/" target="_blank" rel="noopener noreferrer">significant updates to the SparklyR project</a> to support the new Databricks Connect v2, and new Databricks runtimes, are aimed at streamlining workflows, enhancing compatibility, and boosting performance with Databricks services.

<blockquote>Curious about the transformative impact of open-source R/Shiny tools on pharma? Check out this article on <a href="https://appsilon.com/r-shiny-pharma-open-source/" target="_blank" rel="noopener">How Open Source (R and Shiny) Is Transforming Processes in the Pharmaceutical Industry</a>.</blockquote>

<h3 id="future-posit-lakehouse">The Future: Posit Products in Lakehouse Applications</h3>

Posit is setting its sights on incorporating their suite of products into the Lakehouse architecture framework. This strategic move is designed to maintain data within the secure confines of Databricks, while simultaneously empowering data scientists with the ability to effortlessly access, analyze, and share data.

Such an integration marks a pivotal advancement in the realms of data governance, security, and management, reflecting Posit's commitment to evolving with the changing dynamics of data science.

<h2 id="future-directions">Practical Implications and Future Directions</h2>

The collaboration between Posit and Databricks opens up new avenues for data scientists and engineers, particularly those working with R.

These updates not only streamline workflows but also enhance data security, governance, and scalability. This also makes creating Shiny applications that utilize Databricks as a Data Source (or compute source) more appealing and within reach.

As the data science landscape continues to evolve, the Posit and Databricks partnership is poised to play a significant role in shaping future trends and technologies.

<blockquote>Did you find this information useful? Stay informed with the latest R/Shiny developments by <a href="https://appsilon.us16.list-manage.com/subscribe?u=c042d7c0dbf57c5c6f8b54598&id=870d5bfc05" target="_blank" rel="noopener">subscribing to Shiny Weekly</a>.</blockquote>

‍