Databricks To Leverage NVIDIA’s Full Stack To Accelerate Generative AI, Enhancing Data Processing Efficiency

Databricks, a leader in data and AI, integrates NVIDIA CUDA to accelerate computing into its platform. This collaboration aims to enhance data processing efficiency and AI development, with initiatives like native GPU acceleration and the introduction of DBRX as an NVIDIA NIM microservice for generative AI applications.

Databricks, a leading Data and AI company, announced an expanded collaboration with NVIDIA at the Data + AI Summit. This collaboration aims to optimise data and AI workloads by integrating NVIDIA CUDA accelerated computing into the core of Databricks' Data Intelligence Platform. By enhancing data prep, curation, and processing capabilities, the partnership seeks to improve the efficiency, accuracy, and performance of AI development pipelines for modern AI factories.

CUDA is a parallel computing platform and programming model developed by NVIDIA specifically for utilising its GPUs (graphics processing units) in general computing tasks. It allows developers to accelerate compute-intensive applications by exploiting GPU parallelism for parts of the computation that can be parallelised.

The integration of NVIDIA CUDA into the Databricks platform will empower enterprise generative AI by utilising GPU acceleration for enhanced data processing speed and efficiency. 

“Data is the fuel for the generative AI industrial revolution, so reducing data processing energy demands with accelerated computing is essential to sustainable AI platforms”, says Jensen Huang, founder and CEO of NVIDIA. 

This advancement, including the introduction of DBRX as an NVIDIA NIM microservice, aims to simply model deployment and customisation, thereby boosting productivity and enabling organisations to scale their AI initiatives with greater agility and performance.

“Databricks is the pioneer of large-scale data processing. By bringing NVIDIA CUDA acceleration to Databricks’ core computing stack, we’re laying the foundation for customers everywhere to use their data to power enterprise generative AI", adds Jensen. 

Benefits of integration of NVIDIA CUDA in the computing stack of Databricks

Databricks will add direct NVIDIA GPU support to their Data Intelligence Platform as part of their partnership. This will enhance enterprise capabilities in AI, including improving classical machine learning model training, developing generative AI, and optimising digital twins.

Databricks plans to enhance its next-generation vectorised query engine, Photon, with native support for NVIDIA-accelerated computing. Photon powers Databricks SQL, the company's serverless data warehouse known for its industry-leading performance metrics. 

The integration of NVIDIA CUDA in the computing stack of Databricks aims to significantly improve speed and efficiency for customers' data warehousing and analytics workloads while offering enhanced price performance and total cost of ownership.

"We're thrilled to continue growing our partnership with NVIDIA to deliver on the promise of data intelligence for our customers from analytics use cases to AI," said Ali Ghodsi, Co-founder and CEO at Databricks. "Together with NVIDIA, we're excited to help every organisation build their own AI factories on their own private data."

Creating generative AI factories with NVIDIA NIM

COMPUTEX, an annual computer expo co-organised by the government-funded Taiwan External Trade Development Council (TAITRA) and private sector Taipei Computer Association (TCA), where major technology players such as AMD, Intel, Nvidia, Acer, and Asus participate. Databricks introduced its open-source model DBRX as an NVIDIA NIM microservice at COMPUTEX. NVIDIA NIM inference microservices offer pre-optimised containers for deploying generative AI models, boosting developer productivity, and simplifying model integration across applications.

Launched in March 2024, DBRX leverages Databricks' tools and NVIDIA DGX Cloud for training, enabling organisations to customise models with enterprise data or use it as a reference architecture for building custom models.

Databricks has recently expanded its capabilities and market presence through strategic moves. This includes acquiring Tabular, a data management startup co-founded by Ryan Blue, Daniel Weeks, and Jason Reid, known for their expertise in Apache Iceberg™ and Delta Lake technologies. This acquisition aims to improve data compatibility across various formats, enabling seamless data utilisation for organisations. Additionally, Databricks is enhancing its Delta Sharing open ecosystem through innovative products and partnerships, focusing on efficient data integration to support AI innovation and break down data silos.

About NVIDIA

NVIDIA, founded by Jensen Huang, is renowned for revolutionising visual computing and AI technologies. Specialising in GPU-accelerated computing, NVIDIA has played a pivotal role in advancing AI research and development across industries. Its CUDA platform has become synonymous with high-performance computing, enabling faster processing and groundbreaking innovations in AI, graphics, and data science. 

About Databricks

Headquartered in San Francisco, with a global presence through multiple offices, Databricks was founded by the original creators of Lakehouse, Apache Spark™, Delta Lake, and MLflow. This extensive background highlights its strong expertise and leadership in both data management and AI innovation. More than 10,000 organisations worldwide, including prominent names such as Block, Comcast, Conde Nast, Rivian, and Shell, as well as over 60% of the Fortune 500 companies, rely on Databricks' Data Intelligence Platform. The platform empowers enterprises to leverage their data effectively, facilitating advanced analytics, machine learning, and AI-driven insights. 

Databricks saw substantial financial growth, exceeding $1.6 billion in revenue for the fiscal year ending January 31, 2024, marking over 50% year-over-year growth. This underscores their leadership in providing robust data analytics and AI solutions that meet growing enterprise demands.