Databricks and dbt: Understanding the Real Overlap and Differences

Databricks is a lakehouse platform for data, analytics, and AI. dbt is a transformation framework that brings software engineering practices to analytics.

At first glance, their roles seem clear. But as both have evolved, the boundary between them has become less obvious. Databricks now includes native capabilities for pipelines, orchestration, and data quality, while dbt has grown into a more sophisticated system for structuring and managing transformations.

This leads to a common question:

if everything already lives in Databricks, what role does dbt still play?

The answer is not about features. It’s about where each tool operates in the lifecycle of data work.

The core distinction: compile time vs runtime

One of the most important conceptual differences between dbt and Databricks-native features is where they operate in the lifecycle.

• dbt is a compile-time system
• Databricks is a runtime system

dbt defines transformations before execution. It builds a directed acyclic graph (DAG), resolves dependencies, expands macros, and generates SQL. Its job is to describe what should run and in what order.

Databricks executes those transformations. It manages compute resources, runs queries, scales, and persists data. Its job is actually to run the workload.

This distinction explains both the overlap and the limits of that overlap.

dbt structures transformations before execution, while Databricks executes and manages them during runtime.

What dbt actually provides

dbt, short for data build tool, is a transformation framework that allows data teams to define, test, and document transformations using SQL. It introduced a structured approach to the “T” in ELT by applying software engineering practices such as version control, modular design, and testing to analytics workflows.

At its core, dbt is not an execution engine. It is a structured layer for defining transformations.

It introduces:

• Modular SQL models
• Dependency-aware execution (DAG)
• Jinja-based templating and reuse
• Built-in testing and documentation
• Git-driven workflows

Instead of scattered SQL scripts, notebooks, and transformations, it is a compilation and orchestration layer where definitions are defined declaratively and organized into a coherent project.

Developers define models using SQL and Jinja templating, and dbt compiles those definitions into executable SQL statements. It also builds a dependency graph between models and ensures they run in the correct order.

It is worth noting that dbt has expanded beyond pure SQL. Since version 1.3, Python models are natively supported, and when running on Databricks they execute directly on Spark clusters, meaning you don’t lose the platform’s computational power when stepping outside SQL. This closes part of the language gap with Databricks-native tools.

However, the compile-time distinction still holds. Even for Python models, dbt manages the dependency graph, resolves execution order, and coordinates the overall project structure before anything runs. Databricks remains the runtime that actually executes them.

The boundary hasn’t moved. It has just become language-agnostic.

In more recent versions, dbt has evolved further. It now includes state-aware execution, improved incremental processing strategies, and a more advanced compilation engine. These changes reinforce its role as a structured transformation layer rather than just a query runner.

How dbt operates on Databricks

When used with Databricks, dbt acts as a control layer:

• dbt defines models and dependencies
• Databricks executes the generated SQL

In other words, when used with Databricks, dbt connects to either SQL warehouses or Spark clusters and uses them purely as execution backends.

A typical flow still looks familiar:

• Data is ingested into raw (Bronze) tables
• dbt models define transformations into Silver and Gold layers
• dbt compiles SQL and resolves dependencies
• Databricks executes the queries using Spark or SQL warehouses

What is important is that dbt does not replace Spark or SQL. It organizes how they are used. Instead of scattered notebooks or scripts, transformations are defined declaratively, with clear dependencies and structure.

Databricks has strong support for dbt:

In that case, you write dbt models (SQL files), then dbt operates primarily at compile time (building a DAG and generating SQL), while Databricks-native tools operate at runtime (executing pipelines directly)

Native Integration:

• dbt-databricks adapter: official adapter for running dbt projects against Databricks SQL warehouses and clusters
• Databricks Workflows: orchestrate dbt runs as job tasks with built-in scheduling, monitoring, and alerting
• Databricks Asset Bundles (DABs): deploy dbt projects alongside notebooks, jobs, and pipelines as versioned bundles

Key Features:

• Unity Catalog integration: dbt models create managed tables in Unity Catalog with lineage tracking
• Incremental models: leverage Delta Lake’s MERGE for efficient incremental updates
• SQL Warehouse compute: serverless or provisioned compute optimized for SQL transformations
• Git integration: connect dbt repos directly to Databricks Repos for version control

Where Databricks overlaps

Databricks has expanded significantly beyond execution.

Today it includes:

• Lakeflow / Delta Live Tables for declarative pipelines
• Jobs and Workflows for orchestration
• Unity Catalog for governance and lineage
• Native support for incremental processing and data quality

These capabilities overlap with dbt in areas like:

• Pipeline definition
• Dependency management
• Testing and lineage

But the overlap is not complete:

• Databricks-native features are execution-centric. They are designed to define and run pipelines inside the platform.
• dbt remains model-centric. It focuses on structuring transformation logic before execution begins.

Databricks’ native capabilities and where they overlap

Databricks has significantly expanded its native data transformation capabilities. Features such as Lakeflow Declarative Pipelines, evolving from Delta Live Tables, allow teams to define pipelines using SQL or Python with built-in handling for dependencies, incremental processing, and data quality checks.

To make this concrete, consider a simple but realistic example: reading from a Bronze table of raw events and producing a clean, deduplicated daily summary in Silver. This is exactly the kind of transformation that both dbt and Lakeflow can handle, which makes it a useful lens for understanding where they actually differ.

In dbt, you would write something like this:

-- Example reconciliation
-- models/silver/daily_event_summary.sql
-- dbt compiles this into executable SQL and resolves
-- dependencies before anything runs.

{{ config(
materialized='incremental',
unique_key='event_date'
) }}

select
    event_date,
    event_type,
    count(distinct user_id) as unique_users,
    count(*) as total_events
from {{ ref('bronze_events') }}
where status = 'valid'
{% if is_incremental() %}
and event_date > (select max(event_date) from {{ this }})
{% endif %}
group by 1, 2

Notice what dbt is doing here before a single query runs: it resolves the ref(‘bronze_events’) dependency, determines where this model sits in the DAG, expands the Jinja logic based on whether this is an incremental run, and compiles everything into a final SQL statement. Databricks then receives that compiled SQL and executes it. The two tools are operating at entirely different moments in the lifecycle.

The equivalent Lakeflow Declarative Pipeline looks like this:

# pipelines/silver/daily_event_summary.py
# Databricks defines AND executes this pipeline natively.
# There is no separate compile step — definition and
# execution happen inside the same runtime environment.

import dlt
from pyspark.sql import functions as F

@dlt.table(
    comment="Deduplicated daily event summary",
    table_properties={"quality": "silver"}
)
@dlt.expect("valid_date", "event_date IS NOT NULL")
def daily_event_summary():
    return (
        dlt.read("bronze_events")       # dependency resolved at runtime
            .filter(F.col("status") == "valid")
            .groupBy("event_date", "event_type")
            .agg(
                F.countDistinct("user_id").alias("unique_users"),
                F.count("*").alias("total_events")
            )
    )

The result is similar, but something fundamental has shifted. The dependency on bronze_events, the data quality expectation, and the transformation logic are all defined and resolved at runtime, inside the Databricks execution environment. There is no external compilation step, no DAG built independently of the platform, and no separation between what should run and what is running.

This is the distinction that we have been building toward, made visible in code. dbt’s ref() is a compile-time construct! It tells dbt how to order the DAG before execution begins. Lakeflow’s dlt.read() is a runtime construct! It tells Databricks what to connect to when the pipeline actually runs. The two lines look nearly identical on the surface, but they belong to entirely different moments in the data lifecycle.

For a team running a straightforward Silver aggregation like this one, Lakeflow is entirely sufficient and arguably simpler. But as transformation logic grows, with shared macros, reusable models, environment-specific configurations, and cross-project dependencies, dbt’s compile-time layer starts earning its complexity.

The DAG becomes a design tool, not just an execution plan.

Beyond pipelines, Databricks has expanded into areas that further overlap with what teams traditionally relied on dbt or external tools to provide. Databricks Jobs and Workflows enable orchestration of notebooks, SQL tasks, and pipeline runs with built-in scheduling and monitoring. Unity Catalog centralizes governance, permissions, and lineage tracking at the platform level, covering assets that go well beyond what dbt’s own lineage graph can see. And Delta Lake‘s native incremental processing handles many of the update patterns that dbt‘s incremental models were designed to solve.

These capabilities clearly overlap with what dbt provides. Pipeline definitions, dependency management, testing, and lineage can now be handled directly inside Databricks without requiring an external framework. However, the overlap is not exact.

• Databricks-native pipelines are execution-centric: they are designed around defining and running data pipelines within the platform.
• dbt remains model-centric: focusing on how transformations are structured, modularized, and compiled into a coherent DAG before execution even begins.

What can be replaced, and what cannot

Databricks can replace several components traditionally associated with dbt:

• Orchestration (Jobs, Workflows)
• Data quality checks (expectations in pipelines)
• Lineage (Unity Catalog)
• Incremental processing (Delta-based pipelines)

So, Databricks can replace dbt in some architectures, but not as a full 1:1 replacement in most mature setups.

However, it does not fully replicate dbt’s development model. dbt still uniquely provides:

• A modular SQL system (models and macros)
• A compilation layer (Jinja to SQL, and Python since version 1.3)
• A DAG independent of execution
• A consistent project structure

As new capabilities are added to dbt:

• State-aware runs (faster, partial execution)
• Improved incremental strategies (Delta-aware)
• Fusion engine (faster parsing & validation)

So, dbt provides a disciplined way of writing transformations as modular SQL units, supported by macros, reusable patterns, and a clear project structure. Its compilation layer introduces a level of abstraction that separates transformation logic from execution details.

As complexity grows, this distinction becomes more visible, where maintainability, reusability, and clarity of dependencies matter more than just executing pipelines.

When teams move away from dbt

Some teams adopt a fully Databricks-native approach.

This tends to work well when:

• The platform is fully consolidated in Databricks
• Transformations are relatively simple
• Teams prefer fewer tools and tighter integration

In these cases, removing dbt reduces architectural layers.

Why dbt is still widely used

Despite the overlap, dbt remains widely adopted, particularly in more mature data organizations. One reason is its alignment with software engineering practices. It enforces a Git-based workflow, encourages modular design, and provides a clear separation between transformation logic and execution.

Another factor is portability. dbt is not tied to a single platform and can operate across multiple data warehouses. This reduces vendor lock-in and allows teams to maintain consistent transformation logic even if infrastructure changes.

There is also the ecosystem aspect. dbt provides a mature set of tools for testing, documentation, and reusable packages, along with established patterns that many teams already understand.

Finally, dbt’s internal evolution, including improvements in compilation performance and state-aware execution, reinforces its role as more than just a convenience layer. It is increasingly a core part of how transformation logic is defined and managed.

What teams are actually doing
In practice, most teams are not making a binary choice. Instead, hybrid architectures are common. Databricks is used for storage, compute, and ingestion, while dbt handles structured transformations and modeling.

Some teams are experimenting with moving more logic into Databricks-native pipelines, but full migrations away from dbt are not universal. The decision tends to depend more on team preferences and architectural philosophy than on feature availability alone.

What most teams are doing (2025–2026 trend)
The choice is less about features and more about how teams prefer to organize their work.

• Startups / lean teams: go Databricks-native
• Mature data teams: keep dbt
• Hybrids:
  • Airflow loads raw data into Databricks (Delta tables)
  • Databricks for heavy compute
  • dbt for business transformations