What Is Databricks? Architecture, Use Cases, and How It Fits the Modern Data Stack

Databricks is a cloud-based analytics platform built on Apache Spark that unifies data engineering, SQL analytics, and machine learning on a single runtime. At its core is the lakehouse architecture - a design pattern that combines the low-cost, flexible storage of a data lake with the reliability and performance features of a data warehouse. Data lives in open formats (Delta Lake on cloud object storage like S3, ADLS, or GCS), and workloads from batch ETL to real-time ML training run against that same data without copying it between systems.

What makes Databricks distinct is the combination of open formats, multi-cloud deployment (AWS, Azure, GCP), and a single platform for engineers writing Python pipelines, analysts running SQL, and data scientists training models. Organizations adopt it to replace the data lake + data warehouse split that forces duplicate data, duplicate ETL, and duplicate governance. For a detailed comparison with the main alternative, see Databricks vs. Snowflake.

Lakehouse Architecture Explained

The traditional data architecture separates storage into two tiers. A data lake holds raw, unstructured, and semi-structured data at low cost but lacks ACID transactions, schema enforcement, and the query performance analysts expect. A data warehouse provides those guarantees but stores data in proprietary formats, charges premium prices for storage and compute, and cannot handle unstructured data like images, logs, or sensor readings.

The lakehouse pattern eliminates this split. Data lands in cloud object storage (S3, ADLS, GCS) in Parquet format. Delta Lake - an open-source storage layer - adds ACID transactions, schema enforcement, time travel (querying previous versions), and streaming support on top of those Parquet files. The result: one copy of data that supports warehouse-grade queries and lake-grade flexibility.

A concrete example: a retailer storing clickstream data, point-of-sale transactions, and inventory snapshots. In a traditional setup, clickstream goes to S3, transactions go to Redshift, and inventory feeds go to both. With a Databricks lakehouse, all three datasets land in a single Delta Lake. The data engineering team processes them with Spark, the analytics team queries them with SQL, and the data science team trains demand-forecasting models - all against the same governed data, without a single copy pipeline between systems.

Core Components

Delta Lake is the storage layer that makes the lakehouse possible. It wraps Parquet files in a transaction log, giving you ACID guarantees on cloud object storage. You can roll back to previous data versions (time travel), merge streaming and batch data into the same tables, and enforce schemas so malformed records don't corrupt production datasets. Delta Lake is open source, meaning your data is never locked into Databricks - other engines can read it directly.

Apache Spark runtime handles distributed processing. Databricks Runtime, built on Apache Spark, includes the Photon engine - a native C++ vectorized execution engine that accelerates SQL and DataFrame operations. For data engineering workloads, Spark processes anything from gigabytes to petabytes across autoscaling clusters. You write Python, SQL, Scala, or R; the runtime parallelizes execution across available nodes.

Unity Catalog is Databricks' governance layer. It provides centralized access control, audit logging, data lineage tracking, and discoverability across all tables, views, models, and files in the lakehouse. Unity Catalog enforces permissions at the table, column, and row level - critical for organizations in regulated industries. It also records which notebooks, jobs and dashboards read from or write to each table, creating an automated lineage graph.

Databricks SQL gives analysts a familiar SQL interface to query Delta Lake tables. It supports serverless compute (no cluster management), integrates with BI tools like Power BI and Tableau through standard JDBC/ODBC connectors, and includes a built-in visualization layer for ad-hoc exploration. For teams connecting BI tools, see connecting Power BI to Databricks.

MLflow is an open-source platform for managing the ML lifecycle: experiment tracking, model versioning, model registry, and deployment. Databricks integrates MLflow natively, so data scientists can log experiments, compare model performance, promote models to production, and serve predictions - all within the same workspace where the training data lives.

Collaborative notebooks provide an interactive workspace where engineers, analysts, and data scientists work in Python, SQL, Scala, or R within the same notebook. Real-time collaboration, version control integration, and inline visualizations replace the pattern of emailing Jupyter notebooks between teams.

Where Databricks Fits in the Modern Data Stack

Databricks occupies the central "transform and serve" layer of the modern data stack. It sits between ingestion tools and consumption tools, acting as the compute and storage engine where raw data becomes analytics-ready datasets and ML models.

Upstream, ingestion tools like Fivetran and Airbyte pull data from source systems (databases, SaaS APIs, event streams) into the lakehouse. Within Databricks, teams use Spark, SQL, and dbt to transform raw data into clean, modeled tables. Downstream, BI tools (Power BI, Tableau, Looker) connect to Databricks SQL for dashboards and reports. ML models trained in Databricks serve predictions to applications via REST APIs.

Governance spans the full width. Unity Catalog handles access control and lineage within Databricks. For organizations running multiple platforms - Databricks plus Snowflake, plus SaaS tools, plus legacy databases - an external data catalog like Dawiso provides cross-platform governance.

Common Use Cases

Data engineering pipelines. A fintech company ingests transaction data from payment processors, enriches it with customer profiles from a CRM, and writes clean, deduplicated tables to Delta Lake. Spark handles the heavy lifting - deduplication, schema validation, type casting - while Lakeflow pipelines (formerly Delta Live Tables) manage pipeline orchestration and data quality expectations. The resulting tables feed both compliance reports and real-time fraud scoring.

ML model training and serving. A logistics company trains route-optimization models on historical delivery data. Feature engineering, model training, hyperparameter tuning, and experiment tracking all happen within Databricks. MLflow logs every run. When a model beats the current production version, it gets promoted through the model registry and deployed to a serving endpoint - no handoff to a separate ML platform required.

SQL analytics and reporting. A SaaS company uses Databricks SQL to power operational dashboards. Product managers query user engagement metrics through a serverless SQL warehouse connected to Tableau. The same Delta Lake tables that feed the dashboards also feed the churn-prediction model - one copy of data, two workloads, no sync pipeline.

Real-time streaming. A media company processes clickstream events from its platform through Spark Structured Streaming. Events land in Delta Lake within seconds, feeding a real-time recommendation engine and a near-real-time content-performance dashboard. The same architecture handles both the streaming ingestion and the batch aggregations that run overnight for weekly reports.

Pricing and Cost Model

Databricks pricing confuses teams because there are two separate bills. Databricks charges for compute in Databricks Units (DBUs) - a normalized measure of processing power. On classic compute, your cloud provider separately charges for the VMs running those workloads; on serverless, the infrastructure is included in the DBU rate. Storage for your data and any network egress are billed by the cloud provider either way.

DBU rates vary by workload type. Jobs Compute (batch processing) is the cheapest; All-Purpose Compute (interactive notebooks) costs roughly 3-4x more per DBU on AWS Premium list prices ($0.55 vs. $0.15 per DBU). The platform tier also sets the rate. The Standard tier was retired on AWS and GCP on 1 October 2025 and on Azure on 1 October 2026, so the current tiers are Premium and Enterprise on AWS and GCP, and Premium on Azure. Unity Catalog requires Premium; Enterprise adds advanced security and compliance features.

Total cost depends on workload mix, cloud, region and discounts, so there is no typical monthly figure. The biggest cost levers are workload type selection, cluster auto-termination settings, and commit-plan discounts. For a detailed breakdown, see the Databricks pricing guide. For cloud-specific cost differences, see Azure vs. AWS vs. GCP comparison.

Governance and Security

Unity Catalog is Databricks' answer to governance. It provides a centralized metastore that tracks every table, view, ML model, and file across all workspaces. Access control operates at three levels: catalog (top-level namespace), schema (database), and object (table/view/model). Row filters and column masks allow fine-grained restrictions - an analyst in EMEA sees only EMEA customer data, even though the table contains global records.

Data lineage in Unity Catalog is automatic. When a notebook reads from table A, transforms the data, and writes to table B, Unity Catalog records that dependency. This lineage graph surfaces in the UI and through APIs, making impact analysis straightforward: before changing a column in a source table, you can see the downstream tables, Databricks dashboards, jobs and notebooks that depend on it.

Security features include encryption at rest and in transit, VPC/VNet deployment for network isolation, SSO integration with Azure AD, Okta, and other identity providers, and a compliance security profile that supports compliance needs such as HIPAA, HITRUST, PCI-DSS and FedRAMP. Databricks also documents how to prepare Delta tables for GDPR requests such as the right to be forgotten. Audit logs capture every data access, query execution, and permission change.

Unity Catalog extends governance to federated sources through Lakehouse Federation, but its scope centers on the Databricks platform. Organizations running Databricks alongside Snowflake, SaaS tools, and legacy databases often add a cross-platform governance layer - which is where external catalogs like Dawiso fit.

Limitations and Trade-offs

Vendor dependency despite open formats. Delta Lake is open source and Parquet files are portable, but the Databricks runtime optimizations (Photon, adaptive query execution, Lakeflow pipelines) only work within Databricks. Moving away means losing those performance advantages and rewriting pipeline orchestration. Open format is not the same as zero switching cost.

Cost complexity. The two-bill model (DBU + cloud infrastructure) makes cost estimation difficult. A team that spins up All-Purpose clusters for interactive exploration without auto-termination policies can generate unexpectedly large bills. Snowflake bills compute credits, a per-TB storage charge and data transfer as one total; on classic compute, Databricks requires active cost management across two vendors.

Spark learning curve. Teams with deep SQL skills but no Spark experience face a ramp-up period. While Databricks SQL provides a familiar SQL interface, the full power of the platform - pipeline orchestration, streaming, ML - requires understanding distributed computing concepts. This is not an afternoon of training.

Cold-start latency on serverless. Serverless SQL warehouses eliminate cluster management and typically start in 2 to 6 seconds; pro and classic warehouses take minutes. For dashboards with infrequent usage on a pro or classic warehouse, the first query after a stop can mean a noticeable wait for business users accustomed to instant results.

How Dawiso Complements Databricks

Unity Catalog governs the Databricks platform and the sources it federates. Dawiso catalogs metadata across platforms - Databricks alongside Snowflake, BI tools and other connected sources. The two are complementary, not competitive.

Dawiso's data catalog indexes metadata from Databricks Unity Catalog alongside metadata from the other platforms Dawiso connects to, such as Snowflake, dbt and Power BI. Business users search for datasets in one place, whether the data lives in a Delta table or a Snowflake view. Lineage in Dawiso brings together lineage from the Databricks, dbt and Power BI connectors in one catalog.

The business glossary in Dawiso ensures that "revenue," "active customer," and "churn rate" mean the same thing whether the metric is computed in Databricks SQL, queried in Snowflake, or displayed in a Tableau dashboard. Without a shared glossary, different teams build conflicting definitions. Because Dawiso's glossary spans platforms, one definition applies wherever the metric is used.

Through the Model Context Protocol (MCP), AI agents can access Dawiso's catalog programmatically - searching the catalog, tracing lineage, and previewing objects and data samples through a standardized protocol. This is how data governance scales from manual processes to AI-assisted operations across the full data stack.

Conclusion

Databricks solves a real architectural problem: the data lake + warehouse split that creates duplicate data, duplicate pipelines, and governance gaps. The lakehouse pattern - Delta Lake on cloud storage with Spark for compute - eliminates that duplication. For organizations whose primary workloads combine data engineering, SQL analytics, and ML, it is the most integrated option available. The trade-offs are real - cost complexity, Spark learning curve, and vendor-specific optimizations - but for teams willing to invest in the platform, the consolidation payoff is substantial. The governance gap beyond Databricks' own boundary is where a cross-platform catalog like Dawiso becomes essential.

Sources

• Databricks - Manage your subscription
• Microsoft Azure - Azure Databricks pricing
• Microsoft Learn - SQL warehouse types
• Databricks - Where is Delta Live Tables?
• Databricks - What is Photon?
• Databricks - View data lineage using Unity Catalog
• Databricks - Row filters and column masks
• Databricks - Compliance security profile
• Databricks - Prepare your data for GDPR compliance
• Databricks - What is Lakehouse Federation?
• Snowflake - Understanding overall cost
• Forrester / Databricks - Forrester Total Economic Impact Spotlight
• Dawiso Help - Databricks connector
• Dawiso Help - Dawiso Model Context Protocol (MCP)
• Databricks - End of life for Standard tier workspaces on Google Cloud
• Databricks - Databricks pricing