Top 30 Databricks Scenario-Based Interview Questions and Answers — 2026

1. Why Is the Bronze Layer Immutable in Databricks?

The Bronze layer acts as the raw system of record. Data is generally preserved exactly as it arrives so it can be replayed later if Silver or Gold transformations contain defects.

Key benefits include:

• Reproducibility
• Auditing and forensic analysis
• Disaster recovery
• Source-system decoupling
• Historical replay
• Pipeline debugging

Typical metadata added at ingestion includes:

_source_file
_ingestion_timestamp
_batch_id
_source_system

_source_file
_ingestion_timestamp
_batch_id
_source_system

Cleaning and deduplication usually happen in Silver rather than altering the raw Bronze data.

2. How Do You Handle Duplicate and Partial Files in Databricks?

I normally preserve source records in Bronze while attaching metadata such as file name and ingestion timestamp.

During Bronze-to-Silver processing, duplicates can be removed using:

ROW_NUMBER() OVER (
  PARTITION BY business_key
  ORDER BY modified_timestamp DESC,
           ingestion_timestamp DESC
)

ROW_NUMBER() OVER (
  PARTITION BY business_key
  ORDER BY modified_timestamp DESC,
           ingestion_timestamp DESC
)

For partial or corrupt files, I validate the file before promoting data to Silver.

Invalid records or files can be moved into:

Quarantine / Error Layer

Quarantine / Error Layer

The main design objective is idempotency: rerunning a pipeline should not create duplicate business records.

3. What Transformations Are Typically Performed from Bronze to Silver?

Bronze-to-Silver transformations usually include:

• Schema validation
• Data-type conversion
• NULL handling
• Deduplication
• Standardization
• Data-quality validation
• Reference-data enrichment
• Timestamp normalization
• CDC processing
• Invalid-record quarantine

For example:

Raw Customer
     ↓
Validate Schema
     ↓
Standardize Fields
     ↓
Remove Duplicates
     ↓
Validate Business Rules
     ↓
Silver Customer

Raw Customer
     ↓
Validate Schema
     ↓
Standardize Fields
     ↓
Remove Duplicates
     ↓
Validate Business Rules
     ↓
Silver Customer

Silver should represent trustworthy, reusable enterprise data.

4. How Do You Handle Schema Evolution in Databricks?

I first classify schema changes as either compatible or breaking.

Compatible example:

New nullable column added

New nullable column added

Breaking examples:

INTEGER → STRING
Column renamed
Column deleted
Nested structure changed

INTEGER → STRING
Column renamed
Column deleted
Nested structure changed

Bronze should generally tolerate schema changes so raw data remains available.

Silver should enforce stronger contracts.

For compatible Delta Lake changes, schema evolution can be enabled when appropriate.

For breaking changes, I prefer:

Detect
→ Alert
→ Quarantine
→ Review
→ Update Contract
→ Reprocess

Detect
→ Alert
→ Quarantine
→ Review
→ Update Contract
→ Reprocess

Schema changes should be observable rather than silently accepted.

5. How Would You Design a Multi-Region Databricks Medallion Architecture?

I would keep raw ingestion close to each source region.

For example:

US Sources   → US Bronze
EU Sources   → EU Bronze
APAC Sources → APAC Bronze

US Sources   → US Bronze
EU Sources   → EU Bronze
APAC Sources → APAC Bronze

Depending on residency and governance requirements, curated datasets could then be replicated, shared, or consolidated.

Important considerations include:

• Data sovereignty
• Cross-region latency
• Replication
• RTO/RPO
• Egress costs
• Disaster recovery
• Governance
• Catalog consistency

Unity Catalog or another governance layer can provide centralized discovery and security while physical data remains distributed.

6. Where Do You Use Upserts in Databricks?

Upserts are commonly implemented using Delta Lake MERGE.

Typical scenarios include:

• Bronze-to-Silver incremental processing
• CDC pipelines
• SCD Type 1
• SCD Type 2
• Customer master updates
• Gold aggregate refreshes
• Streaming micro-batches

For example:

MERGE INTO customer_target t
USING customer_updates s
ON t.customer_id = s.customer_id
WHEN MATCHED THEN
  UPDATE SET *
WHEN NOT MATCHED THEN
  INSERT *

MERGE INTO customer_target t
USING customer_updates s
ON t.customer_id = s.customer_id
WHEN MATCHED THEN
  UPDATE SET *
WHEN NOT MATCHED THEN
  INSERT *

The main benefit is idempotent incremental processing.

7. Yesterday’s Pipeline Partially Wrote to Silver and Then Failed. What Would You Do Before Rerunning?

First, I would determine whether any partial data was committed.

I would inspect:

• Delta table history
• Batch IDs
• Checkpoints
• Watermarks
• Job logs

If the failed batch committed invalid partial data, I could:

• Remove the affected batch
• Restore a previous Delta version
• Reprocess from Bronze

Conceptually:

Identify last good state
        ↓
Remove/revert bad write
        ↓
Restart from checkpoint
        ↓
Reconcile Bronze vs Silver

Identify last good state
        ↓
Remove/revert bad write
        ↓
Restart from checkpoint
        ↓
Reconcile Bronze vs Silver

The pipeline should be designed so restarting it does not create duplicates.

8. When Would You Ingest Directly from an RDBMS into Silver?

I would generally prefer keeping a Bronze layer.

Direct-to-Silver ingestion may occasionally be considered when:

• Data is already curated
• Volume is very small
• Historical replay is unnecessary
• The workload is non-critical

However, skipping Bronze removes important capabilities such as:

• Raw audit history
• Easy replay
• Schema-evolution buffering
• Source-independent recovery
• Original-data evidence

For enterprise pipelines, maintaining Bronze is normally safer.

9. Multiple Rows Have the Same Modified Timestamp. How Do You Avoid Missing Records?

Using only:

last_timestamp” data-toggle-ms=”1200″ aria-label=”Copy code” data-nosnippet=”” class=”code-copy-btn absolute right-3 top-1/2 -translate-y-1/2 size-8 p-1.5 flex items-center justify-center bg-black/50 text-white rounded-md transition-colors duration-200 cursor-pointer hover:bg-black/70″>

WHERE modified_timestamp > last_timestamp

WHERE modified_timestamp > last_timestamp

can miss records at watermark boundaries.

I would use a combination of:

• Overlap windows
• Business keys
• Ingestion timestamps
• MERGE-based deduplication

For example:

Previous watermark: 10:00:00
Next extraction:
Start from 09:59:55

Previous watermark: 10:00:00
Next extraction:
Start from 09:59:55

The overlapping records are then safely deduplicated in Silver.

Another approach is a compound watermark:

(modified_timestamp, primary_key)

(modified_timestamp, primary_key)

This creates deterministic incremental extraction.

10. JDBC Incremental Reads Are Slow. How Would You Improve Performance?

I would avoid single-threaded large-table JDBC reads.

For Spark JDBC ingestion, I would consider:

partitionColumn
lowerBound
upperBound
numPartitions

partitionColumn
lowerBound
upperBound
numPartitions

For example:

100 million rows
÷
10 JDBC partitions
=
approximately 10 million rows per parallel reader

100 million rows
÷
10 JDBC partitions
=
approximately 10 million rows per parallel reader

I would also:

• Push filters to the source
• Select only required columns
• Use incremental watermarks
• Ensure indexes exist
• Tune fetch size
• Avoid full-table extraction
• Consider native CDC for very large transactional systems

The objective is to minimize pressure on both Databricks and the operational source.

11. How Do You Implement CDC in Databricks?

A common approach is using Delta Lake change capabilities or an external CDC platform.

An external architecture might look like:

PostgreSQL
   ↓
Debezium
   ↓
Kafka/Event Hubs
   ↓
Databricks Structured Streaming
   ↓
Bronze
   ↓
MERGE
   ↓
Silver

PostgreSQL
   ↓
Debezium
   ↓
Kafka/Event Hubs
   ↓
Databricks Structured Streaming
   ↓
Bronze
   ↓
MERGE
   ↓
Silver

CDC events normally contain operation types such as:

INSERT
UPDATE
DELETE

INSERT
UPDATE
DELETE

Silver processing applies these changes deterministically using business keys and sequence information.

Additional Production Databricks Interview Questions

12. What Is the Databricks Medallion Architecture?

Medallion Architecture organizes data into progressively refined layers.

Bronze

Raw source-aligned data.

Silver

Cleaned, deduplicated, standardized, validated data.

Gold

Business-ready data products, KPIs, dimensions, aggregates, and reporting datasets.

Architecture:

Sources
   ↓
Bronze
   ↓
Silver
   ↓
Gold
   ↓
BI / ML / AI / APIs

Sources
   ↓
Bronze
   ↓
Silver
   ↓
Gold
   ↓
BI / ML / AI / APIs

This separation improves maintainability, lineage, recovery, and data quality.

13. What Is Delta Lake?

Delta Lake adds transactional capabilities to data stored in a data lake.

Important features include:

• ACID transactions
• Schema enforcement
• Schema evolution
• Time Travel
• MERGE
• DELETE
• UPDATE
• Transaction history
• Streaming and batch interoperability

It allows data lakes to support many capabilities traditionally associated with database systems.

14. What Is Delta Lake Time Travel?

Time Travel allows users to access previous versions of Delta tables.

For example:

SELECT *
FROM customer_table
VERSION AS OF 125;

SELECT *
FROM customer_table
VERSION AS OF 125;

or conceptually by timestamp.

Common use cases include:

• Debugging
• Auditing
• Comparing versions
• Recovering from bad writes
• Reproducing historical results

It is extremely useful for production incident recovery.

15. What Is Auto Loader in Databricks?

Auto Loader is designed for scalable incremental ingestion of files arriving in cloud object storage.

Typical sources include:

• ADLS
• Amazon S3
• Google Cloud Storage

Conceptually:

Cloud Storage
     ↓
Auto Loader
     ↓
Structured Streaming
     ↓
Bronze Delta

Cloud Storage
     ↓
Auto Loader
     ↓
Structured Streaming
     ↓
Bronze Delta

It is useful when thousands or millions of files continuously arrive because it maintains ingestion state rather than repeatedly rescanning everything.

16. What Is Structured Streaming in Databricks?

Structured Streaming is Spark’s streaming processing framework.

Developers express transformations similarly to batch DataFrame operations while Spark handles incremental execution.

Architecture:

Kafka/Event Hubs
        ↓
Structured Streaming
        ↓
Transformations
        ↓
Delta Lake

Kafka/Event Hubs
        ↓
Structured Streaming
        ↓
Transformations
        ↓
Delta Lake

Important concepts include:

• Checkpoints
• Watermarks
• Micro-batches
• Stateful operations
• Event-time processing

17. What Is a Checkpoint in Structured Streaming?

A checkpoint stores information required for stream recovery.

It can contain:

• Processed offsets
• Streaming progress
• State information
• Metadata

If the streaming job restarts, the checkpoint enables it to continue processing rather than starting blindly from the beginning.

Checkpoint locations should therefore be durable and unique to the streaming query.

18. What Is a Watermark in Spark Structured Streaming?

A watermark helps manage late-arriving events and state.

Suppose:

Current event time = 10:00 AM
Watermark = 15 minutes

Current event time = 10:00 AM
Watermark = 15 minutes

Spark can tolerate events arriving within the configured delay while eventually cleaning old state.

Watermarks are particularly important for:

• Windowed aggregations
• Stream-stream joins
• Deduplication

They balance late-data tolerance against state growth.

19. How Do You Handle Late-Arriving Data?

I would first define the business tolerance for lateness.

For streaming:

Event time
+
Watermark
+
Deduplication

Event time
+
Watermark
+
Deduplication

For batch pipelines, I may reprocess a configurable lookback window.

For example:

Daily job processes:
today
+
previous 2 days

Daily job processes:
today
+
previous 2 days

Then MERGE logic ensures that reprocessed records do not create duplicates.

20. What Is Unity Catalog?

Unity Catalog provides centralized governance for Databricks data and AI assets.

It helps manage:

• Catalogs
• Schemas
• Tables
• Views
• Functions
• Models
• Volumes
• Permissions
• Lineage

A typical hierarchy is:

Metastore
   ↓
Catalog
   ↓
Schema
   ↓
Table / View

Metastore
   ↓
Catalog
   ↓
Schema
   ↓
Table / View

It allows organizations to enforce governance more consistently across workspaces.

21. How Do You Secure Sensitive Data in Databricks?

I would apply several layers of controls.

These may include:

• Unity Catalog permissions
• Role/group-based access
• Row filtering
• Column masking
• Secrets management
• Network controls
• Audit logs
• Data classification

The design should follow least privilege.

For example:

Administrator → Full customer record
Analyst       → Masked SSN
Regional User → Only assigned region

Administrator → Full customer record
Analyst       → Masked SSN
Regional User → Only assigned region

22. How Do You Optimize a Slow Spark Job?

I first use the Spark UI to determine where time is being spent.

I look for:

• Large shuffles
• Data ske
• Spilling
• Long stages
• Too many small tasks
• Uneven task duration
• Expensive joins

Then I may optimize:

• Partitioning
• Join strategy
• Filtering
• File layout
• Data skipping
• Query logic
• Cluster sizing

Scaling compute should come after identifying the actual bottleneck.

23. What Is Data Skew and How Do You Handle It?

Data skew occurs when some partitions contain significantly more data than others.

For example:

Partition 1 = 1 GB
Partition 2 = 1 GB
Partition 3 = 1 GB
Partition 4 = 80 GB

Partition 1 = 1 GB
Partition 2 = 1 GB
Partition 3 = 1 GB
Partition 4 = 80 GB

One executor may become the bottleneck.

Potential techniques include:

• Repartitioning
• Salting skewed keys
• Broadcast joins
• Pre-aggregation
• Adaptive Query Execution

The Spark UI usually makes skew visible through unusually long-running tasks.

24. When Would You Use a Broadcast Join?

A broadcast join is useful when one side of a join is small enough to distribute to workers.

Instead of shuffling both datasets:

Large Table
    +
Small Lookup Table

Large Table
    +
Small Lookup Table

Spark broadcasts the small table to executors.

This can dramatically reduce shuffle operations.

However, broadcasting a table that is too large can cause memory pressure, so it should be used carefully.

25. Why Are Too Many Small Files a Problem?

Millions of tiny files increase:

• File-listing overhead
• Metadata operations
• Task scheduling overhead
• Query startup time
• Storage operations

For example:

1 TB in 1,000 files

1 TB in 1,000 files

is typically easier to process efficiently than:

1 TB in 10 million tiny files

1 TB in 10 million tiny files

I would monitor file sizes and compact small files when appropriate.

26. How Would You Optimize a Large Delta Table?

I would first understand the query access pattern.

Possible optimization areas include:

• Reducing small files
• Improving data layout
• Eliminating unnecessary partitions
• Filtering early
• Selecting only required columns
• Appropriate clustering/layout techniques
• Avoiding excessive full-table scans

The correct optimization depends on whether users typically query by:

date
customer_id
region
account
event_type

date
customer_id
region
account
event_type

I would measure the effect using query profiles rather than optimize blindly.

27. How Would You Implement SCD Type 2 in Databricks?

I would maintain historical versions of dimensional records.

Example:

customer_id
customer_name
status
start_date
end_date
is_current

customer_id
customer_name
status
start_date
end_date
is_current

When a customer attribute changes:

Current row
   ↓
Expire current row
   ↓
Insert new row

Current row
   ↓
Expire current row
   ↓
Insert new row

Example:

101 | Gold     | Jan 1 | Aug 31 | false
101 | Platinum | Sep 1 | NULL   | true

101 | Gold     | Jan 1 | Aug 31 | false
101 | Platinum | Sep 1 | NULL   | true

Delta MERGE and deterministic change detection are often used as part of the implementation.

28. What Is the Difference Between Repartition and Coalesce?

repartition() can increase or decrease partitions and typically involves a shuffle.

Use it when you need more evenly distributed data.

coalesce() is commonly used to reduce the number of partitions with less movement.

Conceptually:

repartition()
→ more expensive
→ better redistribution
coalesce()
→ cheaper
→ mainly reduce partitions

repartition()
→ more expensive
→ better redistribution
coalesce()
→ cheaper
→ mainly reduce partitions

The correct choice depends on the downstream operation and distribution of the data.

29. How Would You Monitor a Production Databricks Pipeline?

I would monitor both technical health and data health.

Technical metrics:

Job duration
Failure rate
Cluster utilization
Shuffle
Spill
Streaming lag
Batch duration

Job duration
Failure rate
Cluster utilization
Shuffle
Spill
Streaming lag
Batch duration

Data metrics:

Rows processed
Rejected rows
Duplicate rate
NULL rate
Data freshness
Source-to-target reconciliation

Rows processed
Rejected rows
Duplicate rate
NULL rate
Data freshness
Source-to-target reconciliation

Operational alerts should answer:

What failed?
Where?
Which batch?
How much data was affected?
Can it be safely replayed?

What failed?
Where?
Which batch?
How much data was affected?
Can it be safely replayed?

Observability is essential for production lakehouse engineering.

30. How Would You Design an Enterprise Databricks Lakehouse Platform?

A production architecture could look like:

SOURCES
│
├─ Databases
├─ APIs
├─ SaaS
├─ Files
└─ Kafka/Event Hubs
       │
       ▼
INGESTION
Auto Loader | CDC | JDBC | Streaming
       │
       ▼
BRONZE
Raw + Immutable
       │
       ▼
DATA QUALITY
Validation + Quarantine
       │
       ▼
SILVER
Clean + Standardized + Deduplicated
       │
       ▼
GOLD
KPIs + Facts + Dimensions + Data Products
       │
 ┌─────┼────────┐
 ▼     ▼        ▼
BI     ML    GenAI/RAG

SOURCES
│
├─ Databases
├─ APIs
├─ SaaS
├─ Files
└─ Kafka/Event Hubs
       │
       ▼
INGESTION
Auto Loader | CDC | JDBC | Streaming
       │
       ▼
BRONZE
Raw + Immutable
       │
       ▼
DATA QUALITY
Validation + Quarantine
       │
       ▼
SILVER
Clean + Standardized + Deduplicated
       │
       ▼
GOLD
KPIs + Facts + Dimensions + Data Products
       │
 ┌─────┼────────┐
 ▼     ▼        ▼
BI     ML    GenAI/RAG

Across the platform I would implement:

Governance

Unity Catalog, RBAC, lineage, masking, and auditing.

Data Quality

Completeness, uniqueness, validity, timeliness, and reconciliation.

Orchestration

Databricks Workflows or enterprise orchestration tools.

Observability

Pipeline metrics, streaming lag, failures, quality checks, and SLA monitoring.

CI/CD

Version-controlled notebooks/code, automated testing, environment promotion, and rollback.

Cost Management

Appropriate compute selection, autoscaling, job clusters, serverless capabilities where appropriate, and workload monitoring.

The goal is not simply to run Spark jobs. It is to create a lakehouse platform that is:

Scalable, governed, recoverable, observable, idempotent, secure, and cost-efficient.