Databricks Data Engineer Interview Questions

Schema evolution lets a table adapt when new columns appear or compatible changes are introduced. I would allow it when the upstream change is expected and controlled, but I would not use it as a substitute for data contracts because uncontrolled schema drift can hide quality problems. The safe approach is to make schema changes explicit when possible.

I would keep Bronze as the raw ingestion layer, Silver as the cleaned and conformed layer, and Gold as the business-ready layer. This separation makes quality checks, lineage, and reprocessing much easier because each stage has a clear responsibility. It also helps teams change transformations without rewriting the ingestion path.

I would check whether the join keys are selective, whether the source is too large, and whether the target table layout is helping or hurting the merge. Often the issue is excessive file rewriting, poor partitioning, or a lack of predicate pushdown. I would also confirm that the merge condition is as narrow as possible so the engine touches fewer files.

OPTIMIZE compacts small files into larger ones, which reduces file listing and metadata overhead. Z-ORDER clusters related values together so queries with selective filters can skip more data. I would use them when table layout is hurting query performance, but I would avoid overusing them because they cost compute and are not free maintenance.

Unity Catalog provides centralized governance for data, metadata, and access control across workspaces. It helps manage permissions consistently, supports fine-grained access control, and improves visibility into lineage and ownership. In a multi-team environment, it is especially useful because governance stops being scattered across ad hoc workspace settings.