Databricks Data Engineer Interview Questions

I would avoid too many small files because they slow down listing, planning, and query execution. At the same time, files should not be so large that they create skew or make updates expensive. The right size depends on the workload, but the main principle is to balance read efficiency with manageable maintenance cost.

AQE re-optimizes Spark query plans at runtime based on actual stage statistics. It dynamically coalesces shuffle partitions, converts sort-merge joins to broadcast joins, and handles skew automatically, making Spark queries much more resilient without manual tuning.

I use mapGroupsWithState or flatMapGroupsWithState with appropriate timeouts and state eviction rules to manage streaming state efficiently without unbounded state growth.

I implement expectations using Delta Live Tables or Great Expectations, routing bad data to dead-letter queues or quarantine tables while letting valid data flow through with clear alerts.

I use single-node or autoscaling clusters with spot instances for non-critical workloads, set tight auto-termination timeouts, and leverage Photon engine for heavy SQL transformations.