Microsoft Data Engineer Interview Questions

I would compare filters, deduplication logic, late-arriving records, and failed records in the reject path. Count mismatches can come from expected business rules, but they can also reveal dropped rows or incorrect joins.

I would include stable identifiers, timestamps, source metadata, and a structure that can tolerate new fields. Event schemas evolve constantly, so I would plan for versioning and explicit compatibility rules rather than assuming the format will stay fixed.

I would validate early, quarantine suspicious records, and keep quality gates between raw ingestion and curated layers. Once bad data reaches widely used tables, cleanup becomes much more expensive.

I would use a streaming or micro-batch ingestion path, store events durably, and maintain incremental aggregates that can be refreshed quickly. The design needs to handle retries, duplicate messages, and late data while keeping latency low enough to be useful.

Batch is simpler and often cheaper when latency is not critical, while streaming gives fresher data but adds operational complexity. I would choose streaming only when near-real-time visibility is a real requirement.