DP-750 — Microsoft Certified: Azure Databricks Data Engineer Associate Quick Reference

Use this Quick Reference as independent review support for Microsoft Certified: Azure Databricks Data Engineer Associate (DP-750). The exam rewards practical decisions: how to ingest data, transform it with Delta Lake, govern it with Unity Catalog, run reliable pipelines, and troubleshoot Azure Databricks workloads.

DP-750 Mental Model

Azure Databricks data engineering questions usually combine four layers:

    flowchart LR
	    A[Sources: files, databases, events, APIs] --> B[Landing storage in ADLS Gen2]
	    B --> C[Ingestion: Auto Loader, COPY INTO, Structured Streaming]
	    C --> D[Bronze Delta tables]
	    D --> E[Silver cleansing and conformance]
	    E --> F[Gold marts and aggregates]
	    F --> G[BI, ML, apps, sharing]
	    H[Unity Catalog] -. governs .-> D
	    H -. governs .-> E
	    H -. governs .-> F
	    I[Jobs / Workflows / Pipelines] -. orchestrate .-> C
	    I -. orchestrate .-> E
	    I -. orchestrate .-> F

Core Azure Databricks Object Map

Medallion Architecture Reference

High-yield distinctions:

Feature Selection Matrix

Ingestion Quick Reference

Batch and File Ingestion

COPY INTO Pattern

COPY INTO prod.bronze.orders_raw
FROM 'abfss://landing@storageacct.dfs.core.windows.net/orders/'
FILEFORMAT = JSON
FORMAT_OPTIONS ('inferSchema' = 'true')
COPY_OPTIONS ('mergeSchema' = 'true');

Exam cues:

Auto Loader Pattern

raw_path = "abfss://landing@storageacct.dfs.core.windows.net/orders/"
schema_path = "abfss://checkpoint@storageacct.dfs.core.windows.net/schemas/orders/"
checkpoint_path = "abfss://checkpoint@storageacct.dfs.core.windows.net/checkpoints/orders/"

(
    spark.readStream
    .format("cloudFiles")
    .option("cloudFiles.format", "json")
    .option("cloudFiles.schemaLocation", schema_path)
    .load(raw_path)
    .writeStream
    .option("checkpointLocation", checkpoint_path)
    .trigger(availableNow=True)
    .toTable("prod.bronze.orders_raw")
)

Auto Loader exam traps:

Streaming Ingestion and Processing

from pyspark.sql.functions import col

updates = (
    spark.readStream.table("prod.bronze.orders_raw")
    .filter(col("order_id").isNotNull())
)

def upsert_orders(batch_df, batch_id):
    batch_df.createOrReplaceTempView("orders_updates")
    spark.sql("""
        MERGE INTO prod.silver.orders AS t
        USING orders_updates AS s
        ON t.order_id = s.order_id
        WHEN MATCHED THEN UPDATE SET *
        WHEN NOT MATCHED THEN INSERT *
    """)

(
    updates.writeStream
    .foreachBatch(upsert_orders)
    .option("checkpointLocation", "abfss://checkpoint@storageacct.dfs.core.windows.net/checkpoints/orders_merge/")
    .start()
)

Delta Lake Operations Reference

MERGE Pattern

MERGE INTO prod.silver.customers AS target
USING prod.bronze.customers_updates AS source
ON target.customer_id = source.customer_id
WHEN MATCHED AND source.operation = 'DELETE' THEN DELETE
WHEN MATCHED THEN UPDATE SET *
WHEN NOT MATCHED THEN INSERT *;

MERGE traps:

Time Travel and History

DESCRIBE HISTORY prod.silver.orders;

SELECT *
FROM prod.silver.orders VERSION AS OF 12;

RESTORE TABLE prod.silver.orders TO VERSION AS OF 12;

Change Data Feed

ALTER TABLE prod.silver.orders
SET TBLPROPERTIES (delta.enableChangeDataFeed = true);

SELECT *
FROM table_changes('prod.silver.orders', 10);

Schema, Tables, and Storage Decisions

Managed vs External Tables

Schema Enforcement and Evolution

Partitioning, Clustering, and File Layout

Lakeflow Declarative Pipelines / Delta Live Tables

Microsoft and Databricks materials may reference Delta Live Tables (DLT) and newer Lakeflow Declarative Pipelines terminology. For exam purposes, focus on the concepts: declarative tables, dependencies, streaming tables, data quality expectations, pipeline monitoring, and managed execution.

Example DLT-style SQL:

CREATE OR REFRESH STREAMING LIVE TABLE bronze_orders
AS
SELECT *
FROM cloud_files(
  "abfss://landing@storageacct.dfs.core.windows.net/orders/",
  "json"
);

CREATE OR REFRESH LIVE TABLE silver_orders
(
  CONSTRAINT valid_order_id EXPECT (order_id IS NOT NULL) ON VIOLATION DROP ROW,
  CONSTRAINT valid_amount EXPECT (amount >= 0) ON VIOLATION DROP ROW
)
AS
SELECT
  order_id,
  customer_id,
  CAST(amount AS DECIMAL(18,2)) AS amount,
  CAST(order_timestamp AS TIMESTAMP) AS order_timestamp
FROM STREAM(LIVE.bronze_orders);

Expectation actions:

DLT / Lakeflow traps:

Unity Catalog and Governance

Namespace and Securable Hierarchy

Use three-level names for clarity:

SELECT *
FROM prod.silver.orders;

Common Unity Catalog Privileges

Basic grant pattern:

GRANT USE CATALOG ON CATALOG prod TO `data-engineers`;
GRANT USE SCHEMA ON SCHEMA prod.silver TO `data-engineers`;
GRANT SELECT ON TABLE prod.silver.orders TO `analysts`;
GRANT SELECT, MODIFY ON TABLE prod.silver.orders TO `etl-service-principal`;

High-yield permission rule: object-level access is not enough if the principal lacks USE CATALOG and USE SCHEMA.

External Locations and Volumes

Example external location pattern:

CREATE EXTERNAL LOCATION raw_orders_location
URL 'abfss://landing@storageacct.dfs.core.windows.net/orders/'
WITH (STORAGE CREDENTIAL adls_storage_credential);

GRANT READ FILES ON EXTERNAL LOCATION raw_orders_location TO `ingestion-engineers`;

Security Decision Table

Unity Catalog Traps

Compute Selection

Access Modes

Cluster Configuration Cues

Jobs, Workflows, and Deployment

Production readiness checklist:

• Use service principals or managed identities for scheduled workloads.
• Parameterize catalog, schema, storage path, and processing date.
• Separate development and production data namespaces.
• Store secrets outside code.
• Use job clusters or governed compute policies.
• Define retry behavior and failure notifications.
• Log row counts, rejected records, and important pipeline metrics.
• Avoid relying on an interactive user’s cluster, credentials, or notebook state.

Performance and Optimization Reference

Table and Query Tuning

Spark Execution Concepts

Optimization Traps

Data Quality and Reliability

Useful metadata columns:

CDC and Slowly Changing Dimensions

SCD Type 2 cues:

Monitoring and Troubleshooting

Where to Look

Common Error Patterns

Triage Order

1. Identify whether the failure is permissions, code, data, compute, or orchestration.
2. Check the job or pipeline run output.
3. Inspect table names, catalog/schema context, and identity used by the run.
4. Validate source paths, external locations, and grants.
5. Review schema changes and data quality expectation failures.
6. Use Spark UI or SQL profile for performance bottlenecks.
7. Fix idempotency before rerunning failed writes.

Azure-Specific Integration Points

SQL and PySpark Syntax Mini-Reference

High-Yield Exam Traps Checklist

• Do not choose legacy mounts when the scenario emphasizes Unity Catalog governance.
• Do not grant SELECT only and forget USE CATALOG and USE SCHEMA.
• Do not use an all-purpose interactive cluster as the default production answer.
• Do not reset or share streaming checkpoints without understanding replay and duplicates.
• Do not use direct file reads when the requirement is incremental file discovery.
• Do not use COPY INTO for continuous event streams.
• Do not use MERGE without a stable key and deduplicated source.
• Do not overpartition high-cardinality columns.
• Do not run VACUUM casually when rollback, time travel, or lagging streams matter.
• Do not store production data in DBFS root as a governance strategy.
• Do not assume a notebook user’s permissions are the same as the job’s service identity.
• Do not hide all data quality logic downstream in BI; validate in silver/pipeline layers.
• Do not choose Python UDFs for simple transformations that Spark SQL functions can handle.
• Do not ignore pipeline event logs, job task logs, Spark UI, and SQL query profiles during troubleshooting.

Last-Mile Practice Plan

Practice DP-750 scenarios by forcing yourself to choose: ingestion pattern, Delta table design, Unity Catalog permissions, compute type, orchestration method, and troubleshooting path. Then implement small end-to-end exercises: Auto Loader to bronze, MERGE to silver, aggregate to gold, secure with Unity Catalog grants, schedule as a job, and diagnose one intentional failure.