DP-750 — Microsoft Certified: Azure Databricks Data Engineer Associate Cheat Sheet

DP-750 preparation should emphasize practical data engineering decisions in Azure Databricks: choosing ingestion patterns, designing Delta Lake tables, building reliable pipelines, applying Unity Catalog governance, troubleshooting jobs, and optimizing performance and cost.

Use this page as a final concept pass, not as a substitute for hands-on practice. The exam is scenario-driven: you need to recognize the best Databricks feature, security boundary, or pipeline pattern from the wording of the question.

    flowchart LR
	    A[Sources: files, databases, events, APIs] --> B[Landing storage in ADLS Gen2]
	    B --> C[Ingestion: Auto Loader, COPY INTO, Structured Streaming]
	    C --> D[Bronze Delta tables]
	    D --> E[Silver cleansing and conformance]
	    E --> F[Gold marts and aggregates]
	    F --> G[BI, ML, apps, sharing]
	    H[Unity Catalog] -. governs .-> D
	    H -. governs .-> E
	    H -. governs .-> F
	    I[Jobs / Workflows / Pipelines] -. orchestrate .-> C
	    I -. orchestrate .-> E
	    I -. orchestrate .-> F

Core Azure Databricks mental model

Azure Databricks data engineering usually follows a lakehouse pattern:

    flowchart LR
	    A[Source systems] --> B[Landing / raw files]
	    B --> C[Bronze Delta tables]
	    C --> D[Silver Delta tables]
	    D --> E[Gold Delta tables]
	    E --> F[BI, ML, apps, downstream jobs]
	
	    G[Unity Catalog] -.governs.-> C
	    G -.governs.-> D
	    G -.governs.-> E
	    H[Jobs / workflows / pipelines] --> C
	    H --> D
	    H --> E

Medallion architecture review

Common mistake: putting complex business transformations directly into bronze. Bronze should support replay and traceability. Silver and gold should carry most cleaning, conforming, and serving logic.

High-yield distinctions:

COPY INTO Pattern

COPY INTO prod.bronze.orders_raw
FROM 'abfss://landing@storageacct.dfs.core.windows.net/orders/'
FILEFORMAT = JSON
FORMAT_OPTIONS ('inferSchema' = 'true')
COPY_OPTIONS ('mergeSchema' = 'true');

Exam cues:

Auto Loader Pattern

raw_path = "abfss://landing@storageacct.dfs.core.windows.net/orders/"
schema_path = "abfss://checkpoint@storageacct.dfs.core.windows.net/schemas/orders/"
checkpoint_path = "abfss://checkpoint@storageacct.dfs.core.windows.net/checkpoints/orders/"

(
    spark.readStream
    .format("cloudFiles")
    .option("cloudFiles.format", "json")
    .option("cloudFiles.schemaLocation", schema_path)
    .load(raw_path)
    .writeStream
    .option("checkpointLocation", checkpoint_path)
    .trigger(availableNow=True)
    .toTable("prod.bronze.orders_raw")
)

Auto Loader exam traps:

Streaming Ingestion and Processing

from pyspark.sql.functions import col

updates = (
    spark.readStream.table("prod.bronze.orders_raw")
    .filter(col("order_id").isNotNull())
)

def upsert_orders(batch_df, batch_id):
    batch_df.createOrReplaceTempView("orders_updates")
    spark.sql("""
        MERGE INTO prod.silver.orders AS t
        USING orders_updates AS s
        ON t.order_id = s.order_id
        WHEN MATCHED THEN UPDATE SET *
        WHEN NOT MATCHED THEN INSERT *
    """)

(
    updates.writeStream
    .foreachBatch(upsert_orders)
    .option("checkpointLocation", "abfss://checkpoint@storageacct.dfs.core.windows.net/checkpoints/orders_merge/")
    .start()
)

MERGE Pattern

MERGE INTO prod.silver.customers AS target
USING prod.bronze.customers_updates AS source
ON target.customer_id = source.customer_id
WHEN MATCHED AND source.operation = 'DELETE' THEN DELETE
WHEN MATCHED THEN UPDATE SET *
WHEN NOT MATCHED THEN INSERT *;

MERGE traps:

Time Travel and History

DESCRIBE HISTORY prod.silver.orders;

SELECT *
FROM prod.silver.orders VERSION AS OF 12;

RESTORE TABLE prod.silver.orders TO VERSION AS OF 12;

Change Data Feed

ALTER TABLE prod.silver.orders
SET TBLPROPERTIES (delta.enableChangeDataFeed = true);

SELECT *
FROM table_changes('prod.silver.orders', 10);

Delta Lake essentials

Delta Lake is central to DP-750 because it provides transactional reliability on cloud object storage.

Delta table choices

MERGE pattern review

Use MERGE when you need deterministic row-level changes into a Delta table.

Common mistake: using MERGE without a stable key. If the match condition is not deterministic, the pipeline may produce duplicates or ambiguous updates.

Schema Enforcement and Evolution

Partitioning, Clustering, and File Layout

Example DLT-style SQL:

CREATE OR REFRESH STREAMING LIVE TABLE bronze_orders
AS
SELECT *
FROM cloud_files(
  "abfss://landing@storageacct.dfs.core.windows.net/orders/",
  "json"
);

CREATE OR REFRESH LIVE TABLE silver_orders
(
  CONSTRAINT valid_order_id EXPECT (order_id IS NOT NULL) ON VIOLATION DROP ROW,
  CONSTRAINT valid_amount EXPECT (amount >= 0) ON VIOLATION DROP ROW
)
AS
SELECT
  order_id,
  customer_id,
  CAST(amount AS DECIMAL(18,2)) AS amount,
  CAST(order_timestamp AS TIMESTAMP) AS order_timestamp
FROM STREAM(LIVE.bronze_orders);

Expectation actions:

DLT / Lakeflow traps:

Use three-level names for clarity:

SELECT *
FROM prod.silver.orders;

Common Unity Catalog Privileges

Basic grant pattern:

GRANT USE CATALOG ON CATALOG prod TO `data-engineers`;
GRANT USE SCHEMA ON SCHEMA prod.silver TO `data-engineers`;
GRANT SELECT ON TABLE prod.silver.orders TO `analysts`;
GRANT SELECT, MODIFY ON TABLE prod.silver.orders TO `etl-service-principal`;

High-yield permission rule: object-level access is not enough if the principal lacks USE CATALOG and USE SCHEMA.

External Locations and Volumes

Example external location pattern:

CREATE EXTERNAL LOCATION raw_orders_location
URL 'abfss://landing@storageacct.dfs.core.windows.net/orders/'
WITH (STORAGE CREDENTIAL adls_storage_credential);

GRANT READ FILES ON EXTERNAL LOCATION raw_orders_location TO `ingestion-engineers`;

Security Decision Table

Unity Catalog Traps

Unity Catalog and governance

Unity Catalog is the central governance model for Databricks data and AI assets. For DP-750, focus on hierarchy, permissions, external access, and least privilege.

Unity Catalog hierarchy

Governance decision rules

Common Unity Catalog traps

• Granting Azure storage permissions but not granting Unity Catalog object privileges.
• Granting Unity Catalog privileges but forgetting the external location/storage credential setup.
• Using legacy workspace-local patterns when the scenario asks for centralized governance.
• Hard-coding storage account keys in notebooks.
• Giving users direct broad access to raw storage instead of governed tables or volumes.
• Assigning permissions to individual users instead of groups.
• Forgetting that production jobs should not rely on a developer’s personal identity.

Access Modes

Cluster Configuration Cues

Production readiness checklist:

• Use service principals or managed identities for scheduled workloads.
• Parameterize catalog, schema, storage path, and processing date.
• Separate development and production data namespaces.
• Store secrets outside code.
• Use job clusters or governed compute policies.
• Define retry behavior and failure notifications.
• Log row counts, rejected records, and important pipeline metrics.
• Avoid relying on an interactive user’s cluster, credentials, or notebook state.

Spark Execution Concepts

Optimization Traps

Table and file layout

Spark execution

Compute choices

Cost trap: an inefficient full recompute on a very large table is usually worse than a slightly more complex incremental design.

Data quality and expectations

Data quality questions usually ask how to detect, drop, fail, quarantine, or report bad records.

Common mistake: silently dropping records without auditability. If the scenario emphasizes compliance, traceability, or reconciliation, keep rejected data and quality metrics.

Common Error Patterns

Triage Order

1. Identify whether the failure is permissions, code, data, compute, or orchestration.
2. Check the job or pipeline run output.
3. Inspect table names, catalog/schema context, and identity used by the run.
4. Validate source paths, external locations, and grants.
5. Review schema changes and data quality expectation failures.
6. Use Spark UI or SQL profile for performance bottlenecks.
7. Fix idempotency before rerunning failed writes.

Monitoring and troubleshooting

Troubleshooting sequence

1. Identify whether the issue is data, code, compute, permissions, or orchestration.
2. Check the earliest failing task, not only the final downstream failure.
3. Review table history and recent schema or data changes.
4. Confirm the job identity has the correct Unity Catalog and storage permissions.
5. Inspect Spark UI or query profile for shuffle, skew, spills, and scan volume.
6. Validate checkpoint and incremental state for streaming or Auto Loader workloads.
7. Re-run safely only if the write path is idempotent.

Final review checklist

Before your next study session, confirm you can:

• Map a source system to the right ingestion pattern.
• Design bronze, silver, and gold Delta tables.
• Apply MERGE, CDF, time travel, OPTIMIZE, and VACUUM appropriately.
• Explain checkpointing and watermarks for streaming workloads.
• Configure jobs with task dependencies, parameters, retries, and alerts.
• Separate development, test, and production concerns.
• Use Unity Catalog for governed tables, views, volumes, and external locations.
• Distinguish Azure permissions from Databricks data permissions.
• Troubleshoot failures using logs, run history, Spark UI, and table history.
• Improve performance without making governance or reliability worse.

Next step: start a focused DP-750 question bank session with topic drills on your weakest area, then review the detailed explanations until each design choice feels automatic.

Ingestion tools at a glance

Ingestion mistakes to avoid

• Using a temporary checkpoint path for a production stream.
• Reusing one checkpoint for multiple unrelated streaming queries.
• Resetting checkpoints without understanding duplicate or replay impact.
• Overwriting bronze data when append-plus-replay would be safer.
• Ignoring schema drift until silver or gold jobs fail.
• Loading files repeatedly because file tracking or idempotent keys were not designed.
• Choosing streaming just because data is periodic; scheduled incremental batch may be simpler.

Batch deduplication patterns

Slowly changing dimensions

Common mistake: using Type 1 logic when the requirement says “preserve history,” “point-in-time reporting,” or “track changes over time.”

Job design checklist

A production-ready job should usually have:

1. A clear owner and run identity.
2. Parameterized paths, table names, and environment settings.
3. A controlled compute choice.
4. Task dependencies instead of manual sequencing.
5. Retries for transient failures.
6. Alerts or notifications for failure and SLA breaches.
7. Idempotent write logic.
8. Logging and run metadata.
9. Source-controlled code.
10. Separate development, test, and production deployment paths.

Do not memorize syntax alone. Practice questions usually test when to use the pattern, what prerequisite is missing, or what risk the command introduces.

Scenario-reading mistakes