Databricks Data Engineer Associate: Data Pipeline Production

Topic snapshot

Sample questions

These questions are original IT Mastery practice items aligned to this topic area. They are designed for self-assessment and are not official exam questions.

Question 1

Topic: Productionizing Data Pipelines

A team currently imports notebooks and recreates the same job manually in each workspace. They want a Git-tracked deployment definition and show this file:

bundle:
  name: orders-pipeline

targets:
  dev:
    mode: development
  prod:
    mode: production

resources:
  jobs:
    orders_job:
      tasks:
        - task_key: daily_load
          notebook_task:
            notebook_path: ../src/daily_load.py

What is the best interpretation of how Databricks Asset Bundles differ from direct workspace deployment?

Options:

• A. Versioned, repeatable deployment of resources across environments

• B. Read-only data sharing with external recipients

• C. Local IDE execution against Databricks compute

• D. Automatic file ingestion with incremental processing

Best answer: A

Explanation: The exhibit shows a bundle file with named targets and a job resource defined as code. That means the deployment can be versioned in Git and applied consistently to development and production, unlike manually creating or updating the workflow directly in each workspace.

Databricks Asset Bundles are a deployment-as-code approach. A bundle file declares Databricks resources, such as jobs, plus target environments, so teams can review changes, store them in version control, and deploy the same definition repeatedly. That is the high-level difference from traditional direct workspace deployment, where notebooks, jobs, or settings are created or edited directly in the workspace UI or imported manually.

In this exhibit, the targets section signals environment-aware deployment, and resources.jobs shows the workflow is defined declaratively in source files. With bundles, the source files become the source of truth; with manual deployment, the workspace often becomes the source of truth. Features for ingestion, local coding, or external sharing do not address this deployment consistency problem.

• Ingestion feature describes Auto Loader, which handles arriving files rather than deployment definitions.
• Local development describes Databricks Connect, which helps write and run code from an IDE but is not a deployment method.
• Data sharing describes Delta Sharing, which provides governed read access to data, not repeatable job deployment.

Question 2

Topic: Productionizing Data Pipelines

A data engineer reviews the Spark UI for a Databricks workflow task. One stage shows:

Stage 12
Tasks: 200
Task time: min 9s, median 12s, max 7m 41s
Shuffle read per task: median 18 MB, max 1.9 GB
Spill (disk): present on longest tasks

Which interpretation is most likely?

Options:

• A. Normal Photon execution with no clear inefficiency

• B. Auto Loader file discovery delaying task start

• C. An undersized driver causing the stage slowdown

• D. A skewed shuffle from an uneven join or aggregation key

Best answer: D

Explanation: The exhibit shows major imbalance across tasks in the same stage. When most tasks finish quickly but a few read far more shuffled data, spill to disk, and run much longer, the most likely cause is data skew during a shuffle.

In Spark UI, data skew appears when a small number of tasks do much more work than the rest. Here, the median task finishes in 12 seconds and reads 18 MB, but the slowest task runs for 7 minutes 41 seconds and reads 1.9 GB. Spill on only the longest tasks strengthens that signal. This usually happens after a shuffle, such as a join or aggregation on unevenly distributed keys, where a few partitions become much larger than others. The best next investigation is the query pattern and the join or grouping keys. A driver issue or general lack of compute would more often affect many tasks more uniformly, not produce a few extreme outliers.

• Driver sizing fails because driver bottlenecks do not usually show up as a few executor tasks with huge shuffle-read outliers.
• Auto Loader delay fails because the exhibit is a stage execution summary, not file-discovery time before processing starts.
• Normal variation fails because the gap between median and maximum task metrics is too large to treat as routine execution variance.

Question 3

Topic: Productionizing Data Pipelines

An operations team manages the same workflow in dev, test, and prod by editing jobs and notebooks directly in each Databricks workspace. A prod run fails again after a release:

Task: load_customers
Status: Failed
Error: Notebook not found: /Shared/etl/load_customers
Note: the notebook was renamed in dev last week
Hotfix: prod job path was edited manually and succeeded
Next release: older job settings were copied back to prod

The team’s main goal is repeatable, source-controlled deployments across workspaces. What is the best next step?

Options:

• A. Use repair runs after releases when notebook paths drift.

• B. Store only notebooks in Git folders and manage jobs in the UI.

• C. Keep editing each workspace manually and export objects after hotfixes.

• D. Deploy a Git-backed Databricks Asset Bundle for the workflow and code.

Best answer: D

Explanation: The recurring failure is caused by workspace drift from manual edits. Databricks Asset Bundles solve that by defining deployable resources and code in source control so the same version can be promoted consistently across environments.

This scenario shows why manual workspace editing is unreliable for production deployment. The notebook rename, manual prod hotfix, and later overwrite created inconsistent job definitions across environments. Databricks Asset Bundles are designed for this exact problem: they let a team define resources such as workflows and the code they deploy in source control, then promote those definitions to dev, test, and prod in a repeatable way.

With a bundle, the team can:

• keep deployment definitions in Git
• version changes with the code
• deploy the same declared resources to each workspace
• reduce accidental drift from UI-only edits

The closest distractor is using Git folders alone, which helps with notebook versioning but does not fully replace declarative deployment of workflow resources.

• Manual hotfixes may restore service temporarily, but they keep environments out of sync and are hard to reproduce reliably.
• Git folders only improve notebook source control, but job and workflow configuration can still drift when managed manually.
• Repair runs are for rerunning failed tasks, not for standardizing how resources are deployed across workspaces.

Question 4

Topic: Productionizing Data Pipelines

A data engineering team must promote the same governed workflow through dev, test, and prod. Their current process is shown.

Exhibit:

Release process
- Notebooks are edited directly in the workspace
- Test job is created by cloning the dev job in the UI
- Prod job is created by cloning the test job in the UI
- Schedules, permissions, and cluster settings are edited by hand
- Last release issue: prod used an outdated notebook path

Which Databricks approach is the best next step?

Options:

• A. Enforce a cluster policy for the existing UI jobs

• B. Use Git folders and keep cloning jobs manually

• C. Define a Databricks Asset Bundle with environment targets

• D. Rebuild it as a Lakeflow Spark Declarative Pipeline

Best answer: C

Explanation: This process shows classic direct-deployment drift: cloned jobs, manual edits, and production using the wrong path. Databricks Asset Bundles are designed to deploy governed workflow resources consistently across environments from a single version-controlled definition.

Databricks Asset Bundles are the stronger choice when a team needs repeatable, governed promotion of workflows across environments. In the exhibit, the problem is not notebook authoring or compute rules alone; it is that jobs are cloned and then changed manually, which causes configuration drift. The outdated production notebook path is a direct example of why traditional direct deployment is weaker here.

A bundle helps by:

• defining jobs and related resources as code
• using targets for dev, test, and prod differences
• deploying the same reviewed definition to each environment

Git folders version notebook code, but they do not by themselves manage the full deployed workflow definition. The key takeaway is that governed production releases need a single source of truth for deployment, not repeated manual UI copying.

• Git only improves notebook versioning, but manual job cloning still leaves schedules, paths, and permissions vulnerable to drift.
• Pipeline redesign focuses on transformation authoring, while the exhibit’s main issue is controlled deployment across environments.
• Cluster policy only can standardize compute settings, but it does not version the full workflow release or prevent manual job drift.

Question 5

Topic: Productionizing Data Pipelines

A Databricks workflow task on serverless compute joins a large orders table with a customers table and now regularly misses its SLA. The Spark UI for the slowest stage shows:

Tasks: 200
199 tasks finished in 12-25 s
1 task ran for 15 min
Longest task shuffle read: 7.8 GB
Most other tasks shuffle read: <100 MB

What is the best next step?

Options:

• A. Repair the workflow run and rerun only the slow task.

• B. Increase driver memory for the workflow task.

• C. Investigate skewed join-key values and repartition data before the join.

• D. Convert the customers table to an external table.

Best answer: C

Explanation: The Spark UI shows one task doing much more work than the rest of the stage. That pattern usually indicates skewed data distribution, so the best next step is to review the join key and repartition or otherwise rebalance the data before the join.

When almost every task in a stage finishes quickly but one task runs much longer and reads most of the shuffle data, Spark is not dividing the work evenly. At Associate depth, this is strong evidence of data skew, often caused by a join key where a few values contain far more rows than others. The practical next step is to inspect the join-key distribution and rebalance the data before the join so partitions are more even.

• Check whether a few key values dominate the data.
• Reduce data earlier if filtering is possible.
• Repartition on a more even distribution before the join.

The key takeaway is that the Spark UI is pointing to uneven partition work, not simply a need to rerun the job or change table ownership.

• Driver sizing misses the signal because one unusually heavy task points to partition imbalance, not a driver-only bottleneck.
• Repair rerun retries the same skewed work and does not change how the data is distributed.
• External table change is unrelated because table storage type does not fix a skewed shuffle stage.

Question 6

Topic: Productionizing Data Pipelines

A data engineer reviews a batch pipeline run in Databricks. The Spark UI for one stage shows:

Stage 14
Tasks: 200
Task time: median 21 s, max 8 min 54 s
Shuffle read: median 42 MB, max 4.9 GB
Near stage completion: 199 tasks finished, 1 task still running

What is the best interpretation of this stage?

Options:

• A. An undersized cluster; add workers because tasks are uniformly slow.

• B. A driver bottleneck; move more logic from the driver to executors.

• C. Small-file overhead; compact source files before running the job.

• D. A skewed shuffle partition; inspect join or grouping key distribution.

Best answer: D

Explanation: The stage shows one clear straggler: almost all tasks finish quickly, but one task runs much longer and reads far more shuffle data. That pattern usually indicates shuffle skew, often caused by uneven join or grouping keys.

Spark UI can reveal skew at a high level without deep Spark internals. Here, the median task finishes in 21 seconds and reads 42 MB, but one task reads 4.9 GB of shuffle data and continues running after the other 199 tasks finish. That large gap means the work is not evenly distributed across partitions. In practice, this often happens after a join, aggregation, or repartition on a skewed key, where one partition receives far more records than the others.

A good next step is to inspect the transformation that caused the shuffle and review the join or grouping key distribution. Adding workers or blaming the driver would not explain why a single task is such an extreme outlier while the rest of the stage completes normally.

• The driver-bottleneck idea fails because the exhibit shows one long executor task, not evidence of driver-side collection or coordination delays.
• The undersized-cluster idea fails because tasks are not uniformly slow; the problem is one extreme outlier task.
• The small-file-overhead idea fails because the exhibit highlights an imbalanced shuffle read, not file-scan startup overhead.

Question 7

Topic: Productionizing Data Pipelines

A data engineering team needs an hourly workflow that uses Auto Loader to ingest new CSV files, applies simple PySpark transformations, and writes Delta tables governed by Unity Catalog. They want a hands-off solution with no cluster sizing or tuning and automatic performance optimization. Which compute choice is best?

Options:

• A. Run the workflow on serverless compute.

• B. Use a classic job cluster with manual autoscaling settings.

• C. Use a SQL warehouse for the entire pipeline.

• D. Keep a shared all-purpose cluster running.

Best answer: A

Explanation: Serverless compute is designed for workloads where the team wants Databricks to manage infrastructure details. This scenario is a scheduled ETL pipeline with straightforward ingestion and transformation steps, so serverless compute is the most hands-off, auto-optimized fit.

The core concept is matching the workload to the right Databricks compute model. For a scheduled data-engineering pipeline that ingests files, runs basic PySpark transformations, and writes governed Delta tables, serverless compute is a strong fit when the team wants minimal operational overhead. Databricks handles provisioning, scaling, and many built-in optimizations, so engineers do not need to choose cluster sizes, tune autoscaling, or manage long-running shared compute.

That makes serverless compute especially appropriate for routine production workflows where simplicity and automation matter more than custom cluster control. Shared all-purpose clusters and classic job clusters can run the workload, but they add administration the team explicitly wants to avoid. The SQL warehouse choice misses the PySpark and Auto Loader pipeline requirement.

• Shared cluster is better for interactive development, but it adds ongoing compute management and is not the most hands-off production option.
• Classic job cluster can run the job, but it still requires explicit cluster configuration instead of fully managed serverless execution.
• SQL warehouse is intended for SQL workloads and is not the best fit for an Auto Loader plus PySpark pipeline.

Question 8

Topic: Productionizing Data Pipelines

A data engineer reviews the following workflow note for a scheduled notebook task:

Workflow: `daily_customer_rollup`
Task type: Notebook
Current compute: New job cluster

Recent successful run
- Cluster start: 3m 20s
- Notebook execution: 58s
- Cluster termination: 1m 05s

Task reads from and writes to Delta tables in Unity Catalog.
No custom libraries or init scripts are required.

The team wants the simplest change to reduce operational overhead for this task. Which recommendation best fits the exhibit?

Options:

• A. Grant broader Unity Catalog privileges to the job’s run identity.

• B. Convert the target Delta table to an external table.

• C. Split the pipeline output into Bronze, Silver, and Gold tables.

• D. Move the task to serverless compute for workflows.

Best answer: D

Explanation: The exhibit shows a short notebook task spending far more time starting and stopping compute than actually processing data. Because the task already succeeds with Unity Catalog Delta tables and needs no custom setup, this is a serverless-compute decision rather than a modeling or governance decision.

Serverless compute is the best fit when a workflow task is short-lived and cluster lifecycle time dominates the run. In the exhibit, the notebook runs for less than a minute, but starting and terminating the job cluster adds several more minutes. The task already reads from and writes to Unity Catalog Delta tables successfully, and there is no sign of missing permissions or a table-design problem.

• The main issue is compute overhead, not query logic.
• Serverless compute is designed to reduce infrastructure management for supported short scheduled tasks.
• Data-modeling changes and governance changes should only be chosen when the exhibit shows schema, table, or access issues.

The closest distractors change storage design or access controls, but neither would remove cluster start and stop time.

• Medallion redesign changes data organization, but the exhibit points to compute lifecycle overhead instead.
• External table change affects storage governance and location, not job-cluster startup or termination time.
• Broader privileges do not fit because the workflow is already successfully reading and writing Unity Catalog tables.

Question 9

Topic: Productionizing Data Pipelines

A data engineering team wants to deploy the same set of Databricks workflows and pipeline definitions to development, test, and production. They want a consistent deployment structure across environments, with only environment-specific settings changing. Which Databricks feature best fits this requirement?

Options:

• A. Delta Sharing

• B. Databricks Repos

• C. Databricks Asset Bundles

• D. Lakeflow Spark Declarative Pipelines

Best answer: C

Explanation: Databricks Asset Bundles are designed for repeatable, code-based deployment of Databricks resources across environments. They support a consistent project structure while allowing controlled differences for targets such as development, test, and production.

Databricks Asset Bundles are the Databricks feature for defining deployment artifacts as code and promoting them consistently across environments. A bundle can describe Databricks resources and deployment targets, so a team can keep one project definition in source control and deploy it to development, test, and production without manually rebuilding the same setup each time.

• Use one bundle definition for the project.
• Define separate targets for each environment.
• Keep only the environment-specific settings different.

This makes bundles the best fit when the main requirement is consistent multi-environment deployment structure.

• Repos are not deployment packaging because they help version-control notebooks and files, but do not by themselves provide structured multi-environment deployment.
• Pipeline authoring is narrower because Lakeflow Spark Declarative Pipelines focuses on building and running data pipelines, not packaging an overall project for consistent promotion.
• Sharing solves a different problem because Delta Sharing is for secure data sharing, not deployment of jobs or pipelines across environments.

Question 10

Topic: Productionizing Data Pipelines

A data engineering team has a notebook-based ingestion job orchestrated with Databricks Workflows in a development workspace. They must promote the same jobs to test and production, keep deployment configuration in source control, and minimize manual workspace changes during releases. What is the best next step?

Options:

• A. Use Databricks Asset Bundles to define the jobs, notebooks, and targets, then deploy them from source control.

• B. Convert the workflow to Lakeflow Spark Declarative Pipelines because pipelines replace deployment tooling.

• C. Store the notebooks in Databricks Repos and manually recreate the workflow settings in each workspace.

• D. Use Delta Sharing to publish the workflow assets to the other workspaces.

Best answer: A

Explanation: Databricks Asset Bundles are designed for source-controlled, repeatable deployment of Databricks resources across environments. Traditional workspace deployment methods can work, but they rely more on direct workspace updates and are more likely to create drift between development, test, and production.

The core difference is deployment as code versus direct workspace changes. Databricks Asset Bundles let a team define resources such as notebooks, jobs, and environment-specific targets in files stored in version control, then deploy the same project consistently to multiple environments. This supports repeatable releases and reduces configuration drift.

Traditional methods usually mean importing code, editing jobs in the workspace, or managing parts of the deployment separately in each environment. Those approaches are more manual and harder to standardize. Repos can help with source control for code, but they do not by themselves package and deploy all required Databricks resources. Features like Delta Sharing and Lakeflow Spark Declarative Pipelines address different needs, not general multi-environment asset deployment.

• Repos only helps version notebook code but still leaves job and workflow configuration to manual workspace updates.
• Data sharing is for securely sharing data, not for deploying notebooks, jobs, or workspace resources.
• Pipeline authoring focuses on defining and running data pipelines, not replacing bundle-based deployment across environments.

Continue with full practice

Use the Databricks Data Engineer Associate Practice Test page for the full IT Mastery route, mixed-topic practice, timed mock exams, explanations, and web/mobile app access.

Try Databricks Data Engineer Associate on Web View Databricks Data Engineer Associate Practice Test

Related focused pages

• Free Databricks Data Engineer Associate Full-Length Practice Exam
• Intelligence Platform
• Development and Ingestion
• Processing and Transforms
• Governance and Quality

Free review resource

Read the Databricks Data Engineer Associate Cheat Sheet on Tech Exam Lexicon, then return to IT Mastery for timed practice.