DATA-ENG-PRO sample questions

Question 1

A developer is defining a Databricks Asset Bundle (DAB) to deploy a multi-task job. The job includes a task that runs a Python script packaged as a wheel file.

Accepted Answer

True

Answer

False

Question 2

**Company Background:**
Streamlytics, a real-time analytics provider, processes IoT sensor data. They have a Spark Structured Streaming job that joins a stream of sensor readings (`sensor_stream`) with a static dimension table of sensor metadata (`sensor_metadata`). The job performs an aggregation to calculate the average temperature per sensor type every 5 minutes.

**Current Architecture:**
The streaming job uses a 10-minute watermark on the event timestamp to handle late-arriving data. The state store is located on cloud object storage, and the pipeline runs on an auto-scaling job cluster. This setup has been running efficiently for several months.

**The Problem:**
After a new set of legacy devices were onboarded, the pipeline's performance has severely degraded. Micro-batch processing times have inflated from 30 seconds to over 15 minutes, causing the stream to fall behind. The state store size has grown to several terabytes. An investigation reveals that the legacy devices occasionally send data that is several days late. These records are critical and must eventually be processed for auditing purposes.

**Business Requirement:**
The engineering lead has tasked you with resolving the performance degradation and the massive state size. The solution must not lose the late-arriving data and should avoid a significant increase in steady-state compute costs.

Accepted Answer

Rearchitect the pipeline to write all raw data to a Delta table. The main streaming job will process data with a tight watermark. A separate, nightly batch job will process data older than the watermark from the raw table and merge it into the final aggregated table.

Answer

Increase the watermark to 7 days to accommodate all late data and provision a larger cluster.

Answer

Enable RocksDB-based state management on instances with local SSDs to improve state access speed.

Answer

Split the input stream into two separate streams. Process the normal data with a 10-minute watermark in one stream, and the legacy data with a 7-day watermark in another, then union the results.

Question 3

A data engineering team has designed a multi-task job in Databricks Workflows to orchestrate their daily ETL process. The job has a linear dependency chain for the main data processing and an independent task for auditing that can run at any time after the start. An engineer needs to add a new task, `generate_reports`, which must run only after both the `validate_data` and `process_data` tasks have successfully completed.

Which task dependency configuration correctly implements this requirement?

```mermaid
graph TD
A[start_job] --> B[ingest_data];
B --> C[validate_data];
C --> D[process_data];
D --> E[load_warehouse];
A --> F[run_audits];
```

Accepted Answer

Set `generate_reports` to depend on both `validate_data` and `process_data`.

Answer

Set `generate_reports` to depend on `process_data`.

Answer

Create two separate jobs: one for the main pipeline and one for `generate_reports` triggered via the API upon completion of `process_data`.

Answer

Set `generate_reports` to run in parallel with `load_warehouse`, with both depending on `process_data`.

Question 4

A data engineer needs to ingest a large volume of continuously arriving JSON files from a cloud storage location. The schema of the JSON files is known to evolve over time, with new columns being added frequently. The pipeline must be resilient to malformed JSON records and should infer and handle schema changes automatically. Which Databricks feature is best suited for this ingestion task?

Accepted Answer

Auto Loader with schema inference and evolution enabled.

Answer

A standard `spark.readStream("json", path)` command.

Answer

A batch job that runs `spark.read.json(path)` every minute.

Answer

Delta Live Tables without Auto Loader.

Question 5

A data pipeline needs to transform a bronze Delta table containing raw customer orders into a silver table. During this transformation, any order record that does not have a valid `order_id` or has a negative `total_amount` should be moved to a quarantine table for later analysis, instead of being loaded into the silver table. The main pipeline should continue processing valid records without interruption. Which Delta Live Tables (DLT) feature should be used to implement this data quality and quarantining process?

Accepted Answer

Multi-hop architecture with expectations and quarantine tables.

Answer

`@dlt.expect_or_fail`

Answer

`@dlt.expect_all_or_drop`

Answer

`@dlt.expect_or_drop`

Question 6

An analytics query on a large, date-partitioned Delta table is running slowly. The query frequently filters on a high-cardinality `user_id` column within specific date ranges. The Spark UI shows that a large number of files are being read even for queries that target a small number of users. The table's files are already compacted to an optimal size (around 1GB). Which optimization technique should be applied to improve data skipping and reduce the number of files scanned?

Accepted Answer

Run the `OPTIMIZE` command with Z-ORDER BY `user_id`.

Answer

Enable Change Data Feed (CDF) on the table.

Answer

Use the `REPARTITION` command on the `user_id` column.

Answer

Enable deletion vectors on the table.

Question 7

What is the primary function of the metastore in the context of the Databricks Lakehouse Platform?

Accepted Answer

To serve as a central repository for metadata about data objects like catalogs, schemas, tables, and views.

Answer

To store the actual data files (e.g., Parquet files) for all tables.

Answer

To manage the lifecycle of clusters, including starting, stopping, and terminating them.

Answer

To execute SQL queries and return results to the user.

Question 8

An engineering team is building a near-real-time ETL pipeline using Delta Live Tables (DLT) to process a Change Data Capture (CDC) feed from a relational database. The source system provides records with `operation_type` ('INSERT', 'UPDATE', 'DELETE'), a primary key, and a timestamp for ordering. The goal is to efficiently replicate these changes into a target Delta table.

Which THREE DLT features or commands are essential for correctly implementing this CDC pipeline? (Select THREE)

Accepted Answer

The `dlt.apply_changes()` function.

Accepted Answer

Specifying the sequence of events using `SEQUENCE BY`.

Accepted Answer

Identifying the operation type using `APPLY AS DELETE WHEN` and `APPLY AS TRUNCATE WHEN`.

Answer

Using `@dlt.expect_or_fail` to validate the primary key.

Answer

Setting the pipeline mode to `Triggered` instead of `Continuous`.

Question 9

A data engineer is investigating a slow-running Databricks job. They need to analyze the execution plan, identify data shuffling, and view detailed metrics for each stage of the job. Which component of the Databricks UI provides the most comprehensive information for this type of performance analysis?

Accepted Answer

The Spark UI.

Answer

The Cluster Metrics page (Ganglia UI).

Answer

The Job Runs page in Databricks Workflows.

Answer

The Query Profile in Databricks SQL.

Question 10

A data provider wants to securely share a specific Delta table named `live_sales_data` with an external partner who is not on Databricks. The provider wants to ensure the partner always has access to the most recent version of the data without creating copies or managing complex data export pipelines. Which Databricks technology is designed for this purpose?

Accepted Answer

Delta Sharing using the open sharing protocol

Answer

Lakehouse Federation

Answer

Databricks-to-Databricks Sharing

Answer

Cloning the table to the partner's cloud storage bucket.

Free DATA-ENG-PRO sample questions

A developer is defining a Databricks Asset Bundle (DAB) to deploy a multi-task job. The job includes a task that runs a Python script packaged as a wheel file.

What is the primary function of the metastore in the context of the Databricks Lakehouse Platform?

10 more free samples are waiting