Free DATA-ENG-ASSOC sample questions

Correct answer: `.option("cloudFiles.schemaEvolutionMode", "rescue")`

The `rescue` mode for `cloudFiles.schemaEvolutionMode` is designed for this exact scenario. It instructs Auto Loader to infer the schema and, if a data type change is detected (like numeric to string), it adds the new column to the `_rescue` column. For schema evolution, the correct option is to enable schema evolution on the write stream using `.option("mergeSchema", "true")` and handle rescued data. However, among the given choices, `rescue` mode is the feature specifically designed to handle columns with mixed data types by rescuing them, which aligns with the observed behavior. While `mergeSchema` is also needed, `schemaEvolutionMode` `rescue` directly addresses the problem of incompatible data types appearing in a column.

Correct answer: Do not grant any privileges on the Bronze tables to any users or groups, but grant `SELECT` and `MODIFY` on the specific Bronze tables to the service principal.

Unity Catalog operates on a default-deny model. To meet the requirement, you should avoid granting any broad permissions at the catalog or schema level. The correct approach is to grant the necessary, specific privileges (`SELECT` to read, `MODIFY` might be needed for the job's operations) directly to the service principal on the target Bronze tables. This ensures that only that principal can access the sensitive data, enforcing the principle of least privilege.

Correct answer: True

This statement is true. The primary purpose of cluster pools is to reduce cluster start and auto-scaling times by maintaining a set of idle, ready-to-use instances. When a job requests a cluster attached to a pool, it gets its nodes from this warm pool instead of requesting new instances from the cloud provider, which significantly shortens the startup latency.

Correct answer: The code fails to apply any data cleansing, validation, or schema enforcement rules before writing to the Silver layer.

The primary purpose of the Silver layer is to store cleansed, validated, and enriched data. This code snippet moves data directly from the source (Bronze) to Silver using a blind overwrite without any intermediate transformation steps. This violates the core principle of the Medallion architecture, as it bypasses the crucial data quality and shaping processes that should occur between the Bronze and Silver layers.

Correct answer: `@dlt.table(comment="Enriched data")` `@dlt.expect_or_drop("valid_key", "join_key IS NOT NULL")`

To create a physical table that can be queried later, `@dlt.table` is the correct decorator. To enforce a quality rule where invalid records are dropped while allowing the pipeline to continue, `@dlt.expect_or_drop` is the appropriate function. `@dlt.expect_or_fail` would halt the pipeline, and `@dlt.expect` would only record the violation statistics without dropping or failing.

Correct answer: Delta Sharing open sharing protocol

Delta Sharing is the open protocol specifically designed for sharing live data from Databricks to any external consumer, regardless of their platform. It works by creating shares and recipients, then providing the recipient with a secure, one-time activation link to download a credential file. This allows them to access the shared data using various connectors (like Power BI, Tableau, pandas) without needing a Databricks account. Lakehouse Federation is for querying external data sources from within Databricks, not sharing data out.

Correct answer: A serverless SQL warehouse.

Serverless SQL warehouses are ideal for this use case. They provide instant compute, automatically scale up and down to handle concurrent query loads, and eliminate the need for administrators to manage cluster configurations. This directly addresses the requirements for performance on variable workloads and reduced management overhead. A job cluster is for automated jobs, and an all-purpose cluster requires manual management and is less efficient for handling concurrent SQL queries.

Correct answer: A `merge` operation with `whenMatchedUpdate` and `whenNotMatchedInsert` clauses.

The `merge` operation is specifically designed for this type of 'upsert' (update or insert) scenario. It allows you to define conditions for matching records between a source DataFrame and a target Delta table. The `whenMatchedUpdate` clause specifies the update logic for existing records, while the `whenNotMatchedInsert` clause handles the insertion of new records, all within a single, atomic transaction.

Correct answer: Use the 'Repair and Rerun' feature from the failed job run page.

The 'Repair and Rerun' feature is designed for this exact purpose. It allows you to resume a failed workflow run from the point of failure. It intelligently skips the tasks that have already completed successfully (like `ingest_data`) and reruns only the failed tasks and their downstream dependencies. This is the most efficient method as it avoids re-processing data unnecessarily.

Correct answer: Resolving complex merge conflicts between two branches.

The Databricks Repos UI provides core Git functionality like commit, pull, push, and branching. However, for complex operations like resolving merge conflicts that Git cannot handle automatically, the user must pull the changes to their local machine, use standard Git tools to resolve the conflicts, and then push the resolved code back to the remote repository.