Databricks Certified Generative AI Engineer Associate Cheat Sheet

This page is not a replacement for hands-on work in Databricks. It is a fast review of the concepts you are likely to need when answering scenario-based questions about building, evaluating, governing, and deploying generative AI applications on the Databricks platform.

Exam-ready mental model

Chunking choices

Recommended chunk table schema

Retrieval tuning

Grounded RAG prompt pattern

System:
You are a Databricks RAG assistant. Use only the provided CONTEXT.
Do not use outside knowledge. If the answer is not supported by CONTEXT,
say "I do not know based on the provided context."

User question:
{question}

CONTEXT:
{context}

Return:
- answer
- citations using source_uri and section

LLM parameter decisions

Prompting traps

Fine-tuning versus RAG versus prompting

A frequent exam decision point is choosing the right adaptation strategy.

Fine-tuning traps

• Fine-tuning to memorize facts that change frequently.
• Fine-tuning without a validation set.
• Fine-tuning before establishing a baseline with prompting and RAG.
• Ignoring cost, latency, governance, and rollback.
• Training on low-quality examples and expecting high-quality behavior.
• Confusing fine-tuning with retrieval: fine-tuning changes model behavior; retrieval supplies external knowledge at inference time.

Exam points:

• Use query_text when the index manages query embedding.
• Use a query vector only when you are managing embeddings yourself.
• Return source metadata needed for citations.
• Apply filters when user role, document type, date, or product scope matters.

Minimal retrieval formatting pattern

def format_docs(docs):
    return "\n\n".join(
        f"Source: {d.get('source_uri')} | Section: {d.get('section')}\n{d.get('chunk_text')}"
        for d in docs
    )

• Do not pass raw objects to the prompt if the model needs readable context.
• Include metadata for traceability.
• Keep formatting consistent for evaluation.

LangChain-style RAG chain pattern

from operator import itemgetter
from langchain_core.prompts import ChatPromptTemplate
from langchain_core.output_parsers import StrOutputParser
from langchain_core.runnables import RunnableLambda, RunnablePassthrough

prompt = ChatPromptTemplate.from_messages([
    ("system", "Answer only from the provided context. Cite sources. If unsupported, say you do not know."),
    ("user", "Question: {question}\n\nContext:\n{context}")
])

rag_chain = (
    {
        "question": itemgetter("question"),
        "context": itemgetter("question") | RunnableLambda(retrieve) | RunnableLambda(format_docs),
    }
    | prompt
    | chat_model
    | StrOutputParser()
)

• itemgetter("question") extracts the user input field.
• Retrieval should happen before prompt construction.
• Output parsing should match the expected serving response.
• Package custom functions, dependencies, and configuration before serving.

Model Serving call pattern

from openai import OpenAI
import os

client = OpenAI(
    api_key=os.environ["DATABRICKS_TOKEN"],
    base_url=f"{os.environ['DATABRICKS_HOST']}/serving-endpoints"
)

response = client.chat.completions.create(
    model="serving-endpoint-name",
    messages=[
        {"role": "system", "content": "Answer using only the provided context."},
        {"role": "user", "content": "Question and context go here."}
    ],
    temperature=0.1
)

• Treat the serving endpoint name as the model target.
• Do not hardcode tokens in notebooks, chains, or app code.
• Keep parameters aligned with the endpoint and provider capabilities.
• Use deterministic settings for evaluation where practical.

MLflow packaging pattern

import mlflow

mlflow.set_registry_uri("databricks-uc")

with mlflow.start_run():
    mlflow.log_param("retriever_top_k", 5)
    mlflow.log_param("prompt_version", "rag_prompt_v3")
    mlflow.log_metric("eval_groundedness", 0.87)

    # Log the chain/model with its dependencies and input example.
    # Register to Unity Catalog for governed deployment.

• Track prompt version, model endpoint, retriever config, chunking config, and evaluation dataset.
• Register production artifacts in Unity Catalog when governance is required.
• Include input examples and signatures so serving can validate requests.
• Logging the notebook alone is not enough for reproducible deployment.

Security and privacy checklist

• Use least privilege for data, indexes, models, functions, and serving endpoints.
• Keep user identity and authorization in mind for retrieval filtering.
• Do not allow a user to retrieve chunks they could not access directly.
• Store sensitive prompts/responses only when logging policy allows it.
• Redact or avoid collecting sensitive data in evaluation datasets when possible.
• Use service principals or supported machine credentials for production jobs.
• Keep credentials out of prompt templates, notebooks, source code, and MLflow params.
• Validate model outputs before using them in downstream systems.
• Treat user input and retrieved context as untrusted text.

Safer tool-calling principles

Safety, guardrails, and prompt injection

GenAI applications need defensive design. Safety is not only about harmful content; it includes data leakage, unauthorized actions, misleading output, and failure to follow policy.

Prompt injection review

Prompt injection occurs when user-provided or retrieved text attempts to override developer/system instructions.

Guardrail checklist

• Separate instructions from untrusted content.
• Do not put secrets in prompts.
• Validate structured outputs.
• Apply permission filters before retrieval context is assembled.
• Use allowlists for tools and actions.
• Add refusal behavior for unsupported, unsafe, or unauthorized requests.
• Monitor safety failures and update tests.

Evaluation dataset design

Offline vs online evaluation

LLM-as-judge traps

Evaluation and monitoring

Evaluation is one of the most important GenAI engineering skills because LLM outputs are probabilistic and application quality is multidimensional.

Offline versus online evaluation

Good evaluation dataset properties

A strong evaluation set includes:

• Common user questions.
• Edge cases and ambiguous requests.
• Questions requiring refusal.
• Questions with no answer in the context.
• Questions requiring exact facts from documents.
• Multi-hop questions, if the application must handle them.
• Representative languages, formats, and user roles.
• Known difficult examples from production logs, if permitted and sanitized.

Evaluation traps

• Evaluating only happy-path examples.
• Using the same examples for prompt design and final evaluation without a holdout set.
• Measuring average quality while ignoring severe safety failures.
• Failing to separate retrieval failures from generation failures.
• Treating an LLM judge as perfect instead of validating judge behavior.
• Not re-running evaluation after changing model, prompt, index, or data source.

Batch vs real-time GenAI

High-yield troubleshooting table

Fast final review checklist

Before practice questions, confirm you can answer these quickly:

• What problem does RAG solve?
• When is RAG better than fine-tuning?
• What causes poor retrieval precision versus poor retrieval recall?
• Why are chunk size and overlap important?
• What metadata should be preserved for RAG?
• How do Unity Catalog permissions affect GenAI application design?
• What should be tracked with MLflow in a GenAI workflow?
• How do you evaluate groundedness and relevance?
• What are common prompt injection defenses?
• How do you make tool-using agents safer?
• What should you monitor after deployment?
• How do latency, token count, model choice, and context size interact?

Prompt engineering decision rules

Prompting is often the cheapest first improvement. Good prompts reduce ambiguity and make evaluation easier.

A useful prompt structure:

1. System instruction: role, boundaries, safety constraints.
2. Task instruction: what to do.
3. Context: retrieved documents, user profile, approved reference text.
4. Output format: schema, bullets, JSON, table, citation style.
5. Quality rules: what to do when context is missing or ambiguous.

Prompting traps

• Asking the model to “be accurate” without giving it trusted context.
• Mixing user-provided text and trusted instructions without clear boundaries.
• Providing contradictory examples.
• Requiring citations but not passing source identifiers.
• Asking for strict JSON but not implementing parsing, validation, and retry logic.
• Using a long prompt that hides the actual user task.
• Treating prompt success on a few examples as proof of production readiness.

RAG component review

Chunking decision rules

Retrieval diagnostics

When a RAG answer is bad, diagnose in this order:

1. Was the right source data available?
2. Was it ingested and indexed correctly?
3. Did the query retrieve the right chunks?
4. Were the right chunks ranked high enough?
5. Was too much irrelevant context included?
6. Did the prompt tell the model how to use the context?
7. Did the model ignore the context or hallucinate?
8. Did evaluation capture the failure clearly?

Do not jump directly to fine-tuning. Many RAG failures are retrieval, chunking, filtering, or prompt assembly failures.

RAG metrics to recognize

Vector search traps

• Using embeddings created by one model with queries embedded by another incompatible model.
• Indexing stale data and wondering why answers reference old policies.
• Dropping metadata needed for citations or access control.
• Retrieving entire documents instead of focused chunks.
• Failing to filter by user permissions before context reaches the LLM.
• Evaluating only the final answer and not retrieval quality.

Unity Catalog governance review

Unity Catalog is high-yield because GenAI applications often touch sensitive enterprise data.

Common governance traps

• Passing sensitive rows to a prompt because retrieval was not permission-filtered.
• Letting an agent call a tool without checking the user’s authorization.
• Logging complete prompts and outputs that contain sensitive data without a retention or redaction plan.
• Treating model access as separate from data access when the application combines both.
• Using a development notebook credential in a production application.

Deployment traps

• Deploying a notebook workflow without packaging configuration, dependencies, and permissions.
• Updating prompts or retrieval settings without re-running evaluation.
• Ignoring token usage until costs spike.
• Serving an application that depends on a vector index not refreshed on the same schedule as source data.
• Assuming a model endpoint is production-ready just because it returns responses.

Evaluation-first habit

Before changing a model or prompt, define what “better” means. Good exam answers often prefer an evaluation-driven change over an ad hoc change.

Ask:

• What dataset represents expected user questions?
• What are the expected answers or judging criteria?
• Do we need human review, automated judges, or both?
• Are we measuring retrieval separately from generation?
• Are we checking safety, privacy, and refusal behavior?
• Is latency/cost part of success?

Tool-use decision rules

• Define tools narrowly with clear input schemas.
• Validate tool inputs before execution.
• Enforce user authorization before tool execution, not after.
• Prefer deterministic tools for calculations, database lookups, and transactions.
• Keep irreversible actions behind confirmation or policy checks.
• Log tool calls and outcomes for troubleshooting.
• Evaluate multi-step traces, not only the final answer.

Agent traps

• Giving an agent broad database access when a narrow function would be safer.
• Allowing the model to construct arbitrary SQL or API calls without validation.
• Not testing what happens when tools fail or return empty results.
• Treating “the agent can reason” as a substitute for deterministic business rules.
• Forgetting that prompt injection can target agents through retrieved documents or user text.

Pattern 2: RAG application

Use when the answer must be grounded in enterprise knowledge.

Pattern 3: Agentic application

Use when the system must perform multi-step work or call tools.