AWS AIP-C01: Testing, Validation, and Troubleshooting

Topic snapshot

Sample questions

These questions are original IT Mastery practice items aligned to this topic area. They are designed for self-assessment and are not official exam questions.

Question 1

Topic: Testing, Validation, and Troubleshooting

A developer is troubleshooting an internal HR policy assistant that uses API Gateway, Lambda, Amazon Bedrock Knowledge Bases, and Bedrock Guardrails. After a parental leave policy update, validation prompts that mention maternity benefits return: “I can’t help with that request.” The team plans to rebuild the vector index.

Exhibit: Bedrock invocation log

Operation: RetrieveAndGenerate
KnowledgeBaseResults: 4
TopResultScore: 0.91
TopResultSource: s3://hr-kb/policies/parental-leave-v3.pdf
PromptTemplateVersion: hr-rag-prod-18
ModelInvocationStatus: Succeeded
GuardrailAction: INTERVENED
Finding: DeniedTopic "medical/benefits advice"

What is the best next step?

Options:

• A. Review the guardrail denied topic and rerun evaluations.

• B. Rebuild the vector index and force a full sync.

• C. Switch to a model with a larger context window.

• D. Add retries around the RetrieveAndGenerate API call.

Best answer: A

Explanation: The decisive signal is GuardrailAction: INTERVENED, not a retrieval, API, prompt, or model failure. The knowledge base returned the expected updated document with a high score, and the model invocation succeeded. The next step is to adjust the governance control and validate it with test cases.

Layered troubleshooting means identifying which component produced the failure signal before changing another layer. In this exhibit, retrieval is healthy because the knowledge base returned four results, including the updated parental leave policy with a TopResultScore of 0.91. The model layer also completed successfully. The blocking behavior is explained by the guardrail finding for the denied topic medical/benefits advice, so the appropriate fix is to review that guardrail configuration, refine the denied topic or exceptions, and rerun validation prompts. Rebuilding embeddings or changing models would target the wrong layer.

• Vector index rebuild fails because the exhibit shows the updated policy was retrieved with a high relevance score.
• Larger context window fails because there is no truncation, token limit, or missing-context signal.
• API retries fail because ModelInvocationStatus: Succeeded indicates the request completed rather than failed transiently.

Question 2

Topic: Testing, Validation, and Troubleshooting

An enterprise search team uses API Gateway and Lambda to call an Amazon Bedrock Prompt Flow that uses prompts from Bedrock Prompt Management and a Bedrock Knowledge Base backed by OpenSearch Service. Prompt, flow, and retrieval configuration changes are promoted by a deployment pipeline. The team must block promotion unless a fixed S3 regression set shows answer correctness and groundedness are not more than 2% worse than the current production baseline, with zero policy violations and auditable evidence. Which implementation meets these requirements?

Options:

• A. Attach Bedrock Guardrails and promote versions with no live blocks.

• B. Run Lambda unit tests and OpenSearch index health checks.

• C. Add a pipeline gate that starts a Step Functions staging evaluation workflow.

• D. Promote candidates to a production canary and monitor latency alarms.

Best answer: C

Explanation: The requirement is continuous regression testing with a quality gate before production promotion. A staging evaluation workflow can compare the candidate prompt, flow, and retrieval configuration with the current production baseline using the fixed dataset, then enforce the stated metric thresholds and preserve audit evidence.

Continuous evaluation for GenAI deployments should run before production alias or configuration promotion. In this scenario, the pipeline should invoke a staging workflow that exercises the candidate Bedrock Prompt Flow and Knowledge Base with the approved S3 regression set, compares results with the production baseline, checks correctness, groundedness, and policy-violation thresholds, and stores results in S3 and CloudWatch. Step Functions is a good orchestration mechanism because it can coordinate application invocations, evaluator calls, Lambda-based metric checks, and a pass/fail result back to the deployment pipeline. Runtime monitoring and guardrails remain useful, but they do not replace pre-release regression gates against a stable dataset.

• Canary monitoring fails because production traffic is exposed before the quality gate and latency alarms do not measure answer quality.
• Guardrails only fails because safety blocks do not prove correctness, groundedness, or retrieval quality against a baseline.
• Infrastructure tests fail because healthy Lambda code and indexes do not validate GenAI output regressions.

Question 3

Topic: Testing, Validation, and Troubleshooting

A team is choosing between two Amazon Bedrock prompt configurations for a customer support summarization workflow. The release gate requires no critical factual accuracy or task-alignment defects.

Exhibit: Evaluation summary

Scale: 1=poor, 5=excellent
Config A: relevance 4.8, factual 2.0, consistency 2.3, fluency 4.9, alignment 2.1
Config B: relevance 4.3, factual 4.5, consistency 4.4, fluency 4.4, alignment 4.6
Reviewer note for A: invents 24x7 phone support and refund promises.

What is the best interpretation and next step?

Options:

• A. Promote Config B and remediate Config A’s unsupported claims.

• B. Run latency tests before making a quality decision.

• C. Raise Config A temperature and re-run the same evaluation.

• D. Promote Config A because its relevance and fluency are highest.

Best answer: A

Explanation: Config A looks polished but fails the most important release criteria. The decisive exhibit details are the low factual accuracy and alignment scores plus the reviewer note that it invents support and refund policies.

FM output evaluation should consider relevance, factual accuracy, consistency, fluency, and task alignment together, not just whether the response reads well. In the exhibit, Config A has strong relevance and fluency, but factual accuracy is 2.0, consistency is 2.3, and alignment is 2.1. The reviewer note confirms the defect: unsupported claims about 24x7 phone support and refunds. Because the release gate blocks critical factual or task-alignment defects, Config A should not be promoted. Config B has slightly lower relevance and fluency but strong factual accuracy, consistency, and alignment, making it the safer production candidate.

• Fluency bias fails because Config A is well written but invents unsupported policy claims.
• Temperature tuning does not directly fix factual grounding or task-alignment failures and may reduce consistency.
• Latency testing first misses that the exhibit already shows a quality gate failure for Config A.

Question 4

Topic: Testing, Validation, and Troubleshooting

A GenAI platform team samples anonymized production prompt-response pairs and wants human reviewers to rate response quality, tag failure modes, and add annotations for later model and prompt comparisons. The architecture includes Amazon SageMaker Ground Truth. What role does Ground Truth serve in this feedback pipeline?

Options:

• A. Records API events for governance investigations.

• B. Applies runtime safety filters to model responses.

• C. Creates managed human labeling and annotation workflows.

• D. Collects distributed traces for request latency analysis.

Best answer: C

Explanation: Amazon SageMaker Ground Truth is used for managed human labeling and annotation workflows. In this GenAI feedback pipeline, it helps collect structured ratings, labels, and reviewer notes from production samples that can feed later evaluation and comparison work.

The core concept is using human annotation workflows to convert production quality signals into usable evaluation data. Ground Truth can manage labeling jobs, reviewer workforces, task instructions, and output annotations. For GenAI applications, teams can route sampled prompt-response pairs to reviewers to score helpfulness, identify hallucinations, classify safety issues, or add comments. Those structured labels can then support prompt regression testing, model comparison, and quality dashboards.

Runtime safety controls, audit logs, and tracing are useful in production architectures, but they do not provide managed human rating and annotation workflows.

• Runtime filtering is the role of guardrail-style controls, not a managed human annotation workflow.
• Audit events help prove who called which API, but they do not capture quality ratings on model outputs.
• Distributed tracing helps diagnose latency and service dependencies, not reviewer labels or annotations.

Question 5

Topic: Testing, Validation, and Troubleshooting

An enterprise runs a customer-support RAG assistant on Amazon Bedrock through API Gateway and Lambda. It uses Amazon Bedrock Knowledge Bases backed by OpenSearch Serverless, a CRM lookup tool, and Bedrock Guardrails. After a release that changed the prompt template and tool timeout, users report unsupported answers, but p95 latency remains below 2 seconds. The team must identify the failing layer, avoid raw PII in logs, keep evidence in the same Region for 90 days, and not add another model-serving stack. Which architecture is the best fit?

Options:

• A. Enable API Gateway and Lambda CloudWatch metrics and use p95 latency, throttles, and 5XX rates as the only debugging signals.

• B. Use X-Ray/CloudWatch plus CloudTrail audit events with redacted prompt versions, model IDs, retrieval hits/scores/filters, guardrail actions, and CRM tool status in same-Region KMS-encrypted logs.

• C. Route requests to a larger Bedrock model and raise token limits before adding retrieval or tool telemetry.

• D. Export full prompts, retrieved documents, and responses to a cross-Region S3 bucket for offline review.

Best answer: B

Explanation: The best design is layered, privacy-preserving observability rather than changing one component. Correlating prompt, model/API, retrieval, guardrail, and tool signals lets the team avoid fixing the wrong layer while meeting retention and locality constraints.

Troubleshooting GenAI applications should preserve the boundary between layers. Normal API latency does not prove the answer path is correct. The winning design keeps a single correlation ID across API Gateway, Lambda, Bedrock invocation, Knowledge Bases retrieval, guardrail evaluation, and CRM tool calls. The records should capture only redacted evidence: prompt/template version, model ID and inference settings, retrieval document IDs, scores, metadata filters, guardrail intervention category, and tool status or error. CloudWatch and X-Ray help trace runtime behavior, while CloudTrail plus KMS-encrypted same-Region retention supports auditability. This isolates whether the release broke the prompt, retrieval, model selection, tool integration, or governance layer without adding a new serving stack.

• API-only metrics show wrapper health but cannot explain unsupported answers when latency and 5XX rates are normal.
• Model-only change assumes model capability is the root cause and hides retrieval, prompt, guardrail, or tool failures.
• Raw cross-Region review violates the PII and data locality constraints and still lacks structured layer attribution.

Question 6

Topic: Testing, Validation, and Troubleshooting

An enterprise is releasing a customer-support Amazon Bedrock agent that can query a knowledge base and call Lambda-backed CRM and refund tools. Before each prompt, model, or tool-schema change, the team must automatically verify task completion, correct tool selection and JSON arguments, adherence to an approved observe-act-stop workflow, and safe stopping after a tool-call budget or ambiguous request. Tests must not write to production systems, and auditors need repeatable evidence. Which architecture best meets these requirements?

Options:

• A. Train a new SageMaker model on historical transcripts and approve releases when validation loss improves.

• B. Run API Gateway load tests against the production agent and approve releases when p95 latency and error rates meet the SLO.

• C. Apply Bedrock Guardrails to the production agent and approve releases when blocked-content counts stay below the threshold.

• D. Use a CodePipeline gate with Step Functions to replay tests against a staging agent alias with sandbox tools, score traces and outputs, and store metrics in S3 and CloudWatch.

Best answer: D

Explanation: The best fit is an automated agent-evaluation gate in the deployment pipeline. Replaying versioned test cases against a staging agent with sandboxed tools provides repeatable evidence while evaluating task completion, tool use, workflow adherence, and safe stopping.

Agent validation should inspect both the final answer and the agent’s visible execution events. A Step Functions workflow can run a fixed evaluation set against a staging Bedrock agent alias, call only sandbox CRM/refund integrations, and capture trace events, tool arguments, stop reasons, and outputs. Deterministic Lambda checks can validate JSON schemas, expected tool calls, tool-call budgets, and stop behavior. A Bedrock judge model can score semantic task completion when exact wording varies. CodePipeline can fail the release when thresholds are not met, while S3 and CloudWatch provide auditable reports and metrics. The key is to test the agent workflow before production release, not just content safety, latency, or model training loss.

• Guardrails only misses task completion, tool-argument quality, workflow adherence, and preproduction release gating.
• Load testing only validates performance, not whether the agent uses tools correctly or stops safely, and it risks production writes.
• Fine-tuning validation loss is a model-development signal, not an agent workflow evaluation for tool usage and safe stopping.

Question 7

Topic: Testing, Validation, and Troubleshooting

In an agent evaluation suite, a developer reviews agent traces and tool-call logs. Which definition best describes safe stopping behavior for a production GenAI agent?

Options:

• A. Caching repeated prompts to reduce model latency

• B. Selecting the highest-scoring retrieved document chunk

• C. Using a judge model to grade answer fluency

• D. Ending once the goal is met, blocked, or handed off safely

Best answer: D

Explanation: Safe stopping is an agent evaluation criterion for whether the workflow terminates at the right time and in the right state. It checks that the agent completes, refuses, asks for help, or hands off safely without unnecessary or risky continued actions.

Agent evaluation should examine more than final answer quality. For safe stopping, reviewers use traces, tool-call logs, and test cases to confirm that the agent stops after task completion, policy refusal, missing required information, escalation, or an iteration limit. This prevents loops, repeated tool calls, unsafe retries, and actions after the workflow should have ended. The closest confusions are semantic caching, retrieval ranking, and LLM-as-judge scoring; those help optimize or evaluate other parts of a GenAI system, but they do not define stopping behavior.

• Caching confusion fails because semantic caching targets latency and cost, not termination correctness.
• Retrieval confusion fails because chunk ranking evaluates RAG relevance, not agent workflow stopping.
• Judge-model confusion fails because LLM-as-judge can score outputs, but fluency alone does not prove safe termination.

Question 8

Topic: Testing, Validation, and Troubleshooting

A RAG application on AWS uses Amazon Bedrock to generate query embeddings and an Amazon OpenSearch Service vector index populated from an S3 document corpus. After a release, the team changed only the query-time embedding model; the indexed document vectors were not regenerated. Keyword search still finds the expected documents, but vector retrieval now returns low-score, unrelated chunks. Which troubleshooting concept best explains this failure?

Options:

• A. Preprocessing defect that removed source metadata

• B. Grounding failure during answer generation

• C. Embedding drift between query and document vectors

• D. Chunking error from oversized document segments

Best answer: C

Explanation: This is an embedding drift or vector-space mismatch problem. The document corpus was embedded with one model, while new queries are embedded with another, so nearest-neighbor similarity scores no longer represent true semantic relevance.

Vector search depends on query vectors and document vectors being produced in the same embedding space. If a release changes the query embedding model without re-embedding the indexed corpus, similarity comparisons can become unreliable even when the underlying documents still contain the correct content. The symptom is retrieval failure: unrelated chunks appear with poor scores while keyword search still works. The fix is to use the same embedding model and preprocessing pipeline for both sides, or rebuild the vector index after changing embeddings. Chunking and grounding issues can also affect RAG quality, but the release change points directly to embedding drift.

• Chunk sizing would usually cause missing or noisy context, not a sudden mismatch after only the query embedding model changed.
• Metadata removal affects filtering, attribution, or governance, but the stem describes semantic vector relevance degradation.
• Grounding failure happens during generation from retrieved context; here the retrieval step itself is returning unrelated chunks.

Question 9

Topic: Testing, Validation, and Troubleshooting

An insurance company uses an API Gateway and Lambda application that calls Amazon Bedrock Knowledge Bases backed by OpenSearch Service and a prompt managed in Amazon Bedrock Prompt Management. In CI, every prompt or chunking change must be evaluated for retrieval relevance, grounding quality, context completeness, and citation usefulness. The evaluation data must stay in AWS and retain auditable evidence. Which implementation should the team use?

Options:

• A. Run a Step Functions evaluation over a labeled query set using the same Knowledge Base and prompt version; store retrieved chunks, scores, answers, and citations in S3; compute metrics with Lambda and Amazon Bedrock; publish pass/fail results to CloudWatch.

• B. Export OpenSearch vector similarity scores after ingestion and approve releases when the average top-k score improves.

• C. Run synthetic conversations through API Gateway and approve releases based on CloudWatch latency, token usage, and 5xx error alarms.

• D. Use Amazon Bedrock to compare final answers to reference answers without storing retrieved chunks or citation mappings.

Best answer: A

Explanation: A RAG evaluation must inspect both what was retrieved and what the model generated from that context. The best implementation runs the same Knowledge Base and prompt path used by the application, captures retrieval and citation evidence, and scores the required dimensions with auditable artifacts.

The core concept is end-to-end RAG validation. Retrieval relevance and context completeness require seeing the top-k chunks, document IDs, metadata, and scores returned for known evaluation queries. Grounding and citation usefulness require comparing the generated answer and citations against those retrieved sources and expected references. A Step Functions workflow can orchestrate the repeatable CI job, Lambda can compute deterministic metrics and format judge prompts, Amazon Bedrock can perform rubric-based judging where needed, S3 can retain evidence, and CloudWatch can hold release-gating metrics.

The key is evaluating the same retrieval and prompt path that production uses, not only the final text or infrastructure signals.

• Operational metrics only miss relevance, grounding, context completeness, and attribution quality.
• Answer-only scoring cannot diagnose whether failures came from retrieval, missing context, or incorrect citations.
• Vector scores alone do not prove the generated answer is grounded or that citations are useful to users.

Question 10

Topic: Testing, Validation, and Troubleshooting

Which statement best defines a quality gate in continuous evaluation and regression testing for a production GenAI application on AWS?

Options:

• A. A tracing configuration that captures token latency and downstream tool calls

• B. An automated CI/CD checkpoint that compares candidate GenAI changes against baseline evaluation thresholds before deployment

• C. A runtime content filter that blocks harmful model inputs and outputs

• D. A prompt version label that records which template is currently in production

Best answer: B

Explanation: A quality gate is a deployment control, not just a monitoring or safety feature. In GenAI CI/CD, it uses automated evaluations and regression tests to decide whether a candidate prompt, model, retrieval configuration, or workflow can be promoted.

Continuous GenAI evaluation uses a repeatable test set and defined metrics to compare a proposed change with the current approved baseline. The quality gate is the pass/fail checkpoint in the release process. It can use tools such as Amazon Bedrock Model Evaluations, LLM-as-judge checks, retrieval relevance tests, grounding checks, and workflow task-completion tests. If the candidate fails required thresholds, the pipeline should stop or require review before production deployment. Runtime guardrails, version labels, and tracing are useful controls, but they do not by themselves establish pre-production regression acceptance criteria.

• Runtime filtering helps enforce safety during inference, but it does not compare candidate releases against baseline evaluation results.
• Prompt versioning supports traceability and rollback, but a label alone does not validate quality.
• Tracing telemetry helps troubleshoot latency and tool behavior, but it is not the release decision mechanism.

Continue with full practice

Use the AWS AIP-C01 Practice Test page for the full IT Mastery route, mixed-topic practice, timed mock exams, explanations, and web/mobile app access.

Try AWS AIP-C01 on Web View AWS AIP-C01 Practice Test

Related focused pages

• Free AWS AIP-C01 Full-Length Practice Exam
• FM Integration and Data
• Implementation and Integration
• AI Safety, Security, and Governance
• Genai Operations

Free review resource

Read the AWS AIP-C01 Cheat Sheet on Tech Exam Lexicon, then return to IT Mastery for timed practice.