Question 520
At Valley Utilities, a customer billing center is improving billing questions during a least-privilege verification (SF-D14-02). The administrator observes that a high-confidence answer cites an incomplete single chunk when the procedure spans several sections. Any fix must ensure that billing adjustments require verified eligibility. Which continuing control best prevents this issue from recurring?
Explanation
Correct Answer
The preventive control directly addresses repeat failures: Review recall and precision with a documented question set; require the check in the next release review at Valley Utilities (SF-D14-02). It complements the corrective action, adjust the retriever result count and chunk design to include required related context, and protects the explicit condition that billing adjustments require verified eligibility.
Incorrect Answers
This safeguard, “Maintain separate index and retriever change records; review it with a documented exception process”, reduces a different source-defined risk: A team creates a search index and expects prompt templates to query it without a retriever. The correction must ensure that billing adjustments require verified eligibility. That risk depends on: Search indexes organize content; retrievers select relevant indexed content for a use case. The recurring problem here is a high-confidence answer cites an incomplete single chunk when the procedure spans several sections. The correction must ensure that billing adjustments require verified eligibility. The originally selected control “Review recall and precision with a documented question set; review it with audit evidence from regression testing” is more specific because retrieval window and chunk boundaries together affect grounding completeness.
This safeguard, “Test representative paraphrases and inspect irrelevant near-matches; review it after configuration or data-source updates”, reduces a different source-defined risk: Users describe a known policy concept using varied language absent from the document title. The correction must ensure that safety-critical changes need an attributable human approval. That risk depends on: Vector retrieval matches semantic similarity rather than requiring exact words. The recurring problem here is a high-confidence answer cites an incomplete single chunk when the procedure spans several sections. The correction must ensure that billing adjustments require verified eligibility. The originally selected control “Review recall and precision with a documented question set; review it with audit evidence from regression testing” is more specific because retrieval window and chunk boundaries together affect grounding completeness.
This safeguard, “Retire obsolete chunks and review source precedence rules; review it with a documented exception process”, reduces a different source-defined risk: Two overlapping libraries contain conflicting versions of the same return policy. The correction must ensure that copyrighted manuscripts remain within the approved workspace. That risk depends on: Duplicate knowledge sources can make retrieval ambiguous even with correct prompts. The recurring problem here is a high-confidence answer cites an incomplete single chunk when the procedure spans several sections. The correction must ensure that billing adjustments require verified eligibility. The originally selected control “Review recall and precision with a documented question set; review it with audit evidence from regression testing” is more specific because retrieval window and chunk boundaries together affect grounding completeness.