Question 220
At Delta Transit, a passenger support center is improving service disruption updates during a least-privilege verification (SF-D12-01). The administrator observes that users describe a known policy concept using varied language absent from the document title. Any fix must ensure that safety-critical changes need an attributable human approval. Which observable result would best demonstrate that the correction actually worked?
Explanation
Correct Answer
This is the direct acceptance signal for the reported failure: In a controlled Service_Alert__c acceptance test, paraphrased questions retrieve the relevant policy paragraph; the record is retained under Delta Transit's rollout (SF-D12-01). The chosen correction was to use semantic vector search over suitable indexed text passages; the measurement therefore tests the relevant source, routing or authorization outcome rather than unrelated activity.
Incorrect Answers
The proposed observation “Test output reflects exact account status with the right record ID; confirm this with production-equivalent source conditions” would check a different original case: An LLM response requires current account status stored in a structured DMO rather than long documents. The correction must ensure that shipment changes remain attributable and reversible. Its mechanism is: Structured record grounding differs from unstructured chunk search. In the present case, users describe a known policy concept using varied language absent from the document title. The correction must ensure that safety-critical changes need an attributable human approval; the more direct acceptance evidence is “Paraphrased questions retrieve the relevant policy paragraph; confirm this using a representative sandbox fixture” because vector retrieval matches semantic similarity rather than requiring exact words.
The proposed observation “A specific fault query retrieves the relevant troubleshooting section rather than the entire guide; confirm this with production-equivalent source conditions” would check a different original case: A twenty-page support guide is indexed as one oversized passage and retrieval is imprecise. The correction must ensure that employee procedures cannot be exposed to anonymous users. Its mechanism is: Chunking trades passage granularity against preservation of useful context. In the present case, users describe a known policy concept using varied language absent from the document title. The correction must ensure that safety-critical changes need an attributable human approval; the more direct acceptance evidence is “Paraphrased questions retrieve the relevant policy paragraph; confirm this using a representative sandbox fixture” because vector retrieval matches semantic similarity rather than requiring exact words.
The proposed observation “Answer traces show internal approved reimbursement sections as the grounding source; confirm this using a representative sandbox fixture” would check a different original case: A private reimbursement policy is being answered with generic public web snippets. The correction must ensure that plant access boundaries are not weakened by chatbot replies. Its mechanism is: Source boundaries should reflect whether knowledge is internal, external, or confidential. In the present case, users describe a known policy concept using varied language absent from the document title. The correction must ensure that safety-critical changes need an attributable human approval; the more direct acceptance evidence is “Paraphrased questions retrieve the relevant policy paragraph; confirm this using a representative sandbox fixture” because vector retrieval matches semantic similarity rather than requiring exact words.