Build a reviewed test set before optimizing prompts. Evaluate retrieval, answers, actions, and operating behavior separately, then judge the complete user task.
1. Define what acceptable behavior means
Choose a bounded task and write the criteria a reviewer can observe. For a support draft, the answer should address the question, reflect the current policy, avoid unsupported commitments, and use the correct account context. For an invoice, field validity and reconciliation matter more than persuasive wording.
Define unacceptable outcomes separately from minor style problems. A wrong customer record is not equivalent to an awkward sentence. Agree on which cases require a clarification, not-found response, or human handover, and make those valid expected outcomes rather than treating every refusal to guess as a failure.
2. Create representative cases with expected evidence
Collect sanitized examples from the intended process. Include common requests, rare but important cases, missing information, conflicting documents, and users with different access. Have someone who understands the domain review the expected behavior.
Keep prompt-development examples separate from held-out evaluation cases. A case record can include the request, authorized context, expected sources, and prohibited actions. The following is an illustrative test record for a knowledge assistant, not an API request format.
{
"case_id": "returns-policy-missing-exception",
"question": "Can this discontinued item be returned after the stated period?",
"user_role": "support_agent",
"expected_sources": ["returns-policy-current"],
"expected_behavior": "Explain the published rule and escalate the unsupported exception",
"must_not": ["Invent an exception", "Issue a refund"]
}3. Check retrieval, answers, and actions independently
For RAG, first verify that the expected passage is retrieved with the correct permissions and version. Then inspect whether the answer is supported, complete, relevant, and accurate. A citation can point to a real document while the sentence attached to it overstates the evidence.
For agents, inspect the selected tool, arguments, authenticated principal, approval state, and target-system result. A final message can look correct even when a tool call failed. Evaluate the user-visible outcome and the operations that produced it, including duplicated or partially completed actions.
4. Use human and automated checks appropriately
Automated checks work well for required fields, schema validity, prohibited tool use, source presence, and arithmetic. Human review is important for usefulness, ambiguous requirements, and meaningful business errors. A model can assist with grading, but its judgments also need calibration against reviewed examples.
Repeat important cases to observe variation, and record the model, prompt, retrieval configuration, and tool versions. Summarize results by request type and failure category as well as overall scores. Average performance can hide a serious weakness in a smaller but important group of cases.
5. Make evaluation part of the operating process
Choose a release gate appropriate to the task and its consequences. Include response time, operating cost, review effort, fallback behavior, and the ability to investigate an error. Launch with a defined audience and a way for users to report problems.
Keep a regression set and rerun it when models, prompts, documents, permissions, or integrations change. Turn meaningful production failures into new reviewed cases. Offline testing and a controlled pilot support a decision; neither guarantees that every future request will be handled correctly.
6. Frequently asked questions
Can an LLM grade another LLM?
It can help assess outputs, but calibrate the grading method against reviewed human examples and inspect disagreements. Automated grading is another component to evaluate, not an independent guarantee.
How many examples are enough?
There is no universal count. Cover the request types, document formats, access rules, and important exceptions your release supports. Expand the set when real usage reveals gaps.
Should we measure only answer accuracy?
No. Evaluate source support, completeness, permitted actions, failure recovery, latency, usage cost, and reviewer effort. The useful measurement depends on the product task.
7. Related articles and next steps
Bring representative inputs, expected outputs, and the systems involved. Discuss your requirements with OdooMatrix.