5 minute read
In this article
- Every score has a reference
- 1. Measured against what?
- 2. Who keeps the reference current?
- 3. Can the judge see the answer you hid?
- 4. What does no score mean?
- 5. Did the evaluator actually run?
- 6. Has the evaluator ever been made to fail?
- A result record that keeps the claims apart
- What this does not show
An agent approves a return. The request returns HTTP 200. Two built-in evaluators score the answer 1.0 for faithfulness and 1.0 for correctness. The return should have been refused.
In a small CloudWatch Omni lab, the agent retrieved a 60-day return policy that a 30-day policy had superseded. The order was 45 days old. Everything downstream agreed with the retrieved document, so everything downstream passed. An explicit freshness check against the current policy was the only signal that failed.
It is tempting to read that as “the evaluators were wrong”. I think that is the wrong lesson. They answered the question they were given. The trouble is that a passing score rarely says which question that was.
This article is a way of reading evaluation scores, built on that one lab. It contains no new experiment. The figures come from a synthetic AWS sandbox run on 27 September 2026, with three attempts per scenario.
Every score has a reference
An evaluator compares an output against something: the retrieved context, supplied ground truth, a rubric, a runtime status. That something is the reference, and a score can only vouch for the output relative to it. The table lays out what each signal in the lab was measured against, and what a pass could not tell me.
| Signal | Measured against | A pass supports | A pass cannot support |
|---|---|---|---|
| HTTP 200 | Request and runtime completion | The call finished | That the answer was right. The stale-policy requests returned 200. |
Lambda Errors metric at zero | Unhandled invocation failures | No unhandled exception escaped | That nothing failed. The denied S3 reads returned 500 while the metric stayed at zero, because the handler caught the exception. |
| Faithfulness | The retrieved context | The answer follows what was retrieved | That the retrieved content is authoritative or current. Both valid stale-policy trials scored 1.0. |
| Generic correctness | Facts supplied in the trace | The answer is consistent with those facts | That those facts are the business truth. It accepted 45 days inside a 60-day window. |
| Freshness check | Independently defined constants | One named rule held | Anything outside that rule. It checks structured fields, not every sentence of the explanation. |
None of these signals is useless. Each answers a real question. The risk is a dashboard that shows them all in the same shade of green, so that “the request completed” and “the policy was current” look like claims of equal weight.
1. Measured against what?
Write the reference next to the score. “Faithfulness: 1.0” becomes “consistent with the document the agent retrieved”. That sentence is true and harmless. The bare number invites a stronger reading.
In the lab, both valid stale-policy trials scored 1.0 on both built-in evaluators, because the answer did follow the retrieved 60-day policy. The correctness explanation accepted eligibility because 45 days fits inside the supplied 60-day window. The reasoning was sound. Its premise was out of date.
2. Who keeps the reference current?
A reference the agent retrieves for itself cannot audit that retrieval. If an evaluator trusts a document’s own claim to be current, it inherits the same failure.
The check that caught the defect used independent constants: the active policy version (2026-09), the active window (30 days) and the order age (45 days). That works for a frozen fixture. In production I would want a governed policy source with effective dates, product scope and exceptions, maintained by the people who own the policy rather than the people who own the agent.
3. Can the judge see the answer you hid?
When you test whether an evaluator catches a wrong answer, check that the right answer is not sitting in the trace. In the lab’s corrupted-answer scenario, the native trace kept the model’s original response beside the replaced, customer-visible one, so the correct answer was recoverable.
I therefore re-ran the two valid traces as derived copies with that metadata removed and the final output replaced by the customer-visible answer. Both built-in evaluators still scored 0.0 on both copies. That supports a narrow claim: the blatant contradiction, 45 days approved under a 30-day rule, was detected without the hidden answer.
It also shows something about the stale case. The same evaluators that passed the stale answer caught the contradiction in the corrupted one. The difference was the evidence available to them, not the tool.
4. What does no score mean?
The tool-failure scenario produced three real 500 responses from an explicit S3 AccessDenied. With no policy evidence, there was nothing meaningful to check, so the custom evaluator reported missing evidence instead of a pass.
That distinction matters. “No failing score” and “passing score” are different states. A pipeline that treats an absent result as success will look healthy in exactly the cases where it has no evidence.
The same discipline applies to trials. Two attempts in the main run returned tool names with unexpected suffixes, which the allowlist rejected. I kept them as invalid trials rather than normalizing them into valid calls. They are excluded from the valid-trial counts, but they stay in the record.
5. Did the evaluator actually run?
Omni’s operator role initially lacked lambda:InvokeFunction for my custom evaluator. The console wrote the two built-in scores and reported the custom failure. A failed evaluator invocation is not a failing quality score. Reading it as one would be a mistake in the opposite direction from the stale-policy case: a broken check dressed up as a verdict.
After I granted invoke access to that one function and retried only the custom evaluator, it wrote the 0.0 result on the stale trace. Record “evaluator error” as its own status, never as a low score.
6. Has the evaluator ever been made to fail?
Before trusting a judge, hand it a case you know is bad, and a case you know is good. The lab had a baseline that passed all three attempts on every check, which shows the checks are not simply failing everything. It also had three deliberately broken setups: a denied tool call, a stale document and a corrupted final answer.
The corrupted-answer scenario is a detector-sensitivity test. It says nothing about how often the model produces wrong answers on its own. Keep those two questions apart, or a good detection result turns into an unearned reliability claim about the agent.
A result record that keeps the claims apart
One practical change follows from all this: stop storing a single number per trace. Store the reference, the status and the score separately. The shape below is illustrative. It is a design suggestion, not something the lab implemented in this form.
{
"trace_id": "...",
"checks": [
{ "name": "faithfulness",
"reference": "retrieved_context",
"status": "scored", "score": 1.0 },
{ "name": "policy_freshness",
"reference": "policy_registry@2026-09",
"status": "scored", "score": 0.0 },
{ "name": "policy_freshness",
"reference": "policy_registry@2026-09",
"status": "unscorable",
"reason": "missing_tool_evidence" },
{ "name": "policy_freshness",
"status": "evaluator_error",
"detail": "AccessDenied on lambda:InvokeFunction" }
]
}The last three entries show different outcomes for the same check, not one trace producing all of them. A dashboard built on this can show “scored”, “unscorable” and “evaluator error” as three different states, and group scores by reference. A pass that is grounded and fresh then stops looking identical to a pass that is only grounded.
What this does not show
This rests on one synthetic scenario: one model, one policy rule, three attempts per scenario and two invalid controls. It is not a benchmark of evaluator accuracy, and it does not estimate how often generic evaluators miss stale evidence in a real system.
The custom check compares structured fields. It does not verify every sentence of the free-text explanation. And the built-in evaluators were never given the current policy, so their 1.0 scores are not an indictment of them. The aim here is narrower: to read a pass as a claim with a stated reference.
The question I would carry into a launch review is short. For every green check on this agent, what is it measured against, and who would notice if that reference went out of date?
Leave a Reply