HTTP 200, Wrong Answer: Testing Agent Evaluations with CloudWatch Omni

The agent approved a return. The request returned HTTP 200. The answer matched the policy it had retrieved. Amazon Bedrock AgentCore’s built-in faithfulness and correctness evaluators both gave it 1.0.

The return was still wrong.

The retrieved document allowed 60 days. The current policy allowed 30. The order was 45 days old. An explicit freshness check, running as a custom AgentCore evaluator, returned 0.0 / FAIL on the same trace.

That is the central result of this small CloudWatch Omni lab: agreement with retrieved evidence does not establish that the evidence is still authoritative. A quality score needs a clearly stated source of truth.

This is an observed result from a synthetic AWS sandbox experiment on 27 September 2026, not a claim about evaluator accuracy in general. I ran three attempts per scenario, retained unsuccessful controls, and separated local deterministic checks from actual managed evaluations.

A small application with one unambiguous business rule

The question was deliberately simple:

Can I return unopened standard order ORD-1001? Check the order and returns policy.

DynamoDB held a synthetic order: standard item, unopened, delivered 45 days ago. S3 held two policy documents:

The current policy was independent test truth. It was not added to the built-in judges’ context when the agent retrieved the old document.

An IAM-authenticated API Gateway endpoint invoked a Python Lambda function. The function ran a bounded tool loop with get_order and get_return_policy, backed by real DynamoDB and S3 calls. The final model was openai.gpt-oss-20b through the regional Bedrock Mantle Chat Completions endpoint in us-east-1, authenticated with temporary role credentials and SigV4.

An illustration of the deployed components, not a screenshot of an automatically discovered application map. The custom trace starts inside Lambda; API Gateway access logs carry the HTTP status and request ID separately.

The original package used Bedrock Converse. Nova model quotas in this sandbox blocked invocation, so I adapted the transport to Mantle while retaining the same fixtures, question and bounded tool loop. This is a custom client-side agent, not the managed Bedrock Agents service. Policy retrieval is a direct S3 document lookup, not vector RAG.

Four scenarios, with controls that can fail

The baseline retrieved the current document. Three controlled failures followed:

• Tool failure: request an S3 object that the Lambda role explicitly denies. This produces a real AccessDenied, rather than a simulated exception.
• Stale policy: retrieve the obsolete 60-day document. The cloud services can succeed and the model can faithfully follow the wrong source.
• Wrong final answer: retrieve the current policy and let the model answer, then deliberately replace the customer-visible result with an incorrect approval. The raw model response remains in the evidence.

The last case is a detector-sensitivity test. It says nothing about how often the model naturally produces wrong answers.

Requests ran serially, with at most four model calls, 384 completion tokens per call, and six tool calls. The runner stopped on an invalid baseline. A sandbox concurrency restriction prevented the planned Lambda reservation, so an atomic DynamoDB lease serialized agent work instead. Its 60-second expiry exceeds the function’s 28-second hard timeout. This limits concurrent agent execution; it is not a guarantee that only one Lambda wrapper can be invoked.

I did not normalize unexpected tool names into valid calls. Two attempts in the main run returned names containing <|channel|> suffixes. The allowlist rejected them. They remain invalid controls in the results below.

Proving ingestion before trusting the dashboard

I treated these as separate checks: a successful model response, a valid application answer, successful export, an ingested trace, agent classification, and a returned evaluator score.

That separation caught a real setup issue. An early request returned an exporter error while Transaction Search was still pending. A local trace ID was not proof that CloudWatch had accepted its spans. After the destination became active, a new baseline produced eight unique spans in aws/spans, and the same trace was queryable in Omni’s Dataset.

The instrumentation used upstream OpenTelemetry with the documented generic OpenInference scope and message attributes. AWS enrichment added agent/model/tool classifications, and Omni’s agent trace view showed the input, final output, model calls, tools, token counts and timeline. This particular custom instrumentation path worked in this lab; that is narrower than claiming universal compatibility. AWS’s generic evaluation integration guide describes the conventions involved.

The Dataset integration forwarded only the lab’s two log groups and aws/spans. Transaction Search settings were snapshotted before changing the trace destination from X-Ray to CloudWatch Logs and indexing from 0% to 100%. No continuous online evaluation was enabled. AWS’s telemetry guide explains the separate forwarding integration and its role permissions.

What actually happened

The main run took place from 13:04:33 to 13:05:08 UTC on 27 September 2026. It contained 12 attempts, in a fixed order. The managed scores below came from on-demand AgentCore Evaluate calls using the actual CloudWatch span records.

The local oracle independently checked structured fields against the retrieved policy and the current policy. It agreed with the business interpretation: the stale answers were grounded in the document but wrong against current policy. Those local checks are not the managed scores in the table.

A separate calibration stale-policy trace produced the same result. I evaluated it again through Omni’s UI so the contrast was visible in the console:

Actual Omni screenshot: the same stale-policy trace receives 1.00 from each built-in evaluator and 0.00 from the explicit current-policy check. The UI run is a separate evaluation of the calibration trace, not another application trial in the 12-request table.

The calibration trace is 0ec3e9df1f1d6256cb6c6606b18fe3b6. Its retrieved policy and answer identify version 2026-06. The correctness evaluator’s explanation accepts eligibility because 45 days is within the supplied 60-day window. It does not establish that this was the current business rule.

That is a useful distinction. The faithfulness evaluator did what its context supported. The generic correctness score also accepted the supplied facts. Neither had an independent policy registry telling it that September’s rule superseded June’s.

The freshness check needs independent truth

I deployed a second Lambda function and registered it as a trace-level, code-based AgentCore evaluator. It reads the root’s customer-visible output and both tool outputs from the supplied spans. Its independent constants are:

ACTIVE_VERSION = "2026-09"
ACTIVE_WINDOW = 30
ORDER_AGE = 45

A pass requires the retrieved policy to match the active version and window, and the final structured answer to identify the order as ineligible under that policy. Missing tool evidence produces MISSING_OR_INVALID_EVIDENCE, rather than an accidental pass. The implementation follows the AgentCore code-based evaluator event and response contract.

This evaluator intentionally has more information than the built-ins. It is not evidence that a custom judge mysteriously reasoned better. It demonstrates the value of checking the specific invariant the generic judges were not given.

In a production system, I would obtain that truth from an independently governed policy source with effective dates, product scope and applicable exceptions. Copying the retrieved document’s own “current” claim into the evaluator would recreate the same dependency. This lab’s exact version comparison is useful for a frozen fixture; it is not a complete policy-governance system. The code also checks structured fields, not every assertion in the free-text explanation.

There was an integration repair here too. The direct API evaluation succeeded, but Omni’s operator role initially lacked lambda:InvokeFunction for the custom evaluator. The UI wrote the two built-in scores and reported the custom failure. I retained that evidence, granted invoke/get access to this one lab function, and retried only the custom evaluator. It then wrote the 0.0 result. A failed evaluator invocation must never be confused with a failing quality score.

Checking that the wrong-answer test was not leaking the answer

The native wrong-answer traces retained the original model response as well as the corrupted customer-visible root output. That makes them useful for debugging, but the correct pre-corruption answer is also available in the native trace.

I therefore ran an additional check on derived copies of the two valid wrong-answer traces. The copies removed harness metadata and replaced the final model-output field with the actual customer-visible answer. Tool evidence and message-call correlations remained. Original CloudWatch records were untouched; the transformation and source hashes were saved.

Both built-in evaluators still returned 0.0 for both blinded copies. This supports the narrow conclusion that the obvious 45-days-within-30-days contradiction was detected without exposing the hidden correct final answer. These are additional evaluator calls on derived inputs, not extra live requests and not a larger accuracy benchmark.

Operational evidence still matters

All three intended tool-failure requests reached the denied S3 object. The API returned 500, and the span evidence identified s3:GetObject, the denied object and the explicit IAM deny.

Lambda’s Errors metric remained zero in the collected minute buckets. That is consistent with this handler: it catches the exception and returns an HTTP response through API Gateway. A handled application failure does not automatically become a Lambda invocation error. The API status, application record and error span carry the operational failure.

Omni’s trace rows exposed these faulted requests. I also observed a view whose aggregate “Traces with errors” summary read zero while individual rows were marked Fault. I preserved the screenshot and used the underlying spans and API records for the experiment’s classification. This lab does not establish the cause of that UI discrepancy.

For the stale-document case, the useful evidence was different: successful dependencies, a 60-day tool result, a matching approval, and the explicit freshness failure. Looking only for infrastructure errors would not establish the business defect.

Where the natural-language investigation stopped

I also asked Omni to find completed requests with failing evaluations in the 13:03–13:06 UTC window, show trace IDs and policy evidence, and expose the queries it used. The background task reported finding 15 traces, but exhausted its query budget before retrieving evaluation scores or policy-evidence details. That window included calibration traffic as well as the 12-request main run.

This was an incomplete investigation, not a successful automated diagnosis. I retained the prompt, response and console screenshot. The evidence for the article comes from the bounded SQL results, saved spans, direct evaluator responses and explicit UI scoring. I did not measure an investigation-speed improvement or keep retrying until the assistant produced a desirable answer.

Reproducing the investigation

The evidence package retains request/response JSON, CloudWatch log events, span exports, native and blinded evaluation requests, returned explanations, Terraform state snapshots, screenshots and a progress log. The implementation passes 24 offline tests covering the tool loop, baseline stop gate, telemetry, transport adapter, serial lease and custom evaluator.

This bounded Omni SQL query retrieved the main run’s spans:

SELECT traceId, name, attributes
FROM traces.default
WHERE `@timestamp` BETWEEN
  to_timestamp('2026-09-27T13:03:00Z')
  AND to_timestamp('2026-09-27T13:06:00Z')
LIMIT 200

In this account, explicit to_timestamp bounds worked; an attempted query with only a lower relative bound was rejected by time-range validation. Keep the returned query and error rather than assuming every documented example behaves identically. The Omni SQL reference describes the signal tables and field access.

To inspect the central example in Omni, open Agent traces, include the recorded time window, search for the calibration trace ID above, and inspect the evaluation badges and get_return_policy output. Direct AgentCore API results were saved locally; I did not assume they automatically populated the Omni evaluation dashboard. The screenshot’s persisted badges came from the explicit Omni UI run.

Cost, limits and the retained walkthrough environment

The authorized sandbox ceiling was $100, while the operational plan kept this run small. There was no load generator, provisioned model capacity, NAT gateway, vector store or continuous evaluation configuration.

The saved direct built-in evaluation responses account for 23 evaluations, 23,803 input tokens and 4,360 output tokens. Applying the rates in AWS’s published pricing example gives about $0.11 for that subset alone. This excludes UI scoring, model inference, custom evaluations, Lambda, telemetry, queries and storage; it is not the total lab bill. AgentCore pricing and Omni pricing should be checked for the chosen Region and usage.

The Billing console refreshed at 13:10 UTC still showed a pending estimated grand total of $0.00 and a highest-region display of $0.05. Those delayed account-level figures are not proof of a free run. Cost Explorer was not enabled, and a final attributable cost is not yet available.

The lab remains deployed for a walkthrough recording at my request. Cleanup is deliberately deferred. The resource inventory and original observability settings are saved so teardown can remove the lab and restore only its tracked changes. Storage and any new queries or invocations can continue to incur charges while it remains available.

This is a small demonstration: three attempts per scenario, two invalid controls, one model and one simple policy rule. There is no detection-rate benchmark, latency percentile claim or measured investigation-speed improvement here.

The result I would carry into a production design is more specific: a grounded answer can still be wrong when its evidence is obsolete. Instrument the evidence, calibrate the judge, and give freshness its own independently defined check.