Skip to content

One agent, three evaluations, nothing hidden.

Pick an evaluation, open any run, see what the signature covers, and compare the first two.

The agent
Fictional. Harbor Returns Agent, its orders and its reviewer are not real.
The output
Real. Every hash, verdict, transcript and signature was produced by the Suncly code path against a local mock. Not a customer evaluation.

Three evaluations of one agent

Same card hash, so the same approved plan runs again. start-return now stops at TASK_STATE_INPUT_REQUIRED on every run, and the second order-status example fails two runs in five: an inconsistency that repeating every test makes visible and one manual prompt would not.

Compare 1 and 2
Sample dataattestation 5f7f23bb · trigger manual

Harbor Returns Agent

Card version 2.3.1 · owner Harbor Commerce platform team (fictional) · started 11 Oct 2026, 21:09 UTC

Evaluation status

Completed

Every planned run was judged, the card was unchanged at the end, a decision was recorded and the attestation was signed.

Approval status

Flag: human review · policyRisk medium

Recorded by the Policy engine. No policy is configured, so a human must review this result; approve and block are never produced automatically in this version.

Needs a human decision

The Policy engine recorded flag. A reviewer has to decide on this evidence.
Pass8runs with every check passed
Fail7runs with a failed check
Inconclusive5never counted as a pass
Runs recorded20 / 20every planned run recorded

Inconsistent behaviour on 1 test case

The same input passed on some runs and failed on others: “Has order 48213 shipped yet?”. One manual prompt would have shown only one of those outcomes.

1 test case undecided

Every run was inconclusive: “How much will I get back if I return order 48213?”. Open the run evidence to see which check Layer 1 could not decide.

Results per test case

Pass, fail and inconclusive side by side, never combined into a score. Select a row to open its runs.

Results per test case: pass, fail and inconclusive counts
SkillTest inputSignalPassFailInconclusive
Order statusorder-status“Where is order 48213?”Passing500
Order statusorder-status“Has order 48213 shipped yet?”Inconsistent320
Start a returnstart-return“Start a return for order 48213, item 2”Failing050
Refund estimaterefund-estimate“How much will I get back if I return order 48213?”Undecided005
Totals across 4 test cases. Counts, not a score.875

What was not tested

Straight from the report. An approval on this evidence is an approval with these gaps.

  • skill without test caseskill 'cancel-order' has no test case; invariant 1 (one test case per declared skill) is not satisfied for it
  • inconclusive runs5 run(s) ended inconclusive; they count neither as pass nor as fail
  • declared capability not exercisedthe card declares streaming; no test case exercises it
  • probesno probe_undeclared, probe_injection or probe_failure test cases exist yet (stage 4)
  • semantic correctnessLayer 1 checks structure, state, output modes and latency; 1 test case(s) carry model checks, but no judge model is configured (judge_model), so their runs are inconclusive
  • production endpointtests ran against the declared sandbox or dry-run endpoint http://127.0.0.1:54312/rpc; the production endpoint itself was not tested (DR-006)

What changed between evaluation 1 and 2

The card hash is identical, so the same approved plan ran both times. The counts are comparable; the agent is not the same.

Earlier

attestation 4049408e

11 Oct 2026, 21:09 UTC

CompletedFlag: human review · policy

Later

attestation 5f7f23bb

11 Oct 2026, 21:09 UTC

CompletedFlag: human review · policy
  • Agent CardUnchangedSame card_hash. An unchanged card is not proof of an unchanged agent; the results below are.
  • Test conditionsSame contractContract version 1, approved by m.lind@harbor.example.
  • Evaluation statuscompleted
  • Test cases2 regressed · 0 improved2 unchanged, 0 mixed, 0 without runs on one side, 0 added, 0 removed
Per test case comparison of pass, fail and inconclusive counts
SkillTest inputEarlier (pass / fail / inconclusive)Later (pass / fail / inconclusive)Change
Order statusorder-status“Where is order 48213?”5 / 0 / 05 / 0 / 0
Unchanged
Order statusorder-status“Has order 48213 shipped yet?”5 / 0 / 03 / 2 / 0
Regressed
Start a returnstart-return“Start a return for order 48213, item 2”5 / 0 / 00 / 5 / 0
Regressed
Refund estimaterefund-estimate“How much will I get back if I return order 48213?”0 / 0 / 50 / 0 / 5
Unchanged

Computed in this browser from the two signed bundles; the command-line tool has no comparison command. What counts as a “drop” for the Policy engine is still an open question in the design documents.

The reviewer's decision

Suncly records flag and stops there. The reviewer records the decision in the workspace and exports it with the signed report, outside the signature.

Block · reviewerHuman decision, outside the signed evidenceSample

start-return no longer completes from a single message: all five runs stop at input-required, while evaluation 1 completed five of five against the same card. order-status is inconsistent on the shipping question (3 pass, 2 fail with upstream timeouts). refund-estimate remains inconclusive pending a semantic check. Block until the returns flow completes without extra input; re-evaluate with the same contract.

m.lind@harbor.example · attestation 5f7f23bb-e9ba-448c-b099-b194fb7f81cd

What remains untested

  • cancel-order: the card declares no examples, so there is no test case. Ask for examples, or write them into a contract file.
  • Whether refund amounts are correct: the check needs the model judge, and this sample ran without one.
  • Streaming: declared by the card, never exercised.
  • Prompt injection and behaviour outside the declared skills: this sample was made before the probes ran by default.
  • The production endpoint: only the declared sandbox was called.

Open them in the review workspace

Load the three evaluations to try the agent history, the comparison and a review note. Everything loaded from here is marked as a sample and stays in your browser.

Bring one agent. Leave with evidence.

Order a verification of one A2A agent or MCP server. Charged only when your report is delivered; usually within 1–2 days.

Get verifiedTalk to us

Or write to team@suncly.com.