


Executive summary
When you give a contractor the keys to your office, you can call their references, check their licence and watch their first day’s work. When an AI agent asks for access to a company’s systems, there is usually one thing to go on: the agent’s own description of itself. An A2A agent lists its skills on its Agent Card. An MCP server lists its tools, with hints saying which ones only read and which ones can destroy data. Testing an agent against those declarations is the fairest way to judge it, but only if the test itself is right. A test that reports a failure when the setup was at fault does nearly as much damage as one that misses a real failure.
This post describes how we check our own verdicts. We built eleven agents that each break one promise in a known way, so that we know in advance what a correct evaluation should say about each of them. In brief:
- The eleven calibration agents, one honest and ten with a single deliberate flaw each, cover four families of failure: broken promises, unreliable delivery, moving targets and leaks, and gaps.
- Only two of the eleven deserve a red verdict. The other nine call for something more precise than pass or fail: inconclusive, mixed, invalidated, redacted or not tested.
- Every case that doesn’t pass is given an owner: the agent, the environment it ran in, or the test itself. In a recent evaluation of a production MCP server with 98 tools, all eight cases that didn’t pass traced to the test account or to our own safety stops. None was a defect in the server.
- Every change to how Suncly judges an agent is run against all eleven before it reaches a customer.
None of this is a new idea, and we don’t present it as one. Software testers have seeded programs with known faults to check their own tests for decades, and the A2A project’s compatibility kit already flags capabilities an agent declares but doesn’t implement. What we add is narrower: verdicts that distinguish an agent’s failure from its environment’s and from our own, and a fixed answer key that every change to our evaluation has to match before it ships.
Today we are sharing three things:
- This post, which sets out the four families and why each matters;
- A technical note describing each Lab agent, its flaw, and the verdict a correct evaluation should give it; and
- The Lab agents themselves, so that anyone building an agent evaluation can check their verdicts against the same answer key.
This is a progress report, not a finished study. We are publishing it so that others can test their evaluations against ours, and tell us where ours fall short.
Introduction
AI agents have moved from answering questions to taking actions: reading a company’s data, creating records, sending messages and calling other agents, often with no person reviewing each step. Two open protocols make this easier. The Agent2Agent protocol lets agents discover and delegate to one another; the Model Context Protocol lets an agent call tools on a server. Both ask the builder to describe what the agent or server does. Very few of those descriptions are ever checked.
Checking them is harder than it looks. Point an evaluation at a real agent and suppose it reports a failure. There are at least three explanations. The agent may have broken its promise. The environment may have got in the way, for example because the test account lacked a permission. Or the evaluation itself may be wrong. Each calls for a different response from the agent’s owner, and an evaluation that can’t tell them apart will be ignored, rightly.
The usual way to check a measuring instrument is to measure something whose value you already know. The Lab agents are our known values.
Why calibration matters
Four problems, in particular, make an uncalibrated evaluation misleading. Each of the eleven agents was built to catch at least one of them.
Problem 1: false reds
An unreachable endpoint, a slow answer or a missing example can all look like failures. None of them says anything about whether the agent keeps its promises. If they are shown in red, honest agents look broken, and the people reading the report learn to discount red, including when it matters.
How calibration helps: the unreachable, slow and no-examples agents are honest in every other respect. If an evaluation shows any of them as broken, the evaluation is wrong.
Problem 2: moving targets
An agent can change while it is being tested. A card that gains a skill halfway through a run, or a tool list that drops a hint, means the results describe something that no longer exists. Averaging them in produces a score for an agent nobody tested.
How calibration helps: the card-changer agent rewrites its own card mid-run. The correct verdict is not a lower score but an invalidated run, and a correct evaluation must notice the change before it reports anything.
Problem 3: leaked secrets
Some agents echo back what they are given, credentials included. An evaluation that faithfully records everything will then store, and pass around, the very secret it was trusted with.
How calibration helps: the leaky agent returns the caller’s credential in its answer. A correct evaluation flags the leak and makes sure the credential appears nowhere in its own transcripts, reports or logs.
Problem 4: gaps that look like passes
A skill with no examples, or a tool that was never called, has not been tested. If it quietly drops out of the count, a report can show a clean result over far less than the agent declares.
How calibration helps: the no-examples agent declares a skill with nothing to test it against. The correct verdict is “not tested”, listed by name, and counted neither for nor against the agent.
The eleven
The table lists each agent, the single flaw it was built with, and the verdict a correct evaluation should give it.
| Agent | What it does wrong | Correct verdict |
|---|---|---|
| honest | Nothing. It does what its card says. | Kept |
| honest-async | Finishes later, so the caller has to poll for the result. | Kept, after polling |
| unreachable | Publishes a card for an endpoint that refuses connections. | Inconclusive |
| lying | Declares skills and output formats it doesn’t honour. | Broken |
| flaky | Succeeds on odd-numbered calls and fails on even ones. | Mixed |
| slow | Answers correctly, but slowly. | Kept, with latency noted |
| direct-message | Replies with a plain message instead of creating a task. | Handled, not counted as a pass |
| interrupted | Always asks for more input and never finishes. | Broken |
| leaky | Echoes the caller’s credential back in its answer. | Flagged, and redacted from every record |
| card-changer | Changes its own card while it is being evaluated. | Run invalidated |
| no-examples | Declares a skill but gives no examples to test it with. | Not tested, listed by name |
What a correct verdict means
Calibration only works if each verdict has a precise meaning. These are the definitions the eleven are built to test:
- Kept means the declaration held on every repetition. A promise kept once and broken once has not been kept; that is Mixed.
- Broken means a confirmed mismatch between what was declared and what was observed. It is the only verdict shown in red, and it is never given on a single run or a split judgment.
- Inconclusive means our checks couldn’t decide. It is reported on its own and never counted as a pass.
- Invalidated means the thing under test changed during the test. The run produces no score.
- Not tested means a declaration we had no way to check. It is listed by name and earns nothing.
The definitions are deliberately strict in both directions. A verdict of Broken needs evidence; a verdict of Kept needs evidence on every run. Doubt either way resolves to Inconclusive, not to whichever answer is more convenient for us or for the owner.
Evaluation
We use the Lab in two ways. Before release, every change to our evaluation is run against all eleven agents and its verdicts compared with the table above; any mismatch blocks the change. After release, we look at real evaluations to see whether the families the Lab covers are the ones that actually occur.
A production MCP server. In a recent evaluation of an MCP server with 98 tools, we ran 36 test cases across four signed runs.1 28 passed, 4 failed and 4 were inconclusive. Each of the eight that didn’t pass had an owner. The four failures traced to the test account: it had no analytics connection for three tools and was on a trial plan for a fourth, and in each case the server refused correctly, with an honest error. The four inconclusive results were our own safety stops. Twice we couldn’t confirm that an object was ours before updating or deleting it, and twice our call would have broken the tool’s own schema, so we refused to send it. No case traced to the server.
What the report showed instead. An uncalibrated evaluation would have reported eight failures. Ours reported no defect, confirmed that every tool declared all four of its hints,2 and named the real limit: coverage. 32 of the 98 tools were called. The other 66, including every update, delete and send tool, were not exercised, and each is listed by name in the report. That coverage gap, not any defect, is what held the server’s score below Clear.3
Related work
Anthropic’s Frontier Red Team recently published progress on provable inference: a way for a model provider to commit to a set of model weights and then prove that each response came from them, without revealing the weights. One of the uses they describe is close to ours. Independent evaluators need to know that the model they tested is the model being served. Suncly faces the same question one layer up. Each of our reports is signed against a snapshot of the exact card or tool list we tested, so a changed declaration is visible to anyone holding the report. What we can’t yet show is that the agent answering in production is the agent we tested. Their work shows what that guarantee takes for a model, and how far the cost of it has fallen.
Calibrating a test against known faults is long-established practice in software testing, where it is called fault seeding or mutation testing. For agents, the A2A project’s compatibility kit checks that an agent conforms to the protocol and flags capabilities a card declares but the agent doesn’t support; it doesn’t test whether declared skills behave as described, which is where our tests begin. For MCP, a recent study of 37,288 runnable servers found that eight security scanners agreed with each other rarely and were right less than half the time, mostly because they judged tools by their descriptions rather than their behaviour. That is the problem calibration exists to prevent.
Work on failures in agent-to-agent systems points in the same direction. A recent taxonomy, drawn from more than 2,500 recorded runs across 488 scenarios in which each agent acts for a different principal, concludes that such systems have to be evaluated agent by agent, against each agent’s own principal. That is how we evaluate them.
Limitations
The Lab is a research tool, and it has the following limitations:
- One flaw at a time. Each Lab agent has a single flaw. Real agents combine them, and some failures appear only under load, with real data, or across long chains of actions.
- Calibrated, not representative. The Lab shows that a known flaw produces the verdict it should. It says nothing about how common each flaw is among the agents being deployed today.
- No injection probes yet. None of the eleven plants hostile text in an agent’s context. Our own behaviour tests for this are in development; until they ship, resilience to injection is reported as not tested.
- Destructive and outbound tools. We test update and delete tools only on objects we can prove we created, and send and invite tools only when the owner gives us a recipient they control. The eleven don’t yet cover either.
- Declarations, not deployments. A signed report binds to the declaration we tested, not to the code that later runs in production. Re-running on every change narrows that gap; it doesn’t close it.
- Judges can disagree. Where a check needs judgment, the model judge answers the same question three times. A split vote becomes Inconclusive, which is safe but not free: it leaves a question for a person to answer.