Skip to content

Check that an MCP server's tools do what they say.

An MCP server gives AI agents a set of tools behind one endpoint: search an inbox, draft a reply, send an email. Each tool describes what it does, and nothing checks that description. Suncly lists every tool, calls only the ones that are safe to call, checks each result against the tool's own definition, and reports what it saw.

Verify an MCP server

From $100 · charged only when your report is delivered · usually within 1–2 days

One endpoint, many tools.

The Model Context Protocol (MCP) is how an AI agent uses tools on a server: one URL, one list of tools. Each tool has a name, a description, an input schema, sometimes an output schema, and annotations such as readOnlyHint that say whether it changes anything. All of it is the server's own claim.

Suncly tests two kinds of target, each in its own way. A2A agents

How Suncly tests an A2A agent and an MCP server
AspectA2A agentMCP server
What is testedAn agent that takes tasks, against the skills its Agent Card declares.The tools a server offers, against each tool's description, schemas and annotations.
Pinned toThe SHA-256 hash of the Agent Card.The SHA-256 hash of the tool snapshot: every tool, every page of the list.
Called by defaultEvery approved test case.Only tools that declare readOnlyHint: true. Any other call needs a person's approval.

One verification, start to finish.

Six steps, in this order. Beside each, what that step recorded when Suncly attested the bundled mock server mcp-mixed.

  1. Take the tool snapshot.

    An isolated Runner process connects to the endpoint, works out which MCP revision the server speaks and lists every tool. The snapshot is hashed, and the contract and the report name that hash. Nothing is called yet.

    Tool snapshot
    Server
    suncly-mock-mcp 1.0.0 (as it reports itself)
    Protocol
    MCP 2026-07-28 (stateless)
    Tools
    6 listed
    Snapshot hash
    sha256:61d8c333…2e7a8

  2. Draft the calls.

    Only tools that declare readOnlyHint: true are drafted. A test case is one call: a tool and its arguments, taken from examples the server publishes in the tool's input schema. A tool that publishes none gets no call unless it takes no arguments, and the report says so.

    Calls, drafted
    • lookup_word{"word": "apple"}1 call
    • failing_lookup{"city": "Paris"}1 call
    • broken_schema{"city": "Oslo"}1 call
    • ask_user{}1 call
    • unannotated_echoleft out: no annotations
    • delete_recordleft out: readOnlyHint: false, destructiveHint: true

  3. A named person approves.

    The approval shows every call in full, each tool's annotations, and every tool that will not be called and why. Nothing is called before a person approves.

    Approval
    Approved by
    alice
    At
    17 Oct 2026, 23:01 UTC
    Plan
    4 calls × 2 runs = 8 runs
    Budget
    16 attempts

  4. Re-check, then call.

    Before every call the Runner lists the tools again. If the list changed, nothing is called and the attestation ends invalidated. The Runner itself refuses a tool that is not declared read-only unless a person approved that call.

    8 runs, re-listed before each
    • lookup_word
    • failing_lookup
    • broken_schema
    • ask_user

  5. Check every result.

    Each run gets pass, fail or inconclusive. A protocol error, a result marked isError, missing content, or structuredContent that breaks the tool's output schema fails the run. Inconclusive never counts as a pass.

    Why runs failed
    • failing_lookupis_error; the tool said: Upstream population service unavailable.
    • broken_schemastructured_content: structuredContent.celsius: 'warm' is not of type 'number'
    • ask_userthe server asked for input instead of answering

  6. Report.

    The snapshot is taken again at the end, so a server that changed its tools during the run shows up. The report holds every count and every transcript, and ends with what was not tested.

    Report
    Snapshot re-check
    unchanged
    Decision
    flag · a person reviews
    suncly-reports/<attestation>/
  report.html
  report.md
  result.json
  manifest.json
  transcripts/<run>.json

What is called, and what never is.

A tool that sends an email is not something a test should fire on its own. Suncly calls a tool by default only if it declares readOnlyHint: true. Every other tool is left out, named in the report with what it declares, and called only when a person approves that one call against a server they declared safe to test.

Example: an email server's toolsAn illustration of the rule, not a server Suncly has tested.
  • search_inboxdeclares readOnlyHint: trueCalleddrafted from the examples in its input schema
  • draft_replydeclares no annotationsNot calledunless a person approves the call
  • send_emaildeclares readOnlyHint: falseNot calledunless a person approves the call

To test a tool that changes state

Run against a server you declared safe to test, add the call to a contract file and set allow_state_change on it. The approval prompt shows the flag next to the tool's annotations, and the report says that calling the tool was approved by you, by name.

  • A tool that declares both readOnlyHint: true and destructiveHint: true contradicts itself and is treated as not read-only.
  • A tool whose execution.taskSupport is required, or whose x-mcp-header breaks the specification's rules, is never called, whatever a person allows.
  • Annotations are the server's claims. Suncly trusts readOnlyHint only to call less, never more, and every report says that whether a read-only tool really changes nothing was not tested.
  • If a server asks for input during a call, Suncly never answers. Suncly declares no client capabilities, so a compliant server does not ask, and the run fails.

What it checks.

Structural checks decide first, on every run. A judge model is used only for what they cannot decide.

Structural checks, on every run

complete_result
The call must end in a complete tool result. A server that asks for input instead fails: Suncly declared no client capabilities, so a compliant server does not ask.
jsonrpc_error
A JSON-RPC error fails the run, unless the test case lists that code as accepted. An error whose fault may be Suncly's is inconclusive instead.
is_error
A result marked isError: true fails, with the tool's own error text in the report.
structured_content
When the tool declares an output schema, structuredContent must be present and match it (MCP 2025-06-18 and later).
valid_result
The result must be a well-formed tool result: a content list whose blocks are valid.
content_present
The result must carry content, not an empty answer.
must_not_contain
Strings you name in a contract file must not appear in the result.
latency_limit
Only when a person sets a limit in a contract file. Otherwise latency is recorded as information and fails nothing; no answer within the timeout (60 seconds by default) is inconclusive.

The tool's promise, with a judge model

With a judge model configured, each call gets one question: does the result do what the tool's description promises, for these arguments? The model can also propose arguments for read-only tools that publish no examples; each is checked against the input schema before anyone sees it. Without a judge model, a pass means a well-formed result only, and the report says so at the top.

The same optional model judge as for agents: your Anthropic API key, in its own process.

One run of the bundled mock server.

mcp-mixed ships with Suncly and misbehaves on purpose. This is what Suncly reported for it: two runs per call, no judge model.

Results for the bundled mock MCP server mcp-mixed, two runs per test case
ToolDeclaresCalledRunsWhat the report says
lookup_word{"word": "apple"}readOnlyHint: trueyesevery structural check passed
failing_lookup{"city": "Paris"}readOnlyHint: trueyesis_error; the tool said: Upstream population service unavailable.
broken_schema{"city": "Oslo"}readOnlyHint: trueyesstructured_content: structuredContent.celsius: 'warm' is not of type 'number'
ask_user{}readOnlyHint: trueyesthe server asked for input instead of answering
unannotated_echono annotationsnonot testednot called: declares no annotations and no person approved calling it
delete_recordreadOnlyHint: false, destructiveHint: truenonot testednot called: declares readOnlyHint: false and no person approved calling it

8 runs: 2 pass, 6 fail, 0 inconclusive (counts, not a score). Tool snapshot re-check: unchanged.

No judge model was configured: a pass means a well-formed result only.

No approval policy configured. A person reviews this result.

What it speaks.

Checked on 15 October 2026 against MCP revisions 2026-07-28 and 2025-11-25.

Transport
Streamable HTTP: one endpoint URL. stdio servers and the deprecated HTTP+SSE transport (2024-11-05) are refused, with what the server speaks named.
Revisions
MCP 2026-07-28 (stateless, server/discover), and 2025-11-25, 2025-06-18 and 2025-03-26 (session-based, initialize), chosen by the specification's backward-compatibility rules. The report names the revision used.
Answers
JSON bodies and event streams. Notifications on a stream are recorded.
Tool lists
Every page of tools/list, up to 1,000 tools and 5 MB per snapshot.
Credentials
A token the server already accepts, as the full Authorization value, or a key in one named header with --auth-header. It stays in the Runner and is redacted from every transcript. Suncly does not run OAuth flows.
Network
Public addresses only. A local sandbox needs allow_private_targets, and the report then says so.
Interoperability
The test suite attests a server built on the official MCP Python SDK (2.2.0) through the real CLI over 2026-07-28, and checks the handshake path against the same server.

Not tested yet.

Every MCP report lists these under “What was NOT tested”, together with each tool that was left out and why.

  • Checking that a tool declared read-only really changes nothing, and that its other annotations are true.
  • Probes for MCP servers: undeclared tools, injected instructions in arguments, bad input.
  • Resources, prompts, completions, subscriptions and the tasks extension. A server that declares them gets each one named in the report.
  • OAuth authorization flows.
  • stdio servers.

The Model Context Protocol (MCP) is an open-source project. Suncly is independent and is not affiliated with or endorsed by the MCP project or its maintainers.

Bring one MCP server. Leave with evidence.

Our team tests the tools you put in scope with normal and edge-case requests and emails you the report. Charged only when your report is delivered.

Verify an MCP serverTalk to us

Or write to team@suncly.com.