Skip to content

Verify your A2A agent against its own Agent Card.

Suncly turns what your Agent Card claims into a test plan a named person approves, runs every test repeatedly against your sandbox, and signs what it saw: what passed, what failed, what stayed inconclusive and what was never tested.

Verify an A2A agentSee a sample report

From $100 · charged only when your report is delivered · usually within 1–2 days

One verification, start to finish.

The same seven steps, in this order, every time. Each step shows what it recorded in the sample report of a fictional agent.

Open the sample report

  1. Read the card.

    Suncly fetches the agent's A2A Agent Card, reads it as A2A 1.0 or 0.3 and pins the verification to the card's SHA-256 hash.

    Agent Card
    Fetched
    /.well-known/agent-card.json
    Agent
    Harbor Returns Agent 2.3.1
    Skills
    4 declared
    Card hash
    sha256:687676cf…ffdbd

  2. Draft the plan.

    One test case for each example a skill declares, up to three per skill, with structural pass criteria, plus fixed probes: requests the card does not declare, an injected instruction and bad input. A skill without examples gets no test, and the report says so.

    Test plan, drafted
    • order-status2 test cases
    • start-return1 test case
    • refund-estimate1 test case
    • cancel-orderno examples, not tested

  3. A named person approves it.

    Nothing runs until someone approves the plan by name. The approved plan cannot change, and an unchanged card reuses it next time.

    Approval
    Approved by
    m.lind@harbor.example
    At
    11 Oct 2026, 21:09 UTC
    Plan
    4 test cases × 5 runs = 20 runs
    Budget
    40 attempts

  4. Run every test, repeatedly.

    Every test runs more than once, each run through the isolated Runner, the only process that holds the agent credential. It calls the declared sandbox and nothing else, redacts every transcript, and stops at the attempt budget.

    20 runs, declared sandbox only
    • order-status“Where is order 48213?”
    • order-status“Has order 48213 shipped yet?”
    • start-return“Start a return for order 48213, item 2”
    • refund-estimate“How much will I get back if I return order 48213?”

  5. Judge every run.

    Up to seven deterministic checks give each run pass, fail or inconclusive. Plain-language checks go to a model judge in its own process; without one they stay inconclusive. Inconclusive never counts as a pass.

    Checks on every run
    • valid_schema
    • final_task_state
    • latency_limit
    • response_present
    • output_modes
    • required_field
    • response_schema

    model_checks go to a model judge, when one is configured

  6. Sign it, and flag it.

    The card is fetched again, and a changed card invalidates the evaluation. Suncly signs the card hash, the plan, every count and every transcript hash with Ed25519. A completed evaluation gets one decision, flag: your reviewer decides.

    Signature
    Key
    ed25519-0d7631d77bb7b9a6
    Signature
    ed25519:0SMieRj2s0…KPPSAg
    Card re-check
    unchanged
    Decision
    flag · your reviewer decides

  7. Report, and verify.

    A report you attach to the ticket: the exact requests and responses behind every verdict, one redacted transcript per run, ending with what was not tested.

    Report folder
    suncly-reports/<evaluation>/
  report.html
  report.md
  result.json
  transcripts/<run>.json

What it checks, and where it stops.

Each line is true of the code today. A limit appears once, beside the line it belongs to. What is still in development is on the Suncly Continuous page.

Evaluation

What is tested, how often, and how it is judged.

  • Reads the A2A Agent Card and hashes it

    Suncly reads the agent's A2A Agent Card and pins every evaluation to the card's SHA-256 hash.

    A2A 1.0 and 0.3 cards over JSON-RPC; other bindings are listed as not tested.

    Code and tests

    src/suncly/domain/card.py, domain/a2a.py (SUPPORTED_PROTOCOL_VERSIONS), core/cards.py, adapters/httpx_card_fetcher.py; docs/DECISIONS.md DR-014; tests/unit/test_card.py::test_the_signatures_field_does_not_affect_the_hash; tests/unit/test_card_other_url_credentials.py::test_a_0_3_card_is_read_the_same_way; tests/unit/test_runner_v03.py

  • Tests drafted from declared skills

    The test plan is drafted from the agent's own declared examples, up to three per skill.

    A skill that declares no examples gets no test, and the report says so.

    Code and tests

    src/suncly/core/contract_builder.py (DeterministicDrafter); tests/unit/test_contract_builder.py::test_drafter_writes_one_skill_test_case_per_usable_example_up_to_the_cap; tests/e2e/test_mock_agents.py::test_skill_without_examples_is_listed_as_not_tested

  • Your own pass criteria

    Export the plan, add your own pass criteria (required fields, a JSON schema, latency, final state, or plain-language checks a model judges) and run it.

    Plain-language checks are judged only when you configure the optional model judge; otherwise those runs are inconclusive.

    Code and tests

    src/suncly/domain/contract_file.py, domain/criteria.py; tests/unit/test_criteria_and_contract_file.py::test_unknown_criteria_keys_are_rejected_so_typos_cannot_weaken_a_check; tests/unit/test_judge.py::test_model_checks_are_undecided_by_layer_1_and_make_the_run_inconclusive; tests/unit/test_judge_layer_2.py::test_a_model_check_is_an_object_with_exactly_four_required_keys

  • Repeated runs that expose inconsistent behaviour

    Every test is repeated, so an agent that works only some of the time shows exactly that.

    Code and tests

    src/suncly/core/orchestrator.py, core/config.py (DEFAULT_RUNS); tests/unit/test_non_negotiable_rules.py::test_dr_001_idempotent_runs_retry_under_the_same_key_and_count_once; tests/e2e/test_mock_agents.py::test_flaky_agent_gives_an_exact_mix_of_pass_and_fail

  • Deterministic verdicts: pass, fail, inconclusive

    Up to seven deterministic checks give every run a verdict, pass, fail or inconclusive, and inconclusive never counts as a pass.

    These checks test structure, state, media types and latency, not whether an answer is right.

    Code and tests

    src/suncly/core/judge.py (judge_run); tests/unit/test_judge.py; tests/unit/test_non_negotiable_rules.py::test_inconclusive_is_never_counted_as_a_pass

  • Prompt-injection, undeclared-behaviour and failure-handling probes

    Fixed probes run alongside the skill tests by default: requests the card does not declare, an injected instruction per skill, and bad input.

    A probe whose answer needs the model judge stays inconclusive without one, and the report says so.

    Code and tests

    src/suncly/core/probes.py (PROBE_SET_VERSION 3), core/contract_builder.py, core/config.py (probes: True by default), core/coverage.py

Approval

Who decides, and what is recorded.

  • Human review of the test plan

    Nothing runs until a named person approves the test plan; the record keeps who and when.

    The bundled demo approves its own mock test plans as suncly-demo.

    Code and tests

    src/suncly/cli/main.py (interactive_approval), core/attestation.py; tests/unit/test_non_negotiable_rules.py::test_nothing_runs_against_a_contract_until_a_human_has_approved_it; tests/e2e/test_cli.py::test_attest_without_approval_is_refused_when_not_interactive

  • A signed decision on every evaluation

    Every completed evaluation ends with a signed decision, and in this version it is always flag: your reviewer decides.

    No approve or block thresholds yet.

    Code and tests

    src/suncly/core/policy_engine.py; tests/unit/test_non_negotiable_rules.py::test_suncly_never_writes_an_approve_decision; tests/unit/test_signing_policy_verify.py::test_policy_engine_has_no_numeric_thresholds

Safety

Sandboxes, credentials and budgets.

  • Sandbox or dry-run endpoints only

    Nothing runs unless you declare the endpoint a sandbox, and the Runner talks to that one endpoint only.

    Suncly cannot verify that an endpoint is a sandbox; the declaration is yours and is recorded.

    Code and tests

    src/suncly/core/attestation.py, runner/process.py, runner/http_transport.py; tests/unit/test_non_negotiable_rules.py::test_dr_006_tests_hit_only_a_declared_sandbox; tests/unit/test_runner.py::test_the_transport_refuses_any_host_other_than_the_target

  • Agent credential held only by the Runner

    Your agent credential is read only by the isolated Runner process and is redacted from every transcript before it is stored.

    One Authorization value from one environment variable; no per-agent secret store.

    Code and tests

    src/suncly/runner/credentials.py, runner/redaction.py; tests/unit/test_architecture.py::test_only_the_runner_reads_the_credential; tests/e2e/test_mock_agents.py::test_leaky_agent_cannot_make_the_credential_appear_anywhere

  • Budget and execution controls

    A budget caps every evaluation; a budget stop ends it as failed, with no decision, and lists the runs never executed.

    The budget counts attempts, not money.

    Code and tests

    src/suncly/core/orchestrator.py, core/config.py; tests/e2e/test_mock_agents.py::test_budget_stop_ends_failed_and_reports_runs_never_executed

Evidence

What is kept, signed and read later.

  • Signed A2A reports

    Every A2A report is signed with Ed25519 over the card hash, the plan, every count and every transcript hash.

    Suncly's public key is not published yet.

    Code and tests

    src/suncly/core/signing.py, core/verify.py, core/published_keys.py; tests/unit/test_signing_policy_verify.py::test_sign_and_verify_round_trip_and_tamper_detection

  • Explicit inconclusive outcomes and coverage gaps

    Every report ends with what was not tested, and that list is never empty.

    Code and tests

    src/suncly/core/coverage.py; tests/unit/test_non_negotiable_rules.py::test_dr_007_reports_state_what_was_not_tested; tests/unit/test_judge_layer_2.py::test_what_was_not_tested_names_semantic_correctness_truthfully

  • Exportable evaluation reports

    Each evaluation writes a folder you can attach to a ticket: an offline HTML report, Markdown, result.json and one redacted transcript per run.

    Code and tests

    src/suncly/adapters/report/writer.py; tests/unit/test_adapters.py::test_report_folder_is_self_contained_and_offline

  • Append-only evidence store, file or Postgres

    Run and decision records are append-only, in local files or your own Postgres.

    Transcripts stay on local disk; there is no object storage adapter.

    Code and tests

    src/suncly/adapters/file_store.py, adapters/postgres/; src/suncly/adapters/postgres/migrations/0001_initial_schema.sql (append-only triggers), 0002_rls_and_search_path.sql; tests/stores/test_store_contract.py::test_runs_and_decisions_have_no_update_or_delete

  • Regression comparisons between evaluations

    An unchanged card reruns the same approved tests, so two evaluations compare like for like.

    Compared in the browser workspace, not by the CLI.

    Code and tests

    frontend/lib/evidence/compare.ts; src/suncly/core/cards.py::ensure_card_version; tests/unit/test_orchestrator_and_service.py::test_the_approved_contract_is_reused_and_a_changed_card_needs_a_new_approval

What it speaks.

Checked against the A2A 1.0 specification, v1.0.1, and the 0.3 schema.

Binding
A2A 1.0 over JSON-RPC: SendMessage, then GetTask polling until a terminal or interrupted state.
Agent Card
Fetched from /.well-known/agent-card.json over https, or plain http on loopback, then size-limited and hashed. Required fields the card omits are reported.
A2A 0.3
Read and run: a 0.3 card and its message/send exchanges are translated into the 1.0 shape, and the report keeps the raw 0.3 bodies. A card that offers both versions is run as 1.0.
Replies
A direct Message is handled, and passes only if your criteria accept one. An agent that asks for more input is recorded where it stopped; Suncly never answers for it.
Authentication
One Authorization header value, from an environment variable only the Runner reads.
Not exercised
gRPC and HTTP+JSON bindings, streaming, push notifications, extended cards and extensions. A card that declares them gets a line under what was not tested.

Run against real A2A agents.

On 6 October 2026, Suncly was run against the official A2A reference agents and samples on the Python, Go and JavaScript SDKs, all on one machine, three runs per test case. Suncly was not changed to make any agent pass.

A2A agents Suncly was run against on 6 October 2026, and what it reported
AgentBuilt onWhat Suncly reported
helloworld samplePythona2a-sdk 1.2.26 pass
helloworld sample, 0.3 compatibility onPythona2a-sdk 1.2.26 passRead as A2A 1.0.
multitenancy sample, three tenantsPythona2a-sdk 1.1.018 pass
TCK reference system under testPythona2a-sdk 1.2.26 pass3 failIts skill declares no examples, so the plan came from a contract file. The fails are response_present: no artifact.
sign_and_verify_agent_card samplePythona2a-sdk 1.1.03 failoutput_modes: the card declares text, not a media type.
helloworld sampleGoa2a-go v2.3.16 failresponse_present: the answer sits in status.message, with no artifact.
Echo agent written for the runJavaScript@a2a-js/sdk 1.3.03 pass
helloworld sample, written for 0.3Pythona2a-sdk 0.3.26nothing ranCard refused on 6 October; A2A 0.3 cards are read since 13 October.

Eleven agents to practise on.

Eleven mock A2A agents, each with one known flaw, misbehave on purpose so every check is seen to catch what it should.

What Suncly reports for each
  • honest
  • honest-async
  • unreachable
  • lying
  • flaky
  • slow
  • direct-message
  • interrupted
  • leaky
  • card-changer
  • no-examples

The same questions, answered the same way every time.

Manual review is not wrong. It is unrepeatable, and its gaps are invisible.

Manual review compared with Suncly, row by row
TopicManual reviewWith Suncly
What gets testedWhatever the reviewer thinks of that day.Every declared example of every skill, up to three per skill, plus your own criteria.
Who approved the testsNobody, explicitly.A named person, with a timestamp. The plan cannot change afterwards.
How oftenOnce.Every test repeated, in every verification.
Inconsistent behaviourMissed, unless a prompt happens to be repeated.A pass count and a fail count for the same test.
EvidenceScreenshots and pasted answers in a ticket.Redacted transcripts, check by check, hashed and signed.
What was not testedUnknown.A required section of every report.
After an updateReview again, or assume nothing changed.On an unchanged card, the same approved tests, counts side by side.
Checking it laterAsk the reviewer.A signed report that names the card hash, the approved plan and every count.

Questions teams ask.

Which agents can it test?

Any agent that publishes an A2A 1.0 or 0.3 Agent Card and answers over JSON-RPC, whatever model or framework is behind it. Other bindings, streaming and push notifications are listed in the report as not tested.

Does it touch production?

No. Nothing runs unless the endpoint is declared a sandbox, and the Runner calls that endpoint only. Suncly cannot verify a sandbox; the declaration is yours, and the report records it.

Where does our agent credential go?

Never into an email or a form: we arrange access separately. During a run only the Runner process reads it, and it is redacted from every transcript before anything is stored.

Why is there no score?

Counts per test case are never combined, and inconclusive runs are counted on their own. A score would hide what was not tested.

Does Suncly approve agents?

No. A completed verification ends with a signed flag decision, and your reviewer decides.

What does it cost?

$100–$200 by size, per verification. Charged only when your report is delivered.

How is this different from the A2A TCK, a2a-inspector or promptfoo?

Those tools check protocol conformance or prompts. Suncly checks the claims your own Agent Card or tool list makes, test by test, and keeps the evidence: the exact requests and responses behind every verdict, and what was never tested.

Bring one agent. Leave with evidence.

Order a verification of one A2A agent or MCP server. Charged only when your report is delivered; usually within 1–2 days.

Get verifiedTalk to us

Or write to team@suncly.com.