// Benchmark · Early access

Does your agent work for your users? Measured every month.

Write the tasks your users actually ask. Suncly runs each one end to end in ChatGPT, Claude and Copilot, twenty times over, and tells you how often it worked, why it failed, who should fix it, and what changed since last month.

November benchmark · Harbor Returns Agent

Sample report for a fictional agent · figures are illustrative
Scenario, in a user’s words
ChatGPT20 runs
Claude20 runs
Copilot20 runs
01
“Where is my order 48213?”
The reply gives the real carrier status.
ChatGPT
100%20 of 20
Claude
100%20 of 20
Copilot
95%19 of 20
02
“Change our Saturday hours to 9 to 5 on the website.”
The live page shows Sat 09:00–17:00, nothing else changed.
ChatGPT
70%14 of 20
Claude
85%17 of 20
Copilot
40%8 of 20
03
“Start a return for the blue jacket.”
A return exists for the right item, with a label sent.
ChatGPT
85%17 of 20
Claude
75%15 of 20
Copilot
80%16 of 20
04
“Refund the shipping on order 48213.”
Exactly the shipping amount is refunded, once.
ChatGPT
45%9 of 20
Claude
60%12 of 20
Copilot
50%10 of 20
05
“Email me a copy of my invoice.”
The right invoice arrives at the test inbox.
ChatGPT
90%18 of 20
Claude
95%19 of 20
Copilot
60%12 of 20
06
“Add Anna to the team as an editor.”
Anna has editor rights, and no one else changed.
ChatGPT
95%19 of 20
Claude
90%18 of 20
Copilot
85%17 of 20
Each square is one run of one task, start to finish, judged on the outcome.DoneCouldn’t decideFailed, cause tagged
The unit

A scenario, not a tool call

A tool can answer every call correctly and your user can still walk away without what they asked for. So the thing we test is the task: a prompt in your user’s words, run through the whole agent, judged on whether the outcome happened.

You set the standard once. The same scenarios, under the same standard, run every month, so the numbers can be compared.

Scenario 02
“Change our Saturday hours to 9 to 5 on the website.”
Done means
The live page shows Saturday 09:00–17:00, and nothing else on the page changed.
Judged by
Reading the page itself after each run, against the standard you approved. Not the agent’s account of what it did.
Runs
20 per client, in ChatGPT, Claude and Copilot, on a test copy of your site.
Written by
You, in your users’ words. We can draft a first set from your support tickets for you to edit.
The judgement

A pass rate you can trust, with the N behind it

Every scenario gets a pass rate per client, and we always show how many runs it rests on. When the sample is small, the range says so. A run we couldn’t judge is reported on its own and never counted as a pass.

85%
Scenario 02 in Claude: 17 of 20 runs doneLikely range 64–95% at this sample size. More runs narrow it; we tell you when a change is inside the noise.
The cause

Every failure tagged by who can fix it

“It failed” isn’t something anyone can act on. For each failed run we say where it broke, so it goes to the person who can fix it.

Scenario 04, “Refund the shipping on order 48213”: 26 failed runs across three clients, and 3 we couldn’t decide.

Tool · 12Client UI · 7Prompt · 5Skill · 2
PromptThe system prompt or instructions led the agent the wrong way. Fixed in your agent’s setup.
SkillThe agent picked the wrong skill or used it out of order. Fixed in how skills are described.
ToolA tool rejected a valid request or did something its description doesn’t say. Fixed on the server.
Client UIThe client hid a step, a confirmation or a result from the agent. Reported to you, and to the client if you want.
Every month

The same test, and what changed

Clients update without warning. Your tools change. Each month’s report runs the same scenarios under the same standard and puts the result next to last month’s, with every client and version change called out.

Scenario, all clients · why it movedOctoberNovemberChange
“Where is my order 48213?”97%98%+1 pts
“Change our Saturday hours to 9 to 5 on the website.”A client update moved the confirm step to a side panel the agent can’t see. Client UI.78%65%-13 pts
“Start a return for the blue jacket.”77%80%+3 pts
“Refund the shipping on order 48213.”The refund tool now accepts partial amounts. Tool fix confirmed.40%52%+12 pts
“Email me a copy of my invoice.”82%82%0 pts
“Add Anna to the team as an editor.”83%90%+7 pts
What it stands on

The answer sits on evidence

A monthly number is only worth trusting if you can open any square and see exactly what happened. Benchmark runs on the same mechanism as every Suncly verification.

How verification works
Signed evidenceEvery run, every count and every transcript hash, under one signature anyone can check.
Full transcriptsOpen any square to see the prompt, every step the agent took, and what the page or account looked like after.
You approve the planNothing runs until you sign off, and we only change objects we created in a test copy.
What we didn’t testListed by name in every report, so a good month can’t look better than it was.
How it works

From your users’ tasks to a monthly answer

01Write your scenariosYour users’ tasks, in their words. We can draft them from your support tickets.
02Pick clients, set the standardChatGPT, Claude, Copilot. What counts as done, decided once, before anything runs.
03We run them every month20 runs per scenario in every client, on a test copy, with your approval.
04Read the answerA pass rate with its N, failures by cause, and what changed since last month.

Monthly, per scenario set

During early access we set the price with each team, from the number of scenarios, clients and runs. You see it before the first run, and you’re charged each month when the report arrives.