Model selection for AI consultancies: measure your client's tasks, not the leaderboard

This is a TerraNet service, working name Separable, now taking pilots. If you already know you want one, request a pilot and skip the argument.

The decision your clients are paying you to make

Every AI engagement includes the same question: which model should this workflow run on? The client usually cannot answer it, which is part of why they hired you. And the answer goes stale each time a vendor ships.

In practice, that choice is often settled by things that have nothing to do with the client's work: which productivity suite they already pay for, which vendor the consultancy partners with, and a tier list absorbed from blog posts. When there is a pilot, it tends to be a handful of prompts run once and judged by eye. The recommendation that comes out of it sounds confident. It is rarely measured.

Why the leaderboard cannot make it for you

In September we published LLM evaluation and the noise floor, which looked at how much the public rankings can actually tell a buyer. The short answer is very little:

• Of the 77 benchmarks in our catalogue, only 2 publish run-to-run confidence intervals. For the rest, you cannot even ask whether a rank difference is real.
• On DeepSWE, one of the two that does, only 3 of 27 adjacent rank steps are statistically separable at 95% confidence. Most of the ranking is sorted noise.
• The same article found that Claude Sonnet 5 costs 263 times as much per task as DeepSeek-V4-Flash, yet their intervals overlap completely. A ranking that separates them is choosing between prices, not capabilities.

A leaderboard also measures someone else's tasks, in someone else's harness. Your client's invoice matching, support triage or document extraction was not on it.

A demo repeats the same mistake

The obvious answer is to test on the client's own work. It is the right answer, with one catch that the original article spelled out: a small set of tasks run once separates nothing. If a public benchmark running hundreds of trials can separate only a handful of its own rank steps, a dozen prompts run once in a sales meeting cannot separate two frontier models.

As that article put it, the case for evaluating your own tasks "is not that it is inherently more precise; it is that it measures the right thing." Precision has to come from somewhere else: repeated trials, explicit error bars, and treating differences within the margin of error as ties.

What we will measure for you

That is the service we are piloting. You bring one client workflow; we do the measurement:

1. Scope. You name one workflow for one client, for example sorting inbound support email into six queues.
2. Examples. You send 20 to 50 real examples with known outcomes, drawn from the client's own history.
3. Targets. You tell us what the client actually needs: the most wrong answers they will accept, what one wrong answer costs them, the response time and peak rate the workflow must meet, a monthly budget, and whether the same input must get the same answer every time. Whatever you can give us; the report says which were set.
4. Prompt. You give us the prompt your deployment will send. Every model gets exactly that, so what you are comparing is the models, not our prompt-writing.
5. Candidates. You pick 3 to 5 model configurations, including at least one cheap one. We estimate the API spend before anything runs.
6. Trials. Every task runs repeatedly per model until the intervals settle or the spend cap is reached.
7. Verdict. Every model runs the same tasks, so models are compared task by task. Where the difference between two models is within the margin of error, they are declared a tie, never ranked. Ties are broken on the client's targets first, then on total cost when a wrong answer has a price, and otherwise on cost per task and latency, all measured on the same runs.

The best fit is a task with a checkable outcome: classification, extraction, structured output and routing.

Request a pilot for one of your clients

Why these numbers hold up

The run is done with our own tooling, and the parts that decide a verdict are built to refuse an answer they have not earned:

• Every task counts once. The thirty-six tasks below, run twenty times each, are thirty-six measurements of a workload, not 720. The range comes from how much the tasks differ, which is the question your client is actually asking: would this hold on more of our work?
• Models are compared task by task, on the same tasks, so a difference counts only when it is larger than the margin of error - and the margin widens as more models are compared, because the leader is picked from the results.
• The tool says when it cannot tell. If a run did not measure every candidate the same way, or nothing passed, it reports that rather than naming a winner.
• It is tested against pilots whose true answer is known. Thousands of simulated runs check that a 95% range contains the truth about 95% of the time, and that one model entered twice under two names comes back tied rather than beating itself.

What you hand your client

The deliverable is a written report, not a dashboard. It has the shape of an engagement's findings, and it is built entirely from the run: no model writes a word of it, so every number in it is the number the statistics produced.

1. Summary. The recommendation, what tied, what was ruled out, how the chosen model stands against your client's targets, and what it costs at their volume.
2. Against your targets. Every candidate against every target the client set, judged by its whole range: it meets a target when the range clears it, misses when the range falls short, and is unproven when the range straddles it, in which case the report prices the tasks that would settle it. With a price on a wrong answer, a total-cost table: model fees and wrong answers together, per month, with its range.
3. Findings. Results with error bars; cost per task, per month and per year; the tasks the models found hard, hardest first, with the expected answer beside each; the answers each model gave instead, which shows you where the queue definitions are blurry; speed, including the slow tail; and how reliable the route was.
4. What is settled and what is not. Every comparison with its difference and range.
5. Options for settling the tie. Take the tie and deploy the recommended model, with the saving or the cost spelled out at the estimate and at both ends of the range. Or more trials, or more tasks, each priced. Or a targeted set built from the tasks where the two models disagreed outright. Then our advice, as a rule your client can apply.
6. Method. Design, what counts as a sample, when the run stopped, how the ranges are built, the correction for comparing several models, how targets break a tie, grading, and the report's limits, each with the run's own numbers.
7. Appendix. The prompt, the settings, the run phases, what the client told us, and how the data was handled.

The verdict figure looks like this:

The scenario behind the run: a small bank's support desk, 10,000 messages a month, no more than one in five misrouted, a misrouted message costing about fifty cents of an agent's time, a $50 monthly budget for model fees, and the same routing on nine tasks in ten. Three excerpts from the report. First, every candidate against those targets:

Target        Your target                  Gemini 3.7 Flash      DeepSeek V4.1 Flash
Accuracy      at least 80.0% right         meets: 92.9%,         unproven: 81.9%,
                                           range 80.3-97.7%      range 67.0-91.0%
Slowest 10%   within 15.0 s (advisory)     meets: 11.6 s         meets: 14.8 s
Peak rate     300 tasks an hour (advisory) meets: 1 at a time    meets: 1 at a time
Model fees    at most $50.00 a month       meets: $3.74          meets: $0.21
Consistency   same answer every time       meets: 91.7%          misses: 72.2%
              on 90% of tasks              of tasks              of tasks

Total cost, model fees and wrong answers together, at $0.50 a wrong answer:

Model                  Fees    Wrong answers   Cost of them   Total a month   Range
Gemini 3.7 Flash      $3.74        708            $354           $358      $119 to $989
DeepSeek V4.1 Flash   $0.21      1,806            $903           $903      $449 to $1,651

Second, the tasks the models found hard, which is where a consultancy earns its fee:

19 of 36 tasks were passed by every model on every trial. The 17
below were not, hardest first. Each cell is passes out of counted
trials.

Task                                    Expected   Gemini  DeepSeek  Nemo
payments-card_payment_wrong_exchange…   payments    0/20     0/20    0/20
payments-cash_withdrawal_not_recogni…   payments    0/20     0/20    0/20
topups-top_up_by_cash_or_cheque         topups     11/20     0/20    0/20
account-fiat_currency_support           account    20/20     0/20    0/20
…

Worth reviewing before anything else: two tasks were failed by
every model on every trial. When every model agrees against the
expected answer, the expected answer or the prompt is the more
likely culprit.

DeepSeek V4.1 Flash failed 130 of 720 counted trials.
  Expected   It answered      Times
  payments   "card"              36
  account    "fees_and_rates"    20
  topups     "payments"          20

Third, the tie-breaker:

Option 1: Take the tie as it stands and deploy Gemini 3.7 Flash

This costs about $42 a year more in model fees than DeepSeek at
10,000 tasks a month. At $0.50 a wrong answer it saves about $545
a month overall on this run's estimate, though at the bottom of
the range DeepSeek would be ahead by $25 a month. DeepSeek misses
your consistency target outright; no amount of testing changes
that without a change to the prompt or the target.

Option 2: Add tasks

More trials of these 36 tasks would not separate the two: your
tasks differ from each other more than the models do. About 67
tasks would, if the 11.0-point gap is real: 31 more, prepared the
same way, about $0.25 in API spend. This is also the option that
would settle the targets marked unproven.

Our advice: deploy Gemini 3.7 Flash. If you would rather give
DeepSeek the chance to prove itself against your accuracy target
before committing, Option 2 at about $0.25 is cheap against what
is at stake.

The report is written to go into your statement of work. "We measured it on your data" is a line a vendor-partner shop recommending whatever it resells cannot use.

Request a pilot for one of your clients

How a pilot works

• Scoped and priced per engagement. The form asks for a budget range so we can size it. Nothing is charged until we have agreed the scope with you.
• Run hands-on by our team while the software is built, on the same statistics code behind our benchmark catalogue. You deal with a person throughout.
• Your client's data stays private. Examples sit in a private Google Cloud bucket, are screened for stray email addresses and card numbers before anything runs, and are deleted thirty days after the memo is delivered. Calls go only to providers that keep no copy, and nothing you send trains a model.
• It measures the model, not a scaffold. If your deployment wraps the model in retrieval or several steps, a pilot measures the model's part of that.

Who this is for

AI consultancies and integrators, roughly five to fifty people, who choose models for small and mid-sized clients and would rather show a measurement than a recommendation. If you have picked a model for a client recently, we would also like to hear how you did it; the form has a place for that.