Evals
Scored against your bar, not the benchmark.
Public benchmarks measure performance on tasks written by someone else. Kethra builds a held-out eval from your own examples and standard, scores every version on three dimensions, and ships nothing until the bar is cleared.
Why public benchmarks are not enough
Most model evaluations are written before the model is built and by someone who does not know your company. They measure things like general reasoning, coding ability or broad factual recall. These are useful signals for choosing a base model. They are not useful for deciding whether a fine-tuned model is ready to represent your brand, answer your customers, or apply your house rules under pressure.
A model that scores well on a public benchmark may still write in the wrong register, give technically correct answers that violate your policies, or refuse when it should not. None of those failures show up in a leaderboard. They show up in production, after the model has shipped.
The solution is an eval built from your examples and your standard, where the prompts are real, the accepted responses are yours, and the failure modes are the ones you actually care about avoiding. That is what Kethra builds before every deployment.
From your examples and your standard
The held-out eval is assembled in four parts, all drawn from your material.
Prompts drawn from your work
The eval is seeded with real prompts from your domain, not synthetic benchmarks. They reflect the tasks your model will actually be asked to do in production.
Accepted responses
For each prompt, you supply one or more responses that would pass your standard. These are the positive targets the model is scored against.
Rejected failure modes
You also supply the responses that would fail: wrong tone, incorrect content, a refusal that should not have been a refusal, or an answer that sounds right but is not. These set the floor.
Weights
Each dimension of the eval may be weighted to reflect what matters most for your use case. A customer-facing model might weight voice highly. A research assistant might weight accuracy above all else.
Three dimensions. All scored before anything ships.
Voice
Does the model sound like your company on its best day? Voice measures tone, register, sentence structure, the words you use and the ones you do not. A model that is accurate but off-brand fails this dimension.
base 58 · tailored 89 · delta +31Accuracy
Does the model get things right? Accuracy measures factual correctness, correct application of your rules and policies, and whether the answer to the prompt is actually the right answer.
base 71 · tailored 84 · delta +13Standard adherence
Does the model follow your written standard under pressure? This dimension measures refusals held, house rules applied, and the absence of the specific failure modes you flagged in your eval suite.
base 62 · tailored 95 · delta +33Base vs tailored, always shown
You see the numbers before anything is deployed. The delta over the base model is always reported, so you know exactly what the fine-tune achieved.
| Dimension | Base | Tailored | Delta |
|---|---|---|---|
| Voice | 58 | 89 | +31 |
| Accuracy | 71 | 84 | +13 |
| Standard | 62 | 95 | +33 |
| Scores out of 100. Delta = tailored minus base on the held-out suite. | |||
Nothing ships until it clears
The eval result determines whether a version is deployed. If a fine-tuned version does not clear the bar set by your examples and your standard, it does not ship. We refine and re-evaluate before returning to you with a passing result.
The bar is not set by us. It is set by the accepted responses and failure modes you supplied. When the tailored model clears it, you have a version you know meets your standard, not one you have to hope does.
The delta is always reported
Every eval report shows the base model score alongside the tailored model score, and the delta between them on each dimension. You know what the fine-tune contributed, and you can use that number to decide whether further refinement is worthwhile. Refinement cycles re-score against the same suite, so the delta is always a fair comparison.
The delta over the base model is the number that matters. It is what the tailoring actually achieved.
Score it before you ship it.
A held-out eval built from your own examples, scored on voice, accuracy and your standard. Nothing deploys until the bar is cleared.