An eval built from your standard, not ours

Every team we meet has a standard. Sometimes it is a style guide. More often it lives in the head of the person who reviews the work and says yes or no. The hardest and most valuable step in tailoring a model is making that standard explicit enough that a model can be scored against it.

Why public benchmarks are not enough

Public benchmarks are useful for comparing base models in the abstract. They are close to useless for deciding whether a model meets your bar, because your bar is not on the benchmark. A model can top a leaderboard and still write in a voice your brand would never sign off, or refuse the exact requests your users make every day.

From a written standard to a held-out eval

1. We read your examples of the work done well and the work done badly, and the note of the standard you hold.
2. We turn that into graded cases: prompts, the response you would accept, the failure modes you would reject, and the weight each carries.
3. We hold a portion out entirely, so the model is scored on cases it never saw during fitting. That held-out set is the bar.

The result is a number that means something inside your company, because it was built from your company. When we say a tailored model scored 47 out of 50 on voice and held its refusals, that is measured against the standard you wrote, not ours.

The delta is the point

We always report the tailored model beside the base model on the same eval. The absolute score matters, but the delta is the argument. It is the difference a fitting made, in the units your reviewers already trust.

base      voice 58   accuracy 71   refusals 62
tailored  voice 89   accuracy 84   refusals 95
delta     +31        +13          +33

Nothing ships until it clears the bar, and the bar is yours.Marielle Voss

An eval built from your standard is not a nicety. It is the thing that lets a private model clear review, because it turns a matter of taste into a measurement your own people agree on.