Notes from building a verifier for finance.
Short, specific pieces on how institutional judgment becomes a signal a model can train against. We show the framework, not the recipe.
One case, worked end to end: a lender, a platform, and a demand shock
A specialist credit fund, a consumer platform whose revenue stopped arriving, four anchor dates, and 117 graded tasks across seven categories of reasoning. Identities transformed by the anonymisation layer; the underlying record is real, with a known outcome.
Where equal-looking models come apart: two frontier families, 282 rollouts, one deal
Claude Sonnet 5 and GLM-5 on the same institutional credit case under deterministic grading: mean reward 0.54 against 0.47, Sonnet 5 ahead on 34 of 47 tasks — and one answer in eight to one in five reasoned from information unavailable at the anchor date, and was penalised for it.
Built backwards from outcomes: ground truth that is a matter of record
Tasks constructed backwards from known outcomes: what a good answer looks like is a matter of record, not an author's judgment — grading criteria freeze before any model runs, and performance cannot be explained by memorisation.
How a prompt's framing quietly answers its own question
Why the grammar of an eval stem can hand a model the shape of its answer, the five trigger families that do the leaking, and the one-word test that catches all of them.