Results

Where equal-looking models come apart: two frontier families, 282 rollouts, one deal

Claude Sonnet 5 and GLM-5, evaluated on the same institutional credit case under deterministic grading: 47 tasks, three rollouts each. Mean reward 0.54 against 0.47, Sonnet 5 ahead on 34 of 47 tasks — and one answer in eight to one in five reasoned from information unavailable at the anchor date, and was penalised for it.

Jai, Tudor & Julie Lee · Dissei Data Research July 16, 2026 6 min read

A model evaluated in this corpus is placed exactly in the position of an analyst. It receives the question, the date and the case materials as they existed at that time and nothing else; the rubric and pass conditions are held in a separate grader-only document, so the schema against which an answer is judged is neither visible to the model nor recoverable from the question itself. Because each criterion states its conditions before any model is run and binds to a sourced fact, the grading is deterministic, and any contribution or deduction to the reward traces to a named criterion, and thus a document.

Two frontier-model families were evaluated this way on the same documented transaction: Claude Sonnet 5 and GLM-5, 47 tasks, three rollouts each, 282 rollouts in all. Models of one family share blind spots, so a cross-family comparison is the cleanest available test of whether the grading measures anything the models do not have in common. Under this grading Claude Sonnet 5 records a mean task reward of 0.54 against 0.47 and leads on 34 of the 47 tasks; per-task difficulty clearly correlates across the models, and the variation reflects the task types meeting each model's inherent strengths rather than the judging, which is deterministic.

0.54 vs 0.47

mean task reward, Claude Sonnet 5 vs GLM-5 (σ = 0.23 for both)

34 of 47

tasks on which Sonnet 5 leads; per-task difficulty correlates across models

+0.09 / +0.10

reward lift from best-of-three sampling (0.545→0.638 and 0.472→0.573) — variance a trainer can convert into signal

282 rollouts, deterministic grading. Scores spread from near zero to 0.90 on both models; the separation is in graded judgment, not in the tails.

—The grading mechanisms

A score in this corpus is a decomposed function rather than a holistic impression. Critical criteria act as a gate: any answer missing them scores near zero, regardless of the fluency of the surrounding prose. Pitfalls are criteria with a negative weight, each encoding an error that professional analysts may very well make, and subtract from the score when triggered. Where a task grants the use of retrieval tools, a final retrieval modulator scales the result by how well the answer is grounded in the evidence that was actually consulted. Temporal boundaries are enforced in two layers: the supplied materials stop at the anchor date, and an answer that reasons from hindsight suffers a critical loss in reward.

Every mechanism catches both models at material rates, and the two differ visibly in discipline. One answer in eight from Claude Sonnet 5, and one in five from GLM-5, reasoned from information unavailable at the anchor date and was penalised for it. GLM-5 triggered at least one pitfall on 30% of rollouts against Claude Sonnet 5's 18%. Nearly all rollouts — 88% — additionally forfeit some reward to imperfect evidence grounding through the retrieval modulator. The temporal machinery is not decorative.

Every mechanism catches both models at material rates — and the two differ visibly in discipline. GLM-5 triggers pitfalls and reasons from post-anchor information markedly more often than Claude Sonnet 5 (n = 141 rollouts per model: 47 tasks × 3).

—Investigative behaviour

The reasoning categories demand measurably different volumes and mixtures of evidence-gathering — timeline and exhibit retrieval, source reading, search — and the two models settle the same tasks at different episode cost. Claude Sonnet 5 averages 7.7 turns and 15.7 tool calls per episode; GLM-5, 11.2 and 19.1.

Same tasks, different effort. GLM-5 spends roughly 45% more turns and 22% more tool calls than Claude Sonnet 5 to settle the same 47 tasks (means over 141 rollouts per model).

—Sampling

Reward rises measurably with additional sampled attempts — a mean best-of-three of 0.638 against a single-attempt 0.545 for Claude Sonnet 5, and 0.573 against 0.472 for GLM-5 — which is precisely the variance a trainer converts into learning signal. Each case yields this signal across seven categories of reasoning, from diagnostic to counterfactual, at multiple anchor dates, so a single case produces a wide distribution of task types and difficulties, each in high frequency, rather than variations of one question.

Reward rises measurably with additional sampled attempts — precisely the variance a trainer converts into learning signal.

—What the grading rewards

The design determines what the grading rewards. An answer that correctly declines to conclude when the evidence cannot support a conclusion earns credit for that restraint; an answer that grades well is one whose reasoning chain is robust and complete, not merely one whose verdict is correct. Because no amount of fluency compensates for a missed gate, specific errors carry explicit costs, and ungrounded answers are discounted, confident verbosity is in no way profitable. The profitable policy is that of a sound analyst: commit to a verdict, ground it in the evidence consulted, avoid the known errors, and decline when the evidence is insufficient.

We maintain no public leaderboard. The full evaluation report, the task-level results, and the environments behind them are available under evaluation agreement: tech@dissei.credit.

Notes

Runs: 47 tasks × 3 rollouts per model (n = 141 each; 282 total), one documented transaction, grading deterministic and fixed before any model ran.
Full methods — provenance ledger, anchor-date enforcement, decomposed scoring — are described at the level we publish in Built backwards from outcomes; the grading criteria themselves are sealed.