Finance LLM benchmarks: what to measure and why

A finance LLM benchmark tests whether a language model can reason about financial situations. That sentence is easy to agree with and almost impossible to execute well. Most benchmarks that claim to test financial reasoning actually test financial knowledge: retrieval of facts, extraction of numbers from tables, classification of sentiment from earnings calls. These are real capabilities, and they are not the ones that matter when a model is put near a capital decision.

The decisions that matter in institutional finance are judgment calls: whether a restructuring thesis holds, whether a covenant breach is material, whether a company's revenue trajectory supports the leverage in its capital structure. These are the questions that language models will increasingly be asked to support, and they are the questions that existing benchmarks do not test. This guide explains what a finance LLM benchmark must actually measure, why the dominant evaluation paradigm falls short, and what outcome-graded environments offer in its place.

What existing finance benchmarks test

The current landscape of finance LLM benchmarks clusters around three capabilities.

Financial knowledge. Multiple-choice questions about accounting standards, valuation methodologies, regulatory frameworks. These test whether the model has absorbed the finance curriculum. They do not test whether the model can apply that knowledge to an ambiguous situation under time pressure, which is what financial judgment actually requires.

Numerical extraction. Pulling specific numbers from financial tables, computing ratios, identifying trends in reported data. This is useful and testable, but it is the part of financial reasoning that is closest to automation without language models. A well-written parser can extract a covenant ratio from a credit agreement. The harder question is what that ratio means in context.

Sentiment classification. Labeling the tone of earnings-call transcripts, press releases, or analyst commentary. This is the thinnest form of financial reasoning: it reduces a complex information environment to a polarity score and tests whether the model agrees with the human annotator's labeling.

None of these are bad tests. They are incomplete tests. A model that scores well on knowledge, extraction, and sentiment and poorly on judgment, temporal reasoning, and sensitivity analysis is a model that looks competent on the leaderboard and fails in the situations where its output matters most.

What a finance LLM benchmark must measure

A benchmark that tests financial judgment must measure at least five capabilities that existing benchmarks largely ignore.

Diagnostic reasoning. Given a set of financial documents, can the model identify what is actually happening in the business? Not what the management says is happening. Not what the analyst consensus believes. What the underlying data supports. Our end-to-end case study includes tasks across seven categories of reasoning, and diagnostic is the foundation: every downstream judgment depends on whether the model reads the situation correctly.

Temporal reasoning. Can the model reason from a point-in-time information set without importing information from the future? This sounds trivial and is not. Our contamination probe shows that models routinely reason from hindsight: 12-20% of answers in a typical evaluation reference information that was not available at the anchor date. A benchmark that does not enforce temporal boundaries is testing a mixture of reasoning and recall, and it cannot distinguish between them.

Sensitivity to framing. Does the model's output change when the same financial situation is presented differently? Our research shows that it does, and the magnitude is alarming. Quadrupling a company's biggest risk factor left the AI's price unchanged; one line of market chatter moved it $45 million. A benchmark that presents each task in a single framing cannot detect this. A robust benchmark must test the same judgment under multiple framings and measure the variance.

The 564-run prompt framing study quantified this precisely: across controlled variants of one synthetic deal, the recommendation never moved, but the price, leverage, and what the memo noticed moved with the packaging of the question. Scaffolding away the surface-level flip did not remove the deeper framing effect. A finance benchmark that does not test for this is measuring the model's performance on the specific question it was asked, not the model's judgment about the underlying situation.

Resistance to anchoring. Does information from one context contaminate the model's judgment in a separate context? In practice, models are used in sessions that span multiple deals, multiple analyses, multiple information environments. Our anchoring study shows that market chatter from one deal's context moves the model's valuation of an unrelated deal by half a turn of EBITDA, and explicit instructions to evaluate independently remove less than a third of the effect. A benchmark that tests each task in isolation misses this entirely.

Grading integrity. The benchmark's own grading function must be tested. A grader that can be gamed produces misleading scores without any visible error. A grader with a high noise floor produces scores that reflect the grading math rather than the model's performance. The rubric must lock before any model sees the task, or the evaluation is describing the model rather than testing it. These are not features of the model being tested. They are properties of the benchmark itself, and a benchmark that has not validated its own grading pipeline is not trustworthy.

Rubric-graded vs outcome-graded evaluation

Most LLM evaluations for finance use rubric-graded scoring: a human evaluator or an LLM judge reads the model's output and assigns a score based on a set of criteria. This approach has two structural problems.

First, the ground truth is the evaluator's judgment, not an external fact. For questions of financial judgment, reasonable evaluators disagree. A rubric score tells you what the evaluator thought, not whether the model was right. Second, rubric-graded evaluations are vulnerable to surface-quality bias: models that produce well-structured, confident, detailed responses score well even when the underlying reasoning is wrong. The precision theatrics study showed this concretely: the model's willingness to repeat a numeric value at a given precision level varied by a factor of 28 across format conditions, while its actual verdict never changed. A rubric-graded evaluation would score the more precise-looking output higher.

Outcome-graded evaluation anchors the scoring to what actually happened. The grading criteria are derived from a documented outcome, not from an evaluator's opinion about what the answer should look like. This does not eliminate subjectivity, but it moves the reference point from opinion to fact. The question is no longer "does this answer look good?" but "would this answer have been correct, given what we now know happened?"

This is a harder standard. It requires that the benchmark be built from real events with documented outcomes, that the tasks be constructed backwards from those outcomes, and that the temporal boundaries be enforced. It also produces a more useful signal: a model that scores well on an outcome-graded benchmark has demonstrated the ability to produce reasoning that aligns with reality, not reasoning that aligns with an evaluator's expectations.

What outcome-graded environments look like in practice

An outcome-graded finance LLM benchmark is an RL environment built from the documentary records of real financial events. Each task is a question about a specific situation at a specific point in time, graded against the outcome that actually occurred.

The construction pipeline works backwards. Start with a documented outcome: a credit event, a restructuring, a covenant breach, a facility drawdown. Gather the complete documentary record: financial exhibits, credit agreements, public filings, board materials. Set an anchor date: the point in time at which the model must reason. Construct tasks that ask about the situation as it stood at that date. Derive grading criteria from the documented outcome. Freeze the rubric. Then, and only then, run the model.

This is the approach described in detail in our piece on constructing evaluation tasks from outcomes. The key property is that the ground truth is external to the evaluation: it is a fact about the world, not a judgment about the model's output.

The practical advantages are specific. Contamination can be tested: if the model's performance drops on anonymized versions of the same tasks, it was recognizing the deal rather than reasoning about it. Temporal leakage can be enforced: answers that reference information unavailable at the anchor date are penalized, not just flagged. The prompt leakage patterns that bias rubric-graded evals can be detected and controlled: five trigger families that hand the model the shape of its answer, caught by a one-word test.

A benchmark whose grading function has not been adversarially tested is not a benchmark. It is a hope.

Choosing or building a finance LLM benchmark

If you are choosing a benchmark for evaluating language models on financial tasks, or building one, the checklist is short.

Does it test judgment or extraction? A benchmark that only tests extraction is useful for extraction. It tells you nothing about how the model will perform on the decisions that actually matter.

Does it enforce temporal boundaries? A benchmark that lets the model reason from hindsight is measuring a blend of judgment and recall, and it cannot tell you which is which.

Does it test sensitivity to framing? A benchmark that presents each task once in one framing is measuring the model's performance on that specific framing, not its judgment about the underlying situation.

Has its grading function been adversarially tested? A benchmark whose grader can be gamed, or whose noise floor has not been measured, is producing numbers that may not mean what they appear to mean.

Is the ground truth a fact or an opinion? Rubric-graded benchmarks anchor to opinion. Outcome-graded benchmarks anchor to documented results. For judgment-heavy tasks, the difference is the difference between measuring what evaluators think and measuring what the model actually gets right.