Saying 70, meaning 40: guessing probabilities with five mid-tier models

We asked five mid-tier models for probabilities on 3,010 resolved stock-price questions, each asked twice, on two scales. Six of the ten sets of stated probabilities told us no more than a coin would. So we looked inside the open-source models; and the probability of answering “yes” turns out to be a different number from the stated probability that yes is true — different by twenty points and more.

—What calibration is

A forecaster is calibrated when the things they call 70% happen seven times in ten. Not once — over many calls. Calibration is the property that makes a probability a measurement rather than a mood.

Language models state a probability whenever you ask for one. Whether the number means anything is an actively studied open problem. A model’s internals encode more about whether an answer is right than the answer shows; and the numbers models state are overconfident, and change with how you ask.

In finance the stakes are mechanical. Position sizing, risk limits and expected-value arithmetic consume probabilities literally, so a confident-sounding ninety that means sixty results in a mispriced position. Whether you should ask a language model for a probability at all is a separate question, and our general answer is no. Either way, you want to know what the numbers it emits are made of.

—The experiment

The question we started with: can an LLM put an accurate probability on a future event in a financial setting? We chose the simplest version of that question we could find. Show a model a stock’s recent price history. Ask for the probability that the price ends above a given level a few weeks out. Equities, because nothing resolves cleaner: thousands of questions, dates certain, and unambiguous answers.

So: 251 US stocks, one 250-day price history each, 3,010 distinct questions. Five mid-tier models — three open-weight, from three different labs, and two closed — answered every question twice, once as a percentage and once on a scale of nought to twenty. The three open-weight models also answered every question a third time, as a bare yes or no, with their internals recorded; that comparison is the last section of this note. We also ran a small test on two frontier models, Claude Opus 5 and GPT‑5.6‑sol: the first twenty questions, percentage only.

Four design guards, because each kills a familiar objection:

• No leakage. Every outcome resolves after every open model’s release date, which we treat as its training cutoff.
• No memory. Each price series and its threshold are multiplied by a shared random constant. The event is identical; the price level is unrecognisable.
• Spread. Thresholds sit at volatility-scaled distances, so correct answers span the whole range instead of piling up at 50‑50.
• No free ride. “Above” and “below” questions are balanced six-and-six per stock, so a rising market cannot gift every question the same resolution. The market did rise over the window our questions resolve in. The balance held the answer key at fifty-one per cent yes.

The three open models were picked to sit within five points of each other on a standard capability index — deliberately matched, so that differences in calibration are not a proxy for one model being smarter.

—The words against reality

At first glance the panel looks fine. Every model’s average lands within a dozen points of the fifty-one per cent base rate. Averages are cheap; hedging gets you there.

The tails are where the money is, and the tails are bad. Haiku 4.5 answered “72” to half of everything it was asked. Those seventy-twos came true fifty-nine times in a hundred. When Kimi K2.5 wrote “0”, the thing it had ruled out happened two times in five. When DeepSeek V3.2 put the chance at nine in ten or better, it happened six times in ten.

Now score everything against the dullest possible competitor: write 50 on every line and go home. Six of the ten stated-probability sets did no better than that, and three did worse. The numbers are not quite pure static: a trace of real ordering runs through them. The signal is garbled, not absent. The rest of this note locates the garble.

These are last cycle’s mid-tier models; the exception is the newest on the panel. GPT‑5.6‑luna, released this July, is different in kind: when it says eighty-five or more, the thing happens ninety-two times in a hundred. Its one vice is humility. It says 74; the answer arrives 81.

In the table above we report the Brier score for each model’s performance, a measure of how accurate its probability predictions were, across all 3,010 questions. Lower is better. The Brier score is simply the mean squared error: the average squared gap between the probability stated and what happened. Writing 50 on every line scores 0.250, and the last column says which gaps from that line are big enough to trust.

On the last row we display the results of a simple, deterministic formula for predicting future stock prices. The formula estimates the stock’s daily volatility from the closes in the prompt, assumes a random walk with no trend, and reads the probability of finishing past the threshold off a normal curve. It is by no means the most advanced way to forecast a price. But the formula’s predictions set a bar — a baseline level of forecasting competence, computed from the prompt alone — and they score 0.144. No mid-tier model reaches the bar. The closest is luna, at 0.184.

The small frontier test says the bar is reachable. On a twenty-question spot check, Claude Opus 5 and GPT‑5.6‑sol both roughly matched the formula’s score, and tracked its probability question by question. So the capability exists, and the failures above are a statement about a tier and a vintage — not about language models. Within our own panel, release date predicts calibration better than price does (the cheapest model was the best) and better than leaderboards do (Kimi and Qwen share a capability score; one is a coin, the other is usable). The tier that gets deployed at scale is the tier that cannot yet price a probability. To be explicit about what the frontier matched: ours is a straightforward textbook formula, not the last word. Sharper volatility estimates exist, and we tested none of them. Matching a textbook formula is not market-beating skill, and nothing in this note claims otherwise.

—The words against each other

Look at the favourite numbers. “72” and “50” are things people write, at frequencies that have nothing to do with this stock. If a model’s number is partly a verbal habit, there is a natural test: take the familiar scale away. So we asked all 3,010 questions again on a scale of nought to twenty — nought impossible, ten an even chance, twenty certain. One scale point is five percentage points. A model that holds a probability and translates it loses nothing, because the ruler is arithmetic.

Things changed. DeepSeek’s average probability rose eight points on the new ruler. Haiku’s rose six. Qwen’s fell six. Kimi and luna barely moved. The favourite numbers did not disappear; they moved house, and twelve-out-of-twenty and fourteen-out-of-twenty became the new sixties and seventies. For Qwen the coarse ruler genuinely helped — its nought-to-twenty arm is the best in the mid panel. For DeepSeek it made the overconfidence worse. Same model, same question, different ruler, different number.

One regularity is worth recording. Rank the five models by how well their two scales agree with each other, and you have also ranked them by how well they agree with reality — the same order, exactly, five for five. Disagreement between a model’s own answers is a tell. This oughtn’t be surprising. A model that falls back on favourite numbers keeps different favourites in percentages and out of twenty, so its two answers drift apart; when they agree, a probability was common to both scales — computed, not recited. And there is one deeper place to look for it.

—The words against the weights

Every answer a language model writes is a draw from a probability distribution over next words. So on a yes/no question, a number exists inside the model before the answer does: if “yes” carries 75%, you hear yes three times in four, and you never see the number. For open-weight models the number can be read directly. So we put every question to the three open models a third time, told them to answer in one word — yes or no — and recorded the probability the model gave each word. The two words absorb over ninety-nine per cent of the distribution.

Here is the plain fact, and it is the reason this note exists. The probability that a model answers “yes” is not the probability it states when you ask for one.

Qwen’s stated numbers average about sixty per cent. Its internal chance of answering yes averages twenty-nine. Ask it yes-or-no, and you would mostly hear no. On the questions Kimi rates better-than-even in words, it would answer no seven times in ten. DeepSeek is the strangest of the three: as its stated probability runs from nought to a hundred, its internal chance of saying yes barely moves — from 0.40 to 0.54.

Which number should you believe? The scores suggest the inner one. Treat the stated probability and the internal probability as two forecasters answering the same questions, and the weights beat the words on every model we opened. Right more often, even at a naive better-than-even cut: fifty-nine against fifty-seven in a hundred for Kimi, sixty-four against fifty-five for DeepSeek, sixty-eight against sixty-one for Qwen. Better at ranking outcomes. Closer to the textbook probability. One caveat: the inner number is mis-centred — biased toward no, where the words lean yes — so it is not ready to use. Nor is it especially good: Brier 0.221 for DeepSeek and 0.230 for Qwen, ahead of their words; 0.258 for Kimi, no better than the coin; all three far from the formula’s 0.144.

Three after-notes.

The crossing point is a supermajority rule. Kimi does not tip past even odds internally until its stated number reaches the nineties; Qwen, around eighty. Asked yes-or-no, these models behave like a cautious committee — say yes only if sure — not like a forecaster reporting the likelier side.

The product implication is concrete. There are two common ways to read “the model’s view”: ask for a percentage, or sample the yes/no answer many times and vote. They read different channels. On these models the channels disagree about the same question, systematically. Two teams shipping “the same model” can ship different opinions.

And the divergence is less surprising in hindsight — this is interpretation, flagged as such. The words are trained on how people write about probability. The disposition is trained on which answers scored well. Nothing ties the two together. What we have shown is that two readouts disagree, and which one is more faithful. Which one is the model’s true belief — whether that is even a well-posed question — we do not claim.

—What to do with it

The words and the weights disagree, and the divergence itself is the finding: one model, two readouts, and no reason yet to trust either. The weights scored better on our questions. Better, but not all that good: the one-line formula still beats them. The stated number is the weak layer, and the weak layer is also where the newest models have visibly improved: luna’s spoken numbers land, and the frontier’s track the textbook figure, on outcomes none of them had seen. This reads as a property of a vintage, not a law of the species.

So. If you consume model probabilities in finance, treat a stated number as untrusted output of an unfinished layer. Measure it per model and per elicitation format against resolved outcomes, or do not consume it at all; we lean towards the latter. A cheap first screen needs no outcomes: ask everything twice, on two scales, and treat self-disagreement as disqualifying. And if you train models, the translation from weights to words is a clean post-training target. Resolved outcomes and a proper scoring rule are objective, cheap, and hard to contaminate. Score the words and the disposition together, so that the words do not improve without the decisions.