Jev’s probabilities, put to the test

Our previous probability article asked whether the numbers language models give us deserve to be treated as probabilities. TypeSafe AI’s new model, Jev, makes that question especially interesting: probabilistic decisions are its interface.

We gave Jev and three language models 1,200 resolved stock-price questions, then tested equivalent questions and newly generated cases with known probabilities. Jev’s historical Brier score was 0.1941, against 0.1126 for a simple volatility calculation. Lower is better. Its answers to identical requests were fairly stable, but complementary questions produced substantial disagreement.

Jev gave us usable numbers. The harder question was whether those numbers behaved like probabilities. That requires separating three things: an interface that returns a value of the right type, accuracy against outcomes, and consistency between related questions.

A useful interface, a separate accuracy question

TypeSafe introduced Jev on 15 September as its first “System One Model”: unstructured state goes in, typed probabilistic decisions come out. The company describes a new architecture, parallel sampling and a training method called Reinforcement Learning for Calibrated Decisions, or RLCD.

For a Yes/No question, its Noul documentation defines the output as the probability that the answer is Yes. That is a useful interface for software which must route a request, apply a threshold or combine uncertain judgments. A language model asked to write a JSON object containing a probability reaches the application through a different interface.

There is an important boundary to this comparison. TypeSafe’s Jev 1.13 limitations page explicitly cautions about numerical precision, recommends keeping arithmetic in code and warns that separate questions need not obey structural invariants. Its launch evidence centres on automation workflows, with reference probabilities supplied by other large models.

Our experiment tests the transfer of Jev’s probability interface to numerical forecasting. It is a demanding use case, outside the semantic judgments the documentation emphasises. We measure API behaviour; we cannot identify the effect of its architecture or training method. We also queried the Gateway alias typesafe-ai/jev, whose responses did not expose a verifiably fixed underlying version. The limitations page is context, not proof that the alias served that exact version.

A retrospective test with no waiting

Each historical question supplied 250 closing prices and asked whether the close five trading sessions later would be above a threshold, or at or below it. We used 50 stocks, four origins in 2017 and six thresholds per stock/date window: 1,200 questions built from 200 windows.

Prices were normalised to end at 100. Tickers and dates were withheld. Every window had three above-threshold and three at-or-below questions, assigned without looking at the future return.

All outcomes had already resolved. This avoids waiting for live events, but it changes the claim we can make. These are retrospective questions. Normalisation and hidden names do not prove that the histories were absent from training data. The archive also imposes membership and complete-case biases. This is a new study, rebuilt without the earlier harness; its scores are not directly comparable with the previous article’s.

Jev returned its native probability. GPT-5.6 Luna, Gemini 3.8 Flash and Qwen3.8 27B returned a requested JSON probability through OpenRouter. Luna and Qwen had reasoning disabled; Gemini used low reasoning. None received tools, retrieval or the future outcome. These are comparisons of those complete configurations, not isolated tests of model architecture.

The reference calculation estimated recent volatility with an exponentially weighted moving average, or EWMA, then assumed Gaussian daily log returns with zero mean. Thresholds were themselves scaled by that volatility, so this defines the kind of probability question being tested. A constant 50% answer supplied a second reference.

Across the historical panel, diagnostics, synthetic cases and token probes, we completed 8,600 planned jobs, yielding 8,599 valid outputs. One Luna synthetic answer reached its output limit and remained a recorded failure. Jev returned a valid probability on all 2,120 of its final questions. Valid output is an interface result; the following comparisons ask what those values mean.

Accuracy: the baseline wins

The Brier score averages (p − y)², where p is the forecast probability and the outcome y is either zero or one. It rewards accurate probabilities. It does not measure calibration alone: a forecast’s ability to distinguish events also matters. An always-50% forecast incurs a loss of 0.25 whichever outcome occurs.

Every system answered all 1,200 primary questions. The comparison below is the pooled historical result, not a ranking across independent market trials.

Jev improved on the constant forecast and Qwen, but trailed the other two models and the calculation. Its Brier disadvantage against EWMA was 0.0816, with a 95% issuer-bootstrap interval of 0.0704 to 0.0927. This is the reported paired estimate, not a subtraction of the rounded displayed means. EWMA scored better at every origin. Gemini also trailed EWMA at every origin, though by much smaller amounts.

These are 50 issuers and four dates, not 1,200 independent market trials. Our intervals resample whole issuers, keeping their dates and thresholds together. They condition on the four origins and do not fully capture common-market dependence.

That qualification matters particularly for close comparisons. Jev trailed Luna by 0.0114. Its issuer interval, 0.0025 to 0.0199, excluded zero; an exploratory bootstrap resampling both issuers and dates, −0.0012 to 0.0241, crossed it. Jev’s much larger gap against EWMA remained positive in that sensitivity check, at 0.0646 to 0.0978.

And what happened when Jev said roughly 70%? Among 178 forecasts in the predefined 65% to below-75% band, its mean forecast was 69.1%, and the event occurred 63.5% of the time. The observed-frequency interval was 55.8% to 71.4%, which includes the mean prediction. That band alone does not establish a calibration error. The pooled accuracy comparison and the consistency tests answer different questions; neither should be reduced to this one band.

Consistency: stable is not the same as coherent

A probability has obligations beyond matching outcomes. If a price must finish either above a threshold or at or below it, the two probabilities must sum to one. Changing the price unit should leave the event unchanged. Equivalent wording should preserve its meaning.

We tested these properties on a fixed subset of 120 questions. Each also received two identical repeats, giving a benchmark for ordinary variation between requests. Repeat noise averages the three pairwise gaps among the original request and its two repeats. Complement error measures distance from a sum of one; wording and rescaling gaps measure disagreement with the parent question. All are expressed in percentage points, but they test different properties.

Jev’s repeats were close: a mean gap of 1.42 percentage points. Its mean complement error was much larger: 11.64 points. Subtracting the repeat benchmark within each question left a gap of 10.23 points, with an issuer interval of 9.03 to 11.35. These are reported paired results; the displayed rounded averages are not sufficient to reconstruct them. The equivalent-wording and rescaling gaps also exceeded its repeat benchmark.

The threshold sequence exposed a related problem. As a threshold rises, the probability of finishing above it should not rise. After converting every answer to that same direction, Jev violated this ordering at least once in 165 of 200 windows, or 82.5%. Gemini did so in nine windows, or 4.5%. This counts any upward step; it does not tell us the size of that step. The supplied results report these rates, not the magnitudes of the violations.

We also supplied a numerical summary: historical returns, volatility and threshold distance. The summary added no future information. Jev’s subset Brier moved from 0.1996 to 0.2085. The paired change was +0.0089, with an interval from −0.0126 to +0.0286. That does not establish improvement or deterioration; the intervals for all four models’ summary effects included zero.

Jev’s small repeat gaps are useful evidence about the service we queried, including its two-decimal output. They do not establish logical coherence. Repeatability, complement consistency and threshold monotonicity are distinct properties. Nor does coherence alone establish forecasting skill: an always-50% answer can satisfy these identities while distinguishing nothing.

Calculation: when the probability is known

Historical forecasting mixes several difficulties: interpreting the input, estimating a distribution and facing a future that may differ from the past. We therefore generated 200 fresh synthetic cases under a simpler rule.

Daily log returns were independent Normal draws. Every model received the exact mean, volatility and generating law, with an instruction to use those parameters. Thresholds covered probabilities from approximately 5% to 95%. The correct answer followed from the Normal distribution.

We could now compare the returned probability directly with the true conditional probability. No lucky or unlucky realised outcome was needed for that comparison. The distinction is between calculating from the disclosed law and estimating parameters from the supplied history.

One Luna answer was truncated. The comparison below uses the 199 cases with valid answers from all four models, not all 200 generated cases. The original available-answer scores, and bounds for the missing answer, are in the results appendix.

Jev’s mean absolute error was 23.93 percentage points, with a case-bootstrap interval of 21.86 to 26.24. Gemini’s was 1.07 points, with an interval of 0.85 to 1.32. The calculation using the known law has zero error by construction; the EWMA baseline must instead estimate volatility from history.

On this control, Jev’s error was close to the constant-50% reference. Its mean squared error against the true probability was 0.08266, compared with 0.08190 for that constant. This quantity is also the excess expected Brier score relative to the exact answer. Unlike the historical Brier score, it compares with a known probability rather than a realised zero-or-one outcome.

The control tests probability calculation under a disclosed law. It does not establish market skill or solve the historical panel’s training-exposure problem. It does show that providing the law and its parameters was insufficient for Jev to return accurate numerical probabilities in this configuration. That result fits the numerical limitations TypeSafe documents.

Readouts: two ways to ask, different answers

The previous article also asked whether a stated probability differs from the probability of generating “Yes”. We revisited that question with Qwen on the same 120 historical parent questions.

We requested a one-word Yes/No answer and obtained actual first-token log probabilities. Both alternatives were present in every case. After normalising their probability mass, the token-based number differed from the stated number by 20.21 percentage points on average.

Qwen’s Brier score improved from 0.2251 to 0.1624 with the token readout. The reported paired difference was −0.0627, with an issuer interval of −0.1112 to −0.0151. The broader issuer/date sensitivity crossed zero, so four dates do not establish that this advantage will generalise.

The requested output changed along with the readout. This comparison therefore cannot separate the effect of the instruction from the effect of extracting token probabilities. It also does not reveal an independently identified “true belief”. It demonstrates that the probability exposed to an application depends materially on how we obtain it.

What we would put into software

Jev’s interface makes a useful promise: software can request a probabilistic decision and receive a value of the required type. Our run delivered that on every final Jev question. Numerical reliability still required separate measurement.

On this benchmark, Jev trailed a simple calculation, disagreed with itself across complementary questions and struggled when the exact probability law was supplied. Those findings concern this numerical task and these tested configurations. They do not settle its value for classification, routing or other semantic judgments.

For this forecasting problem, we would keep the probability calculation in code. For a judgment that requires a model, we would first test its probabilities against relevant outcomes, compare them with a simple reference and check the identities the application relies on.

A native probability interface makes the value easier to consume. Accuracy and coherence tests tell us what the application can reasonably do with it.

Sources and study notes

The numerical results are transcribed from Dissei’s supplied manuscript, dated 25 September 2026, and checked against the study’s supplied results tables, reproduced in the results appendix. Historical scores use 1,200 questions from 200 stock/date windows; consistency and Qwen readout comparisons use 120 matched parent questions; synthetic comparisons use 199 common valid cases from 200 generated cases. Uncertainty and paired estimates are retained as reported, not recomputed from rounded means. The scoring exercise is an illustrative calculation, not a model response or an additional experiment.

A few statements rest on the manuscript alone, because the results tables do not repeat them: the 50-issuer design and question construction, the model configurations, Jev’s 2,120 valid final questions, EWMA’s lead at every origin, Jev’s complement-minus-repeat gap of 10.23 points (issuer interval 9.03 to 11.35) and the issuer/date sensitivity for Qwen’s token readout. The manuscript’s three original figure images, and the analysis note it cites for the matched synthetic subset, were not supplied. This draft therefore presents aggregate comparisons and does not reconstruct reliability bins, origin-level scores, synthetic response curves, monotonicity magnitudes or individual paired readouts.

Preceding article: “Saying 70, meaning 40”. This study uses a rebuilt harness; results are not directly comparable.

TypeSafe: “Introducing System One Models & Jev”, dated 15 September 2026. Source for the interface, architecture and training claims and the scope of TypeSafe’s launch evaluations.

TypeSafe Noul documentation. Defines the Yes/No output as the probability that the answer is Yes.

TypeSafe: Jev 1.13 jaggedness, last reviewed 17 September 2026. Documents numerical and structural-invariance limitations. The tested Gateway alias did not establish an exact underlying version.

Results appendix

These tables reproduce the study’s supplied results in their original units: Brier scores and synthetic probability errors on a 0–1 scale, band frequencies and violation rates in percent, consistency gaps in percentage points. Intervals are 95%. The study charts above draw on the same values; they add no data of their own.