Does the model know the deal, or understand it?

Every one of the 69 tasks in our reference case is built to exist twice: once inside the original deal room, and once inside a byte-level anonymized fork with every identifying detail (the company, the figures, the distinguishing language) stripped and replaced throughout all 30 sources and 18 exhibits. We built the second version because of a plain worry: a model can score well on the first version by recognizing the company, not by reading the documents in front of it.

That worry has a name. Geirhos et al. call it a shortcut¹: solving a task by latching onto a signal correlated with the right answer, without doing the work the task was built to measure. If a company is public, was covered in the trade press, or shows up in a research note scraped into a training corpus years ago, a model can “know” how a deal turned out before it reads a page of the dataroom, and produce a fluent, confident, well-structured answer for a reason that has nothing to do with the documents in front of it. Recognizing a company is not reasoning about a deal.

A model that scores well on a real transaction may be reciting what it already knew rather than reading what you gave it. And until you measure the gap, you cannot tell which one you are grading.

The trouble is that measuring a gap requires an instrument, and an instrument nobody has validated is not evidence of anything. So the question we had to answer before any recognition number meant anything was not “does the model recognize the deal?” It was: does our own probe for recognition actually work?

—The control comes first

The instrument here is an identity probe: something that looks at a model’s answer and estimates whether it is drawing on knowledge of the real, named deal. Run that probe against the anonymized fork and get a clean reading, and it is tempting to conclude the fork carries no identifying signal. But a clean reading from a probe that could never detect identity under any circumstances is not evidence of a clean fork; it is evidence of a broken instrument, and the two look identical unless the instrument has already been tested somewhere it should obviously fire.

That somewhere is the original, unmodified case. The un-anonymized parent runs through the identity probe first, as a positive control, before the probe’s verdict on any fork is trusted. If the probe cannot detect identity in a document that plainly and undeniably contains it, the probe is broken, and every “clean” fork it has ever certified is unverified, not verified. The control comes first. Only once the probe fires correctly on the parent does its silence on the fork mean what we want it to mean.

The fork itself has to earn the same scrutiny the probe does. Anonymization runs at the byte level across every document in a case: not a summary pass, not a find-and-replace on the company name, but a transform fingerprinted against its parent, so every fork is traceable to exactly the document and transform run that produced it. A pipeline that quietly misses one identifying detail is worse than no anonymization at all, because it manufactures confidence in a fork that still leaks; a transform that only partially succeeds never ships. The mechanics of that check remain sealed; the posture it enforces is the point.

—Recognition, measured — and then made moot

With a validated instrument, the measurement is simple to state: run the same task against the original case and against its anonymized fork, and take the score delta. Call that delta the recognition share: the portion of a model’s score on the original that evaporates once everything identifying is gone. Reasoning from the documents should score about the same on both versions. Coasting on recognition should not. We have not measured those deltas yet, and we are not going to estimate them here; the design is what we can show today, and results will be reported only after measurement.

Why does the third version exist at all? Because anonymization alone does not settle the question, even once the instrument is trusted: a capable model does not always need the name to know the company. An unusual margin structure, a specific facility size, a covenant package that shows up in exactly one real deal can be as identifying as a name. That is where counterfactual forks go further than anonymization: instead of only relabeling, we consistently perturb the load-bearing figures throughout a case and regenerate the gold answer from the fact ledger to match. A model that memorized the real answer now gets it wrong, not because the label is missing, but because the correct answer changed and its memorized one didn’t. Hindsight Open, the public tier of our benchmark, is built entirely from cases constructed this way: every task a model outside our walls can run is graded against gold answers regenerated from perturbed figures. Hindsight Sealed, the tier we run ourselves against a submitted model endpoint, applies the same fork logic to cases that never leave our own infrastructure at all, independent cover against the same failure mode, on a set the public can never train against by accident.

—Measured limits

None of this is a claim that contamination is zero. Writing style, document structure, and rare combinations of otherwise-unremarkable facts could in principle still correlate with a known deal in ways neither the anonymization nor the perturbation touches. The honest claim is narrower: we measure the gap on the one axis we can build a positive control for, and we say plainly which axis that is. A positive control validates the instrument for the signal it targets, not for every possible signal.

Our cases begin as eligible records from Dissei’s own lending activity: memos, financials, exhibits, and timelines as they stood at the decision, rebuilt point-in-time before any anonymization or perturbation runs. Secondary licensed origination can add further cases under its own provenance and rights, without changing the fork methodology built on top.

Ask any benchmark vendor for their recognition delta, not just their leaderboard score, and ask what validates their probe before you trust the delta at all. A vendor who has never run the positive control does not know whether their instrument would ever detect the thing it claims to rule out. We hold ourselves to the same standard: the design above is public, and our own deltas will be published as measurements, not estimates.