What are RL environments?

Every reinforcement-learning system has three components: an agent that acts, an environment that responds, and a reward signal that assesses the action. For language models, the environment can include documents, tools, spreadsheets, code, search, iteration, judgement, and a decision, not only a prompt and answer.

This guide covers what reinforcement-learning environments are in the context of language models, how they relate to benchmarks and evaluations, what makes a reward verifiable, and why the source of the underlying data changes everything about whether the training signal is trustworthy.

Reinforcement-learning environments for language models

An RL environment for a language model is a structured setting in which the model receives a task, generates an output, and receives a scalar reward. The environment defines the rules: what information the model can see, what format the answer must take, and how the answer is scored.

In practice, an RL environment consists of four parts:

A task specification. This is the question or instruction the model receives, along with any supporting context: documents, data, constraints, and a defined scope of reasoning. The task must be precise enough that two people reading it would agree on what counts as a good answer.

A context window. The information available to the model. In a finance environment this might be a set of deal documents, financial exhibits, credit-agreement terms, and public filings, all anchored to a specific date. The date matters: an environment that lets information from the future leak into the context is not testing judgment, it is testing recall.

A grading function. The mechanism that takes the model's output and produces a reward. This can be deterministic (rule-based extraction and comparison), semi-deterministic (structured rubric with weighted criteria), or model-based (an LLM judge). Each has failure modes, and those failure modes shape what the training signal actually teaches.

A reward signal. The scalar output of the grading function. In RL, this is what the model's policy optimises against. Noisy reward increases variance and slows learning. A reward that can be gamed teaches the model to game it. The reward is the thing that matters, and the thing that breaks first.

How RL environments differ from benchmarks and evals

The following three terms are often used interchangeably in practice. They should not be. They describe different things, with different failure modes.

A benchmark is a fixed test set with a defined scoring procedure. MMLU, HellaSwag, GSM8K. Benchmarks measure static performance on a frozen set of questions. Their results are often collected into leaderboards, but a leaderboard presents benchmark results; it is not part of the benchmark itself. Benchmarks are useful for comparing models at a point in time, and they degrade as training data absorbs the test set. A benchmark tells you how a model scored. It does not generate a training signal.

An evaluation (eval) is a broader term for any systematic measurement of model capability. An eval can be a benchmark, but it can also be a structured assessment that includes task design, grading criteria, and analysis of failure modes. Evals are diagnostic: they are built to reveal something about how the model reasons, not just what score it achieves. Our research on LLM judge noise across 564 runs is an eval in this sense: a structured experiment designed to surface a specific failure mode in our own scoring math.

An RL environment is an eval that produces a training signal. It includes everything an eval includes, plus the infrastructure to generate reward at scale, reliably enough that a policy gradient can learn from it. This is a higher bar than evaluation. A noisy but informative eval may still be useful for diagnosis. A noisy RL environment trains a policy noisily, increasing the variance of each update and slowing learning. The greater danger is a reward that can be gamed. Optimisation will reinforce whatever earns reward, including shortcuts. Credit must therefore be contingent on correct, evidence-grounded reasoning rather than its appearance.

The key distinction: benchmarks are consumed. Evals are diagnostic. RL environments are generative. They produce the signal that shapes future model behavior. That distinction has consequences for how carefully the reward must be engineered, and how seriously the data underneath it must be sourced.

What makes rewards verifiable

A verifiable reward is one whose correctness can be checked against an external reference, independently of the model that produced the answer and independently of the person who wrote the question. The term RLVR (reinforcement learning with verifiable rewards) describes a training paradigm where the reward signal meets this standard.

In mathematics, verifiable rewards are possible because proof assistants such as Lean can check whether a formal proof is valid. The model is trained for mathematical reasoning and proof, not merely to return the result of a calculation. In code generation, a test suite can play the same role: the code either passes or it does not. These are domains where RLVR works cleanly because correctness can be checked against an external procedure rather than a grader's impression.

In judgment-heavy domains, verifiability is harder. Whether a credit analysis is sound, whether a risk assessment captures the material factors, whether a restructuring thesis holds under stress, these are the questions that matter in finance, and they do not have arithmetic answers. Verifiable rewards in these domains require a different foundation.

We approach this with dated evidence from documented financial events. The model reasons about the situation as it stood at a specific point in time. An LLM judge then scores the answer against a locked, fact-bound rubric; practitioners do not enter the running pipeline for each answer. Later outcomes provide external evidence and calibration, but they do not alone determine reward.

Verifiability is not binary. A reward can be more or less verifiable depending on how tight the link is between the grading criteria and the external reference. A task that asks for a specific financial ratio has a tightly verifiable reward. A task that asks for a judgment call about covenant headroom has a looser link, and the reward function must be engineered to reflect that uncertainty rather than paper over it.

Verifiable rewards (RLVR) in practice

RLVR training loops use the verifiable reward as the policy-gradient signal. The model generates answers, the grading function checks them against the reference, and the reward updates the policy. A common optimiser is Group Relative Policy Optimisation (GRPO): it samples a group of answers to the same prompt and estimates each answer's advantage relative to the group's rewards, avoiding a separately trained value model. The algorithmic details vary, but the reward constraint does not: it must be checkable, reproducible, and resistant to gaming.

The failure modes are specific. A reward function that can be gamed can train the model to exploit the grader rather than improve at the task. Graders therefore receive adversarial tests for shortcut reward. Detailed controls remain sealed.

The noise floor of the reward function also matters. In one 69-task evaluation, 68% of scores landed at exactly zero with a tail to 0.7. The scoring design, not only the model, contributed to that shape. The design changed, and matched post-change validation remains pending.

Reliable rewards demand reliable evidence. The supporting materials must be dated, and the temporal boundary must be enforced. A model that uses information unavailable at the decision date has demonstrated recall, not judgement. Controlled variants and positive controls can test specified recognition channels, within measured limits.

Why real-world data matters for RL environments

RL environments can be built from synthetic data, from curated datasets, or from the documentary records of real events. The choice is not aesthetic. It determines what the reward signal can measure and what the training can teach.

Synthetic data is controllable. You can vary one parameter at a time, isolate specific reasoning steps, and generate large training sets. Our research uses it for diagnostics: the 564-run prompt framing study used a synthetic deal because every variable except the framing intervention had to stay fixed. Its ceiling is the fidelity with which the synthetic case recreates the information in a real financial situation.

That ceiling is hard to raise. To write a fair rubric, the creator must anticipate the valid routes a model might take through incomplete and conflicting evidence. Anticipating those routes is itself financial reasoning in an unfamiliar setting. It is the capability the environment is meant to train. Entirely synthetic finance environments are therefore not yet a substitute for real records. Real, high-quality data and expert human judgement remain necessary.

Real-world data is harder to work with. Documentary records must be sourced, cleaned, anonymised, and anchored in time. Their value is that they preserve the evidence available when a real decision was made, including routes through the evidence that a synthetic author may not anticipate, and provide later evidence about what followed.

Anonymisation removes identifying content without discarding the financial structure that makes a case useful. Case Foundry creates deterministic forks under a frozen name map, scrubs company and figure-level identifiers, and records each fork's lineage. Comparing an original case with its anonymised fork helps distinguish recognition of a known deal from reasoning over the evidence.

This is the thesis behind our approach to environment construction. Judge the decision from the contemporaneous record. Use later outcomes as external evidence and calibration. A sound decision can lose, and a weak decision can get lucky.

How finance RL environments differ

Finance is not uniquely complex. It is unusually suitable for evidence-linked environments because decisions, the information available at the time, and later outcomes often leave a detailed, dated record.

When a credit facility is drawn, the terms are recorded. When a company misses a covenant, the breach is reported. When a restructuring closes, the recovery becomes evidence. This record supports point-in-time assessment and later calibration, while professional judgement remains necessary.

Those records can support different kinds of financial reasoning: diagnostic reasoning (what is happening in this business), temporal reasoning (what was knowable at this date), counterfactual reasoning (what would have changed if this covenant had been enforced), comparative reasoning (how does this facility compare to that one), and valuation reasoning (what is this worth, and on what basis). An environment may validly train only one of them. Covering several within the same anchored case is more efficient: the same documentary record produces a denser training signal across multiple skills.

The reference case in our corpus illustrates the structure: a specialist credit fund, a consumer platform whose revenue stopped arriving, four anchor dates, and 117 graded tasks across seven categories of reasoning. A single documented deal, worked end to end through the environment construction pipeline, produces tasks in the low hundreds. The full corpus currently supports production of more than 20,000 graded tasks.

Anchoring and prompt sensitivity are specific risks in finance environments. Our research on cross-deal anchoring bias showed that a single sentence of market chatter in one deal's context moved the model's valuation of an unrelated deal by half a turn of EBITDA. The study covered one controlled deal sequence, so it does not establish a general rate. It shows why an environment should test this risk instead of assuming isolated tasks remove it.

Inside a Dissei environment

Each task puts the model exactly where an analyst sat. It receives a question, a date, and the case materials as they existed then — nothing else. The materials live behind a small set of retrieval tools: list what exists, search it, read a bounded window of it. A tool budget keeps episodes comparable. The question says plainly: everything after this date is unknowable to you, it is not in the dataroom, and reasoning from hindsight is penalised. Then the model submits its answer, and a grader it never sees scores it against criteria that were locked before the run began.

The anchor date is the whole trick. Try it:

Everything dated after the anchor might as well not exist: the tools enforce the cutoff at retrieval time, and grading enforces it a second time on the answer. A model that writes a beautiful analysis using December’s restructuring agreement while anchored in March has not been clever — it has failed the one rule the task had.

How the reward is shaped

Tasks are built backwards from answers: because each record is complete and its outcome known, what a good answer looks like is a matter of record rather than an author’s opinion. The grading criteria are fixed before any model runs. The score itself is decomposed rather than holistic — four tiers, each bound to sourced facts:

• Gates. The facts without which no answer is acceptable. Miss one and the score collapses, however fluent the prose.
• Criteria. Partial credit defined per criterion in advance, so a partially correct chain of reasoning is worth measurably more than an absent one. This is what keeps the reward dense enough to learn from.
• Penalties. They encode the errors professional analysts actually make. Trigger one and it subtracts.
• Grounding. Where the task grants retrieval tools, the reward scales by whether the answer uses the evidence the case actually required — reading around the dataroom costs you the credit.

Feel how the tiers compose:

The profitable policy that falls out of this shape is that of a sound analyst: commit to a verdict, ground it in the evidence you consulted, avoid the known errors, and decline to conclude when the evidence cannot support a conclusion — restraint earns credit too. Nothing in the scoring rewards confident verbosity. What RL optimises is exactly the behaviour the reward makes optimal.

Building trustworthy RL environments

An RL environment is only as trustworthy as its weakest component. The data must be sourced and temporally anchored. The assessment must match what the dated evidence can support. The grader must receive adversarial tests, and its consistency must be measured.

Those are the general requirements. In our own pipeline, they become a repeated operating discipline rather than a one-time exercise: the rubric freezes before candidate evaluation, high-level controls and measured results are public, and detailed test operations, acceptance rules, and production settings remain sealed.

The quality of an environment is visible in its training signal. Rewards must be dense enough to learn from; task difficulty must create a useful spread of model performance; and the distribution of cases must resemble the real decisions the policy will face. Noise, gaming resistance, and temporal leakage are not secondary controls. They determine whether that signal reflects the reasoning we intended to train.

A useful environment separates models on the reasoning the real task requires.