What are RL environments?

Every reinforcement-learning system has three components: an agent that acts, an environment that responds, and a reward signal that tells the agent how well it did. In robotics the environment is a physical space. In game-playing it is a simulator. In language-model post-training the environment is a task: a prompt, a set of reference materials, a grading function, and a reward that is computed after the model produces its answer.

This guide covers what reinforcement-learning environments are in the context of language models, how they relate to benchmarks and evaluations, what makes a reward verifiable, and why the source of the underlying data changes everything about whether the training signal is trustworthy.

Reinforcement-learning environments for language models

An RL environment for a language model is a structured setting in which the model receives a task, generates an output, and receives a scalar reward. The environment defines the rules: what information the model can see, what format the answer must take, and how the answer is scored.

In practice, an RL environment consists of four parts:

A task specification. This is the question or instruction the model receives, along with any supporting context: documents, data, constraints, and a defined scope of reasoning. The task must be precise enough that two people reading it would agree on what counts as a good answer.

A context window. The information available to the model. In a finance environment this might be a set of deal documents, financial exhibits, credit-agreement terms, and public filings, all anchored to a specific date. The date matters: an environment that lets information from the future leak into the context is not testing judgment, it is testing recall.

A grading function. The mechanism that takes the model's output and produces a reward. This can be deterministic (rule-based extraction and comparison), semi-deterministic (structured rubric with weighted criteria), or model-based (an LLM judge). Each has failure modes, and those failure modes shape what the training signal actually teaches.

A reward signal. The scalar output of the grading function. In RL, this is what the model's policy optimizes against. If the reward is noisy, the model learns noise. If the reward can be gamed, the model learns to game. The reward is the thing that matters, and the thing that breaks first.

How RL environments differ from benchmarks and evals

The three terms are used interchangeably in practice. They should not be. They describe different things, with different failure modes.

A benchmark is a fixed dataset with a leaderboard. MMLU, HellaSwag, GSM8K. Benchmarks measure static performance on a frozen set of questions. They are useful for comparing models at a point in time, and they degrade as training data absorbs the test set. A benchmark tells you how a model scored. It does not generate a training signal.

An evaluation (eval) is a broader term for any systematic measurement of model capability. An eval can be a benchmark, but it can also be a structured assessment that includes task design, grading criteria, and analysis of failure modes. Evals are diagnostic: they are built to reveal something about how the model reasons, not just what score it achieves. Our research on LLM judge noise across 564 runs is an eval in this sense: a structured experiment designed to surface a specific failure mode in our own scoring math.

An RL environment is an eval that produces a training signal. It includes everything an eval includes, plus the infrastructure to generate reward at scale, reliably enough that a policy gradient can learn from it. This is a higher bar than evaluation. An eval that is noisy but informative might be useful for diagnosis. An RL environment that is noisy trains a noisy policy. The tolerance for imprecision is much lower.

The key distinction: benchmarks are consumed. Evals are diagnostic. RL environments are generative. They produce the signal that shapes future model behavior. That distinction has consequences for how carefully the reward must be engineered, and how seriously the data underneath it must be sourced.

What makes rewards verifiable

A verifiable reward is one whose correctness can be checked against an external reference, independently of the model that produced the answer and independently of the person who wrote the question. The term RLVR (reinforcement learning with verifiable rewards) describes a training paradigm where the reward signal meets this standard.

In math, verifiable rewards are straightforward: the answer to a calculation is either correct or it is not, and correctness can be checked by computation. In code generation, a verifiable reward can be a test suite: the code either passes or it does not. These are the domains where RLVR has worked cleanly, because the reward function has a ground truth that is external to any human judgment.

In judgment-heavy domains, verifiability is harder. Whether a credit analysis is sound, whether a risk assessment captures the material factors, whether a restructuring thesis holds under stress, these are the questions that matter in finance, and they do not have arithmetic answers. Verifiable rewards in these domains require a different foundation.

We approach this by building tasks backwards from documented outcomes. When the outcome of a financial event is known and the complete documentary record exists, a task can be constructed that asks the model to reason about the situation as it stood at a specific point in time. The grading criteria are derived from what actually happened, not from what an evaluator believes should have happened. This is what makes the reward verifiable: it is anchored to an external fact, not to an opinion.

Verifiability is not binary. A reward can be more or less verifiable depending on how tight the link is between the grading criteria and the external reference. A task that asks for a specific financial ratio has a tightly verifiable reward. A task that asks for a judgment call about covenant headroom has a looser link, and the reward function must be engineered to reflect that uncertainty rather than paper over it.

Verifiable rewards (RLVR) in practice

RLVR training loops use the verifiable reward as the policy gradient signal. The model generates an answer, the grading function checks it against the reference, and the reward is used to update the policy. This is the same loop as any other RL training, but the reward function is constrained: it must be checkable, reproducible, and resistant to gaming.

The failure modes are specific. A reward function that can be gamed does not crash or error out. It quietly trains the model to exploit the grading function rather than improve at the task. This is reward hacking, and it is close to the default outcome if the grading function is not adversarially tested before deployment. We run three attacks against every grader before its task ships, hunting reward without reasoning. A task that survives ships. A task that does not gets rewritten.

The noise floor of the reward function also matters. If scoring noise is high relative to the true performance signal, the model trains on noise. Our experience with a 69-task eval where 68% of scores landed at exactly zero taught us that the grading math itself can manufacture the noise, independent of the model's actual performance. The fix was a one-line change to the scoring function. The diagnostic that caught it was running the same task 564 times and looking at the distribution.

Verifiable rewards demand verifiable data. A reward that is anchored to an outcome requires that the outcome is documented, the supporting materials are complete, and the temporal boundary is enforced. A model that scores well because it reasons from information that was not available at the anchor date has not demonstrated judgment. It has demonstrated recall. The contamination probe we run on every environment tests exactly this: does the model's performance survive anonymization?

Why real-world data matters for RL environments

RL environments can be built from synthetic data, from curated datasets, or from the documentary records of real events. The choice is not aesthetic. It determines what the reward signal can measure and what the training can teach.

Synthetic data is controllable. You can vary one parameter at a time, isolate specific reasoning steps, and generate arbitrarily large training sets. Our research uses synthetic data for specific diagnostic purposes: the 564-run prompt framing study used a synthetic deal precisely because we needed to control every variable except the one we were testing. But synthetic data has a structural limitation. It teaches the model to reason about synthetic situations. Whether that reasoning transfers to real situations is an empirical question, and the answer is not always yes.

Real-world data is harder to work with. Documentary records must be sourced, cleaned, temporally anchored, and anonymized. The task construction pipeline is slower. But the reward signal has a property that synthetic data cannot match: it is grounded in what actually happened. The model is not learning to produce answers that look reasonable. It is learning to produce answers that would have been correct, judged against an outcome that is a matter of record.

This is the thesis behind our approach to environment construction. Judgment is graded against what actually happened, or it is not graded at all. A rubric score from a human evaluator tells you what the evaluator thought. An outcome-graded reward tells you whether the model's reasoning aligned with reality.

How finance RL environments differ

Finance is not unique because the domain is complex. Every domain is complex. Finance is unusual because the ground truth is documented.

When a credit facility is drawn, the terms are recorded. When a company misses a covenant, the breach is reported. When a restructuring closes, the recovery is known. The documentary record of a financial transaction is detailed, temporally structured, and, for the purposes of environment construction, complete. This is what makes finance a natural domain for RL environments with verifiable rewards: the outcomes exist, they are recorded, and they can be worked backwards into grading criteria.

The reasoning categories that matter in finance are also specific: diagnostic reasoning (what is happening in this business), temporal reasoning (what was knowable at this date), counterfactual reasoning (what would have changed if this covenant had been enforced), comparative reasoning (how does this facility compare to that one), and valuation reasoning (what is this worth, and on what basis). A finance RL environment must test all of these, because a model that excels at extraction but fails at counterfactual reasoning is not useful for the decisions it will be asked to support.

The reference case in our corpus illustrates the structure: a specialist credit fund, a consumer platform whose revenue stopped arriving, four anchor dates, and 117 graded tasks across seven categories of reasoning. A single documented deal, worked end to end through the environment construction pipeline, produces tasks in the low hundreds. The full corpus currently supports production of more than 20,000 graded tasks.

Anchoring and prompt sensitivity are specific risks in finance environments. Our research on cross-deal anchoring bias showed that a single sentence of market chatter in one deal's context moved the model's valuation of an unrelated deal by half a turn of EBITDA. This is not an edge case. It is the default behavior, and an environment that does not test for it is training a model that carries it.

Building trustworthy RL environments

An RL environment is only as trustworthy as its weakest component. The data must be sourced and temporally anchored. The tasks must be constructed so the answer is determinable from the available information. The grading function must be adversarially tested. The reward signal must be measured for noise, bias, and gamability.

This is not a one-time exercise. Every new task enters the pipeline with the same requirements: the rubric freezes before any model sees the task. The grader is attacked before the task ships. The noise floor is measured, not assumed. The contamination probe runs on every environment.

The goal is not a perfect reward signal. The goal is a reward signal whose imperfections are known, measured, and bounded. An RL environment that produces reward with a known noise floor and a tested resistance to gaming is useful. An environment whose reward is assumed to be clean is dangerous, because it trains with confidence on a signal that has not been verified.

Judgment is graded against what actually happened, or it is not graded at all.