Last updated: October 24, 2026

PRACTICE MATTERS · LEGAL RL ENVIRONMENTS

Legal RL environments, built one matter type at a time

We build reinforcement-learning environments for legal work, and nothing else. A tax deficiency, an immigration petition and an M&A closing differ in their sources, their workflow and what counts as a right answer, so each matter type gets its own generation pipeline, data sources, task logic and rule base.

01 · LawyersA network of practising lawyersdesigns every matter type, cross-reviews its rubrics and verifies tasks and grading.

02 · GenerationEnvironments generated continuouslyevery figure computed, every document checked against the record; 2,000 tasks planned, about 5,000 with variations.

03 · EvaluationA research-led evaluation teammeasures what the data teaches: transfer to Harvey LAB, difficulty at pass@k, judge audits.

Evidence so far: a harness searched on 126 of our tasks lifted fully passed runs on Harvey LAB's patent freedom-to-operate work from to .

Our bet

Law is not one domain

Generic pipelines treat legal work as one task type with different nouns. We build each matter type from its own sources, its own way of setting a task and its own rules for what is right.

Tax deficiency

Sources: Internal Revenue Code, Treasury Regulations, Tax Court opinions
Task logic: recompute the deficiency and penalties from the return and the notice
Rule base: code sections, computation rules, burden of proof

Immigration · national interest waiver

Sources: INA § 203(b)(2), 8 C.F.R. § 204.5(k), AAO decisions
Task logic: review a petition's evidence before filing, prong by prong
Rule base: Matter of Dhanasar, 26 I&N Dec. 884 (AAO 2016): three prongs, evidentiary standards

M&A closing

Sources: purchase agreement, disclosure schedules, closing checklist, consents
Task logic: reconcile every condition and deliverable against the agreement
Rule base: conditions precedent, required consents, bring-down of representations

The dataset

2,000tasks planned · about 5,000 with variations

built so far, tested in this report

verified binary criteria

0–3 / 8rollouts meet every criterion, per task (sample test)

What it contains

Tasks span 24 practice areas, contract negotiation, firm knowledge and deal diligence. Each task is a workspace rather than a question[[2,3]]: the client's documents, a partner's request, a hidden rubric of binary criteria usable as a reward[[4,5]], and a separate verifier.

Built per matter type by practising lawyers, from public law, licensed cases and de-identified expert cases.
Verified twice: tasks and the model verifier's judgments are checked by experts.
Fictional parties, real law, cited as it stands.

Sample difficulty test

83–96%criteria met per rollout by Claude Opus 5.5 at maximum reasoning (mean 89%)

No harness, 8 rollouts per task, Claude Opus 5.5 and Codex. Bars: the eight Opus 5.5 rollouts, 80–100% of criteria.

Planned distribution: 2,000 tasks by area

area of each block = its tasks · hover for matters and share · the mix is configurable per client

Sample task

Four tasks from the dataset, one per area, laid out like public sample-task viewers[[12]], as an agent meets them: the request, the matter folder and its tools, the rubric the separate verifier applies, and one recorded run. Select a task on the left.

How a task is built

Eight stages, each checked by code before the next starts; legal experts design each matter type, review the matter design and the rubric, and verify the finished task and the verifier's judgments; a compliance and de-identification pass with human sign-off closes the build. Stages 01–06 build a task; 07–08 are the AutoHarness test. The subset in this report was built the same way.

AutoHarness test

Does training on the data improve an agent on an independent benchmark? We took the built tasks of five areas, searched an agent harness on them alone, following the idea of AutoHarness[[1]] with a natural-language harness (a practice guide in the agent's prompt) improved by reflective proposals[[6]], froze it and ran it on Harvey LAB[[11]] tasks that no task copies. One agent (Claude Sonnet 5.5) and one model judge (Claude Opus 5.5)[[7]] throughout; Harvey scores are never read by the search. The sections below report the result, how the runs change, and why the gain is skill rather than fitting.

Areas in the test · fully passed Harvey LAB runs, without → with the harness

Harvey LAB results

One agent (Claude Sonnet 5.5), one judge (Claude Opus 5.5), the same Harvey LAB tasks with and without the harness. The harness is chosen on practice matters alone and frozen before it touches a Harvey task.

Harvey LAB pass rate (every criterion met)

Share of runs that meet all of a task's criteria, without and with the harness.

Every task

Change in the share of criteria met, per Harvey task (small points); large point: the area mean, band: its 90% bootstrap interval over tasks.

Transfer

Every area gains more on its practice matters than on Harvey LAB. Fully passed tasks rise most where tasks were already close to a pass:

Gain on practice matters against gain on Harvey LAB

x: change in mean score on the practice matters. y: change in criteria met per Harvey task (90% interval). The diagonal is full transfer.

Inside the runs

How the harness runs work

Mean per Harvey run on six measures of how the agent works. Grey dashed: without the harness; colour: with it. Pick an area, or compare all areas with the harness; hover a point for the values.

Ten runs of one task, step by step

one square per tool call · hover a square for the command and what it returned

Four exhibits

Each pairs a run without the harness with a run of the same task with it: the criterion, the harness's own words, what each deliverable says, and the judge's reason. Every quotation is checked by code against its source.

Skill, not fitting

What the agent learns from our tasks is a general legal skill, not a way to game the benchmark. Four checks show the harness carries working habits that transfer to tasks it never saw, rather than any benchmark's answers; the last card notes what the checks leave open.

The subset, task by task

Every matter: one square per criterion. Top row without the harness, bottom row with it.metmissed

Criteria met

Training on the data: an early result

Small-scale validation · in progress, will be updated

The harness test changes the prompt, not the model. The next question is whether the tasks also work as reinforcement-learning data: train an open model's weights on our patent practice matters only, then test it on Harvey LAB tasks it never saw. A first small run says yes.

62.7% → 71.5%criteria met on 11 unseen Harvey LAB patent tasks (Opus judge)

10 of 11tasks improved

+12.7 ptsmean gain on the 4 tasks closest to FTO: claims compared with an accused product

Each unseen Harvey task, before and after training

Share of rubric criteria met. Grey: the base model; blue: after 7 rounds of RL on our tasks. ★ marks the four claim-versus-product infringement tasks. Hover a point for the value.

Qwen3.5-9B · one generation per task · graded by Claude Opus 5.5 on Harvey's own rubric · training logs on GitHub

How it was run

Model: Qwen3.5-9B with a LoRA adapter (rank 16, language model only)
Training data: 82 patent practice matters across 11 kinds of patent work; nothing from Harvey
Method: 7 rounds; each round 32 matters × 4 rollouts, rewarded by Claude Sonnet 5.5 against the matter's rubric, group-relative policy gradient
Test: 11 Harvey LAB patent tasks never seen in training, one generation each, graded by Claude Opus 5.5
Logs: github.com/Lawless-Inc/fto-rl-pilot-logs: the full training logs of this run

Why we trust the signal

The reward tracks the stronger judge. Opus re-graded 64 training outputs: per-output correlation with Sonnet's reward 0.86; from early to late rounds Sonnet saw +6.0 points and Opus +5.2, with 13 of 16 matters improving.
The gain grows with training. After 3 rounds at a small learning rate the Harvey score was flat (+0.6); rounds 4–7 at a higher rate brought the +8.7.
The training tasks improve too. The same practice matter, seen again later in training, scored higher in 68 of 82 cases.

In progress. This is one small run: 11 test tasks, one generation each, one model size. Still to come: held-out practice matters scored with the trained model, repeated generations for confidence intervals, longer training on the full patent set, and the same test in other practice areas. Outputs also grew longer during training (about 15k to 21k characters), which we have not yet separated from the gain. This section will be updated as results arrive.

Configurable Environment

Agent-Native IP Environment as Example

The same idea applied to intellectual property. Priorbit, our US patent and trademark database, is itself the environment: an agent receives a task, acts by calling the database's tools, reads what the environment returns, and is scored once, when it submits. Everything about the world it acts in is a setting.

01 · Agent-nativeEvery operation is a tool with one meaningSearch by title, claims, class, owner or citation; read claims, prosecution history or drawings; resolve a brand to its owners. Eight typed actions serve every task family, return structured results in a second or two, and run as thousands of independent episodes in parallel against one service, with no browser and no third-party site in the loop.

02 · You set the clockThe world is frozen at any date you chooseEach observation is filtered to the episode's snapshot date: patents granted later, applications published later, marks filed later, and status, ownership, citations and office actions decided later are invisible. The live web cannot be rewound; this can, so every run is reproducible and no answer leaks from the future.

03 · Complete and maintainedThe whole US record, kept current9.73 million granted patents since 1976, 8.70 million published applications since 2001, 14.35 million trademark records, 13.1 million office actions since 2007 and 5.2 million independent claims indexed by meaning. It is rebuilt nightly from USPTO bulk files, and each update passes data checks before it is served.

04 · Offline by designThe offline world frontier labs now train inOpenAI reports that all of its frontier RL runs use an "offline world" of indexed and cached web content, and that frontier training is conducted with no internet access[[14]]. A domain database built for agents is that offline world for intellectual property: rich enough to learn real search skills, and contained.

One task, many environments

A cordless stick vacuum, sued over in 2022. The complaint asserted US D853642, granted 2019-07-09. Each row changes one setting of the same task. These five are examples: the settings combine freely (any snapshot date, any subset of tools, any step budget, any reward weights), so one task yields as many environments as you configure, and the configuration travels with the episode.

Setting	Snapshot	What the agent sees	Used for
`default`	2022-05-12	The record as the lawyer had it on the filing date. Status is masked; D853642 shows 1 of the 30 patents that later cite it; a search for DYSON loses 5 of its first 20 marks, filed later.	The fair task
`as_of: "knowable"`	2019-07-09	The first day the answer existed: nothing granted after it, including the competitors' later designs.	The hardest fair setting
`snapshot: "off"`	today	All 30 citing patents, current legal status, later owners.	Ablation: how much hindsight helps
`tools: [−get_drawings]`	2022-05-12	Text only: design patents must be found and judged without their figures.	Action-space ablation
`reward: "recall"`	2022-05-12	Same observations; any correct patent anywhere in the answer scores 1.	Early curriculum, before rank-weighted reward

POST /env/reset  {"task_id": "tro-234", "config": {"as_of": "knowable+30d", "snapshot": "on",
                  "max_steps": 15, "tools": ["search_patents", "get_patents", "resolve_owner"], "reward": "strict"}}

162 tasks from real cases · 3 families · settings: snapshot date, snapshot on/off, step budget, tools, reward

Try it: you are the agent

Pick a task and its settings, then act. Each action is a tool call; the environment answers with feedback filtered to the snapshot date; submit once for the reward. Nothing here calls a model.

Snapshot Hindsight Steps Reward

State s₀—

Action a₁—

Environment—

Feedback o₁—

Reward r—

live against the production environment · the full workbench, with every argument and the raw JSON, is at uselawless.com/env

The paradigm: rebuild the world in a sandbox

Most of the world an agent needs is scattered, gated, messy and not built for agents: records split across registries and court systems, behind logins and rate limits, in scanned images, inconsistent formats and pages written for human eyes. We take that content offline and store it locally. We rebuild it around its own logic as a knowledge graph, so a patent links to its claims, family, citations, owners, office actions, drawings and the marks and companies around it, and we store an embedding for every element, whether an image, a video or a long text.

What the agent sees is not that storage but a small set of semantically general skills and interfaces, the operations a professional actually performs: search, read, resolve, compare, trace. In effect we rebuild the world inside a sandbox, make every property of it configurable (the date it is frozen at, the actions that exist, how work is scored), and expose it the agent-native way, the same environment for training and for inference. Intellectual property is the first world we have rebuilt this way.

References

Papers and books

[1]X. Lou, M. Lázaro-Gredilla, A. Dedieu, et al. AutoHarness: Improving LLM Agents by Automatically Synthesizing a Code Harness. International Conference on Learning Representations (ICLR), 2026. arXiv:2603.03329.
[2]N. Guha, J. Nyarko, D. E. Ho, et al. LegalBench: A Collaboratively Built Benchmark for Measuring Legal Reasoning in Large Language Models. Advances in Neural Information Processing Systems (NeurIPS), Datasets and Benchmarks Track, 2023. arXiv:2308.11462.
[3]B. Vidgen, A. Mann, A. Fennelly, et al. APEX-Agents. arXiv preprint, 2026. arXiv:2601.14242.
[4]A. Gunjal, A. Wang, E. Lau, et al. Rubrics as Rewards: Reinforcement Learning Beyond Verifiable Domains. International Conference on Learning Representations (ICLR), 2026. arXiv:2507.17746.
[5]R. K. Arora, J. Wei, R. S. Hicks, et al. HealthBench: Evaluating Large Language Models Towards Improved Human Health. arXiv preprint, 2025. arXiv:2505.08775.
[6]L. A. Agrawal, S. Tan, D. Soylu, et al. GEPA: Reflective Prompt Evolution Can Outperform Reinforcement Learning. International Conference on Learning Representations (ICLR), 2026. arXiv:2507.19457.
[7]L. Zheng, W.-L. Chiang, Y. Sheng, et al. Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena. Advances in Neural Information Processing Systems (NeurIPS), Datasets and Benchmarks Track, 2023. arXiv:2306.05685.
[8]R. A. Fisher. The Design of Experiments. Oliver and Boyd, Edinburgh, 1935.
[9]W. J. Dixon, A. M. Mood. The Statistical Sign Test. Journal of the American Statistical Association 41(236): 557–566, 1946.
[10]B. Efron. Bootstrap Methods: Another Look at the Jackknife. The Annals of Statistics 7(1): 1–26, 1979.

Web pages

[11]Harvey. Introducing Harvey's Legal Agent Benchmark (LAB). https://www.harvey.ai/blog/introducing-harveys-legal-agent-benchmark. Accessed October 2026.
[12]Mercor. Harvey LAB: leaderboard and sample task (APEX industry partner benchmarks). https://www.mercor.com/apex/partner-benchmarks/harvey-lab-leaderboard/. Accessed October 2026.
[13]Laude Institute. Harbor: a framework for evaluating and optimizing agents. https://github.com/laude-institute/harbor. Accessed October 2026.
[14]OpenAI. Securing our frontier model training environment. October 23, 2026. https://openai.com/index/securing-our-frontier-model-training-environment/.