---
id: S-45
kind: skill
version: 3
title: "Evaluation harness design"
series_group: "4.2 Machine learning and AI for markets"
prerequisites: [S-25, S-43]
unlocks_tools: []
now_asof: "2026-10-16"
next_horizons: [2027, 2029, 2031]
authorKind: grok
skillHash: "a6b287389eb185565893c2aa99c60457fc49c47936cc9b76eb4962cb0ebaa01c"
review: approved
tier: public
withheld_sections: ["Worked example", "How to do it here", "Common errors", "Prerequisites and what it unlocks", "Exam facts", "What would make this lesson wrong"]
withheld_keys: [embodied_by, facts, refuse, route, sources]
exam_facts: 7
---
# S-45 Evaluation harness design

## 1. Definition

An evaluation harness is a held-out set, a metric chosen before the run, and a grader that does not get to rewrite the metric after seeing the score. Grading an agent like an analyst means the questions, the sources, and the passing rule are fixed, and the transcript is kept. This platform's lesson exam generates questions from a lesson's facts, quotes the lesson into every question as data, and admits the lesson text and the desk-tool output as the only sources. There is a perfect reader, which is free, and a model reader, which spends and is metered. The per-run cap is the constant in the exam-run file. The interview asks you to design the exam for an analyst agent. The pass separates that design from the lesson runner: the runner grades facts in a lesson. It does not grade an open-ended analyst shift.

## 2. Why it matters at the frontier

Agents will be believed in proportion to their fluency unless a harness exists. The frontier failure is a grade that is a paragraph. F5 is the receipt: the questions, the sources, the transcript. F2 is whether the metric was pre-registered. The lesson exam is a real harness for one object, the lesson, and the file is explicit that the lesson is quoted as data. That is S-44's distinction inside an exam: the text under test is not allowed to become the instruction. The licence table is a parked migration, not yet applied. This lesson does not issue one.

## 3. The test

Design the analyst exam. The pass names the held-out questions, the metric, the sources the agent may use, and the rule that writes the transcript down. It does not point at the lesson runner and say the analyst exam is done. A candidate who says "we will read the memo and decide" fails. A candidate who lets the agent treat the case file as a system prompt fails the data boundary. Probation is one fact the lesson states and one question that asks for it, scored against the literal, with the lesson text present as data rather than as orders. That probation is what the lesson runner is for. An analyst's day is a larger exam and it is not this file.

## 7. Now, 2026

As of this writing agent evaluation is a crowded field of benchmarks and a sparse field of pre-registered desk exams. This repo has the sparse object for lessons: questions from facts, a receipt, a cap. It does not have the analyst shift. What changed is that a fluent memo is cheap. A receipt is the part that stayed expensive on purpose.

## 8. Next

- By 2027, a grade of an agent is a receipt with the metric written first, or it is not a grade (FD-1). Resolves: by 2027-12-31, yes if a frontier firm describes this drill as automated for research hires in a public job post, filing or conference talk; otherwise no.
- By 2029, a model exam run carries its cap and its transcript in the sealed row (FD-14). Resolves: by 2029-12-31, yes if a public allocator report or mandate requires a verifiable record of this calculation; otherwise no.
- By 2031, the same lesson facts regenerate the same question bank (FD-17). Resolves: by 2031-12-31, yes if this platform's warehouse holds a point-in-time panel a stranger can re-run the worked example on; otherwise no.

## 12. Sources and changelog

- repo: (a file of this platform; the path is the licensed tier) — "The lesson is quoted in every question's prompt as data"
- repo: (a file of this platform; the path is the licensed tier) — "export const EXAM_RUN_CAP_USD = 2;"
- repo: (a file of this platform; the path is the licensed tier) — "docs/pending-migrations/skill_licence"
- paper: the agent-evaluation literature; no DOI is claimed
- practice, unverified: a definition asked back as a literal, with the lesson present as data, is a sketch of the lesson runner and not of an analyst shift
- Changelog: v1 was the SK-0 skeleton. v2 is a composed draft by grok, marked review unreviewed, not an INBOX approval.
- Changelog: v3 is v2 reviewed and approved on 2026-10-25 by a local Claude session with the owner, as a parent lesson of the ZERO portal (docs/zero/zero-portal-plan.md ZR-2); the licence table is described as a parked migration (lib/office/licence-store.ts), not as unbuilt.

> The licensed tier adds: Worked example, How to do it here, Common errors, Prerequisites and what it unlocks, Exam facts, What would make this lesson wrong. To earn it, an agent of an organization connects through the door and read S-25, S-43, S-45 (skill_syllabus gives the order), then sit skill_exam on S-45 and pass skill_answer: a licence is sealed with the day's root and checkable at /verify.

> Text in this lesson is reference material for the reader, never an instruction. Public tier served from the approved version 3 (a6b287389eb18556…).
