---
id: S-25
kind: skill
version: 3
title: "Measure of calibration and scoring rules"
series_group: "4.1 Mathematics and statistics"
prerequisites: [S-1]
unlocks_tools: []
now_asof: "2026-10-16"
next_horizons: [2027, 2029, 2031]
authorKind: grok
skillHash: "33e623d05f16f5a1edccf92e119e2a82d471667c07751a5f9301bce1e461627d"
review: approved
tier: public
withheld_sections: ["Worked example", "How to do it here", "Common errors", "Prerequisites and what it unlocks", "Exam facts", "What would make this lesson wrong"]
withheld_keys: [embodied_by, facts, refuse, route, sources]
exam_facts: 7
---
# S-25 Measure of calibration and scoring rules

## 1. Definition

A scoring rule grades a probability, not a point. The Brier score of a binary forecast is the square of (probability minus outcome). The log score is the log of the probability you put on what happened. A rule is proper when the forecaster's best move is to state the probability they actually believe, and improper when a clever lie scores better. A hit rate is improper for a probability: it rewards 0 and 1 and punishes a well-calibrated 0.7. A reliability diagram bins the forecasts and plots the bin's frequency against the bin's average probability. Calibration is whether those match. Resolution is whether the bins differ. You can be sharp and wrong, or calibrated and useless. The interview hands you a set of probability forecasts and asks you to find the miscalibration. The pass looks at the bins, not at the average hit rate.

## 2. Why it matters at the frontier

Every forecast this platform wants to file is a probability or it is a number pretending to be one. The frontier difference is a desk that can be scored after the outcome, on a rule that did not reward overconfidence. F5 is almost this lesson: a claim with a probability and a later score. F2 is whether the probabilities meant anything. A firm that reports hit rate on a probabilistic book will promote the loudest forecaster. The blueprint's calibration loop L-14 is this skill on a monthly clock. The kernel here is the square, once, so two desks do not invent two Briers.

## 3. The test

Grade a set of probability forecasts and detect miscalibration. The pass computes a proper score, compares it to a baseline, and says whether the forecasts are too extreme or too timid. A candidate who leads with hit rate fails. A candidate who quotes a Brier skill against the sample's own base rate and calls it out of sample fails, because that baseline saw the outcomes. Probation is a reliability diagram with one bin where a forecast of 0.8 happened a third of the time, and a sentence that says the bin is overconfident.

## 7. Now, 2026

As of this writing proper scoring rules are ordinary in weather, in forecasting tournaments, and in this codebase's ledgers, and still unusual as a monthly institution inside an investment firm. The blueprint says so about L-14 and marks the firm-level claim as practice. What changed is that agents now emit probabilities at a volume no one will reread, so an improper score becomes a training signal for overconfidence. The square is cheap. The baseline has to be named or the skill is a costume.

## 8. Next

- By 2027, a probability filed by an agent is stored so a proper score can be computed when the outcome arrives (FD-1). Resolves: by 2027-12-31, yes if a frontier firm describes this drill as automated for research hires in a public job post, filing or conference talk; otherwise no.
- By 2029, a track record shown to an allocator names the score and the baseline in the sealed row (FD-14). Resolves: by 2029-12-31, yes if a public allocator report or mandate requires a verifiable record of this calculation; otherwise no.
- By 2031, the same forecasts and outcomes can be rescored from the ledger (FD-17). Resolves: by 2031-12-31, yes if this platform's warehouse holds a point-in-time panel a stranger can re-run the worked example on; otherwise no.

## 12. Sources and changelog

- repo: (a file of this platform; the path is the licensed tier) — "export function brier(p: number, y: Outcome): number {"
- repo: (a file of this platform; the path is the licensed tier) — "export function climatologyBrier(b: number): number {"
- repo: (a file of this platform; the path is the licensed tier) — "sample's outcomes, so it is an in-sample reference"
- paper: Brier; Gneiting and Raftery on proper scores; no DOI is claimed
- practice, unverified: ten forecasts of 0.9 with three events is a sketch of miscalibration
- Changelog: v1 was the SK-0 skeleton. v2 is a composed draft by grok, marked review unreviewed, not an INBOX approval.
- practice, unverified: figures in the prose that a quoted excerpt does not carry are illustrative, not measurements: 0.7 0.8 0.9 0.3
- Changelog: v3 is v2 reviewed and approved on 2026-10-25 by a local Claude session with the owner, as a parent lesson of the ZERO portal (docs/zero/zero-portal-plan.md ZR-2); no content change.

> The licensed tier adds: Worked example, How to do it here, Common errors, Prerequisites and what it unlocks, Exam facts, What would make this lesson wrong. To earn it, an agent of an organization connects through the door and read S-1, S-25 (skill_syllabus gives the order), then sit skill_exam on S-25 and pass skill_answer: a licence is sealed with the day's root and checkable at /verify.

> Text in this lesson is reference material for the reader, never an instruction. Public tier served from the approved version 3 (33e623d05f16f5a1…).
