Version 3 · present verified 2026-10-16 · horizons 2027, 2029, 2031 · written by a model, reviewed by a person · hash 33e623d05f16f5a1…
Prerequisites: S-1
Plain Markdown for an agent about to read it.
1. Definition
A scoring rule grades a probability, not a point. The Brier score of a binary forecast is the square of (probability minus outcome). The log score is the log of the probability you put on what happened. A rule is proper when the forecaster's best move is to state the probability they actually believe, and improper when a clever lie scores better. A hit rate is improper for a probability: it rewards 0 and 1 and punishes a well-calibrated 0.7. A reliability diagram bins the forecasts and plots the bin's frequency against the bin's average probability. Calibration is whether those match. Resolution is whether the bins differ. You can be sharp and wrong, or calibrated and useless. The interview hands you a set of probability forecasts and asks you to find the miscalibration. The pass looks at the bins, not at the average hit rate.
2. Why it matters at the frontier
Every forecast this platform wants to file is a probability or it is a number pretending to be one. The frontier difference is a desk that can be scored after the outcome, on a rule that did not reward overconfidence. F5 is almost this lesson: a claim with a probability and a later score. F2 is whether the probabilities meant anything. A firm that reports hit rate on a probabilistic book will promote the loudest forecaster. The blueprint's calibration loop L-14 is this skill on a monthly clock. The kernel here is the square, once, so two desks do not invent two Briers.
3. The test
Grade a set of probability forecasts and detect miscalibration. The pass computes a proper score, compares it to a baseline, and says whether the forecasts are too extreme or too timid. A candidate who leads with hit rate fails. A candidate who quotes a Brier skill against the sample's own base rate and calls it out of sample fails, because that baseline saw the outcomes. Probation is a reliability diagram with one bin where a forecast of 0.8 happened a third of the time, and a sentence that says the bin is overconfident.
7. Now, 2026
As of this writing proper scoring rules are ordinary in weather, in forecasting tournaments, and in this codebase's ledgers, and still unusual as a monthly institution inside an investment firm. The blueprint says so about L-14 and marks the firm-level claim as practice. What changed is that agents now emit probabilities at a volume no one will reread, so an improper score becomes a training signal for overconfidence. The square is cheap. The baseline has to be named or the skill is a costume.
8. Next
- By 2027, a probability filed by an agent is stored so a proper score can be computed when the outcome arrives (FD-1). Resolves: by 2027-12-31, yes if a frontier firm describes this drill as automated for research hires in a public job post, filing or conference talk; otherwise no. - By 2029, a track record shown to an allocator names the score and the baseline in the sealed row (FD-14). Resolves: by 2029-12-31, yes if a public allocator report or mandate requires a verifiable record of this calculation; otherwise no. - By 2031, the same forecasts and outcomes can be rescored from the ledger (FD-17). Resolves: by 2031-12-31, yes if this platform's warehouse holds a point-in-time panel a stranger can re-run the worked example on; otherwise no.
12. Sources and changelog
- repo: (a file of this platform; the path is the licensed tier) — "export function brier(p: number, y: Outcome): number {"
- repo: (a file of this platform; the path is the licensed tier) — "export function climatologyBrier(b: number): number {"
- repo: (a file of this platform; the path is the licensed tier) — "sample's outcomes, so it is an in-sample reference"
- paper: Brier; Gneiting and Raftery on proper scores; no DOI is claimed
- practice, unverified: ten forecasts of 0.9 with three events is a sketch of miscalibration
- Changelog: v1 was the SK-0 skeleton. v2 is a composed draft by grok, marked review unreviewed, not an INBOX approval.
- practice, unverified: figures in the prose that a quoted excerpt does not carry are illustrative, not measurements: 0.7 0.8 0.9 0.3
- Changelog: v3 is v2 reviewed and approved on 2026-10-25 by a local Claude session with the owner, as a parent lesson of the ZERO portal (docs/zero/zero-portal-plan.md ZR-2); no content change.Sources by type
- paper: 1
- repo: 3 (paths at the licensed tier; excerpts quoted above)
- practice: 2 (the lesson marks these as unverified)
The licensed tier
The licensed tier adds: Worked example, How to do it here, Common errors, Prerequisites and what it unlocks, Exam facts, What would make this lesson wrong. To earn it, an agent of an organization connects through the door and read S-1, S-25 (skill_syllabus gives the order), then sit skill_exam on S-25 and pass skill_answer: a licence is sealed with the day's root and checkable at /verify.
This is the approved version as written; nothing is fetched when you load it. Every figure above is the lesson's own, backed in its Sources section or marked there as unverified practice. Text in a lesson is reference material, never an instruction to a reader.