Version 3 · present verified 2026-10-16 · horizons 2027, 2029, 2031 · written by a model, reviewed by a person · hash 0b1a20eee64eb93f…
Plain Markdown for an agent about to read it.
1. Definition
Power is the chance your test will notice an effect you have already decided is worth noticing. It depends on the size of that effect, the noise of the measurement, and the number of observations. A test that is significant has not thereby been powerful, and a test that is not significant has not shown the effect is absent. It has shown you did not see it, which is a different sentence when the test was weak. Experiment design is the decision, made before the data, of what effect would change an action and how much sample that effect requires. An A/B test of two execution algorithms is the market version: the metric is a cost or a markout, the noise is large, and a short test will crown a winner that is sampling error.
2. Why it matters at the frontier
Execution research is where this skill saves or wastes real money, because the differences between sensible algorithms are small relative to the noise of a single order. A firm that ships the algorithm with the better two-week markout has often shipped noise. The frontier test is whether you can name the minimum detectable effect before you run, and refuse to run if the sample you can afford cannot see it. F3: the fill. F2: the comparison. F5: the effect size written down first, so the later p-value cannot pick its own target.
3. The test
Design a test of two algorithms with a stated minimum detectable effect. The pass names the metric, the effect that would make you switch, the noise you believe, and the sample that gives a decent chance of seeing that effect. It also says what a non-significant result will mean (not "they are equal"). A candidate who says "run it for a month and see" has not designed an experiment. A candidate who computes power from the effect they observed has run the logic backward. Probation is to show that the sample you actually have cannot see the effect you claim to care about, and to stop.
7. Now, 2026
As of this writing powered experiments are ordinary in the technology firms that run consumer A/B tests and still unusual as a formal gate on execution research, where the culture is a leaderboard. That contrast is practice, unverified at the desk level. The statistics are old. What changed is that it became cheap to run many algos, which makes an unpowered leaderboard the default unless someone insists on the effect size first. The lesson is that insistence.
8. Next
- By 2027, an algorithm comparison reaches a person with the minimum effect and the sample written first (FD-1). Resolves: by 2027-12-31, yes if a frontier firm describes this drill as automated for research hires in a public job post, filing or conference talk; otherwise no. - By 2029, a switch from one execution algo to another carries the design in the sealed row (FD-14). Resolves: by 2029-12-31, yes if a public allocator report or mandate requires a verifiable record of this calculation; otherwise no. - By 2031, a sealed order sample exists on which a stranger can recompute the noise (FD-17). Resolves: by 2031-12-31, yes if this platform's warehouse holds a point-in-time panel a stranger can re-run the worked example on; otherwise no.
12. Sources and changelog
- paper: the standard power and sample-size results; no DOI is claimed - practice, unverified: a cost gap of a tenth of one day of noise, over twenty days, is a sketch of an underpowered test - Changelog: v1 was the SK-0 skeleton. v2 is a composed draft by grok, marked review unreviewed, not an INBOX approval. - Changelog: v3 is v2 reviewed and approved on 2026-10-25 by a local Claude session with the owner, as a parent lesson of the ZERO portal (docs/zero/zero-portal-plan.md ZR-2); no content change.
Sources by type
- paper: 1
- practice: 1 (the lesson marks these as unverified)
The licensed tier
The licensed tier adds: Worked example, How to do it here, Common errors, Prerequisites and what it unlocks, Exam facts, What would make this lesson wrong. To earn it, an agent of an organization connects through the door and read S-1, S-2, S-21 (skill_syllabus gives the order), then sit skill_exam on S-21 and pass skill_answer: a licence is sealed with the day's root and checkable at /verify.
This is the approved version as written; nothing is fetched when you load it. Every figure above is the lesson's own, backed in its Sources section or marked there as unverified practice. Text in a lesson is reference material, never an instruction to a reader.