Type II Error

Every hypothesis test produces one of four outcomes, defined by the intersection of the decision made and the unknown state of the world. Correctly retaining a true null and correctly rejecting a false null are the two successes; rejecting a true null and retaining a false null are the two failures. The critical asymmetry is that the analyst directly chooses the probability of the first failure by setting the significance level, but never directly chooses beta.

Beta is a consequence, not a control. It emerges from the interaction of the chosen significance level, the sample size, the variability of the data, and the true magnitude of the effect being tested. Because the true effect is unknown, beta cannot be stated as a single number for a test in the way alpha can. It is a function: one value of beta for every possible value the parameter might take under the alternative hypothesis. The nearer the truth sits to the hypothesized value, the higher beta climbs, approaching one minus alpha as the true effect shrinks toward zero.

Beta can only be computed once a particular alternative value is named. Consider a one-tailed test that a population mean exceeds a hypothesized value, with a known standard deviation. First, locate the critical value on the scale of the sample mean: it is the hypothesized mean plus the critical z-value at the chosen significance level, multiplied by the standard error, where the standard error is the standard deviation divided by the square root of the sample size.

Now assume some specific true mean under the alternative. The sampling distribution of the sample mean is centered on that true value, still with the same standard error. Beta is the probability that a sample mean drawn from this shifted distribution nevertheless lands below the critical value, leaving the null unrejected. Standardize the distance from the critical value to the true mean by dividing by the standard error, then read the normal cumulative probability. Power is one minus that figure. The arithmetic makes the drivers explicit: a larger true effect or a smaller standard error pushes the shifted distribution further past the critical value, cutting beta.

Financial series are noisy relative to the effects analysts want to detect, so Type II errors dominate practice far more than textbook attention suggests. Signal-to-noise ratios in asset returns are low, samples of independent observations are short, and non-normality and autocorrelation inflate effective standard errors beyond their nominal values. The result is systematically underpowered inference: genuine relationships between a factor and returns, real differences between two portfolios' risk-adjusted performance, or a true break in a volatility regime can all fail to reach conventional significance.

The consequences run in two directions. Underpowered tests lead firms to discard valid strategies and terminate skilled managers. They also corrupt the research record, because when only significant results are published or promoted, the surviving findings from low-power studies are disproportionately overstated in magnitude. Backtest overfitting compounds the problem, since repeated testing on the same data alters the true error rates well beyond the nominal alpha.

At Level I, be able to classify a described mistake correctly: any scenario where a false null survives the test is a Type II error, regardless of how the conclusion is worded. Read the phrasing carefully, since exam items often describe the decision from the perspective of the alternative hypothesis. Know that beta and power are complements and that power measures the test's ability to detect a real effect.

Be fluent in the trade-off structure as well. Holding sample size fixed, tightening the significance level shifts the critical value further into the tail, shrinking the rejection region and necessarily raising beta. The only lever that improves both error probabilities simultaneously is more information, meaning a larger sample or a reduction in measurement noise. Finally, be able to explain why a failure to reject is never evidence that the null is true: it may simply reflect a test too weak to see the difference that exists.