Type I Error

The significance level is not discovered from the data; it is chosen before the test and then converted into a boundary. Once alpha is fixed, the sampling distribution of the test statistic under the null hypothesis determines the critical value or values that cut off the rejection region. If the computed statistic lands beyond that boundary, the null is rejected, and the long-run frequency with which this happens when the null is actually true equals alpha.

The allocation of that probability depends on the alternative hypothesis. A two-tailed test splits alpha evenly between both tails, so a 5% level places 2.5% in each tail and, for a large-sample z test, produces critical values of roughly plus and minus 1.96. A one-tailed test concentrates the entire alpha in the tail indicated by the alternative, lowering the hurdle in that direction to about 1.645 at the 5% level. Same alpha, different geometry, different conclusions from identical data.

Type I and Type II error probabilities move in opposite directions for a fixed sample. Shrinking alpha pushes the critical value further into the tail, which makes rejection harder and therefore raises beta, the chance of failing to detect a real effect. Power, defined as one minus beta, falls accordingly. A test tuned to almost never produce a false positive will routinely miss genuine relationships, which is why an extremely conservative alpha is a choice about which mistake you prefer, not an improvement in accuracy.

The only way to reduce both error probabilities at once is to sharpen the test itself: a larger sample, less noisy data, or a more efficient estimator. Each tightens the sampling distribution of the statistic under both hypotheses, separating them more clearly. Practitioners set alpha by consequence: a costly, irreversible action, such as replacing a manager or committing capital to a strategy, justifies a stricter level than an exploratory screen.

A single test's error rate says nothing about a research programme's error rate. When many hypotheses are examined against the same dataset, the probability that at least one true null is rejected, the familywise error rate, climbs quickly toward certainty. Corrections such as dividing alpha by the number of tests raise each individual hurdle to hold the overall false-positive rate near the intended level, at the direct cost of power.

The issue surfaces wherever repeated testing is cheap: factor discovery, technical trading rule screens, performance attribution across dozens of managers, and event studies run over many windows. It also appears in model validation, where backtesting a value-at-risk model can reject an adequate model purely by chance in a fraction of periods equal to the chosen level. Holdout samples, walk-forward testing and pre-registered hypotheses exist precisely because statistical significance obtained after extensive searching is weak evidence.

Be able to move fluently between the level, the critical value and the decision. Given a stated significance level, a test type and degrees of freedom, identify whether the rejection region is one-tailed or two-tailed, locate the boundary in a table, compare the computed statistic and state the conclusion in terms of the null rather than in terms of proof.

Equally important is classifying mistakes after the fact. If the null is true and you rejected it, the error is Type I; if the null is false and you failed to reject it, the error is Type II. Questions often frame this in economic language, such as concluding a manager adds alpha when he does not, and expect you to name the error, identify whose probability is controlled by the researcher, and explain the effect on power of tightening the level or enlarging the sample.