Trading Dev AcademyFree quant education

Free lesson · Statistics

Multiple testing and the union bound: account for the whole search

Open interactive lessonPractice calculationsExplore labs

Start with the idea

  • Decide which tests belong to the search. Divide the total false-rejection budget by the number of tests, then compare each valid p-value with that smaller cutoff.
Symbols, units & horizon
  • A_j: false rejection event for hypothesis j
  • m: number of prespecified tests
  • j: hypothesis index
  • P: probability
  • p_j: valid null p-value for test j
  • α: desired family-wise error probability
  • all tests concern the declared evaluation horizon

When and why to use this

Use family-wise controls for a declared collection of research tests. Keep a separate untouched chronological evaluation and account for costs and capacity after statistical validation.

Start here · the same idea in everyday words

Technical termEveryday meaningConcrete example
Null hypothesisThe reference claim tested by a statistical procedure.A common reference is that a strategy’s population mean net return is zero.
P-valueUnder the null and test assumptions, the probability of a test statistic at least as extreme as the observed one.A p-value of .01 is not a 1% probability that the null is true.
Family-wise errorThe probability of at least one false rejection across a declared set of tests.Twenty strategy tests create one family only if that is the set you actually declared and tested.

Unpack a technical sentence

Technical wording: Bonferroni controls family-wise error for a fixed family of valid tests.

Read it as: Decide which tests belong to the search. Divide the total false-rejection budget by the number of tests, then compare each valid p-value with that smaller cutoff.

A slower worked example · one step at a time

  1. Before seeing results, declare 20 tests and a total false-rejection budget of .05, or 5%.
  2. Divide .05 by 20. Each test gets a cutoff of .0025, or .25%.
  3. A candidate p-value .01 is below .05, but above .0025. It would pass the unadjusted rule and fail this family-wise rule.
  4. The union bound adds the 20 per-test error budgets: 20 times .0025=.05. It does not require the tests to be independent.
  5. The p-values themselves still need valid tests. Leakage, hidden trials or an unsuitable stopping rule can invalidate them. Recording just the best result does not make the other trials disappear.
Self-assessed: If you tried 20 variations but report only one, is the search a one-test search?

No. Selection used the other results. Record the whole search and choose an evaluation procedure appropriate to it.

Study loop: predict the result, work through the arithmetic, run the Python, then explain which assumption would make the result unreliable.

  • The union bound says that the probability of at least one event is no greater than the sum of event probabilities. It does not require independent events.
  • Bonferroni applies this to valid null p-values: for a fixed family of m hypotheses, reject each only when its p-value is at most α/m. This controls the probability of one or more false rejections at α.
  • A p-value measures tail evidence under a null model; it is not the probability that the null is true or that a strategy will profit. Log every tried specification, including unsuccessful variants. A data-dependent family or adaptive stopping requires an appropriate design rather than silently applying a fixed-family calculation.
Before reading the working, evaluate 0.05/20. Give the result in p-value threshold.

The Bonferroni threshold is .0025 for 20 tests at a 5% family-wise error budget.

P(⋃j=1mAj)≤∑j=1mP(Aj),pj≤αm
Probability inequality and Bonferroni family-wise error control

Multiple testing and the union bound: account for the whole search

  1. Prespecify m=20 hypotheses and family-wise error budget α=.05.
  2. Divide .05 by 20 to obtain per-test threshold .0025.
  3. A candidate p-value .01 passes an unadjusted .05 threshold but fails .0025. The correction controls false rejection under valid tests; it does not establish implementable alpha.
Work it by hand

The Bonferroni threshold is .0025 for 20 tests at a 5% family-wise error budget.

An analogy to remember

Buying many lottery tickets creates many chances for an unusual result. Research searches likewise multiply opportunities for chance findings.

How this becomes a building block

A reproducible research registry records the full candidate family, evaluation rule and unsuccessful trials before selecting a winner.

Python · run and reuse

Architecture pseudocode

# Python 3.10+; standard library only unless NumPy is imported.
# Synthetic teaching inputs; no historical performance claim.
def bonferroni_threshold(alpha, tests):
    if not 0 < alpha < 1 or not isinstance(tests,int) or tests < 1:
        raise ValueError("Probability alpha and positive integer test count required")
    return alpha/tests

print(bonferroni_threshold(.05,20))

# Replay the slower, synthetic example above. Units are stated in its steps.
def plain_language_example():
    from math import exp, e  # Standard library; no extra dependency.
    return {
        "cutoff": .05/20,
        "budget": 20*(.05/20),
    }

print(plain_language_example())
Self-assessed: explain the calculation, its output units, and one condition under which it fails.

Use family-wise controls for a declared collection of research tests. Keep a separate untouched chronological evaluation and account for costs and capacity after statistical validation. Invalid p-values from dependence, leakage or optional stopping are not repaired by dividing alpha by the number of tests. Omitting tried variants understates the search.

Python implementation

Self-contained teaching example. Python 3.10+; dependencies and input conventions are shown in the code and notation. Run in your own Python environment.

# Python 3.10+; standard library only unless NumPy is imported.
# Synthetic teaching inputs; no historical performance claim.
def bonferroni_threshold(alpha, tests):
    if not 0 < alpha < 1 or not isinstance(tests,int) or tests < 1:
        raise ValueError("Probability alpha and positive integer test count required")
    return alpha/tests

print(bonferroni_threshold(.05,20))

# Replay the slower, synthetic example above. Units are stated in its steps.
def plain_language_example():
    from math import exp, e  # Standard library; no extra dependency.
    return {
        "cutoff": .05/20,
        "budget": 20*(.05/20),
    }

print(plain_language_example())

Continue learning

Statistics & Probability — all lessons
  1. Start with counts, probabilities and averages
  2. Random outcomes, sample averages and the limits of the bell curve
  3. Conditional probability: update a belief with evidence
  4. Prediction markets: probability, price and net expected value
  5. Expected value: measure the payoff before choosing the risk
  6. Correlation: how many strategies do you really have?
  7. Conditional probability and Bayes: where the edge actually lives
  8. The central limit theorem: sampling means under explicit assumptions
  9. Law of large numbers: what averaging can and cannot establish
  10. Central limit theorem, standard errors and confidence intervals
  11. Total expectation and total variance: within-regime and between-regime risk
  12. Markov, Chebyshev and Jensen: bounds without a full distribution
  13. Gauss–Markov theorem: what a regression hedge ratio guarantees
  14. Delta method: propagate estimator uncertainty through a smooth function
  15. Multiple testing and the union bound: account for the whole search
  16. Estimation uncertainty and Bayesian updating

Quantitative finance and development glossary · Python resources and libraries · Research sources and limitations