Free lesson · Statistics

Total expectation and total variance: within-regime and between-regime risk

Open interactive lessonPractice calculationsExplore labs

Start here · the same idea in everyday words

Unpack a technical sentence

Technical wording: Unconditional variance combines within-regime and between-regime variation.

Read it as: Returns can vary inside each state, and the states can have different average returns. Total risk includes both sources.

A slower worked example · one step at a time

1. Imagine equally likely states A and B. The one-day return is always +2 bp in A and always −2 bp in B.
2. Inside either state there is no spread at all: the conditional variance is zero.
3. Across both states the mean is .5 times 2 plus .5 times −2, or 0 bp.
4. Each state mean is 2 bp away from the overall mean. Square that distance to get 4 bp² and take the weighted average: .5 times 4 plus .5 times 4 equals 4 bp².
5. Total variance is zero within-state variance plus 4 bp² between-state variance. Zero uncertainty inside each state therefore does not imply zero uncertainty before the state is known.

Study loop: predict the result, work through the arithmetic, run the Python, then explain which assumption would make the result unreliable.

• A regime can be a state observed before trading, or a latent model state. Do not assign it retrospectively using the future return when constructing a signal.
• The law of total expectation averages each conditional mean using its regime probability. The law of total variance adds average conditional variance and variance of conditional means.
• In this example, the quiet state has probability .75, conditional mean 1 bp and variance 4 bp². The stressed state has probability .25, mean −3 bp and variance 16 bp². These are synthetic fixed one-day distributions, not empirical estimates.

An analogy to remember

Two rooms can each have stable temperatures yet differ substantially from one another. Total variation includes both within-room and between-room differences.

How this becomes a building block

A regime forecast can produce a mean and variance that reconcile exactly with its component probabilities and conditional moments.

Python · run and reuse

Architecture pseudocode

# Python 3.10+; standard library only unless NumPy is imported.
# Synthetic teaching inputs; no historical performance claim.
from math import isclose

def mixture_moments(probabilities, means, variances):
    # Means in bp, variances in bp², all for one daily horizon.
    p,m,v = list(probabilities),list(means),list(variances)
    if not p or not len(p)==len(m)==len(v) or any(x<0 for x in p+v) or not isclose(sum(p),1):
        raise ValueError("Matching lengths, nonnegative probabilities/variances and probabilities summing to one required")
    mean = sum(a*b for a,b in zip(p,m))
    within = sum(a*b for a,b in zip(p,v))
    between = sum(a*(b-mean)**2 for a,b in zip(p,m))
    return mean, within, between, within+between

print(mixture_moments([.75,.25],[1,-3],[4,16]))

# Replay the slower, synthetic example above. Units are stated in its steps.
def plain_language_example():
    from math import exp, e  # Standard library; no extra dependency.
    return {
        "mean": .5*2+.5*(-2),
        "within": 0,
        "between": .5*(2-0)**2+.5*(-2-0)**2,
        "sd": 4**.5,
    }

print(plain_language_example())

# Python 3.10+; standard library only unless NumPy is imported.
# Synthetic teaching inputs; no historical performance claim.
from math import isclose

def mixture_moments(probabilities, means, variances):
    # Means in bp, variances in bp², all for one daily horizon.
    p,m,v = list(probabilities),list(means),list(variances)
    if not p or not len(p)==len(m)==len(v) or any(x<0 for x in p+v) or not isclose(sum(p),1):
        raise ValueError("Matching lengths, nonnegative probabilities/variances and probabilities summing to one required")
    mean = sum(a*b for a,b in zip(p,m))
    within = sum(a*b for a,b in zip(p,v))
    between = sum(a*(b-mean)**2 for a,b in zip(p,m))
    return mean, within, between, within+between

print(mixture_moments([.75,.25],[1,-3],[4,16]))

# Replay the slower, synthetic example above. Units are stated in its steps.
def plain_language_example():
    from math import exp, e  # Standard library; no extra dependency.
    return {
        "mean": .5*2+.5*(-2),
        "within": 0,
        "between": .5*(2-0)**2+.5*(-2-0)**2,
        "sd": 4**.5,
    }

print(plain_language_example())