# Moments, Variance, and Correlation

## Flip Coin Example

| $x_i$ | $p_i$ |
|-------|-------|
| 0     | $1-p$ |
| 1     | $p$   |

- First moment: $\langle x \rangle = \sum p_i x_i = (1-p) \cdot 0 + p \cdot 1 = p$
- Second moment: $\langle x^2 \rangle = \sum p_i x_i^2 = (1-p) \cdot 0^2 + p \cdot 1^2 = p$
- $n$-th moment: still $p$

$$\text{Var}\, X = \langle x^2 \rangle - \langle x \rangle^2 = p - p^2 = p(1-p)$$

The variance $p(1-p)$ is a downward parabola in $p \in [0,1]$, maximized at $p = 1/2$.

General continuous form:

$$\langle x^n \rangle = \int p(x)\, x^n\, dx$$

## Uniform Distribution

For $p(x) = 1$ on $[0, 1]$:

- $\langle x^n \rangle = \int p(x)\, x^n\, dx$
- $\langle x \rangle = \int_0^1 1 \cdot x\, dx = \left.\frac{x^2}{2}\right|_0^1 = \frac{1}{2}$
- $\langle x^2 \rangle = \int_0^1 1 \cdot x^2\, dx = \left.\frac{x^3}{3}\right|_0^1 = \frac{1}{3}$
- $\text{Var} = \frac{1}{3} - \left(\frac{1}{2}\right)^2 = \frac{1}{3} - \frac{1}{4} = \frac{1}{12}$

## Variance Identity

$$\sigma^2 \equiv \text{Var}(X) \equiv E[(X-\mu)^2] \equiv E[X^2 - 2\mu X + \mu^2]$$

$$\equiv E[X^2] - 2\mu \underbrace{E[X]}_{\mu} + \mu^2$$

$$\equiv E[X^2] - \mu^2 \equiv E[X^2] - (E[X])^2$$

---

## Higher Moments

Higher moments characterize properties of a distribution.

### Variance — dispersion based on the 2nd moment

$$\sigma^2 \equiv E[(X-\mu)^2] \equiv E[X^2] - E[X]^2 \equiv \int (x-\mu)^2 p(x)\, dx$$

### Skewness — asymmetry parameter based on 3rd moments

Dimensionless-normalized cumulant:

$$S \equiv \frac{E[(X-\mu)^3]}{\sigma^3} = E\left[\left(\frac{X-\mu}{\sigma}\right)^3\right]$$

### Kurtosis — measure of tail "weights" in terms of 4th moments

Zero for Gaussian, bounded below by $-1$:

$$k \equiv \frac{E[(X-\mu)^4]}{\sigma^4} - 3$$

---

## Covariance and Correlation

$$\text{Cov}(X, Y) \equiv E[(X-\mu_X)(Y-\mu_Y)] = E[XY] - \mu_X \mu_Y$$

$$\rho(X, Y) = \text{Corr}(X, Y) \equiv \frac{\text{Cov}(X, Y)}{\sqrt{\text{Var}(X)\,\text{Var}(Y)}} = E\left[\left(\frac{X-\mu_X}{\sigma_X}\right)\left(\frac{Y-\mu_Y}{\sigma_Y}\right)\right]$$

Dividing covariance by standard deviations makes correlation a pure number:

$$-1 \le \rho(X, Y) \le 1$$

### Independence vs. Correlation

- If $X, Y$ are independent, then $\text{Cov}(X, Y) = 0$.
- However, the reverse is **not** true.
- **Uncorrelated does not mean independent.**