Moments and variance

• Definition: The kth moment of a random variable $X$ is $E[X^k] = \displaystyle{ \sum_{x} } x^k \cdot p_X(x)$.

• In practice, only the first and second moment are commonly used.

• Definition: The variance of a random variable $X$ is $Var[X] = \mathbb{E}[(X - \mathbb{E}[X])^2]$.

• The standard deviation ($\sqrt{Var[X]}$) of $X$ is the limiting object of the usual standard deviation we calculate from a dataset.

• Theorem: Let $X$ be a random variable, and $c$ some real constant. Then, the following holds:

  • Moment: $Var[X] = \mathbb{E}[X^2] - \mathbb{E}[X]^2$
  • Scale: $Var[cX] = c^2 \cdot Var[X]$
  • Shift invariant: $Var[c + X] = Var[X]$