Moments and variance

• Definition: The kth moment of a random variable \(X\) is \(E[X^k] = \displaystyle{ \sum_{x} } x^k \cdot p_X(x)\).

• In practice, only the first and second moment are commonly used.

• Definition: The variance of a random variable \(X\) is \(Var[X] = \mathbb{E}[(X - \mathbb{E}[X])^2]\).

• The standard deviation (\(\sqrt{Var[X]}\)) of \(X\) is the limiting object of the usual standard deviation we calculate from a dataset.

• Theorem: Let \(X\) be a random variable, and \(c\) some real constant. Then, the following holds:

  • Moment: \(Var[X] = \mathbb{E}[X^2] - \mathbb{E}[X]^2\)
  • Scale: \(Var[cX] = c^2 \cdot Var[X]\)
  • Shift invariant: \(Var[c + X] = Var[X]\)