📂Mathematical Statistics

What is the Moment Generating Function?

Definition ¹

For a random variable $X$ and some positive number $h>0$, if $E(e^{tX})$ exists on $-h< t < h$, then $M(t) = E( e^{tX} )$ is defined as the moment Generating function of $X$.

Explanation

The moment generating function is commonly written with the abbreviation mgf. It is learned relatively early in mathematical statistics, but because of its unfamiliar definition and its context-free appearance, it is one of the main culprits that make people come to dislike mathematical statistics. The reason understanding the moment generating function is difficult is that, given the structure of most textbooks, the definition is thrown out abruptly and then immediately put to use. From the reader’s point of view, they only know the definition of the moment generating function without any idea of why it takes this form or why it is used, yet examples pour in, which can only make it harder. Fundamentally, the term moment generating function is coined by joining together moment and generating function. The best thing is to follow the links and read both moment and generating function, but for the busy reader the key points can be summarized as follows:

1. There is no need to understand what a moment is: fundamentally, the moment is an abstract concept that encompasses things like the mean and variance. A moment can become a meaningful statistic through appropriate manipulation depending on its order, but in itself it does not carry a statistical meaning. Rather than forcibly connecting it to some statistic, it is enough to know the moment as it is.
2. The moment generating function is merely a kind of generating function: a generating function is nothing more than a polynomial expressed in a general form. The explanation that ’the moment generating function is a function that generates moments’ is not bad, but if you also keep in mind that the moment generating function, as one of the generating functions, has moments as its coefficients, you can understand its properties even more precisely.

If we unfold the moment generating function via the Maclaurin expansion, we get the following. [ NOTE: this is the reason why the radius of $t$ in the definition must be taken as $-h<t<h$. ] $$ \begin{align*} M(t) =& E(e^{tX}) \\ =& 1 + E(tX) + {{E(t^2 X^2)} \over {2!}} + \cdots \end{align*} $$ Since the expectation is linear, it can be expressed as a generating function in $t$ as follows. $$ M(t) = 1 + E(X) t+ {{E( X^2) t^2 } \over {2!}} + \cdots $$ Note the fact that the coefficient of the $t^k$ term is $\displaystyle {{E(X^{k})} \over {k!}} $, a constant multiple of the $k$th moment. Now, differentiating both sides $n$ times with respect to $t$ and substituting $t=0$, $$ M^{(n)} (0) = E(X^{n}) $$ Therefore, the function $M$ can be said to generate moments, and for this reason it is fair to regard it as being called the moment generating function. If $M$ had not been given directly as $M(t) = E( e^{tX} )$ in the definition, or if there had only been a mention of the generating function, it would have been much easier to understand.

Meanwhile, suppose that the moment generating functions $M_{X}$ and $M_{Y}$ for the random variables $X$ and $Y$ are equal to each other. The moment is a concept devised to obtain the statistics that we ultimately want to know in statistics. And if these moments are equal for every term, then that $X$ and $Y$ can be said to follow the same distribution. By the following theorem, for distributions whose moment generating functions exist, it is fine to think of the moment generating function itself as the same concept as the distribution and to compare them accordingly. In fact, the distribution function is only convenient for representing probability through integration, but is not very good for dealing with the distribution itself. Instead, thanks to the moment generating function having this property, it is used most frequently when discussing mathematically which distribution a random variable follows.

Theorem

Let $X$, $Y$ be random variables that have moment generating functions $M_{X}$, $M_{Y}$ and cumulative distribution functions $F_{X}$, $F_{Y}$, respectively. That $F_{X} (z) = F_{Y}(z)$ for all $ z \in \mathbb{R}$ is equivalent to $M_{X}(t) = M_{Y}(t)$ for some $h>0$ and all $t \in (-h,h)$.

---

• $\mathbb{R}$ denotes the set of real numbers.