The moment-generating function

Introduction

We can derive moments of most distributions by evaluating probability functions by integrating or summing values as necessary. However, moment generating functions present a relatively simpler approach to obtaining moments.

In the univariate case, the moment generating function, $$M_X(t)$$ , of a random variable X is given by:

$${ M }_{ X }(t)=E{ [e }^{ tx }]$$

for all values of t for which the expectation exists

Moment generating functions can be defined for both discrete and continuous random variables.

$${ M }_{ X }(t)=E{ [e }^{ tx }]=\sum _{ x }^{ }{ { e }^{ tx }P(X=x) }$$     or     $$\int _{ x }^{ }{ { e }^{ tx }{ f }_{ X }(x)dx }$$

Moment generating functions can be extended to multivariate case, where we use the same underlying concepts. Once the moment generating function is established, we can determine the mean, variance, and other moments.

If $$Y = u(X_1,X_2,\dots,X_n)$$, we say that we can find $$E(Y)$$ by evaluating $$E[u(X_1,X_2,\dots,X_n)]$$. It is also true that we can find $$E[e^{tY}]$$ by evaluating $$E[e^{tu(X_1,X_2,\dots,X_n)}]$$.

Let’s see this in a quick example:

Let $$X_1$$ and $$X_2$$ be independent random variables with uniform distributions on {1,2,3,4,5,6}. Let $$Y = X_1 + X_2$$. For example, $$Y$$ could equal the sum when two fair dice are rolled. The mgf of $$Y$$ is

$$M_Y(t) = E\big(e^{tY}\big) = E\big[e^{t(X_1+X_2)}\big] = E\big(e^{tX_1}e^{tX_2}\big).$$

The independence of $$X_1$$ and $$X_2$$ implies that

$$M_Y(t) = E\big(e^{tX_1}\big)E\big(e^{tX_2}\big)$$

In this example, $$X_1$$ and $$X_2$$ have the same pmf,

$$f(x) = \frac{1}{6}, \qquad x=1,2,3,4,5,6,$$

and thus the same mgf,

$$M_x(t) = \frac{1}{6}e^{t} + \frac{1}{6}e^{2t} + \frac{1}{6}e^{3t} + \frac{1}{6}e^{4t} + \frac{1}{6}e^{5t} + \frac{1}{6}e^{6t}.$$

It then follows that $$M_Y(t) = [M_X(t)]^2$$ equals

$$\frac{1}{36}e^{2t} + \frac{2}{36}e^{3t} + \frac{3}{36}e^{4t} + \frac{4}{36}e^{5t} +\frac{5}{36}e^{6t} + \frac{6}{36}e^{7t} + \frac{5}{36}e^{8t} + \frac{4}{36}e^{9t} + \frac{3}{36}e^{10t} \frac{2}{36}e^{11t} + \frac{1}{36}e^{12t}.$$

Note that the coefficient of $$e^{bt}$$ is equal to the probability $$P(Y = b)$$; that means the probability that the sum of the dice is equal to $$2,5,6,7$$ and so on. As observed, we can find the distribution of $$Y$$ by only determining it’s moment generating function.

This leads us to the following theorem:

If $$X_1,X_2,\cdots,X_n$$ are $$n$$ independent random variables with respective moment generating functions $${ M }_{ { X }_{ i } }(t)=E({ e }^{ t{ X }_{ i } })$$ for $$i = 1, 2, … , n,$$, then the moment-generating function of the linear combination:

$$Y=\sum _{ i=1 }^{ N }{ { a }_{ i }{ X }_{ i } }$$

is

$$M_y(t)= \prod_{i=1}^{n}M_{X_i}(a_it),$$

This theorem explains that if we know the moment generating function of each of the variables that form part of a joint distribution, we can express the moment generating function of this distribution as the product of all the moment generating functions of these variables.

Thanks to the above theorem, the following is also true:

If $$X_1,X_2,\cdots,X_n$$ are observations of a random sample from a distribution with moment-generating function $$M(t)$$, where $$-h < t < h$$, then

1. the moment-generating function of $$Y = \sum_{i=1}^{n}X_i$$ is

$$M_Y(t) = \prod_{i=1}^{n}M(t) = [M(t)]^n,$$

2. the moment-generating function of $$\bar{X} = \sum_{i=1}^{n}(1/n)X_i$$ is

$$M_{\bar{X}}(t) = \prod_{i=1}^{n}M\bigg(\frac{t}{n}\bigg) = \bigg[M\bigg(\frac{t}{n}\bigg)\bigg]^n$$

Examples

(I) Bernoulli Variables

Let $$X_1,X_2,\cdots,X_n$$ denote the outcome of $$n$$ Bernoulli trials, each with probability success $$p$$. The moment generating function of these Bernoulli trials is

$$M(t) = q + pe^t, \quad -\infty < t < \infty.$$

If

$$Y=\sum_{i=1}^{n}X_i,$$

then

$$M_Y(t)=\prod_{i=1}^{n}(q+pe^t)= (q+pe^t)^n, \quad -\infty < t < \infty.$$

Thus, we see that $$Y$$ is a Bernoulli distribution, with $$b(n,p)$$.

(II) Binomial variables

Let $$X$$ and $$Y$$ are independent binomial random variables with parameters $$(n,p)$$ and $$(m,p)$$ respectively. What is the pmf of $$X + Y$$ ?

$${ M }_{ X+Y }(t)={ M }_{ X }(t){ M }_{ Y }(t)$$

$$={ ({ P }{ e }^{ t }+1-P) }^{ n }{ ({ P }{ e }^{ t }+1-P) }^{ m }$$

$$={ ({ P }{ e }^{ t }+1-P) }^{ n+m }$$

To help you understand the example above, below is the moment generating function of a binomial variable $$X$$ with parameters $$n$$ and $$p$$:

$${ M }_{ X }(t)=E({ e }^{ tX })=\sum _{ k=0 }^{ n }{ { e }^{ tx }\left( \begin{matrix} n \\ x \end{matrix} \right) { P }^{ x }{ (1-P) }^{ n-x } }$$

$$=\sum _{ k=0 }^{ n }{ \left( \begin{matrix} n \\ x \end{matrix} \right) { (P{ e }^{ t }) }^{ x }{ (1-P) }^{ n-x } } ={ ({ P }{ e }^{ t }+1-P) }^{ n }$$

It is easy to see why this particular result makes sense. If we toss a coin 5 times in the morning and count the number of heads, the number would be distributed as $$Bin(5, \cfrac { 1 }{ 2 } )$$. If we toss the coin a
further 10 times in the evening, the number of heads will be distributed as $$Bin(10, \cfrac { 1 }{ 2 })$$.
Adding the totals together is obviously the same as the $$Bin(15, \cfrac { 1 }{ 2 } )$$ distribution that we
would expect for the whole day.

Learning Outcome

Topic 3.d: Multivariate Random Variables – Explain and apply joint moment generating functions.