Calculate moments for joint, conditional, and marginal random variables

Moments of a Probability Mass function

The n-th moment about the origin of a random variable is the expected value of its n-th power. Moments about the origin are \(E(X),E({ X }^{ 2 }),E({ X }^{ 3 }),E({ X }^{ 4 }),….\quad\)

For the most part, however, we are going to be looking at moments about the mean, also called central moments. The n-th central moment of a random variable \(X\) is the expected value of the n-th power of the deviation of \(X\) from its expected value. 

  • First central moment: Mean
  • Second central moment: Variance
  • Third central moment: Skewness
  • Fourth central moment: Kurtosis

Moments about the mean describe the shape of the probability function of a random variable.

Properties of Expectation

Recall that the expected value of a random variable \(X\) is defined by

$$ E[X] = \sum_{x} xp(x) $$

where \(X\) is a discrete random variable with probability mass function \(p(x)\), and by

$$ E[X] = \int_{-\infty}^{\infty} xf(x)dx $$

when \(X\) is a continuous random variable with probability density function \(f(x)\). Since \(E[X]\) is a weighted average of the possible values of \(X\), it follows that if \(X\) must lie between a and b, then so must its expected value, i.e,.

If

$$ P(a\leq X \leq b) = 1 $$

then

$$ a \leq E[X] \leq b $$

Expectations of a sum

It follows that,

\(E [ag(X ) + bh(Y )] = aE [g(X )] + bE [h(Y )]\)

where a and b are constants.

What this equation tells us is that handling the expected value of a linear combination of functions is no more difficult than handling the expected values of the individual functions.

This handling also extends to situations where we have more than to variables. Expected values can easily be found from marginal distributions.

Example: Expectation of a sum.

You have been given the following joint pmf. Verify that \(E[{ X }^{ 2 }+3Y]=E[{ X }^{ 2 }]+E[3Y]\)

Solution:

Reading values from the table, we have:

\((E[{ X }^{ 2 }+3Y]=0.1({ 0 }^{ 2 }+3\times 1)+0.1({ 0 }^{ 2 }+3\times 2)+0.2({ 0 }^{ 2 }+3\times 3)\)

                            \(+0.1({ 1 }^{ 2 }+3\times 1)+0.1({ 1 }^{ 2 }+3\times 2)+0.1({ 1 }^{ 2 }+3\times 3)\)

                            \(+0({ 2 }^{ 2 }+3\times 1)+0.2({ 2 }^{ 2 }+3\times 2)+0.1({ 2 }^{ 2 }+3\times 3)=8.1\)

Looking at the terms on the right side above,

\(E[{ X }^{ 2 }]={ 0 }^{ 2 }\times 0.4+{ 1 }^{ 2 }\times 0.3+{ 2 }^{ 2 }\times 0.3=1.5\)

\(E[3Y]=(3\times 1)0.2+(3\times 2)0.4+(3\times 3)0.4=6.6\)

Thus, \(E[{ X }^{ 2 }]+E[3Y]=8.1\), and the reasult has been verified.

Moments of joint random variables

Let \(X,Y\) be a pair of joint random variables with a joint probability function \(f(x,y)\) on the space \(S\). If there exists a function of these two namely \(g(X,Y)\) defined:

$$ E[g(X,Y)] = \sum_{(x,y) \in S} g(x,y)f(x,y) $$

then this function is called the mathematical expectation (or expected value) of \(g(X,Y)\).

This mathematical expectation is known as the first moment of joint random variables, or mean.

The second moment is a derivative of the first moment and it is equal to:

$$ E[g(X,Y)]= E(g(X^2,Y^2)) – (E[g(X,Y)])^2 = Var(X,Y) $$

Example 1: moments of joint random variables

Let \(X\) and \(Y\) have the following pmf:

$$ f(x,y) = \frac{x^2 + 3y}{60} \qquad x = 1,2,3,4\quad y=1,2. $$

Find the expected value with \(g(X,Y) = XY\)

The possible values for this distribution are:

$$(1,1),(1,2),(2,1),(2,2),(3,1),(3,2),(4,1),(4,2)$$

Then we proceed to calculate,

\begin{align} E[XY] & = \sum_{(x,y) \in S} g(x,y) f(x,y)\\ & = \sum_{(x,y) \in S} (xy) \frac{x^2 + 3y}{60}\\ & = (1)\frac{1+3}{60} + (2)\frac{1+6}{60} + (2)\frac{4+3}{60} + (4)\frac{4+6}{60} + \\ & (3)\frac{9+3}{60} + (6)\frac{9+6}{60} + (4)\frac{16+3}{60} + (8)\frac{16+6}{60}\\ & = \frac{4}{60} + \frac{14}{60} + \frac{14}{60} + \frac{40}{60} + \frac{36}{60} + \frac{90}{60} + \frac{76}{60} +\frac{176}{60}\\ & = \frac{15}{2} \end{align}

This process is quite similar to calculating the mean of any mass function, univariate or multivariate.

If \(X\) and \(Y\) are independent, then;

  • The expectation of the product of X and Y is the product of the individual expectations: \(E(XY ) = E(X)E(Y )\). This product formula holds for any expectation of a function \(X\) times a function of \(Y\)
  • The product formula holds for probabilities of the form P(some condition on X, some condition on Y ) (where the comma denotes “and”): For instance, \(P(X\le 3,Y\le 4)=P(X\le 3)P(Y\le 4)\)
  • The variance of the sum of \(X\) and \(Y\) is the sum of the individual variances: \(Var(X + Y ) = Var(X) + Var(Y )\)
Example 2: moments of joint random variables

Let \(X\) and \(Y\) be random variables with joint pmf: \begin{equation*} f(x,y) = \begin{cases} 3(x+y) & 0 < x\leq y\leq 1\\ 0 & \text{otherwise} \end{cases} \end{equation*}

Find \(E(X)\)

This is found by first  working out the marginal distribution function of X as follows:

\({ f }_{ X }(x)=\int _{ x }^{ 1 }{ (3x+3y) } \partial y=|3xy+\cfrac { { 3y }^{ 2 } }{ 2 } |_{ x }^{ 1 }=(3x+\cfrac { 3 }{ 2 } )-({ 3x }^{ 2 }+\cfrac { { 3x }^{ 2 } }{ 2 } )=3x-\cfrac { { 9x }^{ 2 } }{ 2 } +\cfrac { 3 }{ 2 } \)

\(E(X)\quad =\quad \int _{ 0 }^{ 1 }{ x } (3x-\cfrac { { 9x }^{ 2 } }{ 2 } +\cfrac { 3 }{ 2 } )\partial x=\left| \cfrac { { 3x }^{ 3 } }{ 3 } -\cfrac { { 9x }^{ 4 } }{ 8 } +\cfrac { { 3x }^{ 2 } }{ 4 }  \right| _{ 0 }^{ 1 }=1-\cfrac { 9 }{ 8 } +\cfrac { 3 }{ 4 } =\cfrac { 5 }{ 8 } \)

Example 3: moments of joint random variables

Let \(X\) and \(Y\) have the following pmf:

$$ f(x,y) = \frac{x^2 + 3y}{96} \qquad x = 1,2,3,4\quad y=1,2. $$

Find the expected value of \(X\) and \(Y\)

\begin{align} f_x(x) & = \frac{x^2 + 3(1)}{96} + \frac{x^2 + 3(2)}{96}\\ & = \frac{x^2 + 3}{96} + \frac{x^2 + 6}{96}\\ & = \frac{x^2 + x^2 + 3 + 6}{96} = \frac{2x^2+ 9}{96} \end{align}

And

\begin{align} f_y(y) & = \frac{1+3y}{96} + \frac{4+3y}{96} + \frac{9+3y}{96} + \frac{16+3y}{96}\\ & = \frac{12y + 30}{96} \end{align}

Then we can calculate the means for each of the variables:

\begin{align} E(x) & = \sum_{x=1}^{4} xf_x(x) \\ & = \sum_{x=1}^{4} x \frac{2x^2+ 9}{96}\\ & = (1) \frac{2(1)^2+ 9}{60} + (2) \frac{2(2)^2+ 9}{96} + (3) \frac{2(3)^2+ 9}{96} +(4) \frac{2(4)^2+ 9}{96} \\ & = (1)\frac{11}{96} + (2)\frac{17}{96} + (3)\frac{27}{96} + (4)\frac{41}{96} = \frac{145}{48} \end{align}

And the mean for y:

\begin{align} E(y) & = \sum_{y=1}^{2}yf_y(y)\\ & = \sum_{y=1}^{2}y\frac{12y + 30}{96}\\ & = (1) \frac{12(1) + 30}{96} + (2) \frac{12(2) + 30}{96} \\ & = (1) \frac{42}{96} + (2) \frac{54}{96} = \frac{25}{16} \end{align}

With these past examples, we can also proceed to find the variance for the corresponding variables:

Variance = \(V(X)=E({ X }^{ 2 })-{ [E(X)] }^{ 2 }\)

So to work out the variance given a probability function, you need to find \(E({ X }^{ 2 })\) and \(E({ X })\)

Continuing with example 3 above,

\begin{align} V(X) & = \sum_{x=1}^{4} x^2 f_x(x) – [E(X)]^2\\ & = \sum_{x=1}^{4} x^2 \frac{2x^2 + 9}{96} – \bigg(\frac{145}{48}\bigg)^2\\ & = (1)^2 \frac{11}{96} + (2)^2 \frac{17}{96} + (3)^2 \frac{27}{96} + (4)^2\frac{41}{96} – \bigg(\frac{145}{48}\bigg)^2= \frac{163}{16} – \bigg(\frac{145}{48}\bigg)^2 = 1.062 \end{align}

And for \(Y\):

\begin{align} V(Y) &= \sum_{y=1}^{2} y^2f_y(y) – [E(Y)]^2\\ & = \sum_{y=1}^{2} y^2 \frac{12y +30}{96} – \bigg(\frac{25}{16}\bigg)^2\\ & = (1)^2 \frac{42}{96} + (2)^2 \frac{54}{96} – \frac{625}{256}= \frac{43}{16} – \frac{625}{256} = \frac{63}{256} \end{align}

Moments for conditional random variables

Let \(X\) and \(Y\) be random variables with a joint probability function \(f(x,y)\) and marginal functions \(f_x(x)\) and \(f_y(y)\)

Discrete Case:

The conditional pmf of \(Y\) given that \(X=x\) is defined by:

$$h(y|x)=\cfrac { f(x,y) }{ { f }_{ X }(x) }\quad\quad\quad\quad\quad\text{provided that}\quad f_X(x)>0$$

The conditional mean of \(Y\), given that \(X=x\) is defined:

$$ \mu_{Y|x} = E[y|x] = \sum_{y} y h(y|x), $$

and the conditional variance of \(Y\), given that \(X=x\) is defined:

$$ \sigma_{Y|x} = E{(Y – E[Y|x])^2|x} = \sum_{x}(y-E[Y|x])^2 h(y|x) $$

This is simplified to:

$$ \sigma_{Y|x} = E[Y^2|x] – (E[Y|x])^2 $$

Continuous Case

When \(X\) and \(Y\) are continuous random variables, the conditional pdf, mean, and variance are given as follows:

Conditional pdf:

$$h(x|y)=\cfrac { f(x,y) }{ { f }_{ X }(x) } \quad\quad\quad\quad\quad\text{provided that}\quad f_X(x)>0$$

Conditional mean:

$$E(Y|x)=\int _{ -\infty  }^{ \infty  }{ yh(y|x)\partial y } $$

Conditional variance:

$$\text{Var}(Y|x)=E\{ [Y-E(Y|x)]^{ 2 }|x\} $$

                      $$=\int _{ -\infty  }^{ \infty  }{ [y-E(Y|x)]^{ 2 }h(y|x)\partial y } $$

                      $$=E[{ Y }^{ 2 }|x]-{ [E(Y|x] }^{ 2 }$$

Using the same logic, we can come with the conditional pfd/pmf, conditional mean, and conditional variance of \(X\) given that \(Y=y\)

Example: Conditional moments in the discrete case

Consider the joint probability mass function,

$$ f(x,y) = \frac{x^2 + 3y}{60} \qquad x=1,2,3,4,\quad y=1,2. $$

From the joint function, we can get the following marginal pmfs:

$$ f_X(x)= \frac{2x^2+9}{60} \qquad f_Y(y) = \frac{12y+30}{60} $$

We can also find conditional probability mass functions:

$$ g(x|y) = \frac{x^2+3y}{12y+30} \qquad h(y|x)=\frac{x^2+3y}{2x^2+9} $$

With these, suppose we need to find \(E(X|Y=1)\), \(E(Y|X=3)\), as well as the Variance for \(V(X|Y=1)\):

Part 1: Finding \(E(X|Y=1)\):

\begin{align} E(X|Y=1) & = E(g(x|y=1))\\ & = \sum_{x=1}^{4} xg(x|y=1)\\ & = \sum_{x=1}^{4} x \frac{x^2+3(1)}{12(1)+30}\\ & = (1)\frac{1^2+3(1)}{12(1)+30} + (12)\frac{2^2+3(1)}{12(1)+30} + (3)\frac{3^2+3(1)}{12(1)+30} + (4)\frac{4^2+3(1)}{12(1)+30}\\ & = (1)\frac{4}{42} + (2)\frac{7}{42} + (3)\frac{12}{42} + (4)\frac{19}{42} = \frac{65}{21} \end{align}

Part 2: Finding \(E(X|Y=3)\):

\begin{align} E(Y|X=3)& = E(h(y|x=3))\\ & = \sum_{y=1}^{2} yh(y|x=3)\\ & = \sum_{y=1}^{2} y\frac{3^2+3y}{2(3)^2+9}\\ & = (1)\frac{3^2+3(1)}{2(3)^2+9} + (2)\frac{3^2+3(2)}{2(3)^2+9}\\ & = \frac{12}{27}+(2)\frac{15}{27}=\frac{14}{9} \end{align}

Then using the output from example 1, we can compute variance:

Part 3: Finding \(V(X|Y=1)\):

\begin{align} V(X|Y=1) & = E[(X-E(X|Y=1))^2|Y=1]\\ & = \sum_{x=1}^{4}(x-E(X|Y=1))^2g(x|y=1)\\ & = \sum_{x=1}^{4}\bigg(x-\frac{65}{21}\bigg)^2\frac{x^2+3(1)}{12(1)+30}\\ & = \bigg(1-\frac{65}{21}\bigg)^2 \frac{4}{42} + \bigg(2-\frac{65}{21}\bigg)^2 \frac{7}{42} + \bigg(3-\frac{65}{21}\bigg)^2 \frac{12}{42} + \bigg(4-\frac{65}{21}\bigg)^2 \frac{19}{42}\\ & = \bigg(\frac{1936}{441}\bigg)\frac{4}{42} + \bigg(\frac{529}{441}\bigg)\frac{7}{42} + \bigg(\frac{4}{441}\bigg)\frac{12}{42} + \bigg(\frac{361}{441}\bigg)\frac{19}{42} = 0.9909 \end{align}

Thus, we need to calculate the mean to find variance. Now let’s see how to go about a similar problem if the variables involved are continuous. 

Example:  Conditional moments in the continuous case

Let

$$ f(x,y) = \frac{4}{3}(1-xy) \qquad 0\leq x\leq1,\quad 0\leq y\leq1 $$

Part 1: Finding the conditional pdf:

\begin{align} g(x|y)& = \frac{f(x,y)}{f_y(y)} = \frac{\frac{4}{3}(1-xy)}{\int_{0}^{1}\frac{4}{3}(1-xy)dx}=\frac{\frac{4}{3}(1-xy)}{\frac{4}{3}(x-\frac{x^2y}{2})|_{x=0}^{x=1}}=\frac{\frac{4}{3}(1-xy)}{\frac{4}{3}(1-\frac{y}{2})} \end{align}

Part 2: Finding the conditional mean of \(X\)  given \(Y\), \(E(x|y)\)

$$ E(x|y) = \int_{0}^{1}xg(x|y)dx $$

\begin{align} E(x|y)& =\int_{0}^{1} x \frac{(1-xy)}{1-\frac{y}{2}}dx\\ & = \frac{1}{1-\frac{y}{2}} \int_{0}^{1}x(1-xy)dx\\ & = \frac{1}{1-\frac{y}{2}} \int_{0}^{1}(x-x^2y)dx\\\\ & = \frac{1}{1-\frac{y}{2}} \bigg(\frac{x^2}{2}-\frac{x^3y}{3}\bigg)\bigg|_{x=0}^{x=1}\\ & = \frac{1}{1-\frac{y}{2}} \bigg(\frac{1}{2}-\frac{y}{3}\bigg) \end{align}

With this we have the function that will give us the desired expected value, for example if we wanted to find \(E(x|y=1)\), then

$$ E(x|y=1) = \frac{1}{1-\frac{1}{2}} \bigg(\frac{1}{2}-\frac{1}{3}\bigg) = \frac{1}{3} $$

To find variance in the continuous case, we would just integrate over the same region with \(x^2\) instead of \(x\) and then find the difference between this integral and \([E(x|y)]^2\).

Concept Reminders

We compute and define conditional expectations, variances, etc., as usual, but with conditional distributions in place of ordinary distributions:

In the discrete case,

\(E(X|y)=E(X|Y=y)=\sum _{ x }^{  }{ xg(x|y) } \)

\(E({ X }^{ 2 }|y)=E({ X }^{ 2 }|Y=y)=\sum _{ x }^{  }{ { x }^{ 2 }g(x|y) } \)

\(Var(X|y)=Var(X|Y=y)=E({ X }^{ 2 }|y)-[E(X|y)]^{ 2 }\)

and in general,

\(E(X|\text{condition})=\sum _{ x }^{  }{ xP(X=x|\text{condition}) } \)

In the continuous case,

\(E(X|y)=E(X|Y=y)=\int _{  }^{  }{ xg(x|y)\partial x } \)

\(E({ X }^{ 2 }|y)=E({ X }^{ 2 }|Y=y)=\int _{  }^{  }{ { x }^{ 2 }g(x|y)\partial x } \)

\(Var(X|y)=Var(X|Y=y)=E({ X }^{ 2 }|y)-[E(X|y)]^{ 2 }\)

and in general,

\((E(X|{ \text{condition} })=\int _{ -\infty  }^{ \infty  }{ xP(X=x|\text{condition}) } \)

The conditional density (pdf or pmf) of \(X\) given that \(Y = y\) is given by:

\(g(x|y)=\cfrac { f(x,y) }{ { f }_{ Y }(y) } \)

The conditional density of \(Y\) given that \(X=x\) is given by:

\(h(y|x)=\cfrac { f(x,y) }{ { f }_{ X }(x) } \)

Learning Outcome

Topic 3.c: Multivariate Random Variables – Calculate moments for joint, conditional, and marginal random variables.


X