Calculate joint moments, such as the covariance and the correlation coefficient

Covariance and Correlation Coefficient for Joint Random Variables

In learning outcomes covered previously, we have looked at the joint p.m.f. of two discrete/continuous random variables \(X\) and \(Y\), and we have also established the condition required for \(X\) and \(Y\) what to be considered independent. Now are going to look at non-indepencence.

If \(X\) and \(Y\) are two non-independent (dependent) variables, we would want to establish how one varies with respect to the other. If \(X\) increases, for example, does Y also tend to increase or decrease? And if so, how strong is the dependence between the two? Two measures that can help us answer these questions are covariance and correlation coefficient.

Covariance

Covariance is a measure of the directional relationship between two dependent random variables. The covariance \(cov[X,Y ]\) of two random variables \(X\) and \(Y\) is defined by:

\(cov[X,Y ] = E [(X – E[X ])(Y – E[Y ])]\)

which simplifies to:

\(cov[X,Y ] = E[XY ] – E[X ]E[Y ]\)

If you look at the above keenly, you will notice similarities between covariance and variance in the univariate case:

\(Var(X)=E[{ (X-E(X)) }^{ 2 }]=E{ (X }^{ 2 })-{ E }^{ 2 }(X)\)

Note: The units of \(cov(X,Y )\) are the product of those of \(X\) and \(Y\). So for example, if  \(X\) is a time in hours, and Y is a sum of money in $, then cov is in $x hours.
Note also that \(cov[X,X ] = var[X ]\).

It is rather convenient that the mean and variance of any variable can be computed from either the joint pmf (or pdf) or the marginal pmf (or pdf) of same variable. For example in the discrete case for \(X\),

\begin{align*} \mu_X = E(X) & = \sum_{x}\sum_{y}xf(x,y)\\ & = \sum_{x}x\bigg[\sum_{y}f(x,y)\bigg] = \sum_{x}xf_X(x). \end{align*}

However, to compute the covariance, we need the joint pmf(or pdf).

Example 1: covariance given a joint pmf

Calculate the covariance of the random variables \(X\) and \(Y\), given the following joint pmf:

Solution

We will use the formula \(cov(X ,Y) = E[XY] – E[X ]E[Y]\)

Using data from the table,

\(E[XY]=0\times 1\times 0.1+…+2\times 3\times 0.1=2\)

The (marginal) probability mass function of \(X\) is:

 

 

 

 

Thus, \(E(X)=0\times 0.4+1\times 0.3+2\times 0.3=0.9\)

The (marginal) probability mass function of \(Y\) is:

 

 

 

Thus, \(E(Y)=1\times 0.2+2\times 0.4+3\times 0.4=2.2\)

Hence, \(Cocv(X,Y)=2-0.9\times 2.2=0.02\)

Useful Results

  1. \(cov[aX + b,cY + d]= accov[X,Y ]\)
  2. \(cov[X,Y + Z]= cov[X,Y ]+cov[X,Z]\)
  3. If \(X\) and \(Y\) are independent, \(cov[X,Y ] = 0\)

The covariance between X and Y is a measure of the strength of the “linear association” or “linear relationship” between the variables. However, one of its major negative points is that its value is dependent on the units of measurement of the variables. This is corrected by computing the correlation coefficient, a dimensionless (unitless) quantity.

Correlation Coefficient

The correlation coefficient \((X ,Y )\) (usually written as \(corr(X,Y)\) or \(\rho (X,Y)\) of two random variables \(X\) and \( Y\) is defined by:

\(corr(X,Y)=\rho (X,Y)=\cfrac { cov(X,Y) }{ \sqrt { var(X)var(Y) }  } \)

The correlation coefficient takes a value in the range \(-1\le \rho \le 1\). It reflects the degree of association between the two variables. It’s also important to note the following:

  1. If X and Y are independent, then \(corr(X,Y) = 0\);
  2. If \(Y = mX +c\) for some constants \(m\neq 0\) and \(c\), then \(corr(X,Y) = 1\) if \(m > 0\),
    and \(corr(X,Y) = −1\) if \(m < 0\).

Example 2: Let \(X\) and \(Y\) have the joint pmf

$$f(x,y)=\frac{1}{33}(x+2y) \qquad x=1,2\qquad y=1,2,3.$$

The marginal probability mass functions are, respectively,

$$f_x(x) = \sum_{y=1}^{3}\frac{1}{33}(x+2y) = \frac{3x+12}{33}$$

and

$$f_y(y)= \sum_{x=1}^{2}\frac{1}{33}(x+2y)= \frac{4y+3}{33}$$

Since \(f(x,y) \neq f_X(x)f_Y(y)\), \(X\) and \(Y\) are dependent. The mean and the variance of \(X\) are, respectively,

$$\mu_X = \sum_{x=1}^{2}x\frac{3x+12}{33} = \frac{17}{11}$$

and

$$\sigma_{X}^2 = \sum_{x=1}^{2} x^2\frac{3x+12}{33} – \bigg(\frac{17}{11}\bigg)^2 = \frac{29}{11} – \frac{289}{121} = \frac{30}{121}$$

The mean and the variance of \(Y\) are, respectively

$$\mu_Y = \sum_{y=1}^{3}y\frac{4y+3}{33} = \frac{74}{33}$$

And,

$$\sigma_Y^2 = \sum_{y=1}^{3}y^2 \frac{4y+3}{33} – \bigg(\frac{74}{33}\bigg)^2 = \frac{62}{11} – \bigg(\frac{74}{33}\bigg)^2 = 0.607$$

The covariance of \(X\) and \(Y\) is

\begin{align*} Cov(X,Y) & = \sum_{x=1}^{2}\sum_{y=1}^{3}xy\frac{x+2y}{33} – \bigg(\frac{17}{11}\bigg)\bigg(\frac{74}{33}\bigg)\\ & = (1)(1)\frac{3}{33} + (1)(2)\frac{5}{33} + (1)(3)\frac{7}{33} + (2)(1)\frac{4}{33}\\ & + (2)(2)\frac{6}{33} + (2)(3) \frac{8}{33} – \bigg(\frac{17}{11}\bigg)\bigg(\frac{74}{33}\bigg)\\ & = \frac{114}{33} – \bigg(\frac{17}{11}\bigg)\bigg(\frac{74}{33}\bigg) = -\frac{4}{363} \end{align*}

Other important properties of covariance include:

  1. \(Cov(X,Y) = Cov(Y,X)\)
  2. \(Cov(X,X) = Var(X)\)
  3. \(Cov(aX,Y) = a Cov(X,Y)\)
  4. \(Cov\bigg(\sum_{i=1}^{n}X_i,\sum_{j=1}^{m}Y_j\bigg) = \sum_{i=1}^{n}\sum_{j=1}^{m}Cov(X_i,Y_j)\)

The correlation coefficient is a measure of the degree of linearity between \(X\) and \(Y\). A value of \(\rho\) near \(+1\) or \(-1\) indicates a high degree of linearity between \(X\) and \(Y\), whereas a value near 0 indicates that such linearity is absent. A positive value of \(\rho\) indicates that \(Y\) tends to increase when \(X\) does, whereas a negative value indicates that \(Y\) tends to decrease when \(X\) increases. If \(\rho=0\), then \(X\) and \(Y\) are said to be uncorrelated.

Example 3: Determine the covariance and correlation coefficient gievn the following joint probability mass function:

$$ f(x,y) = c(x^2 + 3y) \qquad x=1,2,3,4,\quad y=1,2. $$

Solution:

First, we need to find the value of \(c\) and then proceed to extract the marginal functions:

\begin{align*} f(1,1)+f(1,2)+f(2,1)+f(2,2)+f(3,1)+f(3,2)+f(4,1)+f(4,2) = & 1\\ 4c+7c+7c+10c+12c+15c+19c+22 = & 1\\ 96c = & 1\\ c = & \frac{1}{96} \end{align*}

The marginal functions are:

$$ f_x(x)= \frac{2x^2+9}{96} \qquad f_y(y) = \frac{12y+30}{96} $$

Let’s now compute moments of \(X\) and \Y\)

\begin{align*} \mu_X & = \sum_{x=1}^{4}xf(x)\\ & = \sum_{x=1}^{4}x\frac{2x^2+9}{96} = (1) \frac{11}{96} + (2) \frac{17}{96} + (3)\frac{27}{96} + (4) \frac{41}{96} \\ & = \frac{11}{96}+ \frac{34}{96} + \frac{81}{96} + \frac{164}{96} = \frac{145}{48} \end{align*}

And,

\begin{align*} \sigma_{X}^2 & = \sum_{x=1}^{4}x^2f(x) – [\mu_X]^2\\ & = \sum_{x=1}^{4}x^2\frac{2x^2+9}{96} – \bigg(\frac{145}{48}\bigg)^2\\ & = (1)^2 \frac{11}{96} + (2)^2 \frac{17}{96} + (3)^2 \frac{27}{96} + (4)^2 \frac{41}{96} – \bigg(\frac{145}{48}\bigg)^2\\ & = \frac{11}{96} + \frac{68}{96}+\frac{243}{96} + \frac{656}{96} – \bigg(\frac{145}{48}\bigg)^2\\ & = \frac{163}{16} – \bigg(\frac{145}{48}\bigg)^2 = 1.062 \end{align*}

Then we find the needed values for \(Y\),

\begin{align*} \mu_Y & = \sum_{y=1}^{2}yf_y(y)\\ & = \sum_{y=1}^{2}(y)\frac{12y+30}{96} = (1)\frac{42}{96} + (2)\frac{54}{96}\\ & = \frac{42}{96} + \frac{108}{96} = \frac{25}{16} \end{align*}

And,

\begin{align*} \sigma_{Y}^2 & = \sum_{y=1}^{2}y^2f_y(y) – [\mu_Y]^2 \\ & = \sum_{y=1}^{2} y^2 \frac{12y+30}{96} – \bigg(\frac{25}{16}\bigg)^2 \\ & = (1)^2 \frac{42}{96} + (2)^2 \frac{54}{96} – \bigg(\frac{25}{16}\bigg)^2 \\ & = \frac{42}{96} + \frac{216}{96} – \frac{625}{256} = \frac{43}{16} – \frac{625}{256} = \frac{63}{256} \end{align*}

After we have found these values we can apply what we discussed so far,

\begin{align*} Cov(X,Y) & = \sum_{x=1}^{4}\sum_{y=1}^{2}xyf(x,y) – \mu_X\mu_Y\\ & = \sum_{x=1}^{4}\sum_{y=1}^{2}xy\frac{x^2+3y}{96} – \bigg(\frac{145}{48}\bigg)\bigg(\frac{25}{16}\bigg)\\ & = (1)(1)\frac{4}{96} + (1)(2)\frac{7}{96} + (2)(1)\frac{7}{96} + (2)(2)\frac{10}{96} + (3)(1)\frac{12}{96} \\ & + (3)(2)\frac{15}{96} + (4)(1)\frac{19}{96} + (4)(2)\frac{22}{96} – \bigg(\frac{145}{48}\bigg)\bigg(\frac{25}{16}\bigg)\\ & = (1)\frac{4}{96} + (2)\frac{7}{96} + (2)\frac{7}{96} + (4)\frac{10}{96} + (3)\frac{12}{96} \\ & + (6)\frac{15}{96} + (4)\frac{19}{96} + (8)\frac{22}{96} – \bigg(\frac{145}{48}\bigg)\bigg(\frac{25}{16}\bigg)\\ & = \frac{4}{96} + \frac{14}{96} + \frac{14}{96} +\frac{40}{96} + \frac{36}{96} + \frac{90}{36} + \frac{76}{96} + \frac{176}{96} – \frac{3625}{768} = \frac{75}{16} – \frac{3625}{768} = -\frac{25}{768} \end{align*}

And lastly,

\begin{align*} \rho & = \frac{Cov(X,Y)}{\sqrt{\sigma_{X}^2\sigma_{Y}^2}}\\ & = \frac{-25/768}{\sqrt{(1.062)(63/256)}} = -0.0636 \end{align*}

Covariance for Continuous Random Variables

If \X\) and \Y\) are continuous random variables, we generally embark on a similar path as above so as to calculate covariance and correlation coefficient. 

  • Find \(E(X)\) and \(E(Y)\) at once with iterated integrals
  • Find \(E(XY)\) applying the iterated integrals \(\int_{X}\int_{Y}xyf(x,y)dydx\)
  • Calculate \(Cov(X,Y)\) with \(E(XY) – E(X)E(Y)\)

Example 4: Let

\begin{equation*} f(x,y)= \begin{cases} \frac{2}{3}(2x+y), & 0 < x < 1,0 < y < 1\\ 0, & \text{otherwise} \end{cases} \end{equation*}

Find \(Cov(X,Y)\)
Solution:
\({ f }_{ x }(x)=\cfrac { 2 }{ 3 } \int _{ 0 }^{ 1 }{ (2x+y) } dy=\cfrac { 2 }{ 3 } \left[ 2xy+\cfrac { { y }^{ 2 } }{ 2 }  \right] { |\begin{matrix} 1 \\ 0 \end{matrix}=\cfrac { 2 }{ 3 } (2x+\cfrac { 1 }{ 2 } ) }\)

\(E(X)=\cfrac { 2 }{ 3 } \int _{ 0 }^{ 1 }{ x(2x+\cfrac { 1 }{ 2 } ) } dx=\cfrac { 2 }{ 3 } \left[ \cfrac { { 2x }^{ 3 } }{ 3 } +\cfrac { { x }^{ 2 } }{ 4 }  \right] { |\begin{matrix} 1 \\ 0 \end{matrix}=\cfrac { 2 }{ 3 } (\cfrac { 2 }{ 3 } +\cfrac { 1 }{ 4 } )=\cfrac { 11 }{ 18 }  }\)

\(E({ X }^{ 2 })=\int _{ 0 }^{ 1 }{ { x }^{ 2 }(2x+\cfrac { 1 }{ 2 } )dx=\cfrac { 2 }{ 3 } { [\cfrac { { x }^{ 4 } }{ 2 } +\cfrac { { x }^{ 3 } }{ 6 } ]| } } \begin{matrix} 1 \\ 0 \end{matrix}=\cfrac { 2 }{ 3 } (\cfrac { 1 }{ 2 } +\cfrac { 1 }{ 6 } )=\cfrac { 4 }{ 9 } \)

\(Var(X)=\cfrac{4}{9}-\cfrac{49}{324}=\cfrac{95}{324}\)

\({ f }_{ y }(y)=\cfrac { 2 }{ 3 } \int _{ 0 }^{ 1 }{ (2x+y) } dx=\cfrac { 2 }{ 3 } \left[ { x }^{ 2 }+xy \right] { |\begin{matrix} 1 \\ 0 \end{matrix}=\cfrac { 2 }{ 3 } (1+y) }\)

\(E({ Y })=\int _{ 0 }^{ 1 }{ y(1+y)dy=\cfrac { 2 }{ 3 } [\cfrac { { y }^{ 2 } }{ 2 } +\cfrac { { y }^{ 3 } }{ 3 } ]|\begin{matrix} 1 \\ 0 \end{matrix}=\cfrac { 2 }{ 3 } (\cfrac { 1 }{ 2 } +\cfrac { 1 }{ 3 } )=\cfrac { 5 }{ 9 }  } \)

\(E({ { y }^{ 2 } })=\int _{ 0 }^{ 1 }{ { y }^{ 2 }(1+y)dy=\cfrac { 2 }{ 3 } [\cfrac { { y }^{ 3 } }{ 3 } +\cfrac { { y }^{ 4 } }{ 4 } ]|\begin{matrix} 1 \\ 0 \end{matrix}=\cfrac { 2 }{ 3 } (\cfrac { 1 }{ 3 } +\cfrac { 1 }{ 4 } )=\cfrac { 7 }{ 18 }  } \)

\(Var(Y)=\cfrac{7}{18}-\cfrac{25}{81}=\cfrac{13}{162}\)

\(E(XY)=\cfrac { 2 }{ 3 } \int _{ 0 }^{ 1 }{ \int _{ 0 }^{ 1 }{ xy(2x+y)dxdy= }  } \cfrac { 2 }{ 3 } \int _{ 0 }^{ 1 }{ \int _{ 0 }^{ 1 }{ (2{ x }^{ 2 }y+x{ y }^{ 2 })dxdy } =\cfrac { 2 }{ 3 } \int _{ 0 }^{ 1 }{ \left[ \cfrac { { 2x }^{ 3 }y }{ 3 } +\cfrac { { x }^{ 2 }{ y }^{ 2 } }{ 2 } |\begin{matrix} 1 \\ 0 \end{matrix} \right]  } dy } \)

         \(=\cfrac { 2 }{ 3 } \int _{ 0 }^{ 1 }{ \left[ \cfrac { 2y }{ 3 } +\cfrac { { y }^{ 2 } }{ 2 }  \right]  } dy=\cfrac { 2 }{ 3 } \left[ \cfrac { 2{ y }^{ 2 } }{ 6 } +\cfrac { { y }^{ 3 } }{ 6 }  \right] |\begin{matrix} 1 \\ 0 \end{matrix}=\cfrac { 2 }{ 3 } (\cfrac { 1 }{ 6 } +\cfrac { 1 }{ 6 } )=\cfrac { 4 }{ 18 } \)

Having found all these values we can calculate the covariance for this function:

\(Cov(X,Y) = E[XY] – E[X]E[Y]=\cfrac { 4 }{ 18 }-\cfrac { 11 }{ 18 }\times\cfrac { 5 }{ 9 } = \cfrac{-19}{162}\)

And lastly,

\(\rho =\cfrac { cov(X,Y) }{ \sqrt { var(X)var(Y) }  } =\cfrac { -19/162 }{ \sqrt { \cfrac { 95 }{ 324 } \times \cfrac { 13 }{ 162 }  }  } =-0.76\)

 

Learning Outcome

Topic 3.f: Multivariate Random Variables – Calculate joint moments, such as the covariance and the correlation coefficient.


X