Regression with Multiple Explanatory Variables

After completing this reading, you should be able to:

• Distinguish between the relative assumptions of single and multiple regression.
• Interpret regression coefficients in multiple regression.
• Interpret goodness of fit measures for single and multiple regressions, including R² and adjusted R².
• Construct, apply, and interpret joint hypothesis tests and confidence intervals for multiple coefficients in regression.

Unlike linear regression, multiple regression simultaneously considers the influence of multiple explanatory variables on a response variable Y. In other words, it permits us to evaluate the effect of more than one independent variable on a given dependent variable.

The form of the multiple regression model (equation) is given by:

$$Y_i  = β_0  + β_1 X_1i  + β_2 X_2i  + … + β_k X_ki  + ε_i  \quad ∀i = 1, 2, … n$$

Intuitively, the multiple regression model has k slope coefficients and k+1 regression coefficients. Normally, statistical software (such as Excel and R) are used to estimate the multiple regression model.

Interpreting the Multiple Regression Coefficients

The slope coefficients \({\beta}_k\)  computes the level of variation of the dependent variable Y when the independent variable \(X_j\)  changes by one unit while holding other independent variables constant. The interpretation of the multiple regression coefficients is quite different compared to linear regression with one independent variable. The effect of one variable is explored while keeping other independent variables constant.

For instance, a linear regression model with one independent variable could be estimated as \(\hat{Y}=0.6+0.85X_1\). In this case, the slope coefficient is 0.85, which implies that a 1 unit increase in X₁ results in 0.85 units increases independent variable Y.

Now, assume that we had the second independent variable to the regression so that the regression equation is \(\hat{Y}=0.6+0.85X_1+0.65X_2\). A unit increase in X₁ will not result in a 0.85 unit increase in Y unless X₁ and X₂ are uncorrelated. Therefore, we will interpret 0.85 as one unit of X₁ leads to 0.85 units increase in the dependent variable Y, while keeping X­₂ constant.

OLS Estimators for the Multiple Regression Parameters

Although the multiples regression parameters can be estimated, it is challenging since it involves a huge amount of algebra and the use of matrices. However, we build a foundation of understanding using the multiple regression model with two explanatory variables.

Consider the following multiples regression equation.

$$Y_i  = β_0  + β_1 X_1i  + β_2 X_2i+ε_i$$

The OLS estimator of β₁ is estimated as follows:

The first step is to regress X₁ and X₂  and to get the residual of  \(X_{1i}\)  given by:

$${\epsilon}_{X_{1i}}=X_{1i}-{\hat{\alpha}}_{0}-{\hat{\alpha}}_{1}X_{2i}$$

Where \(\hat{α} _0\) and \(\hat{α} _1\)  are the OLS estimators of \(X_{2i}\).

The next step is to regress Y on X₂ to get the residuals of Y_i, which is intuitively given by:

$${\epsilon}_{Y_i }=Y_i-{\hat{y} _0}-{\hat{y}} _{1} X_{2i}$$

Where \({\hat{γ}} _0\) and  \({\hat{γ}} _1\)  are the OLS estimators of \(X_{2i}\). The final step is to regress the residual of X₁ and Y \( (ϵ_{X_{1i}}  \text{ and } ϵ_{Y_i }) \)  to get:

$$ϵ_{Y_i }=\hat{β} _1 ϵ_{X_{1i} }+ϵ_i $$

Note that we do not have a constant, the expected values of \(ϵ_{Y_i }\)  and \(ϵ_{X_i }\)  are both 0. Moreover, the main purpose of the first and the second regression is to exclude the effect of X₂ from both Y and X₁ by dividing the variable into the fittest value which is correlated with X_2, and the residual error which is uncorrelated with X₂ and thus the two-residual obtained is uncorrelated with X₂  by intuition. The last step of the regression gives the regression between the components of Y and X_1, which is uncorrelated with X₂.

The OLS estimator for \(β_2\)  can be approximated analogously as that of \(β_1\)  by exchanging X₂ for X₁ in the process above. By repeating this process, we can estimate a k-parameter model such as:

$$Y_i  = β_0  + β_1 X_{1i}  + β_2 X_{2i}  + … + β_k X_{ki}  + ε_i  ∀i = 1, 2, \dots n$$

Most of the time, this is done using a statistical package such as Excel and R.

Assumptions of the Multiple Regression Model

Suppose that we have n observations of the dependent variable (Y) and the independent variables (X₁, X₂, . . . , X_k), we need to estimate the equation:

$$Y_i  = β_0  + β_1 X_{1i}  + β_2 X_{2i}  + … + β_k X_{ki}  + ε_i  \quad ∀i = 1, 2, \dots n$$

For us to make a valid inference from the above equation, we need to make classical normal multiple linear regression model assumptions as follows:

1. The relationship between the dependent variable, Y, and the independent variables, X₁, X₂, . . . , X_k, is linear.
2. The independent variables (X₁, X₂, . . . , X_k) are iid. Moreover, there is no definite linear relationship that exists between two or more of the independent variables, X₁, X₂, . . . , X_k.
3. The expectation of value of the error term, conditioned on the independent variables, is 0: E(ϵ| X₁, X₂, . . . , X_k) = 0
4. The variance of the error term is equal for all observations. That is, \(E(ϵ_i^2 )=σ_ϵ^2, i=1,2,…,n\) (homoskedasticity assumption). The assumption enables us to estimate the distribution of the regression coefficients_.
5. The error term ϵ is uncorrelated in all observations. Mathematically put,\(E(ϵ_i ϵ_j )=0  \quad ∀ i≠j\)
6. The error term ϵ is normally distributed. This allows us to test the hypothesis about regression analysis.
7. There are no outliers so that \(E(X_{ji}^{4} )<∞\) for all j=1,2….k

The assumptions are almost the same as those of linear regression with one independent variable, only that the second assumption is tailored to ensure no linear relationships between the independent variables (multicollinearity).

Measures of Goodness of Fit

The goodness of fit of a regression is a measure using the Coefficient of determination (\(R^2\)) and the adjusted coefficient of determination.

The Coefficient of Determination (\(R^2\))

Recall that the standard error estimate gives a percentage at which we are certain of a forecast made by a regression model. However, it does not tell us how suitable is the independent variable in determining the dependent variable. The coefficient of variation corrects this shortcoming.

The coefficient of variation measures a proportion of the total change in the dependent variable explained by the independent variable.  We can calculate the coefficient of variation in two ways:

1.    Squaring the Correlation Coefficient between the Dependent and Independent Variables.

The coefficient of variation can be computed by squaring the correlation coefficient (r) between the dependent and independent variables. That is:

$$R^{2}=r^{2}$$

 Recall that:

$$r=\frac{Cov(X,Y)}{{\sigma}_{X}{\sigma}_{Y}}$$

Where

\(Cov(X, Y)\)-covariance between two variables, X and Y

\(σ_X\)-standard deviation of X

\(σ_Y\)-standard deviation of Y

However, this method only accommodates regression with one independent variable.

Example: Calculating the Coefficient of Determination using Correlation Coefficient

The correlation coefficient between the money supply growth rate (dependent, Y) and inflation rates (independent, X) is 0.7565. The standard deviation of the dependent (explained) variable is 0.050, and that of the independent variable is 0.02. Regression analysis for the ten years was conducted on this variable. We need to calculate the coefficient of determination.

Solution

We know that:

$$r=\frac{Cov(X,Y)}{{\sigma}_{X}{\sigma}_{Y}}=\frac{0.0007565}{0.05\times 0.02}=0.7565$$

So, the coefficient of determination is given by:

$$r^{2}=0.7565^2 =0.5723=57.23%$$

So, in regression, the money supply growth rate explains roughly 57.23% of the variation in the inflation rate over the ten years.

2.    Method for Regression Model with One or More Independent Variables 

If the regression analysis is known, then our best estimate for any observation for the dependent variable would be the mean. Alternatively, instead of using the mean as an estimate of Y_i, we can predict an estimate using the regression equation. The resulting solution will be denoted as:

$$Y_i  = β_0  + β_1 X_1i  + β_2 X_2i  + … + β_k X_ki  + ε_i=\hat{Y} _i+\hat{ϵ} _i$$

So that:

$$Y_i=\hat{Y} _i+\hat{ϵ} _i$$

Now if we subtract the mean of the dependent variable  in the above equation and square and sum on both sides so that:

$$\begin{align*}\sum_{i=1}^{n}{\left(Y_i -\bar{Y}\right)^2}&=\sum_{i=1}^{n}{\left(\hat{Y}_i -\bar{Y}+\hat{\epsilon}_i\right)^2}\\&=\sum_{i=1}^{n}{\left(\hat{Y}_i -\bar{Y}\right)^2}+2\sum_{i=1}^{n}{\hat{\epsilon}_{i} \left(\hat{Y}_i -\bar{Y}\right)}+\sum_{i=1}^{n}{\hat{\epsilon}_{i}^2}\end{align*}$$

Note that:

$$2\sum_{i=1}^{n}{\hat{\epsilon}_{i} \left(\hat{Y}_i -\bar{Y}\right)}=0$$

Since the sample correlation between \(\hat{Y} _i\) and \(\hat{\epsilon} _i\)is 0. The expression, therefore, reduces to,

$$\sum_{i=1}^{n}{\left(Y_i -\bar{Y}\right)^2}=\sum_{i=1}^{n}{\left(\hat{Y}_i -\bar{Y}\right)^2}+\sum_{i=1}^{n}{\hat{\epsilon}_{i}^2}$$

But

$$\hat{\epsilon}^{2}_{i}=\left(Y_i -\hat{Y}\right)^2$$

So, that

$$\sum_{i=2}^{n}{\hat{\epsilon}_{i}^{2}}=\sum_{i=1}^{n}{\left(Y_i -\hat{Y}\right)^2}$$

Therefore,

$$\sum_{i=1}^{n}{\left(Y_i -\bar{Y}\right)^2}=\sum_{i=1}^{n}{\left(\hat{Y}_i -\bar{Y}\right)^2}+\sum_{i=1}^{n}{\left(Y_i -\hat{Y}\right)^2}$$

If the regression analysis is useful for predicting Y_i using the regression equation, then the error should be smaller than predicting Y_i using the mean.

Now let:

Explained Sum of Squares (ESS)=\(\sum_{i=1}^{n} {\left(\hat{Y}_i -\bar{Y}\right)^2}\)

Residual Sum of Squares (RSS)  =\(\sum_{i=1}^{n} {\left(Y_i -\hat{Y}\right)^2}\)

Total Sum of Squares (TSS)=\(\sum_{i=1}^{n} {\left(Y_i -\bar{Y}\right)^2}\)

Then:

 $$TSS=ESS +RSS$$

If we divide both sides by TSS, we get:

$$\begin{align*}1&=\frac{ESS}{TSS}  +\frac{RSS}{TSS}\\&⇒\frac{ESS}{TSS}=1-\frac{RSS}{TSS}\end{align*}$$

Now, recall than the coefficient of determination is the fraction of the overall change that is reflected in the regression. Denoted by R², coefficient of determination is given by:

$$R^2 =\frac{\text{Explained Variation}}{\text{Total Variation}}=\frac{ESS}{TSS}=1-\frac{RSS}{TSS}$$

If a model does not explain any of the observed data, then it has an \(R^2\)  of 0 . On the other hand, if the model perfectly describes the data, then it has an \(R^2\) of 1. Other values are in the range of 0 and 1 and are always positive. For instance, in the above example, the \(R^2\)  is approximately 1 and thus, the money supply growth rate perfectly explains the level of inflation rates in the countries.

Limitations of \(R^2\)

1. As the number of explanatory variables increases, the value of \(R^2\) always increases even if the new variable is almost completely irrelevant to the dependent variable. For instance, if a regression model with one explanatory variable is modified to have two explanatory variables, the new \(R^2\) is greater or equal to that of a single explanatory model. In the case where \(\beta = 0\), adding a variable will not increase \(R^2\). In that case, the RSS will remain the same and so does \(R^2\).
2. The Coefficient of Determination \(R^2\) cannot be compared in different dependent variables. For instance, we cannot compare the \(R^2\) for Y_i and lnY_i .
3. There is no standard value of \(R^2\) that is considered good because its values depend on the nature of the data involved.

Considering the first limitation, we now discuss the adjusted \(R^2\).

The Adjusted R²

Denoted by  \(\bar{R} ^2\), the adjusted-R² measures the goodness of fit, which does not automatically increase if an independent variable is added to the model; that is, it is adjusted for the degrees of freedom. Note that \(\bar{R} ^2\)  is produced by statistical software. The relationship between the R² and   \(\bar{R} ^2\)  is given by:

$$\bar{R}^2 =1-\frac{\left(\frac{RSS}{n-k-1}\right)}{\left(\frac{TSS}{n-1}\right)}$$

$$=1-\left(\frac{n-1}{n-k-1}\right)\left(1-R^2\right)$$

Where

n=number of observations

k=number of the independent variables (Slope coefficients)

The adjusted R-squared can increase, but that happens only if the new variable improves the model more than would be expected by chance. If the added variable improves the model by less than expected by chance, then the adjusted R-squared decreases.

When k≥ 1, then \(R^2>\bar{R}^2\)  since adding an extra new independent variable results in a decrease in \(\bar{R}^2\)  if that addition causes a small increase in \(R^2\). This explains the fact that \(\bar{R}^2\)  can be a negative though \(R^2\)  is always nonnegative.

A point to note is that when we decide to use \(\bar{R}^2\)  to compare the regression models, the dependent variable is defined the same way and that the sample size is the same as that of \(R^2\).

The following are the factors to watch out for when guarding against applying the \(R^2\)  or the \(\bar{R}^2\):

• An added variable doesn’t have to be statistically significant just because the \(R^2\)  or the  \(\bar{R}^2\)  has increased.
• It is not always true that the regressors are a true cause of the dependent variable, just because there is a high \(R^2 \) or \(\bar{R}^2\).
• It is not necessary that there is no omitted variable bias just because we have a high R²  or \(\bar{R}^2\).
• It is not necessarily true that we have the most appropriate set of regressors just because we have a high \(R^2\)  or \(\bar{R}^2\)
• It is not necessarily true that we have an inappropriate set of regressors just because we have a low \(R^2\)  or \(\bar{R}^2\).

\(\bar{R}^2\) does not automatically indicate that regression is well specified due to its inclusion of a right set of variables since a high \(\bar{R}^2\)  could reflect other uncertainties in the data in the analysis. Moreover, \(\bar{R}^2\)  can be negative if the regression model produces an extremely poor fit.

Joint Hypothesis Test on Multiple Regression Parameters

Previously, we had conducted hypothesis tests on individual regression coefficients using the t-test. We need to perform a joint hypothesis test on the multiple regression coefficients using the F-test based on the F-statistic.

In multiple regression, we cannot test the null hypothesis that all the slope coefficients are equal to 0 using the t-test. This is because an individual test on the coefficient does not accommodate the effect of interactions among the independent variables (multicollinearity).

F-test (test of regression’s generalized significance) determines whether the slope coefficients in multiple linear regression are all equal to 0. That is, the null hypothesis is stated as\(H_0:β_1=β_2  = … =β_K= 0\)  against the alternative hypothesis that at least one slope coefficient is not equal to 0.

To accurately compute the test statistic for the null hypothesis that the slope is equal to 0, we need to identify the following:

I. The Sum of Squared Residuals (SSR) given by:

$$\sum_{i=1}^{n} {\left(Y_i -\hat{Y}_i\right)^2}$$

This is also called the residual sum of squares.

II.Explained Sum of Squares (ESS) given by:

$$\sum_{i=1}^{n} {\left(\hat{Y}_i -\bar{Y}_i\right)^2}$$

III. The total number of observations (n).

III. The number of parameters to be estimated. For example, in a regression analysis with one independent variable, there are two parameters: the slope and the intercept coefficients.

Using the above four requirements, we can determine the F-statistic. The F-statistic measures how effective the regression equation explains the changes in the dependent variable. The F-statistic is denoted by F_{(Number of slope parameters, n-(number of parameters))}. For instance, the F-statistic for multiple regression with two slope coefficients (and one intercept coefficient) is denoted as F_{2, n-3}. The value n-3 represents the degrees of freedom for the F-statistic.

The F-statistic is the ratio of the average regression sum of squares to the average amount of squared errors. The average regression sum of squares is the regression sum of squares divided by the number of slope parameters (k) estimated. The average sum of squared errors is the sum of squared errors divided by the number of observations (n) less a total number of parameters estimated ((n – (k + 1)). Mathematically:

$$F=\frac{\text{Average regression sum of squares}}{\text{The average sum of squared errors}}$$

$$=\frac{\frac{\text{Explained sum of squares}}{\text{ESS Slope parameters estimated}}}{\frac{\text{Sum of squared residuals (SSR)}}{n-\text{number of parameters estimated}}}$$

In this case, we are dealing with a multiple linear regression model with k independent variable whose F-statistic is given by:

$$F=\frac{\left(\frac{ESS}{k}\right)}{\left(\frac{SSR}{n-(k+1)}\right)}$$

In regression analysis output (ANOVA part), MSR and MSE are displayed as the first and the second quantities under the MSS (mean sum of the squares) column, respectively. If the overall regression’s significance is high, then the ratio will be large.

If the independent variables do not explain any of the variations in the dependent variable, each predicted independent variable \(\hat{Y}_i )\) possesses the mean value of the dependent variable \((Y)\). Consequently, the regression sum of squares is 0 implying the F-statistic is 0.

So, how do we decide F-test? We reject the null hypothesis at α significance level if the computed F-statistic is greater than the upper α critical value of the F-distribution with the provided numerator and denominator degrees of freedom (F-test is always a one-tailed test).

Example: Conducting F-test

An analyst runs a regression of monthly value-stock returns on four independent variables over 48 months.

The total sum of squares for the regression is 360, and the sum of squared errors is 120.

Test the null hypothesis at a 5% significance level (95% confidence) that all the four independent variables are equal to zero.

Solution

$$H_0:β_1=0,β_2=0,…,β_4=0$$

Versus

$$ H_1:β_j≠0 \text{(at least one j is not equal to zero, j=1,2… k )} $$

ESS = TSS – SSR = 360 – 120 = 240

The calculated test statistic:

$$F=\frac{\left(\frac{ESS}{k}\right)}{\left(\frac{SSR}{n-(k+1)}\right)}=\frac{\frac{240}{4}}{\frac{120}{43}}=21.5$$

\(F_{4,43}^3\) is approximately 2.59 at a 5% significance level.

Decision: Reject H₀.

Conclusion: at least one of the 4 independent variables is significantly different than zero.

Example: Calculating F-statistic and Conducting the F-test

An investment analyst wants to determine whether the natural log of the ratio of bid-offer spread to the price of a stock can be explained by the natural log of the number of market participants and the amount of market capitalization. He assumes a 5% significance level. The following is the result of the regression analysis.

$$\begin{array}{l|c|c|c} & \textbf{Coefficient} & \textbf{Standard Error} & \textbf{t-Statistic} \\ \hline \text{Intercept } & 1.6959 & 0.2375 & 7.0206 \\ \text{Number of market participants} & -1.6168 & 0.0708 & -22.8361 \\ \text{Amount of Capitalization} & -0.4709 & 0.0205 & -22.9707 \end{array}$$

$$\begin{array}{l|c|c|c} \textbf{ANOVA} & \textbf{df} & \textbf{SS} & \textbf{MSS} & \textbf{F} & \textbf{Significance F} \\ \hline \text{Regression} & 2 & 3,730.1534 & 1,865.0767 & 2,217.95 & 0.00 \\ \text{Residual} & 2,797 & 2,351.9973 & 0.8409 & & \\ \text{Total} & 2,799 & 5,801.2051 & & & \end{array}$$

$$ \begin{array}{lc} \text{Residual standard error} & 0.9180 \\ \text{Multiple R-squared}      & 0.6418 \\ \text{Observations}& 2,800 \end{array} $$

We are concerned with the ANOVA (Analysis of variance) results. We need to conduct F-test to determine the significance of regression analysis.

Solution

So, the hypothesis is stated as:

$$H_0 :\hat{\beta}_1 =\hat{\beta}_2=0$$

vs

$$H_1: \text{At least 1}  \hat{β} _j≠0, ∀ j=1,2$$

There are two slope coefficients, k=2 (coefficients on the natural log of the number of market participants and the amount of market capitalization), which is degrees of freedom for the numerator of the F-statistic formula. For the denominator, the degrees of freedom are n- (k + 1) =2800-3= 2,797.

The sum of the squared errors is 2,351.9973, while the regression sum of squares is 3,730.1534. Therefore, the F-statistic is:

$$F_{2,2797}=\frac{\left(\frac{ESS}{k}\right)}{\left(\frac{SSR}{n-(k+1)}\right)}=\frac{\frac{3730.1534}{2}}{\frac{2351.9973}{2797}}=2217.9530$$

Since we are working at a 5% (0.05) significance level, we look at the F-distribution table on the second column which displays the F-distributions with degrees of freedom in the numerator of the F-statistic formula as seen below:

$$\textbf{F Distribution: Critical Values of F (5% significance level)}$$

$$\begin{array}{l|llllllllll}
& \textbf{1} & \textbf{2} & \textbf{3} & \textbf{4} & \textbf{5} & \textbf{6} & \textbf{7} & \textbf{8} & \textbf{9} & \textbf{10} \\ \hline
\textbf{10} & 4.96 & 4.10 & 3.71 & 3.48 & 3.33 & 3.22 & 3.14 & 3.07 & 3.02 & 2.98 \\
\textbf{11} & 4.84 & 3.98 & 3.59 & 3.36 & 3.20 & 3.09 & 3.01 & 2.95 & 2.90 & 2.85 \\
\textbf{12} & 4.75 & 3.89 & 3.49 & 3.26 & 3.11 & 3.00 & 2.91 & 2.85 & 2.80 & 2.75 \\
\textbf{13} & 4.67 & 3.81 & 3.41 & 3.18 & 3.03 & 2.92 & 2.83 & 2.77 & 2.71 & 2.67 \\
\textbf{14} & 4.60 & 3.74 & 3.34 & 3.11 & 2.96 & 2.85 & 2.76 & 2.70 & 2.65 & 2.60 \\
\textbf{15} & 4.54 & 3.68 & 3.29 & 3.06 & 2.90 & 2.79 & 2.71 & 2.64 & 2.59 & 2.54 \\
\textbf{16} & 4.49 & 3.63 & 3.24 & 3.01 & 2.85 & 2.74 & 2.66 & 2.59 & 2.54 & 2.49 \\
\textbf{17} & 4.45 & 3.59 & 3.20 & 2.96 & 2.81 & 2.70 & 2.61 & 2.55 & 2.49 & 2.45 \\
\textbf{18} & 4.41 & 3.55 & 3.16 & 2.93 & 2.77 & 2.66 & 2.58 & 2.51 & 2.46 & 2.41 \\
\textbf{19} & 4.38 & 3.52 & 3.13 & 2.90 & 2.74 & 2.63 & 2.54 & 2.48 & 2.42 & 2.38 \\
\textbf{20} & 4.35 & 3.49 & 3.10 & 2.87 & 2.71 & 2.60 & 2.51 & 2.45 & 2.39 & 2.35 \\
\textbf{21} & 4.32 & 3.47 & 3.07 & 2.84 & 2.68 & 2.57 & 2.49 & 2.42 & 2.37 & 2.32 \\
\textbf{22} & 4.30 & 3.44 & 3.05 & 2.82 & 2.66 & 2.55 & 2.46 & 2.40 & 2.34 & 2.30 \\
\textbf{23} & 4.28 & 3.42 & 3.03 & 2.80 & 2.64 & 2.53 & 2.44 & 2.37 & 2.32 & 2.27 \\
\textbf{24} & 4.26 & 3.40 & 3.01 & 2.78 & 2.62 & 2.51 & 2.42 & 2.36 & 2.30 & 2.25 \\
\textbf{25} & 4.24 & 3.39 & 2.99 & 2.76 & 2.60 & 2.49 & 2.40 & 2.34 & 2.28 & 2.24 \\
\textbf{26} & 4.22 & 3.37 & 2.98 & 2.74 & 2.59 & 2.47 & 2.39 & 2.32 & 2.27 & 2.22 \\
\textbf{27} & 4.21 & 3.35 & 2.96 & 2.73 & 2.57 & 2.46 & 2.37 & 2.31 & 2.25 & 2.20 \\
\textbf{28} & 4.20 & 3.34 & 2.95 & 2.71 & 2.56 & 2.45 & 2.36 & 2.29 & 2.24 & 2.19 \\
\textbf{29} & 4.18 & 3.33 & 2.93 & 2.70 & 2.55 & 2.43 & 2.35 & 2.28 & 2.22 & 2.18 \\
\textbf{30} & 4.17 & 3.32 & 2.92 & 2.69 & 2.53 & 2.42 & 2.33 & 2.27 & 2.21 & 2.16 \\
\textbf{35} & 4.12 & 3.27 & 2.87 & 2.64 & 2.49 & 2.37 & 2.29 & 2.22 & 2.16 & 2.11 \\
\textbf{40} & 4.08 & 3.23 & 2.84 & 2.61 & 2.45 & 2.34 & 2.25 & 2.18 & 2.12 & 2.08 \\
\textbf{50} & 4.03 & 3.18 & 2.79 & 2.56 & 2.40 & 2.29 & 2.20 & 2.13 & 2.07 & 2.03 \\
\textbf{60} & 4.00 & 3.15 & 2.76 & 2.53 & 2.37 & 2.25 & 2.17 & 2.10 & 2.04 & 1.99 \\
\textbf{70} & 3.98 & 3.13 & 2.74 & 2.50 & 2.35 & 2.23 & 2.14 & 2.07 & 2.02 & 1.97 \\
\textbf{80} & 3.96 & 3.11 & 2.72 & 2.49 & 2.33 & 2.21 & 2.13 & 2.06 & 2.00 & 1.95 \\
\textbf{90} & 3.95 & 3.10 & 2.71 & 2.47 & 2.32 & 2.20 & 2.11 & 2.04 & 1.99 & 1.94 \\
\textbf{100} & 3.94 & 3.09 & 2.70 & 2.46 & 2.31 & 2.19 & 2.10 & 2.03 & 1.97 & 1.93 \\
\textbf{120} & 3.92 & \textbf{3.07} & 2.68 & 2.45 & 2.29 & 2.18 & 2.09 & 2.02 & 1.96 & 1.91 \\\textbf{1000} & 3.85 & \textbf{3.00} & 2.61 & 2.38 & 2.22 & 2.11 & 2.02 & 1.95 & 1.89 & 1.84\end{array}$$

As seen from the table, the critical value of the F-test for the null hypothesis to be rejected is between 3.00 and 3.07. The actual F-statistic is 2217.95, which is far higher than the F-test critical value, and thus we reject the null hypothesis that all the slope coefficients are equal to 0.

Calculating the Confidence Interval for the Regression Coefficient

Confidence interval (CI) is a closed interval in which the actual parameter is believed to lie with some degree of confidence. Confidence intervals are used to perform hypothesis tests. For instance, we may want to ascertain stock valuation using the capital asset pricing model (CAPM). In this case, we may wish to hypothesize that the beta possesses the market’s systematic risk or averaged beta.

The same analogy used in the regression analysis with one explanatory variable is also used in a multiple regression model using the t-test.

Example: Calculating the Confidence Interval (CI)

An economist tests the hypothesis that interest rates and inflation can explain GDP growth in a country. Using some 73 observations, the analyst formulates the following regression equation:

$$\text{GDP growth}=\hat{b} _0+\hat{b}_1 (\text{Interest})+\hat{b}_2 (\text{Inflation})$$

The regression estimates are as follows:

$$\begin{array}{l|c|c} & \textbf{Coefficient} & \textbf{Standard Error} \\\hline \text{Intercept} & 0.04 & 0.6\% \\ \text{Interest rates} & 0.25 & 6\% \\ \text{Inflation} & 0.20 & 4\% \end{array}$$

What is the  95% confidence interval for the coefficient on the inflation rate?

A. 0.12024 to 0.27976

B. 0.13024 to 0.37976

C. 0.12324 to 0.23976

D. 0.11324 to 0.13976

Solution

The correct answer is A

From the regression analysis, \(\hat{β}\) =0.20 and the estimated standard error, \(s_\hat{β}\)   =0.04. The number of degrees of freedom is 73-3=70. So, the t-critical value at the 0.05 significance level is =\(t_{\frac{0.05}{2} , 73-2-1}=t_{0.025,70}=1.994\). Therefore, the 95% confidence level for the stock return is:

$$\hat{β}±t_c s_{\hat{β}}=0.2±1.994×0.04= [0.12024, 0.27976]$$

Practice Questions

Question 1

An analyst runs a regression of monthly value-stock returns on four independent variables over 48 months. The total sum of squares for the regression is 360 and the sum of squared errors is 120. Calculate the R².

A. 42.1%

B. 50%

C. 33.3%

D. 66.7%

The correct answer is D.

$$ { R }^{ 2 }=\frac { ESS }{ TSS } =\frac { 360 – 120 }{ 360 } = 66.7%$$

Question 2

Refer to the previous problem and calculate the adjusted R².

A. 27.1%

B. 63.6%

C. 72.9%

D. 36.4%

The correct answer is B.

$$ \begin{align*} \bar { R } ^{ 2 }&=1-\frac { n-1 }{ n-k-1 } \times (1 – { R }^{ 2 } )\\&=1-\frac { 48-1 }{ 48-4-1 } \times (1 – 0.667 )\\&= 63.6\% \end{align*}$$

Question 3

Refer to the previous problem. The analyst now adds four more independent variables to the regression and the new R² increases to 69%. What is the new adjusted R² and which model would the analyst prefer?

A. The analyst would prefer the model with four variables because its adjusted R² is higher.

B. The analyst would prefer the model with four variables because its adjusted R² is lower.

C. The analyst would prefer the model with eight variables because its adjusted R² is higher.

D. The analyst would prefer the model with eight variables because its adjusted R² is lower.

The correct answer is A.

$$ {  \text{ New R } }^{ 2 }=69 \% $$

$$ { \text{ New adjusted R } }^{ 2 }=1-\frac { 48-1 }{ 48-8-1 } \times (1 – 0.69 ) = 62.6 \% $$

The analyst would prefer the first model because it has a higher adjusted R² and the model has four independent variables as opposed to eight.

Question 4

An economist tests the hypothesis that GDP growth in a certain country can be explained by interest rates and inflation.

Using some 30 observations, the analyst formulates the following regression equation:

$$ \text{GDP growth} = { \hat { \beta } }_{ 0 } + { \hat { \beta } }_{ 1 } \text{Interest}+ { \hat { \beta } }_{ 2 } \text{Inflation} $$

Regression estimates are as follows:

$$\begin{array}{l|c|c} & \textbf{Coefficient} & \textbf{Standard Error} \\ \hline \text{Intercept} & 0.10 & 0.5\% \\ \text{Interest Rates} & 0.20 &0.05 \\ \text{Inflation} &0.15 &0.03 \end{array}$$

Is the coefficient for interest rates significant at 5%?

A. Since the test statistic < t-critical, we accept H₀; the interest rate coefficient is not significant at the 5% level.

B. Since the test statistic > t-critical, we reject H₀; the interest rate coefficient is not significant at the 5% level.

C. Since the test statistic > t-critical, we reject H₀; the interest rate coefficient is significant at the 5% level.

D. Since the test statistic < t-critical, we accept H₁; the interest rate coefficient is significant at the 5% level.

The correct answer is C.

We have GDP growth = 0.10 + 0.20(Int) + 0.15(Inf)

Hypothesis:

$$ { H }_{ 0 }:{ \hat { \beta } }_{ 1 } = 0 \quad vs \quad { H }_{ 1 }:{ \hat { \beta } }_{ 1 }≠0 $$

The test statistic is:

$$ t = \left( \frac { 0.20 – 0 }{ 0.05 } \right)  = 4 $$

The critical value is t_{(α/2, n-k-1)} = t_0.025,27 = 2.052 (which can be found on the t-table).

$$\begin{array}{l|cccccc} \textbf{df/p} & \textbf{0.40} & \textbf{0.25} & \textbf{0.10} & \textbf{0.05} & \textbf{0.025} & \textbf{0.01} \\ \hline\textbf{25} & 0.256060 & 0.684430 & 1.316345 & 1.708141 & 2.05954 & 2.48511 \\ \textbf{26} & 0.255955 & 0.684043 & 1.314972 & 1.705618 & 2.05553 & 2.47863 \\ \textbf{27} & 0.255858 & 0.683685 & 1.313703 & 1.703288 & \bf{\textit{2.05183}} & 2.47266 \\ \textbf{28} & 0.255768 & 0.683353 & 1.312527 & 1.701131 & 2.04841 & 2.46714 \\ \textbf{29} & 0.255684 & 0.683044 & 1.311434 & 1.699127 & 2.04523 & 2.46202 \end{array}$$Decision: Since test statistic > t-critical, we reject H₀.

Conclusion: The interest rate coefficient is significant at the 5% level.