Estimating the Parameters of a Simple Linear Regression

While conducting a regression analysis, we start with the dependent variable whose variation we want to explain and the independent variable that explains the changes in the dependent variable. The least-square criterion is used to measure the accuracy of a straight line by minimizing the squared deviations from the line.

$$Y=b_0+b_1X_1+\epsilon$$

Where:

\(Y\) = Dependent Variable

\(b_0\) = Intercept

\(b_1\) = Slope Coefficient

\(X\) = Independent Variable

\(\epsilon\) = Error Term (Noise)

\(b_0\) and \(b_1\) are known as Regression Coefficients.

Since the population parameters values \(\beta_0\) and \(\beta_1\) cannot be observed in a regression model we use only  and   which are estimates, and testing is based on estimated values in relation to the hypothesized population values. When conducting a simple linear regression, the estimated slope \(\hat\beta_0\) and the intercept \(\hat\beta_1\) are such that the sum of the squared vertical distance from the observations to the fitted line is minimized. The sum of squares error (SSE) is:

$$\sum_{i-1}^{n}(Y_{i}-\hat{\beta}_{0}-\hat{\beta}_{1}X_{i})^2$$

$$\sum_{i=1}^{n} e_{i}^{2}$$

With the help of linear squares regression, we can estimate the values of the population parameters of \(\hat\beta_0\) and \(\hat\beta_1\) A line can fit through observations of X and Y, explaining the value need by Y for any value of X. The residual and the dependent variable have the same units of measurement.

To calculate the intercept \(\hat\beta_0\) and the slope \(\hat\beta_1\) for a given sample of (Y, X) pairs of observations, we divide the covariance of Y and X by the variance of X.

$$\hat{\beta}_{1}=\frac{\text{Cov(X,Y)}}{\text{Var(X)}}$$

The intercept is calculated by using the mean of Y and the mean of X.

$$\hat{\beta}_{0}=\hat{Y}-\hat{\beta}_{1}\hat{X}$$

Where:

• \(\hat{Y}\) = Mean of Y
• \(\hat{X}\) = Mean of X

Example:  Estimating Slope and Intercept for the ROA model

$$\small{\begin{array}{l|l|l|l|l|l}\textbf{Company}&{\textbf{ROA}\\ (\textbf{Y}_{\textbf{i}})}&{\textbf{CAPEX}\\ (\textbf{X}_{\textbf{i}})}&{\\(\textbf{Y}_{\textbf{i}}-\bar{\textbf{Y}})^{2}}&{\\ (\textbf{X}_{\textbf{i}}-\bar{\textbf{X}_{\textbf{i}}})^{2}}&{\\(\textbf{Y}_{\textbf{i}}-\bar{\textbf{Y}})^{2}}
\\\hline\text{1} & 7.0 & 0.9 & 31.36 & 23.43 & 27.10 \\ \hline \text{2} & 5.0 & 0.6 & 57.76 & 26.42 & 39.06 \\ \hline \text{3} & 17.0 & 5.2 & 19.36 & 0.29 & -2.38 \\ \hline\text{4} & 22.0 & 12 & 88.36 & 39.19 & 58.84 \\ \hline \text{5} &12.0 & 10 &0.36 & 18.15 & -2.56\\ \hline \textbf{Sum} & 63.0 & 28.7 & 197.2 & 107.48 & 120.06 \\ \hline\text{Mean} & 12.6 & 5.74 & & &\\ \end{array}}$$

Slope coefficient: $$\hat{\beta}_{1}=\frac{\text{120.06}}{\text{107.48}}= 1.12$$ 

Intercept:$$\hat{\beta}_{0}=\text{12.6}-\text{(1.12}\times\text{5.74)}= 6.171$$ 

ROA regression model: $$Y=6.171+1.12X_1+\epsilon$$

Interpreting the regression coefficients

The intercept is the expected mean value of the dependent variable when the value of the independent variable is zero. However, in some contexts, this is not always true. The slope is the one-unit change in the independent variable that will cause a change in the dependent variable. A positive slope indicates that the dependent and independent variables are moving in the same direction. On the other hand, a negative slope indicates that the dependent and independent variables are in opposite directions.

From the previous example, we would interpret the coefficients as follows:

1. The return on assets for a company is 6.171% if the company has no capital expenditure.
2. If Capex increases by one unit, the ROA will increase by 1.12%.

The regression line’s least squares fitting is that the residual term’s expected value is zero. The focus, however, is on minimizing the sum of the squared residual terms.

Cross-Sectional vs. Time-series Regressions.

A cross-sectional regression involves several observations of Y and X for the same period.  The observations for this type of regression can be drawn from different sources.

In a time-series regression, the data is drawn from observation of the same source but in different periods.

Question

Jane estimates a model that regresses her company’s return on equity (ROE) against its growth opportunities (GO). She comes up with the following linear regression:

$$ROE_i=3+1.5GO_i+\epsilon_i$$. The predicted value of the company’s ROE if its GO is 10% is closest to:

1. 18%.
2. 1.5%.
3. 15%.

Solution

The correct answer is A.

$$ROE_i =3+1.5\times10 =18\% $$

B is incorrect. 1.5% is the slope and not the ROE.

C is incorrect. 15% results from the multiplying slope by the independent variable.

Reading 0: Introduction to Linear Regression

LOS 0 (b) Describe the least squares criterion, how it is used to estimate regression coefficients, and their interpretation