Estimating the Parameters of a Simple Linear Regression

Estimating the Parameters of a Simple Linear Regression

While conducting a regression analysis, we start with the dependent variable whose variation we want to explain and the independent variable that explains the changes in the dependent variable. The least-square criterion is used to measure the accuracy of a straight line by minimizing the squared deviations from the line.

$$Y=b_0+b_1X_1+\epsilon$$

Where:

\(Y\) = Dependent Variable

\(b_0\) = Intercept

\(b_1\) = Slope Coefficient

\(X\) = Independent Variable

\(\epsilon\) = Error Term (Noise)

\(b_0\) and \(b_1\) are known as Regression Coefficients.

Since the population parameters values \(\beta_0\) and \(\beta_1\) cannot be observed in a regression model we use only  and   which are estimates, and testing is based on estimated values in relation to the hypothesized population values. When conducting a simple linear regression, the estimated slope \(\hat\beta_0\) and the intercept \(\hat\beta_1\) are such that the sum of the squared vertical distance from the observations to the fitted line is minimized. The sum of squares error (SSE) is:

$$\sum_{i-1}^{n}(Y_{i}-\hat{\beta}_{0}-\hat{\beta}_{1}X_{i})^2$$

$$\sum_{i=1}^{n} e_{i}^{2}$$

With the help of linear squares regression, we can estimate the values of the population parameters of \(\hat\beta_0\) and \(\hat\beta_1\) A line can fit through observations of X and Y, explaining the value need by Y for any value of X. The residual and the dependent variable have the same units of measurement.

To calculate the intercept \(\hat\beta_0\) and the slope \(\hat\beta_1\) for a given sample of (Y, X) pairs of observations, we divide the covariance of Y and X by the variance of X.

$$\hat{\beta}_{1}=\frac{\text{Cov(X,Y)}}{\text{Var(X)}}$$

The intercept is calculated by using the mean of Y and the mean of X.

$$\hat{\beta}_{0}=\hat{Y}-\hat{\beta}_{1}\hat{X}$$

Where:

  • \(\hat{Y}\) = Mean of Y
  • \(\hat{X}\) = Mean of X

Example:  Estimating Slope and Intercept for the ROA model

$$\small{\begin{array}{l|l|l|l|l|l}\textbf{Company}&{\textbf{ROA}\\ (\textbf{Y}_{\textbf{i}})}&{\textbf{CAPEX}\\ (\textbf{X}_{\textbf{i}})}&{\\(\textbf{Y}_{\textbf{i}}-\bar{\textbf{Y}})^{2}}&{\\ (\textbf{X}_{\textbf{i}}-\bar{\textbf{X}_{\textbf{i}}})^{2}}&{\\(\textbf{Y}_{\textbf{i}}-\bar{\textbf{Y}})^{2}}
\\\hline\text{1} & 7.0 & 0.9 & 31.36 & 23.43 & 27.10 \\ \hline \text{2} & 5.0 & 0.6 & 57.76 & 26.42 & 39.06 \\ \hline \text{3} & 17.0 & 5.2 & 19.36 & 0.29 & -2.38 \\ \hline\text{4} & 22.0 & 12 & 88.36 & 39.19 & 58.84 \\ \hline \text{5} &12.0 & 10 &0.36 & 18.15 & -2.56\\ \hline \textbf{Sum} & 63.0 & 28.7 & 197.2 & 107.48 & 120.06 \\ \hline\text{Mean} & 12.6 & 5.74 & & &\\ \end{array}}$$

Slope coefficient: $$\hat{\beta}_{1}=\frac{\text{120.06}}{\text{107.48}}= 1.12$$ 

Intercept:$$\hat{\beta}_{0}=\text{12.6}-\text{(1.12}\times\text{5.74)}= 6.171$$ 

ROA regression model: $$Y=6.171+1.12X_1+\epsilon$$

 

Interpreting the regression coefficients

The intercept is the expected mean value of the dependent variable when the value of the independent variable is zero. However, in some contexts, this is not always true. The slope is the one-unit change in the independent variable that will cause a change in the dependent variable. A positive slope indicates that the dependent and independent variables are moving in the same direction. On the other hand, a negative slope indicates that the dependent and independent variables are in opposite directions.

From the previous example, we would interpret the coefficients as follows:

  1. The return on assets for a company is 6.171% if the company has no capital expenditure.
  2. If Capex increases by one unit, the ROA will increase by 1.12%.

The regression line’s least squares fitting is that the residual term’s expected value is zero. The focus, however, is on minimizing the sum of the squared residual terms.

 

Cross-Sectional vs. Time-series Regressions.

A cross-sectional regression involves several observations of Y and X for the same period.  The observations for this type of regression can be drawn from different sources.

In a time-series regression, the data is drawn from observation of the same source but in different periods.

Question

Jane estimates a model that regresses her company’s return on equity (ROE) against its growth opportunities (GO). She comes up with the following linear regression:

$$ROE_i=3+1.5GO_i+\epsilon_i$$. The predicted value of the company’s ROE if its GO is 10% is closest to:

  1. 18%.
  2. 1.5%.
  3. 15%.

Solution

The correct answer is A.

$$ROE_i =3+1.5\times10 =18\% $$

B is incorrect. 1.5% is the slope and not the ROE.

C is incorrect. 15% results from the multiplying slope by the independent variable.

Reading 0: Introduction to Linear Regression

LOS 0 (b) Describe the least squares criterion, how it is used to estimate regression coefficients, and their interpretation

Shop CFA® Exam Prep

Offered by AnalystPrep

Featured Shop FRM® Exam Prep Learn with Us

    Subscribe to our newsletter and keep up with the latest and greatest tips for success
    Shop Actuarial Exams Prep Shop Graduate Admission Exam Prep


    Daniel Glyn
    Daniel Glyn
    2021-03-24
    I have finished my FRM1 thanks to AnalystPrep. And now using AnalystPrep for my FRM2 preparation. Professor Forjan is brilliant. He gives such good explanations and analogies. And more than anything makes learning fun. A big thank you to Analystprep and Professor Forjan. 5 stars all the way!
    michael walshe
    michael walshe
    2021-03-18
    Professor James' videos are excellent for understanding the underlying theories behind financial engineering / financial analysis. The AnalystPrep videos were better than any of the others that I searched through on YouTube for providing a clear explanation of some concepts, such as Portfolio theory, CAPM, and Arbitrage Pricing theory. Watching these cleared up many of the unclarities I had in my head. Highly recommended.
    Nyka Smith
    Nyka Smith
    2021-02-18
    Every concept is very well explained by Nilay Arun. kudos to you man!
    Badr Moubile
    Badr Moubile
    2021-02-13
    Very helpfull!
    Agustin Olcese
    Agustin Olcese
    2021-01-27
    Excellent explantions, very clear!
    Jaak Jay
    Jaak Jay
    2021-01-14
    Awesome content, kudos to Prof.James Frojan
    sindhushree reddy
    sindhushree reddy
    2021-01-07
    Crisp and short ppt of Frm chapters and great explanation with examples.