Type I and Type II Errors in Manager Selection Process

Candidates may remember their inferential statistics training from CFA Level I. This reading delves deeper into Type I and Type II errors within the context of hiring and firing investment managers. It's important to note that the null hypothesis presumes the manager lacks skill, doesn't meet expectations, or underperforms.

Null Hypothesis: Manager Underperforms

Therefore, the two potential errors are:

• Type I: Hiring or retaining a manager who later underperforms expectations. Rejecting the null hypothesis when it is correct.
• Type II: Not hiring or firing a manager who later outperforms or performs in line with expectations. Not rejecting the null hypothesis when it is incorrect.

Another, perhaps simpler and less scientific way to think about these errors is:

• Type I: Keeping a bad manager.
• Type II: Missing out on a good manager.

Determining whether to avoid type I or type II errors in fund selection varies based on the fund sponsor's preferences. Many prefer steering clear of poor managers, in other words, avoiding type I errors. Here are a few reasons for this choice:

• Financial market participants psychologically try to avoid regret. Type I errors create explicit costs, while Type II errors create opportunity costs. Many individuals give excess weight to explicit costs in their decision-making processes.
• Type I errors are relatively simple to measure and are often directly linked to the decision maker's compensation. Type II errors are less likely to be measured since they represent a more nebulous opportunity cost.
• Type I errors are more noticeable because they bring not only the regret of a wrong decision but also the challenge of explaining it to the investor. In contrast, Type II errors are less conspicuous.

While most investors often focus on type I errors, type II errors are also significant. Monitoring managers who were fired or overlooked can help fund sponsors identify weaknesses in their selection process.

The impact of type I and type II errors is generally smaller in more efficient markets. Market efficiency, especially its tendency to mean-revert, influences the costs associated with these errors. For instance, in a mean-reverting market, firing an underperformer only to see them bounce back represents a Type I error. Conversely, a Type II error would involve retaining strong performers and avoiding managers with weaker short-term track records, which also incurs costs.

Question

Which of the following most accurately describes a type I error?

1. Rejecting the null hypothesis when it is correct.
2. Not rejecting the null hypothesis when it is incorrect.
3. Not rejecting the null hypothesis with it is correct.

Solution

The correct answer is A.

It describes a Type I error accurately. It occurs when a researcher or analyst incorrectly rejects the null hypothesis, which essentially means they conclude that there is a significant effect or difference when, in reality, there isn't one. This is also known as a “false positive.”

B is incorrect. It describes the correct decision in hypothesis testing. When the null hypothesis is incorrect, you should indeed not reject it because you're essentially saying that the data doesn't provide enough evidence to support the alternative hypothesis. This is not a Type I error.

C is incorrect. It is describing the correct decision in hypothesis testing. When the null hypothesis is correct, you should not reject it because it means that the data doesn't provide enough evidence to support the alternative hypothesis. This is not a Type I error.

Remember:

The null hypothesis is that the manager has no skill.

Performance Measurement: Learning Module 2: Investment Manager Selection; Los 2(b) Contrast Type I and Type II errors in manager hiring and continuation decisions