Problems in Backtesting

I. Survivorship Bias

Survivorship bias occurs when a conclusion is drawn from data whose scope only captures companies that survived until the date the backtesting was done. It is worth clarifying that many practitioners fail to quantify the effects of survivorship bias in their backtesting. Several investors suggest that there is nothing wrong with backtesting strategies using the current index constituents. The problem that may arise, however, is that one cannot tell the company which will go under from that which will be amalgamated and subsequently become successful enough to be added to the index. There is yet another problem: The list of surviving firms is also thought to be biased.

When data is available from data vendors, it is strongly recommended that investors use a point-in-time approach to track all companies that have ever existed. That way, the backtesting exercise will yield realistic results.

A low-volatility anomaly is an empirical observation used to illustrate the difference between backtesting with point-in-time data and current index constituents. Low-volatility anomaly argues that low volatility stocks outperform those with high volatility.

II. Look-ahead Bias

Look-ahead bias emanates from the use of unavailable information by an investor during the historical periods over which a backtest is conducted. It is noteworthy that this is the most common mistake made when backtesting is conducted. To avert its occurrence, we use point-in-time data. There are three common forms of look-ahead data.

1. Reporting lag: Occurs when an investor lacks all information since it has not been posted on a company’s website. They, therefore, add several months of reporting lag for every company. This process introduces stale information.
2. Data revisions: Financial statements are often restated. In fact, macroeconomic data can be revised severally. The revised data often replaces the old data, and an analyst will, as such, use information that was unavailable to them when carrying out a backtest.
3. Index additions: Data vendors often add new companies to their database. When they do so, they also include past financial statements dating back to several years. An analyst backtesting with the updated database would use the information on firms that were not originally in the database during the period. The effect of this look-ahead bias is the generation of excessively optimistic results.

III. Data Snooping

Data snooping, also known as p-hacking or selective reporting, involves making an inference after looking at statistical results instead of testing the inference. An analyst does this by selecting data until they obtain the desired result.

Data snooping can take the following forms:

• Dropping outliers after performing analysis.
• Performing an interim analysis to decide if the data collection process will continue.
• Using many variables.
• Deciding which ones to report later.

Data snooping generates false positives. To mitigate data snooping, we often use cross-validation or higher-than-average hurdles, e.g. a high t-statistic.

Question

Cross-validation is useful in the avoidance of which one of the following problems that may affect backtesting?

1. Data snooping.
2. Survivorship bias.
3. Look-ahead bias.

Solution

The correct answer is A.

Cross-validation is often used to mitigate data snooping problems.

B and C are incorrect. Both survivorship and look-ahead bias use point-in-time data to avoid backtesting problems.

Reading 42: Backtesting and Simulation

LOS 42 (d) Identify problems in a backtest of an investment strategy.