Problems in Backtesting

Problems in Backtesting

I. Survivorship Bias

Survivorship bias occurs when a conclusion is drawn from data whose scope only captures companies that survived until the date the backtesting was done. It is worth clarifying that many practitioners fail to quantify the effects of survivorship bias in their backtesting. Several investors suggest that there is nothing wrong with backtesting strategies using the current index constituents. The problem that may arise, however, is that one cannot tell the company which will go under from that which will be amalgamated and subsequently become successful enough to be added to the index. There is yet another problem: The list of surviving firms is also thought to be biased.

When data is available from data vendors, it is strongly recommended that investors use a point-in-time approach to track all companies that have ever existed. That way, the backtesting exercise will yield realistic results.

A low-volatility anomaly is an empirical observation used to illustrate the difference between backtesting with point-in-time data and current index constituents. Low-volatility anomaly argues that low volatility stocks outperform those with high volatility.

II. Look-ahead Bias

Look-ahead bias emanates from the use of unavailable information by an investor during the historical periods over which a backtest is conducted. It is noteworthy that this is the most common mistake made when backtesting is conducted. To avert its occurrence, we use point-in-time data. There are three common forms of look-ahead data.

  1. Reporting lag: Occurs when an investor lacks all information since it has not been posted on a company’s website. They, therefore, add several months of reporting lag for every company. This process introduces stale information.
  2. Data revisions: Financial statements are often restated. In fact, macroeconomic data can be revised severally. The revised data often replaces the old data, and an analyst will, as such, use information that was unavailable to them when carrying out a backtest.
  3. Index additions: Data vendors often add new companies to their database. When they do so, they also include past financial statements dating back to several years. An analyst backtesting with the updated database would use the information on firms that were not originally in the database during the period. The effect of this look-ahead bias is the generation of excessively optimistic results.

III. Data Snooping

Data snooping, also known as p-hacking or selective reporting, involves making an inference after looking at statistical results instead of testing the inference. An analyst does this by selecting data until they obtain the desired result.

Data snooping can take the following forms:

  • Dropping outliers after performing analysis.
  • Performing an interim analysis to decide if the data collection process will continue.
  • Using many variables.
  • Deciding which ones to report later.

Data snooping generates false positives. To mitigate data snooping, we often use cross-validation or higher-than-average hurdles, e.g. a high t-statistic.

Question

Cross-validation is useful in the avoidance of which one of the following problems that may affect backtesting?

  1. Data snooping.
  2. Survivorship bias.
  3. Look-ahead bias.

Solution

The correct answer is A.

Cross-validation is often used to mitigate data snooping problems.

B and C are incorrect. Both survivorship and look-ahead bias use point-in-time data to avoid backtesting problems.

Reading 43: Backtesting and Simulation

LOS 43 (d) Identify problems in a backtest of an investment strategy.

Shop CFA® Exam Prep

Offered by AnalystPrep

Featured Shop FRM® Exam Prep Learn with Us

    Subscribe to our newsletter and keep up with the latest and greatest tips for success
    Shop Actuarial Exams Prep Shop GMAT® Exam Prep


    Daniel Glyn
    Daniel Glyn
    2021-03-24
    I have finished my FRM1 thanks to AnalystPrep. And now using AnalystPrep for my FRM2 preparation. Professor Forjan is brilliant. He gives such good explanations and analogies. And more than anything makes learning fun. A big thank you to Analystprep and Professor Forjan. 5 stars all the way!
    michael walshe
    michael walshe
    2021-03-18
    Professor James' videos are excellent for understanding the underlying theories behind financial engineering / financial analysis. The AnalystPrep videos were better than any of the others that I searched through on YouTube for providing a clear explanation of some concepts, such as Portfolio theory, CAPM, and Arbitrage Pricing theory. Watching these cleared up many of the unclarities I had in my head. Highly recommended.
    Nyka Smith
    Nyka Smith
    2021-02-18
    Every concept is very well explained by Nilay Arun. kudos to you man!
    Badr Moubile
    Badr Moubile
    2021-02-13
    Very helpfull!
    Agustin Olcese
    Agustin Olcese
    2021-01-27
    Excellent explantions, very clear!
    Jaak Jay
    Jaak Jay
    2021-01-14
    Awesome content, kudos to Prof.James Frojan
    sindhushree reddy
    sindhushree reddy
    2021-01-07
    Crisp and short ppt of Frm chapters and great explanation with examples.