{"id":22822,"date":"2021-11-09T20:30:01","date_gmt":"2021-11-09T20:30:01","guid":{"rendered":"https:\/\/analystprep.com\/study-notes\/?p=22822"},"modified":"2026-03-26T08:14:43","modified_gmt":"2026-03-26T08:14:43","slug":"problems-in-backtesting","status":"publish","type":"post","link":"https:\/\/analystprep.com\/study-notes\/cfa-level-2\/problems-in-backtesting\/","title":{"rendered":"Problems in Backtesting"},"content":{"rendered":"<p><script type=\"application\/ld+json\">\r\n{\r\n  \"@context\": \"https:\/\/schema.org\",\r\n  \"@type\": \"QAPage\",\r\n  \"mainEntity\": {\r\n    \"@type\": \"Question\",\r\n    \"name\": \"Cross-validation is useful in avoiding which problem in backtesting?\",\r\n    \"text\": \"Cross-validation is useful in the avoidance of which one of the following problems that may affect backtesting?\\n\\nA. Data snooping.\\n\\nB. Survivorship bias.\\n\\nC. Look-ahead bias.\",\r\n    \"answerCount\": 1,\r\n    \"acceptedAnswer\": {\r\n      \"@type\": \"Answer\",\r\n      \"text\": \"The correct answer is A.\\n\\nCross-validation is used to mitigate data snooping, which occurs when multiple models or hypotheses are tested on the same dataset and only the best-performing results are selected, leading to overly optimistic and misleading conclusions.\\n\\nBy splitting the data into training and validation sets and testing models on unseen data, cross-validation helps ensure that results are not driven by overfitting or repeated data mining.\\n\\nOption B is incorrect because survivorship bias relates to excluding failed entities from the dataset. Option C is incorrect because look-ahead bias occurs when future information is improperly used in model construction.\"\r\n    }\r\n  }\r\n}\r\n<\/script><\/p>\r\n\r\n<p><iframe loading=\"lazy\" src=\"\/\/www.youtube.com\/embed\/FqQEumbr5is\" width=\"611\" height=\"343\" allowfullscreen=\"allowfullscreen\"><\/iframe><\/p>\r\n<h2>I. Survivorship Bias<\/h2>\r\n<p><strong>Survivorship bias<\/strong> occurs when a conclusion is drawn from data whose scope only captures companies that survived until the date the backtesting was done. It is worth clarifying that many practitioners fail to quantify the effects of survivorship bias in their backtesting. Several investors suggest that there is nothing wrong with backtesting strategies using the current index constituents. The problem that may arise, however, is that one cannot tell the company which will go under from that which will be amalgamated and subsequently become successful enough to be added to the index. There is yet another problem: The list of surviving firms is also thought to be biased.<\/p>\r\n<p>When data is available from data vendors, it is strongly recommended that investors use a <strong>point-in-time approach<\/strong> to track all companies that have ever existed. That way, the backtesting exercise will yield realistic results.<\/p>\r\n<p>A <strong>low-volatility anomaly<\/strong> is an empirical observation used to illustrate the difference between backtesting with point-in-time data and current index constituents. Low-volatility anomaly argues that low volatility stocks outperform those with high volatility.<\/p>\r\n<div style=\"text-align: center; margin: 25px 0;\"><a style=\"display: inline-flex; align-items: center; justify-content: center; padding: 10px 18px; border: 2px solid #1a73e8; border-radius: 999px; color: #1a73e8; text-decoration: none; font-weight: 500; background-color: #f5f9ff; white-space: nowrap;\" href=\"https:\/\/analystprep.com\/free-trial\/\" target=\"_blank\" rel=\"noopener\"> Practice backtesting biases with a free trial <\/a><\/div>\r\n<h2>II. Look-ahead Bias<\/h2>\r\n<p><strong>Look-ahead bias<\/strong> emanates from the use of unavailable information by an investor during the historical periods over which a backtest is conducted. It is noteworthy that this is the most common mistake made when backtesting is conducted. To avert its occurrence, we use point-in-time data. There are three common forms of look-ahead data.<\/p>\r\n<ol>\r\n\t<li><strong>Reporting lag<\/strong>: Occurs when an investor lacks all information since it has not been posted on a company\u2019s website. They, therefore, add several months of reporting lag for every company. This process introduces stale information.<\/li>\r\n\t<li><strong>Data revisions<\/strong>: Financial statements are often restated. In fact, macroeconomic data can be revised severally. The revised data often replaces the old data, and an analyst will, as such, use information that was unavailable to them when carrying out a backtest.<\/li>\r\n\t<li><strong>Index additions<\/strong>: Data vendors often add new companies to their database. When they do so, they also include past financial statements dating back to several years. An analyst backtesting with the updated database would use the information on firms that were not originally in the database during the period. The effect of this look-ahead bias is the generation of excessively optimistic results.<\/li>\r\n<\/ol>\r\n<h2>III. Data Snooping<\/h2>\r\n<p><strong>Data snooping<\/strong>, also known as <strong>p-hacking<\/strong> or <strong>selective reporting<\/strong>, involves making an inference after looking at statistical results instead of testing the inference. An analyst does this by selecting data until they obtain the desired result.<\/p>\r\n<p>Data snooping can take the following forms:<\/p>\r\n<ul>\r\n\t<li>Dropping outliers after performing analysis.<\/li>\r\n\t<li>Performing an interim analysis to decide if the data collection process will continue.<\/li>\r\n\t<li>Using many variables.<\/li>\r\n\t<li>Deciding which ones to report later.<\/li>\r\n<\/ul>\r\n<p>Data snooping generates false positives. To mitigate data snooping, we often use cross-validation or higher-than-average hurdles, e.g. a high t-statistic.<\/p>\r\n<blockquote>\r\n<h2>Question<\/h2>\r\n<p>Cross-validation is useful in the avoidance of which one of the following problems that may affect backtesting?<\/p>\r\n<ol style=\"list-style-type: upper-alpha;\">\r\n\t<li>Data snooping.<\/li>\r\n\t<li>Survivorship bias.<\/li>\r\n\t<li>Look-ahead bias.<\/li>\r\n<\/ol>\r\n<h3>Solution<\/h3>\r\n<p><strong>The correct answer is A.<\/strong><\/p>\r\n<p>Cross-validation is often used to mitigate data snooping problems.<\/p>\r\n<p><strong>B and C are incorrect.<\/strong>\u00a0Both survivorship and look-ahead bias use point-in-time data to avoid backtesting problems.<\/p>\r\n<\/blockquote>\r\n<p>Reading 42: Backtesting and Simulation<\/p>\r\n<p><em>LOS 42 (d) Identify problems in a backtest of an investment strategy.<\/em><\/p>\r\n<div style=\"text-align: center; margin: 40px 0;\"><a style=\"display: inline-flex; align-items: center; justify-content: center; padding: 12px 20px; border-radius: 999px; background-color: #1a73e8; color: #ffffff; text-decoration: none; font-weight: 600;\" href=\"https:\/\/analystprep.com\/free-trial\/\" target=\"_blank\" rel=\"noopener\"> Start Free Trial \u2192 <\/a>\r\n<p style=\"font-size: 15px; margin-top: 12px; color: #555;\">Learn how look-ahead bias and survivorship bias distort backtesting results, why point-in-time data is essential, and how to evaluate strategy performance realistically in CFA Level II quantitative methods.<\/p>\r\n<\/div>","protected":false},"excerpt":{"rendered":"<p>I. Survivorship Bias Survivorship bias occurs when a conclusion is drawn from data whose scope only captures companies that survived until the date the backtesting was done. It is worth clarifying that many practitioners fail to quantify the effects of&#8230;<\/p>\n","protected":false},"author":5,"featured_media":0,"comment_status":"closed","ping_status":"closed","sticky":false,"template":"","format":"standard","meta":{"_acf_changed":false,"footnotes":""},"categories":[102,473],"tags":[216,564,568],"class_list":["post-22822","post","type-post","status-publish","format-standard","hentry","category-cfa-level-2","category-portfolio-management","tag-cfa-level-2","tag-portfolio-management","tag-reading-41-backtesting-and-simulation","blog-post","no-post-thumbnail","animate"],"acf":[],"yoast_head":"<!-- This site is optimized with the Yoast SEO plugin v27.4 - https:\/\/yoast.com\/product\/yoast-seo-wordpress\/ -->\n<title>Problems in Backtesting and Biases in Data<\/title>\n<meta name=\"description\" content=\"Learn common backtesting problems, including survivorship bias, look-ahead bias, and data snooping, and how they affect strategy results.\" \/>\n<meta name=\"robots\" content=\"index, follow, max-snippet:-1, max-image-preview:large, max-video-preview:-1\" \/>\n<link rel=\"canonical\" href=\"https:\/\/analystprep.com\/study-notes\/cfa-level-2\/problems-in-backtesting\/\" \/>\n<meta property=\"og:locale\" content=\"en_US\" \/>\n<meta property=\"og:type\" content=\"article\" \/>\n<meta property=\"og:title\" content=\"Problems in Backtesting and Biases in Data\" \/>\n<meta property=\"og:description\" content=\"Learn common backtesting problems, including survivorship bias, look-ahead bias, and data snooping, and how they affect strategy results.\" \/>\n<meta property=\"og:url\" content=\"https:\/\/analystprep.com\/study-notes\/cfa-level-2\/problems-in-backtesting\/\" \/>\n<meta property=\"og:site_name\" content=\"CFA, FRM, and Actuarial Exams Study Notes\" \/>\n<meta property=\"article:published_time\" content=\"2021-11-09T20:30:01+00:00\" \/>\n<meta property=\"article:modified_time\" content=\"2026-03-26T08:14:43+00:00\" \/>\n<meta name=\"author\" content=\"Irene R\" \/>\n<meta name=\"twitter:card\" content=\"summary_large_image\" \/>\n<meta name=\"twitter:label1\" content=\"Written by\" \/>\n\t<meta name=\"twitter:data1\" content=\"Irene R\" \/>\n\t<meta name=\"twitter:label2\" content=\"Est. reading time\" \/>\n\t<meta name=\"twitter:data2\" content=\"3 minutes\" \/>\n<script type=\"application\/ld+json\" class=\"yoast-schema-graph\">{\"@context\":\"https:\\\/\\\/schema.org\",\"@graph\":[{\"@type\":\"Article\",\"@id\":\"https:\\\/\\\/analystprep.com\\\/study-notes\\\/cfa-level-2\\\/problems-in-backtesting\\\/#article\",\"isPartOf\":{\"@id\":\"https:\\\/\\\/analystprep.com\\\/study-notes\\\/cfa-level-2\\\/problems-in-backtesting\\\/\"},\"author\":{\"name\":\"Irene R\",\"@id\":\"https:\\\/\\\/analystprep.com\\\/study-notes\\\/#\\\/schema\\\/person\\\/7002f30d8f174958802c1c30b167eaf5\"},\"headline\":\"Problems in Backtesting\",\"datePublished\":\"2021-11-09T20:30:01+00:00\",\"dateModified\":\"2026-03-26T08:14:43+00:00\",\"mainEntityOfPage\":{\"@id\":\"https:\\\/\\\/analystprep.com\\\/study-notes\\\/cfa-level-2\\\/problems-in-backtesting\\\/\"},\"wordCount\":579,\"keywords\":[\"CFA-level-2\",\"Portfolio management\",\"Reading 41: Backtesting and Simulation\"],\"articleSection\":[\"CFA Level II Study Notes\",\"Portfolio Management\"],\"inLanguage\":\"en-US\"},{\"@type\":\"WebPage\",\"@id\":\"https:\\\/\\\/analystprep.com\\\/study-notes\\\/cfa-level-2\\\/problems-in-backtesting\\\/\",\"url\":\"https:\\\/\\\/analystprep.com\\\/study-notes\\\/cfa-level-2\\\/problems-in-backtesting\\\/\",\"name\":\"Problems in Backtesting and Biases in Data\",\"isPartOf\":{\"@id\":\"https:\\\/\\\/analystprep.com\\\/study-notes\\\/#website\"},\"datePublished\":\"2021-11-09T20:30:01+00:00\",\"dateModified\":\"2026-03-26T08:14:43+00:00\",\"author\":{\"@id\":\"https:\\\/\\\/analystprep.com\\\/study-notes\\\/#\\\/schema\\\/person\\\/7002f30d8f174958802c1c30b167eaf5\"},\"description\":\"Learn common backtesting problems, including survivorship bias, look-ahead bias, and data snooping, and how they affect strategy results.\",\"breadcrumb\":{\"@id\":\"https:\\\/\\\/analystprep.com\\\/study-notes\\\/cfa-level-2\\\/problems-in-backtesting\\\/#breadcrumb\"},\"inLanguage\":\"en-US\",\"potentialAction\":[{\"@type\":\"ReadAction\",\"target\":[\"https:\\\/\\\/analystprep.com\\\/study-notes\\\/cfa-level-2\\\/problems-in-backtesting\\\/\"]}]},{\"@type\":\"BreadcrumbList\",\"@id\":\"https:\\\/\\\/analystprep.com\\\/study-notes\\\/cfa-level-2\\\/problems-in-backtesting\\\/#breadcrumb\",\"itemListElement\":[{\"@type\":\"ListItem\",\"position\":1,\"name\":\"Home\",\"item\":\"https:\\\/\\\/analystprep.com\\\/study-notes\\\/\"},{\"@type\":\"ListItem\",\"position\":2,\"name\":\"Problems in Backtesting\"}]},{\"@type\":\"WebSite\",\"@id\":\"https:\\\/\\\/analystprep.com\\\/study-notes\\\/#website\",\"url\":\"https:\\\/\\\/analystprep.com\\\/study-notes\\\/\",\"name\":\"CFA, FRM, and Actuarial Exams Study Notes\",\"description\":\"Question Bank and Study Notes for the CFA, FRM, and Actuarial exams\",\"potentialAction\":[{\"@type\":\"SearchAction\",\"target\":{\"@type\":\"EntryPoint\",\"urlTemplate\":\"https:\\\/\\\/analystprep.com\\\/study-notes\\\/?s={search_term_string}\"},\"query-input\":{\"@type\":\"PropertyValueSpecification\",\"valueRequired\":true,\"valueName\":\"search_term_string\"}}],\"inLanguage\":\"en-US\"},{\"@type\":\"Person\",\"@id\":\"https:\\\/\\\/analystprep.com\\\/study-notes\\\/#\\\/schema\\\/person\\\/7002f30d8f174958802c1c30b167eaf5\",\"name\":\"Irene R\",\"image\":{\"@type\":\"ImageObject\",\"inLanguage\":\"en-US\",\"@id\":\"https:\\\/\\\/secure.gravatar.com\\\/avatar\\\/33caf1e1bcb63ee970b36351f165c7bc714b19614993ab9c2c8bf36273b7df48?s=96&d=mm&r=g\",\"url\":\"https:\\\/\\\/secure.gravatar.com\\\/avatar\\\/33caf1e1bcb63ee970b36351f165c7bc714b19614993ab9c2c8bf36273b7df48?s=96&d=mm&r=g\",\"contentUrl\":\"https:\\\/\\\/secure.gravatar.com\\\/avatar\\\/33caf1e1bcb63ee970b36351f165c7bc714b19614993ab9c2c8bf36273b7df48?s=96&d=mm&r=g\",\"caption\":\"Irene R\"},\"url\":\"https:\\\/\\\/analystprep.com\\\/study-notes\\\/author\\\/irene\\\/\"}]}<\/script>\n<meta property=\"og:video\" content=\"https:\/\/www.youtube.com\/embed\/FqQEumbr5is\" \/>\n<meta property=\"og:video:type\" content=\"text\/html\" \/>\n<meta property=\"og:video:duration\" content=\"2400\" \/>\n<meta property=\"og:video:width\" content=\"480\" \/>\n<meta property=\"og:video:height\" content=\"270\" \/>\n<meta property=\"ya:ovs:adult\" content=\"false\" \/>\n<meta property=\"ya:ovs:upload_date\" content=\"2021-11-09T20:30:01+00:00\" \/>\n<meta property=\"ya:ovs:allow_embed\" content=\"true\" \/>\n<!-- \/ Yoast SEO plugin. -->","yoast_head_json":{"title":"Problems in Backtesting and Biases in Data","description":"Learn common backtesting problems, including survivorship bias, look-ahead bias, and data snooping, and how they affect strategy results.","robots":{"index":"index","follow":"follow","max-snippet":"max-snippet:-1","max-image-preview":"max-image-preview:large","max-video-preview":"max-video-preview:-1"},"canonical":"https:\/\/analystprep.com\/study-notes\/cfa-level-2\/problems-in-backtesting\/","og_locale":"en_US","og_type":"article","og_title":"Problems in Backtesting and Biases in Data","og_description":"Learn common backtesting problems, including survivorship bias, look-ahead bias, and data snooping, and how they affect strategy results.","og_url":"https:\/\/analystprep.com\/study-notes\/cfa-level-2\/problems-in-backtesting\/","og_site_name":"CFA, FRM, and Actuarial Exams Study Notes","article_published_time":"2021-11-09T20:30:01+00:00","article_modified_time":"2026-03-26T08:14:43+00:00","author":"Irene R","twitter_card":"summary_large_image","twitter_misc":{"Written by":"Irene R","Est. reading time":"3 minutes"},"schema":{"@context":"https:\/\/schema.org","@graph":[{"@type":"Article","@id":"https:\/\/analystprep.com\/study-notes\/cfa-level-2\/problems-in-backtesting\/#article","isPartOf":{"@id":"https:\/\/analystprep.com\/study-notes\/cfa-level-2\/problems-in-backtesting\/"},"author":{"name":"Irene R","@id":"https:\/\/analystprep.com\/study-notes\/#\/schema\/person\/7002f30d8f174958802c1c30b167eaf5"},"headline":"Problems in Backtesting","datePublished":"2021-11-09T20:30:01+00:00","dateModified":"2026-03-26T08:14:43+00:00","mainEntityOfPage":{"@id":"https:\/\/analystprep.com\/study-notes\/cfa-level-2\/problems-in-backtesting\/"},"wordCount":579,"keywords":["CFA-level-2","Portfolio management","Reading 41: Backtesting and Simulation"],"articleSection":["CFA Level II Study Notes","Portfolio Management"],"inLanguage":"en-US"},{"@type":"WebPage","@id":"https:\/\/analystprep.com\/study-notes\/cfa-level-2\/problems-in-backtesting\/","url":"https:\/\/analystprep.com\/study-notes\/cfa-level-2\/problems-in-backtesting\/","name":"Problems in Backtesting and Biases in Data","isPartOf":{"@id":"https:\/\/analystprep.com\/study-notes\/#website"},"datePublished":"2021-11-09T20:30:01+00:00","dateModified":"2026-03-26T08:14:43+00:00","author":{"@id":"https:\/\/analystprep.com\/study-notes\/#\/schema\/person\/7002f30d8f174958802c1c30b167eaf5"},"description":"Learn common backtesting problems, including survivorship bias, look-ahead bias, and data snooping, and how they affect strategy results.","breadcrumb":{"@id":"https:\/\/analystprep.com\/study-notes\/cfa-level-2\/problems-in-backtesting\/#breadcrumb"},"inLanguage":"en-US","potentialAction":[{"@type":"ReadAction","target":["https:\/\/analystprep.com\/study-notes\/cfa-level-2\/problems-in-backtesting\/"]}]},{"@type":"BreadcrumbList","@id":"https:\/\/analystprep.com\/study-notes\/cfa-level-2\/problems-in-backtesting\/#breadcrumb","itemListElement":[{"@type":"ListItem","position":1,"name":"Home","item":"https:\/\/analystprep.com\/study-notes\/"},{"@type":"ListItem","position":2,"name":"Problems in Backtesting"}]},{"@type":"WebSite","@id":"https:\/\/analystprep.com\/study-notes\/#website","url":"https:\/\/analystprep.com\/study-notes\/","name":"CFA, FRM, and Actuarial Exams Study Notes","description":"Question Bank and Study Notes for the CFA, FRM, and Actuarial exams","potentialAction":[{"@type":"SearchAction","target":{"@type":"EntryPoint","urlTemplate":"https:\/\/analystprep.com\/study-notes\/?s={search_term_string}"},"query-input":{"@type":"PropertyValueSpecification","valueRequired":true,"valueName":"search_term_string"}}],"inLanguage":"en-US"},{"@type":"Person","@id":"https:\/\/analystprep.com\/study-notes\/#\/schema\/person\/7002f30d8f174958802c1c30b167eaf5","name":"Irene R","image":{"@type":"ImageObject","inLanguage":"en-US","@id":"https:\/\/secure.gravatar.com\/avatar\/33caf1e1bcb63ee970b36351f165c7bc714b19614993ab9c2c8bf36273b7df48?s=96&d=mm&r=g","url":"https:\/\/secure.gravatar.com\/avatar\/33caf1e1bcb63ee970b36351f165c7bc714b19614993ab9c2c8bf36273b7df48?s=96&d=mm&r=g","contentUrl":"https:\/\/secure.gravatar.com\/avatar\/33caf1e1bcb63ee970b36351f165c7bc714b19614993ab9c2c8bf36273b7df48?s=96&d=mm&r=g","caption":"Irene R"},"url":"https:\/\/analystprep.com\/study-notes\/author\/irene\/"}]},"og_video":"https:\/\/www.youtube.com\/embed\/FqQEumbr5is","og_video_type":"text\/html","og_video_duration":"2400","og_video_width":"480","og_video_height":"270","ya_ovs_adult":"false","ya_ovs_upload_date":"2021-11-09T20:30:01+00:00","ya_ovs_allow_embed":"true"},"_links":{"self":[{"href":"https:\/\/analystprep.com\/study-notes\/wp-json\/wp\/v2\/posts\/22822","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/analystprep.com\/study-notes\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/analystprep.com\/study-notes\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/analystprep.com\/study-notes\/wp-json\/wp\/v2\/users\/5"}],"replies":[{"embeddable":true,"href":"https:\/\/analystprep.com\/study-notes\/wp-json\/wp\/v2\/comments?post=22822"}],"version-history":[{"count":19,"href":"https:\/\/analystprep.com\/study-notes\/wp-json\/wp\/v2\/posts\/22822\/revisions"}],"predecessor-version":[{"id":42935,"href":"https:\/\/analystprep.com\/study-notes\/wp-json\/wp\/v2\/posts\/22822\/revisions\/42935"}],"wp:attachment":[{"href":"https:\/\/analystprep.com\/study-notes\/wp-json\/wp\/v2\/media?parent=22822"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/analystprep.com\/study-notes\/wp-json\/wp\/v2\/categories?post=22822"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/analystprep.com\/study-notes\/wp-json\/wp\/v2\/tags?post=22822"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}