{"id":33624,"date":"2021-10-07T08:11:03","date_gmt":"2021-10-07T08:11:03","guid":{"rendered":"https:\/\/analystprep.com\/cfa-level-1-exam\/?p=33624"},"modified":"2025-03-27T09:53:30","modified_gmt":"2025-03-27T09:53:30","slug":"significance-of-a-test-in-the-context-of-multiple-tests","status":"publish","type":"post","link":"https:\/\/analystprep.com\/cfa-level-1-exam\/quantitative-methods\/significance-of-a-test-in-the-context-of-multiple-tests\/","title":{"rendered":"Significance of a Test in the Context of Multiple Tests"},"content":{"rendered":"\n<p><iframe loading=\"lazy\" src=\"\/\/www.youtube.com\/embed\/twTTWMuEGx8\" width=\"611\" height=\"343\" allowfullscreen=\"allowfullscreen\"><\/iframe><\/p>\n<p>Type I error occurs when we reject a true null hypothesis. Type I error is also referred to as a <strong>false positive<\/strong> result because the null is true, but it is rejected (the false positive). The expected part of a false positive is the <strong>false discovery rate<\/strong> (FDR). For example, if you run 60 tests and use a 5% level of significance, you will get three false positives on average (60 \u00d7 5%). This issue is called the multiple testing problem. This problem can be overcome by using a <strong>false discovery approach<\/strong>.<\/p>\n<p>In the false discovery approach, <em>p<\/em>-values are ranked from various tests, from the lowest to the highest. A comparison is then made, starting with the lowest <em>p<\/em>-value (with k = 1), p(1):<\/p>\n<p>$$<br \/>\\mathrm{p}(1) \\leq \\alpha \\frac{\\text { Rank of } \\mathrm{i}}{\\text { Number of tests }}<br \/>$$<\/p>\n<p>We repeat this comparison until the highest-ranked p(k) for which this condition holds is found.<\/p>\n<h4><strong>Example: Using Benjamini and Hochberg (BH) Criteria<\/strong><\/h4>\n<p>Suppose we hypothesize that the population mean is equal to 25%. The sampling process is repeated 30 times, and 30 sample statistics are calculated. Out of the 30 statistics, the table given below reflects the lowest five <em>p<\/em>-values for five statistics. The level of significance is 5%.<\/p>\n<p>$$<br \/>\\begin{array}{c|c|c|c}<br \/>(1) &amp; (2) &amp; (3) &amp; (4) \\\\<br \/>\\hline \\textbf { P=value } &amp; \\begin{array}{c}<br \/>\\textbf { Rank of the p- } \\\\<br \/>\\textbf { value (lowest } \\\\<br \/>\\textbf { to highest) }<br \/>\\end{array} &amp; \\alpha \\frac{\\textbf { Rank of i }}{\\textbf { Number of tests }} &amp; \\begin{array}{c}<br \/>\\textbf { Is the value in } \\\\<br \/>\\textbf { (1)? } \\leq \\textbf { value in } \\\\<br \/>\\textbf { (3)? }<br \/>\\end{array} \\\\<br \/>\\hline 0.03 &amp; 2 &amp; \\begin{array}{c}<br \/>0.05 \\times\\left(\\frac{2}{30}\\right)=<br \/>0.0033<br \/>\\end{array} &amp; \\text { No } \\\\<br \/>\\hline 0.15 &amp; 4 &amp; 0.05 \\times \\left(\\frac{4 }{ 30}\\right)= 0.0067 &amp; \\text { No } \\\\<br \/>\\hline 0.27 &amp; 5 &amp; 0.05 \\times \\left(\\frac{5}{30}\\right)= 0.0083 &amp; \\text { No } \\\\<br \/>\\hline 0.08 &amp; 3 &amp; 0.05\\times \\left(\\frac{3}{ 30}\\right)=\u00a00.005 &amp; \\text { No } \\\\<br \/>\\hline 0.001 &amp; 1 &amp; 0.05 \\times \\left(\\frac{1}{30}\\right)= \u00a00.0017 &amp; \\text { Yes } \\\\<br \/>\\end{array}<br \/>$$<\/p>\n<p>The above results show that using a significance level of 5%, based only on <em>p<\/em>-values, we would have two tests in which the null hypothesis would be rejected since both 0.001 and 0.03 are smaller than 0.05. In contrast, using the BH criteria, we would have only one significant test since there is only one row where the value in (3) \u2264 the value in (1).<\/p>\n<p>There is a high probability of false positive results when:<\/p>\n<ul>\n<li>The sample size is small or very large.<\/li>\n<li>The power of the test is low.<\/li>\n<li>The test is run multiple times (leading to the risk of data snooping).<\/li>\n<\/ul>\n","protected":false},"excerpt":{"rendered":"","protected":false},"author":15,"featured_media":0,"comment_status":"closed","ping_status":"closed","sticky":false,"template":"","format":"standard","meta":{"_acf_changed":false,"footnotes":""},"categories":[2],"tags":[],"class_list":["post-33624","post","type-post","status-publish","format-standard","hentry","category-quantitative-methods","blog-post","no-post-thumbnail","animate"],"acf":[],"yoast_head":"<!-- This site is optimized with the Yoast SEO plugin v26.9 - https:\/\/yoast.com\/product\/yoast-seo-wordpress\/ -->\n<title>Multiple Hypothesis Testing Significance | AnalystPrep<\/title>\n<meta name=\"description\" content=\"Learn how to assess test significance in multiple hypothesis testing using false discovery rate and p-value ranking. Key CFA Level 1 quantitative concept.\" \/>\n<meta name=\"robots\" content=\"index, follow, max-snippet:-1, max-image-preview:large, max-video-preview:-1\" \/>\n<link rel=\"canonical\" href=\"https:\/\/analystprep.com\/cfa-level-1-exam\/quantitative-methods\/significance-of-a-test-in-the-context-of-multiple-tests\/\" \/>\n<meta property=\"og:locale\" content=\"en_US\" \/>\n<meta property=\"og:type\" content=\"article\" \/>\n<meta property=\"og:title\" content=\"Multiple Hypothesis Testing Significance | AnalystPrep\" \/>\n<meta property=\"og:description\" content=\"Learn how to assess test significance in multiple hypothesis testing using false discovery rate and p-value ranking. Key CFA Level 1 quantitative concept.\" \/>\n<meta property=\"og:url\" content=\"https:\/\/analystprep.com\/cfa-level-1-exam\/quantitative-methods\/significance-of-a-test-in-the-context-of-multiple-tests\/\" \/>\n<meta property=\"og:site_name\" content=\"AnalystPrep | CFA\u00ae Exam Study Notes\" \/>\n<meta property=\"article:published_time\" content=\"2021-10-07T08:11:03+00:00\" \/>\n<meta property=\"article:modified_time\" content=\"2025-03-27T09:53:30+00:00\" \/>\n<meta name=\"author\" content=\"Kosikos Tuitoek\" \/>\n<meta name=\"twitter:card\" content=\"summary_large_image\" \/>\n<meta name=\"twitter:label1\" content=\"Written by\" \/>\n\t<meta name=\"twitter:data1\" content=\"Kosikos Tuitoek\" \/>\n\t<meta name=\"twitter:label2\" content=\"Est. reading time\" \/>\n\t<meta name=\"twitter:data2\" content=\"2 minutes\" \/>\n<script type=\"application\/ld+json\" class=\"yoast-schema-graph\">{\"@context\":\"https:\/\/schema.org\",\"@graph\":[{\"@type\":\"Article\",\"@id\":\"https:\/\/analystprep.com\/cfa-level-1-exam\/quantitative-methods\/significance-of-a-test-in-the-context-of-multiple-tests\/#article\",\"isPartOf\":{\"@id\":\"https:\/\/analystprep.com\/cfa-level-1-exam\/quantitative-methods\/significance-of-a-test-in-the-context-of-multiple-tests\/\"},\"author\":{\"name\":\"Kosikos Tuitoek\",\"@id\":\"https:\/\/analystprep.com\/cfa-level-1-exam\/#\/schema\/person\/73df713e3b6e82ee139e1eff20cebe20\"},\"headline\":\"Significance of a Test in the Context of Multiple Tests\",\"datePublished\":\"2021-10-07T08:11:03+00:00\",\"dateModified\":\"2025-03-27T09:53:30+00:00\",\"mainEntityOfPage\":{\"@id\":\"https:\/\/analystprep.com\/cfa-level-1-exam\/quantitative-methods\/significance-of-a-test-in-the-context-of-multiple-tests\/\"},\"wordCount\":418,\"articleSection\":[\"Quantitative Methods\"],\"inLanguage\":\"en-US\"},{\"@type\":\"WebPage\",\"@id\":\"https:\/\/analystprep.com\/cfa-level-1-exam\/quantitative-methods\/significance-of-a-test-in-the-context-of-multiple-tests\/\",\"url\":\"https:\/\/analystprep.com\/cfa-level-1-exam\/quantitative-methods\/significance-of-a-test-in-the-context-of-multiple-tests\/\",\"name\":\"Multiple Hypothesis Testing Significance | AnalystPrep\",\"isPartOf\":{\"@id\":\"https:\/\/analystprep.com\/cfa-level-1-exam\/#website\"},\"datePublished\":\"2021-10-07T08:11:03+00:00\",\"dateModified\":\"2025-03-27T09:53:30+00:00\",\"author\":{\"@id\":\"https:\/\/analystprep.com\/cfa-level-1-exam\/#\/schema\/person\/73df713e3b6e82ee139e1eff20cebe20\"},\"description\":\"Learn how to assess test significance in multiple hypothesis testing using false discovery rate and p-value ranking. Key CFA Level 1 quantitative concept.\",\"breadcrumb\":{\"@id\":\"https:\/\/analystprep.com\/cfa-level-1-exam\/quantitative-methods\/significance-of-a-test-in-the-context-of-multiple-tests\/#breadcrumb\"},\"inLanguage\":\"en-US\",\"potentialAction\":[{\"@type\":\"ReadAction\",\"target\":[\"https:\/\/analystprep.com\/cfa-level-1-exam\/quantitative-methods\/significance-of-a-test-in-the-context-of-multiple-tests\/\"]}]},{\"@type\":\"BreadcrumbList\",\"@id\":\"https:\/\/analystprep.com\/cfa-level-1-exam\/quantitative-methods\/significance-of-a-test-in-the-context-of-multiple-tests\/#breadcrumb\",\"itemListElement\":[{\"@type\":\"ListItem\",\"position\":1,\"name\":\"Home\",\"item\":\"https:\/\/analystprep.com\/cfa-level-1-exam\/\"},{\"@type\":\"ListItem\",\"position\":2,\"name\":\"Significance of a Test in the Context of Multiple Tests\"}]},{\"@type\":\"WebSite\",\"@id\":\"https:\/\/analystprep.com\/cfa-level-1-exam\/#website\",\"url\":\"https:\/\/analystprep.com\/cfa-level-1-exam\/\",\"name\":\"AnalystPrep | CFA\u00ae Exam Study Notes\",\"description\":\"\",\"potentialAction\":[{\"@type\":\"SearchAction\",\"target\":{\"@type\":\"EntryPoint\",\"urlTemplate\":\"https:\/\/analystprep.com\/cfa-level-1-exam\/?s={search_term_string}\"},\"query-input\":{\"@type\":\"PropertyValueSpecification\",\"valueRequired\":true,\"valueName\":\"search_term_string\"}}],\"inLanguage\":\"en-US\"},{\"@type\":\"Person\",\"@id\":\"https:\/\/analystprep.com\/cfa-level-1-exam\/#\/schema\/person\/73df713e3b6e82ee139e1eff20cebe20\",\"name\":\"Kosikos Tuitoek\",\"image\":{\"@type\":\"ImageObject\",\"inLanguage\":\"en-US\",\"@id\":\"https:\/\/analystprep.com\/cfa-level-1-exam\/#\/schema\/person\/image\/\",\"url\":\"https:\/\/secure.gravatar.com\/avatar\/8260edaa3f7ba04cf6b536b3f7fd769007ecb789b3289ac0cc4c3ab8b3f7f061?s=96&d=mm&r=g\",\"contentUrl\":\"https:\/\/secure.gravatar.com\/avatar\/8260edaa3f7ba04cf6b536b3f7fd769007ecb789b3289ac0cc4c3ab8b3f7f061?s=96&d=mm&r=g\",\"caption\":\"Kosikos Tuitoek\"},\"url\":\"https:\/\/analystprep.com\/cfa-level-1-exam\/author\/kosikos-tuitoek-enockanalystprep-com\/\"}]}<\/script>\n<meta property=\"og:video\" content=\"https:\/\/www.youtube.com\/embed\/twTTWMuEGx8\" \/>\n<meta property=\"og:video:type\" content=\"text\/html\" \/>\n<meta property=\"og:video:duration\" content=\"3528\" \/>\n<meta property=\"og:video:width\" content=\"480\" \/>\n<meta property=\"og:video:height\" content=\"270\" \/>\n<meta property=\"ya:ovs:adult\" content=\"false\" \/>\n<meta property=\"ya:ovs:upload_date\" content=\"2021-10-07T08:11:03+00:00\" \/>\n<meta property=\"ya:ovs:allow_embed\" content=\"true\" \/>\n<!-- \/ Yoast SEO plugin. -->","yoast_head_json":{"title":"Multiple Hypothesis Testing Significance | AnalystPrep","description":"Learn how to assess test significance in multiple hypothesis testing using false discovery rate and p-value ranking. Key CFA Level 1 quantitative concept.","robots":{"index":"index","follow":"follow","max-snippet":"max-snippet:-1","max-image-preview":"max-image-preview:large","max-video-preview":"max-video-preview:-1"},"canonical":"https:\/\/analystprep.com\/cfa-level-1-exam\/quantitative-methods\/significance-of-a-test-in-the-context-of-multiple-tests\/","og_locale":"en_US","og_type":"article","og_title":"Multiple Hypothesis Testing Significance | AnalystPrep","og_description":"Learn how to assess test significance in multiple hypothesis testing using false discovery rate and p-value ranking. Key CFA Level 1 quantitative concept.","og_url":"https:\/\/analystprep.com\/cfa-level-1-exam\/quantitative-methods\/significance-of-a-test-in-the-context-of-multiple-tests\/","og_site_name":"AnalystPrep | CFA\u00ae Exam Study Notes","article_published_time":"2021-10-07T08:11:03+00:00","article_modified_time":"2025-03-27T09:53:30+00:00","author":"Kosikos Tuitoek","twitter_card":"summary_large_image","twitter_misc":{"Written by":"Kosikos Tuitoek","Est. reading time":"2 minutes"},"schema":{"@context":"https:\/\/schema.org","@graph":[{"@type":"Article","@id":"https:\/\/analystprep.com\/cfa-level-1-exam\/quantitative-methods\/significance-of-a-test-in-the-context-of-multiple-tests\/#article","isPartOf":{"@id":"https:\/\/analystprep.com\/cfa-level-1-exam\/quantitative-methods\/significance-of-a-test-in-the-context-of-multiple-tests\/"},"author":{"name":"Kosikos Tuitoek","@id":"https:\/\/analystprep.com\/cfa-level-1-exam\/#\/schema\/person\/73df713e3b6e82ee139e1eff20cebe20"},"headline":"Significance of a Test in the Context of Multiple Tests","datePublished":"2021-10-07T08:11:03+00:00","dateModified":"2025-03-27T09:53:30+00:00","mainEntityOfPage":{"@id":"https:\/\/analystprep.com\/cfa-level-1-exam\/quantitative-methods\/significance-of-a-test-in-the-context-of-multiple-tests\/"},"wordCount":418,"articleSection":["Quantitative Methods"],"inLanguage":"en-US"},{"@type":"WebPage","@id":"https:\/\/analystprep.com\/cfa-level-1-exam\/quantitative-methods\/significance-of-a-test-in-the-context-of-multiple-tests\/","url":"https:\/\/analystprep.com\/cfa-level-1-exam\/quantitative-methods\/significance-of-a-test-in-the-context-of-multiple-tests\/","name":"Multiple Hypothesis Testing Significance | AnalystPrep","isPartOf":{"@id":"https:\/\/analystprep.com\/cfa-level-1-exam\/#website"},"datePublished":"2021-10-07T08:11:03+00:00","dateModified":"2025-03-27T09:53:30+00:00","author":{"@id":"https:\/\/analystprep.com\/cfa-level-1-exam\/#\/schema\/person\/73df713e3b6e82ee139e1eff20cebe20"},"description":"Learn how to assess test significance in multiple hypothesis testing using false discovery rate and p-value ranking. Key CFA Level 1 quantitative concept.","breadcrumb":{"@id":"https:\/\/analystprep.com\/cfa-level-1-exam\/quantitative-methods\/significance-of-a-test-in-the-context-of-multiple-tests\/#breadcrumb"},"inLanguage":"en-US","potentialAction":[{"@type":"ReadAction","target":["https:\/\/analystprep.com\/cfa-level-1-exam\/quantitative-methods\/significance-of-a-test-in-the-context-of-multiple-tests\/"]}]},{"@type":"BreadcrumbList","@id":"https:\/\/analystprep.com\/cfa-level-1-exam\/quantitative-methods\/significance-of-a-test-in-the-context-of-multiple-tests\/#breadcrumb","itemListElement":[{"@type":"ListItem","position":1,"name":"Home","item":"https:\/\/analystprep.com\/cfa-level-1-exam\/"},{"@type":"ListItem","position":2,"name":"Significance of a Test in the Context of Multiple Tests"}]},{"@type":"WebSite","@id":"https:\/\/analystprep.com\/cfa-level-1-exam\/#website","url":"https:\/\/analystprep.com\/cfa-level-1-exam\/","name":"AnalystPrep | CFA\u00ae Exam Study Notes","description":"","potentialAction":[{"@type":"SearchAction","target":{"@type":"EntryPoint","urlTemplate":"https:\/\/analystprep.com\/cfa-level-1-exam\/?s={search_term_string}"},"query-input":{"@type":"PropertyValueSpecification","valueRequired":true,"valueName":"search_term_string"}}],"inLanguage":"en-US"},{"@type":"Person","@id":"https:\/\/analystprep.com\/cfa-level-1-exam\/#\/schema\/person\/73df713e3b6e82ee139e1eff20cebe20","name":"Kosikos Tuitoek","image":{"@type":"ImageObject","inLanguage":"en-US","@id":"https:\/\/analystprep.com\/cfa-level-1-exam\/#\/schema\/person\/image\/","url":"https:\/\/secure.gravatar.com\/avatar\/8260edaa3f7ba04cf6b536b3f7fd769007ecb789b3289ac0cc4c3ab8b3f7f061?s=96&d=mm&r=g","contentUrl":"https:\/\/secure.gravatar.com\/avatar\/8260edaa3f7ba04cf6b536b3f7fd769007ecb789b3289ac0cc4c3ab8b3f7f061?s=96&d=mm&r=g","caption":"Kosikos Tuitoek"},"url":"https:\/\/analystprep.com\/cfa-level-1-exam\/author\/kosikos-tuitoek-enockanalystprep-com\/"}]},"og_video":"https:\/\/www.youtube.com\/embed\/twTTWMuEGx8","og_video_type":"text\/html","og_video_duration":"3528","og_video_width":"480","og_video_height":"270","ya_ovs_adult":"false","ya_ovs_upload_date":"2021-10-07T08:11:03+00:00","ya_ovs_allow_embed":"true"},"_links":{"self":[{"href":"https:\/\/analystprep.com\/cfa-level-1-exam\/wp-json\/wp\/v2\/posts\/33624","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/analystprep.com\/cfa-level-1-exam\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/analystprep.com\/cfa-level-1-exam\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/analystprep.com\/cfa-level-1-exam\/wp-json\/wp\/v2\/users\/15"}],"replies":[{"embeddable":true,"href":"https:\/\/analystprep.com\/cfa-level-1-exam\/wp-json\/wp\/v2\/comments?post=33624"}],"version-history":[{"count":18,"href":"https:\/\/analystprep.com\/cfa-level-1-exam\/wp-json\/wp\/v2\/posts\/33624\/revisions"}],"predecessor-version":[{"id":38624,"href":"https:\/\/analystprep.com\/cfa-level-1-exam\/wp-json\/wp\/v2\/posts\/33624\/revisions\/38624"}],"wp:attachment":[{"href":"https:\/\/analystprep.com\/cfa-level-1-exam\/wp-json\/wp\/v2\/media?parent=33624"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/analystprep.com\/cfa-level-1-exam\/wp-json\/wp\/v2\/categories?post=33624"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/analystprep.com\/cfa-level-1-exam\/wp-json\/wp\/v2\/tags?post=33624"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}