Limited Time Offer: Save 10% on all 2022 Premium Study Packages with promo code: BLOG10 # Significance of a Test in the Context of Multiple Tests

Type I error occurs when we reject a true null hypothesis. Type I error is also referred to as a false positive result because the null is true, but it is rejected (the false positive). The expected part of a false positive is the false discovery rate (FDR). For example, if you run 60 tests and use a 5% level of significance, you will get three false positives on average (60 × 5%). This issue is called the multiple testing problem. This problem can be overcome by using a false discovery approach.

In the false discovery approach, p-values are ranked from various tests, from the lowest to the highest. A comparison is then made, starting with the lowest p-value (with k = 1), p(1):

$$\mathrm{p}(1) \leq \alpha \frac{\text { Rank of } \mathrm{i}}{\text { Number of tests }}$$

We repeat this comparison until the highest-ranked p(k) for which this condition holds is found.

#### Example: Using Benjamini and Hochberg (BH) Criteria

Suppose we hypothesize that the population mean is equal to 25%. The sampling process is repeated 30 times, and 30 sample statistics are calculated. Out of the 30 statistics, the table given below reflects the lowest five p-values for five statistics. The level of significance is 5%.

$$\begin{array}{c|c|c|c} (1) & (2) & (3) & (4) \\ \hline \textbf { P=value } & \begin{array}{c} \textbf { Rank of the p- } \\ \textbf { value (lowest } \\ \textbf { to highest) } \end{array} & \alpha \frac{\textbf { Rank of i }}{\textbf { Number of tests }} & \begin{array}{c} \textbf { Is the value in } \\ \textbf { (1)? } \leq \textbf { value in } \\ \textbf { (3)? } \end{array} \\ \hline 0.03 & 2 & \begin{array}{c} 0.05 \times\left(\frac{2}{30}\right)= 0.0033 \end{array} & \text { No } \\ \hline 0.15 & 4 & 0.05 \times \left(\frac{4 }{ 30}\right)= 0.0067 & \text { No } \\ \hline 0.27 & 5 & 0.05 \times \left(\frac{5}{30}\right)= 0.0083 & \text { No } \\ \hline 0.08 & 3 & 0.05\times \left(\frac{3}{ 30}\right)= 0.005 & \text { No } \\ \hline 0.001 & 1 & 0.05 \times \left(\frac{1}{30}\right)= 0.0017 & \text { Yes } \\ \end{array}$$

The above results show that using a significance level of 5%, based only on p-values, we would have two tests in which the null hypothesis would be rejected since both 0.001 and 0.03 are smaller than 0.05. In contrast, using the BH criteria, we would have only one significant test since there is only one row where the value in (3) ≤ the value in (1).

There is a high probability of false positive results when:

• The sample size is small or very large.
• The power of the test is low.
• The test is run multiple times (leading to the risk of data snooping).
Shop CFA® Exam Prep

Offered by AnalystPrep Level I
Level II
Level III
All Three Levels
Featured Shop FRM® Exam Prep FRM Part I
FRM Part II
FRM Part I & Part II
Learn with Us

Subscribe to our newsletter and keep up with the latest and greatest tips for success
Shop Actuarial Exams Prep Exam P (Probability)
Exam FM (Financial Mathematics)
Exams P & FM
Shop GMAT® Exam Prep Complete Course Sergio Torrico
2021-07-23
Excelente para el FRM 2 Escribo esta revisión en español para los hispanohablantes, soy de Bolivia, y utilicé AnalystPrep para dudas y consultas sobre mi preparación para el FRM nivel 2 (lo tomé una sola vez y aprobé muy bien), siempre tuve un soporte claro, directo y rápido, el material sale rápido cuando hay cambios en el temario de GARP, y los ejercicios y exámenes son muy útiles para practicar. diana
2021-07-17
So helpful. I have been using the videos to prepare for the CFA Level II exam. The videos signpost the reading contents, explain the concepts and provide additional context for specific concepts. The fun light-hearted analogies are also a welcome break to some very dry content. I usually watch the videos before going into more in-depth reading and they are a good way to avoid being overwhelmed by the sheer volume of content when you look at the readings. Kriti Dhawan
2021-07-16
A great curriculum provider. James sir explains the concept so well that rather than memorising it, you tend to intuitively understand and absorb them. Thank you ! Grateful I saw this at the right time for my CFA prep. nikhil kumar
2021-06-28
Very well explained and gives a great insight about topics in a very short time. Glad to have found Professor Forjan's lectures. Marwan
2021-06-22
Great support throughout the course by the team, did not feel neglected Benjamin anonymous
2021-05-10
I loved using AnalystPrep for FRM. QBank is huge, videos are great. Would recommend to a friend Daniel Glyn
2021-03-24
I have finished my FRM1 thanks to AnalystPrep. And now using AnalystPrep for my FRM2 preparation. Professor Forjan is brilliant. He gives such good explanations and analogies. And more than anything makes learning fun. A big thank you to Analystprep and Professor Forjan. 5 stars all the way! michael walshe
2021-03-18
Professor James' videos are excellent for understanding the underlying theories behind financial engineering / financial analysis. The AnalystPrep videos were better than any of the others that I searched through on YouTube for providing a clear explanation of some concepts, such as Portfolio theory, CAPM, and Arbitrage Pricing theory. Watching these cleared up many of the unclarities I had in my head. Highly recommended.