Significance of a Test in the Context of Multiple Tests

Type I error occurs when we reject a true null hypothesis. Type I error is also referred to as a false positive result because the null is true, but it is rejected (the false positive). The expected part of a false positive is the false discovery rate (FDR). For example, if you run 60 tests and use a 5% level of significance, you will get three false positives on average (60 × 5%). This issue is called the multiple testing problem. This problem can be overcome by using a false discovery approach.

In the false discovery approach, p-values are ranked from various tests, from the lowest to the highest. A comparison is then made, starting with the lowest p-value (with k = 1), p(1):

$$
\mathrm{p}(1) \leq \alpha \frac{\text { Rank of } \mathrm{i}}{\text { Number of tests }}
$$

We repeat this comparison until the highest-ranked p(k) for which this condition holds is found.

Example: Using Benjamini and Hochberg (BH) Criteria

Suppose we hypothesize that the population mean is equal to 25%. The sampling process is repeated 30 times, and 30 sample statistics are calculated. Out of the 30 statistics, the table given below reflects the lowest five p-values for five statistics. The level of significance is 5%.

$$
\begin{array}{c|c|c|c}
(1) & (2) & (3) & (4) \\
\hline \textbf { P=value } & \begin{array}{c}
\textbf { Rank of the p- } \\
\textbf { value (lowest } \\
\textbf { to highest) }
\end{array} & \alpha \frac{\textbf { Rank of i }}{\textbf { Number of tests }} & \begin{array}{c}
\textbf { Is the value in } \\
\textbf { (1)? } \leq \textbf { value in } \\
\textbf { (3)? }
\end{array} \\
\hline 0.03 & 2 & \begin{array}{c}
0.05 \times\left(\frac{2}{30}\right)=
0.0033
\end{array} & \text { No } \\
\hline 0.15 & 4 & 0.05 \times \left(\frac{4 }{ 30}\right)= 0.0067 & \text { No } \\
\hline 0.27 & 5 & 0.05 \times \left(\frac{5}{30}\right)= 0.0083 & \text { No } \\
\hline 0.08 & 3 & 0.05\times \left(\frac{3}{ 30}\right)= 0.005 & \text { No } \\
\hline 0.001 & 1 & 0.05 \times \left(\frac{1}{30}\right)=  0.0017 & \text { Yes } \\
\end{array}
$$

The above results show that using a significance level of 5%, based only on p-values, we would have two tests in which the null hypothesis would be rejected since both 0.001 and 0.03 are smaller than 0.05. In contrast, using the BH criteria, we would have only one significant test since there is only one row where the value in (3) ≤ the value in (1).

There is a high probability of false positive results when:

• The sample size is small or very large.
• The power of the test is low.
• The test is run multiple times (leading to the risk of data snooping).