Statistical significance refers to the use of a sample to carry out a statistical test meant to reveal any significant deviation from the stated null hypothesis. We then decide whether to reject or not to reject the null hypothesis. Economic significance entails not just the statistical significance but also the economic effect inherent in the decision made after data analysis and testing.
The need to separate the two arises from the fact that some statistical results may be significant while written down on paper but not economically meaningful. The difference from the hypothesized value may carry some statistical weight but lack economic feasibility, making implementation of the results very unlikely. Perhaps an illustrative example can help you gain a deeper understanding of the two concepts:
Statistical Significance and Economic Feasibility Explained with an Example
A well established pharmaceutical company wishes to assess the effectiveness of a newly developed drug before commercialization. The company’s board of directors commissions a pilot test. The drug is administered to a few patients who haven’t been prescribed any of the existing drugs. A statistical test follows and reveals a significant decrease in the average number of days taken before full recovery. The company considers the evidence sufficient to conclude that the new drug is more effective compared to the existing alternatives.
However, production of the new drug is significantly more expensive because of the scarcity of the active ingredient. Furthermore, the company would have to engage in a year-long lobbying exercise to convince the Food and Drug Administration and the general public that the drug is indeed an improvement to the existing brands. At the end of the day, the management decides to delay commercialization of the drug because of the higher production and introduction costs.
Other factors that may affect the economic feasibility of statistical results include:
- Tax: Financial institutions generally avoid projects that may increase the tax payable. Shareholders always eye increasing returns on investment from one year to the next.
- Risk: We may have a statistically significant project that is too risky. Those that require the setting aside of huge funds in the long term are particularly very risky. In fact, the additional risk is excluded from statistical tests.
Evidence of returns based solely on statistical analysis may not be enough to guarantee implementation. In particular, large samples may produce results that have high statistical significance but very low applicability.
Reading 12 LOS 12e:
Distinguish between a statistical result and an economically meaningful result.