P for ‘Pointless’?

Take a look at any statistical analysis. Chances are, you’d see an innocuous little statistic called p-value in there. Indeed, p-value has its own staunch set of followers. To them, and to most of us, p-value is that discriminatory force that separates the true from the untrue, the right hypothesis from the wrong hypothesis.

How does the p-value have such great power? It all boils down to a sacrosanct value - 0.05. A value of 0.05, technically, means that there is a 5% chance of seeing a value that is seen in the analysis due to random chance. Ergo, there is a 95% chance that the result that is seen in the analysis is not the result of a random chance.

This revered value has its origins in the work of a gigantic figure in statistics - Ronald A. Fisher. Fisher had developed the p-value as a shield against analysis results basing themselves on random chance. In what is an ironical twist, the same value is now used to argue for results based on random chance.

This has given rise to a new data mining “skill” that is lovingly called p-hacking. It uses the prowess of data mining to discover hidden patterns in data. Only, it discovers segments and patterns in the data where a statistically significant p-value can be reported. Once the data has been identified, it is only a matter of quick work to frame a hypothesis and report it.

What is wrong with such p-hacking? Everything. It turns every single good practice of statistics on its head. For starters, statistical analysis is supposed to start with hypothesis and then the data is analysed to look for support for the hypothesis. The reversed way is surely flawed.Widespread misuse had forced the American Statistical Association (ASA) to release a statement on p-values earlier this year. Perhaps more importantly, ASA also released six principles that govern correct use and interpretation of the p-value. The principles are the following:

ASA’s six principles are the following:

1. P-values can indicate how incompatible the data are with a specified statistical model.
2. P-values do not measure the probability that the studied hypothesis is true, or the probability that the data were produced by random chance alone.
3. Scientific conclusions and business or policy decisions should not be based only on whether a p-value passes a specific threshold.
4. Proper inference requires full reporting and transparency.
5. A p-value, or statistical significance, does not measure the size of an effect or the importance of a result.
6. By itself, a p-value does not provide a good measure of evidence regarding a model or hypothesis. 

There is nothing inherently wrong with p-value. It is upon us, the users, to not let it go down to becoming pointless with our repeated misuse.

About the Author 

Soumyadip Pal is a retail analytics professional and a passionate educator with more than 8 years in the industry and more than 7 years in the academia, currently working as a consultant with Manipal Prolearn.

Written By

Soumyadip Pal