Statistical Significance

Intermediate

Statistical significance describes a result that is unlikely to have occurred by chance alone if there were no real effect, usually judged by whether its p-value falls below a chosen threshold, such as 0.05.

When researchers find a difference or relationship in data, they need to know whether it reflects something real or just random variation. A result is called statistically significant when the probability of seeing it, assuming no real effect, is low enough.

For example, if an analysis finds that companies with a certain characteristic earned higher returns, with a p-value of 0.01, the result would be considered statistically significant at the 5% level, and also at the 1% level.

Statistical significance is not the same as practical importance. With very large data sets, tiny effects can be statistically significant but too small to matter. A trading strategy that beats the market by 0.01% a year might be statistically significant but would be wiped out by trading costs. Conversely, an important effect might not reach statistical significance in a small sample.

Significance can also be misleading if many different tests are run, because some will appear significant purely by chance. This problem, sometimes called data mining or p-hacking, is a particular concern in investment research, where many strategies can be tested on the same historical data.

For worked examples and full lessons, visit notesbysara.org.