P-Value

P-values are used in hypothesis testing to judge whether results are likely to reflect a real effect or could easily have happened by chance.

For example, suppose a fund manager claims their strategy beats the market. Their returns over several years are higher than the market's, and a statistical test gives a p-value of 0.03. This means that if the strategy actually had no advantage, there would be only about a 3% chance of seeing outperformance this large or larger purely by luck.

A small p-value suggests the observed result would be unusual if there were no real effect, so it provides evidence against the null hypothesis. Researchers often use a threshold of 0.05: results with a p-value below this are called statistically significant.

P-values are widely misunderstood. A p-value is not the probability that the null hypothesis is true, nor the probability that the result happened by chance. It also says nothing about how large or important an effect is. A tiny, unimportant effect can have a very small p-value in a large data set. Testing many ideas and reporting only those with low p-values can also produce misleading conclusions.