Data Snooping

If enough ideas are tested on the same data, some will look successful purely by luck. Data snooping, also called data mining or data dredging in this sense, happens when researchers or investors keep searching until they find a pattern, then present it as if it were their only hypothesis.

For example, if an analyst tests 100 different trading rules on historical data, around five might appear statistically significant at the 5% level by chance alone, even if none has any real predictive power.

Data snooping is a serious problem in finance, where the same historical market data are analysed by many researchers. Many published patterns in returns have weakened or disappeared after discovery, which may partly reflect data snooping.

Ways to guard against it include setting out hypotheses before testing them, testing results on new data not used in the search, adjusting statistical tests for the number of hypotheses tried, requiring an economic explanation for any pattern and being sceptical of results that seem too good to be true.