Backtesting

Backtesting is used in two main ways. Investors and traders backtest strategies to see how they would have performed in the past. Risk managers backtest risk models, such as Value at Risk, to check whether they predict losses accurately.

For example, a bank's 99% one-day Value at Risk model should be exceeded on about 1% of trading days. Over 250 days, about 2.5 exceptions would be expected. If actual losses exceeded the VaR estimate on 10 days, the model would be underestimating risk. Banking regulators have long used backtesting results, grouping them into green, yellow and red zones, to decide whether banks must hold extra capital.

For investment strategies, backtesting can help reject poor ideas, but it can be misleading. Testing many strategies on the same data may find one that looked good by chance, a problem called data snooping or overfitting. Results can also be distorted by look-ahead bias, survivorship bias and ignoring trading costs.

Good backtesting uses realistic assumptions, includes costs and tests the strategy on separate data it was not designed on.