What is Walk Forward Analysis and which are the Good Practices while Backtesting

Backtest good practices and how to avoid bad performances

Backtests are an excellent tool when evaluating a trading or investing model. A good backtest system is not survivalship biased and it allows us to apply a trading or investing system to historical data to verify how it would have performed during a specified time period. However, many times one can think that a great system was designed, just to find out in out-of-sample that the performance is not correlated to simulation at all. How to avoid that?

Some basic rules, like backtesting an extended period of time (10, 20 or 40 years), where the economy was at different stages, is a starting point, but not enough. One can always data mine a given period and curve-fit it to work well under the past, which is known. But a robust system must be able to continue to perform well in the future, which is unknown, and in most market conditions. It must even have a better adjusted-risk return than the market. That’s why we suggest to always run a walk forward analysis. It’s one of the best ways to evaluate the robustness of a model and forecast it’s future performances.

What is a Walk Forward Analysis and how to use it

One of the best ways to validate the strength of a backtest is to perform walk forward analysis:

“start your backtest with a small period (for example, 10 years), record the performance info and keep adding additional periods to enlarge the total period being tested. Then compare the performance of these different cumulative periods with a full backtest from the initial date until today”.

For example, test from 2000 to 2003, then from 2000 to 2006, then from 2000 to 2009 until 2000 to 2017. Any other approach or combination of backtests is good just like the one presented in the example. The milestone to follow is always the idea of optimising the trading system with a subset of data (in sample) and then run the whole backtest to have a realistic picture of real money trading.

Indeed walk forward analysis means optimising a trading model by using the in sample data and run again the backtest with these parameters fixed, by using another segment of data (out of sample). These two segments of data do not overlap each other and usually the out of sample data are the most recent days.

Thanks to walk forward analysis it’s possible to create robust systems. Walk forward analysis gives the chance to use past data in two ways: thanks to the in sample data, it is possible to find an optimised set of parameters for the trading strategy and thanks to the out of sample ones it’s possible to forward test those parameters. That’s exactly a test on the future by using past data. Walk forward analysis is really important to avoid overfitting, that’s the over-optimization of a model without testing it with data simulating future prices.

How to run walk forward analysis in Metatrader

It is not possible to set your Metatrader to run walk forward analysis, so it’s necessary to set up some steps to simulate a walk forward analysis even for those using Metatrader.

First of all you need to load data in your Metatrader, to be sure to have at-least 10 years of intraday historical data to run a strong backtest.

If you need to know how to load data in Metatrader, here we explain how to do that.

Now that you have all your data loaded, you just need to choose one of your Expert Advisors and go to the testing area. Now flag the “Use Date” button and pick up a date range not bigger than half of your historical dataset. For example if you have 20 years of data, just set 10 years of backtesting. After having flagged even the “Optimization” button, click “Start” and run your backtest to optimise it.

Once you got your best parameters optimised over 10 years of data, you can test again with the last 10 years of historical data without the flag in “Optimization", but with the input parameters of your Expert Advisors set according to the result of the previous backtest, in order to test the parameters without over-optimising your model and not incurring in overfitting.

Now you can run again the backtest with the parameters over the whole period under analysis and compose a strong equity line with a high chance to have similar performances on real trading.

So now you know how to perform a walk forward analysis to validate robustness of a trading system. Furthermore, now you know that a trading system has to be optimized to explore the inefficiencies of the market, but it shouldn’t be over-optimized. That’s one of the issues that leads to over-fitting.

overfitting of a model, what is and how to avoid it

How to avoid over-fitting when designing a model, so it can continue to perform like backtests when out-of-sample?

Many designers think that testing a rule at a time would address this issue - keep testing and checking the results until a system doesn’t perform as good in backtest. However, this is a dangerous approach, as the focus is on process.

A good model must be based on a financially sound idea, not just what worked recently (as in what variables gave a good performance). It should factor metrics that have proven to work for 20, 30, 40 years, with data to support it. There is a lot of research out there elaborating on those principles, and that should be core of any quant investing system. 

Data mining/curve fitting has nothing at all to do with process. It has to do with your intention. If your intention is to build a model based solely on what worked over your sample period(s), then you are curve fitting/data mining. If you build a model based on what you expect, based on logic and common sense, to work in the future, then you are not curve fitting/data mining.

For example:

start with Rule 1: PE<20

Add Rule 2: ROE%>15

I don’t need to test Rule 2 before adding it. In fact, I’m better off not testing it separately. If I test Rule 2 separately, there’s a good chance it will fail. There is no logical reason why shares of companies with strong ROE should do well. Perhaps they are overvalued. Perhaps ROE is going to fall off a cliff. Perhaps they are taking on more debt than they can comfortably afford in order to leverage up a rotten ROA.

If I pair Rule 2 with Rule 1 right off the bat, everything has changed and both rules gain strength. Rule 1 becomes stronger because I’m not limiting my search to low P/E stocks but low P/E on shares of better-quality companies. Rule 2 becomes stronger because I’m following Greenblatt principles; limiting my consideration of better quality companies to those whose shares are reasonably priced.

Therefore:

Rule 1 is one investment idea.

Rule 2 is a completely different investment idea.

Rule 1 + Rule 2 is a completely different idea that has nothing in common with the other two.

I expect rule 1 and 2 to do well together. So I backtest and see how it did in different scenarios. It's different than an approach focused on "let's add this here and see what happens". 

Therefore, a basic quant investing principle is to never, never, never test any rule or factor on it own unless you expect to use that with real money for yourself. Only test full models. If you test one at a time, you are working with a mindset that put you away from where you should be focusing - on a financially sound idea - and testing one at a time will likely cause you to miss out on important things you should be considering.

I hope this article will inspire you to create great trading / investing models.