Curve Fitted Back Testing
Posted: Sun Nov 02, 2014 5:06 pm
Yes... you know you do it.
You have a new scheme. A crazy idea. A wild hair. You have to fit some parameters to the model to see what woulda coulda shoulda been.
But... a curve fit is precarious. You can make any data fit pretty much anything with even just a smidgen of math.
And yet... back testing has to be the first step to test any new idea - if nothing else, just to prove to yourself that it was a crazy, worthless idea.
So... what do you do to ensure that you aren't fooling yourself with regards to backtesting?
Here are my rules. I'd love to hear what others do.
1) Use the longest period of historical data that you can find. Find reliable simulations for assets that are too new to have direct data. Use *daily* data, not monthly. Daily swings can be huge and the max draw down numbers show up in the daily data, not the monthly. For me, I need to simulate the daily swings in order to evaluate whether or not I can stomach them in the future.
2) Limit your rules and conditions as much as possible. I.e. X assets, N rules for reallocation or triggers, etc. The fewer the better.
3) Fit the growth curve of your new scheme to an exponential line. The more it deviates from that line (either up or down) during different economic situations makes your model less robust.
4) If I am doing any curve fitting (i.e. optimizing parameters) I like to set a limit for maximum drawdown. My personal limit is 20%. That is about the most I can stomach before I start to wonder. So... If I have a bunch of data and I am trying to figure out the best asset allocation, I set my optimizer to maximize the CAGR, with the caveat that I never have a maximum draw down more than 20%. As an aside, the smaller this drawdown number, the more likely your total returns will be linear log scale - and thus implying a much more solid model.
5) Do sensitivity analysis on your model. If you have decided on something like 25/25/25/25 asset allocation with 15/35 reallocation bands, then also run some models with the reallocation bands higher and lower. Change the asset allocations by a few percentage points. Do the results still basically come out the same? That's great. If you end up seeing huge swings in the CAGR with minor changes in the rules... then your model won't be very robust in the future.
6) Always calculate the max draw down number of your simulation and the maximum number of days that you are down from a peak. Great that your model shows a 20% historical CAGR. But can you really handle 10 years without a new peak? Or a max draw down of 45%? Max draw down percent and duration are critical things to know about your model. And how do they change when you start doing sensitivity analysis on your parameters?
7) Look at the final data for your simulation and scroll through it day by day. Pretend that you don't know what the next line will show you. Look at the gains and losses each day. How do you feel? Have you lost all hope? Are you ecstatic? Replay your emotions line by line. Could you live with your life assets following such a model?
---
Obviously, the HBPP stands up to this kind of test very well. To me, this impies that the underlying theory and model are quite solid. I think that sort of thinking should go into any new scheme. What is the economic model that makes your idea good. How sensitive is it to unknown variations? How simple can you make it? When you have to pick allocation and trigger and threshold values (i.e. rebalance bands), how sensitive is your model to a 20% difference in your parameter value? (20/30 vs. 15/35). Because for damn sure, you didn't pick exactly right value to start with.
All of this testing and simulation can be done with some pretty basic intermediate Excel work. It is even better if you are comfortable using the solver function in Excel and sort of understand the limitations of global optimizers.
I would love to hear other suggestions for backtesting or checking crazy theories before putting your money on the line.
------
And... this post assumes that you are pursing a crazy idea based on some regular growth potential. If your crazy idea is of the "crap shoot" kind - well that is a different approach and a different post.
You have a new scheme. A crazy idea. A wild hair. You have to fit some parameters to the model to see what woulda coulda shoulda been.
But... a curve fit is precarious. You can make any data fit pretty much anything with even just a smidgen of math.
And yet... back testing has to be the first step to test any new idea - if nothing else, just to prove to yourself that it was a crazy, worthless idea.
So... what do you do to ensure that you aren't fooling yourself with regards to backtesting?
Here are my rules. I'd love to hear what others do.
1) Use the longest period of historical data that you can find. Find reliable simulations for assets that are too new to have direct data. Use *daily* data, not monthly. Daily swings can be huge and the max draw down numbers show up in the daily data, not the monthly. For me, I need to simulate the daily swings in order to evaluate whether or not I can stomach them in the future.
2) Limit your rules and conditions as much as possible. I.e. X assets, N rules for reallocation or triggers, etc. The fewer the better.
3) Fit the growth curve of your new scheme to an exponential line. The more it deviates from that line (either up or down) during different economic situations makes your model less robust.
4) If I am doing any curve fitting (i.e. optimizing parameters) I like to set a limit for maximum drawdown. My personal limit is 20%. That is about the most I can stomach before I start to wonder. So... If I have a bunch of data and I am trying to figure out the best asset allocation, I set my optimizer to maximize the CAGR, with the caveat that I never have a maximum draw down more than 20%. As an aside, the smaller this drawdown number, the more likely your total returns will be linear log scale - and thus implying a much more solid model.
5) Do sensitivity analysis on your model. If you have decided on something like 25/25/25/25 asset allocation with 15/35 reallocation bands, then also run some models with the reallocation bands higher and lower. Change the asset allocations by a few percentage points. Do the results still basically come out the same? That's great. If you end up seeing huge swings in the CAGR with minor changes in the rules... then your model won't be very robust in the future.
6) Always calculate the max draw down number of your simulation and the maximum number of days that you are down from a peak. Great that your model shows a 20% historical CAGR. But can you really handle 10 years without a new peak? Or a max draw down of 45%? Max draw down percent and duration are critical things to know about your model. And how do they change when you start doing sensitivity analysis on your parameters?
7) Look at the final data for your simulation and scroll through it day by day. Pretend that you don't know what the next line will show you. Look at the gains and losses each day. How do you feel? Have you lost all hope? Are you ecstatic? Replay your emotions line by line. Could you live with your life assets following such a model?
---
Obviously, the HBPP stands up to this kind of test very well. To me, this impies that the underlying theory and model are quite solid. I think that sort of thinking should go into any new scheme. What is the economic model that makes your idea good. How sensitive is it to unknown variations? How simple can you make it? When you have to pick allocation and trigger and threshold values (i.e. rebalance bands), how sensitive is your model to a 20% difference in your parameter value? (20/30 vs. 15/35). Because for damn sure, you didn't pick exactly right value to start with.
All of this testing and simulation can be done with some pretty basic intermediate Excel work. It is even better if you are comfortable using the solver function in Excel and sort of understand the limitations of global optimizers.
I would love to hear other suggestions for backtesting or checking crazy theories before putting your money on the line.
------
And... this post assumes that you are pursing a crazy idea based on some regular growth potential. If your crazy idea is of the "crap shoot" kind - well that is a different approach and a different post.