## Curve Fitted Back Testing

**Moderator:** Global Moderator

- Mark Leavy
- Executive Member
**Posts:**643**Joined:**Thu Mar 01, 2012 10:20 pm**Location:**US Citizen, Permanent Traveler

### Curve Fitted Back Testing

Yes... you know you do it.

You have a new scheme. A crazy idea. A wild hair. You have to fit some parameters to the model to see what woulda coulda shoulda been.

But... a curve fit is precarious. You can make any data fit pretty much anything with even just a smidgen of math.

And yet... back testing has to be the first step to test any new idea - if nothing else, just to prove to yourself that it was a crazy, worthless idea.

So... what do you do to ensure that you aren't fooling yourself with regards to backtesting?

Here are my rules. I'd love to hear what others do.

1) Use the longest period of historical data that you can find. Find reliable simulations for assets that are too new to have direct data. Use *daily* data, not monthly. Daily swings can be huge and the max draw down numbers show up in the daily data, not the monthly. For me, I need to simulate the daily swings in order to evaluate whether or not I can stomach them in the future.

2) Limit your rules and conditions as much as possible. I.e. X assets, N rules for reallocation or triggers, etc. The fewer the better.

3) Fit the growth curve of your new scheme to an exponential line. The more it deviates from that line (either up or down) during different economic situations makes your model less robust.

4) If I am doing any curve fitting (i.e. optimizing parameters) I like to set a limit for maximum drawdown. My personal limit is 20%. That is about the most I can stomach before I start to wonder. So... If I have a bunch of data and I am trying to figure out the best asset allocation, I set my optimizer to maximize the CAGR, with the caveat that I never have a maximum draw down more than 20%. As an aside, the smaller this drawdown number, the more likely your total returns will be linear log scale - and thus implying a much more solid model.

5) Do sensitivity analysis on your model. If you have decided on something like 25/25/25/25 asset allocation with 15/35 reallocation bands, then also run some models with the reallocation bands higher and lower. Change the asset allocations by a few percentage points. Do the results still basically come out the same? That's great. If you end up seeing huge swings in the CAGR with minor changes in the rules... then your model won't be very robust in the future.

6) Always calculate the max draw down number of your simulation and the maximum number of days that you are down from a peak. Great that your model shows a 20% historical CAGR. But can you really handle 10 years without a new peak? Or a max draw down of 45%? Max draw down percent and duration are critical things to know about your model. And how do they change when you start doing sensitivity analysis on your parameters?

7) Look at the final data for your simulation and scroll through it day by day. Pretend that you don't know what the next line will show you. Look at the gains and losses each day. How do you feel? Have you lost all hope? Are you ecstatic? Replay your emotions line by line. Could you live with your life assets following such a model?

---

Obviously, the HBPP stands up to this kind of test very well. To me, this impies that the underlying theory and model are quite solid. I think that sort of thinking should go into any new scheme. What is the economic model that makes your idea good. How sensitive is it to unknown variations? How simple can you make it? When you have to pick allocation and trigger and threshold values (i.e. rebalance bands), how sensitive is your model to a 20% difference in your parameter value? (20/30 vs. 15/35). Because for damn sure, you didn't pick exactly right value to start with.

All of this testing and simulation can be done with some pretty basic intermediate Excel work. It is even better if you are comfortable using the solver function in Excel and sort of understand the limitations of global optimizers.

I would love to hear other suggestions for backtesting or checking crazy theories before putting your money on the line.

------

And... this post assumes that you are pursing a crazy idea based on some regular growth potential. If your crazy idea is of the "crap shoot" kind - well that is a different approach and a different post.

You have a new scheme. A crazy idea. A wild hair. You have to fit some parameters to the model to see what woulda coulda shoulda been.

But... a curve fit is precarious. You can make any data fit pretty much anything with even just a smidgen of math.

And yet... back testing has to be the first step to test any new idea - if nothing else, just to prove to yourself that it was a crazy, worthless idea.

So... what do you do to ensure that you aren't fooling yourself with regards to backtesting?

Here are my rules. I'd love to hear what others do.

1) Use the longest period of historical data that you can find. Find reliable simulations for assets that are too new to have direct data. Use *daily* data, not monthly. Daily swings can be huge and the max draw down numbers show up in the daily data, not the monthly. For me, I need to simulate the daily swings in order to evaluate whether or not I can stomach them in the future.

2) Limit your rules and conditions as much as possible. I.e. X assets, N rules for reallocation or triggers, etc. The fewer the better.

3) Fit the growth curve of your new scheme to an exponential line. The more it deviates from that line (either up or down) during different economic situations makes your model less robust.

4) If I am doing any curve fitting (i.e. optimizing parameters) I like to set a limit for maximum drawdown. My personal limit is 20%. That is about the most I can stomach before I start to wonder. So... If I have a bunch of data and I am trying to figure out the best asset allocation, I set my optimizer to maximize the CAGR, with the caveat that I never have a maximum draw down more than 20%. As an aside, the smaller this drawdown number, the more likely your total returns will be linear log scale - and thus implying a much more solid model.

5) Do sensitivity analysis on your model. If you have decided on something like 25/25/25/25 asset allocation with 15/35 reallocation bands, then also run some models with the reallocation bands higher and lower. Change the asset allocations by a few percentage points. Do the results still basically come out the same? That's great. If you end up seeing huge swings in the CAGR with minor changes in the rules... then your model won't be very robust in the future.

6) Always calculate the max draw down number of your simulation and the maximum number of days that you are down from a peak. Great that your model shows a 20% historical CAGR. But can you really handle 10 years without a new peak? Or a max draw down of 45%? Max draw down percent and duration are critical things to know about your model. And how do they change when you start doing sensitivity analysis on your parameters?

7) Look at the final data for your simulation and scroll through it day by day. Pretend that you don't know what the next line will show you. Look at the gains and losses each day. How do you feel? Have you lost all hope? Are you ecstatic? Replay your emotions line by line. Could you live with your life assets following such a model?

---

Obviously, the HBPP stands up to this kind of test very well. To me, this impies that the underlying theory and model are quite solid. I think that sort of thinking should go into any new scheme. What is the economic model that makes your idea good. How sensitive is it to unknown variations? How simple can you make it? When you have to pick allocation and trigger and threshold values (i.e. rebalance bands), how sensitive is your model to a 20% difference in your parameter value? (20/30 vs. 15/35). Because for damn sure, you didn't pick exactly right value to start with.

All of this testing and simulation can be done with some pretty basic intermediate Excel work. It is even better if you are comfortable using the solver function in Excel and sort of understand the limitations of global optimizers.

I would love to hear other suggestions for backtesting or checking crazy theories before putting your money on the line.

------

And... this post assumes that you are pursing a crazy idea based on some regular growth potential. If your crazy idea is of the "crap shoot" kind - well that is a different approach and a different post.

### Re: Curve Fitted Back Testing

Use a "learning" data set and a "testing" data set. That is, fit your model to some subset of the data, and then look at how the predictions work on the rest of the data. For example, if you have 10 years worth of data, fit parameters based on the first 7 years, and see how those parameters work when applied to the final 3.

Another thought (which I've never tried)... If you have daily data, perhaps fit your model based on days 1, 3, 5, etc. and test it on data from days 2, 4, 6, etc.

Sensitivity studies are a must. In fact, I would recommend a Monte Carlo study around the fitted parameter values. (As an aside, it is unfortunate that the Excel Solver does not provide confidence intervals for parameter estimates).

I would also advocate putting some time-of-action uncertainty into the simulation. For example, with the HBPP, one might have a model set up to rebalance upon exceeding 15/35 bands. I would advocate adding a random time delay on the rebalancing (e.g. anywhere between 1 day and 2 weeks) and seeing how that affects results.

All that said, I don't actually do any of this. I generally think doing too much analysis tricks me into thinking I know more than I really do.

Another thought (which I've never tried)... If you have daily data, perhaps fit your model based on days 1, 3, 5, etc. and test it on data from days 2, 4, 6, etc.

Sensitivity studies are a must. In fact, I would recommend a Monte Carlo study around the fitted parameter values. (As an aside, it is unfortunate that the Excel Solver does not provide confidence intervals for parameter estimates).

I would also advocate putting some time-of-action uncertainty into the simulation. For example, with the HBPP, one might have a model set up to rebalance upon exceeding 15/35 bands. I would advocate adding a random time delay on the rebalancing (e.g. anywhere between 1 day and 2 weeks) and seeing how that affects results.

All that said, I don't actually do any of this. I generally think doing too much analysis tricks me into thinking I know more than I really do.

Last edited by tennpaga on Mon Nov 03, 2014 11:50 am, edited 1 time in total.

*

*

https://fs.blog/2009/12/mental-model-greshams-law/

**Gresham's Law**: Bad behavior drives out good.*

**Gresham's corollary**: Avoid participating in systems where good behavior cannot win.https://fs.blog/2009/12/mental-model-greshams-law/

### Re: Curve Fitted Back Testing

I was going to say the same thing.TennPaGa wrote: Use a "learning" data set and a "testing" data set. That is, fit your model to some subset of the data, and then look at how the predictions work on the rest of the data. For example, if you have 10 years worth of data, fit parameters based on the first 7 years, and see how those parameters work when applied to the final 3.

In addition I think inflation-adjusted returns are important since inflation can hide poor real performance and deflation can depress decent/okay returns.

I also like to use real CAGR rolling returns over various periods. This gives a rough idea of the probability of achieving a certain CAGR over a given period.

I have also just learned about the "gain vs pain" parameter. It is basically the sum of the daily/monthly positive returns vs the sum of the daily/monthly negative returns. This is similar to the Sharpe Ratio, but it provides another look at the data. The PP has a "gain vs pain" (using monthly data) of 1.65, while a 60/40 is at 1.52, so PP is better in this regard.

- Mark Leavy
- Executive Member
**Posts:**643**Joined:**Thu Mar 01, 2012 10:20 pm**Location:**US Citizen, Permanent Traveler

### Re: Curve Fitted Back Testing

Some really great comments. Just outstanding.

Thank you Tenn and Gosso.

I'm compiling all of these guidelines into my "scoliosis" folder, in honor of dualstow

Thank you Tenn and Gosso.

I'm compiling all of these guidelines into my "scoliosis" folder, in honor of dualstow

### Re: Curve Fitted Back Testing

Here's something else I did recently... It isn't really curve fitting, but more like your #7 (scroll through the daily data). I took the series of returns and simply reversed the time order. If you are looking at growth of an initial investment, the final value is the same, but the path is of course quite different. And if you are making periodic additions, the final value will be different too.

Here are a couple of plots illustrating the concept. PP is a standard 4-ETF PP with 10% rebalancing bands. BH is a 60/40 VTSMX/VBMFX with 5% rebalancing bands.

Here are a couple of plots illustrating the concept. PP is a standard 4-ETF PP with 10% rebalancing bands. BH is a 60/40 VTSMX/VBMFX with 5% rebalancing bands.

*

*

https://fs.blog/2009/12/mental-model-greshams-law/

**Gresham's Law**: Bad behavior drives out good.*

**Gresham's corollary**: Avoid participating in systems where good behavior cannot win.https://fs.blog/2009/12/mental-model-greshams-law/

### Re: Curve Fitted Back Testing

Gosso,Gosso wrote: I have also just learned about the "gain vs pain" parameter. It is basically the sum of the daily/monthly positive returns vs the sum of the daily/monthly negative returns. This is similar to the Sharpe Ratio, but it provides another look at the data. The PP has a "gain vs pain" (using monthly data) of 1.65, while a 60/40 is at 1.52, so PP is better in this regard.

Can you please explain this further? Does it mean that for every down month in the PP, there are 1.65 up months? Real or nominal? Thanks.

### Re: Curve Fitted Back Testing

You're close, it also measures the magnitudes of the positive months vs negative months. The numbers I used were inflation-adjusted (real), from Jan-1972 to Sept-2014, reinvested dividends and interest, and rebalanced monthly. Here are the numbers:barrett wrote: Gosso,

Can you please explain this further? Does it mean that for every down month in the PP, there are 1.65 up months? Real or nominal? Thanks.

Real PP:

- Sum of positive months = 511.4%

- Sum of negative months = -311.1%

- "Gain to Pain" Ratio = 1.65

Real 60/40:

- Sum of positive months = 674.7%

- Sum of negative months = -444.2%

- "Gain to Pain" Ratio = 1.52

Nominal PP:

- Sum of positive months = 613.4%

- Sum of negative months = -235.9%

- "Gain to Pain" Ratio = 2.60

Nominal 60/40:

- Sum of positive months = 767.0%

- Sum of negative months = -359.7%

- "Gain to Pain" Ratio = 2.13

Nominal Stocks:

- Sum of positive months = 1102.5%

- Sum of negative months = -647.8%

- "Gain to Pain" Ratio = 1.73

Nominal Gold:

- Sum of positive months = 1293.1%

- Sum of negative months = -874.3%

- "Gain to Pain" Ratio = 1.48

Nominal 30 Year:

- Sum of positive months = 818.1%

- Sum of negative months = -451.5%

- "Gain to Pain" Ratio = 1.81

Nominal 1 Year:

- Sum of positive months = 265.3%

- Sum of negative months = -13.4%

- "Gain to Pain" Ratio = 19.79

I don't know if this tell us anything different than standard deviation and shapre ratio. I heard about "gain to pain" from an interview with Jack Schwager (he wrote "Market Wizards", which I haven't read), so I'm guessing he uses it to make hedge funds and managed futures look better.

### Re: Curve Fitted Back Testing

Thanks for that, Gosso. Really good stuff. I am reminded of something that I read about watching a baseball hitter over the course of a season... that you can't really see the difference between a .300 hitter and a .280 hitter if you are sitting in the stands watching every game, because the difference between the two is only one base hit every two weeks. They both seem to be performing at about the same level, but, of course, you'd rather have the .300 hitter.

Figuring out a "pain index" that weighs the effect of recent market swings could be the next great thing for you stat guys. I am amazed (and grateful!) for people like you, TennPa, Mark, Sophie, Tyler, PS, and probably a few others who have the brains and patience to produce these data. Maybe when the PP pain index is high, that should be a signal to invest new money, if possible.

Figuring out a "pain index" that weighs the effect of recent market swings could be the next great thing for you stat guys. I am amazed (and grateful!) for people like you, TennPa, Mark, Sophie, Tyler, PS, and probably a few others who have the brains and patience to produce these data. Maybe when the PP pain index is high, that should be a signal to invest new money, if possible.

- Pointedstick
- Executive Member
**Posts:**8697**Joined:**Tue Apr 17, 2012 9:21 pm-
**Contact:**

### Re: Curve Fitted Back Testing

This brings up sort of a related point… the "pain index" should really only result in pain if you are actually withdrawing from your investments or preparing to. If you still have decades before you expect to use the money, down periods should be completely irrelevant at worst, and a fantastic buying opportunity at best.barrett wrote: Maybe when the PP pain index is high, that should be a signal to invest new money, if possible.

The whole reason why I use the PP is because I expect to achieve financial independence in the next couple of years, so I can't tolerant huge drawdowns. If I was following the more traditional "work for 40 years and retire at 65" career path, I would probably invest with an eye toward generating a much higher CAGR and accepting the additional volatility, and incrementally start converting it all to a PP 15 or 20 years before retirement.

Human behavior is economic behavior. The particulars may vary, but competition for limited resources remains a constant.

- CEO Nwabudike Morgan

- CEO Nwabudike Morgan

### Re: Curve Fitted Back Testing

PointedStick, I hope you let us all know when you start getting close to the early retirement milestone!

I agree completely regarding accepting high volatility vs the PP. 20 years prior to retirement is probably a decent time to make the switch, if you can. I am in that time window as well.

There may still be a rationale to go into the PP even if you are far away from retirement: CAGR and pain/gain isn't the whole story. If your portfolio sustains a big loss, it will take a much larger gain to fight your way back to break-even. Thus, let's say you sustain a 40% loss one year, then follow it with a 40% gain the next year. If you started with $100,000, at the end of two years you'd have only $84,000. Or put another way, you'd need a 66% gain in year two to get back to break even. The smaller drawdowns of the PP minimize this problem.

Craig and MediumTex covered this quite eloquently in their book. In addition of course there's the psychology factor. If you think you can sit through a 40% drawdown and not touch your investments, then more power to you! I already know that I would have a hard time with this, and would be likely to do something stupid like sell and lock in the losses.

I agree completely regarding accepting high volatility vs the PP. 20 years prior to retirement is probably a decent time to make the switch, if you can. I am in that time window as well.

There may still be a rationale to go into the PP even if you are far away from retirement: CAGR and pain/gain isn't the whole story. If your portfolio sustains a big loss, it will take a much larger gain to fight your way back to break-even. Thus, let's say you sustain a 40% loss one year, then follow it with a 40% gain the next year. If you started with $100,000, at the end of two years you'd have only $84,000. Or put another way, you'd need a 66% gain in year two to get back to break even. The smaller drawdowns of the PP minimize this problem.

Craig and MediumTex covered this quite eloquently in their book. In addition of course there's the psychology factor. If you think you can sit through a 40% drawdown and not touch your investments, then more power to you! I already know that I would have a hard time with this, and would be likely to do something stupid like sell and lock in the losses.

"Democracy is two wolves and a lamb voting on what to have for lunch." -- Benjamin Franklin

### Re: Curve Fitted Back Testing

Exactly! And the closer you get to retirement the more terrifying the losses become. Not only because of the percentage amounts, but because the absolute amounts become simply ridiculous. A "mere" 10% loss on a $2M portfolio is $200,000. If you think you can be sanguine about losses like this, I suggest you have another think coming.sophie wrote: PointedStick, I hope you let us all know when you start getting close to the early retirement milestone!

I agree completely regarding accepting high volatility vs the PP. 20 years prior to retirement is probably a decent time to make the switch, if you can. I am in that time window as well.

There may still be a rationale to go into the PP even if you are far away from retirement: CAGR and pain/gain isn't the whole story. If your portfolio sustains a big loss, it will take a much larger gain to fight your way back to break-even. Thus, let's say you sustain a 40% loss one year, then follow it with a 40% gain the next year. If you started with $100,000, at the end of two years you'd have only $84,000. Or put another way, you'd need a 66% gain in year two to get back to break even. The smaller drawdowns of the PP minimize this problem.

Craig and MediumTex covered this quite eloquently in their book. In addition of course there's the psychology factor. If you think you can sit through a 40% drawdown and not touch your investments, then more power to you! I already know that I would have a hard time with this, and would be likely to do something stupid like sell and lock in the losses.

Scene: Wednesday Addams, at school, dressed all in black.

Amanda: Hi, I'm Amanda Buckman. Why are you dressed like that?

Wednesday: Like what?

Amanda: Like you're going to a funeral. Why are you dressed like somebody died?

Wednesday: Wait.

People ask me why I invest like the world might end tomorrow.

I say, "wait".

- technovelist
- Executive Member
**Posts:**5584**Joined:**Wed Sep 15, 2010 11:20 pm

### Re: Curve Fitted Back Testing

Well, I'm down over $400K from my highest ever portfolio balance, and I haven't panicked yet.rickb wrote:Exactly! And the closer you get to retirement the more terrifying the losses become. Not only because of the percentage amounts, but because the absolute amounts become simply ridiculous. A "mere" 10% loss on a $2M portfolio is $200,000. If you think you can be sanguine about losses like this, I suggest you have another think coming.sophie wrote: PointedStick, I hope you let us all know when you start getting close to the early retirement milestone!

I agree completely regarding accepting high volatility vs the PP. 20 years prior to retirement is probably a decent time to make the switch, if you can. I am in that time window as well.

There may still be a rationale to go into the PP even if you are far away from retirement: CAGR and pain/gain isn't the whole story. If your portfolio sustains a big loss, it will take a much larger gain to fight your way back to break-even. Thus, let's say you sustain a 40% loss one year, then follow it with a 40% gain the next year. If you started with $100,000, at the end of two years you'd have only $84,000. Or put another way, you'd need a 66% gain in year two to get back to break even. The smaller drawdowns of the PP minimize this problem.

Craig and MediumTex covered this quite eloquently in their book. In addition of course there's the psychology factor. If you think you can sit through a 40% drawdown and not touch your investments, then more power to you! I already know that I would have a hard time with this, and would be likely to do something stupid like sell and lock in the losses.

Scene: Wednesday Addams, at school, dressed all in black.

Amanda: Hi, I'm Amanda Buckman. Why are you dressed like that?

Wednesday: Like what?

Amanda: Like you're going to a funeral. Why are you dressed like somebody died?

Wednesday: Wait.

People ask me why I invest like the world might end tomorrow.

I say, "wait".

I'll let you know when I do.

Another nod to the most beautiful equation: eiπ + 1 = 0