A Case that all your Backtesting is for Naught

[mmaurice]

To chime a thought here as I have been reading many of the posts on this board, the core aspects that make the PP interesting is it is rooted in a basic understanding on how certain assets act under certain conditions, not past performance as an indicator of future success.

If I understand the history correct, the goal was to design a strategy that the average Joe could put his money and really do nothing except a couple minutes a year of calculations and actions and for which be rewarded (above) average returns with limited variance.  The development of such a system was also at a time when personal retirement savings were switching from the traditional pension plan introduced in the 40's to an individually managed account, where most individuals have not been trained to perform such tasks and thus are set to fail. From HB's perspective given his libertarian/political aspirations, the development of such a system would have seemed to be in alignment with his core principals of serving his constituents.

Therefore, to start, it began with an understanding of the different modes of a modern economy and then what performs in each mode. Given there are four different modes of the economy and that for each mode there is an asset class that shines above all others, the 4x25 split makes the most sense. Furthermore, if you assume the economy, over the broad course of time, spends equal amounts of time in each mode and while in certain situations the economy is contracting but as a whole an economy must expand for a society to survive, then at the very worst, you might assume that the 4x25 portfolio will maintain its principal balance over time.

What then compounds this portfolio is the fact that while in any given mode there is an asset that shines there is then also a secondary asset(s) that expand or hold value. This is what makes the system work, when one or more is contracting; one or more are expanding, at a greater rate than contraction.

Thus, you can backtest to your heart's content and find scenarios that beat others but the legal statements will stand true, "past performance is not an indicator of future success." Reasons for which, as I tried to describe above, you have F'ed up the general principals by which the system was designed.

The only way to truly make a case for any allocation amongst asset classes that seem to succeed in backtesting is to further deliberate the fundamental economic tenants which truly govern why it worked. I might even go further to state that when you look at the modes of an modern economy, they might be changing or at least morphing (which I doubt) and thus you might be able to illustrate better asset choices (than Stocks, Gold, Bonds, Cash) that will succeed in these newly defined economic modes. Of course, to do so puts you in the ranks of serious modern economists such as, Adam Smith, Benjamin Graham, and Milton Friedman.

GLTA, just my ramblings as I see many repetitive posts about the best asset allocation ratios or how wonderful a portfolio might be if you introduce this or that asset class.

[possum]

Just to play devil's advocate, I think it is possible to believe in the four basic modes of a modern economy represented by the PP and still question things like how much and what types of Stocks, Bonds, Gold, and Cash should be included in the portfolio.

[Bean]

Just to play devil's advocate, I think it is possible to believe in the four basic modes of a modern economy represented by the PP and still question things like how much and what types of Stocks, Bonds, Gold, and Cash should be included in the portfolio.

Changing allocations based on back testing assumes the four economic conditions will exist proportionally to your back test.  This is why "optimization" of allocations is impossible, since about the only thing I can guarantee you is the future will not be exactly like the past.

[Pointedstick]

Possum is right. The PP's neutrality is a very pessimistic assessment of the future; assuming that we'll be in tight money recessions as often as periods of prosperity, for example. One could assemble a more "optimistic" PP by replacing some cash with stocks, which would probably begin to approach a Boglehead portfolio, albeit one heavily overweighted with gold.

That said, I don't monkey with the percentages in my portfolio. I've found that the assets can serve useful purposes outside of their corresponding economic conditions. For example, having a lot of cash on hand lets you buy assets quickly and easily as they go on sale, even if tight money recessions have historically been rarer than prosperity.

[possum]

Just to play devil's advocate, I think it is possible to believe in the four basic modes of a modern economy represented by the PP and still question things like how much and what types of Stocks, Bonds, Gold, and Cash should be included in the portfolio.

Changing allocations based on back testing assumes the four economic conditions will exist proportionally to your back test.  This is why "optimization" of allocations is impossible, since about the only thing I can guarantee you is the future will not be exactly like the past.

Part of examining a potential asset allocation is to see how it performed relative to other choices during specific periods of time when different economic conditions existed. That avoids optimizing over a condition that lasts a long time over history. That said, it is hard to come up with a single number that way.

I would argue that the PP's attempts to be neutral may not really be so because I'm not sure I believe that 25% Stocks = 25% Gold = 25% Bonds = 25% Cash. In other words, it may take different amounts of each asset to react in the same way to it's optimum economic condition.

[Pointedstick]

I would argue that the PP's attempts to be neutral may not really be so because I'm not sure I believe that 25% Stocks = 25% Gold = 25% Bonds = 25% Cash. In other words, it may take different amounts of each asset to react in the same way to it's optimum economic condition.

You're certainly right… the question is: how can we know beforehand?

[mmaurice]

I would argue that the PP's attempts to be neutral may not really be so because I'm not sure I believe that 25% Stocks = 25% Gold = 25% Bonds = 25% Cash. In other words, it may take different amounts of each asset to react in the same way to it's optimum economic condition.

I would agree with you on that point but backtesting will not provide you the answer to the question. It will only confirm your answer.

It is true that if you run a regression model you MAY find the best possible allocation for the given backtest period. However, as Bean pointed out, the future would have to act nearly the same (same '87 crash, same wars, same bubbles, etc) in order to present similar outcomes.

I would argue that you must dive deeper into the analysis of the assets themselves (and not just stocks, gold, bonds, cash but also commodities, real estate, etc, etc) and how they correlate to the different modes of the economy. To put it differently you must determine the true systemic risk/return of each asset for each mode of the economy.

The final step then is to build an asset model that provides the best return (not to be confused with overall max return) for the amount variance you are willing to accept in any given economic condition. Thus while I may not beat the market during prosperity, I will not lose all of my money during times of depression 

Now the whole point to the PP is you do not care what the future will hold because as long as fundamentally you have the optimum model your return should be forecastable.

[possum]

I would argue that the PP's attempts to be neutral may not really be so because I'm not sure I believe that 25% Stocks = 25% Gold = 25% Bonds = 25% Cash. In other words, it may take different amounts of each asset to react in the same way to it's optimum economic condition.

You're certainly right… the question is: how can we know beforehand?

No, you can't know. But, we also don't know for the 4x25 PP either...

If it is true that the 4x25 (assets, not economic model) is not neutral, then it is possible a suggested alternative is actually more neutral and shouldn't be discarded out of hand - that was my original point.

[mmaurice]

I would argue that the PP's attempts to be neutral may not really be so because I'm not sure I believe that 25% Stocks = 25% Gold = 25% Bonds = 25% Cash. In other words, it may take different amounts of each asset to react in the same way to it's optimum economic condition.

You're certainly right… the question is: how can we know beforehand?

No, you can't know. But, we also don't know for the 4x25 PP either...

If it is true that the 4x25 (assets, not economic model) is not neutral, then it is possible a suggested alternative is actually more neutral and shouldn't be discarded out of hand - that was my original point.

I agree.

The perfect model might be:
15% Domestic Stocks
10% International Stocks
15% LT Bonds
10% Real Estate
15% Gold
10% Unobtanium
25% ST Bonds

[Tortoise]

Regardless of backtesting, one way of looking at the four-way split is as a max-drawdown-minimization strategy. It minimizes the worst-case drawdown from one of the four assets taking a nosedive. The minute you allocate more than 25% to any of the four assets, you are at risk of a larger drawdown if that particular asset implodes.

Maximizing returns and minimizing the worst-case drawdown are two completely different strategies. Based on Harry Browne's writings and radio shows, I think it's clear that he designed the PP for the latter, not the former.

The fact that the PP has historically provided healthy returns in addition to the superior safety is definitely a pleasant "bonus," but it is not the main reason HB designed the portfolio in the first place.

[MediumTex]

What I would really like to be able to do would be "forward test" a portfolio, but I don't think we know how to do that yet.

As Harry Browne wrote, the best use of backtesting is probably to disprove an allocation theory, at least with respect to how it would have performed in the past.

If something has never worked in the past, I would probably be less inclined to believe that it would work in the future.  If, however, you have an allocation theory and when you backtest it it holds up pretty well, I think that is just one more piece of information to keep in mind when deciding which allocation strategy is right for you.

[Greg]

I would argue that the PP's attempts to be neutral may not really be so because I'm not sure I believe that 25% Stocks = 25% Gold = 25% Bonds = 25% Cash. In other words, it may take different amounts of each asset to react in the same way to it's optimum economic condition.

You're certainly right… the question is: how can we know beforehand?

No, you can't know. But, we also don't know for the 4x25 PP either...

If it is true that the 4x25 (assets, not economic model) is not neutral, then it is possible a suggested alternative is actually more neutral and shouldn't be discarded out of hand - that was my original point.

I think this relates to goes back to a post that Clive wrote at one point regarding every year which asset performed the best. He then had where you'd have your asset allocations based upon the historical percentage that each asset was up for the year. This didn't speak about the magnitude of how much they were up (going on your point about how volatile they are and having one asset pulling the portfolio during a particular economic scenario) but it did say which place would have been the best place to park your money that year.

Obviously this is backtesting but if you could say 50% of the time stocks did the best over all the years collected, even if you gave it a large standard deviation on either side of that and erred on the low end, you'd still probably be above the average 25% asset allocation we have for the plain-vanilla PP.