Bogus backtesting

David MoonBlog

by David Moon

If you own stocks, you should hope Hillary Clinton is our next president.

Our firm has developed the Character Count market system—an investment program that, since 1929, has outperformed the S&P 500 by more than 40 percent annually, in addition to generally avoiding years with below average and negative stock returns.

The system: only own the S&P 500 in years when the president’s last name has six or more characters.

In those high character count years, the S&P 500 increased 14.6 percent annually, compared to only 4.8 percent when the president’s surname contained fewer than six characters.

Another of our proprietary systems, the In the Black model, has returned 180 percent over the past nine years. Without a single losing year. The secret: own the S&P 500 only in years during which the U.S. president is an African-American.

Although these two examples are laughable, they are merely exaggerated versions of a too-common practice in our industry: the improper use of backtested models. That is, advisers and firms cherry picking data and implying (or downright claiming) they had actually produced those returns for clients and investors.

Anyone with a computer can easily create models that appear to be both successful and financially sophisticated, yet are as similarly useless as my two nonsensical examples.

When someone shows you their model mutual fund portfolio and says “this portfolio returned a zillion percent in the past five years,” it doesn’t mean that the adviser’s actual clients earned that return. It means they would have earned a zillion percent if they had owned those hypothetical funds over the past five years.

And if I had owned LaJolla Pharmaceutical shares over the past five years, I would have made 5,200 percent. And if I had worked out 1,800 straight days, I would be 15 pounds lighter.

Backtested models can be useful if they are tested in periods outside of the period used to develop the model. From Patrick Henry until 2007, the In the Black Model didn’t perform so well. It likely won’t in the next four years either. But in the last nine years, it worked like a charm.

Two of the worst backtesting offenders are Unit Investment Trusts (UITs) and mutual fund models. The UIT industry is filled with brokerage firms contriving historic models without testing them outside of the data on which they were based, then selling fixed-life funds to implement the strategy.

Models that select mutual funds, even those tested or implemented after the testing period, suffer from an inability to actually control or determine the underlying securities in the fund, and therefore the assets that ultimately determine an investor’s return. Models based on index funds are an exception, but only if they have been tested in periods other than the period used to develop the model.

David Moon is president of Moon Capital Management, a Knoxville-based investment management firm. This article originally appeared in the News Sentinel (Knoxville, TN).