Preface: Optimizer’s Anonymous
There are some valuable concepts to be taken from the “trend-following” research. Whether its the “turtles,” Ed Seykota, Larry Hite, or Michael Covel, a key concept that is emphasized is “robustness.” If you are a hard-core mathematician type, you tend to scoff at the homely, grandfatherish advice being espoused by the trend-following crew. They spin simplistic tales of creating systems that work across markets, and finding strategies that work over long periods of time. They preach about fat tails, and the unknowable future. The fact is that these guys are really on to something. The field of robust statistics emphasizes a lot of these principles, and truthfully PhD trained economists, physicists and finance geeks tend to ignore them at their own peril. The who’s who in the history of hedge fund blow ups is littered endlessy with ivy league quant gurus. I dare you to show me a single hedge fund that blew up trading the 200-day moving average across asset classes.
The colorful introduction is meant to grab your attention, and shake you up. Deep down in the depths of human psychology we really crave certainty and consistency. Accepting that we can’t control the future is a battle we have to fight every day. Sadly the smarter you are the harder this is to accept. Tell me you haven’t fallen prey to the dreams of creating endless riches by optimizing your favorite indicator. I remember dark moments myself many moons ago trading e-minis dreaming of Freedom 35. I would change my indicator settings daily, desperately trying to find the magic setting that would prevent me from getting whipsawed nearly every day. When i started trying to design stock screens for the first time, i went crazy backtesting every possible combination. I was hoping to find the one combination of P/E ratios or EPS growth that would always win. If this was you, there is hope, and I would like to share with you my story. The first step in Optimizer’s Anonymous is recognizing that you have an optimizing problem!
Fortunately, I had a tremendous advantage over the rest of you: the software i learned on only kept 5 years of historical data and rolled forward every week. As I furiously tested away, I got to see how the screens that i built performed using a new start date. As time rolled on, i got to see how my screens performed in real life as well as their equity curves before and after. It was an incredibly valuable experience that everyone should try for at least a few years. I started realizing though the course of testing that the more specific my parameters were, the noisier they performed out of sample. This problem was made even worse as i expanded the number of parameters. Selecting a price-earnings ratio of 7 or lower was unstable enough, but it was made worse by choosing stocks with an ROE higher than 15, and earnings growth greater than 20%. This led me on the path of using percentiles which i have already become notorious for on this blog—and with my professor friends. Another advantage was that i always tested across sectors simultaneously, under the assumtion that the underlying screen as a concept was more important than the idiosyncracies of a given industry. I tested like this for years even before receiving a formal education, and having that backround allowed me to design models that actually worked in real-life.
Trading strategies should be designed based on concepts, relationships, and logical theories. A theory that is not generalizable is more likely to be valid that one that is highly specific. The greater the number of unique environments that a backtest can survive–the more likely it is to be robust. Building upon these observations are my prescriptions for Optimizer’s Anonymous:
1) Test a given technical trading rule first on at least 5 distinct instruments—try to incorporate commodities, bonds, stocks, countries, sectors, currencies etc.
2) Try to find the parameter that works best on the greatest number of instruments.
3) Map the parameter on a grid to find the broad area of highest and most consistent profitability (local density point)
4) Take the median or average setting
5) Test this setting on specific instruments that you haven’t tested
6) Test this setting out of sample, noting the new area of local density
7) Determine the degree of overlap between the intial local density and the area of local density out of sample.
8) If the overlap is too small you will need to broaden the parameter selection area out to include more combinations.
9) Modify and re-test
CSSA 
I don’t understand your rule #1. Why should it be the case that a strategy that works, say, for financial stocks should also work for currencies? On the contrary I would imagine that different strategies would work better/worse in different sectors.
Russ,
Our confidence that the concept or theory that we are testing is higher if we know that it is generalizable to things that are not exactly the same. Suppose we assume that “undervalued” stocks outperform. This effect should be true across industries, and even asset classes……….and this is in fact the case. The problem with data mining is that it is easy to find a strategy that works for just financial stocks or certain currencies, but if it is not based on a theory or concept, it is probably just a random effect that worked really well for that point in history.
Actually, I’m not convinced. Some undervalued stocks (in terms of PE ratio) are undervalued for good reason. They don’t outperform.
Does any strategy work over long periods in most segments? Perhaps long term trending works; perhaps certain calendar phenomena are reasonably reliable; perhaps option sales are more profitable overall than option purchases. But as Michael Stokes points out, virtually any strategy is subject to change. As he frequently notes Mean Reversion failed in the 1990s but worked so far this century. But that doesn’t mean it will continue to work. You were the first person to comment on his piece today emphasizing that.
So it seems to me that if one can find a strategy that works for a period of time in some area, go for it. It isn’t likely to last anyway. But then neither is anything else.
Ok russ…….things are not black and white, and other things are like apples and oranges—-such as market timing, Michael and others concede that there are no great explanations for a lot of effects that are present in the data. There is no coherent theory to explain certain anomalies such as the mean reversion effect and that makes it more necessary to use proper adaptive methods–A lot of the time you have no idea why something is working, therefore it could stop working at any time. In contrast, certain effects in the finance literature are actually explainable and robust. The fact that low P/E ratios do not work on every stock fits extremely well within the confines of what i was talking about—the effect persists because it is concept or theory. The same applies to relative momentum. Robust statistics aims to identify effects and capture them using strategies that will minimize error in real life.
Of course effects can die, and and be subject to change—and adaptation is still a desirable and very powerful method that i use even for my stock rankings. However, if you know about history, the traders and institutions who went crazy with neural nets in the late 80s and 90s got killed because they over-optimized and didn’t understand what they were doing. As a general rule, good theories provide you with effects that last longer in the real world. Throwing your hands up and saying nothing will last is not the mark of a man of science but rather the mark of a true pessimist. I suggest you stop doing testing and just buy the S&P500 index and forget about it.
Actually, seems like buying and holding the S&P is an optimistic move. Increasingly pessimistic would be cash, gold, guns, salt, … see the fringe blogosphere for details.
Back to Russ, the point of value as a style is not that all selections will outperform, but rather it is one approach that gives an edge when applied to a large number of possible selections. As a starting point, it can provide a better fishing ground.
Great post. As I was reading I was tempted to say, “I’m Bill and I’m an optimizer.” Seriously, this is one of the most valuable lessons I’ve learned in trading, and something the veterans have preached forever… keep it simple.
Of course, I do have an advantage over many who are smarter than me. I don’t have any choice but to keep it simple… 😉
bill
lol welcome to the OA club Bill-thanks for the kind words…….just wanted to emphasize methodology over results for a change! re: simplicity– single-cell organisms survive seismic shifts, using simple but properly applied math with do the same!
toptick says that buying value “is one approach that gives an edge when applied to a large number of possible selections.” Here’s some data.
From until today
IWB (Russell 1000) IWD (Russell 1000 Value)
June 1 2000 -24% -5% (winner)
Jan 1 2003 +19% (winner) +16% (not far behind)
Jul 1 2005 -14% (winner) -20%
Jul 13 2007 -34% (winner) -41%
Mar 6, 2009 +48% +53% (winner)
These data are from http://www.google.com/finance?q=NYSE:IWD.
It’s not clear to me from this data that IWD has a significant or consistent advantage over IWB. Of course IWB includes IWD plus the non-value stocks in the Russell 1000. So perhaps one should double the differences indicated. But the conclusions would be more the same.
Rest assured i have tested everything since 1966, and i am not per se a “value guy” —a combination of factors including value does much better. However, here is a test of a composite percentile ranking including only Price to Sales, Price to Book and Price to Earnings. I took the top 50% of the database of stocks over 100 million in market cap and over a $5 share price with 100,000 shares traded:
Top 50% of the Database Most Undervalued
STATISTICS ex.: $10,000 start Strategy S&P 500
Total Compounded Return 56.3 30.6
Total Compounded Return $ $15,627 $13,061
Compounded Annual Growth Rate 10.70% 6.30%
Win Ratio 67 67
Winning Periods/Total Periods 38 of 57 38 of 57
Avg. # of Stocks Held 1417.5
Avg. Periodic Turnover 20.6
Avg. Return per Period 0.9 0.5
Here is a more concentrated test…….top and bottom 10% by P/E
STATISTICS ex.: $10,000 start Bottom 10% by P/E Top 10% by P/E
Total Compounded Return 72.5 30.6
Total Compounded Return $ $17,253 $13,001
Compounded Annual Growth Rate 13.2% 6.3%
Win Ratio 65 60
Winning Periods/Total Periods 37 of 57 34 of 57
Avg. # of Stocks Held 283.1 283.1
Avg. Periodic Turnover 17.1 19.9
Avg. Return per Period 1.1 0.6
Interesting results!
Sorry about the formatting. It was a lot prettier when I typed it in. I used spaces to space out the table. But the spaces were collapsed. I hope it is still understandable.
Russ: it may be more than you want to know, but http://www.cxoadvisory.com reviews and organizes academic and other research on such issues as value stocks, momentum, seasonality, etc.. For starters, see the ‘blog synthesis’ on Fundamental Valuation or Value Premium.
(It took me weeks to go through the old posts when I discovered that site — a treasure.)
Yes, thanks. It’s already in my read-first blog list. A very good site.