Focus On Robust Concepts Not Optimization
note: a new post on MABI-Modified Absolute Breadth Index will be up quite soon–please stay tuned.
For someone that has looked at data in many more ways that you can imagine, I can tell you one piece of important advice that will help create systems that will work out of sample: Focus on a concept that makes logical sense and is robust-that is, it has worked over long periods of time and multiple cycles and market regimes. Start by creating a hypothesis, and work through the logic and make some predictions in advance. Ideally, you should search for other research done by academics or practioners to build upon what has already been done. You should also carefully observe the testing methodology and indicator or independent variable creation. Next create a test that is as simple as possible and vary the parameters to test for sensitivity. A robust effect in testing will have a few characteristics: 1) it is stable over time 2) in tends to increase or decrease in magnitude in a fairly stable and linear manner with the independent variable 3) the results are not overly sensitive when changing the parameters such as length of measurement or entry/exit criteria. Pass these tests with rigorous testing, and I can virtually assure you that your system will be fairly durable over time as long as the effect persists.
Optimization on the other hand is sort of like collateralized debt obligations (CDOs) and fancy derivatives. Often using this method produces systems you don’t fully understand, that can be so unstable to the untrained idiot, that they could blow up at any time. It is the most dangerous tool in existence for traders or researchers. The biggest error is to take a backtest that is too short, or does not contain multiple regimes. To make matters worse, adding several different variables, “rules” or different indicators will exacerbate the problem substantially. Optimization is a good tool only after at least a dozen thorough criteria have been met. The statistical skill required to fully benefit from optimization is well out of reach for most traders. Use of regression and controls are critical to avoid finding spurious relationships.
But what really gets under my skin is the use and abuse of the concept of designing “trading systems.” It almost bothers me as much as when I read a newspaper article that mutilates the usage of statistical inference. Often I see “systems” that contain only 10 or 15 trades–if you have ever used any rigorous techniques or randomization you should know that it is easy to identify such “systems” and they are generally of no value whatsoever unless the edge or consistency are both ridiculously high (even then i wouldn’t bank on it). Idiots think systems exist in a box when in reality systems are just the expression of some type of theory. For example you may claim to have a “magic” RSI3/6 system that buys when RSI3 is below “x” and exits when RSI6 is above “y” with a stop loss. In reality what you are saying is that you believe mean reversion works, and that short term weakness forecasts above average market returns over a period of a week or 2 weeks. Thus what works specifically on your system should work in general on a variety of RSI or ideally other mean reversion indicators. Furthermore it should produce a consistent edge over that period of time (one week to two weeks).
So I know this is sort of like telling people to eat right and excercise to lose weight, but I know traders would rather use quick optimization procedures that can be run in seconds rather than run “mini-experiments.” Fad diets and magical pills will always be in style because people are inherently lazy and gullible. The optimist in me hopes this is not you!