Skip to content

Focus On Robust Concepts Not Optimization

October 30, 2009

note: a new post on MABI-Modified Absolute Breadth Index will be up quite soon–please stay tuned.

For someone that has looked at data in many more ways that you can imagine, I can tell you one piece of important advice that will help create systems that will work out of sample: Focus on a concept that makes logical sense and is robust-that is, it has worked over long periods of time and multiple cycles and market regimes. Start by creating a hypothesis, and work through the logic and make some predictions in advance. Ideally, you should search for other research done by academics or practioners to build upon what has already been done.  You should also carefully observe the testing methodology and indicator or independent variable creation. Next create a test that is as simple as possible and vary the parameters to test for sensitivity. A robust effect in testing will have a few characteristics: 1) it is stable over time 2) in tends to increase or decrease in magnitude in a fairly stable and linear manner with the independent variable 3) the results are not overly sensitive when changing the parameters such as length of measurement or entry/exit criteria.   Pass these tests with rigorous testing, and I can virtually assure you that your system will be fairly durable over time as long as the effect persists.

Optimization on the other hand is sort of like collateralized debt obligations (CDOs) and fancy derivatives. Often using this method produces systems you don’t fully understand, that can be so unstable to the untrained idiot, that they could blow up at any time. It is the most dangerous tool in existence for traders or researchers. The biggest error is to take a backtest that is too short, or does not contain multiple regimes. To make matters worse, adding several  different variables, “rules” or different indicators  will exacerbate the problem substantially. Optimization is a good tool only after at least a dozen thorough criteria have been met. The statistical skill required to fully benefit from optimization is well out of reach for most traders. Use of regression and controls are critical to avoid finding spurious relationships.

But what really gets under my skin is the use and abuse of the concept of designing “trading systems.” It almost bothers me as much as when I read a newspaper article that mutilates the usage of statistical inference. Often I see “systems” that contain only 10 or 15 trades–if you have ever used any rigorous techniques or randomization you should know that it is easy to identify such “systems” and they are generally of no value whatsoever unless the edge or consistency are both ridiculously high (even then i wouldn’t bank on it). Idiots think systems exist in a box when in reality systems are just the expression of some type of theory. For example you may claim to have a “magic” RSI3/6 system that buys when RSI3 is below “x” and exits when RSI6 is above “y” with a stop loss. In reality what you are saying is that you believe mean reversion works, and that short term weakness forecasts above average market returns over a period of a week or 2 weeks. Thus what works specifically on your system should work in general on a variety of RSI or ideally other mean reversion indicators. Furthermore it should produce a consistent edge over that period of time (one week to two weeks). 

So I know this is sort of like telling people to eat right and excercise to lose weight, but I know traders would rather use quick optimization procedures that can be run in seconds rather than run “mini-experiments.” Fad diets and magical pills will always be in style because people are inherently lazy and gullible. The optimist in me hopes this is not you!

5 Comments leave one →
  1. quant permalink
    October 30, 2009 8:18 am

    Thanks for your comments.

    I wonder about your statement that a system with only 10 to 15 trades is suspect. Suppose a long-term moving average crossover system on stock indices, that goes both long and short, has made only 15 trades over the last 20 years. I could still say that there are 20*250 = 5000 daily return observations, which are nearly independent, even though the positions are highly correlated from day to day. A 2-day RSI system tested over the same period would make a lot more trades but still be based on 5000 observations.

    If you fit a long-term system to something that is noise in reality, you should only lose transaction costs on average. Those costs are much higher for a short-term system.

    • david varadi permalink*
      October 30, 2009 10:42 am

      well, bear in mind the primary difference between the 200 and the RSI in this case is that your trades occured over a wide variety of different regimes so on that front Im obviously inclined to agree with you that they are different. However that said, it is hard to have a lot of confidence in moving average strategies relative to shorter term systems. That is why trend followers test rules across markets to effectively increase their sample size and to verify robustness.


  2. Jeff Pietsch permalink
    October 30, 2009 9:07 am

    A good degrees of freedom post would be in order David. So true. To a degree (no pun intended), need to be careful with regimes for that very same reason. There is a correct and an incorrect way to do it. Good post!

  3. Bill permalink
    October 31, 2009 11:58 am


    waiting for your DVO indicator stats, also would like to see month to month return for last 10 years on DVO ( 5/95 and 50/50) system.


  1. Focus On Robust Concepts Not Optimization | Financial engineering resource center

Leave a Reply

Fill in your details below or click an icon to log in: Logo

You are commenting using your account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s

%d bloggers like this: