Skip to content

The Adaptive Time Machine: The Importance of Statistical Filters

September 16, 2009

Note: to the sophisticated quant crowd, bootstrap tests using White’s Reality check are more approriate measures of confidence to avoid data-mining bias ie random re-sampling, and de-trending the data by subtracting S&P500 returns are done in practice.

Returning to our little experiment and the methodology, recall that there are 50 strategies that our machine can choose from. It can buy or sell on any combination of up or down runs. The question is, which strategies should we focus on? Those who understand the time series of market returns understand that there is a fair amount of randomness that needs to be filtered for to avoid trading noise. One such method to filter strategies is the T-statistic . The basics of the T-stat or T-score is that it helps you derive a % confidence that a strategy has a mean that  is statistically different from zero. This is a good start, as strategies with low levels of confidence are far more likely to be random effects. Typically scientists use 95% as a benchmark for the confidence required to differentiate chance from a systematic effect. Now there are wrinkles in applying this metric to time series data, but that is beyond the scope of this article. Quants like my compadre Michael Stokes at MarketSci and my expert trader friend Frank at Trading the Odds both make frequent use of this statistic and the concept of confidence each in their own way.

To conduct our first test, we selected S&P500 data going back to 1955. First we tested buy and hold and then the baseline strategies traded with either a mean-reversion bias (ie buy on down runs sell on up runs), or a standard follow-through or trend bias. We then created three filters using the t-stat: 1) minimum 50% confidence 2) minimum 75% confidence and 3) minimum 95% confidence. The strategies were then traded in the appropriate direction indicate by the t-test–ie if the t-stat was negative we traded the strategy short and if it was positive we traded it long. Notice that this method is “bias-free” because we don’t care which way the strategies are traded. Confidence figures were calculated using a combination of a 3-year and 1-year time window. Note that even with 95% confidence we ended up trading 7 strategies simultaneously on average, whereas with 75% confidence we traded 24, and 32 for 50% confidence. This is in contrast to selecting the “best method.” It also helps to verify the robustness of the filter. Lets look at how this filter performs vs the various benchmarks:


PerformanceChart101Note that each level of confidence outperforms the other, and the equity curve is very smooth and adapts across a variety of regimes. The annual return and sharpe ratio are much better than buy and hold for 50%, 75%, and 95% confidence. This makes sense, as even 50% confidence will contain strategies that are 95% confidence plus. As a deep thought, the market is essentially a composite equity curve of thousands of strategies, with many being executed on the basis of judgement rather than statistics. Thus the random component of the market is actually fairly high, and avoiding low confidence strategies helps easily beat buy and hold. All strategies also handily beat either a mean reversion or follow through bias. This shows how adaptive this simple algorithm is, learning from what is working and reallocating capital. In the next series of posts we will look at other methods of selecting strategies and improving performance even more.

33 Comments leave one →
  1. September 16, 2009 7:04 pm

    David and Corey, this is just outstanding!

  2. CarlosR permalink
    September 16, 2009 7:26 pm

    Great stuff, David. Really looking forward to the rest of this series!

  3. Ramon permalink
    September 16, 2009 7:44 pm

    Likewise! You guys are taking the quant blogosphere to a new level! Its rare to get this depth of discussion in the public domain – keep up the great work!

  4. Jeff permalink
    September 16, 2009 8:55 pm

    Very good work gentlemen.

  5. quant permalink
    September 17, 2009 7:40 am

    You should graph cumulative returns on a log scale, so that fluctuations early in the sample are apparent.

  6. John permalink
    September 17, 2009 8:26 am

    It is unfair to put all that intelligence in one person’s brain :-). I can’t wait to read your next post.


    • david varadi permalink*
      September 17, 2009 3:00 pm

      intelligence……ha, i just created the “infinite monkey machine” with knowledge of statistics!


  7. Alex permalink
    September 17, 2009 10:58 am

    Any chance you guys can share the excel file so we can pick at it?

    • david varadi permalink*
      September 17, 2009 2:59 pm

      hmm that isn’t going to happen, but i assure you i am not fabricating results.
      if you dig hard enough, you should be able to reproduce them.


  8. Alex permalink
    September 17, 2009 2:17 pm

    How do you get 50 possible strategies? 2 x 2^5 = 64. And I see only 20 of the consecutive day variety, where you take the signal right away. What up/down sequences are excluded?

    • david varadi permalink*
      September 17, 2009 2:58 pm

      hi, its a 5×5 grid of run length, and you can trade that grid both ways ie buy after up run/sell down runs OR buy after down runs/sell after up runs.

      thus 2x5x5 which is 50


      • Alex permalink
        September 17, 2009 3:25 pm

        if we’re talking statistical monkeys, you can also buy down/sell down, and buy up/sell up? except for the ones where the runs =, unless the monkey is really stubborn 🙂

      • david varadi permalink*
        September 17, 2009 3:28 pm

        yes you are correct, all combinations of long short are possible. its an equal opportunity monkey :o)

      • Mike permalink
        September 18, 2009 10:57 pm

        I know this isn’t supposed to be the challenging part, but I still don’t follow. What does “run length” refer to? If it means consecutive up/down closes, then I don’t come up with 25 different cases in the span of 5 days.
        Any assistance would be appreciated.
        And thank you for the outstanding blog, David! I’ve certainly gained some Excel skills while attempting to recreate your indicators!

  9. Alex permalink
    September 17, 2009 6:14 pm

    Does the “well documented daily follow-through” end up being about 40% of all trades (in the 95% scenario)? Not that the rest of the trades lose money.

    I only have 3-4 strategies on average in the 95% scenario (depending on the confidence period), so it looks like i’m doing something wrong.

    • Alex permalink
      September 17, 2009 7:08 pm

      sorry, i was going short in the scenarios above. Now i average 9 strategies, but i think the weakness is that it varies from 3 to 31 active strategies, not that they all give signals, but how do you allocate capital between the signals and how much does the number of potentially active strategies vary for you? Rebalancing at every signal would be too expensive, i’m guessing

      • david varadi permalink*
        September 17, 2009 7:12 pm

        aha, well lets not spoil any surprises just yet! good job on your research alex, impressed with your diligence.


  10. Dwyer permalink
    September 17, 2009 11:36 pm

    I’m mostly a discretionary trader that has recently been afflicted with the “quant bug”.

    I read 2 or 3 research papers every night and by far your exercise seems most in tune to how I conceptualize an adaptive / robust process. I’ve been pondering something like this for months.

    Being a “tape reader” my Achilles Heel is simply producing charts, graphs, tables, back tests, etc. like you have done here. It’s a Possible vs Practical type of dilemma.

    It’s extremely frustrating and it looks like I have a 3 ft thick glass ceiling directly above me. Was this study done in excel or did you utilize other software?

    As a side note … any theory on how this algorithm may perform on a more “immature” index?

    • david varadi permalink*
      September 17, 2009 11:40 pm

      hi, this simple method even with a simple runs analysis seems to work–ie beat buy and hold on almost everything–commodities, stocks etc. It is definitely practical in real life because the strategy entails buying after up or down days and selling after either up or down days in a given sequence. Basically you just constantly do appropriate tests to see what is working using a systematic method, and you simply trade that simple approach until the statistics tell you that it is no longer working well enough to use.
      everything was done in excel using visual basic code.
      thanks for the kind words

  11. MDan permalink
    September 18, 2009 10:13 am

    David, very good work so far.

    But I’m surprised nobody has answered the most obvious question:

    Have you tested the Time Machine on more complex strategies? If yes, were the results as impressive as these?

    • david varadi permalink*
      September 18, 2009 2:08 pm

      :o) of course, and our proprietary method works across the board. But i won’t spoil future posts! thanks mdan


  12. Kevin permalink
    September 18, 2009 2:35 pm

    Hi David, is there any additional gain to going higher than 95%?

    • david varadi permalink*
      September 18, 2009 2:50 pm

      well ranking purely by confidence isn’t optimal so going above 95 isn’t the way to go, but yes it does to answer your question.


  13. Kevin permalink
    September 18, 2009 8:40 pm

    Thanks, looking forward to hear more about ranking.

  14. MDan permalink
    September 20, 2009 8:40 am

    David, it would be interesting to see the results for different periods of time (for instance, the CAGR for periods of 5 years). This way, we could see if the market has become more efficient over time.

    • david varadi permalink*
      September 20, 2009 9:50 am

      hi mdan, actually we plan on showing data like that this week. in a slightly different form. but good suggestion.


  15. November 21, 2010 10:03 am

    Cool test, I tried you concept with RSI-strategies on the swedish stock market and it turn out pretty good. I guess you weight the signals based on their confidence since that showed preformance just like yours.


  16. John Nicholas permalink
    April 11, 2013 8:58 pm

    Are you aware of the need to do a correction for multiple t-tests? Since you are doing many tests you over-estimate the significance of your results. See, for example:


  1. Evening Briefing for September 16th |
  2. Link Feast 3-1-10! « The Edge
  3. (Part 2) Time Machine Test – Non-parametric Statistical Filter « Quantum Financier
  4. Blog Profile: CSS Analytics « Financialfreezeframe's Blog
  5. Coming This Week: The Adaptive Time Machine Returns « CSS Analytics

Leave a Reply

Fill in your details below or click an icon to log in: Logo

You are commenting using your account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s

%d bloggers like this: