# The Adaptive Time Machine: The Importance of Statistical Filters

**Note:** *to the sophisticated quant crowd, bootstrap tests using White’s Reality check are more approriate measures of confidence to avoid data-mining bias ie random re-sampling, and de-trending the data by subtracting S&P500 returns are done in practice.*

Returning to our little experiment https://cssanalytics.wordpress.com/2009/09/14/busting-the-efficient-markets-hypothesis-the-adaptive-market-time-machine/ and the methodology https://cssanalytics.wordpress.com/2009/09/15/creating-the-adaptive-time-machine/, recall that there are 50 strategies that our machine can choose from. It can buy or sell on any combination of up or down runs. The question is, which strategies should we focus on? Those who understand the time series of market returns understand that there is a fair amount of randomness that needs to be filtered for to avoid trading noise. One such method to filter strategies is the T-statistic http://en.wikipedia.org/wiki/T-statistic . The basics of the T-stat or T-score is that it helps you derive a % confidence that a strategy has a mean that is statistically different from zero. This is a good start, as strategies with low levels of confidence are far more likely to be random effects. Typically scientists use 95% as a benchmark for the confidence required to differentiate chance from a systematic effect. Now there are wrinkles in applying this metric to time series data, but that is beyond the scope of this article. Quants like my compadre Michael Stokes at MarketSci http://marketsci.wordpress.com/ and my expert trader friend Frank at Trading the Odds http://www.tradingtheodds.com/ both make frequent use of this statistic and the concept of confidence each in their own way.

To conduct our first test, we selected S&P500 data going back to 1955. First we tested buy and hold and then the baseline strategies traded with either a mean-reversion bias (ie buy on down runs sell on up runs), or a standard follow-through or trend bias. We then created three filters using the t-stat: 1) minimum 50% confidence 2) minimum 75% confidence and 3) minimum 95% confidence. The strategies were then traded in the appropriate direction indicate by the t-test–ie if the t-stat was negative we traded the strategy short and if it was positive we traded it long. Notice that this method is “bias-free” because we don’t care which way the strategies are traded. Confidence figures were calculated using a combination of a 3-year and 1-year time window. Note that even with 95% confidence we ended up trading 7 strategies simultaneously on average, whereas with 75% confidence we traded 24, and 32 for 50% confidence. This is in contrast to selecting the “best method.” It also helps to verify the robustness of the filter. Lets look at how this filter performs vs the various benchmarks:

Note that each level of confidence outperforms the other, and the equity curve is very smooth and adapts across a variety of regimes. The annual return and sharpe ratio are much better than buy and hold for 50%, 75%, and 95% confidence. This makes sense, as even 50% confidence will contain strategies that are 95% confidence plus. As a deep thought, the market is essentially a composite equity curve of thousands of strategies, with many being executed on the basis of judgement rather than statistics. Thus the random component of the market is actually fairly high, and avoiding low confidence strategies helps easily beat buy and hold. All strategies also handily beat either a mean reversion or follow through bias. This shows how adaptive this simple algorithm is, learning from what is working and reallocating capital. In the next series of posts we will look at other methods of selecting strategies and improving performance even more.

David and Corey, this is just outstanding!

Great stuff, David. Really looking forward to the rest of this series!

Likewise! You guys are taking the quant blogosphere to a new level! Its rare to get this depth of discussion in the public domain – keep up the great work!

Very good work gentlemen.

You should graph cumulative returns on a log scale, so that fluctuations early in the sample are apparent.

It is unfair to put all that intelligence in one person’s brain🙂. I can’t wait to read your next post.

John

intelligence……ha, i just created the “infinite monkey machine” with knowledge of statistics!

:o)

dv

Any chance you guys can share the excel file so we can pick at it?

hmm that isn’t going to happen, but i assure you i am not fabricating results.

if you dig hard enough, you should be able to reproduce them.

dv

How do you get 50 possible strategies? 2 x 2^5 = 64. And I see only 20 of the consecutive day variety, where you take the signal right away. What up/down sequences are excluded?

hi, its a 5×5 grid of run length, and you can trade that grid both ways ie buy after up run/sell down runs OR buy after down runs/sell after up runs.

thus 2x5x5 which is 50

dv

if we’re talking statistical monkeys, you can also buy down/sell down, and buy up/sell up? except for the ones where the runs =, unless the monkey is really stubborn🙂

yes you are correct, all combinations of long short are possible. its an equal opportunity monkey :o)

cheers

dv

I know this isn’t supposed to be the challenging part, but I still don’t follow. What does “run length” refer to? If it means consecutive up/down closes, then I don’t come up with 25 different cases in the span of 5 days.

Any assistance would be appreciated.

And thank you for the outstanding blog, David! I’ve certainly gained some Excel skills while attempting to recreate your indicators!

Mike

Does the “well documented daily follow-through” end up being about 40% of all trades (in the 95% scenario)? Not that the rest of the trades lose money.

I only have 3-4 strategies on average in the 95% scenario (depending on the confidence period), so it looks like i’m doing something wrong.

sorry, i was going short in the scenarios above. Now i average 9 strategies, but i think the weakness is that it varies from 3 to 31 active strategies, not that they all give signals, but how do you allocate capital between the signals and how much does the number of potentially active strategies vary for you? Rebalancing at every signal would be too expensive, i’m guessing

aha, well lets not spoil any surprises just yet! good job on your research alex, impressed with your diligence.

:o)

dv

I’m mostly a discretionary trader that has recently been afflicted with the “quant bug”.

I read 2 or 3 research papers every night and by far your exercise seems most in tune to how I conceptualize an adaptive / robust process. I’ve been pondering something like this for months.

Being a “tape reader” my Achilles Heel is simply producing charts, graphs, tables, back tests, etc. like you have done here. It’s a Possible vs Practical type of dilemma.

It’s extremely frustrating and it looks like I have a 3 ft thick glass ceiling directly above me. Was this study done in excel or did you utilize other software?

As a side note … any theory on how this algorithm may perform on a more “immature” index?

hi, this simple method even with a simple runs analysis seems to work–ie beat buy and hold on almost everything–commodities, stocks etc. It is definitely practical in real life because the strategy entails buying after up or down days and selling after either up or down days in a given sequence. Basically you just constantly do appropriate tests to see what is working using a systematic method, and you simply trade that simple approach until the statistics tell you that it is no longer working well enough to use.

everything was done in excel using visual basic code.

thanks for the kind words

dv

David, very good work so far.

But I’m surprised nobody has answered the most obvious question:

Have you tested the Time Machine on more complex strategies? If yes, were the results as impressive as these?

:o) of course, and our proprietary method works across the board. But i won’t spoil future posts! thanks mdan

dv

Hi David, is there any additional gain to going higher than 95%?

well ranking purely by confidence isn’t optimal so going above 95 isn’t the way to go, but yes it does to answer your question.

dv

Thanks, looking forward to hear more about ranking.

David, it would be interesting to see the results for different periods of time (for instance, the CAGR for periods of 5 years). This way, we could see if the market has become more efficient over time.

hi mdan, actually we plan on showing data like that this week. in a slightly different form. but good suggestion.

dv

Cool test, I tried you concept with RSI-strategies on the swedish stock market and it turn out pretty good. I guess you weight the signals based on their confidence since that showed preformance just like yours.

Cheers!

Are you aware of the need to do a correction for multiple t-tests? Since you are doing many tests you over-estimate the significance of your results. See, for example:http://en.wikipedia.org/wiki/Multiple_comparisons