Skip to content

Introducing Composite Mean Reversion and Trend Following Measures: The Aggregate “M” Indicator

November 5, 2009

Note: Apparently there were some questions about how to calculate Aggregate M, please email us at dvindicators@gmail.com to receive spreadsheet later this weekend. Note we use a cleaner version of data than Yahoo Finance adjusted for splits and dividends with verified closing prices. I have been made aware that the backtest results using Yahoo are different from ours (note we calculated the same values as our readers using Yahoo Finance), and that is because their data is not entirely accurate.

The Aggregate M indicator is based on the concept that in the long term the market trends, while in the short-term the market is  noisy, and has a tendency to mean-revert. Why not combine the two concepts to keep life simple?  The Aggregate M  is supposed to reflect an adjusted median that is filtered for short term noise. The median is a far more accurate measure of central tendency than a simple average especially with noisy data.  Taking a superior measure of trend and filtering out some of the noise by adjusting for short-term mean reversion creates an even better median. The Aggregate M is now both trend and mean-reversion rolled into one. In the example below the aggregate M is simply the average of 1) 252 day PERCENTRANK of High, Low and Close values and 2) 10 day (1-PERCENTRANK) of High, Low and Close values. This average is smoothed using a .6 weight on today and .4 weight on yesterday. Is this robust? The S&P500 results over the last 4000 bars speak for themselves–high accuracy, good gains per trade and a nice equity curve. In a separate multimarket test with 20 markets going back to 1984, the Aggregate M did 27% CAGR through 2009. Is it the best method…….probably not—I certainly didn’t spend any time optimizing or digging as I came up with it just yesterday. Furthermore the structure can be substantially improved. But pretty damn good for two common sense parameters!

SPYAggM5050

55 Comments leave one →
  1. roy permalink
    November 6, 2009 12:50 am

    looks too good to be true! ponzi?

  2. Peter Gum permalink
    November 6, 2009 6:29 am

    Are the mathematical (programming) details available anywhere? (C#, Java, Metastock,. . .) The ability to study more detail would help in obtaining a fuller comprehension.

    • david varadi permalink*
      November 6, 2009 10:53 am

      hi peter……i don’t have the programming details in those platforms. I simply did this in Microsoft Excel. The Aggregate M and other indicators will be available on my site http://www.dvindicators.com for free coded for tradestation when I launch (shortly).

      cheers
      dv

      • Peter Gum permalink
        November 7, 2009 9:34 am

        Thanks David. In comment #21 you offer a spreadsheet. I would appreciate a copy: petergum at world.oberlin dot edu

        Thank you. Nice system. -Pete

  3. Frank permalink
    November 6, 2009 6:59 am

    Hello David,

    the result looks great. I of course wanted to reproduce this in my own environment. However i get significantly more trades (+101.) Would you please be so kind and look at these few lines of code of the “Aggregate M” Indicator… i can also send you an XLS if you want.

    Thank you so much!
    Frank

    ————————

    h_rank_long = PERCENTRANK(High,252);
    l_rank_long = PERCENTRANK(Low,252);
    c_rank_long = PERCENTRANK(Close,252);
    h_rank_short = 1-PERCENTRANK(High,10);
    l_rank_short = 1-PERCENTRANK(Low,10);
    c_rank_short = 1-PERCENTRANK(Close,10);

    value = ((h_rank_long + l_rank_long + c_rank_long) + (h_rank_short + l_rank_short + c_rank_short))/6;

    AggregateM = (value[1]*0.4) + (value*0.6);

    • david varadi permalink*
      November 6, 2009 10:55 am

      hi frank, thanks a lot…..please see Ramon’s response which is correct……you are taking the PERCENTRANK of each individual column while I take the percentrank of the entire array including the High Low and Close values.

      cheers
      dv

  4. Damian permalink
    November 6, 2009 7:35 am

    Questions:

    – I assume the percentrank of the high,low,open,close is just that – a percentrank across all four columns?
    – The weighting confuses me – “This average is smoothed using a .6 weight on today and .4 weight on yesterday.” – so it’s not a straight average?

    • david varadi permalink*
      November 6, 2009 10:58 am

      hi Damia, it is the percentrank of the 3 columns of the H, L, C array of price. I think the wording is a little confusing, the aggregate measure of central tendency of the two percentiles is averaged (smoothed) with this weighting. Practically speaking, it was already “averaged” by taking .5 times one percentile and .5 times the other. That is why I call it a smoothing.
      just semantics!

      cheers
      dv

  5. November 6, 2009 8:07 am

    David – this looks rather interesting and novel (to me at least) concept…
    A question on your charts: how do you generate them – is it from a special software/platform. It looks really neat (especially with the 4 plots on one chart)?
    Thanks
    Jez

    • david varadi permalink*
      November 6, 2009 11:00 am

      Hi Jez, Corey my partner in crime generates these charts using Visual Basic code in Excel etc………don’t ask me how he does it because I certainly can’t work this kind of wizardry myself!
      thanks
      cheers
      dv

  6. Ramon permalink
    November 6, 2009 9:28 am

    Frank, this is my understanding:

    rank_Long = PERCENTRANK(HLC 252 period Array, Close);
    rank_Short = PERCENTRANK(HLC 10 period Array, Close);

    ***Note the HLC array is simply three columns of data in stead of one, i.e. =PERCENTRANK(D9:F260,F260)

    Value = (rank_Long + rank_Short)/2
    AggregateM = (value[1]*0.4) + (value*0.6);

    Hope this helps
    Cheers Ramon

    • david varadi permalink*
      November 6, 2009 11:00 am

      thanks ramon as always!
      cheers
      dv

    • Damian permalink
      November 6, 2009 11:20 am

      Hey Ramon – I was looking doing this in AB – the issue is that the PercentRank function you so nicely created can’t handle multiple arrays…any thoughts on that?

  7. Ramon permalink
    November 6, 2009 11:25 am

    Hi Damian, yes you are right – need to tweak that percentRank function a little. I will post some amibroker code I baked earlier here shortly . . .

  8. November 6, 2009 1:25 pm

    Can’t wait to see it Ramon. Thanks!

  9. Jeff permalink
    November 6, 2009 2:09 pm

    Excellent, as usual. Jeff

  10. John French permalink
    November 6, 2009 3:12 pm

    I am using NDO on SPY (actually MOC is not a whole lot different) with Yahoo data (Date, O, H, L, C in columns a-e). Formulas are

    H=PERCENTRANK(C21:E272,E272) (hard-coded to 252 for the time being)
    I=1-PERCENTRANK(C263:E272,E272) (hard-coded to 10 for the time being)
    J=(H272+I272)/2
    =(J272*$K$19)+(J271*$K$18) – where K19=0.6 and k18=0.4

    I get a CAGR of ~12

    • david varadi permalink*
      November 6, 2009 3:22 pm

      hi john, did you also go short? Did you go back to 1993? i can assure you the results i have are correct.
      cheers
      dv

      • John French permalink
        November 6, 2009 3:51 pm

        David, I am shorting and my data does go back to 1993 but due to the lead of 252 days my first trade is a short on 3/17/94. I can send you my ss if you want (if you don’t I understand!)

        I am sure I am doing what Ramon is doing…

      • david varadi permalink*
        November 6, 2009 4:59 pm

        hi if you send me an email i can forward you a spreadsheet to match up with Corey’s
        cheers
        dv

      • Jason permalink
        November 8, 2009 5:35 pm

        Hi David,

        Can u send me a copy of your spreadsheet as well, it looks very interesting.

        my email is lazyboy1628 a t yahoo d ot com

      • Frank permalink
        November 9, 2009 2:41 am

        Hi David,

        i received the XLS, Thanks. Backtesting the strategy/indicator on my own platform will result in a different performance. Largely due to the fact that i’ve got different data.

        What’s YOUR data source / provider?

        Regards,
        Frank

  11. Ramon permalink
    November 7, 2009 7:43 am

    Ok guys, the appropriate High-Low-Close PercentRank function for Amibroker is below. Please note that this function ranks whatever is provided in the ‘Data3’ argument, so an example function call would that ranks the close would be:

    Value1 = PercentRankHLC(High,Low,Close,252);

    Enjoy!

    cheers

    Ramon

    function PercentRankHLC( Data1,Data2,Data3,Periods )
    {
    Count = 0;

    for ( i = 0; i Ref( Data1, -i ), 1, 0 );
    Count = Count + IIf ( Ref( Data3, 0 ) > Ref( Data2, -i ), 1, 0 );
    Count = Count + IIf ( Ref( Data3, 0 ) > Ref( Data3, -i ), 1, 0 );

    }

    return 100 * Count / (Periods*3-1);
    }

    • November 22, 2009 7:24 am

      Hi Ramon,
      Thanks for your function! Seems like you lost a couple of lines when pasting, guess it should be like this?

      function PercentRankHLC( Data1,Data2,Data3,Periods )
      {
      Count = 0;

      for (i = 0; i Ref( Data1, -i ), 1, 0 );
      Count = Count + IIf ( Ref( Data3, 0 ) > Ref( Data2, -i ), 1, 0 );
      Count = Count + IIf ( Ref( Data3, 0 ) > Ref( Data3, -i ), 1, 0 );
      }

      return 100 * Count / (Periods*3-1);
      }

      Regards
      js

      • November 22, 2009 7:28 am

        Paste error, happened to me as well trying again. Regards js

        function PercentRankHLC( Data1,Data2,Data3,Periods )
        {
        Count = 0;

        for (i = 0; i Ref( Data1, -i ), 1, 0 );
        Count = Count + IIf ( Ref( Data3, 0 ) > Ref( Data2, -i ), 1, 0 );
        Count = Count + IIf ( Ref( Data3, 0 ) > Ref( Data3, -i ), 1, 0 );
        }

        return 100 * Count / (Periods*3-1);
        }

      • November 22, 2009 7:34 am

        …i Ref( Data1, -i ), 1, 0 );

      • November 22, 2009 7:46 am

        function PercentRankHLC(Data1, Data2, Data3, Periods)
        {
        Count = 0;
        for (i = 0; i < Periods + 1; i++)
        {
        Count = Count + IIf(Ref(Data3, 0) > Ref(Data1, -i), 1, 0);
        Count = Count + IIf(Ref(Data3, 0) > Ref(Data2, -i), 1, 0);
        Count = Count + IIf(Ref(Data3, 0) > Ref(Data3, -i), 1, 0);
        }
        return 100 * Count / (Periods*3-1);
        }

  12. Ramon permalink
    November 7, 2009 7:48 am

    hmm, looks like the paste of the code didn’t work properly, if you want the code email me at ramon a t minkcapital d ot com

    • November 7, 2009 11:42 am

      U da man Ramon! Thanks…email on its way…

    • darshan permalink
      July 29, 2010 12:50 pm

      Hello Ramon,

      can you please repost the full formula !

      Oh god …i am too late..

      thank you
      *sorry for trouble

  13. Kevin permalink
    November 8, 2009 9:20 pm

    Hi David,

    I note that the results are critically dependent on the data quality. Also, on whether the instrument is tradeable. I once found a simple system with incredible results on the ^DJI index before being disappointed to find that it did not work on the ETF. This was due to the highs and lows on the index calculated as the mean of the component highs and lows without accounting for the time they were made.

    Anyway, could you give a hint on what you mean by improving the structure? Is it to do with taking more trend and countertrend components or something to do with the averaging?

    Thanks

    Kevin

    • david varadi permalink*
      November 9, 2009 2:02 am

      hi kevin, improving the indicator is something that can be done at many levels…….first the choice of periods is somewhat arbitrary, as is the number of different periods to take into account. The weighting placed on each period is also arbitraray and not per se optimal. Furthermore the assumption of mean reversion on a short term window may or may not hold true for a given instrument at a given time.

      i could go on, but i can’t give away all the secrets! :o)

      cheers
      dv

  14. November 9, 2009 9:02 am

    I have run this strategy over my data, (I use Norgate), and I get approximately 12% CAGR. As best as I can tell, there are at least several reasons for the discrepancy between DV’s results and mine.
    1. DV is using dividend adjusted data. When dividends are re-invested and compounded, it is going to make a noticeable difference in returns.
    2. The start date of testing. Unless you have specifically programmed your backtester to use 4000 bars, your results may be different. Also, several market days have passed since this article is published, and so now you may need to use slightly more than 4000 bars to replicate the results.
    3. DV hasn’t covered this, but it is worth noting. For these types of setups, I require a cross of the AggM above 50. What this means is that when the testing starts, without requiring a cross, your platform may see that the AggM value is over 50, and therefore initiate a long trade. In real-time, you wouldn’t have taken the trade on that day. If you are not requiring a cross, this will also affect the beginning of the backtest and may cause your results to be slightly different from others.
    4. Some platforms will make you wait until there are enough bars to calculate the averages, while other platforms, if not specifically told not to, will look back further than 4,000 bars so that the averages will be calculated. This too will cause discrepancies.

    • November 9, 2009 9:24 am

      Whoops, messed up #2.
      What I meant to say was that since some market days have passed, you’ll have to set your backtester to test from 12/20/1993-11/04/09, or else new data will be included.

  15. Kevin permalink
    November 9, 2009 2:53 pm

    Hi David,

    Thanks, that’s very useful. I will do some walk forward optimization and see if I get anything worth sending over. FWIW I get 11% CAGR also using Yahoo data on the SPY.

    Kevin

  16. kostas permalink
    November 10, 2009 12:07 am

    In the algorithm the treatment of the short-term “noise”/mean reversion is “=1-PERCENTRANK(C2:E11,E2)”. David, why the “1-” formulation>

  17. kostas permalink
    November 10, 2009 12:41 am

    Hi DV–A question regarding your testing: Did you model the buys to take place at the open Or the close of the day AFTER the signal switch? (Hopefully NOT on the same day at the close–right?)

  18. Jack permalink
    November 10, 2009 2:56 pm

    Can you list the 20 markets that you used in the separate test ?

    • Corey permalink*
      November 10, 2009 3:14 pm

      Hi Jack, The markets traded in the separate test were:
      (all Futures) S&P 500, Nasdaq 100, Gold, Silver, Copper, Oil, Heating Oil, Natural Gas, Gasoline, Corn, Wheat, Soybeans, Cotton, Coffee, Cocoa, Live Cattle, Lean Hogs, Sugar, Australian Dollar, Pound, Yen, Euro, Swiss Franc.

  19. Dave permalink
    December 8, 2009 7:39 pm

    Hey all

    Looks like I’m late to the party on this one. I’m fascinated by this and would really like to play around with it. I’ve attempted to apply it to excel, but it’s not working for me. I was wondering if a kind soul had an excel sheet with the formula working that I could use as a base. My email is do d*t evans at yahoo d*t co d*t uk

    Thanks in advance

    Dave

  20. Joe Marc permalink
    December 16, 2009 9:23 am

    David/Corey
    I am not anywhere near you folks on math/computers etc. Simple question from a simple guy. I have been daily/manually updating AG M with Excel since I got it (about 7 weeks now), and I have had consistent reading >.5
    Is this correct? Does that jibe with your readings?
    Just checking.
    Many, many thanks
    Joe Marc

  21. February 4, 2010 12:26 am

    Could I get a copy of the XLS for he Aggregate “M” Indicator?
    Thanks
    Scott

  22. Kevin permalink
    March 3, 2010 11:13 am

    David:

    Could you send me your Agg M excel spreadsheet as well?

    Thanks,

    Kevin

  23. Larry permalink
    April 13, 2010 9:21 pm

    David or Ramon, I need clarification on how the PercentRank works across 3 columns. The Excel help file only illustrate its use on one column of data.

    It seems to me that if each ROW is looking for one value to rank, then the Close is superfluous as it always will fall within the values for High and Low.

    Is Percent Rank averaging the three columns? i.e (H+L+C)/3 and then ranking that row as compared to other rows?

    Thanks, keep up the great posts!

  24. Ramon permalink
    April 14, 2010 4:25 am

    Hi Larry,

    When you percentrank across three columns you are actually ranking all the data as if it were in one column or array – therefore the close will have an impact as it will increase the number of data points across which you are ranking.

    Hope this makes sense!

    Ramon

  25. Yo-Man permalink
    April 19, 2010 1:47 pm

    Hi David,

    I discovered this website through Dr. Steenbarger’s Traderfeed. I found the works here very interesting and would like to receive a copy of the XLS spreadsheet if possible. My email is ymanyen@gmail.com

    Thank you,

    Yo-Man

  26. Brian permalink
    July 15, 2010 12:21 am

    Hi David,

    I’m in the process of my thesis study which tests a number of publicly available trading strategies/indicators in Tradestation against the forex market (daily and intraday) and would love to be able to try this indicator out. I have a few questions/comments if I may:
    1. Will Aggregate M work on other markets besides the S&P500?
    2. Will it work intraday as well as it does daily?
    3. If the spreadsheet is still available, I would greatly appreciate a copy. You can reach me at BrianLeip at G m-ail.
    4. It would be preferable to understand some of the methodology behind the indicator (as with all the other strategies/indicators I am using) rather than just relegate it to a “black box”. Are there any blog posts/write-ups re: methodology that I could review?

    Thanks in advance for your time. It looks quite impressive by the way

    – Brian

    • david varadi permalink*
      July 15, 2010 2:32 am

      hi brian, i will get back to you on this. please send me an email personally.
      best
      dv

  27. Omar Westerberg permalink
    July 4, 2012 10:04 am

    Hi, thanks for the post. If the spread sheet is still available I would enjoy a copy to evaluate the methods to use with pairs and some other arbitrage techniques. If available please send to o.westerberg@gmail.com
    Thanks, Omar

  28. May 11, 2013 4:45 pm

    You could certainly see your enthusiasm within the work you
    write. The world hopes for more passionate writers such as you who aren’t afraid to say how they believe. All the time go after your heart.

Trackbacks

  1. Trend or Mean-Reversion? The Strategy Allocation Problem (part 1) « CSS Analytics
  2. Varadi’s Aggregate M Indicator: Get Thee to Cash | System Trading with Woodshedder
  3. Varadi’s AggM Indicator Suggesting a Move to Cash | System Trading with Woodshedder
  4. Weekend Reading |

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s

%d bloggers like this: