S&P500 and Nasdaq Composite Correlations Yield Valuable Information
The S&P500 and the Nasdaq Composite are two very different indices; the S&P is an index of the 500 largest companies by market capitalization, while the Nasdaq Composite is an index of smaller and more growth oriented companies. The two share an interesting and complex relationship–which is a topic for another article. In examining that relationship, we can measure the correlation between the two indices to see if that yields valuable information. In theory a weakening correlation between the two indices would indicate a healthy market where the movement of stocks is somewhat independent of broad market factors. This weak correlation in turn drives down index level volatility, and increases daily persistence as stocks develop lagging relationships with their peers. This allows for different stocks to take turns leading the market—driving the index higher relentlessly. In this sense, a falling correlation regime (such as the one we are in right now) it is a “market of stocks” rather than a stock market and more alpha can be generated by trading individual stocks than at the market level. In contrast, a strong correlation would indicate the presence of fear–each company is treated as a source of cash in periods of mass liquidation, and they are all sold simultaneously with little discrimination between whether they are good or bad. This high correlation drives up market volatility, and the result is a low-return environment with excessive standard deviation. In this environment, alpha is best generated by trading “beta” by harvesting the surface mean-reversion of the index. Lets look at some evidence:
| (1974 to present—last 9000 bars) | ||
| S&P500 Avg Daily | ||
| S&P500 Avg Daily | Buy and Hold Return | |
| return in given condition | (non-compounded) | |
| 20d correlation is above 1yr median | 0.023% | 0.035% |
| 20d correlation is below 1yr median | 0.046% | |
| Nasdaq Comp Avg Daily | ||
| Nasdaq Comp Avg Daily | Buy and Hold Return | |
| return in given condition | (non-compounded) | |
| 20d correlation is above 1yr median | 0.019% | 0.043% |
| 20d correlation is below 1yr median | 0.072% | |
| (1974 to present—last 9000 bars) | ||
| S&P500 Avg Daily | ||
| S&P500 Avg Daily | Buy and Hold Return | |
| return in given condition | (non-compounded) | |
| 20d correlation is rising | 0.012% | 0.035% |
| 20d correlation is falling | 0.058% | |
| Nasdaq Comp Avg Daily | ||
| Nasdaq Comp Avg Daily | Buy and Hold Return | |
| return in given condition | (non-compounded) | |
| 20d correlation is rising | 0.025% | 0.043% |
| 20d correlation is falling | 0.071% | |
As you can clearly see, average daily returns are much higher for both indices when 20-day correlations are either below the median, or falling, far superior to buy and hold returns over the same period. In contrast the rising or above median correlations were not favorable, and underperformed buy and hold returns over the same time period. Now the next logical question is how this effect is moderated by the previous levels of correlation in the recent past. Presumably lower or falling correlations would be even more beneficial when the market correlation was higher in previous periods. A good example would be following a market correction, a fall in correlations brought about by increases in speculation and a decline in fear should signal strong gains. To test this, I looked at whether the 20-day correlation was above or below the median 30-days ago. Here are the results:
| (1974 to present—last 9000 bars) | ||
| S&P500 Avg Daily | ||
| S&P500 Avg Daily | Buy and Hold Return | |
| return in given condition | (non-compounded) | |
| 20d correlation rising, correlation higher | -0.003% | 0.035% |
| than median 30 days ago | ||
| 20d correlation rising, correlation lower | 0.030% | |
| than median 30 days ago | ||
| 20d correlation falling, correlation higher | 0.095% | |
| than median 30 days ago | ||
| 20d correlation falling, correlation lower | 0.023% | |
| than median 30 days ago | ||
| (1974 to present—last 9000 bars) | ||
| Nasdaq Comp Avg Daily | ||
| Nasdaq Comp Avg Daily | Buy and Hold Return | |
| return in given condition | (non-compounded) | |
| 20d correlation rising, correlation higher | 0.004% | 0.043% |
| than median 30 days ago | ||
| 20d correlation rising, correlation lower | 0.048% | |
| than median 30 days ago | ||
| 20d correlation falling, correlation higher | 0.115% | |
| than median 30 days ago | ||
| 20d correlation falling, correlation lower | 0.027% | |
| than median 30 days ago | ||
Consistent with predictions, returns are even stronger when correlation is falling from a higher base. What was interesting was that the most negative situation was when correlations were rising and coming off a higher base. This is in contrast to findings that high and rising volatility are actually positive in the short term as they tend to preclude bottoms. The answer is, as alluded to before in a previous post–correlations actually lead volatility–the increased relationship between previously unrelated stocks mathematically increases the “capture rate” of the average of all stock volatilities. Observing the leading variable–correlations- allows you to see “cracks” in the market, and is thus a valuable timing tool.
CSSA 
Hi, Dave. Thanks for the brain candy!
Attempting to reproduce your work here, I got stronger signals (more significant T-stats and larger return differences) by defining ‘rising’ based on the one-day difference in 20-day correlations rather than the 30-day difference you describe. One-day differencing causes faster regime switching as you might expect.
I’m curious if that provokes any further thoughts.
hi top, i’m not sure if I am confusing what you are referring to, but i did in fact use the one day difference, the 30-day was to identify a past regime.
thoughts?
cheers
dv
I had similar results TopTick. On an aside, I always find it useful to view the equity curves on these assertions to ascertain stability over time.
hi ramon, i agree with looking at equity curves—-i didn’t have corey available to run these tests and throw up a chart—–so we had to settle for the
primitive avg daily returns :o) however I did notice that the effect was strong the last 3000 bars.
cheers
dv
Great stuff again DV
I was about to apply some analysis of a VIX based strategy for the S&P. I noticed that the rolling correlation between the vix & SPX varies between 0 and -1 (chance and 100% negative). So I wondered if you could time a strategy based on the VIX by using the rolling correlation as a filter. You’ve given me some great ideas thanks.
hi, thanks very much—i have done a little testing using the VIX and in general the correlation between the VIX and the SPY is less important than say the relationship between VIX and VXV or VIX and historical volatility. other areas in the short-term that are important are divergences between the SPY daily return vs the VIX—ie if SPY rises and VIX rises, or if the SPY falls but VIX falls also.
best
dv
Hello David – are you using the correlation between the daily % changes or between the index prices themselves? I ran some quick numbers on the above and I’m only getting stark and consistent differences on the index prices. thanks, ms
hi michael im using index prices with a 20 day correlation and a 252-day percentrank of that 20d correlation. happy to send you a spreadsheet if you like.
cheers
dv
Sure, a spreadsheet would be helpful.
I’m curious as to what the implications are of the fact that the observation only really holds true on price rather than daily % change.
Correlation based on price itself is very subject to where the market is today relative to the entire dataset.
Correlation based on daily % change is correlation in the way we traditionally think of it (i.e. to what extent are these two datasets moving t/g day-to-day).
Just thinking aloud.
michael
hi, i think the compounding effect is being captured with the prices themselves versus the returns themselves. thus perhaps capturing the relative strength effect to some degree, but the other results suggest that it works both ways–and appears to be superior to the relative strength effect. your thoughts?
cheers dv
Interesting…not sure what I think yet…will let that brew in the noggin’ for a bit. If you can send that worksheet whenever you get a chance, I’d appreciate it – want to make sure I’m on the same page. tx, michael
Hello, I found your article inspiring. Do you still keep the spreadsheet file? I think it would be good for other readers.