An Interview with Dr. Ernest Chan

“A lot of my ideas are just based on logic”

In the last post I reviewed the Momentum Trading Strategies Course by Quantra (a division of QuantInsti) which I reviewed as part of a recent educational journey to improve my quantitative skill set. The next course that I will be reviewing is Mean-Reversion Strategies in Python which is taught by Dr. Ernest Chan. I have personally read Ernie’s book “Machine Trading” which is very well written and full of interesting and practical ideas. I have also been a follower of his very popular blog which was a pioneer in revealing statistical arbitrage strategies such as pairs-trading. Dr. Chan is a thought leader and industry expert and anyone who is in the quantitative field has inevitably come across his work in one form or another. I reached out to interview him to get a few insights into how to think about quantitative models in the modern era. For those that are unfamiliar with Dr. Chan’s work I have provided his very impressive (and extensive) industry and educational credentials below.

Industry Background: Dr. Chan is the Founder of PredictNow.ai, a financial machine learning SaaS, and also the Managing Member of QTS Capital Management, LLC., a commodity pool operator and trading advisor. His primary focus has been on the development of statistical models and advanced computer algorithms to find patterns and trends in large quantities of data. He has applied his expertise in statistical pattern recognition to projects ranging from textual retrieval at IBM Research, mining customer relationship data at Morgan Stanley, and statistical arbitrage trading strategy research at Credit Suisse, Mapleridge Capital Management, and other hedge funds.

Educational Background

Dr. Chan is an industry expert on ‘Algorithmic Options Trading’ and has conducted seminars and lectures on many international forums. Besides being a faculty in QuantInsti, his academic distributions are available on Quantra and on major web portals. Dr. Chan is also an adjunct faculty at Northwestern University’s Master’s in Data Science program. His courses and publications on finance and machine learning can be found at www.epchan.com. Ernie is the author of “Quantitative Trading: How to Build Your Own Algorithmic Trading Business”, “Algorithmic Trading: Winning Strategies and Their Rationale”, and “Machine Trading”, all published by John Wiley & Sons. He maintains a popular blog “Quantitative Trading” at epchan.blogspot.com. Ernie received his PhD. in Physics from Cornell University.

Interview with Dr. Ernest Chan

1) What do you think about traditional factor investing and trading strategies that use technical indicators? Can they be profitable in the modern environment? 

We don’t use machine learning to generate trading signals, but rather to determine the probability of profit of the existing trading signals generated by a basic, traditional quantitative strategy. This strategy can be a factor model or one based on simple technical indicators. This probability of profit can then be used to determine the order size, which can be zero if the probability is too low. 
Factors and technical indicators are still crucial for the basic strategy. I don’t believe that machine learning can replace human intuition and understanding. In fact, it should be used to enhance such understanding and risk management. The input to a machine learning algorithm is nothing but factors and technical indicators.

2) If you could choose between a Momentum and Mean-Reversion approach to trading which would you choose and why?  

I would trade both. Otherwise the portfolio would not be market neutral since momentum strategies are typically short beta while mean reversion strategies are long. Also, momentum strategies are long “gamma” and “vega”, while mean reversion strategies are short. Note that I put quotation marks around such options Greeks because we are not really trading options nor implied volatility. I am using these terms loosely to indicate an increase in tail movements and realized volatility.

3) Why should traders strongly consider using machine learning in their trading versus hand-coding their own quantitative systems or using more simple statistical tools? For traders that aren’t familiar with coding what do you think is the best way to get started?

Traditional quant strategies are too easily replicated by other equally intelligent traders, hence they suffer more rapid alpha decay. ML strategies have so many parameters and nuances that no two traders can possibly have the same strategy. For traders who are not experts in machine learning or programming can start with a no-code machine learning service such as predictnow.ai. 

4) What do you think is the biggest challenge for newbies trying to design their own machine-learning models?  

Machine learning requires abundant and correctly engineered features as input. I have seen many newbies trying to use 4 or 5 inputs to a ML algorithm. They should instead be using at least 100 inputs.  

5) Do you have a preference in terms of the type of machine learning model you use such as Neural Networks vs KNN or Decision Trees? If so why?

Decision trees, or the more advanced version called random forest, is the preferred ML method for trading. That’s because it doesn’t have as many parameters to fit as a neural network, thus reducing the danger of data snooping bias. Also, the output of a decision tree is a bunch of conditional decision rules, which are much easier to interpret than the nonlinear functions that neural networks use. On the other hand, KNN or logistic regression are too simple – they don’t capture a lot of the nonlinear dependence between different input features and the output return. 

6)  Many traders and market commentators have noticed that markets seem quite a bit different than in the past. The market seems to move much more quickly and reacts to news in ways that are counterintuitive.  Given your vast experience with algorithmic trading what new trends or insights have you gathered in the last few years? Have you made any specific adjustments or recalibrated your models accordingly?

Market patterns often deviate from the “norm” over a short period (e.g. 6 months-1 year), but they often revert to the norm. One needs to diversify so that some strategies are enhanced during such periods, even though others are hurt. Such regime changes can also be detected or predicted to some extent by machine learning.

Thanks Ernie for the interview!