Chapter 1: Backtesting and Automated Execution

Common Pitfalls of Backtesting

Look Ahead Bias:

Look ahead bias occurs when we use prices available in the future to determine today’s trading signals. For instance, using a day’s high or low price to determine the entry signal during the same day results in look ahead bias since we can know a day’s high and low prices only after the day has ended.

Data-Snooping Bias:

Data snooping bias occurs when we have too many free parameters, resulting in an overfitted model that memorizes random noise in the training data and does not perform well on out-of-sample data.

A good way to check for data-snooping bias is to do cross validation.

To avoid data-snooping, make the model as simple as possible, with as few parameters and complicated rules as possible.

There is usually a simpler, linear approximation corresponding to every nonlinear model.

For modelling returns, unless there are strong theoretical and empirical reasons to support a non-Gaussian distribution, we should stick to a Gaussian distribution.

Daniel Kahneman, the Nobel Prize-winning economist, wrote in his bestseller Thinking, Fast and Slow that “formulas that assign equal weights to all the predictors are often superior, because they are not affected by accidents of sampling” (Kahneman, 2011).

Stock Splits and Dividend Adjustments

We need to use prices adjusted for stock splits and dividends. This is because stock prices will drop on the day of the split or dividend payment.

For instance, if ABC is trading at 100 on Day 10 and has a 2-1 split on Day 11, the stock price on Day 11 becomes $50 (absent other market movements). This drop is purely due to the split and does not reflect a change in the market value of the stock. However, such a drop can trigger a trade signal, which is invalid.

We need to adjust all the stock prices prior to Day 11 (i.e. from Day 1 to 10). This is as true in live trading as in backtesting.

You can find historical split and dividend information on earnings.com. If you are interested in historical stock data that are already adjusted for stock splits and dividends, try csidata.com.

Survivorship Bias in Stock Database

Survivorship bias occurs if your historical data do not include delisted stocks.

You can buy reasonably priced historical data that are free of survivorship bias from csidata.com (which provides a list of delisted stocks), kibot.com, tickdata.com, and crsp.com.

Alternatively, you can limit yourself to backtesting only the most recent (e.g. 3 years) data to reduce the chance of survivorship bias.

Primary versus Consolidated Stock Prices

Many U.S. stocks are traded on multiple exchanges, electronic communication networks (ECNs), and dark pools. However, if you use a MOC (market-on-close) or MOO (market-on-open) order, the order will always be routed to the primary exchange only.

Hence, if you have a strategy that relies on market-on-open or market-on-close orders, you need the historical prices from the primary exchange to accurately backtest your model.

A similar consideration applies to using high or low prices of the day. These prices are usually the consolidated highs or lows, not that of the primary exchange. They are often unrepresentative numbers resulting from trades of small sizes on secondary exchanges. Backtest performance will also be inflated if these historical prices are used.

Venue Dependence of Currency Quotes

For the currency markets, a trade executed at one venue need not be at the best bid or ask across all the different venues. Hence, a backtest will be realistic only if we use historical data extracted from the same venue(s) as the one(s) we expect to trade on.

Short-Sale Constraints

Some stocks may be hard or impossible to borrow. This is especially true for small-cap stocks. Sometimes ETFs are as hard to borrow as stocks.

Also, be aware of the alternative “uptick rule” that took effect in 2010. This rule requires a short sale to have a trade price higher than the national best bid when a circuit breaker has been triggered. A circuit breaker for a stock is triggered when that stock traded at 10 percent lower than its previous close. This circuit breaker is in effect for the following day after the initial trigger as well.

Futures Related Considerations

Futures contracts have expiry dates, so a trading strategy on futures is really a trading strategy on many different contracts.

I skipped this section as I have no experience with futures at the moment.

Statistical Significance of Backtesting: Hypothesis Testing

Steps to test if the results of a backtest is statistically significant

1. Compute a certain statistical measure based on the backtest. In the steps below, we’ll assume that this measure is the average daily return.
2. Suppose the true average daily return is actually zero (this is known as the null hypothesis)
3. Suppose the probability distribution of the average daily return is known (e.g. it is a normal distribution).
4. Based on the probability distribution in step 3, compute the probability p that the average daily return will be at least as large as the observed value in the backtest.

This probability p is called the p-value. If it is very small (let’s say smaller than 0.01), that means we can “reject the null hypothesis,” and conclude that the backtested average daily return is statistically significant.

How to determine the probability distribution in step 3?

The first method is to assume that it is a normal distribution. The test statistics is therefore (μ*square root of n)/σ.

In other words, to test if a Sharpe ratio is statistically significant, we only need to multiply the ratio by square root n, where n is the number of days in the backtest.

Another method is to use Monte Carlo simulation to simulate many historical price data with the same moments and same length as the actual price data and run the trading strategy over these simulated price series.

If a large fraction of these price series produces returns greater than or equal to the backtest result, it is highly likely that the return of the strategy is just due to the first few moments of the price distributions, instead of some subtle patterns detected.

The third method is to generate sets of simulated trades, where the number of long and short trades and the average holding period is the same as in the backtest.

We then measure what fraction of such sets of trades has average return greater than or equal to the backtest average return.

Why is hypothesis testing a flawed methodology?

The three methods above illustrate that a null hypothesis is not unique and different null hypotheses can give rise to different estimates of statistical significance.

In addition, we actually want to know the conditional probability that the null hypothesis is true given our observed statistics. However, the procedures above gives us the conditional probability that our observed statistics is true given that the null hypothesis is true.

Also, the predictive power of any backtest rests on the central assumption that the statistical properties of the price series are unchanging, so that the trading rules that were profitable in the past will be profitable in the future. This assumption is invalid in most cases.

Last but not least, there are many instances where we should not backtest a strategy. For instance, we should not backtest high return but low Sharpe ratio strategies. Also, do not bother to backtest strategies with a maximum drawdown duration longer than what you or your investors can possibly endure.

Choosing a Backtesting and Automated Execution Platform

There are three main options:

1. Special purpose backtesting and execution platforms
2. Using a scripting language such as Visual Basic (with Excel), MATLAB, R, or Python
3. Using an advanced programming language such as Java, C++ or C#.

Special purpose backtesting and execution platforms

These platforms allow us to drag-and-drop to create our strategies. We can also use their proprietary language to develop the strategies, or use their APIs to develop strategies using advanced programming languages like Java.

Examples of special purpose platforms include Deltix, Progress Apama, QuantHouse, RTD Tango, MetaTrader, NinjaTrader, Trading Blox, and TradeStation Easy Language.

Using a scripting language

Scripting languages are faster and easier than advanced programming languages and can be used to quickly code and backtest a strategy.

For execution, most brokerages provide APIs written in Java/C++/C#. You can call the functions in these APIs directly if you are familiar with the specified language. Alternatively, you can use commercial products to call these functions.

For instance, for Python, you can use the free and open source software IbPy to connect your trading program to Interactive Brokers.

Using an advanced programming language

Many special purpose trading platforms allow us to use Java/C++/C# directly to develop our strategies.

Alternatively, you can use one of the open source IDEs listed below:

FIX stands for Financial Information eXchange and is a protocol for sending orders to brokers or exchanges. FIX as a broker means that the system can directly access any execution venues via the FIX protocol, regardless of clearing broker.

CEP stands for complex event processing and refers to a program responding to an event instantaneously and taking appropriate action. CEP languages allow us to express complex rules (e.g. sell when the order flow in the last half hour is positive, the price is above the moving average, the volatility is low, and an important news item just arrived) more easily than traditional languages.

Tick-based (sometimes called event driven or stream based) means a trade is triggered by the arrival of a new tick, not by the end of a time bar.

Advantages of Using a Special Purpose Platform

1. These platforms allow for both backtesting and automated execution of strategy. Hence, they help to prevent transcription discrepancies between the backtest code and execution code.
2. Prevents look-ahead bias
3. Enables true tick-based backtesting of high frequency strategies

Chapter 2: The Basics of Mean Reversion

Prices are rarely mean-reverting. Those few price series that are found to be mean reverting are called stationary.

Returns, not prices, are usually the ones that randomly distribute around a mean of zero.

We can often combine two or more individual price series that are not mean reverting into a portfolio whose net market value (i.e., price) is mean reverting. Those price series that can be combined this way are called cointegrating.

There are two types of mean-reversion: time series and cross sectional.

Time series mean reversion = Prices reverting to a mean determined by its own historical prices.

Cross-sectional mean reversion = Cumulative returns of the instruments in a basket revert to the cumulative return of the basket.

Mean Reversion and Stationarity

Mean reversion and stationarity are two equivalent ways of looking at the same type of price series.

To test if a series is mean reverting, we use the ADF test. To test if it is stationary, we compute the Hurst exponent and use the variance ratio test. In both cases, the null hypothesis states that the series is a random walk (and hence not mean reverting or stationary). We require at least 90 percent certainty to reject the null hypothesis.

The tests mentioned above are more statistically significant than a direct backtest of a trading strategy. This is because the tests make use of every bar’s price data for the test, while a backtest usually generates a significantly smaller number of round trip trades for us to collect performance statistics. 

In addition, if we find a price series that passed the tests above or at least one with a short enough half-life (refer to section on half-life of mean reversion), we can be assured that we can eventually find a profitable trading strategy, even if the currently tested strategy is not profitable.

Mean reverting

A price series is mean reverting if (Change of the price series in the next period) is proportional to (current price – mean price).

This can be tested using the ADF test, which tests whether we can reject the null hypothesis that the proportionality constant is zero.

If a price series is mean reverting, we can describe the price changes as

Δy_t = λy_t-1 + μ + ꞵt + 𝛼₁Δy_t-1 + … + 𝛼_kΔy_t-k + ε_t (Equation 1)

The ADF test will find out if λ = 0 (null hypothesis). If the null hypothesis λ = 0 can be rejected, that means the next move Δy_t depends on the current level y_t-1, and therefore it is not a random walk. 

The test statistics is λ/SE(λ), where λ is the regression coefficient (expected to be negative if it is mean reverting) and SE is the standard error of the regression fit.

The critical values themselves depend on the sample size and whether we assume that the price series has a non-zero mean −µ/λ or a steady drift −βt/λ.

For simplicity, we assume the drift term ꞵ to be zero since in practical trading, the drift tends to be of a much smaller magnitude than the daily fluctuations in price.

Stationary price series

A price series is stationary if variance of the log of the prices increases slower than that of a geometric random walk. 

The variance of a random walk should scale proportionally with time. If variance of the log of the prices increases slower than that of a geometric random walk, it is a sublinear function of time. 

This sublinear function is usually approximated by τ^2H, where τ is the time separating two price measurements, and H is the so-called Hurst exponent, which is less than 0.5 if the price series is indeed stationary, equal to 0.5 if the price series is a geometric random walk, and greater than 0.5 if the series is trending. 

The Variance Ratio test can be used to see whether we can reject the null hypothesis that the Hurst exponent is actually 0.5.

Half-Life of Mean Reversion

We can transform the discrete time series equation above to a differential form:

dy_t = (λy_t-1 + µ)dt + dX, where X is some Gaussian noise.

This is known as the Ornstein-Uhlenbeck formula for a mean reverting process.

The expected value of the price decays exponentially to the value -µ/λ and the half life of decay (i.e. the time for the expected value of y to reach the midpoint between the current value and the mean) equals -log(2)/λ.

If λ is positive, the price series is not at all mean reverting. If it is close to zero, the half-life will be very long, and a mean-reverting trading strategy will not be very profitable. 

However, if it is negative enough, even if we cannot reject the null hypothesis that its actual value is zero (with 90 percent certainty in an ADF test), we can possibly trade a mean reversion strategy profitably.

To determine λ, we can run a regression fit with y_t − y_{t − 1} as the dependent variable and y_{t − 1} as the independent variable.

We can use λ to determine a natural time scale for many parameters in our strategy. For example, if the half life is 20 days, we shouldn’t use a look-back of 5 days to compute a moving average or standard deviation for a mean-reversion strategy.

Cointegration

To cointegrate two price series (i.e. combine them with the aim of making the resulting series mean reverting), we can first determine the optimal hedge ratio by running a linear regression fit between two or more price series. We then use this hedge ratio to form a portfolio and run a stationarity test on this portfolio to determine if it is mean reverting.

Alternatively, we can use the cointegrated ADF test.

Cointegrated Augmented Dickey-Fuller Test

Suppose we want to test if we can cointegrate two price series EWC and EWA. We need to choose one series (e.g. EWA) to be the independent variable and the other to be the dependent variable to run the CADF test. 

If we switch the roles of EWA and EWC, the result for the CADF test will differ. The hedge ratio derived from picking EWC as the independent variable will not be the exact reciprocal of the one derived from picking EWA as the independent variable. 

In many cases, only one hedge ratio leads to a stationary portfolio. We should try each variable as independent and see which order gives the best (most negative) t-statistic.

Johansen Test

In order to test for cointegration of more than two variables, we need to use the Johansen test. This test calculates the probability of r <= n, where r is the number of independent portfolios that can be formed by various linear combinations of the cointegrating price series. It calculates r in two different ways based on the eigenvector decomposition of Λ, where Λ is the matrix equivalent of λ in Equation 1. 

Suppose there are 3 price series. The Johansen test produces the trace and eigen statistics for r <= 0, r <= 1 and r <=2. 

Suppose again that we can reject r <= 0 and r <=1. Then, we know that there are at least 2 independent portfolios that can be formed using the three price series.

As a useful by-product, the test produces eigenvectors that can be used as our hedge ratios for the individual price series to form a stationary portfolio.

Pros and Cons of Mean-Reverting Strategies

Pros:

• It is often fairly easy to construct mean-reverting strategies because we are not limited to trading instruments that are intrinsically stationary.
• There is often a good fundamental story behind a mean-reverting pair. For instance, EWA and EWC cointegrate with each other as both the Canadian and the Australian economies are dominated by commodities.
• When a cointegrating pair stops cointegrating, we can often understand the reason.
• Mean-reverting strategies span a great variety of time scales (from seconds to years)

Cons:

• The seemingly high consistency of mean-reverting strategy often lulls traders into overconfidence and overleverage as a result.

Additional Online References

ADF and CADF

https://alpaca.markets/learn/statistically-significant-statarb-01/

Johansen Test

https://blog.quantinsti.com/johansen-test-cointegration-building-stationary-portfolio/