It’s not live money in the live market! So not only is that system not FULLY tested, but also the system creators emotions have not been tested. MANY, MANY, MANY system creators override their system when it gets either close to their max draw down, or they take profits early after an up-run.

In short, people override their own trading rules often, even with automated systems.

The majority of them are “data curve fitted” (over optimized or Data Dredging). Some system creators do this on purpose, and some do it on accident, but either way it hurts YOU the investor.

Here is a quick article on data curve fitting that you can read anytime:

Curve fitting is a process used in machine learning, predictive modeling, and data mining to create a mathematical formula that is able to fit a series of historical data. In other words, we are able to use this formula to recreate a logical path through the data. Once we are able to do this, the idea is that the formula can then be applied to other data series to try and predict future prices.

Predominantly, curve fitting techniques rely on using complex mathematical models to generate predictive formulas. This means we are more likely to be working with linear regression and discrete choice models rather than using stochastic, Bollinger bands and other common indicators. However, their are parallels with systematic trading and it is for this reason that the terms are so prevalent outside of the machine learning community. Those of us who base our entry and exit criteria using a combination of indicators with “if this, then that” logic, are essentially creating a “formula” to fit the “curves” of the historical data. We also believe that this formula has some sort of predictive power. Else why would we bother?

So now we know what overfitting is, how can we take some measures to avoid it? Here are some suggestions and tips to avoid overfitting (listed in no particular order):

Break your test data into two parts. Fit your strategy to the first part of the data set (known as training data). Then test the strategy on the second part (known as the test data). If your strategy is well fit, it should have similar performance on the test data.

Test your strategy against other similar assets. If your strategy has predictive power, it should not perform wildly different on assets that have similar movement, volatility, and driving factors.

Minimize your parameters as much as possible. The more parameters you have, the more likely you can fall into overfitting. This is because you then have that many more combinations of parameters that can change the outcome of the test.

Do not try to catch every price move. Doing so often results in adding more indicators and parameters which as noted above leads to overfitting. It is ok to miss some setups. It is also ok not to have perfect timing.

Basically it explain, how and WHY this happens. Contrary to what ALL system creators think, most of them inadvertently do this.

So, what is the solution? Live money trading the system!

Simple.

But so few do it.... and those that do, even fewer last longer than 3 months.

Most “amazing backtest systems” start losing in the first 60 days! If you have ever tried one or a signal provider then you know exactly what I’m talking about!

Think about this: if someone really had a system that did “574% in 12 months” would they really sell it for $997 or $100 per month?

No!

They would run a large fund like I’m doing.

I’m not saying all of them are lying on purpose, I believe the Newbies are lying on accident and don’t understand my two points above yet.

The question is, do you want backtests hype, or live money trading results?

Thanks,

Tim Mai 

P.S - In case you did not read my full site yet, I NEVER show or use any backtest results!