AlgoTraderAlgoTrader Documentation

Chapter 6. Strategy Backtesting

6.1. Exchange Simulator
6.1.1. In-Process Exchange Simulator
6.2. Simulation Process
6.3. Single Run Simulation
6.4. Automated Parameter Optimization
6.5. Performance Statistics
6.6. Multi Security Simulations

All Esper engine instances are set to use external timing, meaning that Esper time will advance according to supplied market data files. The AdapterCoordinator is responsible for creating the necessary CurrentTimeEvents and send them to the Esper engine instances.

Securities specified within the table subscription or securities subscribed to via the SubscriptionService are fed to the corresponding Esper engine instances. The strategy table field persistent has the following meaning:

The CSV-File containing the market data has to be named and stored according to Section 21.2, “Market Data File Format”

In Simulation Mode, the system provides an Exchange Simulator functionality, where the OrderService itself creates Fills based on placed Order. Currently all orders get filled. MarketOrders get filled at the actual market prices supplied by market data events, whereas LimitOrders get filled at their LimitPrice. It is possible to extend the matching logic of the Exchange Simulator on a startegy specific basis.

In usual simulation process transaction inserts as well as position and cash_balance updates are executed in the database. It is therefore possible to use a standard database reporting tool to perform additional analysis on it.

During a simulation process the following steps are executed sequentially:

To run a strategy in Simulation Mode with the currently defined parameters use the procedure defined in Section 4.1, “Simulation Mode”.

The system allows running multiple simulations in parallel. Using cloud based servers thousands of simulation runs can be carried out in a matter of a few hours. For additional information please visit the full blog post on cloud based trading strategy optimization using algotrader and Amazon Elastic MapReduce.

Using Numerical Optimization functions (i.e. Brent & Newton) optimal parameter ranges can be determined in an automated fashion.

The following options exist:

simulateBySingleParam

One Simulation run with a parameter set to the defined value. The example below will do one run with paramter a set to 0.8

simulateBySingleParam a:0.8

simulateByMultiParam

One Simulation run with multiple parameters set to defined values. The example below will do one run with paramter a set to 0.8 and b set to 12.0

simulateByMultiParam a:0.8,b:12.0

optimizeSingleParamLinear

Multiple Simulation runs by incrementing the value of one parameter within a defined interval. The example below will increment the value of parameter a starting at 0.1 to 0.9, incrementing by 0.1 for each run

optimizeSingleParamLinear a:0.1:0.9:0.1

optimizeSingleParamByValues

Multiple Simulation runs by iterating the value of one parameter according to defined list. The example below will iterate the value of parameter a through the following list: 0.2, 0.8, 0.9 and 1.2

optimizeSingleParamByValues a:0.2:0.8:0.9:1.2

optimizeSingleParam

Multiple Simulation runs by setting the value of one parameter within the defined range and trying to find the maximum SharpeRatio. The optimizer being used is UnivariateRealOptimizer. The example below will set the value of parameter a between 0.1 and 1.0 (accuracy 0.01).

optimizeSingleParam a:0.1:1.0:0.01

optimizeMultiParamLinear

Multiple Simulation runs by doing a matrix Optimization of 2 or 3 parameters by incrementing their values within a defined intervals. The example below will iterate through all possible combinations by incrementing the value of parameter a starting at 0.1 to 0.9 (increment: 0.1), and incrementing the value of parameter b starting at 10.0 to 100.0 (increment: 5.0)

optimizeMultiParamLinear a:0.1:0.9:0.1 b:10.0:100.0:5.0

optimizeMultiParam

Multiple Simulation runs by adjusting the value of multiple parameters around their start values and trying to find the maximum SharpeRatio. The example below will start the optimization by setting the value of parameter a to 85.0 and parameter b to 150.0

optimizeMultiParam SMI a:85.0 b:150.0

In order to process parameters with the correct decimal scale the following VM argument has to be set:

-Dsimulation.roundDigits=4

Note

In order for the parameter optimization to work the following two vm-arguments need to be set:

-Dreport.disabled=true
-Dreport.openBackTestReport=false

At the end of each single simulation run, a CSV and Excel based back test report with performance statistics is created.


The following 4 files are created in the sub-folder /files/report :

  • BackTestReport.xlsm: the Excel based back test report (see image above)

  • MetricReport.csv: contains key performance metrics

  • PortfolioReport.csv: contains daily portfolio values (i.e. netLiqValue, marketValue, realizedPL, unrealizedPL, cashBalance, openPositons & leverage)

  • TradeReport.csv: contains all trades including their profit

The Excel based back test report can be modified in terms of formatting and layout if needed.

In addition when running a single simulation run, statistics will be displayed to the console in the following format:

execution time (min): 1.78dataSet: ux-1day-200801-201301netLiqValue=3'993'712.00
month-year:          Jan-08  Feb-08  Mrz-08  Apr-08  Mai-08  Jun-08  Jul-08  ...
monthlyPerformance:  -0.25%   0.56%   2.77%   3.22%   9.73%  -6.88%  -3.66%  ...
year:                  2008    2009    2010    2011    2012    2012
yearlyPerformance:   46.28%  20.40%  32.41%  30.83%  34.26%  -2.50%
posMonths=38 negMonths=23 avgY=31.31% stdY=18.85%  sharpRatio=1.65
maxMDrawDown=6.88% bestMPerf=21.72% maxDrawDown=13.05% colmar=2.4
WinTrds: cnt=1066(49.8%) avgPrft=13'872.31 avgPrftPct=6.65% avgAge=9.3
LoTrds: cnt=1073(50.1%) avgPrft=-11'158.63 avgPrftPct=-4.90% avgAge=8.9
AllTrds: cnt=2139 avgPrft=1'315.89 avgPrftPct=0.86% avgAge=9.14

When running several simulations in a row, statistics will be displayed in the following summary format showing the current parameter values as well as corresponding performance statistics of one run on one single line:

a=90 avgY=39.86% stdY=20.16% sharpe=1.97 maxDDM=11.29% bestMP=8.35% ...
a=105 avgY=34.60% stdY=20.33% sharpe=1.69 maxDDM=11.56% bestMP=8.39% ... 

In addition to above General Performance statistics, strategy specific performance statistics are printed to the console. These are retrieved by calling the method StrategyService.getSimulationResults of the strategy.

After the simulation the database modifications (mainly transactions and positions) are kept in order to perform further DB based analysis.

The amount of output during the simulation can be adjusted by setting the Log Level according to Chapter 33, Logging.

By default, only those securities will be considered for simulations which have been subscribed to in the INIT or PREFEED phase.

Some strategies that are based on multiple securities need to subscribe and unsubscribe securities during the simulation. A typical example for this would be a Futures bases strategy that needs to unsubscribe an expiring Future and at the same time subscribe to the next Future in the chain. To be able to subscribe and unsubscribe securities during a simulation the following two options exist: