AlgoTraderAlgoTrader Documentation

Chapter 6. Strategy Backtesting

6.1. Exchange Simulator
6.2. In-Process Exchange Simulator
6.3. Simulation Process
6.4. Single Run Simulation
6.5. Automated Parameter Optimization
6.6. Performance Statistics
6.7. Multi Security Simulations

For back testing historical data can be provided to strategies either via .csv files or via Section 22.1, “InfluxDB”.

Securities specified within the table subscription or securities subscribed to via the SubscriptionService are fed to the Strategy.

To feed data from CSV files during a back test please use the following setting inside conf.properties.

# should market data events be feed from CSV files
dataSource.feedCSV = true

For further details on file format and storage location of CSV files please see Section 22.7, “Market Data File Format”.

Note

When feeding historical data with CSV files it is not possible to set a particular time range for the simulation. If this is a requirement please feed data through InfluxDB

To feed data from InfluxDB during a back test please use the following settings inside conf.properties.

# should market data events be feed from the database
dataSource.feedDB = true

# the batch size when feeding from DB (number of market data events)
dataSource.feedBatchSize = 10000

# InfluxDB custom query to use when selecting data for the back test
dataSource.feedCustomQuery = 

# the back test start date when feeding from InfluxDB
dataSource.feedMinDate = 2016-01-01

# the back test end date when feeding from InfluxDB
dataSource.feedMaxDate = 2016-12-31

The tables Subscription, Position, Combination, Component & Property have a field persistent which has the following meaning:

In Simulation Mode, the system provides an Exchange Simulator functionality, where the system executes Order by using an ExecutionModel. An Execution Model contains the logic which decides whether an order gets executed under the current market situation and what portion of the order gets executed. In addition the ExecutionModel also contains the logic to calculate commissions and fees that should be added to an order.

AlgoTrader contains a DefaultExecutionModel which provides a reasonable default logic for executing orders. The DefaultExecutionModel uses the following settings from conf.properties:

# percent slippage that will be added to an order
#{"type":"Double","label":"Percent Slippage"}
execution.slippagePct = 0.0

# execution commission per contract
#{"type":"Double","label":"Commission Per Contract"}
execution.commissionPerContract = 0.0

# execution commission per order
#{"type":"Double","label":"Commission Per Order"}
execution.commissionPerOrder = 0.0

For further details on the DefaultExecutionModel please consult the JavaDoc.

It is possible to replace the DefaultExecutionModel with a custom implementation that implements the interface ExecutionModel. The custom Execution Model needs to be registered as a Spring Bean in the following locations:

In usual simulation process transaction inserts as well as position and cash_balance updates are executed in the database. It is therefore possible to use a standard database reporting tool to perform additional analysis on it.

Executing all transactions in the database during simulation is useful for reporting purposes but also incurs additional processing time. For trivial strategies that do not need to perform any sort of sophisticated querying based on transaction data, an additional in-process / in-memory exchange simulator is available that uses Hash Maps as the underlying storage mechanism. This will allow for significantly faster processing of orders during simulation. The in-process exchange simulator can be used as follows within strategies:

Order order = new MarketOrder.Factory.newInstance();
order.setSecurity(security);
order.setStrategy(strategy);
order.setQuantity(qty);
order.setSide(Side.BUY);

getSimulator().sendOrder(order);

Position position = getSimulator().findPositionByStrategyAndSecurity(strategy, security);

During a simulation process the following steps are executed sequentially by the SimulationExecutorImpl:

  1. Create strategy entries in the database

  2. The database is reset to its original state via the ResetService

  3. An initial amount (USD 1'000'000 per default) is allocated to each strategy (the initial amount can be changed through the simulation.initialBalance setting inside conf.properties)

  4. All server Esper modules are deployed

  5. The life cycle phase INIT is broadcasted to all strategies. During this phase potential initiation steps can be invoked.

  6. All strategy initModules Modules are deployed

  7. The life cycle phase PREFEED is broadcasted to all strategies. During this phase technical indicators can be initialized using historical data

  8. All strategy runModules Modules are deployed

  9. Market data subscriptions are initialized based on entries in the table subscription

  10. The life cycle phase START is broadcasted to all strategies. During this phase eventual actions like security subscriptions can be taken care of

  11. At that time the actual simulation starts and market data events are starting to be sent into the Esper Engines

  12. The life cycle phase EXIT is broadcasted to all strategies. During this phase eventual cleanup actions can be taken care of

  13. At the end of each simulation run, metrics are printed to the console (if enabled), see Chapter 32, Metrics

  14. All open orders are cancelled

  15. All open positions are closed

  16. An EndOfSimulationVO event is sent to all strategies

  17. SimulationResults are retrieved from the strategies

  18. Esper Engines are re-initialized

  19. The In-Process Exchange Simulator is reset

  20. The Market Data Cache is flushed

  21. The second-level cache is cleared

  22. All reports are closed

  23. The Excel based back test report is created and statistics are displayed to the console, see Section 6.6, “Performance Statistics”

To run a strategy in Simulation Mode with the currently defined parameters use the procedure defined in Section 4.1, “Simulation Mode”.

The system allows running multiple simulations in parallel. Using cloud based servers thousands of simulation runs can be carried out in a matter of a few hours. For additional information please visit the full blog post on cloud based trading strategy optimization using algotrader and Amazon Elastic MapReduce.

Using Numerical Optimization functions (i.e. Brent & Newton) optimal parameter ranges can be determined in an automated fashion.

The following options exist:

simulateBySingleParam

One Simulation run with a parameter set to the defined value. The example below will do one run with parameter a set to 0.8

simulateBySingleParam a:0.8

simulateByMultiParam

One Simulation run with multiple parameters set to defined values. The example below will do one run with parameter a set to 0.8 and b set to 12.0

simulateByMultiParam a:0.8,b:12.0

optimizeSingleParamLinear

Multiple Simulation runs by incrementing the value of one parameter within a defined interval. The example below will increment the value of parameter a starting at 0.1 to 0.9, incrementing by 0.1 for each run

optimizeSingleParamLinear a:0.1:0.9:0.1

optimizeSingleParamByValues

Multiple Simulation runs by iterating the value of one parameter according to defined list. The example below will iterate the value of parameter a through the following list: 0.2, 0.8, 0.9 and 1.2

optimizeSingleParamByValues a:0.2:0.8:0.9:1.2

optimizeSingleParam

Multiple Simulation runs by setting the value of one parameter within the defined range and trying to find the maximum Sharpe Ratio. The optimizer being used is UnivariateRealOptimizer. The example below will set the value of parameter a between 0.1 and 1.0 (accuracy 0.01).

optimizeSingleParam a:0.1:1.0:0.01

optimizeMultiParamLinear

Multiple Simulation runs by doing a matrix Optimization of 2 or 3 parameters by incrementing their values within a defined intervals. The example below will iterate through all possible combinations by incrementing the value of parameter a starting at 0.1 to 0.9 (increment: 0.1), and incrementing the value of parameter b starting at 10.0 to 100.0 (increment: 5.0)

optimizeMultiParamLinear a:0.1:0.9:0.1 b:10.0:100.0:5.0

optimizeMultiParam

Multiple Simulation runs by adjusting the value of multiple parameters around their start values and trying to find the maximum Sharpe Ratio. The example below will start the optimization by setting the value of parameter a to 85.0 and parameter b to 150.0

optimizeMultiParam SMI a:85.0 b:150.0

In order to process parameters with the correct decimal scale the following VM argument has to be set:

-Dsimulation.roundDigits=4

Note

In order for the parameter optimization to work the following two VM arguments need to be set:

-DlogLevel=warn
-Dreport.disabled=true
-Dreport.openBackTestReport=false

At the end of each single simulation run, a CSV and Excel based back test report with performance statistics is created.


The following 4 files are created in the sub-folder /files/report :

  • BackTestReport.xlsm: the Excel based back test report (see image above)

  • MetricReport.csv: contains key performance metrics

  • PortfolioReport.csv: contains daily portfolio values (i.e. netLiqValue, marketValue, realizedPL, unrealizedPL, cashBalance, openPositons & leverage)

  • TradeReport.csv: contains all trades including their profit

The Excel based back test report can be modified in terms of formatting and layout if needed.

In addition when running a single simulation run, statistics will be displayed to the console in the following format:

execution time (min): 1.78dataSet: ux-1day-200801-201301netLiqValue=3'993'712.00
month-year:          Jan-08  Feb-08  Mrz-08  Apr-08  Mai-08  Jun-08  Jul-08  ...
monthlyPerformance:  -0.25%   0.56%   2.77%   3.22%   9.73%  -6.88%  -3.66%  ...
year:                  2008    2009    2010    2011    2012    2012
yearlyPerformance:   46.28%  20.40%  32.41%  30.83%  34.26%  -2.50%
posMonths=38 negMonths=23 avgY=31.31% stdY=18.85%  sharpRatio=1.65
maxMDrawDown=6.88% bestMPerf=21.72% maxDrawDown=13.05% colmar=2.4
WinTrds: cnt=1066(49.8%) avgPrft=13'872.31 avgPrftPct=6.65% avgAge=9.3
LoTrds: cnt=1073(50.1%) avgPrft=-11'158.63 avgPrftPct=-4.90% avgAge=8.9
AllTrds: cnt=2139 avgPrft=1'315.89 avgPrftPct=0.86% avgAge=9.14

When running parameter optimizations, statistics will be displayed in the following summary format showing the current parameter values as well as corresponding performance statistics of one run on one single line:

a=90 avgY=39.86% stdY=20.16% sharpe=1.97 maxDDM=11.29% bestMP=8.35% ...
a=105 avgY=34.60% stdY=20.33% sharpe=1.69 maxDDM=11.56% bestMP=8.39% ... 

In addition to above General Performance statistics, strategy specific performance statistics are printed to the console. These are retrieved by calling the method StrategyService.getSimulationResults of the strategy.

The amount of output during the simulation can be adjusted by setting the Log Level according to Chapter 33, Logging.

By default, only those securities will be considered for simulations which have been subscribed to in the INIT or PREFEED phase.

Some strategies that are based on multiple securities need to subscribe and unsubscribe securities during the simulation. A typical example for this would be a Futures bases strategy that needs to unsubscribe an expiring Future and at the same time subscribe to the next Future in the chain. To be able to subscribe and unsubscribe securities during a simulation the following two options exist: