AlgoTraderAlgoTrader Documentation

Chapter 2. Installation and Deployment

2.1. Development Environment Installation
2.1.1. Prerequisites
2.1.2. AlgoTrader Eclipse IDE Installation
2.1.3. AlgoTrader Server Code Installation
2.1.4. Next Steps
2.2. Server Environment Installation
2.2.1. Docker based Installation
2.2.2. Docker Containers
2.2.3. Docker Compose
2.2.4. Docker Management
2.3. VM Arguments
Java JDK

Install the latest Java JDK 1.8 from Oracle

Important

Algotrader requires Java 1.8.x. Do not use Java 1.9 or greater.

It is necessary to have a Java JDK (Java Development Environment), a Java JRE (Java Runtime Environment) will not be sufficient.

Please set the JAVA_HOME environment variable to point at the directory where the Java JDK is installed. You also need to add JAVA_HOME\bin to your PATH variable. Setup java environment variables:

Maven

AlgoTrader uses Apache Maven for handling of dependencies. The AlgoTrader Eclipse IDE already has an embedded Maven installation integrated. In case one wants to use Maven from the command line, it is necessary to download and install the latest version of Maven and setup it`s environment variables according to the link Maven setup.

In particular, please set the MAVEN_HOME environment variable to point at the directory where Maven is installed. You also need to add MAVEN_HOME\bin to your PATH variable.

All AlgoTrader artifacts are located on our Nexus server which is password protected:

https://repo.algotrader.ch/nexus/

It is necessary the create the following file <user-home>\.m2\settings.xml below content there, to make sure Maven can access our Nexus server. Folder .m2 should have been automaticaly created while runing maven for the first time.


<?xml version="1.0" encoding="UTF-8"?>
<settings xmlns="http://maven.apache.org/SETTINGS/1.0.0"
          xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
          xsi:schemaLocation="http://maven.apache.org/SETTINGS/1.0.0 http://maven.apache.org/xsd/settings-1.0.0.xsd">
  <servers>
    <server>
      <id>algotraderrepo</id>
      <username>myusername</username>
      <password>mypassword</password>
    </server>
    <server>
      <id>archetype</id>
      <username>myusername</username>
      <password>mypassword</password>
    </server>
  </servers>

  <profiles>
    <profile>
         <id>algotrader</id>
      <repositories>
        <repository>
            <id>algotraderrepo</id>
            <url>https://repo.algotrader.ch/nexus/repository/general/</url>
        </repository>
        <repository>
          <id>archetype</id>
          <url>https://repo.algotrader.ch/nexus/repository/general/</url>
        </repository>
      </repositories>
      <pluginRepositories>
        <pluginRepository>
            <id>algotraderrepo</id>
            <url>https://repo.algotrader.ch/nexus/repository/general/</url>
        </pluginRepository>
      </pluginRepositories>
    </profile>
  </profiles>

  <activeProfiles>
    <activeProfile>algotrader</activeProfile>
  </activeProfiles>

</settings>

Note

Please replace myusername and mypassword (both appear twice!) with the username and password provided when licensing AlgoTrader.

Git

AlgoTrader uses Git as its source code management system. The AlgoTrader Eclipse IDE already has a Git installation integrated (EGit).

In case one wants to use Git from the command line it is necessary to download and install the latest version of Git.

On Windows use TortoiseGit in combination with Git for Windows.

MySQL

Note

MySQL is not needed for simulations based on the embedded / in-memory database H2

Download and install MySQL Community Server 5.7.

Important

Algotrader requires MySQL 5.7.x. Do not use MySQL 8.0 or greater.

Per default the system uses the user name root and password password. To change username and/or password the following properties need to be updated inside conf-core.properties. Alternatively the properties can be changed via Section 2.3, “VM Arguments”:

# database user name
#{"type":"String","label":"Data Source User name"}
dataSource.user = root

# database password
#{"type":"String","label":"Data Source Password"}
dataSource.password = password

You can create the root user/set the DB password using the following command:

mysqladmin -u myUsername password myPassword

To work with MySQL it is recommended to install a MySQL client. There are many different MySQL clients available to choose from:

Note

The Java MySQL JDBC driver sometimes has issues connecting to the MySQL database depending on the MySQL timezone setting. Java Exceptions like the following are an indication for this issue:

java.sql.SQLException: The server time zone value
        'Coordinated Universal Time' is unrecognized or represents more than one time zone. You must
        configure either the server or JDBC driver (via the serverTimezone configuration property)
        to use a more specifc time zone value if you want to utilize time zone
        support.

To fix the issue it is recommended to change the MySQL timezone setting by executing the following MySQL statement

SET  GLOBAL time_zone =  'UTC' ;

To be sure you do not lose this information, e.g. on reboot, you can set a time zone in the MYSQL my.cnf or my.ini file

[mysqld]
default-time-zone=+00:00
explicit_defaults_for_timestamp=1

If that is not sufficient, follow the instructions under My SQL Time Zone Issues

Flyway

AlgoTrader uses Flyway to manage database updates. Flyway executes database migration scripts to ensure that the database is in the state corresponding to the version of AlgoTrader installed.

Please install the Flyway command line version and add flyway directory to $PATH variable in System Environment settings.

The database migration scripts are located in /bootstrap/conf/flyway.

To apply all necessary migration scripts please execute the following command from inside the flyway directory .

flyway migrate

This will create a schema named algotrader on the local machine including all necessary tables and required reference data.

Default settings can be overwritten as follows

flyway migrate
-url=jdbc:mysql://hostname:port
-user=username
-password=password
-schemas=schemaname

To reset the database please execute the following command

flyway clean migrate

Warning

This operation will erase all database content!

The display the current version of the database schema please execute the following command.

flyway info

For additional information on flyway please visit the flyway documentation.

InfluxDB

Note

InfluxDB is an optional component that is used to store historical data. In back testing it is possible to use CSV files to provide historical data as an alternative to using InfluxDB

Linux/MacOS: Download the latest version of InfluxDB and install it according to the InfluxDB installation instructions( Install it in a directory/tree without spaces in the names).

Windows: Download version 1.5.2 of InfluxDB and unpack the file in a directory/tree without spaces in the names.

Important

On Windows Algotrader requires InfluxDB 1.5.2. Do not use InfluxDB 1.6.0 or greater.

To install InfluxDB as a windows service please follow these steps:

  • download nssm

  • unpack nssm

  • update the following sections inside influxdb.conf by replacing <username> with the username that will run InfluxDB

    [meta]
      # Where the metadata/raft database is stored
      dir = "C:\\Users\\<username>\\.influxdb\\meta"
    
    [data]
      # The directory where the TSM storage engine stores TSM files.
      dir = "C:\\Users\\<username>\\.influxdb\data"
    
      # The directory where the TSM storage engine stores WAL files.
      wal-dir = "C:\\Users\\<username>\\.influxdb\\wal"
  • Go to nssm installed folder, choose win64 or win32 folder and start command propmt. Inside type: .\nssm.exe install InfluxDB <full-path-to-influxd.exe> -config <full-path-to-influxdb.conf>

    (put path in quotation marks if it contains spaces)

    e.g. nssm.exe install InfluxDB c:\AlgoTrader\influxdb-1.5.2\influxd.exe -config c:\AlgoTrader\influxdb-1.5.2\influxdb.conf

    Make sure you run the influxd.exe command, not influx.exe.

  • start InfluxDB service via Windows Service Manager

  • Add an environment variable named HOME pointing to the directory where InfluxDB is installed

All InfluxDB related settings are available within the file conf-influxdb.properties

Per default username/password authentication is disabled. To set username and password based authentication please visit the InfluxDB Authentication and Authorization guide.

There are several client options available to access InfluxDB:

AlgoTrader provides a custom IDE (Integrated Development Environment) based on Eclipse. Other Java IDE's (e.g. IntelliJ) or a standard Eclipse IDE can be used also, but the AlgoTrader Eclipse Plugins will not be available.

To install the AlgoTrader Eclipse IDE:

Important

Prior to installing the AlgoTrader Eclipse IDE please make sure that the necessary prerequisites are installed.

To assist in setting up the AlgoTrader Eclipse IDE, there is a fully configured Eclipse based product which contains all the AlgoTrader bundles and the required Eclipse plugins. Specifically, the AlgoTrader Eclipse product contains:

The AlgoTrader Eclipse Product can be downloaded from:

Username and Password are provided when licensing AlgoTrader.

Instructions:

  • Download the product according to the Operating System

  • Unzip the product zip file

  • Start the product clicking on the AlgoTrader executable file

  • After starting an Eclipse You need to setup a workspace. Select the default workspace or define a folder to it. If folder in the path doesn`t exist, it will be created by default.

  • Then open the Java perspective in the top right corner of the screen by clicking on the icon left of the AlgoTrader icon.

The AlgoTrader server code can be installed either via command line or via Eclipse

To build AlgoTrader from within Eclipse please follow this process.

This will result in the following Eclipse projects:

  • algotrader-conf

  • algotrader-launch

Note

The compilation will show errors, which should go away after the next section has been completed.

AlgoTrader uses Docker for server side installations.

Docker allows packaging of applications with all of its dependencies into a standardized unit for software development.

At the core of the Docker platform is Docker Engine, a lightweight runtime and robust tooling that builds and runs Docker containers.

For an in-depth description of Docker please visit the What is Docker page.

To get started with Docker please visit the Docker Engine Documentation page.

Docker is supported on Linux, Windows, OS X as well as different cloud services (e.g. Amazon AWS or Digital Ocean).

Please follow these setups to setup a Docker based AlgoTrader installation:

Install Docker Engine

according to the Docker Engine installation instructions

Note

On Mac and Windows please install Docker Toolbox that contains the Docker Engine

Install Docker Compose

according to the Docker Compose installation instructions

Note

On Mac and Windows please install the Docker Toolbox that contains Docker Compose

Copy docker compose file

Copy the following file to the server and make changes as necessary according to Section 2.2.3, “Docker Compose”:

https://gitlab.algotrader.ch/main/algotrader/blob/master/docker-compose.yml

Login to Nexus

Login to the Docker Repository with the username and password provided when licensing AlgoTrader.

docker login docker.algotrader.ch
Run docker compose

Invoke the following command inside the directory where the docker-compose.yml file is located:

docker-compose up -d
Open the AlgoTrader HMTL5 Frontend

Open the browser and point it to the following URL using the IP retrieved in the previous step.

http://localhost:9090

The above process will setup an AlgoTrader based system made up of the following Docker containers:

  • AlgoTrader server

  • Interactive Brokers Gateway

  • MySQL (with the AlgoTrader MySQL sample data populated)

To startup AlgoTrader with different startup options please visit this chapter on starting AlgoTrader in a Section 3.3, “Server Environment”

To startup one of the AlgoTrader example strategies please visit the appendix of this documentation.

A typical Docker based AlgoTrader installation is made up of the following Docker containers that can be configured vie a docker-compose.yml file.

The AlgoTrader Docker container contains the AlgoTrader Java code including Flyway (see Chapter 9, Database). The AlgoTrader code is located in the directory /usr/local/algotrader inside the Docker container.

The following Docker environment variables are relevant for the AlgoTrader container:

  • ALGOTRADER_HOST (default: algotrader). This variable is used by strategies to reference the AlgoTrader Docker container.

  • DATABASE_HOST (default: mysql)

  • DATABASE_PORT (default: 3306)

  • DATABASE_NAME (default: algotrader)

  • DATABASE_USER (default: root)

  • DATABASE_PASSWORD (default: password)

  • IB_GATEWAY_HOST (default: ibgateway)

  • IB_GATEWAY_ACCOUNT optional, only needs to be specified if the IB login has multiple accounts associated

  • INFLUXDB_HOST (default: influxdb)

  • VM_ARGUMENTS Additional VM arguments to be added to the java process (e.g. -DlogLevel=INFO)

  • SPRING_PROFILES (default: live,pooledDataSource,iBMarketData,iBNative,iBHistoricalData,embeddedBroker,html5) Spring profiles to be used (comma separated list)

  • STARTER_CLASS (default ch.algotrader.starter.ServerStarter)

In addition the command line switch -i can be used to load the MySQL sample data file (samples/db/mysql/mysql-data.sql) on first start up. The sample data will only be loaded if the security table contains no data. To use the -i switch please use the following directive inside docker-compose.yml:

command: -i

The AlgoTrader Docker container will run through the following process on startup:

  1. Wait for MySQL to be available. When starting up MySQL and AlgoTrader at the same time (using docker compose) it will take the database a few seconds to become available.

  2. Run all Flyway migrate scripts. This happens only the first time the AlgoTrader Docker container is started and does not happen again until the container is updated.

  3. Load MySQL sample data if the -i command line switch is used (see above).

  4. Start the AlgoTrader server

AlgoTrader based trading strategies run in separate Docker containers when running in distributed mode. When running a single strategy in embedded-mode the strategy will run inside the same Docker container as the AlgoTrader server.

The strategy code is located in the directory /usr/local/strategy inside the Docker container.

All strategy Docker containers are based on the AlgoTrader Docker container, so environment variables from the AlgoTrader docker container can be reused inside strategy containers.

To build a strategy Docker container the following Dockerfile has to be added to the root of the project:

FROM docker.algotrader.ch/algotrader/algotrader:latest

ENV STRATEGY_NAME=XYZ

WORKDIR /usr/local/strategy
ADD target/*.jar lib/

ENTRYPOINT ["/usr/local/algotrader/bin/docker-strategy-run.sh"]
CMD ["-e"]

Please replace XYZ with the name of the strategy.

Strategy Docker containers use the /usr/local/algotrader/bin/docker-strategy-run.sh startup script that is provided by the AlgoTrader Docker container.

The startup script supports both embedded and distributed mode, see: Section 3.2, “Live Trading Mode”

To start the strategy Docker container in embedded mode please use the -e command line switch inside the docker-compose.yml file of your strategy:

command: -e

This will cause the system to run through the following process:

  1. Wait for MySQL to be available. When starting up MySQL and AlgoTrader at the same time (using docker compose) it will take the database a few seconds to become available.

  2. Run all Flyway migrate scripts. This happens only the first time the strategy Docker container is started and does not happen again until the container is updated.

  3. Load MySQL data from db/mysql/mysql-data.sql. MySQL is only loaded if the entry in the strategy table for $STRATEGY_NAME is missing

  4. Start the strategy in embedded mode

To start the strategy Docker container in distributed mode please use the -d command line switch inside the docker-compose.yml file of your strategy:

command: -d

This will cause the system to run through the following process:

  1. Wait for the AlgoTrader RMI port (1199) to be available

  2. Wait for the ActiveMQ JMS port (61616) to be available

  3. Start the strategy in distributed mode

When running the system in distributed mode the AlgoTrader server needs to be run in a separate Docker container. Since trading strategies do not have access to the database directly MySQL data needs to be loaded manually by connecting to the database with a MySQL client. It is therefore suggest to follow this process when starting up the system in distributed mode:

  1. Startup MySQL, IB Gateway and AlgoTrader

    docker-compose up -d mysql ibgateway algotrader

    Note

    Please see the following chapter about changing IB API settings (Read-Only API)

  2. Load MySQL data by connecting a MySQL client to port 3306 of the Docker Engine

  3. Start strategies

    docker-compose up -d XYZ

    Please replace XYZ with the name of the strategy.

This container is made up of the following two components:

The following environment variables are relevant for the IB Controller container:

  • TWS_USERNAME (default: edemo)

  • TWS_PASSWORD (default: demouser)

  • TRADING_MODE (default: paper)

To run IB Gateway on a headless server (i.e. the Docker container) an xvfb virtual frame buffer is used.

Unfortunately only few settings of the IB Gateway can be managed via the IB Controller. All other settings have to be managed via the IB Gateway UI itself which is not visible on the Docker container.

This is especially cumbersome for the Read-Only API trading mode that the is set by default. If this mode is active, placement of orders is not allowed.

To change any of the IB Gateway settings (e.g. Read-Only API trading mode) please execute the following steps:

  1. The IB Gateway container stores IB settings inside a Docker Volume. This volume can be mapped to a local directory as follows.

    On Linux and Mac

      volumes:
        - /var/lib/tws:/var/lib/tws

    This will make IB Gateway settings available in the local directory /var/lib/tws

    On Windows

      volumes:
        - c:/Users/Administrator/Documents/tws:/var/lib/tws

    This will make IB Gateway settings available in the local directory c:\Users\Administrator\Documents\tws

  2. Install and start IB Gateway on a regular workstation (Windows, Mac or Linux)

  3. Go to Configure / Settings / API / Settings

  4. Make necessary changes (e.g. deselect the Read-Only API check box) and click OK

  5. Close the IB Gateway

  6. Inside the IB Gateway installation folder there will be one or multiple sub-directories starting which have a name made up of 8-9 characters starting with a the letter d. Please select the directory with the latest time-stamp and makes sure it contains a file named ibg.xml

  7. Copy this directory (e.g. darykqwzr) into the IB Gateway settings directory linked above:

  8. Copy the jts.ini file into the IB Gateway settings directory linked above:

  9. Start the IB Gateway Docker Container:

    docker-compose create start ibgateway

Note

The above steps will not work for the public edemo account which gets reset upon each startup.

MySQL provides a fully configured Docker container. For further details please visit MySQL on Docker Hub

The following environment variables are relevant for the MySQL container:

  • MYSQL_ROOT_PASSWORD (default: password)

  • MYSQL_DATABASE (default; algotrader)

MySQL data is stored inside a Docker Volume. This volume can be mapped to a local directory as follows.

On Linux and Mac

  volumes:
    - /var/lib/mysql:/var/lib/mysql

This will make MySQL data available in the local directory /var/lib/mysql

On Windows

  volumes:
    - c:/Users/Administrator/Documents/mysql:/var/lib/mysql

This will make MySQL data available in the local directory c:\Users\Administrator\Documents\mysql

InfluxDB provides a fully configured Docker container. For further details please visit InfluxDB on Docker Hub

InfluxDB data is stored inside a Docker Volume. This volume can be mapped to a local directory as follows.

On Linux and Mac

  volumes:
    - /var/lib/influxdb:/var/lib/influxdb

This will make MySQL data available in the local directory /var/lib/mysql

On Windows

  volumes:
    - c:/Users/Administrator/Documents/influxdb:/var/lib/influxdb

This will make MySQL data available in the local directory c:\Users\Administrator\Documents\influxdb

Docker based applications typically consist of many small applications that work together. Docker transforms these applications into individual containers that are linked together. Instead of having to build, run and manage each individual container, Docker Compose allows definition of multi-container application with all of its dependencies in a single file, then startup the application in a single command. The application's structure and configuration are held in a single place, which makes starting up applications simple and repeatable everywhere.

For further details regarding Docker Compose please visit the Docker Compose documentation.

Docker Compose uses docker-compose.yml files to configure multi-container applications. AlgoTrader ships with a default docker-compose.yml file located inside the top-level AlgoTrader project directory:

algotrader:
  image: docker.algotrader.ch/algotrader/algotrader
  command: -i
  environment:
    - VM_ARGUMENTS=-Dkeygen.id=...
  links:
    - mysql
    - ibgateway
    - influxdb
  ports:
    - 9090:9090
    - 61614:61614

ibgateway:
  image: docker.algotrader.ch/interactivebrokers/ibgateway
  environment:
    TWS_USERNAME: edemo
    TWS_PASSWORD: demouser
  volumes:
    - /var/lib/tws

mysql:
  image: mysql:5.7.22
  environment:
    MYSQL_ROOT_PASSWORD: password
    MYSQL_DATABASE: algotrader
  ports:
    - 3306:3306
  volumes:
    - /var/lib/mysql

influxdb:
  image: influxdb:1.5.2
  ports:
    - 8086:8086
  volumes:
    - /var/lib/influxdb

Using this docker-compose.yml file will create Docker containers for AlgoTrader, IB Gateway, MySQL and InfluxDB. The following items are present in the file:

  • service name: e.g. algotrader, ibgateway

  • image: the name of the image to use in the format namespace/repository:version (e.g. docker.algotrader.ch/algotrader/algotrader:latest)

  • command: command line argument to pass to the Docker container (e.g. -i). See Section 2.2.2, “Docker Containers” for supported command line arguments.

  • links: services to link to this container (e.g. mysql & ibgateway). Adding a link will make the target container accessible with the correct IP by its link name.

  • ports: ports to map on the host machine. E.g. 3306:3306 will map port 3306 of the MySQL container to 3306 of the host machine

  • environment: environment variables to use. See Section 2.2.2, “Docker Containers” for supported environment variables

  • volumes: Docker Data volumes to create. E.g. c:/Users/mysql:/var/lib/mysql will map the directory /var/lib/mysql inside the container to c:\mysql on the host machine.

All properties within the AlgoTrader *.properties files can be overwritten using the docker environment variable VM_ARGUMENTS as mentioned in Section 2.2.2.1, “AlgoTrader Container”.

If a large number of properties need to be changed or if other config files need to be changed (e.g. fix.cfg) it is recommended to follow this process:

  1. start a local instance of the AlgoTrader container using the command

    docker run -i -t --entrypoint /bin/bash docker.algotrader.ch/algotrader/algotrader
    root@9f9adac97f3c:/usr/local/algotrader#

    Take note of the container ID that has been created, 9f9adac97f3c

  2. make changes to any of the config files inside /usr/local/algotrader/conf

  3. Exit the modified AlgoTrader container using exit

  4. Commit the change to the modified AlgoTrader container using the following command

    docker commit CONTAINER_ID docker.algotrader.ch/algotrader/algotrader:VERSION

    using the CONTAINER_ID retrieved above and setting a new VERSION

It is a Docker best-practice to have only one running process per Docker container. Typically application output is logged directly to the console where it can be viewed using the command docker logs according to theDocker documentation

Sometimes this is not enough and one wishes to write additional information (e.g. fix messages) to a separate log file. To get access to this log file from outside the container it is advised to create an additional volume:

  volumes:
    - ~/fix.log:/usr/local/algotrader/logs/fix.log

Many characteristics of the system can be customized with VM arguments, the following list provides an overview of commonly used VM arguments.

AlgoTrader specific configuration parameters can be changed inside the .properties files. As an alternative configuration parameters can also be provided as VM arguments in which case they will overwrite existing parameters inside *.properties files.

-Dstatement.closePosition=false