AlgoTraderAlgoTrader Documentation

Chapter 2. Installation and Deployment

2.1. Development Environment Installation
2.1.1. Prerequisites
2.1.2. AlgoTrader Eclipse IDE Installation
2.1.3. Next Steps
2.2. Server Environment Installation
2.2.1. Docker based Installation
2.2.2. Docker Containers
2.2.3. Docker Compose
2.2.4. Docker Management
2.3. VM Arguments
Java JDK

Install the latest Java JDK 1.8 from Oracle

Important

It is necessary to have a Java JDK (Java Development Environment), a Java JRE (Java Runtime Environment) will not be sufficient.

Please set the JAVA_HOME environment variable to point at the directory where the Java JDK is installed.

Maven

AlgoTrader uses Apache Maven for handling of dependencies. The AlgoTrader Eclipse IDE already has a Maven installation integrated. All AlgoTrader artifacts are located on our Nexus server which is password protected:

https://repo.algotrader.ch/nexus/

It is necessary the create the following file <user-home>\.m2\settings.xml to make sure Maven can access our Nexus server.


<?xml version="1.0" encoding="UTF-8"?>
<settings xmlns="http://maven.apache.org/SETTINGS/1.0.0"
          xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
          xsi:schemaLocation="http://maven.apache.org/SETTINGS/1.0.0 http://maven.apache.org/xsd/settings-1.0.0.xsd">
  <servers>
    <server>
      <id>algotraderrepo</id>
      <username>myusername</username>
      <password>mypassword</password>
    </server>
    <server>
      <id>archetype</id>
      <username>myusername</username>
      <password>mypassword</password>
    </server>   
  </servers>

  <profiles>
    <profile>
         <id>algotrader</id>
      <repositories>
        <repository>
            <id>algotraderrepo</id>
            <url>https://repo.algotrader.ch/nexus/repository/public/</url>
        </repository>
        <repository>
          <id>archetype</id>
          <url>https://repo.algotrader.ch/nexus/repository/public/</url>
        </repository>
      </repositories>
      <pluginRepositories>
        <pluginRepository>
            <id>algotraderrepo</id>
            <url>https://repo.algotrader.ch/nexus/repository/public/</url>
        </pluginRepository>
      </pluginRepositories>       
    </profile>    
  </profiles>

  <activeProfiles>
    <activeProfile>algotrader</activeProfile>
  </activeProfiles>

</settings>

Note

Please replace myusername and mypassword (both appear twice!) with the username and password provided when licensing AlgoTrader.

In case one wants to use Maven from the command line it is necessary to download and install the latest version of Maven.

Git

AlgoTrader uses Git as its source code management system. The AlgoTrader Eclipse IDE already has a Git installation integrated (EGit).

In case one wants to use Git from the command line it is necessary to download and install the latest version of Git.

On Windows use TortoiseGit in combination with Git for Windows.

MySql

Note

MySql is not needed for simulations based on the embedded / in-memory database H2

Install the latest MySql Community Server.

Per default the system uses the user name root and password password. If a different username and/or password is chosen, the property dataSource.user and/or dataSource.password needs to be set via VM arguments as follows:

-DdataSource.user=myUsername
-DdataSource.password=myPassword

To work with MySql it is recommended to install a MySql client. There are many different MySql clients available to choose from:

Note

The Java MySql JDBC driver sometimes has issues connecting to the MySql database depending on the MySql timezone setting. Java Exceptions like the following are an indication for this issue:

java.sql.SQLException: The server time zone value
        'Coordinated Universal Time' is unrecognized or represents more than one time zone. You must
        configure either the server or JDBC driver (via the serverTimezone configuration property)
        to use a more specifc time zone value if you want to utilize time zone
        support.

To fix the issue it is recommended to change the MySql timezone setting by executing the following MySql statement

SET  GLOBAL time_zone =  'UTC' ;
Flyway

AlgoTrader uses Flyway to manage database updates. Flyway executes database migration scripts to ensure that the database is in the state corresponding to the version of AlgoTrader installed.

Please install the Flyway command line version.

When building AlgoTrader (see Chapter 3, Building AlgoTrader) the database migration scripts will be located in algotrader/core/flyway. Alternatively download the AlgoTrader zip file from the following location and extract the folder algotrader/core/flyway:

https://gitlab.algotrader.ch/main/algotrader/repository/archive.zip

To apply all necessary migration scripts please execute the following command from inside the flyway directory .

flyway migrate

This will create a schema named algotrader on the local machine including all necessary tables and required reference data.

Default settings can be overwritten as follows

flyway migrate
-url=jdbc:mysql://hostname:port
-user=username
-password=password
-schemas=schemaname

To reset the database (in development) please execute the following command

flyway clean migrate

Warning

This operation will erase all database content!

The display the current version of the database schema please execute the following command.

flyway info

For additional information on flyway please visit the flyway documentation.

InfluxDB

Note

InfluxDB is an optional component that is used to store historical data. In back testing it is possible to use CSV files to provide historical data as an alternative to using InfluxDB

Download the latest version of InfluxDB and install it according to the InfluxDB installation instructions.

To install InfluxDB as a windows service please follow these steps:

  • download nssm

  • unpack nssm

  • update the following sections inside influxdb.conf by replacing <username> with the username that will run InfluxDB

    [meta]
      # Where the metadata/raft database is stored
      dir = "C:\\Users\\<username>\\.influxdb\\meta"
    
    [data]
      # The directory where the TSM storage engine stores TSM files.
      dir = "C:\\Users\\<username>\\.influxdb\\data"
    
      # The directory where the TSM storage engine stores WAL files.
      wal-dir = "C:\\Users\\<username>\\.influxdb\\wal"
  • run nssm.exe install InfluxDB <full-path-to-influxd.exe> -config <full-path-to-influxdb.conf

  • start InfluxDB service via Windows Service Manager

  • Add an environment variable named HOME pointing to the directory where InfluxDB is installed

All InfluxDB related settings are available within the file conf-influxdb.properties

Per default username/password authentication is disabled. To set username and password based authentication please visit the InfluxDB Authentication and Authorization guide.

There are several client options available to access InfluxDB:

AlgoTrader has been developed using Eclipse. Other Java IDE's (e.g. IntelliJ) can be used also, but the AlgoTrader Eclipse Plugins will not be available.

To install the AlgoTrader Eclipse IDE the following two options exist:

Important

Prior to installing the AlgoTrader Eclipse IDE please make sure that the necessary prerequisites are installed.

The following sections describe the installation of an AlgoTrader Eclipse IDE from scratch.

AlgoTrader uses Docker for server side installations.

Docker allows packaging of applications with all of its dependencies into a standardized unit for software development.

At the core of the Docker platform is Docker Engine, a lightweight runtime and robust tooling that builds and runs Docker containers.

For an in-depth description of Docker please visit the What is Docker page.

To get started with Docker please visit the Docker Engine Documentation page.

Docker is supported on Linux, Windows, OS X as well as different cloud services (e.g. Amazon AWS or Digital Ocean).

Please follow these setups to setup a Docker based AlgoTrader installation:

Install Docker Engine

according to the Docker Engine installation instructions

Note

On Mac and Windows please install Docker Toolbox that contains the Docker Engine

Install Docker Compose

according to the Docker Compose installation instructions

Note

On Mac and Windows please install the Docker Toolbox that contains Docker Compose

Copy docker compose file

Copy the following file to the server and make changes as necessary according to Section 2.2.3, “Docker Compose”:

https://gitlab.algotrader.ch/main/algotrader/blob/master/docker-compose.yml

Login to Nexus

Login to the Docker Repository with the username and password provided when licensing AlgoTrader.

docker login docker.algotrader.ch
Run docker compose

Invoke the following command inside the directory where the docker-compose.yml file is located:

docker-compose up -d
Open the AlgoTrader HMTL5 Frontend

Open the browser and point it to the following URL using the IP retrieved in the previous step.

http://localhost:9090

The above process will setup an AlgoTrader based system made up of the following Docker containers:

  • AlgoTrader server

  • Interactive Brokers Gateway

  • MySql (with the AlgoTrader MySql sample data populated)

To startup AlgoTrader with different startup options please visit this chapter on starting AlgoTrader in a Section 4.3, “Server Environment”

To startup one of the AlgoTrader example strategies please visit the appendix of this documentation.

A typical Docker based AlgoTrader installation is made up of the following Docker containers that can be configured vie a docker-compose.yml file.

The AlgoTrader Docker container contains the AlgoTrader Java code including Flyway (see Chapter 12, Database). The AlgoTrader code is located in the directory /usr/local/algotrader inside the Docker container.

The following Docker environment variables are relevant for the AlgoTrader container:

  • ALGOTRADER_HOST (default: algotrader). This variable is used by strategies to reference the AlgoTrader Docker container.

  • DATABASE_HOST (default: mysql)

  • DATABASE_PORT (default: 3306)

  • DATABASE_NAME (default: algotrader)

  • DATABASE_USER (default: root)

  • DATABASE_PASSWORD (default: password)

  • IB_GATEWAY_HOST (default: ibgateway)

  • IB_GATEWAY_ACCOUNT optional, only needs to be specified if the IB login has multiple accounts associated

  • INFLUXDB_HOST (default: influxdb)

  • VM_ARGUMENTS Additional VM arguments to be added to the java process (e.g. -DlogLevel=INFO)

  • SPRING_PROFILES (default: live,pooledDataSource,iBMarketData,iBNative,iBHistoricalData,embeddedBroker,html5) Spring profiles to be used (comma separated list)

  • STARTER_CLASS (default ch.algotrader.starter.ServerStarter)

In addition the command line switch -i can be used to load the MySql sample data file (samples/db/mysql/mysql-data.sql) on first start up. The sample data will only be loaded if the security table contains no data. To use the -i switch please use the following directive inside docker-compose.yml:

command: -i

The AlgoTrader Docker container will run through the following process on startup:

  1. Wait for MySql to be available. When starting up MySql and AlgoTrader at the same time (using docker compose) it will take the database a few seconds to become available.

  2. Run all Flyway migrate scripts. This happens only the first time the AlgoTrader Docker container is started and does not happen again until the container is updated.

  3. Load MySql sample data if the -i command line switch is used (see above).

  4. Start the AlgoTrader server

AlgoTrader based trading strategies run in separate Docker containers when running in distributed mode. When running a single strategy in embedded-mode the strategy will run inside the same Docker container as the AlgoTrader server.

The strategy code is located in the directory /usr/local/strategy inside the Docker container.

All strategy Docker containers are based on the AlgoTrader Docker container, so environment variables from the AlgoTrader docker container can be reused inside strategy containers.

To build a strategy Docker container the following Dockerfile has to be added to the root of the project:

FROM docker.algotrader.ch/algotrader/algotrader:latest

ENV STRATEGY_NAME=XYZ

WORKDIR /usr/local/strategy
ADD target/*.jar lib/

ENTRYPOINT ["/usr/local/algotrader/bin/docker-strategy-run.sh"]
CMD ["-e"]

Please replace XYZ with the name of the strategy.

Strategy Docker containers use the /usr/local/algotrader/bin/docker-strategy-run.sh startup script that is provided by the AlgoTrader Docker container.

The startup script supports both embedded and distributed mode.

To start the strategy Docker container in embedded mode please use the -e command line switch.

command: -e

This will cause the system to run through the following process:

To start the strategy Docker container in distributed mode please use the -d command line switch.

command: -d

This will cause the system to run through the following process:

When running the system in distributed mode the AlgoTrader server needs to be run in a separate Docker container. Since trading strategies do not have access to the database directly MySql data needs to be loaded manually by connecting to the database with a MySql client. It is therefore suggest to follow this process when starting up the system in distributed mode:

This container is made up of the following two components:

The following environment variables are relevant for the IB Controller container:

  • TWS_USERNAME (default: edemo)

  • TWS_PASSWORD (default: demouser)

  • TRADING_MODE (default: paper)

To run IB Gateway on a headless server (i.e. the Docker container) an xvfb virtual frame buffer is used.

Unfortunately only few settings of the IB Gateway can be managed via the IB Controller. All other settings have to be managed via the IB Gateway UI itself which is not visible on the Docker container.

This is especially cumbersome for the Read-Only API trading mode that the is set by default. If this mode is active, placement of orders is not allowed.

To change any of the IB Gateway settings (e.g. Read-Only API trading mode) please execute the following steps:

  1. The IB Gateway container stores IB settings inside a Docker Volume. This volume can be mapped to a local directory as follows.

    On Linux and Mac

      volumes:
        - /var/lib/tws:/var/lib/tws

    This will make IB Gateway settings available in the local directory /var/lib/tws

    On Windows

      volumes:
        - /c/Users/Administrator/Documents/tws:/var/lib/tws

    This will make IB Gateway settings available in the local directory c:\Users\Administrator\Documents\tws

  2. Install and start IB Gateway on a regular workstation (Windows, Mac or Linux)

  3. Go to Configure / Settings / API / Settings

  4. Make necessary changes (e.g. deselect the Read-Only API check box) and click OK

  5. Close the IB Gateway

  6. Inside the IB Gateway installation folder there will be one or multiple sub-directories starting which have a name made up of 8-9 characters starting with a the letter d. Please select the directory with the latest time-stamp and makes sure it contains a file named ibg.xml

  7. Copy this directory (e.g. darykqwzr) into the IB Gateway settings directory linked above:

  8. Copy the jts.ini file into the IB Gateway settings directory linked above:

  9. Start the IB Gateway Docker Container:

    docker-compose create start ibgateway

Note

The above steps will not work for the public edemo account which gets reset upon each startup.

MySql provides a fully configured Docker container. For further details please visit MySql on Docker Hub

The following environment variables are relevant for the MySql container:

  • MYSQL_ROOT_PASSWORD (default: password)

  • MYSQL_DATABASE (default; algotrader)

MySql data is stored inside a Docker Volume. This volume can be mapped to a local directory as follows.

On Linux and Mac

  volumes:
    - /var/lib/mysql:/var/lib/mysql

This will make MySql data available in the local directory /var/lib/mysql

On Windows

  volumes:
    - /c/Users/Administrator/Documents/mysql:/var/lib/mysql

This will make MySql data available in the local directory c:\Users\Administrator\Documents\mysql

InfluxDB provides a fully configured Docker container. For further details please visit InfluxDB on Docker Hub

InfluxDB data is stored inside a Docker Volume. This volume can be mapped to a local directory as follows.

On Linux and Mac

  volumes:
    - /var/lib/influxdb:/var/lib/influxdb

This will make MySql data available in the local directory /var/lib/mysql

On Windows

  volumes:
    - /c/Users/Administrator/Documents/influxdb:/var/lib/influxdb

This will make MySql data available in the local directory c:\Users\Administrator\Documents\influxdb

Docker based applications typically consist of many small applications that work together. Docker transforms these applications into individual containers that are linked together. Instead of having to build, run and manage each individual container, Docker Compose allows definition of multi-container application with all of its dependencies in a single file, then startup the application in a single command. The application's structure and configuration are held in a single place, which makes starting up applications simple and repeatable everywhere.

For further details regarding Docker Compose please visit the Docker Compose documentation.

Docker Compose uses docker-compose.yml files to configure multi-container applications. AlgoTrader ships with a default docker-compose.yml file located inside the top-level AlgoTrader project directory:

algotrader:
  image: docker.algotrader.ch/algotrader/algotrader
  command: -i
  links:
    - mysql
    - ibgateway
    - influxdb
  ports:
    - 9090:9090
    - 61614:61614

ibgateway:
  image: docker.algotrader.ch/interactivebrokers/ibgateway
  environment:
    TWS_USERNAME: edemo
    TWS_PASSWORD: demouser
  volumes:
    - /var/lib/tws

mysql:
  image: mysql
  environment:
    MYSQL_ROOT_PASSWORD: password
    MYSQL_DATABASE: algotrader
  ports:
    - 3306:3306
  volumes:
    - /var/lib/mysql

influxdb:
  image: influxdb
  ports:
    - 8086:8086
  volumes:
    - /var/lib/influxdb

Using this docker-compose.yml file will create Docker containers for AlgoTrader, IB Gateway, MySql and InfluxDB. The following items are present in the file:

  • service name: e.g. algotrader, ibgateway

  • image: the name of the image to use in the format namespace/repository:version (e.g. docker.algotrader.ch/algotrader/algotrader:latest)

  • command: command line argument to pass to the Docker container (e.g. -i). See Section 2.2.2, “Docker Containers” for supported command line arguments.

  • links: services to link to this container (e.g. mysql & ibgateway). Adding a link will make the target container accessible with the correct IP by its link name.

  • ports: ports to map on the host machine. E.g. 3306:3306 will map port 3306 of the MySql container to 3306 of the host machine

  • environment: environment variables to use. See Section 2.2.2, “Docker Containers” for supported environment variables

  • volumes: Docker Data volumes to create. E.g. /c/Users/mysql:/var/lib/mysql will map the directory /var/lib/mysql inside the container to c:\mysql on the host machine.

All properties within the AlgoTrader *.properties files can be overwritten using the docker environment variable VM_ARGUMENTS as mentioned in Section 2.2.2.1, “AlgoTrader Container”.

If a large number of properties need to be changed or if other config files need to be changed (e.g. fix.cfg) it is recommended to follow this process:

  1. start a local instance of the AlgoTrader container using the command

    docker run -i -t --entrypoint /bin/bash docker.algotrader.ch/algotrader/algotrader
    root@9f9adac97f3c:/usr/local/algotrader#

    Take note of the container ID that has been created, 9f9adac97f3c

  2. make changes to any of the config files inside /usr/local/algotrader/conf

  3. Exit the modified AlgoTrader container using exit

  4. Commit the change to the modified AlgoTrader container using the following command

    docker commit CONTAINER_ID docker.algotrader.ch/algotrader/algotrader:VERSION

    using the CONTAINER_ID retrieved above and setting a new VERSION

It is a Docker best-practice to have only one running process per Docker container. Typically application output is logged directly to the console where it can be viewed using the command docker logs according to theDocker documentation

Sometimes this is not enough and one wishes to write additional information (e.g. fix messages) to a separate log file. To get access to this log file from outside the container it is advised to create an additional volume:

  volumes:
    - ~/fix.log:/usr/local/algotrader/logs/fix.log

In addition to using the Docker command line, several options exist for management of docker based installations

When running Docker on Windows or Mac Docker Kitematic provides a UI for management of the Docker engine.

Docker recently introduced Docker Cloud, a hosted service for Docker container management. Using Docker Cloud, Docker containers can be deployed to cloud servers or local Linux servers with a single click.

In addition log files, settings, CPU and memory usage can be viewed online from a central location.

With Docker Cloud it's even possible to setup a new Cloud Server (e.g. on Amazon AWS) including AlgoTrader without touching the server itself.

For further details on Docker Cloud please visit the Docker Cloud documentation

Many characteristics of the system can be customized with VM arguments, the following list provides an overview of commonly used VM arguments.

In addition any of the AlgoTrader configuration parameters stored within *.properties files can also be modified via VM argument, see Chapter 29, Configuration and Preferences API.

-DlogLevel

log4j log level (ERROR, WARN, INFO or DEBUG)

-Dspring.profiles.active

list of Spring Profiles to activate (see Section 27.3, “Spring Profiles”)

-Xmx

increase the Java Heap Size to specified amount (e.g. 2048M)

-XX:MaxPermSize

increase the Java Perm Size to specified amount (e.g. 1024M)