CRYPTO TRADING ENGINEERING
SCALABLE BACKTESTING INFRASTRUCTURE
Introduction
Backtesting is applying a trading strategy to historical data to access its profitability and allows you to test trading ideas without risking real funds. If backtesting shows good results, traders may go ahead to apply it to a live environment. Backtesting trading strategies are essential especially when it comes to automated trading. It works with the assumption that what worked in the past may also work in the future, with no guarantee whatsoever.
Business Understanding
A startup called Mela wants to make it simple for everyone to enter the world of cryptocurrencies. It also wants to give investors a reliable source of investment while lowering the risk associated with trading cryptocurrencies. There is a need to design and build a reliable, large-scale trading data pipeline that can run various backtests and store various useful artifacts in a robust data warehouse system.
Data Understanding
The one thing we need to understand is a candlestick. Candlestick is a type of chart used for technical analysis that shows the opening, closing, and high and low prices for securities (tradable financial assets) in a specific period of time. They are mostly used by traders to show chart patterns.
Sample of candlestick data
The data downloaded is Algorand USD financial dataset, from Algorand USD (ALGO-USD) Price History & Historical Data — Yahoo Finance dates; from October 11, 2021, to October 11, 2022. The data shows the historical prices on a daily
basis.
Data details
Candlestick chart from the above Algo data
Fundamental Analysis and Technical Analysis
Fundamental analysis in trading is a method to determine the value of an asset or a security by checking its stocks and evaluating if it’s well valued. Technical analysis on the other hand studies historical trading data and predicts short-term future results.
This project aims to do a technical analysis using backtesting trading strategies, which is in the introduction part of this report. One can either perform backtesting manually, but there exist different backtesting libraries that can automate this task for you, and allow you to create a bot that can buy stocks when low and sell high. Some of these libraries are Vectorbt, Backtrader, freqtrade, etc. For this project, we’ll use Backtrader.
Backtesting using backtrader
Backtrader is an open-source python library that makes it easy for beginner coders to implement backtesting strategies easily and accurately. One can also visualize strategies, and do live trading.
Backtrader uses a class called Cerebro, which is the core of the library, and fetches data, simulates the strategy, and visualizes strategies too.
We can create the Cerebro engine that, adds data, and strategy class, and analyzes and outputs a report.
- The data( described above) was added.
- Strategies/indicators used were SMA(simple moving average) and golden cross
- Analyze to get the Sharpe ratio which is a comparison of the return on investment with its risk. A higher Sharpe ratio is better when comparing portfolios.
Using pyfolio libraries, (a python library for performance and risk analysis of financial portfolios) to generate a report with portfolio details;
- Returns- Daily returns of the strategy, noncumulative
- Positions- daily net position values
- Transactions- Prices and amounts of executed trades, one row per trade.
- Live start date- Point in time when the strategy began live trading
- Round trips- if true, causes the generation of a round trip tear sheet.
To automate the running of backtests(parallelization), i.e a simple JSON file is created from various parameters including some features from the scene. Parallelization is important since it improves efficiency, by running fast, improves functionality, and allows simultaneous processing. This allows live streaming on the frontend dashboard, to be able to view the trading live, with the parameters like dates, strategies, and outputs.
Data pipeline design
All the tools chosen for this pipeline can handle robust, large data and hence are scalable, efficient, and reliable.
REACT FRONTEND
Get scipy’s stories in your inbox
Join Medium for free to get updates from this writer.
There will be numerous backtest requests from Mela’s clients, who will place the requests from a web user interface.
Using Flask APIs, we will receive the requests (on a scene) as a JSON, with the scenes that have occurred. Flask API will also house a producer application that will create a topic that will send the messages to a topic in Kafka. A consumer application will also be housed to request topics from the Kafka server and publish them to the front end.
The frontend features implemented are; User authentication, backtesting, index funding, portfolio and user profile.
BACKEND
The backend was built using Node.js and Express.js (Javascript Backend). It housed the Kafka producer and consumer scripts to produce and consume topics. The other features implemented were; API endpoint, script to query existing runs from the database, and authenticate the user.
KAFKA
Create a cluster that will house the various brokers. Kafka brokers/ servers will be used to log the messages of the scenes on different topics and act as message brokers between the producer and the consumer groups. The topics can be huge, hence Kafka breaks topics into different partitions since it uses a distributed system that runs on a cluster of computers. To scale, we can add the number of consumers to fit our requests.
We then have to create a topic and decide how many partitions we want, depending on the parameters of backtesting. The partitions will house our messages, which will be offset to an ID.
AIRFLOW
Airflow DAGs will orchestrate the scenes. Kafka will also record the airflow orchestrations. Kafka connect would be used in place of Airflow to orchestrate simpler events.
DATABASE SCHEMA
The database chosen in Amazon Relational Database Services (Amazon RDS) makes it simple for us to set up, operate and scale databases due to its microservice architecture. It offers a variety of engines like Amazon Aurora, MySQL, MariaDB, PostgreSQL, Oracle, and Microsoft SQL Server.
For this project, we are going to use PostgreSQL database engine for storing data from backtesting.
Entity relationship diagram and flowchart of the database will be as shown. The main entities of our databases were the backtest metrics, scenes, indicator parameters and users info.
Lessons learned;
- Backtesting metrics using Golden cross and moving average
- Implementing Kafka connect in place of airflow.
- Using Amazon RDS as a database.
What to be implemented Next;
- An ALGO that selects the best-backtested strategy and implements a recommendation based on a combination of best returns, highest Sharpe ratio, and lowest drawdown (or any combination of your choosing).
- An index of the 10 biggest crypto coins by market capitalization (excluding stable coins). The code should allow for the index fund to be recapitalized every month based on the updated total capitalization of all the coins.
- A portfolio of coins using the Modern Portfolio Theory and show what the optimal composition is for the lowest risk and the highest Sharpe ratio
