<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Saeed Rahman on Medium]]></title>
        <description><![CDATA[Stories by Saeed Rahman on Medium]]></description>
        <link>https://medium.com/@usquant349?source=rss-6e65ec0dcdd5------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/2*ZQ6JoYJ5kUP16dBYKInRMw.jpeg</url>
            <title>Stories by Saeed Rahman on Medium</title>
            <link>https://medium.com/@usquant349?source=rss-6e65ec0dcdd5------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Fri, 20 Nov 2026 08:13:51 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@usquant349/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[AI-Assisted Data Catalogs: An LLM Powered by Knowledge Graphs for Metadata Discovery]]></title>
            <link>https://medium.com/datahub-project/llm-and-graph-integration-enhancing-alternative-data-exploration-0fca8aefcd88?source=rss-6e65ec0dcdd5------2</link>
            <guid isPermaLink="false">https://medium.com/p/0fca8aefcd88</guid>
            <category><![CDATA[genai]]></category>
            <category><![CDATA[stock-market]]></category>
            <category><![CDATA[data]]></category>
            <category><![CDATA[llm]]></category>
            <category><![CDATA[finance]]></category>
            <dc:creator><![CDATA[Saeed Rahman]]></dc:creator>
            <pubDate>Mon, 01 Jul 2024 00:08:57 GMT</pubDate>
            <atom:updated>2025-02-28T20:20:55.640Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/772/1*67A0a8G5yxrQKnUGlh1Xeg.png" /></figure><p>Data teams in investment firms play a crucial role in equipping portfolio managers (PMs), quants, and researchers with the tools and insights needed to leverage data assets effectively. A key part of their job is enabling data discovery, ensuring that users can easily locate the necessary data points from a wide range of datasets, including both structured and unstructured alternative data, market data, and reference datasets.</p><p>In larger investment or trading firms, managing these datasets can be particularly challenging. With potentially hundreds or even thousands of datasets to oversee, creating comprehensive documentation and understanding the associated data assets is a complex and resource-intensive task. It is essential for PMs, quants, and researchers to have a deep understanding of the available datasets and how to navigate them, which can consume a significant amount of time and resources.</p><p>Let’s take a look at some of the frequently asked questions patterns that front office professionals have for data teams.</p><ul><li>What are the available datasets for US equities?</li><li>What datasets are available for entity Starbucks?</li><li>What datasets are available for the technology sector?</li><li>What reference datasets can be used with the foot traffic dataset?</li><li>Show me all the columns in different datasets that use zip codes?</li><li>Provide some example Jupiter notebooks for exploring web-traffic data?</li><li>Can you show me examples of analysis done using satellite images dataset?</li><li>Which foot-traffic data vendors have we evaluated?</li><li>What are the data products built using the web-traffic and foot-traffic datasets?</li></ul><p>Though seemingly insignificant, answering some questions can be a daunting task. It often involves sifting through various documents, scouring countless emails, combing through lines of code, and seeking help from colleagues who have experience with the datasets. These seemingly simple questions can become a lengthy and challenging process, highlighting the crucial role of documentation and communication in any project. One way to make information discovery easier is by creating data catalogues, documentation, and example scripts, allowing users to become self-sufficient.</p><p>As LLMs and RAG-based architectures gain popularity, semantic searches on documents have become a common method of gathering information and finding answers. However, these answers are not always straightforward, as they may require information from multiple sources for the LLM to provide logical responses. In such cases, a document-based semantic search on vector databases may not suffice.</p><p>In this article, we will explore a different approach to solving this issue. Through the use of Neo4j knowledge graphs derived from data asset metadata, we can empower LLMs to answer complex questions. This post, along with its accompanying <a href="https://github.com/saeed349/LLM-Powered-Metadata-Graph">GitHub</a> code, will demonstrate this approach and its potential benefits.</p><p>As we move forward, there are several important topics that we will be discussing. However, I want to begin by giving an overview of these objectives.</p><ul><li>Our aim is to use LLM to provide answers to important natural language questions from users about datasets and data assets. A key factor in achieving this is by representing metadata as graphs and utilizing semantic and graph search with the help of LLMs.</li><li>By connecting different alternative datasets through LLM-based entity mapping techniques at the metadata level and integrating this linked information with structured data sources using LLM workflows, we can greatly enhance the discoverability of data.</li></ul><p>Collecting metadata for datasets can be a challenging task as the data is often scattered across different systems. However, instead of manually gathering this information from various sources, a more efficient approach would be to use a data catalogue or observability platform. This not only saves time and effort but also ensures that all the necessary metadata is collected accurately. In my case, I have datasets stored in Snowflake, Databricks, and PostgreSQL, and I use <a href="https://datahubproject.io/">DataHub</a> to retrieve them. By utilizing the platform’s GraphQL API, I am able to easily access the required information and generate graphs in Neo4j.</p><p>In a previous <a href="https://www.linkedin.com/pulse/from-chaos-nirvana-building-scalable-data-hedge-funds-saeed-rahman-uy7cc/">article</a>, I shared a practical guide on how implementing data observability and governance using DataHub can improve data operations and build trust among end users. For those unfamiliar with metadata models and the importance of collecting this information for seamless data operations, I highly recommend reading that <a href="https://www.linkedin.com/pulse/from-chaos-nirvana-building-scalable-data-hedge-funds-saeed-rahman-uy7cc/">article</a>. Join our DataHub Slack Community to ask your questions.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/800/1*NeK8qSmCrbcvCfofmIdRKw.png" /><figcaption><a href="https://pages.acryl.io/slack">https://pages.acryl.io/slack</a></figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/840/0*onpgTIiquD4TOCPT" /></figure><p>For this project, we will utilize the OpenAI API, Neo4j, LangChain, and Chromadb. The OpenAI models will be utilized for the LLM, however, you have the flexibility to switch to any other LLM of your preference. I have tested the compatibility with llama3 in Ollama and it has proven to be just as effective for our use cases.</p><p>Let’s imagine a hypothetical equity discretionary trading firm that possesses few alternative datasets. Most trading firms typically have a security master table to track the securities they trade, which is crucial for their operations and research, our hypothetical firm also has a basic security master table with a limited number of securities.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*b0kvozrlgYIL66Lm" /></figure><p>Note: Each of the datasets mentioned above includes a time component and most are structured to be point-in-time (PiT). Additionally, PiT relationships exist between metadata entities. However, for simplicity, we will not be incorporating this into the ontology design here.</p><p>The initial step is to depict the table metadata entities as graphs. At this stage, we will have two types of nodes — dataset and column. While we could incorporate other data assets from the metadata model (from DataHub) such as notebooks and pipelines, our main focus for now will be on these two.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*34EhoEIjk6vF20eZ" /><figcaption>Datasets and Columns Represented as nodes in Knowledge Graph</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*BvA6Yybmg2nOqKtQ" /><figcaption>Column and Dataset node types with properties</figcaption></figure><p>In addition, we include metadata for these objects in the graphs, such as descriptions and sample values. Furthermore, we will tag the nodes that require embedding, specifically the datasets and reference columns.</p><p>With Neo4j’s vector index and seamless integration with LangChain, we can effortlessly utilize LLM to embed multiple properties in these nodes, resulting in a vector embedding and index that can be utilized for RAG-based queries in the next steps.</p><h3>Simple RAG based questions</h3><p>The LLM can now easily answer simple questions using vector indexes. For example, we can ask about the dataset for measuring retail foot traffic in the US or request to see zip code columns from different datasets. Behind the scenes, the LLM embeds the question and finds nodes in the graph with similar vector embeddings. This context is then used to answer the question accurately, without making assumptions.</p><p>While a simple vector database can also perform similarity searches and retrieve the closest nodes, it may struggle with more complex questions. For instance, if we ask for all columns in the foot-traffic datasets, a vector database may not be able to retrieve them if they are not mentioned in the embedded text. This is where graphs excel because they can retrieve connected nodes that are semantically related to the question. This provides more relevant context to the LLM, making it stand out from traditional vector databases.</p><p>You can find all these questions and their response with the code in this <a href="https://github.com/saeed349/LLM-Powered-Metadata-Graph/blob/main/2.rag_queries.ipynb">notebook</a>.</p><h3>Advanced Questions and using LLM for backend operations.</h3><p>Let’s try asking a more complex question to our agent.</p><p><strong>Can you show me all the datasets related to the company Starbucks?</strong></p><p>or</p><p><strong>Can you display all the datasets related to the consumer discretionary sector?</strong></p><p>However, these questions pose a few challenges that our simple RAG-based approach may not be able to handle.</p><ol><li>The data points are not yet loaded into the graph (only the metadata is loaded), so any questions about entities or datasets themselves cannot be answered from the graph.</li><li>Additionally, we are not aware of the relationships between these datasets and the columns that can be used to connect them.</li></ol><p>I will show how to deal with both these problems using LLM and LangChain. Let’s start by addressing the second issue first, using Starbucks as an example.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*uu5DvEQvf5sXjTA-" /></figure><p>Upon examining the foot-traffic dataset, we can see that the ticker for Starbucks is listed in the symbol column. However, does that mean only the foot_traffic dataset is available for analyzing Starbucks ?</p><p>Upon further investigation, we discover that the stores in the foot_traffic dataset also have post_codes, which are also present in the weather dataset, but are referred to as zip_codes. Despite the difference in terminology, a closer look at the column descriptions and sample values reveals that they represent the same information. This means that both datasets can be used to gather insights about Starbucks locations. In other words, there are two datasets that can be utilized to analyze the stock of Starbucks.</p><p>Having identified the analogues of the ticker and symbol columns in the secmaster and foot-traffic tables, respectively, and the post_code and zip_code columns in the foot-traffic and weather datasets, we can establish a new relationship between these columns in the graph. This will allow us to effectively connect and answer more sophisticated questions.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/961/0*z6zX0GVrwwEcm61H" /></figure><p>Manually sifting through data and linking columns can be a tedious task. However, with the help of LLM, this process can be expedited. The field of entity resolution mapping is vast and complex, and there are numerous techniques that can be used in conjunction with LLMs and graphs. Here are four techniques that have proven to be particularly effective for me.</p><h3>Method 1: Simple vector cosine similarity</h3><p>The simplest approach to finding similar columns in a graph is by using vector cosine similarity. This method is straightforward because we can utilize the existing node vector embeddings without having to access the LLM APIs again. By running a basic cypher query, we can compare pairs of vectors and determine their cosine similarity, which will help us identify similar columns.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*5SL9xVkHOsojT28s" /></figure><h3>Method 2: Simple prompting.</h3><p>The simplest way to match entities is through prompting, as it does not involve the use of embeddings or RAGs. However, this method can be a bit costly in terms of the matching process and LLM API consumption. This is because we need to hit the APIs close to n(n-1)/2 times, where n is the number of columns being compared.</p><pre>You are given two columns from different tables that need to be compared to  determine if they represent the same identifiers. <br>Here are the details of the columns:<br>	Column 1:<br>	name: zip_code,<br>	description: ZIP code of the location where the data was collected.,<br>	values: 10001,<br>	<br>	Column 2:<br>	name: post_code,<br>	description: Postal code where the location is situated.,<br>	values: 10001,73070,<br><br>Compare these columns and answer the following questions:<br>1.Do these columns seem to represent the same type of identifier based on name and the description?<br>2.Based on the sample values, can we infer that these values are similar enough that they are taken from the same identifier ?</pre><h3>Method 3: Simple RAG</h3><p>Another approach to consider is using a rag-based method, where we determine which embedded columns are most similar. This technique can be enhanced by using specific prompts, resulting in successful column matching. However, one drawback is that we begin with a vector search on the input column, which means that if the column or entity cannot be accurately represented by vectors, we may end up retrieving incorrect data points within the context.</p><pre>You are given the below details about a column, find columns that represent the same identifier. There could be more than one match.<br>Use only the information from the given context<br>Compare these columns based on the flowing criteria&#39;s and provide an explanation.<br>1.Do these columns seem to represent the same type of identifier based on name and the description?<br>2.Compare the sample values to infer if these values are similar enough that they are taken from the same identifier, they don&#39;t have to be the same.<br><br>Column details:<br> name: zip_code,<br> description: ZIP code of the location where the data was collected.,<br> values: 10001,<br> <br>Retrieved information:<br>{context}</pre><h3>Method 4: Advanced RAG</h3><p>The RAG method mentioned above can be combined with advanced RAG queries from the graph, where we pull in connected nodes. For instance, if we have a new column called ZIP that represents zip codes, and it matches the column zip_code from the weather dataset in a semantic search, the RAG will also fetch the connected zip_code and postal_code columns from the foot_traffic dataset because it was previously connected. This is in contrast to the simple RAG method (method 3), which would only fetch the zip_code column.</p><p>Now, imagine we are introducing a new dataset with a column called ZP that contains zip code information. By using this connected retrieval process, we are providing the LLM with knowledge from all three previously connected nodes. As we continue to add more columns, the LLM gains more information to map future columns. In my testing, this method has proven to be the most effective and scalable solution.</p><p>You can find the code for all the 4 methods in this <a href="https://github.com/saeed349/LLM-Powered-Metadata-Graph/blob/main/3.column_mapping.ipynb">notebook</a>.</p><p>In terms of LLM API cost, method 1 is the cheapest and method 2 is the most expensive and 3 and 4 is the most effective in terms of cost vs performance in my testing. These techniques are not meant to replace human judgement entirely, but rather to streamline the decision-making process for subject matter experts. The goal is to make it easier for them to determine which columns to connect. It’s not a simple yes or no answer that matters, but rather the detailed argument that the LLM creates to match the columns, providing us with the most information. By involving humans in the process, we can identify false positives and negatives and improve the LLM’s performance by using them as examples for further training.</p><p>One of the advantages of using LLMs is their ability to write SQL queries and Cyphers, Neo4j’s declarative query language. With LangChain, we can generate queries that will provide us with the desired information.</p><p>For instance, if we want to know all the available datasets, a semantic or graph search on the vector indexes we created earlier would not suffice. Instead, we need to construct a Cypher query and use its output as the context for our LLM to obtain the answer. Fortunately, Neo4j and LangChain have the necessary tools to accomplish this.</p><p>Similarly, in the context of our work connecting datasets, we can ask the LLM to create a Cypher query for the question,</p><p>Show me how I can navigate to different datasets from sec_master?</p><pre>MATCH path = (d:dataset {name: &#39;sec_master&#39;})-[r*]-(d2:dataset)<br>UNWIND nodes(path) as n<br>return n, r</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*QIGH0SBV4-5vFiq0" /><figcaption>As we can see, through the column matching exercise, we have successfully established new connections that link datasets to the security master and other datasets.</figcaption></figure><p>Let’s revisit the initial challenge we faced when trying to determine the available datasets for Starbucks. How do we bridge the gap between the metadata graph and the data layer?</p><p>One solution is to import all the data into the graphs, allowing for easier querying of entities and utilization of the methods discussed earlier. However, I must warn you, this task is more complex than it seems. The sheer number of nodes and relationships can be overwhelming, and it may require a complete overhaul of your ontology.</p><p>A more efficient approach would be to create an LLM agentic workflow that integrates both the graphs and the data stores where the information is stored. This will streamline the process and make it more manageable.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/931/0*hwiNKEbARFMLEWwA" /></figure><p>The first step in our process is to identify the entity that the question is referring to. This can be easily accomplished by embedding the sec_master table, where each row represents a document, into a vector database. From there, we can perform a similarity search to determine the entity in question.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/644/0*f0z-QfM7XydN81MC" /></figure><p>We have discovered a method to determine the connections between various datasets. With this information, we can easily create SQL queries for each path and efficiently search for the desired entities in the SQL warehouses where the data is stored. Ask LLM to find all the paths from sec_master and it will give you the below cypher query.</p><p>I need a cypher query to traverse from the sec_master dataset to related datasets? In the cypher query return the type, name and underlying table.</p><pre>MATCH path = (d:dataset {name: &#39;sec_master&#39;})-[*]-(d2:dataset)<br>UNWIND nodes(path) as n<br>RETURN n.type, n.name, n.table;</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/393/0*lSIr8jMg_qpGiKHS" /><figcaption>The cypher output with the paths. This is just one traversal path.</figcaption></figure><p>Using some simple python code, we can convert the output from our cypher query into SQL queries. By adding a where clause on the sec_master table, we can narrow down our search to the specific entity that we initially identified through the vector search.</p><p>And there we can get the below SQL query.</p><pre>select count(*)<br>from graph_db.public.sec_master a <br>inner join graph_db.public.foot_traffic b on a.ticker = b.symbol<br>inner join graph_db.public.foot_traffic c on b.post_code = c.zip_code<br>Where a.ticker = &quot;SBUX&quot;</pre><p>If our snowflake query yields any results, it means that the data exists and we can run similar SQL queries across all traversal paths for the entities or sectors that we are interested in. Thus, we can finally find out how many datasets contain information about Starbucks. In this case, there are two: the foot traffic dataset (which directly mentions Starbucks ticker SBUX) and the weather dataset (which is connected to Starbucks through zip codes in the foot traffic data).</p><p>You can find the code for the above mentioned semantic search and SQL query generation in this <a href="https://github.com/saeed349/LLM-Powered-Metadata-Graph/blob/main/4.secmaster_data_queries.ipynb">notebook</a>.</p><p>At first glance, this may seem daunting, but with the help of frameworks such as LangChain and LangGraph, we can effortlessly incorporate intricate LLM logics. In fact, in all the scenario we discussed, we have to choose between three workflows: a semantic search on the index, creating a cypher to respond to the question, or a more elaborate process involving multiple agents and retrievers. Therefore, it is crucial to utilize frameworks like LangGraph to categorize the questions and develop specific chains based on the question type.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*FWTO3nKH1xVGkVmP" /></figure><p>In this article, I have showcased a few business use cases for discovery. By combining various alternative datasets, we can unlock a whole new world of alpha research potential that goes beyond what has been discussed thus far. With the power of graphs and LLM, we can uncover valuable insights and opportunities that would have been otherwise challenging to attain. Furthermore, our focus has been primarily on basic table metadata. However, there are numerous other elements in the metadata model of data assets that we have not yet explored. These include pipelines, incidents, data users, and quality issues. Expanding our functionality to incorporate these entities would require another post solely dedicated to discussing the impact of utilizing such information sources in data operations. This idea can also be applied to other operations, such as trade or risk operations, limited only by one’s imagination.</p><p>The concept of entity resolution mapping has been a subject of extensive research, especially in the finance and banking sector. From my own experience, I have found that compared to traditional NLP methods, large language models (LLMs) have shown tremendous potential in this field. However, there is still much more to be explored in this area.</p><p>Graph-based RAG is gaining popularity in the RAG community, and I am really excited about MS open-sourcing the <a href="https://www.microsoft.com/en-us/research/project/graphrag/">GraphRAG project</a>. We have only scratched the surface with retrievers in this short project. This is a rapidly evolving area of research, and graph search is undoubtedly the future of RAGs.</p><p>For me these were the key takeaways from my experience working on the project.</p><ul><li>Utilizing the power of metadata graph and LLM to successfully tackle intricate queries related to data assets.</li><li>Connecting different alternative datasets through LLM-based entity resolution mapping.</li><li>The concept of merging structured data and metadata graph through LLM chains and graphs using tools like LangChain and LangGraphs.</li><li>The boundless potential of incorporating semantic and graph search techniques.</li><li>The critical role of maintaining a metadata graph and how it contributes to driving revenue in addition to operational efficiencies.</li></ul><p>Hope you find this useful. Feel free to let me know your thoughts.</p><h3>Connect with me on</h3><p><a href="https://www.linkedin.com/in/saeed-349/">LinkedIn </a>• <a href="https://github.com/saeed349">GitHub</a></p><h3>Connect with DataHub</h3><p><a href="https://slack.datahubproject.io/?utm_source=dhblog_medium">Join us on Slack</a> • <a href="https://mailchi.mp/a097b5cd0790/datahub-community-updates">Sign up for our Newsletter</a> • <a href="https://events.datahubproject.io/?utm_source=dhblog_medium">Subscribe to our Calendar</a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=0fca8aefcd88" width="1" height="1" alt=""><hr><p><a href="https://medium.com/datahub-project/llm-and-graph-integration-enhancing-alternative-data-exploration-0fca8aefcd88">AI-Assisted Data Catalogs: An LLM Powered by Knowledge Graphs for Metadata Discovery</a> was originally published in <a href="https://medium.com/datahub-project">DataHub</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[From Struggle to Scale: DataHub for Hedge Fund DataOps and Beyond]]></title>
            <link>https://medium.com/datahub-project/from-struggle-to-scale-datahub-for-hedge-fund-dataops-and-beyond-3c17092efe3b?source=rss-6e65ec0dcdd5------2</link>
            <guid isPermaLink="false">https://medium.com/p/3c17092efe3b</guid>
            <dc:creator><![CDATA[Saeed Rahman]]></dc:creator>
            <pubDate>Mon, 06 May 2024 12:43:32 GMT</pubDate>
            <atom:updated>2025-02-28T20:40:06.331Z</atom:updated>
            <content:encoded><![CDATA[<p>Data is the new oil and this is especially true for hedge funds and asset management firms. Their competitive edge, or alpha, hinges on superior information sources used effectively and swiftly. Thus, firms relentlessly seek diverse data sources to bolster their positions in the market.</p><p>Data teams within investment firms find cutting-edge analytical work like nowcasting and revenue modelling using ML to be exciting and rewarding. However, the biggest challenge they face is managing the rapid growth in data complexity and quality issues. As more datasets are incorporated, the operational complexity increases exponentially. This often leads to quality problems and delays, which in turn erodes the trust of downstream data consumers within the organization. Based on my experience working across different data teams, I have found that data quality checks and data metadata management are essential for scaling a firm’s data capabilities effectively.</p><p>Managing metadata offers significant opportunities for enhancing both the operational efficiency and strategic direction of a company, as well as accelerating alpha research. However, I’ll reserve the exploration of data strategy and research potentials for another discussion. Here, my emphasis will be on establishing efficient processes and platforms to support the scalability of data operations.</p><p>Among the many challenges faced by data teams in operations, three stand out:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*HgadtR5UMJKYXdvasWGtmw.png" /></figure><p>In this article, I’ll specifically address the issues of discovery and quality, leaving aside broader considerations such as data management. Rather than talking about the benefits of data quality, governance, cataloging, lineage, and observability, I’ll explore their practical application within a hypothetical organization leveraging open-source technologies. Prioritizing value creation at every stage over reaching an elusive endpoint, I seek to demonstrate a smooth, cost-effective journey and propose solutions for various scaling bottlenecks encountered by data operations teams.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/828/1*qXGNtaeXVpc7PmPMt11_Sg.png" /><figcaption>Journey to operational nirvana</figcaption></figure><h3>Hypothetical Hedge Fund</h3><p>Let’s envision a fund grappling with a fragmented data architecture where data operations systems are entangled with other operations such as risk and trading. Compounding this challenge, the data team is understaffed, struggling to meet the growing demands of the business, including onboarding new data and developing innovative tools. Despite their efforts, they find themselves constantly firefighting operational issues, leaving little time for strategic initiatives. While it’s tempting to idealize a firm with flawless architecture, the reality is often far from perfect. So let’s begin from this imperfect world and explore solutions from there.</p><p>In this arrangement, the fund initially deployed an OLTP database (SQL Server) to manage trading and operational requirements, containing certain market and reference datasets. Subsequently, they onboarded Snowflake to function as a data lake for alternative datasets and a warehouse for risk reporting. Data ETL operations happen on-premises and cloud Linux systems via CRON tab, with some ELT tasks executed both within the Snowflake warehouse and SQL Server as stored procedures, and the rest scheduled on Jenkins. Users directly interact with and query the data stores.</p><h3>Pipeline 1.0</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/800/1*_-EOT6iT041pNqgnX9ghBg.gif" /></figure><p>The initial step involves establishing a basic ETL framework before integrating any observability platform. This entails standardizing and centralizing key components, starting with scheduling using Apache Airflow. Focus should be solely on data operations without diversion to reporting or trade operations. Key actions include:</p><ul><li>Centralize scheduling using Apache Airflow and use Airflow operators like SnowflakeOperator and MsSqlOperator for integration and transformation pipelines wherever possible. For complex scripts, utilize SSHOperator or BashOperator instead of CRON to run those processes in the Linux box, as they were before. Add <a href="https://airflow.apache.org/docs/apache-airflow/stable/administration-and-deployment/lineage.html">inlet and outlet</a> definitions for lineage extraction in the subsequent steps.</li><li>Centralize logging across different machines and Airflow, leveraging tools like Filebeat to ship logs to Opensearch/ELK.</li><li>Implement source control for DAG code in Airflow, with CI/CD for ETL planned for later stages.</li><li>Set up Slack or email alerts for DAG failures.</li><li>Consider separating ELT and ETL logic if feasible.</li></ul><h3>Data Quality via Great Expectations</h3><p>Now, we will incorporate data quality assertions to assess the reliability of vendor data and data stores. <a href="https://github.com/great-expectations/great_expectations">Great Expectations</a> (GE) stands out as our preferred solution due to its extensive integrations, adaptable assertion functionalities, and the backing of a helpful community. GE enables us to deploy a variety of assertions, including:</p><ul><li>Row count to identify empty files or missing data.</li><li>Schema changes detection.</li><li>Distributional expectations for anomaly detection.</li><li>Conditional expectations — very handy for alternative and web-collection data.</li></ul><p>GE simplifies expectation checks across datasets as YAML specification and generates human-readable <a href="https://docs.greatexpectations.io/docs/reference/learn/terms/data_docs/">Data Docs</a> for expectation results. Integration into our pipeline can occur through occasional table checks, pre-loading data source checks (e.g., Write Audit Publish pattern for API data), or embedding expectations within ETL tasks themselves. The easiest and the simplest way is to apply expectations checks on data stores (tables, views) as the final task of the DAG. This approach halts downstream DAGs upon expectation failure, ensuring data quality control. Coupled with Airflow and GE alerts in Slack, we can effectively detect data delays and data quality issues.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Bcy9FXShKdGzWAHxDyOe7w.png" /><figcaption>Simple DAG with QE (original slide from DataHub <a href="https://www.youtube.com/watch?v=c7t_d3jj5t4&amp;ab_channel=ApacheAirflow">talk</a>, slightly modified)</figcaption></figure><p>After completing these two steps (Pipeline 1.0 and data quality checks), several benefits emerge:</p><p>- Centralized job monitoring through Airflow.</p><p>- Automated alerts on job statuses.</p><p>- Centralized log monitoring and investigation.</p><p>- Setting data quality expectations at the table level.</p><p>- Receiving alerts when expectations fail.</p><p>- Utilizing Data Docs to understand and address data quality failures effectively.</p><h3>Onboarding DataHub — Choosing a catalogue platform and ingesting metadata:</h3><p>Choosing an observability and cataloging platform can be a daunting task, but open source options like <a href="https://datahubproject.io/">DataHub</a> and <a href="https://open-metadata.org/">Openmetadata</a> offer compelling advantages compared to some paid options like MonteCarlo, Alation, Manta, Octopai, and Collibra. Out of the open-source versions, I’ve found DataHub to be intuitive, customizable, and feature-rich, making it a practical choice for many use cases.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/800/1*eOliCaKNNuNR_q0C1-zb0g.png" /><figcaption><a href="https://pages.acryl.io/slack">https://pages.acryl.io/slack</a></figcaption></figure><p>Another great alternative is <a href="https://www.databricks.com/">Databricks</a>, it’s a comprehensive analytical platform with built-in observability, governance, and data management features. However, its effectiveness is best when most analytical tasks are conducted within it. Unfortunately, real-world data infrastructure is often fragmented, posing challenges for Databricks’ Unity Catalog experience, particularly with external pipeline lineages and OLTP stored procedures. But one area where Databricks excels is in data and policy management, which is a big pain point for many data teams, although not discussed here, it’s worth noting. DataHub fully supports ingesting Databricks metadata, making it an excellent option even for Databricks users, as DataHub provides additional integrations and customization options.</p><p><strong>Metadata Ingestion:</strong></p><p>In this scenario, our initial focus with DataHub is to seamlessly connect our primary systems and automatically ingest metadata. DataHub’s strong integrations with Airflow, Snowflake, and SQL Server (including stored procedures) enable automatic lineage extraction down to the column level for all data assets within these systems. Its intuitive UI and CLI-based ingestion interface simplify the management of ingestion schedules. Additionally, DataHub offers powerful APIs for manually constructing lineages, providing flexibility and customization options. Furthermore, Great Expectations (GE) integrates seamlessly with DataHub, allowing us to utilize the same assertions already in place, ensuring consistent and efficient data quality monitoring across our systems within DataHub’s unified environment.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*rjKCcB8xVV9MLvaMGLIJ_w.png" /><figcaption>Cross-system lineage example within DataHub</figcaption></figure><p>Table and column-level lineage are crucial for diagnosing data problems. If a table has inaccurate or missing data, lineage helps us quickly trace it back to its source. By comparing two point-in-time (PiT) versions of lineage, we can identify changes that caused issues, like alterations in column names. Lineage isn’t just for debugging; it also helps with resource management, migrations, and more. Serving as connections/edges in our metadata graph, lineage enables us to create incident management systems, business catalogs, and reporting frameworks, adding value and improving our data operations.</p><h3>Metadata Graph enrichment — categorizing and classifying data assets</h3><p>Cataloging datasets for exploration and analysis is a fundamental task for the data team, alongside technical and operational duties. Instead of relying solely on manual documentation, generating business metadata from technical metadata is favored. This approach guarantees smooth connections between data assets in the metadata graph and what the business interacts with, fostering discussions and improving comprehension for users.</p><p>In DataHub, organizing data assets into <a href="https://datahubproject.io/docs/dataproducts/">Data Products</a> for each <a href="https://datahubproject.io/docs/domains">Domain</a> or team simplifies the structure. For instance, an Equity Pricing dataset from Bloomberg could be named “Bloomberg Equity Price Dataset” including relevant tables, views, and pipelines. Adding metadata, documentation, and vendor descriptions to data products enhances accessibility and comprehension. <a href="https://datahubproject.io/docs/tags/">Tags</a> and <a href="https://datahubproject.io/docs/api/tutorials/custom-properties/">Custom Properties</a> further help categorize data products by type, data type, asset classes, sector, etc.. . <a href="https://datahubproject.io/docs/glossary/business-glossary">Business Glossary</a> terms also aid in classification and reference management. Utilizing these properties/objects effectively enables data teams to streamline the organization of data assets. Below are some sample tags that we can use for datasets:</p><ul><li>Type: Distinguishing between different types of data products, such as datasets, reports, or analyses.</li><li>Data Type: Market data or Alternative data.</li><li>Asset Classes: Equity, Fixed Income (FI), or Foreign Exchange (FX).</li><li>Sector: Healthcare, Finance, or Technology.</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*QN2aL2v1LeHgDgFFeqEVew.png" /><figcaption>Dataset as a Data Product (original slide from DataHub <a href="https://www.youtube.com/watch?v=vpz62mpvUVs&amp;ab_channel=DataHub">talk</a>, slightly modified)</figcaption></figure><p>Going beyond conventional datasets, data products can include diverse combinations of analytics, providing endless opportunities. For example, the “Retail Whitespace Analysis” can be an analysis that merges foot-traffic and demographic datasets with market datasets to evaluate a retailer’s geographic placement. This showcases the expansive range of possibilities for data product scope, bound only by creativity. Such classification brings considerable value to the research and development process.</p><p>It&#39;s also important to assign owners to the Data products. And DataHub has this <a href="https://datahubproject.io/docs/ownership/ownership-types/">custom ownership type</a> so we can have the roles and user titles according to the firm&#39;s structure. At this point, you can choose to add tags and owners to the individual data items or let them inherit from the data products.</p><h3>Business Catalogue and an opportunity to connect to external workflows</h3><p>The business catalog serves as the gateway for business and end-users to discover and manage available datasets. While a significant portion of the metadata graph contributes to the business catalog, additional aspects related to vendor contractual items, team assignments, sharing terms, costs, etc., are crucial components as well. Acquiring this information often involves digging through contracts, underscoring the importance of cross-team collaboration , documenting, and being able to connect these different aspects together. Now the question arises: how can we achieve this on a large scale? I believe the most effective approach is to broaden the reach of metadata graphs to external systems and functions. This allows us to establish connections and monitor activities efficiently. Through this method, we can seamlessly document and collaborate with cross-functional teams such as project management, vendor management, compliance, accounting, and various front office functions.</p><p>Putting this into practice is no simple task, as each of these teams operates with distinct systems, processes, and terminology. Initially, the approach is to utilize the reference of the relevant metadata graph object of a dataset (Data Product) when collaborating with these functions. For instance, consider using Jira for project management and having a task associated with a specific dataset. If we incorporate the Data Products URN as a property for this task entity in Jira, it becomes effortless to trace tasks, projects, and Jira objects back to the metadata graph and vice versa. This concept can be extended to other functions such as vendor management, data onboarding, compliance, accounting, and beyond. The goal is to establish connections between different functions to drive value across the organization.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/664/1*As7mGoo5aMmzJfpW629d6Q.png" /><figcaption>Metadata objects are the glue connecting different functions</figcaption></figure><p>One aspect of DataHub that I appreciate is its intuitive and straightforward design. Paired with workflow tools like the <a href="https://datahubproject.io/docs/actions/">Actions Framework</a> and extensive programmatic access, it offers ample opportunities to tailor the platform and workflows to our specific needs. I strongly encourage you to explore the <a href="https://www.youtube.com/watch?v=ejcO7hdX0lk&amp;list=PLdCtLs64vZvGCKMQC2dJEZ6cUqWsREbFi&amp;pp=iAQB">DataHub Customer Adoption Stories</a> and their <a href="http://blog.datahubproject.io/">blog</a> to discover the diverse ways in which users have leveraged the platform.</p><h3>Incident Management</h3><p>Late data, quality problems, and system downtimes are inevitable in operations, making incident management a vital aspect of data teams. Establishing a data incident management workflow for your pipelines involves four key steps:</p><ul><li>Incident detection and communication</li><li>Root cause analysis</li><li>Resolution</li><li>Post-mortem analysis</li><li>Managing audit trails.</li></ul><p>While using a service management tool like Jira is common practice, integrating it with an observability platform and catalog would greatly enhance the process and user experience, considering the workflow outlined above. Thankfully, DataHub provides a method to mark the unhealthy status of a data asset using the <a href="https://datahubproject.io/docs/next/incidents/incidents/">Incidents</a> object. Each incident follows a lifecycle, featuring attributes such as state, title, and description.</p><p>Efficient communication is vital for effective incident management. The objective is to promptly detect issues and proactively inform stakeholders, avoiding end-users discovering problems first, which could undermine confidence in data quality and reliability. DataHub’s robust API support enables automatic incident creation triggered by pipeline and assertion failures. <a href="https://www.acryldata.io/">Acryl DataHub</a> ensures asset owners and users receive prompt alerts about incidents, facilitating swift resolution and tagging affected assets within the platform. Additionally, with API support and functionalities like <a href="https://datahubproject.io/docs/actions/">Actions</a>, we can effortlessly develop tailored solutions and integrate them with your customized service management tools.</p><p>Using end-to-end table and column lineage, along with tracking airflow task and assertion statuses in DataHub, and leveraging centralized logging enabled by ELK, we can swiftly pinpoint the root cause of issues. Furthermore, centralized catalog management allows for easy identification of dataset owners (internal/external) and facilitates issue escalation. Additionally, with point-in-time audit trails of incidents, pipeline runs, assertions, and categorized metadata, we can conduct various analyses and generate reports effortlessly. Moreover, ETL processes can be configured to execute only when there are no active critical incidents, thereby boosting operational efficiency and reducing potential data issues.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/908/1*5WpH__4geqq3ppazkdkzPQ.png" /><figcaption>Automatic Incident Detection and Alerting</figcaption></figure><h3>Reporting and data driven decision making</h3><p>Now, with all essential information readily available via the metadata graph, effective communication and decision-making become achievable. SLAs and KPIs can be established for both internal and external stakeholders</p><p>Through the metadata graph, you have the following data points at your disposal:</p><ul><li>Categorized list of data assets with ownership.</li><li>Incident details associated with data assets.</li><li>Usage metrics.</li><li>Pipeline status history.</li><li>Data quality issues.</li><li>SLA’s breaches with external data vendors and internal stakeholders.</li></ul><p>Using these metrics we can easily create data points that can drive</p><ul><li>Vendor negotiations.</li><li>Internal stakeholder meetings.</li><li>Defining KPIs for Data teams.</li><li>Project management for Data team.</li><li>Data operation gap analysis</li><li>Data Steering committee meetings.</li><li>Data Strategy gap analysis.</li></ul><p>If we establish connections with other functions such as vendor management, accounting, and project management through the metadata graph, the potential is even more.</p><h3>Pipeline 2.0:</h3><p>Although significant progress has been made in achieving observability and cataloguing , it’s important to acknowledge that we have only scratched the surface. There’s still much ground to cover and many opportunities for further improvement lie ahead. In the current iteration of Pipeline 1.0, we’ve focused on fundamental aspects to ensure data quality and observability. However, to attain a fully scalable platform, ongoing updates to our systems and processes are essential. Here are a few additions you can implement in the pipeline to enhance its performance, observability and governance even further.</p><ul><li><strong>Data Contract</strong></li></ul><p>The notion of Data Contracts is relatively recent, with different platforms offering diverse interpretations and iterations of it. Essentially, it’s a document that points to a specification outlining the structure, format, and meaning of data exchanged between two parties. Within DataHub, Data Contracts consist of assertions, which are verifiable statements enforced on individual data assets. These assertions pertain to schema-related attributes, service level agreements (SLAs), data freshness, and data quality.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/726/1*qfCVXr7bk4JM-QaU3VIhgQ.png" /><figcaption>Sample Data Contract in DataHub (from DataHub <a href="https://www.youtube.com/watch?v=bVKuLcqv0kI&amp;ab_channel=OSACon">talk</a>)</figcaption></figure><ul><li><strong>Metadata driven ETL</strong></li></ul><p>Initially, we introduced a basic pipeline setup where downstream DAGs fail if upstream DAGs’ assertions fail. However, this setup is complex for several reasons. We’re treating assertions as task failures and creating dependencies on DAGs, which may not always be scalable, especially considering different scheduling needs. An alternative approach could involve confirming the availability of the latest data in the upstream table before initiating downstream DAGs. However, this method may overlook other data quality issues. Therefore, there’s a need for an efficient method to convey assertion status. While some may consider using Xcoms and other Airflow components to address this issue, it could introduce complex dependencies in pipelines. A more straightforward option is to integrate the metadata graph into your pipelines to verify the health and freshness of data assets. Through the utilization of incident statuses and assertion results, we can construct simple pipelines without intricate dependencies between DAGs.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Ltb11WUPvspf8Vswv9JXyQ.png" /><figcaption>Metadata-driven pipelines (original slide from DataHub <a href="https://www.youtube.com/watch?v=c7t_d3jj5t4&amp;ab_channel=ApacheAirflow">talk</a>)</figcaption></figure><ul><li><strong>Refactoring Pipelines</strong>: Start by dissecting each ETL script/process separately, distinguishing between ETL and ELT workloads. Integrate Airflow operators wherever possible to extract entities and trace lineages. Eliminate pipeline dependencies on specific machines by managing and standardizing dependencies and packages, enabling Airflow pipelines to run on scalable infrastructure workers like Celery or Kubernetes.</li><li><strong>Standardizing ELT with DBT</strong>: For those seeking to implement development-like lifecycles for ELT processes, DBT offers an excellent option with its strong integration with DataHub.</li><li><strong>Standardizing ETL Procedures</strong>: Since most vendor data integrations adhere to specific patterns for fetching, processing, and validating assertions and SLAs, it’s advisable to maintain a library of common pipeline templates. Each pipeline should inherit these standardized behaviors.</li><li><strong>Enhancing Source Metadata</strong>: Enhance table and other data asset DDLs with descriptions and comments to ensure they are included in the metadata graph during metadata ingestion.</li><li><strong>Implementing CI/CD</strong>: With standardization comes the need for thorough checks and tests for every code change. This also presents an opportunity to enforce data governance standards and add tests to verify the presence of objects like data contracts, inlets and outlets for unsupported operators etc…</li><li><strong>Compiling Guidelines</strong>: Develop documentation outlining the steps for constructing ETL scripts, DAGs, and similar components to ensure adherence to proper standards.</li></ul><h3>Beyond Graph</h3><p>Graphs present boundless opportunities beyond mere operational duties. I’d like to wrap up by envisioning an intriguing possibility: merging knowledge graphs with the capabilities of LLMs. This amalgamation could offer a wholly new outlook on operations, observability, data strategy, and research. I’ll explore this concept further and share the progress I’ve made using metadata graphs in a separate post.</p><h3>Conclusion</h3><p>When I joined my first Data team, I was tasked with constructing a Catalogue for alternative datasets. It was a thrilling challenge for a newcomer like me. However, gathering the necessary information proved tedious, involving combing through code, documentation, and conversations with colleagues. We had around 50 active datasets, with the top 30 being the most used. Additionally, there were 50 inactive datasets and over 100 trialed datasets. Focusing on cataloging just the top 30 active datasets adds more value than attempting to handle all 200+. This principle also holds true for data quality and observability; typically, 20% of datasets contribute to 80% of the challenges and value. Starting with these critical datasets and addressing major pain points not only saves time but also establishes a robust foundation for future development.</p><p>When you speed ahead, mistakes occur and fires ignite, but now they’re less intimidating because you possess the tools to handle them. It’s a chance to evolve and prepare the platform for genuine innovation. It’s also crucial to share this perspective and educate other firefighting teams within your organization so they can leverage and improve upon what’s already established.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1000/1*iHIDeLBWy456akafTy9lXA.png" /></figure><h3>Connect with me on</h3><p><a href="https://www.linkedin.com/in/saeed-349/">LinkedIn </a>• <a href="https://github.com/saeed349">GitHub</a></p><h3>Connect with DataHub</h3><p><a href="https://slack.datahubproject.io/?utm_source=dhblog_medium">Join us on Slack</a> • <a href="https://mailchi.mp/a097b5cd0790/datahub-community-updates">Sign up for our Newsletter</a> • <a href="https://events.datahubproject.io/?utm_source=dhblog_medium">Subscribe to our Calendar</a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=3c17092efe3b" width="1" height="1" alt=""><hr><p><a href="https://medium.com/datahub-project/from-struggle-to-scale-datahub-for-hedge-fund-dataops-and-beyond-3c17092efe3b">From Struggle to Scale: DataHub for Hedge Fund DataOps and Beyond</a> was originally published in <a href="https://medium.com/datahub-project">DataHub</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Seeking Alpha in Alternative Data]]></title>
            <link>https://usquant349.medium.com/seeking-alpha-in-alternative-data-b520a576af78?source=rss-6e65ec0dcdd5------2</link>
            <guid isPermaLink="false">https://medium.com/p/b520a576af78</guid>
            <category><![CDATA[quant]]></category>
            <category><![CDATA[investment]]></category>
            <category><![CDATA[trading]]></category>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[artificial-intelligence]]></category>
            <dc:creator><![CDATA[Saeed Rahman]]></dc:creator>
            <pubDate>Tue, 30 Apr 2024 17:10:14 GMT</pubDate>
            <atom:updated>2024-04-30T17:11:05.771Z</atom:updated>
            <content:encoded><![CDATA[<p>A retail transaction creates thousands of data points, based on where you shop online or a brick and mortar store. That includes data like payment source, shopping location, your demographics and exactly what choices you make in the store. This might not interest the average Joe but for a Hedge Fund manager this is priceless, alpha as they like to call it. It’s a thing that happens every day: Investors make money off data that you’re giving them for free.</p><p>This article is about my experience working on an Alternative Dataset for the <a href="https://www.linkedin.com/posts/m-science-llc_battlefin-alternativedata-datascience-activity-6545048335059832833-KVz6">Battlefin Alternative Datacombine Competition</a></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*JZLzI558EAQfZBka" /></figure><p>Anyone who have ever used Netflix, Spotify or any other subscription service would have come across a similar email sitting in your inbox. What if we had access to the email inbox of hundreds of thousands of Netflix subscribers, could we use this information to extract potential insights about the company that can drive investment or trading decisions. Well that was the challenge I was given, given the Email Purchase Receipts data of nearly 1 million anonymised US individuals over the span of 3 years, extract insights that could be used for investments decision on Netflix and Spotify.</p><p>Below you can see a mock-up of the dataset that I used for the competition.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*JoBMpuCOepihAMBu" /></figure><p>A natural tendency for many who are starting in the predictive analytics space is to be sidetracked into modelling when dealing with structured dataset, I have done that mistake before but nowadays I listen to what the data has to say, before deciding the direction to navigate. Always remember one thing “if you torture data, it will confess to anything”</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*9fxugklzMMGtaQH3" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*04hPInL-KGrVRUaD" /></figure><p>300K unique users is a very small sample size considering the 60 million active US subscriber base of Netflix, but just like any statistical analysis, we will assume this data has been randomly sampled from the population.</p><h3>Preprocessing</h3><p>There are quite a few caveats before getting to a clean dataset, if we just search through the emails with the word “Netflix” in the email subject line, we would find a lot of transactions that are not a direct contributor to the Netflix revenue, for example you can find Netflix “Gift cards” in Amazon which directly contributes to the revenue of Netflix but also goodies that has the word “Netflix”. They might not contribute directly to the Revenue however they definitely have an impact on brand equity and the trick is accommodating such idiosyncrasies across merchants and deriving appropriate features.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/755/0*pUzduQWGxbOe7ZG_" /></figure><p>Eventhough the dataset had a rich mapping of emails to public companies and their products, it still had a lot of unmapped emails. In a dataset like this you could easily spend couple of days to make it spotless, but the objective should be building a preprocessing pipeline that will generalize all the peculiarities across all the merchant emails.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*TrAKyfsDna-J5pTd" /></figure><h3>The Key Performance Indicators</h3><p>Now that we have clean structured dataset, we can transform this into insights that can be used by the target users. Creating KPIs that are pertinent to the data is the most important step as this is where we merge the domain knowledge and modelling to filter alpha from noise.</p><p>For customer data, one of the most critical KPI category is churn modelling, unfortunately we don’t have enough data on customer characteristics to model the churn. But we still have a lot of information about transactions. Therefore I decided to stick with a few critical KPI’s that I felt suited well for this dataset.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*m645S5s2zmNhKAT_" /></figure><h3>Customer Lifetime Value (CLV)</h3><p>CLV is a measurement of how valuable a customer is to your company with an unlimited time span as opposed to just the first purchase. This metric helps you understand a reasonable contribution per customer. There are a couple of ways we can calculate CLV and I chose the probabilistic models BG/NBD and Gamma-Gamma models due to the relatively small size of the dataset.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/688/0*W5y1gKKyKouA7NKK" /></figure><h3>BG/NBD Model — Calculate expected number of purchases within next n days</h3><p>The basic Pareto/NBD model simulates two events. It uses a “coin” to determine whether a customer churns and then it uses “dice” to determine how many items a customer will order. The coin is modeled using a Pareto distribution and the dice are modeled using a negative binomial distribution. The more information you have on a customer, the better the models can fit them to a specific distribution and the more accurate the predictions end up being.</p><h3>Gamma-Gamma Model — Calculate Average Conditional Transaction Value</h3><p>The Gamma-Gamma model behaves much like the underlying concepts driving the BGF model: heterogeneity. What is the mean transaction value of our customer base? Well we can often estimate that with the sample mean and variance… if our customer base was symmetrically and normal distributed. Since this isn’t the case (1. non-negative transaction value 2. right skew), we can use the Gamma distribution to approximate the distribution of the mean transaction value for each customer. In short, we can calculate the conditional average transaction of our customer base versus the observed average transaction using Gamma-Gamma.</p><h3>The problem definition</h3><p>Now that we have defined our KPI’s, the next step is how can someone use it. If we look from an investing/trading perspective, there are 2 scenarios where this insights would come handy.</p><p>1. To trade earnings:</p><ul><li>Building a position towards earnings day.</li><li>Trading the earning day.</li></ul><p>2. Short Term Investment:</p><ul><li>Portfolios with rebalancing period of 2–3 months.</li><li>Use the CLV and others KPI’s as input into Valuation models.</li></ul><p>Example: Netflix Q1 2016 Earnings:</p><p>By end of Q1 — March 2016, we have the KPI’s available for Q1 and the earnings announcement happened Mid April, so technically you can use the KPI’s to estimate the earnings and other important fundamentals of Netflix before its released on the earnings date.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*PPo90RsxAjqhUhYF" /></figure><h3>Final Feature-Target Space</h3><p>I decided to make the final dataset at quarterly frequency with the feature sets as the KPI’s and the target variables as future returns, so that it can be easily consumed by Research Analysts who are focused on modelling earnings and also Quants who wants to feed it into Supervised Learning Models.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*0_d_NUpPr6vPkoST" /></figure><h3>Insights</h3><p>Since this is a proprietary dataset and there has been quite a few NDA’s that I have signed with the vendor so I will hold off on sharing the full analytics and numbers, but will do share some interesting trends in the KPIs.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/765/0*5ZW-qbZPpYEWEbRB" /></figure><p>Netflix, average spending across customers in the email dataset had a positive correlation with the forward returns (end of quarter till the earnings announcement date)</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/797/0*ruFt32KTSXAxU1J0" /></figure><p>The Netflix drawdown that started in Q2 2018 that lasted till Q1 2019 was predicted by a sluggish CLV growth.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/806/0*WUXsC5QsLdkLZCHS" /></figure><p>The average number of transactions growth vs returns had a high correlation for LinkedIn</p><p>Now if we want to use the same dataset for training Machine Learning models, a good way to go about it would be sticking to simpler models especially since its a small dataset. One important thing to keep in mind is that we are going to end up with a lot of features when we are aggregating to lower frequency (intraday to quarterly) and a feature selection step is critical particularly in investing domain where interpretability is as important as the accuracy of the model.</p><p>Below you can see the ML pipeline I used for this project</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*bzR5U9ssApA00guZ" /></figure><p>Just like any machine learning enthusiast when I started out I used to throw data at advanced models like LSTM and CNN expecting magic. It didn’t take me that long to understand that in the field of investment it takes more than just R-squared and F-score to justify predictions, there needs to be rational economic sense to what the model is predicting. That’s where domain knowledge is crucial in getting insights from these datasets and you can get there by asking questions throughout the process, starting from cleaning and exploration to modelling. Example: A very common customer analytic metric is daily purchase frequency, having that metric for a business who’s revenue is driven by monthly subscription like Netflix is sometimes just noise and asking these important questions throughout the process has helped me to build deeper understanding about the data.</p><p>By 2020 it is expected that each individual living in the digital age would produce 1.7 megabytes of data every second. That’s unfathomable amount of data and individually picking out datasets and getting insights would be next to impossible. And that’s where were we have to use automated analytics and data workflows into the equation so that we can reuse a lot of the processes for similar datasets . Below you can see the overview of the pipeline I used for this project.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*nm6vepTCPe0bZtWX" /></figure><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=b520a576af78" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Deep Reinforcement Learning in Trading]]></title>
            <link>https://usquant349.medium.com/deep-reinforcement-learning-in-trading-2de50f1b2271?source=rss-6e65ec0dcdd5------2</link>
            <guid isPermaLink="false">https://medium.com/p/2de50f1b2271</guid>
            <category><![CDATA[algorithmic-trading]]></category>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[reinforcement-learning]]></category>
            <category><![CDATA[artificial-intelligence]]></category>
            <category><![CDATA[quant]]></category>
            <dc:creator><![CDATA[Saeed Rahman]]></dc:creator>
            <pubDate>Tue, 30 Apr 2024 17:08:04 GMT</pubDate>
            <atom:updated>2024-04-30T17:08:04.763Z</atom:updated>
            <content:encoded><![CDATA[<p>Reinforcement learning is an exponentially accelerating technology inspired by behaviorist psychologist concerned with how agents take actions in an environment so as to maximize some notion of cumulative reward. This technology is the key driver behind AlphaGo, self driving cars and many other Artificially Intelligence (AI) applications.</p><p>Thanks to opensource framework like OpenAI Gym and Deepmind Lab anyone can create cutting edge AI’s and test on thousands of games and real life scenarios, whether it be the game Super Mario or modeling a robot to walk.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/283/0*EPBdQ-fdz_vNOM9_" /></figure><p>In this simple example our agent (Mario) sees an opponent and the most optimal action to do is jump to avoid the opponent, meanwhile if Mario can hit the block or move forward to the end of the level, it would get reward (points or advance to next level) . The only way to win this game is to get as much reward as possible without getting killed. How can an agent do this. That’s where reinforcement learning comes into play, I won’t go into the details of it and make this article boring, but in short its a combination of supervised and unsupervised learning applied in an optimization technique borrowed from control theory. While supervised learning gives us the power of function approximation, reinforcement learning gives us policies, which tell what actions to take.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/391/0*tkDAOUJyRRZDEV8M" /></figure><p>The agent(Mario) is our AI. During the learning process the agent will see the environment (game screen in pixels) and it will take random actions in the beginning, and these random actions would result in some rewards (resulting from scoring points, rescuing the princess, advancing to the next level…) or getting killed. Say the agent test out all the buttons at the beginning of the game and it found that the “right” button gives it the maximum reward, since it moves forward in game. Similarly after each episodes of interaction with the environment, our agent will learn new maneuvers.</p><p>This mechanism of learning might sound so similar, because we have done it and experienced it from our very existence.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/300/0*6JR9qJozHvwSF_gH" /></figure><p>A child (a.k.a. the agent) learns to walk through many failures, but after each failure it is the dopamine and serotonin rush inside its brain invoked by the curiosity seeing other 2 legged agents walking around him that motivates (rewards) him to repeat this action until he nails it. Even when we grow up, all our learning whether it be in school or at home have been incentivised by some reward-penalty structure.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/750/0*YmzqpVp5kpVgZw5Z" /></figure><p>Its not just humans, look around and you can see this learning process in all living beings. Reinforcement learning has adopted it principles from nature and neuroscience.</p><p>Brain’s neuroplasticity is well recognized fact, which states that our brain is plastic and the physical connections between neurons inside the brain changes through an individuals life mainly through learning and its experience’s from the interaction with external world. Similarly in reinforcement learning, our agent has a brain, made using Neural Networks. The agent’s interaction with the environment which changes the weights (or connections) between these neurons gives it the ability to take actions that will maximize the reward.</p><p>Well, why not stock markets? If we can build an agent to play a game, let’s create an agent to trade the market. However, the markets unlike a game, is filled with randomness and complexity. In a game, the states are deterministic, if Mario sees an opponent coming its way, then the rationale thing to do is jump over it and the outcome is predictable, but in the stock markets its not the case, states are random in nature. For example, an agent might have learned a particular policy from a pattern that occurred in the historical data during training, but its not necessarily true that the next time this pattern occur in the market the outcome would be the same, that is the market has a lot of uncertainty in it. But just like in the case of price prediction in general, we don’t have to be always right. When fused with the right risk management techniques (like position sizing, diversification… ) strategies can be profitable in the long run even if its not always right.</p><p>During my final semester at school, I decided to try reinforcement learning in trading. Long story short, I used a Double-Dueling DQN (improvement over the original DQN paper <a href="https://storage.googleapis.com/deepmind-media/dqn/DQNNaturePaper.pdf">Human-level control through Deep Reinforcement Learning</a> published by Google’s Deepmind to play 2600 Atari games) algorithm to design an agent to trade in a single stock environment.</p><p>With a few tweaks and examples from OpenAI’s Gym game environments and some other open source projects(<a href="https://github.com/prediction-machines">Trading environment framework</a>), I was able to setup a single security environment that could ingest historical market data for training and testing.</p><h3>First experiment:</h3><p>The state that the agent receives at each step would be:</p><h3>[price(𝑡−1), price(𝑡), 𝑝𝑜𝑠𝑖𝑡𝑖𝑜𝑛 𝑝𝑟𝑖𝑐𝑒, 𝑝𝑜𝑠𝑖𝑡𝑖𝑜𝑛]</h3><ul><li>𝑝𝑟𝑖𝑐𝑒(𝑡) = current price</li><li>𝑝𝑟𝑖𝑐𝑒(𝑡-1) = price before 1 time period,</li><li>𝑝𝑜𝑠𝑖𝑡𝑖𝑜𝑛 𝑝𝑟𝑖𝑐𝑒 = position entry price</li><li>𝑝𝑜𝑠𝑖𝑡𝑖𝑜𝑛 = current position ( can be 𝑙𝑜𝑛𝑔, 𝑠ℎ𝑜𝑟𝑡 𝑜𝑟 𝑓𝑙𝑎𝑡)</li></ul><p>The agent can take three actions — Buy, Sell or Hold</p><p>However, the agent wasn’t learning much from the two price points. When a trader looks at price chart, they look at 100,1000 or the entire price series before taking a position. Then why not feed 100’s of past price points, but to keep things simple I decided to think in terms of a trader. What does a proprietary trader consider before making a trading decision. Most of the times price is just too complicated to decipher, so traders use function of price and volume commonly known as technical indicators. So I decided to stick with 3 Oscillators, which is already stationary and only scaling is needed before feeding into the neural network.</p><p>The design of the trading environment has been that only realized P&amp;L is passed onto to as rewards, but in the real world a trader also looks at the unrealized P&amp;L to know when to close their positions. So to imitate this, unrealized returns was also included as one of the state factors.</p><h3>New State</h3><p>[𝐴𝐷𝑋(𝑡), 𝑅𝑆𝐼(𝑡), 𝐶𝐶𝐼(𝑡), 𝑉𝑜𝑙(𝑡), 𝑝𝑜𝑠𝑖𝑡𝑖𝑜𝑛, 𝑢𝑛𝑟𝑒𝑎𝑙𝑖𝑧𝑒𝑑 𝑟𝑒𝑡𝑢𝑟𝑛]</p><ul><li>𝐴𝐷𝑋 = 𝐴𝑣𝑒𝑟𝑎𝑔𝑒 𝐷𝑖𝑟𝑒𝑐𝑡𝑖𝑜𝑛𝑎𝑙 𝐼𝑛𝑑𝑒𝑥</li><li>𝑅𝑆𝐼 = 𝑅𝑒𝑙𝑎𝑡𝑖𝑣𝑒 𝑆𝑡𝑟𝑒𝑛𝑔𝑡ℎ 𝐼𝑛𝑑𝑒𝑥</li><li>𝐶𝐶𝐼 = 𝐶𝑜𝑚𝑚𝑜𝑑𝑖𝑡𝑦 𝐶ℎ𝑎𝑛𝑛𝑒𝑙 𝐼𝑛𝑑𝑒𝑥</li><li>𝑉𝑜𝑙 = 𝑉𝑜𝑙𝑢𝑚𝑒</li></ul><p>Reward: Realized P&amp;L — Trading Fee — Time Fee</p><p>To emulate the real world and control the actions of the agent , I also included a commission and a Time Fee (like Rollover rate in Forex or in general time value of money). These rewards are what controls and optimize the agents behaviors during the training phase.</p><p>This new State-Reward structure improved the performance of the agent multi-folds. But, whatever the agent learned wasn’t transferable across different securities or in a way what works for Apple doesn’t always work for Microsoft.</p><p>The agent was trained and tested with all the different stocks in S&amp;P 500 between the year 2013 to 2018 with a 3:1 split and the average results were better than a buy and hold strategy during the same time for these respective stocks.</p><p>There are many improvements that I would make to this project, starting with</p><ul><li>Prioritized experience replay instead of experience replay</li><li>LSTM network instead of Vanilla Neural Network</li><li>Convolutional Neural Network (a bit counter-intuitive since we already have the price, but worth testing out to see if it can overcome the effect of memory loss resulting from forcing stationarity, an idea from Dr. Marcos Lopez)</li><li>A3C (Asynchronous Actor-Critic Agents) algorithm instead of DDDQN.</li><li><a href="https://pathak22.github.io/noreward-rl/">Curiosity-driven Exploration by Self-supervised Prediction</a></li><li>Multi-agent algorithms and many more….</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1005/0*UomSMcGqEkW_O7A6" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/565/0*yWVvnFuq_0Nbz9iB" /></figure><p>A typical Quant strategy workflow involves several risk, portfolio and control steps after the predictive modelling and before the execution. Most of the time these modules are loosely coupled and involves frequent calibration and optimization in the strategy life cycle. But with the use of reinforcement learning, we can engineer these constraints within the reward and state structure.</p><p>A simple example could be an agent dealing with portfolio of stocks. A common requirements among most investors is to build a portfolio with Beta close to 0 (uncorrelated to the market) and a popular method to do this is to build a Long-Short Factor based portfolio. Using reinforcement learning, we can create a similar portfolio in which we penalize the agent during the training phase if Beta doesn’t stay within a range close to zero.</p><p>Supervised and unsupervised learning had made great strides in trading, but I believe the next biggest thing is reinforcement learning and it has limitless potential because it is the closest technology to general intelligence.</p><p>To learn more — Check out my <a href="https://1drv.ms/b/s!AhhAHu3VZUOPhdF23xPByWm64mSf-w">report</a> and <a href="https://github.com/saeed349/Deep-Reinforcement-Learning-in-Trading">repository</a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=2de50f1b2271" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Quant Platform for Cash Equities]]></title>
            <link>https://usquant349.medium.com/quant-platform-for-cash-equities-e817ba967db0?source=rss-6e65ec0dcdd5------2</link>
            <guid isPermaLink="false">https://medium.com/p/e817ba967db0</guid>
            <category><![CDATA[trading]]></category>
            <category><![CDATA[quant]]></category>
            <category><![CDATA[algorithmic-trading]]></category>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[artificial-intelligence]]></category>
            <dc:creator><![CDATA[Saeed Rahman]]></dc:creator>
            <pubDate>Tue, 30 Apr 2024 17:03:16 GMT</pubDate>
            <atom:updated>2024-04-30T17:03:16.770Z</atom:updated>
            <content:encoded><![CDATA[<p>Embarking on a journey through the realms of data analysis, quantitative research, and machine learning has led me beyond the expected boundaries. My passion extends not only to the intricacies of these fields but also to the essential work in quant, infrastructure development, and ETL processes — critical for unlocking the full potential of data analysis and ML endeavors.</p><p>In 2019, my exploration took a turn as I ventured into testing and running systematic strategies for the forex spot market, resulting in the birth of <a href="https://github.com/saeed349/Microservices-Based-Algorithmic-Trading-System">MBATS</a>. Transitioning to the realm of cash equities, scalability became paramount, prompting the migration of key architectural components to the <a href="https://github.com/saeed349/Quant-Trading-Cloud-Infrastructure">cloud</a>. Now, four years later, my research and trading infrastructure has undergone a significant transformation. Following a recent shift from a full-time role, my sabbatical became an opportunity for personal data projects and a revamp of my infrastructure for scalable and cost-efficient operations. This journey also included aiding a former colleague in establishing the data infrastructure for a new fund. Eager to contribute to the community, I aim to share the architecture, design, and insights gleaned from this buildout, driven by valuable feedback received from prior projects such as <a href="https://github.com/saeed349/Microservices-Based-Algorithmic-Trading-System">MBATS</a>.</p><p><a href="https://www.quantconnect.com/">Quantconnect</a> stands as a remarkable Quant trading and research platform and has undergone significant evolution over the years. But sometimes our requirements can get a bit more intricate like working with multiple external datasets that needs staging, and performing resource intensive computations on these dataset (the use cases are explained in details <a href="https://github.com/saeed349/quant_infra/wiki#use-cases">here</a>), then we need to find ways to boost the platform’s performance and capabilities. Fortunately Quantconnect was built on open-source technologies by an amazing community, and they have made the platform very flexible and feature rich that I was able integrate it easily with my external tools.</p><p>The objectives for this setup were pretty straightforward:</p><ul><li>Be able to ingest and wrangle a wide range of market and alternative datasets (structured and unstructured).</li><li>Perform complex computations on these datasets.</li><li>Implement deep learning models, reinforcement learning agents, and intricate simulations.</li><li>Backtest strategies and transition these strategies into production with ease.</li><li>Being able to run fully systematic strategies on US Cash Equities.</li><li>Built tools, reports and alerts to help with discretionary trading.</li><li>Try not to reinvent the wheel — leverage powerhouse platform like Quantconnect handle the heavy lifting wherever possible.</li><li>To be able to do all at the lowest cost possible.</li></ul><p>In this article I will only highlight the important points, but you can read the detailed design in the project <a href="https://github.com/saeed349/quant_infra/wiki">wiki</a>.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*1Tn69E0vvXkkD2AK" /><figcaption>Architecture High Level overview</figcaption></figure><ol><li>First, the data is retrieved from vendor APIs and stored in S3, later transferred to a Data Warehouse such as Databricks/Snowflake.</li><li>Subsequently, all business logic is executed within the warehouse using SQL, Pyspark/Snowpark.</li><li>ML research and modeling is conducted on bare EC2 machines or <a href="https://lambdalabs.com/">Lambda Lab</a> machines with <a href="https://lambdalabs.com/lambda-stack-deep-learning-software">Lambda Stack</a> as the package management layer. Experiment tracking is done on Weights &amp; Biases (hosted) and the model is served via Mlflow.</li><li>Backtesting is performed in <a href="https://www.quantconnect.com/">QuantConnect Cloud</a> for supported markets, while for unsupported markets and datasets, backtests are executed locally via <a href="https://www.quantconnect.com/docs/v2/lean-cli">QuantConnect LEAN CLI.</a></li><li>Live strategies are executed on both the QuantConnect Cloud platform and Lean CLI based on the strategy, market, and resource requirements.</li><li>ETL, ELT, ML model training and inference, strategy deployment &amp; management are all managed using a self-hosted Airflow on AWS EC2.</li><li>Data Warehouse to QuantConnect Cloud data transfer and AWS resource management are handled using AWS Lambda.</li><li>All logs, including live trading and backtesting logs, as well as order files from QuantConnect, are loaded, parsed, and visualized in AWS OpenSearch (analogous to ELK stack) with Filebeats as the shipper.</li><li>Plotly Dash is employed for visualization and charting for discretionary trading.</li><li>Slack serves as the reporting and alert platform. All Airflow DAG success statuses are reported here, and signal, trading, and risk/performance PDF reports are generated and distributed via Slack as well.</li></ol><p>Looking at the diagram above, one might observe that there are a few key components missing — questions arise about optimizers, OMS, risk framework/engines, reconciliation setup, and backoffice systems. I explain the reason why they are missing <a href="https://github.com/saeed349/quant_infra/wiki#other-important-systems">here</a>.</p><h3>Signals to Quantconnect</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/989/0*TtYe-MZvHrJ6dw0q" /></figure><p>Data undergoes processing to generate signals, and if machine learning models are part of the strategy, their inferences are stored in the warehouse. These inferences are then further processed into trading signals. Subsequently, these signals are dispatched to the strategy, either through a REST API (leveraging AWS Lambda) for Quantconnect Cloud or via the data warehouse’s Python package for Quantconnect Lean (via <a href="https://www.quantconnect.com/docs/v2/cloud-platform/api-reference/downloading-data">QC Download</a>). You can read more about 2 sample signals and the possible ways to mitigate look-ahead bias on signals <a href="https://github.com/saeed349/quant_infra/wiki#signal-examples">here.</a></p><h3>Core Equity Datasets</h3><p>In this section we discuss about how we have sourced datasets for this project for the US cash equities market. Since there is so much information to put in the article, please take a look at the wiki <a href="https://github.com/saeed349/quant_infra/wiki/Core-Equity-Datasets">here.</a> The following points are addressed in detail there.</p><ul><li>Core datasets that are required.</li><li>The vendor we are using.</li><li>What identifier to choose for connecting your datasets.</li><li>Common problem faced with examples on share classes, dual-listing, multiple listing, ticker-change, corporate actions etc..</li><li>The importance of Point in Time (PIT).</li><li>Creating a trading universe.</li><li>Why is it better to not source and build everything and use as much as possible from Quantconnect.</li><li>Some other important equity datasets.</li><li>Alternative data and the importance of security master.</li><li>Some other vendors that provide Equity and Futures dataset at an affordable price.</li><li>Most popular institutional grade vendors.</li></ul><h3>Date Warehouse and Data Store</h3><p>The decision between Snowflake and Databricks for your data warehouse is not a straightforward one. While both platforms may seem similar, they differ significantly, making an apples-to-apples comparison challenging. I am personally immersed in the Databricks platform since the inception of my big data journey, it holds a special place for me. I extensively utilize it for work, appreciating its role as not just a data warehouse but also an analytics platform with diverse use cases and applications. But in this project I have gone with Snowflake for a few specific reasons. Ideally its best to have a combination of both these platforms your operation. I go deeper into the comparison in the <a href="https://github.com/saeed349/quant_infra/wiki/Data-Warehouse-and-Data-Store#snowflake-vs-databricks">wiki</a>.</p><p>For this project, I use DuckDB for persisting pandas dataframe during research and you can read more about that <a href="https://github.com/saeed349/quant_infra/wiki/Data-Warehouse-and-Data-Store#datastore">here</a>.</p><h3>Data Ingestion Pipeline</h3><p>Every aspect of Extract, Transform, and Load (ETL) operations unfolds seamlessly within Airflow. Below, I elucidate a straightforward flow illustrating the process of retrieving price updates from a vendor API and subsequently updating a table. While the example is demonstrated on Snowflake, it’s noteworthy that the workflow remains comparably consistent in Databricks as well. You can read details about this design and further <a href="https://github.com/saeed349/quant_infra/wiki/Data-Ingestion-Pipeline">here</a>.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/783/0*V130QBxWDZBWPs50" /><figcaption>Data Ingestion Historical</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/884/0*bYXXrWq4udqlHg8L" /><figcaption>Data Ingestion Updates</figcaption></figure><p><strong>Airflow</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/916/0*RPacxAr67t0W5BDz" /></figure><p>Airflow plays a pivotal role in orchestrating the entire process, and over the years, I have gained extensive experience working with various iterations of Airflow, including those offered by major cloud platforms. Presently, for my personal needs, I deploy Airflow on Docker on an EC2 machine utilizing the <a href="https://airflow.apache.org/docs/apache-airflow/stable/core-concepts/executor/local.html">Local Executor</a>. I anticipate skepticism surrounding the use of LocalExecutor and the feasibility of handling substantial workloads. In the past, I employed Celery executors, necessitating more potent machines for comparable tasks. However, during my previous employment, I encountered a sophisticated workflow by a Quant who efficiently executed intricate workloads on a separate machine via <a href="https://airflow.apache.org/docs/apache-airflow-providers-ssh/stable/_api/airflow/providers/ssh/index.html">ssh hooks</a>, a strategy that resonated well with my objectives. More about Airflow and scaling options discussed <a href="https://github.com/saeed349/quant_infra/wiki/Airflow">here</a>.</p><p><strong>Machine Learning</strong></p><p>I have experimented with machine learning across various platforms, and the choice largely hinges on specific requirements. Factors such as the need for AutoML or a feature store come into play. Personally, my primary criterion is access to cost-effective powerful GPUs. The decision then revolves around whether to opt for a managed ML platform like AWS Sagemaker or GCP Vertex AI.</p><p>A simple comparison of prices for a 16GB T4 NVIDIA GPU machine (ml.g4dn.xlarge) on Sagemaker ($0.7364/hour) versus EC2 ($0.526/hour) underscored that a managed service was unnecessary for my needs. Sagemaker boasts fancy features like automl, data wrangling and others, while on-demand EC2 instances are swift to boot up, resulting in an overall good development experience. The only remaining consideration was package management on the bare metal machine, for which <a href="https://lambdalabs.com/lambda-stack-deep-learning-software">Lambdastack</a> + Anaconda emerged as the ideal solution. This combination can be installed on any Linux machine, automatically managing all required machine learning and deep learning packages. Lambda stack has also addressed the challenge of accessing GPUs inside a docker container, making it my exclusive software stack for machine learning these days.</p><p>Additionally, Lambda Labs provides more cost-effective GPU machines compared to AWS, making it my preferred choice unless I have free credits on AWS. While EC2 machines are occasionally used for inferences, their utilization is limited to when heavy GPU usage is not required. A 24GB A10 NVIDIA GPU instance (g5.8xlarge) costs $2.448/hour on AWS, whereas a similar machine on Lambda Labs is priced at only $0.75/hour. Lambda Labs stands out as one of the few platforms offering H100s and H200s, showcasing their commitment to keeping up with the latest technologies.</p><p>In the realm of experiment tracking, <a href="https://mlflow.org/">Mlflow</a> excels in basic tracking, complemented by several MLOps features. However, <a href="https://wandb.ai/site">Weights and Biases</a> caters specifically to deep learning experiments, offering advanced research features. W&amp;B elevates experiment tracking and note-taking, providing insightful data on CPU and GPU usage. This feature is particularly valuable in reinforcement learning, where understanding resource utilization during simulation is crucial. W&amp;B seamlessly integrates with frameworks like stable-baselines3.</p><p>Read more <a href="https://github.com/saeed349/quant_infra/wiki/Machine-Learning">here</a></p><p><strong>Security</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/889/0*OklPAfCYAJchnEBd" /></figure><p>A poor man’s shot at protecting his resources</p><p>While acknowledging that network security is a broad topic and not my area of expertise and for organizations, its critical to set up a Virtual Private Cloud (VPC) on AWS accounts, implementing identity and access management solutions, Single Sign-On (SSO), deploying a robust firewall, and defining both outbound and inbound rules for the VPC. In the context of my personal deployment, I decided to not go with the aforementioned infrastructure to save cost and to reduce complexity but I have implemented several cheaper measures in this project based on my experience and the best practices observed in other organizations to ensure the projects integrity. Please take a look <a href="https://github.com/saeed349/quant_infra/wiki/Security">here</a> to read more about the policies I have implemented.</p><h3>Project Cost</h3><p>I have tried my best to keep the cost down to a minimal as my personal project was completely bootstrapped and it was quite interesting to find out that how much you can achieve with little. Before I used to run behind managed and serverless technologies, but as I gain more experience I am learning that self-managed technologies can take you a long way.</p><p>If you are an individual or a small team, there are lot of opportunities out there to run things at a very low cost. That might involve things like applying for free cloud credits, using free and lower tier services. So anyone who’s starting out this journey, feel free to reach out to me if you feel intimidated by the cost of some of these services, I can share a few tips that could save you a few bucks.</p><p>You can read the detailed cost breakdown <a href="https://github.com/saeed349/quant_infra/wiki/Cost">here</a>.</p><p>Also, <a href="https://github.com/saeed349/quant_infra/wiki/Project-Management">here</a> are few tips on project management that helped me work on complex projects in general.</p><h3>Credits</h3><p>Throughout my career, I’ve had the privilege of collaborating with exceptionally talented individuals in the quantitative space. The current iteration of my approach draws inspiration from these collective experiences.</p><p>It’s important to note that I’m not asserting this as the definitive approach, as there is no singular right way to tackle these challenges. I’ve witnessed portfolio managers and traders achieving success using tools as simple as Excel to performing complex calculations and analytics directly within transactional databases using SQL. The key takeaway that I learned was that success lies not in adhering to a specific method but in finding the approach that fits the unique nature of your work and your style.</p><p><a href="https://github.com/saeed349/quant_infra/wiki">Link</a> to the Wiki.</p><p>If there is interest, I would be happy to share more on the implementation details and maybe even share a complete walkthrough of the implementation. Do let me know in the comments.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=e817ba967db0" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Poor man’s Serverless Quant Fund Infrastructure]]></title>
            <link>https://usquant349.medium.com/poor-mans-serverless-quant-fund-infrastructure-fd3f1f6b0f70?source=rss-6e65ec0dcdd5------2</link>
            <guid isPermaLink="false">https://medium.com/p/fd3f1f6b0f70</guid>
            <category><![CDATA[google-cloud-platform]]></category>
            <category><![CDATA[algorithmic-trading]]></category>
            <category><![CDATA[big-data]]></category>
            <category><![CDATA[databricks]]></category>
            <category><![CDATA[data-science]]></category>
            <dc:creator><![CDATA[Saeed Rahman]]></dc:creator>
            <pubDate>Sun, 09 May 2021 18:36:20 GMT</pubDate>
            <atom:updated>2021-05-09T18:36:20.455Z</atom:updated>
            <content:encoded><![CDATA[<p>With services like Quantopian shutting its door to the public, I thought it would be useful to share a serverless architecture that I build to do quantitative trading and research. Early last year, I open-sourced a quantitative trading docker based architecture called Microservices Based Algorithmic Trading System (MBATS) (<a href="https://www.linkedin.com/pulse/microservices-based-algorithmic-trading-system-saeed-rahman/">LinkedIn article</a> and <a href="https://github.com/saeed349/Microservices-Based-Algorithmic-Trading-System">GitHub Repo</a>), with MBATS you could run analytical workloads locally using docker but this article is all about scaling MBATS via different Google cloud services provisioned and managed using an open source framework called <a href="https://www.terraform.io/">Terraform</a>.</p><p>Wondering why this title ? well I have designed the architecture in the leanest possible way using cheap/free serverless solutions wherever possible and at the end of the article I will also share some tips to get free cloud credits. Again, this project is not meant for that well funded firm but more for the bootstrapped individuals who wants to get started as quick as possible with a lean scalable analytical platform at the lowest price possible. Also, this project has been structured in a way prioritizing simplicity and maintainability. Therefore all the cloud services chosen are as close to the ones used in MBATS so that its possible to run a hybrid ecosystem (local and public cloud).</p><p>If you are someone like me one who’s impatient and want to see the code, <a href="https://github.com/saeed349/Quant-Trading-Cloud-Infrastructure">here’s</a> Github Repo for the project.</p><h3>Time to get technical…..</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*7kgmnWxSK8t3KXuV5eN1aQ.jpeg" /></figure><p>To begin, I have made quite a few changes to <a href="https://github.com/saeed349/Microservices-Based-Algorithmic-Trading-System">MBATS</a> and released <a href="https://github.com/saeed349/Microservices-Based-Algorithmic-Trading-System-V-2.0">MBATS V2.0</a> where I replaced superset with a multi-page <a href="https://plotly.com/dash/">Dash</a> App and attached <a href="https://airflow.apache.org/docs/apache-airflow/stable/executor/celery.html">Celery executor</a> for Airflow. And finally wrapped all the containers in a <a href="https://docs.nginx.com/nginx/admin-guide/web-server/reverse-proxy/">Nginx reverse proxy</a> server for single access point.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*fFw3dTTXv-9YJlC82BOZAw.png" /></figure><p>In the below image you can see how the different services in MBATS V2.0 has been replaced with the different cloud services to achieve the same functionality but at a scale.</p><h3>Architecture</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*HTuF4p7EDezhakoQnY5O4g.png" /></figure><p>The Terraform code not only creates the different cloud services but also manages loading the application logic into these services via <a href="http://cloud.google.com/build">Cloud build</a>. This way, we can develop the code locally in the <a href="https://github.com/saeed349/Microservices-Based-Algorithmic-%20Trading-System-V-2.0">MBATS V2.0</a> environment and then push the changes to a branch in an online-repository(GitHub for example) and then the Cloud build will automatically update the cloud services.</p><p>I won’t go into the detail of implementation in this article, but will give you a brief idea of how the different cloud services were used.</p><ul><li><a href="https://cloud.google.com/run">Cloud Run</a> is used to deploy containerized applications on a fully managed serverless platform. There are multiple ways we can use this service in this project, and we are mainly using it for processing low timeout data processing jobs that requires multiple technologies working together and that could be containerized. Also, the Dash app is hosted in Cloud Run, and its made public via a domain, the DNS routing and the firewall rules are all setup through <a href="https://www.cloudflare.com/">CloudFlare </a>via Terraform.</li><li><a href="https://cloud.google.com/sql">Cloud SQL</a> is a fully managed PostgreSQL Server. All the security-master, position analytics and risk-analytics data can be stored here. Since its Postgres, making the switch from MBATS to Cloud SQL is seamless.</li><li><a href="https://cloud.google.com/storage">Cloud Storage</a> is a scalable object storage. Minio is replaced with Cloud Storage. And since we are using <a href="https://boto3.amazonaws.com/v1/documentation/api/latest/index.html">Boto3</a> in all the codebase in MBATS, making the switch from local to cloud is also a piece of cake.</li><li><a href="https://cloud.google.com/functions">Cloud Function</a> is a scalable pay-as-you-go functions as a service (FaaS). Cloud Functions can be utilized multiple ways in this project. The primary way we have used is to fetch the data from the data-vendors/brokers rest API and store it to the Cloud SQL. Flask is the underlying framework used to handle incoming HTTP requests. So you could trigger these either by <a href="https://cloud.google.com/scheduler">Cloud Scheduler</a> or via Airflow.</li><li><a href="https://cloud.google.com/compute">Compute Engine</a> is used to create and run virtual machines. Apache Airflow, Celery workers and Flower are hosted in docker compose in the Compute Engine. The end-points are controlled by a reverse-proxy (Nginx) container and they are exposed to a domain via <a href="https://www.cloudflare.com/">CloudFlare</a>. The live strategy needs to be run on a machine that doesn’t have a timeout and that’s the reason we haven’t used Cloud Run (max timeout:15 minutes) or Cloud Function (max timeout:9 minutes) for this particular case.</li></ul><h3>Terraform</h3><p>The best part of docker is that it is infrastructure as code which gives us immense flexibility when developing and deploying. One of the challenges with the cloud architecture is that it involves multiple services, multiple chained and dependent configurations and if you have multiple cloud in a hybrid or multi cloud architecture, then we have whole different level of complexity. Using Terraform we could set these complex rules and architecture as code. Its is a slight upfront investment, but having the architecture as code opens up a lot of opportunity, especially the flexibility to screw up big time but still be able to fix it in one command. For example, for this project I was able to setup all the above services in like 5–10 minutes with just one line of terraform command.</p><h3>Cost and Simplicity</h3><p>Now if you notice all the different services, the only service that’s not truly serverless is the Compute Engine. The reason why this is the case is because the hosted Airflow services in Google Cloud is <a href="https://cloud.google.com/composer">Cloud Composer</a>, this uses Kubernetes as backend and with my current workload and use-case I felt that it was an overkill. On-top of that sometimes serverless technologies can be cost prohibitive as well, so its combination of simplicity and cost that made me to go with <a href="https://cloud.google.com/compute">Compute Engine</a>. To that point there are several other ETL tools available in the Google Cloud ecosystem that could be utilized here for the data ingestion and processing, but the point of this project was not to be fancy but to be practical and I just followed the universal KISS (keep it simple stupid) principal.</p><p>Most Cloud providers have a free tier system and with Google Clouds you can get a 300$ credit when you signup as a new user. With that you could easily run the above set of services for a few weeks. With Terraform, since you are able to setup the infrastructure and destroy it in a few minutes you can be a bit creative and save some money if you wish.</p><p>There are a lot of free resources out there to get free cloud credits from all the big cloud providers as well, <a href="https://blog.ycombinator.com/startup-school-online-registration-and-deals/">here’s </a>one that has worked for me.</p><h3>But what about Big Data ????</h3><p>I can hear a lot of you saying that we won’t be able to process big data in the above architecture. . Market data especially price, reference and fundaments are datasets with very well defined schema and with the above infrastructure I was able to process intraday (minute level) data for around ~5000 active equities across NASDAQ, NYSE with ease. But still I agree that having a power house like spark cluster is always handy. But it really depends on what you would want to do and how complex you want to get.</p><p>At work, I deal with alternative data with millions and billions of records everyday and I use the unified analytical platform <a href="https://databricks.com/">Databricks </a>in AWS environment to do all of that and for that reason I am quite biased towards it. Spark from my perspective is not just an ETL framework but also a good tool for performing analytical work and implementing ML models as well. I won’t go into the details of how to implement the same in Databricks in this article as it can be an project of in itself. But you can expect a future article that would be focused on using Databricks and <a href="https://docs.databricks.com/delta/index.html">Delta Lake</a> to implement a quant-trading infrastructure that would be able to process tick level data using <a href="https://docs.databricks.com/spark/latest/structured-streaming/index.html">Structured Streaming</a> and <a href="https://kafka.apache.org/">Kafka</a>.</p><h3>Value your time</h3><p>The GitHub codes and the multiple repositories I have mentioned here might look a bit intimidating to any who’s beginning in the space. I am a Data Scientist and my expertise are around models and analytics rather than cloud configuration technologies like Terraform. The best strategy in these situations where you face a technology that you are not familiar with but you need to use it for getting things done is to seek help.</p><p>Using portals like <a href="https://www.freelancer.com/">Freelancer</a>, <a href="http://upwork.com/">Upwork </a>and many others you can easily find thousands of experts who would be able to help you out. This is what I did with Terraform, I learned how to implement one or two simple cloud service in Terraform, learned about what it can do and figured out how it would fit into what I am trying to achieve. Then I went out and got help from an amazing freelancer cloud architect, from Brazil, and got all basic code for all the different cloud services from him. Then it was only a matter of putting them together in a way I wanted. What had taken him 5 hours would probably have taken me at least 50–100 hours if I decided to do everything by myself.</p><p>Being an engineer at heart its very tempting to do everything by yourself especially when you are playing around with a new technology and you are excited. In the early phases of my career this has served me well, but soon you will realize there’s only so much you can do in 24 hours. So optimize for goals and deadline and I always stick to the below philosophy.</p><blockquote>Never Mistake the Finger for the Moon The pointing finger is what guides you to the moon. Without the finger, you might not notice the moon. But the pointing finger isn’t what matters most. It only matters because it helps you see the moon for yourself.</blockquote><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=fd3f1f6b0f70" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Microservices Based Algorithmic Trading System]]></title>
            <link>https://usquant349.medium.com/microservices-based-algorithmic-trading-system-83a534c3f132?source=rss-6e65ec0dcdd5------2</link>
            <guid isPermaLink="false">https://medium.com/p/83a534c3f132</guid>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[python]]></category>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[docker]]></category>
            <category><![CDATA[algorithmic-trading]]></category>
            <dc:creator><![CDATA[Saeed Rahman]]></dc:creator>
            <pubDate>Sat, 28 Mar 2020 14:58:29 GMT</pubDate>
            <atom:updated>2020-03-28T14:58:29.728Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*q4Y5VfooTbepLuStgoJb8g.png" /></figure><p>Say that you have an algorithmic trading strategy idea. Whats next ? you need to code it, test it and deploy it in the market. Lets start with downloading market data as csv files and coding a rough version of the strategy to test for its effectiveness, well if the results are promising then you would need to write more code to paper trade it and eventually go live. Maybe later you decide to add Machine Learning to the mix. Before you know this would be pieces of code sticking together leading to a monolithic disaster.</p><p>Instead we can take an easier route of programming your strategy in a platform like Deltix, Algotrader, Metatrader etc… Or even better we can go the SaaS route and opt for services like Quantoconnect, Quantopian or Qunatiacs where you can start coding your strategy without being concerned about data, infrastructure or broker connections. It’s a great way to learn and start your journey, especially the community driven aspects of the SaaS platforms. Either way, most systems come at a cost and you would have to make a few sacrifices like coding up in a proprietary language or not be able to customize it enough to meet your needs and most important of all if the strategy have impressive performance would you want the strategy to be in a server that’s not yours.</p><p>Even though I have worked on multiple predictive modelling projects involving reinforcement learning <a href="https://www.linkedin.com/pulse/deep-reinforcement-learning-trading-saeed-rahman">(article)</a> and alternative data <a href="https://www.linkedin.com/pulse/seeking-alpha-alternative-data-saeed-rahman">(article)</a>, one thing that always troubled me was actually turning those ideas into trading strategies that I can actually backtest, paper trade and eventually go live . I have been playing around with open source backtesters like <a href="https://github.com/quantopian/zipline">Zipline</a>, <a href="https://github.com/mhallsmoore/qstrader">Qstrader</a> and <a href="https://github.com/mementum/backtrader">Backtrader</a> for a while, but they were still not self-sufficient to trade multiple strategies or even to use Machine Learning and lacked a lot of supporting features. But things started falling into place when I stumbled upon this amazing containerization technology called Docker. All of a sudden it opened up an array of opportunities for packaging, reproducing and productionizing software and data-science projects.</p><h3>MBATS</h3><p>From scratch to a functional trading infrastructure under 5 minutes and you can control all aspects of strategy development and production lifecycle using Python; that’s a one line definition of MBATS.</p><p>MBATS (Microservices Based Algorithmic Trading System) is a Docker based trading infrastructure designed with scale and interoperability in mind and one in which Machine Learning is a first class citizen. The project is opensource and all the components used in this project are opensource technologies and they have been chosen in such a way that there is a analogues managed service provided by top Cloud vendors, so that the migration to Cloud can be seamless.</p><h3>Quick start</h3><p>MBATS is based on Docker containers. Running your Infrastructure is as easy as running one command from your terminal. You can either run MBATS on your local machine or on the cloud using docker-compose.</p><p>Setting up the entire infrastructure can be done in 2 lines of script</p><ol><li>Clone the github page. <a href="https://github.com/saeed349/Microservices-Based-Algorithmic-Trading-System">https://github.com/saeed349/Microservices-Based-Algorithmic-Trading-System</a></li><li>Run “docker-compose up -d — build”</li><li>Run “starter_script.bat”</li></ol><iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Fwww.youtube.com%2Fembed%2FhLSGgW4-WC8%3Ffeature%3Doembed&amp;display_name=YouTube&amp;url=https%3A%2F%2Fwww.youtube.com%2Fwatch%3Fv%3DhLSGgW4-WC8&amp;image=https%3A%2F%2Fi.ytimg.com%2Fvi%2FhLSGgW4-WC8%2Fhqdefault.jpg&amp;key=a19fcc184b9711e1b4764040d3dc5c07&amp;type=text%2Fhtml&amp;schema=youtube" width="854" height="480" frameborder="0" scrolling="no"><a href="https://medium.com/media/8d97bc272d900977ad580d5f36a0e03f/href">https://medium.com/media/8d97bc272d900977ad580d5f36a0e03f/href</a></iframe><h3>Architecture</h3><p>Using MBATS, you can easily create Trading Strategies in Backtrader, manage Machine Learning models with MLflow, use Postgres database with pgAdmin for storing and querying Market data. Store files and objects in Minio and use Superset to visualize performance of backtested and live strategies. And tie all the different components together and orchestrate jobs using Airflow and many more features to help you get started from idea to live trading faster and with least effort.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*NN54Mr1ScMAALfuTU_EE0A.png" /></figure><h3><a href="https://www.backtrader.com/">Backtrader</a></h3><p>Backtrader is a python based opensource event-driven trading strategy backtester with support for live trading. The reason why I choose Backtrader over other opensource backtesters like <a href="https://github.com/quantopian/zipline">Zipline</a> and <a href="https://github.com/QuantConnect/Lean">QuantConnect</a> is because of the amazing documentation and the community driven approach. For this project, I had to build quite a few custom Backtrader modules like <a href="https://www.backtrader.com/docu/datafeed-develop-general/datafeed-develop-general/">Datafeeds</a> and <a href="https://www.backtrader.com/docu/analyzers/analyzers/">Analyzers</a>, but due to the clean design and extensibility of Backtrader combined with the community support made most of it quite easy.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/763/1*Y8OVeb0Fwow14SAWCJgw0Q.png" /></figure><h3><a href="https://mlflow.org/">MLflow</a></h3><p>Anyone who has worked in the Datascience field would have heard about <a href="https://spark.apache.org/">Spark</a>, well the founders of Spark have brought a similar disruptive tool to revolutionize the Machine Learning landscape and that is MLflow. Mlflow is an open source platform to manage the ML lifecycle, including experimentation, reproducibility and deployment. It currently offers four components:</p><ul><li>MLflow Tracking</li><li>MLflow Projects</li><li>MLflow Models</li><li>MLflow Registry</li></ul><p>There are a few other organizations that try to address this problem but what separates MLflow from the likes of <a href="https://www.tensorflow.org/tfx">Google-TFX</a>, <a href="https://engineering.fb.com/core-data/introducing-fblearner-flow-facebook-s-ai-backbone/">Facebook-FBLearner Flow</a> and <a href="https://eng.uber.com/michelangelo/">Uber-Michelangelo</a> is that MLFlow try to address the concerns of the crowd rather than a single organization and therefore it is universal and community driven.</p><p>In this project all the Machine Learning models can be tracked, packaged and served by MLflow and the model artifacts are stored in Minio. The ML models are served using MLflow pyfunc, but we also have the option to serve the model as Rest API(code in sample jupyter notebook)</p><p>MLflow is platform, package and language agnostic whereas if you look at TFX which only supports Tensorflow. Moreover the rate at which the community is growing, it is highly likely that it will be the standard tool for Machine Learning lifecycle management in the next few months. If we decide to scale and make the switch to the Cloud for Machine Learning it would be super easy as both <a href="https://www.mlflow.org/docs/latest/python_api/mlflow.sagemaker.html">AWS Sagemaker</a> and <a href="https://docs.microsoft.com/en-us/azure/machine-learning/how-to-use-mlflow">Azure Machine Learning</a> has integration for MLflow. Also MLFlow and Spark has been developed by the Databricks team and therefore its native in their <a href="https://databricks.com/product/managed-mlflow">environment </a>. So in the future if we want to train or serve the Machine Learning models of MBATS in the cloud (Databricks, Azure or AWS) it will be just few lines of code to make the switch.</p><h3><a href="https://airflow.apache.org/">Apache Airflow</a></h3><p>Apache Airflow is an open-source workflow management platform, basically <a href="https://en.wikipedia.org/wiki/Cron">cron</a> on steroids and it has wide array of integration with popular platforms and data stores. In this this project we use airflow for scheduling two tasks. One Task/DAG for downloading daily and minute data into the database and another dynamic DAG for scheduling live strategies.</p><h3><a href="https://superset.incubator.apache.org/">Apache Superset</a></h3><p>From the creators of Apache Airflow, Apache Superset is a Data Visualization tool designed and used within Airbnb and later opensourced. Superset offers interactive Data Exploration that will let you slice, dice and visualize data. Why pay for Tableau and Power-BI when you can use something that is opensource and designed for scale. It might not have all the bells and whistles like <a href="https://looker.com/">Looker</a> but it has enough features for ensuring an amazing user experience. In this project we use Superset to visualize Backtesting and Live trading performance.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*KYhw2pQGJMpgR6zyM3glYg.png" /></figure><h3><a href="https://min.io/">Minio</a></h3><p>MinIO is pioneering high performance object storage. With READ/WRITE speeds of 55 GB/s and 35 GB/s on standard hardware, object storage can operate as the primary storage tier for a diverse set of workloads. Amazon’s S3 API is the defacto standard in the object storage world and represents the most modern storage API in the market. MinIO adopted S3 compatibiity early on and was the first to extend it to support S3 Select. Because of this S3 Compatibility by using Minio we have an upper hand of moving this object store towards cloud (AWS S3, Google Cloud Storage, Azure Storage) with minimal change to the codebase.</p><h3><a href="https://www.postgresql.org/">PostgreSQL</a></h3><p>We have 2 Databases in our PostgreSQL</p><ol><li>Security Master database that stores the Daily and Minute data for Forex Symbols</li><li>Risk Database to store the position information and the performance metrics.</li></ol><p>The Databases can be managed through PgAdmin</p><h3>Current Features</h3><ul><li>Infrastructure as Code — less than 5 minutes from scratch to a fully functional trading infrastructure.</li><li>Backtesting and Live trading Forex using Oanda Broker API (Can be easily be modified to accommodate Interactive Brokers for Equity).</li><li>Machine Learning model development and deployment using MLflow.</li><li>Multiple symbol strategy support.</li><li>Multiple strategy support.</li><li>Market data download and Live trading scheduling using Airflow.</li><li>Superset Dashboard for real-time monitoring of Live trading and Backtest performance results.</li><li>Easily extensible to support any kind of structured data.</li><li>Full code base in Python except for docker-compose setup.</li></ul><h3>Scaling to the Cloud</h3><p>Every technology used in this project has a analogues managed service offered in the cloud. And the best part of scaling a microservices based architecture is that you can approach it in many ways to fits your need. Whether you want to move one functionality to the cloud or if you want to offload the workload of a component to the Cloud but keep all the critical parts on premise, the migration is quite easy when compared to a monolithic architecture. Moreover if the Cloud service is using the same technology then the migration is effortless. A simple example for this is <a href="https://cloud.google.com/composer/">GCP Cloud Composer</a> which is built on top of Apache Airflow and Kubernetes which means that all the tasks/DAG’s that we are using in this project can be used in Cloud Composer as well. In general I have found GCP has a better strategy and technology in place for building a <a href="https://cloud.google.com/solutions/manage-hybrid-cloud/">hybrid Cloud</a> infrastructure and for that reason here’s an architecture if this project has to be transferred entirely into the GCP platform.</p><p>Let me know if you are interested in seeing the below architecture take form using <a href="https://www.terraform.io/">Terraform</a></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*k2jYjfQzQg6hXq2gb6CzHw.png" /></figure><p>MBATS is still in its early stages and need more functionalities and stress testing before it can be used by an a actual trading desk. Instead its meant for educational purpose and it was my curiosity for Machine Learning and Quantitative Trading that drove me to explore this idea and opensource this project so that it could be used by aspiring Quants.</p><h3>Acknowledgments</h3><p><a href="https://community.backtrader.com/">Backtrader community</a></p><p><a href="https://backtest-rookies.com/category/backtrader/">Backtest-rookies</a></p><p><a href="https://www.quantstart.com/">Quantstart</a></p><p><a href="https://www.udemy.com/course/beyond-jupyter-notebooks/">Beyond Jupyter notebook — Udemy course</a></p><p>MBATS is still in its early stages and need more functionalities and stress testing before it can be used by an a actual trading desk. Instead its meant for educational purpose and it was my curiosity for Machine Learning and Quantitative Trading that drove me to explore this idea and opensource this project so that it could be used by aspiring Quants.</p><h3>Links</h3><p><a href="https://github.com/saeed349/Microservices-Based-Algorithmic-Trading-System">saeed349/Microservices-Based-Algorithmic-Trading-System</a></p><p><a href="https://community.backtrader.com/">L</a>inkedin: <a href="https://www.linkedin.com/in/saeed-349/">https://www.linkedin.com/in/saeed-349</a>/</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=83a534c3f132" width="1" height="1" alt="">]]></content:encoded>
        </item>
    </channel>
</rss>