<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Peter Kinder on Medium]]></title>
        <description><![CDATA[Stories by Peter Kinder on Medium]]></description>
        <link>https://medium.com/@petermkinder?source=rss-8f1c869d7f35------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/1*2uHpJCQI6RKLfqWazt9RZQ.jpeg</url>
            <title>Stories by Peter Kinder on Medium</title>
            <link>https://medium.com/@petermkinder?source=rss-8f1c869d7f35------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Fri, 20 Nov 2026 10:51:37 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@petermkinder/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[Spectrograms & Vision Transformers for Time Series Forecasting (Part 2)]]></title>
            <link>https://medium.com/correll-lab/spectrograms-vision-transformers-for-time-series-forecasting-part-2-828733259da0?source=rss-8f1c869d7f35------2</link>
            <guid isPermaLink="false">https://medium.com/p/828733259da0</guid>
            <category><![CDATA[financial-markets]]></category>
            <category><![CDATA[time-series-analysis]]></category>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[transformers]]></category>
            <category><![CDATA[stock-market]]></category>
            <dc:creator><![CDATA[Peter Kinder]]></dc:creator>
            <pubDate>Fri, 20 Dec 2024 08:28:29 GMT</pubDate>
            <atom:updated>2024-12-20T08:28:29.632Z</atom:updated>
            <content:encoded><![CDATA[<h3>Introduction</h3><p>On December 18, 2024, the day before this article was written, a staggering 10,314,175,527 shares, valued at $448,633,834,671, were traded on the NASDAQ. This eye-popping figure — representing a single asset class, over one day, on one exchange, in one country — clearly demonstrates the immense potential financial benefit of accurately forecasting future equity price movements. This article, Part 2 of a series, builds on prior research from J.P. Morgan Research with the ultimate goal of achieving such forecasting capability.</p><p>In the original research paper, titled <a href="https://arxiv.org/pdf/2403.11047"><em>From Pixels to Predictions: Spectrogram and Vision Transformer for Better Time Series Forecasting</em></a>, the use of a vision transformer (ViT) applied to a spectrogram generated from a time series of an equity’s adjusted closing prices to forecast future price movements was explored. While the research showed promising results, it also appeared that there was an opportunity to possibly enhance the approach by incorporating concatenated spectrograms generated from multiple time series as inputs to the ViT rather than relying on just one.</p><p>Part 1 of this series provided a detailed review of the J.P. Morgan Research paper, explaining what spectrograms are and the rationale behind this innovative approach. For a deeper understanding of the foundational concepts underpinning the modeling discussed in this article, please see the link to Part 1 below. However, this installment focuses on the experiments conducted and their resulting outcomes.</p><p><a href="https://medium.com/correll-lab/spectrograms-vision-transformers-for-time-series-forecasting-929163fd8c52">Spectrograms &amp; Vision Transformers for Time Series Forecasting</a></p><h3>Methodology</h3><h4>Data</h4><p>To evaluate whether using multiple time series as inputs for a vision transformer (ViT) could enhance equity price forecasting capabilities, the analysis focused on equities comprising the S&amp;P 500 Index. Two datasets were created for comparison: the first included only spectrograms generated from an equity’s adjusted closing price, while the second expanded on this by including additional spectrograms generated from an equity’s trading volume, the VIX’s adjusted closing price, and the S&amp;P 500 Index’s adjusted closing price.</p><p>Trading volume was included in the second dataset because it is commonly used to assess the strength of a trend in an equity’s price movement. The VIX and S&amp;P 500 Index’s adjusted closing prices were included to provide the model with broader market context, offering insights into overall market sentiment and performance trends alongside the equity’s specific trading volume and adjusted closing price movements.</p><ul><li><a href="https://www.investopedia.com/ask/answers/041015/why-trading-volume-important-investors.asp">Trading Volume: Analysis and Interpretation</a></li><li><a href="https://www.investopedia.com/terms/v/vix.asp">CBOE Volatility Index (VIX): What Does It Measure in Investing?</a></li></ul><p>The time series data, spanning from 2001 to 2023, was sourced from Yahoo Finance. Additionally, since the composition of the S&amp;P 500 Index changes over time, further steps were taken to replicate it throughout this period. A historical list of S&amp;P 500 Index components was obtained from the following GitHub repository:</p><p><a href="https://github.com/fja05680/sp500">GitHub - fja05680/sp500: Current and Historical Lists of S&amp;P 500 components since 1996</a></p><p>It should be noted that some data quality issues arose because an equity’s ticker symbol is not permanent, and Yahoo Finance indexes data by the current ticker. For instance, Meta, previously known as Facebook, is now traded under the ticker META, whereas it was traded under the ticker FB during its time as Facebook. Additionally, discrepancies existed between how tickers were labeled in the GitHub repository and Yahoo Finance. Despite these inconsistencies, and without manually addressing every discrepancy, the vast majority of the S&amp;P 500 Index components were still well-represented in the datasets used.</p><h4>Data Preparation</h4><p>Each time series used to generate the spectrogram inputs spanned 80 consecutive trading days and was normalized to a range between 0 and 1. After normalization, the Python library PyWavelets was employed to perform the wavelet transformation. Specifically, the complex Morlet wavelet was used to conduct the transformation at various scales, consistent with the approach in the original J.P. Morgan Research paper. However, adjustments were required to replicate the same wavelet transformation using PyWavelets. The PyWavelets implementation used for the wavelet transformation included two parameters and was as follows:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/891/1*dZMhwqiOVugk29273ls30g.png" /></figure><p>The equation for the complex Morlet wavelet used in the original J.P. Morgan Research paper was the following:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*c5yHl6Zs-gX7a6Yw.png" /></figure><p>For the required <em>B</em> and <em>C</em> values in the PyWavelets equation, <em>C</em> was set to 1 and <em>B</em> was set to 2. Additionally, to match the scalar multiple used in the equation, the normalized time series was multiplied by a scalar value to achieve the same scalar multiple used in the original J.P. Morgan Research paper.</p><p>Lastly, to make the spectrograms readable to the ViT, the images were converted into RGB arrays. For the single time series data, the RGB array was a 3 channel array. For the multiple time series data, individual 3 channel RGB arrays were concatenated into a single 12 channel array.</p><h4>Experiments</h4><p>To assess whether incorporating spectrograms generated from multiple time series improved model performance, both models were built to forecast whether the price of an equity would rise or fall 20 trading days after the last adjusted closing price in the time series. The models were then trained on a random sample of data spanning a decade and tested on data from the following year. The test data included every equity in the S&amp;P 500 Index for each month, with the last observation of the input time series being the final trading day of the month.</p><p>For instance, a model was trained on a random sample of data from 2001 through 2011, with a 20 trading day shift to avoid any data leakage due to the forecast step size. The trained model was then tested on all S&amp;P 500 Index equities for the year 2012, using the last trading day of each preceding month as the final observation in the input time series. For January 2012, the last observation for predicting price movement could be December 31, 2011; for February 2012, it could be January 31, 2012, and so on. In total, the models were tested on 12 years worth of data ranging from 2012 through 2023.</p><p>For training each of the models using a spectrogram generated from a single time series, a total of 100,000 samples were used. The samples were then split 90/10, with 90,000 samples allocated to the training set and 10,000 samples to the validation set.</p><p>For training each of the models using a spectrogram generated from multiple time series, a total of 25,000 samples were used. The samples were then split 90/10, with 22,500 samples allocated to the training set and 2,500 samples to the validation set.</p><p>The amount of data used was based on how much could be fit onto a 24 GB GPU.</p><p>The architecture and hyperparameter tuning for the model was the same for all experiments and aimed to mimic the model used by J.P. Morgan Research. This architecture and hyperparameter tuning was described in Part 1 of this series and can be seen here:</p><p><a href="https://medium.com/correll-lab/spectrograms-vision-transformers-for-time-series-forecasting-929163fd8c52">Spectrograms &amp; Vision Transformers for Time Series Forecasting</a></p><p>In addition to evaluating the models based on their accuracy, sample portfolios were constructed using predictions that an equity would rise after 20 trading days. These portfolios were compared to assess the potential financial benefits of the modeling approach presented in this article. The portfolios were long-only and allocated based on the quartile in which each stock’s prediction for rising in the given month fell. For example, in January, the portfolio was divided into four groups: one portfolio included equities with a predicted probability of rising in the bottom 25% of all predictions, another included those in the 25%-50% range, and so on. Each portfolio was equally weighted, meaning the portfolio’s return was the average return of all equities in that portfolio. Finally, a baseline was established by averaging the returns of all S&amp;P 500 Index equities for that specific month.</p><h3>Results</h3><p>As is common in financial forecasting, the results on the in-sample data initially sparked dreams of untold riches, only to be crushed by the model’s performance on the out-of-sample data. The in-sample results achieved during the cross-validation phase, using a single spectrogram created from the adjusted closing price time series, were consistent with those in Part 1. However, the results on the out-of-sample test set, on average, were noticeably lower than those in Part 1.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/860/1*OIZwDtc5zzQRN5jybnrwgA.png" /><figcaption>Accuracy metrics for model using a single spectrogram created from an 80 consecutive trading day time series of adjusted closing prices and classifying as either up or down 20 trading days into the future.</figcaption></figure><p>The results from the model using concatenated spectrograms generated from multiple time series were even more disappointing. The validation accuracy on the in-sample data was significantly better than what was achieved using just a single spectrogram as input. However, when evaluated on out-of-sample data, the accuracy was generally worse and much more volatile.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/860/1*bHUseklyA9Y88KCsNelCsg.png" /><figcaption>Accuracy metrics for model using a concatenated spectrograms created from four 80 consecutive trading day time series and classifying as either up or down 20 trading days into the future.</figcaption></figure><p>However, when the predictions were split into quartile portfolios, the accuracy metrics began to rekindle some hope. The quartiles were assigned based on the prediction that the equity would rise after 20 trading days for each month, and accuracy was calculated over the entire year for each quartile group, rather than being calculated monthly and averaged. This approach yielded the following results on the out-of-sample test sets for the model using a spectrogram created from only the adjusted closing price time series</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/869/1*VTVLvDmXB7PVgjeHKpGQiA.png" /><figcaption>Accuracy metrics on test set for model using a single spectrogram created from an 80 consecutive trading day time series of adjusted closing prices and classifying as either up or down 20 trading days into the future. Split into quartiles based on a month’s worth of predictions and averaged across the entire year by quartile.</figcaption></figure><p>The quartile accuracy results for the model using concatenated spectrograms generated from multiple time series were also promising, though to a lesser extent:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/866/1*QPL5ckq9cwG290jjRTBpHg.png" /><figcaption>Accuracy metrics on test set for model using a concatenated spectrograms created from four 80 consecutive trading day time series and classifying as either up or down 20 trading days into the future. Split into quartiles based on a month’s worth of predictions and averaged across the entire year by quartile.</figcaption></figure><p>But yet again, reality set in when evaluating the quartile portfolio returns over time. Monthly returns for each quartile were calculated from 2012 through 2023, covering a total of 12 years, resulting in 144 monthly returns for each quartile portfolio. A starting portfolio value was set at 1, and the cumulative growth of each portfolio was calculated and compared against one another, as well as against the equally weighted portfolio of all equities in the S&amp;P 500 Index. This approach yielded the following results for the model using a single spectrogram created from the adjusted closing price time series:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/968/1*wseBxcCVVTwVGStuY1PPvg.png" /><figcaption>Cumulative quartile portfolio returns for model using a single spectrogram created from an 80 consecutive trading day time series of adjusted closing prices and classifying as either up or down 20 trading days into the future and compared against the equally weight S&amp;P 500 portfolio. Time index 0 represents the starting portfolio value of 1, time index 1 represents the portfolio value after January 2012, and time index 144 represents the final portfolio value after December 2023.</figcaption></figure><p>As observed, the quartile 4 portfolio, which had the highest average accuracy across the 12 test sets, yielded the worst cumulative monthly return. The quartile 3 portfolio achieved the best cumulative monthly return, followed by the quartile 1 portfolio, the equally weighted S&amp;P 500 Index portfolio, and finally the quartile 2 portfolio.</p><p>For comparison, the optimal ordering of the portfolios, from high to low in terms of cumulative monthly return, would have been quartile 4, quartile 3, equally weighted S&amp;P 500 Index, quartile 2, and quartile 1.</p><p>Similarly, a somewhat random ordering of the portfolios also occurred for the model using the concatenated spectrograms generated from multiple time series:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/981/1*VQ76lkgBiLUmkkFQiDOszg.png" /><figcaption>Cumulative quartile portfolio returns for model using a single spectrogram created from an 80 consecutive trading day time series of adjusted closing prices and classifying as either up or down 20 trading days into the future and compared against the equally weight S&amp;P 500 portfolio. Time index 0 represents the starting portfolio value of 1, time index 1 represents the portfolio value after January 2012, and time index 144 represents the final portfolio value after December 2023.</figcaption></figure><p>As observed, the quartile 4 portfolio had the best cumulative monthly return, followed by the quartile 2 portfolio, the equally weighted S&amp;P 500 Index portfolio, the quartile 1 portfolio, and finally the quartile 3 portfolio. All in all, this suggests that the forecasts from both models were essentially no better than random selection.</p><h3>Conclusion</h3><p>While the results were mostly disappointing, there could still be potential value in this approach. The reasoning behind this conclusion is that there were a few shortcomings during data preparation and modeling, largely due to time constraints.</p><p>The first shortcoming was data quality. While the vast majority of the S&amp;P 500 was represented in the data, some equities were missing. Dedicating more time to data preparation and possibly investing in higher-quality data could possibly yield marginally better results.</p><p>The second shortcoming, and likely the greatest one, that needs to be addressed before completely discounting this modeling approach, was the amount of training data used. For the model using a single spectrogram generated from only the adjusted closing price time series, there were only 90,000 training samples and 10,000 validation samples. For the model using concatenated spectrograms generated from multiple time series, there were only 22,500 training samples and 2,500 validation samples. These samples were intended to represent 10 years of data for 500 equities, which in total is approximately 1,260,000 time series. Using only around 8% or 2% of the available data is likely insufficient.</p><p>The last shortcoming was using the model to predict the subsequent 12 months of data. An alternative approach could involve predicting just the following month of data, with the training set bootstrapped to more heavily weight the more recent time series. This approach is likely to more accurately capture the market dynamics present in the test set and could lead to improved out-of-sample performance.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=828733259da0" width="1" height="1" alt=""><hr><p><a href="https://medium.com/correll-lab/spectrograms-vision-transformers-for-time-series-forecasting-part-2-828733259da0">Spectrograms &amp; Vision Transformers for Time Series Forecasting (Part 2)</a> was originally published in <a href="https://medium.com/correll-lab">Toward Humanoids</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Spectrograms & Vision Transformers for Time Series Forecasting]]></title>
            <link>https://medium.com/correll-lab/spectrograms-vision-transformers-for-time-series-forecasting-929163fd8c52?source=rss-8f1c869d7f35------2</link>
            <guid isPermaLink="false">https://medium.com/p/929163fd8c52</guid>
            <category><![CDATA[vision-transformer]]></category>
            <category><![CDATA[spectrogram]]></category>
            <category><![CDATA[jp-morgan]]></category>
            <category><![CDATA[sp500]]></category>
            <category><![CDATA[financial-forecasting]]></category>
            <dc:creator><![CDATA[Peter Kinder]]></dc:creator>
            <pubDate>Sun, 10 Nov 2024 12:03:51 GMT</pubDate>
            <atom:updated>2024-11-15T17:40:34.673Z</atom:updated>
            <content:encoded><![CDATA[<h3>Spectrograms &amp; Vision Transformers for (Financial) Time Series Forecasting</h3><h4>A review and implementation of J.P. Morgan AI Research’s paper submission to the 2023 ACM International Conference on AI in Finance (ICAIF)</h4><p>In 2020, a seminal paper titled <a href="https://arxiv.org/pdf/2010.11929"><em>An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale</em></a> (Dosovitskiy et al., 2020) was published that introduced the Vision Transformer (ViT) to the world. The paper presented a groundbreaking approach that applied transformer models, which had previously been largely limited to natural language processing (NLP) tasks, to image classification. Since being published, the paper has received over 47 thousand citations and has profoundly impacted the computer vision, deep learning and artificial intelligence research communities.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*envhjwhspzXH4PqbgVHMBg.png" /><figcaption>Vision Transformer (ViT) architecture presented by Dosovitskiy et al. (2020).</figcaption></figure><p>In addition to the introduction of ViTs, a novel approach to time series forecasting also recently emerged. The approach consists of transforming numeric time series data into images and applying computer vision algorithms. In the domain of financial forecasting, many of the images were in the form of line, candlestick or recurrence plots. For example, see my earlier article on candlestick pattern classification.</p><p><a href="https://medium.com/correll-lab/clip-for-candlestick-pattern-classification-ec16d125565c">CLIP for Candlestick Pattern Classification</a></p><p>However, performing wavelet transformations on numeric data and converting the transformed data into time-frequency spectrograms has shown promising results. For example, a recent paper titled <a href="https://arxiv.org/pdf/2008.06042"><em>Image Processing Tools for Financial Time Series Classification</em></a><em> </em>(Du et al., 2020) performed discrete and continuous wavelet transformations on five signals chosen via the Maximum Information Coefficient (MIC). These transformed signals were then converted into spectrograms, and a convolutional neural network (CNN) was then used to forecast intraday “Up” and “Down” labels.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*lRz4XAV1tL5c0rAEJP00Ug.png" /><figcaption>A flowchart illustrating the use of discrete and continuous wavelet transformations to denoise five signals selected based on the MIC, which are then transformed into spectrograms and used for classifying intraday “Up” and “Down” labels using a CNN (Du et al., 2020).</figcaption></figure><p>Building on these two papers, and many others, J.P. Morgan AI Research submitted a paper titled <a href="https://arxiv.org/pdf/2403.11047"><em>From Pixels to Predictions: Spectrogram and Vision Transformer for Better Time Series Forecasting</em></a><em> </em>(Zeng et al. 2023) to the 2023 Association for Computing Machinery (ACM) International Conference of AI in Finance (ICAIF). The paper presented the following key contributions:</p><ul><li>Demonstrated the superiority of spectrograms as a visual representation of time series data, leading to enhanced predictions across various domains and diverse datasets (Zeng et al., 2023).</li><li>Leveraged transformer’s ability to learn cross-modality information across time and frequency domain to enhance the forecasting process (Zeng et al., 2023).</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*ovuOL1Efu5rnUMfTpO1dzw.png" /><figcaption>An overview of the proposed approach by Zeng et al. (2023).</figcaption></figure><p>This article delves into J.P. Morgan AI Research’s paper, examining each component of the proposed approach and analyzing the study’s findings. It will then introduce a modified implementation of the approach, with preliminary results based on the adjusted close time series of current S&amp;P 500 constituents. Finally, the article will outline the planned future directions for this research.</p><h3>Paper Review</h3><p>There are four key elements readers of <a href="https://arxiv.org/pdf/2403.11047"><em>From Pixels to Predictions: Spectrogram and Vision Transformer for Better Time Series Forecasting</em></a><em> </em>(Zeng et al., 2023) — which will from here on out be referred to as “the paper” — you should understand. The first is the motivation for using visual time series forecasting, i.e. computer vision on images generated for numerical time series data. The second is what wavelet transformations are and the value of performing them. The third is what a spectrogram (or scalogram) is and how to construct it. The fourth is the benefits of using a ViT instead of a CNN for visual time series forecasting.</p><h4>Visual Time Series Forecasting Motivation</h4><p>In the introduction of the paper, the authors stated that “studies [of visual time series forecasting] are motivated by the fact that traders’ decision making is augmented by the visual representation of financial time series images such as charts, graphs, and plots. In addition, this approach can help to capture additional patterns and dependencies that are beneficial for understanding and forecasting time series data” (Zeng et al., 2023).</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*MtdOtyqy9JK088a00DfDhw.png" /><figcaption>Typical workstation of a professional trader (Photoagriculture/ Shutterstock.com).</figcaption></figure><p>The paper then referenced an array of prior studies that utilized visual time series forecasting, but one study that stood out was also authored by J.P. Morgan AI Research and titled <a href="https://arxiv.org/pdf/2011.09052"><em>Visual Time Series Forecasting: An Image-driven Approach</em></a><em> </em>(Sood et al., 2021). It stood because it demonstrated the capture of additional patterns and dependencies by visual time series forecasting that were missed by traditional time series forecasting. To understand this, consider the following results from the study on out-of-sample data:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*tgBrrHPLLhyQyu7Onm1DvA.png" /><figcaption>Results table on out-of-sample data from Sood et al. (2021).</figcaption></figure><p>As can be seen, the VisualAE (autoencoder on a line plot) did not perform well when evaluated by the <a href="https://en.wikipedia.org/wiki/Symmetric_mean_absolute_percentage_error">SMAPE</a> (symmetric mean absolute percentage error) or <a href="https://en.wikipedia.org/wiki/Mean_absolute_scaled_error">MASE</a> (mean absolute scaled error) metrics. However, it outperformed all of the other models when evaluated by the Intersection-over-Union (<a href="https://en.wikipedia.org/wiki/Jaccard_index">IoU</a>) metric, which is commonly used in object-detection literature and was calculated in the study “by obtaining the 1D bounding boxes of non-zero pixels for each column and then calculating IoU of corresponding columns from the ground-truth and predicted image” (Sood et al., 2021).</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/468/0*OlAqY4DqAkC_mjrS.jpeg" /><figcaption>Intersection-over-Union (IoU) calculation (Subramanyam, 2021).</figcaption></figure><p>To practically understand this, observe a comparison of the metrics across various models on an example of harmonic data (i) and financial data (ii):</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*xxAYshMU9rI9s3nyV8NnQg.png" /><figcaption>Metric comparison across various models on an example of harmonic and financial data (Sood et al., 2021).</figcaption></figure><p>Example (ii) of financial data offers the clearest insight into capturing additional patterns and dependencies when comparing the autoencoder applied to numerical data versus a line plot. While the SMAPE and MASE values are lower with numerical data as input (b), the IoU is significantly higher with the line plot as input (d). This higher IoU indicates that the forecast using the line plot as input more accurately reflects both the pattern and the “Down” label of the time series, underscoring the value of visual time series forecasting as a powerful tool in the toolbox for improving time series predictions.</p><h4>Wavelet Transformation</h4><p>The paper also discussed the use of a continuous wavelet transformation and how it is “suitable for non-periodic and transient signals in stock price time series” (Zeng et al., 2023). The paper then settled on the use of the complex morlet wavelet, which it expressed as</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1018/1*kvgJS3z3lP5rPEuriu6RPQ.png" /></figure><p>where <em>s</em> represents the scale of the wavelet. However, it is possible that there was a bit of an oversight when this equation was prepared for the paper. The inspiration for the use of the complex morlet wavelet was cited from Du et al. (2020) where the complex morlet wavelet, without the scale <em>s,</em> was expressed as</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*20sdCts7qQpSLOpNpD9jiA.png" /></figure><p>The main difference to note here is the paper has (ignore the <em>s</em>)</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*RdR3H_LT_HSx5Lu_SY-4zA.png" /></figure><p>and the cited inspiration has</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*GAIuNea6bLSDjrUD_wkisQ.png" /></figure><p>as the “Gaussian envelope centered at zero” (Zeng et al., 2023). Considering how the paper itself stated that this should represent a Gaussian envelope centered at zero, it seems rather conclusive that the paper made an error expressing the complex morlet wavelet that it used for the continuous wavelet transformation.</p><p>With that clarification out of the way, what is the purpose of a continuous wavelet transformation and how is it computed? Well, as anyone who works with financial data knows, there is a lot of noise baked into the signal. What the continuous wavelet transformation is attempting to do is denoise the signal and extract frequencies of the signal at particular time steps. In a sense, the continuous wavelet transformation serves as a method of feature extraction. For example, consider the following musical analogy from “<a href="https://en.wikipedia.org/wiki/Wavelet">Wavelet</a>” (2024):</p><blockquote>A wavelet could be created to have a frequency of middle C and a short duration of roughly one tenth of a second.</blockquote><blockquote>If this wavelet were to be convolved with a signal created from the recording of a melody, then the resulting signal would be useful for determining when the middle C note appeared in the song.</blockquote><blockquote>Mathematically, a wavelet correlates with a signal if a portion of the signal is similar. Correlation is at the core of many practical wavelet applications.</blockquote><p>So, the continuous wavelet transformation is a convolution</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/468/1*pbWHCrCkAwaEx_V5bjuDpg.gif" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/468/1*6kS8lrK2ov6hAI4yFN_MGA.gif" /><figcaption>Simplified examples of a convolution over a sliding window where the red box is analogous to the wavelet and the blue shapes are analogous to the time series (“Convolution”, 2024).</figcaption></figure><p>where the “coefficients” (also referred to as “magnitudes”) are equivalent to the yellow area under the convolution shown in the examples above. These coefficients are analogous to the correlation between the wavelet and the signal.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*iMLsI7Vm1hZB81IHzXtAoQ.png" /><figcaption>Comparison between convolution, cross-correlation, and autocorrelation. The cross-correlation is equivalent to the reversed convolution or by rotating the wavelet <em>g</em> 180 degrees and then performing the original convolution. The autocorrelation of the signal f is simply convoluting it with itself (“Convolution”, 2024).</figcaption></figure><p>With a complex wavelet, such as the complex morlet wavelet, the magnitude of the absolute value of the coefficient is indicative of the correlation between the wavelet and signal in the time-frequency domain. This absolute value at each time step is then used to create a spectrogram that serves as the input of the paper’s proposed approach.</p><h4>Spectrogram</h4><p>The paper performed continuous wavelet transformations, with different scales <em>s,</em> using the complex morlet wavelet on the various time series it analyzed. The main effect of having a particular scale <em>s</em> in the continuous wavelet transformation is that the scale either stretches or compresses the wavelet, resulting in a different frequency of the wavelet.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/788/1*g7jUpG51NHP6M5Yfq2gyeA.png" /><figcaption>The real morlet wavelet at different scales (Wallisch et al., 2009).</figcaption></figure><p>Going back to the musical analogy, imagine that the real morlet wavelet at scale 1 is a middle C note and at scale 5 is a low B notes. Convolving the audio signal with the real morlet wavelet at scale 1 would be useful for determining when the middle C note appeared in the song. Similarly, convolving the audio signal with the real morlet wavelet at scale 5 would be useful for determining when when the low B note appeared in the song.</p><p>The convolution, using the complex morlet wavelet, was performed at many scales by the paper, and then plotted as a spectrogram.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/998/1*GPk09m2vaNydfoglrThPMA.png" /><figcaption>Various time series and their associated spectrograms (Zeng et al., 2023).</figcaption></figure><p>When looking at the financial time series and the associated spectrogram, it can be seen that the time series is most similar to the complex morlet wavelet of (approximately) scale 300 at time step 39. However, one drawback of the spectrogram is that it does not retain the sign of the signal. To address this, the paper converted the “standardized time series to an image row with intensities represented as integers in the range of [0, 255]” (Zeng et al., 2023) and appended it to the top of the spectrogram.</p><p>One might wonder, “great the time series is similar to the complex morlet wavelet at <em>s</em> scale and <em>t</em> time; what does that practically mean?” Unfortunately, there is no simple answer to that. However, the analogy of a “<strong>spectrogram</strong> is to a <strong>time series</strong> what a <strong>word embedding</strong> is to a <strong>word</strong>” could be offered as consolation. A word embedding and its individual elements’ values are meaningless by themselves, but the projection of the word into a high dimensional space, relative to other words, serves as a useful form of feature extraction for classification and regression tasks.</p><p>For a practical example of this utility, the study by Du et al. (2020), which was cited by the paper as justification for using the complex morlet wavelet, achieved the following results by leveraging spectrograms for classification of intraday “Up” and “Down” classification for the S&amp;P 500 index:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*m-GFbrdG8aoLjQta_qYCEA.png" /><figcaption>Comparative results between visual time series forecasting using both the raw signal and spectrograms (denoised signal), as well as random prediction. The authors concluded that “the performance of this model on the S&amp;P500 index is competitive” (Du et al., 2020)</figcaption></figure><p>While the relative improvement in performance of the spectrogram compared to the raw signal is notable, it should be said that the class balance is 56.76% “Up” and 43.24% “Down.” So, while 57.72% is better than 56.76%, it is only marginally better. However, the takeaway is that the feature extraction via a spectrogram yielded much better results than the raw signal. Additionally, the sign of the signal was not preserved in the study by Du et al. (2020) as it was in the paper.</p><h4>Visual Transformer over Convolutional Neural Network</h4><p>The paper discussed how a key contribution was the demonstration of a “transformer’s ability to learn cross-modality information across time and frequency domain to enhance the forecasting process” (Zeng et al., 2023). This should be readily apparent now that a spectrogram is understood as a magnitude (indicated by the color bar) at a time (on the x-axis) and frequency (1:1 mapping of scale on the y-axis).</p><p>However, an important aspect of why the ViT was more suitable than a CNN for visual time series forecasting was not discussed in depth in the paper. One of the benefits of using a ViT is that there is less inductive bias than a CNN, which facilitates the ability of the ViT to detect longer-term relationships in a time series. Consider the following explanation from the study by Dosovitskiy et al. (2020):</p><blockquote>[A] Vision Transformer has much less image-specific inductive bias than CNNs. In CNNs, locality, two-dimensional neighborhood structure, and translation equivariance are baked into each layer throughout the whole model. In ViT, only MLP layers are local and translationally equivariant, while the self-attention layers are global.</blockquote><p>In a nutshell, the globality of the self-attention layers allows for the position of a pattern in the input to have relevance, while for a CNN the position of a pattern in the input is irrelevant. This relevance of the position of a pattern is of great importance when the input is representing a time series, due to their sequential structure. Furthermore, consider the following chart presented by Dosovitskiy et al. (2020):</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*qNUF2VDgqFPKmW3gApc9lw.png" /><figcaption>Mean attention distance of ViT-L/16 (Dosovitskiy et al., 2020).</figcaption></figure><p>Dosovitskiy et al. (2020) stated that “this ‘attention distance’ is analogous to receptive field size in CNNs.” In the context of a spectrogram, this attention distance can be understood as longer or shorter term relationships over time, or bigger or smaller differences in the scale (frequency) of the wavelet used for the transformation.</p><h4>Results</h4><p>The paper analyzed three datasets in total, ranging from very periodic to non-periodic. The most periodic dataset was a synthetic once computed with the following equation:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*WPuY8AdoLQ0y3vQgUfan_A.png" /></figure><p>All of the variables other than <em>t</em> and <em>T</em> were drawn from either uniform or normal distributions. Overall, there were 150K time series of length 100 generated, with the 20 subsequent time steps predicted using the intial 80 time steps (Zeng et al., 2023).</p><p>The paper also used temperature data gathered by the “Australian Bureau of Meteorology for 422 weather stations across Australia, between 02/05/2015 and 26/04/2017. The dataset was obtained from Monash repo (Rakshitha et al., 2021)” (Zeng et al., 2023). Overall, there were 7,320 time series of length 60 sampled, with the 10 subsequent time steps predicted using the initial 50 time steps (Zeng et al., 2023).</p><p>Lastly, for the non-periodic data, the paper obtained the daily adjusted close prices of S&amp;P 500 constituent stocks through the Yahoo! finance database. Overall, there were 59K time series of length 100 sampled, with the 20 subsequent time steps predicted using the initial 80 time steps (Zeng et al., 2023).</p><p>The paper then assessed multiple models and also performed an ablation study using just the line plot, just the sign intensity color bar located at the top of the spectrograms, and the combination of the sign intensity color bar and spectrogram. The evaluation metrics used were SMAPE, MASE, and sign accuracy with three classes of “going up,” “remaining flat,” and “going down.” The experiments produced the following results:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*MsTHieh9FrZuX7KxVpPtlQ.png" /><figcaption>Results from experiments (Zeng et al., 2023).</figcaption></figure><p>As can be seen, the ViT using a spectrogram and appended sign intensity color bar (ViT-num-spec) as input performed the best in most cases. However, the marginal improvement in performance diminished as the data became less and less periodic, i.e. synthetic to financial. Furthermore, when considering the evaluation using the SMAPE and MASE metrics, the marginal benefits appeared somewhat inconsequential. But, thinking back to the motivation for visual time series forecasting and how the Intersection-over-Union metric demonstrated capture of additional patterns and dependencies, one should not write off the value of these results purely based on the SMAPE and MASE metrics.</p><p>When considering the evaluation using the sign accuracy, it can be seen that the ViT peformed best on the synthetic and financial data, but performed worse than DeepAR on the temperature data. However, the ViT-num-spec did perform much better than the ViT-lineplot when evaluated by the sign accuracy, as well as by SMAPE and MASE. The ViT-num-spec also performed marginally better than the ViT using the sign intensity color bar (ViT-num) as input. This ablation study testifies to the claim made by the authors that they “demonstrated the superiority of spectrograms as a visual representation of time series data, leading to enhanced predictions across various domains and diverse datasets” (Zeng et al., 2023).</p><p>Lastly, for the sign accuracy results on the financial data, it needs to be noted that the authors disclosed that “the ground truth statistics reveal that the distribution of positive, negative, and neutral signs in the dataset is 55.7%, 44.0%, and 0.3% respectively” (Zeng et al., 2023). However, they did not set up the experiment as a classification task, which could explain why the results for the sign accuracy were not particularly impressive.</p><h3>Implementation</h3><p>Based on the results from the paper and the studies cited by the paper, I decided that it would be worthwhile to implement the approach presented in the paper on financial data. However, due to other obligations (and since this article is already pretty long), the implementation walk-though (with code) is still to come as a stand alone article.</p><h4>Notable Differences</h4><p>There are a couple of notable differences that need to be discussed. The first difference is that instead of predicting the subsequent 20 time steps (trading days) using 80 time steps, an “Up” and “Down” classification was performed. The “Up” label was classified as the adjusted close 20 trading days in the future being <strong>greater than</strong> the last adjust close in the time series of 80 trading days; the “Down” label was <strong>less than or equal</strong>.</p><p>The second difference is the composition of the training/validation/test set. The paper used data sampled from the period between 2000 to 2014 for the training/validation set, and a test set with data from the period between 2016 to 2019. This resulted in training/validation set of 46,875 time series with a 0.8 train and validation split, and a test set with 15,625. The implementation used data sampled from the period between 2013 to 2023 for the training/validation set, and a test set with data from the period between 2023 to 2024. This resulted in a training/validation set with 100,000 times series with a 0.9 train and validation split, and a test set with 10,000 time series.</p><p>The third difference is the class balance of the training, validation, and test set. It wasn’t extremely clear whether the paper was referring to the entire dataset or the test set, but it disclosed that “the ground truth statistics reveal that the distribution of positive, negative, and neutral signs in the dataset is 55.7%, 44.0%, and 0.3% respectively” (Zeng et al., 2023). The implementation used sampling to ensure that the training, validation, and test set were all balanced 50/50 between “Up” and “Down.”</p><p>There are a couple other smaller differences that had to due with uncertainty surrounding the paper’s creation of the spectrograms, but they will be discussed in further detail in the soon to come implementation walk-through. However, the complex morlet wavelet discussed in the paper was used, and the PyWavelet library was leveraged to perform the transformation. Below is an example of a spectrogram and sign intensity color bar created with the implementation, as well as the time series from which it was created:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*y5Wndo35zabdtkz_qUxdQw.png" /><figcaption>Time series and associated spectrogram and sign intensity color bar created from implementation.</figcaption></figure><p>A couple of things to note include the scale axis on the spectrogram. The scale should have been inverted, but was not since it wasn’t included in the actual input for the implementation. Additionally, it can be observed that the scale axis is log scale, but the images provided by the paper are not. It is suspected that the paper’s axis should have been log scale, but were not adjusted appropriately. This needs to be explored a bit more in the future work, but that is the current assessment.</p><h4>ViT Architecture</h4><p>The paper stated the following for the ViT model:</p><blockquote>The latent embedding vector has a dimension of 128. We trained the model using a batch size of 128 and the AdamW optimizer with a weight decay of 0.05. The training process was performed for a maximum of 200 epochs, and we implemented an early stopping mechanism with a patience of 10. The base learning rate and learning rate scheduler hyperparameters were tuned for each dataset.</blockquote><p>The implementation maintained all of these decisions, except for the early stopping mechanism with a patience of 10. Instead, the maximum of 200 epochs was maintained, and the model with the highest accuracy on the validation was saved as the optimal model.</p><p>For the learning rate and learning rate scheduler, the implementation performed a grid search over the learning rates of [1e-3, 5e-4, 1e-4, 5e-5, 3e-5, 2e-5, 1e-5]. The implementation used <a href="https://pytorch.org/docs/stable/generated/torch.optim.lr_scheduler.LinearLR.html#torch.optim.lr_scheduler.LinearLR">PyTorch’s linear learning rate scheduler</a> with a start value of 1.0, end values of [1.0, 0.75, 0.5, 0.25], and total iterations of [100, 200]. Lastly, there were three trials for each combination of hyperparameters, resulting of a total of 168 runs.</p><p>The paper also stated:</p><blockquote>In our experiments, we use an input image size of 128 × 128, and image patches of size 16 × 16. The top row of the price time series is scaled to 16 × 128, and the spectrogram is scaled to 112 × 128, ensuring that they are divisible by the patch size.</blockquote><p>The implementation maintained all of these decisions as well.</p><p>The paper made references to the number of layers and heads in the ViT, but didn’t give an explicit number. Instead, it end up pointing to a figure depicting a single transformer encoder block and MLP head. This was taken to suggest that 1 layer and 1 head was used in the ViT, so that is what the implementation used.</p><p>Lastly, using the <a href="https://huggingface.co/docs/transformers/en/model_doc/vit">ViTForImageClassification from Hugging Face</a> was originally considered, but an implementation from scratch provided by a colleague was utilized instead(it’s pretty cool, check it out!):</p><p><a href="https://medium.com/correll-lab/building-a-vision-transformer-model-from-scratch-a3054f707cc6">Building a Vision Transformer Model From Scratch</a></p><h4>Results</h4><p>The optimal model had a training accuracy of 53.52%, a validation accuracy of 53.91%, and a test accuracy of 53.35%. Considering that this was achieved on a 50/50 balanced training, validation, and test set, this is rather impressive compared to the results from the paper. As noted before though, the paper was forecasting the 20 subsequent time steps and the implementation was predicting an “Up” or “Down” label, which could explain the improved sign change accuracy of the implementation.</p><p>The optimal parameters were a learning rate of 3e-5 and a 1.0 starting value with 1.0 ending value with the learning rate scheduler, i.e. effectively no learning rate scheduler. The optimal model was saved on the 195th epoch. Below are the associated loss (CE) and accuracy plots for the for the training and validation sets:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/792/1*MKlgKKd-NNSkF4ax9yBwKQ.png" /><figcaption>Training and validation cross-entropy loss for optimal model.</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/793/1*6-MJRecQWRP673B6lHZWnw.png" /><figcaption>Training and validation accuracy for optimal model.</figcaption></figure><p>For comparison, here are the accuracy plots on the other trials with the same hyperparameters from the grid search:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/803/1*QDHbDxsc-ETpKBL8wgUQaA.png" /><figcaption>Training and validation accuracy for second run using optimal model hyperparameters.</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/811/1*Rpsd_UOUY8MVRbnVoZXGWw.png" /><figcaption>Training and validation accuracy for third run using optimal model hyperparameters.</figcaption></figure><h3>Planned Future Work</h3><h4>Additional Time Series</h4><p>It is reasonable to suspect that additional additional channels representing spectrograms of concurrent time series could provide additional context that would result in improved performance. For example, the S&amp;P 500 index could be introduced to provide context for the individual S&amp;P 500 stock. Additionally, as explored in Du et al. (2020), technical indicators such as the Relative Strength Index (RSI), Exponential Moving Average (EMA), or 60-Day Moving Average (MA60) could be included along with the adjusted closing price.</p><h4>Discrete Wavelet Transformation Denoising</h4><p>The paper performed continuous wavelet transformations, but the paper by Du et al. (2020) that was cited as justification for using the complex morlet wavelet performed discrete wavelet transformations before performing continuous wavelet transformation.</p><h4>Hyperparameter Tuning</h4><p>With the introduction of additional time series and presumably a larger sample size (the sample size in the implementation was kept relatively small so that the data could be loaded onto a single 24GB GPU), there could be a benefit realized by adding more layers or MLP heads to the model architecture. Additionally, it seems from the plots shown in the implementation section that the max epoch value could be increased.</p><h4>Ensembles</h4><p>As can be seen from the plots shown in the implementation section, it appears that there was a good deal of variance in the validation set between epochs. It stands to reason that using an ensemble of ViTs could reduce this variance and perhaps lead to better generalization of the model to the test set.</p><h3>References</h3><p>Du, B., Fernandez-Reyes, D., Barucca, P. (2020). Image Processing Tools for Financial Time Series Classification. <a href="https://doi.org/10.48550/arXiv.2008.06042">https://doi.org/10.48550/arXiv.2008.06042</a></p><p>Dosovitskiy, A. ,Beyer, L., Kolesnikov, A., Weissenborn, D., Zhai, X., Unterthiner, T., Dehghani, M., Minderer, M., Heigold, G., Gelly, S., Uszkoreit, J., Houlsb, N. (2020). An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale. <a href="https://doi.org/10.48550/arXiv.2010.11929">https://doi.org/10.48550/arXiv.2010.11929</a>.</p><p>Sood, S., Zeng, Z., Cohen, N., Balch, T., &amp; Veloso, M. (2021). Visual Time Series Forecasting: An Image-driven Approach. <em>2nd ACM International Conference on AI in Finance</em>, 1–9. <a href="https://doi.org/10.1145/3490354.3494387">https://doi.org/10.1145/3490354.3494387</a></p><p>Subramanyam, V. S. (2021, January 28). <em>IOU(intersection over union)</em>. Medium. <a href="https://medium.com/analytics-vidhya/iou-intersection-over-union-705a39e7acef">https://medium.com/analytics-vidhya/iou-intersection-over-union-705a39e7acef</a></p><p>Wallisch, P., Lusignan, M., &amp; Benayoun, M. (2009). <em>Matlab for Neuroscientists An Introduction to Scientific Computing in Matlab</em>. Elsevier.</p><p>Wikimedia Foundation. (2024, November 4). <em>Convolution</em>. Wikipedia. <a href="https://en.wikipedia.org/wiki/Convolution">https://en.wikipedia.org/wiki/Convolution</a></p><p>Wikimedia Foundation. (2024, September 15). <em>Wavelet</em>. Wikipedia. <a href="https://en.wikipedia.org/wiki/Wavelet">https://en.wikipedia.org/wiki/Wavelet</a></p><p>Zeng, Z., Kaur, R., Siddagangappa, S., Balch, T., &amp; Veloso, M. (2023). From Pixels to Predictions: Spectrogram and Vision Transformer for Better Time Series forecasting. <em>4th ACM International Conference on AI in Finance</em>, 82–90. <a href="https://doi.org/10.1145/3604237.3626905">https://doi.org/10.1145/3604237.3626905</a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=929163fd8c52" width="1" height="1" alt=""><hr><p><a href="https://medium.com/correll-lab/spectrograms-vision-transformers-for-time-series-forecasting-929163fd8c52">Spectrograms &amp; Vision Transformers for Time Series Forecasting</a> was originally published in <a href="https://medium.com/correll-lab">Toward Humanoids</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[CLIP for Candlestick Pattern Classification]]></title>
            <link>https://medium.com/correll-lab/clip-for-candlestick-pattern-classification-ec16d125565c?source=rss-8f1c869d7f35------2</link>
            <guid isPermaLink="false">https://medium.com/p/ec16d125565c</guid>
            <category><![CDATA[ai]]></category>
            <category><![CDATA[clip]]></category>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[candlestick-charts]]></category>
            <category><![CDATA[candlestick-patterns]]></category>
            <dc:creator><![CDATA[Peter Kinder]]></dc:creator>
            <pubDate>Wed, 09 Oct 2024 20:21:15 GMT</pubDate>
            <atom:updated>2024-10-09T20:21:15.940Z</atom:updated>
            <content:encoded><![CDATA[<p>Pattern recognition is essential across many fields, whether guiding a robot’s actions or informing an investor’s next trade. In the world of finance, candlestick pattern recognition is a cornerstone of technical analysis. Given the typically short-term focus of this analysis, the ability to swiftly identify candlestick patterns is crucial for investors. However, as noted in Thomas Bulkowski’s <em>Encyclopedia of Candlestick Charts</em>, there are over 103 known candlestick patterns, and it is safe to assume that many more have been identified since the encyclopedia’s 2008 publication.</p><p>In this tutorial, we are going to generate a dataset a labeled images of candlestick charts using open, high, low and close (OHLC) price data from the current constituents of the S&amp;P 500 and modify a <a href="https://medium.com/correll-lab/building-clip-from-scratch-68f6e42d35f4">CLIP model detailed by my colleague in an earlier article</a>.</p><p><a href="https://medium.com/correll-lab/building-clip-from-scratch-68f6e42d35f4">Building CLIP From Scratch</a></p><p>The notebook for this tutorial can be found <a href="https://colab.research.google.com/drive/1wvpJuSCeng3Kb8r1vI-R_g_nK89Ppo0e?usp=sharing">here</a>. The dataset used to arrive at the results can be found <a href="https://o365coloradoedu-my.sharepoint.com/personal/kinderp_colorado_edu/_layouts/15/onedrive.aspx?id=%2Fpersonal%2Fkinderp%5Fcolorado%5Fedu%2FDocuments%2Fdata%2Ezip&amp;parent=%2Fpersonal%2Fkinderp%5Fcolorado%5Fedu%2FDocuments&amp;ga=1">here</a>.</p><h3>Candlestick Chart Patterns</h3><p>To grasp the concept of a candlestick chart, it’s essential to first understand the candlestick symbol. The symbol includes a stock’s OHLC price data over a discrete period, and the individual OHLC components can be interpreted as shown in the following diagram:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*bN5XIuK1uVV4Q_Ma.png" /><figcaption><a href="https://www.investopedia.com/trading/candlestick-charting-what-is-it/">Investopedia / Julie Bang</a></figcaption></figure><p>Very often the color scheme is represented with red (with the close below the open) and green (with the close above the open) instead of black and white. When represented this way and plotted over multiple periods, a candlestick chart looks as such:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/650/1*Kv_WHblL6G2MYAyF8w3L5w.png" /><figcaption><a href="https://apexcharts.com/angular-chart-demos/candlestick-charts/basic/">ApexCharts</a></figcaption></figure><p>You might be wondering, why do these charts matter to me? Well, candlestick charts are a fundamental tool for technical analysts, and over time, <a href="https://en.wikipedia.org/wiki/Candlestick_pattern">certain patterns</a> have emerged that are believed to signal future price movements of the stock they represent. Some patterns are considered “bullish,” indicating a potential price increase, while others are viewed as “bearish,” suggesting a possible price decrease. A very simple bullish candlestick pattern known as an engulfing can be observed below:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*J0o5ppHG96eoAEnQ.png" /><figcaption><a href="https://www.investopedia.com/trading/candlestick-charting-what-is-it/">Investopedia / Julie Bang</a></figcaption></figure><p>Similarly, a bearish candle stick pattern known as Falling Three Methods can be observed below:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*vtU15hsruI4-CLEu.png" /><figcaption><a href="https://www.investopedia.com/trading/candlestick-charting-what-is-it/">Investopedia / Julie Bang</a></figcaption></figure><h3>Dataset Generation</h3><p>Now that you have a basic understanding of candlestick charts and associated patterns, lets talk about the data used in this tutorial. The full python file used to generate the data can be found <a href="https://github.com/PeterKinder/Candlestick-Pattern-Recognition-with-CLIP">here</a> if you would like to make some modifications.</p><p>The first step in the dataset generation process was to obtain the OHLC price data for the stock current constituents of the S&amp;P 500. The list of the current constituents was obtained from this <a href="https://en.wikipedia.org/wiki/List_of_S%26P_500_companies">Wikipedia page</a>. Once the list was obtained, the ticker symbols were used along with the <a href="https://pypi.org/project/yfinance/">yfinance library</a> to obtain the OHLC price data staring from January 1st, 2010. Lastly, the OHL prices were calculated in relation to the closing price and then multiplied by the adjusted closing price. This was done to account for any possible stock splits which would distort the chart if not accounted for.</p><pre>def get_stock_data(ticker, start_date):<br>    data = yf.download(ticker, start=start_date)<br>    data[&#39;High&#39;] = (1 + (data[&#39;High&#39;] - data[&#39;Close&#39;]) / data[&#39;Close&#39;]) * data[&#39;Adj Close&#39;]<br>    data[&#39;Low&#39;] = (1 + (data[&#39;Low&#39;] - data[&#39;Close&#39;]) / data[&#39;Close&#39;]) * data[&#39;Adj Close&#39;]<br>    data[&#39;Open&#39;] = (1 + (data[&#39;Open&#39;] - data[&#39;Close&#39;]) / data[&#39;Close&#39;]) * data[&#39;Adj Close&#39;]<br>    data[&#39;Close&#39;] = data[&#39;Adj Close&#39;]<br>    return data</pre><p>With the OHLC price data in hand, the next step was to create the candlestick charts and label the images. Now I did not create thousands of candlestick charts and then go through them one by one and manually label them. Instead, I used another python library popular amongst technical analysts: <a href="https://ta-lib.org/">Technical Analysis Library or TA-lib</a>. TA-lib has 61 candlestick patterns that can be recognized using a date indexed Pandas DataFrame with OHLC price data. Yes, yes, I know, if TA-lib can recognize candlestick patterns what is the point of using CLIP to identify them using images of the candlestick charts? Well, it is a proof of concept and TA-lib can only identify 61 candlestick patterns.</p><p>Moving on, a couple important aspects of the labeling process are important to mention:</p><ul><li>The period of each candle stick was one trading day</li><li>There were 10 periods (days) included in each chart</li><li>All patterns were contained within the chart</li><li>Non-trading days were not represented in the charts</li><li>A chart was labeled when the most current (right-most) candlestick created the pattern</li><li>Only charts with a single pattern identified with TA-lib were kept</li><li>Random sampling was used to make the dataset label distribution more balanced</li></ul><p>Lastly, the <a href="https://pypi.org/project/mplfinance/">mplfinance library</a> was used to generate the candlestick charts. The charts also normalized the data to fit it to the designated area and allow apples to apples comparisons. Below is an example of one of the generated candlestick charts:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/118/1*qWzfOsTZ7n70zYIjTLRokw.png" /></figure><p>Here is another example:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/118/1*h2qAw15XHQuC2Y1O8HoM-w.png" /></figure><h3>CLIP Model Adjustments</h3><p>As stated earlier, the CLIP model I used was based on the <a href="https://medium.com/correll-lab/building-clip-from-scratch-68f6e42d35f4">CLIP model detailed by my colleague in an earlier article</a>. The updated code used for this tutorial can be found here, but lets step through a couple important adjustments that were made.</p><p>The first adjustment was made to the default value for the maximum sequence length in the tokenizer function. The tokenizer function tokenizes the provided captions for each candlestick chart in preparation for the text encoder. However, the captions for the candlestick patterns were a bit longer than those for the Fashion-MNIST dataset. Originally the maximum sequence length was 32, but I increased it to 60 which resulted in the following code:</p><pre>def tokenizer(text, encode=True, mask=None, max_seq_length=60):<br>    if encode:<br>        out = chr(2) + text + chr(3) # Adding SOT and EOT tokens<br>        out = out + &quot;&quot;.join([chr(0) for _ in range(max_seq_length-len(out))]) # Adding Padding<br>        out = torch.IntTensor(list(out.encode(&quot;utf-8&quot;))) # Encoding Text<br>        mask = torch.ones(len(out.nonzero()))<br>        mask = torch.cat((mask,torch.zeros(max_seq_length-len(mask)))).type(torch.IntTensor)<br>    else:<br>        out = [chr(x) for x in text[1:len(mask.nonzero())-1]]<br>        out = &quot;&quot;.join(out)<br>        mask = None<br><br>    return out, mask</pre><p>The next adjustment I made was changing the FashionMNIST class that loaded the Fashion-MNIST dataset into the CandlestickPattern class that loaded my custom candlestick pattern dataset. The first main change was changing the source of the load_dataset function to link to the dataset that I had created. The second change was adjusting the captions to reflect the captions of the candlestick patterns. The last change trimming the height of the images and adding some buffer to make the image dimensions more suitable for dividing it into patches. None of the trimming or padded changed the patterns in the images and was done due to some difficulty with having the mplfinance library outputting images of a desired dimension.</p><pre>class CandlestickPattern(Dataset):<br>    def __init__(self, train=True):<br>        self.dataset = load_dataset(&quot;url_here&quot;)<br><br>        self.transform = T.ToTensor()<br><br>        if train:<br>            self.split = &quot;train&quot;<br>        else:<br>            self.split = &quot;test&quot;<br><br><br>        self.captions = captions<br><br><br>    def __len__(self):<br>        return self.dataset.num_rows[self.split]<br><br>    def __getitem__(self,i):<br>        img = self.dataset[self.split][i][&quot;image&quot;]<br>        img = self.transform(img)<br>        img = img[:, 4:-4, :]<br>        img = torch.cat((torch.ones(img.shape[0], img.shape[1], 1), img, torch.ones(img.shape[0], img.shape[1], 1)), dim=2)<br><br>        cap, mask = tokenizer(self.captions[self.dataset[self.split][i][&quot;label&quot;]])<br><br>        mask = mask.repeat(len(mask),1)<br><br>        return {&quot;image&quot;: img, &quot;caption&quot;: cap, &quot;mask&quot;: mask}</pre><p>Another significant adjustment was that of the training parameters. Due to a time constraint a formal hyperparameter grid search was not conducted. But, since the dimension of the images in the Fashion-MNIST dataset was 28x28 and the dimension of the images in my dataset was 260x120 I increased some of the ViT parameters. I also decreased some of the Text Encoder parameters because the number of classes in my dataset was about the same as the Fashion-MNIST dataset, but I felt that the parameters were a bit overkill. The original values are commented out below:</p><pre>emb_dim = 128 #32<br>vit_width = 12 #9<br>img_size = (260,120) #(28,28)<br>patch_size = (20,12) #(14,14)<br>n_channels = 1<br>vit_layers = 6 #3<br>vit_heads = 6 #3<br>vocab_size = 256 <br>text_width = 32<br>max_seq_length = 60 #32<br>text_heads = 4 #8<br>text_layers = 2 #2<br>lr = 1e-3<br>epochs = 50 #10<br>batch_size = 64 #128</pre><p>The last major adjustment was made in the training loop. I commend Matt on putting together the original CLIP model and he did a great job throughout. However, I disagree with how the model was being saved based on the last batch of an epoch loss. I adjusted the code to take the mean of the batch losses for each epoch and use that measure to assess whether the model performed better or not.</p><pre>device = torch.device(&quot;cuda&quot; if torch.cuda.is_available() else &quot;cpu&quot;)<br>print(&quot;Using device: &quot;, device, f&quot;({torch.cuda.get_device_name(device)})&quot; if torch.cuda.is_available() else &quot;&quot;)<br><br>model = CLIP(emb_dim, vit_width, img_size, patch_size, n_channels, vit_layers, vit_heads, vocab_size, text_width, max_seq_length, text_heads, text_layers).to(device)<br><br>optimizer = optim.Adam(model.parameters(), lr=lr)<br><br>best_loss = np.inf<br><br>#array to store the mean loss of each epoch<br>train_losses = []<br><br>for epoch in range(epochs):<br><br>    #array to store each batch loss for the epoch<br>    train_loss = []<br><br>    for i, data in enumerate(train_loader, 0):<br>        img, cap, mask = data[&quot;image&quot;].to(device), data[&quot;caption&quot;].to(device), data[&quot;mask&quot;].to(device)<br>        loss = model(img,cap,mask)<br>        <br>        #add the batch loss to the holding array<br>        train_loss.append(loss.item())<br><br>        optimizer.zero_grad()<br>        loss.backward()<br>        optimizer.step()<br><br>    #add the mean of the batch losses in the epoch to the array<br>    train_losses.append(np.mean(train_loss))<br><br>    #print the last epoch&#39;s mean loss<br>    print(f&#39;Epoch: [{epoch + 1}/{epochs}], Train Loss: {train_losses[-1]}&#39;)<br><br>    #save the model if the epoch&#39;s mean loss is less than the best loss<br>    if train_losses[-1] &lt;= best_loss:<br>        best_loss = train_losses[-1]<br>        torch.save(model.state_dict(), &quot;model/clip.pt&quot;)<br>        print(&quot;Model Saved.&quot;)</pre><p>As a last note, the model can take a while to train. If you have the resources , I would suggest attaching the entire dataset to the GPU for training.</p><h3>Results</h3><p>Similarly to the original <a href="https://medium.com/correll-lab/building-clip-from-scratch-68f6e42d35f4">CLIP model detailed by my colleague in an earlier article</a>, the test set achieved an accuracy of 85%. But, as I stated before, due to time constraints a hyperparameter grid search was not conducted. The notebook to observe these results and try for yourself can be found <a href="https://colab.research.google.com/drive/1wvpJuSCeng3Kb8r1vI-R_g_nK89Ppo0e?usp=sharing">here</a>.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=ec16d125565c" width="1" height="1" alt=""><hr><p><a href="https://medium.com/correll-lab/clip-for-candlestick-pattern-classification-ec16d125565c">CLIP for Candlestick Pattern Classification</a> was originally published in <a href="https://medium.com/correll-lab">Toward Humanoids</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
    </channel>
</rss>