<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Nandita Dwivedi on Medium]]></title>
        <description><![CDATA[Stories by Nandita Dwivedi on Medium]]></description>
        <link>https://medium.com/@nandita_dwivedi?source=rss-9224ae62daca------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/1*1lrFbLjEEvq3WyeXzgJWkw.jpeg</url>
            <title>Stories by Nandita Dwivedi on Medium</title>
            <link>https://medium.com/@nandita_dwivedi?source=rss-9224ae62daca------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Thu, 19 Nov 2026 08:42:03 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@nandita_dwivedi/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[Predicting Stable Portfolios Using Machine Learning]]></title>
            <link>https://medium.com/sfu-cspmp/predicting-stable-portfolios-using-machine-learning-f2e27d6dbbec?source=rss-9224ae62daca------2</link>
            <guid isPermaLink="false">https://medium.com/p/f2e27d6dbbec</guid>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[sharpe-ratio]]></category>
            <category><![CDATA[lstm]]></category>
            <category><![CDATA[deep-learning]]></category>
            <category><![CDATA[portfolio-management]]></category>
            <dc:creator><![CDATA[Nandita Dwivedi]]></dc:creator>
            <pubDate>Sun, 14 Apr 2019 19:39:31 GMT</pubDate>
            <atom:updated>2019-05-10T16:52:28.022Z</atom:updated>
            <content:encoded><![CDATA[<p><a href="https://www.linkedin.com/in/mrafayaleem/">Muhammad Rafay Aleem</a>, <a href="https://www.linkedin.com/in/nandita-dwivedi/">Nandita Dwivedi</a>, <a href="https://www.linkedin.com/in/k-rawat/">Kiran</a></p><h3><strong>1. Motivation and Background</strong></h3><p>Portfolio Management is the process of maximizing the return on a portfolio. Portfolio managers make trading decisions on behalf of their clients depending on their appetite for risk. They analyze different assets, their strengths and weaknesses before making a decision about which equities they should hold in a portfolio for balancing the risk and drawing maximum returns. This makes portfolio management a difficult process. We aim to make this process better and simpler by using predictive modeling and deep learning techniques. We generate stable portfolios on predicted stock prices for next quarter.</p><p><strong>Related Work</strong></p><p>In past a lot of attempts have been made to predict the stock prices for firms, and also to understand how text of news articles, twitter posts, and other platforms influence stock prices. These attempts have involved analysing the impact of sentiments of the mentioned resources and predicting value stock prices. What still remains unexplored is the effect of sentiments of financial reports on these stock markets. Harvard Business School published a working paper, ‘Lazy Prices’ claiming that these financial reports have influence in a firm’s stock price. They stated that a drastic change in similarity (Jaccard and Cosine scores) of these filings can increase or decrease a firm’s market value [1].</p><h3><strong>2. Problem Statement</strong></h3><p>The project required us to first explore finance domain and gather relevant information about stock price movement, trading methods and portfolio management techniques. We understood how impactful data science has been in finance so far and came up with three main problems we wanted to focus -</p><ol><li>To what extent are the quarterly and annual financial reports filed by firms influencing their stock prices?</li><li>How does including the sentiments of historical financial reports change prediction of stock prices?</li><li>How can we built a stable portfolio using the predicted values for a quarter?</li></ol><p>These problems are challenging because we had to do a lot of background research on domain. Using right features and doing proper feature engineering to come up with best predictions had a significant learning curve. In addition to that, the data source and format for these filings is not consistent and therefore scraping and parsing the data was a big challenge. There are two different data sources involved here — one is data from financial reports and other is stock price values. In order to prepare dataset for analysis and to train our prediction model we had to look into various methods to integrate these two sources. These datasets are not a direct match between each other as OHLC data is released daily while SEC filings are released every quarter. To merge them together, we had to carefully match the date ranges and thoughtfully engineer features to map predictions for the next quarter.</p><h3><strong>3. Data Science Pipeline</strong></h3><h4><strong>3.1 Data Collection</strong></h4><p><strong>SEC 10-Q and 10-K filings from EDGAR</strong></p><p>The very first step in our data pipeline was to scrape SEC Edgar (Electronic Data Gathering, Analysis, and Retrieval) database. This is an online database maintained by the U.S. Securities and Exchange Commission (the “SEC”) to track all SEC filings by public companies and now contains over 12 million such filings.</p><p>Since EDGAR doesn’t support any filtering options other than company tickers and central index keys (CIK) at the moment of writing this, we had to scrape all 10-K and 10-Q filings for the S&amp;P 500 [2] companies, regardless of the time periods that we were interested in.</p><p>We collected around 60 GB of this data for S&amp;P 500 tickers from the database. Since EDGAR limits the number of requests per user to 10 per second, we had to add some extra primitives to accommodate for this limitation. Furthermore, given the huge size of dataset, we maintained checkpoints for each downloaded ticker in order to allow resuming at later time in case of failures and avoid any data loss.</p><p><strong>OHLC data from Quandl API</strong></p><p>OHLC (open-high-low-close) is a type of dataset that provides open, high, low, and close prices for a stock for each and every business day. This dataset was one of the most crucial information used to train our models in later parts of our pipeline. We leveraged Stocker [3], a Python abstraction on Quandl API, to retrieve this OHLC data for each S&amp;P 500 company.</p><h4><strong>3.2 Data Preparation</strong></h4><p><strong>SEC 10-Q and 10-K filings from EDGAR</strong></p><p>We found that the scraped 10-K and 10-Q SEC filings were highly unstructured as it contains HTML tags, symbols and numerical tables. We cleaned these files by removing unrelated data and then used them to parse HTML into clean text data which was then used to analyse the sentiments. For cleaning and parsing, we used regex, NumPy and BeautifulSoup.</p><p>As part of data preparation, we also generated CikList list mapping files for 10-K and 10-Q filings for each CIK which contained mappings for dates, SEC types, filenames, CIKs and tickers. Date field was the filing date of the SEC form; SEC type was 10-Q or 10-K, and the filename was the SEC file.</p><p>Each CikList mapping file had approximately 12 rows for 10-K and 34 rows for 10-Q SEC files for a 10 year window. We used these mapping files to keep track of SEC files for every CIK. These saved mappings were later used to calculate sentiments for all SEC files corresponding to each CIK.</p><p><strong>OHLC data from Quandl API</strong></p><p>Python Stocker module makes it relatively easy to get OHLC data for each ticker as a Pandas dataframe. However, since our method makes prediction for the next quarter using a 90 day prior window (reasons described in Methodology section), we had to do some data wrangling before we could feed it to our deep learning models.</p><p>We used a 90 day window (a quarter is 90 days) to construct each row as X containing all OHLC data for that period and put Adj. Close as Y. The method we used to prepare this data is visually explained in Figure 1.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/756/0*JORHoZfZz6hlnEnE" /><figcaption>Figure 1: 90 day windows created on OHLC data</figcaption></figure><h4><strong>3.3 Data Integration</strong></h4><p><strong>Combining Sentiment Scores with OHLC data</strong></p><p>Integrating OHLC data obtained from Quandl API with sentiments from SEC filings based on dates was complicated because of the difference in SEC filing dates and OHLC quarterly dates for the stocks.</p><p>For OHLC data, quarterly stock returns were in months 3rd, 6th, 9th and 12th whereas, for SECs, quarterly data was for months 2nd, 5th, 7th and 10th.</p><p>Thus, to resolve this issue, we used Panda’s “forward” merge method which mapped/joined the sentiment scores for the SEC filing for one quarter with the stock’s price data for the next nearest date. The purpose behind this mapping was to analyse how were the closing prices of tickers affected following the release of a financial report. We have shown first 5 rows of the joined dataframe in Figure 2.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*x0ELS_QgHHJg1bPg" /><figcaption>Figure 2: OHLC data joined with SEC sentiment scores for GWW.</figcaption></figure><p><strong>Standardizing OHLC data for S&amp;P 500 stocks</strong></p><p>We only took 10 years of stock data for the S&amp;P 500 companies. The reason for limiting this data to a recent time window is that it is widely believed that financial data is ‘non-stationary’ and prone to regime changes which might render old data less relevant for prediction [4, 5]. Therefore, we defined this period to be in the range of 1-Jan-2008 to 31-Dec-2018.</p><p>After retrieving this data and limiting it within a 10 year period, we found out that a few companies went public after 1st Jan 2008 and they had to be filtered out to maintain consistency. This filtering reduced the dataset to around 300 tickers. Moreover, there were some cases where few rows of OHLC data were missing for certain tickers. Further investigation didn’t reveal any particular event in those companies’ history so it could have been some bug in Quandl API. To fix this inconsistency, we used Pandas ‘interpolate’ method to construct those missing rows.</p><p>We also used fastai library to convert dates into features such as ‘day of week’, ‘month of year’ and ‘week’ in order to allow the model to learn if there were any temporal relations in stock prices.</p><h3><strong>4. Methodology</strong></h3><p><strong>Sentiment Analysis Using NLTK VADER</strong></p><p>We used NLTK VADER (Valence Aware Dictionary and sEntiment Reasoner) sentiment analyser which is a lexicon and rule-based sentiment analysis tool. In this approach, each of the words in the lexicon is rated as to whether it is positive, negative or neutral and scores are calculated based on how positive, negative or neutral a sentiment is.</p><p>Vader lexicons do not have the financial dictionaries, so we updated it with financial dictionaries from Loughran-McDonald Financial Sentiment Word Lists [6, 7].</p><p>Since the MD&amp;A section of SEC filings focuses on the historical description of the financial indicators from the management’s perspective, if processed correctly, this section can help investors give a better prediction for a company’s future return. In the beginning, we decided to focus on the MD&amp;A section and tried to fetch the content from that section using regex, but due to the disorganised structure of SEC filings, we failed to achieve it.</p><p>Instead, we obtained positive, negative and neutral sentiment scores for the entire sections of each SEC filings (approx. 4600 SEC files). VADER is quite slow with large texts so we split the content into batches of 2000 words which resulted in reduced sentiment calculation time. For each of these batches, we calculated the positive, negative and neutral sentiment scores and took the mean, which was the final sentiment scores for each SEC filing.</p><p><strong>Stock Price Prediction</strong></p><p><em>Initial ideas, failures and success</em></p><p>As mentioned in section 3.3, we had around 300 tickers to work with, given the constraints that we have described in the earlier section. We started training a single model using all 300 tickers by feature engineering them as follows:</p><ul><li>Constructed pandas dataframes for each stock using a 90 day window as described in Figure 1.</li><li>Fit separate scalers on whole data set for each stock to scale values between 0 and 1. Fitting separate scalers was essential because each stock can have different minimum and maximum prices.</li><li>These scalers were then used to transform the respective train, validation and test dataset for each of these stocks.</li><li>After all these transformations we used Keras TimeseriesGenerator to cascade OHLC data for all stocks as described in Figure 3.</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/744/0*IBK7nirG4_nXL08s" /><figcaption>Figure 3: Cascaded OHLC data for each ticker. Each layer denotes dataset for one ticker windowed using the method shown in Figure 1.</figcaption></figure><p>Due to constraints on hardware resources, we failed to train a single model using all the 300 stocks data and had to eventually limit it to the first 50 stocks.</p><p>LSTM models have been studied well and proven to be very effective on time-series data. We compiled neural network models using Keras with two LSTM layers, two dropout layers and a dense layer for the output.</p><p>We obtained very poor results after training a single model over data from all 50 stocks (constructed as described in Figure 3). To investigate it further, we also trained separate models for each of the 50 stocks without the cascading method. Results obtained using the latter proved much more accurate with trend lines that were reflective of the real trends for those stock. Each of these 50 models were trained for 20 epochs and without sentiment scores. Results obtained using this method are discussed in Section 5.</p><p><em>Training with sentiments</em></p><p>Sentiment scores can only be calculated for each quarter, mapping to three 10-Q and one 10-K filing for each financial year considered for a company. Therefore, they needed to be mapped to the daily OHLC data for each stock. We achieved this by considering 90 day OHLC data as one quarter and replicating prior quarter’s sentiment scores for all the rows. Separate models, with exactly same architecture as described above, were trained for 20 epochs for each of the 50 stocks. First training was done with positive sentiment scores and the second was done with negative. Results are discussed in the later section.</p><p><strong>Predicting Stable Portfolios</strong></p><p>We decided to create stable portfolios because we want to hold the portfolio for the predicted next quarter. The strategy we follow in order to create stable portfolios is to keep uncorrelated stocks in same portfolio. We do this to avoid any huge loss in a portfolio, because if one stock goes down the others will balance the loss. For example: If a portfolio has 2 stocks [AAPL, FB] in case they are correlated, if AAPL falls this would mean FB also falls and we get overall loss in portfolio. But if the stocks were uncorrelated, AAPL and FB will not fall together and thus prevent any heavy loss. This creates stable portfolios with balanced risk on which continuous earnings can be drawn for the quarter.</p><p>We started with evaluating the correlation and covariance of each pair combination of stocks, and kept the pairs with weak correlation in same set. We considered a correlation value less than 0.5, and a covariance value less than mean covariance. Then for each set we checked which other stock can be added based on low correlation/covariance values.</p><blockquote>The Sharpe ratio is calculated by subtracting the risk-free rate from the return of the portfolio and dividing that result by the standard deviation of the portfolio’s excess return.</blockquote><figure><img alt="" src="https://cdn-images-1.medium.com/max/998/0*GnGBjoBjQCnu-N2U" /></figure><p>A higher value of Sharpe ratio means better risk-adjusted return. A sharpe ratio greater than 1 is considered good, greater than 2 is considered very good and greater than 3 is considered excellent. We considered risk-free rate of 2% i.e. current risk free rate of US market. We calculated the best sharpe ratio value for each portfolio among randomly selected 200 weights. And based on the Sharpe Ratio obtained we divided our portfolios as good, better and best.</p><h3>5. Evaluation</h3><p><strong>Evaluating the effect of SEC sentiments on the stock prices</strong></p><p>We mapped quarterly OHLC data with the quarterly sentiment scores to analyze how positive and negative sentiments are influencing the closing prices.</p><p>From Figure 4, it can be seen that when positive sentiment increased, GWW’s closing price in the following quarter was trailing the trend and going high. And the opposite trend had been noticed in case of negative sentiments in Figure 5.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*Q2cTXkr_PtkPQ3J3" /><figcaption>Figure 4 : Time series effect of the positive sentiments of SEC files on subsequent close prices for GWW.</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*ENZeaOXm2UfErtgp" /><figcaption>Figure 5 : Time series effect of the negative sentiments of SEC files on subsequent close prices for GWW.</figcaption></figure><p>We were able to see the positive and negative trend between the stock’s closing price and the SEC sentiment scores. It showed that the sentiment scores of the previous quarter had an impact on the stock’s closing price of the next quarter.</p><p>Since stock prices depend on various other factors as well, we were not able to see the relation between sentiments and stock closing price for each quarter.</p><p><strong>Evaluation on stock predictions</strong></p><p>We trained each of our 50 models for three cases; no sentiments, positive sentiments as features and negative sentiments as features. Results for one of the stocks (W W Grainger Inc) for all three cases is shown in Table 1.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*z18X2GQXlqSi4-0-zlGC0w.png" /><figcaption>Table 1: Results for W W Grainger Inc (GWW)</figcaption></figure><p>Although we obtained promising results for the GWW stock, this wasn’t always the case and including sentiments for some of the tickers had little to no effect on the RMSE. This suggests that sentiments from SEC filings are not the only factors affecting stock prices and there can be other potential factors that can be investigated. For example, a sudden announcement of change in energy policy can positively or negatively impact clean energy companies and this change can reflect for the next quarter. If such changes were never addressed in a prior SEC filing, a trained model won’t be able to pick that information from just the sentiment scores of that SEC filing. Improvements and suggestions to circumvent this are discussed in the Future Work section.</p><p>Plots in Figure 6 show Adj. Closing prices for GWW. Green line shows the historical stock performance over a period of 10 years (model training was done using data from this time period). Blue line shows predicted trend without sentiments as features and yellow line shows predicted trend after including negative sentiment scores.</p><p>A magnified view over the test window is shown in Figure 7 and includes actual and predicted trends. Note that although both predicted trends are similar, one that includes negative sentiments appears closer to the real values. This is confirmed by the lower RMSE given in Table 1.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*RTmLYow6ajZPm0pq" /><figcaption>Figure 6: Plots showing Ajd. close prices for GWW.</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*OvwgH4VLgzcW_KvU" /><figcaption>Figure 7: Plots comparing actual Adj. Closing prices for GWW with predicted prices with and without negative sentiment scores.</figcaption></figure><p><strong>Evaluation on Portfolio Optimization</strong></p><p>We analyzed the correlation between 50 stocks using python colormaps. Plot in Figure 8 shows the colormap obtained:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/582/0*Fl0-uQYOVgNKGGGS" /><figcaption>Figure 8: Correlation matrix for 50 stocks</figcaption></figure><p>Blue color is relative to minimum correlation and red is relative to highest. This means that the stocks towards blue can be in the same portfolio. For example: [sbac, msci]</p><p>After getting the stock pairs that have correlation less than 0.5 and covariance less than mean covariance, we analyzed the “pairable” stocks and “unpairable” stocks using the plot in Figure 9:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*j-J90sEVod3_g6Bm" /><figcaption>Figure 9: Plot showing pairable and unpairable stocks</figcaption></figure><p>The plot shows pairable stocks in blue and unpairable stocks in green. For Example: [amgn, mat] qualify to be in same portfolio based on their covariance and correlation values. We generated portfolios based on the above plot and calculated Sharpe Ratio and weight distribution for each portfolio. First few rows of the final dataframe is shown for reference:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*bAoQ1UUMm7O4vqjM" /><figcaption>Figure 10: Final dataframe of constructed portfolios</figcaption></figure><p>Finally, based on Sharpe Ratio, portfolios were divided into good, better and best. The following plots show the Return and Volatility values of Portfolios in “good”, “better” and “best” range. These plots are useful to evaluate the trade-offs between Volatility, Return and Risk and select a portfolio.</p><p>Plot in Figure 11 is generated for “good” portfolios. An example of trade-off by looking at the plot below is [ma, zion, mat, vno, sna, flir, aee, duk, etr, pnr, akam] and [ndaq, mat]. It can be seen that latter gives a better return of 20% with similar risk of 1.7 Sharpe Ratio.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*Iurul09VjqZaZI-9" /><figcaption>Figure 11: Plot of ‘good’ portfolios</figcaption></figure><p>Plot in Figure 12 is for portfolios with Sharpe Ratio &gt;=2 and &lt; 3 (better). Greater sharpe ratio here means that the portfolios are less risky compared to “good” ones. As above, similar interesting trade-offs can be derived in this set as well. The plot shows one such trade-off between [ndaq,flir] and [wfc,rtn] choosing either gives almost the same return but latter is less riskier because of higher Sharpe Ratio.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*-hudUe1csueDlhXe" /><figcaption>Figure 12: Plot of ‘better’ portfolios</figcaption></figure><p>Similar conclusions can be drawn from “best” Sharpe ratio range (Figure 13) as well. This set of portfolios have minimum risk involved.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/866/0*XqlRFf6iAVup_TGS" /><figcaption>Figure 13: Plot of ‘best’ portfolios</figcaption></figure><h3><strong>6. Data Product</strong></h3><p>Our final product is a collection of Jupyter Notebooks. We have divided the product into four modules: <em>SEC Scraper, Sentiment Analyzer, Stock Predictor and Portfolio Generator.</em></p><p><strong>SEC Scraper: </strong>This module scrapes SEC Edgar website to extract 10-Q and 10-K filings for S&amp;P 500 companies. Implements checkpointing so failed downloads can resume later. It also performs all the cleaning on HTML and generates raw text files out of them.</p><p><strong>Sentiment Analyzer:</strong> This module contains two notebooks.</p><ul><li><em>Sentiment Analysis Notebook:</em> Extracts scores for positive, negative and neutral sentiments from SEC files provided in a directory. It saves these extracted sentiments into CSV files which can later be used for model training, visualizations, etc.</li><li><em>Sentiment Visualization Notebook:</em> Generic notebook to plot sentiment trends together with closing prices for a stock.</li></ul><p><strong>Stock Predictor:</strong> This module contains three notebooks</p><ul><li>LSTM Stocks with no sentiments: Performs feature engineering on stock data and trains and evaluates LSTM models for each of the given stocks. Each model is then saved in its respective directory along with plots comparing actual prices with the predicted ones. Training epochs and all directories are configurable.</li><li>LSTM Stocks with sentiments: Performs feature engineering on stock data as well as sentiments extracted from the Sentiment Analyzer module. It then trains and saves each model along with comparison plots.</li><li>Predictions Generator: This notebook can be used after training to load saved models and perform predictions on test data. Predicted data frames can then be exported to CSV files to be used later for various use cases.</li></ul><p><strong>Portfolio Generator and Optimizer:</strong> Constructs portfolios using pairwise selection of weakly correlated stocks. Optimizes portfolios to give highest Sharpe Ratio with respect to the optimal weights assigned to stocks in the portfolio.</p><h3><strong>7. Lessons Learnt</strong></h3><p>Learning about finance domain to understand what can potentially work was definitely a challenge. Moreover, working on how to manipulate time-series data, windowing methods, and training LSTM models using that proved interesting. Our findings also suggested that training a single model for data from all companies is fruitful because of issues such as correlation and covariance.</p><p>Sentiments extracted from SEC filings are significant to predict future stock trends. We learnt to extract sentiments on very large text data (100,000 words in some case) using NTLK VADER.</p><p>We learned how to construct portfolios by leveraging concepts such as correlation, covariance, Sharpe ratio and volatility. Relevant visualizations such as colormaps and correlation matrix were very useful in confirming the obtained results.</p><h3><strong>8. Summary</strong></h3><p>Machine learning in finance is a relatively recent development and it is hard to find the right resources. Moreover, stock price prediction is usually considered a hard problem and one of the reasons why most portfolio managers do not completely rely on it. We have shown neural networks constructed using LSTMs to be reliable for this problem. Sentiment analysis on SEC filings has also proven to be a reliable technique.With cleaner data and more evaluations using tools described in Section 9, we believe that the results can be improved further. Lastly, portfolio construction is hard and stock markets can be volatile. We have shown a method that can be utilized to mitigate risk and construct diversified portfolios based on predicted results.</p><h3><strong>9. Future Work</strong></h3><p><strong>Sentiment Analysis</strong></p><p>For sentiment analysis, we can consider word-embedding models namely: Word2Vec, FastText and Universal Sentence Encoder. We can evaluate if these models can give us more accurate sentiment scores and evaluate if they can help predict better trends on the stock market.</p><p>We extracted only positive, negative and neutral sentiments from SEC filings. There are a lot of other sentiments tied to the financial industry, like uncertainty, litigious, constraining, superfluous, which can be extracted and analyzed further.</p><p><strong>Stock Price Prediction</strong></p><p>Currently, we have trained separate models for each of the stocks considered. This was done because we obtained very poor results after training a single model using the method described in section 4. However, this method of training separate models for each stock can become unscalable if the number of stocks is large. To address this issue, we can consider training models for a set of highly correlated stocks instead of single stock. This can reduce the number of models trained. Moreover, we can leverage transfer learning to retrain the last layer of such models for every new stock that lies within a certain range of colinearity applying to a model.</p><p>We explained earlier that including sentiment scores from SEC filings didn’t necessarily give lower RMSE and better trend lines. To further establish this claim, we can augment our features by including news sentiments for every quarter for each of the respective stocks and evaluate if these scores together with SEC scores will help us make better predictions.</p><p><strong>Portfolio Optimization</strong></p><p>There are a lot of other portfolio construction methods that can be evaluated using our predictions. We can evaluate industry standard techniques such as ‘Global Minimum Variance Portfolio (GMV)’ and ‘Inverse Volatility Portfolio IVP’ [8] and compare those results from real world portfolios for a particular time window.</p><p>Finally, portfolio rebalancing is a standard practice for investment firms. We can extend our methods to implement rebalancing of existing portfolios and compare their performance with the ones that we are constructing for each quarter.</p><p><strong>Github Repository</strong>: <a href="https://github.com/mrafayaleem/equity-portfolio-prediction">https://github.com/mrafayaleem/equity-portfolio-prediction</a></p><p><strong>References:</strong></p><ol><li>Lazy Prices <a href="http://laurenhcohen.com/wp-content/uploads/2017/09/lazyprices.pdf">http://laurenhcohen.com/wp-content/uploads/2017/09/lazyprices.pdf</a></li><li>S&amp;P 500 Companies <a href="https://en.wikipedia.org/wiki/List_of_S%26P_500_companies">https://en.wikipedia.org/wiki/List_of_S%26P_500_companies</a></li><li>Python Stocker <a href="https://github.com/WillKoehrsen/Data-Analysis/tree/master/stocker">https://github.com/WillKoehrsen/Data-Analysis/tree/master/stocker</a></li><li>Universal features of price formation in financial markets: perspectives from Deep Learning <a href="https://arxiv.org/pdf/1803.06917.pdf">https://arxiv.org/pdf/1803.06917.pdf</a></li><li><a href="https://medium.com/datadriveninvestor/why-financial-time-series-lstm-prediction-fails-4d1486d336e0">https://medium.com/datadriveninvestor/why-financial-time-series-lstm-prediction-fails-4d1486d336e0</a></li><li>When Is a Liability Not a Liability? Textual Analysis, Dictionaries, and<strong> </strong>10-Ks <a href="https://www.uts.edu.au/sites/default/files/ADG_Cons2015_Loughran%20McDonald%20JE%202011.pdf">https://www.uts.edu.au/sites/default/files/ADG_Cons2015_Loughran%20McDonald%20JE%202011.pdf</a></li><li>Financial lexicon from Loughran-McDonald Sentiment Word Lists.<a href="https://sraf.nd.edu/textual-analysis/resources/#LM%20Sentiment%20Word%20Lists">https://sraf.nd.edu/textual-analysis/resources/#LM%20Sentiment%20Word%20Lists</a></li><li>The Most Rewarding Portfolio Construction Techniques <a href="https://seekingalpha.com/article/1710142-the-most-rewarding-portfolio-construction-techniques-an-unbiased-evaluation">https://seekingalpha.com/article/1710142-the-most-rewarding-portfolio-construction-techniques-an-unbiased-evaluation</a></li></ol><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=f2e27d6dbbec" width="1" height="1" alt=""><hr><p><a href="https://medium.com/sfu-cspmp/predicting-stable-portfolios-using-machine-learning-f2e27d6dbbec">Predicting Stable Portfolios Using Machine Learning</a> was originally published in <a href="https://medium.com/sfu-cspmp">SFU Professional Computer Science</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[When Deep Learning got BIG!]]></title>
            <link>https://medium.com/sfu-cspmp/when-deep-learning-got-big-a833a69be460?source=rss-9224ae62daca------2</link>
            <guid isPermaLink="false">https://medium.com/p/a833a69be460</guid>
            <category><![CDATA[transfer-learning]]></category>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[deep-learning]]></category>
            <category><![CDATA[intel]]></category>
            <category><![CDATA[bigdl]]></category>
            <dc:creator><![CDATA[Nandita Dwivedi]]></dc:creator>
            <pubDate>Sat, 09 Mar 2019 00:12:43 GMT</pubDate>
            <atom:updated>2019-03-13T20:35:35.752Z</atom:updated>
            <content:encoded><![CDATA[<blockquote>Is it possible to leverage your current Spark cluster to build Deep Learning models?<br>Can terabytes of data stored in HDFS, Hive, HBase be analyzed?</blockquote><p><a href="https://www.linkedin.com/in/mrafayaleem/"><em>Muhammad Rafay Aleem</em></a><em>, </em><a href="https://www.linkedin.com/in/nandita-dwivedi/"><em>Nandita Dwivedi</em></a><em>, </em><a href="https://www.linkedin.com/in/k-rawat/"><em>Kiran</em></a></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*3sD9hYGXLwHIYZW3" /><figcaption>Photo by <a href="https://unsplash.com/@franki?utm_source=medium&amp;utm_medium=referral">Franki Chamaki</a>on <a href="https://unsplash.com/?utm_source=medium&amp;utm_medium=referral">Unsplash</a></figcaption></figure><h4>Diving into Intel’s BigDL</h4><p>Apache Spark has rapidly gotten popular over the past couple of years. This comes from its simplicity, speed and support, also referred as 3 S’s of Spark. Many companies leveraged the Hadoop and Spark environments to build powerful data processing pipelines. These pipelines were built to pre-process huge volumes of data on distributed clusters and draw insights from it for business growth. As Deep Learning gained momentum for its high accuracy predictions and analysis potential, many companies wanted to take advantage of the models that could help improve their businesses further. Intel had such corporate customers with huge data pipelines already built and deployed in their Spark/Hadoop clusters. There were a couple of concerns from these customers on how can deep learning models be applied on these datasets.</p><blockquote>1) Can the same pipeline be leveraged for Deep Learning?</blockquote><blockquote>2) Can Deep Learning be done efficiently for these large datasets, i.e. Deep Learning at scale?</blockquote><blockquote>3) Can the existing Spark/Hadoop cluster be used?</blockquote><p>These questions arose because of the obvious need for a consistent environment where moving these large datasets across different clusters can be avoided. Moreover, moving proprietary datasets from an already existing infrastructure can be a security risk. The earlier experiments for answering these questions involved trying to add existing Deep Learning frameworks over Spark, but they weren’t coherent. This led to the development of BigDL platform on top of Spark, and hence naturally inheriting the 3 S’s: simplicity, speed and support with Deep Learning features.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/500/1*SIKxG7eBfsI61AXEH_u7vA.jpeg" /></figure><p><strong><em>BigDL Library</em></strong></p><p>BigDL is a distributed deep learning library that has rapidly started growing in the field of big data analysis. It is natively integrated with Spark and Hadoop ecosystems because of which it supports significant features like incremental scalability, that allows the models to be trained in any number of clusters depending on the size of data. It provides support to build an end-to-end data-analytics, deep learning and AI pipeline. It performs data-parallel distributed training to achieve high scalability. Since it’s an open source release, BigDL users have constructed numerous analytics and deep learning applications such as visual similarity, parameter synchronization, scaling and convergence etc. on Spark and Big Data platforms.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*yC74N8Yiuq_7fW7i8uVvAw.png" /><figcaption><a href="https://software.intel.com/en-us/ai/frameworks/bigdl/technical-overview">BigDL in Spark Framework</a></figcaption></figure><p>Deep Learning applications can be written as a standard spark program. These libraries that are unified in the Spark framework can read high volumes of data. Moreover, it has support for Python libraries like Numpy, Scipy, NLTK, Pandas, etc. It is integrated with TensorBoard for visualizations and also supports loading of existing Torch models. One of the most important reasons for the enterprise customers wanting to use BigDL and Spark is that in addition to the fact that BigDL is faster than TensorFlow, it also enables them to retrain the models quicker because of parallel computation.</p><p><strong><em>Analytics Zoo Library</em></strong></p><p>Machine Learning in Spark is still in its infancy when compared to numerous standalone libraries available in Python ecosystem. Most of these libraries such as Keras, TensorFlow and PyTorch are not consistent with Spark since they do not support Spark’s underlying core framework that enables distributed computing. Analytics Zoo is a library developed by Intel that is trying to bridge this gap in Spark. It provides a rich set of high-level APIs to seamlessly integrate BigDL, Keras and TensorFlow programs into Spark pipelines. It has several built-in deep learning models for object detection, image classification, text classification, recommendations, etc. The library also provides end-to-end reference use cases such as anomaly detection, fraud detection and image augmentation to apply machine learning on real-world problems.</p><p>To put things into more perspective, the following section provides a brief tutorial on <em>BigDL</em>and <em>Analytics Zoo</em>, showing how easily they can be used to implement transfer learning using pre-trained models and train it on a Spark cluster.</p><h4>Transfer Learning using BigDL and Analytics Zoo</h4><p>In this tutorial, a small dataset containing images of ants and bees is used to perform transfer learning on ResNet-50. This model is pre-trained to classify a given image into one of its 1000 labels. Transfer learning is used to retrain this model for this specific use case and make predictions on the training set containing ants and bees. BigDL and Analytics Zoo enables this training on Spark’s distributed framework. Note that the original ResNet-50 doesn’t contain ants and bees as one of its labels.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/468/1*ORIKv6PWE1rGfE45Zv1jIw.png" /></figure><p>BigDL and Analytics Zoo can be installed using pip as follows:</p><pre>#for Python2<br>pip install BigDL<br>pip install analytics-zoo</pre><pre>#for Python3<br>pip3 install BigDL<br>pip3 install analytics-zoo</pre><p>Before beginning, download the ResNet 50 pre-trained model and training and validation datasets from this <a href="https://github.com/mrafayaleem/transfer-learning-bigdl">repository</a>. More models are available on Analytics Zoo website <a href="https://analytics-zoo.github.io/master/#ProgrammingGuide/image-classification/#download-link">here</a>. You will need to unzip the compressed data.</p><p>Let’s begin by defining imports and initializing spark context using init_nncontext function from Analytics Zoo. Paths to pre-trained model, training and validation data are also defined here.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/7377447f867efd264bdc66b0a9fa5ceb/href">https://medium.com/media/7377447f867efd264bdc66b0a9fa5ceb/href</a></iframe><p>Next, create Spark UDFs to extract filenames. Labels are assigned by checking if the file path contains the keyword ‘ants’ or ‘bees’. Using these two UDFs, construct the training and validation dataframes.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/b43ebd3b95b759c7fe01fe6b08ccf11a/href">https://medium.com/media/b43ebd3b95b759c7fe01fe6b08ccf11a/href</a></iframe><p>To properly construct the model, all images need to be standardized. Analytics Zoo has APIs to create these transformations and chain them so that they are applied in sequence.</p><p>Pre-trained ResNet-50 is loaded using the model file downloaded earlier.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/d76693159e6804499a625fefb9073ce9/href">https://medium.com/media/d76693159e6804499a625fefb9073ce9/href</a></iframe><p>The last 5 layers of ResNet-50 are:</p><pre>res5c_relu<br>pool5<br>Viewf42780f5<br>fc1000<br>prob</pre><p>The last layer in the model is changed to output 2 classes (ants and bees) instead of 1000 that ResNet-50 is trained on. It is constructed with input dimension of 1000 and output dimension of 2 and a softmax activation. This layer is then added as the last element in the pipeline. Note that since transfer learning is being performed, the model can be trained for these 2 new classes in just 25 epochs! This shows the practicality of transfer learning.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/144686a0a66949850b5dda9ff9bcf7f5/href">https://medium.com/media/144686a0a66949850b5dda9ff9bcf7f5/href</a></iframe><p>Now, train the model on training dataframe and get predictions on validation dataframe. Spark allows faster training here across multiple clusters.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/7f359e3556ccb61f8d6bc7095fe2e7c9/href">https://medium.com/media/7f359e3556ccb61f8d6bc7095fe2e7c9/href</a></iframe><p>Finally, predict the classes on test data and show the image with its class.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/f0ab33fd3e69849de75670f19bf1e7c3/href">https://medium.com/media/f0ab33fd3e69849de75670f19bf1e7c3/href</a></iframe><p>An image from the test dataset showing the predicted label on top.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/790/1*ySSPR0S7xmY3rd_7bFr4dQ.png" /></figure><p>If the dataset was a lot larger, say it happens to be stored in HDFS, the same design pipeline can be used and it would have automatically scaled for a much larger cluster. BigDL makes data analytics of these huge datasets a lot faster and efficient. Moreover, other than strong integration with Spark dataframes it is tightly coupled with Spark SQL and Structured Streaming. For example, streaming data from Kafka can be passed to a similar BigDL UDF to make real-time predictions and classifications. Spark SQL can be added to perform querying and filtering at different stages, and even on the predictions.</p><p>A Jupyter notebook with all datasets is provided in this <a href="https://github.com/mrafayaleem/transfer-learning-bigdl">repository</a>.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=a833a69be460" width="1" height="1" alt=""><hr><p><a href="https://medium.com/sfu-cspmp/when-deep-learning-got-big-a833a69be460">When Deep Learning got BIG!</a> was originally published in <a href="https://medium.com/sfu-cspmp">SFU Professional Computer Science</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
    </channel>
</rss>