<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Machine Learning @ NUS Fintech Society on Medium]]></title>
        <description><![CDATA[Stories by Machine Learning @ NUS Fintech Society on Medium]]></description>
        <link>https://medium.com/@nusfintech.ml?source=rss-35fbff2bc022------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/1*MrohZGPG7pXSV2CWo6DfDw.jpeg</url>
            <title>Stories by Machine Learning @ NUS Fintech Society on Medium</title>
            <link>https://medium.com/@nusfintech.ml?source=rss-35fbff2bc022------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Wed, 18 Nov 2026 12:51:26 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@nusfintech.ml/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[Fake Financial News Detection ML Application — Seachain]]></title>
            <link>https://medium.com/@nusfintech.ml/fake-financial-news-detection-ml-application-seachain-45290d3a30b8?source=rss-35fbff2bc022------2</link>
            <guid isPermaLink="false">https://medium.com/p/45290d3a30b8</guid>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[fake-news]]></category>
            <category><![CDATA[large-language-models]]></category>
            <dc:creator><![CDATA[Machine Learning @ NUS Fintech Society]]></dc:creator>
            <pubDate>Tue, 30 Jul 2024 11:59:05 GMT</pubDate>
            <atom:updated>2024-07-30T11:59:05.415Z</atom:updated>
            <content:encoded><![CDATA[<h3>Fake Financial News Detection ML Application — Seachain</h3><h3>Introduction</h3><p>This project (Seachain), undertaken by the FinTech Society, focuses on developing a machine learning-based tool that curates and authenticates financial news for investors. It targets the prevalent issue of information overload in financial markets and the need for reliable, personalized news feeds.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/700/0*En7wFzsVJxdLBGOf.png" /></figure><h3>Problem Statement</h3><p>The financial domain suffers from an influx of data, making it challenging to identify pertinent and credible information. This project addresses these issues by filtering noise and providing tailored trustworthy financial insights.</p><h3>Solution Objective</h3><p>The objective is to utilize machine learning techniques to create a system that not only verifies the authenticity of news articles but also customizes financial news feeds based on user preferences.</p><h3>Scope and Data Sources</h3><p>The project will primarily target financial news, including market trends and company analyses. The data sources for this project will include financial news APIs, user interaction data, and specific datasets like Kaggle’s FakeNewsNet.</p><h3>Implementation Plan</h3><p>The implementation involves multiple phases:</p><ol><li>Literature Review and Data Collection</li><li>Data Cleaning and Feature Engineering</li><li>Model Development</li><li>Website Building</li><li>Alpha Testing</li><li>Final Presentation</li></ol><h3>Problems Encountered</h3><p>Key challenges include:</p><ul><li>Data underfitting or overfitting</li><li>When the ML models are tested on a previously unseen dataset, performance plummeted</li><li>Keeping the model updated with the latest information</li><li>Resource constraints for model training</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/462/0*ceJ9LM-yTbPPWZP9.jpeg" /></figure><h3>Product Market Fit</h3><p>Upon interviewing multiple experienced and inexperienced investors or people who are active in the financial market, everyone recognized the noise and fake news problem in the financial market. They believe a tool that can filter out all the noise and curate information customized for each user would significantly benefit investors in information-gathering.</p><h3>Data</h3><h3>Data Description</h3><p>The aim of our machine learning models is to predict the ‘label’ field in the datasets, i.e. fakeness of articles of unseen news as accurately as possible. To source train data for the models, real and fake news articles are collected from multiple Kaggle datasets and scraped from various news sources and fake news websites with Beautiful Soup and Selenium. To test the performance of several selected machine learning models, we first implement them on a <a href="https://www.kaggle.com/competitions/fake-news/data">Kaggle</a> dataset containing 20,000 entries of real and fake articles. For the models that perform well (i.e. Multinomial Naive Bayes and LSTM), we then expand the train dataset to our existing database of real and fake news for more robust training.</p><p>We compiled the following datasets with 1.32M news entries as ‘general_df’ for training the models. Around 2% of which are fake news.</p><p><strong>Real:</strong> general news, world news, politics news, business news, true_with_source (compiled dataset from multiple Kaggle dataset that include news article with url)</p><p><strong>Fake:</strong> general news, US news, combined_fake_news (scraped from news websites from Appendix 1), government news, conspiracy news, Middle East news, politics news</p><p>We test the models with the test dataset (reuters_ft3_df_nb), which consists of web-scraped data from Reuters and articles from the recent 3 months on Financial Times, and the initial Kaggle dataset.</p><p>It should be noted that for certain models there were trained on a dataset from only one source which may expose it to certain biases the original dataset compiler might have had.</p><h3>Data Preprocessing</h3><p>First, we standardize input formats of the train and test datasets:</p><ol><li>Add labels ‘0’ or ‘1’ to indicate the actual fakeness of the articles, where 0 stands for real and 1 stands for fake.</li><li>Rename columns of data frames to ‘text’, ‘author’, ‘title’ and ‘label’ respectively, and remove other redundant columns like ‘id’, ‘URL’, etc.</li><li>Replace NA values with empty strings ‘’, to handle missing values.</li></ol><p>To improve the quality of input for the models, several common NLTK text preprocessing steps are then implemented on the ‘text’, ‘author’ and ‘title’ columns:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/700/0*oxohxwgoweLqhc0r.png" /></figure><p><em>Before preprocessing</em></p><ol><li>Remove URLs.</li><li>Convert all texts to lowercase, remove punctuations and repeating characters.</li></ol><figure><img alt="" src="https://cdn-images-1.medium.com/max/700/0*3ovEHytlQQd7lJ2n.png" /></figure><p>3. Remove stopwords, i.e. common words like ‘is’, ‘are’, ‘the’, ‘a’, etc, to reduce noise in the data such that models do not lose predictability.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/700/0*9Q-k_4cruHxHHMYL.png" /></figure><p>4. Lemmatization, to break words down to their root forms. For example, from ‘dancer’ to ‘dance’.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/700/0*hrcnTKlB9_J5E_7-.png" /></figure><p>5. Afterwards, these processed texts are tokenized respectively as suitable inputs for different models.</p><h3>Data Challenges</h3><p>One obvious challenge is the imbalance in the real to fake news ratio in our train dataset. This is due to the difficulty in sourcing for fake articles which are well-labeled as fake whereas real news are easily accessible — the available sources for collecting real news are much more plentiful than those for fake news.</p><p>It’s even more challenging to find labeled fake business or financial news which could affect the models’ ability to capture domain-specific signals that suggest news fakeness. Initially, our plan was to predict strictly using real and fake business news, but the performance severely suffered due to the small train dataset size. We address this challenge by collecting fake news related to other topics that are correlated to financial or business events, such as politics and government news. General news is collected such that the models can identify systematic signals that are consistent throughout fake news of all genres. To check if this was a feasible plan, we trained a Naive Bayes classifier with general news to predict politics news later on. Its performance was comparable to predicting politics news with politics news (80–20 train-test split), giving us a confidence booster.</p><p>Some data sources do not have authors, which could potentially be useful predictors of news fakeness. The quality of fake news is generally low as well, with unnecessary repeating characters, punctuations or grammar that’s completely out-of-place. This issue is partially tackled with textual preprocessing steps.</p><h3>Example of Data Processing For Decision Trees and Random Forest</h3><p>The financial data prepared for developing the individual decision trees compiled into the larger random forest model was downloaded from <a href="https://www.kaggle.com/competitions/fake-news/data">Kaggle</a>, a reliable data repository. The uncleaned dataset contained approximately 20,000 rows of data with 5 feature columns, out of which the id column was disposed of as it provided no useful metric for evaluation. The final 4 feature columns kept in the dataset described a financial article’s (per row) title, author, text, and real/fake (0/1) label. Thus, the target variable the model predicted was the label column.</p><p>After preparing the datasets, they are preprocessed by the techniques mentioned above, such as removing duplicate/null entries and NLTK processing techniques.</p><p>Finally, the feature columns were transformed from words to numeric values using a <a href="https://www.kaggle.com/models/google/universal-sentence-encoder/frameworks/tensorFlow2/variations/universal-sentence-encoder/versions/2?tfhub-redirect=true">Universal Sentence Encoder</a> technique that encodes text into high-dimensional numeric vectors called text embeddings. Post-preprocessing, a 70–30 split was observed for the training and testing data respectively.</p><p><em>Note: Other ML models’ datasets (both training and testing) are prepared in a similar way to the Decision Trees and Random Forest.</em></p><h3>ML Models</h3><h3>Decision Trees and Random Forest</h3><p>The random forest classifier is a supervised machine-learning model. This classifier uses an ensembling approach where the predictions from smaller decision tree models are combined to give the final result. As mentioned in the data preparation section, the model used a cleaned dataset of 18,000 entries. The available dataset contained 5 feature columns “id”, “title”, “author”, “text”, and “label.” Out of these “id” was disposed of as it provided no useful evaluation metric and “label” was chosen as the target variable where 0/1 values described real/fake news respectively.</p><p>Regarding the model architecture, the random forest classifier employed an ensemble of decision trees. In this particular setup, three decision trees were built using different combinations of features: (title, author), (author, text), and (title, text). Each decision tree within the random forest was trained independently on its respective set of features. By utilizing different combinations of features in these trees, the overall model gained the ability to capture diverse aspects of the data, reducing overfitting and enhancing predictive performance through the ensemble learning technique.</p><p>In total, it took 1–2 hours to train the model on a training split of 70–30 for training and testing data respectively with a random state seed set to 42.</p><h3>Multinomial Naive Bayes Classifier</h3><p>The Multinomial Naive Bayes classifier is a probabilistic machine learning algorithm based on Bayes’ theorem. It is one of the branches of the Naive Bayes classifier that is designed to take in features that represents counts or frequencies. It calculates the probability of a data point belonging to a particular class based on the probabilities of its features. Despite holding the simplifying assumption of independence between features (which is why it’s ‘naive’), it still performs surprisingly well most of the time and is effective with high-dimensional data. It is commonly used for classification tasks, like text classification or sentiment analysis in Natural Language Processing (NLP).</p><p>Firstly, it is implemented on the Kaggle dataset. It is preprocessed in a similar way as in Decision Trees, and later tokenized with count vectorization, where the maximum number of features is capped at 15,000. The model is then trained and tested with a train-test split of 80–20. All 3 features, title, author and text, are involved. It takes just a few seconds to train the model.</p><p>Then, the train and test datasets are expanded to other news sources collected, as detailed in ‘Data Details’. After preprocessing the data, it is tokenized with count vectorization again with a maximum number of features capped at 20,000. Two other hyperparameters, minimum document frequency (minimum number of documents that a word appears in to be considered a vocabulary) and maximum document frequency (maximum percentage of documents that contains a certain word for it to be not discarded), are kept at 2 and 0.8 respectively. We implemented the classifier on (1) a 80–20 train-test split on ‘general_df’ and (2) trained the model with the entire general_df and tested it with the test dataset reuters_ft3_df_nb. Combinations of using title and text and title, text and author are attempted. It took several minutes to tokenize the textual data but the training time remained at several seconds.</p><h3>LSTM</h3><p>The LSTM model, or Long Short-Term Memory model, is a type of Recurrent Neural Network (RNN) architecture designed to handle long-range dependencies in sequential data. It consists of a memory cell that stores and propagates through the network over time, gates like forget gate, input gate and output gate, as well as the hidden state which contains information to be passed on to subsequent steps. It is widely used for NLP tasks involving sequences, like text classification, machine translation and sentiment analysis, due to its proficiency in learning patterns from sequential data.</p><p>Similar to the workflow in Naive Bayes, the LSTM model is initially implemented on the Kaggle dataset, with a batch size of 64 and 6 epochs. Then, it is implemented on a 80–20 train-test split on ‘general_df’ and trained with a batch size of 512 and only 1 epoch due to memory constraints. All 3 variables, text, title and author, are used.</p><p>The data first passes through a vectorization layer to pad the text to sequences with maximum length of 3000 and a maximum of 500 features, again due to memory constraints of handling large datasets. Then, the padded sequences pass through the following layers:</p><ol><li>Input layer to take in the padded sequences.</li><li>Embedding layer to convert the input sequences to sequences of dense vectors.</li><li>LSTM layer with 64 neurons, containing gates to regulate the flow of information.</li><li>Dense layer (or, fully connected layer) with 256 neurons and a Rectified Linear Unit (‘relu’) activation function.</li><li>Dropout layer, with a dropout rate of 0.5 to prevent overfitting. It is a regularization technique that randomly drops a proportion of neurons to enable the model to learn more robust features from unseen data.</li><li>Output layer, with a sigmoid activation function.</li></ol><figure><img alt="" src="https://cdn-images-1.medium.com/max/700/0*9hBkuwIRHnGPW1uJ.png" /></figure><p><em>LSTM model architecture</em></p><p>Similar procedures are implemented when we use ‘general_df’ to train the LSTM model and ‘reuters_ft3_df_nb’ for testing. It uses the binary cross entropy loss function and optimises with Root Mean Square Propagation (RMSProp). Training the model with G4-GPU takes 5 to 10 minutes, while running it locally takes 1.5 hours.</p><h3>Gradient Boosting Machine + Support Vector Machines + Logistic Regression</h3><p>A Support Vector Machine (SVM) is a powerful supervised machine learning algorithm used for both classification and regression tasks. It aims to find a hyperplane in a space that best separates data points into different classes. Thus, we can exploit SVM to classify news into true news and fake news. We apply the support vector classifier (SVC) from the support vector machine in scikit-learn.</p><p>Logistic Regression is a statistical model used for binary classification tasks. It predicts the probability that an instance belongs to a particular class using a logistic (sigmoid) function and assigns the instance to the class with the highest probability. We apply logistic regression from linear models in scikit-learn.</p><p>A Gradient Boosting Machine (GBM) is an ensemble learning algorithm widely used for regression and classification tasks. It sequentially builds a strong predictive model by combining the outputs of weak learners, often decision trees, in a way that corrects errors made by the previous models. It optimizes predictions by minimizing the residual errors through gradient descent. We apply the gradient boosting classifier from the ensemble module in scikit-learn.</p><h3>BERT LLM</h3><p>BERT (Bidirectional Encoder Representations from Transformers) is a cutting-edge natural language processing model developed by Google in 2018. It utilizes a transformer architecture and is pre-trained on large amounts of unlabeled text data to predict missing words bidirectionally, capturing deep contextualized word representations. After pre-training, BERT can be fine-tuned for specific tasks, such as text classification or question-answering.</p><p>Therefore, in addition, we also tried fine-tuning the BERT model to perform fake news detection.</p><p>The dataset used here is a dataset of over 70 thousand news articles with 49% real and 51% fake news downloaded from Kaggle. It contains four columns: Serial number (starting from 0); Title (about the text news heading); Text (about the news content); and Label (0 for fake and 1 for real). Here we used the title of the news to conduct the detection.</p><p>Basically we freeze the layers of the BERT model, which means we maintain the original BERT model parameters. However, we change the architecture of the BERT model: we add two more layers after the original layers. Thus, what we aim to do is to fine-tune the parameters of the newly added two layers based on our dataset.</p><p>Running one epoch (on GPU) costs about half an hour.</p><h3>Performance Analysis</h3><h4>Multinomial Naive Bayes</h4><p>It is evaluated with mainly with its accuracy, alongside other metrics like precision, recall, f1-score and support. For the Kaggle dataset, its accuracy is at 92%. For ‘general_df’, it achieved accuracies of 90% for both title + text and title + text + author. Training with ‘general_df’ and testing with ‘reuters_ft3_df_nb’ gives an accuracy of 67.6%, indicating that the classifier gives mediocre predictions for articles with contexts different from the training dataset.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/700/0*MN9Z4Y0cc_hA83en.png" /></figure><p><em>Training with general_df and testing with reuters_ft3_df_nb</em></p><p>To illustrate performance evaluation with Receiver Operating Characteristic (ROC) curve, we implemented the model on one of the datasets involved in general_df, true_general.csv.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/700/0*4dfkWKg_eHO1xIZN.png" /></figure><p><em>ROC curve of true_general.csv (title + text + author)</em></p><h4>LSTM</h4><p>LSTM model is evaluated with its accuracy. For the Kaggle dataset, its accuracy is at 99%. Whereas for ‘general_df’ alone, it achieved an accuracy of 97% for both title + text and title + text + author. Training with ‘general_df’ and testing with ‘reuters_ft3_df_nb’ gives an accuracy of 95%. It performs consistently well throughout all 3 trials, thanks to the robustness of its deep learning architecture.</p><p>Alternative performance metrics can be used to evaluate a LSTM model. The diagram below shows the model loss when implemented on true_general.csv.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/700/0*3xzuMjQJ6jTt1ss7.png" /></figure><p><em>LSTM model loss for 6 epochs, true_general.csv</em></p><h4>Decision Trees and Random Forest</h4><p><strong>Evaluation Metrics</strong></p><p>The evaluation metrics used for this model were the accuracy of individual decision trees which was approximately averaged to get the final accuracy of the random forest model.</p><p><strong>Results</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/700/0*9TXAoLoX50gyp8Qc" /></figure><p>The picture above showcases, the accuracy of the random forest model.</p><p><strong>Comparison with Other Models</strong></p><p>Comparing this result with other models such as SVM, Logistic Regression, and Naive Bayes that achieved respective accuracies of approximately 94, 95, and 92%, it showcases the model achieved a lower accuracy in comparison.</p><p><strong>Interpretation</strong></p><p>The result can be interpreted to be lowered due to the addition of a loosely correlated decision tree (title, text) which lowered the overall accuracy of the model. Moreover, the random forest model becomes more robust with the additional trees which in this case only 3 were built.</p><h4>SVM, Logistic Regression and GBM</h4><p>For SVM, Logistic Regression and GBM, they all fit the model quickly. For logistic regression model, it fits the training data almost immediately. For the other two models, they finish fitting in less than 1 minute.</p><p>To measure the performance of the models, we use accuracy. And for SVM, the accuracy can be around 95%, which is quite high. For logistic regression, the accuracy can also reach a high level around 95%. Lastly for GBM, the accuracy of the prediction is able to achieve 96%. The performance of the three models are generally similar. However, it is worth noting that these results are coming the model being trained and tested on the same parent dataset (the dataset where we did the 80/20 or 70/30 train test split). When we tried evaluate the performance of these model on previously unseen dataset, the accuracy score dropped, and becasues these models are too simplistic for our purpose, we didn’t invest too much on these models.</p><h4>BERT LLM</h4><p>To measure the performance of the fine-tuned BERT model, we calculate metrics such as accuracy, precision, recall. For fake news, the precision and recall rate are both 76%. It means 76% of the fake news predictions are actually fake, and 76% of the actual fake news are predicted as fake news.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/700/0*jNPEoqkfNRmvNsc6" /></figure><p>The picture above showcases the assessing metrics (precision, recall, f1-score) of the fine-tuned BERT model.</p><p>Compared to other machine learning models, the performance is not so ideal. Therefore, we planned to use text content of news, instead of only the title, to conduct the detection. However, due to some technical and practical issues (e.g. memory limit, usage of TPU) and time limit, we have not generated a proper result.</p><h3>Model Analysis</h3><h4>SVM, Logistic Regression and GBM</h4><p>SVM, Logistic Regression and GBM, although achieve a relatively high performance score, are too simplistic from the perspective of simple model training and lack of fine-tuning. Instead of using them as primary machine learning models, we can exploit them as quick, straightforward, and supplementary models.</p><h4>Multinomial Naive Bayes and LSTM</h4><p>They perform better than most models involved. For Multinomial Naive Bayes, this is attributable to its strength in handling frequency / counts in classification problems in NLP. Though its performance is slightly subpar to the LSTM model, it is worthnoting that its runtime is extremely short and takes up much less memory than deep learning methods. For LSTM, its advanced Recurrent Neural Network architecture makes it good at capturing long-term text dependencies, explaining for its exceptional performance.</p><p>Yet, as mentioned before, Multinomial Naive Bayes is not as good at handling out-of-sample contexts as LSTM, which makes LSTM a more versatile and flexible method than Naive Bayes. To improve, different plug-ins or dictionaries like Word2Vec can be incorporated in the Multinomial Naive Bayes model to handle out-of-vocabulary words.</p><p>Due to memory and runtime constraints for LSTM, only 1 epoch is involved in training, and we capped parameters at 500 for maximum number of features and 3000 for maximum padded sequence length, which sacrifices some accuracy. Though adding more layers helps improve accuracy, it worsens the memory problem currently faced. One way to improve is to cut down on the number of real news and incorporate more fake news in general_df, which would be more value-adding in boosting its performance given the same train dataset size. More text preprocessing steps can be implemented to further filter away irrelevant texts and improve the input quality.</p><h4>Decision Trees and Random Forest</h4><p>Efficiency Analysis: The model trains in 1–2 hours on 18,000 entries, balancing time and computation. However, its ensemble nature might demand higher computational resources due to multiple decision trees.</p><p>Use Case Relevance: Tailored for fake financial news detection, the model leverages varied textual features (title, author, text) to discern patterns between real and fake articles.</p><p>Strengths: Ensemble design prevents overfitting and bolsters model robustness. Utilizes diverse textual features, enhancing analysis depth. Applies preprocessing and text embeddings for effective data transformation.</p><p>Weaknesses: Shows lower accuracy compared to other classifiers, signaling potential limitations. Dependency on textual features might restrict performance in data-scarce scenarios.</p><p>Potential Improvements: Explore advanced feature engineering beyond text embeddings. Experiment with varied feature combinations or feature importance analysis for refinement. Consider larger ensembles, balancing diversity and overfitting.</p><p>Practical Application: While the current performance is subpar when compared with other models we tested (mainly due to the low amount of decision trees used), with enhancements, the model can be powerful in supporting automated fake financial news detection.</p><h4>Llama2 LLM</h4><p>We combined web scraping and Llama2 to allow users to find and evaluate similar news articles from online than the provided one. This will allow the users to see whether there is a discrepancy in information provided between the different news articles. If there are misalignments in information provided across different news articles, the user should be alert about the credibility of the news.</p><p>Llama 2 is a revolutionary Large Language Model developed by Replicate. For our news corroboration and similarity-checking feature, a data scraper was developed that fetches the user search query and the top 5 article content from the web relating to that query. This fetched input alongside the user-given search query and user-input article content is fed into the Llama 2 LLM using the Replicate API. This LLM then interprets if the article content input by the user matches the web results and appropriately generates a response for the user that confirms or denies the claims in the user’s input article.</p><h3>MVP / Website</h3><p>Below, we present the Seachain Minimum Viable Product (MVP), an intuitive website crafted with precision to serve as your financial news companion. This MVP is comprised of three distinct functions: the News Reliability Checker, the Similar Article Feature, and the Company News Aggregator. Each has been developed to empower users with tools for assessing the credibility of financial news, discovering related articles, and staying informed with the latest company-specific updates.</p><h4>News Reliability Checker</h4><p><strong>News Reliability Check Input Interface</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/700/0*4c6M0t8ePF616D5Q" /></figure><p>This screenshot depicts the user interface of the News Reliability Checker feature. Users are prompted to enter the title and body of a news article into designated text fields. A prominent button labeled “Check News Reliability” invites action to initiate the verification process.</p><p>This component of the MVP serves as the entry point where users interact with the LSTM model to assess the credibility of news content.</p><p><strong>News Reliability Check Process (Example Article)</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/700/0*XUCN4pZmD4diC2h8" /></figure><p>This news article from Yahoo Finance is used as an example.</p><p><strong>News Reliability Check Output Result</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/700/0*YdjOkwnSFJ4gC6pg" /></figure><p>The result screen of the News Reliability Checker is displayed in this screenshot, where the system has labeled the input article as “Real!” This clear and concise feedback provides immediate value to the user. If the news is fake, the News Reliability Checker will return “Fake!” with an accuracy close to 90% based on our testing.</p><h4>Similar Article Feature</h4><p><strong>Similar Article Input Interface</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/700/0*LGIryW8NSjfXWpv1" /></figure><p>This image showcases the interface where users can input an article’s title and content to find similar articles.</p><p>This feature retrieves the top articles on from the web base on the given article’s title, then it uses Llama2 to analyze whether the given article has consistent information with other articles online. This is useful to help the user to find whether the given article’s information is consistent with other articles online. If there appear to be a significant information mismatch, then the user should be wary of the reliability of the given article.</p><p><strong>Similar Article Output Display</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/700/0*5bGLKimMoTJp80mp" /></figure><p>Here we see the output following a user’s request for similar articles. The system presents a list of hyperlinked articles that align with the content of the user’s input. Users can then use the Llama2-generated summary and the related article links to check whether the news information are consistent across various news sources.</p><h4>Company News Aggregator</h4><p><strong>Company News Input Interface</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/700/0*0WAzNZ0bOM58qoiY" /></figure><p>This interface invites users to enter a company name to fetch the latest news. A simple input field and a “View news” button create an intuitive user experience.</p><p>Upon entering a company name, the website utilizes web scraping to aggregate recent news from credible sources such as Forbes, streamlining the information discovery process.</p><p><strong>Company News Scraping Code</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/700/0*VjbKV_LAg1Wlkkr6" /></figure><p>This screenshot reveals a portion of the Python code that powers the news aggregation feature, specifically the function for scraping news links related to a user-entered ticker symbol from Forbes.</p><p><strong>Company News Output List</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/700/0*MHsmRy2YslEXngX0" /></figure><p>Displayed is the output of the company news feature, showing a compiled list of the latest articles from Forbes about the entered company, complete with clickable links.</p><p>This output serves as a quick reference for users, directing them to a curated selection of recent articles from reputable sources on the specified company.</p><h3>Cost Analysis</h3><h4>Resource Utilization</h4><p>The primary paid resource utilized in the project was Google Colab Pro. All other resources, including the use of Llama2 from Replicate and running machine learning models on Google Colab’s free tier, were available at no cost.</p><h4>Google Colab Pro Expenditure</h4><ul><li>Resource: Google Colab Pro</li><li>Purpose: To provide enhanced system RAM, larger disk space, and TPU access for intensive ML model training and data processing.</li><li>Team Members with Access: 1 (One member required the additional computational power for running Bert LLM Model to detect whether the given financial news is fake)</li><li>Duration: 1 month</li><li>Rate: SGD 14.33 per user per month</li><li>Total Cost: SGD 14.33</li></ul><h4>Additional Resources</h4><ul><li>Llama2 (Replicate Credits): Free credits were utilized for accessing Llama2.</li><li>Google Colab (Free Tier): Used for running ML models without the need for enhanced computational power. This resource was leveraged by the majority of the team for regular tasks.</li></ul><h4>Total Expenditure</h4><p>The total expenditure for the Seachain project, over its development period, amounted to SGD 14.33.</p><h3>Conclusion</h3><p>Over the past three months, our team has embarked on an ambitious journey to develop a cutting-edge tool for curating and verifying financial news. The project, Seachain, is designed to address the challenges of information overload in the financial sector and provide investors with reliable, personalized news feeds. Through diligent effort and innovative approaches, we have made significant strides towards achieving this goal.</p><h3>Key Achievements</h3><p>Development of Various ML Models: Our exploration of multiple machine learning models like LSTM, Naive Bayes, Decision Tree, Random Forest, SVM, and Logistic Regression has yielded valuable insights into fake news detection. These models have shown varying degrees of success, with some performing exceptionally well under specific conditions.</p><p>Improvement in Model Accuracy: Initially, our models faced challenges in generalizing to unseen datasets, with accuracy dropping significantly when tested on different data. However, by increasing the diversity and volume of training datasets, we have significantly improved model performance, particularly for LSTM, which now achieves around 90% accuracy.</p><p>Integration of Web Scraping and Llama 2: This innovative approach has proven effective in corroborating news articles, adding another layer of verification to our tool.</p><p>Challenges with Bert Model: While our experiments with the Bert model to detect fake news faced technical hurdles, these challenges have provided valuable learning opportunities and directions for future improvements.</p><p>Development of a Functional Website: Our product has taken shape in the form of a locally hosted website, marking a significant milestone towards making our tool accessible to users.</p><h3>Moving Forward</h3><p>As we continue to refine and enhance Seachain, our focus will remain on improving the accuracy of news detection and enriching the user experience. Future efforts will include:</p><ul><li>Further Refinement of ML Models: Continuous testing and training with more diverse datasets to enhance accuracy and reliability.</li><li>Enhancing User Personalization: Implementing features that actively curate news feeds based on individual user interests.</li><li>Scalability and Accessibility: Transitioning from a local host to a more scalable platform, ensuring wider accessibility for users.</li><li>Continued Innovation: Exploring new technologies and methodologies to stay at the forefront of financial news verification.</li></ul><h3>Concluding Thoughts</h3><p>The journey of the Seachain project has been a testament to the power of collaboration, innovation, and persistence. As we move forward, we are committed to upholding these values and continuously striving to provide a tool that empowers investors with credible and personalized financial news. Our progress thus far fuels our optimism for the future of Seachain and its potential impact on the financial community.</p><h3>Appendix</h3><p><strong>Appendix 1 — Fake news websites</strong></p><ul><li><strong>The Onion</strong>: A satire site that mimics a serious news site and often fools people with its fake stories.</li><li><strong>Faking News</strong>: An Indian satire site that focuses on India and its neighbors and delivers fake news as if it was real news. It is available in both English and Hindi.</li><li><strong>SatireWire</strong>: A satire site that was popular in the early 2000s and returned in 2010. It covers a wide range of topics and has won several Webby Awards.</li><li><strong>The Beaverton</strong>: A Canadian satire site that mixes satire based on reality and comedy. It covers various sections such as News, Sport, Business, Celebrities, Lifestyle, and Video.</li></ul><p><strong>Appendix 2 — Google Drive that contains all the datasets we’ve compiled/used</strong></p><p><a href="https://drive.google.com/drive/folders/1NViChhipgFROMclcJT1u26VyIQWHICZz?usp=sharing">https://drive.google.com/drive/folders/1NViChhipgFROMclcJT1u26VyIQWHICZz?usp=sharing</a></p><p><strong>Appendix 3 — Project Github Repo</strong></p><p><a href="https://github.com/GTG-Llama/Seachain">https://github.com/GTG-Llama/Seachain</a></p><p><strong>Appendix 4 — Project MVP Demo</strong></p><p>News Reliability Checker — <a href="https://youtu.be/SRR1vF44EgI">https://youtu.be/SRR1vF44EgI</a></p><p>Company News Aggregator — <a href="https://youtu.be/wABv_3XMMaM">https://youtu.be/wABv_3XMMaM</a></p><p>Similar Article Feature — <a href="https://youtu.be/v5ODg7qE2I0">https://youtu.be/v5ODg7qE2I0</a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=45290d3a30b8" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[ChatiBank: Simplifying Banking for Seniors and Differently Abled]]></title>
            <link>https://medium.com/@nusfintech.ml/chatibank-simplifying-banking-for-seniors-and-differently-abled-e4e75ccd8de5?source=rss-35fbff2bc022------2</link>
            <guid isPermaLink="false">https://medium.com/p/e4e75ccd8de5</guid>
            <dc:creator><![CDATA[Machine Learning @ NUS Fintech Society]]></dc:creator>
            <pubDate>Tue, 30 Jul 2024 11:50:16 GMT</pubDate>
            <atom:updated>2024-07-30T11:50:16.901Z</atom:updated>
            <content:encoded><![CDATA[<p>Developers: Raman, Nic, Puri, Jae, Jing Jie</p><p>Introduction:<br>The modern UIs and applications used in digital banking often pose a challenge for senior citizens and older generations due to their complexity and overwhelming nature. To address this problem, the NUS Fintech Society’s project, ChatiBank, aims to simplify the digital banking experience by providing a user-friendly and intuitive chat-only interface on top of DBS Digibank. By enabling a human-like and conversational interaction platform, ChatiBank bridges the usability gap and makes banking accessible to everyone.</p><p>Motivation and Problem:<br>Traditional approaches to digital banking lack the familiarity and simplicity that older generations were accustomed to in the past. As a result, older individuals often struggle to navigate and utilize essential services provided by banking apps. Moreover, current chatbots merely serve as retrieval tools rather than acting as true service agents. Senior citizens require a simpler and more intuitive solution to effectively interact with bank services.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/700/0*UCQaIKM9iBSLu0l-.png" /></figure><p>An old man overwhelmed by modern technology, Generated by DeepAI</p><p>Aiming for User-Friendly Interaction:<br>ChatiBank facilitates a chat-only interface that replicates the experience of speaking with a human teller in a physical bank. The goal is to empower senior citizens and provide them with all the information needed without the complexity associated with traditional UIs. Through a series of user prompts and macro-driven web bots, ChatiBank translates conversational requests into banking actions.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/700/0*-6AU5dkf3X_Zxob7.png" /></figure><p>Form Filling within a conversation using ChatiBank</p><p>Challenges:<br>1. Defining the scope of services that can be transformed into chat interactions posed challenges for our team. After careful consideration, functions such as transfers (account, PayNow, bill), as well as a knowledge portion of investments (portfolio, FDs, mutual funds), were identified as suitable candidates for adaptation into the chat-only format.</p><p>2. We also encountered obstacles when attempting to automate web banking functions using chatbots due to factors like bot detection, timeouts, and varying web drivers’ effectiveness. Eventually, the team settled on using undetected-chromedriver and Selenium as the preferred solution for accomplishing these tasks securely and effectively.</p><p>3. As ChatiBank is currently being tested in a secure local environment, partnerships with actual banks would be necessary before deploying the solution at scale. Security is a top priority, especially when considering the integration of systems like ChatGPT, and partnering with established banks would ensure the protection of sensitive user information.</p><p>Actual Implementation:<br>ChatiBank utilizes GPT-4, Selenium, and Streamlit to deliver its chat-only interface on top of DBS Digibank’s website. A combination of message passing and macro-driven web bots enables the system to interpret user queries and carry out bank functions.</p><p>When the language model requires a system action to be performed, it outputs a {ServiceWanted} tag which is parsed by the system and the respective action is carried out. Then, the conversation continues as before. We use a few-shot-prompting approach using 6 example services to help the model learn the flow of actions.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/700/0*Qhlq3F0YeiUHgYez.png" /></figure><p>Training Example: Notice the {‘ServiceWanted’} tag and interaction with ‘system’</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/700/0*Qp5SzYZyBENgKiIP.png" /></figure><p>Conversational Interface allows for dynamic interactions while keeping within professional boundaries</p><p>Choice of Model:<br>We chose GPT-4 as the underlying model for ChatiBank due to its ability to handle conversations effectively and generate accurate responses in a chat-like format.</p><p>Costs and Trade-offs:<br>The cost considerations for running ChatiBank with GPT-4’s base model included both input and output costs per conversation. With an average number of 20,000 input tokens and 500 output tokens, calculations showed an approximate expenditure of 63 cents per conversation.</p><ul><li>One API request rarely exceeds 3500 tokens, making models with longer context windows unnecessary for this project.</li><li>ALTERNATE: chatgpt-3.5-turbo-16k context is also good if one provides more examples for learning. Currently, we use 6 examples which is enough for gpt-4. There is a trade-off in speed since gpt-4 is rate-limited to 10000 tokens per minute (TPM). So an alternative approach with gpt3.5-turbo and more examples is also viable which allows for 60000TPM</li></ul><p>Conclusion:<br>ChatiBank revolutionizes digital banking with its chat-only interface catered specifically to senior citizens and individuals seeking a user-friendly banking experience. By simulating human-like interactions, ChatiBank fills the usability gap and makes modern banking accessible to all.</p><p>Next Steps:<br>1. Currently, the application supports 5 IBanking features, Transfer to PayNow, Transfer to DBS/POSB, Transfer to Other Account, PayNow topup.</p><p>2. Looking to expand into bill payments, and a more complex invest feature which could condense and present information from the DBS Invest Tab.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=e4e75ccd8de5" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Prediction of Customer Lifetime Value through BG-NBD and Gamma-Gamma Models]]></title>
            <link>https://medium.com/@nusfintech.ml/prediction-of-customer-lifetime-value-through-bg-nbd-and-gamma-gamma-models-be1f6c8ef8d4?source=rss-35fbff2bc022------2</link>
            <guid isPermaLink="false">https://medium.com/p/be1f6c8ef8d4</guid>
            <dc:creator><![CDATA[Machine Learning @ NUS Fintech Society]]></dc:creator>
            <pubDate>Tue, 30 Jul 2024 11:47:39 GMT</pubDate>
            <atom:updated>2024-07-30T11:47:39.850Z</atom:updated>
            <content:encoded><![CDATA[<p>Authors: Anthony Tamzil (Tech Lead), Chew Zi Xuan, Mario Francisco Montana, Claudeon Reinard Susanto, Ui Ogino</p><p><em>In this article, we will understand the fundamentals of the Beta-Geometric Negative Binomial Distribution and Gamma-Gamma models and how we utilized them to predict customer lifetime values based on past transaction history.</em></p><p>How does one determine the value of a customer from a business perspective? Many businesses attempt to predict the customer lifetime value (CLV) which is essential in business analytics and customer relationship management. But, what does CLV mean? Well, CLV is the net profit a company can expect to gain from a particular customer over the course of their relationship with the company. With the CLV, businesses would be able to make informed decisions and devise effective strategies to optimize customer acquisition, retention as well as whether to focus their resources on that particular customer. It is a prediction of whether the customer that walks in through the door is able to bring profits for the company.</p><p>Any company that is able to accurately make that prediction would reap the most benefit, thus, most fintech companies use models to predict the CLV and leverage on that knowledge to allocate marketing budget and invest in acquiring customers that are identified to be the most profitable. In the world of business and fintech, accurate prediction of the CLV turns a game of chance and guesswork into a game of calculated risk.</p><h3>Complication</h3><p>Why is predicting the CLV such a massive task? There are two factors that hinder the accurate prediction of CLV. Firstly, there is cost uncertainty where the acquisition cost of a customer and delivery cost of a purchase cannot be predicted exactly. Secondly, there is future purchase uncertainty which poses the biggest issue. When a customer purchases a product from the company in the past, you would not know what or whether they might purchase from said company again in the future. This could be due to changing customer needs or the customer changing supplier which would cause the company to lose a customer. Therefore, with the stated uncertainties, a probabilistic approach is most suitable in order to predict the number and value of future transactions.</p><h3>Project Overview</h3><p>In this project, we decided to take a more probabilistic approach to predicting CLV by using past customer transaction data as a base and probability to predict various factors of the CLV that when combined together, can give the businesses and users a clear CLV. CLV is essentially the product of three factors: (i) the average transaction value of a customer (ii) the number of transactions made by the customer (iii) the period of time that the customer remains an active customer of the business. With machine learning models, we hope to be able to predict the factors that make up the CLV to fill in any gaps or missing variables that businesses potentially may have.</p><p>In terms of machine learning models, there are typically five models that can be used to assist in predicting CLV:</p><ol><li>Pareto-NBD: This model determines which customers are still active and if the customers are still active, it is able to determine how often these customers would purchase from the businesses.</li><li>BG-NBD: This model predicts the expected number of transactions made within a period of time by a customer.</li><li>MBG-NBD: This model removes inconsistencies by allowing non-active customers to remain inactive.</li><li>BG-BB: This model predicts the number of repeat purchases from a single, active customer.</li><li>Gamma-Gamma: This model predicts the average transaction value per customer.</li></ol><p>Upon further discussion and research, we decided to only implement the <strong>BG-NBD</strong> and <strong>Gamma-Gamma</strong> model to assist in predicting the CLV. The BG-NBD model could be used to predict the number of transactions made by the customer while the Gamma-Gamma model could be used to predict the average transaction value of the customer. Combining these two factors, many businesses would be able to fill in the missing variable in the CLV formula and come up with a definitive CLV. Furthermore, these two models are some of the more commonly used models in the field of CLV prediction and are well-documented which would be useful especially during the model optimization and finetuning process.</p><p>As for the dataset, we have decided to use the Online Retail Dataset by UCI Machine Learning Repository which boasts a sizable 541,909 entries with the necessary fields to calculate the value of past transactions as well as essential features like the customer ID, transaction date and time.</p><h3>Beta-Geometric Negative Binomial Distribution (BG-NBD)</h3><p>The BG-NBD model, developed by Fader, Hardie and Lee in 2005, is an influential probabilistic model for describing customer behavior and predicting CLV. By modeling each customer’s purchase behavior’s distribution, the model predicts the expected number of transactions for each customer. In order for the model to work effectively, there are a couple of assumptions and limitations that had to be acknowledged before developing it.</p><h3>Limitations</h3><p>Firstly, the model only tackles one component of CLV calculation, which is the prediction of the number of purchases by the customer within a fixed period of time, and requires the assistance of other models to successfully churn out a definite CLV.</p><p>Secondly, the model is only applicable to non-contractual, continuous purchases which are the most common and challenging to predict.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/650/0*sIb4oY-oFxPHJztx" /><figcaption><em>Figure 1: Examples of different kinds of transactions</em></figcaption></figure><p>Depending on the relationships between the sellers and buyers, a business could either be a contractual business or non-contractual business. A contractual business is essentially one where the buyer-seller relationship is governed by contracts. When either party no longer wants to continue this relationship, the contract is terminated. The contract eliminates all ambiguity as to whether or not someone is, at any given point in time, an active customer. On the other hand, a non-contractual business means that purchases are on a per-need basis without any contract, making it difficult to determine whether a customer, at any given point in time, is still an active customer. A customer is considered to be “deactivated” when the customer ceases all transactions with the company after a successful transaction and never returns.</p><p>A transaction can also occur either under a continuous or discrete setting. In a continuous setting, purchases can occur at any given moment. For example, in Figure 1, a grocery purchase is considered a purchase in a continuous setting as the customer could come in at any time to conduct a transaction. In a discrete setting, purchases can occur periodically with a certain degree of regularity. For example, a magazine subscription could be considered a purchase under a discrete setting as the customer would pay a fixed fee at around the same time every month, following a pattern.</p><p>Generally, the BG-NBD model is applicable to non-contractual, continuous purchases.</p><h3>Customer Assumptions</h3><p>With regards to the customer behavior, two assumptions have to be understood during the development of the model.</p><p>Firstly, each customer has a different purchasing rate. Every customer is unique and similarly, their purchasing pattern or behavior would vary. Therefore, the model needs to be able to find a way to assign a different purchasing rate to each customer.</p><p>Secondly, each customer can deactivate at any time. The customer may never return after a purchase so the deactivation of a customer can be both latent and permanent as the deactivation would not be revealed to the company and the customer will never return after being deactivated. But how do we tell if a customer has been deactivated just by looking at the purchasing pattern?</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/379/0*y8fUV9TiTzsmdGpe" /><figcaption><em>Figure 2: The purchasing pattern of two customers A and B</em></figcaption></figure><p>From Figure 2, it can be deduced that customer A used to shope quite frequently during the first half of the timeline. However, after his last transaction, the time that has passed since then is drastically longer than his typical inter-transaction period, therefore, it is highly likely that A as a customer has been deactivated. However, B is an infrequent customer and the time passed since her last transaction is still within her inter-transaction period. Therefore, even though B’s inter-transaction period is greater than A’s, B remains an active customer while A has been deactivated.</p><h3>Poisson Process</h3><p>The BG-NBD model assumes that as long as the customer is still active, their transactions follow a Poisson process with a constant purchase rate <strong>λ</strong>. The Poisson process has two key characteristics.</p><p>Firstly, events in a Poisson process occur independently of each other, which means that the occurrence of one event does not affect the probability of the next or another event, which is applicable to the customer purchasing behavior in a continuous setting where transactions are independent events of each other.</p><p>Secondly, events occur at a constant average rate (<strong>λ</strong>). The constant rate represents the average number of events that occur in a specified time interval or region of space. In order to predict when the customer’s next transaction will be, it is assumed that there would be a certain hint of regularity to the customer’s behavior, if not, the customer’s behavior is completely random which, in reality, is not the case.</p><p>With the acknowledgement that the customer’s transactions follow a Poisson process, the time-to-next purchase (Δt) can be modeled as an exponential distribution parameterized by <strong>λ</strong>.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/474/1*FcFcTZSdDqA7yvmHsLh3Ig.png" /></figure><p>This exponential distribution function can be further elaborated through an example. Let there be two customers A and B. Customer A buys a cake every day, causing their purchasing rate <strong>λ</strong> to be 1 cake / day. The probability that their next transaction will take place within one day of their current transaction can be found by taking the area under the blue curve as shown in Figure 3 between 0 and 1, which ended up being 0.63. Customer B, on the other hand, only buys a cake every week, causing their purchasing rate <strong>λ</strong> to be 1/7 cake / day. After performing the same integration, it can be seen that the probability that their next transaction will happen within the next day is significantly lower than that of A. A similar concept is being done internally within the BG-NBD model to determine a close approximation of the time to next transaction for a customer.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/468/0*aIY5pff7TZmE6AbA" /><figcaption><em>Figure 3: Probability distribution function of two exponential distributions associated with customer A and B</em></figcaption></figure><h3>Gamma Distribution</h3><p>The BG-NBD model uses a gamma distribution to describe the variation in buying behavior across the population. After all, every customer has a different <strong>λ</strong> and in some way, each customer contributes to a store-wide <strong>λ </strong>distribution. Using gamma distribution, we would be able to model this <strong>λ</strong> distribution.</p><p>Gamma distribution is used mainly because it is not just well-studied, but it is also flexible enough to model different customer bases with different purchasing behaviors. Furthermore, since <strong>λ </strong>can only take in positive real numbers, the distribution chosen must only have positive values which suits gamma distribution.</p><p>Gamma distribution is parameterized by the shape parameter r and the scale parameter α where different combinations of these two parameters would result in the gamma distribution taking distinct shapes. With gamma distribution, we would be able to obtain a quantitative description of the collective purchasing behavior of the customer base.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/550/1*RC4PjXCkE-7hklWr9lc7LQ.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/436/0*20ug42_-RYtGTiWh" /><figcaption><em>Figure 4: Graph of two gamma distribution curves</em></figcaption></figure><p>The gamma distribution can be further elaborated on through an example. In Figure 4, the blue line depicts a downward sloping, left-leaning Gamma distribution that results from setting both r and <strong>λ </strong>to be 1. If this gamma distribution corresponds to the customer base, the heavy left skew means that a majority of the customers have a purchasing rate close to 0, which for the company would mean bad news as most of the customers do not purchase any products.</p><p>However, there is a healthier gamma distribution as portrayed by the orange line that has a peak <strong>λ </strong>at around 2, which means that a majority of the customers purchase two units of the product per day.</p><p>Overall, the combination of Poisson and Gamma distributions that are used to model the customers’ purchasing behavior is known as the Negative Binomial Distribution (NBD).</p><h3>Geometric Process</h3><p>After each purchase, a customer will have to make the decision as to whether or not to deactivate, this decision can be assigned a probability (p). The transaction after which a customer deactivates is distributed according to the shifted geometric distribution and the probability mass function of this distribution is shown below:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/604/1*HtVWG7ggTpFsV3yGYSUpOA.png" /></figure><p>If a customer deactivates after the x-th transaction, it would have meant that they must have remained active after the preceding (x-1) transactions and the probability that they remained active after each transaction is (1-p), hence, the total probability they remained active after the previous (x-1) transactions is (1-p)^(x-1).</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/468/0*sKCNLa6bmLhD8wCZ" /><figcaption><em>Figure 5: Graph of two geometric distribution curves with different probabilities of deactivation</em></figcaption></figure><p>As we can see from the graph above, the higher the probability of deactivation, the greater the probability that the deactivation will happen earlier. As the number of transactions increases, the probability of deactivation decreases. With this geometric distribution, the model is able to deal with the customer’s deactivation process.</p><h3>Beta Distribution</h3><p>Similar to <strong>λ</strong>, it is useful to associate the population of customers with a distribution of their varying probabilities. However, beta distribution is used in this case as it restricts the range of values from 0 to 1 since probabilities can only range from 0 to 1. Shown below is the probability distribution function of the beta distribution where it is parameterized by two positive shape parameters, a and b.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/456/1*A0iip2sAevPBdHjI7TlQnw.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/468/0*nQ4MiyrRQCu8mVYj" /><figcaption><em>Figure 6: Graphs of three beta distributions with varied shape parameters (a, b)</em></figcaption></figure><p>As shown in Figure 6, when the beta distribution is left-skewed (a = 1, b = 3), it suggests that most of the customers have low probability of deactivation and are not likely to deactivate early. The significance is inverted when the beta distribution is right skewed.</p><h3>Gamma-Gamma</h3><p>The Gamma-Gamma model, unlike the BG-NBD model, focuses more on the monetary aspect of the customer transaction by predicting the transaction value of a customer in the future. The Gamma-Gamma model is similar to the BG-NBD in certain ways. It assumes that the monetary value of a customer’s transactions will vary randomly around their average order value and that average order value varies across customers but does not vary that much within the same customer and at the same time, is independent of the transaction process. In order words, the model assumes that there is no relationship between the monetary value of the transaction by the customer and the purchase frequency.</p><h3>Product Development</h3><p>In addition to refining our machine learning models for CLV prediction, we also designed a web application. This web application aims to provide a simpler and more intuitive way for CLV forecasting. This was made possible through a user-friendly interface, making predictive analytics accessible to a broader audience.</p><p>Our tech stack was chosen for its practicality:</p><ul><li>Frontend: HTML, CSS, and JavaScript</li><li>Backend: Flask, Render</li></ul><p>The overall layout of our website is as follows:</p><ul><li>Home Page: Introduces the importance of CLV and its relevance to businesses</li><li>Upload Page: For users to submit transaction data in CSV format.</li><li>Download Page: After predictions, results are available in a clear format, linking CustomerID with their CLV values</li><li>Feedback Page: For users to send feedback for any changes they want to our website’s functionality</li><li>User Guide: A practical and comprehensive manual which helps users navigate our website</li></ul><h3>Challenges Faced</h3><p>[Mario] ​​The biggest challenge in doing this project was the transition from my usual Jupyter Notebook-centric workflow to integrating a web application. It introduced a significant learning curve as I had primarily worked locally before. Learning Flask and web development was something new to me, but it did make me ponder the potential of combining machine learning with user-friendly web elements for a more practical experience.</p><p>[Claudeon] Personally, the most challenging part of the project was researching and implementing the BG-NBD model as it relies on statistical methods that I still do not fully understand. Nevertheless, after reading up more on the model through articles and documentations, the main gist of the model was made much clearer. The documentation also provided methods to evaluate the accuracy of the model by splitting the dataset into training and validation set, which makes it easier to do hyperparameter tuning.</p><p>[Zi Xuan] The most challenging part for me revolved around understanding the models and discovering what we could do with them. With limited resources available online, we had to delve deep into the statistical theories underlying these models to apply them effectively. This required not only a thorough understanding of the customer behavior assumptions they are based on but also a substantial amount of trial and error in fine-tuning the models for our specific dataset.</p><h3>References</h3><p>Baquar, Z. (2022, October 4). <em>How to predict customer lifetime value using machine learning in Python</em>. Medium. <a href="https://blog.devgenius.io/how-to-predict-customer-lifetime-value-using-machine-learning-in-python-4066344d0ab0">https://blog.devgenius.io/how-to-predict-customer-lifetime-value-using-machine-learning-in-python-4066344d0ab0</a></p><p>Baysan. (2021, September 17). <em>Customer life time value prediction by using BG-NBD &amp; gamma-gamma models and applied example in…</em> Medium. <a href="https://medium.com/analytics-vidhya/customer-life-time-value-prediction-by-using-bg-nbd-gamma-gamma-models-and-applied-example-in-997a5ee481ad">https://medium.com/analytics-vidhya/customer-life-time-value-prediction-by-using-bg-nbd-gamma-gamma-models-and-applied-example-in-997a5ee481ad</a></p><p><em>Lifetimes package</em>. lifetimes package — lifetimes 0.11.2 documentation. (n.d.). <a href="https://lifetimes.readthedocs.io/en/latest/lifetimes.html#module-lifetimes.utils">https://lifetimes.readthedocs.io/en/latest/lifetimes.html#module-lifetimes.utils</a></p><p>Matylda. (2023, February 8). <em>Customer lifetime value prediction: The key to higher profits and growth</em>. Spyrosoft. <a href="https://spyro-soft.com/blog/ai-use-case-customer-lifetime-value-prediction#:~:text=Fintech%20companies%20can%20drive%20up,business%20strategies%20of%20fintech%20companies">https://spyro-soft.com/blog/ai-use-case-customer-lifetime-value-prediction#:~:text=Fintech%20companies%20can%20drive%20up,business%20strategies%20of%20fintech%20companies</a></p><p><em>Predict customer lifetime value with machine learning</em>. Idiomatic. (2023, August 21). <a href="https://idiomatic.com/blog/customer-lifetime-value-machine-learning/">https://idiomatic.com/blog/customer-lifetime-value-machine-learning/</a></p><p>Savci, U. (2022, April 5). <em>Customer lifetime value prediction in python</em>. Medium. <a href="https://medium.com/@ugursavci/customer-lifetime-value-prediction-in-python-89e4a50df12e">https://medium.com/@ugursavci/customer-lifetime-value-prediction-in-python-89e4a50df12e</a></p><p>shailaja4247. (2020, April 7). <em>Customer lifetime value prediction</em>. Kaggle. <a href="https://www.kaggle.com/code/shailaja4247/customer-lifetime-value-prediction">https://www.kaggle.com/code/shailaja4247/customer-lifetime-value-prediction</a></p><p>Yao, W. (2021, April 26). <em>An RFM customer segmentation with python</em>. Medium. <a href="https://towardsdatascience.com/an-rfm-customer-segmentation-with-python-cf7be647733d">https://towardsdatascience.com/an-rfm-customer-segmentation-with-python-cf7be647733d</a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=be1f6c8ef8d4" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[A comparison of various classifiers’ performance at credit scoring for P2P loans as an ensemble…]]></title>
            <link>https://medium.com/@nusfintech.ml/a-comparison-of-various-classifiers-performance-at-credit-scoring-for-p2p-loans-as-an-ensemble-c79cb1e67d33?source=rss-35fbff2bc022------2</link>
            <guid isPermaLink="false">https://medium.com/p/c79cb1e67d33</guid>
            <dc:creator><![CDATA[Machine Learning @ NUS Fintech Society]]></dc:creator>
            <pubDate>Mon, 08 Jul 2024 14:14:34 GMT</pubDate>
            <atom:updated>2024-07-08T14:14:34.341Z</atom:updated>
            <content:encoded><![CDATA[<h3><strong>A comparison of various classifiers’ performance at credit scoring for P2P loans as an ensemble learning hard voting classifier</strong></h3><p><a href="https://www.linkedin.com/in/yw-chua-41a251279/">Koby Chua</a> (Tech Lead), <a href="https://www.linkedin.com/in/wen-yeong-lee-979a01256/">Lee Wen Yeong</a> (Neural Network Lead), <a href="https://www.linkedin.com/in/jerryyangg/">Jerry Yang</a> (EDA Lead), <a href="https://www.linkedin.com/in/sparsh-kumar-3bb643200/">Sparsh Kumar</a> (Random Forest co-lead), <a href="https://www.linkedin.com/in/yang-yee-a88a9821a/">Yang Yee</a> (Random Forest co-lead), <a href="https://www.linkedin.com/in/pehtingxuan/">Peh Ting Xuan </a>(Logistic Regression co-lead), <a href="https://www.linkedin.com/in/lionel-see/">Lionel See</a> (Logistic Regression co-lead), <a href="https://www.linkedin.com/in/jingjie-tan">Tan Jing Jie</a> (Neural Network &amp; EDA co-lead), Jolynn Wee (writer)</p><p>Link to source code: <a href="https://github.com/PopKobs/P2P_credit_scoring">https://github.com/PopKobs/P2P_credit_scoring</a></p><p><strong>P2P lending</strong></p><p>Peer-to-Peer (P2P) lending, also known as “Social lending”, is an alternative financing method that has become increasingly popular with the rise of digital platforms that connect borrowers directly with lenders, bypassing traditional financial institutions like banks. Examples of P2P lending platforms that have gained prominence and recognition in recent years include SoFi, Prosper, and LendingClub.</p><p>The P2P lending process typically begins with borrowers creating loan requests detailing the amount needed, purpose, and desired interest rate. These requests are then listed on the platform, where individual lenders can review and choose to fund them. Once funded, the borrower receives the loan amount and begins making repayments, including both principal and interest, according to the agreed-upon terms. Meanwhile, lenders receive returns on their investment as borrowers repay the loan. P2P lending websites often employ algorithms to match borrowers with lenders efficiently while providing transparency and security throughout the transaction process.</p><p>What sets the P2P lending model apart from traditional lending models is its ability to democratize access to credit. For borrowers, especially those deemed high-risk by banks due to poor credit scores, P2P lending opens doors that were previously shut. Borrowers with good credit scores turn to P2P lending to borrow at a lower interest rate than bank rates. Conversely, for lenders, it presents an opportunity to diversify their investment portfolios while potentially reaping returns that surpass interest earned from conventional instruments like bonds.</p><p>The focus of this project would be on the credit scoring algorithms employed by P2P lending websites. Credit scoring is often the final step before a borrower’s loan request is listed on the platform for lenders, often along with the borrower’s credit profile. The borrower’s credit score serves to help lenders make the decision whether to fund all or a portion of the loan to diversify their risk. These algorithms assess the borrower’s creditworthiness with financial indicators like credit scores, loan term, employment, number of loans defaulted, number of criminal charges, and more.</p><p><strong>Research Question</strong></p><p>A good credit scoring algorithm is integral to the success of a P2P lending platform, as it directly impacts investors’ confidence and, subsequently, the platform’s share price. Ideally, it should be able to warn lenders that a borrower has a very poor score despite laws prohibiting the classification of these borrowers as a high default risk. An example of such a law is the “Fresh Start” initiative launched in April 2022 by the Department of Education (DOE) in the US which helps student borrowers in default. On their credit reports, their loan status would be listed as “current” instead of “in collections”, allowing students with very poor credit scores to continue taking loans.</p><p>With a dataset from a realistic P2P loan platform, we want to build a credit scoring model that closely follows the proprietary credit scoring models of platforms like Prosper and SoFi and improves on what these platforms have done. Our model seeks to be more accurate in evaluating loan data and predicting the likelihood of a borrower’s complete repayment of the loan, to better help lenders make decisions about whether to lend to a borrower or the optimal amount to lend a borrower.</p><p>In this project, we compared the accuracy of random forests, neural networks, and logistic regression models. We would then employ a watered-down ensemble learning strategy to tackle the problem.</p><p><strong>Random Forests</strong> operate by constructing multiple decision trees during training and outputting the mode of the classes (classification) or mean prediction (regression) of the individual trees. This approach leverages the concept of bagging (an ensemble learning method), where multiple versions of a predictor are trained on random subsets of the data and then combined to produce a single output. The key advantage of random forests is their ability to reduce overfitting and increase predictive accuracy by averaging out the errors of individual trees.</p><p><strong>Neural Networks</strong>, especially deep learning models, are powerful in capturing complex, non-linear patterns through multiple layers of neurons. These models excel in tasks involving complex data structures and relationships in the data, making it a strong candidate for credit scoring, where borrower behavior can be influenced by numerous, often interrelated factors.</p><p><strong>Logistic Regression </strong>is a statistical method used for binary classification problems, where the outcome variable is categorical with two possible outcomes. The model estimates the probability that a given input belongs to a particular class by using the logistic function.</p><p><strong>Ensemble learning</strong> is a technique that combines multiple models to produce a single, more accurate and robust model.</p><p>The main idea is that by aggregating the predictions of the various models, the simple ensemble that we create could achieve better performance in predicting a borrower’s repayment of a loan than any model could on its own.</p><p><strong>Feature engineering &amp; preprocessing</strong></p><p>Our feature engineering processes attempted to find out the most salient features of our dataset, which was obtained from <a href="https://www.kaggle.com/datasets/henryokam/prosper-loan-data">https://www.kaggle.com/datasets/henryokam/prosper-loan-data</a>. This dataset covers P2P loan transactions on the P2P lending platform Prosper for the years 2009 to 2017. During these years, the American economy, which is where Prosper solely conducts business, was emerging from a global recession from 2009 to 2010. For all other years, the US economy experienced moderate economic growth not exceeding 3%. During 2009 to 2017, the Federal Reserve conducted a program of ‘quantitative easing’ which resulted in lending for all purposes being extremely easy. It is important to note that because P2P lending is viewed by lenders as somewhat of an investment, &amp; borrowers who use P2P lending services are deemed by traditional banks to be of higher credit risk, outside of their tolerance. Therefore, the interest rates of loans on a P2P lending platform will definitely be higher than those at commercial banks.</p><p>The original dataset has 81 variables, &amp; using a process of expert selection, we narrowed down a list of 37 features deemed to be especially relevant to the focus of the research on helping models focus on features that have the highest predictive value on whether a borrower is likely to default on his loan requests or not. This would make each model’s performance measure to be its effectiveness at predicting ‘LoanStatus’. This variable originally has the values Cancelled, Chargedoff, Completed, Current, Defaulted, FinalPaymentInProgress, PastDue. However, as our task is more focused on predicting whether a borrower is likely to repay (1) his loan or not (0), we can perform the following binarisation transformation:</p><pre>data_filtered[&#39;LoanStatus&#39;] = data_filtered[&quot;LoanStatus&quot;].apply(lambda x: 1 if x in[&quot;Completed&quot;, &quot;Current&quot;, &quot;FinalPaymentInProgress&quot;] else 0) </pre><p>The statistical features that our group was interested in was pairwise correlation through bivariate calculations before deeming a feature useful enough to be retained for later advanced feature engineering. Below is an example plot we generated to explore a small subset of features to see which had the highest bivariate correlation with each other.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/622/1*yF0fHXVRK3eKhthRpwii4g.png" /><figcaption>Figure 1</figcaption></figure><p>This list of 37 features was later fed into a process of Mutual information (MI) &amp; later Principal Component Analysis (PCA) to help further condense the list of variables to those that can help the models converge faster &amp; more accurately. This was especially vital for the neural network model being constructed.</p><p>The key concept of MI is that we measure the dependency between two variables. In the context of feature selection, mutual information quantifies the amount of information obtained about one variable through another variable. This requires information about the individual &amp; joint probability distributions of variables X &amp; Y, where X &amp; Y are the variables being analysed pairwise in MI. MI’s benefit to our research comes from its ability to handle non-linearity &amp; feature independence effectively. Also, MI makes no assumption on the distributions of any feature. This is important especially in handling the early process of feature engineering where we will not have enough time to analyze all variables in detail. The figure below shows a plot of feature importance against features shown as a bar chart plotted at the conclusion of MI.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/912/1*ZaLoXuXK1eMIAr36M-ZlTw.png" /><figcaption>Figure 2</figcaption></figure><p>This leads us to corroborate that the features in the original dataset that relate to those of interest rates that the borrower lends for (‘BorrowerAPR’, ‘BorrowerRate’) &amp; the interest rates that the lender is receiving for lending his money (‘LenderYield’) were those of sufficient importance at the early stage of the feature engineering process. Hence, their retention was useful. A surprising observation following the MI process was that the feature ‘LoanCurrentDaysDelinquent’ accounts for over 30% of the feature importance.</p><p>Eventually, the variables selected to be used by all models were those that had feature importance above 0.01. This creates a list of 17 variables that we train each component model on.</p><p>Here are the 17 selected variables:</p><p>● ‘BorrowerAPR’</p><p>● ‘BorrowerRate’</p><p>● ‘LenderYield’</p><p>● ‘EstimatedEffectiveYield’</p><p>● ‘EstimatedLoss’</p><p>● ‘EstimatedReturn’</p><p>● ‘ProsperScore’</p><p>● ‘IsBorrowerHomeowner’</p><p>● ‘IncomeRange’</p><p>● ‘IncomeVerifiable’</p><p>● ‘StatedMonthlyIncome’</p><p>● ‘LoanCurrentDaysDelinquent’</p><p>● ‘LoanOriginalAmount’</p><p>● ‘LP_CustomerPayments’</p><p>● ‘LP_CustomerPrincipalPayments</p><p>● ‘PercentFunded’</p><p>● ‘Investors’</p><p><strong>Logistic Regression</strong></p><p>The Logistic Regression model used a subset of variables, ‘EmploymentStatus’, ‘OpenRevolvingAccounts’, ‘LoanOriginalAmount’, ‘DebtToIncomeRatio’, ‘BorrowerRate’, ‘AvailableBankcardCredit’, ‘ProsperRating (numeric)’, as the feature variables, as they were identified to have greater correlation with the target variable, and are thus more relevant for prediction.</p><p>The target variable, ‘rating’, is a binary classification whereby customers with an above average CreditScore has a value of 1, while those with a below average CreditScore were given a value of 0.</p><p>Further feature engineering was done on the ‘EmploymentStatus’ variable, to convert the values from String to categorical values using a label encoder.</p><p>Missing values in the dataset were replaced with 0 using an imputer.</p><p>In the original dataset, values in certain features were significantly larger than others. We performed Feature Scaling using the StandardScaler() class to standardise the feature variables. This facilitated meaningful comparison between the feature variables, while preserving the meaningful relationship between the data entries.</p><p>We split the dataset into training and testing sets using a 75–25 split, respectively.</p><p>Our initial implementation obtained the following results. Accuracy score: 0.6604177637352993</p><p>Precision score: 0.9290795909293019</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/936/1*R1mnj4RPB2u1fNBbZ0iH0w.png" /><figcaption>Figure 3</figcaption></figure><p>Next, we performed hyperparameter tuning to obtain the best combination of hyperparameters that optimised the model performance as measured by accuracy.</p><pre>param_grid = {<br>&#39;C&#39;: [0.01, 0.1, 1, 10, 100],<br>&#39;penalty&#39;: [&#39;l1&#39;, &#39;l2&#39;, &#39;none&#39;, &#39;elasticnet&#39;]<br>}<br>grid_search = GridSearchCV(model,<br>param_grid, cv = 5,<br>scoring= &#39;accuracy&#39;<br>)</pre><p>Using the following parameter grid, the GridSearchCV trains and tests our Logistic Regression model for each possible combination of hyperparameters provided, and further performs cross-validation for each model tested, returning the best possible model.</p><p>Best Parameters: {‘C’: 1, ‘penalty’: ‘l2’}</p><p>Best Score: 0.7353362371936507</p><p>Through hyper-parameter tuning, we successfully raised our accuracy from 0.6604177637352993 to 0.7353362371936507.</p><p>Lastly, we plotted a Receiver Operating Characteristic curve (ROC curve) to visualise the performance of our regression model at all classification thresholds. This curve plots the relationship between the True Positive Rate and False Positive Rate. Using the roc_auc_score() function, we determined the optimal classification threshold to be 0.828623303233121. Upon re-running the model using the updated classification threshold we further improved our accuracy score to 0.7353362371936507.</p><pre>pred = (model.predict_proba(X_test)[:,1] &gt;= 0.828623303233121).astype(bool)</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/630/1*lpH8iMyaudgQHtyYjZpdZQ.png" /><figcaption>Figure 4</figcaption></figure><p><strong>Neural Network</strong></p><p>In this section, we will walk through the process of splitting data into training, validation, and test sets, standardising the data, and applying Principal Component Analysis (PCA) for dimensionality reduction. This approach helps in handling different scales and variances in the data effectively, leading to better model performance and faster convergence.</p><p>For the neural network model, we started off with Feature Engineering of the 17 variables selected earlier.</p><p>For ‘IncomeRange’, we mapped the income ranges to 7 different levels.</p><pre>income_mapping = {<br>&#39;Not employed&#39;: 0, &#39;$0&#39;: 1,&#39;$1–24,999&#39;: 2, &#39;$25,000–49,999&#39;: 3,<br>&#39;$50,000–74,999&#39;: 4,&#39;$75,000–99,999&#39;: 5, &#39;$100,000+&#39;: 6<br>}<br>df[&#39;IncomeRange&#39;] = df[&#39;IncomeRange&#39;].map(income_mapping)</pre><p>For ‘IsBorrowerHomeowner’ and ‘IncomeVerifiable’, we converted it into an integer datatype, where 1 is true.</p><p>Splitting the Data:</p><p>We proceeded to split the dataset into features (x_df) and the target variable (y_df). Here, we drop the target column, LoanComplete, from the original DataFrame to create our feature set.</p><pre># Split the data into features and target<br>x_df = df.drop(columns=[&quot;LoanComplete&quot;])<br>y_df = df[&quot;LoanComplete&quot;]</pre><p>Next, we split the data into training, validation, and test sets. This is crucial for evaluating the model’s performance and ensuring it generalises well to unseen data. We use an 70–15–15 split, with 70% of the data for training, 15% for validation, and 15% for testing.</p><pre># Split the data into training, validation, and test sets<br>X_train, X_temp, y_train, y_temp = train_test_split(x_df, y_df, test_size=0.3, random_state=42)<br>X_val, X_test, y_val, y_test = train_test_split(X_temp, y_temp, test_size=0.5, random_state=42)</pre><p>Standardising the Data:</p><p>Machine learning algorithms often perform better when the input features are on a similar scale. Standardisation is a common preprocessing step that transforms the data to have a mean of 0 and a standard deviation of 1. This process helps in handling different scales and variances in the data.</p><p>Choosing between normalisation and standardisation to preprocess data were a key consideration as we could only choose one.</p><p>Normalisation:</p><p>Normalisation scales the data to a range of [0, 1] or [-1, 1]. It is often used when you do not assume any distribution of your data. This is particularly useful when the features are on different scales but have a similar range.</p><p>Standardisation:</p><p>Standardisation scales the data to have a mean of 0 and a standard deviation of 1. This method assumes that your data is normally distributed (Gaussian distribution). Standardisation is often more appropriate when the features have different scales and variances.</p><p>Looking at our data, loan amounts and incomes could have very large ranges and different variances compared to features like IsBorrowerHomeowner and IncomeVerifiable, which are binary.</p><p>Standardisation would be better as our features have different scales. Furthermore, for many machine learning algorithms, especially those involving gradient descent, standardisation is generally preferred because it often leads to better performance and faster convergence.</p><p>Given this information, standardisation is typically a good default choice, especially when the data ranges and variances are significantly different.</p><pre># Standardise the data<br>scaler = StandardScaler()<br>X_train_scaled = scaler.fit_transform(X_train)<br>X_val_scaled = scaler.transform(X_val)<br>X_test_scaled = scaler.transform(X_test)</pre><p>In the code above, we fit the StandardScaler on the training data and then apply the same transformation to the validation and test sets. This ensures that all sets are scaled consistently.</p><p>Applying Principal Component Analysis (PCA):</p><p>Dimensionality reduction is an essential technique as we are dealing with high-dimensional data. PCA is a powerful tool that reduces the dimensionality of the data while preserving most of the variance. This helps in reducing computational cost and avoiding the curse of dimensionality.</p><pre># Apply PCA<br>pca = PCA(n_components=0.95)<br>X_train_pca = pca.fit_transform(X_train_scaled)<br>X_val_pca = pca.transform(X_val_scaled)<br>X_test_pca = pca.transform(X_test_scaled)</pre><p>In this code snippet, we apply PCA to the standardised training data, retaining 95% of the variance. The resulting principal components are then used to transform the validation and test sets.</p><p>Evaluating Explained Variance Ratio:</p><p>It’s important to check how much variance is explained by the selected principal components. This gives us an idea of how well the dimensionality reduction preserves the original data’s information.</p><pre># Check explained variance ratio sum<br>explained_variance_ratio_sum = pca.explained_variance_ratio_.sum()</pre><p>In this step, we print the sum of the explained variance ratio to ensure that our PCA transformation retains the desired amount of variance (95% in this case).</p><p>From the sum of explained variance, we can determine that we should retain around 10 principal components for our PCA.</p><p>We introduce 3 different models, each with improvements over the previous in order to achieve as high accuracy and precision as we can. We also utilise tensorboard to visualise the validation accuracy and validation loss at each epoch.</p><p><strong>Model 1:</strong></p><p>A simple feedforward neural network for binary classification is used.</p><pre># Build the neural network<br>model = Sequential([<br>Dense(64, activation=&#39;relu&#39;, input_shape=(X_train.shape[1],)),<br>Dense(32, activation=&#39;relu&#39;),<br>Dense(1, activation=&#39;sigmoid&#39;) # Binary classification, so using sigmoid activation<br>])<br>#Logging the results for the Tensorboard<br>log_dir_model_1 = &quot;logs/fit/model1/&quot; + datetime.datetime.now().strftime(&quot;%Y%m%d-%H%M%S&quot;)<br>tensorboard_callback = tf.keras.callbacks.TensorBoard(log_dir=log_dir_model_1, histogram_freq=1)<br># Compile the model<br>model.compile(optimizer=&#39;adam&#39;, loss=&#39;binary_crossentropy&#39;, metrics=[&#39;accuracy&#39;, f1_metric])<br># Train the model<br>history = model.fit(X_train_pca, y_train, epochs=10, batch_size=32, validation_data=(X_val_pca, y_val), callbacks=[tensorboard_callback])</pre><p>After training it for only 10 epochs, the model achieved an accuracy 99.40% with an F1 accuracy of 99.6%.</p><blockquote>1857/1857 [==============================] — 5s 3ms/step — loss: 0.0268 — accuracy: 0.9939 — f1_metric: 0.9966 — val_loss: 0.0273 — val_accuracy: 0.9937 — val_f1_metric: 0.9965</blockquote><blockquote>After evaluating it on the test set, it achieved a score of 99.4% and 99.6%.</blockquote><blockquote>398/398 [==============================] — 1s 2ms/step — loss: 0.0289 — accuracy: 0.9940 — f1_metric: 0.9967</blockquote><p>Test loss, test accuracy, test F1: [0.028877118602395058, 0.9940289258956909, 0.9966542720794678]</p><p><strong>The tensorboard below shows the training and validation accuracy of Model 1 over 10 epochs:</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/966/1*zcZrUxN4nF9Eaqu8Ds1ZKA.png" /><figcaption>Figure 5</figcaption></figure><p><strong>The tensorboard below shows the training and validation Loss of Model 1 over 10 epochs:</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/966/1*7WTH__4j5HAO0jFGulnSYg.png" /><figcaption>Figure 6</figcaption></figure><p><strong>The tensorboard below shows the training and validation F1 score of Model 1 over 10 epochs:</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/966/1*JnDKaZJAOHqXPD0IAkJCzQ.png" /><figcaption>Figure 7</figcaption></figure><p>From the Tensorboard graph, there is evidence of overfitting. Overfitting occurs when a model performs well on the training data but does not generalise well to unseen data.</p><p>The training loss and accuracy continue to improve, whereas the validation loss stops improving and starts to fluctuate after a few epochs. This divergence between training and validation performance suggests that the model is starting to memorise the training data rather than learning general patterns. Although the validation accuracy remains high, the fluctuations in validation loss suggest that the model’s performance on unseen data is less stable, indicating potential overfitting.</p><p><strong>Model 2:</strong></p><p>As a result of overfitting, Model 1 becomes too specialised in the training data, failing to generalise well to new, unseen cases. We will build a new model, Model 2 and employ several strategies to mitigate this:</p><ol><li><strong>Simplified Architecture:</strong> We opted for a smaller neural network architecture. This means fewer layers and neurons, reducing the model’s capacity to memorise noise in the training data.</li><li><strong>Regularisation:</strong> We incorporated L2 regularisation (also known as Ridge Regression). This technique adds a penalty to the model’s weights, discouraging overly complex parameter values and promoting a smoother decision boundary.</li><li><strong>Dropout:</strong> Dropout layers randomly “deactivate” a fraction of neurons during each training step. This prevents co-adaptation of neurons and encourages the network to learn more robust features.</li><li><strong>Early Stopping:</strong> By monitoring the model’s performance on a validation dataset, we halt training when the validation loss plateaus or starts to increase. This avoids unnecessary iterations that could lead to overfitting.</li><li><strong>Smaller Batch Size:</strong> Reducing the batch size during training can introduce a degree of randomness and make it harder for the model to settle into overly specific patterns.</li></ol><p>Hence, by incorporating the above features into our code:</p><pre>from tensorflow.keras.layers import Dropout<br>from tensorflow.keras import regularizers<br><br># Build the neural network with dropout layers and reduced complexity<br>model2 = Sequential([<br>Dense(32, activation=&#39;relu&#39;, input_shape=(X_train.shape[1],)),<br>Dropout(0.2), # Add dropout layer with 20% dropout rate<br>Dense(16, activation=&#39;relu&#39;, kernel_regularizer=regularizers.l2(0.01)), # Add L2 regularization<br>Dropout(0.2), # Add another dropout layer<br>Dense(1, activation=&#39;sigmoid&#39;) # Binary classification, so using sigmoid activation<br>])<br><br>#Logging the results for the Tensorboard<br>log_dir_model_2 = &quot;logs/fit/model2/&quot; + datetime.datetime.now().strftime(&quot;%Y%m%d-%H%M%S&quot;)<br>tensorboard_callback = tf.keras.callbacks.TensorBoard(log_dir=log_dir_model_2, histogram_freq=1)<br>model2.compile(optimizer=tf.keras.optimizers.Adam(learning_rate=0.001),<br>loss=&#39;binary_crossentropy&#39;,<br>metrics=[&#39;accuracy&#39;, f1_metric])<br><br># Implement early stopping<br>early_stopping = tf.keras.callbacks.EarlyStopping(monitor=&#39;val_loss&#39;, patience=3, restore_best_weights=True)<br><br># Train the model with reduced batch size and early stopping<br>history2 = model2.fit(X_train, y_train, epochs=14, batch_size=16, validation_data=(X_val, y_val), callbacks=[early_stopping,tensorboard_callback])</pre><p>After training it for only 8 epochs, the model achieved an accuracy 98.6% with an F1 accuracy of 99.2%.</p><blockquote>Epoch 10/14</blockquote><blockquote>3713/3713 [==============================] — 14s 4ms/step — loss: 0.0881 — accuracy: 0.9839 — f1_metric: 0.9907 — val_loss: 0.0694 — val_accuracy: 0.9864 — val_f1_metric: 0.9922</blockquote><blockquote>After evaluating it on the test set, it achieved a score of 98.9% and 99.3%, slightly lower than that of model 1.</blockquote><blockquote>398/398 [==============================] — 1s 2ms/step — loss: 0.0520 — accuracy: 0.9891 — f1_metric: 0.9939</blockquote><p>test loss, test acc, test f1: [0.05201537534594536, 0.9890791773796082, 0.9938859939575195]</p><p><strong>The tensorboard below shows the training and validation accuracy of Model 2 over 10 epochs:</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/966/1*AMqpOFBWlGHZ9xbsGvS1QA.png" /><figcaption>Figure 8</figcaption></figure><p><strong>The tensorboard below shows the training and validation Loss of Model 2 over 10 epochs:</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/966/1*O7MwY9sM1eWTIbnTsopsbA.png" /><figcaption>Figure 9</figcaption></figure><p><strong>The tensorboard below shows the training and validation F1 score of Model 2 over 10 epochs:</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/966/1*dzyGsQj2fE1ejxY1X60K8g.png" /><figcaption>Figure 10</figcaption></figure><p>Furthermore, plotting the graph between the training loss and validation loss, the validation loss seems to be below the training loss as opposed to the previous model.</p><p><strong>Model 3:</strong></p><p>This model introduces several refinements that aims to improve the stability of training, accelerate convergence, and potentially boost predictive accuracy. The methods include:</p><ol><li><strong>Batch Normalization:</strong> Stabilizes training and enables faster convergence.</li><li><strong>Learning Rate Scheduling:</strong> Dynamically adjusts the learning rate for optimal performance.</li><li><strong>Expanded Architecture:</strong> Increased capacity for learning complex patterns, balanced by regularization techniques.</li><li><strong>L2 Regularization:</strong> Promotes smoother decision boundaries and prevents overly complex models.</li><li>(On L2 Regularization: Ridge regression is a technique that discourages complex models by adding a penalty to large model coefficients, leading to simpler models with better generalization to new data.)</li><li><strong>Lowered Dropout Rate:</strong> Preserves more information in the network, particularly useful with limited data.</li></ol><pre>from tensorflow.keras.layers import Dropout<br>from tensorflow.keras.layers import BatchNormalization<br>from tensorflow.keras.layers import Activation<br>from tensorflow.keras import regularizers<br><br># Build the neural network with dropout layers and reduced complexity<br>model3 = Sequential([<br>Dense(64, input_shape= (X_train.shape[1],)),<br>BatchNormalization(),<br>Activation(&#39;relu&#39;),<br>Dropout(0.1),<br>Dense(32, kernel_regularizer= regularizers.l2(0.005)),<br>BatchNormalization(),<br>Activation(&#39;relu&#39;),<br>Dropout(0.2), # Add dropout layer with 20% dropout rate<br>Dense(16,kernel_regularizer = regularizers.l2(0.005)), # Add L2 regularization<br>BatchNormalization(),<br>Activation(&#39;relu&#39;),<br>Dropout(0.2), # Add another dropout layer<br>Dense(1, activation=&#39;sigmoid&#39;) # Binary classification, so using sigmoid activation<br>])<br><br>#Logging the results for the Tensorboard<br>log_dir_model_3 = &quot;logs/fit/model3/&quot; + datetime.datetime.now().strftime(&quot;%Y%m%d-%H%M%S&quot;)<br>tensorboard_callback = tf.keras.callbacks.TensorBoard(log_dir=log_dir_model_3, histogram_freq=1)<br>model3.compile(optimizer=tf.keras.optimizers.Adam(learning_rate=0.001),<br>loss=&#39;binary_crossentropy&#39;,<br>metrics=[&#39;accuracy&#39;, f1_metric])<br><br># Implement early stopping<br>early_stopping = tf.keras.callbacks.EarlyStopping(monitor=&#39;val_loss&#39;, patience=3, restore_best_weights=True)<br><br># Learning rate scheduler<br>from tensorflow.keras.callbacks import LearningRateScheduler<br>def scheduler(epoch, lr): # basic LR<br>if epoch &lt; 10:<br>return lr<br>else:<br>return lr * tf.math.exp(-0.1)<br>callback = LearningRateScheduler(scheduler)<br><br># Train the model with reduced batch size and early stopping<br>history3 = model3.fit(X_train, y_train, epochs=10, batch_size=16, validation_data=(X_val, y_val), callbacks=[callback,early_stopping])</pre><p>After training it for only 8 epochs, the model achieved an accuracy 97.9% with an F1 accuracy of 98.8%.</p><blockquote>Epoch 8/10</blockquote><blockquote>3713/3713 [==============================] — 18s 5ms/step — loss: 0.0700 — accuracy: 0.9858 — f1_metric: 0.9917 — val_loss: 0.0767 — val_accuracy: 0.9791 — val_f1_metric: 0.9881 — lr: 0.0010</blockquote><blockquote>After evaluating it on the test set, Model 3 achieved a score of 98.6% and 99.2%, with the accuracy being slightly lower than that of model 2.</blockquote><blockquote>398/398 [==============================] — 1s 2ms/step — loss: 0.0640 — accuracy: 0.9859 — f1_metric: 0.9921</blockquote><p>test loss, test acc, test f1: [0.06395098567008972, 0.9859365224838257, 0.99212247133255]</p><p><strong>The tensorboard shows the training and validation accuracy of Model 3 over 10 epochs:</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/966/1*KQgCo4eEk9Z4vfUK1_DHHg.png" /><figcaption>Figure 11</figcaption></figure><p><strong>The tensorboard shows the training and validation Loss of Model 3 over 10 epochs:</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/966/1*VaKIc0RG-OYDGW4l30P5Lg.png" /><figcaption>Figure 12</figcaption></figure><p><strong>The tensorboard shows the training and validation F1 score of Model 3 over 10 epochs:</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/966/1*Wb9gDiLdXIikQrboujNq0A.png" /><figcaption>Figure 13</figcaption></figure><p>In conclusion, the iterative refinement of neural network models for credit scoring, as demonstrated by Model 3, underscores the importance of precision and meticulous fine-tuning in achieving maximum performance. Model 3’s peak F1 score is 99.6%, suggesting that efforts to improve its performance through the improvements to model 1 &amp; 2 were very effective. By strategically incorporating techniques like Batch Normalization, learning rate scheduling, and a balanced approach to regularization, we can create models that not only learn complex patterns in data but also generalize effectively to unseen cases. However, it is important to note that there are many different ways to create models and that not one model is suitable.</p><p><strong>Random Forest</strong></p><p>The random forest model uses a smaller subset of the 17 variables selected from the feature engineering. The condensed list of 7 variables, ‘EmploymentStatus’, ‘CreditScoreRangeLower’, ‘CreditScoreRangeUpper’, ‘LoanOriginalAmount’, ‘IsBorrowerHomeowner’, ‘CurrentDelinquencies’, ‘AmountDelinquent’, was used after experimentation to attempt to improve model performance measured by F1 score first &amp; accuracy second.</p><p>Further feature engineering was done for 2 of these 7 variables. For ‘IsBorrowerHomeowner’, we converted it into an integer datatype, where 1 is true.</p><p>For ‘EmploymentStatus’, a binary classification treatment similar to ‘LoanStatus’ was done:</p><pre>data_filtered[&#39;EmploymentStatus&#39;] = data_filtered[&#39;EmploymentStatus&#39;].apply(lambda x: 1 if x in [&#39;Employed&#39;, &#39;Full-time&#39;, &#39;Self-employed&#39;, &#39;Part-time&#39;] else 0)</pre><p>We tested model performance by noting error rates as the number of trees (classifiers) in the model increased in steps of ten using the following algorithm:</p><ol><li>Use the RandomForestClassifier from the sklearn.ensemble module to train a Random Forest</li><li>Train the model with different numbers of trees (from 30 to 500, in steps of 10). The out-of-bag (OOB) error and test error are recorded for each model.</li><li>For each number of trees, a RandomForestClassifier model is created and trained on a training dataset (x_train, y_train).</li><li>The OOB error and test error are calculated for each model. The OOB error is the error rate of the model on the training set, while the test error is the error rate of the model on the test set (x_test, y_test). These errors are appended to the respective lists.</li></ol><p>With the above algorithm, another variation was produced to study the effect of different numbers of features used for splitting a node. This variation uses a list, max_features with 3 elements: ‘sqrt’, half the number of features in the training set, and the total number of features in the training set. Hence, we are able to visualise the error rates by varying the max feature (m) in splitting a node. Clearly, in the figure below, by keeping m at its maximum, the lowest test classification errors are seen.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/946/1*_ZF01uwlKIRt6J4fgDYedA.png" /><figcaption>Figure 14</figcaption></figure><p>Next, we perform hyperparameter tuning to find the combination of hyperparameters that gives the best performance, as measured by the F1 score. This process uses the GridSearchCV function from the sklearn.model_selection module to perform hyperparameter tuning on the RandomForestClassifier model. This GridSearchCV is fit to the training data. This will train the model for each combination of hyperparameters in the grid, and perform cross-validation for each model. Then, the best model found by GridSearchCV is extracted.</p><p>Using k = 5 (number of folds), we obtained these as the results of hyperparameter search:</p><p>Best parameters: {‘max_depth’: 10, ‘max_features’: ‘sqrt’, ‘min_samples_leaf’: 2, ‘min_samples_split’: 2, ‘n_estimators’: 200}</p><p>Best cross-validation f1-score: 0.9194596558198883</p><p>Now, we know that the best model chosen has trees that can grow up to a depth of 10 levels, max_features, which is the number of features to consider when looking for the best split is ‘sqrt’, the minimum number of samples required to be at a leaf node is 2, the minimum number of samples required to split an internal node (termination condition for splitting) is 2, &amp; the number of trees in the optimal forest is 200. This model has a cross-validation f1-score of above 91%.</p><p>With this best model, we then calculate and plot its feature importances. We create a variable ‘feature_importances’ to store these feature importances which give a score for each feature of your data, the higher the score implies more importance or relevance that the feature has towards our output variable.’ The standard deviation of the feature importances across all trees in the forest is calculated. This gives an indication of the variability of the feature importance estimates.</p><p>A bar plot that is created with the feature importances is shown below:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/946/1*WNrM_GKJJo2OH45giEh9cg.png" /><figcaption>Figure 15</figcaption></figure><p>With this plot, the best model used the feature ‘CreditScoreAvg’ the most to allow it to have a very good performance. For each node of the tree, the feature that is chosen to split the data is the one that provides the largest decrease in impurity. The feature with the largest mean decrease in impurity is considered the most important.</p><p>This observation suggests that commercial banks’ existing credit scoring processes remain a very good predictor of whether a potential borrower is likely to repay his loan or not. In fact, in the early days of Prosper, researchers found that the Prosper Score was less effective at conveying a borrower’s credit worthiness compared to ‘CreditScoreAvg’ which was the industry standard set by FICO for calculating a borrower’s credit score.</p><p>Another surprising observation was the fact that ‘LoanOriginalAmount’ has a higher mean decrease in impurity than ‘AmountDelinquent’. This suggests that the likelihood of a borrower’s ability to repay a loan is more related to the principal than how much the borrower owes in delinquency, which means that it is a good strategy for P2P platform lenders to first screen the borrower’s total request amount before considering how much the borrower owes in delinquency according to the heuristic that the decision trees of a random forest use for the best performance.</p><p>The results of analysing mean decrease in impurity reveal that conventional factors that commercial banks find useful for credit scoring, like employment status &amp; whether the borrower owns a home are not as relevant compared to the top 3 features of ‘CreditScoreAvg’, ‘LoanOriginalAmount’ &amp; ‘AmountDelinquent’.</p><p>The following are the classification performance metrics that the best random forest classifier managed to achieve:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/716/1*LElzGL5uUZj1YSYiNO09NA.png" /><figcaption>Figure 16</figcaption></figure><p>Analysing these results, with the model achieving a recall of 99%, it shows that the model is very effective at identifying almost all positive instances (e.g., loan defaults) &amp; has a low rate of false negatives, which is crucial in scenarios where missing a positive instance (e.g., predicting a borrower will repay when they won’t) has significant consequences. The model includes a moderate amount of false positives. 15% of the instances your model predicts as positive are actually negative (e.g., predicting a borrower will default when they won’t), resulting in a moderate precision. Looking at the model holistically in terms of F1 score, the model maintains a good balance between precision and recall.</p><p>The higher recall relative to precision suggests that our random forest minimizes the chances of missing positive instances (loan defaults), which is desirable in financial applications where failing to identify a risky loan could result in significant losses. This is a more desirable trait to have than a model which has a higher precision than recall. While having a high precision is valuable, it is not as vital as high recall in credit scoring. A slightly lower precision means that some of the loans the model flags as risky might not actually be risky, leading to some missed opportunities as it seems more conservative in approving loans. Ultimately, we find that the trade-off between precision &amp; recall for the optimal model is acceptable for the random forest to be an effective classifier in our analysis of the 3 types of models on the Prosper lending dataset.</p><p><strong>Observations &amp; comparison of models</strong></p><p><strong>Logistic Regression</strong></p><p>Served as a baseline model due to its simplicity and interpretability. Makes use of 17 features, including numerical and categorical variables. Managed to achieve moderate accuracy and F1 score, providing a reasonable prediction capability. The model is easy to interpret, with clear insights into feature importance via coefficients. It is computationally efficient, making it suitable for quick, initial analyses. However, the linear nature of the model might not capture complex, non-linear relationships in the data, limiting its predictive power compared to more advanced models.</p><p><strong>Neural Networks</strong></p><p>Managed to capture complex patterns in the data, making them a powerful tool for predicting loan defaults. Used preprocessing to standardise numerical variables and encode categorical variables followed by Principal Component Analysis to reduce dimensions to 10. Achieved the highest accuracy and F1 scores among all models, with the best-performing network reaching 99.4% accuracy and 99.6% F1 score. Regularisation techniques like L2 regularisation, dropout, and early stopping were implemented to mitigate overfitting. It excels in identifying non-linear relationships within the data, leading to better performance than other models. However, Neural Networks require substantial computational resources and time for training, especially as the complexity and depth of the network increase. They are less interpretable compared to simpler models, making it challenging to understand the contribution of individual features to the predictions.</p><p><strong>Random Forest</strong></p><p>Uses a smaller subset of seven key features, optimised through extensive hyperparameter tuning.</p><p>Focused on crucial variables like ‘CreditScoreRange’, ‘LoanOriginalAmount’, and ‘AmountDelinquent’. The best model achieved a cross-validation F1 score of 91% and high recall (99%), indicating strong performance in identifying loan defaults. It is capable of capturing complex patterns and interactions between features, leading to robust performance. However, it can be computationally intensive, especially with a large number of trees, requiring careful tuning to balance bias and variance.</p><p><strong>Comparison of Accuracy and F1 Score</strong></p><p>Neural Networks: Achieved the highest accuracy and F1 scores, peaking at 99.4% accuracy and 99.6% F1 score.</p><p>Random Forest: Provided a balanced performance with high recall and acceptable precision. Achieved a cross-validation F1 score of 91% and test accuracy close to 99%.</p><p>Logistic Regression: Served as a reliable baseline with moderate accuracy and F1 scores, but was outperformed by neural networks and Random Forest in capturing complex patterns.</p><p><strong>Interpretability</strong></p><p>Logistic Regression: Most interpretable, with clear insights into feature importance and model coefficients.</p><p>Random Forest: Provided valuable feature importance metrics but was less interpretable due to its ensemble nature.</p><p>Neural Networks: Least interpretable, especially with deeper architectures and complex layers. Visualisations like tensorboard graphs helped, but overall interpretability was lower.</p><p><strong>Computational Efficiency</strong></p><p>Logistic Regression: Most computationally efficient, both in training and prediction.</p><p>Random Forest: Computationally intensive, particularly with a large number of trees and extensive hyperparameter tuning.</p><p>Neural Networks: Required significant computational resources, especially with larger architectures and training iterations.</p><p><strong>Improvements &amp; Follow-up</strong></p><ol><li>The first area of improvement is the recency of the dataset used. The dataset, obtained from Kaggle, covers P2P loan transactions on the Prosper platform from 2009 to 2017. The data’s age may limit the model’s applicability to current market conditions and borrower behaviour, given that economic conditions, regulatory environments, and lending practices have evolved since 2017. Conducting regular model retraining using the most up-to-date data reflecting recent P2P lending trends can help the models better capture contemporary borrower characteristics and market dynamics, improving its predictive power and reliability in current real-world applications.</li><li>We currently employ a Voting Classifier with a Hard Voting approach, whereby each model makes a prediction, and the ensemble’s prediction is determined by the majority vote of the individual classifiers. However, this method does not account for the confidence levels of each model, and is more prone to errors. This is especially the case since the confidence of the Logistic Regression model is lower than that of the Neural Network and Random Forest models.</li><li>A potential follow-up would be the use of a Soft Voting approach, which considers the confidence of each individual model. In this method, the predictions are weighted based on the model confidence, and the final ensemble prediction is determined based on the highest weighted probability. This approach would allow us to leverage on the varying level of confidence of each base model, leading to more accurate predictions and greater robustness to noise in the dataset.</li><li>Incorporating user feedback: in real-world applications, user feedback can be collected to provide valuable insights into how the model performs in real-world scenarios, highlighting areas where the model may be making errors or producing suboptimal results. Models can then be fine-tuned to better align with user expectations and practical requirements. This iterative process of collecting feedback, retraining the model, and validating improvements helps in addressing biases, refining features, and adjusting parameters to improve accuracy and user satisfaction.</li><li>Another potential follow-up would be the use of Stacking Ensemble Learning instead of a voting classifier. Stacking Ensemble Learning involves combining multiple models to produce a single, more powerful predictive model. Numerous base models are trained on the original dataset, and the eventual predictions are fed into a higher-level model (Soni, 2023). This can achieve better performance than any individual model alone by aggregating the predictions of several models. Increasing the diversity of base models used can average out the variance of individual models, further reducing the risk of overfitting. A Stacked Ensemble is also less likely to be affected by the peculiarities or noise in the training data, making them more robust.</li></ol><p><strong>Conclusion</strong></p><p>We hope that this article has allowed you to appreciate the process of credit scoring using various types of classifiers.</p><p>Much change is happening in the fintech industry, including regulatory &amp; business model changes, that makes the rules for data collection &amp; permitted analytics techniques more challenging in the social lending industry compared to traditional lending at commercial banks.</p><p>Also, with the rise of investors increasingly viewing lending others money on a P2P platform as an investment to generate profits, we hope that our work has shed light on the situation that variables such as FICO credit score (traditional scoring standards used by banks) still are of value. Lenders may be distracted by the high potential return on their investment, placing less emphasis on increasing risk of the borrower’s inability to repay with higher interest offered &amp; jeopardising their investment strategy on social lending.</p><p>Hence, the most effective approach to credit scoring in this dynamic environment is to utilize a combination of diverse classifiers. This allows lenders to gain a comprehensive understanding of a borrower’s creditworthiness, beyond just relying on a single metric like the FICO score. By incorporating a range of traditional and alternative data sources, as well as advanced analytical techniques, lenders can make more informed and prudent decisions, ultimately leading to a more sustainable and resilient lending ecosystem.</p><p>As the fintech industry continues to evolve, the need for a multifaceted approach to credit scoring will only become more critical. By recognizing the value of blending various classifiers, lenders can navigate the changing landscape, make better-informed decisions, and build a more robust and responsible lending infrastructure that serves the needs of both borrowers and investors.</p><p><strong>References</strong></p><p>Li, X., &amp; Ching, A. T. (2023). How does a firm adapt in a changing world? The case of prosper marketplace. Marketing Science.</p><p>Serrano-Cinca, C., &amp; Gutiérrez-Nieto, B. (2016). The use of profit scoring as an alternative to credit scoring systems in peer-to-peer (P2P) lending. Decision Support Systems, 89, 113–122.</p><p>Sulastri, R., &amp; Janssen, M. (2023, July). Challenges in designing an inclusive Peer-to-peer (P2P) lending system. In Proceedings of the 24th Annual International Conference on Digital Government Research (pp. 55–65).</p><p>Soni, B. (2023, May 1). Stacking to Improve Model Performance: A Comprehensive Guide on Ensemble Learning in Python. <a href="https://medium.com/@brijesh_soni/stacking-to-improve-model-performance-a-comprehensive-guide-on-ensemble-learning-in-python-9ed53c93ce28#:~:text=Stacking%20is%20a%20powerful%20ensemble,predictions%20of%20many%20base%20models">https://medium.com/@brijesh_soni/stacking-to-improve-model-performance-a-comprehensive-guide-on-ensemble-learning-in-python-9ed53c93ce28#:~:text=Stacking%20is%20a%20powerful%20ensemble,predictions%20of%20many%20base%20models</a>.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=c79cb1e67d33" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Intent classification of texts using Generative Adversarial Networks (GAN) and Bidirectional…]]></title>
            <link>https://medium.com/@nusfintech.ml/intent-classification-of-texts-using-generative-adversarial-networks-gan-and-bidirectional-29ea2f5ef6a4?source=rss-35fbff2bc022------2</link>
            <guid isPermaLink="false">https://medium.com/p/29ea2f5ef6a4</guid>
            <dc:creator><![CDATA[Machine Learning @ NUS Fintech Society]]></dc:creator>
            <pubDate>Mon, 08 Jul 2024 14:01:34 GMT</pubDate>
            <atom:updated>2024-07-08T14:01:34.564Z</atom:updated>
            <content:encoded><![CDATA[<h3><strong>Intent classification of texts using Generative Adversarial Networks (GAN) and Bidirectional Encoder Representations from Transformers (BERT) variations</strong></h3><p>Authors: Anthony Tamzil (Tech Lead), Nguyen Cao Duy, Lee Gene Ee, Nicholas Lee, Kieron Seven Lee</p><p><em>In this article, we will experiment with various BERT variations combined with GANs, performing performance comparison and analysis, identifying the distinct strengths and weaknesses of the various BERT variations by testing them on intent classification using the CLINC150 dataset.</em></p><p>For context, what is the difference between sentiment analysis and intent classification? Sentiment analysis is what most machine learning enthusiasts are familiar with, which is the process of categorizing text based on the sentiment or emotion expressed in a piece of text. For example, a text input like “I love the new features of this app” possesses positive sentiments and is therefore classified as a positive input. On the other hand, intent classification is the task of determining the underlying intention or purpose of a piece of text, commonly used to understand the intention behind the user inputs. For example, a user might type “book a flight to new york” and intent classification would be identifying it as a specific class like “user request”. There can be other classes like “user complaint”, “user feedback”, etc.</p><p>Ultimately, the aim of the project is to explore the various models to see what works and what doesn’t for intent classification. Focus will be placed on the BERT variations and how using GANs for data augmentation to form a hybrid approach can yield a more robust intent classification model.</p><h3>Complication</h3><p>The financial technology (FinTech) industry has witnessed rapid growth in recent years, revolutionizing the way individuals and businesses manage their finances. From online banking and digital payments to lending platforms and investment services, FinTech companies are continuously innovating to provide convenient and efficient financial solutions. However, with innovation comes a unique set of challenges, ranging from customer support automation to regulatory compliance and risk management. In this article, we will explore how intent classification, a powerful natural language processing (NLP) technique, can help the industry overcome these challenges.</p><p>Traditional intent classification models thrive on vast amounts of labeled data. In FinTech, where domain-specific jargon and regulations are common, acquiring and annotating such data is a costly and time-consuming endeavor. This data scarcity cripples the development of accurate and robust intent classification models, which can lead to inaccurate and biased models, or limited functionality.</p><p>Generative Adversarial Networks with Bidirectional Encoder Representations from Transformers (GAN-BERT) seeks to offer a new approach to bridge this data scarcity gap. By leveraging the power of BERT, a state-of-the-art language representation model, and the adversarial training framework of Generative Adversarial Networks (GANs), GAN-BERT aims to generate reliable synthetic training data for intent classification tasks in FinTech. This innovative approach allows for the creation of diverse and realistic data samples, overcoming the limitations of traditional data collection methods.</p><h3>Project Overview</h3><p>In this project, we aim to employ Generative Adversarial Networks (GAN) for data augmentation on the CLINC150 Dataset, consisting of 23,707 samples and 150 intent classes, with a focus on FinTech applications. Our goal is to address the challenge of data scarcity inherent in FinTech-specific datasets by leveraging GAN to generate synthetic data samples that mimic the distribution of real-world FinTech queries and interactions. Subsequently, we aim to modify existing BERT models or train plain BERT on FinTech terms and data to enhance their understanding of domain-specific jargon and regulations commonly encountered in the FinTech industry.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/468/0*-m3LjWP3TEs9SvqZ" /><figcaption><em>Figure 1: Example of CLINC150 dataset sample and label</em></figcaption></figure><p>By combining GAN-based data augmentation with fine-tuning of BERT models on FinTech-specific data, we aim to create a robust intent classification framework tailored to the unique requirements of the FinTech domain. This approach holds the promise of improving the accuracy and effectiveness of intent classification models for FinTech applications, including customer support automation (Chatbots) and risk management.</p><h3>CLINC150 Dataset</h3><p>This section covers the data preparation of the CLINC150 Dataset from downloading the dataset to converting it to an appropriate format for training.</p><ol><li>Cloning the CLINC150 dataset from its GitHub repository:</li></ol><pre># Load the json file downloaded from https://github.com/clinc/oos-eval.git<br>with open(&#39;/content/oos-eval/data/data_full.json&#39;, &#39;r&#39;) as file:<br>    data = json.load(file)</pre><p>2. Loading the dataset and splitting it into train, validation, and test sets:</p><pre>splits = [&quot;train&quot;, &quot;val&quot;, &quot;test&quot;]<br>datasets = {}<br>labels = set()<br>for split in splits:<br>    datasets[split] = [{&quot;text&quot;: item[0], &quot;label&quot;: item[1]} for item in data[split]]<br>    labels.update([item[1] for item in data[split]])</pre><p>3. Converting the data into a format compatible with Hugging Face’s DatasetDict:</p><pre># Convert to a Hugging Face Dataset<br>datasets = DatasetDict({<br>    split: Dataset.from_list(datasets[split]) for split in datasets.keys()<br>})</pre><p>4. Preparing labels for classification and mapping them to unique IDs:</p><pre>label2id = {label: id for id, label in enumerate(labels)}<br>id2label = {id: label for label, id in label2id.items()}<br>len(labels)</pre><p>The CLINC150 dataset does not possess any class imbalance and therefore, there is no need to do any class balancing. The dataset is built to test intent classification model performances, therefore, minimal preprocessing is needed. The data is processed similarly for all the BERT variations in order to ensure fairness and consistency while testing the various models. The same was also done for the GAN-BERT variations. Due to the nature of the GAN in the GAN-BERT variations, we decided to go with a 5% labelled and 95% unlabelled dataset for the GAN to generate and classify data. The labels would then be generated for the unlabelled dataset and generated dataset by the discriminator. However, for the BERT models without any GAN, only the labelled dataset of similar size was used in order to ensure that both the BERT and GAN-BERT started with the same amount of labelled data. The unlabelled data was simply provided for the GAN-BERT to generate more labelled data which would not be possible to achieve for the BERT variations without GAN.</p><h3>Implementation of BERT Variations</h3><p>Developed by Google, BERT (Bidirectional Encoder Representations from Transformers) is pre-trained on a vast corpus of text, leveraging tasks like Masked Language Modeling (MLM) and Next Sentence Prediction (NSP) to learn a rich understanding of language context and structure. This pre-training allows BERT to capture a deep sense of language nuances and relationships, making it highly effective for a wide range of NLP tasks such as sentiment analysis, question answering, and language inference. By fine-tuning BERT on the CLINC150 Dataset, we can build a highly robust intent classification model.</p><p>This section covers the training process of BERT from loading the BERT model, tokenizing the dataset, and setting up the training configuration. The transformers library is used extensively for this purpose.</p><ol><li>Initializing the tokenizer and model for sequence classification:</li></ol><pre>from transformers import AutoTokenizer<br>model_name = &quot;google-bert/bert-base-uncased&quot;<br>tokenizer = AutoTokenizer.from_pretrained(model_name)<br>model = AutoModelForSequenceClassification.from_pretrained(<br>    model_name, num_labels=len(labels), id2label=id2label, label2id=label2id,<br>    ignore_mismatched_sizes=True)</pre><p>2. Preprocessing the dataset by tokenizing texts and converting labels to their corresponding IDs:</p><pre>def preprocess_function(examples):<br>    text = examples[&quot;text&quot;]<br>    labels = examples[&quot;label&quot;]<br><br>    tokenized_text = tokenizer(text, truncation=False)<br>    labels = [label2id[label] for label in labels]<br><br>    return {&quot;input_ids&quot;: tokenized_text[&quot;input_ids&quot;],<br>            &quot;attention_mask&quot;: tokenized_text[&quot;attention_mask&quot;],<br>            &quot;label&quot;: labels}<br><br>tokenized_datasets = datasets.map(preprocess_function, batched=True)</pre><p>3. Configuring training arguments and initializing the Trainer for model training:</p><pre>from transformers import AutoModelForSequenceClassification, TrainingArguments, Trainer<br>data_collator = DataCollatorWithPadding(tokenizer=tokenizer)<br><br>model = AutoModelForSequenceClassification.from_pretrained(<br>    model_name, num_labels=len(labels), id2label=id2label, label2id=label2id,<br>    ignore_mismatched_sizes=True)<br><br>training_args = TrainingArguments(<br>    output_dir=&quot;intent-classification&quot;,<br>    learning_rate=2e-5,<br>    per_device_train_batch_size=32,<br>    per_device_eval_batch_size=32,<br>    num_train_epochs=10,<br>    weight_decay=0.01,<br>    evaluation_strategy=&quot;epoch&quot;,<br>    save_strategy=&quot;epoch&quot;,<br>    load_best_model_at_end=True,<br>    save_total_limit=1,<br>    report_to=&quot;none&quot;<br>)<br><br>trainer = Trainer(<br>    model=model,<br>    args=training_args,<br>    train_dataset=tokenized_datasets[&quot;train&quot;],<br>    eval_dataset=tokenized_datasets[&quot;val&quot;],<br>    tokenizer=tokenizer,<br>    data_collator=data_collator,<br>    compute_metrics=compute_metrics,<br>)<br><br>trainer.train()</pre><p>4. Repeat the same training steps using different models and compare the results using a pipeline:</p><pre>model_names = [<br>    &quot;google-bert/bert-base-uncased&quot;,<br>    &quot;distilbert/distilbert-base-uncased&quot;,<br>    &quot;FacebookAI/roberta-base&quot;,<br>    &quot;microsoft/deberta-base&quot;,<br>    &quot;ProsusAI/finbert&quot;<br>]</pre><p>5. Evaluate the results of the models</p><pre>trainer.evaluate()<br>predictions = trainer.predict(tokenized_datasets[&quot;test&quot;])<br>print(predictions.metrics)</pre><h3>Analysis of BERT Models</h3><h3>BERT-Base</h3><p>BERT-Base stands as a general-purpose variant of the BERT model, trained on the English language corpus. It is architecturally composed of 12 Transformer blocks, encompassing 768 hidden units and 12 self-attention heads, which cumulatively account for approximately 110 million parameters. This extensive configuration empowers BERT-Base with the capability to attain a deep comprehension of language nuances, albeit at the expense of heightened computational demands.</p><p>As compared to traditional NLP techniques, BERT-Base excels by analyzing words in their entirety within sentences. This methodological approach enables the model to grasp the contextual essence of each word, considering its entire surrounding context, both preceding and following, unlike the conventional sequential word processing.</p><p>BERT-Base does this through the execution of two pivotal unsupervised tasks: the Masked Language Model (MLM) and Next Sentence Prediction (NSP). In the MLM task, BERT masks 15% of the words within a sentence, challenging itself to predict these obscured words by relying solely on the context furnished by the remaining, visible words. This process significantly enhances BERT’s contextual understanding. Concurrently, through the NSP task, BERT evaluates pairs of sentences to ascertain if the second sentence logically and sequentially follows the first, further refining its comprehension of sentence relationships.</p><p>In scenarios demanding the depth of language understanding, BERT-Base emerges as the preferred choice, especially when computational resources are not a limiting factor. However, while it ensures accuracy and a sophisticated level of comprehension, it is notably more computationally intensive than its other streamlined counterparts, such as DistilBERT. BERT-Base is also designed for general-purpose applications, whereby further domain-specific fine-tuning may yield superior performance in specialized tasks.</p><h3>DistilBERT</h3><p>DistilBERT is a lightweight variant of BERT that aims to reduce the model size and computational complexity while maintaining high performance on NLP tasks. It achieves a 40% reduction in model size from 110 million parameters in BERT-base to 66 million parameters in DistilBert via knowledge distillation, and is designed to have a faster inference speed compared to the original BERT model. In terms of training time, the DistilBERT has an approximately 30% reduction in training time.</p><p>The key feature of DistilBERT is that the model is trained using knowledge distillation, a technique where a smaller student model (DistilBERT) learns to mimic the behavior of a larger teacher model (BERT), by approximating the full output distributions of the larger BERT model. In neural network terms, knowledge typically refers to the learnt weights and biases. During this distillation process, the student model is trained to minimize the difference between its own outputs and the soft targets produced by the teacher model, which allows the student model to capture the knowledge and generalization capabilities of the student model to achieve comparable performance with fewer parameters.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/850/0*lY0jtBUfo9Xs2Cw3" /><figcaption><em>Figure 2: Visualization of knowledge distillation and comparison of model structure between BERT-base and DistilBERT</em></figcaption></figure><p>In terms of differences between DistilBERT and BERT-base, DistilBERT only has 6 transformer layers while the BERT base model has 12, with the aim to streamline the model while maintaining the model’s core functionality. Furthermore, the token-type embeddings and the pooler present in BERT (not in figure 1) used for next sentence prediction are removed in DistilBERT to simplify architecture and reduce computational overhead, making it more lightweight and efficient. Lastly, a distillation token is added to the input sequence to improve the quality of the sequence representations learnt by the model. Despite these modifications, DistilBERT retains a substantial amount of performance and capabilities from BERT-base.</p><p>DistilBERT achieves a relatively decent performance of 0.6760 test F1-score (25 epochs) for the project considering its smaller size, and is surprisingly able to surpass the BERT-base and FinBERT model in terms of performance. Although it is comparable to the larger BERT-base model and finBERT model, there is evidently a trade-off compared to the other variants of BERT that have seen higher accuracy and performance levels like RoBERTa. In terms of test runtime and inference speed, DistilBERT is consistently the fastest among the BERT variations. The trade-off between model size, computational efficiency and performance makes DistilBERT a great choice in scenarios where resources are limited or if faster inference is required, and has paved the way to inspire further research on model compression and efficient NLP models.</p><h3>RoBERTa</h3><p>RoBERTA is a robustly optimized BERT pretraining approach that improves on BERT-base. In particular, it builds on BERT’s language masking strategy. Architecturally, RoBERTa has almost a similar architecture as compared to BERT.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/828/0*YhdCEruVn_ZlzRJ5" /><figcaption><em>Figure 3: Visualization of RoBERTa’s model structure</em></figcaption></figure><p>RoBERTa modifies certain key hyperparameters in BERT, including removing BERT’s next-sentence pretraining (NSP) loss, allowing RoBERTa to focus and improve on BERT’s MLM objective, leading to better downstream task performance.. The authors of RoBERTa experimented with removing / adding of NSP loss to various versions and concluded that removal of the NSP loss matches or improves NLP task performance. RoBERTa is also trained with much larger mini-batches and learning rates. Originally, BERT is trained for 1 million steps with a batch sizeof 256 sequences, while RoBERTa was trained with 125 steps of 2000 sequences and 31k steps with 8000 sequences, with batch sizes evidently larger than BERT’s. The larger batches not only improves perplexity on the masked language modeling (MLM) objective, but apparently the training process is also easier to parallelize as a result. Furthermore, RoBERTa is trained on a larger corpus of text (10 times larger) compared to BERT which allows it to learn a more robust and nuanced representation of the language. Specifically, RoBERTa is trained on the BOOK corpus, English Wikipedia dataset (both of which were also used for training BERT), CC-NEWS, OPENWEBTEXT and the STORIES dataset. This obviously led to longer training times of about 4 to 5 times longer than that of BERT.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/828/0*gsPdNbsYWRtRkr9g" /><figcaption><em>Figure 4: A comparison between static masking in BERT and dynamic masking in RoBERTa</em></figcaption></figure><p>Lastly, RoBERTa also utilizes dynamic masking. During pre-training, BERT performs masked language modeling by trying to predict a certain percentage of masked tokens. The issue with this implementation is that chosen tokens for masking for a given text sequence across different batches would sometimes be the same. The training dataset is duplicated 10 times, so each sequence was masked in only 10 different ways. With BERT running 40 training epochs, each sequence with the same masking is passed to BERT four times, therefore, dynamic masking was better in the sense that the masking is generated uniquely everytime a sequence is passed to BERT, resulting in less duplicated data during the training and giving the model an opportunity to work with a greater variety of data and masking patterns.</p><p>In terms of performance, RoBERTa got the highest test F1 score (10 and 25 epochs) as advertised. This is mainly due to the differences stated earlier that allow it to have an edge in various NLP tasks compared to BERT-base or DistilBERT (which is a distilled version of BERT). Given that it has been able to top other LLM models in a variety of other NLP tasks, it is no surprise that RoBERTa is one of the higher performing ones here too.</p><h3>DeBERTa</h3><p>DeBERTa is a Decoding-enhanced BERT with Disentangled Attention, seeking to improve on RoBERTa and BERT-base using two novel techniques: a disentangled attention mechanism and an enhanced mask decoder.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/828/0*PpJDjhdl-g9Wqa9T" /><figcaption><em>Figure 5: Visualization of the Disentangled Attention Mechanism in DeBERTa and how it varies from the standard Attention mechanism in base BERT</em></figcaption></figure><p>In BERT, each word in the input layer is represented using a vector which is the sum of its word embedding (content) and position embedding. In DeBERTa, each word is represented using two separate vectors that encode the content and position respectively, and the attention weights among the words are computed using disentangled matrices based on the content and relative positions, therefore the name, disentangled attention. This is because it is believed that the attention weight of a word pair depends on not only their contents but also their relative positions. For example, words that are adjacent have a stronger dependency compared to words that are in different sentences. This then would lead to slight modification in attention score computation. Calculation of cross-attention scores between two embeddings (words) now consists of two vectors that can be decomposed into the sum of four pairwise multiplication of their subvectors which are shown in Figure 6 below.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/828/0*Nr8zTuBquLCTfhj8" /><figcaption><em>Figure 6: Calculation of cross-attention scores between a pair of embeddings</em></figcaption></figure><p>For context, both BERT and DeBERTa are pre-trained using masked language modeling (MLM) which is a fill in the blank task. DeBERTa uses the content and position information of the surrounding words for MLM. The disentangled attention mechanism considers the contents and relative positions of the surrounding words but not the absolute positions of these words which is crucial for the prediction. This is due to the syntactic nuances that heavily depend on the absolute position of the word in the sentence. In order to counter this flaw, DeBERTa incorporates absolute word position embeddings right before the softmax activation layer where the model decodes the masked words based on aggregated contextual embeddings of word contents and positions. This way, DeBERTa captures the relative positions in all the Transformer layers and only uses the absolute positions as complementary information when decoding the masked words.</p><h3>FinBERT</h3><p>FinBERT is a domain-specific BERT model that has been pre-trained on a large corpus of financial text. The domains include English Wikipedia, BooksCorpus (both of which were used to train BERT), FinancialWeb, YahooFinance and RedditFinanceQA. It is designed to better understand and process financial language compared to a standard BERT model.</p><p>FinBERT was first proposed as word embeddings such as word2vec are not effective enough to detect the special language used in the financial field. Moreover, financial text data often consists of named entities like stock, bond types and financial institution names and contain unique vocabulary information, and a more nuanced way of capturing these informations was required. A total of six pre-training tasks were pre-trained through multi-task self-supervised pre-training learning, and the pre-trained model is fine tuned using task specific supervised data to adapt to various language understanding tasks.</p><p>The model is first pre-trained through multi-task learning on financial corpora to transfer knowledge from the financial domain corpora. At the transformer encoder layer, the input embedding is responsible for converting each word into an embedding representation that is fed into the transformer encoder. The transformer encoder is based on a self-attention mechanism, which captures global context information by pairwise correlation.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/910/0*rlTOFk2kfy8yLXvU" /><figcaption><em>Figure 7: Visualization of model architecture for FinBERT</em></figcaption></figure><p>The next step is the multi-task self-supervised pre-training, which is crucial for the model to continuously gain general knowledge. There are a total of six tasks to learn different level of knowledge from the training corpora:</p><ol><li><strong>Span Replace Prediction pre-training task</strong>: 15% of the randomly sampled input texts are dropped out and replaced by a unique masked token to replace all of each successive boundary. The masked span is predicted based on the span boundary.</li><li><strong>Capitalization Prediction pre-training task</strong>: In finance and economics, capital words usually have specific semantic value in a sentence, with advantages such as named entity recognition. The task involves predicting whether the word is capitalized or not.</li><li><strong>Token-Passage Prediction pre-training tas</strong>k: The keywords of a passage appearing in a segment to enable FinBERT to capture topics of a passage are identified. The task involves predicting whether a token appears in segments of the original passage.</li><li><strong>Sentence Deshuffling pre-training tas</strong>k: A paragraph is shuffled by a random permuted order. Using the original de-shuffled sequence as training target, the model is pre-trained to reorganize the permuted segments using a multiclass classification task.</li><li><strong>Sentence Distance pre-training task</strong>: A self-supervised training target is used to predict sentence distance by creating a three-class classification task</li><li><strong>Dialogue Relation pre-training task</strong>: semantic relevance among sentences using question answering data was constructed to learn implicit relationships and semantic relevance.</li></ol><p>FinBERT is one of the poorer performing models for our project with a relatively lower test F1-score compared to the other variations. Due to the nature of the specialization of FinBERT on financial text, its customized train corpora might not sufficiently detect the nuances of “out of scope” queries posed in our dataset, compared to the other better performing models such as RoBERTa or DeBERTa. In terms of performance for intent classification specifically, it would unsurprisingly fall short in terms of performance.</p><h3>Generative Adversarial Networks (GANs)</h3><p>Generative Adversarial Networks (GANs) are a type of deep learning architecture that consists of two neural networks competing against each other: a generator network that generates synthetic data, and a discriminator network that tries to distinguish the generated data from real data. This adversarial training process allows the generator to produce data that resembles the real data distribution. The challenge in this project would be to integrate it with not just a BERT-base model, but BERT variations where documentation is rather scarce.</p><h3>GAN-BERT Hybrid Approach</h3><p>The implementation of GAN-BERT discussed in <a href="https://towardsdatascience.com/semi-supervised-intent-classification-with-gan-bert-934d8659bca2">this article</a> represents a fascinating fusion of GANs and BERT. This integration aims to leverage the power of GANs to enhance the fine-tuning process of BERT, with the goal of achieving improved performance in intent classification. We will adapt this general model structure to test it out on various BERT variations.</p><p>The article outlines a series of experiments conducted to assess the effectiveness of GAN-BERT in the context of intent classification using the comprehensive CLINC150 dataset. Notably, the experiments were conducted in a semi-supervised learning setting, where a portion of the data was labelled while the rest remained unlabelled. This setup is particularly intriguing as it mimics real-world scenarios where labelled data may be limited or expensive to obtain.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/720/0*DCzgb1Xax3y3nGak" /><figcaption><em>Figure 8: GAN-BERT Model Architecture</em></figcaption></figure><p>The training procedure outlined in the article serves as a blueprint for implementing GAN-BERT. The implementation of GAN-BERT can be summarized as follows:</p><ol><li>Pre-train the BERT model on a large corpus of unlabelled text data to capture general language representation.</li><li>The GAN-BERT model is trained with two inputs. Real data which consists of a composition of labelled and unlabelled data is passed into BERT while a 100-dimensional noise vector is passed into the generator to produce fake examples. The discriminator then receives both outputs from the generator and the BERT and then attempts to classify whether the input is part of the real dataset or not. If it predicts that the input is part of the real dataset, it then has to predict which class the input belongs to. This would allow the discriminator network to learn to distinguish between real and generated examples, while the generator network learns to produce more realistic examples that can fool the discriminator. This whole process also generates labelled data which is useful in fine tuning the BERT model.</li><li>Fine-tune the BERT model using a combination of the original labelled data and the synthetic examples generated by the GAN. This augmented training data helps the BERT model learn better representations for the intent classification task, especially when dealing with limited labelled data.</li></ol><p>This accomplishment underscores the potential of GAN-BERT as a robust methodology for improving the performance of NLP tasks, especially in scenarios where labelled data is scarce. By leveraging the generative capabilities of GANs to augment the training data, GAN-BERT offers a promising avenue for enhancing the capabilities of state-of-the-art language models like BERT.</p><p>We will experiment with this model structure and analyze some of the obtained results as shown below.</p><h3>Implementation of GAN-BERT</h3><p>This section covers the training process of GAN-BERT models using the methods detailed in <a href="https://www.aclweb.org/anthology/2020.acl-main.191/">this paper</a>.</p><ol><li>Configuring the transformers parameter which includes batch size and maximum sequence length as well as number of epochs</li></ol><pre>#--------------------------------<br>#  Transformer parameters<br>#--------------------------------<br>max_seq_length = 64<br>batch_size = 64<br><br>#--------------------------------<br>#  GAN-BERT specific parameters<br>#--------------------------------<br># number of hidden layers in the generator,<br># each of the size of the output space<br>num_hidden_layers_g = 1;<br># number of hidden layers in the discriminator,<br># each of the size of the input space<br>num_hidden_layers_d = 1;<br># size of the generator&#39;s input noisy vectors<br>noise_size = 100<br># dropout to be applied to discriminator&#39;s input vectors<br>out_dropout_rate = 0.2<br><br># Replicate labeled data to balance poorly represented datasets,<br># e.g., less than 1% of labeled material<br>apply_balance = True<br><br>#--------------------------------<br>#  Optimization parameters<br>#--------------------------------<br>learning_rate_discriminator = 5e-5<br>learning_rate_generator = 5e-5<br>epsilon = 1e-8<br>num_train_epochs = 10<br>multi_gpu = True<br># Scheduler<br>apply_scheduler = False<br>warmup_proportion = 0.1<br># Print<br>print_each_n_step = 100<br><br>#--------------------------------<br>#  Adopted Transformer model<br>#--------------------------------<br># Since this version is compatible with Huggingface transformers, you can uncomment<br># (or add) transformer models compatible with GAN<br><br># model_name = &quot;google-bert/bert-base-uncased&quot;<br>model_name = &quot;distilbert/distilbert-base-uncased&quot;<br># model_name = &quot;FacebookAI/roberta-base&quot;<br># model_name = &quot;microsoft/deberta-base&quot;<br># model_name = &quot;ProsusAI/finbert&quot;<br><br>#--------------------------------<br>#  Retrieve the CLINC150 Dataset<br>#--------------------------------<br><br>labeled_file = f&quot;{DIRECTORY_PATH}/5_95/labeled.tsv&quot;<br>unlabeled_file = f&quot;{DIRECTORY_PATH}/5_95/unlabeled.tsv&quot;<br>test_filename = f&quot;{DIRECTORY_PATH}/test.tsv&quot;</pre><p>2. Defining the Generator and Discriminator for the GAN</p><pre><br># ------------------------------<br>#   The Generator as in<br>#   https://www.aclweb.org/anthology/2020.acl-main.191/<br>#   https://github.com/crux82/ganbert<br># ------------------------------<br>class Generator(nn.Module):<br>    def __init__(self, noise_size=100, output_size=512, hidden_sizes=[512], dropout_rate=0.1):<br>        super(Generator, self).__init__()<br>        layers = []<br>        hidden_sizes = [noise_size] + hidden_sizes<br>        for i in range(len(hidden_sizes)-1):<br>            layers.extend([nn.Linear(hidden_sizes[i], hidden_sizes[i+1]),<br>                          nn.LeakyReLU(0.2, inplace=True), nn.Dropout(dropout_rate)])<br><br>        layers.append(nn.Linear(hidden_sizes[-1], output_size))<br>        self.layers = nn.Sequential(*layers)<br><br>    def forward(self, noise):<br>        output_rep = self.layers(noise)<br>        return output_rep<br><br># ------------------------------<br>#   The Discriminator<br>#   https://www.aclweb.org/anthology/2020.acl-main.191/<br>#   https://github.com/crux82/ganbert<br># ------------------------------<br>class Discriminator(nn.Module):<br>    def __init__(self, input_size=512, hidden_sizes=[512], num_labels=2, dropout_rate=0.1):<br>        super(Discriminator, self).__init__()<br>        self.input_dropout = nn.Dropout(p=dropout_rate)<br>        layers = []<br>        hidden_sizes = [input_size] + hidden_sizes<br>        for i in range(len(hidden_sizes)-1):<br>            layers.extend([nn.Linear(hidden_sizes[i], hidden_sizes[i+1]),<br>                          nn.LeakyReLU(0.2, inplace=True), nn.Dropout(dropout_rate)])<br><br>        self.layers = nn.Sequential(*layers)  # per il flatten<br>        # +1 for the probability of this sample being fake/real.<br>        self.logit = nn.Linear(hidden_sizes[-1], num_labels+1)<br>        self.softmax = nn.Softmax(dim=-1)<br><br>    def forward(self, input_rep):<br>        input_rep = self.input_dropout(input_rep)<br>        last_rep = self.layers(input_rep)<br>        logits = self.logit(last_rep)<br>        probs = self.softmax(logits)<br>        return last_rep, logits, probs<br><br></pre><p>3. Instantiate the Generator and Discriminator</p><pre># Define the number and width of hidden layers<br>hidden_levels_g = [hidden_size for i in range(0, num_hidden_layers_g)]<br>hidden_levels_d = [hidden_size for i in range(0, num_hidden_layers_d)]<br><br># -------------------------------------------------<br>#   Instantiate the Generator and Discriminator<br># -------------------------------------------------<br>generator = Generator(noise_size=noise_size, output_size=hidden_size,<br>                      hidden_sizes=hidden_levels_g, dropout_rate=out_dropout_rate)<br>discriminator = Discriminator(input_size=hidden_size, hidden_sizes=hidden_levels_d, <br>num_labels=len(labels), dropout_rate=out_dropout_rate)</pre><p>4. Define a function to generate a Dataloader based on the inputs, which preprocesses input exemples, tokenizes them, encodes labels and creates a PyTorch DataLoader for the GAN-BERT variations</p><pre>def generate_data_loader(input_examples, label_masks, label_map, do_shuffle=False, balance_label_examples=False):<br>    &#39;&#39;&#39;<br>    Generate a Dataloader given the input examples, eventually masked if they are<br>    to be considered NOT labeled.<br>    &#39;&#39;&#39;<br>    examples = []<br><br>    # Count the percentage of labeled examples<br>    num_labeled_examples = 0<br>    for label_mask in label_masks:<br>        if label_mask:<br>            num_labeled_examples += 1<br>    label_mask_rate = num_labeled_examples/len(input_examples)<br><br>    # if required it applies the balance<br>    for index, ex in enumerate(input_examples):<br>        if label_mask_rate == 1 or not balance_label_examples:<br>            examples.append((ex, label_masks[index]))<br>        else:<br>            # IT SIMULATE A LABELED EXAMPLE<br>            if label_masks[index]:<br>                balance = int(1/label_mask_rate)<br>                balance = int(math.log(balance, 2))<br>                if balance &lt; 1:<br>                    balance = 1<br>                for b in range(0, int(balance)):<br>                    examples.append((ex, label_masks[index]))<br>            else:<br>                examples.append((ex, label_masks[index]))<br><br>    # -----------------------------------------------<br>    # Generate input examples to the Transformer<br>    # -----------------------------------------------<br>    input_ids = []<br>    input_mask_array = []<br>    label_mask_array = []<br>    label_id_array = []<br><br>    # Tokenization<br>    for (text, label_mask) in examples:<br>        encoded_sent = tokenizer.encode(<br>            text[0], add_special_tokens=True, max_length=max_seq_length, padding=&quot;max_length&quot;, truncation=True)<br>        input_ids.append(encoded_sent)<br>        label_id_array.append(label_map[text[1]])<br>        label_mask_array.append(label_mask)<br><br>    # Attention to token (to ignore padded input wordpieces)<br>    for sent in input_ids:<br>        att_mask = [int(token_id &gt; 0) for token_id in sent]<br>        input_mask_array.append(att_mask)<br><br>    # Convertion to Tensor<br>    input_ids = torch.tensor(input_ids)<br>    input_mask_array = torch.tensor(input_mask_array)<br>    label_id_array = torch.tensor(label_id_array, dtype=torch.long)<br>    label_mask_array = torch.tensor(label_mask_array)<br><br>    # Building the TensorDataset<br>    dataset = TensorDataset(input_ids, input_mask_array,<br>                            label_id_array, label_mask_array)<br><br>    if do_shuffle:<br>        sampler = RandomSampler<br>    else:<br>        sampler = SequentialSampler<br><br>    # Building the DataLoader<br>    return DataLoader(<br>        dataset,  # The training samples.<br>        sampler=sampler(dataset),<br>        batch_size=batch_size)  # Trains with this batch size.</pre><p>5. Convert the input examples into a DataLoader for the training and test data</p><pre># ------------------------------<br>#   Load the train dataset<br># ------------------------------<br>train_examples = labeled_examples<br># The labeled (train) dataset is assigned with a mask set to True<br>train_label_masks = np.ones(len(labeled_examples), dtype=bool)<br># If unlabel examples are available<br>if unlabeled_examples:<br>    train_examples = train_examples + unlabeled_examples<br>    # The unlabeled (train) dataset is assigned with a mask set to False<br>    tmp_masks = np.zeros(len(unlabeled_examples), dtype=bool)<br>    train_label_masks = np.concatenate([train_label_masks, tmp_masks])<br><br>train_dataloader = generate_data_loader(train_examples, train_label_masks, label2id, <br>       do_shuffle=True, balance_label_examples=apply_balance)<br><br># ------------------------------<br>#   Load the test dataset<br># ------------------------------<br># The labeled (test) dataset is assigned with a mask set to True<br>test_label_masks = np.ones(len(test_examples), dtype=bool)<br><br>test_dataloader = generate_data_loader(test_examples, test_label_masks, label2id, <br>do_shuffle=False, balance_label_examples=False)</pre><p>6. Define a training function</p><pre>def training_procedure(is_pooled_output=True):<br>    training_stats = []<br><br>    # Measure the total training time for the whole run.<br>    total_t0 = time.time()<br><br>    # models parameters<br>    transformer_vars = [i for i in transformer.parameters()]<br>    d_vars = transformer_vars + [v for v in discriminator.parameters()]<br>    g_vars = [v for v in generator.parameters()]<br><br>    # optimizer<br>    dis_optimizer = torch.optim.AdamW(d_vars, lr=learning_rate_discriminator)<br>    gen_optimizer = torch.optim.AdamW(g_vars, lr=learning_rate_generator)<br><br>    # scheduler<br>    if apply_scheduler:<br>        num_train_examples = len(train_examples)<br>        num_train_steps = int(num_train_examples / batch_size * num_train_epochs)<br>        num_warmup_steps = int(num_train_steps * warmup_proportion)<br><br>        scheduler_d = get_constant_schedule_with_warmup(dis_optimizer,<br>                                                        num_warmup_steps=num_warmup_steps)<br>        scheduler_g = get_constant_schedule_with_warmup(gen_optimizer,<br>                                                        num_warmup_steps=num_warmup_steps)<br><br>    # For each epoch...<br>    for epoch_i in range(0, num_train_epochs):<br>        # ========================================<br>        #               Training<br>        # ========================================<br>        # Perform one full pass over the training set.<br>        print(&quot;&quot;)<br>        print(<br>            &#39;======== Epoch {:} / {:} ========&#39;.format(epoch_i + 1, num_train_epochs))<br>        print(&#39;Training...&#39;)<br><br>        # Measure how long the training epoch takes.<br>        t0 = time.time()<br><br>        # Reset the total loss for this epoch.<br>        tr_g_loss = 0<br>        tr_d_loss = 0<br><br>        # Put the model into training mode.<br>        transformer.train()<br>        generator.train()<br>        discriminator.train()<br><br>        # For each batch of training data...<br>        for step, batch in enumerate(train_dataloader):<br><br>            # Progress update every print_each_n_step batches.<br>            if step % print_each_n_step == 0 and not step == 0:<br>                # Calculate elapsed time in minutes.<br>                elapsed = format_time(time.time() - t0)<br><br>                # Report progress.<br>                print(&#39;  Batch {:&gt;5,}  of  {:&gt;5,}.    Elapsed: {:}.&#39;.format(<br>                    step, len(train_dataloader), elapsed))<br><br>            # Unpack this training batch from our dataloader.<br>            b_input_ids = batch[0].to(device)<br>            b_input_mask = batch[1].to(device)<br>            b_labels = batch[2].to(device)<br>            b_label_mask = batch[3].to(device)<br><br>            real_batch_size = b_input_ids.shape[0]<br><br>            # Encode real data in the Transformer<br>            model_outputs = transformer(b_input_ids, attention_mask=b_input_mask)<br>            hidden_states = model_outputs[-1]<br><br>            # Generate fake data that should have the same distribution of the ones<br>            # encoded by the transformer.<br>            # First noisy input are used in input to the Generator<br>            noise = torch.zeros(real_batch_size, noise_size,<br>                                device=device).uniform_(0, 1)<br>            # Generate Fake data<br>            gen_rep = generator(noise)<br>            <br>            if not is_pooled_output:<br>                hidden_states = hidden_states[:, 0]<br><br>            # Generate the output of the Discriminator for real and fake data.<br>            # First, we put together the output of the tranformer and the generator<br>            disciminator_input = torch.cat([hidden_states, gen_rep], dim=0)<br>            # Then, we select the output of the disciminator<br>            features, logits, probs = discriminator(disciminator_input)<br><br>            # Finally, we separate the discriminator&#39;s output for the real and fake<br>            # data<br>            features_list = torch.split(features, real_batch_size)<br>            D_real_features = features_list[0]<br>            D_fake_features = features_list[1]<br><br>            logits_list = torch.split(logits, real_batch_size)<br>            D_real_logits = logits_list[0]<br>            D_fake_logits = logits_list[1]<br><br>            probs_list = torch.split(probs, real_batch_size)<br>            D_real_probs = probs_list[0]<br>            D_fake_probs = probs_list[1]<br><br>            # ---------------------------------<br>            #  LOSS evaluation<br>            # ---------------------------------<br>            # Generator&#39;s LOSS estimation<br>            g_loss_d = -1 * \<br>                torch.mean(torch.log(1 - D_fake_probs[:, -1] + epsilon))<br>            g_feat_reg = torch.mean(torch.pow(torch.mean(<br>                D_real_features, dim=0) - torch.mean(D_fake_features, dim=0), 2))<br>            g_loss = g_loss_d + g_feat_reg<br><br>            # Disciminator&#39;s LOSS estimation<br>            logits = D_real_logits[:, 0:-1]<br>            log_probs = F.log_softmax(logits, dim=-1)<br>            # The discriminator provides an output for labeled and unlabeled real data<br>            # so the loss evaluated for unlabeled data is ignored (masked)<br>            label2one_hot = torch.nn.functional.one_hot(b_labels, len(labels))<br>            per_example_loss = -torch.sum(label2one_hot * log_probs, dim=-1)<br>            per_example_loss = torch.masked_select(<br>                per_example_loss, b_label_mask.to(device))<br>            labeled_example_count = per_example_loss.type(torch.float32).numel()<br><br>            # It may be the case that a batch does not contain labeled examples,<br>            # so the &quot;supervised loss&quot; in this case is not evaluated<br>            if labeled_example_count == 0:<br>                D_L_Supervised = 0<br>            else:<br>                D_L_Supervised = torch.div(<br>                    torch.sum(per_example_loss.to(device)), labeled_example_count)<br><br>            D_L_unsupervised1U = -1 * \<br>                torch.mean(torch.log(1 - D_real_probs[:, -1] + epsilon))<br>            D_L_unsupervised2U = -1 * \<br>                torch.mean(torch.log(D_fake_probs[:, -1] + epsilon))<br>            d_loss = D_L_Supervised + D_L_unsupervised1U + D_L_unsupervised2U<br><br>            # ---------------------------------<br>            #  OPTIMIZATION<br>            # ---------------------------------<br>            # Avoid gradient accumulation<br>            gen_optimizer.zero_grad()<br>            dis_optimizer.zero_grad()<br><br>            # Calculate weigth updates<br>            # retain_graph=True is required since the underlying graph will be deleted after backward<br>            g_loss.backward(retain_graph=True)<br>            d_loss.backward()<br><br>            # Apply modifications<br>            gen_optimizer.step()<br>            dis_optimizer.step()<br><br>            # A detail log of the individual losses<br>            # print(&quot;{0:.4f}\t{1:.4f}\t{2:.4f}\t{3:.4f}\t{4:.4f}&quot;.<br>            #      format(D_L_Supervised, D_L_unsupervised1U, D_L_unsupervised2U,<br>            #             g_loss_d, g_feat_reg))<br><br>            # Save the losses to print them later<br>            tr_g_loss += g_loss.item()<br>            tr_d_loss += d_loss.item()<br><br>            # Update the learning rate with the scheduler<br>            if apply_scheduler:<br>                scheduler_d.step()<br>                scheduler_g.step()<br><br>        # Calculate the average loss over all of the batches.<br>        avg_train_loss_g = tr_g_loss / len(train_dataloader)<br>        avg_train_loss_d = tr_d_loss / len(train_dataloader)<br><br>        # Measure how long this epoch took.<br>        training_time = format_time(time.time() - t0)<br><br>        print(&quot;&quot;)<br>        print(&quot;  Average training loss generetor: {0:.3f}&quot;.format(<br>            avg_train_loss_g))<br>        print(&quot;  Average training loss discriminator: {0:.3f}&quot;.format(<br>            avg_train_loss_d))<br>        print(&quot;  Training epcoh took: {:}&quot;.format(training_time))<br><br>        # ========================================<br>        #     TEST ON THE EVALUATION DATASET<br>        # ========================================<br>        # After the completion of each training epoch, measure our performance on<br>        # our test set.<br>        print(&quot;&quot;)<br>        print(&quot;Running Test...&quot;)<br><br>        t0 = time.time()<br><br>        # Put the model in evaluation mode--the dropout layers behave differently<br>        # during evaluation.<br>        transformer.eval()  # maybe redundant<br>        discriminator.eval()<br>        generator.eval()<br><br>        # Tracking variables<br>        total_test_accuracy = 0<br><br>        total_test_loss = 0<br>        nb_test_steps = 0<br><br>        all_preds = []<br>        all_labels_ids = []<br><br>        # loss<br>        nll_loss = torch.nn.CrossEntropyLoss(ignore_index=-1)<br><br>        # Evaluate data for one epoch<br>        for batch in test_dataloader:<br><br>            # Unpack this training batch from our dataloader.<br>            b_input_ids = batch[0].to(device)<br>            b_input_mask = batch[1].to(device)<br>            b_labels = batch[2].to(device)<br><br>            # Tell pytorch not to bother with constructing the compute graph during<br>            # the forward pass, since this is only needed for backprop (training).<br>            with torch.no_grad():<br>                model_outputs = transformer(<br>                    b_input_ids, attention_mask=b_input_mask)<br>                hidden_states = model_outputs[-1]<br>                if not is_pooled_output:<br>                    hidden_states = hidden_states[:, 0]<br>                _, logits, probs = discriminator(hidden_states)<br>                # log_probs = F.log_softmax(probs[:,1:], dim=-1)<br>                filtered_logits = logits[:, 0:-1]<br>                # Accumulate the test loss.<br>                total_test_loss += nll_loss(filtered_logits, b_labels)<br><br>            # Accumulate the predictions and the input labels<br>            _, preds = torch.max(filtered_logits, 1)<br>            all_preds += preds.detach().cpu()<br>            all_labels_ids += b_labels.detach().cpu()<br><br>        # Report the final accuracy for this validation run.<br>        all_preds = torch.stack(all_preds).numpy()<br>        all_labels_ids = torch.stack(all_labels_ids).numpy()<br>        test_accuracy = np.sum(all_preds == all_labels_ids) / len(all_preds)<br>        test_f1score = f1_score(all_labels_ids, all_preds, average=&#39;weighted&#39;)<br>        test_precision = precision_score(<br>            all_labels_ids, all_preds, average=&#39;weighted&#39;)<br>        test_recall = recall_score(all_labels_ids, all_preds, average=&#39;weighted&#39;)<br>        print(&quot;  Accuracy: {0:.3f}&quot;.format(test_accuracy))<br>        print(&quot;  F1-Score: {0:.3f}&quot;.format(test_f1score))<br>        print(&quot;  Recall: {0:.3f}&quot;.format(test_recall))<br>        print(&quot;  Precision: {0:.3f}&quot;.format(test_precision))<br><br>        # Calculate the average loss over all of the batches.<br>        avg_test_loss = total_test_loss / len(test_dataloader)<br>        avg_test_loss = avg_test_loss.item()<br><br>        # Measure how long the validation run took.<br>        test_time = format_time(time.time() - t0)<br><br>        print(&quot;  Test Loss: {0:.3f}&quot;.format(avg_test_loss))<br>        print(&quot;  Test took: {:}&quot;.format(test_time))<br><br>        # Record all statistics from this epoch.<br>        training_stats.append(<br>            {<br>                &#39;epoch&#39;: epoch_i + 1,<br>                &#39;Training Loss generator&#39;: avg_train_loss_g,<br>                &#39;Training Loss discriminator&#39;: avg_train_loss_d,<br>                &#39;Test. Loss&#39;: avg_test_loss,<br>                &#39;Test. Accuracy.&#39;: test_accuracy,<br>                &#39;Test. Weighted F1 Score&#39;: test_f1score,<br>                &#39;Test. Weighted Precision&#39;: test_precision,<br>                &#39;Test. Weighted Recall&#39;: test_recall,<br>                &#39;Training Time&#39;: training_time,<br>                &#39;Test Time&#39;: test_time<br>            }<br>        )<br><br>    print(&quot;\nTraining complete!&quot;)<br>    print(&quot;Total training took {:} (h:mm:ss)&quot;.format(<br>        format_time(time.time()-total_t0)))<br><br>    return training_stats</pre><p>The training function above would run for the other GAN-BERT variations with most of the parameters being kept constant to ensure fair comparison of results.</p><h3>Model Results</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/890/1*Fj1rLhFt8UkNkDrQo8H_ag.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/890/1*HQVrpA_PX3KLIQIqbc-Q1A.png" /></figure><h3>Model Results Analysis of Results</h3><p>Right off the bat, we can notice that GAN-ROBERTa and GAN-DeBERTa are blanked out. This is due to technical issues that we were unable to rectify before week 13, causing the model to fail to converge. The notebook is able to run smoothly otherwise under 10 and 25 epochs. As seen, RoBERTa trumps the other models in terms of performance for non-GAN runs and has the highest overall performance except for test time which is by DistilBERT, which specializes in speed. In terms of performance, DistilBERT performs the best, followed by DeBERTa, FinBERT and then BERT-base. With the GAN augmentation, there is a major increase in absolute performance like F1-score, accuracy and precision although the ranking of the model remains similar. As to why the models might perform differently relative to one another, the reasoning stands to be the same as to why the models perform as such under non-GAN conditions. For the GAN models, we realized that going beyond 10 epochs would result in the model possibly overfitting and therefore, causing the test results to drop a bit.</p><h3>Challenges Faced</h3><p>A difficulty we faced was in deciding the project direction and we initially failed to realize the training process we implemented at first would differ slightly from the training process used for the GAN models and we had to rewrite quite a bit of code to ensure that the training process between the GAN and non-GAN models are similar so as to ensure a fair comparison between them. This eventually led to last minute realization that RoBERTa had issues running together with GAN and left us very little buffer to fix issues.</p><h3>References</h3><p>Croce, D., Castellucci, G., &amp; Basili, R. (2020). GAN-BERT: Generative Adversarial Learning for Robust Text Classification with a Bunch of Labeled Examples. <em>ACL Anthology</em>. <a href="https://doi.org/10.18653/v1/2020.acl-main.191">https://doi.org/10.18653/v1/2020.acl-main.191</a></p><p>Owen, L. (2022, January 28). Semi-supervised Intent Classification with GAN-BERT. <em>Medium</em>. <a href="https://towardsdatascience.com/semi-supervised-intent-classification-with-gan-bert-934d8659bca2">https://towardsdatascience.com/semi-supervised-intent-classification-with-gan-bert-934d8659bca2</a></p><p>Liu, Z., Huang, D., Huang, K., Li, Z., &amp; Zhao, J. (2020). Finbert: A pre-trained financial language representation model for financial text mining. Proceedings of the Twenty-Ninth International Joint Conference on Artificial Intelligence. <a href="https://doi.org/10.24963/ijcai.2020/622">https://doi.org/10.24963/ijcai.2020/622</a></p><p>Context-Aware Legal Citation Recommendation using Deep Learning — Scientific Figure on ResearchGate. Available from: <a href="https://www.researchgate.net/figure/The-RoBERTa-model-architecture_fig2_352642553">https://www.researchgate.net/figure/The-RoBERTa-model-architecture_fig2_352642553</a>[accessed 30 Apr, 2024]</p><h3>Link To Notebooks</h3><p>BERT Variations Notebook:</p><p><a href="https://www.kaggle.com/code/nguyncaoduy/bert-intent-classification">https://www.kaggle.com/code/nguyncaoduy/bert-intent-classification</a></p><p>GANBERT Variations Notebook:</p><p><a href="https://www.kaggle.com/code/nguyncaoduy/ganbert-pytorch/notebook">https://www.kaggle.com/code/nguyncaoduy/ganbert-pytorch/notebook</a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=29ea2f5ef6a4" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Forecasting Tomorrow’s Stock Price with Today’s News]]></title>
            <link>https://medium.com/@nusfintech.ml/forecasting-tomorrows-stock-price-with-today-s-news-2fbeed129b77?source=rss-35fbff2bc022------2</link>
            <guid isPermaLink="false">https://medium.com/p/2fbeed129b77</guid>
            <category><![CDATA[sentiment-analysis]]></category>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[nlp]]></category>
            <category><![CDATA[stock-market]]></category>
            <dc:creator><![CDATA[Machine Learning @ NUS Fintech Society]]></dc:creator>
            <pubDate>Sat, 27 May 2023 14:09:20 GMT</pubDate>
            <atom:updated>2023-05-27T14:13:16.847Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/730/1*cf2Wd-MUldIwD01kOAys1A.jpeg" /></figure><p><strong>Authors:</strong> <a href="https://www.linkedin.com/in/daniel-twh/">Daniel Tan Wei Hao</a> (Tech Lead), <a href="https://sg.linkedin.com/in/kleonang">Kleon Ang Cunrong</a> (Tech Lead), <a href="https://www.linkedin.com/in/teo-wei-ming/">Teo Wei Ming</a>, <a href="https://www.linkedin.com/in/zheng-yilin/">Zheng Yilin</a>, <a href="https://www.linkedin.com/in/chan-zhuo-yang-a164ba1b2/">Chan Zhuo Yang</a>, <a href="https://www.linkedin.com/in/brandon-chiu-xi/">Brandon Chiu Xi</a>, <a href="https://www.linkedin.com/in/hans-neddyanto-tandjung/">Hans Neddyanto Tandjung</a></p><h3>Introduction</h3><p>Can computers understand human language as well as humans can? The field of Natural Language Processing (NLP) seeks to find the answer to that question. Machine Learning is used in NLP to process large amounts of text data and extract useful information from this data. In particular, sentiment analysis is a subfield of NLP that employs such Machine Learning techniques to derive emotions or connotations behind text data.</p><p>The algorithms used in sentiment analysis typically classify text into categories of positive, negative or neutral sentiment. This can alternatively be represented by a sentiment score which can range from -1 (negative sentiment) to 0 (neutral sentiment), up to 1 (positive sentiment).</p><p>In finance, sentiment analysis has numerous applications. In this project, we were interested to explore how the sentiment of news articles on certain stocks can impact their performance across different time horizons. We explored the efficacy of training a variety of Machine Learning models (including Logistic Regression, Random Forest, XGBoost, and BERT) to predict the performance of publicly listed stocks with respect to a market index (S&amp;P500). This analysis was done over 3 short-medium term time periods of 3 months, 6 months, and 1 year.</p><p>We also made an API and interactive web application that is able to summarise the collected data that we used to train our Machine Learning models. This presents information including stock information on given tickers as well as related sentiment scores and news data.</p><h3>Data Preprocessing</h3><p>The financial news was obtained through a get request to one of the API endpoints from Marketaux, a stock market and finance news provider. The API endpoint we have leveraged on is <a href="https://api.marketaux.com/v1/news/all">https://api.marketaux.com/v1/news/all</a> and an example of the parameters we have passed in to obtain all the news for Apple within the past 1 year is as follows:</p><pre>{<br>   “api_token”: &lt;Your API token found on your account dashboard&gt;,<br>   “symbols”: “AAPL”,<br>   “published_before”: 2021-12-16<br>   “Published_after”: 2022-12-16<br>}</pre><p>For more information, please refer to their comprehensive <a href="https://www.marketaux.com/documentation">documentation</a>. Be sure to familiarise yourself with the verbose response object so as to be able to retrieve the fields relevant to your objective. Do also take note of the rate limit for each plan and implement workarounds to circumvent the limitations if you wish to have large amounts of historical financial news to work with.</p><p>The S&amp;P 500 historical prices were obtained from Yahoo Finance, using the yfinance Python library, and from the Center for Research in Security Prices (CRSP) database provided by Wharton Research Data Services. The CRSP database provided us detailed daily and even hourly financial information about the stock. However, we chose to only use the daily historical prices for this project.</p><p>For each stock ticker and date, its price 3 months, 6 months and 1 year after the date is taken (if the future date is a weekend, the next day is taken). The percentage change is calculated. This is compared with the percentage change in the S&amp;P 500 price during the same time period. If the percentage change is less than a certain threshold, the data will be labelled 0. If the change in stock price is at least threshold% more than that of the S&amp;P 500 prices, it has outperformed the S&amp;P 500 and the data is labelled as 1. Else the stock has underperformed and is labelled as 2.</p><p>The threshold helps to enable an even split of data between the 3 different labels. It is determined by iterating through different threshold values to find one that produces the highest entropy.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*ze4aSNBelT47mSp_atGp6w.png" /><figcaption>Time Periods, Thresholds, and corresponding Entropies of our Dataset</figcaption></figure><h4>Preprocessing Steps</h4><p>Both the labelled market data and news data contain the stock ticker and date field. The labels were transferred to the news data by finding the corresponding label for the same ticker and news date in the market data.</p><p>A sample of our final pre-processed data is as follows:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*CXIKRwsm2Sv_5RbuiMXrTA.png" /></figure><h3>Model Training</h3><h4>BM25</h4><p>BM25 (Best Match 25) is a bag-of-words retrieval function which ranks a set of documents based on the appearance of query terms in each document while ignoring the proximity of the query terms. It is a ranking function used by search engines to determine the degree of relevance for a set of documents with respect to a search query and it can be seen as an improvement from TF-IDF.</p><p>The scoring function of BM25 is as follows:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*ig3EssGIHrpEUCUSaaU8lw.png" /><figcaption>Source: <a href="https://en.wikipedia.org/wiki/Okapi_BM25">https://en.wikipedia.org/wiki/Okapi_BM25</a></figcaption></figure><p>To have a better understanding of BM25, we shall dive into the components of the BM25 scoring function.</p><p><strong>Term Frequency (TF)<br></strong>TF is denoted by <em>f(q</em>ᵢ<em>, D)</em> in the formula above. It is the number of times a term appeared in a document. For example, we have the following set of documents:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*qKEhR3ivwGXLugazFDH6yw.png" /></figure><p>The term frequency of ‘a’ for document 1, f(qa , D1) is 2. This is because the term, ‘a’, appeared twice in document 1.</p><p>Intuitively, a document should be more relevant if a keyword appears often. As such, the BM25 score increases with higher TF.</p><p><strong>Document Frequency (DF)<br></strong>DF is the number of documents containing a particular term. Referring back to the previous example, the document frequency for the word ‘a’, DF(‘a’), is 2. This is because the word ‘a’ appeared in documents 1 and 2. Note that we do not count the number of times the word ‘a’ appeared in the set of documents.</p><p><strong>Inverse Document Frequency (IDF)<br></strong>IDF is the logarithmically scaled inverse fraction of the documents that contain the query term.</p><p>To get this, we will first calculate the fraction of the documents that contain the query term. This can be obtained by dividing the total number of documents, N, by DF. Thereafter, we will simply apply the logarithm to the inverse of this fraction.</p><p>Mathematically, it is shown by this formula:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/456/1*ti1X8ajfvKToS0bFYDp9cA.png" /></figure><p>where n(qᵢ) is the DF for query term qᵢ and N is the total number of documents.</p><p>IDF measures the frequency of the search term in all the documents and penalises those search terms which are common (n(qᵢ) is in the denominator).</p><p>This is to account for the scenario where the search terms are common words. For common words, they tend to have high TF and DF values. However, these search terms might not be useful in determining the degree of relevance. Without the IDF portion, the BM25 score will be unfairly high for search terms containing common words due to its high TF value. As such, we have to introduce a penalty (IDF) to reduce the impact of high TF on the BM25 score.</p><p>There are various variants for the calculation for IDF but the core idea remains the same. For BM25, we usually use the probabilistic IDF which is as follows:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/676/1*4F_iKO5IdL41far4iivIPQ.png" /></figure><p><strong>Explanation of (<em>|D| / avgdl</em>) in BM25 Scoring Function<br></strong><em>|D| </em>denotes the length of the current document while avgdl denotes the average length of all documents. This term can be seen as the current document length relative to the average document length and it will have a high value for long documents. As this term is in the denominator, we penalise documents which are long. Intuitively, this makes sense as it is more likely, and less special, for long documents to contain the search terms. Assuming ceteris paribus, a shorter document is more relevant than a long document with the same TF value.</p><p>In summary, the above components make up the BM25 scoring function. The value of the BM25 score will indicate the degree of relevance for the respective documents.</p><h4>TF-IDF</h4><p>TF-IDF measures the relevance of a word in a document amongst a collection of documents. The TF-IDF score of words in the title of each news article were used as features to train the machine learning models.</p><p>TF-IDF is calculated by multiplying two metrics, term frequency (TF) and inverse document frequency (IDF). The term frequency of a word in a document is the number of times the word appears in the document. The inverse document frequency then tells us how common a word is among the collection of documents. It can be calculated by dividing the total number of documents by the number of documents containing the word, then calculating its logarithm. Terms like “is”, “the” and “of” appear frequently among all documents, hence their very low IDF score helps to minimise their weight while scaling up rarer terms that are likely to have a higher impact on a particular document.</p><p>The preprocessing of news titles is as follows. First, all words containing non-english characters were removed. Then, the words are converted to lower case and lemmatised. Finally, TFIDF scores of each word in each news title was obtained using the sklearn.feature_extraction.text.TfidfVectorizer class. These scores are used as features to train the machine learning models, logistic regression, random forest and XGBoost.</p><p>Below are the results:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*nLS01vSD7y2oE4kssBeQ5g.png" /></figure><h4><strong>FinBERT</strong></h4><p>Another model we have explored is FinBERT. FinBERT is a pre-trained NLP model to analyze sentiment of financial text. It is built by further training the BERT language model in the finance domain, using a large financial corpus and thereby fine-tuning it for financial sentiment classification. Specifically, we have leveraged on the pretrained <a href="https://huggingface.co/ProsusAI/finbert">ProsusAI/finbert</a> model on <a href="https://huggingface.co/docs/transformers/index">HuggingFace</a> to perform transfer learning on our labelled dataset mentioned earlier after customising our trainer to account for the imbalance in our dataset.</p><p>The training arguments we eventually settled on are as follows:</p><pre>num_train_epochs=5,<br>warmup_ratio=0.1,<br>weight_decay=0.01,<br>learning_rate=2e-5,<br>per_device_train_batch_size=16,<br>per_device_eval_batch_size=16,<br>evaluation_strategy =&quot;epoch&quot;,<br>save_strategy = &quot;epoch&quot;,<br>save_total_limit = 1, # only best model is saved - older ones overwritten<br>load_best_model_at_end=True,<br>lr_scheduler_type=&#39;cosine&#39;</pre><p>However, we could only manage to achieve an accuracy of 0.74 on the validation set and 0.57 on the test set, which are figures slightly worse than what we expected.</p><h4><strong>BERT</strong></h4><p>We also explored the original BERT base cased model for our news sentiment analysis. BERT is a pre-trained NLP model that is used to predict the sentiment of any general text, and it is not limited to just financial data. We used the BERT model with 12 layers and H=768.</p><p>The training parameters we used are:</p><pre>epochs=10<br>Batch_size=16<br>Adam optimizer (lr = 0.001)</pre><p>Using these parameters, the test accuracy of the models for the different time frame are as follows:</p><ul><li>3 months: 74%</li><li>6 months: 70%</li></ul><h3><strong>Discussions and Comparisons</strong></h3><p>Since we are working with a multilabel classification task, namely to ascribe each financial news text with either positive, neutral or negative sentiments, we choose accuracy — which measures the closeness of each predicted value to its actual value — as our main metric for evaluation. The accuracy metric is simple, robust and easily interpretable for this use case and could be employed indiscriminately across all models which we have built.</p><p>Other metrics that we could use are:</p><ul><li>3x3 confusion matrix to derive the precision, recall and F1 score for each class from the TP, FP, TN, FN values. This helps to gain a more granular insight into the performance of each class and is a more superior evaluation metric for accuracy for an imbalanced dataset.</li><li>AUC score to assess the probability that the model performs better than random guesses. This typically only works with binary classification, thus multilabel classification metrics in our case have to be first converted into a binary one for AUC to be applicable.</li><li>Root Mean Squared Error (RMSE) could also be applied on the raw sentiment scores that were obtained before rounding to the corresponding labels.</li></ul><h4>Performance Comparison</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*r8hi3sVwkwDBmKl4iTWX9g.png" /><figcaption>Bar chart showing accuracy of the models over periods of 3 months, 6 months, and 1 year.</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/876/1*LRJqGwxcAy5Q33aBBny6-g.png" /><figcaption>Table showing accuracy of models over periods of 3 months, 6 months, and 1 year.</figcaption></figure><p>From the bar chart and table above, it is observed that generally the performance of TFIDF models are the worst, followed by BM25 and BERT models respectively. Based on literature readings, we think that the low performance of TFIDF might be due to TFIDF models’ inability to carry semantic meaning. The model simply considers the importance of the words based on its frequency in the text and corpus, thus it misses contextual information of the text. In addition, similar to Bag of Words, TFIDF ignores word order, which makes processing compound nouns that appear in financial texts a lot impossible. This extends to negation such as “doing well” and “not doing well”. Lastly, TFIDF suffers from the curse of dimensionality as the length of TFIDF vectors is equal to the tremendous size of the vocabulary, which makes us unwieldy for clustering purposes (labelling sentiments as -1, 0, or 1)</p><p>BM25 can be thought of as a further improvement of TFIDF, as it also considers the length of the text and term frequency saturation. However, BM25 faces the same problems of inability to carry semantic and contextual meaning, ignorance towards word order, and the curse of dimensionality, just like TFIDF.</p><p>In comparison, BERT models perform best. We hypothesize this difference can be attributed to its ability to take in contextual meanings, which are paramount in analyzing financial and stock sentiments across huge amounts of textual information. By converting words into vectors and numbers, BERT models were able to analyse the contexts and semantic meanings of the words, as opposed to just looking at the frequency of the words appeared and document length.</p><h4><strong>Logistic Regression vs Random Forest vs XGBoost</strong></h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*6PLIXlGxqwa3rzo9j8eOgA.png" /></figure><p>In general, BM25 performs much better than TFIDF for all 3 models. This is because BM25 performs better at predicting the relevance of a word in a document, with its enhanced TF score calculation. For BM25, the impact of term frequency on the TF score is dampened as it approaches a value asymptotically. This helps to account for the problem of diminishing returns as the frequency of a word increases to the point of saturation. Furthermore, BM25 considers the document length in calculating TF score. For shorter documents, the TF score increases much faster and hits the asymptote earlier. This is because shorter documents have fewer terms, and it takes fewer matches to be confident about the relevance of the word.</p><p>Across the three supervised learning models, the accuracies are very similar.</p><h4><strong>BERT vs FinBERT</strong></h4><p>FinBERT is essentially a BERT based language model trained on additional corpus of financial data to obtain a deeper understanding of financial language.</p><p>However, FinBERT is not performing better than BERT in our case. One plausible reason could be that even if FinBERT has been pre-trained and fine-tuned with financial text data, the specific documents that were leveraged in FinBERT may still be significantly different from the financial news text that we are using for transfer learning and prediction. Besides, most of the financial news texts that we are passing in are made up of long sentences and are enriched with an abundance of vocabularies which BERT is able to better cater to as compared to FinBERT.</p><h4><strong>Despite lower performance for BM25 and TFIDF, are there any cases we choose to use it over BERT?</strong></h4><p><strong>Limited computational resources<br></strong>As BERT uses deep learning network as its architecture, runnung a BERT model is computationally expensive. In contrast, TF-IDF/ BM25 does not employ deep learning network. In fact, they make use of rather simple calculation, making them computationally cheap.</p><p><strong>Starting point for a new analysis<br></strong>As TF-IDF/ BM25 are computationally inexpensive, one can make use of TF-IDF/ BM25 as starting algorithms for initial similarity comparison. This will give users a rough estimation of the findings and to let user decide on whether to proceed with more complex algorithms.</p><h4><strong>Future Improvements (How can we potentially boost performance?)</strong></h4><p>In our current model training workflow, we experimented with using different models for a single model training “step”, in which one model was trained to take in text data and directly predict the relative performance of stocks to the index based on the raw text input.</p><p>Instead, we could also have explored using a 2-step approach, whereby one model was used to predict a sentiment score given the raw text input, which is then fed to another model (could be the same class of models) that in turn predicts relative stock performance.</p><p>This alternative approach could allow us to use different models that may be more suited to the two individual sub-tasks — predicting sentiment scores from text and predicting stock performance from sentiment scores — rather than using the same model for both sub-tasks.</p><p>We encourage future studies to explore this approach, especially when models that are well-known to predict sentiment scores from text data accurately (e.g. BERT) are used for the first step. The second step can then be explored using a range of models (e.g. TFIDF/BM25 with Neural Networks, Decision Trees, Naïve Bayes classifiers)</p><h3><strong>Challenges Encountered</strong></h3><p>To anyone who may wish to replicate our project or embark on research of a similar nature, we provide some of the challenges we encountered which may serve as tips to look out for in your own exploration of news sentiment analysis.</p><h4>Data Collection</h4><p>One of the data-related challenges we encountered was the lack of a comprehensive news API that was easily accessible from our sources. Although we had access to databases on the Wharton Research Data Services (WRDS) for this project, we were unable to find a suitable database from the service that had comprehensive historical news data. As a result, we had to use a free online API (marketaux) instead of a commonly used industry standard data service such as WRDS.</p><p>With the free plan on marketaux, we also faced query quotas due to the API’s rate limiting. Hence, we were unable to query all the data that was required from a single API key. To resolve this, we had to collate multiple API keys (registered using the credentials of our team members) and coordinate the news data from the stocks queried amongst the numerous API keys. Nevertheless, there always remains the option of purchasing a higher plan for such online services which would increase or completely remove the API’s querying quotas.</p><h4><strong>Model Training</strong></h4><p>It was challenging to fine tune the pretrained language models to obtain satisfiable accuracy (above the baseline accuracy of simpler machine learning models). It required experimentation with different training parameters such as learning rate, batch size and optimizer. We settled on Adam optimizer with a small learning rate, (10^–5 to 10^-6) and an initial number of warmup steps. Overfitting was also observed, and dropout/early stopping were some of the techniques used to improve model performance on unseen data. Due to limitations in compute resources, we were also unable to try certain configurations such as with large batch sizes (32 or greater) or more complex pretrained models.</p><h3>References</h3><p><strong>TFIDF</strong>: <a href="https://www.capitalone.com/tech/machine-learning/understanding-tf-idf/">https://www.capitalone.com/tech/machine-learning/understanding-tf-idf/</a></p><p><strong>BM25</strong>: <a href="https://kmwllc.com/index.php/2020/03/20/understanding-tf-idf-and-bm-25/">Understanding TF-IDF and BM-25 — KMW Technology (kmwllc.com)</a></p><p><strong>BERT</strong>: <a href="https://towardsdatascience.com/bert-why-its-been-revolutionizing-nlp-5d1bcae76a13">BERT: Why it’s been revolutionizing NLP | by Jerry Wei | Towards Data Science</a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=2fbeed129b77" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Crypto Fraud Detection]]></title>
            <link>https://medium.com/@nusfintech.ml/crypto-fraud-detection-3d99b3298815?source=rss-35fbff2bc022------2</link>
            <guid isPermaLink="false">https://medium.com/p/3d99b3298815</guid>
            <category><![CDATA[blockchain]]></category>
            <category><![CDATA[fraud-detection]]></category>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[cryptocurrency]]></category>
            <dc:creator><![CDATA[Machine Learning @ NUS Fintech Society]]></dc:creator>
            <pubDate>Sat, 27 May 2023 13:21:59 GMT</pubDate>
            <atom:updated>2023-05-27T13:21:59.463Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*DEG-vN2W7a0uui4UnfuYJA.png" /></figure><p><strong>Authors:</strong> <a href="https://www.linkedin.com/in/han-wei-lun-04663519b/">Han Wei Lun</a> (Tech Lead), <a href="https://www.linkedin.com/in/austin-loh-b38239147/">Austin Loh</a>, <a href="https://www.linkedin.com/in/jotham-wong-499951174/">Jotham Wong</a>, <a href="https://www.linkedin.com/in/yienong/">Ong Yi En</a>, <a href="https://www.linkedin.com/in/shaunangcx/">Shaun Ang Chong Xiong</a>, <a href="https://www.linkedin.com/in/elvern-tan-a08405216/">Tan Jian Thong Elvern</a></p><h3><strong>Introduction</strong></h3><h4>Background</h4><p>2021 proved to be a record year for the cryptocurrency market, which briefly surpassed the $3 trillion mark. Bitcoin, the largest cryptocurrency by market value, and ether, the second-largest, hit all-time highs, while altcoins, like meme-inspired dogecoin, surged in popularity. This growth fueled the likes of many retail and institutional investors to enter the space, in hopes of owning a piece of one of the most disruptive technologies of the 21st century.</p><blockquote>There’s a long tail of retail investors who have invested in crypto assets. Do they all understand what they’ve invested in? I think not. For that long tail of retail investors, I’m not sure they do understand.<br>- Jon Cunliffe, deputy governor for the UK central bank</blockquote><p>However, quoting Cunliffe, it seems like most retail investors who “invested” money into digital currencies probably don’t understand exactly what they bought or the risks involved.</p><p>This is a worrying sign, because cryptocurrencies have evolved so quickly that apart from tokens like bitcoin and ethereum, we now have things like non fungible tokens (NFT), in-game assets and digital land. On one end, this is a good sign because the industry is heading towards becoming more utilitarian in nature. However, the flip side is that many don’t even know what these things are or how they work before investing in them, making them more susceptible to getting scammed.</p><p>In fact, it was reported that the amount of money lost in crypto scam worldwide was a whopping USD$7.7 Billion in 2021 alone. The fact that the reported losses to cryptocurrency scams in 2021 was nearly sixty times more than what was reported in 2018 is a further indication that the infrastructure of cryptocurrency security has lagged the growth of the cryptocurrency market.</p><h4>Motivation</h4><p>One of the primary reasons for crypto’s rise to fame is because of its idea of being decentralized, which means that there isn’t a central governing body to dictate how things should be run. This makes a popular idea for many especially since they like the idea of not having all the power lie on 1 party or organization. However, this is a huge issue considering the fact that with no governing body, nobody can revert any mistakes that are made in the decentralized world. To give an example, if you’ve mistakenly made a transaction through the traditional bank, there is a good chance that the bank may be able to recover the money that you’ve sent. In the decentralized world, this would not be possible. If you happen to send some crypto from your wallet to an address, which is something like a bank account, and that address happens to belong to a scammer, there is a very good chance that your crypto will be gone forever since there is no one able to help revert the transaction or retrieve your crypto back. As much as there is visibility of where your crypto is going on the blockchain, there isn’t a central entity you can approach to crack down on stopping the scammer, or ask to not process transactions relating to a particular address, unlike in the traditional banking system.</p><p>Majority of these crypto scams are either investment-related, romance manipulations, or someone acting as a business and government imposter, and their end goal is usually to get the victims to send crypto over to them through the blockchain, which being decentralized, means that victims can never recover any crypto they’ve sent.</p><h4>Goal</h4><p>With this contextual understanding, isn’t it true that prevention (avoiding sending crypto to scammers) is a better option than cure (retrieving sent crypto). This is why the aim of this project is to develop an application whereby users can check whether an address is a potential fraudulent one before making the irreversible decision of sending crypto over to that address.</p><p>This will be done by developing machine learning models which use common attributes of crypto addresses to make calculated decisions on whether an address might be potentially fraudulent.</p><h3>Data Preparation</h3><h4>Data Collection</h4><p>Note that we will classify all kinds of wallets and smart contracts together as addresses from here on.</p><p>For the purpose of this project, we dealt only with addresses from the ethereum chain, since it has the most robust data reporting when it comes to fraudulent transactions. All of our data was obtained from <a href="https://etherscan.io/">etherscan.io</a>, a block explorer where all information about ethereum related addresses and transactions can be found.</p><p>The first part of data collection involved going to etherscan to retrieve the list of fraudulent crypto addresses with labels “Phish/Hack/Exploit/Scam/Fraud”, followed by collecting normal addresses to build a dataset with a mix of fraud and non-fraudulent addresses, in a ratio of around 1:4.</p><h4>Feature Engineering</h4><p>To get started, we had to create an account in etherscan.io to obtain API keys for us to utilize the etherscan API to get our features. The features that were engineered were</p><ol><li>Numerical balance of wallet</li><li>Total transaction count</li><li>Total sent transactions</li><li>Total received transactions</li><li>Total ether sent</li><li>Max, Min, Average and Total Ether Sent</li><li>Max, Min, Average and Total Ether Received</li><li>Number of unique sent address</li><li>Number of unique received address</li><li>Average time difference between each transaction</li><li>Average time difference between each sent transaction</li><li>Average time difference between each received transaction</li><li>Total ERC20 transactions</li><li>Total ERC20 ether sent</li><li>Total ERC20 ether received</li><li>Number of Unique ERC20 received address</li><li>Number of Unique ERC20 sent address</li></ol><p>More information about these features can be found here <a href="https://www.kaggle.com/datasets/vagifa/ethereum-frauddetection-dataset">https://www.kaggle.com/datasets/vagifa/ethereum-frauddetection-dataset</a>.</p><p>We had to read up on the etherscan API documentation to access these features, and the most essential thing we discovered is that we need to figure out which API endpoint to use each feature.</p><p>We have constraints because we are utilizing the free version. We are only permitted to make 5 API calls per second and up to 100,000 API calls per day. Although sufficient, we are unable to perform multithreading due to API restrictions. It took roughly 5–7 hours to get each wallet function with over 20,000 wallet addresses. We divided the work amongst ourselves and completed these features in two weeks.</p><h4>Feature Selection</h4><p>We decided to fill up the missing data with 0’s or the string “None” as we found that most crypto addresses that had missing data were due to very little or even no transactions being made. After filling up our data, we made use of sklearn label encoder to transform categorical values to numerical ones. This was done to allow our models and tests to process the data.</p><p>Random Forest and Logistic Regression models were then trained from the transformed data and from these models, we obtained the features that were deemed important by the models. The 2 models were used as they had in-built functions that allowed us to easily obtain the feature importance. Statistical tests such as Chi-square and F-test were also used to find the important features. By inner joining the results, we chose the 22 most important features that were consistent with all 4 tests.</p><h3>Model Training</h3><h4>Evaluation Metric</h4><p>Ideally, we want to maximize accuracy when training a machine learning model. However, since our models are for fraud detection, the worst prediction to give is a prediction that an address is not fraudulent, when it actually is. This false negative prediction will result in heavier consequences than a false positive prediction. Hence, our goal was to minimize false negative rate, and thus maximize Recall.</p><p>Formula for Recall is:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/802/1*xrj67AT_n1VxQoyZ2-v51g.png" /></figure><h4>Ensemble Models</h4><p>To determine a suitable model for our purpose, we tried a number of models from different ensemble learning methods — namely bagging, stacking and boosting. We mainly used the sklearn library as it provides a comprehensive number of models for us to test and has well-supported documentations.</p><p>For bagging, we used the Random Forest algorithm. For stacking, we made use of Support Vector Machine, Logistic Regression, K nearest Neighbors and HistGradientBoosting from sklearn library. And for boosting, we utilized the HistGradientBoosting algorithm.</p><p>Among these 3 methods, Random Forest and HistGradientBoosting had better recall rates and were approximately the same after using Cross validation to evaluate their performance.</p><p>However, both models’ performance were still slightly inferior to XGBoost’s recall rate.</p><h4>XGBoost</h4><p>XGBoost (Extreme Gradient Boosting) is a scalable, distributed gradient-boosted decision tree machine learning library which attempts to accurately predict a target variable (fraud or non-fraud in this case) by combining estimates of a set of simpler and weaker models.</p><p>We performed hyperparameter tuning to the naive model in order to optimize our model.</p><p>We would like to highlight that, by observing our dataset, we realized that there is an uneven proportion of fraudulent and non-fraudulent addresses. Hence, the dataset is imbalanced. Thus, it is pivotal to tune the ‘scale_pos_weight’ parameter in order to make the dataset more balanced. This parameter allows us to assign different weights, and thus different importance, to the fraudulent and non-fraudulent addresses.</p><p>Upon several iterations of fine tuning, our XGBoost model gave a consistent recall score of over 95% on cross-validation. As for the performance of this model on the validation set, we reflect it through the use of the confusion matrix below.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/762/1*41qA_nZ2d0FlVy-FbKy3DA.png" /></figure><p>We calculate that the recall is 91.95%, which indicates that our model generalizes well to new data and is not an overfitting one. Hence, we proceeded to using this model at hand.</p><h4>LIME</h4><p>As opposed to white-box models such as Linear Regression and Decision Trees, XGBoost is a novel machine-learning algorithm belonging to the family of gradient-boosting machines that is often associated as black-box ML models. Meaning, the XGBoost model reaches a decision — in this case, Fraudulent or not — without explaining or showing how they did so. The internal processes used and the various weighted factors remain unknown. In other words, there is a lack of transparency in this technology. Often, this is also related to what humans are interested in when observing the output of a model. The most common question is probably: why was this prediction made or which variables caused the prediction?</p><p>To answer this question, we adopted the Local Interpretable Model-agnostic Explanations (LIME). It is a technique that approximates a black box machine learning model with a local, interpretable model to explain each individual prediction. The output of LIME is a list of explanations, reflecting the contribution of each feature to the prediction of a data sample, otherwise known as Feature Importance.</p><p>We figured that this is very useful for users, allowing them to verify hypotheses and determine which feature changes will have the most impact on the prediction. Additionally, it sheds light on whether the model is overfitting to noise, which is helpful to our team during the model building, model tuning, and model selection process.</p><h3><strong>Developing A Web Application</strong></h3><h4>Backend</h4><p>For the backend, we utilized Flask, which is a micro web application framework written in Python Language. It is extensively used by many programmers since, unlike other frameworks like Django, it is very pythonic; also, it is simple to use and does not have a steep learning curve.</p><p>We wanted our LIME explainer and model results to be visible on the frontend, therefore loading the explainer and model from the backend was critical. We saved the LIME explainer as a pickle file and the XGBoost model as a “.dat” file. After numerous trial and error, for the web application to work, we had to use the library dill to load the LIME explainer and joblib.load to load the XGBoost model.</p><p>Moving forward, we developed a predict function within the flask script to enable users of a web application to input a wallet of their choice. The function will then return the prediction outcome (Fraud/Non-Fraud), probability that the wallet is a fraud wallet, as well as the weightage of the wallet features that the model considered when coming to its conclusion.</p><h4><em>Frontend</em></h4><p>We created an interactive web application for users to search for the cryptocurrency wallet address, and the prediction (Fraud/Not Fraud), along with the wallet’s features and their importance will be displayed on the webpage.</p><p>The landing page of the web application is the ‘Prediction’ Tab. As an additional feature, users are able to toggle to the ‘Microservice’ tab for an alternative way to obtain a prediction for the cryptocurrency wallet address.</p><p>We used Streamlit, an open-source app framework in Python language, to build this web application. It is compatible with major Python libraries such as NumPy, pandas, as well as XGBoost and LIME, which we utilized.</p><p>A brief overview of the application can be seen in the images below.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*mtKlAfftZN1Hrb6k9ZzN2Q.png" /></figure><h4>Deploying to Cloud</h4><p>With the backend and frontend systems in place, we hosted our application on a cloud server so that others could access this web application. This was done via Google Cloud Run, where we set up containers for two unique Docker images, one for the frontend and one for the backend. Once deployed, when a user logs onto the website and enters a crypto an address, an API call will be made to the backend to get the necessary features, before returning the results in a structured format. An idea of the underlying infrastructure is provided in the image below.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*9fIFFtXEaYFQiCM_CwoqYw.png" /></figure><h3>Conclusion</h3><p>To summarize this article, we’ve motivated the pressing need for solutions that will help prevent crypto users from sending crypto over to fraudulent parties, and described how we could use machine learning to build a predictive system to warn users about potential fraudulent addresses before sending crypto over to them. Finally, we provided an overview of the development of the MVP that we’ve created for this idea.</p><h3>References</h3><ol><li><a href="https://www.cnbc.com/2021/12/27/12-key-moments-that-fueled-cryptos-record-growth-in-2021.html">12 key moments that fueled crypto’s record growth in 2021 (cnbc.com)</a></li><li><a href="https://www.bloomberg.com/news/articles/2022-05-17/crypto-assets-poorly-understood-by-retail-investors-boe-says?leadSource=uverify%20wall">BTC Bitcoin, Crypto Assets Poorly Understood by Retail Investors, BOE Says — Bloomberg</a></li><li><a href="https://www.youtube.com/watch?v=ErIqa72RB8Y">(10) Crypto scams the ‘biggest source of revenue for criminals’ in the industry: Expert — YouTube</a></li><li><a href="https://finance.yahoo.com/news/crypto-scams-conned-thousands-people-181252512.html">Crypto scams conned thousands of people out of over $1 billion since 2021, the most of any type of currency, according to new FTC report (yahoo.com)</a></li></ol><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=3d99b3298815" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Green Fintech: Real or Scam?]]></title>
            <link>https://medium.com/@nusfintech.ml/green-fintech-real-or-scam-f0a95a86b448?source=rss-35fbff2bc022------2</link>
            <guid isPermaLink="false">https://medium.com/p/f0a95a86b448</guid>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[sustainability]]></category>
            <category><![CDATA[green]]></category>
            <dc:creator><![CDATA[Machine Learning @ NUS Fintech Society]]></dc:creator>
            <pubDate>Mon, 20 Jun 2022 15:12:41 GMT</pubDate>
            <atom:updated>2023-06-19T16:10:36.742Z</atom:updated>
            <content:encoded><![CDATA[<p>Authors: <a href="https://www.linkedin.com/in/jack-shi-wl/">Jack Shi Wei Lun</a> (Research Lead), Lee Ren Yi Sean, Lim Hao Shen Keith</p><h3>Introduction</h3><p>Over the decades, the financial sector embraced cutting-edge technology and saw the rise of FinTechs, using smartphones, application of artificial intelligence (AI) and machine learning (ML), data analytics, the Internet of Things (IoT), distributed ledgers (blockchain technologies) and the likes, have entirely rattled the way financial institutions operate and service their customers. FinTech has already shown its ability to disrupt and accelerate innovation in the financial system, and by adding one of our society’s most existential threats to the mix, you create an astonishing combination that might just have the potential to revolutionize the way we think about addressing climate change: green FinTech. Green FinTech and more specifically, climate-related FinTech, gained increased importance on company, national and supranational levels over the years.</p><p>In this article, we aim to give an overview of how green FinTech can play a crucial role in promoting sustainability. The structure of the article will be as follows: Introduction to green FinTech, some successful projects that actively make use of green FinTech, applied technologies and/or application areas for green FinTech and summary. Do note, terms like ‘FinTech’ and ‘finance’ are used interchangeably from here onwards, as well as the terms ‘green’ and ‘sustainable’.</p><h3>What is Green FinTech?</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/660/0*mwGT1p1m--rQijxR" /></figure><p>Green FinTech is a digital financial technology that is catalyzing the decarbonization of our economy. Green FinTech, also synonymous with climate FinTech or sustainable FinTech, is the convergence of three sectors: Climate, Finance and Technology. Green FinTech solutions are digital innovations, applications, and platforms that help institutions and individuals save, spend, and invest in ways that put the planet first.</p><p>Green FinTech already has a massive impact along the whole value chain of financial services covering customer-to-customer (c2c), business-to-customer (b2c), and business-to-business (b2b) services. It connects all relevant participants in the value chain including consumers, (central) banks, insurers, non-banks (startups, big tech firms), (technology) providers, regulators, etc. In addition, as the financial system provides central functions also to other industries, it becomes also increasingly intertwined with other industries enabled by digitization like the energy sector providing digital infrastructures for peer-to-peer payments and many others. There are many potential benefits Green FinTech may bring, or does it? This article seeks to review 3 existing application areas in which green FinTech are emerging: 1) FinTech and general environmental sustainability (i.e. climate, clean energy and water), 2) Blockchain and tokens for environmental sustainability and lastly, 3) FinTech-enabled environmental sustainability for smart cities.</p><h3>Successful projects that made use of Green FinTech</h3><p>Before we go into the 3 application areas of Green FinTech, let us first briefly take a look at some of the successful projects that made use of it and the objectives behind those projects.</p><h3>Project Greenprint (Singapore)</h3><p>Project Greenprint was launched in December 2020 to harness innovation and technology to promote a green finance ecosystem through helping to mobilize capital, monitor sustainability commitments, and measure impact. Project Greenprint is a collection of initiatives that aims to harness technology and data to enable a more transparent, trusted and efficient <a href="https://www.investopedia.com/terms/e/environmental-social-and-governance-esg-criteria.asp">ESG</a> ecosystem to enable green and sustainable finance. Since the announcement of Project Greenprint last December, MAS has engaged the financial industry and other industry sectors to identify potential digital enablers to address the data challenges. These include interoperable data platforms that can aggregate new and existing sustainability data across multiple sectoral platforms and industry players, and enable sharing of the data across different stakeholders.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/641/0*IjmcSkoOiWDToIO4" /></figure><p>Fig. 1, Four common utility platforms to address the financial sector’s sustainability data needs.</p><h3>24 of the most prominent green FinTech startups (Switzerland)</h3><p>Switzerland has great ambitions in green FinTech, as it aims to become a global hub for green digital finance. In order to identify the relevant startups and solutions in Switzerland, a variety of studies, databases, blogs, tweets, alerts and events were analyzed succinctly by <em>Thomas Puschmann, Christian Hugo Hoffmann and Valentyn Khmarskyi, 2020</em>.</p><p>The final analysis in that paper identified 24 out of 361 (traditional FinTech) as the most prominent <em>green</em> FinTech startups in Switzerland, which are briefly described in Table 4 of that paper. Below are just 5 of them (in no particular order) and their respective description of the solution they provide.</p><ol><li><strong>Beedoo</strong> provides an impact investment platform for consumers who aim to invest in sustainable firms and assets.</li><li><strong>Covalence</strong> supports investors in integrating environmental, social and governance (ESG) factors while controlling greenwashing based on an artificial intelligence-based scoring system.</li><li><strong>Energy Web</strong> is a nonprofit organization focusing on a low-carbon, customer-centric electricity system based on blockchain and other decentralized technologies. It focuses on building core infrastructure and shared technology.</li><li><strong>My Drop in the Ocean</strong> develops a global currency platform called DIO. It rewards consumers and businesses for their sustainable actions, in turn, returning value to nature. The platform creates a link between the value of environmental costs and the issuing of a digital currency that can be used by consumers to pay for purchases at participating businesses. DIO is initially issued equitably to individuals as credits, reflecting nature’s shared value to all of us, and are converted to rewards through sustainable actions captured on the online platform.</li><li><strong>Pexapark </strong>develops digital forms of financial power purchase agreements (PPA, also called virtual PPA and synthetic PPA) for firms to purchase renewable energy, which provides financial security for lending institutions, such as banks, to invest in renewables projects.</li></ol><h3>Applied Technologies and/or Application Areas for Green Fintech</h3><p>“When the last tree has been cut down, the last fish caught, the last river poisoned, only then will we realize that one cannot eat money.” — — ‘America Born and Reborn’ by H. Wasserman</p><p>“Sustainable development is a fundamental break that’s going to reshuffle the entire deck. There are companies today that are going to dominate in the future simply because they understand that.” — — François-Henri Pinault</p><p>The only technology that helps to create a plethora of business and employment opportunities while promoting the green environment, is serving good for society and, therefore, sustainability. Here are 3 areas of how green FinTech can be applicable to society.</p><h3>1. FinTech impact on climate change actions, clean energy and clean water</h3><p>The UN Intergovernmental Panel on Climate Change sounded a dire warning in August last year, saying that the world is dangerously close to runaway warming, with greenhouse gas levels in the atmosphere high enough to guarantee climate disruption for decades if not centuries. This sparks huge financial institutions to take climate change actions, like<a href="https://www.businessgreen.com/news/4029982/jp-morgan-citi-pledge-multi-trillion-dollar-green-finance-blitzes"> Citi, JP Morgan Chase</a>,<a href="https://www.environmentalleader.com/2021/03/wells-fargo-to-deploy-additional-500-billion-in-sustainable-finance-by-2030/"> Wells Fargo</a>, <a href="https://www.swissre.com/media/news-releases/nr-20210316-swiss-re-announces-ambitious-climate-targets.html">SwissRe</a>, <a href="https://www.zurich.com/es-es/media/news-releases/2021/2021-0331-01#:~:text=By%202025%2C%20Zurich%20aims%20to,%2Dequivalent%20per%20square%20meter).">Zurich Insurance </a>or <a href="https://group.bnpparibas/en/press-release/bnp-paribas-joins-net-zero-banking-alliance-launched-unep">BNP Paribas</a> pledging trillions to sustainable finance and setting ambitious climate goals.</p><h4>Clean energy and water</h4><p>For many low-income households in the developing world, incomes are highly variable and uncertain. High up-front costs combined with irregular incomes result in unequal access to water and energy access. Households typically can, and should, cover the costs of accessing water and energy resources, but they cannot do this without help. Financial inclusion can help households access water and energy resources. Financial inclusion focuses on ensuring everyone has access to useful and affordable financial products and services. The emerging field of FinTech can help address barriers to financial inclusion in the water sector while potentially reducing or eliminating the need for subsidy. FinTech solutions already address some of the needs of developing-nation households — applications include payments and mobile money, pay-as-you-go (PAYG) models, insurance technology (insurtech), and virtual banks. This <a href="https://www.cgap.org/blog/4-ways-digital-finance-can-help-bring-clean-water-all">article</a> summarized 4 ways digital finance can help bring clean water to all.</p><p>Crowdfunding, another prominent example of FinTech, is also shaping investment forms within the wind and solar PV industry. An increasing amount of solar community projects are installed and financed through crowdfunding. Opportunities for venture capitalists and entrepreneurs in the U.S. to produce clean energy can be accomplished by channeling investments into clean energy and water FinTech start-ups. Plenty of other FinTech innovations also boost the energy transition, such as green bonds, increases in energy efficiency via machine learning, mobile payment solutions, and mobile micro-loans.</p><h3>2. Role of blockchain technology to meet climate change actions such as clean energy production</h3><p>The first major application of blockchain technology was the Bitcoin digital currency in 2009, allowing peer-to-peer transaction of payments online without an intermediary. Since then, research on ways to integrate blockchain into different sectors related to sustainable development is now expanding, and innovation at the intersection of blockchain and climate change is expected to further develop in the coming years.</p><p><strong>Clean energy production</strong></p><p>Blockchain has the potential to promote clean energy production through improved carbon emission trading, facilitated clean energy trading, and better tracking and reporting of greenhouse gas emissions. Blockchain could be used to improve the system of carbon asset transactions, as the recording of carbon assets on a public blockchain would guarantee transparency and ensure that transactions are valid and settled automatically. This is only possible due to the fact that the blockchain is transparent and all transactions on the blockchain can be viewed and audited. Blockchain would also be able to facilitate clean energy trading as the technology would allow for the development of platforms for peer-to-peer renewable energy trade. This would allow consumers to buy, sell and exchange renewable energy with each other using tokens or tradable digital assets, similar to the Bitcoin digital currency. The blockchain technology would also be able to provide more transparency regarding greenhouse gas emissions and make it easier to track and report emission reductions, thereby addressing possible double counting issues and improving the quality and accuracy of data relating to greenhouse gas emissions. These use cases of the blockchain technology would collectively be able to promote clean energy production and improve sustainability.</p><h3>3. Development of smart cities by means of FinTech</h3><p>Smart cities have become increasingly prevalent over the years, where they operate with modern technology to alleviate the daily lives of citizens. Key amenities such as transport, banking and education have been incorporated with such modern technology to improve efficiency and living standards.</p><p>FinTech has multiple roles to play in smart cities, with the most direct one being the modern digital banking system. FinTech facilitates and provides easier, more efficient and personalized banking services to its users. For example, online banking facilities and digital payment services such as contactless payment methods are increasingly being supported by banks with the support of large fintech corporations.</p><p>While such systems, which are only possible due to the advancements in FinTech over the recent years, provide many benefits to the lives of citizens, it also consumes much more energy than other sectors.</p><p><strong>Need for Green FinTech</strong></p><p>Banking has two main forms of transactions — physical and virtual. Physical transactions include ATMs, which rely on electrical power and are often in large numbers distributed all over the country. Furthermore, banks often have multiple branches which require more energy waste in the form of air-conditioning, lights, etc. On the other hand, virtual transactions would be the online digital banking which have large energy requirements for central server hosting and maintenance, coupled with payment security as well. Hence, as cities are becoming increasingly smarter with the incorporation of advanced technology, this also means that energy consumption increases as well.</p><h3>Summary</h3><p>The potentials of such Green FinTech solutions are huge and range from more transparent, blockchain-based supply chains for consumers to investment solutions that only consider sustainable companies and products. However, this outlined development is still in its infancy. So far, existing research has not provided a comprehensive overview on this topic, and there is still a lack of a framework for more detailed analysis of green FinTech solutions and their impact. These 3 points above emphasize the need to explore and further understand the potential benefits of Green FinTech. Although the 3 points above showcased the potential usefulness of green FinTech, further research are required to ensure that there are indeed major long term benefits to it. Companies have to ensure that they do not hand-wave off all negative environmental impacts by simply resorting to green Fintech — an easy way out via greenwashing.</p><p>References:<br>[<a href="https://www.mdpi.com/2071-1050/12/24/10691">https://www.mdpi.com/2071-1050/12/24/10691</a>][<a href="https://unfccc.int/news/how-blockchain-technology-could-boost-climate-action">https://unfccc.int/news/how-blockchain-technology-could-boost-climate-action</a>]<br>[<a href="https://www.straitstimes.com/tech/how-to-tackle-carbon-emissions-climate-change-blockchain-may-hold-the-key">https://www.straitstimes.com/tech/how-to-tackle-carbon-emissions-climate-change-blockchain-may-hold-the-key</a>]<br>[<a href="https://www.sciencedirect.com/science/article/pii/S1364032118307184">https://www.sciencedirect.com/science/article/pii/S1364032118307184</a>][<a href="https://fintechnews.sg/31796/innovation/smart-city-fintech-why/">https://fintechnews.sg/31796/innovation/smart-city-fintech-why/</a>][<a href="https://blog.imarticus.org/the-key-role-of-fintech-in-developing-smart-cities/">https://blog.imarticus.org/the-key-role-of-fintech-in-developing-smart-cities/</a>]</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=f0a95a86b448" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Neural Networks: Theory, Use-cases, Code implementations and more.]]></title>
            <link>https://medium.com/@nusfintech.ml/neural-networks-theory-use-cases-code-implementations-and-more-9fb76b2d0fb4?source=rss-35fbff2bc022------2</link>
            <guid isPermaLink="false">https://medium.com/p/9fb76b2d0fb4</guid>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[neural-networks]]></category>
            <dc:creator><![CDATA[Machine Learning @ NUS Fintech Society]]></dc:creator>
            <pubDate>Mon, 20 Jun 2022 15:12:02 GMT</pubDate>
            <atom:updated>2022-06-20T15:12:02.140Z</atom:updated>
            <content:encoded><![CDATA[<p>Authors: <a href="https://www.linkedin.com/in/yienong/">Ong Yi En</a> (Research Lead), <a href="https://www.linkedin.com/in/victorwongyd/">Victor Wong</a>, <a href="https://github.com/clvnthe">Calvin The Chun Sun</a>, <a href="https://www.linkedin.com/in/leejaeho1997/">Lee Jae Ho</a></p><p>In recent years, Neural Networks has become a powerful tool that is being applied to business decisions. Clearly, finance has become a fertile field for the use of Neural Networks due to the availability of their data intensity, unstructured nature, high degree of uncertainty, and hidden relationships. In most of these applications, neural networks outperformed traditional statistical models, such as discriminant and regression analysis. Furthermore, these applications have shown significant success in Stock Market Predictions, Loan Application Evaluations and Fraud Detections.</p><p>For instance, with the case of the daily NASDAQ stock exchange rate prediction, it was discovered that a network with three hidden layers, coupled with 20–40–20 neurons in equally hidden layers, was the optimized network with an impressive accuracy of 94.08% for validation dataset.</p><p>Now the question is: What exactly are neural networks, and how are they used?</p><h3>Network Architecture</h3><p>Before we talk about a neural network, we need to understand how each neuron works. Below shows what a biological neuron looks like, as compared to an artificial neuron, called Perceptron.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*_eLAzyhTdsDHm45lXlm9PA.png" /><figcaption><em>Figure 1 Comparison of a biological neuron against a Perceptron</em></figcaption></figure><p>A Perceptron is modeled after a biological neuron. Let’s take an example with a sensory neuron inside our hand. Whenever we hold a pen, or pet a cat, the dendrites of many neurons in our hands get fired up with signals. This is equivalent to inputs in a Perceptron.</p><p>However, not all parts of our hand are equally sensitive. For example, we might immediately sense an ant crawling on our palm, but not on our arm. A Perceptron emulates this by placing different weights for each input.</p><p>Of course, we need to take note that a Perceptron represents all inputs and weights as numbers. What we get overall is the sum of the products of inputs and weights, which gives an arbitrary number, like 32. What should 32 even mean for a neuron?</p><p>This is where an activation function comes in. It helps to squeeze the arbitrary value like 32 into a comprehensible signal between 0 and 1 for the next neuron. Below is an example of an activation function.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*mnpnc3FKygiH88_k" /><figcaption><em>Figure 2 Activation function using Sigmoid function</em></figcaption></figure><p>This behavior is exactly like a biological neuron, which eventually has to decide whether it should relay the incoming signal to the next neuron or not.</p><p>For a Perceptron, there is a bias constant which is a useful tool for calibration. For example, our body will regard anything above 20°C as warm and below as cool. Adding a constant that does not depend on input values greatly helps us more accurately represent how our body senses temperature.</p><p>Now that we have understood how an individual neuron works, we can now imagine how a collective of them will work together. Below shows an example of a neural network.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/500/0*KqjGzmkuhTw5rJHJ" /><figcaption><em>Figure 3 Neural network</em></figcaption></figure><p>Like a wall of sensory neurons that line up below the skin of our hand, we will have an input layer of Perceptrons responsible for handling fresh data. For example, when we are using a neural network for image processing, you can imagine each input neuron responsible for processing a specific pixel of an image.</p><p>Following which would be the hidden layers. Each layer processes the signals produced by the previous layer and relays them to the next after calculation. The layers eventually culminate to an output layer, which arrives at a certain conclusion about the input. Using our image processing example, this output layer answers if an image contains a muffin.</p><p>After learning about the theoretical aspect, how can we implement our own simple Neural Network? First, we would need to initialize the weights and bias arrays (usually generated randomly), to start the process of training.</p><pre>def initialize_nn(X):<br>    np.random.seed(999)<br>    weights = np.random.randn(X.shape[1], 1)<br>    bias = 0</pre><p>Next, we will calculate the output ‘outputP’, then use an activation function on the output. In the case of binary classification, we will use the sigmoid activation function to transform the input to a [0, 1] interval. Thereafter, we would calculate the error, and they are used to update the weights of every connection to the output neuron.</p><pre>def Perceptron(input1, input2, output) :<br>    outputP = input1*weights[0]+input2*weights[1]+bias*weights[2]<br>    if outputP &gt; 0 : #activation function (here Heaviside)<br>        outputP = 1<br>    else :<br>        outputP = 0<br>    error = output — outputP<br>    weights[0] += error * input1 * lr<br>    weights[1] += error * input2 * lr<br>    weights[2] += error * bias * lr</pre><p>In order to train the neural network, we just have to create a loop, iterating the Perceptron function for some epochs. Finally, we can now predict the output for new input.</p><p><strong>Backward and Forward Propagation</strong></p><p>Forward and Backward Propagation is the process in which the neural network moves.</p><p>Forward Propagation is the way that the network moves from the Input Layer (based on our diagram above, it is on the left) to the Output Layer (right), in a forward direction. Backward Propagation is how the network moves from the Output Layer (right) to the Input Layer (left), in a backward direction.</p><p>While you are now probably wondering why the network needs a backward step, let us first understand how Forward Propagation first works. This is because Forward Propagation will generate a cost value that will be used in Backward Propagation.</p><p><strong>Forward Propagation</strong></p><p>For a Neural Network of an Input layer, 1 Hidden Layer, and 1 Output layer, this is how the data will be propagated forward.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*5HOewUAWBY36bkCD" /><figcaption><em>Figure 4 Forward Propagation</em></figcaption></figure><p><strong>Input Layer</strong></p><p>At the Input Layer, the data will be taken in and can be represented with A0 = [a1[0], a2[0], a3[0], …]. The subscript corresponds to the width of the Input Layer, while the superscript corresponds to the depth of the Neural Network (Input Layer = 0).</p><p><strong>Input Layer to Hidden Layer</strong></p><p>To transition from the Input Layer to the Hidden Layer, for each neuron (x) in the Hidden Layer, all of the values of the neurons in the Input Layer, captured as A, will be multiplied by a weight factor, W (i.e. W1 = [wX1[0], wX2[0], wX3[0, …]], where x represents the neuron). This will result in Z1 = [z1[1], z2[1], z3[1], …], which are the intermediate values of the neurons in the Hidden Layer.</p><blockquote>i.e. Z1 = A0 * W1</blockquote><p>Once we have the value of Z1, we will use the activation function (elaborated in the previous part) to determine the value of the neurons in the Hidden Layer.</p><blockquote>i.e. A1 = f(Z1), where f() is the activation function</blockquote><p><strong>Hidden Layer to Output Layer</strong></p><p>Similar to the transition from the Input Layer to the Hidden Layer, the transition from the Hidden Layer to the Output Layer will involve the Hidden Layer values (A1) being multiplied by a weight factor, in this case it is W2, and applying this intermediate value (Z2) to the activation function to find the Output Layer value (S).</p><blockquote>Z2 = A1 * W2<br>S = f(Z2)</blockquote><p>The value of S will then be our predicted output (s) of the network.</p><p><strong>Cost</strong></p><p>The cost © of the network is an evaluation metric of how well a predicted output (s) of the model performs to an expected output (y).</p><blockquote>C = cost(s,y)</blockquote><p>The cost() function can be a calculation of its MSE, cross-entropy or any other cost function. Using this C value, the model can adjust its parameters to get closer to the expected output (y). We can do this by using Backward Propagation.</p><p><strong>Backward Propagation</strong></p><p>Backward Propagation is where the model will fine-tune its parameters (weights) based on the cost © obtained in the previous epoch (i.e. iteration). The predicted values will initially have high errors as the weights were randomly initialized, hence, using backward propagation, the weights will be properly tuned to make the model more accurate and reliable.</p><p>We will use Gradient Descent to minimize the cost of the neural network and improve its performance.</p><h3><strong>Gradient Descent</strong></h3><p>Gradient Descent is an optimization algorithm that is used in neural networks to optimize its performance. It is a convex function that tweaks its weight parameters iteratively to minimize the cost at each step. As the algorithm fills the initial weight values randomly, the algorithm will have to iteratively adjust these values with each backward propagation process to minimize the cost.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*qE7isjW61bYUw2D32LaSww.png" /><figcaption><em>Figure 5 Gradient Descent Curve from Hands-on Machine Learning with Scikit-Learn, Keras &amp; TensorFlow (Concepts, Tools, and Techniques to Build Intelligent Systems) — Aurelien Geron</em></figcaption></figure><p>With each step, the algorithm will attempt to decrease the cost function until it eventually converges to a minimum.</p><p>It is also important to note that cost functions are typically way more complex. Instead of just having 1 parameter or dimension, it could go into the millions. Furthermore, the minimum that the algorithm converges to may sometimes represent a local minimum instead of a global minimum that we want to find.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*1PBXSyLspYVFEsnUpslv1g.png" /><figcaption><em>Figure 6 Gradient Vector</em></figcaption></figure><p>This is why gradient vector calculation in deep neural networks is not simple at all. It requires a good computation power and time for the algorithm to run to ensure that the cost function is at its global minimum.</p><p><strong>Applying Gradient Descent to our Neural Network</strong></p><p>Using this understanding of minimizing cost using the gradient of descent of the cost-weight parameter function, we will iteratively change the weight parameters of each layer as we propagate backward.</p><p><strong>Changing the Weight of the Transition from Input Layer to Hidden Layer (W1)</strong></p><p>After determining the gradient of the cost-weight parameter function, d(cost)/d(W1), we can subtract that cost from the existing W1 values to end up with a new W1 (W1new) that should have a more accurate weight value with a lower cost.</p><blockquote>W1new = W1 — d(cost)/d(W1)</blockquote><p>We can do this for every single layer of the network and repeat this step many times until the resultant cost is extremely low.</p><p>This is how Forward and Backward Propagation along with Gradient Descent is used in Neural Networks.</p><h3><strong>Types of Neural Networks</strong></h3><p>In general, we have explored the basics and processes that go into a neural network. However, there is more to be seen with regards to the types of neural networks available. In this section, we will be looking into 3 different types of neural networks and briefly understanding their structures as well as their uses.</p><p><strong>Convolutional Neural Networks</strong></p><p><em>What is a Convolutional Neural Network<br></em>A Convolutional Neural Network(CNN) is a type of feed-forward deep neural network that is often used for classification problems. More commonly, CNNs see their use in image analytics and natural language processing classification problems.</p><p><em>Structure of a CNN</em></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/581/0*e47p2xjvWUCx30ik" /><figcaption><em>Figure 7 Structure of CNN</em></figcaption></figure><p>Typically, CNNs will take in an input image or a text that needs to be classified. This input would be converted into a mathematical matrix where certain features from the input are extracted in the convolution layer. The convolution layer will then proceed to take the dot products of some parts of the inputs where a feature is observed and essentially produce a smaller matrix representing these convolved features. For example, the edges and corners can be extracted from an image for further convolution. This produced convolved matrix is then further fed into the other layers where more features can be extracted and learnt.</p><p>Next, we have the pooling layer where all our convolved features are introduced into. The purpose of the pooling layer is to essentially reduce the size of the outputs from the convolution layer. This is so as to improve computational efficiency.</p><p>Lastly, the processed and extracted features in the form of matrices are then passed on to the fully connected layer where as previously explained, is where neurons are connected and the neural network learns. This is done through forward/backward propagation via gradient descent as seen in the earlier section.</p><p><strong>Recurrent Neural Networks</strong></p><p><em>What is a Recurrent Neural Network<br></em>A Recurrent Neural Network(RNN) is a type of deep neural network that is often used for sequential data or regression problems. More commonly, RNNs see their use in predictive analytics for sequential data like time series or natural language processing word prediction.</p><p><em>How is a CNN different from an RNN<br></em>Before proceeding to the structure of an RNN, the structure of the RNN is also what makes it different and unique from a CNN. The unique feature of an RNN is its ability to have an internal memory where neurons in the neural network can essentially recall information from its preceding calculations. This allows the neural network to use information from its previous input to predict the output. This can only be done for sequential data where data is correlated to one another in a given order.</p><p>This feature is not available for the feed-forward CNN as there is no particular sequence or order to the data such that previous data can affect the subsequent data.</p><p>Lastly, it is important to note that RNNs make use of the same weights for each hidden layer in their network while CNNs have different weights for each layer.</p><p><em>Structure of a RNN</em></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/753/0*6BXxnB_JirjozpbW" /><figcaption><em>Figure 8 Structure of RNN</em></figcaption></figure><p>The Structure of an RNN is very similar to that of the fully connected and output layer of the CNN structure. However, as previously mentioned RNNs have a “temporal memory”, this is done by introducing a connection between the hidden layer back to itself as seen in the diagram above where the Net is linked back to itself. Upon expanding this structure, we can see the expanded recurrent structure where we take in an input X1 into the hidden layer, giving an output O1 but also producing W1 which contains some information learnt that is then passed back into the network.</p><h3><strong>Conclusion</strong></h3><p>So far, we have discussed the concepts of Neural Networks, types of Neural Networks, and touched on some code implementation and use cases. The concepts explained in this article are fundamental to understanding more complex and advanced neural network structures; and we have barely scratched the surface of what Neural Networks can do.</p><p>The neural network field is growing tremendously, with diverse applications in finance, healthcare, and automotive sectors. As these artificial neurons function in a way similar to the human brain. They can be used for image recognition, character recognition and stock market predictions.</p><p>Learning and understanding the concepts in this field is vital in order to be able to work with them. By exploring this field, you can apply neural networks in other areas to solve data problems. Good luck!</p><h3><strong>References</strong></h3><p><strong>Code implementation:</strong></p><p><a href="https://towardsdatascience.com/first-neural-network-for-beginners-explained-with-code-4cfd37e06eaf">https://towardsdatascience.com/first-neural-network-for-beginners-explained-with-code-4cfd37e06eaf</a></p><p><a href="https://gist.github.com/akarsh-saxena/1459b3a2e62c9c931a12f6dbdbbb8e21">https://gist.github.com/akarsh-saxena/1459b3a2e62c9c931a12f6dbdbbb8e21</a></p><p><strong>Introduction &amp; Use case:</strong></p><p><a href="https://dl.acm.org/doi/abs/10.1287/inte.31.5.112.9662">https://dl.acm.org/doi/abs/10.1287/inte.31.5.112.9662</a></p><p><a href="https://www.academia.edu/51155462/Neural_networks_and_statistical_techniques_A_review_of_applications">https://www.academia.edu/51155462/Neural_networks_and_statistical_techniques_A_review_of_applications</a></p><p><a href="http://www.scielo.org.pe/pdf/jefas/v21n41/a07v21n41.pdf">http://www.scielo.org.pe/pdf/jefas/v21n41/a07v21n41.pdf</a></p><p><strong>Network Architecture:</strong></p><p><a href="https://towardsdatascience.com/first-neural-network-for-beginners-explained-with-code-4cfd37e06eaf">https://towardsdatascience.com/first-neural-network-for-beginners-explained-with-code-4cfd37e06eaf</a></p><p><a href="https://www.healthline.com/health/neurons#anatomy">https://www.healthline.com/health/neurons#anatomy</a></p><p><strong>Forward Backward Propogation:</strong></p><p><a href="https://www.youtube.com/watch?v=Tb23YtZ92AE">https://www.youtube.com/watch?v=Tb23YtZ92AE</a></p><p><strong>Gradient Descent:</strong></p><p><a href="https://medium.com/yottabytes/everything-you-need-to-know-about-gradient-descent-applied-to-neural-networks-d70f85e0cc14"><em>https://medium.com/yottabytes/everything-you-need-to-know-about-gradient-descent-applied-to-neural-networks-d70f85e0cc14</em></a></p><p><strong>Types of Neural Networks:</strong></p><p><a href="https://www.upgrad.com/blog/basic-cnn-architecture/">https://www.upgrad.com/blog/basic-cnn-architecture/</a></p><p><a href="https://www.analyticsvidhya.com/blog/2021/05/convolutional-neural-networks-cnn/">https://www.analyticsvidhya.com/blog/2021/05/convolutional-neural-networks-cnn/</a></p><p><a href="https://hackernoon.com/what-is-an-rnn-recurrent-neural-network-in-deep-learning">https://hackernoon.com/what-is-an-rnn-recurrent-neural-network-in-deep-learning</a></p><p><a href="https://machinelearningmastery.com/an-introduction-to-recurrent-neural-networks-and-the-math-that-powers-them/">https://machinelearningmastery.com/an-introduction-to-recurrent-neural-networks-and-the-math-that-powers-them/</a></p><p><a href="https://towardsdatascience.com/what-is-the-difference-between-cnn-and-rnn-719dba4fd5a5#:~:text=CNNs%20employ%20filters%20within%20convolutional,next%20output%20in%20a%20series">https://towardsdatascience.com/what-is-the-difference-between-cnn-and-rnn-719dba4fd5a5#:~:text=CNNs%20employ%20filters%20within%20convolutional,next%20output%20in%20a%20series</a>.</p><p><a href="https://www.researchgate.net/figure/Recurrent-Neural-Network-Structure-The-left-is-the-typical-RNN-structure-The-right-part_fig3_311805526">https://www.researchgate.net/figure/Recurrent-Neural-Network-Structure-The-left-is-the-typical-RNN-structure-The-right-part_fig3_311805526</a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=9fb76b2d0fb4" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[ESG Rating Implementation]]></title>
            <link>https://medium.com/@nusfintech.ml/esg-rating-implementation-c9a11b8e1fa0?source=rss-35fbff2bc022------2</link>
            <guid isPermaLink="false">https://medium.com/p/c9a11b8e1fa0</guid>
            <category><![CDATA[investing]]></category>
            <category><![CDATA[fintech]]></category>
            <category><![CDATA[environment]]></category>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[esg]]></category>
            <dc:creator><![CDATA[Machine Learning @ NUS Fintech Society]]></dc:creator>
            <pubDate>Sun, 12 Jun 2022 10:04:41 GMT</pubDate>
            <atom:updated>2022-06-12T10:04:41.903Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*KuLk9bZYtBYECG2B.jpg" /></figure><p>Authors: <a href="https://www.linkedin.com/in/fumjiaxuan/">Fum Jia Xuan</a> (Tech Lead) , <a href="https://www.linkedin.com/in/han-wei-lun-04663519b/">Han Wei Lun</a>, <a href="https://www.linkedin.com/in/leowweisheng/">Leow Wei Sheng</a>, <a href="https://www.linkedin.com/in/elvern-tan-a08405216/">Elvern Tan</a>, <a href="https://www.linkedin.com/in/brian-wong-6a696a1b4/">Brian Wong</a></p><p><strong>What is Environmental, Social and Governance (ESG) investing?</strong></p><p>ESG investing stands for Environmental, Social and Governance investing and in recent years, more investors have been looking into it for sustainable investing due to the perceived greater long-term value and business resiliency of companies. The COVID-19 pandemic has also highlighted the importance of ESG among investors, policymakers and other key stakeholders as it is perceived as a way to safeguard businesses from future risks.</p><p>The “E” in ESG stands for the Environmental component and it looks at the impact of resource consumption that businesses have on the environment, such as carbon footprint, waste management and energy efficiency. Companies that overlook the environmental effects of their policies and operations may be exposed to higher levels of financial risk.</p><p>The “S” in ESG stands for the Social component, which considers how businesses interact with the communities where it operates at, and also reviews internal policies related to labour, diversity and inclusion policies.</p><p>The “G” in ESG stands for the Governance component which looks at companies’ internal practices and policies that contribute to effective decision making and legal compliance. Some standards observed include a company’s board composition, audit committee structure and political contributions.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/500/0*TKMITNc89oEV6xWz" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/509/0*JOzTPvqpv3ATA5dA" /></figure><p><strong>ESG with machine learning models</strong></p><p>This project seeks to explore whether the ESG scores of companies can be predicted using ESG reports that they have released. The approach was to make use of Natural Language Processing and train different models using the reports that we have collected. We also applied topic modelling to gain insights about topics mentioned by companies in their ESG reports. This was done across each ESG component to narrow down to topics within each component and identify efforts that most companies were taking towards ESG.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/612/0*Xh0-YrPUMOA0-CAS" /></figure><p><strong>The sector picked and how did we obtain our data.</strong></p><p>The sector chosen for this project was the Energy sector as there has been greater expectations on energy companies to focus more on environmental policies and broader ESG measures. There has also been greater pressure on investors to allocate capital exclusively to companies that prioritise ESG. For our dataset, we collected the most recent ESG reports from companies in the Energy sector and used labels of their corresponding ESG risk scores and ratings from Sustainalytics. The ESG reports were pre-processed by parsing the PDFs into text, then performing text cleaning. We used the NLTK package to tokenize texts from the report, remove stop words and lemmatize the texts. We also cleaned the text by removing numbers and punctuations and converting all texts to lowercase, etc.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/612/0*LKwDqn9YuLDBfYWZ" /></figure><p><strong>Models explored with</strong></p><p><strong>Naïve Bayes Classifier</strong></p><p>We first used the Naive Bayes Classifier to predict ESG risk of companies- Low, Medium and High. This classifier works on the Bayes Theorem where independence is assumed among the features. To fit the text data into the model, we first vectorized the text by using the Term Frequency — Inverse Document Frequency (TFIDF) vectorizer. TFIDF is a technique for text vectorization based on the Bag of words (BoW) model. TFIDF was used instead of the BoW model as it considers the importance of the word in a document and reduces weightage given to words of higher frequency in reports.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1000/0*gsrYLxWpzfxb8Y7j" /></figure><p>After vectorizing the text, the data was fitted into a Multinomial Naive Bayes model and the model had a test accuracy of 0.625. Thus this low accuracy score shows that it is ineffective to only use ESG reports and the Naive Bayes model to predict ESG risks of companies. Furthermore, the limitations of using the Naive Bayes model include the assumption of independence among features. This assumption may not have been feasible for our use case since words are likely to be correlated in ESG reports.</p><p>After exploring with a classification model to predict ESG risk score of companies, we looked into the effectiveness of predicting ESG ratings of companies using different regressors.</p><p><strong>Support Vector Regression</strong></p><p>Support Vector Machine (SVM) is a machine learning algorithm used for classification or regression. Since the ESG ratings data are continuous, we used the regression version of SVM, known as Support Vector Regression (SVR). The algorithm works by first mapping the input data into a high dimensional feature space that is hidden from both the input and the output. This is done using a kernel function, which is a linear kernel in our case. Afterwards, an optimal hyperplane is constructed to minimise the L2 loss function. To evaluate our model, we used root mean square error (RMSE).</p><p>To prepare the data, we first used a count vectorizer to create a bag of words. After fitting this data into the model, we realised that the RMSE was very high, which suggests that the features we were using might not be the best. We proceeded to perform feature engineering by exploring ngrams, maximum document frequency, and minimum document frequency when vectorizing the text data. After exploring various combinations, we concluded that reducing the maximum document frequency from 1.0 to 0.1 helped to significantly improve our model performance. This implies that words that are more specific to each document are better predictors of the documents’ ESG ratings.</p><p>Even though the performance of the model improved after feature engineering, we feel that the prediction errors are still not insignificant. One way to look at it is that it is too simplistic to assume that ESG ratings are awarded based on specific words present in each company’s report. Also, there might not be a strong linear relationship between the frequency of certain words and ESG ratings.</p><p><strong>AdaBoost Regressor</strong></p><p>The AdaBoost Regressor is the next model we used to predict the ESG risk of companies. The AdaBoost Regressor works by first fitting a regressor on the dataset, and subsequently fitting more regressors on the dataset. Each regressor is assigned with a certain weight that reduces the error of prediction. After applying the Term Frequency — Inverse Document Frequency (TFIDF) vectorizer to the data, we fit the data into the AdaBoost Regressor model from Python’s sklearn.ensemble package. The training dataset achieved a R-squared score of 0.991 and the test dataset achieved a R-squared score of -0.322. The model is very much overfitted, even after cross-validating. This suggests that the TFIDF data alone is insufficient in providing adequate information for the model to predict ESG ratings. This is because the TFIDF data is unable to estimate the extent of positive impact to society based on the textual data, thereby failing to predict ESG ratings accurately. It is suggested to consider other forms of data pre-processing before fitting the data into the Adaboost Regressor.</p><p><strong>RandomForest Regressor</strong></p><p>The last model we used to predict the ESG risk of companies was the RandomForest Regressor. The way RandomForest works is that it builds multiple decision trees using bagging and feature randomness for each individual tree to create an uncorrelated forest of trees. This forest of trees in turn should lead to more accurate results than that of any individual tree. As for the processing part of the data, it is very similar to the earlier methods, where we first vectorized the text by using the Term Frequency — Inverse Document Frequency (TFIDF) vectorizer.</p><p>Once that was done, the data was fitted into the RandomForestRegressor function which can be found in the sklearn ensemble package. We managed to get a R-squared score of 0.830 for the train dataset while a score of -0.168 for the test dataset, indicating significant overfitting of our model. Even with hyperparameter tuning, we were not able to improve the scores we got by much. This indicates to us that probably due to the high variability of the text data, the random forest regressor was not able to ingest this information well into meaningful predictions. Thus we came to a conclusion that a RandomForest Regressor should not be considered for this particular ESG score prediction problem and the other models might be a better fit.</p><p><strong>Topic Modelling</strong></p><p>Afterwards, we moved on to look into Topic Modelling to gain insights about topics frequently mentioned about ESG and learn more about efforts taken by companies. We used Latent Dirichlet Allocation (LDA), a probabilistic model that represents each model as a distribution of words. LDA is an unsupervised learning model that views documents as a bag of words that picks sets of topics within documents, and picks sets of words that constitutes a topic. We used the Gensim Python Package and used doc2bow to create the corpus for our model before identifying 5 topics from each ESG component from a compilation of all the ESG reports we have collected.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1000/0*EkhC_5mEsMZ1Pn31" /></figure><p><strong>Findings from Topic Modelling for each component</strong></p><p><strong>Environment</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/816/0*SiD7Pg5SClb9qBKO" /></figure><p>For the environment component, some of the topics we identified that were mentioned in ESG reports include firstly describing emissions levels and this is attributed to many companies declaring and justifying their emission levels in ESG reports. Companies also talked about the impact and risks of operations, which shows that companies usually discuss the implications of their operations when addressing environmental concerns. Additionally, companies discussed carbon levels and sustainability which could be attributed to most companies bringing up issues related to carbon levels when addressing environmental concerns.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1000/0*acLaJuQo2AMuIco2" /></figure><p><strong>Social</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/814/0*6ho7S9gCZrW2CZBe" /></figure><p>For the Social component, some topics include discussions about suppliers that companies work with, which shows that many companies mention and justify organisations that they work with in their ESG reports. Most companies also discuss about diversity and talk about projects that they are working on. Some other topics also include discussions about the support companies received to develop their projects.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1000/0*lKdC2yj9r3wDBAEX" /></figure><p><strong>Governance</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/800/0*C4iOPJaRuEX1AoDo" /></figure><p>For the Governance component in ESG reports, some topics include companies discussing about their board of stakeholders and policies imposed. Companies also talked about the safety of their operations, and mentioned about the performance of the company. It was also observed that companies are talking about the implementation of technology and this could be due to the ESG reports being obtained from recent years where technology has been increasingly employed.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1000/0*ymP-HhxnyXJjJjH8" /></figure><p><strong>Conclusion</strong></p><p>In summary, we explored with employing machine learning models and ESG reports to predict companies’ ESG risk scores and ratings. Our results show that using ESG reports alone was not very effective in predicting a company’s ESG scores. For future works to improve the performance of our models, efforts can be made to increase the number of ESG reports collected to expand the size of our dataset. Deep learning models such as Convolutional Neural Networks and Long-Short Term Memory models can also be explored to improve prediction performances.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=c9a11b8e1fa0" width="1" height="1" alt="">]]></content:encoded>
        </item>
    </channel>
</rss>