<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Zulkuf Genc on Medium]]></title>
        <description><![CDATA[Stories by Zulkuf Genc on Medium]]></description>
        <link>https://medium.com/@zulkuf?source=rss-592ba1423f27------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/0*fNXj2i4o4Vl_L1OJ</url>
            <title>Stories by Zulkuf Genc on Medium</title>
            <link>https://medium.com/@zulkuf?source=rss-592ba1423f27------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Mon, 16 Nov 2026 23:07:19 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@zulkuf/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[ProLLM: LLM Benchmarks for Real-World Use-Cases]]></title>
            <link>https://medium.com/prosus-ai-tech-blog/prollm-llm-benchmarks-for-real-world-use-cases-7c180df68306?source=rss-592ba1423f27------2</link>
            <guid isPermaLink="false">https://medium.com/p/7c180df68306</guid>
            <category><![CDATA[llm-benchmarks]]></category>
            <category><![CDATA[llm]]></category>
            <category><![CDATA[llm-evaluation]]></category>
            <dc:creator><![CDATA[Zulkuf Genc]]></dc:creator>
            <pubDate>Thu, 25 Apr 2024 12:04:13 GMT</pubDate>
            <atom:updated>2024-04-30T10:01:30.147Z</atom:updated>
            <content:encoded><![CDATA[<p>The true measure of a Large Language Models (LLMs) capabilities lies beyond the realm of academic benchmarks. Consider the complexities of a customer support chatbot: it must first sift through and leverage a vast array of relevant information related to a customer’s issue. Then, it needs to navigate a cohesive conversation over several rounds, remember past interactions, and dynamically adapt its responses to the unfolding dialogue — all in the service of effectively resolving customer issues. This dynamic setting demands far more than the ability to answer multiple-choice questions, a skill commonly evaluated by academic benchmarks such as MMLU. Additionally, these benchmarks, often compiled by students from various online sources, may include <a href="https://youtu.be/hVade_8H8mE?t=956">questionable content</a>.</p><p>In response to the need for more accurate assessment of LLMs in complex, real-world interactions, we have started creating <strong>ProLLM</strong> benchmarks derived from the diverse use cases of Prosus Group companies, which touch on sectors ranging from EdTech to food delivery and marketplaces. Our aim is to test LLMs against the kind of interactions they will encounter in everyday use as they are powering real products with real customers, not just theoretical ‘classroom’ scenarios.</p><p>We built ProLLM benchmarks on four principles to ensure meaningful and applicable results:</p><ol><li><strong>Usefulness</strong>: We create benchmarks directly from real use-case data and adopt meaningful metrics to measure how well they perform, providing actionable insights into their effectiveness.</li><li><strong>Relevance</strong>: Our benchmarks are the only ones that enable interactive exploration of LLM performance across multiple aspects, tailored to user interests, such as answering advanced debugging questions for JavaScript code.</li><li><strong>Inclusivity</strong>: Our benchmarks are inclusive and dynamic, covering a variety of applications in multiple sectors and languages and evolve with the model capabilities and requirements of use cases.</li><li><strong>Reliability</strong>: Our evaluation sets are not publicly disclosed, ensuring the benchmarks’ integrity, with mirror sets shared for insight and transparency.</li></ol><p>Building on these principles, we benchmarked over 30 leading LLMs, including both proprietary and open-source models, across a variety of tasks. Our initial effort focused on the coding assistant use case leveraging data from Stack Overflow. We are actively expanding our benchmarks using the diverse evaluation sets collected from our group’s companies. This blog provides a preview of our benchmarks. Our <a href="http://prollm.toqan.ai/">interactive benchmarks portal</a> is also live and accessible to the public. Details of our benchmarks and evaluation strategy will be published soon in a paper along with mirror datasets and evaluation scripts.</p><h3>The Reality Check for LLM Benchmarks</h3><h3>The Mismatch</h3><p>Current benchmarks for LLMs, such as <a href="https://huggingface.co/spaces/HuggingFaceH4/open_llm_leaderboard">Hugging Face’s LLM Leaderboard</a>, which focuses mostly on text completion and multiple-choice question answering, are well-suited for demonstrating basic language understanding. However, to better meet the demands of real-world applications, there is a growing need to include benchmarks that assess a model’s proficiency in more practical tasks, such as summarizing key points from a financial report. <a href="https://chat.lmsys.org/?leaderboard">LMSYS Chatbot Arena</a> offers an engaging way to evaluate LLMs through community-driven, randomized battles, using the Elo rating system for comparative analysis. This innovative approach highlights relative model strengths effectively. However, its focus on subjective assessments and comparative rankings may not adequately address specific real-world scenarios, limiting its practical applicability across diverse applications. <a href="https://crfm.stanford.edu/helm/lite/latest/">Stanford’s Helm benchmark</a>, while primarily featuring Q&amp;A evaluations, takes a step in the right direction with its <a href="https://hazyresearch.stanford.edu/legalbench/">LegalBench</a> tasks that more closely mirror the work of legal professionals. However, it only scratches the surface of potential applications. Beyond this, the broader landscape of industry-specific LLM tasks remains significantly underrepresented in benchmarking efforts. Moreover, the vast majority of benchmarks, including those for translation, are conducted in English.</p><p>In software development, benchmarks for evaluating LLMs are few and don’t quite capture the full spectrum of a programmer’s day-to-day tasks. Consider <a href="https://github.com/openai/human-eval">HumanEval</a>, the widely-used coding benchmark, which provides 164 Python coding exercises that are clean-cut and self-contained. These tasks miss the complexity of real-world software development, such as debugging, and code optimizing, where developers often work with a variety of programming frameworks and libraries. Compounding this issue is the prevalent focus on Python across these benchmarks. While Python is a popular choice, as confirmed by its use among 49% of professionals in the latest <a href="https://survey.stackoverflow.co/2023/">Stack Overflow survey</a>, the industry’s reliance on a diverse array of programming languages is not reflected. The mismatch between the current benchmarks and industry requirements can be attributed to several underlying factors:</p><ul><li>The rapid pace of LLM development has outstripped the evolution of benchmarks. Benchmarks, mostly created by researchers, often focus on general and more fundamental linguistic tasks that aren’t necessarily representative of complex, industry-specific problems.</li><li>Initially, language models were primarily an academic pursuit. Only recently have industries significantly invested in leveraging these models, driven by the popularity and success of OpenAI’s ChatGPT.</li><li>Evaluating LLMs with human experts is intensive and not scalable, especially with the breadth of LLM tasks and the constant influx of new models. In response, the field relies on structured tests and surface-level metrics, often focused on literal text matching and multiple-choice answer selection, which do not fully capture the depth of real-world model performance.</li><li>Creating high-quality, industry-specific evaluation datasets is a non-trivial and complex task, requiring extensive resources, the expertise of domain specialists, and data that accurately represent real-world scenarios while meeting strict quality and privacy standards.</li></ul><h3>Aligning Benchmarks with Business</h3><p>Our mission is to redefine the evaluation of LLMs in a way that our benchmarks clearly show their practical usefulness for business in a reliable way. To achieve this, we have concentrated our efforts in the following areas.</p><h4>Crafting Realistic Evaluation Sets</h4><p>To create benchmarks that align with real-world demands, we’ve conducted a thorough analysis of how LLMs are used across Prosus’s vast portfolio. This portfolio comprises over 100 companies worldwide, including Stack Overflow, Udemy, iFood, Swiggy, and OLX. We discovered that LLMs are commonly utilized in higher level for coding assistance and gaining insights into complex documents, as well as a support agent for addressing work-related queries. In lower level, LLMs are often leveraged as a convenient zero-shot machine learning resource for functions like extracting data insights through tagging, classifying, and entity extraction. These common use-cases have been prioritized in the creation of our evaluation sets to keep our benchmarks relevant for business.</p><h4>Preventing Benchmark Data Leakage</h4><p>One of the main hurdles with current public benchmarks is the unintentional leakage of their data into the training sets of LLMs. This data contamination significantly affects their performance, as demonstrated by the performance difference between benchmarks of coding assistance task on historical versus recent Stack Overflow data, shown in <strong>Figure 2</strong>.</p><p>To address this, we’ve implemented a two-step approach for creating our evaluation sets. First, we keep our main evaluation sets private and use them to report our benchmark results. Additionally, we ensure that the LLMs used for automated evaluations are deployed on cloud platforms that give a zero-retention guarantee for our prompts.</p><p>Second, we will offer public mirror evaluation sets that closely match our private sets. These public sets allow LLM developers and other interested parties to test their models on evaluation sets that are similar to our private ones. We are also dedicated to regularly updating our evaluation sets to ensure their quality and relevance.</p><h4>Aligning Evaluation Metrics</h4><p>Over the past three years, our extensive deployment and testing of various LLM use-cases have consistently shown that standard, one-size-fits-all metrics are inadequate for capturing the specific needs and complexities of LLM applications. Each LLM task is unique, requiring customized evaluation criteria that align closely with its individual objectives. These customized metrics are essential for evaluating the model’s performance in line with user expectations.</p><p>To guide decision-makers effectively, these metrics must be intuitive and transparent, distinctly showcasing the model’s success in practical applications. For question-answering tasks, for example, the core metric we focus on is the <strong>Acceptance Rate</strong>. Inspired by the <a href="https://stackoverflow.com/help/accepted-answer">accepted answer</a> concept on Stack Overflow, we refined this metric to evaluate answers on three important dimensions:</p><ul><li><strong>Accuracy</strong>: Is the answer correct, and does it avoid significant inaccuracies that could undermine its validity?</li><li><strong>Completeness</strong>: Does the answer cover all essential points thoroughly?</li><li><strong>Relevance</strong>: Is the answer directly applicable to the core problem posed by the user?</li></ul><p>An answer must score well across all these dimensions to be deemed acceptable<strong>.</strong> If an answer falls short in any of these areas, <strong>leading the user to continue searching for a solution</strong>, it is considered unacceptable.</p><p>For summarization tasks, the metric of <strong>Adherence to Instructions</strong> is particularly significant. It measures how well the LLM output aligns with the user’s specific instructions. For instance, if a user requests three key points, the LLM should ideally provide exactly three key points, assuming they are available in the source material. Additionally, the output must be <strong>accurate</strong>, ensuring that the content is grounded in the original text. The <strong>writing quality </strong>should also be high, maintaining clarity and readability throughout.</p><p>Such tailored metrics are designed to capture what truly matters to the users of these applications.</p><h4>Automating the Evaluation Process</h4><p>To scale up the LLM evaluations, we’ve developed an automated framework that leverages the technology itself. However, using LLMs as evaluators isn’t as simple as it seems. Even the most advanced model to date, GPT-4 Turbo (as of April 2024), may still fall short of domain experts when it comes to complex tasks like evaluating technical answers.</p><p>It is known that to truly improve anything, one must begin by accurately measuring it. With this in mind, we developed an <strong>auto-evaluation</strong> benchmark using a set of 150 diverse questions gathered from Stack Overflow and our internal <a href="http://www.toqan.ai/">AI assistant</a> Toqan, which has processed millions of queries since 2022. We then used different LLMs to generate answers for these questions, which were evaluated and scored by at least three domain experts according to a detailed rubric we defined. This process created a labeled dataset that allows us to accurately quantify the performance of LLMs in evaluating answers.</p><p>Auto-evaluation benchmark has been instrumental in testing our evaluation techniques and facilitating improvement. Moreover, it guided us in establishing rules for creating evaluation sets that are fit for automated assessments, ensuring, for instance, that all questions are self-contained.</p><p>Initially, the highest accuracy levels achieved with our top models ranged from ~70% on our evaluation benchmark. After extensive experimentation and numerous refinements to our automated evaluation methods, we significantly increased the evaluation accuracy above 85%. As illustrated in <strong>Figure 1</strong>, this level of accuracy now surpasses that of the average domain expert.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/933/1*d1DoiY0OmMHbs9MXycfXdw.png" /><figcaption><strong>Figure 1: Auto-evaluation benchmark</strong></figcaption></figure><h3>Benchmarking Across the Spectrum</h3><p>We’ve established an automated evaluation pipeline that enables us to benchmark any new model on the day of its release. This ensures we keep pace with the rapidly expanding field, which includes commercial LLM providers like OpenAI, Anthropic, Mistral and Google, as well as contributions from the open-source community, potentially featuring specialized models in code or other fields.</p><p>Our initial focus lies on a set of coding and Q&amp;A assistant evaluations designed to assist our product teams, both within Prosus and our portfolio companies, including Stack Overflow. We are actively working on expanding our scope of benchmarks in both quantity and diversity. Here’s a snapshot of selected LLM tasks under our evaluation:</p><h4>Coding Assistant</h4><p>Our Coding Assistant benchmark consists of 925 Stack Overflow questions asked over the last five years, covering 25 programming languages and four task types: debugging, implementation, optimization, and conceptual understanding. Each question is selected based on its positive score and multiple high-rated answers, including at least one accepted solution, ensuring reliable evaluations against verified solutions.</p><p>The Coding Assistant — Recent benchmark includes 300 high-quality questions from the past six months (as of October 1, 2023), testing models’ adaptability to new, unseen queries. Significant performance gaps are evident, with GPT4Turbo’s accuracy dropping from 90% in the Coding Assistant benchmark to 64% in the Coding Assistant — Recent benchmark. To facilitate fair comparisons in the Coding Assistant — Recent benchmark, we implemented a Version tag to filter out questions influenced by recent software updates. Despite these adjustments, GPT4Turbo’s performance stabilizes at 76%.</p><p>Additionally, it’s important to note that GPT4, with a training cut-off in April 2023, experiences an even more significant performance drop in the Coding Assistant — Recent benchmark, falling from rank 6 to rank 13. This decline highlights the challenges models face with rapidly evolving content and missing the recent Stack Overflow content in their training data.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Jig4hRMHm1qpHJ05s_7i6A.png" /><figcaption><strong>Figure 2: Coding assistant benchmarks with historical and recent Stack Overflow data.</strong></figcaption></figure><h4>Q&amp;A Assistant</h4><p>We’ve developed an in-house AI assistant, <a href="https://toqan.ai/">Toqan</a>, designed to act as an AI colleague, providing support to our colleagues across the Prosus portfolio companies. Since its launch in 2022, Toqan has addressed over 1.1 million varied queries from thousands of users worldwide. Leveraging this extensive interaction data, we’ve carefully curated an evaluation set of 275 questions with permissive rights for benchmarking. This set includes a balanced mix of technical questions, primarily related to software engineering and IT, and non-technical questions akin to business consultancy. The Q&amp;A Assistant Benchmark evaluates an LLM’s effectiveness as a team member in a business environment by assessing its ability to provide accurate and contextually relevant responses. To ensure a deeper evaluation, we’ve categorized the questions based on their complexity level and type as seen in <strong>Figure 3</strong>. Each question in our evaluation set has a corresponding answer generated by GPT4Turbo, which has received positive ratings and has been validated by domain experts.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*mtIimYI4_dSwrHNhcGpm2g.png" /><figcaption><strong>Figure 3: Q&amp;A Assistant benchmark curated from user queries of our Toqan AI assistant.</strong></figcaption></figure><h4>Other Benchmarks</h4><p>We have recently introduced a <a href="https://prollm.toqan.ai/leaderboard">Summarization Benchmark</a> that evaluates an LLM’s ability and different long-text handling techniques to accurately summarize and extract key insights from diverse sources such as YouTube video transcripts, websites, PDFs, and direct text inputs. This benchmark utilizes approximately 50 user-submitted queries to our Toqan assistant, testing the AI’s proficiency in following detailed instructions to produce concise summaries and perform targeted information extraction tasks.</p><p>Additionally, we are preparing multiple evaluation sets to be released, ranging from an SQL agent to data insights in non-English languages. Keep an eye on our prollm benchmarks portal for updates on their availability.</p><h4>Benchmarking Portal</h4><p>We are proud to provide <a href="https://prollm.toqan.ai/">prollm</a>, our benchmarking portal, as a public and open-access platform, which delivers comprehensive insights into the performance of various LLMs — and soon, agents — across diverse real-world scenarios. It features interactive exploration tools that enable users to thoroughly investigate specific tasks and compare the effectiveness of different models. Users also have the opportunity to create customized benchmarks tailored to their precise evaluation needs. We are dedicated to consistently updating the portal with new benchmarks from both the Prosus ecosystem and external sources, ensuring it keeps pace with the rapidly evolving LLM landscape.</p><h4>Why Interactivity in Benchmarks?</h4><p>Interactive benchmarks offer a dynamic view into the nuanced performance of LLMs, revealing how their effectiveness can vary significantly across different tasks. As a sneak peek into our benchmark insights, we’ve observed that while GPT4-Turbo consistently outperforms other models across many tasks, the degree of its superiority fluctuates.</p><p>For instance, in overall coding assistant tasks, GPT-4 Turbo stands out, performing considerably better than all other models. However, when it comes to answering C and C++ questions, the performance landscape changes. GPT-4 Turbo’s performance drops noticeably, and we also observe a shift in the leaderboard rankings as seen in Figure 4. We see a similar shift in Javascript debugging questions as well.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*oEwrOQHKkq0ekkxbv5plhA.png" /><figcaption><strong>Figure 4: An open-source model outperforms GPT-4 Turbo in answering C and C++ questions on our interactive benchmark.</strong></figcaption></figure><p>These insights are just a glimpse of the in-depth analyses available on our benchmarking portal. The site serves as a resource for anyone interested in the nuanced performance metrics of LLMs across a range of real-world scenarios. By providing a detailed view of how various models perform on different tasks, our benchmarks can guide users to choose the most suitable LLM for their specific needs.</p><h3>Building Better LLM Benchmarks Together</h3><p>We invite organizations worldwide to enhance the benchmarking landscape for LLMs by contributing your unique use cases. Stay informed with real-time performance updates each time a new model is released, allowing your business to effortlessly keep pace with AI advancements. This collaboration is not just a call for partnership; it’s a chance to join a growing AI ecosystem. Together, we can expand our collective knowledge and influence the development of LLMs across industries. Join us to share in this knowledge and gain recognition for your contributions to AI..</p><h3>Acknowledgment</h3><p>All the work presented in this blog — from research, data set curation and tagging, to inference and evaluation pipelines, prompts and interfaces — has been carried out by the Prosus AI Applied LLM team: <a href="https://www.linkedin.com/in/nidhish-s-shah/">Nidhish Shah</a>, <a href="https://www.linkedin.com/in/dogutanaraci5522b5a3/">Doğu Aracı</a> and <a href="https://www.linkedin.com/in/zulkufgenc/">Zülküf Genç</a>.</p><p>We wish to extend our gratitude to our colleagues at Stack Overflow: Ellen Brandenberger, Michael Foree and his team. Their assistance with the creation of the Stack Overflow evaluation sets and their constructive feedback have been instrumental to our initiative. We are also grateful to Euro Beinat and Paul van der Boor for their valuable feedback and for reviewing this blog post. Their insights greatly improved the quality of our work. Lastly, we express our heartfelt appreciation to the team members at Prosus AI and Toqan. Their contributions to the labeling of our LLM-generated evaluations provided significant assistance to our project, for which we are truly grateful.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*7Pe6aPlwjccOeO9P" /><figcaption>A view from our labelling party where Prosus AI and Toqan team members are labelling auto evaluation results.</figcaption></figure><h3>Stay Tuned</h3><p>We are excited about the potential of our efforts to influence the future of LLMs and their real-world uses. If you’re intrigued by our benchmarks and wish to contribute, or if you have any questions or would like to share your feedback, don’t hesitate to contact us at <a href="/service/mailto:zulkuf.genc@prosus.com">zulkuf.genc@prosus.com</a>. For updates on our continued work and new benchmarks, make sure to follow our blog and <a href="https://prollm.toqan.ai/">prollm portal</a>. Thank you!</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=7c180df68306" width="1" height="1" alt=""><hr><p><a href="https://medium.com/prosus-ai-tech-blog/prollm-llm-benchmarks-for-real-world-use-cases-7c180df68306">ProLLM: LLM Benchmarks for Real-World Use-Cases</a> was originally published in <a href="https://medium.com/prosus-ai-tech-blog">Prosus AI Tech Blog</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[FinBERT: Financial Sentiment Analysis with BERT]]></title>
            <link>https://medium.com/prosus-ai-tech-blog/finbert-financial-sentiment-analysis-with-bert-b277a3607101?source=rss-592ba1423f27------2</link>
            <guid isPermaLink="false">https://medium.com/p/b277a3607101</guid>
            <category><![CDATA[naturallanguageprocessing]]></category>
            <category><![CDATA[sentiment-analysis]]></category>
            <category><![CDATA[finance]]></category>
            <category><![CDATA[artificial-intelligence]]></category>
            <category><![CDATA[bert]]></category>
            <dc:creator><![CDATA[Zulkuf Genc]]></dc:creator>
            <pubDate>Fri, 31 Jul 2020 06:53:57 GMT</pubDate>
            <atom:updated>2020-07-31T06:53:57.686Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*99ToSc3FWZWlZ_BPcTe9zQ.png" /></figure><p><a href="https://medium.com/u/c3a1722290a1">Doğu Tan Aracı</a>, <a href="https://medium.com/u/592ba1423f27">Zulkuf Genc</a></p><blockquote>Shares of food delivery companies surged despite the catastrophic impact of coronavirus on global markets.</blockquote><p>The statement above carries a clear positive message about food delivery companies next to a negative one about global markets. It is not an easy task for computers to extract this positive message because it requires understanding of what is positive and negative in the financial context without getting confused with the rest of the sentence.</p><p>In this blog post, we will share a simplified story of how we solved this financial <a href="https://en.wikipedia.org/wiki/Sentiment_analysis">sentiment classification</a> problem with <a href="https://arxiv.org/abs/1810.04805">Bidirectional Encoder Representations from Transformers</a> (BERT) and improved the <a href="https://arxiv.org/pdf/1811.11008.pdf">state-of-the-art</a> performance by 15 percentage points. In addition, we will also share our production settings, code, data and guidelines for you to train and use our financial sentiment classifier.</p><h4>Why did we do this?</h4><p>Prosus is one of the largest technology investors in the world. On a daily basis, we need to sift through an overwhelming amount of information, largely in form of unstructured text data, about the sectors and companies of our interest. Financial sentiment analysis is one of the essential components in navigating the attention of our analysts over such continuous flow of data.</p><p>We quickly noticed that naive approaches such as bag-of-words would simply not cut it, as they generally disregard essential context information in the text. We needed more advanced techniques to crack the financial context and understand what is positive and what is negative from the financial point of view.</p><blockquote>The current state-of-the-art approach to natural language understanding is using pre-trained language models by fine-tuning them for specific (downstream) tasks such as question answering or sentiment analysis.We followed that recipe and developed FinBERT as a BERT-based language model with a deeper understanding of financial language and fine-tuned it for sentiment classification.</blockquote><h4>A New Era in NLP</h4><p>If we were trying to tackle this problem in let’s say 2017, we would have had to come up with a custom model and train it from scratch. In that case we would have needed very large amounts of labeled data for a somehow acceptable performance. Labeled data with good quality isn’t very easy to acquire, especially in a niche domain like finance, where expertise is required and definitely does not come cheap.</p><p>Luckily, <a href="https://ruder.io/nlp-imagenet/">NLP’s ImageNet moment happened</a> in 2018. Starting with <a href="https://arxiv.org/abs/1801.06146">ULMFit</a>, researchers cracked how to do transfer learning efficiently for NLP problems. The idea is simple: First, get textual data that is available in abundance, like Wikipedia. Then, train a language model with that data, which is basically predicting the next word in a sentence. Finally, fine-tune the language model for your task with one or several task-specific layers. The advantage of this approach is that you don’t need a huge dataset for fine-tuning, because the model learns about language during the original language model training. The heavy lifting is already done!</p><p>Many other language models followed ULMFit path but used different training schemas, most significant being <a href="https://arxiv.org/abs/1810.04805">BERT</a>. BERT was the language model that made the whole concept of pre-training and fine-tuning flow very popular. BERT brought two core innovations to language modelling: (1) It borrowed the <strong>transformer</strong> (T of BERT) architecture from <a href="https://papers.nips.cc/paper/7181-attention-is-all-you-need.pdf">machine translation</a>, which does a better job of modelling long-term dependencies than RNN-based ones (excellent overview <a href="https://lilianweng.github.io/lil-log/2018/06/24/attention-attention.html">here</a>). (2) It introduced the Masked Language Modelling (MLM) task, where a random 15% of all tokens are masked and the model predicts them, enabling true <strong>bi-directionality</strong> (B of BERT). Intuitive explanations of how transformers and BERT can be found <a href="https://nostalgebraist.tumblr.com/post/185326092369/the-transformer-explained">here</a> and <a href="https://jalammar.github.io/illustrated-bert/">here</a>.</p><p>BERT achieved state-of-the-art performance in almost all of the down-stream tasks such as text classification and question answering it was applied to. But more importantly, it was a huge step towards democratisation of NLP. Now that the compute-heavy (Google used 16 TPU’s for 4 days to pre-train BERT) language model training is already done, anyone with a decent computing power could train a very accurate NLP model for their niche task based on a pre-trained language model.</p><h4>FinBERT</h4><p>BERT was perfect for our task of financial sentiment analysis. Even with a very small dataset, it was now possible to take advantage of state-of-the-art NLP models. But since our domain — finance is very different from the general purpose corpus BERT was trained on, we wanted to add one more step before going for sentiment analysis. Pre-trained BERT knew how to talk, but now it was time to teach it how to <em>talk like a trader. </em>We took the pre-trained BERT and then further trained it on a purely financial corpus. That corpus is Reuters TRC2, and available upon request <a href="https://trec.nist.gov/data/reuters/reuters.html">here</a>. Our purpose was to achieve better domain adaptation by exposing the model to financial jargon, before fine-tuning for the final task. We further pre-trained BERT using Hugging Face’s excellent library <a href="https://github.com/huggingface/transformers">transformers</a> (back then it was pytorch-pretrained-bert) and <a href="https://github.com/jiesutd/pytorch-pretrained-BERT/tree/master/examples/lm_finetuning">scripts</a> they provided.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*byN4N9ZL1OIjEMbKCj7nMw.png" /><figcaption>The training steps in FinBERT</figcaption></figure><p>Once we had the pre-trained and domain-adapted language model, the next step was to fine-tune it with labeled data for financial sentiment classification. We used the dataset of <a href="https://arxiv.org/pdf/1307.5336.pdf">Financial Phrasebank</a>. It is a very well thought-out and carefully labeled albeit a small dataset. Researchers extracted 4500 sentences from various news articles, which include financial terms. Then 16 experts and master students with finance backgrounds labeled them. They didn’t only report labels but also inter-annotator agreement level for each sentence, which means how many experts labelled as positive, neutral and negative.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/07c9ca2dc4fc2ae8448d361df723bc4a/href">https://medium.com/media/07c9ca2dc4fc2ae8448d361df723bc4a/href</a></iframe><p>Fine-tuning a transformer-based language model for classification is a straight-forward process. A classification layer is added after BERT’s special token [CLS], which is used for sequential tasks like sentence classification or <a href="https://nlp.stanford.edu/projects/snli/">textual entailment</a>. Then the <em>whole model </em>is fine-tuned with classification losses. A visual representation of this structure is on the picture below.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/800/1*-qwGn_js-CjwgOz5e8tWug.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/884/1*WBvoOrHXYXHPy_tx8iwLOQ.png" /><figcaption>The overview of the steps to train FinBERT</figcaption></figure><h4>Results</h4><p>The results were surprisingly good, even for a zealous believer of pre-trained language models. We achieved 97% test-set accuracy in the full inter-annotator agreement part of Financial PhraseBank. That was six percentage points higher than the previous state-of-the-art (<a href="https://ieeexplore.ieee.org/abstract/document/8334488?anchor=keywords">FinSSLX</a>). On the dataset, including sentences without full inter-annotator agreement, accuracy was 86%, 15% percentage points higher than the previous state-of-the-art (<a href="https://arxiv.org/pdf/1811.11008.pdf">HSC</a>).</p><p>We also evaluated how FinBERT performs compared to other deep learning models, with or without transfer learning;</p><ol><li>A plain LSTM model with <a href="https://nlp.stanford.edu/projects/glove/">GloVe</a> embeddings</li><li>LSTM model with <a href="https://allennlp.org/elmo">ELMo</a> embeddings</li><li>ULMFit</li></ol><p>On the table below you can see the results. FinBERT is the best model, though ULMFit is also impressively competitive, considering its much smaller model size.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*hW0Y1p_37we1YP5RUSinjw.png" /><figcaption>Experimental results on the Financial PhraseBank dataset</figcaption></figure><p>And here are some examples of sentences from financial news scored by FinBERT.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/972/1*7gWGvgmhnRg4wPw2fQYvdQ.png" /><figcaption>Some prediction examples from FinBERT</figcaption></figure><h4>Where does FinBERT fail?</h4><p>With 97% accuracy on the subset (100% annotator agreement) of Financial PhraseBank, our curiosity drove us to look into the cases where the model failed. As the PhraseBank <a href="https://arxiv.org/abs/1307.5336">paper</a> indicates, most of the inter-annotator disagreements are between positive and neutral labels (agreement for separating positive-negative, negative-neutral and positive-neutral are 98.7%, 94.2% and 75.2% respectively). It is because of the difficulty of distinguishing “<a href="https://arxiv.org/abs/1307.5336">commonly used company glitter and actual positive statements</a>” and companies trying to spin an objectively neutral statement into a positive one.</p><p>As with most deep learning models, it is not very easy to intuit on failure modes of FinBERT. We still realised there are some re-occurring themes in the errors and thought it would be interesting to share them.</p><p>The first example is actually the most common type of failure. The model sometimes fails to do the math in which figure is higher, and in the absence of words indicative of direction like “increased”, might make the prediction of neutral.</p><blockquote><strong>A. Pre-tax loss totaled euro 0.3 million, compared to a loss of euro 2.2 million in the first quarter of 2005 .</strong></blockquote><p><em>True value: Positive <br>Predicted: Negative</em></p><p>However, there are many similar cases where it does make the true prediction too. Examples <strong>B</strong> and <strong>C </strong>are different versions of the same type of failure. The model fails to distinguish a neutral statement about a given situation from a statement that indicated polarity about the company. In the third example, information about the company’s business would probably help.</p><blockquote><strong>B. This implementation is very important to the operator, since it is about to launch its Fixed to Mobile convergence service in Brazil</strong></blockquote><p><em>True value: Neutral <br>Predicted: Positive</em></p><blockquote><strong>C. The situation of coated magazine printing paper will continue to be weak</strong></blockquote><p><em>True value: Negative <br>Predicted: Neutral</em></p><p>73% of the misclassifications of FinBERT are between positive and neutral labels, while the same number is 5% for negative and positive. That is consistent with both the inter-annotator agreement numbers and common sense. It is easier to differentiate between positive and negative. But it might be more challenging to decide whether a statement indicates a positive outlook or merely an objective observation.</p><h4>Deployment</h4><p>Just having a model with good accuracy was not enough for us. We also wanted it to be used in valuable ways. We came up with and implemented three applications so far:</p><ol><li>A <strong>text editor </strong>that processes text and highlights sentences according to their predicted sentiment. It can be used to refine the message before publication or to assess sentiment in corporate publications, financial disclosures or business blogs. We also shared this text editor as a web interface with our colleagues from other departments, for them to play with FinBERT, and the response was quite positive.</li><li>A <a href="https://www.elastic.co/guide/en/kibana/current/dashboard.html"><strong>Kibana</strong></a><strong> dashboard</strong> that gets tweets from financial news outlets, scores them and generates a live overview of the market in terms of financial sentiment. We thought this type of information might provide an additional perspective for our colleagues.</li><li>And finally an <strong>inference service </strong>deployed on <a href="https://www.kubeflow.org/">Kubeflow</a>, that can be queried by anyone with the endpoint. This enables any data scientist on our portfolio companies to develop their own application with FinBERT, without thinking about deploying the model.</li></ol><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*JhN01dcDnXfW0h1s9G2C2w.png" /><figcaption>The FinBERT text editor</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*1RmJ1wsMgj7x-AEU2PsL6w.png" /><figcaption>The overview of our dashboard pipeline</figcaption></figure><h3>Final Words</h3><p>Our FinBERT journey started as a curiosity-driven small research project after we noticed the existing methods are not performing good enough in financial sentiment classification. Throughout this journey we learned a lot and ended up at a point where we have the state-of-the-art sentiment analyser deployed and creating actual value to our company. We have many ideas for future directions but are also open to your thoughts and suggestions. Please feel free to share in the comments section. Meanwhile, if you want to try it at home (still in lockdown), you can find all the material related to the data and model in our <a href="https://github.com/ProsusAI/finBERT">Github</a>.</p><p><em>We’d like to thank </em><a href="https://medium.com/@nishikantdhanuka"><em>Nishikant Dhanuka</em></a><em> and </em><a href="https://medium.com/@liesbethmd"><em>Liesbeth Dingemans</em></a><em>, our colleagues from the Prosus AI team, for their suggestions and help in editing. Please check the </em><a href="https://arxiv.org/abs/1908.10063"><em>paper</em></a><em>, if you want to learn more about FinBERT. Also if there are any further questions or suggestions, feel free to reach out to us at datascience@prosus.com.</em></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=b277a3607101" width="1" height="1" alt=""><hr><p><a href="https://medium.com/prosus-ai-tech-blog/finbert-financial-sentiment-analysis-with-bert-b277a3607101">FinBERT: Financial Sentiment Analysis with BERT</a> was originally published in <a href="https://medium.com/prosus-ai-tech-blog">Prosus AI Tech Blog</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
    </channel>
</rss>