<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Nicholas Abell on Medium]]></title>
        <description><![CDATA[Stories by Nicholas Abell on Medium]]></description>
        <link>https://medium.com/@nqabell89?source=rss-fbd886da6bfa------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/0*41EkiRJPk2-fQIzm</url>
            <title>Stories by Nicholas Abell on Medium</title>
            <link>https://medium.com/@nqabell89?source=rss-fbd886da6bfa------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Sat, 21 Nov 2026 15:15:12 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@nqabell89/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[7 Key Military Applications of Machine Learning]]></title>
            <link>https://medium.com/@nqabell89/7-key-military-applications-of-machine-learning-9818dfa2ea86?source=rss-fbd886da6bfa------2</link>
            <guid isPermaLink="false">https://medium.com/p/9818dfa2ea86</guid>
            <dc:creator><![CDATA[Nicholas Abell]]></dc:creator>
            <pubDate>Fri, 02 Oct 2020 15:47:11 GMT</pubDate>
            <atom:updated>2020-10-02T15:47:11.578Z</atom:updated>
            <content:encoded><![CDATA[<p>Machine learning is now a critical component in modern warfare systems. Let’s explore 7 key military applications of artificial intelligence today.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/720/1*9nqNvZsZu7A-TmVJQECLPg.png" /><figcaption>Image credit: militaryaerospace.com</figcaption></figure><p>Machine learning has become a critical part of modern warfare, and a major point of interest for me, both as an Army veteran and data scientist. Compared with conventional systems, military systems equipped with ML/DL are capable of handling tremendously larger volumes of data more efficiently. Additionally, AI improves self-control, self-regulation, and self-actuation of combat systems due to its inherent computing and decision-making capabilities; a critical aspect to consider due to the nature of combat.</p><p>AI/ML is being deployed in almost every military application, and increased funding for research and development from military research agencies promises to drive adoption of AI-driven systems in the military sector even further.</p><p>For instance, the US Department of Defense’s (DoD) Defense Advanced Research Projects Agency (DARPA) is financing the development of a robotic submarine system, which is expected to be employed in applications ranging from detection of underwater mines to engagement in anti-submarine operations. Additionally, the US DoD overall spent 7.4 billion on artificial intelligence, Big Data, and cloud in the fiscal year 2017. The market size of military ML solutions is expected to reach 19 billion by 2025.</p><p>Here are seven major military applications where machine learning will prove its importance in the years to come.</p><h3>1. Warfare Platforms</h3><p>Defense forces from different countries across the globe are embedding AI into weapons and other systems used on land, naval, airborne, and space platforms.</p><p>Using AI in systems based on these platforms has enabled the development of efficient warfare systems, which are less reliant on human input. It has also led to increased synergy and enhanced performance of warfare systems while requiring less maintenance. AI is also expected to empower autonomous and high-speed weapons to carry out collaborative attacks.</p><h3>2. Cybersecurity</h3><p>Military systems are often vulnerable to cyber attacks, which can lead to loss of classified military information and damage to military systems. However, systems equipped with AI can autonomously protect networks, computers, programs, and data from any kind of unauthorized access.</p><p>In addition, AI-enabled web security systems can record the pattern of cyber attacks and develop counter-attack tools to tackle them.</p><h3>3. Logistics &amp; Transportation</h3><p>AI is expected to play a crucial role in military logistics and transport. The effective transportation of goods, ammunition, armaments, and troops is an essential component of successful military operations.</p><p>Integrating AI with military transportation can lower transportation costs and reduce human operational efforts. It also enables military fleets to easily detect anomalies and quickly predict component failures. Recently, the US Army collaborated with IBM to use its Watson artificial intelligence platform to help pre-identify maintenance problems in Stryker combat vehicles.</p><h3>4. Target Recognition</h3><p>AI techniques are being developed to enhance the accuracy of target recognition in complex combat environments. These techniques allow defense forces to gain an in-depth understanding of potential operation areas by analyzing reports, documents, news feeds, and other forms of unstructured information. Additionally, AI in target recognition systems improves the ability of these systems to identify the position of their targets.</p><p>Capabilities of AI-enabled target recognition systems include probability-based forecasts of enemy behavior, aggregation of weather and environmental conditions, anticipation and flagging of potential supply line bottlenecks or vulnerabilities, assessments of mission approaches, and suggested mitigation strategies. Machine learning is also used to learn, track, and discover targets from the data obtained.</p><p>For example, DARPA’s Target Recognition and Adaption in Contested Environments (TRACE) program uses machine learning techniques to automatically locate and identify targets with the help of Synthetic-Aperture Radar (SAR) images.</p><h3>5. Battlefield Healthcare</h3><p>In war zones, AI can be integrated with Robotic Surgical Systems (RSS) and Robotic Ground Platforms (RGPs) to provide remote surgical support and evacuation activities. The US in particular is involved in the development of RSS, RGPs, and various other systems for battlefield healthcare. Under difficult conditions, systems equipped with AI can mine soldiers’ medical records and assist in complex diagnosis.</p><p>For instance, IBM’s Watson research team partnered with the US Veterans Administration to develop a clinical reasoning prototype known as the Electronic Medical Record Analyzer (EMRA). This preliminary technology is designed to use machine learning techniques to process patients’ electronic medical records and automatically identify and rank their most critical health problems.</p><h3>6. Combat Simulation &amp; Training</h3><p>Simulation &amp; training is a multidisciplinary field that pairs system engineering, software engineering, and computer science to construct computerized models that acquaint soldiers with the various combat systems deployed during military operations. The US is investing increasingly in the simulation &amp; training applications.</p><p>The US Navy and Army have each been conducting warfare analysis, which has led to the initiation of several sensor simulation programs. The US Navy has enlisted such companies such as Leidos, SAIC, AECOM, and Orbital ATK to support their programs, while the US Army’s programs are supported by firms including SAIC, CACI, Torch Technologies, and Millennium Engineering.</p><h3>7. Threat Monitoring &amp; Situational Awareness</h3><p>Threat monitoring &amp; situational awareness rely heavily on Intelligence, Surveillance, and Reconnaissance (ISR) operations. ISR operations are used to acquire and process information to support a range of military activities.</p><p>Unmanned systems used to carry out ISR missions can either be remotely operated or sent on a pre-defined route. Equipping these systems with AI assists defense personnel in threat monitoring, thereby enhancing their situational awareness.</p><p>Unmanned aerial vehicles (UAVs) — also known as drones — with integrated AI can patrol border areas, identify potential threats, and transmit information about these threats to response teams. Using UAVs can thus strengthen the security of military bases, as well as increase the safety and efficacy of military personnel in battle or at remote locations.</p><h3>Conclusion</h3><p>The mass adoption of AI in military technology, both hardware and software, presents to us an incredible and frightening paradigm shift in modern warfare. It should be no surprise that the largest militaries in the world are focusing more on this technology than anything else, and the winner of this tech race will likely have more global leverage than the US had after developing the atom bomb.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=9818dfa2ea86" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Microsoft’s Exclusive GPT-3 License]]></title>
            <link>https://medium.com/@nqabell89/microsofts-exclusive-gpt-3-license-3a48af54f921?source=rss-fbd886da6bfa------2</link>
            <guid isPermaLink="false">https://medium.com/p/3a48af54f921</guid>
            <dc:creator><![CDATA[Nicholas Abell]]></dc:creator>
            <pubDate>Fri, 25 Sep 2020 17:05:56 GMT</pubDate>
            <atom:updated>2020-09-25T17:05:56.527Z</atom:updated>
            <content:encoded><![CDATA[<p>On September 22nd, Microsoft announced it would be exclusively licensing GPT-3 as part of its ongoing partnership with Elon Musk’s OpenAI.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*8KuR8Fq5s1U1m8oC9x7qzw.jpeg" /><figcaption>Image credit: openai.com</figcaption></figure><p>This came as no surprise. In 2019, Microsoft invested $1 billion in Elon Musk’s OpenAI. The deep learning research lab had this to say; “OpenAI is producing a sequence of increasingly powerful AI technologies, which requires a lot of capital for computational power. The most obvious way to cover costs is to build a product, but that would mean changing our focus. Instead, <strong>we intend to license </strong>some of our pre-AGI technologies, with Microsoft becoming our preferred partner for commercializing them.”</p><p>Here’s what Kevin Scott, Microsoft’s Chief Technology Officer, <a href="https://blogs.microsoft.com/blog/2020/09/22/microsoft-teams-up-with-openai-to-exclusively-license-gpt-3-language-model/">said on Tuesday</a>: “Microsoft is teaming up with OpenAI to exclusively license GPT-3, allowing us to leverage its technical innovations to develop and deliver advanced AI solutions for our customers, as well as create new solutions that harness the amazing power of advanced natural language generation.”</p><p>Microsoft has a vast and established customer base, with over <a href="https://www.statista.com/statistics/983321/worldwide-office-365-user-numbers-by-country/">a million companies</a> using Office 365, including 800,000 in English-speaking countries. Microsoft also owns Bing, the second-most-popular search engine, and the digital assistant Cortana. These are all areas that heavily rely on <a href="https://bdtechtalks.com/2018/02/20/ai-machine-learning-nlg-nlp/">natural language processing</a> (NLP) technology, and GPT-3 is miles ahead of the competition.</p><p>There’s mutual benefit in the licensing as well. Through Microsoft, OpenAI will gain access to hundreds of millions of users, but more importantly, their interactions with the new NLP platform. This will provide key insights and experience that will allow the research lab to improve and build upon the already impressive GPT-3.</p><p>But the real winner will be Microsoft, who will have exclusive access to the most advanced AI technology.</p><p>Per Scott: “The scope of commercial and creative potential that can be unlocked through the GPT-3 model is profound, with genuinely novel capabilities — most of which we haven’t even imagined yet. Directly aiding human creativity and ingenuity in areas like writing and composition, describing and summarizing large blocks of long-form data (including code), converting natural language to another language — the possibilities are limited only by the ideas and scenarios that we bring to the table.”</p><p>GPT-3 has tons of potential applications within the Microsoft ecosystem, specifically in Word, Excel, and Teams, but these are areas where Microsoft already dominates. A more interesting area to observe is the search engine market. Bing currently has a little over <a href="https://www.statista.com/statistics/216573/worldwide-market-share-of-search-engines/">6 percent of the market share</a>, far behind Google’s 87 percent. GPT-3 may enable Microsoft to roll out new features that will redefine how we use a search engine, which could ultimately bring more users to Bing.</p><p>The key benefit of Microsoft’s GPT-3 license is that the company will not be bound to the API pricing model, and will be subsidizing it for its customers. What this means is that Microsoft customers will probably get free access to GPT-3 features. In conclusion, this is huge news for Microsoft as they continue to expand their ability to compete with the other tech giants.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=3a48af54f921" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[MLaaS Showdown: AWS, Azure, and Google Cloud]]></title>
            <link>https://medium.com/@nqabell89/mlaas-showdown-aws-azure-and-google-cloud-91de47cf3049?source=rss-fbd886da6bfa------2</link>
            <guid isPermaLink="false">https://medium.com/p/91de47cf3049</guid>
            <dc:creator><![CDATA[Nicholas Abell]]></dc:creator>
            <pubDate>Fri, 18 Sep 2020 17:32:26 GMT</pubDate>
            <atom:updated>2020-09-18T17:32:26.976Z</atom:updated>
            <content:encoded><![CDATA[<p>A quick comparison of the machine learning as a service (MLaaS) tools offered by the grand IT trio; Amazon, Microsoft, and Google.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/720/1*HL561dzKR6rN545v1LKjKA.png" /><figcaption>Image credit: techaheadcorp.com</figcaption></figure><p>Amazon, Microsoft, and Google; all three companies are market leaders, not only in the machine learning segment, but in all other IT departments as well. Needless to say, they compete with each other on all fronts, constantly working to develop better, faster, and more affordable products. Their approach to machine learning products may differ, but the majority of available options and APIs accomplish the same sets of tasks— translation, text analysis, image recognition, and so on. To save you from spending too much time researching all of the details regarding each machine learning service, here is a short article summarizing the main strengths of each company’s solutions.</p><h3>Amazon AWS</h3><p>Amazon is known for launching services like <a href="https://aws.amazon.com/rekognition/">AWS Rekognition</a> (image recognition) and <a href="https://aws.amazon.com/polly/">AWS Polly</a> (text-to-speech with deep learning), but still their most popular tool is <a href="https://aws.amazon.com/sagemaker/">Amazon SageMaker</a>, which is designed to simplify the process of creating, training, and deploying machine learning models.</p><p><strong>AWS ML verdict</strong>: the company has invested more than anyone into the diversification of its services and tools. This includes machine learning services as well. Amazon is dedicated to enhancing current AI and ML technologies and expanding the current limits of image, voice, and face recognition capabilities. Pricing of MLaaS is difficult to estimate, but if you need massive cloud storage for data on your project, then you should note that Amazon offers very cost-effective options. It’s recommend to use the cost calculator tool on their official website due to the complex structure of pricing calculations, or maybe even consulting with a third-party to pick the most effective options within your budget.</p><h3>Microsoft Azure</h3><p>As for Microsoft, they are aiming to make ML more accessible for everyone with their machine learning services packed into <a href="https://azure.microsoft.com/">Azure</a>. Based on their surprisingly positive improvements in their latest versions of <a href="https://azure.microsoft.com/en-us/products/visual-studio/">Visual Studio</a>, you can say they are doing it pretty well. They are becoming well liked in the industry for their graphic UI that allows building ML models without extensive knowledge or experience with the subject. Also, some companies have reported that this tool significantly reduces the time required to train and deploy accurate models based on their pre-loaded data offerings.</p><p><strong>Azure ML verdict</strong>: compatibility with Microsoft products certainly opens many possibilities for organizations that are already using MS products. Add open source availability to the mix, and you will get a flexible and adjustable platform. Scale your project up or down whenever you want without worrying about the legacy data as it will be saved by Microsoft. The company may have fewer ML tools than Google and Amazon at the moment, but it is not going to stop investing in machine learning. Right now, Azure is probably the most flexible platform in terms of price on machine learning services. However, it requires thorough analysis, planning, and precise calculations for each project to determine the best provider on the basis of cost.</p><h3>Google Cloud</h3><p>Google Cloud advertises its APIs and machine learning services as one of their main advantages. Their range of various ML and AI-based products, including <a href="https://cloud.google.com/automl/">AutoML</a> and <a href="https://cloud.google.com/ml-engine/?authuser=0&amp;hl=RU">Machine Learning Engine</a>, looks truly impressive, allowing users to build and run superior ML models with ease. Regarding AI tools, Google offers access to its <a href="https://cloud.google.com/ai-hub/">AI Hub</a>, which is a hosted repository that contains various plug-and-play AI components. For example, end-to-end AI pipelines, algorithms that can be used right-of-the-box, AI building blocks, and more.</p><p><strong>Google Cloud ML verdict</strong>: Google Cloud is growing fast, and so is its ML segment. Google is working on many new services, while simultaneously enhancing their existing ones. Even in beta, some APIs are already showing fantastic results, allowing Google to compete with time-tested Amazon services. They don’t have much to offer in the IoT department, but if that’s of no concern, and you would rather sacrifice it to have cost-effective ML services, then Google ML might be what you need. Google’s pricing is more transparent and client-friendly. In addition to that, the company has started a trend of giving out major discounts and other bonuses to win market share from Amazon and Microsoft.</p><h3>Conclusion</h3><p>Every ML project is unique with its requirements, features, and design. If one service works best for one organization, it is not guaranteed that it will do the same for yours. Another major factor is pricing — companies have different budgets, so recommending top solutions for small businesses that need simple ML tools may be serious overkill. Nevertheless, I hope this short summary of the ‘big 3’ MLaaS cloud platforms has helped guide you in your research.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=91de47cf3049" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Holt Winters Time Series Forecasting By Hand]]></title>
            <link>https://medium.com/@nqabell89/holt-winters-time-series-forecasting-by-hand-807a13cd608e?source=rss-fbd886da6bfa------2</link>
            <guid isPermaLink="false">https://medium.com/p/807a13cd608e</guid>
            <dc:creator><![CDATA[Nicholas Abell]]></dc:creator>
            <pubDate>Fri, 11 Sep 2020 16:54:10 GMT</pubDate>
            <atom:updated>2020-09-11T16:54:10.930Z</atom:updated>
            <content:encoded><![CDATA[<p>Holt-Winters forecasting is powerful, but often misunderstood. Let’s skip the math and <em>visualize</em> how it works.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/659/1*iF_7h2324CnNzlK49wxfeg.png" /><figcaption>Image credit: magoosh.com</figcaption></figure><p>The usual way to explain Holt-Winters is by showing a bunch of complicated equations. While understanding the underlying mathematics is crucial to proper application of any time series modelling, we’ll skip the math for now and start with breaking it out into simple and visualized steps. We’re going to be working with this time series:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/749/1*R85c9ULL7fY0x01IpcEQvw.png" /></figure><p>The pattern is obvious: the plot repeats the values [0, 1, 0, 0, 0]. Can you tell me what the next 5 values are going to be? Of course you can, because I just told you! They are [0, 1, 0, 0, 0].</p><p>What would it look like if we made the values <em>relative to the average</em> of those 5 points? The average of (0+1+0+0+0)/5 is 0.2, which we’ll draw on the plot as a horizontal line:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/749/1*_72VEeRgfocX70VtVDYsMw.png" /></figure><p>Recall that Holt-Winters has a trend component. If we set its parameter to zero, Holt-Winters ignores the trend (slope), so the model simplifies. Now, it’s just a bunch of values relative to the average. In our plot, the values relative to 0.2 are [-0.2, 0.8, -0.2, -0.2, -0.2]. If we did Holt-Winters without trend, that’s the type of model we’d build.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/749/1*d1_cbR7kllqD7S79nPCPPg.png" /></figure><p>Forecasting with trend is just an enhancement of this. Instead of using a fixed average as the foundation, you just have to incorporate the slope of the line. Here’s an example model that has a trend:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/749/1*PrqfLa1aZHGZgb28zl0Idg.png" /></figure><p>You already know that the example series repeats itself every five points, or the seasonality has 5 periods. What if you didn’t know what the season is for a time series? How can you figure it out? What are the consequences of being wrong?</p><p>The right seasonality is crucial to Holt-Winters forecasting. To illustrate this, let’s see what happens when you use a season of 6 periods, one greater than the actual season of 5 periods:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/749/1*czpRiDWBm_y0MZnzfHx9gg.png" /></figure><p>The forecast, which is the red line in the chart, becomes less accurate and turns into garbage. To get good results, you need to give the model good parameters. This is the second challenge with Holt-Winters forecasting.</p><p>Picking the seasonality is a hard problem. General-purpose forecasting is hard because it has to be ready to use on any dataset, which might have any combination of values, trend, and seasonality. It might not even have some of those components. These are usually unknowns unless you’re manually inspecting the data and customizing the model for it.</p><p>Let’s quantify how good a forecast is. The last forecast was bad, but <em>how bad?</em> The usual way to quantify the accuracy of a forecast is to calculate the differences between the predicted values and the actual values. The blue arrows in the following chart represent how far off the prediction was from the actual value.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/749/1*LB4q-Hb8PBoesEZUxnKazg.png" /></figure><p>To quantify overall accuracy, you can combine these differences into a single value by taking the average or the sum of squared values. The result is a value that is smaller if the forecast is better, and larger if the forecast is worse. This gives you a good way to compare forecast results.</p><h3>Conclusion</h3><p>Holt-Winters forecasting is surprisingly powerful despite its simplicity. It can handle lots of complicated seasonal patterns by simply finding the central value, then adding in the effects of slope and seasonality.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=807a13cd608e" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Microsoft Debuts New BERT-Based NLP Model for PubMed]]></title>
            <link>https://medium.com/@nqabell89/microsoft-debuts-new-bert-based-nlp-model-for-pubmed-216724ce2e1d?source=rss-fbd886da6bfa------2</link>
            <guid isPermaLink="false">https://medium.com/p/216724ce2e1d</guid>
            <dc:creator><![CDATA[Nicholas Abell]]></dc:creator>
            <pubDate>Fri, 04 Sep 2020 18:38:08 GMT</pubDate>
            <atom:updated>2020-09-04T18:38:08.557Z</atom:updated>
            <content:encoded><![CDATA[<p>The open-source nature of machine learning and NLP was bound to lead to one tech giant (Microsoft) building off of another’s work (Google), and now we have PubMedBERT</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/959/1*jp24nDt57auTd2WSXFHq7g.png" /><figcaption>Image credit: <a href="https://www.microsoft.com/en-us/research/">https://www.microsoft.com/en-us/research/</a></figcaption></figure><p>Last month, Microsoft Research <a href="https://www.microsoft.com/en-us/research/blog/domain-specific-language-model-pretraining-for-biomedical-natural-language-processing/">published a paper</a> focusing on a new approach to natural language processing in the field of medical research. Using domain-specific knowledge when pre-training their model, which in this case are biomedical research papers in the medical database PubMed, the team at Microsoft built off of Google’s BERT to create a detailed NLP benchmark, sourced from public data sets to generate more accurate results.</p><p>According to the team, the new model can classify PubMed documents, extract evidence-based medical information, recognize named entities (like specific molecules), and much more. From my experience working with research teams at several major US pharmaceutical companies, I can tell you this has been a source of pain for decades. I have no doubt this will be put to tremendous use in the coming years, as biomedical research continues to grow at a rapid pace.</p><p>As showcased in the image above, while domain-specific training data is essential, previous testing produced less accurate results when combined with “out-of-domain” data. For their new pre-training model, the team showed domain-specific training on its own outperforms the generic mixed ‘in and out’ approach.</p><p>From the research paper;</p><blockquote><em>“To facilitate this study, we compile a comprehensive biomedical NLP benchmark from publicly available datasets and conduct in-depth comparisons of modeling choices for pretraining and task-specific fine-tuning by their impact on domain-specific applications. Our experiments show that domain-specific pretraining from scratch can provide a solid foundation for biomedical NLP, leading to new state-of-the-art performance across a wide range of tasks.”</em></blockquote><p>The vocabulary was generated using the latest PubMed document database; including roughly 15 million summary abstracts, and over 3.2 billion words, which amounted to 21GB of raw text. Using a single Nvidia DGX-2 powered machine with 16 V100 GPUs, Microsoft stated that the training took five days.</p><p>Built on Google’s BERT, the new model is called PubMedBERT, and it has consistently outperformed similar NLP models when applied to medical research papers.</p><blockquote><em>“We show that domain-specific pretraining from scratch substantially outperforms continual pretraining of generic language models, thus demonstrating that the prevailing assumption in support of mixed-domain pretraining is not always applicable.”</em></blockquote><p>I have no doubt that this will end up saving thousands of hours of time, allowing pharma research teams to go beyond the badly dated PubMed website search features to find what they are looking for 100x faster. This will, in turn, allow these companies to perform more efficient research with 1/10th the staff. Either that, or they will free up a considerable amount of time for them to focus on innovating and thinking outside the box. One way or another, this NLP model will lead to lower costs and increased revenues for the pharma and biotech industries.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=216724ce2e1d" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Back to Basics: NLP First Steps]]></title>
            <link>https://medium.com/@nqabell89/back-to-basics-nlp-first-steps-7703824ba32f?source=rss-fbd886da6bfa------2</link>
            <guid isPermaLink="false">https://medium.com/p/7703824ba32f</guid>
            <category><![CDATA[nlp]]></category>
            <dc:creator><![CDATA[Nicholas Abell]]></dc:creator>
            <pubDate>Thu, 27 Aug 2020 20:24:31 GMT</pubDate>
            <atom:updated>2020-08-27T20:24:31.379Z</atom:updated>
            <content:encoded><![CDATA[<p>Whether you’re new to NLP , or a long-time veteran, it can be helpful to review a list of basic steps to consider in the early stages of a project.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*lkZl5-ZogDbf3xU42W0x4g.jpeg" /><figcaption>Image credit: aptude.com</figcaption></figure><h3>The Basics</h3><p>It goes without saying, but the input in any natural language processing operation is a simple string of unicode characters (typically UTF-8). NLP seeks to convert these characters into a sequence of items (words, phrases, and syntactic markers) which can then be used to better understand the content.</p><p>The first step to processing text involves some or all of the following operations:</p><ul><li><strong>Structure extraction</strong> — identifying fields and blocks of content based on tagging (sentences, paragraphs, manual text entries, tweets, legal documents, sections of financial filings, etc)</li><li><strong>Identify sentence, phrase, and paragraph boundaries</strong> — these markers are important when extracting names/entities, and serve as useful breaks for statistical models.</li><li><strong>Language identification</strong> — will detect the language for the entire document and for each paragraph or sentence. Language detectors are critical to determine what linguistic algorithms and dictionaries to apply to the text, although typically this step is reserved for very large projects involving a company’s global operations.</li><li><strong>Tokenization</strong> — to divide up strings into tokens which can be used for further processing, and it’s a foundational step for statistical models that measure the distance between tokens to predict topics and sequences. Tokens can be words, sentences, phrases, numbers, identifiers, or punctuation (depending on the use case).</li><li><strong>Acronym normalization and tagging </strong>— acronyms can be specified as “I.B.M.” or “IBM” so these should be tagged and normalized.</li><li><strong>Lemmatization &amp; Stemming</strong> — reduces word variations to simpler forms that may help increase the effectiveness of NLP utilities. Lemmatization is strongly preferred to stemming, but is often performed one after the other for maximum coverage. For example: ‘Caring’ -&gt; <em>Lemmatization</em> -&gt; ‘Care’, ‘Caring’ -&gt; <em>Stemming</em> -&gt; ‘Car’.</li><li><strong>Decompounding</strong> — for some languages (typically Germanic, Scandinavian, and Cyrillic languages), compound words will need to be split into smaller parts to allow for accurate understanding. For example: “<em>samstagmorgen</em>” is “Saturday Morning” in German.</li><li><strong>Named entity extraction</strong> — identifying and extracting named entities (people, places, companies, street names, etc.) is a necessary step to simplify downstream processing. This can be performed a variety of ways, but some of the more common are <strong><em>Regex Extraction </em></strong>(good for phone numbers, e-mail addresses, URLs, hashtags, credit card numbers, etc), and <strong><em>Dictionary Extraction </em></strong>(uses a dictionary of terms to identify known entities, such as days, months, employees, business groups, products, brands, etc)</li><li><strong>Phrase extraction</strong> — extracts sequences of tokens (phrases) that have a strong meaning which is independent of the words when treated separately. These sequences should be treated as a single unit when doing NLP. For example, “Big Data” has a strong meaning which is independent of the words “big” and “data” when used separately. All companies have these sorts of phrases which are in common usage throughout the organization and are better treated as a single token. Techniques to extract phrases include <strong><em>Part of speech tagging</em></strong> (identifies phrases from noun or verb clauses), and <strong><em>Statistical phrase extraction </em></strong>(identifies token sequences which occur more frequently than statistically allowed by chance).</li></ul><h3>The Conclusion</h3><p>Most of the above steps are built into various text tokenization and cleaning functions/scripts, but should be customized to better fit the type of text you’re working with. It’s also not essential to cover each of these steps in every project, which is why NLP models typically have more “art” than “science”, at least when compared to other machine learning models. Having a firm grasp on the text you are processing should help you figure out which of these basic steps will be needed to achieve your end goals.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=7703824ba32f" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Is Your NLP Project Feasible?]]></title>
            <link>https://medium.com/@nqabell89/is-your-nlp-project-feasible-342c335d9ebe?source=rss-fbd886da6bfa------2</link>
            <guid isPermaLink="false">https://medium.com/p/342c335d9ebe</guid>
            <category><![CDATA[nlp]]></category>
            <dc:creator><![CDATA[Nicholas Abell]]></dc:creator>
            <pubDate>Fri, 21 Aug 2020 16:55:30 GMT</pubDate>
            <atom:updated>2020-08-21T16:55:30.550Z</atom:updated>
            <content:encoded><![CDATA[<p>NLP project feasibility can be difficult to assess, but it is an essential step for any organization looking to avoid the pitfalls of runaway costs commonly associated with poorly planned machine learning initiatives.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/512/1*sRs3bv7df7BrSSS_WShIJg.jpeg" /><figcaption>Image credit: <a href="http://www.uaeauditors.net/index.php/feasibility-study.html">http://www.uaeauditors.net</a>/</figcaption></figure><p>Not all NLP projects are feasible within a reasonable budget and timeline. In this article, we’ll explore a useful flowchart (developed by the folks at Search Technologies Inc, now owned by Accenture), as well as a few key items to consider in your project’s feasibility study.</p><h3>The Flowchart</h3><p>Search Technologies developed a flowchart to decide whether your requirements are likely to be manageable with today’s NLP tools and techniques. It’s a great tool to wrap your head around the various boxes that need to be checked when calculating the risk associated with your project. Check it out below, then continue reading for some more details behind it’s key components.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*nLctk31Kihj8w_2ALhK7jg.jpeg" /><figcaption>Image credit: <a href="https://www.searchtechnologies.com/">https://www.searchtechnologies.com/</a>, now owned by Accenture</figcaption></figure><h3>The Details</h3><h4><strong>Accuracy</strong></h4><p>Accuracy in this context is the percentage of records/outputs where the correct answer is produced by the given prediction algorithm.</p><p>High accuracy systems (&gt; 80%) generally require much more work to handle a larger number of variations in the text. The machine learning and deep learning components of a project that produce highly accurate results also require a considerable amount of time for comparing various algorithms/approaches, and usually requires beefy hardware/computational resources.</p><p>Lower accuracy systems (&gt; 60% &lt; 80%) are often still useful for large-scale analytics and trend analysis. This can especially be true for projects in the financial services domain.</p><h4><strong>Coverage</strong></h4><p>Coverage in this context is the percentage of applicable records for which an answer/output is required. An “applicable record” is a record which contains text that requires a prediction.</p><p>Like high accuracy systems, high coverage systems generally require much more work to handle volume of text variations.</p><h4><strong>Affordability</strong></h4><p>How soon does this project need to be completed? Is the accuracy of the model more or less important than a speedy deployment? What is your budget? Do you have the ability to cover additional costs associated with unforeseen problems with access to data, or data cleanliness? If you talk to anyone who has worked on or payed for a large scale NLP project in the past, they will likely tell you all about how common it is for development time to take up to 3x longer than anticipated.</p><h4><strong>Macro vs. Micro Understanding</strong></h4><p><strong>Macro Understanding</strong> — provides a general understanding of the document as a whole.</p><ul><li>Typically performed with statistical models (like LDA)</li><li>Used for: topic clustering/categorization, text similarity, word clouds, and automatic text summarization.</li></ul><p><strong>Micro Understanding</strong> — extracts understanding from individual phrases or sentences.</p><ul><li>Typically performed with NLP techniques (like stemming, lemmatization)</li><li>It is used for: extracting facts, entities (like names, places, events), entity relationships, actions, and metadata fields</li></ul><p>While micro understanding is generally included in the early stages of macro understanding, the two can be entirely different. For example, a resume/CV may identify a person, overall, as a <strong><em>Data Scientist</em></strong> [macro understanding] but it can also identify them as being<em> </em><strong><em>fluent in French</em></strong> [micro understanding].</p><h4><strong>Training Data</strong></h4><p>Training data is typically required to train statistical models for many types of NLP projects.</p><p>Training data may already be available if:</p><ul><li>The system is replacing a process which was previously done manually.</li><li>The system is filling in gaps for manually entered values, for example, by end users filling out forms.</li><li>Public training data is available from scraping a website or via NLP databases.</li></ul><p>If training data is not available and cannot be inferred, then it will need to be created manually for most types of macro understanding projects.</p><p>Depending on the scope of the project, this can be done by just a few people with skills in scripting languages like Python, or it could require a larger team putting it many hours of manual work.</p><h4><strong>Text Length</strong></h4><p>Generally, short text contains fewer variations and is, therefore, easier to process for micro understanding. Example; tweeted headlines vs. entire news articles.</p><h4>Text Author Similarity</h4><p>If the authors of the text/inputs are similar across the board, then they will typically produce fairly regular text that spans a fairly narrow domain.</p><p>Examples include employees, airline pilots, computer programmers, maintenance engineers, librarians, users trained on a certain product, contract lawyers, etc.</p><p>On the other hand, if the authors cover a wide range of backgrounds, education levels, and language skills (like customers), then typically they will produce a wide range of text variation across a wide domain. This will be more difficult to process.</p><h4><strong>Academic/Journalistic/Narrative Based Text</strong></h4><p>Text written by professional writers tends to be longer, more varied, and have more complex sentence structure, all of which can prove more difficult to understand by machine learning models. This is where custom stop-word lists (words that are common enough to not be of use in the analysis) can be very helpful, for example; common banking terms that would appear in every loan contract, and can be removed across the entire corpus of documents to reduce complexity.</p><h4><strong>Human Intervention &amp; Judgement</strong></h4><p>In some NLP systems, like unsupervised topic modelling, there will be a human required to review the results. Such applications will generally be more tolerant of errors produced by the algorithms, but can of course lead to longer development time.</p><p>For example, a system may extract statements which indicate compliance violations. These would, of necessity, need to be checked by a compliance officer to determine if a rule was actually violated.</p><h4><strong>Known Entities vs. Entity Recognition</strong></h4><p>Known entities come from entity lists that have been previously gathered ahead of time. These can be items like employees (from an employee directory), office locations, public companies, countries, etc.</p><p>Unknown entities are not previously known to the system, and require an extra layer of coding to determine (entity recognition) by looking at names in context of topics and sentence structure.</p><p>If entity-tagged text is not available, then it may need to be manually created. This can be an expensive process, since, usually 500+ examples are required to achieve decent accuracy. To achieve higher accuracy (&gt; 80%), as many as 2,000 to 5,000 examples may be required, and this can obviously add additional time and cost requirements.</p><h3>The Conclusion</h3><p>As you can see, a lot of time and thought has gone into structuring a process for studying the feasibility of an NLP project. This is a direct result of the countless ‘nightmare’ stories of dramatically underestimated costs, but is common in any tech industry as cutting edge as natural language processing.</p><p>If you are working with a consultant, or a third-party machine learning shop, they will likely be doing some form of feasibility analysis like the above, but it will only help you in the long run if you add your own thoughts and answers to these questions before bringing pen to paper.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=342c335d9ebe" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Hypothesis Testing In The Real World]]></title>
            <link>https://medium.com/@nqabell89/hypothesis-testing-in-the-real-world-305a13cc6c20?source=rss-fbd886da6bfa------2</link>
            <guid isPermaLink="false">https://medium.com/p/305a13cc6c20</guid>
            <dc:creator><![CDATA[Nicholas Abell]]></dc:creator>
            <pubDate>Fri, 14 Aug 2020 22:46:11 GMT</pubDate>
            <atom:updated>2020-08-16T10:18:37.505Z</atom:updated>
            <content:encoded><![CDATA[<p>How statistics and Python join forces to explore real business problems.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*D-b7QuJ6GVo0NfUCpamRNQ.jpeg" /><figcaption>Image credit: <a href="http://www.stepupanalytics.com/testing-of-hypothesis-and-its-application-using-r/">http://www.stepupanalytics.com/</a></figcaption></figure><p>This article provides an example of how a data scientist can use hypothesis testing to produce sound summary statistics on any business problem or research project. The example below is a portion of my project to provide home builders and renovators in the state of Illinois with statistical analysis of the economic impact of building homes with solar panel systems, or adding solar panels to existing homes on the market.</p><h3>Hypothesis: Homes with solar panel systems have a higher average value than homes in general.</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/153/1*_UpXKceQW4Nab3o3bg9E6Q.png" /></figure><p>This first test is to compare the average sale price of a subset of new and existing home sales that have solar panel systems, with the average sale price of home sales in general. In this example, our <em>null hypothesis </em>(H0) is that the average sale price of our sample of homes with solar panels installed is <em>the same or less than </em>the average sale price of the population (of illinois single family homes).</p><p>First, I calculate the necessary metrics to perform a <em>Right Tailed Z-Test </em>using data from a database of housing information in the form of Pandas dataframes. The first dataframe contains the entire population of homes (population_df), and the second contains a sample of that population that has solar panels installed (with_df). The feature used in this test is Sale Price (Sale Pr).</p><pre>#population mean, variance, and standard deviation<br>pop_mean = round(population_df[&#39;Sold Pr&#39;].mean(), ndigits=2)<br>pop_var = round(population_df[&#39;Sold Pr&#39;].var(), ndigits=2)<br>pop_std = round(population_df[&#39;Sold Pr&#39;].std(), ndigits=2)</pre><pre>#sample mean, variance, and standard deviation<br>sol_mean = round(with_df[&#39;Sold Pr&#39;].mean(), ndigits=2)<br>sol_var = round(with_df[&#39;Sold Pr&#39;].var(), ndigits=2)<br>sol_std = round(with_df[&#39;Sold Pr&#39;].std(), ndigits=2)</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/405/1*hACU-VjhM9Kiw31CofvONQ.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/354/1*oV_2HfS5H_nlhQHtNKaTzA.png" /></figure><h4>Right Tailed Z-Test</h4><p>A Z-test is any statistical test for which the distribution of the test statistic under the null hypothesis can be approximated by a normal distribution. Right tailed Z-tests compare the mean of a distribution to the mean of a sample group from that distribution, where we expect the <em>alternative hypothesis </em>to be true if the mean of the sample is significantly higher than the mean of the population. For each significance level in the confidence interval, the Z-test has a single critical value.</p><p>Due to the central limit theorem, many test statistics are approximately normally distributed for large samples. Therefore, many statistical tests can be conveniently performed as approximate Z-tests if the sample size is large or the population variance is known, both of which hold true in our example.</p><pre>z = (sol_mean-pop_mean)/(pop_std/np.sqrt(57))<br>p_value = 1 - stats.norm.cdf(z)<br>alpha = 0.05</pre><pre>print(f&#39;P Value: {p_value}&#39;)</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/309/1*LY0I5THZWmGaoQPq0plbVw.png" /></figure><p>As we can see, our P Value is significantly smaller than our alpha of .05, so from this alone we can say that we have enough evidence to reject the null hypothesis. It might help to visualize it just to be sure.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*O1FHC_Cf_nedxRsVyzgSkA.png" /></figure><p>Based on the findings above, we can definitively reject our null hypothesis. It is clear that homes with solar panels sell for a significantly higher average selling price than homes without.</p><p>This does not imply that adding solar panels increases the value of your home. To make this claim, we would need to take into account the dozens of other variables that impact a home’s sale price, and use more sophisticated models to isolate solar panels as a feature.</p><p>The findings make sense, because solar panel systems are expensive, and so home owners who can afford them likely have a more valuable home to begin with.</p><h4>Conclusion</h4><p>Hypothesis testing is one of the first methods learned in any statistics course, and is foundational to any data analysis that may lead to important or expensive business decisions or strategy. Formulating your question into a null and alternative hypothesis can help you make sure that your results in your summary statistics are not simply due to chance, but are in fact valid enough to include in your findings. Thanks for reading!</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=305a13cc6c20" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Better Late Than Never — Private Equity is Being Transformed by Machine Learning]]></title>
            <link>https://medium.com/@nqabell89/better-late-than-never-private-equity-is-being-transformed-by-machine-learning-580ca22a2786?source=rss-fbd886da6bfa------2</link>
            <guid isPermaLink="false">https://medium.com/p/580ca22a2786</guid>
            <dc:creator><![CDATA[Nicholas Abell]]></dc:creator>
            <pubDate>Thu, 06 Aug 2020 17:19:18 GMT</pubDate>
            <atom:updated>2020-08-06T17:19:18.454Z</atom:updated>
            <content:encoded><![CDATA[<h3>Better Late Than Never — Private Equity is Being Transformed by Machine Learning</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/770/1*064TU5N_wk49gr-qcepiIA.jpeg" /><figcaption>Image credit: wealthtriumph.com</figcaption></figure><p>Data science and machine learning has remained one of the top areas in business transformation for a few years now. Companies are increasingly adopting it to transform sales, marketing, talent acquisition, business development, operations, and many other departments across a wide variety of industries. This is partly due to the hype; ML holds the promise of more objective decision-making and a stronger bottom line, but as professionals in the field know, it actually delivers on that hype, perhaps more than any other new technology today.</p><p>When it comes to the world of private equity, however, it’s a different story. Up until 2013/2014, there had been no technological innovation in the private equity workflow since the invention of Microsoft Excel.</p><h4>Blazing A Path for ML in PE</h4><p>Two Six Capital, one of the first PE firms started on the foundation of advanced analytics practices, has essentially pioneered the use of machine learning in private equity, and over the last 7 years has closed over $27 billion worth of transactions. They think of their company as “launching the next wave around data-powered investing.”</p><p>Sajjad Jaffer, co-founder of Two Six, spoke about their methodology at a conference in San Francisco last year. He said his firm’s approach to understanding and projecting a portfolio company’s revenue can be broken down into two distinct pieces; “transaction due diligence, or “understanding a business from the bottom up,” and value creation post-investment, which Jaffer describes as producing “board-level insights … understanding how a portfolio company performs and then driving operational improvements. Using large-scale cloud-based engineering, we can handle very large data sets in very short time-frames. Billions of rows of data.”</p><p>Jaffer also spoke about the difficulty for any investment firm of accurately predicting a portfolio company’s growth, and how data analytics can change the game. “Right now in the industry, when most associates at a private equity firm go to sell a company, they just put ‘12%’ in there. It is a relatively finger-in-the air approach,” he said. “But we — because we count every customer, every transaction — get a much more granular perspective.”</p><p>Two Six Capital relies on 18 machine learning models, built over 28 valuation and analytics methodologies, to understand the vast amounts of data and make decisions in real time.</p><p>They come to their conclusions fast, Jaffer said, asserting that speed is one of Two Six’s competitive advantages. “In commercial diligence, you’ve got four to six weeks to make a go/no-go decision on the deal. We can take in large volumes of data, and quickly come up with a point of view on the question ‘Is this company going to make returns — yes or no?’”</p><h4><strong>Finding Disconnects; Toppling Conventional Wisdom</strong></h4><p>Jaffer also discussed how using data can reveal insights that are completely missed in favor of conventional wisdom; i.e. ‘bad assumptions’. He gave the example of a business that operated in over 120 countries, and after modeling out their business in each individual market, they found each country had a different level of market penetration. In one country that was seen as critical to the business, this company was working alongside Bain on the transaction. “Bain said that Region A was a mature market with a lot of competitors in the space; it’s not going to grow very much.” On the surface, Bain’s argument made a lot of sense, but when the team actually examined the data, a different scenario was uncovered. “The data showed that we were at the beginning of a growth trajectory, and that the growth was very likely going to persist,” said Jaffer. “Because we time-stamp every single time a new customer comes into the system … we knew that the velocity at which customers were entering … was accelerating.”</p><p>Jaffer said Two Six Capital’s prediction turned out to be correct. Region A was the engine of the business, experiencing 100% growth last year.</p><h4>In Conclusion</h4><p>If the growth of adoption of data science in private equity ever fails to increase, it will be due to bad company culture rather than availability. Eventually, like in every other niche of the investment world, the dramatic increase in performance/decrease in costs that comes with the implementation of big data, machine learning, and data science, will separate the winners from the losers. The success of Two Six Capital will likely continue to prove as a shining example.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=580ca22a2786" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Visualizing Unsupervised Text Classification Output with pyLDAvis]]></title>
            <link>https://medium.com/@nqabell89/visualizing-unsupervised-text-classification-output-with-pyldavis-370c1784ed58?source=rss-fbd886da6bfa------2</link>
            <guid isPermaLink="false">https://medium.com/p/370c1784ed58</guid>
            <category><![CDATA[topic-modelling]]></category>
            <category><![CDATA[lda]]></category>
            <category><![CDATA[nlp]]></category>
            <category><![CDATA[unsupervised-clustering]]></category>
            <category><![CDATA[finance]]></category>
            <dc:creator><![CDATA[Nicholas Abell]]></dc:creator>
            <pubDate>Fri, 31 Jul 2020 16:31:26 GMT</pubDate>
            <atom:updated>2020-07-31T16:31:26.956Z</atom:updated>
            <content:encoded><![CDATA[<p>A more efficient review process for unsupervised topic modelling with LDA output visualization package <strong>pyLDAvis.</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*g5OAK61mI74_zpRGN4JebQ.png" /><figcaption>pyLDAvis interactive LDA model output in Jupyter Notebook.</figcaption></figure><p>The purpose of this article is to introduce a tool that allows you to more easily work through the ‘<em>model, check output, tweak parameters, repeat</em>’ component of unsupervised topic modelling; specifically when it comes to using the <strong>Latent Dirichlet Allocation</strong> (LDA) algorithm. This article does <em>not</em> cover an end-to-end approach to unsupervised topic modelling, nor does it discuss the mathematics behind the LDA algorithm, or the early stage processes in NLP, like text cleaning. If you are reading this, I am assuming you have used LDA for topic modelling before, and that you have an understanding of what terms like “tokenization” and “stop-words” mean. Now let’s jump in!</p><h4><strong>Step 0: Domain Knowledge and Assumptions</strong></h4><p>Unsupervised classification in topic modelling is a very unique (and powerful) approach to machine learning, not only because it requires far more manual review of model outputs and human intervention than typical ML solutions, but because it also requires a deep level of domain knowledge surrounding whatever type of documents/text you are attempting to classify.</p><p>You are not working with labeled data, and because of this you cannot score your model outputs in any reliable way. Sure, for algorithms like LDA you can measure Perplexity, or Log-Liklihood, but studies have shown that there is no correlation between optimizing for these scores and human judgement of the success or failure of topic outputs after manual review. Crazy huh? Where in machine learning have you heard that before?</p><p>The result of these truths is that you need a reliable way to run your LDA model, interpret the results by manually reviewing each topic of terms, and deciding which ones are accurately capturing a topic you were expecting to see in the model’s output in the first place.</p><p>In the example below, you will see that I hard-coded 20 topics (n-components parameter) into my LDA model, this is because I spent many hours over many days manually reviewing my documents, and came up with a rough number of distinct topics I would expect to find.</p><p>I was only able to do this because I have extensive understanding of the origin and purpose of these documents; <strong>financial filings of public US companies where they report unscheduled material or financial events (8Ks)</strong>.</p><h4>Step 0.5: Data Cleaning and Tokenization</h4><p>We’re skipping this step, mostly because it’s work that needs to be done in every topic modelling/NLP project.</p><p>Starting in the next step, we’ll be working with a Pandas dataframe (df), and a column in that dataframe called ‘<em>body</em>’, representing the body of cleaned text for each document.</p><h4>Step 1: Count Vectorization of Tokenized Text</h4><p>First we must build a document/topic matrix using sklearn’s <em>CountVectorizer</em> function. It’s best practice to build a custom tokenizer function that best fits your text, and to call that function in the count vectorizer parameters.</p><pre>from sklearn.feature_extraction.text import CountVectorizer</pre><pre>cv = CountVectorizer(tokenizer=custom_tokenizer,    stop_words=&#39;english&#39;)<br>dtm = cv.fit_transform(df[&#39;body&#39;])</pre><h4>Step 2: LDA Model, Fit + Transform Document/Term Matrix (dtm)</h4><pre>from sklearn.decomposition import LatentDirichletAllocation</pre><pre># Build LDA Model with GridSearch params</pre><pre>lda_model = LatentDirichletAllocation(n_components=20,<br>                                      learning_decay=0.5,         <br>                                      max_iter=50,                <br>                                      learning_method=&#39;online&#39;,   <br>                                      random_state=42,            <br>                                      batch_size=5000,            <br>                                      evaluate_every = -1,        <br>                                      n_jobs = -1)                </pre><pre>lda_output = lda_model.fit_transform(dtm)</pre><h4>Step 3: Manual Review of Top Topic Features for Each Topic</h4><p>There are many ways you can accomplish this step, and the overall purpose of this article is to give you a better tool for manually reviewing your topics by the words or n-grams that the LDA model used. However, I still suggested visualizing the top 10 or 15 tokens/features of each topic like I do below.</p><pre>for index,topic in enumerate(lda_model.components_):<br>    print(f&#39;THE TOP 15 WORDS FOR TOPIC #{index}&#39;)<br>    print([cv.get_feature_names()[i] for i in topic.argsort()[-15:]])<br>    print(&#39;\n&#39;)</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*4E625AR7JhU7O8nKu-uuTQ.png" /><figcaption>Example output.</figcaption></figure><h4>Step 4: pyLDAvis-Interactive Visualization of LDA Model Output</h4><p>As stated in the previous step, I recommend taking several approaches in manually reviewing your model outputs, but one of the best tools out there is a python package called pyLDAvis.</p><p>Once you’ve installed it, you’ll be able to see the visual interpretation of how unique/differentiated each topic is (the chart on the left), and you can hover over each circle/cluster representing each topic to see which terms/ngrams were most frequent in that topic on the right.</p><pre>!pip install pyLDAvis</pre><pre>pyLDAvis.enable_notebook()<br>panel = pyLDAvis.sklearn.prepare(lda_model, dtm, cv, mds=&#39;tsne&#39;)<br>panel</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/600/1*_E8bBopZrj55o5cfmoWX9w.png" /><figcaption>Left portion of visualization. Note that hovering over a bubble will highlight in red.</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/864/1*vQzPBAuiwXfcaDdch6pQ_g.png" /><figcaption>After hovering over a topic on the left, you will see the term frequency of the top terms within that topic (red) and across the entire data set (light blue).</figcaption></figure><h4>Step 5: Attribute Labels to Topics</h4><p>In the above step, you will see that we were able to tweak our model to output a set of differentiated topics with no major overlap (the distance map). Once you are confidant in your output, it’s time to manually attribute a label to each topic.</p><p>This is where domain knowledge comes into play. In the example below, I can see that these terms are clearly referring to a specific type of event/topic that I was expecting to see in my results; <strong>a company issuing common stock for purchase by shareholders.</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/623/1*f22ep-c5wy_YimPSF5Gz0A.png" /><figcaption>I was only able to identify what this topic should be labeled as due to my familiarity with the document type and industry (public company finance).</figcaption></figure><h4>Summary</h4><p>In review, we skipped a lot of the unsupervised topic modelling workflow, but only because it mirrors most other NLP machine learning projects, and instead focused on the most unique aspect to this approach; manually reviewing outputs and interjecting human judgement into model success. One of the best ways to accomplish this task is to visualize the uniqueness and ‘distance’ between our topic outputs using the <strong>pyLDAvis</strong> package.</p><p>Thanks for reading! Feel free to share any other approaches you have to reviewing outputs from unsupervised topic modelling in the comments below.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=370c1784ed58" width="1" height="1" alt="">]]></content:encoded>
        </item>
    </channel>
</rss>