<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Josiah Adesola on Medium]]></title>
        <description><![CDATA[Stories by Josiah Adesola on Medium]]></description>
        <link>https://medium.com/@josiah-adesola?source=rss-8d681a6b68c7------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/1*2zBvhPaDuG3gQmnPHn1m0w.jpeg</url>
            <title>Stories by Josiah Adesola on Medium</title>
            <link>https://medium.com/@josiah-adesola?source=rss-8d681a6b68c7------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Sat, 21 Nov 2026 09:06:06 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@josiah-adesola/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[How to Build A Website Query ChatBot: A Step-by-Step Guide]]></title>
            <link>https://ai.plainenglish.io/how-to-build-a-website-query-chatbot-a-step-by-step-guide-4c9c79a3e33f?source=rss-8d681a6b68c7------2</link>
            <guid isPermaLink="false">https://medium.com/p/4c9c79a3e33f</guid>
            <category><![CDATA[langchain]]></category>
            <category><![CDATA[openai]]></category>
            <category><![CDATA[llm]]></category>
            <dc:creator><![CDATA[Josiah Adesola]]></dc:creator>
            <pubDate>Sun, 29 Oct 2023 17:35:11 GMT</pubDate>
            <atom:updated>2024-01-12T04:18:51.058Z</atom:updated>
            <content:encoded><![CDATA[<p>This article demonstrates the creation of a chatbot for querying information from any website. You will acquire the skills to construct a fully operational web interface utilizing LangChain, OpenAI, and Streamlit through a step-by-step guide within this tutorial.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*q2bmkppAAdXZWmQXWZ2nHw.png" /><figcaption>Image from <a href="https://josiah-adesola.medium.com/">Author</a></figcaption></figure><h3>Table of Content</h3><ol><li><a href="#9ea2">Introduction</a></li><li><a href="#f220">Integrating External Knowledge with Large Language Models (LLMs)</a></li><li><a href="#cf75">Building a website query chatbot</a></li><li><a href="#f1bf">Step-by-Step Guide</a></li><li><a href="#a765">Further Development</a></li><li><a href="#ad87">Conclusion</a></li></ol><h3>Introduction</h3><p>ChatGPT, the renowned large language model (LLM), underwent training on a <a href="https://www.forbes.com/sites/bernardmarr/2023/03/03/the-top-10-limitations-of-chatgpt/?sh=268496ae8f35">substantial 570GB dataset sourced from publicly accessible internet</a> text, Wikipedia articles among others.</p><p>Despite this extensive training, it’s important to note that ChatGPT’s knowledge remains limited due to the inherent constraints of its training data. Even the freemium version of <a href="https://help.openai.com/en/articles/6783457-what-is-chatgpt">ChatGPT is bound by knowledge up to 2021</a>.</p><p>Additionally, the GPT-4 iteration, while advanced, still faces certain knowledge limitations, primarily because the internet continually witnesses the emergence of new websites, and some domain-specific knowledge and proprietary data cannot be accessed on the internet.</p><p>In this article, we address this significant limitation and propose solutions. Specifically, we will:</p><ul><li>Introduce the LangChain framework and its capabilities.</li><li>Explore methods for augmenting ChatGPT’s knowledge with external sources.</li><li>Learn about the concept of vector stores and their relevance.</li><li>Guide you through the process of constructing a web-based chatbot using LangChain and the Streamlit library.</li></ul><p>Let’s get to work!</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/500/1*AmI9wRbXrfIWGESx6eEiTw.gif" /><figcaption>Excited GIF from <a href="https://gifer.com/en/2DV">Gipher</a></figcaption></figure><h3>Prerequisites</h3><ul><li>Basic knowledge of Python Programming and LLMs.</li><li><a href="https://openai.com/blog/openai-api">OpenAI API key</a></li><li><a href="https://github.com/">GitHub Account</a></li></ul><h3>Integrating External Knowledge with Large Language Models (LLMs)</h3><p>Large Language Models (LLMs) are generally trained on extensive datasets, but they may encounter limitations when responding to certain queries. In such cases, a common response you may receive is, <em>“I apologize, but I don’t have enough information to answer your question accurately. If you have any other questions or need assistance with a different topic, please feel free to ask, and I’ll do my best to help.”</em></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*kS0y9xaW6WhuBbc9" /><figcaption>Photo by <a href="https://unsplash.com/@siva_photography?utm_source=medium&amp;utm_medium=referral">Levart_Photographer</a> on <a href="https://unsplash.com/?utm_source=medium&amp;utm_medium=referral">Unsplash</a></figcaption></figure><p>To overcome the knowledge limitations, LLMs can tap into external knowledge or domain-specific information using Retrieval Augmented Generation (RAG). <a href="https://research.ibm.com/blog/retrieval-augmented-generation-RAG">RAG is an AI framework</a> that enhances LLMs by facilitating the seamless integration of external knowledge into their responses. This framework allows you to store information for easy retrieval, ensuring that it can be readily accessed by your preferred LLMs.</p><p>Numerous frameworks and applications, including LangChain, enable the implementation of RAG with LLMs. In the following sections, we will delve deeper into how LangChain can be harnessed to create potent applications with LLMs, such as querying a website content.</p><h3>Building A Website Query ChatBot</h3><p>In this tutorial, we’ll explore essential steps for building a website query chatbot using LangChain, OpenAI’s LLM, and Streamlit. While we’ll provide a brief explanation of key concepts for clarity.</p><h4>LangChain</h4><p>LangChain serves as a versatile framework that complements LLMs. It seamlessly integrates with various tools and LLMs,<a href="https://docs.langchain.com/docs/category/use-cases"> allowing the creation of AI applications, including document question-answering applications and chatbots.</a> LangChain proves effective in working with vector stores, embeddings, and more, offering a wide array of use cases, from summarization to API interaction and data extraction.</p><h4>OpenAI LLM</h4><p>OpenAI LLM encompasses a range of model variants, such as Embeddings, which translate text into machine-readable numerical representations. <a href="https://platform.openai.com/docs/models/overview">Notable variants like GPT-3.5 Turbo, GPT-3.5 Turbo-Instruct, and GPT-4 </a>offer diverse token formats and capabilities. In this tutorial, we’ll utilize both embeddings and models to convert queries (questions or prompts) into numerical form and extract valuable responses from websites.</p><h4>Vector Stores</h4><p>Vector stores act as databases for housing external data, breaking it into manageable chunks and embedding it into numerical formats for effortless retrieval. As previously discussed in this article, Retrieval Augmented Generation (RAG) benefits from this approach. Various vector stores, including <a href="https://faiss.ai/index.html">FAISS</a>, <a href="https://www.pinecone.io/">Pinecone</a>, <a href="https://www.trychroma.com/">ChromaDB</a>, and others, are at your disposal for this purpose.</p><blockquote>Let’s build a website query chatbot. Right away!</blockquote><h3>Step-by-Step Guide</h3><h4>1. Clone the repository</h4><p>Clone my repository with the git command into your local machine</p><pre>git clone https://github.com/Josiah-Adesola/Web-Chatter.git</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Jb9wtNUJu3b2M_rSgXIvsg.png" /><figcaption>Github Repo Image from Author</figcaption></figure><ul><li><strong>db: </strong>hosts the vector store or vector database, preserving the numerical representations of text scraped from websites.</li><li><strong>web-chat: </strong>serves as the virtual environment for this project. After forking the project, you can utilize <a href="https://github.com/features/codespaces">GitHub Codespaces</a> to work directly within it.</li><li><strong>README.md:</strong> is a markdown file providing essential information about the repository.</li><li><strong>app.py: </strong>is a Python script encompassing code for scraping, web-text loading, text splitting, embedding, and providing completion, along with the web interface powered by Streamlit.</li><li><strong>requirements.txt: </strong>compiles all the necessary Python libraries for the successful execution of the project.</li></ul><h4>2. Navigate to the project directory</h4><p>Note — This command should be used if you’re using a code editor, but if you’re using codespaces, you’re automatically in the directory of your folder.</p><pre>cd web-chatter</pre><h4>3. Install the required Python Packages</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/931/1*ZQVNQGNI_QZLVh8QKmFsvQ.png" /><figcaption>Requirements.txt Image from Author</figcaption></figure><pre>pip install -r requirements.txt</pre><h4>4. Run the Streamlit App</h4><pre>streamlit run app.py</pre><p>Let’s explore the code within app.py, where you’ll find the essential programs that drive the objectives of this article.</p><p>a. First, we import the necessary libraries for the script, each serving a specific purpose:</p><ul><li><a href="https://docs.langchain.com/docs/components/chains/"><strong>Langchain.chains</strong></a>: Within the LangChain framework, various chains handle distinct operations. The module crucial for this project is RetrievalQA, dedicated to executing the Retrieval Augmented Generation (RAG) operation, previously discussed in this article.</li><li><a href="https://docs.langchain.com/docs/components/models/"><strong>Langchain.chains.chat_models</strong></a>: This import statement brings in the ChatOpenAI model, a component of this module. You’ll find other models available when you refer to the documentation.</li><li><strong>Langchain.document_loaders</strong>: This module imports the WebBaseLoader component, which caters to multiple data sources, not limited to websites but also encompassing text files, PDFs, CSV files, and more.</li></ul><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/f57a6f6fd74477abb66eb31d72d4cb76/href">https://medium.com/media/f57a6f6fd74477abb66eb31d72d4cb76/href</a></iframe><ul><li><a href="https://docs.langchain.com/docs/components/models/text-embedding-model"><strong>Langchain.embeddings</strong>:</a> We import OpenAIEmbeddings,recognized for its effectiveness in converting segmented text chunks into numerical representations, as previously explained.</li><li><a href="https://python.langchain.com/docs/modules/model_io/models/chat/prompts"><strong>Langchain.prompts.chats</strong></a><strong>:</strong> This module aids in the precise formatting of prompts, contributing to the attainment of optimal completions and results for the project.</li><li><a href="https://docs.langchain.com/docs/components/indexing/text-splitters"><strong>LangChain.text_splitter</strong></a><strong>:</strong> While this module offers several text splitter components, our project employs CharacterTextSplitter. Alternatively, you can consider Sentence text splitter, but our experience suggests the former as one of the most effective methods.</li><li><a href="https://python.langchain.com/docs/modules/data_connection/vectorstores/"><strong>LangChain.vectorstores</strong>:</a> Besides Chroma, representing ChromaDb, one of the vector stores mentioned in this article, LangChain provides access to various vector stores.</li></ul><p>b. Before proceeding, make sure to replace OPENAI_API_KEY with your <a href="https://openai.com/blog/openai-api">OpenAI API key</a>, which can be obtained through a free account or purchased if your free $5 credits have expired.</p><p>c. With this setup complete, we proceed to create the prompt template.</p><p>d. Next, let’s develop a function for building the Streamlit web application. We’ll provide further clarification on the script below:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/d6e0d11f92fe700a2e38d0f115c5bc30/href">https://medium.com/media/d6e0d11f92fe700a2e38d0f115c5bc30/href</a></iframe><ul><li><strong>st.title</strong>: In this context, st is the abbreviated representation of the Streamlit library. st.title serves as the primary heading of the website, as depicted in the image below. This element is defined within the main function, as indicated in the script.</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/956/1*AIdtLiimTUZxst6ND_vTew.png" /><figcaption>Image of Streamlit app from Author</figcaption></figure><ul><li><strong>st.subheader</strong>: This function is employed to insert a subheading text beneath the web app title. You have the flexibility to customize the text to your specific text.</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/931/1*kUPMpZARHhHNTg9TBoluWw.png" /><figcaption>Image of Streamlit app from Author</figcaption></figure><ul><li><strong>url &amp; prompt</strong>: The url and prompt are variables designed to hold the outcomes of st.text_input. This function automatically generates an input box, allowing users and visitors to input text. The url variable captures the URL of the website you wish to query, while the prompt represents the query or questions intended for the website, aligning with the core purpose of this article.</li></ul><p>e. Next, create a button to execute all the operations based on the URL and prompt text input:</p><ul><li><strong>st.button</strong>: This component is generated using Streamlit and results in a button element. The label Submit Query represents the text you wish to display on the button, and you have the flexibility to tailor it to your preference. Notably, ABS_PATH and DB_DIR are used to establish the path and directory for embedding storage within the vector database.</li><li><strong>Loading the Data</strong>: This serves as the initial and critical step in querying websites with LangChain. The WebBaseLoader module performs web crawling and data scraping, subsequently populating the data variable through the load functionality within the LangChain framework.</li><li><strong>Data Splitting:</strong> The content extracted from the website is then divided into text segments, each with a chunk_size of 2500 and a chunk_overlap of 40. Incorporating overlap aids in maintaining text consistency.</li></ul><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/0625c448c6451b090a1b839a0983e0de/href">https://medium.com/media/0625c448c6451b090a1b839a0983e0de/href</a></iframe><ul><li><strong>Creating Embeddings</strong>: Embeddings serve the crucial role of translating text chunks into numerical representations, typically in vector format — this is the form that computers comprehend. For this purpose, we’ve opted for OpenAIEmbeddings, a top-tier choice within LangChain’s documentation.</li><li><strong>Establishing a Vector Database</strong>: We embark on the creation of a vector database or vector store. In our case, we employ ChromaDB through LangChain’s Chroma module to craft a database that accommodates the embeddings. This serves our database with a repository of information from our website. Additionally, we configure a retriever to facilitate convenient access to documents or text chunks within the database.</li><li><strong>Leveraging an LLM Model</strong>: Given our objective of building a chatbot, the most suitable model for this task is the ChatOpenAI model. While alternate models can be considered, ChatOpenAI stands as a robust choice for numerous chatbot applications. We employ the llm variable to define this model.</li><li><strong>Creating a Retrieval Chain</strong>: The RetrievalQA module plays a pivotal role, incorporating the chain component as a cornerstone of LangChain. RetrievalQA is configured with llm, a chain_type specified as stuff, and the retriever, which evaluates document similarities concerning any given query. To illustrate, if we pose a query like “When was WHO founded?” and apply it to the WHO website, the system retrieves three text chunks from the WHO website (in this instance, K = 3).</li><li><strong>Executing the Prompt:</strong> Following the completion of these preparatory steps, we retrieve our response using st.success (although st.write is a viable alternative). st.success creates a green background, enhancing the user experience. Finally, we invoke the main function to execute the process.</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/933/1*OLu6xxRAuCeV1-eNEgpiXQ.png" /><figcaption>Streamlit webapp (WebChatter) from Author</figcaption></figure><p>After running this script you should have a ready response from your website.</p><h3>Further Developments</h3><p>For companies and developers, there’s an opportunity to elevate this solution to the next level by seamlessly integrating it into your website or application.</p><p>This enhancement provides users with a convenient means to query information related to your software. Beyond website links, this approach can be extended to handle various document types, expanding its versatility.</p><h3>Conclusion</h3><p>In conclusion, Large Language Models (LLMs) possess remarkable capabilities and intelligence. However, their potential can be further harnessed by granting them access to proprietary data and external bodies of knowledge, enabling users to engage in insightful queries and enhancing their utility. This article shows one of the potential approach to achieving this objective.</p><p><a href="https://github.com/Josiah-Adesola/Web-Chatter">GitHub - Josiah-Adesola/Web-Chatter: This is a chatbot that can help you question anything on your website. It&#39;s built with OpenAI and Langchain.</a></p><p>I hope you found this article really insightful.</p><h3>In Plain English 🚀</h3><p><em>Thank you for being a part of the </em><a href="https://plainenglish.io/"><strong><em>In Plain English</em></strong></a><em> community! Before you go:</em></p><ul><li>Be sure to <strong>clap</strong> and <strong>follow</strong> the writer ️👏<strong>️️</strong></li><li>Follow us: <a href="https://twitter.com/inPlainEngHQ"><strong>X</strong></a><strong> | </strong><a href="https://www.linkedin.com/company/inplainenglish/"><strong>LinkedIn</strong></a><strong> | </strong><a href="https://www.youtube.com/channel/UCtipWUghju290NWcn8jhyAw"><strong>YouTube</strong></a><strong> | </strong><a href="https://discord.gg/in-plain-english-709094664682340443"><strong>Discord</strong></a><strong> | </strong><a href="https://newsletter.plainenglish.io/"><strong>Newsletter</strong></a></li><li>Visit our other platforms: <a href="https://stackademic.com/"><strong>Stackademic</strong></a><strong> | </strong><a href="https://cofeed.app/"><strong>CoFeed</strong></a><strong> | </strong><a href="https://venturemagazine.net/"><strong>Venture</strong></a></li><li>More content at <a href="https://plainenglish.io/"><strong>PlainEnglish.io</strong></a></li></ul><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=4c9c79a3e33f" width="1" height="1" alt=""><hr><p><a href="https://ai.plainenglish.io/how-to-build-a-website-query-chatbot-a-step-by-step-guide-4c9c79a3e33f">How to Build A Website Query ChatBot: A Step-by-Step Guide</a> was originally published in <a href="https://ai.plainenglish.io/">Artificial Intelligence in Plain English</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Universal NER: Named Entity Recognition Across Diverse Domains]]></title>
            <link>https://ai.plainenglish.io/universal-ner-named-entity-recognition-across-diverse-domains-1b881e0b3d99?source=rss-8d681a6b68c7------2</link>
            <guid isPermaLink="false">https://medium.com/p/1b881e0b3d99</guid>
            <category><![CDATA[nlp]]></category>
            <category><![CDATA[ner]]></category>
            <category><![CDATA[chatgpt]]></category>
            <category><![CDATA[universalner]]></category>
            <dc:creator><![CDATA[Josiah Adesola]]></dc:creator>
            <pubDate>Fri, 11 Aug 2023 18:38:57 GMT</pubDate>
            <atom:updated>2023-10-28T13:20:35.309Z</atom:updated>
            <content:encoded><![CDATA[<h4>The Universal NER (UniNER) is a smaller model that performs better than ChatGPT in Named Entity Recognition tasks.</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*gtA3mbZ6rkTksEhl" /><figcaption>Photo by <a href="https://unsplash.com/@ethanjohnson18?utm_source=medium&amp;utm_medium=referral">Ethan Johnson</a> on <a href="https://unsplash.com/?utm_source=medium&amp;utm_medium=referral">Unsplash</a></figcaption></figure><h3>Table of Contents</h3><ul><li><a href="#e045">Introduction</a></li><li><a href="#1a6a">Background and Motivation</a></li><li><a href="#0096">Understanding UniNER</a></li><li><a href="#db26">How UniNER works</a></li><li><a href="#8fb7">Comparison with Existing Approaches</a></li><li><a href="#9321">Demo</a></li><li><a href="#9321">Conclusion</a></li><li><a href="#4465">References</a></li></ul><h3>Introduction</h3><p>Large Language Models (LLMs) are everywhere these days. I really designated 2023 as “<strong><em>The Year of Generative AI.”</em></strong> LLMs are deep learning models that are pre-trained via unsupervised learning on extremely large datasets, which can number in the tens of billions using the transformer architecture and are fine-tuned with self-supervised learning <a href="#7865"><em>[1]</em></a>. They are useful for text generation, information extraction, machine or language translation, and a variety of other activities, and are currently utilized in many large models, such as the <a href="https://chat.openai.com/">ChatGPT</a>, which is created using GPT-3.5 turbo and GPT-4.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*UB6-dc1Gr0pBL7xf" /><figcaption>Photo by <a href="https://unsplash.com/@ilgmyzin?utm_source=medium&amp;utm_medium=referral">ilgmyzin</a> on <a href="https://unsplash.com/?utm_source=medium&amp;utm_medium=referral">Unsplash</a></figcaption></figure><p>This has resulted in LLMs having a high operational cost, being complex, and requiring a large sum of money to pre-train an LLM model due to Graphic Processing Unit (GPU) costs, as well as some having the inability to deliver exact outcomes for tasks—explainability. These challenges give rise to what are known as student models (smaller models), which learn from the predictions of teacher models (larger models) <a href="#0b44">[2]</a> enabling them to reduce the operational cost of training from scratch and occasionally beat the larger model.</p><p>The UniversalNER (UniNER) is a smaller model that learns from ChatGPT’s prediction to generate named entities. It is used on smaller models for specific tasks called targeted distillation using a conversational style of instruction-based tuning for generating named entities on very diverse domains like biomedicine, law, programming, social media, and so on. More on this will be provided in subsequent parts. UniNER’s main accomplishment was outperforming ChatGPT’s Named Entity Recognition (NER) capability by an average of 7–9 absolute F1 points.</p><h3>Background and Motivation</h3><p>A major question arose: How is it possible for a small model to perform better than a larger model? The question of<em> “What if we streamline to just one task and make it perform well”</em> rather than dispersing the knowledge over various NLP tasks</p><p>UniNER responds to the questions raised above. After learning from a larger model, a smaller model with minimal parameters was able to operate on a single NLP task called NER for information extraction. UniNER not only saves computational costs but also demonstrates how target distillation for mission-focused instruction tuning through a conversation style can shine in a single large application.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/538/1*4YCjwbeugRRV2xFlLihDIg.png" /><figcaption>Image from <a href="https://arxiv.org/pdf/2308.03279.pdf">UniNER paper</a></figcaption></figure><p>This broad application was in NER, which is one of the common Natural Language Processing (NLP) jobs capable of extracting information from a document given specific entities. So, instead of constructing a NER for a single or few domains, it was applied to nine (9) different domains and performed well even on zero-shot and few-shot tasks (tasks that it is unaware of or has had limited contact with).</p><h3>Understanding UniNER</h3><p>UniNER is an acronym that stands for Universal Named Entity Recognition. The term <strong><em>“universal”</em></strong> implies that it can conduct named entity recognition for a variety of disciplines, most notably law, finance, biology, programming, social media, and general, using 43 benchmarked NER datasets. <a href="#896e">[3]</a><br>A large number of entities are created every day in various domains, and teaching them about LLMs will take more time and money. Not to mention that, as entities or new terms from various fields expand enormously on a daily basis, LLMs are losing their understanding of them, particularly in specialized domains such as biomedicine or law.</p><p>UniNER is incredibly efficient at generating named entities from any domain by prompting the datasets with natural language-type instructions to produce the output (entities). It employs a conversational tuning method. Taking an input (text), for example, you could ask, <strong><em>“What describes the PERSON in this text?” </em></strong>Then it generates a JSON list representation of all the entities in the text. Just like asking a chatbot a question.</p><blockquote>Let’s get to more details…</blockquote><h3>How UniNER works</h3><p>UniNER operates on the basis of a concept known as “Knowledge Distillation” <a href="#9a4b">[4]</a>. This entails distilling or transferring knowledge of ChatGPT-generated labels of entities into the student model (UniNER) in various domains with the goal of exceeding the ability of the LLM (ChatGPT).</p><p>However, this differs from other works that use student models on a wide range of NLP tasks such as summarization, translation, NER, and others. UniNER is just concerned with one task: recognizing and extracting entities from a given text.</p><p>Instead of performing several tasks, the pre-trained model is tuned for a single broad application, such as open information extraction. One thing to keep in mind is that, whereas many other studies concentrated on increasing the instructions, the UniNER team focused on expanding the input (different domains). After utilizing ChatGPT (gpt-3.5-tubo-0301) <a href="#896e">[3]</a> to produce entities based on different passages, misformatted output was deleted, yielding 45,889 input-output pairs and 13,020 entity types. Without oversight, ChatGPT was urged to construct entity types in this format. Look at the image below.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/557/1*l0jbGSycFtxZrRkXeg32Iw.png" /><figcaption>Image from <a href="https://arxiv.org/pdf/2308.03279.pdf">UniNER paper</a></figcaption></figure><p>The instruction tuning was then applied to smaller models in order to distill the output from the bigger model (which in NER has different entity kinds). This resulted in the retrieval of your entities via NLP queries, such as what is described as entity type (t) from text or passage (X), to provide an output or JSON list of entities denoted as (y).</p><p>In the data construction process, <strong><em>negative sampling</em></strong> was also used, which is a strategy that includes introducing data that is not associated with the goal of the model. The goal of negative sampling is to avoid overfitting and improve the model’s performance. For example, if you include a passage that incorporates entertainment in a collection of data for health and biomedicine, that is a scenario of negative sampling. When you demonstrate the study and UniNER does not recognize an entity, it provides an empty JSON list, similar to this [].</p><h3>Comparison with Existing Approaches</h3><p>The UniNER model was compared to other distillation-method models such as Vicuna-7B and Vicuna-13B (7B and 13B denote 7 billion and 13 billion parameters, respectively).</p><p>After fine-tuning with ChatGPT discussions, Vicuna uses <a href="https://ai.meta.com/llama/">Meta AI’s LLama</a> as the basis model <a href="#3102">[5]</a>.InstructUIE is based on the Flan-T5 11B model, which was fine-tuned on numerous information extraction datasets using a supervised model. All of these models were compared against UniNER, including the ChatGPT (teacher model), and UniNER outperformed all of them. The figure below from the UniNER paper does justice to the above explanation.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/537/1*kO6Zxxp2RWdJbt5A8xzcOw.png" /><figcaption>Image from <a href="https://arxiv.org/pdf/2308.03279.pdf">UniNER paper</a></figcaption></figure><h3>Demo</h3><p>Check out the <a href="https://universal-ner.github.io/">demo</a> here and play with the model.</p><p>I will, however, show you a gif of how to use the model. The link below takes you to the research’s GitHub repo, the article, and any future work and enhancements for the research.</p><p><a href="https://universal-ner.github.io/">UniversalNER</a></p><h3>Conclusion</h3><p>Finally, UniNER uses mission-focused instruction tuning from a big language model like ChatGPT for a specific NLP task called Named Entity Recognition in nine (9) different areas.<br>In terms of average F1, UniNER-7B and UniNER-13B outperformed ChatGPT. They reached 41.7% and 43.4%, respectively, compared to ChatGPT’s 34.9%. This demonstrated the power of targeted distillation in a large application class such as open information extraction, not to mention the relatively small model size.</p><p><em>More content at </em><a href="https://plainenglish.io/"><em>PlainEnglish.io</em></a><em>. Sign up for our </em><a href="http://newsletter.plainenglish.io/"><em>free weekly newsletter</em></a><em>. Follow us on </em><a href="https://twitter.com/inPlainEngHQ"><em>Twitter</em></a>, <a href="https://www.linkedin.com/company/inplainenglish/"><em>LinkedIn</em></a><em>, </em><a href="https://www.youtube.com/channel/UCtipWUghju290NWcn8jhyAw"><em>YouTube</em></a><em>, and </em><a href="https://discord.gg/GtDtUAvyhW"><em>Discord</em></a><em>. Interested in Growth Hacking? Check out </em><a href="https://circuit.ooo/"><em>Circuit</em></a><em>.</em></p><h3>References</h3><ol><li>Kerner, S. M. (2023, April). Large language model (LLM). TechTarget. <a href="https://www.techtarget.com/whatis/definition/large-language-model-LLM">https://www.techtarget.com/whatis/definition/large-language-model-LLM</a></li><li>Knowledge Distillation: Principles, Algorithms, and Applications. Sundeep Teki (2023) Neptune.ai ML Model Development. <a href="https://neptune.ai/blog/knowledge-distillation">https://neptune.ai/blog/knowledge-distillation</a></li><li>Zhou, W., Zhang, S., Gu, Y., Chen, M., &amp; Poon, H. (2023). UniversalNER: Targeted Distillation from Large Language Models for Open Named Entity Recognition. <em>ArXiv</em>. /abs/2308.03279</li><li>Hsieh, C.-Y., Li, C.-L., Yeh, C.-K., Nakhost, H., Fujii, Y., Ratner, A., Krishna, R., Lee, C.-Y., &amp; Pfister, T. (2023). Distilling Step-by-Step! Outperforming Larger Language Models with Less Training Data and Smaller Model Sizes. In Findings of the Association for Computational Linguistics: ACL 2023 (pp. 8003–8017). Toronto, Canada: Association for Computational Linguistics.</li><li>Wei-Lin Chiang, Zhuohan Li, Zi Lin, Ying Sheng, Zhanghao Wu, Hao Zhang, Lianmin Zheng, Siyuan Zhuang, Yonghao Zhuang, Joseph E. Gonzalez, Ion Stoica, and Eric P. Xing. 2023. <a href="https://lmsys.org/blog/2023-03-30-vicuna/">Vicuna: An opensource chatbot impressing gpt-4 with 90%* chatgpt quality.</a></li></ol><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=1b881e0b3d99" width="1" height="1" alt=""><hr><p><a href="https://ai.plainenglish.io/universal-ner-named-entity-recognition-across-diverse-domains-1b881e0b3d99">Universal NER: Named Entity Recognition Across Diverse Domains</a> was originally published in <a href="https://ai.plainenglish.io/">Artificial Intelligence in Plain English</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[BioGPT: The Innovative Generative Pre-trained Transformer (GPT) for Biomedical Text Understanding…]]></title>
            <link>https://ai.plainenglish.io/biogpt-the-innovative-generative-pre-trained-transformer-gpt-for-biomedical-text-understanding-af9243d63fb0?source=rss-8d681a6b68c7------2</link>
            <guid isPermaLink="false">https://medium.com/p/af9243d63fb0</guid>
            <category><![CDATA[ai]]></category>
            <category><![CDATA[biogpt]]></category>
            <category><![CDATA[nlp]]></category>
            <category><![CDATA[technology]]></category>
            <category><![CDATA[gpt]]></category>
            <dc:creator><![CDATA[Josiah Adesola]]></dc:creator>
            <pubDate>Tue, 08 Aug 2023 12:10:57 GMT</pubDate>
            <atom:updated>2023-08-26T10:52:35.654Z</atom:updated>
            <content:encoded><![CDATA[<h3>BioGPT: The Innovative Generative Pre-trained Transformer (GPT) for Biomedical Text Understanding and Generation</h3><h4>A clear explanation of BioGPT and a practical use case of text generation in biomedical tasks</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*EjK7fbcVAnMtEka1" /><figcaption>Photo by <a href="https://unsplash.com/@mycellhub?utm_source=medium&amp;utm_medium=referral">Toon Lambrechts</a> on <a href="https://unsplash.com/?utm_source=medium&amp;utm_medium=referral">Unsplash</a></figcaption></figure><h3>Introduction</h3><p>The world of Large Language Models (LLMs) has lately expanded, from the Google AI team’s <a href="https://huggingface.co/blog/bert-101">BERT</a> to more complex models like OpenAI’s <a href="https://openai.com/gpt-4">GPT-4</a>. LLM has been shown to be effective at a variety of NLP tasks, including text summarization, question answering, text generation, document classification, translation, and many more.</p><p>However, this has frequently been in the general domain. For example, the OpenAI team’s <a href="https://chat.openai.com/">ChatGPT</a> can be used to generate poems, code, and scripts, answer queries, create chatbots, and even translate languages. The problem is that GPT-3 and GPT-4 are trained with a broad scope. This means they have broad knowledge gleaned from a variety of books, articles, websites, and code repositories available on the internet. This has the minor drawback of not being precise in some delicate activities.</p><blockquote>Let’s get more practical!</blockquote><p>This shows a practical comparison between the <a href="https://huggingface.co/gpt2">GPT-2 </a>model hosted and prototyped on the Hugging Face platform and a BioGPT test run on my Google Colab.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*AAZ8CGKYpr--gGSC_j173w.gif" /><figcaption>A comparison between GPT-2 and BioGPT by the <a href="https://josiahadesola.medium.com/">Author</a></figcaption></figure><p>It is very evident that GPT-2 misinterpreted the context, not because it is dumb but because it is not domain-specific. The <a href="https://colab.research.google.com/drive/1vErOTbE5aIFy5hbJNDHZhC0fYJm-Nr5h?usp=sharing">Google Colab Notebook</a> prediction for CP-673451 was highly accurate, but the GPT-2 model on Hugging Face misunderstood the context and misinterpreted the word as a number.<br>BioGPT can be used for a variety of domain-specific activities such as drug-to-target extraction, medical document classification, medical text or science study summarization, and questions and answers on biomedical literature.</p><h3>What is BioGPT?</h3><p><a href="https://github.com/microsoft/BioGPT">BioGPT</a> is an acronym that stands for Biomedical Generative Pretrained Transformers. BioGPT is a <a href="https://www.microsoft.com/en-us/research/publication/biogpt-generative-pre-trained-transformer-for-biomedical-text-generation-and-mining/">Microsoft Research-backed</a> GPT model that has been pre-trained on a variety of biomedical topics, specifically 15 million PubMed abstracts.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*WjKxop2tVxzp3wxu" /><figcaption>Photo by <a href="https://unsplash.com/@turagphotos?utm_source=medium&amp;utm_medium=referral">Turag Photography</a> on <a href="https://unsplash.com/?utm_source=medium&amp;utm_medium=referral">Unsplash</a></figcaption></figure><p>The aim is to perform six biomedical NLP tasks that include: Question-answering on <a href="https://pubmed.ncbi.nlm.nih.gov/">PubMedQA</a>, relation extraction, document classification, and text generation, on benchmark datasets in the Biomedical space. These tasks are grouped into text generation and mining, which can also be called text understanding.</p><p>The datasets used for the BioGPT were gotten from PubMed’s website and cleaned to get only papers with both abstracts and titles. It was trained on GPT-2, which serves as the pillar of BioGPT. The team couldn’t use GPT-3 because it has a lot of parameters, up to 15 Billion.</p><p>It achieved 78.3% accuracy on PubMedQA, which is a great feat in the biomedical research space.</p><h3>Pretraining models</h3><p>Disclaimer: <em>This is not a diversion from the material, but rather an important notion in LLMs.</em><br>Pre-training models, in simple terms, involve training your model on a wide range of datasets that are similar or dissimilar to the specific data for the task from which you want the model to learn. What we call “<em>fine-tuning</em>” comes after pre-training.</p><p>A good example is that, as a college student, you may take some optional courses purely for general information, and you may not be quizzed on them—this is <strong>pre-training</strong>. While fine-tuning occurs after you have gained some broad knowledge and possibly how to study better, you then attend a class tailored to your course of study — this is known as <strong>fine-tuning</strong>.</p><p>It is best to train models from scratch in LLMs before using them for downstream tasks. This is essential for your LLM so that it can acquire knowledge properly and answer questions or execute jobs that are too specific to a niche.</p><p>BioGPT, for example, was pre-trained on 15 million PubMed data items before being fine-tuned by altering the labels in the target sequence to improve output. A nice example is question-answering tasks. “<em>The answer to the question is …</em>” is the desired sequence. BioGPT was also improved in terms of document classification, end-to-end extraction, and text production. The model was trained on eight NVIDIA V100 GPUs with 64 accumulated steps before being fine-tuned on a single NVIDIA V100 GPU with 32 collected steps.</p><h3>Before BioGPT? The technology and some underlying concepts</h3><p>You must have had a thought while reading this post. What was there before BioGPT? Is it the first, or the second? It most obviously is not.</p><blockquote>Let’s dive in quickly!</blockquote><p><a href="https://www.microsoft.com/en-us/research/publication/biogpt-generative-pre-trained-transformer-for-biomedical-text-generation-and-mining/">Microsoft Research’s paper</a> on BioGPT proposes to address text generation restrictions in biomedical NLP problems. Let’s take a quick look at some historical context. Prior to BioGPT, we had a large number of models trained on benchmark datasets for performing biomedical NLP problems. BioBERT, PubMedBERT, and ElECTRAMed are examples of these models.</p><h4>BioBERT and PubMedBERT</h4><p><a href="https://arxiv.org/abs/1810.04805">BERT</a> is an acronym for Bidirectional Encoder Representation for Transformers, which was developed by Google researchers in 2018. Understanding the words before and following a missing word in a phrase is required. It is an upgraded model of the transformer architecture that works with sequential data and can go two-way (back and forth) like a person pacing back and forth.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*CqoZyVCn22FM672Z" /><figcaption>Photo by <a href="https://unsplash.com/@soymeraki?utm_source=medium&amp;utm_medium=referral">Javier Allegue Barros</a> on <a href="https://unsplash.com/?utm_source=medium&amp;utm_medium=referral">Unsplash</a></figcaption></figure><p>BioBERT and PubMedBERT use the BERT model to pre-train large-text biomedical papers and journals; PubMedBERT, on the other hand, is limited to pre-training on PubMed abstracts alone, with the goal of providing an understanding of abstracts or summaries of biological tasks. This performed admirably in sentiment analysis, text summarizing, and a variety of other tasks.</p><p>There was a significant constraint, which was text generation. This is where BioGPT enters the picture. It can complete the aforementioned tasks, outperform BioBERT and PubMedBERT, and generate language. You’re probably wondering why.</p><p>Because BERT works using an underlying principle known as Masked Language Modeling (MLM), the model conducts a “fill in the gap” task in a sentence. For example, “<strong>The cat ____ in the bag</strong>”, the model predicts “<strong>is</strong>”. The model’s inability to generate new words is hampered by its repeated checks for a contextual association between before and afterwards.<br>While BioGPT works using the Causal Language Model (CLM), the term “cause and effect” is drawn from a common expression. The model guesses the next word based on previous words but does not take into account future words. It learns the meaning and flow of different words in a sentence. It only goes one way, as opposed to BERT, which moves in two directions. For example <strong>“Malaria is _____”,</strong> the model predicts <strong>“a mosquito-borne disease.”</strong></p><blockquote>Who wants to code! Let’s get to it</blockquote><h3>Implementation of BioGPT with Hugging Face</h3><p>The BioGPT model was stored on a hugging face and configured for easy use by anyone. I’ll walk you through the code.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/929d02fb4f51357266fadea247b0dbf7/href">https://medium.com/media/929d02fb4f51357266fadea247b0dbf7/href</a></iframe><h3>Code Walkthrough</h3><ol><li><strong>Import the necessary libraries</strong>— The transformer module was first imported with !pip . Then import the pipeline function for text generation, and set_seed to put randomness in check.</li><li><strong>Load the BioGPT model</strong> — The BioGPT language tokenizer and model are imported from the transformers and initialize the class BioGptTokenizer and BioGptForCausalLM from the microsoft/biogpt</li><li><strong>Create the text generation pipeline</strong> — This is created with the pipeline function. Then set_seed(42) is used to maintain reproducibility.</li><li><strong>Text-based Input</strong> — The generator is used to generate text-based input. The parameters include input_text a variable, which you can edit. The max_length equals 200 tokens.</li></ol><p>In summary, the code creates a text-generation pipeline and can produce several things with any given input.</p><p>Here’s my Google Colab Notebook for the demo.</p><p><a href="https://colab.research.google.com/drive/1vErOTbE5aIFy5hbJNDHZhC0fYJm-Nr5h#scrollTo=Xcli5P_oHYOF">Google Colaboratory</a></p><p>Check out the BioGPT explanation and use case on HuggingFace</p><p><a href="https://huggingface.co/docs/transformers/model_doc/biogpt">BioGPT</a></p><h3>Demo</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*JsVfVhhdVHwv__-04U__VQ.gif" /><figcaption>Gif demo of BioGPT text generation ability from the Author’s Colab</figcaption></figure><h3>Conclusion</h3><p>In conclusion, BioGPT is a cutting-edge achievement in NLP for biomedical tasks that has demonstrated improved accuracy on the GPT-2 XL architecture with 1.5B model parameters.<br>BioGPT had an F1 score of 44.98%, 38.42%, and 40.76% on the BC5CDR, KD-DTI, and DDI end-to-end extraction tasks, respectively, and 78.2% accuracy on PubMedQA.When compared to GPT-2 medium, BioBERT, PubMedBERT, and BioLinkBERT base, it demonstrated itself to be a better model.<br>Kudos to the Microsoft Research Team!</p><p><em>More content at </em><a href="https://plainenglish.io/"><em>PlainEnglish.io</em></a><em>. Sign up for our </em><a href="http://newsletter.plainenglish.io/"><em>free weekly newsletter</em></a><em>. Follow us on </em><a href="https://twitter.com/inPlainEngHQ"><em>Twitter</em></a>, <a href="https://www.linkedin.com/company/inplainenglish/"><em>LinkedIn</em></a><em>, </em><a href="https://www.youtube.com/channel/UCtipWUghju290NWcn8jhyAw"><em>YouTube</em></a><em>, and </em><a href="https://discord.gg/GtDtUAvyhW"><em>Discord</em></a><em>. Interested in Growth Hacking? Check out </em><a href="https://circuit.ooo/"><em>Circuit</em></a><em>.</em></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=af9243d63fb0" width="1" height="1" alt=""><hr><p><a href="https://ai.plainenglish.io/biogpt-the-innovative-generative-pre-trained-transformer-gpt-for-biomedical-text-understanding-af9243d63fb0">BioGPT: The Innovative Generative Pre-trained Transformer (GPT) for Biomedical Text Understanding…</a> was originally published in <a href="https://ai.plainenglish.io/">Artificial Intelligence in Plain English</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Automate YouTube Downloads with Python: A Step-by-Step Guide to Streamlining Your Video Downloads]]></title>
            <link>https://python.plainenglish.io/automate-youtube-downloads-with-python-a-step-by-step-guide-to-streamlining-your-video-downloads-98faddc5cc7a?source=rss-8d681a6b68c7------2</link>
            <guid isPermaLink="false">https://medium.com/p/98faddc5cc7a</guid>
            <category><![CDATA[python]]></category>
            <category><![CDATA[automation]]></category>
            <category><![CDATA[productivity]]></category>
            <dc:creator><![CDATA[Josiah Adesola]]></dc:creator>
            <pubDate>Fri, 14 Apr 2023 09:14:16 GMT</pubDate>
            <atom:updated>2023-08-28T02:54:02.132Z</atom:updated>
            <content:encoded><![CDATA[<p>Learn how to use Python to automate your YouTube downloads to your local device and save time with this step-by-step guide</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*T8-KiZqlPXdeT6EVkoViNw.jpeg" /><figcaption><a href="https://www.pexels.com/photo/hand-holding-smartphone-with-internet-access-to-youtube-3227986/">Photo</a> by Szabó Viktor</figcaption></figure><h3>Introduction</h3><p>In today’s digital world, online streaming has become a popular way of accessing video content online. YouTube is a central online streaming platform owned by Google. it has a vast collection of videos ranging from sports and entertainment to technology and is one of the most visited websites on the internet.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*u2jOI5dxh35ToFZM" /><figcaption>Photo by <a href="https://unsplash.com/@rswebsols?utm_source=medium&amp;utm_medium=referral">Souvik Banerjee</a> on <a href="https://unsplash.com/?utm_source=medium&amp;utm_medium=referral">Unsplash</a></figcaption></figure><p>However, streaming video online can be time-consuming, and inconvenient, especially for those with slow internet and limited data plans. In this article, we’ll explore how to download YouTube videos to your local device. With a few lines of Python code, you can save time and make downloading videos more efficient. Let’s dive right in.</p><h3>Step-by-Step Guide</h3><ol><li><strong>Setting up your environment</strong>: Before you start coding on any code editor of your choice, ensure you have installed all the necessary tools and packages needed for the code to work. Here are the steps to get started:</li></ol><ul><li><strong>Install Python</strong>: If you haven’t already installed Python on your computer, download the latest version from the official website (<a href="https://www.python.org/downloads/">https://www.python.org/downloads/</a>). Follow the installation instructions to install Python.</li><li><strong>Install the required libraries</strong>: To download YouTube videos to your local PC using Python code, you need pytube Python library. Run this command on your terminal.</li></ul><pre>pip install pytube</pre><p>Next, you will need to install the progressbar library, this library can also be installed using pip. The library shows the progress, and the estimated time of arrival (ETA), while the video is downloading. Run this command in the command prompt or your terminal</p><pre>pip install progressbar</pre><p>2. Downloading a YouTube video: This is the main part of this article, I’ll explain the codes line by line.</p><ul><li>Import the required Python libraries.</li></ul><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/1071ea2be51b1dcbbd625995110541bc/href">https://medium.com/media/1071ea2be51b1dcbbd625995110541bc/href</a></iframe><p>Thepytubelibrary works well with YouTube; it accesses the links and extracts the necessary information needed for the right download. Urllib.requestworks with the YouTube URL or link inputted for retrieval. Theprogressbar library displays a progress bar while downloading the video, consisting of the percentage of the download completed, a progress bar, and an ETA.</p><ul><li>Fetch YouTube video stream</li></ul><p>Thelink variable allows you to input the YouTube URL. Then, using the try and except code, check if the link is working correctly. The streams variable fetches out the video in the link and works on the resolution. You could change the resolution to 720p or 1080p</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/f8ce363d19dd34b6d9dc45595b409e43/href">https://medium.com/media/f8ce363d19dd34b6d9dc45595b409e43/href</a></iframe><ul><li>Save the video to a file path</li></ul><p>This line of code saves your video in the same folder where you have your code running. It utilizes the os library to work with files on your system. So, if your code is running in the Document folder, the video will be saved in the Document folder in your system.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/a668c5496713bc7e47e1b5f43ac17636/href">https://medium.com/media/a668c5496713bc7e47e1b5f43ac17636/href</a></iframe><ul><li>Download the YouTube video</li></ul><p>The progressbar library uses the Percentage widget to show the bar of the progress of the download, the ETA and the percentage of completion. The urlibretrieve function helps to link the URL to the file path and name and ensure the video is downloaded. pbar.finish() completes the whole download process.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/e0d6bcf4c52e597c1d9a4c939fd33676/href">https://medium.com/media/e0d6bcf4c52e597c1d9a4c939fd33676/href</a></iframe><p>Conclusion</p><p>In conclusion, automating YouTube video downloads using Python can save a lot of time and effort, especially for those who regularly download videos. The step-by-step guide provided in this topic helps beginners to understand the process of automating video downloads and customising the code to suit their specific needs. By utilizing the powerful libraries and tools available in Python, users can streamline their video downloading process and make it more efficient.</p><p>The full code to the script is right here.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/d1cf8e0813db908949a30af316652140/href">https://medium.com/media/d1cf8e0813db908949a30af316652140/href</a></iframe><p>Glad you made it to the end. See you later.</p><h3>In Plain English</h3><p><em>Thank you for being a part of our community! Before you go:</em></p><ul><li><em>Be sure to </em><strong><em>clap</em></strong><em> and </em><strong><em>follow</em></strong><em> the writer! 👏</em></li><li><em>You can find even more content at </em><a href="https://plainenglish.io/"><strong><em>PlainEnglish.io</em></strong></a><strong><em> 🚀</em></strong></li><li><em>Sign up for our </em><a href="http://newsletter.plainenglish.io/"><strong><em>free weekly newsletter</em></strong></a><em>. 🗞️</em></li><li><em>Follow us on </em><a href="https://twitter.com/inPlainEngHQ"><strong><em>Twitter</em></strong></a>, <a href="https://www.linkedin.com/company/inplainenglish/"><strong><em>LinkedIn</em></strong></a>, <a href="https://www.youtube.com/channel/UCtipWUghju290NWcn8jhyAw"><strong><em>YouTube</em></strong></a>, and <a href="https://discord.gg/GtDtUAvyhW"><strong><em>Discord</em></strong></a><strong><em>.</em></strong></li></ul><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=98faddc5cc7a" width="1" height="1" alt=""><hr><p><a href="https://python.plainenglish.io/automate-youtube-downloads-with-python-a-step-by-step-guide-to-streamlining-your-video-downloads-98faddc5cc7a">Automate YouTube Downloads with Python: A Step-by-Step Guide to Streamlining Your Video Downloads</a> was originally published in <a href="https://python.plainenglish.io/">Python in Plain English</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[A step-by-step guide to sending personalized WhatsApp messages using Google Sheets]]></title>
            <link>https://josiah-adesola.medium.com/a-step-by-step-guide-to-sending-personalized-whatsapp-messages-using-google-sheets-a577ba4958a8?source=rss-8d681a6b68c7------2</link>
            <guid isPermaLink="false">https://medium.com/p/a577ba4958a8</guid>
            <category><![CDATA[automation]]></category>
            <category><![CDATA[excel]]></category>
            <category><![CDATA[productivity]]></category>
            <category><![CDATA[whatsapp]]></category>
            <category><![CDATA[spreadsheets]]></category>
            <dc:creator><![CDATA[Josiah Adesola]]></dc:creator>
            <pubDate>Sat, 08 Apr 2023 17:12:01 GMT</pubDate>
            <atom:updated>2024-03-18T23:01:23.618Z</atom:updated>
            <content:encoded><![CDATA[<p>Are you tired of manually sending the same message to multiple contacts on WhatsApp? Do you want to save time and effort by automating the process? Look no further! In this guide, I’ll show you how to use Google Sheets to send personalized WhatsApp messages to multiple recipients with just a few clicks.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*0tDC9X6CPAaaRCGbbuK1LA.jpeg" /><figcaption>Photo by <a href="https://unsplash.com/@alexbemore?utm_source=unsplash&amp;utm_medium=referral&amp;utm_content=creditCopyText">Alexander Shatov</a> on <a href="https://unsplash.com/photos/_qsuER9xYOY?utm_source=unsplash&amp;utm_medium=referral&amp;utm_content=creditCopyText">Unsplash</a></figcaption></figure><h3>Google Sheets</h3><p><a href="https://docs.google.com/spreadsheets/u/0/">Google Sheets </a>is a web-based application spreadsheet that allows you to create, modify, and collaborate with others in real-time. It’s part of the Google Workspace suite that includes Google Docs, Google Forms, Google Sites, Google Keep, G-Mails, and the rest.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*feSA1-TU5gN5Df5gC5UfAQ.jpeg" /><figcaption>Photo by <a href="https://unsplash.com/es/@rubaitulazad?utm_source=unsplash&amp;utm_medium=referral&amp;utm_content=creditCopyText">Rubaitul Azad</a> on <a href="https://unsplash.com/photos/TisvwNLLWA4?utm_source=unsplash&amp;utm_medium=referral&amp;utm_content=creditCopyText">Unsplash</a></figcaption></figure><p>Google Sheets is an efficient tool for anything that involves organizing and calculating information. It has rows and columns for inserting information, can be used for statistical analysis, and also performs several automations, one of which we’ll be learning today.</p><h3>WhatsApp</h3><p>WhatsApp is a cross-platform service for instant messaging and voice calls owned by Meta. It enables users to send text messages, voice messages, video messages, photo messages, and share direct images and documents. It’s available on mobile apps, desktop apps, and web platforms.</p><h3>Step-by-Step Guide</h3><h4><strong>#1- Create a Google spreadsheet.</strong></h4><ul><li>Log in to your Google Account.</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*uK4jxs16Kxon3WLox0CF_g.png" /></figure><ul><li>Go to the <a href="http://sheets.google.com/">Google Sheets </a>homepage (sheets.google.com).</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*OcgW4aujFt-nw5XJqIypFQ.png" /><figcaption>I rubbed off the names of the document from my personal Google sheet.</figcaption></figure><ul><li>Click on the Blank button or the +button to create a new blank spreadsheet.</li><li>Enter data containing names, Phone Number , Message , WhatsApp Link .</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*jE6M7Ml0K87RcZ0vgF8FLA.png" /><figcaption>my google spreadsheet</figcaption></figure><h4>#2- Build a personalized message.</h4><ul><li>Select a cell in the Message column.</li><li>Write a personalized message.</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*QSzWgFeZYT0cYZG398zXTQ.png" /></figure><p>In the cell containing the Message column. Write = to create a formula in Google Sheets. Add the text using “” you have to close the page to save it correctly. Then add a variable representing a container for data or cell.</p><p>We want to add the data in the Name column, for instance, Josiah is on the A2 cell use the &amp; sign to add a variable, so it starts and ends the variable A2 .</p><p>The code is written here.</p><pre>=&quot;Hi, &quot;&amp;A2&amp;&quot;. Kindly send me back a message once you see this.&quot;</pre><ul><li>Duplicate the remaining columns by dragging the lower cross icon on the first cell.</li></ul><h4>#3- Create a WhatsApp link.</h4><ul><li>Create a spreadsheet formula using = characters.</li><li>Hyperlink — represents changing a text to a link.</li><li>Wrap the text in a link.</li><li>Attach the phone numbers.</li><li>Enclose the variable B2 with the &amp; symbol</li><li>The “send message”, serves as a variable to the link, showing on the cell D2:D5</li></ul><pre>=HYPERLINK(&quot;https://api.whatsapp.com/send?phone=&quot;&amp;C2&amp;&quot;&amp;text=&quot;&amp;D2, &quot;Send Message&quot;)</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*vyueIm-To1I-tnJaYBJNzg.png" /></figure><ul><li>Click on the link.</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*zVDn5Ih6SteQk6yY_Ree-A.png" /><figcaption>Whatsapp message</figcaption></figure><h3>Conclusion</h3><p>In conclusion, by following the steps outlined in this guide, you can easily and efficiently send personalized WhatsApp messages using Google Sheets. This powerful combination of tools allows you to streamline your communication and reach your audience in a more targeted and effective way. With a little bit of setup and some basic knowledge of Google Sheets, you can take your messaging to the next level.</p><p>Do well to share and comment if you find this article insightful.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=a577ba4958a8" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[How to Use Named Entity Recognition in Spacy to Analyze Blog Content.]]></title>
            <link>https://josiah-adesola.medium.com/how-to-use-named-entity-recognition-in-spacy-to-analyze-blog-content-c46d50a8eb94?source=rss-8d681a6b68c7------2</link>
            <guid isPermaLink="false">https://medium.com/p/c46d50a8eb94</guid>
            <category><![CDATA[spacy]]></category>
            <category><![CDATA[nlp]]></category>
            <category><![CDATA[medium]]></category>
            <category><![CDATA[named-entity-recognition]]></category>
            <dc:creator><![CDATA[Josiah Adesola]]></dc:creator>
            <pubDate>Fri, 17 Mar 2023 10:35:52 GMT</pubDate>
            <atom:updated>2023-03-17T12:01:31.186Z</atom:updated>
            <content:encoded><![CDATA[<p>Named entity recognition is vital for deriving insights from unstructured data and making the right decisions in no time.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/612/1*U0MbB1Wk9dmCMOh0gcAXtw.png" /><figcaption>photo from Unsplash</figcaption></figure><p>Millions of posts in the form of text, audio, images, videos, and other graphical content are posted on blogs daily. However, it is very difficult for businessmen, marketers, and researchers to derive valuable information from vast amounts of unstructured data (most especially text posts).</p><p>What you’ll learn about:</p><ul><li>Named Entity Recognition (NER)</li><li>Application of NER</li><li>SpaCy for NER</li><li>Building a NER model with SpaCy</li></ul><h3>Named Entity Recognition (NER)</h3><p>Named entity recognition, popularly known as NER, is a modern natural language processing technique that extracts, highlights, or recognizes key entities (mostly proper nouns and keywords) from a group of unstructured text data.</p><p>Named entities in this context, are names of people, organizations, locations, dates, productions, ordinals, money, and geographical areas such as water, and mountains. NER is really efficient for extracting key words you want to derive from a lengthy document, text, article, or blog, running analytics, and making informed decisions.</p><p>Named entities have labels to represent them. For instance, organizations such as Microsoft, and Google are represented with ORG, a person can be represented with PER, and Texas or Lagos can be represented with GEO. Check out this table for more reference.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/410/1*F7X3RYX9BX7N1qv0BM2UzA.png" /><figcaption>Named entities labels</figcaption></figure><h3>Application of Named Entity Recognition</h3><p>NER is not only limited to extracting key features or words from blog content but can also be used in a wide variety of institutions and sectors. I’ll give some best use cases below, but the list is not exhaustive.</p><ol><li><strong>Healthcare</strong>: NER can be used by medical practitioners to extract and classify medical entities such as the name of the patients, illness name, medical procedures, and drug name from the patient record or medical report. This will save a lot of time, identify patterns in patient data, monitor disease outbreaks and help proscribe the right treatments.</li><li><strong>Education</strong>: NER can be used to extract educational entities from a student&#39;s biodata or course profile. The educational entities should include the student name, institution name, academic degrees, age, department name, and course name. This will help educators spend less time trying to extract information, identify patterns in a student and spend more time improving teaching methods.</li><li><strong>Finance</strong>: The finance field is so vast with a lot of entities which can be extracted from finance blogs, news posts, and posts on social media. The entities include stock symbols, company names, and financial metrics. This will help investors make informed decisions quickly and identify key potential opportunities.</li><li><strong>Legal</strong>: The lawyers don’t have to spend so much time trying to get the case name, court name, legal citations, and other legal entities from a document anymore. NER can be used to extract this information and streamline legal research and analysis.</li><li><strong>E-Commerce</strong>: As an e-commerce vendor or an online buyer, many times product descriptions can be so overwhelming and you cannot quickly pick out key entities such as product name, price, short reviews, and specific features. NER can help identify the popular product, monitor customer feedback as the E-commerce business and optimize product offerings.</li><li><strong>Marketing</strong>: NER can be used to analyze customer reviews and feedback. This is one of the most common use cases of NER, it can identify sentiments associated with products, and help build customer satisfaction and loyalty.</li></ol><h3>SpaCy</h3><p><a href="https://spacy.io/">Spacy</a> is a powerful, open-source, Python-based library tool for text processing and analysis. It is an easy-to-use framework, SpaCy is built for industry standards and focuses on usability.</p><p>It has a user-friendly environment, and developers can easily add custom functionalities and features to its library. It takes pride in processing a large amount of text data quickly, due to its high-performance rate.</p><p>Spacy uses statistical modes to predict and analyze text data. It offers several pre-trained models for various languages, such as English, German, Spanish, and others. The great part is that you can easily fine-tune it to your needs.</p><h3>Building a NER model with SpaCy</h3><p>We will be building the named entity recognition model using SpaCy (a Python library). There are three simple steps to building the NER model.</p><ol><li><strong>Import libraries</strong>: In many machine learning projects, it is important to install necessary libraries to import functions necessary to perform specific tasks.</li></ol><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/1b83abf24d94f41222bbd86c85a46387/href">https://medium.com/media/1b83abf24d94f41222bbd86c85a46387/href</a></iframe><ul><li>spacy : is a python library used for text processing and analysis.</li><li>spacy.displacy : is a module in the Spacy library that provides a visual interface for the display of text data. It generates an HTML visualization of the output, it is also used in part-of-speech (POS) tagging and dependency parsing.</li></ul><p>2. <strong>Perform entity extraction:</strong> The text is loaded into a pre-trained Spacy model for English language processing, and then it extracts the named entities and outputs them in a vertical list.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/6f8f05eedded42d853fe9573a2e3d675/href">https://medium.com/media/6f8f05eedded42d853fe9573a2e3d675/href</a></iframe><p>nlp : is a variable used to store the pre-trained model. spacy.load provides the pre-trained model, en_core_web_lg .</p><blockquote>Let’s break it down.</blockquote><p>The en signifies English language, spacy can perform operations on other languages too such as Spanish, finish, german and so on, core signifies it can be used for general-purpose pipelines such as tagging, parsing, lemmatization and named entity recognition. The web signifies that the model was trained on the web. lg signifies a large model, we could have sm — small model or md — a medium model. The larger model takes more time and computation power but it tends to be accurate.</p><p>doc: is a variable used to store the processed text nlp in token forms, entities and more.</p><p>loop : We perform a for-loop operation on doc.ent — the document entities. These are vectorized content and can be used to access the ent.text — exact text to be called a named entity, ent.start_char — the position of the first letter of the text in the word, just like lists. ent.end_char — the position of the last character of the ent.text output. ent.label_ — signifies an abbreviation of the named entity tag, for instance, John Smith is a PERSON .</p><p>Check out the output of the code.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/525/1*Pi__7_LpDPh5Nn_w1wF9fQ.png" /><figcaption>the code output</figcaption></figure><p>3. <strong>Display named entity</strong>: This is the final code for this task. An HTML visualization is needed to properly display the named entity as their labels are abbreviated, person is abbreviated to PER.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/5419c3e86db4abd56e9595627223f17b/href">https://medium.com/media/5419c3e86db4abd56e9595627223f17b/href</a></iframe><p>displacy shows the HTML visualization of the named entities using the serve a function called on the doc — the tokenized form of the text.</p><p>Here, we go. Check out the final output.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*n_dUucs9z28PKIl0gkx8-g.png" /><figcaption>the final NER</figcaption></figure><p>Check out the full code on this <a href="https://www.kaggle.com/code/josiahadesola/named-entity-recognition-ner/">Kaggle</a> link.</p><h3>Conclusion</h3><p>Named entity recognition can be applied in several sectors and for specialized functions. In healthcare, NER can be applied to help doctors easily see the disease and ailments of patients from a medical report, and can further be used for diagnosis and medical classifications. The list of applications is endless, depending on your field.</p><iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Ftenor.com%2Fembed%2F25892483&amp;display_name=Tenor&amp;url=https%3A%2F%2Ftenor.com%2Fen-GB%2Fview%2Fbob-anakshie-gif-25892483&amp;image=https%3A%2F%2Fmedia.tenor.com%2FArsu0w_nD2EAAAAF%2Fbob-anakshie.png&amp;key=a19fcc184b9711e1b4764040d3dc5c07&amp;type=text%2Fhtml&amp;schema=tenor" width="600" height="400" frameborder="0" scrolling="no"><a href="https://medium.com/media/5c180d349cbabb03d39ff0c0311afffa/href">https://medium.com/media/5c180d349cbabb03d39ff0c0311afffa/href</a></iframe><p>See you later. Thanks for reading up until now. Don’t forget to leave a clap, and make a comment if you find this read insightful and helpful. Bye Bye</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=c46d50a8eb94" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[How to Create a Text Summarizer with the Hugging Face Library: A Beginner’s Guide]]></title>
            <link>https://josiah-adesola.medium.com/how-to-create-a-text-summarizer-with-the-hugging-face-library-a-beginners-guide-812d202b1329?source=rss-8d681a6b68c7------2</link>
            <guid isPermaLink="false">https://medium.com/p/812d202b1329</guid>
            <category><![CDATA[hugging-face]]></category>
            <category><![CDATA[text-summarization]]></category>
            <category><![CDATA[nlp]]></category>
            <dc:creator><![CDATA[Josiah Adesola]]></dc:creator>
            <pubDate>Sat, 25 Feb 2023 09:37:06 GMT</pubDate>
            <atom:updated>2023-02-25T09:37:06.616Z</atom:updated>
            <content:encoded><![CDATA[<p>Text summarization is a non-negligible part of writing, So, how can you create a program to summarize a body of text quickly?</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/612/0*bgboTgqc-jCtoNVb" /></figure><p>Natural Language Processing (commonly known as NLP) is a part of Artificial Intelligence that can interpret and perform tasks with human language. The emergence of ChatGPT has led to the use of NLP in various fields such as chatbots, named entity recognition, question-and-answer systems, and many more. Among all the demands, text summarization has particularly been a major one in the NLP space.</p><p>What you’ll learn:</p><ul><li>What is text summarization?</li><li>About the Hugging face platform</li><li>How to use Google Colab</li><li>How to build a text summarizer</li></ul><h3><strong>What is Text Summarization?</strong></h3><p>Text summarization is reducing long articles, a body of texts or paragraphs into shorter and more meaningful statements and sentences. NLP commonly uses this technique to form a consolidated version of articles for proper understanding.</p><p>Applying text summarization results in easy-to-read and quick comprehension for customer reviews, emails, social media messages, and taking minutes of meetings among many others.</p><h3><strong>Hugging Face Platform</strong></h3><p>The <a href="https://huggingface.co/">Hugging Face</a> platform provides an online repository for thousands of NLP models and datasets from individuals and top tech companies like Google, OpenAI, Facebook, and Microsoft. By, offering a lot of pre-trained models, saves users the stress of building an NLP project from scratch.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/650/0*NdsSN65_4uV4ust1" /></figure><p>To train an NLP model properly, you need millions, if not billions of datasets from various sources, which can be costly in terms of time, energy, and resources. The Hugging Face platform offers an alternative by allowing you to use pre-existing datasets, as well as upload your own models and create private datasets. Additionally, you can host demo ML apps using Python SDKs such as <a href="https://gradio.app/">Gradio </a>and <a href="https://streamlit.io/">Streamlit</a>.</p><p><strong>What are Transformers?</strong></p><p>Transformers, a neural network architecture in NLP, uses two major components called encoders and decoders. Encoders convert inputs, such as text for language translation and text summarization, into readable vectors for the machine. Decoders then interpret those vectors from the machine into final results that are easy for people to understand</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/790/0*i_ilYj_dsoaaTd0-" /></figure><p>In 2017, Google Researchers developed Transformers, with the article called “<a href="https://papers.neurips.cc/paper/7181-attention-is-all-you-need.pdf">Attention is All You Need</a>” which revolutionized the way NLP models were trained. Before transformers, NLP models used Recurrent Neural Networks (RNNs), but they had the problem of interpreting words in one part of a document differently than another word in a different part of the same document, similar to memory loss.</p><iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Fembed.deepnote.com%2Fc18ac273-b171-4fee-8bb9-dea68debbd43%2F565542b8632547fba64836db8ac8a47e%2Feaa3dfc602f3452c80c223ea9d7d6ad6%3Fheight%3D152&amp;display_name=Deepnote&amp;url=https%3A%2F%2Fembed.deepnote.com%2Fc18ac273-b171-4fee-8bb9-dea68debbd43%2F565542b8632547fba64836db8ac8a47e%2Feaa3dfc602f3452c80c223ea9d7d6ad6%3Fheight%3D152&amp;image=https%3A%2F%2Fdeepnote.com%2Fstatic%2Fthumbnails%2Fmain.png&amp;key=a19fcc184b9711e1b4764040d3dc5c07&amp;type=text%2Fhtml&amp;schema=deepnote" width="700" height="152" frameborder="0" scrolling="no"><a href="https://medium.com/media/30dba49b546052d1ffb97fdec05f1d99/href">https://medium.com/media/30dba49b546052d1ffb97fdec05f1d99/href</a></iframe><p>Transformers take all the words in a document as a whole, perform a semantic comparison for better understanding and convert them into vectors, which makes them readable for the computer.</p><h3><strong>Building the Text Summarizer in 5 Steps</strong></h3><ol><li><strong>Launch your Google Colab</strong>:<a href="https://colab.research.google.com/"> Google Colab</a> is an ML notebook for coding ML programs in the<a href="https://www.python.org/"> Python</a> programming language. It can be accessed online and it is so efficient for collaboration, and it comes with some pre-installed python libraries.</li><li>Go to the <a href="https://colab.research.google.com/">Google Colab</a> website.</li><li>Sign in with your Gmail account.</li><li>Click on the “New Notebook” text.</li></ol><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*rfYTQpLriw3WyAXO" /></figure><ol><li>Edit the title, do not edit the “.ipynb” text(this is the notebook extension).</li></ol><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*QrmeNTkz0453hUtb" /></figure><p>After doing this successfully. You’re ready to start building your text summarizer program.</p><p>2. <strong>Install the transformer library</strong>: As explained in the previous part of this article, transformers are key components in the NLP. It encodes and decodes the text for your computer to understand.</p><iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Fembed.deepnote.com%2Fe7c888a6-eeab-4b09-a517-1e1acfdaf38c%2Fb1f5a8ca4c824fad887a1dcd81509ae7%2Ff9701faea7dc4abb924ba86c8680cd85%3Fheight%3D101&amp;display_name=Deepnote&amp;url=https%3A%2F%2Fembed.deepnote.com%2Fe7c888a6-eeab-4b09-a517-1e1acfdaf38c%2Fb1f5a8ca4c824fad887a1dcd81509ae7%2Ff9701faea7dc4abb924ba86c8680cd85%3Fheight%3D101&amp;image=https%3A%2F%2Fdeepnote.com%2Fstatic%2Fthumbnails%2Fmain.png&amp;key=a19fcc184b9711e1b4764040d3dc5c07&amp;type=text%2Fhtml&amp;schema=deepnote" width="700" height="101" frameborder="0" scrolling="no"><a href="https://medium.com/media/e2c4bdb1736b9395e727f9e00f1cd6a6/href">https://medium.com/media/e2c4bdb1736b9395e727f9e00f1cd6a6/href</a></iframe><ul><li>The first line of code installs the transformers library from the hugging face platform.</li><li>The second line of code imports the pipeline function from the “transformers” library.</li></ul><p>3. <strong>Build your model</strong>: The “summarizer” is a variable, with a call of the “pipeline” function. I know you are thinking that this is just a single line of code, how simple. Yes, this is because as you can see, the model used here is by Facebook, you can get the link and check on the hugging face platform once you run the code.</p><iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Fembed.deepnote.com%2Fe7c888a6-eeab-4b09-a517-1e1acfdaf38c%2Fb1f5a8ca4c824fad887a1dcd81509ae7%2F1a005fb81edd4c2eaa090c8f596ae3a7%3Fheight%3D228.8000030517578&amp;display_name=Deepnote&amp;url=https%3A%2F%2Fembed.deepnote.com%2Fe7c888a6-eeab-4b09-a517-1e1acfdaf38c%2Fb1f5a8ca4c824fad887a1dcd81509ae7%2F1a005fb81edd4c2eaa090c8f596ae3a7%3Fheight%3D228.8000030517578&amp;image=https%3A%2F%2Fdeepnote.com%2Fstatic%2Fthumbnails%2Fmain.png&amp;key=a19fcc184b9711e1b4764040d3dc5c07&amp;type=text%2Fhtml&amp;schema=deepnote" width="700" height="228" frameborder="0" scrolling="no"><a href="https://medium.com/media/00d1adbac05ecd5edc8e72734c130ac4/href">https://medium.com/media/00d1adbac05ecd5edc8e72734c130ac4/href</a></iframe><p>The model here is a pre-trained model, you do not need to do data cleaning, tokenization, and the rest.</p><p>All these processes and more have been done, you only need to call the pipeline-summarization function, input the model you wish and run the code. While running your code, you notice that the models get downloaded.</p><p>4. <strong>Summarize your article:</strong> The final step is to summarize your article into the desired length you want.</p><iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Fembed.deepnote.com%2Fe7c888a6-eeab-4b09-a517-1e1acfdaf38c%2Fb1f5a8ca4c824fad887a1dcd81509ae7%2F34e7aae5ca0c40a28a9838509a7d0a0c%3Fheight%3D277.60000228881836&amp;display_name=Deepnote&amp;url=https%3A%2F%2Fembed.deepnote.com%2Fe7c888a6-eeab-4b09-a517-1e1acfdaf38c%2Fb1f5a8ca4c824fad887a1dcd81509ae7%2F34e7aae5ca0c40a28a9838509a7d0a0c%3Fheight%3D277.60000228881836&amp;image=https%3A%2F%2Fdeepnote.com%2Fstatic%2Fthumbnails%2Fmain.png&amp;key=a19fcc184b9711e1b4764040d3dc5c07&amp;type=text%2Fhtml&amp;schema=deepnote" width="700" height="277" frameborder="0" scrolling="no"><a href="https://medium.com/media/2377d7f52ba00f0284337f873772ff8e/href">https://medium.com/media/2377d7f52ba00f0284337f873772ff8e/href</a></iframe><p>After running this code, you will have an output of a minimum of 25 words, and a maximum of 50 words. The variables such as “max_len” and “min_len” can be altered to change the maximum and minimum length of the word respectively.</p><p>Text summarization has a lot of applications in summarizing text documents, customer reviews and feedback, research papers, getting the necessary details from product reviews and a host of others.</p><p><strong>Demo</strong></p><p>Watch the <a href="https://drive.google.com/file/d/17C_bS3WqoRS8PNTaOfs1Vyvaz82BBN_q/view?usp=share_link">live demo</a> of the project.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=812d202b1329" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[PyCaret for Beginners: Building Your First Classification Model]]></title>
            <link>https://josiah-adesola.medium.com/pycaret-for-beginners-building-your-first-classification-model-165ad8d25562?source=rss-8d681a6b68c7------2</link>
            <guid isPermaLink="false">https://medium.com/p/165ad8d25562</guid>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[classification]]></category>
            <category><![CDATA[projects]]></category>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[pycaret]]></category>
            <dc:creator><![CDATA[Josiah Adesola]]></dc:creator>
            <pubDate>Mon, 09 Jan 2023 23:09:25 GMT</pubDate>
            <atom:updated>2023-01-10T03:08:02.339Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*2Y1iLNOeAlPAPAK1" /><figcaption>Photo by <a href="https://unsplash.com/@crew?utm_source=medium&amp;utm_medium=referral">Crew</a> on <a href="https://unsplash.com/?utm_source=medium&amp;utm_medium=referral">Unsplash</a></figcaption></figure><p>Are you looking for a way to build a machine learning (ML) model without writing a lot of code? If you have a no-code or low-code background, the PyCaret library can make it easy for you to build ML models like a pro.</p><p>At the end of this article, you will:</p><ul><li>Understand the knowledge of PyCaret.</li><li>Understand machine learning workflow.</li><li>Build a breast cancer classification model using PyCaret.</li></ul><h3>What is PyCaret?</h3><p><a href="http://pycaret.org/">PyCaret </a>is a low-code machine learning Python library. PyCaret eliminates the need for extensive ML processing to create a model. You can complete tasks that would normally take hours in minutes. It is amazing, right?</p><p>PyCaret is designed for individuals with little coding experience. It allows you to easily build and deploy machine learning models. It automates the machine learning workflow with minimal coding. PyCaret is designed to be flexible and scalable, saving you a significant amount of coding time. As a beginner, intermediate, or experienced ML practitioner, PyCaret stands as an excellent choice.</p><h3>Prerequisites</h3><ul><li>Fundamental knowledge of <a href="https://www.python.org/">Python Programming</a>.</li><li>A <a href="https://colab.research.google.com/">Google Colab</a> or <a href="https://www.anaconda.com/products/distribution">Jupyter notebook</a> to run the python code.</li><li>Basic understanding of binary classification.</li><li><a href="https://pycaret.org/">PyCaret</a> 2.0 version or greater.</li></ul><h3>Breast Cancer Classification Model With PyCaret</h3><p>Breast cancer accounts for more than 25% of cancer cases, and millions of people die from it daily. It is caused by the abnormal growth of cells in the breast. This model will predict whether a patient’s breast cancer is <strong>malignant</strong> (deadly) or <strong>benign</strong> (non-deadly).</p><p>The dataset can be found at this <a href="https://www.kaggle.com/datasets/yasserh/breast-cancer-dataset">Kaggle link</a>.</p><ol><li><strong>Download PyCaret: </strong>To use the PyCaret ML library, you can install it in your Jupyter notebook or Colab. This will allow you to access and utilize the library’s functions and features within these environments.</li></ol><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/dd188aea469ffc52f5bb979e553845b9/href">https://medium.com/media/dd188aea469ffc52f5bb979e553845b9/href</a></iframe><p>This line of code installs several Python libraries that are necessary for using PyCaret</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*HvPrro3OoN0HvlfyAmmLoA.png" /><figcaption>Image by author</figcaption></figure><p><strong>2. Import necessary libraries: </strong>For this project, you will need two Python libraries: Pandas and PyCaret. Pandas is already installed on your Google Colab or Jupyter notebook, so you only need to install PyCaret.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/7fe1d64148f106ed1f4c78a7eea41f52/href">https://medium.com/media/7fe1d64148f106ed1f4c78a7eea41f52/href</a></iframe><p><strong>3. Preview the dataset: </strong>To help identify the target variable from the remaining variables, you should check the dataset. Pandas can be used to preview the first five rows of the table.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/a6ea095c40e0fd6c368a09968b3b04f4/href">https://medium.com/media/a6ea095c40e0fd6c368a09968b3b04f4/href</a></iframe><p>The output produced from this code is shown below.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*WkqRcg_HPtRPmm3oGr36Sg.png" /><figcaption>Image by author</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/665/1*V9Jr2aqcaiRirGfD75RRJg.png" /></figure><p>The target variable is the `diagnosis` column, which represents whether the cancer is malignant (M) or benign (B). It is a categorical variable that can be preprocessed with PyCaret. The remaining variables are numerical and will be used to predict the target variable.</p><p>Follow the tutorial above to learn how to import <a href="http://kaggle.com/">Kaggle </a>datasets into your Google Colab in 3 easy steps</p><p><a href="https://josiah-adesola.medium.com/3-simple-steps-to-import-kaggle-dataset-directly-into-google-colab-96428eab9678">3 Simple Steps to Import Kaggle Dataset Directly Into Google Colab</a></p><p><strong>4. Data Preprocessing: </strong>This article does not focus on data preprocessing. However, it is important to note that machine learning models can only understand numerical data. Therefore, we will need to use the Scikit-learn library to convert the ‘diagnosis’ column, which contains the categorical data ‘M’ and ‘B’, to numerical values 1 and 0 respectively.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/8d8bc874272ef1078e09b0821e0321a6/href">https://medium.com/media/8d8bc874272ef1078e09b0821e0321a6/href</a></iframe><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*2taKUMYuTD5WutbYdPZb3g.png" /><figcaption>The diagnosis column has changed to figures</figcaption></figure><p><strong>5. Setup the PyCaret environment: </strong>To use PyCaret’s built-in functions, you must first set up a PyCaret environment. If you are using Colab, you will need to run these lines of code to enable Google Colab. If you do not do this, you may encounter error messages</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/457ad1d9f7c446937d1daae26fdc8c0d/href">https://medium.com/media/457ad1d9f7c446937d1daae26fdc8c0d/href</a></iframe><figure><img alt="" src="https://cdn-images-1.medium.com/max/908/1*P4J1yabedpB94H0ic0deDw.png" /><figcaption>enable colab image by author</figcaption></figure><p>There are three important things to consider when setting up the PyCaret environment:</p><ol><li><strong>Data</strong>: The dataset (df) should be mapped to the ‘data’ variable. Note that ‘df’ is just the variable name I chose for my dataset. You can use any variable name you prefer.</li><li><strong>Target</strong>: Set the target variable column using the ‘target’ parameter.</li><li><strong>Session ID</strong>: This helps to ensure the reproducibility of the code.</li></ol><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/3c64dbf7469b38a44230a939dda929b4/href">https://medium.com/media/3c64dbf7469b38a44230a939dda929b4/href</a></iframe><p>After running these lines of code, you get the output in the image below.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/622/1*-qmasGPAa6VG0MVl4fDzEA.png" /><figcaption>output from the above code by author</figcaption></figure><p>Here is a brief overview of the code output to help with understanding:</p><ul><li><strong>Target Type</strong>: This indicates the type of classification being performed. In this case, we are classifying breast cancer as either malignant or benign, so it is a binary classification.</li><li><strong>Original Data</strong>: This shows the number of rows and columns in the dataset. The dataset in this example has 569 rows and 32 columns, including the target variable.</li><li><strong>Transformed Train and Transformed Test Set</strong>: By default, the original dataset is split into a 67:33 ratio. There are 30 columns instead of 31 because the ‘id’ column has been removed as it is not useful for prediction.</li><li><strong>Missing value</strong>: The output indicates ‘false’, which means there are no missing values in the dataset.</li></ul><p>There is additional information that can be obtained by running this operation in a Jupyter or Google Colab notebook.</p><p><strong>6. Select the best model</strong>: This is a very simple task that took only 20 seconds to complete on Google Colab. The code sorted and printed out over 15 models from highest to lowest using the &quot;sort&quot; parameter for the f1-score evaluation metric. You can try practising with other evaluation metrics such as accuracy and recall.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/c8b77fbe0bcd9fbbb2b84fc20b1f60c1/href">https://medium.com/media/c8b77fbe0bcd9fbbb2b84fc20b1f60c1/href</a></iframe><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*h6MWLcuTqoH6cWcAe8Ss7g.png" /><figcaption>compare_model code output by author</figcaption></figure><p><strong>7. Build a classifier model:</strong> The ‘create_model’ function can be used to build any of the classifier models using any of the acronyms listed in the image above.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/c628407d552c1674cd1b13809b941bf3/href">https://medium.com/media/c628407d552c1674cd1b13809b941bf3/href</a></iframe><figure><img alt="" src="https://cdn-images-1.medium.com/max/842/1*kNBj9EfYZdeOxijCIJdzaw.png" /><figcaption>light gbm classifier model by author</figcaption></figure><p><strong>8. Hyperparameter Tuning</strong>: Hyperparameter tuning is an efficient ML workflow aimed at optimizing your ML model. “The ‘tune_model’ function is used to optimize the hyperparameters of a model through a process called hyperparameter tuning</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/1174e5806956e754ef840e17804da2d3/href">https://medium.com/media/1174e5806956e754ef840e17804da2d3/href</a></iframe><figure><img alt="" src="https://cdn-images-1.medium.com/max/869/1*-NR_CKiEcf6Kkq9dXE917Q.png" /><figcaption>Hyperparameter tuning with PyCaret by author</figcaption></figure><p><strong>9. Feature Importance:</strong> Feature importance is a technique that assigns a value or score to each feature to indicate how significant it is in predicting the target variable. By removing redundant features and utilizing the most important ones, your model can make more accurate predictions.</p><p>PyCaret makes it easy to visualize the feature importance of your model with a graph.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*pzXVgjZX9JkL_VgX_nVcdw.png" /><figcaption>Feature selection with PyCaret by author</figcaption></figure><p><strong>10. Model Evaluation: </strong>Using PyCaret, you can easily evaluate your model with just one line of code. In this example, two model evaluations were performed using the ROC-AUC curve and confusion matrix.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*hxC6KFaH27lrtGWdSPqhaQ.png" /><figcaption>ROC-AUC Curve using PyCaret by author</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*XQQdVeBfzNc3TbvG1fYqcQ.png" /><figcaption>Confusion matrix using PyCaret by author</figcaption></figure><p><strong>11. Prediction: </strong>You can easily make predictions to compare the test dataset and the train dataset, which will help you determine the validity of your evaluation metrics.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/ac09b646708d9c7259c86d5e3fb2996a/href">https://medium.com/media/ac09b646708d9c7259c86d5e3fb2996a/href</a></iframe><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*LspqowPMJb7Xs1nNR3GX1A.png" /><figcaption>predict model by author</figcaption></figure><p>Remember to add a semicolon at the end of the code to produce the above output</p><p><strong>12. Save your model:</strong> Saving your model allows it to be reused in various applications or websites, and it can also be a way to deploy your machine learning solution. PyCaret makes it easy to do this with just a few lines of code</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/d240ba12055d3e657305f3590e86fcd2/href">https://medium.com/media/d240ba12055d3e657305f3590e86fcd2/href</a></iframe><p>After running this line of code, your trained model will be saved in a pickle (pkl) file.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*XTCncz8NzrHcC9ZG7J59yg.png" /><figcaption>save model with PyCaret by author</figcaption></figure><p>Glad you made it this far</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/500/1*-86R6a33-ShZ-fm2YKXQJA.gif" /><figcaption>cheering gif from <a href="https://www.icegif.com/">https://www.icegif.com/</a></figcaption></figure><h3>Conclusion</h3><p>In conclusion, PyCaret is an efficient tool for low-code machine learning practitioners or enthusiasts. It can be applied not only to binary classification problems, but also to multi-classification, regression, clustering, data processing, and even natural language processing problems.</p><p>I encourage you to practice along with this article and share your feedback for support and improvement. You can find the link to the full Jupyter notebook here.</p><p><a href="https://github.com/Josiah-Adesola/Machine-Learning-Articles/tree/main/PyCaret%20for%20Beginners">Machine-Learning-Articles/PyCaret for Beginners at main · Josiah-Adesola/Machine-Learning-Articles</a></p><p>See you later. Bye for now! Don’t forget to give a clap, and make necessary comments for feedback.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=165ad8d25562" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Model Evaluation Metrics For Binary Classification]]></title>
            <link>https://josiah-adesola.medium.com/model-evaluation-metrics-for-binary-classification-a-practical-and-well-explanatory-approach-608c89dce67f?source=rss-8d681a6b68c7------2</link>
            <guid isPermaLink="false">https://medium.com/p/608c89dce67f</guid>
            <category><![CDATA[logistic-regression]]></category>
            <category><![CDATA[machinelerning]]></category>
            <category><![CDATA[classification]]></category>
            <category><![CDATA[model-evaluation]]></category>
            <category><![CDATA[ai]]></category>
            <dc:creator><![CDATA[Josiah Adesola]]></dc:creator>
            <pubDate>Thu, 20 Oct 2022 00:59:58 GMT</pubDate>
            <atom:updated>2023-03-09T02:44:41.899Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/848/1*pkl0VnrfIQN3pnT33gXonQ.png" /></figure><p><strong>Model evaluation is a crucial aspect of any machine learning process. In this article, we used a practical and explanatory approach to explore several evaluation metrics for binary classification, including the F1-score, precision, recall, and accuracy</strong></p><h3>Introduction</h3><p>Classification is a type of supervised machine learning that has many real-world applications, such as detecting spam emails, predicting fraud, and predicting medical conditions.</p><p>Many industries rely on machine learning to solve classification problems in real time, which raises questions about the accuracy and reliability of the models being deployed.</p><p>Model evaluation is the process of answering these questions and ensuring that the predictions being made are accurate and won’t have negative consequences for the business. In this article, we will focus on model evaluation for binary classification problems.</p><p>By the end of this tutorial, you will be able to:</p><ul><li>Understand at least 5 evaluation metrics for classification problems</li><li>Apply the evaluation metrics to a dataset</li><li>Know why certain classification metrics are preferred over the other</li></ul><h3>Prerequisite</h3><p>You’ll need the following:</p><ul><li>A fair knowledge of the <a href="https://www.python.org/">Python</a> Programming Language</li><li>Basic knowledge of the Scikit-learn library for machine learning.</li><li>A <a href="https://www.anaconda.com/products/distribution">Jupyter notebook</a> installed on your machine or <a href="https://colab.research.google.com/">Google Colab </a>Notebook.</li></ul><h3>What is an Evaluation Metric?</h3><p>Have you ever thought about why students take exams or tests after receiving lectures or training in a particular subject? This is because the teacher wants to see how well the student can apply the knowledge to real-life problems. That’s what <strong><em>evaluation metrics</em></strong> are all about in the context of machine learning.</p><p>Evaluation metrics are ways of measuring the performance of your machine learning model by comparing it to other values or test datasets (datasets that were not used to train the model).</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/746/1*sBwCidOOkC818ZnbvFdaYg.png" /><figcaption>model evaluation metrics from javapoint.com</figcaption></figure><p>Model evaluation is an important part of the machine learning process. In this tutorial, we will delve into the evaluation metrics commonly used in binary classification, understand the reasoning behind them, and apply them to a Kaggle dataset.</p><blockquote>Let’s consider various components of evaluation metrics for binary classification models.</blockquote><figure><img alt="" src="https://cdn-images-1.medium.com/max/586/1*Jjkx5bsGjx9NZhxQ4H583A.png" /><figcaption>Image from facebook.com</figcaption></figure><p><strong>True Positive (TP):</strong> This refers to a situation where the model correctly predicts the positive class. For example, correctly predicting that a patient has a certain medical condition.</p><p><strong>True Negative (TN)</strong>: This refers to a situation where the model correctly predicts the negative class. For example, correctly predicting that a patient does not have a certain medical condition.</p><p><strong>False Negative (FN)</strong>: This refers to a situation where the model incorrectly predicts the negative class. For example, predicting that a patient does not have a certain medical condition when they actually do.</p><p><strong>False Positive (FP)</strong>: This refers to a situation where the model incorrectly predicts the positive class. For example, predicting that a patient has a certain medical condition when they actually do not.</p><p><strong>True Positive Rate (TPR)</strong>: TPR, also known as recall or sensitivity, is the ratio of true or correct positive (TP) predictions to the total number of positive predictions (TP and FN).</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/972/1*wJYCo_b2gX2od1HG5Yf-eg.png" /><figcaption>True Positive Rate</figcaption></figure><p><strong>True Negative Rate (TNR): </strong>TNR, also known as <strong>specificity</strong>, is the ratio of true negative (TN) predictions to the total number of negative predictions (TN and FP).</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Z_4llH3AoX5BBF6My03mnQ.png" /><figcaption>True Negative Rate</figcaption></figure><p><strong>False Positive Rate (FPR): The </strong>FPR, also known as the <strong>Type I Error Rate</strong>, is the ratio of false positive (FP) predictions to the total number of negative predictions (FP and TN).</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*RRE15HQ2roF29uOt5xEvmw.png" /><figcaption>False Positive Rate</figcaption></figure><p><strong>False Negative Rate(FNR)</strong>: FNR, also known as the <strong>Type II Error Rate</strong>, is the ratio of false negative (FN) predictions to the total number of positive predictions (FN and TP).”</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*eHvzOn2ZsdkbjDcjJxuGQg.png" /><figcaption>False Negative Rate</figcaption></figure><blockquote>Before going to evaluation metrics, let’s write build our model before using evaluation metrics.</blockquote><h3>Data Pre-processing and Model Building</h3><p>To summarize, the code in this tutorial is used to predict the breast cancer status of a patient, either benign or malignant. The Scikit-Learn library is utilized to perform basic data preprocessing on the target variables, which are represented as M (malignant) and B (benign) using Label Encoding.</p><p>The dataset is then split into train and test sets to prevent overfitting and facilitate model evaluation. The metrics used in this process will be discussed later.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/017af05eca464a9dc86a77475ea98466/href">https://medium.com/media/017af05eca464a9dc86a77475ea98466/href</a></iframe><h3>Evaluation Metrics</h3><ol><li><strong>Accuracy Score: </strong>The accuracy score is the ratio of correct predictions to the total number of predictions made by the model. However, the accuracy score and related metrics are not frequently used in real-world applications due to their limitations.</li></ol><figure><img alt="" src="https://cdn-images-1.medium.com/max/976/1*zbj0KaKTKVYEuM7I78tNoQ.png" /></figure><p>For example, when dealing with an imbalanced dataset such as predicting cancer as either malignant or benign, if the number of benign cases is 100 and the number of malignant cases is 700, the model is likely to predict more malignant cases than benign ones.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*yUlHoOWAYkdbb1ZwULbvug.png" /></figure><p>Now, let’s demonstrate the practical usage of this dataset obtained from Kaggle.</p><p><a href="https://www.kaggle.com/datasets/yasserh/breast-cancer-dataset">Breast Cancer Dataset</a></p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/c67b34d22f2835a38c63dae9851498cd/href">https://medium.com/media/c67b34d22f2835a38c63dae9851498cd/href</a></iframe><figure><img alt="" src="https://cdn-images-1.medium.com/max/773/1*Ue8J_1a6YIwcqwReBpVVUg.png" /><figcaption>The accuracy score</figcaption></figure><p>The image above shows that the model has an accuracy of 0.97 or <strong>97%</strong> using the <strong>accuracy score</strong> evaluation metrics.</p><p><strong>2. Confusion Matrix: </strong>The Confusion Matrix (CM) is a visual representation of the true positive (TP), true negative (TN), false positive (FP), and false negative (FN) values of the predicted and actual classes.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/994/1*hmA2TBPbZmHMzaiRWgexMQ.png" /><figcaption>Confusion Matrix</figcaption></figure><p><strong>CM</strong> is a useful tool for evaluating the performance of a binary or multi-class classification model, as it provides a clear breakdown of the model’s predictions.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/88dbdc65fa08c33aba71d2a56af3c5c6/href">https://medium.com/media/88dbdc65fa08c33aba71d2a56af3c5c6/href</a></iframe><figure><img alt="" src="https://cdn-images-1.medium.com/max/636/1*pAdlYpPDA4lLAk-OZaW5rg.png" /></figure><p>For example, in the confusion matrix above, 110 instances were predicted as false and were actually false (TN), 1 instance was predicted as true but was actually false (FP), 4 instances were predicted as false but were actually true (FN), and 59 instances were predicted as true and were actually true (TP).</p><p><strong>3. Precision: </strong>Precision is the measure of the number of actual positive cases out of all the predicted positive cases. It is calculated by dividing the number of true positive cases (TP) by the sum of the true positive and false positive cases (TP + FP). For example, if the model predicts that 10 people are pregnant, and only 8 of them are actually pregnant, the precision would be 80%.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/833/1*GbIxk-Wa0qtdaZOCFmiOJg.png" /><figcaption>precision formula</figcaption></figure><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/bc950e9bfcf121bd0e3c04014b4e8090/href">https://medium.com/media/bc950e9bfcf121bd0e3c04014b4e8090/href</a></iframe><figure><img alt="" src="https://cdn-images-1.medium.com/max/912/1*lBUuMveLITqIvIvrCVQ-WA.png" /></figure><blockquote>The precision of this model is 98.3%</blockquote><p><strong>4. Recall: </strong>Recall measures the proportion of actual positive cases that were correctly identified by the model. It is the ratio of true positives to all positive results, including those that were correctly predicted and those that were not. For example, if a model correctly identifies 9 out of 10 positive cases, it would have a recall of 90%.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/914/1*tPQl2fmEUKQ-YvgSStQNAw.png" /></figure><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/8c3c4db5091094f07189895e9f429150/href">https://medium.com/media/8c3c4db5091094f07189895e9f429150/href</a></iframe><figure><img alt="" src="https://cdn-images-1.medium.com/max/652/1*SaqnaBkCfEiaxuWXQH7F9Q.png" /></figure><blockquote>The recall is 93.7%</blockquote><p>5. <strong>f1-Score</strong>: The F1 score is the harmonic mean of precision and recall. It is a commonly used metric for evaluating binary classification models, particularly when the classes are imbalanced. The highest possible value for the F1 score is 1, which indicates perfect precision and recall.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*28oE4c6a0pn-_TNG7mbz2Q.png" /></figure><p>The F1 score is more informative than the accuracy score because it takes into account both false negatives and false positives. In real-world applications, it is often important to consider both types of errors.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*X8rntuGfjhb0zHLFQa-ONw.png" /></figure><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/701ca4e06fa0317be6d9914afee9edbe/href">https://medium.com/media/701ca4e06fa0317be6d9914afee9edbe/href</a></iframe><p>For example, in the case of predicting cancer diagnosis, a false negative (predicting a patient does not have cancer when they actually do) can have serious consequences, while a false positive (predicting a patient has cancer when they do not) can also cause unnecessary anxiety and further testing. The F1 score allows us to consider both types of errors in the evaluation of our model.</p><blockquote>To cut the story short, our f1_score is 95.9% accurate.</blockquote><p>6.<strong> AUC-ROC Curve: Receiver Operator Characteristic (ROC)</strong> is a probabilistic curve used to evaluate the performance of a binary classification model by plotting the true positive rate (TPR) against the false positive rate (FPR) at different classification thresholds</p><blockquote>The closer the ROC is to the TPR axis, the more accurate the model is.</blockquote><figure><img alt="" src="https://cdn-images-1.medium.com/max/892/1*niduiRSgRrlLoGverVgu8w.png" /><figcaption>Image from researchgate.net</figcaption></figure><p><strong>The area under the curve (AUC) is a </strong>statistical measure or summary of the curve, which ranges from 0 to 1. A value of 1 indicates perfect prediction or classification between the positive and negative class, while 0 indicates that the positive class was predicted as negative and the negative class was predicted as positive.</p><p>A value of 0.5 indicates that the model is struggling to classify the correct classes accurately. For the best results, the AUC score should be between 0.8 and 1.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/40a872a175d6cd7ce421e167c0e97946/href">https://medium.com/media/40a872a175d6cd7ce421e167c0e97946/href</a></iframe><figure><img alt="" src="https://cdn-images-1.medium.com/max/882/1*hXgE4nMGupALsPEp3wTBSA.png" /></figure><blockquote>The ROC-AUC-SCORE is approximately 96%</blockquote><h3>Conclusion</h3><p>In this tutorial, you learned how to use various evaluation metrics with real data. You now have a good understanding of the different evaluation metrics, including their purpose and how to use them. The full code for this article can be found on my GitHub repository.</p><p><a href="https://github.com/Josiah-Adesola/Machine-Learning-Articles/tree/main/Evaluation-Metrics-Binary-Classification">Machine-Learning-Articles/Evaluation-Metrics-Binary-Classification at main · Josiah-Adesola/Machine-Learning-Articles</a></p><iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Fgiphy.com%2Fembed%2Fl0MYzLLxlJDfYtzy0%2Ftwitter%2Fiframe&amp;display_name=Giphy&amp;url=https%3A%2F%2Fmedia.giphy.com%2Fmedia%2Fl0MYzLLxlJDfYtzy0%2Fgiphy.gif&amp;image=https%3A%2F%2Fi.giphy.com%2Fmedia%2Fl0MYzLLxlJDfYtzy0%2Fgiphy.gif&amp;key=a19fcc184b9711e1b4764040d3dc5c07&amp;type=text%2Fhtml&amp;schema=giphy" width="435" height="238" frameborder="0" scrolling="no"><a href="https://medium.com/media/1b6a66f548cf04ef270bbd3aee36523b/href">https://medium.com/media/1b6a66f548cf04ef270bbd3aee36523b/href</a></iframe><p>Thank you for reading, and I hope to see you in my next article. Bye!</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=608c89dce67f" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[3 Simple Steps to Import Kaggle Dataset Directly Into Google Colab]]></title>
            <link>https://josiah-adesola.medium.com/3-simple-steps-to-import-kaggle-dataset-directly-into-google-colab-96428eab9678?source=rss-8d681a6b68c7------2</link>
            <guid isPermaLink="false">https://medium.com/p/96428eab9678</guid>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[google-colab]]></category>
            <category><![CDATA[kaggle]]></category>
            <category><![CDATA[machine-learning]]></category>
            <dc:creator><![CDATA[Josiah Adesola]]></dc:creator>
            <pubDate>Fri, 14 Oct 2022 23:48:03 GMT</pubDate>
            <atom:updated>2023-01-10T01:43:18.958Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/772/1*TyrcG_YM4qyyDMsyZxNdQA.png" /><figcaption>Image from endtoend.ai</figcaption></figure><h3>Introduction</h3><p>It can be tough to get your Kaggle dataset into Google Colab. Before I found this simple method, I had to download the data from Kaggle, upload it to my Google Drive, and connect to Google Colab.</p><p>Do you want to know the easiest way to get your Kaggle dataset into Google Colab? No more downloading and uploading to your Google Drive, just a simple method to connect Kaggle to Google Colab. Keep reading to find out how!</p><iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Fgiphy.com%2Fembed%2FLTYT5GTIiAMBa%2Ftwitter%2Fiframe&amp;display_name=Giphy&amp;url=https%3A%2F%2Fmedia.giphy.com%2Fmedia%2FLTYT5GTIiAMBa%2Fgiphy.gif&amp;image=https%3A%2F%2Fi.giphy.com%2Fmedia%2FLTYT5GTIiAMBa%2F200.gif&amp;key=a19fcc184b9711e1b4764040d3dc5c07&amp;type=text%2Fhtml&amp;schema=giphy" width="435" height="287" frameborder="0" scrolling="no"><a href="https://medium.com/media/41fec69b168ac266fea2173fd5536f67/href">https://medium.com/media/41fec69b168ac266fea2173fd5536f67/href</a></iframe><p>It’s a strenuous process; you don’t want to imagine it.</p><blockquote>Let’s talk about Kaggle and Google Colab for newbies and beginners in Data Science…</blockquote><h3>Kaggle</h3><p>Kaggle is a fantastic platform for anyone interested in machine learning, data science, data analysis, and data engineering. It is a thriving community of ML enthusiasts and professionals, and one of the largest online communities for data professionals. With Kaggle, you can access a wealth of courses, datasets, and ML notebooks to help you start your journey in the field.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*f1vmOoTJAWQbTDxcoea2Uw.png" /><figcaption>kaggle.com/datasets</figcaption></figure><p>There are also numerous machine learning competitions available, with the chance to earn cash prizes by participating. Kaggle is a great resource for anyone looking to get started or advance in the world of machine learning and data science.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*8lPuh4PxS4U0ZSq6jbWFMw.png" /><figcaption>kaggle.com/competitions</figcaption></figure><h3>Google Colab</h3><p><strong>`Google Colab`</strong> is an online platform that enables machine learning engineers and data scientists to collaborate on projects. It provides a convenient way to share notebooks, view and edit them, and is offered by Google’s Research team.</p><p><a href="https://colab.research.google.com/">Google Colab</a></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*JHmbbbovFh1WcBg3ABcz9g.png" /><figcaption>my google colab</figcaption></figure><blockquote>Let’s get right back to the real deal!!!</blockquote><h3>Import Datasets Directly from Kaggle to Google Colab</h3><ol><li><strong>Install Opendataset Library</strong>: To install the opendataset library, you can use the pip package manager by running the following command: pip install opendataset. This will download and install the opendataset library, which you can then use to easily download datasets from Kaggle.</li></ol><p><a href="https://pypi.org/project/opendatasets/">opendatasets</a></p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/7d7e38eadc34de4846c9bdc7ffb1c6b5/href">https://medium.com/media/7d7e38eadc34de4846c9bdc7ffb1c6b5/href</a></iframe><p>2.<strong> Create a Kaggle API token</strong>: Download your Kaggle API token. The Kaggle API token is a simple file containing your username and key. You will be required by the open dataset to download a dataset.</p><ul><li><strong>Log into your Kaggle account.</strong></li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/940/1*PIF1_-fsKfr2kTujdaYxDg.png" /></figure><ul><li><strong>Click on your profile image</strong></li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*5Xz7Z28W0WpZBC7bKDpbKw.png" /></figure><ul><li><strong>Click on the account menu, and scroll downwards.</strong></li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*xjYOV4Hy_PT77zxTwCIXwA.png" /></figure><ul><li><strong>Click on the “Create New API Token”, it downloads a JSON file</strong></li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*kPL58NcLvdPTjNUAErKyLg.png" /></figure><ul><li><strong>Open the “kaggle.json” file with your Notepad app. Do not share your key with anyone, it’s private.</strong></li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*cRhgyOielaZGJ_DCCuIoRg.png" /></figure><p><strong>3. Connect to Dataset </strong>— Copy and paste the link of your dataset in place of “<em>https://www.kaggle.com/datasets/gaurav2022/mobile-health</em>”.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/632f0fad0c83c3c56c371c2b85f7c023/href">https://medium.com/media/632f0fad0c83c3c56c371c2b85f7c023/href</a></iframe><p>After, running this code, this part pops up. The next step is to fill in your username and key from the JSON file. It starts downloading almost immediately.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/886/1*vCTABadOSc3RDHr0aFUMMA.png" /></figure><p>The file is downloaded directly to your Google Colab.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/921/1*szk4YkolaW-Z80zEtBdTsg.png" /></figure><p>Yayy! You made it to the end</p><iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Ftenor.com%2Fembed%2F3477855&amp;display_name=Tenor&amp;url=https%3A%2F%2Ftenor.com%2Fview%2Fdespicable-me-agnes-cute-minions-excited-gif-3477855&amp;image=https%3A%2F%2Fmedia.tenor.com%2FW3mz5gyVm60AAAAC%2Fdespicable-me-agnes.gif&amp;key=a19fcc184b9711e1b4764040d3dc5c07&amp;type=text%2Fhtml&amp;schema=tenor" width="600" height="400" frameborder="0" scrolling="no"><a href="https://medium.com/media/9849e222e11a9152fcd2a107643c8bc6/href">https://medium.com/media/9849e222e11a9152fcd2a107643c8bc6/href</a></iframe><p>Kindly share, comment and give a clap if you find this content valuable.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=96428eab9678" width="1" height="1" alt="">]]></content:encoded>
        </item>
    </channel>
</rss>