<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Manoj Kumal on Medium]]></title>
        <description><![CDATA[Stories by Manoj Kumal on Medium]]></description>
        <link>https://medium.com/@manojkumal?source=rss-115b91c97fe4------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/1*75kFsAE6ttXMcVWMAztf6g.jpeg</url>
            <title>Stories by Manoj Kumal on Medium</title>
            <link>https://medium.com/@manojkumal?source=rss-115b91c97fe4------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Mon, 16 Nov 2026 23:46:35 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@manojkumal/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[How to Read a Paper Without Losing Your Mind]]></title>
            <link>https://medium.com/@manojkumal/how-to-read-a-paper-without-losing-your-mind-ce3ee9baba80?source=rss-115b91c97fe4------2</link>
            <guid isPermaLink="false">https://medium.com/p/ce3ee9baba80</guid>
            <dc:creator><![CDATA[Manoj Kumal]]></dc:creator>
            <pubDate>Mon, 20 Apr 2026 19:21:37 GMT</pubDate>
            <atom:updated>2026-04-20T19:21:37.506Z</atom:updated>
            <content:encoded><![CDATA[<p>As a grad student, one of the most critical skills is learning how to discover strong research ideas. I’ve been going through some valuable references and reflecting on what I’ve learned from them. One paper, in particular, introduces a highly effective way to read research papers called the <strong>three-pass method</strong>.</p><p>Instead of reading a paper line by line in a single go, this approach breaks the process into three stages each with a distinct purpose helping you understand the work more efficiently and deeply.</p><h3>1. First Pass (5–10 minutes): Get the big picture</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*o1uVsNMz9XTDTRuITjIJ3w.jpeg" /><figcaption>Bird-eye‘s View</figcaption></figure><blockquote>Goal: Decide if the paper is worth reading.</blockquote><p><strong>What to do:</strong></p><ul><li>Read <strong>title, abstract, introduction</strong></li><li>Skim <strong>section and sub-section headings</strong></li><li>Look at <strong>figures and conclusions</strong></li><li>Glance at references</li></ul><p>At the end, answer the <strong>“Five C’s”</strong>:</p><ul><li><strong>Category</strong> — What type of paper is it?</li><li><strong>Context</strong> — What other work is it related to?</li><li><strong>Correctness</strong> — Do assumptions seem reasonable?</li><li><strong>Contributions</strong> — What are the main ideas?</li><li><strong>Clarity</strong> — Is it well written?</li></ul><blockquote>Note: If it’s not relevant, stop here.</blockquote><h3>2. Second Pass (30–60 minutes): Understand the content</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*y1mYKBpFVY7j2HwCpBU81g.png" /><figcaption>Focus Like a Monk</figcaption></figure><blockquote>Goal: Understand the paper, but not deeply.</blockquote><p><strong>What to do:</strong></p><ul><li>Read more carefully</li><li>Focus on <strong>figures, graphs, and results</strong></li><li>Ignore heavy details like proofs</li><li>Take notes and mark unclear parts</li><li>Identify important references for later</li></ul><blockquote>After this, you should understand the <strong>main idea and approach</strong>.</blockquote><h3>Third Pass (deep reading): Fully understand it</h3><blockquote>Goal: Master the paper.</blockquote><p>What to do:</p><ul><li>Try to <strong>reconstruct the paper yourself</strong></li><li>Question assumptions and methods</li><li>Think critically about strengths/weaknesses</li></ul><blockquote>After this, you can explain the paper and evaluate it.</blockquote><p>References:</p><p><a href="https://simon.peytonjones.org/great-research-talk/">How to Give a Great Research Talk</a></p><iframe src="https://drive.google.com/viewerng/viewer?url=https%3A//web.stanford.edu/class/ee384m/Handouts/HowtoReadPaper.pdf&amp;embedded=true" width="600" height="780" frameborder="0" scrolling="no"><a href="https://medium.com/media/357e98076bc9ff35b029deeaf7cf26ca/href">https://medium.com/media/357e98076bc9ff35b029deeaf7cf26ca/href</a></iframe><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=ce3ee9baba80" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Few things can improve your complex RAG project having multiple agents.]]></title>
            <link>https://medium.com/@manojkumal/few-things-can-improve-your-complex-rag-project-having-multiple-agents-ad4eb7537cf0?source=rss-115b91c97fe4------2</link>
            <guid isPermaLink="false">https://medium.com/p/ad4eb7537cf0</guid>
            <dc:creator><![CDATA[Manoj Kumal]]></dc:creator>
            <pubDate>Tue, 18 Jun 2024 12:06:30 GMT</pubDate>
            <atom:updated>2024-06-18T12:06:30.184Z</atom:updated>
            <content:encoded><![CDATA[<p>Retrieval Augmented Generation (RAG) is widely used today for creating chatbots across various domains. Optimizing a RAG system is crucial for its success. However, many RAG projects fail to meet expectations. Here are a few key points to understand when working on a RAG project for a specific domain:</p><h3><strong>Domain Knowledge</strong>:</h3><p>Ensure you have a deep understanding of the specific domain you are working on.For example, if you’re creating a chatbot for the medical domain, it needs to respond to medical-related questions. To do this effectively, you should understand the types of questions users might ask and how detailed the answers need to be.</p><p>Knowing this can help you decide how to break down the text into manageable chunks. For instance, if answers are typically found across several pages, you’ll need to create chunks that are large enough to include all necessary information but small enough to be processed efficiently. Understanding these aspects will guide you in designing a chatbot that provides accurate and comprehensive responses, tailored to the needs of users in the medical domain.</p><h3><strong>Quality of Data</strong></h3><p>High-quality and relevant data are essential. The chatbot’s performance heavily depends on the quality of the data it retrieves and generates from. Data must be accurate and up-to-date. Inaccurate information can mislead users and potentially have harmful consequences.</p><p>Data should be consistent in terms of format, terminology, and style. Inconsistencies can confuse the chatbot and result in inconsistent or conflicting responses. Preprocess the data as needed to ensure the retriever can accurately extract relevant information. This may involve tasks such as cleaning the data, standardizing formats, and removing noise or irrelevant content.</p><h3><strong>Effective Retrieval Mechanism</strong></h3><p>Implement an efficient retrieval mechanism to fetch the most relevant information. Poor retrieval can lead to irrelevant or incorrect responses.There are two primary types of vector search methods: sparse vector search and dense vector search. Each has its own characteristics, and combining both can offer significant advantages.</p><h4>Sparse Vector Search</h4><ul><li>Sparse Vectorsearch relies on traditional term-based retrieval methods such as TF-IDF (Term Frequency-Inverse Document Frequency) or BM25.</li><li>Documents and queries are represented as high-dimensional sparse vectors where each dimension corresponds to a term from the vocabulary.</li><li>The presence or absence of terms and their frequency in the documents determine the vector values.</li></ul><h4>Dense Vector Search</h4><ul><li>Dense Vector search uses dense embeddings generated by neural networks, such as BERT or other transformer models.</li><li>Both queries and documents are embedded into continuous low-dimensional vector spaces.</li><li>Similarity is measured using metrics like cosine similarity or Euclidean distance.</li></ul><p>Using both sparse and dense vector search methods in a hybrid approach can leverage the strengths of each, , providing several benefits:</p><ul><li><strong>Enhanced Relevance</strong>: Combines term-based precision with semantic understanding, improving overall retrieval relevance.</li><li><strong>Robustness</strong>: Handles a wider variety of queries, including those requiring exact matches and those needing semantic interpretation.</li><li><strong>Improved Coverage</strong>: Sparse methods can cover rare terms and specific matches, while dense methods cover broader, context-based matches.</li><li><strong>Balanced Efficiency</strong>: Sparse methods can handle the bulk of retrieval quickly, while dense methods can refine and re-rank results for better accuracy.</li></ul><h3>Chat History and Query Refiner:</h3><p>Integrating chat history and a query refiner into a Retrieval Augmented Generation (RAG) system can significantly enhance the user experience by improving the relevance and accuracy of responses, especially in complex and interactive conversations. However, it is essential to balance the benefits with the potential <strong>costs</strong> and <strong>latency</strong> involved.</p><p>Chat history involves keeping track of the conversation context, including previous user queries and system responses.Enables the chatbot to understand follow-up questions better by referring to previous interactions. This leads to more accurate and contextually relevant responses. It is not required to retrieved information from dataset and makes the process a bit faster.</p><p>A query refiner processes user queries to correct mistakes, clarify ambiguities, and enhance query quality before retrieving information. Implementing a query refiner can increase the computational cost and latency of the system, as additional processing is required before the retrieval stage. In our case, it is adding marginal improvement so we don’t use query refiner.</p><h3><strong>Generation </strong>Quality</h3><p>Focus on the generation aspect of RAG to ensure that the responses are coherent, contextually accurate, and useful.</p><h3>Evaluation</h3><p>Regularly evaluate and test the system to identify and fix issues. This helps in improving the system’s performance and reliability over time. <a href="https://docs.ragas.io/en/stable/">Ragas</a> is a framework that helps you evaluate your Retrieval Augmented Generation (RAG) pipelines. RAG denotes a class of LLM applications that use external data to augment the LLM’s context. There are existing tools and frameworks that help you build these pipelines but evaluating it and quantifying your pipeline performance can be hard. This is where Ragas (RAG Assessment) comes in. You can generate a dataset and stored in <a href="https://langfuse.com/docs/prompts/example-langchain">langufse</a> and evaluate.</p><ol><li>Answer Relevance</li><li>Context Recall</li><li>Context Precision</li><li>Context Relevency</li><li>Context entities Recall</li></ol><h3>Experiement with following things:</h3><h4>Craft the prompt</h4><p>Based on the promblem statement you’re trying to solve, make a prompt for the system. It is also one of the important step and if you didn’t get the as expected output then try to modify the system prompt.</p><h4>Chunk size</h4><p>Choosing an optimal chunk size is crucial for accurately fetching information and generating a correct response. You can start experimenting with chunk sizes from 256 up to 500. Before conducting these experiments, it is important to first integrate the evaluation pipeline using RAGAS. This process ensures that the system is set up correctly to assess the performance of different chunk sizes, enabling the identification of the most effective one for accurate information retrieval and response generation. There are different types of chunking methods such token and character based.</p><h4>Embedding Model</h4><p>Experimenting with different embedding models from the Hugging Face dashboard is essential to enhance the performance of the Retrieval-Augmented Generation (RAG) system</p><h4>Reranker Model</h4><p>When exploring different reranker models, it’s important to consider their impact on <strong>latency</strong>. Selecting an optimal <strong>reranker</strong> <strong>top-k</strong> and <strong>retriever top-k</strong> value can help manage this latency while still improving the performance of the system. By experimenting with various reranker models, you can identify the most effective one for refining the retrieval and ranking process in the system. I have experiemented with following reranker models.</p><ul><li>Cohere reranker</li><li>Cross-encoder reranker</li><li>Flash reranker</li></ul><h3><strong>References:</strong></h3><ul><li><a href="https://www.langchain.com/">LangChain</a></li><li><a href="https://docs.ragas.io/en/stable/">Ragas</a></li><li><a href="https://langfuse.com/">Langfuse</a></li></ul><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=ad4eb7537cf0" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Question Answering over Documents using ️LangChain, Streamlit, Ollama and Pinecone [RAG]]]></title>
            <link>https://blog.tai.com.np/question-answering-over-documents-using-%EF%B8%8Flangchain-streamlit-ollama-and-pinecone-rag-76635e80da13?source=rss-115b91c97fe4------2</link>
            <guid isPermaLink="false">https://medium.com/p/76635e80da13</guid>
            <dc:creator><![CDATA[Manoj Kumal]]></dc:creator>
            <pubDate>Tue, 09 Apr 2024 05:06:49 GMT</pubDate>
            <atom:updated>2024-04-11T11:40:13.782Z</atom:updated>
            <content:encoded><![CDATA[<h3>Introduction</h3><p>The implementation of Gen AI Retrieval Augmented Generation (RAG) merges the capabilities of information retrieval with those of existing Large Language Models (LLMs). RAG guarantees that the generated output is firmly rooted in the provided knowledge base, tailored to the specific use cases it aims to address.</p><blockquote>“RAG (Retrieval-Augmented Generation) can be applied in various use cases or scenarios”</blockquote><p>The situation becomes significantly concerning when we depend on language generation capabilities for critical applications like <strong>summarizing medical records, managing customer support interactions, conducting financial analysis, and offering legal advice.</strong> Even minor inaccuracies could have serious consequences, exposing the limitations of Large Language Models (LLMs) despite their advancements in self-learning and the persistent challenge of <strong>hallucination</strong>, where models generate text that lacks grounding in reality.</p><h4>What is “hallucinations” problem in LLM?</h4><p>The term “hallucination” in the context of Large Language Models (LLMs) refers to situations where the model generates text that is not grounded in reality or lacks factual accuracy.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/784/0*MKfnAdFHhENHm3RP" /><figcaption>Figure: Example of hallucination</figcaption></figure><p>One of the contributing factors to hallucination is the model’s limited grasp of <strong>contextual understanding</strong>.</p><h3><strong>Taxonomy of hallucination mitigation techniques in LLMs</strong></h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*AsTye5_PPdXlFxkE04ME2Q.png" /><figcaption>source: <a href="https://arxiv.org/abs/2401.01313">research paper</a></figcaption></figure><p>As evident from the image above, numerous techniques exist to address the hallucination issue, with one notable approach being RAG and based on various use cases, RAG demonstrates its utility effectively.</p><p>First let’s get familiar with the terms</p><p><a href="https://python.langchain.com/docs/get_started/quickstart/"><strong>LangChain</strong></a>: It is an open-source framework designed specifically to help developers build applications that use large language models (LLMs). In simpler terms, it provides a toolkit for programmers to easily incorporate LLMs into their applications.</p><p>Here are some key aspects of LangChain:-</p><ul><li><a href="https://python.langchain.com/docs/integrations/llms/"><strong>Simplifies LLM Integration</strong></a><strong>:</strong> LangChain provides a standard interface for interacting with various LLMs, regardless of the underlying provider. This makes it easier for developers to switch between different LLM services without having to rewrite code.</li><li><strong>Building Block Components:</strong> LangChain offers a set of pre-built components that handle common LLM tasks. These components can be combined to create complex workflows, like building a chatbot or a question-answering system.</li></ul><p><a href="https://ollama.com/"><strong>Ollama</strong></a>: Ollama is a tool that allows you to run large language models (LLMs) on your own computer, instead of relying on a cloud service. This can be beneficial for a few reasons such as <strong>cost</strong>, <strong>privacy</strong> and <strong>customization</strong>.</p><p>Here’s a breakdown of how Ollama works:-</p><ul><li><strong>Local Execution:</strong> Ollama runs LLMs directly on your computer’s hardware, typically the graphics processing unit (GPU) for faster processing.</li><li><strong>Package Management:</strong> Ollama uses a concept called “Modelfiles” which bundle the LLM’s weights, configuration, and data into a single package. This simplifies setting up and running different LLM models.</li><li><strong>Multiple Access Methods:</strong> Ollama offers different ways to interact with the LLMs. You can directly type prompts and queries to the LLM in a command-line interface.</li></ul><p><a href="https://www.pinecone.io/"><strong>Pinecone</strong></a>:It is a cloud-based vector database designed for swift and effortless storage and retrieval of data via a straightforward API. It facilitates seamless integration of long-term memory into high-performance AI applications.</p><p>To leverage Pinecone’s power within your AI project, consider using the Langchain framework. Langchain provides various <a href="https://python.langchain.com/docs/modules/data_connection/document_loaders/"><strong>loaders</strong></a> specifically designed for different document types, including webpages, PDFs, CSVs, HTML, JSON, and even unstructured data like YouTube videos. These loaders streamline the process of integrating Pinecone into your workflow, allowing you to effortlessly store and retrieve vector representations of your data.</p><h3>Retrieval Augumented Generation(RAG):</h3><p>Here are the steps involved in the process:-</p><ol><li><strong>Loading</strong>: Initially, data loading is essential. Unstructured data can originate from various sources. Utilize the <a href="https://integrations.langchain.com/">LangChain integration hub</a> to access a comprehensive range of <a href="https://python.langchain.com/docs/modules/data_connection/document_loaders/">loaders</a>. Each loader furnishes data in the form of a LangChain Documents.</li><li><strong>Splitting</strong>: The wholeDocuments breakdown into chunks of specified size using <a href="https://python.langchain.com/docs/modules/data_connection/document_transformers/">Text splitters</a>.</li><li><strong>Storage</strong>: We need somewhere to store and index our splits, so that they can later be searched over. This is often done using a <a href="https://python.langchain.com/docs/modules/data_connection/vectorstores/">VectorStore</a> and <a href="https://python.langchain.com/docs/modules/data_connection/text_embedding/">Embeddings</a> model.</li></ol><figure><img alt="" src="https://cdn-images-1.medium.com/max/932/1*K60GVWovPNPLVINi_R68Sw.png" /><figcaption>source: <a href="https://python.langchain.com/docs/use_cases/question_answering/">LangChain Docs</a></figcaption></figure><p>4. <strong>Retrieval</strong>: Given a user input, relevant splits are retrieved from storage using a <a href="https://python.langchain.com/docs/modules/data_connection/retrievers/">Retriever</a>.</p><p>5. <strong>Generation</strong>: A <a href="https://python.langchain.com/docs/modules/model_io/chat/">ChatModel</a> / <a href="https://python.langchain.com/docs/modules/model_io/llms/">LLM</a> produces an answer using a prompt that includes the question and the retrieved data.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/879/1*YRZIBtl61Y5enLvPuRWCtg.png" /><figcaption>source: <a href="https://python.langchain.com/docs/use_cases/question_answering/">LangChain Docs</a></figcaption></figure><p>6. <strong>Conversation</strong> : Hold a multi-turn conversation by adding <a href="https://python.langchain.com/docs/modules/memory/">Memory</a> to your QA chain.</p><p><strong>Let’s Start:</strong></p><p>Before using Pinecone, it is necessary to create a free account by visiting the provided <a href="https://www.pinecone.io/">link</a> and generating API keys.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*YF3xhdV_s0UmQ5trVfS2bg.png" /></figure><h3>Data</h3><p>In this project, I am utilizing <a href="https://www.moljpa.gov.np/en/wp-content/uploads/2018/12/Civil-code.pdf">PDF</a> documents of The National Civil (Code) Act, 2017 (2074), provided by the Ministry of Law, Justice, and Parliamentary Affairs.</p><h3><strong>Loading</strong></h3><p>As I’m working with PDF documents, I’ll be utilizing a document loader. However, you can employ various <a href="https://python.langchain.com/docs/modules/data_connection/document_loaders/">loader</a> depending on your document format, including .doc, .txt, .csv, and more.</p><pre>from langchain.document_loaders import DirectoryLoader<br><br>def load_docs(directory):<br>  loader = DirectoryLoader(directory)<br>  documents = loader.load()<br>  return documents<br><br>directory = &#39;/data&#39;<br>documents = load_docs(directory)</pre><h3>Splitting</h3><p>To prevent exceeding the context window of the LLM, we must split the data into smaller chunks before embedding and storing them in a vector store. We’ll use the RecursiveCharacterTextSplitter for this purpose, but depending on your needs, you may need to explore other splitters. Experimenting with parameters like chunk size and overlap is encouraged.</p><pre>from langchain.text_splitter import RecursiveCharacterTextSplitter<br><br>def split_docs(documents,chunk_size=500,chunk_overlap=20):<br>  text_splitter = RecursiveCharacterTextSplitter(chunk_size=chunk_size, chunk_overlap=chunk_overlap)<br>  docs = text_splitter.split_documents(documents)<br>  return docs<br><br>docs = split_docs(documents)</pre><h3>Storing</h3><p>To facilitate document split retrieval, we must initially store them for future reference. Typically, this involves embedding each document’s content and storing both the embedding and the document in a vector store, where the embedding serves as the document’s index. While we’ll utilize Pinecone as our vector store and SentenceTransformerEmbeddings all-MiniLM-L6-v2 as the embedding model, I recommend exploring alternative embeddings and vector stores as well.</p><p>In the Pinecone console, establish an index and configure the dimensions to 384 for SentenceTransformerEmbeddings(dimensions may vary depending on the chosen embeddings model).</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*U7ezu8Uk2YKPw0QOXhWK6g.png" /></figure><p>Initialise the Pinecone client and the embedding model.</p><pre>import pinecone<br>from langchain.embeddings import SentenceTransformerEmbeddings<br><br>embeddings = SentenceTransformerEmbeddings(model_name=&quot;all-MiniLM-L6-v2&quot;)<br><br>pinecone.init(<br>    api_key=&quot;ab36be90-6b59-42ad-b5ae-6b0933cc5cce&quot;,<br>    environment=&quot;gcp-starter&quot;<br>)<br>index_name=&quot;vectordb&quot;</pre><p>Initialise LLM (Here, I’m using Open sourceMistral )</p><p>Before using Mistral Model from Ollama. you must first <a href="https://ollama.com/">download</a> Ollama onto your local computer.</p><pre># download ollama onto your local computer <br>ollama run Mistral # run which model you want to use.</pre><pre>from langchain_community.llms import Ollama<br><br>llm=Ollama(model=&quot;mistral&quot;)</pre><p>I’ve uploaded both the embedding model and the data to the Pinecone index using the from_documents function. However, you&#39;re free to use other available methods as per your requirements.</p><pre>from langchain.vectorstores import Pinecone<br><br>index=Pinecone.from_documents(docs,embeddings,index_name=index_name)</pre><h3>Retrieval</h3><p>In the realm of similarity search for retrieving relevant splits for any query, vector stores stand out as a popular choice. Within the arsenal of retrievers in LangChain lies a diverse array of options, of which vector stores are just one facet. Each retriever adheres to a standardized method, exemplified by get_relevant_documents() and its asynchronous counterpart aget_relevant_documents().</p><pre>retriever = vectordb.as_retriever()<br><br>def retrieve_query(query,k=2):<br>    matching_results=index.similarity_search(query,k=k)<br>    return matching_results<br><br>retrieve_query(query)</pre><h3>Generation and Conversation</h3><p>In order to facilitate a conversational interface while leveraging document-based responses, we’ll adopt the ConversationalRetrievalChain instead of the standard Retrieval Chain. This allows us to maintain context across multiple exchanges by introducing <a href="https://python.langchain.com/docs/modules/memory/adding_memory/">memory</a> storage through ConversationBufferMemory. By passing this memory as an argument to the ConversationalRetrievalChian, we enable the system to retain and reference previous conversations, enhancing the overall coherence and relevance of responses.</p><p>Additionally, the ConversationalRetrievalChain offers various extensions, such as streaming and source document integration, further augmenting its capabilities. As with any tool in this toolkit, there’s ample room for experimentation and customization. Different prompts and prompt templates can be utilized to tailor each chain to specific requirements and preferences, opening up a realm of possibilities for optimization and fine-tuning.</p><pre>from langchain.chains import ConversationChain<br>from langchain.chains.conversation.memory import ConversationBufferWindowMemory<br><br>if &#39;buffer_memory&#39; not in st.session_state:<br>            st.session_state.buffer_memory=ConversationBufferWindowMemory(k=3,return_messages=True)<br><br><br>system_msg_template = SystemMessagePromptTemplate.from_template(template=&quot;&quot;&quot;Answer the question as truthfully as possible using the provided context, <br>and if the answer is not contained within the text below, say &#39;I don&#39;t know&#39;&quot;&quot;&quot;)<br><br><br>human_msg_template = HumanMessagePromptTemplate.from_template(template=&quot;{input}&quot;)<br><br>prompt_template = ChatPromptTemplate.from_messages([system_msg_template, MessagesPlaceholder(variable_name=&quot;history&quot;), human_msg_template])<br><br>conversation = ConversationChain(memory=st.session_state.buffer_memory, prompt=prompt_template, llm=llm, verbose=True)</pre><p>Employing memory within the system facilitates continuity in conversations, enabling the ability to reference past interactions and context. This allows for the generation of follow-up questions that build upon previous discussions, such as asking for further explanation with prompts like “Can you elaborate on that?” This capability enhances the natural flow of conversation and enables a deeper exploration of topics by leveraging prior context.</p><p>And now we’re ready to make inferences using our chain</p><pre>response = conversation.predict(input=f&quot;Context:\n {context} \n\n Query:\n{query}&quot;)</pre><h3>Query Refiner</h3><p>The query_refiner function is designed to improve the user query by generating a more refined version based on the context provided by a conversation history. It plays a crucial role in improving the effectiveness of conversational interfaces by refining user queries based on contextual information from past interaction.</p><pre>def query_refiner(conversation, query):<br><br>    response = openai.Completion.create(<br>    model=&quot;text-davinci-003&quot;,<br>    prompt=f&quot;Given the following user query and conversation log, formulate a question that would be the most relevant to provide the user with an answer from a knowledge base.\n\nCONVERSATION LOG: \n{conversation}\n\nQuery: {query}\n\nRefined Query:&quot;,<br>    temperature=0.7,<br>    max_tokens=256,<br>    top_p=1,<br>    frequency_penalty=0,<br>    presence_penalty=0<br>    )<br>    return response[&#39;choices&#39;][0][&#39;text&#39;]</pre><h3>Application</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/772/1*ijxaAV1ytvS5IpJC3tZwnQ.png" /></figure><h3>Conclusion:</h3><p>RAG (Retrieval-Augmented Generation) serves multiple purposes. It used to address hallucination issues in language model (LLM) outputs, it has evolved to leverage LLMs’ diverse capabilities, akin to performing Google-like searches on documents. This adaptability renders RAG highly valuable, particularly in dynamic data environments where information undergoes frequent changes.</p><h3>References:</h3><ul><li><a href="https://www.langchain.com/">LangChain</a></li><li><a href="https://www.pinecone.io/">Pinecone</a></li><li><a href="https://ollama.com/">Ollama</a></li></ul><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=76635e80da13" width="1" height="1" alt=""><hr><p><a href="https://blog.tai.com.np/question-answering-over-documents-using-%EF%B8%8Flangchain-streamlit-ollama-and-pinecone-rag-76635e80da13">Question Answering over Documents using 🦜️LangChain, Streamlit, Ollama and Pinecone [RAG]</a> was originally published in <a href="https://blog.tai.com.np/">TAI Blog</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[The Era of 1-Bit LLM]]></title>
            <link>https://medium.com/@manojkumal/the-era-of-1-bit-llm-9a7bd27761c6?source=rss-115b91c97fe4------2</link>
            <guid isPermaLink="false">https://medium.com/p/9a7bd27761c6</guid>
            <dc:creator><![CDATA[Manoj Kumal]]></dc:creator>
            <pubDate>Sun, 10 Mar 2024 11:53:05 GMT</pubDate>
            <atom:updated>2024-03-10T11:53:05.177Z</atom:updated>
            <content:encoded><![CDATA[<h3>Introduction</h3><p>The growing size of big language models has made it hard to use them and raised worries about their environmental impact due to high energy use. New research, like BitNet, is leading to a new type of language model called 1-bit Large Language Models (LLMs). BitNet b1.58 is one such model, where every part of the model uses only three values: -1, 0, or 1. It performs just as well as traditional models but is much cheaper in terms of speed, memory, processing power, and energy. This new type of model sets a new standard for creating efficient and powerful language models. It also allows for new ways of doing calculations and opens up opportunities for making specialized hardware for these models.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/947/1*0bLSvUlczoXf4C9W_zRc_A.png" /><figcaption><a href="https://arxiv.org/pdf/2402.17764.pdf"><strong>Image Source</strong></a></figcaption></figure><h3>The Era of 1-bit LLMs</h3><p>What is 1-bit LLM?</p><p>A 1-bit Large Language Model (LLM) is a type of language model where each parameter or weight of the model is represented using only one bit of information. In other words, each parameter can only take on one of two values: either -1 or 1. This highly compressed representation significantly reduces the memory and computational requirements of the model compared to traditional models, which typically use 16-bit floating-point values (such as FP16 or BF16). Despite the drastic reduction in precision, 1-bit LLMs have shown promising performance in natural language processing tasks, making them an attractive option for deployment in resource-constrained environments and for reducing the environmental and economic impact associated with large language models.</p><ol><li>Large Language Models (LLMs) have grown significantly in size and capabilities, leading to concerns about their environmental and economic impact due to high energy consumption.</li><li>Post-training quantization is one approach to address these challenges, reducing precision to lower bits for inference and lowering memory and computational requirements.</li><li>Recent research on 1-bit model architectures, like <strong>BitNet</strong>, shows promise in reducing the cost of LLMs while maintaining performance.</li><li>BitNet b1.58 is introduced as a 1-bit LLM variant where every parameter is ternary {-1, 0, 1}, offering benefits such as reduced energy consumption, memory footprint, and improved throughput and latency compared to full-precision models.</li><li>BitNet b1.58 retains the benefits of the original BitNet while adding support for feature filtering and matching full-precision baselines in terms of perplexity and end-task performance.</li></ol><h3>Recap of BitNet</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/980/1*XaGwJOrjtkSykas-JbWZXQ.png" /><figcaption><a href="https://arxiv.org/pdf/2310.11453.pdf"><strong>Image Source</strong></a></figcaption></figure><p>BitNet uses the same layout as Transformers, stacking blocks of self-attention and feed-forward networks. Compared with vanilla Transformer, BitNet uses BitLinear instead of conventional matrix multiplication, which employs binarized (i.e., 1-bit) model weights. It leave the other components high-precision, e.g., 8-bit in experiments. The reasons as follows. First, the residual connections and the layer normalization contribute negligible computation costs to large language models. Second, the computation cost of QKV transformation is much smaller than the parametric projection as the model grows larger. Third, we preserve the precision for the input/output embedding because the language models have to use high-precision probabilities to perform sampling.</p><h3>Methodology</h3><h4>BitNet b1.58</h4><p>BitNet b1.58 is based on the BitNet architecture, which is a Transformer that replaces nn.Linear with BitLinear. It is trained from scratch, with 1.58-bit weights and 8-bit activations. Compared to the original BitNet, it introduces some modifications.</p><h4>Quantization Function</h4><p>To constrain the weights to -1, 0, or +1, we adopt an absmean quantization function. It first scales the weight matrix by its average absolute value, and then round each value to the nearest integer among {-1, 0, +1}</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/555/1*LzE7MQelbbGxdZYXRwncxg.png" /><figcaption><a href="https://arxiv.org/pdf/2402.17764.pdf">Image Source</a></figcaption></figure><p>The way it handle activation quantization in BitNet is similar, except it don’t scale the activations before applying non-linear functions. Instead, it scale all activations to a range of -Qb to Qb per token, which eliminates the need for zero-point quantization. This makes implementation and system-level optimization easier and simpler without significantly affecting performance in experiments.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/913/1*p_eTn_-dY0BNnjMgjDK9Fg.png" /><figcaption><a href="https://arxiv.org/pdf/2402.17764.pdf">Image Source</a></figcaption></figure><h3>Results</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/952/1*2NB18bZE0TGciU2Y5lYjiQ.png" /><figcaption><a href="https://arxiv.org/pdf/2402.17764.pdf">Image Source</a></figcaption></figure><p>The performance gap between BitNet b1.58 and LLaMA LLM decreases as model size increases. Importantly, BitNet b1.58 matches the performance of full precision baselines starting from a 3B size. BitNet b1.58 3.9B outperforms LLaMA LLM 3B with lower memory and latency costs, showing it’s a better choice compared to current LLM models.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=9a7bd27761c6" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Summary of different prompt engineering techniques in LLM based on applications.]]></title>
            <link>https://medium.com/@manojkumal/summary-of-different-prompt-engineering-techniques-in-llm-based-on-applications-95b43c361422?source=rss-115b91c97fe4------2</link>
            <guid isPermaLink="false">https://medium.com/p/95b43c361422</guid>
            <dc:creator><![CDATA[Manoj Kumal]]></dc:creator>
            <pubDate>Sat, 17 Feb 2024 13:24:57 GMT</pubDate>
            <atom:updated>2024-02-17T13:24:57.469Z</atom:updated>
            <content:encoded><![CDATA[<h3>Abstract</h3><p>Prompt engineering is a vital technique for boosting the abilities of large language and vision-language models. It works by giving task-specific instructions, called prompts, to the model without changing its core parameters. Instead of tweaking the model itself, prompts direct it on how to behave for a particular task. These prompts can be simple language instructions or learned vectors that activate relevant knowledge, making it easier to use pre-trained models for different tasks.</p><h3>Introduction</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/692/1*_b0TTCLhOvsSNCZDANd1sQ.png" /><figcaption>Figure 1: Visual breakdown of prompt engineering components:<br>LLMs trained on extensive data, instruction and context as pivotal<br>elements shaping the prompt, and a user input interface(<a href="https://arxiv.org/abs/2402.07927">Image source</a>)</figcaption></figure><p>Prompt engineering is a pivotal technique for enhancing the capabilities of pre-trained large language models (LLMs) and vision-language models (VLMs). It involves crafting task-specific instructions, known as prompts, to guide model output without altering parameters. This approach revolutionizes the adaptability of LLMs and VLMs, allowing them to excel across various tasks and domains without the need for extensive retraining. The transformative promise of prompt engineering is evident in its ability to steer model responses, enhancing adaptability and applicability across diverse sectors. With ongoing research revealing innovative approaches, there is a pressing need for a comprehensive survey to understand the landscape of contemporary prompt engineering techniques.</p><h3>Prompt Engineering</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/996/1*j5DdikOnsTTzo3i1pIE6LQ.png" /><figcaption>Figure 2: Prompt engineering techniques in LLMs(<a href="https://arxiv.org/abs/2402.07927">Image source</a>)</figcaption></figure><h3>New Tasks Without Extensive Training</h3><ol><li>Zero-Shot Prompting</li></ol><p>Few-shot prompting capitalizes on the potential of large LLMs by requiring minimal training data. By employing carefully constructed prompts, this technique guides the model to tackle new tasks without extensive training. The model utilizes its pre-existing knowledge to make predictions based on the provided task description, thereby reducing the need for labeled data.</p><p>Here’s an example prompt that ran into GPT 4.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/421/1*qMO3JOjEZLzYFrdn_c3nMQ.png" /><figcaption>Figure 3: Prompt</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/288/1*jyfQPyYBTmc9E-_lDfHONA.png" /><figcaption>Figure 4: Result</figcaption></figure><p>2. Few-shot Prompting</p><p>Few-shot prompting provides models with a few input-output examples to induce an understanding of a given task, unlike zero-shot prompting, where no examples are supplied.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/600/1*hfzlAO04B4LFbuDWHoGmjQ.png" /><figcaption>Figure 5: Prompt</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/289/1*B65lGcFwddSaswI0-62Ngw.png" /><figcaption>Figure 6: result</figcaption></figure><h3>Reasoning and Logic</h3><ol><li>Chain-of-Thought (CoT) Prompting</li></ol><p>Chain-of-Thought (CoT) prompting introduces a method to prompt LLMs, fostering coherent and step-by-step reasoning. This technique stands out for its capacity to elicit more structured and thoughtful responses compared to conventional prompts. Through experiments, CoT prompting demonstrates its ability to guide LLMs through logical reasoning chains, resulting in deeper comprehension of prompts and more accurate responses, akin to human problem-solving processes.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/802/1*AZhw9yuH1gnvt7DStkjbPA.png" /><figcaption>Figure 7: Chain of Thought prompting(<a href="https://arxiv.org/pdf/2201.11903.pdf">Image source</a>)</figcaption></figure><p>2. Automatic Chain-of-Thought (Auto-CoT) Prompting</p><p>Auto-CoT presents an automated approach to instructing LLMs with step-by-step prompts, eliminating the need for manual creation. It enhances robustness by diversely sampling questions and generating multiple reasoning chains for each. This diverse sampling minimizes errors and improves few-shot learning performance, surpassing the traditional CoT paradigm in accuracy improvements on arithmetic and symbolic tasks.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/967/1*KJwVNTNcsqiMiktkxg7PEw.png" /><figcaption>Figure 8: Automatic Chain-of-Thought (Auto-CoT) Prompting(<a href="https://arxiv.org/pdf/2210.03493.pdf">Image source</a>)</figcaption></figure><p>3. Self-Consistency</p><p>Self-consistency, a decoding strategy introduced in CoT prompting, enhances reasoning performance by generating diverse reasoning chains for complex tasks. By sampling from the model’s decoder and identifying the most consistent final answer, it improves accuracy significantly across various benchmarks. This approach capitalizes on the observation that problems requiring thoughtful analysis benefit from greater reasoning diversity, resulting in substantial accuracy improvements compared to baseline CoT prompting.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*wCxZbZCWB2B_M8Z_3LYc1Q.png" /><figcaption>Figure 9: self-consistency(<a href="https://arxiv.org/pdf/2203.11171.pdf">Image source</a>)</figcaption></figure><p>4. Logical Chain-of-Thought (LogiCoT) Prompting</p><p>LogiCoT prompting introduces a neurosymbolic framework to enhance reasoning in LLMs by incorporating principles from symbolic logic. It employs reductio ad absurdum to verify each step of reasoning and provide targeted feedback, reducing logical errors and hallucinations. Experimentation with Vicuna-33b and GPT-4 demonstrates significant improvements over CoT prompting, particularly on datasets like GSM8K and AQuA.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/726/1*1j0SywSfJ3sdKrFEhfQsPQ.png" /><figcaption>Figure 10:Logical Chain-of-Thought (LogiCoT) Prompting(<a href="https://arxiv.org/pdf/2309.13339.pdf">Image source</a>)</figcaption></figure><p>5. Chain-of-Symbol (CoS) Prompting</p><p>CoS as a solution to LLMs’ struggle with spatial tasks, employing condensed symbols for clearer prompts and improved reasoning. Although CoS enhances ChatGPT’s performance significantly, challenges remain in scalability, generalizability, and interpretability of LLM reasoning based on symbols. Notably, CoS achieves up to 92.6% accuracy on Brick World tasks and reduces prompt tokens by up to 65.8%, streamlining the process while maintaining high accuracy.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/683/1*og_h1i_JWs7A6rBeD4ETGg.png" /><figcaption>Figure 11: Chain-of-Thought (CoT) Vs Chain-of-Symbol (COS)(<a href="https://arxiv.org/pdf/2305.10276.pdf">Image source</a>)</figcaption></figure><p>6. Tree-of-Thoughts (ToT) Prompting</p><p>The Tree-of-Thoughts (ToT) framework enhances prompting capabilities for tasks requiring exploration and look-ahead reasoning. It extends CoT prompting by organizing intermediate reasoning steps into a coherent tree structure called “thoughts,” enabling deliberate reasoning and systematic exploration. By integrating model abilities with search algorithms, such as breadth-first or depth-first search, ToT achieved remarkable success rates of 74% in Game of 24 tasks and 60% in word-level tasks, surpassing CoT’s performance significantly.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/877/1*dxTi-4xEN_oBCL4HEoc7kg.png" /><figcaption>Figure 12: Tree-of-Thoughts(<a href="https://arxiv.org/pdf/2305.10601.pdf">Image source</a>)</figcaption></figure><p>7. Graph-of-Thoughts (GoT) Prompting</p><p>A graph-based framework addressing the non-linear nature of human thought processes. This approach allows dynamic interaction, backtracking, and aggregation of ideas, departing from the linear structure of traditional sequential methods like CoT prompting. By modeling reasoning as a directed graph and offering modular architecture, GoT significantly enhances model capabilities, demonstrating notable accuracy improvements over CoT and state-of-the-art methods in tasks such as GSM8K and ScienceQA.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*gqNnZCNi9tIfjqjSMF-7wg.png" /><figcaption>Figure 13: Graph-of-Thoughts (GoT) Prompting(I<a href="https://arxiv.org/abs/2305.16582">mage source</a>)</figcaption></figure><h3>Reduce Hallucination</h3><ol><li>Retrieval Augmented Generation (RAG)</li></ol><p>RAG (Retrieval Augmented Generation) integrates information retrieval into text generation, overcoming the limitations of static training data in LLMs. By analyzing user input and querying a knowledge base, RAG enriches prompts with contextual background, enabling accurate and creative responses. It outperforms traditional models on ODQA benchmarks like TriviaQA and Natural Questions, achieving exact match scores of up to 56.8% and 44.5%, respectively, marking a significant advancement in text generation tasks requiring external knowledge.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/995/1*nO0LULD185HJdeMV59QlEA.png" /><figcaption>Figure 14: RAG(<a href="https://proceedings.neurips.cc/paper/2020/file/6b493230205f780e1bc26945df7481e5-Paper.pdf">Image source</a>)</figcaption></figure><p>2. ReAct Prompting</p><p>ReAct is based on the human ability to blend reasoning and action for learning and decision-making. While existing methods like Chain-of-Thought (CoT) prompting show promise, they have limitations such as being unable to access external information, leading to inaccuracies and error propagation. ReAct addresses these issues by integrating reasoning and action within Large Language Models (LLMs), allowing for dynamic reasoning and interaction with external sources like Wikipedia. This framework enables LLMs to generate verbal reasoning paths and specific actions, leading to more accurate and adaptable decision-making processes.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/707/1*5hu2_brYY6Mf4mYXo96igg.png" /><figcaption>Figure 15: ReAct Prompting(<a href="https://arxiv.org/abs/2210.03629">Image source</a>)</figcaption></figure><p>3. Chain-of-Verification (CoVe) Prompting</p><p>CoVe is a four-step process designed to address hallucinations in Large Language Models (LLMs). It involves generating baseline responses, planning verification questions, answering them independently, and revising responses accordingly. By emulating human verification, CoVe enhances logical reasoning abilities and reduces errors, even with contradictory information. Experimental results across various tasks show that CoVe effectively decreases hallucinations while maintaining factual accuracy by helping models identify and correct inaccuracies through focused verification questions.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/958/1*tGSY8usCgfBf8yncnqqeRg.png" /><figcaption>Figure 16: CoV prompting(I<a href="https://arxiv.org/pdf/2309.11495.pdf">mage source</a>)</figcaption></figure><p>4. Chain-of-Note (CoN) Prompting</p><p>A novel approach enhances Retrieval-augmented Language Models (RALMs) by addressing challenges with external knowledge reliability. CoN systematically evaluates document relevance, filtering out noisy and irrelevant content to improve response precision. Testing across various question-answering datasets shows significant improvements, including higher exact match scores and enhanced rejection rates for questions beyond pre-training knowledge.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/819/1*nnJ-TT9USVTSpbh4aTsTyw.png" /><figcaption>Figure 17: Chain-of-Note(I<a href="https://arxiv.org/pdf/2311.09210.pdf">mage source</a>)</figcaption></figure><h3>References:</h3><p><a href="https://arxiv.org/pdf/2402.07927.pdf">A Systematic Survey of Prompt Engineering in Large Language Models: Techniques and Applications</a>.</p><p>More coming soon!!</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=95b43c361422" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Sliding Window Attention]]></title>
            <link>https://medium.com/@manojkumal/sliding-window-attention-565f963a1ffd?source=rss-115b91c97fe4------2</link>
            <guid isPermaLink="false">https://medium.com/p/565f963a1ffd</guid>
            <dc:creator><![CDATA[Manoj Kumal]]></dc:creator>
            <pubDate>Tue, 06 Feb 2024 04:11:06 GMT</pubDate>
            <atom:updated>2024-02-09T09:00:30.371Z</atom:updated>
            <content:encoded><![CDATA[<p>Before we jump into sliding window attention, let’s review.</p><ol><li>Self-Attention</li><li>Receptive Field</li></ol><h3>Self-Attention</h3><p>Self-Attention allows the model to relate words to each other. Imagine we have the following sentence: “<strong>The cat is on a chair</strong>”<br>Here I show the product of the Q and the K matrix before we apply the softmax.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/915/1*dZG13dlVkWfwDYHGsYNkew.png" /></figure><p>After applying the <strong>causal mask</strong> we apply the <strong>softmax</strong>, which makes the remaining values on the row in such a way that the row sums up to 1.<br>Now, let’s look at the <strong>sliding window attention</strong>.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*LzCCSzgjOUANUloiB_VvTQ.png" /></figure><p>Let’s apply sliding window attention<br>The <strong>sliding window size</strong> is 3.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/994/1*kdTT5AFQlqqkHJtnDG4yuA.png" /></figure><p>After applying the <strong>softmax</strong>, all the –Infinity have become 0, while the other values in the row are changed in such a way that they sum up to one. The output of the softmax can be thought of as a probability distribution.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*5GSgSqb4GBPW3NARWpcUjw.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/733/1*cmslChXGm4M4-qvgNqDumw.png" /></figure><p>The output of the Self-Attention is a matrix of the same shape as the <strong>input sequence</strong>, but where each token now captures information about other tokens according to the mask applied. In our case, the last token of the output captures information about itself and<br>the two preceding tokens.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*991h3Rxs6A0UHpiIh8XsvQ.png" /></figure><p>With a sliding window size 𝑊 = 3, every layer adds information about (𝑊 — 1) = 2 tokens.<br>This means that after 𝑁 layers, we will have an information flow in the order of 𝑊 × 𝑁.</p><h3>Sliding Window Attention: details</h3><ul><li>Reduces the number of <strong>dot-products to perform</strong>, and thus, performance during <strong>training and inference</strong>.<br>•Sliding window attention may lead to degradation in the performance of the model, as some “<strong>interactions</strong>” between tokens will not be captured.<br>•The model mostly focuses on the <strong>local context</strong>, which depending on the size of the window, is enough for most cases.<br><strong>This makes sense if you think about a book</strong>: the words in a paragraph on chapter number 5 depend on the paragraphs in the same chapter but may be totally unrelated to the words used in chapter 1.<br>•Sliding window attention can still allow one token to watch tokens outside the window, using a reasoning similar to the <strong>receptive field</strong> in conventional neural networks.</li></ul><h3>Receptive field in CNNs</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/719/1*yNbN5wggm_MZ2Da2h6cB_Q.png" /></figure><p>This feature depends <strong>directly</strong> on 9 features of the previous layers, but<br><strong>indirectly</strong> on all the features of the initial layers, since each feature of<br>the intermediate layer depends on 9 features of the previous layer. This<br>means that a change in any of the features of the layer 1 will <strong>influence</strong><br>this features as well.<br>It is very similar to the <strong>receptive field of a CNN</strong>.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/489/1*Ry8yg5UBZb0zpcP7Kf2AZA.png" /></figure><h3>Self-Attention during Next Token Prediction Task</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*4qLWngXCcQt2NE07JZmMOA.png" /></figure><p>Let’s break down into steps.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*HagIF_k1_8VdbIb2EOlcBA.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*7MM-dYA1y48Jy2inBOe0Gw.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*ulhBGFr9y-sz3ocfgPnwaw.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*vLE8QZkaSgHfWkCKEO7y1g.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*gCBe2z7nJjp7y0mAVAtVSg.png" /></figure><h3>Self-Attention with KV-Cache</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*AD7rA4kkllC8Ya074cbhVA.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*VmmtkosjT0ZW2g-UTWGW-Q.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*qcbRNE1KTu8y-5u0qQNEZw.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*84r-OE0UlDAsOteRos2YLg.png" /></figure><h3>Rolling Buffer Cache</h3><p>Since we are using Sliding Window Attention (with size W), we don’t need to keep all the previous tokens in the KV-Cache, but we can limit it to the latest W tokens.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*f5l9TtsXz5W1nh4gDaOwIw.png" /></figure><h3>Rolling Buffer Cache: how it works</h3><p>Let’s add the sentence “<strong>The cat is on a chair</strong>”</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/884/1*F-32doMaZs7hJR8eJ71qTA.png" /></figure><p>We add a new token and move the pointer forward<br>Let’s add the sentence “<strong>The cat is on a chair</strong>”</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/881/1*qA4wtXAJc4UaNH_UwZt2NA.png" /></figure><p>We add a new token and move the pointer forward<br>Let’s add the sentence “<strong>The cat is on a chair</strong>”</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/881/1*TcbAKI4LFu88BppsS_8jXg.png" /></figure><p>We add a new token and move the pointer forward<br>Let’s add the sentence “<strong>The cat is on a chair</strong>”</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/893/1*6onyWSxewatylPWFjI2zkg.png" /></figure><p>We add a new token and move the pointer forward<br>Let’s add the sentence “<strong>The cat is on a chair</strong>”</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/875/1*xD6PTqx4hw1ac_Vdq4Qeyw.png" /></figure><p>We add a new token and move the pointer forward<br>Let’s add the sentence “<strong>The cat is on a chair</strong>”</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/887/1*eMVtZl9UYDF7LFzCu1aITw.png" /></figure><p>Now, you can see since window size is 4 and new word ‘a’ is a fifth word so it is replace with the position of ‘The’.</p><h3>Pre-fill and chunking</h3><p>When generating text using a Language Model, we use a prompt and then generate tokens one by one using the previous tokens. When dealing with a KV-Cache, we first need to add all the prompt tokens to the KV-Cache so that we can then exploit it to generate the next<br>tokens.<br>Since the prompt is known in advance (we don’t need to generate it), we can prefill the KV-Cache using the tokens of the prompt. But what if the prompt is very big? We can either add one token at a time, but this can be time-consuming, otherwise we can add all the tokens of the prompt at once, but in that case the attention matrix (which is 𝑁 × 𝑁) may be very big and not fit in the memory.<br>The solution is to use pre-filling and chunking. Basically, we divide the prompt into chunks of a fixed size set to W (except the last one), where W is the size of the sliding window of the attention.<br>Imagine we have a large prompt, with a sliding window size of 𝑊 = 4.<br>For simplicity, let’s pretend that each word is a token.<br>Prompt: “Can you tell me who is the richest man in history”</p><h3>Pre-fill and chunking: first chunk</h3><p>Prompt: “<strong>Can you tell me</strong> who is the richest man in history”</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/466/1*qtVe_oOREDK4o69PkAGjow.png" /></figure><p>At every step, we calculate the attention using the tokens of the<br>KV-Cache + the tokens of the current chunk as Keys and Values,<br>while only the tokens of the incoming chunk as Query.<br>During the first step of pre-fill, the KV-Cache is initially empty.</p><p><strong>After</strong> calculating the attention, we add the tokens of the current<br>chunk to the KV-Cache. This is different from token generation in<br>which we first add the previously-generated token to the KV-Cache<br>and then calculate the attention. <strong>We will see layer why</strong>.</p><h3>Pre-fill and chunking: second chunk</h3><p>Prompt: “Can you tell me who is the richest man in history”</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/655/1*x5KZ_GKvbSVGoZhDl5nQsQ.png" /></figure><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=565f963a1ffd" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Summary of Mistral 7B]]></title>
            <link>https://blog.tai.com.np/summary-of-mistral-7b-1d5ca9a6c17c?source=rss-115b91c97fe4------2</link>
            <guid isPermaLink="false">https://medium.com/p/1d5ca9a6c17c</guid>
            <dc:creator><![CDATA[Manoj Kumal]]></dc:creator>
            <pubDate>Mon, 05 Feb 2024 09:57:20 GMT</pubDate>
            <atom:updated>2024-04-11T11:39:05.825Z</atom:updated>
            <content:encoded><![CDATA[<h3>Abstract</h3><p>The paper introduces Mistral 7B, a language model that performs better and is more efficient than other models. It uses innovative techniques to improve inference speed and handle sequences of any length. They also provide a fine-tuned model, Mistral 7B — Instruct, which outperforms other models in following instructions.Mistral 7B — Instruct, that surpasses Llama 2 13B and models are released under the Apache 2.0 license.</p><h3>Introduction</h3><p>Latest developments in Natural Language Processing and, particularly, in Large Language Models (LLMs) are focused on improving model performance, which often leads to an increase in model size. As one can expect, the escalation in model size also increases computational costs and inference latency, raising barriers when it comes to deploying and using LLMs in real-world scenarios. It offers easy deployment options and integration with platforms like AWS, GCP, Azure, and Hugging Face. It provides superior performance and efficiency through fine-tuning and a chat model that outperforms Llama 2 13B in following instructions. Mistral 7B implements <strong>sliding window attention</strong> (SWA) to handle sequences of arbitrary length, reducing inference costs while maintaining high-quality predictions. It also utilizes a <strong>rolling buffer cache</strong> to optimize memory usage.</p><h3><strong>Architecture Details</strong></h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/731/1*w7NjC_ToRRztmBvjIWYTCg.png" /></figure><h3>Sliding Window Attention</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/674/1*Q8dzOcOkX7-VxeD7LAB8-g.png" /><figcaption>Figure 1: Vanilla Attention Vs Sliding Window Attention</figcaption></figure><p><strong>Sliding Window Attention</strong>. The number of operations in vanilla attention is quadratic in the sequence length, and the memory increases linearly with the number of tokens. At inference time, this incurs higher latency and smaller throughput due to reduced cache availability. To alleviate this issue, we use sliding window attention: each token can attend to at most W tokens from the previous layer (here, W = 3). Note that tokens outside the sliding window still influence next word prediction. At each attention layer, information can move forward by W tokens.<br>Sliding Window Attention leverages the layers of a transformer model to extend its attention beyond a fixed window size, denoted as W. In SWA, the hidden state at position i in layer k can attend to hidden states from the preceding layer within the range of positions i — W to i, allowing access to tokens at a distance of up to W * k tokens. By employing a window size of W = 4096, SWA theoretically achieves an attention span of approximately 131K tokens. In practice with a sequence length of 16K and W = 4096, SWA modifications in <strong>FlashAttention</strong> and <strong>xFormers</strong> result in a 2x speed enhancement compared to vanilla attention methods.</p><p>For detail explanation: <a href="https://medium.com/@manojkumal/sliding-window-attention-565f963a1ffd">Sliding window attention, rolling buffer cache and pre-fill and chunking</a></p><h3>Rolling Buffer Cache</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/709/1*PV11k8MYfincIqruWpZD0A.png" /><figcaption>Figure 2: Rolling Buffer Cache</figcaption></figure><p>The Rolling Buffer Cache utilizes a constant attention span to restrict its size. The cache, set at a fixed size W, preserves keys and values for timestep i at the position i mod W. As i surpasses W, the cache prevents further expansion by overwriting earlier values. As an example, with W = 3 and a 32k-token sequence, the cache effectively reduces memory usage by 8 times without compromising the quality of the model.</p><h3>Pre-fill and chunking</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/629/1*tnj8ai-9Bjs8tOQCZ8vqZw.png" /><figcaption>Figure 3: Pre-fill and chunking</figcaption></figure><p>In sequence generation, tokens are predicted sequentially based on prior context. To optimize efficiency, a (k, v) cache is pre-filled with the known prompt. If the prompt is very long, it is chunked into smaller segments using a chosen window size. Each chunk is used to pre-fill the cache. This approach involves computing attention both within the cache and over the current chunk, Thus aiding in more effective sequence generation.</p><h3>Results</h3><p>Mistral is evaluated against the following benchmarks:</p><ul><li>Commonsense Reasoning (0-shot): Hellaswag, Winogrande, PIQA, SIQA, OpenbookQA, ARC-Easy, ARC-Challenge, CommonsenseQA</li><li>World Knowledge (5-shot): NaturalQuestions, TriviaQA</li><li>Reading Comprehension (0-shot): BoolQ, QuAC</li><li>Math: GSM8K (8-shot) with maj@8 and MATH (4-shot) with maj@4</li><li>Code: Humaneval (0-shot) and MBPP (3-shot)</li><li>Popular aggregated results: MMLU (5-shot), BBH (3-shot), and AGI Eval (3–5-shot, English multiple-choice questions only)</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*eez0QQAZu4rNZjXKMVbQqQ.png" /><figcaption>Figure 4: Performance of Mistral 7B and different Llama models on a wide range of benchmarks</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*NYaoLbmlZsd7DjUddLFodA.png" /><figcaption>Figure 5: Comparison of Mistral 7B with Llama.</figcaption></figure><ul><li>Mistral 7B surpasses Llama 2 13B across all metrics and outperforms Llama 1 34B on most benchmarks.</li><li>In particular, Mistral 7B displays superior performance in code, mathematics, and reasoning benchmarks.</li></ul><h3>Instruction Following</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/546/1*VwthJ3ZoAVcva_Nw3mDCAQ.png" /><figcaption>Figure 6: <em>Comparison of Chat Models</em></figcaption></figure><ul><li>Mistral 7B — Instruct, outperforms all 7B models on MT-Bench, and is comparable to 13B — Chat models.</li></ul><h3>Adding guardrails for front facing applications</h3><p><strong>1. system prompt to enforce guardrails</strong><br>The Mistral system prompt works by providing guidelines and constraints to the model during generation, guiding it to produce output that adheres to acceptable standards and avoids harmful or negative content. It allows users to enforce guardrails to maintain content moderation and integrity while using the language model.</p><figure><img alt="" src="https://cdn-images-1.medium.com/proxy/1*MWRmtgEymVBfDX1pidJbgA.png" /><figcaption>Figure 7: System prompts</figcaption></figure><p>2. <strong>Content moderation with self-reflection</strong><br>Self-reflection plays a role in content moderation by utilizing a self-reflection prompt that enables Mistral 7B to classify prompts or generated answers. This allows the model to assess the appropriateness and acceptability of the content.The benefit of using self-reflection in content moderation is that it provides a mechanism for Mistral 7B to identify and filter out potentially harmful, inappropriate, or misleading content. By applying self-reflection, users can effectively moderate content based on their specific needs and requirements, ensuring a safer and more responsible user experience.</p><figure><img alt="" src="https://cdn-images-1.medium.com/proxy/1*icKXkfdwUExmadoPpViOwQ.png" /><figcaption>Figure 8: Comparison between Mistral and Llama system prompts.</figcaption></figure><p>References</p><p><a href="https://arxiv.org/pdf/2310.06825.pdf">https://arxiv.org/pdf/2310.06825.pdf</a></p><p><a href="https://umarjamil.org/">https://umarjamil.org/</a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=1d5ca9a6c17c" width="1" height="1" alt=""><hr><p><a href="https://blog.tai.com.np/summary-of-mistral-7b-1d5ca9a6c17c">Summary of Mistral 7B</a> was originally published in <a href="https://blog.tai.com.np/">TAI Blog</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Bank Loan Approval Prediction using different Machine Learning Algorithms]]></title>
            <link>https://medium.com/@manojkumal/bank-loan-approval-prediction-using-different-machine-learning-algorithms-f69cbf122423?source=rss-115b91c97fe4------2</link>
            <guid isPermaLink="false">https://medium.com/p/f69cbf122423</guid>
            <dc:creator><![CDATA[Manoj Kumal]]></dc:creator>
            <pubDate>Sat, 08 Jul 2023 11:48:19 GMT</pubDate>
            <atom:updated>2023-07-08T11:50:52.017Z</atom:updated>
            <content:encoded><![CDATA[<h3>Introduction</h3><p>A bank loan is when a bank offer to lend money to customers for a certain time period. As a condition of the bank, the borrower will need to pay a certain amount of interest per month, or per year. The bank will provide a loan for different purpose like business loan, home loan, education loan etc. One of the most challenging parts of a bank is whether the customers will pay back with interest or not because most of the bank comes for a interest. To overcome this problem, we are using machine learning model which help us to predict accurately whether customers are defaulter or not</p><h3>Table of Content</h3><ol><li>Exploratory Data Analysis</li><li>Visualizing Outliers</li><li>Looking at unique value in data</li><li>Handling Imbalance data</li><li>Handing Missing/Null value</li><li>Feature Selection</li><li>Splitting the data for training and testing</li><li>Preprocessing the data for model</li><li>Learning Algorithm</li><li>Metrics Evaluation</li></ol><h3>Exploratory Data Analysis</h3><p>Whenever we are start to do any data science projects. We need to understand what lies inside our data. Understanding about the features/attributes is very first step of the project. It helps determine how best to manipulate data sources to get the answers we need, making it easier for data scientists to discover patterns or check assumptions. The main purpose of EDA is to help look at data before making any assumptions. It can help identify obvious errors, as well as better understand patterns within the data, detect outliers or anomalous events, find interesting relations among the variables. We will look at the features of our dataset.</p><pre>df.columns</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/643/0*iYZPKdYjn7yVh7aQ.png" /></figure><p>we have 18 features where 17 are independent and 1 is dependent variable. i.e Loan Status in our dataset. The features are self explanatory.</p><pre>df.dtypes</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/772/0*rI2lfisxFQhl8tgL.png" /></figure><p>We can see in above figure. There are three types of dataset where there are three types of datatypes. They are float64, int64 and object.</p><ul><li>int64: It represents the integer features in dataset. There are 5 there variables.</li><li>float64: It represents the features having numbers with a fractional part, containing one or more decimals. There are 8 floating variables</li><li>object: It represents the features having multiple datatypes including string. There are 6 object variables.</li></ul><p>There are a variety of descriptive statistics. Numbers such as the mean, median, mode, skewness, kurtosis, standard deviation, first quartile and third quartile, to name a few, each tell us something about our data.</p><ul><li>The minimum — this is the smallest value in our data set.</li><li>The first quartile — this number is denoted Q1 and 25% of our data falls below the first quartile.</li><li>The median — this is the midway point of the data. 50% of all data falls below the median.</li><li>The third quartile — this number is denoted Q3 and 75% of our data falls below the third quartile.</li><li>The maximum — this is the largest value in our data set.</li></ul><pre>df.describe()</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/887/0*nTPfWeaim-5GiAWC.png" /></figure><p>From the above table we can find out an data entry/measurement error in credit score. The maximum value of credit score is 7510 which is impossible. The value of credit score lies within 250–850 or 300–900. So we have to deal with those wrong values (which will be discussed later). The maximum value of current loan amount is also way too higher than it’s median which can be a potential outlier in data. We can also represent our data graphically using histogram and pair plot.</p><pre>df.hist(bins=50,figsize=(25,16))</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1010/0*bSph7naznHQYwz8m.png" /></figure><p>A histogram is one of the most frequently used data visualization techniques in machine learning. It represents the distribution of a continuous variable over a given interval or period of time. Histograms plot the data by dividing it into intervals called ‘bins’.</p><ul><li>It provides us a count of the number of observations in each bin created for visualization.</li><li>From the shape of the bin, we can easily observe the distribution i.e. weather it is Gaussian, skewed or exponential.</li><li>Histograms also help us to see possible outliers.</li></ul><pre>sns.pairplot(X,hue=&#39;Loan Status&#39;)</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/648/0*1AyaOOrBwjuYmbnk.png" /></figure><p>Pairplot are also very useful while visualizing the single variable or relationship between pair of variables. It provides really important insight on which variable separates our classes. But pairplot can’t be useful if we have large number of features because the number of pairplot becomes very large which makes it difficult to go through all the pariplots. We can use seaborn library for pairplot.</p><h3>Visualizing Outliers</h3><p>An outlier is an object that deviates significantly from the rest of the objects. They represent errors in measurement, bad data collection, or simply show variables not considered when collecting the data. Most data mining methods discard outliers noise or exceptions, however, in some applications such as fraud detection, the rare events can be more interesting than the more regularly occurring one and hence, the outlier analysis becomes important in such case. Visualizing data with help of boxplot can be helpful to glance at the outliers. Boxplot is based on five point summary (minimum, maximum, median, first quartile and third quartile). There is also whisker on the both sides box which ranges up to 1.5 times IQR(difference between first and third quartile) from upper quartile and 1.5 times IQR below the lower quartile. All the data points which doesn’t fall within the whisker are considered as outliers.</p><pre>sns.boxplot(x=&#39;Loan Status&#39;, y=&#39;Credit Score&#39;, data=df)</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/468/0*Ms5POQ7fUQ4CXDtW.png" /></figure><p>We see lots of outliers in credit score. As we have discussed earlier, the maximum limit of credit score lies below 900. We have to either remove all the value which contain credit score greater than 900 which will decrease the data point which is not the optimal solution while dealing with outliers or we have to replace the outliers with mean value of credit score. Carefully looking at the value higher than 900.we can see 0 at the rightmost position of value which seem to have been added accidently while entering the value. So we will remove 0 from the respective value. Now the box plot after removing 0. Similarly, we can also see outliers in Current Loan Amount, monthly debt and Maximum Open Credit.</p><pre>sns.boxplot(x=&#39;Loan Status&#39;, y=&#39;Monthly Debt&#39;, data=df)</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/497/0*l0v3M3pBU4kG4N-N.png" /></figure><pre>sns.boxplot(x=&#39;Loan Status&#39;, y=&#39;Maximum Open Credit&#39;, data=df)</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/503/0*16QNgSgRd2AHJUwn.png" /></figure><p>We can see that alot of outlier in Maximum Open Credit are present in only single class Fully Paid i.e who had successfully paid the loan amount. while Monthly debt have also some outliers are present in Charged Off. we remove those outliers from the dataset IQR for detecting outliers</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/700/0*RgWRhW5EO_DkaLJF.png" /></figure><p>A box plot tells us, more or less, about the distribution of the data. It gives a sense of how much the data is actually spread about, what’s its range, and about its skewness. In the above figure,</p><ul><li>minimum is the minimum value in the dataset,</li><li>and maximum is the maximum value in the dataset. So the difference between the two tells us about the range of dataset.</li><li>The median is the median (or center point), also called second quartile, of the data</li><li>Q1 is the first quartile of the data, i.e., to say 25% of the data lies between minimum and Q1.</li><li>Q3 is the third quartile of the data, i.e., to say 75% of the data lies between minimum and Q3. The difference between Q3 and Q1 is called the Inter-Quartile Range or IQR.</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/617/0*u5F_gi_MfRlKqeHK.png" /></figure><ul><li>To detect the outliers using this method, we define a new range, let’s call it decision range, and any data point lying outside this range is considered as outlier. The range is as given below:</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/640/0*hew1pebmgZ2zV42R.png" /></figure><ul><li>Any data point less than the Lower Bound or more than the Upper Bound is considered as an outlier. Now, we are finding the lower and upper bound of monthly debt, Maximum Open Credit to remove outlier.</li></ul><pre>min_threshold,max_threshold = df[&#39;Maximum Open Credit&#39;].quantile([0.001,0.999])<br>min_threshold,max_threshold<br>df = df[(df[&#39;Maximum Open Credit&#39;]&lt;max_threshold) &amp; (df[&#39;Maximum Open Credit&#39;]&gt;min_threshold])]</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/385/0*LVf_Y9wDWn92JeXQ.png" /></figure><pre>min_threshold,max_threshold = df[&#39;Monthly Debt&#39;].quantile([0.001,0.999])<br>min_threshold,max_threshold<br>df = df[(df[&#39;Monthly Debt&#39;]&lt;max_threshold) &amp; (df[&#39;Monthly Debt&#39;]&gt;min_threshold])]</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/418/0*PbsPh-VglqSUF69V.png" /></figure><p>Let’s talk about current loan amount, lower bound and upper bound values are 21538.0 and 999999999.0 so we only take data range between lower and upper bound. This is the method of removing outlier from dataset.</p><h3>Looking at unique value in our data</h3><pre>df_dup = df[df.duplicated()]<br>df_dup.shape</pre><pre>df.drop(df_dup.index, inplace=True)</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/372/0*MTxtvu885tj1ebHd.png" /></figure><p>There are 10215 rows which are duplicated in our dataset which we have to remove. If we look at the unique value in different categorical features we find that the Purpose and Home Ownership feature have duplicate category. We see the other category is duplicated. One begins with small O and another begins with capital O. So, we can replace those features using excel find and replace option. We will also replace the minority category whose presence is less than 0.5% with other category.</p><pre>df[&#39;Purpose&#39;].value_counts()</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/599/0*uq8nobZ7CiXeB2h7.png" /></figure><h3>Handling Missing/Null Values</h3><p>We can find the total number of missing value in each feature using pandas function.</p><pre>df.isna().sum()</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/698/0*wMdsDzktlAyURllN.png" /></figure><pre>sns.heatmap(df.isnull(),yticklabels=False,cbar=False,cmap=&#39;viridis&#39;)</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/540/0*krfVPSQTbhyqsnjW.png" /></figure><p>The percentage of missing value in Months since last delinquent is 48%. Since the percentage of missing value is greater than 30% we will not use this feature in our model. Removing all missing value results in loss of data which we don’t want. We can impute the missing value with mean, median or mode. We can also try KNN imputation. But here we will use mean, median or mode. If our feature are categorical then we use the mode (maximum frequency) to fill missing value. For numerical feature we use mean but if there are outliers then we use median which isn’t sensitive to outliers.</p><pre>df[&#39;Credit Score&#39;] = df[&#39;Credit Score&#39;].fillna(df[&#39;Credit Score&#39;].median())<br>df[&#39;Annual Income&#39;] = df[&#39;Annual Income&#39;].fillna(df[&#39;Annual Income&#39;].mean())<br>df[&#39;Years in current job&#39;] = df[&#39;Years in current job&#39;].fillna(df[&#39;Years in current job&#39;].mean())<br>df[&#39;Months since last delinquent&#39;] = df[&#39;Months since last delinquent&#39;].fillna(df[&#39;Months since last delinquent&#39;].median())<br>df[&#39;Maximum Open Credit&#39;] = df[&#39;Maximum Open Credit&#39;].fillna(df[&#39;Maximum Open Credit&#39;].median())<br>df[&#39;Bankruptcies&#39;] = df[&#39;Bankruptcies&#39;].fillna(df[&#39;Bankruptcies&#39;].mode()[0])<br>df[&#39;Tax Liens&#39;] = df[&#39;Tax Liens&#39;].fillna(df[&#39;Tax Liens&#39;].mode()[0])</pre><pre>sns.heatmap(df.isnull(),yticklabels=False,cbar=False,cmap=&#39;viridis&#39;)</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/511/0*A3TDvW5eLcIUKY6L.png" /></figure><h3>Handling the Imbalance Data</h3><p>Let’s look at the percentage of each class in dataset.</p><pre>df[&#39;Loan Status&#39;].value_counts(normalize=True)</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/514/0*SzYocHAIW-1kV79D.png" /></figure><pre>df[&#39;Loan Status&#39;].value_counts(normalize=True).plot.bar()</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/638/0*XbEY4x0aL6yYjYVb.png" /></figure><p>We see that our dataset is not balanced. We need to balance our data otherwise our model will be biased towards majority class. There are two techniques to handle imbalanced dataset.</p><ol><li>Under Sampling</li><li>Over Sampling</li></ol><p>In under sampling we reduce the number of data points of the majority class to make dataset balance and in over sampling we increase the number of data points of minority class by duplicating the data points. Most of the time we use over sampling because we don’t want to lose our data. We will use over sampling method to balance dataset. Duplicating data points to increase data doesn’t add any new information to the model. Instead, new data points are synthesized from the existing data points. This type of method is referred to as Synthetic Minority Oversampling Technique (SMOTE). SMOTE first select a minority class instance at random and finds its k nearest minority class neighbors. The synthetic instance is then created by choosing one of the k nearest neighbors b at random and connecting a and b to form a line segment in the feature space. The best way of doing SMOTE is not applying SMOTE directly on all data. Once we split our data on train and test then, we only apply SMOTE on train data because all the synthetic data should only be used for training not for validation.</p><h3>Features Selection</h3><p>Feature Selection is the process to automatically select those features which contribute most to our model. Having irrelevant features in our data don’t increase the accuracy of our model. We want to remove those features which doesn’t have significant effect. We can use seaborn library to plot heat map of study the correlation between features.</p><pre>plt.figure(figsize=(12,10))<br>sns.heatmap(df_corr.corr(), annot=True, cmap=&#39;RdYlGn&#39;)<br>plt.show()</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/612/0*20-pzeRmPHm5t2FY.png" /></figure><p>We see our most of the features are not highly correlated with each others. Two features Number of Credit Problem and Bankruptcies are correlated with each other. Instead, if two features are perfectly correlated, then one doesn’t add any additional information. So removing either of the features don’t affect the accuracy of model. We can also use Variance Inflation Factor (VIF) for feature selection. VIF is the measure of the amount of multicollinearity in a set of features. Multicollinearity occurs when two or more independent variables are highly correlated with one another in a regression model. We can find the VIF of respective feature and remove the feature having VIF greater than 10. VIF score of an independent variable represents how well the variable is explained by other independent variables. VIF is calculated by keeping one independent variable as dependent and other independent variable as it is and we fit a linear regression model and calculate coefficient of determination. Coefficient of determination represents the total variation for a dependent variable that is explained by an independent variables. R² value is determined to find out how well an independent variable is described by the other independent variables. A high value of R² means that the variable is highly correlated with the other variables. After calculating coefficient of determination we can find VIF by using the formula</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/472/0*VNSnO6sse3l0Ow9Y.png" /></figure><pre>from statsmodels.stats.outliers_influence import variance_inflation_factor<br>vif = [variance_inflation_factor(df1.values,i) for i in range (df1.shape[1])]<br>for i in range(0,11):<br>    print(&quot;The Vif for {} is {}&quot;.format(df1.columns[i],vif[i]))</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/566/0*iiK3AGcqD_Nncsma.png" /></figure><h3>Splitting the data</h3><p>The fundamental goal of an ML model is to make accurate predictions on future data instances beyond those used to train models. Before using an ML model to make predictions, we need to evaluate the predictive performance of the model. To estimate the quality of an ML models predictions with data it has not seen, so we have to split our data into training and testing data.This allows us to evaluate our model on unseen data.</p><pre>from sklearn.model_selection import train_test_split<br>X_train,X_test,y_train,y_test = train_test_split(<br>    df_final_feature,y,test_size=0.1,stratify=df[&#39;Loan Status&#39;])</pre><h3>Preprocessing the data</h3><p>Data preprocessing is crucial in any data mining process as they directly impact success rate of the project. so, First of all we have to remove the unnecessary feature like Loan ID and Customer ID which are the unique values assigned to each customer. Those features doesn’t add any information to our model. Before feeding our data into model we need to encode the categorical value into numerical value. Categorical value refers to the information that has specific categories within the dataset. There are four categorical features in our dataset i.e Loan Status, Term, Home Ownership, Purpose. we use different technique for treating categorical variables i.e One Hot Encoding. One Hot Encoding creates an additional feature based on the number of unique values in the features. Then every unique value will be treated as feature consisting of binary data either 0 or 1.</p><pre>One_hot = pd.get_dummies(df[[&#39;Term&#39;,&#39;Home Ownership&#39;, &#39;Purpose&#39;]],drop_first=True)<br>One_hot.head()</pre><p>Tree based models doesn’t perform well with one hot encoding if there are too many unique value in feature. This is because they pick the subset of feature while splitting the data. If we have a lot of unique value, then the choose features will be mostly zero which doesn’t produce significant result. So we don’t perform one hot encoding while training tree based models. Linear models don’t suffer from this problem.</p><p>The next step for preprocessing is feature scaling. There are various method to scale the data like standardization, min-max scaler etc. In min-max scaler, we simply subtract the minimum value of feature from current value and divide it by the difference of maximum and minimum value. This technique re-scale a feature value between 0 and 1. Other, technique is to calculate the statistical mean and standard deviation of the attribute values, subtract the mean from each value, and divide the result by the standard deviation. This process is called standardizing a statistical variable and results in a set of values whose mean is zero and standard deviation is one.</p><pre>os = SMOTE(sampling_strategy=0.8,random_state=42)<br>X_train_res, y_train_res = os.fit_resample(X_train,y_train)<br>X_train_res.shape, y_train_res.shape</pre><h3>Evaluation Metrics</h3><p>Evaluation metrics are used to measure the performance of a machine learning model, which is an integral component of any data science project. It aims to estimate the generalization accuracy of a model on the future (unseen/out-of-sample) data. The most frequent classification evaluation metric that we use should be ‘Accuracy’. You might believe that the model is good when the accuracy rate is 99%! However, it is not always true and can be misleading in some situations.</p><h3>Confusion Matrix</h3><p>Evaluation of the performance of a classification model is based on the counts of test records correctly and incorrectly predicted by the model. The confusion matrix provides a more insightful picture which is not only the performance of a predictive model, but also which classes are being predicted correctly and incorrectly, and what type of errors are being made. To illustrate, we can see how the 4 classification metrics are calculated (TP, FP, FN, TN), and our predicted value compared to the actual value in a confusion matrix is clearly presented in the below confusion matrix table.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/695/0*0jAY1Cl8y3h9ZMAe.png" /></figure><ul><li>True Positive (TP): Predicted True and True in reality. i.e Model predict that is positive and actually, they will repay the loan.</li><li>True Negative (TN): Predicted False and False in reality. i.e Model predict that is Negative and actually, they will not repay the loan.</li><li>False Positive (FP): Predicted True and False in reality. i.e Model predict that is positive and actually, they will not repay the loan.</li><li>False Negative (FN): Predicted False and True in reality. i.e Model predict that is Negative and actually, they will repay the loan.</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/743/0*tng-Q-DCXLoH0lps.png" /></figure><ul><li>Accuracy — Accuracy is the most intuitive performance measure and it is simply a ratio of correctly predicted observation to the total observations. One may think that, if we have high accuracy then our model is best. Yes, accuracy is a great measure but only when you have symmetric datasets where values of false positive and false negatives are almost same. Therefore, you have to look at other parameters to evaluate the performance of your model</li><li>Precision — Precision is the ratio of correctly predicted positive observations to the total predicted positive observations.</li><li>Recall (Sensitivity) — Recall is the ratio of correctly predicted positive observations to the all observations in actual class — yes.</li><li>F1 score — F1 Score is the weighted average of Precision and Recall. Therefore, this score takes both false positives and false negatives into account. Intuitively it is not as easy to understand as accuracy, but F1 is usually more useful than accuracy, especially if you have an uneven class distribution. Accuracy works best if false positives and false negatives have similar cost. If the cost of false positives and false negatives are very different, it’s better to look at both Precision and Recall.</li></ul><h3>Learning Algorithm</h3><p>It is obvious that algorithms have been designed to solve specific problems. So, it is important to know what type of problem we are dealing with and what kind of algorithm works best for each type of problem. Choosing the right algorithm is also a important phase of data science project. Since our project is a classification problem we can choose variety of classification algorithm to make prediction. Some of the algorithms we have used are listed below:</p><ol><li>Logistic Regression</li><li>K Nearest Neighbors</li><li>Random Forest Classifier</li><li>Support Vector Machine</li></ol><h3>Logistic Regression</h3><p>Logistic Regression is the supervised machine learning algorithm for classification problem. It is basically used for binary classification. i.e. when the target variable has two classes.Logistic Regression is usually applied to a problem statement where two classification problems can be linearly separable. Logistic regression is named for the function used logistic function. Logistic function is also called sigmoid function. The logistic function gives an “S” shape curve that can take any real value and map it into value between 0 and 1. If the output of the function is greater than 0.5, we classify the output as 1 and if the output is less than 0.5, we classify the output as 0. Below is the figure of sigmoid function.. Below is the figure of sigmoid function. Below is the figure of sigmoid function.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/586/0*ia6CKDP2JD6W_A9o.png" /></figure><p>We will use sklearn library to implement logistic regression. Since our data set is imbalanced evaluating our model based on accuracy gives us misleading result. We use confusion matrix show the correct and incorrect prediction. We want to decrease the false positive while maintaining acceptable false negative. We will use different value of upsampling ratio for training and compare their result.</p><p>The accuracy using upsampling ratio of 0.8 is 65% and confusion matrix is shown in below figure.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/357/0*LIMWI5Qu-1Vqipcv.png" /></figure><p>The accuracy using upsampling ratio of 0.9 is 57% and confusion matrix is shown in below figure.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/367/0*pCgrRo-V1-FqUxg1.png" /></figure><p>The accuracy using upsampling ratio of 1.0 is 51% and confusion matrix is shown in below figure.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/365/0*7fdEV3JPSxT7c_Ai.png" /></figure><p>We can see that while increasing the upsampling ratio the accuracy of the model decrease. The number of false positive also decreases while increasing the upsampling ratio at the same time the number of false negative also increase. In such case we can use some threshold value for false negative and choose the best performing model according to our need.</p><h3>K Nearest Neighbors</h3><p>KNN is a non-parametric classification method. It is used for classification and regression problem. It is one of the simplest algorithms. The key idea of KNN algorithm is nearby points belongs to same class. KNN algorithm makes no assumption about the data. It computes the distance between the testing point to every training example. Then select the k closest instances and their labels. It output the class which is most frequent in labels. The training time of KNN algorithm is zero while the testing time can become very large with large number of datapoints in dataset. This algorithm can be computationally expensive because we need to store all training example and compute distance to all training example for a single prediction. Below is the figure how KNN algorithm works.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/614/0*93ZSEgIlcnGtsjd-.png" /></figure><p>The accuracy using upsampling ratio of 0.8 is 59% and confusion matrix is shown in below figure.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/357/0*NZh-glAcgyPnnl80.png" /></figure><p>The accuracy using upsampling ratio of 0.9 is 59% and confusion matrix is shown in below figure.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/370/0*YLsncF6myAyxjHns.png" /></figure><p>The accuracy using upsampling ratio of 1.0 is 57% and confusion matrix is shown in below figure.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/379/0*JEKNT1y8pa29o3TV.png" /></figure><h3>Random Forest Classifier</h3><p>Random Forest is an ensemble learning method for classification and regression problem. It uses the bagging technique. In bagging, we have multiple base learner models and that is called as decision tree because we basically use decision tree in this model. We will take some sample of rows and feature sample then give it to the decision tree 1 and similarly, we do row sampling and feature sampling with replacement because some sample will be repeated but not all. In this way, we will give some sample of row and feature sample to every base learner models. After getting training data, every base learner model will be trained. Whenever my testing data is come and all my model will gives the output. So, this is call bootstrap. Now, according to the bagging we have to aggregate our output. Then, we are going to use majority of vote. If the more model will give the output as same class then that class will be the final result.</p><p>When I using decision tree and we create tree its completely depth. So, when we do that it basically has low bias and high variance. That mean if I am creating a decision tree to its completely depth then it will get properly execute in trained for our training data. So that training error will very less and high variance is basically say that whenever we get our new test data those decision tree they will give larger amount of error. That mean whenever we are creating a decision tree to its completely depth it leads to something called as over-fitting. When we will combined the multiple decision trees the high variance is convert into the low variance and it will solve the over-fitting.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/543/0*2cpbk3vEt-wGIBj7.png" /></figure><p>We can use the same method described above.</p><p>The accuracy using upsampling ratio of 0.8 is 71% and confusion matrix is shown in below figure.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/374/0*PtgguQdS1FgeeEfj.png" /></figure><p>The accuracy using upsampling ratio of 0.9 is 70% and confusion matrix is shown in below figure.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/373/0*88M6CDEFbP1q4bvB.png" /></figure><p>The accuracy using upsampling ratio of 1.0 is 69% and confusion matrix is shown in below figure.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/375/0*daOyzBC3By73aUoh.png" /></figure><h3>Support Vector Machine</h3><p>Support Vector Machine is supervised learning model that analyze data for classification and regression analysis. It constructs a hyperplane in high dimension space which can be used for classification or regression. It finds a hyperplane which has the largest distance to the nearest training data points. The closest points are called support vectors and hence algorithm is termed as Support Vector Machine. The distance from the nearest training point to hyperplane is called margin. It tries to maximize the margin. So SVM are also called max-margin classifier. Max margin can reduce the effect of mislabelled data than min margin and it generalizes better in future for unseen data. SVM are effective in high dimensional space because data are likely to be linearly separable in high dimension than in low dimension. Below is the figure SVM algorithm.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/434/0*qcjFCXWHXWILHYok.png" /></figure><p>We can use the same method described above. The accuracy using up-sampling ratio of 1.0 is 45% and confusion matrix is shown in below figure</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/368/0*o3oxN86fONXfXTxr.png" /></figure><p>Analyzing the above confusion matrix, we can see the performance of different algorithm varies significantly. SVM doesn’t perform well as it is predicting only a single class for all of the data. At the upsampling ratio of 1.0 logistic regression has the lowest number of false positive as compared to other algorithm which is also the desired objective of our problem. The decrease in false positive came at the cost of increase in false negative</p><h3>Conclusion</h3><p>This article aimed to explore, analysis, and build a machine learning algorithm to correctly identify whether a person, given certain attributes, has a high probability to default on a loan. This type of model could be used by different bank and financial institution to identify certain financial traits of future borrowers that could have the potential to default and not pay back their loan by the designated time. The Random Forest Classifier provided us with an accuracy of 72% while the Logistic Regression provided us with an accuracy of 65% but by using domain knowledge and observation of confusion matrix. Hence, the Random Forest model appears to be a better option for such kind of data. We hope such system can be deployed in real world situation after further refinement which can be very useful and interesting for financial sector.</p><h3>Links</h3><p>The code for the project can be found <a href="https://github.com/Manojkumal/Bank-Loan-Prediction">Here</a></p><p>The original dataset can be found <a href="https://www.kaggle.com/zaurbegiev/my-dataset">Here</a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=f69cbf122423" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Flight Price Prediction using different Machine Learning Algorithms.]]></title>
            <link>https://medium.com/@manojkumal/flight-price-prediction-using-different-machine-learning-algorithms-142c92e4652a?source=rss-115b91c97fe4------2</link>
            <guid isPermaLink="false">https://medium.com/p/142c92e4652a</guid>
            <dc:creator><![CDATA[Manoj Kumal]]></dc:creator>
            <pubDate>Sat, 08 Jul 2023 11:40:24 GMT</pubDate>
            <atom:updated>2023-07-08T11:40:24.713Z</atom:updated>
            <content:encoded><![CDATA[<h3>Introduction</h3><p>With the development of transportations industry and everyone has limited time for travelling, the airplane become more popular way of travelling. The advent of internet everywhere, people can easily check, book and buy air ticket any time with the help of internet. Airlines frequently changes prices of flights to maximize profit and fill more seats. Based on the demand for specific route and travel date, flight distances, service categories, airline may increase or decrease the price of ticket. Generally, costumers want to busy ticket at lowest possible price. However, they are still confuse when to buy ticket or not. Especially those people who are not frequently travel in airplane. To overcome this problem, we are used machine learning model to predict flight price accurately. This would be very useful application for us, which can help passengers to make appropriate purchase decision and save their expense.</p><h3>Table of Contents</h3><ol><li>Data Gathering</li><li>Exploratory Data Analysis</li><li>Feature Engineering</li><li>Feature Selection</li><li>Model Creation</li><li>Model Evaluation</li><li>Conclusion</li></ol><h3>Data Gathering</h3><p>Data is the very essential for any kind of data science projects. First of all, we have to collect data regarding our domain. This data collection exercise often requires domain expert. With the help of domain expert we could collect more accurate data. If we have good dataset then machine learning model can learn more accurately and make better prediction. So, more accurate data is required according to our problem statement. The model performance is depends on how the good data will we provide to our model. To do this project I had take data from kaggle.com. The dataset is look like this below.</p><pre>train_data = pd.read_excel(&quot;Data_Train.xlsx&quot;)<br>pd.set_option(&#39;display.max_columns&#39;,None)<br>train_data.head()</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*LHILjKRFWWKDimPpxeIZ4A.png" /></figure><h3>Exploratory Data Analysis</h3><p>It is a way of visualizing, summarizing and interpreting the information that is hidden in rows and column format. We need understand the various aspects of data and what lies inside our data. Understanding about features/attributes like how many number of numerical features, categorical features and how they are represent etc. First, We will look at the features of our dataset.</p><pre>train_data.columns</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/689/1*KHRgLbkUNt5vIczFfVztJg.png" /></figure><p>We have 10 independent features and 1 dependent feature i.e Price is our target variable in dataset. The features are self explanatory.</p><pre>train_data.datatypes</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/282/1*bPgkwSRNbgKJK6hZv7Ykzw.png" /></figure><p>we can see that their is only two datatypes. They are int64 and object.</p><ul><li>int64: It represents the integer feature in dataset.</li><li>object: It represents the feature having multiple datatypes including string.</li></ul><p>Most of the features are object. which is treated as categorical because it contain string and mathematical notation. The feature Additional Info is unnecessary column because it contain almost 80% of no information and it couldn’t help us to making our conclusion so we can remove this.</p><pre>train_data.drop([&#39;Additiona_Info&#39;],axis=1,inplace=True)<br>train_data.head()</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*F2B050SRt0HdVOhd4N5D6Q.png" /></figure><p>In EDA, we are used different statistical tools for visualizing the data, which help us to reveal hidden information and maximize insight into dataset. We can find the outlier and the relation between independent feature and dependent feature in data. Now, we can see the relation between Airline and price using cat plot.</p><pre># Airline vs Price<br>sns.catplot(y = &quot;Price&quot;, x = &quot;Airline&quot;, data = train_data.sort_values(<br>&quot;Price&quot;, ascending = False), kind=&quot;boxen&quot;, height = 6, aspect = 3)<br>plt.show()</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*uq34IAl7Va4sWWSAmWI_5Q.png" /></figure><p>we can easily see the relation between Airline and price. Jet Airways Business have highest price that means it has more impact on price. Jet Airways also have highest price after Jet Airways Business that means it has more impact on price but less impact than Jet Airways Business. Trujet have lowest price so it has less impact on price. The relation between Price and Source using catplot.</p><pre># Price vs Source<br>sns.catplot(y = &quot;Price&quot;, x = &quot;Source&quot;,data= train_data.sort_values(<br>  &quot;Price&quot;, ascending=False), kind = &quot;boxen&quot;, height = 4, aspect = 3)<br>plt.show()</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*XyLwv4SdU3lcc0mug-UEyA.png" /></figure><p>The Source from banglore have highest price and it has more significant impact on price and source from chennai have less prices so it has less significant impact on price.</p><h3>Feature Engineering</h3><p>These are the list of feature engineering techniques.</p><ol><li>Handling Missing values</li><li>Handling Outliers</li><li>Log Transform</li><li>Extracting Date</li><li>One hot Encoding</li><li>Feature scaling</li><li>Splitting the data for train and test</li></ol><h3>Handling Missing values</h3><p>Missing values are one of the most common problem we can face when we are try to prepare our data for machine learning. The reason for the missing values might be human errors, interruptions in the data flow, privacy concerns, and so on. Whatever is the reason, missing values affect the performance of the machine learning models. The most simple solution for missing value is drop the entire rows and columns but it is not good solution because loss of information and data. We have to Works poorly if the percentage of missing values is high (say 70%), compared to the whole dataset then we can remove it. We can also use mean, median and mode to replace missing values.</p><pre>#looking for Null Values<br>train_data.isnull().sum()</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/465/0*AdTWErCLa-Gw7nb1.png" /></figure><h3>Handling Outliers</h3><p>We can detect outlier with the help of mathematical method by using percentiles. We need to assume certain percent of the value from top or bottom as a outlier. Additionally, one common mistake while using percentiles top 5% is not the values between 96 and 100. Top 5% means here the values that are out of the 95th percentile of data. Another option for handling outliers is to cap them instead of dropping. So you can keep your data size and at the end of the day, it might be better for the final model performance.</p><h3>Log Transform</h3><p>Logarithm transformation is one of the most commonly used mathematical transformations in feature engineering. There are some benefits of log transform. It helps to handle skewed (left skew and right skew) data and after transformation, the distribution becomes more approximate to normal distribution/Gaussian distribution.It also decreases the effect of the outliers, due to the normalization of magnitude differences and the model become more robust.</p><pre>import scipy.stats as stats<br>def diagnostic_plot(train_data, variable):<br># funtion to plot histogram and Q-Q plot<br># side by side, for certain variable<br>    plt.figure(figsize=(16,9))<br>    plt.subplot(1,2,1)<br>    train_data[variable].hist()<br><br>    plt.subplot(1,2,2)<br>    stats.probplot(train_data[variable],dist=&#39;norm&#39;,plot=plt)<br>    plt.show()<br><br>    train_data[&#39;Log_Price&#39;] = np.log(train_data[&#39;price&#39;]+1)<br>    diagnostic_plot(train_data, &#39;Log_Price&#39;)</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/945/0*yKp-1Kx924BkAPf2.png" /></figure><h3>Extracting Date</h3><p>Date columns usually provide very useful information about the model target. However, dates can be present in numerous formats, which make it hard to understand by algorithms, even they are simplified to a format like “24/03/2019”.so, we have to use different type of preprocessing for dates.</p><ul><li>Extracting the parts of the date into different columns: Year, month, day, etc.</li><li>Extracting the time period between the current date and columns in terms of years, months, days, etc.</li><li>Extracting some specific features from the date: Name of the weekday, Weekend or not, holiday or not, etc.</li></ul><p>Date_of_Journey:</p><p>In the column ‘Date_of_Journey’, we can see the date format is given as dd/mm/yyyy and as you can see the datatype is given as object So there is two ways to tackle this column, either convert the column into Timestamp or divide the column into date, Month ,Year. Here , i am splitting the columns.</p><pre>train_data[&#39;Journey_day&#39;] = pd.to_datetime(train_data[&#39;Date_of_Journey&#39;], format = &#39;%d/%m/%Y&#39;).dt.day<br>train_data[&#39;Journey_month&#39;] = pd.to_datetime(train_data[&#39;Date_of_Journey&#39;], format = &#39;%d/%m/%Y&#39;).dt.month<br>train_data.drop([&#39;Date_of_Journey&#39;],axis=1,inplace=True)</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1016/0*jNVFY6nKzptFKSJ7.png" /></figure><p>Arrival_Time:</p><p>In the column ‘Arrival_Time’, if we see we have combination of both time and month but we need only the time details out of it so we split the time into ‘Hours’ and ‘Minute’.</p><pre># Arrival time is when the plane pulls up to the gate.<br># Similar to Date_of_Journey we can extract values from Arrival_Time<br>train_data[&#39;Arrival_hour&#39;] = pd.to_datetime(train_data[&#39;Arrival_Time&#39;]).dt.hour<br>train_data[&#39;Arrival_minute&#39;] = pd.to_datetime(train_data[&#39;Arrival_Time&#39;]).dt.minute<br>train_data.drop([&#39;Arrival_Time&#39;],axis=1,inplace=True)</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/977/0*c5QI2114T4T13mhC.png" /></figure><p>Dep_Time:</p><p>As same as ‘Arrival_time’ .we split this column also in hour and minute.</p><pre># Departure time is when a plane leaves the gate.<br># Similar to Date_of_Journey we can extract values from Dep_Time<br>train_data[&#39;Dep_hour&#39;] = pd.to_datetime(train_data[&#39;Dep_Time&#39;]).dt.hour<br>train_data[&#39;Dep_min&#39;] = pd.to_datetime(train_data[&#39;Dep_Time&#39;]).dt.minute</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/958/0*8pxOq_iX5AU5Tns1.png" /></figure><h3>Dummy Variable Encoding</h3><p>In our dataset, there may be some categorical features that don’t understand by computer so we have to convert those categorical features into numerical feature. Categorical features contain string data type. There are two types of categorical feature in our dataset namely ordinal and nominal. Nominal data are those data which don’t have priority ordering and Ordinal data are those data that has priority ordering with each variable. There are many ways to handle categorical features. In this case, we use dummy variable encoding. Dummy variable encoding represents n categories with n-1 binary value. First of all, we are going convert nominal data types. There are three categorical features in our dataset, which are nominal data types. i.e Airline, Source, Destination.</p><pre>## As Airline is Nominal Categorical data we will perform OneHotEncoding<br>Airline = train_data[&#39;Airline&#39;]<br>Airline = pd.get_dummies(Airline, drop_first=True)<br>Airline.head()</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/954/0*009EDjeG9tStu_ex.png" /></figure><p>For each level of a categorical feature, we create a new variable. Each category is mapped with a binary variable containing The problem with the one hot encoding technique is that it creates redundancy because it. The problem with the one hot encoding technique is that it creates redundancy because it creates an additional feature based on categories in the feature. Tree based models doesn’t perform well with one hot encoding if there are too many unique value in feature. This is because they pick the subset of feature while splitting the data. If we have a lot of unique value, then the chosen features will be mostly zero which doesn’t produce significant result. So we don’t perform one hot encoding while training tree based models. Linear models don’t suffer from this problem.</p><pre>Source = train_data[&#39;Source&#39;]<br>Source = pd.get_dummies(Source, drop_first=True)<br>Source.head()</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/343/1*ccQvPrV04MLwpfUggu17vA.png" /></figure><pre>Destination = train_data[&#39;Destination&#39;]<br>Destination = pd.get_dummies(Destination, drop_first=True)<br>Destination.head()<br> </pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/335/1*2jyjZQwlrUC-xQ753Fvhyw.png" /></figure><p>Ordinal data are those data that has priority ordering with each variable. An ordinal encoding involves mapping each unique label to an integer value. This type of encoding is really only appropriate if there is a known relationship between the categories. In Label encoding, each label (category) is converted into an integer value. Total_stops is ordinal variable. The route and total step are related. The ‘Route’ columns mainly tell us that how many cities they had taken to reach from source to destination .This column is very important because based on the route they took will directly effect on the price of the flight.</p><pre># As this is case of Ordinal Categorical type we perform LabelEncoder<br># Here Values are assigned with corresponding keys<br><br>train_data.replace({&#39;non-stop&#39;:0, &#39;1 stop&#39;: 1, &#39;2 stops&#39;:2, &#39;3 stops&#39;: 3, &#39;4 stops&#39;: 4}, inplace=True)<br>train_data.head()</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/818/0*LiwhlFAZm5epr6Q8.png" /></figure><h3>Feature Scaling</h3><p>In most cases, the numerical features of the dataset do not have same range and they differ from each other. So, we need to scale our feature. This process is not mandatory for many algorithms like tree base algorithms, but it might be still nice to apply. However, the algorithms based on distance calculations such as k-NN or K-means, need to have scaled continuous features as model input.</p><ul><li>Standardization If the standard deviation of features is different, their range also would differ from each other. This reduces the effect of the outliers in the features. In the following formula of standardization, the mean is shown as μ and the standard deviation is shown as σ. z = (x-μ)/σ</li><li>Normalization Normalization (or min-max scaler) scales all values in a fixed range between 0 and 1. This transformation does not change the distribution of the feature and due to the decreased standard deviations, the effects of the outliers increases. Therefore, before normalization, it is recommended to handle the outliers. Xnorm = (X-Xmin)/(Xmax-Xmin).</li></ul><pre>from sklearn.preprocessing import RobustScaler<br>sc = RobustScaler()<br>X_train_sc = sc.fit_transform(X_train)<br>X_test_sc = sc.transform(X_test)</pre><h3>Splitting the data</h3><p>The fundamental goal of an ML model is to make accurate predictions on future data instances beyond those used to train models. Before using an ML model to make predictions, we need to evaluate the predictive performance of the model. To estimate the quality of an ML models predictions with data it has not seen, so we have to split our data into training and testing data. This allows us to evaluate our model on unseen data.</p><pre>from sklearn.model_selection import train_test_split<br>X_train, X_test, y_train, y_test = train_test_split(<br>X, y, test_size = 0.2, random_state = 42)</pre><h3>Feature Selection</h3><p>Feature Selection is the process to automatically select those features which contribute most to our model. Having irrelevant features in our data don’t increase the accuracy of our model. We want to remove those features which doesn’t have significant effect. We can use seaborn library to plot heat map of study the correlation between features.</p><pre># Finds correlation between Independent and dependent attributes<br>plt.figure(figsize = (18,18))<br>sns.heatmap(train_data.corr(), annot = True, cmap = &quot;RdYlGn&quot;)<br>plt.show()</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/528/0*t8uJExGddjoPT4Lc.png" /></figure><p>We see our most of the features are not highly correlated with each others. Two features Number of Total_stops and Price are correlated with each other. Also, The total stops and Duration is highly correlated. Instead, if two features are perfectly correlated, then one doesn’t add any additional information. So removing either of the features don’t affect the accuracy of model. Now, we are using Extra Tree Regressor for feature selection and these are the top 10 important features from dataset.</p><pre># Important feature using ExtraTreesRegressor<br><br>from sklearn.ensemble import ExtraTreesRegressor<br>selection = ExtraTreesRegressor()<br>selection.fit(X, y)<br>print(selection.feature_importances_)<br><br>feat_importances = pd.Series(selection.feature_importances_, index=X.columns)<br>feat_importances.nlargest(10).plot(kind=&#39;barh&#39;)<br>plt.show()</pre><p>Total steps is one of the most important feature and which highly correlated with the Price.</p><h3>Model Creation</h3><p>Choosing the right algorithm is also a important phase of data science project. Since our project is a regression problem we can choose variety of regression algorithm to make prediction. Some of the algorithms we have used are listed below:</p><ol><li>Linear regression</li><li>K Nearest neighbors</li><li>Random Forest</li><li>Decision tree</li><li>Xg boost regressor</li></ol><h3>Linear regression</h3><p>Linear Regression is the machine learning algorithm based on supervised learning. Regression models a target prediction value based on independent variables. It is mostly used for find out the relationship between variables and forecasting. Different regression models differ based on kind of relationship between dependent and independent variables. Linear Regression performs task to predict the dependent variable value (y) based on a given independent variables(x).So, this regression technique finds out a linear relationship between x (input) and y (output). Hence, the name is Linear Regression.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/686/0*GBsbYj2Kf7QTqH3h.png" /></figure><p>In the figure above, X (input) is the work experience and Y (output) is the salary of a person. The straight line in the diagram is the best fit line. The main goal of the simple linear regression is to consider the given data points and plot the best fit line to fit the model in the best way possible.</p><h3>K Nearest neighbors</h3><p>The KNN algorithm is simple, easy to implement supervised machine learning algorithm that can be used to both classification and regression problems. KNN is also a non-parametric learning algorithm because it doesn’t have any assumption about the data. The KNN algorithm uses the feature similarity to predict the value of any new data points. It computes the distance between the new data points and every training data points using Euclidean distance. Then, select the k closest instances and their labels/class. In output, those label is most frequently occur and the new data point will be belong into that class. Below is the figure how KNN algorithm works.</p><p>The KNN takes less amount of time while training and training time is constant. It takes more amount of time while testing. However, testing time is depends on the amount of dataset. This algorithm can be computationally expensive because we need to store all training example and compute distance to all training example for a single prediction.</p><h3>Random Forest</h3><p>Random Forest is the ensemble learning method which is used for both classification as well as regression. It is the one of the most used algorithm because of high accuracy. The main idea of random forest is it builds the multiple decision trees and merges them together to get more accurate and better prediction. First of all, random forest selects the samples randomly from a given dataset. Next, this algorithm will construct the decision tree for every sample. Then, it will get the result from every decision tree. Now, voting will be performed for every predicted result. At last, select the most voted prediction result as the final prediction result.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/577/0*lmmN_ktmwYgcUJdx.png" /></figure><p>In Decision Tree, the over-fitting is main problem which is solved by using random forest. Instead of single decision tree, it creates multiple trees and it reduces the over-fitting by averaging/combining the result.It also work well with large amount of dataset but the complexity is very high. More computational resources are required to implement Random Forest algorithm.</p><h3>Decision Tree</h3><p>Decision Trees are non-parametric supervised learning method used for both classification and regression. Decision Tree builds classification and regression models in the form of tree structure. Decision tree handle the both categorical and numerical data. It iteratively divides attributes into two groups which are the most dominant attribute and others to construct a tree. Then, it calculates the entropy and information gain of each attribute. In this way, the most dominant attribute can be found. After then, the most dominant one is put on tree as decision tree (root node). Entropy and gain score would be again calculated among the other attributes. This procedure will continue until a reaching for that branch.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/700/0*rUX80xHZD8TTk4MW.png" /></figure><p>Decision Tree is look like a simple if else statement. It can handle non linear parameter efficiently and it has so many benefits. However, the main problem with Decision tree is over-fitting because it creates tree completely to the depth. It doesn’t work well with the large amount of dataset.</p><h3>Xg boost regressor</h3><p>XGBoost is a supervised machine learning algorithm that stands for Extreme Gradient Boosting. It is used for both classification and regression problems. It uses the binary tree while constructing the tree and it is highly efficient algorithm. XGBoost is one of the popular algorithms for competition because of its performance. It used the boosting technique which helps to reduce the error made by previous models by adding some weights to the models. It supports the parallel processing and can be run on both single and distributed system like Spark and Hadoop. Its working mechanism is similar like a gradient boosting. The base algorithm reads the data and assigns equal weight to each sample observation. False predictions are assigned to the next base learner with a higher weightage on these incorrect predictions. Repeat unless and until the algorithm can correctly classify the output.</p><h3>Model Evaluation</h3><p>Here are three common evaluation metrics for regression problems:</p><p>Mean Absolute Error (MAE) is the mean of the absolute value of the errors: In equation form, it looks like this</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/483/0*fbxfl02tdU243phu.png" /></figure><p>Mean Squared Error (MSE) is the mean of the squared errors:. In equation form, it looks like this:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/511/0*buSMEeSlrJF0z8gI.png" /></figure><p>Root Mean Squared Error (RMSE) is the square root of the mean of the squared errors:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/565/0*mSdtWtoxYD-Mfj9D.png" /></figure><p>comparing these metrics:</p><ul><li>MAE is the easiest to understand, because it’s the average error.</li><li>MSE is more popular than MAE, because MSE “punishes” larger errors, which tends to be useful in the real world.</li><li>RMSE is even more popular than MSE, because RMSE is interpretable in the “y” units. All of these are loss functions, because we want to minimize them.</li></ul><p><strong>R-Squared</strong></p><p>R-Squared is the statistical measure of how close the data are to the fitted regression line. It is also known as the coefficient of determination or coefficient of multiple determination for multiple regression. R-squared comes with an inherent problem – additional input variables will make the R-squared stay the same or increase (this is due to how the R-squared is calculated mathematically). Therefore, even if the additional input variables show no relationship with the output variables, the R-squared will increase.</p><p>Adjusted R-squared R-squared measures the proportion of the variation in your dependent variable (Y) explained by your independent variables (X) for a linear regression model. Adjusted R-squared adjusts the statistic based on the number of independent variables in the model.R2 shows how well terms (data points) fit a curve or line. Adjusted R2 also indicates how well terms fit a curve or line, but adjusts for the number of terms in a model. If you add more and more useless variables to a model, adjusted r-squared will decrease. If you add more useful variables, adjusted r-squared will increase.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/456/0*aY9BLeEK96HZ1JdT.png" /></figure><p>Evaluation Metrics for Linear Regression</p><pre>from sklearn import metrics<br>print(&quot;MAE&quot;, metrics.mean_absolute_error(y_test,y_pred))<br>print(&quot;MSE&quot;, metrics.mean_squared_error(y_test,y_pred))<br>print(&quot;RMSE&quot;, np.sqrt(metrics.mean_squared_error(y_test,y_pred)))</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/589/0*ahAfNk1GhPIfMlpq.png" /></figure><p>The R-Squared Score is 62%</p><p>Evaluation metrics for KNeighborsRegressor</p><pre>from sklearn import metrics<br>print(&quot;MAE&quot;, metrics.mean_absolute_error(y_test,y_pred))<br>print(&quot;MSE&quot;, metrics.mean_squared_error(y_test,y_pred))<br>print(&quot;RMSE&quot;, np.sqrt(metrics.mean_squared_error(y_test,y_pred)))</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/542/0*hV5E_oGma-Ev05yS.png" /></figure><p>The R-Squared Score is 47%</p><p>Evaluation metrics for Random Forest regressor</p><pre>from sklearn import metrics<br>print(&quot;MAE&quot;, metrics.mean_absolute_error(y_test,y_pred))<br>print(&quot;MSE&quot;, metrics.mean_squared_error(y_test,y_pred))<br>print(&quot;RMSE&quot;, np.sqrt(metrics.mean_squared_error(y_test,y_pred)))</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/637/0*Uo4mJy42ls64EvMh.png" /></figure><p>The R-Squared Score is 81%</p><p>Evaluation metrics for Decision Tree regressor</p><pre>from sklearn import metrics<br>print(&quot;MAE&quot;, metrics.mean_absolute_error(y_test,y_pred))<br>print(&quot;MSE&quot;, metrics.mean_squared_error(y_test,y_pred))<br>print(&quot;RMSE&quot;, np.sqrt(metrics.mean_squared_error(y_test,y_pred)))</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/620/0*qwenEaoflmYLTiWn.png" /></figure><p>The R-Squared Score is 74%</p><p>Evaluation metrics for Xgboost regressor</p><pre>from sklearn import metrics<br>print(&quot;MAE&quot;, metrics.mean_absolute_error(y_test,y_pred))<br>print(&quot;MSE&quot;, metrics.mean_squared_error(y_test,y_pred))<br>print(&quot;RMSE&quot;, np.sqrt(metrics.mean_squared_error(y_test,y_pred)))</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/617/0*zY5gJ6QTwgzVHXTh.png" /></figure><p>The R-Squared Score is 84%</p><h3>Conclusion</h3><p>In this article, a Machine Learning model is developed to predict the airlines fare. In this type of problem Feature Engineering is the most crucial think . You can see how we have handled the categorical and numerical data and also how we build different ML model on the same dataset . We also check the RMSE score of each model so that we can understand how it should perform in our test dataset. With the help of the above techniques, proposed model is able to predict the flight fare with R squared score of 84.59%. However, there is still ways to do improvement in this model. In the future, our model can be predict the flight fare more accurately, if we get some of information such as seat location, when ticket was booked, special occasion on departure date etc.</p><h3>Links</h3><p>The code for the project can be found <a href="https://github.com/Manojkumal/Flight-Fare-Prediction">Here</a></p><p>The original dataset can be found <a href="https://www.kaggle.com/nikhilmittal/flight-fare-prediction-mh">Here</a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=142c92e4652a" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Name Entity Recognition for Nepali text using Spacy]]></title>
            <link>https://medium.com/@manojkumal/name-entity-recognition-for-nepali-text-using-spacy-eafeea0ac00b?source=rss-115b91c97fe4------2</link>
            <guid isPermaLink="false">https://medium.com/p/eafeea0ac00b</guid>
            <dc:creator><![CDATA[Manoj Kumal]]></dc:creator>
            <pubDate>Wed, 12 Oct 2022 06:25:44 GMT</pubDate>
            <atom:updated>2022-10-13T12:52:06.172Z</atom:updated>
            <content:encoded><![CDATA[<h3>Introduction</h3><p>In Nepal, only few people have worked with Nepali language dataset in NER. However, the results were not so good as well as the process was complicated. That’s why we have practised different kinds of approaches for better results and are easy to understand. Our main objective is to create a better NER model for Nepali language which could help to improve the accuracy to extract the information. Meanwhile, we have worked with Nepali Citizenship cards.</p><h3><strong>Table of Contents</strong></h3><ol><li>What is Named Entity Recognition?</li></ol><p>2. How does NER work?</p><p>3. Preparing Nepali Dataset For Training</p><p>4. Training NER with Spacy</p><p>5. Final result</p><p>6. Conclusion</p><h3><strong>What is Named Entity Recognition?</strong></h3><p>Named Entity Recognition (NER) is an application of Natural Language Processing (NLP) that processes large amounts of unstructured human language to locate and classify named entities in text into predefined categories such as the names of persons, organisations, locations, etc. It is a supervised learning problem. This is how a typical NER workflow looks like:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/512/1*8V-AMPbuBdMS0xo-hajsbQ.png" /><figcaption>Figure: Name Entity Recognition</figcaption></figure><h3><strong>Why Named Entities?</strong></h3><p>A named entity is a “real-world object” that’s assigned a name — for example, a person, a country, a product or a book title. SpaCy can recognize various types of named entities in a document, by asking the model for a prediction. Because models are statistical and strongly depend on the examples they were trained on, this doesn’t always work perfectly and might need some tuning later, depending on your use case.</p><h3><strong>How does NER work?</strong></h3><p>The first step in NER is extracting the information by detecting and preparing the named entities mentioned in the document, paragraph, sentence and texts. The entire extraction phase includes tagging the speech, detecting the boundaries of sentence, capitalization rules and coreference in the documents that more important to use and find more specific terms in the search.</p><p>The next process in NER is searching the entity candidates to mention in the document. The searcher maintains the balance with accuracy and recall the right entity while keeping a small set of entities in order to reduce the calculation needed for recognizing such entities.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/512/1*RaiibVY6oKbSGXw162sXiw.png" /><figcaption>Figure: Spacy Library Architecture</figcaption></figure><p><a href="https://blog.vsoftconsulting.com/blog/understanding-named-entity-recognition-pre-trained-models">Image Source</a></p><p>The exact architecture for the SpaCy NER model hasn’t been published yet. However, we can have a look at one of SpaCy’s official videos to understand more about the model. According to the video, the SpaCy NER system contains a word embedding strategy using subword features and “Bloom” embed, and a deep convolutional neural network with residual connections. The system is designed to give a good balance of efficiency, accuracy, and adaptability.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/512/1*Z-5EUggZeY71BLGdFruoOg.png" /><figcaption>Figure: SpaCy Pipeline</figcaption></figure><p>We trained the SpaCy NER model with the custom training dataset we prepared. As we wanted the model to be as generalised as possible, we adopted various training techniques such as dropout and shuffling the data on each iteration. After the training, we evaluated our model on the training dataset and computed various metrics such as accuracy score, precision, recall and f-score.</p><h3><strong>Preparing Nepali Dataset For Training</strong></h3><p>For the training, we have taken 100 citizenship data which we have extracted through tesseract where the extracted information is copied in the text file. The reason behind writing the information in the text file is that in our approach the annotation can be done on text file only.</p><h3><strong>Step 1:</strong></h3><p>First we create the POS tag for our text file. The reason behind creating the tag is that we are labelling the entities so that when we test our model it can predict the correct entities based on tags.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*mTHXB5EkYWKNv4rX1M-ekQ.png" /></figure><p>Based on the nepali citizenship card format we have created 13 tags for our dataset.</p><h3>Step 2:</h3><p>Here we annotate the required entities with the specific tags.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*f7Qaj0Pb6iClaHSPx5Oqag.png" /></figure><h3>Step 3:</h3><p>The annotated file is in Json but spacy version 3.0 does not work with json file so we have to convert the json file in spacy.</p><pre>for text, annot in tqdm(TRAIN_DATA[&#39;annotations&#39;]):</pre><pre>    doc1 = nlp.make_doc(text)</pre><pre>    ents = []</pre><pre>    for start, end, label in annot[&quot;entities&quot;]:</pre><pre>        span = doc1.char_span(start, end, label=label,                    alignment_mode=&quot;contract&quot;)</pre><pre>        if span is None:</pre><pre>             print(&quot;Skipping entity&quot;)</pre><pre>        else:</pre><pre>             ents.append(span)</pre><pre>        doc1.ents = ents</pre><pre>        db.add(doc1)</pre><pre>db.to_disk(&quot;./train_data.spacy&quot;)</pre><h3>Step 4</h3><p>Now we create the pipeline using a config file which you can download directly from spacy. Pipeline is created so that we can add our data and train it. Below you can see how it is done:</p><p>First, we create a config file for our model.</p><pre>!python3 -m spacy init fill-config base_config.cfg config.cfg</pre><p>And in second, we initialise the pipeline by adding our data to be trained.</p><pre>! python3 -m spacy train config.cfg --output ./ --paths.train ./train_data.spacy --paths.dev ./test_data.spacy</pre><h3>Final Result:</h3><p>After our model is trained, we load the best model created while training the pipeline.</p><pre>nlp_ner = spacy.load(&quot;model-best&quot;)</pre><pre>spacy.displacy.render(doc, style=&quot;ent&quot;, jupyter=True)</pre><p>Now, we check how the best-model predicts the entities.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*2pM0yvQbSxSBg9zzbdhOFw.png" /></figure><p>As you can see, the output is quite good. It has extracted maximum entities. It would been better if we had a huge amount of data.</p><p><strong>Note: For the demonstration we have used dummy data instead of actual citizenship information. So, the details presented in above pictures are not of actual person.</strong></p><p>This Project is done by</p><p><a href="https://www.diwaspandey.com.np/">Diwas Panday</a></p><p><a href="https://www.ishparshuprety.com.np/">Ishparsh Uprety</a></p><p><a href="https://medium.com/@manojkumal">Manoj Kumal</a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=eafeea0ac00b" width="1" height="1" alt="">]]></content:encoded>
        </item>
    </channel>
</rss>