<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Abhilasha Thakur on Medium]]></title>
        <description><![CDATA[Stories by Abhilasha Thakur on Medium]]></description>
        <link>https://medium.com/@methakur77?source=rss-e01f7908f0e7------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/1*FNMjOTirFJJCXtGEaQ9JoQ.jpeg</url>
            <title>Stories by Abhilasha Thakur on Medium</title>
            <link>https://medium.com/@methakur77?source=rss-e01f7908f0e7------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Tue, 17 Nov 2026 20:00:14 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@methakur77/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[A Deep Dive into Naive Bayes Classifiers]]></title>
            <link>https://medium.com/@methakur77/a-deep-dive-into-naive-bayes-classifiers-8d3edcff3b83?source=rss-e01f7908f0e7------2</link>
            <guid isPermaLink="false">https://medium.com/p/8d3edcff3b83</guid>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[scikit-learn]]></category>
            <category><![CDATA[naive-bayes-classifier]]></category>
            <category><![CDATA[python]]></category>
            <dc:creator><![CDATA[Abhilasha Thakur]]></dc:creator>
            <pubDate>Sat, 20 Apr 2024 13:49:05 GMT</pubDate>
            <atom:updated>2024-04-20T13:49:05.982Z</atom:updated>
            <content:encoded><![CDATA[<p>In the ever-growing realm of machine learning, classification reigns supreme. It’s the art of sorting data points into predefined categories, a skill crucial for tasks ranging from spam filtering to medical diagnosis. With a vast array of classification algorithms at our disposal, choosing the right one can feel like navigating a labyrinth. But fear not, for this article delves into the world of Naive Bayes classifiers, a surprisingly simple yet extraordinarily effective tool for tackling a multitude of classification challenges.</p><h4><strong>The Fundamental Intuition Behind Naive Bayes</strong></h4><p>At the heart of Naive Bayes lies a cornerstone of probability — Bayes’ theorem. This powerful theorem allows us to calculate the probability of an event (let’s assume for purchase prediction whether a person will “buy or not buy”) occurring given certain conditions (Which day it is?, Free delivery? , On discount?). Imagine you want to automatically classify whether a person will buy or not given certain conditions. Here’s where Bayes’ theorem swoops in:</p><p>P(A|B)= (P(B|A) * P(A)) |P(B)</p><p>Calculate conditional probability of purchase on following combination where B equals Day =Holiday , Discount=Yes , Free delivery=Yes</p><p>Let A= Buy</p><p>Applying Bayes Theorem:-</p><p>(P(Discount=Yes| Buy) * P(Free Delivery=Yes | Buy) * P( Day=Holiday |Buy) * P(Buy)) / P(Discount=Yes) * P(Delivery=Yes) * P(Day=Holiday)</p><h4><strong>The Naive Assumption: A Closer Look</strong></h4><p>There’s a key assumption that underpins Naive Bayes — the features used for classification (like keywords) are independent of each other given the class label (Which day it is?, Free delivery? , On discount?). In the real world, features might be interconnected. This “naive” assumption, where the algorithm gets its name, might raise an eyebrow.</p><p>However, the beauty of Naive Bayes lies in its ability to perform admirably despite this simplification. Why? Because the ease of calculating probabilities with the independence assumption can often outweigh the potential shortcomings arising from slightly interrelated features in real-world scenarios.</p><h4><strong>Variants for Different Data Types</strong></h4><p>One of the strengths of Naive Bayes is its adaptability. Different variants cater to various data types, ensuring a perfect fit for your classification problem:</p><ul><li><strong>Gaussian Naive Bayes</strong>: This variant shines when dealing with continuous features like temperature or height. It assumes a normal distribution (bell-shaped curve) for the continuous features within each class.</li><li><strong>Multinomial Naive Bayes</strong>: This is the go-to choice for discrete features like word counts in emails or document categories. It calculates probabilities based on the frequency of these features appearing within each class.</li><li><strong>Bernoulli Naive Bayes</strong>: This variant tackles binary features like presence or absence of a specific word. It calculates the probability of a feature being present or absent given the class label.</li></ul><p>These variants fine-tune the probability calculations to perfectly match the specific characteristics of the data they’re working with</p><p>Code Example:</p><pre>from sklearn.naive_bayes import MultinomialNB<br>from sklearn.preprocessing import OrdinalEncoder<br><br># Training data<br>features = [<br>    [&quot;Weekday&quot;, &quot;Free Delivery&quot;, &quot;Discount&quot;],<br>    [&quot;Weekday&quot;, &quot;No Free Delivery&quot;, &quot;Discount&quot;],<br>    [&quot;Weekday&quot;, &quot;No Free Delivery&quot;, &quot;No Discount&quot;],<br>    [&quot;Weekend&quot;, &quot;Free Delivery&quot;, &quot;Discount&quot;],<br>    [&quot;Weekend&quot;, &quot;No Free Delivery&quot;, &quot;Discount&quot;],<br>    [&quot;Weekend&quot;, &quot;No Free Delivery&quot;, &quot;No Discount&quot;],<br>    [&quot;Holiday&quot;, &quot;Free Delivery&quot;, &quot;Discount&quot;],<br>    [&quot;Holiday&quot;, &quot;No Free Delivery&quot;, &quot;Discount&quot;],<br>    [&quot;Holiday&quot;, &quot;No Free Delivery&quot;, &quot;No Discount&quot;],<br>]<br><br>labels = [&quot;Buy&quot;, &quot;Not Buy&quot;, &quot;Not Buy&quot;, &quot;Buy&quot;, &quot;Buy&quot;, &quot;Not Buy&quot;, &quot;Buy&quot;, &quot;Buy&quot;, &quot;Not Buy&quot;]<br><br># Encode features<br>encoder = OrdinalEncoder()<br>encoded_features = encoder.fit_transform(features)<br><br># Encode labels<br>label_encoder = OrdinalEncoder()<br>encoded_labels = label_encoder.fit_transform([[label] for label in labels])<br><br># Create and train the classifier<br>classifier = MultinomialNB()<br>classifier.fit(encoded_features, encoded_labels.ravel())  # ravel to convert the labels to 1D array<br><br># Test data<br>test_data = [<br>    [&quot;Weekday&quot;, &quot;Free Delivery&quot;, &quot;Discount&quot;],<br>    [&quot;Weekend&quot;, &quot;No Free Delivery&quot;, &quot;No Discount&quot;],<br>    [&quot;Holiday&quot;, &quot;No Free Delivery&quot;, &quot;Discount&quot;]<br>]<br><br># Encode test data<br>encoded_test_data = encoder.transform(test_data)<br><br># Make predictions<br>predictions = classifier.predict(encoded_test_data)<br><br># Decode predictions<br>decoded_predictions = label_encoder.inverse_transform(predictions.reshape(-1, 1)).ravel()  # Reshape predictions to 2D array before inverse_transform<br><br># Print predictions<br>for i, data in enumerate(test_data):<br>    print(f&quot;Prediction for {data}: {decoded_predictions[i]}&quot;)</pre><h4>Advantages and Limitations</h4><p>One of the key advantages of Naive Bayes Classifier is its simplicity and efficiency, making it suitable for large datasets. It also performs well in high-dimensional spaces and can handle missing data gracefully. However, its assumption of feature independence may not hold true in all scenarios, and it may struggle with correlated features.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=8d3edcff3b83" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Semantic search powered by Large Language Model]]></title>
            <link>https://medium.com/@methakur77/semantic-search-powered-by-large-language-model-c930574d6d39?source=rss-e01f7908f0e7------2</link>
            <guid isPermaLink="false">https://medium.com/p/c930574d6d39</guid>
            <category><![CDATA[python]]></category>
            <category><![CDATA[semantic-search]]></category>
            <category><![CDATA[artificial-intelligence]]></category>
            <category><![CDATA[deep-learning]]></category>
            <category><![CDATA[large-language-models]]></category>
            <dc:creator><![CDATA[Abhilasha Thakur]]></dc:creator>
            <pubDate>Sat, 20 Apr 2024 05:28:22 GMT</pubDate>
            <atom:updated>2024-04-20T05:31:13.873Z</atom:updated>
            <content:encoded><![CDATA[<p>In today’s information age , navigating the ever expanding digital ocean can be overwhelming. Traditional keyword-based search engines often fall short, delivering results based on literal matches rather than understanding the true intent behind a query.</p><p>This is where semantic search comes in. It’s a revolutionary approach that transcends simple keyword matching and delves into the deeper meaning of a user’s search. Here, we’ll explore how semantic search leverages the power of large language models to deliver a more relevant and insightful search experience.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*bIrXkap_qvd2rGz1.jpg" /></figure><h3>Beyond Keywords: Understanding the Why</h3><p>Traditional search engines rely on keywords. You type in a string of words, and the engine returns results containing those exact keywords. But what if you’re phrasing your query differently? Imagine searching for “treating a broken leg at home.” You might not find the most helpful information because the relevant articles might use terms like “fracture care” or “immobilization techniques.”</p><p>Semantic search bridges this gap. It goes beyond literal matches and focuses on understanding the context and intent behind a user’s query. This is achieved through the magic of text embeddings.</p><h3>Cracking the Code: Text Embeddings and LLMs</h3><p>Text embeddings are like secret codes that unlock the meaning of text. Large language models, like the ones you might encounter on Bard: [Google AI blog: Introducing LaMDA] or GPT-3, are trained on massive amounts of text data. This allows them to generate these embeddings, which are essentially numerical representations of a text’s meaning.</p><p>Imagine a multi-dimensional space where words and phrases are positioned based on their semantic relationships. Words like “doctor,” “patient,” and “diagnosis” would be close together, while “car” and “engine” would reside in a different area.</p><p>Semantic search uses these embeddings to represent both user queries and documents within your search database. By analyzing the distances between these embeddings in the multi-dimensional space, the search engine can identify documents that are most semantically similar to the user’s intent, even if they don’t contain the exact keywords.</p><h3>The Power of LLMs in Semantic Search</h3><p>Large language models play a crucial role in semantic search by:</p><ul><li><strong>Generating High-Quality Embeddings:</strong> LLMs are trained on vast amounts of text data, allowing them to capture intricate nuances of language and generate more accurate semantic representations.</li><li><strong>Understanding Context:</strong> LLMs can analyze the context of a query, considering surrounding words and sentence structure to grasp the user’s true intent.</li><li><strong>Handling Synonyms and Paraphrases:</strong> Semantic search powered by LLMs can identify documents that use synonyms or rephrased versions of the keywords in the query, ensuring a broader and more relevant search.</li></ul><h3>Three Major Categories of Language-Model-based Search Systems</h3><h4><strong>Dense Retrieval</strong></h4><p>Imagine a multi-dimensional space where search queries and documents are represented as points. Dense retrieval utilizes LLMs to create these embeddings, allowing the system to identify documents with similar meaning to the user’s query, even if the exact keywords aren’t present. Below code represents dense retrieval using pre-trained LLM like sentence transformers.</p><pre>from sentence_transformers import SentenceTransformer<br><br># Load the pre-trained Sentence Transformer model<br>model = SentenceTransformer(&#39;all-mpnet-base-v2&#39;)<br><br># Define some example queries and documents<br>queries = [&quot;What is the capital of France?&quot;, &quot;Where is the Eiffel Tower located?&quot;]<br>documents = [<br>    &quot;France is a country located in Western Europe. Paris is the capital of France.&quot;,<br>    &quot;The Eiffel Tower is a wrought-iron lattice tower on the Champ de Mars in Paris, France.&quot;<br>]<br><br># Encode the queries and documents using the LLM into embeddings<br>query_embeddings = model.encode(queries)<br>document_embeddings = model.encode(documents)<br><br># Iterate through each query embedding and compare<br># it with all document embeddings using cosine similarity. <br># Cosine similarity measures how similar two vectors are directionally.<br>import numpy as np<br>for i, query_embedding in enumerate(query_embeddings):<br>  max_similarity = -1<br>  closest_document = None<br>  for j, document_embedding in enumerate(document_embeddings):<br>    similarity = np.dot(query_embedding, document_embedding) / (np.linalg.norm(query_embedding) * np.linalg.norm(document_embedding))<br>    if similarity &gt; max_similarity:<br>      max_similarity = similarity<br>      closest_document = documents[j]<br>  print(f&quot;Query: {queries[i]}&quot;)<br>  print(f&quot;Most Relevant Document: {closest_document}&quot;)<br>  print(f&quot;Similarity Score: {max_similarity}\n&quot;)</pre><p>Output:</p><pre>Query: What is the capital of France?<br>Most Relevant Document: France is a country located in Western Europe. Paris is the capital of France.<br>Similarity Score: 0.6987370848655701<br><br>Query: Where is the Eiffel Tower located?<br>Most Relevant Document: The Eiffel Tower is a wrought-iron lattice tower on the Champ de Mars in Paris, France.<br>Similarity Score: 0.7806955575942993</pre><h4><strong>Reranking</strong></h4><p>Not all retrieved documents are created equal. Reranking techniques, powered by LLMs, analyze the retrieved documents and re-order them based on their relevance to the user’s intent. This ensures the most pertinent information surfaces the top.</p><pre>import numpy as np<br>from sentence_transformers import SentenceTransformer<br>from transformers import AutoTokenizer, AutoModelForSequenceClassification<br><br># Load pre-trained models (replace with your desired models)<br>retrieval_model = SentenceTransformer(&#39;all-mpnet-base-v2&#39;)<br>sentiment_model_name = &quot;distilbert-base-uncased-finetuned-sst-2-english&quot;<br>sentiment_tokenizer = AutoTokenizer.from_pretrained(sentiment_model_name)<br>sentiment_model = AutoModelForSequenceClassification.from_pretrained(sentiment_model_name)<br><br>def preprocess_text(text):<br>  &quot;&quot;&quot;<br>  Preprocesses text for both retrieval and sentiment analysis.<br>  &quot;&quot;&quot;<br>  # Text cleaning steps (e.g., lowercase, remove punctuation) can be added here<br>  return text.lower()<br><br>def encode_text(texts):<br>  &quot;&quot;&quot;<br>  Encodes a list of text strings using the retrieval model.<br>  &quot;&quot;&quot;<br>  preprocessed_texts = [preprocess_text(text) for text in texts]<br>  return retrieval_model.encode(preprocessed_texts)<br><br>def retrieve_documents(query, documents, k=10):<br>  &quot;&quot;&quot;<br>  Performs dense retrieval using cosine similarity.<br>  &quot;&quot;&quot;<br>  query_embedding = encode_text([query])[0]<br>  document_embeddings = encode_text(documents)<br>  similarities = np.dot(query_embedding, document_embeddings.T) / (<br>      np.linalg.norm(query_embedding) * np.linalg.norm(document_embeddings, axis=1)<br>  )<br>  top_k_indices = np.argsort(similarities)[-k:]<br>  return [documents[i] for i in top_k_indices[::-1]], similarities[top_k_indices][::-1]<br><br>def predict_sentiment(query, document):<br>  &quot;&quot;&quot;<br>  Predicts sentiment (positive/negative) using the fine-tuned LLM.<br>  &quot;&quot;&quot;<br>  encoded_input = sentiment_tokenizer(query, document, return_tensors=&quot;pt&quot;)<br>  with torch.no_grad():<br>    output = sentiment_model(**encoded_input)<br>  sentiment = output.logits.squeeze().item()  # Extract sentiment score<br>  return sentiment<br><br>def rerank_with_sentiment(query, documents, retrieval_scores, alpha=0.7):<br>  &quot;&quot;&quot;<br>  Reranks documents based on a weighted combination of retrieval score and sentiment score.<br>  &quot;&quot;&quot;<br>  sentiment_scores = [predict_sentiment(query, doc) for doc in documents]<br>  reranked_scores = alpha * retrieval_scores + (1 - alpha) * sentiment_scores<br>  sorted_indices = np.argsort(reranked_scores)[::-1]<br>  return [documents[i] for i in sorted_indices], reranked_scores[sorted_indices]<br><br>def semantic_search(query, documents, k=5):<br>  &quot;&quot;&quot;<br>  Main function for semantic search with retrieval and reranking.<br>  &quot;&quot;&quot;<br>  retrieved_docs, retrieval_scores = retrieve_documents(query, documents, k)<br>  reranked_docs, reranked_scores = rerank_with_sentiment(query, retrieved_docs, retrieval_scores)<br>  return reranked_docs, reranked_scores<br><br># Example usage<br>query = &quot;best laptops for students on a budget&quot;<br>documents = [<br>    &quot;The latest gaming laptops offer powerful performance but can be expensive.&quot;,<br>    &quot;A Chromebook is a great option for students who need a basic laptop for browsing and light tasks.&quot;,<br>    &quot;This mid-range laptop is perfect for students who balance productivity and entertainment.&quot;,<br>    &quot;This article highlights factors to consider when buying a student laptop.&quot;,<br>    &quot;This high-end laptop is ideal for demanding tasks like video editing and design.&quot;<br>]<br><br>reranked_docs, reranked_scores = semantic_search(query, documents)<br><br># Print top reranked results<br>for doc, score in zip(reranked_docs[:3], reranked_scores[:3]):<br>  print(f&quot;Document: {doc}\n Reranked Score: {score}\n&quot;)</pre><h3><strong>Generative Search</strong></h3><p>This approach takes semantic search a step further. Here, LLMs go beyond retrieving existing documents and can potentially generate new content that addresses the user’s query.</p><pre>from transformers import AutoTokenizer, AutoModelForSeq2SeqLM<br><br># Load pre-trained LLM for text generation (replace with your desired model)<br>model_name = &quot;t5-base&quot;  # Example model for text generation<br>tokenizer = AutoTokenizer.from_pretrained(model_name)<br>model = AutoModelForSeq2SeqLM.from_pretrained(model_name)<br><br>def generate_content(query, max_length=500):<br>  &quot;&quot;&quot;<br>  Generates new content based on the user&#39;s query using a generative LLM.<br>  &quot;&quot;&quot;<br>  encoded_prompt = tokenizer(query, return_tensors=&quot;pt&quot;)<br>  output = model.generate(**encoded_prompt, max_length=max_length)<br>  generated_content = tokenizer.decode(output[0], skip_special_tokens=True)<br>  return generated_content.strip()<br><br># Example usage<br>query = &quot;benefits of using a standing desk&quot;<br>generated_content = generate_content(query)<br><br>print(f&quot;Query: {query}&quot;)<br>print(f&quot;Generated Content:\n{generated_content}&quot;)</pre><p>Output:</p><pre>Query: benefits of using a standing desk<br>Generated Content:<br>a standing desk. desk. standing desk. desk. desk. a standing desk............... standing desk. a standing desk. a standing desk. desk. a standing desk. The benefits of using a standing desk are many. The benefits of using a standing desk are many. desk are many. desk. The benefits of using</pre><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=c930574d6d39" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Classifying the Unseen: Zero-Shot Wonders]]></title>
            <link>https://medium.com/@methakur77/classifying-the-unseen-zero-shot-wonders-812c67441ad8?source=rss-e01f7908f0e7------2</link>
            <guid isPermaLink="false">https://medium.com/p/812c67441ad8</guid>
            <category><![CDATA[deep-learning]]></category>
            <category><![CDATA[python]]></category>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[large-language-models]]></category>
            <category><![CDATA[generative-ai-tools]]></category>
            <dc:creator><![CDATA[Abhilasha Thakur]]></dc:creator>
            <pubDate>Sun, 18 Feb 2024 10:01:58 GMT</pubDate>
            <atom:updated>2024-02-18T10:01:58.626Z</atom:updated>
            <content:encoded><![CDATA[<p>Classification is used to categorize data or instances into predefined classes or labels based on their features or attributes. Zero-shot classification refers to the task of classifying instances into classes that the model has never seen during training. In traditional classification tasks, a model is trained on a specific set of classes, and during testing, it is expected to classify instances into one of those known classes. However, in zero-shot classification, the goal is to classify instances into classes that were not present in the training data. This becomes extremely helpful as getting labeled data is a resource-intensive task that can require significant human labor.</p><p>Throughout this article we are going to use an example set - <a href="https://huggingface.co/datasets/rotten_tomatoes">rotten_tomatoes · Datasets at Hugging Face</a>. We are going to use movie reviews and rate them as either positive or negative</p><p>Let us first load the data and convert it into pandas dataframe.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/632/1*hu-KC3GGjUyTX3FhSG6Lcw.png" /></figure><p>We are going to discuss three ways to achieve the zero shot classification — using embeddings, natural language inference, generative models</p><p><strong>USING EMBEDDINGS</strong></p><p>When executing zero shot classification with the help of embeddings we need to be tricky. We need to describe our labels based on what they should represent. For example, a negative label for movie reviews can be described as “This is a negative movie review”. By describing and embedding the labels and documents, we have data that we can work with. To assign labels to documents, we can apply cosine similarity to the document label pairs. Cosine similarity is a similarity measure that checks how similar two vectors are to each other. See diag. below:-</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/512/1*2QIDp8O6Xrym4xgq8KO0GQ.png" /></figure><p>For each document, its embedding is compared to that of each label. The label with the highest similarity to the document is chosen.</p><p>Generate embeddings for the evaluation dataset</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/668/1*Y5gBxYhh_Z-bDexyIvlmEg.png" /></figure><p>Apply Cosine similarity to fit labels</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/697/1*AF2onZZ3tUU50foMSAWBkg.png" /></figure><p>And that is it! We only needed to come up with names for our labels to perform our classification tasks. You can validate how well your model performed as below (f1 score of 0.78):-</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/721/1*UMJ-KIveKtBfq__DuNx_Kg.png" /></figure><p><strong>USING NATURAL LANGUAGE INFERENCE</strong></p><p>Zero-shot classification can also be done using natural language inference (NLI), which refers to the task of investigating whether, for a given premise, a hypothesis is true (entailment) or false (contradiction). We can create a hypothesis of whether the premise is our target label . Example in our case it can be about of positive movie review so that whenever model finds it as entailment we can label it as positive and when contradiction as negative. See diag. below</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/878/1*_rlj95Yvp-T7_DoW-ID0vg.png" /></figure><p>We load in our transformers pipeline and run it on our evaluation dataset:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/807/1*pLvMawSTRQmXzB1JrlvI8Q.png" /></figure><p>Since this is a zero-shot classification task, no training is necessary for us to get the predictions that we are interested in. The predictions variable contains not only the prediction but also a score indicating the probability of a candidate label (hypothesis) to entail the input document (premise).</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/867/1*v5zXbGKYF9rJ4wgqfV_bfA.png" /></figure><p>Without any fine-tuning whatsoever, it received an F1-score of 0.85. We might be able to increase this value depending on how we phrase the candidate labels</p><p><strong>USING GENERATIVE MODELS</strong></p><p>With generative models , we try to guide the model towards the type of output that we are looking for. It is mainly done through prompts. What makes generative model interesting is their ability to learn from the context in prompts.<br>We first create a function that allows us to perform prediction with OpenAI’s GPT models.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/868/1*204w2-ZZsGaTpw8DbEUhvA.png" /></figure><p>Set Open AI API key</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/322/1*SIIIkHeUjn1q3QPWXpTT_A.png" /></figure><p>Define your prompt</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/897/1*LVDzt1RV0YmkN1sWyupjLw.png" /></figure><p>Predict using model</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/602/1*eq6qT0_jsmUhdkic_r2l1A.png" /></figure><p>Next, we can run this for all reviews in the evaluation dataset and look at its performance. Do note though that this requires 300 requests to OpenAI’s API:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/872/1*MufVElAFpAfDXtSk2c8z4Q.png" /></figure><p>CONCLUSION</p><p>We saw many different techniques for performing a zero shot classification tasks. No tuning at all! Classifying textual data is not as straightforward as it may seem on the surface and there is an incredible number of creative techniques for doing so.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=812c67441ad8" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Large Language Model use case design with RAG pipeline]]></title>
            <link>https://medium.com/@methakur77/large-language-model-design-with-rag-pipeline-364f2808b4d7?source=rss-e01f7908f0e7------2</link>
            <guid isPermaLink="false">https://medium.com/p/364f2808b4d7</guid>
            <category><![CDATA[generative-ai-tools]]></category>
            <category><![CDATA[large-language-models]]></category>
            <category><![CDATA[deep-learning]]></category>
            <category><![CDATA[retrieval-augmented]]></category>
            <category><![CDATA[python]]></category>
            <dc:creator><![CDATA[Abhilasha Thakur]]></dc:creator>
            <pubDate>Sat, 17 Feb 2024 15:41:15 GMT</pubDate>
            <atom:updated>2024-02-17T15:43:53.456Z</atom:updated>
            <content:encoded><![CDATA[<p>Large language models (LLMs) have undoubtedly changed the way we interact with information. However, they come with their fair share of limitations as to what we can ask of them. They are only aware of the information that they’ve been trained on and will fall short when we require them to know information beyond that. Retrieval augmented generation (RAG) based LLM applications address this exact issue and extend the utility of LLMs to our specific data sources. In this article we are going to discuss the design of RAG-based LLM application.</p><p>One of the major drawbacks of traditional language models is their inability to effectively handle long-form content. These models often struggle to maintain coherence and relevance when generating longer pieces of text, which can result in poorly constructed sentences and paragraphs</p><h3>What is Retrieval-Augmented Generation (RAG)?</h3><p>The RAG (Retrieval-Augmented Generation) architecture is a revolutionary approach to language modeling that overcomes the limitations of traditional models. It is a hybrid model that combines the best of both worlds — retrieval-based and generator-based methods.</p><p>The benefits of RAG not only include more accurate and relevant responses, but also enhance the ability to verify the sources of information, and a reduction in the likelihood of the AI generating incorrect or misleading information</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/901/1*tqy2CZgCda2r74OsyD7NQw.png" /><figcaption>RAG based LLM application design</figcaption></figure><p>The design above involved two main services — embedding, querying service.</p><p><strong>Embedding service</strong> incorporates the heart of RAG . The principle components are :-</p><p>We have the inflow of contextual data via the data pipelines which is split into chunks and then fed to embedding model to generate embeddings. Embeddings are dense numerical representations of real-world objects and relationships, expressed as a vector. The vector space quantifies the semantic similarity between categories. Embedding vectors that are close to each other are considered similar. These embeddings are then stored into a vector database. Vector databases provide the ability to store and retrieve vectors as high-dimensional points. They add additional capabilities for efficient and fast lookup of nearest-neighbors in the N-dimensional space. They are typically powered by k-nearest neighbor (k-NN) indexes and built with algorithms like the Hierarchical Navigable Small World (HNSW) and Inverted File Index (IVF) algorithms.</p><p><strong>Querying service </strong>incorporates the flow from a user query till the response.</p><p>When a user queries into the system, the query is fed into the prompt generator and optimizer. A copy of query is also fed into the embedding service to retrieve the relevant context. The retriever searches through a large corpus of data to find relevant pieces of information that can be used for text generation. Think of them as specialized librarians who know exactly which ‘books’ to pull off the ‘shelves’ when you ask a question. This relevant context is then combined with the prompt and fed into the LLM model. Without this relevant context that we retrieved, the LLM may not have been able to accurately answer our question. And as our data grows, we can just as easily embed and index any new data and be able to retrieve it to answer questions.</p><h3>Security</h3><p><strong>Multi Tenancy </strong>can be followed in this. It refers to system’s ability to server multiple users (tenants) independently and securely. This means designing a system that not only understands and retrieves information effectively but also strictly adheres to user specific data boundaries.</p><p>The keys lies within metadata. When indexing documents , we incorporate user-specific information (could be a hash of PRC group, the allowed PRC groups can be stored as configuration in db) into metadata before adding it to index. This ensures each document is uniquely tied to a group.</p><p>During query phase , the retriever uses this metadata to filter and only access documents associated with the querying user.</p><h3>Evaluation</h3><p>If we were to change something, such as our chunking logic, embedding model, LLM, etc. how can we know that we have a better configuration than before . A generative task like this is very difficult to quantitatively assess and so we need to develop reliable ways to do so.</p><p>We might need to perform both unit/component and end-to-end evaluation. Component-wise evaluation can involve evaluating our retrieval in isolation (is the best source in our set of retrieved chunks) and evaluating our LLMs response (given the best source, is the LLM able to produce a quality answer). And for end-to-end evaluation, we can assess the quality of the entire system (given the data sources, what is the quality of the response).</p><p>We can use an evaluator LLM to do so as below.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/642/1*fIJ3IcfjSiZqyMZZPX_r6A.png" /></figure><p>CONCLUSION</p><p>As use of LLM model evolves in industry, the use of in-context learning pattern( with RAG pipeline) is an important advancement for all the generative use cases.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=364f2808b4d7" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Scikit Learn — Choosing the right estimator.]]></title>
            <link>https://medium.com/@methakur77/scikit-learn-choosing-the-right-estimator-78e5934a5d8d?source=rss-e01f7908f0e7------2</link>
            <guid isPermaLink="false">https://medium.com/p/78e5934a5d8d</guid>
            <category><![CDATA[python]]></category>
            <category><![CDATA[machine-learning-ai]]></category>
            <category><![CDATA[scikit-learn]]></category>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[estimators]]></category>
            <dc:creator><![CDATA[Abhilasha Thakur]]></dc:creator>
            <pubDate>Fri, 16 Feb 2024 08:13:54 GMT</pubDate>
            <atom:updated>2024-02-16T08:13:54.810Z</atom:updated>
            <content:encoded><![CDATA[<h3>Scikit Learn — Choosing the right estimator.</h3><p>Choosing the right estimator in scikit learn depends on the type of machine learning task you’re working on.</p><p><strong>Types of machine learning problems:-</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/947/1*bJQ7X43mlQjqqoAtHKF0xQ.png" /></figure><p><strong>Classification</strong> models are used to predict a category. Some common examples are — Spam detection in email, Sentiment Analysis, Image Recognition etc.</p><p><strong>Regression</strong> models are used to predict continuous target variables based on one or more input features. Some common examples are — House Price Prediction, Stock Price Forecasting etc.</p><p><strong>Clustering</strong> models are used to group similar data points together based on certain features without knowing the predefined labels of groups. Some common examples are- Anomaly detection, Genomic data analysis, Recommendation systems etc.</p><p><strong>Dimensionality</strong> <strong>Reduction</strong> models are used to reduce the number of features or variables in dataset while preserving its essential information. This is particularly beneficial when dealing with high-dimensional data .</p><p><strong>CLASSIFICATION</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/872/1*REOYK0zzXR4s_DS3M8oYDg.png" /></figure><p>If your data size is less than 100k samples then the estimator, you might want to use is <strong>Linear Support Vector Classifier.</strong> It is suitable for scenarios where relationship between feature and target variable is approximately linear.</p><p>If Linear SVC does not work well for you and you are dealing with text data (ex- articles, emails etc.) then you might want to use <strong>NaiveBayes</strong>. This algorithm is based on Baye’s theorem, with the “naive” assumption of independence between features.</p><p>If it&#39;s not text data and linear SVC doesn’t work for you then you might want to go with <strong>Kneighbors</strong> classifier. It is based on k-nearest neighbors&#39; principle. It classifies a data point by a majority vote of its k-nearest neighbors.</p><p>If Kneighbors also doesn’t work well for you then you might go for other classification techniques, such as <strong>general purpose SVC or ensemble classifiers.</strong></p><p>If you have a very large data set greater than 100k records, then you should opt for <strong>SGD(Stochastic gradient descent)</strong> classifier. It uses stochastic gradient descent optimization to find the weights that minimize the loss function. Suitable when dealing with sparse data.</p><p>If <strong>SGD </strong>does not work well for you then go for kernel approximation.</p><p><strong>REGRESSION</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/872/1*q-gDGKgvMQ53HSzI0AXyFQ.png" /></figure><p>If you have less than 100k samples and only a few features among those samples matter i.e. low dimensionality data, you might choose to go with <strong>Lasso (Least Absolute Shrinkage and Selection Operator) or Elastic Net</strong></p><p>If your data is of high dimensionality i.e. every record has lot of features that are important then you might choose <strong>Ridge Regression</strong> or <strong>SVR (kernel=’linear’)</strong></p><p>If this results in a poor model, then you might choose <strong>SVR (kernel=’rbf’) or Ensemble Regressors</strong></p><p>If you have large data set more than 110k records, then the estimator that works best is <strong>SGD (Stochastic gradient descent) Regressor</strong></p><p><strong>CLUSTERING</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1007/1*wkAw2MoInjdrb1gXEhamPA.png" /></figure><p>If you know the number of categories into which your data will be divided (example — identifying digits from 0–9) and if you have less than 10k samples then you might choose <strong>K-Means</strong> Clustering.</p><p>If K-means do not work, then there are other options to try -<strong>Spectral Clustering</strong> and <strong>GMM (Gaussian mixture models).</strong></p><p>If you know the number of categories and the data has more than 10k records, then go for <strong>MiniBatch K-Means.</strong> It works as K-Means, but it works on batches of data therefore it can scale to very large datasets.</p><p>If you are not aware of categories in your data and you have less than 10k records, then you can choose <strong>MeanShift</strong> or <strong>VBGMM (Variational Bayesian Gaussian mixture models)</strong></p><p>If number of categories are not known and you have a very large data set, then you can choose <strong>DBSCAN (Density-based spatial clustering of application with Noise) </strong>or<strong> OPTICS (Ordering points to identify the clustering structure)</strong></p><p>DIMENSIONALITY REDUCTION</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/677/1*5b7WwXCoWPVg3ZJdng6i-w.png" /></figure><p>For Dimensionality Reduction , you can use <strong>PCA(Principal Component Analysis)</strong></p><p>If this doesn’t work well and you have less than 10k data samples then you can use <strong>Isomap</strong> or <strong>Spectral Embedding</strong>. And if that too does not work, please go for <strong>LLE (Locally linear embedding)</strong>.</p><p>If you have more than 10k data samples, <strong>kernel approximation</strong> technique can work. This technique applies mathematical function to transform your data before applying other algorithms.</p><p>Hope this helps you to choose the right estimator for your use case.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=78e5934a5d8d" width="1" height="1" alt="">]]></content:encoded>
        </item>
    </channel>
</rss>