<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Indradumna Banerjee on Medium]]></title>
        <description><![CDATA[Stories by Indradumna Banerjee on Medium]]></description>
        <link>https://medium.com/@indradumnabanerjee?source=rss-a5f8861043c3------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/0*F0qnjmxij5CvmdEe</url>
            <title>Stories by Indradumna Banerjee on Medium</title>
            <link>https://medium.com/@indradumnabanerjee?source=rss-a5f8861043c3------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Tue, 17 Nov 2026 05:13:02 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@indradumnabanerjee/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[Hallucinations of Multimodal Large Language Models: 2024 perspective]]></title>
            <link>https://medium.com/@indradumnabanerjee/hallucinations-of-multimodal-large-language-models-2024-perspective-fc8d534bf78c?source=rss-a5f8861043c3------2</link>
            <guid isPermaLink="false">https://medium.com/p/fc8d534bf78c</guid>
            <dc:creator><![CDATA[Indradumna Banerjee]]></dc:creator>
            <pubDate>Thu, 15 Aug 2024 04:21:41 GMT</pubDate>
            <atom:updated>2024-08-15T04:21:41.497Z</atom:updated>
            <content:encoded><![CDATA[<p>Large Language Models (LLMs) have revolutionized Natural Language Processing (NLP), showcasing impressive capabilities in language understanding, generation, and reasoning. Building upon LLMs, Multimodal Large Language Models (MLLMs) have emerged, enabling machines to process and understand both text and visual information, opening doors to tasks like image captioning and visual question answering.</p><p>However, a critical concern with the rapid progress of MLLMs is their tendency to hallucinate, generating plausible but factually incorrect content. This phenomenon, stemming from the LLMs they’re built upon, involves producing responses that are either factually incorrect or deviate from user instructions and context. While LLMs primarily experience hallucinations related to factual inconsistencies or unfaithfulness to instructions, MLLMs grapple with an additional challenge: generating text that doesn’t align with the provided visual content. This distinct nature necessitates a separate exploration of hallucination within MLLMs.</p><h3>Causes of Hallucinations</h3><p>Hallucinations in MLLMs stem from a complex interplay of factors, that relate to issues with the data and model, the training process objectives and the inference stage focus drifts.</p><p><strong>Data Issues:</strong></p><p>Insufficient Training Data: If an MLLM is trained on a limited dataset of text and image pairs, it might not encounter enough examples to learn the intricate relationships between the two modalities. This can lead to the model generating text that is factually incorrect or irrelevant to the visual content.</p><p>Low-Quality Training Data: The quality of the training data also plays a crucial role. Data that is noisy, contains errors, or lacks diversity can bias the model towards generating specific types of hallucinations. For instance, an MLLM trained on a dataset consisting primarily of images of cats and dogs might struggle to accurately describe other types of animals.</p><p>Statistical Biases in the Data: Real-world data often reflects societal biases. An MLLM trained on such data might inherit these biases and generate hallucinations that perpetuate stereotypes or misinformation.</p><p><strong>Model Limitations:</strong></p><p>Weaknesses in the Vision Model: The ability of the MLLM to accurately extract meaning from images is critical for generating faithful text descriptions. If the vision model is not powerful enough to capture the nuances of the visual content, the MLLM might resort to hallucinations to fill in the gaps.</p><p>Biases of the Language Model: Just like the training data, the language model itself can harbor biases that influence the hallucinations it produces. These biases can be semantic (e.g., a bias towards using positive language) or syntactic (e.g., a preference for certain sentence structures).</p><p>Alignment Interface Issues: The way the vision model and the language model communicate with each other can also contribute to hallucinations. If the alignment interface is not designed effectively, the MLLM might struggle to integrate information from both modalities, leading to outputs that are inconsistent or nonsensical.</p><p><strong>Training Process:</strong></p><p>Standard Training Objective: The training objective used to train MLLMs might not fully capture the complexities of the task. For example, a common objective is to maximize the likelihood of the generated text being the correct description of the image. However, this objective might not incentivize the MLLM to focus on factual accuracy or coherence, potentially leading to hallucinations.</p><p><strong>Inference Stage:</strong></p><p>Focus Drift During Text Generation: During the process of generating text, the MLLM might lose sight of the original image and start to focus on its own internal language model state. This can lead to hallucinations where the generated text goes off on tangents or introduces irrelevant information.</p><h3>Metrics and Benchmarks</h3><p>Evaluating the extent of hallucinations in MLLMs is an ongoing area of research, with various benchmarks and metrics under development. Here’s a brief overview of some of the most prominent ones:</p><ul><li><strong>CHAIR (CHallenge of Accepting or Rejecting Incoherent hypotheses):</strong> This benchmark presents the MLLM with multiple text passages, some of which are factually consistent with a given image and others that are not. The MLLM’s performance is measured by its ability to discriminate between the factual and hallucinatory passages.</li><li><strong>POPE (POst-processing for Text REfinement):</strong> This metric focuses on identifying hallucinations during the <em>inference</em> stage, after the MLLM has already generated its response. It assesses the coherence and factual soundness of the generated text by comparing it to the original image and external knowledge sources.</li><li><strong>MME (Multimodal Metrics Ego)</strong>: This benchmark incorporates both human evaluation and automatic metrics to assess different aspects of hallucination in MLLMs, including factual correctness, faithfulness to the image, and overall quality.</li></ul><p>These metrics provide valuable insights into the hallucination problem, but each has its own strengths and limitations. For instance, CHAIR relies on manually created datasets, which can be time-consuming and expensive to produce. POPE, on the other hand, requires access to external knowledge sources, which might not always be available or reliable. Ideally, we would like to have a benchmark that is easy to use, computationally efficient, and provides a comprehensive assessment of hallucination across various contexts. Researchers are actively exploring new metrics and combining existing ones to create more robust and informative evaluation methods.</p><h3>Mitigation Strategies</h3><p>While I will reserve the details of Mitigation strategies as part of an upcoming blog post, here is a quick summary of several approaches that are being explored to combat hallucinations in MLLMs:</p><ul><li><strong>Data-centric mitigation:</strong> Improving data quality, introducing negative or counterfactual data, and reducing noise in existing datasets.</li><li><strong>Model-centric mitigation:</strong> Enhancing the resolution and versatility of vision encoders, developing dedicated modules to control language model biases.</li><li><strong>Training-centric mitigation:</strong> Employing auxiliary supervision, reinforcement learning techniques, and even unlearning specific behaviors.</li><li><strong>Inference-centric mitigation:</strong> Implementing interventions during the generation process, such as contrastive or guided decoding, and post-hoc correction methods.</li></ul><h3>Summary</h3><p>The research on hallucinations in MLLMs is still in its early stages. Key challenges and future directions include addressing data-related concerns, improving cross-modal alignment, advancing model architectures, establishing standardized benchmarks, and even exploring the potential of reframing hallucination as a feature rather than a bug.</p><p>Ultimately, tackling the hallucination problem is vital for making MLLMs more reliable and trustworthy. By understanding the causes and developing effective mitigation strategies, we pave the way for MLLMs to revolutionize a wide range of applications while minimizing the risks associated with generating inaccurate or misleading information.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=fc8d534bf78c" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Dockerizing Your Machine Learning Model: A Step-by-Step Guide]]></title>
            <link>https://medium.com/@indradumnabanerjee/dockerizing-your-machine-learning-model-a-step-by-step-guide-12e92f8c960e?source=rss-a5f8861043c3------2</link>
            <guid isPermaLink="false">https://medium.com/p/12e92f8c960e</guid>
            <category><![CDATA[mlops]]></category>
            <category><![CDATA[docker]]></category>
            <category><![CDATA[containers]]></category>
            <dc:creator><![CDATA[Indradumna Banerjee]]></dc:creator>
            <pubDate>Sun, 14 Jul 2024 18:08:35 GMT</pubDate>
            <atom:updated>2024-07-14T18:08:35.376Z</atom:updated>
            <content:encoded><![CDATA[<p>Docker is a powerful tool that enables you to package your machine learning (ML) model, along with all its dependencies (libraries, frameworks, runtime environment), into a self-contained, portable unit aka the container. This container can then be deployed consistently across different environments — from the local machine to development servers to production clusters — without worrying about compatibility issues.</p><h3>How dockerization helps?</h3><ul><li><strong>Reproducibility:</strong> Makes sure that the model runs the same way everywhere, regardless of the underlying operating system or libraries installed on the target machine.</li><li><strong>Isolation:</strong> Prevent conflicts between the model’s dependencies and those of other applications running on the same machine.</li><li><strong>Scalability:</strong> Easily deploy and scale a machine learning model across multiple servers or cloud instances.</li><li><strong>Simplified Deployment:</strong> Share your model as a Docker image, making it easy for others to use your work.</li></ul><h3>The Building Blocks: Dockerfile and Docker Image</h3><ul><li><strong>Dockerfile:</strong> A text file containing a set of instructions that define how the Docker image will be built. And we will see it in action</li><li><strong>Docker Image:</strong> A read-only template created from your Dockerfile, containing everything needed to run your model (code, dependencies, runtime).</li><li><strong>Docker Container:</strong> A running instance of a Docker image, providing an isolated environment for your model to execute.</li></ul><h3>Step-by-Step Dockerization</h3><p><strong>Step 1: Prepare Your Model:</strong></p><p>Save your trained ML model (e.g., in .pkl format for scikit-learn models). Create a Python script (predict.py) that loads your model and exposes it for prediction.</p><p><strong>Step 2: Create a Dockerfile:</strong></p><p>This Dockerfile code snippet shows set up of a container environment for your machine learning model. It installs the necessary dependencies, copies your code and model into the container, and specifies the command to run your application.</p><pre># Use an official Python runtime as a parent image<br>FROM python:3.9-slim</pre><pre># Set the working directory in the container<br>WORKDIR /app</pre><pre># Copy the requirements file into the container<br>COPY requirements.txt .</pre><pre># Install dependencies<br>RUN pip install --no-cache-dir -r requirements.txt</pre><pre># Copy your application code<br>COPY . .</pre><pre># Expose the port for your application<br>EXPOSE 8000</pre><pre># Run the application<br>CMD [&quot;python&quot;, &quot;predict.py&quot;]</pre><p><strong>Step 3: Build the Docker Image:</strong></p><p>Open your terminal and navigate to the directory containing your Dockerfile and model files. Run the following command:</p><pre>docker build -t your_image_name .</pre><p><em>Replace </em><em>your_image_name with a suitable name for your image.</em></p><p><strong>Step 4: Run the Docker Container:</strong></p><p>The RUN command then installs these dependencies using pip, ensuring no cache is used to keep the image size smaller. Afterward, it copies the entire project (model, script, etc.) into the container. The EXPOSE instruction informs Docker that your application will use port 8000, but you&#39;ll still need to map this port to your host machine when running the container.</p><pre>docker run -p 8000:8000 your_image_name</pre><p>The -p 8000:8000 part maps port 8000 from your container to port 8000 on your local machine.</p><p><strong>Step 5: Test Your Model</strong></p><p>Send requests model’s API endpoint (e.g., http://localhost:8000/predict) to get predictions.</p><h3>Pro Tips:</h3><ul><li><strong>Smaller Images:</strong> Use slim base images to reduce image size.</li><li><strong>Environment Variables:</strong> Store sensitive information (API keys, database credentials) as environment variables for better security.</li><li><strong>Multi-Stage Builds:</strong> Use multi-stage builds to optimize Docker images for deployment by separating the build and runtime environments.</li><li><strong>Optimization:</strong> Utilize Docker caching to speed up subsequent builds by only rebuilding layers that have changed.</li></ul><h3>Deployment:</h3><p>Once the model image is built, it can be pushed to a Docker registry (like Docker Hub) to easily share and deploy it to staging and production environments.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=12e92f8c960e" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Getting started with Vision Language Model: Moondream with code]]></title>
            <link>https://medium.com/@indradumnabanerjee/getting-started-with-vision-language-model-moondream-783c264a02b9?source=rss-a5f8861043c3------2</link>
            <guid isPermaLink="false">https://medium.com/p/783c264a02b9</guid>
            <category><![CDATA[computer-vision]]></category>
            <category><![CDATA[artificial-intelligence]]></category>
            <category><![CDATA[generative-ai-tools]]></category>
            <category><![CDATA[vision-language-model]]></category>
            <dc:creator><![CDATA[Indradumna Banerjee]]></dc:creator>
            <pubDate>Sun, 14 Jul 2024 15:52:03 GMT</pubDate>
            <atom:updated>2024-07-14T16:31:46.727Z</atom:updated>
            <content:encoded><![CDATA[<p>In the fast-paced realm of Generative artificial intelligence, vision-language models are emerging as powerful tools for bridging the gap between visual and textual understanding. These models can analyze images and generate human-like descriptions, opening up new possibilities in image captioning, search, and even storytelling.</p><p>Moondream, a tiny vision-language model developed by Vikhyat on GitHub (<a href="https://github.com/vikhyat/moondream">https://github.com/vikhyat/moondream</a>), stands out for its simplicity and efficiency. It leverages the power of the Transformers architecture and pre-trained models to offer an accessible way to explore the world of image description.</p><p><strong>Getting Started with Moondream</strong></p><p>The beauty of Moondream lies in its ease of use. With just a few lines of Python code, you can harness its capabilities to generate insightful descriptions of images. Here’s an expanded code example (and an AI generated Figure 1 to make it look beautiful) that illustrates the process:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/250/0*ct0R0Ubz7odbWJgW" /><figcaption>Fig 1. An AI generated rendering of Moondream</figcaption></figure><pre># Install required libraries<br>!pip3 install transformers einops  <br><br>from transformers import AutoModelForCausalLM, AutoTokenizer<br>from PIL import Image<br><br># Model Configuration<br>model_id = &quot;vikhyatk/moondream2&quot;<br># Specify a model revision for consistency<br>revision = &quot;2024-05-20&quot;  <br><br># Load the Model and Tokenizer<br>model = AutoModelForCausalLM.from_pretrained(<br>    model_id,<br>    trust_remote_code=True, <br>    revision=revision<br>)<br>tokenizer = AutoTokenizer.from_pretrained(model_id, revision=revision)<br><br># Load and Prepare Image<br>image_path = &quot;your_image.jpeg&quot; # Replace with the path to your image<br>try:<br>    # Use PIL to open the image    <br>    image = Image.open(image_path)  <br>except FileNotFoundError:<br>    print(f&quot;Error: Image not found at {image_path}. Please check the path.&quot;)<br>    exit()<br><br># Encode Image and Generate Description<br>encoded_image = model.encode_image(image)<br>description = model.answer_question(encoded_image, &quot;Describe this image.&quot;, tokenizer)<br><br># Display the Description<br>print(&quot;Image Description:\n&quot;, description)</pre><p>First, you’ll need to install the required libraries, transformers and einops, using pip. Then, leverage the Transformers library’s powerful AutoModelForCausalLM and AutoTokenizer classes to load the pre-trained Moondream model and its corresponding tokenizer. Specifying a model revision during this process ensures you&#39;re working with a consistent and well-defined version of the model. Next, use the Python Imaging Library (PIL) to open the image you want Moondream to analyze. The model&#39;s encode_image method then converts this image into a format that the model can understand and process internally. Finally, you can call the answer_question method, providing it with the encoded image and a prompt like &quot;Describe this image.&quot; This will prompt Moondream to generate a textual description of the image, which is then printed to your console for your review.</p><p>By following these steps, you can empower Moondream to act as your own personal image description assistant, helping you understand and analyze visual content in an AI lens.</p><p><strong>Why Moondream?</strong></p><ul><li><strong>Lightweight:</strong> Unlike massive vision-language models that require significant computational resources, Moondream is designed to be compact and efficient.</li><li><strong>Ease of Use:</strong> The straightforward API provided by the Transformers library makes integrating Moondream into your projects a breeze.</li><li><strong>Customizability:</strong> You can experiment with different model revisions or even fine-tune the model on your own dataset to tailor it to specific tasks.</li><li><strong>Research and Exploration:</strong> Moondream provides a valuable starting point for researchers and enthusiasts interested in delving into the world of vision-language models.</li><li><strong>Image Captioning:</strong> Automatically generate captions for images in social media posts, articles, or presentations.</li><li><strong>Visual Question Answering:</strong> Build systems that can answer questions about the content of images.</li><li><strong>Creative Writing:</strong> Use Moondream as a source of inspiration for creative writing projects.</li></ul><p><strong>Conclusion</strong></p><p>Moondream is a testament to the ongoing democratization of AI. It empowers developers, researchers, and curious minds to explore the fascinating intersection of vision and language, paving the way for innovative applications and deeper understanding of the world around us. Give it a try and see what you can create!</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=783c264a02b9" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[CrewAI for Local AI Agents with Ollama: A Hands-On Tutorial for Local AI Agents]]></title>
            <link>https://medium.com/@indradumnabanerjee/crewai-for-local-ai-agents-with-ollama-a-hands-on-tutorial-for-local-ai-agents-a59b6ba32fd1?source=rss-a5f8861043c3------2</link>
            <guid isPermaLink="false">https://medium.com/p/a59b6ba32fd1</guid>
            <category><![CDATA[ollama]]></category>
            <category><![CDATA[crew-ai]]></category>
            <category><![CDATA[llm]]></category>
            <category><![CDATA[agents]]></category>
            <category><![CDATA[agent-based-modeling]]></category>
            <dc:creator><![CDATA[Indradumna Banerjee]]></dc:creator>
            <pubDate>Sun, 07 Jul 2024 19:15:41 GMT</pubDate>
            <atom:updated>2024-07-08T04:30:59.672Z</atom:updated>
            <content:encoded><![CDATA[<h3>CrewAI for Local AI Agents with Ollama: A Hands-On Tutorial</h3><p>Crew AI is a state-of-the-art framework designed to orchestrate role-playing, autonomous AI agents. By facilitating the exchange of information and actions, it enables agents to leverage their unique strengths and work together to achieve shared goals.</p><p>In a scenario where we are seeking control, privacy and customization for our AI projects, integrating CrewAI with Ollama, opens up exciting possibilities. In this tutorial, I will walk you through the process step-by-step, empowering you to create intelligent agents that leverage your own data and models, all while enjoying the benefits of local AI.</p><h3>Step 1: Install Packages</h3><p>Before we begin coding, let’s make sure we have all the necessary libraries in place. Open your terminal or command prompt and run the following commands:</p><pre>!pip3 install crewai<br>!pip3 install langchain<br>!pip3 install streamlit<br>!pip3 install unstructured<br>!pip3 install sentence_transformers</pre><h3>Step 2: Import Essential Libraries</h3><p>With the necessary packages installed, we’re ready to import the essential libraries that will form the backbone of our AI agents. Here’s a breakdown of what we’re bringing in:</p><pre>from crewai import Crew, Agent <br>from langchain.llms import Ollama<br>from langchain.vectorstores import Chroma<br>from langchain_community.embeddings import SentenceTransformerEmbeddings  # Use the wrapper<br>from sentence_transformers import SentenceTransformer<br>from langchain.document_loaders import UnstructuredURLLoader<br>from langchain.text_splitter import RecursiveCharacterTextSplitter<br>from langchain.agents import Tool</pre><h3>Step 3: Data Ingestion</h3><p>Now that we have the necessary libraries, let’s get some information for our agents to work with. We’ll use a real-world example of loading data from Crew AI’s own documentation website.</p><p>We initialize the UnstructuredURLLoader and provide it with the URL of the Crew AI installation guide. This loader is designed to handle various document formats commonly found on websites (HTML, PDF, etc.). The .load() method fetches the content from the specified URL and returns it as a list of Document objects. Each Document contains the text content and some metadata (e.g., source URL). we split the loaded documents into smaller, more manageable chunks using the RecursiveCharacterTextSplitter . This splitter breaks down the text based on character count while ensuring a bit of overlap (chunk_overlap=200) between chunks to maintain context. The result is stored in the texts variable, which is now a list of smaller text chunks.</p><pre># Website Data Ingestion <br>loader = UnstructuredURLLoader(urls=[&quot;https://docs.crewai.com/how-to/Installing-CrewAI/&quot;])<br>data = loader.load()<br>text_splitter = RecursiveCharacterTextSplitter(chunk_size=1000, chunk_overlap=200)<br>texts = text_splitter.split_documents(data)</pre><h3>Step 4: Vector database</h3><p>Now that we’ve divided our website data into manageable chunks, we need to transform them into a format that’s easier for our language models to work with. We will use “embeddings” to convert the text chunks into numerical representations that capture their meaning. We will then store these embeddings in a special kind of database called a “vector database” for efficient retrieval later.</p><p>We choose a pre-trained sentence transformer model (all-MiniLM-L6-v2) for our embeddings. This model is known for its speed and efficiency, making it suitable for many applications. We initialize the SentenceTransformerEmbeddings class, which will handle the transformation of text into embeddings. We use the Chroma.from_documents function to create a vector database from our texts (the chunked website data). We pass in our embeddings object, which will be used to generate embeddings for each text chunk. We specify persist_directory=&quot;./chroma_db&quot; to save the vector database to disk. This allows us to load it later without having to recreate it from scratch. The .persist() method ensures the database is saved.</p><p>Finally, we create a retriever object using the .as_retriever() method of our vector database (db). This retriever is the key to accessing relevant information later. It will be used by our AI agent to search for documents related to the user’s query.</p><pre># Initialize Embeddings<br>model_name = &quot;all-MiniLM-L6-v2&quot;<br>embeddings = SentenceTransformerEmbeddings(model_name=model_name) <br><br># Create and Persist Vector Database<br>db = Chroma.from_documents(texts, embeddings, persist_directory=&quot;./chroma_db&quot;)<br>db.persist()<br><br># Define Retriever from Vector Store<br>retriever = db.as_retriever()<br></pre><h3>Step 5: Create the AI agents</h3><p>With our Ollama language model now integrated into Crew AI’s framework and our knowledge base primed with the CrewAI website data, it’s time to assemble our team of intelligent agents. We’ll start by creating two distinct agents:</p><ol><li><strong>The Researcher:</strong> This agent acts as the primary investigator, delving into the depths of the website knowledge base to retrieve relevant information.</li><li><strong>The Researcher Boss:</strong> Acting as a supervisor and quality control expert, this agent will rigorously challenge and validate the researcher’s findings, ensuring the accuracy and reliability of the information presented.</li></ol><p>By working together, this dynamic duo will not only retrieve information but also critically evaluate it, ultimately delivering more insightful and reliable answers to your queries.</p><pre>researcher = Agent(<br>    ai=ollama_wrapper,<br>    name=&quot;Researcher&quot;,<br>    role=&quot;Researches topics by searching the website data.&quot;,<br>    tools=[<br>        Tool(<br>            name=&quot;Website_Search&quot;,<br>            func=retriever.get_relevant_documents,<br>            description=&quot;useful for when you need to ask with lookup on website data.&quot;<br>        )<br>    ],<br>    goal=&quot;Answer questions by retrieving relevant information from the website&#39;s data.&quot;,  # Add the goal here<br>    backstory=&quot;You are a helpful AI assistant specializing in searching and retrieving information from a website. Use your &#39;Website_Search&#39; tool to find relevant documents when answering questions.&quot;  <br><br>researcher_boss = Agent(<br>    ai=ollama_wrapper,<br>    name=&quot;Researcher Boss&quot;,<br>    role=&quot;Challenges the researcher to bring out the best out of his findings&quot;,<br>    tools=[<br>        Tool(<br>            name=&quot;Website_Search&quot;,<br>            func=retriever.get_relevant_documents,<br>            description=&quot;useful for validating the findings of the researcher&quot;<br>        )<br>    ],<br>    goal=&quot;Ask further questions to the researcher and validates the retrieved relevant information from the website&#39;s data.&quot;,  # Add the goal here<br>    backstory=&quot;You are a helpful AI assistant boss and your job is to make sure the retrieved information is correct. Use your &#39;Website_Search&#39; tool to find relevant documents when answering questions.&quot;  <br>)</pre><h3>Step 6: Execute a mission</h3><p>Now that our agents are primed with knowledge and ready to act, it’s time to give them a mission. In Crew AI, tasks are the specific objectives you assign to your agents. We define two tasks which are the research_task (for the researcher) and boss_task (for the researcher boss). Both tasks have the same description and expected_output, instructing the agents to analyze the Crew AI installation guide URL and extract information about Crew AI&#39;s workings. The agent parameter assigns the respective task to each agent. async_execution=True allows the tasks to be run concurrently. We create a Crew object containing both the researcher and researcher_boss agents. We pass the crew a list of the tasks to execute. The agents will work on their tasks independently and asynchronously.</p><pre>research_task = Task(<br>    description=(<br>        &quot;Analyze the URL provided ({crewai_url}) &quot;<br>        &quot;to extract information about how crewai works. &quot;<br>        &quot;required. Use the tools to gather content and identify &quot;<br>        &quot;and categorize the requirements.&quot;<br>    ),<br>    expected_output=(<br>        &quot;A structured list of crewai specifications, including necessary &quot;<br>        &quot;tools to get started&quot;<br>    ),<br>    agent=researcher,<br>    async_execution=True<br>)<br><br>boss_task = Task(<br>    description=(<br>        &quot;Analyze the URL provided ({crewai_url}) &quot;<br>        &quot;to extract information about how crewai works. &quot;<br>        &quot;required. Use the tools to gather content and identify &quot;<br>        &quot;and categorize the requirements.&quot;<br>    ),<br>    expected_output=(<br>        &quot;A structured list of crewai specifications, including necessary &quot;<br>        &quot;tools to get started&quot;<br>    ),<br>    agent=researcher_boss,<br>    async_execution=True<br>)</pre><h3>Step 7: Create and Execute the Crew</h3><p>After defining agents and tasks, we create a Crew to orchestrate their collaboration. This involves initializing a Crew object with our researcher and boss agents, along with their respective tasks. We enable verbose logging for detailed output. Additionally, we provide extra context through a dictionary, which could include information like the URL to analyze or background details about the agents. Finally, we kick off the crew’s work using research_crew1.kickoff(), and the resulting output will contain the combined findings of both agents, including a list of Crew AI specifications and insights from the boss agent.</p><pre># Create Crew<br>research_crew1 = Crew(<br>    agents=[researcher, researcher_boss],<br>    tasks=[research_task, boss_task],<br>    verbose=True  # This will print logs to the console as the crew works<br>)<br><br># Job Context<br>job_crew_works = {<br>    &#39;crewai_url&#39;: &#39;https://docs.crewai.com/how-to/Installing-CrewAI/&#39;,<br>    &#39;personal_writeup&#39;: &quot;&quot;&quot;Accomplished Researcher <br>    with 18 years of experience, specializing in<br>    setting up CrewAI kind of agent based systems&quot;&quot;&quot;<br>}<br><br># Kickoff the Crew&#39;s Work<br>result = research_crew1.kickoff(inputs=job_crew_works)<br>print(result)</pre><h3>Summary</h3><p>In preparation for building intelligent AI agents with Crew AI and Ollama, we’ve taken the following steps: installed essential libraries, imported crucial modules for agent creation, data handling, and language model interaction, website fetched data using UnstructuredURLLoader and RecursiveCharacterTextSplitter, converted text chunks into numerical representations (embeddings), and stored these embeddings in a Chroma vector database for efficient retrieval. We also define a specialized retriever to access this knowledge base, create a custom wrapper to integrate the Ollama language model with Crew AI, and finally, construct our AI agents. The code can be a starting point for efficient CrewAI and Ollama based systems. Happy coding with CrewAI.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=a59b6ba32fd1" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[LLM Agents: Experiential Learning]]></title>
            <link>https://medium.com/@indradumnabanerjee/experiential-learning-and-llms-0f11779978da?source=rss-a5f8861043c3------2</link>
            <guid isPermaLink="false">https://medium.com/p/0f11779978da</guid>
            <dc:creator><![CDATA[Indradumna Banerjee]]></dc:creator>
            <pubDate>Sun, 09 Jun 2024 04:56:02 GMT</pubDate>
            <atom:updated>2024-06-09T05:19:10.762Z</atom:updated>
            <content:encoded><![CDATA[<p>In this blog, we will take a high level understanding of why its a big deal to build an AI Agent that learns through Experience, as detailed in this recent exciting paper [1]. AI is constantly evolving and the emergence of Large Language Models (LLMs) has been one of the fascinating growth stories for the AI community. These LLM models, like the ones behind ChatGPT, are capable of understanding and generating human-like text, making them incredibly useful for a wide range of applications. However, a major drawback with LLMs is that their knowledge is static on non-evolving with time since the time they have been trained.</p><p>Its a bit similar to a human being who would have stopped learning after school or university. The human being would not be able to adapt to new situations, have the intellect to solve novel problems, or even grow as a person. This is essentially the challenge facing traditional LLMs. Hence, researcher are working on creating AI that learns from its experiences, and that is the summary of all the buzz around the recent AI agent evolution.</p><p><strong>Experiential Learning Agent</strong></p><p>Experiential Learning for LLMs is all about an LLM actively learning from its interactions with the world. The LLM functions much like a human student, gaining knowledge and improving its abilities through practice and reflection.</p><p>The expierential learning process consists of three key stages:</p><ol><li><strong>Experience Gathering:</strong> This is the first stage that makes the LLM explore a variety of tasks and environments, experimenting and interacting with the world around it. It carefully logs its successes and failures, building a repository of valuable experiences.</li><li><strong>Insight Extraction:</strong> In the second stage, LLM works in a high performing student mode, where the student reviews their notes and study materials. Similary, our LLM analyzes its experiences to derive meaningful insights. It learns not only from successful outcomes but also from mistakes, identifying patterns and strategies that lead to better performance.</li><li><strong>Knowledge Application:</strong> In the final stage, when faced with a new challenge, the LLM taps into its pool of insights and past experiences, selecting relevant knowledge to guide its decision-making process. This ability to draw on previous experiences allows Expieriential Learning to adapt to new situations and perform more effectively over time.</li></ol><p><strong>Why Experiential Learning matters</strong></p><p>Experiential learning approach offers several significant advantages over traditional LLM agents:</p><ul><li><strong>Continuous Improvement:</strong> Unlike static LLMs, Experiential Learning is constant evolver and refines its knowledge, leading to ongoing performance improvements.</li><li><strong>Adaptability:</strong> Experiential Learning can transfer its learning to a wide range of tasks and domains, demonstrating impressive versatility and adaptability.</li><li><strong>Transparency:</strong> Experiential Learning extracts are expressed in natural language, making its learning process understandable and interpretable.</li><li><strong>Efficiency:</strong> Experiential learning mechanism avoids the need for expensive and time-consuming retraining of the entire model, making it a more practical and incremental solution.</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1006/1*BrIYVKSdtKyRbDDHxYjgRg.png" /><figcaption><strong><em>Fig1. (Left) </em></strong><em>Experiential Learning is about gathering experiences, trying</em><strong><em> </em></strong><em>different tasks and remember what worked and what didn’t, figure out general tips and tricks from its successes and failures, and use what is learned for similar new tasks to do better than before. (</em><strong><em>Right) </em></strong><em>Experiential Learning practices tasks and tries again if it fails, learning from its mistakes, looks at what it learned and makes a list of helpful tips, focusing on common mistakes and good strategies.</em></figcaption></figure><p><strong>Experiments with Experiential Learning</strong></p><p><strong>Experiential Learning carried out in the paper[1] show that it is superior</strong> and outperforms baseline agent frameworks (ReAct and Act) across various tasks (HotpotQA, ALFWorld, WebShop). The effectiveness is evaluated both in terms of incorporating abstract insights and the ability to recall successful past actions. HotpotQA is a bunch of knowledge intensive tasks, while ALFWorld is a series of action-oriented tasks, and WebShop is a balance of both. During the experimentation of Experiential Learning with these tasks, versus experiments carried out with frameworks in the past with frameworks such as Reflexion, it was seen that Experiential Learning outperformed in all three kinds of tasks. Multiple retries were not needed or minimal compared to other frameworks, and the possibility of transferring knowledge from one task (HotpotQA) to another (FEVER), when “fine-tuned” with a few examples from the target task was also established.</p><p>The implications of Experiential Learning success are far-reaching. Imagine virtual assistants that become more personalized and helpful as they learn your preferences, or educational tools that adapt to each student’s individual learning style. ExpeL’s approach could revolutionize decision-making systems, customer service chatbots, and even autonomous robots.</p><p><strong>Words of Caution</strong></p><p>While Experiential Learning shows great promise, there are several points of caution to consider before widespread adoption. The points include heavy initial data dependency, misinterpretation risks, scalability challenges with clean information, overfitting risks to certain not well-rounded training experiences, and ethical considerations. The recommendations to minimise risks from the potential words of caution includes high quality and careful data curation, human in the loop measures, and further research into potential risks carried by agent based Experiential Learning.</p><p><strong>A Glimpse into the Future</strong></p><p>Experiential Learning represents a significant step forward in AI research, demonstrating its potential to create more intelligent and adaptable agents. As researchers continue to explore and refine this approach, we can expect to see even more impressive capabilities emerge from LLM agents. The future of Generative AI is bright, and experiential learning is poised to play a crucial role in shaping it.</p><p><strong>References</strong></p><p>[1] Zhao, A., Huang, D., Xu, Q., Lin, M., Liu, Y.J. and Huang, G., 2024, March. Expel: Llm agents are experiential learners. In <em>Proceedings of the AAAI Conference on Artificial Intelligence</em> (Vol. 38, №17, pp. 19632–19642).</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=0f11779978da" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Understanding concept drift with a sample Python code]]></title>
            <link>https://medium.com/@indradumnabanerjee/understanding-concept-drift-with-a-sample-python-code-f4a21bc550d7?source=rss-a5f8861043c3------2</link>
            <guid isPermaLink="false">https://medium.com/p/f4a21bc550d7</guid>
            <category><![CDATA[concept-drift]]></category>
            <category><![CDATA[data-drift]]></category>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[mlops]]></category>
            <dc:creator><![CDATA[Indradumna Banerjee]]></dc:creator>
            <pubDate>Sun, 10 Mar 2024 17:21:51 GMT</pubDate>
            <atom:updated>2024-03-10T17:21:51.435Z</atom:updated>
            <content:encoded><![CDATA[<p>Lets say you train a model to identify spam emails based on keywords like “free money” or “urgent action.” Well, while this model works well initially, over time spammers get smarter. They start using different tactics, like emotional manipulation or impersonating legitimate companies. The model that we trained on the old patterns, starts missing these new types of spam. This is what concept drift is. Concept drift refers to a significant change in the underlying relationship between the features (e.g., email text) and the target variable (e.g., spam or not spam). What this means is not just the data distribution is shifting (data drift), but the very concept which we are trying to model has changed underneath.</p><p>While there is no single definitive single way to detect concept drift, here are some approaches:</p><ul><li><strong>Monitor Model Performance:</strong> Track metrics like accuracy or F1 score over time. A sudden drop might indicate concept drift.</li><li><strong>Domain Knowledge:</strong> Leverage your understanding of the problem domain to identify potential external factors causing the change.</li><li><strong>Visualize Feature Distribution:</strong> Compare historical and recent data distributions to see if the relationship between features and target variables has changed.</li></ul><p>Now comes the stage where we suspect there is some concept drift. What do we do next? The first thing that can be done is retrain the model, which means we gather new data that reflects the current relationship and retrain your model. We should have a relook and possibly update feature engineering, like considering modifying how you extract features from the data to capture the new patterns. The good part is there are specialised concept drift adaptation algorithms<strong> </strong>which can continuously adapt to changing relationships. And I will be looking at one such algorithm in an upcoming blog. Last but not the least we need to monitor continuously, that is<strong> e</strong>stablish a process for monitoring both data and model performance for proactive drift detection.</p><p>Now lets try to understand concept drift more intuitively with a sample python code. In this code, We will simulate a data stream with three different distributions and visualize how the statistical properties change over time.</p><pre>import numpy as np<br>import matplotlib.pyplot as plt<br>from matplotlib import gridspec<br><br># Generate data for 3 distributions<br>random_state = np.random.RandomState(seed=42)<br>dist_a = random_state.normal(0.8, 0.05, 1000)<br>dist_b = random_state.normal(0.4, 0.02, 1000)<br>dist_c = random_state.normal(0.6, 0.1, 1000)<br><br># Concatenate data to simulate a data stream with drifts<br>stream = np.concatenate((dist_a, dist_b, dist_c))<br><br># Data plotting function<br>def plot_data(dist_a, dist_b, dist_c, drifts=None):<br>    fig = plt.figure(figsize=(7, 3), tight_layout=True)<br>    gs = gridspec.GridSpec(1, 2, width_ratios=[3, 1])<br>    ax1, ax2 = plt.subplot(gs[0]), plt.subplot(gs[1])<br><br>    ax1.grid()<br>    ax1.plot(stream, label=&#39;Stream&#39;)<br><br>    ax2.grid(axis=&#39;y&#39;)<br>    ax2.hist(dist_a, label=r&#39;$dist_a$&#39;)<br>    ax2.hist(dist_b, label=r&#39;$dist_b$&#39;)<br>    ax2.hist(dist_c, label=r&#39;$dist_c$&#39;)<br><br>    if drifts is not None:<br>        for drift_detected in drifts:<br>            ax1.axvline(drift_detected, color=&#39;red&#39;)<br><br>    plt.show()<br><br># Visualize the data stream<br>plot_data(dist_a, dist_b, dist_c)</pre><p>And this is how the data stream looks like in the plot in Figure1. The blue line represents the data stream, which is a concatenation of the three distributions. On the right, we have histograms for each distribution:</p><ul><li>dist_a: Centered around 0.8 with a standard deviation of 0.05.</li><li>dist_b: Centered around 0.4 with a smaller standard deviation of 0.02.</li><li>dist_c: Centered around 0.6 with a larger standard deviation of 0.1.</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1000/1*uxZ8KWR1raNiP-mMvh2P_Q.png" /><figcaption>Fig.1 Illustration of Concept Drift with three distributions</figcaption></figure><p>In conclusion, I hope the random distributions above helps in an intuitive understanding of concept drift. We believe monitoring both concept drift and data drift are essential core MLOps practises. While concept drift encompasses both changes in data distribution and shifts in the underlying concept, whereas data drift primarily focuses on changes in data distribution. Monitoring and adapting to both types of drift are essential for maintaining accurate models.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=f4a21bc550d7" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Understanding Data Drift with a simple python program]]></title>
            <link>https://medium.com/@indradumnabanerjee/understanding-data-drift-with-a-simple-python-program-b342dfd6e6f1?source=rss-a5f8861043c3------2</link>
            <guid isPermaLink="false">https://medium.com/p/b342dfd6e6f1</guid>
            <category><![CDATA[mlops]]></category>
            <category><![CDATA[data-drift]]></category>
            <category><![CDATA[data-science]]></category>
            <dc:creator><![CDATA[Indradumna Banerjee]]></dc:creator>
            <pubDate>Sun, 10 Mar 2024 14:36:11 GMT</pubDate>
            <atom:updated>2024-03-10T14:36:11.051Z</atom:updated>
            <content:encoded><![CDATA[<p>Data drift describes a shift in the underlying distribution of the data we are working with, over time. This shift can affect the input features and target variables used in the machine learning model. For understanding this better, lets imagine we have built a trained machine learning model to predict customer churn based on their historical purchase behaviour. However, if customer preferences shift, leading to new products becoming popular, the model’s predictions might become less accurate over time.</p><p>Now, let&#39;s try to understand data drift with a simple python program that is based on only two data series. Our code is based on the python functiondetect_drift that takes as input two arguments:</p><ul><li>data: NumPy array containing data points.</li><li>threshold (optional, default=0.05): defines the proportion of data points allowed to deviate from the expected range before drift is considered detected.</li></ul><p>Inside the function detect_drift , we start off by calculating meanor the average value of the data points using np.mean . The standard deviation of the data points is calculated using np.std. Next task is defining the two bounds, expected_upper_bound and expected_lower_bound , which are based on the mean and two standard deviations. These bounds represent the typical range where most data points should fall within statistically. We also define a logic to calculate the cumulative sum of the deviations of each data point from the mean using np.cumsum. The cumulative sum helps track how much the data points are deviating on average over time. The next job to do in this flow is to calculate the number of data points outliers that fall outside the expected bounds calculated earlier. This identifies potential outliers that significantly deviate from the typical data distribution.<br>Now comes the part where we are detecting data drift. The function calculates the proportion of outliers by dividing the number of outliers by the total number of data points. It then compares this proportion to the threshold (default 0.05). If the proportion exceeds the threshold, it indicates a significant deviation, suggesting data drift might have occurred. The function returns True if drift is detected, False otherwise. The entire python code for detect_drift is shown below with two sample distributions data1 which is the original data as well as a sample data where we can see data drift in action.</p><pre>import numpy as np<br>import matplotlib.pyplot as plt<br><br>def detect_drift(data, threshold=0.05):<br>  &quot;&quot;&quot;<br>  Code for detecting data drift<br><br>  Args:<br>      data: sample NumPy array of data points.<br>      threshold: Proportion of data points allowed to fall outside the expected range (default: 0.05).<br><br>  Returns:<br>      A boolean indicating if data drift is detected.<br>  &quot;&quot;&quot;<br>  mean = np.mean(data)<br>  std = np.std(data)<br>  expected_upper_bound = mean + 2 * std<br>  expected_lower_bound = mean - 2 * std<br><br>  # Calculate the cumulative sum of deviations from the mean<br>  deviations = np.cumsum(data - mean)<br><br>  # Identify data points exceeding the expected bounds<br>  outliers = np.sum(deviations &gt; expected_upper_bound) + np.sum(deviations &lt; expected_lower_bound)<br><br>  # Check if the proportion of outliers exceeds the threshold<br>  return outliers / len(data) &gt; threshold<br><br># Original data (no drift)<br>data1 = [1.2, 0.8, -0.5, 2.1, 0.3, 3.5, -1.8, 2.7, 0.9, 4.2]<br><br># New data with drift (higher values)<br>data2 = [5.2, 4.8, 3.5, 6.1, 4.3, 7.5, 3.8, 6.7, 4.9, 8.2]<br><br># Calculate drift for both datasets<br>drift_detected_1 = detect_drift(data1)<br>drift_detected_2 = detect_drift(data2)<br><br>if drift_detected_1:<br>  print(&quot;Data drift detected in original data (not expected)&quot;)<br>else:<br>  print(&quot;No data drift detected in original data (expected).&quot;)<br><br>if drift_detected_2:<br>  print(&quot;Data drift detected in new data (expected)&quot;)<br>else:<br>  print(&quot;No data drift detected in new data (not expected).&quot;)<br></pre><p>I also attach below an additional block for visualization of data drift using matplotlib to understand the process better. The visualisation block plots the original data (data1) in blue and the new data (data2) in red, using circle markers (&#39;o-&#39;). Blue and Red dashed lines represent the expected upper and lower bounds respectively, based on the mean and standard deviation of the original data. Cumulative sums displayed with green lines.</p><pre># Additional module for data drift visualisation<br>plt.figure(figsize=(12, 6))<br>plt.plot(data1, &#39;o-&#39;, label=&#39;Original Data&#39;, color=&#39;blue&#39;)<br>plt.plot(data2, &#39;o-&#39;, label=&#39;New Data (Drift)&#39;, color=&#39;red&#39;)<br>plt.axvline(len(data1) - 1, color=&#39;gray&#39;, linestyle=&#39;--&#39;, label=&#39;Data Split&#39;)<br><br># Plot the cumulative sum (green)<br>plt.plot(np.cumsum(data1 - np.mean(data1)), label=&#39;Cumulative Sum (Original)&#39;, color=&#39;green&#39;)<br>plt.plot(np.cumsum(data2 - np.mean(data2)) + np.cumsum(data1 - np.mean(data1))[-1], label=&#39;Cumulative Sum (New)&#39;, color=&#39;forestgreen&#39;)<br><br># Expected upper and lower bounds (dashed lines)<br>plt.axhline(y=np.mean(data1) + 2 * np.std(data1), color=&#39;red&#39;, linestyle=&#39;--&#39;, label=&#39;Expected Upper Bound&#39;, color=&#39;red&#39;)<br>plt.axhline(y=np.mean(data1) - 2 * np.std(data1), color=&#39;blue&#39;, linestyle=&#39;--&#39;, label=&#39;Expected Lower Bound&#39;, color=&#39;blue&#39;)<br><br># Labels and title<br>plt.xlabel(&#39;Data Point&#39;)<br>plt.ylabel(&#39;Value&#39;)<br>plt.title(&#39;Data Drift Visualization&#39;)<br>plt.legend()<br>plt.grid(True)<br>plt.tight_layout()<br><br># Show the plot<br>plt.show()</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*57RywhXVfCCsxNFRl12Lkg.png" /><figcaption>Fig1: Data drift visualisation based on the python script above that also sets values for the two data series.</figcaption></figure><p>In conclusion, the python script along with sample data above helps in visualizing the data distribution and cumulative sum for a data drift phenomenon, along with the drift detection logic. More sophisticated techniques might be necessary for complex real-world scenarios, but this blog is mainly intended to explain the concept to a ML engineer who is trying to understand data drift from scratch.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=b342dfd6e6f1" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[When to use NLTK vs SpacY]]></title>
            <link>https://medium.com/@indradumnabanerjee/when-to-use-nltk-vs-spacy-6def7662a6ea?source=rss-a5f8861043c3------2</link>
            <guid isPermaLink="false">https://medium.com/p/6def7662a6ea</guid>
            <category><![CDATA[nlp]]></category>
            <category><![CDATA[nltk]]></category>
            <category><![CDATA[generative-ai-tools]]></category>
            <category><![CDATA[spacy]]></category>
            <dc:creator><![CDATA[Indradumna Banerjee]]></dc:creator>
            <pubDate>Sun, 11 Feb 2024 12:59:40 GMT</pubDate>
            <atom:updated>2024-02-11T12:59:40.627Z</atom:updated>
            <content:encoded><![CDATA[<p>Choosing the right Natural Language Processing (NLP) library for your project demands a discerning eye. While both NLTK (Natural Language Toolkit) and spaCy hold considerable sway, their philosophies and strengths diverge. This analysis empowers you to make an informed decision based on your specific needs.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*W3d9gVtrkuZMZvF0TWPZRQ.jpeg" /><figcaption>Choosing between NLTK and SpacY might be a difficult decision for beginners. This writeup hopes to make it somewhat easier on when to choose what.</figcaption></figure><p><strong>Dissecting Capabilities:</strong></p><ul><li><strong>Task-driven Analysis:</strong> Analyze your NLP tasks meticulously. Is efficiency and pre-trained models your priority (e.g., spaCy for named entity recognition)? Or do you value granular control and customizability (e.g., NLTK for research-oriented sentiment analysis)?</li><li><strong>Performance Benchmarks:</strong> Evaluate performance metrics crucial to your project. spaCy often shines in speed and memory usage, while NLTK can be optimized with tools like parallelization. Remember, trade-offs exist, so tailor your choice accordingly.</li><li><strong>Language Landscape:</strong> NLTK boasts support for over 150 languages, exceeding spaCy’s current 7. If multilinguality reigns supreme in your project, NLTK offers wider horizons.</li></ul><p><strong>Embracing Expertise:</strong></p><ul><li><strong>Beginner’s Guide:</strong> spaCy’s intuitive API and pre-trained models make it an approachable starting point. Leverage its simplicity to kickstart your NLP journey.</li><li><strong>Intermediate Focus:</strong> If you have some NLP experience, both libraries present valuable options. Evaluate specific feature needs and experiment with small tasks to determine your preferred fit.</li><li><strong>Advanced Explorer:</strong> NLTK caters to seasoned developers and researchers seeking finer control. Its extensive toolkit empowers experimentation and exploration.</li></ul><p><strong>Community and Documentation:</strong></p><ul><li><strong>Collaboration Counts:</strong> Engage with the active communities surrounding both libraries. spaCy’s larger industrial focus might resonate with practical needs, while NLTK’s research-oriented discussions could prove invaluable for complex explorations.</li><li><strong>Decoding Documentation:</strong> Both offer comprehensive documentation, each tailored to its target audience. spaCy’s leans towards practical usage, while NLTK’s delves deeper into research-oriented concepts. Choose the style that aligns with your learning preferences.</li></ul><p><strong>The Informed Choice:</strong></p><p>Ultimately, the optimal NLP library hinges on your unique project requirements. Consider the interplay between tasks, performance, language needs, and your level of expertise. Remember, experimentation is key. Engage with both NLTK and spaCy on small tasks, delve deeper into their nuances, and let data-driven insights guide your decision.</p><p>By demystifying their strengths and aligning them with your project’s demands, you can unlock the true potential of NLP and empower your initiatives to achieve new heights.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=6def7662a6ea" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Emergent abilities in Large Language Models and the truth behind it]]></title>
            <link>https://medium.com/@indradumnabanerjee/emergent-abilities-in-large-language-models-and-the-truth-behind-it-f873ad6f6304?source=rss-a5f8861043c3------2</link>
            <guid isPermaLink="false">https://medium.com/p/f873ad6f6304</guid>
            <category><![CDATA[metrics]]></category>
            <category><![CDATA[emergence]]></category>
            <category><![CDATA[mlops]]></category>
            <category><![CDATA[large-language-models]]></category>
            <dc:creator><![CDATA[Indradumna Banerjee]]></dc:creator>
            <pubDate>Sun, 31 Dec 2023 15:39:22 GMT</pubDate>
            <atom:updated>2023-12-31T15:39:22.411Z</atom:updated>
            <content:encoded><![CDATA[<p>In the ever-evolving field of artificial intelligence, particularly within the realm of large language models (LLMs), the concept of “emergent abilities” has garnered significant interest. Emergent abilities refer to capabilities that suddenly become apparent in larger-scale models but are absent in their smaller counterparts. These abilities are intriguing due to their unpredictability and the abruptness of their appearance. However, a<a href="https://openreview.net/pdf?id=JRdN9GcI52#:~:text=This%20raises%20the%20possibility%20of%20an%20alternative%20explanation,to%20appear%20wholly%20unable%20to%20perform%20the%20task."> recent study</a> from Stanford University challenges this notion, proposing that what we perceive as emergent abilities might actually be an illusion created by the metrics used to measure model performance.</p><p><strong>The Mirage of Emergent Abilities: </strong>The study argues that emergent abilities are not inherent properties of LLMs but are artifacts of how we evaluate these models. The researchers suggest that the choice of non-linear or discontinuous metrics can distort the apparent performance of a model, creating the illusion of emergent abilities. In contrast, using linear or continuous metrics results in smooth, predictable changes in model performance. This hypothesis was tested using the InstructGPT/GPT-3 family of models and a meta-analysis on BIG-Bench, a comprehensive set of 39 metrics constituting a benchmarking platform for generative models (see Fig 1).</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*aexU2BWZb1SqJYob3ivGCQ.png" /><figcaption>Figure 1: The Occurrence of Emergent Abilities Across Various Metrics and Model Families. A) Emergent abilities are identified in a maximum of 5 out of 39 BIG-Bench metrics. B) Analysis by Wei (2022) using hand-annotated data shows emergent abilities in just 4 metrics. C) Over 92% of emergent abilities are observed in two specific metrics: Multiple Choice Grade and Exact String Match. (Source: <a href="https://openreview.net/pdf?id=JRdN9GcI52#:~:text=This%20raises%20the%20possibility%20of%20an%20alternative%20explanation,to%20appear%20wholly%20unable%20to%20perform%20the%20task.">Schaeffer et. al</a>)</figcaption></figure><p>The team conducted experiments using the InstructGPT/GPT-3 model family, altering the metrics used to evaluate performance on specific tasks. They found that when discontinuous metrics like Multiple Choice Grade or Exact String Match were used, apparent emergent abilities were observed. However, these disappeared when replaced with continuous metrics like the Brier Score. Over 92% of emergent abilities identified in the BIG-Bench tasks were associated with just two metrics: <em>Multiple Choice Grade and Exact String Match</em>. This supports the theory that the choice of evaluation metric significantly influences the perception of emergent abilities.</p><p>To further illustrate their point, the researchers applied their approach to vision tasks, which have not typically shown abrupt transitions in model capabilities. They induced apparent emergent abilities in networks of various architectures, including fully connected, convolutional, and self-attentional models, by defining a discontinuous metric for image reconstruction tasks. By defining a discontinuous metric, which assesses a network’s image reconstruction based on the squared reconstruction error being below a certain threshold, they show that the autoencoder’s performance appears sharply improved and unpredictable. This change in perceived abilities is attributed to the metric’s characteristics, not the model’s inherent capabilities, suggesting that emergent abilities in AI models could be a byproduct of the metrics used in their evaluation.</p><p><strong>Implications and Challenges:</strong> The findings of this study have profound implications for the field of AI and machine learning. They challenge the prevailing understanding of emergent abilities in LLMs, suggesting that these abilities might be more about our perception and measurement methods than about the models themselves. This insight is crucial for accurately evaluating and understanding the capabilities of generative AI models. The study also highlights the challenges inherent in AI evaluation. It underscores the need for careful consideration of metrics and statistical methods when assessing model performance, especially as models become more complex and their capabilities more nuanced.</p><p><strong>Conclusion:</strong> In conclusion, the Stanford study offers a compelling alternative explanation for the phenomenon of emergent abilities in large language models. It shifts the focus from the models themselves to the ways we evaluate them, opening up new avenues for research and development in AI. As the field continues to grow, this perspective will be vital in guiding more accurate and meaningful assessments of AI capabilities.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=f873ad6f6304" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[LMMOPs: Large Multimodal Models in Production and the way forward]]></title>
            <link>https://medium.com/@indradumnabanerjee/lmmops-large-multimodal-models-in-production-and-the-way-forward-7dc4a26e8428?source=rss-a5f8861043c3------2</link>
            <guid isPermaLink="false">https://medium.com/p/7dc4a26e8428</guid>
            <category><![CDATA[lmmops]]></category>
            <category><![CDATA[large-language-models]]></category>
            <category><![CDATA[large-multimodal-models]]></category>
            <category><![CDATA[mlops]]></category>
            <dc:creator><![CDATA[Indradumna Banerjee]]></dc:creator>
            <pubDate>Fri, 22 Dec 2023 16:29:00 GMT</pubDate>
            <atom:updated>2023-12-22T16:29:00.983Z</atom:updated>
            <content:encoded><![CDATA[<p>Imagine a symphony of intelligence, where AI effortlessly brings together vision, audio, and language together. Imagine buildings adapting to our moods, and reacting to our gestures. Surgeons wield robots fearlessly and confidently, guided by real-time analysis of X-rays. Cities are donned with self-driving cars conversing with each other, navigating seamlessly through bustling streets. This is not a distant dream, but a future quite possible scenario , where AI enhances our lives, one good thing at a time.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*tIq_2My91kZ4eOC9F8j-Pw.jpeg" /><figcaption>City of the not too distant future will have AI infused in almost everything and like multimodal AI will be deployed in multiple places. (Image credits: DALLE2)</figcaption></figure><p>One of the key enablers of this envisioned future will be the Large multimodal models (LMM), the multimodal relative of LLM. These models have the ability to process and analyze complex data from multiple modalities, such as text, images, and audio, to generate insights and predictions. However, deploying and managing LLMs in production can be a complex and challenging task. Even though this is a topic at a very nascent stage, in this blog, I will like to point out some of the early foreseeable challenges as we go about implementing the LMMs into production.</p><ol><li><strong>Multimodal data integration:</strong> Integrating different data types such as text, images, and audio requires not just processing them synchronously such that the relationships and contexts across modalities are correctly captured. This synchronization is inherently complex, as each data type has its unique characteristics and requires specific processing techniques. Furthermore, the challenge of multimodal data integration extends to developing a seamless analytical framework that can decipher and leverage the interdependencies and complementarities between different data types. This involves not only the technical aspect of data processing but also the conceptual understanding of how these different modalities interact and contribute to the overall meaning or pattern recognition. Such an integrated approach necessitates advanced algorithms and machine learning techniques that can dynamically adjust to the nuances and variability inherent in each data type, ensuring that the final integrated output is both contextually rich and accurate.</li><li><strong>Computational resources:</strong> Strategic considerations in infrastructure planning, including the choice of hardware components, optimization of parallel processing workflows, and the implementation of scalable solutions, become pivotal. Organizations must navigate the challenge of aligning their computational capabilities with the intricate requirements of LMMs to maximize efficiency and minimize operational costs. In essence, addressing the technical intricacies of computational resource management is fundamental to unlocking the full potential of Large Multimodal Models in practical, real-world scenarios.</li><li><strong>Ethical Dimensions</strong>: The integration of Large Multimodal Models (LMMs) into real-world applications introduces a host of ethical considerations that demand careful attention. Issues pertaining to fairness, transparency, and accountability become paramount. Biases present in training data can inadvertently be learned and perpetuated by LMMs, potentially amplifying existing societal inequalities. Tackling this ethical challenge requires proactive measures, including thorough audits of training datasets, the implementation of bias-mitigation techniques, and a commitment to fairness in algorithmic decision-making. Notably, ethical considerations in deploying LMMs have gained recognition in the broader context of sociotechnical systems, as highlighted by Selbst et al. in their work on “Fairness and Abstraction in Sociotechnical Systems.”</li><li><strong>Model Interpretability and Explainability: </strong>The intricate nature of Large Multimodal Models poses a significant hurdle in terms of interpreting and explaining their decisions. In critical applications such as healthcare and finance, where user trust and interpretability are paramount, the lack of transparency in model outputs becomes a major concern. Ribeiro et al.’s work on “Why should I trust you?” emphasizes the importance of explaining the predictions of complex classifiers. Achieving model interpretability and explainability involves developing methodologies to unravel the decision-making processes within LMMs, fostering trust among end-users, regulators, and other stakeholders. Striking a balance between model complexity and interpretability is crucial for the responsible deployment of LMMs.</li><li><strong>Real-time Inference and Latency: </strong>The demand for real-time analysis and low-latency inference poses a formidable challenge in deploying Large Multimodal Models, particularly in applications such as autonomous vehicles and medical diagnostics. The inherent computational complexity of LMMs can lead to delays in decision-making, compromising the effectiveness of time-sensitive systems. Overcoming this challenge requires the implementation of efficient optimization strategies, ranging from hardware acceleration to algorithmic optimizations. Balancing the need for high-level model accuracy with the imperative for swift responses necessitates a nuanced approach to real-time inference and latency reduction. Addressing these challenges is essential for unlocking the full potential of LMMs in time-critical applications.</li></ol><p>In conclusion, the successful deployment of Large Multimodal Models in real-world applications hinges on addressing a multifaceted set of challenges. These range from the intricate process of data integration and computational resource management to addressing ethical concerns, ensuring model interpretability, and optimizing for real-time performance. Ultimately, the harmonization of these aspects will be pivotal in realizing the transformative potential of LMMs across various sectors, enabling them to effectively contribute to advancements in technology, society, and industry.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=7dc4a26e8428" width="1" height="1" alt="">]]></content:encoded>
        </item>
    </channel>
</rss>