<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by thallyscostalat on Medium]]></title>
        <description><![CDATA[Stories by thallyscostalat on Medium]]></description>
        <link>https://medium.com/@thallyscostalat?source=rss-180b8f46e6e2------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/1*wK-2f6DBFD2U9btynMF2nA.jpeg</url>
            <title>Stories by thallyscostalat on Medium</title>
            <link>https://medium.com/@thallyscostalat?source=rss-180b8f46e6e2------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Thu, 19 Nov 2026 09:11:45 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@thallyscostalat/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[Easy Qdrant Vector Database Deployment on Amazon EC2 using User data Script]]></title>
            <link>https://medium.com/@thallyscostalat/easy-qdrant-vector-database-deployment-on-amazon-ec2-using-user-data-script-5f216497b66d?source=rss-180b8f46e6e2------2</link>
            <guid isPermaLink="false">https://medium.com/p/5f216497b66d</guid>
            <category><![CDATA[qdrant]]></category>
            <category><![CDATA[vector-database]]></category>
            <category><![CDATA[similarity-search]]></category>
            <category><![CDATA[aws]]></category>
            <category><![CDATA[generative-ai-tools]]></category>
            <dc:creator><![CDATA[thallyscostalat]]></dc:creator>
            <pubDate>Sun, 05 Jan 2025 02:34:15 GMT</pubDate>
            <atom:updated>2025-01-05T02:35:28.301Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*NeVzU9okx8X_1cVcb0p9qA.png" /></figure><p>In this post, we will walk through the process of deploying Qdrant, a powerful vector search engine, on an Amazon EC2 instance using a simple user data script. The main goal is to simplify the setup and make it as automated as possible by leveraging EC2’s built-in “User data” feature.</p><h3>What is Qdrant?</h3><p>Qdrant is an open-source vector database designed for working with high-dimensional data. It provides an easy-to-use platform for efficient similarity search, which is a critical functionality for many machine learning and GenAI applications. By storing vectors in a specialized format and supporting fast retrieval based on vector similarity, Qdrant helps you build intelligent applications such as recommendation engines, search engines, and more.</p><h3>Why Use Amazon EC2?</h3><p>Amazon EC2 (Elastic Compute Cloud) provides computing capacity in AWS the cloud, making it an excellent choice for hosting Qdrant. With EC2, you can easily scale up or down based on your needs, and you have full control over the environment and configuration.</p><p>To automate the setup process, we will utilize EC2’s “User data” feature. This allows you to run a shell script when the EC2 instance is launched, eliminating the need to manually install and configure Qdrant.</p><h3>Prerequisites</h3><p>Before proceeding, ensure you have the following:</p><ol><li>An AWS account.</li><li>Basic knowledge of AWS EC2 and Docker.</li><li>A clean EC2 instance running Ubuntu (or a compatible Linux AMI).</li><li>Basic understanding of Qdrant (installing and running Docker containers).</li></ol><h3>Step-by-Step Guide</h3><p>Follow the steps below to deploy Qdrant on an Amazon EC2 instance using a User data script.</p><h4>Step 1: Launch a New EC2 Instance</h4><ol><li><strong>Sign in to the AWS Management Console</strong> and navigate to the EC2 dashboard.</li></ol><figure><img alt="" src="https://cdn-images-1.medium.com/max/944/1*5mJicziH4nDCCuZqEOHXBA.png" /></figure><p>2.<strong> Launch a new EC2 instance</strong> with Ubuntu as the operating system.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*tyHljw6xXXCm-1q7k5siTA.png" /></figure><p>3. Choose an instance type (e.g., t2.micro for testing purposes).</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*RF7qn4Ikij3YfYrZz7yMyA.png" /></figure><p>4. Configure the security group of the EC2 instance on “Network Settings” section.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*lTrz1OrlRJxkZH8huLs0wA.png" /></figure><p>5. In the “Configure Instance” section, find the <strong>User Data</strong> field under “Advanced Details.”</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*1fRnSrSjpt4p7QQUAYLXZA.png" /></figure><ul><li>In the User data field, paste the following script to automatically set up Qdrant when the EC2 instance launches:</li></ul><pre><br>#!/bin/bash<br>sudo apt-get update -y<br>sudo apt-get install -y docker.io<br>sudo systemctl enable docker<br>sudo systemctl start docker<br>sudo docker run -d --name qdrant \<br>  -p 6333:6333 \<br>  -v $(pwd)/qdrant_storage:/qdrant/storage:z \<br>  qdrant/qdrant:latest</pre><p>6. Launch Instance</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/531/1*uFHMsl8X1z-B7fwQB54h8Q.png" /></figure><h4>Step 2: Instance Setup</h4><p>Once the EC2 instance is launched, the script will run automatically, performing the following actions:</p><ol><li><strong>Updates the EC2 instance packages</strong> with the apt-get update -y command to ensure the instance has the latest software updates.</li><li><strong>Installs Docker</strong>, which is required to run Qdrant in a containerized environment.</li><li><strong>Starts the Docker service</strong> to ensure the Docker engine is running.</li><li><strong>Runs the Qdrant container</strong>, exposing the necessary ports (6333), and mounts a volume to store the Qdrant data persistently.</li></ol><p>The docker run command also ensures that Qdrant runs on port 6333, which is the default for the vector search API.</p><h4>Step 3: Access Qdrant</h4><p>After the instance is up and running, you can access Qdrant by using the public IP of the EC2 instance.</p><ol><li>Find the public IP address of the instance in the EC2 Dashboard.</li></ol><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*xWnoK9WbuFAOMWNNBoQorw.png" /></figure><p>2. Open a web browser using the EC2 public IP on port 6333:</p><pre>http://ec2-public-ip:6333/dashboard</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*cjVJw1klS2p1vRn7GAyQ4Q.png" /></figure><p>Or use curl to test the Qdrant API. This should return a response from the Qdrant API if everything is set up correctly.</p><h4>Step 5: Persistent Storage</h4><p>In the script, the volume $(pwd)/qdrant_storage:/qdrant/storage:z ensures that the Qdrant data is stored persistently across container restarts. The qdrant_storage directory is created on the EC2 instance, and the data will be saved there.</p><p>You can modify this path to match your desired storage location, or even attach an EBS volume for larger datasets.</p><h3>Benefits of Using User Data Scripts</h3><p>By using an EC2 User data script, you get several key benefits:</p><ol><li><strong>Automation</strong>: The entire deployment process is automated. Once the EC2 instance starts, the script runs and sets up Qdrant without manual intervention.</li><li><strong>Consistency</strong>: The same script can be used repeatedly, ensuring that every EC2 instance launched has the same configuration.</li><li><strong>Scalability</strong>: You can easily replicate this setup for multiple EC2 instances to handle more traffic or scale your Qdrant infrastructure.</li></ol><h3>Conclusion</h3><p>Deploying Qdrant on Amazon EC2 using a User Data script is a fast, automated, and scalable way to get up and running with a vector database. With Docker containers and Amazon EC2’s flexible computing environment, it’s easy to integrate Qdrant into your machine learning or AI project. Whether you’re building a recommendation system or a search engine, this setup will save you time and resources.</p><p>Give it a try and start exploring the power of vector search with Qdrant on AWS!</p><h3>👏🏽 I hope you enjoy the content! | <a href="https://www.linkedin.com/in/thallys-costalat-147700149/">Follow me on LinkedIn</a>.</h3><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=5f216497b66d" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Building a Powerful Multi-Agent Workflow with CrewAI and Groq]]></title>
            <link>https://medium.com/@thallyscostalat/building-a-powerful-multi-agent-workflow-with-crewai-and-groq-55b4a1ba5cf6?source=rss-180b8f46e6e2------2</link>
            <guid isPermaLink="false">https://medium.com/p/55b4a1ba5cf6</guid>
            <category><![CDATA[llm]]></category>
            <category><![CDATA[crew-ai]]></category>
            <category><![CDATA[groq]]></category>
            <category><![CDATA[multiagent-system]]></category>
            <category><![CDATA[llama-3]]></category>
            <dc:creator><![CDATA[thallyscostalat]]></dc:creator>
            <pubDate>Sun, 19 May 2024 21:49:44 GMT</pubDate>
            <atom:updated>2024-05-19T21:49:44.187Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/800/0*gWUx5KbqmtyC4xqQ" /><figcaption><a href="https://www.google.com/url?sa=i&amp;url=https%3A%2F%2Fwww.linkedin.com%2Fposts%2Fsamwitteveen_llama3-crewai-groq-email-ai-agent-activity-7188367604061736961-ygLo&amp;psig=AOvVaw1gX-Yiwz6SQ6IgvGGY5W7o&amp;ust=1716239008414000&amp;source=images&amp;cd=vfe&amp;opi=89978449&amp;ved=0CBIQjRxqFwoTCLig4N7OmoYDFQAAAAAdAAAAABAH">Source</a></figcaption></figure><p>In the realm of content creation, the traditional model of a single individual working on every aspect of a project has evolved into a more collaborative and specialized approach. Multi-agent workflows, where different agents with distinct roles collaborate to achieve a common goal, have gained prominence due to their effectiveness in enhancing productivity, creativity, and quality.</p><h3>Why Multi-Agent Approaches?</h3><ol><li><strong>Specialization</strong>: Each agent in a multi-agent system specializes in a particular aspect of content creation, such as planning, writing, or editing. By focusing on their respective areas of expertise, agents can deliver higher-quality output more efficiently.</li><li><strong>Efficiency</strong>: Dividing the content creation process into distinct tasks allows agents to work in parallel, reducing the overall time required to complete a project. This parallelism ensures that bottlenecks are minimized, and the workflow remains agile and responsive.</li><li><strong>Collaboration</strong>: Multi-agent workflows foster collaboration among team members, enabling them to leverage each other’s strengths and skills. By sharing knowledge, resources, and feedback, agents can collectively produce content that is more diverse, insightful, and engaging.</li><li><strong>Scalability</strong>: As the complexity and volume of content continue to grow, multi-agent approaches offer scalability by allowing teams to adapt and expand as needed. New agents can be onboarded to handle specific tasks or domains, ensuring that the workflow remains adaptable to changing requirements.</li></ol><p>Now, let’s illustrate the practical application of multi-agent workflows in content creation using CrewAI and Groq.</p><h3>The Workflow</h3><p>Our multi-agent workflow consists of three primary roles: Content Planner, Content Writer, and Editor. Each role is assigned specific tasks aimed at producing engaging and factually accurate content on a given topic. Let’s break down the workflow step by step:</p><ol><li><strong>Content Planning</strong>: The Content Planner’s role is to lay the groundwork for the content creation process. The planner prioritizes relevant trends, identifies the target audience, and develops a comprehensive content outline. This initial stage sets the direction and scope for the subsequent tasks.</li><li><strong>Content Writing</strong>: With the content plan in hand, the Content Writer takes over to craft a compelling blog post. Leveraging Groq’s API for Llama3–70B, the writer incorporates SEO keywords, structures the content with engaging headings and subheadings, and ensures a smooth flow from introduction to conclusion. The emphasis is on delivering valuable insights while maintaining the intended tone and style.</li><li><strong>Editing and Refinement</strong>: Once the initial draft is ready, the Editor steps in to polish the content further. By proofreading for grammatical errors and aligning the text with the brand’s voice, the editor ensures consistency and professionalism. Additionally, the editor evaluates the content from a journalistic perspective, verifying facts and balancing viewpoints where necessary.</li></ol><h3>Understanding the Components</h3><p>Before diving into the workflow, let’s briefly understand the key components involved:</p><ul><li><strong>CrewAI</strong>: CrewAI is a versatile platform designed to facilitate collaborative work among multiple agents, each specializing in different aspects of content creation. It provides a framework for task delegation, communication, and coordination within a team.</li><li><strong>Groq</strong>: Groq, powered by cutting-edge language models, like Llama3–70B and Mixtral 8x7b, offers state-of-the-art natural language processing capabilities. It enables agents to generate high-quality content, perform language-based tasks, and extract insights from vast amounts of data.</li></ul><p>We’ll walk through the process of setting up a multi-agent workflow for content creation using CrewAI and Groq. We’ll explain each block of code in detail to understand how the workflow operates.</p><blockquote>Installation and Importing Libraries</blockquote><pre>!pip install --q crewai==0.28.8 crewai_tools==0.1.6 langchain_community==0.0.29 langchain_groq<br>from crewai import Agent, Task, Crew<br>from langchain_groq import ChatGroq</pre><blockquote>Defining LLM Model (Llama3–70B)</blockquote><pre>llm=ChatGroq(temperature=0,<br>             model_name=&quot;llama3-70b-8192&quot;,<br>             api_key=&#39;&lt;YOUR_GROQ_API_KEY&gt;&#39;)</pre><blockquote>Defining Agents</blockquote><pre>planner = Agent(<br>    llm=llm,<br>    role=&quot;Content Planner&quot;,<br>    goal=&quot;Plan engaging and factually accurate content on {topic}&quot;,<br>    backstory=&quot;You&#39;re working on planning a blog article &quot;<br>              &quot;about the topic: {topic}.&quot;<br>              &quot;You collect information that helps the &quot;<br>              &quot;audience learn something &quot;<br>              &quot;and make informed decisions. &quot;<br>              &quot;Your work is the basis for &quot;<br>              &quot;the Content Writer to write an article on this topic.&quot;,<br>    allow_delegation=False,<br>    verbose=True<br>)<br><br>writer = Agent(<br>    llm=llm,<br>    role=&quot;Content Writer&quot;,<br>    goal=&quot;Write insightful and factually accurate &quot;<br>         &quot;opinion piece about the topic: {topic}&quot;,<br>    backstory=&quot;You&#39;re working on a writing &quot;<br>              &quot;a new opinion piece about the topic: {topic}. &quot;<br>              &quot;You base your writing on the work of &quot;<br>              &quot;the Content Planner, who provides an outline &quot;<br>              &quot;and relevant context about the topic. &quot;<br>              &quot;You follow the main objectives and &quot;<br>              &quot;direction of the outline, &quot;<br>              &quot;as provide by the Content Planner. &quot;<br>              &quot;You also provide objective and impartial insights &quot;<br>              &quot;and back them up with information &quot;<br>              &quot;provide by the Content Planner. &quot;<br>              &quot;You acknowledge in your opinion piece &quot;<br>              &quot;when your statements are opinions &quot;<br>              &quot;as opposed to objective statements.&quot;,<br>    allow_delegation=False,<br>    verbose=True<br>)<br><br>editor = Agent(<br>    llm=llm,    <br>    role=&quot;Editor&quot;,<br>    goal=&quot;Edit a given blog post to align with &quot;<br>         &quot;the writing style of the organization. &quot;,<br>    backstory=&quot;You are an editor who receives a blog post &quot;<br>              &quot;from the Content Writer. &quot;<br>              &quot;Your goal is to review the blog post &quot;<br>              &quot;to ensure that it follows journalistic best practices,&quot;<br>              &quot;provides balanced viewpoints &quot;<br>              &quot;when providing opinions or assertions, &quot;<br>              &quot;and also avoids major controversial topics &quot;<br>              &quot;or opinions when possible.&quot;,<br>    allow_delegation=False,<br>    verbose=True<br>)</pre><ul><li>We instantiate three agents: <strong>planner</strong>, <strong>writer</strong>, and <strong>editor.</strong></li><li>Each agent is initialized with the LLM model Llama3–70B.</li><li>We define the role, goal, backstory, and other attributes for each agent.</li></ul><blockquote>Defining Tasks</blockquote><pre>plan = Task(<br>    description=(<br>        &quot;1. Prioritize the latest trends, key players, &quot;<br>            &quot;and noteworthy news on {topic}.\n&quot;<br>        &quot;2. Identify the target audience, considering &quot;<br>            &quot;their interests and pain points.\n&quot;<br>        &quot;3. Develop a detailed content outline including &quot;<br>            &quot;an introduction, key points, and a call to action.\n&quot;<br>        &quot;4. Include SEO keywords and relevant data or sources.&quot;<br>    ),<br>    expected_output=&quot;A comprehensive content plan document &quot;<br>        &quot;with an outline, audience analysis, &quot;<br>        &quot;SEO keywords, and resources.&quot;,<br>    agent=planner,<br>)<br><br>write = Task(<br>    description=(<br>        &quot;1. Use the content plan to craft a compelling &quot;<br>            &quot;blog post on {topic}.\n&quot;<br>        &quot;2. Incorporate SEO keywords naturally.\n&quot;<br>  &quot;3. Sections/Subtitles are properly named &quot;<br>            &quot;in an engaging manner.\n&quot;<br>        &quot;4. Ensure the post is structured with an &quot;<br>            &quot;engaging introduction, insightful body, &quot;<br>            &quot;and a summarizing conclusion.\n&quot;<br>        &quot;5. Proofread for grammatical errors and &quot;<br>            &quot;alignment with the brand&#39;s voice.\n&quot;<br>    ),<br>    expected_output=&quot;A well-written blog post &quot;<br>        &quot;in markdown format, ready for publication, &quot;<br>        &quot;each section should have 2 or 3 paragraphs.&quot;,<br>    agent=writer,<br>)<br><br>edit = Task(<br>    description=(&quot;Proofread the given blog post for &quot;<br>                 &quot;grammatical errors and &quot;<br>                 &quot;alignment with the brand&#39;s voice.&quot;),<br>    expected_output=&quot;A well-written blog post in markdown format, &quot;<br>                    &quot;ready for publication, &quot;<br>                    &quot;each section should have 2 or 3 paragraphs.&quot;,<br>    agent=editor<br>)</pre><ul><li>We create three tasks: plan, write, and edit.</li><li>Each task includes a description of the steps involved, the expected output, and the corresponding agent responsible for completing the task.</li></ul><blockquote>Creating a Crew</blockquote><pre>crew = Crew(<br>    agents=[planner, writer, editor],<br>    tasks=[plan, write, edit],<br>    verbose=2<br>)</pre><ul><li>We instantiate a Crew object, passing the lists of agents and tasks as parameters.</li></ul><blockquote>Kick Off the Workflow</blockquote><pre>result = crew.kickoff(inputs={&quot;topic&quot;: &quot;Artificial Intelligence&quot;})</pre><ul><li>We provide inputs to the workflow, such as the topic “Artificial Intelligence”, which will be used by the agents during execution.</li></ul><p>A verbose logging will be printed that enable the developer visualize all the steps that the multi-agent system performs to generate a useful result.</p><blockquote>Displaying Markdown Results</blockquote><pre>from IPython.display import Markdown<br>Markdown(result)</pre><p>In this case, a Markdown format will be showed containing the results generated by the agents.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*SEURNy6TWnJWGb8ne9K8ow.png" /><figcaption>Markdown Results Part. 1</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*s8m-QVXEWAh1xGX6zMUDEA.png" /><figcaption>Markdown Results Part. 2</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*J_I1R-KlRUmqo6EHmOod7A.png" /><figcaption>Markdown Results Part. 3</figcaption></figure><p>That’s it! We’ve created a comprehensive multi-agent workflow for content creation. Each agent performs its designated tasks, resulting in a well-coordinated and efficient content creation process.</p><p>In this blogpost, we’ve seen how to leverage CrewAI and Groq to create a streamlined multi-agent workflow for content creation.</p><p>By dividing tasks among specialized agents and utilizing advanced AI capabilities, teams can produce high-quality content efficiently. Multi-agent approaches offer benefits like specialization, efficiency, collaboration, and scalability, making them essential for modern content creation.</p><p>Embracing these tools empowers teams to stay competitive and deliver exceptional results in their content endeavors.</p><h3>👏🏽 I hope you enjoy the content! | <a href="https://www.linkedin.com/in/thallys-costalat-147700149/">Follow me on LinkedIn</a>.</h3><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=55b4a1ba5cf6" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Talk to your data using LangChain CSV Agents and Amazon Bedrock]]></title>
            <link>https://medium.com/@thallyscostalat/talk-to-your-data-using-langchain-csv-agents-and-amazon-bedrock-07ee3d35e9f7?source=rss-180b8f46e6e2------2</link>
            <guid isPermaLink="false">https://medium.com/p/07ee3d35e9f7</guid>
            <category><![CDATA[aws]]></category>
            <category><![CDATA[generative-ai-tools]]></category>
            <category><![CDATA[bedrock]]></category>
            <category><![CDATA[langchain]]></category>
            <category><![CDATA[nlp]]></category>
            <dc:creator><![CDATA[thallyscostalat]]></dc:creator>
            <pubDate>Sun, 05 May 2024 13:27:55 GMT</pubDate>
            <atom:updated>2024-05-05T13:43:12.407Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/960/0*bSESPEp02Zz_TnZP.png" /><figcaption>LangChain and Bedrock. <a href="https://dev.classmethod.jp/articles/amazon-bedrock-with-langchain-invoking-claude3-from-lambda/">Source</a>.</figcaption></figure><p>Have you ever wished you could communicate with your data effortlessly, just like talking to a colleague? With LangChain CSV Agents, that’s exactly what you can do!</p><p>In this article, we’ll explore how you can interact with your CSV data using natural language, leveraging LangChain, an exciting new tool in the field of natural language processing, and a FM from Amazon Bedrock.</p><blockquote><strong>Introduction</strong></blockquote><p>LangChain is a powerful framework that allows you to build conversational agents tailored to your specific data tasks. By combining the capabilities of language models like Claude 3 Sonnet from Anthropic with data processing tools, LangChain enables seamless communication with your datasets.</p><blockquote><strong>Getting Started</strong></blockquote><p>First, let’s search for a CSV dataset on Kaggle to develop our test scenario. In this case, we’ll use <a href="https://www.kaggle.com/datasets/aadarshvelu/heart-failure-prediction-clinical-records?resource=download"><strong>heart_failure_clinical_records</strong></a><strong>.</strong></p><p>To make sure the informations that we have, let’s print the dataframe header:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*epsoWG9duGPNKWBe8yZw1g.png" /></figure><p>So, to summarize, we have the following features:</p><ul><li><strong>age</strong>, <strong>anaemia</strong>, <strong>creatinine_phosphokinase, diabetes, ejection_fraction, high_blood_pressure, platelets, serum_creatinine, serum_sodium, sex, smoking, time, DEATH_EVENT.</strong></li></ul><p>Let’s dive into a practical example to see LangChain and Bedrock in action. We’ll start with a simple Python script that sets up a LangChain CSV Agent and interacts with this CSV file.</p><blockquote><strong>Import Necessary Modules</strong></blockquote><pre>from langchain.agents.agent_types import AgentType<br>from langchain_experimental.agents.agent_toolkits import create_csv_agent<br>from langchain.llms.bedrock import Bedrock<br>import boto3</pre><p>In this section, we import the necessary modules to create and interact with the LangChain CSV Agent. Additionally, we import Bedrock from LangChain for accessing models and boto3 for AWS SDK to communicate with Bedrock service.</p><blockquote><strong>Initializing Bedrock Claude 3 Sonnet FM Model</strong></blockquote><pre>llm=Bedrock(model_id=&quot;anthropic.claude-3-sonnet-20240229-v1:0&quot;,<br>            client=boto3.client(service_name=&#39;bedrock-runtime&#39;, region_name=&#39;us-east-1&#39;),<br>            model_kwargs={&quot;temperature&quot;:0.})</pre><p>Here, we initialize the Bedrock model using the Bedrock class. We specify the model ID which identifies the model version we want to use. Additionally, we provide a boto3 client for communication with the Bedrock runtime service, and specify model kwargs such as temperature.</p><blockquote><strong>Creating the CSV Agent</strong></blockquote><pre>agent = create_csv_agent(<br>    llm,<br>    &quot;heart_failure_clinical_records.csv&quot;,<br>    verbose=True,<br>    agent_type=AgentType.ZERO_SHOT_REACT_DESCRIPTION,<br>    handle_parsing_errors=True<br>)</pre><p>In this section, we create the LangChain CSV Agent using the create_csv_agent() function. We pass the initialized Bedrock model, the path to the CSV file containing the data we want to analyze, and some additional options such as the agent type and whether we want to handle parsing errors.</p><blockquote><strong>Interacting with the Agent</strong></blockquote><pre>agent.run(&quot;how many people have more than 50 years old?&quot;)</pre><p>Finally, we interact with the agent by asking a question about the data. We call the run() method of the agent and pass our question as an argument. The agent processes the question and provides a response based on the data contained in the CSV file.</p><p>In this case, the verbose process will be printed:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*OtLNRJ28mWl_kUJQMSrMtg.png" /><figcaption>CSV Agents Verbose and Answer</figcaption></figure><p>We see on the last line the response based on natural language:</p><pre>3684 people have more than 50 years old in the dataframe df.</pre><p>To confirm the result using Pandas library, use the following code:</p><pre>import pandas as pd<br><br>df = pd.read_csv(&#39;heart_failure_clinical_records.csv&#39;)<br><br>df[df[&#39;age&#39;]&gt;50].shape</pre><p>Response:</p><pre>(3684, 13)</pre><p>LangChain CSV Agents open up exciting possibilities for interacting with your data using natural language. Whether you’re exploring a dataset, generating insights, or performing complex analyses, LangChain makes the process intuitive and efficient. Give it a try and start talking to your data today!</p><p>I hope the insights shared here prove valuable in enhancing your work, regardless of your specific area of focus.</p><h3>👏🏽 I hope you enjoy the content! | <a href="https://www.linkedin.com/in/thallys-costalat-147700149/">Follow me on LinkedIn</a>.</h3><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=07ee3d35e9f7" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Quick Start on RAG (Retrieval-Augmented Generation) for Q&A using AWS Bedrock, ChromaDB, and…]]></title>
            <link>https://medium.com/@thallyscostalat/quick-start-on-rag-retrieval-augmented-generation-for-q-a-using-aws-bedrock-chromadb-and-64c35d966188?source=rss-180b8f46e6e2------2</link>
            <guid isPermaLink="false">https://medium.com/p/64c35d966188</guid>
            <category><![CDATA[genarative-ai]]></category>
            <category><![CDATA[langchain]]></category>
            <category><![CDATA[rags]]></category>
            <category><![CDATA[aws-bedrock]]></category>
            <category><![CDATA[chromadb]]></category>
            <dc:creator><![CDATA[thallyscostalat]]></dc:creator>
            <pubDate>Sun, 28 Jan 2024 12:55:00 GMT</pubDate>
            <atom:updated>2024-01-28T13:08:20.488Z</atom:updated>
            <content:encoded><![CDATA[<h3>Quick Start on RAG (Retrieval-Augmented Generation) for Q&amp;A using AWS Bedrock, ChromaDB, and LangChain</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/801/0*LhiAZT52tfY41mx9" /><figcaption><a href="https://qiita.com/cyberBOSE/items/1977a87a67857db7e6fd">Source</a></figcaption></figure><p>The RAG technique, or Retrieval Augmented Generation, is an advanced approach to questions and answers that combines elements of information retrieval and natural language generation. It stands out for its efficient use of embeddings and Vector DB to provide more contextual and informative responses.</p><p>Benefits of the RAG Technique:</p><ol><li>Advanced Contextualization: The use of embeddings allows the system to understand the semantic context of words, improving the quality of generated responses.</li><li>Efficient Retrieval: The Vector DB optimizes the retrieval of relevant information, contributing to more accurate and contextualized answers.</li><li>Integration of Language Models: The RAG technique integrates natural language generation with information retrieval, delivering more informative and relevant responses.</li></ol><p>In this post, we will explore step by step how to connect to AWS Bedrock, ChromaDB to create a VectorDB, and finally, implement a Q&amp;A retrieval chain using the LangChain library.</p><blockquote><strong>1. Configuring Environment Variables</strong></blockquote><p>To ensure seamless integration with AWS services, it’s crucial to set up the necessary environment variables.</p><p>First and foremost, we need to obtain the environment variables that identify the AWS user to access the required services, in our case, Bedrock via boto3.</p><pre>import os<br>AWS_ACCESS_KEY_ID = os.getenv(&#39;AWS_ACCESS_KEY_ID&#39;)<br>AWS_SECRET_ACCESS_KEY = os.getenv(&#39;AWS_SECRET_ACCESS_KEY&#39;)<br>REGION_NAME = os.getenv(&#39;AWS_REGION&#39;)</pre><p>To configure these variables, you can use the export command:</p><pre>export AWS_DEFAULT_REGION=us-east-1<br><br>export AWS_ACCESS_KEY_ID=&lt;YOUR_ACCESS_KEY&gt;<br><br>export AWS_SECRET_ACCESS_KEY=&lt;YOUR_SECRET_ACCESS_KEY&gt;</pre><blockquote><strong>2. Tools Importing</strong></blockquote><p>In this section, we’re importing essential tools and modules to make the entire process smooth. These include text processing, document loading, AWS Bedrock interaction, and more. This ensures that the subsequent steps are well-supported with the required functionalities.</p><pre>from langchain.text_splitter import RecursiveCharacterTextSplitter<br>from langchain.document_loaders import PyPDFLoader<br>from langchain.prompts import PromptTemplate<br>from langchain.embeddings.bedrock import BedrockEmbeddings<br>from langchain.llms.bedrock import Bedrock<br>from langchain.vectorstores import Chroma<br>from langchain.chains import RetrievalQA<br>import boto3</pre><blockquote><strong>3. AWS Bedrock Client Connection with boto3</strong></blockquote><p>For the most of cases, we’ll use boto3 library to set up a connection with AWS services and performe its functionalities using Python. This library simplifies the process of interacting with AWS services. The script bellow establishes a connection to the &#39;bedrock-runtime&#39; service.</p><pre>bedrock = boto3.client(service_name=&#39;bedrock-runtime&#39;, <br>                       region_name=AWS_DEFAULT_REGION,<br>                       aws_access_key_id=AWS_ACCESS_KEY_ID,<br>                       aws_secret_access_key=AWS_SECRET_ACCESS_KEY)</pre><blockquote><strong>4. PDF Files Loading</strong></blockquote><p>To illustrate the script’s functionality, let’s say we have some PDF files (one, in this case, but can be more than one) related to document analysis. The script efficiently loads these files using the PyPDFLoader, preparing them for further processing.</p><p>In this tutorial, we could use the following file as an example to formulate questions about it: <a href="https://docs.aws.amazon.com/pdfs/bedrock/latest/userguide/bedrock-ug.pdf">https://docs.aws.amazon.com/pdfs/bedrock/latest/userguide/bedrock-ug.pdf</a>.</p><pre>loaders = [PyPDFLoader(&quot;bedrock-ug.pdf&quot;),]</pre><blockquote><strong>5. VectorDB Creation</strong></blockquote><p>This step involves creating a Vector Database (VectorDB) from the loaded documents using ChromaDB library. It intelligently splits the documents and generates embeddings, providing a structured and organized representation of the content. The resulting Vectordb is then stored in a specified directory.</p><pre>docs = []<br><br>for loader in loaders:<br>    docs.extend(loader.load())<br>    <br>r_splitter = RecursiveCharacterTextSplitter(chunk_size=10000,<br>                                            chunk_overlap=100,<br>                                            separators=[&quot;\n\n&quot;, &quot;\n&quot;])<br><br>docs_splitted = r_splitter.split_documents(docs)<br><br>vector_store = Chroma.from_documents(documents=docs_splitted,<br>                                     embedding=BedrockEmbeddings(),<br>                                     persist_directory=&#39;vector_store/chroma/&#39;)</pre><blockquote><strong>6. LangChain Q&amp;A Retrieval Chain</strong></blockquote><p>Now, imagine you have questions related to the content of these documents. The script bellow sets up a powerful question-answering (QA) retrieval chain using LangChain. This chain uses the Bedrock language model (Claude 2) to answer your queries. The retrieval mechanism is enhanced with a pre-built vector store, making the entire QA process efficient.</p><pre>QUERY_PROMPT_TEMPLATE = &quot;&quot;&quot;\<br>H:<br>Answer the question based on the provided context. Do not create false information.<br>{context}<br>Question: {question}<br>A:<br>&quot;&quot;&quot;<br><br>qa_chain = RetrievalQA.from_chain_type(<br>        llm=Bedrock(model_id=&#39;anthropic.claude-v2&#39;, client=bedrock),<br>        retriever=vector_store.as_retriever(search_kwargs={&#39;k&#39;: 5}),<br>        return_source_documents=True,<br>        chain_type_kwargs={&quot;prompt&quot;: PromptTemplate.from_template(QUERY_PROMPT_TEMPLATE)}<br>    )</pre><blockquote><strong>7. Getting the results</strong></blockquote><p>Now, we can make some query related to the document passed and get the answer based on the retrieved documents chunks by the vector store retriever.</p><p>Question:</p><pre>question = &#39;How to use bedrock via boto3?&#39;</pre><p>Query:</p><pre>response = qa_chain({&quot;query&quot;: question})<br><br>response[&#39;result&#39;]</pre><p>Answer:</p><pre>Here is an example of how to use boto3 to call Amazon Bedrock models:<br><br>```python<br>import boto3<br>import json<br><br>bedrock = boto3.client(&#39;bedrock&#39;)<br><br>body = {<br>  &quot;prompt&quot;: &quot;Hello, how are you today?&quot;,<br>  &quot;modelId&quot;: &quot;meta.llama2-13b-chat-v1&quot;,<br>  &quot;parameters&quot;: {<br>    &quot;temperature&quot;: 0.5<br>  }  <br>}<br><br>response = bedrock.invoke_model(<br>  accept=&#39;application/json&#39;,<br>  contentType=&#39;application/json&#39;,<br>  body=json.dumps(body)<br>)<br><br>print(json.loads(response[&#39;body&#39;].read()))<br>```<br><br>The key steps are:<br><br>- Create a Bedrock client using `boto3.client(&#39;bedrock&#39;)`<br>- Construct the request body with the prompt, model ID, and any inference parameters<br>- Call `invoke_model()`, passing the body and setting content type and accept headers<br>- Parse the response body <br><br>The response will contain the model output. You can customize the request by using different models, prompts, and parameters.<br><br>The Bedrock client also provides other methods like `list_models`.</pre><p>This post provides a comprehensive overview of the RAG technique for questions and answers, combining information retrieval and natural language generation. The benefits of RAG include advanced contextualization, efficient retrieval using Vector DB, and the integration of language models for more informative responses.</p><p>I hope the insights shared here prove valuable in enhancing your work, regardless of your specific area of focus.</p><h4>👏🏽 I hope you enjoy the content! | <a href="https://www.linkedin.com/in/thallys-costalat-147700149/">Follow me on LinkedIn</a>.</h4><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=64c35d966188" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Chunking Strategies Optimization for Retrieval Augmented Generation (RAG) in the Context of…]]></title>
            <link>https://medium.com/@thallyscostalat/chunking-strategies-optimization-for-retrieval-augmented-generation-rag-in-the-context-of-e47cc949931d?source=rss-180b8f46e6e2------2</link>
            <guid isPermaLink="false">https://medium.com/p/e47cc949931d</guid>
            <category><![CDATA[retrieval-augmented]]></category>
            <category><![CDATA[nlp]]></category>
            <category><![CDATA[generative-ai-tools]]></category>
            <category><![CDATA[embedding]]></category>
            <category><![CDATA[langchain]]></category>
            <dc:creator><![CDATA[thallyscostalat]]></dc:creator>
            <pubDate>Sun, 14 Jan 2024 12:03:47 GMT</pubDate>
            <atom:updated>2024-01-14T12:03:47.238Z</atom:updated>
            <content:encoded><![CDATA[<h3>Chunking Strategies Optimization for Retrieval Augmented Generation (RAG) in the Context of Generative AI</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/986/0*v-rMAW9xuKERbyJF.png" /><figcaption><a href="https://magazine.sebastianraschka.com/p/ai-and-open-source-in-2023">Source</a></figcaption></figure><p>When handling external documents, the first step involves breaking them down into smaller segments to extract detailed features, which are then embedded to convey their semantics. However, embedding excessively large or overly small text segments can result in less-than-optimal outcomes. Therefore, it is crucial to determine the optimal segment size for documents within the corpus to ensure the accuracy and relevance of the retrieved results.</p><p>Selecting an appropriate segmentation strategy requires careful consideration of several essential factors, including the nature of the indexed content, the embedding model, its optimal block size, the expected length and complexity of user queries, and how the retrieved results are utilized in a specific application. This short paper introduces key chunking strategies, including fixed methods based on characters, recursive approaches balancing fixed sizes and natural language structures, and advanced techniques considering semantic topic changes.</p><blockquote><strong><em>1. Fixed-size (in characters) Overlapping Sliding Window.</em></strong></blockquote><p>This method involves dividing text into fixed-size chunks based on character count. The simplicity of implementation and the inclusion of overlapping segments aim to prevent cutting sentences or thoughts. However, limitations include imprecise control over context size, the risk of cutting words or sentences, and a lack of semantic consideration. Suitable for exploratory analysis but not recommended for tasks requiring deep semantic understanding.</p><p>Example using<em> LangChain</em>:</p><pre>text = &quot;...&quot; # your text<br>from langchain.text_splitter import CharacterTextSplitter<br>text_splitter = CharacterTextSplitter(<br>    chunk_size = 256,<br>    chunk_overlap  = 20<br>)<br>docs = text_splitter.create_documents([text])</pre><blockquote><strong><em>2. Recursive Structure Aware Splitting.</em></strong></blockquote><p>A hybrid method combining fixed-size sliding window and structure-aware splitting. It attempts to balance fixed chunk sizes with linguistic boundaries, offering precise context control. Implementation complexity is higher, with a risk of variable chunk sizes. Effective for tasks requiring granularity and semantic integrity but not recommended for quick tasks or unclear structural divisions.</p><p>Example using<em> LangChain</em>:</p><pre>text = &quot;...&quot; # your text<br>from langchain.text_splitter import RecursiveCharacterTextSplitter<br>text_splitter = RecursiveCharacterTextSplitter(<br>    chunk_size = 256,<br>    chunk_overlap  = 20,<br>    separators = [&quot;\n\n&quot;, &quot;\n&quot;]<br>)<br><br>docs = text_splitter.create_documents([text])</pre><blockquote><strong><em>3. Structure Aware Splitting (by Sentence, Paragraph).</em></strong></blockquote><p>This method considers the natural structure of text, dividing it based on sentences, paragraphs, sections, or chapters. Respecting linguistic boundaries preserves semantic integrity, but challenges arise with varying structural complexity. Effective for tasks requiring context and semantics, but unsuitable for texts lacking defined structural divisions.</p><p>Example:</p><pre>text = &quot;...&quot; # your text<br>docs = text.split(&quot;.&quot;)</pre><blockquote><strong><em>4. Content-Aware Splitting (Markdown, LaTeX, HTML).</em></strong></blockquote><p>This method focuses on content type and structure, especially in structured documents like Markdown, LaTeX, or HTML. It ensures content types are not mixed within chunks, maintaining integrity. Challenges include understanding specific syntax and unsuitability for unstructured documents. Useful for structured documents but not recommended for unstructured content.</p><p>Example for Markdown texts using <em>LangChain</em>:</p><pre>from langchain.text_splitter import MarkdownTextSplitter<br>markdown_text = &quot;...&quot;<br><br>markdown_splitter = MarkdownTextSplitter(chunk_size=100, chunk_overlap=0)<br>docs = markdown_splitter.create_documents([markdown_text])</pre><p>Example for LaTex texts using <em>LangChain</em>:</p><pre>from langchain.text_splitter import LatexTextSplitter<br>latex_text = &quot;...&quot;<br>latex_splitter = LatexTextSplitter(chunk_size=100, chunk_overlap=0)<br>docs = latex_splitter.create_documents([latex_text])</pre><blockquote><strong><em>5. NLP Chunking: Tracking Topic Changes.</em></strong></blockquote><p>A sophisticated approach based on semantic understanding, dividing text into chunks by detecting significant shifts in topics. Ensures semantic consistency but demands advanced NLP techniques. Effective for tasks requiring semantic context and topic continuity but not suitable for high topic overlap or simple chunking tasks.</p><p>Example using NLTK toolkit from <em>LangChain</em>:</p><pre>text = &quot;...&quot; # your text<br>from langchain.text_splitter import NLTKTextSplitter<br>text_splitter = NLTKTextSplitter()<br>docs = text_splitter.split_text(text)</pre><p>In conclusion, effective handling of external documents necessitates a thoughtful segmentation strategy, considering various factors such as the nature of indexed content, the embedding model, user query expectations, and application-specific requirements. The presented chunking strategies offer a spectrum of approaches, each with its strengths and limitations.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=e47cc949931d" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Accelerating Large-Scale Data Processing with Dask and Python]]></title>
            <link>https://medium.com/@thallyscostalat/accelerating-large-scale-data-processing-with-dask-and-python-ce4d59821265?source=rss-180b8f46e6e2------2</link>
            <guid isPermaLink="false">https://medium.com/p/ce4d59821265</guid>
            <category><![CDATA[big-data]]></category>
            <category><![CDATA[python]]></category>
            <category><![CDATA[dask]]></category>
            <category><![CDATA[parallel-computing]]></category>
            <category><![CDATA[high-performance]]></category>
            <dc:creator><![CDATA[thallyscostalat]]></dc:creator>
            <pubDate>Mon, 10 Jul 2023 23:22:31 GMT</pubDate>
            <atom:updated>2023-07-10T23:35:49.296Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*aMIEvhEJCT1-Q9ly.png" /><figcaption>Ultimate guide to handle Big Datasets for Machine Learning using Dask (in Python), <a href="https://www.analyticsvidhya.com/blog/author/aishwaryasingh/">Aishwarya Singh</a>.</figcaption></figure><p>Processing large datasets efficiently is crucial for data scientists, software engineers, and analysts. As data volume and complexity increase, it becomes necessary to find efficient solutions for timely data processing. In this article, we will explore how Dask, a Python library, can accelerate the processing of large datasets and compare it to Pandas, a widely used library for data manipulation and analysis.</p><blockquote><strong>What is Dask?</strong></blockquote><p>Dask is a powerful Python library designed to handle large-scale data processing. It allows for parallel and distributed task execution, providing a scalable and efficient solution. With its familiar API, Dask facilitates the transition from Pandas to distributed computing, making it an attractive choice for processing large datasets.</p><blockquote><strong>Comparison between Dask and Pandas</strong></blockquote><p>While Pandas is a popular library for data manipulation and analysis in Python, its performance may be limited when dealing with large datasets that don’t fit into memory. Dask overcomes this limitation by breaking down computations into smaller tasks and executing them in parallel. It leverages distributed computing techniques to handle large volumes of data efficiently.</p><p>Dask offers two main data structures: Dask Arrays and Dask DataFrames. Dask Arrays extend the functionality of NumPy, allowing for parallel computation on large arrays. Dask DataFrames mimic the Pandas DataFrame API and provide scalable and distributed data processing capabilities.</p><blockquote><strong>Benefits of Dask</strong></blockquote><ol><li>Scalability: Dask enables seamless scalability by leveraging distributed computing resources. It can handle datasets that exceed the memory capacity of a single machine.</li><li>Familiarity: Dask provides a familiar programming interface for those already familiar with Pandas, making it easy to transition existing code to Dask.</li><li>Efficiency: Dask’s ability to parallelize computations across cores or clusters leads to significant performance improvements when processing large datasets.</li></ol><blockquote><strong>Pandas Time Comparison</strong></blockquote><p>To compare the performance of Dask and Pandas, we will use a publicly available dataset called the “Disease Symptoms and Patient Profile Dataset” (<a href="https://www.kaggle.com/datasets/uom190346a/disease-symptoms-and-patient-profile-dataset?resource=download">https://www.kaggle.com/datasets/uom190346a/disease-symptoms-and-patient-profile-dataset?resource=download</a>). This dataset contains a comprehensive compilation of symptoms and patient profiles for a wide range of diseases.</p><p>In the code examples bellow, we load the dataset in both Pandas and Dask. We then apply a filtering operation to select only rows with an Age greater than 50 years and save the filtered dataset to a new CSV file. The time module is used to measure the load time and processing time of the operations.</p><p>Code example using Pandas:</p><pre>import pandas as pd<br>import time<br><br># Loading the CSV file with Pandas<br>start_time = time.time()<br>df = pd.read_csv(&#39;Disease_symptom_and_patient_profile_dataset.csv&#39;)<br>load_time = time.time() - start_time<br><br># Performing processing operations on the DataFrame<br>start_time = time.time()<br>df_filtered = df[df[&#39;Age&#39;] &gt; 50]<br>df_filtered.to_csv(&#39;filtered_ages.csv&#39;, index=False)<br>processing_time = time.time() - start_time<br><br>print(&quot;Loading time: &quot;, load_time)<br>print(&quot;Processing time: &quot;, processing_time)</pre><pre>Loading time:  0.022681713104248047<br>Processing time:  0.08066248893737793</pre><p>Code example using Dask:</p><pre>import dask.dataframe as dd<br>import time<br><br># Loading the CSV file with Dask<br>start_time = time.time()<br>df = dd.read_csv(&#39;Disease_symptom_and_patient_profile_dataset.csv&#39;)<br>load_time = time.time() - start_time<br><br># Performing processing operations on the DataFrame<br>start_time = time.time()<br>df_filtered = df[df[&#39;Age&#39;] &gt; 50]<br>df_filtered.to_csv(&#39;filtered_ages_dask.csv&#39;, index=False)<br>processing_time = time.time() - start_time<br><br>print(&quot;Load time: &quot;, load_time)<br>print(&quot;Processing time: &quot;, processing_time)</pre><pre>Load time:  0.015008926391601562<br>Processing time:  0.026233673095703125</pre><p>By comparing the results obtained with Pandas and Dask, we can observe the performance differences. In many cases, Dask will exhibit faster load times as it doesn’t need to load all the data into memory at once. Additionally, Dask can process operations in parallel, potentially leading to faster processing times compared to Pandas.</p><p>Processing large datasets efficiently is a common challenge in data analysis. Dask provides a powerful solution for handling large-scale data processing, offering scalability and performance improvements over Pandas. By leveraging distributed computing, Dask enables the processing of datasets that exceed the memory capacity of a single machine.</p><p>For those looking to enhance performance and efficiently handle large datasets, Dask is a valuable library to explore and incorporate into data analysis workflows with Python.</p><p>Please note that the dataset used in this example is for illustrative purposes only, and you can adapt the code to use any publicly available dataset of your choice.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=ce4d59821265" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Data Pipeline Automation With Apache Airflow and Python]]></title>
            <link>https://medium.com/@thallyscostalat/easy-data-pipeline-automation-with-apache-airflow-and-python-83a13e8f67e9?source=rss-180b8f46e6e2------2</link>
            <guid isPermaLink="false">https://medium.com/p/83a13e8f67e9</guid>
            <category><![CDATA[etl-pipeline]]></category>
            <category><![CDATA[automation]]></category>
            <category><![CDATA[airflow]]></category>
            <category><![CDATA[data-engineering]]></category>
            <category><![CDATA[python]]></category>
            <dc:creator><![CDATA[thallyscostalat]]></dc:creator>
            <pubDate>Sat, 10 Jun 2023 14:27:29 GMT</pubDate>
            <atom:updated>2023-06-10T23:03:04.456Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/640/0*8o8sz1IAroLqnl4K" /></figure><p>Automating data pipelines is an essential practice for companies dealing with large volumes of information. The process of collecting, transforming, and analyzing data can be complex and time-consuming. However, with the help of appropriate tools like Airflow, it is possible to automate and expedite these tasks.</p><p>In this article, we will explore how to use Airflow to create an automated data pipeline in Python and provide a practical code example. Additionally, we will understand the concept of Directed Acyclic Graphs (DAGs) and how they are fundamental for workflow management.</p><blockquote><strong>What is Apache Airflow?</strong></blockquote><p>Apache Airflow is a workflow management platform that enables scheduling, monitoring, and executing tasks within a data pipeline. It is based on Python and follows a “Workflow as Code” approach, making it easy to create and maintain complex workflows.</p><blockquote><strong>The Concept of DAG</strong></blockquote><p>A key element in the context of Apache Airflow is the concept of a Directed Acyclic Graph (DAG). A DAG is a visual representation of a workflow consisting of a set of interconnected tasks. In this context, tasks are represented by the nodes of the graph, and dependencies between tasks are represented by edges.</p><p>It is important to note that a DAG is acyclic, meaning that there are no cycles in the graph. This characteristic ensures that task dependencies are structured to avoid infinite loops or undesired behaviors. Additionally, the edges of the graph indicate the direction of dependency flow between tasks, i.e., a task depends on other tasks to be executed.</p><p>On the next steps, we will learn how to create a data pipeline that makes a requisition to the Weather API (<a href="https://www.meteomatics.com/en/weather-api/how-to-use-weather-api/">https://www.meteomatics.com/en/weather-api/how-to-use-weather-api/</a>) and saving the data into a text file with a daily schedule.</p><blockquote><strong>Step 1: Setting up the Environment</strong></blockquote><p>Before we begin, it is necessary to have Airflow installed in your environment. You can install it using pip by executing the following command in the terminal:</p><pre>pip install apache-airflow</pre><blockquote><strong>Step 2: Creating an User Admin Account</strong></blockquote><p>After we have installed the Apache Airflow, we need to set up an user admin to monitoring the scheduled jobs. To do that, execute the following command in the terminal to define a generic user admin profile:</p><pre>airflow users  create --role Admin --username admin --email admin --firstname admin --lastname admin --password admin</pre><blockquote><strong>Step 3: Defining the Pipeline</strong></blockquote><p>Now, let’s create a simple example of a data pipeline that involves making a requisition to the Weather API (<a href="https://www.meteomatics.com/en/weather-api/how-to-use-weather-api/">https://www.meteomatics.com/en/weather-api/how-to-use-weather-api/</a>) and saving the data into a text file with a daily schedule.</p><p>Inside the Airflow installing folder, create a folder called ‘<em>dags</em>’. Then, inside the folder ‘<em>dags</em>’, create a file called ‘<em>pipeline.py</em>’ and add the following code:</p><pre>from airflow import DAG<br>from airflow.operators.python_operator import PythonOperator<br>from datetime import datetime<br>import pandas as pd<br>import requests<br><br># Define the function to consume the weather API<br>def fetch_weather_data():<br>    url = &quot;https://api.weatherapi.com/v1/current.json&quot;<br>    params = {<br>        &quot;key&quot;: &quot;YOUR_API_KEY&quot;,<br>        &quot;q&quot;: &quot;City&quot;  # Replace with your preferred city<br>    }<br>    response = requests.get(url, params=params)<br>    data = response.json()<br>    return data<br><br># Define the function to save data to a file<br>def save_to_file(data):<br>    with open(&quot;weather_data.txt&quot;, &quot;a&quot;) as file:<br>        file.write(data)<br>        file.write(&quot;\n&quot;)<br><br># Define the Airflow DAG (Directed Acyclic Graph)<br>dag = DAG(<br>    &#39;pipeline_dados&#39;,<br>    description=&#39;Pipeline de Dados&#39;,<br>    schedule_interval=&#39;@daily&#39;,<br>    start_date=datetime.utcnow(),<br>)<br><br># Define the tasks in the DAG<br>task_fetch_weather_data = PythonOperator(<br>    task_id=&#39;fetch_weather_data&#39;,<br>    python_callable=fetch_weather_data,<br>    dag=dag<br>)<br><br>task_save_to_file = PythonOperator(<br>    task_id=&#39;save_to_file&#39;,<br>    python_callable=save_to_file,<br>    op_kwargs={&#39;data&#39;: &#39;{{ ti.xcom_pull(task_ids=&quot;fetch_weather_data&quot;) }}&#39;},  # Get data from the previous task<br>    dag=dag<br>)<br><br># Define the dependencies between tasks<br>task_fetch_weather_data &gt;&gt; task_save_to_file</pre><p>In the above code, we added two new functions:</p><ol><li>fetch_weather_data(): This function consumes a weather API using the requests library. Make sure to replace &#39;YOUR_API_KEY&#39; with your own valid API key and &#39;City&#39; with a real city that you want to retrieve data for.</li><li>save_to_file(data): This function receives the data returned by the weather API and saves the information to a log file named &#39;<em>weather_data.txt&#39;</em>. You can customize the file name and format as needed.</li></ol><p>In the task definition of task_save_to_file, we use op_kwargs to pass the data returned by the previous task (fetch_weather_data()) to the save_to_file(data) function.</p><blockquote><strong>Step 4: Running the Pipeline</strong></blockquote><p>After defining the pipeline in code, it’s time to execute it. Open a terminal window and navigate to the directory where the ‘<em>pipeline.py</em>’ file is located. Then, run the following command:</p><pre>airflow scheduler</pre><p>This command starts the Airflow scheduler, which is responsible for scheduling and executing the pipeline tasks.</p><p>In another terminal window, run the following command:</p><pre>airflow webserver</pre><p>This command starts the Airflow web server, allowing you to visualize and monitor the progress of your pipeline through a web interface.</p><blockquote><strong>Step 5: Activating the Pipeline</strong></blockquote><p>First, we need to login into the Airflow web server environment through out the web interface, using the credentials defined on the Step 2.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*VhD1jnCR_aTnuxLW_nVZTA.png" /><figcaption>Username: admin, Password: admin</figcaption></figure><p>After that, the DAGs window will appear with all DAGs registered in the Airflow server, as the following image bellow:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*UOYFcmY0CSBMZhlbDhAZuQ.png" /><figcaption>Airflow DAGs Window</figcaption></figure><p>Now, we need to search for the DAG created in this tutorial named ‘<em>pipeline_dados</em>’, and then activate it switching up the left side bar as the following image bellow:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*YvZg9GY-IrX-jwzrHoC03A.png" /></figure><p>Now, our data pipeline is already activated. Daily, it will storage the API requisitions in a .txt file named ‘<em>weather_data.txt</em>’.</p><p>In this article, we explored how to automate a data pipeline using Airflow in Python. Airflow is a powerful tool that allows you to create and manage complex workflows. We also understood the concept of DAGs, which are essential in Airflow for defining task dependencies within a pipeline. By using Airflow, you can schedule, monitor, and execute your tasks in an automated manner, enhancing efficiency and productivity in data manipulation.</p><p>In the provided code example, we added the functionality to consume a weather API and save the information to a log file. This demonstrates how Airflow can be used to integrate various data sources and automate tasks efficiently. You can adapt this example to fit your specific needs, such as saving the data to a database instead of a file.</p><p>Continue exploring the official Apache Airflow documentation (<a href="https://airflow.apache.org/docs/">https://airflow.apache.org/docs/</a>) and available resources to deepen your knowledge and make the most of this data pipeline automation tool. With Airflow, you’ll be prepared to tackle complex challenges and drive efficient data analysis.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=83a13e8f67e9" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[10 Essential SQL Commands that Every Data Professional Should Know]]></title>
            <link>https://medium.com/@thallyscostalat/10-essential-sql-commands-every-data-professional-should-know-49cecd79bb0b?source=rss-180b8f46e6e2------2</link>
            <guid isPermaLink="false">https://medium.com/p/49cecd79bb0b</guid>
            <category><![CDATA[database]]></category>
            <category><![CDATA[data-engineering]]></category>
            <category><![CDATA[query-language]]></category>
            <category><![CDATA[sql]]></category>
            <category><![CDATA[tabular-data]]></category>
            <dc:creator><![CDATA[thallyscostalat]]></dc:creator>
            <pubDate>Thu, 18 May 2023 14:04:23 GMT</pubDate>
            <atom:updated>2023-05-20T11:27:46.117Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/800/0*F0Tb3LtoIPzHRcUx" /><figcaption>Image: How to set up and learn SQL on Mac, By Lucy Hattersley</figcaption></figure><p>The SQL (Structured Query Language) plays a crucial role in the world of data. It is a powerful tool that allows you to extract, manipulate, and manage information in relational databases. For data professionals, mastering SQL commands is essential to perform complex queries and gain valuable insights. In this article, we will highlight the ten essential SQL commands that every data professional should know.</p><blockquote>1. SELECT</blockquote><p>The SELECT command is the foundation of SQL. It allows you to select and retrieve data from one or more tables. With SELECT, you can specify the columns you want to return and apply filters and conditions to refine your query.</p><p>Example:</p><pre>SELECT name, age<br>FROM customers<br>WHERE age &gt; 30;</pre><p>In this example, we select the names and ages of customers whose age is greater than 30. This allows us to retrieve specific information about older customers in our database.</p><blockquote>2. WHERE</blockquote><p>The WHERE command is used in conjunction with SELECT to filter data based on one or more conditions. It allows you to specify criteria to return only the rows that meet those conditions, making your queries more precise and relevant.</p><p>Example:</p><pre>SELECT name, surname<br>FROM customers<br>WHERE city = &#39;São Paulo&#39;;</pre><p>In the given example, we select the names and surnames of customers who reside in São Paulo. Using WHERE, we filter only the records that match this specific geographical condition.</p><blockquote>3. GROUP BY</blockquote><p>The GROUP BY command is used to group records based on one or more columns. It is often combined with aggregation functions like SUM, COUNT, AVG, among others, to perform calculations on specific groups of data. This is particularly useful for analysis and summary reports.</p><p>Example:</p><pre>SELECT department, COUNT(*) as total_employees<br>FROM employees<br>GROUP BY department;</pre><p>In the provided example, we group employees by department and count the total number of employees in each department. The result will be a summary that allows us to identify how many employees exist in each work area.</p><blockquote>4. JOIN</blockquote><p>The JOIN command is essential for combining data from multiple tables. With it, you can merge related information between tables using common columns as joining points. There are different types of JOIN, such as INNER JOIN, LEFT JOIN, RIGHT JOIN, and FULL JOIN, each with its own specificities.</p><p>Example:</p><pre>SELECT orders.number, customers.name<br>FROM orders<br>JOIN customers ON orders.customer_id = customers.id;</pre><p>In the presented example, we join order information with customer information based on the “customer_id” field. This allows us to retrieve the order numbers along with the corresponding customer names in a single query.</p><blockquote>5. ORDER BY</blockquote><p>The ORDER BY command allows you to sort the results of a query based on one or more columns. You can specify whether the sorting should be in ascending (ASC) or descending (DESC) order. This is useful for organizing the results according to specific criteria and facilitating data interpretation.</p><p>Example:</p><pre>SELECT name, age<br>FROM customers<br>ORDER BY age DESC;</pre><p>In the given example, we are ordering customers by age in descending order. Thus, we will obtain a list of the oldest customers first, which can be useful for identifying relevant age segments in our analysis.</p><blockquote>6. INSERT INTO</blockquote><p>The INSERT INTO command is used to add new data to an existing table. With it, you can specify the columns and values you want to insert. This is crucial for updating your database with relevant and up-to-date information.</p><p>Example:</p><pre>INSERT INTO customers (name, age)<br>VALUES (&#39;John&#39;, 35);</pre><p>In the provided example, we are inserting a new customer with the name “John” and age of 35 into our customers table.</p><blockquote>7. UPDATE</blockquote><p>The UPDATE command allows you to modify existing data in a table. You can update specific values in certain columns based on defined conditions. This is useful when you need to fix errors, update records, or make bulk changes.</p><p>Example:</p><pre>UPDATE customers<br>SET age = 36<br>WHERE name = &#39;John&#39;;</pre><p>In the presented example, we are updating the age of the customer with the name “John” to 36 years. This allows us to keep our data up-to-date and correct incorrect information.</p><blockquote>8. DELETE</blockquote><p>The DELETE command is used to remove records from a table. With it, you can delete specific rows based on defined criteria. However, it is important to use it with caution and be aware that deleted data cannot be easily recovered.</p><p>Example:</p><pre>DELETE FROM customers<br>WHERE name = &#39;John&#39;;</pre><p>In the provided example, we are deleting the customer with the name “John” from our customers table. This will result in the permanent removal of that customer’s data.</p><blockquote>9. CREATE TABLE</blockquote><p>The CREATE TABLE command is used to create a new table in a database. You can define the table name, columns, their data types, and constraints such as primary and foreign keys. Mastering this command is essential for database modeling and structuring.</p><p>Example:</p><pre>CREATE TABLE customers (<br>    id INT PRIMARY KEY,<br>    name VARCHAR(50),<br>    age INT<br>);</pre><p>In the presented example, we are creating a new table called “customers” with three columns: “id” (integer primary key), “name” (text with a maximum of 50 characters), and “age” (integer).</p><blockquote>10. ALTER TABLE</blockquote><p>The ALTER TABLE command is used to modify the structure of an existing table. With it, you can add columns, change data types, rename columns, add constraints, and make other changes to the table structure. This command is useful when you need to adjust the table schema to accommodate new requirements or fix issues.</p><p>Example:</p><pre>ALTER TABLE customers<br>ADD address VARCHAR(100);</pre><p>In the provided example, we are adding a new column called “address” to the customers table. This allows us to store address information for each customer.</p><p>Mastering SQL commands is essential for data professionals working with relational databases. The provided examples illustrate the syntax and usage of the ten mentioned commands. By mastering these commands, you will be able to extract valuable information, perform powerful analysis, and make informed decisions based on the data. Invest time and effort into improving your SQL skills, opening a world of possibilities to make the most of the available data in your work environment.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=49cecd79bb0b" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Quick Start On Machine Learning Models Deployment with Flask API]]></title>
            <link>https://medium.com/@thallyscostalat/deploying-machine-learning-models-with-flask-api-14fdc05f417d?source=rss-180b8f46e6e2------2</link>
            <guid isPermaLink="false">https://medium.com/p/14fdc05f417d</guid>
            <category><![CDATA[deployment]]></category>
            <category><![CDATA[python]]></category>
            <category><![CDATA[flask]]></category>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[api-development]]></category>
            <dc:creator><![CDATA[thallyscostalat]]></dc:creator>
            <pubDate>Thu, 11 May 2023 12:22:57 GMT</pubDate>
            <atom:updated>2024-01-28T13:16:35.639Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*e1fG1lf-NRG3rmsQ" /></figure><p>Deploying machine learning models is a crucial step in bringing data science and AI solutions to the real world. In this article, we will explore how to create a Flask API to deploy a machine learning model. Flask is a micro web framework written in Python that is perfect for building RESTful APIs. We will use the scikit-learn library to train a linear regression model on a cars dataset, and then deploy the model using Flask API.</p><blockquote>Step 1: Prepare the dataset</blockquote><p>The first step is to prepare the dataset. For this example, we will use the cars dataset from the seaborn library. We will remove any missing values and select the relevant features. The code for this step is shown below.</p><pre>import pandas as pd<br>from sklearn.linear_model import LinearRegression<br>import joblib<br><br># Load the dataset<br>cars = pd.read_csv(&#39;https://raw.githubusercontent.com/selva86/datasets/master/Cars93_miss.csv&#39;)<br><br># Remove missing values<br>cars = cars.dropna()<br><br># Select features<br>X = cars[[&#39;MPG.city&#39;, &#39;EngineSize&#39;, &#39;Horsepower&#39;, &#39;RPM&#39;, &#39;Rev.per.mile&#39;, &#39;Fuel.tank.capacity&#39;, &#39;Length&#39;, &#39;Width&#39;, &#39;Turn.circle&#39;, &#39;Weight&#39;]]<br>y = cars[&#39;Price&#39;]</pre><blockquote>Step 2: Train and save the machine learning model</blockquote><p>The next step is to train and save the machine learning model. We will use scikit-learn’s LinearRegression class to create a linear regression model. The code for this step is shown below.</p><pre># Train the model<br>model = LinearRegression()<br>model.fit(X, y)<br><br># Save the model<br>joblib.dump(model, &#39;model.pkl&#39;)</pre><blockquote>Step 3: Create the Flask API</blockquote><p>Now that we have trained the machine learning model, we can create a Flask API to deploy it. The Flask API will listen for HTTP requests and respond with predictions from the machine learning model. We will save and call this API script as <em>‘deploy.py’</em>. The code for this step is shown below.</p><pre>from flask import Flask, request, jsonify<br>import joblib<br>import numpy as np<br><br>app = Flask(__name__)<br><br># Load the model<br>model = joblib.load(&#39;model.pkl&#39;)<br><br># Define the endpoint<br>@app.route(&#39;/predict&#39;, methods=[&#39;POST&#39;])<br>def predict():<br>    # Get the data from the request<br>    data = request.get_json()<br><br>    # Transform the data into a numpy array<br>    X = np.array([data[&#39;MPG.city&#39;], data[&#39;EngineSize&#39;], data[&#39;Horsepower&#39;], data[&#39;RPM&#39;],<br>                  data[&#39;Rev.per.mile&#39;], data[&#39;Fuel.tank.capacity&#39;], data[&#39;Length&#39;], data[&#39;Width&#39;],<br>                  data[&#39;Turn.circle&#39;], data[&#39;Weight&#39;]])<br><br>    # Make a prediction<br>    prediction = model.predict(X.reshape(1, -1))<br><br>    # Return the prediction as a JSON object<br>    response = {&#39;prediction&#39;: float(prediction)}<br>    return jsonify(response)<br><br>if __name__ == &#39;__main__&#39;:<br>    app.run(port=8000)</pre><p>The above code creates a Flask application that listens for POST requests to the /predict endpoint. When it receives a request, it extracts the data from the request, transforms it into a numpy array, and passes it to the machine learning model to make a prediction. Finally, the prediction is returned as a JSON object.</p><blockquote>Step 4: Test the Flask API</blockquote><p>To run your Flask API’s deploy.py file in the terminal, you can use the following command:</p><pre>python deploy.py</pre><p>This command will start your Flask application and it will be available at the URL <a href="http://localhost:5000/">http://localhost:8000</a>.</p><p>To test your API’s /predict endpoint, you can use curl or the requests library in Python to send a POST request with the input data.</p><p>Using curl, you can execute the following command in a terminal:</p><pre>curl -X POST -H &quot;Content-Type: application/json&quot; -d &quot;{\&quot;MPG.city\&quot;: 25, \&quot;EngineSize\&quot;: 4, \&quot;Horsepower\&quot;: 150, \&quot;RPM\&quot;: 5500, \&quot;Rev.per.mile\&quot;: 2150, \&quot;Fuel.tank.capacity\&quot;: 16, \&quot;Length\&quot;: 190, \&quot;Width\&quot;: 67, \&quot;Turn.circle\&quot;: 37, \&quot;Weight\&quot;: 3053}&quot; http://localhost:8000/predict</pre><p>In this article, we covered how to create a Flask API to serve a machine learning model. We started by training a simple model on a dataset, then saved the model using joblib. We then created a Flask API and loaded the saved model. We defined an endpoint to receive JSON data and make a prediction using the loaded model, returning the prediction as a JSON response.</p><p>Deploying machine learning models as APIs can be a powerful way to make predictions and integrate with other applications. With Flask, it’s easy to get started building and serving your own machine learning models.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=14fdc05f417d" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[How to Use the GPT Chatbot API with Python and Build Your Own Intelligent Virtual Assistant.]]></title>
            <link>https://medium.com/@thallyscostalat/how-to-use-the-gpt-chatbot-api-with-python-and-build-your-own-intelligent-virtual-assistant-14b5c21051a5?source=rss-180b8f46e6e2------2</link>
            <guid isPermaLink="false">https://medium.com/p/14b5c21051a5</guid>
            <category><![CDATA[artificial-intelligence]]></category>
            <category><![CDATA[python]]></category>
            <category><![CDATA[chatgpt]]></category>
            <category><![CDATA[generative-ai]]></category>
            <category><![CDATA[chatbot-design]]></category>
            <dc:creator><![CDATA[thallyscostalat]]></dc:creator>
            <pubDate>Sat, 29 Apr 2023 13:13:10 GMT</pubDate>
            <atom:updated>2023-04-29T16:35:19.407Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/554/1*xXTtSmJQJI0bCOyQqdr6Rw.png" /></figure><p>Artificial intelligence has rapidly advanced in recent years, enabling a wide range of incredible applications. One such application is the creation of chatbots capable of answering questions and interacting with users in a natural manner. In this article, we will explore how to use the Chatbot GPT-3.5 API to perform queries using Python, allowing you to build your own intelligent virtual assistants.</p><blockquote><strong>What is GPT-3.5?</strong></blockquote><p>GPT-3.5 is an artificial intelligence language model developed by OpenAI. It can generate text in response to a given input text, making it ideal for constructing advanced chatbots. Through the GPT-3.5 API, we can interact with the model and receive real-time responses.</p><blockquote><strong>Setting up the Environment</strong></blockquote><p>Before we can start using the GPT-3.5 API, we need to set up the development environment. Make sure you have Python installed on your machine and install the “openai” library using pip:</p><pre>pip install openai</pre><blockquote><strong>Authentication</strong></blockquote><p>To use the GPT-3.5 API, you need a valid API key from OpenAI. You can obtain it from the official OpenAI website (<a href="https://platform.openai.com/">https://platform.openai.com/</a>). Once you have the key, you can configure the authentication in your Python code:</p><pre>import openai<br><br>openai.api_key = &#39;YOUR_API_KEY&#39;</pre><blockquote><strong>Performing queries</strong></blockquote><p>With the “openai” library installed and authentication configured, we are ready to make queries to GPT-3.5. We will use the openai.Completion.create() function to send a request to the API and receive a response. Here&#39;s an example:</p><pre>response = openai.Completion.create(<br>  engine=&quot;text-davinci-003&quot;,<br>  prompt=&quot;What is the capital of Brazil?&quot;,<br>  max_tokens=50,<br>  n=1,<br>  stop=None,<br>  temperature=0.7<br>)<br><br>answer = response.choices[0].text.strip()<br>print(answer)</pre><p>In this example, we are using the “text-davinci-003” model, providing the question “What is the capital of Brazil?” as input. We also set the max_tokens parameter to limit the response size to 50 tokens. The response will be stored in the answer variable and printed to the console.</p><blockquote><strong>Customizing Queries</strong></blockquote><p>You can customize queries to GPT-3.5 according to your needs. For instance, you can provide context or add specific instructions to obtain more accurate responses. Experiment with different configurations and parameters to achieve even better results.</p><p>The GPT-3.5 API is a powerful tool for creating intelligent chatbots. In this article, you learned how to set up the development environment, authenticate with the API, and perform queries using Python. Now, you can further explore this technology and build your own advanced virtual assistants. Harness the potential it offers and unleash your creativity!</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=14b5c21051a5" width="1" height="1" alt="">]]></content:encoded>
        </item>
    </channel>
</rss>