<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Gokberk Ozsoy on Medium]]></title>
        <description><![CDATA[Stories by Gokberk Ozsoy on Medium]]></description>
        <link>https://medium.com/@gokberkozsoy?source=rss-d25b4d8fc601------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/0*Tm_2brgJIQbN1dn-</url>
            <title>Stories by Gokberk Ozsoy on Medium</title>
            <link>https://medium.com/@gokberkozsoy?source=rss-d25b4d8fc601------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Thu, 19 Nov 2026 01:47:27 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@gokberkozsoy/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[Serverless HuggingFace on AWS]]></title>
            <link>https://medium.com/@gokberkozsoy/serverless-huggingface-on-aws-504ccdfcff5c?source=rss-d25b4d8fc601------2</link>
            <guid isPermaLink="false">https://medium.com/p/504ccdfcff5c</guid>
            <category><![CDATA[hugging-face]]></category>
            <category><![CDATA[mlops]]></category>
            <category><![CDATA[lambda]]></category>
            <category><![CDATA[aws]]></category>
            <category><![CDATA[machine-learning]]></category>
            <dc:creator><![CDATA[Gokberk Ozsoy]]></dc:creator>
            <pubDate>Thu, 15 Aug 2024 20:07:59 GMT</pubDate>
            <atom:updated>2024-08-15T20:07:59.675Z</atom:updated>
            <content:encoded><![CDATA[<h4>Cost-effective ML endpoints via Lambda, API gateway &amp; Docker</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*OSWdqvpd2UXeyM6D" /><figcaption>Photo by <a href="https://unsplash.com/@riccardomion?utm_source=medium&amp;utm_medium=referral">Riccardo Mion</a> on <a href="https://unsplash.com/?utm_source=medium&amp;utm_medium=referral">Unsplash</a></figcaption></figure><h4><strong>Introduction</strong></h4><p>HuggingFace Model Hub is home to thousands of publicy available pretrained transformer models. One can easily find specialized models even for uncommon tasks, which eliminates the hustle of further finetuning the base weights. Experimenting with these models in a colab notebook is fun, but the real challenge is serving them in production as scalable and cost-effective as possible. There is no one-size-fits-all approach here, as the deployment strategy needs to be adjusted for each use-case (model size and inference latency).</p><p>A solution that may help with most of the scenarios, tough, is Serverless HuggingFace on AWS. Backed by Lambda, API gateway and ECR, it does not need any management of underlying infrastructure, can automatically scale-out and -in and allows fully customized ML workflows. Lambda supports custom (Docker) container images up to 10 GB as runtime, and timeout period of 15 mins. This is enough for many lightweight ML tasks where a CPU is sufficient. Here, API gateway routes the incoming traffic to our Lambda function, and ECR stores the container image on AWS. Thus, all you need is wrapping your custom ML code inside a container. Now, let’s implement these from scratch!</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/978/1*n0sxZx6adYB4SCipg-sK9A.jpeg" /><figcaption>Serverless Huggingface pipeline on AWS</figcaption></figure><h4>Tutorial</h4><p>Our use-case is detecting objects in uploaded images using facebook/detr-resnet-50. We will follow these steps:</p><ul><li>Develop your ML logic locally</li><li>Create your Docker container suitable for Lambda</li><li>Upload your Docker container to ECR</li><li>Create Lambda function and API gateway endpoint and test via Postman</li></ul><p>Code is available in <a href="https://github.com/gozsoy/creative-ads">this</a> Github repo.</p><p><strong>Develop your ML logic locally</strong></p><p>Here is my ML logic: For each uploaded image, I want to detect objects in it, filter out the ones with confidence scores smaller than 90%, and report only once per object type. Finally, I want to return a dynamic html template html_string, typing the detected objects and scores with a given coloralong with the uploaded image encoded_image itself.</p><p>I find dealing images as base64 strings easier than files (see API gateway section). Downloading the model to a folder ./hf_cache in the same directory with our code is vital (see Docker container section).</p><pre>import io<br>import base64<br>from PIL import Image<br>import torch<br>from transformers import DetrImageProcessor, DetrForObjectDetection<br><br># dummy valid inputs<br>color = &quot;#ebe999&quot; # hexcode<br>encoded_image = &quot;/9j/4AAQSkZJRgABAQABLAEsAAD/...&quot; # base64 string<br><br># initialize processor and model<br>processor = DetrImageProcessor.from_pretrained(&quot;facebook/detr-resnet-50&quot;,<br>                                               cache_dir=&#39;./hf_cache&#39;)<br>model = DetrForObjectDetection.from_pretrained(&quot;facebook/detr-resnet-50&quot;,<br>                                               cache_dir=&#39;./hf_cache&#39;)<br><br># decode image string to byte<br>image_bytes = base64.b64decode(encoded_image)<br><br># Convert bytes to a BytesIO object<br>init_image = Image.open(io.BytesIO(image_bytes))<br><br># forward pass<br>inputs = processor(images=init_image, return_tensors=&quot;pt&quot;)<br>outputs = model(**inputs)<br><br># let&#39;s only keep detections with score &gt; 0.9<br>target_sizes = torch.tensor([init_image.size[::-1]])<br>results = processor.\<br>    post_process_object_detection(outputs, <br>                                  target_sizes=target_sizes, <br>                                  threshold=0.9)[0]<br># save detected object&#39;s and scores<br>detected_labels = []<br>prompt = &quot;&quot;<br>for score, label in zip(results[&quot;scores&quot;], results[&quot;labels&quot;]):<br>    if model.config.id2label[label.item()] not in detected_labels:<br>        prompt += f&quot;Detected {model.config.id2label[label.item()]}\<br>              with confidence {round(score.item(), 3)}. &lt;br&gt;&quot;<br>        detected_labels.append(model.config.id2label[label.item()])<br><br># just a fancy way to show outputs<br>html_string = f&quot;&quot;&quot;<br>    &lt;html&gt;<br>        &lt;head&gt;<br>            &lt;meta charset=&quot;UTF-8&quot;&gt;<br>            &lt;meta name=&quot;viewport&quot; content=&quot;width=device-width, initial-scale=1.0&quot;&gt;<br>            &lt;style&gt;<br>                .container {{ text-align: center; }}<br>                h1 {{color: {color}; }}<br>            &lt;/style&gt;<br>        &lt;/head&gt;<br>        &lt;body&gt;<br>            &lt;div class=&quot;container&quot;&gt;<br>            &lt;h1&gt;{prompt}&lt;/h1&gt;<br>            &lt;img src=&quot;data:image/jpeg;base64,{encoded_image}&quot; alt=&quot;Base64 Image&quot; /&gt;<br>        &lt;/div&gt;<br>        &lt;/body&gt;<br>    &lt;/html&gt;<br>&quot;&quot;&quot;</pre><p><strong>Create your Docker container suitable for Lambda</strong></p><p>Now that our custom code is ready, we need to convert it to a function within an AWS Lambda compatible Docker container. When deployed, this function will be triggered via API gateway with different inputs.</p><p>handler accepts two inputs: event and context. event holds information about HTTP request, made by client to API gateway. By design,color and encoded_image sent as body parameters in our HTTP request.</p><ul><li><em>Why do we download the model to container?</em> Lambda provides a max storage of 512 MB, much less than the size of our transformer models. Plus, downloading the model during function run incurs unnecessary delays. Instead, we cache our model inside the container that is placed in memory during runtime. Lambda supports max memory of 10 GB, enough for many transformer models.</li><li><em>Why do not we initialize model inside the function? </em>Any code run outside of handler persist across concurrent calls. That means, first function invocation will initialize processor and model with a delay (cold start), subsequent ones only run inside of handler (warm start).</li><li><em>Why do we return json response?</em> This way Lambda informs API gateway that the request was successful, and the response should be treated as html.</li></ul><pre>import io<br>import ast<br>import base64<br>from PIL import Image<br>import torch<br>from transformers import DetrImageProcessor, DetrForObjectDetection<br><br><br>processor = DetrImageProcessor.from_pretrained(&quot;facebook/detr-resnet-50&quot;, <br>                                               cache_dir=&#39;./hf_cache&#39;)<br>model = DetrForObjectDetection.from_pretrained(&quot;facebook/detr-resnet-50&quot;, <br>                                               cache_dir=&#39;./hf_cache&#39;)                                             <br><br>def handler(event, context):<br>    # parse body json<br>    body = ast.literal_eval(event[&#39;body&#39;])<br>    <br>    encoded_image = body[&#39;encoded_image&#39;]<br>    color = body[&#39;color&#39;]<br><br>    # decode image string to byte<br>    image_bytes = base64.b64decode(encoded_image)<br><br>    # Convert bytes to a BytesIO object<br>    init_image = Image.open(io.BytesIO(image_bytes))<br><br>    # run object detection model<br>    inputs = processor(images=init_image, return_tensors=&quot;pt&quot;)<br>    outputs = model(**inputs)<br><br>    # only keep detections with score &gt; 0.9<br>    target_sizes = torch.tensor([init_image.size[::-1]])<br>    results = processor.post_process_object_detection(outputs, target_sizes=target_sizes, threshold=0.9)[0]<br><br>    # report detected objects in html file<br>    detected_labels = []<br>    prompt = &quot;&quot;<br>    for score, label in zip(results[&quot;scores&quot;], results[&quot;labels&quot;]):<br>        if model.config.id2label[label.item()] not in detected_labels:<br>            prompt += f&quot;Detected {model.config.id2label[label.item()]} with confidence {round(score.item(), 3)}. &lt;br&gt;&quot;<br>            detected_labels.append(model.config.id2label[label.item()])<br>    <br>    # just a fancy way to show outputs<br>    html_string = f&quot;&quot;&quot;<br>        &lt;html&gt;<br>            &lt;head&gt;<br>                &lt;meta charset=&quot;UTF-8&quot;&gt;<br>                &lt;meta name=&quot;viewport&quot; content=&quot;width=device-width, initial-scale=1.0&quot;&gt;<br>                &lt;style&gt;<br>                    .container {{ text-align: center; }}<br>                    h1 {{color: {color}; }}<br>                &lt;/style&gt;<br>            &lt;/head&gt;<br>            &lt;body&gt;<br>                &lt;div class=&quot;container&quot;&gt;<br>                &lt;h1&gt;{prompt}&lt;/h1&gt;<br>                &lt;img src=&quot;data:image/jpeg;base64,{encoded_image}&quot; alt=&quot;Base64 Image&quot; /&gt;<br>            &lt;/div&gt;<br>            &lt;/body&gt;<br>        &lt;/html&gt;<br>    &quot;&quot;&quot;<br>    <br>    # api gateway compatible output<br>    return {<br>      &quot;isBase64Encoded&quot;: False,<br>      &quot;statusCode&quot;: 200,<br>      &quot;body&quot;: html_string,<br>      &quot;headers&quot;: {<br>        &quot;content-type&quot;: &quot;text/html&quot;<br>      }<br>    }<br></pre><p>Below is our Dockerfile. FROM uses an Lambda compatible python base image. COPY puts necessary files and model weights into container. CMD executes handler whenever this container starts running. Note that I intentionally install pytorch with only cpu support, as normal install fetches large GPU support files, that inflates our container size.</p><pre>FROM public.ecr.aws/lambda/python:3.12<br><br># Copy requirements.txt<br>COPY requirements.txt ${LAMBDA_TASK_ROOT}<br><br># Install the specified packages<br>RUN pip install torch torchvision --index-url https://download.pytorch.org/whl/cpu<br>RUN pip install -r requirements.txt<br><br># Copy function code<br>COPY lambda_function.py ${LAMBDA_TASK_ROOT}<br># Copy model<br>COPY hf_cache ${LAMBDA_TASK_ROOT}/hf_cache<br><br># Set the CMD to your handler<br>CMD [ &quot;lambda_function.handler&quot; ]</pre><p>Finally, lets build our container image:</p><pre>docker build -t adcreative-objectdet .</pre><p><strong>Upload your Docker container to ECR</strong></p><p>When you create a repo in ECR, it shows you the push commands:</p><pre>aws ecr get-login-password --region eu-west-2 | docker login --username AWS --password-stdin 103335980965.dkr.ecr.eu-west-2.amazonaws.com<br><br>docker tag adcreative-objectdet:latest 103335980965.dkr.ecr.eu-west-2.amazonaws.com/adcreative-objectdet:latest</pre><p>Now that, we logged in, and tagged the image appropriately, I want to check my image size to ensure it is good for Lambda.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*e-0dyll0DeVHs8t-6PZy4w.png" /><figcaption>Screenshot from Docker Desktop. Image size is 1.95 GB.</figcaption></figure><p>As it is, push it to cloud via:</p><pre>docker push 103335980965.dkr.ecr.eu-west-2.amazonaws.com/adcreative-objectdet:latest</pre><p><strong>Create Lambda function and API gateway endpoint and test via Postman</strong></p><p>Next, create a Lambda function using Container Image option. Container Image URI is the id of the image you have just pushed to ECR. After creation is successful, go to Configuration tab, and increase Timeout and Memory to right amount.</p><p>Finally, create a new REST API in API gateway service, with the following configurations:</p><ul><li><em>Create resource </em>Name of your endpoint. I called it /generation</li><li><em>Create method </em>HTTP request type. For us, it is POST</li><li><em>Method request</em> No authorization, Request body Content type application/json</li><li><em>Integration request</em> Integration type: Lambda function (copy&amp;paste your function’s arn), Execution role: assign your API gateway a simple IAM role with AWSLambda_FullAccess permission.</li><li><em>Method response </em>HTTP status code 200, Response body Content type application/json</li></ul><p>Click on Deploy API to make our endpoint alive. Locate Invoke URL, the url we will send our requests to.</p><p>Now, if you refresh your Lambda function’s page, you will see an API gateway trigger attached like:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/996/1*QcwMJDb3d7pkM3gbsDnQ8g.png" /><figcaption>Screenshot from AWS Console. API gateway as Lambda trigger.</figcaption></figure><p>Let’s see our end-to-end architecture in action. In Postman:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*e_pJ1wpkt1BomvTlu2YohQ.png" /><figcaption>Screenshot Postman POST request with json payload and HTML response.</figcaption></figure><p>color is a hex string, encoded_image is a base64 string. I know encoding each image before Postman might be cumbersome, and I implement a client.py code for this in the repo. Note that, first call takes some 10 sec, but subsequent one are immediate as discussed above.</p><p>Wow! We successfully deployed our demo ML app to AWS in a serverless fashion. The same idea can be transferred to many CV and NLP applications effortlessly.</p><h4>Related Products</h4><p>AWS has a diverse set of managed ML services: Comprehend for classic NLP such as NER, sentiment etc., Textract for OCR, Rekognition for object detection, Kendra for embeddings, and others. I like to describe our pipeline as a more general and customizable way of doing these.</p><p>Huggingface Transformers.js allows client side computation, suitable for using in single page applications via React. As the models run on edge devices, only tasks doable by lightweight models are supported (e.g. no text-to-image). Similarly, our system is designed for server side lightweight ML tasks.</p><p>Huggingface Inference API (Serverless) is a free, rate limited API service for a selection of models hosted in Model Hub. It is designed for demo and testing, not production, and it does not allow customization.</p><p>Huggingface Inference Endpoints (Dedicated) is the paid version, where you choose the underlying infrastructure (cpu or gpu), and host cloud provider. It feels like serverless, but you still pay for the EC2 instance hourly, and therefore ideal for scenarios where your service has predictible an steady load.</p><p>Our solution (Serverless Huggingface on AWS) is ideal for traffic with idle periods, occasionally burst loads, or smaller order of magnitudes than 100k number of total calls.</p><p>AWS Sagemaker has real time inference endpoints and serverless inference. Former resembles to Huggingface Inference Endpoints, latter to our solution.</p><p>Finally, let’s make an basic cost comparison:</p><ul><li><em>Our pipeline </em>For 10k requests per day, each with 2.5 sec computation time and 2.5GB container size, total cost = $32.34/month</li><li><em>HF Inference Endpoints </em>AWS CPU intel-spr x2 (2x vCPUs 4GB RAM) with 2 replicas, total cost = $0.064/hr * (730hr * 2 replica) = $93.44/month</li><li><em>Sagemaker Serverless Inference </em>with same conditions with our pipeline, total cost = $60.00/month</li></ul><h4><strong>Bonus: Asynchronous Inference</strong></h4><p>AWS Sagemaker provides asynchronous inference option for long processing times. On the contrary, our pipeline only supports synchronous inference as API gateway waits for a response of max 29 secs. Hence, any Lambda function running beyond this limit is not reachable. We can easily overcome this by decoupling components with SQS. Here is the updated pipeline:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*HAf_aFMiVsFS9Lw_Wl0-hQ.jpeg" /><figcaption>Asynchronous Serverless ML pipeline</figcaption></figure><p>I hope you enjoyed your reading. Please comment your thoughts, and reach me on <a href="http://www.linkedin.com/in/gozsoy">Linkedin</a> for discussing more!</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=504ccdfcff5c" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[LangChain Memory and Agents Simplified]]></title>
            <link>https://medium.com/@gokberkozsoy/langchain-memory-and-agents-simplified-129b065259e5?source=rss-d25b4d8fc601------2</link>
            <guid isPermaLink="false">https://medium.com/p/129b065259e5</guid>
            <category><![CDATA[llm]]></category>
            <category><![CDATA[chatgpt]]></category>
            <category><![CDATA[computer-science]]></category>
            <category><![CDATA[prompt-engineering]]></category>
            <category><![CDATA[langchain]]></category>
            <dc:creator><![CDATA[Gokberk Ozsoy]]></dc:creator>
            <pubDate>Wed, 26 Jun 2024 09:46:13 GMT</pubDate>
            <atom:updated>2024-06-26T09:49:52.954Z</atom:updated>
            <content:encoded><![CDATA[<h4>Pure Python implementations of two prompt engineering methods</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*PwxyRbSH4e-OEH79qcpnPw.png" /><figcaption>source: Stable Diffusion. caption: a comic drawing of a parrot sitting on top of a chain.</figcaption></figure><p><em>This article assumes some familiarity with language modeling and prompt engineering.</em></p><h4>Introduction</h4><p>Prompt engineering is all the efforts we put into manipulating LLMs to fulfill complex tasks such as chatting, reasoning and question-answering. Some of these efforts are:</p><ul><li>few-shot learning (<em>showing how to perform via a few examples)</em></li><li>retrieval augmented generation (<em>fetching necessary info from private files</em>)</li><li>conversation memory (<em>reminding what was talked previously</em>)</li><li>agents -<em>a.k.a. tool-use- </em>(<em>providing external tools as helper</em>)</li></ul><p>LangChain 🦜️🔗 is an exciting framework implementing these in detail. It is a go-to library for developers building LLM applications. I, personally, used it to gain lots of hands-on experience in this topic. However, it abstracts away many concepts from the start, inheriting a steep learning curve (See these <a href="https://www.reddit.com/r/datascience/comments/16ni0h7/is_it_just_me_or_is_langchain_is_too_complicated/">reddit</a> and <a href="https://news.ycombinator.com/item?id=36645575">ycombinator</a> posts).</p><p>In this article, I implement two prompt engineering concepts, namely conversation memories and agents in pure Python. I aim to show how easy it is to code these powerful methods, and customize them for different use-cases. Code available in <a href="https://github.com/gozsoy/langchain-from-scratch">github</a>.</p><h4><strong>Setting up the environment</strong></h4><p>My LLM choice is <em>OpenAI’s gpt-3.5-turbo. </em>I simply put my API key in a .env file. You may go with an open-source model like <em>HuggingFaceH4/zephyr-7b-beta.</em></p><pre>import os<br>import ast<br>from dotenv import load_dotenv<br>import arxiv<br>from openai import OpenAI<br><br># parse .env file and load all variables<br>load_dotenv()<br><br>OPENAI_API_KEY = os.getenv(&#39;OPENAI_API_KEY&#39;)<br><br># initialize open ai client for api calls<br>client = OpenAI()</pre><h4>Conversation Memories</h4><p>LLMs are stateless, i.e. they process each prompt independently. We need to embed past interactions in each new prompt to create a chat-like conversation. There are several ways we store these past interactions:</p><p><strong>1- </strong><a href="https://python.langchain.com/v0.1/docs/modules/memory/types/buffer/"><strong>ConversationBufferMemor</strong></a><strong>y:</strong> A straightforward and suitable memory type for most cases. It simply stores the full conversation history in a variable (<em>history</em>), and puts it in the prompt as <em>Current Conversation </em>for each new <em>input.</em></p><pre>prompt = &quot;&quot;&quot;System: This is a conversation between a software engineer and <br>intellectual AI bot. AI bot is talkative and teaches concepts to the engineer.<br>Current Conversation:<br>{history}<br>Human: {input}<br>AI:&quot;&quot;&quot;<br><br>history = &#39;&#39;<br><br>def talk(question):<br><br>    global history<br>    <br>    # put current values of the variables into prompt string<br>    prompt_after_formatting = prompt.format(input=question,<br>                                            history=history)<br><br>    # make API call<br>    completion = client.chat.completions.create(<br>        model=&quot;gpt-3.5-turbo&quot;,<br>        messages=[<br>            {&quot;role&quot;: &quot;user&quot;, &quot;content&quot;: prompt_after_formatting}<br>        ]<br>    )<br><br>    # processing open ai&#39;s response<br>    output = completion.choices[0].message.content<br><br>    # my way of adding human and ai messages into history<br>    history += f&quot;Human: {question}\nAI: {output}\n&quot;<br><br>    return output</pre><p>Let’s see this in action. Below is a prompt example after asking 3 questions to LLM. You can observe it uses memory effectively to answer the second question.</p><pre>talk(&quot;Do you know anything about OOP?&quot;)<br>talk(&quot;Sorry. What did I ask you?&quot;)<br>talk(&quot;What is SQL?&quot;)<br>print(prompt.format(input=&quot;current question&quot;,history=history))</pre><pre>System: This is a conversation between a software engineer and intellectual AI bot. AI bot is talkative and teaches concepts to the engineer.<br>Current Conversation:<br>Human: Do you know anything about OOP?<br>AI: Yes, Object-Oriented Programming (OOP) is a programming paradigm that revolves around the concept of objects, which can contain data in the form of fields (attributes or properties) and code in the form of procedures (methods). OOP focuses on organizing code into reusable, modular components, allowing for easier maintenance and scalability. Is there a specific aspect of OOP you&#39;d like more information on?<br>Human: Sorry. What did I ask you?<br>AI: You asked me if I know anything about OOP (Object-Oriented Programming).<br>Human: What is SQL?<br>AI: SQL stands for Structured Query Language, and it is used to communicate with and manipulate databases. SQL allows users to perform tasks such as retrieving data, creating, updating, and deleting records, and managing a database&#39;s structure. It is a standard language for interacting with relational databases and is essential for anyone working with data management and storage. Is there a specific aspect of SQL you&#39;d like to know more about?<br><br>Human: current question<br>AI:</pre><p><strong>2- </strong><a href="https://python.langchain.com/v0.1/docs/modules/memory/types/summary/"><strong>ConversationSummaryMemory</strong></a><strong>: </strong>ConservationBufferMemory fills up input token limit too quickly for a long chat history. Even though <em>gpt-3.5-turbo </em>has a huge limit (16k), longer prompts cost higher. For this, we can summarize the full conversation history using a second LLM to extract only the most important points.</p><pre>prompt = &quot;&quot;&quot;System: This is a conversation between a software engineer and <br>intellectual AI bot. AI bot is talkative and teaches concepts to the engineer.<br>Current Conversation Summary:<br>{summary}<br>Human: {input}<br>AI:&quot;&quot;&quot;<br><br>summary_prompt = &quot;&quot;&quot;System: Your task is to summarize below conversation with <br>emphasis on key points. If nothing given, return &#39;&#39;.<br>{history}<br>Summary:&quot;&quot;&quot;<br><br>history = &#39;&#39;<br><br>def talk(question):<br><br>    global history<br><br>    # put history variable into summary prompt string<br>    summary_prompt_after_formatting = summary_prompt.format(history=history)<br><br>    # make API call for summarization<br>    summary_completion = client.chat.completions.create(<br>        model=&quot;gpt-3.5-turbo&quot;,<br>        messages=[<br>            {&quot;role&quot;: &quot;user&quot;, &quot;content&quot;: summary_prompt_after_formatting}<br>        ]<br>    )<br>    <br>    # processing open ai&#39;s response<br>    summary = summary_completion.choices[0].message.content<br><br>    # put current values of the variables into prompt string<br>    prompt_after_formatting = prompt.format(input=question,<br>                                            summary=summary)<br>    <br>    # make API call for answering question<br>    completion = client.chat.completions.create(<br>        model=&quot;gpt-3.5-turbo&quot;,<br>        messages=[<br>            {&quot;role&quot;: &quot;user&quot;, &quot;content&quot;: prompt_after_formatting}<br>        ]<br>    )<br><br>    # processing open ai&#39;s response<br>    output = completion.choices[0].message.content<br>    print(prompt_after_formatting)<br>    # my way of adding human and ai messages into history<br>    history += f&quot;Human: {question}\nAI: {output}\n&quot;<br><br>    return output</pre><p>Let’s see this in action. You can see that summary includes all key points necessary to answer upcoming questions.</p><pre>talk(&quot;How are you today?&quot;)<br>talk(&quot;Sorry. What did I ask you?&quot;)<br>talk(&quot;do you know about football?&quot;)<br>talk(&quot;when did england compete in euros first?&quot;)<br>talk(&quot;what were the other teams in the same competition?&quot;)</pre><pre>input prompt for last question:<br><br>System: This is a conversation between a software engineer and intellectual AI bot. AI bot is talkative and teaches concepts to the engineer.<br>Current Conversation Summary:<br>The conversation includes inquiries about the AI&#39;s knowledge of football and the Euros, specifically when England competed in the Euros for the first time in 1968 and the other teams that participated in the same competition. The AI provides detailed information about the tournament and the teams involved. It also emphasizes that it is ready to assist with any other questions related to football history or the Euros.<br>Human: what were the other teams in the same competition?<br>AI:<br>---<br>response for last question:<br><br>&#39;In the 1968 European Championship, besides England, the other teams that competed were Italy, Yugoslavia, and the Soviet Union. Italy emerged as the champions of the tournament after defeating Yugoslavia in the final. If you have any more questions about the competition or football history, feel free to ask!&#39;</pre><h4><a href="https://python.langchain.com/v0.1/docs/use_cases/tool_use/">Agents and Tool-use</a></h4><p>Agent is just a cool name for an LLM that can call an auxiliary function (tool) with the help of a descriptive prompt. LLMs’ reasoning capabilities and knowledge base are limited for answering any question flawlessly. However, they act superior when supported with necessary tools such as calculator, google search, math functions, weather api, and etc. Given a question, instead of trying to come up with an answer possibly including hallucinations and wrong computations, LLM acts as an orchestrator calling necessary functions step-by-step to reach the correct answer. This way we balance LLM’s reasoning and acting capabilities, and do not exploit its -already limited- reasoning power alone. See <a href="https://react-lm.github.io/">ReAct paper</a> for more on agents.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/950/1*92eVspeFUpwTWXzU1hTNHg.png" /><figcaption>source: <a href="https://react-lm.github.io/">ReAct paper</a></figcaption></figure><p>Let’s start coding the above mechanism. First, I implement the functions my LLM might use for answering my questions. As a proof-of-concept, I only added two functions, though in reality we would have tens of them.</p><pre>tools = {<br>    &#39;convert_time&#39;: &#39;A function to convert a time string with format H:MM:SS to seconds, args: {&quot;time&quot;: {&quot;type&quot;: &quot;string&quot;}}&#39;,<br>    &#39;arxiv_search&#39;: &#39;A function to get information about a scientific article or articles, args: {&quot;query&quot;: {&quot;type&quot;: &quot;string&quot;}, &quot;id&quot;: {&quot;type&quot;: &quot;string&quot;}}&#39;<br>}<br><br>def arxiv_search(args):<br>    # Construct the default API client.<br>    client = arxiv.Client()<br><br>    # Search for the paper with ID<br>    search_by_id = arxiv.Search(query=args[&#39;query&#39;],<br>                                id_list=[args[&#39;id&#39;]])<br>    # Reuse client to fetch the paper, then print its title.<br>    first_result = next(client.results(search_by_id))<br>    <br>    return first_result.title<br><br>def convert_time(args):<br>    temp = args[&#39;time&#39;].split(&#39;:&#39;)<br><br>    return int(temp[0])*3600 + int(temp[1])*60 + int(temp[2])</pre><p>Our prompt introduces available tools in <em>tools, </em>dictates the reasoning format as Thought-Action-Observation and answer format as Final Answer. <em>agent_scratchpad </em>is a variable storing LLM’s all previously generated reasoning steps as a reference to be used in the next step (i.e. internal memory).</p><pre>system_prompt = &quot;&quot;&quot;<br>Answer the following questions as best you can. You have access to the <br>following tools:<br><br>{tools}<br><br>ALWAYS use the following format:<br><br>Question: the input question you must answer<br>Thought: you should always think about what to do. only one action at a time.<br>Action: the action to take, should be one of {tool_names}<br>Action Input: the input to the action<br>Observation: the result of the action<br>... (this Thought/Action/Action Input/Observation can repeat N times)<br>Thought: I now know the final answer<br>Final Answer: the final answer to the original input question<br><br>Begin! Reminder to ALWAYS use the exact characters `Final Answer:` when you <br>provide a definitive answer.<br><br>Question: {input}<br>Thought:{agent_scratchpad}<br>&quot;&quot;&quot;</pre><p>When LLM requests a call of an helper tool, we understand it by observing <em>Observation: </em>string. Hence it becomes our stopping criteria. That implicitly means LLM is waiting for the result of the <em>Action</em> called with <em>Action Inputs</em>. We continue this loop until LLM generates <em>Final Answer: </em>string.</p><pre>agent_scratchpad = &#39;&#39;<br><br>def ask(question):<br><br>    global agent_scratchpad<br><br>    # convert function description into suitable string for prompt<br>    tools_str = &#39;&#39;<br>    for func, desc in tools.items():<br>        tools_str += f&quot;{func} : {desc}\n&quot;<br>    <br>    # put current values of the variables into prompt string<br>    prompt_after_formatting = system_prompt.format(input=question, <br>                                                   agent_scratchpad=agent_scratchpad, <br>                                                   tools=tools_str, <br>                                                   tool_names=list(tools.keys()))<br>    print(prompt_after_formatting)<br><br>    #  make API call for answering question<br>    # stop generation when see Observation:<br>    # as it signs that we need a call a function externally<br>    completion = client.chat.completions.create(<br>        model=&quot;gpt-3.5-turbo&quot;,<br>        messages=[<br>            {&quot;role&quot;: &quot;user&quot;, &quot;content&quot;: prompt_after_formatting}<br>        ],<br>        stop=[&quot;Observation:&quot;]<br>    )<br><br>    # processing open ai&#39;s response<br>    output = completion.choices[0].message.content<br>    print(output)<br>    <br>    # keep reasoning until see Final Answer: string<br>    while output.find(&quot;Final Answer: &quot;)==-1:<br><br>        # append previous reasoning step<br>        agent_scratchpad += output<br><br>        # parse LLM output string into function and its arguments<br>        function_name = output.split(&quot;Action: &quot;)[1].split(&#39;\n&#39;)[0]<br>        function_args = ast.literal_eval(output.split(&quot;Action Input: &quot;)[1].split(&#39;\n&#39;)[0])<br><br>        # call function externally<br>        result = globals()[function_name](function_args)<br><br>        # append function output to logs<br>        agent_scratchpad += f&quot;Observation: {result}\n&quot;<br>        print(f&quot;Observation: {result}\n&quot;)<br>        <br>        # put current values of the variables into prompt string<br>        prompt_after_formatting = system_prompt.format(input=question, <br>                                                    agent_scratchpad=agent_scratchpad, <br>                                                    tools=tools_str, <br>                                                    tool_names=list(tools.keys()))<br>        <br>        # make the next API call, this time with the result of <br>        # previously requested function by LLM <br>        completion = client.chat.completions.create(<br>            model=&quot;gpt-3.5-turbo&quot;,<br>            messages=[<br>                {&quot;role&quot;: &quot;user&quot;, &quot;content&quot;: prompt_after_formatting}<br>            ],<br>            stop=[&quot;Observation:&quot;]<br>        )<br><br>        # processing open ai&#39;s response<br>        output = completion.choices[0].message.content<br>        <br>        print(output)  <br>    <br>    return output</pre><p>Let’s see this in action. Here is a 2-step question asked to our agent. Observe that it does not diverge from the required format, and keeps correct reasoning with the light of its past actions.</p><pre>ask(&quot;how many seconds in 4:00:01. What&#39;s the paper 2311.01606 about?&quot;)</pre><pre>Answer the following questions as best you can. You have access to the following tools:<br><br>convert_time : A function to convert a time string with format H:MM:SS to seconds, args: {&quot;time&quot;: {&quot;type&quot;: &quot;string&quot;}}<br>arxiv_search : A function to get information about a scientific article or articles, args: {&quot;query&quot;: {&quot;type&quot;: &quot;string&quot;}, &quot;id&quot;: {&quot;type&quot;: &quot;string&quot;}}<br><br><br>ALWAYS use the following format:<br><br>Question: the input question you must answer<br>Thought: you should always think about what to do. only one action at a time.<br>Action: the action to take, should be one of [&#39;convert_time&#39;, &#39;arxiv_search&#39;]<br>Action Input: the input to the action<br>Observation: the result of the action<br>... (this Thought/Action/Action Input/Observation can repeat N times)<br>Thought: I now know the final answer<br>Final Answer: the final answer to the original input question<br><br>Begin! Reminder to ALWAYS use the exact characters `Final Answer:` when you provide a definitive answer.<br><br>Question: First answer how many seconds in 4:00:01. Then answer What&#39;s the paper 2311.01606 about?<br>Thought:<br>I need to convert the time 4:00:01 to seconds first before searching for the paper.<br>Action: convert_time<br>Action Input: {&quot;time&quot;: &quot;4:00:01&quot;}<br><br>Observation: 14401<br><br>Action: arxiv_search<br>Action Input: {&quot;query&quot;: &quot;&quot;, &quot;id&quot;: &quot;2311.01606&quot;}<br><br>Observation: KG-FRUS: a Novel Graph-based Dataset of 127 Years of US Diplomatic Relations<br><br>Final Answer: 14401 seconds. The paper 2311.01606 is about &quot;KG-FRUS: a Novel Graph-based Dataset of 127 Years of US Diplomatic Relations&quot;.</pre><h4>Conclusion</h4><p>Prompt engineering is how we play with inputs and outputs of LLMs. It boosts LLMs’ effectiveness, providing room for ever intelligent products. Yet, it is not hard to implement various prompt engineering methods once understood the fundamentals.</p><p>I hope you enjoyed your reading. You can reach me on <a href="http://www.linkedin.com/in/gozsoy">Linkedin</a> for discussing more. Next time, I will be posting for proving how easy RAG is!</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=129b065259e5" width="1" height="1" alt="">]]></content:encoded>
        </item>
    </channel>
</rss>