<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Marco Bertelli on Medium]]></title>
        <description><![CDATA[Stories by Marco Bertelli on Medium]]></description>
        <link>https://medium.com/@marco.bertelli?source=rss-75f167fb6874------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/1*9rlc-i8umqRTVqC1lZLngQ.jpeg</url>
            <title>Stories by Marco Bertelli on Medium</title>
            <link>https://medium.com/@marco.bertelli?source=rss-75f167fb6874------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Mon, 16 Nov 2026 15:00:51 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@marco.bertelli/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[The Questions You Should Ask Before Letting AI Build Your Next Feature]]></title>
            <link>https://levelup.gitconnected.com/the-questions-you-should-ask-before-letting-ai-build-your-next-feature-93661f8c3a50?source=rss-75f167fb6874------2</link>
            <guid isPermaLink="false">https://medium.com/p/93661f8c3a50</guid>
            <category><![CDATA[software-development]]></category>
            <category><![CDATA[claude-code]]></category>
            <category><![CDATA[ai]]></category>
            <category><![CDATA[programming]]></category>
            <category><![CDATA[artificial-intelligence]]></category>
            <dc:creator><![CDATA[Marco Bertelli]]></dc:creator>
            <pubDate>Sun, 15 Nov 2026 18:26:45 GMT</pubDate>
            <atom:updated>2026-11-15T18:26:45.307Z</atom:updated>
            <content:encoded><![CDATA[<p><em>I built Filo, a Claude Code plugin that helps you work through a feature’s flow, one question at a time, before you implement it.</em></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/860/1*vAILz985gbPK4u_atLJIPg.gif" /><figcaption>Filo Preview</figcaption></figure><blockquote>Add the ability to share a booking.</blockquote><p>That sounds like a reasonably clear request. There’s an object, an action, and an expected outcome. A coding agent can get to work: a button, an endpoint, permissions, a screen update.</p><p>Then a question comes up.</p><p>The booking is shared with a colleague. Overnight, it expires. Who tells the colleague?</p><p>And another: if the meeting falls through, can that colleague cancel the booking? Or are they only allowed to view it?</p><p>The code can work perfectly while these decisions remain unresolved. But someone will make them. Perhaps the agent, implicitly. Perhaps you, during a review. Perhaps a user will expose the gap by opening a support ticket after turning up at a room they thought was booked.</p><p><strong>I built Filo to bring those questions into the development session, before they become code.</strong></p><p>When we describe a feature, we tend to describe the path we have in mind. The user arrives, takes an action, and gets a result.</p><p>The product also needs to behave sensibly when someone doesn’t respond. When a permission changes. When a background job runs while nobody is watching. When two people see the same object but are allowed to do different things.</p><p>Those details are easy to leave implicit. Each one leaves room for interpretation.</p><p>The idea behind Filo is simple: make those interpretations visible, work through them, and keep the resulting decisions with the project.</p><p>Filo is a Claude Code plugin that starts with your repository. It reads roles, screens, states, errors, background jobs, and decisions you’ve already documented. It uses that context to guide you through an interview about how the feature should behave.</p><p>One question at a time.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/584/1*Y1V9pO9tZITjVJH7cZIl8w.png" /><figcaption>Example Workflow made with Filo</figcaption></figure><p>That detail matters. A twenty-point questionnaire is easy to rush through. A concrete situation gives you a chance to stop and imagine what someone using the product will actually experience.</p><p>Take the booking example. A question inspired by the example in the repository might look like this:</p><p>&gt; Anna shares a booking with Marco. Nobody confirms it, and it expires overnight. What should Marco see the next day?</p><p>&gt; A. A notification telling him it expired.</p><p>&gt; B. The updated status in his booking list.</p><p>&gt; C. The booking disappears from his list.</p><p>Each answer changes the experience. A notification adds another behavior to build. Updating the status assumes Marco opens the app. Removing the booking makes the list simpler, but may leave him wondering where it went.</p><p>Filo offers options, recommends one, and explains the consequences. You can pick an option, answer in your own words, or delegate the decision.</p><p>The most interesting moment comes when two answers don’t fit together.</p><p>Earlier, you decided that a colleague could only view a booking. A few questions later, you ask for that colleague to be able to cancel it when the meeting falls through.</p><p>Those instructions conflict.</p><p>Filo is designed to flag that conflict and ask which decision should stand. It checks against earlier answers, recorded decisions, and the behavior it can reconstruct from the code.</p><p>That’s where I want it to earn its place: making a contradiction visible while resolving it still means changing a decision.</p><p>Of course, the quality of that analysis also depends on the model running it. Filo can help you spot a problem; its conclusions still need your judgment.</p><p>At the end of the interview, the flow becomes a page you can review.</p><p>You can read the explanation, follow a map of the handoffs between people, inspect the main object’s lifecycle, and look through a storyboard of the screens that change. The page also brings together the decisions and the cases identified during the session.</p><p>It makes clear which choices you made and which ones you delegated to the agent.</p><p>You can say “you choose” when a detail doesn’t deserve more of your time. That choice stays visible, so you can revisit it before approving the result.</p><p>Once you approve the flow, Filo saves it as a Markdown file in your project, normally at product/flows/&lt;name&gt;.md, along with acceptance scenarios. It then proposes a short implementation plan.</p><p><strong>The conversation becomes a set of decisions your next coding session can pick up.</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*jegf7Aqy-E7_BGb2HUkGEA.png" /><figcaption>Another example of output articat with Filo</figcaption></figure><p>To try it, you’ll need Claude Code with plugin support. Install Filo with:</p><pre>/plugin marketplace add marco-bertelli/filo<br>/plugin install filo@filo</pre><p>Start a new session in your product’s repository and run:</p><pre>/filo:flow your description here</pre><p>Then describe the feature you want to work through. The session can run in your language.</p><p>My suggestion is to choose something involving at least two people or multiple screens: an invitation, an approval, a sharing flow, a handoff. An existing feature that keeps generating questions is also a good place to start.</p><p>Try a prompt like this:</p><p>&gt; I want to review our team invitation flow. Before we implement any changes, help me decide what happens for the person sending the invitation and the person receiving it, including expiration, revocation, and invitations that have already been accepted.</p><p>As you try it, watch for one thing: <strong>which question makes you decide something your original request left open?</strong></p><p>That gives you a concrete way to judge whether it helped.</p><p>For a refactor or a small, isolated fix, a flow interview may be unnecessary. Filo is intended for features where actions, people, and states need to stay consistent with one another.</p><p>The project is in beta. The file format and review page may still change. An automated check comparing the implemented feature against the approved flow is not available yet. For now, the saved scenarios are the reference to use during implementation and verification.</p><p>If you try it, I’d especially like to hear where the interview helps and where it loses the thread. A vague question, a missed contradiction, or an option that makes no sense is useful feedback.</p><p>The repository is here: <a href="https://github.com/marco-bertelli/filo">https://github.com/marco-bertelli/filo</a>, including a demo and a guide to your first session: <a href="https://github.com/marco-bertelli/filo/blob/main/docs/getting-started.md">https://github.com/marco-bertelli/filo/blob/main/docs/getting-started.md</a>.</p><p>Pick a feature you’re about to build. Before asking your agent to implement it, take it through Filo.</p><p>You might discover that the missing piece was a decision.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=93661f8c3a50" width="1" height="1" alt=""><hr><p><a href="https://levelup.gitconnected.com/the-questions-you-should-ask-before-letting-ai-build-your-next-feature-93661f8c3a50">The Questions You Should Ask Before Letting AI Build Your Next Feature</a> was originally published in <a href="https://levelup.gitconnected.com/">Level Up Coding</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[The Architecture of a Multi-Agent System That Can Actually Take Action]]></title>
            <link>https://levelup.gitconnected.com/the-architecture-of-a-multi-agent-system-that-can-actually-take-action-2fef26dd80d8?source=rss-75f167fb6874------2</link>
            <guid isPermaLink="false">https://medium.com/p/2fef26dd80d8</guid>
            <category><![CDATA[ai-agent]]></category>
            <category><![CDATA[ai]]></category>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[artificial-intelligence]]></category>
            <category><![CDATA[software-engineering]]></category>
            <dc:creator><![CDATA[Marco Bertelli]]></dc:creator>
            <pubDate>Wed, 16 Sep 2026 16:23:21 GMT</pubDate>
            <atom:updated>2026-09-16T16:23:21.389Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*SOAXrZGzqfwPY5bx1PVsNw.png" /><figcaption>How complex an agent system can be</figcaption></figure><p><em>Orchestrators and specialist agents are only the beginning. Here’s how tools, state, approvals, retries, and verification turn an AI workflow into a system you can trust.</em></p><p>The most revealing moment in an AI demo often comes after the impressive answer. Someone types:</p><blockquote>“Great. Go ahead.”</blockquote><p>Three words.</p><p>Until that moment, the system only had to understand a request and produce a useful response.</p><p>Now it has to <strong>do something</strong>.</p><p>It needs to remember exactly what it proposed. Identify the right record. Decide which specialist should handle the task. Check whether the action is still valid. Verify that the user is allowed to approve it. Call another system. Handle failures.</p><p>And then it needs to answer a surprisingly difficult question:</p><p><strong>Did the action actually happen?</strong></p><p>Most multi-agent demos focus on the part before this.</p><p>An orchestrator receives a request.</p><p>It delegates work to specialist agents.</p><p>Those agents call tools.</p><p>A result comes back.</p><p>That architecture is useful.</p><p>But for systems that take real-world actions, orchestration is only the beginning.</p><p>The moment a user says <em>“go ahead,”</em> the system suddenly needs explicit state, authorization, durable workflows, revalidation, idempotency, recovery, and evidence of completion.</p><p>That is where a multi-agent workflow becomes an architectural problem.</p><h3>One Request, Several Responsibilities</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/780/1*2E2mD7qCim-0AyYAAcNRmw.png" /><figcaption>Described flow</figcaption></figure><p>Consider a request you might give a customer support assistant:</p><blockquote><em>“Check whether order #4821 qualifies for a refund. If it does, prepare the refund for my approval.”</em></blockquote><p>A person can immediately picture the work:</p><ol><li>Understand what the customer wants.</li><li>Retrieve the order.</li><li>Find the relevant refund policy.</li><li>Compare the order with the policy.</li><li>Resolve anything that is still unknown.</li><li>Calculate the refund.</li><li>Prepare a proposal.</li><li>Ask for approval.</li><li>Execute the refund.</li><li>Confirm that it actually happened.</li></ol><p>For an AI system, every one of those transitions needs somewhere to happen.</p><p>A useful architecture might look like this:</p><p><strong>User → Orchestrator → Specialist Agent → Evidence → Proposal → Approval → Execution → Verification</strong></p><p>The orchestrator decides where the request belongs.</p><p>A specialist agent performs the domain-specific investigation.</p><p>Tools connect the system to documentation and current records.</p><p>Explicit state preserves the action being proposed.</p><p>The user approves a specific proposal.</p><p>Application code executes the action.</p><p>And verification establishes whether the intended outcome actually occurred.</p><p>There is one principle behind the entire design:</p><blockquote><strong><em>The model can decide what should happen. The application must decide what is allowed to happen.</em></strong></blockquote><p>Let’s follow the refund request through that architecture.</p><p><em>The code below consists of simplified architectural sketches. Model setup, storage, and service integrations are omitted; names such as </em><em>llm and </em><em>policy_retriever represent application components.</em></p><h3>Start With an Orchestrator, Not an Agent That Does Everything</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/748/1*qs0otuHChuPenmloh8v0pA.png" /><figcaption>How work an Orchestrator, like claude code</figcaption></figure><p>The simplest way to build an agent is to give it every tool available.</p><p>Search documentation.</p><p>Look up orders.</p><p>Check payments.</p><p>Issue refunds.</p><p>Send emails.</p><p>Generate reports.</p><p>For a small application, that can be completely reasonable.</p><p>But as responsibilities grow, the agent’s job becomes harder to describe.</p><p>A request about delivery times and a request to reverse a payment begin to share the same increasingly crowded workspace.</p><p>A useful boundary is to separate:</p><p><strong>understanding where a request belongs</strong></p><p>from:</p><p><strong>performing specialized work</strong></p><p>For our support assistant, we can introduce an orchestrator that handles general questions and routes refund-related requests to a specialist.</p><pre>from llama_index.core.agent.workflow import FunctionAgent</pre><pre>refund_specialist = FunctionAgent(<br>    name=&quot;refund_specialist&quot;,<br>    description=&quot;Checks refund eligibility and prepares proposals.&quot;,<br>    system_prompt=(<br>        &quot;Check the order and the applicable policy. &quot;<br>        &quot;Explain your evidence before proposing a refund. &quot;<br>        &quot;Hand back unrelated requests to the coordinator.&quot;<br>    ),<br>    tools=[get_order, search_policy, prepare_refund],<br>    llm=llm,<br>    can_handoff_to=[&quot;coordinator&quot;],<br>)</pre><p>The description is not cosmetic.</p><p>It is part of the architecture.</p><p>“Helpful assistant” tells us almost nothing.</p><p>“Checks refund eligibility and prepares proposals” gives the agent a responsibility that we can inspect, test, and reason about.</p><p>LlamaIndex’s FunctionAgent and AgentWorkflow support this kind of explicit handoff between agents.</p><p>But more agents do not automatically produce a better system.</p><p>Every specialist should earn its place.</p><p>For this example, one coordinator and one refund specialist are enough to create a clear boundary without introducing unnecessary coordination overhead.</p><h3>Give the Orchestrator a Clear Route</h3><p>Now we can connect the coordinator and the specialist.</p><pre>from llama_index.core.agent.workflow import AgentWorkflow</pre><pre>coordinator = FunctionAgent(<br>    name=&quot;coordinator&quot;,<br>    description=&quot;Answers general questions and routes refund requests.&quot;,<br>    system_prompt=(<br>        &quot;Use documentation for general questions. &quot;<br>        &quot;Hand refund requests to the refund specialist.&quot;<br>    ),<br>    tools=[search_policy],<br>    llm=llm,<br>    can_handoff_to=[&quot;refund_specialist&quot;],<br>)</pre><pre>workflow = AgentWorkflow(<br>    agents=[coordinator, refund_specialist],<br>    root_agent=&quot;coordinator&quot;,<br>)</pre><p>Our request now has a route.</p><p>The coordinator recognizes that the user is asking about a refund.</p><p>The refund specialist retrieves the order.</p><p>It searches the relevant policy.</p><p>It identifies missing information.</p><p>And once it has enough evidence, it prepares a proposal.</p><p>The exact conversational path can vary.</p><p>One customer might provide the delivery date immediately.</p><p>Another might start by asking about the refund policy and mention the order three messages later.</p><p>The agents can interpret those variations.</p><p>The application still defines which capabilities exist and the boundaries around their use.</p><p>That distinction matters:</p><blockquote><strong><em>Conversation can remain flexible while consequential operations remain controlled.</em></strong></blockquote><h3>Specialist Agents Still Need Evidence</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/780/1*1-1RbKmdskVLXv3CvO7J3g.png" /><figcaption>Evidence or sources visual image</figcaption></figure><p>Routing a request to the correct agent does not make its decision correct.</p><p>The specialist still needs evidence.</p><p>Imagine the refund policy says:</p><blockquote><em>Unopened products can be returned within 30 days.</em></blockquote><p>And the order system says:</p><blockquote><em>Order #4821 was delivered 12 days ago.</em></blockquote><p>Those facts come from different systems.</p><p>Both matter.</p><p>The policy describes the rule.</p><p>The order describes the situation.</p><p>Eligibility depends on bringing the two together.</p><p>This is where retrieval becomes part of the workflow.</p><p>The specialist can search the relevant policy while separately retrieving the current order.</p><p>A deliberately small policy-search tool might look like this:</p><pre>async def search_policy(question: str) -&gt; list[dict]:<br>    &quot;&quot;&quot;Find policy passages relevant to a customer request.&quot;&quot;&quot;</pre><pre>    matches = await policy_retriever.aretrieve(question)</pre><pre>    return [<br>        {<br>            &quot;text&quot;: match.node.get_content(),<br>            &quot;source&quot;: match.node.metadata.get(&quot;title&quot;),<br>            &quot;section&quot;: match.node.metadata.get(&quot;section&quot;),<br>        }<br>        for match in matches<br>    ]</pre><p>Notice what travels with the text:</p><p><strong>its source.</strong></p><p>If the assistant says the order is inside the refund window, the eventual explanation should be able to point to the policy that established that window.</p><p>But there is an important distinction:</p><blockquote><strong><em>A source does not make an interpretation correct. It makes the interpretation inspectable.</em></strong></blockquote><p>Suppose the policy also requires the item to be unopened.</p><p>The order system does not contain that information.</p><p>The right answer is not to guess.</p><p>The right next step is to ask.</p><p>A useful architecture needs to represent missing information instead of quietly turning uncertainty into confidence.</p><h3>Orchestration Is Not State</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/748/1*1auZdSN3Rs_Yp6tKX6-Pjw.png" /><figcaption>Conversation history vs application state</figcaption></figure><p>Now imagine the investigation is complete.</p><p>The specialist says:</p><blockquote><em>“Order #4821 qualifies for a €79 refund. Would you like me to proceed?”</em></blockquote><p>The user replies:</p><blockquote><em>“Yes, go ahead.”</em></blockquote><p>The orchestrator successfully routed the request.</p><p>The specialist successfully investigated it.</p><p>The tools successfully retrieved the evidence.</p><p>And yet the system still has a problem.</p><p>What exactly did the user approve?</p><p>Which order?</p><p>Which amount?</p><p>Which currency?</p><p>Which version of the proposal?</p><p>What if two orders were being discussed?</p><p>What if the refund amount changed after the proposal was generated?</p><p>This is why consequential actions should not exist only inside the chat transcript.</p><p>The application needs explicit state.</p><p>For example:</p><pre>pending_action = {<br>    &quot;proposal_id&quot;: &quot;refund-proposal-17&quot;,<br>    &quot;order_id&quot;: &quot;4821&quot;,<br>    &quot;amount_minor&quot;: 7900,<br>    &quot;currency&quot;: &quot;EUR&quot;,<br>    &quot;status&quot;: &quot;awaiting_approval&quot;,<br>    &quot;evidence&quot;: [<br>        &quot;returns-policy:section-3&quot;,<br>        &quot;order:4821&quot;,<br>    ],<br>}</pre><p>The important field is not only status.</p><p>It is proposal_id.</p><p>Approval should refer to the exact proposal the user reviewed.</p><p>If the amount changes, the proposal changed.</p><p>If the order changes, the proposal changed.</p><p>If new evidence changes the recommendation, the proposal changed.</p><p>This gives us one of the most useful distinctions in agent architecture:</p><blockquote><strong><em>Conversation history is for understanding. Application state is for acting.</em></strong></blockquote><p>The transcript can help the model understand what “go ahead” means conversationally.</p><p>Structured state gives the application something precise to execute.</p><h3>Approval Is a Data Structure, Not a Sentence</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/780/1*lANNGxOhYLiZz-0wcwnRMA.png" /><figcaption>Approval as structured state</figcaption></figure><p>It is tempting to represent approval as:</p><pre>user said yes</pre><p>But a safer representation is closer to:</p><pre>user X approved proposal Y,<br>containing action Z,<br>under the current conditions.</pre><p>That difference becomes critical as soon as an agent can affect the outside world.</p><p>Approval should bind the user to a specific proposed action.</p><p>Not to whatever action the model happens to infer later.</p><p>This is why the proposal should contain the operational details that matter:</p><ul><li>target</li><li>amount</li><li>currency</li><li>evidence</li><li>status</li><li>proposal identifier</li></ul><p>The natural-language conversation explains the action.</p><p>The structured proposal defines it.</p><h3>Make Waiting Part of the Workflow</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/748/1*nGTP1Fx4POvuSpvpgl8mTA.png" /><figcaption>Approval flow</figcaption></figure><p>Human approval introduces another complication:</p><p>time.</p><p>The user might approve immediately.</p><p>They might reply after lunch.</p><p>They might close the browser and return tomorrow.</p><p>If a workflow depends on human approval, waiting is not an exception.</p><p>It is part of the workflow.</p><p>The system therefore needs to preserve pending work and resume it later.</p><p>LlamaIndex provides workflow context and human-input mechanisms that help structure this interaction.</p><p>But durable storage, authorization, expiration rules, and the surrounding application lifecycle still need to be designed.</p><p>For a consequential action such as a refund, I would let the agent prepare the proposal and let application code execute the approved operation.</p><p>For example:</p><pre>async def approve_refund(proposal_id, user):<br>    proposal = await proposals.load(proposal_id)</pre><pre>    await permissions.require(<br>        user,<br>        &quot;approve_refund&quot;,<br>        proposal,<br>    )</pre><pre>    await proposals.require_pending(proposal)</pre><pre>    await refunds.revalidate(proposal)</pre><pre>    receipt = await payments.refund(<br>        order_id=proposal.order_id,<br>        amount_minor=proposal.amount_minor,<br>        idempotency_key=proposal.id,<br>    )</pre><pre>    await proposals.mark_completed(<br>        proposal.id,<br>        receipt,<br>    )</pre><pre>    return receipt</pre><p>There are several important boundaries hidden inside this function.</p><p>The proposal is loaded from stored state.</p><p>Authorization is checked outside the model.</p><p>The application confirms that the proposal is still pending.</p><p>The refund is revalidated.</p><p>The external operation receives a stable idempotency key.</p><p>And the result is stored.</p><p>The chain now looks like this:</p><p><strong>stored proposal → authorized approval → revalidation → execution → recorded result</strong></p><p>That chain matters more than whether the agent generated the perfect sentence.</p><h3>Revalidate Before You Act</h3><p>Why validate again after the user already approved the refund?</p><p>Because the world may have changed while the system was waiting.</p><p>Perhaps the order was already refunded through another support channel.</p><p>Perhaps the payment status changed.</p><p>Perhaps somebody modified the order.</p><p>Perhaps another workflow already performed the operation.</p><p>The proposal represents what was believed to be valid when it was created.</p><p>Execution happens later.</p><p>Those are different moments.</p><p>Before performing a consequential operation, the application should verify that the assumptions behind the proposal still hold.</p><p>A useful rule is:</p><blockquote><strong><em>Approval authorizes an action. It does not freeze reality.</em></strong></blockquote><p>The longer a workflow can remain pending, the more important revalidation becomes.</p><h3>Retries Must Not Repeat the Action</h3><p>There is another failure mode.</p><p>The user approves the refund.</p><p>The application sends the request.</p><p>The payment provider processes it.</p><p>But the network connection drops before the application receives the response.</p><p>The client retries.</p><p>Should the system issue another €79 refund?</p><p>Obviously not.</p><p>This is where idempotency matters.</p><p>A stable operation identifier allows an external service that supports idempotency to recognize that two requests represent the same logical operation.</p><p>Instead of interpreting the retry as:</p><blockquote><em>issue another refund</em></blockquote><p>the service can interpret it as:</p><blockquote><em>continue or return the result of refund proposal #17</em></blockquote><p>This rarely appears in flashy multi-agent demos.</p><p>But once an AI system can move money, send messages, modify records, create tickets, or trigger other real-world effects, retries become part of the architecture.</p><h3>“I Don’t Know Whether It Worked” Is a Real State</h3><p>Now consider the most uncomfortable moment in the workflow.</p><p>The payment service times out.</p><p>Did the refund fail?</p><p>Or did the service process it before the connection disappeared?</p><p>From the user’s perspective, the difference is €79.</p><p>The wrong response would be:</p><blockquote><em>“The refund failed. Trying again.”</em></blockquote><p>The system does not know that.</p><p>It only knows that it did not receive confirmation.</p><p>Those are different things.</p><p>A reliable workflow should preserve that uncertainty.</p><p>Instead of immediately repeating the operation, it can check the status of the existing transaction.</p><p>Until the result is known, the assistant can say something accurate:</p><blockquote><em>“I couldn’t confirm whether the refund completed, so I’m checking its status.”</em></blockquote><p>The same principle applies earlier in the workflow.</p><p>A failed order lookup does not mean the order does not exist.</p><p>It means the order status is unknown.</p><p>A failed policy search does not mean there is no relevant policy.</p><p>It means the policy question is unresolved.</p><p>A timeout does not necessarily mean an operation failed.</p><p>It means completion is unconfirmed.</p><p>Reliable systems preserve those distinctions.</p><h3>A Tool Call Is Not the Definition of Done</h3><p>This is the point where the architecture extends beyond orchestration.</p><p>An agent calls a tool.</p><p>The tool returns.</p><p>Is the job finished?</p><p>Not necessarily.</p><p>If the payment API returns an error, it is not finished.</p><p>If the request times out, it is not finished.</p><p>If the application cannot determine whether the refund happened, it is definitely not finished.</p><p>A stronger definition of completion is:</p><blockquote><strong><em>The system has enough evidence to state that the intended outcome occurred.</em></strong></blockquote><p>That changes the contract.</p><p>The goal is not:</p><pre>call refund()</pre><p>The goal is:</p><pre>establish that refund proposal #17<br>for order #4821<br>was successfully processed exactly once</pre><p>Those are very different guarantees.</p><p>And that is why orchestrators and specialist agents are only the beginning.</p><h3>Before You Let a Multi-Agent System Act, Ask These 9 Questions</h3><p>Before connecting an AI workflow to real-world actions, I would check these nine things.</p><p><strong>1. Does every agent have a bounded responsibility?</strong></p><p>Can you explain what the orchestrator and each specialist are responsible for in one sentence?</p><p><strong>2. Can you identify the evidence behind important decisions?</strong></p><p>Can the system show which documents and records informed the proposal?</p><p><strong>3. Can missing information remain explicitly unknown?</strong></p><p>Or will uncertainty quietly become a confident recommendation?</p><p><strong>4. Is the proposed action represented as structured state?</strong></p><p>The operational details should not exist only in the conversation.</p><p><strong>5. Is the user approving one specific proposal?</strong></p><p>“Yes” should authorize something precise.</p><p><strong>6. Are permissions enforced outside the model?</strong></p><p>The model can recommend an operation. Application code should determine whether it is allowed.</p><p><strong>7. Is the proposal revalidated immediately before execution?</strong></p><p>Reality may have changed since the proposal was created.</p><p><strong>8. Can execution safely survive retries?</strong></p><p>For consequential operations, idempotency is part of correctness.</p><p><strong>9. Does the system verify the outcome before saying the job is complete?</strong></p><p>Calling a tool is not evidence that the intended result occurred.</p><p>If you cannot answer these questions clearly, you may still have an impressive multi-agent demo.</p><p>But you probably do not yet have a system you should trust to act.</p><h3>The Architecture Behind “Done”</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*D-FdD-GShNuliacEn-Rtkw.png" /><figcaption>Complete recap</figcaption></figure><p>Return to the original request:</p><blockquote><em>“Check whether order #4821 qualifies for a refund. If it does, prepare the refund for my approval.”</em></blockquote><p>Now every part of the request has somewhere to go.</p><p>The orchestrator routes the request.</p><p>The specialist gathers domain-specific evidence.</p><p>Tools connect the system to documentation and current records.</p><p>Explicit state preserves the proposal.</p><p>The user approves one specific action.</p><p>Application code verifies permissions and revalidates the proposal.</p><p>The external system performs the operation.</p><p>The application records and verifies the result.</p><p>Only then can the final answer be wonderfully boring:</p><blockquote><em>“Your €79 refund for order #4821 has been processed. Here is the confirmation.”</em></blockquote><p>That sentence is useful because of everything behind it.</p><p>As multi-agent systems grow, I would keep returning to the same questions:</p><p>Does every capability have a clear owner?</p><p>Can we inspect the evidence behind a decision?</p><p>Can the workflow preserve uncertainty?</p><p>Can it recover when a dependency fails?</p><p>Can a person understand exactly what they are approving?</p><p>Can the system prove that the intended outcome occurred?</p><p>Those questions help us decide where another specialist agent adds value, where an ordinary function is enough, and where the workflow needs a stronger boundary.</p><p>The most satisfying multi-agent experience may end with a very ordinary sentence:</p><p><strong>“Done.”</strong></p><p>The architecture is what lets you trust it.</p><p><em>I’m considering turning this architecture into a complete LlamaIndex implementation: orchestrator, specialist agents, persistent state, approval events, idempotent execution, failure recovery, and verification included.</em></p><p><em>If that would be useful, leave a comment with the part you’d most like to see implemented next.</em></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=2fef26dd80d8" width="1" height="1" alt=""><hr><p><a href="https://levelup.gitconnected.com/the-architecture-of-a-multi-agent-system-that-can-actually-take-action-2fef26dd80d8">The Architecture of a Multi-Agent System That Can Actually Take Action</a> was originally published in <a href="https://levelup.gitconnected.com/">Level Up Coding</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Why Does Hello World Cost $134/Month on AWS?]]></title>
            <link>https://levelup.gitconnected.com/why-does-hello-world-cost-134-month-on-aws-50900884ee70?source=rss-75f167fb6874------2</link>
            <guid isPermaLink="false">https://medium.com/p/50900884ee70</guid>
            <category><![CDATA[software-engineering]]></category>
            <category><![CDATA[devops]]></category>
            <category><![CDATA[money]]></category>
            <category><![CDATA[aws]]></category>
            <category><![CDATA[software-development]]></category>
            <dc:creator><![CDATA[Marco Bertelli]]></dc:creator>
            <pubDate>Wed, 09 Sep 2026 17:50:29 GMT</pubDate>
            <atom:updated>2026-09-11T14:28:33.965Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/860/1*YPAMNRgrVt49zcQNupXPTg.png" /><figcaption>AWS can be expensive</figcaption></figure><p><em>A cost breakdown of private containers, redundant infrastructure, and controlled releases. And when they’re actually worth paying for.</em></p><p><strong><em>Author’s note:</em></strong><em> I used AI to help translate this article into English. The architecture, opinions, and examples come from my own experience. Prices are estimates, not charges from a live deployment.</em></p><p>The application returns `{“message”:”Hello World”}`. The monthly estimate for the AWS platform around it is <strong>$133.85</strong>. About $21 pays for the two containers. Almost $49 pays for the private connections they use to pull their images and send logs.</p><p><strong>Before we build anything, should we build this?</strong></p><p>There is a tempting way to start a new project: design the infrastructure for the company you hope it will become. Add redundancy, private connectivity, several environments, a database, a sophisticated delivery pipeline and enough monitoring to watch all of it. Each choice has a reasonable explanation. Put them together and you can end up operating a small platform before you know whether anyone needs the product.</p><p>That is the question behind this article. <strong>How much architecture does the project need today, and what are we paying to prepare for a future that is still uncertain?</strong></p><p>Consider a prototype with a handful of testers. The team is still changing the API, discovering what users want and deciding whether to continue. A brief interruption during a planned update might be acceptable. Spending several days designing a deployment that can tolerate failures the project has never encountered may deliver less value than improving the feature those testers are struggling to use.</p><p>Now consider the same small API handling work that a customer depends on every morning. Its request volume might still be low, but the consequences of an interruption are different. Some of the infrastructure that looked excessive for the prototype may now be justified. Traffic alone cannot tell us how much reliability to buy; the impact of failure matters too.</p><p>An architecture can be technically sound and still be a poor economic fit. The problem is often timing. You pay for the additional resources and take responsibility for them immediately, while the benefit may depend on growth, contractual commitments or operating requirements that have not arrived.</p><p>Our $133.85 estimate makes that visible. Under the same monthly assumptions, it represents about $1,606 over a year, before the excluded charges and anyone’s engineering time. That may be an easy expense to justify for an established service. For a small experiment with no revenue, it deserves a conversation. And this example is still modest: there is no required database, container host fleet or multi-Region deployment in that number.</p><p>Overengineering can make the mismatch much larger. A database added for a feature that does not exist yet, extra services kept running in every environment and recovery infrastructure built around an unagreed availability target can turn a simple API into a collection of recurring charges. Each component looks affordable in isolation. The total can become disproportionate to the work the application actually performs, especially when several copies sit idle most of the time.</p><p>The invoice only captures part of the cost. Every component also needs configuration, permissions, updates and someone who understands its failure modes. A new engineer must learn those relationships. A failed deployment requires tracing them. An unused database still needs an owner. An alarm still needs a decision about who responds and what they should do.</p><p>That complexity can work against the reliability it was intended to improve. A team may struggle to diagnose a system with too many dependencies, or keep broad permissions because nobody is confident enough to narrow them. Adding more controls creates more configuration to get right. The operating burden belongs in the design review alongside the monthly estimate.</p><p>Starting with a smaller design still requires care. Credentials, access to private data and recovery of anything you cannot afford to lose need attention from the beginning. The choices you can defer depend on the application: a disposable prototype and a service holding irreplaceable customer records have very different obligations. Simplicity should come from removing unnecessary responsibilities and making accepted limitations explicit.</p><p>A useful way to defer a component is to name the event that would justify adding it. Perhaps a customer commitment makes deployment interruptions unacceptable. Perhaps measured load calls for another instance. Perhaps the application gains persistent data and now needs a tested restore procedure. A concrete trigger gives the team a reason to revisit the design, rather than either building everything on day one or postponing it indefinitely.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/674/1*lZPtS7JS1AiET2WRcFD2Jw.jpeg" /><figcaption>Explained with a meme</figcaption></figure><p><strong>The purpose of this article is to make those choices easier to judge.</strong> We will examine a fairly substantial AWS setup around the smallest application possible, so its infrastructure costs and responsibilities are easy to see. For every major expense, we will ask what it provides, what work it leaves with the team and when that tradeoff might be worthwhile.</p><p>The platform uses two private containers in a two-Availability-Zone design, a public HTTPS load balancer, a web application firewall, restricted AWS service access, protected Terraform state and an independently owned audit destination. You may finish the explanation and decide that your project needs only some of those properties. That is a useful outcome. The tutorial is here for when you have a reason to build this particular design.</p><p>We will follow the money through the architecture and examine what each major component provides. If the tradeoffs make sense for your use case, the complete Terraform implementation and hands-on deployment guide are available in the companion repository and Wiki.</p><p>You can read the first half as an architecture review and stop at the decision about whether to use it. The second half takes you from a local container to an AWS deployment, including inspection, a failure exercise and retirement. The <a href="https://github.com/marco-bertelli/aws-terraform-sample">repo</a> contains the source.</p><p><strong>The estimate, before the explanation</strong></p><p>Here are the assumptions behind the headline: <br>- EU London, `eu-west-2`; Linux/x86 Fargate;<br>- 730 hours; two tasks, each requesting 0.25 vCPU and 0.5 GiB of memory.<br>- The traffic model assumes one million requests, one average load-balancer capacity unit and 10 GB processed through interface endpoints.</p><p>Prices were checked on <strong>9 September 2026</strong>. There are no credits, Free Tier benefits or discounts in the calculation. The reference data contains regional AWS SKU identifiers and source publication dates.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*IFRgIqUxUb8_UKWehlpAMA.png" /><figcaption>Estimated Prices</figcaption></figure><p><strong>Total, calculated before rounding $133.85</strong></p><p>The $20 allowance is deliberately labelled. It is not a quoted AWS bundle, and it cannot limit actual charges. To turn this into a budget for your workload, replace the allowance with expected log ingestion, retention, image storage and request quantities.</p><p>Internet and applicable cross-AZ data transfer, domain registration, CI runners, support, taxes and optional databases are outside this total. More public ALB addresses, retained recovery resources or heavy logging can increase it. State and audit costs shared by several environments should be allocated once across the estate.</p><p>There is an easy way to misread this table. You could conclude that AWS charges $134 to execute a few lines of JavaScript. That would miss most of the resources we provisioned. You could also conclude that the additional spending makes the deployment production-ready. The table cannot establish that either.</p><p>What it gives us is a set of choices to examine.</p><p><strong>The largest line is a network decision</strong></p><p>A client resolves the API hostname through DNS and opens an HTTPS connection to the Application Load Balancer. The ALB evaluates the request with the associated WAF configuration and forwards permitted traffic to a healthy private task. Node handles the request on port 8080 and returns the greeting.</p><p>The task has no public IP address. Its security group accepts port 8080 only from the ALB security group. You cannot bypass the load balancer by discovering the task’s private address and connecting to it from the public internet.</p><p>That is a useful, narrow property. It does not authenticate API users. The example’s two endpoints are intentionally public and contain no private information.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*PgKwB8HV6V-BXSxzluviIw.png" /></figure><p><em>The normal request path ends at Node. Image pulls and log delivery use supporting connections outside that request path.</em></p><p>A <strong>VPC</strong> is the virtual network in which we define the workload’s address space and connectivity. A <strong>subnet</strong> is a range within it, placed in one Availability Zone. Route tables determine where packets can go; security groups decide which connections are permitted at the resources they protect. Those are different controls.</p><p>For example, a public subnet has a route to an internet gateway. That route alone does not make every resource in the subnet reachable. Addressing and access rules still matter. Our application subnets have no general IPv4 default route to the internet and the tasks have no public addresses. The public and application subnets therefore serve different purposes.</p><p>Security groups are stateful: response traffic for an allowed connection is handled as part of that connection. We do not add a broad outbound internet rule simply so Node can return an ALB request. The required application ingress is the specific ALB-to-task relationship on port 8080.</p><p>So far, the API looks pleasantly isolated. Then ECS tries to start it.</p><p>The container image lives in ECR. Its layers must be downloaded. The logging driver needs to reach CloudWatch Logs. A task can fail before Node runs because one of those supporting paths is missing.</p><p>We provide three interface endpoint services: `ecr.api`, `ecr.dkr` and `logs`. Each has a placement in both AZs. The ECR image-layer download also uses an S3 gateway endpoint. These are connections to AWS services, not extra application servers.</p><p>The hourly calculation is:</p><p><em>3 services × 2 AZs × 730 hours × $0.011 = $48.18</em></p><p>That is the largest individual line in the estimate. The endpoints keep their hourly cost even when the API is idle. AWS bills interface endpoint placement by AZ; deleting the endpoint ends that hourly charge.</p><p>Why accept it? Because this design can start its containers and deliver logs without a general workload internet route. Its permitted startup dependencies are easy to identify. If a future change needs a third-party API, the missing path becomes an explicit design decision.</p><p>That restriction has a practical cost beyond dollars. A developer cannot add an external HTTP dependency and assume it will work. A package installation attempted during container startup will fail. The build environment must produce an image that is ready to execute, and the image must already exist in the private ECR repository.</p><p>Those are useful constraints for some systems. For others, they are friction with little benefit.</p><p>A NAT-based design would give private workloads a path to public destinations and would need its own availability, processing-cost and egress-control review. NAT alone does not provide a destination allowlist. A comparison should include the endpoint services you still need, actual traffic, placement and permitted destinations. “No NAT” is a description of this implementation, not a general claim that endpoints are always cheaper.</p><p>There is another limit worth keeping visible: private connectivity is only one layer. Endpoint policies and IAM still determine service access. Here, the interface endpoint policies constrain callers to the workload account, while the execution roles narrow repository and log permissions. The S3 gateway policy permits the regional ECR layer-bucket path. A private endpoint with an overly broad authorization model would still be a poor boundary.</p><p>If I removed this $48.18 line during a review, I would want a replacement explanation for image pulls, logs and outbound access. The saving is real. So is the work of choosing what replaces it.</p><p><strong>Twenty-one dollars of compute, and a more interesting promise</strong></p><p>ECS uses a <strong>task definition</strong> to describe what runs: image, CPU, memory, roles, logging and health checks. A <strong>task</strong> is one execution. A <strong>service</strong> maintains the desired number of those executions. Fargate supplies the compute environment, so we do not manage an EC2 host fleet for this example.</p><p>One small task for a planning month costs:</p><p><em>730 × (0.25 × $0.04656 + 0.5 × $0.00511) = approximately $10.36</em></p><p>Two cost about $20.72. Fargate charges for allocated resources and runtime, including the startup interval described in its pricing rules. A mostly idle process does not make the requested allocation free.</p><p>The second task is interesting because of what happens around it.</p><p>Suppose we replace an image. ECS can start new tasks while healthy old tasks remain available. The load balancer can route around a target that has not become healthy. If one task exits, the service attempts to restore the desired count.</p><p>That gives us a mechanism for replacement. Whether users see an interruption depends on startup time, task placement, target health, available capacity and how the application shuts down. A desired count of two in a plan does not prove any of those observations. We inspect the deployed AZ placement and test the service while changing it.</p><p>The production example requires at least two tasks. Its rolling limits retain 100 percent healthy capacity and allow up to 200 percent of the desired count. With two steady-state tasks, a rollout can temporarily use four. The monthly estimate describes steady state; it does not price unlimited deployment overlap.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*hgohERFSqREgbwAvGurkiQ.png" /></figure><p>The API also has a small runtime contract. It listens on port 8080, runs as user 10001, accepts a read-only root filesystem and handles SIGTERM. The container health command calls `/health` locally. The ALB performs its own health request through the target network path.</p><p>These two checks can disagree for a good reason. Local health may succeed while a security-group mistake blocks the ALB. A healthy target can return a greeting while running the wrong image. That is why the release checks compare the selected digest as well as the response.</p><p>The ECS deployment circuit breaker adds another useful behavior: a revision that fails to stabilize can roll back to a previous completed deployment. The first deployment has no successful predecessor. A rollback also does not magically edit the Terraform variable that still names the rejected image; the operator must reconcile the intended configuration afterward.</p><p>The compute spending buys a place to run and enough configured capacity to attempt this workflow. Keeping a known-good artifact, investigating a failure and deciding when to halt a rollout remain team responsibilities.</p><h3><strong>A public entrance with private backends</strong></h3><p>The load-balancer portion of the example is $19.32 for provisioned hours and $6.13 for the assumed average capacity. Add $7.30 for two public IPv4 addresses. That gives the public entrance a noticeable cost before WAF or DNS.</p><p>An <strong>Application Load Balancer</strong> routes HTTP requests to targets and evaluates their health. <strong>ACM</strong> supplies the managed certificate used by the HTTPS listener, and <strong>Route 53</strong> points the hostname to the load balancer. These services let the public hostname remain stable while individual task addresses change.</p><p>An ALB capacity unit is based on measured dimensions such as connections, processed bytes and rule evaluations. One million requests is therefore insufficient information to predict its capacity charge. Long-lived connections and large responses can produce a very different result from short greetings. The example’s one average LCU is an assumption you should replace with measurements.</p><p>Likewise, two public IPv4 addresses are a minimum assumption for this topology, not a fixed cap on future allocation. Check the actual address inventory and usage.</p><p>There is a deliberate boundary in the encryption story. The client uses HTTPS to the ALB. The ALB forwards HTTP to the private task. If your requirements call for TLS on that internal hop too, this implementation needs a change. Describing it as “encrypted end to end” would be incorrect.</p><p>WAF contributes $9.60 in this traffic model: one web ACL, four billed rule entries and a million requests. The configuration uses three standard AWS managed rule groups plus an IP-based rate rule. It also sends redacted WAF logs to CloudWatch and disables sampled requests.</p><p>A web application firewall evaluates traffic against rules. It can reject requests that match selected patterns before those requests reach Node. It can also reject a legitimate request, so enabling rules creates a tuning and investigation responsibility.</p><p>The sample has no database query to inject into and no customer login to bypass. The SQL-injection rule group is part of the broader reference configuration, rather than a fix for an existing Hello World query. It is reasonable to question its inclusion for this exact application. A future API should choose rules against its actual request shapes and threats instead of inheriting a set without review.</p><p>An IP-based rate rule is also a coarse traffic control. Many legitimate users can share a public address, and attackers can distribute traffic across addresses. It is not a precise per-user quota or a spending limit.</p><p>For this platform, WAF establishes a controlled place to apply web filtering. It does not establish application authorization, prevent every attack or guarantee that unwanted traffic will cost nothing. The recurring charge needs an owner who knows which rules are active and how to investigate a block.</p><h3><strong>The controls with small bills and large consequences</strong></h3><p>Some of the most consequential decisions barely show up as dedicated lines in the estimate.</p><p>Take the role attached to the API. It has <strong>no IAM permissions policy</strong>. The server returns public JSON and calls no AWS APIs, so there is no reason to give its application identity access to S3, secrets or infrastructure changes.</p><p>ECS still needs permission to pull the image and write logs. That belongs to the <strong>execution role</strong>, which supports startup and service integration. The <strong>task role</strong> is the identity available to code inside the container. Keeping those responsibilities separate avoids granting application code permissions merely because the platform needs them during startup.</p><p>A role’s trust policy describes who may assume it. Its permissions describe what the resulting session may do. Human federation and GitHub OIDC can supply temporary sessions, but the duration of a credential does not tell you whether its authority is appropriate. A temporary administrator is still an administrator.</p><p>The optional image publisher can push to the API repository. It cannot update ECS, read Terraform state or publish the privileged database maintenance image. A provisioner selects the image digest and applies the infrastructure plan. That identity is powerful enough to change the controls themselves, and we describe it as administration.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*us6L3RtJXKz2_iG9Cli4ag.png" /></figure><p><em>Follow the identity that can choose the code, as well as the identity the code receives.</em></p><p>Imagine a policy review that proves the Node task cannot retrieve an administrator secret. Useful. Now ask who can register a new task definition, select arbitrary code and pass a secret-bearing execution role to ECS. That operator may have a path to the same secret without directly calling its retrieval API.</p><p>This is why `iam:PassRole` deserves attention. It participates in delegating authority to a service. The database lab makes that path explicit, while the ordinary API has no secret to receive.</p><p>Terraform state has a different job. It records the relationship between configuration and managed resources, and may contain sensitive operational information. The bootstrap creates a versioned S3 bucket, KMS encryption and native S3 locking. Separate environment state keys and working directories reduce accidental overlap. Account restrictions make a mismatched credential fail before a normal apply proceeds.</p><p>A saved plan ties review to a proposed change. It is not a substitute for understanding the plan, and the file itself needs private handling. The companion wrapper requires a saved plan for apply rather than silently planning a new change at execution time.</p><p>For audit records, the workload trail delivers management events into another account. An ordinary workload identity receives no read or delete access there. This changes who controls evidence already delivered after a workload incident.</p><p>It leaves two important questions. Who can stop future logging in the workload account? Who can bypass governance retention in the audit account? Those identities still exist in the trust model. Separate ownership improves the boundary; it does not eliminate the need to administer it.</p><p>CloudTrail, ALB logs, application logs and VPC flow logs describe different events. A management trail does not record every HTTP call or SQL query. Our Node logs omit request headers, bodies and query strings, but ALB access logs have their own fields and can include request URLs. “The app does not log it” is not a promise about every log in the platform. Keep secrets out of URLs and review access and retention for each log destination.</p><p>The small monthly key allocation is easy to approve. Defining human access, retaining state correctly, confirming audit delivery and practicing emergency access take more thought. None of that labor is included in $133.85.</p><h3><strong>The database is a separate decision</strong></h3><p>It would be easy to add PostgreSQL to the drawing because production architecture diagrams often have a database. Hello World has nothing to store, so the default Terraform configuration does not create one.</p><p>There is an optional RDS lab for readers who want to work through credentials, private SQL access and recovery. It creates one synthetic checkpoint row and a reader identity. The API remains independent of it.</p><p><strong>RDS</strong> operates a database engine while leaving you responsible for capacity, network access, database grants and recovery decisions. A traditional Multi-AZ instance deployment maintains a standby for availability. That standby is not a read replica for application traffic. <strong>Secrets Manager</strong> stores and manages secret versions; <strong>KMS</strong> controls encryption-key use. IAM database authentication supplies another login mechanism, with SQL privileges still governing the actions permitted after login.</p><p>For 730 hours, the optional lab adds:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Bos8sIfNZ3sMl28YYSXwQw.png" /><figcaption>Lab additional costs</figcaption></figure><p>The two-task baseline with the Multi-AZ lab comes to about <strong>$216.85 per month</strong> under these assumptions. This is an addition you can choose, not a prerequisite for the main API.</p><p>The interesting lesson is the credential path. One approved initializer task receives the RDS-managed administrator secret. It creates a `<em>course_probe</em>` database user and grants it read access to a synthetic table. A different task uses IAM authentication to read that table and verify that an update is denied. Only the initializer execution role can retrieve the administrator secret.</p><p><em>The extra database cost funds an independent lab. It adds no startup dependency to Hello World.</em></p><p>This makes several failure modes observable. A timeout is a network investigation. A TLS verification error concerns the certificate path or hostname. An IAM login failure is different from a SQL permission denial. Giving the task more AWS permissions will not repair an incorrect PostgreSQL table grant.</p><p>It also gives rotation a concrete meaning. A newly started initializer fetches the current managed password. A container already running does not have its environment rewritten when a secret rotates. The probe generates fresh IAM authentication material when it connects. An expired login token does not terminate a database session that already authenticated.</p><p>For a real stateful application, these mechanisms would need to be integrated into its connection lifecycle. In this lab, the short-lived tasks let you study them without writing that application first.</p><h3><strong>Would I choose this for a real project?</strong></h3><p>For an occasional public greeting, this would be a hard design to justify. The fixed services dominate the budget, and the team still has to understand IAM, Terraform state, networking and release behavior.</p><p>For a service that must run containers privately, has a clear need to separate publishing from deployment, and needs an observable replacement procedure, the same spending becomes easier to explain. A team may value a repeatable hosting pattern across applications more than the lowest possible cost for one idle endpoint.</p><p>The deciding factor is the requirement behind each resource.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*xLu-72cm3SkRXLigGRecSA.png" /></figure><p>The request-driven alternative is a different operating model, with service-specific quotas and pricing to review; this article does not assert a cheaper total without a workload comparison.</p><p>There are also three useful cost profiles within the current design:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Z7FSg1VHc_lIqvS2kBfxZw.png" /></figure><p>The allowances differ between profiles. So do traffic assumptions. The difference between the last two totals is not just the cost of the second container; that container alone adds about $10.36 at a full month of runtime.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*mnQfo9UvT4mRMzYnUW-FNg.png" /></figure><p><em>The 48-hour figure requires actual retirement of hourly resources. Leaving the foundation running changes the estimate.</em></p><p>Before removing a resource, write down what changes operationally. Removing a second task means accepting less replacement capacity. Removing WAF changes where request filtering happens. Removing endpoints requires a different startup and log-delivery path. Reducing log retention changes which incidents you can investigate later.</p><p>Some reductions are entirely sensible. I would rather see a small, understood deployment with explicit limitations than an elaborate one whose owner cannot explain the plan. The objective is to make the tradeoff visible before it appears as an incident or an unexpected charge.</p><p>If this particular set of tradeoffs fits what you want to learn or operate, the rest of the article builds it.</p><h3>Build it, if the tradeoff fits</h3><p>At this point, the architecture review is complete. We know where most of the fixed cost comes from, what each component contributes, and which responsibilities remain with the team even after AWS has been paid.</p><p>If those tradeoffs fit something you want to operate or simply something you want to study in a realistic environment the full implementation is available as a separate hands-on guide:</p><p><a href="https://github.com/marco-bertelli/aws-terraform-sample/wiki/Build-it:-from-a-local-container-to-AWS"><strong>Build it: from a local container to AWS</strong></a></p><p>The guide starts with the local Node container and follows the platform through Terraform bootstrap, private networking, ECR publication, ECS activation, release inspection, failure exercises, audit validation, optional RDS recovery work and final resource retirement.</p><p>The separation is intentional.</p><p>The architecture discussion should stand on its own even if you never deploy it. Conversely, if you do deploy it, the exercise should be detailed enough to test the properties we have been talking about rather than stopping when the first 200 OK appears.</p><h3>What did the $133.85 actually buy?</h3><p>It did not buy 134 dollars’ worth of JavaScript execution.</p><p>The application still does almost nothing.</p><p>Most of the bill comes from keeping infrastructure available around it.</p><p>The interface endpoints buy a restricted startup and logging path for private tasks without giving those workloads a general route to the internet.</p><p>The second task buys additional replacement capacity, although availability still has to be observed rather than inferred from desired_count = 2.</p><p>The load balancer provides a stable public HTTPS entry point while private task addresses can change underneath it.</p><p>WAF provides a place to evaluate selected request filters before traffic reaches the application.</p><p>Separate execution and task identities let us distinguish permissions required by the platform from permissions available to application code.</p><p>Protected Terraform state and independently owned audit records create boundaries around two things that become especially important when something goes wrong: who can change the infrastructure and who can change the evidence of what happened.</p><p>None of those properties makes the application automatically secure, highly available or production-ready.</p><p>They are mechanisms.</p><p>Their value depends on requirements, configuration, testing and the people operating them.</p><p>That distinction matters because architecture diagrams make capabilities look binary. A box labelled “WAF” can suggest security. Two tasks can suggest high availability. A private subnet can suggest isolation. A backup can suggest recovery.</p><p>The more useful questions are operational:</p><ul><li>Can we show what traffic the workload is allowed to initiate?</li><li>Can we identify the exact image that is running?</li><li>Can we replace a task and observe what clients experience?</li><li>Does the expected responder actually receive the alarm?</li><li>Can an identity outside the workload account inspect the audit evidence?</li><li>Can we restore the state we said was recoverable and use the result?</li></ul><p>Those answers tell us more than the presence of the boxes.</p><h3>What remains after the greeting</h3><p>The initial estimate looks strange because the application is almost empty. That is exactly what makes the example useful: there is nowhere for infrastructure spending to hide behind business logic.</p><p>We can point at roughly $48 of private connectivity and explain the startup dependencies and outbound restrictions behind it.</p><p>We can point at the second container and explain the replacement behavior we expect to observe.</p><p>We can point at the load balancer and explain why a stable public endpoint exists independently of individual tasks.</p><p>We can point at state, roles and audit storage and discuss who is allowed to change the system, deploy code and retain evidence.</p><p>Some projects genuinely need those properties.</p><p>Others should use a much smaller design.</p><p>And even this architecture leaves important questions open. It runs in one Region. Task counts are fixed. TLS terminates at the load balancer before HTTP is forwarded to the private application target. Human identity governance remains outside the platform. No monthly AWS bill resolves those decisions for us.</p><p>So before approving an architecture like this, I would ask its owner for something more concrete than a diagram.</p><ul><li>Show me the image digest that is actually running.</li><li>Show me what happened when a task disappeared.</li><li>Show me an alarm that reached its intended responder.</li><li>Show me the resources that are still billing and reconcile them with the estimate.</li><li>Show me which identities can change the application, the infrastructure and the audit trail.</li></ul><p>Those answers explain what the architecture is buying.</p><p>The Node response will still be just:</p><p>{&quot;message&quot;:&quot;Hello World&quot;}</p><p>The useful result is being able to explain everything required to keep it there and deciding which of those requirements your next application actually has.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=50900884ee70" width="1" height="1" alt=""><hr><p><a href="https://levelup.gitconnected.com/why-does-hello-world-cost-134-month-on-aws-50900884ee70">Why Does Hello World Cost $134/Month on AWS?</a> was originally published in <a href="https://levelup.gitconnected.com/">Level Up Coding</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Stop Building Toy RAGs: A Guide to Self-Hosting LLMs on AWS SageMaker]]></title>
            <link>https://levelup.gitconnected.com/stop-building-toy-rags-a-guide-to-self-hosting-llms-on-aws-sagemaker-58e9ec4af024?source=rss-75f167fb6874------2</link>
            <guid isPermaLink="false">https://medium.com/p/58e9ec4af024</guid>
            <category><![CDATA[artificial-intelligence]]></category>
            <category><![CDATA[software-development]]></category>
            <category><![CDATA[technology]]></category>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[machine-learning]]></category>
            <dc:creator><![CDATA[Marco Bertelli]]></dc:creator>
            <pubDate>Tue, 16 Dec 2025 18:45:00 GMT</pubDate>
            <atom:updated>2025-12-19T03:22:07.334Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*3US2471ZKxR1Ew6eecjX3Q.png" /><figcaption>Build a RAG System with self hosted LLM</figcaption></figure><h3>Introduction</h3><p>The “Model Wars” are in full swing. Every single day seems to bring a new open-source release, accompanied by a flood of public benchmarks and KPIs claiming to beat the competition.</p><p>the real challenge for engineers isn’t just picking a winner — it’s figuring out what to do with it. <strong>How do we actually put one of these models into production?</strong> Or, more immediately: how do we efficiently test a model that demands significant resources just to start up?</p><p>If that first question sounds difficult, the follow-up is even trickier: <strong><em>Once I have my model hosted, how do I integrate it into a real-world RAG application effectively?</em></strong></p><p>Before we dive in, let’s clarify a few critical points. This is important to set the expectations for what we are about to build together.</p><ul><li><strong>This is not another “toy” tutorial.</strong> You’ve probably seen dozens of guides on spinning up a tiny, quantized model locally on a laptop for a simple QA bot. That is <em>not</em> what we are doing here. This article is designed as a research piece — a deep dive into answering the complex infrastructure and integration questions that everyone else seems to be ignoring.</li><li><strong>Prerequisites are required.</strong> I won’t be covering the absolute basics of RAG here. You need to understand the fundamental architecture of a Retrieval-Augmented Generation system. If you are new to this, check out the series on my profile where I build a RAG system from scratch using open-source tech. It covers the theory you’ll need to follow along today.</li><li><strong>Fresh Code for 2025.</strong> If you’ve followed my previous articles, you know the theory holds up, but the code needed a refresh. I have updated the repository with a modern, 2025-ready stack. All the code we analyze in this article is available in a dedicated branch, ready for you to download, inspect, and test.</li></ul><h3>Theory: The &quot;Why&quot; and &quot;What&quot; of Self-Hosting</h3><p>Let’s start with a bit of theory. When do you actually need a self-hosted LLM, and what does that term really mean in this context?</p><p>In the real world, clients often request open-source models for two main reasons: they anticipate <strong>massive traffic</strong>, or they are deeply concerned about <strong>data privacy</strong>. However, often when a client asks for this, they don&#39;t fully grasp the operational complexity or the sheer cost of taking a raw LLM into production.</p><p><strong>The Reality Check:</strong> In most cases, you probably <strong>don&#39;t</strong> need a self-hosted LLM. If the primary concern is privacy or GDPR compliance, enterprise solutions like Azure OpenAI Service already offer robust, compliant environments where your data isn&#39;t used for model training. If you can use a managed API, you usually should.</p><p><strong>So, when is self-hosting the right call?</strong> We are here because sometimes those easy answers don&#39;t apply. Maybe you have a project with extremely specific high-traffic requirements where token costs on APIs would be prohibitive, or perhaps strict legal constraints require the entire architecture—model included—to live solely within the client&#39;s private cloud account. In these scenarios, self-hosting is the only path forward.</p><p>By &quot;self-hosted,&quot; I mean taking an open-source model (like Llama 3, Mistral, etc.) and running it on infrastructure that you control and manage, rather than calling an external API.</p><h3>Why I Choose AWS SageMaker</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*7x6Z_K_3UiyEzZ7qXJDEPw.jpeg" /><figcaption>Sagenaker AWS Service</figcaption></figure><p>For this architecture, I’ve chosen <strong>AWS SageMaker</strong>.</p><p>Why not just spin up a standard EC2 instance with a GPU? SageMaker offers a massive advantage: it provides pre-built, optimized environments (Deep Learning Containers) for the most popular models.</p><p>Trust me on this: if you try to manually configure an environment for a modern LLM on a raw EC2 instance, handling CUDA drivers, library dependencies, and model sharding from scratch, you are looking at two weeks of frustration. And even after that effort, your result will likely be less optimized than the default SageMaker containers designed by AWS architects. Unless you have a dedicated MLOps team, manual setup is a trap for a non expert person.</p><p>The deal AWS offers is simple: they provide the &quot;JumpStart&quot; list of models with ready-to-deploy environments, and they handle the heavy lifting—like autoscaling, health checks, and suggesting the ideal instance type for your specific model. This lets us focus on the application logic, not the plumbing.</p><h3>Step-by-Step: From Console to Deployment</h3><p>So, how do we actually do this?</p><p>Head over to your AWS console and navigate to the <strong>SageMaker</strong> service. From there, open <strong>SageMaker Studio</strong> (if you haven’t created a domain yet, you’ll need to set that up first — it takes just a few clicks).</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*bSJEU7y_Jz5RE8dWHba4Gg.png" /><figcaption>Sample Image of what you will see</figcaption></figure><p>Inside Studio, look for the <strong>JumpStart</strong> section. This is AWS’s “App Store” for models, featuring a vast library of popular open-source LLMs. Here, you can browse, select, and deploy with a single click.</p><p>For this project, I have chosen <strong>Mistral-Nemo</strong>. Here is why this specific model is the perfect fit for our use case:</p><ul><li><strong>Language Proficiency:</strong> It is excellent at Italian. This is crucial: most benchmarks you see online are English-centric. If your documents and users are Italian (or any other non-English language), a generic model might struggle with nuances. Always choose a model with strong multilingual training.</li><li><strong>The “Goldilocks” Size:</strong> It’s not too big, not too small. Before writing this article, I tested 5 or 6 different models. The results were clear: models that are too small often hallucinate or produce inconsistent reasoning, while the massive 70B+ parameter models are overkill and incredibly expensive to host. Mistral-Nemo hits that sweet spot of performance vs. cost. <em>(Note: Sizing models is a complex topic with many variables. if you are interested, let me know in the comments and I can write a dedicated deep-dive on it).</em></li><li><strong>Instruct Tuned:</strong> We are using the “Instruct” version of the model. For those new to this world, this means the model has been fine-tuned specifically to follow instructions and answer questions, rather than just autocompleting text. This is essential for RAG.</li></ul><h3>crucial Infrastructure Tips for 2025</h3><p>When you select your model, SageMaker will ask you which instance type (Virtual Machine) to use. Here is a pro-tip for the current landscape in late 2025:</p><p><strong>Try to use </strong><strong>ml.g6 instances.</strong> Why? Because the older ml.g5 instances are facing a global capacity shortage on AWS. It sounds crazy for a cloud giant, but it is the reality. If you try to deploy on ml.g5, you will likely hit an InsufficientCapacityError. The ml.g6 family is newer, often more available, and offers better price-performance.</p><p><strong>Don’t forget the Service Quota!</strong> These GPU machines are powerful and expensive. By default, AWS sets your quota for these instances to <em>zero</em> to prevent accidental billing spikes. Before you can deploy anything, you must request a <strong>Service Quota Increase</strong> for the specific instance type you plan to use. Do this immediately — support tickets can take a day or two to process.</p><h3>The Missing Piece: Embeddings</h3><p>Once you click deploy, you will have a <strong>SageMaker Endpoint</strong> online for your LLM. (Remember to delete it if you are just testing, or your credit card will suffer!).</p><p>But for a RAG system, an LLM is not enough. We need one more endpoint: the <strong>Embeddings Model</strong>. Just like before, search for an embedding model in JumpStart. I personally recommend <strong>BGE-M3</strong>. It is part of the “Large” family, has fantastic multilingual support, and handles dense retrieval exceptionally well. Deploy this on a smaller instance to create your second endpoint.</p><p><strong>The Big Questions:</strong> Now we have two endpoints running. But what hardware is actually powering them? And the question everyone is afraid to ask: <strong>How much is this going to cost me per month?</strong></p><h3>Hardware Specs &amp; The Bill</h3><p>Let’s get down to the numbers. High performance comes with a price tag, and transparency is key here. We are moving away from the “free tier” experiments into enterprise-grade infrastructure.</p><p>Here is the breakdown of the hardware powering our two endpoints and what it costs to keep the lights on.</p><h4>The Embeddings Machine</h4><p>For the embedding model (like BGE-M3), we need a machine capable of fast, dense vector calculations. Based on the specs, we are looking at an instance class equivalent to the <strong>ml.g5.2xlarge</strong>.</p><p><strong>Technical Specifications</strong></p><pre>Instance Class:   ml.g5.2xlarge (approximate match)<br>---------------------------------------------------<br>vCPU:             8 (AMD EPYC 7R32 – 2.8 GHz)<br>Memory (RAM):     32 GiB<br>GPU:              1x NVIDIA A10G<br>Video Memory:     24 GiB GDDR6<br>Storage:          1x 450 GiB NVMe SSD (Fast ephemeral)<br>Network Bandwidth: Up to 10 Gbps</pre><p><strong>Estimated Costs</strong></p><pre>Purchase Model      Hourly Price     Est. Monthly Cost<br>------------------------------------------------------<br>On-Demand           $1.212           ~$885<br>Spot (Variable)     ~$0.426          ~$311<br>Reserved (1 Year)   ~$0.764          ~$558<br>Reserved (3 Year)   ~$0.524          ~$383</pre><h4>The LLM Host (Mistral-Nemo)</h4><p>Now for the main event. For the LLM, AWS suggests deploying on the <strong>ml.g6.12xlarge</strong>. This is a powerhouse machine designed to handle the massive matrix multiplications required by Generative AI.</p><p><strong>Technical Specifications</strong></p><pre>Instance Class:   ml.g6.12xlarge<br>---------------------------------------------------<br>vCPU:             48 (AMD EPYC 7R13)<br>Memory (RAM):     192 GiB<br>GPU:              4x NVIDIA L4 Tensor Core<br>Video Memory:     96 GiB Total (4x 24 GiB GDDR6)<br>Architecture:     Ada Lovelace (Successor to Ampere)<br>Storage:          7.5 TB NVMe SSD<br>Network Bandwidth: 25 Gbps</pre><p><strong>Estimated Costs</strong></p><pre>Purchase Model      Hourly Price     Est. Monthly Cost<br>------------------------------------------------------<br>On-Demand           $3.924           ~$2,865<br>Spot (Variable)     ~$1.648          ~$1,203<br>Reserved (1 Year)   ~$2.472          ~$1,805<br>Reserved (3 Year)   ~$1.695          ~$1,237</pre><p><strong><em>A quick note on Spot Instances:</em></strong><em> You will notice the Spot prices are significantly lower. However, </em><strong><em>do not use Spot instances for a production endpoint</em></strong><em> unless you have a robust failover system. AWS can reclaim these machines with only a 2-minute warning, which means your RAG system would simply go offline randomly. For production, stick to On-Demand or Reserved.</em></p><p><strong>So, what is the final damage</strong>? If we run both endpoints 24/7 using On-Demand instances (which is the standard for production availability), here is your total monthly bill:</p><pre>Monthly Cost Recap (On-Demand)<br>------------------------------------------------------<br>Embeddings Endpoint (ml.g5.2xlarge):   $   885<br>LLM Endpoint (ml.g6.12xlarge):         $ 2,865<br>------------------------------------------------------<br>TOTAL MONTHLY ESTIMATED:               $ 3,750<br>TOTAL YEARLY ESTIMATED:                $45,000</pre><p><strong>Let that sink in.</strong> We are talking about roughly <strong>$45,000 per year</strong> just for the raw compute to keep the lights on. This doesn’t include data transfer costs, storage fees, or the salary of the engineers managing it.</p><p>This brings us back to the introduction: <strong>Self-hosting is a commitment.</strong> You are paying this premium for total data sovereignty, zero data leakage to third-party APIs, and guaranteed throughput. If your project doesn’t strictly require these features, you might want to reconsider managed APIs. But if you <em>do</em> need them, this is the price of independence.</p><h3>Real World Performance: How Many Users?</h3><p>We have talked about the specs and the scary price tag (~$3,750/mo total). Now, the most critical question for an architect: <strong>What does that money actually buy me in terms of traffic?</strong></p><p>Mathematical estimates on paper are fine, but they rarely survive contact with reality. So, we didn’t guess. We performed a real-world load test using <strong>Apache Benchmark</strong> against the <strong>ml.g6.12xlarge</strong> endpoint. We wanted to see exactly how the inference engine behaves under stress and where the breaking point lies.</p><p>Here are the results of our stress test, simulating concurrent users hitting the RAG system simultaneously.</p><p><strong>Apache Benchmark Load Test Results</strong></p><pre>Scenario: Mistral-Nemo on ml.g6.12xlarge<br>-----------------------------------------------------------------------<br>Users   Avg Response Time  Status    Notes<br>-----------------------------------------------------------------------<br>1       ~4.3s               ✅ OK     Baseline inference speed.<br>                                      Fast and snappy.<br><br>10      ~16.7s              ✅ OK     Initial Saturation.<br>                                      The GPU is working hard to parallelize.<br>                                      Response time slows (4s -&gt; 16s).<br><br>20      ~29.5s              ⚠️ LIMIT  Total Saturation.<br>                                      Wait times double linearly.<br>                                      We reached the UX &quot;patience limit&quot;<br>                                      (30s wait is rarely acceptable).<br><br>30      N/A                 ❌ FAIL   System Crash / Timeout.<br>                                      The system cannot respond within<br>                                      acceptable HTTP timeout windows.</pre><h3>Analyzing the Limits</h3><p>These results tell us a very clear story about the capacity of a single node.</p><ul><li><strong>The ideal Spot (1–5 Users):</strong> If you have a few concurrent users, the system flies. 4 seconds for a complex RAG answer is a great user experience.</li><li><strong>The “Queue” Effect (10 Users):</strong> As we hit 10 concurrent users, you can see the queue building up. The GPU is powerful, but it has to time-slice resources. 16 seconds is noticeable, but for an internal enterprise tool, it is often acceptable.</li><li><strong>The Hard Limit (20 Users):</strong> This is your red line. At 20 concurrent users, the wait time hits ~30 seconds. In the world of web UX, 30 seconds is an eternity. Users will assume the application is broken and refresh the page (which actually makes the load worse).</li><li><strong>The Breaking Point (30+ Users):</strong> Beyond this, requests simply time out.</li></ul><p>A single <strong>ml.g6.12xlarge</strong> instance can realistically handle a maximum of <strong>15–20 concurrent users</strong> before the user experience degrades significantly. If your application expects 50 or 100 concurrent users (users hitting “Enter” at the exact same second), you will need to configure <strong>Auto-Scaling</strong> in SageMaker to spin up a second or third instance automatically. And yes, that implies multiplying the monthly cost we calculated earlier.</p><h3>LlamaIndex &amp; SageMaker (code implementation)</h3><p>We have the hardware running. The endpoints are live. Now, we need to make our application talk to them.</p><p>For build a RAG System i choose Llamaindex, in my opinion on of the best framework’s currently available. If you are coming from the world of OpenAI API, you are used to a “plug-and-play” experience. You just change the API key, and it works. <strong>SageMaker is different.</strong> SageMaker endpoints are raw HTTP endpoints. They don’t have a standardized API schema like OpenAI. One model might expect input as {&quot;text&quot;: &quot;...&quot;}, while another expects {&quot;inputs&quot;: &quot;...&quot;}.</p><p>This is why is soo important to not only try a lot of model’s but try to understand each model feature, structures and limitations.</p><p>To bridge this gap using <strong>LlamaIndex</strong>, we need to write <strong>Custom I/O Handlers</strong>. These are small classes that tell LlamaIndex exactly how to format the request (Serialize) and how to read the answer (Deserialize) for our specific models.</p><h4>The Embeddings Handler (BGE-M3)</h4><p>For the embedding model, the task is simple: send text, get a vector. We create a class BGEHandler that inherits from BaseIOHandler.</p><p><em>File: </em><em>handlers/embeddings_handler.py</em></p><pre>import json<br>from typing import TYPE_CHECKING, List<br>from llama_index.embeddings.sagemaker_endpoint.utils import BaseIOHandler<br><br>if TYPE_CHECKING:<br>    from botocore.response import StreamingBody<br>class BGEHandler(BaseIOHandler):<br>    content_type: str = &quot;application/json&quot;<br>    accept: str = &quot;application/json&quot;<br>    def serialize_input(self, request: List[str], model_kwargs: dict) -&gt; bytes:<br>       <br>        request_str = json.dumps(<br>            {&quot;text_inputs&quot;: request, &quot;mode&quot;: &quot;embedding&quot;, **model_kwargs}<br>        )<br>        return request_str.encode(&quot;utf-8&quot;)<br>    def deserialize_output(self, response: &quot;StreamingBody&quot;) -&gt; List[List[float]]:<br>        <br>        response_json = json.loads(response.read().decode(&quot;utf-8&quot;))<br>        return response_json[&quot;embedding&quot;]</pre><h4>The LLM Handler (Mistral-Nemo)</h4><p>The LLM is more complex. We need to handle generation parameters (temperature, max tokens) and, critically, the <strong>Prompt Template</strong>. Mistral uses a specific syntax ([INST] ... [/INST]) to distinguish between user and system roles. If we send raw text without this formatting, the model will get confused.</p><p>Here is our MistralIOHandler:</p><p><em>File: </em><em>handlers/llm_handler.py</em></p><pre>from llama_index.llms.sagemaker_endpoint.utils import BaseIOHandler<br>from llama_index.core.llms import ChatMessage, MessageRole<br>import json<br><br>class MistralIOHandler(BaseIOHandler):<br>    content_type: str = &quot;application/json&quot;<br>    accept: str = &quot;application/json&quot;<br>    def serialize_input(self, request: str, model_kwargs: dict) -&gt; bytes:<br>        params = model_kwargs.copy()<br>        <br>        if &quot;max_new_tokens&quot; not in params:<br>            params[&quot;max_new_tokens&quot;] = 1024<br>        if &quot;temperature&quot; not in params:<br>            params[&quot;temperature&quot;] = 0.3<br>            <br>        request_body = {<br>            &quot;inputs&quot;: request,<br>            &quot;parameters&quot;: params,<br>        }<br>        return json.dumps(request_body).encode(&quot;utf-8&quot;)<br>    def deserialize_output(self, response):<br>        response_json = json.loads(response.read().decode(&quot;utf-8&quot;))<br>    <br>        return response_json.get(&quot;generated_text&quot;, &quot;&quot;)<br><br># The Prompt Formatter that use the custom ministral format<br>def messages_to_prompt_mistral(messages, system_prompt=None):<br>    &quot;&quot;&quot;<br>    Converts a list of messages into the specific Mistral [INST] format<br>    Example: &lt;s&gt;[INST] User Question [/INST] Model Answer&lt;/s&gt;<br>    &quot;&quot;&quot;<br>    prompt = &quot;&lt;s&gt;&quot;<br>    for message in messages:<br>        if message.role == MessageRole.USER:<br>            prompt += f&quot;[INST] {message.content} [/INST]&quot;<br>        elif message.role == MessageRole.ASSISTANT:<br>            prompt += f&quot; {message.content}&lt;/s&gt;&quot;<br>        elif message.role == MessageRole.SYSTEM:<br>            # We prepend system instruction to the first user message<br>            prompt += f&quot;[INST] {message.content}\n\n&quot;<br>            <br>    return prompt</pre><h3>The Assembly: Putting It All Together</h3><p>Now that we have our “translators” (the handlers), we can configure the main application logic.</p><p>In previous versions of this project, we might have initialized an OpenAI object or a Perplexity class. Now, we switch to SageMakerLLM and SageMakerEmbedding provided by llama-index.</p><p>The beauty of this approach is that the rest of your RAG pipeline (vector store, ingestion, query engine) <strong>doesn’t need to change</strong>. You are simply swapping the brain and the eyes of the system.</p><p>Here is how we update index_manager.py:</p><p><em>File: </em><em>index_manager.py</em></p><pre>import os<br>from llama_index.core import Settings<br>from llama_index.embeddings.sagemaker_endpoint import SageMakerEmbedding<br>from llama_index.llms.sagemaker_endpoint import SageMakerLLM<br>from handlers.llm_handler import MistralIOHandler, messages_to_prompt_mistral<br>from handlers.embeddings_handler import BGEHandler<br><br>ENDPOINT_NAME = os.environ.get(&quot;SAGEMAKER_ENDPOINT_NAME&quot;)<br>EMBED_ENDPOINT_NAME = os.environ.get(&quot;SAGEMAKER_EMBED_ENDPOINT_NAME&quot;)<br>REGION_NAME = os.environ.get(&quot;AWS_REGION_NAME&quot;, &quot;eu-west-1&quot;)<br><br># Initialize the Custom Handlers<br># These are the classes we wrote in the previous step<br>ministral_handler = MistralIOHandler()<br>content_handler = BGEHandler()<br><br># Configure the LLM<br>llm = SageMakerLLM(<br>    endpoint_name=ENDPOINT_NAME,<br>    content_handler=ministral_handler,<br>    aws_access_key_id=os.environ.get(&quot;AWS_ACCESS_KEY_ID&quot;),<br>    aws_secret_access_key=os.environ.get(&quot;AWS_SECRET_ACCESS_KEY&quot;),<br>    aws_region_name=REGION_NAME,<br>    # Crucial: This tells LlamaIndex how to format the prompt for Mistral<br>    messages_to_prompt=messages_to_prompt_mistral<br>)<br><br># Configure the Embeddings<br>embed_model = SageMakerEmbedding(<br>    endpoint_name=EMBED_ENDPOINT_NAME,<br>    content_handler=content_handler,<br>    region_name=REGION_NAME,<br>)<br><br># Apply Global Settings<br># From now on, any index construction or query will use these models<br>Settings.llm = llm<br>Settings.embed_model = embed_model</pre><p><strong>Things to notice from previous code:</strong></p><ul><li><strong>Decoupling:</strong> By injecting the content_handler, we keep the main logic clean. If you switch from Mistral to Llama 3 tomorrow, you just swap the handler, not the entire application logic.</li><li><strong>Prompt Formatting:</strong> Notice the messages_to_prompt argument in the LLM. This is where many RAG systems fail. If you don&#39;t map the abstract &quot;User/System&quot; messages to the specific string format the model was trained on (like [INST]), the model&#39;s performance will degrade significantly.</li></ul><h3>Conclusions: The Price of Independence</h3><p>We started this journey with a simple question: <em>How do we take an open-source model and actually make it work in production?</em></p><p>As we have seen, the answer is far from a “Hello World” tutorial. We navigated through the complex landscape of <strong>AWS SageMaker</strong>, discovered the importance of choosing the right instance class (ml.g6 vs ml.g5), and faced the cold hard truth of costs: <strong>~$45,000 a year</strong> for a basic, production-ready setup.</p><p>We also looked at the code, realizing that standard libraries often need custom “drivers” (our Handlers) to talk effectively with raw infrastructure.</p><p><strong>So, is it worth it?</strong> If you are a small startup with no strict data constraints, probably not. Managed APIs like OpenAI or Anthropic are cheaper and easier. But if you are an enterprise where <strong>Data Sovereignty</strong>, <strong>GDPR compliance</strong>, and <strong>Predictable Throughput</strong> are non-negotiable, then yes. This architecture gives you the keys to the castle. You own the model, you own the infrastructure, and no data ever leaves your VPC.</p><p>Oh yes, as always the code is open sourced and available here: <a href="https://github.com/marco-bertelli/rag.flask-start">https://github.com/marco-bertelli/rag.flask-start</a></p><p>keep in mind that that code is not on master branch, soo double check it!</p><h3>🤝 Sponsored by Runelab (my awesome company)</h3><p>Writing deep-dive research articles like this takes time and resources. This guide was made possible by <strong>Runelab</strong>.</p><p>At Runelab, we specialize in building complex, high-performance AI architectures for the enterprise. We don’t just write about it; we build these systems for production environments every day.</p><p>If your company is looking to implement a secure, scalable RAG system — or if you need to migrate from a prototype to a self-hosted architecture — we are here to help.</p><p>📩 <strong>Contact us at:</strong> <a href="https://runelab.it/">https://runelab.it/</a></p><p>or: marco.bertelli@runelab.it</p><h3>What’s Next?</h3><p>This article is part of a larger journey. If you got lost in the technical things or need a refresh on how the Retrieval part works, check out the <strong>previous articles on my profile</strong>. I have covered everything from the basics of Vector Databases to building a RAG system from scratch.</p><p><strong>Support:</strong> If you found this “reality check” useful, leave a clap (or 50!) and drop a comment. It helps me understand what you want to read next.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=58e9ec4af024" width="1" height="1" alt=""><hr><p><a href="https://levelup.gitconnected.com/stop-building-toy-rags-a-guide-to-self-hosting-llms-on-aws-sagemaker-58e9ec4af024">Stop Building Toy RAGs: A Guide to Self-Hosting LLMs on AWS SageMaker</a> was originally published in <a href="https://levelup.gitconnected.com/">Level Up Coding</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Scheduling chrone job’s for production application’s using Node JS]]></title>
            <link>https://levelup.gitconnected.com/scheduling-chrone-jobs-for-production-application-s-using-node-js-4c2d4acdc658?source=rss-75f167fb6874------2</link>
            <guid isPermaLink="false">https://medium.com/p/4c2d4acdc658</guid>
            <category><![CDATA[software-development]]></category>
            <category><![CDATA[nodejs]]></category>
            <category><![CDATA[github]]></category>
            <category><![CDATA[open-source]]></category>
            <category><![CDATA[javascript]]></category>
            <dc:creator><![CDATA[Marco Bertelli]]></dc:creator>
            <pubDate>Sat, 04 Oct 2025 17:06:27 GMT</pubDate>
            <atom:updated>2025-10-10T16:12:21.405Z</atom:updated>
            <content:encoded><![CDATA[<h4><strong>Why we forked Agenda and why you should care</strong></h4><p><em>A production-ready, MongoDB-backed job scheduler for Node.js that’s actively maintained and battle-tested</em></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/720/1*kDx3Y2oIFUspVK1utqXqLQ.png" /></figure><h3>The Open Source Software Graveyard</h3><p>Picture this: you’re building a critical application that needs reliable job scheduling. You find the perfect library — Agenda—only to discover it&#39;s been quietly abandoned. No updates, no bug fixes, and no support for the latest MongoDB drivers. Sound familiar?</p><p>This is the graveyard of promising but forgotten open-source projects. And it’s exactly why <strong>Chronos</strong> exists.</p><h3>Meet Chronos: Agenda Reborn 🔥</h3><p><strong>Chronos</strong> is a lightweight yet powerful job scheduling library for Node.js. It began as a fork of Agenda but with one crucial difference: <strong>it&#39;s actively maintained, battle-tested, and ready for the future.</strong></p><p>We took the solid foundation of Agenda and rebuilt it for the modern development landscape.</p><h3>Why Choose Chronos?</h3><h4>💾 Leverage Your Existing MongoDB Stack</h4><p>If you’re already running on MongoDB, why add the complexity of Redis just for scheduling? Chronos delivers enterprise-grade job scheduling using the database infrastructure you already know and love. That means fewer dependencies, less overhead, and a more consistent stack.</p><h4>🚀 Actively Maintained and Community-Driven</h4><p>Unlike its predecessor, Chronos is a living project. Maintained by <a href="https://www.google.com/search?q=your-personal-github-or-linkedin-link">Marco Bertelli</a> (that’s me!), it’s regularly updated with the latest drivers and backed by responsive community support. You’re not alone; we’re here to build something reliable together.</p><h3>Battle-Tested for Production 🔧</h3><p>Chronos isn’t an experiment. It’s a proven solution packed with all the features you’d expect from an enterprise-grade scheduler:</p><ul><li><strong>Configurable concurrency</strong> to manage your workload.</li><li><strong>Job priorities</strong> to ensure critical tasks run first.</li><li><strong>Failure handling and automatic retries</strong>.</li><li><strong>Event-driven architecture</strong> for maximum flexibility.</li><li><strong>Graceful shutdowns</strong> to ensure no jobs are lost.</li></ul><h3>The Technical Evolution: What We’ve Improved Since Forking Agenda</h3><p>Here’s a look at the key upgrades we’ve made:</p><p>✅ <strong>Modern MongoDB Drivers</strong> Full support for the latest versions of the mongodb driver, guaranteeing better performance, enhanced security, and compatibility with modern MongoDB features.</p><p>✅ <strong>Complete and Clear Documentation</strong> We created a dedicated <a href="https://chronos-docs.vercel.app/">documentation site</a> to make onboarding as fast and painless as possible. No more digging through outdated GitHub issues.</p><p>✅ <strong>Modern JavaScript (ESM &amp; Async/Await)</strong> The codebase has been refactored to use modern JavaScript features, making it cleaner, more performant, and easier to contribute to.</p><p>✅ <strong>First-Class TypeScript Support</strong> Chronos is written in TypeScript, offering robust type safety and a superior developer experience right out of the box.</p><h3>Ready to Make the Switch?</h3><p>It’s time to migrate to a scheduling solution you can trust.</p><ul><li>⭐ <strong>Star us on GitHub:</strong> <a href="https://github.com/marco-bertelli/chronos">github.com/marco-bertelli/chronos</a></li><li>📖 <strong>Check out the docs:</strong> <a href="https://chronos-docs.vercel.app/">chronos-docs.vercel.app</a></li><li>💬 <strong>Join the discussion:</strong> Got questions, issues, or feature requests? We’re here to help in our <a href="https://github.com/marco-bertelli/chronos/discussions">GitHub Discussions</a>.</li></ul><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=4c2d4acdc658" width="1" height="1" alt=""><hr><p><a href="https://levelup.gitconnected.com/scheduling-chrone-jobs-for-production-application-s-using-node-js-4c2d4acdc658">Scheduling chrone job’s for production application’s using Node JS</a> was originally published in <a href="https://levelup.gitconnected.com/">Level Up Coding</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Exploring Multimodal RAG with LlamaIndex and GPT-4 or the New Anthropic Sonnet Model]]></title>
            <link>https://levelup.gitconnected.com/exploring-multimodal-rag-with-llamaindex-and-gpt-4-or-the-new-anthropic-sonnet-model-96705c877dbb?source=rss-75f167fb6874------2</link>
            <guid isPermaLink="false">https://medium.com/p/96705c877dbb</guid>
            <category><![CDATA[artificial-intelligence]]></category>
            <category><![CDATA[programming]]></category>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[technology]]></category>
            <category><![CDATA[data-science]]></category>
            <dc:creator><![CDATA[Marco Bertelli]]></dc:creator>
            <pubDate>Sun, 01 Sep 2024 14:00:39 GMT</pubDate>
            <atom:updated>2024-09-02T00:32:12.317Z</atom:updated>
            <cc:license>http://creativecommons.org/licenses/by/4.0/</cc:license>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/982/1*1fZwj-_Dr09LhetzE99TTg.png" /><figcaption>Multi-modal Rag’s Workflow</figcaption></figure><p>Today, we’ll explore how to build a Multimodal Retrieval-Augmented Generation (RAG) system using LlamaIndex in combination with GPT-4 or the new Anthropic Sonnet model. But first, what exactly is a Multimodal RAG?</p><p>Multimodal Retrieval-Augmented Generation (RAG) is an advanced AI framework that integrates multiple data modalities — such as text, images, or audio — with retrieval-augmented generation. This approach enhances the generation of responses by incorporating relevant information retrieved from external sources or databases, like a vector database in our case.</p><h3>In a Multimodal RAG system:</h3><ol><li><strong>Retrieval</strong>: The model retrieves pertinent information from external databases or knowledge sources based on a given query. This retrieval process can involve various types of data, such as images, text, or other formats.</li><li><strong>Augmented Generation</strong>: The model then uses this retrieved information to enrich and improve the generation of responses. It synthesizes the information to produce coherent and contextually relevant outputs across multiple modalities.</li></ol><p>For instance, a multimodal RAG model might respond to a text query by pulling relevant images or audio clips from a database and integrating them into the final output. This approach results in richer, more informative responses that leverage different data types.</p><p><strong>New to the World of RAG?</strong></p><p>If you’re new to RAG, I recommend starting with the basics before diving into this advanced topic. Check out the article “<a href="https://medium.com/gitconnected/frontend-rag-mastery-deploying-with-aws-github-and-legacy-integration-3cdd875f5c5e"><strong>RAG: From Theory to Production</strong></a>” where you’ll find comprehensive resources, from the theoretical foundations of RAG to deploying a fully functional RAG server in production.</p><p><strong>Implementing the RAG System with LlamaIndex</strong></p><p>So, how do we go about implementing this RAG system? We’ll be using LlamaIndex. I found a foundational example from Jerry Liu, which I’ve adapted and expanded for this tutorial. Here’s what we will cover:</p><ol><li><strong>Retrieving the Page and Document Name</strong>: We’ll start by setting up the system to identify where the LLM (Large Language Model) will locate the information necessary to answer our queries. Essentially, we’ll pinpoint that the response was found on page X of document Y.</li><li><strong>Building a Multimodal Query Engine</strong>: From scratch, we’ll construct a Multimodal Query Engine that analyzes and processes queries involving both images and text. This engine will utilize a multimodal model to generate more complex and nuanced responses.</li><li><strong>Creating an Agent with Reranking Capabilities</strong>: Using our Multimodal Query Engine, we’ll build an agent that incorporates reranking. Reranking refers to refining and enhancing the quality of documents or passages retrieved from a knowledge source (such as a database or large corpus) before the model generates the final response.</li></ol><p>Now that we’ve outlined the key points we’ll explore today, it’s time to dive into the code. As we go through each step, I’ll also provide a theoretical explanation of the underlying concepts. The following code is presented in a Medium notebook, and I’ll share a link later so you can interact with it directly.</p><pre>%pip install llama-index llama-parse llama-index-multi-modal-llms-openai git+https://github.com/openai/CLIP.git llama_index.postprocessor.cohere_rerank</pre><p>First of all we need to install some dependencies for our notebook.</p><pre>!wget https://github.com/user-attachments/files/16461058/data.zip -O data.zip<br>!unzip -o data.zip<br>!rm data.zip<br><br>!mkdir files<br><br>!cp ./data/fredde.pdf ./files</pre><p>Next, we need to download the files we want to parse. For our example, we’ve chosen an IKEA product manual. This is an excellent multimodal example because these manuals typically contain little to no text but are rich in images, making it a challenging dataset to process and respond to. Here’s a sample page:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/799/1*uOtB2kJlegFxsWFscrywYQ.png" /><figcaption>Sample Page That Show A Complex Layout To Parse</figcaption></figure><pre>import os<br><br>os.environ[&quot;OPENAI_API_KEY&quot;] = &quot;xxx&quot;<br>os.environ[&quot;LLAMA_CLOUD_API_KEY&quot;] = &quot;xxx&quot;<br>os.environ[&quot;COHERE_API_KEY&quot;] = &quot;xxx&quot;</pre><p>For our sample before start make sure to have those api key’s.</p><pre>from llama_parse import LlamaParse<br><br>parser = LlamaParse(<br>    result_type=&quot;markdown&quot;,<br>    parsing_instruction=&quot;You are given IKEA assembly instruction manuals&quot;,<br>    use_vendor_multimodal_model=True,<br>    vendor_multimodal_model_name=&quot;openai-gpt4o&quot;,<br>    show_progress=True,<br>    verbose=True,<br>    invalidate_cache=True,<br>    do_not_cache=True,<br>    num_workers=8,<br>    # Setting language<br>    language=&quot;en&quot;<br>)</pre><p>Now it’s time to start parsing the files. To do this, we’ll use the <strong>LlamaParse </strong>library. Why choose this library? Because it offers a range of prebuilt features that are perfect for our needs. Here’s how it helps in our example:</p><ul><li><strong>Prompt Instruction for Parsing</strong>: We can instruct the parser that we’re working with an IKEA manual, allowing it to optimize the parsing process accordingly.</li><li><strong>OCR Support</strong>: Since we’re building a Multimodal Query Engine, we need both text and images. LlamaParse makes this easy by providing prebuilt tools for extracting images from the pages and the text within those images.</li><li><strong>Language Support</strong>: If your files are in different languages, you can specify the language to improve the parser’s performance and accuracy.</li></ul><pre>DATA_DIR = &quot;files&quot;<br><br><br>def get_data_files(data_dir=DATA_DIR) -&gt; list[str]:<br>    files = []<br>    for f in os.listdir(data_dir):<br>        fname = os.path.join(data_dir, f)<br>        if os.path.isfile(fname):<br>            files.append(fname)<br>    return files<br><br><br>files = get_data_files()<br><br>print(files[0])</pre><p>This is a small helper to get all the files list.</p><pre>md_json_objs = parser.get_json_result(files)<br>image_dicts = parser.get_images(md_json_objs, download_path=&quot;data_images&quot;)</pre><p>Now it’s time to use our previously instantiated parser to extract both JSON results and images from the document. We’ll also specify the path where the extracted images should be saved. To extract text from images, LlamaParse utilizes OCR technology under the hood:</p><p><strong>OCR </strong>stands for <strong>Optical Character Recognition</strong>. It is a technology that enables the conversion of different types of documents, such as scanned paper documents, PDFs, or images captured by a camera, into machine-readable and editable text.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/800/1*Z-mX5P9cLJNTma2dP14cng.jpeg" /><figcaption>OCR Workflow</figcaption></figure><h3>How OCR Works:</h3><ol><li><strong>Image Acquisition</strong>: OCR begins with acquiring an image that contains text. This could be from a scanner, a photograph, or any other image source.</li><li><strong>Preprocessing</strong>: The image may undergo preprocessing to enhance its quality for better recognition. This can include steps like noise reduction, binarization (converting the image to black and white), deskewing (correcting any tilt), and other image enhancement techniques.</li><li><strong>Text Detection</strong>: The OCR system detects and locates the regions in the image where text is present. In some cases, it also involves segmenting the text into characters, words, or lines.</li><li><strong>Character Recognition</strong>: The core of OCR involves recognizing the individual characters within the detected text regions. This is typically done by comparing the detected shapes of characters to a library of character shapes, or by using machine learning models trained on large datasets of text images.</li><li><strong>Post-processing</strong>: After recognizing the characters, OCR software often applies post-processing techniques to improve accuracy. This might include spell-checking, context analysis, and formatting restoration.</li><li><strong>Output</strong>: The final step is to convert the recognized text into a machine-readable format, such as a plain text file, PDF, Word document, or other editable formats.</li></ol><pre>import re<br>from pathlib import Path<br>import typing as t<br>from llama_index.core.schema import TextNode<br><br><br>def get_page_number(file_name):<br>    &quot;&quot;&quot;Gets page number of images using regex on file names&quot;&quot;&quot;<br>    match = re.search(r&quot;-page-(\d+)\.jpg$&quot;, str(file_name))<br>    if match:<br>        return int(match.group(1))<br>    return 0<br><br><br>def _get_sorted_image_files(image_dir):<br>    &quot;&quot;&quot;Get image files sorted by page.&quot;&quot;&quot;<br>    raw_files = [f for f in list(Path(image_dir).iterdir()) if f.is_file()]<br>    sorted_files = sorted(raw_files, key=get_page_number)<br>    return sorted_files<br><br><br>def get_text_nodes(md_json_objs, image_dir) -&gt; t.List[TextNode]:<br>    &quot;&quot;&quot;Creates nodes from json + images&quot;&quot;&quot;<br><br>    nodes = []<br><br>    for result in md_json_objs:<br>      json_dicts = result[&quot;pages&quot;]<br>      document_name = result[&quot;file_path&quot;].split(&#39;/&#39;)[-1]<br><br>      print(json_dicts)<br><br>      docs = [doc[&quot;md&quot;] for doc in json_dicts]  # extract text<br>      image_files = _get_sorted_image_files(image_dir)  # extract images<br><br>      for idx, doc in enumerate(docs):<br>          # adds both a text node and the corresponding image node (jpg of the page) for each page<br>          node = TextNode(<br>              text=doc,<br>              metadata={&quot;image_path&quot;: str(image_files[idx]), &quot;page_num&quot;: idx + 1, &quot;document_name&quot;: document_name},<br>          )<br>          nodes.append(node)<br><br>    return nodes<br><br><br>text_nodes = get_text_nodes(md_json_objs, &quot;data_images&quot;)</pre><p>Next, we define a series of functions to create nodes from our JSON object and images folder. For each result, a node is created, and each node contains useful metadata that will be utilized later. Here’s a breakdown of the key elements:</p><ol><li><strong>Image Path</strong>: This metadata indicates the location of the actual image file. It allows us to retrieve and reference the image when needed.</li><li><strong>Page Number</strong>: This metadata stores the page number from which the image was extracted. This information will be crucial for referencing and constructing results later in our process.</li><li><strong>Image Information</strong>: Two additional helper fields are included to capture specific details about the images, such as their dimensions or format.</li></ol><p>To provide visibility into the structure of our nodes, we also include a print statement. This will allow you to see how each node is composed, including the metadata and any other relevant details.</p><p>This setup ensures that each node is well-defined and contains all the necessary information for subsequent processing and querying.</p><pre>from llama_index.core import (<br>    VectorStoreIndex,<br>    StorageContext,<br>    load_index_from_storage,<br>    Settings,<br>)<br>from llama_index.embeddings.openai import OpenAIEmbedding<br>from llama_index.llms.openai import OpenAI<br><br>embed_model = OpenAIEmbedding(model=&quot;text-embedding-3-large&quot;)<br>llm = OpenAI(&quot;gpt-4o&quot;)<br><br>Settings.llm = llm<br>Settings.embed_model = embed_model<br><br>if not os.path.exists(&quot;storage_manuals&quot;):<br>    index = VectorStoreIndex(text_nodes, embed_model=embed_model)<br>    index.storage_context.persist(persist_dir=&quot;./storage_manuals&quot;)<br>else:<br>    ctx = StorageContext.from_defaults(persist_dir=&quot;./storage_manuals&quot;)<br>    index = load_index_from_storage(ctx)<br><br>retriever = index.as_retriever()</pre><p>Now that we have our parsed nodes with all the necessary metadata, the next step is to create an index. For our multimodal model, we’re using <strong>GPT-4o</strong> for the generative capabilities and the latest OpenAI text-embedding-3-large for embeddings. These choices ensure high-quality performance in both understanding and generating responses. For detailed guidance on selecting models and embeddings, refer to the earlier articles mentioned in the introduction.</p><p>Once the index is created, we need to instantiate a retriever from it. Typically, you would directly instantiate the query engine. However, in this case, we’re building the query engine from scratch, which requires setting up a lower-level retriever instance. This approach allows for more granular control and customization of the retrieval process, tailored specifically to our needs.</p><p>In summary:</p><ol><li><strong>Create the Index</strong>: Use the chosen model (GPT-4o) and embeddings (text-embedding-3-large) to build the index from our parsed nodes.</li><li><strong>Instantiate the Retriever</strong>: Develop a retriever instance from the index, which will be used to facilitate detailed querying and retrieval in our custom-built query engine.</li></ol><p>This setup ensures that we have both a robust indexing system and a flexible retrieval mechanism to support our multimodal RAG system.</p><pre>from llama_index.core.query_engine import CustomQueryEngine<br>from llama_index.core.retrievers import BaseRetriever<br>from llama_index.multi_modal_llms.openai import OpenAIMultiModal<br>from llama_index.core.schema import NodeWithScore, MetadataMode, QueryBundle<br>from llama_index.core.base.response.schema import Response<br>from llama_index.core.prompts import PromptTemplate<br>from llama_index.core.schema import ImageNode<br><br>from typing import Any, List, Optional, Tuple<br>from llama_index.core.postprocessor.types import BaseNodePostprocessor<br><br>QA_PROMPT_TMPL = &quot;&quot;&quot;\<br>You are a chatbot that will help users to get technical responses about and ikea product manual.<br><br>Below we give parsed text from slides in two different formats, as well as the image.<br><br>We parse the text in both &#39;markdown&#39; mode as well as &#39;raw text&#39; mode. Markdown mode attempts \<br>to convert relevant diagrams into tables, whereas raw text tries to maintain the rough spatial \<br>layout of the text.<br><br>Use the image information first and foremost. ONLY use the text/markdown information <br>if you can&#39;t understand the image.<br><br>When you reply dosen&#39;t send images links, but only text explaination of that.<br><br>Context:<br>---------------------<br>{context_str}<br>---------------------<br><br>Given the context information and not prior knowledge, answer the query using ONLY Context informations, if you dosen&#39;t find the answer in the Context NOT try to answer, reply that you dosen&#39;t know and give a page and document name where the user can find similar response.<br>Give the page&#39;s number and the document name where you find the response based on the Context.<br><br>Query: {query_str}<br>Answer: &quot;&quot;&quot;<br><br>QA_PROMPT = PromptTemplate(QA_PROMPT_TMPL)<br><br>gpt_4o_mm = OpenAIMultiModal(model=&quot;gpt-4o&quot;)<br><br><br>class MultimodalQueryEngine(CustomQueryEngine):<br>    qa_prompt: PromptTemplate<br>    retriever: BaseRetriever<br>    multi_modal_llm: OpenAIMultiModal<br>    node_postprocessors: Optional[List[BaseNodePostprocessor]]<br><br>    def __init__(<br>        self,<br>        qa_prompt: PromptTemplate,<br>        retriever: BaseRetriever,<br>        multi_modal_llm: OpenAIMultiModal,<br>        node_postprocessors: Optional[List[BaseNodePostprocessor]] = [],<br>    ):<br>        super().__init__(<br>            qa_prompt=qa_prompt,<br>            retriever=retriever,<br>            multi_modal_llm=multi_modal_llm,<br>            node_postprocessors=node_postprocessors<br>        )<br><br>    def custom_query(self, query_str: str):<br>        # retrieve most relevant nodes<br>        nodes = self.retriever.retrieve(query_str)<br><br>        for postprocessor in self.node_postprocessors:<br>            nodes = postprocessor.postprocess_nodes(<br>                nodes, query_bundle=QueryBundle(query_str)<br>            )<br><br><br>        # create image nodes from the image associated with those nodes<br>        image_nodes = [<br>            NodeWithScore(node=ImageNode(image_path=n.node.metadata[&quot;image_path&quot;]))<br>            for n in nodes<br>        ]<br><br>        # create context string from parsed markdown text<br>        ctx_str = &quot;\n\n&quot;.join(<br>            [r.node.get_content(metadata_mode=MetadataMode.LLM).strip() for r in nodes]<br>        )<br><br>        # prompt for the LLM<br>        fmt_prompt = self.qa_prompt.format(context_str=ctx_str, query_str=query_str)<br><br>        # use the multimodal LLM to interpret images and generate a response to the prompt<br>        llm_repsonse = self.multi_modal_llm.complete(<br>            prompt=fmt_prompt,<br>            image_documents=[image_node.node for image_node in image_nodes],<br>        )<br>        return Response(<br>            response=str(llm_repsonse),<br>            source_nodes=nodes,<br>            metadata={&quot;text_nodes&quot;: text_nodes, &quot;image_nodes&quot;: image_nodes},<br>        )</pre><p>This is the core of our implementation: building the <strong>MultiModalQuery Engine</strong>. Let’s break down its components and functionality:</p><h3>Core Components:</h3><p><strong>Prompt</strong>: At the heart of our reasoning system is the prompt. In this prompt, we instruct the model to prioritize images as the primary source of information and to avoid responding with URLs. We also specify that the query engine should focus solely on the context of the query, meaning it won’t handle unrelated questions. This approach is particularly useful for QA systems. Additionally, we direct the model to include information about the page number in its responses.</p><p><strong>CustomQueryEngine</strong>: We use a LlamaIndex class called CustomQueryEngine for our query engine. This class requires several arguments:</p><ul><li><strong>Prompt</strong>: The one we defined earlier, which guides the model’s responses.</li><li><strong>Retriever</strong>: The storage system we’ve set up to handle our indexed data.</li><li><strong>Multimodal Model</strong>: The model used for processing both text and images.</li><li><strong>Node Postprocessors</strong>: These are used to refine the retrieved nodes. A detailed list of available node postprocessors can be found on the LlamaIndex website. For this tutorial, we’ll focus on the reranking node postprocessor, which we’ll discuss in detail later.</li></ul><p><strong>Custom Query Function</strong>: After setting up the CustomQueryEngine, we define a custom_query function. This function performs the following steps:</p><ul><li><strong>Retrieve Nodes</strong>: It uses our retriever to fetch a list of nodes based on the query.</li><li><strong>Apply Node Postprocessors</strong>: If any node postprocessors are defined, they are applied to the retrieved nodes to enhance their quality.</li><li><strong>Create Image Nodes and Context String</strong>: We separate the images into image_nodes and compile all text information into a context_string.</li><li><strong>Construct Prompt</strong>: We build the prompt for the LLM by combining our pre-defined prompt with the retrieved nodes.</li><li><strong>Invoke Multimodal LLM</strong>: Finally, we make a call to the multimodal LLM, providing both text and images, and obtain a response.</li></ul><p>This setup ensures that our MultiModalQuery Engine is well-equipped to handle complex queries involving both text and images, delivering precise and contextually relevant responses.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*6rUq_UHvziEKmb5UjZjcqg.png" /><figcaption>Reraning Workflow For Undestand Better</figcaption></figure><p>Reranking in the context of <strong>Retrieval-Augmented Generation (RAG)</strong> is an advanced technique used to improve the relevance of the documents or passages that are retrieved before they are passed to the generation model. Here’s how it fits into the RAG workflow:</p><h3>RAG Overview:</h3><ol><li><strong>Query or Input</strong>: The process starts with a query or input prompt.</li><li><strong>Document Retrieval</strong>: The system retrieves a set of potentially relevant documents or passages from a large corpus using a retrieval model (e.g., BM25, Dense Passage Retriever).</li><li><strong>(Optional) Reranking</strong>: The retrieved documents are reranked to better prioritize the most relevant ones.</li><li><strong>Generation</strong>: The top-ranked documents are then used by the generation model (usually a transformer-based model) to produce a final response.</li></ol><h3>Reranking Process in RAG:</h3><ol><li><strong>Initial Retrieval</strong>: When a query is issued, the retriever model identifies a list of documents or passages that are most likely relevant to the query based on some scoring mechanism (like cosine similarity in vector space models).</li><li><strong>Reranking Model</strong>: After the initial retrieval, a reranking model is applied. This model takes the top-N documents (where N might be 10, 20, etc.) and reorders them based on a more nuanced evaluation of their relevance. The reranking model might use additional features, such as:</li></ol><ul><li>Deeper contextual understanding (e.g., semantic relationships).</li><li>Query-document relevance computed with a more sophisticated model, like a cross-encoder that evaluates the query and document together.</li><li>Other signals like metadata, document length, or recency.</li></ul><p><strong>Selection for Generation</strong>: After reranking, the most relevant documents are selected to be fed into the generator model. This helps ensure that the generation model has access to the most pertinent information, leading to better, more accurate responses.</p><h3>Why Reranking is Important in RAG:</h3><ul><li><strong>Improves Response Quality</strong>: By reranking, the system can ensure that the most contextually relevant information is prioritized, which directly impacts the quality of the generated response.</li><li><strong>Handles Ambiguity Better</strong>: Reranking helps in cases where the initial retriever might return documents that are only tangentially related to the query, ensuring that truly relevant documents are used.</li><li><strong>Efficient Use of Resources</strong>: Since generation models can be computationally expensive, reranking ensures that only the best candidates are used, optimizing resource use.</li></ul><pre>from llama_index.postprocessor.cohere_rerank import CohereRerank<br><br>api_key = os.environ[&quot;COHERE_API_KEY&quot;]<br>cohere_rerank = CohereRerank(api_key=api_key, top_n=3, model=&quot;rerank-multilingual-v3.0&quot;)<br><br># Insert reranking here only if after some test it increase the accuracy<br>query_engine = MultimodalQueryEngine(<br>    qa_prompt=QA_PROMPT,<br>    retriever=index.as_retriever(similarity_top_k=9),<br>    multi_modal_llm=gpt_4o_mm,<br>    node_postprocessors=[]<br>)</pre><p>Now it’s time to instantiate our <strong>MultimodalQueryEngine</strong> and prepare the node postprocessor for reranking. Although we won’t be using the reranker for our initial tests, it’s important to understand its role and configuration.</p><h3>Setting Up the MultimodalQueryEngine:</h3><ol><li><strong>Instantiate MultimodalQueryEngine</strong>: Create an instance of the MultimodalQueryEngine using the previously defined components: the prompt, retriever, multimodal model, and any necessary node postprocessors.</li><li><strong>Prepare Node Postprocessor for Reranking</strong>: Even though we won’t utilize the reranker in our test, it’s useful to configure it for future use. If you decide to implement a reranker, remember to increase the similarity_top_k number. This parameter controls how many of the top nodes are considered by the reranker. As previously discussed, the reranker will ultimately use only a subset of the nodes passed to it, so adjusting this number can impact the quality of your results.</li></ol><h3>Final Step: Testing</h3><p>With the MultimodalQueryEngine and reranking configuration in place, it&#39;s time to perform some tests. This will help you verify that the engine is functioning correctly and producing the desired results. Testing will also give you insights into how well the engine handles queries and integrates text with images.</p><pre>from IPython.display import display, Markdown<br><br>response = query_engine.query(&quot;What parts are included in the Uppspel?&quot;)<br>display(Markdown(str(response)))</pre><p>Response: I don’t have information about the parts included in the Uppspel. However, you can refer to the document “fredde.pdf” on page 2 for tools required and general assembly instructions, and other pages for specific steps and parts used in the assembly process.</p><p>Amazing, it dosen’t know the info but guide us to the page</p><pre>response = query_engine.query(&quot;What does the FREDDE look like?&quot;)<br>display(Markdown(str(response)))</pre><p>Response: The FREDDE desk is a multi-functional desk with a modern design. It features a main desk surface with additional shelves and brackets for extra storage and organization. The desk has a sturdy frame with multiple levels, including a top shelf and side shelves. The design allows for efficient use of space, making it suitable for various activities such as working on a computer, studying, or gaming.</p><p>For more detailed visual instructions on assembling the FREDDE desk, you can refer to the following pages in the “fredde.pdf” document:</p><ul><li>Page 1</li><li>Page 23</li><li>Page 24</li><li>Page 29</li><li>Page 30</li><li>Page 31</li><li>Page 32</li></ul><p>Amazing again, it was able to describe a page and also give us pages where to find more info of a “complete” product.</p><pre>response = query_engine.query(&quot;What should I do if I&#39;m confused with reading the manual?&quot;)<br>display(Markdown(str(response)))</pre><p>Response: If you are confused with reading the manual, you should contact IKEA for assistance. This information can be found on page 2 of the document “fredde.pdf”.</p><h4>Perfect for QA</h4><p>Our system effectively answers queries by providing the exact solution, including the page number and document location for the IKEA contact information.</p><h3>Creating an Agent with Our Query Engine</h3><p>Now, let’s move on to the final part of our article: constructing an <strong>Agent</strong> using our query engine.</p><p>An <strong>RAG Agent</strong> is an advanced AI system that integrates Retrieval-Augmented Generation (RAG) techniques with agent-based frameworks. This combination allows the agent to:</p><ol><li><strong>Retrieve Relevant Information</strong>: The agent can access and extract pertinent information from a large dataset using retrieval techniques. This ensures that the information is accurate and relevant to the query.</li><li><strong>Generate Contextually Accurate Responses</strong>: Based on the retrieved information, the agent can generate responses or take actions that are contextually appropriate. This makes the system highly effective for applications requiring detailed and accurate answers, such as customer support or complex information retrieval tasks.</li></ol><p>By leveraging our custom MultimodalQueryEngine, the RAG Agent can handle both text and image data, providing rich, contextually informed responses.</p><pre>from llama_index.core.agent import FunctionCallingAgentWorker<br>from llama_index.core.tools import QueryEngineTool<br><br>query_engine_tool = QueryEngineTool.from_defaults(<br>    query_engine=query_engine,<br>    name=&quot;query_engine_tool&quot;,<br>    description=&quot;Useful for retrieving specific context from the data. Do NOT select if question asks for a summary of the data.&quot;,<br>)<br>agent = FunctionCallingAgentWorker.from_tools(<br>    [query_engine_tool], llm=llm, verbose=True<br>).as_agent()</pre><p>As you can see, using LlamaIndex simplifies the process significantly. With just a few steps, we can transform our query engine into a powerful tool for real-world applications. Here’s how it works:</p><h4>Building the Query Engine Tool</h4><ol><li><strong>Create </strong><strong>query_engine_tool</strong>: Using the previously defined query_engine, we build a query_engine_tool. This tool encapsulates the query engine&#39;s functionality, allowing us to easily integrate it into various applications.</li><li><strong>Instantiate </strong><strong>FunctionCallingAgentWorker</strong>: With the query_engine_tool in place, we can create a FunctionCallingAgentWorker. This agent utilizes the tool to perform complex tasks, such as executing function calls and generating responses based on queries.</li></ol><h4>Real-World Test</h4><p>Let’s see this setup in action with a real-world test. By applying the FunctionCallingAgentWorker to a practical scenario, we can observe how effectively it handles queries and generates contextually relevant responses using the multimodal capabilities of our query engine.</p><pre>response = agent.chat(<br>    &quot;How do I assemble the Fredde, the first 3 steps?&quot;<br>)<br>display(Markdown(str(response)))</pre><p>Reponse:</p><h4>Step 1</h4><ol><li>Insert the screw (100181) into the hole of the metal frame.</li><li>Tighten the screw.</li><li>Ensure the screw is properly secured and not loose.</li></ol><h4>Step 2</h4><ol><li>Align the metal rod with the frame.</li><li>Insert the rod into the frame.</li><li>Rotate the rod to secure it in place.</li></ol><h4>Step 3</h4><ol><li>Insert 2x part 108430 into the designated slots.</li></ol><p>For more detailed instructions and visual aids, refer to pages 5 and 15 of the document “fredde.pdf”.</p><h3>Conclusion</h3><p>Once again, the results are impressive. While the information may not be 100% accurate, it’s important to remember that much of it is derived from images — an inherently challenging data type to process.</p><blockquote>That wraps up our exploration for today! If you found this article helpful and want to support me, please leave some claps, a comment, and subscribe to boost the article’s visibility on Medium. Your feedback and support are greatly appreciated!</blockquote><p>If you have any questions or need further assistance, feel free to ask in the comments. I’m here to help and hope you found this article useful. For those interested in diving deeper, you can access the full article code here: <a href="https://colab.research.google.com/drive/1PcuVqUQjacMt18p8LwODnjbsXOFMurwa?usp=sharing">Full Article Code</a>.</p><p>Thank you for your time!</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=96705c877dbb" width="1" height="1" alt=""><hr><p><a href="https://levelup.gitconnected.com/exploring-multimodal-rag-with-llamaindex-and-gpt-4-or-the-new-anthropic-sonnet-model-96705c877dbb">Exploring Multimodal RAG with LlamaIndex and GPT-4 or the New Anthropic Sonnet Model</a> was originally published in <a href="https://levelup.gitconnected.com/">Level Up Coding</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Step-by-Step Guide to Building and Deploying Custom Image Classification Models with Amazon…]]></title>
            <description><![CDATA[<div class="medium-feed-item"><p class="medium-feed-image"><a href="https://levelup.gitconnected.com/step-by-step-guide-to-building-and-deploying-custom-image-classification-models-with-amazon-d63549968b5d?source=rss-75f167fb6874------2"><img src="https://cdn-images-1.medium.com/max/1000/1*uyBkfBPi_Ci9VxGnws86HQ.png" width="1000"></a></p><p class="medium-feed-snippet">Image classification is a fundamental task in the field of computer vision. It involves training a computer algorithm to recognize and&#x2026;</p><p class="medium-feed-link"><a href="https://levelup.gitconnected.com/step-by-step-guide-to-building-and-deploying-custom-image-classification-models-with-amazon-d63549968b5d?source=rss-75f167fb6874------2">Continue reading on Level Up Coding »</a></p></div>]]></description>
            <link>https://levelup.gitconnected.com/step-by-step-guide-to-building-and-deploying-custom-image-classification-models-with-amazon-d63549968b5d?source=rss-75f167fb6874------2</link>
            <guid isPermaLink="false">https://medium.com/p/d63549968b5d</guid>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[programming]]></category>
            <category><![CDATA[artificial-intelligence]]></category>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[technology]]></category>
            <dc:creator><![CDATA[Marco Bertelli]]></dc:creator>
            <pubDate>Mon, 10 Jun 2024 19:33:57 GMT</pubDate>
            <atom:updated>2024-06-11T15:46:26.671Z</atom:updated>
        </item>
        <item>
            <title><![CDATA[Frontend RAG Mastery: Deploying with AWS, GitHub, and Legacy Integration]]></title>
            <description><![CDATA[<div class="medium-feed-item"><p class="medium-feed-image"><a href="https://levelup.gitconnected.com/frontend-rag-mastery-deploying-with-aws-github-and-legacy-integration-3cdd875f5c5e?source=rss-75f167fb6874------2"><img src="https://cdn-images-1.medium.com/max/1920/1*Y4IjEMGFS0d_r3b0I-AUBw.jpeg" width="1920"></a></p><p class="medium-feed-snippet">In the final section of our series, we will discuss deploying the frontend to a production-ready environment. The two primary challenges&#x2026;</p><p class="medium-feed-link"><a href="https://levelup.gitconnected.com/frontend-rag-mastery-deploying-with-aws-github-and-legacy-integration-3cdd875f5c5e?source=rss-75f167fb6874------2">Continue reading on Level Up Coding »</a></p></div>]]></description>
            <link>https://levelup.gitconnected.com/frontend-rag-mastery-deploying-with-aws-github-and-legacy-integration-3cdd875f5c5e?source=rss-75f167fb6874------2</link>
            <guid isPermaLink="false">https://medium.com/p/3cdd875f5c5e</guid>
            <category><![CDATA[artificial-intelligence]]></category>
            <category><![CDATA[technology]]></category>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[programming]]></category>
            <dc:creator><![CDATA[Marco Bertelli]]></dc:creator>
            <pubDate>Fri, 19 Apr 2024 18:54:55 GMT</pubDate>
            <atom:updated>2024-04-30T17:46:52.362Z</atom:updated>
        </item>
        <item>
            <title><![CDATA[Deploying a Production-Ready RAG Server: A Comprehensive Guide with LlamaIndex]]></title>
            <link>https://levelup.gitconnected.com/deploying-a-production-ready-rag-server-a-comprehensive-guide-with-llamaindex-dbe57cc960df?source=rss-75f167fb6874------2</link>
            <guid isPermaLink="false">https://medium.com/p/dbe57cc960df</guid>
            <category><![CDATA[programming]]></category>
            <category><![CDATA[artificial-intelligence]]></category>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[technology]]></category>
            <category><![CDATA[data-science]]></category>
            <dc:creator><![CDATA[Marco Bertelli]]></dc:creator>
            <pubDate>Wed, 27 Mar 2024 19:47:38 GMT</pubDate>
            <atom:updated>2024-04-30T17:46:56.762Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/600/1*owg5RPtazedwH8fxpZF_vg.png" /><figcaption>Deploy to production</figcaption></figure><p>Previous Articles:</p><ul><li><strong>Intro</strong>: <a href="https://medium.com/@marco.bertelli/build-a-complete-opensource-llm-rag-qa-chatbot-an-in-depth-journey-introduction-c630b16c330c">here</a></li><li><strong>Choose the Model</strong>: <a href="https://medium.com/@marco.bertelli/build-a-complete-opensource-llm-rag-qa-chatbot-choose-the-model-25968144d7a6">here</a></li><li><strong>Flask Server Setup</strong>: <a href="https://medium.com/@marco.bertelli/build-a-complete-opensource-llm-rag-qa-chatbot-flask-server-dc7938239a6b">here</a></li><li><strong>ChatEngine Construction</strong>: <a href="https://medium.com/@marco.bertelli/build-a-complete-opensource-llm-rag-qa-chatbot-chatengine-b0b7dfab0dda">here</a></li><li><strong>Advanced RAG performance</strong>: <a href="https://medium.com/@marco.bertelli/revolutionizing-chatbot-performance-unleashing-three-potent-strategies-for-rag-enhancement-c1188e395d9d">here</a></li><li><strong>Dynamic Sources</strong>: <a href="https://medium.com/@marco.bertelli/empowering-your-chatbot-unveiling-dynamic-knowledge-sources-with-advanced-integration-e8353e85099c">here</a></li><li><strong>Frontend Construction: </strong><a href="https://medium.com/@marco.bertelli/unveiling-the-power-of-rag-building-an-interactive-chatbot-with-react-a-comprehensive-guide-99c409a5f69a">here</a></li></ul><p><strong>Big News</strong>: Before publishing this update, I’ve ensured that all sample repositories are now updated to the latest version of LlamaIndex!</p><p>In our previous articles, we delved into constructing a fully production-ready chat system using LlamaIndex and Python. If you’ve missed any of those, be sure to catch up before diving into this segment.</p><p>Now, while we’ve successfully developed a robust system, it’s crucial to take the next step: <strong>deployment in the cloud</strong>. A production-ready system isn’t truly operational until it’s deployed and capable of handling real-world traffic. By “production traffic,” we mean that the code, which has so far been tested in our local environment, must perform reliably when accessed by multiple users simultaneously.</p><p>Imagine launching your fantastic Chatbot online, only to have it crash when 20 users try to engage with it concurrently. That scenario is far from ideal. We need a server infrastructure that can gracefully handle numerous user requests simultaneously without compromising performance or stability.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*7N2BUjF9gP06TlNa5-Vs-g.png" /><figcaption>Our Manual managed server</figcaption></figure><p>Fortunately, building a server from scratch is not necessary. In 2024, numerous providers offer a wide range of production-ready configurations. For this series, we’ve chosen Heroku. Why Heroku over other options?</p><p>Well, it’s simple to use and provides a solid, well-managed platform. Additionally, Heroku offers a plethora of plugins with seamless integration, requiring zero code adjustments. In my opinion, <em>this simplicity is crucial at the outset of a project</em>. It allows you to focus on product development rather than server management. Plus, when you’re ready, transitioning away from Heroku is straightforward.</p><p>Let’s get started. First, you’ll need a Heroku account. Once you’ve created an account and logged in, navigate to the top-right corner and click the “New” button:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*GdVL8kbr159uZaQBNkNkkg.png" /><figcaption>The Button position</figcaption></figure><p>After selecting the “New” button, you’ll be prompted to choose a name for your server and the region where it will be hosted. Once you’ve made these selections, it’s time to configure your server.</p><p>Begin by navigating to the “Resources” tab. Here, you’ll choose the size of your server. For demo or development servers, opt for the “<strong>Eco</strong>” or “<strong>Basic</strong>” tiers. These options provide sufficient resources for testing and experimentation.</p><p>However, if you’re setting up a production server, it’s essential to select a higher tier based on the expected user load of your application. Consider factors such as the number of concurrent users and the complexity of your application. Choosing an appropriate tier ensures that your server can handle the demands of production traffic effectively.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*fa_aXJB0D9GoFw3h3Ehkfw.png" /><figcaption>Tiers</figcaption></figure><p>Once you’ve configured the server size, the next step is to set up <strong>Papertrail</strong>.</p><p><strong>Papertrail</strong> allows you to view real-time logs of your application, providing the same debugging experience as your local environment. You can find Papertrail in the “<em>Add-ons</em>” section of the same tab. Simply search for it, and the basic plan, which offers essential features, is free!</p><p>After adding Papertrail, navigate to the “<em>Settings</em>” tab. Here, you’ll set the Python buildpack, which is the prerequisite pack for Python applications.</p><p>Additionally, you’ll find the public URL of your application in this tab. This URL is crucial as it allows you to access and test your server using tools like POSTMAN in a manner similar to your local environment. With this setup, you’ll be able to seamlessly develop and debug your application on Heroku.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*I6YKyCdeInXEyM4NtWSTBw.png" /><figcaption>Buildpack and url</figcaption></figure><p>The Heroku Python buildpack provides a seamless deployment experience for Python web applications. Under the hood, this buildpack utilizes a specific <strong>WSGI (Web Server Gateway Interface)</strong> server.</p><p>But what exactly is a <a href="https://www.fullstackpython.com/wsgi-servers.html">WSGI server</a>? In essence, it’s a framework designed to handle the web server side of the WSGI interface, allowing Python web applications to run smoothly on the internet. A WSGI server comes with a multitude of pre-configured settings, making it easier to serve your code over the web.</p><p>In practical terms, when you deploy your Python application on Heroku using the Python build-pack, it automatically configures and sets up the WSGI server for you. This ensures that your application can handle HTTP requests and interact with web clients effectively.</p><p>For more detailed information on WSGI servers and how they work, you can explore the provided previos link.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/640/1*IxkqZgfe5fu7oFA8MctmCg.png" /><figcaption>WSGI basic workflow</figcaption></figure><p>Now that we’ve configured our Heroku environment, it’s time to deploy our code. But how do we do it? Do we need to download the Heroku CLI and push the code manually every time? Not necessarily.</p><p>Instead, we can set up a Continuous Integration and Continuous Deployment <strong>(CI/CD) pipeline</strong> to automate the deployment process. CI/CD involves a series of steps taken to deliver a new version of software seamlessly.</p><p>Fortunately, most Git online providers offer their own pipeline systems for CI/CD. For this guide, we’ll use one of the most popular ones: <em>GitHub Actions</em>. With GitHub Actions, we can automate the deployment of our code to Heroku every time we push changes to our repository.</p><p>By setting up this CI/CD pipeline, we ensure that our deployment process is efficient, reliable, and consistent, allowing us to focus more on developing our application rather than worrying about manual deployment tasks.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/800/1*HEpPo-STwDCnLm9LfellCQ.png" /><figcaption>Basic CI/CD pipeline workflow</figcaption></figure><p>Let’s begin building our CI/CD pipeline. In your repository, add a folder named .github. Within this folder, create another folder named workflows. Finally, create a file called main.yml inside the workflows folder, and write the following code:</p><pre>name: Deploy<br><br>on:<br>  push:<br>    branches:<br>      - master<br><br>jobs:<br>  heroku-deploy:<br>    runs-on: ubuntu-latest<br>    steps:<br>      - name: Check out repository<br>        uses: actions/checkout@v3<br>      - name: Deploy to Heroku<br>        uses: akhileshns/heroku-deploy@v3.12.12<br>        with:<br>          heroku_api_key: ${{ secrets.HEROKU_API_KEY }}<br>          heroku_app_name: ${{ secrets.HEROKU_APP_NAME }}<br>          heroku_email: marcobert37@gmail.com</pre><p>The name field in our YAML file represents the name of our pipeline. This pipeline will be triggered every time we push changes to the selected branches (in our case, only the master branch). Next, we define some jobs. A job is executed every time the pipeline is triggered. We&#39;ve named our job heroku-deploy, and it runs on the latest version of the Ubuntu operating system.</p><p>To deploy our code to Heroku, we’re using a pre-built pipeline (in this case, akhileshns/heroku-deploy@v3.12.12). We need to pass some environment variables to this pre-built pipeline: our Heroku API key and the name of our Heroku application, which we chose in the earlier steps.</p><p>To set up these environment variables in your GitHub repository, follow these steps:</p><h4><strong>Heroku API_KEY:</strong></h4><ul><li>Go to the Heroku dashboard.</li><li>Click on your profile icon and navigate to the “Account settings”.</li><li>Select the “API Key” section and generate a new API key if you haven’t already.</li></ul><h4>GitHub Secrets:</h4><ul><li>Go to your GitHub repository.</li><li>Navigate to the “Settings” tab.</li><li>In the left-hand menu, select “Secrets” or “Secrets and variables” → “Actions”.</li><li>Click on “New repository secret”.</li><li>Set the name of the secret as HEROKU_API_KEY and paste your Heroku API key as the value.</li><li>Additionally, set another secret named HEROKU_APP_NAME and assign the name of your Heroku application as its value.</li></ul><p>By setting up these secrets in your GitHub repository, the pipeline will be able to access them during execution and deploy your code to Heroku seamlessly.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*yYBMfq91-kLCZH4S2XIrdA.png" /><figcaption>Setted keys</figcaption></figure><p>After setting up the environment variables in our GitHub repository, Heroku needs to know how to start our server once the deployment is complete. To do this, we need to create a file at the root level of our project called Procfile. This file is a Heroku standard that specifies what command to run to start our server after deploying new code.</p><p>Inside the Procfile, add the following line of code:</p><pre>web: gunicorn --preload --max-requests 500 --max-requests-jitter 5 -t 3 --worker-class gthread --timeout 120 index:app</pre><p>Here’s a breakdown of each component:</p><ul><li>web: This indicates the process type. In this case, it&#39;s specifying that this process type will handle incoming web requests.</li><li>gunicorn: This is the command to start the Gunicorn server, which serves as the WSGI server for our application.</li><li>index:app: With this part of the command, we&#39;re telling Gunicorn that our application is located in a file named index.py and that the variable app within that file represents our WSGI application.</li><li>--preload: This flag is crucial for memory usage optimization. It instructs Gunicorn to load the application code and dependencies into memory before starting the worker processes. This helps reduce memory usage by ensuring that all workers share the same memory space.</li><li>--max-requests=500: This parameter tells Gunicorn to automatically restart a worker process after it has served 500 requests. This is a preventive measure against memory leaks and other potential issues that may arise from long-running worker processes.</li><li>--worker-class=gthread: This specifies the class of worker processes to be used by Gunicorn. In this case, gthread is chosen, indicating that Gunicorn should use the threaded worker class. Other available options include sync, eventlet, and gevent. However, in many cases, the gthread option tends to be more stable and reliable, particularly in Python web applications.</li></ul><p>Understanding these configuration options is essential for ensuring the smooth and efficient operation of our application when deployed with Gunicorn on Heroku.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*o4iNDsshEM9u72UR1OR2Wg.jpeg" /><figcaption>The Graphic explanation of our workflow until now</figcaption></figure><p>Now that everything is set up, we’re ready to deploy our application by pushing our changes to the master branch. Let&#39;s see what happens when we do that:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*BL2SrzEfDV6WWJxY6gHMEg.png" /><figcaption>The build fail</figcaption></figure><p><strong>As always in the IT field</strong>, things don’t always go as expected. In this case, our pipeline failed. But why? Well, in the earlier chapters of the series, we opted to use a local embeddings module. While everything worked fine on our local machines without any limits, deploying to Heroku posed a challenge. Heroku imposes constraints on the size of the application, allowing only 500 MB, whereas our embeddings model alone consumed a hefty 2.8 GB!</p><p>So, do we need to switch to using some API for embeddings? Not necessarily. Our goal throughout this series has been to utilize open-source solutions and minimize costs wherever possible. Luckily, some brilliant individuals have developed a solution for this exact problem: <strong>FastEmbed</strong>.</p><h4>But what exactly is FastEmbed?</h4><p>It’s a lightweight library with minimal external dependencies. Unlike other solutions, FastEmbed doesn’t require a GPU and doesn’t entail downloading gigabytes of PyTorch dependencies. Instead, it leverages the power of ONNX Runtime. This makes FastEmbed an ideal choice for serverless runtimes or small servers like those on Heroku.</p><p>Now, you might be wondering, what is ONNX Runtime? ONNX Runtime inference provides faster customer experiences and lower costs by supporting models from various deep learning frameworks like PyTorch and TensorFlow/Keras, as well as classical machine learning libraries such as scikit-learn, LightGBM, and XGBoost. ONNX Runtime is compatible with different hardware, drivers, and operating systems. It achieves optimal performance by utilizing hardware accelerators where available, alongside graph optimizations and transformations.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*U2R_mtWpwRa_S7CNBUrEEA.png" /><figcaption>ONNX Runtime workflow</figcaption></figure><p>Great! With FastEmbed, we can deploy our local embeddings model selected in the previous steps while significantly reducing the amount of space it consumes. However, to make this work seamlessly with LlamaIndex, we need to make some changes to our code. Let’s navigate to the index_manager.py file and update the way we use the embeddings:</p><pre>from llama_index.core import (<br>    VectorStoreIndex,<br>    StorageContext,<br>    ServiceContext,<br>    SimpleDirectoryReader<br>)<br><br>from llama_index.core.indices.loading import load_index_from_storage<br>from utils.vector_database import build_pinecone_vector_store, build_mongo_index<br>from mongodb.index import getExistingLlamaIndexes<br><br>from llama_index.llms.perplexity import Perplexity<br>from llama_index.embeddings.fastembed import FastEmbedEmbedding<br><br>import os<br><br>llm = Perplexity(<br>    api_key=os.getenv(&quot;PERPLEXITY_API_KEY&quot;), model=&quot;mixtral-8x7b-instruct&quot;, temperature=0.2<br>)<br><br>embed_model = FastEmbedEmbedding(model_name=&quot;BAAI/bge-small-en-v1.5&quot;)<br><br>index_store = build_mongo_index()<br>vector_store = build_pinecone_vector_store()<br><br>service_context = ServiceContext.from_defaults(<br>    llm=llm,<br>    embed_model=embed_model,<br>)</pre><p>Indeed, the updated code is elegantly simple, thanks to LlamaIndex’s seamless integration with FastEmbed. This integration provides us with a straightforward interface to utilize the power of FastEmbed without any hassle.</p><p>Now, let’s put our changes to the test by redeploying our application. The best way to do this is by pushing the updated code to the master branch of our repository. This will trigger our CI/CD pipeline once again, allowing us to verify that our changes are successfully deployed to Heroku.</p><p>With just a simple push to the master branch, we can ensure that our application remains up-to-date and fully functional, ready to serve our users with the latest enhancements and optimizations.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*mIlezdTYHL_n1AGKBr4WlQ.png" /><figcaption>Pipeline Work!</figcaption></figure><p>This time, our pipeline worked smoothly. Before pushing any changes, ensure that you have also updated your requirements.txt file with the newly installed libraries. Heroku relies on this file to download the required packages, and any discrepancies can lead to strange dependency errors.</p><p>But are we done yet? Not quite. We still need to take care of two last things. The first, and perhaps the easiest, is to monitor the logs. If everything is functioning correctly, you should see something like this:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*thkxNFosTpODRRrd09yh3A.png" /><figcaption>logs of a correct boot</figcaption></figure><p>As you can observe, all the downloads occur before the workers start due to our utilization of the <strong>preload</strong> flag. This flag, employed to conserve space and memory, ensures that all necessary downloads and setups are completed before the worker processes begin their tasks.</p><p>Another crucial aspect to understand is the concept of workers and how to configure their numbers.</p><blockquote>In the context of Gunicorn, a worker refers to a process responsible for handling incoming requests. The most basic and default type of worker is the synchronous worker. Synchronous workers operate by handling one request at a time, sequentially processing each request before moving on to the next.</blockquote><p>Understanding the behavior and configuration of workers is essential for optimizing the performance and scalability of our application. By appropriately adjusting the number and type of workers, we can effectively manage resource utilization and ensure smooth handling of incoming traffic.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/512/1*zItJAj3I832hyZfOYtSF6w.png" /><figcaption>Worker graphic explanation in the detail</figcaption></figure><p>Before, we set the worker-class flag to gthread and configured the -t 3 flag, allowing each worker to handle up to 3 requests concurrently. Now, you might be wondering how to determine the number of workers needed. By default, Heroku dynamically adjusts the number of workers based on the memory allocation of the chosen instance type. However, you can override this default behavior using an environment variable called WEB_CONCURRENCY.</p><p>The formula to determine the number of requests that can be handled simultaneously is:</p><blockquote>Number of workers * -t number.</blockquote><p>For example, if you have 3 workers and set 3 threads per worker, you can handle 9 requests concurrently.</p><p>With our deployment now complete and many questions and errors resolved, we’re ready for the final step: deploying our frontend to enable a production-ready interface. You can find the code for this part on my GitHub repository <a href="https://github.com/marco-bertelli/rag.flask-start">here</a>.</p><p>If you found this post helpful, please consider sharing it on social networks like LinkedIn, and don’t forget to leave a clap and a comment! Your engagement helps promote the post (yeah medium love the interactions with it’s alghoritms).</p><p>If you have any questions or feedback, feel free to reach out to me on Discord at sl33p1420. Special thanks to James Pham for alerting me about a bug. Your contributions and feedback are greatly appreciated!</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=dbe57cc960df" width="1" height="1" alt=""><hr><p><a href="https://levelup.gitconnected.com/deploying-a-production-ready-rag-server-a-comprehensive-guide-with-llamaindex-dbe57cc960df">Deploying a Production-Ready RAG Server: A Comprehensive Guide with LlamaIndex</a> was originally published in <a href="https://levelup.gitconnected.com/">Level Up Coding</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Unveiling the Power of RAG: Building an Interactive Chatbot with React — A Comprehensive Guide]]></title>
            <description><![CDATA[<div class="medium-feed-item"><p class="medium-feed-image"><a href="https://levelup.gitconnected.com/unveiling-the-power-of-rag-building-an-interactive-chatbot-with-react-a-comprehensive-guide-99c409a5f69a?source=rss-75f167fb6874------2"><img src="https://cdn-images-1.medium.com/max/600/1*lQxVlJoLFDqbl-2pZZVs-g.png" width="600"></a></p><p class="medium-feed-snippet">Previous Articles:</p><p class="medium-feed-link"><a href="https://levelup.gitconnected.com/unveiling-the-power-of-rag-building-an-interactive-chatbot-with-react-a-comprehensive-guide-99c409a5f69a?source=rss-75f167fb6874------2">Continue reading on Level Up Coding »</a></p></div>]]></description>
            <link>https://levelup.gitconnected.com/unveiling-the-power-of-rag-building-an-interactive-chatbot-with-react-a-comprehensive-guide-99c409a5f69a?source=rss-75f167fb6874------2</link>
            <guid isPermaLink="false">https://medium.com/p/99c409a5f69a</guid>
            <category><![CDATA[artificial-intelligence]]></category>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[programming]]></category>
            <category><![CDATA[technology]]></category>
            <category><![CDATA[react]]></category>
            <dc:creator><![CDATA[Marco Bertelli]]></dc:creator>
            <pubDate>Sun, 18 Feb 2024 15:34:52 GMT</pubDate>
            <atom:updated>2024-04-30T17:46:58.916Z</atom:updated>
        </item>
    </channel>
</rss>