<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by David Ben Shabat on Medium]]></title>
        <description><![CDATA[Stories by David Ben Shabat on Medium]]></description>
        <link>https://medium.com/@davidbenshabat?source=rss-afdb10106dcc------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/1*5pmU0Jl_xHKRDa-Xkyrk_Q.png</url>
            <title>Stories by David Ben Shabat on Medium</title>
            <link>https://medium.com/@davidbenshabat?source=rss-afdb10106dcc------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Wed, 18 Nov 2026 03:16:24 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@davidbenshabat/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[Building a GenAI Co-Pilot: The Critical Role of Data Boundaries and Output Accuracy]]></title>
            <link>https://medium.com/@davidbenshabat/building-a-genai-co-pilot-the-critical-role-of-data-boundaries-and-output-accuracy-3f6f51c90c57?source=rss-afdb10106dcc------2</link>
            <guid isPermaLink="false">https://medium.com/p/3f6f51c90c57</guid>
            <category><![CDATA[co-pilot]]></category>
            <category><![CDATA[llm]]></category>
            <category><![CDATA[ai]]></category>
            <category><![CDATA[genai]]></category>
            <category><![CDATA[iac]]></category>
            <dc:creator><![CDATA[David Ben Shabat]]></dc:creator>
            <pubDate>Mon, 09 Sep 2024 04:24:27 GMT</pubDate>
            <atom:updated>2024-09-09T04:34:34.926Z</atom:updated>
            <content:encoded><![CDATA[<h3>Introduction</h3><p>In the era of large-scale cloud automation, developing a Generative AI (GenAI) Co-Pilot powered by Large Language Models (LLMs) has become a game-changer for infrastructure management. These AI-driven systems can optimize operations, streamline decision-making, and even suggest new configurations for cloud environments. However, ensuring that these systems operate within well-defined boundaries and produce accurate, coherent responses is critical. Without proper safeguards, LLMs can generate irrelevant or nonsensical outputs, potentially disrupting critical workflows. In this article, we explore the importance of maintaining data boundaries and ensuring output accuracy when building a GenAI Co-Pilot for cloud infrastructure management.</p><blockquote>The future of cloud infrastructure lies in such AI-driven systems, but the key to success will always be ensuring that the LLMs guiding them remain grounded in well-defined, reliable data and processes.</blockquote><h3>Data Segregation: The Foundation of Responsible AI</h3><p>One of the most significant challenges in developing an LLM-based Co-Pilot is managing data across different stages of the AI lifecycle. Data segregation ensures that sensitive production data does not get mixed with training or validation data. By separating these data types, developers can ensure the Co-Pilot only accesses vetted information, minimizing the risk of errors or data leaks.</p><p>In infrastructure management, tools like <strong>Terraform</strong> and <strong>Ansible</strong> are invaluable for maintaining version-controlled Infrastructure-as-Code (IaC), making it easier to manage and track data boundaries. These practices also prevent your LLM from training on sensitive data, safeguarding compliance and reducing privacy risks.</p><h3>Model and Process Boundaries: Keeping LLMs on Task</h3><p>Large Language Models can sometimes generate unexpected or unrelated content if not properly guided. To ensure that your GenAI Co-Pilot remains focused on cloud infrastructure tasks, it’s vital to establish strong model and process boundaries. This means training the LLM on specific datasets related to cloud provisioning and IaC patterns.</p><p>Open-source tools like <strong>DVC</strong> (<a href="https://github.com/iterative/dvc">Data Version Control</a>) and <a href="https://github.com/iterative/datachain">DataChain</a> can assist in versioning both models and data, helping to control the lifecycle of the AI model itself. By isolating different tasks — such as automating AWS infrastructure versus handling GCP environments — you ensure that the Co-Pilot doesn’t generate off-topic or nonsensical results.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*-zR-RZaSbG9wYUuGmkThmA.gif" /></figure><h3>Ensuring Output Accuracy: No Room for Nonsense</h3><p>LLMs are powerful but can produce nonsensical answers if not carefully guided. Ensuring the accuracy of the Co-Pilot’s outputs requires multiple layers of checks and validation. Techniques like <strong><em>temperature tuning</em></strong> and <strong><em>beam search</em></strong> can help control the variability of the model’s output, ensuring it generates relevant, high-quality responses.</p><blockquote>By isolating different tasks — such as automating AWS infrastructure versus handling GCP environments — you ensure that the Co-Pilot doesn’t generate off-topic or nonsensical results</blockquote><p>Additionally, integrating <strong><em>output validation</em></strong> mechanisms — where the generated content is checked against predefined rules or patterns — can catch errors before they cause disruptions. For example, when generating cloud infrastructure blueprints, you can use <strong>OPA</strong> (<a href="https://blog.openpolicyagent.org/opa-1-0-is-coming-heres-what-you-need-to-know-c8fb0d258368">Open Policy Agent</a>) to validate that the proposed configurations are compatible with the existing cloud services.</p><h3>Compatibility and Feedback Loops: Refining the Co-Pilot</h3><p>The key to building a reliable GenAI Co-Pilot is not just generating accurate outputs but also ensuring compatibility with real-world cloud environments. The Co-Pilot should suggest only those configurations that are known to work seamlessly within your cloud provider’s ecosystem. Creating feedback loops that allow users to validate and refine the model’s outputs helps improve accuracy over time. This <strong>human-in-the-loop</strong> process is vital for reducing errors and ensuring that the Co-Pilot continues to learn from its successes and mistakes.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*7gm89Q4GvxwPF2wxMWYSLQ.gif" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*DjmFwqKjuuMnzum3uil4AQ.gif" /></figure><h3>Conclusion</h3><p>Building a GenAI Co-Pilot for cloud infrastructure management involves much more than just training an LLM. It requires careful attention to data boundaries, model specialization, and output validation to prevent the model from generating irrelevant or nonsensical results. By segregating data, enforcing process boundaries, validating outputs, and ensuring compatibility with real-world cloud environments, developers can create a robust and reliable Co-Pilot that enhances operational efficiency while maintaining high levels of accuracy and security. The future of cloud infrastructure lies in such AI-driven systems, but the key to success will always be ensuring that the LLMs guiding them remain grounded in well-defined, reliable data and processes.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=3f6f51c90c57" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Standardizing Cloud Deployment for RAG | Accelerating AI Adoption with Blueprints]]></title>
            <link>https://medium.com/@davidbenshabat/standardizing-cloud-deployment-for-rag-accelerating-ai-adoption-with-blueprints-20829d5591b1?source=rss-afdb10106dcc------2</link>
            <guid isPermaLink="false">https://medium.com/p/20829d5591b1</guid>
            <category><![CDATA[platform-engineering]]></category>
            <category><![CDATA[infrastructure-as-code]]></category>
            <category><![CDATA[cloud-services]]></category>
            <category><![CDATA[ai]]></category>
            <category><![CDATA[llm]]></category>
            <dc:creator><![CDATA[David Ben Shabat]]></dc:creator>
            <pubDate>Wed, 04 Sep 2024 08:04:05 GMT</pubDate>
            <atom:updated>2024-09-04T12:55:08.306Z</atom:updated>
            <content:encoded><![CDATA[<p>When deploying Retrieval-Augmented Generation (RAG) systems in the cloud, in AWS for instance, selecting and configuring the right resources is essential. These resources provide the infrastructure needed to enable AI-driven solutions, such as vector searches and large-scale data processing, and must be carefully standardized to ensure consistency and efficiency.</p><p><strong>VPC and Security Groups, ALB, OpenSearch, Elastic Container Service (ECS)</strong>, and <strong>Elastic Container Registry (ECR)</strong> are few key cloud resources essential for creating a scalable, secure, and efficient infrastructure.</p><p>Blueprints become a valuable tool in this process, allowing organizations to standardize the deployment of key resources in their RAG workflows. Below, we explore the role of specific AWS services and how Blueprints facilitate their integration in RAG systems.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*CFtW52QSa5NX7DraDfOohA.png" /></figure><h3>VPC and Security Groups</h3><p>Amazon Virtual Private Cloud (VPC) enables organizations to launch resources in a logically isolated network that they can fully control. Security groups act as virtual firewalls to control inbound and outbound traffic to resources within the VPC. Standardizing VPC configurations via Blueprints ensures that all RAG environments are deployed with consistent network settings, enhancing security and reducing misconfigurations.</p><p>For example, by defining Blueprints, you can set up consistent rules for private subnets, internet gateways, and security groups across environments. This approach ensures that OpenSearch, ECS, and other services are protected by secure, predefined rules, which streamline the infrastructure setup and prevent security gaps.</p><h3>Application Load Balancer (ALB) for Scalable RAG Infrastructure</h3><p>In a RAG system, scalability and efficient distribution of traffic are critical, especially when handling multiple user requests or high demand for queries and document uploads. This is where AWS <strong>Application Load Balancer (ALB)</strong> plays a crucial role. ALB is responsible for distributing incoming traffic across multiple targets, such as the servers running the application, ensuring that the system remains responsive even under heavy load.</p><p>Using an ALB in our RAG system helps to:</p><ol><li><strong>Distribute Traffic Efficiently</strong>: ALB routes incoming requests — whether they are document uploads or chatbot queries — to the appropriate server instance, balancing the load across the system.</li><li><strong>Scale Automatically</strong>: When traffic increases, ALB can automatically distribute requests to additional instances, ensuring that the system scales without manual intervention. This is important for AI-powered systems like RAG, which may experience spikes in traffic during periods of high demand.</li><li><strong>Enhance Availability and Fault Tolerance</strong>: By distributing traffic across multiple instances, ALB ensures that the system remains available even if one or more instances fail. This improves the reliability and availability of the RAG system.</li><li><strong>Secure Access</strong>: ALB can be configured to use <strong>SSL/TLS certificates</strong>, ensuring secure communication between users and the application. This is critical when handling sensitive data or running queries over encrypted communication channels.</li></ol><p>By incorporating ALB into the RAG infrastructure, we ensure that the system is not only capable of handling high traffic but also remains secure and fault-tolerant. Standardizing ALB configurations through Blueprints further simplifies the setup process, ensuring that traffic management is consistently optimized across all environments.</p><h3>Amazon OpenSearch as an Embedding Vector Database</h3><p>A key component in a RAG architecture is the embedding vector database, which stores and retrieves document vectors during AI query processes. Amazon OpenSearch is a robust service for this purpose, as it supports scalable indexing and searching of vector embeddings, enabling efficient retrieval of relevant documents based on similarity metrics.</p><p>One essential step in configuring OpenSearch for RAG use cases is the creation of a K-Nearest Neighbors (KNN) index. KNN is crucial for performing similarity searches on vector data, which powers the RAG system’s ability to retrieve documents most relevant to a given query. This is achieved by defining the vector embeddings that represent the content to be searched.</p><p>A typical KNN index configuration looks like this:</p><pre>PUT /rag_index<br>{<br>  &quot;settings&quot;: {<br>    &quot;index&quot;: {<br>      &quot;knn&quot;: true<br>    }<br>  },<br>  &quot;mappings&quot;: {<br>    &quot;properties&quot;: {<br>      &quot;my_vector1&quot;: {<br>        &quot;type&quot;: &quot;knn_vector&quot;,<br>        &quot;dimension&quot;: &lt;num_dimesions_req&gt;<br>      }<br>    }<br>  }<br>}</pre><p>In this example, the index rag_index is created with KNN enabled, and a <strong><em>knn_vector</em></strong> field <strong><em>my_vector1</em></strong> is defined with a the required dimensions (based on the embeddings). The dimension represents the number of values in each vector that will be compared during the search. This structure allows OpenSearch to perform fast and accurate similarity searches, making it a critical step in optimizing RAG deployments.</p><p>Blueprint build-block, not only provides a consistent provisioning of such stores, but also simplify and standardize the setup of these KNN indices in OpenSearch, ensuring that each environment leverages best practices for indexing and search performance.</p><blockquote>One key challenge is that only a few in the organization may have the deep knowledge required to provision such infrastructure, let alone put everything together in response to multiple, high-demand requests.</blockquote><h3>Elastic Container Service (ECS)</h3><p>AWS ECS is a fully managed container orchestration service that simplifies deploying and scaling containerized applications, making it ideal for RAG deployments. By standardizing container deployments, organizations can ensure that the infrastructure is optimized for running AI models and data processing workloads.</p><p>Blueprints can define cluster configurations, service tasks, and auto-scaling policies for ECS, ensuring that each environment is consistently configured to handle RAG workloads efficiently. This reduces the complexity of managing containers and ensures that deployments can seamlessly scale to meet demand.</p><h3>Elastic Container Registry (ECR)</h3><p>Amazon Elastic Container Registry (ECR) is a fully managed Docker container registry that stores container images for deployment on ECS. Standardizing the ECR setup through Blueprints ensures that all teams are using consistent container images in their RAG systems.</p><p>Blueprints can be used to define access policies, versioning strategies, and image tagging conventions for ECR, making it easier to manage container versions and ensuring the integrity of the RAG system’s application images. This approach reduces the chances of errors when deploying new versions of AI models or updating infrastructure components.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*HThPGiDQ2R1etvLdGhk1Jg.png" /><figcaption>Cloud-Infrastructure orchestration for RAG</figcaption></figure><h3>Start experimenting with a “Platform Agnostic Approach”</h3><p>The system is designed to work across different cloud platforms. For this example, we are used <strong>AWS</strong>. The platform agnostic approach allows for flexibility, making it easier to adapt the system for other cloud providers in the future.</p><h4>User Interaction</h4><ul><li><strong>Uploading Files</strong>: The system should provide a simple interface where developers can upload documents. After uploading, the files should transformed into embeddings, chunked (split into smaller sections), and stored in the vector database.</li><li><strong>Chatbot Interaction</strong>: Users interact with a chatbot that retrieves relevant documents based on their query. The chatbot presents the context from those documents to an AI model (like GPT-3.5), which generates a well-informed answer using the information found.</li></ul><h4>Search and Response</h4><p>When the developer submits a question, the system looks through the stored document embeddings and retrieves the most relevant ones. These documents are then used to build a prompt for the <strong>LLM</strong>, which constructs a response based on the relevant context from the retrieved documents.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*BJ7Fd9rmdORTSRCxeeqsjw.png" /><figcaption>Standard RAG Flow</figcaption></figure><h3>Conclusion</h3><p>By using Blueprints to standardize key AWS resources like VPC, Security Groups, ALB, ECS, and ECR, organizations can ensure that their RAG deployments are consistent, secure, and scalable. This approach simplifies the complexity of cloud infrastructure setup, allowing teams to focus on refining their AI models and delivering results more efficiently.</p><p>For companies taking their first steps into AI, this standardized approach to cloud deployment provides a critical framework for rapid adoption, allowing them to confidently build and scale RAG systems. <strong>One key challenge is that only a few in the organization may have the deep knowledge required to provision such infrastructure, let alone put everything together in response to multiple, high-demand requests</strong>. Blueprints are especially valuable in this context, as they abstract and bridge the complexity gap, making it easier for teams to consistently deploy infrastructure without needing deep expertise. <strong>This reusability and consistency ensure that teams can rapidly scale RAG systems to meet demand while maintaining reliability and best practices</strong>.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=20829d5591b1" width="1" height="1" alt="">]]></content:encoded>
        </item>
    </channel>
</rss>