<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Shrishs on Medium]]></title>
        <description><![CDATA[Stories by Shrishs on Medium]]></description>
        <link>https://medium.com/@shrishs?source=rss-748916ab05a5------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/1*7LRIcXoLD80XJKESpfkVyA.jpeg</url>
            <title>Stories by Shrishs on Medium</title>
            <link>https://medium.com/@shrishs?source=rss-748916ab05a5------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Tue, 17 Nov 2026 17:45:11 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@shrishs/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[Building a RAG + MCP Powered Customer Intelligence Agent with Llama Stack]]></title>
            <link>https://medium.com/@shrishs/building-a-rag-mcp-powered-customer-intelligence-agent-with-llama-stack-1eef5b5aca9f?source=rss-748916ab05a5------2</link>
            <guid isPermaLink="false">https://medium.com/p/1eef5b5aca9f</guid>
            <category><![CDATA[agentic-ai]]></category>
            <category><![CDATA[agentic-rag]]></category>
            <category><![CDATA[llama-stack]]></category>
            <category><![CDATA[red-hat-openshift-ai]]></category>
            <category><![CDATA[openshift]]></category>
            <dc:creator><![CDATA[Shrishs]]></dc:creator>
            <pubDate>Mon, 23 Feb 2026 22:02:00 GMT</pubDate>
            <atom:updated>2026-02-23T22:02:00.145Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*s_tfQChvbewuQby6RZa1og.png" /></figure><p>In my previous article, <em>“</em><a href="https://medium.com/@shrishs/building-agentic-ai-on-openshift-from-chat-completions-to-the-responses-api-with-llama-stack-3f53c228a57f"><em>Building Agentic AI on OpenShift: From Chat Completions to the Responses API with Llama Stack</em></a><em>”</em>, I walked through setting up Llama Stack and explored how the Responses API enables tool orchestration for agentic workflows.</p><p>In this article, I take that foundation further.</p><p>I build a <strong>Customer Intelligence Agent</strong> using Llama Stack, a Milvus-backed vector database, and an MCP server to dynamically enrich customer data. Instead of treating the model as a standalone , we combine retrieval, tool-calling, and external business logic to create a structured, production-style AI workflow.</p><p>The result is a system that can:</p><ul><li>Retrieve customer records from a CSV stored in a vector store</li><li>Invoke an external MCP service to determine customer tier (e.g., Gold status)</li><li>Merge and return structured output in real time</li></ul><h4>Integrating an Embedding Service into Llama Stack for RAG</h4><p>For Retrieval-Augmented Generation (RAG), the model does not directly search raw CSV files or documents. Instead, the data must first be converted into <strong>vector embeddings</strong> — numerical representations of text that capture semantic meaning.</p><p>To enable this, we deploy a dedicated <strong>embedding inference service</strong> and integrate it with Llama Stack. This service is responsible for:</p><ul><li>Generating embeddings for documents during ingestion</li><li>Generating embeddings for user queries at runtime</li><li>Enabling semantic similarity search within the vector store (Milvus)</li></ul><p>Once integrated, Llama Stack uses this embedding service automatically when creating or querying vector stores.</p><p>Importantly, embedding generation does <strong>not</strong> require an expensive GPU. Lightweight embedding models such as BGE-small or similar transformer-based models can efficiently run on CPU with sufficient performance for most enterprise RAG workloads. Since embedding computation is not latency-critical in the same way as large language model inference, CPU-based deployment significantly reduces infrastructure cost without impacting system effectiveness.</p><ul><li>Create a inference service for the embedding service.</li></ul><pre>apiVersion: serving.kserve.io/v1beta1<br>kind: InferenceService<br>metadata:<br>  name: embedding-service<br>  annotations:<br>    serving.kserve.io/deploymentMode: RawDeployment<br>    security.opendatahub.io/enable-auth: &quot;false&quot;<br>  labels:<br>    networking.kserve.io/visibility: exposed<br>    opendatahub.io/dashboard: &quot;true&quot;<br>spec:<br>  predictor:<br>    minReplicas: 1<br>    maxReplicas: 1<br>    automountServiceAccountToken: false<br>    containers:<br>      - name: tei<br>        image: ghcr.io/huggingface/text-embeddings-inference:cpu-1.8.1<br>        args:<br>          - &quot;--model-id=BAAI/bge-small-en-v1.5&quot;<br>          - &quot;--port=3000&quot;<br>          - &quot;--auto-truncate&quot;<br>        ports:<br>          - containerPort: 3000<br>        env:<br>          # Force all caches into a writable path<br>          - name: HOME<br>            value: /tmp<br>          - name: HF_HOME<br>            value: /tmp/hf<br>          - name: HUGGINGFACE_HUB_CACHE<br>            value: /tmp/hf/hub<br>          - name: TRANSFORMERS_CACHE<br>            value: /tmp/hf/transformers<br>        volumeMounts:<br>          - name: tmp<br>            mountPath: /tmp<br>        resources:<br>          requests:<br>            cpu: &quot;2&quot;<br>            memory: 4Gi<br>          limits:<br>            cpu: &quot;4&quot;<br>            memory: 8Gi<br>    volumes:<br>      - name: tmp<br>        emptyDir: {}</pre><ul><li>Set up the following parameter.</li></ul><pre>export INFERENCE_MODEL=&quot;redhataillama-31-8b-instruct&quot;<br>export VLLM_URL=&quot;http://redhataillama-31-8b-instruct-predictor.demo-ai.svc.cluster.local:8080/v1&quot;<br>export VLLM_TLS_VERIFY=&quot;false&quot;  <br>export VLLM_API_TOKEN=&quot;NA&quot;<br>export VLLM_MAX_TOKENS=16384<br><br>export EMBEDDING_MODEL=&quot;BAAI/bge-small-en-v1.5&quot;<br>export EMBEDDING_PROVIDER_MODEL_ID=&quot;BAAI/bge-small-en-v1.5&quot;<br>export VLLM_EMBEDDING_URL=&quot;http://embedding-service-predictor.demo-ai.svc.cluster.local:3000/v1&quot;<br>export VLLM_EMBEDDING_API_TOKEN=&quot;NA&quot;<br>export VLLM_EMBEDDING_MAX_TOKENS=512<br>export VLLM_EMBEDDING_TLS_VERIFY=&quot;true&quot;</pre><pre>oc create secret generic llama-stack-secret -n demo-ai \<br>  --from-literal=INFERENCE_MODEL=&quot;$INFERENCE_MODEL&quot; \<br>  --from-literal=VLLM_URL=&quot;$VLLM_URL&quot; \<br>  --from-literal=VLLM_TLS_VERIFY=&quot;$VLLM_TLS_VERIFY&quot; \<br>  --from-literal=VLLM_API_TOKEN=&quot;$VLLM_API_TOKEN&quot; \<br>  --from-literal=VLLM_MAX_TOKENS=&quot;$VLLM_MAX_TOKENS&quot; \<br>  --from-literal=EMBEDDING_MODEL=&quot;$EMBEDDING_MODEL&quot; \<br>  --from-literal=EMBEDDING_PROVIDER_MODEL_ID=&quot;$EMBEDDING_PROVIDER_MODEL_ID&quot; \<br>  --from-literal=VLLM_EMBEDDING_URL=&quot;$VLLM_EMBEDDING_URL&quot; \<br>  --from-literal=VLLM_EMBEDDING_TLS_VERIFY=&quot;$VLLM_EMBEDDING_TLS_VERIFY&quot; \<br>  --from-literal=VLLM_EMBEDDING_API_TOKEN=&quot;$VLLM_EMBEDDING_API_TOKEN&quot; \<br>  --from-literal=VLLM_EMBEDDING_MAX_TOKENS=&quot;$VLLM_EMBEDDING_MAX_TOKENS&quot;</pre><ul><li>Create the Llama Stack Distribution.</li></ul><pre>apiVersion: llamastack.io/v1alpha1<br>kind: LlamaStackDistribution<br>metadata:<br>  name: lsd-llama-milvus-inline<br>  namespace: demo-ai<br>spec:<br>  replicas: 1<br>  server:<br>    containerSpec:<br>      resources:<br>        requests:<br>          cpu: &quot;250m&quot;<br>          memory: &quot;500Mi&quot;<br>        limits:<br>          cpu: 4<br>          memory: &quot;12Gi&quot;<br>      env:<br>        # PostgreSQL metadata store (required in {productname-short} 3.2)<br>        - name: POSTGRES_HOST<br>          value: example-primary.demo-ai.svc<br>        - name: POSTGRES_PORT<br>          value: &quot;5432&quot;<br>        - name: POSTGRES_DB<br>          value: example<br>        - name: POSTGRES_USER<br>          value: example<br>        - name: POSTGRES_PASSWORD<br>          valueFrom:<br>            secretKeyRef:<br>              name: example-pguser-example<br>              key: password<br><br>        # Remote LLM configuration<br>        - name: INFERENCE_MODEL<br>          valueFrom:<br>            secretKeyRef:<br>              name: llama-stack-secret<br>              key: INFERENCE_MODEL<br>        - name: VLLM_URL<br>          valueFrom:<br>            secretKeyRef:<br>              name: llama-stack-secret<br>              key: VLLM_URL<br>        - name: VLLM_TLS_VERIFY<br>          valueFrom:<br>            secretKeyRef:<br>              name: llama-stack-secret<br>              key: VLLM_TLS_VERIFY<br>        - name: VLLM_API_TOKEN<br>          valueFrom:<br>            secretKeyRef:<br>              name: llama-stack-secret<br>              key: VLLM_API_TOKEN<br>        - name: VLLM_MAX_TOKENS<br>          valueFrom:<br>            secretKeyRef:<br>              name: llama-stack-secret<br>              key: VLLM_MAX_TOKENS<br><br>        # Remote embedding configuration<br>        - name: EMBEDDING_MODEL<br>          valueFrom:<br>            secretKeyRef:<br>              name: llama-stack-secret<br>              key: EMBEDDING_MODEL<br>        - name: EMBEDDING_PROVIDER_MODEL_ID<br>          valueFrom:<br>            secretKeyRef:<br>              name: llama-stack-secret<br>              key: EMBEDDING_PROVIDER_MODEL_ID<br>        - name: VLLM_EMBEDDING_URL<br>          valueFrom:<br>            secretKeyRef:<br>              name: llama-stack-secret<br>              key: VLLM_EMBEDDING_URL<br>        - name: VLLM_EMBEDDING_TLS_VERIFY<br>          valueFrom:<br>            secretKeyRef:<br>              name: llama-stack-secret<br>              key: VLLM_EMBEDDING_TLS_VERIFY<br>        - name: VLLM_EMBEDDING_API_TOKEN<br>          valueFrom:<br>            secretKeyRef:<br>              name: llama-stack-secret<br>              key: VLLM_EMBEDDING_API_TOKEN<br>        - name: VLLM_EMBEDDING_MAX_TOKENS<br>          valueFrom:<br>            secretKeyRef:<br>              name: llama-stack-secret<br>              key: VLLM_EMBEDDING_MAX_TOKENS<br><br>        - name: FMS_ORCHESTRATOR_URL<br>          value: &quot;http://localhost&quot;<br>      name: llama-stack<br>      port: 8321<br>    distribution:<br>      name: rh-dev<br>    storage:<br>      size: 5Gi</pre><ul><li>Verify the Llama Stack Deployment if both the embedding model and the LLM model are correctly registered.</li></ul><pre>curl https://$LLAMA_STACK_URL/v1/models |jq<br>{<br>  &quot;data&quot;: [<br>    {<br>      &quot;identifier&quot;: &quot;vllm-embedding/BAAI/bge-small-en-v1.5&quot;,<br>      &quot;provider_resource_id&quot;: &quot;BAAI/bge-small-en-v1.5&quot;,<br>      &quot;provider_id&quot;: &quot;vllm-embedding&quot;,<br>      &quot;type&quot;: &quot;model&quot;,<br>      &quot;metadata&quot;: {<br>        &quot;embedding_dimension&quot;: 768<br>      },<br>      &quot;model_type&quot;: &quot;embedding&quot;<br>    },<br>    {<br>      &quot;identifier&quot;: &quot;vllm-inference/redhataillama-31-8b-instruct&quot;,<br>      &quot;provider_resource_id&quot;: &quot;redhataillama-31-8b-instruct&quot;,<br>      &quot;provider_id&quot;: &quot;vllm-inference&quot;,<br>      &quot;type&quot;: &quot;model&quot;,<br>      &quot;metadata&quot;: {},<br>      &quot;model_type&quot;: &quot;llm&quot;<br>    }<br>  ]<br>}</pre><pre>curl https://$LLAMA_STACK_URL/v1/providers|jq<br>  &quot;data&quot;: [<br>    {<br>      &quot;api&quot;: &quot;inference&quot;,<br>      &quot;provider_id&quot;: &quot;vllm-inference&quot;,<br>      &quot;provider_type&quot;: &quot;remote::vllm&quot;,<br>      &quot;config&quot;: {<br>        &quot;url&quot;: &quot;http://redhataillama-31-8b-instruct-predictor.demo-ai.svc.cluster.local:8080/v1&quot;,<br>        &quot;max_tokens&quot;: 16384,<br>        &quot;api_token&quot;: &quot;********&quot;,<br>        &quot;tls_verify&quot;: false<br>      },<br>      &quot;health&quot;: {<br>        &quot;status&quot;: &quot;Not Implemented&quot;,<br>        &quot;message&quot;: &quot;Provider does not implement health check&quot;<br>      }<br>    },<br>    {<br>      &quot;api&quot;: &quot;inference&quot;,<br>      &quot;provider_id&quot;: &quot;vllm-embedding&quot;,<br>      &quot;provider_type&quot;: &quot;remote::vllm&quot;,<br>      &quot;config&quot;: {<br>        &quot;url&quot;: &quot;http://embedding-service-predictor.demo-ai.svc.cluster.local:3000/v1&quot;,<br>        &quot;max_tokens&quot;: 512,<br>        &quot;api_token&quot;: &quot;********&quot;,<br>        &quot;tls_verify&quot;: true<br>      },<br>      &quot;health&quot;: {<br>        &quot;status&quot;: &quot;Not Implemented&quot;,<br>        &quot;message&quot;: &quot;Provider does not implement health check&quot;<br>      }<br>    },<br>    {<br>      &quot;api&quot;: &quot;vector_io&quot;,<br>      &quot;provider_id&quot;: &quot;milvus&quot;,<br>      &quot;provider_type&quot;: &quot;inline::milvus&quot;,<br>      &quot;config&quot;: {<br>        &quot;db_path&quot;: &quot;/opt/app-root/src/.llama/distributions/rh/milvus.db&quot;,<br>        &quot;persistence&quot;: {<br>          &quot;backend&quot;: &quot;kv_default&quot;,<br>          &quot;namespace&quot;: &quot;vector_io::milvus&quot;<br>        }<br>      },<br>      &quot;health&quot;: {<br>        &quot;status&quot;: &quot;Not Implemented&quot;,<br>        &quot;message&quot;: &quot;Provider does not implement health check&quot;<br>      }</pre><h4>Preparing and Ingesting Customer Data into the Vector Store</h4><ul><li>Create a customer.csv file</li></ul><pre>customer_id,name,email,status,country<br>30,John Doe,john.doe@example.com,Germany<br>31,Jane Smith,jane.smith@example.com,France<br>32,Michael Brown,michael.brown@example.com,USA<br>33,Emily Davis,emily.davis@example.com,Canada<br>34,Daniel Wilson,daniel.wilson@example.com,Australia<br>35,Sophia Taylor,sophia.taylor@example.com,UK<br>36,James Anderson,james.anderson@example.com,Germany<br>37,Olivia Thomas,olivia.thomas@example.com,Italy<br>38,William Jackson,william.jackson@example.com,Spain<br>39,Ava White,ava.white@example.com,Netherlands</pre><ul><li>Creating a Vector Store and Ingesting Customer Data.</li></ul><pre>vector_store = client.vector_stores.create(<br>    name=&quot;customer_record_1&quot;,<br>    extra_body={<br>        &quot;embedding_model&quot;: embedding_model_id,<br>        &quot;embedding_dimension&quot;: embedding_dimension,<br>        &quot;provider_id&quot;: &quot;milvus&quot;,<br>    },<br>)<br>vector_store_id = vector_store.id<br><br>import requests<br><br>filename = &quot;customers.csv&quot;<br><br>with open(filename, &quot;rb&quot;) as f:<br>    file_info = client.files.create(<br>        file=(filename, f),<br>        purpose=&quot;assistants&quot;,<br>    )<br><br>vector_store_file = client.vector_stores.files.create(<br>    vector_store_id=vector_store_id,<br>    file_id=file_info.id,<br>    chunking_strategy={<br>        &quot;type&quot;: &quot;static&quot;,<br>        &quot;static&quot;: {<br>            &quot;max_chunk_size_tokens&quot;: 800,<br>            &quot;chunk_overlap_tokens&quot;: 400,<br>        },<br>    },<br>)<br>print(vector_store_file)</pre><ul><li>Verify if the vector store is created.</li></ul><pre>curl https://$LLAMA_STACK_URL/v1/vector_stores |jq<br>{<br>  &quot;object&quot;: &quot;list&quot;,<br>  &quot;data&quot;: [<br>    {<br>      &quot;id&quot;: &quot;vs_86919b78-44d1-44f0-a2b2-6843a91bb6ae&quot;,<br>      &quot;object&quot;: &quot;vector_store&quot;,<br>      &quot;created_at&quot;: 1771755235,<br>      &quot;name&quot;: &quot;customer_record_1&quot;,<br>      &quot;usage_bytes&quot;: 0,<br>      &quot;file_counts&quot;: {<br>        &quot;completed&quot;: 1,<br>        &quot;cancelled&quot;: 0,<br>        &quot;failed&quot;: 0,<br>        &quot;in_progress&quot;: 0,<br>        &quot;total&quot;: 1<br>      },<br>      &quot;status&quot;: &quot;completed&quot;,<br>      &quot;expires_after&quot;: null,<br>      &quot;expires_at&quot;: null,<br>      &quot;last_active_at&quot;: 1771755235,<br>      &quot;metadata&quot;: {<br>        &quot;provider_id&quot;: &quot;milvus&quot;,<br>        &quot;provider_vector_store_id&quot;: &quot;vs_86919b78-44d1-44f0-a2b2-6843a91bb6ae&quot;<br>      }<br>    },<br>  ],<br>  &quot;first_id&quot;: &quot;vs_86919b78-44d1-44f0-a2b2-6843a91bb6ae&quot;,<br>  &quot;last_id&quot;: &quot;vs_6f28292d-e98c-4d9e-90ac-9cca16e0b704&quot;,<br>  &quot;has_more&quot;: false<br>}</pre><h4>Retrieving and Enriching Customer Data with Tools</h4><ul><li>Retrieve the Vector Store Id.</li></ul><pre>STORE_NAME = &quot;customer_record_1&quot;<br><br>stores_page = client.vector_stores.list()<br>stores = stores_page.data  # list[VectorStore]<br><br>match = next((s for s in stores if getattr(s, &quot;name&quot;, None) == STORE_NAME), None)<br><br>if not match:<br>    raise ValueError(f&quot;Vector store named &#39;{STORE_NAME}&#39; not found. Available: {[getattr(s,&#39;name&#39;,None) for s in stores]}&quot;)<br><br>VECTOR_STORE_ID = match.id<br>print(&quot;VECTOR_STORE_ID =&quot;, VECTOR_STORE_ID)</pre><h4>Combining File Search and MCP Tools in a Single Workflow</h4><p>In this step, we pass both the <strong>file_search tool</strong> (RAG over the vector store) and the <strong>MCP tool</strong> (external customer validation service) in a single responses.create() call.It will</p><ul><li>Retrieve customer records from the vector store.</li><li>For each customer, call the MCP server to check whether they are a GOLD customer.</li><li>Stop once the first GOLD customer is found.</li></ul><pre>system_instructions=&quot;&quot;&quot;<br>You are a Customer assistant.<br>You will be first get customer records from file search tool.<br>- call the MCP server tool to check whether the customer is a GOLD customer.<br>Rules:<br>- Call MCP server with the customer id <br>- once you find first gold customer ,stop calling MCP Server.<br>&quot;&quot;&quot;<br><br><br><br>query = &quot;Get me the first gold customer records&quot;<br><br><br>response = client.responses.create(<br>    model=model_id,<br>    input=query,<br>    instructions=system_instructions,<br>    tools=[   <br>        {<br>            &quot;type&quot;: &quot;file_search&quot;,<br>            &quot;vector_store_ids&quot;: [VECTOR_STORE_ID],<br>        },             <br>        {<br>            &quot;type&quot;: &quot;mcp&quot;,<br>            &quot;server_url&quot;: mcp_customer_endpoint,<br>            &quot;server_label&quot;: &quot;Customer-MCP-Server&quot;<br>        },<br>    ],<br>)<br><br>print(response.output_text)</pre><pre>INFO:httpx:HTTP Request: POST https://$LLAMA_STACK_URL/v1/responses &quot;HTTP/1.1 200 OK&quot;<br>The first gold customer is John Doe with customer id 30.</pre><h4>Rebuilding the Workflow Using the Agent Abstraction</h4><p>While the previous section demonstrated multi-tool orchestration using a single responses.create() call, Llama Stack also provides a higher-level <strong>Agent abstraction</strong>.</p><p>The Agent model introduces:</p><ul><li>Persistent sessions</li><li>Stateful interactions</li><li>Structured tool management</li><li>More extensible workflow design</li></ul><pre>from llama_stack_client import Agent<br><br>system_instructions=&quot;&quot;&quot;<br>You are a Customer assistant.<br>You will get customer records from file search tool.<br>&quot;&quot;&quot;<br><br># system_instructions = &quot;&quot;&quot;<br># You are a data retrieval assistant.<br># You MUST use the content returned by the file_search tool.<br># The data is coming from a CSV file stored in a vector store.<br># &quot;&quot;&quot;<br><br><br>query = &quot;Get me the customer records&quot;<br><br>agent = Agent(<br>    model=model_id,<br>    instructions=system_instructions,<br>    client=client,<br>    tools=[   <br>        {<br>            &quot;type&quot;: &quot;file_search&quot;,<br>            &quot;vector_store_ids&quot;: [VECTOR_STORE_ID],<br>        },             <br>        {<br>            &quot;type&quot;: &quot;mcp&quot;,<br>            &quot;server_url&quot;: mcp_customer_endpoint,<br>            &quot;server_label&quot;: &quot;Customer-MCP-Server&quot;<br>        },<br>    ],<br>)<br><br><br>import uuid<br>session_id = agent.create_session(f&quot;nps_session-{uuid.uuid4().hex}&quot;)<br><br>## Step1<br><br>agent_response1 = agent.create_turn(<br>    messages=[{&quot;role&quot;: &quot;user&quot;, &quot;content&quot;: query}],<br>    session_id=session_id,<br>    stream=False,<br>)<br><br>print(agent_response1)<br><br>## Step 2<br><br>query = &quot;which one is gold customer&quot;<br><br>agent_response2 = agent.create_turn(<br>    messages=[{&quot;role&quot;: &quot;user&quot;, &quot;content&quot;: query}], # &quot;Which of these is happening the soonest?&quot;<br>    session_id=session_id,<br>    stream=False,<br>)<br><br>print(agent_response2)</pre><pre>INFO:httpx:HTTP Request: POST https://$LLAMA_STACK_URL/v1/conversations &quot;HTTP/1.1 200 OK&quot;<br>INFO:httpx:HTTP Request: POST https://$LLAMA_STACK_URL/v1/responses &quot;HTTP/1.1 200 OK&quot;<br>INFO:httpx:HTTP Request: POST https://$LLAMA_STACK_URL/v1/responses &quot;HTTP/1.1 200 OK&quot;<br>ResponseObject(id=&#39;resp_ce4a036a-c8a5-459a-b479-fae3935d6a74&#39;, created_at=1771760853, model=&#39;vllm-inference/redhataillama-31-8b-instruct&#39;, object=&#39;response&#39;, output=[OutputOpenAIResponseOutputMessageMcpListTools(id=&#39;mcp_list_12ae32c5-0a3c-4ec7-ba7d-c3c0699dc3af&#39;, server_label=&#39;Customer-MCP-Server&#39;, tools=[OutputOpenAIResponseOutputMessageMcpListToolsTool(input_schema={&#39;properties&#39;: {&#39;customer_id&#39;: {&#39;type&#39;: &#39;string&#39;}}, &#39;required&#39;: [&#39;customer_id&#39;], &#39;type&#39;: &#39;object&#39;}, name=&#39;get_customer_details&#39;, description=None)], type=&#39;mcp_list_tools&#39;), OutputOpenAIResponseOutputMessageMcpCall(id=&#39;fc_2355f054-6e48-478f-83ab-2161d32056b8&#39;, arguments=&#39;{&quot;customer_id&quot;: &quot;customer records&quot;}&#39;, name=&#39;get_customer_details&#39;, server_label=&#39;Customer-MCP-Server&#39;, type=&#39;mcp_call&#39;, error=None, output=&#39;{&quot;customer_id&quot;:&quot;customer records&quot;,&quot;name&quot;:&quot;John Doe&quot;,&quot;email&quot;:&quot;john.doe@example.com&quot;,&quot;status&quot;:&quot;Gold&quot;,&quot;country&quot;:&quot;Germany&quot;}&#39;), OutputOpenAIResponseOutputMessageFileSearchToolCall(id=&#39;fc_e263888e-8780-47cf-a02d-f0568c9a6999&#39;, queries=[&#39;customer records&#39;], status=&#39;completed&#39;, type=&#39;file_search_call&#39;, results=[OutputOpenAIResponseOutputMessageFileSearchToolCallResult(attributes={}, file_id=&#39;file-bdb2e101ae774acf86f6aafd03e35454&#39;, filename=&#39;file-bdb2e101ae774acf86f6aafd03e35454&#39;, score=0.6902071237564087, text=&#39;customer_id,name,email,status,country\n30,John Doe,john.doe@example.com,Germany\n31,Jane Smith,jane.smith@example.com,France\n32,Michael Brown,michael.brown@example.com,USA\n33,Emily Davis,emily.davis@example.com,Canada\n34,Daniel Wilson,daniel.wilson@example.com,Australia\n35,Sophia Taylor,sophia.taylor@example.com,UK\n36,James Anderson,james.anderson@example.com,Germany\n37,Olivia Thomas,olivia.thomas@example.com,Italy\n38,William Jackson,william.jackson@example.com,Spain\n39,Ava White,ava.white@example.com,Netherlands&#39;)]), OutputOpenAIResponseOutputMessageFileSearchToolCall(id=&#39;fc_3f0e6011-d590-4cda-b896-6d221d751b54&#39;, queries=[&#39;customer records file search tool&#39;], status=&#39;completed&#39;, type=&#39;file_search_call&#39;, results=[OutputOpenAIResponseOutputMessageFileSearchToolCallResult(attributes={}, file_id=&#39;file-bdb2e101ae774acf86f6aafd03e35454&#39;, filename=&#39;file-bdb2e101ae774acf86f6aafd03e35454&#39;, score=0.6301706433296204, text=&#39;customer_id,name,email,status,country\n30,John Doe,john.doe@example.com,Germany\n31,Jane Smith,jane.smith@example.com,France\n32,Michael Brown,michael.brown@example.com,USA\n33,Emily Davis,emily.davis@example.com,Canada\n34,Daniel Wilson,daniel.wilson@example.com,Australia\n35,Sophia Taylor,sophia.taylor@example.com,UK\n36,James Anderson,james.anderson@example.com,Germany\n37,Olivia Thomas,olivia.thomas@example.com,Italy\n38,William Jackson,william.jackson@example.com,Spain\n39,Ava White,ava.white@example.com,Netherlands&#39;)]), OutputOpenAIResponseOutputMessageFileSearchToolCall(id=&#39;fc_2864543b-920d-487c-97a1-ba8e768454bc&#39;, queries=[&#39;customer records from file search tool&#39;], status=&#39;completed&#39;, type=&#39;file_search_call&#39;, results=[OutputOpenAIResponseOutputMessageFileSearchToolCallResult(attributes={}, file_id=&#39;file-bdb2e101ae774acf86f6aafd03e35454&#39;, filename=&#39;file-bdb2e101ae774acf86f6aafd03e35454&#39;, score=0.6196300387382507, text=&#39;customer_id,name,email,status,country\n30,John Doe,john.doe@example.com,Germany\n31,Jane Smith,jane.smith@example.com,France\n32,Michael Brown,michael.brown@example.com,USA\n33,Emily Davis,emily.davis@example.com,Canada\n34,Daniel Wilson,daniel.wilson@example.com,Australia\n35,Sophia Taylor,sophia.taylor@example.com,UK\n36,James Anderson,james.anderson@example.com,Germany\n37,Olivia Thomas,olivia.thomas@example.com,Italy\n38,William Jackson,william.jackson@example.com,Spain\n39,Ava White,ava.white@example.com,Netherlands&#39;)]), OutputOpenAIResponseOutputMessageMcpCall(id=&#39;fc_d08e8a09-b68c-4e69-a55e-75a36ec67bb6&#39;, arguments=&#39;{&quot;customer_id&quot;: &quot;customer records from file search tool&quot;}&#39;, name=&#39;get_customer_details&#39;, server_label=&#39;Customer-MCP-Server&#39;, type=&#39;mcp_call&#39;, error=None, output=&#39;{&quot;customer_id&quot;:&quot;customer records from file search tool&quot;,&quot;name&quot;:&quot;John Doe&quot;,&quot;email&quot;:&quot;john.doe@example.com&quot;,&quot;status&quot;:&quot;Gold&quot;,&quot;country&quot;:&quot;Germany&quot;}&#39;), OutputOpenAIResponseMessage(content=[OutputOpenAIResponseMessageContentUnionMember2(annotations=[], text=&#39;The final answer to the user\&#39;s prompt is:\n\n&quot;customer_id: customer records from file search tool\nname: John Doe\nemail: john.doe@example.com\nstatus: Gold\ncountry: Germany&quot;&#39;, type=&#39;output_text&#39;)], role=&#39;assistant&#39;, type=&#39;message&#39;, id=&#39;msg_581a1cf7-cfa8-4d74-90ed-ddefaa25c2ec&#39;, status=&#39;completed&#39;)], parallel_tool_calls=False, status=&#39;completed&#39;, text=Text(format=TextFormat(type=&#39;text&#39;, description=None, name=None, schema_=None, strict=None)), error=None, previous_response_id=None, temperature=None, tools=[ToolOpenAIResponseInputToolFileSearch(type=&#39;file_search&#39;, vector_store_ids=[&#39;vs_86919b78-44d1-44f0-a2b2-6843a91bb6ae&#39;], filters=None, max_num_results=10, ranking_options=None), ToolOpenAIResponseToolMcp(server_label=&#39;Customer-MCP-Server&#39;, type=&#39;mcp&#39;, allowed_tools=None)], top_p=None, truncation=None, usage=Usage(input_tokens=5877, output_tokens=149, total_tokens=6026, input_tokens_details=None, output_tokens_details=None), instructions=&#39;\nYou are a Customer assistant.\nYou will get customer records from file search tool.\n&#39;)</pre><h3>Conclusion</h3><p>In this article, we built a <strong>Customer Intelligence Agent</strong> using Llama Stack, Milvus, vLLM, and an MCP server.</p><p>We demonstrated how to:</p><ul><li>Ingest structured customer data into a vector store</li><li>Retrieve it using RAG (file_search)</li><li>Enrich it dynamically through an MCP validation service</li><li>Apply conditional logic to identify the first GOLD customer</li></ul><p>We implemented the same workflow in two ways:</p><ul><li><strong>Single-call orchestration</strong> using the Responses API</li><li><strong>Stateful orchestration</strong> using the Agent abstraction</li></ul><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=1eef5b5aca9f" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Agentic AI on OpenShift]]></title>
            <link>https://medium.com/@shrishs/agentic-ai-on-openshift-8b5c38762a6d?source=rss-748916ab05a5------2</link>
            <guid isPermaLink="false">https://medium.com/p/8b5c38762a6d</guid>
            <category><![CDATA[openshift]]></category>
            <category><![CDATA[kubernetes]]></category>
            <category><![CDATA[mcp-server]]></category>
            <category><![CDATA[red-hat-openshift-ai]]></category>
            <category><![CDATA[agentic-ai]]></category>
            <dc:creator><![CDATA[Shrishs]]></dc:creator>
            <pubDate>Mon, 26 Jan 2026 17:35:51 GMT</pubDate>
            <atom:updated>2026-01-26T17:35:51.777Z</atom:updated>
            <content:encoded><![CDATA[<p>This page serves as a curated index of my articles on <strong>agentic AI on OpenShift AI 3.x</strong>. The posts cover platform foundations, model serving, MCP servers, the Responses API, and Llama Stack, with a consistent focus on building and operating agentic systems on OpenShift.</p><ul><li><a href="https://medium.com/@shrishs/building-agentic-ai-on-openshift-from-chat-completions-to-the-responses-api-with-llama-stack-3f53c228a57f">Building Agentic AI on OpenShift: From Chat Completions to the Responses API with Llama Stack</a></li><li><a href="https://medium.com/@shrishs/creating-an-mcp-model-context-protocol-server-on-openshift-85dceac65c07">Creating an MCP (Model Context Protocol) Server on OpenShift</a></li><li><a href="https://medium.com/@shrishs/openshift-ai-3-0-model-deployment-ai-asset-endpoints-and-the-gen-ai-playground-9a2b6be6377d">OpenShift AI 3.0: Model Deployment, AI Asset Endpoints, and the Gen AI Playground</a></li><li><a href="https://medium.com/@shrishs/models-as-a-service-how-to-deploy-and-govern-llm-apis-on-openshift-ai-ed965acc7036">Models-as-a-Service: How to Deploy and Govern LLM APIs on OpenShift AI</a></li><li><a href="https://medium.com/@shrishs/gpu-considerations-for-llm-deployment-on-openshift-ai-3-x-e8999d85591c">GPU Considerations for LLM Deployment on OpenShift AI 3.x</a></li><li><a href="https://medium.com/@shrishs/designing-an-ai-native-developer-hub-06942a5d05ab">Designing an AI-Native Developer Hub</a></li><li><a href="https://medium.com/@shrishs/bring-your-own-model-openshift-mcp-server-from-vs-code-bcd4437e81e8">Bring Your Own Model: OpenShift MCP Server from VS Code</a></li></ul><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=8b5c38762a6d" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Building Agentic AI on OpenShift: From Chat Completions to the Responses API with Llama Stack]]></title>
            <link>https://medium.com/@shrishs/building-agentic-ai-on-openshift-from-chat-completions-to-the-responses-api-with-llama-stack-3f53c228a57f?source=rss-748916ab05a5------2</link>
            <guid isPermaLink="false">https://medium.com/p/3f53c228a57f</guid>
            <category><![CDATA[openshift]]></category>
            <category><![CDATA[agentic-ai]]></category>
            <category><![CDATA[red-hat-openshift-ai]]></category>
            <category><![CDATA[kubernetes]]></category>
            <category><![CDATA[llama-stack]]></category>
            <dc:creator><![CDATA[Shrishs]]></dc:creator>
            <pubDate>Mon, 26 Jan 2026 16:59:41 GMT</pubDate>
            <atom:updated>2026-01-26T16:59:41.453Z</atom:updated>
            <content:encoded><![CDATA[<p>Most generative AI applications start with a <strong>Chat Completions–style API</strong>:</p><pre>messages → model → text</pre><p>This model is easy to understand and works well for simple chatbots. But it quickly falls apart when you need anything beyond text generation — calling tools, retrieving data, handling multi-step reasoning, or operating in an enterprise environment. Developers are forced to build complex control loops in client code, making systems fragile and hard to scale.</p><p><strong>Responses API</strong> takes a different approach.Instead of producing text, it treats each request as an <strong>execution</strong> — a structured process where reasoning, tool calls, and observations are first-class concepts:</p><pre>input<br> → reasoning<br> → tool calls<br> → observations<br> → final output</pre><p>By shifting orchestration to the platform, the Responses API enables true agentic behavior: systems that can reason, act, and respond reliably, without pushing complexity into the client.</p><p>Llama Stack <a href="https://developers.redhat.com/articles/2025/12/09/your-ai-agents-evolved-modernize-llama-stack-agents-migrating-responses-api?utm_source=chatgpt.com#">adopts</a> the <a href="https://developers.openai.com/blog/responses-api/">Responses API</a> to align with the industry’s shift toward agentic reasoning. Rather than coordinating multiple tool endpoints and identifiers in client code, developers make a single API call that automatically manages tool discovery, execution planning, and response generation.</p><p>In this article, we will set up Llama Stack and an MCP server, then use the Responses API to build an agent that demonstrates the real power of agentic systems.</p><h4>Setting up Llama Stack</h4><p>With <a href="https://docs.redhat.com/en/documentation/red_hat_openshift_ai_self-managed/3.2/html-single/release_notes/index#enhancements">openshift AI 3.2</a> ,PostgreSQL is now the only supported database for production Llama Stack deployments in OpenShift AI.</p><ul><li>For the demo purpose I deployed the Crunchy Postgres Operator.</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*2N5izEcNNcsFpRy8HskgcQ.png" /></figure><ul><li>Created a Postgres cluster.</li></ul><pre>kind: PostgresCluster<br>apiVersion: postgres-operator.crunchydata.com/v1beta1<br>metadata:<br>  name: example<br>  namespace: demo-ai<br>spec:<br>  backups:<br>    pgbackrest:<br>      repos:<br>        - name: repo1<br>          volume:<br>            volumeClaimSpec:<br>              accessModes:<br>                - ReadWriteOnce<br>              resources:<br>                requests:<br>                  storage: 1Gi<br>  instances:<br>    - dataVolumeClaimSpec:<br>        accessModes:<br>          - ReadWriteOnce<br>        resources:<br>          requests:<br>            storage: 1Gi<br>      replicas: 1<br>  postgresVersion: 16</pre><p>Once the Postgres is up.This also creates <strong>example-pguser-example</strong> secret.Give some specific permission to default example DB.</p><pre>SELECT current_user, current_database();<br>CREATE SCHEMA IF NOT EXISTS llamastack AUTHORIZATION &quot;example&quot;;<br>ALTER ROLE &quot;example&quot; SET search_path TO llamastack, public;</pre><ul><li>Create the inference service as described in my <a href="https://medium.com/@shrishs/openshift-ai-3-0-model-deployment-ai-asset-endpoints-and-the-gen-ai-playground-9a2b6be6377d">previous article</a>.</li><li>Set up the environment variable.</li></ul><pre>export INFERENCE_MODEL=&quot;redhataillama-31-8b-instruct&quot;<br>export VLLM_URL=&quot;http://redhataillama-31-8b-instruct-predictor.demo-ai.svc.cluster.local:8080/v1&quot;<br>export VLLM_TLS_VERIFY=&quot;false&quot;  <br>export VLLM_API_TOKEN=&quot;NA&quot;<br>export VLLM_MAX_TOKENS=16384<br><br>export EMBEDDING_MODEL=&quot;nomic-embed-text-v1-5&quot;<br>export EMBEDDING_PROVIDER_MODEL_ID=&quot;nomic-embed-text-v1-5&quot;<br>export VLLM_EMBEDDING_URL=&quot;&lt;embedding-endpoint&gt;/v1&quot;<br>export VLLM_EMBEDDING_API_TOKEN=&quot;&lt;embedding-token&gt;&quot;<br>export VLLM_EMBEDDING_MAX_TOKENS=8192<br>export VLLM_EMBEDDING_TLS_VERIFY=&quot;true&quot;</pre><ul><li>Create the requieed secret.</li></ul><pre>oc create secret generic llama-stack-secret -n demo-ai \<br>  --from-literal=INFERENCE_MODEL=&quot;$INFERENCE_MODEL&quot; \<br>  --from-literal=VLLM_URL=&quot;$VLLM_URL&quot; \<br>  --from-literal=VLLM_TLS_VERIFY=&quot;$VLLM_TLS_VERIFY&quot; \<br>  --from-literal=VLLM_API_TOKEN=&quot;$VLLM_API_TOKEN&quot; \<br>  --from-literal=VLLM_MAX_TOKENS=&quot;$VLLM_MAX_TOKENS&quot; \<br>  --from-literal=EMBEDDING_MODEL=&quot;$EMBEDDING_MODEL&quot; \<br>  --from-literal=EMBEDDING_PROVIDER_MODEL_ID=&quot;$EMBEDDING_PROVIDER_MODEL_ID&quot; \<br>  --from-literal=VLLM_EMBEDDING_URL=&quot;$VLLM_EMBEDDING_URL&quot; \<br>  --from-literal=VLLM_EMBEDDING_TLS_VERIFY=&quot;$VLLM_EMBEDDING_TLS_VERIFY&quot; \<br>  --from-literal=VLLM_EMBEDDING_API_TOKEN=&quot;$VLLM_EMBEDDING_API_TOKEN&quot; \<br>  --from-literal=VLLM_EMBEDDING_MAX_TOKENS=&quot;$VLLM_EMBEDDING_MAX_TOKENS&quot;</pre><ul><li>Create LLamaStack Distribution</li></ul><pre>apiVersion: llamastack.io/v1alpha1<br>kind: LlamaStackDistribution<br>metadata:<br>  name: lsd-llama-milvus-inline<br>  namespace: demo-ai<br>spec:<br>  replicas: 1<br>  server:<br>    containerSpec:<br>      resources:<br>        requests:<br>          cpu: &quot;250m&quot;<br>          memory: &quot;500Mi&quot;<br>        limits:<br>          cpu: 4<br>          memory: &quot;12Gi&quot;<br>      env:<br>        # PostgreSQL metadata store (required in {productname-short} 3.2)<br>        - name: POSTGRES_HOST<br>          value: example-primary.demo-ai.svc<br>        - name: POSTGRES_PORT<br>          value: &quot;5432&quot;<br>        - name: POSTGRES_DB<br>          value: example<br>        - name: POSTGRES_USER<br>          value: example<br>        - name: POSTGRES_PASSWORD<br>          valueFrom:<br>            secretKeyRef:<br>              name: example-pguser-example<br>              key: password<br><br>        # Remote LLM configuration<br>        - name: INFERENCE_MODEL<br>          valueFrom:<br>            secretKeyRef:<br>              name: llama-stack-secret<br>              key: INFERENCE_MODEL<br>        - name: VLLM_URL<br>          valueFrom:<br>            secretKeyRef:<br>              name: llama-stack-secret<br>              key: VLLM_URL<br>        - name: VLLM_TLS_VERIFY<br>          valueFrom:<br>            secretKeyRef:<br>              name: llama-stack-secret<br>              key: VLLM_TLS_VERIFY<br>        - name: VLLM_API_TOKEN<br>          valueFrom:<br>            secretKeyRef:<br>              name: llama-stack-secret<br>              key: VLLM_API_TOKEN<br>        - name: VLLM_MAX_TOKENS<br>          valueFrom:<br>            secretKeyRef:<br>              name: llama-stack-secret<br>              key: VLLM_MAX_TOKENS<br><br>        # Remote embedding configuration<br>        - name: EMBEDDING_MODEL<br>          valueFrom:<br>            secretKeyRef:<br>              name: llama-stack-secret<br>              key: EMBEDDING_MODEL<br>        - name: EMBEDDING_PROVIDER_MODEL_ID<br>          valueFrom:<br>            secretKeyRef:<br>              name: llama-stack-secret<br>              key: EMBEDDING_PROVIDER_MODEL_ID<br>        - name: VLLM_EMBEDDING_URL<br>          valueFrom:<br>            secretKeyRef:<br>              name: llama-stack-secret<br>              key: VLLM_EMBEDDING_URL<br>        - name: VLLM_EMBEDDING_TLS_VERIFY<br>          valueFrom:<br>            secretKeyRef:<br>              name: llama-stack-secret<br>              key: VLLM_EMBEDDING_TLS_VERIFY<br>        - name: VLLM_EMBEDDING_API_TOKEN<br>          valueFrom:<br>            secretKeyRef:<br>              name: llama-stack-secret<br>              key: VLLM_EMBEDDING_API_TOKEN<br>        - name: VLLM_EMBEDDING_MAX_TOKENS<br>          valueFrom:<br>            secretKeyRef:<br>              name: llama-stack-secret<br>              key: VLLM_EMBEDDING_MAX_TOKENS<br><br>        - name: FMS_ORCHESTRATOR_URL<br>          value: &quot;http://localhost&quot;<br>      name: llama-stack<br>      port: 8321<br>    distribution:<br>      name: rh-dev<br>    storage:<br>      size: 5Gi</pre><ul><li>Ensure LLama stack is up &amp; running</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/924/1*ScHIxyylXNbGfKy0MDLE-g.png" /></figure><ul><li>Deploy the customer MCP Server as described in my <a href="https://medium.com/@shrishs/creating-an-mcp-model-context-protocol-server-on-openshift-85dceac65c07">previous article</a>.</li><li>Create a .env file with the following endpoint</li></ul><pre>LLAMASTACK_END_POINT=&quot;https://LLama_Stack_ROUTE_URL&quot;<br>MCP_CUSTOMER_END_POINT_SSE=&quot;https://MCP_ROUTE_URL/sse/sse&quot;</pre><h4>Responses API to build an agent</h4><ul><li>Create a python file with the following content &amp; run it.</li></ul><pre># Install a specific Llama Stack client version used in this example<br>%pip install -q llama_stack_client==0.3.5<br><br># Load environment variables (Llama Stack endpoint + MCP endpoints)<br>from dotenv import load_dotenv<br>import os<br>load_dotenv()<br><br>base = os.environ[&quot;LLAMASTACK_END_POINT&quot;]<br>mcp_customer_endpoint = os.environ[&quot;MCP_CUSTOMER_END_POINT&quot;]<br>mcp_customer_endpoint_sse = os.environ[&quot;MCP_CUSTOMER_END_POINT_SSE&quot;]<br><br># Import core Llama Stack client primitives<br>from llama_stack_client import Agent, AgentEventLogger, LlamaStackClient<br><br># Create a client pointing to the Llama Stack server<br># Note: this client does NOT execute agent logic locally.<br># It only sends requests to the server-side agent runtime.<br>client = LlamaStackClient(base_url=base)<br><br># Select an LLM model dynamically from the model registry<br># This avoids hard-coding model IDs<br>model_id = next(m.identifier for m in models if m.model_type == &quot;llm&quot;)<br>print(model_id)<br><br># Define the agent *declaratively*<br># This specification is sent to the Llama Stack server,<br># where reasoning, tool invocation, and execution happen.<br>agent = Agent(<br>    model=model_id,<br>    instructions=(<br>        &quot;You are a helpful Customer Details assistant. &quot;<br>        &quot;Use the registered MCP tool to answer the queries.&quot;<br>    ),<br>    client=client,<br>    tools=[{<br>        &quot;type&quot;: &quot;mcp&quot;,<br>        &quot;server_url&quot;: mcp_customer_endpoint,<br>        &quot;server_label&quot;: &quot;Customer-MCP-Server&quot;,<br>    }]<br>)<br><br>import uuid<br><br># Create a new session<br># A session represents shared conversational context on the server,<br># not a running agent process on the client.<br>session_id = agent.create_session(f&quot;nps_session-{uuid.uuid4().hex}&quot;)<br><br># -------------------------------<br># RESPONSE 1<br># -------------------------------<br># This call creates a full Responses API execution:<br># input → reasoning → MCP tool call → observation → final output<br>agent_response1 = agent.create_turn(<br>    messages=[{<br>        &quot;role&quot;: &quot;user&quot;,<br>        &quot;content&quot;: &quot;Get me details of Customer id 30&quot;<br>    }],<br>    session_id=session_id,<br>    stream=False,<br>)<br><br># Inspecting the response shows:<br># - the MCP tool call<br># - the tool output<br># - the final synthesized answer<br>pretty_print(agent_response1)<br><br># -------------------------------<br># RESPONSE 2<br># -------------------------------<br># This is a NEW execution (not a continuation of response 1),<br># but it runs in the SAME session.<br>#<br># Even though the user does not repeat the customer ID,<br># the agent infers `customer_id = 30` from session context.<br># The tool is called again to ground the answer with fresh evidence.<br>agent_response2 = agent.create_turn(<br>    messages=[{<br>        &quot;role&quot;: &quot;user&quot;,<br>        &quot;content&quot;: &quot;Is it a Gold Customer?&quot;<br>    }],<br>    session_id=session_id,<br>    stream=False,<br>)</pre><ul><li>In the above example, the second question does not repeat the customer ID(<em>Is it a Gold Customer</em>), yet the agent automatically resolves it from the ongoing session. This happens because the Responses API preserves <strong>conversational context</strong>, allowing the agent to infer that the user is still referring to the same customer.</li><li>At the same time, if you see the output below, each request is treated as a fresh execution, and <strong>MCP tool is called again</strong>.This balance between contextual understanding and execution-level correctness is what makes the Responses API well suited for building reliable, agentic systems.</li><li>Response for the first call <em>pretty_print(agent_response1) </em>is as follows</li></ul><pre>{<br>  &quot;id&quot;: &quot;resp_54d9de11-6956-4fcb-975b-37636a104947&quot;,<br>  &quot;created_at&quot;: 1769435398,<br>  &quot;model&quot;: &quot;vllm-inference/redhataillama-31-8b-instruct&quot;,<br>  &quot;object&quot;: &quot;response&quot;,<br>  &quot;output&quot;: [<br>    {<br>      &quot;id&quot;: &quot;mcp_list_17affc9a-1c3e-4fc9-8b14-ba377e1890c4&quot;,<br>      &quot;server_label&quot;: &quot;Customer-MCP-Server&quot;,<br>      &quot;tools&quot;: [<br>        {<br>          &quot;input_schema&quot;: {<br>            &quot;properties&quot;: {<br>              &quot;customer_id&quot;: {<br>                &quot;type&quot;: &quot;string&quot;<br>              }<br>            },<br>            &quot;required&quot;: [<br>              &quot;customer_id&quot;<br>            ],<br>            &quot;type&quot;: &quot;object&quot;<br>          },<br>          &quot;name&quot;: &quot;get_customer_details&quot;,<br>          &quot;description&quot;: null<br>        }<br>      ],<br>      &quot;type&quot;: &quot;mcp_list_tools&quot;<br>    },<br>    {<br>      &quot;id&quot;: &quot;fc_f7965b9f-ab82-4977-a557-f73853732efe&quot;,<br>      &quot;arguments&quot;: &quot;{\&quot;customer_id\&quot;: \&quot;30\&quot;}&quot;,<br>      &quot;name&quot;: &quot;get_customer_details&quot;,<br>      &quot;server_label&quot;: &quot;Customer-MCP-Server&quot;,<br>      &quot;type&quot;: &quot;mcp_call&quot;,<br>      &quot;error&quot;: null,<br>      &quot;output&quot;: &quot;{\&quot;customer_id\&quot;:\&quot;30\&quot;,\&quot;name\&quot;:\&quot;John Doe\&quot;,\&quot;email\&quot;:\&quot;john.doe@example.com\&quot;,\&quot;status\&quot;:\&quot;Gold\&quot;,\&quot;country\&quot;:\&quot;Germany\&quot;}&quot;<br>    },<br>    {<br>      &quot;content&quot;: [<br>        {<br>          &quot;annotations&quot;: [],<br>          &quot;text&quot;: &quot;Environment: ipython\nCutting Knowledge Date: December 2023\nToday Date: 26 Jul 2024\n\nThe response is a JSON object containing the customer details for customer id 30.&quot;,<br>          &quot;type&quot;: &quot;output_text&quot;<br>        }<br>      ],<br>      &quot;role&quot;: &quot;assistant&quot;,<br>      &quot;type&quot;: &quot;message&quot;,<br>      &quot;id&quot;: &quot;msg_313553ee-d8d4-4db8-8216-c6e05a16e787&quot;,<br>      &quot;status&quot;: &quot;completed&quot;<br>    }<br>  ],<br>  &quot;parallel_tool_calls&quot;: false,<br>  &quot;status&quot;: &quot;completed&quot;,<br>  &quot;text&quot;: {<br>    &quot;format&quot;: {<br>      &quot;type&quot;: &quot;text&quot;,<br>      &quot;description&quot;: null,<br>      &quot;name&quot;: null,<br>      &quot;schema_&quot;: null,<br>      &quot;strict&quot;: null<br>    }<br>  },<br>  &quot;error&quot;: null,<br>  &quot;previous_response_id&quot;: null,<br>  &quot;temperature&quot;: null,<br>  &quot;tools&quot;: [<br>    {<br>      &quot;server_label&quot;: &quot;Customer-MCP-Server&quot;,<br>      &quot;type&quot;: &quot;mcp&quot;,<br>      &quot;allowed_tools&quot;: null<br>    }<br>  ],<br>  &quot;top_p&quot;: null,<br>  &quot;truncation&quot;: null,<br>  &quot;usage&quot;: {<br>    &quot;input_tokens&quot;: 444,<br>    &quot;output_tokens&quot;: 61,<br>    &quot;total_tokens&quot;: 505,<br>    &quot;input_tokens_details&quot;: null,<br>    &quot;output_tokens_details&quot;: null<br>  }<br>}</pre><ul><li>Response for the second call <em>pretty_print(agent_response2) </em>is as follows</li></ul><pre>INFO:httpx:HTTP Request: POST https://llamastack-demo-ai.appxxxxxxxx/v1/responses &quot;HTTP/1.1 200 OK&quot;<br>{<br>  &quot;id&quot;: &quot;resp_87929bb3-56cf-4e2a-a55f-55696ff9dad4&quot;,<br>  &quot;created_at&quot;: 1769435399,<br>  &quot;model&quot;: &quot;vllm-inference/redhataillama-31-8b-instruct&quot;,<br>  &quot;object&quot;: &quot;response&quot;,<br>  &quot;output&quot;: [<br>    {<br>      &quot;id&quot;: &quot;mcp_list_6cffb58b-9361-472d-a4fd-2ee14740da82&quot;,<br>      &quot;server_label&quot;: &quot;Customer-MCP-Server&quot;,<br>      &quot;tools&quot;: [<br>        {<br>          &quot;input_schema&quot;: {<br>            &quot;properties&quot;: {<br>              &quot;customer_id&quot;: {<br>                &quot;type&quot;: &quot;string&quot;<br>              }<br>            },<br>            &quot;required&quot;: [<br>              &quot;customer_id&quot;<br>            ],<br>            &quot;type&quot;: &quot;object&quot;<br>          },<br>          &quot;name&quot;: &quot;get_customer_details&quot;,<br>          &quot;description&quot;: null<br>        }<br>      ],<br>      &quot;type&quot;: &quot;mcp_list_tools&quot;<br>    },<br>    {<br>      &quot;id&quot;: &quot;fc_efe68252-7647-425a-ba7f-f97aaff2f524&quot;,<br>      &quot;arguments&quot;: &quot;{\&quot;customer_id\&quot;: \&quot;30\&quot;}&quot;,<br>      &quot;name&quot;: &quot;get_customer_details&quot;,<br>      &quot;server_label&quot;: &quot;Customer-MCP-Server&quot;,<br>      &quot;type&quot;: &quot;mcp_call&quot;,<br>      &quot;error&quot;: null,<br>      &quot;output&quot;: &quot;{\&quot;customer_id\&quot;:\&quot;30\&quot;,\&quot;name\&quot;:\&quot;John Doe\&quot;,\&quot;email\&quot;:\&quot;john.doe@example.com\&quot;,\&quot;status\&quot;:\&quot;Gold\&quot;,\&quot;country\&quot;:\&quot;Germany\&quot;}&quot;<br>    },<br>    {<br>      &quot;content&quot;: [<br>        {<br>          &quot;annotations&quot;: [],<br>          &quot;text&quot;: &quot;The customer with id 30 is a Gold customer.&quot;,<br>          &quot;type&quot;: &quot;output_text&quot;<br>        }<br>      ],<br>      &quot;role&quot;: &quot;assistant&quot;,<br>      &quot;type&quot;: &quot;message&quot;,<br>      &quot;id&quot;: &quot;msg_76e3bd34-490f-4ae9-a712-c789924c2300&quot;,<br>      &quot;status&quot;: &quot;completed&quot;<br>    }<br>  ],<br>  &quot;parallel_tool_calls&quot;: false,<br>  &quot;status&quot;: &quot;completed&quot;,<br>  &quot;text&quot;: {<br>    &quot;format&quot;: {<br>      &quot;type&quot;: &quot;text&quot;,<br>      &quot;description&quot;: null,<br>      &quot;name&quot;: null,<br>      &quot;schema_&quot;: null,<br>      &quot;strict&quot;: null<br>    }<br>  },<br>  &quot;error&quot;: null,<br>  &quot;previous_response_id&quot;: null,<br>  &quot;temperature&quot;: null,<br>  &quot;tools&quot;: [<br>    {<br>      &quot;server_label&quot;: &quot;Customer-MCP-Server&quot;,<br>      &quot;type&quot;: &quot;mcp&quot;,<br>      &quot;allowed_tools&quot;: null<br>    }<br>  ],<br>  &quot;top_p&quot;: null,<br>  &quot;truncation&quot;: null,<br>  &quot;usage&quot;: {<br>    &quot;input_tokens&quot;: 672,<br>    &quot;output_tokens&quot;: 31,<br>    &quot;total_tokens&quot;: 703,<br>    &quot;input_tokens_details&quot;: null,<br>    &quot;output_tokens_details&quot;: null<br>  }<br>}</pre><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=3f53c228a57f" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Creating an MCP (Model Context Protocol) Server on OpenShift]]></title>
            <link>https://medium.com/@shrishs/creating-an-mcp-model-context-protocol-server-on-openshift-85dceac65c07?source=rss-748916ab05a5------2</link>
            <guid isPermaLink="false">https://medium.com/p/85dceac65c07</guid>
            <category><![CDATA[agentic-ai]]></category>
            <category><![CDATA[kubernetes]]></category>
            <category><![CDATA[red-hat-openshift-ai]]></category>
            <category><![CDATA[fastmcp]]></category>
            <category><![CDATA[mcp-server]]></category>
            <dc:creator><![CDATA[Shrishs]]></dc:creator>
            <pubDate>Mon, 05 Jan 2026 10:41:18 GMT</pubDate>
            <atom:updated>2026-01-06T08:58:53.403Z</atom:updated>
            <content:encoded><![CDATA[<p>Large Language Models are increasingly being augmented with <strong>external tools, data sources, and actions</strong>.The <strong>Model Context Protocol (MCP)</strong> is emerging as a standard way to expose such capabilities in a structured and model-agnostic way .</p><p>In this article, we walk through <strong>how to create and run an MCP server on OpenShift</strong>, treating it like a real production microservice. And Test it using Openshift AI 3.x playground.Testing setup is mentioned in my previous article <a href="https://medium.com/@shrishs/openshift-ai-3-0-model-deployment-ai-asset-endpoints-and-the-gen-ai-playground-9a2b6be6377d"><em>OpenShift AI 3.0: Model Deployment, AI Asset Endpoints, and the Gen AI Playground</em></a></p><h4>Choosing the MCP Server Implementation</h4><p><a href="https://gofastmcp.com/">FastMCP</a> is a popular framework for building servers that use the open <strong>Model Context Protocol (MCP)</strong>, which provides a standardized way for AI language models (LLMs) to access external data and tools.</p><h4>Local Implementation</h4><p>In this example we create simple MCP server on the developer machine.</p><ul><li>server.py</li></ul><pre>from fastmcp import FastMCP<br>from fastapi import FastAPI<br>from fastapi.middleware.cors import CORSMiddleware<br>import uvicorn<br><br>mcp = FastMCP(&quot;Customer Tools&quot;)<br><br>@mcp.tool<br>def get_customer_details(customer_id: str) -&gt; dict:<br>    return {<br>        &quot;customer_id&quot;: customer_id,<br>        &quot;name&quot;: &quot;John Doe&quot;,<br>        &quot;email&quot;: &quot;john.doe@example.com&quot;,<br>        &quot;status&quot;: &quot;Gold&quot;,<br>        &quot;country&quot;: &quot;Germany&quot;,<br>    }<br><br><br># Create BOTH ASGI apps<br>mcp_http_app = mcp.http_app(transport=&quot;streamable-http&quot;)<br>mcp_sse_app  = mcp.http_app(transport=&quot;sse&quot;)   # yes, same factory, different transport<br><br># Pick one lifespan to pass to FastAPI.<br># (Streamable-http is the critical one for OpenShift AI.)<br>app = FastAPI(lifespan=mcp_http_app.lifespan)<br><br>@app.get(&quot;/healthz&quot;)<br>def healthz():<br>    return {&quot;ok&quot;: True}<br><br><br><br># Mount endpoints<br>app.mount(&quot;/mcp&quot;, mcp_http_app)     # OpenShift AI uses this<br>app.mount(&quot;/sse&quot;, mcp_sse_app)      # MCP Inspector/debug uses this<br><br># Added for testing with the Browser based application<br>app.add_middleware(<br>    CORSMiddleware,<br>    allow_origins=[&quot;*&quot;],<br>    allow_methods=[&quot;*&quot;],<br>    allow_headers=[&quot;*&quot;],<br>)<br><br><br>if __name__ == &quot;__main__&quot;:<br>    uvicorn.run(app, host=&quot;0.0.0.0&quot;, port=8080)<br></pre><ul><li>requirements.txt</li></ul><pre>fastmcp<br>fastapi<br>uvicorn</pre><ul><li>Run the python code.</li></ul><pre>$ python server.py <br>INFO:     Started server process [4840]<br>INFO:     Waiting for application startup.<br>INFO:     Application startup complete.<br>INFO:     Uvicorn running on http://0.0.0.0:8080 (Press CTRL+C to quit)</pre><ul><li>Verify it.</li></ul><pre># curl localhost:8080/healthz<br>{&quot;ok&quot;:true}<br><br><br># curl localhost:8080/sse/sse<br>event: endpoint<br>data: /sse/messages/?session_id=cb1d705707714fa79ef8ca9a4caa3b8d<br>: ping - 2026-01-06 08:38:59.551426+00:00<br><br><br># curl -i -N -X POST localhost:8080/mcp/mcp -H &quot;Content-Type: application/json&quot;   -H &quot;Accept: application/json, text/event-stream&quot;   -d &#39;{&quot;jsonrpc&quot;:&quot;2.0&quot;,&quot;id&quot;:1,&quot;method&quot;:&quot;initialize&quot;,&quot;params&quot;:{&quot;protocolVersion&quot;:&quot;2024-11-05&quot;,&quot;clientInfo&quot;:{&quot;name&quot;:&quot;curl&quot;,&quot;version&quot;:&quot;0.1&quot;},&quot;capabilities&quot;:{}}}&#39;<br>HTTP/1.1 200 OK<br>date: Tue, 06 Jan 2026 08:37:34 GMT<br>server: uvicorn<br>cache-control: no-cache, no-transform<br>connection: keep-alive<br>content-type: text/event-stream<br>mcp-session-id: 96618d51b8aa4f6eb5518853046a4b87<br>x-accel-buffering: no<br>Transfer-Encoding: chunked<br>event: message<br>data: {&quot;jsonrpc&quot;:&quot;2.0&quot;,&quot;id&quot;:1,&quot;result&quot;:{&quot;protocolVersion&quot;:&quot;2024-11-05&quot;,&quot;capabilities&quot;:{&quot;experimental&quot;:{},&quot;prompts&quot;:{&quot;listChanged&quot;:true},&quot;resources&quot;:{&quot;subscribe&quot;:false,&quot;listChanged&quot;:true},&quot;tools&quot;:{&quot;listChanged&quot;:true}},&quot;serverInfo&quot;:{&quot;name&quot;:&quot;Customer Tools&quot;,&quot;version&quot;:&quot;2.14.2&quot;}}}</pre><ul><li>MCP Inspector can also be run to verify it.</li></ul><pre>npx -y @modelcontextprotocol/inspector@latest<br>Starting MCP inspector...<br>⚙️ Proxy server listening on localhost:6277<br>🔑 Session token: 7a3326852db47d6beb07c7f1ebf9a265b02dbecacffb858375e2593224f3e0d2<br>   Use this token to authenticate requests or set DANGEROUSLY_OMIT_AUTH=true to disable auth<br><br>🚀 MCP Inspector is up and running at:<br>   http://localhost:6274/?MCP_PROXY_AUTH_TOKEN=7a3326852db47d6beb07c7f1ebf9a265b02dbecacffb858375e2593224f3e0d2<br><br>🌐 Opening browser...</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*4chRroeiPXzIjAno-O0WWg.png" /></figure><h4>Containerizing the MCP Server</h4><ul><li>Comment the server.py run part,And CORS settings.</li></ul><pre>from fastmcp import FastMCP<br>from fastapi import FastAPI<br>from fastapi.middleware.cors import CORSMiddleware<br>import uvicorn<br><br>mcp = FastMCP(&quot;Customer Tools&quot;)<br><br>@mcp.tool<br>def get_customer_details(customer_id: str) -&gt; dict:<br>    return {<br>        &quot;customer_id&quot;: customer_id,<br>        &quot;name&quot;: &quot;John Doe&quot;,<br>        &quot;email&quot;: &quot;john.doe@example.com&quot;,<br>        &quot;status&quot;: &quot;Gold&quot;,<br>        &quot;country&quot;: &quot;Germany&quot;,<br>    }<br><br><br># Create BOTH ASGI apps<br>mcp_http_app = mcp.http_app(transport=&quot;streamable-http&quot;)<br>mcp_sse_app  = mcp.http_app(transport=&quot;sse&quot;)   # yes, same factory, different transport<br><br># Pick one lifespan to pass to FastAPI.<br># (Streamable-http is the critical one for OpenShift AI.)<br>app = FastAPI(lifespan=mcp_http_app.lifespan)<br><br>@app.get(&quot;/healthz&quot;)<br>def healthz():<br>    return {&quot;ok&quot;: True}<br><br><br><br># Mount endpoints<br>app.mount(&quot;/mcp&quot;, mcp_http_app)     # OpenShift AI uses this<br>app.mount(&quot;/sse&quot;, mcp_sse_app)      # MCP Inspector/debug uses this<br><br># Added for testing with the Browser based application<br>app.add_middleware(<br>    CORSMiddleware,<br>    allow_origins=[&quot;*&quot;],<br>    allow_methods=[&quot;*&quot;],<br>    allow_headers=[&quot;*&quot;],<br>)<br><br><br>#if __name__ == &quot;__main__&quot;:<br>    #uvicorn.run(app, host=&quot;0.0.0.0&quot;, port=8080)<br><br></pre><ul><li>Create a Containerfile with <em>ubi9/python-311 </em>as base image.Move the execution part(uvicorn..) from server.yaml to this file.</li></ul><pre>FROM registry.access.redhat.com/ubi9/python-311<br><br>WORKDIR /opt/app-root/src<br><br>COPY requirements.txt .<br>RUN pip install --no-cache-dir -r requirements.txt<br><br>COPY server.py .<br><br>EXPOSE 8080<br><br># Run MCP SSE server and bind to 0.0.0.0:8080<br># Start uvicorn directly (proxy-aware) and serve the Starlette app object &quot;app&quot;<br>CMD [&quot;uvicorn&quot;, &quot;server:app&quot;, &quot;--host&quot;, &quot;0.0.0.0&quot;, &quot;--port&quot;, &quot;8080&quot;, &quot;--proxy-headers&quot;, &quot;--forwarded-allow-ips&quot;, &quot;*&quot;]</pre><ul><li>Build it.</li></ul><pre>podman build --no-cache --platform=linux/amd64 -t customer-mcp-sse:0.1 -f Containerfile .</pre><ul><li>Tag it and push it you available registry</li></ul><pre>podman tag localhost/customer-mcp-sse:0.1 &lt;REPO_URL&gt;/customer-mcp-sse:0.1<br>podman push &lt;REPO_URL&gt;/customer-mcp-sse:0.1</pre><h4>Kubernetes Scaffolding for Deploying an MCP Server</h4><pre>apiVersion: apps/v1<br>kind: Deployment<br>metadata:<br>  name: customer-mcp<br>  labels:<br>    app: customer-mcp<br>spec:<br>  replicas: 1<br>  selector:<br>    matchLabels:<br>      app: customer-mcp<br>  template:<br>    metadata:<br>      labels:<br>        app: customer-mcp<br>    spec:<br>      containers:<br>        - name: customer-mcp<br>          image: &lt;REPO_URL&gt;/customer-mcp-sse:0.1<br>          imagePullPolicy: IfNotPresent<br>          ports:<br>            - name: http<br>              containerPort: 8080<br>          readinessProbe:<br>            httpGet:<br>              path: /healthz<br>              port: 8080<br>            initialDelaySeconds: 3<br>            periodSeconds: 10<br>          livenessProbe:<br>            httpGet:<br>              path: /healthz<br>              port: 8080<br>            initialDelaySeconds: 10<br>            periodSeconds: 20<br>          resources:<br>            requests:<br>              cpu: 50m<br>              memory: 128Mi<br>            limits:<br>              cpu: 500m<br>              memory: 512Mi</pre><pre>apiVersion: v1<br>kind: Service<br>metadata:<br>  name: customer-mcp<br>  labels:<br>    app: customer-mcp<br>spec:<br>  selector:<br>    app: customer-mcp<br>  ports:<br>    - name: http<br>      port: 8080<br>      targetPort: 8080<br>  type: ClusterIP</pre><ul><li>Apply the above definition to the Openshift cluster and create a edge route to access it externally from the cluster.</li></ul><pre>curl -k https://&lt;Openshift_Edge_Route&gt;/healthz<br>{&quot;ok&quot;:true}</pre><p><strong>NOTE</strong>: To test the below part ,Openshift AI 3.x and LLamaStack enablement is prerequisite.</p><ul><li>Create a ConfigMap</li></ul><pre>apiVersion: v1<br>kind: ConfigMap<br>metadata:<br>  name: gen-ai-aa-mcp-servers<br>  namespace: redhat-ods-applications<br>data:<br>  Customer-MCP-Server: |<br>    {<br>      &quot;url&quot;: &quot;https://&lt;Openshift_Edge_Route&gt;/mcp/mcp&quot;,<br>      &quot;description&quot;: &quot;The Customer MCP server provide customer details.&quot;<br>    }   </pre><ul><li>Go to OpenshiftAI Dashboard,Try it in playground.</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*U_gOs42Ua4JExNXP3tk61Q.png" /></figure><ul><li>Request the customer details with some id .</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*EtYtDLOb9yPAq0j9M0CbuA.png" /></figure><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=85dceac65c07" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Designing an AI-Native Developer Hub]]></title>
            <link>https://medium.com/@shrishs/designing-an-ai-native-developer-hub-06942a5d05ab?source=rss-748916ab05a5------2</link>
            <guid isPermaLink="false">https://medium.com/p/06942a5d05ab</guid>
            <category><![CDATA[backstage]]></category>
            <category><![CDATA[watsonx-orchestrate]]></category>
            <category><![CDATA[openshift]]></category>
            <category><![CDATA[ai-agent]]></category>
            <category><![CDATA[agent-development-kit]]></category>
            <dc:creator><![CDATA[Shrishs]]></dc:creator>
            <pubDate>Fri, 26 Dec 2025 20:15:17 GMT</pubDate>
            <atom:updated>2025-12-26T20:15:17.825Z</atom:updated>
            <content:encoded><![CDATA[<p>As organizations adopt AI at scale, one challenge becomes increasingly clear:<br> <strong>we don’t just need more AI — we need visibility, structure, and governance around it.</strong></p><p>Models, agents, prompts, tools, and orchestration logic are being created across teams, repositories, and platforms. Without a clear system of record, the AI landscape quickly becomes fragmented.</p><p>In this article, I describe how I approached this problem by combining:</p><ul><li><a href="https://c4model.com/"><strong>C4 architecture thinking</strong></a> to structure the platform</li><li><strong>A Developer Hub</strong> as the control plane</li><li><strong>Cataloging the entire AI estate</strong> as first-class assets</li><li><strong>Dynamically importing agents from Watson Orchestrate</strong>, exposed through MCP</li></ul><h4>Starting with C4: A Simple Way to Design a Developer Hub</h4><p>The <strong>C4 model</strong> provides a simple mental framework by progressively zooming in:</p><ul><li><strong>Context</strong> — Who uses the system and why?</li><li><strong>Containers</strong> — What runs the system?</li><li><strong>Components</strong> — What are the main building blocks?</li><li><strong>Code</strong> — How is it implemented?</li></ul><p>This approach keeps discussions grounded in purpose rather than technology.</p><p>At the <strong>context level</strong>, the need is straightforward:<em>Developers and AI agents need a single place to discover, understand, and interact with the organization’s AI capabilities.</em>That “single place” is the <strong>Developer Hub.</strong></p><p><a href="https://docs.redhat.com/en/documentation/red_hat_developer_hub/1.8/html/about_red_hat_developer_hub/index">Red Hat Developer Hub</a> is a customizable developer portal with enterprise-level support and a centralized software catalog that one can use to build high-quality software efficiently in a streamlined development environment.It’s upstream project is <a href="https://backstage.io/">backsatge</a>.</p><p><strong>Installing Red Hat Developer Hub</strong></p><p>This blog is more focussed on installing in developer mode .</p><ul><li>Clone <a href="https://github.com/redhat-developer/rhdh">https://github.com/redhat-developer/rhdh</a></li><li>start the Backend</li></ul><pre>yarn - cwd packages/backend start</pre><ul><li>Start the Front-End</li></ul><pre>yarn --cwd packages/app start   </pre><ul><li>Above command open the page in the browser.</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*4TYNsa3TWQhSjblAFhWU7Q.png" /></figure><h4>Cataloging the Entire AI Estate</h4><p>This is the pivotal shift:<strong><em>AI is not a special case — it is part of the software estate.</em></strong>Cataloging the AI estate means treating AI assets the same way we treat services, APIs, and infrastructure.</p><p><strong>What does “AI estate” include?</strong></p><ul><li>Foundation models</li><li>AI agents</li><li>MCP Tool integrations</li></ul><p>Each of these becomes a <strong>first-class entity</strong> in the Developer Hub.</p><ul><li>Define the System Definition for <a href="https://github.com/shrishs/ai-platform-catalog/blob/main/catalog/system-ai-models.yaml">ai-models</a>.</li></ul><pre>apiVersion: backstage.io/v1alpha1<br>kind: System<br>metadata:<br>  name: ai-models<br>  title: AI Models<br>  description: System for model services &amp; artifacts.<br>  tags: [ai, models]<br>spec:<br>  owner: group:ml-eng<br>  lifecycle: production</pre><ul><li>Same way define it for Agent Catalog and <a href="https://github.com/shrishs/ai-platform-catalog/blob/main/catalog/system-mcp-integration.yaml">MCP Integration Layer</a>.Edit the app-config.local.yaml file and edit this as location.</li></ul><pre>catalog:<br>  database:<br>    client: better-sqlite3<br>    connection:<br>      filename: ./database/catalog.sqlite<br>  import:<br>    entityFilename: catalog-info.yaml<br>    pullRequestBranchName: backstage-integration<br>  rules:<br>    - allow: [Component, System, Group, Resource, Location, Template, API]<br>  locations:<br>    - type: file<br>      target: ../../catalog-entities/all.yaml<br>    - type: url<br>      target: https://github.com/shrishs/ai-platform-catalog/blob/main/catalog/location.yaml      </pre><ul><li>Restart the servers and you see all the System displayed</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*gb12dgqYGS3xcbdAg0w8YA.png" /></figure><ul><li>Create the component for model as explained in this <a href="https://developers.redhat.com/articles/2024/11/12/llms-developer-hub-catalog-ai-assets#">articles</a>.And define your <a href="https://github.com/shrishs/model-catalog-example/blob/main/developer-model-service/catalog-info.yaml">catalog-info.yaml</a> and link it with the existing system(ai-models).</li></ul><pre>apiVersion: backstage.io/v1alpha1<br>kind: Component<br>metadata:<br>  name: developer-model-service<br>  annotations:<br>    backstage.io/techdocs-ref: dir:./<br>  description: &#39;A vLLM and 3scale-based model service providing models for developer tools. A single model (IBM Granite Code 8b) is deployed on it through Red Hat OpenShift AI, and accessed over a secured API.&#39;<br>  links:<br>    - url: https://model-service.apps.domain.com<br>      title: Access<br>      type: website<br>      icon: WebAssett<br>    - url: https://ibm-granite-8b-code-instruct-vllm.apps.domain.com<br>      title: API URL<br>      type: website<br>      icon: WebAsset<br>  tags:<br>    - genai<br>    - ibm-granite<br>    - vllm<br>    - llm<br>    - developer-model-service<br>    - authenticated<br>    - gateway<br>spec:<br>  type: model-server<br>  owner: group:ml-eng<br>  system: ai-models <br>  lifecycle: production<br>  providesApis:<br>    - model-service-api<br>  dependsOn:<br>    - resource:ibm-granite-8b-code-instruct<br>    - api:model-service-api<br>  profile:<br>    displayName: &quot;Developer Model Service&quot;</pre><ul><li>Now click on AI Model and ,it displays all the model listed.</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*WwoQx6OlruuIEEpL1FcfMQ.png" /></figure><ul><li>Define the component of for <a href="https://github.com/shrishs/kubernetes-mcp-server/blob/main/catalog-info.yaml">MCP Server</a>.</li></ul><h4>Templates to Intelligent Agents: Scaffolding AI Agents with watsonx Orchestrate <a href="https://developer.watson-orchestrate.ibm.com/"><strong>Agent development Kit</strong></a></h4><p>To scale agent development, manual creation is not enough. Agent creation can be standardized through templates that define structure, skills, tooling, and metadata upfront. These templates act as scaffolding, enabling developers to rapidly bootstrap new agents in a consistent and governed way. Integrated into the Developer Hub, this approach turns agent creation into a repeatable, self-service workflow rather than a bespoke engineering effort.</p><ul><li>Create a <a href="https://github.com/shrishs/rhdh-templates/blob/main/templates/orchestrate-agent/template.yaml">template</a>.</li></ul><pre>apiVersion: scaffolder.backstage.io/v1beta3<br>kind: Template<br>metadata:<br>  name: orchestrate-agent<br>  title: Create a watsonx Orchestrate Agent<br>  description: A simple Backstage template to scaffold a watsonx Orchestrate ADK agent definition.<br>  tags: [watsonx, orchestrate, adk]<br>spec:<br>  type: template<br>  lifecycle: development<br>  owner: group:platform-ai<br>  system: ai-agent<br>  parameters:<br>    - title: Agent information<br>      required: [agentName, description]<br>      properties:<br>        agentName:<br>          title: Agent Name<br>          type: string<br>          description: Name of the agent (e.g., hello-agent)<br>        description:<br>          title: Description<br>          type: string<br>          description: What does this agent do?<br>        lifecycle:<br>          title: Choose Environment<br>          type: string<br>          enum: [development, staging ,production]<br>          default: development<br>        mcpServer:<br>          title: Choose MCP Server<br>          type: string<br>          description: Select an available MCP server from the catalog<br>          ui:field: EntityPicker<br>          ui:options:<br>            catalogFilter:<br>              kind: Component<br>              spec.type: mcp-server<br>            defaultKind: Component<br>        model:<br>          title: Choose Model<br>          type: string<br>          description: Select an available model from the catalog<br>          ui:field: EntityPicker<br>          ui:options:<br>            catalogFilter:<br>              kind: Resource<br>              spec.type: ai-model<br>            defaultKind: Resource<br>        owner:<br>          title: Owner<br>          type: string<br>          description: Backstage owner entity (e.g., group:platform)<br>        repoUrl:<br>          title: Repository<br>          type: string<br>          description: github.com?owner=your-org&amp;repo={{ parameters.agentName }}<br><br>  steps:<br>    - id: fetch<br>      name: Fetch skeleton<br>      action: fetch:template<br>      input:<br>        url: ./skeleton<br>        values:<br>          agentName: ${{ parameters.agentName }}<br>          description: ${{ parameters.description }}<br>          lifecycle: ${{ parameters.lifecycle }}<br>          mcpServer: ${{ parameters.mcpServer }}<br>          model: ${{ parameters.model }}<br>          owner: ${{ parameters.owner }}<br>          repoUrl: ${{ parameters.repoUrl }}<br><br>    - id: publish<br>      name: Publish to GitHub<br>      action: publish:github<br>      input:<br>        repoUrl: github.com?owner=${{ parameters.owner }}&amp;repo=${{ parameters.agentName }}<br>        defaultBranch: main<br>        repoVisibility: private<br><br>    - id: register<br>      name: Register in Backstage<br>      action: catalog:register<br>      input:<br>        repoContentsUrl: ${{ steps.publish.output.repoContentsUrl }}<br>        catalogInfoPath: /catalog-info.yaml<br><br>  output:<br>    links:<br>      - title: Repository<br>        url: ${{ steps.publish.output.remoteUrl }}<br>      - title: Backstage entity<br>        url: ${{ steps.register.output.entityRef }}</pre><ul><li>Define the Agent skelton.</li></ul><pre>spec_version: v1<br>kind: native<br>name: ${{ values.agentName }}<br>description: ${{ values.description }}<br>instructions: &quot;This agent uses ${{ values.mcpServer }} with available tools: {% for tool in values.mcpTools %}{{ tool.name }}{% if not loop.last %}, {% endif %}{% endfor %}&quot;<br>llm: ${{ values.model }}<br>mcp_server: ${{ values.mcpServer }}<br>lifecycle: ${{ values.lifecycle }}<br>style: default<br>collaborators: []<br>tools:<br>{% for tool in values.mcpTools %}<br>  - name: {{ tool.name }}<br>    description: {{ tool.description | replace(&#39;\n&#39;, &#39; &#39;) }}<br>    type: mcp<br>    server: ${{ values.mcpServer }}<br>{% endfor %}</pre><ul><li>Go to developer Hub portal and create a agent using the template.</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*nSe_gumVG-9NK4FwQok9qw.png" /></figure><ul><li>Launch the template.</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*4FLeI_ljne6ns8TxhZzVrg.png" /></figure><ul><li>This creates all the required scaffolding in your github repo and is ready to be imported in watsonx orchestrate.This is detailed in my <a href="https://medium.com/@shrishs/build-and-run-agents-locally-with-the-watsonx-orchestrate-agent-development-kit-adk-868caa959d65">previous article</a>.</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*qjFIUGNVFDU6quf4ttg6Kg.png" /></figure><h4>Retrieving watsonx Orchestrate Agents in Red Hat Developer Hub (RHDH)</h4><p>It is implemented using a <strong>custom Backstage EntityProvider</strong> that:</p><ol><li>Calls the watsonx Orchestrate Agents API</li><li>Transforms each agent into a Backstage Resource</li><li>Registers those resources dynamically with the Catalog</li><li>Periodically refreshes them using the Backstage scheduler</li></ol><ul><li>Create backend/src/modules/OrchestrateAgentsEntityProvider.ts</li></ul><pre>import { EntityProvider, EntityProviderConnection } from &#39;@backstage/plugin-catalog-node&#39;;<br>import { LoggerService } from &#39;@backstage/backend-plugin-api&#39;;<br><br>type OrchestrateAgent = {<br>  id?: string; name?: string; displayName?: string; description?: string; version?: string;<br>};<br><br>export default class OrchestrateAgentsEntityProvider implements EntityProvider {<br>  private connection?: EntityProviderConnection;<br><br>  constructor(<br>    private readonly opts: { id: string; baseUrl: string; token: string; logger: LoggerService }<br>  ) {}<br><br>  getProviderName() {<br>    return `orchestrate-agents-provider:${this.opts.id}`;<br>  }<br><br>  async connect(connection: EntityProviderConnection) {<br>    this.connection = connection;<br>    this.opts.logger.info(&#39;[orchestrate] Catalog connection established&#39;);<br>  }<br><br>  private async fetchAgents(): Promise&lt;OrchestrateAgent[]&gt; {<br>    const url = `${this.opts.baseUrl.replace(/\/$/, &#39;&#39;)}/api/v1/orchestrate/agents?include_hidden=false`;<br>    this.opts.logger.info(`[orchestrate] Fetching agents from ${url}`);<br><br>    const res = await fetch(url, {<br>      headers: { accept: &#39;application/json&#39;, authorization: `Bearer ${this.opts.token}` },<br>    });<br>    if (!res.ok) {<br>      const body = await res.text().catch(() =&gt; &#39;&#39;);<br>      throw new Error(`Orchestrate API ${res.status}: ${body}`);<br>    }<br><br>    const data = await res.json();<br>    const arr =<br>      (Array.isArray(data) &amp;&amp; data) ||<br>      data?.agents ||<br>      data?.items ||<br>      data?.data ||<br>      [];<br><br>    const count = Array.isArray(arr) ? arr.length : 0;<br>    this.opts.logger.info(`[orchestrate] Normalized to ${count} agent(s)`);<br>    return Array.isArray(arr) ? arr : [];<br>  }<br><br>  async run() {<br>    if (!this.connection) {<br>      this.opts.logger.warn(&#39;[orchestrate] run() called before catalog connection established&#39;);<br>      return;<br>    }<br><br>    this.opts.logger.info(&#39;[orchestrate] Starting sync run...&#39;);<br>    try {<br>      const agents = await this.fetchAgents();<br>      this.opts.logger.info(`[orchestrate] Received ${agents.length} agent(s) from Orchestrate`);<br><br>      const entities = agents.map((a, i) =&gt; {<br>        const id = a.id ?? `no-id-${i}`;<br>        const title = a.displayName ?? a.name ?? `Agent ${id}`;<br>        const desc = a.description ?? &#39;watsonx Orchestrate agent&#39;;<br>        const metadataName = `orch-agent-${String(id).toLowerCase().replace(/[^a-z0-9-]/g, &#39;-&#39;)}`.slice(0, 63);<br><br>        return {<br>          apiVersion: &#39;backstage.io/v1alpha1&#39;,<br>          kind: &#39;Resource&#39;,<br>          metadata: {<br>            name: metadataName,<br>            title,<br>            description: desc,<br>            annotations: {<br>              &#39;orchestrate.ibm.com/agent-id&#39;: String(id),<br>              &#39;orchestrate.ibm.com/version&#39;: a.version ?? &#39;unknown&#39;,<br>              &#39;backstage.io/managed-by-location&#39;: `url:orchestrate-provider/${id}`,<br>              &#39;backstage.io/managed-by-origin-location&#39;: `url:orchestrate-provider/${id}`,              <br>            },<br>            tags: [&#39;watsonx&#39;, &#39;orchestrate&#39;, &#39;agent&#39;],<br>          },<br>          spec: { type: &#39;agent&#39;, owner: &#39;orchestrate&#39;, system: &#39;ai-agent&#39; },<br>        };<br>      });<br><br>      await this.connection.applyMutation({<br>        type: &#39;full&#39;,<br>        entities: entities.map(entity =&gt; ({ entity, locationKey: this.getProviderName() })),<br>      });<br><br>      this.opts.logger.info(`[orchestrate] Synced ${entities.length} agent(s) into the catalog`);<br>    } catch (e) {<br>      this.opts.logger.error(<br>        `[orchestrate] Failed to refresh Orchestrate agents: ${e instanceof Error ? e.message : String(e)}`<br>      );<br>    }<br>  }<br>}</pre><ul><li>Create a Module backend/src/modules/orchestrateAgentsModule.ts</li></ul><pre>import { coreServices, createBackendModule } from &#39;@backstage/backend-plugin-api&#39;;<br>import { catalogProcessingExtensionPoint } from &#39;@backstage/plugin-catalog-node/alpha&#39;;<br>import OrchestrateAgentsEntityProvider from &#39;./OrchestrateAgentsEntityProvider&#39;;<br><br>const orchestrateAgentsModule = createBackendModule({<br>  pluginId: &#39;catalog&#39;,<br>  moduleId: &#39;orchestrate-agents&#39;,<br>  register(env) {<br>    env.registerInit({<br>      deps: {<br>        config: coreServices.rootConfig,<br>        logger: coreServices.logger,<br>        scheduler: coreServices.scheduler,<br>        processing: catalogProcessingExtensionPoint,<br>      },<br>      async init({ config, logger, scheduler, processing }) {<br>        const baseUrl = config.getString(&#39;orchestrate.baseUrl&#39;);<br>        const token = config.getString(&#39;orchestrate.token&#39;);<br><br>        const provider = new OrchestrateAgentsEntityProvider({<br>          id: &#39;default&#39;,<br>          baseUrl,<br>          token,<br>          logger,<br>        });<br><br>        // Register with the catalog first (so connect() is called)<br>        processing.addEntityProvider(provider);<br><br>        // Periodic refresh<br>        await scheduler.scheduleTask({<br>          id: &#39;orchestrate-agents-refresh&#39;,<br>          frequency: { minutes: 5 },<br>          timeout: { minutes: 2 },<br>          fn: async () =&gt; provider.run(),<br>        });<br><br>        // Kick off a first run shortly after startup so the connection is ready<br>        setTimeout(() =&gt; {<br>          logger.info(&#39;[orchestrate] Triggering first sync after catalog startup&#39;);<br>          provider.run();<br>        }, 3000);<br>      },<br>    });<br>  },<br>});<br><br>export default orchestrateAgentsModule;</pre><ul><li>Update this module in backend/src/modules/index.ts</li></ul><pre>export * from &#39;./authProvidersModule&#39;;<br>export * from &#39;./rbacDynamicPluginsModule&#39;;<br>export * from &#39;./healthcheck&#39;;<br>export { default as orchestrateAgentsModule } from &#39;./orchestrateAgentsModule&#39;;</pre><ul><li>Also add this in backend/src/index.ts</li></ul><pre>const backend = createBackend();<br>backend.add(import(&#39;@backstage/plugin-mcp-actions-backend&#39;));<br>backend.add(orchestrateAgentsModule);</pre><ul><li>Verify how many Agents are running in watsony Orchestrate.</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*87YU2FmJb3UJCvOGvN592g.png" /></figure><ul><li>Restart the backend and in the log you see agents are getting retrieved.</li></ul><pre>yarn --cwd packages/backend start<br>25-12-26T20:09:07.768Z search warn Postgres search engine is not supported, skipping registration of search-backend-module-pg <br>2025-12-26T20:09:07.769Z search info Added DefaultCatalogCollatorFactory collator factory for type software-catalog <br>2025-12-26T20:09:07.800Z events info Database is not PostgreSQL, using memory store <br>2025-12-26T20:09:07.828Z auth info Enabled Provider Factories : {} <br>2025-12-26T20:09:07.828Z auth info Configuring &quot;database&quot; as KeyStore provider <br>2025-12-26T20:09:07.839Z catalog info Created new signing key 84808d16-2e99-4c47-9ac3-db467a12a916 <br>2025-12-26T20:09:07.844Z catalog info Task worker starting: orchestrate-agents-refresh, {&quot;version&quot;:2,&quot;cadence&quot;:&quot;PT5M&quot;,&quot;timeoutAfterDuration&quot;:&quot;PT2M&quot;} task=&quot;orchestrate-agents-refresh&quot;<br>2025-12-26T20:09:07.846Z catalog info Performing database migration <br>2025-12-26T20:09:07.851Z auth info Configuring auth provider: guest <br>2025-12-26T20:09:07.898Z catalog info [orchestrate] Catalog connection established <br>2025-12-26T20:09:07.913Z permission warn RBAC backend plugin was disabled by application config permission.enabled: false <br>2025-12-26T20:09:07.913Z permission warn Permission backend started with permissions disabled. Enable permissions by setting permission.enabled=true. <br>2025-12-26T20:09:07.914Z search info Starting all scheduled search tasks. <br>2025-12-26T20:09:07.915Z backstage info Plugin initialization complete, newly initialized: &#39;mcp-actions&#39;, &#39;healthcheck&#39;, &#39;proxy&#39;, &#39;search&#39;, &#39;events&#39;, &#39;dynamic-plugins-info&#39;, &#39;scalprum&#39;, &#39;translations&#39;, &#39;licensed-users-info&#39;, &#39;scaffolder&#39;, &#39;auth&#39;, &#39;user-settings&#39;, &#39;app&#39;, &#39;catalog&#39;, &#39;permission&#39; type=&quot;initialization&quot;<br>2025-12-26T20:09:07.917Z catalog info Task worker starting: catalog_orphan_cleanup, {&quot;version&quot;:2,&quot;cadence&quot;:&quot;PT30S&quot;,&quot;timeoutAfterDuration&quot;:&quot;PT24S&quot;} task=&quot;catalog_orphan_cleanup&quot;<br>2025-12-26T20:09:07.994Z search info Task worker starting: search_index_software_catalog, {&quot;version&quot;:2,&quot;cadence&quot;:&quot;PT10M&quot;,&quot;initialDelayDuration&quot;:&quot;PT3S&quot;,&quot;timeoutAfterDuration&quot;:&quot;PT15M&quot;} task=&quot;search_index_software_catalog&quot;<br>2025-12-26T20:09:08.012Z rootHttpRouter info [2025-12-26T20:09:08.012Z] &quot;GET /api/catalog/.backstage/auth/v1/jwks.json HTTP/1.1&quot; 200 994 &quot;-&quot; &quot;node&quot; type=&quot;incomingRequest&quot; date=&quot;2025-12-26T20:09:08.012Z&quot; method=&quot;GET&quot; url=&quot;/api/catalog/.backstage/auth/v1/jwks.json&quot; status=200 httpVersion=&quot;1.1&quot; userAgent=&quot;node&quot; contentLength=994 trace_id=&quot;a72d0940a1c70fecb743ef63af08bf73&quot; span_id=&quot;7d6f13f16fc31de2&quot; trace_flags=&quot;01&quot;<br>2025-12-26T20:09:08.019Z rootHttpRouter info [2025-12-26T20:09:08.019Z] &quot;GET /api/catalog/.backstage/auth/v1/jwks.json HTTP/1.1&quot; 200 994 &quot;-&quot; &quot;jose/v5.10.0&quot; type=&quot;incomingRequest&quot; date=&quot;2025-12-26T20:09:08.019Z&quot; method=&quot;GET&quot; url=&quot;/api/catalog/.backstage/auth/v1/jwks.json&quot; status=200 httpVersion=&quot;1.1&quot; userAgent=&quot;jose/v5.10.0&quot; contentLength=994 trace_id=&quot;a72d0940a1c70fecb743ef63af08bf73&quot; span_id=&quot;486b2adc863ceed0&quot; trace_flags=&quot;01&quot;<br>2025-12-26T20:09:08.021Z rootHttpRouter info [2025-12-26T20:09:08.021Z] &quot;PUT /api/events/bus/v1/subscriptions/catalog.catalog HTTP/1.1&quot; 201 0 &quot;-&quot; &quot;node&quot; type=&quot;incomingRequest&quot; date=&quot;2025-12-26T20:09:08.021Z&quot; method=&quot;PUT&quot; url=&quot;/api/events/bus/v1/subscriptions/catalog.catalog&quot; status=201 httpVersion=&quot;1.1&quot; userAgent=&quot;node&quot; trace_id=&quot;a72d0940a1c70fecb743ef63af08bf73&quot; span_id=&quot;1e6f01db3b4dc9b0&quot; trace_flags=&quot;01&quot;<br>2025-12-26T20:09:10.845Z catalog info [orchestrate] Triggering first sync after catalog startup <br>2025-12-26T20:09:10.845Z catalog info [orchestrate] Starting sync run... <br>2025-12-26T20:09:10.845Z catalog info [orchestrate] Fetching agents from http://localhost:4321/api/v1/orchestrate/agents?include_hidden=false <br>2025-12-26T20:09:10.887Z catalog info [orchestrate] Normalized to 3 agent(s) <br>2025-12-26T20:09:10.887Z catalog info [orchestrate] Received 3 agent(s) from Orchestrate <br>2025-12-26T20:09:10.887Z catalog info [orchestrate] Synced 3 agent(s) into the catalog </pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*vItwPEo86TlLmy9jDjX1mg.png" /></figure><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=06942a5d05ab" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Models-as-a-Service: How to Deploy and Govern LLM APIs on OpenShift AI]]></title>
            <link>https://medium.com/@shrishs/models-as-a-service-how-to-deploy-and-govern-llm-apis-on-openshift-ai-ed965acc7036?source=rss-748916ab05a5------2</link>
            <guid isPermaLink="false">https://medium.com/p/ed965acc7036</guid>
            <category><![CDATA[red-hat-openshift-ai]]></category>
            <category><![CDATA[llm]]></category>
            <category><![CDATA[gateway]]></category>
            <category><![CDATA[model-as-a-service]]></category>
            <category><![CDATA[kuadrant]]></category>
            <dc:creator><![CDATA[Shrishs]]></dc:creator>
            <pubDate>Tue, 09 Dec 2025 19:24:29 GMT</pubDate>
            <atom:updated>2025-12-09T19:28:12.829Z</atom:updated>
            <content:encoded><![CDATA[<p>OpenShift AI 3.0, Red Hat has introduced <a href="https://docs.redhat.com/en/documentation/red_hat_openshift_ai_self-managed/3.0/html-single/release_notes/index#developer-preview-features_relnotes"><strong>Model-as-a-Service</strong></a><strong> (MaaS)</strong> — a powerful pattern that solves a very real enterprise problem: how do you serve LLMs efficiently <em>and</em> enforce proper governance?</p><p>In this article, we break down how MaaS works behind the scenes and walk through one of its most important features: applying <strong>user-level rate limits</strong> to manage and protect your model workloads.</p><p><strong>Gateway API:</strong><br> Starting with Openshift <a href="https://docs.redhat.com/en/documentation/openshift_container_platform/4.19/html-single/release_notes/index#ocp-4-19-networking-gateway-api-controller_release-notes">release 4.19</a>, managing ingress traffic with Gateway API is fully supported and GA. Gateway API offers a standardized, Kubernetes-native way to control L4 and L7 traffic across the cluster.</p><p><strong>Red Hat Connectivity Link:</strong><br> Built on the <a href="https://kuadrant.io/">Kuadrant</a> project, Connectivity Link provides the control plane for configuring Gateway API–based ingress. It adds Kubernetes-native policies for TLS, authentication, authorization, rate limiting, DNS, and multicluster health checks.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/669/1*Wp2Qm1BQMVGDJ5yQIi27GQ.png" /></figure><p><strong>llm-d</strong></p><p>Distributed Inference with llm-d is a Kubernetes-native, open-source framework designed for serving large language models (LLMs) at scale. One can use Distributed Inference with llm-d to simplify the deployment of generative AI, focusing on high performance and cost-effectiveness across various hardware accelerators.It Integrates into a standard Kubernetes environment, where it leverages specialized components like the Envoy proxy to handle networking and routing</p><p>In the next section we will</p><ul><li>Deploy the LLM Model using <em>Distributed Inference Server with llm-d</em></li><li>How does the Gateway object is created.</li><li>How the RateLimiting Policy on this Gateway object(HTTPRoute) is applied using Kuadrant.</li></ul><h4>Deploy the model using Distributed Inference</h4><p>Prerequisites are defined <a href="https://docs.redhat.com/en/documentation/red_hat_openshift_ai_self-managed/3.0/html/deploying_models/deploying_models#deploying-models-using-distributed-inference_rhoai-user">here</a>.</p><ul><li>Configuring authentication with Red Hat Connectivity Link.</li></ul><pre>apiVersion: kuadrant.io/v1beta1<br>kind: Kuadrant<br>metadata:<br>  name: kuadrant<br>  namespace: kuadrant-system<br>spec: {}</pre><pre>oc get pods -n kuadrant-system<br>NAME                                  READY   STATUS    RESTARTS   AGE<br>authorino-56df78f5fb-sbtvs            1/1     Running   0          77s<br>limitador-limitador-546f7c5bb-bw4rx   1/1     Running   0          77s<br><br>oc get svc -n kuadrant-system<br>NAME                                TYPE        CLUSTER-IP       EXTERNAL-IP   PORT(S)              AGE<br>authorino-authorino-authorization   ClusterIP   172.30.25.47     &lt;none&gt;        50051/TCP,5001/TCP   111s<br>authorino-authorino-oidc            ClusterIP   172.30.165.172   &lt;none&gt;        8083/TCP             111s<br>authorino-controller-metrics        ClusterIP   172.30.12.42     &lt;none&gt;        8080/TCP             111s<br>limitador-limitador                 ClusterIP   None             &lt;none&gt;        8080/TCP,8081/TCP    111s</pre><ul><li>Create the Gateway Class and Gateway</li></ul><pre>apiVersion: gateway.networking.k8s.io/v1<br>kind: GatewayClass<br>metadata:<br>  annotations:<br>    platform.opendatahub.io/instance.name: default-gateway<br>    platform.opendatahub.io/instance.uid: 49d4f4d1-087f-4802-ac15-6cd2536747a5<br>    platform.opendatahub.io/type: OpenShift AI Self-Managed<br>    platform.opendatahub.io/version: 3.0.0<br>  name: data-science-gateway-class<br>  labels:<br>    platform.opendatahub.io/part-of: gatewayconfig<br>spec:<br>  controllerName: openshift.io/gateway-controller/v1</pre><pre>apiVersion: gateway.networking.k8s.io/v1<br>kind: Gateway<br>metadata:<br>  name: openshift-ai-inference<br>  namespace: openshift-ingress<br>  labels:<br>    istio.io/rev: openshift-gateway<br>    platform.opendatahub.io/part-of: gatewayconfig<br>spec:<br>  gatewayClassName: data-science-gateway-class<br>  listeners:<br>    - allowedRoutes:<br>        namespaces:<br>          from: Selector<br>          selector:<br>            matchExpressions:<br>              - key: kubernetes.io/metadata.name<br>                operator: In<br>                values:<br>                  - openshift-ingress<br>                  - redhat-ods-applications<br>                  - demo-ai<br>      name: https<br>      port: 443<br>      protocol: HTTPS<br>      tls:<br>        certificateRefs:<br>          - group: &#39;&#39;<br>            kind: Secret<br>            name: default-gateway-tls<br>        mode: Terminate</pre><ul><li>A corresponding svc is also created.</li></ul><pre>bash-3.2$ oc get svc openshift-ai-inference-data-science-gateway-class -n openshift-ingress<br>NAME                                                TYPE           CLUSTER-IP      EXTERNAL-IP                                                             PORT(S)                         AGE<br>openshift-ai-inference-data-science-gateway-class   LoadBalancer   172.30.43.253   ac1be10b222d54eb18ca5b48ad5d98b2-95298441.us-east-2.elb.amazonaws.com   15021:30782/TCP,443:32544/TCP   9h</pre><ul><li>Above two objects automatically creates the authpolicy</li></ul><pre>apiVersion: kuadrant.io/v1<br>kind: AuthPolicy<br>metadata:<br>  name: openshift-ai-inference-authn<br>  namespace: openshift-ingress<br>  ownerReferences:<br>    - apiVersion: gateway.networking.k8s.io/v1<br>      blockOwnerDeletion: true<br>      controller: true<br>      kind: Gateway<br>      name: openshift-ai-inference<br>  labels:<br>    app.kubernetes.io/component: llminferenceservice-policies<br>    app.kubernetes.io/managed-by: odh-model-controller<br>    app.kubernetes.io/name: llminferenceservice-auth<br>    app.kubernetes.io/part-of: llminferenceservice<br>spec:<br>  rules:<br>    authentication:<br>      kubernetes-user:<br>        credentials: {}<br>        kubernetesTokenReview:<br>          audiences:<br>            - &#39;https://kubernetes.default.svc&#39;<br>        metrics: false<br>        priority: 0<br>    authorization:<br>      inference-access:<br>        kubernetesSubjectAccessReview:<br>          authorizationGroups:<br>            expression: auth.identity.user.groups<br>          resourceAttributes:<br>            group:<br>              value: serving.kserve.io<br>            name:<br>              expression: &#39;request.path.split(&quot;/&quot;)[2]&#39;<br>            namespace:<br>              expression: &#39;request.path.split(&quot;/&quot;)[1]&#39;<br>            resource:<br>              value: llminferenceservices<br>            subresource: {}<br>            verb:<br>              value: get<br>          user:<br>            expression: auth.identity.user.username<br>        metrics: false<br>        priority: 1<br>  targetRef:<br>    group: gateway.networking.k8s.io<br>    kind: Gateway<br>    name: openshift-ai-inference</pre><ul><li>Deploy a Model -<em>RedHatAI/Qwen3–8B-FP8-dynamic</em></li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*psUIkpx7wWIMaDrbyV6K4g.png" /></figure><ul><li>Select the Serving Runtime as -<em>Distributed Inference Server with llm-d</em></li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*OkUc_fCotpLDOsKU8qOOKA.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*cTTcw8NG7Lubf9-23mYaVw.png" /></figure><ul><li>Distributed Inference Server with llm-d creates LLMInferenceService</li></ul><pre>apiVersion: serving.kserve.io/v1alpha1<br>kind: LLMInferenceService<br>metadata:<br>  annotations:<br>    opendatahub.io/connections: secret-mt1uy1<br>    opendatahub.io/hardware-profile-name: gpu-small<br>    opendatahub.io/hardware-profile-namespace: redhat-ods-applications<br>    opendatahub.io/model-type: generative<br>    openshift.io/display-name: RedHatAI/Qwen3-8B-FP8-dynamic<br>  name: redhataiqwen3-8b-fp8-dynamic<br>  namespace: demo-ai<br>  finalizers:<br>    - serving.kserve.io/llmisvc-finalizer<br>  labels:<br>    opendatahub.io/dashboard: &#39;true&#39;<br>    opendatahub.io/genai-asset: &#39;true&#39;<br>spec:<br>  model:<br>    name: redhataiqwen3-8b-fp8-dynamic<br>    uri: &#39;hf://RedHatAI/Qwen3-8B-FP8-dynamic&#39;<br>  replicas: 1<br>  router:<br>    gateway: {}<br>    route: {}<br>    scheduler: {}<br>  template:<br>    containers:<br>      - name: main<br>        resources:<br>          limits:<br>            cpu: &#39;2&#39;<br>            memory: 8Gi<br>            nvidia.com/gpu: &#39;1&#39;<br>          requests:<br>            cpu: &#39;2&#39;<br>            memory: 8Gi<br>            nvidia.com/gpu: &#39;1&#39;<br>    tolerations:<br>      - effect: NoSchedule<br>        key: nvidia.com/gpu<br>        operator: Exists<br>status:<br>  addresses:<br>    - url: &#39;https://ac1be10b222d54eb18ca5b48ad5d98b2-95298441.us-east-2.elb.amazonaws.com/demo-ai/redhataiqwen3-8b-fp8-dynamic&#39;<br>  conditions:<br>    - lastTransitionTime: &#39;2025-12-09T10:01:42Z&#39;<br>      severity: Info<br>      status: &#39;True&#39;<br>      type: HTTPRoutesReady<br>    - lastTransitionTime: &#39;2025-12-09T10:01:42Z&#39;<br>      severity: Info<br>      status: &#39;True&#39;<br>      type: InferencePoolReady<br>    - lastTransitionTime: &#39;2025-12-09T18:48:00Z&#39;<br>      severity: Info<br>      status: &#39;True&#39;<br>      type: MainWorkloadReady<br>    - lastTransitionTime: &#39;2025-12-09T09:51:17Z&#39;<br>      severity: Info<br>      status: &#39;True&#39;<br>      type: PresetsCombined<br>    - lastTransitionTime: &#39;2025-12-09T18:48:00Z&#39;<br>      status: &#39;True&#39;<br>      type: Ready<br>    - lastTransitionTime: &#39;2025-12-09T10:01:42Z&#39;<br>      status: &#39;True&#39;<br>      type: RouterReady<br>    - lastTransitionTime: &#39;2025-12-09T09:51:54Z&#39;<br>      severity: Info<br>      status: &#39;True&#39;<br>      type: SchedulerWorkloadReady<br>    - lastTransitionTime: &#39;2025-12-09T18:48:00Z&#39;<br>      status: &#39;True&#39;<br>      type: WorkloadsReady<br>  observedGeneration: 2<br>  url: &#39;https://ac1be10b222d54eb18ca5b48ad5d98b2-95298441.us-east-2.elb.amazonaws.com/demo-ai/redhataiqwen3-8b-fp8-dynamic&#39;</pre><ul><li>Corrsponding HTTPRoute is also created</li></ul><pre>apiVersion: gateway.networking.k8s.io/v1<br>kind: HTTPRoute<br>metadata:<br>  name: redhataiqwen3-8b-fp8-dynamic-kserve-route<br>  namespace: demo-ai<br>  ownerReferences:<br>    - apiVersion: serving.kserve.io/v1alpha1<br>      blockOwnerDeletion: true<br>      controller: true<br>      kind: LLMInferenceService<br>      name: redhataiqwen3-8b-fp8-dynamic<br>      uid: ea1c8538-30a2-43b5-b515-50c8bb97de80<br>  labels:<br>    app.kubernetes.io/component: llminferenceservice-router<br>    app.kubernetes.io/name: redhataiqwen3-8b-fp8-dynamic<br>    app.kubernetes.io/part-of: llminferenceservice<br>spec:<br>  parentRefs:<br>    - group: gateway.networking.k8s.io<br>      kind: Gateway<br>      name: openshift-ai-inference<br>      namespace: openshift-ingress<br>  rules:<br>    - backendRefs:<br>        - group: inference.networking.x-k8s.io<br>          kind: InferencePool<br>          name: redhataiqwen3-8b-fp8-dynamic-inference-pool<br>          port: 8000<br>          weight: 1<br>      filters:<br>        - type: URLRewrite<br>          urlRewrite:<br>            path:<br>              replacePrefixMatch: /v1/completions<br>              type: ReplacePrefixMatch<br>      matches:<br>        - path:<br>            type: PathPrefix<br>            value: /demo-ai/redhataiqwen3-8b-fp8-dynamic/v1/completions<br>      timeouts:<br>        backendRequest: 0s<br>        request: 0s<br>    - backendRefs:<br>        - group: inference.networking.x-k8s.io<br>          kind: InferencePool<br>          name: redhataiqwen3-8b-fp8-dynamic-inference-pool<br>          port: 8000<br>          weight: 1<br>      filters:<br>        - type: URLRewrite<br>          urlRewrite:<br>            path:<br>              replacePrefixMatch: /v1/chat/completions<br>              type: ReplacePrefixMatch<br>      matches:<br>        - path:<br>            type: PathPrefix<br>            value: /demo-ai/redhataiqwen3-8b-fp8-dynamic/v1/chat/completions<br>      timeouts:<br>        backendRequest: 0s<br>        request: 0s<br>    - backendRefs:<br>        - group: &#39;&#39;<br>          kind: Service<br>          name: redhataiqwen3-8b-fp8-dynamic-kserve-workload-svc<br>          port: 8000<br>          weight: 1<br>      filters:<br>        - type: URLRewrite<br>          urlRewrite:<br>            path:<br>              replacePrefixMatch: /<br>              type: ReplacePrefixMatch<br>      matches:<br>        - path:<br>            type: PathPrefix<br>            value: /demo-ai/redhataiqwen3-8b-fp8-dynamic<br>      timeouts:<br>        backendRequest: 0s<br>        request: 0s<br>status:<br>  parents:<br>    - conditions:<br>        - lastTransitionTime: &#39;2025-12-09T09:51:17Z&#39;<br>          message: &#39;Object affected by AuthPolicy [openshift-ingress/openshift-ai-inference-authn]&#39;<br>          observedGeneration: 1<br>          reason: Accepted<br>          status: &#39;True&#39;<br>          type: kuadrant.io/AuthPolicyAffected<br>        - lastTransitionTime: &#39;2025-12-09T14:51:10Z&#39;<br>          message: &#39;Object affected by RateLimitPolicy [demo-ai/redhataiqwen3-8b-fp8-dynamic-rl]&#39;<br>          observedGeneration: 1<br>          reason: Accepted<br>          status: &#39;True&#39;<br>          type: kuadrant.io/RateLimitPolicyAffected<br>      controllerName: kuadrant.io/policy-controller<br>      parentRef:<br>        group: gateway.networking.k8s.io<br>        kind: Gateway<br>        name: openshift-ai-inference<br>        namespace: openshift-ingress<br>    - conditions:<br>        - lastTransitionTime: &#39;2025-12-09T10:01:32Z&#39;<br>          message: Route was valid<br>          observedGeneration: 1<br>          reason: Accepted<br>          status: &#39;True&#39;<br>          type: Accepted<br>        - lastTransitionTime: &#39;2025-12-09T09:51:17Z&#39;<br>          message: All references resolved<br>          observedGeneration: 1<br>          reason: ResolvedRefs<br>          status: &#39;True&#39;<br>          type: ResolvedRefs<br>      controllerName: openshift.io/gateway-controller/v1<br>      parentRef:<br>        group: gateway.networking.k8s.io<br>        kind: Gateway<br>        name: openshift-ai-inference<br>        namespace: openshift-ingress</pre><ul><li>Test the inference service</li></ul><pre>TOKEN=$(oc whoami -t)<br>curl -k     https://ac1be10b222d54eb18ca5b48ad5d98b2-95298441.us-east-2.elb.amazonaws.com/demo-ai/redhataiqwen3-8b-fp8-dynamic/v1/completions     -H &quot;Authorization: Bearer $TOKEN&quot;     -H &quot;Content-Type: application/json&quot;     -d &#39;{&quot;model&quot;:&quot;redhataiqwen3-8b-fp8-dynamic&quot;,&quot;prompt&quot;:&quot;capital of France&quot;,&quot;max_tokens&quot;:20}&#39;  <br>{&quot;choices&quot;:[{&quot;finish_reason&quot;:&quot;length&quot;,&quot;index&quot;:0,&quot;logprobs&quot;:null,&quot;prompt_logprobs&quot;:null,&quot;prompt_token_ids&quot;:null,&quot;stop_reason&quot;:null,&quot;text&quot;:&quot; is Paris, and the capital of Italy is Rome.  The capital of Spain is Madrid. &quot;,&quot;token_ids&quot;:null}],&quot;created&quot;:1765307782,&quot;id&quot;:&quot;cmpl-3308ccf3-dca3-494c-acef-3fa7f86ac282&quot;,&quot;kv_transfer_params&quot;:null,&quot;model&quot;:&quot;redhataiqwen3-8b-fp8-dynamic&quot;,&quot;object&quot;:&quot;text_completion&quot;,&quot;service_tier&quot;:null,&quot;system_fingerprint&quot;:null,&quot;usage&quot;:{&quot;completion_tokens&quot;:20,&quot;prompt_tokens&quot;:3,&quot;prompt_tokens_details&quot;:null,&quot;total_tokens&quot;:23}}</pre><h4>RateLimiting</h4><ul><li>Creat a RateLimitPolicy</li></ul><pre>apiVersion: kuadrant.io/v1<br>kind: RateLimitPolicy<br>metadata:<br>  name: redhataiqwen3-8b-fp8-dynamic-rl<br>  namespace: demo-ai<br>spec:<br>  targetRef:<br>    group: gateway.networking.k8s.io<br>    kind: HTTPRoute<br>    name: redhataiqwen3-8b-fp8-dynamic-kserve-route<br><br>  limits:<br>    per-user:<br>      rates:<br>        - limit: 5      # 5 requests per minute per user<br>          window: 1m<br>      counters:<br>        - expression: &quot;auth.identity.user.username&quot;</pre><ul><li>Try the request more tan 5 time in a min.And it will deny the request ny saying “<strong>Too Many Requests</strong>”</li></ul><pre>bash-3.2$ curl -k     https://ac1be10b222d54eb18ca5b48ad5d98b2-95298441.us-east-2.elb.amazonaws.com/demo-ai/redhataiqwen3-8b-fp8-dynamic/v1/completions     -H &quot;Authorization: Bearer $TOKEN&quot;     -H &quot;Content-Type: application/json&quot;     -d &#39;{&quot;model&quot;:&quot;redhataiqwen3-8b-fp8-dynamic&quot;,&quot;prompt&quot;:&quot;capital of France&quot;,&quot;max_tokens&quot;:20}&#39;  <br>Too Many Requests</pre><p>What’s Next: In the next article we will be configuring <a href="https://opendatahub-io.github.io/maas-billing/0.0.1/configuration-and-management/tier-configuration/#1-configure-tier-mapping">different Tiers</a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=ed965acc7036" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[OpenShift AI 3.0: Model Deployment, AI Asset Endpoints, and the Gen AI Playground]]></title>
            <link>https://medium.com/@shrishs/openshift-ai-3-0-model-deployment-ai-asset-endpoints-and-the-gen-ai-playground-9a2b6be6377d?source=rss-748916ab05a5------2</link>
            <guid isPermaLink="false">https://medium.com/p/9a2b6be6377d</guid>
            <category><![CDATA[large-language-models]]></category>
            <category><![CDATA[model]]></category>
            <category><![CDATA[openshift]]></category>
            <category><![CDATA[red-hat-openshift-ai]]></category>
            <category><![CDATA[kubernetes]]></category>
            <dc:creator><![CDATA[Shrishs]]></dc:creator>
            <pubDate>Tue, 02 Dec 2025 19:15:39 GMT</pubDate>
            <atom:updated>2025-12-02T19:15:39.409Z</atom:updated>
            <content:encoded><![CDATA[<p>OpenShift AI 3.0 brings some genuinely exciting new capabilities that reshape how teams discover, test, and interact with AI models on the platform. Two of the most impactful additions are the <strong>AI Available Assets</strong> page and the new <strong>Generative AI (GenAI) Playground</strong>, both designed to make model consumption dramatically easier for engineers and application developers.</p><p>In this article I will:</p><ul><li>Enable Gen AI Studio</li><li>Deploy the <strong>llama-31–8b-instruct model on OpenshiftAI 3.0</strong></li><li>Register it in AI Asset Point and Test in the playground</li></ul><h4>Enabling Gen AI Studio</h4><ul><li>Enable <em>genAiStudio: true in </em>OdhDashboardConfig as described in the <a href="https://docs.redhat.com/en/documentation/red_hat_openshift_ai_self-managed/3.0/html-single/experimenting_with_models_in_the_gen_ai_playground/index#cluster_administrator_prerequisites">document</a>.</li></ul><pre>apiVersion: opendatahub.io/v1alpha<br>kind: OdhDashboardConfig<br>metadata:<br>  name: odh-dashboard-config<br>  namespace: redhat-ods-applications<br>  labels:<br>    app: rhods-dashboard<br>    app.kubernetes.io/part-of: rhods-dashboard<br>    app.opendatahub.io/rhods-dashboard: &#39;true&#39;<br>    platform.opendatahub.io/part-of: dashboard<br>spec:<br>  dashboardConfig:<br>    genAiStudio: true</pre><ul><li>Enable the llamastackoperator in DataScienceCluster as decribe in the <a href="https://docs.redhat.com/en/documentation/red_hat_openshift_ai_self-managed/3.0/html/working_with_llama_stack/activating-the-llama-stack-operator_rag">docs</a>.</li></ul><pre>apiVersion: datasciencecluster.opendatahub.io/v2<br>kind: DataScienceCluster<br>metadata:<br>  labels:<br>    app.kubernetes.io/name: datasciencecluster<br>  name: default-dsc<br>spec:<br>  components:<br>    llamastackoperator:<br>      managementState: Managed</pre><ul><li>Now one can see the Gen AI Studio enabled.</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*mc6GX1t-3UMT67LeabMqcg.png" /></figure><h4>Deploy the <strong>llama-31–8b-instruct model</strong></h4><p>In my previous article <a href="https://medium.com/@shrishs/gpu-considerations-for-llm-deployment-on-openshift-ai-3-x-e8999d85591c">GPU Considerations for LLM Deployment on OpenShift AI 3.x</a> ,I have deployed it using the UI.In this article I will use the yaml definition.</p><ul><li>Creating a HardwareProfile remain same as in the <a href="https://medium.com/@shrishs/gpu-considerations-for-llm-deployment-on-openshift-ai-3-x-e8999d85591c">previous article</a>.</li><li>Create the ServingRuntime</li></ul><pre>apiVersion: serving.kserve.io/v1alpha1<br>kind: ServingRuntime<br>metadata:<br>  annotations:<br>    opendatahub.io/accelerator-name: &#39;&#39;<br>    opendatahub.io/apiProtocol: REST<br>    opendatahub.io/recommended-accelerators: &#39;[&quot;nvidia.com/gpu&quot;]&#39;<br>    opendatahub.io/runtime-version: v0.9.1.0<br>    opendatahub.io/serving-runtime-scope: global<br>    opendatahub.io/template-display-name: vLLM NVIDIA GPU ServingRuntime for KServe<br>    opendatahub.io/template-name: vllm-cuda-runtime<br>    openshift.io/display-name: redhataillama-31-8b-instruct<br>  name: redhataillama-31-8b-instruct<br>  labels:<br>    opendatahub.io/dashboard: &#39;true&#39;<br>spec:<br>  annotations:<br>    opendatahub.io/kserve-runtime: vllm<br>    prometheus.io/path: /metrics<br>    prometheus.io/port: &#39;8080&#39;<br>  containers:<br>    - args:<br>        - &#39;--port=8080&#39;<br>        - &#39;--model=/mnt/models&#39;<br>        - &#39;--served-model-name={{.Name}}&#39;<br>      command:<br>        - python<br>        - &#39;-m&#39;<br>        - vllm.entrypoints.openai.api_server<br>      env:<br>        - name: HF_HOME<br>          value: /tmp/hf_home<br>      image: &#39;registry.redhat.io/rhaiis/vllm-cuda-rhel9@sha256:ad756c01ec99a99cc7d93401c41b8d92ca96fb1ab7c5262919d818f2be4f3768&#39;<br>      name: kserve-container<br>      ports:<br>        - containerPort: 8080<br>          protocol: TCP<br>      volumeMounts:<br>        - mountPath: /dev/shm<br>          name: shm<br>  multiModel: false<br>  supportedModelFormats:<br>    - autoSelect: true<br>      name: vLLM<br>  volumes:<br>    - emptyDir:<br>        medium: Memory<br>        sizeLimit: 2Gi<br>      name: shm</pre><ul><li>Create the InferenceService.<strong><em>Give</em> <em>Attention to the labe</em></strong><em>l: opendatahub.io/genai-asset: ‘true’. </em>This ensures that Model is registered as AI Asset.</li></ul><pre>apiVersion: serving.kserve.io/v1beta1<br>kind: InferenceService<br>metadata:<br>  annotations:<br>    serving.kserve.io/stop: &#39;false&#39;<br>    security.opendatahub.io/enable-auth: &#39;false&#39;<br>    openshift.io/description: &#39;&#39;<br>    openshift.io/display-name: RedHatAI/Llama-3.1-8B-Instruct<br>    serving.kserve.io/deploymentMode: RawDeployment<br>    opendatahub.io/hardware-profile-namespace: redhat-ods-applications<br>    opendatahub.io/hardware-profile-name: gpu-small<br>    opendatahub.io/model-type: generative<br>  name: redhataillama-31-8b-instruct<br>  finalizers:<br>    - odh.inferenceservice.finalizers<br>    - inferenceservice.finalizers<br>  labels:<br>    networking.kserve.io/visibility: exposed<br>    opendatahub.io/dashboard: &#39;true&#39;<br>    opendatahub.io/genai-asset: &#39;true&#39;<br>spec:<br>  predictor:<br>    automountServiceAccountToken: false<br>    maxReplicas: 1<br>    minReplicas: 1<br>    model:<br>      args:<br>        - &#39;--max-model-len=65536&#39;<br>      modelFormat:<br>        name: vLLM<br>      name: &#39;&#39;<br>      resources:<br>        limits:<br>          cpu: &#39;2&#39;<br>          memory: 8Gi<br>          nvidia.com/gpu: &#39;1&#39;<br>        requests:<br>          cpu: &#39;2&#39;<br>          memory: 8Gi<br>          nvidia.com/gpu: &#39;1&#39;<br>      runtime: redhataillama-31-8b-instruct<br>      storageUri: &#39;oci://registry.redhat.io/rhelai1/modelcar-llama-3-1-8b-instruct-fp8-dynamic:1.5&#39;<br>    tolerations:<br>      - effect: NoSchedule<br>        key: nvidia.com/gpu<br>        operator: Exists</pre><ul><li>Verify it into Openshift Ai Console</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*v7L6Dcz5YSDMVfA8DmUlOA.png" /></figure><ul><li>Add to the PlayGround.This would create the playground in which you can communicate with the deployed model.</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*_pAU6Ps1BkUjPo2WlpfEgg.png" /></figure><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=9a2b6be6377d" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[GPU Considerations for LLM Deployment on OpenShift AI 3.x]]></title>
            <link>https://medium.com/@shrishs/gpu-considerations-for-llm-deployment-on-openshift-ai-3-x-e8999d85591c?source=rss-748916ab05a5------2</link>
            <guid isPermaLink="false">https://medium.com/p/e8999d85591c</guid>
            <category><![CDATA[kubernetes]]></category>
            <category><![CDATA[large-language-models]]></category>
            <category><![CDATA[vllm]]></category>
            <category><![CDATA[openshift]]></category>
            <category><![CDATA[red-hat-openshift-ai]]></category>
            <dc:creator><![CDATA[Shrishs]]></dc:creator>
            <pubDate>Mon, 01 Dec 2025 12:11:29 GMT</pubDate>
            <atom:updated>2025-12-01T12:11:29.784Z</atom:updated>
            <content:encoded><![CDATA[<p>Deploying Large Language Models (LLMs) in <strong>OpenShift AI</strong> isn’t as simple as attaching a GPU and pressing “deploy.”Modern models — especially long-context architectures like <strong>Meta-Llama-3.1 8B/70B</strong> — place very specific, often misunderstood demands on GPU memory.</p><p>In this article ,I will walks through <strong>exactly what you must check</strong> before deploying LLMs in OpenShift AI — starting from GPU discovery, then diving into memory architecture and real-world best practices.</p><h4>Verify GPU Availability on Your OpenShift Nodes</h4><pre>oc get csv -A |grep -i &quot;NVIDIA GPU Operator&quot;<br>nvidia-gpu-operator gpu-operator-certified.v25.10.0     NVIDIA GPU Operator                              25.10.0               gpu-operator-certified.v25.3.0                     Succeeded</pre><pre>oc describe node/ip-10-0-24-62.us-east-2.compute.internal | grep -A8 Capacity<br>Capacity:<br>  cpu:                8<br>  ephemeral-storage:  125238252Ki<br>  hugepages-1Gi:      0<br>  hugepages-2Mi:      0<br>  memory:             32499868Ki<br>  nvidia.com/gpu:     1<br>  pods:               250</pre><h4>Identify the GPU Model and VRAM Size</h4><ul><li>Go to the terminal of the node, run the <strong>nvidia-smi </strong>command.</li></ul><pre>$ nvidia-smi    <br>+-----------------------------------------------------------------------------------------+<br>| NVIDIA-SMI 580.95.05              Driver Version: 580.95.05      CUDA Version: 13.0     |<br>+-----------------------------------------+------------------------+----------------------+<br>| GPU  Name                 Persistence-M | Bus-Id          Disp.A | Volatile Uncorr. ECC |<br>| Fan  Temp   Perf          Pwr:Usage/Cap |           Memory-Usage | GPU-Util  Compute M. |<br>|                                         |                        |               MIG M. |<br>|=========================================+========================+======================|<br>|   0  NVIDIA A10G                    On  |   00000000:00:1E.0 Off |                    0 |<br>|  0%   32C    P0             89W /  300W |   21413MiB /  23028MiB |      0%      Default |<br>|                                         |                        |                  N/A |<br>+-----------------------------------------+------------------------+----------------------+<br><br>+-----------------------------------------------------------------------------------------+<br>| Processes:                                                                              |<br>|  GPU   GI   CI              PID   Type   Process name                        GPU Memory |<br>|        ID   ID                                                               Usage      |<br>|=========================================================================================|<br>|    0   N/A  N/A              86      C   VLLM::EngineCore                      21404MiB |<br>+-----------------------------------------------------------------------------------------+</pre><p>Above output tells you the model and how much VRAM you have.In the above example,I have already deployed a model in my cluster.</p><ul><li><strong>GPU model:</strong> A10G</li><li><strong>VRAM:</strong> ~23 GB</li></ul><h4>Understand GPU Memory Layout for LLMs (Weights vs Runtime vs KV Cache)</h4><p>Consider the example of deploying <a href="https://huggingface.co/meta-llama/Llama-3.1-8B-Instruct"><strong>Meta-Llama-3.1–8B-Instruct-FP8-dynamic</strong></a> model.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/949/1*SvLCowRDhkSsFalzxVsnCg.png" /></figure><ol><li><strong>Model Weights (FP8)</strong></li></ol><p>FP8 = 8-bit floating point → <strong>1 byte per parameter</strong>.</p><p><strong>Total Model Weight in Bytes</strong>=8B Param * 1 Byte= 8B Bytes~<strong>8 GB</strong></p><p>Once you deploy the model ,you can also verify it inside the model pod logs.</p><pre>(EngineCore_DP0 pid=86) INFO 12-01 06:54:20 [gpu_model_runner.py:2653] Model loading took 8.4915 GiB and 34.539810 seconds</pre><p><strong>2. KV Cache Dominates GPU Memory</strong></p><figure><img alt="From HuggingFace" src="https://cdn-images-1.medium.com/max/778/1*Xty4a8WLYDK_OZ-3UEt-jA.png" /></figure><p>In the above pic the default context length is <strong>128k.</strong> I did some hit &amp; trial with this model while deploying it .As you see in the below log ,each token requires ≈ <strong>128 KB of KV cache.</strong>There are some <a href="https://developer.nvidia.com/blog/mastering-llm-techniques-inference-optimization/">better way</a> of calculating<strong> </strong>the KV cache size.Here I am just taking an example.</p><pre>(EngineCore_DP0 pid=86) INFO 12-01 06:54:58 [gpu_worker.py:298] Available KV cache memory: 10.13 GiB<br>(EngineCore_DP0 pid=86) INFO 12-01 06:54:58 [kv_cache_utils.py:1087] GPU KV cache size: 82,992 tokens</pre><p>Based on the example <strong>each token requires ≈ 128 KB of KV cache ,</strong>I created the following table</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/713/1*_ZWjhy2-QA-nosmXuIIz-Q.png" /></figure><p>As my GPU VRAM is 23 GB and 8GB is reserved for the Model Weights ,It is clear that we can not use the default context length of 128K.Either we need to use some other GPU like <a href="https://www.colfax-intl.com/nvidia/nvidia-h100">NVIDIA-H100</a> ,which has 80GB VRAM or use the lower context length.</p><h4>The Most Important Setting in vLLM: --max-model-len</h4><p>Apart from Model Weight and KV Cache , We also need to consider that vLLM need memory for some other stuff also like <strong>graph capture.</strong></p><pre>(EngineCore_DP0 pid=86) INFO 12-01 06:55:09 [gpu_model_runner.py:3486] Graph capturing finished in 10 secs, took 0.66 GiB<br>(EngineCore_DP0 pid=86) INFO 12-01 06:55:09 [core.py:210] init engine (profile, create kv cache, warmup model) took 48.25 seconds</pre><p>Final Calculation:</p><ul><li><strong>Total VRAM ~ 23GB</strong></li><li><em>Model weight ~ 8GB</em></li><li><em>Considering context Length around 64K ~ 8GB</em></li></ul><p>Above configuration gives around 7GB for the vLLM and other buffers.Now we can proceed to Model deployment .To set the context Length around 64K ,It is required to pass following parameter.</p><pre>--max-model-len=65536</pre><h4>Let’s Deploy the Model:</h4><ul><li>Create a HardwareProfile</li></ul><pre>apiVersion: infrastructure.opendatahub.io/v1<br>kind: HardwareProfile<br>metadata:<br>  annotations:<br>    opendatahub.io/dashboard-feature-visibility: &#39;[]&#39;<br>    opendatahub.io/description: &#39;1x NVIDIA GPU, 2–4 CPUs, 8–16Gi memory.&#39;<br>    opendatahub.io/disabled: &#39;false&#39;<br>    opendatahub.io/display-name: gpu-small<br>    opendatahub.io/managed: &#39;false&#39;<br>  name: gpu-small<br>  namespace: redhat-ods-applications<br>  labels:<br>    app.kubernetes.io/part-of: hardwareprofile<br>    app.opendatahub.io/hardwareprofile: &#39;true&#39;<br>spec:<br>  identifiers:<br>    - defaultCount: 2<br>      displayName: CPU<br>      identifier: cpu<br>      maxCount: 4<br>      minCount: 2<br>      resourceType: CPU<br>    - defaultCount: 8Gi<br>      displayName: Memory<br>      identifier: memory<br>      maxCount: 16Gi<br>      minCount: 8Gi<br>      resourceType: Memory<br>    - defaultCount: 1<br>      displayName: GPU<br>      identifier: nvidia.com/gpu<br>      maxCount: 1<br>      minCount: 1<br>      resourceType: Accelerator<br>  scheduling:<br>    node:<br>      tolerations:<br>        - effect: NoSchedule<br>          key: nvidia.com/gpu<br>          operator: Exists<br>    type: Node</pre><ul><li>Go to Openshift AI — --&gt;Catalog →Llama-3.1–8B-Instruct</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*V_0WnqPDR4RrLGQBEP5Edg.png" /></figure><ul><li>Click Deploy Model</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*2JxcruhPWqq64qerJwCufw.png" /></figure><ul><li>Select the GPU ServingRuntime →Harware Profile →</li><li>Set — <strong>max-model-len=65536</strong> as configuration parameter</li><li>Deploy</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/753/1*ahMoEBqS3lGzXdkV4OBsGQ.png" /></figure><ul><li>Look into your project.</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*vMFDo-Nuoqmlt9T_fZfQKQ.png" /></figure><ul><li>curl $EXTERNAL_URL/v1/models</li></ul><pre>{<br>  &quot;object&quot;: &quot;list&quot;,<br>  &quot;data&quot;: [<br>    {<br>      &quot;id&quot;: &quot;redhataillama-31-8b-instruct&quot;,<br>      &quot;object&quot;: &quot;model&quot;,<br>      &quot;created&quot;: 1764590563,<br>      &quot;owned_by&quot;: &quot;vllm&quot;,<br>      &quot;root&quot;: &quot;/mnt/models&quot;,<br>      &quot;parent&quot;: null,<br>      &quot;max_model_len&quot;: 65536,<br>      &quot;permission&quot;: [<br>        {<br>          &quot;id&quot;: &quot;modelperm-81db68825c944c71802fbf085db0789a&quot;,<br>          &quot;object&quot;: &quot;model_permission&quot;,<br>          &quot;created&quot;: 1764590563,<br>          &quot;allow_create_engine&quot;: false,<br>          &quot;allow_sampling&quot;: true,<br>          &quot;allow_logprobs&quot;: true,<br>          &quot;allow_search_indices&quot;: false,<br>          &quot;allow_view&quot;: true,<br>          &quot;allow_fine_tuning&quot;: false,<br>          &quot;organization&quot;: &quot;*&quot;,<br>          &quot;group&quot;: null,<br>          &quot;is_blocking&quot;: false<br>        }<br>      ]<br>    }<br>  ]<br>}</pre><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=e8999d85591c" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Building a Self-Contained RHEL Bastion Host for PXE Booting MicroShift]]></title>
            <link>https://medium.com/@shrishs/building-a-self-contained-rhel-bastion-host-for-pxe-booting-microshift-4b534713c478?source=rss-748916ab05a5------2</link>
            <guid isPermaLink="false">https://medium.com/p/4b534713c478</guid>
            <category><![CDATA[kubernetes]]></category>
            <category><![CDATA[ostree]]></category>
            <category><![CDATA[bastion-host]]></category>
            <category><![CDATA[microshift]]></category>
            <category><![CDATA[pxe-boot]]></category>
            <dc:creator><![CDATA[Shrishs]]></dc:creator>
            <pubDate>Sun, 16 Nov 2025 19:25:20 GMT</pubDate>
            <atom:updated>2025-11-27T15:34:43.017Z</atom:updated>
            <content:encoded><![CDATA[<p>When deploying <strong>Red Hat MicroShift</strong> or <strong>RHEL for Edge</strong> in disconnected or air-gapped environments, a <strong>Bastion host</strong> acts as the control hub — providing DHCP, TFTP, and HTTP services for PXE boot, Kickstart automation .</p><p>In my previous articles ,I talked about how to created <a href="https://medium.com/@shrishs/embedding-microshift-in-a-rhel-for-edge-image-eff404023bfd">microshift for edge</a> and how to <a href="https://medium.com/@shrishs/embedding-user-applications-into-a-microshift-enabled-rhel-for-edge-image-9ea182abf859">deploy the applications</a>.</p><p>In this post, I’ll walk you through how to <strong>build a reusable Bastion ISO</strong> that automates the entire provisioning flow.</p><pre> <br>                     ┌────────────────────────────────────────────────────────┐<br>                     │                    Bastion Host                        │<br>                     │         (RHEL for Edge + MicroShift PXE Server)        │<br>                     │────────────────────────────────────────────────────────│<br>                     │  Core Services:                                        │<br>                     │   • dnsmasq → DHCP + PXE metadata + (built-in) TFTP    │<br>                     │   • httpd   → serves kernel/initrd, KS, OSTree, apps   │<br>                     │                                                        │<br>                     │                                                        │<br>                     │  Key HTTP Paths (/var/www/html/microshift):            │<br>                     │   images/{vmlinuz, initrd.img, install.img}            │<br>                     │   microshift-ostree.ks                                 │<br>                     │   ostree/repo/                                         │<br>                     │   applications/                                        │<br>                     │                                                        │<br>                     │  TFTP Root (/var/lib/tftpboot):                        │<br>                     │   pxelinux.0, pxelinux.cfg/default, microshift/{...}   │<br>                     │                                                        │<br>                     │  dnsmasq PXE config (/etc/dnsmasq.d/pxe.conf ):        │<br>                     │  ────────────────────────────────────────────────────  │<br>                     │  dhcp-range=${DHCP_START},${DHCP_END},${LEASE_TIME}    │<br>                     │  dhcp-boot=pxelinux.0   # PXE bootloader (BIOS/LEGACY) │<br>                     │  enable-tftp                                           │<br>                     │  tftp-root=/var/lib/tftpboot                           │<br>                     │  ────────────────────────────────────────────────────  │<br>                     └──────────────────────────┬─────────────────────────────┘<br>                                                │<br>                              DHCP lease + PXE info + HTTP fetches<br>                                                │<br>        ┌──────────────────────────────────────────────────────────────────────┐<br>        │                                                                      │<br>   ┌────────────────────────────┐                            ┌────────────────────────────┐<br>   │        PXE Client 1        │                            │        PXE Client 2        │<br>   │  (RHEL Edge + MicroShift)  │                            │  (RHEL Edge + MicroShift)  │<br>   └────────────────────────────┘                            └────────────────────────────┘<br>          │                                                                  │<br>          │ ① DHCP Request (Who gives me IP + boot info?)                   │<br>          │──────────────────────────────────────────────────────────────▶   │<br>          │                                                                  │<br>          │ ② DHCP Reply + boot file info (pxelinux.0)                      │<br>          ◀──────────────────────────────────────────────────────────────│   │<br>          │                                                                  │<br>          │ ③ TFTP download bootloader (pxelinux.0)                         │<br>          │──────────────────────────────────────────────────────────────▶   │<br>          │                                                                  │<br>          │ ④ Bootloader requests menu, kernel, initrd                      │<br>          │──────────────────────────────────────────────────────────────▶   │<br>          │                                                                  │<br>          │ ⑤ Installer starts &amp; downloads installer bits                   │<br>          │     from http://&lt;bastion-ip&gt;/microshift/                         │</pre><h4>Directory Layout</h4><ul><li>Here’s the directory structure we’ll work with:</li></ul><pre>bastion/<br>├── input-image/<br>│   └── rhel-9.6-x86_64-dvd.iso<br>├── kickstart.ks<br>├── mirror/<br>│   ├── microshift/<br>│   │   ├── microshift-ostree.ks<br>│   │   └── microshift-x86_64.iso<br>└── output-image/<br>    └── rhel9-6-bastion.iso</pre><p>Each folder plays a specific role:</p><ul><li><strong>input-image/</strong> → your RHEL base ISO</li><li><strong>mirror/microshift/</strong> → Microshift iso and kickstart file</li><li><strong>output-image/</strong> → where the final Bastion ISO will be generated</li></ul><h4>Prepare kickstart.ks</h4><ul><li><strong>Extract MicroShift ISO boot files and repo</strong>.This steps extract the following file from Microshift ISO image.<br>- vmlinuz: <strong>Linux kernel image</strong> used to boot the OS.<br>- initrd.img: <strong>init</strong>ial <strong>r</strong>am<strong>d</strong>isk: a <strong>temporary, minimal root filesystem</strong> that lives entirely in RAM.</li><li>Add the following in the kickstart file.</li></ul><pre>%post --nochroot --log=/mnt/sysroot/root/ks-post-microshift.log --erroronfail<br>set -euo pipefail<br><br>ISO=&quot;/run/install/repo/mirror/microshift/microshift-x86_64.iso&quot;<br>ISODIR=&quot;/run/install/repo/mirror/microshift&quot;<br>DSTWEB=&quot;/mnt/sysroot/var/www/html/microshift&quot;<br>DSTTFTP=&quot;/mnt/sysroot/var/lib/tftpboot/microshift&quot;<br><br>if [ -f &quot;$ISO&quot; ]; then<br>  mkdir -p &quot;$DSTWEB&quot; &quot;$DSTTFTP&quot;<br>  cp &quot;$ISODIR&quot;/microshift-ostree.ks &quot;$DSTWEB&quot;/<br>  TMPDIR=&quot;$(mktemp -d)&quot;<br>  mount -o loop &quot;$ISO&quot; &quot;$TMPDIR&quot;<br><br>  # Use cp -a (rsync is NOT guaranteed in installer env)<br>  cp -a &quot;$TMPDIR&quot;/. &quot;$DSTWEB&quot;/<br><br>  # PXE kernel + initrd<br>  if [ -f &quot;$TMPDIR/images/pxeboot/vmlinuz&quot; ] &amp;&amp; [ -f &quot;$TMPDIR/images/pxeboot/initrd.img&quot; ]; then<br>    cp -f &quot;$TMPDIR/images/pxeboot/vmlinuz&quot;    &quot;$DSTTFTP/&quot;<br>    cp -f &quot;$TMPDIR/images/pxeboot/initrd.img&quot; &quot;$DSTTFTP/&quot;<br>  else<br>    echo &quot;ERROR: pxeboot files not found in ISO&quot; &gt;&gt; /mnt/sysroot/root/ks-post-microshift.log<br>  fi<br><br>  umount &quot;$TMPDIR&quot;<br>  rmdir &quot;$TMPDIR&quot;<br>else<br>  echo &quot;ERROR: MicroShift ISO not found at $ISO&quot; &gt;&gt; /mnt/sysroot/root/ks-post-microshift.log<br>fi<br>%end</pre><ul><li>This results in creating the following structure.</li></ul><pre>[root@bastion ~]# ls -al /var/lib/tftpboot/<br>total 392<br>drwxr-xr-x.  4 root root    136 Nov 12 09:35 .<br>drwxr-xr-x. 28 root root   4096 Nov 12 11:00 ..<br>-rw-r--r--.  1 root root 115844 Nov 12 09:35 ldlinux.c32<br>-rw-r--r--.  1 root root 179456 Nov 12 09:35 libcom32.c32<br>-rw-r--r--.  1 root root  23508 Nov 12 09:35 libutil.c32<br>-rw-r--r--.  1 root root  26148 Nov 12 09:35 menu.c32<br>drwxr-xr-x.  2 root root     39 Nov 12 09:35 microshift<br>-rw-r--r--.  1 root root  42686 Nov 12 09:35 pxelinux.0<br>drwxr-xr-x.  2 root root     21 Nov 12 09:35 pxelinux.cfg<br><br>[root@bastion ~]# ls -al /var/lib/tftpboot/microshift/<br>total 176316<br>drwxr-xr-x. 2 root root        39 Nov 12 09:35 .<br>drwxr-xr-x. 4 root root       136 Nov 12 09:35 ..<br>-rw-r--r--. 1 root root 165619497 Nov 12 09:35 initrd.img<br>-rw-r--r--. 1 root root  14923896 Nov 12 09:35 vmlinu</pre><ul><li><strong>Create Syslinux bootloader &amp; MicroShift-only PXE menu</strong><br>- specifying the kernel file location to load(microshift/vmlinuz), <em>relative to the TFTP root</em> (/var/lib/tftpboot).<br>- Where to find the <strong>installer runtime (</strong><strong>install.img)</strong>.It is running http web server <a href="http://${bastion_ip}/microshift">http://${BASTION_IP}/microshift</a></li></ul><pre>[root@bastion ~]# ls -al /var/www/html/microshift<br>total 16<br>drwxr-xr-x. 7 root root  164 Nov 12 11:16 .<br>drwxr-xr-x. 3 root root   24 Nov 12 09:35 ..<br>drwxr-xr-x. 3 1001 1002   45 Oct  6 08:57 applications<br>-rwxr--r--. 1 root root   55 Oct  7 15:05 .discinfo<br>drwxr-xr-x. 3 root root   18 Oct  7 14:49 EFI<br>drwxr-xr-x. 3 1001 1002  163 Oct  7 15:05 images<br>drwxr-xr-x. 2 root root  175 Oct  7 15:05 isolinux<br>-rw-r--r--. 1 1001 1002 3627 Oct  7 14:36 kickstart.ks<br>-rw-r--r--. 1 root root 4022 Nov 12 11:16 microshift-ostree.ks<br>-rw-r--r--. 1 root root  107 Oct  7 15:05 osbuild.ks<br>drwxr-xr-x. 3 root root   18 Oct  7 15:05 ostree</pre><p>Add the following in the kickstart file.</p><pre>%post --log=/root/ks-post-pxelinux.log --erroronfail<br>set -e<br><br># Copy syslinux PXE bits<br>install -D -m0644 /usr/share/syslinux/pxelinux.0   /var/lib/tftpboot/pxelinux.0 || true<br>install -D -m0644 /usr/share/syslinux/menu.c32     /var/lib/tftpboot/menu.c32   || true<br>install -D -m0644 /usr/share/syslinux/ldlinux.c32  /var/lib/tftpboot/ldlinux.c32 || true<br>install -D -m0644 /usr/share/syslinux/libcom32.c32 /var/lib/tftpboot/libcom32.c32 || true<br>install -D -m0644 /usr/share/syslinux/libutil.c32  /var/lib/tftpboot/libutil.c32  || true<br>mkdir -p /var/lib/tftpboot/pxelinux.cfg<br><br># Detect bastion IP to embed in the menu (fallback to first IP)<br>BASTION_IP=&quot;$(ip -4 route get 1.1.1.1 | awk &#39;{for(i=1;i&lt;=NF;i++) if($i==&quot;src&quot;){print $(i+1); exit}}&#39;)&quot; || true<br>[ -n &quot;$BASTION_IP&quot; ] || BASTION_IP=&quot;$(hostname -I | awk &#39;{print $1}&#39;)&quot;<br><br># Create PXE menu (MicroShift only)<br>cat &gt;/var/lib/tftpboot/pxelinux.cfg/default &lt;&lt;EOF<br>DEFAULT menu.c32<br>PROMPT 0<br>TIMEOUT 50<br>MENU TITLE Bastion PXE Menu<br><br>LABEL microshift<br>  MENU LABEL Install RHEL for Edge + MicroShift<br>  KERNEL microshift/vmlinuz<br>  APPEND initrd=microshift/initrd.img ip=dhcp rd.neednet=1 inst.stage2=http://${BASTION_IP}/microshift inst.ks=http://${BASTION_IP}/microshift/microshift-ostree.ks<br>EOF<br><br># Enable services to start on next boot<br>systemctl enable httpd<br>systemctl enable tftp.socket<br><br># Open firewall for HTTP/TFTP using offline tool (installer environment)<br>if command -v firewall-offline-cmd &gt;/dev/null 2&gt;&amp;1; then<br>  firewall-offline-cmd --add-service=http<br>  firewall-offline-cmd --add-service=tftp<br>fi<br>%end</pre><ul><li><strong>dnsmasq: Bastion provides DHCP + TFTP for PXE (MicroShift):</strong>This create the dnsmasq file specifying<br>- DHCP Range<br>- TFTP Service (tftp-root=/var/lib/tftpboot)</li></ul><pre>%post --log=/root/ks-post-dnsmasq.log --erroronfail<br>set -euo pipefail<br><br># ---- EDIT THESE to match your network ----<br>IFACE=&quot;$(nmcli -t -f DEVICE,STATE dev | awk -F: &#39;$2==&quot;connected&quot;{print $1; exit}&#39;)&quot; || true<br>[ -n &quot;$IFACE&quot; ] || IFACE=&quot;eth0&quot;<br><br># DHCP pool on your PXE VLAN / subnet<br>DHCP_START=&quot;192.168.122.100&quot;<br>DHCP_END=&quot;192.168.122.200&quot;<br>LEASE_TIME=&quot;12h&quot;<br><br># Default gateway &amp; DNS handed to PXE clients<br># (Set these to your site router/DNS; fallback to bastion IP if unknown)<br>BASTION_IP=&quot;$(ip -4 addr show dev &quot;$IFACE&quot; | awk &#39;/inet /{print $2}&#39; | cut -d/ -f1)&quot;<br>ROUTER=&quot;${ROUTER:-$BASTION_IP}&quot;<br>DNS=&quot;${DNS:-$BASTION_IP}&quot;<br><br># Write dnsmasq PXE config (BIOS/legacy path)<br>cat &gt;/etc/dnsmasq.d/pxe.conf &lt;&lt;EOF<br># Listen only on the PXE interface<br>interface=${IFACE}<br>bind-interfaces<br><br># DHCP address pool<br>dhcp-range=${DHCP_START},${DHCP_END},${LEASE_TIME}<br>dhcp-option=3,${ROUTER}     # default gateway<br>dhcp-option=6,${DNS}        # DNS server<br><br># PXE bootloader for BIOS clients<br>dhcp-boot=pxelinux.0<br><br># TFTP service<br>enable-tftp<br>tftp-root=/var/lib/tftpboot<br><br># Helpful logging<br>log-dhcp<br>EOF<br><br># Below line is to ensure that above dnsmasq is the only tftp provider<br>systemctl disable --now tftp.socket tftp.service || true<br><br># Enable dnsmasq; it will start next boot (safer during install)<br>systemctl enable dnsmasq<br><br><br># Open firewall for DHCP/TFTP/HTTP using offline tool (installer phase)<br>if command -v firewall-offline-cmd &gt;/dev/null 2&gt;&amp;1; then<br>  firewall-offline-cmd --add-service=dhcp<br>  firewall-offline-cmd --add-service=tftp<br>  firewall-offline-cmd --add-service=http<br>fi<br><br><br># Fix TFTP file permissions and SELinux labels<br>chmod 0644 /var/lib/tftpboot/microshift/{vmlinuz,initrd.img} || true<br>restorecon -RF /var/lib/tftpboot || true<br><br>echo &quot;[ks] dnsmasq PXE (DHCP+TFTP) configured on ${IFACE} with pool ${DHCP_START}-${DHCP_END}, router ${ROUTER}, dns ${DNS}&quot;<br>%end</pre><ul><li>Build the bastion Host</li></ul><pre>sudo mkksiso --add mirror/ --ks kickstart.ks input-image/rhel-9.6-x86_64-dvd.iso output-image/rhel9-6-bastion.iso</pre><ul><li>Deploy the Bastion Host</li></ul><pre>sudo virt-install --name bastion-vm --ram 8096 --vcpus 4 --disk size=20,format=qcow2 --cdrom output-image/rhel9-6-bastion.iso --os-variant rhel9.0 --graphics vnc --noautoconsole</pre><ul><li>Login to above vm</li></ul><pre>ssh admin@$BASTION_HOST<br>password: admin123</pre><ul><li>change the bastion host ip</li></ul><pre>[admin@bastion ~]$ sudo grep 192.168 /var/www/html/microshift/microshift-ostree.ks<br>ostreesetup --osname=rhel --remote=edge --url=http://192.168.122.132/microshift/ostree/repo --ref=rhel/9/x86_64/edge --nogpg<br>BASE=&quot;http://192.168.122.132/microshift&quot;   # &lt;-- hardcoded for now<br>BASE=&quot;http://192.168.122.132/microshift&quot;   # </pre><ul><li>Edit the above file and change the above IP with the BASTION_HOST</li></ul><pre>sudo systemctl restart httpd<br>sudo systemctl restart dnsmasq</pre><p><strong>Verify the PXE boot : </strong>Create a new VM.Both the VM needs to on same network.</p><pre>sudo virt-install --connect qemu:///system   --name microshift-pxe-client   --ram 8096   --vcpus 6   --disk size=12   --network network=default   --os-variant rhel9.0   --boot hd,network,menu=on   --noautoconsole</pre><ul><li>Above VM will be deployed via PXE boot( — boot hd,network).First time it will be booted via network where it finds the PXE boot server and once installation is done .it boots the microshift from hd.</li></ul><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=4b534713c478" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Embedding User Applications into a MicroShift-Enabled RHEL for Edge Image]]></title>
            <link>https://medium.com/@shrishs/embedding-user-applications-into-a-microshift-enabled-rhel-for-edge-image-9ea182abf859?source=rss-748916ab05a5------2</link>
            <guid isPermaLink="false">https://medium.com/p/9ea182abf859</guid>
            <category><![CDATA[kubernetes]]></category>
            <category><![CDATA[edge-computing]]></category>
            <category><![CDATA[microshift]]></category>
            <category><![CDATA[openshift]]></category>
            <category><![CDATA[ostree]]></category>
            <dc:creator><![CDATA[Shrishs]]></dc:creator>
            <pubDate>Sun, 16 Nov 2025 17:02:41 GMT</pubDate>
            <atom:updated>2025-11-27T15:35:13.179Z</atom:updated>
            <content:encoded><![CDATA[<p><a href="https://medium.com/@shrishs/embedding-microshift-in-a-rhel-for-edge-image-eff404023bfd"><strong>In my previous post</strong></a><strong>, we walked through how to build a MicroShift-enabled RHEL for Edge image using Image Builder and OSTree. </strong>Now that you have a working edge image with MicroShift integrated, it’s time to take the next step: embedding your own application directly into that image<strong>.</strong> In this post, we’ll break down exactly how to package, include, and deliver your custom application as part of the ISO file— so every edge device boots with your software already built in and ready to run .</p><p>In this articles ,we are using kustomization for creating the application.At every start, MicroShift searches the following manifest directories for Kustomize manifest files:</p><ul><li>/etc/microshift/manifests</li><li>/etc/microshift/manifests.d/*</li><li>/usr/lib/microshift/</li><li>/usr/lib/microshift/manifests.d/*</li></ul><p>MicroShift automatically runs the equivalent of the kubectl apply -k command to apply the manifests to the node .</p><h4>Application Packaging</h4><ul><li>Create the following application structure.</li></ul><pre>├── applications<br>│   ├── kustomization.yaml<br>│   └── myapp<br>│       └── myapp.yaml<br>├── images<br>│   └── images.ini</pre><ul><li>kustomization.yaml</li></ul><pre>$ cat applications/kustomization.yaml <br>apiVersion: kustomize.config.k8s.io/v1beta1<br>kind: Kustomization<br>resources:<br>  - myapp/myapp.yaml</pre><ul><li>myapp.yaml: This is a sample application to deploy</li></ul><pre>cat applications/myapp/myapp.yaml <br>apiVersion: v1<br>kind: Namespace<br>metadata:<br>  name: myapp<br>---<br>apiVersion: apps/v1<br>kind: Deployment<br>metadata:<br>  name: frontend<br>  namespace: myapp<br>spec:<br>  replicas: 1<br>  selector: {matchLabels: {app: frontend}}<br>  template:<br>    metadata: {labels: {app: frontend}}<br>    spec:<br>      containers:<br>      - name: frontend<br>        image: registry.redhat.io/jboss-webserver-6/jws61-openjdk21-openshift-rhel8:6.1.2<br>        imagePullPolicy: IfNotPresent</pre><ul><li>we also need to provide the image the required for the application(jws61-openjdk21-openshift-rhel8:6.1.2) as we dont have internet connectivity at the edge.These files need to be provided as tar file</li></ul><pre>cat images/images.ini <br>registry.redhat.io/jboss-webserver-6/jws61-openjdk21-openshift-rhel8:6.1.2</pre><p>One can automate the process by reading the images from the above file.</p><pre>      export REGISTRY_AUTHFILE=/root/.config/containers/auth.json<br>      mkdir -p images<br>      while IFS= read -r IMG || [ -n &quot;$IMG&quot; ]; do<br>        case &quot;$IMG&quot; in &#39;&#39;|\#*) continue;; esac<br>        echo &quot;Pulling $IMG&quot;<br>        sudo podman pull &quot;$IMG&quot;<br>        OUT=&quot;images/$(printf &#39;%s&#39; &quot;$IMG&quot; | sed &#39;s#[/:@]#_#g&#39;).tar&quot;<br>        echo &quot;Saving to $OUT&quot;<br>        sudo podman save --format docker-archive -o &quot;$OUT&quot; &quot;$IMG&quot;<br>      done &lt; images/images.ini</pre><ul><li>We inject the above structure(images,applications and kickstart.ks) using mkksiso, producing a customized microshift-x86_64.iso that installs RHEL for Edge fully automatically.</li></ul><pre>sudo mkksiso --add ./images --add ./applications --ks kickstart.ks ${BUILDID}-installer.iso microshift-x86_64.iso</pre><ul><li>Test the image by creating a virtual machine.</li></ul><pre>sudo virt-install --name microshift-vm --ram 4096 --vcpus 2 --disk size=13,format=qcow2 --cdrom /var/lib/libvirt/boot/microshift-x86_64.iso --os-variant rhel9.0 --network network=default --graphics vnc --noautoconsole</pre><ul><li>Login to Microshift &amp;verify the status of Microshift</li></ul><pre>ssh admin@VM_IP<br>password: Microshift123!</pre><pre>[admin@localhost ~]$ ls -al /etc/microshift/manifests<br>-rw-r--r--. 1 root root  95 Oct  6 04:57 kustomization.yaml<br>drwxr-xr-x. 2 root root  24 Nov 10 05:17 myapp</pre><ul><li>Verify if the pods in myapp namespace is running successfully</li></ul><pre>sudo -i<br>export KUBECONFIG=/var/lib/microshift/resources/kubeadmin/kubeconfig</pre><pre>oc get pods -A<br>NAMESPACE                              NAME                                       READY   STATUS    RESTARTS   AGE<br>kube-system                            csi-snapshot-controller-74795dc54f-bwnnz   1/1     Running   0          4m42s<br>myapp                                  frontend-fb8c4bbd8-7kvnx                   1/1     Running   0          4m22s<br>openshift-dns                          dns-default-n7nvd                          2/2     Running   0          4m17s<br>openshift-dns                          node-resolver-gk457                        1/1     Running   0          4m34s<br>openshift-ingress                      router-default-59f4f4c47b-b6g57            1/1     Running   0          4m41s<br>openshift-operator-lifecycle-manager   catalog-operator-7496db5f56-cks7c          1/1     Running   0          4m31s<br>openshift-operator-lifecycle-manager   olm-operator-754d9fccf4-b55cf              1/1     Running   0          4m31s<br>openshift-ovn-kubernetes               ovnkube-master-87mvq                       4/4     Running   0          4m34s<br>openshift-ovn-kubernetes               ovnkube-node-wskxt                         1/1     Running   0          4m34s<br>openshift-service-ca                   service-ca-755884c9cf-d5knh                1/1     Running   0          4m41s<br>openshift-storage                      lvms-operator-65d4bdf577-zjzmr             1/1     Running   0          4m45s<br>openshift-storage</pre><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=9ea182abf859" width="1" height="1" alt="">]]></content:encoded>
        </item>
    </channel>
</rss>