<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Prabhat Racherla on Medium]]></title>
        <description><![CDATA[Stories by Prabhat Racherla on Medium]]></description>
        <link>https://medium.com/@prabhatracherla?source=rss-a6205bc08f7e------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/proxy/1*TGH72Nnw24QL3iV9IOm4VA.png</url>
            <title>Stories by Prabhat Racherla on Medium</title>
            <link>https://medium.com/@prabhatracherla?source=rss-a6205bc08f7e------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Sat, 21 Nov 2026 07:49:17 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@prabhatracherla/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[Run & Manage Florence LLM Locally]]></title>
            <link>https://medium.com/@prabhatracherla/run-manage-florence-llm-locally-2e754598fa25?source=rss-a6205bc08f7e------2</link>
            <guid isPermaLink="false">https://medium.com/p/2e754598fa25</guid>
            <category><![CDATA[vision-model]]></category>
            <category><![CDATA[localhost]]></category>
            <category><![CDATA[chatbots]]></category>
            <category><![CDATA[llm]]></category>
            <category><![CDATA[ai]]></category>
            <dc:creator><![CDATA[Prabhat Racherla]]></dc:creator>
            <pubDate>Fri, 09 Jan 2026 15:49:01 GMT</pubDate>
            <atom:updated>2026-06-26T14:47:43.527Z</atom:updated>
            <content:encoded><![CDATA[<p>This blog post provides a deep dive into building a production-ready AI stack using Microsoft’s Florence-2 Image Vision AI, supported by a robust service layer for secrets, storage, and observability — built with 100% Open Source software and designed to run locally</p><p><strong>From Notebook to Production: Building a Robust AI Stack for Microsoft Florence-2</strong></p><p>Deploying Vision-Language Models (VLMs) often feels like a choice between two extremes: running a messy local script or paying for expensive, opaque cloud services. This project demonstrates a middle ground — a high-performance, containerized, and production-grade wrapper for Microsoft Florence-2 that you can run on your own hardware.</p><p>By decoupling the inference engine from the web interfaces and utilizing modern infrastructure tools (SeaweedFS, Logfire, Redis), we create a modular architecture that is both scalable and easy to manage.</p><p><strong>The Architecture: Application vs. Infrastructure</strong></p><p>The system is divided into two primary interfaces and a set of backend infrastructure services.</p><h4>The Interfaces</h4><ul><li><strong>FastAPI Backend:</strong> A programmatic REST API (/v1/predict) ideal for integrating into other applications. It features a flexible store_image flag, allowing you to choose between persistent storage (returning S3 URLs) or transient, memory-only inference (returning Base64 strings).</li><li><strong>Chainlit Frontend:</strong> A user-friendly, interactive chat interface for real-time image analysis.</li></ul><p><strong>The Engine Room (Infrastructure)</strong></p><ul><li><strong>SeaweedFS:</strong> High-performance, S3-compatible object storage used to persist input and output images, featuring native Object Lifecycle Management for auto-purging old blobs.</li><li><strong>Logfire:</strong> A unified observability platform (by Pydantic) for structured logging and monitoring, providing Grafana-like insights.</li><li><strong>Redis:</strong> A message broker that enables a singleton model worker to serve multiple concurrent API or Chat requests, drastically reducing memory overhead.</li></ul><p>Key Enhancements: Why This Stack?</p><h4>1. Singleton Model Worker &amp; Scalable Backend</h4><p>Unlike typical implementations that load the model into every API worker, this stack uses a dedicated model_worker.py. Only one instance of the 1.6GB Florence-2 model resides in memory. You can scale your API throughput by increasing the API_WORKER_COUNT without multiplying your RAM usage.</p><h4>2. Universal Hardware Support (CPU, AMD ROCm, &amp; GPU)</h4><p>This wrapper is hardware-agnostic.</p><ul><li><strong>CPU:</strong> Run on any standard laptop or server with system RAM.</li><li><strong>AMD ROCm:</strong> Fully supported with automatic hardware detection tasks to map GPU render nodes and GFX versions, allowing for hardware-accelerated inference on consumer AMD GPUs.</li></ul><h4>3. Efficient Two-Stage Docker Builds</h4><p>AI models have massive dependencies (PyTorch, Transformers). This project uses a <strong>Two-Stage Build Strategy</strong>:</p><ul><li>Dockerfile.base: Contains heavy dependencies (PyTorch, CUDA/ROCM libraries), built once.</li><li>Dockerfile: A lightweight layer containing only application logic, saving hours of build time during development.</li></ul><h4>4. Intelligent Storage Management</h4><p>Local environments often suffer from rapid disk depletion. This stack uses <strong>SeaweedFS</strong> with configurable BUCKET_TTL, automating the cleanup of old inference results so your local storage remains lean.</p><p><strong>Quick Start</strong></p><p>Once the stack is up, you have a p<strong>Configure:</strong> Create a .env file based on the provided template (including your Logfire token if remote monitoring is desired).</p><ol><li><strong>Hardware Config:</strong> If on AMD/Linux, run the “Florence: Auto-Detect Hardware Config” VS Code task to automatically map your GPU permissions.</li><li><strong>Build:</strong> Run docker build -f Dockerfile.base -t florence-base:latest . followed by the application build.</li><li><strong>Launch:</strong> Use docker compose up -d (for GPU) or docker compose -f docker-compose.cpu.yaml up -d (for CPU/RAM-only).</li></ol><p><strong>Demo &amp; Usage</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*t0UdL2AK_mqE9ckXU9zqoQ.png" /><figcaption>Chainlit Demo: Task given was Object Detection. You can see it detects table, Nintendo Switch, bottle, table, curtains</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*XWkFc_tFRVxdv4D6gSrHGg.png" /><figcaption>Rest API: This feature enables this project get linked with other projects.</figcaption></figure><p>Credits &amp; References</p><p>This project was built upon the foundations laid by the <a href="https://github.com/askaresh/MS-Florence2/tree/main/app">original MS-Florence2 app</a>. The goal was to take that powerful engine and wrap it in a “production-grade” chassis and also to enable this project in machines owned by people who can’t afford costly Nvidia GPU but have 16Gb+ RAM</p><p>Check out the full source code here:</p><ul><li><a href="https://github.com/rb1811/FastAPI-Wrapper-for-MS-Florence">Application Layer: FastAPI-Wrapper-for-MS-Florence</a></li><li><a href="https://github.com/rb1811/infra-monitoring">Service Layer: Infra-Monitoring</a></li></ul><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=2e754598fa25" width="1" height="1" alt="">]]></content:encoded>
        </item>
    </channel>
</rss>