<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Sharath Pai on Medium]]></title>
        <description><![CDATA[Stories by Sharath Pai on Medium]]></description>
        <link>https://medium.com/@sharathpai107?source=rss-ecc09fa8060a------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/0*aPpmQjncOnPmE9D5</url>
            <title>Stories by Sharath Pai on Medium</title>
            <link>https://medium.com/@sharathpai107?source=rss-ecc09fa8060a------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Sun, 22 Nov 2026 10:29:34 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@sharathpai107/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[Building a Resilient Scraper for Quick-Commerce Product Data]]></title>
            <link>https://medium.com/eightinity/building-a-resilient-scraper-for-quick-commerce-product-data-4618d9c0b4b2?source=rss-ecc09fa8060a------2</link>
            <guid isPermaLink="false">https://medium.com/p/4618d9c0b4b2</guid>
            <category><![CDATA[python]]></category>
            <category><![CDATA[rate-limiting]]></category>
            <category><![CDATA[tls]]></category>
            <category><![CDATA[web-scraping]]></category>
            <category><![CDATA[anti-bot]]></category>
            <dc:creator><![CDATA[Sharath Pai]]></dc:creator>
            <pubDate>Thu, 11 Jun 2026 12:03:31 GMT</pubDate>
            <atom:updated>2026-06-11T12:03:31.313Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*e-0KjuwcZm5psaRrsLhV_g.png" /></figure><p>When building a food product intelligence database, we needed to collect roughly 100,000 product pages across two major quick-commerce platforms. Both platforms enforce strict rate limits, and building a scraper that stays within those limits while completing a multi-day collection run takes more than a simple retry loop.</p><p>Here’s the full stack we landed on, what failed along the way, and why each layer matters.</p><p>— -</p><h3><strong>A Note on Responsible Data Collection</strong></h3><p>This project was scoped to publicly accessible product listing pages, limited to food and grocery items available to any user browsing the platform without authentication. We respected platform rate limits, kept request rates well below what a human browsing session would produce, collected no personal or user data, and ensured all collected data was used for internal product research only. If you’re running a similar project, check the platform’s terms of service and robots.txt before proceeding.</p><h3><strong>The Problem With Naive Scrapers</strong></h3><p>The first version was simple: a for loop, requests.get(), write to JSONL. It got blocked within seconds. One platform returns 429 immediately if you make more than a handful of requests per minute from the same IP. The other is subtler, it serves a valid-looking HTML page but strips all the product data from the RSC payload.</p><p>Both problems share the same root cause: <strong>requests that don&#39;t match a browser&#39;s network fingerprint are treated as automated traffic and either rate-limited or served degraded responses</strong>.</p><p>— -</p><h4><strong>Layer 1: TLS Fingerprinting</strong></h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*QRx6tQV7YslCcul7ugeIIA.png" /></figure><p>The first and most impactful fix wasn’t retry logic at all. It was curl_cffi.</p><p>Modern platforms inspect your TLS handshake — the cipher suites offered, their order, and your HTTP/2 SETTINGS frame — and compare them against known browser fingerprints (called JA3/JA4). Python’s requests library uses the OS TLS stack, which produces a fingerprint that doesn’t match any real browser. The result is degraded or blocked responses before the application layer even processes your URL.</p><p>curl_cffi wraps libcurl compiled with BoringSSL and produces browser-compatible TLS behavior:</p><pre>from curl_cffi import requests as creqs<br><br>session = creqs.Session(impersonate=&quot;chrome110&quot;)<br>resp = session.get(&quot;https://example-commerce.com/product/some-product/pvid/12345&quot;)</pre><p>This alone took our success rate on one platform from 0% to ~85%. The page started returning full RSC payloads with productInfo, ingredients, nutrition data, everything. Same Python code, just a different HTTP client underneath.</p><p>The second platform was similar. Their internal API returns a 403 with requests and a full JSON payload with curl_cffi.</p><p>— -</p><h4><strong>Layer 2: Exponential Backoff With Global Cooldown</strong></h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*nGaDLKbM7onXTbD4QdgIqg.png" /></figure><p>Even with the right TLS fingerprint, we’d still hit 429s under load. The retry logic needed to be smarter than a fixed sleep.</p><pre>import threading, time, random<br><br>_global_cooldown_until: float = 0.0<br>_global_cooldown_lock = threading.Lock()<br><br>def _set_global_cooldown(seconds: float) -&gt; None:<br>    global _global_cooldown_until<br>    with _global_cooldown_lock:<br>        _global_cooldown_until = max(<br>            _global_cooldown_until,<br>            time.monotonic() + seconds<br>        )<br><br>def get_page(url: str) -&gt; str:<br>    for attempt in range(4):<br>        # Honour global cooldown before each attempt<br>        remaining = _global_cooldown_until - time.monotonic()<br>        if remaining &gt; 0:<br>            time.sleep(remaining)<br><br>        r = session.get(url, timeout=20)<br><br>        if r.status_code == 429:<br>            wait = (2 ** attempt) * 2 + random.uniform(0, 1)<br>            _set_global_cooldown(wait)  # pause ALL threads<br>            time.sleep(wait)<br>            continue<br><br>        if r.status_code &gt;= 500:<br>            wait = (2 ** attempt) + random.uniform(0, 1)<br>            time.sleep(wait)<br>            continue<br><br>        r.raise_for_status()<br>        return r.text<br><br>    raise Exception(f&quot;Max retries exceeded for {url}&quot;)</pre><p>The key insight is the <strong>global cooldown</strong>. We ran 16 concurrent threads. Without it, when one thread received a 429, the other 15 would continue sending requests for the next few seconds, extending the backoff window and increasing the chance of further rate-limiting. Broadcasting the cooldown to all threads immediately reduces total request pressure while the platform recovers.</p><p>The jitter (random.uniform(0, 1)) prevents all 16 threads from waking up simultaneously after the cooldown, which would recreate the same burst.</p><p>— -</p><h4><strong>Layer 3: Hard Timeouts for Hanging Connections</strong></h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*DWuW_Tt-6raA86dyAr4CKA.png" /></figure><p>curl_cffi’s timeout parameter governs socket inactivity. If a server begins responding and then stalls mid-stream, the request never times out. This silently blocks a worker thread indefinitely.</p><p>We wrapped every request in a ThreadPoolExecutor to enforce a Python-level hard deadline:</p><pre>from concurrent.futures import ThreadPoolExecutor, TimeoutError as FuturesTimeout<br><br>_req_executor = ThreadPoolExecutor(max_workers=16)<br><br>def _get_with_hard_timeout(session, url, headers, timeout_sec=20):<br>    fut = _req_executor.submit(session.get, url, headers=headers, timeout=timeout_sec)<br>    try:<br>        return fut.result(timeout=timeout_sec + 2)<br>    except FuturesTimeout:<br>        fut.cancel()<br>        raise Exception(f&quot;Hard timeout after {timeout_sec}s on {url}&quot;)</pre><p>This converts silent stalls into retryable exceptions, keeping all 16 workers active.</p><p>— -</p><h4><strong>Layer 4: Concurrency Tuning</strong></h4><p>We started at 8 workers, then bumped to 16. The sweet spot depended on the platform:</p><p>- <strong>Platform A</strong>: 16 workers, ~99.9% first-pass success rate. The global cooldown and backoff kept burst traffic low enough to stay within the rate limit window.</p><p>- <strong>Platform B</strong>: 16 workers, ~85% first-pass success rate. More aggressive rate limiting on burst traffic meant more retries. After retries, final coverage reached ~97%.</p><p>The working formula: <strong>concurrency × requests_per_second &lt; rate limit threshold</strong>, with a safety margin. We estimated the threshold empirically — start conservatively, watch the 429 rate, reduce concurrency if it exceeds ~20%.</p><p>— -</p><h4><strong>Layer 5: Geography and Infrastructure</strong></h4><p>For the cloud portion of the collection (handled through an actor-based setup), the problem shifted. curl_cffi isn’t available in Node.js, so we used a full browser via Playwright. A subtler issue emerged: without pinning the proxy country, the residential proxy pool was routing requests through non-Indian IPs, and one platform returns empty product fields for requests originating outside India, even though those pages are publicly accessible.</p><pre>// Playwright actor config — pin to Indian residential proxies<br>const proxyConfiguration = await Actor.createProxyConfiguration({<br>    useApifyProxy: true,<br>    apifyProxyGroups: [&#39;RESIDENTIAL&#39;],<br>    apifyProxyCountry: &#39;IN&#39;,  // empty product data without this<br>});</pre><p>The broader principle: for platforms that serve region-specific content, your infrastructure must originate from the right geography. Residential proxies from the correct country also tend to produce fewer challenges than datacenter IPs, because the request profile is closer to normal consumer traffic.</p><p>Getting RSC product data in the cloud required a two-step approach: let Playwright navigate to the page fully (establishing a valid session), then use page.context().request.get() to fetch the raw RSC streaming payload. Using APIRequestContext alone — without the prior browser navigation — produces incomplete payloads because the platform checks session state when deciding how much data to include in the streaming response.</p><p>— -</p><h3><strong>Results</strong></h3><p>After all these layers:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/645f3c40048be8609827dc09e628823e/href">https://medium.com/media/645f3c40048be8609827dc09e628823e/href</a></iframe><p>The remaining ~3% gap on Platform B consists largely of combo or bundle products that don’t have individual SSR product pages, and a small number of products that were genuinely out of stock or unlisted during the collection window. These are not rate-limiting failures.</p><p>— -</p><h3><strong>The Hierarchy of Defences</strong></h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*N03Ct4_mlE51MN3BxLZ2uA.jpeg" /></figure><p>In order of impact:</p><p>1. <strong>Browser-compatible TLS</strong>: curl_cffi or Playwright instead of requests. This is the difference between receiving complete payloads and empty ones, before any retry logic applies.</p><p>2. <strong>Global cooldown across threads</strong>: when one thread is rate-limited, pause all of them immediately to reduce total pressure.</p><p>3. <strong>Exponential backoff with jitter</strong>: avoid fixed retry intervals; add randomness to prevent coordinated bursts across threads.</p><p>4. <strong>Hard timeouts</strong>: stalled connections silently kill worker throughput without them.</p><p>5. <strong>Right concurrency</strong>: more workers isn’t always faster — find the rate limit ceiling empirically.</p><p>6. <strong>Region-correct infrastructure</strong>: residential proxies from the appropriate country for platforms that gate content by geography.</p><p>The instinct when hitting 429s is to add sleep. The real fix is usually closer to the bottom of the stack: what does your request <em>look like</em> to the platform, and is it sending from the right place?</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=4618d9c0b4b2" width="1" height="1" alt=""><hr><p><a href="https://medium.com/eightinity/building-a-resilient-scraper-for-quick-commerce-product-data-4618d9c0b4b2">Building a Resilient Scraper for Quick-Commerce Product Data</a> was originally published in <a href="https://medium.com/eightinity">Eightinity</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[The Web Scraper’s Cheat Code Nobody Talks About]]></title>
            <link>https://medium.com/eightinity/the-web-scrapers-cheat-code-nobody-talks-about-2ca47efbdfcf?source=rss-ecc09fa8060a------2</link>
            <guid isPermaLink="false">https://medium.com/p/2ca47efbdfcf</guid>
            <category><![CDATA[data-collection]]></category>
            <category><![CDATA[seo]]></category>
            <category><![CDATA[web-scraping]]></category>
            <category><![CDATA[web-crawling]]></category>
            <category><![CDATA[understanding-sitemaps]]></category>
            <dc:creator><![CDATA[Sharath Pai]]></dc:creator>
            <pubDate>Tue, 09 Jun 2026 12:49:05 GMT</pubDate>
            <atom:updated>2026-06-09T13:02:17.245Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*-CK0jss0-VeajGBoIIqNqQ.png" /></figure><h4><strong>I thought I was done. I was not done.</strong></h4><p>I was scraping a major quick-commerce platform, one of those 10-minute grocery delivery apps, and I was feeling pretty good about myself. I’d built a category crawler that walked their navigation tree, pulled subcategory pages, handled pagination, and collected product URLs. It ran overnight. When I checked the output in the morning: <strong>5,287 products</strong>.</p><p>The platform’s own website says they carry 45,000+ products.</p><p>I had found 12% of the catalog. My crawler had essentially done the equivalent of walking into a supermarket, looking at the items on the endcap displays near the entrance, and declaring the inventory complete.</p><p>The naive fix is obvious: crawl harder. More categories, deeper pagination, more concurrency. But before I started that, I did what I should have done at the start. I checked robots.txt .</p><p>What I found there changed how I approach every scraping project now.</p><p>— -</p><h4><strong>Why Category Pages Are Lying to You</strong></h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*onQAGoqShNgBchishn6wIg.png" /></figure><p>Here’s the thing about category pages on e-commerce sites: they’re merchandised. They’re not database dumps. A product manager decided what shows up on the “Dairy &amp; Breakfast” landing page. An algorithm ranked those items by conversion rate. The pagination you’re crawling is a <strong><em>curated view</em></strong> of the catalog, not the catalog itself.</p><p>On this platform, a typical subcategory page surfaced 30–50 products. Some categories had “load more” buttons, some had pagination, some silently truncated at 48 items with no indication there was more. If you’re not paying close attention, you scrape those 48 items and move on, never knowing the category actually had 300 SKUs.</p><p>The other problem: subcategory depth. Navigation trees are designed for humans browsing at a high level. The “Snacks” category links to “Chips &amp; Crisps”, “Biscuits”, “Namkeen”, but there’s no nav link to “Protein Bars &gt; Whey-Based &gt; 40g Single Serve”. That stuff exists in the catalog. It just isn’t surfaced in the nav tree because it would make the UI unusable.</p><p>Category crawling finds what marketing decided to show. Sitemaps find what the database actually contains.</p><p>— -</p><h4><strong>The Sitemap Discovery</strong></h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*RkheKMFBfovktWV81eddJA.png" /></figure><p>robots.txt is the file at domain.com/robots.txt that tells web crawlers which paths they’re allowed to access. Every serious web developer knows it. Most scrapers check it to avoid getting blocked. What they often skip is the last section.</p><p>At the bottom of many robots.txt files, there’s a Sitemaps (sitemap.xml)directive that points to the XML sitemap. This is the file that sites submit to Google to say “here’s everything on my website, please index all of it.”</p><p>The platform’s `robots.txt` pointed to their sitemap index. The index contained dozens of shard files, each with up to 10,000 product URLs. The full catalog — every product the platform carries — was sitting there in static XML, no login required, no browser automation needed.</p><p>The numbers told a stark story. My overnight category crawler found a fraction of what the sitemap exposed in minutes — across two different platforms. The full comparison:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/1d4dad328f01dec8c7899b60f62febb0/href">https://medium.com/media/1d4dad328f01dec8c7899b60f62febb0/href</a></iframe><p>No Playwright. No JavaScript rendering. No waiting for React to hydrate. Just HTTP requests to static XML files. Something was also clearly off about the total counts — the sitemap had far more entries than the platform claims to carry. Something was very wrong, or very interesting. (It turned out to be both. More on that later.)</p><p>— -</p><h4><strong>Why Companies Maintain Sitemaps So Carefully</strong></h4><p>Here’s the reason this works so well: sitemaps are for Google.</p><p>Getting indexed by Google is existential for an e-commerce company. If your product pages aren’t in Google’s index, nobody finds them via search. So there’s an entire SEO operation inside these companies whose job is to make sure the sitemap is comprehensive, accurate, and regularly updated. Product added to catalog? It goes in the sitemap within hours. Product gets a new image? Sitemap gets a &lt;lastmod&gt; update.</p><p>Category pages are maintained by product teams and UI designers. Sitemaps are maintained by people who are religiously accountable to Google Search Console. These are different standards of quality.</p><p>When I’m scraping a site’s category navigation, I’m at the mercy of whatever UI decisions got made. When I’m reading their sitemap, I’m reading the file they trust Google with. That’s a much better source.</p><p>— -</p><h4><strong>The Implementation</strong></h4><p>First, find the sitemap. Don’t hardcode it, parse robots.txt:</p><pre>import re<br>from curl_cffi import requests as creqs<br><br>session = creqs.Session(impersonate=&quot;chrome110&quot;)<br><br>def find_sitemap(domain):<br>    r = session.get(f&quot;https://{domain}/robots.txt&quot;)<br>    for line in r.text.splitlines():<br>        if line.lower().startswith(&quot;sitemap:&quot;):<br>            return line.split(&quot;:&quot;, 1)[1].strip()<br>    return f&quot;https://{domain}/sitemap.xml&quot;</pre><p>I’m using curl_cffi here instead of the standard requestslibrary because it impersonates a real browser’s TLS fingerprint. Some CDNs will block requests that look like Python’s default HTTP client. curl_cffi with impersonate=&quot;chrome110&quot; gets through in almost every case I’ve encountered.</p><p>Next, the sitemap index gives you shard URLs:</p><pre>def get_shards(index_url):<br>    r = session.get(index_url)<br>    return re.findall(r&quot;&lt;loc&gt;(https://[^&lt;]+)&lt;/loc&gt;&quot;, r.text)</pre><p>I’m using regex instead of an XML parser here deliberately. Sitemaps are supposed to be well-formed XML, but in practice they sometimes aren’t, mismatched namespaces, encoding issues, stray characters. A simple regex that grabs everything between &lt;loc&gt; tags is more resilient than a parser that throws on malformed input.</p><p>Then parse each shard:</p><pre>def parse_shard(shard_url):<br>    r = session.get(shard_url)<br>    products = []<br>    for block in re.finditer(r&quot;&lt;url&gt;(.*?)&lt;/url&gt;&quot;, r.text, re.DOTALL):<br>        b = block.group(1)<br>        loc = re.search(r&quot;&lt;loc&gt;(https://[^&lt;]+)&lt;/loc&gt;&quot;, b)<br>        img = re.search(r&quot;&lt;image:loc&gt;([^&lt;]+)&lt;/image:loc&gt;&quot;, b)<br>        if loc:<br>            products.append({<br>                &quot;url&quot;: loc.group(1),<br>                &quot;image_url&quot;: img.group(1) if img else &quot;&quot;<br>            })<br>    return products</pre><p>Run this across all shards with light concurrency, I used concurrent.futures.ThreadFuturePoolExecutor with 8 workers, and you’re done in 4 minutes with a CSV of every product URL and image URL on the platform.</p><p>— -</p><h4><strong>What You Get for Free</strong></h4><p>Two things come out of the sitemap that you’d normally have to scrape from product pages:</p><p><strong>Image URLs:</strong> The &lt;image:loc&gt; extension tag is standard in Google’s image sitemap spec. Many platforms populate it for every product. So before you’ve made a single product page request, you already have the CDN URL for every product image. If your use case only needs images and slugs (say, you’re building a product catalog or doing image deduplication analysis), you might not need to scrape product pages at all.</p><p><strong>Category from shard path:</strong> One of the platforms I scraped structured their shard URLs like this:</p><pre>/sitemaps/products/dairy-breakfast/butter/sitemap-dairy-breakfast-butter.xml</pre><p>The category and subcategory are right there in the path. You can extract them with a single regex split, which means you get category labels for every product in the catalog without touching a single product page. That’s the kind of information that would normally require either scraping the PDP’s breadcrumbs or maintaining a mapping table by hand.</p><p>The other platform’s shard URLs were just numbered, sitemap-products-1.xml, sitemap-products-2.xml, no category encoded. But the image URLs were clean, and the product slugs themselves contained useful category signals.</p><p>— -</p><h4><strong>The Twist: 237,760 Is Not 237,760 Products</strong></h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*h7cCRHy2SP03M-a9uYT0QQ.png" /></figure><p>Remember how the platform claims 45,000 products but the sitemap had 237,760 entries? I said this was both wrong and interesting. Here’s the interesting part.</p><p>When I started deduplicating by slug, I noticed something: tomato-local appeared <strong>50 times</strong>. Not as duplicates, each entry had a different pvid (product variant ID) in the URL and a different image URL pointing to what was presumably a slightly different photo.</p><p>Quick-commerce platforms operate on a dark store model. They have hundreds of micro-warehouses across a city. Each dark store sources locally, the tomatoes at one location come from a different supplier than those at another. Each supplier relationship produces a distinct product entry with its own ID and its own product image.</p><p>When you deduplicate by slug, the sitemap inflation becomes obvious, the same product slug appears many times, each with a different variant ID and sometimes a different product image. Local produce is the worst offender; commodity products with supplier variants can appear dozens of times.</p><p>The math works out: the claimed catalog size × average dark-store variants per product ≈ sitemap entry count. It fits. (The full deduplication breakdown is in the table above.)</p><p>This isn’t a data quality problem. It’s a business architecture that leaked into the sitemap. Once you understand it, you can deduplicate by slug for product-level analysis, or keep the variant IDs if you need location-level granularity.</p><p>Combo packs are also counted separately, which inflates the number further. A “Butter 100g” and “Butter 3-Pack” are distinct entries. Reasonable. They’re different products, but worth knowing when you’re comparing catalog sizes.</p><p>— -</p><p><strong>The Two-Phase Pipeline</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*55YR1xo2xRdnDFxj_Sdq7Q.png" /></figure><p>Understanding this led me to restructure my entire scraping pipeline. The old approach was monolithic: crawl category pages, find product URLs, scrape PDPs, all in one pass. The new approach splits into two clean phases.</p><p><strong>Phase 1: Discovery.</strong> Download all sitemap shards and build a catalog CSV. Columns: product_id, url, slug, image_url. This is pure HTTP, no browser, no rate limiting needed (these are static XML files served from CDN). Takes 4 minutes, produces a complete map of everything on the platform.</p><p><strong>Phase 2: Extraction.</strong> Use the catalog CSV as input to a targeted PDP scraper. Now you’re making one request per product page to get price, stock status, ingredients, weight, and whatever else you need. This phase needs rate limiting and concurrency controls, but it’s operating on a known-good list of URLs rather than discovering them on the fly.</p><p>The separation matters for a few reasons. If Phase 2 fails halfway through (rate limited, IP blocked, whatever), you haven’t lost your discovery work. Restart from the CSV at the URL where you left off. Phase 1 can run daily to pick up new products without re-scraping all the PDPs. And the phases can be handed off: one person builds and maintains the catalog crawler, another works on the PDP extraction logic.</p><p>It also makes the product count question trivially answerable at any time. Want to know how many products a platform has today? Run Phase 1. Done in 4 minutes, no product page requests needed.</p><p>— -</p><h4><strong>The Caveats</strong></h4><p>Nothing this clean comes without gotchas.</p><p><strong>Stale entries.</strong> Sitemaps lag reality. We found product IDs that returned 404s when we actually requested the page, products that had been discontinued but whose sitemap entries hadn’t been cleaned up. Not many, maybe 1–2% of entries, but enough to account for in your pipeline. Always check HTTP status before treating a sitemap entry as live inventory.</p><p><strong>Not all sitemaps are product-level.</strong> Some sites have sitemaps that only list category pages or blog posts. A quick scan of the shard URLs or a sample of the &lt;loc&gt; values tells you immediately whether you’re dealing with product URLs or not. If the URLs look like /category/dairy-breakfast rather than /products/butter-100g/id/1234, you’re looking at a category sitemap and you’ll need to crawl from there.</p><p><strong>Sitemap ≠ ground truth. </strong>The sitemap tells you what exists. It doesn’t tell you if it’s in stock, what the current price is, or whether the product is available in your delivery zone. For a live price/stock feed, you still need product page scraping. The sitemap gives you the <strong><em>universe</em></strong> of things to scrape, what it doesn’t give you is the <strong><em>state</em></strong> of those things right now. Think of it as a highly reliable table of contents, not a data extract.</p><p>— -</p><h4><strong>Check robots.txt First. Every Time</strong></h4><p>I’ve changed my personal scraping checklist based on this. Before I write a single line of crawler code, I do three things:</p><p>Open robots.txt. Read the whole thing. Note every Sitemap: directive. Then open the sitemap index and look at the shard structure before deciding on an approach.</p><p>This 10-minute investigation would have saved me the time building a category crawler that found 12% of the catalog. For other platforms, the payoff will be different, but you’ll know immediately whether the sitemap approach is viable before you’ve committed to anything.</p><p>The SEO team has already done the hard work of comprehensively enumerating every URL on the site. They did it for Google. You’re just reading the same file.</p><p>It’s the closest thing to a cheat code I’ve found in web scraping.</p><p>— -</p><p>All scraping was done for product catalog research purposes. The code snippets above use curl_cffi for HTTP and standard Python re for XML parsing. The full pipeline including PDP extraction and deduplication is a bit more involved, happy to cover that in a follow-up post if there’s interest.</p><p><strong>Eightinity Engineering</strong><br>We share real-world AI and mobile engineering insights. Before you go:</p><p>👏 Show your support by <strong>clapping</strong> and <strong>following</strong> the <strong>author</strong><br>🧠 Discover more AI, OpenAI, and SAM model articles<br>📱 Explore iOS, Android, and UI engineering blogs<br>🚀 Learn how we build AI-powered products at Eightinity<br>🔔 Follow us: <a href="https://www.linkedin.com/company/eightinity-technologies"><strong>LinkedIn</strong></a> | <a href="https://x.com/8inityStudio"><strong>X (Twitter)</strong></a> | <a href="https://www.eightinity.in/"><strong>Website</strong></a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=2ca47efbdfcf" width="1" height="1" alt=""><hr><p><a href="https://medium.com/eightinity/the-web-scrapers-cheat-code-nobody-talks-about-2ca47efbdfcf">The Web Scraper’s Cheat Code Nobody Talks About</a> was originally published in <a href="https://medium.com/eightinity">Eightinity</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Performance Metrics for evaluating LLMs]]></title>
            <link>https://medium.com/eightinity/performance-metrics-for-evaluating-llms-2b32aa4297fd?source=rss-ecc09fa8060a------2</link>
            <guid isPermaLink="false">https://medium.com/p/2b32aa4297fd</guid>
            <category><![CDATA[performance-metrics]]></category>
            <category><![CDATA[ai-ml-services]]></category>
            <category><![CDATA[large-language-models]]></category>
            <category><![CDATA[generative-ai-use-cases]]></category>
            <dc:creator><![CDATA[Sharath Pai]]></dc:creator>
            <pubDate>Thu, 15 Jan 2026 10:37:08 GMT</pubDate>
            <atom:updated>2026-01-15T10:37:08.966Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*7rDFkImbrmfJG3SNy5IA1g.png" /></figure><p>LLMs don’t answer like traditional software. They improvise. They paraphrase. They hallucinate. They return JSON inside essays and essays inside JSON.<br>So the real challenge isn’t just <em>making</em> an LLM answer something — it’s knowing whether the answer was any good. If you’re building anything serious with LLMs, evaluation becomes the unsung backbone of the entire system.</p><p>This article breaks down the different kinds of metrics used today to measure LLM performance — illustrated with clean, Figma-style diagrams that you can recreate visually.</p><h3>1. String-Level Metrics: The Old School Basics</h3><p>These are the simplest metrics — exact match, edit distance, token overlap. They work for short direct answers like “4” or “Paris,” but fall apart when the model’s phrasing changes.</p><pre>┌──────────────────────────────┐<br>│ Expected: &quot;4&quot;                │<br>│ Model:    &quot;4&quot;                │<br>│ Score: 100%                  │<br>└──────────────────────────────┘</pre><p>Useful, but too shallow for modern LLM tasks.</p><h3>2. Set-Based Metrics (Jaccard Similarity)</h3><p><em>Perfect for entity extraction and lists.</em></p><p>Most extraction tasks care about <strong>what</strong> was extracted, not the <strong>order</strong>.</p><pre>Expected: {milk, sugar, cocoa}<br>Model:    {sugar, cocoa, milk, salt}</pre><pre>┌────────────────────────────────────────────────────┐<br>│      Expected Set        ∩        Model Set        │<br>│  {milk, sugar, cocoa}        {sugar, cocoa, salt,  │<br>│                                      milk}         │<br>│                                                    │<br>│  Jaccard = |Intersection| / |Union| = 3/4 = 0.75   │<br>└────────────────────────────────────────────────────┘</pre><p>Jaccard rewards partial correctness and cleanly exposes hallucinations.</p><h3>3. Weighted Precision &amp; Recall</h3><p><em>Because some mistakes matter more.</em></p><p>Not all items are equal. Missing <em>gluten</em> is worse than missing <em>salt</em>; hallucinating <em>peanut</em> can be dangerous.</p><pre>┌───────────────────────────────────────┐<br>│ Item       Weight     Comment         │<br>│ salt         1        low importance  │<br>│ sugar        1        neutral         │<br>│ gluten       5        critical        │<br>└───────────────────────────────────────┘</pre><p>Weighted metrics feel closer to real-world risk.</p><h3>4. Ordinal Scoring</h3><p><em>Used when the answer is on a scale (e.g., 1–5, 1–10).</em></p><p>Being slightly wrong should not be punished like being wildly wrong.</p><pre>Scale: 1──2──3──4──5</pre><pre>True: 5<br>Model: 4</pre><pre>Distance = 1</pre><pre>┌───────────────────────────────────────────────┐<br>│   1     2     3     4     5                   │<br>│                         ↑ Model (4)           │<br>│                               ↑ True (5)      │<br>│   Ordinal Penalty = 1 step                    │<br>└───────────────────────────────────────────────┘</pre><p>Perfect for preference prediction, ratings, or rubrics.</p><h3>5. Structured Output Scoring</h3><p><em>When the model returns JSON, lists, or messy mixed formats.</em></p><p>Models often respond like:</p><pre>{<br> &quot;ingredients&quot;: &quot;Sugar, eggs , milk &quot;<br>}</pre><p>or</p><pre>Ingredients: sugar | milk | cocoa</pre><p>A robust metric pipeline:</p><ul><li>parses even broken JSON</li><li>extracts lists from sentences</li><li>coerces types</li><li>compares fields loosely</li><li>gives partial credit</li></ul><pre>┌───────────────────────┐<br>│ Raw Output            │<br>│ &quot;milk ,  sugar.&quot;      │<br>└───────────┬───────────┘<br>            ↓<br>┌───────────────────────┐<br>│ Normalized            │<br>│ [&quot;milk&quot;,&quot;sugar&quot;]      │<br>└───────────────────────┘</pre><p>Normalization determines whether evaluation is fair or meaningless.</p><h3>6. Composite Metrics</h3><p><em>Most real-world tasks require multiple dimensions of scoring.</em></p><p>You often score on:</p><ul><li>structural correctness</li><li>factual accuracy</li><li>completeness</li><li>hallucinations</li><li>formatting</li></ul><p>Then combine everything into one weighted score.</p><pre>┌───────────────────────────────────┐<br>│ Composite Score Breakdown         │<br>│                                   │<br>│ Structure         20%             │<br>│ Accuracy          40%             │<br>│ Recall            20%             │<br>│ Hallucination     20%             │<br>│                                   │<br>│ Final Score = 0.87                │<br>└───────────────────────────────────┘</pre><p>This is the closest thing to a “report card” for an LLM.</p><h3>7. LLM-as-a-Judge</h3><p><em>The most important modern evaluation technique.</em></p><p>Instead of comparing strings, a separate LLM acts as a judge and evaluates:</p><ul><li>reasoning</li><li>coherence</li><li>factuality</li><li>format adherence</li><li>groundedness</li><li>hallucinations</li></ul><pre>┌───────────────┐      ┌───────────────────┐<br>│ Primary Model │ ───► │   Judge Model     │ ───► Score + Feedback<br>└───────────────┘      └───────────────────┘</pre><p>This enables evaluation of things no simple metric can measure.</p><h3>8. Evidence-Based Feedback (GEPA-style)</h3><p><em>The judge explains its score using evidence from the model’s output.</em></p><pre>┌──────────────────────────────────────────────┐<br>│ Judge Feedback                               │<br>│ - Found correct items: sugar, cocoa          │<br>│ - Missing: milk                              │<br>│ - Hallucinated: almonds                      │<br>│ - Final Score: 7/10                          │<br>└──────────────────────────────────────────────┘</pre><p>This kind of feedback is gold for debugging and improving prompts.</p><h3>Why Good Metrics Matter</h3><p>LLMs don’t fail loudly.<br>They fail quietly, subtly, and often confidently.</p><p>Bad evaluation means:</p><ul><li>hallucinations slip into production</li><li>structured tasks break silently</li><li>improvements become impossible to measure</li><li>safety can’t be guaranteed</li></ul><p>Good metrics do more than judge — they reveal, guide, and protect.</p><p>They turn LLM development from guesswork into engineering.</p><p><strong>Eightinity Engineering</strong><br>We share real-world AI and mobile engineering insights. Before you go:</p><p>👏 Show your support by <strong>clapping</strong> and <strong>following</strong> the <strong>author</strong><br>🧠 Discover more AI, OpenAI, and SAM model articles<br>📱 Explore iOS, Android, and UI engineering blogs<br>🚀 Learn how we build AI-powered products at Eightinity<br>🔔 Follow us: <a href="https://www.linkedin.com/company/eightinity-technologies"><strong>LinkedIn</strong></a> | <a href="https://x.com/8inityStudio"><strong>X (Twitter)</strong></a> | <a href="https://www.eightinity.in/"><strong>Website</strong></a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=2b32aa4297fd" width="1" height="1" alt=""><hr><p><a href="https://medium.com/eightinity/performance-metrics-for-evaluating-llms-2b32aa4297fd">Performance Metrics for evaluating LLMs</a> was originally published in <a href="https://medium.com/eightinity">Eightinity</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[DSPy in Practice: Orchestrating Multi-Agent, Multi-Turn Conversations]]></title>
            <link>https://medium.com/eightinity/dspy-in-practice-orchestrating-multi-agent-multi-turn-conversations-898412ccfc88?source=rss-ecc09fa8060a------2</link>
            <guid isPermaLink="false">https://medium.com/p/898412ccfc88</guid>
            <category><![CDATA[dspy]]></category>
            <category><![CDATA[prompt-engineering]]></category>
            <category><![CDATA[multi-agent-systems]]></category>
            <category><![CDATA[aiml]]></category>
            <category><![CDATA[react]]></category>
            <dc:creator><![CDATA[Sharath Pai]]></dc:creator>
            <pubDate>Wed, 07 Jan 2026 08:26:00 GMT</pubDate>
            <atom:updated>2026-01-07T08:26:00.590Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/700/0*rwbXZEvUHJBxHwHc" /></figure><p>As conversational AI systems mature, the challenge is no longer generating a single good response — it’s sustaining coherent, goal-driven conversations over multiple turns. Many existing multi-agent designs still treat agents as passive tools, with a central controller deciding every handoff. This creates a bottleneck where reasoning becomes centralized, limiting flexibility and emergent collaboration between agents.</p><p>In this article, we propose a different approach: agent-driven orchestration, where agents actively interact with each other across turns while sharing conversational state. Using DSPy, we formalize this interaction as a programmable workflow rather than a collection of fragile prompts. The result is a system that is easier to reason about, easier to extend with new agents, and better suited for complex, real-world conversational tasks where decisions evolve dynamically.</p><h3>Approach</h3><p>We’ll create agent definitions using dspy.Signature which serve as declarative contracts specifying the expected inputs and outputs of an agent. The signatures also serve as system prompts where task-specific instructions can be provided.</p><p>Each agent signature includes the following fields: user_input, history, response_to_user, and handoff_agent. The user_input captures the current user query, while history maintains conversational context, enabling stateful interactions. The response_to_user field represents the agent’s generated output, and handoff_agent allows the agent to explicitly indicate whether control should be transferred to another agent.</p><h3>Architecture</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*-V3o88rAVnO0gtT7F3mGrg.png" /></figure><p>Each signature will be wrapped in a dspy.Module which act as building blocks abstracting a <strong>prompting technique. </strong>We’ll be using Predict technique for simple predictions or theReAct technique in case the agent is using some tool.</p><p>Since we are designing this architecture for multi-turn conversations which is a chatbot workflow, we must store conversations in form of some memory, here dspy.History comes into the picture where there will be a shared history across the entire conversation.</p><p>Each conversation will be a session based conversation where the each user_input goes through the TopLevelModule. The outcome of the first turn of the conversation is always decided by TopLevelModule which hands-off to a specific agent based on the user input. The handed-off agent is set as the active agent for the subsequent turn. If the query of the next turn is suitable to be handled by the active agent, then the active agent itself gives the response or else it hands-off to some other agent which can handle the task and it gives the final response. The other agent is set as an active agent for the subsequent turn in case the the query is handed off from another agent. The process remains the same for the entire conversation</p><h3>Tool Definitions</h3><p><strong>Web Search Tool: </strong>Searching the web when required. We’re going to use Exa API for web search</p><pre>def web_search_tool(query: str) -&gt; dict:<br>    &quot;&quot;&quot;Search the web using Exa API.&quot;&quot;&quot;<br>    api_key = os.getenv(&quot;EXA_API_KEY&quot;)<br>    if not api_key:<br>        raise ValueError(&quot;EXA_API_KEY environment variable is not set&quot;)<br><br>    exa = Exa(api_key=api_key)<br>    result = exa.search_and_contents(<br>        query,<br>        livecrawl=&quot;never&quot;,<br>        num_results=3,<br>        text={&quot;max_characters&quot;: 1000},<br>        type=&quot;auto&quot;,<br>    )<br>    return {&quot;results&quot;: result}</pre><p><strong>Load Product Details Tool:</strong> Getting information about a product such as ingredients, nutritional information by including the product or brand name in the query. We’re going to use OpenFoodFacts API for search for information related to food products.</p><pre>def load_product_details(query: str) -&gt; Dict[str, Any]:<br>    &quot;&quot;&quot;Call the OpenFoodFacts API to fetch product metadata for a food query.&quot;&quot;&quot;<br><br>    search_terms = (query or &quot;&quot;).strip()<br>    if not search_terms:<br>        raise ValueError(&quot;query must include a non-empty search term.&quot;)<br><br>    params = {<br>        &quot;search_terms&quot;: search_terms,<br>        &quot;fields&quot;: &quot;code,product_name,brands,ingredients_text&quot;,<br>        &quot;search_simple&quot;: 1,<br>        &quot;action&quot;: &quot;process&quot;,<br>        &quot;json&quot;: 1,<br>        &quot;page_size&quot;: 10,<br>    }<br>    encoded_params = parse.urlencode(params, safe=&quot;,&quot;)<br>    url = f&quot;https://world.openfoodfacts.org/cgi/search.pl?{encoded_params}&quot;<br><br>    try:<br>        with request.urlopen(url, timeout=15) as response:<br>            if response.status != 200:<br>                raise RuntimeError(f&quot;OpenFoodFacts returned status {response.status}&quot;)<br>            payload = json.loads(response.read().decode(&quot;utf-8&quot;))<br>    except Exception as exc:<br>        raise RuntimeError(f&quot;Failed to load product details: {exc}&quot;) from exc<br><br>    products = payload.get(&quot;products&quot;) or []<br>    summarized = [<br>        {<br>            &quot;code&quot;: item.get(&quot;code&quot;),<br>            &quot;name&quot;: item.get(&quot;product_name&quot;),<br>            &quot;brands&quot;: item.get(&quot;brands&quot;),<br>            &quot;ingredients&quot;: item.get(&quot;ingredients_text&quot;),<br>        }<br>        for item in products<br>    ]<br><br>    return {<br>        &quot;query&quot;: query,<br>        &quot;count&quot;: payload.get(&quot;count&quot;, len(summarized)),<br>        &quot;page&quot;: payload.get(&quot;page&quot;),<br>        &quot;page_size&quot;: payload.get(&quot;page_size&quot;),<br>        &quot;products&quot;: summarized,<br>    }</pre><h3>Agent Definitions</h3><p><strong>Food Agent:</strong> Handles all food related to queries. If asked about any particular food product, it calls the load_product_details tool which returns metadata of the product based on the query.</p><pre>from __future__ import annotations<br><br>import dspy<br>from tools import load_product_details<br><br>class FoodAgent(dspy.Signature):<br>    &quot;&quot;&quot;Signature for the food agent: recipes, meal ideas, nutrition, and cooking help.&quot;&quot;&quot;<br><br>    user_input: str = dspy.InputField(<br>        desc=&quot;User question or request about food, cooking, recipes, or nutrition.&quot;<br>    )<br>    history: dspy.History = dspy.InputField(<br>        desc=&quot;Conversation history carried across turns. Provide the running dspy.History instance.&quot;<br>    )<br><br>    response_to_user: str = dspy.OutputField(<br>        desc=&quot;Concrete food-related advice or answer; keep it concise and actionable.&quot;<br>    )<br>    handoff_agent: str = dspy.OutputField(<br>        desc=(<br>            &quot;If another specialist should take over, return one of {&#39;catchall&#39;,&#39;literature&#39;,&#39;news&#39;}; &quot;<br>            &quot;otherwise return &#39;&#39;. Choose &#39;catchall&#39; for general greetings/feedback, &#39;literature&#39; for research framing, &quot;<br>            &quot;and &#39;news&#39; for current events context.&quot;<br>        )<br>    )<br><br><br>class FoodModule(dspy.Module):<br>    &quot;&quot;&quot;Help with food topics: recipes, meal ideas, nutrition, and cooking advice.&quot;&quot;&quot;<br><br>    def __init__(self):<br>        super().__init__()<br>        self.tools = [<br>            dspy.Tool(load_product_details),<br>        ]<br>        self.strategy = dspy.ReAct(FoodAgent, tools=self.tools, max_iters=1)<br><br>    def forward(self, user_input: str, history: dspy.History):<br>        prediction = self.strategy(user_input=user_input, history=history)<br>        self.output = prediction<br>        return prediction</pre><p><strong>News Agent: </strong>Searching for ground truth across the internet using web_search_tool.</p><pre>from __future__ import annotations<br>from typing import List<br>import dspy<br>from tools import web_search_tool<br><br>class NewsAgent(dspy.Signature):<br>    &quot;&quot;&quot;Signature for the news agent: summarize current events with citations.&quot;&quot;&quot;<br><br>    user_input: str = dspy.InputField(<br>        desc=&quot;User request about current events, market updates, or news stories.&quot;<br>    )<br>    history: dspy.History = dspy.InputField(<br>        desc=&quot;Conversation history carried across turns. Provide the running dspy.History instance.&quot;<br>    )<br><br>    response_to_user: str = dspy.OutputField(<br>        desc=&quot;Concise news summary with key facts and cited links.&quot;<br>    )<br>    handoff_agent: str = dspy.OutputField(<br>        desc=(<br>            &quot;If another specialist should take over, return one of {&#39;catchall&#39;,&#39;food&#39;,&#39;literature&#39;}; &quot;<br>            &quot;otherwise return &#39;&#39;. Choose &#39;catchall&#39; for greetings/feedback, &#39;food&#39; for nutrition topics, &quot;<br>            &quot;and &#39;literature&#39; for deeper research contexts.&quot;<br>        )<br>    )<br><br><br>class NewsModule(dspy.Module):<br>    &quot;&quot;&quot;Aggregate up-to-date news using ReAct with the web search tool.&quot;&quot;&quot;<br><br>    def __init__(self):<br>        super().__init__()<br>        self.tools = [<br>            dspy.Tool(web_search_tool),<br>        ]<br>        self.strategy = dspy.ReAct(NewsAgent, tools=self.tools, max_iters=1)<br><br>    def forward(self, user_input: str, history: dspy.History):<br>        prediction = self.strategy(user_input=user_input, history=history)<br>        self.output = prediction<br>        return prediction</pre><p><strong>Literature Agent:</strong> Answers queries related to academic articles, books, and scholarly content.</p><pre>from __future__ import annotations<br><br>import dspy<br><br><br>class LiteratureAgent(dspy.Signature):<br>    &quot;&quot;&quot;Signature for the literature agent: prior work, citations, and framing.&quot;&quot;&quot;<br><br>    user_input: str = dspy.InputField(<br>        desc=&quot;User request related to literature, references, or prior work framing.&quot;<br>    )<br>    history: dspy.History = dspy.InputField(<br>        desc=&quot;Conversation history carried across turns. Provide the running dspy.History instance.&quot;<br>    )<br><br>    response_to_user: str = dspy.OutputField(<br>        desc=&quot;Summary, explanation, or citation-style output related to literature.&quot;<br>    )<br>    handoff_agent: str = dspy.OutputField(<br>        desc=(<br>            &quot;If another specialist should take over, return one of {&#39;catchall&#39;,&#39;food&#39;,&#39;news&#39;}; otherwise return &#39;&#39;. &quot;<br>            &quot;Choose &#39;catchall&#39; for conversational replies, &#39;food&#39; for nutrition topics, and &#39;news&#39; for current events.&quot;<br>        )<br>    )<br><br><br>class LiteratureModule(dspy.Module):<br>    &quot;&quot;&quot;Help with related work, citations, and prior art framing.&quot;&quot;&quot;<br><br>    def __init__(self):<br>        super().__init__()<br>        self.strategy = dspy.Predict(LiteratureAgent)<br><br>    def forward(self, user_input: str, history: dspy.History):<br>        prediction = self.strategy(user_input=user_input, history=history)<br>        self.output = prediction<br>        return prediction</pre><p><strong>Catch-All Agent:</strong> Handles generic interactions such as greetings, user feedback, and other non-domain-specific queries. This agent also serves as a fallback to ensure graceful handling of inputs that do not clearly map to the specialized agents.</p><pre>from __future__ import annotations<br><br>import dspy<br><br><br>class CatchAllAgent(dspy.Signature):<br>    &quot;&quot;&quot;Signature for the catch-all agent: greetings, feedback, and general chit-chat.&quot;&quot;&quot;<br><br>    user_input: str = dspy.InputField(<br>        desc=&quot;User message that is conversational, contains feedback, or doesn&#39;t match another specialist.&quot;<br>    )<br>    history: dspy.History = dspy.InputField(<br>        desc=&quot;Running conversation history (dspy.History). Use it to keep tone consistent.&quot;<br>    )<br><br>    response_to_user: str = dspy.OutputField(<br>        desc=&quot;Friendly acknowledgement or clarification. Thank users for feedback, greet politely, and keep it brief.&quot;<br>    )<br>    handoff_agent: str = dspy.OutputField(<br>        desc=(<br>            &quot;If the message really needs another specialist, return one of {&#39;food&#39;,&#39;literature&#39;,&#39;news&#39;}; &quot;<br>            &quot;otherwise return &#39;&#39;. Choose &#39;news&#39; for current events, &#39;literature&#39; for research framing, and &quot;<br>            &quot;&#39;food&#39; for culinary requests.&quot;<br>        )<br>    )<br><br><br>class CatchAllModule(dspy.Module):<br>    &quot;&quot;&quot;Default fallback for greetings, thanks, or off-topic chatter.&quot;&quot;&quot;<br><br>    def __init__(self):<br>        super().__init__()<br>        self.strategy = dspy.Predict(CatchAllAgent)<br><br>    def forward(self, user_input: str, history: dspy.History):<br>        prediction = self.strategy(user_input=user_input, history=history)<br>        self.output = prediction<br>        return prediction</pre><p>In any multi-agent based applications, this agent is a must have so that the agentic system remains centralized and it can handle all types of queries. You can prompt it to restrict answering to some queries and it can return a warning message as a response.</p><p>All the modules can be composed into a single TopLevelModule which will act as a compiled program</p><pre>class TopLevelModule(dspy.Module):<br>    &quot;&quot;&quot;Top-level router that coordinates the research agents with handoffs.&quot;&quot;&quot;<br><br>    def __init__(self):<br>        super().__init__()<br>        # Use module classes (wrapping ReAct over signatures) as the routed agents.<br>        self.modules: Dict[str, dspy.Module] = {<br>            &quot;catchall&quot;: CatchAllModule(),<br>            &quot;food&quot;: FoodModule(),<br>            &quot;literature&quot;: LiteratureModule(),<br>            &quot;news&quot;: NewsModule(),<br>        }<br>        self.active_module_name: Optional[str] = None<br><br>    def forward(<br>        self,<br>        user_input: str,<br>        history: Optional[dspy.History] = None,<br>        active_module_name: Optional[str] = None,<br>    ):<br>        if active_module_name:<br>            self.active_module_name = active_module_name<br><br>        if self.active_module_name is None:<br>            # Default to the first registered module; callers can override via active_module_name.<br>            self.active_module_name = next(iter(self.modules))<br><br>        previous_module = self.active_module_name<br><br>        result, updated_module = route_with_handoff(<br>            user_input=user_input,<br>            history=history,<br>            modules=self.modules,<br>            active_module_name=self.active_module_name,<br>        )<br>        self.active_module_name = updated_module<br>        active_module = updated_module or previous_module<br><br>        handoff_agent = &quot;&quot;<br>        if previous_module and updated_module and updated_module != previous_module:<br>            handoff_agent = updated_module<br><br>        enriched_result = dict(result)<br>        enriched_result[&quot;active_module&quot;] = active_module<br>        enriched_result[&quot;handoff_agent&quot;] = handoff_agent<br><br>        self.output = enriched_result<br>        return enriched_result</pre><p>The TopLevelModule won’t have any signature, its job is to handoff to an agent for just the first query.</p><h3>Storing History across the agents</h3><p>We are going to store the history of the conversations in dspy.History. It will contain all the signature fields</p><pre>history = dspy.History(messages=[])<br><br>history.messages.append(<br>            {<br>                &quot;user_input&quot;: query,<br>                &quot;response_to_user&quot;: assistant_reply,<br>                &quot;handoff_agent&quot;: handoff_agent,<br>            }<br>        )</pre><p>This way, each query gets a context of the past conversations and we don’t need to implement RAG.</p><h3>Environment Variables</h3><p>Set the following API Keys in your environment variables</p><pre>EXA_API_KEY = &quot;...&quot;<br>LANGFUSE_SECRET_KEY=&quot;...&quot;<br>LANGFUSE_PUBLIC_KEY=&quot;...&quot;<br>LANGFUSE_HOST = &quot;http://localhost:3000&quot;<br><br>USER_ID = &#39;your-name&#39;<br><br>OPENAI_API_KEY = &#39;...&#39;</pre><p>You can use any other API other than OpenAI as per your liking. And Langfuse keys are for tracing the LM calls on Langfuse.</p><h3>Tracking Multi-Agent Conversations in Langfuse</h3><p>We will be using a self-hosted instance of Langfuse. Follow these steps to set it up:</p><pre># Clone the latest Langfuse repository<br>git clone https://github.com/langfuse/langfuse.git<br>cd langfuse<br>docker compose up</pre><p><strong>Note: </strong>Running docker compose up requires Docker Desktop to be installed and running on your system.</p><h4><strong>Example: A 5-Turn Conversation</strong></h4><h4>Turn 1</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*FWg-goQtmQnQKlz-5wTgAw.png" /><figcaption>Turn 1</figcaption></figure><p><strong><em>Query: “What does Cheeselings contain?”</em></strong><br>This initial food-related query is routed by the orchestrator (TopLevelModule) to the FoodAgent. Since the question asks for product details, the `load_product_details` tool is triggered. The FoodAgent is now set as the active agent.</p><h4>Turn 2</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*A4SYxuI0SDcJHhTv7RtFAg.png" /><figcaption>Turn 2</figcaption></figure><p><strong><em>Query: “Is it generally healthy?”</em></strong><br>Using conversation history, i.e. dspy.History, the context is retained. The query remains within the food domain, so the FoodAgent responds. The TopLevelModule returns continuation=True, keeping the FoodAgent active.</p><h4><strong>Turn 3</strong></h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*UrY2GTGTSUfdySaCwOxKoQ.png" /><figcaption>Turn 3</figcaption></figure><p><strong><em>Query: “What was Jean-Baptiste Clamence’s profession in the novel The Fall by Albert Camus?”</em></strong><br>Although the FoodAgent is currently active, this literature question is outside its scope. The FoodAgent acknowledges it cannot handle the query and initiates a handoff to the LiteratureAgent, which provides the final response and becomes the new active agent.</p><h4>Turn 4</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*oddgjNQ8n_WHaptJMbn05A.png" /><figcaption>Turn 4</figcaption></figure><p><strong><em>Query: “Who won the recent ODI series between India and South Africa?”</em></strong></p><p>This query requires current information, so the LiteratureAgent hands off to the NewsAgent. The NewsAgent uses web_search_tool to fetch the latest details and is set as the active agent.</p><h4>Turn 5</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*xwxA6HDI4e_rS347iava1Q.png" /><figcaption>Turn 5</figcaption></figure><p><strong><em>Query: “Thank you, I love how you have diverse knowledge as an AI agent.”</em></strong></p><p>This is a general acknowledgment/greeting. The NewsAgent hands off to the CatchAllAgent, which handles the final interaction.</p><h3>Best Practices for DSPy in Multi-Agent Systems</h3><p>When building multi-agent architectures with DSPy, avoid hardcoding too much programmatic logic—such as manual keyword-based routing—as it defeats the purpose of using a framework like DSPy. Instead:</p><ul><li>Keep dspy.Signature fields descriptive, not instructional. If the model struggles, you can include instructions within the signature or—better yet—create a dataset of observed interaction patterns.</li><li>Use <strong>optimizers</strong> like BootstrapFewShot or GEPAto train your program on this dataset. This allows the system to learn routing and behavior without constant manual signature adjustments.</li><li>Iterate with new patterns. When encountering a new query pattern, add corresponding examples to your dataset and re-optimize. This maintains a consistent system prompt and improves handling over time.</li></ul><h3>Current Limitation</h3><p>This architecture is designed to invoke <strong>only one agent per query</strong>. A compound question like <em>“Who won the recent ODI series between India and South Africa and what does Cheeselings contain?”</em> would trigger the CatchAllAgent instead of correctly routing to both the NewsAgent and FoodAgent.</p><h3>Link to the Code</h3><p>The entire code can be found on my <a href="https://github.com/Sharath8inity/dspy-multi-agent-architecture">GitHub</a></p><p>This concludes our overview. A <strong>Part II</strong> of this blog is coming soon, where I will demonstrate optimizing this program with a DSPy optimizer and showcase the final signature structure. I hope you found this useful.</p><p><strong>Eightinity Engineering</strong><br>We share real-world AI and mobile engineering insights. Before you go:</p><p>👏 Show your support by <strong>clapping</strong> and <strong>following</strong> the <strong>author</strong><br>🧠 Discover more AI, OpenAI, and SAM model articles<br>📱 Explore iOS, Android, and UI engineering blogs<br>🚀 Learn how we build AI-powered products at Eightinity<br>🔔 Follow us: <a href="https://www.linkedin.com/company/eightinity-technologies"><strong>LinkedIn</strong></a> | <a href="https://x.com/8inityStudio"><strong>X (Twitter)</strong></a> | <a href="https://www.eightinity.in/"><strong>Website</strong></a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=898412ccfc88" width="1" height="1" alt=""><hr><p><a href="https://medium.com/eightinity/dspy-in-practice-orchestrating-multi-agent-multi-turn-conversations-898412ccfc88">DSPy in Practice: Orchestrating Multi-Agent, Multi-Turn Conversations</a> was originally published in <a href="https://medium.com/eightinity">Eightinity</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Running HuggingFace language models with Langchain]]></title>
            <link>https://medium.com/@sharathpai107/running-huggingface-language-models-with-langchain-687b633ba31d?source=rss-ecc09fa8060a------2</link>
            <guid isPermaLink="false">https://medium.com/p/687b633ba31d</guid>
            <category><![CDATA[openai-open-source-model]]></category>
            <category><![CDATA[langchain]]></category>
            <category><![CDATA[hugging-face]]></category>
            <category><![CDATA[large-language-models]]></category>
            <dc:creator><![CDATA[Sharath Pai]]></dc:creator>
            <pubDate>Fri, 08 Aug 2025 14:33:01 GMT</pubDate>
            <atom:updated>2025-08-08T14:33:01.424Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*Wx3LqWitx6pOkSn9.png" /></figure><p>There are some big names in the open-source LLM market, such as Deepseek, Qwen, and GPT OSS. We all have had the desire to run the biggest of their models but we don’t have the setup.</p><p>I’ve got you covered. You don’t need any GPU to run these models. The partnership of HuggingFace with Langchain has solved our woes. The langchain-huggingface package provides some functions with which you can run upto 1000 language models supported by the HuggingFace API.</p><p>Let’s dive deep into the Python code for a better understanding.</p><h3>Setup</h3><p>Create a HuggingFace Token which you can find in <a href="https://huggingface.co/settings/tokens">https://huggingface.co/settings/tokens</a>. Create a fine grained token and allot all the permissions.</p><p>Create an .env file and add the token as</p><pre>HUGGINGFACE_API_KEY = &quot;hf_......&quot;</pre><p>Create a virtual environment to avoid package conflicts</p><pre>python -m venv myenv<br>myenv\Scripts\activate</pre><p>Install the following dependencies</p><pre>pip install langchain langchain-huggingface python-dotenv transformers torch</pre><h3>Code</h3><p>Create a file main.py. We’ll go part by part.</p><p>Import all the packages with their functions and define environment variables.</p><pre>import os<br>from langchain_huggingface import ChatHuggingFace, HuggingFaceEndpoint<br>from dotenv import load_dotenv<br>from langchain_core.messages import HumanMessage, SystemMessage<br><br>load_dotenv(override=True)<br>os.environ[&quot;HUGGINGFACE_API_TOKEN&quot;] = os.getenv(&quot;HUGGINGFACE_API_TOKEN&quot;)</pre><p>Create a class LangchainHuggingFace and define all the functions in the class</p><pre>class LangchainHuggingFace:</pre><p>Initialize your model and a couple of function names. Here we’ll be using openai/gpt-oss-120b model which was recently introduced by OpenAI.</p><pre>    def __init__(self):<br>        self.model_id = &quot;openai/gpt-oss-120b&quot;<br>        self.llm = self.generate_text()<br>        self.chat_model = self.chat_model()</pre><p>Using the HuggingFace endpoint to initialize model attributes and task. Here we’re performing text generation.</p><pre>    def generate_text(self):<br>        return HuggingFaceEndpoint(<br>            repo_id=self.model_id,<br>            task=&quot;text-generation&quot;,<br>            max_new_tokens=512,<br>            do_sample=False,<br>            repetition_penalty=1.03,<br>            provider=&quot;auto&quot;,  # let Hugging Face choose the best provider for you<br>        )</pre><p>Embed the generate_textfunction in the ChatHuggingface function</p><pre>    def chat_model(self):<br>        return ChatHuggingFace(llm=self.llm)</pre><p>Initialize the chat template with query as an argument to be taken in the main function and invoke the messages.</p><pre>    def get_response(self, query):<br>        messages = [<br>            SystemMessage(content=&quot;You&#39;re a helpful assistant&quot;),<br>            HumanMessage(<br>                content=query<br>            ),<br>        ]<br><br>        response = self.chat_model.invoke(messages)<br>        return response.content</pre><p>Outside the LangchainHuggingFace class, define the main block where you can run test cases by calling the class. Just change the query as per your liking.</p><pre>if __name__ == &quot;__main__&quot;:<br>    langchain_hf = LangchainHuggingFace()<br>    query = &quot;What is the capital of India?&quot;<br>    response = langchain_hf.get_response(query)<br>    print(response)</pre><p>Final code</p><pre>import os<br>from langchain_huggingface import ChatHuggingFace, HuggingFaceEndpoint<br>from dotenv import load_dotenv<br>from langchain_core.messages import HumanMessage, SystemMessage<br><br>load_dotenv(override=True)<br>os.environ[&quot;HUGGINGFACE_API_TOKEN&quot;] = os.getenv(&quot;HUGGINGFACE_API_TOKEN&quot;)<br><br>class LangchainHuggingFace:<br>    def __init__(self):<br>        self.model_id = &quot;openai/gpt-oss-120b&quot;<br>        self.llm = self.generate_text()<br>        self.chat_model = self.chat_model()<br><br>    def generate_text(self):<br>        return HuggingFaceEndpoint(<br>            repo_id=self.model_id,<br>            task=&quot;text-generation&quot;,<br>            max_new_tokens=512,<br>            do_sample=False,<br>            repetition_penalty=1.03,<br>            provider=&quot;auto&quot;,  # let Hugging Face choose the best provider for you<br>        )<br><br>    def chat_model(self):<br>        return ChatHuggingFace(llm=self.llm)<br>    <br>    def get_response(self, query):<br>        messages = [<br>            SystemMessage(content=&quot;You&#39;re a helpful assistant&quot;),<br>            HumanMessage(<br>                content=query<br>            ),<br>        ]<br><br>        response = self.chat_model.invoke(messages)<br>        return response.content<br>    <br>if __name__ == &quot;__main__&quot;:<br>    langchain_hf = LangchainHuggingFace()<br>    query = &quot;What is the capital of India?&quot;<br>    response = langchain_hf.get_response(query)<br>    print(response)</pre><h3>Run the code</h3><p>Inside the virtual environment, run</p><pre>python main.py</pre><h3>Checking the list of available models</h3><p>Go to <a href="https://huggingface.co/api/models">https://huggingface.co/api/models</a>. There are 1000 models that you can run via the Inference API.</p><p>Search for the model which you want by pressing Ctrl+F and searching. “id”: “openai/</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Tew2q_jfnTIDa5vm8lq60Q.png" /></figure><p>As you can see, there are 13 models provided by OpenAI comprising different tasks. Here’s the model card of the openai/gpt-oss-120b model that we used.</p><pre>{<br>    &quot;_id&quot;: &quot;68913522f16f3c8aaffccf1f&quot;,<br>    &quot;id&quot;: &quot;openai/gpt-oss-120b&quot;,<br>    &quot;likes&quot;: 2939,<br>    &quot;trendingScore&quot;: 2939,<br>    &quot;private&quot;: false,<br>    &quot;downloads&quot;: 237866,<br>    &quot;tags&quot;: [<br>      &quot;transformers&quot;,<br>      &quot;safetensors&quot;,<br>      &quot;gpt_oss&quot;,<br>      &quot;text-generation&quot;,<br>      &quot;vllm&quot;,<br>      &quot;conversational&quot;,<br>      &quot;license:apache-2.0&quot;,<br>      &quot;autotrain_compatible&quot;,<br>      &quot;endpoints_compatible&quot;,<br>      &quot;8-bit&quot;,<br>      &quot;mxfp4&quot;,<br>      &quot;region:us&quot;<br>    ],<br>    &quot;pipeline_tag&quot;: &quot;text-generation&quot;,<br>    &quot;library_name&quot;: &quot;transformers&quot;,<br>    &quot;createdAt&quot;: &quot;2025-08-04T22:33:06.000Z&quot;,<br>    &quot;modelId&quot;: &quot;openai/gpt-oss-120b&quot;<br>  },</pre><p>The pipeline-tag attribute specifies the task that we mentioned in the generate_text function.</p><p>In this way, you can search for other models and model providers such as Qwen, Deepseek, etc., as per your required task. You can run a model of any parameters. It just should support Inference API.</p><h3>Reference</h3><p>Langchain documentation: <a href="https://python.langchain.com/docs/integrations/chat/huggingface/">https://python.langchain.com/docs/integrations/chat/huggingface/</a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=687b633ba31d" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Ways to run Large Language Models]]></title>
            <link>https://medium.com/@sharathpai107/ways-to-run-large-language-models-2dca2cab4e13?source=rss-ecc09fa8060a------2</link>
            <guid isPermaLink="false">https://medium.com/p/2dca2cab4e13</guid>
            <category><![CDATA[large-language-models]]></category>
            <category><![CDATA[genai]]></category>
            <category><![CDATA[hugging-face]]></category>
            <dc:creator><![CDATA[Sharath Pai]]></dc:creator>
            <pubDate>Mon, 30 Jun 2025 08:05:41 GMT</pubDate>
            <atom:updated>2025-06-30T08:05:41.885Z</atom:updated>
            <content:encoded><![CDATA[<p>This blog explores the differences between closed-source and open-source LLMs, how to run them on local systems or cloud platforms, and techniques like quantization to optimize performance. Whether you’re handling sensitive data or managing costs, this guide provides insights to help you navigate the world of LLMs effectively.</p><h3>Closed Source LLMs</h3><p>Closed source LLMs are those LLMs whose data remains with the AI providers which includes giants such as OpenAI, Google, Anthropic, Groq, Meta, Together.ai.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/700/0*_MJ7quFCipiZT6nb" /><figcaption>Four AI giant providers of LLMs</figcaption></figure><p><strong>How to run them?</strong> Each provider has its own documentation. You need to create an API key and check for the various supported models by the provider. This way you can run the LLM on your local system or Google Colab with the help of the code template provided by the provider or with the help of Langchain. Below is the code template from the provider Open AI</p><pre>from openai import OpenAI<br>client = OpenAI()</pre><pre>completion = client.chat.completions.create(<br>    model=&quot;gpt-4o&quot;,<br>    messages=[<br>        {&quot;role&quot;: &quot;developer&quot;, &quot;content&quot;: &quot;You are a helpful assistant.&quot;},<br>        {<br>            &quot;role&quot;: &quot;user&quot;,<br>            &quot;content&quot;: &quot;Write a haiku about recursion in programming.&quot;<br>        }<br>    ]<br>)</pre><pre>print(completion.choices[0].message)</pre><p>Note that most of these providers don’t provide their APIs for free. For free APIs, you can use Groq.</p><h3>Open source LLMs</h3><p>Open source LLMs are those LLMs whose source code is available and we can self host it to ensure that the data remains with it. Open source is the go-to way in case of sensitive data such as aadhar cards, etc. Open source LLMs are free of cost and are available on HuggingFace.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/700/0*aTDaM4Q9PfS8IZ0Z.png" /><figcaption>Open Source Models on HuggingFace</figcaption></figure><h3>Ways to run open source models:</h3><h3>a. A hardware setup (GPU)</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/700/0*w77qY-gtJ3CB_u8l.png" /><figcaption>NVIDIA A100 GPU Software Interface</figcaption></figure><p>Most of the open source LLMs are larger in size. So it cannot be run locally. So you need a GPU to host your LLM which you can use it through your organization.</p><p>To host a LLM, you can use vLLM, llama.cpp, ollama and check which LLMs are compatible with them through their documentation.</p><p>They come with commands which you can use inside your GPU to host them and use their endpoints in your local system.</p><p>You don’t need to have knowledge about GPU hosting. These things can be learnt inside organization.</p><h3>b. Google Colab</h3><p>HuggingFace also has APIs for most of the LLMs called inference APIs. Its URL endpoint is like this</p><pre>https://api-inference.huggingface.co/models/&lt;provider&gt;/&lt;model-name&gt;</pre><pre># For the below model, the Inference API is<br>https://api-inference.huggingface.co/models/mistralai/Mistral-7B-Instruct-v0.3</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/391/0*jSql9he1BfFI3Pfv.png" /><figcaption>HuggingFace Inference API for a Mistral model</figcaption></figure><p>You can use the inference API on Google Colab and you can run the LLM through its code template provided in HuggingFace or GitHub. You need to generate a HuggingFace token for running inference APIs.</p><p>Below is an example code snippet of running LLMs via Inference API</p><pre>HUGGINGFACE_API_URL = &quot;https://api-inference.huggingface.co/models/mistralai/Mistral-7B-Instruct-v0.3&quot;<br>HUGGINGFACE_API_KEY = os.getenv(&quot;HUGGINGFACE_TOKEN&quot;)</pre><pre># Function to generate story segment using Hugging Face API<br>def generate_story_segment(prompt):<br>    &quot;&quot;&quot;Generate a story segment based on the prompt using the Hugging Face API.&quot;&quot;&quot;<br>    headers = {<br>        &quot;Authorization&quot;: f&quot;Bearer {HUGGINGFACE_API_KEY}&quot;<br>    }<br>    payload = {<br>        &quot;inputs&quot;: prompt,<br>        &quot;parameters&quot;: {<br>            &quot;max_length&quot;: 500,  # Longer output for detailed storytelling<br>            &quot;temperature&quot;: 0.7,<br>            &quot;do_sample&quot;: True,<br>            &quot;top_p&quot;: 0.9<br>        }<br>    }<br>    response = requests.post(HUGGINGFACE_API_URL, headers=headers, json=payload)<br>    if response.status_code == 200:<br>        generated_text = response.json()[0][&quot;generated_text&quot;]<br>        return generated_text.strip()<br>    else:<br>        print(f&quot;Error: {response.status_code} - {response.text}&quot;)<br>        return &quot;Error generating story segment.&quot;</pre><p>Also don’t forget to change the runtime type to T4 GPU in Colab. It will help in downloading of model files faster.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/523/0*caBg8K6IuE7n3abU.png" /><figcaption>Switching to T4 GPU on Google Colab</figcaption></figure><p>Note that Google Colab comes with 16GB VRAM, so you can run LLMs upto 14B parameters in it.</p><h3>c. Code Editor</h3><p>The process is same as Colab with the only difference being the API limit from HuggingFace. You can run models upto 10GB from HuggingFace</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/700/0*FlA1ABbBynH1nKvd.png" /><figcaption>Code Editor showing rate limits for models &gt; 10GB</figcaption></figure><h3>d. Ollama</h3><p>Ollama is an open source tool made to open source run LLMs locally on its platform. This is useful for testing LLMs for your applications and the data remains with you and its for free! On top of that models are uncensored.</p><p>You need to download Ollama from its official website and you can run the models on your cmd. If you need an interactive UI, you can install OpenWebUI via Python or Docker and you’ll see an interface like ChatGPT on 8080 with a LLM from Ollama. Additionally you can use tools such as web search from OpenWebUI to get latest search results.</p><p>This interface is just for testing purposes. For running LLMs from Ollama in your code, you can use Langchain’s chat completions or OpenAI’s chat completions whose endpoint is localhost:11434/v1 which is activated when you run an Ollama model. You can host Ollama on a server when you’re building production level applications.</p><p>However note that you can run upto 7B parameters models on your local machine if your RAM is 8 GB.</p><h3>Quantization of LLMs</h3><p>As we saw we can only run smaller models on Google Colab or locally. We need to have a GPU for running large models. But can we do it without GPU? The answer is yes.</p><p>There is a process called quantization which is used to reduce the size and computation of LLMs wrt to their parameters. Quantization is a great innovation in the field of deep learning, which allows you to run large models locally.</p><p>We can quantize models with the help of tensorflow or pytorch. But there is an alternative to this as well.</p><p>There is a popular user on HuggingFace named “TheBloke” who has quantized around 4000 models reducing the model size by 70% which are used by many users who don’t have a GPU setup. His models are officially documented by vLLM and we can run it on GPU as well. Models originally around 405B parameters can be used in GPU.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/700/0*YxQlQnm5469lb2cI.png" /><figcaption>TheBloke’s quantized models on HuggingFace</figcaption></figure><p>Quantization may hamper the output of the LLM.</p><h3>Understanding the cost of LLMs based on usage</h3><p>If you’re using LLMs from closed source providers such as OpenAI, Anthropic, Together.ai, etc, you need to track the cost of their APIs since they are paid.</p><p>There is a technique called tokenization in NLP where your input prompt and output results are word tokenized. For example,</p><pre>Input: What is the capital of India? <br>Tokenized Input: [&#39;What&#39;, &#39;is&#39;, &#39;the&#39;, &#39;capital&#39;, &#39;of&#39;, &#39;India&#39;, &#39;?&#39;] <br>Input tokens: 7</pre><pre>Output: The capital of India is New Delhi.<br>Tokenized output: [&#39;The&#39;, &#39;capital&#39;, &#39;of&#39;, &#39;India&#39;, &#39;is&#39;, &#39;New&#39;, &#39;Delhi&#39;, &#39;.&#39;] <br>Output tokens: 8<br>Total tokens: 15</pre><p>If the API costs $0.02 per 1000 tokens, this request would cost</p><pre>(15 tokens/1000) * $0.02 = $0.003</pre><p>We don’t have time to calculate the cost on every request. So we use dashboards such as Langfuse which tracks the cost of LLMs based on their usage. Langfuse needs to be hosted on your cloud platform and you can track your LLMs cost using LiteLLM as a client in your code.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/700/0*HVrYZOOEpmHOzDn_.jpeg" /><figcaption>Using Langfuse to track model costs</figcaption></figure><p>In this way you can set up your LLMs for any project by keeping in mind the type of model you’re gonna use and its specifications.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=2dca2cab4e13" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Performing Agentic RAG with MongoDB and Agno]]></title>
            <link>https://medium.com/@sharathpai107/performing-agentic-rag-with-mongodb-and-agno-010ceef5141b?source=rss-ecc09fa8060a------2</link>
            <guid isPermaLink="false">https://medium.com/p/010ceef5141b</guid>
            <category><![CDATA[agentic-ai]]></category>
            <category><![CDATA[mongodb]]></category>
            <category><![CDATA[vector-database]]></category>
            <category><![CDATA[agno]]></category>
            <category><![CDATA[retrieval-augmented-gen]]></category>
            <dc:creator><![CDATA[Sharath Pai]]></dc:creator>
            <pubDate>Mon, 09 Jun 2025 10:22:06 GMT</pubDate>
            <atom:updated>2025-06-10T07:28:45.282Z</atom:updated>
            <content:encoded><![CDATA[<p>When we have a large collection of data in the form of a PDF or on a website, and if we want the source to answer a specific question, we perform vector search.</p><p>Agents are used to fetch real time data which is not possible only with Large Language Models. So we use a combination of Large Language models with tools, knowledge, memory, vector databases which together form an agent.</p><p>Vector search requires a <strong>knowledge base</strong> which is the domain specific information that enables the agent to make better decisions, also called Agentic RAG. We also use an <strong>embedding model</strong> that embeds all the text into the form of vectors and stores it in a <strong>vector database, </strong>which is efficient in vector search. Then we use a <strong>language model, </strong>which will answer the query using vector search.</p><h3>Setting up MongoDB</h3><p>Assuming you have <a href="https://www.mongodb.com/try/download/community">MongoDB Community Server</a> and <a href="https://www.mongodb.com/try/download/shell">MongoDB Shell</a> installed, we now need a connection string, which we get from CLI with a command called mongosh. After installation, run mongosh in cmd which will enable us to enter inside Mongo CLI.</p><pre>mongosh</pre><p>You’ll get a connection string which looks something like this mongodb://localhost/?directConnection=true</p><p>If you use this string, <strong>you won’t be able to store the embeddings</strong>. This connection string is useful for other projects, but when we perform vector search, we require a connection string from <a href="https://www.mongodb.com/cloud/atlas/register">MongoDB Atlas</a> which comes with a username and a password.</p><p>After entering Atlas, you need to create a new project as shown below</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*phKBOZLj4vu0f_tc1cpf3Q.png" /></figure><p>Click on new project and name your project Agentic RAG and click Next.</p><p>You can add email addresses of people working with you and click on Create Project.</p><p>Your project has been successfully created and you’ve been directed to the home page. Click on Create to create a cluster. Select Free tier and click on Create Deployment.</p><p>You’ll get a pop-up containing username and password. Store them safely somewhere and click on Choose a connection method.</p><p>In Access your data through tools, select Shell.</p><p>Select “I have the MongoDB Shell installed” and copy the connection string, and click Done. Now inside command prompt, in a new terminal paste the connection string.</p><pre>mongosh &quot;mongodb+srv://cluster0.anc.mongodb.net/&quot; --apiVersion 1 --username &lt;db_username&gt;</pre><p><strong>We’ll always be using the above command for activating the Mongo CLI</strong>. Also, don’t forget to replace the &lt;db_username&gt; with the username you got in the popup.</p><p>Press enter and enter the password.</p><p>You’ll get a new connection string which looks like mongodb+srv://&lt;username&gt;:&lt;password&gt;@cluster0.mongodb.net/?retryWrites=true&amp;w=majority. We’ll be using this connection string in our code.</p><p>Since we’re inside Mongo CLI, we’ll be create a database and a collection inside our cluster. Run the following commands inside the activated Mongo CLI.</p><p>Show databases</p><pre>show dbs</pre><p>Switch to a database named agno (which will also create a database called agno)</p><pre>use agno</pre><p>Create a collection called vector-embeddings</p><pre>db.createCollection(&#39;vector-embeddings&#39;)</pre><p>We’ll be storing all the embeddings in this collection inside the agno database.</p><h3>Configuring MongoDB Atlas for Vector Search</h3><p>We have set up MongoDB which will help us in storing the embeddings, but to perform vector search and answer the query, we need to create a vector search index in Atlas.</p><p>On the Atlas homepage, on the left-side menu, under services, click Atlas Search.</p><p>Next, click on Create Search Index. Select the Search Type as Vector Search.</p><p>Set index name as vector-search.</p><p>In Database and Collection, under agno database, select vector-embeddings collection.</p><p>Select the Configuration method as JSON and click Next.</p><p>In the JSON editor, paste the below JSON and click Next and click Create Vector Search Index.</p><pre>{<br>  &quot;fields&quot;: [<br>    {<br>      &quot;type&quot;: &quot;vector&quot;,<br>      &quot;path&quot;: &quot;embedding&quot;,<br>      &quot;numDimensions&quot;: 1536,<br>      &quot;similarity&quot;: &quot;cosine&quot;<br>    }<br>  ]<br>}</pre><p>Our vector search index with the name vector-search has been created and we can use this index name in our code.</p><h3>Setting up Ollama</h3><p>We’ll be using an embedding model from Ollama called openhermes. You can install Ollama with the help of <a href="https://medium.com/@sharathpai107/running-llms-locally-on-ollama-and-openwebui-72d275782fa2">this guide</a>. After you’re done installing Ollama, we’ll pull openhermes.</p><pre>ollama pull openhermes</pre><p>When you install Ollama, it is by default hosted on <a href="http://localhost:11434/">http://localhost:11434/</a> We’ll be using this host in our code. Check whether the host is running in your browser. If you get the message as Ollama is running, it means that the Ollama server is running.</p><h3>Installing dependencies</h3><p>We’re using Python in our code. We need to install the necessary packages that will run the code. But firstly, we need to install a virtual environment to avoid package conflicts. Inside the terminal of your code folder, run the following commands:</p><pre>python -m venv myenv</pre><pre>myenv\Scripts\activate</pre><p>Now that your virtual environment has been activated, install the following packages:</p><pre>pip install python-dotenv agno pymongo pypdf ollama</pre><h3>Adding secrets to .env</h3><p>Create a .env file inside your code and add the following secret:</p><pre>MONGO_CONNECTION_STRING = &#39;mongodb+srv://&lt;username&gt;:&lt;password&gt;@cluster0.mongodb.net/?retryWrites=true&amp;w=majority&#39;</pre><h3>Let’s code</h3><p>We’ll be using Agno framework inside our code to build the agent. We’ll be writing production level code. Name the file as main.py</p><p>First we’ll import the packages</p><pre>from os import getenv<br>from dotenv import load_dotenv<br>from agno.agent import Agent<br>from agno.knowledge.pdf_url import PDFUrlKnowledgeBase<br>from agno.vectordb.mongodb import MongoDb<br>from agno.embedder.ollama import OllamaEmbedder</pre><p>We’ll create a class called PDFUrlKnowledgeAgent, all the rest of the code will be inside this class</p><pre>class PDFUrlKnowledgeAgent:</pre><p>Next, we’ll be using __init__() function to assign the necessary variables.</p><pre>def __init__(self, urls: list[str]):<br>        load_dotenv(override=True)<br>        self.collection_name = &quot;vector-embeddings&quot;<br>        self.mongo_connection_string = getenv(&quot;MONGO_CONNECTION_STRING&quot;)<br>        self.database_name = &quot;agno&quot;<br>        self.search_index_name = &quot;vector-search&quot;<br>        self.embedder = OllamaEmbedder(id=&quot;openhermes&quot;, host=&#39;http://localhost:11434/&#39;, timeout=1000.0)<br>        self.vector_db = self._init_vector_db()<br>        self.knowledge_base = self._init_knowledge_base(urls)<br>        self.agent = self._init_agent()</pre><p>We’re passing urls as input because we’re gonna add the urls inside the main() function.</p><p>Next, we’ll be initializing the MongoDB vector database</p><pre>def _init_vector_db(self) -&gt; MongoDb:<br>      return MongoDb(<br>          collection_name=self.collection_name,<br>          db_url=self.mongo_connection_string,<br>          database=self.database_name,<br>          search_index_name=self.search_index_name,<br>      )</pre><p>As you can see we’re using the agno database and the vector-embedding collection that we set up earlier</p><p>Now, we’re using PDFUrlKnowledgeBase from Agno as our knowledge base which reads PDFs from urls, converts them into vector embeddings and loads them to the vector database.</p><pre>def _init_knowledge_base(self, urls: list[str]) -&gt; PDFUrlKnowledgeBase:<br>        return PDFUrlKnowledgeBase(<br>            urls=urls,<br>            vector_db=self.vector_db,<br>            embedder=self.embedder<br>        )</pre><p>Here we’re using the embedding model from Ollama that we pulled earlier.</p><p>Now we’ll create the agent which combines the embedder, knowledge base and vector database.</p><pre>    def _init_agent(self) -&gt; Agent:<br>        return Agent(<br>            knowledge=self.knowledge_base,<br>            show_tool_calls=True<br>        )</pre><p>Usually we use a Large Language Model inside the agent, but PDFUrlKnowledgeBase has a built in LLM for query-based answering, so we don’t need to use a LLM seperately.</p><p>Next, we’ll load the documents into the the vector database and run a query which we’ll pass in main function.</p><pre>    def load_documents(self, recreate: bool = False):<br>        self.knowledge_base.load(recreate=recreate)<br><br>    def query(self, prompt: str, markdown: bool = True):<br>        self.agent.print_response(prompt, markdown=markdown)</pre><p>Now, inside the main function, we’ll add the pdf url, call the class and run the query.</p><pre>if __name__ == &quot;__main__&quot;:<br>    urls = [<br>        &quot;https://www.scollingsworthenglish.com/uploads/3/8/4/2/38422447/garth_stein_-_the_art_of_racing_in_the_rain.pdf&quot;<br>    ]<br><br>    runner = PDFUrlKnowledgeAgent(urls=urls)<br>    runner.load_documents(recreate=False)<br>    runner.query(&quot;How did Eve die?&quot;, markdown=True)</pre><p>The entire code should look like this</p><pre>from os import getenv<br>from dotenv import load_dotenv<br>from agno.agent import Agent<br>from agno.knowledge.pdf_url import PDFUrlKnowledgeBase<br>from agno.vectordb.qdrant import Qdrant<br>from agno.vectordb.mongodb import MongoDb<br>from agno.embedder.ollama import OllamaEmbedder<br><br>class PDFUrlKnowledgeAgent:<br>    def __init__(self, urls: list[str]):<br>        load_dotenv(override=True)<br>        self.collection_name = &quot;vector-embeddings&quot;<br>        self.mongo_connection_string = getenv(&quot;MONGO_CONNECTION_STRING&quot;)<br>        self.database_name = &quot;agno&quot;<br>        self.search_index_name = &quot;vector-search&quot;<br>        self.embedder = OllamaEmbedder(id=&quot;openhermes&quot;, host=&#39;http://localhost:11434/&#39;, timeout=1000.0)<br>        self.vector_db = self._init_vector_db()<br>        self.knowledge_base = self._init_knowledge_base(urls)<br>        self.agent = self._init_agent()<br><br>    def _init_vector_db(self) -&gt; MongoDb:  <br>        return MongoDb(<br>            collection_name=self.collection_name,<br>            db_url=self.mongo_connection_string,<br>            database=self.database_name,<br>            search_index_name=self.search_index_name,<br>        )<br><br>    def _init_knowledge_base(self, urls: list[str]) -&gt; PDFUrlKnowledgeBase:<br>        return PDFUrlKnowledgeBase(<br>            urls=urls,<br>            vector_db=self.vector_db,<br>            embedder=self.embedder<br>        )<br><br>    def _init_agent(self) -&gt; Agent:<br>        return Agent(<br>            knowledge=self.knowledge_base,<br>            show_tool_calls=True<br>        )<br><br>    def load_documents(self, recreate: bool = False):<br>        self.knowledge_base.load(recreate=recreate)<br><br>    def query(self, prompt: str, markdown: bool = True):<br>        self.agent.print_response(prompt, markdown=markdown)<br><br><br>if __name__ == &quot;__main__&quot;:<br>    urls = [<br>        &quot;https://www.scollingsworthenglish.com/uploads/3/8/4/2/38422447/garth_stein_-_the_art_of_racing_in_the_rain.pdf&quot;<br>    ]<br><br>    runner = PDFUrlKnowledgeAgent(urls=urls, database=database)<br>    runner.load_documents(recreate=False)<br>    runner.query(&quot;How did Eve die?&quot;, markdown=True)</pre><h3>Running the program</h3><p>Inside the virtual environment in the terminal, run the file</p><pre>python main.py</pre><p>Now the code will run the PDFUrlKnowledgeAgent which will store all the vector embeddings of the PDF’s content into the vector-embedding collection of the agno database, and answer the query from the document by performing vector search.</p><p>You can check all the embeddings by adding the connection string in your MongoDB compass.</p><p>This is how we performed Agentic RAG. Follow for more such blogs!!</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=010ceef5141b" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Running LLMs locally on Ollama and OpenWebUI]]></title>
            <link>https://medium.com/@sharathpai107/running-llms-locally-on-ollama-and-openwebui-72d275782fa2?source=rss-ecc09fa8060a------2</link>
            <guid isPermaLink="false">https://medium.com/p/72d275782fa2</guid>
            <category><![CDATA[open-source]]></category>
            <category><![CDATA[large-language-models]]></category>
            <category><![CDATA[artificial-intelligence]]></category>
            <category><![CDATA[ai]]></category>
            <category><![CDATA[ollama]]></category>
            <dc:creator><![CDATA[Sharath Pai]]></dc:creator>
            <pubDate>Fri, 07 Feb 2025 10:56:22 GMT</pubDate>
            <atom:updated>2025-02-07T20:39:15.995Z</atom:updated>
            <content:encoded><![CDATA[<p>Recently I heard that TCS has blocked all Generative AI websites. When we use ChatGPT and other chat tools, there is a concern of data security. Since these LLMs are closed source, the data goes to the provider’s servers. We don’t want to risk using these tools when we’re working with organizational data. The best way is to run LLMs locally so that the data is safe.</p><p>For this purpose, we use Ollama which is an open source tool to run LLMs locally. It comes with text models, embedding models, vision models and tools, all from HuggingFace. It is a collection of open source models. Let’s see how we can use Ollama</p><h3>Downloading Ollama from its official website</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*bJ-AHiGei6wb88-j2XyrVQ.png" /></figure><p>Go to <a href="https://ollama.com/download">Ollama’s official website</a> and download Ollama for your operating system. I’ll be downloading it for Windows.</p><p>Finish the installation and open command prompt and type the following command.</p><pre>ollama</pre><p>If it shows the list of commands, this means your installation is successful.</p><h3>Choosing a model</h3><p>Now you need to choose a model to run. Go to <a href="https://ollama.com/search">Ollama’s website</a> and search for a model</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*oSoSqpDPB9YnI3Oy95iVMA.png" /></figure><p>We are using deepseek-r1 model for this tutorial. This model contains several parameters such as 1.5b, 7b, etc. These parameters are proportional to the RAM of your system.</p><p><strong>You should have at least 8 GB of RAM available to run the 7B models, 16 GB to run the 13B models, and 32 GB to run the 33B models.</strong></p><p>For this tutorial we’ll run the 1.5B parameters model. You can run 7B or any other parameter’s model based on your requirements.</p><h3>Running the model</h3><p>Go back to cmd and type the following commands.</p><pre>ollama run deepseek-r1:1.5b</pre><p>This will download all the model files of the model and it will take some time. It will occupy the storage of your local disk.</p><p>After all the files are downloaded you’ll see three arrows in the next line.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/404/1*gzzWs8bnNa7XPiGyEyMZhw.png" /></figure><p>Here you have to type the prompt.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*FRFXvb8VjnhCYWRToF0F3Q.png" /></figure><p>Since deepseek r1 is a reasoning model, you may get lengthy answers for your prompts.</p><p>You can type more prompts and in this way we have successfully learnt to run a LLM locally using Ollama!</p><h3>Using Ollama models in code</h3><p>We are running deepseek-r1 model now. Don’t close the command prompt as when you’re running an Ollama model, you’re running the Ollama server which is by default hosted on localhost:11434. Open this port on your browser. You should get a response as Ollama is running. If you don’t get this response you need to run the model using the run command above</p><p>We can use this port as an endpoint to run models in our code.</p><p>We’ll be writing a simple Python script to test the model. <strong>Use Python 3.11 and above.</strong></p><p>First install Open AI using pip in you code editor’s terminal.</p><pre>pip install openai</pre><p>Name the file as ollama.py and paste the below code.</p><pre>from openai import OpenAI<br><br>client = OpenAI(<br>    base_url = &#39;http://localhost:11434/v1&#39;,<br>    api_key=&#39;ollama&#39;, # required, but unused<br>)<br><br>response = client.chat.completions.create(<br>  model=&quot;deepseek-r1:1.5b&quot;,<br>  messages=[<br>    {&quot;role&quot;: &quot;system&quot;, &quot;content&quot;: &quot;You are a helpful assistant.&quot;},<br>    {&quot;role&quot;: &quot;user&quot;, &quot;content&quot;: &quot;What is artificial intelligence?&quot;},<br>  ]<br>)<br>print(response.choices[0].message.content)</pre><p>Run the file in the terminal</p><pre>python ollama.py</pre><p>You’ll get a response in your terminal</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*6tVBtYxEun3zJWa-8ih72Q.png" /></figure><h3>Some Ollama commands</h3><p>Pull a model from the Ollama registry.</p><pre>ollama pull deepseek-r1:1.5b</pre><p>After using ollama pull, we have to use ollama run to run the model. Whereas if we directly run ollama run, it pulls plus runs the model simultaneously. That’s why i don’t use the pull command much.</p><p>Display the information of a model</p><pre>ollama show deepseek-r1:1.5b</pre><p>Display list of downloaded models</p><pre>ollama ls</pre><p>Show all the currently running models.</p><pre>ollama ps</pre><p>Remove a model</p><pre>ollama rm deepseek-r1:1.5b</pre><p>You need to remove a model if you’re no longer using it for future use to free up the memory. The LLM will take a lot of time to generate the response.</p><h3>Using OpenWebUI as a web interface</h3><p>Running a model on command line interface can be boring for some. We need a ChatGPT like interface where we can use our models. We’ll be using OpenWebUI as a web interface.</p><p>For this tutorial we’ll be running OpenWebUI in Python. You can also run it using Docker. Refer the <a href="https://github.com/open-webui/open-webui">GitHub documentation</a>.</p><p>Let’s install OpenWebUI with pip. <strong>Use Python 3.11 and above. </strong>Open a new terminal on the command prompt and type the following commands.</p><pre>pip install open-webui</pre><p>Run the server.</p><pre>open-webui serve</pre><p>OpenWebUI server is by default hosted on localhost:8080. Open it in your browser. You’ll be required to create an admin account and then the following screen appears.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*PvaDaUgJC6DLukh-NOhqDQ.png" /></figure><p>And that’s it. You can use OpenWebUI like ChatGPT for Ollama models and you can choose models above, provided models are pulled and downloaded on Ollama.</p><p>Additionally, don’t forget to enable the Memory option by clicking on your username on bottom left, then Settings &gt; Personalization.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/669/1*LWITwuyNzsP_ooXxWOtJyA.png" /></figure><p>Enabling memory allows the LLM to remember the previous conversations and we don’t need to give the same context again and again.</p><h3>Web Search in OpenWebUI</h3><p>We also use OpenWebUI to perform web search when required since Ollama can’t perform web search on CLI.</p><p>OpenWebUI acts as an agent when we use LLMs with certain tools. You might have seen ChatGPT and Deepseek Chat using web search. Both of them are agents which use gpt-4o and deepseek-r1 as LLMs and web search as tools.</p><p>Here we’ll see how we can integrate web search tool in OpenWebUI.</p><p>On the bottom left click on your name then Settings &gt; Admin Settings &gt; Web Search</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*lhGnmGTjlkMkp-6FR7dtGQ.png" /></figure><p>Enable Web Search and select any Web Search Engine as per your liking. I have selected bing. Add the Bing Search V7 Endpoint as https://api.bing.microsoft.com/v7.0/search and enter your Bing Search V7 Subscription Key which you can obtain on Microsoft’s website by searching Bing API Key. Click on Save.</p><p>Now click on New Chat and in the area to enter prompt, you’ll get a Web Search Option. Click on it to enable Web Search. You can run a query regarding the any recent news and it will search the web and return the response.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*qnZqt4kBEQLdmfVUOvNu4g.png" /></figure><h3>Demerits</h3><p>Since most of us are running models upto 7B parameter models on an 8 GB RAM system, the LLM responses may not be accurate since these models are quantized models of their original models of hundreds of parameters. Quantization is performed to reduce the size of the original large model but it reduces acccuracy due to loss of precision in model weights.</p><p>If we want to achieve greater accuracy, we need to use higher parameter models of over 33B parameters which is achievable using GPUs.</p><p>The good thing is we can use these LLMs for our daily work without worrying about data security since we are running these models on our local server and it can be used in small to medium scale applications and it’s for free since it’s open source!</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=72d275782fa2" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Capabilities of OpenAI]]></title>
            <link>https://medium.com/@sharathpai107/capabilities-of-openai-3a3122643bf8?source=rss-ecc09fa8060a------2</link>
            <guid isPermaLink="false">https://medium.com/p/3a3122643bf8</guid>
            <category><![CDATA[gpt]]></category>
            <category><![CDATA[openai]]></category>
            <category><![CDATA[llm]]></category>
            <dc:creator><![CDATA[Sharath Pai]]></dc:creator>
            <pubDate>Fri, 10 Jan 2025 13:35:29 GMT</pubDate>
            <atom:updated>2025-01-10T13:43:03.682Z</atom:updated>
            <content:encoded><![CDATA[<p>OpenAI is undoubtedly the best LLM provider with a variety of LLMs for different tasks. What sets OpenAI and its LLMs apart from other providers, let’s find out.</p><h3>GPT-4o being the best multimodal model</h3><p>GPT-4o accepts image and text as inputs and returns the text as output. Gemini already had an image-to-text plus there were lot other traditional OCRs. But GPT-4o has beaten them all. Other OCRs and VLMs may mispell some characters, have censored models which accept a particular set of images, may not work well in case of low-contrast images. GPT-4o has taken care of all the limitations and extracts text perfectly almost all time and do I even need to talk about the wonders of text-to-text of GPT?</p><h3>DALL-E’s unprecedented creativity and versatility</h3><p>DALL-E is OpenAI’s text-to-image model which generates high quality, high resolution imaginative images from a mere surreal prompt. If you mention a prompt like “a cat made of sushi”, it generative a highly creative and plausible image. Earlier models struggled to understand the context and produced low-quality images.</p><h3>Whisper standing out among open-source speech-to-text models</h3><p>Whisper is one of the only open source models of OpenAI. Whisper is far ahead from its speech-to-text competitors. It supports upto 90 languages of ISO 639 whereas other models support upto 50. It can perform translation of speech from one language to other as well as transcription of speech and is highly robust to noise. It can transcribe even a low resource language like Swahili without fine-tuning the data. The best!</p><h3>Master of system prompt</h3><p>If you’re a LLM engineer or researcher, you might often wonder how it returns such accurate results in both playground and API? It’s because of how well it’s system prompt is written. System prompt is a set of precise instructions for a LLM at the backend.</p><h3>GPT-4o as an agent</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/923/1*0hXkNe4sVN0F8-7EexaAzQ.png" /><figcaption><em>GPT-4o giving web search results for latest information</em></figcaption></figure><p>Agents are a combination of LLMs with tools such as web search, youtube search, etc. Older GPTs didn’t provide up to date information. To be able to achieve this, GPT-4o has been combined with a web search tool to act as an agent to provide up to date information from the web like Gemini and Perplexity. GPT-4o uses Microsoft Bing as its web search tool. If a web search tool is combined with a good LLM, it produces more precise results. This way GPT-4o agent can take down Google in upcoming years.</p><h3>OpenAI as a proxy client</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*_j5qGOo61LE0pJo99GKfzA.png" /><figcaption>OpenAI as a client for vLLM supported models</figcaption></figure><p>You must have hosted open source LLMs using vLLM. vLLM uses OpenAI as the default client for all the LLMs since OpenAI is widely used and it’s well documented. You don’t need an OpenAI API key for it, just use OpenAI as a client.</p><h3>Collaborations</h3><p>OpenAI has shared its models with multiple LLM providers. Since Whisper model is open-source, it is available on HuggingFace. If you want to use its API, it is also available on Groq. Additionally GPT models are also available on Azure which can be accessed through your organization.</p><h3>Demerits</h3><p>Even the best have certain demerits.</p><ul><li><strong>Pricing:</strong> OpenAI’s API is expensive. Alternatives: Groq (almost free)</li><li><strong>Rate Limits:</strong> GPT can’t take longer texts in its user prompt and can max accept 3 images in input for 1 day. Alternatives: Deepseek (no rate limits on prompt length), Gemini (can supply as much images as you want)</li><li><strong>Coding:</strong> while GPT is in top 5 for understanding the context of code, it’s not the best. Alternatives: Codestral 22B (the king for coding)</li></ul><p>Apart from its demerits, OpenAI has been the king of LLM providers. All the other LLM providers use OpenAI’s pretrained models to create their LLMs. This is the amount of reputation OpenAI has achieved</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=3a3122643bf8" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Deploying a Gradio Application on  Spaces]]></title>
            <link>https://medium.com/@sharathpai107/deploying-a-streamlit-application-on-spaces-6138ed4d3268?source=rss-ecc09fa8060a------2</link>
            <guid isPermaLink="false">https://medium.com/p/6138ed4d3268</guid>
            <category><![CDATA[github-actions]]></category>
            <category><![CDATA[hugging-face-tutorials]]></category>
            <category><![CDATA[hugging-face]]></category>
            <dc:creator><![CDATA[Sharath Pai]]></dc:creator>
            <pubDate>Fri, 29 Nov 2024 09:00:53 GMT</pubDate>
            <atom:updated>2025-08-07T18:26:14.622Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*hjwRhVACxVg1w_8L.png" /></figure><p>In this tutorial, I’ll tell you about the easiest way of deploying your Gradio application, without any cloud-based knowledge and tools and for absolutely free of cost.</p><p>HuggingFace Spaces acts like a playground to many open-source applications where users can test models for free without code. This saves time as if we code, we may need to download dependencies, models which takes a lot of time and some models may be larger than the CPU RAM (8 GB RAM in my case).</p><p>I recently created a Gen AI project on Gradio and will be telling you the steps to deploy it. Let’s break it down into steps</p><h3><strong>Creating a HuggingFace Space</strong></h3><p>Before starting to code, go to <a href="https://huggingface.co/spaces">https://huggingface.co/spaces</a> and create an account if you haven’t created it yet.</p><p>Next, click on “Create New Space”. Give space name as per you requirements and select Gradio as the Space SDK. Leave rest of the options as it is and click on “Create Space”.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/425/1*N2XIA9zlBss1pb6sD87W7Q.png" /></figure><p>A repository will be created like below.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*7X52bKBGBoWthIqemUspfw.png" /></figure><p>Clone the repository in your project directory</p><pre>git clone https://huggingface.co/spaces/&lt;username&gt;/&lt;repository-name&gt;<br>cd repository-name</pre><h3><strong>Using GitHub to host your code</strong></h3><p>Now you can start coding on the cloned repository. My directory structure looks like this.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/626/1*9vPbEoExHthXHUTpbt8ePw.png" /></figure><p>For any Gradio based app, app.py,requirements.txtand README.mdare the main files required for deployment.</p><p>app.py: Acts as the main python file of your Gradio application</p><p>requirements.txt: Contains the packages or dependencies you used in the project.</p><p>README.md: Acts like a yaml containing about the framework used in the project. Do not change anything in it, else the deployment will fail. You can add more details explaining about your setup but don’t remove any line which was provided by HF Spaces.</p><p>Now let’s create a repository on GitHub</p><p>Go to https://github.com/&lt;your-username&gt; . Go to the repository section and click on ‘New’. Give your repository a name and keep rest of the fields as it is.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/481/1*_cSVCenn05cW87AauiCd5w.png" /></figure><p>Then click on Create repository and you’ll be directed to a screen that has the folllowing commands.</p><pre>git remote remove origin<br>git init<br>git add .<br>git commit -m &quot;first commit&quot;<br>git branch -M main<br>git remote add origin https://github.com/&lt;username&gt;/&lt;repository-name&gt;.git<br>git push -u origin main</pre><p>Run these commands inside the terminal of your code editor. Don’t forget to change the remote origin which can be found on your repository.</p><p>The repository link is <a href="https://github.com/Sharath1036/engineering-branch-explorer">https://github.com/Sharath1036/engineering-branch-explorer</a>. You can go through the contents of each file for a better understanding.</p><h3>Using Git LFS for tracking large files (Optional)</h3><p>If you’re using a model or a file which exceeds 10MB, GitHub isn’t able to deploy it into production. Hence we use Git LFS which tracks large files so that it can be easily deployed into production.</p><p>To check the file size of files in a directory, use the following command (for windows),</p><pre>Get-ChildItem | Select-Object Name, @{Name=&quot;Size(MB)&quot;;Expression={[math]::Round($_.Length / 1MB, 2)}} | Format-Table -AutoSize</pre><p>This command gives you the file size of all files in MB. Do this for all directories.</p><p>If you find any file exceeding 10 MB, we need to track it using Git LFS. Let’s say we have files named haarcascade_frontalface_default.xml and files having .t7 extenstion exceeding 10 MB. Let’s track them using Git LFS.</p><p>Initialize Git LFS in your repository</p><pre><br>cd your-repository<br>git lfs install</pre><p>Track your large files</p><pre>git lfs track &quot;haarcascade_frontalface_default.xml&quot; # tracks haarcascade file<br>git lfs track &quot;*.t7&quot; # tracks files having .t7 extension</pre><p>A file named .gitattributes will be created. Make sure to track it.</p><pre>git add .gitattributes</pre><p>Add and commit your large files and push it</p><pre>git add .<br>git commit -m &quot;Add large files using Git LFS&quot;<br>git push -u origin main</pre><p>Verify if large files are tracked</p><pre>git lfs ls-files</pre><h3>Generating HuggingFace Token</h3><p>Go to <a href="https://huggingface.co/settings/tokens">https://huggingface.co/settings/tokens</a> and create a new token. Copy it and store it somewhere safely.</p><p>In your GitHub repository, go to Settings &gt; Security &gt; Secrets and Variables &gt; Actions. Click on New Repository Secret . Set the name as HF_TOKEN and its value that you stored earlier in the Secret field and click on Add Secret.</p><h3>Creating a CI/CD pipeline using GitHub Actions</h3><p>Whenever we make changes in our code and push it to GitHub, we need a gateway that will detect the changes in GitHub and will update the application on HuggingFace Spaces with the changes. That’s where CI/CD pipeline comes into picture.</p><p>The workflow of the pipeline is that it will push the code from GitHub to our HuggingFace Space and trigger the changes from GitHub to the space.</p><p>In your GitHub repository, go to Actions and click on set up a workflow youself.</p><p>Add the below code</p><pre>name: Sync to Hugging Face hub<br>on:<br>  push:<br>    branches: [main]<br><br>  # to run this workflow manually from the Actions tab<br>  workflow_dispatch:<br><br>jobs:<br>  sync-to-hub:<br>    runs-on: ubuntu-latest<br>    steps:<br>      - uses: actions/checkout@v3<br>        with:<br>          fetch-depth: 0<br>          lfs: true<br>      - name: Push to hub<br>        env:<br>          HF_TOKEN: ${{ secrets.HF_TOKEN }}<br>        run: git push --force https://HF_USERNAME:$HF_TOKEN@huggingface.co/spaces/HF_USERNAME/SPACE_NAME main</pre><pre>Replace HF_USERNAME and SPACE_NAME with your HuggingFace username and space name respectively. For more details about the pipeline, referhttps://huggingface.co/docs/hub/en/spaces-github-actions</pre><p>The changes will be pushed into the files section of the HuggingFace Space. We are force pushing changes because the files section of the space ain’t empty and has a couple of files.</p><h3>Adding Environment Variables to the HuggingFace Space (optional)</h3><p>In case you are using any API keys important for running the application, you need to add that key into the environment for the users to be able to use it. Be aware about the costs and rate limits of the API you’re using since your space is being made available to the public to use it.</p><p>In your HuggingFace Space, go to Settings &gt; Secrets and Variables. Add the API key and you’re good to go.</p><p>Now your HuggingFace Space acts as a playground for users to use and test it without needing to code.</p><p>Note: I have updated the article by changing Streamlit to Gradio as Streamlit has been really bad in processing PDF files. So, I suggest everyone to deploy on Gradio.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=6138ed4d3268" width="1" height="1" alt="">]]></content:encoded>
        </item>
    </channel>
</rss>