<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Shubham Mishra on Medium]]></title>
        <description><![CDATA[Stories by Shubham Mishra on Medium]]></description>
        <link>https://medium.com/@shubhammishra892?source=rss-34d7ac76b1ae------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/1*-Qq4m_StBzdKx8lyvz5osg.jpeg</url>
            <title>Stories by Shubham Mishra on Medium</title>
            <link>https://medium.com/@shubhammishra892?source=rss-34d7ac76b1ae------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Sat, 21 Nov 2026 09:14:15 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@shubhammishra892/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[NVSentinel: Running your GPUs 24x7…]]></title>
            <link>https://medium.com/@shubhammishra892/nvsentinel-running-your-gpus-24x7-8f6e4fc62d8a?source=rss-34d7ac76b1ae------2</link>
            <guid isPermaLink="false">https://medium.com/p/8f6e4fc62d8a</guid>
            <category><![CDATA[nvidia-gpu]]></category>
            <category><![CDATA[llm-training]]></category>
            <category><![CDATA[kubernetes]]></category>
            <category><![CDATA[gpu-reporting]]></category>
            <category><![CDATA[gpuaas]]></category>
            <dc:creator><![CDATA[Shubham Mishra]]></dc:creator>
            <pubDate>Mon, 25 May 2026 16:27:30 GMT</pubDate>
            <atom:updated>2026-05-25T16:27:30.086Z</atom:updated>
            <content:encoded><![CDATA[<p>It has been long since I wrote my experiences here, partly have been busy with customers in helping them achieve their Gen AI goals as much as I can. That being said, I met a customer, a neo cloud, GPUaaS provider last week and one of the things they mentioned to me, was they were having tough time maintaining existing fleets of H200 SXM GPUs. This provider has 4000+ H200 all connected with state of the art IB backend in proper RAIL topology. Provider has left no stone un-turned to optimize the compute, network, storage etc. to ensure the end users have best outcomes they can for a fair price. However one of the key challenges, this provider often faces is GPU FATIGUE… Yes, even GPUs do get tired! When they do, the outcome is less than ideal:</p><ul><li><strong>Silent data corruption</strong> (faulty GPUs producing wrong results).</li><li><strong>Cascading job failures</strong> (one bad node kills multi-day training runs).</li><li><strong>Idle healthy GPUs</strong> while waiting for manual triage.</li><li><strong>Pager fatigue</strong> for on-call teams (hours/days of manual diagnosis).</li><li><strong>Low utilization &amp; SLA risk</strong> in shared/multi-tenant environments.</li></ul><p>Provider is looking for a solution which just does not report on GPU setup and current utilization (DCGM) but also helps in remediating the issue and automatically removing the challenged GPUs from the production. This is where <a href="https://github.com/nvidia/nvsentinel">NVSentinel </a>comes into play, it is NVIDIA’s open-source, Kubernetes-native fault detection, classification, and automated remediation system purpose-built for large-scale GPU-accelerated clusters. It continuously monitors GPU nodes (and NVSwitch fabric) for hardware and software faults, then automatically quarantines unhealthy nodes, drains workloads, and triggers repair workflows — without human intervention.</p><h3><strong>Key features include:</strong></h3><ul><li><strong>Real-time, multi-source monitoring</strong> (GPU via DCGM, system logs, cloud provider events, custom Kubernetes objects).</li><li><strong>Policy-driven fault classification</strong> using CEL (Common Expression Language) rules.</li><li><strong>Automated node lifecycle</strong>: cordon → drain (with per-namespace strategies) → remediation CRD → optional reboot/terminate.</li><li><strong>Modular &amp; pluggable</strong> architecture (use only the pieces you need: monitoring only, full closed-loop, or custom monitors).</li><li><strong>High availability</strong> (replicas + leader election, MongoDB change-stream driven).</li><li><strong>Metadata enrichment</strong> (cloud provider context, GPU topology, DCGM/driver versions).</li><li><strong>Event export</strong> in CloudEvents format for external systems.</li><li><strong>Graceful eviction</strong> with configurable timeouts and Slony drain support.</li><li><strong>Persistent event store</strong> (MongoDB) for auditing and pattern analysis.</li><li><strong>Kubernetes-native</strong> (NodeConditions, labels, CRDs for maintenance).</li></ul><h3><strong>Deep Dive Architecture:</strong></h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*JGj3fljz-fT8Zzx4.png" /></figure><p>NVSentinel uses a <strong>disaggregated microservices design</strong> with independent modules coordinated via <strong>MongoDB </strong>change streams + Kubernetes API. All health monitors speak gRPC to a central Platform Connector.</p><ol><li><strong>Health Monitors</strong> (pluggable, extensible):</li></ol><ul><li><strong>GPU Health Monitor</strong> → DCGM diagnostics (thermal, ECC correctable/uncorrectable, XID errors, etc.).</li><li><strong>Syslog Health Monitor</strong> → journalctl for driver crashes, kernel faults, hardware errors.</li><li><strong>CSP Health Monitor</strong> → AWS/GCP/OCI maintenance &amp; hardware events.</li><li><strong>Kubernetes Object Monitor</strong> → CEL-based policies on any K8s resource.</li></ul><p><strong>2. Core Processing Modules</strong>:</p><ul><li><strong>Platform Connectors</strong> — gRPC server, validation, persistence, NodeCondition updates.</li><li><strong>Fault Quarantine</strong> — CEL rules → cordon node.</li><li><strong>Node Drainer</strong> — graceful pod eviction with namespace-specific strategies.</li><li><strong>Fault Remediation</strong> — creates Maintenance CRDs to hand off to external repair systems.</li><li><strong>Janitor</strong> — cloud-provider reboots/terminations.</li><li><strong>Health Events Analyzer</strong> — pattern detection &amp; recommended actions.</li><li><strong>Labeler + Metadata Collector</strong> — auto-labels nodes, gathers GPU/NVSwitch topology.</li><li><strong>Log Collector</strong> — bundles diagnostics for troubleshooting.</li><li><strong>Event Exporter</strong> — streams to external SIEM/observability.</li></ul><p><strong>3. Data Flow</strong> (for the errors encountered, issues recorded):</p><ul><li>Monitors detect → gRPC event.</li><li>Platform Connector persists to MongoDB + updates K8s.</li><li>Watchers (Quarantine/Drainer/etc.) react via change streams.</li><li>Actions executed on Kubernetes API + external hooks.</li></ul><p>If NVSentinel detects an error or an issue, from one of its connectors (Continuous polling + event-driven (DCGM, logs, CSP APIs)), it goes through below process to make sure the administrator is aware of the issue and the unhealthy node is cordoned or removed from the fleet.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*8XU2O0VL3LKpcQBFlYkKdg.jpeg" /><figcaption><a href="https://developer-blogs.nvidia.com/wp-content/uploads/2025/12/nvsentinel-remediation-workflow-png.webp">https://developer-blogs.nvidia.com/wp-content/uploads/2025/12/nvsentinel-remediation-workflow-png.webp</a></figcaption></figure><h3><strong>Supported Environments and Compatibility:</strong></h3><ul><li><strong>Kubernetes</strong>: 1.25+ (tested on bare-metal, cloud, DGX Cloud).</li><li><strong>GPUs</strong>: All DCGM-supported data-center GPUs (H100, A100, B200, GB200/GB300, A30/A40, older Tesla, etc.).</li><li><strong>Platforms</strong>: DGX A100/H100, HGX A100/H100/B200, NVSwitch systems.</li><li><strong>Required</strong>: <strong>NVIDIA GPU Operator + DCGM (included)</strong>.</li><li><strong>Optional</strong>: cert-manager, Prometheus, cloud-provider credentials (AWS/GCP/OCI), external repair systems.</li><li><strong>Integrations</strong>: <a href="https://rafay.co/ai-and-cloud-native-blog/automated-gpu-health-monitoring-with-nvidia-nvsentinel-on-the-rafay-platform">Rafay </a>(via Cluster Blueprints for fleet-wide consistency), <a href="https://blogs.nvidia.com/blog/optional-data-center-fleet-management-software/">NVIDIA GPU Health Service</a> (fleet-level telemetry complement).</li></ul><p>So for this GPUaaS provider we went ahead and have deployed this solution, allowing them to monitor and manage GPU failures and issues using NVSentinel. While this sounds pretty slick however as far as I know this tool has no support for SLURM driven fleets (correct me in comments, if I missed this), also this does not work with AMD GPUs (like obviously!). We expect issue detetcion and isolation to be reduced from <strong>Minutes vs. hours</strong> hence ensuring h<strong>igher utilization</strong> and better SLAs for GPU clouds.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/784/0*W8poO-cLElRD5WoI.jpg" /></figure><p>It seems like this has proven itself at scale since NVIDIA runs it on &gt;40k GPUs internally; 13+ releases and external contributions by early 2026, more details <a href="https://developer.nvidia.com/blog/automate-kubernetes-ai-cluster-health-with-nvsentinel/">here</a>. If you have large NVIDIA GPU fleets, this tool is worth a try, from personal experience I can say, pretty staight forward to deploy and manage as well, if you are GPU provider, K8s is not an issue for you, and this tool integrates very well ! <br>If you like my work, feel free to get me a <a href="https://buymeacoffee.com/livelyshubham">cappuccino</a>!</p><figure><a href="https://buymeacoffee.com/livelyshubham"><img alt="" src="https://cdn-images-1.medium.com/max/700/0*lafzmhNdSOKGA4Yw.png" /></a></figure><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=8f6e4fc62d8a" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[NVIDIA’s Most Efficient Open Model (yet): Nemotron 3 Super Redefines Agentic AI]]></title>
            <link>https://medium.com/@shubhammishra892/nvidias-most-efficient-open-model-yet-nemotron-3-super-redefines-agentic-ai-60b99ee02270?source=rss-34d7ac76b1ae------2</link>
            <guid isPermaLink="false">https://medium.com/p/60b99ee02270</guid>
            <category><![CDATA[nvidia]]></category>
            <category><![CDATA[llm]]></category>
            <category><![CDATA[agentic-ai]]></category>
            <category><![CDATA[ai]]></category>
            <category><![CDATA[ai-model-training]]></category>
            <dc:creator><![CDATA[Shubham Mishra]]></dc:creator>
            <pubDate>Tue, 17 Mar 2026 03:12:22 GMT</pubDate>
            <atom:updated>2026-03-17T03:12:22.876Z</atom:updated>
            <content:encoded><![CDATA[<p>NVIDIA released Nemotron 3 Super just few days before GTC 2026 and for open model enthusiasts, it was a blast! There is a lot of uncover and how models like these change the way inferencing will be done for complex multi-agent workflows. Nemotron 3 Super, is a 120 billion (active 12 billion) parameter hybrid Mamba-Attention Mixture-of-Experts model. As per NVIDIA it is the first model to:</p><ol><li>be pre-trained in NVFP4.</li><li>leverage LatentMoE, a new Mixture-of-Experts architecture that optimizes for both <strong>accuracy per FLOP and accuracy per parameter</strong> and</li><li>include MTP layers for inference acceleration through native speculative decoding.</li></ol><p>The final model supports up to <strong>1M context length</strong> and achieves comparable accuracy on common benchmarks, while also achieving up to 2.2× and 7.5× higher inference throughput compared to GPT-OSS-120B and Qwen3.5–122B, respectively. Nemotron 3 Super datasets, along with the base, post-trained, and quantized checkpoints, are open-sourced on HuggingFace.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*kMJkcc5ef3er3hnqawRzNg.png" /><figcaption><a href="https://research.nvidia.com/labs/nemotron/files/NVIDIA-Nemotron-3-Super-Technical-Report.pdf">https://research.nvidia.com/labs/nemotron/files/NVIDIA-Nemotron-3-Super-Technical-Report.pdf</a></figcaption></figure><p>Checkpoints for the Model can be found at:</p><ol><li><a href="https://huggingface.co/nvidia/NVIDIA-Nemotron-3-Super-120B-A12B-NVFP4"><strong>Nemotron 3 Super 120B-A12B NVFP4</strong></a> : post-trained and NVFP4 quantized model.</li><li><a href="https://huggingface.co/nvidia/NVIDIA-Nemotron-3-Super-120B-A12B-FP8"><strong>Nemotron 3 Super 120B-A12B FP8</strong></a> : post-trained and FP8 quantized model.</li><li><a href="https://huggingface.co/nvidia/NVIDIA-Nemotron-3-Super-120B-A12B-BF16"><strong>Nemotron 3 Super 120B-A12B BF16</strong></a> : post-trained model.</li><li><a href="https://huggingface.co/nvidia/NVIDIA-Nemotron-3-Super-120B-A12B-Base-BF16"><strong>Nemotron 3 Super 120B-A12B Base BF16</strong></a> : base model.</li><li><a href="https://huggingface.co/nvidia/Qwen3-Nemotron-235B-A22B-GenRM-2603"><strong>Qwen3-Nemotron-235B-A22B-GenRM-2603</strong></a> : GenRM used for RLHF</li></ol><p>Datasets on which the model was trained can be found at:</p><ol><li><a href="https://huggingface.co/datasets/nvidia/Nemotron-Pretraining-Specialized-v1.1"><strong>Nemotron-Pretraining-Specialized-v1.1</strong></a> : a collection of synthetic datasets aimed to improve LLM capabilities in code concepts and algorithms, formal logic, economics, and multiple choice questions.</li><li><a href="https://huggingface.co/collections/nvidia/nemotron-post-training-v3"><strong>Nemotron-Super-Post-Training-Data</strong></a> : a collection of RL environments and SFT datasets targeting a broad range of agentic capabilities.</li></ol><p><strong>Model Architecture:</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/913/1*_JBySzHTsgLew1e_JfsJEg.png" /><figcaption><a href="https://research.nvidia.com/labs/nemotron/files/NVIDIA-Nemotron-3-Super-Technical-Report.pdf">https://research.nvidia.com/labs/nemotron/files/NVIDIA-Nemotron-3-Super-Technical-Report.pdf</a></figcaption></figure><ul><li>Uses <strong>LatentMoE</strong>, a new MoE design that squeezes data into a smaller “latent” space for expert processing, reducing memory and communication costs while boosting accuracy (more on this below).</li><li>Incorporates <strong>Multi-Token Prediction (MTP)</strong> layers to predict multiple future tokens at once, improving quality and enabling “speculative decoding” for faster inference (up to 3–7 tokens guessed ahead, verified in one go).</li><li>Hybrid layering: Mostly Mamba for speed (linear time, small state), with periodic attention “anchors” for global context. Supports up to 1 million token context length.</li><li>Total layers: 88, with 512 experts per MoE layer, activating top-22 per token.</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/771/1*luFx_ghoUYg8t9fh33PoaQ.png" /></figure><p><strong>Understanding Latent MoE:</strong></p><p>Okay, imagine you’re building a super-smart AI like a giant brain made of “experts” — each expert is a mini-brain specialized in something, like one for math, one for coding, one for history. In a regular Mixture-of-Experts (MoE) setup, the AI picks a few experts for each word (or “token”) it processes, but this gets expensive because experts deal with tons of data (think high-res images vs. thumbnails). It <strong>uses a lot of memory and “bandwidth”</strong> (like downloading huge files over slow Wi-Fi), especially when the AI has to shuffle data between experts across computers.</p><p>Latent MoE is like a smarter, cost-cutting version. Before sending data to the experts, <strong>the AI “compresses” it — like zipping a file or turning a detailed essay into bullet points</strong> (from dimension ‘d’ to a smaller ‘ℓ’, say from 4096 to 1024 bits). Now, the experts work on this tiny, compressed version, which saves a ton on memory (less data to load) and communication (less to send around). To keep the AI just as smart, they use the savings to add <em>more</em> experts (from N to N * d/ℓ) and activate <em>more</em> per token (from K to K * d/ℓ). It’s like hiring more workers but giving them shorter shifts — overall work gets done better without extra cost.</p><p>Why is this relevant? In real hardware (like GPUs), <strong>the bottleneck is often loading/sending data, not the math itself</strong>. Latent MoE fixes that by shrinking the “hidden” data highway, but keeps non-expert parts (like routing) full-size to avoid quality drops. <strong>Result: Better accuracy per byte/FLOP, meaning the AI thinks smarter and faster on the same hardware</strong>. Think of it as upgrading from a clunky old laptop to a sleek gaming rig without buying new parts — just optimizing how it handles tasks. Cool for big models like this one!</p><p><strong>What is Multi-Token Prediction (MTP)?</strong></p><p>Normally, when training an AI like this, it learns by predicting just the <em>next single word</em> (or “token”) in a sentence, one step at a time — like guessing “cat” after “The quick brown fox jumps over the lazy dog and sees a…”. But MTP flips that: instead of one token, the model is trained to predict <em>multiple future tokens</em> all at once from each position in the text. For example, it might try to guess the next 2, 3, or even 7 words ahead. This does two big things:</p><ol><li><strong>Improves Quality</strong>: By forcing the model to think ahead further, it builds better “understanding” of longer patterns and dependencies in language. Think of it like practicing chess by planning 5 moves ahead instead of 1 — it makes the AI better at complex reasoning, math, code, or stories, reducing errors and improving benchmarks (like scoring higher on tests for validation loss or accuracy).</li><li><strong>Boosts Speed (Speculative Decoding)</strong>: During actual use (inference), those extra prediction “heads” (mini-parts of the model) act like a built-in “guesser.” The AI drafts several tokens ahead speculatively, then verifies them in one quick pass. If the guess is right, it accepts them; if not, it corrects. This speeds up generation by 2–3x or more without needing extra hardware or a separate “draft” model. In Nemotron 3 Super, they share weights across these heads to make guessing more reliable even for longer drafts, avoiding mismatches between training and real use.</li></ol><p>In the paper, they tested it on SPEED-Bench (a benchmark for this), where Nemotron 3 Super gets an average “acceptance length” of 3.45 tokens per verification step — better than competitors like DeepSeek-R1 (2.70) or Qwen3-Next (3.33). It shines in areas like coding (3.78) and math (3.73). Overall, it’s like giving the AI a crystal ball for faster, sharper responses while keeping quality high. Complete details can be found <a href="https://research.nvidia.com/labs/nemotron/files/NVIDIA-Nemotron-3-Super-Technical-Report.pdf">here</a>.</p><p><strong>What it means for Multimodal and Agentic Inferencing?</strong></p><ul><li><strong>Native Speculative Decoding with 3.45 x Average Acceptance Length</strong> Shared-weight MTP heads (trained on multiple offsets) achieve the highest acceptance rate on SPEED-Bench (3.45 tokens accepted per verification step). In agentic loops this means <strong>2–3.5 x fewer forward passes</strong>; in multimodal settings it speeds up text continuation after a single vision encoding pass.</li><li><strong>Robust Recursive Drafting for Long-Horizon Tasks</strong> Unlike independent-offset heads, shared parameters regularize the model across prediction horizons (4–7 tokens). This gives stable acceptance even at deeper drafts — critical for agentic trajectories with many tool calls or for multimodal interleaved sequences (image analysis → multi-step plan → action).</li><li><strong>Stronger Multi-Step Dependency Modeling</strong> Predicting multiple future tokens during training creates richer hidden states that capture longer-range structure. This directly improves multi-step reasoning and planning in agentic workflows (SWE-Bench, terminal use) and in multimodal reasoning chains that span image context + text actions.</li><li><strong>Better Throughput–Latency Pareto Frontier (Especially at Small Batch Sizes)</strong> Adding MTP draft depth (D=1 → D=3) shifts the entire TPS vs. median latency curve upward (see below figure).</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/512/1*8xGOhenyetH8pP2AH7oYnw.png" /></figure><p>Agentic applications (real-time tool use) and multimodal interactive agents (chat-with-vision) benefit most because they typically run at batch size 1–8.</p><ul><li><strong>Zero Extra Parameters or External Draft Model Overhead</strong> MTP reuses the same model weights (just 2 extra shared heads). This keeps VRAM footprint identical — a huge win for multimodal models (which already burn memory on vision encoders + long KV cache) and for agentic deployments on edge or cost-sensitive hardware. No need for a separate smaller draft model like in classic speculative decoding.</li></ul><p>By open-sourcing the full model checkpoints in <strong>BF16, FP8, and NVFP4 formats</strong>, the specialized pre-training datasets, the rich agentic post-training data, and the complete training recipe on Hugging Face and the Nemotron Developer Repository, NVIDIA is making frontier-level technology truly accessible. This release showcases their dedication to not just building powerful models, but to sharing the innovations — from LatentMoE and Multi-Token Prediction to efficient hybrid Mamba-Transformer architectures — with the broader community. In a world increasingly dominated by closed models, NVIDIA’s transparent and hardware-optimized approach stands out, empowering developers to create the next generation of capable and efficient agentic AI systems.</p><p>If you like my work, feel free to get me a <a href="https://buymeacoffee.com/livelyshubham">cappuccino</a>!</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/875/0*9Xb4fbHoVPpGhbRi.png" /></figure><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=60b99ee02270" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[SemiAnalysis GPUaaS ClusterMAX 2.0 Results]]></title>
            <link>https://medium.com/@shubhammishra892/semianalysis-gpuaas-clustermax-2-0-results-dd3df07d0d30?source=rss-34d7ac76b1ae------2</link>
            <guid isPermaLink="false">https://medium.com/p/dd3df07d0d30</guid>
            <category><![CDATA[llm-inference]]></category>
            <category><![CDATA[gpu]]></category>
            <category><![CDATA[llm-training]]></category>
            <category><![CDATA[gpuaas]]></category>
            <dc:creator><![CDATA[Shubham Mishra]]></dc:creator>
            <pubDate>Tue, 18 Nov 2025 10:25:49 GMT</pubDate>
            <atom:updated>2025-11-18T11:00:52.819Z</atom:updated>
            <content:encoded><![CDATA[<p>If you are into GPUaaS solutions as much as I am, you should absolutely read the complete report <a href="https://www.clustermax.ai/v2#">here</a>. This requires a 500 USD subscription for complete report and all the details in it. Just in case, you do not have the time or patience to go about it, I have written down some key take aways from the report, Happy Reading ahead!</p><p>ClusterMAX 2.0 is the second major version of the ClusterMAX™ Rating System, an independent benchmarking and rating framework developed by SemiAnalysis (a semiconductor and AI industry research firm). Launched on November 6, 2025, it serves as a comprehensive buyer’s guide and “power ranking” for GPU cloud providers (often called “Neoclouds” or AI hyperscalers), specifically evaluating their ability to support large-scale AI workloads like model training and inference. SemiAnalysis also does InferenceMAX where it rates different GPUs versus each other on various metrics. I would say this helps AI developers, enterprises, and buyers compare GPU cloud platforms objectively. Unlike simple price-per-GPU comparisons, it assesses quality across technical, operational, and usability dimensions to identify providers capable of handling demanding, production-grade AI clusters reliably.</p><p>This year they have rated 84 neo-clouds in detail which 26 more than the last 1.0 version. The report claims they covered 95% of rental GPU market by compute volume. The GPUaaS providers were scored in these 10 catergories:</p><ul><li><strong>Security</strong>: Enterprise-grade security and compliance.</li><li><strong>Lifecycle</strong>: Deployment and support experience.</li><li><strong>Orchestration</strong>: SLURM and Kubernetes support.</li><li><strong>Storage</strong>: Performance and scalability.</li><li><strong>Networking</strong>: Architecture and interconnect quality.</li><li><strong>Reliability</strong>: Uptime and service guarantees.</li><li><strong>Monitoring</strong>: Observability and management tools.</li><li><strong>Pricing</strong>: Total cost of ownership analysis.</li><li><strong>Partnerships</strong>: Ecosystem and integration support.</li><li><strong>Availability</strong>: Resource access and capacity planning.</li></ul><p>There are rating tiers as well, ranging from Platinum (highest, only CoreWeave achieved this in 2.0), Gold, Silver, Bronze, etc. SemiAnalysis arrives at these results after hands-on testing, customer surveys (140+), provider submissions, and strict requirements. Criteria were shared with providers in advance for fairness (as per SemiAnalysis). Below is the final standings for the Neo-clouds as per SemiAnalaysis as of 6 Nov 2025.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*YTr3h3m54em4q8Ag.png" /><figcaption>Courtesy: SemiAnalysis | ClusterMAX 2.0 Results</figcaption></figure><p>Since CoreWeave is the only provider to achieve <strong>Platinum </strong>status in ClusterMAX 2.0, just as it was the only Platinum recipient in version 1.0. According to SemiAnalysis, no other provider came particularly close. To reach Platinum, a provider essentially has to score 90+ (out of 100) in nearly every category with almost no weak points. CoreWeave’s key strengths that secured and retained Platinum as per report:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/c5d36eb76df8e7495b2b4411e83b3e19/href">https://medium.com/media/c5d36eb76df8e7495b2b4411e83b3e19/href</a></iframe><p>SemiAnalysis explicitly stated that CoreWeave is the only provider that “checks almost every single box for running the absolute largest and most demanding training and inference workloads reliably in production.” Below is the technical stack from hardware and software perspective which CoreWeave seems to use:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*QkwYKcS9xcCvjLFSYJlzhQ.png" /></figure><p>There are few more changes I am seeing personally from a GPUaaS provider perspective, there is a shift in the market, where these neo-clouds are now even providing Inference as a service, rather than just renting GPU in form of K8s pod, or VMs, charging end users on basis of number of tokens, much like what hyper-scalers do. The only difference being neo-clouds have better setup than the hyper-scalers, which for sure is changng fast as I write this.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*lOcUdJG1uY5_LmXWXJYTbQ.png" /><figcaption>Various GPUaaS Consumption Models</figcaption></figure><p>When I am meeting GPUaaS providers, I hear them telling me, that all the three consumption models are in demand by end users, however there is a larger demand of token based pricing, specifically from end users who are not MLOps savy. This makes sense since a lot of end users want to leverage LLMs for Enterprise operations, however do not have the right skillsets for the same, this Inference / Tokens as a service, removes that barrier.</p><p>In conclusion, the ClusterMAX 2.0 report leaves no room for ambiguity: when the goal is to train or serve the largest, most demanding frontier models with maximum reliability and performance, only one GPU cloud provider currently sits alone at the top. The report also mentioned in detail the technical capabilities neo-clouds need to add to stay or get on top. For startups and enterprises alike, this matters more than ever: as models scale to trillions of parameters and training runs stretch into months, the difference between a “good enough” cluster and a truly production-grade one can mean the difference between breaking new ground and burning millions on failed jobs. If you’re spending serious money on GPU cloud, ClusterMAX 2.0 is now the closest thing the industry has to a definitive buyer’s guide. The bar has been set; the rest of the market is racing to catch up.</p><p>If you like my work, feel free to get me a <a href="https://buymeacoffee.com/livelyshubham">cappuccino</a>!</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*isrO9H4TzHusEeCb6OjvSA.png" /></figure><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=dd3df07d0d30" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Small Language Models and Agentic AI: A Match Made in Heaven]]></title>
            <link>https://medium.com/@shubhammishra892/small-language-models-and-agentic-ai-a-match-made-in-heaven-672a6514a3dd?source=rss-34d7ac76b1ae------2</link>
            <guid isPermaLink="false">https://medium.com/p/672a6514a3dd</guid>
            <category><![CDATA[agentic-applications]]></category>
            <category><![CDATA[llm-vs-slm]]></category>
            <category><![CDATA[llm]]></category>
            <category><![CDATA[genai]]></category>
            <category><![CDATA[agentic-ai]]></category>
            <dc:creator><![CDATA[Shubham Mishra]]></dc:creator>
            <pubDate>Sun, 14 Sep 2025 17:20:37 GMT</pubDate>
            <atom:updated>2025-09-17T12:37:38.396Z</atom:updated>
            <content:encoded><![CDATA[<h4>SLMs for Agentic AI</h4><p>Talking to a customer earlier last week in Asia, I learned quite a bit about the way enterprises are leveraging Small Language Models or SLMs (more on this later in the blog). It seems like SLMs and Agentic AI are meant for each other. This is even more accentuated by increasing size of GPUs and enterprise’s hunger for them. SLM is any Language Model which is less than 10B parameters in size — as per <a href="https://arxiv.org/pdf/2506.02153">https://arxiv.org/pdf/2506.02153</a>. These can be any models, text generation, Vision Models, etc.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*VyRyfD0siYq51WrU.png" /></figure><p>GPUs and AI infrastructure around them is NOT cheap, more expensive are the MLOps and Data Science folks who can actually materialize the value of this solution. Models such as 671Bn parameter DeepSeek would need minimum 1.4 TB of GPU vRAM over NVLINK (NVIDIA) or on Infinity Fabric (AMD) to make the model ‘sit’ in the GPU vRAM to make the inference / fine tuning possible, we are not even discussing the vRAM required for KV cache or dataset loading for fine tuning. So I totally get that LLMs are not the ultimate panacea for Gen AI applications, let alone for Agentic AI applications, where there is possibility of multiple models being used simultaneously. The infrastructure requirements and their management can scale up quite fast.</p><p>LLMs are also not ‘VERY’ good at specific tasks, they are more generalized and would really bite dust unless fine-tuned on enterprise specific data, which again brings up to the challenges which I mentioned earlier. SLMs, may also need fine-tuning, however due to the nature of smaller number of parameters, fine-tuning them is much faster / easier compared to a LLM. Let us look at some reasons why Small Language Models may be the way to go for Agentic AI or generic AI application as well.</p><ul><li>The scaling curve between model size and capabilities is becoming increasingly steeper, implying that the capabilities of newer small language models are much closer to those of previous large language models. Indeed, recent advances show that well-designed small language models can meet or exceed the task performance previously attributed only to much larger models. A lot of comparitive work has been done for LLMs and SLMs, however for Agentic deployment <strong>aptitude for commonsense reasoning</strong> (an indicator of basic understanding), <strong>tool calling and code generation</strong> (both indicators of the ability to correctly communicate across the model → tool/code interface, and instruction following is needed, which SLMs can do very well now.</li><li>SLMs are more <strong>ECONOMICAL </strong>for Agentic AI, let me just start by stating, that <strong>serving a 7bn SLM is 10–30× cheaper (in latency, energy consumption, and FLOPs) than a 70–175bn LLM, enabling real-time agentic responses at scale, have a good read </strong><a href="https://arxiv.org/pdf/2501.05465"><strong>here</strong></a>. A LLAMA 405B FP16 model would need around 810 GB of GPU vRAM, means ideally you will not be able to fit this model in a Eight H100 GPU system! You would need 2 servers, each with Eight H100 GPUs, connected with dedicated 400 GbE network. I did not add, the specific K8s tooling you would need to manage the workloads and fine-tuning across the nodes. This would look something like this below. We are not even discussing the power, cooling required for something like this (each GPU server would need min 11 kW), and also the management servers for this.</li></ul><figure><img alt="Two Servers — each having Eight H100 GPUs connected via 400G network, dedicated for GPU communication." src="https://cdn-images-1.medium.com/proxy/1*PjT1mITbsVY-HzbcrcBjYQ.png" /><figcaption>GPU Networking</figcaption></figure><ul><li>SLMs require less or no parallelization across GPUs and nodes, the maintenance and operation of the serving infrastructure comes at a lower expense as well, this means that managing inferencing on SLMs is much cheaper, softwares like Dynamo and vLLM can make this even more easy. <strong>Fine-tuning — </strong>Parameter-efficient (e.g., LoRA and DoRA) and full parameter finetuning for SLMs require <strong>only a few GPU-hours</strong>, allowing behaviors to be added, fixed, or specialized overnight rather than weeks. <strong>Edge deployment — </strong>Advances in on-device inference systems demonstrate local execution of SLMs on consumer-grade GPUs, showcasing real-time, offline agentic inference with lower latency and stronger data control. <strong>Parameter utilization — </strong>LLMs are the new monoliths involving a large amount of parameters representing swathes of compressed information in the production of their outputs. However, much of the signals passing through these systems is sparse, engaging only a fraction of their parameters for any single input. This behavior is not seen in SLMs which means they may be fundamentally more efficient by the virtue of having a smaller proportion of their parameters contribute to the inference cost without a tangible effect on the output.</li><li>Due to small size and the associated reduction in pre-training and fine-tuning costs, SLMs are inherently more flexible than their large counterparts when appearing in agentic systems. As such, it becomes much more affordable and practical to train, adapt, and deploy multiple specialized expert models for different agentic routines. This efficiency enables rapid iteration and adaptation, making it feasible to address evolving user needs, including supporting new behaviors, meeting new output formatting requirements, and complying with changing local regulation in selected markets. You could train something like Whisper AI over a single server with Eight MI300X over night for a specific language and its tonalities.</li><li>A typical AI agent has frequent interactions with code, be it through LM tool calling or by returning output that is to be parsed by a piece of agentic code that makes the LM call. It is essential for the success of these interactions that the generated tool call and the generated output conform to <strong>strict formatting requirements</strong> imposed on it by the order, typing, and nature of the tool’s parameters, and the expectation of the code invoking the LM, respectively. In such cases, it becomes unnecessary for the model to handle multiple different formats (e.g. JSON/XML/Python for tool calls and XML/YAML/Markdown/Latex for output), as only one would be chosen for consistency across the agentic application. It is also undesirable for the model to make the occasional hallucinatory mistake and respond in a format different from that being expected by the “code parts” of the agentic system. It is because of this that the SLM trained with a single formatting decision enforced during its post-training or encouraged through additional fine-tuning at a low cost is preferable over a general-purpose LLM in the context of AI agents.</li></ul><p>I am sure there are more reasons for the use of SLMs for Agentic AI and will be applicable as well as we move forward in the AI application development. For one of my customer, who is trying to automate the call centre operations. Now this customer is in Asia and in a geography where each tonality of a word means ‘many’ different contexts. To add to complexity, customer wants the call to be transcripted by Whisper AI to text, a LLM like LLAMA to gather context and then respond using relevant data from a vector DB to customer using a model like VITS (Text to Voice). It looks something like this:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/881/1*tkDMpiO_wEBgofyEz9ob3g.jpeg" /></figure><p>As you can see all of this can fit in a single server with Eight MI300X GPUs from AMD or even Eight H100 / H200 from NVIDIA, leaving enough room for KV cache, hosting Vector DB as much as possible on GPU, or using GPUs for search and indexing of data related to Vector DBs. If not, then host multiple instances of this above setup to increase processing of multiple simultaneous requests from users without increasing TTFT or even E2E latency. Customer has plans to add ‘loggers’ after WhisperAI, LLAMA and VITS to understand the behaviour and response type of these models so customer can fine tune these as per the responses these are getting from each other.</p><p>The very nature of agentic applications enables them to eventually switch from using LLM generalists to using SLM specialists at many of their interfaces. From what it seems, LLM to SLM conversion will be inevitable as more and more Agentic AI applications come to Life!</p><p>If you like my work, feel free to get a <a href="https://buymeacoffee.com/livelyshubham">cappuccino</a>!</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*isrO9H4TzHusEeCb6OjvSA.png" /><figcaption><a href="https://buymeacoffee.com/livelyshubham">https://buymeacoffee.com/livelyshubham</a></figcaption></figure><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=672a6514a3dd" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Why do you need Infiniband for Large Scale GPU Clusters…]]></title>
            <link>https://medium.com/@shubhammishra892/why-would-you-need-infiniband-for-large-scale-gpu-clusters-07929ede7e59?source=rss-34d7ac76b1ae------2</link>
            <guid isPermaLink="false">https://medium.com/p/07929ede7e59</guid>
            <category><![CDATA[gpu]]></category>
            <category><![CDATA[ai-architecture]]></category>
            <category><![CDATA[ai]]></category>
            <category><![CDATA[amd]]></category>
            <category><![CDATA[nvidia]]></category>
            <dc:creator><![CDATA[Shubham Mishra]]></dc:creator>
            <pubDate>Sun, 20 Jul 2025 09:35:08 GMT</pubDate>
            <atom:updated>2025-10-23T04:49:27.483Z</atom:updated>
            <content:encoded><![CDATA[<p>Last week I met a customer who wanted to use Llama 405B parameter model at FP8 with NVIDIA GPUs and wanted to know whether they should go for Infiniband or ethernet for their 32 GPU cluster across four nodes. Now there has been a lot written and experimented on the debate of infiniband vs Etherent for GPU network, and if you ask different architects you may get different answers, below is just what I think should be done. Once again your setup or your customer’s setup is dependent on number of users, AI or HPC use case, concurrent users, workload type (training, Fine Tuning or Inferencing) etc. etc.</p><p>In large-scale GPU setups, such as those used for AI training, high-performance computing (HPC), or supercomputing clusters, GPU-to-GPU communication (also called East-West Communication) is essential for tasks like model parallelism, data parallelism, and collective operations (e.g., all-reduce for gradient synchronization). Below is a reference figure of a GPU to GPU communications for a 256 GPU cluster. This requires 32 servers each with eight GPUs and Eight Leaf switches (show as L1–L8) and four spine switches (S1, S2, S3 and S4). All of these are 400 Gbps connections. NVIDIA documentation talks in great details for a larger cluster which you can read <a href="https://docs.nvidia.com/dgx-superpod/reference-architecture/scalable-infrastructure-h200/latest/_downloads/bbd08041e98eb913619944ead1f92373/RA-11336-001-DSPH200-ReferenceArch.pdf">here</a>.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/508/1*4G0L1BUgOCL92t4NupFFOw.png" /></figure><p>Large AI clusters often involve thousands or tens of thousands of GPUs working in concert, requiring massive data transfers, low-latency synchronization, and minimal overhead to maintain high utilization and efficient job completion times. Without optimized interconnects, bottlenecks can lead to <strong>underutilized</strong> GPUs, <strong>longer </strong>training times, and <strong>increased</strong> costs. InfiniBand is a specialized networking technology designed specifically for high-throughput, low-latency environments like HPC and AI clusters. NVIDIA, which acquired InfiniBand pioneer Mellanox in 2019, recommends it for large-scale GPU setups because it addresses the unique demands of these workloads more effectively than alternatives. <strong>AMD </strong>on the other hand primarily recommends <strong>Ethernet-based networking</strong> solutions, such as RDMA over Converged Ethernet (RoCE), for large-scale AI clusters using their GPUs. This approach is positioned as delivering performance comparable to InfiniBand while offering lower costs and leveraging standard Ethernet architecture for better scalability and <a href="https://instinct.docs.amd.com/projects/gpu-cluster-networking/en/develop/how-to/roce-network-config.html">interoperability</a>. AMD’s official strategy emphasizes commitment to Ethernet for connecting GPU nodes, pods, and clusters, with no explicit recommendation for InfiniBand in their AI networking <a href="https://www.amd.com/content/dam/amd/en/documents/pensando-technical-docs/article/amd-ai-networking-direction-and-strategy.pdf">documentation</a>. Instead, they highlight advancements in Ethernet speeds (e.g., 400 Gb/s and 800 Gb/s) and technologies like the Ultra Ethernet Consortium, of which AMD is a founding member, to support high-bandwidth, low-latency requirements in AI workloads. While InfiniBand can technically be used with AMD GPUs (e.g., Instinct MI series) for AI clusters, there are considerations around compatibility, performance optimization, and software ecosystem support, as it’s not the focus of AMD’s optimizations like RCCL (ROCm Collective Communication Library). AMD’s push for Ethernet aligns with industry trends toward open standards to avoid proprietary dependencies, contrasting with competitors like NVIDIA who heavily rely on InfiniBand for their clusters.</p><h3>Key Advantages of InfiniBand for GPU-to-GPU Communication</h3><ul><li><strong>Ultra-Low Latency</strong>: GPU communications in large clusters (upwards of 128 GPUs) involve frequent, small messages (e.g., for synchronization or parameter updates). InfiniBand delivers end-to-end latencies in the <strong>sub-microsecond</strong> range, far lower than standard Ethernet. This is critical for reducing idle time during collective operations, where even minor delays compound across thousands of GPUs. For instance, in AI training, low latency ensures faster iteration cycles and better scaling efficiency.</li><li><strong>High Bandwidth and Throughput</strong>: Modern InfiniBand (e.g., NDR at 400Gbps or higher) provides massive bandwidth for transferring large tensors or datasets between GPUs. In setups with 100,000+ GPUs, this enables sustained high-speed data movement without bottlenecks, supporting workloads like large language model (LLM) training that involve terabytes of data shuffling.</li><li><strong>Native Lossless Transmission</strong>: InfiniBand is inherently lossless, using credit-based flow control to prevent packet drops even under heavy congestion. This reliability is vital in GPU clusters, where lost packets would require retransmissions, stalling computations and reducing overall system efficiency.</li><li><strong>Efficient RDMA (Remote Direct Memory Access)</strong>: InfiniBand supports native RDMA, allowing direct GPU-to-GPU data transfers without involving the host CPU or OS kernel. This “zero-copy” mechanism minimizes overhead, CPU utilization, and latency — key for GPU-direct technologies like NVIDIA’s GPUDirect. In contrast, non-RDMA setups require multiple memory copies, which can degrade performance by 2x or more in communication-intensive tasks.</li><li><strong>In-Band Network Computing (e.g., SHARP)</strong>: InfiniBand supports NVIDIA’s Scalable Hierarchical Aggregation and Reduction Protocol (SHARP), which offloads collective operations (like all-reduce or all-gather) to the network switches themselves. This accelerates ML training by performing data reductions in-network, reducing GPU load and improving scalability in large clusters. SHARP can boost job completion times by up to 30% in large GPU setups.</li><li><strong>Superior Scalability and Reliability at Large Scales</strong>: InfiniBand’s switched fabric architecture handles massive interconnections (e.g., 32,000+ GPUs) with minimal performance degradation, better congestion management, and higher message rates. It’s optimized for rail-optimized topologies common in NVIDIA DGX or HGX systems, ensuring consistent performance even in failure-prone environments. Features like adaptive routing and multi-rail support (using multiple host channel adapters per node) enhance fault tolerance and throughput.</li></ul><h3>Why InfiniBand Over Ethernet, Even Though Ethernet is Supported?</h3><p>Ethernet is indeed supported for GPU clusters via protocols like RoCEv2 (RDMA over Converged Ethernet), which NVIDIA offers through products like Spectrum-X and is supported networking option for AMD based AI clusters. It’s more ubiquitous, cost-effective for smaller setups, and leverages existing infrastructure, making it appealing for cloud providers or mixed workloads. However, NVIDIA recommends InfiniBand for large-scale, performance-critical setups due to Ethernet’s inherent limitations:</p><ul><li><strong>Higher Latency and Overhead</strong>: Ethernet typically has higher baseline latency (due to its shared-medium design and additional protocol layers) and requires tuning for RDMA (e.g., Priority Flow Control and Explicit Congestion Notification) to mimic lossless behavior. This can lead to variability and slower job times in communication-heavy workloads.</li><li><strong>Congestion and Lossiness</strong>: Standard Ethernet is lossy and stochastic, prone to packet drops under load, which disrupts GPU synchronization. While RoCEv2 mitigates this, it doesn’t match InfiniBand’s native reliability, especially at scales where congestion amplifies.</li><li><strong>Lack of Native Advanced Features</strong>: Ethernet doesn’t natively support SHARP or equivalent in-band computing, limiting its efficiency for AI collectives. It also scales less predictably in massive clusters, often requiring more switches and complex configurations.</li><li><strong>Performance Trade-offs in Real-World Tests</strong>: In benchmarks, InfiniBand often outperforms Ethernet by 10–30% in job completion for large AI tasks, <strong>though Ethernet can achieve parity in optimized, smaller clusters or with recent advancements (e.g., Ultra Ethernet Consortium efforts)</strong>. Read more <a href="https://techblog.comsoc.org/2024/08/28/will-ai-clusters-be-interconnected-via-infiniband-or-ethernet-nvidia-doesnt-care-but-broadcom-sure-does/">here</a>.</li></ul><p>In summary, while Ethernet suffices for mid-scale or cost-sensitive setups I would say up to 64 GPUs in a cluster for NVIDIA we can use Ethernet for GPU network. For AMD cluster since they are creating a more open eco-system, they do support ethernet for GPU network for both small and large setup. InfiniBand’s specialized design maximizes GPU utilization in massive clusters, justifying NVIDIA’s recommendation for frontier AI/HPC applications. For the largest systems (e.g., 100,000+ GPUs), the performance gains translate to faster time-to-insight and lower operational costs despite higher upfront expenses.<br>For most use cases, where the number of GPUs range from 8–64 (anything for inferencing and small size Fine Tuning) <strong>ethernet </strong>would be a more cost effective and manageable solution. IB comes with its own baggage which we can unpack some other day. Needless to say my customer is also going ahead with ethernet for GPU switching which both NVIDIA and AMD support. Remember, just cause something is best at mega-scale does not necessarily mean, it would be best at small scale as well, as someone said, One Size Does Not Fit All!</p><p>If you learned something and liked it, consider buying me a coffee!</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/700/1*NqY1jOdD7Avjm1I6J_151g.png" /></figure><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=07929ede7e59" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[AMD’s Advancing AI — 2025 Event]]></title>
            <link>https://medium.com/@shubhammishra892/amds-advancing-ai-2025-event-a88825960bd8?source=rss-34d7ac76b1ae------2</link>
            <guid isPermaLink="false">https://medium.com/p/a88825960bd8</guid>
            <category><![CDATA[genai]]></category>
            <category><![CDATA[amd]]></category>
            <category><![CDATA[gpu]]></category>
            <category><![CDATA[llm]]></category>
            <category><![CDATA[mlops]]></category>
            <dc:creator><![CDATA[Shubham Mishra]]></dc:creator>
            <pubDate>Fri, 20 Jun 2025 18:36:15 GMT</pubDate>
            <atom:updated>2025-06-20T18:36:15.480Z</atom:updated>
            <content:encoded><![CDATA[<h3>AMD’s Advancing AI — 2025 Event</h3><p>Yes, I know I am a week and a day late for this post, but believe me when I say, I do not get any time to write my thoughts here cause of my present work situation, not complaining but everyone wants an AI SOLUTION, I believe it is a good problem to have. Anyway, on 12 June 2025, AMD announced a bunch of new stuff for their GPU line up for enterprises and what lies in future. Needless to say, NVIDIA needs to pull up their HBM (High Bandwidth Memory) game, cause these new AMD GPUs look very appetizing. This is a summay of keynote presentation and partner discussions led by Dr. Lisa Su, Chair and CEO of AMD, at the Advancing AI 2025 event. You can watch the whole video <a href="https://youtu.be/zCOqJx3Yst4">here</a> or just read below. The focus was on AMD’s leadership and strategy in AI hardware, software, and ecosystems as they advance into a future powered by agentic AI — a new AI paradigm involving autonomous agents performing complex reasoning and multistep tasks.</p><p>AMD announced its latest AI hardware offerings, including the <a href="https://www.amd.com/en/products/accelerators/instinct/mi350/mi350x.html">MI350</a> series and previews the forthcoming <a href="https://www.amd.com/en/newsroom/press-releases/2025-6-12-amd-unveils-vision-for-an-open-ai-ecosystem-detai.html">MI400</a>, which together deliver unprecedented leaps in AI compute power, memory capacity, and efficiency, designed to meet the demands of large-scale training, inference, and distributed workloads. Alongside these hardware advances, AMD details its robust <a href="https://rocm.docs.amd.com/en/latest/">ROCm</a> software platform, tailored to provide a developer-first experience supporting key AI frameworks and models with accelerated performance, broad accessibility, and open source collaboration. The event started with Dr. Lisa stating and I quote, “With all the new use cases and reasoning models we now expect that <strong>inference will grow by more than 80% a year for the next few years</strong>, becoming the largest driver of AI compute.” There she announced the complete AMD line up for AI hardware solutions.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*DMRgPj6Zf6KNuhklU4t7eg.png" /></figure><p>This event was decked with prominent partners across the AI and cloud ecosystem including <strong>xAI, Meta, Oracle Cloud Infrastructure (OCI), Microsoft, Cohere, Red Hat, and OpenAI</strong>. Each of them shared insights on how AMD’s hardware and software enable them to scale state-of-the-art AI workloads efficiently, securely, and cost-effectively. AMD’s roadmap for AI also addresses emerging trends such as agentic AI with high demands across the stack: GPUs for inference and reasoning, CPUs for orchestration, and advanced interconnects providing secure, low-latency data flow. The introduction of open standards like the Ultra Accelerator Link (UALink) and Ultra Ethernet Consortium (UEC) enable next-generation scale-up and scale-out AI platforms, fostering industry-wide collaboration and avoiding vendor lock-in. Current AMD GPU portfolio looks like this:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Dh1TkfS8fcBZjd0n5Tkkwg.png" /></figure><p>AMD released their latest GPU series in form of MI350X and MI355X (which is same GPU as MI350X but can bear more temperature and power). Its specs are mentioned below. With an enterprise server having eight MI350X we can have 2.3 TB of HBM, we can run almost any model in a single server. This is available in air and liquid cooled versions. MI355X is 35 times the throughput compared to MI300X! Dr. Lisa also mentioned that MI350X can deliver upto <strong>40% more tokens/dollar vs B200, which makes it a go to choice for inferencing.</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*ZmpvcsjsFVx5wDNKqy8GsQ.png" /></figure><p>They also announced, racks built completely on AMD technology, these have 5th Gen EPYC CPUs, MI350X GPUs and can have 64 GPUs in a single rack, all of them air cooled, or upto 128 GPUs in a single rack liquid cooled, these all are OCP compliant designs:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/695/1*A-4yBXnjR3ho1kNH7HVkDg.png" /></figure><p>AMD also annouced ROCm 7 release to be made available soon. This release brings capabilities like distributed inference, large scale training and faster deployment of models effortlessly. Some of the new features of ROCm 7 for inferencing are below:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*YnAbYEF03EkTqSOF51faXw.png" /></figure><p>AMD also stressed on the open-source inferencing (vLLM, SGLANG) being much faster than proprietary solutions. One of the key things to optimize and ensure the model runs in its bet shape is to make sure that we are taking advantage of all the memory on the GPU chip, MI350X with ROCm 7 will allow to run larger models, with their weights and KV cache in the GPU to allow for faster inferencing and even training / fine tuning. AMD also mentioned that ROCm 7 will support many new training frameworks and will add new features to exisiting ones.</p><h3><strong>New Era of AI</strong></h3><p>AMD also shared how they see this new era of AI unfolding with Agentic AI in use and how it can affect business processes. The key in Agentic AI is to connetc the power of LLM models to the business’s data, tools and people. Agentic flow will include many models, including specialized models each performing own rool to complete tasks, like below</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*5_ZDnBdnNNBBhNi-s2WTpw.png" /></figure><p>These agents will access many resources, and will not just run on isolated GPUs, and will also run on CPUs running these applications and processing the data in and out of GPUs, using secured network infrastructure.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*WigqR4heQkOxRoKMmzwPog.png" /></figure><p>Since this agentic behaviour will touch every aspect of the datacentre, AMD is prepared with solutions to resolve the challenges of Agentic Applications, powered by Trillion Parameter models and petabytes of data in future. Some of these solutions and shown below:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*rJQKhm_vY71YVVcoyBqd7g.png" /></figure><p>Since AMD uses Infinity Fabric to connect CPUs and GPUs together, this sets the machine for exemplary performance for Agentic AI use cases, which leverage both CPUs and GPUs. AMD also mentioned their Ultra Accelerator Link — UA Link technology and how it can help scale to 1000+ GPU cluster, since this is completely open in architecture, this can work with any CPU and GPU.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*kAXJ-j1G2aI_yYg7F2p_Xw.png" /></figure><h3>Helios AI Rack</h3><p>Helios AI Rack is we architected as a unified system. That’s combining AMD CPUs, AMD GPUs, AMD Pensando NICs, and ROCm software all together in one platform. And it’s purpose built for the most demanding AI workloads, from training, to the largest, models, to scaling inference across thousands of nodes. This is delivered in an open OCP compliant rack that supports both Ultra Ethernet and UALink. It connects <strong>up to 72 GPUs with 260 terabytes per second of scale-up bandwidth</strong>. It enables <strong>2.9 exaflops of FP4 performance</strong>. Compared to the competition, we support 50% more HBM4 memory, memory bandwidth, and scale-out bandwidth, and what this translates in is faster training, higher inference throughput, and the ability to really handle massive models. The CPU VENICE is built on 2 nm process and has up to<strong> 256 Zen 6 cores</strong>, it is optimized to run with MI400X GPU series which is designed to run trillion parameter models. We deliver up to 40 petaflops of FP4 performance.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*rdPvxnD17tH91Nh-yxprhQ.png" /></figure><p><strong>MI400X has 432 gigabytes of HBM4 and supports 300 gigabytes per second of scale-out bandwidth to connect across racks and clusters.</strong> Below are some number for MI400X to amaze you!</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*rR0L__ElR4VmCGV_M0g2fw.png" /></figure><p>AMD also introduced Volcano, next generation scale-out AI NIC. Volcano is fully UEC 1.0 compliant. It supports PCIe and UAL interfaces to connect directly both GP CPUs and GPUs, and it delivers 800 gigabits per second of line rate throughput to scale for the largest systems. This all shrouded in ROCm software is ready to deliver millions of tokens at a fraction of dollar cost compared to competition.</p><p>There were so many tech icons who were at this session that their collectivce knowledge and presence shows, how important AMD is to them, <a href="https://www.amd.com/en/corporate/events/advancing-ai.html#featured-speakers">here</a> is the list of all who were present. Sam Altman was here and he tried his best but he did in the end talk about <a href="https://www.mckinsey.com/featured-insights/mckinsey-explainers/what-is-artificial-general-intelligence-agi"><strong>AGI</strong></a>. You can read more about it <a href="https://www.amd.com/en/corporate/events/advancing-ai.html">here</a>.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=a88825960bd8" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[KV Cache for LLMs — Zero to Hero]]></title>
            <description><![CDATA[<div class="medium-feed-item"><p class="medium-feed-image"><a href="https://medium.com/@shubhammishra892/kv-cache-for-llms-zero-to-hero-eeaa852b0115?source=rss-34d7ac76b1ae------2"><img src="https://cdn-images-1.medium.com/max/2523/1*BnyR9FQCBEUkjJz5rJpdbg.png" width="2523"></a></p><p class="medium-feed-snippet">ATTENTION is all you need, but KV cache is all what ATTENTION needs.</p><p class="medium-feed-link"><a href="https://medium.com/@shubhammishra892/kv-cache-for-llms-zero-to-hero-eeaa852b0115?source=rss-34d7ac76b1ae------2">Continue reading on Medium »</a></p></div>]]></description>
            <link>https://medium.com/@shubhammishra892/kv-cache-for-llms-zero-to-hero-eeaa852b0115?source=rss-34d7ac76b1ae------2</link>
            <guid isPermaLink="false">https://medium.com/p/eeaa852b0115</guid>
            <category><![CDATA[genai]]></category>
            <category><![CDATA[llm]]></category>
            <category><![CDATA[kv-cache]]></category>
            <category><![CDATA[mlops]]></category>
            <category><![CDATA[attention]]></category>
            <dc:creator><![CDATA[Shubham Mishra]]></dc:creator>
            <pubDate>Fri, 02 May 2025 17:57:53 GMT</pubDate>
            <atom:updated>2025-05-03T13:37:33.935Z</atom:updated>
        </item>
        <item>
            <title><![CDATA[ROCm 6.4, released on April 11, 2025, represents AMD’s latest advancement in its open-source GPU…]]></title>
            <link>https://medium.com/@shubhammishra892/rocm-6-4-released-on-april-11-2025-represents-amds-latest-advancement-in-its-open-source-gpu-c8eaddfe4811?source=rss-34d7ac76b1ae------2</link>
            <guid isPermaLink="false">https://medium.com/p/c8eaddfe4811</guid>
            <category><![CDATA[rocm]]></category>
            <category><![CDATA[gpu]]></category>
            <category><![CDATA[gpu-computing]]></category>
            <category><![CDATA[amd]]></category>
            <category><![CDATA[llm]]></category>
            <dc:creator><![CDATA[Shubham Mishra]]></dc:creator>
            <pubDate>Sat, 19 Apr 2025 10:15:21 GMT</pubDate>
            <atom:updated>2025-04-19T10:15:21.054Z</atom:updated>
            <content:encoded><![CDATA[<p>ROCm 6.4, released on April 11, 2025, represents AMD’s latest advancement in its open-source GPU computing platform, bringing significant improvements to AI development, HPC workloads, and enterprise deployment. This update introduces new features for performance optimization, expanded framework support, and improved deployment tools, while continuing to address compatibility across AMD’s GPU portfolio.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/466/1*SaizGrsBlkJ-Qmqg6mez6g.png" /></figure><p><strong>Core Architecture Improvements and Software Modularity</strong></p><p>ROCm 6.4 establishes a more modular software foundation, creating clearer separation between user space and driver space components. This architectural enhancement improves compatibility between ROCm’s user-space libraries and the AMDKFD kernel driver, allowing for operation across a wider range of Linux kernels. The release introduces an important shift in the versioning scheme for the Instinct driver, preparing for future independent versioning from ROCm releases. This separation provides several operational benefits:</p><ul><li>Independent upgrades of Instinct drivers and ROCm user space components</li><li>Targeted bug fixes released separately between driver and user space components.</li><li>Expanded internal testing covering more user and kernel version combinations.</li></ul><p><strong>Memory Configuration Advancements</strong></p><p>A notable feature in ROCm 6.4 is support for CPX (<strong>Core Partitioned X-celerator</strong>) mode with NPS4 (<strong>NUMA Per Socket 4</strong>) memory mode on AMD Instinct MI300X systems. This configuration:</p><ul><li>Can be applied to SR-IOV host or bare metal environments</li><li>Enables superior performance with smaller language models (13B parameters or less) that fit within one CPX GPU</li><li>Provides optimal memory bandwidth scenarios on MI Instinct accelerators.</li></ul><p><strong>AI and Deep Learning Framework Enhancements PyTorch Optimizations</strong></p><p>ROCm 6.4 introduces significant enhancements for PyTorch, including support for versions <strong>2.5 </strong>and <strong>2.6</strong>. Performance improvements in the PyTorch framework include:</p><ul><li><strong>Optimized Flex Attention</strong>: Delivering competitive performance comparable with CUDA implementations.</li><li><strong>TopK Enhancements</strong>: Achieving 3x faster inference, leading to more responsive LLMs while maintaining output quality.</li><li><strong>SDPA Optimization</strong>: Improving efficiency in scaled dot-product attention, enabling faster and more memory-efficient LLM inference in long-context scenarios.</li></ul><p><strong>The release includes pre-built, optimized Docker containers for AI training and inference workflows, supporting models like Llama 3.1 (8B, 70B), Llama 2 (70B), and FLUX.1-dev </strong><a href="https://hub.docker.com/r/rocm/vllm-dev"><strong>to download click here</strong></a>.</p><p><strong>Megatron-LM Framework Support</strong></p><p>The Megatron-LM framework for ROCm receives significant upgrades with three new fused kernels:</p><ul><li>Fused Attention (QKV).</li><li>Fused Layer Norm.</li><li>Fused ROPE</li></ul><p>These enhancements speed up transformer model training by combining multiple operations into single GPU passes, improving training efficiency for large-scale language models on AMD GPUs.</p><p><strong>Mathematical Libraries and Compute Performance</strong></p><p>ROCm 6.4 delivers significant improvements to mathematical libraries essential for AI and HPC workloads:</p><p><strong>WMMA (Wave Matrix Multiply-Accumulate) Enhancements</strong></p><ul><li>Interleaved GEMM optimization: Improving data locality and ensuring better memory access patterns.</li><li>Stream-K algorithm: Providing more consistent performance across various GEMM problem sizes.</li><li>Infrastructure to support interleaved wave-tiles for better GEMM performance.</li></ul><p><strong>Sparse Computation Improvements</strong></p><p>The rocSPARSE library sees key enhancements:</p><ul><li>Optimized CSR Matrix-Vector Multiplication: Accelerating sparse workloads by reducing memory overhead.</li><li>Expanded SpMM support: Improving efficiency for applications processing sparse matrices.</li></ul><p>These improvements reduce computation time and enhance scalability for workloads requiring sparse data handling.</p><p><strong>GPU Communication Optimization</strong></p><ul><li>rocSHMEM Intra-Node GPU Communication: Facilitating GPU-initiated data transfers for single-node jobs without network dependencies.</li><li>Topology Optimization in RCCL: Eliminating the need for Tier-2 spine switches and enabling efficient rail topology communication with only Tier-1 switches.</li></ul><p>These improvements optimize both intra-node and inter-node communication, reducing network congestion, lowering latency, and improving scalability for large-scale AI and HPC workloads.</p><p><strong>Kubernetes and Enterprise Deployment</strong></p><p>ROCm 6.4 enables new Kubernetes-based GPU management features:</p><ul><li>Automatic GPU scheduling and driver management: Eliminating manual configuration efforts for seamless workload deployment.</li><li>Support for Red Hat OpenShift (v4.16-v4.17) &amp; Ubuntu 22.04/24.04: Ensuring broad compatibility across cloud and enterprise environments.</li><li>Automatic driver upgrades &amp; workload orchestration: Including node cordon, drain, and reboot automation for smooth rolling updates with minimal downtime.</li><li>Air-gapped and proxy deployments: Allowing GPU clusters to function efficiently in restricted network environments.</li></ul><p><strong>Device Metrics Exporter: Real-Time GPU Health Monitoring</strong></p><p>The release introduces Prometheus-based telemetry for tracking critical GPU metrics:</p><ul><li>ECC errors, power consumption, and GPU/Memory utilization: Providing real-time monitoring for early anomaly detection.</li><li>Custom health thresholds: Configurable via Kubernetes ConfigMaps to automatically flag or remove GPUs before system instability occurs.</li></ul><p>More details can be found at, official ROCm <a href="https://rocm.docs.amd.com/en/latest/about/release-notes.html">link</a> and GitHub releases <a href="https://github.com/ROCm/ROCm/releases">page </a>. Complete lists of all ROCm components: <a href="https://rocm.docs.amd.com/en/latest/about/release-notes.html#rocm-components">https://rocm.docs.amd.com/en/latest/about/release-notes.html#rocm-components</a></p><p>ROCm 6.4 represents a significant step forward in AMD’s GPU computing ecosystem, introducing important performance optimizations, expanded framework support, and improved deployment tools. The architectural shift toward more modular components and better compatibility between user space and kernel drivers is a promising direction potentially enabling more flexible deployment and upgrade paths with AMD GPU technologies.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=c8eaddfe4811" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Local RAG with DeepSeek R1]]></title>
            <description><![CDATA[<div class="medium-feed-item"><p class="medium-feed-image"><a href="https://medium.com/@shubhammishra892/local-rag-with-deepseek-r1-008984143be0?source=rss-34d7ac76b1ae------2"><img src="https://cdn-images-1.medium.com/max/2600/1*ZAJ0FErTtRpHdzfYTJ1sgQ.png" width="2712"></a></p><p class="medium-feed-snippet">In my last write up I did a very simple setup where I ran DeepSeek on my ollama setup and used OpenWebUI to create &#x201C;kind of&#x201D; my own&#x2026;</p><p class="medium-feed-link"><a href="https://medium.com/@shubhammishra892/local-rag-with-deepseek-r1-008984143be0?source=rss-34d7ac76b1ae------2">Continue reading on Medium »</a></p></div>]]></description>
            <link>https://medium.com/@shubhammishra892/local-rag-with-deepseek-r1-008984143be0?source=rss-34d7ac76b1ae------2</link>
            <guid isPermaLink="false">https://medium.com/p/008984143be0</guid>
            <category><![CDATA[large-language-models]]></category>
            <category><![CDATA[llm]]></category>
            <category><![CDATA[ollama]]></category>
            <category><![CDATA[genai]]></category>
            <category><![CDATA[deepseek-r1]]></category>
            <dc:creator><![CDATA[Shubham Mishra]]></dc:creator>
            <pubDate>Sat, 01 Mar 2025 17:11:20 GMT</pubDate>
            <atom:updated>2025-03-01T17:13:32.386Z</atom:updated>
        </item>
        <item>
            <title><![CDATA[Personal ChatGPT using Ollama and OpenWebUI]]></title>
            <link>https://medium.com/@shubhammishra892/personal-chatgpt-using-ollama-and-openwebui-37a033058785?source=rss-34d7ac76b1ae------2</link>
            <guid isPermaLink="false">https://medium.com/p/37a033058785</guid>
            <category><![CDATA[chatgpt]]></category>
            <category><![CDATA[deepseek-r1]]></category>
            <category><![CDATA[llm]]></category>
            <category><![CDATA[ollama]]></category>
            <dc:creator><![CDATA[Shubham Mishra]]></dc:creator>
            <pubDate>Tue, 25 Feb 2025 10:56:46 GMT</pubDate>
            <atom:updated>2025-02-25T10:56:46.381Z</atom:updated>
            <content:encoded><![CDATA[<p>In few simple steps and a GPU powered laptop we can have our own ChatGPT whether connected to internet or not. The process is fairly simple and can takes less than 30 mins to get it rolling.</p><p>We would need below to begin with:</p><ol><li><strong>GPU powered Laptop / Desktop</strong> — the GPU can be NVIDIA, AMD etc. what would really affect the performance in the amount of memory in GPU, the more the merrier. Higher memory capacity, can help you keep larger LLM in the GPU memory leading to better inferencing and less latency. This also means you can use larger LLMs like Llama 70B etc. or even DeepSeek R1 with high quantization. You can check your GPUs memory using either nvidia-smi or rocm-smi for NVIDIA and AMD GPUs respectively. As you can see I have a NVIDIA RTX 5000 ADA with 16GB of internal GPU memory.</li></ol><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*31zJN-FZarK6pQDiZstC_A.png" /></figure><ol><li>We would need <strong>Ollama </strong>— Ollama is in simple words democratization of large language model (LLM) technologies, enabling developers, researchers, and organizations to harness the power of advanced AI systems while maintaining full control over data privacy and computational resources. This framework bridges the gap between proprietary cloud-based AI services and locally hosted solutions, offering a versatile platform for experimentation, development, and deployment of LLM-driven applications. We wouldse this to run LLM of our choice. This is free to download and can be installed on variety of OS platforms. <a href="https://ollama.com/download">Download here.</a> You can also check out various models you can download from <a href="https://ollama.com/models">here</a>. Once you have downloaded Ollama, you can check its version and download the necessary model using:</li></ol><pre>ollama --version  ## to check ollama version.<br>ollama list ## to check the list of all LLMs currently installed in system.<br>ollama pull &lt;model name&gt; ## To download and install model from Ollama on Windows.</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*NYgn8msTXD76MPk6k_dDuw.png" /></figure><p>You can run ollama list again to see that deepseek is listed in the model name. You can then run below command to run your model:</p><pre>ollama run &lt;modelname&gt; ## to run the model<br>ollama run deepseek-r1:14b --verbose ## to give detailed metrics on the model.</pre><p>Below is an example:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*nqxQQ7GnBaNyL7HVDX5wRw.png" /><figcaption>Check the bottom right, where no. of token are mentioned, with LLM output of a Ollama poem.</figcaption></figure><p>2. <strong>LLM </strong>— this will depend on your use case, if you are just looking for a basic reasoning inference, you can download DeepSeek R1 or even a specialized model such as <a href="https://ollama.com/library/qwen2.5-coder">QwenCoder</a> etc. depending on your use case. One more thing to be mindful is the amount of memory your GPU has, and parameters your model has. The more the parameters the model has, more accurate it is, also uses more memory and lesser the parameters, lesser the memory it uses, so depening on your resource availability, you need to choose accordingly. As you can see Deepseek-r1 with 1.5 billion parameters needs 1.1 GB of memory, however 671 billion needs 404 GB.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*koMfBNYEITh5DA7GzDWz5Q.png" /></figure><p>For our case we are using DeepSeek-r1 14 Billion parameter model which is 9 GB in size.</p><p>4. We would need <a href="https://openwebui.com/"><strong>OpenWeb UI</strong></a>, which can be installed for Windows, Mac etc. Using this you can effortlessly integrate OpenAI-compatible APIs for versatile conversations alongside Ollama models. Customize the OpenAI API URL to link with LMStudio, GroqCloud, Mistral, OpenRouter, and more. You should have python installed in your setup. You can use below commands to install and start openwebui from CMD prompt or bash shell.</p><pre>pip install open-webui<br>open-webui serve</pre><p>5. Connect Open WebUI to DeepSeek: Once Open WebUI is running, navigate to the web interface in your browser. Select the deepseek model from the dropdown menu to start interacting with it. You can now use the chat interface to interact with DeepSeek just like you would with ChatGPT. Check below image:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*IsQUYf_PaeeTm7z9N_5XwQ.png" /><figcaption>Check the last line, thats the address of your configured CHAT GPT like application, put it in a supported browser.</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*YFnM9F1VYNzt53NUKwSDYA.png" /></figure><p>This Chat GPT like interface is running on your machine, end to end, you can remove internet and will work still. Have a look at the video below:</p><iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Fwww.youtube.com%2Fembed%2FtaC51RT41Ms%3Ffeature%3Doembed&amp;display_name=YouTube&amp;url=https%3A%2F%2Fwww.youtube.com%2Fwatch%3Fv%3DtaC51RT41Ms&amp;image=https%3A%2F%2Fi.ytimg.com%2Fvi%2FtaC51RT41Ms%2Fhqdefault.jpg&amp;type=text%2Fhtml&amp;schema=youtube" width="640" height="480" frameborder="0" scrolling="no"><a href="https://medium.com/media/3b28d9ee56b48147e7c01ea2c9074903/href">https://medium.com/media/3b28d9ee56b48147e7c01ea2c9074903/href</a></iframe><p>If you like my work, consider buying me a coffee <a href="https://buymeacoffee.com/livelyshubham">here</a>.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=37a033058785" width="1" height="1" alt="">]]></content:encoded>
        </item>
    </channel>
</rss>