<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Vinodbokare on Medium]]></title>
        <description><![CDATA[Stories by Vinodbokare on Medium]]></description>
        <link>https://medium.com/@vinodbokare0588?source=rss-66aeb228608b------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/0*6_WUH-bHIwmrAViK</url>
            <title>Stories by Vinodbokare on Medium</title>
            <link>https://medium.com/@vinodbokare0588?source=rss-66aeb228608b------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Tue, 17 Nov 2026 08:15:29 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@vinodbokare0588/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[How AwsLambda’s 90-Min Timeout Reframes Agentic AI Design]]></title>
            <description><![CDATA[<div class="medium-feed-item"><p class="medium-feed-image"><a href="https://medium.com/@vinodbokare0588/how-awslambdas-90-min-timeout-reframes-agentic-ai-design-155b10eae573?source=rss-66aeb228608b------2"><img src="https://cdn-images-1.medium.com/max/1390/1*wSLAMu5HlsNFi8HhfYH2xA.png" width="1390"></a></p><p class="medium-feed-snippet">If an AI workload might take more than 15 minutes, Lambda probably isn&#x2019;t the right place for it.</p><p class="medium-feed-link"><a href="https://medium.com/@vinodbokare0588/how-awslambdas-90-min-timeout-reframes-agentic-ai-design-155b10eae573?source=rss-66aeb228608b------2">Continue reading on Medium »</a></p></div>]]></description>
            <link>https://medium.com/@vinodbokare0588/how-awslambdas-90-min-timeout-reframes-agentic-ai-design-155b10eae573?source=rss-66aeb228608b------2</link>
            <guid isPermaLink="false">https://medium.com/p/155b10eae573</guid>
            <category><![CDATA[ai-agent]]></category>
            <category><![CDATA[aws-lambda]]></category>
            <category><![CDATA[ai]]></category>
            <category><![CDATA[amazon-bedrock]]></category>
            <dc:creator><![CDATA[Vinodbokare]]></dc:creator>
            <pubDate>Sun, 13 Sep 2026 06:14:59 GMT</pubDate>
            <atom:updated>2026-09-13T06:14:59.220Z</atom:updated>
        </item>
        <item>
            <title><![CDATA[Stop Asking One AI Everything. Build an AI Team Instead]]></title>
            <link>https://medium.com/@vinodbokare0588/stop-asking-one-ai-everything-build-an-ai-team-instead-69434e65ba3c?source=rss-66aeb228608b------2</link>
            <guid isPermaLink="false">https://medium.com/p/69434e65ba3c</guid>
            <category><![CDATA[software-engineering]]></category>
            <category><![CDATA[software-development]]></category>
            <category><![CDATA[claude-code-sub-agents]]></category>
            <category><![CDATA[ai-agent]]></category>
            <category><![CDATA[ai]]></category>
            <dc:creator><![CDATA[Vinodbokare]]></dc:creator>
            <pubDate>Mon, 29 Jun 2026 17:45:49 GMT</pubDate>
            <atom:updated>2026-06-29T17:45:49.448Z</atom:updated>
            <content:encoded><![CDATA[<h4>How Fan Out Subagents are changing software engineering by turning one prompt into a team of AI specialists.</h4><p>For the longest time, I approached AI coding assistants the same way most developers do: I gave them a large prompt and expected one model to analyze everything.</p><p>Sometimes it worked.</p><p>Often it didn’t.</p><p>The AI would miss edge cases, overlook architectural problems, ignore security concerns, or spend thousands of tokens reasoning about one issue while completely forgetting another.</p><p>Then I discovered a different way of thinking.</p><p>Instead of asking <strong>one AI to do everything</strong>, ask <strong>many specialized AIs to do one thing each.</strong></p><p>This concept is often called <strong>“fan out subagents.”</strong></p><h3>What Does “Fan Out Subagents” Mean?</h3><p>Imagine you’re building a production-grade application.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1016/1*ccciQmpGBxV3J-gVzhaDtQ.png" /><figcaption>how fan out subagents</figcaption></figure><p>Would you hire one engineer to be:</p><ul><li>Software Architect</li><li>Backend Engineer</li><li>Security Engineer</li><li>Performance Expert</li><li>QA Engineer</li><li>Technical Writer</li><li>DevOps Engineer</li></ul><p>Of course not.</p><p>Real engineering teams specialize.</p><p>AI can work exactly the same way.</p><p>When you <strong>fan out subagents</strong>, your AI orchestrator creates multiple specialized workers in parallel.</p><p>Instead of one giant reasoning chain, you get multiple focused investigations happening simultaneously.</p><h3>A Real Example</h3><p>Suppose you ask:</p><blockquote><em>“Review my enterprise FastAPI project.”</em></blockquote><p>Instead of producing one generic review, the work can be divided into specialists.</p><h3>Architecture Agent</h3><p>Looks only at:</p><ul><li>System design</li><li>Service boundaries</li><li>Scalability</li><li>Domain modeling</li><li>Event-driven architecture</li></ul><h3>Code Quality Agent</h3><p>Focuses on:</p><ul><li>Python quality</li><li>SOLID principles</li><li>Design patterns</li><li>Maintainability</li><li>Refactoring opportunities</li></ul><h3>Security Agent</h3><p>Searches for:</p><ul><li>SQL Injection</li><li>Authentication flaws</li><li>Authorization gaps</li><li>Secrets management</li><li>Dependency vulnerabilities</li></ul><h3>Performance Agent</h3><p>Examines:</p><ul><li>Database queries</li><li>Async implementation</li><li>Memory usage</li><li>Caching</li><li>API latency</li></ul><h3>QA Agent</h3><p>Generates:</p><ul><li>Unit tests</li><li>Integration tests</li><li>Edge cases</li><li>Failure scenarios</li><li>Missing validation</li></ul><h3>Documentation Agent</h3><p>Reviews:</p><ul><li>README</li><li>API documentation</li><li>Deployment guide</li><li>Missing diagrams</li><li>Developer onboarding</li></ul><p>Finally…</p><p>An orchestrator merges every report into one comprehensive recommendation.</p><p>Instead of one opinion…</p><p>You receive an engineering review from an entire AI team.</p><h3>Why This Produces Better Results</h3><p>Large prompts force one model to constantly switch context.</p><p>It has to think about:</p><ul><li>architecture</li><li>testing</li><li>code</li><li>documentation</li><li>deployment</li><li>security</li></ul><p>all at the same time.</p><p>That increases cognitive load — even for AI.</p><p>Specialized agents stay focused.</p><p>Each one goes deeper because it has one responsibility.</p><p>The result is:</p><ul><li>fewer missed issues</li><li>deeper reasoning</li><li>parallel execution</li><li>easier verification</li><li>more comprehensive output</li></ul><h3>This Isn’t Just About Coding</h3><p>The same idea applies everywhere.</p><h3>Product Development</h3><p>One agent researches competitors.</p><p>Another estimates pricing.</p><p>Another analyzes customer interviews.</p><p>Another designs the roadmap.</p><h3>Cloud Architecture</h3><p>One reviews IAM.</p><p>One checks networking.</p><p>One validates Terraform.</p><p>One estimates AWS costs.</p><p>One reviews observability.</p><h3>Research</h3><p>One gathers academic papers.</p><p>One summarizes.</p><p>One fact-checks.</p><p>One identifies contradictions.</p><p>One creates visual explanations.</p><h3>Think Like a Tech Lead</h3><p>The biggest mindset shift isn’t using a better model.</p><p>It’s learning to delegate.</p><p>Instead of saying:</p><blockquote><em>“Analyze my project.”</em></blockquote><p>Ask for specialized reviews.</p><p>Instead of one giant prompt…</p><p>Create multiple focused objectives.</p><p>That’s exactly what experienced engineering managers do with people.</p><p>Now we can do the same with AI.</p><h3>The Future Isn’t Bigger Models</h3><p>For years, AI progress was measured by model size.</p><p>Today, the biggest leap is orchestration.</p><p>Modern AI systems increasingly behave less like a single assistant and more like a coordinated team of specialists that collaborate on your behalf.</p><p>The value isn’t only in intelligence.</p><p>It’s in coordination.</p><h3>Final Thoughts</h3><p>The next time you’re tackling a complex problem, don’t ask one AI to wear ten hats.</p><p>Ask ten specialists to wear one hat each.</p><p>Then combine their expertise.</p><p>Sometimes the fastest way to get a better answer isn’t a smarter model.</p><p>It’s a better team.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=69434e65ba3c" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Loop Engineering: The AI Skill Replacing Prompt Engineering]]></title>
            <description><![CDATA[<div class="medium-feed-item"><p class="medium-feed-image"><a href="https://medium.com/@vinodbokare0588/loop-engineering-the-ai-skill-replacing-prompt-engineering-0e108d3713be?source=rss-66aeb228608b------2"><img src="https://cdn-images-1.medium.com/max/1017/1*aZPFmbV21KZrHR-wjLiHvA.png" width="1017"></a></p><p class="medium-feed-snippet">Why the best engineers have stopped typing prompts &#x2014; and started building machines that type for them</p><p class="medium-feed-link"><a href="https://medium.com/@vinodbokare0588/loop-engineering-the-ai-skill-replacing-prompt-engineering-0e108d3713be?source=rss-66aeb228608b------2">Continue reading on Medium »</a></p></div>]]></description>
            <link>https://medium.com/@vinodbokare0588/loop-engineering-the-ai-skill-replacing-prompt-engineering-0e108d3713be?source=rss-66aeb228608b------2</link>
            <guid isPermaLink="false">https://medium.com/p/0e108d3713be</guid>
            <category><![CDATA[ai]]></category>
            <category><![CDATA[ai-agent]]></category>
            <category><![CDATA[software-engineering]]></category>
            <category><![CDATA[loop-engineering]]></category>
            <dc:creator><![CDATA[Vinodbokare]]></dc:creator>
            <pubDate>Sat, 27 Jun 2026 18:00:34 GMT</pubDate>
            <atom:updated>2026-06-27T18:00:34.022Z</atom:updated>
        </item>
        <item>
            <title><![CDATA[How to build Your Own Free AI Coding Assistant on Windows Using Ollama, Qwen3, Aider, and PyCharm]]></title>
            <description><![CDATA[<div class="medium-feed-item"><p class="medium-feed-image"><a href="https://medium.com/@vinodbokare0588/how-to-build-your-own-free-ai-coding-assistant-on-windows-using-ollama-qwen3-aider-and-pycharm-3435ada5e27c?source=rss-66aeb228608b------2"><img src="https://cdn-images-1.medium.com/max/600/1*kdARhOZ3o-rxlGfFO1f8yw.png" width="600"></a></p><p class="medium-feed-snippet">Last week I was experimenting with AI-assisted development and quickly realized how fast usage costs can add up. Between large code&#x2026;</p><p class="medium-feed-link"><a href="https://medium.com/@vinodbokare0588/how-to-build-your-own-free-ai-coding-assistant-on-windows-using-ollama-qwen3-aider-and-pycharm-3435ada5e27c?source=rss-66aeb228608b------2">Continue reading on Medium »</a></p></div>]]></description>
            <link>https://medium.com/@vinodbokare0588/how-to-build-your-own-free-ai-coding-assistant-on-windows-using-ollama-qwen3-aider-and-pycharm-3435ada5e27c?source=rss-66aeb228608b------2</link>
            <guid isPermaLink="false">https://medium.com/p/3435ada5e27c</guid>
            <category><![CDATA[ollama]]></category>
            <category><![CDATA[ai]]></category>
            <category><![CDATA[local-llm]]></category>
            <category><![CDATA[aider]]></category>
            <dc:creator><![CDATA[Vinodbokare]]></dc:creator>
            <pubDate>Sat, 20 Jun 2026 22:20:17 GMT</pubDate>
            <atom:updated>2026-06-20T22:22:31.233Z</atom:updated>
        </item>
        <item>
            <title><![CDATA[AI Agents Can’t Read Your Database — Build This Layer]]></title>
            <description><![CDATA[<div class="medium-feed-item"><p class="medium-feed-image"><a href="https://medium.com/@vinodbokare0588/ai-agents-cant-read-your-database-build-this-layer-d6d36983a26d?source=rss-66aeb228608b------2"><img src="https://cdn-images-1.medium.com/max/1774/1*BlG4lAgPRRyydmvnZXfjAQ.png" width="1774"></a></p><p class="medium-feed-snippet">Why text-to-SQL agents hallucinate on real databases &#x2014; and how an open-source, read-only semantic layer grounds them. With architecture.</p><p class="medium-feed-link"><a href="https://medium.com/@vinodbokare0588/ai-agents-cant-read-your-database-build-this-layer-d6d36983a26d?source=rss-66aeb228608b------2">Continue reading on Medium »</a></p></div>]]></description>
            <link>https://medium.com/@vinodbokare0588/ai-agents-cant-read-your-database-build-this-layer-d6d36983a26d?source=rss-66aeb228608b------2</link>
            <guid isPermaLink="false">https://medium.com/p/d6d36983a26d</guid>
            <category><![CDATA[ai-database]]></category>
            <category><![CDATA[artificial-intelligence]]></category>
            <category><![CDATA[ai]]></category>
            <category><![CDATA[postgres-ai]]></category>
            <category><![CDATA[ai-agent]]></category>
            <dc:creator><![CDATA[Vinodbokare]]></dc:creator>
            <pubDate>Sat, 06 Jun 2026 16:35:33 GMT</pubDate>
            <atom:updated>2026-06-06T16:35:33.793Z</atom:updated>
        </item>
        <item>
            <title><![CDATA[One HTTP Header Could Hijack Your AI Agent]]></title>
            <link>https://medium.com/@vinodbokare0588/one-http-header-could-hijack-your-ai-agent-cd957759f766?source=rss-66aeb228608b------2</link>
            <guid isPermaLink="false">https://medium.com/p/cd957759f766</guid>
            <category><![CDATA[python]]></category>
            <category><![CDATA[ai]]></category>
            <category><![CDATA[cybersecurity]]></category>
            <category><![CDATA[mcp-server]]></category>
            <category><![CDATA[ai-security]]></category>
            <dc:creator><![CDATA[Vinodbokare]]></dc:creator>
            <pubDate>Thu, 28 May 2026 07:23:56 GMT</pubDate>
            <atom:updated>2026-05-28T07:23:56.581Z</atom:updated>
            <content:encoded><![CDATA[<h4>The newly discovered “BadHost” vulnerability in Python’s Starlette framework exposes millions of AI agents, FastAPI apps, and MCP servers to credential theft and authentication bypass attacks.</h4><h3>The Setup: The Python Framework Powering Millions of AI Agents</h3><p>When developers build AI agents and APIs in Python, they often choose FastAPI.<br>To serve large language models at scale, many rely on vLLM.<br>For routing requests across multiple AI providers, LiteLLM has become a popular choice.</p><p>What most developers don’t realize is that all of these tools share the same underlying foundation: Starlette.</p><p><a href="https://starlette.dev/"><strong>Starlette </strong></a>is a lightweight ASGI (Asynchronous Server Gateway Interface) framework that quietly powers a massive portion of the modern Python ecosystem. It is fast, elegant, and deeply embedded in today’s AI infrastructure — with more than 325 million downloads every week.</p><p>On May 27, 2026, security researchers at Secwest disclosed CVE-2026–48710, nicknamed “BadHost,” a high-severity vulnerability affecting Starlette that had remained unnoticed across millions of deployments.</p><h3>What Exactly Is BadHost?</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*twt0KvZq794Qqr_ivXNbHg.png" /><figcaption>How, different parts of the framework interpreted this request differently e.g. GET /admin HTTP/1.1<br>Host: attacker.com:<a href="http://twitter.com/victim">@victim</a>.com</figcaption></figure><p>The diagram above shows how a single malformed Host header can cause Starlette’s routing layer and security layer to interpret the same request differently.</p><p>Normally, when a browser or API client sends a request, it includes a Host header identifying the target domain. Starlette uses this value internally while reconstructing the request URL and generating the request.url.path value used by routers, middleware, and authentication systems.</p><p>The problem begins in <strong>Step 1</strong> of the diagram.</p><p>An attacker sends a specially crafted HTTP request with a malformed Host value:</p><pre>GET /admin HTTP/1.1<br>Host: attacker.com:@victim.com</pre><p>At this point, everything depends on how Starlette reconstructs the URL internally.</p><p>In <strong>Step 2</strong>, Starlette rebuilds the request URL using the manipulated Host header. But different parts of the framework interpret that reconstructed URL differently.</p><ul><li>The <strong>routing layer</strong> sees the request as:</li><li>/admin</li><li>While the <strong>security layer</strong> interprets it as:</li><li>/@victim.com/admin</li></ul><p>That tiny difference becomes catastrophic.</p><p>As shown in <strong>Step 3</strong>, the routing system believes the request matches a legitimate application route and forwards it to the endpoint. Meanwhile, the authentication or middleware layer evaluates a different internal path and fails to apply the correct security policy.</p><p>The result: authentication checks can be bypassed entirely.</p><p>No stolen credentials.<br>No remote code execution.<br>No sophisticated exploit chain.</p><p>Just a carefully malformed HTTP request that causes two different parts of the framework to see two different realities.</p><p>That is the essence of BadHost.</p><p>Researchers at X41 D-Sec summarized the vulnerability bluntly: the routing system and the security layer were effectively evaluating different URLs for the same request.</p><h3>Why AI Agents Make This More Dangerous</h3><p>A normal authentication bypass is already serious.</p><p>But in AI systems, the impact becomes much bigger.</p><p>Modern AI agents don’t just answer questions anymore. They connect to real systems using MCP (Model Context Protocol) servers. These servers help AI agents access tools, APIs, databases, emails, calendars, cloud systems, and business applications.</p><p>To do that, MCP servers store sensitive credentials like:</p><ul><li>API keys</li><li>OAuth tokens</li><li>Service passwords</li></ul><p>That means an AI agent may already have access to:</p><ul><li>Email and calendar accounts</li><li>Customer databases</li><li>Cloud infrastructure</li><li>SaaS platforms</li><li>Internal company tools</li></ul><p>BadHost creates a way to reach those systems.</p><p>Not by stealing passwords.<br>Not by breaking encryption.<br>But by tricking Starlette into skipping authentication checks because different parts of the framework interpret the request differently.</p><p>That is what makes this vulnerability so dangerous.</p><p><em>Researchers at Secwest warned that the official severity score does not fully capture the real-world impact, especially for AI systems connected to sensitive infrastructure.</em></p><h3>Why the Problem Is So Large</h3><p>One of the biggest issues is something called a <strong>transitive dependency</strong>.</p><p>Most developers using FastAPI, LiteLLM, or vLLM never installed Starlette directly. It was automatically included underneath those frameworks.</p><p>So many teams may not even realize they are using a vulnerable version of Starlette.</p><p>Even though a fix is available in Starlette v1.0.1, production systems are often slow to update. Internal tools, research projects, and early AI products can remain unpatched for a long time.</p><p>That increases the overall risk.</p><h3>How to Protect Your Systems</h3><p>The good news: a fix already exists.</p><h3>Immediate Steps</h3><h3>1. Upgrade Starlette</h3><pre>pip install --upgrade starlette</pre><p>Make sure you are running <strong>v1.0.1 or later</strong>.</p><h3>2. Check Your Dependencies</h3><pre>pipdeptree | grep starlette</pre><p>Also check if your system uses:</p><ul><li>FastAPI</li><li>LiteLLM</li><li>vLLM</li></ul><p>These frameworks may include Starlette automatically.</p><h3>3. Scan Your Environment</h3><p>Researchers released a free scanner at:<br><strong>badhost.org</strong></p><h3>4. Protect MCP Servers</h3><p>Do not expose MCP servers directly to the internet without proper authentication and firewall protection.</p><h3>5. Rotate Credentials</h3><p>If you were running a vulnerable version, rotate:</p><ul><li>API keys</li><li>OAuth tokens</li><li>Service credentials</li></ul><p>especially for systems connected to AI agents.</p><h3>The Bigger Lesson</h3><p>BadHost is not just a single framework bug.</p><p>It shows a larger problem in modern AI infrastructure.</p><p>Today, developers build AI systems very quickly using open-source libraries and frameworks. But many teams do not fully track the dependencies running underneath their applications.</p><p>As AI agents gain access to real business systems, security becomes much more important.</p><p>A small bug inside a hidden dependency can suddenly expose emails, databases, cloud systems, and internal tools.</p><p>AI infrastructure needs the same level of security review and monitoring as any production system handling sensitive data.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=cd957759f766" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[ Amazon S3 Files — Principal AWS Deep Dive]]></title>
            <link>https://medium.com/@vinodbokare0588/amazon-s3-files-principal-aws-deep-dive-7cef49d059a0?source=rss-66aeb228608b------2</link>
            <guid isPermaLink="false">https://medium.com/p/7cef49d059a0</guid>
            <category><![CDATA[cloud-storage]]></category>
            <category><![CDATA[amazon-s3]]></category>
            <category><![CDATA[s3-files]]></category>
            <category><![CDATA[cloud-computing]]></category>
            <dc:creator><![CDATA[Vinodbokare]]></dc:creator>
            <pubDate>Sat, 11 Apr 2026 18:46:20 GMT</pubDate>
            <atom:updated>2026-04-11T18:46:20.289Z</atom:updated>
            <content:encoded><![CDATA[<h3>1. Core Understanding</h3><h3>What Is It?</h3><p>S3 Files is AWS’s answer to a 15-year-old question: <em>“Why can’t I just mount my S3 bucket like a file system?”</em> Previous answers (s3fs-fuse, goofys, Mountpoint for S3) all had the same problem — they were FUSE proxies that translated every file operation into synchronous S3 API calls, with no caching and brutal latency on metadata operations.</p><p>S3 Files is a managed NFS file system that uses an S3 bucket as its backing store. Unlike FUSE-based solutions that translate file operations into S3 API calls on every request, S3 Files maintains a high-performance caching layer that serves active data locally while keeping the full dataset in S3. <a href="https://computingforgeeks.com/amazon-s3-files-guide-setup-performance-pricing/">ComputingForGeeks</a> The result is file-system-speed access for hot data, with S3 economics for everything cold.</p><h3>Internal Architecture</h3><p>S3 Files uses a two-tier architecture. When you create an S3 Files file system, AWS provisions a caching layer built on EFS infrastructure. This layer stores the active working set: recently written files, recently read files, and metadata. Small file reads (below the configurable threshold, default 128 KB) are served entirely from this layer with sub-millisecond latency. <a href="https://computingforgeeks.com/amazon-s3-files-guide-setup-performance-pricing/">ComputingForGeeks</a></p><p>The second tier is your existing S3 bucket. Large reads (1 MB and above) stream directly from S3, bypassing the cache entirely. This is a deliberate design choice: S3 already delivers high throughput for large sequential reads, so caching those would waste money without meaningful latency improvement. <a href="https://computingforgeeks.com/amazon-s3-files-guide-setup-performance-pricing/">ComputingForGeeks</a></p><p>The write path is equally important: when you write data, your writes are sent to the highly durable high-performance storage and then synced back to S3 to keep your bucket consistent. Data that hasn’t been accessed within a configurable window (1 to 365 days, defaulting to 30) automatically expires from this storage, so you pay only for what you’re actively using while your authoritative data always remains in S3. <a href="https://aws.amazon.com/s3/features/files/">amazon</a></p><p><strong>Protocol:</strong> NFS 4.1 and 4.2 protocol support with full POSIX permissions, advisory file locking, and read-after-write consistency. Two-way synchronization between the file system view and the S3 bucket — write a file via NFS, read it via the S3 API (after sync). Upload an object via S3, access it through the mount (on first access). <a href="https://computingforgeeks.com/amazon-s3-files-guide-setup-performance-pricing/">ComputingForGeeks</a></p><p><strong>Consistency model:</strong> NFS close-to-open consistency — not strong POSIX consistency. Concurrent writers to the same file need application-level coordination.</p><h3>Key Features &amp; Hard Limits</h3><p>DimensionValueMax IOPS per bucket10M+Aggregate read throughput4 TB/s+Max simultaneous connections25,000Supported computeEC2, Lambda, ECS, EKS, Fargate, BatchCache expiry window1–365 days (default: 30)Small-file threshold (configurable)Default 128 KBProtocolNFS 4.1/4.2OS requirementLinux-based onlyConsistencyNFS close-to-open</p><p><strong>Hard limitations engineers will hit:</strong></p><ul><li><strong>Linux only</strong> — Windows workloads can’t mount it. No SMB/CIFS.</li><li><strong>Close-to-open NFS semantics</strong> — not POSIX strong consistency. Multiple concurrent writers to the same file = undefined behavior without app-level locking.</li><li><strong>Newly launched</strong> — no battle-tested runbooks yet, limited community troubleshooting history.</li><li><strong>Cache is regional</strong> — cross-region access still pays S3 cross-region data transfer.</li><li><strong>Minimum billable cache size: 6 KiB</strong> — thousands of tiny files inflate cache storage cost unexpectedly.</li><li><strong>Writes sync asynchronously</strong> — there is a window where a crash could lose writes not yet flushed to S3.</li></ul><h3>When to Use vs. When NOT to Use</h3><p><strong>Use S3 Files when:</strong></p><ul><li>You have petabytes in S3 but only actively use a fraction at a time (active working set ≪ total dataset)</li><li>Your toolchain is file-based (ML frameworks, bioinformatics pipelines, CAD, render farms) and you’re tired of copy-stage-process-push loops</li><li>You need shared access to the same dataset from 100s–25,000 concurrent compute resources</li><li>You’re building multi-agent AI systems that need a shared working directory</li><li>You already pay for both EFS and S3 and want to eliminate one</li></ul><p><strong>Do NOT use S3 Files when:</strong></p><ul><li>All your data is hot all the time (EFS Standard is cheaper and simpler at that point)</li><li>You need Windows/SMB access — this doesn’t exist here</li><li>Your workload is pure random-write, high-concurrency with strong consistency requirements (use EFS or a database)</li><li>You have millions of tiny files under 6 KB — minimum billing size will destroy your economics</li><li>You need sub-millisecond latency for cold data — first-access still pays S3 retrieval latency</li></ul><h3>2. Cost Model Deep Dive</h3><p>S3 Files pricing trips people up because it layers three separate cost components on top of each other. You pay standard S3 storage and request costs (which you already have), plus two new charges for the high-performance cache layer and data access. <a href="https://computingforgeeks.com/amazon-s3-files-pricing-cost-calculator/">ComputingForGeeks</a></p><h3>Pricing Dimensions (us-east-1)</h3><p><strong>Layer 1 — S3 Standard Storage (existing, unchanged):</strong></p><ul><li>$0.023/GB-month for the first 50 TB, $0.022/GB-month for the next 450 TB, $0.021/GB-month beyond 500 TB. <a href="https://go-cloud.io/amazon-s3-pricing/">GoCloud</a></li><li>PUT/COPY/POST/LIST: $0.005 per 1,000 requests</li><li>GET/SELECT: $0.0004 per 1,000 requests</li></ul><p><strong>Layer 2 — S3 Files Cache Storage (NEW):</strong></p><ul><li>~$0.30/GB-month for data actively cached on the high-performance storage (EFS-backed). <a href="https://computingforgeeks.com/amazon-s3-files-pricing-cost-calculator/">ComputingForGeeks</a> This is ~13× the cost of raw S3 Standard storage.</li><li>Minimum billable size: 6 KiB per file</li><li>Billed hourly, prorated</li></ul><p><strong>Layer 3 — S3 Files Data Access (NEW):</strong></p><ul><li>Cache reads (small files, &lt;1 MB): ~$0.01/GB</li><li>Cache writes (via NFS mount): ~$0.01/GB</li><li>Large reads (1 MB and above) that stream directly from S3 incur only standard S3 GET request costs with no additional S3 Files charge. <a href="https://computingforgeeks.com/amazon-s3-files-guide-setup-performance-pricing/">ComputingForGeeks</a> This is the key cost escape hatch.</li></ul><p><strong>Hidden / Commonly Missed Costs:</strong></p><ol><li><strong>Cache storage for metadata</strong> — File metadata (inodes, directory entries) is cached too, not just file content. A bucket with 10 million small objects generates significant metadata cache overhead.</li><li><strong>Write sync back to S3</strong> — Every NFS write triggers a PUT back to S3 eventually. High-frequency small writes = high PUT request costs.</li><li><strong>EventBridge notifications</strong> — Event notifications are billed directly through Amazon EventBridge. <a href="https://aws.amazon.com/s3/features/files/">amazon</a> If you have file-change listeners, this adds up.</li><li><strong>S3 Versioning on the backing bucket</strong> — Writes via S3 Files create new object versions if versioning is enabled. That’s storage cost for every version.</li><li><strong>Cross-AZ data transfer</strong> — If your compute is in a different AZ than the EFS cache layer, you pay cross-AZ transfer fees (~$0.01/GB each direction).</li><li><strong>Incomplete multipart uploads</strong> — The sync-back mechanism can leave partial uploads if interrupted; lifecycle rules are mandatory.</li><li><strong>Early cache deletion</strong> — Deleting data early results in early deletion charges <a href="https://www.hyperglance.com/blog/aws-s3-pricing-guide/">Hyperglance</a> similar to S3 IA classes. Files deleted before the retention window expires may incur prorated charges.</li></ol><h3>Pricing Formula</h3><pre>Monthly S3 Files Cost =<br>  (Total bucket GB × $0.023)                            # S3 Standard storage<br>+ (Active working set GB × $0.30)                       # Cache storage<br>+ (Small read GB × $0.01)                               # Cache read access<br>+ (Write GB × $0.01)                                    # Cache write access<br>+ (Large read requests × $0.0004/1000)                  # S3 GET for large files<br>+ (Write-back PUT requests × $0.005/1000)               # S3 PUT from cache sync<br>+ (EventBridge events × EventBridge pricing)</pre><h3>3. Cost Optimization Strategies</h3><p><strong>1. Tune the small-file threshold aggressively</strong> The default 128 KB threshold determines what goes through cache vs. direct S3. For ML training on Parquet files averaging 50 MB, raise this threshold to 1 MB or higher so nearly all reads bypass the cache entirely and hit only standard S3 GET costs.</p><p><em>Trade-off:</em> First-byte latency for medium files increases. Fine for batch workloads, problematic for interactive queries.</p><p><strong>2. Shrink the cache retention window</strong> Default is 30 days. If your pipeline runs weekly and your working set is only “hot” for 3 days post-load, set the window to 4–5 days. This keeps your cached GB-month low and forces cold data back to raw S3 pricing.</p><p><em>Trade-off:</em> Cache misses on re-access require reloading from S3. Adds latency if your access pattern isn’t purely sequential.</p><p><strong>3. Use prefix scoping, not full-bucket mounts</strong> Use a prefix to scope exactly what your file-based workloads need, so only your active dataset is placed onto the file system’s high-performance storage. <a href="https://aws.amazon.com/s3/features/files/">amazon</a> If you mount the entire bucket but only 10% of it is ever accessed, you’re paying cache storage for 100% of touched objects (including all the metadata).</p><p><em>Trade-off:</em> Requires application-level awareness of prefix layout.</p><p><strong>4. Write large files; avoid chatty small writes</strong> Every NFS write eventually becomes a PUT request to S3. A workload writing 100,000 × 10 KB files costs the same in PUT requests as writing 100,000 × 10 MB files — but the latter actually does useful work. Batch small writes into larger objects before flushing.</p><p><em>Trade-off:</em> Application refactoring required. Write buffering adds complexity.</p><p><strong>5. Disable S3 Versioning on the backing bucket (or use lifecycle rules)</strong> Each NFS write creates a new S3 object version. With versioning on, your bucket’s storage grows faster than your actual data because previous versions accumulate. Use lifecycle rules to expire noncurrent versions within 1–3 days.</p><p><em>Trade-off:</em> Lose fine-grained version history. Accept coarser point-in-time recovery via S3 Versioning with TTLs.</p><p><strong>6. Separate hot and cold buckets</strong> Don’t mount a multi-petabyte data lake as a single S3 Files system. Create a smaller “working” bucket (hot data only), mount that via S3 Files, and use S3 Replication or Batch Operations to move results back to the cold archive bucket. Only the hot bucket incurs S3 Files cache costs.</p><p><em>Trade-off:</em> Adds operational complexity and data movement overhead.</p><p><strong>7. Anti-patterns that silently inflate cost:</strong></p><ul><li>Mounting a bucket with millions of tiny objects (&lt; 6 KB) — minimum billing per-file creates massive overhead</li><li>Running ls -laR or recursive find on a large mount — generates thousands of READDIR calls, each hitting cache metadata</li><li>Enabling S3 Access Logging on the backing bucket — creates new objects per request, which re-enters the cache as new metadata</li><li>Setting retention window to max (365 days) without profiling actual access patterns</li></ul><h3>4. Usage Pattern &amp; Edge Cases</h3><p><strong>Small files vs. large files:</strong> This is the most critical dimension. For a bucket of 10 million 10 KB files: nearly every read goes through cache (below threshold), cache storage cost is enormous (6 KiB minimum × 10M = ~60 GB cache minimum just for metadata), and cost spirals. For a bucket of 100,000 × 1 GB files: most reads bypass cache entirely at 1 MB threshold, you pay only S3 GET costs for the vast majority of data, and S3 Files is dramatically cheaper than EFS.</p><p><strong>Burst vs. steady load:</strong> The cache layer auto-scales (it’s EFS under the hood). A burst workload that hits 10 TB of data over 6 hours and then goes quiet for a week will accumulate 10 TB × $0.30/GB × (6hrs/720hrs) = ~$25 for that burst, then pay almost nothing until next burst. Compare to EFS which charges $0.30/GB-month on all 10 TB regardless of whether you’re using it.</p><p><strong>Read-heavy vs. write-heavy:</strong> Read-heavy is the optimal case. Large sequential reads (ML training, genomics alignment) bypass cache and incur only S3 GET costs. Write-heavy workloads (checkpoint-heavy training, high-frequency appends) incur both cache write charges and the eventual S3 PUT write-back, plus potential S3 versioning overhead.</p><p><strong>Scaling behavior:</strong> Cost scales sub-linearly with total data size but linearly with active working set size. A 1 PB bucket where only 1 TB is ever active costs: (1 PB × $0.023 ≈ $24,000 for S3) + (1 TB × $0.30 = $300 for cache). EFS for the same 1 PB would cost $307,200/month at standard EFS pricing — 12× more expensive.</p><p><strong>Edge case — Lambda access:</strong> Lambda functions mounting S3 Files via NFS require VPC configuration. Cold-start Lambda invocations that mount and immediately access uncached files will see elevated first-access latency. For Lambda-heavy workloads, keep the working set pre-warmed with a keep-alive process.</p><h3>5. Real-World Architecture Example</h3><p><strong>Use Case: Quantitative Research Firm — Multi-Researcher Backtesting Platform</strong></p><p>Production scenario inspired by firms like QRT (mentioned in AWS customer quotes): 50 quant researchers running simultaneous backtests on petabytes of tick-level market data. Previously: each researcher maintained a local NFS server with a local copy of relevant data, synchronized weekly from S3. Result: 50× data duplication, sync errors, stale data bugs, 4-hour “copy from S3” waits before each new backtest.</p><p><strong>Architecture:</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/449/1*hlP8NdnGqTb0SToLo3hnNA.png" /></figure><p><strong>Where cost is incurred:</strong></p><ul><li>S3 Standard storage: 15 TB total × $0.023 = <strong>$345/month</strong></li><li>S3 Files cache: 200 GB active set × $0.30 = <strong>$60/month</strong></li><li>Cache data access (small reads — metadata, index files): 100 GB/month × $0.01 = <strong>$1/month</strong></li><li>Large reads bypass cache (tick data files are 50–500 MB each): 50 researchers × 20 GB/day × 30 days × $0.0004/1000 GET ≈ <strong>$18/month</strong> in S3 GETs</li><li>Write-back PUTs (result files): 100,000 PUT/month × $0.005/1000 = <strong>$0.50/month</strong></li><li><strong>Total: ~$425/month</strong></li></ul><p><strong>Optimizations applied:</strong></p><ul><li>Retention window set to 7 days (researchers’ active dataset has a weekly cycle)</li><li>Threshold set to 512 KB (most tick data files are &gt;&gt; 512 KB, bypassing cache)</li><li>Prefix-scoped mount to /equity-us/ only — avoids caching the options or futures data that equity researchers never touch</li><li>S3 Versioning with 7-day noncurrent expiry to avoid unbounded storage growth</li></ul><h3>6. Monthly Cost Estimation (MANDATORY)</h3><p><strong>Workload:</strong> ML training platform at a mid-size AI company. 100 TB total training dataset in S3; 5 TB actively used per month by training jobs. Daily training runs reading 500 GB of data; 50 GB/day new data written (checkpoints + preprocessed features). 20 concurrent GPU instances.</p><p>ComponentUnit CostMonthly UsageMonthly CostS3 Standard storage (100 TB)$0.023/GB102,400 GB<strong>$2,355</strong>S3 Files cache storage (5 TB active set)$0.30/GB5,120 GB<strong>$1,536</strong>Large file reads bypass cache (480 GB/day × 30)$0.0004/1000 GETs~300K GETs<strong>$0.12</strong>Small file reads via cache (20 GB/day × 30)$0.01/GB600 GB<strong>$6</strong>NFS writes to cache (50 GB/day × 30)$0.01/GB1,500 GB<strong>$15</strong>S3 PUT write-backs (50 GB/day, avg 100 MB files)$0.005/100015,000 PUTs<strong>$0.075</strong>S3 Versioning storage (7-day retention)$0.023/GB~350 GB (versions)<strong>$8</strong>EventBridge notifications (file events)$1/million events~200K events<strong>$0.20TOTAL~$3,920/month</strong></p><p><em>Compared to running the same 5 TB active dataset on EFS Standard ($0.30/GB × 5,120 GB = $1,536) PLUS the original 100 TB in S3 Standard ($2,355) = $3,891 — nearly identical cost but with the critical advantage that data lives in one place and never needs ETL sync.</em></p><h3>7. Naive vs. Optimized Cost Comparison (MANDATORY)</h3><h3>Naive Architecture</h3><p><strong>What they built:</strong> A team mounts the entire 100 TB bucket with default settings (128 KB threshold, 30-day retention), runs ls -R in deployment scripts to pre-warm, keeps versioning on without noncurrent expiry, and does frequent 1–50 KB checkpoint writes.</p><pre>Naive Setup:<br>- Full bucket mount (100 TB visible to cache)<br>- 30-day cache retention<br>- 128 KB threshold (most checkpoint files &lt; 128 KB → hit cache)<br>- 5 million checkpoint files × 6 KB min = 30 GB cache minimum for metadata alone<br>- No noncurrent version expiry<br>- 100K writes/day of 10 KB checkpoint files<br><br>Cost:<br>- S3 storage (100 TB): $2,355/month<br>- Cache storage (50 TB touched over 30 days): $15,360/month<br>- Cache writes (100K × 10KB/day × 30): 30 GB × $0.01 = $0.30/month<br>- Cache read access (small files): 200 GB × $0.01 = $2/month<br>- S3 versioned storage accumulation: +$500/month after 90 days<br>NAIVE TOTAL: ~$18,217/month</pre><h3>Optimized Architecture</h3><p><strong>What changed:</strong> Prefix-scoped mount to /checkpoints/current/ only; retention window 3 days; threshold raised to 1 MB; checkpoints batched into 50 MB rollup files before write; noncurrent version expiry lifecycle rule at 3 days.</p><pre>Optimized Setup:<br>- Prefix-scoped mount (500 GB active prefix)<br>- 3-day retention window<br>- 1 MB threshold (checkpoints batched to 50 MB files → bypass cache)<br>- Lifecycle: noncurrent versions expire in 3 days<br>- Batched writes: 100K × 10KB → 20 × 50MB files/day<br>Cost:<br>- S3 storage (100 TB): $2,355/month (unchanged)<br>- Cache storage (500 GB × $0.30): $150/month<br>- Large file reads bypassing cache: $1/month<br>- Cache writes (batched, minimal): $2/month<br>- S3 storage version churn (controlled): $15/month<br>OPTIMIZED TOTAL: ~$2,523/month</pre><p>NaiveOptimizedSavingsMonthly cost$18,217$2,523<strong>$15,694/month</strong>Annual$218,604$30,276<strong>$188,328/year</strong>Reduction — — <strong>86%</strong></p><p><strong>What drove the savings:</strong> Cache storage was the killer — mounting the entire bucket and keeping 30-day retention meant 50 TB of cache at $0.30/GB. Scoping to active prefix + 3-day window cut cache to 500 GB. Batching checkpoints eliminated small-file penalty entirely.</p><h3>8. Monitoring &amp; Cost Observability (MANDATORY)</h3><h3>Key CloudWatch Metrics</h3><p>MetricNamespaceWhat It Tells YouCacheStorageBytesAWS/S3FilesActive cache size — primary cost driver; alert if growing unexpectedlyCacheHitRateAWS/S3FilesLow hit rate means you&#39;re bypassing cache (good for large files, bad for small)DataReadIOBytesAWS/S3FilesGB served from cache vs. direct S3; maps directly to access chargesDataWriteIOBytesAWS/S3FilesBytes written through NFS; maps to write access charges + S3 PUT costMetadataIOOperationsAWS/S3FilesSpike here = someone running recursive scans; directly inflates costBucketSizeBytesAWS/S3Track S3 Versioning growth; filter by StorageType = AllStorageTypesNumberOfObjectsAWS/S3Object count growth; small objects at scale = cache min-size billing5xxErrorsAWS/S3FilesElevated errors often precede runaway retry storms that inflate request costs</p><h3>Cost Explorer Setup</h3><ul><li><strong>Group by:</strong> Service = S3, then Usage Type — look specifically for S3Files-CacheStorage-* and S3Files-DataAccess-* line items (newly added in April 2026)</li><li><strong>Filter to:</strong> Usage Type Group = S3 Files to isolate new charges from legacy S3 costs</li><li><strong>Time granularity:</strong> Daily — cache storage is hourly-prorated, so daily view catches anomalies faster than monthly</li><li><strong>Tag strategy:</strong> Tag your S3 Files file system with Team, Workload, CostCenter at creation; these propagate to all associated charges</li></ul><h3>Budget &amp; Alarm Configuration</h3><pre>Budget 1: S3 Files Cache Storage<br>  Threshold: $500/month<br>  Alert: 80% actual ($400), 100% forecast<br>  Action: SNS → Slack #infra-cost channel<br><br>Budget 2: Total S3 (including Files)<br>  Threshold: $5,000/month<br>  Alert: 90% actual<br>  Action: SNS → PagerDuty P3<br><br>CloudWatch Alarm: CacheStorageBytes<br>  Threshold: &gt; 2 TB (your expected max)<br>  Period: 1 hour<br>  Action: SNS notification to on-call</pre><h3>Logs/Metrics Indicating Cost Inefficiency</h3><ul><li>CacheStorageBytes growing monotonically → retention window too long, or writes never expire</li><li>MetadataIOOperations spike without corresponding DataReadIOBytes → someone is running find / ls -R on the mount</li><li>DataWriteIOBytes &gt;&gt; DataReadIOBytes → write-heavy workload; evaluate whether NFS writes vs. direct S3 SDK would be cheaper</li><li>S3 NumberOfObjects growing 10× faster than BucketSizeBytes → small-file explosion, minimum billing penalty accumulating</li></ul><h3>9. Cost Anomaly Scenarios (MANDATORY)</h3><pre>Scenario 1: The Accidental Full-Bucket Mount<br>Trigger: Developer mounts entire 500 TB data lake bucket to debug<br>  a single pipeline issue.<br>Root Cause: All accessed objects (even from browsing directories)<br>  get pulled into the cache layer. A 2-hour debug session touching<br>  10 TB of objects leaves 10 TB stuck in cache for 30 days.<br>  10,000 GB × $0.30 = $3,000 surprise on next bill.<br>Early Detection: CacheStorageBytes alarm crossing expected threshold<br>  within hours of mount creation.<br>Prevention: Enforce prefix-scoped mounts via IAM policy; require<br>  a /active/ prefix structure; set cache window to 3–7 days;<br>  audit new file system creations via CloudTrail.</pre><pre>Scenario 2: Recursive Scan by Monitoring Agent<br>Trigger: A third-party monitoring tool (Datadog, Prometheus exporter,<br>  custom script) runs `find / -name &quot;*.log&quot;` across all mounted paths<br>  hourly, including S3 Files mounts.<br>Root Cause: Each directory traversal fires thousands of READDIR/GETATTR<br>  NFS calls, all of which hit the metadata cache layer and count as<br>  MetadataIOOperations charges. At 50M metadata ops/month the bill<br>  can exceed $500 from a monitoring agent alone.<br>Early Detection: MetadataIOOperations CloudWatch metric spikes on an<br>  hourly cadence; no corresponding DataRead spike.<br>Prevention: Exclude S3 Files mount paths from recursive monitoring tools.<br>  Mount with noatime option. Use S3 Inventory instead of filesystem scans.</pre><pre>Scenario 3: S3 Versioning + High-Frequency Checkpoint Writes<br>Trigger: ML training job writes model checkpoints every 60 seconds<br>  (100 MB each) to the NFS mount. Versioning is ON.<br>Root Cause: Each NFS write → 1 new S3 object version. 1,440 writes/day<br>  × 100 MB = 144 GB/day of new S3 versions being created. At $0.023/GB<br>  that&#39;s $3.30/day = $100/month just from one training job&#39;s checkpoints.<br>  With 50 concurrent jobs: $5,000/month in version storage, compounding.<br>Early Detection: BucketSizeBytes growing 10× faster than your expected<br>  dataset growth rate. Check StorageType breakdown in Cost Explorer.<br>Prevention: Lifecycle rule: expire noncurrent versions after 3 days.<br>  Or better: checkpoint to a separate non-versioned bucket, replicate<br>  the best checkpoint to the versioned bucket only.</pre><pre>Scenario 4: Lambda Cold Start Mount Storm<br>Trigger: Auto-scaling event triggers 500 Lambda cold starts<br>  simultaneously, each mounting S3 Files and reading a shared<br>  10 MB config file.<br>Root Cause: 500 parallel first-access reads on a cold cache cause<br>  500 concurrent S3 GET requests. Lambda functions are billed for<br>  the full duration including the NFS mount + S3 cold-load latency<br>  (can be 2–5 seconds). 500 × 5s × 3 GB Lambda = expensive.<br>  Plus the cache layer spikes causing EFS burst credit burn.<br>Early Detection: Lambda Duration P99 spike; ConcurrentExecutions<br>  alarm; S3Files DataReadIOBytes sudden spike.<br>Prevention: Pre-warm cache with a scheduled job. For config files,<br>  use SSM Parameter Store or S3 direct SDK — don&#39;t go through NFS<br>  for config reads that don&#39;t need file semantics.</pre><pre>Scenario 5: Multipart Upload Abandonment During Cache Sync<br>Trigger: A large write (50 GB ML output) via NFS is interrupted<br>  mid-sync (EC2 spot termination, OOM kill). The sync-back creates<br>  a partial multipart upload in S3 that never completes.<br>Root Cause: Incomplete multipart uploads are charged at S3 Standard<br>  rates indefinitely. A 50 GB incomplete MPU sitting for 30 days<br>  costs 50 GB × $0.023 = $1.15 — trivially. But at scale (100s<br>  of interrupted jobs/month) this accumulates to hundreds of dollars.<br>Early Detection: S3 Storage Lens dashboard showing MPU bytes growing;<br>  Cost Explorer showing unexpected S3 Standard storage growth without<br>  matching object count growth.<br>Prevention: Lifecycle rule on ALL buckets: abort incomplete MPUs<br>  after 3 days. This is table stakes for any S3 bucket, and<br>  especially critical for S3 Files-backed buckets.</pre><h3>10. Interview-Level Insights</h3><p><strong>Q1: S3 Files vs. Mountpoint for S3 — when would you still choose Mountpoint?</strong></p><p>A: Mountpoint for S3 is open-source, runs entirely in-process on your EC2 instance, and has zero additional cost beyond standard S3 request charges. For read-only workloads on large files where you only need sequential access — like a single EC2 training job pulling data it won’t re-access — Mountpoint delivers comparable throughput at lower cost because there’s no cache layer billing. S3 Files wins when you need: write support with sync-back, shared multi-client concurrent access (Mountpoint is single-host), POSIX-compatible metadata operations, or low-latency repeated access to a hot working set. The moment you have 2+ compute nodes needing the same data as files, Mountpoint falls apart and S3 Files makes sense.</p><p><strong>Q2: Explain the write path in detail. What are the consistency guarantees between NFS clients?</strong></p><p>A: Write via NFS → buffered in EFS-backed cache layer → periodically flushed to S3 (asynchronously). Two clients with the same file open get NFS close-to-open consistency: after Client A closes a file, Client B opening it will see A’s writes. But while both have the file open simultaneously, there’s no locking enforcement beyond NFS advisory locks. If both clients write to the same file concurrently without coordination, you have undefined behavior. This is different from EFS which also only gives close-to-open, but at least the NFS cache is shared and the race window is narrower. The important engineering implication: S3 Files is safe for patterns like “one writer, many readers” or “independent writers to separate files,” but requires application-level coordination for concurrent writers to the same file.</p><p><strong>Q3: How does S3 Files interact with S3 Intelligent-Tiering on the backing bucket?</strong></p><p>A: Tricky one. If your backing bucket has S3 Intelligent-Tiering enabled and objects get moved to Infrequent Access tier, when S3 Files needs to load them into cache, you pay the S3-IA retrieval fee ($0.01/GB) on top of the S3 Files data access charges. Worse, if objects get moved to Archive Access tier, cache loading requires a restore operation (hours) which completely breaks the file-system access model. The correct setup is: use S3 Standard as the storage class for the S3 Files-backing bucket (or the prefix mounted via S3 Files), and use Intelligent-Tiering only on prefixes not mounted via S3 Files.</p><p><strong>Q4: A customer says “S3 Files delivered 90% cost reduction.” Walk me through when this claim is valid vs. when it’s misleading.</strong></p><p>A: Valid when: the customer was previously maintaining a separate EFS filesystem containing a copy of their S3 data. EFS Standard costs $0.30/GB-month on ALL data. A 10 TB EFS + 10 TB S3 duplicate = $3,072 + $230 = $3,302/month. With S3 Files: 10 TB in S3 ($230) + 500 GB cache for active set ($150) = $380/month. That’s ~88% reduction. The claim is misleading when: the customer only had S3 before and is now adding S3 Files cache costs on top. In that scenario, they’re paying MORE than before, not less — they’re buying file-system access that previously didn’t exist at all. Journalists and AWS marketing conflate these two situations. As a principal engineer, always ask: “90% compared to what baseline?”</p><p><strong>Q5: How do you design for disaster recovery with S3 Files?</strong></p><p>A: The authoritative copy is always S3. Recovery is: recreate the S3 Files file system, mount it, and the data is immediately accessible — S3 is the source of truth. The cache is ephemeral and reconstruction is automatic on first access. The risk is the write-back window: writes that are in the cache but not yet flushed to S3 are vulnerable to cache-layer failure. Mitigate with: (1) use NFS sync mount option to force synchronous write-through to S3 on every write (latency penalty but eliminates the window), (2) use S3 Versioning with cross-region replication on the backing bucket for regional DR, (3) treat the cache as a performance layer only, never as the authoritative store — architect your application to re-derive/re-process rather than relying on checkpoint recovery from cache.</p><p><strong>Q6: What happens to cost when your working set exceeds your expected cache size?</strong></p><p>A: S3 Files auto-scales the cache — there’s no capacity cap you configure (unlike EFS Provisioned Throughput). If your working set grows from 500 GB to 5 TB, the cache grows to 5 TB and your cache storage bill grows 10× with it. The danger is that there’s no hard limit you set; cost scales elastically. Monitoring CacheStorageBytes with an alarm and a budget threshold is not optional. A pathological case: a daily batch job that touches a new 10 TB slice each day but has a 30-day retention window will accumulate 300 TB of cache storage after a month (30 days × 10 TB/day), costing $90,000/month in cache alone.</p><p><strong>Q7: Common mistakes engineers make:</strong></p><ol><li><strong>Leaving the retention window at 30 days without profiling</strong> — Most workloads have a working set that’s only “hot” for 1–7 days. Default 30 days = 4× the necessary cache cost.</li><li><strong>Mounting the entire bucket instead of a prefix</strong> — Every object you touch enters the cache. Scope aggressively.</li><li><strong>Not setting a lifecycle rule for incomplete MPUs</strong> — The first interrupted large write becomes a phantom S3 charge that nobody notices for months.</li><li><strong>Using S3 Files for Windows workloads</strong> — Linux-only. Teams that discover this after architectural planning lose weeks.</li><li><strong>Assuming POSIX strong consistency</strong> — NFS close-to-open is not POSIX. Concurrent writers to the same file without application locks will corrupt data silently.</li></ol><h3>11. Decision Framework</h3><h3>When Should a Principal Engineer Choose S3 Files?</h3><p>Choose S3 Files when <strong>all three</strong> of these are true:</p><ol><li>Data lives in S3 (or should) as the authoritative store</li><li>Compute tools expect file system semantics (POSIX paths, standard file I/O)</li><li>Active working set is a fraction of total dataset size (ideally &lt; 20%)</li></ol><h3>Comparison vs. Alternatives</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*vW1RLnokCQAL5KlWFh_QzA.png" /></figure><h3>Decision Checklist</h3><p>Before committing to S3 Files, answer these:</p><ul><li>Is my authoritative data store S3 (or do I want it to be)? If not, use EFS.</li><li>Do I need shared multi-client file access (&gt;1 host)? If single-host read-only, use Mountpoint instead and save the cache cost.</li><li>Is my active working set &lt; 30% of total data? If active set ≈ total data, EFS is simpler and comparably priced.</li><li>Are my files generally &gt; 128 KB? Small-file-dominated workloads pay a disproportionate cache penalty.</li><li>Am I on Linux? If Windows workloads are involved, this is a hard blocker today.</li><li>Have I profiled my actual hot data window? Set retention window to match, not default.</li><li>Do I have S3 Versioning enabled? If yes, do I have noncurrent version expiry lifecycle rules?</li><li>Is the service available in my required region? As of April 2026, check AWS regional availability — newly launched services often roll out to additional regions over 3–6 months.</li></ul><p>#Aws #S3 #S3Files</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=7cef49d059a0" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Amazon S3 Files: Bridging the Gap Between Object Storage and File Systems]]></title>
            <link>https://medium.com/@vinodbokare0588/amazon-s3-files-bridging-the-gap-between-object-storage-and-file-systems-1011290adeaf?source=rss-66aeb228608b------2</link>
            <guid isPermaLink="false">https://medium.com/p/1011290adeaf</guid>
            <category><![CDATA[cloud-computing]]></category>
            <category><![CDATA[s3]]></category>
            <category><![CDATA[data-engineering]]></category>
            <category><![CDATA[aws-s3]]></category>
            <category><![CDATA[aws]]></category>
            <dc:creator><![CDATA[Vinodbokare]]></dc:creator>
            <pubDate>Wed, 08 Apr 2026 09:12:51 GMT</pubDate>
            <atom:updated>2026-04-08T09:12:51.188Z</atom:updated>
            <content:encoded><![CDATA[<h4>Access your S3 data as a file system — without duplication, without pipelines, and without changing your applications</h4><p>For years, teams using <strong>Amazon S3</strong> faced a common challenge:<br> S3 is highly scalable and cost-effective, but it is not a file system.</p><p>Applications that rely on file operations — like reading, writing, or appending files — often required additional layers such as file systems or data pipelines. This led to duplicated data, increased costs, and unnecessary complexity.</p><p>Amazon S3 Files changes this model.</p><p>It allows you to access your S3 data as a file system while keeping the data exactly where it is — inside S3. No movement, no synchronization, and no new tools to learn.</p><h3>What is Amazon S3 Files?</h3><p><strong>Amazon S3 Files</strong> is a new capability that allows you to use your S3 data as a <strong>shared file system</strong>.</p><p>Traditionally, S3 stored data as objects. With S3 Files, the same data can now be accessed like a <strong>file system with folders, paths, and standard file operations</strong>.</p><p>Key idea:</p><ul><li>Your data <strong>remains in S3</strong></li><li>But applications can access it <strong>as files instead of objects</strong></li></ul><p>This means:</p><ul><li>No data movement</li><li>No duplication</li><li>No new APIs to learn</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*chZm6CeLydb1dBHQgBw8Wg.png" /></figure><h3>Why AWS introduced S3 Files</h3><p>Many applications are built to work with <strong>file systems, not object storage</strong>. This created common challenges:</p><ul><li>Teams had to <strong>copy data between S3 and file systems</strong> (like EFS)</li><li>Additional pipelines were needed to <strong>sync data</strong></li><li>Increased <strong>cost, complexity, and latency</strong></li></ul><p>S3 Files solves this by:</p><ul><li>Allowing <strong>file-based access directly on S3</strong></li><li>Eliminating <strong>data duplication</strong></li><li>Removing the need for <strong>synchronization pipelines</strong></li></ul><p>In short, AWS introduced S3 Files to <strong>bridge the gap between object storage and file systems</strong>.</p><h3>3. What S3 Files enables</h3><h3>Direct file access to S3 data</h3><p>Applications, scripts, and tools can now:</p><ul><li>Read/write data as files</li><li>Use standard file paths and directories</li><li>Work without code changes</li></ul><h3>High-performance file workloads</h3><ul><li>Low latency access for frequently used data</li><li>Intelligent caching for active datasets</li><li>High throughput for large-scale workloads</li></ul><h3>Store once, use everywhere</h3><ul><li>One copy of data in S3</li><li>Accessible as both:</li><li>Object storage (APIs, SDKs)</li><li>File system (NFS interface)</li></ul><p>No more:</p><ul><li>Duplicate storage</li><li>Data sync jobs</li></ul><h3>Cost-efficient scaling</h3><ul><li>No infrastructure to manage</li><li>Pay only for what you use</li><li>Optimized for active working data</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*qeqJOM80ygqIC3MgKuMg3w.png" /><figcaption>comparison of Old S3 with New S3 Files</figcaption></figure><p>Amazon S3 Files is a significant step forward. It combines:</p><ul><li>The <strong>scalability and durability of S3</strong></li><li>With the <strong>simplicity of a file system</strong></li></ul><p>This removes a long-standing trade-off in cloud architectures:</p><blockquote><em>Choosing between object storage and file systems</em></blockquote><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=1011290adeaf" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[The Week-Long Bug That Would Take 10 Seconds Today]]></title>
            <description><![CDATA[<div class="medium-feed-item"><p class="medium-feed-snippet">Five years ago, when I was working on Morningstar systems, I got stuck on a bug that quietly took over my entire week.</p><p class="medium-feed-link"><a href="https://medium.com/@vinodbokare0588/the-week-long-bug-that-would-take-10-seconds-today-6f2a929b4b30?source=rss-66aeb228608b------2">Continue reading on Medium »</a></p></div>]]></description>
            <link>https://medium.com/@vinodbokare0588/the-week-long-bug-that-would-take-10-seconds-today-6f2a929b4b30?source=rss-66aeb228608b------2</link>
            <guid isPermaLink="false">https://medium.com/p/6f2a929b4b30</guid>
            <category><![CDATA[debugging]]></category>
            <category><![CDATA[ai]]></category>
            <category><![CDATA[productivity]]></category>
            <category><![CDATA[lessons-learned]]></category>
            <dc:creator><![CDATA[Vinodbokare]]></dc:creator>
            <pubDate>Fri, 03 Apr 2026 03:51:01 GMT</pubDate>
            <atom:updated>2026-04-03T03:51:01.607Z</atom:updated>
        </item>
        <item>
            <title><![CDATA[Why Single-Threaded Redis Beats Your 16-Thread Database]]></title>
            <link>https://medium.com/@vinodbokare0588/why-single-threaded-redis-beats-your-16-thread-database-16e3694c519c?source=rss-66aeb228608b------2</link>
            <guid isPermaLink="false">https://medium.com/p/16e3694c519c</guid>
            <category><![CDATA[system-design-interview]]></category>
            <category><![CDATA[redis]]></category>
            <category><![CDATA[threads]]></category>
            <dc:creator><![CDATA[Vinodbokare]]></dc:creator>
            <pubDate>Tue, 31 Mar 2026 19:34:09 GMT</pubDate>
            <atom:updated>2026-03-31T19:34:09.565Z</atom:updated>
            <content:encoded><![CDATA[<blockquote>The surprising reality of performance and concurrency: doing less—but doing it intelligently—almost always delivers better results.</blockquote><p><strong>The interviewer leans back and asks</strong>: <em>“If multi-threading is supposed to make things faster, why does single-threaded Redis beat most databases? Your database has 16 threads. Redis has 1. Redis is still winning. How?”</em></p><p>This is one of those questions that filters engineers from juniors instantly. <strong>The weak answer is “because Redis is in-memory.”</strong> That’s true but incomplete — it misses the entire architectural philosophy. Let’s go deep.</p><p>“<strong>Threading doesn’t make things faster</strong>. Threading makes things concurrent. And concurrency has a price.” — The core insight every interviewer is waiting to hear</p><h3>First: What Does “Fast” Actually Mean?</h3><p>Before answering, you need to reframe the question. Speed has two axes:</p><ul><li><strong>Throughput </strong>— how many operations can you complete per second total?</li><li><strong>Latency </strong>— how long does a single operation take from start to finish?</li></ul><p>Multi-threading can improve throughput by doing work in parallel. But it often <em>kills</em> latency due to overhead. Redis is optimized for both, and that’s the trick.</p><h3>The Hidden Tax of Multi-Threading</h3><p>Here’s what most textbooks miss: threads aren’t free. When multiple threads access shared data, there’s always a cost. Let’s see what that looks like in a typical multi-threaded database.</p><h3>1. Lock Contention</h3><p>When 16 threads all want to read or write the same row, they fight for a lock. Thread 1 grabs it, threads 2–16 wait. That waiting is called lock contention, and at high concurrency it becomes the bottleneck — not compute, not memory, just threads waiting for other threads.</p><blockquote><em>“Imagine 16 chefs trying to use the same chopping board. Yes, you have 16 chefs. But only 1 can chop at a time, and 15 are just standing around elbowing each other. Now compare to a restaurant with 1 chef, an extremely well-organized kitchen, and zero coordination overhead. The single chef wins for most orders.”</em></blockquote><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Sx_d6pE2wJUHHI5a6x--1w.png" /><figcaption>16 chef’s vs 1 well organized chef</figcaption></figure><h3>2. Context Switching</h3><p>The OS constantly switches between threads — saving one thread’s state, loading another’s. This is called a context switch, and it’s expensive: 1–10 microseconds per switch. With 16 threads competing for CPU time, you can easily burn millions of microseconds per second just on bookkeeping.</p><pre>// What a context switch looks like under the hood<br>// CPU must save:<br>  - Program counter (where am I?)<br>  - All registers (what values do I have?)<br>  - Stack pointer<br>  - TLB entries (memory mappings)<br><br>// Then restore all of this for the next thread<br>// ~1,000–10,000 ns per switch × millions of switches/sec = real overhead</pre><h3>3. Cache Invalidation</h3><p>Each CPU core has its own L1/L2 cache. When Thread A on Core 1 modifies data, Core 2 (running Thread B) must be told its cached copy is stale. This cache coherence traffic flies across the CPU bus constantly in multi-threaded programs, slowing everything down.</p><h4>Multi-Threaded DB (16 threads)</h4><ul><li>Lock/unlock on every operation</li><li>Context switch overhead</li><li>Cache coherence bus traffic</li><li>Deadlock avoidance complexity</li><li>Thread scheduler unpredictability</li><li>Memory barriers slow writes</li></ul><h4>Redis (1 thread)</h4><ul><li>Zero locks needed (1 writer)</li><li>Zero context switches</li><li>Cache always hot and valid</li><li>Deadlocks literally impossible</li><li>100% predictable execution</li><li>No memory sync overhead</li></ul><h3>The Redis Architecture: The Event Loop</h3><p>Redis doesn’t use threads to handle concurrency. It uses an event loop — the same pattern Node.js, Nginx, and most high-performance servers use.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*BODfEqMtVqfuqjdZOJI-Ng.png" /><figcaption>redis architecture</figcaption></figure><p>The OS kernel watches all sockets using epoll (Linux) or kqueue (macOS). When client data arrives, the kernel notifies Redis: &quot;Hey, client C3 has data.&quot; Redis reads it, processes it — entirely in memory — and replies. Then moves to the next ready client. No waiting. No blocking. No locks.</p><p>The key insight: Redis never <em>waits</em> on I/O. It only processes sockets when they’re already ready to be read or written. The OS does the waiting — Redis only runs when there’s real work to do. This is called non-blocking I/O with multiplexing.</p><h3>Why In-Memory Is The Multiplier</h3><p>Now add the other half: everything Redis touches is in RAM.</p><blockquote>Redis GET (in-memory)~0.1ms</blockquote><blockquote>PostgreSQL SELECT (index hit)~1–5ms</blockquote><blockquote>MySQL SELECT (disk read)~10–50ms</blockquote><p>Disk I/O takes 10,000–100,000× longer than RAM access. A multi-threaded database uses threads partly to hide that latency — while Thread 1 waits for disk, Thread 2 can run. But Redis never goes to disk for its primary data, so it never needs that trick.</p><h3>The Mental Model That Wins Interviews</h3><p>Here’s the framing that impresses interviewers. Threads are a solution to waiting. When you’re waiting on something (disk, network, CPU from another process), threads let you do other useful work in the meantime.</p><p>Redis removed the waiting entirely. No disk I/O to wait on. No lock contention. No blocking calls. When there’s no waiting, threads become pure overhead — coordination cost with zero benefit.</p><p>It’s like asking: “Why does a Formula 1 car use only 1 driver instead of 2?” Because the bottleneck isn’t the number of people — it’s how fast one person can react and steer. Adding a second driver doesn’t make it go faster; it just adds coordination complexity on a 200mph machine.</p><h3>What About Modern Redis? (Redis 6+)</h3><p>Strong answer bonus: mention that Redis 6.0 (2020) introduced I/O threads — but only for network I/O (reading/writing sockets), not for command processing. The actual data structure operations are still single-threaded. This shows you know the architecture deeply, not just the “single-threaded” headline.</p><pre># redis.conf — Redis 6+ I/O threading<br>io-threads 4          # 4 threads to read/write sockets<br>io-threads-do-reads yes<br># But ALL command execution is still:<br>single-threaded → no locks, no contention, no bugs<br># The bottleneck shifted: network I/O was the ceiling,<br># not command processing. So they threaded *that* part.</pre><h3>The Complete Interview Answer</h3><p>“Redis beats multi-threaded databases for three compounding reasons:</p><p><strong>First</strong>, all data lives in RAM — so there’s nothing slow to wait on. Multi-threading in databases largely exists to hide disk I/O latency; Redis doesn’t have that problem.</p><p><strong>Second</strong>, single-threaded means zero locking overhead. Multi-threaded databases pay a constant tax: mutexes, spinlocks, memory barriers, and cache coherence traffic between CPU cores. Redis pays none of that. One thread means one owner of the data — no synchronization needed.</p><p><strong>Third</strong>, Redis uses a non-blocking event loop with I/O multiplexing (epoll/kqueue). It never blocks waiting for a client — it only acts when I/O is ready. Thousands of connections are served by one tight loop with no context switching.</p><p><strong>The result</strong>: Redis executes ~100,000–1,000,000 ops/second with sub-millisecond latency. That’s not despite being single-threaded — it’s because of it. Threads solve the problem of waiting. Redis architecturally eliminated the waiting.”</p><h3>Bonus: Follow-Up Questions They’ll Ask</h3><h3>Q: When would Redis lose to a multi-threaded DB?</h3><p>When data doesn’t fit in RAM (Redis has memory limits), when you need complex queries with JOINs and aggregations (Redis isn’t a relational engine), or when you need ACID durability guarantees (Redis persistence is async by default).</p><h3>Q: Can Redis use multiple CPUs?</h3><p>Yes — by running multiple Redis instances (one per core) with consistent hashing across them, or using Redis Cluster. You get horizontal scaling without the concurrency headache.</p><h3>Q: Why doesn’t every database do this?</h3><p>Because most databases serve different workloads — large datasets on disk, complex SQL queries, transactions across many rows. Those genuinely require multi-threading. Redis is purpose-built for fast key-value lookups; the constraint enabled the optimization.</p><h3>Conclusion</h3><p><em>Redis wins because it eliminated the problem threads were invented to solve. No disk wait = no need for threads. No threads = no locks, no cache misses, no context switches. An event loop handles 10,000 concurrent connections on 1 thread cleanly. Single-threaded + in-memory + event loop = performance that multi-threading can’t touch for this use case.</em></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=16e3694c519c" width="1" height="1" alt="">]]></content:encoded>
        </item>
    </channel>
</rss>