<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Keval Rabadiya on Medium]]></title>
        <description><![CDATA[Stories by Keval Rabadiya on Medium]]></description>
        <link>https://medium.com/@kevalrabadiya27?source=rss-cfbc1dd2a8b1------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/1*nlPTdtRDiA6OAWcjX_o5zw.png</url>
            <title>Stories by Keval Rabadiya on Medium</title>
            <link>https://medium.com/@kevalrabadiya27?source=rss-cfbc1dd2a8b1------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Tue, 17 Nov 2026 12:38:20 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@kevalrabadiya27/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[I Moved 4TB From S3 to Backblaze B2. Here’s the Math That Convinced Me.]]></title>
            <link>https://medium.com/@kevalrabadiya27/i-moved-4tb-from-s3-to-backblaze-b2-heres-the-math-that-convinced-me-37fac6d5e477?source=rss-cfbc1dd2a8b1------2</link>
            <guid isPermaLink="false">https://medium.com/p/37fac6d5e477</guid>
            <category><![CDATA[aws]]></category>
            <category><![CDATA[cloud-infrastructure]]></category>
            <category><![CDATA[cost-optimization]]></category>
            <category><![CDATA[s3]]></category>
            <dc:creator><![CDATA[Keval Rabadiya]]></dc:creator>
            <pubDate>Thu, 25 Jun 2026 05:57:45 GMT</pubDate>
            <atom:updated>2026-06-25T05:57:45.508Z</atom:updated>
            <content:encoded><![CDATA[<h3>A developer’s honest breakdown of AWS S3 vs Backblaze B2 — and why egress, not storage, is the number that actually matters</h3><p>If you’ve shipped anything that stores user files — images, backups, exports, video — you already know the AWS S3 story. You reach for it because it’s the default. It’s reliable, it’s everywhere, every tutorial uses it, and for a while the bill looks fine.</p><p>Then your app gets traction. People start <em>downloading</em> what you stored. And one month you open the AWS console and the egress line item is bigger than everything else combined.</p><p>This is the part nobody warns you about when they recommend S3: <strong>storage is cheap, leaving is expensive.</strong> That asymmetry is the whole reason Backblaze B2 exists, and it’s why — for most workloads that aren’t deeply AWS-native already — it deserves a serious look before you reflexively spin up another bucket.</p><p>This isn’t a “everything AWS does is bad” post. It’s a cost-and-architecture breakdown for developers who want to know exactly where the money goes.</p><h3>The headline numbers</h3><p>Here’s what storing and serving data actually costs on each platform, using current published pricing:</p><p>AWS S3 (Standard) Backblaze B2 Storage $0.023/GB/mo (first 50TB) ~$0.006/GB/mo ($6/TB) Egress (download) $0.09/GB after first 100GB/mo free Free up to 3x stored data; free indefinitely via Cloudflare API Native S3 API S3-compatible API 100TB stored, modest egress ~$2,250/mo storage alone ~$600/mo storage alone</p><p>That storage gap — roughly 3.8x — is the number everyone quotes. It’s real, but it’s not the interesting number. The interesting number is egress, because that’s where S3 bills quietly spiral.</p><h3>Why egress is the real story</h3><p>Here’s a scenario that’s extremely common and extremely underestimated: you build something that lets users upload and then <em>re-download</em> files — photos, documents, video, whatever. Storage grows slowly. Downloads grow with usage, and usage is the thing you actually want to grow.</p><p>On S3, every gigabyte that leaves AWS to the public internet costs $0.09 after a small free allowance. That’s not a typo-level number on day one. It’s a compounding-resentment number by month six, when your “storage” service has quietly become a “bandwidth” service and bandwidth is priced like a premium product.</p><p>Backblaze’s model flips the incentive. Egress is free up to <strong>3x</strong> whatever you have stored — so if you’re storing 1TB, you can serve 3TB out per month before paying anything. And if you put Backblaze behind Cloudflare, egress to Cloudflare is free <em>indefinitely</em>, full stop, through the Bandwidth Alliance partnership the two companies run. For anything content-heavy — image hosting, video delivery, public datasets — that one detail can be the difference between a sustainable side project and a surprise four-figure invoice.</p><h3>“But is it actually S3, though?”</h3><p>This is the fair pushback, and the answer is: mostly, yes.</p><p>Backblaze B2 has shipped an S3-compatible API since 2018. In practice this means your existing boto3 code, your existing AWS CLI workflows, and most S3-aware tooling work against B2 with essentially one change — the endpoint.</p><pre>import boto3</pre><pre>s3 = boto3.resource(<br>    &#39;s3&#39;,<br>    endpoint_url=&#39;https://s3.us-west-002.backblazeb2.com&#39;,<br>    aws_access_key_id=&#39;&lt;b2_keyId&gt;&#39;,<br>    aws_secret_access_key=&#39;&lt;b2_appKey&gt;&#39;,<br>)</pre><pre>s3.create_bucket(Bucket=&#39;my-bucket&#39;)</pre><p>That’s it. Same SDK, same method calls, same mental model — just a different endpoint and credentials. If your code already isn’t doing anything AWS-proprietary (no Lambda triggers on bucket events, no tight IAM policy coupling, no Glacier lifecycle rules), migrating is often an afternoon of work, not a rewrite.</p><p>Where it’s <em>not</em> a perfect clone: B2 only supports v4 signatures, ACLs are limited to private/public-read (no complex policy documents), object tagging isn&#39;t fully supported, and there&#39;s no SSE-KMS. For 90% of &quot;store a file, serve a file&quot; use cases, none of this matters. If you&#39;re leaning on IAM-level fine-grained permissions or KMS-managed keys specifically, that&#39;s a real constraint to check before you migrate.</p><h3>Where S3 still wins, honestly</h3><p>I’m leaning toward Backblaze in this piece, but pretending S3 has no advantages would make this a worse post, not a better one. S3 is the right call when:</p><ul><li><strong>You’re already deep in AWS</strong> — Lambda triggers on object creation, CloudFront distributions, tight IAM integration. The cost of <em>leaving</em> that ecosystem can exceed the cost of the storage bill itself.</li><li><strong>You need real storage tiering.</strong> S3 has Standard, Infrequent Access, Glacier, and Deep Archive, with lifecycle policies that automatically age data down to fractions of a cent per GB. B2 has one tier. If you have genuinely cold compliance archives sitting untouched for years, S3’s tiering can beat B2 on total cost.</li><li><strong>You need specific compliance certifications</strong> at a scale where AWS’s enterprise paper trail matters more than the bill.</li><li><strong>Ecosystem breadth matters more than price</strong> — more SDKs, more third-party integrations, more “it just works with this random tool” coverage.</li></ul><p>None of these are nothing. They’re just narrower than “default to S3” makes them feel.</p><h3>Where Backblaze wins</h3><ul><li><strong>Storage cost</strong>, straightforwardly — roughly a third of S3 Standard at equivalent volume.</li><li><strong>Egress-heavy workloads</strong> — anything where people download more than they upload. This is the category most indie projects, content sites, and backup tools fall into.</li><li><strong>Predictability.</strong> One tier, one rate, free egress up to 3x stored. You can forecast a B2 bill on a napkin. S3 bills require a spreadsheet and a prayer.</li><li><strong>Backup and archive use cases</strong> specifically — data that’s written once and rarely touched is exactly what B2’s pricing model rewards.</li></ul><h3>The actual decision</h3><p>If you’re already running compute on AWS and need tight service integration or real cold-storage tiering, stay on S3 — the egress savings elsewhere won’t offset the architectural cost of leaving.</p><p>For almost everything else — and especially for anything where users download more than they upload — run the numbers on B2 before you assume S3 is the only serious option. The API is close enough that switching isn’t a rewrite, and the egress model alone has saved real projects from the kind of bill that makes you reconsider the whole product.</p><p>The honest version of this advice: don’t pick a storage provider because it’s the default. Pick it because you did the arithmetic for <em>your</em> access pattern — and for most developers outside the AWS ecosystem, that arithmetic points at Backblaze.</p><p><em>Pricing referenced is current as of mid-2026 for standard US regions and excludes request fees, minimum retention charges, and regional variation — always check current rates before making an infrastructure decision based on a blog post.</em></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=37fac6d5e477" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Golem Network: Decentralizing Compute Power with Blockchain]]></title>
            <link>https://medium.com/@kevalrabadiya27/golem-network-decentralizing-compute-power-with-blockchain-8339af944e66?source=rss-cfbc1dd2a8b1------2</link>
            <guid isPermaLink="false">https://medium.com/p/8339af944e66</guid>
            <category><![CDATA[cloud-computing]]></category>
            <category><![CDATA[decentralization]]></category>
            <category><![CDATA[blockchain]]></category>
            <category><![CDATA[aws]]></category>
            <category><![CDATA[golem]]></category>
            <dc:creator><![CDATA[Keval Rabadiya]]></dc:creator>
            <pubDate>Mon, 23 Mar 2026 06:16:43 GMT</pubDate>
            <atom:updated>2026-03-23T06:16:43.453Z</atom:updated>
            <content:encoded><![CDATA[<h3>Introduction</h3><p>In today’s digital world, computing power is one of the most valuable resources. From rendering graphics and training AI models to running complex simulations, the demand for compute resources continues to grow rapidly. However, most of this power is controlled by centralized cloud providers, making it expensive and sometimes inaccessible.</p><p>This is where <strong>Golem Network</strong> comes in.</p><p>Golem Network is a decentralized marketplace for computing power, built on blockchain technology. It allows users to <strong>rent out their unused computing resources</strong> or <strong>buy compute power on demand</strong>, creating a peer-to-peer ecosystem that is more efficient, flexible, and cost-effective.</p><h3>What is Golem Network?</h3><p>Golem Network is a blockchain-based platform that enables users to share and access computing resources in a decentralized manner.</p><p>Instead of relying on centralized services like AWS or Google Cloud, Golem connects users directly:</p><ul><li><strong>Providers</strong> → Users who share their idle computing power</li><li><strong>Requestors</strong> → Users who need computing resources</li></ul><p>These interactions are powered by blockchain, ensuring <strong>secure, transparent, and trustless transactions</strong>.</p><h3>How Golem Network Works</h3><p>The working of Golem can be understood in a few simple steps:</p><ol><li>A requestor submits a task (e.g., rendering, AI processing)</li><li>The task is split into smaller subtasks</li><li>These subtasks are distributed across multiple providers</li><li>Providers process the tasks using their machines</li><li>Results are returned and combined</li><li>Payment is made using Golem’s native token</li></ol><p>This distributed approach allows Golem to utilize <strong>global idle resources</strong>, making computation faster and more scalable.</p><h3>Key Features</h3><h3>1. Decentralized Computing</h3><p>Golem removes the need for centralized cloud providers. Anyone with a computer can participate in the network.</p><h3>2. Cost Efficiency</h3><p>Since resources come from individuals, costs are often lower compared to traditional cloud platforms.</p><h3>3. Scalability</h3><p>The network grows as more users join, increasing available computing power dynamically.</p><h3>4. Flexibility</h3><p>Golem supports various workloads such as:</p><ul><li>CGI rendering</li><li>Machine learning tasks</li><li>Scientific computations</li></ul><h3>5. Security &amp; Transparency</h3><p>Blockchain ensures that all transactions are:</p><ul><li>Verifiable</li><li>Tamper-proof</li><li>Transparent</li></ul><h3>Use Cases</h3><h3>🎬 Rendering (CGI &amp; Animation)</h3><p>Artists can render high-quality graphics faster by distributing workloads across multiple machines.</p><h3>🤖 Artificial Intelligence</h3><p>AI developers can train models using distributed compute resources at a lower cost.</p><h3>🔬 Scientific Research</h3><p>Researchers can run simulations without investing in expensive infrastructure.</p><h3>🧠 Data Processing</h3><p>Large-scale data analysis can be performed efficiently using distributed computing.</p><h3>Token Economy (GLM)</h3><p>Golem uses its native token called <strong>GLM (Golem Network Token)</strong>.</p><ul><li>Requestors pay providers using GLM</li><li>Providers earn GLM by contributing resources</li><li>Acts as an incentive mechanism for network participation</li></ul><p>This token-based economy ensures that the network remains <strong>self-sustaining</strong>.</p><h3>Advantages of Golem Network</h3><ul><li>✅ Decentralized and trustless</li><li>✅ Lower cost compared to cloud providers</li><li>✅ Efficient use of idle resources</li><li>✅ Open and permissionless network</li></ul><h3>Challenges &amp; Limitations</h3><p>Despite its potential, Golem also faces some challenges:</p><ul><li>❌ Adoption is still growing</li><li>❌ Performance may vary depending on providers</li><li>❌ Not yet as mature as centralized cloud platforms</li><li>❌ Requires technical understanding for setup</li></ul><h3>Future of Golem Network</h3><p>The future of Golem looks promising as the demand for decentralized infrastructure increases.</p><p>With the rise of:</p><ul><li>Web3</li><li>AI workloads</li><li>Distributed systems</li></ul><p>Golem has the potential to become a <strong>key player in decentralized cloud computing</strong>.</p><p>As more developers and enterprises adopt blockchain-based solutions, Golem could evolve into a strong alternative to traditional cloud services.</p><h3>Conclusion</h3><p>Golem Network represents a powerful idea: <strong>turning the world’s unused computing power into a global decentralized supercomputer</strong>.</p><p>While it is still evolving, its core concept addresses major issues in traditional cloud computing — cost, centralization, and accessibility.</p><p>For developers, researchers, and businesses, Golem offers an innovative way to access compute resources efficiently, while also enabling individuals to monetize their idle hardware.</p><p>👉 In simple terms:<br><strong>Golem is building the future of decentralized computing — one node at a time.</strong></p><h3>Final Thought</h3><p>If blockchain is the future of finance, platforms like Golem could be the future of <strong>computing infrastructure</strong>.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=8339af944e66" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[How Trading View Infrastructure Works Behind the Scenes]]></title>
            <link>https://medium.com/@kevalrabadiya27/how-trending-view-infrastructure-works-behind-the-scenes-53ed1229c43c?source=rss-cfbc1dd2a8b1------2</link>
            <guid isPermaLink="false">https://medium.com/p/53ed1229c43c</guid>
            <category><![CDATA[distributed-systems]]></category>
            <category><![CDATA[tradingview]]></category>
            <category><![CDATA[real-time-analytics]]></category>
            <category><![CDATA[systems-thinking]]></category>
            <category><![CDATA[trading]]></category>
            <dc:creator><![CDATA[Keval Rabadiya]]></dc:creator>
            <pubDate>Fri, 16 Jan 2026 19:03:03 GMT</pubDate>
            <atom:updated>2026-03-23T06:21:02.279Z</atom:updated>
            <content:encoded><![CDATA[<h3>(Data, Scale, Speed, and Money Flow)</h3><p>When you open a trading app and see <strong>“Trending Stocks”</strong>, <strong>“Hot Coins”</strong>, or <strong>“Top Movers”</strong>, you are seeing the output of a <strong>high-performance real-time system</strong>.</p><p>Behind that simple UI lies:</p><ul><li>Streaming data pipelines</li><li>Real-time computation</li><li>Ranking engines</li><li>Fault-tolerant infrastructure</li><li>Millisecond-level decision systems</li></ul><p>Let’s break down <strong>how trending view systems work behind the scenes</strong>.</p><h3>1️⃣ High-Level System Architecture</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/720/0*4XtSdAqxeYjqNkWk.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1000/0*yWNWI7VP_IVRPddq.png" /></figure><p>At a high level:</p><pre>Market Data Sources<br>        ↓<br>Streaming Ingestion Layer<br>        ↓<br>Real-Time Processing Engine<br>        ↓<br>Trend &amp; Money Flow Scoring Engine<br>        ↓<br>Cache + Database<br>        ↓<br>API Layer<br>        ↓<br>Frontend (Trending View)</pre><p>Each layer is optimized for <strong>speed, accuracy, and scale</strong>.</p><h3>2️⃣ Market Data Ingestion (The Firehose)</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/936/0*N3-bY9s6gQ3FvAK1" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*Tiv6oBOpuTaIJQ_P.jpg" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*rFPFDKLqLEPRbTdb.png" /></figure><h3>Data Sources</h3><ul><li>Stock exchanges (NSE, NYSE, NASDAQ)</li><li>Crypto exchanges (Binance, Coinbase)</li><li>Order book feeds</li><li>Trade ticks</li><li>Volume &amp; depth data</li></ul><h3>Infra Reality</h3><ul><li>Data arrives via:</li><li>WebSockets</li><li>FIX protocol</li><li>TCP multicast</li><li>Millions of events per second</li></ul><p>👉 This layer must <strong>never block</strong> or slow down.</p><h3>3️⃣ Streaming Layer (Real-Time Backbone)</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*WUoNgk5dYwHb_ss9.jpg" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/680/0*zeSs6hokvc6lTQ26" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/671/0*1ro29XWXodVfd51N.png" /></figure><p>Raw market data is pushed into a <strong>streaming platform</strong>:</p><p>Common choices:</p><ul><li>Kafka</li><li>Pulsar</li><li>Kinesis</li></ul><h3>Why streaming?</h3><p>✔ Decouples producers &amp; consumers<br> ✔ Handles bursts safely<br> ✔ Replays historical data<br> ✔ Horizontal scaling</p><p>Each event looks like:</p><pre>{<br>  &quot;symbol&quot;: &quot;ABC&quot;,<br>  &quot;price&quot;: 102.5,<br>  &quot;volume&quot;: 12000,<br>  &quot;timestamp&quot;: 1700000000<br>}</pre><h3>4️⃣ Real-Time Processing Engine</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/725/0*0aqNq4A5U_TV0a7N.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/333/0*Btp9A2TDkrlAHtOp" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*h7sC0CZKhQBcvefk" /></figure><p>This is where <strong>trending logic begins</strong>.</p><h3>Responsibilities:</h3><ul><li>Aggregate ticks into:</li><li>1m / 5m / 15m candles</li><li>Calculate:</li><li>Price change %</li><li>Volume delta</li><li>Volatility</li><li>Detect breakouts &amp; momentum</li></ul><h3>Tools used:</h3><ul><li>Apache Flink</li><li>Spark Streaming</li><li>Custom in-memory engines</li></ul><p>⚠️ Latency target: <strong>&lt; 100 ms</strong></p><h3>5️⃣ Acute Trend Detection Engine (Early Signals)</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*x67JJw6BICo6FOwN.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/869/0*ODACFI5tYwhpSi5e.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/932/0*DkmLWWi3XwUYKG4J" /></figure><p><strong>Acute trends</strong> = early explosive moves.</p><h3>Infra logic:</h3><ul><li>Sliding time windows</li><li>Z-score / deviation analysis</li><li>Sudden volume expansion detection</li><li>Momentum acceleration checks</li></ul><p>Example logic:</p><pre>If:<br> volume(last 1m) &gt; 3x avg volume<br> AND price breaks resistance<br> AND volatility expands<br>→ Acute Trend Flag</pre><p>These assets are marked <strong>high-risk / high-reward</strong> internally.</p><h3>6️⃣ Money Flow &amp; Capital Tracking Layer</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/698/0*TXCeCSYupw7AHhrW.jpg" /></figure><p>This layer answers:</p><blockquote><em>“Is real money entering or leaving?”</em></blockquote><h3>What is tracked:</h3><ul><li>Buy vs sell volume</li><li>Order imbalance</li><li>Accumulation vs distribution</li><li>Net capital inflow</li></ul><h3>Why infra is complex:</h3><ul><li>Requires <strong>order-level data</strong></li><li>Needs micro-batch aggregation</li><li>Sensitive to latency</li></ul><p>👉 Trending view <strong>prioritizes money flow over price spikes</strong></p><h3>7️⃣ Trend Scoring &amp; Ranking Engine</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*XCSNIJUCqdfBfcGN.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/800/0*5OWpYbPB6Evaghks" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*YmBnDAf8jNQ4l1bL.png" /></figure><p>Every asset gets a <strong>Trend Score</strong>.</p><h3>Typical inputs:</h3><ul><li>Price momentum</li><li>Volume strength</li><li>Money flow</li><li>Timeframe alignment</li><li>Risk volatility</li></ul><h3>Output:</h3><pre>{<br>  &quot;symbol&quot;: &quot;ABC&quot;,<br>  &quot;trendScore&quot;: 87.4,<br>  &quot;trendType&quot;: &quot;acute&quot;,<br>  &quot;direction&quot;: &quot;bullish&quot;<br>}</pre><p>Assets are sorted by score → <strong>Trending View list</strong>.</p><h3>8️⃣ Cache Layer (Speed Is Everything)</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*ZsMUCwkL1sr56Bzh.jpeg" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*wE_e6u8-DUK6B3Fo.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/614/0*vVVJp178N84oVSsj.png" /></figure><p>Trending data is <strong>read-heavy</strong>.</p><h3>Why caching is critical:</h3><ul><li>Frontend refreshes every few seconds</li><li>Millions of users</li><li>Real-time updates</li></ul><h3>Common setup:</h3><ul><li>Redis</li><li>In-memory LRU caches</li></ul><p>🔥 Reads must be <strong>&lt; 5 ms</strong></p><h3>9️⃣ Database Layer (History &amp; Analytics)</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/520/0*BYHi3NV4jcenccYp.gif" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*BLlSUfj685Yb2mjm.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/600/0*SE4gsdQrHBjM2bP7.jpg" /></figure><p>Used for:</p><ul><li>Historical trend analysis</li><li>Backtesting</li><li>User insights</li></ul><p>Databases:</p><ul><li>Time-series DBs (ClickHouse, Influx)</li><li>Cold storage (S3, HDFS)</li></ul><p>👉 Real-time systems <strong>never query DB directly</strong></p><h3>🔟 API &amp; Frontend Delivery</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*qxequHmFnNNOUovW.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*gQuTdcYBMAJolxTD.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*O48cHK2a4JUBMLIO.png" /></figure><h3>APIs:</h3><ul><li>REST for snapshots</li><li>WebSocket for live updates</li></ul><h3>Frontend:</h3><ul><li>Polls or subscribes</li><li>Shows:</li><li>Trending rank</li><li>Direction</li><li>Strength</li><li>Risk tag</li></ul><p>Trending view refreshes <strong>without page reload</strong>.</p><h3>1️⃣1️⃣ Reliability, Scale &amp; Safety</h3><h3>Infra must handle:</h3><ul><li>Market open spikes</li><li>Sudden crashes</li><li>Data gaps</li><li>Exchange outages</li></ul><h3>Safety mechanisms:</h3><ul><li>Circuit breakers</li><li>Data validation</li><li>Fallback feeds</li><li>Rate limiting</li></ul><p>❌ Wrong trend = financial loss<br> ✔ Accuracy &gt; speed (but both matter)</p><h3>🧠 Final Mental Model</h3><blockquote><strong><em>Trending View is a real-time ranking system built on streaming data, money flow analysis, and low-latency infrastructure.</em></strong></blockquote><p>It is <strong>not just charts</strong> — it’s:</p><ul><li>Distributed systems</li><li>Real-time analytics</li><li>Financial risk control</li><li>Scalable infra engineering</li></ul><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=53ed1229c43c" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Fan-out Architectures in System Design: Scalability Through Smart Distribution]]></title>
            <link>https://medium.com/@kevalrabadiya27/fan-out-architectures-in-system-design-scalability-through-smart-distribution-aff650202378?source=rss-cfbc1dd2a8b1------2</link>
            <guid isPermaLink="false">https://medium.com/p/aff650202378</guid>
            <category><![CDATA[system-design-concepts]]></category>
            <category><![CDATA[aws]]></category>
            <category><![CDATA[backend-development]]></category>
            <dc:creator><![CDATA[Keval Rabadiya]]></dc:creator>
            <pubDate>Sat, 28 Jun 2025 21:10:50 GMT</pubDate>
            <atom:updated>2025-06-28T21:13:49.581Z</atom:updated>
            <content:encoded><![CDATA[<p>As systems scale and user demands grow, developers often face a critical challenge: <strong>how to handle a single event that must trigger multiple downstream processes simultaneously</strong>. This is where <strong>fan-out architectures</strong> step in.</p><p>In this blog, we’ll explore what fan-out means in system design, why it’s important, common implementation strategies (like queues and serverless), and real-world use cases that show how fan-out helps scale modern applications.</p><h3>🚀 What is Fan-out in System Design?</h3><p><strong>Fan-out</strong> refers to a design pattern where a <strong>single input or event leads to multiple outputs or actions</strong>.</p><p>In simpler terms:</p><blockquote><em>One request → Many tasks triggered in parallel.</em></blockquote><p>It’s the opposite of fan-in, where multiple sources feed into a single destination.</p><p><strong>Example:</strong><br> When a user uploads a photo to Instagram, the platform might:</p><ul><li>Generate thumbnails</li><li>Notify followers</li><li>Extract metadata</li><li>Push to search indexing</li><li>Store to CDN<br> …all triggered by that single upload. This is fan-out in action.</li></ul><h3>🧱 Why Fan-out Matters</h3><p>Fan-out solves a few critical problems in modern distributed systems:</p><ul><li><strong>Parallelism</strong>: Execute tasks simultaneously to reduce latency.</li><li><strong>Scalability</strong>: Each downstream task can scale independently.</li><li><strong>Decoupling</strong>: Systems become modular, easier to test, maintain, and evolve.</li><li><strong>Reliability</strong>: Failures in one part don’t necessarily affect others.</li></ul><h3>🛠️ How to Implement Fan-out</h3><p>There are multiple ways to implement fan-out architectures, depending on the system’s complexity and scale. Here are some of the most common:</p><h3>1. Message Queues (Pub/Sub)</h3><p>Use messaging systems like:</p><ul><li><strong>Amazon SNS/SQS</strong></li><li><strong>Google Pub/Sub</strong></li><li><strong>RabbitMQ</strong></li><li><strong>Kafka</strong></li></ul><p>A publisher sends a message to a topic, and multiple subscribers (or queues) receive and process it independently.</p><p><strong>Pros</strong>:</p><ul><li>Loose coupling between services</li><li>Good for event-driven architectures</li></ul><p><strong>Cons</strong>:</p><ul><li>Delivery ordering and guarantees vary</li><li>Requires proper error handling &amp; dead-letter queues</li></ul><h3>2. Serverless Fan-out (AWS Lambda, Azure Functions)</h3><p>With services like AWS Lambda:</p><ul><li>Trigger a Lambda function on an event (e.g., S3 upload)</li><li>This Lambda then asynchronously invokes multiple other Lambdas or services</li></ul><p>Or use <strong>AWS Step Functions</strong> to orchestrate the flow.</p><p><strong>Pros</strong>:</p><ul><li>Auto-scaling and cost-effective</li><li>Easy to manage concurrency</li></ul><p><strong>Cons</strong>:</p><ul><li>Timeout constraints</li><li>Cold starts in some environments</li></ul><h3>3. Worker Pools with Task Queues</h3><p>Classic approach using:</p><ul><li>Task queue (e.g., Celery with Redis/RabbitMQ)</li><li>Worker processes consume and execute tasks</li></ul><p><strong>Pros</strong>:</p><ul><li>Full control over task processing</li><li>Can retry or schedule jobs</li></ul><p><strong>Cons</strong>:</p><ul><li>Needs more infrastructure</li><li>Scaling workers manually or with orchestration (K8s, ECS)</li></ul><h3>4. Stream Processing (Kafka, Kinesis)</h3><p>Fan-out can also be implemented via <strong>streams</strong>, where data flows through a pipeline and multiple consumers read and react independently.</p><p>Best for:</p><ul><li>Real-time analytics</li><li>ETL pipelines</li><li>Continuous data enrichment</li></ul><h3>⚙️ Real-World Use Cases</h3><p>Use Case Fan-out Example Social Media On post: notify followers, update feed, index post E-commerce On order: send confirmation, update inventory, log analytics, trigger shipment Video Upload Transcode video, extract audio, notify user, update catalog IoT Sensor event: alerting, logging, machine learning, dashboard updates</p><h3>🧩 Fan-out vs Eventual Consistency</h3><p>One key tradeoff in fan-out designs is <strong>eventual consistency</strong>. Since tasks are processed asynchronously, not all components may update at the same time.</p><p><strong>Design Tip:</strong><br> Use <strong>idempotent functions</strong> and <strong>retry-safe</strong> operations to ensure reliable and consistent fan-out execution.</p><h3>🧠 Best Practices</h3><ul><li>Use <strong>correlation IDs</strong> to trace a single request across multiple systems</li><li>Monitor fan-out paths individually (e.g., CloudWatch, OpenTelemetry)</li><li>Implement retries, DLQs, and alerts</li><li>Understand limits on concurrency and throttling.</li><li>Consider <strong>backpressure</strong> to avoid overwhelming downstream services</li></ul><h3>📌 Final Thoughts</h3><p>Fan-out is a powerful architectural pattern that helps systems scale, evolve, and respond to user actions in parallel. Whether you’re building a microservices backend, a serverless workflow, or an event-driven platform — mastering fan-out design is essential.</p><p>Next time you’re building a feature that needs to “do a bunch of things when X happens,” consider fan-out architecture. It could be the key to making your system fast, scalable, and resilient.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=aff650202378" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[ Deploying a Node.js App on AWS with PM2, NGINX, and SSL]]></title>
            <link>https://medium.com/@kevalrabadiya27/deploying-a-node-js-app-on-aws-with-pm2-nginx-and-ssl-35498c9de7b1?source=rss-cfbc1dd2a8b1------2</link>
            <guid isPermaLink="false">https://medium.com/p/35498c9de7b1</guid>
            <category><![CDATA[pm2]]></category>
            <category><![CDATA[firebase]]></category>
            <category><![CDATA[nginx]]></category>
            <category><![CDATA[aws]]></category>
            <category><![CDATA[backend]]></category>
            <dc:creator><![CDATA[Keval Rabadiya]]></dc:creator>
            <pubDate>Tue, 25 Feb 2025 16:01:58 GMT</pubDate>
            <atom:updated>2025-02-25T16:01:58.498Z</atom:updated>
            <content:encoded><![CDATA[<h3>Introduction</h3><p>Deploying a Node.js application efficiently requires a reliable process. In this guide, we’ll walk through deploying a Node.js app to <strong>DigitalOcean</strong>, using <strong>PM2</strong> for process management, <strong>NGINX</strong> as a reverse proxy, and securing it with <strong>Let’s Encrypt SSL</strong>.</p><h3>💻 Step 1: Create and Launch an EC2 Instance</h3><p>For this tutorial, we will use an <strong>Ubuntu 20.04</strong> instance. If you’re using <strong>DigitalOcean</strong>, create a <strong>Droplet</strong> with at least <strong>1GB RAM (preferably 2GB for better performance)</strong>.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/260/1*wvgcx6IdsRwfJHzYq48Q5Q.jpeg" /></figure><pre>ssh root@your-server-ip</pre><p><em>If using AWS, use the PEM key:</em></p><pre>ssh -i your-key.pem ubuntu@your-server-ip</pre><h3>📦 Step 2: Install Node.js and NPM</h3><p>Ensure Node.js and NPM are installed on your server:</p><pre>curl -sL https://deb.nodesource.com/setup_18.x | sudo -E bash -<br>sudo apt install -y nodejs</pre><p>Verify installation:</p><pre>node --version<br>npm --version</pre><h3>🔗 Step 3: Clone Your Project from GitHub</h3><p>Navigate to the directory where you want to deploy your app and clone the repository:</p><pre>git clone https://github.com/piyushgargdev-01/short-url-nodejs<br>cd short-url-nodejs</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/235/1*JRepaNw5nhGKahIFlrpRww.jpeg" /></figure><h3>⚙️ Step 4: Install Dependencies and Setup PM2</h3><p>We will use <strong>PM2</strong> to manage our application and ensure it runs continuously:</p><pre>sudo npm install -g pm2<br>pm2 start index.js  # Start your app</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/417/1*JE8wO9pgG2fwkNPN1Ezc7A.jpeg" /></figure><p>Other useful PM2 commands:</p><pre>pm2 show index     # View app details  <br>pm2 status         # Check running apps  <br>pm2 restart index  # Restart app  <br>pm2 stop index     # Stop app  <br>pm2 logs           # Show logs  <br>pm2 flush          # Clear logs</pre><p>To keep your app running after a reboot:</p><pre>pm2 startup ubuntu<br>pm2 save</pre><h3>🔥 Step 5: Setup Firewall (UFW) for Security</h3><p>Enable <strong>UFW (Uncomplicated Firewall)</strong> to prevent unauthorized access:</p><pre>sudo ufw enable<br>sudo ufw status<br>sudo ufw allow ssh     # Allow SSH (Port 22)<br>sudo ufw allow http    # Allow HTTP (Port 80)<br>sudo ufw allow https   # Allow HTTPS (Port 443)</pre><h3>🌐 Step 6: Install and Configure NGINX</h3><p>Install <strong>NGINX</strong> as a reverse proxy:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/347/1*1CnGkpgL_jjJubZV0bNSAQ.png" /></figure><pre>sudo apt install nginx</pre><p>Open the <strong>NGINX configuration file</strong>:</p><pre>sudo nano /etc/nginx/sites-available/default</pre><p>Modify the server block:</p><pre>server {<br>    listen 80;<br>    server_name yourdomain.com www.yourdomain.com;<br><br>    location / {<br>        proxy_pass http://localhost:8001;  # Change to your app’s port<br>        proxy_http_version 1.1;<br>        proxy_set_header Upgrade $http_upgrade;<br>        proxy_set_header Connection &#39;upgrade&#39;;<br>        proxy_set_header Host $host;<br>        proxy_cache_bypass $http_upgrade;<br>    }<br>}<br><br></pre><p>Save and test your configuration:</p><pre>sudo nginx -t</pre><p>If there are no errors, restart NGINX:</p><pre>sudo nginx -s reload</pre><h3>🔒 Step 7: Secure Your App with Let’s Encrypt SSL</h3><p>To encrypt traffic, install <strong>Certbot</strong> and obtain a <strong>free SSL certificate</strong>:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/318/1*Fc2EsnoCccgsYq2nXv2_-A.jpeg" /></figure><pre>sudo add-apt-repository ppa:certbot/certbot<br>sudo apt-get update<br>sudo apt-get install python3-certbot-nginx</pre><p>Now, generate the SSL certificate for your domain:</p><pre>sudo certbot --nginx -d yourdomain.com -d www.yourdomain.com</pre><p>Test auto-renewal (certificates expire every 90 days):</p><pre>certbot renew --dry-run</pre><h3>🎉 Deployment Complete!</h3><p>Your Node.js app is now live, secure, and running efficiently on AWS! 🎊</p><h3>🚀 Quick Recap</h3><p>✅ Setup a cloud instance (EC2/Droplet)<br>✅ Installed Node.js and cloned the project<br>✅ Used PM2 to manage the Node.js process<br>✅ Configured NGINX as a reverse proxy<br>✅ Secured the app with Let’s Encrypt SSL</p><h3>💡 Next Steps</h3><p>Want to improve performance and security further? Consider:</p><ul><li><strong>Using a Load Balancer</strong> for scalability</li><li><strong>Monitoring logs with PM2 &amp; Logrotate</strong></li><li><strong>Setting up a CI/CD pipeline</strong> for automatic deployments</li></ul><p>💬 Have any questions or issues? Drop them in the comments below! 🚀</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=35498c9de7b1" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Understanding Node.js Process Events: A Comprehensive Guide]]></title>
            <link>https://medium.com/@kevalrabadiya27/understanding-node-js-process-events-a-comprehensive-guide-7da15d11cc3d?source=rss-cfbc1dd2a8b1------2</link>
            <guid isPermaLink="false">https://medium.com/p/7da15d11cc3d</guid>
            <category><![CDATA[events]]></category>
            <category><![CDATA[backend]]></category>
            <category><![CDATA[web-development]]></category>
            <category><![CDATA[nodejs]]></category>
            <category><![CDATA[process]]></category>
            <dc:creator><![CDATA[Keval Rabadiya]]></dc:creator>
            <pubDate>Tue, 10 Sep 2024 05:25:01 GMT</pubDate>
            <atom:updated>2024-09-10T05:25:01.724Z</atom:updated>
            <content:encoded><![CDATA[<blockquote>Node.js is widely known for its non-blocking, event-driven architecture. This unique approach allows Node.js to handle numerous concurrent connections efficiently. Among the many features that power this model, one stands out: the process module. If you’re looking to become a proficient Node.js developer, understanding the process module and its events is crucial.</blockquote><h4>What is the process Module?</h4><p>The process module in Node.js provides information and control over the current Node.js process. It&#39;s a global module, meaning you don’t need to require it in your scripts—Node.js makes it available everywhere.</p><p>The process object is essential for interacting with the environment in which a script is being executed. It allows developers to manage input and output, interact with the operating system, handle environment variables, and more.</p><h4>Key Process Events</h4><p>Node.js provides several built-in process events. These events are emitted by the process object itself. Let’s explore the most important ones:</p><p><strong>exit</strong></p><ul><li>The exit event is emitted when the Node.js process is about to exit. It is the last event emitted by the process, and after this event, the event loop stops processing any further events.</li></ul><pre>process.on(&#39;exit&#39;, (code) =&gt; {<br>  console.log(`Process exited with code: ${code}`);<br>});</pre><ul><li>This can be particularly useful for cleanup tasks or logging.</li></ul><p><strong>uncaughtException</strong></p><ul><li>Emitted when an uncaught JavaScript exception is thrown and not handled. If you don’t handle exceptions properly, it could crash your entire application.</li></ul><pre>process.on(&#39;uncaughtException&#39;, (err) =&gt; {<br>  console.error(&#39;There was an uncaught error&#39;, err);<br>  // Do some cleanup and exit gracefully<br>  process.exit(1);<br>});</pre><p><strong>SIGINT</strong></p><ul><li>Emitted when the process receives a SIGINT signal (usually when you press Ctrl+C). You can listen for this event to perform cleanup before shutting down your application.</li></ul><pre>process.on(&#39;SIGINT&#39;, () =&gt; {<br>  console.log(&#39;Received SIGINT. Gracefully shutting down...&#39;);<br>  process.exit();<br>});</pre><p><strong>beforeExit</strong></p><ul><li>Emitted when Node.js empties its event loop and has no additional work scheduled, but is not yet exiting. It can be used to schedule more work asynchronously.</li></ul><pre>process.on(&#39;beforeExit&#39;, (code) =&gt; {<br>  console.log(`Process will exit soon with code: ${code}`);<br>});</pre><p>The beforeExit event is a good place to perform final asynchronous operations.</p><p><strong>warning</strong></p><ul><li>Emitted when a warning is generated. Warnings are often non-critical issues but should still be addressed to improve the codebase.</li></ul><pre>process.on(&#39;warning&#39;, (warning) =&gt; {<br>  console.warn(warning.name);    // Print warning name<br>  console.warn(warning.message); // Print warning message<br>  console.warn(warning.stack);   // Print stack trace<br>});</pre><h4>Why Should You Care About Process Events?</h4><p>Understanding process events allows you to:</p><ul><li><strong>Gracefully handle shutdowns</strong>: Clean up resources, close connections, and save state before exiting.</li><li><strong>Error handling</strong>: Capture unhandled exceptions or rejected promises to prevent crashes.</li><li><strong>Optimize performance</strong>: Use beforeExit to perform any last-minute operations or logging.</li><li><strong>Security</strong>: Handle signals and errors to avoid unexpected behavior in production.</li></ul><h4>Practical Tips for Using Process Events</h4><ol><li><strong>Always Handle </strong><strong>uncaughtException with Caution</strong>: It should be a last resort. Ideally, you should manage errors in your application logic rather than relying on this event.</li><li><strong>Graceful Shutdowns</strong>: Listening to events like SIGINT or SIGTERM can help you implement graceful shutdowns, a critical aspect of building resilient applications.</li><li><strong>Monitor Warnings</strong>: Pay attention to warnings emitted by the process object. They are not errors but can indicate potential issues in your codebase.</li></ol><h4>Conclusion</h4><p>Node.js process events provide a powerful way to interact with the environment in which your code is running. Whether handling unexpected errors, capturing signals, or optimizing your application’s performance, these events offer essential tools for managing your Node.js applications effectively.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=7da15d11cc3d" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Secure Image Uploads to Amazon S3 with Presigned URLs: A Step-by-Step Guide]]></title>
            <link>https://medium.com/@kevalrabadiya27/secure-image-uploads-to-amazon-s3-with-presigned-urls-a-step-by-step-guide-2100ad0f1956?source=rss-cfbc1dd2a8b1------2</link>
            <guid isPermaLink="false">https://medium.com/p/2100ad0f1956</guid>
            <category><![CDATA[node]]></category>
            <category><![CDATA[aws]]></category>
            <category><![CDATA[s3-bucket]]></category>
            <category><![CDATA[image-upload]]></category>
            <category><![CDATA[aws-s3]]></category>
            <dc:creator><![CDATA[Keval Rabadiya]]></dc:creator>
            <pubDate>Mon, 09 Sep 2024 09:53:48 GMT</pubDate>
            <atom:updated>2024-09-09T09:54:27.566Z</atom:updated>
            <content:encoded><![CDATA[<p>In this blog, we will explore how to securely upload images to Amazon S3 using presigned URLs. This method allows clients to directly upload files to S3 without exposing your server credentials or making your server handle the file upload, resulting in faster and more secure file transfers.</p><h4>1. Introduction to Amazon S3 and Presigned URLs</h4><ul><li><strong>Amazon S3</strong>: Amazon Simple Storage Service (S3) is a scalable object storage service that allows you to store and retrieve any amount of data at any time.</li><li><strong>Presigned URLs</strong>: A presigned URL gives a user temporary access to upload or download an object to/from an S3 bucket. The URL is created by your server and can be shared with the client, allowing direct uploads to S3.</li></ul><h4>2. Why Use Presigned URLs for Uploads?</h4><ul><li><strong>Security</strong>: Limits access to your S3 bucket by providing a time-bound URL for uploads.</li><li><strong>Performance</strong>: Allows clients to directly upload to S3, reducing the load on your server.</li><li><strong>Flexibility</strong>: You can control access by specifying permissions and expiration times.</li></ul><h4>3. Prerequisites</h4><p>Before getting started, make sure you have:</p><ul><li>An AWS account with S3 configured.</li><li>Node.js installed on your machine.</li><li>AWS SDK for JavaScript (<a href="http://twitter.com/aws">@aws</a>-sdk/client-s3) installed.</li><li>A basic understanding of Node.js and Express.js (if you’re using a backend server).</li></ul><h4>4. Setting Up Amazon S3 Bucket</h4><ul><li><strong>Step 1</strong>: Log in to the <a href="https://aws.amazon.com/">AWS Management Console</a>.</li><li><strong>Step 2</strong>: Navigate to <strong>S3</strong> and click on <strong>“Create Bucket.”</strong></li><li><strong>Step 3</strong>: Name your bucket (e.g., my-image-uploads) and choose a region.</li><li><strong>Step 4</strong>: Configure settings like versioning, logging, and permissions as required.</li><li><strong>Step 5</strong>: Click <strong>“Create Bucket.”</strong></li></ul><h4>5. Configuring <a href="http://twitter.com/aws">aws</a>-sdk/client-s3 in Node.js</h4><ul><li>Install the AWS SDK/CLIENT-S3:</li></ul><pre>npm install @aws-sdk/client-s3<br>npm install @aws-sdk/s3-request-presigner</pre><ul><li>Create a generatePresignedUrl.js file and configure your AWS credentials:</li></ul><pre>import { S3Client, S3ClientConfig } from &quot;@aws-sdk/client-s3&quot;;<br>import { AWS_S3 } from &quot;../helper/constants&quot;;<br><br>const config: S3ClientConfig = {<br>  credentials: {<br>    accessKeyId: AWS_S3.API_KEY ?? &quot;&quot;,<br>    secretAccessKey: AWS_S3.SECRET ?? &quot;&quot;<br>  },<br>  region: AWS_S3.REGION<br>};<br><br>export const client = new S3Client(config);</pre><h4>6. Generating a Presigned URL for Image Upload</h4><ul><li>Create a function to generate a presigned URL:</li></ul><pre>import { S3Client, PutObjectCommand } from &quot;@aws-sdk/client-s3&quot;;<br>import { AWS_S3 } from &quot;../helper/constants&quot;;<br>import { getSignedUrl } from &quot;@aws-sdk/s3-request-presigner&quot;;<br><br>export const imageUpload = async (req: Request, res: Response) =&gt; {<br>  try {<br>    const getPresignedUrl = async (client: S3Client) =&gt; {<br>      const fileName =<br>        Date.now().toString() + (Math.random() * 100000).toFixed(0);<br>      const fileUrl = `https://${AWS_S3.NAME}.s3.${AWS_S3.REGION}.amazonaws.com/${fileName}`;<br>      const command = new PutObjectCommand({<br>        Bucket: AWS_S3.NAME,<br>        Key: fileName<br>      });<br>      const preSignedUrl = await getSignedUrl(client, command);<br>      return {<br>        fileUrl,<br>        preSignedUrl<br>      };<br>    };<br>    const count = req.query.count ?? 1;<br>    const presignedUrlArray = [];<br>    for (let index = 0; index &lt; +count; index++) {<br>      presignedUrlArray.push(getPresignedUrl(client));<br>    }<br>    const presignedUrlResult = await Promise.allSettled(presignedUrlArray);<br>    const uploadUrls = [];<br>    for (const result of presignedUrlResult) {<br>      if (result.status === &quot;fulfilled&quot;) {<br>        uploadUrls.push(result.value);<br>      }<br>    }<br>    return res.status(200).json({<br>      success: true,<br>      data: uploadUrls,<br>      message: &quot;preSignedUrl Generated...&quot;<br>    });<br>  } catch (error) {<br>    logger.error(&quot;Error in imageUpload.&quot; + error);<br>    return res<br>      .status(500)<br>      .json({ success: false, message: &quot;preSigned URL failed:&quot; + error });<br>  }<br>};</pre><h3>Code Explanation</h3><ol><li><strong>Function Overview</strong> The imageUpload function handles the generation of presigned URLs for uploading images to an AWS S3 bucket. The URLs allow clients to securely upload files directly to S3 without exposing server-side credentials.</li><li><strong>Generate a Presigned URL</strong> The getPresignedUrl function creates a presigned URL for uploading a file to an S3 bucket:</li></ol><ul><li>Generates a unique fileName using a combination of the current timestamp and a random number.</li><li>Constructs the S3 file URL using the bucket name, region, and generated filename.</li><li>Uses the AWS SDK PutObjectCommand to define the parameters for the S3 object upload.</li><li>Calls getSignedUrl to get a presigned URL for the specified S3 command.</li></ul><p><strong>3. Handle Multiple URLs Generation</strong></p><ul><li>The count parameter from the request query determines the number of presigned URLs to generate (defaults to 1).</li><li>A loop creates the desired number of presigned URLs using the getPresignedUrl function.</li><li>Uses Promise.allSettled to wait for all presigned URL generation to complete, handling both resolved and rejected promises.</li></ul><p><strong>4. Response Construction</strong></p><ul><li>Collects all successfully generated URLs into an uploadUrls array.</li><li>Returns a JSON response with the list of URLs on success, or an error message on failure.</li></ul><h3>7. Set your Route</h3><pre>router.get(&quot;/upload&quot;, imageUpload);</pre><h3>8. Generate preSignedUrl</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*SwiGkDNKnBA3Uk1rSNMcZw.png" /><figcaption>postman image</figcaption></figure><p>Note:Count=1 is number of Generate preSignedURL</p><h3>9. Image Upload in PreSignUrl Using Put http</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*DRJCb_-3PsU5AtUFkk7LBg.png" /></figure><h3>Next Steps</h3><ol><li><strong>Test the Implementation</strong>: Deploy the code and test it by making requests to your server endpoint to generate presigned URLs.</li><li><strong>Secure the Endpoint</strong>: Implement authentication and authorization checks to ensure only authorized users can generate presigned URLs.</li><li><strong>Monitor Usage</strong>: Use AWS CloudWatch or other monitoring tools to track S3 usage and detect any unusual patterns.</li></ol><p>Would you like more details on any specific part of the code or further improvements?</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=2100ad0f1956" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[A Beginner’s Guide to SSH Keys and Connecting to a Remote Server]]></title>
            <link>https://medium.com/@kevalrabadiya27/a-beginners-guide-to-ssh-keys-and-connecting-to-a-remote-server-7f94dd6b963d?source=rss-cfbc1dd2a8b1------2</link>
            <guid isPermaLink="false">https://medium.com/p/7f94dd6b963d</guid>
            <category><![CDATA[linux]]></category>
            <category><![CDATA[ssh-keygen]]></category>
            <category><![CDATA[ssh]]></category>
            <category><![CDATA[aws]]></category>
            <category><![CDATA[server-connection]]></category>
            <dc:creator><![CDATA[Keval Rabadiya]]></dc:creator>
            <pubDate>Tue, 20 Aug 2024 06:10:46 GMT</pubDate>
            <atom:updated>2024-08-20T06:10:46.125Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/261/1*DD5j-ZIG_ivv5vBjsOBhjQ.png" /></figure><h3>Introduction to SSH</h3><p>SSH (Secure Shell) is a cryptographic network protocol that enables secure communication between two systems over an unsecured network. It is widely used to access remote servers securely, providing a way to manage and configure them.</p><p>One of the most important tools in SSH is the use of SSH keys. These keys eliminate the need to repeatedly enter passwords while maintaining a high level of security. In this guide, we’ll explore the basics of SSH keys and walk you through how to use them to connect to a remote server.</p><h3>What are SSH Keys?</h3><p>SSH keys are a pair of cryptographic keys used to authenticate a user to an SSH server. They consist of two parts:</p><ul><li><strong>Public Key</strong>: This key is shared with the server and stored in a file that authorizes you as a trusted user.</li><li><strong>Private Key</strong>: This is kept secret on your local machine and is used to verify your identity when connecting to the server.</li></ul><p>The relationship between these two keys ensures that only someone with the correct private key can access the server, even if the public key is available to anyone.</p><h3>Why Use SSH Keys Over Passwords?</h3><p>Using SSH keys offers several advantages:</p><ol><li><strong>Increased Security</strong>: Unlike passwords, SSH keys are nearly impossible to guess. Even if someone knows your username, they cannot access your server without the correct private key.</li><li><strong>Convenience</strong>: Once you’ve set up SSH keys, you won’t have to enter your password every time you connect to a server.</li><li><strong>Automation</strong>: SSH keys allow for easier automation of tasks between servers, which is particularly useful in cloud infrastructure and DevOps workflows.</li></ol><h3>Generating SSH Keys</h3><p>To connect to a server using SSH keys, you’ll first need to generate a key pair on your local machine. This process is straightforward and only takes a minute.</p><ol><li><strong>Open your terminal</strong>: The process is similar on Linux, macOS, and Windows (with tools like Git Bash or WSL).</li></ol><p><strong>2. Generate your SSH key pair</strong>: Run the following command:</p><pre>ssh-keygen -t rsa -b 4096 -C &quot;your_email@example.com&quot;</pre><p>Here’s what the flags mean:</p><ul><li>-t rsa: Specifies the RSA algorithm for the key.</li><li>-b 4096: Defines the key length (4096 bits is recommended for security).</li><li>-C &quot;your_email@example.com&quot;: This is a label for the key, usually your email.</li></ul><p><strong>3 . Save the keys</strong>: You’ll be prompted to choose a file to save your key. You can press Enter to accept the default location (~/.ssh/id_rsa).</p><p><strong>4. Set a passphrase</strong> (optional): You can choose to protect your private key with a passphrase. This adds an extra layer of security, but you’ll need to enter the passphrase each time you use the key.</p><p>Your keys are now generated! The public key is stored in ~/.ssh/id_rsa.pub, and the private key is stored in ~/.ssh/id_rsa.</p><h3>Copying the Public Key to Your Server</h3><p>To use your SSH key for authentication, you need to copy the public key to the server you want to connect to. There are a few ways to do this:</p><h4>Method 1: Using ssh-copy-id</h4><p>If you already have password access to the server, you can use the ssh-copy-id command to automatically install your public key on the server:</p><pre>ssh-copy-id username@your_server_ip</pre><p>This command appends your public key to the ~/.ssh/authorized_keys file on the server, which is where the server looks for authorized public keys.</p><h4>Method 2: Manually Copying the Public Key</h4><p>If ssh-copy-id isn’t available, you can manually copy your public key:</p><ol><li>Display the public key on your local machine:</li></ol><pre>cat ~/.ssh/id_rsa.pub</pre><p>2. Log in to your server via password authentication and navigate to the ~/.ssh/ directory:</p><pre>mkdir -p ~/.ssh</pre><p>3. Edit the authorized_keys file:</p><pre>nano ~/.ssh/authorized_keys</pre><p>4. Paste the contents of your public key (id_rsa.pub) into the file and save it. Make sure the file permissions are correct:</p><pre>chmod 600 ~/.ssh/authorized_keys</pre><h3>Connecting to the Server Using SSH Keys</h3><p>Once your public key is installed on the server, connecting is simple. Run the following command:</p><pre>ssh username@your_server_ip</pre><p>If everything is set up correctly, you will be logged into your server without needing to enter a password!</p><h3>Troubleshooting Common Issues</h3><ul><li><strong>Permission Denied (Public Key)</strong>: If you encounter this error, ensure the public key was copied correctly to the server and that the file permissions for ~/.ssh/authorized_keys are set to 600.</li><li><strong>Missing SSH Directory</strong>: If you don’t have an ~/.ssh/ directory on your server, create one and set the appropriate permissions (chmod 700 ~/.ssh).</li><li><strong>Passphrase Prompt</strong>: If you opted to add a passphrase when generating the key, you’ll be prompted to enter it when connecting to the server.</li></ul><h3>Conclusion</h3><p>SSH keys are an essential tool for securely accessing and managing remote servers. They enhance security, streamline workflows, and are easy to set up. Whether you’re managing a single server or a large-scale infrastructure, SSH keys should be a part of your toolkit.</p><p>Now that you’ve learned how to generate and use SSH keys, you can connect to your servers with ease and peace of mind.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=7f94dd6b963d" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Integrating Clerk Authentication with Next.js Using clerkMiddleware()]]></title>
            <link>https://medium.com/@kevalrabadiya27/integrating-clerk-authentication-with-next-js-using-clerkmiddleware-1969b11103bc?source=rss-cfbc1dd2a8b1------2</link>
            <guid isPermaLink="false">https://medium.com/p/1969b11103bc</guid>
            <category><![CDATA[web-development]]></category>
            <category><![CDATA[middleware]]></category>
            <category><![CDATA[front-end-development]]></category>
            <category><![CDATA[nextjs]]></category>
            <category><![CDATA[clerk-auth]]></category>
            <dc:creator><![CDATA[Keval Rabadiya]]></dc:creator>
            <pubDate>Tue, 20 Aug 2024 05:42:55 GMT</pubDate>
            <atom:updated>2024-08-20T05:42:55.786Z</atom:updated>
            <content:encoded><![CDATA[<p>In this guide, we’ll explore how to seamlessly integrate Clerk authentication into your Next.js application using clerkMiddleware(). This method is compatible with both the App and Pages routers in Next.js, offering a flexible approach to securing your routes and ensuring only authorized users can access specific parts of your application.</p><h3>Setting Up clerkMiddleware()</h3><p>To get started, create a middleware.ts file at the root of your project (or within your src/ directory if you have one). This file will house the clerkMiddleware() configuration.</p><p>Here’s a basic setup:</p><pre>import { clerkMiddleware } from &#39;@clerk/nextjs/server&#39;;<br><br>export default clerkMiddleware();<br>export const config = {<br>  matcher: [<br>    &#39;/((?!_next|[^?]*\\.(?:html?|css|js(?!on)|jpe?g|webp|png|gif|svg|ttf|woff2?|ico|csv|docx?|xlsx?|zip|webmanifest)).*)&#39;,<br>    &#39;/(api|trpc)(.*)&#39;,<br>  ],<br>};</pre><p>By default, clerkMiddleware() will not protect any routes, making all routes public unless you explicitly configure protection.</p><h3>Protecting Routes with createRouteMatcher()</h3><p>Clerk provides a createRouteMatcher() helper to protect multiple routes efficiently. This function accepts an array of routes and checks whether the user is attempting to access one of them.</p><p>Here’s an example of protecting /dashboard and /forum routes:</p><pre>import { clerkMiddleware, createRouteMatcher } from &#39;@clerk/nextjs/server&#39;;<br><br>const isProtectedRoute = createRouteMatcher([&#39;/dashboard(.*)&#39;, &#39;/forum(.*)&#39;]);<br><br>export default clerkMiddleware((auth, req) =&gt; {<br>  if (!auth().userId &amp;&amp; isProtectedRoute(req)) {<br>    return auth().redirectToSignIn();<br>  }<br>});<br><br>export const config = {<br>  matcher: [<br>    &#39;/((?!_next|[^?]*\\.(?:html?|css|js(?!on)|jpe?g|webp|png|gif|svg|ttf|woff2?|ico|csv|docx?|xlsx?|zip|webmanifest)).*)&#39;,<br>    &#39;/(api|trpc)(.*)&#39;,<br>  ],<br>};</pre><h3>Advanced Route Protection Based on Authorization</h3><p>Beyond simple authentication, you can protect routes based on user authorization, such as roles or permissions. Here’s how to restrict admin routes to users with specific permissions:</p><h3>Protecting All Routes and Debugging</h3><p>You might want to protect all routes in your application by default, except for specific public routes like /sign-in and /sign-up. This can be achieved by inverting the logic in the clerkMiddleware() function.</p><pre>import { clerkMiddleware, createRouteMatcher } from &#39;@clerk/nextjs/server&#39;;<br><br>const isPublicRoute = createRouteMatcher([&#39;/sign-in(.*)&#39;, &#39;/sign-up(.*)&#39;]);<br><br>export default clerkMiddleware((auth, req) =&gt; {<br>  if (!isPublicRoute(req)) {<br>    auth().protect();<br>  }<br>});<br><br>export const config = {<br>  matcher: [<br>    &#39;/((?!_next|[^?]*\\.(?:html?|css|js(?!on)|jpe?g|webp|png|gif|svg|ttf|woff2?|ico|csv|docx?|xlsx?|zip|webmanifest)).*)&#39;,<br>    &#39;/(api|trpc)(.*)&#39;,<br>  ],</pre><h3>Combining Middleware</h3><p>Finally, you can combine Clerk’s middleware with other middlewares, like internationalization, to build a robust and comprehensive solution.</p><pre>import { clerkMiddleware, createRouteMatcher } from &#39;@clerk/nextjs/server&#39;;<br>import createMiddleware from &#39;next-intl/middleware&#39;;<br><br>import { AppConfig } from &#39;./utils/AppConfig&#39;;<br>const intlMiddleware = createMiddleware({<br>  locales: AppConfig.locales,<br>  localePrefix: AppConfig.localePrefix,<br>  defaultLocale: AppConfig.defaultLocale,<br>});<br>const isProtectedRoute = createRouteMatcher([&#39;dashboard/(.*)&#39;]);<br>export default clerkMiddleware((auth, req) =&gt; {<br>  if (isProtectedRoute(req)) auth().protect();<br>  return intlMiddleware(req);<br>});<br>export const config = {<br>  matcher: [<br>    &#39;/((?!_next|[^?]*\\.(?:html?|css|js(?!on)|jpe?g|webp|png|gif|svg|ttf|woff2?|ico|csv|docx?|xlsx?|zip|webmanifest)).*)&#39;,<br>    &#39;/(api|trpc)(.*)&#39;,<br>  ],<br>};</pre><h3>Debugging Middleware</h3><p>If you encounter issues, enabling debugging in clerkMiddleware() can be incredibly helpful. You can enable debugging conditionally based on your environment:</p><pre>import { clerkMiddleware } from &#39;@clerk/nextjs/server&#39;;<br><br>export default clerkMiddleware(<br>  (auth, req) =&gt; {<br>    // Custom logic for debugging<br>  },<br>  { debug: process.env.NODE_ENV === &#39;development&#39; },<br>);<br>export const config = {<br>  matcher: [<br>    &#39;/((?!_next|[^?]*\\.(?:html?|css|js(?!on)|jpe?g|webp|png|gif|svg|ttf|woff2?|ico|csv|docx?|xlsx?|zip|webmanifest)).*)&#39;,<br>    &#39;/(api|trpc)(.*)&#39;,<br>  ],<br>};</pre><p>With clerkMiddleware() and the powerful tools Clerk provides, you can easily secure and manage authentication and authorization in your Next.js applications, ensuring a smooth and secure user experience.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=1969b11103bc" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Syncing Data Seamlessly: Leveraging Clerk Webhooks and Convex in Next.js 14]]></title>
            <link>https://medium.com/@kevalrabadiya27/syncing-data-seamlessly-leveraging-clerk-webhooks-and-convex-in-next-js-14-b70b946f71fa?source=rss-cfbc1dd2a8b1------2</link>
            <guid isPermaLink="false">https://medium.com/p/b70b946f71fa</guid>
            <category><![CDATA[real-time-communication]]></category>
            <category><![CDATA[clerk-auth]]></category>
            <category><![CDATA[nextjs]]></category>
            <category><![CDATA[middleware]]></category>
            <category><![CDATA[convex]]></category>
            <dc:creator><![CDATA[Keval Rabadiya]]></dc:creator>
            <pubDate>Sat, 10 Aug 2024 04:22:06 GMT</pubDate>
            <atom:updated>2024-08-10T04:22:06.880Z</atom:updated>
            <content:encoded><![CDATA[<p>In modern web applications, seamless data synchronization across different services is critical. Imagine managing user authentication with Clerk and wanting to sync this data to Convex in real-time. With Next.js 14, this process is simpler than ever, thanks to Clerk Webhooks and the powerful clerkMiddleware function.</p><h3>Prerequisites</h3><p>Before diving in, ensure you have the following:</p><ul><li><strong>Next.js 14</strong>: Ensure your project is running on the latest Next.js version.</li><li><strong>Clerk Account</strong>: You’ll need to set up a Clerk project.</li><li><strong>Convex Account</strong>: This is essential for syncing data.</li><li><strong>Basic Understanding of Webhooks</strong>: Familiarity with how webhooks work will be helpful.</li></ul><h3>Step 1: Setting Up Clerk in Next.js</h3><p>First, let’s set up Clerk in your Next.js 14 application.</p><ol><li><strong>Install Clerk SDK</strong>:</li></ol><pre>npm install @clerk/nextjs</pre><p><strong>2. Initialize Clerk</strong>: In your _app.tsx, initialize Clerk.</p><pre>import { ClerkProvider } from &#39;@clerk/nextjs&#39;;<br>import &#39;../styles/globals.css&#39;;<br><br>function MyApp({ Component, pageProps }) {<br>  return (<br>    &lt;ClerkProvider {...pageProps}&gt;<br>      &lt;Component {...pageProps} /&gt;<br>    &lt;/ClerkProvider&gt;<br>  );<br>}<br>export default MyApp;</pre><p><strong>3. Set Up Clerk Environment Variables</strong>: In your .env.local file, add your Clerk API keys.</p><pre>NEXT_PUBLIC_CLERK_FRONTEND_API=&lt;your-frontend-api&gt;<br>CLERK_API_KEY=&lt;your-backend-api-key&gt;</pre><h3>Step 2: Configuring Convex</h3><p>Now, let’s configure Convex to store and sync the user data.</p><ol><li><strong>Install Convex SDK</strong>:</li></ol><pre>npm install convex</pre><p><strong>2. Initialize Convex</strong>: Set up Convex in your project.</p><pre>npx convex init</pre><p><strong>3. Create a Schema</strong>: Define a schema to store the user data. In your convex/schema.ts:</p><pre>import { defineSchema, defineTable, s } from &quot;convex/schema&quot;;<br><br>export default defineSchema({<br>  users: defineTable({<br>    clerkId: s.string(),<br>    email: s.string(),<br>    username: s.string(),<br>  }),<br>});</pre><h3>Step 3: Setting Up Clerk Webhooks</h3><p>Webhooks allow Clerk to notify your application when a user is created or updated. Let’s configure a webhook in Clerk.</p><ol><li><strong>Create a Webhook in Clerk</strong>:</li></ol><ul><li>Go to the Clerk dashboard.</li><li>Navigate to Webhooks and create a new one.</li><li>Set the webhook URL to point to your Next.js API route, e.g., /api/webhooks/clerk.</li></ul><p><strong>2. Handle Webhooks in Next.js</strong>: Create an API route to handle the webhook events.</p><pre>import { clerkMiddleware } from &quot;@clerk/nextjs/api&quot;;<br>import { mutation } from &quot;convex/_generated/server&quot;;<br><br>export default clerkMiddleware(async (req, res) =&gt; {<br>  if (req.method === &quot;POST&quot;) {<br>    const { id, email_addresses, username } = req.body.data.object;<br><br>    // Sync data with Convex<br>    const addUser = mutation(&quot;users.add&quot;);<br>    await addUser({ clerkId: id, email: email_addresses[0].email_address, username });<br><br>    res.status(200).json({ message: &quot;User data synced with Convex!&quot; });<br>  } else {<br>    res.status(405).json({ message: &quot;Method not allowed&quot; });<br>  }<br>});</pre><h3>Step 4: Syncing Data with Convex</h3><p>Now, let’s create a Convex function to handle adding or updating user data.</p><ol><li><strong>Define a Convex Mutation</strong>: In your convex/functions.ts:</li></ol><pre>import { mutation } from &quot;convex/_generated/server&quot;;<br><br>export const addUser = mutation(async ({ db }, { clerkId, email, username }) =&gt; {<br>  await db.insert(&quot;users&quot;, {<br>    clerkId,<br>    email,<br>    username,<br>  });<br>});</pre><p><strong>2. Call the Mutation in the Webhook</strong>: As shown in the previous step, invoke this mutation within the Clerk webhook handler.</p><h3>Step 5: Testing the Integration</h3><p>Finally, let’s test the setup:</p><ol><li><strong>Create a New User in Clerk</strong>: Manually create a user in your Clerk dashboard.</li><li><strong>Check Convex</strong>: Verify that the user data has been synced to Convex.</li></ol><p>If everything is set up correctly, you should see the user data stored in Convex!</p><h3>Conclusion</h3><p>With Clerk, Convex, and Next.js 14, syncing user data across services can be handled efficiently. Clerk Webhooks, paired with clerkMiddleware and Convex, provide a seamless way to keep your user data up-to-date in real time.</p><p>This integration not only simplifies your development workflow but also ensures that your application’s data is always consistent across different platforms.</p><p>Feel free to extend this setup by handling more webhook events, or syncing additional data fields as required by your application. Happy coding!</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=b70b946f71fa" width="1" height="1" alt="">]]></content:encoded>
        </item>
    </channel>
</rss>