<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Tuhin Banerjee on Medium]]></title>
        <description><![CDATA[Stories by Tuhin Banerjee on Medium]]></description>
        <link>https://medium.com/@geektuhin?source=rss-ad140a453883------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/0*SLDMUldAKOmG9kCF</url>
            <title>Stories by Tuhin Banerjee on Medium</title>
            <link>https://medium.com/@geektuhin?source=rss-ad140a453883------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Sun, 22 Nov 2026 03:14:02 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@geektuhin/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[Your AI agents have a credit card. Who’s holding the limit?]]></title>
            <description><![CDATA[<div class="medium-feed-item"><p class="medium-feed-image"><a href="https://medium.com/@geektuhin/your-ai-agents-have-a-credit-card-whos-holding-the-limit-cd823aa3830a?source=rss-ad140a453883------2"><img src="https://cdn-images-1.medium.com/max/2560/1*_tZMub3MMgrhda7f3NhL7g.png" width="2560"></a></p><p class="medium-feed-snippet">Why monthly spend caps don&#x2019;t protect you from runaway agents, and what a per-run budget looks like.</p><p class="medium-feed-link"><a href="https://medium.com/@geektuhin/your-ai-agents-have-a-credit-card-whos-holding-the-limit-cd823aa3830a?source=rss-ad140a453883------2">Continue reading on Medium »</a></p></div>]]></description>
            <link>https://medium.com/@geektuhin/your-ai-agents-have-a-credit-card-whos-holding-the-limit-cd823aa3830a?source=rss-ad140a453883------2</link>
            <guid isPermaLink="false">https://medium.com/p/cd823aa3830a</guid>
            <category><![CDATA[guardrail]]></category>
            <category><![CDATA[token]]></category>
            <category><![CDATA[ai-agent]]></category>
            <category><![CDATA[ai]]></category>
            <category><![CDATA[security]]></category>
            <dc:creator><![CDATA[Tuhin Banerjee]]></dc:creator>
            <pubDate>Sat, 21 Nov 2026 03:16:21 GMT</pubDate>
            <atom:updated>2026-11-21T03:16:21.488Z</atom:updated>
        </item>
        <item>
            <title><![CDATA[Jev Isn’t Here to Replace Your LLM. It’s Here to Make Decisions Cheap.]]></title>
            <description><![CDATA[<div class="medium-feed-item"><p class="medium-feed-image"><a href="https://medium.com/@geektuhin/jev-isnt-here-to-replace-your-llm-it-s-here-to-make-decisions-cheap-fec18dd17b21?source=rss-ad140a453883------2"><img src="https://cdn-images-1.medium.com/max/2400/1*9Dadlnule-9XGZhTEX0rLw.png" width="2400"></a></p><p class="medium-feed-snippet">What TypeSafe&#x2019;s new System One model actually does, what independent testing shows so far, and where I&#x2019;d put it inside an AI agent.</p><p class="medium-feed-link"><a href="https://medium.com/@geektuhin/jev-isnt-here-to-replace-your-llm-it-s-here-to-make-decisions-cheap-fec18dd17b21?source=rss-ad140a453883------2">Continue reading on Medium »</a></p></div>]]></description>
            <link>https://medium.com/@geektuhin/jev-isnt-here-to-replace-your-llm-it-s-here-to-make-decisions-cheap-fec18dd17b21?source=rss-ad140a453883------2</link>
            <guid isPermaLink="false">https://medium.com/p/fec18dd17b21</guid>
            <category><![CDATA[jevs]]></category>
            <category><![CDATA[llm]]></category>
            <category><![CDATA[ai]]></category>
            <category><![CDATA[pm]]></category>
            <category><![CDATA[ai-agent]]></category>
            <dc:creator><![CDATA[Tuhin Banerjee]]></dc:creator>
            <pubDate>Wed, 04 Nov 2026 05:15:28 GMT</pubDate>
            <atom:updated>2026-11-04T05:15:28.912Z</atom:updated>
        </item>
        <item>
            <title><![CDATA[How I Think About Designing AI Agents for Production]]></title>
            <description><![CDATA[<div class="medium-feed-item"><p class="medium-feed-image"><a href="https://medium.com/@geektuhin/how-i-think-about-designing-ai-agents-for-production-43b1a8408f57?source=rss-ad140a453883------2"><img src="https://cdn-images-1.medium.com/max/2440/1*yh55QzOTtFPMquGUNVkXJg.png" width="2440"></a></p><p class="medium-feed-snippet">Five layers, one running example, and a few architecture diagrams. From scoping the problem to deciding when extra complexity is actually&#x2026;</p><p class="medium-feed-link"><a href="https://medium.com/@geektuhin/how-i-think-about-designing-ai-agents-for-production-43b1a8408f57?source=rss-ad140a453883------2">Continue reading on Medium »</a></p></div>]]></description>
            <link>https://medium.com/@geektuhin/how-i-think-about-designing-ai-agents-for-production-43b1a8408f57?source=rss-ad140a453883------2</link>
            <guid isPermaLink="false">https://medium.com/p/43b1a8408f57</guid>
            <category><![CDATA[ai]]></category>
            <category><![CDATA[agentic-ai]]></category>
            <category><![CDATA[artitecture]]></category>
            <category><![CDATA[ai-agent]]></category>
            <category><![CDATA[generative-ai-tools]]></category>
            <dc:creator><![CDATA[Tuhin Banerjee]]></dc:creator>
            <pubDate>Sun, 01 Nov 2026 11:18:04 GMT</pubDate>
            <atom:updated>2026-11-01T11:18:04.475Z</atom:updated>
        </item>
        <item>
            <title><![CDATA[Your AI Model Isn’t the Product. The Harness Is.]]></title>
            <description><![CDATA[<div class="medium-feed-item"><p class="medium-feed-snippet">What I&#x2019;ve learned about harness engineering, and why I think it should be built once and reused across teams.</p><p class="medium-feed-link"><a href="https://medium.com/@geektuhin/your-ai-model-isnt-the-product-the-harness-is-b570ca57229f?source=rss-ad140a453883------2">Continue reading on Medium »</a></p></div>]]></description>
            <link>https://medium.com/@geektuhin/your-ai-model-isnt-the-product-the-harness-is-b570ca57229f?source=rss-ad140a453883------2</link>
            <guid isPermaLink="false">https://medium.com/p/b570ca57229f</guid>
            <category><![CDATA[software-engineering]]></category>
            <category><![CDATA[ai]]></category>
            <category><![CDATA[agentic-ai]]></category>
            <dc:creator><![CDATA[Tuhin Banerjee]]></dc:creator>
            <pubDate>Sat, 31 Oct 2026 15:38:16 GMT</pubDate>
            <atom:updated>2026-10-31T15:38:16.914Z</atom:updated>
        </item>
        <item>
            <title><![CDATA[I Built an AI Agent That Reads Your API Spec and Executes Tasks Autonomously]]></title>
            <description><![CDATA[<div class="medium-feed-item"><p class="medium-feed-image"><a href="https://medium.com/@geektuhin/i-built-an-ai-agent-that-reads-your-api-spec-and-executes-tasks-autonomously-e420e2fae0b7?source=rss-ad140a453883------2"><img src="https://cdn-images-1.medium.com/max/1006/1*ZfaGWpnXBBpFChL7ikpfVA.png" width="1006"></a></p><p class="medium-feed-snippet">No more seed scripts. No more brittle fixtures. Just describe what you want &#x2014; and watch it happen.</p><p class="medium-feed-link"><a href="https://medium.com/@geektuhin/i-built-an-ai-agent-that-reads-your-api-spec-and-executes-tasks-autonomously-e420e2fae0b7?source=rss-ad140a453883------2">Continue reading on Medium »</a></p></div>]]></description>
            <link>https://medium.com/@geektuhin/i-built-an-ai-agent-that-reads-your-api-spec-and-executes-tasks-autonomously-e420e2fae0b7?source=rss-ad140a453883------2</link>
            <guid isPermaLink="false">https://medium.com/p/e420e2fae0b7</guid>
            <category><![CDATA[customer-service]]></category>
            <category><![CDATA[ai]]></category>
            <category><![CDATA[automation]]></category>
            <category><![CDATA[agents]]></category>
            <category><![CDATA[data]]></category>
            <dc:creator><![CDATA[Tuhin Banerjee]]></dc:creator>
            <pubDate>Fri, 03 Apr 2026 05:27:48 GMT</pubDate>
            <atom:updated>2026-04-04T19:42:16.447Z</atom:updated>
        </item>
        <item>
            <title><![CDATA[Tabula Studio: Revolutionary Browser-Based Data Analytics Platform]]></title>
            <link>https://medium.com/bitsrc/tabula-studio-revolutionary-browser-based-data-analytics-platform-a37768aa615c?source=rss-ad140a453883------2</link>
            <guid isPermaLink="false">https://medium.com/p/a37768aa615c</guid>
            <category><![CDATA[neon]]></category>
            <category><![CDATA[data-analytics]]></category>
            <category><![CDATA[data-visualization]]></category>
            <category><![CDATA[analytics]]></category>
            <category><![CDATA[duckdb]]></category>
            <dc:creator><![CDATA[Tuhin Banerjee]]></dc:creator>
            <pubDate>Sun, 27 Jul 2025 19:27:26 GMT</pubDate>
            <atom:updated>2025-07-28T07:09:58.541Z</atom:updated>
            <content:encoded><![CDATA[<p><em>Breaking the barriers between data and insights with zero-installation enterprise analytics</em></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/894/1*n08KqhuhpdtxpHdInn4r0Q.png" /></figure><h3>The Problem: Traditional Analytics is Broken</h3><p>Enterprise data teams face an impossible choice: spend months setting up complex data infrastructure or compromise on security and performance with cloud-based solutions. Data scientists waste 80% of their time on setup and configuration, rather than actual analysis. Meanwhile, sensitive enterprise data travels through multiple servers, creating security vulnerabilities and compliance headaches.</p><p><strong>What if there was a better way?</strong></p><h3>Introducing Tabula Studio: Analytics Reimagined</h3><p><a href="https://tabulastudio.com/">Tabula Studio</a> is the <strong>browser-only enterprise data analytics platform</strong> that processes millions of rows in seconds while keeping your data completely private. No servers. No installations. No compromises.</p><h3>🚀 Core Value Proposition</h3><p><strong>Traditional Setup</strong>: 6–8 weeks, $50K+ infrastructure costs, ongoing maintenance <strong>Tabula Studio</strong>: 5 minutes, $0 infrastructure, zero maintenance</p><ul><li><strong>99% faster deployment</strong> than traditional analytics stacks</li><li><strong>100% data privacy</strong> — your data never leaves your browser</li><li><strong>90% cost reduction</strong> compared to cloud analytics platforms</li><li><strong>Zero IT overhead</strong> — no servers to manage or maintain</li><li><strong>Writeback — </strong>data to Databases like Neon and Supabase and more.</li></ul><h3>The Modern Data Pipeline: Bronze to Gold</h3><p>Tabula Studio revolutionizes the traditional data engineering workflow by bringing the entire pipeline into your browser:</p><h3>Current Data Flow Architecture</h3><p><strong>Bronze Layer</strong> → <strong>Tabula Studio Processing</strong> → <strong>Gold Layer Export</strong></p><p>Raw data from various sources gets ingested, transformed with AI-powered insights, visualized for analysis, and exported as refined gold-standard data back to your enterprise systems.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*-J5EmLhWHWPsI1liX2ym_A.png" /></figure><h3>Technical Architecture: Engineering Excellence</h3><h4>Browser-First Design Philosophy</h4><p>Tabula Studio leverages cutting-edge web technologies to deliver desktop-class performance directly in your browser:</p><h3>Key Technical Innovations</h3><h4>1. Lightning-Fast Query Engine</h4><ul><li><strong>DuckDB WASM</strong>: Process 10M+ rows in under 3 seconds</li><li><strong>Columnar storage</strong>: 100x faster aggregations</li><li><strong>Parallel processing</strong>: Utilizes all CPU cores in your browser</li><li><strong>Smart caching</strong>: Instant results for repeated queries</li></ul><h4>2. Enterprise-Grade Security</h4><ul><li><strong>Browser fingerprinting</strong> for unique device keys</li><li><strong>Automatic token refresh</strong> with secure key rotation</li><li><strong>Memory cleanup</strong> on session end</li></ul><h4>3. AI-Powered Analytics (Planned)</h4><ul><li><strong>Smart chart recommendations</strong> based on data patterns</li><li><strong>Natural language queries</strong>: “Show me sales trends by region”</li><li><strong>Automated insights</strong>: Detect anomalies and correlations</li><li><strong>Query optimization</strong>: AI suggests faster query patterns</li></ul><h3>Feature Spotlight: What Makes Us Different</h3><h3>🔒 100% Privacy Guaranteed</h3><p>Your data never leaves your browser. Ever. We mathematically cannot access your information.</p><h3>⚡ Instant Performance</h3><ul><li>Process files up to 2GB entirely in-browser</li><li>Sub-second query responses on million-row datasets</li><li>No network latency for data processing</li></ul><h3>🔌 Smart Connectivity</h3><p>Currently supports modern cloud databases with more coming soon:</p><ul><li><strong>Database support</strong>: Neon (PostgreSQL), Supabase</li><li><strong>File formats</strong>: CSV, Excel, Parquet, JSON</li><li><strong>Coming soon</strong>: Snowflake, Databricks, BigQuery, and more</li></ul><h3>🎯 Smart Insights</h3><ul><li>AI suggests the best visualizations for your data</li><li>Automatically detects data quality issues</li><li>Generates executive summaries from complex datasets</li></ul><h3>📊 Jupyter-Style Notebook Experience</h3><p>Tabula Studio provides a familiar notebook interface that data professionals love:</p><p><strong>SQL Notebook Cells:</strong></p><ul><li>Write complex SQL queries with syntax highlighting</li><li>Execute queries with sub-second response times</li><li>View results in interactive data tables</li></ul><p><strong>Instant Visualizations:</strong></p><ul><li>AI-powered chart recommendations (“Low cardinality categorical data — pie chart shows proportions clearly”)</li><li>One-click chart generation (Bar, Line, Pie, Scatter, Heatmaps)</li><li>Interactive charts with drill-down capabilities</li><li>Real-time chart updates as you modify queries</li></ul><p><strong>Professional Export Options:</strong></p><ul><li><strong>Charts</strong>: PNG, JPG, SVG, PDF for presentations</li><li><strong>Data</strong>: CSV, Excel, JSON, Parquet for further analysis</li><li><strong>Reports</strong>: Complete dashboard exports</li><li><strong>Code</strong>: Shareable SQL snippets and notebook files</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*UVDaH152QkLDnO0c0yDhpg.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*uB2jRUxqBWhDsbOc19VHiA.png" /></figure><h3>📝 Powerful Spreadsheet Capabilities</h3><blockquote>Tabula Studio provides modern spreadsheet functionality directly in your browser</blockquote><p>- <strong>Interactive Data Editing</strong>: Edit, transform, and create data directly in spreadsheet-like interfaces<br>- <strong>Formula Support</strong>: Use familiar spreadsheet formulas for quick calculations and data manipulation<br>- <strong>Direct Database Integration</strong>: Changes in sheets can be automatically synced to your browser DuckdDB, and thereby available for export.<br>- <strong>Smart Data Handling</strong>: Automatically detects data types and formatting for easier analysis</p><h3>💰 Massive Cost Savings</h3><p>Traditional Stack Tabula Studio Savings Infrastructure: $10K+/month $0/month $120K+/year Setup time: 8 weeks 5 minutes $50K in dev costs Maintenance: 2 FTEs 0 FTEs $200K+/year <strong>Total Annual Savings: $370K+</strong></p><h3>Current Status &amp; Roadmap</h3><h3>✅ Currently Supported (V1.0)</h3><ul><li><strong>Data Sources</strong>: Neon (PostgreSQL), Supabase, Local Files</li><li><strong>File Formats</strong>: CSV, Excel, JSON, Parquet</li><li><strong>Processing</strong>: 2GB+ files, 10M+ rows/second</li><li><strong>Export</strong>: CSV, Excel, JSON formats, Neon, and Supabase tables.</li></ul><h3>🚀 Coming Soon (V2.0 — Q3 2026)</h3><ul><li><strong>Enterprise Sources</strong>: Snowflake, Databricks, BigQuery</li><li><strong>Traditional Databases</strong>: MySQL, PostgreSQL, SQL Server</li><li><strong>Advanced Exports</strong>: Direct database writes, API integrations</li><li><strong>Real-time Streaming</strong>: Live data processing capabilities</li><li><strong>Sheets: </strong>Create data in TabulaStudio</li><li><strong>Save your work: </strong>Retain your queries forever</li></ul><h3>The Future of Analytics is Here</h3><p>Tabula Studio represents a fundamental shift in how we think about data analytics. By moving computation to the browser, we’ve eliminated the traditional tradeoffs between performance, security, and ease of use.</p><p><strong>Your data deserves better than legacy analytics platforms.</strong></p><p>Ready to experience analytics without limits? <a href="https://tabulastudio.com/"><strong>Try Tabula Studio today</strong></a> and join thousands of data professionals who’ve already made the switch.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=a37768aa615c" width="1" height="1" alt=""><hr><p><a href="https://medium.com/bitsrc/tabula-studio-revolutionary-browser-based-data-analytics-platform-a37768aa615c">Tabula Studio: Revolutionary Browser-Based Data Analytics Platform</a> was originally published in <a href="https://medium.com/bitsrc">Bits and Pieces</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Understanding Trino Architecture: Unleashing the Power of Distributed SQL Queries]]></title>
            <link>https://medium.com/@geektuhin/understanding-trino-architecture-unleashing-the-power-of-distributed-sql-queries-e996aa3eb39f?source=rss-ad140a453883------2</link>
            <guid isPermaLink="false">https://medium.com/p/e996aa3eb39f</guid>
            <category><![CDATA[trino-presto]]></category>
            <category><![CDATA[data-engineering]]></category>
            <category><![CDATA[analytics]]></category>
            <category><![CDATA[duckdb]]></category>
            <category><![CDATA[sql]]></category>
            <dc:creator><![CDATA[Tuhin Banerjee]]></dc:creator>
            <pubDate>Sun, 01 Oct 2023 06:44:14 GMT</pubDate>
            <atom:updated>2023-10-01T06:44:14.390Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*d8taLndrGpb2B978Nrfp8A.jpeg" /></figure><p>Trino is a distributed SQL query engine designed for high-performance, ad-hoc analytics, and data federation</p><p>This article assumes that you have Trino installed and configured. We will now delve into the internal workings of Trino to provide a comprehensive understanding of its architecture and query-processing capabilities.</p><h4>Coordinator and Workers in a Cluster</h4><p>When installing Trino, you use only a single machine to run everything. For the desired scalability and performance, this deployment is not suitable.</p><p>Trino is a distributed SQL query engine resembling massively parallel processing (MPP) style databases and query engines. Rather than relying on vertical scaling of the server running Trino, it is able to distribute all processing across a cluster of servers in a horizontal fashion. This means that you can add more nodes to gain more processing power.</p><p>Leveraging this architecture, the Trino query engine is able to process SQL queries on large amounts of data in parallel across a cluster of computers, or nodes. Trino runs a single-server process on each node. Multiple nodes running Trino, configured to collaborate with each other, make up a Trino cluster.</p><p>Let’s see how the communication within the cluster happens between the coordinator and the workers, as well as from one worker to another.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*QhbqCzHvb5oQw1iKjScxFg.png" /></figure><p>The coordinator talks to workers to assign work, update status, and fetch the top-level result set to return to the users. The workers talk to each other to fetch data from upstream tasks, running on other workers. The workers retrieve result sets from the data source.</p><p>A <em>coordinator </em>is a Trino server that handles incoming queries and manages the <em>workers </em>to execute the queries.<br>A <em>worker </em>is a Trino server responsible for executing tasks and processing data.</p><p>The <em>discovery service </em>typically runs on the coordinator and allows workers to register to participate in the cluster.</p><p><em>Note: All communication and data transfer between clients, coordinators, and workers uses REST-based interactions over HTTP/HTTPS.</em></p><h4>Query Planning</h4><p>Let&#39;s try to understand the Trino Query planner by an example.</p><pre>SELECT<br>(<br>SELECT name <br>FROM region r <br>WHERE regionkey = n.regionkey<br>) AS region_name,<br>n.name AS nation_name,<br>sum(totalprice) orders_sum<br>FROM nation n, orders o, customer c <br>WHERE n.nationkey = c.nationkey<br>AND c.custkey = o.custkey<br>GROUP BY n.nationkey, regionkey, n.name ORDER BY orders_sum DESC<br>LIMIT 5;</pre><ul><li>A SELECT query using three tables in the FROM clause implicitly defines a CROSS</li><li>JOIN between the nation, orders, and customer tables</li><li>A WHERE condition to retain the matching rows from the nation, orders, and customer tables</li><li>An aggregation using GROUP BY regionkey to aggregate values of orders for each nation</li><li>A subquery, (SELECT name FROM region WHERE regionkey = n.regionkey), to pull the region name from the region table; note that this query is correlated as if it was supposed to be executed independently for each row of the containing result set</li><li>An ordering definition, ORDER BY orders_sum DESC, to sort the result before returning</li><li>A limit of five rows is defined to return only nations with the highest order sums and filter out all others.</li></ul><h4>Step 1: Parsing and Analysis</h4><p>Before a query can be planned for execution, it needs to be parsed and analyzed.</p><p>Trino performs 3 key steps</p><p><em>a. Identifying tables used in a query</em></p><p><em>b. Identifying columns used in a query</em></p><p><em>c. Identifying references to fields within ROW values</em></p><h4>Step 2: Initial Query Planning</h4><p>A query plan can be viewed as a program that produces query results. Recall that SQL is a declarative language: the user writes a SQL query to specify the data they want from the system. Unlike an imperative program, the user does not specify how to process the data to get the result. This part is left to the query planner and optimizer to determine the sequence of steps to process the data for the desired result.</p><p>This sequence of steps is often referred to as the <em>query plan</em>. Theoretically, an exponential number of query plans could yield the same query result</p><p>The performance of the plans varies dramatically, and this is where the Trino planner and optimizer try to determine the optimal plan. Plans that always produce the same results are called <em>equivalent plans</em>.</p><pre> Limit[5]<br>  - Sort[orders_sum DESC]<br>    - LateralJoin[2]<br>      - Aggregate[by nationkey...; orders_sum := sum(totalprice)]<br>       - Filter[c.nationkey = n.nationkey AND c.custkey = o.custkey]<br>          - CrossJoin<br>            - CrossJoin<br>              - TableScan[nation]<br>              - TableScan[orders]<br>            - TableScan[customer]<br>      - EnforceSingleRow[region_name := r.name]<br>        - Filter[r.regionkey = n.regionkey]<br>          - TableScan[region]</pre><p>Let’s now consider the computational complexity of this query plan.</p><p>If <em>N</em>, <em>O</em>, <em>C</em>, and <em>R </em>represent the number of rows in nation, orders, customer, and region tables, respectively, we can observe the following:</p><ul><li>TableScan[orders] reads the orders table, returning <em>O </em>rows, so its complexity is Ω(<em>O</em>). Similarly, the other two TableScans return <em>N </em>and <em>C </em>rows; thus their complexity is Ω<em>N</em> and ΩC, respectively.</li><li>CrossJoin above TableScan[nation] and TableScan[orders] combine the data from nation and orders tables; therefore, its complexity is Ω(<em>N </em>× <em>O</em>).</li><li>The CrossJoin above combines the earlier CrossJoin, which produced <em>N </em>× <em>O </em>rows, with TableScan[customer] so with data from the customer table, therefore its complexity is Ω(<em>N </em>× <em>O </em>× <em>C</em>).</li><li>TableScan[region] at the bottom has complexity Ω(<em>R</em>). However, because of the LateralJoin, it is invoked <em>N </em>times, with <em>N </em>as the number of rows returned from the aggregation. Thus, in total, this operation incurs Ω(<em>R </em>× <em>N</em>) computational cost.</li><li>The Sort operation needs to order a set of <em>N </em>rows, so it cannot take less time than proportional to <em>N </em>× log(<em>N</em>).</li></ul><p>Enough of algebraic formulas. It’s time to see what this means in practice! Let’s consider an example of a popular shopping site with 100 million customers from 200 nations who placed 1 billion orders in total. The CrossJoin of these two tables needs to materialize 20 quintillion (20,000,000,000,000,000,000) rows. For a moderately beefy 100-node cluster, processing 1 million rows a second on each node, it would take over 63 centuries to compute the intermediate data for our query.</p><p>Of course, Trino does not even try to execute such a naive plan. A naive plan has its role, though. The initial plan serves as a bridge between two worlds: the world of SQL language and its semantic rules, and the world of query optimizations. The role of query optimization is to transform and evolve the initial plan into an equivalent plan that can be executed as fast as possible, at least in a reasonable amount of time, given the finite resources of the Trino cluster.</p><h4>Trino&#39;s Optimization Rule</h4><p><em>Predicate pushdown </em>is probably the single most important optimization and easiest to understand. Its role is to move the filtering condition as close to the source of the data as possible. As a result, data reduction happens as early as possible during query execution. In our case, it transforms a Filter into a simpler Filter and an InnerJoin above the same CrossJoin condition, leading to a plan shown</p><pre>...<br>- Aggregate[by nationkey...; orders_sum := sum(totalprice)]<br>  - Filter[c.nationkey = n.nationkey AND c.custkey = o.custkey]  // original filter<br>    - CrossJoin<br>      - CrossJoin<br>        - TableScan[nation]<br>        - TableScan[orders]<br>      - TableScan[customer]<br>...<br>...<br>- Aggregate[by nationkey...; orders_sum := sum(totalprice)]<br>  - Filter[c.nationkey = n.nationkey]<br>    - InnerJoin[o.custkey = c.custkey]<br>      - CrossJoin<br>        - TableScan[nation]<br>        - TableScan[orders]<br>      - TableScan[customer]</pre><p>The “bigger” join that was present is now converted into InnerJoin on an equality condition. Without going into details, let’s assume for now that such a join can be efficiently implemented in a distributed system, with computational complexity equal to the number of produced rows. This means that predicate pushdown replaced an “at least” Ω(<em>N </em>× <em>O </em>× <em>C</em>) CrossJoin with a Join that is “exactly” Θ(<em>N </em>× <em>O</em>).</p><h3>DuckDB and Trino for a Modern Data Stack</h3><p>When you head to the racetrack, you bring a high-performance car. You don’t take your family-friendly, but very versatile SUV. While we may not usually think of ducks as high-performance, that is exactly what <strong>DuckDB</strong> is for — fast queries on large data sets. It is the racecar of databases — you can’t fit anything in it, it seats two, and your golf clubs might not even fit in the back, but it is perfect for the racetrack. The features it lacks don’t detract from its strengths. This minimalism is a central design principle in the containerized world of the modern data stack.</p><p>However, it’s essential to know DuckDB’s limits. DuckDB isn’t designed for transactional workloads, which makes it inappropriate for high-concurrency write scenarios. DuckDB offers two concurrency options: one process can both read and write to the database, or multiple processes can read only simultaneously.</p><p>Another aspect to consider is the absence of user management features, rendering DuckDB more suited for individuals rather than collaborative team environments, which may require user access controls and permissions.</p><p>Despite these limitations, DuckDB shines in analytical tasks, provided they align with its strengths and intended use.</p><h4>How about we use it with Trino?</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/647/1*rQ_PKh8pdgMu4d_PxabkIA.png" /></figure><p>To facilitate collaborative workloads, we can employ Trino in conjunction with a DuckDB source connector. Please note that as of now, Trino does not offer a native DuckDB source connector; however, we can utilize the Iceberg connector as an alternative.</p><h4>Conclusion: Unlocking the Full Potential of Trino</h4><p>In this deep dive into Trino’s internal architecture, we’ve uncovered the intricate mechanisms that power this versatile distributed SQL query engine. Assuming you’ve got Trino installed and configured, you now have a clearer understanding of how it processes queries, optimizes performance, handles fault tolerance, and ensures security. Trino’s ability to seamlessly connect to various data sources through connectors like Iceberg opens up a world of possibilities for data-driven collaboration and analytics.</p><p>As the world of data continues to evolve, Trino stands as a valuable tool for organizations seeking to harness the power of their data. With its ever-growing community and the flexibility to extend its capabilities, Trino remains at the forefront of modern data analytics.</p><p>We encourage you to further explore Trino, experiment with its advanced features, and engage with the vibrant Trino community. As you continue your journey in the realm of data analytics, may Trino serve as a robust and reliable companion in your quest for insights and innovation.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=e996aa3eb39f" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[A Comprehensive Guide to Redis Cluster]]></title>
            <link>https://medium.com/bitsrc/a-comprehensive-guide-to-redis-cluster-how-it-works-communication-management-and-more-a79954c698e7?source=rss-ad140a453883------2</link>
            <guid isPermaLink="false">https://medium.com/p/a79954c698e7</guid>
            <category><![CDATA[docker-compose]]></category>
            <category><![CDATA[node]]></category>
            <category><![CDATA[docker]]></category>
            <category><![CDATA[bulljs]]></category>
            <category><![CDATA[redis]]></category>
            <dc:creator><![CDATA[Tuhin Banerjee]]></dc:creator>
            <pubDate>Tue, 01 Aug 2023 17:21:28 GMT</pubDate>
            <atom:updated>2023-08-02T09:46:25.580Z</atom:updated>
            <content:encoded><![CDATA[<h4>How Redis Cluster works — communication, management, and more and learn how to set up a Redis Cluster locally.</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/836/1*75Lq6gwKe6NlPtgZ8Zds1A.jpeg" /></figure><p><strong>Redis Cluster: An Introduction</strong></p><p>Redis is an open-source, in-memory data structure store known for its blazing-fast performance and versatility. Redis Cluster is a distributed implementation of Redis that enables horizontal scaling and high availability. It was introduced to address the limitations of single-node Redis deployments when dealing with large datasets and high traffic.</p><p>Redis Cluster allows you to distribute data across multiple nodes, providing fault tolerance and seamless data sharding. By leveraging Redis Cluster, you can achieve both high performance and data redundancy, making it an ideal choice for modern applications with demanding performance requirements.</p><p><strong>How Redis Cluster Works</strong></p><p><strong>1. Slots and Nodes:</strong><br>Redis Cluster uses the concept of slots to partition data across multiple nodes in a cluster. A Redis cluster consists of multiple nodes, each responsible for a subset of the total slots. By default, there are 16,384 slots in Redis Cluster, and each key is hashed to one of these slots using the CRC16 algorithm. As nodes join or leave the cluster, the slot allocation dynamically adjusts to ensure data distribution and availability.</p><p><strong>2. Cluster Communication:</strong><br>Nodes in a Redis Cluster communicate with each other using a binary protocol called the Redis Cluster Bus. This communication enables nodes to coordinate various operations, such as slot mapping updates, failover, and cluster state sharing. When a client connects to a node and issues a command that involves keys from different slots, the node will internally forward the requests to the appropriate nodes responsible for those slots, ensuring that the command is executed seamlessly across the cluster.</p><p><strong>3. Cluster Management:</strong><br>Redis Cluster employs a distributed consensus protocol, RAFT, to manage cluster configuration and state. RAFT ensures all nodes agree on the current cluster state and slot allocation. If a node fails or becomes unreachable, the cluster uses an automatic failover mechanism to promote one of the slave nodes as the new master for the failed node’s slots. This ensures that the cluster remains operational and available even in the face of node failures.</p><pre>                   +----------------------------------------------+<br>                   |                 Redis Cluster                 |<br>                   |                                              |<br>                   |         +-------------------+                |<br>                   |         |      Node 1       |                |<br>                   |         | +---------------+ |                |<br>                   |         | |    Slot 0     | |                |<br>                   |         | +---------------+ |                |<br>                   |         |      ...        | |                |<br>                   |         | +---------------+ |                |<br>                   |         | | Slot N / 2 - 1 | |                |<br>                   |         | +---------------+ |                |<br>                   |         +-------------------+                |<br>                   |                                              |<br>                   |         +-------------------+                |<br>                   |         |      Node 2       |                |<br>                   |         | +---------------+ |                |<br>                   |         | | Slot N / 2     | |                |<br>                   |         | +---------------+ |                |<br>                   |         |      ...        | |                |<br>                   |         | +---------------+ |                |<br>                   |         | | Slot N - 1     | |                |<br>                   |         | +---------------+ |                |<br>                   |         +-------------------+                |<br>                   |                                              |<br>                   |                     ...                      |<br>                   +----------------------------------------------+</pre><p><strong><em>How can we set up the Redis cluster locally?</em></strong></p><pre># Copyright VMware, Inc.<br># SPDX-License-Identifier: APACHE-2.0<br><br>version: &#39;2&#39;<br>services:<br>  redis-node-0:<br>    image: docker.io/bitnami/redis-cluster:7.0<br>    volumes:<br>      - redis-cluster_data-0:/bitnami/redis/data<br>    environment:<br>      - &#39;REDIS_PASSWORD=bitnami&#39;<br>      - &#39;REDIS_NODES=redis-node-0 redis-node-1 redis-node-2 redis-node-3 redis-node-4 redis-node-5&#39;<br>    networks:<br>      - default<br><br>  redis-node-1:<br>    image: docker.io/bitnami/redis-cluster:7.0<br>    volumes:<br>      - redis-cluster_data-1:/bitnami/redis/data<br>    environment:<br>      - &#39;REDIS_PASSWORD=bitnami&#39;<br>      - &#39;REDIS_NODES=redis-node-0 redis-node-1 redis-node-2 redis-node-3 redis-node-4 redis-node-5&#39;<br>    networks:<br>      - default<br><br>  redis-node-2:<br>    image: docker.io/bitnami/redis-cluster:7.0<br>    volumes:<br>      - redis-cluster_data-2:/bitnami/redis/data<br>    environment:<br>      - &#39;REDIS_PASSWORD=bitnami&#39;<br>      - &#39;REDIS_NODES=redis-node-0 redis-node-1 redis-node-2 redis-node-3 redis-node-4 redis-node-5&#39;<br>    networks:<br>      - default<br><br>  redis-node-3:<br>    image: docker.io/bitnami/redis-cluster:7.0<br>    volumes:<br>      - redis-cluster_data-3:/bitnami/redis/data<br>    environment:<br>      - &#39;REDIS_PASSWORD=bitnami&#39;<br>      - &#39;REDIS_NODES=redis-node-0 redis-node-1 redis-node-2 redis-node-3 redis-node-4 redis-node-5&#39;<br>    networks:<br>      - default<br><br>  redis-node-4:<br>    image: docker.io/bitnami/redis-cluster:7.0<br>    volumes:<br>      - redis-cluster_data-4:/bitnami/redis/data<br>    environment:<br>      - &#39;REDIS_PASSWORD=bitnami&#39;<br>      - &#39;REDIS_NODES=redis-node-0 redis-node-1 redis-node-2 redis-node-3 redis-node-4 redis-node-5&#39;<br>    networks:<br>      - default<br><br>  redis-node-5:<br>    image: docker.io/bitnami/redis-cluster:7.0<br>    volumes:<br>      - redis-cluster_data-5:/bitnami/redis/data<br>    depends_on:<br>      - redis-node-0<br>      - redis-node-1<br>      - redis-node-2<br>      - redis-node-3<br>      - redis-node-4<br>    environment:<br>      - &#39;REDIS_PASSWORD=bitnami&#39;<br>      - &#39;REDISCLI_AUTH=bitnami&#39;<br>      - &#39;REDIS_CLUSTER_REPLICAS=1&#39;<br>      - &#39;REDIS_NODES=redis-node-0 redis-node-1 redis-node-2 redis-node-3 redis-node-4 redis-node-5&#39;<br>      - &#39;REDIS_CLUSTER_CREATOR=yes&#39;<br>    networks:<br>      - default<br><br>  redis-connection:<br>    build:<br>      context: ../<br>      dockerfile: &#39;docker/Dockerfile.redis&#39;<br>    depends_on:<br>      - redis-node-0<br>      - redis-node-1<br>      - redis-node-2<br>      - redis-node-3<br>      - redis-node-4<br>      - redis-node-5<br><br>volumes:<br>  redis-cluster_data-0:<br>    driver: local<br>  redis-cluster_data-1:<br>    driver: local<br>  redis-cluster_data-2:<br>    driver: local<br>  redis-cluster_data-3:<br>    driver: local<br>  redis-cluster_data-4:<br>    driver: local<br>  redis-cluster_data-5:<br>    driver: local<br><br>networks:<br>  default: {}</pre><p>The Docker Compose file defines six Redis nodes (redis-node-0 to redis-node-5) that will be part of the Redis Cluster. Each node is running the Bitnami Redis Cluster image version 7.0. It also sets up a Redis client container (redis-connection) that can interact with the Redis Cluster.</p><p>Each Redis node is associated with a separate volume (redis-cluster_data-0 to redis-cluster_data-5) to persist the data. The ‘driver: local’ attribute indicates that the data will be stored locally on the host machine.</p><p>For each Redis node, the following configurations are defined as environment variables:</p><p>1. `REDIS_PASSWORD`: Sets the password for the Redis nodes to “bitnami.”<br>2. `REDIS_NODES`: Specifies the list of Redis nodes in the cluster that the current node will join.</p><p>The `depends_on` attribute ensures that the Redis-connection container starts only after all the Redis nodes are up and running.</p><p>The Redis Cluster requires at least three master nodes to function correctly. In this setup, redis-node-5 acts as the cluster creator and has a replica for each of the other nodes (REDIS_CLUSTER_REPLICAS=1). This means that the cluster will have three master nodes (redis-node-0 to redis-node-2) and three replica nodes.</p><p>The `redis-connection` container is used to connect to the Redis Cluster. The `build` attribute specifies the context and Dockerfile used to build the image. The container depends on all the Redis nodes to ensure that they are running before it starts.</p><blockquote>Dockerfile.Redis</blockquote><pre># Use the official Node.js base image<br>FROM node:14<br><br># Set the working directory inside the container<br>WORKDIR /usr/src/app<br><br># Copy the Node.js script to the container<br>COPY redis.js .<br><br># Install any dependencies (if needed)<br>RUN npm install ioredis<br>RUN npm install bull<br># Command to run your Node.js script<br>CMD [&quot;node&quot;, &quot;redis.js&quot;]</pre><p>To start the Redis Cluster, navigate to the directory containing the Docker Compose file and run the following command:</p><blockquote>docker-compose up -d</blockquote><p>This will start the Redis Cluster and make it available for interaction using the Redis CLI or any other Redis client.</p><p>To stop the Redis Cluster, run the following command:</p><blockquote>docker-compose down</blockquote><p>That’s it! With this Docker Compose configuration, you can easily set up a local Redis Cluster for testing and development purposes. Remember that this setup is not suitable for production use as it runs all the Redis nodes on a single machine, and data persistence is done locally using volumes. For production scenarios, consider deploying Redis Cluster in a distributed and fault-tolerant manner across multiple servers or cloud instances.</p><p>You might have noticed a docker container configuration in the docker-compose file.</p><pre>redis-connection:<br>    build:<br>      context: ../<br>      dockerfile: &#39;docker/Dockerfile.redis&#39;<br>    depends_on:<br>      - redis-node-0<br>      - redis-node-1<br>      - redis-node-2<br>      - redis-node-3<br>      - redis-node-4<br>      - redis-node-5</pre><p>This section aims to run a test code showcasing the connection mechanism to the Redis Cluster using the Node.js ioredis module. We will explore this process comprehensively, including understanding any potential errors, particularly those related to the Redis cluster with Bull.</p><p>If you require greater control over your local Redis cluster, you can utilize the provided docker-compose file.</p><pre>version: &#39;3.3&#39;<br>services:<br><br>  # Web UI to browse through our Redis data across all nodes<br>  redis_commander:<br>    image: rediscommander/redis-commander:latest<br>    container_name: redis_web<br>    environment:<br>      REDIS_HOSTS: &#39;local:redis_1:6379,local:redis_2:6379,local:redis_3:6379&#39;<br>    ports:<br>      - &#39;5002:8081&#39;<br>    depends_on:<br>      - redis_1<br>      - redis_2<br>      - redis_3<br>      - redis_4<br>      - redis_5<br>      - redis_6<br>      - cluster_initiator<br>    networks:<br>      redis_cluster_net:<br>        ipv4_address: 173.18.0.9<br><br>  # Ephemeral container to create the Redis cluster connections.<br>  # Once the setup is done, this container shuts down<br>  # and the cluster can be used by the service app container<br>  cluster_initiator:<br>    container_name: cluster_initiator<br>    build:<br>      context: redis<br>      dockerfile: Dockerfile<br>    tty: true<br>    depends_on:<br>      - redis_1<br>      - redis_2<br>      - redis_3<br>      - redis_4<br>      - redis_5<br>      - redis_6<br>    networks:<br>      redis_cluster_net:<br>        ipv4_address: 173.18.0.8<br><br>  # Here we have six Redis containers with Cluster mode enabled,<br>  # three of them will work as master nodes and each one of<br>  # will have a replica, so in case of failures, the replica becomes the master.<br>  # They are configured by the `cluster_initiator` container.<br>  redis_1:<br>    image: &#39;redis&#39;<br>    container_name: redis_1<br>    ports:<br>      - &#39;6379:3000&#39;<br>    volumes:<br>      - redis_1_data:/data<br>      - ./redis/redis.conf:/usr/local/etc/redis/redis.conf<br>    command: [&#39;redis-server&#39;, &#39;/usr/local/etc/redis/redis.conf&#39;]<br>    networks:<br>      redis_cluster_net:<br>        ipv4_address: 173.18.0.2<br><br>  redis_2:<br>    image: &#39;redis&#39;<br>    container_name: redis_2<br>    ports:<br>      - &#39;6379:3001&#39;<br>    volumes:<br>      - redis_2_data:/data<br>      - ./redis/redis.conf:/usr/local/etc/redis/redis.conf<br>    command: [&#39;redis-server&#39;, &#39;/usr/local/etc/redis/redis.conf&#39;]<br>    networks:<br>      redis_cluster_net:<br>        ipv4_address: 173.18.0.3<br><br>  redis_3:<br>    image: &#39;redis&#39;<br>    container_name: redis_3<br>    ports:<br>      - &#39;6379:3002&#39;<br>    volumes:<br>      - redis_3_data:/data<br>      - ./redis/redis.conf:/usr/local/etc/redis/redis.conf<br>    command: [&#39;redis-server&#39;, &#39;/usr/local/etc/redis/redis.conf&#39;]<br>    networks:<br>      redis_cluster_net:<br>        ipv4_address: 173.18.0.4<br><br>  redis_4:<br>    image: &#39;redis&#39;<br>    container_name: redis_4<br>    ports:<br>      - &#39;6379:3003&#39;<br>    volumes:<br>      - redis_4_data:/data<br>      - ./redis/redis.conf:/usr/local/etc/redis/redis.conf<br>    command: [&#39;redis-server&#39;, &#39;/usr/local/etc/redis/redis.conf&#39;]<br>    networks:<br>      redis_cluster_net:<br>        ipv4_address: 173.18.0.5<br><br>  redis_5:<br>    image: &#39;redis&#39;<br>    container_name: redis_5<br>    ports:<br>      - &#39;6379:3004&#39;<br>    volumes:<br>      - redis_5_data:/data<br>      - ./redis/redis.conf:/usr/local/etc/redis/redis.conf<br>    command: [&#39;redis-server&#39;, &#39;/usr/local/etc/redis/redis.conf&#39;]<br>    networks:<br>      redis_cluster_net:<br>        ipv4_address: 173.18.0.6<br><br>  redis_6:<br>    image: &#39;redis&#39;<br>    container_name: redis_6<br>    ports:<br>      - &#39;6379:3005&#39;<br>    volumes:<br>      - redis_6_data:/data<br>      - ./redis/redis.conf:/usr/local/etc/redis/redis.conf<br>    command: [&#39;redis-server&#39;, &#39;/usr/local/etc/redis/redis.conf&#39;]<br>    networks:<br>      redis_cluster_net:<br>        ipv4_address: 173.18.0.7<br><br><br>volumes:<br>  app-db-data: null<br>  mssql-server-linux-data: null<br>  mssql-user-linux-data: null<br>  redis_1_data: {}<br>  redis_2_data: {}<br>  redis_3_data: {}<br>  redis_4_data: {}<br>  redis_5_data: {}<br>  redis_6_data: {}<br><br># Rename the default network so we can easily identify it<br># Across all containers<br>networks:<br>  redis_cluster_net:<br>    driver: bridge<br>    ipam:<br>      driver: default<br>      config:<br>        - subnet: 173.18.0.0/16</pre><h4>Redis Cluster will Bull</h4><p>If you are trying to add a job to the queue connecting to the Redis node, you might encounter this error.</p><blockquote>ReplyError: CROSSSLOT Keys in request don’t hash to the same slot</blockquote><p><strong>What does this error mean?</strong></p><p>The error message “ReplyError: CROSSSLOT Keys in request don’t hash to the same slot” is also related to Redis, and it occurs when you try to perform a Redis operation that involves multiple keys that are located in different Redis cluster slots.</p><p>In Redis Cluster, data is distributed across multiple slots using a sharding mechanism based on the keys. Each key is mapped to a specific slot using a hash function, and Redis ensures that keys that are part of the same operation (e.g., a multi-key command like MSET, MGET, or a transaction) are all assigned to the same slot? This way, Redis can guarantee that all keys involved in a single operation are located on the same node in the cluster.</p><p>When you encounter the “CROSSSLOT Keys in request don’t hash to the same slot” error, it means that the keys you are using in your command are mapped to different slots, and therefore, the operation cannot be executed in a single step.</p><p><em>Note: to fix this error we need to create a Redis cluster client while initializing the bull queue.</em></p><pre>const Queue  = require(&#39;bull&#39;);<br>const Redis = require(&#39;ioredis&#39;);<br><br><br>// Redis cluster nodes configuration<br>const nodes = [<br>  {<br>    host: &#39;redis-node-0&#39;,<br>    port: 6379,<br>  },<br>  {<br>    host: &#39;redis-node-1&#39;,<br>    port: 6379,<br>  },<br>  {<br>    host: &#39;redis-node-2&#39;,<br>    port: 6379,<br>  },<br>  {<br>    host: &#39;redis-node-3&#39;,<br>    port: 6379,<br>  },<br>  {<br>    host: &#39;redis-node-4&#39;,<br>    port: 6379,<br>  },<br>  {<br>    host: &#39;redis-node-5&#39;,<br>    port: 6379,<br>  },<br>];<br><br>// Create a new Redis cluster instance<br>const redisCluster = new Redis.Cluster(nodes, {<br>  redisOptions: {<br>    password: &#39;bitnami&#39;, // Replace with your Redis cluster password if set<br>  },<br>});<br><br>// Test the connection<br>redisCluster.on(&#39;connect&#39;, () =&gt; {<br>  console.log(&#39;Connected to Redis cluster.&#39;);<br>});<br><br>redisCluster.on(&#39;error&#39;, (err) =&gt; {<br>  console.error(&#39;Error connecting to Redis cluster:&#39;, err);<br>});<br><br>// Sample usage: Set and get a value<br>async function testRedis() {<br>  try {<br>    await redisCluster.set(&#39;myKey&#39;, &#39;Hello, Redis Cluster!&#39;);<br>    const result = await redisCluster.get(&#39;myKey&#39;);<br>    console.log(&#39;Value from Redis:&#39;, result);<br>    // const myFirstQueue = new Queue(&#39;my-first-queue&#39;, {<br>    //   redis: { host: &quot;redis-node-5&quot;, port: 6379, password: &#39;bitnami&#39; },<br>    //   prefix: &#39;{myprefix}&#39;<br>    // });<br>    const clusterQueue = new Queue(&#39;cluster-testing&#39;, {<br>      prefix: &#39;{clusterQueue}&#39;,<br>      createClient: () =&gt;<br>      new Redis.Cluster(nodes, {<br>        redisOptions: {<br>          password: &#39;bitnami&#39;, // Replace with your Redis cluster password if set<br>        },<br>      })<br>    });<br>    clusterQueue.add({foo: &#39;bar&#39;  });<br>    console.log(`clusterQueue: job added`);<br>  } catch (error) {<br>    console.error(&#39;Error in Redis operation:&#39;, error);<br>  } finally {<br>    // Close the Redis cluster connection when done.<br>    redisCluster.quit();<br>  }<br>}<br><br>// Call the sample function<br>testRedis();</pre><p>This code demonstrates how to connect to a <a href="https://github.com/taskforcesh/bullmq/discussions/410">Redis Cluster</a> and perform basic operations using the `ioredis` library for Node.js along with the `bull` library for handling queues.</p><blockquote>What if I am getting MOVED error?</blockquote><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*uodFAg2UYuU_rQQ_YF5YGA.png" /></figure><p><strong>Let&#39;s try to understand the error</strong></p><p>The error message you provided, “ReplyError: MOVED 10905 turing-redis-0001–001.turing-redis.tsz6ei.use2.cache.amazonaws.com:6379,” is related to Redis, a popular in-memory data structure store often used as a caching layer or database in various applications.</p><p><strong>ReplyError</strong>: This is the type of error returned by Redis when there is an issue with a command or operation sent to the server.</p><p><strong>MOVED</strong>: This part of the error message indicates that the data you are trying to access or interact with has been moved to a different Redis node. Redis uses a concept called “cluster” to distribute data across multiple nodes to achieve better performance, scalability, and fault tolerance. When data is moved between nodes, Redis returns this error to inform the client that the data is no longer available on the current node.</p><p><strong>10905</strong>: This number represents the hash slot that your data should belong to in the Redis cluster. Redis uses a partitioning mechanism to assign different data items to specific hash slots, and each node in the cluster is responsible for handling certain hash slots.</p><p><strong>Possible reasons for the “MOVED” error:</strong></p><p>1. <strong>Cluster Resharding</strong>: If you are using a Redis cluster, the cluster might be going through a resharding process. During resharding, data is moved between nodes to balance the load. As a result, some keys might be temporarily unavailable in their previous locations, leading to the “MOVED” error.</p><p>2. <strong>Cluster Reconfiguration</strong>: The Redis cluster’s configuration might have changed, causing the data to be moved to different nodes. This can happen when nodes are added or removed from the cluster, or when the cluster’s hash slot allocation changes.</p><p>3.<strong> Client Misconfiguration</strong>: Sometimes, this error occurs when the client is not correctly configured to handle Redis clusters. If the client is not cluster-aware and tries to access a Redis cluster, it may not be able to interpret the “MOVED” response correctly.</p><blockquote>Note: Bull internals require atomic operations that span different keys. This behavior breaks Redis’s rules for cluster configurations</blockquote><p><strong>Possible Solution for the MOVED error.</strong></p><p>A standard queue requires <strong>3 connections</strong> to the Redis server. In some situations, you might want to re-use connections</p><ul><li>bclient connections <a href="https://github.com/OptimalBits/bull/issues/880">cannot be re-used</a>, so you should return a new connection each time this is called.</li><li>client and subscriber connections can be shared and will not be closed when the queue is closed. When you are shutting down the process, first close the queues, then the shared connections (if they are shared).</li><li>If you are not sharing connections but still using createClient to do some custom connection logic, you may still need to keep a list of all the connections you created so you can manually close them later when the queue shuts down if you need a graceful shutdown for your process</li><li>do not set a keyPrefix on the connection you create, use Bull&#39;s built-in prefix feature if you need a key prefix</li></ul><pre>const Queue  = require(&#39;bull&#39;);<br>const Redis = require(&#39;ioredis&#39;);<br><br><br>// Create a new Redis cluster instance<br>const redisCluster =  new Redis({<br>  host: &quot;clustercfg.*********&quot;,<br>  port: 6379, // Redis default port<br>  password: &quot;*********&quot;, // Redis AUTH password<br>  tls: {<br>    // Enable TLS (optional)<br>    rejectUnauthorized: false, // Adjust this depending on your TLS setup<br>  },<br>});<br><br>// Test the connection<br>redisCluster.on(&#39;connect&#39;, () =&gt; {<br>  console.log(&#39;Connected to Redis cluster.&#39;);<br>});<br><br>redisCluster.on(&#39;error&#39;, (err) =&gt; {<br>  console.error(&#39;Error connecting to Redis cluster:&#39;, err);<br>});<br><br>// Sample usage: Set and get a value<br>async function testRedis() {<br>  try {<br>    await redisCluster.set(&#39;myKey&#39;, &#39;Hello, Redis Cluster!&#39;);<br>    const result = await redisCluster.get(&#39;myKey&#39;);<br>    console.log(&#39;Value from Redis:&#39;, result);<br>    // const myFirstQueue = new Queue(&#39;my-first-queue&#39;, {<br>    //   redis: { host: &quot;redis-node-5&quot;, port: 6379, password: &#39;bitnami&#39; },<br>    //   prefix: &#39;{myprefix}&#39;<br>    // });<br>    console.log(&quot;----bull connection&quot;)<br>    const clusterQueue = new Queue(&#39;cluster-testing&#39;, {<br>      prefix: &#39;{clusterQueue}&#39;,<br>      createClient: function (type) {<br>        switch (type) {<br>          case &quot;client&quot;: return redisCluster;<br>          case &quot;subscriber&quot;: return redisCluster;<br>          case &quot;bclient&quot;: return redisCluster;<br>          default: return new Redis(); // Fallback to the default client<br>        }<br>      },<br>    });<br>    clusterQueue.add({}, { repeat: { cron: `*/1 * * * *` } });<br>  } catch (error) {<br>    console.error(&#39;Error in Redis operation:&#39;, error);<br>  }<br>}<br><br>// Call the sample function<br>testRedis();</pre><p><strong>Advantages of Using Redis Cluster</strong></p><p>1. High Availability: Redis Cluster’s automatic failover mechanism ensures that the cluster remains operational even if some nodes fail, reducing the risk of data loss and application downtime.</p><p>2. Horizontal Scalability: With Redis Cluster, you can easily scale your data storage and processing capabilities by adding more nodes to the cluster. This allows you to handle increasing amounts of data and traffic without sacrificing performance.</p><p>3. Seamless Data Sharding: Redis Cluster’s slot-based data partitioning distributes data evenly across nodes, eliminating the need for manual data sharding. This simplifies data management and allows for efficient data retrieval.</p><p>4. In-Memory Performance: Redis is an in-memory database, which means that it stores data in RAM, enabling lightning-fast read and write operations. Redis Cluster inherits this performance advantage, making it an excellent choice for applications requiring real-time data processing.</p><p><strong>Disadvantages of Using Redis Cluster</strong></p><p>1. Complexity: Setting up and managing a Redis Cluster can be more complex than a single Redis instance. Properly configuring the cluster, handling failover scenarios, and monitoring cluster health require additional effort and expertise.</p><p>2. Limited Durability: Since Redis Cluster stores data primarily in memory, there is a risk of data loss in the event of a complete cluster failure. Although Redis offers snapshotting and append-only file (AOF) persistence options, these methods may impact performance or cause data inconsistencies.</p><p>3. Network Overhead: Redis Cluster relies heavily on inter-node communication. This can introduce network overhead and may affect performance, especially in geographically distributed clusters.</p><blockquote><em>💡</em> As an aside, for big projects, consider using a <a href="https://blog.bitsrc.io/component-driven-development-and-composable-applications-a-guide-7a0934e60936"><strong>component-driven approach</strong></a> to developing your apps by breaking down complex logic into smaller reusable components, making your codebase more modular, scalable and maintable. Then, with <a href="https://bit.cloud/"><strong>Bit</strong></a>, you can build, test, version, document and publish these independent components to Bit’s <a href="https://bit.cloud/components"><strong>component-sharing platform</strong></a> from where you (or others) can import them into any project with a simple bit import your.username/your-component command.</blockquote><h4>Learn more:</h4><p><a href="https://bit.dev/blog/sharing-javascript-utility-functions-across-projects-l557lr9k/">Sharing JavaScript Utility Functions Across Projects</a></p><p><strong>Conclusion</strong></p><p>Redis Cluster is a powerful distributed database solution that offers high availability, horizontal scalability, and efficient data sharding. Its ability to automatically manage failover and distribute data across nodes makes it a reliable choice for high-performance applications.</p><p>However, setting up and managing a Redis Cluster requires careful consideration and expertise due to its inherent complexity and potential durability concerns. By understanding its advantages and disadvantages, you can make an informed decision about whether Redis Cluster is the right choice for your specific use case.</p><p>Remember to always evaluate your application’s requirements and consider the trade-offs before adopting any technology, including Redis Cluster, in your production environment. With proper planning and implementation, Redis Cluster can be a valuable asset to boost the performance and scalability of your applications.</p><h3>Build composable apps with reusable components, just like Lego</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/700/0*_ptpwFYC91UlqPLG.png" /></figure><p><a href="https://bit.cloud/"><strong>Bit</strong></a><strong> </strong>is an open-source toolchain for the development of composable software.</p><p>With Bit, you can develop any piece of software — a modern web app, a UI component, a backend service or a CLI script — as an independent, reusable and composable unit of software. Share any component across your applications to make it easier to collaborate and faster to build.</p><p><strong>Join the 100,000+ developers building composable software together.</strong></p><p>Get started with these tutorials:</p><h3>→ Micro-Frontends: <a href="https://youtu.be/zBuBlExo3xA">Video</a> // <a href="https://blog.bitsrc.io/how-we-build-micro-front-ends-d3eeeac0acfc">Guide</a></h3><h3>→ Code Sharing: <a href="https://youtu.be/9vS86xgG3ak">Video</a> // <a href="https://bit.cloud/blog/how-to-reuse-react-components-across-your-projects-l4pz83f4">Guide</a></h3><h3>→ Modernization: <a href="https://www.youtube.com/watch?v=c4X1HkFfq1k">Video</a> // <a href="https://blog.bitsrc.io/modernizing-frontend-applications-a-guide-b0bac99be932">Guide</a></h3><h3>→ Monorepo: <a href="https://www.youtube.com/watch?v=5wxyDLXRho4&amp;t=2041s">Video</a> // <a href="https://bit.dev/blog/painless-monorepo-dependency-management-with-bit-l4f9fzyw/">Guide</a></h3><h3>→ Microservices: <a href="https://youtu.be/5wxyDLXRho4">Video</a> // <a href="https://bit.dev/blog/component-driven-microservices-with-nodejs-and-bit-l64shurc/">Guide</a></h3><h3>→ Design System: <a href="https://youtu.be/YG4kpUXRlG4">Video</a> // <a href="https://blog.bitsrc.io/how-we-build-our-design-system-15713a1f1833">Guide</a></h3><h3>Recommended Articles:</h3><ul><li><a href="https://bit.dev/blog/deploying-a-composable-react-app-to-netlify-l7rlluzs/">Deploying a Composable React App to Netlify</a></li><li><a href="https://bit.dev/blog/creating-indepenent-bit-components-in-a-nextjs-project-ljy3mt2y/">Creating Indepenent Bit Components in a NextJS Project</a></li><li><a href="https://blog.bitsrc.io/creating-a-developer-website-with-bit-components-3f4083a7f050">Creating a Developer Website with Bit components</a></li><li><a href="https://bit.dev/blog/how-to-create-a-composable-react-app-with-bit-l7ejpfhc/">How to Create a Composable React App with Bit</a></li><li><a href="https://blog.bitsrc.io/component-driven-development-and-composable-applications-a-guide-7a0934e60936">Composable Applications and Component-Driven Development: a Guide</a></li><li><a href="https://blog.bitsrc.io/why-building-with-components-is-the-best-way-to-ensure-clean-code-aa840b31c03d">Building With Components: Unleashing the Power of Clean Code and Maintainability</a></li><li><a href="https://blog.bitsrc.io/maximizing-code-reusability-with-bit-and-node-js-aece9eb94496">Maximizing Code Reusability with Bit and Node.js</a></li><li><a href="https://blog.bitsrc.io/sharing-components-at-the-enterprise-12f9e5f1bb67">Sharing Components at The Enterprise</a></li></ul><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=a79954c698e7" width="1" height="1" alt=""><hr><p><a href="https://medium.com/bitsrc/a-comprehensive-guide-to-redis-cluster-how-it-works-communication-management-and-more-a79954c698e7">A Comprehensive Guide to Redis Cluster</a> was originally published in <a href="https://medium.com/bitsrc">Bits and Pieces</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Building a Scalable Multi-DataSource ETL Platform with AWS Glue]]></title>
            <link>https://medium.com/cloud-native-daily/building-a-scalable-multi-datasource-etl-platform-with-aws-glue-22f1549995f4?source=rss-ad140a453883------2</link>
            <guid isPermaLink="false">https://medium.com/p/22f1549995f4</guid>
            <category><![CDATA[data-engineering]]></category>
            <category><![CDATA[duckdb]]></category>
            <category><![CDATA[etl-pipeline]]></category>
            <category><![CDATA[aws]]></category>
            <category><![CDATA[aws-glue]]></category>
            <dc:creator><![CDATA[Tuhin Banerjee]]></dc:creator>
            <pubDate>Sun, 30 Jul 2023 10:45:23 GMT</pubDate>
            <atom:updated>2023-07-31T15:02:52.930Z</atom:updated>
            <content:encoded><![CDATA[<h3>Building a Scalable Multi-DataSource ETL Platform with AWS Glue, DuckDB+WASM</h3><h4>Explore a <strong><em>high-level system design</em></strong> for a multi-data source ETL platform using AWS Glue, DuckDB and WebAssembly.</h4><h3>Introduction</h3><p>In the age of big data, organizations are dealing with vast amounts of information coming from various sources, ranging from databases to data lakes. Extract, Transform, and Load (ETL) processes play a critical role in consolidating and preparing this data for analytics and business intelligence purposes. AWS Glue, a serverless ETL service offered by Amazon Web Services (AWS), simplifies the process of data integration and transformation.</p><p>In this blog post, we’ll explore a <strong><em>high-level system design</em></strong> for a multi-data source ETL platform using AWS Glue. This architecture enables you to efficiently extract data from diverse sources, transform it, and load it into a central data store for further analysis.</p><p><strong>System Architecture Overview</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*SOm2myGpNQrXq8y-up56Lw.png" /></figure><p>1. Data Sources:<br>The ETL process starts by ingesting data from multiple sources. These sources can include Amazon S3, databases (like PostgreSQL or others hosted on AWS RDS), data warehouses, APIs, or any other data storage systems.</p><p>2. AWS Glue:<br>AWS Glue is a fully managed and serverless ETL service provided by AWS. It allows you to discover, catalog, and transform data from various sources without the need to manage infrastructure. Glue supports a variety of data formats and provides tools to handle schema changes, making it suitable for dynamic data environments.</p><p>3. Data Transformation:<br>Once data is ingested, AWS Glue performs the data transformation process. This step involves cleaning, enriching, and aggregating data as required by the target data model or analytics needs. Glue provides a visual ETL workflow that allows you to build, schedule, and monitor ETL jobs efficiently.</p><p>4. Intermediate Data Storage:<br>During the transformation process, it’s common to store intermediate results. AWS Glue can store these interim datasets in Amazon S3, ensuring data integrity and facilitating auditing and debugging.</p><p>5. Data Loading:<br>After data transformation, AWS Glue loads the processed data into the target data store. Depending on your use case, this data store can be another S3 bucket, an AWS Redshift data warehouse, Amazon RDS, or any other destination supported by AWS Glue.</p><p>6. Cataloging with AWS Glue Data Catalog:<br>AWS Glue provides a Data Catalog service, which automatically crawls and catalogs metadata about data assets in various data stores. The Data Catalog stores table definitions, statistics, and schema information, providing a centralized location to query and discover data assets.</p><p>7. Querying with AWS Athena:<br>AWS Athena is a serverless query service that allows you to perform ad-hoc SQL queries directly on data stored in Amazon S3. By utilizing the metadata cataloged by AWS Glue, data scientists and analysts can easily query and analyze the transformed data using familiar SQL statements without the need to move data or provision servers.</p><h3>Benefits of the Architecture</h3><p>1. Scalability and Flexibility:<br>The use of AWS Glue, a serverless ETL service, ensures automatic scalability and flexibility to handle varying data workloads. As your data sources grow or change, Glue can seamlessly adapt to accommodate these changes.</p><p>2. Cost Efficiency:<br>By adopting serverless services like AWS Glue and Athena, you only pay for the resources consumed during the data processing and querying operations, eliminating the need for upfront infrastructure investment.</p><p>3. Simplified Maintenance:<br>AWS Glue abstracts away much of the underlying infrastructure management, reducing the operational burden on your team. Additionally, the automatic cataloging of data assets simplifies the data discovery process for your data scientists.</p><p>4. Real-time Data Availability:<br>With periodic triggers, the ETL pipeline can run at predefined intervals, ensuring that the data in the target data store is always up-to-date.</p><p>Once you have successfully reached the GOLD data stage in your ETL pipeline, the curated and optimized dataset in the form of Parquet files is now ready for further analysis and exploration. In this section, we will introduce two powerful technologies, DuckDB and WebAssembly (WASM), that can be leveraged as part of your data analytics stack to dynamically transform, slice, and dice your data efficiently.</p><blockquote><strong>What is DuckDB?</strong></blockquote><p>DuckDB is an open-source analytical database management system that excels at analytical queries on large datasets. It is designed to deliver high-performance and low-latency responses, making it an ideal candidate for interactive data analysis tasks. DuckDB is characterized by its ability to efficiently handle complex analytical queries, including joins, aggregations, and window functions while consuming minimal resources.</p><p>Key features of DuckDB include:</p><p>1. Columnar Storage: DuckDB stores data in a columnar format, which is highly optimized for analytical workloads. This storage layout allows for efficient compression and faster query performance.</p><p>2. Vectorized Execution: DuckDB employs vectorized query execution, processing data in batches, which significantly reduces CPU overhead and enhances query performance.</p><p>3. In-Memory Processing: By keeping frequently accessed data in memory, DuckDB minimizes disk I/O, resulting in faster query processing times.</p><p>4. SQL Compatibility: DuckDB supports standard SQL syntax, making it easy for data analysts and SQL-savvy users to interact with the database.</p><p>By utilizing DuckDB as part of your data analytics stack, you can perform complex analytical queries on your GOLD data with remarkable speed and efficiency, enabling quick exploration and analysis of large datasets.</p><blockquote><strong>What is WebAssembly (WASM)?</strong></blockquote><p>WebAssembly is a binary instruction format designed to enable high-performance execution on web browsers. However, its capabilities extend beyond the web domain, and it can be utilized in various other contexts, including data analytics and processing. WebAssembly allows developers to run code written in multiple programming languages at near-native speeds, making it an attractive option for compute-intensive tasks, such as data analysis.</p><p>Key features of WebAssembly include:</p><p>1. Portability: Web assembly code can run on multiple platforms, including web browsers, servers, and edge devices, enabling seamless execution across various environments.</p><p>2. Language Agnostic: WebAssembly supports multiple programming languages, including C, C++, Rust, and more, giving developers the flexibility to choose their preferred language for computation.</p><p>3. Security: WebAssembly runs in a sandboxed environment, ensuring code safety and mitigating potential security risks associated with executing untrusted code.</p><p>When combined with DuckDB, WebAssembly can be used to create data analytics applications that perform complex computations efficiently, even in resource-constrained environments. WebAssembly-based applications can leverage the power of DuckDB for querying data stored in Parquet files, enabling dynamic transformation, slicing, and dicing of data in real-time, right in the user’s web browser or other runtime environments.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*9Eq3QMisz9UEYRzLdj3ORQ.png" /></figure><p>Sharing some inspirations to build the app</p><ul><li><a href="https://observablehq.com/@cmudig/duckdb">DuckDB in WebAssembly</a></li><li><a href="https://github.com/franloza/coches-net-dashboard">GitHub - franloza/coches-net-dashboard: Sample project that use Dagster, dbt, DuckDB and Dash to visualize car and motorcycle Spanish market</a></li><li><a href="https://franloza.medium.com/building-a-data-application-with-duckdb-43823070730">Building a data application with DuckDB</a></li></ul><h3>Conclusion</h3><p>Building a robust multi-data source ETL platform using AWS Glue empowers organizations to efficiently manage their data integration and transformation needs. The serverless nature of Glue ensures scalability, flexibility, and cost efficiency, while the AWS Glue Data Catalog and Athena enable data scientists and analysts to explore and analyze data without the complexities of managing the underlying infrastructure.</p><p>By adopting this high-level system design, organizations can focus on deriving valuable insights from their data, enhancing decision-making processes, and staying ahead in today’s data-driven world.</p><p>Furthermore, Integrating <a href="https://duckdb.org/2021/10/29/duckdb-wasm.html">DuckDB</a> and <a href="https://duckdb.org/2021/10/29/duckdb-wasm.html">WebAssembly</a> into your data analytics stack enhances your ability to perform interactive and complex data analysis tasks on the GOLD data stage. DuckDB’s speed and efficiency in handling analytical queries combined with WebAssembly’s portability and performance make for a powerful duo. With this technology stack in place, your data analysts and users can dynamically explore, transform, and analyze data with ease, empowering them to derive valuable insights and make data-driven decisions effectively.</p><p>I would love to hear from you if you are building in the DuckDB ecosystem.</p><h4>Further Reading:</h4><ul><li><a href="https://gethelios.dev/blog/serverless-observability/?utm_source=medium&amp;utm_medium=cloud+native+daily">Serverless observability, monitoring, and debugging explained</a></li><li><a href="https://gethelios.dev/opentelemetry-a-full-guide/?utm_source=medium&amp;utm_medium=cloud+native+daily">OpenTelemetry: A full guide</a></li><li><a href="https://aws.plainenglish.io/top-tools-for-aws-lambda-monitoring-8ea51b1f095c">Top Tools for AWS Lambda Monitoring</a></li><li><a href="https://gethelios.dev/blog/combinining-opentelemetry-traces-with-prometheus-metrics-to-build-a-powerful-alerting-mechanism/?utm_source=medium&amp;utm_medium=referral&amp;utm_campaign=cloud+native+daily">Combining OTel and Prometheus metrics for alerting machine</a></li><li><a href="https://gethelios.dev/blog/lambda-monitoring-combining-the-three-pillars-of-observability-to-reduce-mttr/?utm_source=medium&amp;utm_medium=cloud+native+daily">Lambda monitoring: Combining the three pillars of observability to reduce MTTR</a></li></ul><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=22f1549995f4" width="1" height="1" alt=""><hr><p><a href="https://medium.com/cloud-native-daily/building-a-scalable-multi-datasource-etl-platform-with-aws-glue-22f1549995f4">Building a Scalable Multi-DataSource ETL Platform with AWS Glue</a> was originally published in <a href="https://medium.com/cloud-native-daily">Cloud Native Daily</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Demystifying OOM Killer in Kubernetes: Tracking Down Memory Issues]]></title>
            <link>https://medium.com/cloud-native-daily/title-demystifying-oom-killer-in-kubernetes-tracking-down-memory-issues-b5a4973fbd56?source=rss-ad140a453883------2</link>
            <guid isPermaLink="false">https://medium.com/p/b5a4973fbd56</guid>
            <category><![CDATA[oomkill]]></category>
            <category><![CDATA[docker]]></category>
            <category><![CDATA[k8s]]></category>
            <category><![CDATA[monitoring]]></category>
            <category><![CDATA[pods]]></category>
            <dc:creator><![CDATA[Tuhin Banerjee]]></dc:creator>
            <pubDate>Fri, 09 Jun 2023 18:00:57 GMT</pubDate>
            <atom:updated>2023-06-12T06:33:07.925Z</atom:updated>
            <content:encoded><![CDATA[<h4>Unravelling the mysteries of the OOM killer, delve into its inner workings, and learn how to track down memory issues that lead to OOM kills.</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*SIsp12a3foOF6shG7VAQcg.jpeg" /></figure><h3>Introduction</h3><p>Have you ever encountered the dreaded OOM (Out-of-Memory) killer in Kubernetes? This critical component plays a vital role in managing memory resources efficiently within your cluster. However, when a pod exceeds its memory limit and consumes excessive memory, the OOM killer steps in, terminating the pod to free up memory for other critical processes. In this blog, we will unravel the mysteries of the OOM killer, delve into its inner workings, and learn how to track down memory issues that lead to OOM kills. So, let’s dive in!</p><h3>Understanding the OOM Killer</h3><p>The OOM killer is an essential mechanism in Kubernetes that helps maintain the system&#39;s stability and prevent memory exhaustion. It acts as the last line of defense when memory resources are critically low. In such cases, the OOM killer identifies the process or pod responsible for the memory overload and terminates it to free up memory for the rest of the system. By sacrificing one process, the OOM killer prevents a complete system crash, ensuring the overall stability of the cluster.</p><h3>How OOM Kills Are Triggered</h3><p>When a pod in Kubernetes exceeds its specified memory limit, it triggers an OOM event. The container runtime, such as Docker, reports the memory usage to the Kubernetes kubelet. The kubelet, in turn, monitors the memory usage of all pods and compares it against their respective limits. If a pod breaches its limit, the kubelet initiates the OOM killer to terminate the offending pod, freeing up memory resources for other critical workloads.</p><h3>Understanding Memory Metrics and the OOM Decision</h3><p>To make informed decisions about OOM kills, the OOM killer relies on memory metrics obtained from cAdvisor (Container Advisor) and exposed to Kubernetes. The primary metric used by the OOM killer is `<strong><em>container_memory_working_set_bytes</em></strong>`. It represents an estimate of the memory that cannot be evicted, considering the memory pages actively used by the container. This metric acts as the baseline for the OOM killer to decide whether a pod should be terminated or not.</p><h3>Differentiating Between Memory Metrics</h3><p>While `<strong>container_memory_usage_bytes</strong>` may seem like an obvious choice for monitoring memory utilization, it includes cached items, such as the filesystem cache, which can be evicted under memory pressure. Hence, it does not accurately reflect the memory observed and acted upon by the OOM killer. On the other hand, `container_memory_working_set_bytes` provides a more reliable indication of memory usage, aligning with what the OOM killer monitors. It focuses on the memory that cannot be easily reclaimed.</p><h3>How can I debug the memory consumption?</h3><blockquote>To track memory growth within your application, you can monitor the specific file that provides memory usage information. By deploying the following code snippet as part of your application, you can invoke it in DEBUG mode to print the memory usage:</blockquote><pre>const fs = require(&#39;fs&#39;);<br><br>// Function to read memory usage<br>function readMemoryUsage() {<br>  try {<br>    const memoryUsage = fs.readFileSync(&#39;/sys/fs/cgroup/memory/memory.usage_in_bytes&#39;, &#39;utf8&#39;);<br>    console.log(`Memory Usage: ${memoryUsage}`);<br>  } catch (error) {<br>    console.error(&#39;Error reading memory usage:&#39;, error);<br>  }<br>}<br><br>// Call the function to read memory usage<br>readMemoryUsage();</pre><p>In this code, we utilize the fs.readFileSync method to synchronously read the contents of the /sys/fs/cgroup/memory/memory.usage_in_bytes file. The file is read with the &#39;utf8&#39; encoding to interpret the data as a string.</p><p>The readMemoryUsage the function reads the file and logs the memory usage to the console. If an error occurs during the reading process, it will be caught and logged as well.</p><p>Please note that accessing system files /sys/fs/cgroup/memory/memory.usage_in_bytes typically requires elevated privileges. Make sure to run the Node.js script with the necessary permissions or as a privileged user.</p><h4>code to get node heap usage</h4><pre>const fs = require(&#39;fs&#39;);<br><br>// Function to track memory growth<br>function trackMemoryGrowth() {<br>  const memoryUsage = process.memoryUsage();<br>  console.log(`Memory Usage (RSS): ${memoryUsage.rss}`);<br>  console.log(`Memory Usage (Heap Total): ${memoryUsage.heapTotal}`);<br>  console.log(`Memory Usage (Heap Used): ${memoryUsage.heapUsed}`);<br>}<br><br>  trackMemoryGrowth();<br>});</pre><h3>How to monitor Memory and CPU in K8s using kubectl top?</h3><p>The `kubectl top` command is a powerful tool in Kubernetes that enables you to monitor the resource usage of pods and nodes in your cluster. It provides real-time information about memory and CPU utilization, allowing you to identify potential bottlenecks, troubleshoot performance issues, and make informed decisions regarding resource allocation. Let’s explore how to use the `kubectl top` command to monitor pod and node resource usage.</p><blockquote>Monitoring Pod Resource Usage:<br>To monitor the memory and CPU usage of pods, you can use the following command:<br><br><strong>kubectl top pod<br></strong><br>This command provides an overview of the resource usage for all pods in the current namespace. It displays the pod name, CPU usage, memory usage, and the corresponding percentage of resource utilization.</blockquote><blockquote>If you want to focus on a specific pod, you can use the pod name as an argument:<br><br><strong><em>kubectl top pod &lt;pod-name&gt;</em></strong><br><br>This command provides detailed resource usage information for the specified pod.</blockquote><blockquote>Monitoring Node Resource Usage<br>To monitor the memory and CPU usage of nodes in your cluster, you can use the following command:<br><br><strong>kubectl top node</strong><br><br>This command gives you an overview of resource usage for all nodes in the cluster. It provides information about the node name, CPU usage, memory usage, and the corresponding percentage of resource utilization.</blockquote><blockquote>Similar to monitoring pods, you can specify a particular node to retrieve detailed resource usage information:<br><br><strong><em>kubectl top node &lt;node-name&gt;</em></strong></blockquote><p>In conclusion, the `kubectl top` command is a valuable tool that allows you to monitor the memory and CPU usage of pods and nodes in your Kubernetes cluster. By using this command, you can gain insights into resource utilization, detect performance bottlenecks, and optimize resource allocation, ensuring the efficient operation of your applications.</p><h3>Wait, what if I want the same for my docker environment?</h3><p>docker stats memory display collects data from the path /sys/fs/cgroup/memory</p><pre><br># similar to top<br>docker stats --no-stream &lt;container id&gt;</pre><p>On Linux, the <a href="https://docs.docker.com/engine/reference/commandline/stats/">Docker CLI</a> reports memory usage by subtracting cache usage from the total memory usage. The API does not perform such a calculation but rather provides the total memory usage and the amount from the cache so that clients can use the data as needed. The cache usage is defined as the value of total_inactive_file field in the memory.stat file on cgroup v1 hosts.</p><p>On Docker 19.03 and older, the cache usage was defined as the value of cache field. On cgroup v2 hosts, the cache usage is defined as the value of inactive_file field.</p><p>memory_stats.usage is from /sys/fs/cgroup/memory/memory.usage_in_bytes. memory_stats.stats.inactive_file is from /sys/fs/cgroup/memory/memory.stat.</p><h3>Give me the node code to print the memory usage for the docker container</h3><p>To use the docker stats use the <strong><em>docker-stats-api</em></strong> library in Node.js and print the container usage, you can follow these steps:</p><p>Install the <strong><em>docker-stats-api </em></strong>library as a dependency in your Node.js project by running the following command:</p><pre>npm install docker-stats-api</pre><p>Define a function to print the container usage:</p><pre>function printContainerUsage(containerId) {<br>  dockerStats.getStats(containerId)<br>    .then(stats =&gt; {<br>      console.log(`Container Usage (CPU): ${stats.cpu_percent}`);<br>      console.log(`Container Usage (Memory): ${stats.mem_usage}`);<br>    })<br>    .catch(error =&gt; {<br>      console.error(&#39;Error retrieving container stats:&#39;, error);<br>    });<br>}</pre><p>Here’s the complete example:</p><pre>const DockerStats = require(&#39;docker-stats-api&#39;);<br><br>const dockerStats = new DockerStats();<br><br>function printContainerUsage(containerId) {<br>  dockerStats.getStats(containerId)<br>    .then(stats =&gt; {<br>      console.log(`Container Usage (CPU): ${stats.cpu_percent}`);<br>      console.log(`Container Usage (Memory): ${stats.mem_usage}`);<br>    })<br>    .catch(error =&gt; {<br>      console.error(&#39;Error retrieving container stats:&#39;, error);<br>    });<br>}<br><br>const containerId = &#39;YOUR_CONTAINER_ID&#39;;<br>printContainerUsage(containerId);</pre><p>By executing this code, you will be able to retrieve and print the CPU and memory usage of a specific container using the `docker-stats-api` library in Node.js. Remember to replace `’YOUR_CONTAINER_ID’` with the actual ID of the container you wish to monitor.</p><h3>Conclusion</h3><p>By understanding the role and functioning of the OOM killer in Kubernetes, we gain valuable insights into memory management within our clusters. In this blog, we explored how the OOM killer triggers OOM kills and discussed the importance of memory metrics, particularly `container_memory_working_set_bytes`, in making OOM decisions. Armed with this knowledge, you’re well-equipped to monitor and troubleshoot memory issues that lead to OOM kills effectively.</p><h4>Further Reading:</h4><ul><li><a href="https://gethelios.dev/blog/kubernetes-monitoring-opentelemetry/?utm_source=medium&amp;utm_medium=content&amp;utm_campaign=cloudnativedaily">Kubernetes Monitoring with OpenTelemetry</a></li><li><a href="https://gethelios.dev/blog/opentelemetry-tracing/?utm_source=medium&amp;utm_medium=content&amp;utm_campaign=cloudnativedaily">OpenTelemetry Tracing: Everything you need to know</a></li><li><a href="https://medium.com/cloud-native-daily/distributed-tracing-a-guide-for-2023-a40a1ee218b5">Distributed Tracing: A Guide for 2023</a></li><li><a href="https://javascript.plainenglish.io/9-best-distributed-tracing-tools-for-developers-185e415b7101">9 Best Distributed Tracing Tools for Developers</a></li></ul><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=b5a4973fbd56" width="1" height="1" alt=""><hr><p><a href="https://medium.com/cloud-native-daily/title-demystifying-oom-killer-in-kubernetes-tracking-down-memory-issues-b5a4973fbd56">Demystifying OOM Killer in Kubernetes: Tracking Down Memory Issues</a> was originally published in <a href="https://medium.com/cloud-native-daily">Cloud Native Daily</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
    </channel>
</rss>