<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Rafi Syed on Medium]]></title>
        <description><![CDATA[Stories by Rafi Syed on Medium]]></description>
        <link>https://medium.com/@raphy.26.007?source=rss-4e4f9b071bde------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/0*9vacGmsAl4vCZrBJ</url>
            <title>Stories by Rafi Syed on Medium</title>
            <link>https://medium.com/@raphy.26.007?source=rss-4e4f9b071bde------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Fri, 20 Nov 2026 10:11:19 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@raphy.26.007/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[Navigating HikariCP Connection Pool Issues: When Your Database Says “No More Connections!”]]></title>
            <link>https://medium.com/@raphy.26.007/navigating-hikaricp-connection-pool-issues-when-your-database-says-no-more-connections-3203217a14a0?source=rss-4e4f9b071bde------2</link>
            <guid isPermaLink="false">https://medium.com/p/3203217a14a0</guid>
            <category><![CDATA[hikari]]></category>
            <category><![CDATA[application]]></category>
            <category><![CDATA[java]]></category>
            <dc:creator><![CDATA[Rafi Syed]]></dc:creator>
            <pubDate>Thu, 30 Jan 2025 06:43:52 GMT</pubDate>
            <atom:updated>2025-01-30T06:43:52.932Z</atom:updated>
            <content:encoded><![CDATA[<p>Database connection management is a critical aspect of application performance and reliability. When your application suddenly can’t create new database connections — even though HikariCP’s connection pool is configured — it can lead to downtime, slow response times, or cascading failures. One common error message you might encounter in this scenario is:</p><blockquote><strong><em>“remaining connection slots are reserved for non-replication superuser connections”</em></strong></blockquote><p>This article dives into why this happens, how HikariCP settings like idle-timeout and max-lifetime influence connection reuse, and actionable steps to resolve these issues.</p><h3>Understanding HikariCP’s Idle Timeout and Max Lifetime</h3><h3>Idle Timeout: The Parking Lot Rule</h3><p>Imagine a parking lot (connection pool) where cars (database connections) park when unused. The <strong>idle timeout</strong> is like a rule that says: <em>“If a car isn’t driven for 10 minutes, tow it to free up space.”</em></p><ul><li><strong>What It Does</strong>: Closes connections that remain idle (unused) for a configured duration.</li><li><strong>Example</strong>: Setting idle-timeout=600000 (10 minutes) ensures unused connections are closed after 10 minutes of inactivity.</li></ul><h3>Max Lifetime: The Car Replacement Policy</h3><p>Now imagine every car in the lot must be replaced after 30 minutes, regardless of usage. This is the <strong>max lifetime</strong> setting.</p><ul><li><strong>What It Does</strong>: Closes and replaces connections after a fixed duration, even if active.</li><li><strong>Example</strong>: max-lifetime=1800000 (30 minutes) refreshes connections to avoid staleness or network issues.</li></ul><h3>Key Difference</h3><ul><li><strong>Idle Timeout</strong> manages <em>unused</em> connections.</li><li><strong>Max Lifetime</strong> manages <em>age</em> of connections, active or idle.</li></ul><h3>Why New Connections Are Created Despite Idle Ones</h3><p>If your application is creating new connections while idle ones exist, here’s why:</p><ol><li><strong>Connection Validation Failures</strong>:<br>Idle connections are validated before reuse. If validation fails (e.g., network hiccup), HikariCP discards them and creates new ones.</li><li><strong>High Concurrency Spikes</strong>:<br>During traffic surges, if all idle connections are in use, HikariCP spawns new ones up to maximum-pool-size.</li><li><strong>Misconfigured Pool Limits</strong>:</li></ol><p>minimum-idle too low: The pool may not retain enough warm connections.</p><p>maximum-pool-size exceeding the database’s max_connections limit.</p><p>4<strong>. Connection Leaks</strong>:<br>Unclosed connections (e.g., unreturned to the pool) remain marked as “in use,” forcing new connections.</p><p>5<strong>. Transaction Isolation</strong>:<br>Long-running transactions hold connections hostage, preventing reuse.</p><h3>Diagnosing the “Remaining Connection Slots” Error</h3><p>This error occurs when your database (e.g., PostgreSQL) runs out of connection slots. Here’s how to troubleshoot:</p><h3>Step 1: Check Active Connections</h3><p>Run this query to see who’s consuming connections:</p><pre>SELECT datname, usename, state, COUNT(*)<br>FROM pg_stat_activity<br>GROUP BY datname, usename, state;</pre><h3>Step 2: Identify Leaks or Long-Running Queries</h3><p>Look for:</p><ul><li><strong>Idle connections</strong>: Connections unused but not closed.</li><li><strong>Active connections</strong>: Queries taking too long (e.g., missing indexes).</li></ul><h3>Step 3: Terminate Problematic Connections</h3><p>Free slots by closing idle connections:</p><pre>SELECT pg_terminate_backend(pid)<br>FROM pg_stat_activity<br>WHERE state = &#39;idle&#39; AND now() - state_change &gt; interval &#39;10 minutes&#39;;</pre><h3>Fixing the Root Cause: Configuration and Best Practices</h3><h3>1. Tune HikariCP Settings</h3><p>Align pool settings with your database’s capacity:</p><pre>spring.datasource.hikari:<br>  maximum-pool-size: 20       # Match DB&#39;s max_connections - reserved slots<br>  minimum-idle: 5             # Keep some connections warm<br>  idle-timeout: 60000         # Close idle connections after 1 minute<br>  max-lifetime: 1800000       # Replace connections every 30 minutes<br>  leak-detection-threshold: 2000  # Log leaks after 2 seconds</pre><h3>2. Increase PostgreSQL’s max_connections</h3><p>Edit postgresql.conf and restart:</p><pre>max_connections = 200         # Default is typically 100</pre><p><strong>Caution</strong>: More connections = higher memory usage. Monitor your DB server’s RAM.</p><h3>3. Use PgBouncer for Efficient Pooling</h3><p>PgBouncer acts as a middleware connection pooler, reducing the load on PostgreSQL:</p><pre>max_client_conn = 500         # Client connections to PgBouncer<br>default_pool_size = 20        # Actual PostgreSQL connections</pre><h3>4. Fix Connection Leaks in Code</h3><ul><li><strong>Close resources explicitly</strong>: Use try-with-resources for streams, resultsets, and connections.</li><li><strong>Avoid long transactions</strong>: Keep @Transactional methods short; avoid external API calls inside transactions.</li></ul><p><strong>Example Fix</strong>:</p><pre>// Before: Connection leak risk!<br>Stream&lt;Data&gt; stream = jdbcTemplate.queryForStream(&quot;SELECT ...&quot;);</pre><pre>// After: Automatically close the stream<br>try (Stream&lt;Data&gt; stream = repository.getAllStream()) {<br>    return stream.map(...).toList();<br>}</pre><h3>5. Monitor and Optimize Queries</h3><ul><li>Log slow queries:</li></ul><pre>ALTER DATABASE mydb SET log_min_duration_statement = 2000; # Log &gt;2s queries</pre><p>Use indexes to reduce query execution time.</p><h3>Why Idle Connections Aren’t Reused During High Load</h3><p>Under heavy traffic, even with idle connections available, HikariCP might create new ones because:</p><ul><li><strong>Validation Overhead</strong>: Checking idle connections for validity takes time.</li><li><strong>Concurrency Spikes</strong>: 100 requests hitting a pool of 20 connections will force new ones.</li><li><strong>Pool Warm-Up Delay</strong>: If minimum-idle is too low, the pool isn’t prepped for sudden demand.</li></ul><p><strong>Solution</strong>:</p><ul><li>Increase minimum-idle to keep more connections ready.</li><li>Lower connection-timeout to fail fast instead of waiting for validation.</li></ul><h3>Key Takeaways</h3><ol><li><strong>Configure HikariCP Carefully</strong>: Match maximum-pool-size to your database’s limits.</li><li><strong>Monitor Connections</strong>: Use tools like PgBouncer and PostgreSQL’s pg_stat_activity.</li><li><strong>Prevent Leaks</strong>: Always close resources and avoid long transactions.</li><li><strong>Optimize Queries</strong>: Slow queries hog connections — indexes and tuning are your friends.</li><li><strong>Plan for Spikes</strong>: Adjust minimum-idle and max-lifetime to handle traffic bursts gracefully.</li></ol><p>By understanding HikariCP’s behavior and aligning it with your database’s capabilities, you can prevent connection slot exhaustion and keep your application running smoothly — even under pressure.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=3203217a14a0" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Elevate Your Search RAG with Structured Queries]]></title>
            <link>https://medium.com/@raphy.26.007/elevate-your-search-rag-with-structured-queries-6e409cf26c63?source=rss-4e4f9b071bde------2</link>
            <guid isPermaLink="false">https://medium.com/p/6e409cf26c63</guid>
            <category><![CDATA[rags]]></category>
            <category><![CDATA[elasticsearch]]></category>
            <dc:creator><![CDATA[Rafi Syed]]></dc:creator>
            <pubDate>Mon, 23 Dec 2024 06:48:03 GMT</pubDate>
            <atom:updated>2024-12-23T06:48:41.150Z</atom:updated>
            <content:encoded><![CDATA[<p><strong>1. Introduction</strong></p><p>In the world of search, from e-commerce to knowledge management, the precision of results can make all the difference. While <strong>free-text</strong> queries are a common approach, they often miss the mark when it comes to understanding user intent, especially with numeric or specific attribute constraints. That’s where <strong>Retrieval-Augmented Generation (RAG)</strong> paired with <strong>Gemini</strong> comes into play.</p><p>Large Language Models (LLM), can parse natural language queries into structured filters, which, when combined with <strong>Elasticsearch</strong>, deliver pinpoint accuracy. In this guide, we’ll:</p><ol><li>Contrast <strong>free-text</strong> with <strong>Gemini-structured</strong> queries.</li></ol><p>2. Provide <strong>real code snippets</strong> and <strong>cURL</strong> commands to illustrate the implementation.</p><p>3. Measure <strong>time</strong> and <strong>relevancy</strong> to demonstrate the superiority of structured queries.</p><p>Let’s dive into enhancing your search game!</p><p><strong>2. Dataset &amp; Index Setup</strong></p><p>Let’s start with a small but illustrative product dataset, featuring these fields:</p><ul><li><strong>brand</strong></li><li><strong>color</strong></li><li><strong>location</strong></li><li><strong>price</strong></li><li><strong>category</strong></li><li><strong>description</strong></li></ul><p><strong>Sample Document</strong>:</p><p>json</p><pre>{<br>  &quot;id&quot;: 1,<br>  &quot;brand&quot;: &quot;TechBrand&quot;,<br>  &quot;color&quot;: &quot;silver&quot;,<br>  &quot;location&quot;: &quot;New York&quot;,<br>  &quot;price&quot;: 999,<br>  &quot;category&quot;: &quot;laptop&quot;,<br>  &quot;description&quot;: &quot;Sleek silver laptop under $1k.&quot;<br>}</pre><p><strong>Elasticsearch Index</strong> (products_index):js</p><pre>{<br>  &quot;mappings&quot;: {<br>    &quot;properties&quot;: {<br>      &quot;brand&quot;: {&quot;type&quot;: &quot;text&quot;},<br>      &quot;color&quot;: {&quot;type&quot;: &quot;text&quot;},<br>      &quot;location&quot;: {&quot;type&quot;: &quot;text&quot;},<br>      &quot;price&quot;: {&quot;type&quot;: &quot;integer&quot;},<br>      &quot;category&quot;: {&quot;type&quot;: &quot;text&quot;},<br>      &quot;description&quot;: {&quot;type&quot;: &quot;text&quot;}<br>    }<br>  }<br>}</pre><p><strong>3. Free-Text Query (Multi Match)</strong></p><p><strong>Example:</strong></p><p>“Find me a silver laptop under $1000 in New York.”</p><p>Here’s the naive approach using Elasticsearch’s multi_match:</p><pre>{<br>  &quot;query&quot;: {<br>    &quot;multi_match&quot;: {<br>      &quot;query&quot;: &quot;Find me a silver laptop under $1000 in New York&quot;,<br>      &quot;fields&quot;: [&quot;brand&quot;, &quot;color&quot;, &quot;location&quot;, &quot;category&quot;, &quot;description&quot;]<br>    }<br>  }<br>}</pre><p><strong>Issue</strong>:</p><ul><li>Words like “under $1000” are treated as literal text, not recognized as a price filter.</li><li>This can lead to irrelevant results or miss precise matches.</li></ul><p><strong>4. Gemini-Structured Query</strong></p><p><strong>Intent:</strong></p><p>Transform the free-text query into actionable filters:</p><ul><li><strong>color</strong>: “silver”</li><li><strong>location</strong>: “New York”</li><li><strong>category</strong>: “laptop”</li><li><strong>price</strong>: “under $1000” (in reality, we’d convert this into a numeric range)</li></ul><p><strong>Structured Elasticsearch Bool Query</strong> (simplified):</p><pre>{<br>  &quot;query&quot;: {<br>    &quot;bool&quot;: {<br>      &quot;must&quot;: [<br>        {&quot;match&quot;: {&quot;color&quot;: &quot;silver&quot;}},<br>        {&quot;match&quot;: {&quot;location&quot;: &quot;New York&quot;}},<br>        {&quot;match&quot;: {&quot;category&quot;: &quot;laptop&quot;}}<br>      ]<br>    }<br>  }<br>}</pre><p>Note: We’ve omitted numeric range logic for brevity, but you’d use {“range”: {“price”: {“lte”: 1000}}} for actual implementation.</p><p><strong>5. Harnessing the Gemini API with cURL</strong></p><p>To automate this transformation, you can engage Gemini directly with a cURL command:</p><pre>curl -X POST &quot;https://generativelanguage.googleapis.com/v1/models/gemini-pro:generateContent?key=YOUR_GEMINI_API_KEY&quot; \<br>  -H &quot;Content-Type: application/json&quot; \<br>  -d &#39;{<br>    &quot;contents&quot;: [<br>      {<br>        &quot;role&quot;: &quot;user&quot;,<br>        &quot;parts&quot;: [<br>          {<br>            &quot;text&quot;: &quot;Transform this user query into a structured product filter:\n\&quot;Find me a silver laptop under $1000 in New York\&quot;\nFormat: { \&quot;color\&quot;: \&quot;&lt;color&gt;\&quot;, \&quot;price\&quot;: \&quot;&lt;price&gt;\&quot;, \&quot;location\&quot;: \&quot;&lt;location&gt;\&quot;, \&quot;category\&quot;: \&quot;&lt;category&gt;\&quot; }&quot;<br>          }<br>        ]<br>      }<br>    ],<br>    &quot;generationConfig&quot;: {<br>      &quot;temperature&quot;: 0.7,<br>      &quot;maxOutputTokens&quot;: 1024<br>    }<br>  }&#39;</pre><p><strong>Expected Response</strong>:</p><pre>{<br>  &quot;candidates&quot;: [<br>    {<br>      &quot;content&quot;: {<br>        &quot;parts&quot;: [<br>          {<br>            &quot;text&quot;: &quot;{ \&quot;color\&quot;: \&quot;silver\&quot;, \&quot;price\&quot;: \&quot;under $1000\&quot;, \&quot;location\&quot;: \&quot;New York\&quot;, \&quot;category\&quot;: \&quot;laptop\&quot; }&quot;<br>          }<br>        ],<br>        &quot;role&quot;: &quot;model&quot;<br>      }<br>    }<br>  ]<br>}</pre><p>From here, you construct your Elasticsearch query with precision.</p><p><strong>6. Measuring Time &amp; Relevancy</strong></p><p>We focus on:</p><ol><li><strong>Retrieval Time</strong>: How quickly can we find what we need?</li><li><strong>Relevancy</strong>: How well do our results match our expectations?</li></ol><p><strong>Interpretation</strong>:</p><ul><li>Structured queries not only found the perfect match but did so faster, avoiding the noise of misinterpreted terms.</li></ul><p><strong>7. Python in Action</strong></p><p>Here’s a quick Python script to compare both approaches:</p><p>python</p><pre>import requests<br>import time</pre><pre>ES_URL = &quot;https://my-es-domain.com/products_index&quot;</pre><pre>def free_text_search():<br>    body = {<br>        &quot;query&quot;: {<br>            &quot;multi_match&quot;: {<br>                &quot;query&quot;: &quot;Find me a silver laptop under $1000 in New York&quot;,<br>                &quot;fields&quot;: [&quot;brand&quot;, &quot;color&quot;, &quot;location&quot;, &quot;category&quot;, &quot;description&quot;]<br>            }<br>        }<br>    }<br>    return requests.post(f&quot;{ES_URL}/_search&quot;, json=body)</pre><pre>def gemini_structured_search():<br>    # Assuming Gemini&#39;s response: {&quot;color&quot;:&quot;silver&quot;,&quot;location&quot;:&quot;New York&quot;,&quot;category&quot;:&quot;laptop&quot;}<br>    body = {<br>        &quot;query&quot;: {<br>            &quot;bool&quot;: {<br>                &quot;must&quot;: [<br>                    {&quot;match&quot;: {&quot;color&quot;: &quot;silver&quot;}},<br>                    {&quot;match&quot;: {&quot;location&quot;: &quot;New York&quot;}},<br>                    {&quot;match&quot;: {&quot;category&quot;: &quot;laptop&quot;}}<br>                ]<br>            }<br>        }<br>    }<br>    return requests.post(f&quot;{ES_URL}/_search&quot;, json=body)</pre><pre># Timing the search<br>start = time.time()<br>resp_ft = free_text_search()<br>time_ft = time.time() - start</pre><pre>start = time.time()<br>resp_st = gemini_structured_search()<br>time_st = time.time() - start</pre><pre>print(f&quot;Free-text time: {time_ft:.2f} seconds&quot;)<br>print(f&quot;Gemini-structured time: {time_st:.2f} seconds&quot;)</pre><p><strong>8. Conclusion</strong></p><p>Free-text might seem like a straightforward approach, but it often falls short with nuanced queries. By leveraging <strong>Gemini</strong> to interpret and structure user queries before feeding them to <strong>Elasticsearch</strong>, we achieve:</p><ul><li><strong>Higher Precision</strong>: Fewer irrelevant results, more hits on target.</li><li><strong>Better Performance</strong>: Elasticsearch focuses its efforts.</li><li><strong>Enhanced RAG</strong>: Gemini’s understanding enriches the retrieval process, setting the stage for more context-aware generation.</li></ul><p>Next time your search system struggles with user intent, consider this blend of LLM-powered query parsing and Elasticsearch’s powerful bool filtering. It’s a match made in search heaven.</p><pre>import requests<br>import json<br>import time<br><br>###############################################################################<br># CONFIGURATION<br>###############################################################################<br><br>ES_URL = &quot;https://vpc-kepler-es-integration-v1-gsffeklbxeuvx3zx5t3qm3xht4.us-east-1.es.amazonaws.com&quot;<br>INDEX_NAME = &quot;products_index&quot;<br><br>GEMINI_API_URL = &quot;https://generativelanguage.googleapis.com/v1/models/gemini-pro:generateContent&quot;<br>GEMINI_API_KEY = &quot;API_J&quot;  # Replace with your valid Gemini key<br>HEADERS_JSON = {&quot;Content-Type&quot;: &quot;application/json&quot;}<br>HEADERS_NDJSON = {&quot;Content-Type&quot;: &quot;application/x-ndjson&quot;}<br><br>###############################################################################<br># HELPER FUNCTIONS<br>###############################################################################<br><br>def handle_request(url, method, payload=None, headers=None):<br>    &quot;&quot;&quot;<br>    Generic HTTP request handler. Returns JSON response or {} on error.<br>    &quot;&quot;&quot;<br>    try:<br>        response = requests.request(method, url, json=payload, headers=headers)<br>        response.raise_for_status()<br>        return response.json()<br>    except requests.RequestException as e:<br>        print(f&quot;[ERROR] {method.upper()} request failed: {e}&quot;)<br>        if &#39;response&#39; in locals():<br>            print(&quot;[DEBUG] Status Code:&quot;, response.status_code)<br>            print(&quot;[DEBUG] Response:&quot;, response.text)<br>        return {}<br><br>def measure_time(func, *args, **kwargs):<br>    &quot;&quot;&quot;<br>    Measure the execution time of a function call.<br>    Returns (result_of_function, elapsed_time_in_seconds).<br>    &quot;&quot;&quot;<br>    start = time.time()<br>    result = func(*args, **kwargs)<br>    end = time.time()<br>    return result, (end - start)<br><br>###############################################################################<br># ELASTICSEARCH SETUP<br>###############################################################################<br><br>def cleanup_index():<br>    &quot;&quot;&quot;<br>    Delete the existing index if it exists, so we can start fresh.<br>    &quot;&quot;&quot;<br>    delete_url = f&quot;{ES_URL}/{INDEX_NAME}&quot;<br>    print(f&quot;[INFO] Deleting index `{INDEX_NAME}` if it exists...&quot;)<br>    result = handle_request(delete_url, &quot;delete&quot;)<br>    if result == {} or (result.get(&#39;acknowledged&#39;) is True):<br>        print(&quot;[INFO] Index cleaned up (or did not exist).&quot;)<br>    else:<br>        print(&quot;[WARNING] Index deletion might not be confirmed. Response:&quot;, result)<br><br>def create_index():<br>    &quot;&quot;&quot;<br>    Create or update the &#39;products_index&#39; with fields for brand, color, location, etc.<br>    &quot;&quot;&quot;<br>    index_url = f&quot;{ES_URL}/{INDEX_NAME}&quot;<br>    mapping = {<br>        &quot;mappings&quot;: {<br>            &quot;properties&quot;: {<br>                &quot;brand&quot;:      { &quot;type&quot;: &quot;text&quot; },<br>                &quot;color&quot;:      { &quot;type&quot;: &quot;text&quot; },<br>                &quot;location&quot;:   { &quot;type&quot;: &quot;text&quot; },<br>                &quot;price&quot;:      { &quot;type&quot;: &quot;integer&quot; },<br>                &quot;category&quot;:   { &quot;type&quot;: &quot;text&quot; },<br>                &quot;description&quot;:{ &quot;type&quot;: &quot;text&quot; }<br>            }<br>        }<br>    }<br>    print(f&quot;[INFO] Creating/updating `{INDEX_NAME}` index...&quot;)<br>    result = handle_request(index_url, &quot;put&quot;, payload=mapping, headers=HEADERS_JSON)<br>    if result.get(&#39;acknowledged&#39;):<br>        print(f&quot;[SUCCESS] Index `{INDEX_NAME}` created/updated.&quot;)<br>    else:<br>        print(f&quot;[ERROR] Could not create/update index. Response:&quot;, result)<br><br>def get_sample_data():<br>    &quot;&quot;&quot;<br>    Returns a list of sample product data.<br>    Each doc has a unique &#39;id&#39; (used as _id in ES).<br>    &quot;&quot;&quot;<br>    return [<br>        {<br>            &quot;id&quot;: 1,<br>            &quot;brand&quot;: &quot;TechBrand&quot;,<br>            &quot;color&quot;: &quot;silver&quot;,<br>            &quot;location&quot;: &quot;New York&quot;,<br>            &quot;price&quot;: 999,<br>            &quot;category&quot;: &quot;laptop&quot;,<br>            &quot;description&quot;: &quot;Sleek silver laptop under $1k.&quot;<br>        },<br>        {<br>            &quot;id&quot;: 2,<br>            &quot;brand&quot;: &quot;BudgetTech&quot;,<br>            &quot;color&quot;: &quot;black&quot;,<br>            &quot;location&quot;: &quot;New York&quot;,<br>            &quot;price&quot;: 800,<br>            &quot;category&quot;: &quot;laptop&quot;,<br>            &quot;description&quot;: &quot;Budget-friendly black laptop.&quot;<br>        },<br>        {<br>            &quot;id&quot;: 3,<br>            &quot;brand&quot;: &quot;FashionCo&quot;,<br>            &quot;color&quot;: &quot;silver&quot;,<br>            &quot;location&quot;: &quot;Los Angeles&quot;,<br>            &quot;price&quot;: 1200,<br>            &quot;category&quot;: &quot;laptop&quot;,<br>            &quot;description&quot;: &quot;High-end laptop in LA.&quot;<br>        },<br>        {<br>            &quot;id&quot;: 4,<br>            &quot;brand&quot;: &quot;MusicCo&quot;,<br>            &quot;color&quot;: &quot;white&quot;,<br>            &quot;location&quot;: &quot;San Francisco&quot;,<br>            &quot;price&quot;: 200,<br>            &quot;category&quot;: &quot;headset&quot;,<br>            &quot;description&quot;: &quot;White wireless headset for music lovers.&quot;<br>        },<br>        {<br>            &quot;id&quot;: 5,<br>            &quot;brand&quot;: &quot;MusicCo&quot;,<br>            &quot;color&quot;: &quot;black&quot;,<br>            &quot;location&quot;: &quot;Chicago&quot;,<br>            &quot;price&quot;: 250,<br>            &quot;category&quot;: &quot;headset&quot;,<br>            &quot;description&quot;: &quot;Black noise-cancelling headset.&quot;<br>        },<br>        {<br>            &quot;id&quot;: 6,<br>            &quot;brand&quot;: &quot;TabWorld&quot;,<br>            &quot;color&quot;: &quot;gold&quot;,<br>            &quot;location&quot;: &quot;New York&quot;,<br>            &quot;price&quot;: 599,<br>            &quot;category&quot;: &quot;tablet&quot;,<br>            &quot;description&quot;: &quot;Gold tablet with stylus support.&quot;<br>        },<br>        {<br>            &quot;id&quot;: 7,<br>            &quot;brand&quot;: &quot;TechBrand&quot;,<br>            &quot;color&quot;: &quot;silver&quot;,<br>            &quot;location&quot;: &quot;New York&quot;,<br>            &quot;price&quot;: 1099,<br>            &quot;category&quot;: &quot;laptop&quot;,<br>            &quot;description&quot;: &quot;Slightly above $1k silver laptop.&quot;<br>        },<br>        {<br>            &quot;id&quot;: 8,<br>            &quot;brand&quot;: &quot;BudgetTech&quot;,<br>            &quot;color&quot;: &quot;silver&quot;,<br>            &quot;location&quot;: &quot;Boston&quot;,<br>            &quot;price&quot;: 950,<br>            &quot;category&quot;: &quot;laptop&quot;,<br>            &quot;description&quot;: &quot;Affordable silver laptop just under $1k.&quot;<br>        }<br>    ]<br><br>def insert_data():<br>    &quot;&quot;&quot;<br>    Bulk-insert product documents into `products_index`.<br>    &quot;&quot;&quot;<br>    bulk_url = f&quot;{ES_URL}/{INDEX_NAME}/_bulk&quot;<br>    docs = get_sample_data()<br>    lines = []<br>    for doc in docs:<br>        # Action line: index with `_id`<br>        lines.append(json.dumps({&quot;index&quot;: {&quot;_id&quot;: doc[&quot;id&quot;]}}))<br>        # Document line<br>        lines.append(json.dumps(doc))<br><br>    bulk_data_str = &quot;\n&quot;.join(lines) + &quot;\n&quot;<br><br>    print(&quot;[INFO] Inserting documents with _bulk...&quot;)<br>    response = requests.post(bulk_url, data=bulk_data_str, headers=HEADERS_NDJSON)<br>    if response.status_code == 200:<br>        result = response.json()<br>        if result.get(&quot;errors&quot;):<br>            print(&quot;[ERROR] Some documents failed to index:&quot;)<br>            for item in result[&quot;items&quot;]:<br>                if item.get(&quot;index&quot;, {}).get(&quot;status&quot;, 0) &gt;= 400:<br>                    print(&quot; -&quot;, item[&quot;index&quot;])<br>        else:<br>            print(&quot;[SUCCESS] All documents inserted successfully.&quot;)<br>    else:<br>        print(f&quot;[ERROR] Bulk insert failed. Status: {response.status_code}&quot;)<br>        print(&quot;[DEBUG] Response:&quot;, response.text)<br><br>def verify_data():<br>    &quot;&quot;&quot;<br>    Check how many docs exist via match_all, just to confirm insertion.<br>    &quot;&quot;&quot;<br>    search_url = f&quot;{ES_URL}/{INDEX_NAME}/_search&quot;<br>    body = {&quot;query&quot;: {&quot;match_all&quot;: {}}}<br>    print(&quot;[INFO] Verifying data with match_all...&quot;)<br>    resp = requests.post(search_url, json=body, headers=HEADERS_JSON)<br>    if resp.status_code == 200:<br>        es_json = resp.json()<br>        total = es_json[&quot;hits&quot;][&quot;total&quot;][&quot;value&quot;]<br>        print(f&quot;[SUCCESS] Found {total} docs in `{INDEX_NAME}`.&quot;)<br>    else:<br>        print(f&quot;[ERROR] Verification search failed: {resp.status_code}&quot;)<br>        print(&quot;[DEBUG] Response:&quot;, resp.text)<br><br>###############################################################################<br># GEMINI PREPROCESS (TRANSFORM FREE TEXT -&gt; STRUCTURED JSON)<br>###############################################################################<br><br>def gemini_preprocess(query):<br>    &quot;&quot;&quot;<br>    Convert user_query into a structured product filter<br>    { &quot;color&quot;: &quot;&lt;color&gt;&quot;, &quot;price&quot;: &quot;&lt;price&gt;&quot;, &quot;location&quot;: &quot;&lt;location&gt;&quot;, &quot;category&quot;: &quot;&lt;category&gt;&quot; }<br>    using Gemini Pro.<br>    &quot;&quot;&quot;<br>    prompt = f&quot;&quot;&quot;<br>    Transform this user query into a structured product filter:<br>    &quot;{query}&quot;<br>    Format: {{<br>      &quot;color&quot;: &quot;&lt;color&gt;&quot;,<br>      &quot;price&quot;: &quot;&lt;price&gt;&quot;,<br>      &quot;location&quot;: &quot;&lt;location&gt;&quot;,<br>      &quot;category&quot;: &quot;&lt;category&gt;&quot;<br>    }}<br>    &quot;&quot;&quot;<br><br>    payload = {<br>        &quot;contents&quot;: [<br>            {<br>                &quot;role&quot;: &quot;user&quot;,<br>                &quot;parts&quot;: [{&quot;text&quot;: prompt}]<br>            }<br>        ],<br>        &quot;generationConfig&quot;: {<br>            &quot;temperature&quot;: 0.7,<br>            &quot;maxOutputTokens&quot;: 1024<br>        }<br>    }<br><br>    print(&quot;\n[INFO] Sending query to Gemini for preprocessing...&quot;)<br>    print(&quot;[DEBUG] Prompt to Gemini:&quot;, prompt)<br><br>    gemini_url = f&quot;{GEMINI_API_URL}?key={GEMINI_API_KEY}&quot;<br>    response = requests.post(gemini_url, headers=HEADERS_JSON, json=payload)<br>    if response.status_code == 200:<br>        try:<br>            resp_json = response.json()<br>            text_out = resp_json[&quot;candidates&quot;][0][&quot;content&quot;][&quot;parts&quot;][0][&quot;text&quot;]<br>            structured_q = json.loads(text_out)<br>            print(&quot;[DEBUG] Gemini structured output:&quot;, structured_q)<br>            return structured_q<br>        except Exception as e:<br>            print(&quot;[ERROR] Parsing Gemini response:&quot;, e)<br>            return {}<br>    else:<br>        print(f&quot;[ERROR] Gemini API call failed: {response.status_code}&quot;)<br>        print(&quot;[DEBUG] Response:&quot;, response.text)<br>        return {}<br><br>###############################################################################<br># EXAMPLE SEARCH QUERIES<br>###############################################################################<br><br>def free_text_search(query):<br>    &quot;&quot;&quot;<br>    Example free-text multi_match query.<br>    Also prints the ES query body for debugging.<br>    &quot;&quot;&quot;<br>    print(&quot;\n[INFO] Executing FREE-TEXT QUERY (multi_match)...&quot;)<br>    body = {<br>        &quot;query&quot;: {<br>            &quot;multi_match&quot;: {<br>                &quot;query&quot;: query,<br>                &quot;fields&quot;: [&quot;brand&quot;, &quot;color&quot;, &quot;location&quot;, &quot;category&quot;, &quot;description&quot;]<br>            }<br>        }<br>    }<br>    print(&quot;[DEBUG] Free-Text ES Query:\n&quot;, json.dumps(body, indent=2))<br><br>    search_url = f&quot;{ES_URL}/{INDEX_NAME}/_search&quot;<br>    resp = requests.post(search_url, json=body, headers=HEADERS_JSON)<br>    if resp.status_code == 200:<br>        return resp.json()<br>    else:<br>        print(f&quot;[ERROR] Free-text search failed: {resp.status_code}&quot;)<br>        return {}<br><br>def structured_search_from_gemini(structured_q):<br>    &quot;&quot;&quot;<br>    Build a bool query from the structured_q output of gemini_preprocess.<br>    For a real scenario, parse &#39;price&#39; if needed. Here, we handle color, location, category.<br>    Also prints the ES query body for debugging.<br>    &quot;&quot;&quot;<br>    must_clauses = []<br>    if &quot;color&quot; in structured_q:<br>        must_clauses.append({&quot;match&quot;: {&quot;color&quot;: structured_q[&quot;color&quot;]}})<br>    if &quot;location&quot; in structured_q:<br>        must_clauses.append({&quot;match&quot;: {&quot;location&quot;: structured_q[&quot;location&quot;]}})<br>    if &quot;category&quot; in structured_q:<br>        must_clauses.append({&quot;match&quot;: {&quot;category&quot;: structured_q[&quot;category&quot;]}})<br><br>    # If desired, parse structured_q[&quot;price&quot;] for a numeric range.<br>    # e.g. &quot;under $1000&quot; =&gt; &quot;range&quot;: {&quot;price&quot;: {&quot;lte&quot;: 1000}}<br><br>    body = {<br>        &quot;query&quot;: {<br>            &quot;bool&quot;: {<br>                &quot;must&quot;: must_clauses<br>            }<br>        }<br>    }<br>    print(&quot;\n[INFO] Executing GEMINI-STRUCTURED QUERY (bool must)...&quot;)<br>    print(&quot;[DEBUG] Structured ES Query:\n&quot;, json.dumps(body, indent=2))<br><br>    search_url = f&quot;{ES_URL}/{INDEX_NAME}/_search&quot;<br>    resp = requests.post(search_url, json=body, headers=HEADERS_JSON)<br>    if resp.status_code == 200:<br>        return resp.json()<br>    else:<br>        print(f&quot;[ERROR] Structured search failed: {resp.status_code}&quot;)<br>        return {}<br><br>###############################################################################<br># MAIN<br>###############################################################################<br><br>if __name__ == &quot;__main__&quot;:<br>    # 1. Clean up / create index / insert data<br>    cleanup_index()<br>    create_index()<br>    insert_data()<br>    verify_data()<br><br>    # 2. Our user query (the one we want to test)<br>    user_query = &quot;Find me a silver laptop under $1000 in New York&quot;<br><br>    # 3. Free-text approach<br>    (ft_resp, ft_time) = measure_time(free_text_search, user_query)<br>    ft_hits = ft_resp.get(&quot;hits&quot;, {}).get(&quot;hits&quot;, [])<br>    print(f&quot;[INFO] Free-text found {len(ft_hits)} hits in {ft_time:.3f}s&quot;)<br>    for d in ft_hits:<br>        print(&quot;  -&quot;, d[&quot;_source&quot;])<br><br>    # 4. Use Gemini to parse the query =&gt; structured fields =&gt; ES bool query<br>    (gemini_struct, gemini_time) = measure_time(gemini_preprocess, user_query)<br><br>    if gemini_struct:<br>        (st_resp, st_time) = measure_time(structured_search_from_gemini, gemini_struct)<br>        st_hits = st_resp.get(&quot;hits&quot;, {}).get(&quot;hits&quot;, [])<br>        print(f&quot;[INFO] Gemini-structured found {len(st_hits)} hits in {st_time:.3f}s (plus {gemini_time:.3f}s for Gemini).&quot;)<br>        for d in st_hits:<br>            print(&quot;  -&quot;, d[&quot;_source&quot;])<br>    else:<br>        st_hits = []<br>        st_time = 0<br><br>    # 5. Summary<br>    print(&quot;\n=== COMPARISON SUMMARY ===&quot;)<br>    print(f&quot;User Query: &#39;{user_query}&#39;&quot;)<br>    print(f&quot;Free-text =&gt; {len(ft_hits)} hits in {ft_time:.3f}s&quot;)<br>    print(f&quot;Gemini =&gt; {len(st_hits)} hits in {gemini_time+st_time:.3f}s (Gemini + structured query time)&quot;)<br>    print(&quot;[DONE]&quot;)<br></pre><pre>[INFO] Deleting index `products_index` if it exists...<br>[INFO] Index cleaned up (or did not exist).<br>[INFO] Creating/updating `products_index` index...<br>[SUCCESS] Index `products_index` created/updated.<br>[INFO] Inserting documents with _bulk...<br>[SUCCESS] All documents inserted successfully.<br>[INFO] Verifying data with match_all...<br>[SUCCESS] Found 4 docs in `products_index`.<br><br>[INFO] Executing FREE-TEXT QUERY (multi_match)...<br>[DEBUG] Free-Text ES Query:<br> {<br>  &quot;query&quot;: {<br>    &quot;multi_match&quot;: {<br>      &quot;query&quot;: &quot;Find me a silver laptop under $1000 in New York&quot;,<br>      &quot;fields&quot;: [<br>        &quot;brand&quot;,<br>        &quot;color&quot;,<br>        &quot;location&quot;,<br>        &quot;category&quot;,<br>        &quot;description&quot;<br>      ]<br>    }<br>  }<br>}<br>[INFO] Free-text found 6 hits in 0.854s<br>  - {&#39;id&#39;: 3, &#39;brand&#39;: &#39;FashionCo&#39;, &#39;color&#39;: &#39;silver&#39;, &#39;location&#39;: &#39;Los Angeles&#39;, &#39;price&#39;: 1200, &#39;category&#39;: &#39;laptop&#39;, &#39;description&#39;: &#39;High-end laptop in LA.&#39;}<br>  - {&#39;id&#39;: 7, &#39;brand&#39;: &#39;TechBrand&#39;, &#39;color&#39;: &#39;silver&#39;, &#39;location&#39;: &#39;New York&#39;, &#39;price&#39;: 1099, &#39;category&#39;: &#39;laptop&#39;, &#39;description&#39;: &#39;Slightly above $1k silver laptop.&#39;}<br>  - {&#39;id&#39;: 8, &#39;brand&#39;: &#39;BudgetTech&#39;, &#39;color&#39;: &#39;silver&#39;, &#39;location&#39;: &#39;Boston&#39;, &#39;price&#39;: 950, &#39;category&#39;: &#39;laptop&#39;, &#39;description&#39;: &#39;Affordable silver laptop just under $1k.&#39;}<br>  - {&#39;id&#39;: 1, &#39;brand&#39;: &#39;TechBrand&#39;, &#39;color&#39;: &#39;silver&#39;, &#39;location&#39;: &#39;New York&#39;, &#39;price&#39;: 999, &#39;category&#39;: &#39;laptop&#39;, &#39;description&#39;: &#39;Sleek silver laptop under $1k.&#39;}<br>  - {&#39;id&#39;: 2, &#39;brand&#39;: &#39;BudgetTech&#39;, &#39;color&#39;: &#39;black&#39;, &#39;location&#39;: &#39;New York&#39;, &#39;price&#39;: 800, &#39;category&#39;: &#39;laptop&#39;, &#39;description&#39;: &#39;Budget-friendly black laptop.&#39;}<br>  - {&#39;id&#39;: 6, &#39;brand&#39;: &#39;TabWorld&#39;, &#39;color&#39;: &#39;gold&#39;, &#39;location&#39;: &#39;New York&#39;, &#39;price&#39;: 599, &#39;category&#39;: &#39;tablet&#39;, &#39;description&#39;: &#39;Gold tablet with stylus support.&#39;}<br><br>[INFO] Sending query to Gemini for preprocessing...<br>[DEBUG] Prompt to Gemini:<br>    Transform this user query into a structured product filter:<br>    &quot;Find me a silver laptop under $1000 in New York&quot;<br>    Format: {<br>      &quot;color&quot;: &quot;&lt;color&gt;&quot;,<br>      &quot;price&quot;: &quot;&lt;price&gt;&quot;,<br>      &quot;location&quot;: &quot;&lt;location&gt;&quot;,<br>      &quot;category&quot;: &quot;&lt;category&gt;&quot;<br>    }<br><br>[DEBUG] Gemini structured output: {&#39;color&#39;: &#39;silver&#39;, &#39;price&#39;: {&#39;max&#39;: 1000}, &#39;location&#39;: &#39;New York&#39;, &#39;category&#39;: &#39;laptop&#39;}<br><br>[INFO] Executing GEMINI-STRUCTURED QUERY (bool must)...<br>[DEBUG] Structured ES Query:<br> {<br>  &quot;query&quot;: {<br>    &quot;bool&quot;: {<br>      &quot;must&quot;: [<br>        {<br>          &quot;match&quot;: {<br>            &quot;color&quot;: &quot;silver&quot;<br>          }<br>        },<br>        {<br>          &quot;match&quot;: {<br>            &quot;location&quot;: &quot;New York&quot;<br>          }<br>        },<br>        {<br>          &quot;match&quot;: {<br>            &quot;category&quot;: &quot;laptop&quot;<br>          }<br>        }<br>      ]<br>    }<br>  }<br>}<br>[INFO] Gemini-structured found 2 hits in 0.820s (plus 2.764s for Gemini).<br>  - {&#39;id&#39;: 7, &#39;brand&#39;: &#39;TechBrand&#39;, &#39;color&#39;: &#39;silver&#39;, &#39;location&#39;: &#39;New York&#39;, &#39;price&#39;: 1099, &#39;category&#39;: &#39;laptop&#39;, &#39;description&#39;: &#39;Slightly above $1k silver laptop.&#39;}<br>  - {&#39;id&#39;: 1, &#39;brand&#39;: &#39;TechBrand&#39;, &#39;color&#39;: &#39;silver&#39;, &#39;location&#39;: &#39;New York&#39;, &#39;price&#39;: 999, &#39;category&#39;: &#39;laptop&#39;, &#39;description&#39;: &#39;Sleek silver laptop under $1k.&#39;}</pre><p><strong>Thanks for reading!</strong> Share your experiences or questions in the comments below — happy searching!</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=6e409cf26c63" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Mastering Kafka Consumer Configurations: Understanding max.poll.interval.ms and Beyond]]></title>
            <link>https://medium.com/@raphy.26.007/mastering-kafka-consumer-configurations-understanding-max-poll-interval-ms-and-beyon-9eec493be744?source=rss-4e4f9b071bde------2</link>
            <guid isPermaLink="false">https://medium.com/p/9eec493be744</guid>
            <category><![CDATA[kafka]]></category>
            <category><![CDATA[confluent-kafka]]></category>
            <category><![CDATA[java]]></category>
            <dc:creator><![CDATA[Rafi Syed]]></dc:creator>
            <pubDate>Wed, 06 Nov 2024 10:13:54 GMT</pubDate>
            <atom:updated>2024-11-07T09:12:08.592Z</atom:updated>
            <content:encoded><![CDATA[<p><em>A comprehensive guide to optimizing Kafka consumer configurations for real-world applications</em></p><p>Kafka is a powerhouse for real-time data streaming, enabling systems to handle vast amounts of data with remarkable efficiency. However, to harness its full potential, especially when using platforms like Confluent Kafka, understanding and fine-tuning consumer configurations is essential. In this article, we’ll delve deep into one critical configuration — max.poll.interval.ms—and explore scenarios where polling intervals might not remain periodic.</p><h3>What You’ll Learn</h3><ul><li>Essential Kafka consumer configurations</li><li>How to handle non-periodic polling intervals</li><li>Practical examples with real-world numbers</li><li>Best practices for optimizing your Kafka setup</li></ul><h3>Kafka Basics Recap</h3><p>Before we dive into the complexities of configurations, let’s quickly revisit some foundational Kafka concepts that will frame our discussion:</p><ul><li><strong>Producer</strong>: Sends records (messages) to Kafka topics</li><li><strong>Consumer</strong>: Reads records from Kafka topics</li><li><strong>Topic</strong>: A category or feed name to which records are sent</li><li><strong>Partition</strong>: A subdivision of a topic that allows for parallelism</li><li><strong>Consumer Group</strong>: A group of consumers that work together to consume a topic’s partitions</li></ul><h3>Understanding Producer and Consumer Rates</h3><p>Let’s consider a typical scenario to frame our discussion:</p><pre>Producer Rate: 200 records per second<br>Consumer Processing Rate: 100 records per 30 minutes</pre><p>This translates to approximately 3.33 records processed per second per consumer — a relatively slow processing rate that calls for optimization.</p><h3>Key Kafka Consumer Configurations</h3><h3>1. max.poll.records</h3><p>This configuration controls the maximum number of records a consumer fetches in a single poll.</p><pre>max.poll.records=100</pre><h3>2. max.poll.interval.ms</h3><p>This is the maximum delay between two consecutive poll() calls before the consumer is considered failed.</p><pre>max.poll.interval.ms=2000000  # About 33 minutes</pre><h3>3. heartbeat.interval.ms &amp; session.timeout.ms</h3><p>These configurations work together to maintain consumer health monitoring:</p><pre>heartbeat.interval.ms=3000<br>session.timeout.ms=10000</pre><h3>When Polling Isn’t Periodic</h3><blockquote><em>“In the real world, processing times aren’t always predictable.”</em></blockquote><p>There are several scenarios where polling intervals might become non-periodic:</p><ol><li><strong>Variable Processing Times</strong></li></ol><ul><li>Processing time fluctuates based on data complexity</li><li>Occasional spikes from 30 to 40 minutes</li></ul><p><strong>2. Long-Running Operations</strong></p><ul><li>Heavy data transformations</li><li>External API calls</li><li>Database operations</li></ul><p><strong>3. Resource Contention</strong></p><ul><li>High system load</li><li>Memory pressure</li><li>I/O bottlenecks</li></ul><h3>Practical Solutions</h3><h3>Strategy 1: Adjust Configuration Values</h3><pre># For processing times up to 40 minutes<br>max.poll.interval.ms=2700000  # 45 minutes</pre><p>In some cases consumers will take more time in processing its better we can tweak the configurations on max.poll.records (Suggest have smaller value for batch size in-case of the time taking consumers to avoid consumer-lags), also adjust on the max.poll.interval.ms (sugggest give more value than processing time), adjust also the session.time.ms and heartbeat.ms</p><h3>Strategy 2: Implement Parallel Processing</h3><pre>// Example pseudo-code<br>while (true) {<br>    ConsumerRecords&lt;K, V&gt; records = consumer.poll(Duration.ofMillis(100));<br>    ExecutorService executor = Executors.newFixedThreadPool(3);<br>    <br>    for (ConsumerRecord&lt;K, V&gt; record : records) {<br>        executor.submit(() -&gt; processRecord(record));<br>    }<br>}</pre><h3>Strategy 3: Monitor and Scale</h3><p>Set up monitoring for:</p><ul><li>Consumer lag</li><li>Processing times</li><li>Offset commit rates</li><li>System resource utilization</li></ul><h3>Example of Real-World Challenges: Offset Commit Failures Due to Rebalancing</h3><p>In practice, Kafka consumers may encounter situations where processing takes longer than expected, leading to challenges in committing offsets and potential rebalances. Here’s a log snippet that illustrates a common issue when max.poll.interval.ms or other configuration settings aren’t aligned with the processing time:</p><pre>22:52:56.896 [Consumer clientId=consumer-recon-data-set-unit-sync-data-60, groupId=recon-data-set-unit-sync-data] Failing OffsetCommit request since the consumer is not part of an active group<br>22:52:56.896 Successfully processed and acknowledged unit sync data messages.<br>22:52:56.962 [Consumer clientId=consumer-recon-data-set-unit-sync-data-60, groupId=recon-data-set-unit-sync-data] Failing OffsetCommit request since the consumer is not part of an active group<br>22:52:56.964 [Consumer clientId=consumer-recon-data-set-unit-sync-data-60, groupId=recon-data-set-unit-sync-data] Rebalance failed.<br>22:52:56.964 [Consumer clientId=consumer-recon-data-set-unit-sync-data-60, groupId=recon-data-set-unit-sync-data] (Re-)joining group</pre><p>These logs indicate that, despite successfully processing a batch of messages, the consumer couldn’t commit the offsets because it was no longer part of the active group, causing a rebalance. This issue can lead to unintended message reprocessing.</p><h3>Practical Solutions for Handling Long Processing Times</h3><p>When consumers take longer to process, it’s essential to tweak configurations to prevent consumer lag and frequent rebalances:</p><ol><li><strong>max.poll.records</strong>: Set a smaller batch size for time-intensive consumers to control how much data each poll retrieves, preventing excessive lag.</li><li><strong>max.poll.interval.ms</strong>: Ensure this value is slightly above the maximum processing time to give enough buffer for processing.</li><li><strong>session.timeout.ms</strong> and <strong>heartbeat.interval.ms</strong>: Adjust these settings to ensure the consumer remains in the group without timing out during long processing operations.</li></ol><h3>The Traffic Signal Analogy</h3><p>Think of Kafka’s polling mechanism like a city’s traffic management system:</p><ul><li>Traffic signals = poll() calls</li><li>City monitoring = max.poll.interval.ms</li><li>Traffic flow = data processing</li></ul><p>Just as a city’s traffic system needs regular updates to maintain flow, your Kafka consumers need consistent polling to maintain data processing efficiency.</p><h3>Best Practices Checklist</h3><ul><li>Set max.poll.interval.ms higher than maximum processing time</li><li>Implement proper error handling and retry mechanisms</li><li>Monitor consumer lag and processing times</li><li>Scale horizontally when needed</li><li>Use appropriate batch sizes</li><li>Implement graceful shutdown procedures</li></ul><h3>Conclusion</h3><p>Mastering Kafka consumer configurations is crucial for building robust data streaming applications. By understanding and properly configuring settings like max.poll.interval.ms, you can ensure your consumers handle varying loads efficiently and reliably.</p><p>Remember: The key is finding the right balance between processing time, batch size, and polling intervals for your specific use case.</p><p><em>Did you find this article helpful? What Kafka configurations have you struggled with? Share your experiences in the comments below!</em></p><p><strong>About the Author</strong> [Your bio here]</p><p><strong>Further Reading</strong></p><ul><li><a href="https://kafka.apache.org/documentation/">Apache Kafka Documentation</a></li><li><a href="https://docs.confluent.io/">Confluent Platform Documentation</a></li><li><a href="https://kafka.apache.org/documentation/#consumerconfigs">Kafka Consumer Configurations Guide</a></li></ul><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=9eec493be744" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Click house issues with Zookeeper]]></title>
            <link>https://medium.com/@raphy.26.007/click-house-issues-with-zookeeper-1982475f03f5?source=rss-4e4f9b071bde------2</link>
            <guid isPermaLink="false">https://medium.com/p/1982475f03f5</guid>
            <category><![CDATA[clickhouse]]></category>
            <category><![CDATA[confluent-kafka]]></category>
            <dc:creator><![CDATA[Rafi Syed]]></dc:creator>
            <pubDate>Fri, 26 Jul 2024 17:30:51 GMT</pubDate>
            <atom:updated>2024-07-26T17:30:51.110Z</atom:updated>
            <content:encoded><![CDATA[<p>Click house issues with Zookeeper</p><h4>Overview</h4><p>On July 10th, 2024, we encountered significant issues with our ClickHouse server and Kafka connectors. This article provides a detailed Root Cause Analysis (RCA), observations made during the incident, solutions implemented, and actions taken to enhance monitoring and prevent future occurrences.</p><h4>ClickHouse Server Issue</h4><p>Issue</p><p>The ClickHouse server experienced disruptions when one of the ZooKeeper nodes in our otherwise healthy cluster became unavailable due to memory issues.</p><p>Observations</p><p>As we didnt configure the System alerts on the Clickhouse below things we observed</p><ol><li><strong>Exceptions on ZooKeeper</strong>: During the incident, exceptions were logged on the ZooKeeper nodes. And also One of Clickhouse zookeeper nodes was totally down</li><li><strong>Tables in Read Mode</strong>: System tables in ClickHouse indicated that the tables switched to read-only mode, which impeded normal operations like Engine tables were unable to read the Confluent kafka stream, Explicitly we tried updating the tables problems were persisting</li></ol><h4>Kafka Connector Issue</h4><p>Simultaneously, we faced issues related to Kafka connectors, which exacerbated the situation.</p><h4>Solution Implemented</h4><p>Increase PVC Size for ClickHouse-ZooKeeper</p><p><strong>Action</strong>: The DevOps team increased the Persistent Volume Claim (PVC) size for ClickHouse-ZooKeeper from 20GB to 50GB for each of the two ZooKeeper nodes. This action was critical in ensuring that memory issues were mitigated.</p><h4>References</h4><ul><li><a href="https://github.com/ClickHouse/ClickHouse/issues/44597"><strong>ClickHouse Issue #44597</strong></a></li><li><a href="https://github.com/ClickHouse/ClickHouse/issues/31052"><strong>ClickHouse Issue #31052</strong></a></li><li><a href="https://stackoverflow.com/"><strong>Stack Overflow Discussion</strong></a></li></ul><p>These references and actions have been pivotal in improving the resilience and early detection of issues related to ClickHouse and Kafka connectors, ensuring smoother operations and quicker responses to potential incidents.</p><p>Also there were some issues as the data was missing on the clickhouse completely we performed simple procedure to recover the 3 days of data</p><h3>1. Dry Run Reset the Consumer Group</h3><pre>kafka-consumer-groups -bootstrap-server confluent-kafka-server:9092 \ <br>-command-config /Users/xyz/kafka/client_config.properties \ <br>-group unit-metrics-data-group-new \ -reset-offsets <br>-to-datetime 2024–07–06T00:00:00.000 \ <br>-topic unit-metrics-topic -timeout 100000 — dry-run</pre><h3>2. Real query for resetting the consumer group</h3><pre>kafka-consumer-groups — bootstrap-server your_broker_list:9092 \ <br>-command-config /Users/xyz/kafka/client_config.properties \ <br>-group unit-metrics-data-group-new \ <br>— reset-offsets -to-datetime 2024–07–06T00:00:00.000 \ <br>— topic unit-metrics-topic — timeout 100000 — execute</pre><h3>3. Create a New Kafka Engine Table</h3><p>First, we need to create a new Kafka engine table to capture streaming data. This table will receive data from Kafka topics and store it in ClickHouse.</p><pre>CREATE TABLE my_database.new_kafka_table (<br> `id` String, <br> `event_time` DateTime64(3, ‘UTC’), <br> `metric` Float32<br>) ENGINE = Kafka <br>SETTINGS kafka_broker_list = ‘your_broker_list:9092’, <br>kafka_topic_list = ‘your_topic_list’, <br>kafka_group_name = ‘your_group_name’, <br>kafka_skip_broken_messages = 1, <br>kafka_format = ‘JSONEachRow’, <br>kafka_num_consumers = 5;</pre><h3>4. Describe Consumer Group</h3><p>To check the details of the consumer group, use the following Kafka command:</p><pre><br>kafka-consumer-groups — bootstrap-server your_broker_list:9092 \<br>-describe — group your_group_name \<br>-command-config /path/to/client_config.properties \<br>-timeout 100000</pre><h3>5. Create a Temporary Table</h3><p>Next, create a temporary table in ClickHouse. This table will store the intermediate data before it is processed and inserted into the main table.</p><pre><br>CREATE TABLE my_database.temp_table (<br> `id` String, <br> `event_time` DateTime64(3, ‘UTC’), <br> `metric` Float32<br>) ENGINE = ReplicatedReplacingMergeTree(‘/clickhouse/tables/0/temp_table’, ‘ch1’) <br>PARTITION BY toDate(event_time) <br>PRIMARY KEY id <br>ORDER BY (id, event_time) <br>SETTINGS index_granularity = 8192;<br></pre><h3>6. Create Materialized View</h3><p>Create a materialized view to populate the `temp_table` with data from the Kafka engine table.</p><pre><br>CREATE MATERIALIZED VIEW my_database.mv_temp_table TO my_database.temp_table AS<br>SELECT <br> id, <br> event_time, <br> metric <br>FROM my_database.new_kafka_table;<br></pre><h3>7. Check for Data Differences Before Inserting</h3><p>Before inserting new data into the main table, check for differences between new and existing data.</p><pre><br>SELECT <br> new_table.id, <br> new_table.event_time, <br> new_table.metric<br>FROM <br> my_database.temp_table AS new_table <br>WHERE <br> new_table.event_time &gt; ‘2024–07–07’ <br> AND new_table.event_time &lt; ‘2024–07–10’ <br>EXCEPT <br>SELECT <br> old_table.id, <br> old_table.event_time, <br> old_table.metric<br>FROM <br> my_database.unit_metrics AS old_table <br>WHERE <br> old_table.event_time &gt; ‘2024–07–07’ <br> AND old_table.event_time &lt; ‘2024–07–10’;<br></pre><h3>8. Insert the Data</h3><p>Insert new records into the `unit_metrics` table by selecting from the `temp_table` and excluding already existing records.</p><pre><br>INSERT INTO my_database.unit_metrics (id, event_time, metric)<br>SELECT <br> new_table.id, <br> new_table.event_time, <br> new_table.metric<br>FROM <br> my_database.temp_table AS new_table <br>WHERE <br> new_table.event_time &gt; ‘2024–06–01’ <br> AND new_table.event_time &lt; ‘2024–07–10’ <br>EXCEPT <br>SELECT <br> old_table.id, <br> old_table.event_time, <br> old_table.metric<br>FROM <br> my_database.unit_metrics AS old_table <br>WHERE <br> old_table.event_time &gt; ‘2024–06–01’ <br> AND old_table.event_time &lt; ‘2024–07–10’;</pre><h3>9. Monitor Kafka Lag</h3><p>To monitor the lag of the consumer group, use the following Kafka command:</p><pre><br>kafka-consumer-groups — bootstrap-server your_broker_list:9092 \<br> — describe — group your_group_name \<br> — command-config /path/to/client_config.properties \<br> — timeout 100000</pre><h3>10. Clean Up Temporary Tables</h3><p>Once the data is successfully migrated, drop the temporary tables and materialized view to free up resources.</p><pre><br>DROP TABLE IF EXISTS my_database.new_kafka_table; <br>DROP TABLE IF EXISTS my_database.temp_table; <br>DROP MATERIALIZED VIEW IF EXISTS my_database.mv_temp_table;<br></pre><h3>Conclusion</h3><p>By following these steps, you can effectively create and manage Kafka engine tables in ClickHouse, process streaming data, and ensure that new data is accurately inserted into the main tables. Monitoring and cleanup steps help maintain the health and efficiency of the system, ensuring smoother operations and quicker response to potential incidents.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=1982475f03f5" width="1" height="1" alt="">]]></content:encoded>
        </item>
    </channel>
</rss>