<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Holidu Tech Blog on Medium]]></title>
        <description><![CDATA[Stories by Holidu Tech Blog on Medium]]></description>
        <link>https://medium.com/@holidu?source=rss-12d4c8214aa6------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/1*gO3Xm0RiBZIedMcSDP4CKA.png</url>
            <title>Stories by Holidu Tech Blog on Medium</title>
            <link>https://medium.com/@holidu?source=rss-12d4c8214aa6------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Fri, 20 Nov 2026 10:16:37 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@holidu/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[Circuit Breakers at Scale]]></title>
            <link>https://medium.com/@holidu/circuit-breakers-at-scale-85075f193b09?source=rss-12d4c8214aa6------2</link>
            <guid isPermaLink="false">https://medium.com/p/85075f193b09</guid>
            <category><![CDATA[circuit-breaker]]></category>
            <category><![CDATA[microservices]]></category>
            <category><![CDATA[backend-development]]></category>
            <category><![CDATA[design-systems]]></category>
            <category><![CDATA[distributed-systems]]></category>
            <dc:creator><![CDATA[Holidu Tech Blog]]></dc:creator>
            <pubDate>Mon, 15 Dec 2025 11:36:48 GMT</pubDate>
            <atom:updated>2025-12-21T07:01:27.407Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*KfzSQkXkBklGDR9-" /><figcaption>Photo by <a href="https://unsplash.com/@findway_jerry?utm_source=medium&amp;utm_medium=referral">Jerry Wei</a> on <a href="https://unsplash.com/?utm_source=medium&amp;utm_medium=referral">Unsplash</a></figcaption></figure><h4><strong>TL;DR</strong></h4><p>At Holidu, we process approximately 300 million pricing requests daily across hundreds of provider APIs. When a provider starts failing, our platform remains stable, but users experience failures or slower responses, reducing engagement and losing bookings. We built an automated circuit breaker system using AWS Athena that detects failures within minutes, isolates unhealthy providers, and recovers them automatically using progressive backoff. The solution eliminates manual intervention, reduces wasted compute, and protects critical user journeys like booking.</p><h3>The Problem</h3><p>In the vacation rental industry, Holidu aggregates inventory from hundreds of property management companies and channel managers. Each provider’s API has different reliability characteristics: some experience periodic maintenance windows, others suffer timeout spikes during peak traffic, and some have intermittent failures due to infrastructure issues.</p><p>Before implementing our circuit breaker system, a failing provider API would continue receiving requests, creating several cascading problems:</p><ul><li><strong>Response times degraded</strong> as the system waited for timeouts from unhealthy providers</li><li>Guest experience suffered when <strong>pricing calls failed at checkout, </strong>the most critical moment in the booking funnel</li><li>Engineers had to <strong>manually detect</strong> and respond to failures, pulling them away from feature work</li><li>We continued spending <strong>AWS compute costs</strong> on requests destined to fail.</li></ul><p>Our detection mechanism relied on business alerts configured in Looker. When failures occurred, engineers had to manually analyze the issue and decide to set a provider offline. This process proved unreliable, alerts were sometimes delayed by minutes or hours. Looker alerts occasionally failed to trigger entirely, and manual response time varied depending on engineer availability.</p><p>The engineering challenge was clear: detect failures quickly enough to matter (within minutes, not hours) while avoiding false positives that would unnecessarily disable healthy providers.</p><h3>The Solution: Circuit Breakers</h3><p>Circuit breakers are one of the most common software patterns for handling external service failures. They serve as a “smart proxy” between your system and external dependencies, preventing cascading failures when services become unavailable.</p><p>The classic circuit breaker has three states:</p><ul><li><strong>CLOSED</strong>: Normal operation, requests flow through</li><li><strong>OPEN</strong>: Failures exceeded threshold, requests are blocked/fast-failed</li><li><strong>HALF-OPEN</strong>: Testing phase, allows limited requests to check if service recovered</li></ul><p>Our implementation automatically detects failures and stops making requests to failing providers, protecting the user experience while allowing automatic recovery.</p><h4>Requirements</h4><p>Before building the solution, we established clear requirements based on our operational experience:</p><p><strong>Detection Speed</strong>: Failures must be detected within 10–15 minutes of occurrence, not hours. This timeframe balances quick response against the risk of false positives from momentary spikes.</p><p><strong>Scale</strong>: The system must handle analysis of 300 million daily requests (~1k-3k requests/sec) without requiring dedicated infrastructure that scales with request volume.</p><p><strong>Automation</strong>: Once a failure is detected, the system must automatically isolate the provider and attempt recovery without human intervention for standard cases.</p><p><strong>Differentiated Thresholds</strong>: Different user journeys have different failure tolerances. Checkout failures are catastrophic, search failures are more tolerable.</p><p><strong>Observability</strong>: Engineers and stakeholders need visibility into circuit breaker state, historical patterns, and alerts for exceptional cases requiring human review.</p><p><strong>Cost Efficiency</strong>: The solution should leverage existing infrastructure where possible and minimize additional operational costs.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/448/1*UV-ctIKAISc2bcOvbV3Ztw.jpeg" /></figure><h4>Where does it live?</h4><p>The circuit breaker lives in our Provider Live Service (PLS). PLS makes requests to provider APIs during real-time user journeys. When the search backend needs availability or pricing data, it calls PLS, which communicates with the respective provider APIs.</p><p>One crucial design decision: we map provider errors to generic response types. This is communicated via a STATUS field — PROVIDER_FAILURE, TIMEOUT, etc. This normalization allows us to detect patterns across diverse provider implementations.</p><h3>Detection: Why Athena?</h3><p>At 300 million pricing requests per day, traditional in-memory circuit breaker patterns don’t scale well. The classic circuit breaker pattern maintains failure counters per downstream service, typically in application memory. At our scale, this approach presents several challenges.</p><p>Maintaining real-time failure counters per provider across distributed service instances would require coordination overhead, having a shared cache like Redis. With hundreds of providers and multiple service instances, the memory footprint becomes significant. In-memory state is lost during deployments or instance failures, potentially causing circuit breakers to reset.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*AUyqBJl2qqAz1Mv0gWZgtg.png" /></figure><p>The tradeoff is latency: Athena queries take seconds versus milliseconds for in-memory lookups. For circuit breaker detection, this is acceptable. We’re optimizing for minutes-scale detection, not milliseconds.</p><h4><strong>Data Pipeline: From Request to Detection</strong></h4><p>Every API call made by PLS flows through our data pipeline before being analyzed.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*SZ15vi41qCT3eIUwf1EHGg.jpeg" /></figure><p><strong>Why this design?</strong> Firehose flushes data when <strong>any</strong> of these conditions is met first: 600 seconds elapsed, 128MB accumulated, or 500,000 records buffered. At our scale (~2,000 requests/second), we hit the <strong>500K record limit in about 120 seconds, </strong>well before the time buffer. The 10–12 minute query window accounts for this plus processing time, ensuring we analyze complete data rather than partial windows that could cause false positives.</p><h4>Different Thresholds</h4><p>The circuit breaker takes a centralized, scheduled approach rather than the traditional per-request pattern. Every <strong>10 minutes</strong>, a scheduled job queries AWS Athena to analyze PLS request logs from the past 12 minutes. Different request types have different tolerance levels based on the use case.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Z_eqKoI2vtxcPYHk3dVwsA.jpeg" /></figure><p><strong>Why different thresholds?</strong> A 30% failure rate on checkout is catastrophic (lost bookings), while a 50% rate on search is more tolerable (users see other properties). The minimum request counts prevent false positives from low-traffic providers.</p><h3>High-Level Architecture</h3><p>The circuit breaker system operates as a scheduled monitoring layer alongside the real-time request flow. The architecture separates concerns cleanly. The request path remains unchanged, while detection runs asynchronously on response data.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/991/1*JUMdV0pW1HXqi1wx-r6yWg.png" /></figure><p><strong>Components</strong>:</p><ol><li><strong>Provider Live Service (PLS)</strong>: Makes requests to provider APIs, logs outcomes</li><li><strong>Kinesis Firehose</strong>: Buffers and batches log data efficiently</li><li><strong>Amazon S3</strong>: Stores log data in queryable format with time-based partitioning</li><li><strong>AWS Athena</strong>: Analyzes failure patterns using SQL queries</li><li><strong>Scheduler</strong>: Orchestrates detection, state management, and alerting</li><li><strong>Redis</strong>: Persists circuit breaker state</li><li><strong>Provider Status API</strong>: Exposes provider state to Search Engine</li></ol><h4>Circuit Breaker State Machine</h4><p>The circuit breaker follows a classic state machine pattern with progressive backoff. We have two schedulers running every 10 minutes:</p><ul><li>Open CB Scheduler: Detects failures from Athena queries and puts the provider offline (<strong>OPEN</strong> state = Provider is <strong>TEMPORARY_OFFLINE</strong>)</li><li>Closed CB Scheduler:<strong> </strong>Gets Redis state timeline of the provider and decides whether it’s time for the provider to be brought back online (<strong>CLOSED</strong> state = Provider is <strong>LIVE</strong>)</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*0EV5K9dMiKkgR9Zhq0LvwA.jpeg" /></figure><p>When a provider trips, it starts with a 1-hour offline period. If it trips again after recovery, the offline period escalates to 2 hours, then 3 hours. After multiple trips, the system escalates to L1 support for manual intervention.</p><p>The states are persisted in Redis so that the Online Scheduler can recover the Provider depending on the back off stage.</p><h4>Channel Manager Grouping</h4><p>Many providers operate through channel managers, which manage multiple sub-providers through a single API connection. The circuit breaker groups all sub-providers of a channel manager together.</p><p>This approach reflects the reality that failures typically stem from the shared API infrastructure rather than individual property issues.</p><h3>Observability &amp; Alerting</h3><h4>Slack Integration</h4><p>When a circuit trips, detailed alerts are sent via our Slack bot app. The alerts include provider states and timeline information, useful for both Engineers and Technical Account Managers (TAMs). Slack threads are updated with current status using color-coded indicators showing circuit state.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*cdMe0FpDBYd2N8A73g6SRw.png" /></figure><p>Also, Slack threads are created to update about current status. It also maintains a color coded approach indicating the status of the circuit in the main messsage.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/426/1*Qs5LoWBxaW0EtYjKI2moEg.png" /></figure><h4><strong>Grafana Alerts</strong></h4><p>We built Grafana dashboards providing visibility into:</p><ul><li><strong>Current State Overview:</strong> To view providers which were triggered, time during state change and trip count</li><li><strong>Historical Patterns: </strong>Circuit trips over time to identify recurring issues</li><li><strong>System monitoring:</strong> Health of scheduler</li><li><strong>L1 Alerts:</strong> Triggered after 4th detection for manual intervention. Provider stays offline.</li></ul><h4>Rolling out</h4><p>We rolled it out in incremental steps:</p><p><strong>Phase I:</strong> Operates during <strong>business hours</strong> (9:00–20:00). Monitor carefully.</p><p><strong>Phase II: Add Timeout Detection</strong> Currently monitoring Provider failures status. Adding Timeout detection will catch slow-responding providers that technically return responses but degrade user experience.</p><p><strong>Phase III: 24/7 Coverage</strong> Extending to round-the-clock monitoring</p><h3>Results and Impact</h3><p>Since launching in June 2025:</p><ul><li><strong>Eliminated manual developer intervention: </strong>No more waiting for Looker alerts or manually setting providers offline</li><li><strong>Improved checkout success rate</strong> by preventing users from reaching checkout with unhealthy providers</li><li><strong>Faster incident response:</strong> Detection within 10–12 minutes vs. hours with manual monitoring</li><li><strong>Automatic recovery: </strong>Healthy providers come back online without human intervention</li><li><strong>Reduced wasted compute</strong> by stopping requests to failing providers</li></ul><h3>Conclusion</h3><p>The Provider Circuit Breaker transforms provider reliability from a manual operational burden into a managed, observable system property. By leveraging Holidu’s existing data pipeline (Kinesis Firehose → S3 → Athena), the system achieves near-real-time failure detection with minimal additional infrastructure.</p><p>The key architectural insight: <strong>Circuit Breakers don’t need to be in the request path.</strong> For systems at our scale, centralized detection on logged data provides sufficient speed while dramatically simplifying implementation.</p><p>For vacation rental marketplaces operating at scale, automated circuit breakers protect guests from checkout failures while giving partners clear visibility into their API health.</p><p><em>Written by </em><a href="https://www.linkedin.com/in/swetharnaik-mec/"><strong><em>Swetha Ramesh Naik</em></strong></a><em>, Senior Backend Engineer</em></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=85075f193b09" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Automating Cross-Account AWS Secrets Backup and Log Indexing with Lambda, S3, and Elasticsearch]]></title>
            <link>https://medium.com/@holidu/automating-cross-account-aws-secrets-backup-and-log-indexing-with-lambda-s3-and-elasticsearch-aede74039105?source=rss-12d4c8214aa6------2</link>
            <guid isPermaLink="false">https://medium.com/p/aede74039105</guid>
            <category><![CDATA[aws]]></category>
            <category><![CDATA[secrets]]></category>
            <category><![CDATA[aws-secrets-manager]]></category>
            <category><![CDATA[disaster-recovery]]></category>
            <dc:creator><![CDATA[Holidu Tech Blog]]></dc:creator>
            <pubDate>Wed, 02 Jul 2025 12:10:17 GMT</pubDate>
            <atom:updated>2025-07-02T12:18:21.929Z</atom:updated>
            <content:encoded><![CDATA[<h3>TL;DR</h3><p>When it comes to security, there’s no such thing as too many backups — especially when dealing with sensitive information like AWS Secrets. At Holidu, we needed a reliable, automated way to back up secrets from production AWS accounts into a secure account — an isolated AWS environment with strict access controls, designed specifically for safely storing sensitive and critical data.</p><p>There is currently no native AWS solution for cross-account secrets backup. We also couldn’t find any readily available third-party tools that offer this functionality out of the box.</p><p>So we built our own.</p><p>In this post, we’ll walk through our custom solution that:<br> • Copies secrets cross-account between AWS Secrets Manager.<br> • Stores logs in an S3 bucket.<br> • Uses a scheduled Lambda function to read logs and index them into an Elasticsearch cluster.<br> • Ties everything together with EventBridge.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*BKw6EhmI8tVj-ZxTgtZSNg.png" /><figcaption>Architecture</figcaption></figure><h3>Requirements</h3><p>Before we began designing our secrets backup system, we laid out a set of clear requirements based on our internal security policies, reliability expectations, and observability needs. The goal was to create a fully automated solution that reduces manual intervention while remaining secure and transparent.</p><p>Here’s what the ideal system needed to achieve:</p><ol><li><strong>Cross-Account Secrets Backup</strong></li></ol><p>Automatically back up all secrets from our production AWS account into a secure, access-restricted backup account — without exposing credentials or granting overly broad permissions.</p><p><strong>2. Immutable, Auditable Logging</strong></p><p>Every operation in the backup process should be logged in structured JSON format and stored in a tamper-proof way (e.g., S3 with limited write access and lifecycle policies).</p><p><strong>3. Separation of Concerns</strong></p><p>The secure account is used solely for backups. Monitoring and alerting should happen entirely from the production account to maintain access restrictions.</p><p><strong>4. Near Real-Time Observability</strong></p><p>Logs should be indexed into a searchable system (e.g., Elasticsearch), with dashboards and alerts to help us understand the system’s health and catch failures early.</p><p><strong>5. Scheduled and Reliable Execution</strong></p><p>The entire pipeline should run on a fixed schedule (daily), using native AWS services that are serverless, reliable, and cost-effective.</p><p><strong>6. Infrastructure as Code (Optional but Preferred)</strong></p><p>Wherever possible, deployment and configuration should be handled using Infrastructure as Code tools like <strong>Terraform</strong>, enabling repeatability, version control, and ease of maintenance across accounts.</p><h3>System Design</h3><p>If you’ve already had a chance to look into our schema, you might be wondering why we architected this solution with two separate Lambda functions deployed across two different AWS accounts. The short answer: security boundaries and observability.</p><p>Our organization maintains a secure backup account that’s intentionally locked down. Only a small, trusted group has access to it. On the other hand, our production accounts are more accessible, allowing for day-to-day troubleshooting and monitoring. This separation ensures that even if the production environment is compromised, our backups remain safe and untampered.</p><p>Here’s how the system works:</p><h4><strong>1. Lambda in the Secure Account</strong></h4><p>This Lambda function is the core of the backup process. It:</p><ul><li>Has cross-account IAM permissions to <strong>read secrets</strong> from the production account.</li><li>Writes the secrets into the <strong>AWS Secrets Manager</strong> of the secure account, with a unique prefix to track origin.</li><li>Logs every step of the process in structured <strong>JSON format.</strong></li><li>Stores those logs in a designated <strong>S3 bucket</strong> under the path format logs/YYYY/MM/DD/.</li></ul><p>Everything is written in <strong>Python</strong>, ensuring clean, modular logic that’s easy to maintain.</p><h4><strong>2. Lambda in the Production Account</strong></h4><p>Once the log files are written to S3, an <strong>S3 Event Notification</strong> triggers the second Lambda function — this one running in the production account. Its job is to:</p><ul><li>Read the log entries from S3.</li><li>Parse and format them.</li><li>Index them into our Elasticsearch cluster.</li></ul><p>This allows us to monitor the backup process in near real-time — <strong>without needing direct access to the secure account</strong>.</p><h4><strong>3. Dashboards and Alerts via Grafana</strong></h4><p>With <strong>Elasticsearch as our primary observability layer</strong>, we integrated <strong>Grafana</strong> to build dashboards that provide a high-level view of the backup operations. Using the <strong>Elasticsearch data source</strong>, we track logs with ERROR status to quickly identify and investigate any failures in the process. In parallel, we use the <strong>CloudWatch data source</strong> to monitor Lambda invocations, ensuring that our backup function runs on schedule every day. Together, these dashboards and alerts give us confidence that the system is both reliable and transparent.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*d8KYs5wmW42hQA53t49J4Q.png" /><figcaption>Grafana Errors Rate and Invocation panels</figcaption></figure><h4><strong>4. Automating the Pipeline with EventBridge Scheduled Rules</strong></h4><p>To kick off this entire pipeline, we use an Amazon EventBridge scheduled rule (cron job) that triggers the secrets backup Lambda on a daily basis. EventBridge integrates seamlessly with Lambda, requires no additional infrastructure, and is extremely cost-efficient for our use case.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*IBRkEfm4WAvP1jYn2UMRPw.png" /><figcaption>EventBridge schedule configuration</figcaption></figure><p>EventBridge’s pricing is incredibly generous for low-frequency schedules like ours. Here’s the breakdown:</p><ul><li><strong>First 100,000 events per month</strong> are free during the <strong>12-month AWS Free Tier</strong> period.</li><li>After the first year, AWS charges <strong>$1.00 per 1 million events</strong> or <strong>$0.000001 per event</strong>.</li></ul><p>Since our secrets backup job runs once daily:</p><ul><li><strong>365 events per year × $0.000001 = $0.000365 per year</strong></li></ul><p>That’s less than one-tenth of a cent per year, making this one of the most <strong>cost-efficient scheduling solutions</strong> available in the AWS ecosystem.</p><h3>Implementation</h3><blockquote>For the purposes of the following steps, let’s assume the <strong>target secure account</strong> has account_id = 111111111111, and the <strong>live account</strong> has account_id = 222222222222.</blockquote><blockquote>Please note that the implementation of the Python functions referenced in this article is <strong>left to you</strong>. You are free to choose the approach you’re most comfortable with — whether that’s a <strong>manual process</strong>, <strong>Terraform</strong>, <strong>Pulumi</strong>, or another method.</blockquote><blockquote>We provide the <strong>core idea</strong> and <strong>examples of cross-account permissions</strong>, which in our implementation have been <strong>automated using Terraform</strong>. However, the actual code and design will vary depending on your organization’s needs and system architecture.</blockquote><h4>Implementation steps (LIVE Account)</h4><ol><li>Create cross-account role SecretsReadRoleForSecureAccount and permissions policy ReadOnlySecretsManager to read AWS secrets in Live account by Lambda function hosted in Secure account. Role’s trusted entities should look like:</li></ol><pre>{<br>    &quot;Version&quot;: &quot;2012-10-17&quot;,<br>    &quot;Statement&quot;: [<br>        {<br>            &quot;Effect&quot;: &quot;Allow&quot;,<br>            &quot;Principal&quot;: {<br>                &quot;AWS&quot;: &quot;arn:aws:iam::111111111111:role/lambda_backup_secrets_from_live_to_secure&quot;<br>            },<br>            &quot;Action&quot;: &quot;sts:AssumeRole&quot;<br>        }<br>    ]<br>}</pre><p>ReadOnlySecretsManager permissions policy to be created and assigned to the role:</p><pre>{<br>    &quot;Statement&quot;: [<br>        {<br>            &quot;Action&quot;: [<br>                &quot;secretsmanager:ListSecrets&quot;,<br>                &quot;secretsmanager:GetSecretValue&quot;,<br>                &quot;secretsmanager:DescribeSecret&quot;<br>            ],<br>            &quot;Effect&quot;: &quot;Allow&quot;,<br>            &quot;Resource&quot;: &quot;*&quot;<br>        }<br>    ],<br>    &quot;Version&quot;: &quot;2012-10-17&quot;<br>}</pre><p>2. Create S3CrossAccountRoleForLambdaToSecureAccount role and necessary permissions policies to let Lambda function get access to S3 bucket logs located in Secure account. Role’s trusted entities:</p><pre>{<br>    &quot;Version&quot;: &quot;2012-10-17&quot;,<br>    &quot;Statement&quot;: [<br>        {<br>            &quot;Effect&quot;: &quot;Allow&quot;,<br>            &quot;Principal&quot;: {<br>                &quot;Service&quot;: &quot;lambda.amazonaws.com&quot;<br>            },<br>            &quot;Action&quot;: &quot;sts:AssumeRole&quot;<br>        }<br>    ]<br>}</pre><p>ReadOnlyLambdaToSecureS3 permissions policy to be created and assigned to the role:</p><pre>{<br>    &quot;Statement&quot;: [<br>        {<br>            &quot;Action&quot;: [<br>                &quot;s3:GetObject&quot;,<br>                &quot;s3:ListBucket&quot;<br>            ],<br>            &quot;Effect&quot;: &quot;Allow&quot;,<br>            &quot;Resource&quot;: [<br>                &quot;arn:aws:s3:::YOUR_S3_BUCKET_NAME&quot;,<br>                &quot;arn:aws:s3:::YOUR_S3_BUCKET_NAME/*&quot;<br>            ]<br>        }<br>    ],<br>    &quot;Version&quot;: &quot;2012-10-17&quot;<br>}</pre><p>AWSLambdaBasicExecutionRole managed AWS policy to be assigned to the role:</p><pre>{<br>    &quot;Version&quot;: &quot;2012-10-17&quot;,<br>    &quot;Statement&quot;: [<br>        {<br>            &quot;Effect&quot;: &quot;Allow&quot;,<br>            &quot;Action&quot;: [<br>                &quot;logs:CreateLogGroup&quot;,<br>                &quot;logs:CreateLogStream&quot;,<br>                &quot;logs:PutLogEvents&quot;<br>            ],<br>            &quot;Resource&quot;: &quot;*&quot;<br>        }<br>    ]<br>}</pre><p>3. Create lambda_function.py in Live account which will be performing reading cross-account logs, parse and inject them into ElasticSearch. Make sure to link your Lambda function to the correct VPC within your infrastructure and increase default timeout value to 5 mins.</p><p>4. Push Lambda function code in ZIP archive via CLI, because by default, AWS Lambda <strong>does not include external libraries</strong> like elasticsearch, so you need to package them yourself.</p><pre>import boto3<br>import os<br>import json<br>from elasticsearch import Elasticsearch, helpers<br>from urllib.parse import unquote_plus<br>from datetime import datetime<br>from zoneinfo import ZoneInfo <br>import logging<br>import uuid<br><br>logger = logging.getLogger()<br>logger.setLevel(logging.INFO)<br><br>def connect_to_elasticsearch():<br>    es_host = os.environ[&#39;ES_HOST&#39;]<br><br>    es = Elasticsearch(<br>        hosts=[es_host],<br>        use_ssl=es_host.startswith(&quot;https&quot;),<br>        verify_certs=False<br>    )<br><br>    return es<br><br>def lambda_handler(event, context):<br>    s3 = boto3.client(&#39;s3&#39;)<br>    es = connect_to_elasticsearch()<br><br>    for record in event.get(&#39;Records&#39;, []):<br>        bucket = record[&#39;s3&#39;][&#39;bucket&#39;][&#39;name&#39;]<br>        key = unquote_plus(record[&#39;s3&#39;][&#39;object&#39;][&#39;key&#39;])<br><br>        try:<br>            response = s3.get_object(Bucket=bucket, Key=key)<br>            content = response[&#39;Body&#39;].read().decode(&#39;utf-8&#39;)<br>            log_data = json.loads(content)<br><br>            if not isinstance(log_data, list):<br>                log_data = [log_data]<br><br>            logger.info(f&quot;Read {len(log_data)} entries from {key}&quot;)<br><br>            # Local timezone<br>            local_time = datetime.now(ZoneInfo(&quot;Europe/Berlin&quot;))<br>            index_name = f&quot;YOUR_INDEX_NAME-{local_time.strftime(&#39;%Y.%m.%d&#39;)}&quot;<br><br>            actions = [<br>                {<br>                    &quot;_index&quot;: index_name,<br>                    &quot;_id&quot;: str(uuid.uuid4()),<br>                    &quot;_source&quot;: entry<br>                } for entry in log_data<br>            ]<br><br>            success, failed = helpers.bulk(es, actions, stats_only=True, raise_on_error=False)<br><br>            logger.info(f&quot;Bulk indexing done: {success} succeeded, {failed} failed.&quot;)<br><br>            if failed &gt; 0:<br>                for entry in log_data:<br>                    try:<br>                        es.index(index=index_name, body=entry)<br>                    except Exception as e:<br>                        logger.error(f&quot;Failed to index single document: {e}&quot;)<br><br>            logger.info(f&quot;Successfully processed {key}&quot;)<br><br>        except Exception as e:<br>            logger.error(f&quot;Error processing {key} from bucket {bucket}: {str(e)}&quot;, exc_info=True)<br>            continue<br><br>    return {<br>        &#39;statusCode&#39;: 200,<br>        &#39;body&#39;: json.dumps(&#39;Processing complete!&#39;)<br>    }</pre><pre>python3 -m venv venv<br>source venv/bin/activate<br><br>pip install --upgrade pip<br>pip install -r requirements.txt<br><br>mkdir python_package<br>cp python_package<br>cp -r ../venv/lib/python3.*/site-packages/* .<br>cp -r ../venv/lib/python3.*/site-packages/.* . 2&gt;/dev/null || true # Include hidden files<br>cp ../lambda_function.py .<br><br>zip -r9 ../lambda_function.zip .<br>aws lambda update-function-code --function-name lambda_secrets_logs_es_ingestion_from_secure_account --zip-file fileb://lambda_function.zip</pre><p>5. Set the ES_HOST environment variable to your Elasticsearch URL on Lambda function.</p><p>6. Create <strong>Resource-based policy statements</strong> for Lambda located in Live account.</p><pre>aws lambda add-permission \<br>--function-name lambda_secrets_logs_es_ingestion_from_secure_account \<br>--statement-id AllowS3InvokeFromLiveAccount \<br>--action lambda:InvokeFunction \<br>--principal s3.amazonaws.com \<br>--source-account 111111111111 \<br>--source-arn arn:aws:s3:::YOUR_S3_BUCKET_NAME</pre><h4>Implementation steps (Secure Account)</h4><ol><li>Create lambda_backup_secrets_from_live_to_secure role with necessary permissions to assume SecretsReadRoleForSecureAccount role in Live, write permissions to local AWS Secrets Manager and to S3 bucket. Role’s trusted entities:</li></ol><pre>{<br>    &quot;Version&quot;: &quot;2012-10-17&quot;,<br>    &quot;Statement&quot;: [<br>        {<br>            &quot;Effect&quot;: &quot;Allow&quot;,<br>            &quot;Principal&quot;: {<br>                &quot;Service&quot;: &quot;lambda.amazonaws.com&quot;<br>            },<br>            &quot;Action&quot;: &quot;sts:AssumeRole&quot;<br>        }<br>    ]<br>}</pre><p>ReadWriteSecretsManagerPolicy permissions policy to be created and assigned to the role:</p><pre>{<br>    &quot;Statement&quot;: [<br>        {<br>            &quot;Action&quot;: [<br>                &quot;secretsmanager:CreateSecret&quot;,<br>                &quot;secretsmanager:UpdateSecret&quot;,<br>                &quot;secretsmanager:PutSecretValue&quot;,<br>                &quot;secretsmanager:ListSecrets&quot;,<br>                &quot;secretsmanager:GetSecretValue&quot;,<br>                &quot;secretsmanager:DescribeSecret&quot;,<br>                &quot;secretsmanager:TagResource&quot;<br>            ],<br>            &quot;Effect&quot;: &quot;Allow&quot;,<br>            &quot;Resource&quot;: &quot;*&quot;<br>        },<br>        {<br>            &quot;Action&quot;: &quot;sts:AssumeRole&quot;,<br>            &quot;Effect&quot;: &quot;Allow&quot;,<br>            &quot;Resource&quot;: &quot;arn:aws:iam::222222222222:role/SecretsReadRoleForSecureAccount&quot;<br>        }<br>    ],<br>    &quot;Version&quot;: &quot;2012-10-17&quot;<br>}</pre><p>LambdaWriteToS3 permissions policy to be created and assigned to the role:</p><pre>{<br>    &quot;Statement&quot;: [<br>        {<br>            &quot;Action&quot;: [<br>                &quot;s3:PutObject&quot;<br>            ],<br>            &quot;Effect&quot;: &quot;Allow&quot;,<br>            &quot;Resource&quot;: &quot;arn:aws:s3:::YOUR_S3_BUCKET_NAME/*&quot;,<br>            &quot;Sid&quot;: &quot;AllowWriteLogsToS3&quot;<br>        }<br>    ],<br>    &quot;Version&quot;: &quot;2012-10-17&quot;<br>}</pre><p>2. Create S3 bucket YOUR_S3_BUCKET_NAME with the following bucket policy and blocked public access:</p><pre>{<br>    &quot;Version&quot;: &quot;2012-10-17&quot;,<br>    &quot;Statement&quot;: [<br>        {<br>            &quot;Sid&quot;: &quot;AllowLambdaToPutLogs&quot;,<br>            &quot;Effect&quot;: &quot;Allow&quot;,<br>            &quot;Principal&quot;: {<br>                &quot;AWS&quot;: &quot;arn:aws:iam::111111111111:role/lambda_backup_secrets_from_live_to_secure&quot;<br>            },<br>            &quot;Action&quot;: &quot;s3:PutObject&quot;,<br>            &quot;Resource&quot;: &quot;arn:aws:s3:::YOUR_S3_BUCKET_NAME/*&quot;<br>        },<br>        {<br>            &quot;Effect&quot;: &quot;Allow&quot;,<br>            &quot;Principal&quot;: {<br>                &quot;AWS&quot;: &quot;arn:aws:iam::222222222222:role/S3CrossAccountRoleForLambdaToSecureAccount&quot;<br>            },<br>            &quot;Action&quot;: [<br>                &quot;s3:GetObject&quot;,<br>                &quot;s3:ListBucket&quot;<br>            ],<br>            &quot;Resource&quot;: [<br>                &quot;arn:aws:s3:::YOUR_S3_BUCKET_NAME&quot;,<br>                &quot;arn:aws:s3:::YOUR_S3_BUCKET_NAME/*&quot;<br>            ]<br>        }<br>    ]<br>}</pre><p>3. Create lambda_function.py in Secure account which will be performing reading secrets from Live and coping them in Secrets Manager on target account. Make sure to link your Lambda function to the correct VPC within your infrastructure and increase default timeout value to 5 mins. You might need to set additionally some environment variables - it strictly depends on how you design you python code logic.</p><pre>import boto3<br>import os<br>import json<br>from botocore.exceptions import ClientError<br>from datetime import datetime<br>from zoneinfo import ZoneInfo<br><br># Initialize logs list<br>logs = []<br><br>def log(message, status=&quot;INFO&quot;):<br>    local_time = datetime.now(ZoneInfo(&quot;Europe/Berlin&quot;))<br>    timestamp = local_time.strftime(&quot;%Y-%m-%dT%H:%M:%S%z&quot;)<br>    log_entry = {<br>        &quot;timestamp&quot;: timestamp,<br>        &quot;status&quot;: status,<br>        &quot;message&quot;: message<br>    }<br>    logs.append(log_entry)<br>    print(f&quot;[{status}] {message}&quot;)<br><br>def assume_role(role_arn: str, session_name: str = &quot;CrossAccountSecretSyncSession&quot;) -&gt; boto3.Session:<br>    sts_client = boto3.client(&#39;sts&#39;)<br>    response = sts_client.assume_role(RoleArn=role_arn, RoleSessionName=session_name)<br>    creds = response[&#39;Credentials&#39;]<br>    return boto3.session.Session(<br>        aws_access_key_id=creds[&#39;AccessKeyId&#39;],<br>        aws_secret_access_key=creds[&#39;SecretAccessKey&#39;],<br>        aws_session_token=creds[&#39;SessionToken&#39;]<br>    )<br><br>def list_secrets(client) -&gt; list:<br>    secrets = []<br>    paginator = client.get_paginator(&#39;list_secrets&#39;)<br>    for page in paginator.paginate():<br>        secrets.extend(page[&#39;SecretList&#39;])<br>    return secrets<br><br>def get_secret_value(client, secret_id: str):<br>    try:<br>        response = client.get_secret_value(SecretId=secret_id)<br>        if not response.get(&quot;SecretString&quot;) and not response.get(&quot;SecretBinary&quot;):<br>            log(f&quot;Secret {secret_id} has no value. Skipping.&quot;, &quot;WARNING&quot;)<br>            return None<br>        return {<br>            &#39;SecretString&#39;: response.get(&#39;SecretString&#39;),<br>            &#39;SecretBinary&#39;: response.get(&#39;SecretBinary&#39;)<br>        }<br>    except ClientError as e:<br>        log(f&quot;Failed to fetch {secret_id}: {e}&quot;, &quot;ERROR&quot;)<br>        return None<br><br>def get_secret_tags(client, secret_id: str):<br>    try:<br>        response = client.describe_secret(SecretId=secret_id)<br>        return response.get(&quot;Tags&quot;, [])<br>    except ClientError as e:<br>        log(f&quot;Could not get tags for {secret_id}: {e}&quot;, &quot;ERROR&quot;)<br>        return []<br><br>def get_existing_secret_value(client, secret_id: str):<br>    try:<br>        response = client.get_secret_value(SecretId=secret_id)<br>        return {<br>            &#39;SecretString&#39;: response.get(&#39;SecretString&#39;),<br>            &#39;SecretBinary&#39;: response.get(&#39;SecretBinary&#39;)<br>        }<br>    except client.exceptions.ResourceNotFoundException:<br>        return None<br>    except ClientError as e:<br>        log(f&quot;Failed to fetch existing secret {secret_id} in target: {e}&quot;, &quot;ERROR&quot;)<br>        return None<br><br>def create_or_update_secret(client, secret_name: str, secret_value: dict, tags=None):<br>    kwargs_create = {&quot;Name&quot;: secret_name}<br>    kwargs_update = {&quot;SecretId&quot;: secret_name}<br><br>    if secret_value.get(&quot;SecretString&quot;) is not None:<br>        kwargs_create[&quot;SecretString&quot;] = secret_value[&quot;SecretString&quot;]<br>        kwargs_update[&quot;SecretString&quot;] = secret_value[&quot;SecretString&quot;]<br>    elif secret_value.get(&quot;SecretBinary&quot;) is not None:<br>        kwargs_create[&quot;SecretBinary&quot;] = secret_value[&quot;SecretBinary&quot;]<br>        kwargs_update[&quot;SecretBinary&quot;] = secret_value[&quot;SecretBinary&quot;]<br>    else:<br>        log(f&quot;No usable content for secret {secret_name}&quot;, &quot;ERROR&quot;)<br>        return<br><br>    if tags:<br>        kwargs_create[&quot;Tags&quot;] = tags<br><br>    try:<br>        client.create_secret(**kwargs_create)<br>        log(f&quot;Created secret: {secret_name}&quot;, &quot;INFO&quot;)<br>    except client.exceptions.ResourceExistsException:<br>        existing_value = get_existing_secret_value(client, secret_name)<br>        if existing_value == secret_value:<br>            log(f&quot;Skipped updating secret {secret_name} — value unchanged&quot;, &quot;INFO&quot;)<br>            return<br>        try:<br>            client.update_secret(**kwargs_update)<br>            log(f&quot;Updated secret: {secret_name}&quot;, &quot;INFO&quot;)<br>            if tags:<br>                try:<br>                    client.tag_resource(SecretId=secret_name, Tags=tags)<br>                    log(f&quot;Updated tags for secret: {secret_name}&quot;, &quot;INFO&quot;)<br>                except ClientError as e:<br>                    log(f&quot;Failed to update tags for {secret_name}: {e}&quot;, &quot;ERROR&quot;)<br>        except ClientError as e:<br>            log(f&quot;Error updating secret {secret_name}: {e}&quot;, &quot;ERROR&quot;)<br>    except ClientError as e:<br>        log(f&quot;Error creating secret {secret_name}: {e}&quot;, &quot;ERROR&quot;)<br><br>def sync_secrets(source_client, target_client):<br>    secrets = list_secrets(source_client)<br>    log(f&quot;Found {len(secrets)} secrets to sync&quot;, &quot;INFO&quot;)<br><br>    for secret in secrets:<br>        name = secret[&quot;Name&quot;]<br><br>        if &quot;!&quot; in name:<br>            log(f&quot;Skipping AWS-managed secret &#39;{name}&#39; due to presence of special character &#39;!&#39; indicating a managed secret&quot;, &quot;WARNING&quot;)<br>            continue<br><br>        clean_name = name.lstrip(&quot;/&quot;)<br>        new_name = f&quot;live-backup/{clean_name}&quot;<br><br>        secret_value = get_secret_value(source_client, name)<br>        tags = get_secret_tags(source_client, name)<br>        if not secret_value:<br>            continue<br><br>        create_or_update_secret(target_client, new_name, secret_value, tags=tags)<br><br>def upload_logs_to_s3(bucket_name, log_data):<br>    try:<br>        s3_client = boto3.client(&#39;s3&#39;)<br>        timestamp = datetime.now(ZoneInfo(&quot;Europe/Berlin&quot;)).strftime(&quot;%Y/%m/%d/%H%M%S&quot;)<br>        log_filename = f&quot;logs/{timestamp}_sync_log.json&quot;<br><br>        s3_client.put_object(<br>            Bucket=bucket_name,<br>            Key=log_filename,<br>            Body=json.dumps(log_data),<br>            ContentType=&#39;application/json&#39;<br>        )<br>        log(f&quot;Successfully uploaded logs to S3: {log_filename}&quot;, &quot;INFO&quot;)<br>    except Exception as e:<br>        log(f&quot;Failed to upload logs to S3: {e}&quot;, &quot;ERROR&quot;)<br><br>def lambda_handler(event, context):<br>    try:<br>        role_arn = os.environ[&quot;SOURCE_ACCOUNT_ROLE_ARN&quot;]<br><br>        log(f&quot;Assuming role: {role_arn}&quot;, &quot;INFO&quot;)<br>        source_session = assume_role(role_arn)<br>        source_client = source_session.client(&quot;secretsmanager&quot;)<br>        target_client = boto3.client(&quot;secretsmanager&quot;)<br><br>        log(&quot;Starting full secret sync&quot;, &quot;INFO&quot;)<br>        sync_secrets(source_client, target_client)<br><br>        log(&quot;Secret sync completed successfully.&quot;, &quot;INFO&quot;)<br>        return {<br>            &quot;statusCode&quot;: 200,<br>            &quot;body&quot;: &quot;Secret sync completed&quot;<br>        }<br><br>    except Exception as e:<br>        log(f&quot;Unexpected error: {e}&quot;, &quot;ERROR&quot;)<br>        return {<br>            &quot;statusCode&quot;: 500,<br>            &quot;body&quot;: f&quot;Error: {str(e)}&quot;<br>        }<br><br>    finally:<br>        try:<br>            upload_logs_to_s3(os.environ[&quot;S3_BUCKET_NAME&quot;], logs)<br>        except Exception as upload_err:<br>            print(f&quot;Final log upload failed: {upload_err}&quot;)</pre><p>4. Pack and push Lambda function code in ZIP archive via CLI as we have already made it in previous steps for Live account.</p><p>5. Configure the SOURCE_ACCOUNT_ROLE_ARN and S3_BUCKET_NAME environment variables within your Lambda function settings</p><p>6. Create <strong>Event notifications </strong>TriggerLambdaOnLogUpload in S3 bucket (<strong>Properties</strong>):</p><ul><li>Triggers a Lambda function (lambda_secrets_logs_es_ingestion_from_secure_account)</li><li>On <strong>all object creation events</strong></li><li>For objects with prefix logs/</li><li>And names the notification: TriggerLambdaOnLogUpload</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*swgsL1ECm57duSQCumEqbQ.png" /><figcaption>S3 Event Notifications to trigger Lambda on log upload</figcaption></figure><p>7. Create EventBridge trigger to run Lambda on Secure account by a schedule.</p><ul><li><strong>Name:</strong> secrets-manager-daily-backup</li><li><strong>Occurrences: </strong>Recurring schedule</li><li><strong>Timezone:</strong> Europe/Berlin</li><li><strong>Cron expression</strong>: 0 23 * * ? *</li><li><strong>Target:</strong> aws_secrets_manager_backup_from_live_to_secure_account</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*IBRkEfm4WAvP1jYn2UMRPw.png" /><figcaption>EventBridge schedule configuration</figcaption></figure><h3>Conclusion</h3><p>Because there’s no built-in AWS support for cross-account secrets backup, we developed our own solution — a custom pipeline that protects sensitive data while providing clear visibility and strong reliability, all without compromising security.</p><p>By combining Lambda, EventBridge, S3, Elasticsearch, and Grafana, we’ve automated the entire process — from secrets replication to log indexing and monitoring. Each component was chosen for its simplicity, scalability, and cost-effectiveness. For instance, EventBridge cron jobs allow us to schedule backups for virtually zero cost, and the use of Elasticsearch with Grafana gives us deep observability without exposing our secure backup account.</p><p>Another core strength of this solution is its clear separation of responsibilities: all sensitive operations are handled within a tightly restricted secure account, while observability and alerting are managed from the more accessible production account.</p><p>Depending on your system architecture and the Infrastructure as Code (IaC) standards within your organization, you might choose to use Terraform to manage and automate the deployment. In our case, Terraform helped us streamline the provisioning of IAM roles and cross-account permissions. While not strictly required, using IaC tools like Terraform can make the setup more consistent, auditable, and easier to maintain — especially as the system grows or evolves.</p><p>If you’re managing secrets in a multi-account AWS environment, or if you’re simply looking to improve your backup and monitoring practices, we hope this guide offers you a solid foundation to build upon — or adapt for your own use case.</p><p>Have thoughts, suggestions, or similar experience? Feel free to connect or drop a comment — we’re always looking to iterate and learn from the community!</p><p><em>Written by </em><a href="https://www.linkedin.com/in/pidumenk/overlay/about-this-profile/"><strong><em>Pavel Dumenko</em></strong></a><em>, DevOps engineer</em></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=aede74039105" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Elasticsearch Autoscaler]]></title>
            <link>https://medium.com/@holidu/elasticsearch-autoscaler-59bad612b69c?source=rss-12d4c8214aa6------2</link>
            <guid isPermaLink="false">https://medium.com/p/59bad612b69c</guid>
            <category><![CDATA[aws]]></category>
            <category><![CDATA[load-testing]]></category>
            <category><![CDATA[elasticsearch]]></category>
            <category><![CDATA[performance]]></category>
            <category><![CDATA[scaling]]></category>
            <dc:creator><![CDATA[Holidu Tech Blog]]></dc:creator>
            <pubDate>Wed, 28 May 2025 15:14:43 GMT</pubDate>
            <atom:updated>2025-06-04T12:49:02.785Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*VBBzgQyrm9ndB8F76nx3nw.png" /></figure><h3>TL;DR</h3><p><strong>The Problem:</strong> COVID-19 caused 3x traffic spikes that put lots of stress on our Elasticsearch cluster. No existing autoscaling solutions worked for our needs.</p><p><strong>The Solution:</strong> Built custom autoscaling using AWS Auto Scaling Groups + replica shard management. Scales 30–100+ nodes automatically based on CPU metrics.</p><p><strong>The Results:</strong> 4+ years stable, sub-700ms search performance, reduced AWS costs, zero manual intervention.</p><p><strong>What You’ll Learn:</strong> Complete code walkthrough, AWS integration, Elasticsearch shard strategies, and production gotchas.</p><h3>Introduction</h3><p>Our story began in 2020 during the COVID-19 pandemic. There was a wave of lockdowns, travel restrictions, uncertainties about the future, a tsunami of booking cancellations, while the marketing costs had already been incurred. “So what? We all suffered from it” — would say some readers. Holidu, as the leading European booking platform for vacation rentals, took the front row seat at this pandemic roller coaster: some weeks all the bookings got cancelled; a few months later, the daily traffic of users was 3x higher than in previous years only to drop to almost 0 once a new lock down would come. Now you can see why 2020 must have been very tough for us, and, boy, it was!</p><p>Luckily, with every crisis, opportunities arise. We have been closely following the ease of travel restrictions on the regional level and launched performance marketing in full force within hours, outpacing the behemoths of the travel world. This strategy brought millions of new users to our websites and 2020–2022 were very successful years. At the same time, the traffic fluctuations were brought to a whole new level.</p><h3>Traffic Patterns</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*p2Fi7l0U9tC9lzvFGM1WTQ.png" /></figure><p>I would like to share a small but powerful automation we built for managing the Elasticsearch infrastructure, as we couldn’t find any off-the-shelf solutions back then. And even now, if you face similar challenges, your options are severely limited.</p><p>At Holidu, we have a very seasonal business:</p><ul><li>Within a year: Germans rush to book vacations on the second day after Christmas, after visiting their Opas and Omas on both sides, while Southern European customers would book it just one month in advance.</li><li>Within a week: Customers commonly search for and book vacations on Sundays, or at the latest, by Monday. From our experience, people tend to delay decision-making, but the end of the week typically serves as a deadline for finalizing plans.</li><li>Within a day: Most users search for vacation rentals in the evenings, and since we focus on European vacation rental market, our users mostly search in the evenings CET.</li></ul><h3>High-level architecture</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*7GfnhGibjuBMUYs6B3fQjg.png" /></figure><p>In the Guest Experience department, we optimize for search and performance, and care about the latency of APIs and a snappy Web and App experience. Let me explain what the search experience look like and the setup of the Elasticsearch.</p><h4>Search Experience</h4><p>We offer more than 15 million vacation rentals to potential guests. Our products can find what customers want within just 500–700 ms, while searching through all those 15 million vacation rentals, applying filters, geo-searches, personalized and history rankings and many more advanced features. The machinery behind it is called the Search Engine, and its heart is an Elasticsearch cluster.</p><h4>Elasticsearch</h4><p>We run a self-managed Elasticsearch cluster, that consists of variable amount of data nodes (from 30 all the way up to ~100). Elasticsearch nodes run on EC2 instances, while all configurations and required software are baked into a custom AMI. The nodes themselves and auxiliary infrastructure (like EBS volumes and LBs) are provisioned by Terraform (the IaC part is irrelevant in this case, it could have been pulumi or CloudFormation). We organize Elasticsearch data nodes into logical units referred to as racks.</p><p>We have around 100 indices, which we host on a specific rack, based on the use case. The most important index is called inventory, and it&#39;s used on the search page. The index has at the moment 4 primary shards and multiple replica shards. We keep them on a dedicated rack inventory-rack — one shard per node. Additionally, we run one standby node, which doesn’t have any shards (it’s needed in case one of the EC2 nodes will be terminated because of the external reasons).</p><h3>Autoscaling</h3><p>As the number of users on our website and apps increases, so does the number of searches per second against the inventory index, which brings more load on EC2 instances from inventory-rack. An Elasticsearch cluster comes with fantastic features for running resilient operations, but there is nothing to make an index underlying servers scalable based on the load.</p><p>Elasticsearch scales horizontally through shards. When you add more nodes, Elasticsearch only makes use of them effectively if there’s enough shard-level parallelism.</p><p>If your cluster cannot cope with the current load, you would need to follow these steps:</p><ol><li>Add more nodes (EC2 instances in our case) to the cluster</li><li>Increase total number of shards</li><li>Wait for the relocation to finish</li></ol><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*gOxN9TKFP3h359kE6R90yA.png" /></figure><p>Once the load decreases and you don’t need that much compute power, it’s reasonable to free up a couple of EC2 instances and terminate them. These would be the steps:</p><ol><li>Reduce the total number of shards (that will make several nodes “empty”)</li><li>Terminate empty nodes</li></ol><p>Now, in terms of changing total number of shards, there are two options:</p><ol><li>Change the number of primary shards — very time-consuming, as it require re-indexing data into a new index</li><li>Change the number of replica shards — this is way faster, as it only requires copying data from each primary shard onto a new node</li></ol><p>Therefore at Holidu we decided to keep the number of primary shards semi-static (we only modify it manually after load testing and when the content of the main index changes significantly) and we rather scale by increasing and decreasing the number of replica shards.</p><p>If there is a monitoring of the Elasticsearch cluster, one could write a bunch of scripts and connect them to metrics and create an autoscaling Elasticsearch cluster.</p><p>How hard could it be?</p><h3>Enough theory, let the code speak</h3><h4>1. Scale out logic</h4><p>Scaling out always sounds like a safer option compared to scaling in — you are just adding more nodes and not risking losing any shards. Nevertheless there are a couple of precautionary measures we found useful from our four years production experience with such an automation:</p><ul><li>Once the new nodes are provisioned, one must wait till all of them join the cluster and only then replicas can be increased</li><li>Once the number of replicas is increased, it’s advisable to wait till reallocation is fully finished and then check if the all the operations have successfully finished. If not — raise an alert right away, most likely the wave of users will need more compute power just in some minutes</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Rcx7luGyaNiXSMJ2KD-HVA.png" /></figure><p>Back in the days we had to manually implement all of steps, which are basically just calls of some REST APIs or AWS SDK. That took some time and effort, and put a certain price tag on the solution. Let’s see how fast we could implement it nowadays with vibe-coding!</p><p><strong>1.1 Big picture implementation</strong></p><pre>export async function action({ request }: ActionFunctionArgs) {<br>  if (request.method !== &quot;POST&quot;) {<br>    return json({ message: &quot;Method not allowed. Please use POST.&quot; }, { status: 405 });<br>  }<br><br>  try {<br>    // Step 1: Calculate desired number of replicas<br>    const desiredNumberOfReplicas = await calculateReplicasForScaleOut();<br>    // Step 2: Provision new nodes<br>    const increaseCapacityResult = await increaseDesiredCapacity()<br>    // Step 3: Add more replica shards<br>    await setNumberOfReplicas(desiredNumberOfReplicas);<br>    // Step 4: Wait for shards reallocation<br>    await waitForReallocationFinished();<br>    return json(<br>      {<br>        success: true,<br>        message: &quot;Scale-out process completed successfully.&quot;<br>      },<br>      { status: 200 }<br>    );<br>  } catch (error) {<br>    // handle errors<br>  }<br>}</pre><p><strong>1.2 Adding more nodes</strong></p><p>Let’s assume we have a dedicated inventory-rack set of nodes, that run on EC2 instances from an AWS Autoscaling group with a custom AMI image. Let’s assume inventory index has X primary shards. That means every time we scale out, we have to increase the size of the Autoscaling group by X instances</p><p>NOTE: This approach works only for pre 8.X Elastic version, as 8.X versions have <a href="https://www.elastic.co/guide/en/elasticsearch/reference/8.18/targz.html#_enroll_nodes_in_an_existing_cluster">a special enrollment process</a>. One would have to write even more code to automate it.</p><pre>async function ensureRackNodeCount(<br>  desiredNodeCount: number<br>): Promise&lt;void&gt; {<br>  const { rackId, pollingDelayMs } = fetchAutoscalingConfig();<br><br>  let currentAttempt = 0;<br>  while (true) {<br>    currentAttempt++;<br>    logger.info(`Attempt #${currentAttempt}: Checking nodes in rack &quot;${rackId}&quot;...`);<br><br>    try {<br>      //here we just need to call /_nodes endpoint <br>      const onlineNodes = await getOnlineNodesInRack(config);<br>      if (onlineNodes === desiredNodeCount) {<br>        return;<br>      }<br>    } catch (error) {<br>      // error handling<br>    }<br>    await new Promise(resolve =&gt; setTimeout(resolve, config.pollingDelayMs));<br>  }<br>}<br><br>async increaseDesiredCapacity(): Promise&lt;IncreaseCapacityResult&gt; {<br>    const { autoScalingGroupName, increaseByAmount } = fetchAutoscalingConfig();<br>    const asg = await this.getAutoScalingGroup(autoScalingGroupName)<br>    const newDesiredCapacity = asg.currentDesiredCapacity() + increaseByAmount;<br><br>    if (newDesiredCapacity &gt; asg.maxSize()) {<br>      // raise an alert for the on-call engineer<br>    }<br><br>    try {<br>      const updateCommand = new UpdateAutoScalingGroupCommand({<br>        AutoScalingGroupName: autoScalingGroupName,<br>        DesiredCapacity: newDesiredCapacity,<br>      });<br>      await this.client.send(updateCommand);<br>      await ensureRackNodeCount(newDesiredCapacity);<br>      return {<br>        autoScalingGroupName,<br>        oldDesiredCapacity: asg.currentDesiredCapacity(),<br>        newDesiredCapacity: newDesiredCapacity,<br>        maxSize: asg.maxSize(),<br>        message: `Successfully increased desired capacity for &quot;${autoScalingGroupName}&quot; from ${asg.currentDesiredCapacity()} to ${newDesiredCapacity}.`,<br>      };<br>    } catch (error) {<br>      //handle errors<br>    }<br>  }<br>}</pre><p><strong>1.3 Adding more replicas</strong></p><p>Here we will first increase the number of replicas of the index inventory and then wait till re allocation is finished</p><pre>async function calculateReplicasForScaleOut(): Promise&lt;number&gt; {<br>  const { host, index } = fetchAutoscalingConfig();<br><br>  const url = `${host}/${index}/_settings`;<br><br>  try {<br>    const response = await fetch(url);<br>    if (!response.ok) {<br>      // handle error<br>    }<br><br>    const settingsResponse: IndexSettingsResponse = await response.json();<br>    const indexName = Object.keys(settingsResponse)[0];<br>    const replicasString = settingsResponse[indexName]?.settings?.index?.number_of_replicas;<br>    const currentNumberOfReplicas = parseInt(replicasString, 10);<br>    return currentNumberOfReplicas + 1;<br>  } catch (error) {<br>    // handle error<br>  }<br>}<br><br>async setNumberOfReplicas(desiredNumberOfReplicas: number): Promise&lt;void&gt; {<br>    const endpoint = `${this.config.host}/${this.config.index}/_settings`;<br>    const payload = {<br>      index: {<br>        number_of_replicas: desiredNumberOfReplicas,<br>      },<br>    };<br>    const requestHeaders = {<br>      &#39;Content-Type&#39;: &#39;application/json&#39;<br>    };<br><br>    try {<br>      const response = await this.httpClient.put(endpoint, payload, {<br>        headers: requestHeaders,<br>      });<br>    } catch (error) {<br>      // error handling<br>  }<br>}</pre><p><strong>1.4 Waiting for shards reallocation</strong></p><p>Here we observe the status of _cat/recovery for the index in question so that all shards are fully allocated.</p><pre>async function waitForReallocationFinished(): Promise&lt;void&gt; {<br>  const {<br>    host,<br>    index,<br>    initialDelayMs = DEFAULT_INITIAL_DELAY_MS,<br>    pollingIntervalMs = DEFAULT_POLLING_INTERVAL_MS,<br>    logger = DEFAULT_LOGGER,<br>  } = fetchAutoscalingConfig();<br><br>  const recoveryUrl = `${host}/_cat/recovery?index=${index}-v*&amp;h=st&amp;format=text`;<br><br>  while (true) {<br>    let reallocationText: string;<br>    try {<br>      const response = await fetch(recoveryUrl);<br>      if (!response.ok) {<br>        //error handling<br>      }<br>      reallocationText = await response.text();<br>    } catch (error) {<br>      //error handling<br>    }<br><br>    // Process the response text: remove &quot;done&quot; and newline characters.<br>    // If the result is an empty string, it means all shards reported &quot;done&quot; or there were no matching shards.<br>    const processedReallocation = reallocationText<br>      .split(&#39;\n&#39;)<br>      .map(line =&gt; line.trim())<br>      .filter(line =&gt; line &amp;&amp; line !== &#39;done&#39;) // Filter out empty lines and &quot;done&quot; lines<br>      .join(&#39;&#39;);<br><br>    if (processedReallocation === &#39;&#39;) {<br>      logger.info(`No active reallocation for ${index}-v*. Reallocation finished.`);<br>      break; // Exit the loop<br>    }<br><br>    logger.info(`Reallocation in progress for ${index}-v*. Current status lines: ${processedReallocation.length &gt; 0 ? processedReallocation.split(/\s+/).length : 0}`);<br>    await new Promise(resolve =&gt; setTimeout(resolve, pollingIntervalMs));<br>  }<br>}</pre><h4>2. Scale in logic</h4><p>That’s a scarier bit, as there are plenty of opportunities to break the cluster. We will be more cautious and follow these steps:</p><ol><li>Reduce number of replicas</li><li>Wait till reallocation finished</li><li>Exclude last X nodes (where X is the number of primary shards) from routing allocation — this triggers shards reallocation and frees up the last X nodes while moving their shards onto vacant nodes from the first step</li><li>Wait till reallocation finished</li><li>Reduce number of instances in the Autoscaling group</li></ol><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*o2ftS1h5f0F7m9bPC8C_tA.png" /></figure><p>Let’s vibe-code again</p><p><strong>2.1 Big picture implementation</strong></p><pre>export async function action({ request }: ActionFunctionArgs) {<br>  if (request.method !== &quot;POST&quot;) {<br>    return json({ message: &quot;Method not allowed. Please use POST.&quot; }, { status: 405 });<br>  }<br><br>  try {<br>    // Step 1: Reduce number of replicas<br>    const desiredNumberOfReplicas = await calculateReplicasForScaleIn();<br>    await setNumberOfReplicas(desiredNumberOfReplicas);<br>    // Step 2: Wait for shards reallocation<br>    await waitForReallocationFinished();<br>    // Step 3: Exclude last X node<br>    await manageNodeExclusionByCount();<br>    // Step 4: Wait for shards reallocation<br>    await waitForReallocationFinished();<br>    // Step 5: Reduce number of nodes in ASG<br>    await decreaseDesiredCapacity()<br>    return json(<br>      {<br>        success: true,<br>        message: &quot;Scale-in process completed successfully.&quot;<br>      },<br>      { status: 200 }<br>    );<br>  } catch (error) {<br>    // handle errors<br>  }<br>}</pre><p><strong>2.2 Reduce number of replicas</strong></p><pre>async function calculateReplicasForScaleIn(): Promise&lt;number&gt; {<br>...<br>// it should be very similar to<br>// async function calculateReplicasForScaleOut(): Promise&lt;number&gt; {}<br>// but we return return currentNumberOfReplicas - 1;<br>}<br><br>// we will reuse <br>// - async setNumberOfReplicas(desiredNumberOfReplicas: number): Promise&lt;void&gt;</pre><p><strong>2.3 Wait till reallocation finished</strong></p><pre>// we will reuse <br>// - async function waitForReallocationFinished(): Promise&lt;void&gt;</pre><p><strong>2.4 Exclude last X nodes</strong></p><pre>async function manageNodeExclusionByCount(): Promise&lt;void&gt; {<br>  try {<br>    const config = fetchAutoscalingConfig();<br>    // 1. Fetch all node data from the cluster<br>    const allNodesResponse = await fetchAllNodesData(config.host);<br>    const allNodesMap = allNodesResponse.nodes;<br><br>    // 2. Filter nodes that belong to the specified rack<br>    const nodesInTargetRack: ElasticsearchNode[] = [];<br>    for (const nodeId in allNodesMap) {<br>      if (<br>        allNodesMap[nodeId].attributes &amp;&amp;<br>        allNodesMap[nodeId].attributes.rack === config.rack<br>      ) {<br>        nodesInTargetRack.push(allNodesMap[nodeId]);<br>      }<br>    }<br><br>    // 3. Sort the rack nodes by their ordinal number (extracted from name)<br>    nodesInTargetRack.sort((nodeA, nodeB) =&gt; {<br>      const ordinalA = extractOrdinalFromNodeName(nodeA.name);<br>      const ordinalB = extractOrdinalFromNodeName(nodeB.name);<br>      return ordinalA - ordinalB; // Ascending order<br>    });<br><br>    // 4. Determine which nodes to exclude<br>    const nodesToActuallyExclude = nodesInTargetRack.slice(nodesInTargetRack.length - config.decreaseByAmount);<br><br>    // 5. Get the IP addresses of the nodes selected for exclusion<br>    const ipsToExclude = nodesToActuallyExclude.map(node =&gt; node.ip);<br>    await updateClusterExcludeAllocation(ipsToExclude, config.host);<br><br>  } catch (error) {<br>    // handle errors<br>  }<br>}<br><br>async function fetchAllNodesData(host: string): Promise&lt;NodesApiResponse&gt; {<br>  const response = await fetch(`${host}/_nodes`);<br>  return response.json() as Promise&lt;NodesApiResponse&gt;;<br>}<br><br>function extractOrdinalFromNodeName(nodeName: string): number {<br>  const numericMatches = nodeName.match(/\d+/g); // Find all sequences of digits<br>  if (numericMatches &amp;&amp; numericMatches.length &gt; 0) {<br>    return parseInt(numericMatches[numericMatches.length - 1], 10);<br>  }<br>  return 0;<br>}<br><br>async function updateClusterExcludeAllocation(<br>  ipsToExclude: string[],<br>  host: string,<br>): Promise&lt;void&gt; {<br>  // Ensure unique IPs and format them as a comma-separated string<br>  const uniqueIps = Array.from(new Set(ipsToExclude));<br>  const ipsListString = uniqueIps.join(&#39;,&#39;);<br><br>  const settingsPayload = {<br>    transient: {<br>      cluster: {<br>        routing: {<br>          allocation: {<br>            enable: &quot;all&quot;, // Or other appropriate value like &quot;primaries&quot;, &quot;new_primaries&quot;<br>            exclude: {<br>              _ip: ipsListString,<br>            },<br>          },<br>        },<br>      },<br>    },<br>  };<br><br>  const response = await fetch(`${host}/_cluster/settings`, {<br>    method: &#39;PUT&#39;,<br>    headers: {<br>      &#39;Content-Type&#39;: &#39;application/json&#39;<br>    },<br>    body: JSON.stringify(settingsPayload),<br>  });<br><br>  const responseData = await response.json(); // Assuming JSON response<br>  console.info(<br>    `Successfully updated cluster settings. Excluded IPs: ${ipsListString}. Response:`,<br>    responseData,<br>  );<br>}</pre><p><strong>2.5 Wait till reallocation finished</strong></p><pre>// we will reuse <br>// - async function waitForReallocationFinished(): Promise&lt;void&gt;</pre><p><strong>2.6 Reduce number of instances in the Autoscaling group</strong></p><pre>// it&#39;s almost like increaseDesiredCapacity(), we just need to decrease the capacity</pre><h3>More considerations</h3><p>To build a fully fledged and stable system, more things need to be considered. We won’t deep dive into each of them, but want to highlight other parts that need to be tackled to build a production ready autoscaler</p><ul><li>Scaling operations should have a lock (to avoid running the same actions twice or opposite actions in parallel)</li><li>Scaling operations should have a cooldown period to let the metrics of the cluster stabilize</li><li>There should be no ingestion into the index during the scaling operations (in simple words: you don’t want to copy data from A to B, while you keep writing into A)</li><li>It’s good to have &quot;allocation.total_shards_per_node:1&quot;</li><li>There should be a special logic to handle edge cases of scaling out (hitting limits of ASG) and scaling in like having a standby node or having the minimum size of the cluster</li><li>Good logging and alerting are a must</li><li>Test different EC2 Instance Types to figure out which ones perform better for your use case</li><li>Load test your index to figure out the best sharding and allocation settings for your index — maybe the nodes in your cluster can have 3 shards of the index and you can have a different scaling pattern</li></ul><h3>Scaling in action</h3><p>Once you have a small node.js app in place, you can deploy it and connect to your preferred systems for monitoring metrics.</p><p>At Holidu we use Prometheus+Thanos+Grafana stack for metrics monitoring — have a read about our setup:</p><p><a href="https://medium.com/@holidu/how-to-handle-terabytes-of-metrics-in-kubernetes-monitoring-a3056adf92b">How to Handle Terabytes of Metrics in Kubernetes Monitoring</a></p><p>The sequence of steps is as follows:</p><ol><li>Create new contact points for both actions</li></ol><ul><li>scale-out contact point connected to /scale-out endpoint of the autoscaler</li><li>scale-in contact point connected to /scale-in endpoint of the autoscaler</li></ul><p>2. Create 2 panels</p><ul><li>for scaling out we are measuring CPU of top 4 nodes</li><li>for scaling in we are measuring average CPU</li></ul><p>3. Create alerts in both panels and wire them to corresponding Contact Points</p><ul><li>the thresholds for going off is chosen after some experimentation</li></ul><h3>Alternatives</h3><p>Are there any ready solutions for such a problem?</p><p>In the time of writing, there is an option to use serverless OpenSearch, and it would work perfectly out of the box for some cases. However, you should get to know potential caveats such as:</p><ul><li>There is no guarantee it will be more cost-efficient compared to self-managed cluster and a bit of automation (even when Total Cost of Ownership is applied) — the maintenance of Elasticsearch cluster is low, the main effort actually goes into migrating to the next Java Client, which keeps changing with every major version</li><li>Plugin ecosystem and third-party integrations are more constrained for OpenSearch</li><li>OpenSearch often receives performance improvements and new indexing optimizations months or even years after they appear in Elasticsearch</li><li>There are some benchmarks indicating that performance of Elastic might be better than OpenSearch</li></ul><p>One other options would be Elastic Cloud Kubernetes, as it supports autoscaling. However, it comes with the following question marks</p><ul><li>one would have to commit to running Elasticsearch cluster on K8S</li><li>it only works with Elastic Cloud Hosted, Elastic Cloud Enterprise, and Elastic Cloud on Kubernetes</li></ul><p>Alternatively, <a href="https://github.com/zalando-incubator/es-operator">es-operator</a> from Zalando should be capable of achieving autoscaling out of the box … as long as you are fine with running the Elasticsearch cluster on K8S.</p><h3>Closing thoughts and next steps</h3><p>When we first implemented Elasticsearch autoscaling, there was nothing similar available on the Internet. That worked and still works super smoothly, providing great search experience to our customers while maintaining reasonable AWS bills for Michael, our CTO.</p><p>We believe our solution might be interesting to other companies, that because of multiple reasons cannot use or afford using OpenSearch.</p><p>Currently we are preparing for Elastic 8.X upgrade. Our main bottleneck is rewriting ALL queries again, as the Java SDK has been changed again. In the past 2 year we were ̶c̶o̶p̶i̶n̶g̶ ̶w̶i̶t̶h̶ ̶P̶T̶S̶D̶ ̶o̶f̶ ̶7̶.̶X̶ ̶u̶p̶g̶r̶a̶d̶e̶ strategically waiting till LLMs become capable to pulling off the 7.X-&gt;8.X rewrite. The upgrade will also require the extension/rewrite of the ES autoscaler, because of changes in bringing nodes to the cluster. And then, who knows — we might even give running an Elasticsearch cluster on Kubernetes a try.</p><p>All of that and even more we will be publishing in the blog, as we are finally back to talk about challenges and technologies at Holidu.</p><p>Till then Happy Scaling!</p><p><em>Written by </em><a href="https://www.linkedin.com/in/konstantin-pasko/"><em>Konstantin Pasko</em></a><em>, Director of Engineering</em></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=59bad612b69c" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[How to Handle Terabytes of Metrics in Kubernetes Monitoring]]></title>
            <link>https://medium.com/@holidu/how-to-handle-terabytes-of-metrics-in-kubernetes-monitoring-a3056adf92b?source=rss-12d4c8214aa6------2</link>
            <guid isPermaLink="false">https://medium.com/p/a3056adf92b</guid>
            <category><![CDATA[grafana]]></category>
            <category><![CDATA[prometheus]]></category>
            <category><![CDATA[thanos]]></category>
            <category><![CDATA[kubernetes]]></category>
            <category><![CDATA[monitoring]]></category>
            <dc:creator><![CDATA[Holidu Tech Blog]]></dc:creator>
            <pubDate>Thu, 10 Mar 2022 15:09:27 GMT</pubDate>
            <atom:updated>2022-03-10T16:51:34.872Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="Kubernetes monitoring with Prometheus, Thanos and Grafana at Holidu" src="https://cdn-images-1.medium.com/max/500/1*UsHbWJDwV6rpu4L-xQmfoQ.png" /></figure><p>Monitoring is the backbone of robust infrastructure which acts as a saviour during outages. It provides a strong root for proactively identifying the forthcoming issues and avoiding possible downtime. Having better monitoring paves the way for proper decision making of future development and new feature validations.</p><p>Constantly increasing customer traffic at Holidu resulted in ever growing infrastructure. Generating approx. 40k metrics samples every second it overwhelmed our initial monitoring setup and increased the operational cost (as the required storage size increased to accommodate the growing number of metrics every day).</p><p>We clearly needed a new scalable and stable monitoring system that is cost-efficient and able to handle tons of generated metrics. We used Prometheus, Thanos and Grafana as a new setup that smoothly handles the ~40k metric samples generated every second. Now we are able to store and visualise yearly historical metrics with easy maintenance and upgrades with low operational cost. In this blog post, our engineering team shares some insights and learnings on our journey to a state-of-the-art monitoring setup.</p><h3><strong>The Old Setup</strong></h3><figure><img alt="A complex and hard to maintain monitoring setup with Prometheus and Grafana at Holidu" src="https://cdn-images-1.medium.com/max/669/0*Stv3Dutz5WPcaKf0" /></figure><p>Initially we started with a single prometheus instance to collect metrics for our entire kubernetes cluster but due to increased traffic this instance was quickly overwhelmed and became unstable. To mitigate the problem at the time we decided to split our Prometheus instance into several instances per namespace each with a dedicated EBS volume as storage to reduce the load on a single instance and to limit the “blast radius”. Having Prometheus instances per team/namespace made upgrades and maintenance difficult and time-consuming. This is because every team might not consider the same upgrade window due to them having different release schedules and sprint priorities.</p><p>The operational cost was also growing day by day as we had to accommodate the growing size of metrics on EBS. One can imagine the cost of EBS when persisting 1 terabyte of data in total.</p><h3><strong>The New Setup</strong></h3><figure><img alt="Advanced Kubernetes monitoring with Prometheus, Thanos and Grafana at Holidu" src="https://cdn-images-1.medium.com/max/1024/0*jXXF1F1ggW54Gkkr" /><figcaption><strong>Thanos Sidecar</strong>: Thanos sidecar is to ship Prometheus data into an AWS S3 Bucket. <strong>Thanos Compactor</strong>: The compactor scans the S3 bucket and processes compaction where required. At the same time, it is responsible for creating down-sampled copies of data to speed up queries. <strong>Thanos Store</strong>: Thanos Store queries historical data by implementing the gRPC data API and queries directly from S3 to fetch the data. <strong>Thanos Query</strong>: Thanos Query is a global query layer to evaluate PromQL queries and query real-time data from Prometheus and historical data from Thanos-Store.</figcaption></figure><figure><img alt="Advanced Kubernetes monitoring setup with Prometheus, Thanos and Grafana at Holidu" src="https://cdn-images-1.medium.com/max/736/0*X30ZL-DitivTjnLC" /><figcaption><strong>Grafana</strong> queries the <strong>Thanos-query</strong> component to get the time series data and display it on panels. Thanos-query fetches live metrics directly from <strong>Prometheus</strong>. For historical metrics, Thanos-query reaches to <strong>Thanos-Store</strong> which queries directly from <strong>S3</strong> using gRPC data API. Finally, the metrics are shown on the grafana panel.</figcaption></figure><p>Our new monitoring setup moves back to a single centralised Prometheus instance per cluster, but significantly improves on the storage costs and query performance by introducing Thanos. The Centralised Prometheus instance collects metrics from all namespaces and stores them temporarily in EBS (50 GB). These metrics are collected and compressed into a block. Further, these blocks are automatically shipped to AWS S3 periodically (every 3 hours) and the EBS volume is again reclaimed. This operation ensures that EBS is never out of space and there is no need to increase the EBS volume size. It reduced the operational cost drastically as storing data in S3 is way cheaper than storing it in EBS. Now we can retain the metrics for the past 1 year and using Thanos, metrics can be queried directly from S3.</p><p>With further enhancements, there are now two separate monitoring setups and both serve a single Grafana instance. The first one handles all the application metrics and the second one handles all the system metrics. Each setup is provided with sufficient resources (memory and CPU) to handle the growing load. See below for details on why we chose to split the metrics like this. This split ensures reduced response time of queries and also adds some redundancy to our monitoring setup such that a complete failure of the monitoring is less likely., then there is no complete outage.</p><h3><strong>Deploy the setup</strong></h3><p>We use the Prometheus Community helm chart to deploy the stack. However, the current chart does not provide Thanos Query, Compactor and Store which has to be deployed separately.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/69e6b25523a7cc63f6988181aff138df/href">https://medium.com/media/69e6b25523a7cc63f6988181aff138df/href</a></iframe><p>Reference: <a href="https://github.com/prometheus-community/helm-charts/tree/main/charts/kube-prometheus-stack">https://github.com/prometheus-community/helm-charts/tree/main/charts/kube-prometheus-stack</a></p><p>To deploy the Thanos component you can either use the kubernetes manifest (1) or the helm chart (2).</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/e2d5507f5d4ee166a68dd6c5a3269e6b/href">https://medium.com/media/e2d5507f5d4ee166a68dd6c5a3269e6b/href</a></iframe><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/ada73c89e82fec6ff22b746935600d6a/href">https://medium.com/media/ada73c89e82fec6ff22b746935600d6a/href</a></iframe><p>Reference: <a href="https://github.com/thanos-io/kube-thanos/tree/main/examples/all/manifests">https://github.com/thanos-io/kube-thanos/tree/main/examples/all/manifests</a></p><h3><strong>Benefits from New Setup</strong></h3><h4><strong>Prometheus able to handle growing load</strong></h4><p>With the improved setup Prometheus is able to handle more metrics samples ingested without pods getting killed / restarted, as compared to our previous setup: the new monitoring system is more stable and we expect it to scale well with expected increased load in the future.</p><h4>Cost-effective</h4><p>The new setup relies on S3 for the bulk of our storage rather than EBS. We have a limited EBS storage (50GB) which reduces the cost when compared to the old setup where the EBS volumes had to grow in size with the growing number of metrics.</p><h4>Long retention of metrics</h4><p>Having S3 as a main storage allows us to retain the historical data longer and thus visualise metrics from the last one year. It helps us to better analyse the behaviour of applications, track the results of improvements and new feature validation on a quarterly / half-yearly / yearly basis. This helps in better decision making of future developments.</p><h4>Avoid monitoring outage</h4><p>Having two separate setups for application metrics and system metrics ensures that there is no complete monitoring blackout in case of any unexpected breakdown. Even if one of the systems is down, we will still have another set of metrics monitoring unaffected.</p><h3><strong>Challenges faced</strong></h3><h4>1. Prometheus pod killing with OOM and constantly restarts</h4><p>As soon as the new stack was deployed initially, it became unstable and further restarted several times due to Out of Memory (OOM). Resource consumption of Prometheus pod depends on a lot of factors like:</p><ul><li>How big a cluster is in terms of the number of nodes?</li><li>How many applications does Prometheus scrape the metrics from?</li><li>Number of Kubernetes system components being monitored, for example, Node metrics, Kubelet metrics, CoreDNS etc.</li></ul><p><strong>Solution:</strong></p><p>At Holidu, we operate our Kubernetes cluster of ~90 nodes with 150+ applications being scrapped, along with system components metrics like kubelet and node metrics. Prometheus is high on memory consumption when it comes to bulk processing. Below are the approaches we used to figure out resource consumption to keep Prometheus stable and avoid getting killed due to OOM.</p><p><strong>Analyse Prometheus resource consumption</strong></p><p><em>Approach 1</em></p><p>Initially, we made Prometheus stable by setting higher limits of memory and CPU. Once the setup was stable then the Grafana dashboard was created to analyse the actual consumption. After that, the resource request and limits were adjusted in values.yaml.</p><figure><img alt="Prometheus memory usage metrics at Holidu" src="https://cdn-images-1.medium.com/max/1024/0*1FVa_ZA5FCwyUwuP" /></figure><p>Memory Usage Prometheus Query</p><pre>sum(container_memory_working_set_bytes{container=&quot;prometheus&quot;, namespace=~&quot;monitoring&quot;, pod=&quot;centralized-system-metrics-prometheus-0&quot;}) by (id,pod)</pre><p>Updated Resource Block in helm chart values.yaml</p><pre>requests:<br>  memory: &quot;4000Mi&quot;<br>  cpu: &quot;256m&quot;<br>limits:<br>  memory: &quot;10500Mi&quot;<br>  cpu: &quot;768m&quot;</pre><p><em>Approach 2</em></p><p>We checked samples/sec ingested in Prometheus and number of chunks created per hour and here are the numbers:</p><ul><li>Number of Samples per second: 39692</li><li>Number of Series created per second: 20</li><li>Number of Head Chunks per second: 370</li><li>Number of Label Pairs: 43130</li><li>Number of Targets: 15+</li></ul><p>Below are the Prometheus queries to identify the above data:</p><pre>Samples ingested per second:<br>  max(rate(prometheus_tsdb_head_samples_appended_total[1m]))</pre><pre>Series created per second:<br>  max(rate(prometheus_tsdb_head_series_created_total[1m]))</pre><pre>Chunks created per second:<br>  max(rate(prometheus_tsdb_head_chunks_created_total[1m]))</pre><h4><strong>2. Prometheus is very slow in returning query results</strong></h4><p>Grafana graph took 20+ seconds to show data in the panel. Volume of system metrics generated was very high (~35k samples per second) when compared to application metrics (~5k samples per second). Having both the metrics in the same instance made the response time of queries higher than usual.</p><p><strong>Solution:</strong></p><p>As mentioned earlier, to solve this issue, we split the setup into 2 separate Prometheus instances. One is dedicated to application metrics and another for system metrics. Now Prometheus response time is reduced and the setup is more stable. With this setup, we also decoupled both types of metrics. If any one of the Prometheus instances is down, it will not result in a complete outage and we will still have one set of the metrics available for analysis.</p><h4><strong>3. Helm chart failed to install as cluster roles already exist with the same name due to the old Prometheus setup</strong></h4><p>We had an old setup running which could not be removed unless the new monitoring setup was tested. The helm chart failed to deploy as the Kube-State-Metrics sub-chart tried to create Clusterroles and ClusterRolebindings with the same names, which were already created by the old setup helm chart.</p><p><strong>Solution:</strong></p><p>We reuse the existing ClusterRoles and ClusterRoleBindings by passing below values in helm deploy command:</p><pre>--set kube-state-metrics.rbac.create=false</pre><pre>--set kube-state-metrics.rbac.useExistingRole=prometheus-kube-state-metrics</pre><h3>Closing thoughts</h3><p>In this post, we covered how we moved from a quite complex and hard to maintain monitoring setup based on individual Prometheus instances and storage per namespace to a more advanced, stable and scalable centralised solution. With new monitoring, we now successfully handle ~40K metrics samples per second and are able to retain the historical data of the past one year which amounts to close to 1.5 terabytes. Operational cost reduced as we now rely on S3 rather than EBS as a storage solution for the bulk of the data. Metrics are split into two fragments and a separate setup reduces the risk of a complete outage. Having a centralised monitoring setup improves overall maintenance and enables upgrades in much shorter time.</p><p>We also discussed the challenges faced during the installation and their solutions. We hope this solution will be helpful in setting up and improving Kubernetes workloads and infrastructure monitoring for other teams as well</p><p>We will be back soon with our next post with exciting cutting-edge technologies at Holidu.</p><p>Till then Happy Monitoring!</p><p><em>Written by </em><a href="https://www.linkedin.com/in/samarthkanungo/"><strong><em>Samarth Kanungo</em></strong></a><em>, Senior DevOps engineer</em></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=a3056adf92b" width="1" height="1" alt="">]]></content:encoded>
        </item>
    </channel>
</rss>