<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Manjula Wanasinghe on Medium]]></title>
        <description><![CDATA[Stories by Manjula Wanasinghe on Medium]]></description>
        <link>https://medium.com/@manjula.wanasinghe?source=rss-42c8abcbaab5------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/1*dmbNkD5D-u45r44go_cf0g.png</url>
            <title>Stories by Manjula Wanasinghe on Medium</title>
            <link>https://medium.com/@manjula.wanasinghe?source=rss-42c8abcbaab5------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Sat, 21 Nov 2026 20:16:57 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@manjula.wanasinghe/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[A Guide to Deploying Elasticsearch 9.0.0 Securely on RHEL 9]]></title>
            <link>https://medium.com/axonect-blog/a-guide-to-deploying-elasticsearch-9-0-0-securely-on-rhel-9-6c01b967b23c?source=rss-42c8abcbaab5------2</link>
            <guid isPermaLink="false">https://medium.com/p/6c01b967b23c</guid>
            <category><![CDATA[elasticsearch]]></category>
            <category><![CDATA[elk-stack]]></category>
            <category><![CDATA[rhel]]></category>
            <category><![CDATA[rhel-9]]></category>
            <dc:creator><![CDATA[Manjula Wanasinghe]]></dc:creator>
            <pubDate>Fri, 07 Aug 2026 11:02:50 GMT</pubDate>
            <atom:updated>2026-08-10T09:43:06.480Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*vuNWfRZQXPiYiEBLeSQY0A.jpeg" /></figure><p>The ELK Stack — comprising Elasticsearch, Logstash, and Kibana — is one of the most widely adopted platforms for centralised log management, observability, and operational analytics. By collecting, processing, indexing, and visualising data from applications, servers, and infrastructure, it enables organisations to detect issues faster, improve system reliability, and gain actionable insights from large volumes of machine data. At the core of the stack is Elasticsearch, a distributed search and analytics engine that delivers near real-time indexing, powerful full-text search, and horizontal scalability. Its ability to store, search, and analyse massive datasets quickly makes it the foundation of modern monitoring, security (SIEM), and business intelligence solutions. This guide comprehensively detail how Elasticsearch can be securely and efficiently deployed as the backbone of a production-ready ELK environment.</p><h3><strong>Background:</strong></h3><p>This guide provides a production-hardened roadmap for deploying the Elastic Stack 9.0.0 on RHEL 9 in a secure and optimised manner. The plan is designed for real-world enterprise environments, focusing on high availability, security, and performance tuning. <br> <br>We will build a 3-node Elasticsearch cluster (one master/data node and two data nodes) to ensure resilience and scalability. Kibana will be installed on the master node for visualization, while Logstash will handle centralised log processing. Additionally, an application server will run Filebeat, which collects system and application logs and forwards them to Logstash for enrichment and indexing into Elasticsearch. <br> <br>The deployment uses the automated enrollment method for cluster security, eliminating manual certificate handling and simplifying node onboarding. All services will run under a non-root user (intgw) for better security. The guide also includes an architecture diagram, systemised service configurations, and performance tuning tips for handling large-scale workloads.</p><h3><strong>Infrastructure Overview</strong></h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/822/1*ME0QstquCTBj9APeWwgIAQ.png" /></figure><h3><strong>Step 1: System-Wide Preparation (All Nodes)</strong></h3><p>Prepare the operating system across all cluster nodes by creating a dedicated non-root service account (intgw), increasing kernel virtual memory mapping limits (vm.max_map_count), setting file descriptor and memory locks in system limits, and creating the target installation directory. These system-level prerequisites prevent out-of-memory errors and ensure Elasticsearch operates reliably under heavy production workloads.</p><pre># Run as root<br>useradd -m -s /bin/bash intgw<br>echo &quot;vm.max_map_count=262144&quot; &gt;&gt; /etc/sysctl.conf &amp;&amp; sysctl -p<br>cat &lt;&lt;EOF &gt;&gt; /etc/security/limits.conf<br>intgw soft nofile 65535<br>intgw hard nofile 65535<br>intgw soft memlock unlimited<br>intgw hard memlock unlimited<br>EOF<br>mkdir -p /opt/elk_stack &amp;&amp; chown -R intgw:intgw /opt/elk_stack</pre><h3><strong>Step 2: Elasticsearch Essential Configuration</strong></h3><p>Download and extract the Elasticsearch 9.0.0 binaries on every node. Configure elasticsearch.yml to set cluster and node identifiers, set network bindings, enforce memory locks, and activate X-Pack security features with automated enrollment enabled. Finally, define a systemd service file to manage process execution securely under the non-root user account.</p><p><strong>1. Binary Extraction</strong></p><pre>[intgw@all-es-nodes ~]$<br>cd /opt/elk_stack<br>wget https://artifacts.elastic.co/downloads/elasticsearch/elasticsearch-9.0.0-linux-x86_64.tar.gz<br>tar -xzf elasticsearch-9.0.0-linux-x86_64.tar.gz</pre><p><strong>2. Configuration File: /opt/elk_stack/elasticsearch-9.0.0/config/elasticsearch.yml</strong></p><pre>cluster.name: production-deployment<br>node.name: ${HOSTNAME}<br>path.data: /opt/elk_stack/elasticsearch-9.0.0/data<br>path.logs: /opt/elk_stack/elasticsearch-9.0.0/logs<br>network.host: 0.0.0.0<br>http.port: 9200<br>transport.port: 9300<br>bootstrap.memory_lock: true<br>xpack.security.enabled: true<br>xpack.security.enrollment.enabled: true</pre><p><strong>3. Systemd Service Unit: /etc/systemd/system/elasticsearch.service</strong></p><pre>[Unit]<br>Description=Elasticsearch 9.0.0<br>Documentation=https://www.elastic.co/guide/en/elasticsearch/reference/current<br>Wants=network-online.target<br>After=network-online.target<br><br>[Service]<br>User=intgw<br>Group=intgw<br>ExecStart=/opt/elk_stack/elasticsearch-9.0.0/bin/elasticsearch<br>Restart=always<br>LimitMEMLOCK=infinity<br>Environment=ES_JAVA_OPTS=&quot;-Xms8g -Xmx8g&quot;<br>WorkingDirectory=/opt/elk_stack/elasticsearch-9.0.0<br><br>[Install]<br>WantedBy=multi-user.target</pre><h3><strong>Step 3: Deep Dive into Automated Enrollment</strong></h3><p>Securely join additional data nodes (es-node2 and es-node3) to the cluster using Elasticsearch’s automated enrollment mechanism. This feature eliminates manual certificate generation and distribution by using secure, short-lived enrollment tokens generated on the initial bootstrap node.</p><p><strong>The 8-Step Automated Window</strong></p><p><strong>1. Bootstrap Node 1: </strong>Start Node 1. It generates a cluster CA and superuser password.</p><p><strong>2. Generate Token: </strong>Create the invitation string on Node 1.</p><p><strong>3. Initialize New Node: </strong>Start Node 2/3 using — enrollment-token.</p><p><strong>4. Handshake: </strong>Node 2 connects to Node 1 via HTTPS.</p><p><strong>5. Validation: </strong>Node 2 verifies identity via the token’s CA fingerprint.</p><p><strong>6. Certificate Issuance: </strong>Node 1 issues unique TLS certs to Node 2.</p><p><strong>7. Auto-Config: </strong>Node 2 updates its own elasticsearch.yml automatically.</p><p><strong>8. Cluster Join: </strong>Node 2 joins the verified cluster members.</p><p><strong>Node Enrollment Command Examples:</strong></p><pre>Start Node 1. <br># On Node 1 (Start Elasticsearch for the first time as interactive)<br>/opt/elk_stack/elasticsearch-9.0.0/bin/elasticsearch<br>  Note: On first startup, Elasticsearch outputs generated credentials and CA security certificate details to the terminal log.<br><br># On Node 1 (generate an enrollment token for nodes)<br>/opt/elk_stack/elasticsearch-9.0.0/bin/elasticsearch-create-enrollment-token -s node<br><br># Copy the printed token (valid briefly) securely to the joining nodes.<br><br># On Node 2 (join cluster using the token)<br>/opt/elk_stack/elasticsearch-9.0.0/bin/elasticsearch --enrollment-token &lt;TOKEN&gt;<br><br># On Node 3 (join cluster using the token)<br>/opt/elk_stack/elasticsearch-9.0.0/bin/elasticsearch --enrollment-token &lt;TOKEN&gt;</pre><h3><strong>Step 4: Kibana Installation &amp; Systemd (Node 1)</strong></h3><p>Install Kibana on es-node1 to serve as the visualization and cluster management interface. Connect Kibana to Elasticsearch securely using a Kibana-specific enrollment token, and configure a systemd service to manage its lifecycle.</p><pre>[intgw@es-node1 ~]$<br>wget https://artifacts.elastic.co/downloads/kibana/kibana-9.0.0-linux-x86_64.tar.gz<br>tar -xzf kibana-9.0.0-linux-x86_64.tar.gz<br><br># On Node 1 (generate Kibana enrollment token)<br>/opt/elk_stack/elasticsearch-9.0.0/bin/elasticsearch-create-enrollment-token -s kibana<br><br># On the Kibana host (Node 1)<br>/opt/elk_stack/kibana-9.0.0/bin/kibana-setup --enrollment-token &lt;KIBANA_TOKEN&gt;</pre><p><strong>Systemd Unit: /etc/systemd/system/kibana.service</strong></p><pre>[Unit]<br>Description=Kibana 9.0.0<br>After=elasticsearch.service<br><br>[Service]<br>User=intgw<br>Group=intgw<br>Environment=NODE_OPTIONS=&quot;--max-old-space-size=2048&quot;<br>ExecStart=/opt/elk_stack/kibana-9.0.0/bin/kibana<br>Restart=always<br><br>[Install]<br>WantedBy=multi-user.target</pre><h3><strong>Step 5: Logstash Installation &amp; Essential Config (Node 1)</strong></h3><p>Set up Logstash on es-node1 to function as the centralized ingestion and data processing engine. Configure an ingestion pipeline (logstash.conf) that accepts incoming log events from Filebeat, parses and enriches JSON payloads, and routes the formatted data to the Elasticsearch cluster securely over HTTPS.</p><pre>[intgw@es-node1 ~]$<br>wget https://artifacts.elastic.co/downloads/logstash/logstash-9.0.0-linux-x86_64.tar.gz<br>tar -xzf logstash-9.0.0-linux-x86_64.tar.gz</pre><p><strong>Systemd Unit: /etc/systemd/system/logstash.service</strong></p><pre>[Unit]<br>Description=Logstash 9.0.0<br>After=elasticsearch.service<br><br>[Service]<br>User=intgw<br>Group=intgw<br>ExecStart=/opt/elk_stack/logstash-9.0.0/bin/logstash -f /opt/elk_stack/logstash-9.0.0/config/logstash.conf<br>Restart=always<br>StandardOutput=journal<br>StandardError=inherit<br><br>[Install]<br>WantedBy=multi-user.target</pre><p><strong>Pipeline Configuration: /opt/elk_stack/logstash-9.0.0/config/logstash.conf</strong></p><pre>input {<br>  beats {<br>    port =&gt; 5044<br>    # Uncomment to enable TLS between Filebeat and Logstash<br>    # ssl =&gt; true<br>    # ssl_certificate =&gt; &quot;/opt/elk_stack/logstash-9.0.0/config/certs/logstash.crt&quot;<br>    # ssl_key =&gt; &quot;/opt/elk_stack/logstash-9.0.0/config/certs/logstash.key&quot;<br>  }<br>}<br><br>filter {<br>  # Parse JSON logs when &#39;message&#39; contains JSON<br>  json {<br>    source =&gt; &quot;message&quot;<br>    skip_on_invalid_json =&gt; true<br>  }<br>  # Add common fields<br>  mutate {<br>    add_field =&gt; { &quot;env&quot; =&gt; &quot;prod&quot; }<br>  }<br>}<br><br>output {<br>  elasticsearch {<br>    hosts =&gt; [&quot;https://es-node1:9200&quot;, &quot;https://es-node2:9200&quot;, &quot;https://es-node3:9200&quot;]<br>    index =&gt; &quot;app-logs-%{+YYYY.MM.dd}&quot;<br>    user =&gt; &quot;elastic&quot;<br>    password =&gt; &quot;&lt;ELASTIC_PASSWORD&gt;&quot;<br>    ssl =&gt; true<br>    cacert =&gt; &quot;/opt/elk_stack/elasticsearch-9.0.0/config/certs/http_ca.crt&quot;<br>    ilm_enabled =&gt; true<br>    ilm_rollover_alias =&gt; &quot;app-logs&quot;<br>    ilm_pattern =&gt; &quot;{now/d}-000001&quot;<br>  }<br>}</pre><p><strong>Optional: Generate Logstash Self-signed TLS Cert</strong></p><pre>openssl req -x509 -nodes -days 365 -newkey rsa:4096 \<br>  -keyout /opt/elk_stack/logstash-9.0.0/config/certs/logstash.key \<br>  -out /opt/elk_stack/logstash-9.0.0/config/certs/logstash.crt \<br>  -subj &quot;/CN=es-node1&quot;<br><br>chown -R intgw:intgw /opt/elk_stack/logstash-9.0.0/config/certs<br>chmod 600 /opt/elk_stack/logstash-9.0.0/config/certs/logstash.key</pre><h3><strong>Step 6: Filebeat Installation &amp; Essential Config (App Server)</strong></h3><p>Install Filebeat on application servers (app-srv-01) to collect log files. Filebeat efficiently ships logs to Logstash while supporting multiline events, metadata enrichment, buffering, and reliable log delivery.</p><pre>[intgw@app-srv-01 ~]$<br>wget https://artifacts.elastic.co/downloads/beats/filebeat/filebeat-9.0.0-linux-x86_64.tar.gz<br>tar -xzf filebeat-9.0.0-linux-x86_64.tar.gz</pre><p><strong>Systemd Unit: /etc/systemd/system/filebeat.service</strong></p><pre>[Unit]<br>Description=Filebeat 9.0.0<br>Documentation=https://www.elastic.co/guide/en/beats/filebeat/current<br>Wants=network-online.target<br>After=network-online.target<br><br>[Service]<br>User=intgw<br>Group=intgw<br>ExecStart=/opt/elk_stack/filebeat-9.0.0/filebeat -c /opt/elk_stack/filebeat-9.0.0/filebeat.yml<br>Restart=always<br><br>[Install]<br>WantedBy=multi-user.target</pre><p><strong>Configuration File: /opt/elk_stack/filebeat-9.0.0/filebeat.yml</strong></p><pre>filebeat.inputs:<br>  - type: log<br>    enabled: true<br>    paths:<br>      - /var/log/*.log<br>    multiline:<br>      pattern: &#39;^\[&#39;<br>      negate: true<br>      match: after<br><br>processors:<br>  - add_host_metadata: ~<br>  - add_cloud_metadata: ~<br>  - add_fields:<br>      target: &#39;&#39;<br>      fields:<br>        env: prod<br>  - drop_fields:<br>      fields: [&quot;agent.ephemeral_id&quot;, &quot;input.type&quot;]<br><br>output.logstash:<br>  hosts: [&quot;192.168.8.10:5044&quot;]<br>  # ssl.enabled: true<br>  # ssl.certificate_authorities: [&quot;/etc/pki/tls/certs/logstash_ca.crt&quot;]<br><br>queue.mem:<br>  events: 4096<br>  flush.min_events: 512<br>  flush.timeout: 5s</pre><h3><strong>Step 7: Finalizing Services</strong></h3><p>Enable all ELK services using systemd and verify they start automatically after system reboots. This final step ensures the complete logging pipeline is operational and ready for production workloads.</p><pre># On Node 1 (es-node1)<br>systemctl daemon-reload<br>systemctl enable --now elasticsearch kibana logstash<br><br># On Application Server (app-srv-01)<br>systemctl daemon-reload<br>systemctl enable --now filebeat</pre><h3><strong>Architecture Diagram</strong></h3><p><strong>ELK Stack Architecture (3-Node Cluster Ingestion &amp; Visualization Flow)</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/838/1*vE02OyEluHXluFd9UPOfkg.png" /></figure><h3><strong>Performance Tuning Tips for Elasticsearch</strong></h3><p>Below are proven, production-grade tuning recommendations for Elasticsearch 9.x on RHEL 9. Adjust values to match your hardware and workload (ingest vs. search-heavy).</p><p><strong>1) JVM Heap &amp; GC</strong></p><p>Set 50% of system RAM but &lt;= 32GB to preserve compressed OOPs. Match Xms and Xmx.</p><pre>-Xms8g<br>-Xmx8g<br>-XX:+UseG1GC<br>-XX:InitiatingHeapOccupancyPercent=30<br>-XX:+AlwaysPreTouch</pre><p><strong>2) OS &amp; Memory</strong></p><p>Lock memory to prevent swapping and disable Transparent Huge Pages (THP) at boot.</p><pre>cat &lt;&lt;EOF &gt;&gt; /etc/sysctl.d/99-elastic.conf<br>vm.swappiness=1<br>vm.dirty_ratio=10<br>vm.dirty_background_ratio=5<br>EOF<br>sysctl --system</pre><p><strong>3) Shards, Replicas &amp; Indices</strong></p><p>Target ~1 primary shard per 20–50GB of index size. Avoid many tiny shards. Use ILM (Index Lifecycle Management) to rollover daily/size-based.</p><pre>PUT /_index_template/app-logs<br>{<br>  &quot;index_patterns&quot;: [&quot;app-logs-*&quot;],<br>  &quot;template&quot;: {<br>    &quot;settings&quot;: {<br>      &quot;number_of_shards&quot;: 1,<br>      &quot;number_of_replicas&quot;: 1,<br>      &quot;refresh_interval&quot;: &quot;30s&quot;<br>    }<br>  }<br>}</pre><p><strong>4) Ingest &amp; Refresh</strong></p><p>Increase refresh_interval to 30s (or higher) during heavy ingest. Use _bulk with action groups of 5–15MB payloads.</p><p><strong>5) Thread Pools &amp; Queues</strong></p><p>Inspect thread pools and monitor queues for saturation. If bulk queues saturate, reduce bulk size or add data nodes.</p><pre>GET _nodes/thread_pool<br>GET _cat/thread_pool?v</pre><p><strong>6) Disk I/O &amp; Storage Layout</strong></p><p>Use NVMe/SSD for data paths. Separate OS and data disks. Prefer XFS over ext4 for Elasticsearch data volumes and mount with noatime.</p><p><strong>7) Search Performance</strong></p><p>Use keyword fields for aggregations, avoid text unless full-text search is required. Leverage pre_filter_shard_size and request_cache.</p><p><strong>8) Logstash &amp; Beats Throughput Tips</strong></p><p>Set pipeline.workers close to available CPU cores for parsing-heavy workloads and increase batch size gradually.</p><pre>- pipeline.id: main<br>  path.config: &quot;/opt/elk_stack/logstash-9.0.0/config/logstash.conf&quot;<br>  pipeline.workers: 2<br>  pipeline.batch.size: 125<br>  pipeline.batch.delay: 5</pre><p><strong>Troubleshooting Essentials</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/949/1*EmDRRBCDfnE8Hv62UtEQzA.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/946/1*KstzIvqBjeBlUazf2fBHxQ.png" /></figure><p><strong>Appendix: Key Directory Reference</strong></p><p><strong>• Elasticsearch Base Directory: </strong>/opt/elk_stack/elasticsearch-9.0.0/</p><p><strong>• Logstash Configuration Directory: </strong>/opt/elk_stack/logstash-9.0.0/config/</p><ul><li><strong>Filebeat Configuration File: </strong>/opt/elk_stack/filebeat-9.0.0/filebeat.yml</li></ul><p><em>The views and opinions expressed in this article are those of the author and do not necessarily reflect the official policy or position of Axonect</em></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=6c01b967b23c" width="1" height="1" alt=""><hr><p><a href="https://medium.com/axonect-blog/a-guide-to-deploying-elasticsearch-9-0-0-securely-on-rhel-9-6c01b967b23c">A Guide to Deploying Elasticsearch 9.0.0 Securely on RHEL 9</a> was originally published in <a href="https://medium.com/axonect-blog">Axonect Blog</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
    </channel>
</rss>