<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Prithvi Raju Alluri on Medium]]></title>
        <description><![CDATA[Stories by Prithvi Raju Alluri on Medium]]></description>
        <link>https://medium.com/@alluri.prithvi?source=rss-85b7dbaf2e61------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/1*wewTTbhaqFt06STtAvcypw.jpeg</url>
            <title>Stories by Prithvi Raju Alluri on Medium</title>
            <link>https://medium.com/@alluri.prithvi?source=rss-85b7dbaf2e61------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Tue, 17 Nov 2026 17:29:48 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@alluri.prithvi/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[LogMate: The AI Sidekick I Built To Tame Kubernetes Logs Without Leaving My Cluster]]></title>
            <link>https://medium.com/@alluri.prithvi/logmate-the-ai-sidekick-i-built-to-tame-kubernetes-logs-without-leaving-my-cluster-9e2f4148738f?source=rss-85b7dbaf2e61------2</link>
            <guid isPermaLink="false">https://medium.com/p/9e2f4148738f</guid>
            <category><![CDATA[kubernetes]]></category>
            <category><![CDATA[llmops]]></category>
            <category><![CDATA[observability]]></category>
            <category><![CDATA[devops]]></category>
            <category><![CDATA[aiops]]></category>
            <dc:creator><![CDATA[Prithvi Raju Alluri]]></dc:creator>
            <pubDate>Mon, 13 Oct 2025 18:07:53 GMT</pubDate>
            <atom:updated>2025-10-14T05:41:46.232Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Dgj02c69C0eAc3U_WqBiuw.png" /></figure><p><strong>Picture this: it’s 2:13 a.m.,</strong> Grafana is bleeding red, and kubectl logs -n kube-system is returning an endless firehose of doom. I’d been through that war room one time too many, copy‑pasting log fragments into yet another SaaS tool, praying that network latency or redacted data wouldn’t hide the smoking gun. That night I decided to build a guardian that would live where the chaos happens — inside the cluster — and fight back with AI on my own terms. Meet LogMate.</p><p>GitHub: <a href="http://github.com/prithvidbox/k8s_logs">github.com/prithvidbox/k8s_logs</a></p><h3>The Pain That Sparked It</h3><ul><li>Latency kills insight. Every hop to a managed log platform adds seconds of round‑trip purgatory when you should be triaging.</li><li>Data sovereignty matters. Shipping pod output off-prem was a nonstarter for regulated workloads.</li><li>Visual visibility was missing. A pile of stderr spew isn’t a dashboard, and my tired 2 a.m. brain wanted color-coded clues, not grep gymnastics.</li></ul><p>I wanted real anomaly detection, semantic intelligence, and slick charts — but powered by my hardware, not someone else’s endpoints.</p><h3>Building LogMate: My Local-First AI Stack</h3><ol><li>Streamlit UI (but not your beginner demo). Custom gradient theming, metric cards, tabbed insights, and a manual refresh cadence so the Python runtime doesn’t thrash between polls.</li><li>AI in reach. I embed every log line with all-MiniLM-L6-v2, caching the embeddings locally. FAISS gives millisecond similarity search — perfect when an obscure error line whispers “I’ve seen this before.”</li><li>Anomaly brain that never leaves home. TF‑IDF + K-Means identifies low-population clusters — the “what just happened?” moments — without exchanging a single byte with external APIs. It’s autonomous AI living in the same namespace as your workloads.</li><li>Taxonomy intelligence. Regex-driven classifiers tag logs as “Critical Failure”, “Scheduling”, “Networking” and more at render time. Control-plane meltdown? It stands out instantly.</li><li>SQLite persistence. Every fetch lands in a local SQLite store. No cloud bucket, no data exhaust, just a rolling corpus you can snapshot or grep later.</li><li>Prometheus-tight integration. kubernetes_log_count, kubernetes_error_logs, and kubernetes_anomalies are emitted from a single Prometheus server thread (port 8090). Grafana gets the last-mile glamour, while LogMate stays the command center.</li></ol><p>All of this sits behind one command: streamlit run <a href="http://app.py/">app.py</a>. No agents, no gRPC sidecars, no billing surprises. Bring your kubeconfig; LogMate handles the rest.</p><p><a href="https://vimeo.com/1127058996?share=copy">Verify to continue</a></p><h3>A Night in the Life With LogMate</h3><p>The next incident hit during a routine rollout. I triggered LogMate and watched:</p><ul><li>Namespace Footprint chart erupt as one tenant flooded kube-system.</li><li>Anomaly Detection tab flash a high-severity banner: “24 anomalies found.”</li><li>The AI-backed detection table stacked a wall of CrashLoopBackOff logs tagged “Container Lifecycle.” Problem solved before PagerDuty escalated.</li></ul><p>No external API keys, no throttled rate limits. Just me, the cluster, and a local AI that didn’t need an internet connection to do its job.</p><h3>Tech Deep Dive</h3><p>LogMate stacks a Kubernetes Python client fetcher, SQLite storage, sentence-transformers embeddings (all-MiniLM-L6-v2), FAISS CPU similarity search, scikit-learn TF-IDF + KMeans anomaly engine, Streamlit UI, and prometheus_client metrics — each chosen for low-latency, local-first observability that respects your kubeconfig, keeps data on disk, serves sub-millisecond semantic queries, surfaces deterministic anomalies, delivers a polished control panel, and emits Grafana-ready telemetry without external dependencies.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1000/0*jInLk2svRwCSZJOY" /></figure><p>I tuned everything for local execution. Peak memory is tame, CPU usage sits comfortably under a core on a MacBook Pro, and the only network call is to the Kubernetes API server you already trust.</p><h3>Try It Yourself</h3><pre>git clone https://github.com/prithvidbox/k8s_logs.git<br>cd k8s_logs<br>pip install -r requirements.txt</pre><pre>cat &lt;&lt;&#39;ENV&#39; &gt; .env<br>METRICS_SERVER=True<br>ANOMALY_THRESHOLD=3<br>ENV</pre><pre>streamlit run app.py</pre><pre>Visit http://localhost:8501. Want metrics? Point Prometheus at localhost:8090 and build dashboards around the three custom time-series.</pre><p>I built LogMate because I was tired of shipping sensitive logs to someone else to get “AI-powered insights.” If you feel the same — if local-first AI resonates with your resilience strategy — give LogMate a spin, star the repo, or help me push the roadmap forward.</p><p>👉 GitHub: <a href="http://github.com/prithvidbox/k8s_logs">github.com/prithvidbox/k8s_logs</a></p><p>Let’s keep the data close, the insights sharp, and the on-call pager quiet.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=9e2f4148738f" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Everything about Cloud Security]]></title>
            <link>https://medium.com/@alluri.prithvi/everything-about-cloud-security-6f394749c4b2?source=rss-85b7dbaf2e61------2</link>
            <guid isPermaLink="false">https://medium.com/p/6f394749c4b2</guid>
            <category><![CDATA[access-control]]></category>
            <category><![CDATA[aws]]></category>
            <category><![CDATA[cloud]]></category>
            <category><![CDATA[security]]></category>
            <category><![CDATA[hacking]]></category>
            <dc:creator><![CDATA[Prithvi Raju Alluri]]></dc:creator>
            <pubDate>Sat, 01 Aug 2020 12:48:19 GMT</pubDate>
            <atom:updated>2020-08-01T12:53:58.623Z</atom:updated>
            <content:encoded><![CDATA[<p>Cloud data security is vital, as you will want to be sure that your data is safe while stored in the cloud. A number of high-profile hacking cases mean that this issue is topical for many business owners, but the reality is that your data is much safer in the cloud, and security is an extremely high priority for all cloud storage services.</p><p>Let me start with an incident from the past.</p><blockquote>Hackers have stolen the personal information of about 57 million customers and drivers from Uber in the fall of 2016</blockquote><p><strong>What happened :</strong> This happened when two hackers gained access to Uber’s private GetHub account. Upon gaining access to Uber’s GetHub data, the hackers were able to extract the company’s AWS credentials.</p><p><strong>The problem</strong> was not related to any sort of AWS configuration issue, but instead was related to a leaked password. The most obvious way to prevent the problem would have been for Uber to use stronger protection for its GetHub account, and avoid storing its AWS passwords within GitHub.</p><blockquote><strong>Fun Fact :</strong> Uber tried to hide this news with a 10000$ agreement with the hacker.</blockquote><p>Hacker need not be a pro to hack into systems in these scenarios, one can just scan all commits happening on GitHub catch matching RegEx using a 20 line code and get access to every company proprietary details.</p><p><strong>Try for your self :</strong> <a href="https://shhgit.darkport.co.uk/">https://shhgit.darkport.co.uk/</a></p><p>More examples from the past caused to due to lack of security and access managment on public clouds :</p><p>Feb 2017 — CloudPets — 583k user accounts, password hashes and children’s voice messages</p><p>Sep 2018 — Veeam — 445M customer records and marketing details</p><p>Apr 2019 — Rubrik — 10+ gigabytes of customer information</p><p>Oct 2019 — Adobe — 7M creative cloud user accounts</p><p>Oct 2019 — Best Western — 179Gb of govt user booking reservation details, including personal information</p><blockquote>Problem</blockquote><h4><strong>7 common mistakes</strong> on AWS</h4><p><strong>Not knowing who is in charge of security</strong> — Security is a shared responsibility, AWS does not own for mistakes you do creating resources with default configurations is a big NO.</p><p><strong>Giving away too many privileges</strong> — While it is tempting to give developers with Admin access, you shouldn’t. More than 35 percent of privileged users in AWS have full access to a wide variety of services, including the ability to bring down the whole customer AWS environment. Policies should include the least amount of permissions to get a job done.</p><p><strong>Having powerful users and broad roles</strong> — AWS Identity and Access Management (IAM) is critical for securing AWS deployments, A common misconfiguration is to assign access to the complete set of permissions for each AWS item. If the application needs the ability to write files to Amazon S3 and it has full access to S3, it can read, write, and delete every single file in S3 for that account. More over this service is free to use.</p><p><strong>Exposed secrets and keys</strong> — Credentials are often found hard-coded into application source code, or configuration files containing keys and passwords are stored in publicly accessible locations. AWS keys have been exposed in public repositories over the years. GitHub now regularly scans public repositories to alert developers about exposed AWS credentials. Keys should be regularly rotated. Don’t be the administrator who lets too much time pass between rotations. IAM is powerful, but many of its features are frequently ignored.</p><p><strong>Not taking root seriously </strong>— Admins forget to disable Root API access — a highly risky practice. No one should be using the AWS root account and associated keys, let alone sharing them across users and applications. Keys to access AWS resources directly should be used sparingly, as the keys need to be tracked, managed, and protected.</p><p><strong>Putting everything in one VPC or account </strong>— The more teams and workloads added to an account or Virtual Private Cloud (VPC), the more likely you are to meet the lowest common denominator. AWS has very generous limits on VPCs and accounts. There’s no reason not to <a href="https://docs.aws.amazon.com/AmazonVPC/latest/UserGuide/VPC_Appendix_Limits.html">isolate workloads and teams</a> into different regions, VPCs, or even accounts. The simplest way to start is to make sure that development, testing, and production are in different accounts.</p><p><strong>Leaving wide open connections</strong> — Too many admins enable global permissions to instances. When you use 0.0.0.0/0, you are giving every machine everywhere the ability to connect to your AWS resources. “You wouldn’t leave the front door to your house open, why do you use 0.0.0.0/0?” One-third of the top 30 common AWS configuration mistakes identified involve open ports. Thanks to high-quality automation tools such as OpsWorks, Chef, Ansible, and Puppet, there’s no reason to allow remote access — such as SSH or RDP — to EC2 instances.</p><blockquote>SOLUTION</blockquote><p>There is no solution for human errors, You should be implementing below practices to have a governance on cloud assets and access management and therefore your/client data.</p><p><strong>Strengthen Authentication Controls </strong>— Use single sign-on to facilitate account management and improve the user experience. Supplement it with multifactor authentication to prevent the reuse of stolen credentials.</p><p><strong>Deploy Endpoint Protection </strong>— Next-generation endpoint protection is a critical security control if you run full operating systems in the cloud. If you’re running containerized computing, make sure that you’ve deployed appropriate controls there as well.</p><p><strong>Implement IAM Best Practices</strong> — Assign identities to every person, application and object in your environment with an identity and access management tool, and then create policies that restrict access based upon time of day, length of access, geolocation and other criteria.</p><p><strong>Enforce the Principle of Least Privilege</strong> — Limit administrative users to modifying cloud objects within their specific scope of responsibility. Avoid the use of shared administrative accounts to maintain accountability.</p><p><strong>Monitor Your Cloud Environment Carefully</strong> — Deploy technology that reaches directly into public cloud environments such as Microsoft Azure and Amazon Web Services to monitor the security of your infrastructure. This is particularly important for environments that must be compliant with the Payment Card Industry Data Security Standard, HIPAA, ISO 27001 and other security standards.</p><p><strong>Gain Visibility and Control with a CASB Tool </strong>— Cloud access security broker (CASB) tools offer active scanning and enforcement of security policies in a consistent manner across cloud services.</p><p><strong>Capture and Monitor NetFlow Data</strong> — Understanding the flow of data between cloud servers helps you identify abnormal activity. This is a customer responsibility under the shared responsibility model.</p><p><strong>Conduct Penetration Testing </strong>— Periodic testing provides visibility into security gaps and helps you assess the state of your security posture.</p><p><strong>Correlate Logs with SIEM Technology</strong> — Security information and event management (SIEM) technology can identify suspicious patterns of activity across multiple systems that might otherwise go undetected.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=6f394749c4b2" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[A DevOps episode in Monolith To Microservices journey]]></title>
            <link>https://medium.com/@alluri.prithvi/a-devops-episode-in-monolith-to-microservices-journey-6c3697ead2bf?source=rss-85b7dbaf2e61------2</link>
            <guid isPermaLink="false">https://medium.com/p/6c3697ead2bf</guid>
            <category><![CDATA[technology]]></category>
            <category><![CDATA[startup]]></category>
            <category><![CDATA[microservices]]></category>
            <category><![CDATA[scale]]></category>
            <category><![CDATA[devops]]></category>
            <dc:creator><![CDATA[Prithvi Raju Alluri]]></dc:creator>
            <pubDate>Wed, 14 Nov 2018 12:43:13 GMT</pubDate>
            <atom:updated>2018-11-14T13:35:01.703Z</atom:updated>
            <content:encoded><![CDATA[<p>Past couple of years I’m working closely with teams and organisations that are looking to solve problems with scale using proven methodologies.</p><p>In this blog I will present my view on Micro services and DevOps. I have built this opinion on analysing and working on multiple case studies over a period of time</p><p><a href="https://eng.uber.com/building-tincup/">Uber case study</a> on Microservices implementation is a big influencer, which can be summarised in a quote</p><blockquote>“Your team setup says more about your architecture than the architecture itself.” — <a href="https://www.youtube.com/watch?v=Lwdhw8aI8es">Raffi Krikorian</a>, Uber</blockquote><p>There is no formal definition for microservices and there is no standard model that represents every system based on this architectural style.</p><p>If you are starting with a greenfield application and have a small team of developers, you are free to start building using microservices and never look back. You don’t have organisational constraints because for all practical purposes, you don’t have any organisation. In such a situation, it’s easy to focus only on the software engineering side of your architecture, because the human engineering side is trivial, obvious, and intuitive. It’s easy to think that such will be the case for any organisation of any size — an easy mistake.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/640/0*Y4AMFWjyo9WRWaqV.png" /></figure><p><strong>What if you already have a huge team and a large monolithic application? What if you want to move towards microservices?<em> Enter Conway’s Law.</em></strong></p><p>Many years ago, Melvin Conway had observed that how organisations were structured would have a strong impact on any systems they created.</p><p><em>“Any organization that designs a system (defined more broadly here than just information systems) will inevitably produce a design whose structure is a copy of the organization’s communication structure.”</em></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/638/0*oXUTX5oJua6dC0nc" /></figure><p>Put simply, there is no way for you to succeed unless you change your development team such that individual teams of developers are responsible for each microservice. If you want to avoid chaos, you better consider whether or not you want a complex multi-dimensional mapping, a simple hierarchy, or some combination. The accountability has to be clear and someone has to own the overall design. Otherwise, you will end up with a mush of itty bitty services that are redundant, inconsistent, and ultimately of poor quality.</p><blockquote><strong>In the current generation of technology a organisation that appreciated failures and values speed over values triumphs</strong></blockquote><h4>Introducing Devops practices to handle the transition and chaos</h4><p>DevOps practices run deep in modern application architectures. DevOps practices have helped create a space for developers and engineers to build new ways to optimise resources and scale-out application architectures through continuous delivery practices.</p><p>Cloud-native applications are based on a set of loosely coupled components, or microservices, that run for the most part on containers, and are managed with orchestration engines such as <a href="https://kubernetes.io/">Kubernetes</a>. However, they are also beginning to run as a set of discrete functions in serverless architectures. Services or functions are defined by developer and engineering teams, then continuously built, rebuilt and improved by increasingly cross-functional teams. Operations are now less focused on the infrastructure and more on the applications that run light workloads. The combined effect is a shaping of automated processes that yield better efficiencies.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/750/0*qGEu_XXe2O_LRE04.jpg" /></figure><blockquote><strong>Tools Simplify Processes but Don’t Solve Problems, but the process built around the tools will !</strong></blockquote><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=6c3697ead2bf" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Application Logs from Kubernetes to S3 and Elasticsearch using Fluentd]]></title>
            <link>https://medium.com/@alluri.prithvi/application-logs-from-kubernetes-to-s3-and-elasticsearch-using-fluentd-2f1b09a9872e?source=rss-85b7dbaf2e61------2</link>
            <guid isPermaLink="false">https://medium.com/p/2f1b09a9872e</guid>
            <category><![CDATA[logging]]></category>
            <category><![CDATA[docker]]></category>
            <category><![CDATA[kubernetes]]></category>
            <category><![CDATA[fluentd]]></category>
            <dc:creator><![CDATA[Prithvi Raju Alluri]]></dc:creator>
            <pubDate>Mon, 18 Jun 2018 21:16:01 GMT</pubDate>
            <atom:updated>2018-06-18T21:35:34.907Z</atom:updated>
            <content:encoded><![CDATA[<h4><strong>Problem Statement :</strong></h4><p>Application logging is an important part of software development lifecycle, deploying a solution for log management in Kubernetes is simple when log’s are written to stdout ( Best practise ). However when an monolithic stack is converted to container and pushed to Kubernetes, there are many traditional log paths that should be used to collect and aggregate logs.</p><p>I have tried many helm chart’s for logging kubernetes, there are many articles and repo’s for collelcting logs from K8 stack but there is no easy way to pull it from within a container.</p><h4><strong>Scenario</strong></h4><p>Multiple micro services running in Kubernetes cluster deployed in AWS, Application logs are being stored on different paths inside docker containers ( not stdout ). These logs are needed in S3 for log analytics and long term retention and in Elasticsearch for real time log aggregation and visualisation.</p><h4><strong>Solution :</strong></h4><p>Easy to deploy with customisations. No change needed in application logging. Out of box log tool set’s deployed along with application containers but isolated and not impacting performance of app workload. Globally defined log functions which can be plugged into any new application that is being deployed. Secured aggregation pipeline.</p><h4><strong>Stack Used :</strong></h4><p>Kubernetes Side Car containers, Fluentd, AWS Elasticsearch, S3 and Obviously Dockers</p><h4><strong>Architecture :</strong></h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*93zHyhexMQ6n93jhck-_sQ.png" /></figure><p>Source code and reference deployment manifests can be found <a href="https://github.com/prithviraju/kubernetes-fluentd-logging">here</a></p><p><a href="https://github.com/prithviraju/kubernetes-fluentd-logging">prithviraju/kubernetes-fluentd-logging</a></p><blockquote>If you know what you are doing after looking at the Repo ! <strong>don’t scroll down,</strong> it’s just me explaining steps for deploying this solution for newbies.</blockquote><h3><strong>Implementation :</strong></h3><p>I’m going to assume that you have a Kubernetes cluster at hand for this walkthrough.</p><p><em>Note : Good news is I do not have any pv,pvc’s,nfs included in this architecture , We would be reading logs on the fly and pushing them indexed data out side kubernetes cluster, I did not use any state full set’s here.</em></p><pre>➜  Documents git clone  <a href="https://github.com/prithviraju/kubernetes-fluentd-logging.git">https://github.com/prithviraju/kubernetes-fluentd-logging.git</a><br>Cloning into &#39;kubernetes-fluentd-logging&#39;...<br>remote: Counting objects: 14, done.<br>remote: Compressing objects: 100% (11/11), done.<br>remote: Total 14 (delta 0), reused 11 (delta 0), pack-reused 0<br>Unpacking objects: 100% (14/14), done.</pre><pre>➜  kubernetes-fluentd-logging git:(master) tree<br>.<br>├── Dockerfile<br>├── LICENSE<br>├── README.md<br>├── fluent.conf<br>├── kubernetes.conf<br>├── sample_deployment.yaml<br>└── systemd.conf</pre><p>There are no big changes needed in the configuration files. You should be looking at fluent.conf file to add your environment specify variables.</p><p>Update fluent.conf by replacing items marker with ###</p><pre><a href="http://twitter.com/include">@include</a> systemd.conf<br><a href="http://twitter.com/include">@include</a> kubernetes.conf</pre><pre>## App 1 config <br>&lt;source&gt;<br>  <a href="http://twitter.com/type">@type</a> tail<br>  <a href="http://twitter.com/id">@id</a> XXXX ### Uniq ID<br>  path XXXX ### Log path<br>  pos_file XXXX ### pos file<br>  tag XXXXX ### App tag <br>  read_from_head true<br>  &lt;parse&gt;<br>    <a href="http://twitter.com/type">@type</a> regexp<br>    keep_time_key true<br>    expression (?&lt;logtime&gt;.*?) - (?&lt;names&gt;[^ ]*) - (?&lt;log_level&gt;[^ ]*) - (?&lt;message&gt;.*)<br>    time_format %Y/%m/%d %H:%M:%S,%6N%z<br>  &lt;/parse&gt;<br>&lt;/source&gt;</pre><pre>## App 2 config</pre><pre>&lt;source&gt;<br>  <a href="http://twitter.com/type">@type</a> tail<br>  <a href="http://twitter.com/id">@id</a> XXXXX ## Uniq ID<br>  path XXXX ### Log path <br>  pos_file XXXX ### pos file   <br>  tag XXXXX ## App tag <br>  read_from_head true<br>  &lt;parse&gt;<br>    <a href="http://twitter.com/type">@type</a> regexp<br>    keep_time_key true<br>    expression (?&lt;logtime&gt;.*?) - (?&lt;names&gt;[^ ]*) - (?&lt;log_level&gt;[^ ]*) - (?&lt;message&gt;.*)<br>    time_format %Y/%m/%d %H:%M:%S,%6N%z<br>  &lt;/parse&gt;<br>&lt;/source&gt;</pre><pre># Store Data in Elasticsearch and S3<br>&lt;match **&gt;<br>  <a href="http://twitter.com/type">@type</a> copy<br>  deep_copy true<br>  &lt;store&gt;</pre><pre><a href="http://twitter.com/type">@type</a> s3<br>  aws_key_id XXXXXXXXXXXXXXXXX # Access Key <br>  aws_sec_key XXXXXXXXXXXXXXXXXXXXXX # Secret Key <br>  s3_bucket XXXXXX # S3 Bucket<br>  s3_region XXXXX # Bucket Region <br>  path &quot;#{ENV[&#39;S3_LOGS_BUCKET_PREFIX&#39;]}&quot;<br>  buffer_path /var/log/fluent/s3<br>  s3_object_key_format %{path}%{time_slice}/cluster-log-%{index}.%{file_extension}<br>  time_slice_format %Y%m%d-%H<br>  time_slice_wait 10m<br>  flush_interval 60s<br>  buffer_chunk_limit 256m<br>  &lt;/store&gt;<br> <br> &lt;store&gt;<br>   <a href="http://twitter.com/type">@type</a> elasticsearch<br>   <a href="http://twitter.com/id">@id</a> out_es<br>   log_level info<br>   include_tag_key true<br>   host &quot;#{ENV[&#39;FLUENT_ELASTICSEARCH_HOST&#39;]}&quot;<br>   port &quot;#{ENV[&#39;FLUENT_ELASTICSEARCH_PORT&#39;]}&quot;<br>   scheme &quot;#{ENV[&#39;FLUENT_ELASTICSEARCH_SCHEME&#39;] || &#39;https&#39;}&quot;<br>   ssl_verify &quot;#{ENV[&#39;FLUENT_ELASTICSEARCH_SSL_VERIFY&#39;] || &#39;true&#39;}&quot;<br>   reload_connections &quot;#{ENV[&#39;FLUENT_ELASTICSEARCH_RELOAD_CONNECTIONS&#39;] || &#39;true&#39;}&quot;<br>   logstash_prefix &quot;#{ENV[&#39;FLUENT_ELASTICSEARCH_LOGSTASH_PREFIX&#39;] || &#39;applogs&#39;}&quot;<br>   logstash_format true<br>   &lt;buffer&gt;<br>     flush_thread_count 8<br>     flush_interval 5s<br>     chunk_limit_size 2M<br>     queue_limit_length 32<br>     retry_max_interval 30<br>     retry_forever true<br>   &lt;/buffer&gt;<br>  &lt;/store&gt;<br>&lt;/match&gt;</pre><p>Take a moment in understanding this configuration file.</p><p>We are ready to build our image and use it in our Kubernetes deployment manifest.</p><p>I would be using ECR to store my images.</p><pre>➜  kubernetes-fluentd-logging git:(master) docker build  -t 123456789.dkr.ecr.ap-south-1.amazonaws.com/app-prod:logging-v1 .</pre><pre>➜  kubernetes-fluentd-logging git:(master) docker push 123456789.dkr.ecr.ap-south-1.amazonaws.com/app-prod:logging-v1</pre><h4>Updating your Deployment file</h4><p>Look sample_deployment.yam lwhich is an extacr from one of my production deployment</p><p>In this example I’m interested in logging two log files from my container /var/log/access/app.log and /var/log/error/error.log.</p><p>Take a minute to understand below file.</p><pre>---<br>apiVersion: extensions/v1beta1<br>kind: Deployment<br>metadata:<br>  annotations:<br>  labels:<br>    app: app1<br>  name: app1<br>  namespace: app1<br>spec:<br>  replicas: 3<br>  strategy:<br>    rollingUpdate:<br>      maxSurge: 1<br>      maxUnavailable: 1<br>    type: RollingUpdate<br>  template:<br>    metadata:<br>      labels:<br>        app: app1<br>    spec:<br>      affinity:<br>        podAntiAffinity:<br>          preferredDuringSchedulingIgnoredDuringExecution:<br>            - podAffinityTerm:<br>                topologyKey: kubernetes.io/hostname<br>              weight: 100<br>      containers:<br>        - name: app1<br>          image: &quot;IMAGE:VERSION&quot;<br>          env:<br>            - name: NODE_APP<br>              value: &quot;&quot;<br>          imagePullPolicy: IfNotPresent<br>          livenessProbe:<br>            httpGet:<br>              path: /health-check<br>              port: 80<br>            initialDelaySeconds: 30<br>            periodSeconds: 15<br>            timeoutSeconds: 5</pre><pre>readinessProbe:<br>            httpGet:<br>              path: /health-check<br>              port: 80<br>            initialDelaySeconds: 30<br>            periodSeconds: 15<br>            timeoutSeconds: 5<br>          volumeMounts:<br>          - name: applog-error<br>            mountPath: /var/log/access<br>          - name: applog-error<br>            mountPath: /var/log/error<br>          terminationMessagePath: /dev/termination-log<br>          terminationMessagePolicy: File</pre><pre>- name: fluentd<br>          image: IMAGE:VERSION_LOG<br>          env:<br>            - name: INSTANCE<br>              value: &quot;app1&quot;<br>            - name:  FLUENT_ELASTICSEARCH_HOST<br>              value: &quot;&quot;<br>            - name:  FLUENT_ELASTICSEARCH_PORT<br>              value: &quot;443&quot;<br>            - name: FLUENT_ELASTICSEARCH_SCHEME<br>              value: &quot;https&quot;<br>            - name: S3_LOGS_BUCKET_PREFIX<br>              value: &quot;app-logs/app1/&quot;<br>            - name: FLUENT_ELASTICSEARCH_LOGSTASH_PREFIX<br>              value: &quot;app1&quot;<br>            - name: FLUENT_ELASTICSEARCH_SSL_VERIFY<br>              value: &quot;true&quot;<br>            - name: FLUENT_ELASTICSEARCH_USER<br>              value: &quot;&quot;<br>            - name: FLUENT_ELASTICSEARCH_PASSWORD<br>              value: &quot;&quot;<br>            - name: FLUENT_ELASTICSEARCH_RELOAD_CONNECTIONS<br>              value: &quot;false&quot;<br>          imagePullPolicy: Always<br>          resources:<br>            limits:<br>              memory: 100Mi<br>            requests:<br>              cpu: 50m<br>              memory: 100Mi<br>          volumeMounts:<br>          - name: applog-error<br>            mountPath: /log/access<br>          - name: applog-error<br>            mountPath: /log/error<br>      dnsPolicy: ClusterFirst<br>      restartPolicy: Always<br>      schedulerName: default-scheduler<br>      securityContext: {}<br>      terminationGracePeriodSeconds: 30<br>      volumes:<br>      - name: applog-access<br>        emptyDir: {}<br>      - name: applog-error<br>        emptyDir: {}</pre><p>Here /var/log/access from my app container is /log/access on my side car and /var/log/error is /log/error.</p><blockquote>Briefly we have two volumes applog-access and applog-error mounted on both the containers.</blockquote><p>The volume attached to this pod is a emptyDir, which means the life of the volume attached to both the containers are only until the pod is in running state.</p><p>Make sure your fluent.conf is looking at these paths for logs in path section of the file.</p><p>You will not get this right on the first deployment, at-least I did not! below kubectl command can be used to tail logs from the pod.</p><pre>kubectl get pods -n $NAMESPACE<br>kubectl logs $POD_ID -n $NAMESPACE -c $APP/FLUENTD -f </pre><p>If no error’s you should be seeing logs in S3 and Elasticsearch</p><pre>S3 directory strcuture. </pre><pre>├── S3-BUCKET<br>    |-- APP1<br>         |-- 20180613-19<br>                |-- cluster-log-0.gz<br>                |-- cluster-log-1.gz<br>         |-- 20180613-20<br>    |-- APP2</pre><p>You can access Kibana to get realtime log visualisation.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*3nUYqj_vNTC63HgKyWaZMA.png" /></figure><p>Happy Logging !</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=2f1b09a9872e" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[ELB Logs to ELK | Serverless]]></title>
            <link>https://medium.com/@alluri.prithvi/elb-logs-to-elk-serverless-5df239e05f1c?source=rss-85b7dbaf2e61------2</link>
            <guid isPermaLink="false">https://medium.com/p/5df239e05f1c</guid>
            <category><![CDATA[devops]]></category>
            <category><![CDATA[automation]]></category>
            <category><![CDATA[serverless]]></category>
            <category><![CDATA[elk]]></category>
            <category><![CDATA[aws]]></category>
            <dc:creator><![CDATA[Prithvi Raju Alluri]]></dc:creator>
            <pubDate>Wed, 24 Jan 2018 09:24:20 GMT</pubDate>
            <atom:updated>2018-01-24T14:39:20.835Z</atom:updated>
            <content:encoded><![CDATA[<h3>Problem Statement :</h3><p>Multiple services spread across different datacenters, hundreds of servers and millions of log records. Debugging an issue requires logging into each individual server to look at log’s.</p><p>As the system scales to more servers, this will produce even more logs. It is a challenge to log, index, search, store, visualise, and analyse these logs every day. Also security in terms of access control, storage and transport has to be implemented.</p><h3>Solution</h3><p>Collecting all log’s from Loadbalancers to a centralised place and provision access to developers/operations team to query log’s on demand seamlessly. This will help engineering team take better decisions that are data driven.</p><h3>Choice of Solution.</h3><ul><li>OpenSource and Serverless</li><li>Highly avilable and enachanced security for data stored and in trasit.</li><li>User Friendly visulization dashboards and reports.</li><li>Ease of log managment, archival and retreval.</li><li>No Operational overhead.</li><li>Immutable and autoscaled.</li></ul><h3>What is ELK stack?</h3><p>ELK stands for Elasticsearch, Logstash and Kibana.</p><h4>Brief definitions:</h4><p><strong>Logstash:</strong> It is a tool for managing events and logs. You can use it to collect logs, parse them, and store them for later use (like, for searching). Speaking of searching, logstash comes with a web interface for searching and drilling into all of your logs. It is fully free and fully open source.</p><p><strong>Elasticsearch:</strong> Elasticsearch is a search server based on Lucene. It provides a distributed, multitenant-capable full-text search engine with a RESTful web interface and schema-free JSON documents.</p><p><strong>Kibana: </strong>A nifty tool to visualize logs and timestamped data.</p><p><strong>ELBLogs: </strong>Elastic Load Balancing publishes a log file for each load balancer node at the interval you specify. You can specify a publishing interval of either 5 minutes or 60 minutes when you enable the access log for your load balancer.</p><p><strong>Lambda Functions: </strong>AWS Lambda is a compute service that lets you run code without provisioning or managing servers. AWS Lambda executes your code only when needed and scales automatically, from a few requests per day to thousands per second.</p><h3>Problem with traditional ELK | EC2 deployments:</h3><p>Installing ELK is usually hassel free, however as the system grows connecting dot’s between log managment peices will be a pain. Setting ELK stack in autoscaling mode is quite a task. Log managment Machines that aren’t delivering the product but still need to be managed (and paid for)</p><blockquote>Let’s not worry about monitoring monitoring tools.</blockquote><blockquote><strong>Truely serverless</strong> | ELB -&gt; S3 -&gt; Lambda -&gt; AWS ES -&gt; AWS Kibana</blockquote><blockquote>There are tons of approaches to achieve this; we choose to use a combination of various AWS resources for aggregating log’s seamlessly with a server less technique.</blockquote><figure><img alt="" src="https://cdn-images-1.medium.com/max/721/1*923NtTgliVFeBXVmbf51_Q.png" /></figure><h3>Enabling ELB Logging</h3><p>ELB logging is turned off by default, so our first step is to enable logging on all required ELB’s and point them to a s3 bucket .</p><p>Below guide should help with logging setup.</p><p><a href="https://docs.aws.amazon.com/elasticloadbalancing/latest/classic/enable-access-logs.html">Enable Access Logs for Your Classic Load Balancer - Elastic Load Balancing</a></p><blockquote>Note : We choose to push all ELB logs to a single bucket with different prefix. This helped us in smooth data archival and retrieval.</blockquote><p>If the logging pipeline is setup correctly, you should see a test log file in the bucket after a few minutes. The access logs themselves take a while longer to be sent into the bucket.</p><h3>AWS Elastic search Service</h3><p>We have used Elastic service version(5.1) that was created without VPC in AWS Mumbai Region</p><p>Below guide will help you deploy ES service in AWS</p><p><a href="https://docs.aws.amazon.com/elasticsearch-service/latest/developerguide/es-vpc.html">VPC Support for Amazon Elasticsearch Service Domains - Amazon Elasticsearch Service</a></p><h3>Prepare you local machine</h3><p>We would be building a node app that will be executed as a function in Lambda to parse logs</p><ul><li>Download and Install <a href="https://nodejs.org/en/download/">NodeJS</a></li><li>Clone this git <a href="https://github.com/prithviraju/aws-elb-logs-to-elasticsearch">repo</a></li></ul><p><a href="https://github.com/prithviraju/aws-elb-logs-to-elasticsearch">prithviraju/aws-elb-logs-to-elasticsearch</a></p><ul><li>Go to root folder of the repository and install node dependencies by running</li></ul><pre>npm install</pre><ul><li>Verify that these are installed within the node_modules​ subdirectory</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*WJNXHaqR7c9-7Lu0nyUeoA.png" /></figure><ul><li>Update the index.js​ file with elastic search service endpoint, region and index name</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*5u3QbHKX3DtktZ_UxqXjdA.png" /></figure><ul><li>Create a zip file to package the index.js and the node_modules directory (Example : elb.zip)</li></ul><h3>Setting up Lambda function and IAM Role</h3><p>IAM Role for Lambda</p><p>For creating a s3 Read only access role for lambda to read ELB logs from s3, Go to IAM and click on Role ​and choose Lambda​ under AWS service and and click Next</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*F-VxxwosRKhDHE-sR5wa-g.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*R9QQMJWSUoaZ_MESu0yafg.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*LyZ-gejty1ghNkkTAmPmIg.png" /></figure><blockquote>Note ARN that is create in the above step.</blockquote><p>Creating a Lambda function</p><ul><li>Create a lambda function by choosing <strong>author from scratch</strong> option</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*N6CJmFRIrTRFKw5ZCXrRSw.png" /></figure><ul><li>Upload zip file ( elb.zip ) that was creating on your local machine.</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*hwGIPRjrA1sgljoRWjO4ZQ.png" /></figure><ul><li>In add trigger field choose s3.</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*-IPetRgCnNrNQjzbyludrw.png" /></figure><ul><li>In configuration triggers choose the bucket details as follows.</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*D17ZOJz3vwm-ADldQC38mg.png" /></figure><ul><li>Add trigger</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*fDRsSMmwmt3cGo-8_qNmkg.png" /></figure><ul><li>Save the function</li></ul><blockquote><strong>Note : Create a Lambda function in the same region as S3 bucket</strong></blockquote><h3><strong>Securing Elastic Search and Kibana</strong></h3><p>It is very important to restrict access to Kibana and Elastic search, as the best practice allow only from VPN IP and deny from the world. This can also be protected with a Nginx + access reverse proxy deployment.</p><p>Here we would be restricting access to Elastic search and Kibana to a specific IP. More details here..</p><p><a href="https://aws.amazon.com/blogs/security/how-to-control-access-to-your-amazon-elasticsearch-service-domain/">How to Control Access to Your Amazon Elasticsearch Service Domain | Amazon Web Services</a></p><p>Go to elastic search and choose modify policy option.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*nkfwYLlBnnA6eK7JeN3vlA.png" /></figure><ul><li>In Add or edit the access policy​ add the policy like below</li></ul><pre>{ &quot;Version&quot;: &quot;2012-10-17&quot;, &quot;Statement&quot;: [ { &quot;Sid&quot;: &quot;IP restriction&quot;, &quot;Effect&quot;: &quot;Allow&quot;, &quot;Principal&quot;: { &quot;AWS&quot;: &quot;*&quot; }, &quot;Action&quot;: &quot;es:*&quot;, &quot;Resource&quot;: &quot;arn:aws:es:ap-south-1:XXXXXXXXXXXX:domain/elblogs/*&quot;, &quot;Condition&quot;: { &quot;IpAddress&quot;: { &quot;aws:SourceIp&quot;: [ &quot;XXX.XXX.XXX.XXX&quot; ] } } }, { &quot;Sid&quot;: &quot;Lambda-Role restriction&quot;, &quot;Effect&quot;: &quot;Allow&quot;, &quot;Principal&quot;: { &quot;AWS&quot;: &quot;arn:aws:iam::XXXXXXXXXXXX:role/Lambda-s3-readonly&quot; }, &quot;Action&quot;: &quot;es:*&quot;, &quot;Resource&quot;: &quot;arn:aws:es:ap-south-1:XXXXXXXXXXXX:domain/elblogs/*&quot; } ] }</pre><ul><li>Save it and access Kibana ! Happy logging.</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*E6KlQF9RFqJ67lNIuT1p6g.png" /><figcaption>Sample Kibana Dashboard.</figcaption></figure><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=5df239e05f1c" width="1" height="1" alt="">]]></content:encoded>
        </item>
    </channel>
</rss>