Kubernetes Logging Monitoring and Alerting using EFK Stack

Logging, monitoring, and alerting are critical components of any system, particularly in software development.

We add logging statements in the application to record events, actions and other information related to the application which helps in troubleshooting later in case of any issues. Commonly, logs are stored in text files, but when the application is receiving heavy traffic, checking logs from the text files seems to be a hectic task. Running kubectl logs also isn’t practical.

To address this, we need an efficient logging, monitoring and alerting system. There are multiple choices available for the same, especially if the application is running in cloud. AWS X-Ray and Azure Application Insights are powerful tools to address the issue, but for a more cost-effective solution, EFK Stack could be used.

Logging with EFK

EFK stands for Elasticsearch, Fluent Bit and Kibana.

Elasticsearch is a distributed, RESTful search and analytics engine designed for horizontal scalability, reliability, and real-time search capabilities.

Fluent Bit is an open-source and lightweight log processor and forwarder designed to collect, filter, and forward logs and metrics data

Kibana is an open-source data visualization and exploration tool specifically designed for Elasticsearch

To start, let’s create a namespace for our EFK setup. You can even deploy in the same namespace where your application is running, but it’s a good practice to create a separate namespace for EFK

kubectl create namespace efk-ns

To verify the creation of namespace, we can give the following command

kubectl get namespaces

You should be able to see the efk-ns namespace in the list

We will create a secret for elasticsearch password, which will be used later

apiVersion: v1
kind: Secret
metadata:
  name: kibana-password
  namespace: efk-ns
type: Opaque
data:
  password: yourPasswordHere

Save the file as kibana-password-secret.yaml and apply with the following command

kubectl apply -f kibana-password-secret.yaml

Now let’s create an headless service for elastic search

kind: Service
apiVersion: v1
metadata:
  name: elasticsearch
  namespace: efk-ns
  labels:
    app: elasticsearch
spec:
  selector:
    app: elasticsearch
  clusterIP: None
  ports:
    - port: 9200
      name: rest
    - port: 9300
      name: inter-node

Let’s save the file as elastic-svc.yaml. After saving, we can enter the following command to apply the file

kubectl apply -f elastic-svc.yaml

We need a PersistentVolume and PersistentVolumeClaim where we will store our logs. This can be done in any available storage space. For this example, we are using Azure Storage Account with 5 TB storage space. For most applications with moderate traffic, around 30–50GB storage per day would be required to store the logs. We can set a Log Purge Policy as well in Kibana, which we will explain in the later stage

apiVersion: v1
kind: PersistentVolume
metadata:
  name: efk-pvc
  namespace: efk-ns
spec:
  capacity:
    storage: 5Ti
  accessModes:
    - ReadWriteMany
  azureFile:
    secretName: azure-file-secret
    shareName: efk
    readOnly: false
    secretNamespace: efk-ns
  mountOptions:
    - dir_mode=0777
    - file_mode=0777
    - uid=1000
    - gid=1000

Save the file as efk-pv.yaml and apply using following command

kubectl apply -f efk-pv.yaml

Now we will create PVC

apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: efk-pvc
  namespace: efk-ns
spec:
  accessModes:
    - ReadWriteMany
  storageClassName: ""
  resources:
    requests:
      storage: 5Ti

Save the file as efk-pvc.yaml and apply using following command

kubectl apply -f efk-pvc.yaml

Now let’s create a StatefulSet for elasticsearch. We are using StatefulSet pods for elasticsearch so that the data will not be lost during deployments.

apiVersion: apps/v1
kind: StatefulSet
metadata:
  name: es-cluster
  namespace: efk-ns
spec:
  serviceName: elasticsearch
  replicas: 3
  selector:
    matchLabels:
      app: elasticsearch
  template:
    metadata:
      labels:
        app: elasticsearch
    spec:
      containers:
      - name: elasticsearch
        image: docker.elastic.co/elasticsearch/elasticsearch:7.2.0
        resources:
          limits:
            cpu: 2000m
            memory: 6Gi
          requests:
            cpu: 1000m
            memory: 2Gi
        ports:
        - containerPort: 9200
          name: rest
          protocol: TCP
        - containerPort: 9300
          name: inter-node
          protocol: TCP
        volumeMounts:
        - name: data
          mountPath: /efk
        env:
          - name: cluster.name
            value: k8s-logs
          - name: node.name
            valueFrom:
              fieldRef:
                fieldPath: metadata.name
          - name: discovery.seed_hosts
            value: "es-cluster-0.elasticsearch,es-cluster-1.elasticsearch,es-cluster-2.elasticsearch"
          - name: cluster.initial_master_nodes
            value: "es-cluster-0,es-cluster-1,es-cluster-2"
          - name: network.host
            value: "0.0.0.0"
          - name: xpack.security.enabled
            value: "false"
          - name: xpack.monitoring.collection.enabled
            value: "true"   
          - name: ES_JAVA_OPTS
            value: "-Xms5g -Xmx5g" 
          - name: path.data
            value: /efk
          - name: node.max_local_storage_nodes
            value: "3"
          - name: ELASTIC_PASSWORD
            valueFrom:
              secretKeyRef:
                name: kibana-password
                key: password     
      initContainers:
      - name: fix-permissions
        image: busybox
        command: ["sh", "-c", "chown -R 1000:1000 /efk"]
        securityContext:
          privileged: true
        volumeMounts:
        - name: data
          mountPath: /efk
      - name: increase-vm-max-map
        image: busybox
        command: ["sysctl", "-w", "vm.max_map_count=262144"]
        securityContext:
          privileged: true
      - name: increase-fd-ulimit
        image: busybox
        command: ["sh", "-c", "ulimit -n 65536"]
        securityContext:
          privileged: true
      volumes:
       - name: data
         persistentVolumeClaim:
           claimName: efk-pvc

Here, we can set the CPU and memory limits as well. Java Heap memory is set using the ES_JAVA_OPTS environment variable. For setting the heap , a good rule-of-thumb is to ensure you keep the number of shards per node below 20 per GB heap it has configured. A node with a 30GB heap should therefore have a maximum of 600 shards, but the further below this limit you can keep it the better.

Lets save the above file as elastic-sts.yaml and apply the same with the following command

kubectl apply -f elastic-sts.yaml

Now, to check if the pods are up, we can use the following command

kubectl get pods -n efk-ns

Now lets move to Kibana. First we need to create a service for Kibana

apiVersion: v1
kind: Service
metadata:
  name: kibana
  namespace: efk-ns
  labels:
    app: kibana
spec:
  ports:
  - port: 5601
  selector:
    app: kibana

Save as kibana-service.yaml and apply using following command

kubectl apply -f kibana-service.yaml

Now we need to create deployment for Kibana

apiVersion: apps/v1
kind: Deployment
metadata:
  name: kibana
  namespace: efk-ns
  labels:
    app: kibana
spec:
  replicas: 1
  selector:
    matchLabels:
      app: kibana
  template:
    metadata:
      labels:
        app: kibana
    spec:
      containers:
      - name: kibana
        image: docker.elastic.co/kibana/kibana:7.2.0
        resources:
          limits:
            cpu: 1000m
            memory: 1Gi
          requests:
            cpu: 700m
            memory: 1Gi
        env:
          - name: ELASTICSEARCH_URL
            value: http://elasticsearch:9200
          - name: ELASTICSEARCH_USERNAME
            value: elastic
          - name: ELASTICSEARCH_PASSWORD
            valueFrom:
              secretKeyRef:
                name: kibana-password
                key: password
        ports:
        - containerPort: 5601
          name: kibana
          protocol: TCP

Save as kibana-deployment.yaml and apply using following command

kubectl apply -f kibana-deployment.yaml

Now let’s move to FluentBit. First we need to create a ServiceAccount for FluentBit

apiVersion: v1
kind: ServiceAccount
metadata:
  name: fluent-bit-sa
  namespace: efk-ns
  labels:
    app: fluent-bit-sa

Save as fluentbit-sa.yaml and apply using following command

kubectl apply -f fluentbit-sa.yaml

Now we need to create a ClusterRole for FluentBit

apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  name: fluent-bit-role
  labels:
    app: fluent-bit-role
rules:
  - nonResourceURLs:
      - /metrics
    verbs:
      - get
  - apiGroups: [""]
    resources:
      - namespaces
      - pods
      - pods/logs
    verbs: ["get", "list", "watch"]

Save as fluentbit-clusterrole.yaml and apply using following command

kubectl apply -f fluentbit-clusterrole.yaml

Now we need to create ClusterRoleBinding for FluentBit

kind: ClusterRoleBinding
apiVersion: rbac.authorization.k8s.io/v1
metadata:
  name: fluent-bit-rb
roleRef:
  kind: ClusterRole
  name: fluent-bit-role
  apiGroup: rbac.authorization.k8s.io
subjects:
- kind: ServiceAccount
  name: fluent-bit-sa
  namespace: efk-ns

Save as fluentbit-rb.yaml and apply using following command

kubectl apply -f fluentbit-rb.yaml

Now let’s create the config map for FluentBit

apiVersion: v1
kind: ConfigMap
metadata:
  name: fluent-bit-config
  namespace: efk-ns
  labels:
    k8s-app: f-bit-pod
data:
  # Configuration files: server, input, filters and output
  # ======================================================
  fluent-bit.conf: |
    [SERVICE]
        Flush         1
        Log_Level     info
        Daemon        off
        Parsers_File  parsers.conf
        HTTP_Server   On
        HTTP_Listen   0.0.0.0
        HTTP_Port     2020
    @INCLUDE input-kubernetes.conf
    @INCLUDE filter-kubernetes.conf
    @INCLUDE output-elasticsearch.conf

  input-kubernetes.conf: |
    [INPUT]
        Name              tail
        Tag               kube.*
        Path              /var/log/containers/*.log
        Parser            docker
        DB                /var/log/flb_kube.db
        Mem_Buf_Limit     5MB
        Skip_Long_Lines   On
        Refresh_Interval  10
  filter-kubernetes.conf: |
    [FILTER]
        Name                kubernetes
        Match               kube.*
        Kube_URL            https://kubernetes.default.svc:443
        Kube_CA_File        /var/run/secrets/kubernetes.io/serviceaccount/ca.crt
        Kube_Token_File     /var/run/secrets/kubernetes.io/serviceaccount/token
        Kube_Tag_Prefix     kube.var.log.containers.*
        Merge_Log           On
        Merge_Log_Key       log_processed
        K8S-Logging.Parser  On
        K8S-Logging.Exclude Of

  output-elasticsearch.conf: |
    [OUTPUT]
        Name            es
        Match           *
        Host            ${FLUENT_ELASTICSEARCH_HOST}
        Port            ${FLUENT_ELASTICSEARCH_PORT}
        HTTP_User       ${FLUENT_ELASTICSEARCH_USER}
        HTTP_Passwd     ${FLUENT_ELASTICSEARCH_PASSWORD}
        Logstash_Format On
        Replace_Dots    On
        Retry_Limit     False

  parsers.conf: |
    [PARSER]
        Name   apache
        Format regex
        Regex  ^(?<host>[^ ]*) [^ ]* (?<user>[^ ]*) \[(?<time>[^\]]*)\] "(?<method>\S+)(?: +(?<path>[^\"]*?)(?: +\S*)?)?" (?<code>[^ ]*) (?<size>[^ ]*)(?: "(?<referer>[^\"]*)" "(?<agent>[^\"]*)")?$
        Time_Key time
        Time_Format %d/%b/%Y:%H:%M:%S %z
    [PARSER]
        Name   apache2
        Format regex
        Regex  ^(?<host>[^ ]*) [^ ]* (?<user>[^ ]*) \[(?<time>[^\]]*)\] "(?<method>\S+)(?: +(?<path>[^ ]*) +\S*)?" (?<code>[^ ]*) (?<size>[^ ]*)(?: "(?<referer>[^\"]*)" "(?<agent>[^\"]*)")?$
        Time_Key time
        Time_Format %d/%b/%Y:%H:%M:%S %z
    [PARSER]
        Name   apache_error
        Format regex
        Regex  ^\[[^ ]* (?<time>[^\]]*)\] \[(?<level>[^\]]*)\](?: \[pid (?<pid>[^\]]*)\])?( \[client (?<client>[^\]]*)\])? (?<message>.*)$
    [PARSER]
        Name   nginx
        Format regex
        Regex ^(?<remote>[^ ]*) (?<host>[^ ]*) (?<user>[^ ]*) \[(?<time>[^\]]*)\] "(?<method>\S+)(?: +(?<path>[^\"]*?)(?: +\S*)?)?" (?<code>[^ ]*) (?<size>[^ ]*)(?: "(?<referer>[^\"]*)" "(?<agent>[^\"]*)")?$
        Time_Key time
        Time_Format %d/%b/%Y:%H:%M:%S %z
    [PARSER]
        Name   json
        Format json
        Time_Key time
        Time_Format %d/%b/%Y:%H:%M:%S %z
    [PARSER]
        Name        docker
        Format      json
        Time_Key    time
        Time_Format %Y-%m-%dT%H:%M:%S.%L
        Time_Keep   On
    [PARSER]
        Name        syslog
        Format      regex
        Regex       ^\<(?<pri>[0-9]+)\>(?<time>[^ ]* {1,2}[^ ]* [^ ]*) (?<host>[^ ]*) (?<ident>[a-zA-Z0-9_\/\.\-]*)(?:\[(?<pid>[0-9]+)\])?(?:[^\:]*\:)? *(?<message>.*)$
        Time_Key    time
        Time_Format %b %d %H:%M:%S

Here, Kube_URL should be the URL of your kubernetes instance

Get Junaid’s stories in your inbox

Join Medium for free to get updates from this writer.

Remember me for faster sign in

Save the file as fluent-bit-configmap.yaml and apply using following command

kubectl apply -f fluent-bit-configmap.yaml

Now let’s create a Daemonset for Fluent Bit. This will collect logs from the nodes and send them to Elasticsearch

apiVersion: apps/v1
kind: DaemonSet
metadata:
  name: f-bit-pod
  namespace: efk-ns
  labels:
    k8s-app: fluent-bit-logging
    version: v1
    kubernetes.io/cluster-service: "true"
spec:
  selector:
    matchLabels:
      k8s-app: fluent-bit-logging
  template:
    metadata:
      labels:
        k8s-app: fluent-bit-logging
        version: v1
        kubernetes.io/cluster-service: "true"
      annotations:
        prometheus.io/scrape: "true"
        prometheus.io/port: "2020"
        prometheus.io/path: /api/v1/metrics/prometheus
    spec:
      containers:
      - name: fluent-bit
        image: fluent/fluent-bit:1.6.0
        imagePullPolicy: Always
        ports:
        - containerPort: 2020
        env:
        - name: FLUENT_ELASTICSEARCH_HOST
          value: "elasticsearch"
        - name: FLUENT_ELASTICSEARCH_PORT
          value: "9200"
        - name: FLUENT_ELASTICSEARCH_USER
          value: "elastic"
        - name: FLUENT_ELASTICSEARCH_PASSWORD
          valueFrom:
            secretKeyRef:
              name: kibana-password
              key: password
        volumeMounts:
        - name: varlog
          mountPath: /var/log
        - name: varlibdockercontainers
          mountPath: /var/lib/docker/containers
          readOnly: true
        - name: fluent-bit-config
          mountPath: /fluent-bit/etc/
      terminationGracePeriodSeconds: 10
      volumes:
      - name: varlog
        hostPath:
          path: /var/log
      - name: varlibdockercontainers
        hostPath:
          path: /var/lib/docker/containers
      - name: fluent-bit-config
        configMap:
          name: fluent-bit-config
      serviceAccountName: fluent-bit-sa
      tolerations:
      - key: node-role.kubernetes.io/master
        operator: Exists
        effect: NoSchedule
      - operator: "Exists"
        effect: "NoExecute"
      - operator: "Exists"
        effect: "NoSchedule"

Save the file as fluentbit-ds.yaml and apply using following command

kubectl apply -f fluentbit-ds.yaml

Verify that all the pods are running by running the following command

kubectl get pods -n efk-ns -o wide

Now if there are no errors, FluentBit will be forwarding the logs to Elasticsearch and we will be able to see the same through Kibana UI. You will be able to access the Kibana UI by the IP address of the pod of Kibana, with port number as 5601.

The UI will look like this

We will be able to search for logs by inputting the keyword in the search field

These are the options available in the Kibana UI. For monitoring the EFK stack, we can go to the Stack Monitoring field. If everything is working fine, the status will be Green.

Normally, logs of each day will be a shard (eg. logstash-2024.04.19)

We can have the Log Purge Policy from the Index Lifecycle Policies under Management section

We can specify the number of days after which the logs should be deleted

Alerting with EFK

An easy way to achieve alerts with EFK is to have a filter in Fluent Bit config map and output to our preferred mode of receiving the alert. For this example, I am using a slack channel to receive the alert.

We would mainly require alert in case of resource exhaustion or any specific type of error message in the log.

Let’s take an example of 2 microservices, and one is sending a heartbeat request to another every 2 minutes. This ensures that there is connectivity between these 2 services. In case if there is any error, we would be printing the error log which indicates that connectivity is lost.

In payment systems and applications, this is critical and we would require immediate alert for the same. To achieve this, we can put an alert based on the error message.

    [FILTER]
        Name         rewrite_tag
        Match        kube.*
        Rule         $log Error\sin\sheartbeat failed_destination false

The above filter will catch the text “Error in heartbeat” and sends it to failed_destination

    [OUTPUT]
        name     slack
        match    failed_destination
        webhook  https://hooks.slack.com/services/yourSlackWebhookUrl

Any match in failed_destination will be sent to the webhook we specify. With the above changes, the modified Fluent-Bit Config map is given below

apiVersion: v1
kind: ConfigMap
metadata:
  name: fluent-bit-config
  namespace: efk-ns
  labels:
    k8s-app: f-bit-pod
data:
  # Configuration files: server, input, filters and output
  # ======================================================
  fluent-bit.conf: |
    [SERVICE]
        Flush         1
        Log_Level     info
        Daemon        off
        Parsers_File  parsers.conf
        HTTP_Server   On
        HTTP_Listen   0.0.0.0
        HTTP_Port     2020
    @INCLUDE input-kubernetes.conf
    @INCLUDE filter-kubernetes.conf
    @INCLUDE output-elasticsearch.conf
    @INCLUDE output-slack.conf
  input-kubernetes.conf: |
    [INPUT]
        Name              tail
        Tag               kube.*
        Path              /var/log/containers/*.log
        Parser            docker
        DB                /var/log/flb_kube.db
        Mem_Buf_Limit     5MB
        Skip_Long_Lines   On
        Refresh_Interval  10
  filter-kubernetes.conf: |
    [FILTER]
        Name                kubernetes
        Match               kube.*
        Kube_URL            https://kubernetes.default.svc:443
        Kube_CA_File        /var/run/secrets/kubernetes.io/serviceaccount/ca.crt
        Kube_Token_File     /var/run/secrets/kubernetes.io/serviceaccount/token
        Kube_Tag_Prefix     kube.var.log.containers.*
        Merge_Log           On
        Merge_Log_Key       log_processed
        K8S-Logging.Parser  On
        K8S-Logging.Exclude Off
    [FILTER]
        Name         rewrite_tag
        Match        kube.*
        Rule         $log Error\sin\sheartbeat failed_destination false

  output-elasticsearch.conf: |
    [OUTPUT]
        Name            es
        Match           *
        Host            ${FLUENT_ELASTICSEARCH_HOST}
        Port            ${FLUENT_ELASTICSEARCH_PORT}
        HTTP_User       ${FLUENT_ELASTICSEARCH_USER}
        HTTP_Passwd     ${FLUENT_ELASTICSEARCH_PASSWORD}
        Logstash_Format On
        Replace_Dots    On
        Retry_Limit     False
  output-slack.conf: |
    [OUTPUT]
        name     slack
        match    failed_destination
        webhook  https://hooks.slack.com/services/yourSlackWebhookUrl
  parsers.conf: |
    [PARSER]
        Name   apache
        Format regex
        Regex  ^(?<host>[^ ]*) [^ ]* (?<user>[^ ]*) \[(?<time>[^\]]*)\] "(?<method>\S+)(?: +(?<path>[^\"]*?)(?: +\S*)?)?" (?<code>[^ ]*) (?<size>[^ ]*)(?: "(?<referer>[^\"]*)" "(?<agent>[^\"]*)")?$
        Time_Key time
        Time_Format %d/%b/%Y:%H:%M:%S %z
    [PARSER]
        Name   apache2
        Format regex
        Regex  ^(?<host>[^ ]*) [^ ]* (?<user>[^ ]*) \[(?<time>[^\]]*)\] "(?<method>\S+)(?: +(?<path>[^ ]*) +\S*)?" (?<code>[^ ]*) (?<size>[^ ]*)(?: "(?<referer>[^\"]*)" "(?<agent>[^\"]*)")?$
        Time_Key time
        Time_Format %d/%b/%Y:%H:%M:%S %z
    [PARSER]
        Name   apache_error
        Format regex
        Regex  ^\[[^ ]* (?<time>[^\]]*)\] \[(?<level>[^\]]*)\](?: \[pid (?<pid>[^\]]*)\])?( \[client (?<client>[^\]]*)\])? (?<message>.*)$
    [PARSER]
        Name   nginx
        Format regex
        Regex ^(?<remote>[^ ]*) (?<host>[^ ]*) (?<user>[^ ]*) \[(?<time>[^\]]*)\] "(?<method>\S+)(?: +(?<path>[^\"]*?)(?: +\S*)?)?" (?<code>[^ ]*) (?<size>[^ ]*)(?: "(?<referer>[^\"]*)" "(?<agent>[^\"]*)")?$
        Time_Key time
        Time_Format %d/%b/%Y:%H:%M:%S %z
    [PARSER]
        Name   json
        Format json
        Time_Key time
        Time_Format %d/%b/%Y:%H:%M:%S %z
    [PARSER]
        Name        docker
        Format      json
        Time_Key    time
        Time_Format %Y-%m-%dT%H:%M:%S.%L
        Time_Keep   On
    [PARSER]
        Name        syslog
        Format      regex
        Regex       ^\<(?<pri>[0-9]+)\>(?<time>[^ ]* {1,2}[^ ]* [^ ]*) (?<host>[^ ]*) (?<ident>[a-zA-Z0-9_\/\.\-]*)(?:\[(?<pid>[0-9]+)\])?(?:[^\:]*\:)? *(?<message>.*)$
        Time_Key    time
        Time_Format %b %d %H:%M:%S

The alert would come in the Slack channel as follows

Monitoring with EFK

Apart from Logging and alerting, monitoring is also important for the smooth functioning of the application. For monitoring, Elastic APM (Application Performance Monitoring) can be used. Elastic APM is designed to monitor the performance of applications and services in real-time, providing insights into their behavior, identifying bottlenecks, and helping developers optimize their code and infrastructure.

For APM, we need to have an APM Server and APM Agents. The detailed instruction on how to do this is visible in the Kibana UI as well.

Based on our OS, we can install the APM Server. For this example, we are using Ubuntu (DEB)

Download and unpack APM Server

curl -L -O https://artifacts.elastic.co/downloads/apm-server/apm-server-7.2.0-amd64.deb
sudo dpkg -i apm-server-7.2.0-amd64.deb

Specify the credentials in apm-server.yaml (Mostly in /etc/apm-server)

output.elasticsearch:
    hosts: ["<es_url>"]
    username: <username>
    password: <password>

Start the server using the following command

service apm-server start

For APM Agent, based on our programming language, we can install the package. The detail description on how to do this is also given in the Kibana UI

Once this is done, we will be able to monitor our application from the APM section in Kibana

We will be able to see the Avg Response time, 95th percentile and 99th percentile of requests, transactions per minute, errors etc using this

We can also view the message details of each incoming API hits.

With this, we will be able to meet our requirements of Logging, monitoring and alerting in Kubernetes with EFK Stack.

Hope this helps!