Set up EFK (Elasticsearch, Fluentd and Kibana) on kubernetes cluster
In the world of micro‑services, logs are more than just lines of text – they’re the heartbeat of your applications. Without a central place to collect and analyse them, chasing down issues becomes a game of guesswork. That’s where the EFK stack comes in.
EFK combines Fluentd, a lightweight data collector that tails your container logs on every node; Elasticsearch, a distributed search engine that indexes those logs and makes them searchable; and Kibana, the slick web interface for sifting through, visualising and making sense of the noise.
In this lab you’ll learn how to stand up an EFK stack from scratch, wire it into your Kubernetes cluster and start turning raw log files into actionable insights.
Steps :
1. Create namespace
2. Create Elasticsearch resources (Statefulset and Service)
3. Create fluentd resources (DaemonSet , ServiceAccount, ClusterRole, ClusterRoleBinding and ConfigMap)
4. Create Kibana resources (Deployment and Service)
5. Access to EFK via browser
6. Configure logstash in the index pattern of kibana dashboard
7. Create a pod that generates some logs
8. Check and filter the logs from the kibana dashboard
- Create namespace
2. Create Elasticsearch resources (Statefulset and Service)
apiVersion: apps/v1
kind: StatefulSet
metadata:
name: es-cluster
namespace: kube-logging
spec:
serviceName: elasticsearch
replicas: 3
selector:
matchLabels:
app: elasticsearch
template:
metadata:
labels:
app: elasticsearch
spec:
containers:
- name: elasticsearch
image: docker.elastic.co/elasticsearch/elasticsearch:7.2.0
resources:
limits:
cpu: 1000m
requests:
cpu: 100m
ports:
- containerPort: 9200
name: rest
protocol: TCP
- containerPort: 9300
name: inter-node
protocol: TCP
volumeMounts:
- name: data
mountPath: /usr/share/elasticsearch/data
env:
- name: cluster.name
value: k8s-logs
- name: node.name
valueFrom:
fieldRef:
fieldPath: metadata.name
- name: discovery.seed_hosts
value: "es-cluster-0.elasticsearch,es-cluster-1.elasticsearch,es-cluster-2.elasticsearch"
- name: cluster.initial_master_nodes
value: "es-cluster-0,es-cluster-1,es-cluster-2"
- name: ES_JAVA_OPTS
value: "-Xms512m -Xmx512m"
initContainers:
- name: fix-permissions
image: busybox
command: ["sh", "-c", "chown -R 1000:1000 /usr/share/elasticsearch/data"]
securityContext:
privileged: true
volumeMounts:
- name: data
mountPath: /usr/share/elasticsearch/data
- name: increase-vm-max-map
image: busybox
command: ["sysctl", "-w", "vm.max_map_count=262144"]
securityContext:
privileged: true
- name: increase-fd-ulimit
image: busybox
command: ["sh", "-c", "ulimit -n 65536"]
securityContext:
privileged: true
volumeClaimTemplates:
- metadata:
name: data
labels:
app: elasticsearch
spec:
accessModes: [ "ReadWriteOnce" ]
storageClassName: do-block-storage
resources:
requests:
storage: 20Gi
---
kind: Service
apiVersion: v1
metadata:
name: elasticsearch
namespace: kube-logging
labels:
app: elasticsearch
spec:
selector:
app: elasticsearch
clusterIP: None
ports:
- port: 9200
name: rest
- port: 9300
name: inter-node
3. Create fluentd resources
a. DaemonSet , ServiceAccount, ClusterRole, ClusterRoleBinding
---
apiVersion: v1
kind: ServiceAccount
metadata:
name: fluentd
namespace: kube-logging
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
name: fluentd
namespace: kube-logging
rules:
- apiGroups:
- ""
resources:
- pods
- namespaces
verbs:
- get
- list
- watch
---
kind: ClusterRoleBinding
apiVersion: rbac.authorization.k8s.io/v1
metadata:
name: fluentd
namespace: default
roleRef:
kind: ClusterRole
name: fluentd
apiGroup: rbac.authorization.k8s.io
subjects:
- kind: ServiceAccount
name: fluentd
namespace: kube-logging
---
apiVersion: apps/v1
kind: DaemonSet
metadata:
name: fluentd
namespace: kube-logging
labels:
k8s-app: fluentd-logging
version: v1
spec:
selector:
matchLabels:
k8s-app: fluentd-logging
version: v1
template:
metadata:
labels:
k8s-app: fluentd-logging
version: v1
spec:
securityContext:
runAsUser: 0
serviceAccountName: fluentd
tolerations:
- key: node-role.kubernetes.io/master
effect: NoSchedule
containers:
- name: fluentd
image: fluent/fluentd-kubernetes-daemonset:v1-debian-elasticsearch
env:
- name: FLUENT_ELASTICSEARCH_HOST
value: "elasticsearch"
- name: FLUENT_ELASTICSEARCH_PORT
value: "9200"
resources:
limits:
memory: 200Mi
requests:
cpu: 100m
memory: 200Mi
volumeMounts:
- name: fluentd-config
mountPath: /fluentd/etc
- name: varlog
mountPath: /var/log
- name: varlibdockercontainers
mountPath: /var/lib/docker/containers
readOnly: true
# declare the volumes here – names must match the volumeMounts above
volumes:
- name: fluentd-config
configMap:
name: fluentd-config # ConfigMap must exist in kube-logging
- name: varlog
hostPath:
path: /var/log
type: DirectoryOrCreate
- name: varlibdockercontainers
hostPath:
path: /var/lib/docker/containers
type: DirectoryOrCreateb. ConfigMap
apiVersion: v1
kind: ConfigMap
metadata:
name: fluentd-config
namespace: kube-logging
data:
fluent.conf: |-
################################################################
# This source gets all logs from local docker host
@include pods-kind-fluent.conf
@include elastic-fluent.conf
pods-kind-fluent.conf: |-
<source>
@type tail
read_from_head true
tag kubernetes.*
path /var/log/containers/*.log
pos_file /var/log/fluentd-containers.log.pos
exclude_path ["/var/log/containers/fluent*"]
<parse>
@type regexp
expression ^(?<time>\d{4}-\d{2}-\d{2}T\d{2}:\d{2}:\d{2}.[^Z]*Z)\s(?<stream>[^\s]+)\s(?<character>[^\s])\s(?<message>.*)$
</parse>
</source>
<filter kubernetes.**>
@type kubernetes_metadata
@id filter_kube_metadata
kubernetes_url "#{ENV['FLUENT_FILTER_KUBERNETES_URL'] || 'https://' + ENV.fetch('KUBERNETES_SERVICE_HOST') + ':' + ENV.fetch('KUBERNETES_SERVICE_PORT') + '/api'}"
verify_ssl "#{ENV['KUBERNETES_VERIFY_SSL'] || true}"
ca_file "#{ENV['KUBERNETES_CA_FILE']}"
skip_labels "#{ENV['FLUENT_KUBERNETES_METADATA_SKIP_LABELS'] || 'false'}"
skip_container_metadata "#{ENV['FLUENT_KUBERNETES_METADATA_SKIP_CONTAINER_METADATA'] || 'false'}"
skip_master_url "#{ENV['FLUENT_KUBERNETES_METADATA_SKIP_MASTER_URL'] || 'false'}"
skip_namespace_metadata "#{ENV['FLUENT_KUBERNETES_METADATA_SKIP_NAMESPACE_METADATA'] || 'false'}"
</filter>
elastic-fluent.conf: |-
<match **>
@type elasticsearch
host "#{ENV['FLUENT_ELASTICSEARCH_HOST'] || 'elasticsearch'}"
port "#{ENV['FLUENT_ELASTICSEARCH_PORT'] || '9200'}"
logstash_format true # create daily indices like logstash‑2025.08.28
logstash_prefix logstash # prefix for the index name
include_timestamp true
suppress_type_name true # don’t send a custom type_name to ES 7+
log_es_400_reason true # capture the reason if ES rejects a document:contentReference[oaicite:2]{index=2}
</match>
4. Create Kibana resources (Deployment and Service)
apiVersion: v1
kind: Service
metadata:
name: kibana
namespace: kube-logging
labels:
app: kibana
spec:
ports:
- port: 5601
selector:
app: kibana
---
apiVersion: apps/v1
kind: Deployment
metadata:
name: kibana
namespace: kube-logging
labels:
app: kibana
spec:
replicas: 1
selector:
matchLabels:
app: kibana
template:
metadata:
labels:
app: kibana
spec:
containers:
- name: kibana
image: docker.elastic.co/kibana/kibana:7.2.0
resources:
limits:
cpu: 1000m
requests:
cpu: 100m
env:
- name: ELASTICSEARCH_URL
value: http://localhost:9200
ports:
- containerPort: 5601
5. Access to EFK via browser
Get Ennebet Mohammed’s stories in your inbox
Join Medium for free to get updates from this writer.
Check the resources creation first then forward the elasticsearch and kibana pods
6. Configure logstash in the index pattern of kibana dashboard
Go to Management > Index pattern under Kibana section and search for logstash-* then click on Next step
Add timestamp and create the index pattern
7. Create a pod that generates some logs
apiVersion: v1
kind: Pod
metadata:
name: medcounter
namespace: med
spec:
containers:
- name: medcounter
image: busybox
args: [/bin/sh, -c,
'i=0; while true; do echo "$i: Hello Mohammed here is the date $(date)"; i=$((i+1)); sleep 1; done']> Kubectl create ns med
> Kubectl create –f medcounterPod.yaml
8. Check and filter the logs from the kibana dashboard
In Kibana go to Discover , select the index pattern logstash-* and put in the filters section the folowing : kubernetes.namespace_name : med and kubernetes.container_name : medcounter
then refresh
At the end note that EFK is one of the robuste and popular open-source toolkit for centralising and analysings logs. Fluentd acts as a unified data collector running on each node to capture container and application logs. Those logs are then shipped to Elasticsearch, a distributed search and analytics engine that indexes the data for fast querying. Kibana sits on top as the visual layer, giving you dashboards and search tools to explore your logs in real time.Together, these three components provide a scalable way to turn raw log streams into actionable insights across a complex, distributed system.
