Deploying Elastic Stack (ELK) on Kubernetes with Helm for Centralized Microservice Logging

In modern cloud-native Kubernetes environments, distributed microservices produce millions of log lines across hundreds of ephemeral container pods every hour. When a customer reports a transactional checkout failure or an authentication outage, manually running kubectl logs across dozens of shifting pods is hopelessly slow and ineffective. If a pod crashes and restarts, its local log files vanish forever. In 2026, enterprise Kubernetes clusters require a centralized, immutable logging pipeline.

The industry gold-standard logging architecture remains the Elastic Stack (Elasticsearch, Logstash/Filebeat, and Kibana) deployed via Helm charts. By running Filebeat as a lightweight DaemonSet on every cluster node, streaming structured JSON logs into an Elasticsearch cluster, and visualizing error spikes in Kibana dashboards, SRE teams achieve instant operational visibility. This production engineering guide walks you through the entire Kubernetes deployment.

The Kubernetes Logging Pipeline Architecture

The centralized logging pipeline consists of four coordinated components:

Step 1: Adding the Official Elastic Helm Repository

Install and verify the official Elastic Helm charts:

# Add and update Elastic Helm repo
helm repo add elastic https://helm.elastic.co
helm repo update

# Create dedicated namespace for centralized observability
kubectl create namespace logging

Step 2: Deploying Elasticsearch StatefulSet (`values-elasticsearch.yaml`)

Configure a resilient 3-node master-eligible Elasticsearch cluster with persistent volume claims:

# values-elasticsearch.yaml
clusterName: "enterprise-logs"
nodeGroup: "master"
replicas: 3

# Resource bounds preventing node memory starvation
resources:
  requests:
    cpu: "1000m"
    memory: "2Gi"
  limits:
    cpu: "2000m"
    memory: "4Gi"

# JVM Heap size (always set to exactly 50% of container memory)
esJavaOpts: "-Xmx2g -Xms2g"

volumeClaimTemplate:
  accessModes: ["ReadWriteOnce"]
  storageClassName: "gp3" # AWS NVMe EBS storage class
  resources:
    requests:
      storage: 50Gi

antiAffinity: "hard" # Guarantees each ES pod runs on a distinct physical node

# Deploy Elasticsearch via Helm
helm install elasticsearch elastic/elasticsearch -n logging -f values-elasticsearch.yaml

Step 3: Deploying Filebeat DaemonSet (`values-filebeat.yaml`)

Deploy Filebeat across every Kubernetes node to collect container stdout logs:

# values-filebeat.yaml
daemonset:
  extraEnvs:
    - name: "ELASTICSEARCH_HOST"
      value: "elasticsearch-master"
    - name: "ELASTICSEARCH_PORT"
      value: "9200"

filebeatConfig:
  filebeat.yml: |
    filebeat.inputs:
    - type: container
      paths:
        - /var/log/containers/*.log
      processors:
        - add_kubernetes_metadata:
            host: ${NODE_NAME}
            matchers:
            - logs_path:
                logs_path: "/var/log/containers/"

    output.elasticsearch:
      host: '${NODE_NAME}'
      hosts: '${ELASTICSEARCH_HOST}:${ELASTICSEARCH_PORT}'
      index: "k8s-logs-%{[agent.version]}-%{+yyyy.MM.dd}"

# Deploy Filebeat DaemonSet
helm install filebeat elastic/filebeat -n logging -f values-filebeat.yaml

Step 4: Managing Data Lifecycle with Index Lifecycle Management (ILM)

To prevent Elasticsearch from exhausting persistent disk volumes, configure automated Index Lifecycle Management (ILM) rules via Dev Tools in Kibana:

• Hot Phase (Days 0 – 7): High-speed NVMe write and search access.
• Warm Phase (Days 8 – 30): Replicas reduced to 1; indexes shrink-wrapped for storage efficiency.
• Cold Phase (Days 31 – 90): Indices moved to low-cost magnetic storage.
• Delete Phase (Day 91): Automated deletion of logs older than 90 days.

Conclusion

Centralized logging is a non-negotiable prerequisite for enterprise Kubernetes stability. By deploying the Elastic Stack using structured Helm charts, engineering teams gain real-time visibility across microservices, reduce mean time to resolution (MTTR) by 75%, and maintain complete regulatory compliance.