Elasticsearch in Kubernetes: HelmRelease, ingest pipelines, and bootstrap

Published: 2026-03-14

Elasticsearch runs on every cluster. It stores APISIX access logs and application logs from Vector. Getting it right requires more than just a HelmRelease — there's a bootstrap script that initializes ingest pipelines, index templates, and sets replica counts correctly for a single-node setup.

The HelmRelease

Elasticsearch is deployed from the charts bundled in the git repo itself — not from an external Helm registry:

yamlapiVersion: helm.toolkit.fluxcd.io/v2
kind: HelmRelease
metadata:
  name: elasticsearch
spec:
  interval: 1h
  chart:
    spec:
      chart: ./charts/elastic/elasticsearch
      version: "7.17.3"
      sourceRef:
        kind: GitRepository
        name: flux-system
        namespace: flux-system
  install:
    remediation:
      retries: -1
  upgrade:
    remediation:
      retries: -1

retries: -1 means Flux will keep retrying indefinitely on failure. This is intentional for stateful services — a transient PVC provisioning delay shouldn't permanently mark the release as failed.

Per-environment patches

The base HelmRelease contains no hostname or storage config. Patches add environment-specific values:

yaml# projects/dev/kustomization/hub/patches/elasticsearch.yaml
apiVersion: helm.toolkit.fluxcd.io/v2
kind: HelmRelease
metadata:
  name: elasticsearch
  namespace: dev
spec:
  values:
    master:
      persistence:
        storageClass: local-path
      resources:
        requests:
          memory: "4Gi"
          cpu: "500m"
        limits:
          memory: "6Gi"
    esConfig:
      elasticsearch.yml: |
        xpack.security.enabled: true
        xpack.monitoring.enabled: false
    ingress:
      hosts:
        - host: elasticsearch.dev.example.com
          paths:
            - path: /

YC clusters use yc-network-ssd instead of local-path.

Bootstrap script

After first install, Elasticsearch is a blank slate. The bootstrap script creates the necessary configuration:

bashES_USER=elastic ES_PASS='<password>' ./scripts/bootstrap-elasticsearch.sh

It runs against all clusters and does:

1. Ingest pipeline for APISIX logs:

json{
  "description": "Parse APISIX access log JSON",
  "processors": [
    { "date": { "field": "timestamp", "formats": ["UNIX_MS"], "target_field": "@timestamp" } },
    { "remove": { "field": "timestamp" } },
    { "geoip": { "field": "remote_addr", "ignore_missing": true } }
  ]
}

2. Index template for APISIX:

json{
  "index_patterns": ["apisix*"],
  "settings": {
    "number_of_replicas": 0,
    "default_pipeline": "apisix"
  }
}

3. Index templates for application logs:

json{
  "index_patterns": ["vector*", "app*"],
  "settings": { "number_of_replicas": 0 }
}

number_of_replicas: 0 is essential for single-node Elasticsearch. With replicas > 0, indices stay yellow forever.

4. Patch existing indices:

bashcurl -X PUT "http://${ES_HOST}:9200/vector*/_settings" \
  -H 'Content-Type: application/json' \
  -d '{"index": {"number_of_replicas": 0}}'

This handles the case where Vector created indices before the template was applied.

Checking cluster health

bashkubectl port-forward -n logging svc/elasticsearch 9200:9200

# Cluster health
curl -u elastic:$ES_PASS http://localhost:9200/_cluster/health?pretty

# Index list with status
curl -u elastic:$ES_PASS http://localhost:9200/_cat/indices?v

# Check all indices have 0 replicas (single node should be all green)
curl -u elastic:$ES_PASS http://localhost:9200/_cat/indices?v&h=index,health,rep

All indices should be green with 0 replicas on a single-node cluster.

Index lifecycle

APISIX logs write to apisix (single index with the ingest pipeline). Application logs write to vector-YYYY-MM-DD (daily rolling indices).

No ILM is configured — old indices are deleted when disk fills up. If you want automation:

bash# Delete indices older than 30 days
curl -X DELETE "http://localhost:9200/vector-$(date -d '30 days ago' +%Y.%m.%d)"

Or configure a CronJob to run this periodically.

Kibana

Kibana runs only on the infra cluster and connects to the infra Elasticsearch instance. It provides the APM UI, Discover, and dashboards for access log analysis.

For spoke clusters, raw Elasticsearch queries via port-forward or the Elasticsearch Exporter + Grafana dashboard are sufficient.

Elasticsearch Exporter

The Prometheus Elasticsearch exporter scrapes index stats, cluster health, and JVM metrics:

yamlapiVersion: helm.toolkit.fluxcd.io/v2
kind: HelmRelease
metadata:
  name: elasticsearch-exporter
  namespace: observability
spec:
  chart:
    spec:
      chart: prometheus-elasticsearch-exporter
      sourceRef:
        kind: HelmRepository
        name: prometheus-community
  values:
    es:
      uri: http://elasticsearch.logging.svc.cluster.local:9200
      username: elastic
      password: ""  # use secretRef in practice
    serviceMonitor:
      enabled: true
      labels:
        release: kube-prom-stack

Key metrics: elasticsearch_cluster_health_status, elasticsearch_indices_docs, elasticsearch_jvm_memory_used_bytes.

Troubleshooting

Yellow cluster health: Check replica count — number_of_replicas must be 0 on single-node.

OOM kills: Elasticsearch is memory-hungry. The JVM heap is set to 50% of container memory limit. A 4Gi limit gives 2Gi heap — adequate for moderate log volumes.

Slow queries / high CPU: Check the Slow Query Log:

json{"index": {"search.slowlog.threshold.query.warn": "2s"}}

Index not using pipeline: Check the default_pipeline is set in the index template, not just the index settings.

weblog