Elasticsearch Index Lifecycle Management

Published: 2026-03-22

Without ILM, Elasticsearch indices grow indefinitely. On a single-node cluster with 50GB PVC, that's a disk-full within weeks. ILM automates the lifecycle: rollover when an index gets too big, merge and shrink on older data, delete after a retention period.

The four phases

For a single-node setup: skip shrink (requires multiple primary shards), skip freeze (adds overhead), use hot → delete only.

Creating the ILM policy

bashcopycurl -X PUT "http://elasticsearch:9200/_ilm/policy/vector-logs" \
  -H "Content-Type: application/json" \
  -d '{
    "policy": {
      "phases": {
        "hot": {
          "min_age": "0ms",
          "actions": {
            "rollover": {
              "max_primary_shard_size": "20gb",
              "max_age": "7d"
            },
            "set_priority": {
              "priority": 100
            }
          }
        },
        "delete": {
          "min_age": "30d",
          "actions": {
            "delete": {}
          }
        }
      }
    }
  }'

Rolls over the write index when it reaches 20GB or 7 days old. Indices older than 30 days from rollover are deleted.

Index template

The template applies the ILM policy to all new indices automatically:

bashcopycurl -X PUT "http://elasticsearch:9200/_index_template/vector-logs" \
  -H "Content-Type: application/json" \
  -d '{
    "index_patterns": ["vector-*"],
    "template": {
      "settings": {
        "index.lifecycle.name": "vector-logs",
        "index.lifecycle.rollover_alias": "vector-logs",
        "number_of_replicas": 0,
        "number_of_shards": 1
      },
      "mappings": {
        "properties": {
          "@timestamp": {"type": "date"},
          "message": {"type": "text"},
          "level": {"type": "keyword"},
          "kubernetes.namespace": {"type": "keyword"},
          "kubernetes.pod.name": {"type": "keyword"}
        }
      }
    }
  }'

number_of_replicas: 0 — essential for single-node cluster. Elasticsearch keeps shards yellow if replicas can't be placed. number_of_shards: 1 — one shard per index for small-to-medium data volumes.

Bootstrap the write alias

ILM requires a write alias pointing to the initial index:

bashcopycurl -X PUT "http://elasticsearch:9200/vector-logs-000001" \
  -H "Content-Type: application/json" \
  -d '{
    "aliases": {
      "vector-logs": {
        "is_write_index": true
      }
    }
  }'

Vector's Elasticsearch sink writes to the alias vector-logs, not to a specific index. ILM creates vector-logs-000002, vector-logs-000003, etc. on rollover.

Bootstrap script as a Kubernetes Job

bashcopy#!/bin/bash
set -e

ES_URL="http://elasticsearch-master.observability.svc.cluster.local:9200"

# Wait for ES
until curl -sf "$ES_URL/_cluster/health?wait_for_status=yellow"; do
  echo "waiting for elasticsearch..."
  sleep 5
done

# Apply ILM policy
curl -sf -X PUT "$ES_URL/_ilm/policy/vector-logs" \
  -H "Content-Type: application/json" \
  --data-binary @/bootstrap/ilm-policy.json

# Apply index template
curl -sf -X PUT "$ES_URL/_index_template/vector-logs" \
  -H "Content-Type: application/json" \
  --data-binary @/bootstrap/index-template.json

# Create initial index (idempotent: ignore 400)
curl -sf -X PUT "$ES_URL/vector-logs-000001" \
  -H "Content-Type: application/json" \
  --data-binary @/bootstrap/initial-index.json \
  || true

echo "Bootstrap complete"

The || true on the last step makes the Job idempotent — re-running it after a cluster rebuild doesn't fail if the index already exists.

Monitoring ILM

bashcopy# Check which phase each index is in
curl http://elasticsearch:9200/vector-*/_ilm/explain?filter_path=indices.*.phase,indices.*.age

# Check for ILM errors
curl http://elasticsearch:9200/vector-*/_ilm/explain | jq '.indices | to_entries[] | select(.value.step_info.type != null)'

# Manually trigger rollover
curl -X POST "http://elasticsearch:9200/vector-logs/_rollover"

# Force ILM to re-evaluate
curl -X POST "http://elasticsearch:9200/vector-logs-000001/_ilm/retry"

Shards sizing guide

Over-sharding is a common mistake. Each shard has overhead (file handles, memory). For small datasets, 1 shard per index is optimal.

Troubleshooting

Alias doesn't exist error on rollover: The bootstrap script wasn't run or the initial index doesn't have the alias. Run bootstrap again.

Index yellow: Usually number_of_replicas > 0 on a single-node cluster. Set replicas to 0.

ILM not progressing: ILM checks every 10 minutes by default. Use /_ilm/explain to see the current step and any errors. Use /_ilm/retry to unstick a failed step.