Loki and LogQL: logs without Elasticsearch overhead

Published: 2026-03-21

Loki indexes only log labels, not log content. This makes it dramatically cheaper than Elasticsearch for raw log storage — 10× less disk, minimal CPU for ingestion. The tradeoff: full-text search is slower because it regex-scans log lines rather than looking up an inverted index. For most use cases — finding errors, tracing requests, debugging — LogQL is fast enough and the cost savings are real.

Install with Helm

yamlapiVersion: helm.toolkit.fluxcd.io/v2
kind: HelmRelease
metadata:
  name: loki
  namespace: observability
spec:
  chart:
    spec:
      chart: loki
      version: "6.x"
      sourceRef:
        kind: HelmRepository
        name: grafana
  values:
    deploymentMode: SingleBinary
    loki:
      commonConfig:
        replication_factor: 1
      storage:
        type: filesystem
      auth_enabled: false
      limits_config:
        retention_period: 30d
        ingestion_rate_mb: 16
        ingestion_burst_size_mb: 32
    singleBinary:
      replicas: 1
      persistence:
        enabled: true
        storageClass: local-path
        size: 50Gi
    monitoring:
      serviceMonitor:
        enabled: true
        labels:
          release: kube-prom-stack

SingleBinary mode runs all Loki components in one pod. Fine for a single-cluster setup. For HA, use SimpleScalable or Distributed mode with object storage.

Grafana Alloy (Promtail replacement)

Promtail is deprecated in favor of Grafana Alloy. Alloy ships logs to Loki, metrics to Prometheus, and traces to Tempo from the same agent:

yamlapiVersion: helm.toolkit.fluxcd.io/v2
kind: HelmRelease
metadata:
  name: alloy
  namespace: observability
spec:
  chart:
    spec:
      chart: alloy
      version: "0.x"
      sourceRef:
        kind: HelmRepository
        name: grafana
  values:
    alloy:
      configMap:
        content: |
          discovery.kubernetes "pods" {
            role = "pod"
          }

          discovery.relabel "pods" {
            targets = discovery.kubernetes.pods.targets
            rule {
              source_labels = ["__meta_kubernetes_pod_node_name"]
              target_label  = "node"
            }
            rule {
              source_labels = ["__meta_kubernetes_namespace"]
              target_label  = "namespace"
            }
            rule {
              source_labels = ["__meta_kubernetes_pod_label_app"]
              target_label  = "app"
            }
          }

          loki.source.kubernetes "pods" {
            targets    = discovery.relabel.pods.output
            forward_to = [loki.write.default.receiver]
          }

          loki.write "default" {
            endpoint {
              url = "http://loki.observability.svc.cluster.local:3100/loki/api/v1/push"
            }
          }

LogQL basics

LogQL starts with a log stream selector and optionally pipes through filters and parsers.

Stream selectors:

logql{namespace="app", app="my-service"}
{namespace="app"} |= "ERROR"
{namespace="app"} |~ "timeout|refused"
{namespace="app"} != "healthz"

|= line contains string
|~ line matches regex
!= line does not contain string
!~ line does not match regex

JSON parsing:

logql{namespace="app", app="my-service"}
  | json
  | level="error"
  | line_format "{{.message}}"

| json parses each log line as JSON and extracts fields. | level="error" filters on the parsed level field. line_format controls what's shown.

Rate queries for Grafana panels:

logqlrate({namespace="app"} |= "ERROR" [5m])

Errors per second over the last 5 minutes. Use in dashboards to show error rate alongside metrics.

Count errors by service:

logqlsum by (app) (
  rate({namespace="app"} |= "ERROR" [5m])
)

Grafana datasource

yamladditionalDataSources:
  - name: Loki
    type: loki
    url: http://loki.observability.svc.cluster.local:3100
    access: proxy
    isDefault: false
    jsonData:
      derivedFields:
        - name: TraceID
          matcherRegex: '"traceId":"(\w+)"'
          url: "${__value.raw}"
          datasourceUid: jaeger

The derivedFields config makes trace IDs in log lines clickable links that open in Jaeger. This bridges logs and traces without a full APM solution.

Loki vs Elasticsearch

Feature	Loki	Elasticsearch
Storage cost	Low (label-only index)	High (full inverted index)
Ingestion CPU	Minimal	Significant
Full-text search	Regex scan (slower)	Inverted index (fast)
Structured field filtering	Via JSON parser	Native
Integration	Grafana	Kibana
Best for	Kubernetes logs	Application search

Troubleshooting

Too many streams error: Loki has a default limit on unique label combinations. Avoid high-cardinality labels like pod names or request IDs. Keep labels to: namespace, app, job, level.

Ingestion rate limit: Increase ingestion_rate_mb in limits_config or apply per-tenant limits.

Queries timing out: Add |= "search term" to reduce the number of log lines before parsing. The stream selector filters on labels (fast), then filters narrow by content.

weblog