Kubernetes Production Stack

Enterprise-grade observability setup for Kubernetes deployments using Prometheus Operator, Grafana, and Jaeger. Includes automatic metric scraping, distributed tracing, and managed dashboards.

Architecture

WSO2 Integrator Pods
├── Metrics (9797) ──▶ Prometheus Operator (auto-discovers ServiceMonitor)
│                          ▼
│                     Prometheus StatefulSet
│                          ▼
└─ Traces (4317) ──▶ Jaeger Deployment (OTLP receiver)
                          ▼
                     Elasticsearch

Grafana Deployment ◀── Prometheus, Jaeger

Prerequisites

• Kubernetes cluster 1.25+
• Helm 3+
• kubectl configured

Installation Steps

Step 1: Create Observability Namespace

kubectl create namespace observability

Step 2: Add Helm Repositories

# Prometheus Operator
helm repo add prometheus-community https://prometheus-community.github.io/helm-charts

# Grafana
helm repo add grafana https://grafana.github.io/helm-charts

# Jaeger
helm repo add jaegertracing https://jaegertracing.github.io/helm-charts

helm repo update

Step 3: Install Prometheus Stack

Create prometheus-values.yml:

prometheus:
  prometheusSpec:
    retention: 15d
    enableAdminAPI: true
    resources:
      requests:
        cpu: 500m
        memory: 2Gi
    storageSpec:
      volumeClaimTemplate:
        spec:
          accessModes: ["ReadWriteOnce"]
          resources:
            requests:
              storage: 50Gi

grafana:
  enabled: true
  adminPassword: "admin-password"

alertmanager:
  enabled: true
  config:
    global:
      resolve_timeout: 5m
    route:
      group_by: ['alertname', 'cluster', 'service']
      group_wait: 10s
      group_interval: 10s
      repeat_interval: 12h
      receiver: 'null'

The chart provisions Grafana's Prometheus data source, so you don't need to add one.

helm install prometheus prometheus-community/kube-prometheus-stack \
  -n observability \
  -f prometheus-values.yml

Step 4: Install Jaeger

Create jaeger-values.yml. It configures Jaeger v2 to receive OTLP traces and store them in Elasticsearch. Replace http://elasticsearch:9200 with the URL of your Elasticsearch cluster.

userconfig:
  service:
    extensions: [jaeger_storage, jaeger_query, healthcheckv2]
    pipelines:
      traces:
        receivers: [otlp]
        processors: [batch]
        exporters: [jaeger_storage_exporter]
  extensions:
    healthcheckv2:
      use_v2: true
      http:
        endpoint: 0.0.0.0:13133
    jaeger_query:
      storage:
        traces: primary_store
    jaeger_storage:
      backends:
        primary_store:
          elasticsearch:
            server_urls: ["http://elasticsearch:9200"]
  receivers:
    otlp:
      protocols:
        grpc:
          endpoint: 0.0.0.0:4317
        http:
          endpoint: 0.0.0.0:4318
  processors:
    batch: {}
  exporters:
    jaeger_storage_exporter:
      trace_storage: primary_store

helm install jaeger jaegertracing/jaeger \
  -n observability \
  -f jaeger-values.yml

Step 5: Deploy Your Integration

Replace myorg/wso2-integrator:latest with your integration's container image. The integration must import ballerinax/prometheus and ballerinax/jaeger and be built with observabilityIncluded = true, as described in Metrics and Jaeger distributed tracing. The environment variables below enable metrics and tracing without a Config.toml.

apiVersion: apps/v1
kind: Deployment
metadata:
  name: wso2-integrator
  namespace: default
spec:
  replicas: 3
  selector:
    matchLabels:
      app: wso2-integrator
  template:
    metadata:
      labels:
        app: wso2-integrator
      annotations:
        prometheus.io/scrape: "true"
        prometheus.io/port: "9797"
        prometheus.io/path: "/metrics"
    spec:
      containers:
        - name: integrator
          image: myorg/wso2-integrator:latest
          ports:
            - name: http
              containerPort: 9090
            - name: metrics
              containerPort: 9797
          env:
            - name: BAL_CONFIG_VAR_BALLERINA_OBSERVE_METRICSENABLED
              value: "true"
            - name: BAL_CONFIG_VAR_BALLERINA_OBSERVE_METRICSREPORTER
              value: "prometheus"
            - name: BAL_CONFIG_VAR_BALLERINAX_PROMETHEUS_PORT
              value: "9797"
            - name: BAL_CONFIG_VAR_BALLERINA_OBSERVE_TRACINGENABLED
              value: "true"
            - name: BAL_CONFIG_VAR_BALLERINA_OBSERVE_TRACINGPROVIDER
              value: "jaeger"
            - name: BAL_CONFIG_VAR_BALLERINAX_JAEGER_AGENTHOSTNAME
              value: "jaeger.observability.svc.cluster.local"
            - name: BAL_CONFIG_VAR_BALLERINAX_JAEGER_AGENTPORT
              value: "4317"
          resources:
            requests:
              cpu: 250m
              memory: 512Mi
            limits:
              cpu: 500m
              memory: 1Gi
          livenessProbe:
            tcpSocket:
              port: http
            initialDelaySeconds: 30
            periodSeconds: 10
          readinessProbe:
            tcpSocket:
              port: http
            initialDelaySeconds: 10
            periodSeconds: 5

---
apiVersion: v1
kind: Service
metadata:
  name: wso2-integrator
  namespace: default
  labels:
    app: wso2-integrator
spec:
  type: ClusterIP
  ports:
    - name: http
      port: 9090
      targetPort: 9090
    - name: metrics
      port: 9797
      targetPort: 9797
  selector:
    app: wso2-integrator

Step 6: Create ServiceMonitor for Prometheus

apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata:
  name: wso2-integrator
  namespace: default
  labels:
    release: prometheus
spec:
  selector:
    matchLabels:
      app: wso2-integrator
  endpoints:
    - port: metrics
      path: /metrics
      interval: 30s

Verifying the Setup

Check Prometheus Targets

kubectl port-forward -n observability svc/prometheus-operated 9090:9090
# Visit http://localhost:9090/targets

Check Grafana

kubectl port-forward -n observability svc/prometheus-grafana 3000:80
# Visit http://localhost:3000 (admin / admin-password)

Check Jaeger UI

kubectl port-forward -n observability svc/jaeger 16686:16686
# Visit http://localhost:16686

Creating Alert Rules

Create a PrometheusRule for alerting:

apiVersion: monitoring.coreos.com/v1
kind: PrometheusRule
metadata:
  name: wso2-integrator-alerts
  namespace: default
  labels:
    release: prometheus
spec:
  groups:
    - name: wso2-integrator.rules
      interval: 30s
      rules:
        - alert: HighErrorRate
          expr: |
            (sum(rate(requests_total_value{http_status_code_group="5xx"}[5m])) by (service)) /
            (sum(rate(requests_total_value[5m])) by (service)) > 0.05
          for: 5m
          labels:
            severity: critical
            service: "{{ $labels.service }}"
          annotations:
            summary: "High error rate on {{ $labels.service }}"
            description: "Error rate is {{ $value | humanizePercentage }}"

        - alert: HighLatency
          expr: |
            max(response_time_seconds{quantile="0.95", timeWindow="300000"}) by (service) > 2
          for: 10m
          labels:
            severity: warning
            service: "{{ $labels.service }}"
          annotations:
            summary: "High P95 latency on {{ $labels.service }}"
            description: "P95 latency is {{ $value }}s"

        - alert: PodCrashLooping
          expr: |
            rate(kube_pod_container_status_restarts_total{pod=~"wso2-integrator.*"}[15m]) > 0.1
          for: 5m
          labels:
            severity: critical
          annotations:
            summary: "Pod {{ $labels.pod }} is crash looping"

Creating Grafana Dashboards

Import pre-built dashboard for Kubernetes:

# Download the dashboard and point it at the Prometheus data source
curl -s https://grafana.com/api/dashboards/7249/revisions/1/download \
  | sed 's/${DS_PROMETHEUS}/Prometheus/g' > kubernetes.json

# Install dashboard configmap
kubectl create configmap grafana-dashboard-kubernetes -n observability \
  --from-file=kubernetes.json
kubectl label configmap grafana-dashboard-kubernetes -n observability \
  grafana_dashboard=1

Grafana's dashboard sidecar loads ConfigMaps that have the grafana_dashboard=1 label.

Scaling Considerations

• Prometheus Storage: Use persistent volumes sized for your retention period
• Grafana: Stateless, can scale replicas
• Jaeger: Uses Elasticsearch backend; ensure sufficient storage
• Retention: Set appropriate retention for logs and metrics

Backup and Recovery

Backup Prometheus Data

Take a TSDB snapshot through the admin API (enabled by enableAdminAPI: true in Step 3):

kubectl port-forward -n observability svc/prometheus-operated 9090:9090
# In another terminal:
curl -X POST http://localhost:9090/api/v1/admin/tsdb/snapshot

Prometheus writes the snapshot to the snapshots directory on its data volume. Back up that volume with your volume snapshot or backup tool.

Backup Grafana Dashboards

Export dashboards through the Grafana HTTP API:

kubectl port-forward -n observability svc/prometheus-grafana 3000:80
# In another terminal, list the dashboards and note the uid of each one:
curl -s -u admin:admin-password "http://localhost:3000/api/search?type=dash-db"
curl -s -u admin:admin-password "http://localhost:3000/api/dashboards/uid/<uid>" \
  > grafana-dashboard-backup.json

Troubleshooting

Prometheus not scraping metrics:

# Check ServiceMonitor:
kubectl get servicemonitor -A

# Check Prometheus targets:
kubectl port-forward -n observability svc/prometheus-operated 9090:9090
# Visit http://localhost:9090/targets

Jaeger not receiving traces:

# Check Jaeger:
kubectl get pod -n observability -l app.kubernetes.io/name=jaeger

# Check logs:
kubectl logs -n observability -l app.kubernetes.io/name=jaeger

Grafana not showing metrics:

# Check that Prometheus is reachable inside the cluster:
kubectl run -n observability prometheus-check --rm -i --restart=Never \
  --image=curlimages/curl -- curl -s http://prometheus-operated:9090/-/healthy

What's Next

• ELK Stack Recipe – Add advanced log aggregation
• Integration Control Plane – Centralized WSO2 monitoring
• Metrics Overview – Learn more about metrics
• Jaeger distributed tracing – Learn more about tracing