Kubernetes Tutorial #8: Monitoring with Prometheus and Grafana

Your Kubernetes app is running. But is it healthy? Is it slow? Are any Pods crashing? How much memory is it using?

Without monitoring, you find out about problems when users complain. With monitoring, you know before they do.

The industry standard for Kubernetes monitoring is Prometheus + Grafana. Prometheus collects and stores metrics. Grafana visualizes them in dashboards.

The Observability Stack

Observability has three pillars:

Metrics — numbers over time (CPU usage, request count, error rate)
Logs — what happened and when (application output, events)
Traces — how a request flows through multiple services

This tutorial focuses on metrics with Prometheus and Grafana. Logging (Fluent Bit) and tracing (OpenTelemetry + Jaeger) are separate topics.

How Prometheus Works

Prometheus uses a pull model. Instead of your app sending metrics to Prometheus, Prometheus scrapes metrics from your app on a schedule.

Your app → exposes /metrics endpoint
Prometheus → scrapes /metrics every 15 seconds
Grafana → queries Prometheus to build dashboards

Your app exposes metrics in Prometheus format:

# HELP http_requests_total Total HTTP requests
# TYPE http_requests_total counter
http_requests_total{method="GET",status="200"} 1234
http_requests_total{method="POST",status="500"} 5

Kubernetes components (kubelet, API server, nodes) already expose metrics in this format.

Installing kube-prometheus-stack

The easiest way to set up Prometheus, Grafana, and all required components is with the kube-prometheus-stack Helm chart. It installs:

Prometheus (metrics storage and alerting)
Grafana (dashboards)
Alertmanager (alert routing)
node-exporter (node-level metrics)
kube-state-metrics (Kubernetes object metrics)
Pre-built dashboards for Kubernetes

# Add the Prometheus community Helm repository
helm repo add prometheus-community https://prometheus-community.github.io/helm-charts
helm repo update

# Create a namespace for monitoring
kubectl create namespace monitoring

# Install kube-prometheus-stack
helm install prometheus prometheus-community/kube-prometheus-stack \
  --namespace monitoring \
  --set grafana.adminPassword=securepassword123

Wait for all Pods to be ready:

kubectl get pods -n monitoring -w

NAME                                                  READY   STATUS    RESTARTS   AGE
prometheus-grafana-7d6b4b5c8-4xkzp                    3/3     Running   0          2m
prometheus-kube-state-metrics-7d4f96c-wqmnr            1/1     Running   0          2m
prometheus-prometheus-node-exporter-abcde              1/1     Running   0          2m
prometheus-prometheus-operator-5f9c6b-abc12            1/1     Running   0          2m
prometheus-prometheus-kube-prometheus-prometheus-0     2/2     Running   0          90s
prometheus-alertmanager-0                              2/2     Running   0          90s

Accessing Grafana

Port-forward the Grafana service to your local machine:

kubectl port-forward -n monitoring svc/prometheus-grafana 3000:80

Open http://localhost:3000 in your browser.

Username: admin
Password: securepassword123 (what you set above)

Pre-built Kubernetes Dashboards

Grafana has a library of pre-built dashboards. The most useful for Kubernetes:

Dashboard ID	Name	What it shows
315	Kubernetes cluster monitoring	CPU/memory per node and pod
3662	Prometheus stats	Prometheus itself
6417	Kubernetes cluster	High-level cluster view
13332	Kubernetes Resource Overview	All namespace resources

To import a dashboard:

Click the “+” menu in Grafana → Import
Enter the dashboard ID (e.g., 315)
Click Load → Select your Prometheus datasource → Import

You immediately get graphs for CPU usage, memory, pod counts, and node health.

Prometheus Metric Types

Understanding metric types helps you query and interpret them correctly.

Counter — only increases. Use for total requests, total errors.

http_requests_total

Gauge — can go up or down. Use for current memory, replica count.

container_memory_usage_bytes

Histogram — measures distribution (e.g., request duration). Use for latency percentiles.

http_request_duration_seconds_bucket

Summary — like histogram but calculated on the client. Less common.

PromQL Basics

Prometheus Query Language (PromQL) is how you query metrics. Grafana uses PromQL to build graphs.

# Current CPU usage per pod (as a rate over 5 minutes)
rate(container_cpu_usage_seconds_total{namespace="default"}[5m])

# Total memory used per pod
container_memory_usage_bytes{namespace="default"}

# HTTP request rate per second
rate(http_requests_total[5m])

# 95th percentile request duration
histogram_quantile(0.95, rate(http_request_duration_seconds_bucket[5m]))

# Sum memory by namespace
sum by (namespace) (container_memory_usage_bytes)

# Number of pod restarts in the last hour
increase(kube_pod_container_status_restarts_total[1h])

Key Kubernetes Metrics to Monitor

# Pods NOT in Running state
kube_pod_status_phase{phase!="Running"}

# CPU throttling (bad if high)
rate(container_cpu_throttled_seconds_total[5m])

# Pods in CrashLoopBackOff
kube_pod_container_status_waiting_reason{reason="CrashLoopBackOff"}

# Nodes not ready
kube_node_status_condition{condition="Ready",status="true"} == 0

# Persistent Volume almost full
(kubelet_volume_stats_used_bytes / kubelet_volume_stats_capacity_bytes) > 0.85

Adding Application Metrics

Your app can expose custom metrics using a Prometheus client library.

Python example (with prometheus-client):

from prometheus_client import Counter, Histogram, start_http_server
import time

# Define metrics
REQUEST_COUNT = Counter('http_requests_total', 'Total HTTP requests',
                        ['method', 'endpoint', 'status'])
REQUEST_LATENCY = Histogram('http_request_duration_seconds', 'Request latency',
                            ['endpoint'])

# In your request handler:
start = time.time()
# ... handle request ...
REQUEST_COUNT.labels(method='GET', endpoint='/api', status='200').inc()
REQUEST_LATENCY.labels(endpoint='/api').observe(time.time() - start)

# Expose /metrics endpoint
start_http_server(8080)

Go example (with prometheus/client_golang):

import (
    "github.com/prometheus/client_golang/prometheus"
    "github.com/prometheus/client_golang/prometheus/promhttp"
    "net/http"
)

var httpRequests = prometheus.NewCounterVec(
    prometheus.CounterOpts{
        Name: "http_requests_total",
        Help: "Total HTTP requests",
    },
    []string{"method", "status"},
)

func init() {
    prometheus.MustRegister(httpRequests)
}

// Expose /metrics
http.Handle("/metrics", promhttp.Handler())

Tell Prometheus to scrape your app by adding a ServiceMonitor:

# servicemonitor.yaml
apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata:
  name: myapp-monitor
  namespace: monitoring
spec:
  selector:
    matchLabels:
      app: myapp
  endpoints:
    - port: http
      path: /metrics
      interval: 30s
  namespaceSelector:
    matchNames:
      - default

kubectl apply -f servicemonitor.yaml

Prometheus will now scrape your app’s /metrics endpoint every 30 seconds.

Setting Up Alerts

Alertmanager routes alerts to Slack, PagerDuty, email, or other destinations.

Define alert rules with PrometheusRule:

# alertrules.yaml
apiVersion: monitoring.coreos.com/v1
kind: PrometheusRule
metadata:
  name: myapp-alerts
  namespace: monitoring
spec:
  groups:
    - name: myapp
      rules:
        - alert: HighErrorRate
          expr: rate(http_requests_total{status="500"}[5m]) > 0.1
          for: 5m
          labels:
            severity: warning
          annotations:
            summary: "High error rate on {{ $labels.instance }}"
            description: "Error rate is {{ $value }} req/s"

        - alert: PodCrashLooping
          expr: increase(kube_pod_container_status_restarts_total[15m]) > 5
          for: 5m
          labels:
            severity: critical
          annotations:
            summary: "Pod {{ $labels.pod }} is crash looping"

kubectl apply -f alertrules.yaml

Accessing Prometheus Directly

Port-forward to the Prometheus UI for debugging queries:

kubectl port-forward -n monitoring svc/prometheus-kube-prometheus-prometheus 9090:9090

Open http://localhost:9090. Use the expression browser to run PromQL queries directly.

Common Mistakes

No retention limit on Prometheus

The kube-prometheus-stack Helm chart defaults to 10 days retention. Vanilla Prometheus defaults to 15 days. On busy clusters, this can fill your disk quickly. Set an explicit retention limit:

helm upgrade prometheus prometheus-community/kube-prometheus-stack \
  --namespace monitoring \
  --set prometheus.prometheusSpec.retention=7d \
  --set prometheus.prometheusSpec.retentionSize=10GB

High-cardinality labels

Avoid using user IDs, request IDs, or other unbounded values as Prometheus labels. Each unique label combination creates a new time series. 1 million unique user IDs = 1 million time series = Prometheus crashes.

# BAD — user_id has unbounded cardinality
REQUEST_COUNT = Counter('requests', 'Requests', ['user_id'])

# GOOD — status has bounded cardinality (200, 404, 500, ...)
REQUEST_COUNT = Counter('requests', 'Requests', ['status'])

No resource limits on Prometheus

Prometheus can be memory-hungry on large clusters. Always set resource requests and limits:

helm upgrade prometheus prometheus-community/kube-prometheus-stack \
  --namespace monitoring \
  --set prometheus.prometheusSpec.resources.requests.memory=1Gi \
  --set prometheus.prometheusSpec.resources.limits.memory=4Gi

What’s Next?

Your cluster is observable. The next step is automating deployments — setting up a CI/CD pipeline with GitHub Actions.

Next: Kubernetes Tutorial #9: CI/CD with Kubernetes and GitHub Actions

The Observability Stack#

How Prometheus Works#

Installing kube-prometheus-stack#

Accessing Grafana#

Pre-built Kubernetes Dashboards#

Prometheus Metric Types#

PromQL Basics#

Key Kubernetes Metrics to Monitor#

Adding Application Metrics#

Setting Up Alerts#

Accessing Prometheus Directly#

Common Mistakes#

What’s Next?#

Related Articles#