Monitoring and Logging Kubernetes: metrics-server, Prometheus, Grafana and Log Collection
Three Kinds of Signals
| Signal | Question | Common tools |
|---|---|---|
| Metrics | How much? How fast? (CPU, memory, requests per second) | metrics-server, Prometheus, Grafana |
| Logs | What happened? (text lines) | kubectl logs, Fluent Bit, Loki, Elasticsearch |
| Traces | Where did the request spend time? | OpenTelemetry, Jaeger, Tempo |
| Events | What did Kubernetes do? | kubectl get events, event exporters |
metrics-server: The Built-in Resource Metrics
kubectl top and the Horizontal Pod Autoscaler need the Metrics API. It is provided by metrics-server, which collects CPU and memory from every kubelet. It is not installed by default in kubeadm clusters.
bash
# Install from the official manifest (check the project page for the current release URL)
kubectl apply -f https://github.com/kubernetes-sigs/metrics-server/releases/latest/download/components.yaml
kubectl -n kube-system get deploy metrics-server
kubectl get apiservice v1beta1.metrics.k8s.io
kubectl top nodes
kubectl top pods -A --sort-by=cpu | head
kubectl top pod web --containers
# In a lab with self-signed kubelet certificates you may need the argument --kubelet-insecure-tls
# (never use it in production; fix kubelet serving certificates instead)
What metrics-server is not: It keeps only the latest values in memory. It is not a monitoring system, has no history and no alerts. For history, use Prometheus.
Prometheus Architecture
text
targets (pods, nodes, kube-state-metrics, apps) <── scrape /metrics every 15-30 s ── Prometheus (TSDB) ──► Alertmanager ──► email/Slack/pager
│
└──► Grafana (dashboards, PromQL)
| Component | Provides |
|---|---|
| cAdvisor (built into kubelet) | Container CPU, memory, network, disk metrics |
| kube-state-metrics | State of Kubernetes objects (desired vs available replicas, pod phases, node conditions) |
| node-exporter | Machine metrics (CPU, memory, disk, network of the node) |
| Prometheus | Collects, stores and queries metrics (PromQL) |
| Alertmanager | Groups, silences and routes alerts |
| Grafana | Dashboards |
| Prometheus Operator | Manages Prometheus with CRDs: Prometheus, ServiceMonitor, PodMonitor, PrometheusRule |
Install kube-prometheus-stack With Helm
bash
helm repo add prometheus-community https://prometheus-community.github.io/helm-charts
helm repo update
helm install monitoring prometheus-community/kube-prometheus-stack \
-n monitoring --create-namespace \
--set grafana.adminPassword='change-me' \
--set prometheus.prometheusSpec.retention=15d
kubectl -n monitoring get pods
kubectl -n monitoring port-forward svc/monitoring-grafana 3000:80 # http://localhost:3000
kubectl -n monitoring port-forward svc/monitoring-kube-prometheus-prometheus 9090
Scrape Your Own App: ServiceMonitor
yaml
apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata:
name: shop-api
namespace: shop
labels:
release: monitoring # must match the Prometheus serviceMonitorSelector of your install
spec:
selector:
matchLabels: {app: api} # selects the SERVICE
endpoints:
- port: http # the NAME of the service port
path: /metrics
interval: 30s
Useful PromQL Queries
text
# CPU used per pod (cores)
sum(rate(container_cpu_usage_seconds_total{container!=""}[5m])) by (namespace, pod)
# Memory working set per pod
sum(container_memory_working_set_bytes{container!=""}) by (namespace, pod)
# Pods not ready
sum(kube_pod_status_ready{condition="false"}) by (namespace)
# Restarts in the last hour
increase(kube_pod_container_status_restarts_total[1h]) > 3
# Deployment with missing replicas
kube_deployment_spec_replicas != kube_deployment_status_replicas_available
# Node memory available percent
100 * node_memory_MemAvailable_bytes / node_memory_MemTotal_bytes
# CPU throttling ratio
sum(rate(container_cpu_cfs_throttled_periods_total[5m])) by (pod) / sum(rate(container_cpu_cfs_periods_total[5m])) by (pod)
# API server error rate
sum(rate(apiserver_request_total{code=~"5.."}[5m])) / sum(rate(apiserver_request_total[5m]))
Alerts to Start With
yaml
apiVersion: monitoring.coreos.com/v1
kind: PrometheusRule
metadata:
name: basic-alerts
namespace: monitoring
labels: {release: monitoring}
spec:
groups:
- name: workloads
rules:
- alert: PodCrashLooping
expr: increase(kube_pod_container_status_restarts_total[15m]) > 3
for: 5m
labels: {severity: warning}
annotations:
summary: "{{ $labels.namespace }}/{{ $labels.pod }} is restarting often"
- alert: DeploymentReplicasMismatch
expr: kube_deployment_spec_replicas != kube_deployment_status_replicas_available
for: 10m
labels: {severity: warning}
- alert: NodeNotReady
expr: kube_node_status_condition{condition="Ready",status="true"} == 0
for: 5m
labels: {severity: critical}
- alert: PVCAlmostFull
expr: kubelet_volume_stats_available_bytes / kubelet_volume_stats_capacity_bytes < 0.15
for: 15m
labels: {severity: warning}
| Cluster alert | Why |
|---|---|
| Node NotReady, disk or memory pressure | Capacity and health |
| API server error rate or latency high | Control plane health |
| etcd leader changes, fsync latency, DB size | State store health |
| Certificates expiring in under 30 days | Avoid outages |
| Pods Pending for long, CrashLoopBackOff | Application health |
| PVC above 85% full | Avoid stuck apps |
Container Logs
bash
kubectl logs web # stdout/stderr of the container
kubectl logs web --since=15m --timestamps
kubectl logs -l app=web --all-containers --prefix --tail=20 # many pods
kubectl logs deploy/web -f
# Where logs live on a node
ls /var/log/pods/ # per pod, per container, rotated by the kubelet
ls /var/log/containers/ # symlinks named pod_namespace_container-id.log
- Applications should log to stdout and stderr in a structured format (JSON). The runtime writes them to files on the node
- The kubelet rotates container logs (
containerLogMaxSize,containerLogMaxFiles), so logs vanish when a pod is deleted or the node is lost - Therefore ship logs to a central system
Central Logging: Fluent Bit and Loki
bash
helm repo add grafana https://grafana.github.io/helm-charts
helm install loki grafana/loki -n logging --create-namespace -f loki-values.yaml # check chart docs for values
helm repo add fluent https://fluent.github.io/helm-charts
helm install fluent-bit fluent/fluent-bit -n logging -f fluent-bit-values.yaml # DaemonSet reading /var/log/containers
# In Grafana add Loki as a data source, then query with LogQL:
# {namespace="shop", app="api"} |= "error" | json | level="error"
| Stack | Notes |
|---|---|
| Fluent Bit / Fluentd + Elasticsearch/OpenSearch + Kibana | Powerful search, heavier to run |
| Fluent Bit / Promtail / Alloy + Loki + Grafana | Lightweight, indexes labels only, cost efficient |
| Cloud services (CloudWatch, Azure Monitor, Cloud Logging) | Managed, integrated with managed Kubernetes |
Kubernetes Events
bash
kubectl get events -A --sort-by=.lastTimestamp
kubectl events -n shop --for pod/web # newer command
kubectl get events -A --field-selector type=Warning
# Events are kept for about 1 hour by default. Export them with an event exporter if you need history.
Traces With OpenTelemetry (Overview)
- The OpenTelemetry Collector runs as a DaemonSet or Deployment and receives traces, metrics and logs from apps (OTLP protocol)
- It sends them to Tempo, Jaeger, Prometheus or a vendor
- Add the OpenTelemetry SDK or auto-instrumentation to your apps
Practice Tasks
- Install metrics-server and use
kubectl top nodesandkubectl top pods - Install kube-prometheus-stack and open Grafana
- Write PromQL for pods restarting more than 3 times in an hour
- Add a ServiceMonitor for an app that exposes
/metrics - Ship container logs to Loki and search for an error string
Conclusion
metrics-server gives live CPU and memory for kubectl top and autoscaling. Prometheus with kube-state-metrics and node-exporter gives history and alerts, Grafana shows them, and a log collector such as Fluent Bit sends logs to a central store like Loki before pods disappear.
Quick Summary: metrics-server for top/HPA; Prometheus+kube-state-metrics+node-exporter; ServiceMonitor; alerts; logs to stdout then Fluent Bit->Loki; events 1 hour.

