Monitoring and Logging Kubernetes: metrics-server, Prometheus, Grafana and Log Collection

Certified Kubernetes Administrator 30 min min read Updated: Nov 17, 2026 Intermediate
Monitoring and Logging Kubernetes: metrics-server, Prometheus, Grafana and Log Collection
Intermediate Topic 1 of 2

Three Kinds of Signals

SignalQuestionCommon tools
MetricsHow much? How fast? (CPU, memory, requests per second)metrics-server, Prometheus, Grafana
LogsWhat happened? (text lines)kubectl logs, Fluent Bit, Loki, Elasticsearch
TracesWhere did the request spend time?OpenTelemetry, Jaeger, Tempo
EventsWhat did Kubernetes do?kubectl get events, event exporters

metrics-server: The Built-in Resource Metrics

kubectl top and the Horizontal Pod Autoscaler need the Metrics API. It is provided by metrics-server, which collects CPU and memory from every kubelet. It is not installed by default in kubeadm clusters.

bash # Install from the official manifest (check the project page for the current release URL) kubectl apply -f https://github.com/kubernetes-sigs/metrics-server/releases/latest/download/components.yaml kubectl -n kube-system get deploy metrics-server kubectl get apiservice v1beta1.metrics.k8s.io kubectl top nodes kubectl top pods -A --sort-by=cpu | head kubectl top pod web --containers # In a lab with self-signed kubelet certificates you may need the argument --kubelet-insecure-tls # (never use it in production; fix kubelet serving certificates instead)
What metrics-server is not: It keeps only the latest values in memory. It is not a monitoring system, has no history and no alerts. For history, use Prometheus.

Prometheus Architecture

text targets (pods, nodes, kube-state-metrics, apps) <── scrape /metrics every 15-30 s ── Prometheus (TSDB) ──► Alertmanager ──► email/Slack/pager │ └──► Grafana (dashboards, PromQL)
ComponentProvides
cAdvisor (built into kubelet)Container CPU, memory, network, disk metrics
kube-state-metricsState of Kubernetes objects (desired vs available replicas, pod phases, node conditions)
node-exporterMachine metrics (CPU, memory, disk, network of the node)
PrometheusCollects, stores and queries metrics (PromQL)
AlertmanagerGroups, silences and routes alerts
GrafanaDashboards
Prometheus OperatorManages Prometheus with CRDs: Prometheus, ServiceMonitor, PodMonitor, PrometheusRule

Install kube-prometheus-stack With Helm

bash helm repo add prometheus-community https://prometheus-community.github.io/helm-charts helm repo update helm install monitoring prometheus-community/kube-prometheus-stack \ -n monitoring --create-namespace \ --set grafana.adminPassword='change-me' \ --set prometheus.prometheusSpec.retention=15d kubectl -n monitoring get pods kubectl -n monitoring port-forward svc/monitoring-grafana 3000:80 # http://localhost:3000 kubectl -n monitoring port-forward svc/monitoring-kube-prometheus-prometheus 9090

Scrape Your Own App: ServiceMonitor

yaml apiVersion: monitoring.coreos.com/v1 kind: ServiceMonitor metadata: name: shop-api namespace: shop labels: release: monitoring # must match the Prometheus serviceMonitorSelector of your install spec: selector: matchLabels: {app: api} # selects the SERVICE endpoints: - port: http # the NAME of the service port path: /metrics interval: 30s

Useful PromQL Queries

text # CPU used per pod (cores) sum(rate(container_cpu_usage_seconds_total{container!=""}[5m])) by (namespace, pod) # Memory working set per pod sum(container_memory_working_set_bytes{container!=""}) by (namespace, pod) # Pods not ready sum(kube_pod_status_ready{condition="false"}) by (namespace) # Restarts in the last hour increase(kube_pod_container_status_restarts_total[1h]) > 3 # Deployment with missing replicas kube_deployment_spec_replicas != kube_deployment_status_replicas_available # Node memory available percent 100 * node_memory_MemAvailable_bytes / node_memory_MemTotal_bytes # CPU throttling ratio sum(rate(container_cpu_cfs_throttled_periods_total[5m])) by (pod) / sum(rate(container_cpu_cfs_periods_total[5m])) by (pod) # API server error rate sum(rate(apiserver_request_total{code=~"5.."}[5m])) / sum(rate(apiserver_request_total[5m]))

Alerts to Start With

yaml apiVersion: monitoring.coreos.com/v1 kind: PrometheusRule metadata: name: basic-alerts namespace: monitoring labels: {release: monitoring} spec: groups: - name: workloads rules: - alert: PodCrashLooping expr: increase(kube_pod_container_status_restarts_total[15m]) > 3 for: 5m labels: {severity: warning} annotations: summary: "{{ $labels.namespace }}/{{ $labels.pod }} is restarting often" - alert: DeploymentReplicasMismatch expr: kube_deployment_spec_replicas != kube_deployment_status_replicas_available for: 10m labels: {severity: warning} - alert: NodeNotReady expr: kube_node_status_condition{condition="Ready",status="true"} == 0 for: 5m labels: {severity: critical} - alert: PVCAlmostFull expr: kubelet_volume_stats_available_bytes / kubelet_volume_stats_capacity_bytes < 0.15 for: 15m labels: {severity: warning}
Cluster alertWhy
Node NotReady, disk or memory pressureCapacity and health
API server error rate or latency highControl plane health
etcd leader changes, fsync latency, DB sizeState store health
Certificates expiring in under 30 daysAvoid outages
Pods Pending for long, CrashLoopBackOffApplication health
PVC above 85% fullAvoid stuck apps

Container Logs

bash kubectl logs web # stdout/stderr of the container kubectl logs web --since=15m --timestamps kubectl logs -l app=web --all-containers --prefix --tail=20 # many pods kubectl logs deploy/web -f # Where logs live on a node ls /var/log/pods/ # per pod, per container, rotated by the kubelet ls /var/log/containers/ # symlinks named pod_namespace_container-id.log
  • Applications should log to stdout and stderr in a structured format (JSON). The runtime writes them to files on the node
  • The kubelet rotates container logs (containerLogMaxSize, containerLogMaxFiles), so logs vanish when a pod is deleted or the node is lost
  • Therefore ship logs to a central system

Central Logging: Fluent Bit and Loki

bash helm repo add grafana https://grafana.github.io/helm-charts helm install loki grafana/loki -n logging --create-namespace -f loki-values.yaml # check chart docs for values helm repo add fluent https://fluent.github.io/helm-charts helm install fluent-bit fluent/fluent-bit -n logging -f fluent-bit-values.yaml # DaemonSet reading /var/log/containers # In Grafana add Loki as a data source, then query with LogQL: # {namespace="shop", app="api"} |= "error" | json | level="error"
StackNotes
Fluent Bit / Fluentd + Elasticsearch/OpenSearch + KibanaPowerful search, heavier to run
Fluent Bit / Promtail / Alloy + Loki + GrafanaLightweight, indexes labels only, cost efficient
Cloud services (CloudWatch, Azure Monitor, Cloud Logging)Managed, integrated with managed Kubernetes

Kubernetes Events

bash kubectl get events -A --sort-by=.lastTimestamp kubectl events -n shop --for pod/web # newer command kubectl get events -A --field-selector type=Warning # Events are kept for about 1 hour by default. Export them with an event exporter if you need history.

Traces With OpenTelemetry (Overview)

  • The OpenTelemetry Collector runs as a DaemonSet or Deployment and receives traces, metrics and logs from apps (OTLP protocol)
  • It sends them to Tempo, Jaeger, Prometheus or a vendor
  • Add the OpenTelemetry SDK or auto-instrumentation to your apps

Practice Tasks

  1. Install metrics-server and use kubectl top nodes and kubectl top pods
  2. Install kube-prometheus-stack and open Grafana
  3. Write PromQL for pods restarting more than 3 times in an hour
  4. Add a ServiceMonitor for an app that exposes /metrics
  5. Ship container logs to Loki and search for an error string

Conclusion

metrics-server gives live CPU and memory for kubectl top and autoscaling. Prometheus with kube-state-metrics and node-exporter gives history and alerts, Grafana shows them, and a log collector such as Fluent Bit sends logs to a central store like Loki before pods disappear.

Quick Summary: metrics-server for top/HPA; Prometheus+kube-state-metrics+node-exporter; ServiceMonitor; alerts; logs to stdout then Fluent Bit->Loki; events 1 hour.

Get Newsletter

Subscibe to our newsletter and we will notify you about the newest updates on Edugators