DevOps Project 3: Production-Style Platform on Kubernetes with GitOps, Observability and Security

DevOps Certification 40 min min read Updated: Nov 17, 2026 Advanced
DevOps Project 3: Production-Style Platform on Kubernetes with GitOps, Observability and Security
Advanced Topic 3 of 3

Project Goal

Create a small but realistic internal platform on Kubernetes. Everything is installed from a Git repository using Argo CD. Developers deploy only by merging pull requests. The platform provides metrics, logs, dashboards, alerts, security policies and safe canary releases.

text GitHub: platform-config repo | (Argo CD pulls) v kind cluster |- platform namespace group: ingress-nginx, cert-manager, kube-prometheus-stack, loki, argo-rollouts, kyverno |- app namespace: shop-api (Rollout, canary, ServiceMonitor, NetworkPolicy) |- policies: no :latest, non-root, signed images (audit), default-deny network '- alerts: SLO burn rate -> Alertmanager (webhook/Slack)
Resources: Run this on a machine with 4 CPUs and 8 GB RAM or more, or use a small cloud cluster. Create a kind cluster with the config below. Delete it when you are done.

Step 1: Create the Cluster

yaml # kind-config.yaml kind: Cluster apiVersion: kind.x-k8s.io/v1alpha4 nodes: - role: control-plane kubeadmConfigPatches: - | kind: InitConfiguration nodeRegistration: kubeletExtraArgs: node-labels: "ingress-ready=true" extraPortMappings: - { containerPort: 80, hostPort: 80, protocol: TCP } - { containerPort: 443, hostPort: 443, protocol: TCP } - role: worker - role: worker
bash kind create cluster --name platform --config kind-config.yaml kubectl get nodes

Step 2: Config Repository Layout

text platform-config/ bootstrap/ root-app.yaml <- the only thing you apply by hand platform/ <- Argo CD Application manifests (one per component) ingress-nginx.yaml cert-manager.yaml monitoring.yaml <- kube-prometheus-stack loki.yaml argo-rollouts.yaml kyverno.yaml policies.yaml <- points at policies/ policies/ disallow-latest.yaml require-nonroot.yaml default-deny-netpol.yaml apps/ shop-api/ rollout.yaml service.yaml servicemonitor.yaml analysis.yaml networkpolicy.yaml shop-api.yaml <- Argo CD Application for the app

Step 3: Install Argo CD and the Root Application

bash kubectl create namespace argocd kubectl apply -n argocd -f https://raw.githubusercontent.com/argoproj/argo-cd/stable/manifests/install.yaml kubectl -n argocd rollout status deploy/argocd-server --timeout=180s
yaml # bootstrap/root-app.yaml ("app of apps") apiVersion: argoproj.io/v1alpha1 kind: Application metadata: name: root namespace: argocd spec: project: default source: repoURL: https://github.com/YOUR_USER/platform-config.git targetRevision: main path: platform destination: server: https://kubernetes.default.svc namespace: argocd syncPolicy: automated: { prune: true, selfHeal: true }
bash kubectl apply -f bootstrap/root-app.yaml kubectl -n argocd get applications -w

Step 4: Platform Components as Applications

Each file in platform/ is an Argo CD Application. Examples (replace <chart-version> with a current version from each project's repository):

yaml # platform/ingress-nginx.yaml apiVersion: argoproj.io/v1alpha1 kind: Application metadata: name: ingress-nginx namespace: argocd annotations: { argocd.argoproj.io/sync-wave: "-2" } spec: project: default source: repoURL: https://kubernetes.github.io/ingress-nginx chart: ingress-nginx targetRevision: "<chart-version>" helm: values: | controller: hostPort: { enabled: true } service: { type: NodePort } nodeSelector: { ingress-ready: "true" } tolerations: - { key: node-role.kubernetes.io/control-plane, operator: Equal, effect: NoSchedule } destination: { server: https://kubernetes.default.svc, namespace: ingress-nginx } syncPolicy: automated: { prune: true, selfHeal: true } syncOptions: [CreateNamespace=true] --- # platform/monitoring.yaml apiVersion: argoproj.io/v1alpha1 kind: Application metadata: name: monitoring namespace: argocd spec: project: default source: repoURL: https://prometheus-community.github.io/helm-charts chart: kube-prometheus-stack targetRevision: "<chart-version>" helm: values: | grafana: adminPassword: change-me-lab-only ingress: { enabled: true, ingressClassName: nginx, hosts: [grafana.localtest.me] } prometheus: prometheusSpec: serviceMonitorSelectorNilUsesHelmValues: false # pick up all ServiceMonitors destination: { server: https://kubernetes.default.svc, namespace: monitoring } syncPolicy: automated: { prune: true, selfHeal: true } syncOptions: [CreateNamespace=true, ServerSideApply=true]
localtest.me: Names such as grafana.localtest.me resolve to 127.0.0.1 through public DNS, so you can use hostnames with Ingress on your laptop without editing /etc/hosts.

Create similar Applications for loki (chart loki-stack or the current Loki chart with Promtail or Alloy), argo-rollouts, kyverno and cert-manager. Add Loki as a Grafana data source through the monitoring chart values (grafana.additionalDataSources).

Step 5: Security Policies as Code

yaml # policies/disallow-latest.yaml apiVersion: kyverno.io/v1 kind: ClusterPolicy metadata: { name: disallow-latest-tag } spec: validationFailureAction: Audit # change to Enforce after fixing violations background: true rules: - name: require-pinned-tag match: { any: [{ resources: { kinds: [Pod] } }] } validate: message: "Use a specific image tag, not :latest" pattern: spec: containers: - image: "!*:latest" --- # policies/require-nonroot.yaml apiVersion: kyverno.io/v1 kind: ClusterPolicy metadata: { name: require-run-as-nonroot } spec: validationFailureAction: Audit rules: - name: run-as-non-root match: any: - resources: kinds: [Pod] namespaces: [shop] validate: message: "Containers must set runAsNonRoot: true" pattern: spec: containers: - securityContext: { runAsNonRoot: true } --- # apps/shop-api/networkpolicy.yaml apiVersion: networking.k8s.io/v1 kind: NetworkPolicy metadata: { name: default-deny-ingress, namespace: shop } spec: podSelector: {} policyTypes: [Ingress] --- apiVersion: networking.k8s.io/v1 kind: NetworkPolicy metadata: { name: allow-ingress-controller, namespace: shop } spec: podSelector: { matchLabels: { app: shop-api } } ingress: - from: - namespaceSelector: { matchLabels: { kubernetes.io/metadata.name: ingress-nginx } } - namespaceSelector: { matchLabels: { kubernetes.io/metadata.name: monitoring } }
CNI note: kind's default network plugin does not enforce NetworkPolicy. Install Calico or Cilium in the cluster (create the cluster with disableDefaultCNI: true) if you want enforcement, otherwise the policies are applied but not enforced.

Step 6: The Application with Canary and Analysis

yaml # apps/shop-api/rollout.yaml apiVersion: argoproj.io/v1alpha1 kind: Rollout metadata: { name: shop-api, namespace: shop } spec: replicas: 4 selector: { matchLabels: { app: shop-api } } template: metadata: { labels: { app: shop-api } } spec: containers: - name: api image: ghcr.io/YOUR_USER/shop-api:1.0.0 ports: [{ name: http, containerPort: 8000 }] securityContext: { runAsNonRoot: true, allowPrivilegeEscalation: false } readinessProbe: { httpGet: { path: /health, port: http } } resources: { requests: { cpu: 50m, memory: 64Mi }, limits: { memory: 128Mi } } strategy: canary: canaryService: shop-api-canary stableService: shop-api trafficRouting: { nginx: { stableIngress: shop-api } } steps: - setWeight: 20 - pause: { duration: 1m } - analysis: { templates: [{ templateName: success-rate }] } - setWeight: 60 - pause: { duration: 1m } --- # apps/shop-api/servicemonitor.yaml apiVersion: monitoring.coreos.com/v1 kind: ServiceMonitor metadata: { name: shop-api, namespace: shop } spec: selector: { matchLabels: { app: shop-api } } endpoints: [{ port: http, path: /metrics, interval: 15s }]

Use the instrumented version of the API from the monitoring lesson (it exposes /metrics). Create the shop-api and shop-api-canary Services, an Ingress for shop.localtest.me, and an AnalysisTemplate named success-rate that queries Prometheus for the success ratio.

Step 7: SLO Alerts and Runbook

yaml # apps/shop-api/alerts.yaml apiVersion: monitoring.coreos.com/v1 kind: PrometheusRule metadata: { name: shop-api-slo, namespace: shop, labels: { release: monitoring } } spec: groups: - name: shop-api-slo rules: - alert: ShopApiErrorBudgetFastBurn expr: | (sum(rate(http_requests_total{namespace="shop",status=~"5.."}[5m])) / sum(rate(http_requests_total{namespace="shop"}[5m]))) > (14.4 * 0.001) for: 2m labels: { severity: page } annotations: summary: "shop-api is burning its 99.9 percent SLO budget quickly" runbook_url: https://github.com/YOUR_USER/platform-config/blob/main/runbooks/shop-api-errors.md
text # runbooks/shop-api-errors.md ## Symptom: ShopApiErrorBudgetFastBurn 1. Open Grafana dashboard "shop-api" and check error rate by path. 2. Check the rollout: kubectl argo rollouts get rollout shop-api -n shop 3. If a release is in progress or was recent: kubectl argo rollouts abort shop-api -n shop, then revert the commit in Git. 4. Look at logs in Grafana Explore (Loki): {namespace="shop"} |= "ERROR" 5. Check dependencies and node health. Escalate to the platform team if unresolved after 15 minutes.

Step 8: End-to-End Demo Script

  1. git push a change to the config repo that bumps the image from 1.0.0 to 1.1.0
  2. Watch Argo CD sync: argocd app get shop-api or the UI
  3. Watch the canary: kubectl argo rollouts get rollout shop-api -n shop --watch
  4. Generate traffic: while true; do curl -s shop.localtest.me/products > /dev/null; done
  5. Open Grafana and see request rate and latency for stable and canary
  6. Release a bad version (return HTTP 500 for 30 percent of requests). The analysis fails and the rollout aborts automatically
  7. Show the alert firing in Alertmanager and follow the runbook
  8. Try to deploy a Pod with image: nginx:latest into the shop namespace and show the Kyverno audit report (kubectl get policyreport -A)
  9. Delete a Deployment manually with kubectl and watch Argo CD self-heal it back

Troubleshooting Tips

ProblemFix
Applications stay OutOfSync for CRDsUse ServerSideApply=true and sync waves so CRDs install before resources that use them
Prometheus does not see the ServiceMonitorCheck the release label and serviceMonitorSelector of the Prometheus resource
Ingress returns 404Check ingressClassName, host name and that the controller Pod is running on the node with the port mapping
Out of memory on laptopDisable Loki or lower replica counts and retention in the chart values
Kyverno blocks Argo CD itselfExclude platform namespaces from policies

What To Put in Your Portfolio

  • The Git repository with a clear README and an architecture diagram
  • A short screen recording or screenshots of: Argo CD tree, Grafana dashboard, canary abort, policy report
  • A written postmortem of the intentionally bad release, with timeline and action items
  • A list of trade-offs you chose (kind vs cloud, Audit vs Enforce, Loki vs OpenSearch)
  • Estimated cost if run in a real cloud and how you would harden it for production

Conclusion

This capstone connects the whole course: GitOps delivery with Argo CD, metrics and logs, SLO alerts, policy as code, canary releases with automated analysis and a runbook. It shows you can build and operate a platform, not just use tools.

Quick Summary: app-of-apps bootstrap; platform as Argo apps; kube-prometheus-stack + Loki; Kyverno audit then enforce; canary + analysis abort; SLO alert + runbook.

Get Newsletter

Subscibe to our newsletter and we will notify you about the newest updates on Edugators