DevOps Project 3: Production-Style Platform on Kubernetes with GitOps, Observability and Security
Project Goal
Create a small but realistic internal platform on Kubernetes. Everything is installed from a Git repository using Argo CD. Developers deploy only by merging pull requests. The platform provides metrics, logs, dashboards, alerts, security policies and safe canary releases.
text
GitHub: platform-config repo
| (Argo CD pulls)
v
kind cluster
|- platform namespace group: ingress-nginx, cert-manager, kube-prometheus-stack, loki, argo-rollouts, kyverno
|- app namespace: shop-api (Rollout, canary, ServiceMonitor, NetworkPolicy)
|- policies: no :latest, non-root, signed images (audit), default-deny network
'- alerts: SLO burn rate -> Alertmanager (webhook/Slack)
Resources: Run this on a machine with 4 CPUs and 8 GB RAM or more, or use a small cloud cluster. Create a kind cluster with the config below. Delete it when you are done.
Step 1: Create the Cluster
yaml
# kind-config.yaml
kind: Cluster
apiVersion: kind.x-k8s.io/v1alpha4
nodes:
- role: control-plane
kubeadmConfigPatches:
- |
kind: InitConfiguration
nodeRegistration:
kubeletExtraArgs:
node-labels: "ingress-ready=true"
extraPortMappings:
- { containerPort: 80, hostPort: 80, protocol: TCP }
- { containerPort: 443, hostPort: 443, protocol: TCP }
- role: worker
- role: worker
bash
kind create cluster --name platform --config kind-config.yaml
kubectl get nodes
Step 2: Config Repository Layout
text
platform-config/
bootstrap/
root-app.yaml <- the only thing you apply by hand
platform/ <- Argo CD Application manifests (one per component)
ingress-nginx.yaml
cert-manager.yaml
monitoring.yaml <- kube-prometheus-stack
loki.yaml
argo-rollouts.yaml
kyverno.yaml
policies.yaml <- points at policies/
policies/
disallow-latest.yaml require-nonroot.yaml default-deny-netpol.yaml
apps/
shop-api/
rollout.yaml service.yaml servicemonitor.yaml analysis.yaml networkpolicy.yaml
shop-api.yaml <- Argo CD Application for the app
Step 3: Install Argo CD and the Root Application
bash
kubectl create namespace argocd
kubectl apply -n argocd -f https://raw.githubusercontent.com/argoproj/argo-cd/stable/manifests/install.yaml
kubectl -n argocd rollout status deploy/argocd-server --timeout=180s
yaml
# bootstrap/root-app.yaml ("app of apps")
apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
name: root
namespace: argocd
spec:
project: default
source:
repoURL: https://github.com/YOUR_USER/platform-config.git
targetRevision: main
path: platform
destination:
server: https://kubernetes.default.svc
namespace: argocd
syncPolicy:
automated: { prune: true, selfHeal: true }
bash
kubectl apply -f bootstrap/root-app.yaml
kubectl -n argocd get applications -w
Step 4: Platform Components as Applications
Each file in platform/ is an Argo CD Application. Examples (replace <chart-version> with a current version from each project's repository):
yaml
# platform/ingress-nginx.yaml
apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
name: ingress-nginx
namespace: argocd
annotations: { argocd.argoproj.io/sync-wave: "-2" }
spec:
project: default
source:
repoURL: https://kubernetes.github.io/ingress-nginx
chart: ingress-nginx
targetRevision: "<chart-version>"
helm:
values: |
controller:
hostPort: { enabled: true }
service: { type: NodePort }
nodeSelector: { ingress-ready: "true" }
tolerations:
- { key: node-role.kubernetes.io/control-plane, operator: Equal, effect: NoSchedule }
destination: { server: https://kubernetes.default.svc, namespace: ingress-nginx }
syncPolicy:
automated: { prune: true, selfHeal: true }
syncOptions: [CreateNamespace=true]
---
# platform/monitoring.yaml
apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
name: monitoring
namespace: argocd
spec:
project: default
source:
repoURL: https://prometheus-community.github.io/helm-charts
chart: kube-prometheus-stack
targetRevision: "<chart-version>"
helm:
values: |
grafana:
adminPassword: change-me-lab-only
ingress: { enabled: true, ingressClassName: nginx, hosts: [grafana.localtest.me] }
prometheus:
prometheusSpec:
serviceMonitorSelectorNilUsesHelmValues: false # pick up all ServiceMonitors
destination: { server: https://kubernetes.default.svc, namespace: monitoring }
syncPolicy:
automated: { prune: true, selfHeal: true }
syncOptions: [CreateNamespace=true, ServerSideApply=true]
localtest.me: Names such as grafana.localtest.me resolve to 127.0.0.1 through public DNS, so you can use hostnames with Ingress on your laptop without editing /etc/hosts.
Create similar Applications for loki (chart loki-stack or the current Loki chart with Promtail or Alloy), argo-rollouts, kyverno and cert-manager. Add Loki as a Grafana data source through the monitoring chart values (grafana.additionalDataSources).
Step 5: Security Policies as Code
yaml
# policies/disallow-latest.yaml
apiVersion: kyverno.io/v1
kind: ClusterPolicy
metadata: { name: disallow-latest-tag }
spec:
validationFailureAction: Audit # change to Enforce after fixing violations
background: true
rules:
- name: require-pinned-tag
match: { any: [{ resources: { kinds: [Pod] } }] }
validate:
message: "Use a specific image tag, not :latest"
pattern:
spec:
containers:
- image: "!*:latest"
---
# policies/require-nonroot.yaml
apiVersion: kyverno.io/v1
kind: ClusterPolicy
metadata: { name: require-run-as-nonroot }
spec:
validationFailureAction: Audit
rules:
- name: run-as-non-root
match:
any:
- resources:
kinds: [Pod]
namespaces: [shop]
validate:
message: "Containers must set runAsNonRoot: true"
pattern:
spec:
containers:
- securityContext: { runAsNonRoot: true }
---
# apps/shop-api/networkpolicy.yaml
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata: { name: default-deny-ingress, namespace: shop }
spec:
podSelector: {}
policyTypes: [Ingress]
---
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata: { name: allow-ingress-controller, namespace: shop }
spec:
podSelector: { matchLabels: { app: shop-api } }
ingress:
- from:
- namespaceSelector: { matchLabels: { kubernetes.io/metadata.name: ingress-nginx } }
- namespaceSelector: { matchLabels: { kubernetes.io/metadata.name: monitoring } }
CNI note: kind's default network plugin does not enforce NetworkPolicy. Install Calico or Cilium in the cluster (create the cluster with disableDefaultCNI: true) if you want enforcement, otherwise the policies are applied but not enforced.
Step 6: The Application with Canary and Analysis
yaml
# apps/shop-api/rollout.yaml
apiVersion: argoproj.io/v1alpha1
kind: Rollout
metadata: { name: shop-api, namespace: shop }
spec:
replicas: 4
selector: { matchLabels: { app: shop-api } }
template:
metadata: { labels: { app: shop-api } }
spec:
containers:
- name: api
image: ghcr.io/YOUR_USER/shop-api:1.0.0
ports: [{ name: http, containerPort: 8000 }]
securityContext: { runAsNonRoot: true, allowPrivilegeEscalation: false }
readinessProbe: { httpGet: { path: /health, port: http } }
resources: { requests: { cpu: 50m, memory: 64Mi }, limits: { memory: 128Mi } }
strategy:
canary:
canaryService: shop-api-canary
stableService: shop-api
trafficRouting: { nginx: { stableIngress: shop-api } }
steps:
- setWeight: 20
- pause: { duration: 1m }
- analysis: { templates: [{ templateName: success-rate }] }
- setWeight: 60
- pause: { duration: 1m }
---
# apps/shop-api/servicemonitor.yaml
apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata: { name: shop-api, namespace: shop }
spec:
selector: { matchLabels: { app: shop-api } }
endpoints: [{ port: http, path: /metrics, interval: 15s }]
Use the instrumented version of the API from the monitoring lesson (it exposes /metrics). Create the shop-api and shop-api-canary Services, an Ingress for shop.localtest.me, and an AnalysisTemplate named success-rate that queries Prometheus for the success ratio.
Step 7: SLO Alerts and Runbook
yaml
# apps/shop-api/alerts.yaml
apiVersion: monitoring.coreos.com/v1
kind: PrometheusRule
metadata: { name: shop-api-slo, namespace: shop, labels: { release: monitoring } }
spec:
groups:
- name: shop-api-slo
rules:
- alert: ShopApiErrorBudgetFastBurn
expr: |
(sum(rate(http_requests_total{namespace="shop",status=~"5.."}[5m])) / sum(rate(http_requests_total{namespace="shop"}[5m]))) > (14.4 * 0.001)
for: 2m
labels: { severity: page }
annotations:
summary: "shop-api is burning its 99.9 percent SLO budget quickly"
runbook_url: https://github.com/YOUR_USER/platform-config/blob/main/runbooks/shop-api-errors.md
text
# runbooks/shop-api-errors.md
## Symptom: ShopApiErrorBudgetFastBurn
1. Open Grafana dashboard "shop-api" and check error rate by path.
2. Check the rollout: kubectl argo rollouts get rollout shop-api -n shop
3. If a release is in progress or was recent: kubectl argo rollouts abort shop-api -n shop, then revert the commit in Git.
4. Look at logs in Grafana Explore (Loki): {namespace="shop"} |= "ERROR"
5. Check dependencies and node health. Escalate to the platform team if unresolved after 15 minutes.
Step 8: End-to-End Demo Script
git push a change to the config repo that bumps the image from 1.0.0 to 1.1.0
- Watch Argo CD sync:
argocd app get shop-api or the UI
- Watch the canary:
kubectl argo rollouts get rollout shop-api -n shop --watch
- Generate traffic:
while true; do curl -s shop.localtest.me/products > /dev/null; done
- Open Grafana and see request rate and latency for stable and canary
- Release a bad version (return HTTP 500 for 30 percent of requests). The analysis fails and the rollout aborts automatically
- Show the alert firing in Alertmanager and follow the runbook
- Try to deploy a Pod with
image: nginx:latest into the shop namespace and show the Kyverno audit report (kubectl get policyreport -A)
- Delete a Deployment manually with kubectl and watch Argo CD self-heal it back
Troubleshooting Tips
| Problem | Fix |
Applications stay OutOfSync for CRDs | Use ServerSideApply=true and sync waves so CRDs install before resources that use them |
| Prometheus does not see the ServiceMonitor | Check the release label and serviceMonitorSelector of the Prometheus resource |
| Ingress returns 404 | Check ingressClassName, host name and that the controller Pod is running on the node with the port mapping |
| Out of memory on laptop | Disable Loki or lower replica counts and retention in the chart values |
| Kyverno blocks Argo CD itself | Exclude platform namespaces from policies |
What To Put in Your Portfolio
- The Git repository with a clear README and an architecture diagram
- A short screen recording or screenshots of: Argo CD tree, Grafana dashboard, canary abort, policy report
- A written postmortem of the intentionally bad release, with timeline and action items
- A list of trade-offs you chose (kind vs cloud, Audit vs Enforce, Loki vs OpenSearch)
- Estimated cost if run in a real cloud and how you would harden it for production
Conclusion
This capstone connects the whole course: GitOps delivery with Argo CD, metrics and logs, SLO alerts, policy as code, canary releases with automated analysis and a runbook. It shows you can build and operate a platform, not just use tools.
Quick Summary: app-of-apps bootstrap; platform as Argo apps; kube-prometheus-stack + Loki; Kyverno audit then enforce; canary + analysis abort; SLO alert + runbook.