Project 2: Patching and Fleet Management With Systems Manager, Maintenance Windows and Compliance Reporting
Goal
Patch a mixed dev and prod fleet every month with scan first, install later, reboot safely, report automatically and with no SSH access.
Design
| Item | Choice |
|---|---|
| Targeting | Tags: env=dev|prod, patch-group=web|app, os=al2023|windows |
| Baselines | corp-linux-security: Security + Bugfix, Critical/Important, auto-approve after 3 days. corp-windows: Critical + Security updates after 5 days |
| Schedule | Scan daily (State Manager) → Install on dev second Tuesday 22:00 → Install on prod the following Saturday 22:00 after dev passes |
| Safety | MaxConcurrency 10%, MaxErrors 1, ASG-backed servers drained by load balancer, snapshot or AMI before patching prod |
| Reporting | Patch compliance dashboard + Resource Data Sync → S3 → Athena + weekly SNS summary |
Step 1: Make Everything Managed
bash
aws ssm describe-instance-information --query 'InstanceInformationList[?PingStatus!=`Online`].[InstanceId,PingStatus]'
# Quick Setup (Host Management): keeps SSM agent updated, collects inventory, enables patch scans
# Or create an association manually:
aws ssm create-association --name AWS-UpdateSSMAgent --targets Key=InstanceIds,Values=* --schedule-expression "rate(14 days)"
Step 2: Baselines and Patch Groups
bash
aws ssm create-patch-baseline --name corp-linux-security --operating-system AMAZON_LINUX_2023 --approved-patches-compliance-level CRITICAL \
--approval-rules 'PatchRules=[{PatchFilterGroup={PatchFilters=[{Key=CLASSIFICATION,Values=[Security,Bugfix]},{Key=SEVERITY,Values=[Critical,Important]}]},ApproveAfterDays=3,ComplianceLevel=CRITICAL}]'
aws ssm register-patch-baseline-for-patch-group --baseline-id pb-0aaa --patch-group web
aws ssm get-patch-baseline-for-patch-group --patch-group web
aws ec2 create-tags --resources i-0abc --tags Key="Patch Group",Value=web
- Instances use the baseline of their **
Patch Grouptag** (enable tag use on instances through instance metadata tags if needed), or the default baseline for the OS - Use
RejectedPatchesfor a known bad patch and **ApprovedPatches** to force a hotfix
Step 3: Maintenance Windows
bash
WID=$(aws ssm create-maintenance-window --name patch-prod --schedule "cron(0 22 ? * SAT *)" --schedule-timezone "Asia/Kolkata" --duration 4 --cutoff 1 --no-allow-unassociated-targets --query WindowId --output text)
TID=$(aws ssm register-target-with-maintenance-window --window-id $WID --resource-type INSTANCE --targets Key=tag:env,Values=prod Key=tag:patch-group,Values=web --query WindowTargetId --output text)
aws ssm register-task-with-maintenance-window --window-id $WID --targets Key=WindowTargetIds,Values=$TID --task-arn AWS-RunPatchBaseline --task-type RUN_COMMAND \
--max-concurrency 10% --max-errors 1 --priority 1 --task-invocation-parameters '{"RunCommand":{"Parameters":{"Operation":["Install"],"RebootOption":["RebootIfNeeded"]},"ServiceRoleArn":"arn:aws:iam::111111111111:role/SSMMaintenanceRole","NotificationConfig":{"NotificationArn":"arn:aws:sns:ap-south-1:111111111111:ops-alerts","NotificationEvents":["Failed","TimedOut"],"NotificationType":"Invocation"}}}'
aws ssm describe-maintenance-window-executions --window-id $WID
Step 4: Protect Production
- Before: create an AMI or snapshot (Automation task priority 0 in the same window using
AWS-CreateImage) - During: rolling 10 percent at a time. For ASG instances use standby or rely on ALB health checks with deregistration delay
- After: a verification task runs a health check script (
curl -fsS localhost/health). Failures raise an OpsItem - Rollback option: replace the instance from the previous AMI (instance refresh) or restore the snapshot. Patches cannot be cleanly uninstalled for every OS
Step 5: Compliance and Reporting
bash
aws ssm describe-patch-group-state --patch-group web
aws ssm describe-instance-patch-states-for-patch-group --patch-group web --query 'InstancePatchStates[].[InstanceId,InstalledCount,MissingCount,FailedCount,Operation]' --output table
aws ssm list-compliance-summaries --filters Key=ComplianceType,Values=Patch
aws ssm create-resource-data-sync --sync-name fleet-inventory --s3-destination BucketName=ops-ssm-sync,Region=ap-south-1,SyncFormat=JsonSerDe
sql
-- Athena over the Resource Data Sync: instances with missing critical patches
SELECT resourceid, patchstate, count(*) AS n
FROM aws_complianceitem
WHERE compliancetype = 'Patch' AND status = 'NON_COMPLIANT' AND severity = 'CRITICAL'
GROUP BY resourceid, patchstate ORDER BY n DESC;
- Create an EventBridge rule on
Maintenance Window Execution State-changeandCommand Status-changefor alerts - Export a weekly CSV through a scheduled Lambda to S3 and email the summary
- Use Explorer for a cross-account dashboard and OpsCenter for failures
Step 6: Immutable Alternative
- For ASG fleets build a golden AMI weekly with EC2 Image Builder (patched by
update-linuxcomponent), test, publish AMI id to Parameter Store, then run instance refresh. Patch Manager is then used mostly for scan and compliance - This avoids long-lived drift and gives a quick rollback by switching the launch template back
Troubleshooting Patching
| Problem | Fix |
|---|---|
| Instance missing from the window | Not Online in SSM, tag mismatch, window target selection wrong |
Failed with no patches installed | No internet or repository access: allow Amazon Linux repositories through NAT or S3 endpoint, or configure a mirror |
| Patches installed but compliance shows non-compliant | Reboot pending, baseline different from expectation, scan not yet run (Scan after install) |
| Window never starts | Time zone and cron, window disabled, role permissions |
| Application broken after patch | Roll back through AMI or snapshot, pin package versions, test in dev first |
Conclusion
A safe patching program uses tags, baselines with approval delays, scans first and installs in rate-controlled maintenance windows (dev before prod), takes backups before changes, verifies health afterward, reports with compliance dashboards and Athena, and moves toward immutable AMIs for autoscaled fleets.
Quick Summary: goal; design; managed nodes; baselines; windows; protection; reporting; immutable approach; troubleshooting.

