Amazon CloudWatch for Developers: Metrics, Logs, Logs Insights, Embedded Metric Format, Alarms and Dashboards
The Observability Trio
| Signal | Service | Answers |
| Metrics | CloudWatch Metrics | How much? How fast? (numbers over time) |
| Logs | CloudWatch Logs | What happened? (text and JSON events) |
| Traces | AWS X-Ray / OpenTelemetry | Where did the time go across services? |
| Events / alerts | Alarms, EventBridge, SNS | Tell me when something is wrong |
Metrics
| Term | Meaning |
| Namespace | Container for metrics, for example AWS/EC2, AWS/Lambda, or your own Shop/Orders |
| Metric | A time-ordered set of data points, such as CPUUtilization |
| Dimension | Name-value pair that identifies a series: InstanceId=i-0abc. Each unique combination of dimensions is a separate metric |
| Statistic | Sum, Average, Minimum, Maximum, SampleCount, percentiles (p99) |
| Period | Aggregation window in seconds (60, or multiples; 1 to 30 seconds for high-resolution) |
| Unit | Count, Seconds, Bytes, Percent... |
- Standard resolution = 1 minute granularity. High resolution custom metrics = down to 1 second
- Retention: 1-second data for 3 hours, 1-minute data for 15 days, 5-minute for 63 days, 1-hour for 455 days (15 months). Older data rolls up automatically
- EC2 default metrics are every 5 minutes. Detailed monitoring gives 1 minute (extra cost)
- EC2 does not publish memory or disk usage. Use the CloudWatch agent for those
- Many services publish metrics for free (Lambda:
Invocations, Errors, Duration, Throttles, ConcurrentExecutions)
Custom Metrics
python
import boto3, datetime
cw = boto3.client("cloudwatch")
cw.put_metric_data(
Namespace="Shop/Orders",
MetricData=[{
"MetricName": "OrdersPlaced", "Dimensions": [{"Name": "Env", "Value": "prod"}],
"Timestamp": datetime.datetime.utcnow(), "Value": 1, "Unit": "Count",
"StorageResolution": 1, # 1 = high resolution (1 second), 60 = standard
}])
Watch dimension explosion Dimensions such as userId or orderId create a new metric for each value, which is slow and expensive. Use low-cardinality dimensions (environment, service, operation) and put ids in logs.
Embedded Metric Format (EMF)
PutMetricData is an API call (cost and latency). EMF lets you write one structured JSON log line that CloudWatch automatically turns into metrics. It is ideal for Lambda (stdout goes to logs), and the log line also keeps the high-cardinality details for debugging.
python
import json, time
def emit(order_id, amount, ms):
print(json.dumps({
"_aws": {"Timestamp": int(time.time()*1000),
"CloudWatchMetrics": [{"Namespace": "Shop/Orders",
"Dimensions": [["Env"]],
"Metrics": [{"Name": "OrderAmount", "Unit": "None"}, {"Name": "ProcessingTime", "Unit": "Milliseconds"}]}]},
"Env": "prod", "OrderAmount": amount, "ProcessingTime": ms,
"orderId": order_id # not a dimension: stays in the log only
}))
CloudWatch Logs
| Concept | Meaning |
| Log group | Collection of streams with shared retention, encryption and access (for Lambda /aws/lambda/NAME) |
| Log stream | Sequence of events from one source (one Lambda environment, one instance) |
| Retention | Default never expires. Set it (1 day to 10 years). Unbounded logs cost money |
| Encryption | Optional KMS key on the log group |
| Classes | Standard and Infrequent Access (cheaper, fewer features) |
- Lambda needs
logs:CreateLogGroup, logs:CreateLogStream, logs:PutLogEvents (the basic execution role has them)
- On EC2 and on premises send logs with the CloudWatch agent. ECS/Fargate uses the
awslogs driver. API Gateway, VPC Flow Logs, CloudTrail and Route 53 can send logs here
- Structured JSON logs make queries much easier
Metric Filters
A metric filter counts patterns in log events and publishes a metric (for example the number of ERROR lines), which you can alarm on.
bash
aws logs put-metric-filter --log-group-name /aws/lambda/orders --filter-name errors \
--filter-pattern '{ $.level = "ERROR" }' \
--metric-transformations metricName=AppErrors,metricNamespace=Shop/Orders,metricValue=1,defaultValue=0
- Filters apply to new events only, not to old ones
- Pattern examples:
ERROR, [ip, user, ts, request, status=5*, size] for space-delimited, { $.status = 500 } for JSON
Logs Insights
text
# Top 10 slowest requests in the last hour
fields @timestamp, orderId, duration
| filter ispresent(duration)
| sort duration desc
| limit 10
# Error count per 5 minutes
filter level = "ERROR"
| stats count() as errors by bin(5m)
# Lambda cold starts and memory use
filter @type = "REPORT"
| stats count(*) as invocations, avg(@duration) as avg_ms, max(@maxMemoryUsed/1000000) as max_mb,
sum(ispresent(@initDuration)) as coldStarts by bin(1h)
- Pick the log groups and time range, then query. You pay per GB scanned, so narrow the range
- Commands:
fields, filter, stats, sort, limit, parse (extract fields with a pattern), dedup, display
- Lambda's
REPORT line has duration, billed duration, memory used and initDuration (cold start)
Subscription Filters and Exports
- Subscription filters stream log events in real time to Kinesis Data Streams/Firehose, Lambda or OpenSearch for processing or central storage
- Cross-account log sharing uses subscription destinations
- Export old logs to S3 (
CreateExportTask) for cheap long-term storage
Alarms
bash
aws cloudwatch put-metric-alarm --alarm-name orders-lambda-errors \
--namespace AWS/Lambda --metric-name Errors --dimensions Name=FunctionName,Value=orders \
--statistic Sum --period 60 --evaluation-periods 3 --datapoints-to-alarm 2 \
--threshold 5 --comparison-operator GreaterThanOrEqualToThreshold \
--treat-missing-data notBreaching --alarm-actions arn:aws:sns:ap-south-1:111111111111:oncall
| Topic | Detail |
| States | OK, ALARM, INSUFFICIENT_DATA |
| M out of N | datapoints-to-alarm of evaluation-periods reduces flapping |
| Missing data | notBreaching, breaching, ignore, missing. Choose carefully for sparse metrics |
| Actions | Notify SNS, run Auto Scaling policy, EC2 actions (stop, terminate, reboot, recover), Systems Manager OpsItem, create incident |
| Composite alarms | Combine alarms with AND/OR/NOT to cut noise (page only when errors and latency are bad) |
| Anomaly detection alarms | Dynamic threshold from a machine learning band |
| Metric math | Compute rates and ratios: errors / invocations * 100 |
Dashboards, Synthetics and Application Signals
- Dashboards give a single view across Regions and accounts (widgets for metrics, logs, alarms, text)
- CloudWatch Synthetics runs canary scripts (Node.js or Python with Puppeteer or Selenium) on a schedule to test endpoints and user flows
- Application Signals and Container/Lambda Insights provide service-level maps, SLOs and ready dashboards
- CloudWatch RUM measures real user performance in browsers
- Contributor Insights shows top talkers (for example top IPs in a log)
What to Monitor in a Typical App
| Layer | Golden signals |
| API Gateway / ALB | 5XXError, 4XXError, Latency, Count, TargetResponseTime, unhealthy hosts |
| Lambda | Errors, Throttles, Duration p99, ConcurrentExecutions, IteratorAge (streams), DLQ depth |
| DynamoDB | ThrottledRequests, SystemErrors, ConsumedReadCapacityUnits, SuccessfulRequestLatency |
| SQS | ApproximateAgeOfOldestMessage, ApproximateNumberOfMessagesVisible, DLQ messages |
| RDS | CPUUtilization, DatabaseConnections, FreeStorageSpace, ReplicaLag |
| Business | Orders per minute, payment failures, signups |
Practice Tasks
- Publish a custom metric with
PutMetricData and graph it
- Rewrite it with EMF and compare
- Create a metric filter for ERROR logs and an alarm that emails you
- Write a Logs Insights query for Lambda cold starts
- Build a dashboard with three widgets and a composite alarm
Conclusion
CloudWatch stores metrics (dimensions define series, EC2 needs the agent for memory), logs (set retention, use JSON), and alarms. Use EMF for cheap custom metrics from logs, metric filters to count log patterns, Logs Insights to investigate, and alarms with M-out-of-N and composite logic to reduce noise.
Quick Summary: signals; metrics; custom metrics; EMF; logs; metric filters; Insights; subscription; alarms; dashboards; what to monitor.