Troubleshooting AWS Applications: Common Errors, Throttling, Timeouts, Permissions and a Debugging Playbook
A Debugging Method
- Define the symptom: what exactly fails, since when, for whom, what changed (deployment, config, traffic)
- Find the layer: client, edge (CloudFront/WAF), API, compute, data store, network, permissions
- Check dashboards and alarms: errors, latency, throttles, saturation
- Read logs (CloudWatch Logs Insights) and traces (X-Ray) for the failing request id
- Check recent changes in CloudTrail and CloudFormation events
- Form a hypothesis and test one change at a time
- Fix, verify, add an alarm or test so it is caught earlier next time
Lambda
| Symptom | Likely cause | Fix |
|---|---|---|
Task timed out after 3.00 seconds | Timeout too low or slow dependency (cold start, DB, external API, VPC without NAT) | Raise timeout and memory, add client timeouts, check VPC routes and endpoints, trace calls |
Runtime.ImportModuleError / Cannot find module | Wrong handler path, missing dependency in the package or layer, wrong architecture build | Check handler string, package contents, build with sam build --use-container |
AccessDeniedException calling another service | Execution role lacks permission (or a KMS permission) | Add the exact action and resource to the execution role |
Caller sees AccessDenied invoking Lambda | Missing resource policy permission for the caller (S3, API Gateway, SNS...) | aws lambda add-permission |
TooManyRequestsException (429), Throttles metric | Account or reserved concurrency exhausted, burst limit | Raise limits, remove too-low reserved concurrency, smooth with SQS, provisioned concurrency |
Out of memory (Runtime exited ... signal: killed) | Memory too low or large payload in memory | Increase memory, stream data, check REPORT Max Memory Used |
| Function works when invoked manually but not from the event | Event shape differs, wrong trigger, permission, filter | Log the event, check trigger configuration and CloudWatch logs |
| Duplicate processing | At-least-once delivery, retries | Make handler idempotent (conditional writes, idempotency keys) |
| Async invocation lost events | Retries exhausted, no destination | Configure DLQ / on-failure destination |
| VPC function cannot reach internet or AWS APIs | Private subnet without NAT or endpoint | Add NAT gateway or VPC endpoints |
| Stream (Kinesis/DynamoDB) processing stuck | A bad record fails the batch repeatedly | BisectBatchOnFunctionError, max retries, on-failure destination, check IteratorAge |
API Gateway
| Code | Check |
|---|---|
| 403 Missing Authentication Token | Wrong URL or method or stage not deployed (REST). Resource does not exist |
| 403 Forbidden | Authorizer deny, resource policy, WAF block, missing API key |
| 401 | Token missing/expired/wrong audience |
| 429 | Throttling or usage plan quota. Add backoff, raise limits |
| 502 | Malformed Lambda response (wrong proxy format) or Lambda crash |
| 504 | Integration slower than the timeout (29 s default). Use async patterns |
| CORS error in browser | Missing CORS headers on the response (also on 4xx/5xx) or OPTIONS route |
DynamoDB
| Symptom | Cause and fix |
|---|---|
ProvisionedThroughputExceededException | Over capacity or hot partition. Use on-demand, adaptive capacity, better keys, caching (DAX), SDK backoff, spread traffic |
ValidationException | Reserved word without #name, wrong types, missing key, item over 400 KB, empty string key value |
ConditionalCheckFailedException | Condition not met. This is often expected (duplicate or concurrent update) |
TransactionCanceledException | Read CancellationReasons for the failing item |
| Query returns fewer items than expected | 1 MB page limit: use LastEvaluatedKey. Filter expressions run after reading |
| Stale read right after write | Eventual consistency. Use ConsistentRead=True (not on GSIs) |
| GSI write throttling slows the table | Under-provisioned GSI capacity |
Messaging and Streams
| Symptom | Cause and fix |
|---|---|
| SQS messages processed twice | Visibility timeout shorter than processing time, at-least-once delivery. Increase timeout, idempotent consumer |
| Messages go to DLQ unexpectedly | maxReceiveCount too low, consumer error, poison message |
| Queue grows | Consumers too slow or failing. Scale consumers, check Lambda concurrency and errors |
| SNS to SQS no delivery | Missing queue policy allowing the topic, wrong filter policy |
Kinesis ProvisionedThroughputExceeded | Shard limit or hot key. More shards, better partition key, backoff, on-demand mode |
| Kinesis consumer lag | High IteratorAge. Increase parallelization factor, shards, optimize consumer |
Permissions: S3, KMS and Cross-Account
- **S3
AccessDeniedon an existing object: check IAM, bucket policy (explicit deny), Block Public Access, Organizations SCP, KMS key permissions** (kms:Decrypt), VPC endpoint policy, and object ownership for cross-account uploads - 404 vs 403: S3 returns 403 instead of 404 when you lack
s3:ListBucket - **
kms:DecryptAccessDenied**: key policy and IAM must both allow cross-account use - Wrong identity: run
aws sts get-caller-identity. Remember the role the service uses (task role vs execution role, Lambda execution role) - Decode
Encoded authorization failure messagewithsts decode-authorization-message
Containers (ECS and EKS)
| Symptom | Check |
|---|---|
Task stops immediately: Essential container exited | Container logs (awslogs), exit code, command, environment variables, memory limit (exit 137 = killed for memory) |
CannotPullContainerError | ECR permissions on the task execution role, no route to ECR (NAT or VPC endpoints), wrong image tag |
| Service keeps replacing tasks | Failing ALB health check (path, port, security group, grace period too short) |
ResourceInitializationError for secrets | Execution role cannot read the secret or network cannot reach Secrets Manager |
Pod CrashLoopBackOff (EKS) | kubectl logs, kubectl describe pod, resources, probes, IRSA role |
Network
- Timeout (no response) is usually network: security group, NACL, route table, missing NAT, wrong subnet, DNS
- Connection refused means the host is reachable but nothing listens on that port
- Use VPC Flow Logs (
REJECTlines) and Reachability Analyzer to see where traffic stops - Security group chain: ALB SG to app SG to DB SG on the right ports
- Resolve names with the VPC DNS (enable DNS hostnames/support for endpoints)
CloudFormation and Deployments
| Symptom | Cause and fix |
|---|---|
ROLLBACK_COMPLETE on create | A resource failed. Read stack events for the first CREATE_FAILED. Delete the stack and redeploy |
UPDATE_ROLLBACK_FAILED | Resource changed manually or dependency issue. Fix and continue update rollback |
Requires capabilities: [CAPABILITY_IAM] | Add the capability flag |
| Resource name already exists | Remove hard-coded names or delete the old resource |
| Export in use | Another stack imports it. Update or remove the importer first |
| CodeDeploy failed on ValidateService | Check hook script output in /opt/codedeploy-agent/deployment-root/.../logs/scripts.log |
| Beanstalk deployment fails | eb logs, .ebextensions errors, health, immutable rollback |
Cost and Capacity Surprises
- Bill spike: Cost Explorer by service and tag, look for NAT data transfer, CloudWatch Logs ingestion, Lambda loops (S3 trigger recursion), idle resources
- Service quota errors (
LimitExceededException): check Service Quotas and request an increase early
Exam-Style Scenarios
| Scenario | Answer |
|---|---|
| Lambda writes to DynamoDB but throttles during bursts | Use on-demand mode or increase capacity, SDK retries with backoff, buffer with SQS |
| Users get intermittent 504 from API Gateway for long reports | Return 202 and process asynchronously |
| Encrypted S3 reads fail for a new role | Add kms:Decrypt on the key policy and role |
| ECS tasks cannot pull images in private subnets | Add NAT or ECR and S3 VPC endpoints |
| Lambda in VPC cannot call DynamoDB | Add a DynamoDB gateway endpoint or NAT |
| Duplicate orders created when SQS redelivers | Idempotent consumer with a conditional write |
| Need to know who changed a security group | CloudTrail event history |
Practice Tasks
- Break a Lambda function's role on purpose and trace the error to the missing permission
- Reproduce an API Gateway 502 and fix the response format
- Force DynamoDB throttling and compare provisioned and on-demand behavior
- Find a REJECT in VPC Flow Logs for a blocked port
- Practise with the exam-style table: answer without looking, then check
Conclusion
Use a method: define the symptom, locate the layer, read metrics, logs and traces, check recent changes, test one hypothesis at a time. Know the signature of common failures: Lambda timeouts and throttles, API Gateway 502 vs 504, DynamoDB hot keys and 1 MB pages, SQS visibility problems, KMS and policy denials, ECS pull and health check issues, network timeouts, and CloudFormation rollbacks.
Quick Summary: method; Lambda; API Gateway; DynamoDB; messaging; permissions; containers; network; CloudFormation; cost; exam scenarios.

