Project 2: Event-Driven Order Processing with SNS, SQS, EventBridge, Step Functions and Dead-Letter Queues

AWS Developer Associate 70 min min read Updated: Nov 17, 2026 Advanced
Project 2: Event-Driven Order Processing with SNS, SQS, EventBridge, Step Functions and Dead-Letter Queues
Advanced Topic 2 of 3

Goal

When a customer places an order the system must reserve stock, charge payment, create a shipment and notify the customer. Parts can fail or be slow, so we use events, queues and a workflow instead of one long synchronous call.

text POST /orders ─► API Gateway ─► OrderApi Lambda ─► DynamoDB Orders (status=PENDING, transactional write) └─► EventBridge bus "shop": OrderCreated EventBridge rules: OrderCreated ─► Step Functions "OrderWorkflow" (orchestrates the critical path) OrderCreated ─► SNS "order-events" ─► SQS analytics ─► Lambda (reports) (fan-out, non-critical) ─► SQS email ─► Lambda (notifications) OrderWorkflow: ReserveStock → ChargePayment → CreateShipment → MarkConfirmed on failure: RefundPayment ← ReleaseStock ← MarkFailed (saga compensation) Every queue has a DLQ + alarm. Every consumer is idempotent.

Design Decisions

DecisionReason
API returns 202 Accepted quicklyCustomer does not wait for the full workflow
EventBridge as the main busContent-based routing, many targets, archive and replay
Step Functions for stock → payment → shipmentOrdered steps, retries, branches and compensation visible in one place
SNS + SQS for side effectsEach consumer has its own queue, own speed and own DLQ
DynamoDB transaction for order + outboxNo lost event if the process fails after saving
Idempotency keys everywhereAt-least-once delivery means duplicates will occur

Step 1: Create the Order and Publish the Event Reliably

A common bug: save to the database, then crash before publishing the event, and the order is stuck. The outbox pattern writes both in one transaction, and a stream publishes later.

python import json, os, uuid, time, boto3 ddb = boto3.client("dynamodb") TABLE = os.environ["TABLE"] def handler(event, context): body = json.loads(event["body"]) order_id = body.get("orderId") or str(uuid.uuid4()) now = int(time.time()) detail = {"orderId": order_id, "customerId": body["customerId"], "items": body["items"], "total": body["total"]} try: ddb.transact_write_items(TransactItems=[ {"Put": {"TableName": TABLE, "ConditionExpression": "attribute_not_exists(pk)", "Item": {"pk": {"S": f"ORDER#{order_id}"}, "sk": {"S": "HEADER"}, "status": {"S": "PENDING"}, "createdAt": {"N": str(now)}, "customerId": {"S": body["customerId"]}, "total": {"N": str(body["total"])}}}}, {"Put": {"TableName": TABLE, # outbox item: a stream publishes it to EventBridge "Item": {"pk": {"S": f"OUTBOX#{order_id}"}, "sk": {"S": "OrderCreated"}, "detail": {"S": json.dumps(detail)}}}}]) except ddb.exceptions.TransactionCanceledException: pass # duplicate request: same order id, safe to return 202 again return {"statusCode": 202, "body": json.dumps({"orderId": order_id, "status": "PENDING"})}
python # Outbox publisher: DynamoDB Stream -> Lambda -> EventBridge import json, boto3 from boto3.dynamodb.types import TypeDeserializer eb = boto3.client("events"); td = TypeDeserializer() def handler(event, context): entries, failures = [], [] for rec in event["Records"]: if rec["eventName"] != "INSERT": continue img = {k: td.deserialize(v) for k, v in rec["dynamodb"]["NewImage"].items()} if img["pk"].startswith("OUTBOX#"): entries.append({"EventBusName": "shop", "Source": "shop.orders", "DetailType": img["sk"], "Detail": img["detail"]}) for i in range(0, len(entries), 10): # PutEvents accepts 10 entries r = eb.put_events(Entries=entries[i:i+10]) if r["FailedEntryCount"]: raise RuntimeError("PutEvents failed") # the stream retries the batch

Step 2: EventBridge Rules

yaml ToWorkflow: Type: AWS::Events::Rule Properties: EventBusName: !Ref ShopBus EventPattern: {source: [shop.orders], detail-type: [OrderCreated]} Targets: - Id: workflow Arn: !Ref OrderWorkflow RoleArn: !GetAtt EventsToSfnRole.Arn DeadLetterConfig: {Arn: !GetAtt EventsDlq.Arn} RetryPolicy: {MaximumRetryAttempts: 8, MaximumEventAgeInSeconds: 3600} ToFanout: Type: AWS::Events::Rule Properties: EventBusName: !Ref ShopBus EventPattern: {source: [shop.orders], detail-type: [OrderCreated]} Targets: [{Id: sns, Arn: !Ref OrderEventsTopic}]

Step 3: Fan-Out with SNS and SQS

yaml OrderEventsTopic: {Type: AWS::SNS::Topic} EmailQueue: Type: AWS::SQS::Queue Properties: VisibilityTimeout: 180 # 6 x the 30 s Lambda timeout RedrivePolicy: {deadLetterTargetArn: !GetAtt EmailDlq.Arn, maxReceiveCount: 4} EmailDlq: {Type: AWS::SQS::Queue, Properties: {MessageRetentionPeriod: 1209600}} EmailSub: Type: AWS::SNS::Subscription Properties: {TopicArn: !Ref OrderEventsTopic, Protocol: sqs, Endpoint: !GetAtt EmailQueue.Arn, RawMessageDelivery: true} EmailQueuePolicy: Type: AWS::SQS::QueuePolicy Properties: Queues: [!Ref EmailQueue] PolicyDocument: Statement: [{Effect: Allow, Principal: {Service: sns.amazonaws.com}, Action: sqs:SendMessage, Resource: !GetAtt EmailQueue.Arn, Condition: {ArnEquals: {'aws:SourceArn': !Ref OrderEventsTopic}}}]
python # Idempotent notification consumer with partial batch failure import json, boto3 from botocore.exceptions import ClientError t = boto3.resource("dynamodb").Table(os.environ["DEDUP_TABLE"]) def handler(event, context): failures = [] for rec in event["Records"]: try: msg = json.loads(rec["body"]) try: # claim this order for email; second attempt sees the item and skips t.put_item(Item={"pk": f"email#{msg['detail']['orderId']}", "expiresAt": int(time.time()) + 86400}, ConditionExpression="attribute_not_exists(pk)") except ClientError as e: if e.response["Error"]["Code"] == "ConditionalCheckFailedException": continue raise send_email(msg) except Exception: failures.append({"itemIdentifier": rec["messageId"]}) return {"batchItemFailures": failures}

Step 4: The Order Workflow (Saga)

json { "StartAt": "ReserveStock", "States": { "ReserveStock": {"Type": "Task", "Resource": "arn:aws:states:::lambda:invoke", "Parameters": {"FunctionName": "${ReserveStockFn}", "Payload.$": "$.detail"}, "ResultPath": "$.stock", "ResultSelector": {"ok.$": "$.Payload.ok"}, "Retry": [{"ErrorEquals": ["Lambda.ServiceException", "Lambda.TooManyRequestsException"], "IntervalSeconds": 2, "MaxAttempts": 4, "BackoffRate": 2}], "Catch": [{"ErrorEquals": ["States.ALL"], "ResultPath": "$.error", "Next": "MarkFailed"}], "Next": "ChargePayment"}, "ChargePayment": {"Type": "Task", "Resource": "arn:aws:states:::lambda:invoke", "Parameters": {"FunctionName": "${ChargeFn}", "Payload": {"orderId.$": "$.detail.orderId", "amount.$": "$.detail.total"}}, "ResultPath": "$.payment", "Retry": [{"ErrorEquals": ["PaymentTimeout"], "IntervalSeconds": 3, "MaxAttempts": 3, "BackoffRate": 2}], "Catch": [{"ErrorEquals": ["States.ALL"], "ResultPath": "$.error", "Next": "ReleaseStock"}], "Next": "CreateShipment"}, "CreateShipment": {"Type": "Task", "Resource": "arn:aws:states:::lambda:invoke", "Parameters": {"FunctionName": "${ShipFn}", "Payload.$": "$.detail"}, "ResultPath": "$.shipment", "Catch": [{"ErrorEquals": ["States.ALL"], "ResultPath": "$.error", "Next": "RefundPayment"}], "Next": "MarkConfirmed"}, "MarkConfirmed": {"Type": "Task", "Resource": "arn:aws:states:::dynamodb:updateItem", "Parameters": {"TableName": "${Table}", "Key": {"pk": {"S.$": "States.Format('ORDER#{}', $.detail.orderId)"}, "sk": {"S": "HEADER"}}, "UpdateExpression": "SET #s = :c", "ExpressionAttributeNames": {"#s": "status"}, "ExpressionAttributeValues": {":c": {"S": "CONFIRMED"}}}, "End": true}, "RefundPayment": {"Type": "Task", "Resource": "arn:aws:states:::lambda:invoke", "Parameters": {"FunctionName": "${RefundFn}", "Payload.$": "$.detail"}, "ResultPath": null, "Next": "ReleaseStock"}, "ReleaseStock": {"Type": "Task", "Resource": "arn:aws:states:::lambda:invoke", "Parameters": {"FunctionName": "${ReleaseFn}", "Payload.$": "$.detail"}, "ResultPath": null, "Next": "MarkFailed"}, "MarkFailed": {"Type": "Task", "Resource": "arn:aws:states:::dynamodb:updateItem", "Parameters": {"TableName": "${Table}", "Key": {"pk": {"S.$": "States.Format('ORDER#{}', $.detail.orderId)"}, "sk": {"S": "HEADER"}}, "UpdateExpression": "SET #s = :f", "ExpressionAttributeNames": {"#s": "status"}, "ExpressionAttributeValues": {":f": {"S": "FAILED"}}}, "End": true} } }
  • Each activity function must also be idempotent (use the order id as the idempotency key with payment and shipping providers)
  • Retry only transient errors. Catch business failures and run compensation
  • Use a Standard workflow for auditability and the visual history. Use Express only for very high volume short flows

Step 5: Failure Handling

FailureWhat happens
Lambda throttled or transient errorStep Functions retries with backoff
Payment declinedCatch → release stock → mark order FAILED, customer notified
Email provider downMessage retried by SQS; after 4 receives goes to DLQ; alarm fires
EventBridge cannot reach the targetRetries up to the policy, then target DLQ
Bad deployment of a consumerMessages wait in the queue; fix and the backlog drains
bash # Look at failed messages, fix the bug, then move them back aws sqs receive-message --queue-url $EMAIL_DLQ --max-number-of-messages 5 aws sqs start-message-move-task --source-arn $EMAIL_DLQ_ARN --destination-arn $EMAIL_QUEUE_ARN # DLQ redrive # Replay events from an EventBridge archive aws events start-replay --replay-name fix-nov17 --event-source-arn $ARCHIVE_ARN \ --destination '{"Arn":"'$BUS_ARN'"}' --event-start-time 2026-11-17T08:00:00Z --event-end-time 2026-11-17T10:00:00Z

Step 6: Monitoring

  • Alarms on every DLQ depth greater than 0
  • ApproximateAgeOfOldestMessage for each queue (consumers falling behind)
  • Step Functions ExecutionsFailed, ExecutionThrottled, ExecutionTime
  • EventBridge FailedInvocations and ThrottledRules
  • X-Ray tracing across API, Lambdas and Step Functions
  • A business metric: orders confirmed per minute versus created

Test Plan

  1. Unit test each Lambda with moto
  2. Run the workflow with test input in the Step Functions console and inject failures (a Lambda that raises PaymentTimeout, one that raises a business error)
  3. Send the same OrderCreated event twice and confirm there is one email and one charge
  4. Kill the email Lambda and confirm messages queue and drain later
  5. Load test 1,000 orders and watch queue age and throttles

Extensions

  1. Add a wait for callback step for manual fraud review using waitForTaskToken
  2. Add EventBridge Pipes from the stream to the bus instead of the outbox Lambda
  3. Add Kinesis Firehose to store all events in S3 for Athena analytics
  4. Add FIFO queue for strict per-customer ordering
  5. Add X-Ray groups and a CloudWatch dashboard

Conclusion

You built a decoupled order pipeline: transactional order plus outbox, EventBridge routing, a Step Functions saga with retries and compensation for the critical path, SNS and SQS fan-out for side effects, idempotent consumers with partial batch failures, DLQs with redrive, archive replay, and alarms on queue age and failures.

Quick Summary: goal; design decisions; outbox; EventBridge rules; fan-out; consumer; saga; failure table; redrive/replay; monitoring; test plan.

Get Newsletter

Subscibe to our newsletter and we will notify you about the newest updates on Edugators