Project 2: Event-Driven Order Processing with SNS, SQS, EventBridge, Step Functions and Dead-Letter Queues
Goal
When a customer places an order the system must reserve stock, charge payment, create a shipment and notify the customer. Parts can fail or be slow, so we use events, queues and a workflow instead of one long synchronous call.
text
POST /orders ─► API Gateway ─► OrderApi Lambda ─► DynamoDB Orders (status=PENDING, transactional write)
└─► EventBridge bus "shop": OrderCreated
EventBridge rules:
OrderCreated ─► Step Functions "OrderWorkflow" (orchestrates the critical path)
OrderCreated ─► SNS "order-events" ─► SQS analytics ─► Lambda (reports) (fan-out, non-critical)
─► SQS email ─► Lambda (notifications)
OrderWorkflow: ReserveStock → ChargePayment → CreateShipment → MarkConfirmed
on failure: RefundPayment ← ReleaseStock ← MarkFailed (saga compensation)
Every queue has a DLQ + alarm. Every consumer is idempotent.
Design Decisions
| Decision | Reason |
| API returns 202 Accepted quickly | Customer does not wait for the full workflow |
| EventBridge as the main bus | Content-based routing, many targets, archive and replay |
| Step Functions for stock → payment → shipment | Ordered steps, retries, branches and compensation visible in one place |
| SNS + SQS for side effects | Each consumer has its own queue, own speed and own DLQ |
| DynamoDB transaction for order + outbox | No lost event if the process fails after saving |
| Idempotency keys everywhere | At-least-once delivery means duplicates will occur |
Step 1: Create the Order and Publish the Event Reliably
A common bug: save to the database, then crash before publishing the event, and the order is stuck. The outbox pattern writes both in one transaction, and a stream publishes later.
python
import json, os, uuid, time, boto3
ddb = boto3.client("dynamodb")
TABLE = os.environ["TABLE"]
def handler(event, context):
body = json.loads(event["body"])
order_id = body.get("orderId") or str(uuid.uuid4())
now = int(time.time())
detail = {"orderId": order_id, "customerId": body["customerId"], "items": body["items"], "total": body["total"]}
try:
ddb.transact_write_items(TransactItems=[
{"Put": {"TableName": TABLE, "ConditionExpression": "attribute_not_exists(pk)",
"Item": {"pk": {"S": f"ORDER#{order_id}"}, "sk": {"S": "HEADER"}, "status": {"S": "PENDING"}, "createdAt": {"N": str(now)},
"customerId": {"S": body["customerId"]}, "total": {"N": str(body["total"])}}}},
{"Put": {"TableName": TABLE, # outbox item: a stream publishes it to EventBridge
"Item": {"pk": {"S": f"OUTBOX#{order_id}"}, "sk": {"S": "OrderCreated"}, "detail": {"S": json.dumps(detail)}}}}])
except ddb.exceptions.TransactionCanceledException:
pass # duplicate request: same order id, safe to return 202 again
return {"statusCode": 202, "body": json.dumps({"orderId": order_id, "status": "PENDING"})}
python
# Outbox publisher: DynamoDB Stream -> Lambda -> EventBridge
import json, boto3
from boto3.dynamodb.types import TypeDeserializer
eb = boto3.client("events"); td = TypeDeserializer()
def handler(event, context):
entries, failures = [], []
for rec in event["Records"]:
if rec["eventName"] != "INSERT": continue
img = {k: td.deserialize(v) for k, v in rec["dynamodb"]["NewImage"].items()}
if img["pk"].startswith("OUTBOX#"):
entries.append({"EventBusName": "shop", "Source": "shop.orders", "DetailType": img["sk"], "Detail": img["detail"]})
for i in range(0, len(entries), 10): # PutEvents accepts 10 entries
r = eb.put_events(Entries=entries[i:i+10])
if r["FailedEntryCount"]:
raise RuntimeError("PutEvents failed") # the stream retries the batch
Step 2: EventBridge Rules
yaml
ToWorkflow:
Type: AWS::Events::Rule
Properties:
EventBusName: !Ref ShopBus
EventPattern: {source: [shop.orders], detail-type: [OrderCreated]}
Targets:
- Id: workflow
Arn: !Ref OrderWorkflow
RoleArn: !GetAtt EventsToSfnRole.Arn
DeadLetterConfig: {Arn: !GetAtt EventsDlq.Arn}
RetryPolicy: {MaximumRetryAttempts: 8, MaximumEventAgeInSeconds: 3600}
ToFanout:
Type: AWS::Events::Rule
Properties:
EventBusName: !Ref ShopBus
EventPattern: {source: [shop.orders], detail-type: [OrderCreated]}
Targets: [{Id: sns, Arn: !Ref OrderEventsTopic}]
Step 3: Fan-Out with SNS and SQS
yaml
OrderEventsTopic: {Type: AWS::SNS::Topic}
EmailQueue:
Type: AWS::SQS::Queue
Properties:
VisibilityTimeout: 180 # 6 x the 30 s Lambda timeout
RedrivePolicy: {deadLetterTargetArn: !GetAtt EmailDlq.Arn, maxReceiveCount: 4}
EmailDlq: {Type: AWS::SQS::Queue, Properties: {MessageRetentionPeriod: 1209600}}
EmailSub:
Type: AWS::SNS::Subscription
Properties: {TopicArn: !Ref OrderEventsTopic, Protocol: sqs, Endpoint: !GetAtt EmailQueue.Arn, RawMessageDelivery: true}
EmailQueuePolicy:
Type: AWS::SQS::QueuePolicy
Properties:
Queues: [!Ref EmailQueue]
PolicyDocument:
Statement: [{Effect: Allow, Principal: {Service: sns.amazonaws.com}, Action: sqs:SendMessage, Resource: !GetAtt EmailQueue.Arn,
Condition: {ArnEquals: {'aws:SourceArn': !Ref OrderEventsTopic}}}]
python
# Idempotent notification consumer with partial batch failure
import json, boto3
from botocore.exceptions import ClientError
t = boto3.resource("dynamodb").Table(os.environ["DEDUP_TABLE"])
def handler(event, context):
failures = []
for rec in event["Records"]:
try:
msg = json.loads(rec["body"])
try: # claim this order for email; second attempt sees the item and skips
t.put_item(Item={"pk": f"email#{msg['detail']['orderId']}", "expiresAt": int(time.time()) + 86400},
ConditionExpression="attribute_not_exists(pk)")
except ClientError as e:
if e.response["Error"]["Code"] == "ConditionalCheckFailedException": continue
raise
send_email(msg)
except Exception:
failures.append({"itemIdentifier": rec["messageId"]})
return {"batchItemFailures": failures}
Step 4: The Order Workflow (Saga)
json
{
"StartAt": "ReserveStock",
"States": {
"ReserveStock": {"Type": "Task", "Resource": "arn:aws:states:::lambda:invoke",
"Parameters": {"FunctionName": "${ReserveStockFn}", "Payload.$": "$.detail"}, "ResultPath": "$.stock", "ResultSelector": {"ok.$": "$.Payload.ok"},
"Retry": [{"ErrorEquals": ["Lambda.ServiceException", "Lambda.TooManyRequestsException"], "IntervalSeconds": 2, "MaxAttempts": 4, "BackoffRate": 2}],
"Catch": [{"ErrorEquals": ["States.ALL"], "ResultPath": "$.error", "Next": "MarkFailed"}], "Next": "ChargePayment"},
"ChargePayment": {"Type": "Task", "Resource": "arn:aws:states:::lambda:invoke",
"Parameters": {"FunctionName": "${ChargeFn}", "Payload": {"orderId.$": "$.detail.orderId", "amount.$": "$.detail.total"}},
"ResultPath": "$.payment",
"Retry": [{"ErrorEquals": ["PaymentTimeout"], "IntervalSeconds": 3, "MaxAttempts": 3, "BackoffRate": 2}],
"Catch": [{"ErrorEquals": ["States.ALL"], "ResultPath": "$.error", "Next": "ReleaseStock"}], "Next": "CreateShipment"},
"CreateShipment": {"Type": "Task", "Resource": "arn:aws:states:::lambda:invoke",
"Parameters": {"FunctionName": "${ShipFn}", "Payload.$": "$.detail"}, "ResultPath": "$.shipment",
"Catch": [{"ErrorEquals": ["States.ALL"], "ResultPath": "$.error", "Next": "RefundPayment"}], "Next": "MarkConfirmed"},
"MarkConfirmed": {"Type": "Task", "Resource": "arn:aws:states:::dynamodb:updateItem",
"Parameters": {"TableName": "${Table}", "Key": {"pk": {"S.$": "States.Format('ORDER#{}', $.detail.orderId)"}, "sk": {"S": "HEADER"}},
"UpdateExpression": "SET #s = :c", "ExpressionAttributeNames": {"#s": "status"}, "ExpressionAttributeValues": {":c": {"S": "CONFIRMED"}}}, "End": true},
"RefundPayment": {"Type": "Task", "Resource": "arn:aws:states:::lambda:invoke", "Parameters": {"FunctionName": "${RefundFn}", "Payload.$": "$.detail"}, "ResultPath": null, "Next": "ReleaseStock"},
"ReleaseStock": {"Type": "Task", "Resource": "arn:aws:states:::lambda:invoke", "Parameters": {"FunctionName": "${ReleaseFn}", "Payload.$": "$.detail"}, "ResultPath": null, "Next": "MarkFailed"},
"MarkFailed": {"Type": "Task", "Resource": "arn:aws:states:::dynamodb:updateItem",
"Parameters": {"TableName": "${Table}", "Key": {"pk": {"S.$": "States.Format('ORDER#{}', $.detail.orderId)"}, "sk": {"S": "HEADER"}},
"UpdateExpression": "SET #s = :f", "ExpressionAttributeNames": {"#s": "status"}, "ExpressionAttributeValues": {":f": {"S": "FAILED"}}}, "End": true}
}
}
- Each activity function must also be idempotent (use the order id as the idempotency key with payment and shipping providers)
- Retry only transient errors. Catch business failures and run compensation
- Use a Standard workflow for auditability and the visual history. Use Express only for very high volume short flows
Step 5: Failure Handling
| Failure | What happens |
| Lambda throttled or transient error | Step Functions retries with backoff |
| Payment declined | Catch → release stock → mark order FAILED, customer notified |
| Email provider down | Message retried by SQS; after 4 receives goes to DLQ; alarm fires |
| EventBridge cannot reach the target | Retries up to the policy, then target DLQ |
| Bad deployment of a consumer | Messages wait in the queue; fix and the backlog drains |
bash
# Look at failed messages, fix the bug, then move them back
aws sqs receive-message --queue-url $EMAIL_DLQ --max-number-of-messages 5
aws sqs start-message-move-task --source-arn $EMAIL_DLQ_ARN --destination-arn $EMAIL_QUEUE_ARN # DLQ redrive
# Replay events from an EventBridge archive
aws events start-replay --replay-name fix-nov17 --event-source-arn $ARCHIVE_ARN \
--destination '{"Arn":"'$BUS_ARN'"}' --event-start-time 2026-11-17T08:00:00Z --event-end-time 2026-11-17T10:00:00Z
Step 6: Monitoring
- Alarms on every DLQ depth greater than 0
ApproximateAgeOfOldestMessage for each queue (consumers falling behind)
- Step Functions
ExecutionsFailed, ExecutionThrottled, ExecutionTime
- EventBridge
FailedInvocations and ThrottledRules
- X-Ray tracing across API, Lambdas and Step Functions
- A business metric: orders confirmed per minute versus created
Test Plan
- Unit test each Lambda with moto
- Run the workflow with test input in the Step Functions console and inject failures (a Lambda that raises
PaymentTimeout, one that raises a business error)
- Send the same
OrderCreated event twice and confirm there is one email and one charge
- Kill the email Lambda and confirm messages queue and drain later
- Load test 1,000 orders and watch queue age and throttles
Extensions
- Add a wait for callback step for manual fraud review using
waitForTaskToken
- Add EventBridge Pipes from the stream to the bus instead of the outbox Lambda
- Add Kinesis Firehose to store all events in S3 for Athena analytics
- Add FIFO queue for strict per-customer ordering
- Add X-Ray groups and a CloudWatch dashboard
Conclusion
You built a decoupled order pipeline: transactional order plus outbox, EventBridge routing, a Step Functions saga with retries and compensation for the critical path, SNS and SQS fan-out for side effects, idempotent consumers with partial batch failures, DLQs with redrive, archive replay, and alarms on queue age and failures.
Quick Summary: goal; design decisions; outbox; EventBridge rules; fan-out; consumer; saga; failure table; redrive/replay; monitoring; test plan.