The Trigger Chain: From DynamoDB Row to Running Pipeline
How a single DynamoDB write cascades through Streams, EventBridge Pipe, SQS, and a consumer Lambda to start a Step Functions execution — and the subtle data normalization problem hiding inside.
In Part 1 we established the core bet: API Gateway writes directly to DynamoDB, and no Lambda is in the synchronous path. A podcast record lands in DynamoDB with status: "pending". Now what?
Something has to notice that record and start the generation pipeline. The naive answer is a Lambda triggered by the DynamoDB write. But that Lambda would be in the hot path of ingestion — which we just eliminated. The answer Lumcast uses is a four-hop chain: DynamoDB Stream → EventBridge Pipe → SQS → consumer Lambda. Each hop solves a distinct problem, and removing any one of them breaks something specific.
The Four Hops
graph LR
A[DynamoDB\nNEW_IMAGE Stream] -->|INSERT events| B[EventBridge Pipe\nfilter + flatten]
B -->|status=pending only| C[SQS Queue\nbuffer + DLQ]
C -->|batch=1| D[Lambda\nsqs-consumer]
D -->|StartExecution| E[Step Functions]
Hop 1 — DynamoDB Stream: The lumcast-podcasts table has Streams enabled with NEW_IMAGE. Every write produces a record containing the new item’s full attribute map in DynamoDB wire format ({"S": "value"}, {"N": "1"}). This stream is the event bus — it fires for every insert and update.
Hop 2 — EventBridge Pipe: The pipe subscribes to the stream with two jobs: filter and transform. Filter: eventName = INSERT AND dynamodb.NewImage.status.S = "pending". Updates (e.g., when the callback writes status: ready) do not trigger the pipeline a second time. Transform: a VTL input template flattens the DynamoDB wire format into plain JSON, extracting the fields the consumer needs.
Hop 3 — SQS: The pipe delivers to an SQS queue. This is the decoupling point. If the consumer Lambda is cold, throttled, or failing, messages wait rather than dropping. A DLQ catches messages after 3 failed deliveries, with 14-day retention — enough time to diagnose and replay a bad batch.
Hop 4 — Lambda consumer: Batch size = 1. One podcast per invocation. The consumer does three things: quota preflight, jobs table initialisation, Step Functions start. Post 5 covers the quota system in depth.
Why EventBridge Pipe Over Lambda-on-Stream?
The older pattern is: DynamoDB Stream → Lambda trigger directly. That Lambda would filter INSERT events and invoke Step Functions. EventBridge Pipe eliminates it entirely.
The gain is narrow but real. You remove a Lambda cold start from the trigger path (this one is not user-facing, but it does add latency between ingestion and pipeline start). You move the filter into infrastructure configuration, not application code. And you get native SQS batching without writing any glue.
The cost: EventBridge Pipe is a less common service and harder to test locally. The filter syntax is JSON path expressions, not Python. The input template is VTL again. If your team is stronger in Python than in VTL, the Lambda-on-stream approach is more maintainable.
SQS: Reliability, Not Just Queueing
SQS is easy to treat as a simple buffer — write here, read there. But its reliability properties are doing meaningful work:
| Property | Value | Why |
|---|---|---|
| Visibility timeout | 300s | Matches Lambda timeout; message reappears if Lambda crashes |
| Message retention | 1 day | Podcast requests older than 1 day are stale anyway |
| Receive count before DLQ | 3 | Two transient retries before human inspection |
| DLQ retention | 14 days | Enough for a weekday incident to be caught and replayed |
| Batch size | 1 | One podcast per Lambda invocation; failure is isolated |
Batch size = 1 is a deliberate choice. If the consumer processed 10 messages per invocation and crashed after processing 3, SQS would redeliver all 10. With batch size = 1, a crash redelivers exactly one message. The cost is higher Lambda invocation overhead at scale; for a podcast generator, that tradeoff is correct.
Execution Naming
The consumer starts a Step Functions execution with a deterministic name:
# Pseudocode — constructing a valid SFN execution name
prefix = podcast_id[:40]
suffix = request_id.replace("-", "")[:16]
execution_name = f"{prefix}-{suffix}"
Step Functions execution names must match [A-Za-z0-9_.-]+ and cannot exceed 80 characters. A raw UUID (podcast-id-with-hyphens-...) fails the character set requirement. Stripping hyphens from aws_request_id and capping at 16 characters gives a unique suffix without blowing the limit. This name appears in CloudWatch, Step Functions console, and X-Ray traces — making it easy to correlate a specific podcast generation with its execution.
Apply This
1. EventBridge Pipe as a zero-Lambda filter. For stream-to-queue routing with a simple filter condition, Pipe eliminates a Lambda entirely. The pitfall: test the input template carefully. Missing attributes render as empty strings, not absent keys. Always validate what the Pipe actually sends before building a consumer that assumes clean nulls.
2. SQS as a reliability contract, not just a queue. Tune visibility timeout to your Lambda timeout. Tune DLQ threshold to your acceptable retry count. Tune message retention to your data freshness requirements. These three numbers are the operational contract for your async path.
3. Batch size = 1 for idempotency-sensitive work. When each message triggers expensive, non-idempotent work (starting a Step Functions execution, charging a payment, sending an email), batch size = 1 minimises the blast radius of partial-batch failures. Scale batch size only when the work is idempotent and cost of re-processing is low.
4. Embed the trigger in the data model.
The EventBridge Pipe filter on status = "pending" means the data model is the trigger. No separate notification, no webhook, no polling. Updating a field is the event. This couples the data model to the pipeline trigger — a constraint worth being explicit about. Post 3 shows what the pipeline does once it starts.