Orchestrating AI: Step Functions as a Generation Pipeline

Why Step Functions STANDARD workflows are the right orchestrator for a multi-stage AI pipeline, how Map states handle parallel synthesis, and how to treat error handling as a first-class state.

AWS Step Functions Serverless AI Orchestration

Part 2 ended with the SQS consumer calling StartExecution on a Step Functions state machine. That state machine is where the actual work happens: ten states, eight Lambda invocations, one parallel Map, and a terminal routing decision. This post is about why Step Functions is the right orchestrator here, how the pipeline is structured, and the two design patterns — error handling as a state, and Map state composition — that make it maintainable.

Why Step Functions Over Chained Lambdas?

The alternative to Step Functions is chaining Lambda calls directly: each Lambda invokes the next, passing state through the return value. This works at small scale and has zero infrastructure overhead. It falls apart for a few specific reasons:

State is invisible. When a chained Lambda pipeline fails halfway through, you have to reconstruct what happened from CloudWatch logs across multiple invocations. Step Functions STANDARD gives you a full execution history in the console — every state’s input, output, and error, queryable after the fact.

Retry logic is duplicated. Each Lambda in a chain must implement its own retry + backoff for downstream failures. In Step Functions, retry and catch are declared in the state machine definition, once, and apply uniformly.

Partial failure is hard to handle. If a Lambda in the middle of a chain fails after doing some work, what do you do? The next Lambda doesn’t know. Step Functions Catch lets every state redirect to a failure handler that has access to the error cause.

The cost of STANDARD workflows is per-state-transition. For a 10-state pipeline processing tens to hundreds of executions per day, this is negligible. The operational visibility is worth it at this scale. (Express workflows are cheaper per execution but lose the full history — the right choice for high-volume, short-duration pipelines. Switch if cost becomes a concern at higher volume.)

The Pipeline Structure

graph TD
    A[AssemblePrompt] --> B[GenerateScript]
    B --> C[SegmentChapters]
    C --> D[FormatSSML]
    D --> E[SynthesizeChunks\nMap, MaxConcurrency=3]
    E --> F[ConcatenateAudio]
    F --> G[UploadArtifacts]
    G --> H[FinalizeJob]
    H --> I{CheckIfSeries?}
    I -->|seriesId != null| J[SummarizeEpisode]
    I -->|standalone| K[GenerationComplete]
    J --> K
    
    A -.->|Catch: States.ALL| Z[HandleFailure]
    B -.->|Catch: States.ALL| Z
    C -.->|Catch: States.ALL| Z
    D -.->|Catch: States.ALL| Z
    E -.->|Catch: States.ALL| Z
    F -.->|Catch: States.ALL| Z
    G -.->|Catch: States.ALL| Z
    H -.->|Catch: States.ALL| Z

Eight task states build the audio. One Choice state routes based on whether this is a series episode. One terminal failure state handles every error path. SummarizeEpisode is designed to fail gracefully — even if it errors, the route is GenerationComplete, not HandleFailure. The summary is useful context; it is not required for the podcast to be playable.

State Composition and Pass-Through

Each Lambda in the pipeline receives the full state from the previous Lambda’s output. This creates a natural pass-through: fields added early in the pipeline (like jobId, userId, podcastId) travel forward without any Lambda needing to re-fetch them from DynamoDB.

// Pseudocode — how pipeline state accumulates across states
// After AssemblePrompt:
{ "jobId": "...", "userId": "...", "systemPrompt": "...", "userPrompt": "...", "modelId": "...", "maxTokens": 4200 }

// After GenerateScript adds its output:
{ "jobId": "...", "userId": "...", "systemPrompt": "...", "rawScript": "...", "actualWords": 3100, "modelUsed": "haiku" }

// After SegmentChapters adds its output:
{ "jobId": "...", "userId": "...", "rawScript": "...", "chapters": [...], "totalChunks": 47 }

The FinalizeJob state is the only one that uses explicit Parameters mapping rather than pass-through. It selectively extracts fields from the accumulated state — s3AudioKey, durationSeconds, chaptersJson — and passes only those to the callback Lambda. This prevents the callback from receiving the entire script and all intermediate data, which would exceed Lambda payload limits for long podcasts.

Parallel Synthesis: The Map State

Audio synthesis is the most time-consuming step. Each SSML chunk (up to 47 chunks for a 30-minute podcast) must be synthesised by Polly independently. Running them sequentially would take minutes. The Map state runs them with MaxConcurrency: 3.

graph LR
    A[FormatSSML\noutput: 47 ssmlChunks] -->|ItemsPath: $.ssmlChunks| B[Map State\nMaxConcurrency=3]
    B --> C1[Polly ch00_ck000]
    B --> C2[Polly ch00_ck001]
    B --> C3[Polly ch01_ck000]
    B --> C4[... 44 more]
    C1 & C2 & C3 & C4 --> D[ResultPath: $.synthesizedChunks]
    D --> E[ConcatenateAudio]

MaxConcurrency: 3 is a deliberate constraint. Polly has account-level concurrency quotas. Three concurrent synthesis requests leaves headroom for multiple users generating podcasts simultaneously. More concurrency would be faster for a single user but risks throttling for the account. The right number depends on your Polly quota and expected concurrent users.

Each Map item receives the chunk’s SSML plus context spread in via Parameters: userId, podcastId, voiceConfig. The Parameters block in the Map state merges per-item data with static context:

// Pseudocode — Map state Parameters merging chunk + context
{
  "chapterIndex.$": "$.chapterIndex",
  "chunkIndex.$": "$.chunkIndex",
  "ssml.$": "$.ssml",
  "userId.$": "$$.Execution.Input.userId",
  "podcastId.$": "$$.Execution.Input.podcastId",
  "voiceConfig.$": "$$.Execution.Input.voiceConfig"
}

The $$.Execution.Input path accesses the original execution input regardless of current state — this is how static context travels into a Map iterator without being passed through every prior state’s output.

Error Handling as a First-Class State

Every task state in the pipeline has this catch:

// Pseudocode — catch on every pipeline state
"Catch": [{
  "ErrorEquals": ["States.ALL"],
  "Next": "HandleFailure",
  "ResultPath": "$.error"
}]

HandleFailure is a Lambda invocation, not a terminal state. It calls the sfn-callback Lambda with status: failed and errorMessage: $.error.Cause. This updates both the jobs table and the podcasts table, so the client’s polling loop receives a terminal status with a human-readable error message.

The ResultPath: "$.error" is important. Without it, Step Functions replaces the entire state input with the error object. With it, the error is merged into the existing state, so HandleFailure still has access to jobId, userId, and podcastId — the fields it needs to write the failure record.

This pattern means the failure path is deterministic. No matter which state fails, HandleFailure runs. No Lambda in the pipeline needs to write its own failure record. No podcast stays stuck in processing status because a Lambda crashed without cleaning up.

The Retry Configuration

Each task state also has:

// Pseudocode — standard retry on every pipeline state
"Retry": [{
  "ErrorEquals": ["Lambda.ServiceException", "Lambda.TooManyRequestsException"],
  "IntervalSeconds": 2,
  "MaxAttempts": 2,
  "BackoffRate": 2.0
}]

Two retries with exponential backoff handle transient Lambda throttling. The GenerateScript state has a 120-second timeout; SynthesizeChunks allows 180 seconds per item. These are not arbitrary — they reflect the actual p99 latency of each operation under load.

Apply This

1. STANDARD workflows for pipelines you need to debug. The per-state-transition cost is real but low at moderate volume. The execution history is invaluable when a production podcast fails at state 6 of 10 and you need to know what the input to FormatSSML looked like. Use Express only when you’ve measured that STANDARD is too expensive.

2. Error handling as a named state, not an error response. Every pipeline failure should route to a handler that writes a clean record. If a Lambda crashes, the user should see failed with a message — not a stuck processing status. Model your failure handler before you model your success path.

3. Use ResultPath to preserve state on error. "ResultPath": "$.error" merges the error into the existing state object. Without this, your failure handler arrives with only the error object and no jobId to write against. Every Catch clause should set ResultPath.

4. Map state MaxConcurrency is a capacity contract. It’s not a performance knob — it’s a promise to downstream services. Set it based on the quota of the service being called (Polly, Bedrock, external API), not based on how fast you want things to go. Over-concurrency causes throttling that retries cannot fully absorb.

5. Use $$.Execution.Input to inject static context into Map iterators. Don’t thread static fields (userId, podcastId, configuration) through every state output just so they’re available inside a Map iterator. Access them directly from the execution input. This keeps intermediate state objects small and avoids accidental field shadowing. Post 4 covers what happens inside each of those Polly synthesis Lambdas.

← Back to projects