The Async Pipeline, part 4

From Script to Audio: Chunking, SSML, and Parallel Synthesis

How Lumcast turns a raw LLM script into a multi-chapter MP3: the 2900-character chunk problem, SSML voice matrices, parallel Polly synthesis, and lossless ffmpeg concatenation.

AWS / Polly / Bedrock / SSML / Audio Processing

Part 3 established the Step Functions pipeline and its error handling model. The middle four states of that pipeline — SegmentChapters, FormatSSML, SynthesizeChunks, ConcatenateAudio — are where a raw LLM-generated script becomes a playable MP3. This is where the interesting constraints live.

The Script Arrives as Prose with Markers

The Bedrock Lambda returns a script that looks like this:

// Pseudocode — raw LLM script format
[CHAPTER: Introduction]
Jazz emerged in New Orleans in the early twentieth century...

[CHAPTER: The Bebop Revolution]
By the 1940s, musicians like Charlie Parker and Dizzy Gillespie...

The chapter markers are a prompt-level contract. The prompt instructs the model to use exactly this format. The segmenter Lambda parses it with a capturing regex split, collecting (marker, text) pairs. Text before the first marker becomes an “Intro” chapter. This is the only structural convention the pipeline relies on from the model — everything else is treated as opaque prose.

The 2900-Character Problem

Polly’s SSML input has a hard limit: 2900 characters per API call. A 30-minute podcast script contains roughly 4,000–5,000 words. A single chapter can easily exceed 2900 characters. So the segmenter must split each chapter into chunks that fit within Polly’s limit.

The chunking algorithm is greedy with a fallback:

# Pseudocode — greedy sentence-boundary chunker
def chunk_text(text, max_chars):
    sentences = split_on_sentence_boundaries(text)  # split at .?! + whitespace
    current_chunk = []
    current_len = 0

    for sentence in sentences:
        if current_len + len(sentence) <= max_chars:
            current_chunk.append(sentence)
            current_len += len(sentence)
        else:
            if current_chunk:
                yield " ".join(current_chunk)
            if len(sentence) > max_chars:
                yield hard_split(sentence, max_chars)  # no choice
                current_chunk, current_len = [], 0
            else:
                current_chunk = [sentence]
                current_len = len(sentence)

    if current_chunk:
        yield " ".join(current_chunk)

The limit is set to 2900, not 3000, leaving 100-character headroom for the SSML wrapper tags (<speak>, <voice>, break tags) that will be added in the next stage. The hard-split fallback handles edge cases — a single URL or compound word that exceeds the limit — without crashing.

The output is a nested structure: chapters → chunks → text. A 30-minute podcast might produce 5 chapters with 8–12 chunks each. The SSML formatter flattens this into a single array for the Map state.

SSML: The Voice Layer

Plain text synthesised by Polly sounds acceptable. SSML-annotated text sounds natural. The SSML formatter wraps each chunk and adds two types of pauses:

  • Sentence pauses (<break time="400ms"/>) after ., ?, ! — gives the listener breathing room between ideas
  • Chapter transition pauses (<break time="1200ms"/>) at the start of the first chunk in a new chapter — signals a topic shift

The voice selection is non-trivial. Polly has three quality tiers — standard, neural, generative — and the voice IDs differ by language and tier:

LanguageStandardNeuralGenerative
EnglishMatthewMatthewMatthew
ItalianCarlaAdrianoAdriano (neural fallback)
GermanMarleneDanielDaniel (neural fallback)
FrenchCelineRemiRemi (neural fallback)
SpanishConchitaSergioSergio (neural fallback)

Generative synthesis is only available for English in Polly. For other languages, a generative request silently routes to neural. The SSML formatter hardcodes this fallback logic — if language != "en" and quality == "generative", use the neural voice and neural engine.

There is a critical constraint on neural and generative engines: <prosody rate> and <prosody pitch> tags are not supported. The formatter never adds them. This sounds obvious but trips up engineers who write SSML for standard voices and then switch to neural without reading the Polly engine documentation.

Parallel Synthesis and S3 Part Layout

Each chunk is synthesised as a separate Polly MP3 and stored in S3 at a deterministic path:

{userId}/{podcastId}/parts/ch00_ck000.mp3
{userId}/{podcastId}/parts/ch00_ck001.mp3
{userId}/{podcastId}/parts/ch01_ck000.mp3
...

The zero-padded naming is load-bearing. The concatenator sorts parts by key name to assemble chapters in order. If you use ch0_ck0.mp3 and ch10_ck0.mp3, lexicographic sort puts chapter 10 before chapter 2.

Each Polly synthesiser Lambda receives one chunk, synthesises it, uploads to S3, and returns { chapterIndex, chunkIndex, s3PartKey }. The Map state collects these into synthesizedChunks — an array that the concatenator uses to download and assemble.

Concatenation Without Re-encoding

The concatenator Lambda has three jobs: download all parts in parallel, concatenate them in order, and measure the final duration.

# Pseudocode — parallel download then sequential concat
def concatenate(parts, bucket):
    with ThreadPoolExecutor(max_workers=10) as pool:
        futures = {pool.submit(download, p["s3PartKey"]): p for p in parts}
        local_files = [futures_result_in_order(futures, parts)]

    # Write ffmpeg concat list
    write_file("concat.txt", "\n".join(f"file '{f}'" for f in local_files))

    subprocess.run([
        "/opt/bin/ffmpeg",
        "-f", "concat", "-safe", "0",
        "-i", "concat.txt",
        "-acodec", "copy",        # stream copy: no re-encoding
        "output.mp3"
    ])

    duration = probe_duration("/opt/bin/ffprobe", "output.mp3")
    return upload(bucket, "output.mp3"), duration

The key is -acodec copy. All Polly MP3 parts have identical encoding parameters (22050 Hz, same bitrate). Stream copy concatenates the bitstreams directly without decoding and re-encoding. For a 30-minute podcast with 47 parts, this takes under a second instead of tens of seconds, and introduces zero quality loss.

ffmpeg is available via a Lambda layer deployed from the AWS Serverless Application Repository. The binaries land at /opt/bin/ffmpeg — not in the PATH by default, so the Lambda calls the full path. The concatenator is the only Lambda with 1024 MB memory (vs 512 for all others); it holds all audio parts in /tmp simultaneously during assembly.

After a successful upload, the concatenator batch-deletes the parts from S3. This is cleanup, not part of the success contract — it’s logged but not retried if it fails.

The Transcript

Parallel to the audio, the UploadArtifacts Lambda uploads a transcript JSON to S3:

// Pseudocode — transcript structure
{
  "podcastId": "...",
  "topic": "The history of jazz",
  "language": "en",
  "durationSeconds": 1843,
  "generatedAt": "2026-05-31T14:22:10Z",
  "chapters": [
    { "index": 0, "title": "Introduction", "text": "Jazz emerged in New Orleans..." },
    { "index": 1, "title": "The Bebop Revolution", "text": "By the 1940s..." }
  ],
  "fullText": "Jazz emerged in New Orleans... By the 1940s..."
}

Chapter text is reconstructed by joining the chunks for that chapter. fullText is all chapters concatenated — useful for search indexing in a future iteration.

Apply This

1. Chunk at a semantic boundary, not a byte boundary. Polly’s limit is per-call, not per character in the final audio. Chunking at sentence boundaries means each synthesis call contains complete thoughts. Chunking at arbitrary byte offsets produces audio that sounds cut off mid-sentence. Always find the nearest semantic boundary at or below the limit.

2. Zero-pad numeric identifiers in sort keys. If any downstream operation sorts by key name (S3 object keys, filenames, log entries), zero-pad numbers. ch00 before ch09 before ch10 sorts correctly. Without padding, lexicographic sort breaks at 10. This is boring to implement and expensive to fix in production.

3. Prefer stream copy over re-encoding when format is uniform. If all your input files share the same codec, bitrate, and sample rate, -acodec copy is the right ffmpeg flag. It is not an optimisation — it is the correct operation. Re-encoding introduces quality loss and latency with no benefit. Confirm your source format is uniform before using it.

4. Build voice matrices as data, not code. The voice selection logic — language × quality tier → (voiceId, engine) — is a 5×3 lookup table. Express it as a dictionary, not as nested if/elif chains. Tables are easier to extend (add a new language, add a new quality tier), easier to review, and easier to unit test. Nest your conditionals and you will introduce a bug adding the sixth language.

5. Separate part storage from final storage. The parts/ prefix in S3 is temporary work product. The final audio.mp3 is the deliverable. Keeping them in separate prefixes makes lifecycle rules, presigned URLs, and cleanup operations unambiguous — you can delete parts/* without risk of touching the final output. Post 5 covers what happens after the audio is stored: how the quota system deducts cost only after confirming successful generation.