D5 Voiceover, Subtitles, and Audio Tracks: Multi-Character Voices, Timeline Alignment, and Subtitle Files

Today's Goals

Assign a voice per character and batch-synthesize dialogue, producing audio files ready to use directly
Align audio duration with shot duration, and generate subtitle files with a correct timeline
Explain the parameters in speech synthesis that most affect the finished feel, and the boundaries for handling background music

Yesterday you got a batch of moving shots. Today you give them voices — and settle a question the first four days kept deferring: how long should this shot actually be? When you finish, scroll back up and tick the three goals off.

Plain-Language Walkthrough

Break one default assumption first, or today's whole chapter reads as the only option available.

Not every video model produces a silent film. Some return picture only, with an empty audio track; some include ambient sound; some will generate lip-sync along with spoken lines written into the prompt, and the official docs show exactly that form, sometimes allowing a reference audio clip to fix the timbre. So "once you have the shot you must dub it" depends on which model you used — before you start, check whether the output even has an audio stream.

If the model can speak for itself, why does this course still run a separate synthesis step? Four engineering differences:

The lines are controllable word for word. The script's dialogue is fixed. Let the model improvise and it will never say exactly that line — and in drama, one wrong word is a wrong line.
The timbre stays stable across shots. A character appears in six shots; if each clip generates its own voice, one person ends up with six voices. This is the audio version of the face-swap problem, and audiences are more sensitive to voice than to face.
The timeline becomes computable. The whole pipeline's timeline is derived backwards from voice duration (that is the second half of today). With speech baked into the video you cannot measure when each line starts or how long it runs — subtitle timecodes cannot be generated, and music ducking cannot be aligned.
Changing one word differs by four orders of magnitude. Re-synthesizing a line costs a fraction of a cent; re-generating a shot costs dollars.

Conversely, if your piece explicitly wants no subtitles, the lines need not be exact, and lip-sync matters more, letting the video model speak is the easier choice — its lip movement is generated with the picture, and post cannot match that. Today teaches the controllable path; knowing the other one exists is how you know what you are choosing.

One voice per character

In a recording booth the voice director's first job is not to roll, it is casting: who voices the lead, who voices the second female. Once decided, it goes on the wall chart and does not change all season. Audiences remember voices better than you expect — change a character's voice in episode two and the comments immediately ask who this is.

So step one of dubbing is an assignment table, and that table should not be improvised on dubbing day. It belongs to the character card defined on D2: each character has a voiceId field holding the vendor's system voice identifier. Cross-episode consistency comes from that record, not from a temporary map inside the dubbing code — which is why this course calls the character card a record rather than a parameter.

MiniMax's system voices include a set tuned for short drama, named after the archetype: badao_shaoye the domineering heir, junlang_nanyou the handsome boyfriend, lengdan_xiongzhang the aloof senior, chunzhen_xuedi the innocent junior, bingjiao_didi the possessive little brother; on the female side wumei_yujie, danya_xuejie, tianxin_xiaoling, qiaopi_mengmei, diadia_xuemei. There is also a general set such as male-qn-jingying and female-chengshu, good for narration and background characters.

Beyond the voice itself, three parameters directly shape the finished feel:

Emotion (emotion) is the most overlooked and the most audible. Its values are a fixed enumeration: happy, sad, angry, fearful, disgusted, surprised, calm, fluent, whisper. The same line — "that phone is not yours" — is a cool warning under calm and a confrontation under angry; two completely different scenes. The engineering move is to give each character a baseline emotion and let individual lines override it. Short-form drama's emotional turns live in those overrides.

Speed (speed) ranges from 0.5 to 2, and it has a hidden effect: speed changes audio duration, and duration propagates into the shot. So speed cannot be nudged casually at this step; nudge it and you re-align the timeline.

Volume and pitch (vol, pitch) usually stay at their defaults. To make a character sound younger or heavier, change the voice, do not pull the pitch — pitched audio has an obvious synthetic edge.

Finally one return-value detail you must know: data.audio is a hex string by default, not base64. In Node that is Buffer.from(json.data.audio, "hex"). Decoding it as base64 gives you a file that saves fine and will not play, and the error message points nowhere near the cause.

tts.js

async function synthesize({ text, voiceId, emotion, speed, outPath }) {
  const res = await fetch(`${process.env.MINIMAX_BASE_URL}/v1/t2a_v2`, {
    method: 'POST',
    headers: {
      Authorization: `Bearer ${process.env.MINIMAX_API_KEY}`,
      'Content-Type': 'application/json',
    },
    body: JSON.stringify({
      model: 'speech-2.8-hd',
      text,
      stream: false,
      voice_setting: { voice_id: voiceId, speed, vol: 1, pitch: 0, emotion },
      audio_setting: { sample_rate: 32000, bitrate: 128000, format: 'mp3', channel: 1 },
    }),
  })
  const json = await res.json()
  if (json.base_resp?.status_code) throw new Error(`synthesis failed ${json.base_resp.status_code}`)
  // data.audio is a hex string by default, not base64 - decoding as base64 gives an unplayable file
  await writeFile(outPath, Buffer.from(json.data.audio, 'hex'))
  // audio_length is in milliseconds and goes straight into the timeline
  return { path: outPath, durationMs: json.extra_info.audio_length }
}

tts.py

import os
import httpx
 
 
def synthesize(text: str, voice_id: str, emotion: str, speed: float, out_path: str) -> dict:
    res = httpx.post(
        f"{os.environ['MINIMAX_BASE_URL']}/v1/t2a_v2",
        headers={"Authorization": f"Bearer {os.environ['MINIMAX_API_KEY']}"},
        json={
            "model": "speech-2.8-hd",
            "text": text,
            "stream": False,
            "voice_setting": {"voice_id": voice_id, "speed": speed, "vol": 1, "pitch": 0, "emotion": emotion},
            "audio_setting": {"sample_rate": 32000, "bitrate": 128000, "format": "mp3", "channel": 1},
        },
        timeout=120,
    )
    data = res.json()
    if data.get("base_resp", {}).get("status_code"):
        raise RuntimeError(f"synthesis failed {data['base_resp']['status_code']}")
    # data.audio is a hex string by default, not base64 - base64 gives an unplayable file
    with open(out_path, "wb") as f:
        f.write(bytes.fromhex(data["data"]["audio"]))
    # audio_length is in milliseconds and goes straight into the timeline
    return {"path": out_path, "duration_ms": data["extra_info"]["audio_length"]}

Voices cast, parameters tuned, audio on disk — and now the real problem appears: this clip runs 3.3 seconds, but the shot you generated yesterday is 3 seconds. Who gives way?

Duration is a two-way constraint

This is the core of the chapter, and the reason the pipeline is ordered the way it is.

Shot duration has two sources. One comes from the script: the writer marked this as a quick cut, planned at 3 seconds. The other comes from the line: saying it takes 3.3 seconds. When the two disagree, one must give — and that choice decides the shape of your pipeline.

This course's ruling is dialogue first: stretch the shot, never trim the speech.

The reason is direct: an audience can hear a line being cut, and cannot see that a shot ran 0.8 seconds longer than planned. A clipped half-sentence is a defect; eight-tenths of a second is a slightly slower beat. And speeding up delivery to protect the planned length gives you an episode that sounds rushed, with speed-induced artifacts that are especially audible on voice.

Hence the order of the line: generate the picture, then synthesize the speech, then write shot duration back from speech duration. That looks roundabout — why not synthesize first and then generate a video of exactly that length? Because video duration comes in fixed tiers (six seconds by default); you cannot ask for exactly 6.34 seconds. So the practical arrangement is: video generated at planned length, timeline laid out by speech duration, and any shortfall in picture made up on the editing bay (freeze or extend) — D6's craft.

The arithmetic: a shot's required duration is lead-in silence + total speech + inter-line gaps + tail silence. The silences are not optional decoration — if the line starts the instant the shot does and the cut lands the instant it ends, the episode feels throttled. The final duration is the larger of planned and required, and any stretched shot gets flagged for the human review desk on D10.

One iron rule: the duration written into the timeline must be measured from the file on disk, never estimated. Character count times a coefficient is only good enough for scheduling. Real speech varies with speed, emotion and punctuation pauses, and a few hundred milliseconds per line is normal — while subtitle drift accumulates line by line: 200 milliseconds off on line one is two seconds off by line ten, and the audience sees subtitles racing the picture.

align.js

const LEAD_MS = 250 // lead-in silence
const TAIL_MS = 400 // tail silence
const GAP_MS = 180 // pause between two lines inside one shot
 
// Dialogue first: stretch the shot, never trim the speech
export function planShot(shot, clips) {
  const plannedMs = Math.round(shot.durationSec * 1000)
  // durationMs in clips comes from the file on disk, not a character-count estimate
  const spoken = clips.reduce((sum, c) => sum + c.durationMs, 0)
  const gaps = Math.max(0, clips.length - 1) * GAP_MS
  const requiredMs = clips.length === 0 ? plannedMs : LEAD_MS + spoken + gaps + TAIL_MS
  return {
    shotId: shot.id,
    plannedMs,
    requiredMs,
    finalMs: Math.max(plannedMs, requiredMs),
    conflict: requiredMs > plannedMs, // stretched shots get flagged for human review
  }
}

align.py

LEAD_MS = 250  # lead-in silence
TAIL_MS = 400  # tail silence
GAP_MS = 180  # pause between two lines inside one shot
 
 
# Dialogue first: stretch the shot, never trim the speech
def plan_shot(shot: dict, clips: list[dict]) -> dict:
    planned_ms = round(shot["durationSec"] * 1000)
    # duration_ms in clips comes from the file on disk, not a character-count estimate
    spoken = sum(c["duration_ms"] for c in clips)
    gaps = max(0, len(clips) - 1) * GAP_MS
    required_ms = planned_ms if not clips else LEAD_MS + spoken + gaps + TAIL_MS
    return {
        "shot_id": shot["id"],
        "planned_ms": planned_ms,
        "required_ms": required_ms,
        "final_ms": max(planned_ms, required_ms),
        "conflict": required_ms > planned_ms,  # stretched shots get flagged for human review
    }

Where subtitle timing comes from

Subtitle timestamps have two possible sources, and this comes up in interviews more often than you would think.

One: the API gives them. The speech API has a subtitle_enable switch and a subtitle_type (per sentence or per word), and returns a download link to a subtitle file. That sounds convenient, but it has two practical problems: you spend an extra HTTP request fetching the content, and more importantly its timestamps are relative to that one audio clip, while your subtitles must sit on the episode's timeline, separated by the offset of "which second this shot starts at." So even using it, you cannot skip laying things out yourself.

Two: local alignment. You already hold each clip's real duration and each shot's start time; accumulate them and you have the subtitle timeline. It needs no extra request, depends on no vendor field, and you control the line breaks — one line of dialogue per cue, which is exactly the rhythm short drama wants.

This course fixes on the second path and computes every subtitle from the local timeline. The reason is practical: the field-by-field structure of the downloaded subtitle file is not published in the official docs, and we do not build a course on unverified fields. If you want to use it in your own project, measure it first.

Alignment has exactly one key: the subtitle cursor and the shot cursor must be the same variable. In pseudocode: the episode has one cursor starting at 0; for each shot, record its start, then lay out cues from "start + lead-in," advancing the cursor by each line's real duration plus the inter-line gap; when the shot is done, advance the episode cursor by that shot's final duration. Two cursors sharing one origin is the only guarantee against drift.

One easily missed self-check: every cue must fall inside the shot it belongs to. An out-of-bounds cue raises no error; it just floats the previous shot's line over the next shot's picture, and the audience sees words appearing while no mouth moves. The check is cheap and belongs hard-wired into the pipeline.

Subtitle file format details

SRT is the most universal subtitle format, simple enough to hand-write: an index starting at 1, a line of timecode, one or two lines of text, and a blank line between entries.

TextText

1
00:00:00,250 --> 00:00:02,230
Why did the phone light up by itself
 
2
00:00:06,250 --> 00:00:09,550
News from five minutes in the future,
pushed to me right now

It looks simple, but three mistakes make players reject the file, and usually with no error at all — just no subtitles. Those are painfully slow to diagnose, because you first suspect the encoding, then the path, then the player.

First, the millisecond separator is a comma, not a period. 00:00:01.320 is WebVTT; SRT needs 00:00:01,320. This is the most common one.

Second, hours, minutes and seconds must be zero-padded, and milliseconds must be three digits. Most players reject 0:0:1,32 outright.

Third, indexes start at 1 and stay contiguous. Start at 0 or skip a number and some players silently drop every subsequent cue.

srt.js

// SRT timecode: HH:MM:SS,mmm - the separator is a comma, not a period, and every part is padded
export function formatTimecode(ms) {
  const p = (n, w = 2) => String(n).padStart(w, '0')
  const h = Math.floor(ms / 3_600_000)
  const m = Math.floor((ms % 3_600_000) / 60_000)
  const s = Math.floor((ms % 60_000) / 1000)
  return `${p(h)}:${p(m)}:${p(s)},${p(ms % 1000, 3)}`
}
 
export function toSrt(cues) {
  const blocks = cues.map((c) =>
    // Indexes start at 1 and stay contiguous; a gap makes some players drop everything after it
    [`${c.index}`, `${formatTimecode(c.startMs)} --> ${formatTimecode(c.endMs)}`, wrap(c.text)].join('\n')
  )
  return blocks.join('\n\n') + '\n'
}

srt.py

# SRT timecode: HH:MM:SS,mmm - the separator is a comma, not a period, and every part is padded
def format_timecode(ms: int) -> str:
    h, rest = divmod(ms, 3_600_000)
    m, rest = divmod(rest, 60_000)
    s, millis = divmod(rest, 1000)
    return f"{h:02d}:{m:02d}:{s:02d},{millis:03d}"
 
 
def to_srt(cues: list[dict]) -> str:
    blocks = []
    for cue in cues:
        # Indexes start at 1 and stay contiguous; a gap drops everything after it on some players
        blocks.append(
            "\n".join([
                str(cue["index"]),
                f"{format_timecode(cue['start_ms'])} --> {format_timecode(cue['end_ms'])}",
                wrap(cue["text"]),
            ])
        )
    return "\n\n".join(blocks) + "\n"

Then line breaking. A vertical frame is narrow — roughly fourteen CJK characters, or about thirty Latin ones, fill a line, and past that the text either smears across the frame or gets cropped at both ends. Long lines break into two, preferring a punctuation mark as the break point (comma, period), falling back to a hard break. Hard breaks have one classic ugly outcome: a second line holding a single orphaned word. So search backwards for punctuation all the way to a sensible minimum line length — uneven lines beat an orphan.

Stacking tracks: voice, ambience, music

An episode's sound usually has three layers: voice is the lead, ambience (rain, street noise, the hum of a convenience-store cooler) makes the picture feel real, and music drives emotion.

Their relationship in one sentence: voice is always loudest, ambience sits where you can feel it without it competing, music sits a step below that and drops further whenever someone speaks. That last move is called ducking, and it is what makes dialogue intelligible — without it viewers feel "the music is too loud, I cannot hear them" but cannot say why.

How many decibels apart is not a single fixed answer, and it should not be hard-coded into a command today. Today's job is to record the intent: voice is the 0 dB reference, ambience one step down, music one more step down, with music pushed further down during speech. Those numbers travel with the timeline to the editing bay, and D6 turns them into actual ffmpeg parameters — the timeline and the mix plan are data, and the command is a function of the data. That division of labor recurs throughout the second half of this course.

One diagnostic aside: if the finished piece sounds muffled on the voice, the first suspect is not the loudness balance but mismatched sample rates. If synthesis outputs one sample rate and the video's audio track another, the resampling on mix eats the high end. Unifying the sample rate helps far more than adjusting volume.

The copyright red line: you cannot just grab background music

This section has the least technology in it and is the likeliest place on this pipeline to actually blow up.

Almost all background music is copyrighted. Ripped from a video site, recorded off someone else's drama, or even downloaded from a stock site marked "free" — any of it may carry terms that forbid commercial use. Once a drama is out and has views, a rights holder showing up is a realistic outcome, and the platform's usual response is a takedown plus throttling. What you lose is not the song; it is the whole piece's reach.

There are only three safe options: use a library with explicit commercial licensing and keep the license records; use a generative music service and confirm its terms allow commercial use; or use ambience only and no score. Short drama's emotion is largely carried by dialogue and pacing, so going without a score is not fatal.

The same family has one more red line: do not generate the likeness or the voice of a real person. Describing a character's appearance by naming a public figure, or cloning a real person's timbre, touches likeness and voice rights, and is more serious than the music issue. Every character in this course is fictional, and the appearance notes in the character card use generic features only.

What labeling generated content requires, what metadata exports should carry, and what filings short drama needs before release form a whole compliance topic that D11 covers, with the official text as the authority there. Today, just hold those two red lines and make "where did this music come from" a required field in your project — traceable provenance is the precondition for compliance at all.

Source Reading

Hands-On Lab

🧪 D5 lab: a dubbing stage that batch-voices by character and emits aligned subtitle files

Code location: labs/ai-drama-pipeline/day-05-voice-subtitle

Acceptance criteria:

In step 1 the assignment table gives the two characters two different voices, matching the fields in their character cards.
In step 2 the duration printed for each line comes from the file on disk, not a character-count estimate.
In step 3 the shot whose lines do not fit is marked as stretched, and the storyboard record's duration has actually been changed to the corrected value.
Step 5 prints a passing self-check: shots are contiguous, cues do not overlap, every cue sits inside its own shot, and every timecode is well-formed.
pnpm typecheck passes with no any.

starter/ leaves four exercises: two in dubbing (voice assignment, real duration) and two in the timeline (duration correction, timecode format). Everything runs offline under MOCK=1; placeholder audio length is derived from the line's character count, but the duration is still measured out of the file with ffprobe — so no line of the alignment logic is bypassed. Run it once unmodified first: the last step lists a string of problems — out-of-bounds cues, malformed timecodes, two characters sharing one voice — and that is your to-do list, shrinking as each exercise lands.

Wire the character card's voice field into the assignment table, run once, and confirm the two characters have different voice identifiers.
Batch-synthesize every line to disk, print each clip's real duration, and compare it against the character-count estimate.
Correct shot durations with the real values, find the stretched shot, and confirm the storyboard record was actually rewritten.
Generate the subtitle file from the timeline, load it in a player, and confirm the subtitles track the picture with no global offset.
Run the timecode self-check and drive out-of-bounds, overlap and format problems all to zero.

Interview Questions

Today's 3 questions are in the question bank below, focused on how synthesis parameters relate to finished feel, the ordering of the two-way duration constraint, and the two sources of subtitle timestamps. Read the analysis before the key points — question 1's answer must include why that side gives way; a bare conclusion invites a follow-up. The cn / global labels let you pick by target market.

Checklist and Tomorrow

I can assign a voice per character and batch-synthesize dialogue, producing audio files ready to use directly
I can align audio duration with shot duration, and generate subtitle files with a correct timeline
I can explain the parameters in speech synthesis that most affect the finished feel, and the boundaries for handling background music
I can explain why the timeline must use measured audio duration rather than a character-count estimate
I can name the three easiest mistakes in SRT timecodes
All 5 lab acceptance criteria pass
I can answer at least 2 of the 3 interview questions without looking at the key points

Tomorrow (D6) we enter the editing bay: shot videos, voice, subtitles and music get assembled along today's timeline into one complete vertical cut, plus a cover frame. The reason the timeline comes before the edit is that the editing bay should not compute time itself — it only translates a structured table into a string of ffmpeg parameters. You will also meet a very practical engineering habit: the program probes whether the local ffmpeg has a given capability before choosing burned-in or soft subtitles, rather than assuming the reader's environment matches yours.

Voiceover, Subtitles, and Audio Tracks: Multi-Character Voices, Timeline Alignment, and Subtitle Files