The Editing Bay: Assembling Footage Into One Vertical Cut With ffmpeg

Two paths for joining: one absurdly fast, one painfully slow

The first is the concat demuxer. You write a manifest, one segment per line, and let ffmpeg read straight through:

# concat.txt contents: one line per segment, file '/absolute/path'
ffmpeg -f concat -safe 0 -i concat.txt -c copy -movflags +faststart joined.mp4

-c copy is the whole point: no decoding, no re-encoding, just moving packets in order into a new container. A twenty-one-second cut appears in a fraction of a second with not one bit of quality lost.

But it has an unforgiving precondition: every segment's encoding parameters must be identical. Resolution, frame rate, pixel format, sample rate, channel count — one mismatch and at best the second segment tears and loses sound, at worst you get Non-monotonous DTS and a file with a scrambled duration. And AI-generated material is exactly the kind most likely to be inconsistent: different models from one vendor can output different sizes, and image-to-video output frequently has no audio track at all.

The second path is the filter graph: feed all inputs into a directed graph, let ffmpeg decode and compute, and re-encode the output. It can do anything — scaling, padding, mixing, transitions — at the cost of re-encoding the whole piece: a tenfold slowdown and one generation of quality loss.

The beginner's instinct is "the filter graph can do everything, so always use it." That is wrong. The right move is to chain both paths:

The normalization filter chain looks like this. You do not need to read it word for word — just see that a program assembled it:

ffmpeg -i clip.mp4 -i voice.mp3 -filter_complex \
  "[0:v]scale=1080:1920:force_original_aspect_ratio=decrease,\
   pad=1080:1920:-1:-1:color=black,fps=24,trim=duration=6.000,setpts=PTS-STARTPTS[v];\
   [0:a]volume=0.25,aformat=sample_fmts=fltp:sample_rates=44100:channel_layouts=stereo,\
   apad,atrim=duration=6.000,asetpts=PTS-STARTPTS[amb];\
   [1:a]aformat=sample_fmts=fltp:sample_rates=44100:channel_layouts=stereo,\
   adelay=200|200,apad,atrim=duration=6.000,asetpts=PTS-STARTPTS[vo];\
   [amb][vo]amix=inputs=2:duration=longest:normalize=0[a]" \
  -map "[v]" -map "[a]" -c:v libx264 -crf 20 -pix_fmt yuv420p -r 24 \
  -c:a aac -b:a 128k -ar 44100 -ac 2 -t 6.000 seg-s01.mp4

Transitions are a special case of this choice. The frames of a cross-fade between two shots are newly computed, so there is no way to have a transition without re-encoding. Wanting transitions means the filter graph, stacking segments pairwise with xfade. And there is one very easy miscalculation: each segment's start offset must subtract the transition time already consumed before it. Four segments with three 300-millisecond transitions make the cut 900 milliseconds shorter than the timeline — forget to correct the subtitle timecodes and they drift further the later you go, evenly, looking like "everything is slightly late," which is the drift most often blamed on the player.

// The choice between the two paths happens in exactly one place, and it is a pure function:
// give it segments and whether a transition is wanted, get back an argument list.
// The arguments are computed, not string-concatenated, and certainly not model-generated.
export function buildConcatArgs(segments, durationsMs, { fadeMs = 0 } = {}) {
  if (fadeMs === 0) {
    // Path one: identical parameters, straight stream copy. The caller writes the manifest first.
    return ['-f', 'concat', '-safe', '0', '-i', 'concat.txt', '-c', 'copy', 'joined.mp4']
  }
  // Path two: an xfade chain. The offset subtracts all transition time consumed before it.
  const fade = (fadeMs / 1000).toFixed(3)
  const chains = []
  let v = '0:v'
  let offsetMs = 0
  for (let i = 1; i < segments.length; i += 1) {
    offsetMs += durationsMs[i - 1] - fadeMs
    chains.push(`[${v}][${i}:v]xfade=transition=fade:duration=${fade}:offset=${(offsetMs / 1000).toFixed(3)}[v${i}]`)
    v = `v${i}`
  }
  return [...segments.flatMap((p) => ['-i', p]), '-filter_complex', chains.join(';'), '-map', `[${v}]`, 'joined.mp4']
}

# The choice between the two paths happens in exactly one place, and it is a pure function:
# give it segments and whether a transition is wanted, get back an argument list.
# The arguments are computed, not string-concatenated, and certainly not model-generated.
def build_concat_args(segments: list[str], durations_ms: list[int], fade_ms: int = 0) -> list[str]:
    if fade_ms == 0:
        # Path one: identical parameters, straight stream copy. The caller writes the manifest.
        return ["-f", "concat", "-safe", "0", "-i", "concat.txt", "-c", "copy", "joined.mp4"]
 
    # Path two: an xfade chain. The offset subtracts all transition time consumed before it.
    fade = f"{fade_ms / 1000:.3f}"
    chains, v, offset_ms = [], "0:v", 0
    for i in range(1, len(segments)):
        offset_ms += durations_ms[i - 1] - fade_ms
        chains.append(f"[{v}][{i}:v]xfade=transition=fade:duration={fade}:offset={offset_ms / 1000:.3f}[v{i}]")
        v = f"v{i}"
 
    inputs = [arg for p in segments for arg in ("-i", p)]
    return [*inputs, "-filter_complex", ";".join(chains), "-map", f"[{v}]", "joined.mp4"]

The vertical canvas: 1080 by 1920

Short drama is watched upright, on a fixed 1080x1920 canvas at 9:16. The number itself is unremarkable; what is worth discussing is what to do when the material does not match the canvas, because that is close to inevitable.

Two options: scale up to fill and crop the excess, or scale down to fit and pad the edges with black. The force_original_aspect_ratio=decrease plus pad in the command above is the second — preserve the content first, then pad. Why is that the default? Because cropping cuts off the top of a head or a chin, and in AI-generated frames the face's position is uncontrollable to begin with, so cropping goes wrong easily. Black bars are at least predictably ugly rather than randomly ruinous.

One thing gets overlooked even more than resolution: the safe area. On a phone, the very top and bottom of a vertical video are not fully visible — notches, rounded corners, the scrubber, and the platform's overlaid username and hashtags all eat into it. So subtitles must not hug the bottom edge, and the subject must not touch the top. Placing subtitles in the band roughly one-tenth to one-sixth up from the bottom, with ten percent margins on each side, is a safe default.

How many characters fit on a subtitle line also needs pinning down. A 1080-wide frame holds about thirteen or fourteen CJK characters per line before the font shrinks past comfortable phone reading. Long lines must be broken when the subtitles are generated, not left to the player.

Which brings us to today's most important engineering habit.

Probe the capability, then degrade

Faced with "this feature may not exist," the beginner writes it into the docs: "please make sure your ffmpeg supports the subtitles filter." That sentence achieves roughly nothing — readers do not read it, and if they do, they do not know how to check, and they give up on an error message they cannot parse.

The right approach is for the program to probe, take another route if the probe fails, and tell the user which route it took. ffmpeg lists every supported filter under ffmpeg -filters, so the probe is one command and one match:

ffmpeg -hide_banner -filters | grep -w subtitles

Found: burn in. Not found: fall back to soft subtitles — muxing the SRT into the mp4 container as a separate subtitle track using mov_text, the subtitle codec mp4 understands:

ffmpeg -i with-bgm.mp4 -i episode-1.srt -c copy -c:s mov_text \
  -metadata:s:s:0 language=eng episode-1.mp4

The two produce different things. Burned-in subtitles are pixels no player can turn off, which is usually what short-drama distribution wants; soft subtitles are a toggleable track, keeping the master clean and letting you swap in another translation, at the price that many platform players do not surface the toggle. So the default policy is "burn if you can, mux if you cannot, and say clearly which one happened", not a hard-coded choice of one.

import { execFile } from 'node:child_process'
import { promisify } from 'node:util'
const run = promisify(execFile)
 
async function hasFilter(name) {
  try {
    const { stdout } = await run('ffmpeg', ['-hide_banner', '-filters'])
    return new RegExp(`(^|\\s)${name}\\s`, 'm').test(stdout)
  } catch {
    return false // no ffmpeg at all, so certainly no such filter
  }
}
 
export async function attachSubtitles(video, srt, out) {
  if (await hasFilter('subtitles')) {
    const escaped = srt.replace(/:/g, '\\:').replace(/'/g, `\\'`)
    await run('ffmpeg', ['-y', '-i', video, '-vf', `subtitles='${escaped}'`, '-c:a', 'copy', out])
    return 'burned'
  }
  // Fallback: a soft subtitle track. The cut still comes out; the viewer toggles it on.
  await run('ffmpeg', ['-y', '-i', video, '-i', srt, '-c', 'copy', '-c:s', 'mov_text', out])
  return 'soft'
}

import re
import subprocess
 
 
def has_filter(name: str) -> bool:
    try:
        out = subprocess.run(["ffmpeg", "-hide_banner", "-filters"], capture_output=True, text=True).stdout
    except FileNotFoundError:
        return False  # no ffmpeg at all, so certainly no such filter
    return re.search(rf"(^|\s){name}\s", out, re.MULTILINE) is not None
 
 
def attach_subtitles(video: str, srt: str, out: str) -> str:
    if has_filter("subtitles"):
        escaped = srt.replace(":", r"\:").replace("'", r"\'")
        subprocess.run(["ffmpeg", "-y", "-i", video, "-vf", f"subtitles='{escaped}'", "-c:a", "copy", out], check=True)
        return "burned"
 
    # Fallback: a soft subtitle track. The cut still comes out; the viewer toggles it on.
    subprocess.run(["ffmpeg", "-y", "-i", video, "-i", srt, "-c", "copy", "-c:s", "mov_text", out], check=True)
    return "soft"

This pattern reaches far beyond subtitles. Whether a hardware encoder exists, whether a decoder for some format is present, which system fonts are installed — all follow the same shape: probe once, cache the result, degrade, announce. It works far better than a line of "please make sure" in a README.

Out of sync: check these three places

The first thing to do with a finished cut is listen. Audio-video desync is the most frequent problem at this stage, and every instance looks the same — mouth and sound half a beat apart — while the causes differ completely. Check in this order and three steps locate it almost every time.

First, check the timeline table. Subtract each record's startMs from its endMs and compare with the real duration of that shot's video file. If the table says six seconds and the file is 5.8, every sound from that shot onward arrives 0.2 seconds early, and the error accumulates. This is the most common one. The root cause is usually upstream: the video API's returned duration is not exactly what you requested.

Second, check whether the voice was truncated. A twelve-character line at roughly 0.22 seconds per CJK character needs 2.64 seconds, plus 0.2 seconds of breathing at each end, so just over 3 seconds; if the shot is only 2 seconds long, the last few characters are chopped off. That is not desync, but it sounds almost identical. So the conflict should be detected and warned about while generating the timeline, not discovered by ear in the finished cut. Today's BREAK_SHOT switch lets you cause one deliberately.

Third, check the join. If the first two are clean, the join introduced it. Under stream copy, misaligned audio and video start timestamps across segments produce offsets after joining; with transitions the cut gets shorter overall, and subtitles that were not recomputed show up as "worse the further in you go."

The cover: do not use the first frame

Once an episode is out, the overwhelming majority of people see exactly one thing: the cover. It decides the click rate, and the click rate decides whether the episode gets watched at all.

Beginners grab frame one, which is almost always wrong. The opening frame is usually black, or the start of a push-in, or a wide shot that has not settled. Worse, the first frame of an AI-generated video is often its least stable one, with a face that has not fully formed.

A few directly usable rules: prefer close-ups, because a thumbnail is thumbnail-sized on a phone and nothing in a wide shot reads; take the midpoint of that shot, not its start; if the episode has a clear emotional peak, take it there. In code that is a pure function: find the first shot whose size is close and take the midpoint of its start and end.

The grab itself is simple, but one detail is worth stating:

ffmpeg -ss 8.500 -i episode-1.mp4 -frames:v 1 -q:v 2 cover-1.jpg

Putting -ss before -i is keyframe-level fast seeking, nearly instant; after -i it is frame-accurate, at the cost of decoding from the start. A cover needs no such accuracy, so put it before. This is one of the few places in ffmpeg where argument position changes behavior, and it is worth remembering.

What size the cover should be exported at, whether to add title text, and what ratio each platform requires belongs to distribution, and D13 covers it. Today, one 1080x1920 image matching the cut is enough.

Let the model fill parameters, not write commands

One design question last, and it matters more than every ffmpeg trick above.

The thought will occur to you: ffmpeg parameters are so complicated, why not just have the model generate the command? Hand it the timeline and the requirements, get back a full command line, execute it. Very Agent-like.

Do not. Three reasons, in order of severity.

First, it is a command injection surface. You are handing a string to a shell, and part of that string came from a sentence a user typed and a script a model wrote. One unescaped quote and arbitrary commands run on your server.

Second, it is not reproducible. Given the same timeline, the command generated today may differ from tomorrow's: changed encoding parameters, reordered filters, a dropped flag. Re-run one episode and the output differs, and you cannot even say what changed. A production line needs the same input to yield the same output always, and tomorrow that becomes a hard requirement.

Third, it is not debuggable. When the command is wrong you get one cryptic ffmpeg error and then have to guess why the model wrote it that way; when your own pure function generates the arguments, a wrong result is a wrong function, and an assertion tests it.

Is the model useless here, then? No — but its place is parameter filler, not command generator: is this shot cold or warm, hard cut or fade, which shot for the cover. It emits structured choices only, selected from an enumeration you fixed in advance. Validate the result against a whitelist, then feed it to your own argument-building function.

// The model only picks from an enumeration, and the pick still passes a whitelist.
// So even if the model is prompt-injected, the worst it can do is choose an ugly transition.
const TRANSITIONS = new Set(['none', 'fade'])
const CROPS = new Set(['pad', 'crop'])
 
export function sanitizePlan(raw) {
  return {
    transition: TRANSITIONS.has(raw?.transition) ? raw.transition : 'none',
    crop: CROPS.has(raw?.crop) ? raw.crop : 'pad',
    // The cover shot must actually exist in this episode's timeline, else fall back to default.
    coverShotId: typeof raw?.coverShotId === 'string' ? raw.coverShotId : null,
  }
}

# The model only picks from an enumeration, and the pick still passes a whitelist.
# So even if the model is prompt-injected, the worst it can do is choose an ugly transition.
TRANSITIONS = {"none", "fade"}
CROPS = {"pad", "crop"}
 
 
def sanitize_plan(raw: dict) -> dict:
    return {
        "transition": raw.get("transition") if raw.get("transition") in TRANSITIONS else "none",
        "crop": raw.get("crop") if raw.get("crop") in CROPS else "pad",
        # The cover shot must actually exist in this episode's timeline, else fall back.
        "coverShotId": raw["coverShotId"] if isinstance(raw.get("coverShotId"), str) else None,
    }

This boundary is worth reusing in any scenario where an Agent touches system resources: the model judges, the program executes; the model's output must land in a constrained structure before it becomes an action. Hold on to that sentence and you will not go far wrong designing any Agent that does real work.

Source Reading

Hands-On Lab

Confirm two things before starting: ffmpeg -version and ffprobe -version both run on this machine, and yesterday's dubbing output is still there. The lab needs no API key — MOCK=1 has ffmpeg generate placeholder material on the spot — but the business logic (timeline computation, SRT generation, argument assembly, capability probing) all runs for real. If you get stuck, look at the full ffmpeg command printed in the terminal and run it standalone; the error will be far clearer.

1. Run the full flow once, open the cut, and confirm the four shots join in order, each has voice, and the subtitles can be turned on in a player.
2. Inspect the cut with ffprobe: three streams, 1080x1920, duration matching the timeline, and reconcile those three numbers against the table printed in the terminal.
3. Turn on TRANSITION=fade and run again, compare the elapsed time, and confirm the subtitle timecodes shifted earlier by three transitions' worth.
4. Open the generated cover, confirm it is not a black frame, and find the line in the code that decided which second to grab.
5. Use BREAK_SHOT to shorten one shot, watch the warning and the change in the cut, then trace the warning back to that record in the timeline.

Interview Questions

Today's 3 questions are in the question bank below, focused on the division of labor between the timeline data structure and command generation, the order of desync diagnosis, and the risk of letting a model generate commands. Read the analysis before the key points — practicing the derivation beats memorizing the points. The cn / global labels let you pick by target market.

Checklist and Tomorrow

• I can use ffmpeg to splice multiple shots along a timeline into one vertical finished cut
• I can add subtitles, an audio track, transitions, and a cover to the finished cut, and output encoding parameters the target platform accepts
• I can have the program generate the ffmpeg command from a structured timeline, rather than handing the command to the model to improvise
• I can state the precondition and the cost of stream-copy joining versus filter-graph joining, and why normalization comes first
• I can recite the three-step desync diagnosis and say why step one is the likeliest hit
• All 5 lab acceptance criteria pass
• I can answer at least 2 of the 3 interview questions without looking at the key points

Tomorrow (D7) we connect all six stages end to end, running one command from a sentence to a finished cut, and measure how much time and money each stage consumed. Why wait until day seven to connect them? Because only once all six run individually are the problems that surface problems caused by the connecting itself — artifact paths, intermediate state, log flooding, and the most painful one: step four blows up and the money spent on the first three is gone. D7 does not fix that; it makes you see it clearly, because only then does D8's workflow engine mean anything.