From Shot to Footage: Image-to-Video, Polling Async Tasks, and Retrying Failures

The first frame: actually using yesterday's output

The image-to-video field is first_frame_image. It takes a public URL or a Base64 data URL, accepts JPG, PNG and WebP, and must be under 20MB. Any approved still in yesterday's asset library goes straight in, as long as it can become an address the API can fetch.

The certainty it buys is larger than it sounds. Once the first frame is given, the clip's opening frame is that image — composition, lighting, wardrobe and face all settled, and the model only pushes six seconds forward from there. Yesterday's reference image governs "do these images look like the same person"; today's first frame governs "where does this shot start acting from." They are a relay, not a repetition; character consistency was covered yesterday and is not reopened here.

One practice matters when writing alongside a first frame: describe what happens next, not what the frame already shows. The rainy night, the scooter and the blue work jacket are already there; repeating them just spends characters. Write "he slowly straightens up and looks toward the lit window upstairs" instead.

And one switch must go off up front: prompt_optimizer defaults to true in the video API (the image API defaults it to false — exactly opposite, so don't mix them up). Left on, the server rewrites your prompt invisibly and the same input can produce completely different results twice. For reproducibility, set it to false.

const SHOT_SIZE_TEXT = { wide: 'wide shot', medium: 'medium shot', close: 'close-up' }
 
// Order: picture, framing, camera, style. Truncation eats the tail, so style goes last
function shotPrompt(shot) {
  return `${shot.visual}. ${SHOT_SIZE_TEXT[shot.shotSize]}, ${shot.camera}. ${STYLE}`
}
 
async function submit(shot, firstFrameUrl) {
  const res = await fetch(`${process.env.MINIMAX_BASE_URL}/v1/video_generation`, {
    method: 'POST',
    headers: {
      Authorization: `Bearer ${process.env.MINIMAX_API_KEY}`,
      'Content-Type': 'application/json',
    },
    body: JSON.stringify({
      model: 'MiniMax-Hailuo-2.3',
      prompt: shotPrompt(shot),
      first_frame_image: firstFrameUrl, // the approved still picked yesterday
      duration: Math.round(shot.durationSec),
      resolution: '768P',
      prompt_optimizer: false, // defaults to true here and quietly rewrites your prompt
    }),
  })
  const json = await res.json()
  if (json.base_resp?.status_code) throw new Error(`submit failed ${json.base_resp.status_code}`)
  return json.task_id // only an identifier comes back; not one frame exists yet
}

import os
import httpx
 
SHOT_SIZE_TEXT = {"wide": "wide shot", "medium": "medium shot", "close": "close-up"}
 
 
# Order: picture, framing, camera, style. Truncation eats the tail, so style goes last
def shot_prompt(shot: dict) -> str:
    return f"{shot['visual']}. {SHOT_SIZE_TEXT[shot['shotSize']]}, {shot['camera']}. {STYLE}"
 
 
def submit(shot: dict, first_frame_url: str) -> str:
    res = httpx.post(
        f"{os.environ['MINIMAX_BASE_URL']}/v1/video_generation",
        headers={"Authorization": f"Bearer {os.environ['MINIMAX_API_KEY']}"},
        json={
            "model": "MiniMax-Hailuo-2.3",
            "prompt": shot_prompt(shot),
            "first_frame_image": first_frame_url,  # the approved still picked yesterday
            "duration": round(shot["durationSec"]),
            "resolution": "768P",
            "prompt_optimizer": False,  # defaults to True here and rewrites your prompt
        },
        timeout=60,
    )
    data = res.json()
    if data.get("base_resp", {}).get("status_code"):
        raise RuntimeError(f"submit failed {data['base_resp']['status_code']}")
    return data["task_id"]  # only an identifier comes back; not one frame exists yet

The shape of an async task: four steps, every one of which can fail

Look at the last line of that code: what comes back is a task_id, not a video.

This is the real subject of the chapter. The image API is synchronous — send, wait a few seconds, take the result. Video cannot be: it runs for minutes, and no HTTP connection wants to hang that long. So the shape becomes four steps:

1. Submit: POST /v1/video_generation, receive a task_id.
2. Poll: GET /v1/query/video_generation?task_id=, checking status repeatedly. Note task_id is a query parameter, not a path parameter. The status values are capitalized: Preparing, Queueing, Processing, Success, Fail.
3. Collect: on success, take the file_id out of the query result and call GET /v1/files/retrieve?file_id= to exchange it for a download address.
4. Download: put the file on disk immediately.

In crew terms: submitting hands the call sheet to the production office and the task_id is that sheet's number; polling is walking over to ask whether your shot is done; collecting is taking the number to the vault for the tape. Each step has its own way of going wrong — the sheet gets lost, the take is unusable, the vault has no such tape, the network drops on the way back.

One trap to name early: the same vendor may run two video APIs with different shapes at once. Besides the v1 line used here, MiniMax has a v2 line. The difference is shape, not capability — query is a path parameter, the status values are lowercase, and success returns the result address directly, with no collect step at all. Parse one line's response against the other's docs and you get a pile of nulls plus a suspicion that your network is broken. This course uses v1 only; in your own project, pin which line you use in both code and docs.

The VideoProvider.generate() from D1 wraps those four steps into a single await, which is pleasant to call. Today you take the wrapper apart, because what breaks is always the layer inside it: in production you see "this shot never came out," and fixing it means knowing which step it is stuck on. A bonus: each step becomes injectable, so every failure can be rehearsed offline — which is what today's lab does.

The manners and the cost of polling

Now write that loop. It looks simple, but three parts will bite you in production.

First, the interval cannot be fixed. Polling a five-minute queue at one second is three hundred useless requests. The query endpoint has its own rate limit, so you easily rate-limit yourself, then read "video generation failed: rate limited" and blame the generation endpoint. Use exponential backoff: start at five seconds and multiply by a factor (say 1.5) each round.

Second, the backoff needs a ceiling. Uncapped, by round ten you are checking every few minutes — the task finished long ago and you are still asleep. Around 20 seconds is reasonable.

Third, the timeout check goes before the sleep. The common version sleeps first and then checks the budget, which always oversleeps by a full round — 20 wasted seconds at a 20-second interval. The correct test is: would sleeping take me past the deadline? If so, give up now.

And one ordering issue: check the terminal states before the timeout. If a task returns success on the very last query and you discard it as a timeout, you have paid for it and lost the output.

export async function pollUntilDone(taskId, opts) {
  const startedAt = Date.now()
  const deadline = startedAt + opts.timeoutMs
  let waitMs = opts.initialWaitMs // 5000
 
  for (let attempt = 1; ; attempt += 1) {
    const { status, fileId } = await query(taskId)
 
    // Terminal states first: a task that succeeds on the last query must not be killed
    if (status === 'Success') {
      if (!fileId) throw new Error('task succeeded but returned no file_id')
      return fileId
    }
    if (status === 'Fail') throw new TaskFailedError(taskId)
 
    // The test is "would the sleep cross the deadline", not "did we overrun after sleeping"
    if (Date.now() + waitMs > deadline) {
      throw new TaskTimeoutError(taskId, Date.now() - startedAt)
    }
 
    await sleep(waitMs)
    waitMs = Math.min(Math.round(waitMs * opts.factor), opts.maxWaitMs) // backoff needs a cap
  }
}

import time
 
 
def poll_until_done(task_id: str, timeout_ms: int, initial_wait_ms: int = 5000,
                    factor: float = 1.5, max_wait_ms: int = 20000) -> str:
    started_at = time.monotonic() * 1000
    deadline = started_at + timeout_ms
    wait_ms = initial_wait_ms
 
    while True:
        status, file_id = query(task_id)
 
        # Terminal states first: a task that succeeds on the last query must not be killed
        if status == "Success":
            if not file_id:
                raise RuntimeError("task succeeded but returned no file_id")
            return file_id
        if status == "Fail":
            raise TaskFailedError(task_id)
 
        now = time.monotonic() * 1000
        # The test is "would the sleep cross the deadline", not "did we overrun after sleeping"
        if now + wait_ms > deadline:
            raise TaskTimeoutError(task_id, int(now - started_at))
 
        time.sleep(wait_ms / 1000)
        wait_ms = min(round(wait_ms * factor), max_wait_ms)  # backoff needs a cap

That code handles one task. A real episode has forty shots, and pushing forty at once hits the vendor's concurrency and quota limits, so you need a gate on how many run at a time. That gate and quota accounting are D9's material; today, get one task right.

The result file is temporary

Once the task succeeds you hold a file_id, and what that buys is not a video either, but a temporary download address.

The docs do not say how long it lives, so the only safe assumption is that it can expire at any moment. There is exactly one correct behavior: download to disk the instant you have it, and store the local path in your database and indexes — never the URL.

The violation is subtle. Everything works for months, because the download always happens seconds after the address arrives. Then you add an optimization that "collects all the addresses first, then downloads them together," and the incident arrives. Put the download inside the same function as the collect step, so the two are structurally inseparable; that beats a reminder comment.

After it lands, record two more things: which task_id this shot came from, and what prompt was used. Video is the most expensive artifact on this line, and being unable to answer "how was this generated" means paying again.

The failure classification table

The last section is about failure, the part of this chapter interviews ask about most. The criterion is not the leading digit of the status code but three questions: will waiting help? will changing the input help? or does a human have to step in?

The table's main value is isolating the class that burns money for nothing. Invalid parameters and moderation repeat the same error on every retry while consuming the rate limit that genuinely retryable calls need. Classifying those two correctly saves more than any amount of clever retry logic. Why moderation triggers and how to rewrite a prompt is D11's compliance topic; today, just get the class right.

Write the table as a function and the retry logic collapses to a single check:

// Back off only for errors classified as retry; everything else throws at once, wasting no budget
export async function withRetry(fn, { attempts, initialWaitMs, maxWaitMs }) {
  let waitMs = initialWaitMs
  for (let attempt = 1; ; attempt += 1) {
    try {
      return await fn()
    } catch (err) {
      const verdict = classify(err)
      if (verdict.action !== 'retry' || attempt >= attempts) throw err
      console.log(`attempt ${attempt} failed (${verdict.kind}), retrying in ${waitMs}ms`)
      await sleep(waitMs)
      waitMs = Math.min(waitMs * 2, maxWaitMs)
    }
  }
}

import time
 
 
# Back off only for errors classified as retry; everything else raises at once
def with_retry(fn, attempts: int, initial_wait_ms: int, max_wait_ms: int):
    wait_ms = initial_wait_ms
    for attempt in range(1, attempts + 1):
        try:
            return fn()
        except Exception as err:
            verdict = classify(err)
            if verdict["action"] != "retry" or attempt >= attempts:
                raise
            print(f"attempt {attempt} failed ({verdict['kind']}), retrying in {wait_ms}ms")
            time.sleep(wait_ms / 1000)
            wait_ms = min(wait_ms * 2, max_wait_ms)

One class is missing from the table and needs more care than any row in it: timeouts. A timeout is not a failure; it is "we do not know whether it succeeded." The task may still be queued, or may already have finished. Resubmit straight away and you may pay twice for the same shot. The correct order is: check the idempotency key for a finished artifact first, resubmit only if there is none. Idempotency keys and resuming from checkpoints are D8's workflow engine; today, write that sentence into the timeout log line so your future self knows the trap is there.

Finally, one batch-level ruling, consistent with yesterday: a single failed shot must not abort the batch. If shot seven of forty is blocked, record it, keep running, and report the failure list at the end. Aborting throws away the money spent on the first six — and since one shot costs a hundred-odd times one image, that arithmetic is harsher than yesterday's.

Source Reading

Hands-On Lab

Fault injection uses the course-wide INJECT variable, with five values today: timeout keeps the task queued forever, ratelimit makes the first submit hit the rate limit, blocked hits moderation, badrequest hits an invalid parameter, and taskfail drives the task to a failed terminal state. Injection applies only at the network egress layer, so the polling state machine and the classifier still run for real — you test your own code, not the stub.

starter/ leaves four exercises: two in the polling loop (timeout budget, backoff), one in the classification table, one in prompt assembly. Everything runs offline under MOCK=1 — the offline queue really walks the chain from queued to success and can be switched to "queued forever," so no line of your polling code is bypassed. Run it once unmodified first: you will see a constant interval, a timeout that leaves one second early, and a table that is retry all the way down. Those three symptoms are your to-do list.

1. Write submit, poll, collect and download as four separate functions, run the happy path once, and confirm the log shows the status moving from queued to success.
2. Run it again using a D3 approved still as the first frame, and compare the composition against the run without one.
3. Change polling to exponential backoff with a ceiling, inject a permanently queued task, and confirm the program gives up at the moment the budget runs out rather than early or never.
4. Write the classifier so the table's six codes land in retry, fix-input and alert respectively, and inject a rate limit to verify the backoff retry really happens.
5. Inject a moderation failure and confirm the program does not retry and gives a readable recommendation instead.

Interview Questions

Today's 3 questions are in the question bank below, focused on writing an async task client correctly, polling backoff and timeout cancellation, and error classification for generative APIs. Read the analysis before the key points — question 1 is close to mandatory for any role touching generative APIs, and listing the failure cases completely matters more than answering elegantly. The cn / global labels let you pick by target market.

Checklist and Tomorrow

• I can generate a single shot from a first-frame image plus a prompt, and explain the trade-off between text-to-video and image-to-video
• I can write a correct async task-polling loop that handles queuing, timeouts, backoff, and cancellation
• I can classify video API failures, distinguishing which are worth retrying from which retrying won't fix
• I can explain why a timeout is not a failure, and what must happen before any retry
• I can explain why the download has to live in the same step as the collect
• All 5 lab acceptance criteria pass
• I can answer at least 2 of the 3 interview questions without looking at the key points

Tomorrow (D5) we give these shots a voice. Each character gets its own timbre, the lines are synthesized in bulk, and then we do something whose ordering matters: use the measured real speech duration to correct the shot durations in reverse. Today's shots were generated at planned lengths, but how many seconds a line actually takes is only knowable after synthesis — and that "picture first or sound first" loop decides the ordering of the whole pipeline. D5 also produces the subtitle file from that timeline, the sole input to the editing bay the day after.