Concurrency and Quotas: Starting Multiple Episodes at Once Without Blowing Through Any Provider's Limits

The unit of parallelism: per episode or per shot

Before installing gates, settle one thing: what is the unit of parallelism?

Per episode is the easy option: five processes, each running one episode's six stages. Isolation is clean and one episode's collapse does not touch the other four; the downside is a large failure radius. An episode that blows up at stage five leaves two hours of artifacts hanging, and the money is already spent.

Per shot cuts finer: every shot of every episode enters one pool, and whatever's dependencies are satisfied runs. Utilization is high — one episode waits on its script while another's forty shots saturate the speech quota; the downside is orchestration complexity: you maintain the dependency relationships yourself and must still answer "what percentage of episode three is complete?"

This course chooses parallel by shot, reported by episode. Execution granularity is the shot, because only that saturates quota; but external reporting, the cost ledger and failure re-runs are per episode, because readers care whether episode three can ship, not how shot seventeen of episode three is doing. The combination costs one extra layer: tasks carry a label for the episode they belong to, and statistics aggregate by label. Those few dozen extra lines are worth it.

And a counterintuitive judgment in passing: parallelism should not be decided by your CPU core count. There is almost no local computation on this line; it is all waiting on the network. What really decides parallelism is quota, plus how much you are willing to pay for "how much has to be redone when one thing fails."

One gate per provider

A gate has two distinct parts, and people often merge them into one and then debug a pile of inexplicable behavior.

The first is the concurrency cap, governing how many requests are in flight at any instant. You choose that number, to protect your own process, memory and wallet. The second is the token bucket, governing how many requests go out per minute — that number is the vendor's, straight from the RPM table above.

Why not just one? With only a concurrency cap, if every request returns quickly you can send hundreds a minute and still hit the limit. With only a token bucket, twenty per minute is correct but all twenty might be in flight at once, leaving twenty pending video downloads in memory. You need both.

The token bucket's shape deserves a sentence. The laziest rate limiter sleeps before each request — twenty RPM means sleep three seconds each time. That does limit the rate, and it also throws away burst capacity: the real quota lets you fire twenty times at the start of a minute and stay quiet for fifty seconds. A token bucket preserves the burst, at the cost of maintaining a window and a counter.

// Token bucket: capacity tokens issued per window. The key is that tryTake does not block -
// on failure it returns false at once so the scheduler can do something else instead of waiting.
class TokenBucket {
  constructor(capacity, windowMs = 60_000) {
    this.capacity = capacity
    this.windowMs = windowMs
    this.tokens = capacity
    this.windowStart = Date.now()
  }
 
  tryTake() {
    const now = Date.now()
    if (now - this.windowStart >= this.windowMs) {
      this.windowStart = now
      this.tokens = this.capacity
    }
    if (this.tokens <= 0) return false
    this.tokens -= 1
    return true
  }
}
 
// RPM comes from the official rate-limit page; the concurrency cap is yours. Two different things.
const QUOTA = {
  image: { rpm: 10, concurrency: 2 },
  tts: { rpm: 20, concurrency: 3 },
  video: { rpm: 20, concurrency: 2 },
}

import time
 
 
class TokenBucket:
    """capacity tokens per window. try_take never blocks: on failure the scheduler moves on."""
 
    def __init__(self, capacity: int, window_s: float = 60.0) -> None:
        self.capacity = capacity
        self.window_s = window_s
        self.tokens = capacity
        self.window_start = time.monotonic()
 
    def try_take(self) -> bool:
        now = time.monotonic()
        if now - self.window_start >= self.window_s:
            self.window_start = now
            self.tokens = self.capacity
        if self.tokens <= 0:
            return False
        self.tokens -= 1
        return True
 
 
# RPM comes from the official rate-limit page; the concurrency cap is yours. Two different things.
QUOTA = {
    "image": {"rpm": 10, "concurrency": 2},
    "tts": {"rpm": 20, "concurrency": 3},
    "video": {"rpm": 20, "concurrency": 2},
}

Note that tryTake is non-blocking. That single detail decides whether the whole scheduler works, and section four shows the consequence.

Rate limiting is not an error, it is a signal

MiniMax's response body carries base_resp.status_code, where 0 is success; today's relevant codes are 1002 rate limiting and 1039 (rate limiting on the TPM dimension). Many people bundle those with 1004 authentication failure, 1008 insufficient balance and 2013 invalid parameter into one catch that logs a line, which is a waste.

Split the codes into two classes and it becomes clear: those a retry might fix (rate limiting, server errors) and those a retry will never fix (authentication, balance, parameters, moderation). Back off and retry the first; never retry the second even once — retrying just commits the same error five times in a minute while occupying quota.

For the retryable class, backoff must do three things, and skipping any one means it is not done:

First, exponential backoff with jitter. If five episodes hit the limit together and all retry after a fixed one second, they wake together and collide again. Jitter multiplies each backoff by a random factor between 0.7 and 1.3 to spread them out.

Second, do not hold an execution slot while backing off. That is the next section's subject.

Third, slow down deliberately after being rate limited. This is the one most often missed. Being limited means your send rate exceeded what the vendor will accept right now, so charging back at the old speed after the backoff just collides again. The right move is to penalize the token bucket: issue half the tokens for the next window or two, and restore once the collisions stop.

// Exponential backoff plus jitter. Jitter stops five episodes waking together and colliding again.
const backoffMs = (attempt) => Math.round(300 * 2 ** (attempt - 1) * (0.7 + Math.random() * 0.6))
 
function onFailure(task, err, gate) {
  const rateLimited = err.statusCode === 1002 || err.statusCode === 1039
  if (rateLimited) {
    gate.stats.rateLimited += 1
    gate.bucket.penalize() // half tokens for the next two windows
  }
  // auth 1004, balance 1008, params 2013, moderation 1026/1027 are never retried
  if (!err.retryable || task.attempt + 1 >= 5) return scheduler.fail(task, err)
  scheduler.requeue(task, backoffMs(task.attempt + 1))
}

import random
 
 
def backoff_s(attempt: int) -> float:
    """Exponential backoff plus jitter, so five episodes do not wake together and collide."""
    return 0.3 * 2 ** (attempt - 1) * random.uniform(0.7, 1.3)
 
 
def on_failure(task, err, gate, scheduler) -> None:
    if err.status_code in (1002, 1039):
        gate.stats.rate_limited += 1
        gate.bucket.penalize()  # half tokens for the next two windows
    # auth 1004, balance 1008, params 2013, moderation 1026/1027 are never retried
    if not err.retryable or task.attempt + 1 >= 5:
        scheduler.fail(task, err)
        return
    scheduler.requeue(task, backoff_s(task.attempt + 1))

How a single async task is submitted, polled and abandoned on timeout was D4's material and is not repeated; today is only about how the gate opens when many tasks arrive at once.

Priority and fairness: three tiers, and how not to starve

Rush jobs always exist. Operations says at midnight that episode six is needed tomorrow morning, so that episode jumps the line. The mechanism is direct: three task tiers — rush, normal, low — scheduled by tier first and, within a tier, first come first served by enqueue time.

The real trap is the next step: low-priority tasks may never get scheduled at all. As long as rush jobs keep arriving, the earliest low-priority batch stays at the tail forever. That is starvation.

The cheapest anti-starvation mechanism is aging: the longer a task waits in the queue, the higher its effective priority. Wait one tier's worth of time and it rises one tier. Even under a constant stream of rush jobs, long-waiting tasks float up.

Aging needs one hard-coded constraint: promotion has a ceiling and must never reach the rush tier. Otherwise after half an hour the queue is all rush jobs and the tier means nothing. This course lets low rise only to normal, reserving rush for human line-jumping.

There is a subtler trap that makes priority completely ineffective while looking fine in the logs. If tasks are dispatched first and then wait for quota inside, then when all three worker slots hold low-priority tasks that are all waiting on tokens, a rush job never even gets dispatched — being first in the queue is useless if nobody comes to collect it.

That is why the previous section stressed that tryTake must be non-blocking. The correct structure is that the scheduler asks the gate for permission before handing out work: it dispatches only on success, and skips to the next candidate on failure. Then, when quota is tight, tokens go by priority to the task that most deserves them.

Long tasks should not hold a worker slot

Video generation is the only long task on this line. Submit, receive a task_id, then wait — the vendor publishes no typical duration, so you work from measurement, and it fluctuates with the queue.

The problem is that if your code is one await video.generate(...), that execution flow is pinned by a task doing nothing but waiting. Three worker slots and eighteen video tasks: the moment the first three videos are submitted, the whole line stalls. Scripts could run, dubbing could run, the timeline could run — and no slot is free to run them.

The fix splits a long task's lifecycle in two: submission occupies a worker slot (it is a short HTTP request), and waiting does not. The gate's ticket stays held during the wait — the vendor really does have a task pending, and you should not lie to yourself about that — but your execution flow should return to the queue immediately for other work.

// start() returning undefined means done; returning a function means "handed to the vendor, now just waiting".
const task = {
  id: 'ep6-clip-s01',
  gate: 'video',
  start: async () => {
    // Note: no await. Wrap the promise in a function and hand it back; the slot is returned now.
    const inflight = providers.video.generate({ prompt, outPath, durationSec: 6 })
    return async () => {
      await inflight
      run.record({ nodeId: 'ep6-clip-s01', kind: 'video', units: 6 })
    }
  },
}
 
const pending = await task.start()
if (pending) {
  log('submitted to the vendor, releasing the worker slot for other work')
  background.push(pending().then(() => scheduler.complete(task)))
} else {
  scheduler.complete(task)
}

import asyncio
 
 
async def start_clip(providers, run, prompt, out_path):
    """None means done; a coroutine means "handed to the vendor, now just waiting"."""
    # create_task attaches it to the event loop, and this execution flow returns immediately
    inflight = asyncio.create_task(
        providers.video.generate(prompt=prompt, out_path=out_path, duration_s=6)
    )
 
    async def wait() -> None:
        await inflight
        run.record(node_id="ep6-clip-s01", kind="video", units=6)
 
    return wait
 
 
pending = await task.start()
if pending is not None:
    log("submitted to the vendor, releasing the worker slot for other work")
    background.append(asyncio.create_task(finish(pending, task)))
else:
    scheduler.complete(task)

The same logic applies to any wait: waiting for a token, waiting out a backoff, waiting on a downstream dependency — none should hold a worker slot. The test is easy to remember: if your code does nothing during that interval, it should not be holding a slot.

How this relates to the message-queue material

Readers with a distributed-systems background will find this familiar: is this not a priority work queue plus a rate limiter? Yes. In production, what you wrote today is usually a Redis Streams consumer group, or an off-the-shelf task queue, plus rate limiting at the gateway.

So why write it yourself? Because the judgment this course teaches is not in the queue itself but in what dimension the gate buckets by. An off-the-shelf queue gives you concurrency control, but it does not know that your images and speech come from two different quota pools, nor that your video tasks spend eighty percent of their time waiting rather than computing. Only you know those, and only you can configure them.

If you are building a real production system, the recommended boundary is: use something off the shelf for queue persistence, consumer groups and failure redelivery, and write the gate and priority policy yourself — it is one or two hundred lines and is the part that most needs tuning against your bill. This platform's "30 Days from Frontend Engineer to Agent Engineer" course has a day on Redis Streams consumer groups and the pending list; if you want to replace today's in-memory queue with a cross-process one, that is the cheapest place to pick it up.

Source Reading

Hands-On Lab

The lab compresses the rate window to six seconds by default so the whole flow finishes in a dozen seconds, and no quota number is changed; setting RATE_WINDOW_MS to 60000 gives real behavior. The starter leaves five exercises, and running it unmodified shows five of six checks failing, one flipping to a pass per exercise.

Fault injection uses the course-wide INJECT variable, with one value today:

MOCK=1 pnpm start                    # happy path
INJECT=ratelimit MOCK=1 pnpm start   # first 1 image and first 2 speech calls return status_code=1002

Injection applies only at the provider egress, and the gate, scheduling, priority and aging logic all run normally — that is what distinguishes it from skipping the step.

1. Implement non-blocking admission with a token bucket and a concurrency semaphore, run once, and confirm "blocked by token" is no longer zero and peak in-flight stays within the cap.
2. Queue five episodes at once and watch the image row in the gate statistics — it will be the first to jam, matching the arithmetic in the walkthrough.
3. Run again with INJECT=ratelimit, watch the backoff log, and confirm the backoff has jitter, the bucket is penalized, and tasks eventually succeed.
4. Add priority ordering and aging, and watch the rush job's jump count and the starvation-protection log appear together.
5. Change video tasks to return the worker slot after submission, and compare total elapsed time before and after.

Interview Questions

Today's three questions are in the question bank below, focused on parallel granularity versus failure radius, per-provider rate-limit design, and correct handling of rate-limit responses. Read the analysis before the key points — practicing the derivation beats memorizing them. The cn / global labels let you pick by target market.

Checklist and Tomorrow

• I can run multiple episodes in parallel with a queue, and cap concurrency and rate separately per provider
• I can prioritize tasks so an urgent rush job doesn't starve a queued long-running task
• I can make the right backoff and degradation decisions among over-limit, rate-limited, and queue-timeout conditions
• I can explain what the concurrency cap and the token bucket each govern, and why both are needed
• I can explain why waiting for quota must not hold a worker slot, or priority silently stops working
• All 5 lab acceptance criteria pass
• I can answer at least 2 of the 3 interview questions without looking at the key points

Tomorrow (D10) we build the screening room: a web console for reviewing images and cuts shot by shot, editing lines, and regenerating a single shot. The order is deliberate — today we scaled output to five parallel episodes, and once volume rises, human checking becomes the new bottleneck; and for a human to intervene, the first requirement is being able to see. More importantly, you now know that every regeneration is deducted from quota, so tomorrow's arithmetic of "which nodes must re-run when one line changes" finally means something: get it wrong and the time saved is repaid in duplicate calls.