Why controllable AI video looks more like compilation than prompting

TLDR: Reliable multi-shot AI video requires treating creative concepts as compilable programs—decomposed into explicit shot specifications, visual anchors, and selective continuity memory—rather than relying on single prompts to handle pacing, camera work, and identity. This structured separation makes production failures cheaper to isolate and fix by allowing teams to inspect intermediate states, maintain persistent narrative facts across cuts, and locally regenerate individual shots without restarting the entire video.

A text-to-video prompt can produce an impressive clip. It is a poor production plan.

A finished creative has structure. It introduces something, changes the viewer's understanding, moves between viewpoints and lands on a deliberate final state. Products and characters must remain recognisable. Actions need to occur in the right order. Some details must persist across cuts while others are supposed to change.

A single prompt leaves the model to decide all of this at once: the shots, their duration, the camera, the transitions, the identity of recurring elements and the allocation of the story across time. When the result is wrong, the only obvious repair is to prompt again and hope for a better sample.

There is another way to frame the task. Treat the creative as a program that has to be compiled into a sequence of shots. Each shot receives visual anchors, motion instructions and constraints. The generation model executes those local specifications. Evaluation checks whether the execution still matches the original intent.

This adds machinery, and not every video needs it. But when fidelity, narrative order or review cost matters, decomposition gives us something a one-shot prompt does not: a place to inspect and repair the work.

A video is not one continuous instruction

Most real videos are assembled from shots. A close-up, a wide view and a reverse angle can depict the same event without sharing continuous pixels. The viewer understands the relationship because characters, objects, actions and context carry across the cuts.

Current video generators are strongest at producing short, continuous clips. Multi-shot generation asks for a different capability. The system has to decide which information persists after a cut, retrieve it when an entity returns, and distinguish a deliberate change of viewpoint from an accidental change of identity.

Recent research approaches this in several ways.

OneStory generates the next shot conditioned on a compact selection of relevant frames from previous shots. Its frame-selection and adaptive-conditioning modules act as a memory: they retain useful visual context without forcing the model to attend to every earlier frame. [4]

MultiShotMaster changes how a transformer represents position across shots. It introduces one positional scheme for narrative order and another for placing reference and grounding signals in space and time. [2]

STAGE keeps the workflow modular. It turns a story into a structured storyboard, predicts start-and-end frames for each shot, and hands those visual anchors to an image-to-video model. [1]

These are different technical bets: retrieve previous context, teach one generator an explicit multi-shot representation, or compile the story into smaller generation tasks. None has removed the core requirement. The system must represent what persists, what changes and when.

Start with a shot specification, not a prompt

The first frame of a shot defines a visual state, as discussed in the previous post. A multi-shot workflow also needs to define the relationships between those states.

A useful shot specification might contain:

shot_id: 03
purpose: reveal the obstacle
subject: hero_A
required_state:
  - costume_variant = blue
  - tool = hammer
  - environment = workshop
start_anchor: frame_03_start.png
end_anchor: frame_03_end.png
subject_motion: hero turns toward the blocked exit
camera: slow push-in, eye level
must_preserve:
  - hero identity
  - tool geometry
  - environment colours
must_change:
  - gaze direction
  - distance to camera
handoff_to_next_shot:
  - obstacle remains unresolved

The schema is not important. The separation is.

The shot's narrative purpose is different from its visual state. The visual state is different from motion. Invariants are different from intended changes. The handoff to the next shot is different from the local end frame.

Putting all of that into prose makes it difficult to validate. Keeping it structured allows each field to drive the appropriate tool, generation control or check.

At Replai, our workflow already separates ideation, scripting, scene decomposition, storyboards or frames, animation and evaluation. The point of those stages is not to produce a longer agent trace. Each intermediate artefact should expose a decision that would otherwise remain hidden inside a generated video.

Visual anchors are necessary, but insufficient

Storyboards are useful because they turn narrative intent into inspectable images. A reviewer can catch a wrong character, missing product state or invalid composition before paying to animate the shot.

One anchor per shot still leaves most of the motion unspecified. The model may preserve the opening frame and invent the wrong ending. It may follow both anchors but connect them with a morph that no camera or object could produce.

Start-and-end pairs reduce that ambiguity. STAGE uses them at two levels: the pair within a shot describes its progression, while the end of one shot and the start of the next describe the cut or transition. [1] Commercial generation APIs also expose first-and-last-frame control for individual clips, so this pattern is not confined to research systems. [6]

More anchors do not guarantee a better video. KeyFrame-Compass evaluates whether keyframes appear, remain faithful, occur in order, appear at the intended time, persist appropriately and avoid accidental duplication. Its experiments found that denser constraints can reduce instruction adherence. Models that closely reproduce anchors may produce worse transitions, while smoother outputs can drift from the visual plan. [5]

The practical lesson is to assign anchors to decisions, not to arbitrary timestamps. Use one when the state must be visible and verifiable: a reveal, a product change, the outcome of an action or the composition required for the next cut.

Cross-shot consistency needs selective memory

Passing the complete history to every shot is tempting. It is also expensive and noisy.

The next shot rarely needs every earlier pixel. It may need the main character's identity from shot one, an object introduced in shot two and the environment state at the end of shot four. Other details should be forgotten because the scene, time or viewpoint has changed.

This makes cross-shot context a retrieval problem as much as a generation problem. A production system can maintain an explicit continuity state containing:

• reference images or asset identifiers for recurring entities;
• attributes that must remain invariant;
• state changes that have already occurred;
• relevant end frames from earlier shots;
• the current environment, time and camera relationship;
• unresolved narrative conditions needed later.

Before generating a shot, the system retrieves the subset required by that shot's specification. After accepting the result, it updates continuity state with the verified changes.

This resembles OneStory's use of semantically relevant frames from prior shots, although a production workflow can also store symbolic state and approved assets outside the model. [4] The distinction matters. A model's latent memory is difficult to inspect. An explicit continuity record can be reviewed, versioned and corrected.

The failure mode to avoid is accidental persistence. A model may carry an irrelevant background object forward because it appeared recently, or preserve a costume that the script says should change. Consistency is not “keep everything the same.” It is preserve the right facts across deliberate change.

Camera and editing belong in the specification

A sequence of consistent shots can still be badly directed.

Camera motion affects what geometry the model has to invent. Shot scale controls what details are visible. A cut determines whether discontinuity is an error or the intended editing language. These are not cosmetic parameters added after generation.

ShotDirector combines explicit camera parameters, including six-degree-of-freedom pose and intrinsic settings, with prompts that describe editing patterns. [3] Motion Prompting and related trajectory-conditioned methods address a similar issue within a shot: text alone is a weak representation for precise movement. [7]

Most production teams will not author a complete camera path numerically. They do need an intermediate representation that can be translated into model-specific controls. “Slow push-in,” for example, may become a camera trajectory for one generator and a constrained prompt plus start/end framing for another.

This translation layer also preserves provider optionality. The creative specification remains stable while adapters map it to whichever controls a model exposes. That is useful only if the adapters are tested; a generic abstraction that hides meaningful differences between models creates false portability.

Local generation makes local repair possible

The strongest operational argument for decomposition is not that each clip will be better on the first attempt. It is that failure becomes cheaper to isolate.

If shot four changes a product detail, the system can regenerate shot four. If the transition between shots four and five is implausible, it can revise the end anchor, the next start anchor or the transition instruction. It does not need to discard an otherwise acceptable opening.

This only works if the system tracks dependencies. Changing an accepted frame can invalidate later shots that used it as a reference. A continuity state can become stale. A repaired action may alter the story timing.

A practical orchestration layer therefore needs:

• versioned scripts, shot specifications and anchors;
• provenance linking each clip to its inputs, model and parameters;
• dependency tracking between shots;
• separate checks for local quality and whole-video coherence;
• a policy for which downstream artefacts must be regenerated after a change.

Without this, decomposition replaces one opaque generation with a collection of loosely related files.

Evaluate the plan and the execution separately

A visually good video can execute the wrong plan. A faithful execution can still look poor.

The evaluation should preserve that distinction.

At the shot level, check reference fidelity, required entities, motion, camera behaviour and technical artefacts. At the sequence level, check shot order, continuity state, narrative progression, transitions and pacing. Then apply the business or product criteria that determine whether the creative is actually usable.

This makes evaluator output actionable. A missing narrative beat points back to the script or shot plan. Identity drift suggests a reference or continuity problem. A correct pair of anchors connected by implausible motion points to the generator or motion control. A valid sequence that violates a product rule belongs to domain evaluation.

A single overall quality score hides these distinctions. It may rank outputs, but it will not tell the workflow what to repair.

One model can still execute the plan

Explicit planning does not require every shot to be generated by a separate model call. Native multi-shot generation is becoming credible. HoloCine generates an entire scene while localising prompts to particular shots and using dense attention within shots but sparse communication between them. [8] Seedance 1.0 reports native multi-shot generation using interleaved multimodal positional encoding and shot-level captions. [9] The latter is a vendor-authored technical report, so its capability claims should be treated as reported evidence rather than independent validation.

These systems weaken any categorical claim that shot-by-shot generation will always be better. A holistic model can share context across cuts and may avoid seams created by independently generated clips. It also creates a larger retry unit and can expose fewer intermediate surfaces for approval or repair.

The durable recommendation is therefore narrower: keep the shot plan, continuity state and acceptance criteria explicit, even when one model executes several shots internally. The execution boundary can change as models improve without forcing the production specification to disappear with it.

When decomposition is the wrong choice

A one-shot generator is often enough for a short atmospheric clip, a disposable concept sketch or a scene where variation is welcome. Splitting a five-second shot into a graph of versioned artefacts would add more work than it saves.

Decomposition becomes useful when at least one of these is true:

• the video has several distinct narrative beats;
• recurring subjects or product states must stay consistent;
• reviewers need to approve intermediate decisions;
• local regeneration is materially cheaper than restarting;
• failures can be mapped to explicit rules or controls;
• the workflow will be repeated often enough to justify the tooling.

The architecture should earn its cost through accepted-output rate, lower review effort, cheaper repair or greater experimental capacity. More stages are not evidence of a better system.

What this post does not cover

This post describes inference-time planning and orchestration. It does not argue that one fixed pipeline fits every video model, and it does not claim that modular generation will always outperform a future end-to-end multi-shot model.

The two approaches are already converging. End-to-end models are acquiring references, memory, shot structure and camera controls. Modular systems are using stronger multimodal models to generate and revise their plans. Even if one model eventually produces the complete sequence, an explicit plan may still be useful for review, provenance and repair.

There is also a point where better prompts, references and orchestration stop solving a repeated domain-specific failure. A model may consistently misunderstand a particular visual language, asset family or motion pattern. That is where adaptation and fine-tuning become worth considering, along with their data, evaluation and maintenance costs.

That is the next technical question in this series.

Sources

1. STAGE: Storyboard-Anchored Generation for Cinematic Multi-shot Narrative
2. MultiShotMaster: A Controllable Multi-Shot Video Generation Framework
3. ShotDirector: Directorially Controllable Multi-Shot Video Generation with Cinematographic Transitions
4. OneStory: Coherent Multi-Shot Video Generation with Adaptive Memory
5. KeyFrame-Compass: Towards Comprehensive Evaluation of Keyframe-Conditioned Video Generation
6. Wan first-and-last-frame API documentation
7. Motion Prompting: Controlling Video Generation with Motion Trajectories
8. HoloCine: Holistic Generation of Cinematic Multi-Shot Long Video Narratives
9. Seedance 1.0 technical report