The first frame is a contract, not a thumbnail
Why controllable AI video starts by deciding what must remain true
TLDR: An input image in image-to-video AI serves as a visual specification that defines the intended starting state, meaning production success depends on carefully preparing and constraining this opening frame rather than just refining motion prompts. Moving forward, reliable video workflows can progressively layer structural controls like trajectories and multi-view references to balance motion freedom against identity preservation.
A good image-to-video demo can look like magic. Upload an image, describe some motion and receive a moving scene a few minutes later.
That interface hides the most important design decision. The input image is not simply inspiration for the video. It is the first concrete specification of the world the model is being asked to continue.
It establishes the subject, composition, colours, textures, proportions, camera position and visible geometry. The model then has to infer everything the image does not show: the back of an object, what exists outside the frame, how parts move, how the camera moves and which details should remain unchanged.
This makes the first frame much closer to a contract than a thumbnail. It defines what the generated video starts from. It does not guarantee that the model will honour the contract, and it cannot specify every future state, but ambiguity or errors in that frame propagate into the rest of the process.
For production systems, this changes where the engineering work belongs. The objective is not merely to write a better motion prompt. It is to construct a sufficiently explicit visual state, choose the right controls for what remains uncertain and verify that the generated clip preserves the details that matter.
What the model actually receives
Modern video generators commonly compress frames into a latent representation and learn to denoise a sequence of noisy video tokens. A text prompt supplies semantic guidance. In image-to-video systems, the reference image is encoded as an additional condition, often through latent concatenation, cross-attention or a visual encoder. [1] [4]
The practical consequence matters more than the architectural vocabulary: the model does not reconstruct a known future. It samples one plausible continuation from a learned distribution.
Suppose the first frame contains a stylised game character holding a tool. The visible pixels may specify the character's face, costume, tool and pose. They do not fully specify:
- the character's appearance after turning around;
- the geometry hidden behind the tool;
- how cloth or hair should react to motion;
- whether the camera or the character should move;
- which small visual details identify the character;
- what should appear when the camera reveals the edge of the scene.
Text can reduce some ambiguity, but prose is a poor format for exact geometry and timing. “The character turns quickly while the camera moves left” leaves many valid trajectories. How far does the camera travel? Which motion starts first? What remains fixed? What does “quickly” mean over a five-second clip?
This is one reason emerging systems add controls such as reference images, poses, depth, camera parameters, masks and point trajectories instead of trying to express everything in a prompt. [1] Motion Prompting, for example, represents movement as tracks through space and time. Those tracks can describe object motion, camera motion or both, giving the generator a signal that text cannot express precisely. [3]
The first-frame paradox
A stronger visual anchor usually gives more control over appearance. It can also make motion harder.
If conditioning is too weak, the model has freedom to produce dynamic motion but may change the subject, colours or geometry. If conditioning is too strong, it may preserve the image by barely moving it, copy parts of the first frame into later frames or fight the action described in the prompt. Research on “conditional image leakage” gives one mechanism for this behaviour: at highly noisy denoising steps, the clean reference can be much more informative than the noisy future-video latent, so the model learns to over-rely on the image and underproduce motion. [7]
Research on subject-consistent video treats this as an alignment problem between visual and textual conditions. The image says what the subject looks like; the text says what should happen. The model has to preserve the right information from one while following the other. [2] These demands can conflict. A reference portrait encodes not only identity but also pose, expression and lighting. Preserving all of it would prevent the requested performance.
The problem is even clearer for products and other rigid objects. A single view can show a recognisable object while leaving its three-dimensional structure underdetermined. As the viewpoint changes, a generator may warp the shape, merge parts, alter materials or invent hidden surfaces. Recent object-centric work addresses this by augmenting the opening image with additional views and explicitly encoding geometric as well as semantic features. [4]
That points to a useful distinction:
- Identity is what must remain recognisably the same.
- State is what may change during the shot.
- Motion describes how that state changes over time.
- Presentation describes how the camera observes it.
A single image entangles all four. Better production controls make them more explicit.
Build the opening frame for animation
An attractive still is not necessarily a good animation source.
Before sending a frame to a video model, it is worth checking five things.
1. Is the identity information visible?
If a character, product or game element must remain recognisable, its distinguishing features should be legible at the target resolution. A tiny logo or partially hidden tool may be obvious to a human who knows the asset, but weak conditioning for the model.
For important subjects, multiple reference views may be more useful than another paragraph of prompting. They reduce the amount of geometry the model must invent when the viewpoint changes.
2. Does the composition leave room for the requested motion?
A character placed against the edge of the frame has nowhere to move in that direction without revealing unseen content. A close crop may work for a subtle expression change but not for a wide camera orbit. The opening composition should anticipate the action and camera path.
3. Are the constraints compatible?
The frame, text and other controls should describe the same shot. If the frame implies a static close-up while the prompt requests a rapid full-body action and a large camera move, the model must choose which instruction to violate.
This is a specification error before it is a model error.
4. What is allowed to change?
Not every pixel needs to remain fixed. A useful shot specification names the invariants, such as character identity, product geometry, brand colours and gameplay state. It separates them from properties that should evolve, such as pose, expression, particles, lighting or camera angle.
This also improves evaluation. “Looks similar” is difficult to debug. “The shield shape and costume colours remain stable while the character moves from pose A to pose B” is testable.
5. Does the frame contain accidental commitments?
Image generators sometimes produce details that are visually plausible but wrong: malformed text, extra props, inconsistent UI, or a product feature that does not exist. Animation tends to preserve, reinterpret or amplify those errors. The opening frame should pass product and brand checks before the more expensive video stage begins.
At Replai, this is why storyboards and frames are intermediate artefacts rather than disposable previews. They create a point where creative intent, product fidelity and production rules can be checked before animation introduces another class of failure.
One frame is becoming several controls
The interface is already moving beyond “image plus prompt.”
First-and-last-frame generation lets a creator specify both the initial and desired final state, leaving the model to synthesize the transition. Commercial APIs now expose this directly. [6] It is useful when the end state matters: an object must arrive at a particular position, a camera move must finish on a specific composition, or a scene must connect to the next shot.
Multiple keyframes go further by specifying several states along the timeline. This can improve control over event order and narrative beats, but it does not remove the hard part. The model still has to produce plausible motion between anchors. Recent evaluation across keyframe-conditioned systems found a recurring trade-off: systems that reproduce keyframes faithfully can generate abrupt transitions, while systems that create smoother motion may drift away from the supplied frames. Performance also degrades as the number of constraints increases. [5]
More control is therefore not automatically better. Each additional condition can conflict with the others. Dense keyframes can overconstrain motion. A last frame that is geometrically incompatible with the first can force a morph rather than a believable action. A trajectory can demand movement that the initial composition cannot support.
The practical pattern is progressive constraint:
- start with the smallest set of conditions that express the shot;
- generate and inspect representative candidates;
- identify the actual failure mode;
- add a control that targets that failure;
- verify that the new control did not damage motion or overall quality.
This is more reliable than adding every available conditioning signal from the start.
Evaluate the contract, not only the clip
Generic video-quality scores are insufficient for production. A clip can be smooth, detailed and aesthetically pleasing while depicting the wrong product or changing a game mechanic.
Evaluation should separate at least four questions:
- Reference fidelity: Did the required identity, geometry and visual attributes survive?
- Motion adherence: Did the requested object and camera movement occur?
- Temporal quality: Is the motion coherent, or do details flicker, warp or disappear?
- Creative validity: Does the result still satisfy the brief and domain rules?
The distinction is operationally important. Each failure points to a different intervention. Poor reference fidelity may require a better opening frame, additional views, or a different model. Poor motion adherence may require trajectories or a simpler action. Temporal artefacts may be specific to the generator or sampling configuration. A creatively invalid result may reveal a problem upstream in the script or storyboard.
The useful metric is not how many videos the system can generate. It is the cost and review effort required to produce an accepted clip while preserving the required fidelity.
When the simple path is enough
This level of control has a cost. Extra references need to be prepared, checked and stored. Motion paths take time to define. Every new condition creates another opportunity for conflict and another variable to debug.
For an exploratory mood clip, a background element or a disposable internal sketch, one image and a short prompt may be the sensible choice. A team should add structure when a failure has a meaningful cost: the asset has to match a product, survive expert review, connect to another shot or be regenerated repeatedly.
The aim is not to construct the most elaborate conditioning stack. It is to spend control where variation would be expensive.
What this post does not cover
This is not a ranking of video models. Their capabilities and interfaces change too quickly, and the right choice depends on the shot. It also does not cover model fine-tuning, which becomes relevant when prompting and inference-time controls repeatedly fail on the same domain-specific requirement.
The remaining question here is how to carry a controlled visual state across more than one shot.
The boundary of the first frame
The first frame is powerful because it turns an abstract prompt into a concrete visual state. It is limited because a video is more than a sequence of appearances. It contains change, timing, causality, camera language and narrative structure.
For a single short shot, a strong opening frame plus an appropriate motion control may be enough. For a multi-shot creative, it is not. Each shot has to inherit some facts from earlier shots, deliberately change others and connect to the next part of the story.
That leads to the next problem: instead of asking one model to improvise the complete video, how do we compile a creative idea into shots, visual anchors, motion constraints and checks that can be generated and repaired independently?
That is the subject of the next post.
Sources
- Controllable Video Generation: A Survey
- Phantom: Subject-Consistent Video Generation via Cross-Modal Alignment
- Motion Prompting: Controlling Video Generation with Motion Trajectories
- ConsID-Gen: View-Consistent and Identity-Preserving Image-to-Video Generation
- KeyFrame-Compass: Towards Comprehensive Evaluation of Keyframe-Conditioned Video Generation
- Wan first-and-last-frame API documentation
- Identifying and Solving Conditional Image Leakage in Image-to-Video Diffusion Model