Text-to-video is a generative AI technique that creates a video clip from a written prompt. It extends text-to-image generation across time: the model must generate many frames that stay consistent in objects, lighting, style and motion. Most leading systems use diffusion models built on transformers, often called diffusion transformers.
This page explains how the technique works. For the broader field, including avatars, dubbing and editing, see What is AI Video Generation?
Key takeaways
- Text-to-video generates motion, not just a still image.
- Temporal consistency, keeping things the same across frames, is the core challenge.
- Many models process video as space-time patches with a transformer.
- Prompts should describe camera movement and action, not only the scene.
- Clips are typically short; longer videos are assembled from multiple generations.
Text-to-video explained in simple terms
A video is a flipbook of images. Generating one good picture is hard; generating dozens that flow smoothly, where a person walks without their shirt changing colour, is much harder. A text-to-video model plans the whole flipbook together, making sure each page follows naturally from the last, while matching your description.
How text-to-video works
- Encode the prompt with a text encoder.
- Compress video into a smaller latent space with a video autoencoder, shrinking both frame size and the number of frames to process.
- Split into space-time patches: small cubes covering an area of the frame across several moments.
- Denoise with a transformer: starting from noise, a diffusion transformer removes noise over many steps. Attention across patches lets the model relate each part of the video to every other part, in space and time, which helps consistency.
- Decode the final latent into full frames.
- Optional stages: upscaling, frame interpolation for smoother motion, and audio generation.
OpenAI’s Sora preview in February 2024 brought wide attention to the diffusion transformer approach for video, and other labs, including Google with Veo, released competing models.
Prompting for video
Video prompts need more than a scene description. Include:
- Subject and action: “a golden retriever runs across a beach”.
- Camera: “slow tracking shot”, “drone shot rising”, “static close-up”.
- Environment and time: “at sunset, light waves”.
- Style: “cinematic, 35mm film look” or “3D animation”.
- Pacing: “slow motion”, “timelapse”.
A worked example
Prompt: “Slow drone shot rising above a Pune street market at dawn, vendors setting up colourful fruit stalls, soft golden light, light mist, cinematic.”
- The camera instruction (rising drone shot) tells the model how the viewpoint should move.
- The action (vendors setting up) adds believable motion.
- The lighting and mist set a consistent mood across frames.
If the first result has inconsistent stalls, generating from a reference image (image-to-video) or shortening the clip usually improves consistency.
Related techniques
| Technique | Input | Why use it |
|---|---|---|
| Image-to-video | Still image plus prompt | Better control of the first frame and the look |
| Video-to-video | Existing clip | Restyle footage while keeping motion |
| Extension | A generated clip | Continue a shot beyond its original length |
| Keyframe control | Start and end images | Guide the motion between two states |
Limitations
- Physics errors: objects may merge, float or change shape.
- Hands, faces in motion and on-screen text are common failure points.
- Consistent characters across multiple shots are hard to maintain.
- Fine control over timing and choreography is limited.
- Generation is compute-heavy and can be slow.
- Realistic outputs carry deepfake risks; see synthetic media.