TLDR: TTOM (Test-Time Optimization and Memorization) is a novel, training-free framework that significantly improves compositional video generation by Video Foundation Models. It achieves this by optimizing lightweight parameters during inference, guided by spatiotemporal layouts derived from text prompts. Crucially, TTOM incorporates a parametric memory to store and reuse historical optimization contexts, enhancing efficiency, scalability, and generalization. Experimental results demonstrate substantial improvements in text-video alignment and overall quality on leading benchmarks.
Video Foundation Models (VFMs) have made incredible strides in creating realistic and vivid videos. However, these advanced models often face significant challenges when asked to generate complex scenes with multiple objects, specific motions, or intricate spatial relationships – what researchers call ‘compositional scenarios’. Imagine asking a model to show ‘four pandas munching bamboo in a garden’ or ‘a robot and a sorcerer stealthily approaching each other’; current models can struggle to accurately represent these detailed interactions.
A new research paper introduces a framework called Test-Time Optimization and Memorization (TTOM) to address these limitations. TTOM is a training-free approach designed to enhance the ability of VFMs to align their outputs with precise spatial and temporal layouts during the video generation process itself, rather than requiring extensive pre-training.
How TTOM Works: Two Core Components
TTOM operates on two main principles: Test-Time Optimization (TTO) and a Parametric Memory mechanism.
Test-Time Optimization (TTO): Unlike methods that directly alter the video’s underlying data or attention mechanisms, TTOM integrates and optimizes new, lightweight parameters within the VFM during inference. This optimization is guided by a ‘layout-attention objective’. Essentially, when you give the model a text prompt, a large language model (LLM) first translates it into a detailed spatiotemporal layout – a plan of where objects should be, how they should move, and their relationships over time. TTOM then uses this layout to fine-tune the new parameters, ensuring the generated video adheres closely to the prescribed scene. This approach avoids disrupting the VFM’s core visual generation capabilities, which can sometimes lead to degraded video quality in other methods.
Parametric Memory: One of TTOM’s most innovative features is its ability to remember past optimization contexts. In real-world scenarios, users often generate a stream of related videos. Traditional methods treat each generation as an isolated event, discarding valuable learned information. TTOM, however, formulates video generation within a ‘streaming setting’. After a successful optimization for a particular compositional pattern (like a specific motion or object interaction), the learned parameters are stored in a parametric memory, indexed by keywords from the prompt. If a similar prompt appears later, TTOM can retrieve these stored parameters, use them as a starting point, and even further optimize them for better adaptation. This mechanism supports flexible operations like inserting new patterns, reading existing ones, updating them, and deleting less frequently used entries, making the process more efficient and scalable over time.
Also Read:
- Unlocking Deeper Reasoning in AI Models with Test-Time Adaptation
- MetaVLA: A Unified Training Approach for Generalist Robot Agents
Benefits and Performance
The researchers found that TTOM effectively disentangles compositional world knowledge, leading to powerful transferability and generalization. This means the model can apply learned patterns to new, unseen scenarios and perform well across a wide range of complex prompts.
Extensive experiments were conducted on two major text-to-video benchmarks: T2V-CompBench and VBench. TTOM consistently surpassed baseline models, demonstrating significant improvements in compositional scenarios. For instance, it achieved relative improvements of 34% over CogVideoX-5B and 14% over Wan2.1-14B on T2V-CompBench’s overall average performance. The gains were particularly pronounced in challenging categories like ‘Motion’ and ‘Numeracy’, with improvements of up to 63% and 37% respectively on CogVideoX-5B, and even higher on Wan2.1-14B.
Beyond compositional accuracy, TTOM also showed clear improvements in semantic consistency on VBench, enhancing aspects like object classification, multi-object handling, and color/spatial relation fidelity. This indicates that TTOM not only gets the complex relationships right but also maintains overall visual quality and coherence.
The framework offers a flexible trade-off between video quality and efficiency, making it a practical solution for complex video generation tasks. For more details, you can read the full research paper: TTOM: Test-Time Optimization and Memorization for Compositional Video Generation.
