Matrix-Game 2.0 generates interactive video at 25 FPS on a single H100

Interactive world models are a way to teach AI to sense the world rather than only describe it in words. Until recently, three obstacles stood in the way: there was not enough quality data with precise action labels; classic video diffusion models were too slow to compute and "forgot" the start of the clip; and errors compounded from frame to frame. Matrix-Game 2.0 offers a clear, practical answer: an autoregressive model with a small number of steps, streaming generation on the fly at 25 FPS, and tight control by mouse and keyboard.

Why model the world in video at all

• So that agents in games and simulators can plan by seeing the consequences of key presses instead of reasoning from a schematic.
• So that training for driving or navigation goes faster and more safely.
• So that LLMs and multi-agent systems get a visual simulator where hypotheses can be tested from prompts, without the physical world.

Instead of long prompts, Matrix-Game 2.0 keys on what actually governs the dynamics: visual frames and real actions.

Where the training data comes from

The team built two data-production pipelines, one in Unreal Engine and one in GTA5. Both emit video along with frame-accurate logs of key presses, camera motion and physics.

• Unreal Engine: a navigation mesh lays out trajectories that never get stuck, agents are trained with PPO under rewards for diversity and collision avoidance, and the camera is driven with near-zero rotation error. Rendering and recording run in parallel threads, which raises throughput even on a single GPU.
• GTA5: a custom plugin records RGB, mouse and keyboard, and vehicle motion parameters at the same time. Camera views, weather and traffic density are switchable, and the plugin helps keep behavior inside drivable areas.

Together that comes to roughly 1,200 hours of varied clips with actions tied precisely to frames.

What is inside the model

Matrix-Game 2.0 is a world model with no text input: it takes a single reference image plus actions and outputs video that obeys the physics of the scene. A 3D Causal VAE compresses frames across space and time, saving compute. The diffusion transformer (DiT) receives:

• continuous mouse actions, embedded through an MLP and temporal self-attention;
• discrete key presses, fed in through cross-attention with RoPE positional features so that long-range dependencies do not break.

The key to real time is distilling a bidirectional teacher into a causal few-step student with Self-Forcing. The student learns to predict the next tokens from its own past output rather than from ground truth, which cuts error accumulation. A KV cache holds the most recent tokens and actions in a sliding window: that gives continuous streaming generation without cost growing quadratically with length.

How it feels in practice

• Speed: 25 frames per second on a single H100, and several minutes of continuous video without losing its memory of the scene.
• Control: precise frame-by-frame response to keyboard and mouse, which matters for games and simulators.
• Quality: on the GameWorld Score benchmark the model clearly beats Oasis in Minecraft (better image, action and stability over long stretches), and in complex scenes it holds style and dynamics at or above YUME while running faster and resisting frozen frames.

To hit 25 FPS without giving up quality, the team combined three techniques: caching the VAE decoder, slimming down the action modules, and cutting diffusion steps to three. The metrics stay stable throughout.

One unexpected lesson

Too large a local KV cache size spoils long sequences: artifacts pile up and the model retrains on its own flaws. A moderate window preserves context and leaves room to correct course on the fly.

Where the limits are today

• Out-of-domain scenes are still hard: a long camera climb or a very long forward move leads to degradation.
• A resolution of 352×640 is not 4K, and some tasks want more.
• Long-term memory of objects and events over horizons of tens of minutes calls for lightweight memory modules or external retrieval that does not cost speed.

Why it matters

Open code and weights make it possible to test ideas quickly: plugging in planners for multi-agent systems, wiring a world model to an LLM through a prompt, training new agents on rich interactive trajectories. Matrix-Game 2.0 turns video diffusion into a working tool for real systems.

In short, the work shows that a world generated by AI can have a tactile quality: you press a key and the world moves. Fast, stable and accurate enough to build the next generations of interactive AI on. Code and weights are to be released, which means the main experiment is only beginning.

💾 Code