Streaming Your World-Action Model for Real-Time Robot Manipulation.

Insights from Our Exploration

Across our experiments, we identified three practical observations that shaped Streaming-WAM: video and action distillation behave differently, action conditioning benefits from strict temporal alignment, and partial action context needs an explicit representation.

Distillation behaves differently across modalities. On RoboTwin 2.0, the open-source FastWAM-Joint model achieves 87.0 overall success. Distilling both video and action generation to one step—the 1V1A setting—reduces the score to approximately 77.8, while keeping two steps for action generation—the 1V2A setting—recovers it to 86.0. This suggests that the loss comes mainly from aggressive action distillation. Our interpretation is that video latents contain spatial and temporal redundancy, whereas actions are the final control outputs: small biases in a 32-step, 14-dimensional dual-arm trajectory can directly affect contact, coordination, and grasp stability. One-step action generation may therefore require additional training structure or conditioning rather than serving as a drop-in replacement for an iterative sampler.

Action conditioning should respect the action-to-frame correspondence. Our design pairs 32 action steps with nine video frames sampled every four control steps. With a temporal-compression factor of four, the video stream contains the observed group Z₀ and two future groups, Z₁, Z₂, each corresponding to a 16-action interval. The condition stream likewise covers 16 positions: eight known condition slots C_K encoding the committed actions, followed by eight unknown condition slots C_U. We therefore add action-conditioned attention only to Z₁, whose temporal range contains those committed actions. Applying the same context to Z₂ would mix actions and visual states from different time ranges.

Condition slots make partial action context usable. The fixed 16-slot condition stream combines eight known condition slots with eight learned unknown placeholders. A learned state embedding distinguishes an unavailable action from a zero-valued or no-op command, while the fixed layout keeps attention routing and compiled inference shape-stable. These slots provide structured context to the visual branch without replacing the normal action stream. Together, these findings connect efficient distillation, temporally aligned conditioning, and explicit uncertainty into a coherent design for Streaming-WAM.

World Action Models and the Real-Time Gap

World Action Models (WAMs) jointly predict future visual representations and action chunks, allowing robot policies to connect action generation with object motion, physical interaction, and task progress [1]. The visual branch provides a temporally structured account of how the scene may evolve, while the action branch turns that prediction into a multi-step control plan. This coupling has produced strong manipulation performance, but iterative generation over both modalities is often much slower than the robot control cycle.

Under synchronous deployment, inference and execution are serialized: the robot finishes its current chunk, waits for the next prediction, and only then resumes motion. Model latency therefore becomes idle time or repeated stale commands. Because the observation that launched inference is also aging during this pause, recent changes in object state, contact, or robot motion cannot influence the policy until the next update.

Longer action horizons reduce the frequency of model calls but commit the robot to a staler open-loop plan. Faster inference shortens the wait, yet any remaining latency still lies on the critical path. The deployment gap is therefore not only a question of model speed, but also of how prediction and execution are scheduled. Real-time WAM deployment requires prediction to overlap ongoing motion without losing consistency between the predicted world evolution and the actions actually executed.

Asynchronous Execution Beyond Action Continuity

Asynchronous execution starts the next prediction before the current action chunk is exhausted. This creates a temporal overlap in which committed actions are reused across adjacent chunks, and the incoming prediction must remain consistent with the motion already underway. Accurate temporal alignment is therefore essential: even individually valid chunks can be joined at the wrong steps if the handoff time is misidentified [2].

Real-time chunking (RTC) constrains the incoming chunk at inference time through inpainting-style guidance of the denoising trajectory [3]. Because this guidance is recomputed at every denoising step, it adds inference cost; as a soft constraint, it can also leave residual disagreement in the delay region [2]. Prefix-conditioned methods move the constraint into learned generation: delay-region actions are provided as clean context, and Training-Time RTC learns to produce the remaining action continuation from that context [2,4]. When runtime latency falls outside the delay distribution represented during training, the clean context may no longer match the actual handoff [4].

Both approaches align action trajectories rather than predicted world evolution. RTC steers the overlap between adjacent chunks, while prefix conditioning fixes the known action segment, but neither explicitly tells the visual branch how the committed actions transform the scene. This distinction matters in a WAM because the next action is generated together with a predicted account of the world. LingBot-VA 2.0 addresses the visual state through FDM grounding. It refreshes the visual cache with the latest observation, appends the executing action, and predicts that action's visual outcome as context for decoding the next action [5]. This extra grounding pass adds future-state prediction before each action update, a practical cost at high control rates.

Streaming-WAM instead couples action continuity with world prediction inside a joint WAM. It reuses committed actions across adjacent chunks and routes them both directly into each Stream Update and into known condition slots. Together with the unknown action slots, the resulting condition slots guide future visual generation in the same joint update that produces the next visual-action chunk, while robot execution continues.