DENG Lab ↗ · MLSys Team ↗

Wave Forcing

Wave Your
Streaming Videoto Speed-of-Light.

Wave Forcing keeps Rolling Forcing's mixed-noise schedule, but makes cross-chunk attention block-causal. Wave Runtime (WaveRT) — our custom execution engine on SGLang Diffusion's model-layer backend — executes this DAG through Wave Parallelism: full-model GPU replicas hide one-way KV communication and stream completed chunks through a balanced VAE pipeline.

Run the pipeline View code ↗

117.7 E2E FPS≈13.56 s · full 1596-frame pipeline

125.7 FPS(p50)best E2E config · pooled steady ticks

7.9×vs single-GPU E2E

Explore

One video—or eight arriving in waves.

4-step denoising · in training

Stay tunedWave Forcing 4-step live comparison will be posted here

5-step denoising

Timeline

Rolling Forcing

0.0 s

1 video in 134.0 s

Wave Forcing

0.0 s

8 videos generated in 152.4 s

5-step results · BF16 · WaveRT causal paged KV exchange · torch_sdpa attention · 6 DiT + 2 VAE stages on 8×H200

01 · The design gap

Streaming exists. The missing piece is a pipeline with live context.

Prior work opens two strong routes: Rolling Forcing keeps a live mixed-noise window but creates layer-wise synchronization, while Live Avatar builds an efficient stage pipeline by giving up fresh cross-stage history. Wave Forcing keeps the useful half of both designs.

Streaming starting point

Few-step causal generation is still block-first.

CausVid · Causal Forcing · Self Forcing / Self-Forcing++ make streaming practical, but every denoising stage of chunk c finishes before chunk c+1 starts. That leaves a sequential chain inside each request.

A · Live frontierRolling Forcing

Fresh context, cyclic execution.

RF advances mixed-noise chunks together and emits one clean chunk per joint call after pipeline fill.

Keeps: Live bidirectional active-window context
Costs: Reverse edges close a cycle. Even exact Ulysses or Ring execution still coordinates global attention at every Transformer layer

Mixed noise ✓Fresh context ✓Layer barriers

B · Stage-local frontierLive Avatar

Clean pipeline, fixed-state history.

Fixed denoising stages pass latents forward in a simple anti-diagonal hardware pipeline.

Gains: Latent-only stage communication without layer-wise global attention
Trades: Each stage keeps history at a fixed noise state; its KV cache is not refreshed by cleaner representations from later stages. This is a modeling trade-off to measure, not just a runtime shortcut

Anti-diagonal ✓Latent-only ✓Stage-local history

C · Directed live frontierWave Forcing

Fresh context, acyclic execution.

Wave Forcing keeps RF's mixed-noise schedule: full attention within each chunk, block-causal attention across chunks, and fresh cleaner-to-noisier context.

Changes: Remove only the reverse noisy-to-clean edges; rollout-aligned distillation trains on the same directed frontier used at inference
Unlocks: Full-model GPU replicas publish layer-wise KV in one direction, with no global collective barrier

Mixed noise ✓Fresh context ✓No global barrier ✓

Key observation

The mixed-noise window is not the cycle.
The reverse noisy → clean edge is.

Full RF

clean⇄noisy

cycle · synchronize

→

Wave Forcing

clean→noisy

DAG · pipeline

02 · Interactive pipeline

Two pipelines work at two different scales.

Chunk–stage tasks form a wave across full-model GPU replicas. Inside every Transformer layer, one-way KV communication is prefetched and hidden under the longer attention computation of the critical consumer. Two ideas make this wave possible — Wave Parallelism, the core mechanism, and Wave Runtime (WaveRT), the engine that executes it.

Core mechanism

Wave Parallelism

Wave Parallelism is Context Parallelism specialized into an async form by algorithm co-design. Block-causal attention lets each denoising step start on its own GPU rank the moment it is ready — ranks publish KV one-way instead of meeting at a layer-wise all-gather — so the wave advances one GPU–stage pair per tick.

Today: decoupled implementation · Next: fused form

The execution engine

Wave Runtime (WaveRT)

WaveRT is our custom execution engine, built on SGLang Diffusion's model-layer backend. It organizes warmup and steady-state runs, and integrates the Wave Parallel optimizations that follow on this page — FP8 KV, copy-engine one-sided writes, paged buffers, and the measured-time VAE split.

Inside one diffusion tick

The slow rank is not just a bottleneck.
It is our overlap window.

Compute KV transfer Exposed wait

Critical consumer · r08-chunk context

Later, noisier chunk sees every cleaner predecessor → more compute per layer

shorter ranks publish first←

Cleaner producers · r1–store≈5-chunk context

Earlier chunks have shorter legal context → fresh KV is ready before r0 needs it

0 μs

A · synchronized

Compute, stop, communicate.

Barrier every layer

Layer LL+1L+2L+3L+4L+5

r0 · 8 chunks

r1 · 7 chunks

r2 · 6 chunks

r3 · 5 chunks

store · clean KV

NVLink collective

Communication stays exposed. Faster ranks wait at all_gather; the next layer cannot begin until transfer and synchronization finish.

B · overlapped

Prefetch, compute, gate.

Depth-1 async P2P

Layer LL+1L+2L+3L+4L+5

r0 · 8 chunks

r1 · 7 chunks

r2 · 6 chunks

r3 · 5 chunks

store · clean KV

NVLink P2P

Each rank publishes early; six consecutive KV transfers stay under r0's compute windows

Communication moves off the critical path. At the point of use, r0 only gates on any residual transfer instead of paying the full communication cost.

Measured 14B steady stager0 · 646 ms

≈70 ms of downstream slack

r1 / r2 / r3 / store575–581 ms

Across consecutive video chunks

Now zoom out: the whole wavefront.

Read the schedule by GPU ID vertically and Stage ID horizontally. Follow one color diagonally to track the same chunk through diffusion, context publication, and VAE decode — that diagonal is Wave Parallelism advancing one GPU–stage pair per tick.

GPU × Stage wavefrontsame color · same chunk

↘ Step +1 moves each chunk to the next GPU–stage pair△ Light upper triangle = cleaner-stage KV has arrived at each noisier DiT consumer

Layer-wise KV transport inside the diffusion wavecopy engine · one-sided direct write

producer KV

consumer slots

NowFilling the pipeline

The first noisy chunk enters GPU 0 / D₁. Later chunks follow one tick apart and move diagonally across the matrix.

1 active ranks 0 chunks decoded ≈95 ms best 1.3B steady tick

Layer-wise context exchange

At every Transformer layer, cleaner ranks publish fresh KV in one direction. The longer-context consumer gates only when those states are used.

↝

Deterministic re-noise

A deterministic seed per chunk and stage preserves reproducibility while chunks travel through independent ranks.

≋

Two disjoint NCCL worlds

Diffusion and VAE use separate communicators, connected by a CPU queue, avoiding cross-pipeline deadlocks.

Δt

Measured-time VAE split

Stages are partitioned by observed runtime instead of FLOPs, balancing memory-bound high-resolution decoder blocks.

03 · End-to-end KV transport

Optimize bytes, movement, and destination.

Communication is not one cost. Wave Parallelism attacks three different taxes: how many bytes move, who coordinates the move, and whether the consumer must rebuild the buffer afterward.

Producer GPULayer L · K/V ready

NCCL collectiveBF16 · 2 bytes / value

Qquant / dequant

matching collective wall

generation flag

Consumer GPU · r0Receive, then assemble

torch.catassembly wall

Baseline · synchronized BF16

Every rank joins; every byte is full width.

Blocking all_gather moves BF16 KV through a matching collective. The consumer then concatenates received segments into an attention buffer at every layer.

Critical-path communication≈79 ms14B diagnostic

01 · Precision

FP8 halves DMA bytes.

→

Raw communication falls from roughly 79 to 40 ms. But conversion is not free: with SageAttention, requantization costs more than the saved transfer time.

Effective standalone · experimental with Sage

02 · Transport

Copy Engine removes the collective wall.

GPU ACE / NVLinkGPU B

The producer writes remote IPC memory using cuMemcpyPeerAsync. A device-side generation flag replaces matching NCCL send/recv synchronization.

0 SM copy path · 343 GB/s microbenchmark

03 · Layout

Paged writes remove the assembly wall.

Each producer writes directly into its fixed slot in the consumer's contiguous attention buffer. Attention reads the prefix in place—no per-layer reconstruction.

torch.cat: 18.49 → 4.08 ms · recommended

Why this is not one linear stack

FP8 KV is a byte-reduction experiment; it is distinct from the sweep's W8A8 DiT tier. In the final pooled-p50 report, the best 14B route is Sage + W8A8 + causal paged at 421.1 ms p50 / 28.1 E2E FPS. The 14B run uses shape-accurate random-init dummy weights.

04 · VAE without bubbles

Overlap work across GPUs,
not on the same SMs.

Once 1.3B diffusion is optimized to the limit, decode becomes the final barrier. Launching the next chunk before the current decode finishes sounds parallel—but on one GPU, both chunks compete for the same SMs and each becomes slower.

A · tempting, but slower

Concurrent chunks on one GPU

SM contention

GPU 5 · shared SM pool160% requested

0time→

Chunk C1decode C1

Chunk C2decode C2

both kernels fight for SMs

Overlapping two compute-bound decodes does not create free capacity. It stretches both jobs and makes the pipeline tick longer.

B · Wave Parallelism

One chunk per GPU stage

No same-GPU overlap

Tick T

T+1

T+2

T+3

GPU 5 · early

C1C2C3C4

GPU 6 · middle

C1C2C3

GPU 7 · output

C1C2

Each GPU owns a contiguous decoder stage and its temporal cache. Consecutive chunks overlap spatially across GPUs, never by contending on one GPU.

Why equal FLOPs are not equal time

Partition the measured latency,
not the spreadsheet.

High-resolution upsample blocks are memory-bound at 480×832. Blocks with similar FLOPs can take roughly 6× different wall time, so a FLOPs split leaves one stage overloaded and the others waiting.

AlgorithmContiguous DP min–max partition

CorrectnessBit-exact vs. the original split

FLOPs-balanced split≈1.8× time imbalance

GPU 5Stage A

GPU 6Stage B · 146 ms barrier

GPU 7Stage C

pipeline waits here

measure every decoder unit↓

Time-balanced split99 ms max stage

GPU 5Stage A · 95 ms

GPU 6Stage B · 99 ms

GPU 7Stage C · 96 ms

After extreme 1.3B optimization

The balanced VAE becomes the final cadence—not an idle bubble.

Best E2E config · DiT95.4 ms p50 / tick

Balanced VAE≈99 ms / chunk≈3.5 ms final barrier · all stages remain occupied

chunk arrives

next chunk

05 · New benchmark sweep

A full matrix,
not a single lucky run.

178 recorded runs across two model scales, two pipeline topologies, three compute tiers, and six KV exchange modes. The 54 non-relay configurations are complete. Steady ticks use per_tick_ms[12:-4], pooled across three repeats before taking p50.

Best full pipeline117.7 E2E FPSSage · causal paged · 7.9× single-GPU

399-latent E2E time≈13.56 s1596 output frames · includes streaming VAE

Fastest valid steady tick126.5 FPS(p50)94.9 ms · 4+3 · Sage + W8A8 · causal paged

Best 14B pipeline28.1 E2E FPS421.1 ms p50 · Sage + W8A8 · causal paged

Model / topology1.3B · 4+3

KV semanticsCausal

AggregationPooled p50 · 3 reps

Single-GPU baseline106.462 s

Compute tier	Sync	Overlap	One-sided	Staggered	Paged
Loading benchmark summary…

Steady FPS(p50) uses 12 output frames per tick. DiT/E2E FPS use 1596 frames over their respective wall clocks.Relay excluded: 14 recorded timeouts · rerun pending

Why 5+2 plateaus

Five fast DiT ranks feed only two coarse VAE stages.

The diffusion side can exceed 120 FPS, but a two-way VAE split leaves too much decoder work in each stage. Its longest stage runs past the next chunk's arrival, so decode cannot be fully hidden behind generation.

120+DiT FPS available

→

85.1best reliable E2E FPS

Chunk cadenceExposed decoder tail

DiT arrivals

CₙCₙ₊₁Cₙ₊₂Cₙ₊₃

2-stage VAE

VAE AVAE Bnot hidden

When the next chunk enters before the previous decode finishes, adjacent chunks contend on the same two VAE GPUs. SM competition stretches both jobs instead of increasing throughput.

4+3 remedyThree smaller, time-balanced VAE stages approach the DiT cadence and leave only a small residual barrier.

Source & scope

Final 178-record report: 164 successful runs and 14 recorded relay timeouts. FPS(p50) is steady tick throughput; DiT FPS includes diffusion fill/drain; E2E FPS also includes VAE. The 14B sweep uses random-init dummy weights. In 5+2, the two-stage VAE sets a structural ceiling near 85 E2E FPS; orange Sage/SageFP8 async cells show an additional resource-contention slowdown despite healthy DiT throughput.

06 · VBench quality

Acceleration doesn't trade
away quality.

On VBench-Long with ~100-second videos, Wave Forcing delivers quality on par with Rolling Forcing.

Model	Temporal Flickering	Subject Consistency	Background Consistency	Motion Smoothness	Aesthetic Quality	Imaging Quality	Quality Drift ↓
Rolling Forcing	97.81	97.34	96.42	98.85	53.83	69.58	0.618
Rolling Forcing on WaveRT	97.24	96.82	96.14	98.39	55.18	66.82	2.825
Wave Forcing	97.86	98.00	96.79	98.57	54.93	69.66	0.030

Model	Temporal Flickering	Subject Consistency	Background Consistency	Motion Smoothness	Aesthetic Quality	Imaging Quality	Quality Drift ↓
Rolling Forcing^*	97.78	97.20	96.23	98.84	53.62	69.62	1.111
Rolling Forcing on WaveRT^*	97.36	96.69	96.11	98.50	54.52	66.32	2.680
Wave Forcing^*	97.55	97.66	96.69	98.28	54.31	68.89	2.761
Causal Forcing (Long)	94.15	95.37	95.07	97.73	49.92	69.24	2.032
Wave Forcing 4-step	Stay tunedWave Forcing 4-step · results will be posted here

* 5-step-trained checkpoints, evaluated here with forced 4-step inference.

Model	Runtime
Loading quality summary…

Protocol

The 5-step table runs all three arms on the same 50 MovieGen prompts, one 1593-frame video per prompt at 16 FPS / 99.56 s. Rolling Forcing on WaveRT is the same Rolling Forcing checkpoint decoded with Wave Forcing's block-causal inference. The 4-step table forces the same three 5-step checkpoints to a 4-step schedule and adds Causal Forcing (Long), which runs its native 4-step long-video inference; Wave Forcing 4-step results are pending and will be posted as the sweep completes. The algorithm × runtime table uses 10 aligned prompts for VBench-Long quality; FPS is DiT-only throughput from a one-shot 399-latent-frame run. Quality Drift is computed following the Rolling Forcing paper, lower-is-better; every other quality metric is higher-is-better.

Wave Forcing × Wave Parallel