The Library of Congress holds a copy pressed from the original master and has digitized it for its collection "American Leaders Speak: Recordings from World War I." It is an acoustic recording, made without a microphone, and the noise of the disc sits on top of Pershing's voice. For Veterans Day, 108 years after the armistice of November 11, 1918, we ran it through Diffio AI, our speech restoration tool. This page puts the original and the restoration side by side.

Listen: the 1918 recording as it survives

Listen: the same recording, restored with Diffio

Both files are the whole disc, matched for loudness, so what changes is the noise and the voice, not the volume. What Pershing says:

"Three thousand miles from home, an American army is fighting for you. Everything you hold worthwhile is at stake. Only the hardest blows can win against the enemy we are fighting. Invoking the spirit of our forefathers, the army asks your unshrinking support, to the end that the high ideals for which America stands may endure upon the earth."

The Library of Congress catalog quotes the phrase as "unshrinking support." On both the original and the restoration, a speech recognition model hears "unflinching," which shows how hard the word is to make out on the disc. We have kept the catalog's wording.

Watch: before and after

What changed

Diffio restores recorded speech. It separates the voice from everything else in the recording and rebuilds it so it sounds closer to a clean microphone. On this disc the surface noise drops away and Pershing's voice comes forward.

We checked the result the same way we check our own work: we ran a speech recognition model (Whisper) on the original and on the restoration. On the original it heard the last line as "they endure upon the earth"; on the restoration it heard "may endure upon the earth," which is what Pershing said. Everything else matched.

One honest note from the process. On our first pass the restoration lost the opening words, "thousand miles from home," because the speech starts in the very first second of the file. Adding two seconds of silence before the recording fixed it, and the version on this page is that second pass. If you restore a recording that starts mid-word, give it a moment of lead-in.

The models are generative, which is why the result can be so much clearer than a noise filter manages on audio this old. It is also why we publish the original next to the restoration. Treat the restored file as a listening copy: the original stays exactly as the Library of Congress digitized it, and anyone quoting Pershing should check the words against it.

More voices from the same years

Earlier this fall we restored Warren G. Harding's 1921 Hoboken address for the American war dead, which he recorded for Victor in 1922, and a set of early presidential voices from 1908 to 1922. The same kind of restoration works on recordings much closer to home: a grandfather's cassette about his service, a reel-to-reel of a Veterans Day service, an oral history taped in a noisy hall. You can try Diffio on three uploads without a card.

For journalists, educators and museums

The Library of Congress records that this disc was sold commercially in 1918, and U.S. sound recordings published before 1923 entered the public domain in 2022. The restored audio and the video on this page are free to use for press, classroom and museum work. Write to me at nathan.harmon@trydiffio.com if you'd like WAV files. Please credit "Audio: Gen. John J. Pershing, 'From the Battlefields of France,' Nation's Forum (1918), Library of Congress; restored with Diffio AI (diffio.ai)". The original is at the Library of Congress.

Nathan Harmon is the founder of Diffio AI, based in Colorado Springs.

The discs survive, digitized by the Library of Congress for its National Jukebox. They are acoustic recordings, made without a microphone, and the voices sit under a century of hiss and crackle. For Election Day we ran three of them through Diffio AI, our speech restoration tool. This page puts each original next to its restoration so anyone can hear both. Each pair is the same passage, matched for loudness, so what changes is the noise and the voice, not the volume.

Watch: Roosevelt, before and after

Theodore Roosevelt: "They dread you, the people"

From "Why the Trusts and Bosses Oppose the Progressive Party" (Victor 35250-A). Roosevelt is answering Standard Oil's John D. Archbold and Senator Boies Penrose, who had testified against him that year, and turns their attacks into his case for the Progressive platform:

"The fundamental motive that induces them to act as they have acted in this manner is not merely that they dislike me, but far more because they dread you. They dread you, the people, you and those like you who make up the people of the United States. They know that their time has come if once the people obtain real power. We stand for the rights of the people. We stand for the rights of the wage worker. We stand for his right to a living wage."

Original (Victor 35250-A, recorded September 22, 1912)

Restored with Diffio

Woodrow Wilson on the third party

From "Woodrow Wilson on the Third Party" (Victor 35251-A). Wilson argues that Roosevelt's plan to regulate the trusts would make partners of the men it meant to control:

"Mr. Roosevelt puts forth an admirable platform of what he would like to do for the people. But how is he going to do it? He proposes in his platform not to abolish monopoly but to take it under the legal protection of the government and to regulate it. In other words, to take the very men into partnership who have been making it impossible to carry out these great programs by which all of us wish to help the people."

Original (Victor 35251-A, recorded September 24, 1912)

Restored with Diffio

William Howard Taft: "Who are the people?"

From "Who Are the People" (Victor 35256-A). Taft, the incumbent, answers both challengers' claim to speak for the people:

"Who are the people? They are not alone the unfortunate and the weak. They are the weak and the strong, the poor and the rich, and the many who are neither. The wage earner and the capitalist, the farmer and the professional man, the merchant and the manufacturer, the storekeeper and the clerk, the railroad manager and the employee. They all make up the people."

Original (Victor 35256-A, recorded October 1, 1912)

Restored with Diffio

Wilson won the election on November 5, 1912. Roosevelt came second, the only third-party candidate ever to do so, and Taft third. More than a century later, their argument about who speaks for "the people" still sounds familiar.

What changed

Diffio restores recorded speech. It separates the voice from everything else in the recording and rebuilds it so it sounds closer to a clean microphone. On these discs the surface noise drops away and the voices come forward.

The models are generative, which is why the result can be so much clearer than a noise filter manages on audio this old. It is also why we publish each original next to its restoration. We treat the restored files as listening copies: the originals stay exactly as the Library of Congress digitized them, and anyone quoting these speeches should check the words against the originals. The quotations on this page are from our own transcription; the full discs are in the National Jukebox.

The same kind of restoration works on recordings much closer to home: a family cassette, an interview recorded on a phone, a radio program saved on reel-to-reel. If you have one, you can try Diffio on three uploads without a card.

For journalists, educators and museums

The original recordings are in the public domain: they were first published in 1912, and U.S. sound recordings published before 1923 entered the public domain in 2022. The restored audio and the video on this page are free to use for press, classroom and museum work. Write to me at nathan.harmon@trydiffio.com if you'd like WAV files or the full restored discs. Please credit "Audio: Victor Talking Machine Co. (1912), Library of Congress National Jukebox; restored with Diffio AI (diffio.ai)".

Nathan Harmon is the founder of Diffio AI, based in Colorado Springs.

That disc survives, digitized by the Library of Congress for its National Jukebox. It is an acoustic recording, made without a microphone, and a century of surface noise sits on top of it: a roar of hiss and crackle with Harding's voice somewhere underneath. For Veterans Day we ran it through Diffio AI, our speech restoration tool. This page puts the original and the restoration side by side so anyone can hear both.

Listen: the 1922 recording as it survives

Listen: the same passage, restored with Diffio

Both clips are the same 35 seconds, the passage the speech is remembered for, and both are matched for loudness, so what changes is the noise and the voice, not the volume:

"One's words fail, his understanding is halted, his emotions are stirred beyond control when contemplating these thousands of beloved dead. I find a hundred thousand sorrows touching my heart, and there is ringing in my ears, like an admonition eternal, an insistent call, 'It must not be again. It must not be again.' God grant that it will not be, and let a practical people join in cooperation with God to the end that it shall not be."

Watch: 20 seconds, before and after

The full address

The whole 4 minute 25 second disc, before and after:

Original (Victor 35718-A, 1922)

Restored with Diffio

The 1922 recording is a little shorter than the speech Harding gave in Hoboken. A short passage near the end ("They gave all which men and women can give...") is not on the disc; the Library of Congress prints the full text with the omitted lines marked.

Correction (November 3, 2026): checking the full restoration word by word against the original disc, we found one passage the model got wrong. About 3 minutes 27 seconds in, Harding says "of a nation being so righteous as never to"; the words are clear on the original, but in the restored file they come out garbled, closer to "even". This is the kind of error a generative model can make, and it is why the original stays the reference: quote Harding from the original, not the restoration. The 35-second excerpt above does not include this passage, and we checked it against the published text.

What changed

Diffio restores recorded speech. It separates the voice from everything else in the recording and rebuilds it so it sounds closer to a clean microphone. On this disc the surface noise drops away and Harding's voice, which on the original sits underneath it, comes forward.

The models are generative, which is why the result can be so much clearer than a noise filter manages on audio this old. It is also why we publish the original next to the restoration. We treat the restored file as a listening copy: the original stays exactly as the Library of Congress digitized it, and anyone quoting Harding should check the words against it and against the published text.

Six months later: the Unknown Soldier

On November 11, 1921, the third anniversary of the armistice, Harding presided at the burial of the Unknown Soldier at Arlington National Cemetery. This Veterans Day marks 105 years of the Tomb. The Hoboken address is the closest thing we have to hearing him speak about the war dead in his own voice, and the line at its center, "It must not be again," is one that every generation since has had reason to repeat.

The same kind of restoration works on recordings much closer to home: a grandfather's cassette about his service, a reel-to-reel of a Veterans Day service, an oral history taped in a noisy hall. If you have one, you can try Diffio on three uploads without a card.

For journalists, educators and museums

The original recording is in the public domain: it was first published in 1922, and U.S. sound recordings published before 1923 entered the public domain in 2022. The restored audio and the 20-second video on this page are free to use for press, classroom and museum work. Write to me at nathan.harmon@trydiffio.com if you'd like WAV files. Please credit "Audio: Warren G. Harding, Victor 35718-A (1922), Library of Congress National Jukebox; restored with Diffio AI (diffio.ai)".

Nathan Harmon is the founder of Diffio AI, based in Colorado Springs.

We ran the broadcast through Diffio AI, our speech restoration tool, and published the result on the diffio.ai homepage. The Daily Mail covered it on October 24, 2026. This page puts the original and the restored version side by side, so anyone can hear both and judge for themselves.

Listen: the original 1935 broadcast

Listen: the same passage, restored with Diffio

Both clips are the same 32 seconds. The restored one is louder because the noise that sat under the voice is gone; turn your volume down a little before you switch.

What changed

Diffio restores recorded speech. It separates the voice from everything else in the recording and rebuilds the voice so it sounds closer to a clean microphone: background noise and hiss are removed, room echo is reduced, and supported clipping and distortion are repaired. On this broadcast that means the hiss and crackle drop away and Earhart's voice comes forward.

The models are generative, which is why the result can sound so much clearer than older noise filters manage on audio this old. It is also why we publish the original next to the restoration. For historical recordings we treat the restored file as a listening copy: the original stays exactly as it was, and anyone quoting Earhart should check the words against it.

Why old voices matter

"History feels different when you can hear someone speak," I told the Daily Mail. "With Amelia Earhart, it's a chance to hear the person behind the photographs and the extraordinary story." Most people know her from black-and-white pictures and from the search for her Electra, which went back to Nikumaroro this October. Far fewer have heard her talk, partly because the recordings that survive are so rough. Making them easier to listen to is a small way to bring her a little closer.

The same applies to recordings in family closets, church archives and local history collections: cassette interviews with grandparents, reel-to-reel sermons, oral histories taped in noisy rooms. If you have one of those, you can try Diffio on three uploads without a card.

For journalists and educators

The original and restored audio of this broadcast are cleared for press, classroom and museum use, along with a 10-second before-and-after video. Write to me at nathan.harmon@trydiffio.com and I'll send the files. Please credit "Audio: Amelia Earhart, 1935 radio broadcast, restored with Diffio AI (diffio.ai)".

Nathan Harmon is the founder of Diffio AI, based in Colorado Springs.

A guided step is three independent forward passes

The model is a diffusion transformer with a Qwen2 trunk. It restores audio in overlapping 30 s windows, each conditioned on semantic features from the Granite speech encoder, on observation latents (the degraded input encoded by the Ming AudioVAE codec) and on the transcript text. The samplers are stochastic rectified-flow samplers (flow matching covers the velocity field they integrate) with classifier-free guidance. A guided step evaluates three branches: u (no conditions), so (semantic and observation, no text) and sot (all conditions). Late steps are unguided and evaluate sot alone.

The branches do not interact until the guidance arithmetic combines their velocities, so a step can be divided by branch across engines and lasts as long as the slowest engine's share.

guided step    (~235 ms)   GPU + CPU       sot    each MLP split between GPU (bf16) and CPU (fp32)
                           Neural Engine   so, u  one after the other
unguided step  (~215 ms)   GPU + CPU       sot
                           Neural Engine   codec work for neighbouring windows
per window                 GPU (PyTorch)   Granite, the condition encoders and the text encoder

MLX's GEMMs reach 89% of the GPU's peak, so kernels were not the bottleneck

Moving the transformer from PyTorch's MPS backend to MLX was the first large gain, because MLX fuses the elementwise work around each GEMM into fewer kernels; it also cut the relative L2 error of the output velocity against an fp32 forward pass from 6e-2 to 5e-3. Compiling each layer's attention block with mx.compile helped a little more. The GEMMs themselves had nothing left to give: Metal 4's tensor-op multiply (matmul2d) ran within 1% of MLX at our shapes, and PyTorch's torch.compile made the MPS forward pass 70% slower.

Splitting each MLP between the GPU and the CPU's SME units makes all CPU work compete with the model

MLX evaluates a GPU stream and a CPU stream concurrently. We partition each MLP along its intermediate dimension: the GPU computes 70% of the intermediate columns in bf16, the CPU's SME units compute the other 30% in fp32 (bf16 elementwise math measured four times slower than fp32 on the CPU), and the two partial down-projections are summed. In a microbenchmark this cut an unguided forward pass from 236 ms to 208 ms. The best share, found by sweep, gives the GPU more than the standalone rates would, because neither engine keeps its standalone rate while both run: they share one power budget, and the GPU's clock drops from 1578 MHz to 1.43–1.54 GHz while the CPU computes.

Because the CPU is part of every layer, any other CPU work lengthens the steps it overlaps. The codec decoder's final ISTFT stage cost 110 ms of numpy per window, and moving it to the GPU (11 ms) sped up the steps it had overlapped. A helper process computing the next window's Granite features on the CPU would have cost more than it saved, and on the efficiency cores at background priority the same work was far too slow.

Core ML runs the transformer on the Neural Engine only as fixed-shape fp16 chunks with overflow guards

Core ML, the only public route to the Neural Engine, compiles fixed-shape fp16 graphs for it, and when its compiler rejects a graph it runs it on the CPU without an error. The full 28-layer model and our first codec conversion (rank-5 tensors in its attention) both fell back; Core ML's compute plan (MLComputePlan, which reports each operation's device) and power readings (the Neural Engine at 0 W) catch it. The text-free branches run as chunks of consecutive layers. The small pieces around the chunks (input projection, timestep embedding, per-layer adaptive-norm vectors, output projection) run in numpy in a separate process, because Core ML's predict holds Python's GIL.

fp16 tops out at 65504, and this model exceeds it in attention-sink scores and in the squares inside RMSNorm. The conversion normalizes each RMSNorm row by its maximum absolute value before squaring; for attention it scales the queries by an extra 1/1024, subtracts each row's maximum, and scales back before the softmax. Both rewrites are exact in fp32. With them the Neural Engine's branches are as accurate as the GPU's: on inputs captured from real guided steps, relative L2 error against fp32 is 4.7–6.4e-3 on the Neural Engine and 5.2–7.5e-3 for GPU bf16.

The GPU's error comes mostly from its bf16 GEMM inputs: the same forward pass with fp16 GEMMs and an fp32 residual stream is 1.9–2.6e-3 from fp32. The Neural Engine gains fp16's three extra mantissa bits in its GEMMs and gives most of them back by keeping the residual stream in fp16. Quantizing its weights to int8 would have doubled the error for a 6% speedup.

Converting Granite exposed two fp16 precision losses: activations fused after a GEMM, and the overflow guard

A SiLU that directly follows a GEMM on the Neural Engine loses precision that the same SiLU alone does not: in a probe, relative error was 2.8e-4 standalone and 2.9e-3 after a linear layer. On Granite's own activations, writing the sigmoid as 1/(1+exp(-x)) brought a feed-forward block from 3.9e-3 to 2.4e-3, where tanh after a linear layer had reached 1.8e-2. The overflow guard costs precision where it is not needed: without it, Granite's attention error fell from 3.9e-2 to 1.6e-3. The likely cause is that the extra 1/1024 puts small query values near fp16's smallest normal number, where fewer mantissa bits remain. The transformer's text-free branches need the guard and did not change when their SiLU and softmax were rewritten.

A branch without text attends to one null token, so its cross-attention is a constant

A branch without transcript text still has a text cross-attention input: a learned null embedding, with every other position masked. A softmax over a single unmasked key is 1 for every query, so the cross-attention output is the null token's value projection passed through the output projection, a constant vector per layer. The text-free branches skip the query projection, the attention and the per-position output projection, and text keys and values are computed only for branches that have text. In a single forward pass the conditioned branch stayed bit-identical and the unconditional branch moved by at most one bf16 unit in the last place. The shortcut also makes so and u the same graph: their prompts have the same shape and differ only in token ids, whose embeddings are computed outside the Core ML chunks.

Partitioning the MLP along its intermediate dimension let the Neural Engine run two branches

Before partitioning, one text-free branch took 244 ms on the Neural Engine, and batching two branches into one call cost 2.17x the time of one, so the GPU kept two branches per guided step. A microbenchmark of one MLP at our shape showed where the time went: partitioned into independent blocks of intermediate columns, with the blocks' down-projections summed, the same MLP ran more than twice as fast at the same error against fp32.

Throughput rises until a block's weights plus its activations fit in about 32 MB, then flattens. That is the size of the Neural Engine's on-chip SRAM, where reverse-engineering measurements found throughput falling once a matrix multiply's operands no longer fit, although at our shapes the penalty was larger than the 30% they report. The ReLU MLP at 2 blocks, 27.5 MB of weights but about 33 MB with activations, sits between the extremes. We did not observe the compiler's schedule, so this reading fits the data without proving the cause. Apple's guidance for transformers on the Neural Engine recommends chunking large intermediate tensors, which the partition also does, for cache residency and to spread work across the engine's cores.

Partitioning alone was worth 1.7x; chunks longer than 7 layers and more than 8 blocks did not help. At 117 ms per branch, the Neural Engine runs so and then u while the GPU and CPU run sot, and guided steps fell from about 355 ms to about 235 ms. Clips gained less than that step saving predicts, flash least: the codec work that had filled the Neural Engine's idle time on guided steps lost it, and flash has the fewest unguided steps to absorb it.

Duplicate weight copies, not the model, made unified memory swap

Unified memory removes copies between engines, not copies within a program. With the Neural Engine running two branches, mid-run free memory fell to 13% and the system swapped 5–6 GB. The main process held 14 GB, mostly GPU memory (MLX's weights and buffers plus PyTorch's allocator), including original MLP weights that MLX kept after building the split GPU and CPU copies. Each set of Core ML chunks also holds its own weights, and u had its own set. Running u on so's chunks and dropping the original MLP weights kept swap under 0.7 GB and removed most of the run-to-run timing variance. Capping MLX's cache of freed buffers, and mx.set_wired_limit, which keeps MLX's buffers resident so the memory compressor never touches the weights, bought a few percent more.

Memory also blocked the remaining engine move. Unguided steps would take 156 ms on the Neural Engine instead of 215 ms on the GPU, but they condition on text and need a third set of chunks, which brought back 2–4 GB of swap. Core ML multifunction packages, which let two fixed-shape functions share one deduplicated weight file, reduced that to transient swap, and the configuration then only tied the default on this 24 GB machine.

The codec runs in Neural Engine idle time that the main process declares

The codec encodes each window's input into the observation condition and decodes each finished window into audio. Both run in the Neural Engine process as Core ML chunks of 30–60 ms, and a chunk starts only while the main process has declared the Neural Engine idle: for 0.18 s on each unguided step, when the GPU works alone, and for 0.5 s after each window. Each window is decoded while the next one samples, and the next window's observation is encoded ahead of time. A file's first window has no earlier window to overlap, so its encode starts at once and runs while the GPU computes the Granite features.

Declared idle time still costs something. Running Granite on the Neural Engine as well removed its work from the GPU but made clips slower overall. We did not isolate the cause; the most likely one is the CPU work every Neural Engine call brings (dispatch and conversion of large fp32 arrays), which slows the split MLP like any other CPU work. Splitting one branch's layers between the GPU and the Neural Engine failed the same way: guided steps got faster, but the per-step hand-off and the lost idle time for the codec left clip time unchanged.

Measured effect of each change on clip time

Each entry comes from an interleaved A/B at the time, repeated until the direction held (run-to-run spread was 0.1–0.3 s per clip); the entries are not strictly additive.

Quality matches the CUDA run except a ScoreQ offset that predates the Neural Engine

The benchmark is the one the CUDA production run uses. ScoreQ-NR is a no-reference speech-quality predictor on a 1–5 scale; WER is the word error rate of an ASR transcript of the restored speech against the reference transcript; speaker similarity is the cosine between speaker embeddings of the restored and clean speech; UTMOS is a predicted mean opinion score. The Mac draws its sampler noise from CPU generators, so it does not reproduce CUDA's noise, and per-clip differences include seed variation of about ±0.01 for ScoreQ and speaker similarity and ±0.02 for WER. Intervals are bootstrap intervals over per-clip differences.

Only the ScoreQ interval excludes zero, at one to two times the seed-to-seed noise. The first full run, before any branch ran on the Neural Engine, had the same offset (−0.025 pro, −0.018 flash), and it did not grow afterwards.

Every change was smoke-tested on two clips before its timing counted and re-scored on all 78 clips when it moved outputs beyond rounding. The final rewrite produces bit-identical audio to the benchmarked pipeline. From a cold start, a 3-minute file restores in 32 s, including compilation of the first window's graphs and excluding an 8 s model load.

Mac mini setup changes

The software was macOS 27, MLX 0.32, coremltools 9.0 and PyTorch 2.10. Core ML models were converted as fp16 ML programs for macOS 15 with CPU-and-Neural-Engine compute units and take fp32 NumPy inputs. Arrays cross the pipe to the Neural Engine process at under 1 MB per guided step, plus about 5 MB of conditions per window. These balances were measured on the M4 Pro; the M5 generation adds matrix units to the GPU, which would shift them.

Sources

• Apple Machine Learning Research, Deploying Transformers on the Apple Neural Engine (2022): the (B, C, 1, S) layout with 1x1 convolutions and chunking of intermediate tensors for cache residency and multicore use, in the partitioning section.
• Inside the M4 Apple Neural Engine, Part 2: ANE Benchmarks: the ~32 MB SRAM cliff and the ~30% penalty for matrix multiplies whose operands exceed it, in the partitioning section.
• Orion: Characterizing and Programming Apple's Neural Engine for LLM Training and Inference (arXiv:2603.06728): the 32 MB SRAM, the ~19 TFLOPS fp16 peak through private APIs and fp16's range, in the engine table and the Core ML section.
• Hello SME! Generating Fast Matrix Multiplication Kernels Using the Scalable Matrix Extension (arXiv:2409.18779): the M4 CPU's SME units, in the CPU section.
• MLX documentation: mlx.core.set_wired_limit and compilation: wired memory and mx.compile, in the memory and MLX sections.
• Core ML Tools: multifunction models: weight sharing between functions, in the memory section.
• Apple Machine Learning Research, Exploring LLMs with MLX and the Neural Accelerators in the M5 GPU: the M5 GPU's matrix units, in the setup section.

Related Diffio Posts

• Classifier-Free Guidance: The Guidance Scale Is an Exponent on an Implicit Classifier: why a guided step needs forward passes with and without conditions.
• Flow Matching: Regress the Field the Sampler Integrates: the velocity field the samplers integrate.

The prediction target is the first generator constraint. It strongly shapes the loss family, the sampler, the latency path, the representation boundary, and the kinds of errors that can compound at inference time. A generator that emits token IDs learns a categorical distribution. A generator that samples mel frames works over a continuous acoustic field, even if the network head predicts velocity or noise rather than the frame itself. A generator that emits semantic tokens still needs another model to render timbre, timing, prosody, and residual acoustic detail. A generator that emits continuous latents needs a sampler and a latent space that remain stable under prediction.

Backbone choice matters, but many recent TTS systems share familiar Transformer, DiT, Llama, Qwen, Ministral, or flow-matching machinery. The clearer first map is by sampled representation, with generation schedule tracked beside it.

The sampled representation constrains the generator

For multi-stage systems, track both the long-range sampled representation and the local acoustic-rendering representation. The sampled representation is neither the final waveform nor every internal tensor predicted by a neural network head. It is the representation that carries the acoustic modeling burden before a vocoder or codec decoder turns it into waveform.

Generation schedule is the second classification axis. Autoregressive, masked, non-autoregressive, delay-pattern, within-frame residual-depth, and ODE samplers create different critical paths over the same representation. Frame rate is not the same thing as total work or total bitrate.

The table separates sampled representations and sampler choices rather than ranking systems. A "language model" over audio can mean token IDs, continuous vectors, semantic codes, or a mixed frame. A "diffusion" or "flow" model can be the main generator, the acoustic renderer, or a local head attached to a token model. Semantic tokens are content-biased but can retain speaker and prosodic information.

Discrete codec tokens make TTS a categorical sequence-modeling problem

Discrete codec TTS makes speech look like language modeling. A neural codec converts waveform into a grid of codebook IDs. The generator predicts those IDs, and the decoder converts a legal code stack back into audio.

VALL-E is the canonical version of this setup. It uses an EnCodec representation with 75 frames per second and eight residual vector quantization codebooks. VALL-E predicts the first codebook autoregressively and fills the remaining codebooks with non-autoregressive models.

Discrete codec tokens let the generator use cross-entropy, prompting, and sampling temperature, but quality and controllability remain bounded by codec rate, residual depth, and decoder behavior. A correct-looking token sequence can still sound wrong if the codec does not preserve the cue the task needs.

MOSS-TTS is the low-frame-rate, high-RVQ-depth case. It lowers the sequential frame rate to 12.5 frames per second while increasing residual quantization depth to 32 layers. That does not make acoustic modeling cost disappear. It changes the critical path from many temporal decisions to fewer temporal decisions with deeper within-frame reconstruction.

The codec-token design choice is how to allocate modeling capacity across temporal steps, residual codebooks, and within-frame decoding. Once the time axis is compressed, autoregressive temporal prediction is only one part of the problem. The model must also predict the residual-depth position within each frame.

Mel-frame systems generate continuous acoustic features

Mel-frame systems leave discrete neural codec tokens out of the acoustic generator's sampled representation. The target is a dense time-frequency representation, usually an 80- or 100-bin log-mel frame sequence, and a separate vocoder turns that representation into waveform. The vocoder is not neutral: mel targets discard phase and rely on the vocoder prior for final waveform detail.

Voicebox, Matcha-TTS, and F5-TTS generate log-mel frames with non-autoregressive flow-matching models before a vocoder. Voicebox uses 80-dimensional log-mel frames at 100 Hz, and F5-TTS uses 100-channel log-mel frames before Vocos. Their inference cost is controlled by the number of function evaluations, or NFEs, rather than by autoregressive token count.

The model samples continuous acoustic features before vocoding. This avoids dependence on a codec codebook layout, but duration, alignment, conditioning, vector-field sampling, and vocoder behavior remain part of the design.

For TTS, flow matching shifts the design problem from vocabulary construction to vector-field modeling.

Semantic-token cascades separate content modeling from acoustic rendering

Semantic-token systems factor speech generation into semantic-token prediction and acoustic rendering. First, sample a content-biased representation. Then, render acoustics from that representation plus speaker, prompt, style, or text conditioning. "Semantic" can mean ASR-supervised units, self-supervised clusters, phoneme-like codes, or a semantic branch inside a codec.

CosyVoice is a representative supervised version. It extracts semantic tokens from an ASR-oriented model, uses a language model to generate those tokens from text, then uses a flow-matching acoustic model and HiFi-GAN vocoder to synthesize speech. MaskGCT is a masked-generation version: it uses a semantic tokenizer for content and an acoustic codec for waveform detail, then generates both stages with iterative masked prediction rather than a single left-to-right token pass.

The appeal is modularity. Text correctness and acoustic rendering are no longer forced into one token stream. Semantic tokens reduce the sequence length between text conditioning and acoustic rendering by replacing waveform-length acoustic detail with lower-rate content-biased units. The acoustic model can then focus on speaker identity, pitch, rhythm, and fine acoustic detail.

WER can overstate a semantic tokenizer's suitability for generative TTS. Word error rate, or WER, measures lexical recoverability but not whether the tokens preserve timing and acoustic correlations needed by a generator. The WER Trap paper shows speech tokens can score well on ASR-like tests while still failing to preserve the acoustic variation a generator needs. It proposes flow-probe diagnostics that ask whether a flow model can reconstruct acoustic trajectories from the tokens, not just whether an external recognizer can read them.

The generator constraint for a semantic cascade is two-sided. The first stage must preserve enough content to guide the utterance, and the second stage must be able to recover the missing acoustics without inventing the wrong speaker, rhythm, or affect.

Continuous latents replace codebook indices with vector prediction targets

Continuous-latent TTS removes the categorical codebook interface. Instead of sampling token IDs, the generator samples vectors from a learned audio latent space. These are not just mel frames with a different name: mel bins are hand-designed acoustic features, while continuous latents are learned compressed vectors intended to be reconstructable and predictable. That makes the sampled representation a regression, diffusion, or flow problem rather than a softmax problem.

LatentLM is the cleanest general statement of this idea. It replaces next-token classification with next-token diffusion over continuous latents. For speech synthesis, it evaluates low-rate speech latents down to 3.75 Hz, making the context length look more like text while keeping the prediction target continuous.

VibeVoice is the long-form TTS example: a 7.5 Hz continuous acoustic latent plus a next-token diffusion head shifts the budget from sequence length to latent geometry. Its coupled shared-latent tokenizer ablation reduces perceptual speech quality (PESQ) and objective speaker similarity (SIM-O), which is why the related VibeVoice and latent-encoder posts treat rate, width, variance, and semantic alignment as generator constraints.

SemaVoice makes the same point from another direction. It uses continuous speech tokens at 15 Hz, and its frame-rate study shows that simply increasing temporal resolution is not automatically better. The denser 60 Hz setup degrades sharply in that study, especially without alignment. Without the right alignment and capacity, a denser latent stream can make prediction harder rather than easier.

A representation is both a compression format and the target distribution the generator must learn.

Hybrid frames combine autoregressive semantic tokens with local acoustic generation

Recent systems increasingly combine the earlier choices rather than choosing only one. Voxtral TTS is a useful marker. Its codec runs at 12.5 Hz and represents each frame with one 8192-entry semantic VQ token plus 36 finite-scalar-quantized acoustic coordinates. The decoder backbone generates the semantic stream autoregressively. A flow-matching acoustic head models those 36 acoustic coordinates in continuous space, runs eight NFEs by default, then discretizes the result back to the 21-level FSQ interface.

Voxtral combines a codec representation with a flow-matching acoustic head. The semantic stream keeps a language-model-style global ordering. The acoustic head avoids making all fine acoustic detail a long sequence of categorical decisions. Streaming then becomes a coordination problem between the autoregressive semantic loop, the local acoustic generator, and the audio decoder.

MOSS and Voxtral address the same residual acoustic modeling problem with different allocations: MOSS models RVQ depth within each frame, while Voxtral models local acoustic coordinates with a per-frame flow model. VibeVoice offers another answer by keeping the low-rate acoustic representation continuous.

Reducing sequential frame rate relocates acoustic detail into residual depth, a semantic-acoustic cascade, a local flow head, a mel-frame sampler, or a continuous latent space.

Five fields locate a TTS system

A new TTS system can be classified by five fields.

1. Which representation does the primary acoustic generator sample before waveform rendering?
2. Which generation schedule determines the critical path for that representation?
3. At what frame rate does the system make sequential decisions?
4. Where is acoustic detail represented when the frame rate is low?
5. Which metric tests the representation itself rather than only the final audio?

These fields distinguish systems that share a frame rate, training objective, or tokenizer metric. A 12.5 Hz codec-token model and a 12.5 Hz hybrid flow model share a temporal rate, but they do not share a sampler or loss. A flow-matching mel model and a flow-matching acoustic-token head share training math, but not the sampled representation. A semantic tokenizer with strong WER is not automatically a good TTS latent. A continuous latent encoder that reconstructs well is not automatically easy for a language model to predict.

Sources

• VALL-E: discrete-codec TTS, 75 Hz EnCodec representation, eight residual codebooks, and autoregressive plus non-autoregressive codebook generation.
• Voicebox: non-autoregressive flow matching over 80-dimensional log-mel frames at 100 Hz and the NFE tradeoff for sampling.
• Matcha-TTS: optimal-transport conditional flow matching for non-autoregressive acoustic modeling.
• F5-TTS: direct non-autoregressive flow matching over 100-channel log-mel frames with Vocos as vocoder.
• CosyVoice: supervised semantic tokens, text-to-token language modeling, and flow-matching acoustic rendering.
• MaskGCT: semantic and acoustic token stages with masked generative codec Transformers.
• MOSS-TTS: low-rate 12.5 Hz codec tokens, 32 residual codebooks, and residual-depth modeling.
• Voxtral TTS: hybrid semantic-token plus finite-scalar-quantized acoustic-code architecture, Figure 2, Voxtral Codec, and per-frame flow-matching acoustic generation.
• VibeVoice: 7.5 Hz semantic and continuous acoustic tokenizers, next-token diffusion head, and tokenizer ablations.
• The WER Trap: evidence that semantic-token WER can miss generative acoustic suitability.
• LatentLM: next-token diffusion over continuous latents and low-rate speech-latent experiments.
• SemaVoice: continuous autoregressive speech synthesis and frame-rate sensitivity for continuous speech tokens.

Related Diffio posts

• Flow Matching
• MOSS-TTS: RVQ Depth Is The Bottleneck
• VibeVoice: Next-Token Diffusion For Long-Form Speech
• Training Latent Audio Encoders

Reconstruction quality still matters, as a ceiling: nothing generated through the decoder can sound better than the encoder-decoder round trip, so across large fidelity gaps it ranks systems coarsely. It cannot order competitive candidates: four sources show this directly by holding the downstream model fixed and varying only the latent space, and the autoencoder that reconstructs best is routinely not the one whose latents generate best. The remaining design axes, frame rate and width, regularization, semantic alignment, and training schedule, each target properties that reconstruction training does not measure.

Four fixed-generator comparisons decouple reconstruction from generation

Each source trains one downstream generator configuration per latent space, so the latent is the only deliberate variable.

The first two rows come from one SAME ablation. Metrics: MELlog1p is a multi-resolution log-mel reconstruction distance (lower is better); FAD-CLAP is Fréchet audio distance in the embedding space of CLAP, a contrastive audio-text model (lower is better); STOI is a 0–1 intelligibility measure and PESQ a reference-based speech quality score, both computed against the original audio; WER is word error rate of an ASR model run on the synthesized speech; UTMOS and DNSMOS are model-predicted mean-opinion quality scores; text-audio similarity is a CLAP-embedding match score and KAD a kernel-based distribution distance between real and generated audio embeddings; SIM is speaker similarity, the cosine between speaker-embedding vectors.

LongCat-AudioDiT, Figure 3, p. 11. Solid lines are the Wav-VAE's reconstruction metrics, dashed lines the paired TTS model's, against latent dimension; the intelligibility panel plots negated WER, so falling means worse. The decoupling is sharpest in the intelligibility and quality panels; TTS speaker similarity peaks at 128 before dropping at 256, and the naturalness metrics decline on both sides.

The rows differ in what they isolate. The SAME pair puts the disagreement in distributional structure that the bottleneck's fixed statistics do not capture; the SAME post traces which auxiliary losses put that structure in. LongCat shows the failure is not a generator-capacity problem: aside from a marginal speaker-similarity gain, scaling the TTS backbone to 3.5B on the 128-dim VAE still loses to 3.5B on the 64-dim VAE, so under a fixed training recipe the extra width is a modeling burden that added parameters do not offset. Target-KL shows the disagreement inside a single architecture family swept along one axis, which rules out confounds between recipes. And Semantic-VAE shows the converse: a latent change that improves generation substantially while reconstruction does not move at all.

The direction of the trade is not universal: AudioLDM's mel-spectrogram VAE reconstructs comparably across its compression sweep while generation quality falls as compression rises, the opposite of LongCat's frame-rate result below, so the optimum is task- and architecture-dependent. The durable conclusion is the selection rule: only a paired downstream model ranks latents as generation targets. Every sweep above paid for a full generator per candidate; whether a small jointly trained probe would produce the same ranking at a fraction of the cost is untested.

Frame rate and width set the prediction problem

Frame rate and channel width are fixed before any training loss applies, and they define the object the generator must predict: rate sets the sequence length and how much information each step carries, width sets the dimensionality of the continuous vector the diffusion head or distribution predictor must model per step. Sequence length is what an autoregressive generator pays for in context and compute, which is why long-form TTS systems push rates down to 7.5–15 Hz; the VibeVoice post works out that budget arithmetic. The two axes convert: in LatentLM's compression ablation, dropping the frame rate from 37.5 to 15 Hz at fixed width costs reconstruction similarity (0.866 to 0.700), and doubling the width restores it (0.870), with downstream TTS similarity following the same pattern. Information removed from the time axis must be carried by added latent dimensions.

Two sweeps locate the best shape for autoregressive speech at the same information rate, 480 floats per second:

LatentLM, Tables 4-5; SemaVoice, Table 4. LatentLM sweeps downward from 15 Hz and loses speaker similarity as the rate falls; SemaVoice sweeps upward and loses intelligibility as the rate rises, while its reconstruction stays roughly stable. Jointly the two bracket 15 Hz as the optimum at this budget, approached from opposite directions; the sweeps differ in data, benchmark, backbone, and head, so they do not trace a controlled curve. And because the budget is fixed, rate and width move together: the 60 Hz failure happens in 8-dim frames, so the high end cannot separate a rate ceiling from a width floor. LongCat's rate sweep points the same way from the non-autoregressive side: lowering the frame rate at fixed dimension 64 costs VAE reconstruction similarity and PESQ while improving the paired TTS model, except for TTS speaker similarity, which peaks at 11.72 Hz, the rate LongCat ships.

LongCat-AudioDiT, Figure 4, p. 12. Solid lines are the Wav-VAE's reconstruction metrics, dashed lines the paired TTS model's, against latent frame rate; the quality panel shows the two moving in opposite directions, and the similarity panel the TTS peak at the middle rate.

Width gets two defensible answers. LongCat's response to its dimension sweep is to keep the latent narrow, 64-dim at 11.72 Hz. SAME keeps a wide 256-dim latent at a frame rate close to LongCat's and restores predictability with auxiliary semantic and generation losses. LongCat's sweep varies width alone and measures the cost of raw width; SAME demonstrates that shaped width is usable, though its own table complicates a pure width story: the unshaped wide VAE already beats the narrow configuration on generation, a comparison that confounds width with frame rate. No source varies width and shaping independently. Geometry beyond the uniform time grid is also in play: Music2Latent2 packs each audio chunk into a set of unordered summary embeddings so global features are not re-encoded at every frame, and TADA collapses 50 Hz frames into one vector per text token, trading the time grid for text alignment.

Regularization decides which component absorbs prediction error

A predicted latent is always slightly off the encoder's output manifold. Regularization choices decide which component absorbs that error, and the systems here give four answers that shipping recipes combine.

The latent distribution absorbs it. A vanilla VAE lets per-channel posterior variances collapse toward zero, leaving sampled latents brittle as autoregressive context. A sigma-VAE takes the variance out of the encoder's hands, training the decoder and the generator under latent noise of a designed scale so that prediction errors smaller than that scale stay in-distribution; the exposure bias post develops the mechanism and its evidence. LatentLM, VibeVoice, and SemaVoice all build their speech tokenizers on the recipe.

The decoder absorbs it. Injecting Gaussian noise into latents during autoencoder training builds the same robustness into a different component, and it is nearly free. LatentFlowSR's ablation isolates the effect: under inference-time latent noise its plain autoencoder collapses (LSD 3.42, ViSQOL 2.36, where LSD is log-spectral distance, lower better, and ViSQOL a 1–5 perceptual quality estimate) while the noise-trained version barely moves (0.92, 4.22), and on clean audio the noise-trained autoencoder reconstructs slightly better, not worse. SAME trains with latent noise at 5e-2 and keeps a smaller 1e-3 noise on the latent at inference.

The generator absorbs it. An under-regularized latent reconstructs well but is sharp and perturbation-sensitive, so the full difficulty lands on the predictor; an over-regularized latent is smooth but caps fidelity. Most recipes pick a point on that trade implicitly through an ad hoc KL weight (Stable Audio Open down-weights KL by 1e-4 and accepts whatever rate emerges); Target-KL replaces the weight with a specified information rate, a squared penalty pulling measured KL toward a chosen value, which is what made the sweep in the table above possible. Its TTS results carry one caveat: some high-bitrate VAEs scored deceptively low WER because the diffusion model copied information from unregularized prompt latents, with monotonous-sounding output. KALL-E occupies the deliberately smooth end, keeping a high KL weight (32) and accepting worse reconstruction for a latent its autoregressive distribution predictor can fit.

A separate pathway absorbs it. VibeVoice routes semantics around the generated latent entirely: its acoustic tokenizer stays purely reconstructive, and at each step a frozen ASR-trained encoder re-encodes the audio just generated and adds the result to the autoregressive history.

SAME goes one step further: rather than only making errors survivable, it trains a small flow-matching model jointly with the encoder and backpropagates that model's loss, so the latent space itself is optimized to be easy for a generator to model. The SAME post covers the mechanism; flow matching is the objective both that probe and most of the downstream generators here train with.

Semantic alignment strength must match latent capacity

Semantic objectives change what the latent encodes rather than how much, and whether they conflict with acoustic fidelity depends on the alignment's strength and the latent's capacity, not on semantic alignment per se.

A hard semantic objective on a small bottleneck degrades both sides. VibeVoice pushed a full transcript-decoding ASR objective through a 7.5 Hz, 64-dim latent and collapsed reconstruction and speaker similarity together (PESQ 1.92 against 2.98 for its reconstruction-only tokenizer); the mechanism, two objectives competing for fixed capacity, is analyzed in the VibeVoice post. The WER Trap measures the same competition from the semantic side with a discrete 5 Hz ASR-supervised token: adding a reconstruction loss costs character error rate (11.98 to 14.32) and audio-QA accuracy. Both land on decoupling the objectives.

A gradient-bounded alignment loss improves both sides. SemaVoice aligns its 15 Hz latent to frozen WavLM features with a frame-wise cosine loss plus a pairwise self-similarity loss, and sets the alignment weight each step so its gradient norm is half the reconstruction loss's, so the alignment can never dominate. It improves WER and speaker similarity together, costs almost nothing on reconstruction, and its value grows with frame rate: at 60 Hz, where the aligned sweep has already degraded to WER 14.71, removing the alignment doubles the damage to 28.06. Loss form matters as much as weight: in Semantic-VAE, cosine alignment to a late WavLM layer improves TTS, while L1 and MSE alignment to the same teacher are worse than no alignment at all (WER 3.12 and 4.37 against the 2.65 baseline); magnitude-constraining losses over-restrict the latent where directional alignment only orients it. CALM extends to its full continuous latent the WavLM cosine distillation that Mimi, the discrete tokenizer it derives from, applies only to its first codebook.

A weak objective on a wide latent is nearly free. SAME applies linear probes regressing DSP features plus a contrastive text critic to its 256-dim latent and takes the best generation scores in its ablation at a small reconstruction cost. The availability of a teacher explains why the recipes differ across domains: speech has WavLM-class self-supervised content teachers, so SemaVoice, Semantic-VAE, and CALM align to them; music has no equivalent, so SAME builds its semantic targets from the signal (chromagrams, interaural level difference) and from paired text. No single source varies alignment strength at fixed rate and width, so where soft becomes hard, and how width moves that threshold, remains unmeasured.

Staged training routes adversarial gradients away from the encoder

The sources here disagree about freezing schedules but agree about routing: losses that shape the latent for generation (semantic alignment, SAME's diffusion probe) are deliberately sent into the encoder, while adversarial and feature-matching gradients that fit fine-grained waveform detail are kept off it during late fine-tuning. RAVE shows why: it trains its VAE for representation first, then freezes the encoder and fine-tunes only the decoder adversarially, and its appendix shows that letting the stage-two feature-matching loss reach the encoder dramatically inflates the latent space's estimated dimensionality (by SVD, its 128 nominal dimensions hold about 16–24 informative ones when the encoder is protected). RAVE measures the harm as lost compactness, not generation quality, but spreading information across more dimensions is exactly the raw width whose cost the LongCat sweep measured. The pattern at scale: Stable Audio Open trains end to end with adversarial losses for 183 hours, then spends 273 hours training the decoder alone against a frozen encoder, and SAME freezes its encoder for two decoder-only stages.

Sources

• SAME: A Semantically-Aligned Music Autoencoder: the fixed-budget bottleneck ablation rows in the decoupling table (Table 2, Sec. 5.1); noise injection scales (Sec. 2.3); the linear-probe and contrastive losses and the joint diffusion probe (Sec. 3.3); frozen-encoder decoder stages (Sec. 4.2).
• LongCat-AudioDiT: High-Fidelity Diffusion Text-to-Speech in the Waveform Latent Space: the dimension and frame-rate sweeps with paired TTS models, the 3.5B scaling check, and Figures 3 and 4 (Sec. 5.3.2, pp. 11-12).
• Taming Audio VAEs via Target-KL Regularization: target-KL as a specified information rate (Secs. 1-2), the rate-distortion curve (Fig. 1), the text-to-sound generation peak and the TTS prompt-copying caveat (Tables 2-3, Sec. 4).
• Semantic-VAE: Semantic-Alignment Latent Representation for Better Speech Synthesis: reconstruction-equivalent VAEs with diverging F5-TTS results (Tables 1-2) and the cosine/L1/MSE loss-form ablation (Table 3).
• Multimodal Latent Language Modeling with Next-Token Diffusion (LatentLM): the sigma-VAE recipe (Sec. 2.3), the frame-rate sweep (Tables 4-5), and the compression/width conversion ablation (Table 6).
• SemaVoice: Semantic-Aware Continuous Autoregressive Speech Synthesis: the gradient-ratio-capped WavLM alignment (Eqs. 3-7) and the granularity sweep with and without alignment (Table 4).
• VibeVoice Technical Report: the coupled-tokenizer collapse, the hybrid acoustic/semantic routing, and the semantic feedback path (Tables 5 and 7, Appendices B and D).
• The WER Trap: the reconstruction/semantics competition measured from the discrete-token side (Tables 1-2).
• LatentFlowSR: High-Fidelity Audio Super-Resolution via Noise-Robust Latent Flow Matching: the latent-noise-injection robustness ablation (Tables 4-5).
• RAVE: A Variational Autoencoder for Fast and High-Quality Neural Audio Synthesis: two-stage training with the frozen encoder (Sec. 3.1.2), the SVD dimensionality estimate (Sec. 5.3, Fig. 2), and the unfrozen-encoder dimensionality inflation (Annex E, Fig. 8).
• Stable Audio Open: the standard loss substrate, the 1e-4 KL weight, and the joint/frozen-phase compute split (Sec. 2).
• AudioLDM: Text-to-Audio Generation with Latent Diffusion Models: the compression-level sweep where reconstruction holds and generation degrades (Tables 4 and 8).
• KALL-E: Autoregressive Speech Synthesis with Next-Distribution Prediction: the deliberately high KL weight and its stated rationale (experimental setup and Table 1 discussion).
• Music2Latent2: Audio Compression with Summary Embeddings and Autoregressive Decoding: unordered summary embeddings (Secs. 1-3).
• TADA: A Generative Framework for Speech Modeling via Text-Acoustic Dual Alignment: text-token-aligned acoustic vectors (Sec. 3).
• Continuous Audio Language Models (CALM): WavLM distillation over the full continuous speech latent, contrasted with Mimi's first-codebook distillation (Sec. 4.1).

Related Diffio posts

• SAME Makes Generation Difficulty an Autoencoder Training Loss: the single-system deep dive on shaping a latent with a jointly trained generation probe.
• VibeVoice: Frame Rate Is the Context Budget for Long-Form TTS: the frame-rate budget arithmetic and the coupled-tokenizer capacity analysis.
• Exposure Bias: Corrupt the Context, Keep the Target Correct: why autoregressive consumption makes latent variance an error-tolerance radius.
• Flow Matching: Regress the Field the Sampler Integrates: the training objective of most generators these latents are built for.
• TTS Models: A Map Of The Modern Speech Stack: where continuous-latent systems sit among the TTS architecture families.

A 4x shorter, 4x wider latent at the same scalar rate

Raising temporal downsampling shortens the sequence a downstream generator must model; at fixed latent dimension it also raises compression difficulty, so SAME widens the latent to compensate. At 10.8 Hz and 256 channels the latent carries about 2,760 floats per second, exactly the scalar rate of a conventional 1024x, 64-channel audio VAE: the budget is unchanged, reshaped into a 4x shorter, 4x wider sequence, because sequence length is what a generator pays for in compute and context. A 6-minute track is about 3,900 latent frames. Early latent-diffusion practice kept latent dimension small on the belief that wide latents are hard to generate in; SAME instead cites the representation-autoencoder (RAE) results from images, where large dimensions are tractable when the latent is semantically structured.

Patching and transformer resampling replace strided convolution

SAME paper, Figure 1, p. 2. The auxiliary losses attach at the bottleneck, on the latent itself; the reconstruction and adversarial losses act on the decoded waveform.

There are no learned convolutions on the signal path. A parameter-free patching pretransform reshapes stereo waveform (B, 2, T) into (B, 2P, T/P) with P=256, so each embedding concatenates 256 left-channel and 256 right-channel samples: 256x downsampling as a pure reshape. The remaining 16x comes from a Transformer Resampling Block (TRB): the patch sequence is partitioned into non-overlapping 16-embedding segments, one learnable output embedding is appended to each, a transformer stack processes the interleaved sequence, and only the output embeddings are kept. The decoder TRB reverses the roles, pairing each latent with 16 learnable outputs.

SAME paper, Figure 2, p. 3, shown with stride 2 for clarity; the released models use stride 16. Downsampling happens through attention into one learned token per segment, not through pooling or strided convolution.

In SAME-L the transformer layers use differential attention with RoPE and per-head QK-normalization, and the last 8 decoder layers swap their feed-forward activations for sin(pi x), a periodic basis for waveform detail; SAME-S drops both for CPU throughput. Both variants replace LayerNorm with Dynamic Tanh, a learnable tanh activation, avoiding normalization statistics that near-silent audio segments corrupt. Because training segments are a few seconds but inference must handle arbitrary lengths, full attention is unsuitable: SAME-L uses sliding-window attention, and SAME-S uses chunked attention whose chunk grid shifts by half a chunk midway through the stack, so first-half chunk boundaries fall in second-half chunk interiors and stop producing audible boundary artifacts; CPU inference libraries such as LiteRT do not support sliding windows. The all-transformer design buys speed: SAME-L runs at about twice the real-time factor (audio duration over encode-plus-decode wall clock) of smaller convolutional VAE baselines despite its 852M parameters, which the paper attributes to well-optimized transformer primitives plus the high compression ratio.

Soft-normalization: a deterministic bottleneck that pins statistics

Where most continuous audio codecs put a VAE, SAME puts a deterministic normalization with no sampled posterior. The encoder output passes through a learnable per-channel affine transform, then is divided by a running standard deviation tracked by exponential moving average. A penalty of Gaussian KL form pushes means and variances toward zero and one along two axes independently, per channel over time and per timestep across channels, so neither individual channels nor individual frames drift to extreme statistics. Gaussian noise scaled by the running standard deviation is added to the latent (5e-2 during training, 1e-3 at inference), smoothing the latent manifold so the decoder tolerates the imperfect latents a downstream generator will hand it.

L_diff: an auxiliary diffusion model trains jointly and backpropagates into the encoder

When building discrete tokenizers it is common to jointly train a small autoregressive prior whose loss backpropagates into the encoder, keeping the token distribution easy for that prior to model; for continuous latents under diffusion the paper calls the equivalent move rarer, citing a single recent precedent. SAME's version is the loss it calls L_diff: a 4-layer, 768-dimensional diffusion transformer trains on the latents with the standard straight-path flow-matching objective, noising z_t = (1-t)z + t*eps and regressing the velocity eps - z, with timesteps drawn from a truncated logistic-normal distribution. The auxiliary model warms up on detached latents; after warmup, its gradients flow into the encoder.

The auxiliary model is unconditional and exists only as a training signal. L_diff rises when the current latent distribution is hard for a flow-matching model to fit, so minimizing it with respect to the encoder pushes the latent toward geometries that are easier to generate in. An unconditional model this small is a coarse proxy for the text-conditioned generator that will actually use the latent; Stable Audio 3 pre-trains on the same frozen latents with the same straight-path velocity parameterization (its full objective adds minibatch optimal-transport coupling and an inpainting mask), so the encoder is shaped during its own training by a scaled-down version of the downstream generator.

L_sem and L_con: linear chroma and stereo regressors, plus a contrastive text critic

Speech tokenizers usually buy semantic structure by aligning the latent to a self-supervised teacher such as WavLM. SAME aligns to no pretrained teacher; its semantic regression loss, L_sem, supervises with quantities computed from the signal itself. Single 1x1-convolution regressors predict three octave-banded chromagrams (per-frame energy by pitch class) and an interaural level difference (the per-band left/right log-magnitude gap that encodes the stereo image) directly from the latent; because the regressors are linear, pitch-class and spatial structure are pressured to stay linearly decodable rather than nonlinearly entangled. A separate contrastive loss, L_con, trains a critic to score whether a latent sequence, a wavelet decomposition of the audio, and a T5Gemma embedding of the paired text description come from the same input, with negatives formed by re-pairing audio and text within the batch and heavy masking of latent and feature positions to block shortcut cues. Like the auxiliary diffusion model, the critic warms up on detached latents before its gradients reach the encoder.

The pressure is mild: linear regressors and a warmed-up critic on a 256-dimensional latent. VibeVoice found that the opposite setting, a hard transcript-decoding objective pushed through a 7.5 Hz, 64-dimensional speech latent, degraded reconstruction and collapsed speaker similarity.

The bottleneck ablation: reconstruction pinned, generation decoupled

The ablation holds everything fixed except the latent space: 50k autoencoder steps with the SAME-L backbone and no adversarial losses (removing GAN dynamics as a confounder), then a roughly 1.4B-parameter diffusion transformer (DiT) trained with flow matching on each latent space for 50k steps. VAE variants use a single, unswept KL weight (1e-4). Both budgets are short, so the comparison measures how readily a DiT fits each latent space at fixed compute, not converged quality.

SAME paper, Table 2, p. 8. MELlog1p is a multi-resolution log-mel reconstruction error (lower is better); FAD-CLAP is Fréchet audio distance in the embedding space of CLAP, a contrastive audio-text model, computed on the DiT's caption-conditioned outputs (lower is better); MuQEval is a model-based, reference-less musical quality score (higher is better).

Configs A and B reconstruct identically on the table's metric and generate very differently: FAD-CLAP 0.651 against 1.061. The statistics soft-normalization pins (means, variances, magnitudes) survived the swap, so whatever the DiT needed lives in distributional structure beyond those statistics, which a reconstruction metric cannot see. Adding the auxiliary diffusion model (config C) more than repays the simplification, beating the VAE baseline on all three metrics at no reconstruction cost; the table cannot say whether the gain is diffusion-specific shaping or a more generic smoothing pressure, and the untested VAE-plus-L_diff cell is what would separate the bottleneck's contribution from the auxiliary model's. Adding the semantic losses (config D) trades a little reconstruction for the best generation scores, with a MuQEval jump the paper reads as musical structure becoming easier to model; L_sem and L_con are added together, so their individual contributions are unknown.

Config E is the conventional 1024x/64 shape: the table's best reconstruction, and worse generation than A, C, and D. All generation evidence in the paper is this internal ablation; no generator is trained on a competitor's latent space. The same reconstruction/generation decoupling has been measured independently across speech and audio systems.

Full-scale reconstruction: best mel error and listening score from a 4x shorter sequence

The released models train in three stages on about 19,500 hours of licensed production audio (a 66/25/9 percent music, sound-effects, instrument-stems mix): an end-to-end stage with all losses, then two decoder-only stages with the encoder frozen, the second swapping in a transformer-based discriminator ensemble and synthetic linear frequency sweeps against aliasing.

SAME paper, Table 1, p. 8; baselines are recent open-weights continuous audio autoencoders. Objective metrics use 446 captioned music tracks from the Song Describer Dataset: SI-SDR is scale-invariant signal-to-distortion ratio in dB (higher is better), STFTlog1p a log-magnitude STFT distance (lower is better), CCPC cross-channel phase coherence for stereo-image fidelity (higher is better). RTF is averaged over 50 two-minute tracks, FP16 on one H100. MUSHRA is a 0-100 listening test on excerpts, here 36 valid trials from 12 participants after filtering, with the hidden reference (the clean original, rated blind) at 97.6; CoDiCodec was excluded from it.

SAME-L takes the best mel error and the best mean listening score (82.2 against 77.6 for the strongest baseline, with overlapping confidence intervals), while eps-ar-VAE, the conventional 1024x/64 shape, stays marginally ahead on SI-SDR, the magnitude-STFT distance, and stereo phase coherence. Every number here is music reconstruction; despite the sound-effects share in training, non-music quality is never evaluated in the paper.

SAME-S, the 108M CPU variant, matches the Stable Audio Open VAE on objective reconstruction at 6-7x the speed; its listening score is lower (66.1 to SAO's 73.3). Its distillation aligns student latents to the frozen SAME-L teacher's with an L1 loss and applies reconstruction and adversarial losses to cross-decoded outputs, student decoder on teacher latents and the reverse, so the two models share one latent space with interchangeable latents. That is what lets Stable Audio 3 train its small model on SAME-S and its larger models on SAME-L.

Sources

• SAME: A Semantically-Aligned Music Autoencoder: all architecture and training details (Secs. 2-4), the bottleneck ablation (Table 2, Sec. 5.1), reconstruction and MUSHRA results (Table 1, Secs. 5.3-5.4), Figures 1 and 2.
• Stable Audio 3: the generator trained on SAME latents; model tiers on SAME-S/SAME-L and the flow-matching pre-training objective (Secs. 2.1, 3.2).
• Diffusion Transformers with Representation Autoencoders: the image-domain wide-latent argument SAME cites for choosing d=256.
• SAME open weights and audio examples

Related Diffio Posts

• Flow Matching: Regress the Field the Sampler Integrates
• Latent Audio Encoders: Reconstruction Sets the Ceiling, Not the Ranking
• VibeVoice: Frame Rate Is the Context Budget for Long-Form TTS

L_FM = E_{t, x ~ p_t} || v_theta(x, t) - u_t(x) ||^2

Two properties make this the default way to train continuous generators. The regression target is the same field the ODE solver integrates at sampling time, and its scale can be made uniform over t, which together make training markedly more robust than score matching at identical architecture and compute. And the path is a free choice rather than the byproduct of a diffusion process, which admits the straight optimal-transport path: a velocity target that is constant in time during training, and, empirically, the same sample quality at a fraction of the solver steps.

Per-example velocity targets optimize the intractable marginal objective

The loss above cannot be computed as written: for any path that ends at the data distribution, u_t depends on the unknown data density, so there is nothing to regress onto. Conditional flow matching (CFM) is the construction that fixes this. Condition on one data sample x_1 and define a per-example Gaussian path p_t(x | x_1) that starts at standard noise at t = 0 and ends at a concentrated Gaussian N(x_1, sigma_min^2 I) around the sample at t = 1, with sigma_min a small constant, typically 1e-5. Marginalizing over the data distribution q(x_1) yields a path whose endpoint approximates the data. Writing x_t for a sample on the path at time t, the field that generates this marginal path is the average of the conditional fields over every x_1 that could have produced the observed x_t, weighted by the posterior:

u_t(x) = E[ u_t(x | x_1) | x_t = x ]

The CFM loss regresses the network onto the per-example targets instead:

L_CFM = E_{t, x_1 ~ q, x ~ p_t(x | x_1)} || v_theta(x, t) - u_t(x | x_1) ||^2

This works because of what least squares does with targets that vary given the input: the minimizing function is their conditional expectation, which by the equation above is exactly the marginal field. Lipman et al. prove the two statements that make this exact: the marginal field assembled from conditional fields generates the marginal path (Theorem 1), and L_CFM and L_FM differ by a constant independent of theta, so their gradients are identical (Theorem 2). Training on closed-form per-example targets therefore optimizes the intractable marginal objective exactly, in expectation. The construction is the same one denoising score matching uses for diffusion models, generalized from scores to arbitrary velocity fields.

For a Gaussian conditional path N(mu_t(x_1), sigma_t(x_1)^2 I), the affine map psi_t(x) = sigma_t(x_1) x + mu_t(x_1) pushes standard noise onto the path, and Theorem 3 derives the unique field whose flow is that map (primes denote time derivatives):

u_t(x | x_1) = (sigma_t'(x_1) / sigma_t(x_1)) (x - mu_t(x_1)) + mu_t'(x_1)

A training step is then: sample t uniformly on [0, 1], a data sample x_1, and noise x_0; form x_t = sigma_t(x_1) x_0 + mu_t(x_1); take a gradient step on the squared error against the closed-form target. No ODE is solved during training. Papers do not share a time convention (the flow-matching paper puts noise at t = 0 and data at t = 1; much of the diffusion literature and several flow-matching systems run the other way), and velocity signs flip with the convention, so equations cannot be copied across papers without checking it.

The path is a free choice: diffusion paths are special cases, the OT path is the alternative

The other half of the framework is that mu_t and sigma_t are free choices. Setting them to the schedule of the variance-preserving (VP) diffusion process recovers that diffusion path exactly, and the conditional field Theorem 3 assigns it coincides with the drift of score-based diffusion's probability-flow ODE (Appendix D of the paper). Deterministic diffusion sampling and flow-matching sampling are therefore the same operation, an ODE solve, and on the same path the ideal field is identical. What distinguishes the methods is the regressed quantity, its loss weighting, and which paths the construction can express: diffusion paths must arise from a forward noising process, and one inherited consequence is that they never reach the noise distribution in finite time, so the prior at t = 0 is only ever approximated.

The optimal-transport (OT) path makes mean and standard deviation linear in time:

mu_t(x_1) = t x_1          sigma_t(x_1) = 1 - (1 - sigma_min) t
x_t = (1 - (1 - sigma_min) t) x_0 + t x_1
u_t(x_t | x_1) = x_1 - (1 - sigma_min) x_0

The conditional flow is the optimal-transport displacement map between the two endpoint Gaussians: each conditional sample moves along a straight line at constant speed, the minimal-cost interpolation under squared distance. Along its own trajectory, the regression target is the constant x_1 - (1 - sigma_min) x_0; as a field over all of space, u_t factors as g(t) h(x | x_1), so its direction at any point never changes and only its scale depends on t, a simpler regression target than one that rotates as t advances. And conditional sampling trajectories stay straight, where diffusion-path trajectories can overshoot the endpoint and backtrack.

Source: Flow Matching for Generative Modeling, Figure 2, p. 6. What to see: across the four time panels, the diffusion path's conditional score (left) changes direction and magnitude as t advances, while the OT path's conditional vector field (right) keeps a constant direction.

Source: Flow Matching for Generative Modeling, Figure 3, p. 6. What to see: trajectories of the diffusion conditional field (left) curve and backtrack; the OT conditional trajectories (right) run straight to the same endpoints.

Conditional optimality does not make the marginal flow an optimal-transport map. Where conditional paths cross, the marginal field is a posterior average of disagreeing conditional velocities, so marginal trajectories still curve; the paper claims only that the marginal field stays relatively simple, and closing this gap is what rectified flow's reflow procedure and minibatch OT coupling target.

The score target's scale diverges; the velocity loss measures the field the solver integrates

Two mechanisms separate the objectives. The first is target scale. For a Gaussian conditional path, the score (the gradient of the log density, which diffusion training regresses onto) is -(x - mu_t) / sigma_t^2 = -x_0 / sigma_t, with expected squared norm d / sigma_t^2 in d dimensions. As the path concentrates onto the data (sigma_t toward 0), the target's scale diverges, so an unweighted score regression is dominated by the times nearest the data, and every score-matching recipe inserts a per-time loss weight lambda(t) to compensate. The weightings are heuristics: Grad-TTS, a score-based TTS model, weights its per-time losses by lambda_t and states the rule it follows, that weights should be proportional to 1 / E||target||^2. The OT velocity target needs no weighting; its distribution is the same at every t. The same arithmetic says why a path ending in a near-point mass is trainable under velocity regression and pathological as a score target, whose norm at the endpoint scales as 1 / sigma_min.

The second is what the loss measures. The CFM loss is the squared error of the field the ODE solver will integrate, weighted uniformly over t. Score matching trains a score network, and the sampler then assembles the drift from it, with the score entering scaled by beta(1-t) / 2 on the VP path (beta is the VP noise schedule; the 1-t argument is the time-convention flip noted above). Converting squared score error into squared drift error therefore means weighting by (beta(1-t) / 2)^2, and neither standard recipe uses that weight: original score matching takes lambda(t) = sigma_t^2 and ScoreFlow takes beta(1-t). The trained loss and the error the solver accumulates are different time-weightings of the same error, so one can be low while the other is not.

Grad-TTS documents what that looks like: its diffusion loss converges early and then barely moves while output quality keeps improving across 1.7 million training iterations, and the authors' explanation is that score estimates inaccurate on a small subset of t barely move the weighted loss but break the ODE solve whenever the solver lands there. That much is true of any time-integrated loss, flow matching's included; what flow matching removes is the systematic mismatch. With an adaptive solver, the number of function evaluations (NFE, how many times the solver calls the network) needed to sample a score-matching model changes drastically over training, while the flow-matching models' sampling cost stays near its final value from the first epochs.

Source: Flow Matching for Generative Modeling, Figure 10, p. 21 (CIFAR-10, dopri5 solver at tolerance 1e-5). What to see: the score-matching curve (green) is the unstable one, spiking past 1,000 evaluations and drifting for hundreds of epochs, while both flow-matching curves stay near their final cost, the OT model (orange) cheapest.

Two ablations separate the objective from the path

The flow-matching paper trains the same U-Net architecture with the same hyperparameters under five configurations (three diffusion losses, flow matching on two paths), with the baselines allowed more training iterations to converge.

Source: Flow Matching for Generative Modeling, Table 1, p. 8. FID is Frechet Inception Distance, lower is better; the NFE column is the adaptive solver's evaluation count averaged over 50K samples. DDPM regresses the added noise; ScoreFlow is score matching with a likelihood-motivated weighting. None of these unconditional models is tuned for absolute state of the art (the paper suggests the architecture was not optimized for CIFAR-10 at all), so read the columns as controlled within-table contrasts.

FM with the OT path is best on every dataset and metric in the paper's full Table 1, test likelihood included. The within-path contrast carries the objective claim: holding the VP diffusion path fixed, the two score parameterizations degrade sharply on CIFAR-10 while the two constant-scale regression targets, DDPM's noise (a standard Gaussian at every t) and FM's velocity, stay stable. A well-conditioned target buys the stability whether it is noise or velocity; what noise matching does not buy is cheap integration, a roughly 40% higher adaptive-solver cost than FM on the identical path. The objective's advantage is robustness rather than uniform dominance: score matching edges FM-on-diffusion on ImageNet-32 (FID 5.68 against 6.37, with a lower adaptive NFE as well). Training cost moves the same direction, though only across papers: the ImageNet-128 FM-OT model reached its reported FID on a third fewer total training images than the ADM baseline of Dhariwal and Nichol, with a model 25% larger and otherwise differently tuned.

Voicebox replicates the factorization in speech: one model, one dataset, trained three ways (FM with the OT path, FM with the VP diffusion path, score matching with the same diffusion path), evaluated as zero-shot TTS by WER, the word error rate of an ASR model transcribing the synthesized speech. The comparison runs in a reduced setup with a lowered learning rate so that all three objectives converge.

Source: Voicebox, the objective-comparison tables in Section 5.8 (Tables 8 and 9 in the arXiv version). For scale, ground-truth recordings score WER 2.2 on this evaluation.

At a 50K-update budget only the OT path yields a usable model; both diffusion-path variants still fail to produce intelligible speech. At 150K updates the objective swap on the identical path cuts WER roughly in half, and the path swap closes the rest of the gap to ground truth. In the low-NFE column both factors matter, but only the OT model stays usable at four evaluations, and score matching is still degraded at the table's maximum of 32.

Source: Flow Matching for Generative Modeling, Figure 4, p. 7. What to see: on the left, the 2D checkerboard pattern emerges much earlier along the flow for FM with OT (bottom row) than for the diffusion-path models above it; on the right, FM-OT samples remain clean at NFE 4 to 20 while the other objectives' samples degrade.

Sampling: the NFE is chosen at inference time, and the path shifts the quality/NFE trade-off

Any off-the-shelf ODE solver turns a noise draw into a sample by integrating the learned field, and with a fixed-step solver the NFE is proportional to the step count, so the quality/latency operating point is chosen at inference time, per run, with no retraining. Deterministic diffusion samplers share that property; what the path changes is where the trade-off sits. On ImageNet-32 with the same fixed-step solvers, the FM-OT model reaches the same numerical error as the diffusion-path models with roughly 60% of the function evaluations and keeps usable FID at very low NFE, and Voicebox's NFE=4 column above is the speech version of the same gap.

Source: Flow Matching for Generative Modeling, Figure 7, p. 9 (ImageNet-32, fixed-step solvers). What to see: in the left panel the FM-OT model (green) reaches a given numerical error at the fewest evaluations, and in the FID panels FM-OT sits near its converged quality at NFEs where score matching has not yet left the high-FID regime.

Classifier-free guidance transfers unchanged: train with condition dropout, then mix the conditional and unconditional fields at every solver step, which doubles the network evaluations for a fixed NFE. The mechanism, a guidance scale acting as an exponent on an implicit classifier, is covered in Classifier-Free Guidance: The Guidance Scale Is an Exponent on an Implicit Classifier. In audio the objective appears at more than one interface: Voicebox generates whole mel-spectrogram sequences with it, and a growing family generates the latents of a pretrained audio autoencoder, the setting whose encoder-side requirements are covered in Latent Audio Encoders: Reconstruction Sets the Ceiling, Not the Ranking. The objective also appears inside autoencoder training itself: SAME Makes Generation Difficulty an Autoencoder Training Loss trains a small flow-matching model jointly with a music autoencoder and backpropagates its loss into the encoder, shaping the latent space the downstream generator inherits.

Sources

• Lipman et al., Flow Matching for Generative Modeling (arXiv:2210.02747): FM and CFM objectives with Theorems 1-3, Sections 2-4 (objective section); Gaussian path family, VP and OT instances, and Figures 2 and 3, Section 4.1 (path section, both figures reproduced); probability-flow coincidence, Appendix D, and the score/noise-matching loss definitions with their weightings and the sampling drift, Appendix E (path and training-gap sections); Table 1, Figure 4, Figure 10, and the ImageNet-128 training budget, Section 6.1 (ablation section; Figure 4 reproduced as header and body figure, Figure 10 reproduced); the 60% NFE result and Figure 7, Section 6.2 (sampling section, figure reproduced).
• Le et al., Voicebox: Text-Guided Multilingual Universal Speech Generation at Scale (arXiv:2306.15687): flow matching with the OT path and classifier-free guidance for vector fields, Sections 3.1-3.5; the objective and path ablations, Section 5.8, Tables 8 and 9, with the reduced training setup in Appendix B.1 and the full grid in Table B4 (ablation and sampling sections); the ground-truth WER reference point in the English zero-shot TTS results table of Section 5.1.
• Popov et al., Grad-TTS: A Diffusion Probabilistic Model for Text-to-Speech (arXiv:2105.06337): the score-matching loss and its lambda_t weighting with the stated weighting heuristic, Sections 2-3.3; the loss-plateau observation and the small-subset-of-t explanation, Section 4 and Figure 3 (training-gap section).

Related Diffio Posts

• Classifier-Free Guidance: The Guidance Scale Is an Exponent on an Implicit Classifier: the sampling-time guidance mechanism flow-matching systems inherit.
• Latent Audio Encoders: Reconstruction Sets the Ceiling, Not the Ranking: what the latent space must provide when flow matching generates autoencoder latents instead of waveforms or spectrograms.
• SAME Makes Generation Difficulty an Autoencoder Training Loss: the flow-matching objective used as an auxiliary loss inside an audio autoencoder, shaping the latent space its generator trains on.

The finding with the widest scope is in the RL stage: running DAPO with the standard 1 - WER reward moved little over the supervised checkpoint, and the gains appeared after the reward was rebuilt to score how a transcript fails, not how many words it gets wrong. The rest of the pipeline encodes the same idea at the data level.

Severe degradation changes the failure mode, not just the error rate

The paper's organizing observation is a regime boundary. At WER at or below roughly 30%, recognition errors are predominantly word-level confusions: acoustically plausible substitutions a reward can grade locally. Beyond that threshold, the paper reports, the dominant failure switches to sentence-level breakdowns: fluent hallucinations, large omissions, and empty outputs. The model has lost acoustic grounding, and the language model fills the gap with whatever its prior finds likely. The evidence for the boundary is observational (training-time inspection and case studies), and the same 30/50/70 numbers reappear as the pipeline's hyperparameters, so read the specific threshold as the paper's chosen operating point rather than a measured phase transition.

Mega-ASR paper, Figure 6, page 10. The far-field panel shows the failure-mode shift: on the same heavily degraded clip, Qwen3-ASR returns an empty transcript (WER 100%), Gemini-3.1-Pro fabricates fluent content unrelated to the audio (WER 86.1%), and Mega-ASR recovers the reference exactly. Neither baseline failure is a word confusion.

Every pipeline stage is built around this boundary. The dataset discards generated samples above the 70% cap, which the authors observed destabilize training. The first supervised phase orders its curriculum by the same thresholds. The RL split is enriched in medium- and high-WER examples, selected by each example's stored base-model WER. And the RL reward switches its emphasis at a WER gate.

Voices-in-the-Wild-2M: seven effects compose into 54 scenarios under one severity value

Stacking degradations is standard augmentation practice; what the released robust-ASR corpora the paper compares against lack is compositional difficulty, since they mostly cover one or two isolated conditions that the backbone already transcribes at 4-10% WER. The corpus is built as a three-level compositional hierarchy. Eight primitive signal operators (additive noise, echo delay, reverberation, nonlinear distortion, resampling, spectral filtering, loudness transformation, frame-level stutter) compose into seven atomic effects. Each atomic effect is an ordered operator chain with one dominant operator plus the secondary artifacts that physically co-occur with it: far-field is reverb, then low-pass filtering, then a loudness drop, because a distant speaker is simultaneously more reverberant, spectrally attenuated, and quieter; obstructed (speech through a wall, door, or mask) is dominated by repeated low-pass filtering. The clean speech underneath comes from public English and Mandarin corpora (LibriSpeech and Common Voice; WenetSpeech and AISHELL-1), with noise drawn from MUSAN and similar collections.

The atomic effects then compose into 54 compound scenarios under an anchor-modifier rule. Far-field, echo&reverb, and obstructed define mutually exclusive propagation geometries and never combine with each other; noise, recording coloration, electronic distortion, and transmission dropout are modifiers that attach to any geometry. The enumeration is rule-based but not exhaustive: at the three-effect level only 9 of the 18 possible anchor-plus-two-modifier combinations are kept, and the introduction credits an "agentic check" for physical plausibility that the paper never specifies.

Mega-ASR paper, Figure 2, page 4. The seven atomic-effect tiles expand into the 54 hybrid scenarios; the header panel carries the corpus scale (2.4M clips, 11K hours).

Each sample draws a single severity value shared by every active operator; every operator parameter declares which direction is harder and is set from that one value. Independent per-effect sampling would produce incoherent samples (severe reverberation over an otherwise pristine recording); the shared value makes a hard sample hard everywhere. Linear severity sampling beat hard-biased, easy-biased, and middle-biased alternatives in the paper's 50K-sample pilot runs. The per-effect parameters were tuned against real recordings via SFT validation, though the paper names neither the calibration set nor the fit objective, and its own difficulty numbers conflict: the comparison table puts the backbone at 18.42 average WER on the corpus, the text says 35%. Either figure is well above the corpora it compares against.

A2S-SFT: ground the encoder and aligner before the language prior adapts

Recognition under heavy degradation has two coupled bottlenecks: extracting reliable acoustic evidence from a corrupted signal, and using the language model's prior to reconstruct the transcript when that evidence is only partially reliable. The supervised stage trains them in that order, with rank-8 LoRA throughout. Phase one updates only the encoder and aligner, on a curriculum that expands through the three WER thresholds in order. Phase two freezes the acoustic side and adapts the LLM on the full targeted split, no curriculum, activating semantic recovery. Phase three updates everything jointly, with the acoustic side on a lower learning rate to preserve phase one. The schedule itself is the standard align-then-joint recipe for multimodal LLMs; the additions are the phase-one curriculum and a measurement of how much the staging is worth.

The order matters for the same reason the high-WER failures look the way they do. In the authors' preliminary variants, joint training from the start was unstable on medium- and high-WER samples because the LLM adapted to acoustic representations that were still unreliable, and adapting the LLM before the acoustic warm start made the model more prone to relying on its language prior. A prior that learns to override a noisy encoder during training is the same mechanism that produces fluent off-audio transcripts at inference. In the ablation table below, unstaged SFT on the same data recovers only about half of the staged schedule's improvement.

DG-WGPO: a reward built to keep ranking transcripts after the WER reward saturates

The RL stage samples a group of transcript candidates per audio clip and updates with GRPO-style group-relative advantages under the DAPO loss: each candidate's advantage is its reward measured against the group's, so the learning signal lives entirely in within-group reward differences. Our GRPO/DAPO post covers the general mechanism and DAPO's changes to GRPO. A reward that stops discriminating between candidates therefore stops producing gradient, and the paper reports exactly that failure for the plain WER reward: under heavy degradation it saturates, collapsing reward dispersion within the rollout group precisely where the policy most needs signal. Sequence-discriminative training against expected word error (MWER, sMBR) is decades-old ASR practice and works in its regime; what breaks here is the group-relative estimator.

Mega-ASR paper, Figure 4, page 5. Left: the three A2S-SFT phases. Center: the policy's rollout group, scored by a static rule-based reward and by the dual-granularity dynamic reward whose two terms a WER gate mixes. Right: the combined rewards become group-relative advantages.

The redesigned reward keeps R_static = R_rep * (1 - WER) as an anchor, where the repetition gate R_rep zeroes any rollout that collapses into repeated n-grams. On top of it sits a dynamic reward with two terms. The token-level term R_fine splits substitutions by character-level edit similarity: a near-miss with similarity at or above 0.5 is discounted to 0.4 of an error, while insertions and deletions always count fully, since unlike a near-miss substitution neither is an acoustic confusion. The sentence-level term R_struc ignores token identity and scores structural preservation of the reference: half from longest-common-subsequence recall, half from a length-integrity term that reaches zero when the hypothesis is empty or runs away. R_struc is what separates an empty output from a truncated-but-faithful one from a fluent fabrication, distinctions that all live in the saturated tail of 1 - WER.

A WER gate, evaluated on each rollout's own WER against the reference, swaps the two weights at a threshold: below it the token-level term dominates the mix, above it the sentence-level term does, and the final reward weights the dynamic pair over the static anchor. The gate threshold is 0.3 in the main text and its sweep but 0.5 in the appendix and in the framework figure above, and the two halves of the paper also disagree on the RL learning rate, the rollout count (16 versus 12), and the SFT learning-rate scheme, so the released numbers pin down the design but not the exact operating point.

The ablation: reward design, not the optimizer, carried the RL stage

The table below is measured on the medium- and high-difficulty subsets of VOiCES (a far-field room-microphone corpus) and NOIZEUS (noisy speech at controlled signal-to-noise ratios).

Mega-ASR paper, Table 5, page 8. WER (%), lower is better, on the VOiCES and NOIZEUS medium- and high-difficulty splits. Single runs, no variance reported; elsewhere the paper lists the full system at 7.53 on VOiCES for what is nominally this same configuration, so treat differences of a tenth of a point as direction, not magnitude.

First, the supervised rows dominate: the corpus alone (unstaged SFT) and the staging contribute roughly equally, and together they account for around four fifths of the total improvement. Second, WER-only RL contributed little: vanilla GRPO lands slightly worse than no RL on VOiCES, and vanilla DAPO leaves VOiCES essentially unchanged while capturing about a quarter of the eventual NOIZEUS gain. Every variant of the redesigned reward moves both benchmarks where WER-only RL moved only one, and the optimizer choice (DAPO over GRPO) is worth less than the reward redesign. Third, among the reward components, removing the sentence-level term costs the most, consistent with the regime boundary. The gating mechanism contributes little: removing the gated fusion costs almost nothing, and the gate-threshold sweep (reported on NOIZEUS only) is nearly flat end to end.

The reward also did not need a model in the loop. Replacing the dynamic reward with a Gemini-2.5-flash-lite judge score changed WER by about 0.1 in both directions across the test sets while roughly tripling the time per RL step. Where the task has a reference transcript, a verifiable rule-based reward already captures what the judge provides.

The router: degradation is cheap to detect, so the robust weights apply as a LoRA delta

Always-on robust weights cost accuracy on clean input.

Mega-ASR paper, Table 3, page 7. WER (%), lower is better. LibriSpeech is read clean English speech; FLEURS is a multilingual read-speech benchmark.

The fix exploits an asymmetry: classifying audio as degraded is far easier than transcribing it. A single-layer Transformer over log-Mel features (a standard spectrogram-style featurization) reaches over 99.5% clean/degraded accuracy. Routing then swaps weights by adding or subtracting precomputed LoRA delta tensors instead of reloading the model, and measured end-to-end runtime on CHiME-4 (a noisy-speech benchmark) changes by -0.8%, within noise. The routed system restores the clean numbers above while keeping most of the robustness.

The asymmetry claim comes with a scope limit: the classifier's accuracy is measured against its own training mixture, clean corpora versus the simulator's output, so it may be detecting the simulator's artifacts rather than degradation in general. The behavioral evidence points the same way. On the NOIZEUS full-benchmark average (a different split from the ablation table), always-on weights stay better than routed, 7.52 against 7.90, and on babble noise (overlapping crowd chatter) at 0 dB the routed system is worse than even the unrouted backbone, a pattern consistent with the router misrouting real degraded audio it never saw in training.

Results: gains concentrate where the baseline collapses

Mega-ASR paper, Table 2, page 7. Average WER (%) across the three robustness benchmarks; selected rows of the paper's 13-system table. Qwen3-ASR already led this comparison, so Mega-ASR extends the field's best baseline rather than rescuing a weak one.

The average understates how localized the improvement is. On VOiCES, the far-microphone conditions in the two hardest rooms carry almost all of the gain; the worst subset, room-4 far-microphone babble, falls from 54.01 to 45.69 WER. Close-microphone subsets change little in absolute terms, the easiest NOIZEUS conditions improve mostly in relative terms, and a few moderate CHiME-4 subsets regress. That distribution is what training against the high-WER regime predicts: large recoveries where the baseline loses grounding, small movement where it never did.

On the paper's own Voices-in-the-Wild-Bench, mixed-degradation WER falls from 3.30/5.39 to 2.73/4.57 (real/simulated splits), with Kimi-Audio, the closest external system on that split, about a point and a half behind on both. But 70% of the benchmark comes from the same simulator as the training data, so the simulated columns partly measure distribution fit. An LLM-as-judge evaluation supports the failure-mode story more directly, with the judged hallucination rate falling by about a third and missed content by more than half against the backbone, though the judge protocol is under-specified, so treat those as directional. What remains open is transfer: every result is on Qwen3-ASR-1.7B, and the recipe's value on a backbone with a different acoustic encoder or a stronger language prior is untested.

Sources

• Mega-ASR: Towards In-the-wild² Speech Recognition via Scaling up Real-world Acoustic Simulation (arXiv 2605.19833): all claims and numbers; the regime boundary and reward design (Sec. 4.2), A2S-SFT staging and preliminary variants (Sec. 4.1, Appendix E.1, Table 24), dataset construction (Sec. 3, Appendix C, Table 17), router (Sec. 4.3, Appendix D), ablation (Table 5), reward-versus-judge comparison (Table 6), clean-cost and robustness tables (Tables 2-3, 13), LLM-judge metrics (Table 7), and Figures 2, 4, 6 reproduced above.
• Voices-in-the-Wild-2M dataset (Hugging Face): the released training corpus described in the dataset section (the card lists about 646K examples, a subset of the paper's 2.4M clips).
• Voices-in-the-Wild-Bench (GitHub): the released evaluation benchmark cited in the results section.
• Qwen3-ASR technical report (arXiv 2601.21337): the backbone model fine-tuned throughout.
• DAPO: An Open-Source LLM Reinforcement Learning System at Scale (arXiv 2503.14476): the policy-optimization loss used by DG-WGPO.

Related Diffio posts

• GRPO and DAPO: Within-Group Reward Dispersion Gates the Learning Signal: the group-relative policy optimization mechanism the RL section builds on.

DAPO (Decoupled Clip and Dynamic sAmpling Policy Optimization), from ByteDance Seed and Tsinghua AIR, is the published account of what it takes to run this recipe at scale: four training-detail fixes separate a naive GRPO run on Qwen2.5-32B from a score above DeepSeek's R1 reasoning-RL recipe on competition math.

GRPO replaces the learned critic with the statistics of a rollout group

In RLHF-style PPO, the advantage of each generated token comes from generalized advantage estimation over per-token rewards, using a learned value model, with the KL penalty against the reference policy (the frozen model RL starts from) folded into the per-token reward. DeepSeekMath gives two reasons to drop the value model. It is typically another model of comparable size as the policy, a substantial memory and compute burden (argued, not benchmarked: the paper reports no PPO counterpart at 7B). And the reward model scores only completed responses, so a value function that is accurate at every intermediate token must be learned from terminal-only supervision, a credit-assignment problem GRPO avoids setting up at all.

Source: DeepSeekMath paper, Figure 4, p. 13. What to see: PPO (top) routes each output through reward, value, and GAE blocks to get per-token advantages, training two models (yellow); GRPO (bottom) samples G outputs per question, scores them with the reward model (no value model), and a group-computation block turns the G rewards into the G advantages, training only the policy.

GRPO's substitute is the group. For each question, sample G outputs from the current policy, score each with the reward model, and set the advantage of every token of output i to the group-normalized reward:

A_i = (r_i - mean(r_1..r_G)) / std(r_1..r_G)

Under this outcome supervision the advantage is constant across an output's tokens, so GRPO is a sequence-level update: each whole response is reinforced or penalized in proportion to how far its reward sits from the group mean, measured in group standard deviations. The paper also argues the group baseline matches how reward models are built, since they are trained on comparisons between outputs to the same question, and a within-question relative advantage asks the reward model only the kind of question it was trained to answer.

Two further changes accompany the group baseline. The KL penalty moves out of the per-token reward and into the loss as an explicit regularizer, estimated with the unbiased, guaranteed-positive estimator from Schulman's KL-approximation note (the k3 form), which keeps the advantage a pure function of the rewards. And the PPO clip survives in the objective but does no work in the original paper's runs: DeepSeekMath takes a single gradient update after each sampling stage, so the policy ratio is 1 at update time and the clip never binds (the paper's own appendix derivations drop the clip under exactly this assumption). The clip binds only in the off-policy regime DAPO runs.

Source: DeepSeekMath Section 4.2, p. 15; DAPO Sections 2.3, 2.4, 3 and 4.1, pp. 3-8.

Uniform rewards across a group yield zero gradient from the prompt

The advantage formula has an exact consequence: if all G outputs of a question earn the same reward, every advantage is zero (and the standard deviation in the denominator is zero, so the normalization is undefined). The question contributes no policy gradient beyond the KL term. The learning signal exists only on prompts the policy sometimes solves and sometimes fails at the current sampling temperature; prompts the model has mastered and prompts beyond its reach alike contribute nothing. The exact zero is a discrete-reward phenomenon: under a binary verifier, uniform groups are common, while under DeepSeekMath's continuous reward model exact ties are rare and the failure arrives instead as low dispersion, with the std division amplifying whatever reward noise remains. That is consistent with DeepSeekMath never naming the failure mode and specifying no handling for zero-variance groups, and with DAPO, which trains against a binary rule reward, naming it and building a fix into the objective.

A well-calibrated PPO critic produces a soft version of the same zeroing, since a value function that correctly predicts a saturated prompt's reward also sends advantages toward zero. The group baseline zeroes out exactly, by construction, computed from the same batch; because the zero is exact, zero-dispersion groups are countable in every batch, and DAPO counts them: the fraction of prompts whose rollouts all come back correct climbs throughout training and reaches roughly half of each batch late in the run (Figure 3b, p. 5), exactly when the model is strongest. A critic, by contrast, still emits per-token signal on the saturated groups GRPO discards: the critic-based VAPO recipe later reported 60.4 on the same Qwen-32B AIME setup, more than ten points above DAPO, so dropping the value model trades training signal for simplicity, and the comparison is not settled.

The same coupling constrains reward design: a reward whose metric saturates in the regime where the model operates yields no policy gradient even when it is perfectly correct on average. The two classes of fix follow from the two sides of the dependence: resample prompts until every group disperses (DAPO's dynamic sampling, below), or reshape the reward so dispersion survives where the task needs gradient, the route Mega-ASR's DG-WGPO reward takes for severely degraded speech.

Dividing by the group's standard deviation has a second effect: each prompt's update is measured in units of its own dispersion, so a nearly-uniform group that is not exactly uniform gets amplified advantages. Dr. GRPO later named this a question-level difficulty bias and removed the std term from the advantage entirely.

DeepSeekMath's RL gains are reweighting: Maj@K rises, Pass@K does not

On DeepSeekMath-Instruct 7B, GRPO over roughly 144K math prompts lifts GSM8K from 82.9 to 88.2 and MATH from 46.8 to 51.7 (Table 5, p. 12), with gains on out-of-domain benchmarks that contributed no RL data. The interpretive result is Figure 7: RL raises Maj@K (accuracy of the majority answer over K samples) across K up to 64, but leaves Pass@K (whether any of K samples is correct) essentially unchanged on both benchmarks. The paper's own reading is that the improvement comes from "boosting the correct response from TopK rather than the enhancement of fundamental capabilities", that is, from reweighting probability mass toward answers the SFT model could already produce. The result is scoped by its setup, which the authors flag themselves: RL prompts came from the instruction-tuning distribution and exploration used naive nucleus sampling. It does not bound RL from a base model: DAPO starts from Qwen2.5-32B scoring near zero on AIME and its pass@32 climbs throughout training alongside avg@32 (DAPO Figure 1, p. 1), with reflection and backtracking behaviors its case study reports as absent early in training.

Source: DeepSeekMath paper, Figure 7, p. 21. What to see: the Maj@K curves separate (orange RL above purple Instruct) on both benchmarks, while the Pass@K curves (blue RL, green Instruct) track each other, with RL slightly below Instruct at large K on both benchmarks.

The paper's Section 5.2 makes the reweighting view systematic by writing SFT, rejection-sampling fine-tuning (RFT), DPO (preference optimization on chosen/rejected response pairs), PPO, and GRPO as one gradient expression that differs only in where the data comes from and what scalar coefficient multiplies each token's log-probability gradient: 1 for SFT, a binary correct/incorrect indicator for RFT, a sigmoid of the preference margin for DPO, the critic's advantage for PPO, the group z-score plus a per-token pull toward the reference model for GRPO. Its 1.3B-scale comparison then isolates two axes: sampling from the current policy beats reusing stale SFT-model samples once the policy has drifted (the RL form of training on self-generated contexts, the distribution-shift mechanism covered in Exposure Bias: Corrupt the Context, Keep the Target Correct), and reward-magnitude-aware coefficients beat binary ones. GRPO sits at the online, graded corner of that grid.

DAPO: a rule-based reward, no KL term, and four fixes between 30 and 50

DAPO's starting claim is that R1-style results failed to reproduce not for lack of compute but for lack of training details omitted from the published reports: its naive GRPO baseline on Qwen2.5-32B scores 30 on AIME 2024 (a 30-problem competition-math benchmark scored as avg@32, mean accuracy in percent over 32 samplings of the test set) against DeepSeek's reported 47, with entropy collapse, reward noise, and training instability as the named failure modes. The full system reaches 50 using half the training steps of DeepSeek-R1-Zero-Qwen-32B; all DAPO evidence is this one benchmark on this one base model, without seed variance. The reward is deliberately minimal, +1 for a final answer equivalent to ground truth and -1 otherwise, over a 17K-prompt math set rewritten so every answer is an integer, which makes the rule verifiable without a formula parser.

DAPO also deletes GRPO's KL term entirely. The KL penalty exists in RLHF to keep an aligned model near its initial policy; a long chain-of-thought reasoning model's distribution necessarily diverges far from the base model during training, so the paper removes the penalty rather than penalize an inevitable divergence. The deletion is also cheaper here than in RLHF generally: the KL penalty doubles as protection against over-optimizing a learned reward model, and a rule-based verifier leaves no reward model to over-optimize.

Source: DAPO Table 1, p. 9. The rows are cumulative in this order, not independent ablations, so each delta is conditional on everything above it; the table supports the mechanism analysis, not a performance ranking.

Clip-higher: a symmetric clip zeroes gradient on exactly the exploration tokens

The first failure is entropy collapse: generation entropy decays toward zero early in training, the G rollouts of each prompt become near-identical, and exploration stops. DAPO's diagnosis is that PPO's symmetric clip is asymmetric in probability space. Clipping zeroes the gradient of tokens whose policy ratio leaves the trust region rather than bounding the ratio itself, and the threshold where gradient stops is multiplicative in the old probability. With eps = 0.2 and a positive advantage, a 0.01-probability token stops receiving gradient once it reaches 0.012, while a 0.9-probability token keeps its gradient up to a nominal 1.08, a bound above probability 1 that can never bind. The upper clip therefore zeroes the gradient of exactly the rare tokens exploration depends on and never touches confident ones. The measurement matches: the mean probability of up-clipped tokens settles below 0.2 after an early transient (Figure 3a, p. 5).

The fix decouples the two clip bounds, raising the upper bound to 0.28 while keeping the lower at 0.2, because raising the lower bound would let negative-advantage updates push token probabilities to zero and collapse the sampling space from the other side. With clip-higher, generation entropy stabilizes around 0.4 instead of decaying to zero, and accuracy climbs past the baseline's plateau.

Source: DAPO paper, Figure 2, p. 3. What to see: on the right, the baseline's entropy (cyan) decays to near zero by step 1,000 while the clip-higher run (purple) holds around 0.4; on the left, the baseline's accuracy plateaus around 0.25 while the clip-higher run keeps improving.

The clip binds at all only because DAPO takes 16 gradient updates per rollout batch, so by the later minibatches the policy has moved off the rollout distribution and the clip binds throughout training. Clipping, and therefore clip-higher, is the price of reusing rollouts; how hard the clip works is set by how off-policy the update schedule is.

Dynamic sampling: refill the batch until every group has dispersion

The second failure is the fraction of prompts whose groups have zero reward dispersion and therefore zero advantage. DAPO's fix is a constraint written directly into the objective: the number of correct responses in each group must be strictly between 0 and G. Operationally, over-sample prompts, discard groups that come back all-correct or all-wrong, and keep sampling until the batch is full of prompts with non-degenerate dispersion. Every prompt in every batch then carries gradient. It is the largest single step in Table 1's cumulative ablation, and the dynamic-sampling run reaches the unfiltered baseline's best accuracy in roughly a third of the training steps (Figure 6, p. 8).

Source: DAPO paper, Figure 3, p. 5. What to see: the right panel's all-correct prompt fraction climbing through training, the zero-dispersion share dynamic sampling removes; the left panel's up-clipped tokens sitting at low probabilities, the evidence behind clip-higher.

The filtering is not free: discarded groups cost generation. The paper argues the wall-clock impact is limited because synchronized rollout time is dominated by the longest samples anyway, but Figure 6 counts steps, not GPU-hours, and reports no sampling-overhead measurement, so the efficiency claim is about gradient quality per step, not compute.

Token-level loss: per-response averaging shrinks per-token gradient as responses grow

GRPO's loss averages per token within each response, then across responses, giving every response equal weight regardless of length. Each token of a long response therefore contributes proportionally less gradient than each token of a short one. In long chain-of-thought training this misweights both directions at once: the reasoning patterns inside high-quality long responses are under-taught, and the gibberish and repetition that low-quality long responses accumulate are under-penalized. DAPO observes the consequence as an unhealthy climb in both entropy and mean response length, with length spiking near 5,000 tokens before swinging erratically (Figure 4, p. 6).

The fix normalizes the summed per-token loss by the total token count across the group, so a generation pattern is promoted or suppressed by the same amount regardless of the length of the response it appears in. In the cumulative ablation it adds a single point, within plausible single-seed noise on a 30-problem benchmark, and the paper credits it with training stability and healthier length growth rather than score. Token-count normalization has its own bias, since a single very long response can dominate its group's update: Dr. GRPO instead removes the per-response length normalization, tracing it to length inflation in incorrect responses, and GSPO later moved the importance ratio and the clip to the sequence level entirely, reporting more stable training, notably for mixture-of-experts models.

Overlong reward shaping: truncation penalties are reward noise

Generation runs under a hard length cap, and the default treatment punishes truncated samples. That punishment is reward noise: a sound reasoning process gets penalized solely for its length, which, in the paper's words, can confuse the model about the validity of its own reasoning. Simply masking truncated samples out of the loss (Overlong Filtering) is the first row of the cumulative ablation and its second-largest delta, and removing it produces an entropy explosion late in training (Figure 5, p. 7). The shipped version replaces the step penalty with a graded one: reward is untouched up to the soft cap, then decreases linearly to -1 across the penalty band before the hard cap (lengths in the operating-point table). The penalty is then a smooth function of length rather than a truncation event correlated with reasoning depth (the cumulative ablation applies the graded penalty on top of the filtering row; whether the mask is kept in the final recipe the paper does not say).

Sources

• DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Models (arXiv:2402.03300): GRPO objective, advantage normalization, and critic-removal rationale, Section 4.1, pp. 11-14 (GRPO section, including Figure 4, p. 13, reproduced as the header and first body figure); RL hyperparameters and single-update schedule, Section 4.2, p. 15, with the clip-dropping simplification in Appendix A.1.5-A.1.6 (operating-point table and clip-never-binds claim); GSM8K/MATH results, Table 5, p. 12; Maj@K/Pass@K finding and quote, Section 5.2.2 and Figure 7, p. 21 (reweighting section, including the reproduced figure); unified gradient-coefficient paradigm, Section 5.2.1, Eq. 5 and Table 10, pp. 18-19, and Appendix A.1.
• DAPO: An Open-Source LLM Reinforcement Learning System at Scale (arXiv:2503.14476): naive-GRPO baseline, headline result, and pass@32 curve, Section 1 and Figure 1, pp. 1-2; KL removal, Section 2.3; rule-based reward and dataset, Sections 2.4 and 3.5; clip-higher mechanism and evidence, Section 3.1, Figures 2 and 3a, pp. 3-5 (both reproduced); dynamic sampling, Section 3.2, Figures 3b and 6 (Figure 3 reproduced); token-level loss, Section 3.3, Figure 4; overlong shaping, Section 3.4, Figure 5; hyperparameters, Section 4.1, p. 8; cumulative ablation, Table 1, p. 9; emergent reflection case study, Section 4.4.
• Schulman, 2020, Approximating KL Divergence: the k3 estimator GRPO uses for its loss-side KL term (GRPO section).
• Liu et al., 2025, Understanding R1-Zero-Like Training: A Critical Perspective / Dr. GRPO (arXiv:2503.20783): removal of the std normalization (question-level difficulty bias) and the per-response length normalization (dispersion and token-level loss sections).
• Yue et al., 2025, VAPO: Efficient and Reliable Reinforcement Learning for Advanced Reasoning Tasks (arXiv:2504.05118): value-based 60.4 AIME 2024 result on Qwen-32B under settings matching DAPO (dispersion section's critic trade-off).
• Zheng et al., 2025, Group Sequence Policy Optimization (arXiv:2507.18071): sequence-level importance ratios and clipping, MoE stability claim (token-level loss section).

Related Diffio Posts

• Exposure Bias: Corrupt the Context, Keep the Target Correct: the train/inference context-distribution mismatch behind the online-beats-offline finding in the unified-paradigm comparison.
• Mega-ASR: Group-Relative RL Needs a Reward That Ranks Failed Transcripts: the reward-reshaping route in practice, where a saturating WER reward is rebuilt to rank sentence-level failures.

The MOSS comparison is internal: the report evaluates an 8B delay-pattern generator and a 1.7B Local Transformer under the same tokenizer and pretraining recipe. The comparison is not parameter matched, so it does not isolate causal RVQ depth from every local-module difference. It shows that in this released comparison, the smaller local architecture outperformed the larger delay backbone on the reported cloning metrics.

Low frame rate shifts modeling cost to RVQ depth

MOSS-Audio-Tokenizer makes long context possible by using a low frame rate and multiple RVQ layers per frame. It encodes 24 kHz audio at 12.5 frames per second.

Each frame contains 32 residual vector quantization (RVQ) layers. RVQ means the first codebook quantizes a coarse approximation, the next codebook quantizes what the first missed, and later codebooks keep refining the residual.

The report's final pretraining phase measures context in backbone time steps rather than scalar codec decisions. At 12.5 frame steps per second, its 64k-step training window covers about 85 minutes of audio before text, prompt tokens, and the delay model's 31-step edge overhead. The remaining modeling problem is the 32 ordered decisions attached to every frame.

The tokenizer can trade bitrate for residual depth because random quantizer dropout trains the decoder after randomly dropping later RVQ layers, so prefixes of the 32-layer stack remain valid codes. At this frame rate, each additional RVQ layer adds 125 bps.

MOSS-TTS Technical Report Table 2. SIM is cosine-scale speaker-embedding similarity; PESQ-WB is a wideband perceptual speech-quality score. These are tokenizer-oracle reconstruction rows, not a generator-factorization result.

The delay pattern preserves sequence length by interleaving RVQ layers

The delay pattern keeps the language-model sequence close to the audio frame count. Instead of flattening the RVQ stack into separate audio positions, it shifts each residual layer by its layer index. A T-frame utterance becomes T + 31 delayed backbone steps.

Each aligned step has one text/pad channel and one channel per RVQ layer. Channel 0 emits a normal text token on text-only steps and a dedicated pad symbol on audio-frame steps; the audio channels emit RVQ codes. Invalid delayed or padding positions are masked from the loss. In the delay model, one backbone hidden state feeds all channel heads.

This factorization is efficient for serving. The backbone is evaluated once per delayed step, and the output heads map the hidden state to token logits. This is why the report treats the delay architecture as the simpler route for long contexts, duration control, pronunciation control, and optimized inference backends.

The cost is that within-frame RVQ dependencies are represented across multiple delayed steps. Lower layers of a frame are emitted at earlier delayed steps than higher layers. By the time a higher layer is predicted, lower layers can influence it only through intervening input vectors, each built by adding codebook-specific embeddings from whatever delayed layers are present at that step. At any one delayed step, the audio heads are predicting channels from different original frames. The backbone is therefore not just modeling speech over time; it is also tracking incomplete RVQ code stacks across delayed positions.

The Local Transformer factorizes each frame autoregressively over RVQ layers

The Local Transformer keeps the frame aligned in time and adds a local autoregressive loop over channels. The temporal backbone produces one latent for the frame, then a lightweight local module predicts the text/pad channel and the RVQ layers in order. The text/pad channel keeps that stream in the same factorization as the audio layers. For audio frames it is the pad decision, followed by RVQ layers that condition on earlier layers from the same frame and on previous aligned steps.

That is the inductive bias the MOSS comparison supports for this residual stack. Later RVQ layers are not exchangeable labels; they are refinements of a residual left by earlier layers. Giving layer j the sampled lower-depth codes as direct context makes within-frame RVQ conditioning explicit. The delay pattern can represent that dependency only after the temporal backbone learns to route it through delayed positions.

Moshi is the immediate lineage for this choice: its RQ-Transformer predicts tokens bottom to top inside an audio step, and MOSS adapts that idea to the tokenizer's full RVQ stack. The tradeoff is steady-state cost. The local model runs an inner loop over channels, while the delay model cannot stream-decode a complete frame until the final RVQ layer arrives about 2.5 seconds later.

MOSS-TTS Technical Report, Figure 2, p. 8. Delay pattern on the left; frame-local autoregressive loop on the right.

The smaller local model wins the within-MOSS cloning comparison

Seed-TTS-eval reports WER for English word error rate, CER for Chinese character error rate, and SIM for speaker-embedding similarity on a percentage scale. This SIM scale differs from the 0-1 reconstruction SIM above.

Seed-TTS-eval, MOSS-TTS Technical Report Table 3. Lower WER/CER (%) is better; higher SIM (%) is better. The report gives no confidence intervals or listening-test variance for this table.

The Local Transformer outperforms the delay model in all four SIM cells, and the highest MOSS similarity value in the table is the local continuation model's Chinese SIM. In Clone mode it also lowers both content-error metrics. In Continuation mode the delay model has slightly lower WER/CER, but both models sit in the low-error regime where the report says manual review found many remaining mismatches to be ASR errors rather than audible pronunciation failures.

CV3-Eval extends the same direction to multilingual cloning, but only for content error. The Local Transformer has lower CER/WER than the delay model in every same-language comparison. Its average absolute improvement over the delay rows is about half a percentage point in both Clone and Continuation mode. That does not establish better speaker preservation outside English and Chinese, because CV3-Eval does not report speaker similarity, but it rules out the narrow explanation that the local model merely trades intelligibility for timbre.

The smaller model outperformed the larger one on the cloning cells the report uses to characterize the local-vs-delay tradeoff, so the evidence supports a scoped design claim: for this tokenizer and release, local conditioning over RVQ depth was the better cloning architecture.

Control and ultra-long results are delay-only

The control sections evaluate only the 8B delay model. Duration control, pronunciation control, and ultra-long generation are not reported for the Local Transformer.

The delay architecture supports the control and long-context experiments because it has a simpler single-backbone parameterization. For duration control, the prompt includes a requested audio-token count. The delay model's absolute relative duration error, the generated-duration miss divided by the requested duration, is 0.712% in Chinese and 0.723% in English, using paired pretraining examples with and without explicit token-count prompts rather than a separate duration-control fine-tune.

The ultra-long internal set separates content from speaker drift. In the English 50000+ input-character bucket, Clone WER reaches 17.49% and Continuation WER reaches 29.52%, while speaker similarity falls to 44.4% for Clone and 51.2% for Continuation.

MOSS-TTS Technical Report, Figure 6, p. 21. The English longest-bucket curves separate early and remain low, showing that long-horizon degradation appears as cumulative speaker drift before every sample reaches a lexical failure.

Sources

• MOSS-TTS Technical Report: tokenizer geometry, Figure 2 architecture diagram, delay and Local Transformer objectives, Seed-TTS-eval Table 3, CV3-Eval Table 4, duration Table 5, ultra-long Table 6.
• MOSS-Audio-Tokenizer: CAT tokenizer motivation, end-to-end Transformer tokenizer context, and the tokenizer's role as a scalable audio-modeling interface.
• MusicGen: Simple and Controllable Music Generation: delay/interleaving-pattern lineage for multi-codebook audio-token modeling.
• Moshi: a speech-text foundation model for real-time dialogue: RQ-Transformer lineage and frame-local bottom-to-top token modeling.

Related Diffio posts

• TTS Models: A Map Of The Modern Speech Stack
• VibeVoice: Frame Rate Is the Context Budget for Long-Form TTS

VibeVoice is Microsoft Research's open-weights system for zero-shot, multi-speaker, long-form speech synthesis: give it a short voice sample per speaker and a script, and a Qwen2.5 backbone generates podcast-style conversation, turn-taking and breaths included, in one streaming left-to-right pass. In our map of the modern TTS stack it sits in the continuous-latent autoregressive family. The abstract claims up to 90 minutes of audio with up to 4 speakers; the quantitative evaluation stops at 30 minutes, and the paper never reconciles the two figures. Podcast generation itself is table stakes (NotebookLM, MoonCast, and FireredTTS2 all ship it).

The load-bearing choice is the speech tokenizer's frame rate: 7.5 frames per second, a 3200x downsampling of the 24 kHz waveform, far below the discrete codecs in the paper's comparison tables (40 to 600 tokens per second). That one rate is what fits a 90-minute conversation into a language model's context window. Everything distinctive downstream exists to make the rate survivable: the latent widens to 64 continuous dimensions, a diffusion head replaces the softmax, and semantic content gets its own encoder.

A 7.5 Hz tokenizer fits 90 minutes in context; the cost lands in channel width

The acoustic tokenizer emits one 64-dimensional continuous latent per 133 ms of audio, so 90 minutes of speech is 40,500 positions: inside the 65,536-token training context (the final phase of a length curriculum that starts at 4,096) with room left for the script. Long-form generation is then dense attention over one flat sequence, with no chunking, sliding windows, or hierarchical outline model.

What compression removes from the time axis has to reappear in the channel dimension if reconstruction quality is to hold, so the latent must widen as the rate falls. LatentLM, the paper VibeVoice takes its tokenizer recipe from, measures the conversion directly, and the compensation has a ceiling: its deepest compression setting degrades despite a wider latent. VibeVoice's 7.5 Hz / 64-dim operating point is the same one LatentLM ships for speech. SAME makes the same trade for music at 4096x, and our latent audio encoder post maps the general design space: rate, width, and latent geometry jointly define the prediction problem the downstream generator faces.

A 64-dim continuous channel is also what retires the softmax. A softmax head picks one codebook entry per step, and no practical codebook covers the information that survives 3200x compression. The discrete route shards each frame across a stack of residual vector quantization (RVQ) codebooks, each quantizing what the previous ones missed: MOSS-TTS, the strongest discrete system at comparable ambition, spends 32 codebooks on every 12.5 Hz frame and pays for them with an inner per-frame decoding loop. VibeVoice samples the whole 64-dim vector jointly through a diffusion head, paying an inner loop of denoising steps instead. A continuous head does not have to be a diffusion head (MELLE regresses mel frames with a variance-augmented head), but denoising from noise samples the full conditional distribution where a regression head commits to a parametric family.

What 7.5 Hz costs in reconstruction:

LibriTTS test-clean, from the paper's Table 3. PESQ is an intrusive perceptual speech-quality score with a 4.5 ceiling, STOI a 0–1 intelligibility score computed against the reference, UTMOS a learned 1–5 predictor of human quality ratings.

The continuous tokenizer leads PESQ and UTMOS at one tenth of WavTokenizer's token rate, but token rate is a sequence-length unit, not an information unit: at 16-bit floats the 64-dim latent stream is roughly 7.7 kbps, above every discrete row in the table. The win is positions in the LM's context, not bitrate efficiency.

The weak column is STOI, which the authors attribute to training on podcast audio that deliberately skipped noise reduction. LibriTTS test-clean is clean, so the mismatch runs through the decoder's learned prior: at 3200x the decoder synthesizes fine detail rather than copying it, and detail synthesized under a noisy-podcast prior diverges from the clean reference at exactly the short-time envelope level STOI compares.

The sigma-VAE tokenizer sets latent variance for the generator that consumes it

The tokenizer is a mirror-symmetric encoder-decoder pair, roughly 340M parameters each, built from seven stages of Transformer blocks whose self-attention is replaced by 1D depth-wise causal convolutions, so encoding and decoding stream; training follows DAC's adversarial recipe. The paper calls it a tokenizer, but it is a VAE variant from LatentLM, the same paper that supplies the generation recipe: the encoder predicts only a mean, and the latent is

z = mu + sigma * epsilon,   epsilon ~ N(0,1),   sigma ~ N(0, C_sigma)

with sigma a scalar shared across all 64 channels, drawn fresh per example; VibeVoice sets C_sigma = 0.5.

Taking variance away from the encoder makes the latent space's noise floor a design parameter instead of a training outcome. A vanilla VAE under reconstruction pressure lets per-channel variances collapse toward zero, and once they collapse, every property the generator depends on, the scale of the latents, the spacing between codes, the smoothness of the distribution the diffusion head must learn to sample, is whatever reconstruction happened to leave behind. Fixing sigma pins those properties: information must be carried by mean displacements large enough to clear a noise band of known, channel-uniform width. LatentLM tuned the knob against the consumer rather than the reconstruction: across a sweep of tokenizer variances, generation quality for a next-token-diffusion model tracks the variance while an image-level diffusion model over the same latents is indifferent to it. The band is also the system's tolerance to its own sampling error: every generated latent re-enters the context, and an error smaller than the trained band lands inside the distribution the decoder and the LM already know. Exposure bias, and how variance shaping buys margin against it, gets a full treatment in its own post.

A diffusion head replaces the softmax; the backbone stays a standard autoregressive decoder

The prompt sequence concatenates, per speaker, an identifier and that speaker's voice-prompt latents (encoded by the acoustic encoder only; the script never includes the prompt's transcript), followed by speaker-tagged script text and a start-of-speech token. From there the backbone runs the standard autoregressive decoding loop with one substitution. At each step its hidden state conditions a diffusion head, 4 layers and roughly 123M parameters in the 1.5B configuration, in the style MAR introduced for image generation: trained with the standard DDPM noise-prediction L2 objective, it denoises Gaussian noise into the next acoustic latent while the backbone separately predicts whether to emit a termination token. The paper calls each step's output a speech segment without stating whether that is one 7.5 Hz frame or a packed group of frames, the report's largest reproduction gap.

VibeVoice paper, Figure 1, p. 2. Follow the right-hand loop: the diffusion head (D) emits an acoustic latent, the acoustic decoder (A) renders audio, and the semantic encoder (S) re-encodes that audio and adds it back into the next step's input.

At inference the head applies classifier-free guidance, extrapolating away from an unconditional prediction (derived here from the start-of-speech token's hidden state) to sharpen conditioning, at scale 1.3 with 10 denoising steps under DPM-Solver++, a fast ODE solver for diffusion sampling.

The cost structure keeps the language-model serving contract. The backbone runs autoregressively with a KV cache and dominates per-step latency at every profiled setting; the diffusion head's cost scales linearly with its step count, and the feedback path described next adds an acoustic decode and a semantic re-encode inside the loop at around 15 ms each. With everything included, the real-time factor (synthesis time over audio duration; below 1.0 is faster than real time) stays between 0.62 and 0.97 on a single A6000 across both model sizes, generating strictly left-to-right.

Semantic content comes from a second frozen encoder and feeds back as input, never as a generation target

The second tokenizer mirrors the acoustic encoder's architecture without the VAE machinery and is trained on an automatic speech recognition (ASR) proxy task: Transformer decoder layers learn to predict transcripts from its output, then the decoder is thrown away and the encoder is frozen. VibeVoice never generates semantic features. Instead, after the model produces a speech segment and decodes it to waveform, the history entry for that step combines two views of the same audio:

z_p,i = W_a * z_a,i + W_s * SemanticEnc(y_i)

where z_a is the generated acoustic latent, y is the rendered waveform, and z_p is the hybrid entry that actually enters the LLM's context, built from learned projections of both views.

The semantic entry is a deterministic function of the latent just produced, so it adds no new information to the context. What it adds is a representation: the phonetic content of that latent made explicit, where the acoustic latent buries it in texture optimized for reconstruction. The feedback closes a loop in which the LLM hears the phonetic content of its own output, so when generated audio drifts from the script, the drift is visible in the context rather than something the backbone must learn to extract. The loop is only closed at inference; during training the feedback path reads ground-truth audio, so any corrective use of it is learned implicitly.

That speech generation wants two representations is established lineage: AudioLM generated semantic tokens before acoustic ones, and SpeechTokenizer and Mimi distill semantic content into the first RVQ level of a discrete codec. What VibeVoice's ablation localizes is what goes wrong at a single continuous bottleneck, tested at matched model size and context (1.5B, 16K):

Short-set podcast generation (the benchmark's under-12-minute subset), paper's Table 5; the overall columns pool one to four speakers. WER-W is the word error rate of a Whisper transcription of the generated audio against the input script; because the training data was itself filtered by Whisper agreement, the paper cross-checks with a second ASR (WER-N), which preserves the ranking. SIM-O is the cosine similarity between speaker embeddings of the generated audio and the original reference recording, dominated by timbre, the spectral character that makes a voice recognizable.

The acoustic-only row says reconstruction-grade latents preserve timbre (SIM-O matches the best setting) but lose content anchoring exactly when dialogue structure appears: single-speaker error is excellent, and at three speakers it explodes by an order of magnitude.

The coupled row tests the obvious repair, one encoder trained with reconstruction and transcript losses on a single latent, and shows the two objectives competing for fixed capacity. Content stabilizes where dialogue had broken it, but single-speaker error gets worse and the voice goes: overall SIM-O collapses to 0.45, the lowest of all settings in the paper, with the coupled tokenizer's raw reconstruction falling alongside it (PESQ 1.92 against the dedicated acoustic encoder's 2.98 on LibriSpeech test-clean). A 64-dim latent at 3200x has no slack for both objectives, and hard transcript-decoding pressure displaces exactly the texture that makes a voice identifiable. The shipped fix is more capacity, split by objective: a second full encoder, so reconstruction and transcript pressure never compete in one bottleneck.

The dilemma is narrower than the table makes it look. The hybrid pays a visible tax (worse single-speaker error and similarity than acoustic-only), and it binds only because VibeVoice keeps acoustic latents in the LM's context at all: cascades in the CosyVoice style dodge it by generating semantic tokens alone and restoring timbre in a reference-conditioned acoustic decoder, at the price of a second generation stage. SemaVoice serves both objectives with a single latent by softening the pressure, a cosine alignment to frozen WavLM features (a self-supervised speech encoder) at 15 Hz rather than transcript decoding at 7.5 Hz. And the ablation never tests other vehicles for semantic pressure, feeding back ASR text among them, so a separate frozen encoder is the tested fix, not the proven requirement.

Enhancement-free training data and few denoising steps protect the same signal: recording texture

The training corpus is roughly 80B tokens of internal pseudo-labeled podcast audio (the paper does not say how the count splits between text and audio frames), labeled by a pipeline of standard parts: voice-activity segmentation, Whisper transcription, diarization (assigning each segment to a speaker) via WeSpeaker embeddings and HDBSCAN clustering, then filtering by cross-ASR disagreement. The pipeline's one deliberate omission is speech enhancement: denoising removes interjections and prosodic cues along with the noise, so the tokenizer and the LM train on audio with its noise floor, reverb, and mouth sounds intact.

The same trade reappears at inference. The paper's Appendix J frames the diffusion head as a progressive denoiser: more solver steps refine detail but bias the output toward an idealized clean-speech distribution. The step/guidance sweep shows the two metrics pulling in opposite directions: word error is best near 10 steps, while SIM-O is highest at 5 steps and declines monotonically as steps increase.

VibeVoice paper, Figure 3, p. 8. The global WER minimum (left) sits at 10 steps, while SIM-O (right) is highest in the 5-step column at every guidance scale and falls as steps grow.

The Appendix J spectrograms show what the extra steps remove: at 50 steps the model strips the prompt's environmental texture (room reverb, noise floor, microphone character), producing audio that is technically cleaner but spectrally mismatched from the reference the speaker embedding is computed on. A speaker's measured identity includes their recording environment, and over-denoising deletes evidence the metric expects to find. How much of that loss is perceptual rather than metric artifact is open: SIM-O is timbre-centric, and in an earlier round of the paper's human evaluation CosyVoice2 outscored VibeVoice-1.5B on SIM-O (0.68 vs 0.55) while human raters judged VibeVoice more similar to the prompt.

Results on VIBEVOICE-Eval and a 24-listener study: the 7B model leads, on an internal benchmark

Both evaluations are internal, so the podcast results locate the system rather than certify it: VIBEVOICE-Eval is 108 podcast samples (1 to 30 minutes, built with the same annotation pipeline from sources held out of training), and the listener study covers eight LLM-written two-speaker dialogues averaging 7.3 minutes, rated by 24 annotators.

Paper's Table 1; the subjective average pools realism, richness, and preference scores from the listener study. The paper reports no SIM-O for Gemini.

Scale buys consistency: the 7B leads every subjective dimension and posts the best speaker similarity, while the 1.5B keeps the lowest word error. On the long subset (12 to 30 minutes) the 7B holds WER-W 1.24 with SIM-O 0.75, where MoonCast, limited to two speakers and about ten minutes, crashes on long and 3-plus-speaker cases. Nothing past 30 minutes is measured, and the 7B never trained at the 65,536-token context phase (dropped for resource limits), so the 90-minute figure rests on the 1.5B's final training phase and the context arithmetic. Short-utterance generalization transfers: on the SEED benchmark's Chinese set the 1.5B stays competitive (character error rate 1.16) against systems running 25 to 50 Hz frame rates.

Sources

• VibeVoice: Expressive Podcast Generation with Next-Token Diffusion (OpenReview, ICLR 2026): architecture and training setup (Sections 2–3, Appendix F), reconstruction comparison (Table 3, Table 7), tokenizer ablations (Table 5, Appendices B and D), inference sweep and spectrograms (Figure 3, Appendix J), evaluation (Tables 1–2, Appendices G–I), inference cost (Table 8, Appendix E). Figures 1 and 3 reproduced from this paper. The shorter arXiv technical report (2508.19205) omits the appendices and ablation tables cited here.
• microsoft/VibeVoice on GitHub: released code and checkpoints.
• Multimodal Latent Language Modeling with Next-Token Diffusion (LatentLM, arXiv 2412.08635): sigma-VAE construction (Section 2.3), the tokenizer-variance sweep against DiT (Figure 6), compression-vs-width ablation (Table 6) cited in the frame-rate section.
• SemaVoice: Semantic-Aware Continuous Autoregressive Speech Synthesis (arXiv 2605.16964): the single-latent counterpoint cited in the tokenizer-split section.

Related Diffio Posts

• Exposure bias: why models trained on ground-truth history are brittle to their own samples, and how latent variance shaping buys margin.
• Classifier-free guidance: the conditioning-sharpening mechanism the diffusion head uses at inference.
• Latent Audio Encoders: Reconstruction Sets the Ceiling, Not the Ranking: the design space of continuous latents built for downstream generators.
• MOSS-TTS: Modeling The RVQ Stack: the discrete counterpoint, 32 codebooks per frame instead of one wide continuous latent.
• TTS Models: A Map Of The Modern Speech Stack: where the continuous-latent autoregressive family sits.
• SAME Makes Generation Difficulty an Autoencoder Training Loss: the same rate-for-width trade in music autoencoders.

The mitigation literature looks like a grab bag: scheduled sampling, sequence-level reinforcement learning, interactive imitation learning, target relabeling, and, in modern continuous-latent systems, latent variance shaping and noise injection. One distinction organizes most of it. Sound mitigations do one of two things: they move training onto the context distribution inference actually produces while supplying supervision that is still valid there, or they widen the training distribution of contexts until inference-time errors land inside it. Both respect the same correctness criterion: whatever supervision is attached to a self-generated or corrupted context must still be correct for that context. The criterion is necessary rather than sufficient (how much corruption to budget is a separate question), but the most famous mitigation, scheduled sampling, is the one that violates it, and it is the one with a provably improper objective.

Teacher forcing fits every conditional under data prefixes; inference queries model prefixes

Autoregressive maximum likelihood factorizes the sequence probability into per-step conditionals and fits each one at ground-truth prefixes. Teacher forcing is not an approximation bolted onto this objective; it is the objective, and it is what makes training cheap: every position becomes an independent supervised problem, a Transformer scores all positions in one forward pass, and the gradient never depends on sampling.

The mismatch appears at decode time. Each conditional was fit under prefixes drawn from the data distribution but is queried under prefixes drawn from the model itself. If the model matched the data exactly, the two prefix distributions would coincide and there would be no bias; exposure bias is entirely a property of model error. That is also why the teacher-forced training and validation losses cannot register it: both are computed under data prefixes, which measure the per-step error seed but not the compounding. Arora et al. later measured this directly, showing that errors accumulate over model-generated context while perplexity fails to capture the accumulation.

Model error makes the mismatch self-amplifying. The first sampling error produces a context slightly off the training distribution; on off-distribution contexts the model's per-step error rate is higher than anything the loss controlled, which pushes the context further off. Bengio et al. put it directly: a mistake made early "can be quickly amplified because the model might be in a part of the state space it has never seen at training time."

Imitation learning supplies the formal version, because teacher forcing is behavior cloning of the data distribution and exposure bias is its covariate shift. Ross et al. prove that a policy with some per-step error rate under the expert's state distribution can accumulate cost growing quadratically with horizon, while training on the learner's own visited states (with an expert relabeling them) brings the bound back to linear. Quadratic versus linear in sequence length is the worst-case size of the problem.

The realized gap scales with model error: domain shift is where it gets expensive

Because a perfect model has no exposure bias, the bias concentrates wherever per-step error is highest: long generations, weak models, and inputs far from the training distribution. The worst-case quadratic bound is not the typical case. He et al. compare generation under data prefixes against generation under the model's own prefixes and find the induced distortion limited and non-compounding, crediting a self-recovery ability of language models, so in-domain damage for strong models is often modest.

Domain shift is where the gap gets expensive. Wang and Sennrich's translation models hallucinate (produce fluent output detached from the source) on 1 to 2 percent of in-domain inputs but 33 to 35 percent of out-of-domain inputs, and they attribute part of that gap to exposure bias: minimum risk training, a sequence-level objective computed on the model's own samples, cuts the out-of-domain hallucination rate to 26 to 29 percent without consistently improving in-domain scores. The practical readout: short in-domain benchmarks are nearly blind to exposure bias, which surfaces in long-form generation and off-distribution prompts.

Scheduled sampling: model contexts, data targets, improper objective

Scheduled sampling (Bengio et al., 2015) attacks the prefix mismatch directly. During training, each input token is the ground-truth previous token with some probability and the model's own previous prediction otherwise, with the probability annealed from 1 toward 0 on a linear, exponential, or inverse-sigmoid schedule. The sampling decision is treated as a fixed input; no gradient flows through it. It improved image captioning, parsing, and speech benchmarks, and it was used in the winning entry to the 2015 MSCOCO captioning challenge.

The flaw is that only the context was fixed. The target at each position stays the ground-truth token at that position of the original sequence, regardless of what the sampled prefix now says. Ranzato et al. make the misalignment concrete (their example is aimed at data-as-demonstrator, an earlier method with the same target rule, and applies to scheduled sampling unchanged): the ground truth is "I took a long walk", the model samples "walk" where the data had "long", and the next position's target is still "walk", so the model is taught to emit "walk walk". The corrupted context changed what the correct continuation is, and the supervision did not follow.

Huszar formalized the damage: the scheduled sampling objective is improper, meaning even infinite data and capacity do not recover the data distribution. In his analysis of length-2 sequences, the fully annealed objective is globally minimized by the product of per-position marginals rather than the joint, and the optimal sequence model "learns to pay no attention whatsoever to the content of the sequence prefix", using its hidden state as a position counter. The theory matches an empirical detail in the original paper: always sampling from the model, the fully annealed limit, performed very poorly. The method helped anyway because the schedules never reach the degenerate limit in practice: at intermediate mixing rates the procedure widens the prefix distribution around the data (the second mechanism's benefit, obtained without its correctness) while the misaligned-target damage is still diluted.

Training on self-generated contexts: rewards, relabeled targets, matched dynamics

The repair scheduled sampling needed is supervision that remains correct on model-generated prefixes. Sequence-level training buys it by abandoning per-position targets. MIXER (Ranzato et al., 2016) treats generation as reinforcement learning: the model samples its own continuations, and REINFORCE propagates a sequence-level score (BLEU or ROUGE, n-gram overlap metrics against reference outputs) that is well defined for any sampled sequence. Training prefixes now come from the model by construction, and the objective simultaneously fixes the second mismatch Ranzato identified, training on word-level likelihood while evaluating on sequence-level metrics. The price is a sparse, high-variance learning signal: MIXER only works by warm starting with cross-entropy, keeping the first positions of each sequence teacher-forced while REINFORCE trains the tail, and shrinking the teacher-forced span over training. It improved over the cross-entropy baseline on summarization, translation, and captioning, with greedy decoding beating the baseline's beam search on two of the three tasks.

The move does change what is being optimized: reward maximization rather than distribution matching, so the characteristic failure shifts from compounding drift to metric gaming. Modern RLHF-style post-training inherits both properties: the policy trains on its own rollouts, and its failure mode is reward hacking, not exposure bias.

When the task metric admits computing the best continuation from an arbitrary prefix, per-position targets can be rescued instead of abandoned. Optimal Completion Distillation samples prefixes from the model and sets each target to the tokens that begin edit-distance-optimal suffixes, computable by dynamic programming; this is DAgger's move (roll out the learner, relabel its visited states with an expert) with the dynamic program standing in for the expert, and it reached strong speech recognition results on Wall Street Journal without any likelihood pretraining. Open-ended generation has no such computable expert, which is why rewards became the dominant form there. A third classical family, Professor Forcing, skips supervision pairs entirely and adversarially matches the network's hidden-state dynamics between teacher-forced and free-running modes; it targets the diverging state distributions directly rather than constructing valid training pairs.

Continuous latent context turns occasional wrong branches into dense drift

A growing family of generators is autoregressive over continuous vectors rather than discrete tokens: GIVT predicts VAE latents with a Gaussian-mixture head, MAR and LatentLM denoise the next latent with a small diffusion head conditioned on the backbone's hidden state, and TTS systems such as VibeVoice build on the LatentLM recipe (next-token diffusion: the causal Transformer runs once per position, the head iterates only over its own small denoising chain, and the sampled latent feeds back as the next input).

Discreteness was quietly absorbing error all along. Sampling from a softmax snaps the context back onto the lattice of legal token sequences, the way digital signaling regenerates a degraded pulse into a clean symbol: corruption cannot accumulate at the symbol level, only at the distributional level, where legal sequences wander into vanishing-density regions (degenerate repetition loops are exactly such wandering). A sampled continuous vector has no lattice to snap to. It is almost never exactly a point the encoder would have produced, so against a tightly concentrated latent distribution the model is slightly off-distribution at every position, and the design question becomes whether the representation absorbs that error or lets it compound. The contrast is also why this family puts genuine samplers at each step: GIVT's mixture head and MAR's diffusion head both exist, by their authors' statement, because autoregressive generation needs a per-token distribution it can draw samples from. An MSE regression head returns only the conditional mean, which averages modes, and feeding averaged outputs back as context tends to drive generation progressively toward an oversmoothed mean.

Variance shaping: sigma-VAE makes sampling error land inside the trained band

Because the context is now a vector in a space the system designer controls, the second mitigation mechanism becomes available: widen the training distribution itself, an option discrete tokens never had. LatentLM's sigma-VAE is the explicit version. The encoder predicts only a latent mean; the noise scale is removed from its control entirely. A single scalar, drawn afresh for each training example from a zero-mean Gaussian of width C_sigma (a hyperparameter), multiplies unit Gaussian noise added to every channel; the scalar's sign is immaterial since it scales symmetric noise. With the noise scale no longer learnable, the KL term of the ELBO reduces to a penalty on the mean, and training minimizes reconstruction error plus a weighted squared mean norm. A vanilla VAE under reconstruction pressure lets the learned variance of some channels collapse toward zero, leaving a near-deterministic autoencoder whose decoder has only ever seen points on the clean latent manifold. Sigma-VAE makes collapse impossible and the noise floor a knob.

Training latents fill a band of controlled width around each mean, the decoder learns to map the whole band to the correct output, and information is carried by mean displacements large enough to stand clear of the noise. An autoregressive sampling error smaller than the band keeps the context inside the distribution the system was trained on; variance sets the error-tolerance radius of the latent code. Note what does and does not change: the generator itself still trains teacher-forced, and the robustness is bought entirely in the representation underneath it. The corruption also satisfies the correctness criterion by construction, because the noisy latent is a genuine posterior sample, so the reconstruction target is exactly right for it; isotropic noise carries no alternative semantics, and nothing analogous to the scheduled-sampling target misalignment can occur.

Source: LatentLM paper, Figure 6, p. 8. What to see: in the left panel (no classifier-free guidance) the orange LatentLM curve improves as tokenizer variance grows while the black DiT curve stays nearly flat; in the center and right panels, guidance bends the LatentLM curves into a U whose high-variance arm rises again. In every panel the star markers, tokenizers tuned for image-level latent diffusion with tiny variance, sit far above the LatentLM curve and roughly on the DiT curve.

LatentLM trains image tokenizers across a sweep of fixed variances and evaluates each latent space under two generators: DiT, a non-autoregressive diffusion Transformer that denoises all positions jointly and so never feeds a sampled latent back as conditioning context for later positions, and LatentLM, which does exactly that at every position. FID (Frechet Inception Distance, an image-generation quality score, lower is better) is nearly insensitive to tokenizer variance for DiT but, without classifier-free guidance (CFG, a sampling-time knob that amplifies conditioning), improves monotonically with variance for LatentLM. The small-variance tokenizers that serve image-level diffusion perfectly are the worst latent spaces for autoregressive consumption. The two generators differ in architecture and objective wholesale, so this is directional evidence rather than a single-variable ablation, but the variance sensitivity tracks exactly the presence of cross-position feedback. The U-shaped curves under guidance are the budget side of the knob: past the optimum, variance spent on robustness is capacity taken from the code. The sweep is image-domain with fixed rather than sampled variances; nothing isolates the knob for speech.

GIVT reports the same trade through a different knob. Raising the VAE's KL weight makes reconstruction worse but autoregressive sampling better, and since reconstruction quality is a ceiling generation cannot exceed, the knob trades ceiling height for how closely the generator approaches it. This is the same decoupling between reconstruction and generation usability documented for non-autoregressive generators in SAME Makes Generation Difficulty an Autoencoder Training Loss. How to budget the trade across KL targets, noise injection, normalization, and decoder tolerance is the subject of Latent Audio Encoders: Reconstruction Sets the Ceiling, Not the Ranking, which treats latent regularization as deciding which component absorbs prediction error.

VibeVoice is the production endorsement. Its frozen acoustic tokenizer is a sigma-VAE using the sampled-sigma scheme with C_sigma = 0.5, adopted, in the paper's words, to mitigate variance collapse in autoregressive modeling settings, and the system generates up to 90 minutes of multi-speaker audio, the regime where dense latent drift has the longest lever arm. That working value is the only published speech-side variance datapoint we know of in this family; treat it as a starting point, not a tuned optimum.

Diffusion samplers have the same mismatch, and the same fix works

The exposure-bias template is not specific to autoregression over positions; it applies wherever a model consumes self-generated state. A diffusion sampler is the clearest second instance: training denoises ground-truth noised inputs, but at inference each denoising step receives the previous step's prediction, a mismatch Ning et al. describe as the exposure bias problem shared with autoregressive methods, with prediction errors accumulating along the chain. Their fix is the sigma-VAE move at the sampler level: perturb the training input with extra Gaussian noise to simulate inference-time prediction error, leaving the target untouched. The perturbation is semantics-free, so the original target remains a valid answer for the perturbed input (the same property that makes the noisy posterior sample a correct training pair above), and the retrained ADM-IP model matches its baseline's quality with far fewer sampling steps (80 versus 1,000 in their headline comparison). Wherever a model consumes self-generated state, this is the property to check first: the supervision attached to a corrupted or self-generated context must still be correct for that context.

Sources

• Ranzato et al., 2016, Sequence Level Training with Recurrent Neural Networks (arXiv:1511.06732): naming and definition of exposure bias (intro and teacher forcing section); the misaligned-target example against data-as-demonstrator (scheduled sampling section); MIXER mechanism and results (sequence-level training section).
• Bengio et al., 2015, Scheduled Sampling for Sequence Prediction with Recurrent Neural Networks (arXiv:1506.03099): error-amplification framing (teacher forcing section); the sampling mechanism, decay schedules, no-backprop detail, task results, and the always-sampling failure (scheduled sampling section).
• Huszar, 2015, How (not) to Train your Generative Model: Scheduled Sampling, Likelihood, Adversary? (arXiv:1511.05101): improperness of the scheduled sampling objective, the product-of-marginals optimum, and the prefix-ignoring optimal model (scheduled sampling section).
• Ross, Gordon, Bagnell, 2011, A Reduction of Imitation Learning and Structured Prediction to No-Regret Online Learning (arXiv:1011.0686): quadratic-versus-linear compounding bounds, Theorems 2.1 and 3.2, and the DAgger mechanism (teacher forcing and sequence-level training sections).
• Arora et al., 2022, Why Exposure Bias Matters: An Imitation Learning Perspective of Error Accumulation in Language Generation (arXiv:2204.01171): measured error accumulation and perplexity's blindness to it (teacher forcing section).
• He et al., 2021, Exposure Bias versus Self-Recovery: Are Distortions Really Incremental for Autoregressive Text Generation? (arXiv:1905.10617): the self-recovery finding bounding in-domain damage (model-error section).
• Wang and Sennrich, 2020, On Exposure Bias, Hallucination and Domain Shift in Neural Machine Translation (arXiv:2005.03642): in-domain versus out-of-domain hallucination rates and the minimum risk training result (model-error section).
• Sabour, Chan, Norouzi, 2018, Optimal Completion Distillation for Sequence Learning (arXiv:1810.01398): edit-distance-optimal target relabeling on model-sampled prefixes (sequence-level training section).
• Lamb et al., 2016, Professor Forcing: A New Algorithm for Training Recurrent Networks (arXiv:1610.09038): adversarial matching of teacher-forced and free-running dynamics (sequence-level training section).
• Sun et al., 2024, Multimodal Latent Language Modeling with Next-Token Diffusion / LatentLM (arXiv:2412.08635): next-token diffusion definition (continuous-latent section); sigma-VAE formulation, Section 2.3; the variance sweep, Section 3.1.3 and Figure 6, p. 8 (variance shaping section, including the reproduced figure).
• Tschannen, Eastwood, Mentzer, 2023, GIVT: Generative Infinite-Vocabulary Transformers (arXiv:2312.02116): Gaussian-mixture head (continuous-latent section); the KL-weight versus sampling-quality trade and the reconstruction ceiling (variance shaping section).
• Li et al., 2024, Autoregressive Image Generation without Vector Quantization / MAR (arXiv:2406.11838): diffusion loss as the per-token distribution model for continuous tokens (continuous-latent section).
• VibeVoice: Expressive Podcast Generation with Next-Token Diffusion (OpenReview, ICLR 2026): sigma-VAE adoption rationale, Section 2.1; C_sigma = 0.5, Appendix F; 90-minute multi-speaker generation, abstract (variance shaping section). The arXiv technical report (arXiv:2508.19205) omits the appendix carrying the C_sigma value.
• Ning et al., 2023, Input Perturbation Reduces Exposure Bias in Diffusion Models (arXiv:2301.11706): the diffusion-sampler instance and the input-perturbation fix (final section).

Related Diffio Posts

• Latent Audio Encoders: Reconstruction Sets the Ceiling, Not the Ranking: the regularization-as-error-absorption framing this post's variance section builds on.
• SAME Makes Generation Difficulty an Autoencoder Training Loss: the reconstruction-versus-generation decoupling in a non-autoregressive setting.

The scale is widely described as conditioning strength. That framing predicts the benefit, stronger adherence to the condition, and none of the costs. By Bayes rule, the difference between the conditional and unconditional predictions estimates the gradient of a classifier the generative model already contains, and the guidance weight is an exponent on that classifier's probability. That one fact is what makes the rest predictable: CFG sharpens the sampling distribution rather than improving the model, costs a second forward pass per step, trades diversity for per-sample fidelity, pushes the sampler off its training distribution at high scales, and has no portable "good value", because the exponent multiplies a quantity whose magnitude is system-specific.

Classifier guidance needed a separate noise-aware classifier

CFG replaced an earlier construction. A diffusion model's denoising network eps(x_t, c) predicts the noise mixed into the current iterate x_t, and up to a known negative scale factor that prediction estimates the score, the gradient of the log density of the noised data. Bayes rule splits the conditional score into two terms:

grad log p_t(x | c) = grad log p_t(x) + grad log p_t(c | x)

Dhariwal and Nichol exploited the second term directly: train a separate classifier p(c | x_t) and add its input-gradient, multiplied by a scale s, to the diffusion model's score at every sampling step. The classifier must be trained on noised inputs at every noise level the sampler visits, because a clean-data classifier produces meaningless gradients on the noisy iterates the sampler actually queries.

The scale was already an exponent there. Multiplying the gradient by s turns p(c | x) into p(c | x)^s inside the log, and as a function of x that term up-weights the inputs the classifier assigns to c most confidently, more aggressively as s grows. The exponent was not optional. Guiding an unconditional ImageNet model at s = 1, which is exactly the Bayes decomposition, produced samples that did not visibly match their classes even though the classifier scored them near 50%: the estimated gradient is too weak in practice to carry the conditioning alone. Pushing s from 1 to 10 on ImageNet traded recall (a coverage measure of diversity) from 0.52 down to 0.32 while precision rose from 0.82 to 0.88. The construction worked but carried an operational burden: a separate noise-aware classifier per conditioning type, and for a condition richer than a class label, a free-text prompt, there is no off-the-shelf classifier to train.

Bayes rule removes the classifier network, not the classification

Ho and Salimans inverted the same identity. The implicit classifier inside any conditional generative model is p(c | x) proportional to p(x | c) / p(x), so its gradient is the difference between the conditional and unconditional scores. A diffusion model that can produce both predictions does not need the external classifier; the subtraction eps(x_t, c) minus eps(x_t) plays its role.

Both predictions come from one network. During training the condition c is replaced with probability p_uncond by a null token, a learned embedding standing for "no condition", so the same weights learn p(x | c) and p(x) jointly. In their ablation, dropout rates of 0.1 and 0.2 perform about equally and 0.5 is consistently worse: the unconditional branch needs only a minority of training capacity. Sampling combines the two outputs:

eps_guided = (1 + w) * eps(x_t, c)  -  w * eps(x_t)

with guidance strength w, where w = 0 is plain conditional sampling. Most codebases reparameterize with s = 1 + w:

eps_guided = eps_uncond + s * (eps_cond - eps_uncond)

so s = 1 means guidance off. The two conventions differ by exactly one, both are in active use, and published "guidance scale" numbers are not comparable without the accompanying equation.

The cost is structural: every sampling step evaluates the full network twice, once per branch, usually run as a doubled batch. When that matters, the guided pair can be distilled into a single student network that matches the combined output directly (Meng et al. 2022), moving the cost from every deployment step to a one-time training job.

The scale tilts the sampling distribution toward the implicit classifier's modes

Substituting the implicit classifier back into the classifier-guidance formula shows what the sampler is being pointed at, per noise level:

grad log [ p_t(x | c) * p_implicit(c | x)^w ]

Each denoising step moves toward a tilted distribution: the model's own conditional, multiplied by the implicit classifier raised to the guidance power. Geometrically, the update extrapolates past the conditional prediction, directly away from the unconditional one: away from what the model expects regardless of the condition, deeper into the region that only the condition explains.

Two gaps separate that picture from exact sampling, and Ho and Salimans state the first themselves: the epsilon outputs are unconstrained network outputs, so their difference is not in general the gradient of any actual classifier. The second is that even with exact scores, steering each noise level toward its tilted marginal does not compose into sampling the tilted data distribution; Bradley and Nakkiran later showed that CFG instead matches a predictor-corrector sampler, alternating a denoising step with a sharpening correction. CFG is a per-step correction that reliably lands in high-confidence regions of the implicit classifier, not exact inference over a known distribution.

The exponent reading predicts the measured trade-offs. Concentrating mass on classifier modes should hurt distribution-level metrics first and per-sample metrics last, and it does: on ImageNet 64x64, FID (a distribution distance that punishes diversity loss and per-sample artifacts alike) is best at the barely-on w = 0.1 and degrades about seventeenfold by w = 4, while Inception Score, which rewards confidently recognizable individual samples, rises monotonically over the same sweep. Ho and Salimans describe the visual version plainly: sample variety decreases as individual sample fidelity increases.

Pushed further, guidance breaks the sampler's own input distribution. The Imagen authors traced high-guidance failure to a train-test mismatch: the guided update implies clean-image estimates outside the [-1, 1] range of training images, the sampler iterates on its own out-of-range outputs, and images come out saturated and unnatural or diverge outright. Their fix, dynamic thresholding, re-clamps the implied clean-image estimate at every step. The portable lesson is the mechanism, not the fix: extrapolation queries the network off its training manifold, and what breaks first is whatever your domain's analog of pixel range is.

The same extrapolation runs per token on logits in autoregressive models

Nothing in the construction is diffusion-specific; it needs only two predictions from one network that differ in conditioning. In discrete autoregressive models the combination runs once per decoding step, on logits. Koel-TTS, an encoder-decoder TTS token model, trains with both the text and the reference speaker audio dropped together at probability 0.1, then decodes with

logits_guided = logits_uncond + gamma * (logits_cond - logits_uncond)

before the softmax, where gamma sits in the s-convention (gamma = 1 is guidance off). Here the tilt is exact where diffusion's is heuristic: the softmax renormalizes, so each token is sampled from precisely p_cond^gamma times p_uncond^(1-gamma), normalized. The exactness is per token: the product of per-token tilts is not the tilted distribution over whole sequences, where the same caveat reappears one level up. The unconditional branch consumes the same generated token history with no text or speaker prefix, so its prediction answers "how would this audio continue with no instructions", and the guided difference amplifies exactly what the instructions explain. Text LLMs use the same per-step recipe to amplify a prompt, contrasting predictions with and without it (Sanchez et al. 2023), and the two branches' different prefixes mean AR CFG doubles the KV cache as well as the per-token compute.

The trade-off lands differently in TTS than in images. Sweeping gamma from 1 to 3, Koel-TTS finds character error rate and speaker-embedding similarity improve together, with the optimum at 2.5: both metrics measure adherence to a conditioning input being amplified, and much of the "diversity" being burned is the unconditioned failure mode, hallucinated words and drifting voice. The cost surfaces where the paper notes it, as reduced variation at higher scales, not in the adherence metrics.

The unconditional branch defines what you extrapolate away from

What the second branch conditions on is a design choice, and systems diverge on it. Koel-TTS drops text and speaker context together, so guidance amplifies both at once. VibeVoice's report describes no trained null token: its diffusion head's unconditional branch conditions on the hidden state of the start-of-sequence token, a minimally conditioned input the model computes anyway. SemaVoice drops the LLM hidden state but keeps the previously generated acoustic patch in the unconditional branch, so patch-to-patch continuity stays unguided and only the semantic conditioning is sharpened. The negative-prompt field in image generators is this same equation with the null condition replaced by an unwanted description, turning "away from what is typical" into "away from what is specified".

Two refinements push the lever further. InstructPix2Pix decomposes the condition and runs two differences with two scales, one amplifying agreement with the input image and one with the edit instruction, the practical alternative to dropping all conditions jointly the way Koel-TTS does. Autoguidance (Karras et al. 2024) replaces the unconditional branch with a smaller, less-trained version of the same model, so the extrapolation points away from that weaker model's errors rather than away from the condition's absence, and recovers fidelity without the diversity collapse. In every case the guided sample is defined as much by the negative branch as by the positive one.

Tuned scales do not transfer across systems, tasks, or notation conventions

Speech systems that all cite the same CFG paper (we locate these architectures in our map of the modern speech stack) ship guidance settings spanning a factor of three, at different denoising step counts:

Source: VibeVoice Sec. 3.2 and Fig. 3; Koel-TTS Secs. 2.5 and 3.4; SemaVoice Sec. 3.2.3 (which uses the (1+w) convention, hence the conversion); LatentLM Secs. 3.1 and 3.3.6.

LatentLM is one architecture family, and it needs 1.65 for class-conditional ImageNet but 4 for TTS: neither the architecture nor the codebase pins the value, the task does. And SemaVoice's published 2.5 is 3.5 in the convention the other rows use, because its paper writes the Ho-Salimans (1+w) form; copying a guidance number without its equation silently changes the strength by one full unit.

The spread follows from the exponent reading: the scale multiplies the implicit classifier's log-likelihood ratio, and the magnitude of that ratio depends on how strongly the conditioning pins the output (a transcript plus a speaker prompt constrains speech far harder than a class label constrains an image), on what the unconditional branch conditions on, and on the sampler's step count. The guidance scale is a coefficient on a system-specific signal, not an absolute strength in shared units.

Published sweeps differ in shape, not just in optimum:

LatentLM paper, Figure 10, p. 14: intelligibility (1 - WER-C, where WER-C is word error rate from a Conformer-Transducer ASR transcription) and speaker similarity over CFG scales 1 to 8 and over diffusion sampling steps. The jump from scale 1 (guidance off) to 2 dwarfs everything after it, and the optimum at 4 sits on a broad plateau.

VibeVoice paper, Figure 3, p. 8: word error rate and SIM-O (speaker-embedding similarity to the reference audio) over CFG scales 1.25 to 1.35 and denoising steps 5 to 35. Within that narrow scale window, WER still swings from 3.83 to 1.55 depending on the scale-step pair, and the WER optimum (scale 1.25 at 10 steps) disagrees with the similarity optimum (5 steps), so the shipped setting of 1.3 at 10 steps is a compromise.

The operational consequence: treat the guidance scale and step count as one jointly swept pair per system, and when reporting the result, publish the combination equation next to the number. A scale without its equation is not a reproducible setting.

Sources

• Ho & Salimans, Classifier-Free Diffusion Guidance (arXiv 2207.12598): condition-dropout training and p_uncond ablation (Table 1), the guided combination and w convention (Eq. 6), the implicit-classifier derivation and the not-a-real-gradient caveat, the two-evaluation cost, and the ImageNet 64x64 FID/IS sweep.
• Dhariwal & Nichol, Diffusion Models Beat GANs on Image Synthesis (arXiv 2105.05233): classifier guidance mechanics, noise-aware classifier requirement, gradient scaling as the exponent on p(c|x), the s = 1 failure on an unconditional model, and the precision/recall trade (Table 4).
• Saharia et al., Imagen (arXiv 2205.11487): high-guidance train-test mismatch, out-of-range clean-image estimates, saturation, and dynamic thresholding.
• Bradley & Nakkiran, Classifier-Free Guidance is a Predictor-Corrector (arXiv 2408.09000): CFG does not sample the tilted distribution; supports the exact-sampling caveat.
• Koel-TTS (arXiv 2502.05236): logit-space CFG for an AR TTS model (Sec. 2.5), 10% joint dropout, the gamma sweep and optimum (Sec. 3.4, Fig. 4), and the doubled effective batch.
• LatentLM, Multimodal Latent Language Modeling with Next-Token Diffusion (arXiv 2412.08635): TTS CFG and step ablations (Fig. 10), image-generation guidance settings (Sec. 3.1), WER-C metric definition (Sec. 3.3).
• VibeVoice (OpenReview, ICLR 2026): diffusion-head CFG with the start-of-sequence unconditional branch (Eq. 6), shipped settings (Sec. 3.2), and the scale-step heatmaps (Fig. 3). The shorter arXiv technical report (2508.19205) omits the heatmap figure.
• SemaVoice (arXiv 2605.16964): LLM-state dropout with retained previous patch, (1+w) convention at w = 2.5 (Sec. 3.2.3).
• Sanchez et al., Stay on Topic with Classifier-Free Guidance (arXiv 2306.17806): per-step logit CFG in pure text LLMs.
• Brooks et al., InstructPix2Pix (arXiv 2211.09800): two-condition CFG with independent guidance scales (Eq. 3).
• Karras et al., Guiding a Diffusion Model with a Bad Version of Itself (arXiv 2406.02507): autoguidance; replacing the unconditional branch with a degraded model.
• Meng et al., On Distillation of Guided Diffusion Models (arXiv 2210.03142): removing the two-pass cost by distilling the guided pair.

Related Diffio Posts

• TTS Models: A Map of the Modern Speech Stack