[{"data":1,"prerenderedAt":72},["ShallowReactive",2],{"blog-post-moss-tts-rvq-depth":3},{"slug":4,"path":5,"title":6,"description":7,"publishedAt":8,"updatedAt":9,"author":10,"category":11,"tags":12,"featureImage":15,"ogImage":20,"canonical":21,"readingTimeMinutes":22,"bodyHtml":23,"showFeatureImage":24,"relatedPosts":25},"moss-tts-rvq-depth","/blog/moss-tts-rvq-depth/","MOSS-TTS Shows Local RVQ Conditioning Can Beat Delay Modeling","MOSS-TTS, a fully discrete text-to-speech system, shows that conditioning RVQ codec layers locally can beat the delay pattern for predicting multi-codebook audio tokens.","2026-06-12T20:03:14Z","2026-06-13T16:07:44Z","Nathan Harmon","research",[13,14],"TTS","ML",{"src":16,"alt":17,"widthPixels":18,"heightPixels":19},"/media/blog/moss-tts-rvq-depth/figure-2.webp","MOSS-TTS Figure 2: delay pattern and local transformer architectures",1324,726,{"src":16,"alt":17,"widthPixels":18,"heightPixels":19},"https://diffio.ai/blog/moss-tts-rvq-depth/",7,"\u003Cp>MOSS-TTS is a fully discrete text-to-speech system: text and prompt audio are serialized into language-model inputs, the model predicts audio codec tokens autoregressively, and a neural codec decoder turns those tokens back into waveform. In the \u003Ca href=\"/blog/tts-models-a-map-of-the-modern-speech-stack/\">Diffio map of modern TTS systems\u003C/a>, that places it in the discrete codec-LM family, where the generated object is not a mel frame or a continuous latent but a matrix of codec tokens.\u003C/p>\n\u003Cp>The MOSS comparison is internal: the report evaluates an 8B delay-pattern generator and a 1.7B Local Transformer under the same tokenizer and pretraining recipe. The comparison is not parameter matched, so it does not isolate causal RVQ depth from every local-module difference. It shows that in this released comparison, the smaller local architecture outperformed the larger delay backbone on the reported cloning metrics.\u003C/p>\n\u003Ch2 id=\"low-frame-rate-shifts-modeling-cost-to-rvq-depth\">Low frame rate shifts modeling cost to RVQ depth\u003C/h2>\n\u003Cp>MOSS-Audio-Tokenizer makes long context possible by using a low frame rate and multiple RVQ layers per frame. It encodes 24 kHz audio at 12.5 frames per second.\u003C/p>\n\u003Cp>Each frame contains 32 residual vector quantization (RVQ) layers. RVQ means the first codebook quantizes a coarse approximation, the next codebook quantizes what the first missed, and later codebooks keep refining the residual.\u003C/p>\n\u003Cp>The report's final pretraining phase measures context in backbone time steps rather than scalar codec decisions. At 12.5 frame steps per second, its 64k-step training window covers about 85 minutes of audio before text, prompt tokens, and the delay model's 31-step edge overhead. The remaining modeling problem is the 32 ordered decisions attached to every frame.\u003C/p>\n\u003Cp>The tokenizer can trade bitrate for residual depth because random quantizer dropout trains the decoder after randomly dropping later RVQ layers, so prefixes of the 32-layer stack remain valid codes. At this frame rate, each additional RVQ layer adds 125 bps.\u003C/p>\n\u003Cdiv class=\"blog-table\" role=\"region\" aria-label=\"Table 1\" tabindex=\"0\">\u003Ctable>\n\u003Cthead>\n\u003Ctr>\n\u003Cth style=\"text-align:right\">Bitrate\u003C/th>\n\u003Cth style=\"text-align:right\">RVQ layers used\u003C/th>\n\u003Cth style=\"text-align:right\">English/Chinese SIM\u003C/th>\n\u003Cth style=\"text-align:right\">English/Chinese PESQ-WB\u003C/th>\n\u003C/tr>\n\u003C/thead>\n\u003Ctbody>\n\u003Ctr>\n\u003Ctd style=\"text-align:right\">1000 bps\u003C/td>\n\u003Ctd style=\"text-align:right\">8\u003C/td>\n\u003Ctd style=\"text-align:right\">0.88 / 0.81\u003C/td>\n\u003Ctd style=\"text-align:right\">2.87 / 2.43\u003C/td>\n\u003C/tr>\n\u003Ctr>\n\u003Ctd style=\"text-align:right\">4000 bps\u003C/td>\n\u003Ctd style=\"text-align:right\">32\u003C/td>\n\u003Ctd style=\"text-align:right\">0.97 / 0.93\u003C/td>\n\u003Ctd style=\"text-align:right\">3.69 / 3.30\u003C/td>\n\u003C/tr>\n\u003C/tbody>\n\u003C/table>\n\u003C/div>\u003Cp>MOSS-TTS Technical Report Table 2. SIM is cosine-scale speaker-embedding similarity; PESQ-WB is a wideband perceptual speech-quality score. These are tokenizer-oracle reconstruction rows, not a generator-factorization result.\u003C/p>\n\u003Ch2 id=\"the-delay-pattern-preserves-sequence-length-by-interleaving-rvq-layers\">The delay pattern preserves sequence length by interleaving RVQ layers\u003C/h2>\n\u003Cp>The delay pattern keeps the language-model sequence close to the audio frame count. Instead of flattening the RVQ stack into separate audio positions, it shifts each residual layer by its layer index. A \u003Ccode>T\u003C/code>-frame utterance becomes \u003Ccode>T + 31\u003C/code> delayed backbone steps.\u003C/p>\n\u003Cp>Each aligned step has one text/pad channel and one channel per RVQ layer. Channel 0 emits a normal text token on text-only steps and a dedicated pad symbol on audio-frame steps; the audio channels emit RVQ codes. Invalid delayed or padding positions are masked from the loss. In the delay model, one backbone hidden state feeds all channel heads.\u003C/p>\n\u003Cp>This factorization is efficient for serving. The backbone is evaluated once per delayed step, and the output heads map the hidden state to token logits. This is why the report treats the delay architecture as the simpler route for long contexts, duration control, pronunciation control, and optimized inference backends.\u003C/p>\n\u003Cp>The cost is that within-frame RVQ dependencies are represented across multiple delayed steps. Lower layers of a frame are emitted at earlier delayed steps than higher layers. By the time a higher layer is predicted, lower layers can influence it only through intervening input vectors, each built by adding codebook-specific embeddings from whatever delayed layers are present at that step. At any one delayed step, the audio heads are predicting channels from different original frames. The backbone is therefore not just modeling speech over time; it is also tracking incomplete RVQ code stacks across delayed positions.\u003C/p>\n\u003Ch2 id=\"the-local-transformer-factorizes-each-frame-autoregressively-over-rvq-layers\">The Local Transformer factorizes each frame autoregressively over RVQ layers\u003C/h2>\n\u003Cp>The Local Transformer keeps the frame aligned in time and adds a local autoregressive loop over channels. The temporal backbone produces one latent for the frame, then a lightweight local module predicts the text/pad channel and the RVQ layers in order. The text/pad channel keeps that stream in the same factorization as the audio layers. For audio frames it is the pad decision, followed by RVQ layers that condition on earlier layers from the same frame and on previous aligned steps.\u003C/p>\n\u003Cp>That is the inductive bias the MOSS comparison supports for this residual stack. Later RVQ layers are not exchangeable labels; they are refinements of a residual left by earlier layers. Giving layer \u003Ccode>j\u003C/code> the sampled lower-depth codes as direct context makes within-frame RVQ conditioning explicit. The delay pattern can represent that dependency only after the temporal backbone learns to route it through delayed positions.\u003C/p>\n\u003Cp>Moshi is the immediate lineage for this choice: its RQ-Transformer predicts tokens bottom to top inside an audio step, and MOSS adapts that idea to the tokenizer's full RVQ stack. The tradeoff is steady-state cost. The local model runs an inner loop over channels, while the delay model cannot stream-decode a complete frame until the final RVQ layer arrives about 2.5 seconds later.\u003C/p>\n\u003Cfigure class=\"blog-figure blog-figure--image\">\u003Cimg src=\"/media/blog/moss-tts-rvq-depth/figure-2.webp\" alt=\"MOSS-TTS Figure 2: delay pattern and local transformer architectures\" width=\"1324\" height=\"726\" loading=\"lazy\" decoding=\"async\">\u003C/figure>\n\n\u003Cp>MOSS-TTS Technical Report, Figure 2, p. 8. Delay pattern on the left; frame-local autoregressive loop on the right.\u003C/p>\n\u003Ch2 id=\"the-smaller-local-model-wins-the-within-moss-cloning-comparison\">The smaller local model wins the within-MOSS cloning comparison\u003C/h2>\n\u003Cp>Seed-TTS-eval reports WER for English word error rate, CER for Chinese character error rate, and SIM for speaker-embedding similarity on a percentage scale. This SIM scale differs from the 0-1 reconstruction SIM above.\u003C/p>\n\u003Cdiv class=\"blog-table\" role=\"region\" aria-label=\"Table 2\" tabindex=\"0\">\u003Ctable>\n\u003Cthead>\n\u003Ctr>\n\u003Cth>Model\u003C/th>\n\u003Cth>Mode\u003C/th>\n\u003Cth style=\"text-align:right\">Params\u003C/th>\n\u003Cth style=\"text-align:right\">EN WER\u003C/th>\n\u003Cth style=\"text-align:right\">EN SIM\u003C/th>\n\u003Cth style=\"text-align:right\">ZH CER\u003C/th>\n\u003Cth style=\"text-align:right\">ZH SIM\u003C/th>\n\u003C/tr>\n\u003C/thead>\n\u003Ctbody>\n\u003Ctr>\n\u003Ctd>MOSS-TTS\u003C/td>\n\u003Ctd>Clone\u003C/td>\n\u003Ctd style=\"text-align:right\">8B\u003C/td>\n\u003Ctd style=\"text-align:right\">1.92\u003C/td>\n\u003Ctd style=\"text-align:right\">69.31\u003C/td>\n\u003Ctd style=\"text-align:right\">1.46\u003C/td>\n\u003Ctd style=\"text-align:right\">76.21\u003C/td>\n\u003C/tr>\n\u003Ctr>\n\u003Ctd>MOSS-TTS-Local-Transformer\u003C/td>\n\u003Ctd>Clone\u003C/td>\n\u003Ctd style=\"text-align:right\">1.7B\u003C/td>\n\u003Ctd style=\"text-align:right\">1.87\u003C/td>\n\u003Ctd style=\"text-align:right\">71.74\u003C/td>\n\u003Ctd style=\"text-align:right\">1.33\u003C/td>\n\u003Ctd style=\"text-align:right\">77.24\u003C/td>\n\u003C/tr>\n\u003Ctr>\n\u003Ctd>MOSS-TTS\u003C/td>\n\u003Ctd>Continuation\u003C/td>\n\u003Ctd style=\"text-align:right\">8B\u003C/td>\n\u003Ctd style=\"text-align:right\">1.84\u003C/td>\n\u003Ctd style=\"text-align:right\">70.86\u003C/td>\n\u003Ctd style=\"text-align:right\">1.37\u003C/td>\n\u003Ctd style=\"text-align:right\">76.98\u003C/td>\n\u003C/tr>\n\u003Ctr>\n\u003Ctd>MOSS-TTS-Local-Transformer\u003C/td>\n\u003Ctd>Continuation\u003C/td>\n\u003Ctd style=\"text-align:right\">1.7B\u003C/td>\n\u003Ctd style=\"text-align:right\">1.93\u003C/td>\n\u003Ctd style=\"text-align:right\">73.28\u003C/td>\n\u003Ctd style=\"text-align:right\">1.44\u003C/td>\n\u003Ctd style=\"text-align:right\">79.62\u003C/td>\n\u003C/tr>\n\u003C/tbody>\n\u003C/table>\n\u003C/div>\u003Cp>Seed-TTS-eval, MOSS-TTS Technical Report Table 3. Lower WER/CER (%) is better; higher SIM (%) is better. The report gives no confidence intervals or listening-test variance for this table.\u003C/p>\n\u003Cp>The Local Transformer outperforms the delay model in all four SIM cells, and the highest MOSS similarity value in the table is the local continuation model's Chinese SIM. In Clone mode it also lowers both content-error metrics. In Continuation mode the delay model has slightly lower WER/CER, but both models sit in the low-error regime where the report says manual review found many remaining mismatches to be ASR errors rather than audible pronunciation failures.\u003C/p>\n\u003Cp>CV3-Eval extends the same direction to multilingual cloning, but only for content error. The Local Transformer has lower CER/WER than the delay model in every same-language comparison. Its average absolute improvement over the delay rows is about half a percentage point in both Clone and Continuation mode. That does not establish better speaker preservation outside English and Chinese, because CV3-Eval does not report speaker similarity, but it rules out the narrow explanation that the local model merely trades intelligibility for timbre.\u003C/p>\n\u003Cp>The smaller model outperformed the larger one on the cloning cells the report uses to characterize the local-vs-delay tradeoff, so the evidence supports a scoped design claim: for this tokenizer and release, local conditioning over RVQ depth was the better cloning architecture.\u003C/p>\n\u003Ch2 id=\"control-and-ultra-long-results-are-delay-only\">Control and ultra-long results are delay-only\u003C/h2>\n\u003Cp>The control sections evaluate only the 8B delay model. Duration control, pronunciation control, and ultra-long generation are not reported for the Local Transformer.\u003C/p>\n\u003Cp>The delay architecture supports the control and long-context experiments because it has a simpler single-backbone parameterization. For duration control, the prompt includes a requested audio-token count. The delay model's absolute relative duration error, the generated-duration miss divided by the requested duration, is 0.712% in Chinese and 0.723% in English, using paired pretraining examples with and without explicit token-count prompts rather than a separate duration-control fine-tune.\u003C/p>\n\u003Cp>The ultra-long internal set separates content from speaker drift. In the English \u003Ccode>50000+\u003C/code> input-character bucket, Clone WER reaches 17.49% and Continuation WER reaches 29.52%, while speaker similarity falls to 44.4% for Clone and 51.2% for Continuation.\u003C/p>\n\u003Cfigure class=\"blog-figure blog-figure--image\">\u003Cimg src=\"/media/blog/moss-tts-rvq-depth/figure-6.webp\" alt=\"MOSS-TTS Figure 6: speaker similarity drift under ultra-long generation\" width=\"1318\" height=\"301\" loading=\"lazy\" decoding=\"async\">\u003C/figure>\n\n\u003Cp>MOSS-TTS Technical Report, Figure 6, p. 21. The English longest-bucket curves separate early and remain low, showing that long-horizon degradation appears as cumulative speaker drift before every sample reaches a lexical failure.\u003C/p>\n\u003Ch2 id=\"sources\">Sources\u003C/h2>\n\u003Cul>\n\u003Cli>\u003Ca href=\"https://arxiv.org/abs/2603.18090\">MOSS-TTS Technical Report\u003C/a>: tokenizer geometry, Figure 2 architecture diagram, delay and Local Transformer objectives, Seed-TTS-eval Table 3, CV3-Eval Table 4, duration Table 5, ultra-long Table 6.\u003C/li>\n\u003Cli>\u003Ca href=\"https://arxiv.org/abs/2602.10934\">MOSS-Audio-Tokenizer\u003C/a>: CAT tokenizer motivation, end-to-end Transformer tokenizer context, and the tokenizer's role as a scalable audio-modeling interface.\u003C/li>\n\u003Cli>\u003Ca href=\"https://arxiv.org/abs/2306.05284\">MusicGen: Simple and Controllable Music Generation\u003C/a>: delay/interleaving-pattern lineage for multi-codebook audio-token modeling.\u003C/li>\n\u003Cli>\u003Ca href=\"https://arxiv.org/abs/2410.00037\">Moshi: a speech-text foundation model for real-time dialogue\u003C/a>: RQ-Transformer lineage and frame-local bottom-to-top token modeling.\u003C/li>\n\u003C/ul>\n\u003Ch2 id=\"related-diffio-posts\">Related Diffio posts\u003C/h2>\n\u003Cul>\n\u003Cli>\u003Ca href=\"/blog/tts-models-a-map-of-the-modern-speech-stack/\">TTS Models: A Map Of The Modern Speech Stack\u003C/a>\u003C/li>\n\u003Cli>\u003Ca href=\"/blog/vibevoice-next-token-diffusion/\">VibeVoice: Frame Rate Is the Context Budget for Long-Form TTS\u003C/a>\u003C/li>\n\u003C/ul>\n",false,[26,42,57],{"slug":27,"path":28,"title":29,"description":30,"publishedAt":31,"updatedAt":32,"author":10,"category":11,"tags":33,"featureImage":34,"ogImage":39,"canonical":40,"readingTimeMinutes":41},"vibevoice-next-token-diffusion","/blog/vibevoice-next-token-diffusion/","VibeVoice: Frame Rate Is the Context Budget for Long-Form TTS","VibeVoice is Microsoft Research's open-weights system for zero-shot, multi-speaker, long-form speech synthesis: give it a short voice sample per speaker and a script, and a Qwen2.5 backbone generates podcast-style conversation, turn-taking and breaths included, in one streaming left-to-right pass.","2026-06-12T16:09:38Z","2026-06-13T17:03:26Z",[13,14],{"src":35,"alt":36,"widthPixels":37,"heightPixels":38},"/media/blog/vibevoice-next-token-diffusion/figure-1.webp","VibeVoice architecture: tokenizers, LLM, diffusion head, and the semantic feedback path",1109,599,{"src":35,"alt":36,"widthPixels":37,"heightPixels":38},"https://diffio.ai/blog/vibevoice-next-token-diffusion/",13,{"slug":43,"path":44,"title":45,"description":46,"publishedAt":47,"updatedAt":48,"author":10,"category":11,"tags":49,"featureImage":50,"ogImage":55,"canonical":56,"readingTimeMinutes":41},"exposure-bias","/blog/exposure-bias/","Exposure Bias: Corrupt the Context, Keep the Target Correct","Exposure bias is the mismatch between teacher-forced training and free-running inference in autoregressive models. The fix: corrupt the context, keep the target correct.","2026-06-12T16:06:23Z","2026-06-13T16:07:46Z",[14],{"src":51,"alt":52,"widthPixels":53,"heightPixels":54},"/media/blog/exposure-bias/figure-6.webp","LatentLM Figure 6: FID versus tokenizer variance for DiT and LatentLM at three CFG scales",1124,466,{"src":51,"alt":52,"widthPixels":53,"heightPixels":54},"https://diffio.ai/blog/exposure-bias/",{"slug":58,"path":59,"title":60,"description":61,"publishedAt":62,"updatedAt":48,"author":10,"category":11,"tags":63,"featureImage":64,"ogImage":69,"canonical":70,"readingTimeMinutes":71},"classifier-free-guidance","/blog/classifier-free-guidance/","Classifier-Free Guidance: The Guidance Scale Is an Exponent on an Implicit Classifier","Classifier-free guidance trains one network with condition dropout and extrapolates between its conditional and unconditional predictions. The guidance scale is an exponent on an implicit classifier.","2026-06-12T15:56:16Z",[14],{"src":65,"alt":66,"widthPixels":67,"heightPixels":68},"/media/blog/classifier-free-guidance/latentlm-figure-10.webp","LatentLM ablation of CFG scale and sampling steps for zero-shot TTS",1120,458,{"src":65,"alt":66,"widthPixels":67,"heightPixels":68},"https://diffio.ai/blog/classifier-free-guidance/",11,1793728206638]