[{"data":1,"prerenderedAt":74},["ShallowReactive",2],{"blog-post-classifier-free-guidance":3},{"slug":4,"path":5,"title":6,"description":7,"publishedAt":8,"updatedAt":9,"author":10,"category":11,"tags":12,"featureImage":14,"ogImage":19,"canonical":20,"readingTimeMinutes":21,"bodyHtml":22,"showFeatureImage":23,"relatedPosts":24},"classifier-free-guidance","/blog/classifier-free-guidance/","Classifier-Free Guidance: The Guidance Scale Is an Exponent on an Implicit Classifier","Classifier-free guidance trains one network with condition dropout and extrapolates between its conditional and unconditional predictions. The guidance scale is an exponent on an implicit classifier.","2026-06-12T15:56:16Z","2026-06-13T16:07:46Z","Nathan Harmon","research",[13],"ML",{"src":15,"alt":16,"widthPixels":17,"heightPixels":18},"/media/blog/classifier-free-guidance/latentlm-figure-10.webp","LatentLM ablation of CFG scale and sampling steps for zero-shot TTS",1120,458,{"src":15,"alt":16,"widthPixels":17,"heightPixels":18},"https://diffio.ai/blog/classifier-free-guidance/",11,"\u003Cp>Classifier-free guidance (CFG) trains a conditional generative model with condition dropout, so one network can predict both with and without the conditioning signal. At sampling time the two predictions are combined by extrapolating along their difference, scaled by a guidance weight. Nearly every modern conditional generator exposes that weight: image diffusion models, the per-token diffusion heads that autoregressive TTS systems run on top of an LLM backbone, and, per decoding step, token language models.\u003C/p>\n\u003Cp>The scale is widely described as conditioning strength. That framing predicts the benefit, stronger adherence to the condition, and none of the costs. By Bayes rule, the difference between the conditional and unconditional predictions estimates the gradient of a classifier the generative model already contains, and the guidance weight is an exponent on that classifier's probability. That one fact is what makes the rest predictable: CFG sharpens the sampling distribution rather than improving the model, costs a second forward pass per step, trades diversity for per-sample fidelity, pushes the sampler off its training distribution at high scales, and has no portable "good value", because the exponent multiplies a quantity whose magnitude is system-specific.\u003C/p>\n\u003Ch2 id=\"classifier-guidance-needed-a-separate-noise-aware-classifier\">Classifier guidance needed a separate noise-aware classifier\u003C/h2>\n\u003Cp>CFG replaced an earlier construction. A diffusion model's denoising network eps(x_t, c) predicts the noise mixed into the current iterate x_t, and up to a known negative scale factor that prediction estimates the score, the gradient of the log density of the noised data. Bayes rule splits the conditional score into two terms:\u003C/p>\n\u003Cpre class=\"blog-code\">\u003Ccode>grad log p_t(x | c) = grad log p_t(x) + grad log p_t(c | x)\n\u003C/code>\u003C/pre>\n\u003Cp>Dhariwal and Nichol exploited the second term directly: train a separate classifier p(c | x_t) and add its input-gradient, multiplied by a scale s, to the diffusion model's score at every sampling step. The classifier must be trained on noised inputs at every noise level the sampler visits, because a clean-data classifier produces meaningless gradients on the noisy iterates the sampler actually queries.\u003C/p>\n\u003Cp>The scale was already an exponent there. Multiplying the gradient by s turns p(c | x) into p(c | x)^s inside the log, and as a function of x that term up-weights the inputs the classifier assigns to c most confidently, more aggressively as s grows. The exponent was not optional. Guiding an unconditional ImageNet model at s = 1, which is exactly the Bayes decomposition, produced samples that did not visibly match their classes even though the classifier scored them near 50%: the estimated gradient is too weak in practice to carry the conditioning alone. Pushing s from 1 to 10 on ImageNet traded recall (a coverage measure of diversity) from 0.52 down to 0.32 while precision rose from 0.82 to 0.88. The construction worked but carried an operational burden: a separate noise-aware classifier per conditioning type, and for a condition richer than a class label, a free-text prompt, there is no off-the-shelf classifier to train.\u003C/p>\n\u003Ch2 id=\"bayes-rule-removes-the-classifier-network-not-the-classification\">Bayes rule removes the classifier network, not the classification\u003C/h2>\n\u003Cp>Ho and Salimans inverted the same identity. The implicit classifier inside any conditional generative model is p(c | x) proportional to p(x | c) / p(x), so its gradient is the difference between the conditional and unconditional scores. A diffusion model that can produce both predictions does not need the external classifier; the subtraction eps(x_t, c) minus eps(x_t) plays its role.\u003C/p>\n\u003Cp>Both predictions come from one network. During training the condition c is replaced with probability p_uncond by a null token, a learned embedding standing for "no condition", so the same weights learn p(x | c) and p(x) jointly. In their ablation, dropout rates of 0.1 and 0.2 perform about equally and 0.5 is consistently worse: the unconditional branch needs only a minority of training capacity. Sampling combines the two outputs:\u003C/p>\n\u003Cpre class=\"blog-code\">\u003Ccode>eps_guided = (1 + w) * eps(x_t, c)  -  w * eps(x_t)\n\u003C/code>\u003C/pre>\n\u003Cp>with guidance strength w, where w = 0 is plain conditional sampling. Most codebases reparameterize with s = 1 + w:\u003C/p>\n\u003Cpre class=\"blog-code\">\u003Ccode>eps_guided = eps_uncond + s * (eps_cond - eps_uncond)\n\u003C/code>\u003C/pre>\n\u003Cp>so s = 1 means guidance off. The two conventions differ by exactly one, both are in active use, and published "guidance scale" numbers are not comparable without the accompanying equation.\u003C/p>\n\u003Cp>The cost is structural: every sampling step evaluates the full network twice, once per branch, usually run as a doubled batch. When that matters, the guided pair can be distilled into a single student network that matches the combined output directly (Meng et al. 2022), moving the cost from every deployment step to a one-time training job.\u003C/p>\n\u003Ch2 id=\"the-scale-tilts-the-sampling-distribution-toward-the-implicit-classifiers-modes\">The scale tilts the sampling distribution toward the implicit classifier's modes\u003C/h2>\n\u003Cp>Substituting the implicit classifier back into the classifier-guidance formula shows what the sampler is being pointed at, per noise level:\u003C/p>\n\u003Cpre class=\"blog-code\">\u003Ccode>grad log [ p_t(x | c) * p_implicit(c | x)^w ]\n\u003C/code>\u003C/pre>\n\u003Cp>Each denoising step moves toward a tilted distribution: the model's own conditional, multiplied by the implicit classifier raised to the guidance power. Geometrically, the update extrapolates past the conditional prediction, directly away from the unconditional one: away from what the model expects regardless of the condition, deeper into the region that only the condition explains.\u003C/p>\n\u003Cp>Two gaps separate that picture from exact sampling, and Ho and Salimans state the first themselves: the epsilon outputs are unconstrained network outputs, so their difference is not in general the gradient of any actual classifier. The second is that even with exact scores, steering each noise level toward its tilted marginal does not compose into sampling the tilted data distribution; Bradley and Nakkiran later showed that CFG instead matches a predictor-corrector sampler, alternating a denoising step with a sharpening correction. CFG is a per-step correction that reliably lands in high-confidence regions of the implicit classifier, not exact inference over a known distribution.\u003C/p>\n\u003Cp>The exponent reading predicts the measured trade-offs. Concentrating mass on classifier modes should hurt distribution-level metrics first and per-sample metrics last, and it does: on ImageNet 64x64, FID (a distribution distance that punishes diversity loss and per-sample artifacts alike) is best at the barely-on w = 0.1 and degrades about seventeenfold by w = 4, while Inception Score, which rewards confidently recognizable individual samples, rises monotonically over the same sweep. Ho and Salimans describe the visual version plainly: sample variety decreases as individual sample fidelity increases.\u003C/p>\n\u003Cp>Pushed further, guidance breaks the sampler's own input distribution. The Imagen authors traced high-guidance failure to a train-test mismatch: the guided update implies clean-image estimates outside the [-1, 1] range of training images, the sampler iterates on its own out-of-range outputs, and images come out saturated and unnatural or diverge outright. Their fix, dynamic thresholding, re-clamps the implied clean-image estimate at every step. The portable lesson is the mechanism, not the fix: extrapolation queries the network off its training manifold, and what breaks first is whatever your domain's analog of pixel range is.\u003C/p>\n\u003Ch2 id=\"the-same-extrapolation-runs-per-token-on-logits-in-autoregressive-models\">The same extrapolation runs per token on logits in autoregressive models\u003C/h2>\n\u003Cp>Nothing in the construction is diffusion-specific; it needs only two predictions from one network that differ in conditioning. In discrete autoregressive models the combination runs once per decoding step, on logits. Koel-TTS, an encoder-decoder TTS token model, trains with both the text and the reference speaker audio dropped together at probability 0.1, then decodes with\u003C/p>\n\u003Cpre class=\"blog-code\">\u003Ccode>logits_guided = logits_uncond + gamma * (logits_cond - logits_uncond)\n\u003C/code>\u003C/pre>\n\u003Cp>before the softmax, where gamma sits in the s-convention (gamma = 1 is guidance off). Here the tilt is exact where diffusion's is heuristic: the softmax renormalizes, so each token is sampled from precisely p_cond^gamma times p_uncond^(1-gamma), normalized. The exactness is per token: the product of per-token tilts is not the tilted distribution over whole sequences, where the same caveat reappears one level up. The unconditional branch consumes the same generated token history with no text or speaker prefix, so its prediction answers "how would this audio continue with no instructions", and the guided difference amplifies exactly what the instructions explain. Text LLMs use the same per-step recipe to amplify a prompt, contrasting predictions with and without it (Sanchez et al. 2023), and the two branches' different prefixes mean AR CFG doubles the KV cache as well as the per-token compute.\u003C/p>\n\u003Cp>The trade-off lands differently in TTS than in images. Sweeping gamma from 1 to 3, Koel-TTS finds character error rate and speaker-embedding similarity improve together, with the optimum at 2.5: both metrics measure adherence to a conditioning input being amplified, and much of the "diversity" being burned is the unconditioned failure mode, hallucinated words and drifting voice. The cost surfaces where the paper notes it, as reduced variation at higher scales, not in the adherence metrics.\u003C/p>\n\u003Ch2 id=\"the-unconditional-branch-defines-what-you-extrapolate-away-from\">The unconditional branch defines what you extrapolate away from\u003C/h2>\n\u003Cp>What the second branch conditions on is a design choice, and systems diverge on it. Koel-TTS drops text and speaker context together, so guidance amplifies both at once. VibeVoice's report describes no trained null token: its diffusion head's unconditional branch conditions on the hidden state of the start-of-sequence token, a minimally conditioned input the model computes anyway. SemaVoice drops the LLM hidden state but keeps the previously generated acoustic patch in the unconditional branch, so patch-to-patch continuity stays unguided and only the semantic conditioning is sharpened. The negative-prompt field in image generators is this same equation with the null condition replaced by an unwanted description, turning "away from what is typical" into "away from what is specified".\u003C/p>\n\u003Cp>Two refinements push the lever further. InstructPix2Pix decomposes the condition and runs two differences with two scales, one amplifying agreement with the input image and one with the edit instruction, the practical alternative to dropping all conditions jointly the way Koel-TTS does. Autoguidance (Karras et al. 2024) replaces the unconditional branch with a smaller, less-trained version of the same model, so the extrapolation points away from that weaker model's errors rather than away from the condition's absence, and recovers fidelity without the diversity collapse. In every case the guided sample is defined as much by the negative branch as by the positive one.\u003C/p>\n\u003Ch2 id=\"tuned-scales-do-not-transfer-across-systems-tasks-or-notation-conventions\">Tuned scales do not transfer across systems, tasks, or notation conventions\u003C/h2>\n\u003Cp>Speech systems that all cite the same CFG paper (we locate these architectures in our \u003Ca href=\"/blog/tts-models-a-map-of-the-modern-speech-stack/\">map of the modern speech stack\u003C/a>) ship guidance settings spanning a factor of three, at different denoising step counts:\u003C/p>\n\u003Cdiv class=\"blog-table\" role=\"region\" aria-label=\"Table 1\" tabindex=\"0\">\u003Ctable>\n\u003Cthead>\n\u003Ctr>\n\u003Cth>System\u003C/th>\n\u003Cth>CFG site\u003C/th>\n\u003Cth>Published setting\u003C/th>\n\u003Cth>In s-convention (s = 1 is off)\u003C/th>\n\u003Cth>Head denoising steps\u003C/th>\n\u003C/tr>\n\u003C/thead>\n\u003Ctbody>\n\u003Ctr>\n\u003Ctd>VibeVoice\u003C/td>\n\u003Ctd>per-token diffusion head\u003C/td>\n\u003Ctd>1.3\u003C/td>\n\u003Ctd>1.3\u003C/td>\n\u003Ctd>10\u003C/td>\n\u003C/tr>\n\u003Ctr>\n\u003Ctd>Koel-TTS\u003C/td>\n\u003Ctd>AR token logits\u003C/td>\n\u003Ctd>gamma = 2.5\u003C/td>\n\u003Ctd>2.5\u003C/td>\n\u003Ctd>none (token model)\u003C/td>\n\u003C/tr>\n\u003Ctr>\n\u003Ctd>SemaVoice\u003C/td>\n\u003Ctd>per-patch diffusion head\u003C/td>\n\u003Ctd>w = 2.5\u003C/td>\n\u003Ctd>3.5\u003C/td>\n\u003Ctd>not stated\u003C/td>\n\u003C/tr>\n\u003Ctr>\n\u003Ctd>LatentLM (TTS)\u003C/td>\n\u003Ctd>per-token diffusion head\u003C/td>\n\u003Ctd>4\u003C/td>\n\u003Ctd>4\u003C/td>\n\u003Ctd>5\u003C/td>\n\u003C/tr>\n\u003Ctr>\n\u003Ctd>LatentLM (ImageNet)\u003C/td>\n\u003Ctd>per-token diffusion head\u003C/td>\n\u003Ctd>1.65\u003C/td>\n\u003Ctd>1.65\u003C/td>\n\u003Ctd>not stated\u003C/td>\n\u003C/tr>\n\u003C/tbody>\n\u003C/table>\n\u003C/div>\u003Cp>Source: VibeVoice Sec. 3.2 and Fig. 3; Koel-TTS Secs. 2.5 and 3.4; SemaVoice Sec. 3.2.3 (which uses the (1+w) convention, hence the conversion); LatentLM Secs. 3.1 and 3.3.6.\u003C/p>\n\u003Cp>LatentLM is one architecture family, and it needs 1.65 for class-conditional ImageNet but 4 for TTS: neither the architecture nor the codebase pins the value, the task does. And SemaVoice's published 2.5 is 3.5 in the convention the other rows use, because its paper writes the Ho-Salimans (1+w) form; copying a guidance number without its equation silently changes the strength by one full unit.\u003C/p>\n\u003Cp>The spread follows from the exponent reading: the scale multiplies the implicit classifier's log-likelihood ratio, and the magnitude of that ratio depends on how strongly the conditioning pins the output (a transcript plus a speaker prompt constrains speech far harder than a class label constrains an image), on what the unconditional branch conditions on, and on the sampler's step count. The guidance scale is a coefficient on a system-specific signal, not an absolute strength in shared units.\u003C/p>\n\u003Cp>Published sweeps differ in shape, not just in optimum:\u003C/p>\n\u003Cfigure class=\"blog-figure blog-figure--image\">\u003Cimg src=\"/media/blog/classifier-free-guidance/latentlm-figure-10.webp\" alt=\"LatentLM ablation of CFG scale and sampling steps for zero-shot TTS\" width=\"1120\" height=\"458\" loading=\"lazy\" decoding=\"async\">\u003C/figure>\n\n\u003Cp>LatentLM paper, Figure 10, p. 14: intelligibility (1 - WER-C, where WER-C is word error rate from a Conformer-Transducer ASR transcription) and speaker similarity over CFG scales 1 to 8 and over diffusion sampling steps. The jump from scale 1 (guidance off) to 2 dwarfs everything after it, and the optimum at 4 sits on a broad plateau.\u003C/p>\n\u003Cfigure class=\"blog-figure blog-figure--image\">\u003Cimg src=\"/media/blog/classifier-free-guidance/vibevoice-figure-3.webp\" alt=\"VibeVoice ablation heatmaps of CFG scale and denoising steps on WER and speaker similarity\" width=\"1114\" height=\"436\" loading=\"lazy\" decoding=\"async\">\u003C/figure>\n\n\u003Cp>VibeVoice paper, Figure 3, p. 8: word error rate and SIM-O (speaker-embedding similarity to the reference audio) over CFG scales 1.25 to 1.35 and denoising steps 5 to 35. Within that narrow scale window, WER still swings from 3.83 to 1.55 depending on the scale-step pair, and the WER optimum (scale 1.25 at 10 steps) disagrees with the similarity optimum (5 steps), so the shipped setting of 1.3 at 10 steps is a compromise.\u003C/p>\n\u003Cp>The operational consequence: treat the guidance scale and step count as one jointly swept pair per system, and when reporting the result, publish the combination equation next to the number. A scale without its equation is not a reproducible setting.\u003C/p>\n\u003Ch2 id=\"sources\">Sources\u003C/h2>\n\u003Cul>\n\u003Cli>\u003Ca href=\"https://arxiv.org/abs/2207.12598\">Ho & Salimans, Classifier-Free Diffusion Guidance (arXiv 2207.12598)\u003C/a>: condition-dropout training and p_uncond ablation (Table 1), the guided combination and w convention (Eq. 6), the implicit-classifier derivation and the not-a-real-gradient caveat, the two-evaluation cost, and the ImageNet 64x64 FID/IS sweep.\u003C/li>\n\u003Cli>\u003Ca href=\"https://arxiv.org/abs/2105.05233\">Dhariwal & Nichol, Diffusion Models Beat GANs on Image Synthesis (arXiv 2105.05233)\u003C/a>: classifier guidance mechanics, noise-aware classifier requirement, gradient scaling as the exponent on p(c|x), the s = 1 failure on an unconditional model, and the precision/recall trade (Table 4).\u003C/li>\n\u003Cli>\u003Ca href=\"https://arxiv.org/abs/2205.11487\">Saharia et al., Imagen (arXiv 2205.11487)\u003C/a>: high-guidance train-test mismatch, out-of-range clean-image estimates, saturation, and dynamic thresholding.\u003C/li>\n\u003Cli>\u003Ca href=\"https://arxiv.org/abs/2408.09000\">Bradley & Nakkiran, Classifier-Free Guidance is a Predictor-Corrector (arXiv 2408.09000)\u003C/a>: CFG does not sample the tilted distribution; supports the exact-sampling caveat.\u003C/li>\n\u003Cli>\u003Ca href=\"https://arxiv.org/abs/2502.05236\">Koel-TTS (arXiv 2502.05236)\u003C/a>: logit-space CFG for an AR TTS model (Sec. 2.5), 10% joint dropout, the gamma sweep and optimum (Sec. 3.4, Fig. 4), and the doubled effective batch.\u003C/li>\n\u003Cli>\u003Ca href=\"https://arxiv.org/abs/2412.08635\">LatentLM, Multimodal Latent Language Modeling with Next-Token Diffusion (arXiv 2412.08635)\u003C/a>: TTS CFG and step ablations (Fig. 10), image-generation guidance settings (Sec. 3.1), WER-C metric definition (Sec. 3.3).\u003C/li>\n\u003Cli>\u003Ca href=\"https://openreview.net/forum?id=FihSkzyxdv\">VibeVoice (OpenReview, ICLR 2026)\u003C/a>: diffusion-head CFG with the start-of-sequence unconditional branch (Eq. 6), shipped settings (Sec. 3.2), and the scale-step heatmaps (Fig. 3). The shorter arXiv technical report (2508.19205) omits the heatmap figure.\u003C/li>\n\u003Cli>\u003Ca href=\"https://arxiv.org/abs/2605.16964\">SemaVoice (arXiv 2605.16964)\u003C/a>: LLM-state dropout with retained previous patch, (1+w) convention at w = 2.5 (Sec. 3.2.3).\u003C/li>\n\u003Cli>\u003Ca href=\"https://arxiv.org/abs/2306.17806\">Sanchez et al., Stay on Topic with Classifier-Free Guidance (arXiv 2306.17806)\u003C/a>: per-step logit CFG in pure text LLMs.\u003C/li>\n\u003Cli>\u003Ca href=\"https://arxiv.org/abs/2211.09800\">Brooks et al., InstructPix2Pix (arXiv 2211.09800)\u003C/a>: two-condition CFG with independent guidance scales (Eq. 3).\u003C/li>\n\u003Cli>\u003Ca href=\"https://arxiv.org/abs/2406.02507\">Karras et al., Guiding a Diffusion Model with a Bad Version of Itself (arXiv 2406.02507)\u003C/a>: autoguidance; replacing the unconditional branch with a degraded model.\u003C/li>\n\u003Cli>\u003Ca href=\"https://arxiv.org/abs/2210.03142\">Meng et al., On Distillation of Guided Diffusion Models (arXiv 2210.03142)\u003C/a>: removing the two-pass cost by distilling the guided pair.\u003C/li>\n\u003C/ul>\n\u003Ch2 id=\"related-diffio-posts\">Related Diffio Posts\u003C/h2>\n\u003Cul>\n\u003Cli>\u003Ca href=\"/blog/tts-models-a-map-of-the-modern-speech-stack/\">TTS Models: A Map of the Modern Speech Stack\u003C/a>\u003C/li>\n\u003C/ul>\n",false,[25,42,58],{"slug":26,"path":27,"title":28,"description":29,"publishedAt":30,"updatedAt":31,"author":10,"category":11,"tags":32,"featureImage":34,"ogImage":39,"canonical":40,"readingTimeMinutes":41},"tts-models-a-map-of-the-modern-speech-stack","/blog/tts-models-a-map-of-the-modern-speech-stack/","TTS Models: Sampled Acoustic Representations Set the Generator Constraints","A map of modern text-to-speech systems by the representation their generators sample: codec tokens, mel frames, semantic-acoustic cascades, continuous latents, and hybrid frames.","2026-06-13T15:43:43Z","2026-06-13T16:07:41Z",[33,13],"TTS",{"src":35,"alt":36,"widthPixels":37,"heightPixels":38},"/media/blog/tts-models-a-map-of-the-modern-speech-stack/voxtral-figure-2.webp","Voxtral TTS combines autoregressive semantic generation with a flow-matching acoustic token head. The figure is Figure 2, page 3, from the Voxtral TTS technical report.",1006,833,{"src":35,"alt":36,"widthPixels":37,"heightPixels":38},"https://diffio.ai/blog/tts-models-a-map-of-the-modern-speech-stack/",9,{"slug":43,"path":44,"title":45,"description":46,"publishedAt":47,"updatedAt":31,"author":10,"category":11,"tags":48,"featureImage":50,"ogImage":55,"canonical":56,"readingTimeMinutes":57},"training-latent-audio-encoders","/blog/training-latent-audio-encoders/","Latent Audio Encoders: Reconstruction Sets the Ceiling, Not the Ranking","Why an audio autoencoder's reconstruction quality sets the ceiling for the generator trained on its latents, but does not decide which latent space ranks best.","2026-06-13T00:12:05Z",[49,13],"Audio",{"src":51,"alt":52,"widthPixels":53,"heightPixels":54},"/media/blog/training-latent-audio-encoders/longcat-figure-3.webp","LongCat-AudioDiT Figure 3: four panels plotting VAE reconstruction metrics and TTS metrics against latent dimension.",1166,724,{"src":51,"alt":52,"widthPixels":53,"heightPixels":54},"https://diffio.ai/blog/training-latent-audio-encoders/",13,{"slug":59,"path":60,"title":61,"description":62,"publishedAt":63,"updatedAt":64,"author":10,"category":11,"tags":65,"featureImage":66,"ogImage":71,"canonical":72,"readingTimeMinutes":73},"same-audio-encoder","/blog/same-audio-encoder/","SAME Makes Generation Difficulty an Autoencoder Training Loss","Stability AI's SAME autoencoder, the latent space Stable Audio 3 generates in, makes generation difficulty part of the autoencoder's training loss.","2026-06-12T23:15:08Z","2026-06-13T16:07:42Z",[49,13],{"src":67,"alt":68,"widthPixels":69,"heightPixels":70},"/media/blog/same-audio-encoder/figure-1.webp","SAME architecture: patching, encoder TRB, soft-normalization bottleneck, decoder TRB, unpatching, with reconstruction, adversarial, and auxiliary loss families.",728,264,{"src":67,"alt":68,"widthPixels":69,"heightPixels":70},"https://diffio.ai/blog/same-audio-encoder/",10,1793728206636]