[{"data":1,"prerenderedAt":67},["ShallowReactive",2],{"blog-post-efficient-model-deployment-modal":3},{"slug":4,"path":5,"title":6,"description":7,"publishedAt":8,"updatedAt":9,"author":10,"category":11,"tags":12,"featureImage":16,"ogImage":16,"canonical":17,"readingTimeMinutes":18,"bodyHtml":19,"showFeatureImage":20,"relatedPosts":21},"efficient-model-deployment-modal","/blog/efficient-model-deployment-modal/","Smaller Model State Did Not Establish Faster Cold Requests for Diffio on Modal","What Diffio’s Modal experiments revealed about GPU snapshots, initialization, quantization, CUDA graphs, transcription dependencies, and cold-request latency.","2026-09-09T00:42:46Z","2026-10-22T23:00:32Z","Nathan Harmon","infrastructure",[13,14,15],"ML","Inference","Modal",null,"https://diffio.ai/blog/efficient-model-deployment-modal/",14,"\u003Cp>Diffio’s deployment experiments ran from July through September 2026 on our \u003Ca href=\"/\">audio-restoration service\u003C/a>. Diffio 2.0 uses ahead-of-time compiled model packages; Diffio 3.5 also has an autoregressive language-model path and a diffusion component. The general \u003Ca href=\"/blog/vibevoice-next-token-diffusion/\">next-token diffusion architecture\u003C/a> explains why the autoregressive loop and diffusion head offer different optimization boundaries.\u003C/p>\n\u003Cdiv class=\"blog-table\" role=\"region\" aria-label=\"Table 1\" tabindex=\"0\">\u003Ctable>\n\u003Cthead>\n\u003Ctr>\n\u003Cth>September research environment\u003C/th>\n\u003Cth>Configuration\u003C/th>\n\u003C/tr>\n\u003C/thead>\n\u003Ctbody>\n\u003Ctr>\n\u003Ctd>GPU\u003C/td>\n\u003Ctd>RTX PRO 6000 Blackwell\u003C/td>\n\u003C/tr>\n\u003Ctr>\n\u003Ctd>Runtime\u003C/td>\n\u003Ctd>PyTorch 2.9, CUDA 12.8\u003C/td>\n\u003C/tr>\n\u003C/tbody>\n\u003C/table>\n\u003C/div>\u003Ch2 id=\"a-cold-request-includes-work-that-a-gpu-profiler-cannot-see\">A cold request includes work that a GPU profiler cannot see\u003C/h2>\n\u003Cp>Our restoration workflow includes source preparation, transcription, dispatch, platform waiting, worker startup, model execution, publication, and a completion join that waits for all required outputs. Some stages overlap, so adding their elapsed times can double-count the same wall-clock interval.\u003C/p>\n\u003Cdiv class=\"blog-table\" role=\"region\" aria-label=\"Table 2\" tabindex=\"0\">\u003Ctable>\n\u003Cthead>\n\u003Ctr>\n\u003Cth>Measurement\u003C/th>\n\u003Cth>What it establishes\u003C/th>\n\u003C/tr>\n\u003C/thead>\n\u003Ctbody>\n\u003Ctr>\n\u003Ctd>Snapshot creation\u003C/td>\n\u003Ctd>The cost of building a reusable initialized worker state.\u003C/td>\n\u003C/tr>\n\u003Ctr>\n\u003Ctd>Restored cold request\u003C/td>\n\u003Ctd>The first request handled by a newly restored worker.\u003C/td>\n\u003C/tr>\n\u003Ctr>\n\u003Ctd>Same-process warm request\u003C/td>\n\u003Ctd>Reuse of a worker that has already handled a request.\u003C/td>\n\u003C/tr>\n\u003Ctr>\n\u003Ctd>First playable audio\u003C/td>\n\u003Ctd>In the final study, the client downloaded and decoded the first AAC segment; this did not measure audible browser playback.\u003C/td>\n\u003C/tr>\n\u003Ctr>\n\u003Ctd>Complete request\u003C/td>\n\u003Ctd>The instrumented client observed the complete workflow, including required transcript and artifact publication.\u003C/td>\n\u003C/tr>\n\u003Ctr>\n\u003Ctd>GPU kernel interval union\u003C/td>\n\u003Ctd>The elapsed union of GPU kernel intervals, counting overlapping kernels once. It excludes CPU and remote waiting.\u003C/td>\n\u003C/tr>\n\u003C/tbody>\n\u003C/table>\n\u003C/div>\u003Cp>A restore flag alone cannot classify a request as cold: the second request in that process still inherits a worker that originally restored from a snapshot. We required distinct worker identities, evidence that the previous worker was gone, and platform restore events. Snapshot-creation attempts remained in the attempt journal but outside the restored cohort.\u003C/p>\n\u003Cp>The boundary labeled “physical start” also needed verification. The final harness preferred Modal’s restore-start system log; it was not invariably an operating-system process-creation timestamp. Its interval to the start of application finalization is therefore a restore-to-finalization-start proxy, not an isolated measurement of checkpoint copying. Submission to that event can include placement, image preparation, retries, and other platform work.\u003C/p>\n\u003Ch2 id=\"gpu-snapshots-avoided-package-loading-and-scoreq-initialization\">GPU snapshots avoided package loading and ScoreQ initialization\u003C/h2>\n\u003Cp>GPU snapshots become valuable when restoring initialized state costs less than repeating the computation that constructed it. In the July probe, application finalization after a CPU snapshot took about 22.5 seconds, compared with about 0.4 seconds after a GPU-plus-CPU snapshot. That small historical comparison measured finalization, not total request latency, and used separate groups of six CPU-snapshot and eight GPU-snapshot observations.\u003C/p>\n\u003Cp>Our early 2.0 worker still loaded its compiled packages on the first restored request, despite preparing their files during image construction. That first control trace spent 6.97 seconds loading the packages.\u003C/p>\n\u003Cp>ScoreQ, the learned audio-quality scorer used by the service, had an even more expensive first-use path. Its first production-shape ONNX inference took roughly 35–36 seconds, overlapping word-to-audio alignment enough to make that stage look slow. Warming alignment alone did not remove the delay. Exercising ScoreQ’s actual input shape before capture moved its convolution-search and initialization work out of the first restored request.\u003C/p>\n\u003Cp>After restoration, the worker created fresh credentials, network clients, runtime threads, and random identity. Retaining reusable computation while refreshing externally meaningful state avoided reusing captured connections or a creator’s random stream.\u003C/p>\n\u003Cp>\u003Ca href=\"https://modal.com/docs/guide/memory-snapshots\">Modal’s current snapshot documentation\u003C/a> places pre-capture work in \u003Ccode>@modal.enter(snap=True)\u003C/code> and describes GPU snapshots as an alpha feature. Snapshots use the same distributed storage system as Images and Volumes, so storage-bandwidth-limited weight loading may gain little.\u003C/p>\n\u003Cp>Snapshot creation still has a cost. Our candidate spent longer preparing its initial reusable state, and creation can recur when code, configuration, or platform conditions change.\u003C/p>\n\u003Ch2 id=\"the-earlier-bundle-lowered-median-completion-but-delayed-the-3-5-preview\">The earlier bundle lowered median completion but delayed the 3.5 preview\u003C/h2>\n\u003Cp>The September combined optimization changed several components together: compiled-package preload, ScoreQ warmup, redundant model copies, allocator state, CPU memory reservation, and the 3.5 diffusion execution path. Its 60-second-input comparison used ten matched restored pairs per model. This bundled design does not isolate any individual change’s contribution.\u003C/p>\n\u003Cdiv class=\"blog-table\" role=\"region\" aria-label=\"Table 3\" tabindex=\"0\">\u003Ctable>\n\u003Cthead>\n\u003Ctr>\n\u003Cth>Model\u003C/th>\n\u003Cth style=\"text-align:right\">Median complete request, control → bundle\u003C/th>\n\u003Cth style=\"text-align:right\">Mean complete request, control → bundle\u003C/th>\n\u003Cth style=\"text-align:right\">Faster candidate pairs\u003C/th>\n\u003C/tr>\n\u003C/thead>\n\u003Ctbody>\n\u003Ctr>\n\u003Ctd>Diffio 2.0\u003C/td>\n\u003Ctd style=\"text-align:right\">61.934 → 33.093 s\u003C/td>\n\u003Ctd style=\"text-align:right\">64.100 → 33.124 s\u003C/td>\n\u003Ctd style=\"text-align:right\">10/10\u003C/td>\n\u003C/tr>\n\u003Ctr>\n\u003Ctd>Diffio 3.5\u003C/td>\n\u003Ctd style=\"text-align:right\">85.350 → 45.604 s\u003C/td>\n\u003Ctd style=\"text-align:right\">85.232 → 85.243 s\u003C/td>\n\u003Ctd style=\"text-align:right\">7/10\u003C/td>\n\u003C/tr>\n\u003C/tbody>\n\u003C/table>\n\u003C/div>\u003Cp>The 3.5 mean stayed flat despite its lower median because capacity-related outliers remained. Median time to an advertised HLS (HTTP Live Streaming) playlist or segment increased from 35.892 to 46.096 seconds; this measured advertisement, not successful download and decoding. The preview subset contained eight observations per arm. The preview and completion medians use different subsets, so their relative ordering is not a per-request ordering.\u003C/p>\n\u003Ch2 id=\"repository-pruning-did-not-measure-snapshot-shrinkage\">Repository pruning did not measure snapshot shrinkage\u003C/h2>\n\u003Cp>The model repository was our versioned release source, not a download repeated on every restored request. We published immutable Hugging Face revisions, verified their file hashes, and synchronized them into append-only Modal Volume directories. Image construction prepared compatible executables and caches. Restored trials ran with Hugging Face downloads disabled.\u003C/p>\n\u003Cp>Removing unused legacy GPU-architecture packages saved about 4.04 GB in the 2.0 repository, but those files were never active state on the tested GPU. Pruning an unused semantic tokenizer and connector from 3.5 removed about 694 MB of parameters while leaving retained tensors unchanged. These studies measured repository, compiled-constant, or device-memory reductions; they did not measure serialized snapshot size.\u003C/p>\n\u003Cp>Immutable revisions also protect restores and rollbacks. Hugging Face’s \u003Ca href=\"https://huggingface.co/docs/huggingface_hub/main/guides/download\">download API\u003C/a> supports explicit commit revisions. Modal documents that changing a Volume does not invalidate its snapshots, and deleting files needed by a restore can break it. We therefore preserved old revision directories while live or rollback deployments could still depend on them.\u003C/p>\n\u003Cp>The release identity combined source, model revision, image, and configuration. Development deployment and fixture checks preceded any promotion decision.\u003C/p>\n\u003Ch2 id=\"compiled-packages-failed-after-the-server-passed-health-checks\">Compiled packages failed after the server passed health checks\u003C/h2>\n\u003Cp>Ahead-of-time compilation, which produces executables before serving, adds compatibility requirements beyond tensor shapes. Our packages targeted SM120, NVIDIA’s architecture identifier for the tested Blackwell GPU. A fallback GPU needed its own compatible package; an existing package for another NVIDIA architecture was not evidence of compatibility.\u003C/p>\n\u003Cp>Three packaging failures only became visible when we exercised the real model:\u003C/p>\n\u003Cul>\n\u003Cli>\u003Cstrong>System-library compatibility.\u003C/strong> The compiled shared libraries required \u003Ccode>GLIBC_2.38\u003C/code>; the older image supplied glibc 2.36.\u003C/li>\n\u003Cli>\u003Cstrong>Executable-stack flags.\u003C/strong> A newer image’s hardened loader rejected flags on compiled wrapper libraries. We cleared them in image-local copies, rebuilt and checked the archives, and left pinned source artifacts intact.\u003C/li>\n\u003Cli>\u003Cstrong>Execution-provider selection.\u003C/strong> Dependencies could install both CPU and GPU ONNX Runtime distributions. The final build installed the pinned GPU distribution last and required \u003Ccode>CUDAExecutionProvider\u003C/code>, the backend that runs supported ONNX operations on CUDA.\u003C/li>\n\u003C/ul>\n\u003Cp>The canary therefore loaded every required package and ran real audio. A successful import, HTTP health check, or tensor allocation could miss all three failures.\u003C/p>\n\u003Ch2 id=\"cuda-graphs-reduced-diffusion-runtime-broader-compilation-added-first-use-work\">CUDA graphs reduced diffusion runtime; broader compilation added first-use work\u003C/h2>\n\u003Cp>A CUDA graph records a GPU execution sequence for replay with stable memory addresses, reducing repeated launch overhead. We captured the deterministic 3.5 diffusion trajectory, generated its initial random noise outside capture, and kept graph inputs private. Moving decoder cache indices to the CPU also avoided repeated GPU-to-CPU lookups.\u003C/p>\n\u003Cp>The integrated service exposed a failure that isolated graph code missed: alignment performed CUDA work on another thread during global capture. Thread-local capture restricted unsafe-operation error checks to the capturing thread, resolving this observed capture failure while preserving a single denoising scheduler and private buffers. It was validated as a complete configuration; changing the capture mode alone does not make shared buffers safe. \u003Ca href=\"https://docs.pytorch.org/docs/2.9/generated/torch.cuda.graph.html\">PyTorch’s capture-mode documentation\u003C/a> describes the different cross-thread error checks.\u003C/p>\n\u003Cp>The selected graph matched baseline decoded PCM, the waveform’s numerical samples, across all 232 evaluation clips with frozen transcripts. Comparing WAV file bytes would have produced false differences because metadata timestamps changed. For a minute of input, median Diffio 3.5 local-runtime request time fell from 16.445 to 12.916 seconds. This included local transcription, alignment, and generation, and excluded the deployed storage, scoring, and gateway path.\u003C/p>\n\u003Cp>Broader compilation did not produce the same result. Compiling the whole language-model path introduced roughly 117 seconds of first-use work and 62 seconds on a new shape. The tested whole-Whisper compilation also added substantial compilation overhead on first use and on new input shapes. Compilation cost alone would not reject a configuration whose compilation could finish before capture; these tested configurations did not establish a reduction in complete-request latency.\u003C/p>\n\u003Ch2 id=\"quantized-storage-successful-reload-and-acceptable-audio-are-separate-gates\">Quantized storage, successful reload, and acceptable audio are separate gates\u003C/h2>\n\u003Cp>Post-training quantization replaces trained weights with lower-precision representations, optionally using calibration recordings to choose the mapping, without fine-tuning the model.\u003C/p>\n\u003Cp>Our INT4 export initially failed to preserve packed storage. Tensor metadata could describe a low-bit object while copying into an ordinary floating-point destination reconstructed a larger representation. We inspected the physical packed backing storage and compiled constants, then reloaded each candidate in a fresh process. Earlier serialized candidates that reconstructed the expected tensor class had still failed output parity.\u003C/p>\n\u003Cp>Even correct packing did not establish an audio-quality pass. We compared frozen inputs and transcript conditioning, first against the current deployed model and then against the original BF16 model to measure cumulative change. The full evaluation contained 232 clips; the final acoustic-INT4 candidate also passed a fresh 32-recording confirmation set.\u003C/p>\n\u003Cp>Quality decisions used UTMOS and WVMOS, learned predictors of perceived speech quality, averaged into a dataset-balanced blend, together with ScoreQ as a separate learned score. The provisional limits allowed at most a 0.02 drop in each aggregate and a 0.05 drop on any dataset. An improved aggregate could not override a failed dataset threshold. Those automated gates did not establish perceptual identity; no human listening verdict was recorded. Recognition-based checks likewise could flag numerical-word disagreements but could not resolve them without checking what was actually spoken.\u003C/p>\n\u003Cp>MLP refers to feed-forward multilayer perceptron sublayers. HQQ, GPTQ, and AWQ-lite name the tested quantization recipes; NVFP4 is a four-bit floating-point format.\u003C/p>\n\u003Cdiv class=\"blog-table\" role=\"region\" aria-label=\"Table 4\" tabindex=\"0\">\u003Ctable>\n\u003Cthead>\n\u003Ctr>\n\u003Cth>Tested family\u003C/th>\n\u003Cth>Result that determined the decision\u003C/th>\n\u003C/tr>\n\u003C/thead>\n\u003Ctbody>\n\u003Ctr>\n\u003Ctd>HQQ INT4 language projections, including smaller-group and MLP-only variants\u003C/td>\n\u003Ctd>Candidates that passed the initial small evaluation set failed broader quality evaluation.\u003C/td>\n\u003C/tr>\n\u003Ctr>\n\u003Ctd>Calibrated NVFP4 with GPTQ and INT4 with AWQ-lite\u003C/td>\n\u003Ctd>Successful packing and calibration still failed the tested audio-quality gates.\u003C/td>\n\u003C/tr>\n\u003Ctr>\n\u003Ctd>Selective INT8 language MLPs\u003C/td>\n\u003Ctd>Lost 0.283 blend points on SOVA despite a favorable aggregate.\u003C/td>\n\u003C/tr>\n\u003Ctr>\n\u003Ctd>Other tested lower-precision acoustic or codec variants\u003C/td>\n\u003Ctd>Tested variants failed quality or execution checks.\u003C/td>\n\u003C/tr>\n\u003Ctr>\n\u003Ctd>Acoustic MLPs changed from INT8 to INT4, retaining INT8 attention\u003C/td>\n\u003Ctd>Advanced to restored cold requests.\u003C/td>\n\u003C/tr>\n\u003C/tbody>\n\u003C/table>\n\u003C/div>\u003Cp>Positive deltas below favor the candidate.\u003C/p>\n\u003Cdiv class=\"blog-table\" role=\"region\" aria-label=\"Table 5\" tabindex=\"0\">\u003Ctable>\n\u003Cthead>\n\u003Ctr>\n\u003Cth>Full evaluation comparison\u003C/th>\n\u003Cth style=\"text-align:right\">Blend delta\u003C/th>\n\u003Cth style=\"text-align:right\">ScoreQ delta\u003C/th>\n\u003C/tr>\n\u003C/thead>\n\u003Ctbody>\n\u003Ctr>\n\u003Ctd>Acoustic INT4 vs current deployed model\u003C/td>\n\u003Ctd style=\"text-align:right\">+0.01678\u003C/td>\n\u003Ctd style=\"text-align:right\">+0.01284\u003C/td>\n\u003C/tr>\n\u003Ctr>\n\u003Ctd>Acoustic INT4 vs original BF16 model\u003C/td>\n\u003Ctd style=\"text-align:right\">+0.00514\u003C/td>\n\u003Ctd style=\"text-align:right\">+0.00254\u003C/td>\n\u003C/tr>\n\u003Ctr>\n\u003Ctd>Worst dataset change vs original BF16\u003C/td>\n\u003Ctd style=\"text-align:right\">−0.03587\u003C/td>\n\u003Ctd style=\"text-align:right\">−0.02554\u003C/td>\n\u003C/tr>\n\u003C/tbody>\n\u003C/table>\n\u003C/div>\u003Cp>These are component- and recording-specific results, not a ranking of quantization methods.\u003C/p>\n\u003Ch2 id=\"the-final-smaller-state-candidates-did-not-meet-the-cold-latency-gate\">The final smaller-state candidates did not meet the cold-latency gate\u003C/h2>\n\u003Cp>The final study compared candidates against controls with the earlier deployment improvements already in place. Acceptance required the specified median completion improvement, a paired interval supporting improvement, passing quality, and no clear first-playable regression.\u003C/p>\n\u003Cdiv class=\"blog-table\" role=\"region\" aria-label=\"Table 6\" tabindex=\"0\">\u003Ctable>\n\u003Cthead>\n\u003Ctr>\n\u003Cth>Final cold-study protocol\u003C/th>\n\u003Cth>Requirement\u003C/th>\n\u003C/tr>\n\u003C/thead>\n\u003Ctbody>\n\u003Ctr>\n\u003Ctd>Input duration\u003C/td>\n\u003Ctd>60 seconds\u003C/td>\n\u003C/tr>\n\u003Ctr>\n\u003Ctd>Practical median completion target\u003C/td>\n\u003Ctd>At least 1 second and 5% faster\u003C/td>\n\u003C/tr>\n\u003Ctr>\n\u003Ctd>Planned main comparison\u003C/td>\n\u003Ctd>10 matched pairs per candidate\u003C/td>\n\u003C/tr>\n\u003C/tbody>\n\u003C/table>\n\u003C/div>\u003Cp>Pairs used the same frozen input and workflow settings, interleaving control and candidate and reversing their order on alternating pairs. The allocator candidate asked ONNX Runtime to release unused ScoreQ arena memory while retaining its session and caches. The acoustic candidate quantized the original BF16 weights directly, avoiding requantization of the lossy INT8 representation.\u003C/p>\n\u003Cdiv class=\"blog-table\" role=\"region\" aria-label=\"Table 7\" tabindex=\"0\">\u003Ctable>\n\u003Cthead>\n\u003Ctr>\n\u003Cth>Candidate\u003C/th>\n\u003Cth style=\"text-align:right\">Measured reduction\u003C/th>\n\u003Cth style=\"text-align:right\">Matched cold pairs\u003C/th>\n\u003Cth style=\"text-align:right\">Median paired completion change, with 95% bootstrap interval\u003C/th>\n\u003C/tr>\n\u003C/thead>\n\u003Ctbody>\n\u003Ctr>\n\u003Ctd>ScoreQ allocator shrink, 3.5\u003C/td>\n\u003Ctd style=\"text-align:right\">512 MiB device memory in the local probe\u003C/td>\n\u003Ctd style=\"text-align:right\">10\u003C/td>\n\u003Ctd style=\"text-align:right\">−6.20 s [−63.00, +77.84]\u003C/td>\n\u003C/tr>\n\u003Ctr>\n\u003Ctd>Acoustic INT4 MLPs, 2.0\u003C/td>\n\u003Ctd style=\"text-align:right\">130,170,880 compiled constant bytes\u003C/td>\n\u003Ctd style=\"text-align:right\">7\u003C/td>\n\u003Ctd style=\"text-align:right\">+5.74 s [+4.47, +66.32]\u003C/td>\n\u003C/tr>\n\u003C/tbody>\n\u003C/table>\n\u003C/div>\u003Cp>Changes are candidate minus matched control; negative favors the candidate. The intervals describe the paired median and come from resampling whole matched pairs. The acoustic candidate’s interval describes a sample stopped under a futility rule introduced during collection; it does not have guaranteed fixed-sample coverage. Creator counters repeated after a restore are frozen metadata rather than new memory observations.\u003C/p>\n\u003Cp>The allocator result was inconclusive. Its mean completion change was slower by 11.66 seconds, and its median restore-to-finalization-start proxy changed by only −0.096 seconds. The favorable completion median did not establish that less retained memory restored faster.\u003C/p>\n\u003Cp>Acoustic-INT4 completion was slower in six of seven pairs, and median first-playable time also worsened. We stopped further dispatches when the planned ten-pair median could no longer improve, even with arbitrarily good remaining pairs. The best possible median was still +4.54 seconds. One request already in flight finished and remained unmatched, outside the paired statistics.\u003C/p>\n\u003Cp>Neither candidate qualified for the planned additional cold tests at other input durations or a cold-latency release.\u003C/p>\n\u003Cp>The acoustic candidate’s local warm slowdown was much smaller than its complete request difference. The alternating-process RunPod diagnostic measured model execution, excluding transcription, ScoreQ, Modal restoration, transport, and output-file writing.\u003C/p>\n\u003Cdiv class=\"blog-table\" role=\"region\" aria-label=\"Table 8\" tabindex=\"0\">\u003Ctable>\n\u003Cthead>\n\u003Ctr>\n\u003Cth>Input duration\u003C/th>\n\u003Cth style=\"text-align:right\">Current model\u003C/th>\n\u003Cth style=\"text-align:right\">Acoustic INT4\u003C/th>\n\u003Cth style=\"text-align:right\">Added model time\u003C/th>\n\u003C/tr>\n\u003C/thead>\n\u003Ctbody>\n\u003Ctr>\n\u003Ctd>3 s\u003C/td>\n\u003Ctd style=\"text-align:right\">2.126 s\u003C/td>\n\u003Ctd style=\"text-align:right\">2.334 s\u003C/td>\n\u003Ctd style=\"text-align:right\">0.208 s\u003C/td>\n\u003C/tr>\n\u003Ctr>\n\u003Ctd>30 s\u003C/td>\n\u003Ctd style=\"text-align:right\">2.128 s\u003C/td>\n\u003Ctd style=\"text-align:right\">2.305 s\u003C/td>\n\u003Ctd style=\"text-align:right\">0.177 s\u003C/td>\n\u003C/tr>\n\u003Ctr>\n\u003Ctd>60 s\u003C/td>\n\u003Ctd style=\"text-align:right\">2.131 s\u003C/td>\n\u003Ctd style=\"text-align:right\">2.318 s\u003C/td>\n\u003Ctd style=\"text-align:right\">0.187 s\u003C/td>\n\u003C/tr>\n\u003Ctr>\n\u003Ctd>300 s\u003C/td>\n\u003Ctd style=\"text-align:right\">6.469 s\u003C/td>\n\u003Ctd style=\"text-align:right\">7.034 s\u003C/td>\n\u003Ctd style=\"text-align:right\">0.565 s\u003C/td>\n\u003C/tr>\n\u003C/tbody>\n\u003C/table>\n\u003C/div>\u003Cp>The diagnostic made 48 calls. Excluding the first call per duration in each process left 32 calls, with four observations per table cell.\u003C/p>\n\u003Cp>The compiled model processes fixed 120-second windows, padding shorter inputs. The three shorter fixtures therefore each execute one window; the longest executes three, explaining the nearly flat short-input timings.\u003C/p>\n\u003Cp>We retained the candidate’s artifacts so its lower storage requirement and higher execution time could be reproduced. Attributing the entire cold-request regression to low-bit kernels would conflate the local compute penalty with platform and workflow variability.\u003C/p>\n\u003Ch2 id=\"waiting-before-restoration-dominated-the-largest-observed-delay\">Waiting before restoration dominated the largest observed delay\u003C/h2>\n\u003Cp>One allocator-control request took about 845 seconds to complete, with roughly 776 seconds between GPU submission and the explicit restore-start log. Transcription had already completed while Modal showed a pending call with no worker task binding. After the restore event, the observed interval to finalization start was about six seconds.\u003C/p>\n\u003Cp>The timing places most of that delay before the recorded restore event without identifying which platform operation consumed it. The tested state reductions did not establish an improvement in that waiting interval. Both arms experienced long waits, and we retained them in the paired sample.\u003C/p>\n\u003Cp>An earlier region-pinned arm had worse scheduling observations and was stopped. A cheaper GPU fallback offered no measured advantage and introduced a different architecture to support. Reducing the CPU memory reservation was eventually viable for the changed bundle, but an earlier comparison had favored the larger reservation’s scheduling. Initialization peaks, steady memory, placement, and cost were separate constraints.\u003C/p>\n\u003Ch2 id=\"independent-transcription-let-diffio-2-0-release-the-gpu-before-the-job-finished\">Independent transcription let Diffio 2.0 release the GPU before the job finished\u003C/h2>\n\u003Cp>Moving speech recognition off the GPU removed a separate ASR and alignment package, about 1.89 GB per model. ASR means automatic speech recognition; the acoustic Whisper encoders used by restoration still remained necessary. Single-run GPU profiles showed less kernel execution after offload, but paired warm service measurements showed slower completion because the worker waited for remote transcripts.\u003C/p>\n\u003Cdiv class=\"blog-table\" role=\"region\" aria-label=\"Table 9\" tabindex=\"0\">\u003Ctable>\n\u003Cthead>\n\u003Ctr>\n\u003Cth>Approximately 32-second input\u003C/th>\n\u003Cth style=\"text-align:right\">Old warm completion mean\u003C/th>\n\u003Cth style=\"text-align:right\">External-ASR warm completion mean\u003C/th>\n\u003C/tr>\n\u003C/thead>\n\u003Ctbody>\n\u003Ctr>\n\u003Ctd>Diffio 2.0\u003C/td>\n\u003Ctd style=\"text-align:right\">13.628 s\u003C/td>\n\u003Ctd style=\"text-align:right\">29.076 s\u003C/td>\n\u003C/tr>\n\u003Ctr>\n\u003Ctd>Diffio 3.5\u003C/td>\n\u003Ctd style=\"text-align:right\">22.145 s\u003C/td>\n\u003Ctd style=\"text-align:right\">32.777 s\u003C/td>\n\u003C/tr>\n\u003C/tbody>\n\u003C/table>\n\u003C/div>\u003Cp>Each mean uses two subsequent calls in the same process; the kernel profiles were separate single observations, not a performance distribution. Moving work to another service changed which resource performed it, without automatically removing it from the completion dependency graph.\u003C/p>\n\u003Cp>Diffio 2.0 allowed restoration and transcription to execute independently because its restored audio does not require the external transcript. We dispatched audio restoration independently, let it publish and release its GPU invocation, and completed the job on the CPU after both audio and transcript were valid. Tests that deliberately held back transcript-provider responses verified that ordering for short and long inputs, rather than measuring a new latency distribution. Diffio 3.5 still depended on canonical transcript conditioning, so it could not inherit the same independence. An earlier evaluation using external transcripts also failed a dataset quality threshold.\u003C/p>\n\u003Cp>The CPU completion coordinator checked that audio and transcript belonged to the same operation before marking the job successful. It prevented uncertain dispatch from duplicating GPU work, respected cancellation and later transcript edits, and charged usage once. Audio availability and full-job completion became separate states.\u003C/p>\n\u003Cp>CPU preparation could remain slow even on a reused instance. One roughly ten-second extraction/upload interval occurred after that instance had handled 157 earlier requests. It did not support attributing the delay to an empty container’s initialization. Phase telemetry was needed to distinguish media work, storage round trips, and time between stages.\u003C/p>\n\u003Ch2 id=\"retention-and-testing-budgets-depend-on-reuse-and-information-gained\">Retention and testing budgets depend on reuse and information gained\u003C/h2>\n\u003Cp>Keeping a container warm allows reuse if another request arrives before shutdown. In a small production sample of nine events, the shortest observed gap exceeded 22 minutes. That incomplete sample gave no evidence that retaining workers for a few additional minutes would often avoid a restore.\u003C/p>\n\u003Cp>Modal’s \u003Ca href=\"https://modal.com/docs/guide/scale\">autoscaler controls\u003C/a> expose minimum warm containers, an active buffer, and an idle scale-down window.\u003C/p>\n\u003Cdiv class=\"blog-table\" role=\"region\" aria-label=\"Table 10\" tabindex=\"0\">\u003Ctable>\n\u003Cthead>\n\u003Ctr>\n\u003Cth>Retention estimate, September 8, 2026 listed rates\u003C/th>\n\u003Cth>Value\u003C/th>\n\u003C/tr>\n\u003C/thead>\n\u003Ctbody>\n\u003Ctr>\n\u003Ctd>Requested resources\u003C/td>\n\u003Ctd>RTX PRO 6000, 8 physical CPU cores, 32 GiB memory\u003C/td>\n\u003C/tr>\n\u003Ctr>\n\u003Ctd>Combined requested-resource rate\u003C/td>\n\u003Ctd>$0.001018/second\u003C/td>\n\u003C/tr>\n\u003C/tbody>\n\u003C/table>\n\u003C/div>\u003Cp>Keeping that allocation for five extra minutes would cost about $0.31; actual metered usage and billing policy determine the invoice.\u003C/p>\n\u003Cp>We used the already allocated RunPod GPU for calibration, packing, fresh reload, audio quality, profiling, and duration checks. Modal calls were reserved for the behavior that local processes could not reproduce: actual snapshot creation/restoration, platform waiting, and the complete deployed workflow. Interleaved controls, frozen revisions, explicit exclusions, and preserved failures made those paid calls interpretable.\u003C/p>\n\u003Cp>The final three-app Modal billing checkpoint for the cold-quantization study was $4.43. It could lag usage and excluded RunPod, the separate CPU workflow, and external ASR.\u003C/p>\n\u003Ch2 id=\"sources\">Sources\u003C/h2>\n\u003Cul>\n\u003Cli>\u003Ca href=\"/media/blog/efficient-model-deployment-modal/diffio-modal-measurements-reviewed.json\">Diffio deployment measurements\u003C/a>: original study methods and quality gates, anonymized final paired timings, initialization and packaging evidence, the earlier bundled comparison, warm-duration and transcription results, and the cold-study billing checkpoint. The experiments are dated; they are not a benchmark of every Modal configuration.\u003C/li>\n\u003Cli>\u003Ca href=\"https://modal.com/docs/guide/memory-snapshots\">Modal Memory Snapshots\u003C/a>: lifecycle hooks, CPU/GPU snapshot behavior, storage-bandwidth limitations, captured randomness, and Volume dependencies in the snapshot and artifact sections.\u003C/li>\n\u003Cli>\u003Ca href=\"https://docs.pytorch.org/docs/2.9/generated/torch.cuda.graph.html\">PyTorch 2.9 CUDA graph API\u003C/a>: replay and capture-mode semantics in the CUDA graph section.\u003C/li>\n\u003Cli>\u003Ca href=\"https://onnxruntime.ai/docs/execution-providers/CUDA-ExecutionProvider.html\">ONNX Runtime CUDA execution provider\u003C/a>: CUDA provider selection and convolution-algorithm search underlying the ScoreQ and packaging sections.\u003C/li>\n\u003Cli>\u003Ca href=\"https://huggingface.co/docs/huggingface_hub/main/guides/download\">Hugging Face Hub downloads\u003C/a>: immutable revision selection in the artifact-release section.\u003C/li>\n\u003Cli>\u003Ca href=\"https://modal.com/docs/guide/scale\">Modal scaling\u003C/a> and \u003Ca href=\"https://modal.com/pricing\">pricing\u003C/a>: retention controls and the dated resource-cost calculation.\u003C/li>\n\u003C/ul>\n\u003Ch2 id=\"related-diffio-posts\">Related Diffio Posts\u003C/h2>\n\u003Cul>\n\u003Cli>\u003Ca href=\"/blog/vibevoice-next-token-diffusion/\">VibeVoice: Frame Rate Is the Context Budget for Long-Form TTS\u003C/a>: the general autoregressive-plus-diffusion architecture referenced above.\u003C/li>\n\u003C/ul>\n",false,[22,33,48],{"slug":23,"path":24,"title":25,"description":26,"publishedAt":27,"updatedAt":28,"author":10,"category":11,"tags":29,"featureImage":16,"ogImage":16,"canonical":31,"readingTimeMinutes":32},"mac-mini-inference-serving","/blog/mac-mini-inference-serving/","Overlapping I/O With Inference Around One Shared Model, Not Batching, Raised Mac Mini Throughput on I/O-Bound Restores","How Diffio raised Mac mini throughput on I/O-bound speech restores by overlapping downloads, encoding and uploads with inference around one shared model, rather than batching.","2026-10-07T16:33:19Z","2026-10-22T23:00:30Z",[13,14,30],"Apple Silicon","https://diffio.ai/blog/mac-mini-inference-serving/",26,{"slug":34,"path":35,"title":36,"description":37,"publishedAt":38,"updatedAt":39,"author":10,"category":11,"tags":40,"featureImage":41,"ogImage":46,"canonical":47,"readingTimeMinutes":18},"diffio-apple-silicon","/blog/diffio-apple-silicon/","Partitioning MLPs Along the Intermediate Dimension Let the Apple Neural Engine Run Two of Three Guidance Branches","How Diffio 4.0 reached 7.8x (pro) and 10.7x (flash) realtime on an M4 Pro Mac mini by splitting guidance branches across the GPU, the CPU's SME units and the Neural Engine, and what memory, precision and scheduling taught us.","2026-09-23T15:03:54Z","2026-10-22T23:00:31Z",[13,14,30],{"src":42,"alt":43,"widthPixels":44,"heightPixels":45},"/media/blog/diffio-apple-silicon/feature.webp","Diagram of one guidance step on Apple silicon: one branch on the GPU and CPU, two on the Neural Engine with the MLP split into quarters, combined into guidance",1400,933,{"src":42,"alt":43,"widthPixels":44,"heightPixels":45},"https://diffio.ai/blog/diffio-apple-silicon/",{"slug":49,"path":50,"title":51,"description":52,"publishedAt":53,"updatedAt":54,"author":10,"category":55,"tags":56,"featureImage":59,"ogImage":64,"canonical":65,"readingTimeMinutes":66},"pershing-1918-battlefields-of-france-restored","/blog/pershing-1918-battlefields-of-france-restored/","Hear General Pershing's 1918 message from the battlefields of France, restored","A 33-second disc General John J. Pershing recorded in France in March 1918, restored for Veterans Day, with the original alongside.","2026-11-02T08:40:45Z","2026-11-02T09:13:55Z","restoration",[57,58],"Audio restoration","History",{"src":60,"alt":61,"widthPixels":62,"heightPixels":63},"/media/blog/pershing-1918-battlefields-of-france-restored/feature.webp","Waveform of Gen. John J. Pershing's 1918 recording \"From the Battlefields of France,\" restored with Diffio AI",1280,720,{"src":60,"alt":61,"widthPixels":62,"heightPixels":63},"https://diffio.ai/blog/pershing-1918-battlefields-of-france-restored/",3,1793728206637]