Serving Sakana Namazu: Quantization, Speculative Decoding and GPU Scaling

Serving Sakana Namazu, a roughly one-trillion-parameter model built for Japanese business use, required more than fitting its weights onto GPUs. The deployment team had to preserve Japanese instruction following while reducing memory use, speed up generation, and handle unpredictable early demand. Modal and Sakana AI approached those constraints together, tuning the model and serving system around the workload.

The work combined NVFP4 quantization, a draft model trained on Sakana-specific token data, serving-kernel changes and elastic GPU capacity. Modal says the quantized model matched Sakana’s production checkpoint on Japanese instruction following, within the noise of Sakana’s published benchmarks. The account gives several performance measurements, though it doesn’t report absolute GPU counts, serving costs or benchmark scores.

Keeping Japanese performance through quantization

A model of this size has a large memory footprint at FP16 or BF16 precision, making lower-precision weights a practical requirement for economical production serving. But calibration based on general English text may miss the language register and tool-calling patterns that matter to a model optimized for Japanese business contexts.

Modal’s team profiled how Namazu’s weights responded to NVFP4, a format designed for NVIDIA Blackwell GPUs, then applied compression selectively. Routed experts, which contain most of the model’s parameters, were quantized. Attention, the router and shared experts stayed in BF16. Calibration used conversations representative of Namazu’s style, with a strong emphasis on Japanese. It covered all 23,040 experts before their quantization scales were set.

That approach reportedly reduced memory use and increased throughput while keeping Japanese instruction performance close to the production checkpoint. The source doesn’t give a percentage for the memory reduction or throughput gain, so the practical size of those improvements can’t be compared directly with other deployments.

Drafting from the model’s actual tokens

Speculative decoding uses a smaller draft model to propose several tokens for the main model to check in a single step. Generation can speed up when the main model accepts more of those proposals. A generic draft model performed poorly on Namazu’s Japanese business register, with acceptance rates below 40%, so Modal trained a custom DFlash draft model using representative traffic.

The training data preserved the exact token IDs generated by Namazu. Decoding those IDs into text and tokenizing the text again can change token boundaries in tool calls, JSON schemas and reasoning markers. That mismatch can make draft tokens less likely to match what the main model accepts. Modal built a capture path that sent raw token IDs from production into a training corpus of more than 30 billion tokens.

Against a generic draft model, the first custom drafter increased speculative token acceptance by 25%, reduced per-token decode latency by about 35%, and allowed roughly 40% more traffic per replica before queues began to grow. After launch, the team trained another drafter from scratch on roughly one million real requests. This version used DFlash2, an architecture described as better suited to nonzero-temperature sampling, which covers most Namazu traffic.

In a replay of 1,000 held-out requests using customer sampling settings, average acceptance length rose from 3.51 tokens to 5.01, a 43% increase. The new drafter had not trained on those requests. P99 latency fell from 68 seconds to 52 seconds. These results come from the reported replay and don’t establish the same gains for other request mixes.

Removing stalls and supporting structured output

The teams tested serving-engine updates on the same hardware and request batches. One change removed a CPU–GPU synchronization stall in the speculative decoding loop: each step had paused while the CPU read results from the GPU. Removing the wait cut GPU idle time per step by more than 90%.

New kernels for NVFP4 expert matrix multiplication also shortened decode steps. Those operations account for about 60% of each step, according to Modal, and the tighter numerical behavior increased draft acceptance by 9% with the same drafter. Caching prefill dispatch and expert-runner setup avoided repeating work on every request, with most of the benefit going to time to first token.

Namazu uses tensor parallelism across four GPUs. During speculative verification, the GPUs must agree on which drafted tokens the main model accepts. Instrumentation exposed a rare path where ranks could disagree; the team made acceptance a shared decision, eliminating the resulting class of stalls without changing output quality.

The team also made grammar-constrained generation work within speculative decoding. That lets requests using JSON schemas for tool use receive the decoding speedup while maintaining schema validity, according to the source.

Routing requests and scaling GPU capacity

Modal enabled its KV-aware router on the deployment without changes to Sakana’s code. Instead of distributing requests evenly, it favors replicas that are likely to have the request’s prompt prefix in cache, while accounting for load as replicas change. Modal reports that time-to-first-token spikes of several minutes disappeared, per-replica queues fell close to zero, and end-to-end tail latency improved. Per-token decode speed stayed the same.

Elastic GPU capacity also helped Sakana handle launch demand without committing to reserved capacity before the API had seen customer traffic. Masaya Ohagi, an applied research engineer, said the team needed to adjust GPU capacity while preserving model quality, and that Modal’s engineers worked with Sakana on performance changes introduced by quantization.

The deployment’s results reflect several coordinated choices: selective low-precision weights, calibration matched to Japanese use, draft training on raw production tokens, kernel and routing changes, and capacity that can grow with demand. For teams serving large specialist models, the central lesson is that compression and decoding gains depend on measuring the actual workload. The reported figures offer a useful case study, while the absence of absolute cost and hardware data leaves open how the approach compares economically across deployments.