Moonshot Unveils Kimi K3: A 2.8 Trillion Parameter Multimodal Model with Substantial Performance Gains

Moonshot has officially launched Kimi K3, an advanced multimodal model possessing an impressive 2.8 trillion parameters, a context window of 1 million tokens, and robust native vision capabilities. On the day of its release, users can interact with the model at a remarkable processing rate of 460 tokens per second, showcasing its efficiency.

This new offering is backed by a collaborative effort with Modal, providing token-based pricing for its Shared API, as well as a dedicated Auto Endpoint for optimized capacity. The team has also integrated DFlash, a speculator specifically tuned for K3’s architecture, to enhance its operational performance further.

Performance Benchmarks and Structure

Kimi K3 stands out as the most powerful open model currently available, ranking fourth in intelligence indexes primarily occupied by proprietary models. Its architecture is grounded in a mixture-of-experts design, activating only 16 out of 896 experts per token processed. This innovative model is directed towards long-term, agentic tasks.

Moonshot’s internal testing revealed that K3 effectively managed substantial kernel optimization tasks, underscoring its design's viability. The incorporation of Kimi Delta Attention significantly reduces attention cost as sequence lengths increase, while the Attention Residuals mechanism allows deeper layers to reference prior attention outputs, suggesting a considerable leap in scalability compared to its predecessor, K2.

Deployment Constraints and Solutions

Despite its advanced capabilities, K3's implementation isn't without challenges. Moonshot prioritized user accessibility, tackling the complexities involved in operating such a high-parameter model. They implemented quantization-aware training from the supervised fine-tuning (SFT) phase onward, utilizing MXFP4 weights and MXFP8 activations. This approach permits K3 to be run on diverse hardware setups, thereby improving its versatility.

An important part of their development process involved rebalancing expert parallelism to maintain throughput under large-scale conditions. The team also addressed issues with the conventional prefix caching that KDA encountered, leading to the creation of a new implementation, which was made available to the vLLM community prior to K3's launch. Such foundational, albeit low-profile, work has enabled a 3 trillion parameter open model to function effectively in production environments.

Interactivity and Throughput Enhancements

Utilizing Modal's infrastructure, immediate support for a custom DFlash speculator tailored to K3's framework has been established. Given the model's tendency to generate a considerable volume of tokens per task, the decode time becomes a significant factor in user wait times. By optimizing the decode process through speculative methods, the enhancements have yielded substantial advancements in interactive performance and overall throughput.

• Interactivity improved by 360%, moving from 100 to 460 tokens per second.
• Throughput increased by 88%, rising from 800,000 to 1.5 million tokens per minute per GPU.

Speculative Decoding for Enhanced Inference

K3 represents the largest target against which a speculator has been trained to date. Custom draft model architectures, such as DFlash, can remarkably speed up production inference when precisely trained. The DFlash training process is primarily a data generation exercise, with K3 serving as the teacher model. The finalized training utilized 32 B300 nodes, where 28 nodes operated K3 on TP8 for generating hidden states, while the remaining 4 nodes trained against these outputs.

Several design considerations guided the development of this drafter, and further insights into these approaches will be shared in subsequent communications. In the meantime, extensive discussion about the commitment to speculative decoding can be found in our related posts.

Accessing Kimi K3 on Modal

Kimi K3 is now accessible as an OpenAI-compatible Shared API, available with token-based pricing for both Team and Enterprise customers. It can also be deployed as a dedicated Auto Endpoint under any plan, enabling organizations to harness its capabilities effectively.

Experience Kimi K3 today by accessing it through Modal's platform, and engage with its groundbreaking features. Your feedback will be invaluable as we continue to refine and enhance this model.