Paper: SLAI T-Rex: Full-Parameter Post-training of the DeepSeek-V4 Family on Ascend SuperPOD

Papers
Page content

Listen to this article.

Audio is available for 30 days and will be removed automatically.

Problem

Training massive, trillion-parameter Mixture of Experts (MoE) language models like DeepSeek-V4 presents significant engineering challenges when using distributed training systems. The paper highlights issues including intense memory usage, communication bottlenecks, and inefficient processing during the post-training phase—specifically, Full Parameter Post-Training (CPT) and Supervised Fine Tuning (SFT). While most existing solutions rely on GPU clusters, this research explores an alternative approach leveraging Ascend Neural Processing Units (NPUs).

Method

The authors introduce “SLAI T-Rex,” a hierarchical optimization framework built for the Ascend NPU SuperPOD. This framework tackles the distributed training problem at multiple levels: model parallelism, efficient orchestration of computation and communication, and optimizations within low-level kernel execution. To demonstrate this system’s capabilities, they applied it to the DeepSeek-V4 family and further created specialized CPT and SFT data pipelines tailored for Operations Research (OR) tasks. These pipelines combine real-world OR domain resources with synthetically generated optimization documents validated by solvers.

Results & Limitations

The authors report impressive results: SLAI T-Rex achieved a 34.22% Model FLOPs Utilization (MFU), representing a 2.93x performance improvement over an open-source baseline on the Ascend NPU SuperPOD. Further, the specialized model trained using their OR data achieves state-of-the-art zero-shot Pass@1 scores of 71.81%, surpassing GPT-5.4-Mini and the base DeepSeek-V4-Flash by significant margins. However, based solely on the abstract, it’s unclear how reproducible these results are outside of the specific Ascend NPU SuperPOD environment. The complexity of the hierarchical optimization framework might also present a barrier for broader adoption.

Why It Matters

This paper is relevant to ML/data practitioners working with very large language models and those exploring alternative hardware beyond traditional GPUs. It demonstrates that NPUs, specifically Ascend NPUs, can be effectively used for training trillion-parameter models – potentially offering performance advantages. Furthermore, the approach of combining domain resources with solver-verified synthetic data for specialized task fine-tuning (like Operations Research) offers a promising direction for creating highly effective LLMs in specific application areas. The 45 upvotes on Hugging Face suggest significant community interest.

References