Model details and offline inference

The DeepInfer repository supports only one model, DeepSeek-V4.1-Flash, a large language model with 552 billion parameters. This model was released on September 10 and features a mixture of experts architecture, which ActuIA described in detail. The repository includes documentation that describes the release as enabling offline inference, where the engine, kernels, and scheduler are available but no API server is documented.

Supporting libraries and API alignment

The repository is accompanied by deepseek-recipe, a set of Rust libraries that convert API Chat Completions, Responses, and Messages requests into token sequences. These libraries also analyze the model's outputs, including reasoning and tool calls. DeepSeek states that the logic of deepseek-recipe aligns with its official API, so a third-party host could achieve the same API behavior. However, the host would still need to provide inference, tool execution, and HTTP transport.

Revenue and scale of service

A Q&A document published with the code outlines the scale of the service. DeepSeek states that a single cluster processes 155,000 billion tokens per day, with a flat rate revenue of $38 per GPU-hour, which is half the peak rate. This figure represents revenue and provides no information about costs or margins. The document does not specify the cluster size, the chip used, the proportion of input tokens, or whether the revenue is billed or theoretical.

Comparing revenue figures

On March 1, 2025, DeepSeek published a theoretical revenue of $562,027 over 24 hours for models V3 and R1. This figure is based on an average of 226.75 nodes, each with eight H800s, and 776 billion tokens, combining input and output. This results in $12.9 per GPU-hour at the R1 rate, calculated by ActuIA. The $38 figure is nearly three times higher. However, the comparison remains indirect due to differences in models, pricing grids, and the fact that the 2025 figure is theoretical.

Components needed for deployment

The README lists additional components needed for a complete service, including a KV cache system and the separation of image encoding, prefill, and decode phases onto distinct machines. Load balancing between prefill and decode replicas is also required. Prefill processes the input prompt, while decode generates the response token by token. The README states that most of the required engine features are already implemented.

Internal infrastructure and open-source components

DeepSeek notes that the KV cache relies on 3FS, a distributed file system it open-sourced in February 2025. The storage has approximately 100 petabytes of SSD, with a retention period of three days for agent traffic and seven days for chatbot traffic. The inference system uses four load balancers, with the expert load balancer included in the published code. The document indicates that the load balancing achieves about 80% of a perfectly balanced offline benchmark.

Optimal deployment configuration

The document suggests that one prefill instance and one decode instance suffice for a deployment. The recommended decode configuration distributes the experts across 64 to 128 GPUs, with a prefill/decode ratio of approximately 1 to 1. The most efficient unit is 64 GPUs for prefill and 64 for decode. DeepSeek indicates that its inference nodes are spread across distinct but neighboring data centers, and the company advises planning for approximately 400 Gbit/s of interconnection per group of eight GPUs.

Decision to delay engine open-sourcing

In April 2025, DeepSeek decided not to open its internal engine, citing three reasons: an old fork of vLLM, tight coupling with internal infrastructure, and the lack of a small research team to maintain a large project. The company promised to contribute to existing projects. Eighteen months later, the DeepInfer README still credits vLLM as the original base, with SGLang and TensorRT-LLM for inspiration. The same argument is used to exclude the KV cache and load balancing.

Future plans and chip support

Two new elements appear in DeepSeek's texts: the engine targets multiple chip suppliers, including NVIDIA and Huawei, and the inference code for future models will continue to be published. On the NVIDIA side, only the SM100 architecture, used in Blackwell data center GPUs, is listed. The H100 and H200 are not included. On the Huawei side, DeepInfer requires the CANN 9.2 toolset. DeepSeek had ported DeepGEMM-Ascend and DeepEP-Ascend to the Ascend 950 on September 30.

Performance and compatibility differences

Batch-invariant inference and deterministic sampling are available only on NVIDIA. The two performance charts in the README, prefill on 4 GPU SM100 and decode on 64, are measured on NVIDIA. The text provides no numbers. No independent benchmark was published on October 20, according to Bloomberg. The Hugging Face page for DeepSeek-V4.1-Flash-1017, the default checkpoint of DeepInfer published on October 17, details a deployment with vLLM and SGLang, without mentioning DeepInfer. V4.1-Pro, which the V4.1-Flash announcement mentions will be coming, has no date, and DeepSeek does not name it in relation to DeepInfer.