deepseek-ai/DeepInfer

选用指南

AI 根据 README 整理 · 2026/10/20 17:26:46

使用方式

通过命令行构建和运行，使用 Python 脚本进行离线推理，支持通过环境变量配置模型路径和各项功能。

查看 README 依据

- Linux - GCC >= 13.0 - Python 3.12 - CUDA 13 (NVIDIA only) - CANN 9.2 (Ascend only) ### Build from source Before building DeepInfer, please create and activate an empty Python 3.12 environment. ``` bash # Use `ascend` instead for the Ascend platform # Keep it exported in the environment where you run DeepInfer export DEEPINFER_ACCELERATOR=nvidia ./build.sh ``` ### Offline inference The default checkpoint, [`deepseek-ai/DeepSeek-V4.1-Flash-1017`](https://huggingface.co/deepseek-ai/DeepSeek-V4.1-Flash-1017), is

downloaded into the Hugging Face cache on the first run; set `DEEPINFER_HF_MODEL_DIR=/path/to/DeepSeek-V4.1-Flash-1017` to run from a local copy instead. ``` bash python examples/offline_inference.py ``` See [examples](examples/README.md) for details. ## Performance benchmarks | Prefill | Decode | | :---: | :---: | | [](doc/images/prefill_pareto.png) | [](doc/images/decode_pareto.png) |

部署要求

需要 Linux 系统、GCC 13.0 或更高版本、Python 3.12，NVIDIA 平台需 CUDA 13，Ascend 平台需 CANN 9.2。

查看 README 依据

- Linux - GCC >= 13.0 - Python 3.12 - CUDA 13 (NVIDIA only) - CANN 9.2 (Ascend only) ### Build from source Before building DeepInfer, please create and activate an empty Python 3.12 environment. ``` bash # Use `ascend` instead for the Ascend platform # Keep it exported in the environment where you run DeepInfer export DEEPINFER_ACCELERATOR=nvidia ./build.sh ``` ### Offline inference The default checkpoint, [`deepseek-ai/DeepSeek-V4.1-Flash-1017`](https://huggingface.co/deepseek-ai/DeepSeek-V4.1-Flash-1017), is

已知限制

本发布版不包含完整的推理服务所需组件，如 KV cache 系统、VE/Prefill/Decode 分离和跨数据并行副本的负载均衡。

查看 README 依据

- **Structured outputs:** constrain generation to a regex, JSON schema, or EBNF grammar. Pass `so_args={"schema_type": ..., "schema": ...}` to `SamplingParams`; see [examples](examples/_structured_output.py). </details> Supported hardware: - NVIDIA SM100 GPUs - Huawei Ascend 950 NPUs Required for a complete inference service (not included in this release): - A KV cache system - VE/Prefill/Decode disaggregation (most required engine functionality is already implemented) - Load balancing across data-parallel replicas for both prefill and decode ## Getting started ### Prerequisites