HalfVulpes/Deepinfer-SM80

选用指南

AI 根据 README 整理 · 2026/10/21 13:21:44

使用方式

通过应用git补丁或复制模块文件到vLLM安装目录来使用，并可运行提供的微基准测试脚本。

查看 README 依据

``` --- ## Quick Start & Installation ### Option A: Apply the Unified Git Patch to vLLM From the root of your vLLM checkout: ```bash git apply /path/to/Deepinfer-SM80/patches/vllm-sm80-deepinfer.patch ``` ### Option B: Bind-Mount or Copy Drop-In Modules Copy the 6 drop-in files directly into your `vllm` Python package directory (or bind-mount them as read-only overlays in Docker): ```bash cp -r vllm_sm80_deepinfer/vllm/* /path/to/site-packages/vllm/ ``` ### Running the Isolated SM80 Kernel Microbenchmarks

To verify bit-exactness and measure kernel latency on your own `A100` / `RTX 3090` / `A6000` GPU: ```bash python3 benchmarks/microbench_sm80_kernels.py python3 benchmarks/microbench_sm80_part2.py python3 benchmarks/microbench_sm80_part3.py python3 benchmarks/generate_charts.py ``` --- ## Multi-GPU Topology Guide for 1M Context on SM80 (`TP2 × PP3` vs `PP6` vs `TP2 + TP4`)

部署要求

需要NVIDIA Ampere架构GPU（如A100、RTX 3090、RTX A6000），并运行vLLM V1引擎。

查看 README 依据

**DeepInfer-SM80** ports and adapts core architectural concepts and kernel optimizations from DeepSeek's [`DeepInfer`](https://github.com/deepseek-ai/DeepInfer) (`SM100` / Blackwell `tcgen05.mma` + `TMEM` + `CuteDSL`) to **NVIDIA Ampere (`SM80` / `SM86`) GPUs**—including **NVIDIA A100 (`40GB / 64GB / 80GB`)**, **A800**, **RTX 3090**, and **RTX A6000**—running on the **vLLM V1** engine.

已知限制

仅支持NVIDIA Ampere（SM80/SM86）GPU，不支持Blackwell架构的硬件特性。

查看 README 依据

While upstream `DeepInfer` relies on Blackwell-exclusive hardware features (`tcgen05` warp-group instructions, Tensor Memory `TMEM`, hardware `UE8M0` scale instructions, and 228 KB shared memory per SM), **DeepInfer-SM80** re-engineers the critical execution path around Ampere's `mma.sync.aligned.m16n8k16` Tensor Cores, 164 KB L1/SRAM capacity (`SM80`), IEEE-754 integer bitshift `UE8M0` scale decoding, SM-count-aware grid scheduling, and YOCO (`L20`) shared-KV reuse. --- ## Highlights & Production Benchmark Results