eagle-hq/EAGLE-4

选用指南

AI 根据 README 整理 · 2026/10/21 06:51:39

目标用户

面向需要加速LLM推理的开发者或研究人员。

查看 README 依据

Figure 1(a) of the EAGLE-4 paper: Qwen3.8-27B on the SPEED-Bench qualitative split, 4× NVIDIA B300, one request, temperature 0. The numbers this release measured with EAGLE-4 are in the <a href="docs/support-matrix.md">support matrix</a>. </sub> </p> ## EAGLE-4 EAGLE-4 (Extrapolation Algorithm for Greater Language-model Efficiency) is the latest generation of the EAGLE family for accelerating large language model inference. One drafter adapts its speculative decoding strategy to the batch size, the hardware and the workload, and keeps the target model's output distribution.

使用方式

通过pip安装eagle_harness包，并与SGLang、vLLM或TokenSpeed引擎插件配合使用。

查看 README 依据

The EAGLE-4 draft uses a sliding-window draft layer, a low-rank candidate head and an append-only draft cache. Dense and latent-MoE draft structures are implemented. Qwen3.8-27B with its [EAGLE-4 draft](https://huggingface.co/EAGLE-corp/EAGLE4-Qwen3.8-27B) is the measured target pair. ## Engines This repository is `eagle-harness`, a pip-installable package (`eagle_harness`) that serves EAGLE-4 drafts on SGLang, vLLM and TokenSpeed through engine plugins. EAGLE-4 sizes the speculative tree from measured draft and

部署要求

需要Python 3.12（CUDA引擎）或3.10（ROCm），以及NVIDIA B300或AMD MI355X等特定硬件。

查看 README 依据

verification costs for the batch size, hardware and workload it is running on. It registers through hooks the engine exposes and does not patch the engine at runtime. The patched engines ship as pinned wheels and container images, not as source trees in this repository: | Engine | Hardware | Wheel index | Image | Source | | --- | --- | --- | --- | --- | | SGLang | NVIDIA B300, CUDA 13.0 | [cu130](https://eagle-hq.github.io/EAGLE-4/whl/cu130/) | `ghcr.io/eagle-hq/eagle-4-sglang:0.1.0` | [eagle-hq/sglang](https://github.com/eagle-hq/sglang) |

Use Python 3.12 for the CUDA engines and Python 3.10 for the ROCm wheel. The commands read their pins from the release settings file in the checkout. Run them, and start servers, from the directory that contains the checkout. Inside it, `python -m` would import the source tree's `eagle_harness/` instead of the installed wheel. ```bash git clone https://github.com/eagle-hq/EAGLE-4 . EAGLE-4/docs/release.env ``` SGLang on CUDA: ```bash python3.12 -m venv .venv-sglang && source .venv-sglang/bin/activate python -m pip install --upgrade pip

已知限制

仅支持文本请求，不支持流水线并行，TokenSpeed为实验性且仅限CUDA。

查看 README 依据

eagle-harness hooks. On SGLang and vLLM a draft top-k of 1 gives a chain and a larger top-k gives a tree. TokenSpeed implements static and calibrated trees, chains and sampled trees, without prompt logprobs or native no-speculation rounds, and its coverage is limited to the configurations in the support matrix. - GPUs: NVIDIA sm_80 and newer, and AMD CDNA2 to CDNA4, through Triton kernels. The kernel tests run on B300 and the other architectures are compile-checked. MI355X is covered by the serving measurements in the support matrix. See [kernels](docs/kernels.md).

- Requests: text only. vLLM tree verification handles greedy and sampled requests and needs an unquantized KV cache. Constrained output takes the correctness fallback in [compatibility](docs/compatibility.md). - Batch: SGLang and vLLM trees and chains admit up to 512 running requests within the engine's KV and recurrent-state capacity. The SGLang tree default is 32, so set `--max-running-requests` for larger batches. No pipeline parallelism. Admission is not a throughput or correctness guarantee. Check the