NVIDIA/flare-local-inference

选用指南

AI 根据 README 整理 · 2026/10/21 09:11:17

适用场景

适用于在特定 NVIDIA GPU 上本地运行模型，支持长对话、编码代理和重复工具使用等场景。

查看 README 依据

<p align="center"> <img src="docs/images/flare-logo.png" alt="Flare — ASCII wordmark above a solar flare" width="912"> </p> > **Coding agents:** skip this README and read [AGENTS.md](AGENTS.md) directly. Flare is a local inference server for NVIDIA SM120/SM121 GPUs, built for long conversations, coding agents, and repeated tool use. Its goal is to make the memory, bandwidth, and compute in workstations and compact systems such as DGX Spark go further, while keeping generation responsive. The design rests on a few principles:

使用方式

通过命令行界面进行配置和使用，具体可参考 CLI 指南。

查看 README 依据

[vLLM also uses CUDA graphs](https://docs.vllm.ai/en/latest/design/cuda_graphs/). Compare the engines on your actual model and workload. Start with [model preparation and serving](docs/model-preparation.md). See the [CLI guide](docs/cli-design.md) for configuration and the [core design](docs/core-design.md) for the architectural reasoning.

部署要求

需要 NVIDIA SM120/SM121 GPU。

查看 README 依据

<p align="center"> <img src="docs/images/flare-logo.png" alt="Flare — ASCII wordmark above a solar flare" width="912"> </p> > **Coding agents:** skip this README and read [AGENTS.md](AGENTS.md) directly. Flare is a local inference server for NVIDIA SM120/SM121 GPUs, built for long conversations, coding agents, and repeated tool use. Its goal is to make the memory, bandwidth, and compute in workstations and compact systems such as DGX Spark go further, while keeping generation responsive. The design rests on a few principles:

已知限制

与 vLLM 相比，Flare 支持的模型、硬件和部署模式范围更窄，专注于选定的模型和本地工作负载。

查看 README 依据

hardware, and deployment patterns. Flare trades that breadth for a focused serving contract around selected models and local workloads. That focus lets model preparation, kernels, memory management, and scheduling be designed together. It also means supported recipes and prepared capacities determine what you can run. Think of Flare as a specialized option when its hardware and model coverage fit your machine. Rust gives the serving runtime explicit resource ownership; performance still depends on the GPU work, memory traffic, and scheduling.