ninjahawk/livenerf

选用指南

AI 根据 README 整理 · 2026/10/16 21:11:36

适用场景

适用于需要长期跟踪特定模型（如 Claude Opus 5.5）在发布后性能变化的场景，例如验证模型是否被降级或能力漂移。

查看 README 依据

livenerf is a small, boring, append-only benchmark for one question: does a model get worse after it ships? For months there have been reports that Anthropic "nerfs" models some days or weeks after release. That could mean quantization, a smaller model behind the same name, lower effort, or routing changes. It could also mean nothing happened and people are pattern-matching on noise. Nobody has had a clean day-0 baseline to check against, so every argument ends up as vibes versus vibes. Claude Opus 5.5 came out on 2026-09-22, so this is a chance to start the clock on launch day and keep it

The main thing this repo will maintain is a running 10-day table of how Opus 5.5 does on the calibrated benchmark panel relative to its launch-week baseline. A negative delta means worse than launch week. The table reports improvements just as loudly as regressions. | # | window | samples | score | Δ vs baseline ± SE | output tok (median) | control Δ | CLI | decision | |---|------|---------|-------|--------------------|---------------------|-----------|-----|----------| | 0 | days 1–10 (from 2026-09-24) | 780 | 0.584 ± 0.036 | baseline | 588 | baseline | 2.1.280 | baseline |

目标用户

面向对模型性能变化感兴趣的研究者或用户，尤其是关注模型发布后是否变差的人群。

查看 README 依据

livenerf is a small, boring, append-only benchmark for one question: does a model get worse after it ships? For months there have been reports that Anthropic "nerfs" models some days or weeks after release. That could mean quantization, a smaller model behind the same name, lower effort, or routing changes. It could also mean nothing happened and people are pattern-matching on noise. Nobody has had a clean day-0 baseline to check against, so every argument ends up as vibes versus vibes. Claude Opus 5.5 came out on 2026-09-22, so this is a chance to start the clock on launch day and keep it

使用方式

通过命令行使用，需要 Python 3.11+、uv 和已登录的 Claude Code 安装，支持 Linux、macOS 和 Windows。

查看 README 依据

You need Python 3.11+, [uv](https://docs.astral.sh/uv/), and a logged-in [Claude Code](https://code.claude.com/docs/en/overview) install. v0 was built around a Max subscription, but anything that can run `claude -p` works. Linux, macOS and Windows are all supported. ```bash git clone https://github.com/ninjahawk/livenerf cd livenerf uv sync ``` This installs a pinned [Inspect](https://inspect.aisi.org.uk/) and registers the `claudecode` model provider. The provider wraps a hermetic `claude -p` call, so Inspect treats your Max subscription like any other model API.

部署要求

需要 Python 3.11+、uv 和已登录的 Claude Code 安装，并建议使用 Claude Max 订阅（但任何能运行 `claude -p` 的方式均可）。

查看 README 依据

You need Python 3.11+, [uv](https://docs.astral.sh/uv/), and a logged-in [Claude Code](https://code.claude.com/docs/en/overview) install. v0 was built around a Max subscription, but anything that can run `claude -p` works. Linux, macOS and Windows are all supported. ```bash git clone https://github.com/ninjahawk/livenerf cd livenerf uv sync ``` This installs a pinned [Inspect](https://inspect.aisi.org.uk/) and registers the `claudecode` model provider. The provider wraps a hermetic `claude -p` call, so Inspect treats your Max subscription like any other model API.

已知限制

该基准测试无法检测同系列模型的小规模替换（如 Opus 5 替换为 Opus 5.5），且测量的是通过 Claude Code 订阅服务的模型，而非原始 API 模型。

查看 README 依据

- effort low: −62% output tokens, −8.3 ± 4.5 points of accuracy; - effort medium: −26% tokens, −4.2 ± 3.9 points. - **The limit.** Swapping in Opus 5 was *not* distinguishable from Opus 5.5 at 99% (−3.8 ± 6.3 points, −23% tokens). This instrument can't detect a same-family model swap of that size in a validation's worth of samples. A 10-day window has about 2.5 times as many samples, but that hasn't been shown to be enough. - **The questions themselves.** A report-only audit of the 78 questions, plus the 2 later

The important thing to note is that livenerf measures **Opus 5.5 as served through Claude Code on a subscription**. That is the thing most nerf reports are actually about, and it is *not* the same as the raw API model. The launch-week baseline is also a reference point, not ground truth. Launch week could easily be the *worst* week: new serving stack, capacity strain, launch bugs. The 2025 quality incidents turned out to be infrastructure bugs, not deliberate downgrades. So livenerf tests for change in either direction and does not assume a mechanism. ## Pre-registration