google-research/harness2

选用指南

AI 根据 README 整理 · 2026/10/18 04:06:29

使用方式

通过命令行使用，提供 setup、check、run、judge、report 等子命令，支持并行和顺序递归模式。

查看 README 依据

alternatives, and the [Codex executor](docs/SETUP.md#codex-executor). ## Run a benchmark The following example installs LAB and runs one task, then evaluates and reports its result. Model-backed runs incur API costs; start with a small task selection. ```bash harness2-setup lab harness2-check --bench lab harness2 --bench lab --k 1 --mode parallel \ --only-domains antitrust-competition --max-tasks 1 harness2-judge --bench lab --k 1 --mode parallel harness2-report --bench lab --k 1 --mode parallel ``` Sequential recursion runs the same commands with `--mode sequential`. `--k` is

the task-level recursion budget; see [Experiments](#experiments). Judging and reporting must use the same benchmark, executor, subset, `--k`, `--mode`, and seed as the refinement run. Add `--dry-run` to `harness2` to preview task selection without model calls. | Benchmark | Tasks | Install | Guide | |---|---|---|---| | JobBench | Professional work | `harness2-setup jb` | [JobBench](docs/BENCHMARKS.md#jobbench) | | LAB | Legal work | `harness2-setup lab` | [LAB](docs/BENCHMARKS.md#lab) |

部署要求

需要 Linux、Python 3.12 和 uv。模型基准测试需要 Node.js 20+、Google Cloud CLI，以及各基准的依赖（LAB 需要 Pandoc 和 Poppler，WorkBuddy-Bench 需要 Docker）。

查看 README 依据

<img src="assets/components.png" alt="Eight harness components grouped by runtime role: system prompt, guardrails, memory, and skills construct context; tool guidance and scripts define the tool interface; subagents orchestrate delegated work; plugins control middleware." width="100%"> </a> </p> ## Install Use **Linux, Python 3.12, and [uv](https://docs.astral.sh/uv/getting-started/installation/)**. From the repository root: ```bash uv venv --python 3.12 .venv source .venv/bin/activate uv pip install -e '.[render]' harness2-demo --output runs/demo --k 2 ```

The local demo runs both recursion modes with deterministic editors and real Python executions. It requires no model credentials. For model-backed benchmarks, install **Node.js 20+, the Google Cloud CLI**, and the [benchmark dependencies](docs/SETUP.md#requirements). LAB needs Pandoc and Poppler; WorkBuddy-Bench needs Docker. Then configure your models: ```bash cp .env.example .env # Edit .env with your Vertex AI project and model IDs. gcloud auth application-default login ``` See the [setup guide](docs/SETUP.md) for model settings, authentication

已知限制

模型支持的运行会产生 API 费用；执行器网络访问不受限制，Codex 沙箱被禁用，需自行配置网络限制以防止访问基准答案。

查看 README 依据

alternatives, and the [Codex executor](docs/SETUP.md#codex-executor). ## Run a benchmark The following example installs LAB and runs one task, then evaluates and reports its result. Model-backed runs incur API costs; start with a small task selection. ```bash harness2-setup lab harness2-check --bench lab harness2 --bench lab --k 1 --mode parallel \ --only-domains antitrust-competition --max-tasks 1 harness2-judge --bench lab --k 1 --mode parallel harness2-report --bench lab --k 1 --mode parallel ``` Sequential recursion runs the same commands with `--mode sequential`. `--k` is

Executor network access is unrestricted, and Codex runs with its sandbox disabled. Public benchmark rubrics may be accessible online. Before evaluation, configure [network restrictions](docs/EXPERIMENTS.md#web-access) to prevent access to benchmark answers and grading criteria. ## Results and resume Results are saved under `runs/`, organized by benchmark and subset where applicable. They include task summaries, deliverables, adopted harnesses, reflection notes, and evaluation results. Repeat the same command to resume. Use a new `HARNESS2_RUNS_DIR` when changing