google-research/rrsi

选用指南

AI 根据 README 整理 · 2026/10/14 13:46:08

目标用户

面向对 LLM 代理自动优化感兴趣的研发人员和研究人员，需要熟悉 Python 和实验评估。

查看 README 依据

flow, tools, memory and context management around a frozen model. Evolving the harness against a fixed evolve set is effective but overfits: the harness memorizes the training tasks, and large in-distribution gains shrink or vanish out of distribution. RRSI keeps the harness edit space open and regularizes the search trajectory through it instead. On the proposal side, an annealed budget caps how many independent edits one candidate may bundle, the proposer is conditioned on the full edit history so a falsified hypothesis is not redrawn, and a stalled run is redirected toward

The search is not tied to one policy family: with Gemini 3.5 Flash as the frozen policy, the same coding instance goes from 64.6 to 78.7 on Terminal-Bench 2.1 and from 76.8 to 79.0 on SWE-bench Verified. ## 🧱 Adding a domain A domain is one module, `domains/<name>/adapter.py`, exporting `DOMAIN`, an instance of `rrsi.domain.Domain` that implements: * `evolve_ids`, `heldout_ids`, `smoke_ids`: the task splits; * `run(root, runs_dir, job, ids, k)` and `score(runs_dir, job, ids, k)`: run the harness checked out under `root` and return per-task trial rewards (Evaluate);

使用方式

通过命令行运行，使用 `python3 rrsi.py --domain <name> {smoke, baseline, run}` 等命令进行测试、评估和迭代。

查看 README 依据

export VERTEX_PROJECT="your-project-id" VERTEXAI_PROJECT="your-project-id" export VERTEX_LOCATION=global VERTEXAI_LOCATION=global export RRSI_VERTEX_PROJECTS="your-project-id" ``` ### 2. Run an instance Every instance follows the same shape: ```bash python3 rrsi.py --domain <coding|workspace|eng> smoke # liveness: compile, construct, a couple of tasks python3 rrsi.py --domain <name> baseline # Evaluate(H_0), seed runs/<name>/frontier.json python3 rrsi.py --domain <name> run # rounds 0..T-1, resumable; touch runs/<name>/STOP to stop

部署要求

需要 Python 3.10 或更高版本，并安装依赖和运行环境（如 Docker、git worktrees）。需要配置 LLM 模型（如 Claude Opus 4.8 或 Gemini 3.5 Flash），以及 Vertex AI 项目设置。

查看 README 依据

| Critic (leakage screen before evaluation) | `rrsi/critic.py` (domain regex denylist plus LLM review, bounded repair) | | Evaluate in parallel | `rrsi/evaluate.py`, `Run.round` thread pool | | Noise-adjusted floor, cost rule, within-band rule, argmax | `rrsi/selection.py` | | Novelty nu_t (structural component types never in a winning edit) | `rrsi/components.py: novelty` over K_str = client_tool, skill, memory, subagent | | Prune set B_t | `History.prune_set`, handed to the proposer with the accepted machinery to remove |

| Noise band delta | fixed per instance in `rrsi.json` (0.017 / 0.004 / 0.020); `rrsi/calibrate.py` re-estimates it when `delta` is `null` (bootstrap over trials of the base evaluation, or repeated base evaluations) | | Non-compensatory domain criteria | `Domain.guards` (engineering: valid-rate drop, no-submission rise) | --- ## ⚡️ Quickstart ### 0. Install ```bash git clone https://github.com/google-research/rrsi.git && cd rrsi pip install -e ".[dev]" # the search core (Python 3.10 or newer) python3 -m pytest tests ```

The benchmark runners live in their own environments: harbor for the coding instance (`domains/coding/.venv`), and a Python 3.11 environment with `pip install -e ".[agentic]"` for the workspace and engineering instances (`RRSI_AGENT_PYTHON`). ### 1. LLM configuration The proposer, the analyst, the critic and the frozen policy are Claude Opus 4.8 on Vertex AI (`policy_model` in `domains/coding/rrsi.json`, `ORCHESTRATOR_MODEL` for the other two instances; any LiteLLM model string works). The Harvey LAB judge is Gemini 3.5 Flash. ```bash gcloud auth application-default login

已知限制

作者声明项目不是官方支持的 Google 产品，且不参与漏洞奖励计划；需要配置外部服务和模型。

查看 README 依据

This is not an officially supported Google product. This project is not eligible for the [Google Open Source Software Vulnerability Rewards Program](https://bughunters.google.com/open-source-security).