Interactive-Active-AI/NextTransformer

选用指南

AI 根据 README 整理 · 2026/11/5 16:47:04

适用场景

适用于在固定训练预算下比较不同 Transformer 架构，或通过提交插件加入新方法。

查看 README 依据

Parameters, memory and speed are outcomes, not controls. The [protocol](https://interactive-active-ai.github.io/NextTransformer/docs/protocol/overview/) explains every rule and the [compute accounting](https://interactive-active-ai.github.io/NextTransformer/docs/protocol/compute/) what is counted. ## Add your method New methods and Arena entries follow the same submission process. 1. [Open a proposal issue](https://github.com/Interactive-Active-AI/NextTransformer/issues/new/choose) to discuss the method and run plan.

**When the training budget is fixed, how should a language model spend its computation?** NextTransformer trains recurrent depth, sparse and linear attention, residual-stream variants and hybrids on the same data, for the same 49.152B tokens, at the same theoretical training FLOPs per token, and evaluates them at the same checkpoints. Every result is an audited, file-backed record, and AI agents can research new architectures under the very same rules. ## News

使用方式

通过命令行工具进行训练和评估，支持单机8卡或多节点训练。

查看 README 依据

cp site.native.example.json site.local.json # set runtime_dir, resources and data_dir hf download karpathy/climbmix-400b-shuffle --repo-type dataset \ --revision 915333b4f8b8684f39aeaafea600fea6f43fb703 --local-dir /path/to/climbmix-400b-shuffle source launch/env.sh --site-config site.local.json python -m nexttransformer.prepare --site-config site.local.json ``` **3. Train on one node with 8 GPUs:** ```bash launch/train.sh --config configs/baseline/dense_full.json --site-config site.local.json \ --output runs/dense_full --micro-batch 8 --formal --resume auto ```

**4. Multi-node training (2 nodes × 8 GPUs)** Install `uv pip install accelerate==1.15.0` in the training environment. Submit one distributed job with the following per-node entry point. Map the platform's node rank, main node address and port to the three variables shown: ```bash export FLA_DISABLE_BACKEND_DISPATCH=1 # registered FLA Triton path (linear and hybrid methods) accelerate launch --config_file launch/accelerate.yaml \ --machine_rank "$NODE_RANK" --main_process_ip "$MASTER_ADDR" --main_process_port "$MASTER_PORT" \

部署要求

需要 Linux、NVIDIA GPU、uv 和 Python 3.10-3.13，数据集约600GB。

查看 README 依据

<!-- /generated:leaderboard --> ## Quick start **1. Install** (Linux, NVIDIA GPUs, [uv](https://docs.astral.sh/uv/)): ```bash git clone https://github.com/Interactive-Active-AI/NextTransformer.git && cd NextTransformer uv sync --frozen --extra gpu --extra upstream && source .venv/bin/activate uv pip install --no-deps --require-hashes -r requirements/linear.txt # linear-attention kernels ``` **2. Prepare the data** (ClimbMix at a pinned revision, about 600 GB, plus the tokenizer and CORE bundle): ```bash

<img alt="Python 3.10-3.13" src="https://img.shields.io/badge/python-3.10--3.13-3776AB?style=flat-square&logo=python&logoColor=white"> <img alt="PyTorch 2.9.1" src="https://img.shields.io/badge/PyTorch-2.9.1-EE4C2C?style=flat-square&logo=pytorch&logoColor=white"> <a href="https://discord.gg/k2XPXW6Jz"><img alt="Discord" src="https://img.shields.io/badge/Discord-join-74638C?style=flat-square&logo=discord&logoColor=white"></a>

已知限制

Arena 仍在开发中，Arena v2 计划于2026年11月14日发布，当前仅支持部分功能。

查看 README 依据

<a href="https://interactive-active-ai.github.io/NextTransformer/docs/arena/overview/"><img src=".github/assets/readme/arena-en.webp" alt="Arena v2, by November 14, 2026: agent-native research, end to end" width="100%"></a> > [!NOTE] > **The arena is in active development.** Arena v2, with full support for autonomous agent research (a research gym > with MCP tools, metered compute, research traces and self-improving research loops), ships by **November 14, 2026**. Any LLM, with or without a research framework, can already enter an architecture today: it is trained under the same