HazyResearch/ncm

选用指南

AI 根据 README 整理 · 2026/10/26 00:56:21

使用方式

通过命令行运行评估脚本，需要下载检查点和数据集。

查看 README 依据

git clone https://github.com/HazyResearch/ncm.git cd ncm pip install -r requirements.txt ``` Python 3.10 or newer and a CUDA-capable PyTorch installation are recommended. ## Evaluation Download the checkpoints and evaluation inputs: ```bash hf download hazyresearch/ncm-checkpoints --local-dir artifacts hf download hazyresearch/ncm-datasets --repo-type dataset --local-dir data ``` Then run: ```bash python evaluate.py --config config/eval/<task>.yaml --checkpoint-dir artifacts/<task> --data-dir data/<task> # <task> is one of xsum, wmt14, sudoku-easy, sudoku-medium or sudoku-hard

部署要求

需要 Python 3.10 或更高版本，以及支持 CUDA 的 PyTorch 安装。

查看 README 依据

git clone https://github.com/HazyResearch/ncm.git cd ncm pip install -r requirements.txt ``` Python 3.10 or newer and a CUDA-capable PyTorch installation are recommended. ## Evaluation Download the checkpoints and evaluation inputs: ```bash hf download hazyresearch/ncm-checkpoints --local-dir artifacts hf download hazyresearch/ncm-datasets --repo-type dataset --local-dir data ``` Then run: ```bash python evaluate.py --config config/eval/<task>.yaml --checkpoint-dir artifacts/<task> --data-dir data/<task> # <task> is one of xsum, wmt14, sudoku-easy, sudoku-medium or sudoku-hard

已知限制

评估脚本在可见 GPU 上运行生成分片，并合并结果。

查看 README 依据

python evaluate.py --config config/eval/<dataset>.yaml --checkpoint-dir artifacts/<dataset> # <dataset> is tinystories, owt or lm1b-<size> with <size> one of small, medium or large ``` This runs one generation shard per visible GPU, merges them, and prints ROUGE-1/2/L and generative perplexity for XSum; cased and lowercased sacreBLEU, chrF and length ratio for WMT14; solved, cell and valid-grid rates for Sudoku; and generative perplexity under GPT-2 large with unigram entropy, over 5 seeds of 256 unconditional samples, for TinyStories, OpenWebText and LM1B, the latter under the FLM scoring