roboterax/video-prediction-policy-2

选用指南

AI 根据 README 整理 · 2026/10/16 22:16:36

使用方式

通过命令行和Python脚本使用，支持零样本视频预测和策略训练。

查看 README 依据

python -m pip install -r requirements-train.txt -c environment-reference.txt ``` The scripts load VPP2 directly from this checkout; installing the `vpp2` package is unnecessary. For inference and data processing only, use `requirements.txt`. ## Get started ### Zero-shot video prediction | Task | What is included | Guide | |---|---|---| | Video prediction | Input image processing, prompt enhancement, and single-image or batch video prediction with [sample inputs](assets/random_samples/). | [Video prediction guide](docs/video_prediction.md) | ### Policy learning

Our pipeline has three stages: (1) **event-level video pretraining**, (2) **single-step visual planning through post-training and distillation**, and (3) **action learning with a mixture-of-transformers (MoT) architecture**. This codebase includes: - **[Zero-shot video prediction](docs/video_prediction.md).** Generate videos from an initial image and a text instruction using the **Stage-1 (49 frames)** or **Stage-2 (17 frames)** video model, with input preparation, prompt enhancement, and ready-to-run examples.

部署要求

需要Python 3.10，并安装CUDA/PyTorch依赖。

查看 README 依据

<a href="assets/teaser.png"><img src="assets/teaser.png" alt="VPP2 paper overview: event-level video prediction, instruction following, and robot policy learning." width="100%"></a> </p> ## Installation ```bash git clone https://github.com/roboterax/video-prediction-policy-2.git cd video-prediction-policy-2 conda create -n vpp2 python=3.10 -y conda activate vpp2 ``` Install the CUDA/PyTorch dependencies using the [installation guide](docs/robodojo.md#policy-environment), then install the dependencies from the repository root: ```bash

已知限制

ModelScope模型下载需要账户访问权限。

查看 README 依据

| Benchmark | What is included | Guide | |---|---|---| | RoboDojo | Joint Video + Action2B training, checkpoint export, policy serving, and closed-loop evaluation | [RoboDojo guide](docs/robodojo.md) | | LIBERO | Video-stage training, action training, and evaluation on LIBERO, LIBERO-OOD, and LIBERO-PRO | [LIBERO guide](docs/libero.md) | ## Model Weights Weights are available on [Hugging Face](https://huggingface.co/Haodong082399/VPP2) and [ModelScope](https://modelscope.cn/models/haodong123/VPP2). The Hugging Face repository is public; ModelScope requires an account with access.