Overview

OpenTau is a PyTorch toolkit for training vision-language-action (VLA) models while fully leveraging the key techniques introduced in the \(\pi\)-series papers.

Motivation

The \(\pi\) series policies are by far the most popular models in today’s VLA community. While many training techniques have been described in the \(\pi\) papers, only a small fraction of these techniques have been implemented in open-source repositories.

Whether you use the official OpenPi codebase or LeRobot’s reimplementation, you are missing out on a lot of the key components that make the \(\pi\) models perform so well.

OpenTau (\(\tau\)) is a tool developed by Tensor to bridge this gap.

Key Features

OpenTau aims to make it easy to train VLAs on your own datasets while including the techniques that matter, such as:

  • Co-training on an adjustable mixture of heterogeneous datasets

  • Discrete actions for fast VLM convergence in \(\pi_{0.5}\)

  • Knowledge insulation between the VLM backbone and the action expert

  • Response prediction in \(\pi_{0.5}\) for training on VQA datasets and high level planning while inferencing.

  • Dropout layers in the VLM that prevent overfitting

  • A reinforcement learning pipeline described in \(\pi^*_{0.6}\)

  • Multi-node and multi-GPU training

  • Simulation environments for evaluating models

Quick Start

If you are familiar with LeRobot, getting started with OpenTau is very easy. Because OpenTau is a fork of the popular LeRobot repository, any LeRobot-compliant policy and dataset can be used directly with OpenTau.

Checkpoints

We provide fully functioning \(\pi_{0.5}\) checkpoints trained on the LIBERO dataset with high success rates.

  • TensorAuto/tPi0.5-libero: A \(\pi_{0.5}\) model checkpoint trained on the LIBERO dataset with discrete actions and knowledge insulation.

Acknowledgements

We would like to thank the original authors of the \(\pi\) series papers for their groundbreaking work in the VLA field.

We also acknowledge the contributions of the open-source community, especially LeRobot, for their efforts in re-implementing the \(\pi\) models and standardizing training infrastructure.

OpenTau builds upon these foundations to provide a more accessible and comprehensive tool for training vision-language agents.